Publishing a Book with Leanpub and Org Mode

Posted on 2017-04-09 by jcs

Angelo Basile has a nice post on publishing a book with Leanpub and Org Mode. I’ve written about this before when I discussed Lakshmi Narasimhan’s workflow with Leanpub. Narasimhan used Dropbox, which some may find inconvenient so Basile employs an alternative method that uses BitBucket (or GitHub if you prefer).

I’ve never published with Leanpub so I don’t know how they are to work with but you certainly couldn’t ask for an easier publishing pipeline. If you use Basile’s method, you simply

  • Set up a new book with Leanpub (they say it takes 30 seconds)
  • Set up a repository for the book on BitBucket or GitHub
  • Add a file, Book.txt, that tells Leanpub the names (and order) of your chapters
  • Write your book adding the file or files to Book.txt
  • Use Leanpub’s publishing mechanism to preview or publish your book with a single click

Leanpub wants their input in Markdown but Juan Reyero has an Org mode to Leanpub Markdown exporter that will take your Org files and output them in Leanpub compatible Markdown (Markua). You can keep both sets of files in your repository as long as Book.txt has only the names of the Markdown files in them.

Be sure to take a look at Narasimhan’s post too because he has some Elisp to automate almost everything but the writing. If you want to self publish, it’s hard to imagine an easier way of doing so.

Posted in General | Tagged , | Leave a comment

Hacker Archetypes

Posted on 2017-04-08 by jcs

Eric Raymond (ESR) has an interesting post on hacker archetypes. The list grew out of a discussion between ESR and one of his regular readers, Susan Sons. At the moment, they’ve identified 8 archetypes. You’ll want to read his post to see what they are and where you fit in but here’s an example to give you the flavor.

Algorithmicists:

  • Good at algorithms and intricate coding
  • High tolerance for complexity
  • Good mathematical intuition
  • Likes correctness proofs and thinks in terms of invariants
  • Tends to gravitate to compiler writing and crypto
  • Often has poor social skills
  • Tends to fail by excessive cleverness

That’s just the first archetype; there are 7 more and all are described in more detail than just the list of traits I gave above. ESR says that no hacker is only one of the archetypes. He describes himself as exhibiting the traits of three of them although one does dominate. That’s probably true of most of us.

Head on over to Armed and Dangerous and see which archetypes apply to you.

Posted in General | Tagged | Leave a comment

A Simple Programming Praxis Solution in Elisp

Posted on 2017-04-07 by jcs

Programming Praxis has a simple problem that is meant to explore how easy your favorite programming language makes it to work with strings. The problem is to double every blank in a string. That’s pretty simple but some languages make you deal with string allocation or perhaps, in the case of languages with immutable strings, repeatedly allocate new strings.

I have two solutions for this using Elisp. The first is a sort of direct solution that doesn’t think too hard about the special functions Emacs has for dealing with strings. 

(defun double-blanks (string)
  (with-temp-buffer
    (dolist (ch (string-to-list string))
      (insert ch)
      (if (eql ch ? ) (insert ? )))
    (buffer-string)))

This code deals with the fact that strings in Elisp are immutable by doing all the work in a temporary buffer as I discussed in this post. Except for the temporary buffer trick, it’s pretty much what any programmer would write. Of course, that string-to-list call is going to do a lot of consing, so for a really long string it might make sense to get the string characters by hand.

Of course, this is Emacs so there’s a prebuilt solution. 

(defun double-blanks (string)
  (replace-regexp-in-string " " "  " string))

The replace-regexp-in-string function works by building up a list of matches and combining the list into a single string at the end so it also avoids the problem a immutable strings. If there are a lot of blanks, replace-regexp-in-string is going to do a lot of consing too.

If you’re dealing with large strings with lots of blanks, the best strategy is probably to combine the two approaches above: use a regular expression search for the blanks and build the output string in a buffer as you find them. That wouldn’t be hard and makes a good exercise for anyone interested in learning Elisp.

UPDATE [2017-04-10 Mon 13:49]: In the comments Phil remarks that string-to-list isn’t very efficient and suggests another strategy. I replied that it would be interesting to see if my (unimplemented) strategy or his would be more efficient and said that perhaps I’d try them out and report back. Phil, being Phil, took care of that before I could get around to it and go some interesting results.

For some reason his main post with the results isn’t showing even though it appears in the recent comments bar. Here’s the meat of that comment. I have to agree with Phil that it’s a bit surprising that the split-string method was the winner but that goes to show that one’s intuition about these things is most often wrong. 

(length teststr)    
2070    

(length (split-string teststr " "))    
553    

(benchmark-run 10000 (double-blanks1 teststr)) ;; string-to-list    
(28.203663436 967 6.414597634999999) ;; uncompiled    
(23.110737061000002 968 8.26656119499986) ;; compiled    

(benchmark-run 10000 (double-blanks2 teststr)) ;; replace-regexp-in-string    
(21.614643389999998 1913 11.981117643999973) ;; uncompiled    
(25.317938649 1913 15.727532714000041) ;; compiled    

(benchmark-run 10000 (double-blanks3 teststr)) ;; search-forward/insert    
(8.588276816 149 0.9523469339999999) ;; uncompiled    
(8.355207331 147 1.2536677860000367) ;; compiled    

(benchmark-run 10000 (double-blanks4 teststr)) ;; split-string/mapconcat    
(7.9514880880000005 476 3.2033875379999657) ;; uncompiled    
(7.953855983 477 3.237625087999996) ;; compiled
Posted in Programming | Tagged , | 6 Comments

Mu4e Configuration

Posted on 2017-04-06 by jcs

This post is the latest in my series on installing and configuring mu4e on a Mac and using Apple as a mail provider. As I wrote in my post on mbsync configuration, setting up mu4e is simple by comparison. On my MacBook Pro, I compiled the mu package from source. On my iMac I installed it with homebrew. Either way works fine. The advantage of homebrew is that it will load all the libraries that you need but aren’t included in the base macOS install.

My configuration is an adaption of Charl Botha’s, which uses some of the features in the latest release, with the addition of some ideas from Ben Maughan and a few bits of my own: 

  1: (require 'mu4e)
  2: 
  3: ;; Call EWW to display HTML messages
  4: (defun jcs-view-in-eww (msg)
  5:   (eww-browse-url (concat "file://" (mu4e~write-body-to-html msg))))
  6: 
  7: ;; Arrange to view messages in either the default browser or EWW
  8: (add-to-list 'mu4e-view-actions '("ViewInBrowser" . mu4e-action-view-in-browser) t)
  9: (add-to-list 'mu4e-view-actions '("Eww view" . jcs-view-in-eww) t)
 10: 
 11: ;; From Ben Maughan: Get some Org functionality in compose buffer
 12: (add-hook 'message-mode-hook 'turn-on-orgtbl)
 13: (add-hook 'message-mode-hook 'turn-on-orgstruct++)
 14: 
 15: ;; Call mu every 5 minutes to update and index Maildir
 16: (setq mu4e-update-interval 300)
 17: 
 18: ;; Set format=flowed
 19: ;; mu4e sets up visual-line-mode and also fill (M-q) to do the right thing
 20: ;; each paragraph is a single long line; at sending, emacs will add the
 21: ;; special line continuation characters.
 22: (setq mu4e-compose-format-flowed t)
 23: 
 24: ;; every new email composition gets its own frame! (window)
 25: ;;(setq mu4e-compose-in-new-frame t)
 26: 
 27: ;; give me ISO(ish) format date-time stamps in the header list
 28: (setq mu4e-headers-date-format "%Y-%m-%d %H:%M")
 29: 
 30: ;; show full addresses in view message (instead of just names)
 31: ;; toggle per name with M-RET
 32: (setq mu4e-view-show-addresses 't)
 33: 
 34: ;;store org-mode links to messages
 35: (require 'org-mu4e)
 36: ;;store link to message if in header view, not to header query
 37: (setq org-mu4e-link-query-in-headers-mode nil)
 38: 
 39: ;;rename files when moving
 40: ;;NEEDED FOR MBSYNC
 41: (setq mu4e-change-filenames-when-moving t)
 42: 
 43: ;; Try to show images
 44: (setq mu4e-view-show-images t
 45:       mu4e-show-images t
 46:       mu4e-view-image-max-width 800)
 47: 
 48: ;; path to our Maildir directory
 49: (setq mu4e-maildir "~/Maildir")
 50: 
 51: ;; the next are relative to `mu4e-maildir'
 52: ;; instead of strings, they can be functions too, see
 53: ;; their docstring or the chapter 'Dynamic folders'
 54: (setq mu4e-sent-folder   "/icloud/Sent Messages"
 55:       mu4e-drafts-folder "/icloud/Drafts"
 56:       mu4e-trash-folder  "/icloud/Trash")
 57: 
 58: ;; the maildirs you use frequently; access them with 'j' ('jump')
 59: (setq   mu4e-maildir-shortcuts
 60:         '(("/icloud/Archive"             . ?a)
 61:           ("/icloud/inbox"               . ?i)
 62:           ("/icloud/Saved"               . ?v)
 63:           ("/icloud/Sent Messages"       . ?s)))
 64: 
 65: ;; the list of all of my e-mail addresses
 66: (setq mu4e-user-mail-address-list '("XXX@mac.com"
 67:                                     "YYY@irreal.org"))
 68: 
 69: ;; the headers to show in the headers list -- a pair of a field
 70: ;; and its width, with `nil' meaning 'unlimited'
 71: ;; (better only use that for the last field.
 72: ;; These are the defaults:
 73: (setq mu4e-headers-fields
 74:     '( (:date          .  25)    ;; alternatively, use :human-date
 75:        (:flags         .   6)
 76:        (:from          .  22)
 77:        (:subject       .  nil))) ;; alternatively, use :thread-subject
 78: 
 79: ;; Program to get mail.
 80: ;; Called when 'U' is pressed in main view.
 81: 
 82: ;; If you get your mail without an explicit command,
 83: ;; use "true" for the command (this is the default)
 84: ;; when I press U in the main view, or C-c C-u elsewhere,
 85: ;; this command is called, followed by the mu indexer
 86: (setq mu4e-get-mail-command "mbsync icloud")
 87: 
 88: ;; not using smtp-async yet
 89: ;; some of these variables will get overridden by the contexts
 90: (setq
 91:  send-mail-function 'smtpmail-send-it
 92:  message-send-mail-function 'smtpmail-send-it)
 93: 
 94: ;; don't keep message buffers around
 95: (setq message-kill-buffer-on-exit t)
 96: 
 97: ;; Contexts: One for each mail personality.
 98: (setq mu4e-contexts
 99:       `( ,(make-mu4e-context
100:            :name "a XXX@mac.com"
101:            :enter-func (lambda () (mu4e-message "Enter XXX@mac.com context"))
102:            :leave-func (lambda () (mu4e-message "Leave XXX@mac.com context"))
103:            ;; Match based on the contact-fields of the message (that we are replying to)
104:            :match-func (lambda (msg)
105:                          (when msg 
106:                            (mu4e-message-contact-field-matches msg 
107:                                                                :to "XXX@mac.com")))
108:            :vars '( ( user-mail-address      . "XXX@mac.com"  )
109:                     ( user-full-name         . "Jon Snader" )
110:                     ( smtpmail-smtp-server   . "smtp.mail.me.com" )
111:                     ( smtpmail-smtp-service  . 587)
112:                     ( smtpmail-stream-type   . starttls)))
113: 
114:          ,(make-mu4e-context
115:            :name "i YYY@irreal.org"
116:            :enter-func (lambda () (mu4e-message "Enter YYY@irreal.org context"))
117:            :leave-func (lambda () (mu4e-message "Leave YYY@irreal.org context"))
118:            ;; we match based on the contact-fields of the message
119:            :match-func (lambda (msg)
120:                          (when msg 
121:                            (mu4e-message-contact-field-matches msg 
122:                                                                :to "YYY@irreal.org")))
123:            :vars '( ( user-mail-address       . "YYY@irreal.org" )
124:                     ( user-full-name          . "Jon Snader" )
125:                     ( smtpmail-smtp-server    . "smtp-server.tampabay.rr.com" )
126:                     ( smtpmail-stream-type    . plain)
127:                     ( smtpmail-smtp-service   . 25)))))
128: 
129: 
130: ;; start with the first (default) context; 
131: (setq mu4e-context-policy 'pick-first)
132: 
133: ;; compose with the current context if no context matches;
134: (setq mu4e-compose-context-policy nil)
135: 
136: (global-set-key (kbd "C-<f6>") 'mu4e)

As you can see, it’s very similar to Botha’s. Some emails, especially commercial ones (like those from Amazon) and those from professional groups like ACM, don’t render very well so I prefer to view them in a browser environment. Usually, EWW is sufficient but sometimes I need the full Safari treatment. Lines 3–9 show how to do that. If I press a while viewing a message, it will bring up a menu with various viewing options. If I choose E the message is displayed in EWW. V displays it in the default browser, Safari in my case.

The other noteworthy thing about the configuration are the contexts, which are configured on lines 97–134. All my emails are filtered through the Apple server so the contexts allow me to send mail either from my Apple account or my Irreal account. When I respond to an email, the correct “From:” address is chosen from the “To:” line of the original message.

Finally, there’s the matter of sending messages. Many writers recommend third party packages to handle multiple accounts but I’ve found the contexts in mu4e handles that nicely. Unless you have special needs, like Ben Maughan, using the built-in smtpmail functionality is fine. You’ll want to set up an .authinfo file to list the servers, accounts, and passwords. The line for the apple server looks like 

machine smtp.mail.me.com login XXX@mac.com password same-password-as-for-mbsync

You’ll want to encrypt the file for security, but Emacs only asks for the password one time per (Emacs) session so that’s not burdensome.

Posted in General | Tagged , | Leave a comment

Configuring mbsync for Apple Mail

Posted on 2017-04-05 by jcs

As a first step in realizing my goal of running mu4e as my email client, I had to get mbsync running. Mbsync is a process that communicates with one or more IMAP servers to keep your email clients in sync. My primary email is through Apple so one of my goals was to be able to use mu4e on my Macs but still have fully synced access to mail on my iPhone and iPad. That’s pretty easy because Apple mail is IMAP based and keeping multiple clients in sync is the point of IMAP.

There are two problems to solve for getting mbsync working on the Mac. The first is specifying your mail server password to mbsync. You could, of course, just put the password in your mbsync configuration file but that’s horribly insecure and, happily, unnecessary. Here’re the directions for setting up a password in your keychain from the sample configuration that comes with mbsync. 

# On Mac OS X, run "KeyChain Access" -- File->New Password Item. Fill out form using
#  "Keychain Item Name" http://IMAPSERVER  (note: the "http://" is a hack)
#  "Account Name" USERNAME
#  "Password" PASSWORD

That brings us to the second problem: what to use for a password. You might think your icloud password is what’s needed here but if you have two-factor authentication enabled—which you almost certainly do if you’re running Sierra or later—you need to generate an application specific password. You can do that by following these instructions. Failure to understand this point was the source of almost all my problems in getting things working.

I keep all my saved emails in a single folder for reasons I’ve discussed before. Your setup may be a little different but if so you just need to adjust the list of mailboxes on the Patterns line in the configuration below. I added an icloud subdirectory under Maildir because I intend to add a separate mail repository for Gmail later. Right now, the configuration communicates only with the Apple mail server.

Here is my .mbsyncrc file. It normally lives in your home directory. 

# Based on http://www.macs.hw.ac.uk/~rs46/posts/2014-01-13-mu4e-email-client.html
IMAPAccount icloud
Host imap.mail.me.com
User XXX #not XXX@me.com etc.
PassCmd "security find-generic-password -s mbsync-icloud-password -w"
Port 993
SSLType IMAPS
SSLVersions TLSv1.2
AuthMechs PLAIN

IMAPStore icloud-remote
Account icloud

MaildirStore icloud-local
Path ~/Maildir/icloud/
Inbox ~/Maildir/icloud/inbox
Trash Trash

#
# Channels and Groups 
# (so that we can rename local directories and flatten the hierarchy)
#
#
Channel icloud-folders
Master :icloud-remote:
Slave :icloud-local:
Patterns "INBOX" "Saved" "Drafts" "Archive" "Sent*" "Trash"
Create Both
Expunge Both
SyncState *

Group icloud
Channel icloud-folders

Once you get everything set up you can try retrieving your mail with 

mbsync -a

or, if you have more than one mail server configured 

mbsync icloud #or whatever your mail group name is

In the case of Apple, the server doesn’t like users sucking up bandwidth so if you have a large repository on the server, it may drop off after a while. That’s not fatal because you can just restart the download and it will take up where it left off. You can help a bit by rate limiting your network I/O as described by Ben Maughan in this post. That post also has a nice configuration for Gmail if that’s what you need. After the initial download everything will work smoothly.

Shortly, I’ll publish my next post in this series about mu4e with a discussion of my mu4e configuration.

Posted in General | Tagged , | 1 Comment

Summer!

Posted on 2017-04-04 by jcs

As many of you know, the Irreal International Headquarters is in Tampa, Florida. That means that this Kontra tweet resonated:

Kontra is, I think, based in New York City. He should try a Florida summer.

Posted in General | Tagged | Leave a comment

Mu4e at Irreal

Posted on 2017-04-03 by jcs

Ever since I read Ben Maughan’s post on mastering your inbox with mu4e and Org mode back in 2015, I’ve been insanely jealous of his method and of his use of mu4e for dealing with emails directly in Emacs. Right after I read his post, I combined all my email folders into one and started following his procedures for dealing with emails.

Still, it was a bit clumsy with Apple’s mail app and the searching involved a lot of mouse clicking. And, of course, it wasn’t happening inside of Emacs. Now, finally, I have mu4e up and running and am using it as my mail client. The search capability is amazing: it’s lightning fast and very flexible. You can form complex queries without touching the mouse or being limited to canned searches. The best part is that because it’s all IMAP based my old client and more importantly, my iPhone and iPad mail apps still work and keep everything in sync.

Setting up mu4e turned out to be pretty easy. The tough part was getting mbsync to work with Apple. That’s mostly because I forgot about the two-factor authentication that Apple recently added in the last macOS release so you can chalk up the difficulties to my stupidity.

In the near future, I will post my mbsync configuration and what I had to do to get it to talk to Apple’s IMAP server. That may be useful because it’s really hard to find examples of setting it up with Apple. After that, I will write a post on my mu4e setup and configuration, although most of that was stolen from Ben and Charl Botha.

Next up is moving my RSS to elfeed. Fortunately, Mike Zamansky has three excellent videos on that so it shouldn’t take nearly as long. After that, browsing and iMessage will be my only frequently used applications that don’t run under Emacs. I don’t see much hope for moving either of them to Emacs anytime soon so after elfeed my workflow will be pretty much as Emacs-centric as it’s going to get.

Posted in General | Tagged , | 3 Comments

Updating save-receipt-link

Posted on 2017-04-02 by jcs

This is tax season here in the United States. My Org-based tax record keeping enabled me to send everything to my accountant as soon as I had all the proper forms from employers, banks, and other sources of income to the vast Irreal empire. As usual, once I’d finished with all the paperwork, I reviewed my procedures to see if there was anything I could do to reduce friction and make things easier.

Five years ago, I wrote about one such improvement: a bit of Elisp to save links to scanned receipts in my Org files. Here, for reference, is that Elisp from the above link: 

(defconst current-tax-file "tax12")

(defun save-receipt-link (doc)
  "Make a link to a receipt in the current tax file."
  (interactive "sDocument Name: ")
  (insert (concat "[[file:~/tax/" current-tax-file "/receipts/" doc
                  "][receipt]]")))

The save-receipt-link function worked well and I’ve been using it throughout the last 5 years as I get receipts that I need to save for tax purposes. There are, however, a couple of problems with it. The first is that I often get the file name of the scanned receipt wrong. Most often this involves leaving off the .pdf at the end. I could solve most instances of that problem by just programmatically adding the .pdf but some documents have a different extension and sometimes I just mistype the name so appending the .pdf doesn’t really solve the problem.

The other problem appeared at the beginning of the year. I was still getting receipts from the previous year but also getting receipts for the current year. That meant I had to hand-edit the receipt links.

Obviously, since I’ve been using this method for 5 years, neither of these problems is fatal but they do inject a bit of friction. This year I decided to fix things. I did that by changing save-receipt-link to be 

(defconst prev-tax-file "~/tax/tax16/receipts")
(defconst current-tax-file "~/tax/tax17/receipts")

(defun save-receipt-link (prevp)
  "Make a link to a receipt in the current or previous tax file.
Use the previous tax file if called with the universal argument."
  (interactive "P")
  (insert (format "[[file:%s][receipt]]"
                  (completing-read "Document: "
                                   (directory-files
                                    (if prevp prev-tax-file current-tax-file) t)))))

By adding the completing-read, I get a list of the files and merely have to pick the correct one. That makes the entry faster while eliminating the errors I kept making with the old version.

If I call save-receipt-link with the universal argument, I will use last year’s directory instead of the current year’s. That eliminates the confusion at the change of year.

None of this is going to make dealing with taxes pleasurable, of course, but it does make it a little easier.

Posted in General | Tagged | Leave a comment

AI Loosely Defined

Posted on 2017-04-01 by jcs

Since this is April 1st and no post will be taken seriously anyway, I offer this piece of wisdom, which Jean-Philippe Paradis retweeted:

Posted in General | Tagged , | Leave a comment

The Emacs rx Macro

Posted on 2017-03-31 by jcs

I’ve long been aware of the rx macro in Emacs as a way of lispifying regular expressions but it always seemed too complicated to learn. Now Francis Murillo (I think) has published an excellent post on using the rx macro. The post has an explanation of the syntax and several worked examples.

If you are one of those people who have a hard time writing or reading regular expressions, you should take a look. It may make writing those regular expressions easier, especially if you feel comfortable with Elisp.

Posted in General | Tagged | Leave a comment