Putting Shell Output In The Current Buffer

I’ve long used 【Meta+|】 to pipe a region to a shell command. I use it all the time to pipe troff input to groff and display the results with gxditview. Recently I learned that if you specify a prefix arg it will place the results in the current buffer. The easiest way to do that is simply to type 【Meta+1 Meta+|】. The same trick works with 【Meta+!】. For example, if I want to insert a directory listing into the current buffer I could type 【Meta+1 Meta+!ls.

Summary of Key Sequences

  • Meta+|
    Pipe the highlighted region to a shell command and place the output in a ∗Shell Command Output∗ buffer.
  • Meta+1 Meta+|
    Pipe the highlight region to a shell command and place the output in the current buffer.
  • Meta+!
    Run a shell command and place the output in a ∗Shell Command Output∗ buffer.
  • Meta+1 Meta+!
    Run a shell command and place the output in the current buffer.

Update: prefix tag → prefix arg

Posted in General | Tagged | 3 Comments

The New Luddites Redux

It’s been a long time since I’ve written about the New Luddites—the last time was early on in my old blog (1, 2, 3)—but sadly their silliness has surfaced again. I follow Shelly Palmer’s Digital Living blog, which is geared towards the non-technical user of consumer electronics but often has interesting posts. Yesterday he reported on a CNN article that discussed smartphone users obsessively checking their phones. That article, in turn, mentioned a study published in the Journal of Personal and Ubiquitous Computing.

Palmer begins by poking fun at the study for belaboring the obvious but then goes on to take its conclusions seriously and offer “solutions” for our addictive behavior. The study itself is not particularly compelling. Its methodology is ad hoc and its sample sizes are ludicrously small (the largest of the 3 studies had 36 participants, another 12, and the last studied 3 groups with single digit participants).

What are the study’s results that have so alarmed Palmer? The participants checked their smartphones an average of 34 times a day. Really? Assuming they were sleeping 8 hours a day that amounts to checking their phones every half hour. That’s addictive behavior?

First of all, it’s silly to use the word addictive with its overtones of alcoholism or hard drug use. Those habits clearly are harmful both physically and emotionally. The most harmful thing pointed to by the CNN article was that one man’s frequent checking of his smart phone annoyed his wife. Many people today choose to live a digital life: they read ebooks, get their bills as emails, keep electronic diaries, maintain friendships through social media, use Google as a sort of associative memory, and most of all stay connected all the time. The device that enables this lifestyle is the smartphone. Of course these people check their phones frequently.

To devise tactics to reduce smartphone use as Palmer suggests makes no sense except as a bizaro world yes-I-can type of experiment. Twenty years ago the equivalent exercise would have been to go without electricity and air conditioning for a week just to prove you could. Just as it was silly to worry about becoming habituated to electricity and A/C, it’s silly to forgo the advantages of smartphones on the grounds that using them twice an hour is addictive behavior. The New Luddites are always with us, of course, but I was surprised and saddened to see the author of Overcoming the Digital Divide: How to use Social Media and Digital Tools to reinvent yourself and your career give them aid and comfort.

Update: bizzaro → bizaro

Posted in General | Leave a comment

Scrolling With Lion

OS X 10.7 (Lion) has been out for about a week and although that’s probably not enough time for any subtle bugs to surface, it is, apparently, enough time for my natural paranoia to be overcome by my desire to give it a spin. So over the last couple of days I’ve updated my machines with Lion—the upgrades went extraordinarily smoothly; thanks for asking.

I’m still getting used to the OS so I don’t have much of substance to say right now but I can comment on one issue that’s getting mentioned a lot in the reviews that I’ve read: scrolling. If you’ve been paying any attention to Lion, you know by now that Apple has “reversed” the scrolling. Swiping down with two fingers used to scroll towards the bottom of the page; now it scrolls towards the top of the page. It’s easy to reinstate the old behavior with the System Preferences pane but that hasn’t stopped the howls of anguish—not to say outrage—from many of the commenters.

The idea is that the new behavior mimics what happens with iOS when you scroll: you’re grabbing the screen and pushing it up or down. Despite having a small problem with muscle memory, I like the change. The old behavior never made sense to me and only muscle memory and the fact that it had always been that way kept me from being totally confused. If I thought too hard about it I would do the “wrong” thing. After only a few hours the new scrolling already feels natural. Sometimes I start to move in the wrong direction but that is already starting to die out. If you’re having trouble with the change, just imagine that you are putting your fingers on the text and pushing it in the direction you want to go.

I’m sure there’ll be lots of kvetching and dark mumbling about Apple oppressing their users and upsetting the natural order of things but, really, all Apple has done is finally have scrolling make sense. And, you know, if you really, really don’t like it, it’s easy to change it back.

Posted in General | Tagged | Leave a comment

Using set-mark-command To Remember Locations

A couple of days ago I wrote a note to myself post to help me remember the align-regexp functionality. Today’s post is another note to myself. A recent post by Xah Lee reminded me of some Emacs functionality that I am always forgetting. Every time I see the set-mark-command function described I think, “Boy, that’s really useful, I need to remember it.” Then I promptly forget about it again.

The set-mark-command (bound to 【Ctrl+Space】) sets the mark at the point’s position. The normal way of invoking it to remember locations is to type 【Ctrl+Space Ctrl+Space】. The first 【Ctrl+Space】 sets the mark and the second deactivates it so that no region is highlighted. Emacs also pushes the mark onto the buffer’s mark ring and onto the global mark ring.

This is useful because we can use the marks recorded on the mark rings to return to a remembered location. If we want to return to a previous mark in the same buffer, we type 【Ctrl+u Ctrl+Space】 We can move back several locations by repeating the 【Ctrl+u Ctrl+Space】.

If we want to return to a mark which may be in another buffer we can type 【Ctrl+x Ctrl+Space】. This can also be repeated to move back several marks.

This may seem confusing but it boils down to just three key sequences

  • Ctrl+Space Ctrl+Space】 to push the current position onto the mark rings.
  • Ctrl+u Ctrl+Space】 to return to a previous position in the same buffer.
  • Ctrl+x Ctrl+Space】 to return to the previous position, which may or may not be in the same buffer.
Posted in General | Tagged | 6 Comments

More On Borders’ Closing

David Hill over at Singularity Hub has an interesting take on the closing of Borders. While agreeing that Borders’ downfall is a shame, Hill argues that it will actually be a huge win for authors and readers. It will benefit authors, he says, by accelerating the changes in the publishing industry that are putting the authors in control. Those same changes will hasten the move to digital books with all the advantages that brings to readers.

Hill’s post has a link to a post by Alan Rinzier, an industry insider, that is also interesting. Rinzier confirms the trend of greater author control and says, basically, that the publishing industry has no idea what it’s doing and is coming to realize that. As evidence for this, he reports that 80–90 percent of all published books lose money.

Rinzier says that while it’s as hard as ever to write a good book, authors who can now have the opportunity to self publish and control their own destiny. As I suggested in my Future of Books post, the publishing industry is undergoing great change but appears to remain in denial. If they wish to avoid Borders’ fate, they had better get a clue. They still provide a valuable service but they are no longer the gatekeepers that they once were. If their tactics to defend their business models become too annoying, authors and readers will simply route around them and they will learn first hand the meaning of disintermediation. As I also said in that post, I hope that doesn’t happen but I see little reason to be sanguine about their prospects.

Posted in General | Leave a comment

Aligning Text With Regular Expressions in Emacs

This is really a note to myself. A while ago I ran across this post over at RAW SYNTAX about the Emacs command align-regexp. I don’t require this functionality very often but when I do, align-regexp is just what I need. Unfortunately, I keep forgetting about it so I’m writing this post in the hopes that it will help me remember (or at least give me a place to find the name of the function)

The simplest, and probably most common, use case is to align on a particular character. The canonical example is a telephone number list. We just mark the region to align, type 【Meta+xalign-regexp, and then ( when prompted for the regular expression

# Original list

Joe (555) 111-2222
Mary  (555) 222-1212
Alexandra 123-4567
Francis (555) 333-2121

# After align-regexp

Joe     (555) 111-2222
Mary    (555) 222-1212
Alexandra 123-4567
Francis (555) 333-2121

That’s not quite what we want, though, because Alexandra’s number doesn’t have an area code so there is no open parenthesis to align on. To fix this, we have to use the more complicated version of align-regexp by giving the invocation a prefix arg: 【Ctrl+u Meta+xalign-regexp. This time we specify .+?\( +\).+ as the regular expression, choose 1 as the group to modify, 1 as the amount of spacing, and no as to whether we should repeat the rule throughout the line. The result is:

Joe       (555) 111-2222
Mary      (555) 222-1212
Alexandra 123-456
Francis   (555) 333-2121

That’s better but what we really want is to align on the -. If we do that directly using the simple case, we get

Joe (555) 111     -2222
Mary  (555) 222   -1212
Alexandra 123     -4567
Francis (555) 333 -2121

which is definitely not what we want. Instead we use the regular expression .+? +\(.*\) and specify -1 as the group to modify or justify. That gives us

Joe        (555) 111-2222
Mary       (555) 222-1212
Alexandra        123-4567
Francis    (555) 333-2121

which is just what we want.

Update: lower case the key sequences.

Update 2: prefix tag → prefix arg

Posted in General | Tagged | 2 Comments

Traveling Without A Computer

Heh heh. OK, that was a teaser title. Of course, I would never go anywhere without a computer; my days of being ABEND are long over. My normal routine is to take my MacBook Pro and associated equipment with me in a backpack. The MacBook essentially mirrors my main iMac machine so I can do anything on the road that I can do at home. Still, flying with the laptop is a hassle. I use the excellent SwissGear ScanSmart Backpack so at least I don’t have to take the computer out of the backpack but it’s still one more thing to deal with at the TSA checkpoints.

Last week, I was in New York visiting family. I decided to try leaving the MacBook at home and depending on my iPad instead. Naturally that precluded any development or serious writing but this was a family visit so, realistically, I wouldn’t be doing much of either. I could have tried depending on my iPhone and avoiding large computers altogether but I wanted to keep up with blogs and news and it’s just too painful to read that sort of thing on the iPhone’s small screen.

Traveling with the iPad is definitely easier. I have a small case for it and the only support equipment it requires is the charging cable. I always travel with the wonderful Scottevest Travel Vest which, with its 22 pockets, has plenty of room for the cable, my iPhone, keys, and all the other stuff I would normally throw in the backpack. When I got to the checkpoint, my shoes and iPad went into a bin and I put the travel vest on top. Just one bin and the iPad is effectively hidden from anyone tempted to get a free computer as they come out of the X-ray machine.

As I’ve mentioned before, I keep up with my feeds with Reeder, which has a very nice and easy to use iPad implementation so it was easy staying current with blogs and news. The iPad has a nice email client that I use for my main email inbox. Mail from mailing lists and things like that go to a gmail account and gmail on the iPad is so good that I prefer to read gmail on the iPad even when I’m home.

The only other thing I needed to do was to manage my blog. I queued up a couple of posts so I didn’t have to worry about writing any entries while I was on the road (I do all that in Emacs anyway so the iPad is not a place to write posts). I was able to moderate forum spam and respond to legitimate comments easily.

Touch typing just isn’t going to happen on the iPad, but I can do a bit better than hunt and peck on the virtual keyboard so responding to emails and blog comments was relatively painless. I suppose I could get a portable keyboard to use with the iPad but, honestly, I don’t feel the need. For the little amount of typing I do on it, the virtual keyboard is fine.

All in all the experiment was a success. Traveling with the iPad was easier than bringing along the MacBook. The few chores I needed to perform were easily handled on the iPad so I didn’t miss having the MacBook. And as a bonus, I didn’t have to carry a book to read on the plane—I just fired up iBooks and enjoyed an ebook.

Update: Over at IT World Brian Proffitt has a post on using (only) an iPad for 7 days.

Posted in General | Tagged | Leave a comment

Xah’s Challenge (Part 2)

Yesterday, I wrote about Xah Lee’s programming challenge and gave a solution similar to Lee’s own. Today I will give another, possibly simpler, solution. We start with some constants and a couple of helper functions:

(defconst brackets '((?( . ?))
                     (?{ . ?})
                     (?[ . ?])
                     (?“ . ?”)
                     (?‹ . ?›)
                     (?« . ?»)
                     (?【 . ?】)
                     (?〈 . ?〉)
                     (?《 . ?》)
                     (?「 . ?」)
                     (?『 . ?』)))

(defconst not-brackets "^(){}[]“”‹›«»【】〈〉《》「」『』")

(defvar filen nil)

(defun brkt-closes-p (brkt obrkt)
  "Predicate to test if brkt closes obrkt."
  (let ((bpair (assoc obrkt brackets)))
    (and bpair (char-equal brkt (cdr bpair)))))

(defun brkt-open-p (brkt)
  "Predicate to test if brkt is an opening bracket."
  (assoc brkt brackets))

As before, we have a list of the beginning and ending brackets but this time they are given as characters rather than strings. The not-brackets constant will be used in a call to skip-chars-forward. The ^ at the beginning of the string specifies all characters except the brackets.

The brkt-closes-p function is a predicate that tests whether its first argument is the closing bracket for the second argument. It just uses the brackets constant as an alist.

The brkt-open-p function is even simpler. It’s a predicate that tests whether its first argument is an opening bracket. That will be true if it’s found in a key position of the brackets alist.

As in the first solution, check-file-for-mismatches gets called to check a single file but all the work is done in brkt-match. The check-file-for-mismatches function merely sets up the temporary buffer for the file and calls brkt-match.

(defun check-file-for-mismatches (fpath)
  "Check a file for mismatched brackets."
  (let ((fb (get-buffer-create "*Temp*")))
    (set-buffer fb)
    (insert-file-contents fpath nil nil nil t)
    (goto-char 1)
    (setq filen fpath)
    (brkt-match ?\f)
    (kill-buffer fb)))

The real checking is done in brkt-match:

 1:  (defun brkt-match (obrkt)
 2:    "Match OBRKT with its closing bracket."
 3:    (let ((open-pos (1- (point))))
 4:      (catch 'exit
 5:        (while t
 6:          (skip-chars-forward not-brackets)
 7:          (if (not (eobp))
 8:              (forward-char))
 9:          (cond
10:           ((eobp) (if (not (char-equal ?\f obrkt))
11:                       (print (format "Unclosed %c in %s at %d"
12:                                      obrkt filen open-pos)))
13:            (throw 'exit t))
14:           ((brkt-closes-p (char-before) obrkt) (throw 'exit t))
15:           ((brkt-open-p (char-before)) (brkt-match (char-before)))
16:           (t (print (format "Mismatch in %s at %d" filen (1- (point))))))))))
17:  

The idea is that brkt-match is passed an opening bracket and searches for the closing bracket (the ?\f passed in by check-file-for-mismatches is a special marker indicating the beginning of file—it matches end-of-buffer). Rather than locate the brackets with a regular expression, we find them by skipping over everything else with skip-chars-forward on line 6.

The processing is done in the while loop on lines 5–16. The catch on line 4 is so that we can exit the while if we match the input or reach end-of-buffer. The cond on lines 9–16 makes the checks:

  1. Lines 10–13
    If we are at end-of-buffer and the input was not ?\f then output an error message about an unclosed bracket. In any case, exit the loop and therefore the function.
  2. Line 14
    If the current character closes the input character, exit the loop and function.
  3. Line 15
    If this is an opening bracket, recursively call brkt-match to try to match it.
  4. Line 16
    If the other tests fail then this is a mismatched bracket. Output an error message and keep looking for the closing bracket matching the input.

We can use the exact same do-files from the first solution to drive check-file-for-mismatches so I won’t repeat it here.

This solution appears radically different from the first but it isn’t really; only the details differ. We use characters instead of strings, skip-chars-forward instead of regular expressions, and recursion instead of an explicit stack but the strategy is the same:

  • Search for the match to an opening bracket
  • If we find another open bracket, save what we’ve done so far and start over with the new opening bracket
  • When we match an opening bracket, pick up where we left off.

It’s an interesting question as to which method is faster. I haven’t tested that aspect but I’m pretty sure the second method is faster. It can do simple character compares instead of the more complex string compares and regular expressions have to do at least as much work as skip-chars-forward.

Update: test → tests. Fixed line numbers in text.

Posted in Programming | Tagged | 3 Comments

Xah’s Challenge

A couple of days ago Xah Lee posed a programming challenge. The problem was to write a miniparser in your favorite language to detect mismatched bracketing characters in a set of text files. The set of brackets to be checked were: () {} [] “” ‹› «» 【】 〈〉 《》 「」 『』. His solution is here.

I solved this problem in two “different” ways, both in Emacs lisp. This post is my first solution. I’ll post the second solution tomorrow.

This solution is similar to Lee’s but written in a more functional style. The strategy is to hunt down the brackets with search-forward-regexp, push the opening brackets onto a stack, and check that each closing bracket matches the top of the stack. If it does, we pop the stack and keep going. If it doesn’t, we record the error and write it to an error buffer. If there is anything left on the stack when we finish, that’s also an error because there is an unmatched opening bracket.

The main function, check-file-for-mismatches uses a complicated regular expression to locate the brackets and an alist to type the brackets and identify the matching bracket. In a production system, these would be computed before run time to save on execution time. As Paul Graham has said, the nice thing about Lisp is that all of Lisp is available all the time. That means that we can use Lisp as we are writing the code to compute the regexp and alist. In this example, I compute them at run time so that you can see how I did it.

We start with the list of brackets:

(defvar brackets '(("(" . ")")
                   ("{" . "}")
                   ("[" . "]")
                   ("“" . "”")
                   ("‹" . "›")
                   ("«" . "»")
                   ("【" . "】")
                   ("〈" . "〉")
                   ("《" . "》")
                   ("「" . "」")
                   ("『" . "』")))

and then compute the regexp with:

(defun make-br-regexp ()
  "Make a regexp to check for any of the delimiters in brackets"
  (mapconcat (lambda (bp)
               (concat (regexp-quote (car bp)) "\\|" (regexp-quote (cdr bp))))
             brackets "\\|"))

and the alist with:

(defun make-br-alist ()
  "Make an alist of the brackets. Each entry is of the form:
\(bracket open-or-close matching-bracket\)"
  (reduce #'append (mapcar (lambda (bp)
                             (list (list (car bp) 'open (cdr bp))
                                   (list (cdr bp) 'close (car bp))))
                           brackets)))

Each entry in the alist has the form (bracket type matching-bracket) where type is either open or close and indicates whether the bracket is an opening or closing bracket.

As I said, all of this could be done at coding time and the results made into constants. Instead we define a couple of constants on the fly:

(defconst br-regexp (make-br-regexp))
(defconst br-alist (make-br-alist))

All the work is done in check-file-for-mismatches.

 1:  (defun check-file-for-mismatches (fpath)
 2:    "Check a file for mismatched brackets."
 3:    (let ((fb (get-buffer-create "*Temp*"))
 4:          (stack nil)
 5:          (mismatches nil))
 6:      (set-buffer fb)
 7:      (insert-file-contents fpath nil nil nil t)
 8:      (goto-char 1)
 9:      (while (search-forward-regexp br-regexp nil t)
10:        (let* ((bpos (point))
11:               (brk (buffer-substring-no-properties (1- bpos) bpos))
12:               (blist (assoc brk br-alist))
13:               (btype (cadr blist)))
14:          (cond
15:           ((eq btype 'open)  (push (cons brk (1- bpos)) stack))
16:           ((eq btype 'close) (if (or (null stack)
17:                                      (not (string= (caar stack) (caddr blist))))
18:                                  (push (1- bpos) mismatches)
19:                                (pop stack)))
20:           (t (error "FAIL--found %s instead of bracket" brk)))))
21:      (if mismatches
22:          (mapc (lambda (e)
23:                  (print (format "Mismatch in %s at position %d"
24:                                 fpath e)))
25:                (reverse mismatches)))
26:      (if stack
27:          (mapc (lambda (e)
28:                  (print (format "%s has an unclosed %s at position %d"
29:                                 fpath (car e) (cdr e))))
30:                (reverse stack)))
31:      (kill-buffer fb)))
32:  

The actual checking occurs in the while loop on lines 9–20. We find a bracket with the regexp search, record its position and type and then check to see if it’s an opening or closing bracket. For opening brackets, we just push it on the stack. For closing brackets we check for the matching opening bracket on the stack and record an error if it’s not a match.

On lines 21–25, we print any mismatches that we found in the file. On lines 26–30 we print an error message for any open brackets still on the stack. Almost half of check-file-for-mismatches is concerned with error reporting. It’s worth noting that after the first error the parsing tends to get out of sync and you can get cascading errors but the error report tells you what happened so that you can fix the real errors and rerun the program as a final check.

To run check-file-for-mismatches we need a function to feed it the files and establish the error buffer. The sample function, do-files that I show below checks files with the extension txt in the ~/elisp directory. What you actually do depends on how you are selecting your files.

(defun do-files ()
  "Check a list of files for bracket mismatches"
  (interactive)
  (with-output-to-temp-buffer "*Mismatch Report*"
    (mapc #'check-file-for-mismatches (directory-files "~/elisp" t "\.txt$"))))

Many of the ideas for these functions come from Lee’s Emacs Lisp Tutorial. This page, in particular, has several useful templates for handling files and buffers in a way similar to what we’ve done here.

Posted in Programming | Tagged | Leave a comment

Common iPhone Pins

I’ve written several (1, 2, 3) posts about the analysis of passwords divulged by groups like LulzSec. The results were terrifyingly consistent: 123456 and password were almost always the most frequently used passwords. Now Daniel Amitay author of the Big Brother Camera Security App for the iPhone has some interesting results for iPhone pins.

Because his app has a lock screen nearly identical to the iPhone phone lock screen, he collected (anonymized) statistics on what codes were being used for his app in the hopes that it could tell us something about what how users are selecting their iPhone lock codes. He collected 204,508 codes and calculated their frequencies. Among other things, Amitay found that the top ten most common pins accounted for 15% of all passcodes.

Without reading further, try to guess the top four most popular pins. Hint: It’s not hard. The number one pin is, of course, 1234 followed by 0000. The next two are 2580 (going down the middle column) and 1111. The next 6 aren’t much harder to guess. Another common scheme is to use the user’s birth year.

Amity also looks at the most common digit for each of the four positions. The most common first digit is 1 by a large margin. If you squint a bit, it almost looks as if the pins obey Benford’s law. The second, third, and fourth digits are more uniform.

The post is an interesting, if depressing, read. As Amitay points out, for a random iPhone a cracker has a 15% chance of unlocking the phone before the automatic data wipe feature is activated. Considering that smartphones like the iPhone often contain confidential data or access to confidential data this is bad news.

In a sad endnote, Amitay has had his app removed from the App Store for his trouble. Apparently Apple was concerned that he was harvesting the Apple lock codes or something. Amitay is discussing the matter with Apple and we can hope that he will have the whole thing straightened out soon.

Posted in General | Tagged | Leave a comment