Irreal

This is the final post of my series on storing data as Lisp programs (see here, here, here, here, and here). I hope that those of you have read the whole series have a new or renewed appreciation for the concept of data as code that is one half of the data/code duality that Lisp provides. Today, I want discuss, just a bit, how I leveraged the power of Emacs to develop and debug the sample code in the series.

I’ve written before about how you can think of Emacs As A Lisp Machine: that is, the power of Lisp is always available to you. As an example of this, consider the “program” file that I used to develop code for the series. Here’s a screen shot of almost the entire buffer. Cut off at the top are 3 more records and an enclosing (log ...) sexpr that is assigned to log-file.

The thing to notice is that there are no real programs there, just a bunch of record and function definitions, a list of the tag names for the record structure and a few other miscellaneous pieces of code.

Walking through the process for the third problem of the previous post will make the process clear. I began by copying the list of tag names to the bottom of the file, narrowing to the list, and then running a keyboard macro on the narrowed buffer that changed each name into a call to defalias that defined the name as an alias for identity. Then I cut exception, frame, and record out of the list and changed the 'identity to 'list. All that’s just basic editing but with Emacs it involved almost no work: probably about 50 keystrokes total.

The next thing I did was to highlight the new function definitions and run eval-region. After that the new functions were defined and I could go to the end of one of the records and type 【Ctrl+x Ctrl+e】 to execute the record. That’s where the block showing the arguments to record came from.

Finally, I wrote the final 6 functions in the file and ran eval-region on them. That completed the solution to the third problem. I tested the solution first on each of the 4 records by running 【Ctrl+x Ctrl+e】 on each of them and then an actual filtering run by typing 【Ctrl+x Ctrl+e】 at the end of

(setq result (remove-if-not 'eval (cdr log-file)))

That resulted in the two records that passed the criteria being put in result.

The whole process is much more trouble to write down then it was to actually do. Emacs’ superb editing powers allowed me to write the support functions with an absolute minimum of effort and then to test and refine them interactively. Notice that there wasn’t even a REPL involved except for the implicit REPL that Emacs is always running as you type things into a buffer. The ability to select some code and execute it is tremendously powerful and liberating. And fast. During this development, I never left the filter-log.el buffer. There was no need to go to a separate REPL buffer, or to run a compile, or anything else.

I don’t program like this all the time, of course. Sometimes you just write a program or part of a program and then test it but when you’re feeling your way along—when you’re doing exploratory programming—it’s very nice to be able to write and try a snippet of code as small as a single statement. After a while, you begin to understand the problem better and you can drop back to the normal process of writing larger segments of code. At least until you need to do a little exploring again. Emacs supports both styles seamlessly and it’s easy to switch back and forth as needed. The power of Emacs can be astounding, even to those of us who are used to it.

My last two posts mentioned the ease with which s-expressions can be converted to XML and vice versa. Out of curiosity, I decided to write an sexpr to XML function in Elisp.

The implementation consists of two functions. The first, sexpr->xml, does the actual conversion, while the second, convert-to-xml, takes care of Emacs bookkeeping details and acts as a driver. For convenience, convert-to-xml takes an sexpr as its input but it would be easy to have it operate on a region or even a whole buffer.

We’ll use the example sexpr from Yegge’s The Emacs Problem post. Note that, again for simplicity, I am dealing with only a single record but everything would work exactly the same if there were multiple records wrapped in a '(log …) sexpr.

'(record
  (date "2005-02-21T18:57:39")
  (millis 1109041059800)
  (sequence 1)
  (logger nil)
  (level 'SEVERE)
  (class "java.util.logging.LogManager$RootLogger")
  (method 'log)
  (thread 10)
  (message "A very very bad thing has happened!")
  (exception
    (message "java.lang.Exception")
    (frame
      (class "logtest")
      (method 'main)
      (line 30))))

The converter is very simple.

1:  (defun sexpr->xml (sexpr)
2:    (let ((tag (car sexpr)))
3:      (princ (format "<%s>" tag))
4:      (dolist (o (cdr sexpr))
5:        (if (atom o)
6:            (princ (format "%s " o))
7:          (sexpr->xml o)))
8:      (princ (format "</%s>" tag))))

It’s passed an sexpr whose first symbol is the XML tag. That gets saved away in the let on line 2 and printed as the opening tag on line 3. The dolist loop in lines 4–7 looks at each of the other objects in the sexpr. If an object is not another sexpr, it is printed with a trailing space. If it is another sexpr, sexpr->xml is called recursively on line 7 to process it. When all the objects in the sexpr have been processed, the end tag is printed on line 8.

The output of sexpr->xml is a single line of XML with no formatting at all. Also, any nils will appear explicitly in the XML instead of the tag pair being empty and all the quoted symbols will be wrapped in <quote> … </quote> tags because the lisp reader turns 'symbol into (quote symbol).

Now let’s look at the driver function:

1:  (defun convert-to-xml (sexpr)
2:    (with-output-to-temp-buffer "*XML*"
3:      (sexpr->xml sexpr)
4:      (set-buffer "*XML*")
5:      (xml-mode)
6:      (replace-regexp "\\bnil\\b\\|<quote>\\|</quote>" "" nil (point-min) (point-max))
7:      (sgml-pretty-print (point-min) (point-max))))

Lines 2 and 3 call sexpr->xml and arrange for its output to go into a buffer named *XML*. When sexpr->xml returns, the *XML* buffer is selected and set to xml-mode (really nxml-mode). The replace-regexp on line 6 deletes any occurrences of nil and gets rid of <quote> and </quote> tags. Finally, sgml-pretty-print is called to format the XML nicely.

That’s a lot of work for not very much code. We could, of course, take care of the formatting and fixing up the nil and <quote> problems right in sexpr->xml but I wanted to show how simple the conversion can be without a lot of busy details. Besides, we have the power of Emacs so it would foolish not to use it.

The final result of running convert-to-xml on our sample sexpr is

<record>
  <date>2005-02-21T18:57:39
  </date>
  <millis>1109041059800
  </millis>
  <sequence>1
  </sequence>
  <logger>
  </logger>
  <level>SEVERE
  </level>
  <class>java.util.logging.LogManager$RootLogger
  </class>
  <method>log
  </method>
  <thread>10
  </thread>
  <message>A very very bad thing has happened!
  </message>
  <exception>
    <message>java.lang.Exception
    </message>
    <frame>
      <class>logtest
      </class>
      <method>main
      </method>
      <line>30
      </line>
    </frame>
  </exception>
</record>

Any Elisp coders out there might want to consider what a translator in the other direction (XML to s-expressions) would look like. Not having s-expressions to leverage makes the processing a little more difficult but not by much.

Almost all Emacs users are familiar with Steve Yegge’s Effective Emacs post. If you’re an Emacs user and haven’t read it, stop what you’re doing and go read it right now. As it happens, Yegge wrote several posts about Emacs and one of my favorites, which I’ve just reread is The Emacs Problem.

That post begins with a claim by one of Yegge’s colleagues that Lisp is not a good language for text processing, that Emacs goes to show that, and that Emacs should be rewritten with Ruby as the interpreter. I know, I know; me too. But Yegge examines that claim seriously and in the process generates an wonderfully readable case that nothing but Lisp is really any good at text processing.

He begins by noting that when we think of text processing we almost always think of regular expressions as the main tool. He notes that Lispers are generally skeptical of regexps and say things like, “Regexps aren’t useful for tree structured data and why are you storing your data as text in the first place instead of s-expressions (that is, as Lisp)?” Then he goes into a comedy routine about log files and how they’re usually just a line or two of text that is most easily processed with regexps and that those Lisp losers appear not to know that.

But then he says that he just noticed that his java.util.logging output had suddenly changed to being XML (this was with Java 1.5 in 2005). He gives an example of a short text-based log entry and the corresponding entry in XML and concedes that sometimes the extra metadata in the XML can be useful and that tools like Xpath provide a powerful way of processing and querying XML data.

Next, he gives the same data as an sexpr. It’s shorter, clearer, and much easier to read than the XML and can be operated on by Xpath-like tools available for Lisp. In fact, you can think of the data as executable and write functions or macros that can make it transform or process itself. He then identifies the Text-Processing Conundrum:

1. You want to be able to store and process text data.
2. Doing this effectively requires the data be tree structured in all but trivial cases.
3. The only good, general tool for doing this is XML.
4. XML processing is supposed to be easy but rapidly becomes complex when you start using tools like XSLT and XQuery or worse yet write your own transformations using a SAX or DOM processor in the language of your choice.
5. But those are your only options.

Unless you’re using Lisp. With Lisp, data is code and so you can store your data as a Lisp program. Querying and transforming it are almost trivial.

He goes on to consider other text processing such as configuration files and finds that the same principles apply only more so. By writing your configuration file in Lisp, it ceases being a configuration file and becomes part of your program.

Finally, he considers the question of rewriting Emacs in some other language. He goes through all the usual reasons why that’s not practical (at least until guile becomes the interpreter for Emacs if it ever does) and recites the usual litany of political problems surrounding Emacs and RMS.

To my mind, though, he misses the main point. Why would you want to write it in something besides a Lisp-like language. As he amply demonstrates in the first part of the post, Lisp is a great language for text processing and the only one without serious problems. Sure, a lot of the text that Emacs deals with isn’t tree structured, but a lot of it is and Emacs has a powerful regexp system for the text that isn’t. Just read through Xah Lee’s excellent series of articles on text processing with Emacs, for instance. A recurring theme in Lee’s posts is text-soup automation in which he uses Elisp to transform arbitrary text.

Yegge’s post is excellent and if you haven’t read it, you’re missing out on a treat. Also be sure to read the comments. He develops some of the ideas in the post further in responses to the commenters.