Irreal

My last two posts mentioned the ease with which s-expressions can be converted to XML and vice versa. Out of curiosity, I decided to write an sexpr to XML function in Elisp.

The implementation consists of two functions. The first, sexpr->xml, does the actual conversion, while the second, convert-to-xml, takes care of Emacs bookkeeping details and acts as a driver. For convenience, convert-to-xml takes an sexpr as its input but it would be easy to have it operate on a region or even a whole buffer.

We’ll use the example sexpr from Yegge’s The Emacs Problem post. Note that, again for simplicity, I am dealing with only a single record but everything would work exactly the same if there were multiple records wrapped in a '(log …) sexpr.

'(record
  (date "2005-02-21T18:57:39")
  (millis 1109041059800)
  (sequence 1)
  (logger nil)
  (level 'SEVERE)
  (class "java.util.logging.LogManager$RootLogger")
  (method 'log)
  (thread 10)
  (message "A very very bad thing has happened!")
  (exception
    (message "java.lang.Exception")
    (frame
      (class "logtest")
      (method 'main)
      (line 30))))

The converter is very simple.

1:  (defun sexpr->xml (sexpr)
2:    (let ((tag (car sexpr)))
3:      (princ (format "<%s>" tag))
4:      (dolist (o (cdr sexpr))
5:        (if (atom o)
6:            (princ (format "%s " o))
7:          (sexpr->xml o)))
8:      (princ (format "</%s>" tag))))

It’s passed an sexpr whose first symbol is the XML tag. That gets saved away in the let on line 2 and printed as the opening tag on line 3. The dolist loop in lines 4–7 looks at each of the other objects in the sexpr. If an object is not another sexpr, it is printed with a trailing space. If it is another sexpr, sexpr->xml is called recursively on line 7 to process it. When all the objects in the sexpr have been processed, the end tag is printed on line 8.

The output of sexpr->xml is a single line of XML with no formatting at all. Also, any nils will appear explicitly in the XML instead of the tag pair being empty and all the quoted symbols will be wrapped in <quote> … </quote> tags because the lisp reader turns 'symbol into (quote symbol).

Now let’s look at the driver function:

1:  (defun convert-to-xml (sexpr)
2:    (with-output-to-temp-buffer "*XML*"
3:      (sexpr->xml sexpr)
4:      (set-buffer "*XML*")
5:      (xml-mode)
6:      (replace-regexp "\\bnil\\b\\|<quote>\\|</quote>" "" nil (point-min) (point-max))
7:      (sgml-pretty-print (point-min) (point-max))))

Lines 2 and 3 call sexpr->xml and arrange for its output to go into a buffer named *XML*. When sexpr->xml returns, the *XML* buffer is selected and set to xml-mode (really nxml-mode). The replace-regexp on line 6 deletes any occurrences of nil and gets rid of <quote> and </quote> tags. Finally, sgml-pretty-print is called to format the XML nicely.

That’s a lot of work for not very much code. We could, of course, take care of the formatting and fixing up the nil and <quote> problems right in sexpr->xml but I wanted to show how simple the conversion can be without a lot of busy details. Besides, we have the power of Emacs so it would foolish not to use it.

The final result of running convert-to-xml on our sample sexpr is

<record>
  <date>2005-02-21T18:57:39
  </date>
  <millis>1109041059800
  </millis>
  <sequence>1
  </sequence>
  <logger>
  </logger>
  <level>SEVERE
  </level>
  <class>java.util.logging.LogManager$RootLogger
  </class>
  <method>log
  </method>
  <thread>10
  </thread>
  <message>A very very bad thing has happened!
  </message>
  <exception>
    <message>java.lang.Exception
    </message>
    <frame>
      <class>logtest
      </class>
      <method>main
      </method>
      <line>30
      </line>
    </frame>
  </exception>
</record>

Any Elisp coders out there might want to consider what a translator in the other direction (XML to s-expressions) would look like. Not having s-expressions to leverage makes the processing a little more difficult but not by much.

Almost all Emacs users are familiar with Steve Yegge’s Effective Emacs post. If you’re an Emacs user and haven’t read it, stop what you’re doing and go read it right now. As it happens, Yegge wrote several posts about Emacs and one of my favorites, which I’ve just reread is The Emacs Problem.

That post begins with a claim by one of Yegge’s colleagues that Lisp is not a good language for text processing, that Emacs goes to show that, and that Emacs should be rewritten with Ruby as the interpreter. I know, I know; me too. But Yegge examines that claim seriously and in the process generates an wonderfully readable case that nothing but Lisp is really any good at text processing.

He begins by noting that when we think of text processing we almost always think of regular expressions as the main tool. He notes that Lispers are generally skeptical of regexps and say things like, “Regexps aren’t useful for tree structured data and why are you storing your data as text in the first place instead of s-expressions (that is, as Lisp)?” Then he goes into a comedy routine about log files and how they’re usually just a line or two of text that is most easily processed with regexps and that those Lisp losers appear not to know that.

But then he says that he just noticed that his java.util.logging output had suddenly changed to being XML (this was with Java 1.5 in 2005). He gives an example of a short text-based log entry and the corresponding entry in XML and concedes that sometimes the extra metadata in the XML can be useful and that tools like Xpath provide a powerful way of processing and querying XML data.

Next, he gives the same data as an sexpr. It’s shorter, clearer, and much easier to read than the XML and can be operated on by Xpath-like tools available for Lisp. In fact, you can think of the data as executable and write functions or macros that can make it transform or process itself. He then identifies the Text-Processing Conundrum:

1. You want to be able to store and process text data.
2. Doing this effectively requires the data be tree structured in all but trivial cases.
3. The only good, general tool for doing this is XML.
4. XML processing is supposed to be easy but rapidly becomes complex when you start using tools like XSLT and XQuery or worse yet write your own transformations using a SAX or DOM processor in the language of your choice.
5. But those are your only options.

Unless you’re using Lisp. With Lisp, data is code and so you can store your data as a Lisp program. Querying and transforming it are almost trivial.

He goes on to consider other text processing such as configuration files and finds that the same principles apply only more so. By writing your configuration file in Lisp, it ceases being a configuration file and becomes part of your program.

Finally, he considers the question of rewriting Emacs in some other language. He goes through all the usual reasons why that’s not practical (at least until guile becomes the interpreter for Emacs if it ever does) and recites the usual litany of political problems surrounding Emacs and RMS.

To my mind, though, he misses the main point. Why would you want to write it in something besides a Lisp-like language. As he amply demonstrates in the first part of the post, Lisp is a great language for text processing and the only one without serious problems. Sure, a lot of the text that Emacs deals with isn’t tree structured, but a lot of it is and Emacs has a powerful regexp system for the text that isn’t. Just read through Xah Lee’s excellent series of articles on text processing with Emacs, for instance. A recurring theme in Lee’s posts is text-soup automation in which he uses Elisp to transform arbitrary text.

Yegge’s post is excellent and if you haven’t read it, you’re missing out on a treat. Also be sure to read the comments. He develops some of the ideas in the post further in responses to the commenters.

Xah Lee as an instructive post on converting and scaling images with Emacs. Unlike him, I don’t often need to scale or convert even a single image, let alone many, but he does show an interesting technique to pass several files to an Elisp function. The idea is to use dired to mark the files you’re interested in and then use dired-get-marked-files to pass them to the function.

The template for performing some action on each of the marked files is

(defun process-each-file (files)
  "Perform an action on each file in list FILES."
  (interactive (list (dired-get-marked-files)))
  (mapc (lambda (f) (perform-action-on-file f)) files))

For example, if you want to perform perform-action-on-file on each JPG file you can use 【*.】 .jpg in dired to mark all the JPG files and then call process-each-file.

The trick that makes this work is passing a list to the interactive declaration. I’ve written about that before. This is a handy technique that can often be used to advantage.