Irreal

A few days ago, I posed an Elisp programming challenge that asked you to make the record

(record
  (date "2005-02-21T18:57:39")
  (millis 1109041059800)
  (sequence 1)
  (logger nil)
  (level 'SEVERE)
  (class "java.util.logging.LogManager$RootLogger")
  (method 'log)
  (thread 10)
  (emessage "A very very bad thing has happened!")
  (exception
    (emessage "java.lang.Exception")
    (frame
      (class "logtest")
      (method 'main)
      (line 30))))

executable in such a way that it transforms itself into the equivalent XML record when executed.

On the day I posed the challenge, I woke up thinking that making the record transform itself into XML was the proper way to turn it into XML rather than the way I described in Converting S-Expression To XML In Emacs. It seemed like a direct application of the ideas in my More Fun With Log Files Stored As Lisp post and I decided to program it just for fun with no idea of turning it into a post.

A couple of hours later I was still flailing around so maybe the application wasn’t as direct as I thought. Actually, the idea is the same but because you’re not transforming it back into Lisp it’s a little trickier.

In any event, here’s how I solved the problem. First define all the tags except record to be functions of the form

(defun date (&rest args) (toxml 'date args))

As I explained in the Exploratory Programming In Emacs post that involved little more than copying the list of tags and a keyboard macro. The toxml function is

(defun toxml (tag args)
  (with-output-to-string
    (princ (format "\n<%s>" tag))
    (mapc 'princ args)
    (princ (format "</%s>" tag))))

The idea is that each sexpr returns a string of the arguments wrapped in <tag>…</tag> where tag is the symbol in the function slot of the sexpr. That is, (date "2005-02-21T18:57:39") is transformed to "<date>2005-02-21T18:57:39</date>". If we temporarily alias record to list and executed the record sexpr, we get

("
<date>2005-02-21T18:57:39</date>" "
<millis>1109041059800</millis>" "
<sequence>1</sequence>" "
<logger>nil</logger>" "
<level>SEVERE</level>" "
<class>java.util.logging.LogManager$RootLogger</class>" "
<method>log</method>" "
<thread>10</thread>" "
<emessage>A very very bad thing has happened!</emessage>" "
<exception>
<emessage>java.lang.Exception</emessage>
<frame>
<class>logtest</class>
<method>main</method>
<line>30</line></frame></exception>")

which shows what the record function will see when it gets called. Note that each string begins with a newline so the ” ” at the end of each line is a close quote followed by an open quote.

Finally, we need to define a record function to stitch the strings together and get rid of that annoying nil in <logger>nil</logger>.

(defun record (&rest args)
  (princ "<record>")
  (princ (replace-regexp-in-string ">nil<" "><" (apply 'concat args)))
  (princ "</record>"))

The princ on the third line gets rid of the quote marks that the concat places around the final result of the apply.

Now when we execute the record we get

<record>
<date>2005-02-21T18:57:39</date>
<millis>1109041059800</millis>
<sequence>1</sequence>
<logger></logger>
<level>SEVERE</level>
<class>java.util.logging.LogManager$RootLogger</class>
<method>log</method>
<thread>10</thread>
<emessage>A very very bad thing has happened!</emessage>
<exception>
<emessage>java.lang.Exception</emessage>
<frame>
<class>logtest</class>
<method>main</method>
<line>30</line></frame></exception></record>

Update: Let me mention again that the inspiration for this series of posts and the sample record sexpr are from Steve Yegge’s The Emacs Problem. Be sure to give it a read if you haven’t already.

In Converting S-Expressions To XML In Emacs, I showed how to take a log record expressed as Lisp and turn it into the equivalent XML. In More Fun With Log Files Stored As Lisp, I showed how to take that same log record and make it executable so that it transformed one of the fields when the record was executed.

Here is a challenge for Elisp programmers: take the record

(record
  (date "2005-02-21T18:57:39")
  (millis 1109041059800)
  (sequence 1)
  (logger nil)
  (level 'SEVERE)
  (class "java.util.logging.LogManager$RootLogger")
  (method 'log)
  (thread 10)
  (emessage "A very very bad thing has happened!")
  (exception
    (emessage "java.lang.Exception")
    (frame
      (class "logtest")
      (method 'main)
      (line 30))))

and turn it into XML like this:

<record>
<date>2005-02-21T18:57:39</date>
<millis>1109041059800</millis>
<sequence>1</sequence>
<logger></logger>
<level>SEVERE</level>
<class>java.util.logging.LogManager$RootLogger</class>
<method>log</method>
<thread>10</thread>
<emessage>A very very bad thing has happened!</emessage>
<exception>
<emessage>java.lang.Exception</emessage>
<frame>
<class>logtest</class>
<method>main</method>
<line>30</line></frame></exception></record>

Do this by making the Lisp record executable so that it transforms itself into the XML when executed. Your XML doesn’t have to be formatted exactly like mine and it’s sufficient to have it print out in the echo area or the *Messages* buffer.

I’ll give my solution in a few days so that you have time to work on it. If you get stuck, the wrap-up post has some techniques that may help.

Yesterday, I mentioned that Xah Lee had a short post on dramatically differing run times for processing several files with find-file versus processing the same files using with-temp-buffer. His post was fairly short and didn’t have a lot of details. Happily, he’s published an extended post that shows the code and detailed performance of the two approaches.

This is really interesting and if you write Elisp that looks at lots of files at once, you should check out his post.

Xah Lee has a short post on find-file vs. with-temp-buffer for loading a large number of files. The results were sort of surprising, at least to me, so you should take a look if you need to load several files in Emacs. His post was basically a little benchmark and I vaguely wondered how he did the timing. Scheme and Common Lisp have functions for this but I had never seen anything for Elisp.

Just now, while looking at something completely unrelated I came across the benchmark-run command. It does just exactly what its name suggests: if you want to time some Elisp code you just wrap it with benchmark-run and go. It will report the elapsed time for execution, the number of garbage collections that ran, and the time spent in the garbage collector.

For example, suppose we want to run some-function on several files that are marked in a Dired buffer. We could do something like

(benchmark-run 1 (mapc 'some-function (dired-get-marked-files)))

to time how long it takes. The first argument (1) says to run the benchmark 1 time. You can set that to a larger number if you want to time several runs to get a better idea of average performance.

That’s a pretty nifty thing to know and I don’t recall ever seeing it anywhere before. That’s one of the nice things about Emacs: there’s always something new to learn.

Update: Lisp → Common Lisp

This is the final post of my series on storing data as Lisp programs (see here, here, here, here, and here). I hope that those of you have read the whole series have a new or renewed appreciation for the concept of data as code that is one half of the data/code duality that Lisp provides. Today, I want discuss, just a bit, how I leveraged the power of Emacs to develop and debug the sample code in the series.

I’ve written before about how you can think of Emacs As A Lisp Machine: that is, the power of Lisp is always available to you. As an example of this, consider the “program” file that I used to develop code for the series. Here’s a screen shot of almost the entire buffer. Cut off at the top are 3 more records and an enclosing (log ...) sexpr that is assigned to log-file.

The thing to notice is that there are no real programs there, just a bunch of record and function definitions, a list of the tag names for the record structure and a few other miscellaneous pieces of code.

Walking through the process for the third problem of the previous post will make the process clear. I began by copying the list of tag names to the bottom of the file, narrowing to the list, and then running a keyboard macro on the narrowed buffer that changed each name into a call to defalias that defined the name as an alias for identity. Then I cut exception, frame, and record out of the list and changed the 'identity to 'list. All that’s just basic editing but with Emacs it involved almost no work: probably about 50 keystrokes total.

The next thing I did was to highlight the new function definitions and run eval-region. After that the new functions were defined and I could go to the end of one of the records and type 【Ctrl+x Ctrl+e】 to execute the record. That’s where the block showing the arguments to record came from.

Finally, I wrote the final 6 functions in the file and ran eval-region on them. That completed the solution to the third problem. I tested the solution first on each of the 4 records by running 【Ctrl+x Ctrl+e】 on each of them and then an actual filtering run by typing 【Ctrl+x Ctrl+e】 at the end of

(setq result (remove-if-not 'eval (cdr log-file)))

That resulted in the two records that passed the criteria being put in result.

The whole process is much more trouble to write down then it was to actually do. Emacs’ superb editing powers allowed me to write the support functions with an absolute minimum of effort and then to test and refine them interactively. Notice that there wasn’t even a REPL involved except for the implicit REPL that Emacs is always running as you type things into a buffer. The ability to select some code and execute it is tremendously powerful and liberating. And fast. During this development, I never left the filter-log.el buffer. There was no need to go to a separate REPL buffer, or to run a compile, or anything else.

I don’t program like this all the time, of course. Sometimes you just write a program or part of a program and then test it but when you’re feeling your way along—when you’re doing exploratory programming—it’s very nice to be able to write and try a snippet of code as small as a single statement. After a while, you begin to understand the problem better and you can drop back to the normal process of writing larger segments of code. At least until you need to do a little exploring again. Emacs supports both styles seamlessly and it’s easy to switch back and forth as needed. The power of Emacs can be astounding, even to those of us who are used to it.