find-file Vs. with-temp-bufferYesterday, I mentioned that Xah Lee had a short post on dramatically differing run times for processing several files with find-file versus processing the same files using with-temp-buffer. His post was fairly short and didn’t have a lot of details. Happily, he’s published an extended post that shows the code and detailed performance of the two approaches.
This is really interesting and if you write Elisp that looks at lots of files at once, you should check out his post.
Xah Lee has a short post on find-file vs. with-temp-buffer for loading a large number of files. The results were sort of surprising, at least to me, so you should take a look if you need to load several files in Emacs. His post was basically a little benchmark and I vaguely wondered how he did the timing. Scheme and Common Lisp have functions for this but I had never seen anything for Elisp.
Just now, while looking at something completely unrelated I came across the benchmark-run command. It does just exactly what its name suggests: if you want to time some Elisp code you just wrap it with benchmark-run and go. It will report the elapsed time for execution, the number of garbage collections that ran, and the time spent in the garbage collector.
For example, suppose we want to run some-function on several files that are marked in a Dired buffer. We could do something like
(benchmark-run 1 (mapc 'some-function (dired-get-marked-files)))
to time how long it takes. The first argument (1) says to run the benchmark 1 time. You can set that to a larger number if you want to time several runs to get a better idea of average performance.
That’s a pretty nifty thing to know and I don’t recall ever seeing it anywhere before. That’s one of the nice things about Emacs: there’s always something new to learn.
Update: Lisp → Common Lisp
In view of the fact that my last few posts have concentrated on the use of S-expressions to represent data—in particular, to represent log file entries—this seems like a good time to link to S-expressions: The most powerful data structure available over at The (λ) Lambda Meme — all things Lisp. This post argues that sexprs are the most powerful data structure currently available and that they can represent arbitrarily complex data.
We saw a small example of that in our exploration of using sexprs to represent log data. Although the record structure we used was fairly simple, it’s obvious that we could represent any log record structure with an sexpr. Combine that will the ability to make them executable in a direct and easy way and you have a powerful tool.
The reason for the power of the sexpr, The Lambda meme argues, is that sexprs can be used to represent graphs and graphs can model the associations among multiple data. They also say that S-expressions are really all there is to understand about Lisp and once you do understand them, the rest is just a few details. I’m not sure it’s quite that simple but it is true that S-expressions are the soul of Lisp.
In any event, the linked post is short and interesting and worth a read.
This is the final post of my series on storing data as Lisp programs (see here, here, here, here, and here). I hope that those of you have read the whole series have a new or renewed appreciation for the concept of data as code that is one half of the data/code duality that Lisp provides. Today, I want discuss, just a bit, how I leveraged the power of Emacs to develop and debug the sample code in the series.
I’ve written before about how you can think of Emacs As A Lisp Machine: that is, the power of Lisp is always available to you. As an example of this, consider the “program” file that I used to develop code for the series. Here’s a screen shot of almost the entire buffer. Cut off at the top are 3 more records and an enclosing (log ...) sexpr that is assigned to log-file.
The thing to notice is that there are no real programs there, just a bunch of record and function definitions, a list of the tag names for the record structure and a few other miscellaneous pieces of code.
Walking through the process for the third problem of the previous post will make the process clear. I began by copying the list of tag names to the bottom of the file, narrowing to the list, and then running a keyboard macro on the narrowed buffer that changed each name into a call to defalias that defined the name as an alias for identity. Then I cut exception, frame, and record out of the list and changed the 'identity to 'list. All that’s just basic editing but with Emacs it involved almost no work: probably about 50 keystrokes total.
The next thing I did was to highlight the new function definitions and run eval-region. After that the new functions were defined and I could go to the end of one of the records and type 【Ctrl+x Ctrl+e】 to execute the record. That’s where the block showing the arguments to record came from.
Finally, I wrote the final 6 functions in the file and ran eval-region on them. That completed the solution to the third problem. I tested the solution first on each of the 4 records by running 【Ctrl+x Ctrl+e】 on each of them and then an actual filtering run by typing 【Ctrl+x Ctrl+e】 at the end of
(setq result (remove-if-not 'eval (cdr log-file)))
That resulted in the two records that passed the criteria being put in result.
The whole process is much more trouble to write down then it was to actually do. Emacs’ superb editing powers allowed me to write the support functions with an absolute minimum of effort and then to test and refine them interactively. Notice that there wasn’t even a REPL involved except for the implicit REPL that Emacs is always running as you type things into a buffer. The ability to select some code and execute it is tremendously powerful and liberating. And fast. During this development, I never left the filter-log.el buffer. There was no need to go to a separate REPL buffer, or to run a compile, or anything else.
I don’t program like this all the time, of course. Sometimes you just write a program or part of a program and then test it but when you’re feeling your way along—when you’re doing exploratory programming—it’s very nice to be able to write and try a snippet of code as small as a single statement. After a while, you begin to understand the problem better and you can drop back to the normal process of writing larger segments of code. At least until you need to do a little exploring again. Emacs supports both styles seamlessly and it’s easy to switch back and forth as needed. The power of Emacs can be astounding, even to those of us who are used to it.
Yesterday’s post discussed some rudimentary methods of having log files stored as Lisp filter themselves. Today I planned to finish that discussion and remark on how Emacs provides a powerful environment for exploratory programming. The discussion of the 3 challenge questions from yesterday turned out to be longer than I anticipated, so I’ll discuss them today and make my final comments about Emacs and exploratory programming in a subsequent post.
The first question wants us to combine queries with or rather than and. In particular, how would you filter on
level equals SEVERE or line > 30
One way of doing this is to make the definitions
(defun level (s) (not (eq s 'SEVERE))) (defun line (l) (<= line 30))
and then use remove-if instead of remove-if-not to filter the records.
The second question asks how you would filter on the method field that is part of the frame sexpr. The easiest way of doing that is to change the definition of frame to return the truth value of the condition. For example, if you wanted to filter on method being ‘main, you could define frame as
(defun frame (class method line) (eq method 'main))
The third question shows that we’ve just been fooling around so far and that what we really need is access to all the fields. Our record structure looks like
and the record function can’t see the emessage1, class, method, and line fields that are underneath the exception node. To get a general solution we need to arrange for record to have access to all the fields.
We start by having the leaf nodes just return their value. Thus, we define
(defalias 'date 'identity)
and similarly for millis, sequence, logger, level, class, thread, emessage, method, and line.
Next, we define the internal nodes exception and frame to return a list of their leaf values:
(defalias 'exception 'list)
and similarly for frame.
With these definitions, record has access to all the fields. To see what record‘s arguments look like, define record as list and execute one of the records. We get
("2005-02-21T18:57:39" 1109041059800 2 nil SEVERE "java.util.logging.LogManager$RootLogger" log 10 "A very very bad thing has happened!" ("java.lang.Exception" ("logtest" main 120)))
To get access to the leaves under exception, we need some accessor functions.
(defun exception.emessage () (car exception)) (defun exception.frame () (cadr exception)) (defun exception.frame.class () (car (exception.frame))) (defun exception.frame.method () (cadr (exception.frame))) (defun exception.frame.line () (caddr (exception.frame)))
With these definitions we can filter on conditions by writing the appropriate record function. To filter on
(level equals SEVERE or line > 30) and thread equals 10
as the third question asked, we would define record as
(defun record (date millis sequence logger level class method thread emessage exception) (and (or (eq level 'SEVERE) (> (exception.frame.line) 30)) (= thread 10)))
and then evaluate
(remove-if-not 'eval (cdr log-file))
as before.
Note that given some record structure, everything but the record function is fixed so that filtering on different criteria means that only the record function needs to be changed.
UPDATE: would → wouldn’t
1 I changed the message field of the record to emessage so that I wouldn’t have to redefine Emacs’ message function.
Yesterday, I wrote about transforming log files stored as Lisp. Today, I want to continue that theme by considering how we can have the log file filter itself so that we can pick out records meeting certain criteria. I wasn’t going to bother with this but it’s so easy once you have the infrastructure set up, I thought it might be fun to explore it a bit. Yesterday, I worked in Common Lisp, today, just for variety, I’ll work in Emacs Lisp.
For the benefit of those just joining the discussion, here is a typical record (from Steve Yegge’s The Emacs Problem).
(record (date "2005-02-21T18:57:39") (millis 1109041059800) (sequence 1) (logger nil) (level 'SEVERE) (class "java.util.logging.LogManager$RootLogger") (method 'log) (thread 10) (message "A very very bad thing has happened!") (exception (message "java.lang.Exception") (frame (class "logtest") (method 'main) (line 30))))
To help me in developing code for this post, I duplicated this record 4 times, changing some of the fields, wrapped it in a (log ...) sexpr, and put it in the variable log-file. As we did yesterday, we start by making each of the tags executable. To start, we define record, exception, and frame as functions that perform an and operation on their arguments. The record function is defined as
(defun record (&rest args) (every 'identity args))
and the other two are the same except for their names. See Applying and And or In Emacs for why something like (apply 'and args) won’t work. The other tags are defined as functions that just return t. For example, date is
(defun date (&rest args) t)
Now let’s suppose we only want to see records that have a level of SEVERE. We redefine level as
(defun level (s) (eq s 'SEVERE))
and filter out the other records by running
(remove-if-not 'eval (cdr log-file))
The remove-if-not function filters out any records that fail the “predicate” eval. Because eval evaluates the record sexpr it returns a t or nil value for each record. In this case, records without a level of SEVERE will have level return nil and therefore record will also return nil and therefore be filtered out.
We can build up more complicated conditions if needed. For example, if we want to see only records with a level of SEVERE and a line greater than 30, in addition to redefining level, we would also redefine line as
(defun line (l) (> l 30))
and run the remove-if-not function on log-file.
If you’re interested in this sort of thing, here are some problems to think about
level equals SEVERE or line > 30
method appears twice in the record. How would you filter on the one that’s in the frame sexpr? (level equals SEVERE or line > 30) and thread equals 10
I’ll give my answers to those questions in the next installment.
It might seem like this is a lot of work to run a query given that there are general tools available for doing so but it doesn’t take nearly as much effort as you might think. First of all, you would have a lot of these functions already defined so that you can just paste them into your buffer, redefine the filtering conditions, and run the filter. In the next installment, I’ll talk about how writing these functions, even if you don’t already have them, is really easy when you leverage the power of Emacs. I’ll also discuss how I used Emacs to develop and test the code interactively as I went along.
A couple of days ago, I wrote about Converting S-Expressions To XML in Emacs. That post was occasioned by my recent rereading of Steve Yegge’s The Emacs Problem in which he talks about, among other things, the utility of storing data as a Lisp program. His thesis was that doing so allows you to easily perform all sorts of useful manipulations on your data.
Certainly, as my post on transforming sexprs to XML shows, it’s trivial to convert a log file stored as a Lisp program to XML. But that post didn’t really leverage the fact that the log record from Yegge’s example is (or can be made) executable.
If you look at it again closely, you’ll notice that it’s just a complicated sexpr and therefore executable as long as the “tags” in the function slots are defined as functions.
(record (date "2005-02-21T18:57:39") (millis 1109041059800) (sequence 1) (logger nil) (level 'SEVERE) (class "java.util.logging.LogManager$RootLogger") (method 'log) (thread 10) (message "A very very bad thing has happened!") (exception (message "java.lang.Exception") (frame (class "logtest") (method 'main) (line 30))))
To show how we can make the record transform itself, let’s start by defining the tags to just reproduce the sexpr that they head. The idea is simple. We could, for example, define record as
(defun record (&rest args) (cons 'record args))
With this definition, an expression such as
(record "abc" 123)
would evaluate to itself. In order to evaluate the full record from Yegge, we need to define functions for each of the function tags. To make that a bit easier we define a helper macro, prself.
(defmacro prself (f) `(defun ,f (&rest args) (self-print ',f args)))
so that
(prself record)
get transformed to
(defun record (&rest args) (self-print 'record args))
Unfortunately, self-print is a bit more complicated than our initial example because we have to quote symbols. That is, we want the sexpr
(level 'SEVERE)
to return an sexpr with SEVERE quoted instead of
(level SEVERE)
To do that, we need to quote all the symbols except nil and t.
(defun self-print (f args) (cons f (mapcar (lambda (a) (if (and (symbolp a) (not (eq a nil)) (not (eq a t))) `(quote ,a) a)) args)))
Next we turn record, date, millis, …, line into functions with (prself record), (prself date), (prself millis), …, (prself line). At this point, the record is executable, and if we execute it directly we get
(RECORD (DATE "2005-02-21T18:57:39") (MILLIS 1109041059800) (SEQUENCE 1) (LOGGER NIL) (LEVEL 'SEVERE) (CLASS "java.util.logging.LogManager$RootLogger") (METHOD 'LOG) (THREAD 10) (MESSAGE "A very very bad thing has happened!") (EXCEPTION (MESSAGE "java.lang.Exception") (FRAME (CLASS "logtest") (METHOD 'MAIN) (LINE 30))))
That’s nice, but not very interesting. But suppose we want to transform the milliseconds field (millis) into seconds. We can do that by redefining millis as
(defun millis (m) `(seconds ,(/ m 1000.0)));
and if we execute record again, we get
(RECORD (DATE "2005-02-21T18:57:39") (SECONDS 1.109041E+9) (SEQUENCE 1) (LOGGER NIL) (LEVEL 'SEVERE) (CLASS "java.util.logging.LogManager$RootLogger") (METHOD 'LOG) (THREAD 10) (MESSAGE "A very very bad thing has happened!") (EXCEPTION (MESSAGE "java.lang.Exception") (FRAME (CLASS "logtest") (METHOD 'MAIN) (LINE 30))))
Again, that’s a trivial transformation, but it shows how easy it is to transform fields. Also notice that if we had a whole log of the form
(log (record (date...)...) (record (date...)...) ... (record (date...)...))
and ran prself on log, then we could transform the whole log at once without worrying about writing any looping code or anything else. Just execute the (log ...) sexpr and you’re done.
This is a long post but it shows the power of having your data be executable. Incidentally, I wrote this code using Common Lisp but the same functions would work just fine in Emacs Lisp.
But if you want a visualization of quick sort, they’ve got you covered.
There’s a new version of Org mode available. You can download it from here. The download page also has instructions for cloning the Git repository so that you can stay as up-to-date as you like.
The main changes appear to be with Babel processing. Some of these are incompatible so if you’re a Babel user be sure to take a look. The changes page has an Elisp function to automate updating your old Babel buffers to the new syntax.
Concurrent with the new release, the Org team has also updated the site with a completely new look.
My last two posts mentioned the ease with which s-expressions can be converted to XML and vice versa. Out of curiosity, I decided to write an sexpr to XML function in Elisp.
The implementation consists of two functions. The first, sexpr->xml, does the actual conversion, while the second, convert-to-xml, takes care of Emacs bookkeeping details and acts as a driver. For convenience, convert-to-xml takes an sexpr as its input but it would be easy to have it operate on a region or even a whole buffer.
We’ll use the example sexpr from Yegge’s The Emacs Problem post. Note that, again for simplicity, I am dealing with only a single record but everything would work exactly the same if there were multiple records wrapped in a '(log …) sexpr.
'(record (date "2005-02-21T18:57:39") (millis 1109041059800) (sequence 1) (logger nil) (level 'SEVERE) (class "java.util.logging.LogManager$RootLogger") (method 'log) (thread 10) (message "A very very bad thing has happened!") (exception (message "java.lang.Exception") (frame (class "logtest") (method 'main) (line 30))))
The converter is very simple.
1: (defun sexpr->xml (sexpr) 2: (let ((tag (car sexpr))) 3: (princ (format "<%s>" tag)) 4: (dolist (o (cdr sexpr)) 5: (if (atom o) 6: (princ (format "%s " o)) 7: (sexpr->xml o))) 8: (princ (format "</%s>" tag))))
It’s passed an sexpr whose first symbol is the XML tag. That gets saved away in the let on line 2 and printed as the opening tag on line 3. The dolist loop in lines 4–7 looks at each of the other objects in the sexpr. If an object is not another sexpr, it is printed with a trailing space. If it is another sexpr, sexpr->xml is called recursively on line 7 to process it. When all the objects in the sexpr have been processed, the end tag is printed on line 8.
The output of sexpr->xml is a single line of XML with no formatting at all. Also, any nils will appear explicitly in the XML instead of the tag pair being empty and all the quoted symbols will be wrapped in <quote> … </quote> tags because the lisp reader turns 'symbol into (quote symbol).
Now let’s look at the driver function:
1: (defun convert-to-xml (sexpr) 2: (with-output-to-temp-buffer "*XML*" 3: (sexpr->xml sexpr) 4: (set-buffer "*XML*") 5: (xml-mode) 6: (replace-regexp "\\bnil\\b\\|<quote>\\|</quote>" "" nil (point-min) (point-max)) 7: (sgml-pretty-print (point-min) (point-max))))
Lines 2 and 3 call sexpr->xml and arrange for its output to go into a buffer named *XML*. When sexpr->xml returns, the *XML* buffer is selected and set to xml-mode (really nxml-mode). The replace-regexp on line 6 deletes any occurrences of nil and gets rid of <quote> and </quote> tags. Finally, sgml-pretty-print is called to format the XML nicely.
That’s a lot of work for not very much code. We could, of course, take care of the formatting and fixing up the nil and <quote> problems right in sexpr->xml but I wanted to show how simple the conversion can be without a lot of busy details. Besides, we have the power of Emacs so it would foolish not to use it.
The final result of running convert-to-xml on our sample sexpr is
<record> <date>2005-02-21T18:57:39 </date> <millis>1109041059800 </millis> <sequence>1 </sequence> <logger> </logger> <level>SEVERE </level> <class>java.util.logging.LogManager$RootLogger </class> <method>log </method> <thread>10 </thread> <message>A very very bad thing has happened! </message> <exception> <message>java.lang.Exception </message> <frame> <class>logtest </class> <method>main </method> <line>30 </line> </frame> </exception> </record>
Any Elisp coders out there might want to consider what a translator in the other direction (XML to s-expressions) would look like. Not having s-expressions to leverage makes the processing a little more difficult but not by much.