As Irreal regulars know, I’m now reading my feeds with Elfeed. It put one more frequent task under Emacs and generally makes organizing and dealing with the feeds easier. Because of my Elfeed adoption, I’ve been paying more attention to news and articles about it.
One such article from last December is by Chris Wellons himself and describes one of the internal features of Elfeed. When an XML record describing an article or post arrives, the first thing that happens is that it is converted to an equivalent sexpr structure. From there, the necessary data is extracted and put into Elfeed’s database. That should be easy and, in fact, one could imagine making the sexpr process the data itself as I described back in 2011.
Sadly, things are more complicated. The RSS and Atom standards are a bit like the situation with Markdown: no one interprets them in exactly the same way. Data may be in one of several places and may or may not be encoded as you expect. Therefore, parsing the sexpr is non trivial. In order to avoid writing the same complicated code over and over, Wellons implemented a DSL that takes a description of the desired field within the structure. The DSL, which is sort of like a pattern matching description, is then interpreted by a pair of functions.
This can take some time. The example Wellons gives is searching for the data of an Atom entry. This may exist in 5 possible fields so he has a DSL entry to check each of those fields. As he added more features, the number of DSL entries that had to be checked grew and the process became slower.
This is where Wellons showed his genius. Rather than interpret the DSLs—parsing them over and over—he compiled them into Elisp byte code. That’s easy to do in Elisp because it’s precisely what macros are for. Basically, he replaced the two functions that interpreted the DSL with two macros that generated the same Elisp he would have written by hand. That change sped up the DSL processing by an order of magnitude and the entire XML processing by 25%. The processing is now dominated by the conversion of XML to the sexpr, which uses a function from the XML library and is out of his control.
See Wellons’ post for the details. If you like Lisp or ever wondered what the big deal about macros is, you’ll enjoy reading it. If you were wondering about macros, it will open your eyes. Really, go read it.