Filtering Log Files Stored As Lisp With Emacs

Yesterday, I wrote about transforming log files stored as Lisp. Today, I want to continue that theme by considering how we can have the log file filter itself so that we can pick out records meeting certain criteria. I wasn’t going to bother with this but it’s so easy once you have the infrastructure set up, I thought it might be fun to explore it a bit. Yesterday, I worked in Common Lisp, today, just for variety, I’ll work in Emacs Lisp.

For the benefit of those just joining the discussion, here is a typical record (from Steve Yegge’s The Emacs Problem).

(record
  (date "2005-02-21T18:57:39")
  (millis 1109041059800)
  (sequence 1)
  (logger nil)
  (level 'SEVERE)
  (class "java.util.logging.LogManager$RootLogger")
  (method 'log)
  (thread 10)
  (message "A very very bad thing has happened!")
  (exception
    (message "java.lang.Exception")
    (frame
      (class "logtest")
      (method 'main)
      (line 30))))

To help me in developing code for this post, I duplicated this record 4 times, changing some of the fields, wrapped it in a (log ...) sexpr, and put it in the variable log-file. As we did yesterday, we start by making each of the tags executable. To start, we define record, exception, and frame as functions that perform an and operation on their arguments. The record function is defined as

(defun record (&rest args) (every 'identity args))

and the other two are the same except for their names. See Applying and And or In Emacs for why something like (apply 'and args) won’t work. The other tags are defined as functions that just return t. For example, date is

(defun date (&rest args) t)

Now let’s suppose we only want to see records that have a level of SEVERE. We redefine level as

(defun level (s) (eq s 'SEVERE))

and filter out the other records by running

(remove-if-not 'eval (cdr log-file))

The remove-if-not function filters out any records that fail the “predicate” eval. Because eval evaluates the record sexpr it returns a t or nil value for each record. In this case, records without a level of SEVERE will have level return nil and therefore record will also return nil and therefore be filtered out.

We can build up more complicated conditions if needed. For example, if we want to see only records with a level of SEVERE and a line greater than 30, in addition to redefining level, we would also redefine line as

(defun line (l) (> l 30))

and run the remove-if-not function on log-file.

If you’re interested in this sort of thing, here are some problems to think about

1. How would you filter on

   level equals SEVERE or line > 30
   

2. Notice that method appears twice in the record. How would you filter on the one that’s in the frame sexpr?
3. How would you handle general queries such as

   (level equals SEVERE or line > 30) and thread equals 10
   

I’ll give my answers to those questions in the next installment.

It might seem like this is a lot of work to run a query given that there are general tools available for doing so but it doesn’t take nearly as much effort as you might think. First of all, you would have a lot of these functions already defined so that you can just paste them into your buffer, redefine the filtering conditions, and run the filter. In the next installment, I’ll talk about how writing these functions, even if you don’t already have them, is really easy when you leverage the power of Emacs. I’ll also discuss how I used Emacs to develop and test the code interactively as I went along.