Mickey Petersen author of the the excellent book Mastering Emacs and the blog of the same name has another of his comprehensive and informative articles on his blog. This time it’s about tree sitter and its application to syntax highlighting in Emacs.
Even though we take it mostly for granted, syntax highlighting is actually a devilishly difficult problem. Emacs and almost every other editor does it with a witches’ brew of regular expressions and ad hoc functions. The results are usually, but not always, good enough. The correct method, of course, is to build a syntax tree of the code and use that to inform the highlighting. There’s a whole host of problems with that approach. In the first place, the code you’re editing probably isn’t syntactically correct because it’s incomplete. Secondly, the naive solution of rebuilding the tree with every keystroke is too slow to be practical and there are other problems as well.
Mickey has a solution: use tree sitter to drive the highlighting. If you don’t know about tree sitter or what it’s used for, head on over to Mickey’s post for the details. Tree sitter can do more than just figure out syntax highlighting. Mickey mentions the excellent ParEdit and its use in Lispy languages. Mickey wanted to extend that capability to other languages, Python in particular.
He’s built a package to do just that. He calls it combobulate. It’s still alpha software but you can try it out if you like. Mickey says he’ll put it on Melpa as soon as he works out a few more issues. Mickey believes that the tree sitter approach is the future for addressing these types of problems.
I know I keep saying this but tree sitter and Emacs’ embrace of it once again puts the lie to the notion that Emacs is moribund technology. The reverse is true. Emacs keeps introducing new capabilities that other editors copy and then use to support their claims of superiority.