Importing Data from a RESTful Website into an Org Table

Charl Botha has a really interesting post on importing data from a RESTful Webservice into an Org table. He gives the example of downloading cryptocurrency trading values and updating a table showing the total value of a user’s holdings but it’s easy to imagine all sorts of useful applications for the strategy.

Besides the idea of using the request package to retrieve data from a Website, Botha also demonstrates a couple of useful techniques that I didn’t know about. The first is the cl-function macro that allows you to use the full set of Common Lisp function argument forms. You probably won’t use that very often but it’s just what’s needed for Botha’s application.

The second technique is the use of the let-alist form. That’s a new macro from Artur Malabarba in Emacs 25 that makes it easy to access data in alists in a concise and natural way. Take a look at Botha’s post for a nice example of its use.

So even if you don’t have a need to import Web data into an Org table, it’s still worthwhile studying Botha’s post for the useful techniques.

Posted in General | Tagged , | Leave a comment

Semantic Locality

I’m an old time Unix guy who’s still more comfortable on the command line than a big GUI. That’s probably why I enjoy living in Emacs to the extent possible. One of the most powerful concepts from Unix is the idea of small tools that do one thing (usually to a text stream) well and to connect those tools with pipes to do a larger, more complicated task.

I was looking through my queue of blog ideas and came across this Eric Raymond post from last March. In it, he introduces the concept of “semantic locality” as a way of explaining when and how the set of small tools operating on a text stream works. You really need to read his post to understand his argument but the TL;DR is that text stream concept works when the data has semantic locality. That is when you can do useful work on small pieces of data that are (mostly) contiguous. You can see right away why data with semantic locality is susceptible to the small tools approach. As esr points out, even more it tells you when that approach probably won’t work.

Semantic locality is really just a formalism of our intuition. Even if you’ve never heard of it, you can—if you’re an experienced Unix hand—probably decide when a pipeline is a good idea and when it isn’t. It’s useful though because it gives us a systematic way of thinking about the problem. I found the post interesting and if you’re a Unix guy or gal you probably will too.

Posted in General | Tagged , | Leave a comment

Fuco1 on Anchored Fontlocking

Matus Goljer (Fuco1) has a really meaty post on one of the dark corners in Emacs. By dark corner I mean little understood and complicated. Fuco1 doesn’t think very much of the font lock logic, depending, as it does, on regular expressions1. Never-the-less, he admits, it does work very well in practice. Fuco1 had a use case that he thought would serve as an example of a problem that the existing font lock logic couldn’t handle. When he looked into it a bit more, he discovered that the mechanism is a little more flexible than is generally believed and that it could, in fact, solve his problem.

I’ll let you read Fuco1’s post for the details. If you’re like me, you’ll probably never have occasion to need the full flexibility of the font lock mechanism but the details of his solution are fascinating anyway. It’s not documented very well so Fuco1 had to tease out the details by reading the code. Even the commentary in font-lock.el, which I read using the wonderful finder-commentary, doesn’t tell the whole story.

My takeaway from all this is that Emacs has some (more or less) hidden capabilities that the average user won’t ever stumble on and that sometimes you have to get your hands dirty and read the code to find out what’s going on. It’s yet another reason to spend a little time learning to read Elisp.

Footnotes:

1

Ironically, one of authors of the font lock package is Jamie Zawinski.

Posted in General | Tagged | Leave a comment

Some Scimax Enhancements

I’ve been seeing a lot of tweets and posts asking about how to do Python development in Emacs. There are a bunch of good solutions, of course, but one of the best resources is John Kitchin. Kitchin is a Chemical Engineering professor at Carnegie Mellon and integrates Python and Emacs into his workflow at a very fundamental level. He’s collected his solutions to integrating Python and Emacs into the publicly available Scimax package for Emacs. I’ve written about Scimax a couple of times and if you aren’t familiar with it, you should check out his posts about it.

Kitchin has just released some enhancements to the package. In a sense, they aren’t earth shattering but they do make working with Python, and iPython in particular, more enjoyable. If you follow the link, you can see Kitchin’s explanation of the enhancements but, as he says, it’s probably easier to see what’s going on by watching the video included in the post.

The thing about Kitchin’s Scimax is that he is using it everyday to do real research and write his papers. You know it has to be good because he’s depending on it for his own work.

Posted in General | Tagged , | Leave a comment

Another Emacs Mail Solution

I’ve recently blogged about Emacs solutions to handling email. As most of you know by now, I’ve settled on mu/mu4e and have been extraordinarily happy with it. Still, not everyone has the same workflow or likes the same things. Andy C over at yakshaving.co.uk has another solution leveraging Gnus.

He was using mu4e and elfeed like but didn’t like that they each had their own UI that required him to learn different key sequences to do his work. He decided to settle on Gnus for email, RSS, and mailing lists. There were a few problems to get things working well and you should read his post to see how he solved them.

For me, the big advantage of mu/mu4e is the instantaneous search facility. I no longer worry about sorting messages into special folders; I just dump everything into a single folder and use the search function to find what I need. Andy C has pretty much the same workflow—although he stores emails by year—and had to install workarounds to get decent search with Gnus.

Oddly, even though I have often complained about using more than one editor because of muscle memory problems associated with different key sequences, I have no problem keeping the key sequences between mu4e and elfeed, for instance, separate. Even if I did, it’s trivial to rebind them to be the same so I don’t see the advantage to switching to Gnus because of it.

Of course, many people, including our fearless leader John Wiegley, swear by Gnus. It serves their workflows and makes them productive. But that’s a little different from trying to make Gnus look like some other solution. Regardless, Emacs, as I often say, allows us to have it our own way. If you like Gnus but want a decent way to search your emails, perhaps Andy C’s post can help you achieve it. If you don’t have special needs, like Wiegley does, mu4e provides an excellent solution. Emacs can provide whatever works for you.

Posted in General | Tagged | 3 Comments

Notmuch Video

A while ago I wrote about notmuch and astroid as an alternative to mu/mu4e. Since then lots of folks have told me how much they love notmuch and what a great solution it is for email. There’s even an Emacs client.

I’m very happy with mu and mu4e and don’t have any urge to try something else but if they don’t work for you or if you aren’t an Emacs user, notmuch may be a good solution. It works with Mutt, has its own Emacs client, and, of course, works nicely with Astroid. If you’d like to know more about notmuch, Carl Worth, the notmuch developer, has a video that explains its rationale.

As far as I can see, notmuch and mu are pretty much similar except for tags. Notmuch uses tags as one of its principal organizing methods in addition to search. You can, if you like, essentially recreate a folder structure using tags but I’m not sure why you’d want to. Mu, on the other hand, doesn’t really support tags. You can add a tagging function similar to Gmail but there’re apparently problems with Email servers that aren’t Gmail. In my experience, searching is really all you need, especially since the searching in both applications is excellent and fine grained.

The video is from 2011 when notmuch was still young so there’s been many improvements since then but it’s still useful for giving you an idea of what notmuch can do and why Worth developed it. The video is about 45 minutes long so plan accordingly.

Posted in General | Tagged | Leave a comment

How to Read the Commentary Section of Emacs Packages

Alexandro de Oliveira has a problem that I’ve also experienced. He wants to know how he can read a complete description of a package such as that given on its Elpa page. He asked on Stack Exchange and gave as an example the documentation for undo-tree. I’ve had that exact problem. I’m always forgetting some of the undo-tree keystrokes and that means I have to bring up the source to read the commentary at the front of the source file.

It turns out that there’s a much better way: as suggested by wasamasa, simply use finder-commentary to extract the commentary from the source file. That function is part of the finder package written by Eric Raymond back in 1992 so it’s been around for a long time but this is the first time I’ve heard about it.

If you’re writing a package, this tip should encourage you to spend some time on the commentary section. It can really be useful to your users, especially if the package’s UI is the least bit complicated. As usual, I’m always learning something new about Emacs.

Posted in General | Tagged | 4 Comments

The FBI Proves the Iron Law of Data Collection

About a year ago, I wrote about the FBI’s use of §702 data—supposedly collected to fight terrorism—for routine law enforcement having nothing to do with terrorism. I called it another confirmation of the Iron Law of Data Collection: the idea that once data is collected it will be used for purposes other than the reason it was collected and that, finally, it will be abused.

Newly unclassified FISA Court transcripts show that the situation is much worse than we thought. So much worse that even the FISA Court is worried that things might have gone too far. You should definitely read the Circa article linked above but among other things they report that

  • The FBI illegally shared §702 data with third parties not entitled to have it.
  • The FBI routinely trawls through the NSA’s raw data with no concern for the legally mandated minimization procedures.
  • The FBI collected privileged conversations between attorneys and their clients.

All of this despite the fact that James Comey testified before Congress that “Nobody gets to see FISA information of any kind unless they’ve had the appropriate training and have the appropriate oversight.” The FBI has consistently denied that they’re doing any of this but the FISA documents show that this is untrue.

The §702 legislation is expiring this year and for the first time there appears to be bipartisan skepticism about renewing it. Probably not enough skepticism to prevent renewal but perhaps enough to write firm restrictions on how it’s used into the law.

Posted in General | Tagged | Leave a comment

An Update on Microsoft’s Move to Git

A few months ago, I wrote about Microsoft’s move to git and the scaling problems they were having to overcome. Now Brian Harry has posted an update on their work. They have what they claim is the largest git repository on the planet. At 3.5 million files and 300 GB, there’s no reason to doubt that description.

When Harry first wrote about the move back in February, they had a working system but hadn’t yet moved an appreciable number of engineers over to it. As a result, they didn’t know how well it would scale when thousands of Windows engineers started using it. They now have about 3,500 of the roughly 4,000 Windows engineers switched over and have collected some useful performance statistics. They also discovered that some of their assumptions about how to speed things up were off a bit and have since fixed them.

I’ll let you read Harry’s post (or this Ars Technica article) to see where they stand now. The TL;DR is that the speeds are similar to their old system but they are still working to improve them. One huge improvement is, as you’d expect from git, in branching. That went from an ordeal almost too painful to endure to a simple and easy process. The other news is that Microsoft is making their work available to the public and is moving to an open development model where users can contribute if they wish.

Give Harry’s new post a read. You almost certainly don’t have the problems with your repository that Microsoft does with theirs but it’s interesting none-the-less and shows what can be done to use git with large code bases.

Posted in General | Tagged | 1 Comment

Fontifying Broken Org Links

John Kitchin answers an Emacs Stack Exchange question and provides a handy tip. The question was how to display a broken Org link in the Org-warning face. That turns out to be really simple. If you’re dealing with particularly volatile links, this may be useful to you. Or even if you aren’t, I suppose.

Posted in General | Tagged , | 1 Comment