AWK Examples

Posted on 2017-10-26 by jcs

Before Perl and other such scripting languages, there was AWK. AWK is a great language for doing text manipulation chores. It has regular expressions, associative arrays, automatic splitting of input lines into fields, and other features to make working with text easy. The traditional reference for the language is The AWK Programming Language. It’s out of print now but you can still get a copy if you don’t mind spending an exorbitant amount. There are, of course, PDF copies available but I have no idea of their legal status.

Even if you can’t get a copy of the book, there’s the excellent The GNU Awk User’s Guide that serves as a reference manual for the GNU’s AWK implementation. If you’re looking for a quick go-by or set of recipes, Sundeep Agarwal has a nice set of AWK examples. The examples make it easy to get going and use AWK in your everyday work.

AWK excels in “one-liners” and is therefore often used in pipelines. It can also be used to build larger programs. Before Org mode, I used an AWK program to take a list of tasks and the times spent on them to typeset a time sheet with Groff. I still prefer it to Perl for most jobs that I can’t do with Org mode and Emacs. If you’re looking for a quick and easy introduction to AWK, take a look at Agarwal’s examples.

Posted in General | Tagged , , | Leave a comment

Tables and Org Mode Code Blocks

Posted on 2017-10-25 by jcs

I have lots of Org files that collect data and have an embedded code block to generate a summary table. Here are a couple of examples that others may find useful. The first thing to know is that tables are stored as a list of rows with each row being a list of the row’s elements. If we think of a table as an \(m \times n\) matrix, the table is stored as

\begin{matrix}
((T_{1,1} T_{1,2} \cdots T_{1,n})
(T_{2,1} T_{2,2} \cdots T_{2,n})
\cdots
(T_{m,1} T_{m,2} \cdots T_{m,n}))
\end{matrix}

It’s pretty easy to build a table from scratch and make it the output of a code block. As a simple example, suppose you have a long table of 3 columns that tracks data over time. You don’t want to export the whole table when you output the document because a lot of the data is old and not that interesting anymore. Let’s suppose that the table is named (via #+NAME:) “data-table”. Here is a code block to output the last 3 entries. 

#+BEGIN_SRC emacs-lisp :var tab=data-table :exports results :results table
  (cons '("Col. 1" "Col. 2" "Col. 3") (cons 'hline (last tab 3)))
#+END_SRC

The (last tab 3) generates a list of the last 3 rows. The two cons’s put the header and horizontal line on the front of the table. The result is

Col. 1 Col. 2 Col. 3
1 2 3
4 5 6
7 8 9

As a second example, suppose we want to use the entire input table for our calculations. Org mode is especially intelligent about this. By default, if you have a header (indicated by the horizontal line), Org will remove the header before sending the table to your code block. It’s Emacs so of course that’s configurable. You can control what happens to the header with the :colnames header argument but most of the time you’ll want the default action.

Now suppose we want to build a summary table containing the low and high values of columns 2 and 3. We can do something like 

#+BEGIN_SRC emacs-lisp :var tab=data-table :exports results :results table
  (let (col2-low col2-high col3-low col3-high)
    ;; routine calculations to set col1-low, etc.
    `((" " "Low" "High") hline
      ("Col. 2" ,col2-low ,col2-high)
      ("Col. 3" ,col3-low ,col3-hi)))
#+END_SRC

Assuming the Low and High values are as shown, the last quasiquoted expression is output as the table

  Low High
Col. 2 2 8
Col. 3 3 9

The nice thing is that the “routine calculations” don’t have to worry about the header when accessing the data; it’s as if it didn’t exists.

Posted in General | Tagged , | 3 Comments

EXWM

Posted on 2017-10-24 by jcs

I mentioned the Emacs X Window Manager (WXWM) in a previous post but didn’t say very much about it. Now, Phil Hagelberg has a post in which he explains his use case for EXWM. Like me and many other Emacs users, Hagelberg likes to stay in Emacs as much as he can. He even remaps his Firefox keys to be consistent with Emacs. Sadly, the next Firefox version will not support the plugin that he uses to accomplish this and he was wondering what he should do.

He discovered that EXWM has a feature called “simulation keys” that allows him to do the same thing. So, even though he had always thought of EXWM as something of a toy, he installed and started using it. He’s been very happy with it and other Irreal readers who like staying in the Emacs experience as much as possible should read his post. Perhaps it will work for you too.

As I said in my previous post, I’m a macOS user so EXWM isn’t available to me but macOS has built-in key remapping so Emacs keys work globally for me. Still, I’d like to try EXWM just to see what the experience is like. If you’re using an X-based GUI front end, you can try it out if you’re curious.

Posted in General | Tagged | Leave a comment

The Unix Philosophy

Posted on 2017-10-23 by jcs

For many people, the Unix Philosophy is captured by the aphorism, “Programs should do one thing well.” That’s an important part of the Unix philosophy but the actual philosophy is much richer and more subtle. Actually, there’s no formal philosophy but rather it comprises a series of zen-like mandates contributed by several people over a series of years.

The other day I wrote about an excerpt from Eric Raymond’s The Art of Unix Programming that discussed Unix history. Today, I want to mention another section from that book that discusses the basics of the Unix philosophy. It’s a really excellent distillation of the wisdom gained by the early Unix developers. It’s still important today because it points to ways to write simpler, more robust programs—ideas that are needed more than ever in today’s environment.

Esr abstracts 17 rules from the writings and practices of the Unix elders. If you’re writing software, I urge you to spend 10 minutes reading the section from esr’s book. If you take its lessons to heart, you will write better, more easily maintained programs.

Posted in General | Tagged , | 1 Comment

Fraud in Digital Advertising

Posted on 2017-10-22 by jcs

I’ve written many times (1, 2, 3, 4) about adtech and the associated fraud. The sooner the advertising industry stops tracking us the better it will be for them, the advertisers, and us. A recent BuzzFeed article indicates that the problem is much more widespread and serious than most of us believed. It turns out the fraud is rampant in digital advertising and even “legitimate” players are aware of and profiting from it. The article is long but it’s worth a read.

Tracking is infuriating and needs to end but this article calls digital advertising itself into question. In addition to the fraud, many advertisers are finding that digital advertising is largely ineffective. Proctor and Gamble, for example, recently decided to forego an large advertising buy and found it made little difference to its business.

All this got me thinking about what happens if advertisers give up on digital advertising altogether. Will the content that we enjoy and have come to depend on disappear too? Probably not because the Internet is eating all the other content delivery systems so advertisers don’t have anywhere else to go. What has to change is that the advertising agencies have to clean up their act and provide ads that consumers are willing to watch or read and to provide real value to their clients, the advertisers. It’s pretty clear that that means—just for starters—no more tracking and no more fraud.

Posted in General | Tagged | Leave a comment

Practical GPG

Posted on 2017-10-21 by jcs

Over at Hackaday, Pedro Umbelino has a nice article on the practical mechanics of using GPG and, more generally, public key cryptography. Rather than looking at its integration into a particular email client, Umbelino demonstrates everything on the command line. That’s handy because sometimes you want to sign, verify, or encrypt documents that aren’t email. For me, this most often this comes up when I want to verify a signed software download, such as the latest Emacs distribution from GNU.

The hardest part of public key cryptography is the verification of public keys. It’s almost certainly the reason that encrypted emails have never gained traction. Umbelino’s article covers this in more depth than most similar posts but at the end of the day, you’re pretty much stuck with the web of trust and pubic key servers. If you have a few people with whom you need to exchange encrypted documents, you can sign each others keys and the system works well.

If you’ve wanted to try public key cryptography, Umbelino’s article provides a good go-by. You’ll probably want to explore how it’s handled by your email client because unless you’re using it only to verify signed downloads, you’ll likely use it mostly with email. The article is definitely worth a read.

Posted in General | Tagged | 2 Comments

Lists in Word

Posted on 2017-10-20 by jcs

So true. So very, very true:

Note: Unfortunately, the embedded tweet has the formatting messed up a bit. Click on it to see the original.

Posted in General | Tagged , | 2 Comments

Which-key and Calc

Posted on 2017-10-19 by jcs

I ran across this excellent video on Emacs calc and was preparing to write about it. During my research, I discovered that I’d already written about it. That was a couple of years ago so you can consider this a reminder to watch it if you haven’t already. Calc is much much more than a simple calculator and can do some pretty astounding things. Karthik C covers only a few of them but the video does serve to show you powerful it is. For example, if it’s been a few years since you last sat in a Calculus class and you need to know the indefinite integral \(\int x^{2}e^{x}\,dx\), just power up calc, type in x^2 exp(x), press a i and discover the integral is \(x^{2}e^x+2e^{x}-2xe^{x}\). As I say, watch the video. If you want to start using calc, you should also download a copy of Sue D. Nymme’s cheat sheet. It’s, by far, the best one I know of.

Once I discovered that I’d already written about the video, I abandoned the post but started playing around with calc just for fun. It was then that I discovered something new. If you’re familiar with calc or watch the video, you know that you can get help by hitting a prefix key and ?. For example, if you can’t remember how to integrate, you can type a ? to get a list of options. However, with which-key installed, that all happens automatically. If you type a and pause before hitting another key, which-key will pop up one of its normal help buffers with all the options. That very nice and reason enough to install which-key if you’re a calc user. Actually, unless you have a photographic memory you should install which-key.

Posted in General | Tagged | Leave a comment

Emacs Line Modes

Posted on 2017-10-18 by jcs

LigerLearn has another nice video up. This time it’s about line wrapping, line truncation, and visual line mode. If you’ve ever been confused about visual line mode and what it’s for, this video will help clear things up for you.

The video begins with a demonstration of how many Emacs commands don’t act in the expected way when operating on long lines that are wrapped on the screen. It then goes on to show how visual line mode solves these problems and gives you a better user experience both visually and operationally.

The video is short—5 minutes, 40 seconds—so it’s easy to find time for it. If your understanding of visual line mode is that it just automatically adjusts line wrapping as you edit, you should take a few minutes to watch this video.

Posted in General | Tagged | Leave a comment

Remote Sudo

Posted on 2017-10-17 by jcs

If your workflow involves administering remote systems or something similar, you probably have the following burned into your muscle memory. If, on the other hand, you’re like me and don’t have occasion to need root access on a remote machine very often, you can probably use a reminder. If you’re in the second category, here, courtesy of abo-abo, you go:

If you click on the tweet, you’ll see the next tweet, which explains that “cloud” is the name of a remote system defined in abo-abo’s ~/.ssh/config. Defining often used remote systems like this is something I do too and it saves me a bunch of time and mental cycles trying to remember domains or, worse yet, IP addresses.

Posted in General | Tagged | 8 Comments