Script Kiddies At Work

Kura over at SYSLOG has a really great animation showing the origins of SSH attacks on a single server during a 24 hour period. It’s another reminder, if you needed one, to lock down your machines. It’s pretty obvious these are just scripts trying a range of IP addresses but that makes it worse in a way. These aren’t targeted attacks. They’re automated scripts looking for a vulnerable machine. It’s easy to think, “I’m not a high profile site so no one will bother with me” but these guys will bother anyone foolish enough to leave the door open.

The animation is less than a minute but still fascinating. Recommended.

Posted in General | Tagged | 1 Comment

JSON Versus S-Expressions (Again)

I’ve written several posts concerning the relative merits of JSON vs S-expressions (1, 2, 3, 4) and concluded that while s-expressions are more powerful and flexible, JSON might be a better choice because it’s supported by virtually all programming languages. A couple of days ago, Eli Bendersky published a post that reached similar conclusions. While his conclusions pretty much align with mine his reasoning does not and I disagree with it.

Bendersky approaches the question from the standpoint of parsimony and says that while JSON is a win over XML because it’s more “fat free,” you can’t go further and replace JSON with sexprs to get rid of even more fat because there’s no easy way to handle dictionary structures. He says that the equivalent S-expression to the JSON expression

'[["oranges", 2], ["apples", 6], ["pears" 5]]'

is

'((ORANGES 2) (APPLES 6) (PEARS 5))

which is indeed less punctuation but that as soon as you have the JSON expression

'{"oranges": 2, "apples": 6, "pears": 5}'

S-expressions lose because there’s no way to express a dictionary structure with sexprs without adding abstraction.

That’s certainly not true. Here, for example, is a Common Lisp reader macro that does just that.

(defun |#H-reader| (stream sub-char numarg)
  (declare (ignore sub-char numarg))
  (let* ((kv (read stream t nil t))
         (h (make-hash-table :test 'equal :size (* 2 (length kv)))))
    (mapc (lambda (p)
            (destructuring-bind (key value) p
              (setf (gethash (symbol-name key) h) value)))
          kv)
    h))

(set-dispatch-macro-character #\# #\H #'|#H-reader|)

Notice how short and easy it is to write. Now we can treat that sexpr as a dictionary simply by prepending #H.

CL-USER> (setq dict #H((oranges 2) (apples 6) (pears 5)))
#<HASH-TABLE :TEST EQUAL size 3/31 #x3020009740CD>
CL-USER> (gethash "APPLES" dict)
6
T

Or we can prepend ' and treat it as an association list or simply an unordered list.

“But I don’t use Lisp” I hear you say. I could respond that your choice to use a less powerful language is not my problem but I won’t do that precisely because that’s the reason JSON may make more sense: it’s supported in all programming languages.

Bendersky remarks that he can read

'{"oranges": 2, "apples": 6, "pears": 5}'

directly into Python or Javascript and get his dictionary-like structure. As I showed above, you can do that in Lisp too. Now what about all those other languages? How is JSON more natural and direct in, say, C++ than an S-expression? Sure, there are JSON libraries for them but that’s adding abstraction and if we can do it for JSON we can also do it for S-expressions.

I believe JSON may be a better solution because it represents a sort of lowest acceptable common denominator. It’s simple, everyone understands it, and there are libraries for every imaginable language. But that doesn’t mean it’s better in some absolute or esthetic sense. It isn’t. S-expressions are more powerful (because of the duality of code and data), more flexible, and have less syntactic sugar.

In any event, Bendersky’s post is interesting and makes some good points. You should really take a look and decide whether you agree with him or not.

Update: mapcar → mapc. The mapcar works, of course, but there’s no reason to build the list of values and then throw it away.

Update 2: associative array → association list

Posted in General | Tagged , | 1 Comment

DMR Stories

Mike Maney over at Maney Digital has a very nice tribute up for Dennis Ritchie. While he was at the Mobile World Congress in Barcelona, Maney talked to some of dmr’s colleagues and captured their words on his Flip. The production values, as he admits, are not great but it is great to hear people who knew and worked with dmr tell stories about him and what he was like. It’s not long at all, so take 4 minutes and give it a look.

Posted in General | 1 Comment

Reversing An Emacs Lisp List In-Place

Yesterday, Xah Lee put up a post that showed how to reverse arrays and lists in various languages, including Emacs Lisp. His Elisp examples worked only for arrays so I added one for lists in the comments. Later I realized that although it was pretty efficient it wasn’t really a solution because it built a separate list.

Here’s a less efficient method that does do the reverse in-place

(let ((list '(a b c d e f g)))
  (do ((i 0 (1+ i))
       (j (1- (length list)) (1- j)))
      ((>= i j) list)
    (rotatef (nth i list) (nth j list))))

If you’re not familiar with (or don’t like) the do macro, here’s the same strategy rendered with a while.

(let* ((list '(a b c d e f g)) (l 0) (r (1- (length list))))
  (while (< l r)
    (rotatef (nth l list) (nth r list))
    (incf l)
    (decf r))
  list)

We can also use rotatef to reverse an array

(let ((vec ["a" "b" "c" "d" "e" "f" "g"]))
  (do ((i 0 (1+ i))
       (j (1- (length vec)) (1- j)))
      ((>= i j) vec)
    (rotatef (aref vec i) (aref vec j))))

Of course, all this is a bit silly because Elisp already has this covered. The best and fastest way to reverse a list in-place is

(let ((list '(a b c d e f g)))
  (nreverse list))
Posted in Programming | Tagged , | 5 Comments

The Emacs set-variable Command

I learned a new Emacs command today. What do you do when you want to set an Emacs variable? Until today my answer was 【Meta+:】 and then

(setq some-variable some-value)

That works well and there’s nothing wrong with it but it’s often faster to use 【Meta+xset-variable instead. When you do that, Emacs will prompt you for the variable and its value. One of the reasons that this may be faster is that you can 【Tab】 to complete variable names.

Posted in General | Tagged | Leave a comment

AT&T To Cap Unlimited Data Plans

The Wall Street Journal is reporting that AT&T is going to cap unlimited data plans. Truth to tell, it’s not much of a change because they were already slowing download speeds for the top 5% of their bandwidth consumers. Now they will slow download speeds for anyone exceeding 3 gigs/5 gigs for 3G/4G users. In view of the study that I wrote about last week that showed there is essentially no difference between the data usage of “the top 5 percent” and the top tier of the metered users, it’s hard to see how this is justified.

I wouldn’t be surprised to see some push back. After all, AT&T sold their customers an unlimited data plan and charged more for it. At the time there was no talk of capping the plans or slowing speeds. Just pay us the extra money please. That arguably amounts to a contract to provide the unlimited service as sold. It certainly amounts to a moral obligation.

If AT&T feels that they can’t make money with their current plans then they’re free not to offer them but they’re not free to simply abrogate those contracts—even partially—simply for their own convenience. And certainly not as a device to force all their customers onto a tiered plan as the researchers that I wrote about last week suspect. Back in the old days, everyone hated Ma Bell. This is the kind of behavior that reminds us why.

Update: According to a Wired Gadget Lab article, the change was in response to customer complaints that the “top 5% rule” was catching people who were using no more data than the tiered customers, who weren’t being throttled.

Posted in General | Leave a comment

Yegge On Dynamic Languages

I stumbled upon a nice talk that Steve Yegge gave at Stanford in 2008. The talk, entitled Dynamic Languages Strike Back, debunks some of the myths about the performance, optimization, and tools for dynamic languages such as Python, Ruby, Perl, and even our favorites such as Lisp and Scheme.

Yegge argues that despite the claims of static typing partisans, you can get better performance and optimization with dynamic languages. That requires a some work, much of it yet to be done, but we are already making great strides. The idea is that you have much better information about program behavior at run time than you do at compile time so better decisions can be made. Yegge remarks that the Java byte compiler, for example, does all sorts of optimization but at run time the VM converts the byte stream back to a tree and throws out all the optimizations in favor of its own, which are superior. The talk is worth listening to just to hear him expound on this point.

Yegge predicts that the languages we have today and their relative popularity will remain pretty much constant for the foreseeable future. It’s too hard for new languages to gain traction even if they are better and many developers today know only one language and resist learning another, suggesting the static ranking.

During the question period, someone asked if we are any better off today than we were with the Lisp machines. Yegge said he didn’t think so and remarked that the only Lisp machine left today is Emacs and despite the fact that Elisp isn’t the world’s best Lisp it still provides an amazingly powerful environment.

This is a great talk and I recommend it to anyone interested in dynamic languages. It’s about 68 minutes so plan accordingly.

Posted in Programming | Leave a comment

ESR: An Open Letter To Chris Dodd

Eric Raymond has posted An Open Letter to Chris Dodd concerning Big Media’s approach to the Internet. It is, I think, an important letter and one that most hackers and Irreal readers will agree with. That’s not the same, of course, as Dodd taking its message to heart or even reading it but I urge you to give it a read if you haven’t already.

Posted in General | Leave a comment

Reproducible Science

Ars Technica has an interesting article up entitled Nature Editorial: If you want reproducible science, the software needs to be open source. I’ve written a couple of times about reproducible research and how Emacs and Org Mode make it particularly easy so I was interested in what the article had to say.

The article talks about an editorial in Nature that strongly urges authors of papers to include the source code for the software that supports their papers. I was a bit taken aback because it would never occur to me not to do this. Perhaps it’s just being part of the open source movement or maybe it’s a reaction to the East Anglia CRU fiasco but if you are doing science then you must enable others to reproduce (and thereby validate) your results. My position—and that of most scientists not doing climate research—is that if you don’t make your data and methodology available then you’re not doing science.

With the really great tools that are available today, like Emacs and Org mode, there really is no excuse not to do this. Keeping everything together in a single file would, in most cases, makes the scientists’ research easier and then there is only a single file to submit to the journal. It has the LaTeX source for the paper itself, research notes, data, and software in it. In some cases involving huge data sets or code bases the author may want to add a link to the actual data or code rather than include it but the principle is the same.

The editorial makes a good case that even offering an executable version of the software is not enough. Often the software plays a fundamental roll in the research and should, itself, be examined for errors or other problems. As Irreal readers know to their sorrow, bugs are a part of any non-toy piece of software so it is not a trivial objection to say that an executable version does not suffice.

One of the alternatives to open sourcing the code used by some journals is to have the author provide a “rigorous” natural language description of it in terms of the algorithms and mathematics used. I’m sure that any Irreal reader would find this a non-starter for obvious reasons but the editorial has a large section that discusses and dismisses this as impractical.

The editorial presents and discusses several objections or difficulties to providing the code but I found only one persuasive: concern on the part of the researchers or their organizations that the code represents a marketable commodity and should therefore be withheld. The editorial authors say that’s a tough problem and that perhaps such papers should be marked as not having the necessary items for complete reproducibility. Actually, I’d be fine with saying, “We don’t accept papers that don’t include everything needed for reproducibility.” In the case of publicly funded research, this should not even come up. The public paid for it, the public owns it. Of course, I’m not holding my breath for that to happen.

Posted in General | Tagged , | 1 Comment

Multiple Editors And Markdown

I’ve written many, many times about how much I love writing in Org mode. I write a plain text file that I can then export to HTML, LaTeX, or several other formats. I can even, theoretically, write an Org mode document with some editor other than Emacs. But Org mode isn’t the only way of doing this. Another is Markdown a similar system that probably inspired the Org mode mark-up syntax.

One of the advantages of Markdown is that you can use it with any editor. Many editors, including Emacs, even have special Markdown modes that enable highlighting and interface with the Markdown script. I’d certainly be using it if I didn’t have Org mode so I read The Markdown Mindset by Hilton Lipschitz over at the The Hiltmon blog with particular interest. Lipschitz gives the usual list of benefits for Markdown: editor agnostic, plain text, flexible, exports to multiple formats, and so on.

But here’s what I don’t get. Lipschitz goes on about how he writes his programming notes using Markdown in BBEdit, his blog posts using Markdown in Byword, his manuals and related documents using Markdown in Scrivener, and his readme files using Markdown in TextMate. I don’t object to him doing this—whatever works for him—but I can tell you for sure that I’d be drooling and babbling incoherently if I tried something like that. After 5 years, I still have flashbacks to Vim and try to go up a line with 【Ctrl+k】 or something similarly hilarious. I simply can’t imagine trying to use that many editors. One of the advantages, to me, of Emacs is that I can stay in the same editing tool almost all the time. On my Mac I even have a special file that gives me (mostly) Emacs keybindings so that everything works pretty much the same even when I’m not in Emacs. Love them or hate them, there’s only one set of bindings to worry about. (Yes, yes, I know that statement gets muddied a little when considering some modes but it’s basically true).

I’d be interested in hearing what others think. Can you easily handle multiple editors routinely the way Lipschitz does or does leaving Emacs to use some other editing tool make you uncomfortable and less efficient?

Posted in General | Tagged , | 9 Comments