Irreal

Cory Doctorow has a splendid article in Publishers Weekly about the futility of DRM in the publishing industry. In it he compares the industry’s belief in DRM to the disastrous beliefs of Trofim Lysenko, the Stalin-era Russian pseudo-geneticist, that led to mass starvation in the Soviet Union. In some sense, there’s not much point in telling this to Irreal readers: they all know the arguments and despise DRM already. Still it’s a good article and points out some interesting facts.

Most of the line staff at the publishers know, like Stalin-era geneticists knew about Lysenkoism, that DRM won’t work and that it’s really hurting the publishing industry while failing to do anything to stop illegal copying. The problem is that their leaders, like Stalin, do believe in it and refuse to consider the possibility that their faith in DRM might be misplaced.

Doctorow tells the story of how one publisher is planning on sending all its ARCs out as encrypted PDFs. ARCs are the Advanced Reader Copies that publishers send to reviewers before the book is actually published in the hopes of getting a good review. As Doctorow remarks, most reviewers get far more requests for a review than they can honor (Doctorow says 100-to-1 in his case) and that the idea a reviewer would put up with the annoyance of dealing with a “book” that couldn’t be loaded into whatever e-reader he happens to use or couldn’t be conveniently read on a plane, say, is wishful thinking. Instead, those crippled PDFs will be ignored and the publisher and author will miss out on advanced publicity.

Again, there’s nothing in the article that we don’t all know but it’s a good take on the problem with some interesting inside information. Well worth a read.

Regular Irreal readers know that publishing and its travails are an enduring interest of mine. As I’ve said many times, I like publishers and wish them well but at the same time I recognize that they are making the same tragic mistakes that the music industry did. Unfortunately, they are less well positioned to survive them.

Case in point: TechCrunch has an article claiming that Amazon is purposely killing off rival publishers by giving authors huge advances and very favorable deals with the intention of driving the traditional publishers into bankruptcy. Although it’s not well known, Amazon is into publishing in a big way and has several of its own imprints. TechCrunch goes on to say that Amazon enjoys a number of benefits such as inexpensive print-on-demand facilities and, of course, the Kindle installed base.

All-in-all, the TechCrunch article is pretty depressing not only for the publishers but for those of us who think it a good thing for there to be more than one publishing outlet. It is, I think, dangerous to have only one source of published works but that’s the direction we’re headed in. I hope the traditional publishers can adjust to the times and improve their prospects but I have to say that I am less hopeful than ever.

H S Dent has a set of endlessly fascinating demand curves that show the demand for various kinds of consumer goods by the age of the consumer. Many of these are pretty much what you’d expect: demand for bicycles, for example, grows until about age 40 and then drops off to nothing a little after age 80. Still, there’s a lot of surprises.

Children’s clothing (of all sorts) drops off with increasing age as you’d expect but then rises again at a little after age 80. You see this again and again in everything from boys’ underwear to girls uniforms. I have no idea why this is. Oddly, adult clothing doesn’t follow the same pattern: you generally see the expected drop off with increasing age. Not all the curves show an increase to some age and then a decrease. Some rise and fall repeatedly throughout the age span—college tuition is an example of this. Some, like safe deposit box rental and prescription drugs shows a steady increase throughout the age span.

I doubt any of this is of particular importance to the typical Irreal reader but I find the graphs entertaining and in some cases puzzling. It well worth a look—you may be surprised at how interesting they can be.

As long time readers know, I am a big fan of Russ Cox’s work. In particular, his 3 part series on implementing regular expression search is just outstanding. Now Cox has added a fourth article to the series that describes how he implemented Google Code Search.

There is, of course, too much code on the Web to do a grep over the entire corpus. Instead, they built a trigram index of the documents and then analyzed each regexp to decide which trigrams would need to be present in a document to satisfy the regexp. Then they used the trigram index to retrieve and grep just those documents. Cox’s paper discusses how they did the analysis—it more complicated than you might imagine—and presents some results on a smaller version for which the code is available. (Google discontinued Code Search so the standalone version is all there is).

One example is searching for “hello world” in the 3.1.3 Linux kernel. There are 36,972 files to search but only 25 of them contain the necessary trigrams: “ wo”, “ell”, “hel”, “llo”, “lo ”, “o w”, “orl”, “rld”, “wor”. That cuts the search time by 2 orders of magnitude.

This paper is a beautiful example of using some elementary data structures and some smart analysis to bring a huge problem down to a manageable size. I really recommend the article for anyone interested in programming.