I’ve always been fascinated by the typesetting process and have a particular fondness for the old mechanical typesetters such as the Linotype. I wrote about that in this post from a couple of years ago. That post has a link to a wonderful film that describes in detail how the Linotype worked. It really was a mechanical marvel.
Over at Practically Efficient Eddie Smith has a long post that describes the special problems that mathematical typesetting presents. If you think of a reasonably complicated mathematical expression, the Cauchy Integral Formula, say, \[f(a)=\frac {1}{2\pi{}i}\oint_\Gamma \frac{f(z)}{z-a}\,dz\] you can see right away why it was such a problem. There are Greek letters, fractions, and special symbols that can take up either multiple lines or have to be set in part of one of those lines. It was difficult and expensive to typeset.
Smith discusses some of the solutions to these problems but until fairly recently, you had a choice of expensive, essentially hand-set type or something that looked ugly on the page. It was that “ugly on the page” part that provoked Don Knuth to take a decade off from writing The Art of Computer Programming and develop \(\TeX\). I have the original version of Volume 2 of AOCP that was produced with the traditional hot lead Monotype typesetter. I remember seeing the revised edition that was produced by an early phototypesetter in a book store. It looked terrible. That book was the impetus for \(\TeX\).
Now, of course, typesetting mathematics isn’t much harder than typesetting anything else. \(\TeX\) and especially \(\LaTeX\) have made it so easy that many, or maybe even most, authors typeset their work themselves. The publisher may tweak the \(\LaTeX\) a bit but essentially all the hard work is already done.