I taught myself LATEX this summer to get some lecture notes in shape. Having originally used the LATEX Book class I have now moved to the memoir class and can thoroughly recomend it. The attached is a two page spread chosen pretty much at random and it shows that you can integrate text, math and figures easily with a pretty professional looking book layout that is the default in memoir.
The learning curve is reasonable and all the fancy headers, figure numbering, contents etc etc defaults are well chosen. The TEX set up on a PC was easy and free - I use the LED editor and the MikTex 2.7 package.
This is typeset onto B5 paper in PDF format and I have now found a pretty good quality short print run digital printers in the UK (this is not POD which is generally low quality, the shortest print run is 50 copies) to produce final book output. I'll keep you posted how the whole process goes.
The attached JPG is deliberately low resolution - its mainly to show the two page spread.