GNU things are GNU - Simplicity of GNU Make

GNU things are GNU. If you're a hacker you'll understand that this is meant to be a fun interjection. It really depends on who reads this what you'll make of that interjection. As a kernel hacker perhaps you'll cringe and maybe even cry yourself to sleep in fetal position (as I think Johannes Berg [G+] may have today) of the thought of the completion of a project such as the GNU Hurd. As a GNU'er maybe you'll pride yourself with the amount of documentation, portability, and righteousness and of course recursion after the implications of such a statement. If you're not a hacker just understand that GNU has a bible and a church.

As reported by Hauke Mehrtens, I recently broke parallel building in compat, a Linux kernel backport module, when I added autoconf.h-like support support for it. The autoconf.h support allows us to push build time configuration options onto the C and header files we are using at build time. GNU Make typically builds things serially but if you ask it to build things in parallel, by specifying the -j command argument, GNU Make will build all objects through independent tasks and later combine them. For any modern system this is crucial otherwise you system's resources will not be used efficiently at build time and each object will only build until the last one will have finished, even though each one could have been built independently. The component that manages parallel building in GNU Make is called, the jobserver.

I'll confess that I have been avoiding looking into how GNU Make jobserver works given that the man page really doesn't provide much details about its implementation nor does it clarify exactly what that pesky "+" thing is when it fails on using the job server:

hauke@hauke:~/compat-wireless/compat-wireless$ make -j5
make[1]: warning: jobserver unavailable: using -j1.  Add `+' to parent
make rule.

Peter D. Smith is the maintainer of GNU Make and I'm fortunate to have found an entry on his page where he documents in careful detail history and evolution of the implementation details of GNU Make's jobserver. After reading that I now appreciate GNU Make a bit more in consideration for all the "GNU" objectives it strives for, and it being a GNU project utility. Felix Fietkau tells me that this appreciation says more about other GNU utilities than about GNU Make... perhaps he's right but nevertheless I now really like the simplicity of GNU Make's manpage, its apparent simplicity, and its side little optimizations that I have found after mucking around with it a bit more on my hunt to understand its jobserver.

I hate long blog entries so I'm going to now summarize very quickly for hackers my findings, if you're not a hacker, stop reading here.



The jobserver works using pipes, and the jobserver will barf if its heuristics implementation fails at understanding if a target has a command that may not be another make process (called sub-make process, in GNU Make language). The heuristics implementation is very simple, this means GNU Make is inherently dumb and you need to guide it, the only thing it knows is itself, and to the GNU pride even about recursive calls against itself. The job server will know what to do if a sub-make process is issued with the same binary, to be clear the same make binary -- and this is pretty darn cool. You make emphasis that a sub-process is an explicit GNU Make sub-process and ensure that this propagates properly by using the MAKE variable. It is imperative that sub-processes on a target that are not identical GNU Make sub-processes are annotated as such, you do this by pre-pending a "+" to each command that is not a GNU Make sub-process. Since the jobserver is braindead stupid by design GNU Make will halt the jobserver completely when issuing these specific commands.



What if you have a project that needs some header file but you want that header file to be generated through the build process automatically ? GNU Make is dumb but fortunately smart enough to look for header files that are being included on targets, even if the target is not listed as a dependency to a final required build target. GNU Make is also dumb but smart enough to know that if it had to build a header file that was being included it will call itself again, automatically after building the required target file, in fact it will not fatally fail in this case even if you asked it to fail (by not specifying that the include file can be missing by prefexing the include command with the "-"). This is really cool and shows some real GNU recursive philosophy in mind.

Now to the stupid part. When trying to optimize your Makefiles take some time to review all the ways that you have left GNU Make go astray and be stupid by looking at the make -d output. What I found was that there were tons of implicit checks on targets that needed to be generated, this included even the god damn Makefile itself and possible derivatives of it. I counted 18 stupid derivatives. The simple solution is to optimize build time by adding to .PHONY even your own Makefile.

If you'd like to see examples you can review my jobserver and general GNU Make optimization patches to compat. These patches illustrate all these optimizations at the Linux kernel module level, along with some build time optimizations for external Linux kernel modules. I hope someone out there appreciates the recursion between the link to this blog post on the patches posted and the linking from this blog post to those patches. I wonder what Google calls this when scraping the web, there must be a name for it.