Optimising compile times with precompiled headers

Inside niven, I've been using precompiled headers since the first version. Read on for some insights I gained over the years.

Platform

The following is written while I was using the Microsoft Visual C++ 8.0 compiler. It is probably also true for earlier versions, but I cannot check that.

Start point

niven had precompiled headers right from the beginning. They used to contain often used internal headers and large parts of the STL (map,vector, etc.). Over the time, a lot of headers from Boost found their way into the precompiled header. The last version I had compiled to a 47 MiB large .pch file. Compilation time was roughly one minute for the core library, which was already pretty good given the size and complexity of the source code. I've been using the best practices for physically decoupling the code. This means each header forward declared as much as possible and includes were delayed so most of the happened in the .cpp file (where they should be). In addition, often used headers were put into a single precompiled header and compiled just once for the application.

The Fast & the Curious

Well, as always, fast was not fast enough for me (1 minute for a full recompile on my notebook - as if I would have a spare minute :) ). I've taken a deep look at what should really go into the precompiled header, commenting everything out first. Step by step, I could strip down approximately half of my precompiled header, reducing the compile time down to 37 seconds - now my notebook is just as fast as my desktop previously. The .pch file is now 30 MiB - a 33% reduction.

Lessons learned

The most expensive headers are those taking huge parts of the STL and other template magic with them. For example, including <boost/bind.hpp> can easily double the compile time per source file! Compared to this, files with a lot of preprocessor magic (in my case, <boost/signals.hpp>) don't make a visible difference. I've moved them out and back of the precompiled header, but I could not measure any improvement - seems the preprocessor is really fast. Also, be aware that some headers tend to include stuff you don't think about usually. For example, <boost/signals.hpp> includes <list> - after removing the former from the precompiled header, some parts of my code complained about not finding the latter. An interesting result of this is that eventually, the compiler is able to process a lot of files per second now. You don't see individual files popping up in the output window but batches of three or even more files - woohoo! Link time and precompiled header generation is now dominating the build time. I presume most of it is spent in the interprocedural optimization step, as the core library is linking basically against the bare minimum you have to link (C-Runtime, C++-Runtime, some Windows specific libraries). As soon as I get a chance, I'll try with the Intel C++ 9.1 Compiler to see how big the difference is over there.

Virtualization and updates

Since monday, I've become a pretty happy virtualization user. Background: At university, our latest homework is released in partly binary form (for x86); moreover, it is configured to work on university computers only. It is recommended to work on it directly on a university computer ... Another annoyance was that - just for using X-forwarding over SSH - I had previously to reboot my computer in order to run Linux, as I'm a usually running windows.

A long and winding road

The first and obvious solution was to get Cygwin, which comes with a built-in X-server. It works, quite good even, but somehow the X-Server wasn't stable on my machine (sometimes silently crashing). In order to start it reliably, I had to log-off and log-in again and then run the X-Server. Hmpf! There had to be something better. Well, and there is: What I wanted was the comfort of a real Linux without the hassle of rebooting just because of it - so virtualization was the way to go. My first try was Microsoft Virtual PC. Unfortunately, neither the 2004 SP1 nor the 2007 Beta worked as I wanted. Both had serious problems (graphics stretched, various Linux distributions refused to install at all). I presume they're better suited at virtualizing Windows. The next logical choice was VMware. They provide their host runtime (the VMware Player) for free. All you need to use it is a tool which creates .vmx files that contain the virtual machine settings (actually, they're simple text files, so you could write them by hand, but I'm lazy ;) ). I've found the free VMX Builder, which is a simple GUI similar to VMware Workstation. A friend suggested to use the - also free - VMware Server for creating the files (ask him about that solution). Unlike Virtual PC, VMware worked right out-of-the-box with Fedora Core 6. Performance is surprisingly good (running on a 1.8 GHz Turion Notebook with 1 GiB RAM with 320 MiB dedicated to the VM). The full installation took roughly 30 minutes and uses 3 GiB.

Updates

Adobe has chosen to fix a security problem in the Adobe Reader 7.0.8 the hard way - they've released the Reader 8.0. It sports a new GUI and a much faster start-up compared to 7.x.

Working on ...

... some stuff with OpenMP - as soon as I have some more experience with it I'll blog about it here. VPlan is more or less ready for release and just waiting (hope to get it done during the next week). Some rather scary homework is still waiting untouched; and my maths homework is pleading for a second and third look. Moreover, I'm reading a big pile of books and trying some new things in C++ which actually deserve an own post on their own. In case you did not notice it yet - I'm busy as usual :) Short side note, I've got some brand new music from Christina Aguilera (Back to basics - a few astonishing songs there), Faithless (Forever faithless - "I can't get no sleep") and Pink (I'm not dead - yeah, absolutely). I can recommend each of those, but if you have to choose one, try "Forever faithless" - you won't regret it (as long as you have a decent subwoofer).

niven build scripts updated

I finally managed to get some time to update the niven build scripts (possibly 50% rewritten ;) ). Now, it still supports Windows only, but it will support x64 and x86. The x64 support is currently held back until Boost 1.34 comes out. I've also used a conditional statement in Python for the first time (suffix = 'd' if conf == 'debug' else '') :)