Build systems: FASTbuild

The last weeks we've been steadily increasing the complexity of our build tools -- from bare-bones Make to WAF which had high-level concepts like include directory export. This week, we're going back to the roots again with FASTBuild -- you might have seen it Ubisoft's presentation at CppCon 2014.

Overview

FASTBuild is in many ways similar to Make. It's a very focused tool for low-level execution, without much fluff on top of it. The main difference between FASTBuild and the other tools we've looked at so far is the highly hierarchical configuration setup. In FASTBuild, you build up your configuration step by step, adding all options directly like in Make.

Project files & syntax

FASTBuild uses a custom language for build files. It's a very bare-bones languages, providing structured programming through #if similar to a C preprocessor, and various ways to manipulate variables.

The key concepts are inheritance and composition. This allows scaling to large builds by reusing a lot of the configuration without having global state that applies to all projects. Let's look at how you specify two different target platforms in FASTBuild. Instead of a single option you need to set, you'd specify multiple configurations:

.ConfigX86 =
[
    .Compiler = "compilers/x86/cl.exe"
    .ConfigName = "x86"
]

.ConfigX64 =
[
    .Compiler = "compilers/x64/cl.exe"
    .ConfigName = "x64"
]

.Configs =
{
    .ConfigX86,
    .ConfigX64
}

This sets up two structures and an array Configs containing them both. Later on, you'd pass around the configuration array to your targets, and just iterate over the array, building a target with each of them manually. This approach allows FASTBuild to generate multiple platforms and configurations at the same time.

Functions

Functions provide the actual node graph in FASTBuild. For instance, calling the function Executable will register a new executable to be built, which can depend on different libraries. The build graph is built from those dependencies and cannot be expressed directly in the build file language -- there's no way to create a new build node without resorting to C++ and extending the core of FASTBuild.

... and more

So far, FASTBuild sounds like a very bare-bones replacement of Make, but there are various interesting capabilities built into FASTBuild: Distribution, parallel execution and caching. All of these are related: As FASTBuild knows all dependencies precisely, and has a global understanding of the project, it can automatically distribute the build across multiple cores and even nodes in a network, and it can also cache things which don't require to be rebuild.

Sample project

For FASTBuild, we're going to start with the root build file which contains all the project configuration. As mentioned above, FASTBuild is very explicit about all settings and it requires quite a bit of setup before it can get going. The sample project supports only Windows, and only Visual Studio 2015, but all those settings would be in a separate file for a production build. Building a project on Windows requires a lot of options to be passed to the compiler, I'm picking just the compiler options here as an example:

.CompilerOptions    = '"%1"' // Input
                + ' /Fo"%2"' // Output
                + ' /Z7' // Debug format (in .obj)
                + ' /c' // Compile only
                + ' /nologo' // No compiler spam
                + ' /W4' // Warning level 4
                + ' /WX' // Warnings as errors

Here we can see build-time variable substitutions at work, FASTBuild will replace %1 with the name of the input file automatically.

Slightly down below, you'll notice the first time I'm taking advantage of the structured nature of FASTBuild to specify the DLLOptions. Those are simply the default options, but with minor tweaks:

.DLLOptions =
[
    .CompilerOptions        + ' /DLL /MT'
    .LinkerOptions          + ' /DLL'
]

Later on, we'll see how this way of setting things comes in handy, but let's start with the static library. It consists of two nodes -- an Exec node which invokes Python to generate the table, and a Library node which requires the table to be generated already. The Exec node is very similar to a basic Make rule:

Exec ("tablegen")
{
    .ExecExecutable = "C:\Program Files\Python 3.5\python.exe"
    .ExecInput = "statlib\tablegen.py"
    .ExecOutput = "statlib\table.cpp"
    .ExecArguments = "%1"
    .ExecUseStdOutAsOutput = true
}

The dynamic library is the first one where we're going to use a structure to pass in parameters. Instead of setting the linker options directly, we just pull in the DLLOptions we defined above using the Using command:

DLL("dynlib")
{
    Using (.DLLOptions)

    .LinkerOutput = "dynlib.dll"
    .Libraries = {"statlib" ,"dynlib-obj"}
}

We could have written .LinkerOptions + ' /DLL' as well, but then we'd have to duplicate it everywhere in our project where we want to build a shared library. Notice that the dependency to the static library is established directly by name, but we still need to manually set the include path as there's no communication between targets by default.

Finally, the executable itself has nothing surprising any more, and our sample project is complete (and as usual, available in the repository for your viewing pleasure.) I'm a bit on the edge regarding FASTBuild -- I like the fact that it's very focused on building C++ code fast, but I wish it would allow for some more flexibility and extensibility. For instance, it would be interesting to be able to define new functions in the build language. Even if slower than the built-in nodes, this would make it possible to build more complex tasks going beyond the simple property setting & inheritance which is at the core right now.

That's all for this week, I hope you liked it, and next week we'll go into the completely opposite direction and look at a very high-level build tool. Stay tuned!

Build systems: Waf

Another week, another build system -- this week, we're going to look at Waf. It's another Python-based build system, and it's one with a few new concepts, so let's dive into it right away.

Overview

The key idea behind Waf is to have the build split up into separate phases: Configure, then build. Waf is not the first build system ever with a configure step -- that honor probably goes to automake -- but it's the first in this series, and as such, we'll take the opportunity to investigate what configure is good for.

The idea behind a separate configure step is that there's a lot of setup code which only needs to be run once to identify the platform. Think of searching for header files and setting build definitions based on this, which usually requires firing up a compiler just to store the result whether it worked or not. Same goes for platform-specific file generation -- what's the name of the operating system, what configuration options did the user specify, and so on.

Instead of running this every time you build, configure runs once, does all the expensive work and persists it, and subsequent build calls don't even need to execute the configuration steps.

Project files

Waf project files are plain Python scripts, same as we saw in SCons last week. The key difference is that we need to expose a couple of functions per script which are called by Waf, instead of typing in our commands directly and calling Waf on our own.

We need at least a configure and a build function, and both get a context object passed into them. During configure, we'll do the usual things like setting up options for the compiler. This is straightforward as the environment exposes the flags directly, and we can manipulate them using Python, which is very well suited for dictionaries and lists.

Task oriented

Waf's execution graph is task-based -- each task is the atomic unit of work which takes some inputs and transforms them into the outputs. Once the tasks are created, then the scheduler takes over and runs the tasks. Integrating new custom build steps is done through new tasks, as we'll see below in the sample project. One great feature of the Waf scheduler is that it will automatically extract execution dependencies. If we specify a task which produces files, and we have another task which consumes them, Waf will take care of scheduling them in-order without extra work.

C++ support

Waf provides C++ support along other languages like D and Fortran, and it's integrated through language specific tasks. The language integration provides basic build targets, and it has first-class support for C++ specific concepts like include paths. Of all the build systems we've looked at so far, Waf is the first one which allows for true modularity of the build files as it exports & imports include directories across targets.

In one file, we can create a new shared library which specifies the export_includes directory, and if we consume that library elsewhere using use, Waf will take care of setting the correct include directory at the use site. We can thus move around projects freely, and things will just work -- no need to hard-code paths any more.

Sample project

Let's start with the static library project, which requires some code generation. As mentioned above, we'll use a task to generate the lookup table. There's two things to a task, the definition, and then the invocation. Here's the definition:

class tablegen(Task):
    def run(self):
        return self.exec_command ('python {} > {}'.format (
            self.inputs [0].abspath (),
            self.outputs [0].abspath ()
        ))

What we notice right away is that there's no need for special escape variables, the inputs & outputs are specified as member variables. This also implies the task is instantiated for every execution, instead of being reused. This happens very explicitly in the build step:

tg = tablegen (env=ctx.env)
tg.set_inputs (ctx.path.find_resource ('tablegen.py'))
tg.set_outputs (ctx.path.find_or_declare ('table.cpp'))

ctx.add_to_group(tg)

We specify both the inputs and outputs, which initializes the self.inputs, self.outputs members we saw above. Finally, we add it to the current build group, which is the list of tasks that gets executed for this project. Thanks to the auto-discovery of dependencies, it's enough to declare the output table.cpp and use that in the subsequent stlib call to get the execution order right.

The rest is very simple:

ctx.stlib (
    source = 'StaticLibrarySource.cpp table.cpp',
    includes = '.',
    export_includes = '.',
    target = 'statlib',
    cxxflags=ctx.env.CXXFLAGS_cxxshlib)

This adds a static library, uses the specified include directory and source files, gets a name, and exports include directories as explained above. The last line needs a bit more explanation. The stlib has the usual -fPIC problem for Linux, and unfortunately Waf cannot resolve this automatically. We need to specify the flags manually -- the recommended solution is to use the cxxshlib flags for a static library. That's what's happening here in the last line.

The remainder of the sample project is rather boring -- Waf doesn't require a lot of magic to set up. One of the highlights of Waf is that it builds out-of-source by default. All build files, temporaries, and targets get placed into a build folder by default, which is trivial to ignore for source-control systems and avoids polluting the source tree with intermediate files. Even the lookup table we generated above gets built there. There's only one minor caveat here which is: Waf doesn't copy shared libraries into the same folder as the binary, nor does it provide an easy build command line which would set up all paths.

As usual, the sample project is online, and it should work on both Linux and Windows. That's all for this week, see you again next week!

Build systems: SCons

This week we'll cover SCons, a Python-based build system. SCons is a full build system, with built-in support for different programming languages, automated dependency discovery, and other higher-level utilities. Some of you might have heard of it as it has been used in Doom 3 for the Linux builds. That's at least how I heard about SCons for the first time ☺ Apparently, at some point, Frostbite was using SCons as well.

Overview

SCons is written in Python -- and the build files are actually Python scripts. This is great as there's no "built-in" scripting, you can just do whatever you want in Python, and you call SCons like any other library.

SCons at its core is similar bare-bones to Make. You have node objects internally which represent the build graph. Usually, those nodes get generated under-the-hood -- for example, you write a SCons file like this:

Program(['main.c'])

Here, main.c is an Object (as in a C++ object file, not something related to object-oriented programming, mind you). The code above is equivalent to:

Program ([Object ('main.c')])

This creates one C++ Program node, with one C++ Object as the input. You can look at the dependency tree using scons --tree all. For the example above, it would produce something like this:

+-.
+-Sconstruct
+-main
| +-main.o
| | +-main.cpp
| | +-/usr/bin/g++
| +-/usr/bin/g++
+-main.cpp
+-main.o
  +-main.cpp
  +-/usr/bin/g++

That's the full dependency tree. SCons is not timestamp based (at least by default), and the files above will be hashed and checked as dependencies when you ask for a build. When you run the build, you'll see how SCons processes the tree bottom-up:

scons: Building targets ...
g++ -o main.o -c main.cpp
g++ -o main main.o
scons: done building targets.

That's it for the core functionality of SCons. Notice that SCons is aware of all intermediate targets, which means that it can also clean everything by default.

C++ integration

C++ is yet another target language for SCons, and shares functionality with C, D and Fortran. All of them have the same targets and a similar build model. You specify a Program, and based on the list of provided source files, SCons figures out what compiler to use.

There's limited support for discovering libraries and functions. It will search the default paths for it, but I haven't seen a way to register custom search modules. Of course, that's only a minor limitation, as you can just write the search script in Python, but it's a bit disappointing that SCons doesn't come with a set of search scripts.

SCons can optionally generate Visual Studio project files. Optionally, as there's no simple "generate project files" from a given SCons node graph. Instead you build them in a declarative way just like any other target. What you get from SCons is the actual Xml file generation. This makes it easier than writing it from scratch, but it requires still some repetition -- for instance, the source files need to be provided to the target you want to build, and then again to the Visual Studio project. The Visual Studio project generator doesn't query them from the target, this part is left to the user. If you're interested in the details, the documentation has some example code which shows a simple project setup.

Sample project

Let's see how SCons fares with our sample project. On the good news side, it's the first system we're looking at which supports both Linux and Windows with the same build script.

import os

# This is needed so we get the python from PATH
env = Environment(ENV = os.environ)

# include path is relative to root directory (indicated as #)
env.Append (CPPPATH='#')
p = os.path.abspath ('./tablegen.py')

This is just some basic setup so we can later use the same python executable as we used to invoke SCons. SCons doesn't inherit the environment of the surrounding shell by default, so we need to do this manually. This is for Windows, so we can actually find a Python interpreter if we have it in the PATH of the calling script.

pyexec = 'python' if os.name == 'nt' else 'python'
env.Command ('table.cpp', 'tablegen.py', '{} {} > $TARGET'.format (pyexec, p))

Now we're finally coming to the meat of this build:

env.StaticLibrary('statlib', [
    # This adds fPIC in a portable way
    SharedObject ('StaticLibrarySource.cpp'),
    SharedObject ('table.cpp')])

Here we're using the nodes directly, as we'd have to edit the compiler configuration manually otherwise to add -fPIC. Notice that SCons doesn't notice on it's own that this static library is consumed in a shared library, so we need to handle this manually. From here on, we can reference the target statlib in other project files inside the same build, which simplifies the linking -- no hard-coded paths. However, we can't export/import include paths directly, so our dynamic library project will still end up specifying the path for includes manually:

SharedLibrary ('dynlib', ['DynamicLibrarySource.cpp'],
  LIBS=['statlib'], LIBPATH=['../statlib'],
  CPPPATH=['.', '../statlib'], CPPDEFINES = ['BUILD_DYNAMIC_LIBRARY=1'])

That's it for SCons and this week! As always, make sure to look at the sample project for a full, working example.

In my opinion, the main advantages of SCons are twofold. First, it lets you freely mix various languages -- you can build a combined D and C++ and Java program without breaking a sweat. Second: By virtue of being a Python module, it easily integrates with the typical "glue" code in a build without having to change languages. I currently have a lot of Python code running as part of my build, and having the build system written in Python and using Python for the build files would simplify many things. It also removes the context switch between build system language and utility language. Personally, I'd probably consider it for very heterogenous builds, as SCons makes customization and scripting really simple compared to the other tools we've look at so far -- and also compared to quite a few tools that are yet to come. Enough teasing for today, see you again next week!