JIRA & Confluence with systemd on CentOS

I used to run JIRA & Confluence "bare metal" on an Ubuntu machine, but recently I've decided to put them into a VM. Backing up JIRA and Confluence is always a bit of a pain, especially if you want to move across servers. In my case the biggest pain point was authentication, I'm using the JIRA user database for Confluence, and when restoring from backup, your application link from Confluence to JIRA won't get restored, so the first thing you do is you lock yourself out of Confluence. Fixing that requires some fun directly in the database, but that's a story for another day (it's also well documented on the Atlassian support page.)

Setting the stage

While deciding which OS to use for the VM, I ended up with CentOS, as that's the only one on which the installer is officially supported. It turns out though that the JIRA (and Confluence) installers set up some scripts in /etc/init.d (for System V init), while CentOS is a systemd based distribution. That on its own wouldn't bother me much, but what happened is that occasionally Confluence would start before the PostgreSQL database was online, then exit, and then JIRA would fail to see the application link (as Confluence was down). Long story short, there's a startup order which should be followed, and instead of mixing the systems and playing around with runlevels until things work, I've decided to move everything to systemd and solve it once and for all.

Getting rid of the init scripts

The first thing which really confused me was that systemctl would show up a jira and confluence service, but no such service was present in any of the systemd configuration directories. It turns out, systemd will automatically generate services for init scripts. With that knowledge, it's easy to see how we can fix this. A service with the same name will take precedence over a System V init script, so we could just set up some services and rely on that. I wanted to get rid of the init scripts wholesale though to clean everything up.

Unfortunately, it turns out chkconfig doesn't know about jira and confluence. I.e. running chkconfig --del jira will tell you:

service jira does not support chkconfig

This can be solved by changing the scripts to make them chkconfig compatible, but we might as well do the steps chkconfig --del performs manually. First, I got rid of jira and confluence in /etc/init.d. That leaves the symlinks in /etc/rc3.d (and so on -- one per runlevel.) First, I stopped the services using service stop jira and service stop confluence. Then I simply searched the 6 run level folders and removed all S95jira, K95jira, S95confluence and K95confluence entries there. After a reboot, nothing was started automatically any more -- time for systemd.

Moving to systemd

systemd requires a service configuration file which contains the commands to execute, as well as dependencies -- just what we need. According to the Red Hat documentation, services added by an administrator go into /etc/systemd/system. Let's create one file per service then!

For JIRA, I created /etc/systemd/system/jira.service with the following contents:

[Unit]
Description=Jira Issue & Project Tracking Software
Wants=nginx.service postgresql.service
After=network.target nginx.service postgresql.service

[Service]
Type=forking
User=jira
PIDFile=/opt/atlassian/jira/work/catalina.pid
ExecStart=/opt/atlassian/jira/bin/start-jira.sh
ExecStop=/opt/atlassian/jira/bin/stop-jira.sh

[Install]
WantedBy=multi-user.target

This assumes all the default paths. The only interesting lines are the Wants and After lines, which specify that JIRA has to come online after the postgresql.service, and it requires the postgresql.service to start with. For Confluence, the file looks virtually the same, except I make it dependent on JIRA as well -- otherwise, users can't log in anyway. Here's the corresponding /etc/systemd/system/confluence.service:

[Unit]
Description=Confluence Team Collaboration Software
Wants=postgresql.service nginx.service jira.service
After=network.target jira.service postgresql.service nginx.service

[Service]
Type=forking
User=confluence
PIDFile=/opt/atlassian/confluence/work/catalina.pid
ExecStart=/opt/atlassian/confluence/bin/start-confluence.sh
ExecStop=/opt/atlassian/confluence/bin/stop-confluence.sh

[Install]
WantedBy=multi-user.target

I later found out that there are virtually identical service definitions here, but they're missing the Wants/After dependencies. All that is left is to actually enable and run the services. This is rather straightforward:

$> systemctl daemon-reload
$> systemctl enable jira
$> systemctl enable confluence
$> systemctl start confluence

As Confluence depends on JIRA, it will start that automatically as well. Now we have both running through systemd, with dependencies properly specified. Just run systemctl list-dependencies jira to see it depend on PostgreSQL (and nginx.) With that, there are no more failures due to funky start ordering, and as a bonus, everything uses systemd instead of having to deal with some compatibility modes.

Docker, KVM and iptables

A quick blog post as I've been recently setting up a server with KVM and docker. If you're using a bridge interface to have your VMs talk to your network, you might notice that after a docker installation, your VMs have suddenly no connection. So what is going on?

It turns out, docker adds a bunch of iptables rules by default which prevent communication. These will interfere with an already existing bridge, and suddenly your VMs will report no network. There are two ways to solve it:

  • Add a new rule for your bridge.
  • Stop docker from adding iptables rules.

I'm assuming Ubuntu 17.04 for the commands below; they should be similar on any Debian based system.

Solution 1: Add a new rule

In my case, my bridge is called br0. docker changes the default for FORWARD from accept to drop, and adds a few exceptions for itself. Adding a new forward rule for br0 will allow your bridge (and the devices) behind it to get back into your network and not get dropped:

iptables -A FORWARD -i br0 -o br0 -j ACCEPT

Unfortunately, this won't be persistent -- you'll need the iptables-persistent package on Linux to make it persistent, plus some extra setup. It's good for a quick test though! (Source)

Solution 2: Stop docker

In my case, the server is not on the public internet, and I've got no need for the extra security. It turns out that the docker service adds the rules on startup, unless --iptables=false is used. This can be either added to the default docker configuration, or, slightly cleaner in my opinion, to the daemon.json configuration file (see the documentation for all options). Create a file /etc/docker/daemon.json with the following contents:

{
    "iptables" : false
}

That'll stop docker from adding new rules, and everything will work as it did before.

Build systems: Conclusion

We've made it -- the last post in this series! It's been quite a ride, over the last weeks, we've investigated nine different build systems. Let's summarize what we've seen. I've broken down the features into a couple of grand categories:

Overview

  • Portable: Is the build definition portable across different compilers? This requires some kind of abstraction level from the underlying compiler, for instance, include directories being passed through variables, etc.
  • Modular: Does the build system scale to larger builds? Can you write parts of the build independently of each other and then combine them into a bigger solution?
  • Ubiquitous: Can you expect this build system to be available?
  • Type: Does the build tool actually build on its own (in that case, you'll see "Build" specified), or does it rely on other tools ("Generator")?
Tool Portable Modular Ubiquitous Type
Make No No Yes Build
MSBuild No No Yes Build
SCons Yes No No Build
Waf Yes Yes No Build
FASTbuild No No No Build
Bazel Yes Yes No Build
Buck Yes Yes No Build
Premake Yes No No Generator
CMake Yes Yes Yes Generator

Survey results

Last week I've also started survey to get a better idea what people are using out there "in the wild". From the responses, the most popular build systems by far are MSBuild (including Visual Studio), followed by CMake and make. Given over 95% indicated they use Windows, this should be no surprise, as MSBuild is the default system for the extremely popular Visual Studio IDE. Still, CMake is showing up as a very strong competitor, second to MSBuild in both availability and usage.

I've asked which language is used for the "build system glue" as well, and that answer is interesting as well. 50% use whatever their build system uses, Python, or shell scripts. Combining all shell script languages makes shell the most popular solution for the extra tasks the build systems don't cover. The interesting bit here is although Python was really popular in this category, Python based build systems don't seem to be interesting for developers.

Wrapping up

We've looked at various build systems over the last couple of weeks, and if there's one thing we've learned, then this: There's no "one size fits all" solution, and for C++, there might as well never be until C++ gets a standard ABI which will make library reuse a reality. CMake has tackled this problem head-on with the "find module" concept which is in my opinion one of the main reasons for its popularity. It's going to be interesting to see if other projects will just embrace CMake's approach or just migrate to CMake. Microsoft has heavily invested in CMake providing a C++ package manager dubbed vcpkg which is completely based on CMake. At the same time, large projects like the Unreal Engine 4 use multiple fully customized build tools like the Unreal Build System. I'm personally very curious to see how the ecosystem will evolve going forward, and what features and concepts future build systems will bring to the table.

To this end, I hope that you got a good idea of what we have today in terms of build tools and their concepts, so you'll be ready for the future. That's it for this series, thanks a lot for reading, and I'd also like to thank my tireless reviewers for their help, in particular, Baldur and Jasper. Thanks guys!

Build systems: CMake

Welcome to the last build system we're going to look at in this cycle! As hinted last week, there's one problem left: How can we find binaries and other dependencies already present in the host environment? This is a tricky question, as we have to solve two problems at the same time: First, we need to be able to find things in a cross-platform manner, which means we need things like "invoke a binary" to query version numbers for instance. Second, we need a rather precise search to ensure that our build system has a good view of the dependency we're about to integrate. If it's just a binary, this is usually not a problem, but a C++ library might require special compile definitions, a link library, an include path and more to be set on the target project.

Fortunately, I've got just the build system for you to demonstrate this -- CMake. CMake is in many ways the union of various build systems we've seen so far. It's a build system generator just like Premake. It comes with its own scripting language, just like Bazel. It's called "as-if" it's a library just like Scons. And of top of all of this, it provides two new concepts: Find modules and imported targets.

Find modules

One of the two big features of CMake are find modules, which allow you to find an existing library or binary with a simple command. Instead of trying to hard-code paths to the Python interpreter, we can just call find_package(PythonInterp 3.5 REQUIRED) and CMake will handle everything. After the command has executed, CMake will populate a few variables so we can consume Python directly. In the case of the module above, PYTHON_EXECUTABLE will point to the Python binary, and we can immediately invoke it in our generator step.

The find modules don't end with binaries though. You can also search for libraries, and even search for library components. This is super helpful when dealing with large libraries like Boost or Qt, which are very large and require searching up many paths. The search method is just the same, but there are two ways in which the results can be returned. The "old" way is to populate a couple of variables, which are then manually added to your project. The "new" way is to provide an imported target, which handles all of the required settings in a transparent way.

Imported targets

An imported target in CMake is nothing else than a "normal" target, but assembled manually, instead of being generated as part of the current CMake build. Let me elaborate a bit on this. In CMake, if you add a new target like using for example add_library, it produces a named object you can reference elsewhere -- just like we saw it in Waf. CMake allows to specify settings on a target which will affect the consumers. For example, you can specify an include directory which is automatically added to everyone linking against this target.

An imported target is now a target which is imported from elsewhere, but internally behaves "as if" it was a normal CMake target. That is, it gets all the goodness of injecting include directories, compile settings and more we mentioned above. There are two ways to get an imported target. The simple one is to let CMake handle this: If your project is set up correctly for installation, CMake can generate a project description which can be imported directly. This is somewhat similar to how Bazel handles large builds, except that for CMake, you bundle up the project output with the build definition.

The other way to create an imported target is to assemble it by hand, which is what most of the modern find modules do. What happens is that an imported target is created out of thin air, and the various components are then specified manually. If you're curious about the details, the add_library documentation and the sample FindModule got you covered!

Sample project

Let's get started with our sample project then, and the way we generate our lookup table:

find_package(PythonInterp 3.5 REQUIRED)

add_custom_command(
    OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/lookup_table.cpp
    COMMAND ${PYTHON_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/tablegen.py > ${CMAKE_CURRENT_BINARY_DIR}/lookup_table.cpp
    DEPENDS tablegen.py)

In the first line, we let CMake find the Python interpreter. The next command registers a custom command which will be executed to produce the lookup_table.cpp file. Note that the file needs to be consumed somewhere for this to make sense, otherwise it's just a leaf in the build graph with no dependency on it. For a custom command, we can specify the dependencies manually, as CMake doesn't try to run them in a sandbox and log I/O access or anything like that.

Next, we'll use that generated file in our static library definition:

add_library(statlib STATIC
    StaticLibrarySource.cpp ${CMAKE_CURRENT_BINARY_DIR}/lookup_table.cpp)

We still need to set up the include directories. For this project, we only have one include directory -- the current directory -- but targets consuming the static library need to use that include directory as well. In CMake, that's called a PUBLIC include directory:

target_include_directories(statlib
    PUBLIC ${CMAKE_CURRENT_SOURCE_DIR})

We're nearly done, there's only one bit left, which is specifying the -fPIC flag. Unfortunately, CMake doesn't handle this automatically, nor does it warn when we try to consume a static library without it enabled. On the upside, CMake provides a compiler agnostic way of specifying it:

set_property (TARGET statlib PROPERTY POSITION_INDEPENDENT_CODE TRUE)

With that, our project is complete. In the dynamic library, we have a case where we need to set a compile definition but only for the library. This is a PRIVATE setting -- just as we had PUBLIC above:

target_compile_definitions(dynlib
    PRIVATE BUILD_DYNAMIC_LIBRARY=1)

This way, we can have very fine-grained visibility of the settings, though we can't restrict project visibility as we could in Bazel. As usual, you can find the whole project in the sample repository.

Before we wrap up, some small remarks regarding the various features. The module finding is something which is aimed at Linux environments, which provide all libraries in standard locations. On Windows, the find experience for libraries is such that you typically have to specify the path to the library. It's not a big problem in practice, as you can specify the various paths on the first invocation of CMake.

Another thing to keep in mind that unlike Premake, which generates portable build files using relative paths to the source, CMake bakes in the full directories. Your build description is yet another build artifact and should be treated as such, so it's not a tool you can use to generate Visual Studio project files. The developer must have CMake installed, and there's no easy way to package up CMake into a single executable for instance like there is with Premake.

With this, we're nearly at the end of the series! For the last blog post, I'd like to gather some information about what build systems you use, and to this end, I've set up a survey. It should only take a minute or so to fill it out. Thanks for your help and see you again next week for the grand finale!

Build systems: Premake

Welcome back to another post in the build system series! So far, we've looked at many different build systems, but all of them had one thing in common: They did build the project all by themselves, no other tool was required. I've hinted a bit at generating build files in the Make entry, and today, we're going to look at a build tool which is actually a "meta" build tool -- all it does is generate build files for other build tools. I've picked Premake to showcase this new concept, as it's literally the name of the game here ☺

Build generators?

It might seem a bit weird at first to build a build tool which then requires another build tool to be invoked, but if we look a bit closer, there are good reasons for this separation. As I mentioned in the intro, the goal of a build system is to transform some data using some commands. We saw how Make and MSBuild focus on implementing that goal and not much else. By separating the build execution from the build script generation, we can have two focused tools, instead of mixing high- and low-level concepts together. The other build tools we looked at all have some task and dependency engine somewhere, but it might not be directly accessible, making it hard to understand what exactly is being executed in the end.

There are various advantage of going "meta", one being portability between systems. On Linux, Make is super popular, on Windows, MSBuild is the default, and if you want to target both, it makes sense to generate the files from a single source. Especially once it comes to supporting multiple versions of the same tool -- Visual Studio 2013, 2015, 2017, for example -- being able to just generate the files reduces the maintenance burden a lot.

Another major advantage of splitting the generation from the execution is performance. We already saw build systems splitting the initial configuration into a separate step from the actual build when we looked at Waf. In general, the build description will change only rarely, and by making the build description minimal, we can improve the daily build experience. In fact, this very idea is what lead to the development of Ninja, a super low-level tool similar to Make, which is designed to execute machine generated build scripts.

Sample project

Let's see how Premake differs from the other tools we've used so far. On the build file side, Premake uses a scripting language to allow for easy high-level customization. Here, it's Lua, and the build files are actually Lua scripts which are interpreted at generation time. After generation, Premake writes a build file for the tool you've selected, which then needs to build once more. One feature of Premake is that it writes portable build files, so you can use it as a generator for build files which are then shipped, removing the need to invoke Premake on the developer's machine. Next week, we'll see a different approach to build files where they are treated as an intermediate output only.

The actual build description looks very similar to what we've seen before. As usual, we'll start with the static library here:

project "statlib"
    kind "StaticLib"
    language "C++"

We tell Premake we're building a static library in C++, nothing special so far. Next up, we define the source files:

files {"StaticLibrarySource.cpp", "%{cfg.buildtarget.directory}/table.cpp"}

He we have our generated file, specified just "as if" it already exists. We don't specify the actual generation script for it, instead, we use a "pre-build" command which will be executed before the build runs to generate this file. This way, we side-step the issue of specifying build-time dependencies inside Premake. Premake will write a build description which assumes the file to exist, and ask the underlying build system to produce it in a pre-build step, but Premake is not going to provide the actual dependency to the build system. This means that it will get rebuilt no matter what. This can be also seen from how we specify this:

p = path.getabsolute (os.getcwd ())
prebuildcommands { "\"C:\\Program Files\\Python36\\python.exe\" " .. p .. "/tablegen.py > %{cfg.buildtarget.directory}/table.cpp" }

We invoke Lua to get the current working directory, so we can run the Python executable with the correct path. We don't tell Premake about what is required to build the table, and what outputs it produces, we just pass on the raw command line and ask Premake to execute it.

That's it for the static library -- unsurprisingly, Premake requires just as much information as any other tool we looked at to generate correct build files. That's it for Premake, the reminder of the build description will look very familiar to you if you've been following this series so far. As always, head to the sample repository for the build definition.

With this, what is left in terms of build systems? There's still one problem we had in all the systems so far, which is finding binaries and dependencies, that none of the build systems have tackled so far. Next week, we're going to investigate this issue in a lot of detail -- stay tuned!