JIRA & Confluence with systemd on CentOS

I used to run JIRA & Confluence “bare metal” on an Ubuntu machine, but recently I’ve decided to put them into a VM. Backing up JIRA and Confluence is always a bit of a pain, especially if you want to move across servers. In my case the biggest pain point was authentication, I’m using the JIRA user database for Confluence, and when restoring from backup, your application link from Confluence to JIRA won’t get restored, so the first thing you do is you lock yourself out of Confluence. Fixing that requires some fun directly in the database, but that’s a story for another day (it’s also well documented on the Atlassian support page.)

Setting the stage

While deciding which OS to use for the VM, I ended up with CentOS, as that’s the only one on which the installer is officially supported. It turns out though that the JIRA (and Confluence) installers set up some scripts in /etc/init.d (for System V init), while CentOS is a systemd based distribution. That on its own wouldn’t bother me much, but what happened is that occasionally Confluence would start before the PostgreSQL database was online, then exit, and then JIRA would fail to see the application link (as Confluence was down). Long story short, there’s a startup order which should be followed, and instead of mixing the systems and playing around with runlevels until things work, I’ve decided to move everything to systemd and solve it once and for all.

Getting rid of the init scripts

The first thing which really confused me was that systemctl would show up a jira and confluence service, but no such service was present in any of the systemd configuration directories. It turns out, systemd will automatically generate services for init scripts. With that knowledge, it’s easy to see how we can fix this. A service with the same name will take precedence over a System V init script, so we could just set up some services and rely on that. I wanted to get rid of the init scripts wholesale though to clean everything up.

Unfortunately, it turns out chkconfig doesn’t know about jira and confluence. I.e. running chkconfig --del jira will tell you:

service jira does not support chkconfig

This can be solved by changing the scripts to make them chkconfig compatible, but we might as well do the steps chkconfig --del performs manually. First, I got rid of jira and confluence in /etc/init.d. That leaves the symlinks in /etc/rc3.d (and so on – one per runlevel.) First, I stopped the services using service stop jira and service stop confluence. Then I simply searched the 6 run level folders and removed all S95jira, K95jira, S95confluence and K95confluence entries there. After a reboot, nothing was started automatically any more – time for systemd.

Moving to systemd

systemd requires a service configuration file which contains the commands to execute, as well as dependencies – just what we need. According to the Red Hat documentation, services added by an administrator go into /etc/systemd/system. Let’s create one file per service then!

For JIRA, I created /etc/systemd/system/jira.service with the following contents:

[Unit] 
Description=Jira Issue & Project Tracking Software
Wants=nginx.service postgresql.service
After=network.target nginx.service postgresql.service

[Service] 
Type=forking
User=jira
PIDFile=/opt/atlassian/jira/work/catalina.pid
ExecStart=/opt/atlassian/jira/bin/start-jira.sh
ExecStop=/opt/atlassian/jira/bin/stop-jira.sh

[Install] 
WantedBy=multi-user.target

This assumes all the default paths. The only interesting lines are the Wants and After lines, which specify that JIRA has to come online after the postgresql.service, and it requires the postgresql.service to start with. For Confluence, the file looks virtually the same, except I make it dependent on JIRA as well – otherwise, users can’t log in anyway. Here’s the corresponding /etc/systemd/system/confluence.service:

[Unit] 
Description=Confluence Team Collaboration Software
Wants=postgresql.service nginx.service jira.service
After=network.target jira.service postgresql.service nginx.service

[Service] 
Type=forking
User=confluence
PIDFile=/opt/atlassian/confluence/work/catalina.pid
ExecStart=/opt/atlassian/confluence/bin/start-confluence.sh
ExecStop=/opt/atlassian/confluence/bin/stop-confluence.sh

[Install] 
WantedBy=multi-user.target

I later found out that there are virtually identical service definitions here, but they’re missing the Wants/After dependencies. All that is left is to actually enable and run the services. This is rather straightforward:

$> systemctl daemon-reload
$> systemctl enable jira
$> systemctl enable confluence
$> systemctl start confluence

As Confluence depends on JIRA, it will start that automatically as well. Now we have both running through systemd, with dependencies properly specified. Just run systemctl list-dependencies jira to see it depend on PostgreSQL (and nginx.) With that, there are no more failures due to funky start ordering, and as a bonus, everything uses systemd instead of having to deal with some compatibility modes.