+++ title = "building a multi instance postgres systemd service" date = "2012-09-28T11:00:00+00:00" author = "Gibheer" draft = false +++

Just out of curiosity I tried to build a service for PostgreSQL and the systemd init system. Before that, I only read the service files of postgres and dhcp delivered with Archlinux. What I wanted to build is a service file able to start multiple instances of postgres with separate configuration files.

This was much easier than I thought it would be.

Systemd supports that pretty well and the only thing to do, is add an '@' to the service file name. Everything after '@' is then put into a variable %I, which can be used in the service file. So my service file was named 'pg@.service' and I put it into /etc/systemd/system. Another possible location is /usr/lib/systemd/system/.

The service file looks like an .ini file. It has the three sections Unit, Service and Install. The section Install in which target the service is installed. Targets are like run levels in other init systems. The postgres service gets installed into the multi-user target, which is started after the network:

[Install]                                                                       
WantedBy=multi-user.target

The next part is Unit. This section describes the service with a short description and a description of the dependencies. Postgres just needs the network up, so this section looks like this:

[Unit]                                                                          
Description=run PostgreSQL instance %I                                          
After=network.target

There you can also see the %I, which is replaced with the part after '@' from the name in systemd.

The next section is a bit larger and describes everything needed to manage the service itself, like start, stop and reload.

[Service]                                                                       
User=postgres                                                                   
Group=postgres                                                                  
TimeoutSec=120                                                                  
Type=forking                                                                       
                                                                               
EnvironmentFile=/etc/conf.d/pg.%I                                                  
                                                                               
SyslogIdentifier=postgres-%i                                                       
                                                                               
ExecStartPre=/usr/bin/postgresql-check-db-dir ${pgdata}                            
ExecStart= /usr/bin/pg_ctl -s -D ${pgdata} start -w -t 120                         
ExecReload=/usr/bin/pg_ctl -s -D ${pgdata} reload                                  
ExecStop=  /usr/bin/pg_ctl -s -D ${pgdata} stop -m fast                                       
                                                                            
OOMScoreAdjust=-200

Okay, this is a bit longer than the other parts. The first Couple of options handle the user to start with and he startup timeout. The timeout can't be replaced with a variable because all options from the config will be loaded as environment variables on execution. The Type option is very important, because it can't be set to anything else as forking for postgres, because it will fork to the background. So if you start it as a simple service systemd would loose the handler to postgres and stop it immediately.

The next options are EnvironmentFile and SyslogIdentifier. The first is for a small config file in /etc/conf.d/pg.instance where you replace instance with the instance name. As you can see with the %I in place, it will fill up the full name with the instance identifier. So you can use different config files for different instances. The same happens to the SyslogIdentifier. I thought it would be awesome if the log can be showed per instance and this is what you need to make it happen.

The option OOMScoreAdjust is just an option for the OOMKiller, that it should leave postgres alone as much as possible.

The option ExecStartPre calls a script which is delivered with postgres on Archlinux and does a check for the data dir. If it does not exist, it will log a line on how to create it. Pretty neat. ExecStart, ExecStop and ExecReload describe the actions to be done, when the service should be started, stopped or reloaded. As you can see, the script uses ${pgdata} to determine where to look and that variable comes from the EnvironmentFile, which looks for my first instance like this

pgdata=/tmp/ins1

The file is saved as /etc/conf.d/pg.ins1 and is really nothing more than this. The rest can handle postgres itself.

Now how do we get the service file into systemd? You do a

systemctl --system daemon-reload

and then

systemctl start pg@ins1.service

This creates your first service and tries to start it. You will get an error message like the following

Job for pg@ins1.service failed. See 'systemctl status pg@ins1.service' and 'journalctl' for details.

If you run the status, you will see that it failed, how it failed and the log message from the check script. After that, you can create the instance and start it anew there it is.

# systemctl status pg@ins1.service
pg@ins1.service - PostgreSQL database server
          Loaded: loaded (/etc/systemd/system/pg@ins1.service; disabled)
          Active: active (running) since Tue, 25 Sep 2012 09:27:54 +0200; 3 days ago
         Process: 372 ExecStop=/usr/bin/pg_ctl -s -D ${PGROOT}/data stop -m fast (code=exited, status=0/SUCCESS)
         Process: 624 ExecStart=/usr/bin/pg_ctl -s -D ${PGROOT}/data start -w -t 120 (code=exited, status=0/SUCCESS)
         Process: 619 ExecStartPre=/usr/bin/postgresql-check-db-dir ${PGROOT}/data (code=exited, status=0/SUCCESS)
        Main PID: 627 (postgres)
          CGroup: name=systemd:/system/postgresql.service
                  ├ 627 /usr/bin/postgres -D /var/lib/postgres/data
                  ├ 629 postgres: checkpointer process   
                  ├ 630 postgres: writer process   
                  ├ 631 postgres: wal writer process   
                  ├ 632 postgres: autovacuum launcher process   
                  └ 633 postgres: stats collector process

Now if you want to see some logging, you can ask journcalctl and give it the log string.

journcalctl SYSLOG_IDENTIFIER=postgres-instance1

That's all there is to multi instance services on syslog. To figure everything out actually took not even much time, as the documentation is pretty good. Just a hint, don't look in the web for documentation but in the man pages. The best starting point to look for documentation is man systemd and then take a look at the SEE ALSO section.

Have fun!