Managing puma with the systemd user instance and monit
Many guides to deploying Rails with Capistrano will use systemd to have it auto-started when the system boots. This is often done using the system instance of systemd which by default can only be controlled by root.
The typical workaround for this is either to grant our Capistrano deployment user passwordless sudo access or to grant them passwordless sudo access to just the commands required to restart the rails (and potentially sidekiq) systemd services.
This can be avoided by using the systemd user instance, which allows persistent services to be managed as a non-root user. This is compatible with the default systemd configuration in Ubuntu 20.04.
There are multiple locations systemd user instance units can be located, there's more here, in this case we'll be using:
In here we'll put a systemd unit file similar to the following:
[Unit] Description=Puma HTTP Server for RAILS APP NAME (ENVIRONMENT) After=network.target [Service] Type=simple WorkingDirectory=/home/deploy/apps/APP_NAME/current ExecStart=/usr/local/rbenv/bin/rbenv exec bundle exec puma -C /home/deploy/apps/APP_NAME/shared/puma.rb ExecReload=/bin/kill -USR1 $MAINPID ExecStop=/bin/kill -TSTP $MAINPID StandardOutput=append:/home/deploy/apps/APP_NAME/shared/log/puma_access.log StandardError=append:/home/deploy/apps/APP_NAME/shared/log/puma_error.log Environment=RBENV_VERSION=3.0.0 Environment=RBENV_ROOT=/usr/local/rbenv Restart=always RestartSec=1 SyslogIdentifier=APP_NAME [Install] WantedBy=default.target
capistrano-puma Gem can auto-generate this file and the
capistrano-cookbook gem provides and overridden version of the template which fixes some rbenv compatibility issues and allows for zero downtime deploys (as well as also generating all other capistrano configuration automatically).
You can see the most recent Capistrano Cookbooks unit file template - which may be useful as a reference - here which is a tweaked version of the version in
Note a few things about this unit file:
- There is no
Userdirective, user services will run as the user in question, including a
Userdirective may lead to non-descriptive
service start request repeated too quickly, refusing to starttype errors
- Our Environment variables are not included in the
ExecStartcommand, they're in separate
Environmentlines, this is explained here
WantedByis set to
default.targetwhich is the correct value for user services if we want them to be started at boot
In order for our service to be started at boot, we then need to enable this service with:
systemctl --user enable UNIT_FILE_NAME
Note that this is different to starting the unit. We can start a unit immediately with
systemctl --user start UNIT_FILE_NAME but this does not set the unit to be started on boot, so we must enable it as well. This is taken care of automatically if you're using the
deploy:setup_config task from
Our next challenge is that by default, user instance systemd services are only started when the user starts a session and will continue to run only while the user in question has an active session.
To resolve this we must enable lingering, lingering ensures that a manager for the user in question in spawned on boot so that the user can manage long run services.
We can enable lingering with:
loginctl enable-linger USERNAME
Where USERNAME is the capistrano deployment user. This is taken care of automatically if you're using the
deploy:setup_config task from
Finally we may want to monitor our systemd service with Monit. While there is crossover between systemd and monit, both will monitor that a process is running and start it if not.
Monit however adds some additional capabilities on top of systemd, it can allow for significantly more complex checks such as making sure that our service is responding on a given port and even check the contents of certain healthcheck responses and issuing restarts if these aren't matched.
Monit however runs as root and we need it to control a systemd user service.
We may initially think we can use something like:
start program = "/usr/bin/systemctl --user start SYSTEMD_SERVICE_UNIT_FILE" as uid "deploy" and gid "deploy"
As our start program where
deploy is our Capistrano user. We might expect this to be equivalent to running
systemctl --user start as the deploy user in a shell. While the command will be run as that user, due to some missing environment variables, we're likely to get an error along the lines of:
Failed to get D-bus connection: no such file or directory
This is due to
XDG_RUNTIME_DIR not being set correctly when users are switched in this way. The same issue can happen if we try and use
su in Capistrano to change users before executing a
systemctl --user command.
We can resolve is by modifying our start command to set this environment variable manually. So a simple monit definition might look like this:
check process APP_NAME with pidfile "/home/deploy/apps/APP_NAME/shared/tmp/pids/puma.pid" start program = "/bin/bash -c 'XDG_RUNTIME_DIR=/run/user/$(id -u) /usr/bin/systemctl start --user SYSTEMD_SERVICE_UNIT_FILE'" as uid "deploy" and gid "deploy" stop program = "/bin/bash -c 'XDG_RUNTIME_DIR=/run/user/$(id -u) /usr/bin/systemctl stop --user SYSTEMD_SERVICE_UNIT_FILE'" as uid "deploy" and gid "deploy"
You can see the most recent version of
capistrano cookbooks monit definition here which may be useful as a reference.
With all of this completed, we should now have puma being managed by a systemd service, which will auto start at boot, and is monitored with monit.