This post will cover why it’s necessary to restart Sidekiq for each deployment as well as how to achieve it.

Why it’s necessary to restart Sidekiq for each deployment

Example: database column name change

Let’s say you have a model in your Rails application called Customer. The corresponding customers table has a column that someone has called fname.

Because you’re not a fan of making your code harder to work with through overabbreviation, you decide to alter the customers table and rename fname to first_name. (You also change any instances of fname to first_name in the Rails code.) You deploy this change and the database migration runs in production.

Problem: out-of-date Sidekiq code

Unfortunately, unless you restart the Sidekiq process, there will be a problem. At the time the Sidekiq process was started (and by “Sidekiq process” I of course mean the thing that runs when you run bundle exec sidekiq), the Rails code referred to a column called fname.

If you changed the column name without restarting Sidekiq, the code the Sidekiq process is running will be referring to the non-existent fname column and your application will break.

The fix is simple: restart Sidekiq each time you deploy. The way to achieve this, however, is not so simple.

How to get Sidekiq to restart

systemd and systemctl

Before we get into the specifics of how to get Sidekiq to restart, a little background is necessary.

You’ve probably run Linux commands before that look something like sudo service nginx restart. I’ve been running such commands for many years without understanding what the “service” part is all about.

The “service” part is related to a pair of Linux concepts, systemd and systemctl. In the words of the DigitalOcean article I just linked, systemd is “an init system and system manager” and systemctl is “the central management tool for controlling the init system”. To put it in something closer to layperson’s terms, systemd and systemctl are a way to manage processes in a convenient way.

Using systemd to manage Sidekiq

If we register Sidekiq as a service with systemd, we gain the ability to run commands like service sidekiq start, service sidekiq restart and service sidekiq stop.

Once we have that ability, restarting Sidekiq on each deployment is as simple as adding service sidekiq restart as a step in our deployment script.

Adding a sidekiq.service file

The first step in telling systemd about Sidekiq is to put a sidekiq.service file in a certain directory. Which directory depends on which Linux distribution you’re using. I use Ubuntu so my file goes at /lib/systemd/system/sidekiq.service.

For the contents of the file you can use this sidekiq.service file from the Sidekiq repo as a starting point.

Important modifications to make to sidekiq.service

Change WorkingDirectory from /opt/myapp/current to the directory where your app is served from.

Change User=deploy and Group=deploy to whatever user and group you use for deployment.

If you’re using a version of Sidekiq that’s earlier than 6.0.6, you’ll need to change Type=notify to Type=simple and remove the WatchdogSec=5 line.

Bringing in environment variables

If your application depends on environment variables, which it of course almost certainly does, the Sidekiq process will need to be aware of those environment variable values in order to run.

There are multiple ways to pull environment variables into the Sidekiq service. In my case I did so by creating a file called /etc/profile.d/sidekiq_env (the name and location are arbitrary) and adding a line EnvironmentFile=/etc/profile.d/sidekiq_env to my sidekiq.service. Here’s a sample sidekiq_env file so you know how it’s formatted:

RAILS_ENV=production
RAILS_FORCE_SSL=true
RAILS_SKIP_MIGRATIONS=false
# etc

Trying out your Sidekiq service

Once you’ve done all the above you can run systemctl enable sidekiq and then service sidekiq start. If you get something other than no output, something is wrong. If you get no output, the service theoretically started successfully, although you won’t know for absolute certain until you verify by running a job while tailing the logs.

You can tail the logs by running journalctl -u sidekiq -f. I verified my Sidekiq systemd setup by watching the output of those logs while invoking a Sidekiq job.

Troubleshooting

If everything is working at this point, congratulations. In the much more likely case that something is wrong, there are a couple ways you can troubleshoot.

First of all, you should know that you need to run systemctl daemon-reload after each change to sidekiq.service in order for the change to take effect.

One way is to use the output from the same journalctl -u sidekiq -f command above. Another is to run systemctl status sidekiq and see what it says.

If Sidekiq doesn’t run properly via systemd, try manually running whatever command is in ExecStart, which, if you used the default, is /usr/local/bin/bundle exec sidekiq -e production. That command should of course work, so if it doesn’t, then there’s a clue.

But it is possible for the command in ExecStart to work and for the systemd setup to still be broken. If, for example, you have your environment variables loaded properly in the shell but you don’t have environment variables loaded properly in your sidekiq.service file, the service start sidekiq command won’t work. Examine any Rails errors closely to determine whether the problem might be due to a missing environment variable value.

Bonus: restarting Sidekiq via Ansible

If you happen to use Ansible to manage your infrastructure like I do, here’s how you can add a task to your Ansible playbooks for restarting Sidekiq.

The restart task itself looks like this:

- name: Restart sidekiq
  service:
    name: sidekiq
    state: restarted
    daemon_reload: yes

This task alone wasn’t enough for me though. I wanted to add a task to copy the sidekiq.service file. I also needed to add a task to enable the Sidekiq service.

tasks:
  - name: Copy sidekiq.service
    template:
      src: sidekiq.service
      dest: /lib/systemd/system/sidekiq.service
      force: yes
      owner: root
      group: root
      mode: 0644

  - name: Enable sidekiq
    service:
      name: sidekiq
      enabled: yes

  - name: Restart sidekiq
    service:
      name: sidekiq
      state: restarted
      daemon_reload: yes

Good luck

Good luck, and please leave a comment if you have troubles or need clarification.