This post will cover why it’s necessary to restart Sidekiq for each deployment as well as how to achieve it.
Why it’s necessary to restart Sidekiq for each deployment
Example: database column name change
Let’s say you have a model in your Rails application called
Customer. The corresponding
customers table has a column that someone has called
Because you’re not a fan of making your code harder to work with through overabbreviation, you decide to alter the
customers table and rename
first_name. (You also change any instances of
first_name in the Rails code.) You deploy this change and the database migration runs in production.
Problem: out-of-date Sidekiq code
Unfortunately, unless you restart the Sidekiq process, there will be a problem. At the time the Sidekiq process was started (and by “Sidekiq process” I of course mean the thing that runs when you run
bundle exec sidekiq), the Rails code referred to a column called
If you changed the column name without restarting Sidekiq, the code the Sidekiq process is running will be referring to the non-existent
fname column and your application will break.
The fix is simple: restart Sidekiq each time you deploy. The way to achieve this, however, is not so simple.
How to get Sidekiq to restart
systemd and systemctl
Before we get into the specifics of how to get Sidekiq to restart, a little background is necessary.
You’ve probably run Linux commands before that look something like
sudo service nginx restart. I’ve been running such commands for many years without understanding what the “service” part is all about.
The “service” part is related to a pair of Linux concepts, systemd and systemctl. In the words of the DigitalOcean article I just linked,
systemd is “an init system and system manager” and
systemctl is “the central management tool for controlling the init system”. To put it in something closer to layperson’s terms,
systemctl are a way to manage processes in a convenient way.
Using systemd to manage Sidekiq
If we register Sidekiq as a service with
systemd, we gain the ability to run commands like
service sidekiq start,
service sidekiq restart and
service sidekiq stop.
Once we have that ability, restarting Sidekiq on each deployment is as simple as adding
service sidekiq restart as a step in our deployment script.
Adding a sidekiq.service file
The first step in telling
systemd about Sidekiq is to put a
sidekiq.service file in a certain directory. Which directory depends on which Linux distribution you’re using. I use Ubuntu so my file goes at
For the contents of the file you can use this sidekiq.service file from the Sidekiq repo as a starting point.
Important modifications to make to sidekiq.service
/opt/myapp/current to the directory where your app is served from.
Group=deploy to whatever user and group you use for deployment.
If you’re using a version of Sidekiq that’s earlier than 6.0.6, you’ll need to change
Type=simple and remove the
Bringing in environment variables
If your application depends on environment variables, which it of course almost certainly does, the Sidekiq process will need to be aware of those environment variable values in order to run.
There are multiple ways to pull environment variables into the Sidekiq service. In my case I did so by creating a file called
/etc/profile.d/sidekiq_env (the name and location are arbitrary) and adding a line
EnvironmentFile=/etc/profile.d/sidekiq_env to my
sidekiq.service. Here’s a sample
sidekiq_env file so you know how it’s formatted:
RAILS_ENV=production RAILS_FORCE_SSL=true RAILS_SKIP_MIGRATIONS=false # etc
Trying out your Sidekiq service
Once you’ve done all the above you can run
systemctl enable sidekiq and then
service sidekiq start. If you get something other than no output, something is wrong. If you get no output, the service theoretically started successfully, although you won’t know for absolute certain until you verify by running a job while tailing the logs.
You can tail the logs by running
journalctl -u sidekiq -f. I verified my Sidekiq
systemd setup by watching the output of those logs while invoking a Sidekiq job.
If everything is working at this point, congratulations. In the much more likely case that something is wrong, there are a couple ways you can troubleshoot.
First of all, you should know that you need to run
systemctl daemon-reload after each change to
sidekiq.service in order for the change to take effect.
One way is to use the output from the same
journalctl -u sidekiq -f command above. Another is to run
systemctl status sidekiq and see what it says.
If Sidekiq doesn’t run properly via
systemd, try manually running whatever command is in
ExecStart, which, if you used the default, is
/usr/local/bin/bundle exec sidekiq -e production. That command should of course work, so if it doesn’t, then there’s a clue.
But it is possible for the command in
ExecStart to work and for the
systemd setup to still be broken. If, for example, you have your environment variables loaded properly in the shell but you don’t have environment variables loaded properly in your
sidekiq.service file, the
service start sidekiq command won’t work. Examine any Rails errors closely to determine whether the problem might be due to a missing environment variable value.
Bonus: restarting Sidekiq via Ansible
If you happen to use Ansible to manage your infrastructure like I do, here’s how you can add a task to your Ansible playbooks for restarting Sidekiq.
The restart task itself looks like this:
- name: Restart sidekiq service: name: sidekiq state: restarted daemon_reload: yes
This task alone wasn’t enough for me though. I wanted to add a task to copy the
sidekiq.service file. I also needed to add a task to enable the Sidekiq service.
tasks: - name: Copy sidekiq.service template: src: sidekiq.service dest: /lib/systemd/system/sidekiq.service force: yes owner: root group: root mode: 0644 - name: Enable sidekiq service: name: sidekiq enabled: yes - name: Restart sidekiq service: name: sidekiq state: restarted daemon_reload: yes
Good luck, and please leave a comment if you have troubles or need clarification.