Part of a series on configuring and running the eigenmagic.net Mastodon instance.
Sidekiq is an early scaling bottleneck with the default configuration for Mastodon. Here’s what we’ve done to help scale Sidekiq for eigenmagic.net.
We’ve configured eigenmagic.net to use multiple Sidekiq processes, which we start using systemd unit files. We’ve also collected the unit files into a new systemd target.
Here’s how to configure this setup.
Creating a new systemd target
We chose to collect all our systemd units into a new target to help group things together logically. Adding a target is relatively straightforward. You create a new .target
file in /lib/systemd/system
:
[Unit]
Description=Mastodon app server
Requires=network.target
Wants=multi-user.target
After=network.target
[Install]
WantedBy=multi-user.target
The [Unit]
pieces define the unit file, while [Install]
tells systemd how to install this target into its tree of targets.
WantedBy=multi-user.target
tells systemd that our mastodon.target
should be run at the same time as the multi-user.target
.
Requires=network.target
tells systemd not to start this mastodon.target
if the network.target
hasn’t been reached successfully. If there’s no network, we don’t want to try starting the Mastodon units.
Wants=multi-user.target
is the Unit version of the Install definition WantedBy
. It tells systemd to wait until multi-user.target
is being started to start Mastodon, but that it can start at the same time.
After=network.target
tells systemd that, unlike Wants
, it should wait until after the network.target
is finished before starting Mastodon.
Mastodon Unit Files
We have three kinds of unit files:
- The streaming frontend service
- The web frontend service
- The Sidekiq queue processors
The streaming service
The streaming service has a unit file in /lib/systemd/system/mastodon-streaming.service
with the following contents:
[Unit]
Description=mastodon-streaming
After=network.target
[Service]
Type=simple
User=mastodon
WorkingDirectory=/home/mastodon/live
Environment="NODE_ENV=production"
Environment="PORT=4000"
Environment="BIND=::"
Environment="STREAMING_CLUSTER_NUM=2"
ExecStart=/home/mastodon/bin/node ./streaming
TimeoutSec=15
Restart=always
# Capabilities
CapabilityBoundingSet=
# Security
NoNewPrivileges=true
# Sandboxing
ProtectSystem=strict
PrivateTmp=true
PrivateDevices=true
PrivateUsers=true
ProtectHostname=true
ProtectKernelLogs=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectControlGroups=true
RestrictAddressFamilies=AF_INET
RestrictAddressFamilies=AF_INET6
RestrictAddressFamilies=AF_NETLINK
RestrictAddressFamilies=AF_UNIX
RestrictNamespaces=true
LockPersonality=true
RestrictRealtime=true
RestrictSUIDSGID=true
RemoveIPC=true
PrivateMounts=true
ProtectClock=true
# System Call Filtering
SystemCallArchitectures=native
SystemCallFilter=~@cpu-emulation @debug @keyring @ipc @memlock @mount @obsolete @privileged @resources @setuid
SystemCallFilter=pipe
SystemCallFilter=pipe2
ReadWritePaths=/home/mastodon/live
[Install]
WantedBy=mastodon.target
I won’t describe all these parameters as you can look them up. I’ll just focus on the ones used to define the service and link it to our target.
The [Install]
section tells systemd that this service unit is part of the mastodon.target
so when it gets installed, systemd will create a link to it under /etc/systemd/system/mastodon.target.wants
. More on that in a moment.
In the [Unit]
definition, After=network.target
tells systemd not to start this unit until the network.target
is finished, because without a network, the streaming service is pointless, and it tends to spin-wait and log a lot of complaints about not being able to access the database (which is on a different system, accessed over the network).
Environment="STREAMING_CLUSTER_NUM=n"
is an important tuning parameter that defines how many streaming processes get started. This is a candidate for templating, but we haven’t needed to do that yet, unlike the sidekiq units that we’ll get to shortly.
The web frontend service
The web frontend service unit looks very similar to the streaming service unit definition. It goes in /lib/systemd/system/mastodon-web.service
with the following contents:
[Unit]
Description=mastodon-web
After=network.target
[Service]
Type=simple
User=mastodon
WorkingDirectory=/home/mastodon/live
Environment="RAILS_ENV=production"
Environment="PORT=3000"
Environment="LD_PRELOAD=libjemalloc.so"
Environment="BIND=localhost"
Environment="WEB_CONCURRENCY=3"
Environment="MAX_THREADS=10"
ExecStart=/home/mastodon/.rbenv/shims/bundle exec puma -C config/puma.rb
ExecReload=/bin/kill -SIGUSR1 $MAINPID
TimeoutSec=15
Restart=always
# Capabilities
CapabilityBoundingSet=
# Security
NoNewPrivileges=true
# Sandboxing
ProtectSystem=strict
PrivateTmp=true
PrivateDevices=true
PrivateUsers=true
ProtectHostname=true
ProtectKernelLogs=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectControlGroups=true
RestrictAddressFamilies=AF_INET
RestrictAddressFamilies=AF_INET6
RestrictAddressFamilies=AF_NETLINK
RestrictAddressFamilies=AF_UNIX
RestrictNamespaces=true
LockPersonality=true
RestrictRealtime=true
RestrictSUIDSGID=true
RemoveIPC=true
PrivateMounts=true
ProtectClock=true
# System Call Filtering
SystemCallArchitectures=native
SystemCallFilter=~@cpu-emulation @debug @keyring @ipc @mount @obsolete @privileged @setuid
SystemCallFilter=@chown
SystemCallFilter=pipe
SystemCallFilter=pipe2
ReadWritePaths=/home/mastodon/live
[Install]
WantedBy=mastodon.target
The systemd [Install]
and [Unit]
definitions are essentially the same as for the streaming service; only the description has changed.
I do want to explain a few important scaling parameters in the service definition.
Environment="LD_PRELOAD=libjemalloc.so"
is important if you’re using Linux, because if Ruby uses the default malloc()
library, it tends to consume more memory than it really needs due to the way malloc()
works. If you tell Puma (which is written in Ruby) to use jemalloc
instead (via this LD_PRELOAD
environment variable) then you should see lower memory consumption on Linux for the web frontend.
Environment="WEB_CONCURRENCY=n"
defines how many puma worker processes to start. Each process will use up to Environment="MAX_THREADS=m"
threads. The configuration above gives us 30 threads in total, spread across 3 processes.
The primary author of Puma advises that you should set WEB_CONCURRENCY
to 1.5*server_core_count
and then tune the threads.
Ruby has a global lock (much like Python) so there is a limit to how many threads you can add to a single process before you hit diminishing returns. Between 25-50 seems to be the rule of thumb. After that, you’re better off scaling by adding more processes. This is the technique we use with Sidekiq.
Sidekiq queue services
Because of the thread scaling challenge above, we’ve split out each sidekiq queue into its own process. Mastodon has 6 queues at time of writing:
- default – the default queue, does what it says on the tin.
- scheduler – schedules jobs for other code to deal with later.
- mailers – handles email notifications, such as for new account signups and password reset requests.
- ingress – receives inbound messages from remote instances.
- push – sends messages to remote instances.
- pull – pulls in certain remote notification messages.
We’ve set up a single templated unit file that we use for all the sidekiq queues. It looks like this:
[Unit]
Description=Mastodon Sidekiq %j queue
After=network.target
[Service]
Type=simple
User=mastodon
WorkingDirectory=/home/mastodon/live
Environment="RAILS_ENV=production"
Environment="DB_POOL=%i"
Environment="MALLOC_ARENA_MAX=2"
Environment="LD_PRELOAD=libjemalloc.so"
ExecStart=/home/mastodon/.rbenv/shims/bundle exec sidekiq -c %i -q %j
TimeoutSec=15
Restart=always
[Install]
WantedBy=mastodon.target
The [Install]
section should be familiar to you by now. It tells systemd that this unit file is wanted by the mastodon.target
we’ve defined.
The [Unit]
section has an important parameter in the Description
field: %j
. This parameter gets replaced by whatever comes between the last ‘-‘ character in the installed unit filename and the ‘@’ symbol. This lets us use the same unit content for multiple sidekiq unit definitions.
You should also notice the %i parameter used a couple of places. %i
gets replaced by whatever comes immediately after the ‘@
‘ symbol once the unit is installed, and we use it to semi-dynamically set the number of threads used by the process.
Some important tuning parameters
We once again set Environment="LD_PRELOAD=libjemalloc.so"
so that Ruby uses the more efficient jemalloc()
memory allocator on Linux.
We also set Environment="MALLOC_ARENA_MAX=2"
which reduces the memory consumption of Sidekiq dramatically. According to the description in the sidekiq Github wiki here, “[o]n Linux, glibc clashes very badly with Ruby’s multithreading and bloats quickly.”
We saw the memory consumption of the sidekiq queues drop by half when we added this environment setting. It looks well worth trying if you’re seeing a lot of memory consumption by sidekiq queues.
Creating the templates
If you put this content into a file called /lib/systemd/system/mastodon-sidekiq-queue@.service
it will be a template file we can use for creating multiple queue
processes with systemd commands. We have the following files in /lib/systemd/system/
to define our sidekiq unit templates, which are all just copies of the same content as above.
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
Using the templates
To use these templates, we need to tell systemd about them first:
sudo systemctl daemon-reload
Then we enable the target:
sudo systemctl enable mastodon.target
Then we enable the streaming and web units:
sudo systemctl enable mastodon-streaming.service
sudo systemctl enable mastodon-web.service
Then we enable each sidekiq queue with a defined queue size, like so:
sudo systemctl enable [email protected]
sudo systemctl enable [email protected]
sudo systemctl enable [email protected]
sudo systemctl enable [email protected]
sudo systemctl enable [email protected]
sudo systemctl enable [email protected]
Then you can start all the services with something like:
sudo systemctl start mastodon-*
Say we wanted to increase the number of threads the default service is using from 25 to 40. We would use these commands:
sudo systemctl stop mastodon-sidekiq-default
sudo systemctl disable mastodon-sidekiq-default@25
sudo systemctl enable mastodon-sidekiq-default@40
sudo systemctl start mastodon-sidekiq-default
We can also use multiple processes by copying an existing sidekiq unit definition to a new one, like so:
sudo cp /lib/systemd/system/[email protected] /lib/systemd/system/[email protected]
sudo systemctl enable mastodon-sidekiq-2-push@40
sudo systemctl start mastodon-sidekiq-2-push
Though we’d probably move to a better naming convention like mastodon-sidekiq-nn-<queuename>
I hope that helps a few instance admins out there!