Part of a series on configuring and running the eigenmagic.net Mastodon instance.
Sidekiq is an early scaling bottleneck with the default configuration for Mastodon. Here’s what we’ve done to help scale Sidekiq for eigenmagic.net.
We’ve configured eigenmagic.net to use multiple Sidekiq processes, which we start using systemd unit files. We’ve also collected the unit files into a new systemd target.
Here’s how to configure this setup.
Creating a new systemd target
We chose to collect all our systemd units into a new target to help group things together logically. Adding a target is relatively straightforward. You create a new
.target file in
[Unit] Description=Mastodon app server Requires=network.target Wants=multi-user.target After=network.target [Install] WantedBy=multi-user.target
[Unit] pieces define the unit file, while
[Install] tells systemd how to install this target into its tree of targets.
WantedBy=multi-user.target tells systemd that our
mastodon.target should be run at the same time as the
Requires=network.target tells systemd not to start this
mastodon.target if the
network.target hasn’t been reached successfully. If there’s no network, we don’t want to try starting the Mastodon units.
Wants=multi-user.target is the Unit version of the Install definition
WantedBy. It tells systemd to wait until
multi-user.target is being started to start Mastodon, but that it can start at the same time.
After=network.target tells systemd that, unlike
Wants, it should wait until after the
network.target is finished before starting Mastodon.
Mastodon Unit Files
We have three kinds of unit files:
- The streaming frontend service
- The web frontend service
- The Sidekiq queue processors
The streaming service
The streaming service has a unit file in
/lib/systemd/system/mastodon-streaming.service with the following contents:
[Unit] Description=mastodon-streaming After=network.target [Service] Type=simple User=mastodon WorkingDirectory=/home/mastodon/live Environment="NODE_ENV=production" Environment="PORT=4000" Environment="BIND=::" Environment="STREAMING_CLUSTER_NUM=2" ExecStart=/home/mastodon/bin/node ./streaming TimeoutSec=15 Restart=always # Capabilities CapabilityBoundingSet= # Security NoNewPrivileges=true # Sandboxing ProtectSystem=strict PrivateTmp=true PrivateDevices=true PrivateUsers=true ProtectHostname=true ProtectKernelLogs=true ProtectKernelModules=true ProtectKernelTunables=true ProtectControlGroups=true RestrictAddressFamilies=AF_INET RestrictAddressFamilies=AF_INET6 RestrictAddressFamilies=AF_NETLINK RestrictAddressFamilies=AF_UNIX RestrictNamespaces=true LockPersonality=true RestrictRealtime=true RestrictSUIDSGID=true RemoveIPC=true PrivateMounts=true ProtectClock=true # System Call Filtering SystemCallArchitectures=native [email protected] @debug @keyring @ipc @memlock @mount @obsolete @privileged @resources @setuid SystemCallFilter=pipe SystemCallFilter=pipe2 ReadWritePaths=/home/mastodon/live [Install] WantedBy=mastodon.target
I won’t describe all these parameters as you can look them up. I’ll just focus on the ones used to define the service and link it to our target.
[Install] section tells systemd that this service unit is part of the
mastodon.target so when it gets installed, systemd will create a link to it under
/etc/systemd/system/mastodon.target.wants. More on that in a moment.
After=network.target tells systemd not to start this unit until the
network.target is finished, because without a network, the streaming service is pointless, and it tends to spin-wait and log a lot of complaints about not being able to access the database (which is on a different system, accessed over the network).
Environment="STREAMING_CLUSTER_NUM=n" is an important tuning parameter that defines how many streaming processes get started. This is a candidate for templating, but we haven’t needed to do that yet, unlike the sidekiq units that we’ll get to shortly.
The web frontend service
The web frontend service unit looks very similar to the streaming service unit definition. It goes in
/lib/systemd/system/mastodon-web.service with the following contents:
[Unit] Description=mastodon-web After=network.target [Service] Type=simple User=mastodon WorkingDirectory=/home/mastodon/live Environment="RAILS_ENV=production" Environment="PORT=3000" Environment="LD_PRELOAD=libjemalloc.so" Environment="BIND=localhost" Environment="WEB_CONCURRENCY=3" Environment="MAX_THREADS=10" ExecStart=/home/mastodon/.rbenv/shims/bundle exec puma -C config/puma.rb ExecReload=/bin/kill -SIGUSR1 $MAINPID TimeoutSec=15 Restart=always # Capabilities CapabilityBoundingSet= # Security NoNewPrivileges=true # Sandboxing ProtectSystem=strict PrivateTmp=true PrivateDevices=true PrivateUsers=true ProtectHostname=true ProtectKernelLogs=true ProtectKernelModules=true ProtectKernelTunables=true ProtectControlGroups=true RestrictAddressFamilies=AF_INET RestrictAddressFamilies=AF_INET6 RestrictAddressFamilies=AF_NETLINK RestrictAddressFamilies=AF_UNIX RestrictNamespaces=true LockPersonality=true RestrictRealtime=true RestrictSUIDSGID=true RemoveIPC=true PrivateMounts=true ProtectClock=true # System Call Filtering SystemCallArchitectures=native [email protected] @debug @keyring @ipc @mount @obsolete @privileged @setuid [email protected] SystemCallFilter=pipe SystemCallFilter=pipe2 ReadWritePaths=/home/mastodon/live [Install] WantedBy=mastodon.target
[Unit] definitions are essentially the same as for the streaming service; only the description has changed.
I do want to explain a few important scaling parameters in the service definition.
Environment="LD_PRELOAD=libjemalloc.so" is important if you’re using Linux, because if Ruby uses the default
malloc() library, it tends to consume more memory than it really needs due to the way
malloc() works. If you tell Puma (which is written in Ruby) to use
jemalloc instead (via this
LD_PRELOAD environment variable) then you should see lower memory consumption on Linux for the web frontend.
Environment="WEB_CONCURRENCY=n" defines how many puma worker processes to start. Each process will use up to
Environment="MAX_THREADS=m" threads. The configuration above gives us 30 threads in total, spread across 3 processes.
The primary author of Puma advises that you should set
1.5*server_core_count and then tune the threads.
Ruby has a global lock (much like Python) so there is a limit to how many threads you can add to a single process before you hit diminishing returns. Between 25-50 seems to be the rule of thumb. After that, you’re better off scaling by adding more processes. This is the technique we use with Sidekiq.
Sidekiq queue services
Because of the thread scaling challenge above, we’ve split out each sidekiq queue into its own process. Mastodon has 6 queues at time of writing:
- default – the default queue, does what it says on the tin.
- scheduler – schedules jobs for other code to deal with later.
- mailers – handles email notifications, such as for new account signups and password reset requests.
- ingress – receives inbound messages from remote instances.
- push – sends messages to remote instances.
- pull – pulls in certain remote notification messages.
We’ve set up a single templated unit file that we use for all the sidekiq queues. It looks like this:
[Unit] Description=Mastodon Sidekiq %j queue After=network.target [Service] Type=simple User=mastodon WorkingDirectory=/home/mastodon/live Environment="RAILS_ENV=production" Environment="DB_POOL=%i" Environment="MALLOC_ARENA_MAX=2" Environment="LD_PRELOAD=libjemalloc.so" ExecStart=/home/mastodon/.rbenv/shims/bundle exec sidekiq -c %i -q %j TimeoutSec=15 Restart=always [Install] WantedBy=mastodon.target
[Install] section should be familiar to you by now. It tells systemd that this unit file is wanted by the
mastodon.target we’ve defined.
[Unit] section has an important parameter in the
%j. This parameter gets replaced by whatever comes between the last ‘-‘ character in the installed unit filename and the ‘@’ symbol. This lets us use the same unit content for multiple sidekiq unit definitions.
You should also notice the %i parameter used a couple of places.
%i gets replaced by whatever comes immediately after the ‘
@‘ symbol once the unit is installed, and we use it to semi-dynamically set the number of threads used by the process.
Some important tuning parameters
We once again set
Environment="LD_PRELOAD=libjemalloc.so" so that Ruby uses the more efficient
jemalloc() memory allocator on Linux.
We also set
Environment="MALLOC_ARENA_MAX=2" which reduces the memory consumption of Sidekiq dramatically. According to the description in the sidekiq Github wiki here, “[o]n Linux, glibc clashes very badly with Ruby’s multithreading and bloats quickly.”
We saw the memory consumption of the sidekiq queues drop by half when we added this environment setting. It looks well worth trying if you’re seeing a lot of memory consumption by sidekiq queues.
Creating the templates
If you put this content into a file called
/lib/systemd/system/mastodon-sidekiq-queue@.service it will be a template file we can use for creating multiple
queue processes with systemd commands. We have the following files in
/lib/systemd/system/ to define our sidekiq unit templates, which are all just copies of the same content as above.
[email protected] [email protected] [email protected] [email protected] [email protected] [email protected]
Using the templates
To use these templates, we need to tell systemd about them first:
sudo systemctl daemon-reload
Then we enable the target:
sudo systemctl enable mastodon.target
Then we enable the streaming and web units:
sudo systemctl enable mastodon-streaming.service sudo systemctl enable mastodon-web.service
Then we enable each sidekiq queue with a defined queue size, like so:
sudo systemctl enable [email protected] sudo systemctl enable [email protected] sudo systemctl enable [email protected] sudo systemctl enable [email protected] sudo systemctl enable [email protected] sudo systemctl enable [email protected]
Then you can start all the services with something like:
sudo systemctl start mastodon-*
Say we wanted to increase the number of threads the default service is using from 25 to 40. We would use these commands:
sudo systemctl stop mastodon-sidekiq-default sudo systemctl disable [email protected] sudo systemctl enable [email protected] sudo systemctl start mastodon-sidekiq-default
We can also use multiple processes by copying an existing sidekiq unit definition to a new one, like so:
sudo cp /lib/systemd/system/[email protected] /lib/systemd/system/[email protected] sudo systemctl enable [email protected] sudo systemctl start mastodon-sidekiq-2-push
Though we’d probably move to a better naming convention like
I hope that helps a few instance admins out there!