Scaling Mastodon with systemd template units

Part of a series on configuring and running the eigenmagic.net Mastodon instance.

Sidekiq is an early scaling bottleneck with the default configuration for Mastodon. Here’s what we’ve done to help scale Sidekiq for eigenmagic.net.

We’ve configured eigenmagic.net to use multiple Sidekiq processes, which we start using systemd unit files. We’ve also collected the unit files into a new systemd target.

Here’s how to configure this setup.

Creating a new systemd target

We chose to collect all our systemd units into a new target to help group things together logically. Adding a target is relatively straightforward. You create a new .target file in /lib/systemd/system:

[Unit]
Description=Mastodon app server
Requires=network.target
Wants=multi-user.target
After=network.target

[Install]
WantedBy=multi-user.target

The [Unit] pieces define the unit file, while [Install] tells systemd how to install this target into its tree of targets.

WantedBy=multi-user.target tells systemd that our mastodon.target should be run at the same time as the multi-user.target.

Requires=network.target tells systemd not to start this mastodon.target if the network.target hasn’t been reached successfully. If there’s no network, we don’t want to try starting the Mastodon units.

Wants=multi-user.target is the Unit version of the Install definition WantedBy. It tells systemd to wait until multi-user.target is being started to start Mastodon, but that it can start at the same time.

After=network.target tells systemd that, unlike Wants, it should wait until after the network.target is finished before starting Mastodon.

Mastodon Unit Files

We have three kinds of unit files:

  • The streaming frontend service
  • The web frontend service
  • The Sidekiq queue processors

The streaming service

The streaming service has a unit file in /lib/systemd/system/mastodon-streaming.service with the following contents:

[Unit]
Description=mastodon-streaming
After=network.target

[Service]
Type=simple
User=mastodon
WorkingDirectory=/home/mastodon/live
Environment="NODE_ENV=production"
Environment="PORT=4000"
Environment="BIND=::"
Environment="STREAMING_CLUSTER_NUM=2"
ExecStart=/home/mastodon/bin/node ./streaming
TimeoutSec=15
Restart=always
# Capabilities
CapabilityBoundingSet=
# Security
NoNewPrivileges=true
# Sandboxing
ProtectSystem=strict
PrivateTmp=true
PrivateDevices=true
PrivateUsers=true
ProtectHostname=true
ProtectKernelLogs=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectControlGroups=true
RestrictAddressFamilies=AF_INET
RestrictAddressFamilies=AF_INET6
RestrictAddressFamilies=AF_NETLINK
RestrictAddressFamilies=AF_UNIX
RestrictNamespaces=true
LockPersonality=true
RestrictRealtime=true
RestrictSUIDSGID=true
RemoveIPC=true
PrivateMounts=true
ProtectClock=true
# System Call Filtering
SystemCallArchitectures=native
SystemCallFilter=~@cpu-emulation @debug @keyring @ipc @memlock @mount @obsolete @privileged @resources @setuid
SystemCallFilter=pipe
SystemCallFilter=pipe2
ReadWritePaths=/home/mastodon/live

[Install]
WantedBy=mastodon.target

I won’t describe all these parameters as you can look them up. I’ll just focus on the ones used to define the service and link it to our target.

The [Install] section tells systemd that this service unit is part of the mastodon.target so when it gets installed, systemd will create a link to it under /etc/systemd/system/mastodon.target.wants. More on that in a moment.

In the [Unit] definition, After=network.target tells systemd not to start this unit until the network.target is finished, because without a network, the streaming service is pointless, and it tends to spin-wait and log a lot of complaints about not being able to access the database (which is on a different system, accessed over the network).

Environment="STREAMING_CLUSTER_NUM=n" is an important tuning parameter that defines how many streaming processes get started. This is a candidate for templating, but we haven’t needed to do that yet, unlike the sidekiq units that we’ll get to shortly.

The web frontend service

The web frontend service unit looks very similar to the streaming service unit definition. It goes in /lib/systemd/system/mastodon-web.service with the following contents:

[Unit]
Description=mastodon-web
After=network.target

[Service]
Type=simple
User=mastodon
WorkingDirectory=/home/mastodon/live
Environment="RAILS_ENV=production"
Environment="PORT=3000"
Environment="LD_PRELOAD=libjemalloc.so"
Environment="BIND=localhost"
Environment="WEB_CONCURRENCY=3"
Environment="MAX_THREADS=10"
ExecStart=/home/mastodon/.rbenv/shims/bundle exec puma -C config/puma.rb
ExecReload=/bin/kill -SIGUSR1 $MAINPID
TimeoutSec=15
Restart=always
# Capabilities
CapabilityBoundingSet=
# Security
NoNewPrivileges=true
# Sandboxing
ProtectSystem=strict
PrivateTmp=true
PrivateDevices=true
PrivateUsers=true
ProtectHostname=true
ProtectKernelLogs=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectControlGroups=true
RestrictAddressFamilies=AF_INET
RestrictAddressFamilies=AF_INET6
RestrictAddressFamilies=AF_NETLINK
RestrictAddressFamilies=AF_UNIX
RestrictNamespaces=true
LockPersonality=true
RestrictRealtime=true
RestrictSUIDSGID=true
RemoveIPC=true
PrivateMounts=true
ProtectClock=true
# System Call Filtering
SystemCallArchitectures=native
SystemCallFilter=~@cpu-emulation @debug @keyring @ipc @mount @obsolete @privileged @setuid
SystemCallFilter=@chown
SystemCallFilter=pipe
SystemCallFilter=pipe2
ReadWritePaths=/home/mastodon/live

[Install]
WantedBy=mastodon.target

The systemd [Install] and [Unit] definitions are essentially the same as for the streaming service; only the description has changed.

I do want to explain a few important scaling parameters in the service definition.

Environment="LD_PRELOAD=libjemalloc.so" is important if you’re using Linux, because if Ruby uses the default malloc() library, it tends to consume more memory than it really needs due to the way malloc() works. If you tell Puma (which is written in Ruby) to use jemalloc instead (via this LD_PRELOAD environment variable) then you should see lower memory consumption on Linux for the web frontend.

Environment="WEB_CONCURRENCY=n" defines how many puma worker processes to start. Each process will use up to Environment="MAX_THREADS=m" threads. The configuration above gives us 30 threads in total, spread across 3 processes.

The primary author of Puma advises that you should set WEB_CONCURRENCY to 1.5*server_core_count and then tune the threads.

Ruby has a global lock (much like Python) so there is a limit to how many threads you can add to a single process before you hit diminishing returns. Between 25-50 seems to be the rule of thumb. After that, you’re better off scaling by adding more processes. This is the technique we use with Sidekiq.

Sidekiq queue services

Because of the thread scaling challenge above, we’ve split out each sidekiq queue into its own process. Mastodon has 6 queues at time of writing:

  • default – the default queue, does what it says on the tin.
  • scheduler – schedules jobs for other code to deal with later.
  • mailers – handles email notifications, such as for new account signups and password reset requests.
  • ingress – receives inbound messages from remote instances.
  • push – sends messages to remote instances.
  • pull – pulls in certain remote notification messages.

We’ve set up a single templated unit file that we use for all the sidekiq queues. It looks like this:

[Unit]
Description=Mastodon Sidekiq %j queue
After=network.target

[Service]
Type=simple
User=mastodon
WorkingDirectory=/home/mastodon/live
Environment="RAILS_ENV=production"
Environment="DB_POOL=%i"
Environment="MALLOC_ARENA_MAX=2"
Environment="LD_PRELOAD=libjemalloc.so"
ExecStart=/home/mastodon/.rbenv/shims/bundle exec sidekiq -c %i -q %j
TimeoutSec=15
Restart=always

[Install]
WantedBy=mastodon.target

The [Install] section should be familiar to you by now. It tells systemd that this unit file is wanted by the mastodon.target we’ve defined.

The [Unit] section has an important parameter in the Description field: %j. This parameter gets replaced by whatever comes between the last ‘-‘ character in the installed unit filename and the ‘@’ symbol. This lets us use the same unit content for multiple sidekiq unit definitions.

You should also notice the %i parameter used a couple of places. %i gets replaced by whatever comes immediately after the ‘@‘ symbol once the unit is installed, and we use it to semi-dynamically set the number of threads used by the process.

Some important tuning parameters

We once again set Environment="LD_PRELOAD=libjemalloc.so" so that Ruby uses the more efficient jemalloc() memory allocator on Linux.

We also set Environment="MALLOC_ARENA_MAX=2" which reduces the memory consumption of Sidekiq dramatically. According to the description in the sidekiq Github wiki here, “[o]n Linux, glibc clashes very badly with Ruby’s multithreading and bloats quickly.”

We saw the memory consumption of the sidekiq queues drop by half when we added this environment setting. It looks well worth trying if you’re seeing a lot of memory consumption by sidekiq queues.

Creating the templates

If you put this content into a file called /lib/systemd/system/mastodon-sidekiq-queue@.service it will be a template file we can use for creating multiple queue processes with systemd commands. We have the following files in /lib/systemd/system/ to define our sidekiq unit templates, which are all just copies of the same content as above.

[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]

Using the templates

To use these templates, we need to tell systemd about them first:

sudo systemctl daemon-reload

Then we enable the target:

sudo systemctl enable mastodon.target

Then we enable the streaming and web units:

sudo systemctl enable mastodon-streaming.service
sudo systemctl enable mastodon-web.service

Then we enable each sidekiq queue with a defined queue size, like so:

sudo systemctl enable [email protected]
sudo systemctl enable [email protected]
sudo systemctl enable [email protected]
sudo systemctl enable [email protected]
sudo systemctl enable [email protected]
sudo systemctl enable [email protected]

Then you can start all the services with something like:

sudo systemctl start mastodon-*

Say we wanted to increase the number of threads the default service is using from 25 to 40. We would use these commands:

sudo systemctl stop mastodon-sidekiq-default
sudo systemctl disable mastodon-sidekiq-default@25
sudo systemctl enable mastodon-sidekiq-default@40
sudo systemctl start mastodon-sidekiq-default

We can also use multiple processes by copying an existing sidekiq unit definition to a new one, like so:

sudo cp /lib/systemd/system/[email protected] /lib/systemd/system/[email protected]
sudo systemctl enable mastodon-sidekiq-2-push@40
sudo systemctl start mastodon-sidekiq-2-push

Though we’d probably move to a better naming convention like mastodon-sidekiq-nn-<queuename>

I hope that helps a few instance admins out there!

Bookmark the permalink.

Comments are closed.