Connecting NGINX Reverse Stream Proxy To Docker With Traefik

This post continues on from the first and second post in this series on setting up a reverse proxy lab.

In the last post we configured NGINX to forward traffic for three different URLs to different backend servers:

1. https://playnice.eigenmagic.net, an existing project running on its own web stack on other01(10.10.1.11/24)
2. https://project-eschatron.eigenmagic.net, an existing internal project running on int01 on (10.10.1.201/24)
3. https://webification.eigenmagic.net, a new project running on ext01 (10.10.1.101/24)

In this post we’re going to look closer at how we configure NGINX and Traefik to play nicely with each other.

The playnice project is the simplest and easiest to describe, so we’ll start there, then move on to project-eschatron which is a bit more complex, and finally webification which combines some of the techniques from the previous two.

Playnice

To make playnice work, we combine the stream proxy from the last post with a simple NGINX server listening on port 443 for inbound connections for server_name playnice.eigenmagic.net.

The stream proxy upgrades all connections on port 80 to port 443, and then inspects the SNI heading to see which host the traffic is trying to reach. For playnice.eigenmagic.net, a PROXY connection is made to 10.10.1.11 port 443, which is running on the other01 VM.

On this VM, we have an NGINX virtual host configuration listening for PROXY traffic like so:

map $http_upgrade $connection_upgrade {
  default upgrade;
  ''      close;
}

log_format proxied '$proxy_protocol_addr - $remote_user [$time_local] '
                   '"$request" $status $body_bytes_sent '
                   '"$http_referer" "$http_user_agent"';

server {
  listen 80 proxy_protocol;
  listen [::]:80 proxy_protocol;
  server_name playnice.eigenmagic.net;
  root /var/www/playnice;

  location / { return 301 https://eigenmagic.net$request_uri; }

  access_log /var/log/nginx/playnice.access.log proxied;
}

server {
  listen 443 ssl http2 proxy_protocol;
  listen [::]:443 ssl http2 proxy_protocol;
  server_name playnice.eigenmagic.net;

  ssl_protocols TLSv1.3;
  ssl_ciphers HIGH:!MEDIUM:!LOW:!aNULL:!NULL:!SHA;
  ssl_prefer_server_ciphers on;
  ssl_session_cache shared:SSL:10m;

  ssl_certificate     /etc/letsencrypt/live/playnice.eigenmagic.net/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/playnice.eigenmagic.net/privkey.pem;

  access_log /var/log/nginx/mastodon.access.log proxied;
}

The key part here is the server{} block with the listen directive that includes proxy_protocol to turn on PROXY protocol support, combined with the server_name directive to make this a virtual server.

The rest of the server{} block is just the usual location{} blocks and various settings related to regular webserver operations that don’t affect the way the traffic proxying works. The only new addition to the configuration required to support the NGINX stream proxy we set up is adding proxy_protocol to the listen directive.

Project Eschatron

For https://project-eschatron.eigenmagic.net/ we have a slightly more complex setup involving a 3 container web stack (nginx-web, wordpress, db) with a frontend Traefik proxy/load-balancer container, all running in Docker.

The NGINX stream proxy forwards traffic for project-eschatron.eigenmagic.net on port 443 to 10.10.1.201 on port 443 where Traefik is listening.

The relevant bits of the Traefik config are in a docker-compose.yml file, like so:

version: '3.3'

services:
  traefik:
    # The official Traefik docker image
    image: traefik:latest
    container_name: traefikint
    command: 
      - "--api=true"
      - "--api.dashboard=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--providers.file.directory=/config/"
      - "--providers.file.watch=true"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--entrypoints.adminsecure.address=:4443"

      # Make all HTTP switch to HTTPS
      - "--entrypoints.web.http.redirections.entryPoint.to=websecure"
    labels:
      # Enable the Traefik dashboard, securely on a specific hostname
      traefik.enable: "true"
      traefik.http.routers.traefik.rule: "Host(`traefik-internal.eigenmagic.net`)"
      traefik.http.routers.traefik.entrypoints: websecure
      traefik.http.routers.traefik.service: "api@internal"
      traefik.http.routers.traefik.tls: "true"
      
      traefik.http.routers.traefik.middlewares: "traefik-auth"
      traefik.http.middlewares.traefik-auth.ipwhitelist.sourcerange: "10.10.1.2/32"
    ports:
      # The HTTP port
      - "80:80"
      # HTTPS port
      - "443:443"

    volumes:
      # So that Traefik can listen to the Docker events
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ${APPVOLBASE}/traefik:/config
      - ${APPVOLBASE}/letsencrypt:/letsencrypt
      - ${APPVOLBASE}/traefik/logs:/logs

    restart: ${RESTART_POLICY}

    networks:
      - proxy

networks:
  proxy:
    external: true

The entrypoints define the ports that Traefik will pay attention to: port 80 (for unencrypted requests) and port 443 (encrypted requests). We add the web.http.redirections.entryPoint to force all attempts to use port 80 to upgrade to TLS encryption on port 443.

These ports are only inside Docker at this point, so we need to connect them to the outside world using the ports: directive, which connects port 80 outside to port 80 inside, and the same for port 443.

Note that this is only inside the proxy network, which is a bridge network defined externally as part of the Docker configuration. This Docker network is how we will connect requests from the outside world into Traefik, and from there into other services that join the proxy network. All other containers will be kept in isolated Docker networks that can’t be seen outside Docker. All traffic has to come in via Traefik.

Then we use labels to get Traefik to respond to Docker events and connect entrypoints to services. We define one for Traefik itself here, which exposes the traefik-internal.eigenmagic.net site on port 443 (websecure) but only allows a single host (10.10.1.2/32) to access it. Everyone else will get an HTTP 403 error.

Now, to join our Project Eschatron WordPress web stack into Traefik, we spin up a set of containers with a docker-compose.yml file like so:

version: '3.3'

secrets:
  wp_pe_db_password:
    file: ${APPVOLBASE}/secrets/wp_pe_db_password

services:

  db:
    image: mysql
    restart: ${RESTART_POLICY}
    expose:
      - 3306
    secrets:
      - wp_pe_db_password
    environment:
      MYSQL_DATABASE: wp_eschatron
      MYSQL_USER: wp_eschatron
      MYSQL_PASSWORD_FILE: /run/secrets/wp_pe_db_password
      MYSQL_RANDOM_ROOT_PASSWORD: '1'
    cap_add:
      - SYS_NICE # Gets rid of the mbind: Operation not permitted log errors
    volumes:
      - ${APPVOLBASE}/wp-eschatron/mysql:/var/lib/mysql
    networks:
      - default

  wordpress:
    depends_on:
      - db
    image: wordpress:fpm
    restart: ${RESTART_POLICY}
    labels:
      traefik.enable: "false"

    secrets:
      - wp_pe_db_password

    environment:
      WORDPRESS_DB_HOST: db:3306
      WORDPRESS_DB_USER: wp_eschatron
      WORDPRESS_DB_PASSWORD_FILE: /run/secrets/wp_pe_db_password
      WORDPRESS_DB_NAME: wp_eschatron
      WORDPRESS_CONFIG_EXTRA: |
        define('WP_SITEURL', 'https://project-eschatron.eigenmagic.net/');
        define('WP_HOME','https://project-eschatron.eigenmagic.net/);"

    volumes:
      - ${APPVOLBASE}/wp-eschatron/wordpress:/var/www/html

    networks:
      - default

  # NGINX to drive the wordpress php-fpm container
  nginx:
    depends_on:
      - wordpress
    image: nginx:alpine
    restart: ${RESTART_POLICY}
    labels:
      traefik.enable: "true"
      traefik.http.routers.nginx.rule: "Host(`project-eschatron.eigenmagic.net`)"

      traefik.http.routers.nginx.entrypoints: "websecure"
      traefik.http.routers.nginx.tls: "true"
      traefik.http.services.nginx.loadbalancer.server.port: "80"

      traefik.backend: wordpress

    expose:
      - 80

    volumes:
      - ${APPVOLBASE}/wp-pivotnine/nginx/conf:/etc/nginx/conf.d
      - ${APPVOLBASE}/wp-pivotnine/nginx/logs:/var/log/nginx
      - ${APPVOLBASE}/wp-pivotnine/wordpress:/var/www/html
      
    networks:
      - default
      - proxy

networks:
  proxy:
    external: true

The key parts here are the NGINX container configuration.

The NGINX container exposes port 80 and connects to networks proxy and default. The proxy network connection allows Traefik to forward web traffic to it, and the default network lets it talk to the other containers in the web stack. We don’t further isolate the wordpress or database containers from the nginx proxy, though we could.

We use labels to tell Traefik how to set things up for us. We define a router (called ‘nginx’) and say it should use this router if the hostname matches project-eschatron.eigenmagic.net. The router connects the websecure entrypoint (port 443 which is connected to the outside world) to a single-port loadbalancer service on port 80, which is where our nginx service is listening. We also tell Traefik to use TLS on its port 443.

Note that the nginx container isn’t using TLS here. The TLS connection is terminated at Traefik and the traffic inside Docker is routed around in the clear, but that’s all internal to Docker so we’re not adding extra overhead of encrypting the inter-container communications.

Webification

Now for the final piece in the puzzle: the webification project!

This project lives alongside the NGINX stream proxy in Docker on the ext01 virtual machine, so its Traefik setup needs to be a little different, mostly to do with the ports it uses to communicate to NGINX.

Because NGINX is using ports 80 and 443 to listen to the outside world, Traefik needs to listen on different ports for HTTP and HTTPS traffic. Other than that, the configuration is pretty much the same as for the Traefik container on int01:

version: '3.3'

services:
  traefik:
    # The official Traefik docker image
    image: traefik:latest
    command: 
      - "--api.dashboard=true"
      - "--accessLog.filepath=/logs/traefik.log"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--providers.file.directory=/config/"
      - "--providers.file.watch=true"
      # These are internal docker ports
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--entryPoints.websecure.proxyProtocol.trustedIPs=10.10.1.101/32"

      # Make all HTTP switch to HTTPS
      - "--entrypoints.web.http.redirections.entryPoint.to=websecure"

    labels:
      # Enable the Traefik dashboard, securely on a specific hostname
      traefik.enable: "true"
      traefik.http.routers.traefik.rule: "Host(`traefik-external.eigenmagic.net`)"
      traefik.http.routers.traefik.entrypoints: websecure
      traefik.http.routers.traefik.service: "api@internal"
      traefik.http.routers.traefik.tls: "true"
    
      traefik.http.routers.traefik.middlewares: "traefik-auth"
      traefik.http.middlewares.traefik-auth.ipwhitelist.sourcerange: "10.10.1.2/32"

    ports:
      # The HTTP port
      - "8080:80"
      # HTTPS port
      - "8443:443"

    volumes:
      # So that Traefik can listen to the Docker events
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ${APPVOLBASE}/letsencrypt:/letsencrypt
      - ${APPVOLBASE}/traefik-prodext:/config
      - ${APPVOLBASE}/traefik-prodext/logs:/logs

    networks:
      - proxy
    restart: ${RESTART_POLICY}

networks:
  proxy:
    external: true

The first thing to notice is the different ports: directive. Note how we’re connecting outside port 8080 to port 80 internally, and port 8443 to port 443. If you go and check the NGINX stream proxy definitions for the upstream servers, you’ll notice we set the webification_backend to be 10.10.1.101:8443 (not port 443) and this is why.

Traefik already understands the PROXY protocol, but we want it to rewrite the client addresses, which means telling it to trust the PROXY traffic coming from the NGINX stream proxy. That’s what setting entryPoints.websecure.proxyProtocol.trustedIPs=10.10.1.101/32 does.

The End!

That’s it!

Hopefully this worked example helps you to understand how this stuff works a bit better. I expect I’ll be referring back to this set of posts when future me wants to understand why some doofus set all this stuff up this way. Hopefully future me appreciates all the effort I went through to document things.

It’s also helped me to consolidate what I learned, and I definitely understand it better now. If you’re struggling with something, maybe try writing down an explanation to future you. I find it helps.