A load balancer after Docker

docker run and docker-compose up felt like enough yesterday. Then real traffic arrived. One container is choking. You start a second, a third. Docker does not answer the next question: who gets the following request, and what if one of them is already dead.

A load balancer is a proxy that takes incoming requests and spreads them across copies of the app.

Three cases where you already need one

A shop. Weekdays 100 RPS, one container lives. Friday night 800 RPS, the same container dies. Five copies behind a balancer hold about 700 RPS, updates roll without a window.

An update without one: stop the container, ship the new one, start it. The site is down 30-60 seconds. With one: start the new copies, check them, switch traffic, kill the old ones.

One container dies of OOM or a network blip. Without a balancer the user sees 502. With one, requests go only to live instances.

How it decides

It checks backend health. It does not give ten jobs to one driver while the others sit. When needed it remembers who the client already spoke to.

User → Load Balancer (port 80/443) → Backend servers (containers)
              ↓
      Health checks every N seconds
              ↓
      Routing by algorithm

Which algorithm

Round robin: around the circle. Identical containers, requests of roughly the same weight. A mobile API where every call costs about the same.

Least connections: the next request goes where there are fewer live connections. A 50 ms GET sits next to a five-second POST.

IP hash: one IP always hits one container. Only makes sense if state still lives in process memory. That is a crutch. Sessions belong in Redis or Postgres.

Weighted round robin: some containers get more than others. Uneven hardware, or a canary: the new version gets 10%, the old one 90%.

Nginx, HAProxy, Traefik

Nginx: HTTP server, proxy, static files, cache, SSL. Config in files, apply with nginx -s reload. The default for most jobs.

upstream backend {
    server app1:8000;
    server app2:8000;
    server app3:8000;
}
 
server {
    listen 80;
    server_name example.com;
 
    location / {
        proxy_pass http://backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

HAProxy: 100k+ RPS on one machine, TCP and HTTP, can balance Postgres and Redis, metrics out of the box. It can do static files and SSL, that is not its job. The syntax is its own.

frontend http_front
    bind *:80
    default_backend app_servers
 
backend app_servers
    balance roundrobin
    option httpchk GET /health
    server app1 app1:8000 check
    server app2 app2:8000 check
    server app3 app3:8000 check

Traefik: finds services in Docker and Kubernetes by itself, fetches Let's Encrypt certs, config via labels. Heavier than Nginx and HAProxy, too much for one service.

version: "3"
 
services:
  traefik:
    image: traefik:v2.10
    command:
      - "--providers.docker=true"
      - "--entrypoints.web.address=:80"
    ports:
      - "80:80"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
 
  app:
    image: myapp:latest
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.app.rule=Host(`example.com`)"
    deploy:
      replicas: 3

Three copies behind Nginx in five minutes

version: "3.8"
 
services:
  nginx:
    image: nginx:alpine
    ports:
      - "80:80"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - app1
      - app2
      - app3
 
  app1:
    image: hashicorp/http-echo
    command: ["-text", "Hello from app1"]
    expose:
      - "5678"
 
  app2:
    image: hashicorp/http-echo
    command: ["-text", "Hello from app2"]
    expose:
      - "5678"
 
  app3:
    image: hashicorp/http-echo
    command: ["-text", "Hello from app3"]
    expose:
      - "5678"

events {
    worker_connections 1024;
}
 
http {
    upstream backend {
        server app1:5678;
        server app2:5678;
        server app3:5678;
    }
 
    server {
        listen 80;
 
        location / {
            proxy_pass http://backend;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
 
        location /health {
            access_log off;
            return 200 "OK\n";
            add_header Content-Type text/plain;
        }
    }
}

docker-compose up

# Make several requests
curl http://localhost
# Hello from app1
 
curl http://localhost
# Hello from app2
 
curl http://localhost
# Hello from app3
 
curl http://localhost
# Hello from app1 (cycle repeats)

Thirty lines. Kill one container: the other two keep answering.

Health check

The balancer is not a psychic. It needs an endpoint that only lies if the app is actually ready: the database answers, Redis is up, critical dependencies are there. A bare 200 OK will send traffic to a process that no longer serves people.

from fastapi import FastAPI
 
app = FastAPI()
 
@app.get("/health")
async def health():
    # Here you can check DB, Redis, etc.
    return {"status": "ok"}

upstream backend {
    server app1:8000 max_fails=3 fail_timeout=30s;
    server app2:8000 max_fails=3 fail_timeout=30s;
}

backend app_servers
    option httpchk GET /health
    http-check expect status 200
    server app1 app1:8000 check inter 5s fall 3 rise 2
    server app2 app2:8000 check inter 5s fall 3 rise 2

check inter 5s: every five seconds. fall 3: after three failures the server is dead. rise 2: after two successes it is alive again.

Typical holes

The user logged in on app1, the next request went to app2, no session there. Either sticky by cookie / IP, or (better) sessions in Redis.

The app sees the balancer IP, analytics lie.

proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;

The app has to read X-Real-IP or X-Forwarded-For.

The balancer itself also dies. I watch its uptime, how many backends are healthy, latency, error rate, RPS. Prometheus and Grafana, and in the cloud whatever is already paid for.

You do not need one when there is a single container and steady load, when this is local development, when there is nothing to balance: one Postgres, one Redis master. Read replicas are a different conversation.

What this opens

Today addresses are written by hand: app1, app2, app3. In Kubernetes that is replicas: 3, services find each other, a rolling update is one command. Same ideas, different scale.

See also:

• Load Testing
• Proxmox