Skip to main content

Docker Compose and Kubernetes: when to move

Constantin Potapov
18 min

Three servers, SSH, a prayer. 5000 RPS. After that Compose becomes support hell. Helm, kind, and when it is better to stay.

The API held about 5000 RPS at peak. Compose ran eight copies on three servers. Deploy: SSH to each, git pull, docker-compose up -d --build, a prayer. It usually fell over.

Rolling update by hand, 30-60 seconds of downtime per server. A machine died at night, the balancer kept sending traffic there, we learned in the morning from customers. No autoscaling: I spin up a server, an hour later it sits idle, I pay for air. The health check asked whether the container was alive, not whether the app was.

Scaling Compose further meant feeding support hell. We needed an orchestrator. Not necessarily Kubernetes.

Compose is fine locally and on one to three servers up to about 500 RPS. After that downtime starts to cost money.

What Compose does not close

With Compose even docker-compose up -d --no-deps --build service leaves a gap of a few seconds. A user sees it.

In Kubernetes new pods start, pass readiness, take traffic, old ones leave. Rollback: kubectl rollout undo.

# Deploy new version
kubectl set image deployment/myapp myapp=myapp:v2
 
# Rollback if something went wrong
kubectl rollout undo deployment/myapp

A container dies: restart: unless-stopped brings it back. A server dies: SSH to a new one and stand the stack up. In k8s a pod dies, the scheduler starts another. A node dies, pods move to live ones.

In Compose you look at state over SSH. In k8s you write replicas: 5 and kubectl apply. Git is the source of truth. ArgoCD or Flux apply it. Rollback: git revert.

# Commit changes to manifests
git commit -m "Scale app to 10 replicas"
git push
 
# ArgoCD or Flux automatically applies changes
# Rollback? Just git revert and push

healthcheck in Compose: the container answered /health. The database is down, the container is up, users get 500. In k8s two probes. Liveness: kill and start again. Readiness: no traffic until ready.

livenessProbe:
  httpGet:
    path: /health/live
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10
 
readinessProbe:
  httpGet:
    path: /health/ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5

Autoscaling in Compose: by hand or cron. Horizontal Pod Autoscaler watches CPU, memory, or a custom metric from Prometheus.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: myapp-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
Docker Compose
Kubernetes
Deployment
SSH + prayer
kubectl apply (zero-downtime)
Downtime
30-60 sec per server
0 seconds
100%
Autoscaling
Manual or hacks
HPA automatic
Recovery
Manual at 3 AM
Self-healing built-in

I leave Compose for local work, a monolith on one or two servers at 100-200 RPS, an MVP, and a team without DevOps. There Render or Fly.io is simpler than a cluster.

Kubernetes solves hard problems. If the problems are simple, it creates new ones.

The minimum

You tell the cluster: five copies, 2 CPU, 4 GB. It decides where to put them, watches, restarts.

Pod: one or more containers on one node.

apiVersion: v1
kind: Pod
metadata:
  name: myapp-pod
spec:
  containers:
    - name: myapp
      image: myapp:latest
      ports:
        - containerPort: 8080

Deployment: how many replicas, which image, how to update.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
        - name: myapp
          image: myapp:v1.0.0
          ports:
            - containerPort: 8080

Service: a stable IP and DNS. ClusterIP inside, NodePort outside, LoadBalancer in the cloud.

apiVersion: v1
kind: Service
metadata:
  name: myapp-service
spec:
  selector:
    app: myapp
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
  type: LoadBalancer

Ingress: HTTP from outside in.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp-ingress
spec:
  rules:
    - host: myapp.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: myapp-service
                port:
                  number: 80

ConfigMap and Secret instead of variables in compose.

apiVersion: v1
kind: ConfigMap
metadata:
  name: myapp-config
data:
  DATABASE_URL: "postgresql://user@db:5432/mydb"
  LOG_LEVEL: "info"
---
apiVersion: v1
kind: Secret
metadata:
  name: myapp-secrets
type: Opaque
data:
  DB_PASSWORD: cGFzc3dvcmQxMjM= # base64 encoded

k8s places pods, scales, heals, gives DNS, rolling updates, secrets. It does not give monitoring, CI, backups or security. That is Prometheus, ArgoCD, Velero, RBAC on the side.

Helm

One app: Deployment, Service, Ingress, ConfigMap, Secret. Two hundred lines. Three environments with different replicas and domains. Without Helm: copy-paste and Find & Replace.

myapp/
  Chart.yaml
  values.yaml
  templates/
    deployment.yaml
    service.yaml
    ingress.yaml
    configmap.yaml

Before:

version: "3.8"
services:
  app:
    image: myapp:latest
    ports:
      - "8080:8080"
    environment:
      DATABASE_URL: postgresql://user:pass@db:5432/mydb
      REDIS_URL: redis://redis:6379
    depends_on:
      - db
      - redis
 
  db:
    image: postgres:15
    environment:
      POSTGRES_PASSWORD: secretpass
    volumes:
      - db_data:/var/lib/postgresql/data
 
  redis:
    image: redis:7-alpine
 
volumes:
  db_data:

After, values.yaml:

replicaCount: 3
 
image:
  repository: myapp
  tag: "v1.0.0"
  pullPolicy: IfNotPresent
 
service:
  type: LoadBalancer
  port: 80
  targetPort: 8080
 
ingress:
  enabled: true
  host: myapp.example.com
 
env:
  DATABASE_URL: postgresql://user@postgres:5432/mydb
  REDIS_URL: redis://redis:6379
 
resources:
  limits:
    cpu: 500m
    memory: 512Mi
  requests:
    cpu: 250m
    memory: 256Mi
 
autoscaling:
  enabled: true
  minReplicas: 3
  maxReplicas: 10
  targetCPUUtilizationPercentage: 70
apiVersion: apps/v1
kind: Deployment
metadata:
  name: { { include "myapp.fullname" . } }
spec:
  replicas: { { .Values.replicaCount } }
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
        - name: myapp
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          ports:
            - containerPort: { { .Values.service.targetPort } }
          env:
            - name: DATABASE_URL
              value: { { .Values.env.DATABASE_URL } }
            - name: REDIS_URL
              value: { { .Values.env.REDIS_URL } }
          resources: { { - toYaml .Values.resources | nindent 10 } }
          livenessProbe:
            httpGet:
              path: /health
              port: { { .Values.service.targetPort } }
            initialDelaySeconds: 30
          readinessProbe:
            httpGet:
              path: /health/ready
              port: { { .Values.service.targetPort } }
            initialDelaySeconds: 5
# Install chart
helm install myapp ./myapp
 
# Upgrade with new parameters
helm upgrade myapp ./myapp --set image.tag=v1.1.0
 
# Rollback to previous version
helm rollback myapp
 
# Uninstall
helm uninstall myapp
# Dev (1 replica, small resources)
helm install myapp-dev ./myapp -f values-dev.yaml
 
# Staging (3 replicas)
helm install myapp-staging ./myapp -f values-staging.yaml
 
# Production (10 replicas, HPA)
helm install myapp-prod ./myapp -f values-prod.yaml
KubernetesHelmDockerkubectl

Ready charts on Artifact Hub.

# PostgreSQL
helm repo add bitnami https://charts.bitnami.com/bitnami
helm install postgres bitnami/postgresql
 
# Redis
helm install redis bitnami/redis
 
# Nginx Ingress Controller
helm install nginx-ingress ingress-nginx/ingress-nginx
 
# Prometheus + Grafana (monitoring)
helm install monitoring prometheus-community/kube-prometheus-stack

Locally

A full cluster is heavy on a laptop. kind, minikube, k3d.

Criterionkindminikubek3d
FoundationDocker containersVM (VirtualBox/Docker)k3s in Docker
Startup speedVery fast (10-20 sec)Slow (1-2 min)Fast (20-30 sec)
RAMLow (~2GB)High (~4-8GB)Low (~1-2GB)
Multi-node clusterYesYes (harder)Yes
LoadBalancer supportVia MetalLBOut of boxOut of box
Closeness to prodVery closeClosek3s is not full k8s
Best forCI/CD, testingLearning, devFast daily dev

kind for CI and closeness to prod k8s. minikube for learning if RAM is cheap. k3d every day.

brew install kind kubectl
 
# Or via binary
curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.20.0/kind-darwin-arm64
chmod +x ./kind
mv ./kind /usr/local/bin/kind
# Basic single-node cluster
kind create cluster --name dev
 
# Multi-node cluster (1 control-plane + 2 workers)
cat <<EOF | kind create cluster --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
EOF
kubectl cluster-info
kubectl get nodes
# Load local image into kind
kind load docker-image myapp:latest --name dev
 
# Deploy
kubectl apply -f deployment.yaml
kubectl get pods
kubectl logs -f <pod-name>
 
# Port forward for access
kubectl port-forward deployment/myapp 8080:8080
# Now available at http://localhost:8080
kind delete cluster --name dev
brew install k3d
 
# Or curl
curl -s https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash
# Cluster with 3 worker nodes and LoadBalancer on port 8080
k3d cluster create dev \
  --agents 3 \
  --port "8080:80@loadbalancer"
kubectl apply -f deployment.yaml
kubectl apply -f service.yaml
 
# If Service type: LoadBalancer, available at localhost:8080
curl http://localhost:8080
k3d cluster stop dev
k3d cluster start dev

Minimum 8 GB RAM and 4 cores, better 16 and 6. I cut Docker Desktop to 6 GB and 4 CPUs. Limits in the manifests. I do not start the frontend if I am testing the API. After tests kubectl delete: a stopped pod still eats RAM.

If the laptop is choking, a $10-20 cloud dev cluster is cheaper than a day of reboots.

How to see what broke

# Get pod list
kubectl get pods
 
# Detailed pod info
kubectl describe pod <pod-name>
 
# Pod logs
kubectl logs <pod-name>
kubectl logs <pod-name> -f  # follow (real-time)
kubectl logs <pod-name> --previous  # logs of previous crashed container
 
# Logs from all deployment pods
kubectl logs -l app=myapp --all-containers=true
 
# Exec into pod (like docker exec)
kubectl exec -it <pod-name> -- /bin/bash
 
# Port forward
kubectl port-forward pod/<pod-name> 8080:8080
 
# Cluster events
kubectl get events --sort-by='.lastTimestamp'
 
# Top (CPU/Memory usage)
kubectl top nodes
kubectl top pods

Pending: describe, usually not enough CPU/memory or nodes. CrashLoopBackOff: logs --previous. Running and silent: readiness. Empty Service: kubectl get endpoints, pods are not ready.

# Add repository
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
 
# Install kube-prometheus-stack (Prometheus + Grafana + Alertmanager)
helm install monitoring prometheus-community/kube-prometheus-stack
 
# Get Grafana password
kubectl get secret monitoring-grafana -o jsonpath="{.data.admin-password}" | base64 --decode
 
# Port forward for access
kubectl port-forward svc/monitoring-grafana 3000:80
# Grafana available at http://localhost:3000 (admin / <password>)
helm repo add grafana https://grafana.github.io/helm-charts
helm install loki grafana/loki-stack \
  --set grafana.enabled=false \
  --set promtail.enabled=true
 
# Add Loki as data source in Grafana
# URL: http://loki:3100

In Explore: {app="myapp"} and {app="myapp"} |= "ERROR".

If not Kubernetes

Nomad: one binary, 50 MB RAM, Docker and not only Docker, Consul and Vault next door. Smaller ecosystem, no Ingress, no managed offer from the clouds. Makes sense for a team under ten on the HashiCorp stack.

job "myapp" {
  datacenters = ["dc1"]
 
  group "app" {
    count = 3
 
    task "web" {
      driver = "docker"
 
      config {
        image = "myapp:v1.0.0"
        ports = ["http"]
      }
 
      resources {
        cpu    = 500
        memory = 512
      }
 
      service {
        name = "myapp"
        port = "http"
 
        check {
          type     = "http"
          path     = "/health"
          interval = "10s"
          timeout  = "2s"
        }
      }
    }
  }
}

ECS: if you are already all-in on AWS. Fargate without servers. Lock-in and price, especially Fargate.

Fly.io: fly deploy, Heroku with Docker, points around the world, three free VMs. Less control, the bill grows with load.

# Install CLI
brew install flyctl
 
# Login
fly auth login
 
# Initialize app
fly launch
 
# Deploy
fly deploy
 
# Scaling
fly scale count 5
fly scale vm shared-cpu-2x

Docker Swarm: if you know Compose you will learn it in a day. docker stack deploy -c docker-compose.yml. Docker Inc barely develops it.

# Initialize Swarm
docker swarm init
 
# Deploy compose file
docker stack deploy -c docker-compose.yml myapp
 
# Scaling
docker service scale myapp_web=5
CriterionKubernetesNomadAWS ECSFly.ioDocker Swarm
ComplexityHighMediumLowVery lowLow
EcosystemHugeMediumAWS-onlySmallDead
FlexibilityMaximumHighMediumLowMedium
Vendor lock-inNoNoAWSFly.ioNo
CostDIY cheap, managed expensiveDIY cheapExpensiveGrowsCheap
Best forMedium/large teamsSmall teams, HashiCorp stackAWS-native projectsStartupsSmall projects

Compose is enough: I stay. Outgrown: Fly or Nomad first. Kubernetes when I need its flexibility, not a line on a CV.

Cost of the move

To deploy: 20-40 hours. To run production: 100-200. Architect: 500+.

Managed EKS/GKE/AKS: control plane $70-150, nodes $50-200, addons yours. Self-hosted: software free, the team is not.

Five microservices, 20-30 pods, about 2000 RPS. GKE $250, 10-15 hours of DevOps a month, two weeks of onboarding. It pays if you save on hand scaling and downtime.

I do not go to k8s if there are fewer than three developers, a monolith on one server, no DevOps, this is an MVP, or the budget is under $500. Managed is expensive, your own cluster without expertise is dangerous.

How I moved

A week or two: the kubernetes.io tutorial, local kind, an audit of services and secrets, managed or own, a registry in CI.

Then the simplest stateless service.

api:
  image: myapi:latest
  ports:
    - "8080:8080"
  environment:
    DATABASE_URL: postgresql://db:5432/mydb
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
        - name: api
          image: myapi:v1.0.0
          ports:
            - containerPort: 8080
          env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: api-secrets
                  key: database-url
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 500m
              memory: 512Mi
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
          readinessProbe:
            httpGet:
              path: /health/ready
              port: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: api
spec:
  selector:
    app: api
  ports:
    - port: 80
      targetPort: 8080
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: main-ingress
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - api.example.com
      secretName: api-tls
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: api
                port:
                  number: 80

I do not rush databases into k8s. RDS and Cloud SQL are simpler. If you still drag them: StatefulSet, PVC, backup, a restore drill. Redis can be a Bitnami chart. Files: S3 or a ReadWriteMany PVC.

Then kube-prometheus-stack and Loki. Traffic 10%, a week of metrics, 50%, 100%. Old Compose lives another two weeks.

I moved in 2021. The first two months were pain and rakes. Then infrastructure that scales without me, stands itself up, and gives ten deploys a day without a window. That is worth the time if the job is hard enough. If not, Compose and sleep are calmer.

See also: