The API held about 5000 RPS at peak. Compose ran eight copies on three servers. Deploy: SSH to each, git pull, docker-compose up -d --build, a prayer. It usually fell over.
Rolling update by hand, 30-60 seconds of downtime per server. A machine died at night, the balancer kept sending traffic there, we learned in the morning from customers. No autoscaling: I spin up a server, an hour later it sits idle, I pay for air. The health check asked whether the container was alive, not whether the app was.
Scaling Compose further meant feeding support hell. We needed an orchestrator. Not necessarily Kubernetes.
Compose is fine locally and on one to three servers up to about 500 RPS. After that downtime starts to cost money.
What Compose does not close
With Compose even docker-compose up -d --no-deps --build service leaves a gap of a few seconds. A user sees it.
In Kubernetes new pods start, pass readiness, take traffic, old ones leave. Rollback: kubectl rollout undo.
# Deploy new version
kubectl set image deployment/myapp myapp=myapp:v2
# Rollback if something went wrong
kubectl rollout undo deployment/myappA container dies: restart: unless-stopped brings it back. A server dies: SSH to a new one and stand the stack up. In k8s a pod dies, the scheduler starts another. A node dies, pods move to live ones.
In Compose you look at state over SSH. In k8s you write replicas: 5 and kubectl apply. Git is the source of truth. ArgoCD or Flux apply it. Rollback: git revert.
# Commit changes to manifests
git commit -m "Scale app to 10 replicas"
git push
# ArgoCD or Flux automatically applies changes
# Rollback? Just git revert and pushhealthcheck in Compose: the container answered /health. The database is down, the container is up, users get 500. In k8s two probes. Liveness: kill and start again. Readiness: no traffic until ready.
livenessProbe:
httpGet:
path: /health/live
port: 8080
initialDelaySeconds: 30
periodSeconds: 10
readinessProbe:
httpGet:
path: /health/ready
port: 8080
initialDelaySeconds: 5
periodSeconds: 5Autoscaling in Compose: by hand or cron. Horizontal Pod Autoscaler watches CPU, memory, or a custom metric from Prometheus.
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: myapp-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: myapp
minReplicas: 3
maxReplicas: 20
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70I leave Compose for local work, a monolith on one or two servers at 100-200 RPS, an MVP, and a team without DevOps. There Render or Fly.io is simpler than a cluster.
Kubernetes solves hard problems. If the problems are simple, it creates new ones.
The minimum
You tell the cluster: five copies, 2 CPU, 4 GB. It decides where to put them, watches, restarts.
Pod: one or more containers on one node.
apiVersion: v1
kind: Pod
metadata:
name: myapp-pod
spec:
containers:
- name: myapp
image: myapp:latest
ports:
- containerPort: 8080Deployment: how many replicas, which image, how to update.
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
spec:
replicas: 3
selector:
matchLabels:
app: myapp
template:
metadata:
labels:
app: myapp
spec:
containers:
- name: myapp
image: myapp:v1.0.0
ports:
- containerPort: 8080Service: a stable IP and DNS. ClusterIP inside, NodePort outside, LoadBalancer in the cloud.
apiVersion: v1
kind: Service
metadata:
name: myapp-service
spec:
selector:
app: myapp
ports:
- protocol: TCP
port: 80
targetPort: 8080
type: LoadBalancerIngress: HTTP from outside in.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: myapp-ingress
spec:
rules:
- host: myapp.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: myapp-service
port:
number: 80ConfigMap and Secret instead of variables in compose.
apiVersion: v1
kind: ConfigMap
metadata:
name: myapp-config
data:
DATABASE_URL: "postgresql://user@db:5432/mydb"
LOG_LEVEL: "info"
---
apiVersion: v1
kind: Secret
metadata:
name: myapp-secrets
type: Opaque
data:
DB_PASSWORD: cGFzc3dvcmQxMjM= # base64 encodedk8s places pods, scales, heals, gives DNS, rolling updates, secrets. It does not give monitoring, CI, backups or security. That is Prometheus, ArgoCD, Velero, RBAC on the side.
Helm
One app: Deployment, Service, Ingress, ConfigMap, Secret. Two hundred lines. Three environments with different replicas and domains. Without Helm: copy-paste and Find & Replace.
myapp/
Chart.yaml
values.yaml
templates/
deployment.yaml
service.yaml
ingress.yaml
configmap.yaml
Before:
version: "3.8"
services:
app:
image: myapp:latest
ports:
- "8080:8080"
environment:
DATABASE_URL: postgresql://user:pass@db:5432/mydb
REDIS_URL: redis://redis:6379
depends_on:
- db
- redis
db:
image: postgres:15
environment:
POSTGRES_PASSWORD: secretpass
volumes:
- db_data:/var/lib/postgresql/data
redis:
image: redis:7-alpine
volumes:
db_data:After, values.yaml:
replicaCount: 3
image:
repository: myapp
tag: "v1.0.0"
pullPolicy: IfNotPresent
service:
type: LoadBalancer
port: 80
targetPort: 8080
ingress:
enabled: true
host: myapp.example.com
env:
DATABASE_URL: postgresql://user@postgres:5432/mydb
REDIS_URL: redis://redis:6379
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 250m
memory: 256Mi
autoscaling:
enabled: true
minReplicas: 3
maxReplicas: 10
targetCPUUtilizationPercentage: 70apiVersion: apps/v1
kind: Deployment
metadata:
name: { { include "myapp.fullname" . } }
spec:
replicas: { { .Values.replicaCount } }
selector:
matchLabels:
app: myapp
template:
metadata:
labels:
app: myapp
spec:
containers:
- name: myapp
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
ports:
- containerPort: { { .Values.service.targetPort } }
env:
- name: DATABASE_URL
value: { { .Values.env.DATABASE_URL } }
- name: REDIS_URL
value: { { .Values.env.REDIS_URL } }
resources: { { - toYaml .Values.resources | nindent 10 } }
livenessProbe:
httpGet:
path: /health
port: { { .Values.service.targetPort } }
initialDelaySeconds: 30
readinessProbe:
httpGet:
path: /health/ready
port: { { .Values.service.targetPort } }
initialDelaySeconds: 5# Install chart
helm install myapp ./myapp
# Upgrade with new parameters
helm upgrade myapp ./myapp --set image.tag=v1.1.0
# Rollback to previous version
helm rollback myapp
# Uninstall
helm uninstall myapp# Dev (1 replica, small resources)
helm install myapp-dev ./myapp -f values-dev.yaml
# Staging (3 replicas)
helm install myapp-staging ./myapp -f values-staging.yaml
# Production (10 replicas, HPA)
helm install myapp-prod ./myapp -f values-prod.yamlReady charts on Artifact Hub.
# PostgreSQL
helm repo add bitnami https://charts.bitnami.com/bitnami
helm install postgres bitnami/postgresql
# Redis
helm install redis bitnami/redis
# Nginx Ingress Controller
helm install nginx-ingress ingress-nginx/ingress-nginx
# Prometheus + Grafana (monitoring)
helm install monitoring prometheus-community/kube-prometheus-stackLocally
A full cluster is heavy on a laptop. kind, minikube, k3d.
| Criterion | kind | minikube | k3d |
|---|---|---|---|
| Foundation | Docker containers | VM (VirtualBox/Docker) | k3s in Docker |
| Startup speed | Very fast (10-20 sec) | Slow (1-2 min) | Fast (20-30 sec) |
| RAM | Low (~2GB) | High (~4-8GB) | Low (~1-2GB) |
| Multi-node cluster | Yes | Yes (harder) | Yes |
| LoadBalancer support | Via MetalLB | Out of box | Out of box |
| Closeness to prod | Very close | Close | k3s is not full k8s |
| Best for | CI/CD, testing | Learning, dev | Fast daily dev |
kind for CI and closeness to prod k8s. minikube for learning if RAM is cheap. k3d every day.
brew install kind kubectl
# Or via binary
curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.20.0/kind-darwin-arm64
chmod +x ./kind
mv ./kind /usr/local/bin/kind# Basic single-node cluster
kind create cluster --name dev
# Multi-node cluster (1 control-plane + 2 workers)
cat <<EOF | kind create cluster --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
EOFkubectl cluster-info
kubectl get nodes# Load local image into kind
kind load docker-image myapp:latest --name dev
# Deploy
kubectl apply -f deployment.yaml
kubectl get pods
kubectl logs -f <pod-name>
# Port forward for access
kubectl port-forward deployment/myapp 8080:8080
# Now available at http://localhost:8080kind delete cluster --name devbrew install k3d
# Or curl
curl -s https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash# Cluster with 3 worker nodes and LoadBalancer on port 8080
k3d cluster create dev \
--agents 3 \
--port "8080:80@loadbalancer"kubectl apply -f deployment.yaml
kubectl apply -f service.yaml
# If Service type: LoadBalancer, available at localhost:8080
curl http://localhost:8080k3d cluster stop dev
k3d cluster start devMinimum 8 GB RAM and 4 cores, better 16 and 6. I cut Docker Desktop to 6 GB and 4 CPUs. Limits in the manifests. I do not start the frontend if I am testing the API. After tests kubectl delete: a stopped pod still eats RAM.
If the laptop is choking, a $10-20 cloud dev cluster is cheaper than a day of reboots.
How to see what broke
# Get pod list
kubectl get pods
# Detailed pod info
kubectl describe pod <pod-name>
# Pod logs
kubectl logs <pod-name>
kubectl logs <pod-name> -f # follow (real-time)
kubectl logs <pod-name> --previous # logs of previous crashed container
# Logs from all deployment pods
kubectl logs -l app=myapp --all-containers=true
# Exec into pod (like docker exec)
kubectl exec -it <pod-name> -- /bin/bash
# Port forward
kubectl port-forward pod/<pod-name> 8080:8080
# Cluster events
kubectl get events --sort-by='.lastTimestamp'
# Top (CPU/Memory usage)
kubectl top nodes
kubectl top podsPending: describe, usually not enough CPU/memory or nodes. CrashLoopBackOff: logs --previous. Running and silent: readiness. Empty Service: kubectl get endpoints, pods are not ready.
# Add repository
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
# Install kube-prometheus-stack (Prometheus + Grafana + Alertmanager)
helm install monitoring prometheus-community/kube-prometheus-stack
# Get Grafana password
kubectl get secret monitoring-grafana -o jsonpath="{.data.admin-password}" | base64 --decode
# Port forward for access
kubectl port-forward svc/monitoring-grafana 3000:80
# Grafana available at http://localhost:3000 (admin / <password>)helm repo add grafana https://grafana.github.io/helm-charts
helm install loki grafana/loki-stack \
--set grafana.enabled=false \
--set promtail.enabled=true
# Add Loki as data source in Grafana
# URL: http://loki:3100In Explore: {app="myapp"} and {app="myapp"} |= "ERROR".
If not Kubernetes
Nomad: one binary, 50 MB RAM, Docker and not only Docker, Consul and Vault next door. Smaller ecosystem, no Ingress, no managed offer from the clouds. Makes sense for a team under ten on the HashiCorp stack.
job "myapp" {
datacenters = ["dc1"]
group "app" {
count = 3
task "web" {
driver = "docker"
config {
image = "myapp:v1.0.0"
ports = ["http"]
}
resources {
cpu = 500
memory = 512
}
service {
name = "myapp"
port = "http"
check {
type = "http"
path = "/health"
interval = "10s"
timeout = "2s"
}
}
}
}
}ECS: if you are already all-in on AWS. Fargate without servers. Lock-in and price, especially Fargate.
Fly.io: fly deploy, Heroku with Docker, points around the world, three free VMs. Less control, the bill grows with load.
# Install CLI
brew install flyctl
# Login
fly auth login
# Initialize app
fly launch
# Deploy
fly deploy
# Scaling
fly scale count 5
fly scale vm shared-cpu-2xDocker Swarm: if you know Compose you will learn it in a day. docker stack deploy -c docker-compose.yml. Docker Inc barely develops it.
# Initialize Swarm
docker swarm init
# Deploy compose file
docker stack deploy -c docker-compose.yml myapp
# Scaling
docker service scale myapp_web=5| Criterion | Kubernetes | Nomad | AWS ECS | Fly.io | Docker Swarm |
|---|---|---|---|---|---|
| Complexity | High | Medium | Low | Very low | Low |
| Ecosystem | Huge | Medium | AWS-only | Small | Dead |
| Flexibility | Maximum | High | Medium | Low | Medium |
| Vendor lock-in | No | No | AWS | Fly.io | No |
| Cost | DIY cheap, managed expensive | DIY cheap | Expensive | Grows | Cheap |
| Best for | Medium/large teams | Small teams, HashiCorp stack | AWS-native projects | Startups | Small projects |
Compose is enough: I stay. Outgrown: Fly or Nomad first. Kubernetes when I need its flexibility, not a line on a CV.
Cost of the move
To deploy: 20-40 hours. To run production: 100-200. Architect: 500+.
Managed EKS/GKE/AKS: control plane $70-150, nodes $50-200, addons yours. Self-hosted: software free, the team is not.
Five microservices, 20-30 pods, about 2000 RPS. GKE $250, 10-15 hours of DevOps a month, two weeks of onboarding. It pays if you save on hand scaling and downtime.
I do not go to k8s if there are fewer than three developers, a monolith on one server, no DevOps, this is an MVP, or the budget is under $500. Managed is expensive, your own cluster without expertise is dangerous.
How I moved
A week or two: the kubernetes.io tutorial, local kind, an audit of services and secrets, managed or own, a registry in CI.
Then the simplest stateless service.
api:
image: myapi:latest
ports:
- "8080:8080"
environment:
DATABASE_URL: postgresql://db:5432/mydbapiVersion: apps/v1
kind: Deployment
metadata:
name: api
spec:
replicas: 3
selector:
matchLabels:
app: api
template:
metadata:
labels:
app: api
spec:
containers:
- name: api
image: myapi:v1.0.0
ports:
- containerPort: 8080
env:
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: api-secrets
key: database-url
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 500m
memory: 512Mi
livenessProbe:
httpGet:
path: /health
port: 8080
readinessProbe:
httpGet:
path: /health/ready
port: 8080
---
apiVersion: v1
kind: Service
metadata:
name: api
spec:
selector:
app: api
ports:
- port: 80
targetPort: 8080apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: main-ingress
annotations:
cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
ingressClassName: nginx
tls:
- hosts:
- api.example.com
secretName: api-tls
rules:
- host: api.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: api
port:
number: 80I do not rush databases into k8s. RDS and Cloud SQL are simpler. If you still drag them: StatefulSet, PVC, backup, a restore drill. Redis can be a Bitnami chart. Files: S3 or a ReadWriteMany PVC.
Then kube-prometheus-stack and Loki. Traffic 10%, a week of metrics, 50%, 100%. Old Compose lives another two weeks.
I moved in 2021. The first two months were pain and rakes. Then infrastructure that scales without me, stands itself up, and gives ten deploys a day without a window. That is worth the time if the job is hard enough. If not, Compose and sleep are calmer.
See also:
