Envoy Gateway: 15 lines instead of 47 annotations

Summer 2024. Friday, 6:30 p.m. The production canary is broken.

The job is simple: 10% of traffic to the new API. Nginx Ingress, the example from the docs:

# What I thought would work
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-canary
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "10"
spec:
  rules:
    - host: api.company.com
      http:
        paths:
          - path: /v2
            backend:
              service:
                name: api-v2
                port:
                  number: 8080

Applied it. One hundred percent to the new version. Changed weight to 90. Still one hundred.

Four hours: the docs seven times, search "nginx ingress canary not working", controller 0.21 instead of the required 0.22+, the upgrade broke two more Ingress resources, a separate resource with the same host, the path was regex not prefix.

At 11 p.m. it worked. 47 lines of YAML and 12 annotations. I did not understand half of it.

Monday: "let's add JWT to that endpoint".

The same canary a month later

I tried Envoy Gateway on a dev cluster, skeptically.

# What works on the first try
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-route
spec:
  parentRefs:
    - name: my-gateway
  hostnames:
    - api.company.com
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /v2
      backendRefs:
        - name: api-v1 # 90% traffic
          port: 8080
          weight: 90
        - name: api-v2 # 10% traffic (canary)
          port: 8080
          weight: 10

15 lines. Zero annotations. First try.

A week later they asked for JWT. With Nginx that is more annotations, an external auth service and an evening on tokens. Here:

# Twelve more lines, JWT works
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: jwt-policy
spec:
  targetRef:
    kind: HTTPRoute
    name: api-route
  jwt:
    providers:
      - name: auth0
        issuer: https://company.auth0.com/
        audiences:
          - api://myapp
        remoteJWKS:
          uri: https://company.auth0.com/.well-known/jwks.json

Applied. Worked. I have not gone back to Nginx Ingress annotations.

What it is made of

Envoy Proxy from Lyft. Istio runs on it. HTTP/1.1, HTTP/2, HTTP/3, gRPC, WebSocket. Least request and consistent hashing. Circuit breaking, retry, timeout without plugins. Rate limit and JWT native. Metrics to Prometheus, traces to Jaeger.

Gateway API: the official Kubernetes API for ingress traffic. Ingress is older and lives on strings in annotations. Each controller has a dialect. Moving means rewriting those strings. Here the fields are typed, the error shows before apply, changing the implementation is changing gatewayClassName.

The control plane reads Gateway and HTTPRoute, translates to xDS, runs Envoy pods, reloads config without a restart, talks to cert-manager, external-dns, Prometheus.

Envoy ProxyKubernetesGateway APIcert-manager

Метрика

Nginx Ingress

Envoy Gateway

Lines of YAML

47 lines

15 lines

68%

Annotations

12 magic strings

100%

Setup time

6 hours + Google

10 minutes

67%

JWT and a limit without Redis

A hundred requests a minute per user. In Nginx the limit often wants Redis and Lua. Here two manifests.

# JWT: 12 lines
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: jwt-policy
spec:
  targetRef:
    kind: HTTPRoute
    name: api-route
  jwt:
    providers:
      - name: auth0
        issuer: https://mycompany.auth0.com/
        audiences:
          - api://myapp
        remoteJWKS:
          uri: https://mycompany.auth0.com/.well-known/jwks.json
 
---
# Rate Limiting: 15 lines
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
  name: rate-limit
spec:
  targetRef:
    kind: HTTPRoute
    name: api-route
  rateLimit:
    type: Local
    local:
      rules:
        - clientSelectors:
            - headers:
                - name: x-user-id
                  type: Distinct
          limit:
            requests: 100
            unit: Minute

JWKS caches itself. Counters live in memory. Metrics show how many were cut, how many passed.

gRPC

Nginx can do gRPC. Health checks and per-method metrics you build yourself. Envoy was born at Lyft around gRPC.

apiVersion: gateway.networking.k8s.io/v1
kind: GRPCRoute
metadata:
  name: user-service
spec:
  parentRefs:
    - name: my-gateway
  hostnames:
    - grpc.example.com
  rules:
    - matches:
        - method:
            service: myapp.v1.UserService
            method: GetUser
      backendRefs:
        - name: user-service
          port: 9090

Latency of GetUser and errors of CreateUser show separately. Health check, retry, circuit breaking sit next to them.

Nginx added gRPC later. For Envoy this is home ground.

Do not take Envoy Gateway if you have one or two services, the cluster is older than 1.25, the team is not ready to rewrite Ingress, you need mTLS between every service (that is Istio), or you want one Ingress for the whole cluster and to forget.

Take it if you need canary, JWT, limits, portable manifests and metrics out of the box.

How it is wired

┌─────────────────────────────────────────────────────┐
│  Kubernetes Cluster                                 │
├─────────────────────────────────────────────────┤
│                                                     │
│  ┌─────────────────────────────────┐                │
│  │ Control Plane (envoy-gateway)   │                │
│  │                                 │                │
│  │  ┌──────────────────────────┐   │                │
│  │  │ Gateway API Controller   │   │                │
│  │  │ (watches Gateway, Route) │   │                │
│  │  └────────┬─────────────────┘   │                │
│  │           │                     │                │
│  │           ▼                     │                │
│  │  ┌──────────────────────────┐   │                │
│  │  │ xDS Translator           │   │                │
│  │  │ (Gateway API → Envoy cfg)│   │                │
│  │  └────────┬─────────────────┘   │                │
│  │           │ xDS (gRPC)          │                │
│  └───────────┼─────────────────────┘                │
│              │                                      │
│              ▼                                      │
│  ┌─────────────────────────────────┐                │
│  │ Data Plane (envoy-proxy pods)   │                │
│  │                                 │                │
│  │  ┌────────┐  ┌────────┐         │                │
│  │  │ Envoy  │  │ Envoy  │  ...    │                │
│  │  │ Pod 1  │  │ Pod 2  │         │                │
│  │  └───┬────┘  └───┬────┘         │                │
│  │      │           │              │                │
│  └──────┼───────────┼──────────────┘                │
│         │           │                               │
│         │  Ingress Traffic (HTTP/gRPC)              │
│         ▼           ▼                               │
│  ┌─────────────────────────────────┐                │
│  │ Backend Services (Pods)         │                │
│  └─────────────────────────────────┘                │
│                                                     │
└─────────────────────────────────────────────────────┘

One control-plane pod in envoy-gateway-system. N Envoy pods take traffic. The data plane scales on its own. Config is hot. xDS is shared: you can change the controller.

Nginx Ingress reads YAML itself. Here the control plane reads YAML and tells the proxy over gRPC what to do.

From zero to canary

You need Kubernetes 1.25+ and kubectl.

# Gateway API CRDs
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/standard-install.yaml
 
# Envoy Gateway
kubectl apply -f https://github.com/envoyproxy/gateway/releases/download/latest/install.yaml
 
# Check
kubectl get pods -n envoy-gateway-system
 
# Should see pod: envoy-gateway-xxxxx (STATUS: Running)

After install a GatewayClass named envoy-gateway appears.

# Create echo-app.yaml
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Namespace
metadata:
  name: demo
 
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo-v1
  namespace: demo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: echo
      version: v1
  template:
    metadata:
      labels:
        app: echo
        version: v1
    spec:
      containers:
        - name: echo
          image: hashicorp/http-echo:latest
          args: ["-text=Hello from v1"]
          ports:
            - containerPort: 5678
 
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo-v2
  namespace: demo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: echo
      version: v2
  template:
    metadata:
      labels:
        app: echo
        version: v2
    spec:
      containers:
        - name: echo
          image: hashicorp/http-echo:latest
          args: ["-text=Hello from v2 (canary!)"]
          ports:
            - containerPort: 5678
 
---
apiVersion: v1
kind: Service
metadata:
  name: echo-v1
  namespace: demo
spec:
  selector:
    app: echo
    version: v1
  ports:
    - port: 80
      targetPort: 5678
 
---
apiVersion: v1
kind: Service
metadata:
  name: echo-v2
  namespace: demo
spec:
  selector:
    app: echo
    version: v2
  ports:
    - port: 80
      targetPort: 5678
EOF
 
# Check
kubectl get pods -n demo
# Should see 4 pods: echo-v1-xxx (2), echo-v2-xxx (2)

cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: demo-gateway
  namespace: demo
spec:
  gatewayClassName: envoy-gateway
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: Same
EOF
 
# Wait for Gateway to be ready
kubectl wait --for=condition=Programmed gateway/demo-gateway -n demo --timeout=300s
 
# Get IP/Hostname
export GATEWAY_IP=$(kubectl get gateway demo-gateway -n demo -o jsonpath='{.status.addresses[0].value}')
echo "Gateway IP: $GATEWAY_IP"

On GKE, EKS, AKS the Gateway gets an external IP. On minikube and kind you need kubectl port-forward.

cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: echo-route
  namespace: demo
spec:
  parentRefs:
    - name: demo-gateway
  hostnames:
    - echo.example.com
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: echo-v1
          port: 80
          weight: 90  # 90% traffic to v1
        - name: echo-v2
          port: 80
          weight: 10  # 10% traffic to v2 (canary)
EOF
 
# Check status
kubectl get httproute -n demo
# STATUS: Accepted

# If you have external IP:
for i in {1..20}; do
  curl -H "Host: echo.example.com" http://$GATEWAY_IP/
done
 
# If using port-forward (local cluster):
kubectl port-forward -n demo svc/demo-gateway-envoy-gateway 8080:80 &
for i in {1..20}; do
  curl -H "Host: echo.example.com" http://localhost:8080/
done
 
# Result (approximately):
# Hello from v1 (18 out of 20 ≈ 90%)
# Hello from v2 (canary!) (2 out of 20 ≈ 10%)

A limit of five requests a minute on x-user-id:

cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
  name: rate-limit
  namespace: demo
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: echo-route
  rateLimit:
    type: Local
    local:
      rules:
        - clientSelectors:
            - headers:
                - name: x-user-id
                  type: Distinct
          limit:
            requests: 5
            unit: Minute
EOF
 
# Test (make 10 requests with same user_id)
for i in {1..10}; do
  curl -H "Host: echo.example.com" -H "x-user-id: user123" http://$GATEWAY_IP/
done
 
# First 5 requests: HTTP 200 OK
# Next 5: HTTP 429 Too Many Requests

No Redis. Counters in memory, synced between replicas. In production, several Gateways can switch to Global plus Redis.

Traps

Gateway API is alive. Gateway, HTTPRoute, GRPCRoute are stable as v1. SecurityPolicy and BackendTrafficPolicy are experimental, check the matrix. Pin the Envoy Gateway version in production.

Certificates by hand every 90 days are not needed. cert-manager:

# Install cert-manager
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.0/cert-manager.yaml
 
# ClusterIssuer for Let's Encrypt
cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: admin@example.com
    privateKeySecretRef:
      name: letsencrypt-prod-key
    solvers:
      - http01:
          gatewayHTTPRoute:
            parentRefs:
              - name: demo-gateway
                namespace: demo
EOF
 
# Gateway with TLS
cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: demo-gateway
  namespace: demo
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  gatewayClassName: envoy-gateway
  listeners:
    - name: https
      protocol: HTTPS
      port: 443
      hostname: "*.example.com"
      tls:
        mode: Terminate
        certificateRefs:
          - name: example-com-tls
EOF

Envoy exports metrics itself: envoy_http_downstream_rq_total, envoy_http_downstream_rq_xx, envoy_http_downstream_rq_time, envoy_cluster_upstream_rq_retry. Dashboard: Envoy Gateway Overview.

In production at least two Envoy replicas, CPU and memory limits, TLS, a request limit, circuit breaking, preStop, manifests in Git.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: production-gateway
  namespace: production
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  gatewayClassName: envoy-gateway
  listeners:
    # HTTP → HTTPS redirect
    - name: http
      protocol: HTTP
      port: 80
      hostname: "*.example.com"
 
    # HTTPS with automatic certificates
    - name: https
      protocol: HTTPS
      port: 443
      hostname: "*.example.com"
      tls:
        mode: Terminate
        certificateRefs:
          - name: wildcard-tls
 
---
# Automatic redirect to HTTPS
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: https-redirect
  namespace: production
spec:
  parentRefs:
    - name: production-gateway
      sectionName: http
  hostnames:
    - "*.example.com"
  rules:
    - filters:
        - type: RequestRedirect
          requestRedirect:
            scheme: https
            statusCode: 301

Moving from Nginx

I do not move thirty Ingress resources in one day. Envoy next to it, a separate LoadBalancer, one non-critical service, a week of metrics. Then DNS 10/90, 25, 50, 75, 100. Rollback: switch DNS. Nginx stays another two weeks.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/rate-limit: "100"
spec:
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /api(/|$)(.*)
            backend:
              service:
                name: api-service
                port:
                  number: 8080

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-route
spec:
  parentRefs:
    - name: production-gateway
  hostnames:
    - api.example.com
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /api
      filters:
        - type: URLRewrite
          urlRewrite:
            path:
              type: ReplacePrefixMatch
              replacePrefixMatch: /
      backendRefs:
        - name: api-service
          port: 8080
 
---
# Rate limiting via separate policy
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
  name: api-rate-limit
spec:
  targetRef:
    kind: HTTPRoute
    name: api-route
  rateLimit:
    type: Local
    local:
      rules:
        - limit:
            requests: 100
            unit: Minute

Annotations became fields. Regex became PathPrefix. The limit moved into a policy you can hang on another route.

I am not learning "Nginx Ingress annotations". I am learning Gateway API. The same knowledge works with Istio Gateway, Cilium, Kong.

A local cluster and fifteen minutes on the example above is usually enough to decide whether to drag this into production.

See also:

Envoy Gateway: 15 lines instead of 47 annotations

Table of Contents

The same canary a month later

What it is made of

JWT and a limit without Redis

gRPC

How it is wired

From zero to canary

Traps

Moving from Nginx

Related posts

Docker Compose and Kubernetes: when to move

A load balancer after Docker

Husky in a working project