Load testing as a financial audit

A colleague ran $15k a day in ads. Production lay down for half a day. About $7.5k gone on a flat stretch of road. Since then I do not run load for graphs. I look for the point where the system starts eating revenue.

Marketplaces on Black Friday and 11.11, government portals on the first day of QR passes: they tested. They tested the wrong thing the wrong way. A local curl returns 50 ms. At 1000 RPS the same endpoints take seconds and return 500s. Nine out of ten of those things show up only under load.

A one-second delay at 1000 RPS: every real second, users wait a thousand seconds in total. Those are abandoned carts, not an "awkward UX".

Two questions, not four test types

When I start a run I keep two questions.

How fast is the system at the target load? That is p95, latency, throughput.

How and when does it break, how does it degrade, and what does that cost? That is the limit in money.

The names after that are clerical. Load: how many we can serve before we start losing money. Stress: where the floor burns through. Spike: a Hacker News hit, at which RPS we fall. Soak: what an hour of peak costs if memory leaks.

Why k6

The tool is secondary. The report has to answer "how much we lose on this architecture", not "what a pretty graph".

k6 lives in the repo next to the product, gets reviewed, runs in CI. Gatling and JMeter are fine if they meet the same bar: test as code, one pipeline step, thresholds, a report someone other than the author can read.

JMeter was born for a separate QA team and clicks in a GUI. k6 was born where the developer and the engineer are the same person.

If not k6: JMeter when the team does not write JS. Gatling when the team is JVM and scenarios have state. Locust if you already live in Python. Artillery if YAML is enough and you need WebSocket.

The order without which a graph is useless

First I price 1% errors and one second of latency on the key flow. Then I pick a tool. Then two or three money flows and hypotheses about where it will break. Then a profile with a gradual ramp and SLA thresholds. Then test metrics against the metal. Then a report: hypothesis, data, action, the same run again.

A taxi service, flow "search → order → pay". Analytics: p95 of car search above 3 s, conversion drops 15%. Peak 1000 RPS: 150 lost requests a second. Average order $20, 10% become an order. About $300 of revenue a minute. The test said the geocoder cracks at 800 RPS. An hour of peak before the fix: about $2400.

Before the run the team writes where the system will break first. "The DB will hit IOPS at 300 RPS." "The cache dies at 500 VU." "The payment gateway starts timing out." The test buries those sentences or confirms them.

In the first run I touch two or three flows that make most of the revenue or the peak. Auth and dashboard. Search, cart, checkout. Create a record and list it.

import http from "k6/http";
import { group, check } from "k6";
 
export default function () {
  group("User Journey: Login → Dashboard → Logout", () => {
    // 1. Login
    const loginRes = http.post("https://api.example.com/auth/login", {
      email: "test@example.com",
      password: "password123",
    });
 
    check(loginRes, { "login success": (r) => r.status === 200 });
    const token = loginRes.json("token");
 
    // 2. Get dashboard
    const headers = { Authorization: `Bearer ${token}` };
    const dashRes = http.get("https://api.example.com/dashboard", { headers });
 
    check(dashRes, { "dashboard loaded": (r) => r.status === 200 });
 
    // 3. Logout
    http.post("https://api.example.com/auth/logout", null, { headers });
  });
}

Ten thousand VU at once give fake failures. I ramp in steps.

export const options = {
  stages: [
    { duration: "1m", target: 50 }, // Warmup
    { duration: "3m", target: 50 }, // Baseline
    { duration: "2m", target: 150 }, // Growth (peak hours)
    { duration: "5m", target: 150 }, // Peak load
    { duration: "2m", target: 300 }, // Stress test
    { duration: "3m", target: 300 }, // Hold stress
    { duration: "2m", target: 0 }, // Gradual ramp-down
  ],
};

Thresholds turn a run into an SLA check.

export const options = {
  thresholds: {
    // Latency
    http_req_duration: [
      "p(50)<200", // Median < 200ms
      "p(95)<500", // 95th percentile < 500ms
      "p(99)<1000", // 99th percentile < 1s
    ],
 
    // Availability
    http_req_failed: ["rate<0.01"], // < 1% errors
 
    // Throughput
    http_reqs: ["rate>100"], // Minimum 100 RPS
 
    // Custom checks
    "checks{type:auth}": ["rate>0.99"], // 99% successful logins
  },
};

I run in CI so the conditions match. Locally only for debugging.

After the run I look at p50/p95/p99, errors by code, RPS, how many VU we held. Next to that: CPU and memory, database queries, cache hit, network.

Numbers that decide

One Grafana screen: latency versus throughput, the latency / error rate / throughput triangle, CPU, IO and network saturation.

p95: the SLA for the main mass. p99: conscience. If p95 is 200 ms and p99 is 2000 ms, there is a core and a tail. The tail is usually slow queries, locks, GC.

1% errors at 1000 RPS: 10 failed requests a second. Ten minutes of test: 6000 errors. That is an incident, not noise.

RPS alone says nothing. I watch latency as RPS grows. If it spikes, we found the bottleneck ceiling.

RPS at 90-100% CPU or IOPS: the metal will crack, not the code.

Apdex without Satisfied / Tolerating / Frustrated thresholds is an empty number. For trading, Satisfied is closer to 50 ms, for a CMS to 500 ms. Formula: (Satisfied + Tolerating/2) / Total.

CPU above 80% for a long time: bottleneck. Memory grows linearly: a leak. Connection pool empty: tune it. Low cache hit: the cache strategy is lying.

How I find the root:

• High p99 and CPU/IO spikes on the DB: slow query log, EXPLAIN ANALYZE, pt-query-digest.
• 5xx with no CPU growth: connection limits, pools, external API timeouts.
• Throughput stuck, CPU low: global locks, network, rate limit.
• RPS grows, latency jumps, cache hit drops: misses and TTL.

A report that becomes a backlog

# Load Test: API v2.0
 
## Goal
 
Verify API readiness for 3x load increase (from 500 to 1500 RPS).
 
## Scenario
 
- User Journey: Login → Get Dashboard → Logout
- Load profile: 50 → 150 → 300 VU
- Duration: 18 minutes
 
## Results
 
### Passed Thresholds
 
- p95 latency: 420ms (threshold: < 500ms)
- Error rate: 0.3% (threshold: < 1%)
- Throughput: 1200 RPS (expected: 1000 RPS)
 
### Issues
 
- p99 latency: 3.2s (threshold: < 1s)
- CPU on DB: 92% at peak
- Slow queries: `/users` endpoint → N+1 queries
 
## Bottlenecks (by priority)
 
1. **Database N+1 queries:** `/users` makes 50+ DB queries
   - **Action:** add eager loading
   - **ETA:** 2 days
   - **Effect:** p99 latency → < 800ms
 
2. **CPU on DB:** reaches 92% at 300 VU
   - **Action:** upgrade instance (4 → 8 vCPU)
   - **ETA:** 1 day
   - **Effect:** headroom to 500 VU
 
3. **Cache hit rate:** only 65% for `/dashboard`
   - **Action:** increase cache TTL from 5min to 15min
   - **ETA:** 1 day
   - **Effect:** reduce DB load by 20%
 
## Recommendations
 
- Optimize N+1 queries **critical before release**
- Upgrade DB instance **recommended**
- Tune cache **can be postponed**
 
## Hypothesis and Verification (mandatory for each issue)
 
- **Problem:** p99 latency: 3.2s on `/users`
- **Hypothesis:** DB logs show N+1 queries on this endpoint
- **Action:** add eager loading
- **Verification:** rerun same test and confirm p99 < 800ms
 
## Graphs
 
[Attach Grafana dashboard screenshot]
 
## Next Steps
 
- [ ] Fix N+1 queries
- [ ] Rerun test after optimization
- [ ] Stress test to find limit (500+ VU)

Numbers without a next action are a museum.

A typical useless report: "we hit 5000 RPS, p95 2.3 s, optimize the DB". Unclear whether we can live at 5000 RPS (at that p95, no), what to fix, or whether it is worth the money. The answer needed: at which RPS we stay inside p95, how much we lose above the line, and which component breaks first.

Pitfalls

Staging weaker than prod, synthetic data, external APIs stubbed. The results are water. The environment has to be close to prod, the data representative, dependencies either stood up or real prod with the owner's consent.

You hit Cloudflare or Nginx, not the code. Raise or drop limits for the runner IP.

First requests are slow because of a cold cache and JIT. Warm up:

export const options = {
  stages: [
    { duration: "1m", target: 10 }, // Warm-up
    { duration: "5m", target: 100 }, // Main test
  ],
};

k6 is green, the server still dies: no CPU, memory, disk, network next to the test metrics.

One two-hour run: unclear in which minute it broke. Better micro-tests: baseline 50 VU 5 min, peak 150 VU 10 min, stress 300 VU 5 min.

CI

load-test:
  stage: test
  image: grafana/k6:latest
  script:
    - k6 run --out json=results.json tests/load/api.js
  artifacts:
    reports:
      junit: results.json
  only:
    - main
  when: manual # Run manually before release

name: Load Test
 
on:
  workflow_dispatch: # Manual trigger
 
jobs:
  load-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
 
      - name: Run k6 test
        uses: grafana/k6-action@v0.3.1
        with:
          filename: tests/load/api.js
          cloud: true
          token: ${{ secrets.K6_CLOUD_TOKEN }}

Money

A day or two of an engineer for the test. Hours of peak downtime cost an order of magnitude more. Incident formula:

(Average order × Conversion × Affected traffic share × Outage duration) + (Support cost × Ticket count) + (Reputation cost × Loss coefficient)

The test plugs in live numbers: at 300 RPS and 10% checkout errors we lose X a minute.

Critical before a sale, a content launch, mass payouts, a press mention. Same questions: what error rate we can stand in absolute requests, at which p95 conversion drops, whether it is cheaper to raise the architecture or budget 0.5% loss.

The first serious run almost always finds two or three bottlenecks nobody knew about. Better that way than from mail after the peak.

See also:

• Proxmox: Home Data Center
• Self-host Supabase