How Things Work

Performance & Load Testing

Measuring how a system behaves under load — virtual users, throughput vs latency, finding the breaking point, percentiles over averages, and enforcing budgets with k6/JMeter in CI.

How It Works

Performance testing measures how a system behaves under concurrent load — its throughput, latency, and error rate as virtual users ramp up. Load, stress, soak, and spike tests each probe a different limit. The aim is to find the breaking point and bottlenecks before users do, track percentile latency (not averages), and enforce performance budgets as CI thresholds so regressions fail the build.

Simulate Real Load

Performance tests generate concurrent virtual users (VUs) hitting your system and measure how it responds: throughput (requests/sec), latency (response times), and error rate. The goal is to understand behavior under load before real users find the limits for you.

Test Types

Load testing checks behavior at expected peak. Stress testing pushes past it to find the breaking point. Soak (endurance) testing runs moderate load for hours to surface memory leaks and resource exhaustion. Spike testing slams sudden surges. Each answers a different reliability question.

Throughput, Latency & the Knee

Below capacity, adding users increases throughput while latency stays flat. At the 'knee' of the curve the system saturates: throughput plateaus, latency climbs steeply, and errors appear as queues back up. Finding that point tells you the real capacity and where to set safe operating limits.

Percentiles, Baselines & CI

Always measure percentiles (p95/p99), not averages — averages hide the slow tail that users actually feel. Establish a baseline, set thresholds/SLOs (e.g. p95 < 300ms, errors < 1%), and run tests like k6 or JMeter in CI so a performance regression fails the build, just like a functional one.

Key Concepts

👥Virtual Users (VUs)

Simulated concurrent clients generating load against the system under test.

📈Throughput

Requests served per second. Scales with load until the system saturates, then plateaus.

📊Latency Percentiles

p95/p99 response times — the tail users feel. Far more meaningful than the average.

📐The Knee

The load level where the system saturates: throughput flattens, latency spikes, errors rise.

🧪Test Types

Load (peak), stress (past peak), soak (endurance), spike (sudden surge) — each probes a different limit.

🎯Baseline & SLO

A reference measurement plus targets (p95 budget, error rate) enforced as CI thresholds.

k6 load test with thresholds

tsx

1// k6 load test — ramp virtual users and enforce thresholds in CI.
2import http from "k6/http";
3import { check, sleep } from "k6";
4
5export const options = {
6  stages: [
7    { duration: "1m", target: 50 },    // ramp up
8    { duration: "3m", target: 200 },   // sustained load (past capacity?)
9    { duration: "1m", target: 0 },     // ramp down
10  ],
11  thresholds: {
12    http_req_duration: ["p(95)<300"],  // p95 latency budget → fails build if breached
13    http_req_failed:   ["rate<0.01"],  // <1% errors
14  },
15};
16
17export default function () {
18  const res = http.get("https://api.example.com/products");
19  check(res, { "status is 200": (r) => r.status === 200 });
20  sleep(1);
21}
22// Measure PERCENTILES (p95/p99), not averages — averages hide the slow tail.

💡

Why This Matters

Performance and scalability problems cause outages and lost users, and they're invisible to functional tests. Knowing how to load test, read throughput/latency curves, reason in percentiles, and set performance SLOs is core to building systems that survive real traffic — and a frequent topic in system-design and SRE interviews.

Common Pitfalls

⚠Reporting averages instead of p95/p99 — averages hide the slow tail that users actually experience.

⚠Load testing against an unrealistic environment (tiny data, no caching cold-start, single instance) — misleading results.

⚠Only testing peak load, never soak — missing memory leaks and resource exhaustion that appear over time.

⚠No baseline or CI thresholds, so performance silently regresses until it's a production incident.

⚠Ignoring the test client/network as the bottleneck — measuring your load generator, not the system.

Real-World Use Cases

1Discovering Capacity Before the Big Launch

Scenario

A team is about to launch a marketing campaign expecting 10× normal traffic. Nobody knows whether the system can handle it, and finding out in production would be catastrophic.

Problem

Without load testing, capacity is a guess. A surge could saturate the system, spike latency, and cause an outage at the worst possible moment — during the campaign that was supposed to drive growth.

Solution

Run a load/stress test that ramps VUs to and beyond the expected peak, identifying the knee (e.g. ~150 VUs) and the bottleneck (a database connection pool). They raise capacity, add caching, re-test to confirm headroom, and set autoscaling thresholds below the breaking point.

💡

Takeaway: Load testing turns capacity from a guess into a known number. Find the breaking point and bottleneck in a controlled test, fix it, and launch with verified headroom instead of hope.

2The Latency Regression Caught in CI

Scenario

A seemingly harmless code change (an extra DB query in a hot endpoint) doubled p95 latency. Functional tests passed, so it nearly shipped — until the performance gate flagged it.

Problem

Functional tests verify correctness, not speed. Performance regressions sneak through normal testing and only surface as a gradual, hard-to-trace slowdown in production.

Solution

Add a k6 performance test to CI with a threshold (p95 < 300ms). The change breached the budget and failed the build, pointing straight at the regression. The team fixed the N+1 query before merge, keeping latency within SLO.

💡

Takeaway: Treat performance like correctness: codify latency/error budgets as CI thresholds. Automated performance tests catch regressions at the PR stage, before users ever feel the slowdown.

Join the discussion

Comments

Loading comments...

Cookie preferences

We use essential cookies for sign-in and preferences. With your permission, we also use basic analytics to improve lessons.