Stress testing · Crust docs

Goal: put controlled pressure on a service — a burst, a paced scenario, a long soak, or a fixture glob at volume — and read the latency distribution out of honest numbers.

The one-liner (fixed count)

For a plain GET hammer, skip fixtures entirely — the load stages are shell-native:

range(0, 999) | parallel 50 | GET :3000/health | expect 200 | stats
# {"count":1000,"wallMs":2188,"rps":457,"status":{"200":1000},"p50":53,"p95":66.2,"p99":126.7,"meanMs":53.6}

parallel N sets the fan-out for the next http stage, GET fires one timed request per upstream item, expect NNN fails the pipeline (exit 1) naming the mismatch count, and stats yields one summary with real latency percentiles, rps and a status histogram. range answers “how fast can we serve N requests” — as fast as possible, no pacing. When the question is “how do we behave at this rate for this long”, use load.

parallel streams results in completion order, not input order. That’s deliberate: downstream stages see each result the moment it settles, which is what makes windowed soak stats meaningful — a barrier variant would dump everything into the final window. Sort downstream if you need input order.

Scenarios — duration & rate (`load`)

load <dur> <rate> is a paced source: one tick per scheduled slot, for a wall-clock duration. Durations in ms/s/m, rates as N/s or N/m (decimals allowed). The minimal scenario is one phase:

load 30s 100/s | parallel 50 | GET :3000/health | stats

Slots are anchored to absolute times, so pacing doesn’t drift, and each phase ends on its wall-clock regardless of how many ticks got out.

Ramps are comma-separated phases in one stream — warm up gently, then push:

load 10s 50/s, 30s 200/s | parallel 50 | GET :3000/health | stats

Write-path load works because ticks are real objects — {n, phase, scheduledAt, lagMs} — so a body-factory lambda between load and the verb has material to build unique payloads from:

load 10s 20/s | (t => ({name: "load-" + t.n, kind: "widget"})) \
  | parallel 8 | POST :3000/api/things -H "authorization: Bearer $TOKEN" \
  | expect 201 | stats

The parallel modifier puts any verb in load mode: output becomes {status, ms, url} timing records, bodies are drained, and a network error yields a status: 0 record instead of killing the run. (parallel 1 | POST … is the explicit opt-in for serial-but-timed. Putting parallel before anything that isn’t an http verb, lambda, or registered fn — or leaving it trailing — is a loud parse error, not a silent no-op.)

Reading the drop report

When downstream can’t keep up — the parallel pool is saturated — stale slots are skipped, never burst. Crust counts them and reports the shortfall on stderr when the stream drains:

load: target 150 ticks — emitted 40, dropped 110 (downstream saturated; raise parallel N?), achieved 13.2/s

That line is the difference between a load tool and a flattery tool. If the target rate wasn’t sustained, you find out — and stats.rps is always the measured rate, so crust structurally cannot report a rate it didn’t achieve (the classic coordinated-omission trap: pause the clock while the server chokes, then advertise the target as the result). Dropped slots are never bursted later, because a burst would measure a traffic shape you didn’t ask for.

Sizing `parallel` — Little’s law

The pool you need is rate × latency: at 100/s against a ~150ms endpoint, about 15 requests are in flight at any instant, so parallel 20 gives headroom; parallel 8 guarantees drops. Start there, and let the drop report tell you if you guessed low — a nonzero dropped count with a healthy server means the pool is the bottleneck, not the service.

The generator itself doesn’t cap the rate: each wakeup emits every due slot as a batch, so tick emission sustains 5000/s and beyond (verified in crust’s own suite). Your real ceiling is downstream — the parallel pool × service time. A due slot is dropped only when consumer backpressure has let it go stale beyond maxLagMs (default 1s; TS API load(phases, {maxLagMs})) or the phase clock ran out — catch-up of due slots is the schedule, never a burst beyond it. Still single-process CI smoke-load and soak tooling, not a distributed load rig:

load 5s 400/s | parallel 100 | GET :3000/health --timeout 2s | expect 200 | stats
# {"count":2000,"wallMs":4999,"rps":400,"status":{"200":2000},"p50":0.2,"p95":0.6,"p99":1.2,"meanMs":0.2}

400/s offered, 400/s measured — the pool, not the pacing, is what you size.

Guarding against a hung upstream — `--timeout`

A load run has one failure mode worse than drops: an upstream that stops answering but keeps the sockets open. Without a bound, every hung request pins a parallel worker forever — the pool wedges, every later slot goes stale, and the run stalls instead of finishing. --timeout <dur> (durations ms/s/m) bounds every request the stage makes:

load 2s 20/s | parallel 10 | GET :3000/export --timeout 500ms | stats
# {"count":40,"wallMs":2457,"rps":16,"status":{"0":40},"p50":500.8,"p95":502.8,"p99":503.5,"meanMs":501}

Against an endpoint that never responds, the run still completes: each worker cycles every 500ms, and every timed-out request yields a {status: 0, timedOut: true, ms, url} record — it lands in the stats histogram under "0", fails expect 200, and the timedOut flag distinguishes it from a connection refusal (also status: 0, no flag). This is a per-request bound on http stages — not to be confused with test-fixture/test-pipes --timeout (integer ms, per fixture/line). Typo’d --flags on http stages are loud errors, so a misspelled --timeuot can’t silently run unbounded.

Soak testing — `stats --every N`

One cumulative summary hides drift: a warming cache, a leaking pool, a slowly saturating queue. stats --every N emits a delta summary per N-second window while the run is still going, then one cumulative summary tagged final:

range(0, 99999) | parallel 50 | GET :3000/health | stats --every 5
# {"window":1,"count":2412,"wallMs":5001,"rps":482,"status":{"200":2412},"p50":51.2,"p95":63.0,"p99":88.4,"meanMs":52.0}
# {"window":2,"count":2380,"wallMs":5000,"rps":476,"status":{"200":2380},"p50":52.8,"p95":66.1,"p99":94.0,"meanMs":53.6}
# {"window":3,"count":2118,"wallMs":5002,"rps":423,"status":{"200":2101,"503":17},"p50":58.9,"p95":112.7,"p99":301.2,"meanMs":66.1}
# ...
# {"final":true,"count":100000,"wallMs":214180,"rps":467,"status":{"200":99801,"503":199},"p50":53.1,"p95":71.9,"p99":140.2,"meanMs":55.0}

Each window is a fresh accumulator — window 3’s rising p99 and 503s are this window’s problem, visible while it happens, not averaged away by the first two minutes. Windows flush on the item path, so a fully stalled upstream holds the next window until an item arrives. Because results stream in completion order, a slow request lands in the window it finished in, and ms values are true durations — the percentiles stay honest.

Gating a soak

Windows are plain objects, so a chained assert turns “watch the soak” into “fail the soak the moment a window degrades”. Guard the predicate with the tag keys to scope it:

… | stats --every 5 | assert (s => !s.window || s.p95 < 400)   # gate each window
… | stats --every 5 | assert (s => !s.final  || s.p95 < 200)   # gate only the final summary

A bare predicate would run against every window object AND the final summary; the !s.window || / !s.final || guards scope it to one or the other (both are also correct without --every — the plain summary carries neither key). The whole threshold story — chained asserts, baselines, exit codes, CI snippets — is its own recipe: CI load gates.

Fixtures at volume — `--count` / `--threads`

--count N runs every matched fixture N times. Combine with --threads M for concurrency. When N > 1 the report adds a stress block per fixture: p50, p95, p99, mean, min, max, plus the status-code distribution. Each result is tagged with its iter index so failures point at the offending iteration. Reach for this over load when the traffic that matters is your fixtures — auth’d, DB-verified, multi-assertion requests — rather than a URL.

# 1000 iterations, 32 concurrent workers, JSON report
test-fixture stress.crust.ts -n1000 -j32 -o report.json

# Same idea, glob of fixtures, Markdown summary
test-fixture 'fixtures/**/*.crust.ts' -n500 -j16 -o report.md

# Long runs: cap each request, stop scheduling on first failure
test-fixture stress.crust.ts -n10000 -j32 -t5000 -b

// Programmatic runner — embed in CI, gate on percentiles
import { runFixtures } from "crust/testFixture";

const report = await runFixtures({
  target: "stress.crust.ts",
  count: 1000,
  threads: 32,
  out: "report.json",
});

if (report.stress.p95 > 250) {
  console.error(`p95 regression: ${report.stress.p95}ms`);
  process.exit(1);
}

Randomized inputs — `random`

To vary inputs across iterations, use thunks in input together with the random helper. A thunk (zero-argument function) is resolved + awaited per iteration; a function with at least one parameter is treated as a predicate matcher over the actual output value.

Watch the arity. data: () => true reads like “any value here” but a zero-argument function is a thunk supplying the expected value, so it means “the body must equal true”. Since 0.2.4 that is a hard error naming the fix rather than a silent mis-comparison. Write (v) => ... for a predicate, or the literal directly.

# Run the randomized fixture against your service
test-fixture stress.crust.ts -n1000 -j32 -o report.json

// stress.crust.ts
import { random } from "crust/testFixture/random";

const userIds = [1, 2, 3, 4, 5, 6, 7, 8];

export default {
  input: {
    url: () => `http://localhost:3000/users/${random.choice(userIds)}`,
    method: "POST",
    body: () => ({
      name: random.string(8),
      age: random.int(18, 99),
      tier: random.weighted([
        ["free", 7] as const,
        ["pro", 3] as const,
      ]),
      id: random.uuid(),
    }),
  },
  output: { status: (s: number) => s < 500 },
};

`random` helpers

random.int(min, max);                 // inclusive integer
random.float(min, max);
random.bool(p?);                      // weighted coin, default 0.5
random.choice(arr);                   // uniform element
random.from(iter);                    // works with any iterable
random.weighted([[v, w], ...]);       // weighted choice
random.string(len, alphabet?);
random.uuid();
random.shuffle(arr);

Reading the report

The report format is picked from --out’s extension: .json, .md, anything else is plain text. With no --out, you get a colored, folder-grouped summary on stdout.

{
  "name": "POST /users randomized",
  "iterations": 1000,
  "threads": 32,
  "stress": {
    "mean": 12.4,
    "min": 4,
    "max": 187,
    "p50": 9,
    "p95": 31,
    "p99": 62
  },
  "statusCodes": { "201": 994, "409": 6 }
}

Exit codes: 0 all pass, 1 any failure or error, 2 no files matched or bad args.

Runner knobs for long runs

--timeout <ms> — fail any fixture whose request runs longer. A fixture’s own input.signal wins; --timeout only fills the gap. Without it, one hung socket stalls a 10k-iteration run indefinitely.
--bail — stop starting new iterations after the first fail/error; in-flight ones finish and are reported. The right default for CI stress gates, where iteration 12 failing makes the next 9,988 noise.

Teardown isolation. Under --threads/--count, many iterations are mid-flight at once. A teardown must touch ONLY what its own ctx created — a “global cleanup” sweep (DELETE FROM users) shreds every iteration still running. Full resets belong outside the runner.

Tips

Bake-out runs: --count 1 first to confirm the fixture works at all. Once green, bump to --count 100 to find flakiness, then --count 1000+ for steady-state load.
Thread counts: start with --threads = vCPUs. Higher only helps if your service is I/O-bound.
Coverage via random.weighted: weight rare branches up during stress runs so they actually fire (e.g. tier pro 3:7 above ensures it shows up).
AOT runner: when installed via install.sh, the runner is AOT-compiled to a host-arch bytecode binary at ~/.crust/bin/crust-test-fixture for fast cold-start. The shell builtin execs that binary if present and falls back to in-process dynamic import otherwise.

CI load gates — thresholds, baselines, and exit codes for everything on this page.
API smoke tests — same fixtures, single-iteration mode.
Mock server — pair stress mode against an OpenAPI mock when the real backend isn’t ready.

The one-liner (fixed count)

Scenarios — duration & rate (load)

Reading the drop report

Sizing parallel — Little’s law

Guarding against a hung upstream — --timeout

Soak testing — stats --every N