Generated negative fixtures

Goal: stop hand-writing the boring-but-critical rejection tests. If your OpenAPI spec documents what the server must reject, gen-fixtures can derive the whole negative matrix — unauthenticated, wrong-scope, unknown-id, per-field validation, and per-field boundary cases — as .crust.ts files that test-fixture runs like any other fixture. And for every qualifying collection it also derives a CRUD flow suite: a capture-chained .pipes file that creates, reads, updates, deletes, and checks the tombstone.

The shape of it

# Spec in; one <tag>.gen.crust.ts per OpenAPI tag out, plus flow suites
gen-fixtures ./openapi.json
# generated 23 cases across 1 files -> /app/tests/gen
# generated 1 CRUD flows -> /app/tests/gen/flows/flows.gen.pipes

# Run the negative matrix like any other fixtures
test-fixture 'tests/gen/*.gen.crust.ts' -j8 -t5000

# Run the generated CRUD flows — the sibling setup module is auto-detected
test-pipes tests/gen/flows/flows.gen.pipes

// Programmatic — regenerate as part of a CI step
import { generateFixtures } from "crust/genFixtures";

const { outDir, files, totalCases } = await generateFixtures({
  swagger: "./openapi.json",          // URL or path; Swagger 2.0 auto-converted
  out: "tests/gen",
  setup: "./tests/gen-setup.ts",
});
console.log(`${totalCases} cases in ${files.length} files -> ${outDir}`);

The --out directory is deleted and recreated on every run — never edit generated files; change the spec or the setup module and regenerate.

The regen-and-diff workflow

Because generation is deterministic, the generated directory works like a lockfile: check it in, regenerate on spec changes, and read the diff as a review artifact. A new required field shows up as new 400 cases; a loosened maxLength shows up as a deleted boundary case. If the diff surprises you, the spec change did something you didn’t intend — you caught it before the server did.

gen-fixtures ./openapi.json
git diff --stat tests/gen         # what did the spec change actually change?

In CI, the same idea catches a stale checked-in matrix:

crust -c 'gen-fixtures ./openapi.json'
git diff --exit-code tests/gen    # fails the job if someone changed the spec without regenerating

Regenerating a spec that predates newer derivation rules yields a purely additive diff — existing case names, order and bodies are untouched, so the diff stays reviewable.

What gets derived

For every operation the spec documents:

Every 400 case perturbs a schema-valid base body — required fields are synthesized from the schema (format-aware: emails, uuids, dates, simple digit-pattern sampling) so exactly one thing is wrong per case. The wrong-type case for a nullable field is a value of a genuinely wrong type, never null: the union permits null, so sending it would assert nothing. Authz cases (401/403) also carry a valid body, because routes that validate before the auth middleware would otherwise 400 ahead of the status under test.

Response schemas ride along — output.schema

When the spec documents a response schema for a case’s expected status, the generated fixture also carries it as output.schema — the reserved test-fixture key that checks response-body conformance with per-field pointer failures. A spec whose 400 response documents the {error, code, fieldErrors} shape yields cases like:

output: {
  status: 400,
  data: (d: { code?: string; fieldErrors?: Record<string, unknown> }) =>
    d.code === "validation" && d.fieldErrors !== undefined && "name" in d.fieldErrors,
  schema: {"type":"object","required":["error","code","fieldErrors"],"properties":{"error":{"type":"string"},"code":{"type":"string","enum":["validation"]},"fieldErrors":{"type":"object"}}},
},

So the matrix doesn’t just prove the server rejects — it proves the rejection body matches the documented error contract. Cases whose expected status documents no schema (a bare 401: { description: … }) get no schema key — nothing is invented.

The boundary matrix, case by case

Given one schema —

{
  "type": "object",
  "additionalProperties": false,
  "required": ["name", "kind"],
  "properties": {
    "name":     { "type": "string", "minLength": 3, "maxLength": 40 },
    "kind":     { "type": "string", "enum": ["widget", "gadget"] },
    "priority": { "type": "integer", "minimum": 1, "maximum": 5 },
    "sku":      { "type": "string", "pattern": "^[A-Z]{3}-[0-9]{4}$" }
  }
}

— the POST operation alone yields this matrix (real case names from a generated run):

POST /things without credentials -> 401
POST /things missing required 'name' -> 400
POST /things wrong type for 'name' -> 400
POST /things missing required 'kind' -> 400
POST /things wrong type for 'kind' -> 400
POST /things invalid enum for 'kind' -> 400
POST /things too short 'name' -> 400          # minLength: sends a 2-char name
POST /things too long 'name' -> 400           # maxLength: sends 41 chars
POST /things below minimum 'priority' -> 400  # optional fields get cases too
POST /things above maximum 'priority' -> 400
POST /things pattern violation 'sku' -> 400
POST /things unexpected extra property -> 400 # additionalProperties: false

The rules that keep the matrix sane:

• Boundary cases run in fixed per-field order (too short, too long, below, above, pattern), so regeneration diffs are stable.
• An object whose properties live in a union branch is built from that branch. A node declaring type: "object" alongside a oneOf/anyOf used to produce {} — the explicit type sent it straight to the object case, which composes through allOf only. The same node without a type worked, which is what kept it hidden.
• exclusiveMinimum/exclusiveMaximum are honoured in both spellings — 3.0’s boolean modifier on minimum/maximum, and 3.1’s number — so {maximum: 1, exclusiveMaximum: true} is never answered with 1, the one value it excludes. Where a format and a pattern are both declared, the format constant is used only if it satisfies the pattern: format: email beside a pattern whose TLD is 2–5 letters rejects gen@crust.fixture, which has seven.
• The field-name heuristics yield to anything the schema actually states — an explicit pattern, an explicit format, and equally a length bound: client_id: {type: "string", maxLength: 20} never mentions uuid, and the 36-character one does not fit. An explicit format is different: there the schema asked for the value itself, and a maxLength too small for it is the spec contradicting itself, which crust leaves visible.
• The enum member picked satisfies the schema’s own declared type — real specs write {type: "string", enum: [true, false]}. And allOf is merged as an intersection: properties and required are unioned across the branches, and where two bounds disagree the stricter governs (minLength: 3 beside a branch’s minLength: 40 means 40). That reaches the shape where a node carries the type and leaves its refinements to branches.
• Formats the generator has no constant of its own for — uri/url above all — fall through to the same defaults the mock uses; its four fixed constants (email, uuid, date, date-time) keep their values so existing matrices do not churn. An explicit pattern outranks the field-name heuristics: a field called job_id carrying ^job-[0-9]{3}$ gets a value matching the pattern, not the stable uuid, because a guess from a name must not beat what the schema says.
• The valid base body honours the schema’s own bounds — maxLength, minLength, minItems/maxItems, minimum/maximum. Every 400-case perturbs that body, so where it was already invalid the expected 400 could arrive for the wrong reason: a false pass, which is the one thing crust must never produce. The fixed format values are deliberately unchanged, since checked-in matrices are CI-diffed against a regeneration.
• A wrong-type case is only generated when the value is actually rejected. A field whose schema constrains nothing — {properties: {…}} with no type, a description-only node — has no wrong value, so such a case would assert -> 400 for a request a correct API answers 200: a test that fails against a correct implementation. 6,101 were derivable from a 4,138-spec corpus. crust owns the validator, so it asks it rather than guessing.
• A missing type is inferred from the keywords present, sharing one implementation with the mock — so {minLength: 3} with no type now yields the boundary case it silently skipped before.
• Every $ref is inlined once and shared, and inlining stops after 200,000 nodes. Copying at each occurrence turned one real few-MB spec into a 2.03 GB structure and ran out of memory; past the budget a $ref inlines as {}, exactly as a cyclic one does, and gen-fixtures says so.
• maxLength above 4096 is skipped — checked-in files stay reviewable.
• maximum: Number.MAX_SAFE_INTEGER is treated as an “unbounded” sentinel and skipped.
• Zod-style nullable wrappers (anyOf: [X, {type: "null"}]) are unwrapped, so nullable fields get their boundary cases too.
• The pattern case is deduped when the wrong-type case already sends an unparseable string.
• The extra-property case asserts only status + code === "validation" — unknown-key naming in fieldErrors varies too much by server to pin.

Two practical notes from running these against real servers:

• Dereference your spec first. The generator reads requestBody.content.application/json.schema directly — a top-level $ref there means no base body and no 400/boundary cases for that op (you’ll see only the auth/404 cases). If your toolchain emits $refs into components.schemas, bundle/inline the spec before feeding it in.
• Validate before you look up. The PATCH/PUT boundary cases hit a random uuid with an invalid body and expect 400. A handler that checks existence first will 404 and fail the case. Spec-conformant servers validate the body before touching storage — the generated matrix enforces that ordering.

Generated CRUD flows

Unless you pass --no-flows, qualifying collection paths also get a flow suite: --out/flows/flows.gen.pipes plus a sibling flows.gen.setup.ts that test-pipes auto-detects — zero extra flags to run. Here’s a generated file, line by line:

# GENERATED by crust gen-fixtures — DO NOT EDIT.
# CRUD flows derived from the OpenAPI spec; run with test-pipes (the sibling
# flows.gen.setup.ts is auto-detected and seeds $GEN_AUTH_HEADER/$GEN_URL_*;
# capture stages write $GEN_ID_* at run time).
# NOTE: SQL assertions are not derivable from a spec, so none are emitted —
# add DB-level checks in a hand-written .pipes file if you need them.

# flow: /things  (create -> read -> update -> delete -> read-after-delete)
{"name":"gen-value-x","kind":"widget"} | POST $GEN_URL_THINGS -H "$GEN_AUTH_HEADER" | assert (r => r.status === 201) | (r => r.json()) | capture GEN_ID_THINGS (b => b.thing.id)
GET $GEN_URL_THINGS/$GEN_ID_THINGS -H "$GEN_AUTH_HEADER" | assert (r => r.status === 200) | (r => r.json()) | assert (j => JSON.stringify(j).includes(process.env.GEN_ID_THINGS ?? " "))
{"name":"gen-value-x"} | PATCH $GEN_URL_THINGS/$GEN_ID_THINGS -H "$GEN_AUTH_HEADER" | expect 200
DELETE $GEN_URL_THINGS/$GEN_ID_THINGS -H "$GEN_AUTH_HEADER" | expect 204
GET $GEN_URL_THINGS/$GEN_ID_THINGS -H "$GEN_AUTH_HEADER" | expect 404

• Line 1 — create. The body is the same schema-valid base body the 400 cases perturb. The status comes from the op’s lowest documented 2xx. The tail parses the response and captures the new id into $GEN_ID_THINGS — the generator derived where the id lives (b.thing.id) from the POST’s 2xx response example/schema.
• Line 2 — read. Interpolates the captured id into the item URL and asserts the id appears in the response body (capture values live in process.env, which is how a JS predicate reads them).
• Line 3 — update. Prefers PATCH over PUT when both exist; sends a valid partial body.
• Line 4 — delete. Emitted only when the spec documents a DELETE.
• Line 5 — tombstone. The read-after-delete 404 — emitted only when the item GET documents a 404 and a DELETE exists.

The generated flows.gen.setup.ts adapts your existing setup module — it calls shared() and headersFor(ctx, "member") to seed $GEN_AUTH_HEADER, and resolvePath per flow to seed $GEN_URL_<T>. Same contract, no extra exports.

Qualification — a collection path P gets a flow when: it has a POST with an application/json request schema and a documented 2xx; an item path P/{param} with at least one of GET/PUT/PATCH/DELETE exists; P carries no path params beyond the scope param (nested collections are skipped with a stdout notice); and the created id’s location is derivable from the POST’s 2xx response — the media example, else schema.properties: top-level id, else the first object-valued property containing an id. Not derivable → skipped with a notice naming the path.

gen-fixtures ./openapi.json --no-flows   # matrix only

Wiring flows into the suite — a flow file is a normal .pipes file, so it runs alongside your hand-written ones:

# everything at once: hand-written suites AND generated flows
test-pipes 'tests/**/*.pipes' -b -t5000

Each .pipes file is hermetic (env snapshotted/restored per file), so generated flows can’t leak $GEN_* vars into your hand-written suites — and since the spec can’t know your table names, add sql-assert lines in a hand-written .pipes file when you want the DB-level counterpart of a flow.

The setup module contract

Generated files are app-agnostic: they import only from your --setup module. It carries every app-specific detail — how to authenticate, what “scope” means, where the API lives. Required exports:

A minimal real module:

// tests/gen-setup.ts
const BASE = process.env.BASE ?? "http://localhost:3000";

export const JSON_HEADERS = { "content-type": "application/json" };
export const scopeParam = "buildingId";
export const scopeRoots = ["/api/buildings"];

// Promise-cached: every generated fixture shares ONE scenario.
let scenario: Promise<{
  buildingId: string;
  memberToken: string;
  outsiderToken: string;
}> | null = null;

export function shared() {
  scenario ??= (async () => {
    const member = await signupAndLogin("member@gen.test");
    const building = await createBuilding(member.token);
    const outsider = await signupAndLogin("outsider@gen.test"); // no membership
    return {
      buildingId: building.id,
      memberToken: member.token,
      outsiderToken: outsider.token,
    };
  })();
  return scenario;
}

type Ctx = Awaited<ReturnType<typeof shared>>;

export function headersFor(ctx: Ctx, role: "none" | "member" | "outsider") {
  if (role === "none") return JSON_HEADERS;
  const token = role === "member" ? ctx.memberToken : ctx.outsiderToken;
  return { ...JSON_HEADERS, authorization: `Bearer ${token}` };
}

export function resolvePath(ctx: Ctx, template: string) {
  const path = template
    .replace("{buildingId}", ctx.buildingId)
    .replace(/\{\w+\}/g, () => crypto.randomUUID()); // unknown ids -> 404 bait
  return `${BASE}${path}`;
}

The import specifier in generated files is rewritten relative to --out, so a path-like --setup works from anywhere; a bare package specifier is kept as-is.

The crust repo ships a complete runnable module to copy at examples/gen-setup.ts — the same shape as above with working fetch-based auth helpers to replace with your API’s flow.

crust does not ship ./tests/gen-setup.ts — that path is the default by convention in your repo. Run gen-fixtures without one and it says so, naming the path it looked at, the template to copy, and the --setup flag.

What a generated file looks like

// GENERATED by crust gen-fixtures — DO NOT EDIT.
import { JSON_HEADERS, headersFor, resolvePath, shared } from "../gen-setup.ts";

type Ctx = Awaited<ReturnType<typeof shared>>;

export default [
  {
    name: "POST /api/buildings without credentials -> 401",
    setup: shared,
    input: (ctx: Ctx) => ({
      url: resolvePath(ctx, "/api/buildings"),
      method: "POST",
      headers: JSON_HEADERS,
      body: "{\"name\":\"gen-value-x\"}",
    }),
    output: { status: 401 },
  },
  {
    name: "POST /api/buildings missing required 'name' -> 400",
    setup: shared,
    input: (ctx: Ctx) => ({
      url: resolvePath(ctx, "/api/buildings"),
      method: "POST",
      headers: headersFor(ctx, "member"),
      body: "{}",
    }),
    output: {
      status: 400,
      data: (d: { code?: string; fieldErrors?: Record<string, unknown> }) =>
        d.code === "validation" && d.fieldErrors !== undefined && "name" in d.fieldErrors,
    },
  },
];

Plain test-fixture fixtures — nothing magic. --threads, --timeout, --bail, and report formats all apply.

Tips

• Descriptions are the trigger. The 401 derivation keys off the response description (/not authenticated|log in/i). If your middleware 401s aren’t matching, that’s the string to fix in the spec.
• Run against the real server, not the mock. The whole point is proving the server rejects; mock-server accepts anything that routes — unless you boot it with --validate, in which case it rejects with 422, which still isn’t your server’s documented 400 contract.
• Zero cases for an op means the spec is silent. Documenting the 400/401/403/404 responses — and the minLength/maximum/pattern constraints — is what buys you the tests.

Related

• API smoke tests — the positive-path counterpart: one-liners, capture-chained .pipes suites, .crust.ts fixtures.
• Mock server — the spec’s other uses: mocking, request validation, and conformance-proxying.
• Stress testing — the same runner at volume.