Testing

Implements: TEST-01, TEST-02, TEST-03, TEST-04, TEST-05, TEST-06, TEST-07, TEST-08, TEST-09 — see CHARTER.md Part 4 for the requirements and their status.

How ESI.ts is tested: what each tier owns, where it lives, how to run it, where CI runs it, and the signal that proves it can fail. This is the canonical testing guide. What blocks a merge, what runs nightly and what files an issue is in QUALITY-GATES.md. Mutation testing, including the ratchet, the shards and how to kill a survivor, is in Mutation testing. How to write a Rule is in tests/bdd/README.md (the rules) and tests/bdd/GUIDE.md (the walkthrough).

The stance

Three positions explain every choice below.

1. A tier that cannot fail manufactures confidence. Each tier owns one class of failure and carries a signal that proves it can go red: a negative fixture, a killed mutant, an injected leak, a deliberately broken package. A green run from a tier with no such signal says nothing, so the one tier that has none today (API fuzz) is listed as such rather than counted as protection.
2. Every gate is a one-way ratchet. Floors, baselines and known-gap lists start at the value measured on the day they were introduced and move in one direction. Each shrink-only list fails when an entry is added, fails when a listed entry stops reproducing, and fails closed when the base ref it compares against cannot be read. A fix cannot leave a stale exception behind, and a pull request cannot exempt itself.
3. The specification executes. Behaviour is stated as EARS requirements, one shall per Gherkin Rule:, verified by scenarios that mock only the HTTP transport. npm run spec:audit fails a pull request that weakens the wording, npm run bdd:report fails one in which a scenario did not run, and the BDD-only mutation ratchet fails when the scenarios stop killing mutants (charter posture 6).

Where the suite stands

Measured on 2026-09-28 on master at 9486fe2c (v10.2.3; 11.0.0 in progress), on a 4-core machine. The command in the first column reproduces each row.

Command	Config	Suites	Tests	Result
npm test	config/jest/unit.config.cjs	291	8,308	All pass, about 141 s
of which unit, tests/tdd (not composition)	same	210	7,524
of which specification, tests/bdd	same	74	742	31 legacy *.steps.ts files and 43 *.spec.ts entries
of which composition, tests/tdd/composition	same	7	42
npm run fuzz	config/jest/fuzz.config.cjs	15	1,240	All pass, about 22 s with four workers
npm run faults	config/jest/faults.config.cjs	2	148	All pass
npm run contract:replay	config/jest/contract.replay.config.cjs	5	121	All pass, over 86 recorded fixtures
npm run test:integration	config/jest/integration.config.cjs	6	228	20 mocked pass; 208 live tests skip without ESI_LIVE_TESTS, ESI_GATED_TESTS or sde-data/
Offline total		319	10,045	9,837 run, 0 failures

Coverage (npm run coverage)	Measured	Floor (config/jest/unit.config.cjs)
Statements	97.79%	90%
Branches	95.88%	80%
Functions	93.57%	75%
Lines	97.85%	90%

Coverage is collected from src/**/*.ts, excluding .d.ts, src/types/, *.generated.ts and src/clients/generated/. The floors sit well below the measured values on purpose (TEST-04): they catch an untested module without turning one bad week into a broken gate. Mutation, not coverage, is what says the tests assert anything.

Specification (npm run spec:audit:verbose)	Count
Feature files	77 (core/ 52, integration/ 1, performance/ 1, sde/ 23)
Rule: requirements	664
Scenarios	863 (848 Scenario, 15 Scenario Outline)
Audit result	77 of 77 pass

Other counts, each reproducible with ls or find: 13 tsd files in tests/typetests/, 9 Jest configs (ls config/jest/*.config.cjs), and 29 workflows in .github/workflows/ (30 files including its README.md).

The tiers

This is the canonical tier table. Every other file names tiers by these names; none of them uses a number. Seventeen tiers, each owning one class of failure.

Tier	Location	Files	Config · command	Where CI runs it	Signal that it can fail
Static analysis	config/eslint/*.config.mjs, scripts/*/*-lint.ts	4 lints and export coverage	lint:suite-health, lint:bdd-seam, lint:determinism, lint:layers, test:export-coverage	Push (ci-fast: seam, suite health, layers); PR (lint-and-build, spec-audit, static-analysis)	One negative fixture per rule, linted at its real path by a suite in tests/tdd/
Unit	tests/tdd/	240 suites	config/jest/unit.config.cjs · npm test	Push, Node 22 (ci-fast); PR, Node 22/24 (unit-tests), coverage, full-test-suite	Stryker mutation, per-directory floors
Composition and concurrency	tests/tdd/composition/	7 suites	config/jest/unit.config.cjs · npm test	Push; PR; nightly-interleave.yml (four calls, seeded random schedules)	Pinned schedule counts; interleave.test.ts finds and replays a planted race
Specification (EARS/BDD)	tests/bdd/	77 features, 78 suites	config/jest/unit.config.cjs · npm run bdd	Push (inside npm test); PR (bdd-tests, spec-audit); ears.yml, advisory	bdd:report fails on a scenario that did not execute; BDD-only mutation floors
Type tests	tests/typetests/	13	tsd · npm run test:types	PR (full-test-suite, through test:all)	Type mutation, nightly, per-entry-point floors
Properties and fuzz	tests/fuzz/	15 suites	config/jest/fuzz.config.cjs · npm run fuzz	PR (fuzz-tests, 100 runs per property); nightly-properties.yml (10,000)	Each model-based property must fail against registered known-bad implementations
Fault injection	tests/faults/	2 suites, plus 1 nightly	config/jest/faults.config.cjs · npm run faults; npm run faults:nightly	PR (fault-catalogue); nightly-faults.yml	fixtures/weak-fault.ts must be rejected on every count
Recorded replay	tests/contract/replay/	5 suites, 86 fixtures	config/jest/contract.replay.config.cjs · npm run contract:replay	PR (contract-replay); nightly-recorded-payloads.yml re-records	A recording edited to violate its schema must be rejected
Live contract	tests/contract/	2 suites	config/jest/contract.live.config.cjs · npm run contract:live	PR (contract-tests, soft-skips on HTTP 503); weekly in maintenance.yml	An unknown spec mismatch hard-fails; known ones sit in named exception sets
Integration, mocked	tests/integration/full-stack.test.ts	1	config/jest/integration.config.cjs · npm run test:integration	PR (full-test-suite, through test:all)	None of its own; it overlaps the unit and specification tiers, which are mutation-scored
Integration, live	tests/integration/: live-esi, client-integration, esi-spec-contract, sde/	4	ESI_LIVE_TESTS=true npm run test:integration; npm run test:integration:live	sde/ nightly in nightly-sde.yml; the other three by no workflow	A live response of the wrong shape fails; test:integration:live fails when its variable is missing
Integration, gated auth	tests/integration/gated-auth.test.ts	1	npm run test:integration:gated (reads .env)	Not run by any workflow	Fails when ESI_GATED_TESTS=true and no token is set
Consumer contract	tests/consumer/	1 package	npm run test:consumer	PR (consumer-contract, four rows); consumer-matrix-nightly.yml; release.yml	Five broken packages must fail the matrix; a clean control must pass
Documentation examples	tests/doc-examples/	every ts block in the docs	npm run test:docs-examples	PR (doc-examples)	Negative fixtures in tests/tdd/doc-examples/
Benchmarks and heap soak	tests/benchmark/	18 tasks and the soak	npm run bench:ab, npm run bench:compare, npm run soak	PR (benchmarks, only when a hot path changes); nightly-benchmarks.yml	An injected slowdown and an injected leak must be flagged
Mutation	src/**	unit, BDD-only and type runs	Stryker · npm run mutation:pr, mutation, mutation:bdd; npm run test:type-mutation	PR (mutation-pr, changed files); nightly-mutation.yml	tests/mutation-fixture/ must leave survivors and kill at least one mutant
API fuzz	Prism mock and Schemathesis	none	npm run fuzz:api (Docker)	nightly-schemathesis.yml	None. It uploads a report and files an api-fuzz issue on failure; count it as a probe, not a gate

The charter's Part 4 table groups the same ground as twelve rows. This table splits its contract row into recorded replay and live contract, and adds four the charter counts under gates: static analysis, composition, consumer contract and documentation examples.

The shrink-only lists that hold today's known gaps live beside the tier that owns them: tests/faults/known-gaps.json (empty), tests/contract/fixtures/known-mismatches.json (empty) and unrecordable.json (one entry), scripts/spec/spec-audit-exceptions.json (unconverted empty, legacyStepFiles 38, unchartedFeatures empty), scripts/*/*-baseline.json, config/mutation/unit-thresholds.json, config/mutation/bdd-thresholds.json and config/mutation/type-thresholds.json.

The test system in C4

Three views, in the notation of guides/ARCHITECTURE.md: what the suite talks to, what runs when, and how a test reaches the code under test.

Level 1: Context

Who runs the tests, and the outside things they depend on. Only the live contract, the live and gated integration suites, payload recording, the nightly examples and the post-publish canary reach ESI. The test tiers among them run only when their environment variable is set.

flowchart TB
    contributor(["Contributor"])
    reviewer(["Reviewer"])

    subgraph boundary [" "]
        suite["ESI.ts test suite<br/>17 tiers, one per failure class"]
    end

    esi[/"ESI API<br/>live, public + authenticated"/]
    spec[/"ESI OpenAPI document"/]
    registry[/"npm registry"/]
    actions[/"GitHub Actions"/]

    contributor -- "npm test, npm run check:local, ..." --> suite
    suite -- "recordings, live contract, manual live runs" --> esi
    spec -. "generated types, cache TTLs, drift" .-> suite
    suite -- "packed tarball installed by the consumer matrix" --> registry
    suite -- "jobs, ratchet files, nightly issues" --> actions
    actions -- "one required check: ci-success" --> reviewer

    style contributor fill:#08427b,color:#fff,stroke:#073b6f
    style reviewer fill:#08427b,color:#fff,stroke:#073b6f
    style suite fill:#1168bd,color:#fff,stroke:#0e5aa7
    style esi fill:#999,color:#fff,stroke:#888
    style spec fill:#999,color:#fff,stroke:#888
    style registry fill:#999,color:#fff,stroke:#888
    style actions fill:#999,color:#fff,stroke:#888
    style boundary fill:none,stroke:#1168bd,stroke-width:2px,stroke-dasharray:5

Level 2: Containers

Each tier is a container with its own runner and trigger. A tier moves right as it gets slower: what cannot fit a pull request runs nightly and gates the next pull request through a ratchet file rather than by blocking.

flowchart LR
    subgraph local ["Local: pre-commit, pre-push, check:local"]
        direction TB
        staged["lint-staged<br/>ESLint + Prettier"]
        checklocal["npm run check:local<br/>every offline CI tier"]
    end

    subgraph pr ["Pull request, gate: ci-success"]
        direction TB
        unit["Unit + BDD + composition<br/>config/jest/unit.config.cjs"]
        props["Properties and fuzz<br/>config/jest/fuzz.config.cjs"]
        faults["Fault catalogue<br/>config/jest/faults.config.cjs"]
        replay["Recorded replay<br/>config/jest/contract.replay.config.cjs"]
        live["Live contract<br/>soft-skips on 503"]
        types["tsd · api-extractor · publint · attw · size-limit"]
        consumer["Consumer matrix<br/>ESM/CJS x node16/nodenext/bundler"]
        docs["Documentation examples"]
        mutpr["Mutation of changed files<br/>Stryker --incremental"]
        bench["Benchmarks base vs head<br/>only when hot paths change"]
        statics["Static analysis<br/>export coverage · determinism · layers · suite health · seam lint"]
    end

    subgraph night ["Nightly: reports and issues, never blocks directly"]
        direction TB
        nmut["Full mutation, unit + BDD + type"]
        nprops["Properties at 10 000 runs"]
        nfaults["Payload fuzz, every endpoint"]
        nrec["Re-record payloads, diff, open a PR"]
        nbench["Benchmarks vs pinned reference + heap soak"]
        nex["Examples against live ESI"]
        nflake["No-retry run · interleavings · audit · spec drift · Schemathesis"]
    end

    subgraph weekly ["Weekly"]
        direction TB
        wlive["Live contract + oasdiff<br/>maintenance.yml"]
    end

    subgraph release ["Release"]
        direction TB
        rmatrix["Consumer matrix against the signed tarball"]
        rcanary["Post-publish canary"]
    end

    contributor(["Contributor"]) --> local
    local --> pr
    pr --> release
    night -- "updates ratchet files, opens one issue per tier" --> pr

    style contributor fill:#08427b,color:#fff,stroke:#073b6f
    style local fill:none,stroke:#1168bd,stroke-width:2px
    style pr fill:none,stroke:#1168bd,stroke-width:2px
    style night fill:none,stroke:#999,stroke-width:2px,stroke-dasharray:5
    style weekly fill:none,stroke:#999,stroke-width:2px,stroke-dasharray:5
    style release fill:none,stroke:#999,stroke-width:2px

Level 3: Components of a test run

How a test reaches the code. Everything above the seam is test-owned; everything below it is the real library. Nothing between a public client method and fetch is stubbed, which is what lets a scenario fail for a bug anywhere in src/.

flowchart TB
    subgraph drivers ["Test drivers"]
        direction LR
        steps["BDD steps<br/>tests/bdd/steps"]
        interleave["Interleaving scheduler<br/>tests/tdd/composition"]
        catalogue["Fault catalogue<br/>tests/faults"]
        fixtures["Recorded payloads<br/>tests/contract/fixtures"]
        properties["Property runner<br/>tests/fuzz"]
    end

    seam["Transport seam: queueResponse() over jest-fetch-mock<br/>strict: an unrequested or unconsumed response fails the test"]

    subgraph lib ["ESI.ts, running for real"]
        direction TB
        client["Domain client method"]
        pipeline["Request pipeline<br/>rate limiter · circuit breaker · dedupe · retry"]
        cache["ETag cache and pagination"]
        validate["Zod validation"]
    end

    fetchboundary[/"fetch()"/]

    assertions["Assertions<br/>expect · assertNoProblems · assertThat · named invariants"]

    steps --> seam
    interleave --> seam
    catalogue --> seam
    fixtures --> seam
    properties --> seam
    seam --> client
    client --> pipeline
    pipeline --> cache
    cache --> validate
    pipeline -. "only the live tiers get here" .-> fetchboundary
    validate --> assertions
    seam -- "recorded request sequence" --> assertions

    style drivers fill:none,stroke:#1168bd,stroke-width:2px
    style lib fill:none,stroke:#0e5aa7,stroke-width:2px
    style seam fill:#1168bd,color:#fff,stroke:#0e5aa7
    style assertions fill:#1168bd,color:#fff,stroke:#0e5aa7
    style fetchboundary fill:#999,color:#fff,stroke:#888

Test layout

tests/
  tdd/                 unit tests: one directory per domain client or core area,
                       plus the self-tests of every gate (spec-audit, bdd-seam,
                       suite-health, determinism-lint, layers, mutation-ratchet,
                       type-mutation, consumer-contract, doc-examples,
                       package-lint, export-coverage, benchmark, verify-local, scripts)
    composition/       pipeline stages under overlapping calls, interleaving scheduler
    helpers/           describeClientErrors and shared test utilities
  bdd/
    features/          the .feature files: core/, integration/, performance/, sde/
    specs/             47 spec entries: 23 core and 23 SDE features, plus step-library.spec.ts (the dry run)
    steps/             given/ when/ then/, one step per file
    step-definitions/  31 legacy defineFeature files, shrink-only (legacyStepFiles)
    support/           binder, transport seam, World, per-domain fixtures
  fuzz/                15 fast-check suites; 7 model-based *.property.test.ts (two for the SDE)
  faults/              fault catalogue, its self-test, nightly payload fuzz
  contract/            live deep contract and snapshot, recorder, shape diff
    replay/            5 replay suites
    fixtures/          recorded/ (86), recorded-negative/, unrecordable.json, known-mismatches.json
  integration/         full-stack (mocked), live-esi, client-integration,
                       esi-spec-contract, gated-auth, sde/sde-real-data
  typetests/           7 tsd files
  consumer/            a private downstream package for the consumer contract
  doc-examples/        prelude and stub fetch for test:docs-examples
  benchmark/           mitata harness, 18 tasks, heap soak
  mutation-fixture/    the known-weak clamp for Stryker's self-test
  setup/               requireLiveTests.ts, the global setup of the live configs
  support/             shared assertions

Each of the tier directories tests/fuzz, tests/faults, tests/contract, tests/benchmark, tests/bdd and tests/tdd/composition has an AGENTS.md with its rules. Read it before changing the tier.

Running tests

npm test                     # unit + BDD + composition (config/jest/unit.config.cjs)
npm run test:watch           # the same, in watch mode
npm run coverage             # the same, with coverage and the floors
npm run bdd                  # the BDD suites only
npm run bdd:<domain>         # one BDD suite: bdd:market, bdd:resilience, bdd:sde, ... (npm run help -- bdd)
npm run bdd:steps            # BDD dry run: every step matches one definition, none unused
npm run spec:audit           # EARS and Gherkin audit of the feature files
npm run ears                 # one PASS/FAIL/NOT RUN verdict per Rule
npm run fuzz                 # properties and fuzz
npm run fuzz:properties      # the model-based properties and the SDE schema fuzz
npm run faults               # fault catalogue
npm run faults:nightly       # seeded payload fuzz over every endpoint
npm run contract:replay      # recorded payloads through the pipeline, no network
npm run test:integration     # mocked integration; live suites skip
npm run test:types           # tsd
npm run test:all             # test, bdd, test:integration, fuzz, test:types (what full-test-suite runs)
npm run check:local          # every offline CI tier in one run (below)

Tiers that need a build, a tarball, Docker, the network or minutes of CPU:

npm run test:consumer        # pack, install into a clean consumer, type-check and run
npm run test:docs-examples   # type-check every ts block in the docs against the tarball
npm run test:type-mutation   # after npm run build; -- --ratchet gates
npm run mutation:pr          # Stryker over your changed src/ files
npm run bench:ab -- --base ../base-worktree --head . && npm run bench:compare
npm run soak                 # heap soak, 100 000 requests
ESI_LIVE_TESTS=true npm run contract:live
ESI_LIVE_TESTS=true npm run contract:record
npm run fuzz:api             # Schemathesis against a Prism mock (Docker)

Running the gates locally

npm run check:local runs every tier ci.yml gates on that works offline, and keeps going after a failure so the answer is the whole list. check:local:fast skips the build and the tiers that need dist/; check:local:all adds type mutation. The tier list is TIERS in scripts/quality/verify-local-core.ts, and tests/tdd/verify-local/verify-local.test.ts fails when ci.yml gains an npm run or a directly invoked tool that is neither in the list nor excluded with a written reason. What it leaves out and why is in QUALITY-GATES.md.

Static analysis

The lints in this tier read tests or source without running them. Each has a Jest suite in tests/tdd/ that lints negative fixtures at their real paths and requires exactly the expected findings, so a rule that stops firing fails npm test.

Command	What it rejects	Baseline	Self-test
npm run lint:suite-health	.only, .skip, .todo, assertion-free tests and Then steps, swallowed assertions, unrestored console mocks, in tests/	None; every finding fails	tests/tdd/suite-health/
npm run lint:bdd-seam	Any spyOn of an ESI client, domain client, client prototype or ApiClient, and reassigned client methods, in tests/bdd/	None	tests/tdd/bdd-seam/
npm run lint:determinism	Date.now(), timers, performance.now(), Math.random() in src/ outside the clock modules (src/core/clock.ts; src/sde/clock.ts for the SDE, which may import only the ports)	scripts/quality/determinism-baseline.json, shrink-only	tests/tdd/determinism-lint/
npm run lint:layers	Imports in src/ that point outward: core into the layers above it, ports importing anything, generated code importing anything but the ports, the SDE and the pipeline importing each other	BASELINE in config/eslint/layers.rules.cjs, shrink-only	tests/tdd/layers/
npm run test:export-coverage	A public export no file under tests/ references	scripts/package/export-coverage-baseline.json, shrink-only	tests/tdd/export-coverage/

The rule tables and reasoning are in QUALITY-GATES.md and QUALITY-GATES.md. npm run lint also covers tests/ with the src/ rule set, under the relaxations the tests/** block of eslint.config.mjs declares; no-floating-promises, no-misused-promises and await-thenable stay errors there (TEST-09, #271). lint:suite-health and lint:bdd-seam add the test-only rules.

Time and randomness

An assertion about Expires, retry backoff or circuit half-open timing is only deterministic if the code under test takes its time from something the test controls. npm run lint:determinism (config/eslint/determinism.rules.cjs, driven by scripts/quality/determinism-lint.ts) restricts these in src/: Date.now(), new Date() and Date() with no arguments, performance.now(), process.hrtime, Math.random(), setTimeout, setInterval, setImmediate, queueMicrotask (bare or through globalThis, global, window, self) and imports of timers / timers/promises. Parsing a date (new Date(header), Date.parse) is allowed. The one allow-listed path is the clock module, src/core/clock.ts; inline eslint-disable comments are ignored.

The sites that exist today are counted per file and construct in scripts/quality/determinism-baseline.json. The baseline only shrinks: a count above its entry fails (a new site), a count below its entry fails until the entry is lowered (npm run lint:determinism -- --update, which never raises one), and an entry above origin/master's fails. With no base ref resolvable (set DETERMINISM_BASE_REF, or fetch master) the check fails closed. Until call sites move to an injected clock, tests of existing timing code keep using jest.useFakeTimers(). They assert the exact delay or boundary (fresh at the TTL, expired one millisecond later) rather than sleeping for real and bounding Date.now(): a test that waits on the wall clock kills a boundary mutant on one runner and not another, which is how 43 mutants came to flip between nightlies (#382, "Timing tests made deterministic" under The ratchet). A test that drives the whole request pipeline and needs only Date frozen uses useFakeDate() from tests/tdd/helpers/fakeDate.ts.

Each restricted construct has a negative fixture in tests/tdd/determinism-lint/fixtures/violations/, linted through ESLint's Node API by tests/tdd/determinism-lint/determinism-lint.test.ts, alongside compliant fixtures that must produce no findings and the ratchet's added, stale and no-base-ref cases.

Package lint and size budgets

Run: npm run lint:package (publint and Are The Types Wrong on the npm pack tarball) and npm run size (size-limit, after a build) CI: package-lint in ci.yml, inside ci-success. Negative fixtures: tests/tdd/package-lint/, part of npm test.

The consumer contract installs and runs the tarball; these check it statically. The linters block on any finding outside scripts/package/package-lint-baseline.json under node16 CJS, node16 ESM and bundler resolution, and .size-limit.cjs sets a ceiling for the ESM and CJS build of every exports sub-path. The unit suite runs both tools against a package with a broken exports map and a clean control, and runs size-limit against a budget set below its fixture's size, so a check that stops firing fails npm test. Rules, the baseline ratchet and how to raise a budget: QUALITY-GATES.md.

Unit

Location: tests/tdd/Config: config/jest/unit.config.cjsRun: npm test

182 suites and 6,648 tests (npm test, 2026-09-27). One directory per domain client, one per core area (core/, schemas/, auth/, sde/, ports/, adapters/, config/), and one per gate self-test. The groups that matter most:

• Domain clients. One test file per client. Each mocks fetch and checks URL construction, response parsing and the result type. The shared describeClientErrors helper adds the five HTTP error paths (401, 403, 404, 429, 500) to every non-trivial client.
• Core infrastructure. Circuit breaker, rate limiter, offset and cursor pagination, the ETag cache, request deduplication, retry with backoff, the middleware pipeline, endpoint definitions, validation, timeout, diagnostics and configuration.
• Public API surface. publicApiSurface.test.ts and apiSurfaceSnapshots.test.ts snapshot the public exports and the shapes of EsiError, RateLimiter status and CircuitBreaker stats. A rename or removal fails here before it ships under the wrong SemVer.
• Schemas. schemaRejection.test.ts is table-driven over every schema file: valid data is accepted, a wrong type is rejected, an extra field is preserved (looseObject). tests/tdd/testing/TestDataFactory.schemas.test.ts checks every factory default passes the schema its endpoint is validated with.
• Security and resilience. Token scoping, HTTPS and the host allowlist, path injection, query length and non-finite numbers (security.test.ts); 429/420 handling, malformed and truncated bodies, stale-on-error, retry and deduplication under load (resilience.test.ts).
• Gate self-tests. spec-audit/, bdd-seam/, suite-health/, determinism-lint/, layers/, mutation-ratchet/, type-mutation/, consumer-contract/, doc-examples/, package-lint/, export-coverage/, benchmark/, verify-local/ and scripts/. These are what make the other tiers' signals real.

The TDD pattern

Each domain client test builds an ApiClient, mocks the fetch response, calls the method and asserts the result.

import { AllianceClient } from '../../../src/clients/AllianceClient';
import { ApiClientBuilder } from '../../../src/core/ApiClientBuilder';
import { getBody } from '../../../src/core/util/testHelpers';
import fetchMock from 'jest-fetch-mock';

describe('AllianceClient', () => {
  let allianceClient: AllianceClient;

  beforeEach(() => {
    fetchMock.resetMocks();
    const client = new ApiClientBuilder()
      .setClientId('test')
      .setLink('https://esi.evetech.net')
      .build();
    allianceClient = new AllianceClient(client);
  });

  it('returns alliance info', async () => {
    fetchMock.mockResponseOnce(
      JSON.stringify({
        alliance_id: 99005338,
        name: 'Goonswarm Federation',
        ticker: 'CONDI',
      }),
    );

    const result = await getBody(() =>
      allianceClient.getAllianceById(99005338),
    );
    expect(result.name).toBe('Goonswarm Federation');
  });

  it('throws on 404', async () => {
    fetchMock.mockResponseOnce('Not Found', { status: 404 });
    await expect(allianceClient.getAllianceById(99999999)).rejects.toThrow(
      'Resource not found',
    );
  });
});

The shared error helper

describeClientErrors (tests/tdd/helpers/clientErrorTests.ts) generates a describe block that tests the five HTTP error codes against the exact messages in ApiRequestHandler.STATUS_MESSAGES and checks the thrown error carries the status code. Each case builds a fresh ApiClient and passes it to the callback, which must construct the domain client from it: a 429 blocks the endpoint's rate-limit group for 60 seconds on the client that received it, and a shared client would make every later test in the file time out once jest --randomize puts the 429 case first.

import { describeClientErrors } from '../helpers/clientErrorTests';

describe('MarketClient', () => {
  describeClientErrors('MarketClient', (apiClient) =>
    new MarketClient(apiClient).getMarketPrices(),
  );
});

Composition and concurrency

Location: tests/tdd/composition/ (agent notes in its AGENTS.md) Config: config/jest/unit.config.cjs, so it runs in npm test and Stryker covers it Run: npx jest --config config/jest/unit.config.cjs --testPathPatterns=compositionNightly: nightly-interleave.yml

Owns one failure class: request pipeline stages that are each correct alone but wrong together when calls overlap. Each scenario drives a real EsiClient with every stage on (rate limiter out of test mode, circuit breaker, deduplication, retry, ETag cache, Zod validation) and mocks only HTTP, at the BDD transport seam. The interleaving scheduler (support/interleave.ts) holds each request at the seam and chooses step by step which call starts, which response arrives and when fake time advances, then checks named invariants over the sequence of requests (method, path, If-None-Match) and outcomes.

Scenario	Interaction
retryCircuit.test.ts	Retry inside a circuit that has just opened
staleRefresh.test.ts	Stale-on-error while the stale entry is being refreshed
dedupeRejection.test.ts	Deduplication when the shared in-flight request fails
etagOrdering.test.ts	A 304 for an old ETag arriving after a 200 for a new one
errorLimit.test.ts	The ESI error-limit back-off holding back every endpoint
writeInvalidation.test.ts	A write invalidating the cache while a read is in flight

On every PR each scenario runs every schedule of two and three calls; the schedule counts are pinned, so a change that stops calls overlapping fails instead of silently testing less. Nightly, four calls run in seeded random order (ESI_INTERLEAVE_RUNS schedules, 5,000 by default). A failure prints the broken invariant, the event log and the command that replays the schedule (ESI_INTERLEAVE_REPLAY=..., plus ESI_INTERLEAVE_MODE=random for a nightly failure). The scheduler's own tests (interleave.test.ts) show it enumerating exactly the schedules that exist, finding and replaying a planted race, and failing closed on deadlock, nondeterminism and malformed environment variables.

Specification (EARS/BDD)

Location: tests/bdd/Config: config/jest/unit.config.cjs (its testMatch includes tests/bdd/step-definitions/**/*.steps.ts and tests/bdd/specs/**/*.spec.ts) Run: npm run bdd, or npm test, which includes it

The 77 feature files state 664 requirements, one per Rule:, verified by 863 scenarios (npm run spec:audit:verbose). Every scenario mocks at the transport seam (TEST-03), so the rate limiter, retry, deduplication, ETag cache, JSON parsing and Zod validation all execute. Coverage by area:

• Core (features/core/, 45 files): one feature per domain client, plus the cross-cutting ones numbered from 0050: ETag caching, resilience, response headers, runtime validation, token management and request headers.
• Integration (features/integration/): cross-domain workflows such as character profile assembly and fleet operations.
• Performance (features/performance/): smoke checks that responses are waited for, concurrent calls overlap rather than queue, and large or repeated payloads come back complete and in order. These scenarios state no latency or throughput budget. The only time bounds they keep separate overlapping requests from serial dispatch; how fast a path is, and whether a change slowed it, is the benchmark tier's job.
• SDE (features/sde/, 23 files): the Static Data Export module, every family of the provider port, loading, the optional peers, the memory entry and ingestion. sde/TESTING.md is the module's scorecard.

Three gates decide whether a Rule protects anything

A Rule is protection only when all three hold (tests/bdd/README.md, "When a Rule is protection"):

• Well-formed: npm run spec:audit holds every Rule to one shall, one of the five EARS patterns, a named system, no vague language, at least one Scenario under it, no Scenario outside a Rule, and a tracker tag (@esi-<bead> or @gh-<issue>) beside every @bug (TEST-02). Every feature also carries a charter tag, the id of the guides/CHARTER.md requirement it verifies (@DES-08), on the Feature or on the Rule that backs it; the tag must name a requirement the charter defines, and a feature a requirement's "Verified by" line cites must carry that id. Features no requirement covers are listed under unchartedFeatures in scripts/spec/spec-audit-exceptions.json, a shrink-only list (empty since 2026-10-12). tests/bdd/README.md rule 10 lists the tags in use.
• Executed: mkdir -p reports/bdd, npm run bdd -- --json --outputFile=reports/bdd/jest-results.json, then npm run bdd:report joins the run to the feature files. It fails when any scenario did not execute (feature-not-run, scenario-not-executed), and writes reports/bdd/junit.xml with each test case named Feature › Rule › Scenario, and reports/bdd/scenarios.json with one record per scenario: feature file and line, Feature, Rule, Scenario, the charter tags it inherits, its status (passed, failed or not-run) and the Jest test file that ran it. The bdd-tests job in ci.yml runs both and uploads reports/bdd/ as the bdd-junit artifact.
• Able to fail: npm run mutation:bdd:ratchet floors the BDD-only mutation score per source directory in config/mutation/bdd-thresholds.json: 19 floors, from 0% (src/schemas) to 66.6% (src/adapters). Every scored directory has one, and a directory without one fails the ratchet. See Where the scores stand.

npm run validate:spec-consistency (scripts/spec/rule-schema-check.ts) adds a fourth, narrower check in the spec-audit job: a Rule that names a response field the schema marks optional must say so.

npm run charter:audit (scripts/quality/charter-audit.ts) holds guides/CHARTER.md to the same form (PROC-06): every #### requirement block has one shall, names its system, follows one of the five EARS patterns with the header's pattern matching its leading keyword, and contains no vague language; a block whose status is Enforced must name, in backticks, a script (npm run <script>), a job or workflow under .github/workflows/, or a file in the repository that exists (branch protection is the one mechanism accepted by name, since it lives in GitHub's settings). Each "Verified by" opens every clause with Test: (a test under tests/), Gate: (a script, job, tool or setting that fails on a violation), Review: or To add:, and an Enforced block needs a Test: or Gate: clause. Every Jest test file a "Verified by" cites, by path or by bare file name, must name the block's ID in a describe, it or test title, as describe('SEC-03: workflow token permissions', ...) does; the audit reads the titles from the source, including describeIfGated(...) and (canRun ? describe : describe.skip)(...), but not a title .each builds from a table. A bare file name two test files share fails until the row gives the path. It runs in the same spec-audit job, in check:all and in check:local, and its checks in scripts/quality/charter-audit-core.ts are unit-tested against fixtures in tests/tdd/scripts/charter-audit.test.ts.

The standalone EARS check

npm run ears runs the first two gates on their own, outside npm test, and answers per requirement rather than per scenario. It runs the spec audit, runs the BDD scenarios with Jest's JSON output, joins the run to the feature files, and gives every Rule: one verdict:

• PASS (verified): every scenario under the Rule ran and passed.
• FAIL (failing): at least one scenario under it ran and failed.
• NOT RUN (unverified): nothing failed, but a scenario under it did not execute, so the Rule is documentation, not protection.

It exits 1 when the audit fails or any requirement is not PASS, and writes reports/ears/:

• ears-report.md: the totals, the requirements not verified and why, the exclusion register, advisory feedback on the specification (the EARS pattern mix, how many requirements rest on a single scenario, and which features state no If …, then … shall unwanted-behaviour requirement), then every requirement by feature;
• ears-report.json: the same data for tooling;
• junit.xml: the scenario-level JUnit report bdd:report writes.

Each requirement has an id, <feature file stem>#R<n> (for example 0023-market#R4), counting Rules from 1 in file order, so a verdict can be quoted and found.

The exclusion register. What the client deliberately does not do is a requirement too (CHARTER TEST-11): an exclusion is written as an unwanted-behaviour Rule whose response is negated, If <condition>, then the <system> shall not <response>., with a scenario that proves the absence. The report's "Exclusion register" section lists every such Rule with its id, verdict and the scenarios under it, and the console totals count them (Exclusions (shall not)); ears-report.json carries exclusion: true on each. The register is a listing, not a gate: it cannot know what the client should ignore, so its completeness is a review question. An exclusion stated only in prose (a guide, a code comment) does not appear and protects nothing. tests/bdd/GUIDE.md has a worked example of writing one.

npm run ears                          # everything
npm run ears -- --only=market         # features whose path contains "market"
npm run ears -- --results=<jest json> # report an existing run without re-running it
npm run ears -- --skip-audit          # verdicts only

The dashboard. npm run ears:dashboard turns the same sources into one self-contained HTML page, reports/ears/ears-dashboard.html, for a reader who wants to browse the specification rather than read 77 feature files. It has three tabs. Overview gives the totals, the EARS pattern mix, each area of tests/bdd/features/, the charter ids the features verify, and what needs attention. Requirements lists every Rule by feature, searchable and filterable by pattern, area, verdict and charter id, each with its source as written, the spec entry or step file that runs it, its verdict and the reason for it. Write a requirement explains the shape of a Rule and takes the shortest clean, verified Rule of each pattern from the specification as its example. Verdicts come from reports/ears/ears-report.json, so run npm run ears first; without a report the page shows every requirement marked "no run". The audit findings it shows on a Rule come from the spec audit's own checks (scripts/spec/spec-audit-checks.ts). The documentation site publishes it at lgriffin.github.io/ESI.ts/requirements/ on every release (npm run docs:ears, inside docs:site). --since=<ref> marks each Rule added or changed after a git ref and lists the ones removed, matched by file and title.

npm run ears && npm run ears:dashboard     # this run's verdicts
npm run ears:dashboard -- --since=v12.0.0  # plus what changed since the release

.github/workflows/ears.yml runs the same command on pull requests that touch src/, tests/bdd/ or the EARS scripts, and on demand from the Actions tab; the report and the dashboard are the ears-report artifact, and the job summary carries the report. It is advisory: bdd-tests and spec-audit in ci.yml already gate the same ground.

The BDD pattern

A Rule states the requirement; the scenario under it describes one case as an HTTP exchange. From tests/bdd/features/core/0023-market.feature:

Rule: When current market prices are requested, the Market client shall return each entry with a type_id, and a numeric average_price and an adjusted_price when present.
  The price endpoint is the cheapest way to value an arbitrary item. The two
  prices differ in meaning, so both are carried through rather than collapsed
  into one figure.

  Scenario: Price list returns average and adjusted prices per type
    Given the market system is operational
    When the client requests current market prices
    Then the client shall return price data for all tradeable items

The feature is bound by a spec entry at the same path under tests/bdd/specs/, whose whole content is bindFeature(__filename). Each step is one file under tests/bdd/steps/<keyword>/, named after its pattern, and found by its text from any feature:

// tests/bdd/steps/given/the-market-system-is-operational.ts
import { marketFixtures, marketPaths } from '../../support/market';
import { queueResponse } from '../../support/transport';
import { Given } from '../../support/steps';

Given('the market system is operational', function () {
  queueResponse({
    match: marketPaths.prices,
    body: marketFixtures.priceList(),
  });
});

// tests/bdd/steps/when/the-client-requests-current-market-prices.ts
import { When } from '../../support/steps';

When('the client requests current market prices', async function () {
  this.result = await this.client.market.getMarketPrices();
});

// tests/bdd/steps/then/the-client-shall-return-price-data-for-all-tradeable-items.ts
import { lastRequest } from '../../support/transport';
import { Then } from '../../support/steps';

Then('the client shall return price data for all tradeable items', function () {
  expect(lastRequest().method).toBe('GET');
  expect(this.result).toEqual([
    { type_id: 34, average_price: 5000000.0, adjusted_price: 5100000.0 },
    { type_id: 35, average_price: 15000000.0, adjusted_price: 15200000.0 },
  ]);
});

The Given queues what ESI sends back; the When calls the real client method; the Then asserts on the result and on what went over the wire. The transport is strict: a request with no queued response fails the scenario, and so does a queued response nobody requested. jest.spyOn on a client method is rejected by npm run lint:bdd-seam, because a scenario that mocks the method it exists to exercise cannot fail for any bug in it. The rules are in tests/bdd/README.md; the long-form walkthrough, including how to choose an EARS pattern, is tests/bdd/GUIDE.md. Agents follow the ears-gherkin-dev skill, which enforces Red before Green (TEST-01).

The legacy step-file migration

Each feature file is run by exactly one of two things, and the audit fails a feature with neither or both. Twenty-three core features are converted to spec entries and the global step library (tests/bdd/specs/core/, 23 files, plus step-library.spec.ts, the dry run); the 23 SDE features in tests/bdd/specs/sde/ were written that way. The other 31 features still run through legacy defineFeature files in tests/bdd/step-definitions/, each listed in legacyStepFiles in scripts/spec/spec-audit-exceptions.json. That list ratchets against the integration branch: it gains no entry, every entry must exist, and a legacy file not on it fails the audit. Converting a domain moves each step into its own file, moves fixtures and paths into tests/bdd/support/<domain>.ts, adds the spec entry, deletes the legacy file and removes its line from the list. The unconverted list in the same file, for features not yet in Rule form, is empty.

Type tests

Location: tests/typetests/ (7 files: index, createClient, endpoint-args, branded-ids, error-guards, result-type, domain-responses) Run: npm run test:types (tsd) CI: full-test-suite in ci.yml, through npm run test:all

Compile-time assertions on the consumer-facing type surface (TEST-06): endpoint argument inference, branded IDs, error guards, the result envelope and domain response types. Half the value of the library is at the type level, and a refactor can break inference without failing a runtime test. The tier's own signal is type mutation.

Properties and fuzz

Location: tests/fuzz/ (read tests/fuzz/AGENTS.md first) Config: config/jest/fuzz.config.cjsRun: npm run fuzz (12 suites, 957 tests, 2026-09-27); npm run fuzz:properties for the model-based ones and the SDE schema fuzz CI: fuzz-tests in ci.yml; nightly-properties.yml at 10,000 runs per property

Unit tests check the inputs someone thought of. Properties check invariants across every input fast-check generates, including the ones nobody would write by hand: an object whose toString returns an array, a string with an embedded null byte, an integer at MAX_SAFE_INTEGER. The first fuzz run found that validatePathParam throws a raw TypeError rather than an EsiError for such an object; that is still open (see Known gaps).

The model-based properties (*.property.test.ts) register through describeProperty in support/property.ts, drive real domain clients against a fake ESI at the fetch seam (support/fakeEsi.ts), and must fail against known-bad implementations registered beside them:

File	Invariant
circuit-breaker-model.property.test.ts	CircuitBreaker matches a closed/open/half-open reference model under any command sequence, including overlapping calls
pagination-assembly.property.test.ts	Eager, fetchAll* and cursor walks return every page once, in order, for any page count, failure pattern and arrival order
cache-key.property.test.ts	Cache keys are deterministic, injective, in canonical query order, and scoped to the access token
etag-cache-model.property.test.ts	The ETag cache predicts every request, If-None-Match, result and stored entry across calls, identities, faults and time
backoff.property.test.ts	retryDelay is finite, in [0, maxDelayMs] and monotonic in attempt for any jitter value

The older fuzz files are plain fast-check tests:

File	Invariant
parameter-fuzz.test.ts	validatePathParam() and validateQueryParam(): positive integers pass and return their string form; null, undefined and empty strings are rejected; strings with an unsafe path character (/\?#@!$&'()*+,;=<>{}|^` ) are rejected; NaN and ±Infinity are rejected; objects and arrays never pass
url-construction-fuzz.test.ts	buildEndpointPath(): no {param} placeholder survives substitution; slashes and the dot segments . and .. are rejected; other special characters are percent-encoded; the body is extracted
schema-fuzz.test.ts	Every Zod schema in src/schemas/: safeParse() never throws for any input from fc.anything(), and non-object primitives are rejected where an object is expected
pagination-fuzz.test.ts	The page parameter: zero, negative, float, NaN, Infinity, large and numeric-string values are accepted or rejected as the validator says
domain-property-fuzz.test.ts	Generated military-campaign and corporation-project bodies parse under their schemas
response-validation-fault-injection.test.ts	A body that violates an endpoint's responseSchema, served through the BDD transport seam, rejects with an EsiValidationError (direction: 'response', status 0, the request URL) whose Zod issues match the schema's own verdict, after exactly one request; safe mode returns it as { ok: false }; validateResponse: false returns the body unchanged
run-settings.test.ts	FC_NUM_RUNS, FC_SEED and FC_PATH fail closed: a malformed value stops the run instead of falling back to the defaults

A failing property prints the shrunk counter-example, the violated invariant, the seed, the path and the exact replay command (FC_SEED=... FC_PATH=... npx jest --config config/jest/fuzz.config.cjs --testPathPatterns ...). A counter-example is committed as a named example test in the same file before the fix; a generator or invariant is never weakened to make one go away.

Fault injection

Location: tests/faults/ (read tests/faults/AGENTS.md first) Config: config/jest/faults.config.cjs, config/jest/faults.nightly.config.cjsRun: npm run faults (2 suites, 148 tests, 2026-09-27), npm run faults:nightlyCI: fault-catalogue in ci.yml; nightly-faults.yml

This tier owns what the client does when the network misbehaves. It serves faults through the BDD transport seam, so the whole pipeline runs: rate limiter, retry, deduplication, ETag cache, JSON parsing and validation. Timers and Date run on Jest's fake clock, driven by the runner, so a 60-second rate-limit block costs nothing and elapsed time is exact.

The catalogue (catalogue.ts) is a list of named faults, each a transformation of a target's good exchange: truncated, empty or HTML bodies on a 200; the wrong Content-Type; Expires in the past or unparseable; a 304 with no ETag or no cached entry; a reset before the headers, a reset part way through the body, a stalled body, no response before the timeout; 5xx with an HTML body with and without a cached entry; 502 without a reason phrase; 400 and 500 with ESI's error JSON; 420 with and without X-ESI-Error-Limit-*; 429 with Retry-After in seconds, as an HTTP date and absent; X-Pages that grows or shrinks between pages; schema violations, null, unknown fields and schema-valid absurd values.

Every fault cites the Rule: (or guide section) that specifies the answer and asserts the whole outcome: the error class, status or code and an anchored message, or the resolved value and meta.stale; the exact request count; what the cache holds afterwards, probed through the transport; the elapsed virtual time; and every warn/error log entry. npm run faults applies the catalogue to five targets: status.getStatus (public GET), location.getCharacterLocation (authenticated GET), market.getMarketOrders (X-Pages), freelanceJobs.getFreelanceJobs (cursor) and universe.postNamesAndCategories (POST).

The signal. catalogue.selftest.test.ts holds the tier's ratchet: the number of faults without a resolvable Rule or a fully specified outcome stays 0. It proves the gate with fixtures/weak-fault.ts, which must be rejected on every count, and proves the runner by running a real fault with wrong expectations and requiring every broken invariant to be reported. Faults that expose an unfixed bug sit in known-gaps.json with their bead; the list only shrinks against origin/master (it fails closed without the ref), a listed fault that starts passing fails the run, and the list is empty today.

Nightly payload fuzz (payload-fuzz.nightly.test.ts) covers every endpoint definition with a responseSchema. zodArbitrary.ts derives fast-check arbitraries from the Zod schema; each generated body must come back unchanged, and each single-point mutation must either reject with an EsiValidationError carrying one Zod issue at the mutated path (a dropped required field, a wrong type, a null) or, for an unknown field, resolve with the field preserved. The seed is printed; FAULTS_SEED replays it and FAULTS_RUNS sets cases per endpoint (default 100). nightly-faults.yml runs it with the catalogue and keeps one issue, "Nightly fault tier failing", open while it fails.

Decisions the tier pins, so a change to them is deliberate: Content-Type is not trusted (the body decides); Expires is ignored for freshness; page 1's X-Pages is authoritative; and schema-valid but absurd values (a negative volume_remain) pass through unchanged and unlogged, because schemas check shape and ESI is the source of truth. Each is also an exclusion Rule in the register (0052-response-headers, 0050-etag-caching, 0053-runtime-validation).

Recorded replay

Location: tests/contract/replay/, fixtures in tests/contract/fixtures/recorded/, agent notes in tests/contract/AGENTS.mdConfig: config/jest/contract.replay.config.cjsRun: npm run contract:replay (5 suites, 121 tests, 2026-09-27; no network). One endpoint: npm run contract:replay -- -t "market.getMarketOrders". Record: ESI_LIVE_TESTS=true npm run contract:record [-- --only=<endpoint key>] (refuses to run without ESI_LIVE_TESTS=true) CI: contract-replay in ci.yml; nightly-recorded-payloads.yml re-records

The live contract checks the spec against the endpoint definitions. This tier checks the other truth: the bodies ESI actually sends. tests/contract/record.ts records one sanitised response per public (unauthenticated) GET endpoint definition, calling the real client method with a capturing fetch so the URL, User-Agent and X-Compatibility-Date are the client's own, then sending that request itself one at a time (stopping on a 420, waiting out a 429 once, pausing when the error limit runs low). Each fixture keeps the ETag, Expires, Last-Modified, Cache-Control, Content-Type and X-Pages headers, the compatibility date, the hash of the OpenAPI document served for it and the operation's x-cache-age. Arrays, maps and long strings are truncated, every cut is listed in truncated, paginated endpoints keep two pages with X-Pages rewritten to match and the upstream count kept in upstreamPages. Fixtures are capped at 24 KiB each and 256 KiB together (tests/contract/recorded/policy.ts); the 86 fixtures take 166 KiB (cat tests/contract/fixtures/recorded/*.json | wc -c).

Each replay goes through the BDD transport seam, so the whole pipeline runs, and checks that:

• the endpoint's Zod schema accepts the recorded body;
• the returned value keeps every key path of the recording, and a field added to the body survives validation;
• offset pagination sends one request per recorded page;
• within the recorded x-cache-age a second call is served from the cache, and after it the client revalidates with If-None-Match set to the recorded ETag.

coverage.test.ts fails when a public GET endpoint has neither a fixture nor an entry in tests/contract/fixtures/unrecordable.json, when an entry or fixture is stale, when a fixture was recorded under a different compatibility date than the client sends, and when the size budget is exceeded. unrecordable.json (one entry: dogma.getDynamicItemInfo, which needs a mutated item ID no public endpoint lists) and known-mismatches.json (empty) only shrink against origin/master, and the check fails closed when no base ref can be read. The failure signal is tests/contract/fixtures/recorded-negative/status.getStatus.json, a recording with players edited to a string, which the replay must reject with an EsiValidationError naming players.

nightly-recorded-payloads.yml re-records every night, restores fixtures whose shape did not change (npm run contract:shape-diff -- --revert-unchanged: status, which cache headers are sent, and the JSON type at each key path; values such as prices, IDs and ETags are ignored), replays the rest and opens a pull request with the shape diff and the replay result. Nothing merges automatically. A failed run keeps an issue labelled recorded-payloads-check-failed open.

Live contract

Location: tests/contract/esi-contract.test.ts, tests/contract/esi-snapshot.test.ts, helpers in tests/contract/helpers.tsConfig: config/jest/contract.live.config.cjs (extends config/jest/contract.config.cjs with a global setup that fails unless ESI_LIVE_TESTS=true) Run: ESI_LIVE_TESTS=true npm run contract:liveCI: contract-tests in ci.yml, soft-skipping on HTTP 503 (TEST-08); weekly in maintenance.yml

npm run contract uses the base config, whose suites skip without ESI_LIVE_TESTS; CI runs contract:live so that a dropped variable fails instead of passing an empty run. This is the tier that tells the project CCP moved something (TEST-05).

Deep contract (esi-contract.test.ts) extracts every endpoint definition from the *Endpoints.ts files and validates it against the live ESI OpenAPI 3.1 document in eight dimensions:

• Path parameters: {param} names in the spec path match the definition's pathParams.
• Query parameters: the spec's required: true query parameters appear in queryParams.
• Request body: spec requestBody presence matches hasBody / bodyBuilder.
• Auth: spec OAuth security matches requiresAuth.
• HTTP method: the definition's method matches the spec operation.
• Response schema coverage (advisory): GET endpoints with JSON responses have a Zod responseSchema.
• Pagination contract (advisory): spec X-Pages headers align with the pagination metadata.
• Deprecation sync (advisory): spec deprecated: true matches the definition's deprecated.

Known deviations sit in three named exception sets at the top of the file, each entry with its reason. An unknown mismatch hard-fails. The four deviations the first run found are all still listed:

Endpoint	Exception set	Mismatch
getCorporationStarbaseDetail	KNOWN_QUERY_PARAM_EXCEPTIONS	The spec requires query parameter system_id; the definition passes it in the path
createFleetWing	KNOWN_BODY_EXCEPTIONS	The definition sends an empty body; the spec has no requestBody
getCorporationHistory	KNOWN_AUTH_EXCEPTIONS	The definition sets requiresAuth: true; the spec is public
getMarketOrdersInStructure	KNOWN_AUTH_EXCEPTIONS	The spec requires auth; the definition sets requiresAuth: false

Snapshot comparison (esi-snapshot.test.ts) compares tests/contract/snapshots/esi-openapi.snapshot.json with the live spec and reports path additions and removals, schema count changes and cache TTL drift as advisory warnings.

Spec-drift tools

Tool	Command	What it does	Where it runs
oasdiff	npm run contract:diff (Docker)	Breaking changes between the committed snapshot and the live spec	Weekly, maintenance.yml
Snapshot	npm run contract:snapshot (scripts/spec/snapshot-openapi.ts)	Refreshes the committed spec baseline	Weekly, maintenance.yml, before oasdiff
Prism	npm run mock:esi (Docker)	A spec-conformant ESI mock on port 4010	Locally, and under Schemathesis
Schemathesis	npm run fuzz:api (scripts/spec/run-schemathesis.sh, Docker)	Downloads the spec, strips security requirements, serves it with Prism and fuzzes it; fails only on failures in its JUnit report	Nightly 01:00 UTC, nightly-schemathesis.yml

Schemathesis is the API fuzz tier. It fuzzes a mock generated from the spec, so it finds contradictions inside the spec, not bugs in this client, and it has no negative fixture proving it can fail. It uploads schemathesis-report, and a failed night opens or comments on the issue titled "api-fuzz: nightly Schemathesis run failed", which the next green night closes. Missing endpoints, stale generated types and schema drift are checked nightly by nightly-spec-drift.yml, described in QUALITY-GATES.md.

Integration

Integration tests live in tests/integration/ and run with config/jest/integration.config.cjs, outside npm test. npm run test:integration runs all six files: the mocked suite passes and the live ones skip unless their environment is present (6 suites, 177 tests: 20 pass, 157 skip, 2026-09-27).

File	Tests	Needs	What it covers
full-stack.test.ts	20	nothing	The full request lifecycle with mocked fetch: ETag round trip, circuit breaker trip and recovery, middleware order, 401 refresh and retry, pagination assembly, error propagation, construction patterns, per-client cache isolation
live-esi.test.ts	40	ESI_LIVE_TESTS=true	Real HTTP to 42 public endpoints across status, alliances, market, universe, dogma, wars, industry, insurance, incursions, sovereignty, route, faction warfare and killmails, plus the rate-limit headers; each checks HTTP 200 and the response shape
client-integration.test.ts	11	ESI_LIVE_TESTS=true	The real EsiClient against live ESI: ETag round trip, rate-limit tracking, multi-page assembly of universe/types, a 404, route calculation, diagnostics
esi-spec-contract.test.ts	10	ESI_LIVE_TESTS=true	Surface drift against the live spec. Phantom endpoints, method mismatches and type drift warn; cache TTL and scope drift hard-fail, because they mean the generated files need npm run generate:types
gated-auth.test.ts	33	ESI_GATED_TESTS=true and a token	Authenticated endpoints with a real OAuth token: location, skills, wallet, assets, characters, clones, contacts, killmails, mail, fittings, industry, market, loyalty, contracts, calendar, search, faction warfare
sde/sde-real-data.test.ts	114	sde-data/ on disk	The SDE provider against a real extracted Static Data Export; nightly-sde.yml runs it against CCP's current build

npm run test:integration                                  # mocked passes, live skips
ESI_LIVE_TESTS=true npm run test:integration              # live smoke, client integration, spec contract
npm run test:integration:live                             # live-esi.test.ts only; fails without ESI_LIVE_TESTS=true
ESI_GATED_TESTS=true ESI_ACCESS_TOKEN=<token> npm run test:integration:gated
npx cross-env ESI_LIVE_TESTS=true npm run test:integration   # Windows (PowerShell)

# One suite
npx jest --config config/jest/integration.config.cjs --testPathPatterns=full-stack
npx jest --config config/jest/integration.config.cjs --testPathPatterns=live-esi
npx jest --config config/jest/integration.config.cjs --testPathPatterns=client-integration
npx jest --config config/jest/integration.config.cjs --testPathPatterns=esi-spec-contract

npm run token:create writes a PKCE token to .env, which test:integration:gated reads. Live tests have 30-second timeouts and fail when ESI is down.

Where they run. The mocked suite runs on every pull request, inside full-test-suite (the test:all tiers). No workflow runs the live smoke, client integration, spec contract, gated auth or real-SDE suites; they are for a contributor to run by hand. The live checks CI does run are contract-tests on pull requests, maintenance.yml weekly, and the public examples in nightly-examples.yml.

Published results

Each CI job that runs a Jest tier also writes Jest's --json results to reports/test-results/ and uploads them, whether the tier passed or failed, so a tool can read which tests ran and how they ended rather than only the job's status (#639, #640):

Artifact	Job	Files
test-results-full-suite	full-test-suite	unit.json, bdd.json, integration.json, fuzz.json, types.json
test-results-contract-live	contract-tests	contract-live.json (written on a 503 soft-skip too, with what ran)
test-results-contract-replay	contract-replay	contract-replay.json
test-results-fuzz	fuzz-tests	fuzz.json
test-results-faults	fault-catalogue	faults.json

tsd has no JSON output, so npm run test:types:report (scripts/quality/type-test-report.ts) runs the same type tests as test:types and writes types.json: one record per tests/typetests/*.test-d.ts file, failed when tsd reports an error against it, with the diagnostics. The live tiers no workflow runs (smoke, client integration, spec contract, gated auth) show up in integration.json as skipped, which is what they are. The BDD results per scenario are in the bdd-junit artifact (see Specification).

Why three integration tiers

• Mocked (full-stack.test.ts) runs on every pull request. It is fast and deterministic and catches orchestration bugs between the client, the pipeline and the cache.
• Live (live-esi, client-integration, esi-spec-contract) catches what no mock can: URL construction against the real router, response shapes CCP changed, and real HTTP behaviour such as rate-limit headers.
• Gated auth (gated-auth) is the only tier that exercises token scoping against real authenticated endpoints. It needs a personal token, so it can never run unattended.

Why a deep contract as well as a spec contract

esi-spec-contract.test.ts checks surface drift: endpoint counts, type field names, and whether the generated TTL and scope files are stale. The deep contract checks each endpoint definition structurally: does pathParams match the {param} names in the spec path, does the definition require auth where the spec does. Those are harder failures that catch real bugs; the first run found the four mismatches in the table above.

Examples against live ESI

examples/ holds runnable scripts against the real API (npm run example:<name>; npm run help -- example lists them). nightly-examples.yml type-checks every example and runs each one tagged @nightly public or @nightly mixed against live ESI at 04:15 UTC, opening one issue per failing example. A failure Tranquility being down explains (a gateway 5xx, or a service still starting after the 11:00 UTC downtime) is a warning, not an issue, and the runner first waits for /status to answer, because GitHub can delay the schedule into the downtime. tests/tdd/scripts/examples-nightly.test.ts fails npm test when a public endpoint has no example calling it. Details in QUALITY-GATES.md.

Consumer contract

Location: tests/consumer/ (a private downstream package), driven by scripts/quality/consumer-contract.ts, with the matrix checks in scripts/quality/consumer-contract-core.tsRun: npm run test:consumer (-- --skip-build packs the existing dist/, -- --tarball <path> tests a tarball already packed, -- --typescript oldest|repo|latest|next|<version> picks the compiler, -- --keep keeps the workspace) CI: consumer-tarball packs once, then consumer-contract in ci.yml installs that tarball on four rows: Node 22 with the oldest supported TypeScript and with npm latest, Node 24 with the repository's TypeScript and with latest. Both jobs are inside ci-success. consumer-matrix-nightly.yml adds TypeScript next and current Node, release.yml runs the four rows against the tarball it signs before anything is published, and post-publish-canary.yml installs what the registry serves after a release. Negative fixtures: tests/tdd/consumer-contract/, part of npm test.

Every other tier imports from src/, so none of them sees the package a consumer installs. This one does:

1. Installs the tarball, plus typescript at the chosen version, a matching @types/node and esbuild, into a copy of tests/consumer/ in a temporary directory outside the repository, so resolution cannot fall back to the repo's node_modules.
2. Fails if the packed exports map does not list exactly the documented sub-paths (DOCUMENTED_SUBPATHS, the list in SEMVER.md), or if one is not imported by each consumer source (src/require.cts, src/import.mts, bundler/index.mts).
3. Type-checks six cells with skipLibCheck: false, so the shipped declarations are checked too. Each cell pairs an ES module or CommonJS consumer with node16, nodenext or bundler resolution, and each one includes a generated probe that imports every documented sub-path. The ES module probe also fails if a namespace type has a default member, which is what CommonJS declarations behind the import condition produce (esi-23g.29). Every cell runs, and the step summary shows the table.
4. Loads every documented sub-path through require and import on the running Node, then runs the emitted CommonJS and ES module consumers: a real EsiClient against a stubbed fetch, a malformed body rejected with EsiValidationError, and a 404, both recognised by the classes and guards imported from ./errors. It also runs schemas, TestDataFactory and the SDE providers.
5. runtime/parity.mjs loads every sub-path under both require and import. It fails if the CJS and ESM builds export different names, or if two sub-paths in one build export the same name as different values. A class exported from . and ./errors must be one class, and the root schemas namespace is compared with ./schemas.
6. Bundles import { isEsiError } from '@lgriffin/esi.ts/errors' with esbuild. The bundle must not contain an endpoint path from the root entry, must not include dist/index.mjs, and must stay under 1,600 B. A bundle of EsiClient from . is the control and must contain both.
7. runtime/sde-optional-peers.mjs covers js-yaml and adm-zip, the optional peer dependencies of ./sde. Steps 2 to 6 run without them installed. Before that step, absent checks that ./sde loads and that fromDirectory and fromZip throw an SdeError naming the missing package. At the end the runner installs both, and present loads real YAML and ZIP files through the CJS and ESM builds.

The fixture suite writes small dual packages into a temporary node_modules and runs the same cell, runtime and tree-shaking checks against them with the repository's TypeScript and esbuild. The correct package must pass every check. Each broken package must be rejected by the check that owns its defect:

• A documented sub-path missing from exports: the exports check, every cell and the runtime probe.
• A .d.mts with an extensionless relative specifier: the node16 and nodenext ES module cells.
• import types pointing at .d.ts in a "type": "commonjs" package: every ES module cell.
• require types pointing at .d.mts: the node16 CommonJS cell.
• A light entry re-exported through the heavy one: the tree-shaking check.

Defects the contract finds are recorded as known issues against their beads: each logs while it reproduces and fails the run once it stops, so the fix has to remove the workaround. There are none open; esi-v2s.15 (error class identity across sub-paths) and esi-v2s.16 (./sde peer dependencies) were the last two.

Documentation examples

Location: scripts/docs/doc-examples.ts and scripts/docs/doc-examples-core.ts; prelude and stub fetch in tests/doc-examples/; self-tests and fixtures in tests/tdd/doc-examples/Run: npm run test:docs-examples (-- --skip-build packs the existing dist/, -- --keep keeps the workspace) CI: doc-examples in ci.yml, Node 22, inside ci-success. Not part of npm test; the unit suite checks the annotations, the baseline and the fixtures against a stub package.

Packs the library as the consumer contract does and type-checks every fenced ts/typescript block in README.md, guides/*.md, guides/sde/*.md and src/sde/README.md as its own module under nodenext and bundler resolution, then runs the blocks marked runnable against a stubbed fetch. A contributor snippet that imports from src/ or tests/, like the ones in this guide, carries <!-- doc-example: no-check <reason> --> on the line above its fence. The convention, the prelude and the shrink-only known-broken baseline (scripts/docs/doc-examples-baseline.json, empty) are in DOCUMENTATION.md.

Benchmarks and the heap soak

Location: tests/benchmark/ (see tests/benchmark/AGENTS.md) Run: npm run benchmark, npm run bench:ab, npm run bench:compare, npm run soakCI: benchmarks in ci.yml (only when a hot path changes); nightly-benchmarks.yml

This tier owns one failure class: the client got slower, or started holding memory. Bundle size is owned by the size-limit budgets.

The Jest benchmark suites that used to live here asserted raw wall-clock upper bounds (expect(elapsed).toBeLessThan(500)) that sat 10 to 100 times above the real cost. A bound that loose cannot see a 30% regression, and a tighter one flakes on a shared runner, so they were retired rather than converted. No test asserts a latency budget. The specification's performance feature keeps only the time bounds that separate overlapping requests from serial dispatch, each at or above half the serial figure.

Micro-benchmarks. tests/benchmark/tasks.ts holds 24 tasks: 18 over the paths a client pays for on every call, and six SDE tasks from tests/benchmark/sde.bench.ts (below). The client tasks: parse and Zod-validate a small object, a 1000-order market page and a nested colony layout; ETag cache hit, miss, write and write-at-capacity; cache-key derivation; the spec TTL lookup; response-header parsing (ETag, Expires, X-Pages, rate-limit headers); the rate limiter's acquire; the circuit breaker's check; batchFetch; and two whole-pipeline requests against an instant transport. harness.ts runs them with mitata, which batches fast operations so a 50 ns call is not lost in timer resolution, forces a collection before each task, and reports per-sample nanoseconds.

Comparison. A benchmark number alone means nothing; the question is "slower than what". npm run bench:ab bundles the harness for two trees, typically a pull request's base tip and its head, and runs them in alternating processes on one machine, so runner-to-runner noise cancels. The observation per task is one per-process median; npm run bench:compare applies a one-sided Mann-Whitney U test per task, Holm-adjusted across tasks at alpha = 0.05, and calls a task regressed only if the ratio of medians is also at least 1.10 and the absolute difference at least 2 ns/op. Improvements are reported. A missing baseline, too few rounds or a dropped task fails closed. The decision logic is unit-tested against synthetic distributions in tests/tdd/benchmark/.

SDE tasks. tests/benchmark/sde.bench.ts writes a seeded export of 50,000 types (SDE_BENCH_TYPES changes the count; SDE_BENCH_DIR points at a real export instead) as CCP-shaped YAML once per process and times SdeDataProvider.fromDirectory, getType by ID, getTypesByGroup with the foreign-key index built and cold (the index cleared before each call), and searchTypesByName with a fragment many names share (stops at the limit) and with one whole name (scans the table). Task names carry the data set's label, so the synthetic series and the real export never share a name in a comparison.

Heap soak. tests/benchmark/soak.ts drives 100,000 requests through a real EsiClient against an in-process transport, with a bounded ETag cache and a distinct key per request. Under --expose-gc it forces a full collection at 50 sample points and fails when a least-squares fit over the second half projects growth beyond the threshold, when the cache exceeds its bound, or when timers or process listeners survive shutdown(). npm run soak -- --inject-leak --expect-fail runs the same soak with a response interceptor that retains every response and passes only if the leak is flagged; the unit suite runs that fixture too, so the detector is checked on every pull request.

SDE soak. tests/benchmark/sde-soak.ts (npm run soak -- --sde) loads the export, answers 100,000 lookups in a fixed mix (eight in ten getType, one in ten a group or category, one in ten a foreign-key list, one in a hundred a name search), closes the provider and repeats, three cycles by default. After each close() and two full collections the heap must be back within the larger of 8 MiB and 5% of the loaded heap of the baseline measured before the first load, and the last cycle's loaded heap must not sit above the first's by more than that, so a provider that keeps a previous export alive across reloads fails. The table records load time, heap and RSS after load, peak heap during the lookups and heap after close per cycle; the verdict is unit-tested on synthetic runs and the driver on a small generated set in tests/tdd/benchmark/sde-bench.test.ts.

Where it runs. The benchmarks job in ci.yml runs the A/B comparison (10 rounds) only when src/core/, src/schemas/, the harness, the bench scripts or the lockfile change, and otherwise reports success with a summary line. nightly-benchmarks.yml compares master with a pinned reference commit (15 rounds), runs the soak, publishes the trend to the bench-data branch, and keeps one performance-nightly issue open while either fails. The SDE tasks ride both, on the generated set; nightly-sde.yml's performance job runs them and the SDE soak against CCP's current export (two rounds) and publishes both tables to the step summary, without a comparison, because those numbers describe a build rather than a commit.

Mutation testing

Coverage says a line ran. Mutation says a test would notice if the line were wrong. Stryker makes small changes ("mutants") to the source and checks whether any test fails. A test that fails kills the mutant. A mutant every test passes survived: a bug the suite would ship. A suite can reach 100% line coverage without asserting anything; mutation is the signal that it asserts, and the tier that proves the others can fail. Three runs, each held by a one-way ratchet:

Run	Command	Scope	Floors	Where
Unit	npm run mutation:pr (PR), npm run mutation (nightly)	src/core/** and src/sde/**, minus endpoints, interfaces and test fixtures	config/mutation/unit-thresholds.json, 11 directories	PR mutation-pr (changed files); nightly, 6 shards
BDD-only	npm run mutation:bdd	all of src/, step definitions as the only tests	config/mutation/bdd-thresholds.json, 16 directories	Nightly, 32 shards
Type	npm run test:type-mutation -- --ratchet	the built dist/**/*.d.ts	config/mutation/type-thresholds.json, 6 entry points	Nightly

Every pull request first runs npm run mutation:fixture, which must see the known-weak fixture in tests/mutation-fixture/ leave survivors and lose at least one mutant, so the gate can still go red on a run that measures nothing. A floor may only rise; lowering one fails mutation-pr in the pull request that tries. Type mutation is described at the end of this section; the rest covers the two Stryker runs: how the pull request run picks its files and baseline, why the nightly is sharded, how the floors were seeded, where the scores stand, and how to kill a survivor.

Running it

npm run mutation             # Full unit-suite run over the mutate scope
npm run mutation:ratchet     # Score reports/mutation/mutation.json per directory
npm run mutation:pr          # What CI runs on a pull request: only your changed src/ files
npm run mutation:fixture     # The tier's self-test on a known-weak fixture
npm run mutation:bdd         # BDD-only run (see below); one shard with BDD_MUTATION_SHARD=<name>
npm run mutation:bdd:merge   # Put the shard reports back together for the ratchet
npm run mutation:bdd:ratchet # Score the BDD-only report per directory

Reports are written to reports/mutation/ (mutation.html, mutation.json and, with --incremental, stryker-incremental.json).

Where mutation testing runs

Where	What	Gate
Pull request (mutation-pr.yml)	Known-weak fixture, then the changed src/ files in scope, incrementally	Advisory since 2026-09-27 (its own check, outside ci-success, until the target floors hold, after 11.0.0; ROADMAP.md): touched directories vs their floors, and the fixture. A cold run that times out warns
Nightly (nightly-mutation.yml)	Every file in scope (--incremental --force), one job per shard in config/mutation/unit-shards.json	Fails the run: every directory vs its floor, scored on the merged report
Nightly, BDD-only matrix	All of src/, BDD step definitions only, one job per shard in config/mutation/bdd-shards.json	Fails the run: every scored directory vs its floor in config/mutation/bdd-thresholds.json

The pull request job owns "this change weakened the tests of the code it touched". The nightly owns everything a pull request cannot see: test-only changes, merges that interact, and directories no pull request touched.

Pull requests

npm run mutation:pr (scripts/mutation/mutation-pr.ts) does, in order:

1. Base. MUTATION_BASE_REF (CI sets HEAD^1, the base tip of the pull request merge commit, with fetch-depth: 2), otherwise the merge base with origin/master or master. If none resolves, it fails closed (exit 2).
2. Ratchet direction. config/mutation/unit-thresholds.json is compared with the base copy. Raising or adding a floor passes; lowering or removing one fails (exit 1). A missing or unparsable head file, or an unparsable base copy, fails closed. The only case with nothing to compare is a base commit that predates the file.
3. Plan. git diff --name-only --diff-filter=d <base> -- src/, intersected with the mutate patterns in config/mutation/stryker.config.mjs. If nothing is left, the job summary says "Skipped" with the reason (no src/ change, or only files outside the scope) and the step exits 0. The job still ran the thresholds check and the fixture, so it reports success honestly rather than being skipped, which ci-success counted as a failure while the job was inside it and will again once the release gate moves it back.
4. Run. stryker run config/mutation/stryker.config.mjs --incremental --mutate <files>. The files are the changed ones plus any file in the same score directories that the restored nightly report cannot vouch for: absent from it, or with a source that has changed since. Stryker drops (rather than re-runs) stale mutants in files outside --mutate, so without this a directory score would silently shrink. With no restored report, every file in the touched directories is mutated from scratch: slower, never wrong. When the pull request changes or deletes a test file (*.test.ts), the run drops from the restored report every mutant whose verdict rests on that file: killed by one of its tests, or covered by one and survived or timed out. Stryker runs exactly those again and reuses every other result, so a weakened or deleted test lowers a score and a strengthened one raises it from the pull request, and a one-test change to a hot-path file no longer re-runs the whole file (#380, #571). When it changes any other file under tests/ (a helper, setup file or fixture, which can change what any test does), or the report records no per-test results, every reused file whose mutants a changed test covered is mutated again with --force, as before. A test the report does not know (a new file, or coverage it newly gained) can only raise a score, and the next nightly records that; re-mutating whole directories for it does not fit the job's time budget. A report without per-test coverage retests every reused file of the touched directories. A test-only pull request mutates the directories its tests reach: those the restored report shows the test covering, and for a test under tests/tdd/ the directory its path mirrors; one the run can tie to no directory, or whose tests covered nothing in the report, is reported and skipped.
5. Gate. Each touched directory is scored from the merged report (fresh results for changed code, nightly results for the rest) and fails if it is below its floor, or has no floor at all. The job summary lists the directory table, per-file scores for the changed files, and every surviving or uncovered mutant in them with its line, mutator and replacement. The HTML and JSON reports are uploaded as mutation-pr-report.

Run it locally the same way; it diffs your working tree (committed or not) against the merge base with master:

npm run mutation:pr -- --concurrency 4

To reuse a nightly baseline locally, download the mutation-report artifact from the latest nightly and put its stryker-incremental.json in reports/mutation/. Stryker's incremental mode keys a mutant's result on the source and the covering tests' names, so a test you strengthen under the same name would reuse the old Survived verdict on a plain --incremental run; mutation:pr passes --force for you when the diff touches tests/, and a direct npm run mutation -- --incremental --force --mutate <file> does the same by hand.

When there is no baseline

npm run mutation:pr:gate runs the check under a deadline and decides what the outcome means, because two very different things look alike from outside: a run that measured a regression, and a run that measured nothing.

Outcome	With the nightly baseline	Without it
A directory below its floor	blocks	blocks
The check cannot read its thresholds or base ref	blocks	blocks
The run does not finish in time	blocks	warns

A cold run mutates every file in the touched directories, not just the changed ones. #355 hit an eight-minute wall at 48% of 589 mutants, having killed every one it reached. Nothing was wrong with the change, and failing the pull request for it teaches people to read a red mutation job as noise. A timeout with the baseline restored still blocks: that run should only have had the changed files to mutate, so it is broken rather than slow.

The tier keeps a hard signal either way. npm run mutation:fixture runs first in the same job and fails when the known-weak fixture stops leaving survivors, so the gate can still fail even on a run that measures nothing. The policy is classifyPrMutationRun in scripts/mutation/mutation-ratchet-core.ts, pinned by tests/tdd/mutation-ratchet/mutation-pr.test.ts; esi-23g.52 covers why a baseline can be missing in the first place.

When a runner is reclaimed

GitHub-hosted runners intermittently drop a long job with The runner has received a shutdown signal and exit 143. On this repository: the unsharded BDD run at 30 minutes, the core-pipeline shard at 36, the core-rest shard at 45, while core-rest had itself finished at 59 minutes earlier the same day. (core-pipeline and core-rest are the names of BDD shards on 17 and 18 September 2026; they have since been split into the shards config/mutation/bdd-shards.json lists today.) It is random rather than a length limit, and nothing inside the job can defend against it: the runner goes, not the process.

Because the merge refuses an incomplete set, one reclaim costs every shard's score. Two things blunt that:

• Shorter shards. A reclaim is roughly proportional to how long a job runs, so splitting the longest ones makes each loss smaller and each re-run cheaper. Between 4 and 7 October 2026 a 37 to 138 minute shard was reclaimed on both attempts most nights, so on 7 October the BDD run was re-balanced to 32 shards that each aim to finish inside 30 minutes, packed by mutant count weighted by each area's observed minutes per mutant (the method is in the $balance note of config/mutation/bdd-shards.json). A shard may name single files as well as directories; each area keeps a catch-all shard under its old name that excludes the files split out of it, so a new file is still mutated. src/core/rateLimiter/RateLimiter.ts alone is about 400 mutants and stays one longer shard.
• nightly-mutation-retry.yml. On workflow_run, if the nightly finished as a failure on its first attempt, it re-runs the failed jobs once. gh run rerun --failed re-runs their dependents too, so the merge and the ratchet run again with the artifacts the surviving shards already uploaded, which persist across attempts of one run.

It has to be a separate workflow: a job inside a run cannot re-run its own run. It is bounded to one extra attempt, because a shard that fails twice is not a reclaimed runner and the second failure should be read.

Why the unit run is sharded

A single job over all of src/core took almost exactly two hours on 14, 15 and 16 September 2026 (119m40s, 120m08s, 119m26s) and then stopped finishing inside its 240-minute timeout. Nothing about the mutants changed: the unit suite grew from ~4,957 tests to 6,468, and with coverageAnalysis: perTest every added test slows every mutant. A nightly that never completes scores nothing and publishes no baseline, which is how the pull request gate came to mutate from scratch and time out as well (esi-23g.52).

config/mutation/unit-shards.json splits src/core five ways, balanced by mutant count, and gives src/sde a shard of its own (Track S Run 2), so the SDE is scored, ratcheted and restored on its own. Counts are a property of the source and the mutator config rather than of the test suite, so these are the same numbers the BDD shards use:

Shard	Directories	Mutants
core-backoff	rateLimiter, circuitBreaker	594
core-request-pipeline	requestPipeline	503
core-root	src/core itself, and endpoints, which the globs exclude	453
core-cache	cache, pagination, middleware	448
core-support	logger, util	387
sde	src/sde, ingestion included; the test-data factory excluded	1,317

The core shards sum to 2,385, which is what the unsharded run over src/core instrumented when they were balanced; with sde the unsharded run instruments 3,702. The arithmetic is the check that nothing fell between them.

UNIT_MUTATION_SHARD=<name> npm run mutation runs one, writing to reports/mutation/shards/<name>/. npm run mutation:unit:merge puts them back together for the ratchet and refuses a run with a shard missing, empty or overlapping another. tests/tdd/mutation-ratchet/unitShards.test.ts asserts the shards partition src/core, so a regrouping cannot drop a directory: orphaning src/core/endpoints fails 46 of its cases.

Each shard saves its own incremental file under stryker-unit-shard-<name>-…. A shard's file is a whole baseline for its own directories and knows nothing about the others, so the pull request job restores one only when it covers every file the run may mutate: npm run mutation:pr:shard plans the run before any baseline exists, names the one shard that claims all of those files (baselineShardFor in scripts/mutation/mutation-merge-core.ts), and mutation-pr.yml restores that shard's cache and copies it to reports/mutation/stryker-incremental.json. The common pull request touches one directory and gets a warm run whose timeout fails the job. A change that spans shards (say src/core/RetryStrategy.ts and src/core/cache/) restores nothing and runs cold, which warns rather than blocks on a timeout: restoring one shard there would claim a baseline the run only half has (esi-23g.55).

The incremental baseline

Each nightly shard runs npm run mutation -- --incremental --force: every mutant runs, and Stryker also writes an incremental file, which the job saves with actions/cache/save under stryker-unit-shard-<name>-<sha>-<run id>.

mutation-pr restores the one shard's file that covers every file it will mutate, preferring the entry saved for the pull request's base commit, then the newest for that shard (see Why the unit run is sharded for how the shard is chosen). When no single shard covers the run, nothing is restored and the run is cold.

Caches written on master are readable by pull requests into master. Pull requests never save one.

Stryker reuses a result only when the mutant's code is unchanged and, for a killed mutant, its killing test is unchanged, or, for a survivor, no test was added. The older the baseline, the more mutants re-run: a one-day-old baseline on an active branch reused 51 of 169 mutants in ETagCacheManager.ts.

That rule has a blind spot in the local loop. Stryker treats a test as new only by its name, so strengthening an existing test (same name, a sharper assertion) does not invalidate a survivor it now kills: the next --incremental run still reports the mutant as Survived. In #375, AsyncPaginationIterator.ts:17 (body !== undefined becoming true) stayed Survived after toEqual([]) became toHaveLength(0), although applying the mutant by hand showed the edited test killed it. It fails safe (the score reads low, never high), but it sends people after mutants that are already dead. Before you chase a survivor you believe a strengthened test kills, re-run that file with --force, which runs every mutant in the --mutate files and ignores the incremental results for them (#380):

npm run mutation -- --incremental --force --mutate src/core/pagination/AsyncPaginationIterator.ts

The nightly always runs with --force, so it clears such stale survivors every night. A pull request that only strengthens tests can still see its directory score read low until then, the same fail-safe way.

The ratchet

The unit floors live in config/mutation/unit-thresholds.json, one per score directory: src/core for files directly in core, src/core/<sub> below it (the same directoryOf as the BDD ratchet in scripts/mutation/mutation-ratchet-core.ts). Values are Stryker's mutation score (detected / (detected + undetected)), rounded down to one decimal.

• A pull request that raises a directory's score leaves its floor alone; a follow-up raises it once two nightlies have measured the new tests (npm run mutation:ratchet -- --update --also <earlier report>, see "Re-seeding" below). The pull request gate cannot prove the raise itself: it re-mutates only the changed src/ files and keeps the nightly's results for the rest, so a directory improved by new tests alone is still scored on the survivors those tests kill. Re-mutating the whole directory instead does not fit the pull request budget for code that everything imports: on #374, src/core/logger's mutants are each covered by thousands of tests and the run timed out. State the new score in the follow-up's body in one line; ratchet bumps without a reason are how ratchet fatigue starts.
• A floor never goes down. If a change truly has to lower one (for example, deleting dead code that only had killed mutants), that is a reviewed exception in its own pull request, and it fails mutation-pr there on purpose.
• A new directory in scope must arrive with its floor; the failure message says which value to add.
• npm run mutation:ratchet -- --update (after a full npm run mutation) raises every floor to today's score and adds missing ones. It never lowers a floor.

Seeding. Each floor is the lower of two real runs: a full local run of this branch's base (npm run mutation -- --incremental --force, concurrency 4, 54 minutes) and the last nightly that produced a report (16 September 2026, bc563d4d, run 35067980626). They agree within two points for most directories but not all: src/core/logger scored 55.5% on the nightly and 33.3% locally, and src/core/rateLimiter 73% against 70.9%, mostly because a mutant that times out counts as detected, and how many time out depends on the machine. Taking the lower value means the first pull request to touch a directory is not failed by that spread. The nightly raises nothing on its own; floors move up only in reviewed pull requests.

Re-seeding, 19 September 2026. The floors were raised from the first two complete sharded nightlies (runs 35352679266 and 35428381994), which differ only in a two-line change to src/core/constants.ts. Taking the lower of the two scores was not enough: 43 mutants changed status between the runs on the same code, 14 of them in src/core/cache, mostly a mutant that timed out on one runner and survived on the other. So each floor is the score of a report in which a mutant counts as detected only if both runs detected it. That is below either run's own score, which is the point: a third run on the same code cannot score below it unless a mutant neither run left alive survives. No floor was lowered; src/core stays at 81.8% where that rule gave 81.7%.

Timing tests made deterministic (#382). The flips traced back to unit tests that measured real time: setTimeout sleeps before checking a cache entry had expired, circuit breaker cleanup that passed or not depending on whether a millisecond ticked between two calls, and retry and rate-limit waits that were slept through for real and then bounded loosely (toBeGreaterThanOrEqual(40) for a 50ms delay). Those tests in tests/tdd/core now run on Jest's fake clock (jest.useFakeTimers(), or useFakeDate() from tests/tdd/helpers/fakeDate.ts where the whole request pipeline runs and only Date should be frozen) or record sleep() calls with Math.random pinned, and each asserts the exact delay or boundary: an entry is fresh at its TTL and expired one millisecond later, a circuit is open at resetTimeoutMs - 1 and half-open at resetTimeoutMs, a back-off is [10, 20, 40], not "positive". A mutant of one of those comparisons or delays is now killed by an assertion on every machine rather than by a timeout on a slow one. New tests of timing code follow the same rule (see Time and randomness): no real sleeps, no Date.now() bounds.

Re-seeding once the timing tests have landed. This step needs real nightlies and has not been done yet; the floors above are still the 19 September both-runs values. With the time-dependent mutants deterministic, two nightlies on the same src/core should agree mutant for mutant, and then detectedByEveryRun stops being what holds a floor down: seeding from one run with npm run mutation:ratchet -- --update gives the same floor as seeding from two with --also. To confirm that before relying on it:

1. Wait for two complete sharded unit nightlies after the change, with no src/core change between them. Download each run's shard reports and merge them (npm run mutation:unit:merge) into two mutation.json files.
2. Run npm run mutation:ratchet -- --update --also <earlier report> against the later report, as on 19 September. Count the mutants whose status differs between the two reports; the issue's 43 is the number to beat, per directory as in the table below.
3. If the gap between each directory's score and its new floor is near zero, record the new floors and the flip count here in one line, and later re-seeds may use a single complete nightly without --also. Directories that still flip keep the two-run rule, and their flipping mutants are the next tests to make deterministic.

Where the scores stand

The SDE directories were seeded from local runs rather than a nightly. Track S Run 2 (2026-09-27) measured src/sde 87.6% and src/sde/ingestion 73.8% on the unit suite (1,317 mutants) and 22.6% / 0% BDD-only. Run 12 (2026-09-28) moved the files into ports/, domain/, providers/, ingestion/ and testing/, and re-measured: unit src/sde 93.7% (15/16), src/sde/ingestion 73.8% (161/218) and src/sde/providers 89.5% (86/96) over 838 mutants; BDD-only, once #523 let the dry run find the step-library scenarios, src/sde 31.2% (5/16), src/sde/ingestion 58.0% (133/229) and src/sde/providers 58.5% (58/99) over 1,007 mutants. src/sde/domain holds only type declarations and schemas and yields no valid mutants; src/sde/testing is excluded from both runs like src/testing. Those seeds sat two points under the measured scores: unit 85.6 / 71.8 / 87.5, BDD 29.2 / 56.0 / 56.5. Track S Run M (#458, 2026-10-07 and 08) then took every SDE directory past 90 on both runs, one directory per pull request, with a verdict for every survivor: killed by a new test or scenario, or marked // Stryker disable next-line with the reason it is equivalent. Unit: src/sde/providers 100% (92/92, #607), src/sde/ingestion 100% (184/184, #608), src/sde already 93.8% (15/16). BDD-only: src/sde/providers 90.5% (86/95, #610), src/sde 100% (15/15, #611), src/sde/ingestion 97.6% (162/166, #612). The floors now stand at unit 85.6 / 98 / 98 and BDD 93 / 97 / 90 (src/sde, ingestion, providers); the unit src/sde floor keeps its seed until a nightly raises it, because Run M changed no unit test there. Those scores were measured locally with the nightly's own configs, and the table below, which lists the nightly-measured core, does not carry them; sde/TESTING.md does.

The first two complete runs of the sharded unit matrix: 18 September 2026 (run 35352679266) and 19 September 2026 (run 35428381994). Five shards, 38 to 82 minutes each, merged into one report of 37 mutated files. The 18 September run was the first time src/core had been scored since 16 September: the unsharded job stopped finishing inside its 240-minute timeout as the suite grew.

Directory	18 Sept	19 Sept	Detected / valid (19 Sept)	Floor	Floor before
src/core	83.3%	83.3%	215/258	81.8%	81.8%
src/core/cache	73.3%	70.1%	87/124	65.3%	53.3%
src/core/circuitBreaker	77.7%	79.3%	100/126	77.7%	75.2%
src/core/logger	59.2%	62.9%	17/27	55.5%	33.3%
src/core/middleware	100%	100%	26/26	100%	92.3%
src/core/pagination	73.9%	73.9%	88/119	73.9%	72.2%
src/core/rateLimiter	78.3%	79.4%	213/268	77.9%	70.9%
src/core/requestPipeline	77.6%	76.0%	184/242	75.2%	67.5%
src/core/util	96.1%	95.4%	148/155	95.4%	95.3%

A snapshot, not a source of truth: the live numbers are whatever the last nightly published, and this table is here to say where the floors sat relative to reality when the matrix first worked.

The gap between a directory's score and its floor is now the flakiness of its mutants, not slack. Before the re-seeding, four directories sat 7 to 26 points above their floors, a measurement the ratchet was not holding: a change could have given back twenty points of src/core/cache and no gate would have noticed. What is left between score and floor is the mutants that are detected on one runner and not another. Killing those deterministically (fake timers and an exact assertion, rather than a timeout that depends on machine speed) is what lets a floor rise to the score. The timing tests named in #382 have moved to fake timers (see "Timing tests made deterministic" above); the re-seed that measures the effect is still to run.

Two caveats on comparing this table with an earlier one. A mutant that times out counts as detected, and how many time out depends on the machine, so a local run and a nightly disagree by a point or two on the same code (the seeding note above records src/core/logger at 55.5% nightly against 33.3% locally for exactly that reason), and two nightlies disagree too. And these are five separate shards merged, so each directory's score comes from the one shard that owns it rather than from a single process.

The BDD-only floors in config/mutation/bdd-thresholds.json come from the same two nights by the same rule. 19 September was the first run in which every BDD shard finished; on 18 September the core-rest shard (since split into core-support and core-root) lost a runner and the merge refused the incomplete set, but the seven shards that did finish still count, so a mutant any of them left alive counts as undetected. Most directories were therefore measured once, and BDD-only scores are low by design: the scenarios exercise behaviour through the transport seam, not every branch. The 19 floors today run from 0% (src/schemas, where a schema declaration has little for a scenario to kill) and 11.7% (src/core/rateLimiter) to 42.8% (src/core/util) on the core; the SDE's three run from 90% to 97% since Run M, because its scenarios drive the providers and the ingestion pipeline directly rather than through a transport. src/adapters (66.6%) and src/client (40.9%) were added at the scores the 29 September nightly measured; cat config/mutation/bdd-thresholds.json lists them.

The known-weak fixture

tests/mutation-fixture/weakClamp.ts is a clamp function whose fixture test only checks an in-range value. npm run mutation:fixture mutates it with config/mutation/stryker.fixture.config.mjs and fails unless the report shows at least one killed mutant (the run can detect a fault), at least one survivor (a weak test is visible), and a ratchet failure for it against a 100% floor. mutation-pr runs it before the real run, so every pull request proves the pipeline can still go red. Do not strengthen that test. The last local run: 5 killed, 4 survived, 2 without coverage, score 45.4%, 15 seconds. Pointing the fixture's Jest roots at the original tree instead of the sandbox, the same mistake the module mapper used to make, turns that into 0 killed and 11 survived, and the check exits 1 with "no fixture mutant was killed".

tests/tdd/mutation-ratchet/ holds the unit tests for the ratchet itself: a directory below its floor fails, a missing or unreadable thresholds file or base ref fails closed, a lowered or removed floor is rejected, a pull request with no in-scope src/ change skips, and the plan widens to whole directories when the baseline cannot vouch for a file.

Sharding the BDD-only run

One job mutating all of src/ against the BDD suite alone does not finish: 4,495 mutants, and the only attempt was killed at 30 minutes, which is why config/mutation/bdd-thresholds.json was empty for as long as it was. nightly-mutation.yml therefore runs one job per shard in config/mutation/bdd-shards.json; BDD_MUTATION_SHARD=<name> npm run mutation:bdd mutates that shard alone and writes reports/mutation-bdd/shards/<name>/.

A split run only means the same thing as the single run it replaces if nothing falls between the shards, so two checks hold it together:

• tests/tdd/mutation-ratchet/bddShards.test.ts asserts the shards partition src/: every TypeScript file belongs to exactly one. src/sde has a shard of its own since Track S Run 2 (1,536 mutants), so the SDE's BDD scenarios are scored, ratcheted and restored apart from rest. A file claimed by none is never mutated and its directory's score quietly improves; a file claimed by two is counted twice. The exclusions in config/mutation/stryker.bdd.config.mjs are shared by every shard, so the union of the shards mutates exactly what the unsharded glob did.
• npm run mutation:bdd:merge (scripts/mutation/mutation-merge-core.ts) refuses to merge a run with a shard missing, a shard that mutated nothing, or two shards reporting one file. Without that, a shard whose job died would leave its directories scored on whatever else ran, which reads as a pass.

Seed the floors from a completed run: dispatch the workflow with seed_bdd_thresholds, which prints and uploads config/mutation/bdd-thresholds.json raised to that run's scores. Nothing commits it; --update never lowers a floor.

Why the BDD-only run is not on pull requests

Running the step definitions as the dry run for a src/core/requestPipeline change would take most of the 12-minute pull request budget on its own, and there is no incremental baseline to restore until the nightly matrix has published one. Once it has, a pull request run scoped to src/core/requestPipeline is the natural next step.

How it works

1. Instrumentation: Stryker parses all files matching the mutate glob and identifies possible mutations (2,385 in the unit scope when the shards were balanced on 18 September 2026; see Why the unit run is sharded).
2. Dry run: Stryker runs the tests once to establish a baseline and map which tests cover which code. The unit run narrows the dry run to the tests Jest's module graph relates to the mutated files (enableFindRelatedTests); the BDD run cannot, because the step-library specs bind their steps at run time through bindFeature(__filename) and relate to nothing statically (the sde shard found no tests at all once Run 3 converted the SDE steps), so it runs every BDD spec and lets per-test coverage do the narrowing.
3. Mutation: For each mutant, Stryker modifies the source and runs only the tests that cover the changed code (perTest coverage analysis). The TypeScript checker discards mutants that do not compile first.
4. Scoring: Each mutant is classified:

Status	Meaning
Killed	A test failed: the mutation was detected
Survived	All tests passed: a potential blind spot
Timeout	A test timed out, likely an infinite loop; counts as detected
NoCoverage	No test executes this code path
CompileError	The TypeScript checker rejected it; not counted
Ignored	Excluded mutator (StringLiteral) or static code; not counted

Configuration

Config files: config/mutation/stryker.config.mjs (unit suite, nightly and pull requests), config/mutation/stryker.bdd.config.mjs (BDD-only), config/mutation/stryker.fixture.config.mjs (known-weak fixture).

Scope

The unit run mutates src/core/**/*.ts and src/sde/**/*.ts with these exclusions:

• src/core/endpoints/**: endpoint definitions are data declarations, not logic
• Interface-only files (ILogger.ts, ICache.ts, IRateLimiter.ts, IRetryStrategy.ts, ICircuitBreaker.ts, IDeduplicator.ts) and the requestPipeline barrel and dependency wiring
• src/sde/testing/**: SdeTestDataFactory, a test fixture shipped for consumers' tests; mutating it would score the fixture, not the module

Files NOT in scope (and why):

Excluded	Reason
src/clients/**	Thin delegation layers tested via BDD scenarios
src/schemas/**	Zod schema declarations, no branching logic
src/types/**	Type-only files, no runtime code
src/config/**	Configuration setup, not core logic
src/EsiClient.ts, src/EsiClientBuilder.ts	High-level orchestration tested via integration tests
*.generated.ts	Auto-generated from OpenAPI spec

A pull request that changes only out-of-scope src/ files skips the mutation step and names those files in the summary.

Thresholds

The unit config has no global break; the per-directory floors in config/mutation/unit-thresholds.json are the gate. high: 80 and low: 60 only colour the HTML report.

A directory is src/<area> for most of the tree and src/core/<sub> or src/sde/<sub> inside the core and the SDE, whose parts differ enough that one number would hide a weak one (directoryOf in scripts/mutation/mutation-ratchet-core.ts). The SDE floors were seeded from local runs (2026-09-27, Track S Run 2) and raised by Track S Run M (2026-10-08) to just under the local scores its pull requests measured with the nightly's configs; the nightly ratchet raises them further and never lowers one. Fifteen SDE mutants crash the test runner rather than fail a test (a missing-file check removed, a transform recursing on itself) and count as RuntimeError, which no score counts.

Sandbox and module resolution

Stryker runs tests in an isolated sandbox (.stryker-tmp/sandbox-XXX/) containing mutated source files. Since tests live outside the sandbox at tests/, a moduleNameMapper redirects relative imports like ../../../src/core/..., and a bare ../../../src (the package root), to the sandbox's mutated source:

moduleNameMapper: {
  '^(?:\\.\\./)+src(/.*)?$': '<rootDir>/src$1',
}

Without this, tests would import the original (un-mutated) source and every mutant would survive or show as "NoCoverage". Before the root import was mapped, tests/tdd/auth/index.test.ts compared classes loaded from the sandbox with classes loaded from the original tree, and the nightly's dry run failed on it (17 September 2026).

Stryker links the project's node_modules into the sandbox only when it is a real directory: in a git worktree whose node_modules is a symlink to the main checkout's, the sandbox gets none, and every test file fails to compile with Cannot find name 'jest'. Make the worktree's node_modules a directory of per-package symlinks instead (Track S Run 12, 28 September 2026).

Static mutants

ignoreStatic: true is enabled. Static mutants are mutations in module-level code (e.g., default values, constant expressions) that are only executed once during module initialization. These are expensive to test (they require re-running ALL tests since every test loads the module) and represent only ~1% of mutants. They are reported as "Ignored" in the output.

Reading the report

The HTML report at reports/mutation/mutation.html shows:

• Per-file scores with color coding (green ≥ 80%, yellow ≥ 60%, red < 60%)
• Individual mutants with their status, the original code, and the mutation applied
• Covering tests for each mutant (which tests would need to kill it)

Current per-directory floors are in config/mutation/unit-thresholds.json; npm run mutation:ratchet prints today's scores next to them.

Killing a survivor

When a mutant survives, it means changing that line doesn't break any test. To kill it:

1. Open the HTML report (or the mutation-pr job summary) and find the survived mutant
2. Read what the mutation does (e.g., a > b changed to a >= b)
3. Write a test case where the original behavior and mutated behavior produce different results
4. Re-run the file with --force to confirm the mutant is killed (an --incremental run can keep reporting it as Survived when you strengthened an existing test rather than adding one; see The incremental baseline)
5. Raise the directory's floor in config/mutation/unit-thresholds.json to the new score, with a one-line reason in the pull request

The weakest directories at seeding were src/core/cache (ETagCacheManager.ts), src/core/logger and src/core/requestPipeline (statusHandling.ts, cachePolicy.ts).

Runtime

Run	Wall time
Pull request, one-function change to ETagCacheManager.ts, local, concurrency 4, same-day baseline	1 min 9 s (169 mutants instrumented, 1 of 1,845 files mutated)
The same change against a one-day-old baseline (51 of 169 mutants reused)	2 min 22 s
Known-weak fixture, local	15 s
Known-weak fixture, GitHub runner	9 s
Pull request job on a GitHub runner with no cached baseline (the whole src/core/cache directory mutated)	5 min 12 s for the job, 4 min 41 s for the mutation step
Full unit run, local, concurrency 4 (1,907 mutants, the seeding run of 16 September 2026)	54 min
Full unit run, nightly, unsharded (GitHub runner, concurrency 6), until 16 September 2026	about 2 h, then no longer finished inside 240 min
Full unit run, nightly, five shards (18 and 19 September 2026)	38 to 82 min per shard

The pull request step has a 30-minute deadline inside a 37-minute job. It was 8 minutes until #434: src/core/ApiRequestHandler.ts has 110 mutants, and a change to it re-tests about 70 of them even with a fresh baseline, because every test that covers a changed line invalidates the mutants it reaches. At about 7 s a mutant on two test runners, plus 30 s per timeout, one such run reached 54 of 69 at a 15-minute deadline and 59 of 69 at 20, slowed further by test runners running out of memory and restarting. A change that invalidates most of a large directory (for example, a rewrite of RateLimiter.ts with no baseline) can still exceed 30; that fails the job rather than passing it.

Type mutation

Location: scripts/mutation/type-mutation.ts (CLI), scripts/mutation/type-mutation-core.ts (operators, sampling, ratchet), scripts/mutation/type-mutation-run.ts (workspaces, tsd) Run: npm run build && npm run test:type-mutation (-- --ratchet gates, -- --update raises floors, --max, --seed, --workers) CI: type-mutation-testing in nightly-mutation.yml. Not a pull request job.

The tsd suite is only as good as the promises it pins. Type mutation checks that the way Stryker checks the unit suite: it makes one deliberate edit to a copy of the built declarations and runs the tsd tests against it, with their ../../src imports pointed at the copy. Mutants come from the declarations the package.json exports entries reach, found with the TypeScript compiler API:

Operator	Edit
return-unknown	A function, method, accessor or call signature returns unknown
drop-readonly	A readonly modifier or readonly T[] loses readonly
optional-to-required	x?: becomes x: (properties and parameters)
required-to-optional	x: becomes x?: (parameters only when nothing required follows)
union-drop-member	The first, last or a nullish member of a union is dropped
widen-literal	A literal, or a union of same-kind literals, becomes string, number or boolean
remove-overload	One signature of an overload group is removed
constraint-unknown	T extends X becomes T extends unknown

A mutant is killed when tsd reports a failure in a type test, invalid when the mutated declarations themselves no longer compile (excluded from the score), and survives when tsd passes. A survivor is a missing tsd case. The score per entry point is killed / (killed + survived); the floors in config/mutation/type-thresholds.json run from 0 (./testing) to 32 (./sde), with ./errors at 24 and ./sde/memory at 25, which is the honest measure of how much of the type surface the tsd suite pins today.

About eight thousand candidates exist, so at most 500 run. Entry points take turns picking their next mutant in order of a seeded hash of the mutant's id (built from file, symbol, operator and the mutated text, not offsets), so the same seed and surface always give the same sample, and each mutant a change adds displaces at most one sampled mutant instead of reshuffling the rest. --ratchet refuses a non-default --seed or --max, because the floors were measured on the default sample. The report is reports/type-mutation/type-mutation.{json,md}.

tests/tdd/type-mutation/ holds the negative fixture: a one-interface package whose tsd test pins Widget.id and never mentions Widget.label. The suite runs the real mutation against it and fails unless making id optional is killed and making label optional survives.

How the tests are wired

Jest configurations

Nine Jest configs (ls config/jest/*.config.cjs):

Config	testMatch	Timeout	Notes
config/jest/unit.config.cjs	tests/tdd/**/*.test.ts, BDD *.steps.ts and *.spec.ts	default	npm test; coverage floors; setup enables jest-fetch-mock
config/jest/fuzz.config.cjs	tests/fuzz/**/*.test.ts	30 s
config/jest/faults.config.cjs	tests/faults/**/*.test.ts except *.nightly.test.ts	30 s
config/jest/faults.nightly.config.cjs	tests/faults/**/*.nightly.test.ts	600 s
config/jest/contract.replay.config.cjs	tests/contract/replay/**/*.test.ts	30 s	No network
config/jest/contract.config.cjs	tests/contract/**/*.test.ts except replay/	60 s	Suites skip without ESI_LIVE_TESTS
config/jest/contract.live.config.cjs	as config/jest/contract.config.cjs	60 s	Global setup fails unless ESI_LIVE_TESTS=true; what CI runs
config/jest/integration.config.cjs	tests/integration/**/*.test.ts	30 s	Mocked suite runs; live and gated suites skip without their environment
config/jest/integration.live.config.cjs	tests/integration/live-esi.test.ts	30 s	Global setup fails unless ESI_LIVE_TESTS=true

config/jest/unit.config.cjs loads tests/setup/jest.setup.ts (enables jest-fetch-mock, builds a shared ApiClient with the rate limiter in test mode, resets both before each test) and the global setup and teardown beside it.

Mocking

Unit tests mock fetch directly with jest-fetch-mock. The specification, composition, fault, replay and property tiers mock at the transport seam instead: tests/bdd/support/transport.ts (and tests/fuzz/support/fakeEsi.ts for properties) install a handler on jest-fetch-mock that serves queued or computed responses and records every request. No test outside the live tiers makes a real HTTP request.

import fetchMock from 'jest-fetch-mock';

fetchMock.mockResponseOnce(JSON.stringify({ name: 'Jita' }));
const result = await universeClient.getSystemById(30000142);
expect(result.name).toBe('Jita');

fetchMock.mockResponseOnce('Not Found', { status: 404 });
await expect(client.getAllianceById(99999999)).rejects.toThrow(
  'Resource not found',
);

Test helpers

src/core/util/testHelpers.ts provides getBody(), which unwraps a client result in TDD tests.

src/testing/TestDataFactory.ts (the ./testing sub-path) builds mock data with sensible defaults and optional overrides:

import { TestDataFactory } from '../../../src/testing/TestDataFactory';

const alliance = TestDataFactory.createAllianceInfo();
// => { alliance_id: 99005338, name: 'Goonswarm Federation', ticker: 'CONDI', ... }

const custom = TestDataFactory.createAllianceInfo({
  name: 'My Alliance',
  ticker: 'TEST',
});

const notFound = TestDataFactory.createError(404);
// => EsiError { statusCode: 404, message: 'Resource not found' }

Available factory methods:

Method	Returns
createAllianceInfo()	AllianceInfo
createAllianceContact()	AllianceContact
createAllianceContactLabel()	AllianceContactLabel
createCharacterInfo()	CharacterInfo
createCharacterPortrait()	CharacterPortrait
createCharacterAttributes()	CharacterAttributes
createCharacterSkill()	CharacterSkill
createCharacterRoles()	Roles object
createCharacterLocation()	Location object
createCharacterSkills()	Skills summary
createCharacterAsset()	Asset object
createCharacterMarketOrder()	Character market order
createCharacterOrderHistory()	Order history entry
createCharacterMedal()	Medal object
createCharacterNotification()	Notification object
createCorporationInfo()	CorporationInfo
createCorporationHistoryEntry()	Corp history entry
createCorporationMemberRoles()	Member roles object
createCorporationAsset()	Corp asset object
createCorporationStructure()	Structure object
createCorporationWallet()	Wallet division
createMarketOrder()	MarketOrder
createMarketPrice()	Price object
createMarketHistory()	History entry
createWalletTransaction()	WalletTransaction
createCorporationWalletTransaction()	CorporationWalletTransaction
createWalletJournalEntry()	Journal entry
createContract()	Contract
createPublicContract()	PublicContract
createFleetInfo()	Fleet object
createFleetMember()	Fleet member
createFleetWing()	Fleet wing
createIndustryJob()	Industry job
createCorporationIndustryJob()	Corporation industry job
createBlueprint()	Blueprint object
createSolarSystem()	System object
createStation()	Station object
createStructure()	Structure object
createItemType()	Type object
createItemGroup()	Group object
createStar()	Star object
createPlanet()	Planet object
createSearchResults()	Search result set
createEntityName()	Named entity
createSovereigntySystem()	Sovereignty system (combined)
createSovereigntyHub()	Sovereignty hub
createOrbitalSkyhook()	Orbital skyhook
createRaidableSkyhook()	Raidable skyhook
createMercenaryDen()	Mercenary den
createMercenaryTacticalOperation()	Mercenary tactical operation
createAccessList()	Access list
createError(statusCode)	EsiError
createTestScenarios()	Full test scenario set
createPerformanceTestData(size)	Bulk test data
createRealisticTestData()	Linked alliance/corp/character

Every payload builder's default output passes the Zod schema its endpoint is validated with. tests/tdd/testing/TestDataFactory.schemas.test.ts enforces this, so a new builder must be added to its map. The same test checks that builders leave out fields ESI never sends (an entity's own ID on its detail route, for example) and include the fields its spec marks required.

Schema validation, level by level

Zod validation is tested at four levels, each owning a different question:

Level	Where	Question
Schema	tests/tdd/schemas/ (schemaRejection.test.ts and others)	Does each schema accept a valid body, reject a wrong type and preserve an extra field?
Property	tests/fuzz/schema-fuzz.test.ts	Does safeParse() never throw, whatever it is given?
Pipeline	tests/fuzz/response-validation-fault-injection.test.ts, the fault catalogue, recorded replay	What does a consumer receive when ESI sends a body the schema rejects, or a real body it must accept?
Requirement	tests/bdd/features/core/0053-runtime-validation.feature, npm run validate:spec-consistency	What does the specification promise, and does every Rule agree with the schema about optional fields?

Testing your application

An application's own tests run without ESI by handing the runtime a mock transport. createMockTransport() from @lgriffin/esi.ts/testing is an HttpTransport: it answers requests from a table of routes and records every request it saw. Everything between the application's call and the transport is the SDK's real pipeline (URL building, headers, retries, the cache, pagination, response validation), so a test written against it can fail for a bug in the SDK as well as in the application. The specification is 0057-mock-transport.feature.

import { createEsi, identityFromToken } from '@lgriffin/esi.ts/client';
import { createMockTransport } from '@lgriffin/esi.ts/testing';

const transport = createMockTransport()
  .respond({
    method: 'GET',
    path: '/status',
    body: {
      players: 12,
      server_version: '1',
      start_time: '2026-09-16T11:00:00Z',
      vip: false,
    },
  })
  .respond({
    method: 'GET',
    path: '/characters/{character_id}/wallet',
    body: 1234567.89,
    times: 1,
  })
  .respond({
    method: 'POST',
    path: /\/universe\/names/,
    status: 200,
    body: [],
  });

const esi = createEsi({ userAgent: 'my-app/1.0 (you@example.com)', transport });
const wallet = await esi
  .as(identityFromToken(token))
  .character(characterId)
  .wallet.get();

transport.sent[0]?.headers['authorization']; // "Bearer <token>"
transport.unrouted; // [] when every request had a route
transport.reset(); // forget the routes and the record, keep the runtime

A route is one respond() call:

Field	Meaning
method	The HTTP method, compared without regard to case. Omitted, any method.
path	The ESI path template as the spec writes it ({character_id} stands for one segment, a trailing slash is ignored, the query string is not compared), or a regular expression tested against the whole URL.
status	200 to 599. Default 200.
headers	Response headers, such as x-pages for a paginated route or etag.
body	An object, array, number or boolean is JSON-encoded as application/json; a string is sent verbatim; omitted, no body.
times	How many matching requests the route answers before it is retired. Omitted, every one.

Routes are tried in the order added and the first match answers. A request no route answers is rejected with an EsiConfigurationError (code CONFIGURATION_ERROR) naming the request and the route table. The pipeline passes an EsiError a transport throws through unchanged, so the call fails at once under that name, without a retry and without falling back to a stale cache entry as an HTTP 5xx would; the transport lists the request under unrouted. sent holds every request in order, header names in lower case, the body as a string or undefined.

reset() empties the route table and the record, not the runtime built over the transport: once a route answered with an etag header, the runtime's response cache serves that request again without reaching the transport. A test file that repeats a request across a reset builds a runtime per test, or passes enableETagCache: false to createEsi.

TestDataFactory, from the same entry, builds response bodies the schemas accept (Test helpers).

Adding tests

1. A behaviour change starts with an EARS Rule and a scenario that fails before the implementation exists (TEST-01). Add the Rule to the right feature under tests/bdd/features/, bind a new feature with a spec entry under tests/bdd/specs/, and add one file per new step under tests/bdd/steps/<keyword>/. Queue HTTP at the transport seam. Confirm red, implement, confirm green, run npm run spec:audit. Never add a legacy step file.
2. A unit test goes in tests/tdd/<area>/. Mock fetch, call the method, assert the result; add describeClientErrors to a new client's file.
3. An invariant that no example can cover goes in tests/fuzz/ as a *.property.test.ts, with a known-bad implementation it must fail against.
4. A network fault goes in the catalogue in tests/faults/, citing the Rule that specifies the outcome.
5. A new public GET endpoint needs a recorded fixture (ESI_LIVE_TESTS=true npm run contract:record -- --only=<key>) or an unrecordable.json entry with a reason, and a public or mixed example that calls it.
6. Test data goes in src/testing/TestDataFactory.ts and in the schema map of TestDataFactory.schemas.test.ts.
7. A live test goes in tests/integration/, gated with LIVE ? describe : describe.skip (never a committed .skip), idempotent and read-only against ESI (TEST-08).

Before pushing, npm run check:local runs every offline tier CI will.

CI schedule

Every pull request to master runs ci.yml, whose single required check is ci-success. Every push to any branch runs ci-fast.yml. The scheduled runs, in UTC:

Workflow	When	What it runs	On failure
ci-fast.yml	Every push	lint, lint:layers, lint:bdd-seam, lint:suite-health, format:check, build, typecheck, typecheck:examples, typecheck:isolated, npm test on Node 22	Status only
ci.yml	Every pull request	Every tier marked PR in the tier table; ci-success fans in all of its jobs	Blocks the merge
ears.yml	Pull requests touching the specification or src/; manual	npm run ears	Status; advisory
nightly-schemathesis.yml	Daily 01:00	npm run fuzz:api equivalent against a Prism mock	Artifact; issue "api-fuzz: nightly Schemathesis run failed"
nightly-mutation.yml	Daily 02:00	Unit mutation (6 shards), BDD-only mutation (32 shards), type mutation; each ratcheted	Fails the run; issue "mutation: nightly run failed"
nightly-mutation-retry.yml	When nightly-mutation.yml fails on its first attempt	Re-runs the failed shards once	Status only
nightly-no-retry.yml	Daily 03:00	npm test in random order with a fresh seed; fails if any test ran more than once	Fails the run; no-retry-report artifact
nightly-properties.yml	Daily 03:30	npm run fuzz:properties with FC_NUM_RUNS=10000 and a seed per night	Issue "Nightly property run failed"
nightly-interleave.yml	Daily 03:30	The composition tier with four calls in seeded random order	Fails the run
nightly-faults.yml	Daily 03:30	npm run faults:nightly and npm run faults	Issue "Nightly fault tier failing"
nightly-examples.yml	Daily 04:15; PRs touching examples/	Type-checks every example; runs the public and mixed ones against live ESI	One issue per failing example
nightly-benchmarks.yml	Daily 04:30	Benchmarks against a pinned reference; the heap soak and its injected leak	performance-nightly issue
consumer-matrix-nightly.yml	Daily 04:45	The consumer contract on TypeScript next/latest and current Node	Fails the run
nightly-audit.yml	Daily 05:00	npm audit against the accepted-advisory list	security-audit issue
nightly-live-health.yml	Daily 05:45	npm run health:live: refresh the maintained SSO token, call /status and /characters/{id}/online through the real client	Issue "Live health: the Tranquility smoke failed"; an outage only warns
nightly-sde.yml	Daily 05:15	CCP's current SDE export: tests/integration/sde with SDE_REQUIRE_DATA=1, sde:drift, the sde examples, the sde/ benchmarks and the SDE soak	Issue "sde: nightly real-data run failed or the export drifted"
nightly-spec-drift.yml	Daily 06:00	Missing endpoints, generated-type freshness, schema drift, against the newest compatibility date	spec-drift issue
nightly-recorded-payloads.yml	Daily 06:30	contract:record, contract:shape-diff -- --revert-unchanged, contract:replay	A pull request with the shape diff; recorded-payloads-check-failed issue
maintenance.yml	Mondays 09:00	contract:snapshot, contract:live, contract:diff (oasdiff), outdated, audit, coverage	Artifacts only
post-publish-canary.yml	A GitHub release is published	Installs the published version from the registry, verifies provenance, loads every sub-path, calls ESI once	release-verification issue

Nothing runs the live smoke, client integration, spec contract or gated auth suites on a schedule; the real-SDE integration suite runs nightly in nightly-sde.yml. Every workflow, its jobs and what blocks where are in QUALITY-GATES.md.

Known gaps

Gap	Severity	Notes
Contract deviations	Medium	Four SDK-to-spec mismatches in the deep contract's exception sets (Live contract), unfixed since the first run
Non-primitive path parameter	Low	validatePathParam throws a raw TypeError, not an EsiError, for an object whose toString returns a non-primitive (the template literal in src/core/util/validation.ts), and accepts other non-primitives by their string form (a plain object as [object Object]); the fuzz properties in tests/fuzz/parameter-fuzz.test.ts hold only that what it returns is a safe, non-empty segment
Type drift	Medium	Fields in the spec not yet in hand-written types, reported as warnings by esi-spec-contract.test.ts; schema drift is ratcheted in scripts/spec/schema-drift-baseline.json
Live suites unscheduled	Medium	No workflow runs live-esi, client-integration, esi-spec-contract, gated-auth or sde-real-data; the last skips wherever sde-data/ is absent
Corporate auth endpoints	Medium	Gated tests cover character-level auth, not corporation director endpoints
DNS and TLS failures	Low	The fault tier covers resets, stalls and timeouts; DNS and TLS failures reach the client as the same fetch failed and are not told apart
Mutation flakiness	Medium	Mutants that time out on one runner and survive on another (#382); stale incremental results (#380)
API fuzz has no signal	Low	Schemathesis has no negative fixture, so a green night does not prove it can fail

Debugging

# One test file
npx jest --config config/jest/unit.config.cjs tests/tdd/alliances/AllianceClient.test.ts

# Tests matching a name
npx jest --config config/jest/unit.config.cjs --testNamePattern="should return valid alliance"

# One path pattern (Jest 30 spells it --testPathPatterns)
npx jest --config config/jest/unit.config.cjs --testPathPatterns=composition

# Replay a nightly no-retry order
npx jest --config config/jest/unit.config.cjs --randomize --seed=<seed>

# Node inspector
node --inspect-brk node_modules/.bin/jest --config config/jest/unit.config.cjs --runInBand

File reference

Path	Purpose
config/jest/*.config.cjs	The nine Jest configs (table)
tests/setup/jest.setup.ts	Enables jest-fetch-mock, shared ApiClient, rate-limiter reset
tests/tdd/helpers/clientErrorTests.ts	describeClientErrors, the five HTTP error cases
tests/bdd/support/transport.ts	The transport seam: queueResponse, queueError, sentRequests
tests/bdd/support/binder.ts	Binds a feature to the step library under Jest
scripts/spec/spec-audit.ts	The EARS and Gherkin audit
scripts/spec/spec-audit-exceptions.json	unconverted (empty) and legacyStepFiles (38), shrink-only
scripts/quality/bdd-report.ts	The execution check and the JUnit report
scripts/quality/ears.ts	The standalone EARS check
tests/fuzz/support/property.ts	describeProperty, run settings and known-bad registration
tests/faults/catalogue.ts	The named transport faults
tests/contract/record.ts	The payload recorder
tests/contract/esi-contract.test.ts	Deep contract, with the known exception sets
tests/contract/snapshots/esi-openapi.snapshot.json	The committed spec baseline
scripts/spec/run-schemathesis.sh	Prism and Schemathesis runner
scripts/quality/consumer-contract.ts	The consumer contract driver
scripts/docs/doc-examples.ts	The documentation example checker
scripts/bench/bench-ab.ts, scripts/bench/bench-compare-core.ts	Alternating runs; Mann-Whitney U, Holm, the verdict
scripts/bench/soak-core.ts	Heap trend, cache bound, timer and listener checks
scripts/quality/verify-local-core.ts	The check:local tier list and its written exclusions
config/mutation/stryker.config.mjs, config/mutation/stryker.bdd.config.mjs, config/mutation/stryker.fixture.config.mjs	Unit, BDD-only and fixture mutation configs
src/testing/TestDataFactory.ts	Mock data factory