Contributing · 4. Testing
cargo test --locked --workspace is what you run before every push, it needs no daemon and no network, and it covers 5,969 tests. Everything else in this chapter is a tier you reach for deliberately.
This is reference material. Section 4.13 is the short version if you only want the pre-push command.
4.1 The measured shape of the suite
Measured on main at 93b41a7, 2026-07-27, with grep -rE "#\[(tokio::)?test\]" crates --include="*.rs" | wc -l:
5,969 test attributes across crates/.
| Crate | Tests | Crate | Tests |
|---|---|---|---|
corecruxd | 2,244 | crux-claude-hooks | 222 |
corecruxctl | 1,043 | corecrux-storage | 147 |
crux-mcp | 769 | crux-session | 89 |
corecrux-memory | 288 | crux-config-wizard | 74 |
corecrux-projections | 276 | crux-observe | 66 |
corecrux-receipts | 275 | corecrux-segment | 62 |
docs/testing-and-coverage.mdis stale. Its headline figures and its per-crate table are a 2026-06-18 snapshot: 4,489 tests total,corecruxd1,537,corecruxctl933,crux-mcp622. The real numbers are about 33% higher. The document is labelled a snapshot, but the numbers are the first thing a reader takes away, so treat this chapter as authoritative and the older document as history. Refreshing it is good first contribution number 4.
The tiers at a glance
| Tier | Command | Needs a live daemon? | Needs network? | Roughly how long |
|---|---|---|---|---|
| 1: unit, inline | cargo test --locked --workspace | No | No | About 10 minutes on a warm CI runner. A single leaf crate is 2 s |
2, per-crate tests/ | Same command; they are part of the workspace run | No | No | Included in the above |
| 3, integration | ./scripts/run-integration-tests.sh | It spawns its own | No | Not measured. 55 tests, each spawning a real daemon process |
| 4, fuzzing | cargo fuzz run <target> -- -runs=1 | No | No | Seconds for a smoke run. Needs a nightly toolchain |
| 5, mutation | cargo mutants --file <f> -p <crate> --timeout 120 | No | No | Hours for a full crate. Scope it to one file |
| Coverage | cargo llvm-cov --workspace ... | No | No | Not measured. Adds a third target/ tree, section 1.5 |
| Ignored by design | cargo test -- --ignored | Some | Some | See section 4.10 |
4.2 Tier 1: unit tests, inline
The convention is an inline #[cfg(test)] mod tests at the bottom of the module it tests. crates/corecruxd/src/problem.rs is a small, readable example. The exception in scale is crates/corecruxd/src/http/tests.rs, a single test module of over 18,000 lines covering the HTTP surface, declared at http/mod.rs:1929.
cargo test --locked --workspace # everything
cargo test -p corecruxd # one crate
cargo test -p corecruxd route_auth # one filter
cargo test -p corecruxd http_boundary_contractscorecruxd is a binary crate with no lib target, so -p corecruxd runs its inline tests plus its tests/ directory.
No live daemon. No network. This is what you run before every push.
4.3 Tier 2: per-crate tests directories
These run as part of cargo test --workspace; they are listed separately because knowing what exists tells you where to add yours.
| Path | Covers |
|---|---|
crates/corecrux-segment/tests/corruption_matrix.rs | Tamper rejection: magic, version, CRC, record hash, TOC corruption |
crates/corecrux-memory/tests/sync_low_hanging.rs | Memory sync paths |
crates/corecrux-retrieval/tests/cat12_validation.rs | Retrieval quality against the checked-in cat12-corpus.json |
crates/corecruxctl/tests/ | cose_cli.rs, deploy_audit_cli.rs, fixtures.rs, ingest_dry_run.rs, plus three fixture directories |
crates/corecruxd/tests/mutation_path_receipt_audit.rs | Asserts every mutation path emits a receipt |
crates/corecruxd/tests/route_spec_drift.rs | Route spec against the console api.js; also hosts the regen_api_js writer |
crates/crux-claude-hooks/tests/ | hook_e2e.rs, llm_shim_e2e.rs, cloud_witness_e2e.rs, snapshot_egress_e2e.rs |
crates/crux-config-wizard/tests/end_to_end.rs | Profile-fragment composition |
crates/crux-integrations/tests/ | community_packs.rs plus two Vault and C2PA suites, both #[ignore] |
crates/crux-mcp/tests/ | memory_recall_bench.rs, token_bench_determinism.rs |
crates/crux-observe/tests/leak_canary.rs | Redaction leak canary |
crates/crux-session/tests/ | always_store.rs, ce_full_parity.rs, ce_migration_round_trip.rs, end_to_end.rs, golden.rs, invocation_chain.rs |
4.4 Tier 3: integration tests
crates/crux-integration-tests holds four suites, 55 tests total: daemon.rs (37), grpc.rs (13), e2e.rs (4), ledger.rs (1).
./scripts/run-integration-tests.shThat script is exactly three steps:
cargo build -p corecruxd --manifest-path <repo>/Cargo.toml
export CORECRUXD_BINARY=<repo>/target/debug/corecruxd
cargo test --manifest-path <repo>/Cargo.toml -p crux-integration-testsOr by hand, which is what CONTRIBUTING.md documents:
cargo build --bin corecruxd
CORECRUXD_BINARY=target/debug/corecruxd cargo test -p crux-integration-testsThese tests need no live daemon of yours and no network. The harness spawns and tears down its own.
4.5 How the integration harness works
Worth understanding before you add a test, because the design decisions are load-bearing. All in crates/crux-integration-tests/src/lib.rs.
| Property | How |
|---|---|
| One real daemon per test | TestDaemon::start() spawns an actual corecruxd process; start_with_agent_token() is the authed variant |
| Parallel-safe | Three distinct random ports per daemon via portpicker::pick_unused_port(). No hardcoded ports |
| Isolated state | A fresh tempfile::tempdir() per daemon |
| Deterministic environment | The harness pins the daemon env: CORECRUXD_AUTH_MODE=off, CORECRUXD_LOG_LEVEL=warn, text search, graph expand, time range and BUILD_CCXI all on |
| Never touches the network | CORECRUXD_UPDATE_CHECK_ENABLED=0 is pinned by the harness |
| Cannot inherit your shell | CRUX_AGENT_TOKEN and CRUX_AGENT_TOKENS are explicitly removed from the child environment |
| Fails loudly | Startup is retried with health polling; on total failure it panics with every attempt's stderr |
| Works under coverage | The CORECRUXD_BINARY override exists because cargo llvm-cov uses a non-standard target directory |
If you add an integration test, copy this shape. Do not hardcode a port.
4.6 The root tests directory is fixtures, not tests
The repository root has no [package], so tests/ at the root is not a cargo test directory. It holds shared data consumed by crate tests.
| Path | Consumed by |
|---|---|
tests/fixtures_segments/minimal/minimal.ccxseg | crates/corecrux-segment/src/builder.rs via include_bytes!, crates/corecrux-storage/src/tests.rs, crates/corecruxctl/src/storage.rs, and corecruxctl fixture-digest |
tests/fixtures_v3/minimal | crates/corecruxctl/tests/fixtures.rs |
tests/fixtures_v1/minimal | The stage-1 to v3 import bridge, corecruxctl import-v1 |
tests/bench/replay_gates/ | Baseline and candidate JSON for the replay gate logic |
tests/bench/replay_many_gates/ | The same, for the multi-replay gate |
tests/bench/perf_regression_gates/ | Baseline and candidate JSON for the perf-regression gate |
Adding a .rs file here does nothing. It will not compile and it will not run.
4.7 Tier 4: fuzzing
fuzz/ is a separate cargo workspace (it has its own [workspace] table). Four targets:
| Target | Fuzzes |
|---|---|
segment_decode | corecrux_segment::decode_segment_v1 |
storage_scan_frames | The storage block frame scanner, via the crate's fuzzing feature |
receipt_verify_cbor | Receipt body and signature CBOR verification decode |
rcx_canonical_token | RCX canonical CBOR decode plus typed token validation and verification |
A local smoke run. This needs a nightly toolchain; that is a cargo-fuzz requirement, not a project choice:
cargo install cargo-fuzz
cargo fuzz run segment_decode -- -runs=1
cargo fuzz run storage_scan_frames -- -runs=1
cargo fuzz run receipt_verify_cbor -- -runs=1
cargo fuzz run rcx_canonical_token -- -runs=1The rule from fuzz/README.md: "These targets must not use production data or network access."
In CI, .github/workflows/fuzz.yml runs a short PR-time smoke when fuzz/** or the trust-core crate paths change, plus a nightly run at 03:30 UTC with corpus persistence and crash-artifact upload. The runner-safety posture is spelled out in the workflow: max-parallel: 1, explicit -rss_limit_mb and -malloc_limit_mb bounds, a canonical-CBOR length-prefix OOM in rcx_canonical_token would otherwise kill the runner, a corpus capped by cmin plus a size trim, and a per-run CARGO_HOME reclaimed unconditionally.
Fuzzing is not a required check.
4.8 Tier 5: mutation testing
Two workflows, neither required.
| Workflow | Name | Trigger | Shape |
|---|---|---|---|
mutants.yml | Mutation (trust core) | Nightly cron 02:30 UTC, plus manual dispatch | 8 shards, max-parallel: 3, CARGO_BUILD_JOBS=4. Jobs: shard <n> and merge + ratchet |
mutants-diff.yml | Mutation (PR diff) | Pull requests touching crates/corecrux-{receipts,segment,storage}/** | cargo-mutants --in-diff, pinned to cargo-mutants@27.1.0 |
The nightly cron time and the sharding both come from an incident: the previous single 3-to-6-hour job was OOM-killed on roughly 29 of 31 nights and still reported green. The version pin exists because the mutants.out format changes between versions and the nightly and PR legs must agree.
Exit-code handling is deliberate: 0 clean, 2 survivors, 3 timeouts, all ratcheted. Anything else (4 is a baseline failure; others are infrastructure) hard-fails.
The ratchet is .github/mutants-baseline.txt, 755 lines: corecrux-storage 524, corecrux-segment 164, corecrux-receipts 65. A new survivor turns the run red. Survivors that become caught are listed so the baseline can shrink.
To burn one down:
cargo mutants --file <file> -p <crate> --timeout 120Then delete its line from .github/mutants-baseline.txt. Only genuinely inert mutations, logging, metrics, belong in the baseline long term; prefer #[mutants::skip] with a comment explaining why.
The report tool is scripts/mutants-report.py:
python3 scripts/mutants-report.py --outs mutants.out --baseline .github/mutants-baseline.txt4.9 Verifying the claims yourself
AGENTS.md:34 invites you to verify rather than trust, and Apache-2.0's Grant of Copyright License (LICENSE:66) covers exactly this: reproducing, running, and inspecting the source is a licensed right, not a favour. Four commands:
cargo test --workspace
corecruxctl verify-store --strict
corecruxctl replay --strict
cargo fuzz run segment_decode -- -runs=1Two shell gates go further, and both run offline against a binary you built:
CORECRUXCTL=target/debug/corecruxctl bash scripts/demo-receipt-tamper.sh
CORECRUXCTL=target/debug/corecruxctl bash scripts/assert-context-custody.shassert-context-custody.sh is the interesting one. It is a self-run exit test: seed a CROWN receipt into a fresh temporary data directory, run context export, run context verify --json asserting ok=true with all four checks passing and a zero exit, then tamper one byte inside memory.cruxpack and re-verify, asserting ok=false with a hash-mismatch failure and a non-zero exit. Its own header states the point: "This proves the verifier is real, not a rubber stamp that always returns ok."
demo-receipt-tamper.sh does the equivalent for verify-store: an on-disk byte flip must be caught.
Both also run PR-time inside the Test job, so they are gates as well as demonstrations.
What these do not prove: they exercise the local daemon in this repository. They say nothing about hosted backend behaviour, and a CROWN receipt is a verifiable record of what was stored and retrieved, not an attestation of what an agent did. See chapter 8 and docs/assurance-coverage-matrix.md.
4.10 Tests that need something extra
Everything here is #[ignore]d and will not run in a normal cargo test. Run them with cargo test -- --ignored, understanding what each one wants.
| Test | Needs | Why it is ignored |
|---|---|---|
crates/corecruxd/src/witness_submit.rs | Live network; it hits Rekor staging | #[ignore = "hits live Rekor staging over the network; run with --ignored"] |
crates/crux-integrations/tests/vault_c2pa_m4_integration.rs | A live hashicorp/vault dev server with a pki-c2pa mount and a c2pa-leaf role | External service |
crates/crux-integrations/tests/vault_c2pa_m4_evidence.rs | A running vault -dev plus c2patool | External service |
crates/crux-session/tests/golden.rs | A CueCrux-Shared sibling checkout that you do not have | #[ignore = "pre-existing: decoder/schema drift vs v2 fixtures; fixture path requires CueCrux-Shared sibling"] |
crates/corecruxd/src/http/tests.rs | Over 35 s of wall clock | #[ignore = "long-running"] |
crates/crux-observe/src/redact.rs | Release mode to mean anything | #[ignore = "perf bench"] |
The crux-session golden tests are four skips you cannot fix. They are permanently ignored on a private sibling checkout path. Seeing them skip is expected; do not spend an evening on it.
One CI-only suite has its own prerequisites: .github/workflows/audit-vectors.yml needs Python 3.12 with cryptography>=41 and zstandard>=0.22. It runs tools/verify_audit_bundle_v1.py against every vector under crates/corecrux-receipts/vectors/audit-bundle-v1/, in both unpacked-directory and .tar.zst form, then regenerates them with tools/gen_audit_bundle_vectors.py to assert no drift. It is not a required check.
4.11 Writer tests and serialised tests
Two tests are writers: they regenerate committed artefacts and are #[ignore]d by design so they never fire accidentally.
cargo test -p corecruxd --test route_spec_drift -- --ignored regen_api_js
cargo test -p crux-integrations --test community_packs -- --ignored regen_studio_board_exampleRun regen_api_js whenever you add a route the console calls. Its non-ignored twin, generated_api_js_is_in_sync (route_spec_drift.rs:784), fails if you forget.
Serialisation: there are 342 serial_test and #[serial] references across more than 20 modules, heaviest in corecruxd/src/http/tests.rs, config.rs, auth.rs and the workspace-scan modules, plus much of corecruxctl. These are tests that mutate environment variables or the current working directory. serial_test = "3" is a workspace dependency (Cargo.toml:85).
If your new test reads or writes an environment variable, mark it #[serial]. If you can avoid the environment variable entirely, do that instead, the pattern the codebase prefers is a flag-checking public wrapper delegating to a flag-free handle_inner, so the core is testable without env races. See 6.2.
4.12 Known-flaky and environment-sensitive areas
These are real incidents with real fixes, recorded so you recognise the shape if you hit it.
| Area | What happens | The fix in tree |
|---|---|---|
| Port collisions in CI smoke | Concurrent Test jobs share a runner host. On the default 14800/14801 an AddrInUse daemon dies silently while the probes happily test another job's daemon, a false green | Per-run strided ports (ci.yml:206). Any new smoke probe must do the same |
assert-no-phone-home.sh cannot run PR-time | It binds fixed ports (MCP 24801) and collides with leftover daemons on the shared self-hosted runner | It runs only on the ephemeral ubuntu-latest release leg |
SIGPIPE races under set -o pipefail | Three separate CI bugs from piping curl into grep -q, printf into head, and echo into grep -q. A grep -q exiting at its first match can SIGPIPE the writer, and a negated test then misclassifies | Stage to a variable, use a herestring, or use bash string matching. See ci.yml:81 |
| Runner disk | Incident 2026-06-15: 263 orphan CARGO_HOME directories, 118 GiB, the data array at 100%, runner listeners crash-looping | Every job creates a per-job CARGO_HOME and reclaims it in an if: always() step (ci.yml:139) |
| Runner OOM | Incident 2026-06-22 on the mutation jobs | CARGO_BUILD_JOBS=4 on those jobs |
crux-session golden tests | Four permanent skips on a private path | None available to you. Section 4.10 |
4.13 What to run before you push
The short version. The full gate list, including the lint and policy checks, is in 5.11.
cargo fmt --check
cargo clippy --locked --workspace -- -D warnings
cargo test --locked --workspace
./scripts/run-integration-tests.sh # if you touched the daemon surfaceSources
- crates/crux-integration-tests/src/lib.rs,
TestDaemon, port picking, pinned env - crates/corecruxd/tests/route_spec_drift.rs:784,
generated_api_js_is_in_sync - crates/corecruxd/tests/route_spec_drift.rs:816,
regen_api_js, the writer - crates/corecruxd/src/http/mod.rs:1929, the HTTP test module declaration
- Cargo.toml:85,
serial_test = "3" - scripts/run-integration-tests.sh, the three-step integration runner
- scripts/assert-context-custody.sh, the tamper exit test
- scripts/demo-receipt-tamper.sh,
verify-storebyte-flip demonstration - .github/mutants-baseline.txt, the 755-line survivor ratchet
- .github/workflows/fuzz.yml, fuzz schedule and runner-safety bounds
- .github/workflows/mutants.yml, nightly sharding rationale
- .github/workflows/ci.yml:139, per-job
CARGO_HOMEreclaim - AGENTS.md:34, verify the claims yourself
- LICENSE:66, Grant of Copyright License, the audit right in practice