CueCruxVaultCrux

Operations · 8. Troubleshooting

Start with the symptom you can see, not the subsystem you suspect. This chapter is a how-to. Each entry gives the check that discriminates, the cause, and the fix. They are ordered by how often each actually happens.

Two commands answer more incidents than everything else here combined:

df -h /path/to/data_dir
curl -s http://127.0.0.1:14800/readyz

Run both before forming a theory.

8.1 The daemon will not start

It exits immediately with a message and no log lines

Check: read the message on stderr. That is the whole diagnosis.

Cause: you are in the pre-tracing part of boot. Nothing has been logged yet, by design.

Fix: match the message against the 25 refusal conditions in chapter 4 of the internals set §4.3. Each row carries the exact operator fix.

CORECRUXD_AUTH_MODE must be set explicitly

Cause: there is no default. This is intentional.

Fix: set the variable, or set daemon.auth_mode in the config file. If you believe you set it in a file, read the next entry; that failure is silent.

Configuration in a YAML file appears to be ignored

Check: set CORECRUXD_CONFIG_PATH explicitly and validate the YAML separately.

Cause: three possibilities, all silent. XDG_CONFIG_HOME is unset so no file is read at all, and there is no ~/.config fallback. Or the file has a syntax error and the parse failure is discarded. Or the path is wrong, which looks identical to no file.

Fix: name the path explicitly and check the YAML parses.

auth mode Off may not bind to non-loopback addresses…

Cause: the bind-posture rail. A dev auth mode cannot bind a non-loopback address.

Fix: switch to a JWT mode, keep the bind on loopback, or set CORECRUXD_ALLOW_INSECURE_DEV_AUTH_BIND=1 knowingly. A stock Helm install trips this, see chapter 16 B2.

A boolean flag you set has no effect

Cause: nine incompatible parsing rules across the flag surface, and six flags do not trim whitespace. A trailing newline from a systemd EnvironmentFile= or a Kubernetes configMapKeyRef reads as off.

Fix: use =1 and =0. They work under every rule.

A command-line flag does nothing

Cause: corecruxd accepts no runtime configuration flags and silently ignores unrecognised ones. corecruxd --data-dir /x starts normally and ignores it.

Fix: use environment variables.

8.2 /readyz returns 503

The body names a gate

Check: read checks[]. Nine gates exist (health.rs:276).

Failing gateCauseFix
data_dir_capacityFree ratio below the emergency threshold, 0.10 by defaultFree disk. See §8.3
data_dir_lock_heldThe lock is not heldAnother daemon holds it, or the path changed
routing_loadedThe shard map did not loadCheck meta/routing/
replicated_commit_dataplaneReplicatedCommit selected on a build with no dataplaneThis gate cannot pass in this edition. Do not select that mode
replicated_commit_topologyReplication topology invalidSame
read_retry_failed_thresholdRead retries past thresholdStorage-level problem
projection_snapshots_validA projection snapshot is invalidThe response samples the offending ids
corruption_state_clearcorecrux_segment_corrupt_total incrementedRequires operator clearance after investigation
control_evidence_okThe CONTROL.json evidence check failedRead CONTROL.json

Everything unrelated is failing and the error is unhelpful

Check: df -h on the data partition. Do this first, always.

Cause: the data_dir_capacity gate takes an otherwise-healthy daemon out of rotation below 10% free. Downstream tooling reports that as a bare timeout with empty stderr, every test failing at once, across unrelated changes, with no message.

Fix: free space. Reap shards/*/quarantine/ and observations/*.jsonl; neither has any GC path.

The failure body has no per-gate breakdown

Cause: CORECRUXD_PUBLIC_PROBES_MINIMAL=1 strips it, along with routing and valve detail from /healthz.

Fix: unset it, or read the gate from metrics instead. Note the flag does not cover /metrics.

8.3 Storage and growth

Ingest is paused and you did not pause it

Check: read CONTROL.json, field valves.pauseIngest.actor.

Cause: capacity_guard means the background guard paused ingest on disk pressure.

Fix: free space. It resumes at CORECRUXD_CAPACITY_RESUME_FREE_RATIO (config.rs:1112). It will not stomp an operator-set pause.

facts.jsonl is growing unexpectedly

Check: Memory › Facts. Compare visible against stored, then turn the reserved __\* toggle on and look at the session-binding prefix.

Cause: the most common one is a stateless MCP bridge re-initialising on every poll, each initialisation writing one durable session-binding fact. Unbounded without the ephemeral GC.

Fix: CORECRUXD_EPHEMERAL_GC=1, then restart; it is read once at boot. It caps session bindings at 32 per passport and drops reverify receipts older than 30 days, and it never touches user facts (ephemeral_gc.rs:23).

Something is growing and nothing reclaims it

Cause: six artefacts have no automatic reclamation path: observation journals, shard quarantine directories, the routing shard-map history, the integrations audit journal, the activity journal and the cost report journal.

Fix: CORECRUXD_OBS_RETENTION_DAYS archives observation sessions hourly, unset means retain forever. The rest are manual. See chapter 17 §17.9.

8.4 The console

The console is blank: no rail, no content

Check, in order:

curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:14800/console
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:14800/console-v2/api.js
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:14800/console-v2/render.js
ResultCauseFix
/console 404CORECRUXD_CONSOLE_ENABLED=0; it removes every console route (console.rs:294)Unset it
/console 200, a module 404The asset name is not one of the seven served (console.rs:109)A proxy is rewriting paths
Everything 200, page still blankA script threw. Open the browser consoleThe renderer prints Renderer module unavailable when a module is missing
Blank only on one originThe CORS allowlistSee the next entry

The console loads no external resource of any kind. If your browser reports a blocked external request, something in front of the daemon is injecting it.

The console works on one host and not another

Check: CORECRUXD_CONSOLE_ALLOWED_ORIGINS.

Cause: the console asset routes use an explicit allowlist, never a wildcard. Unset, three defaults apply: https://crux.cuecrux.com, http://100.70.12.73, http://crux (console.rs:60).

Fix: set it to your real origin, comma-separated. Entries that are not a real http(s)://host are dropped, including * and null, and if nothing usable survives, the defaults apply (console.rs:248).

A red banner: "Your session has expired"

Cause: an origin-wide session gate bounced the boot read to a sign-in page, or returned 401/403 (shell.html:4864).

Fix: click Sign in, the link carries a return URL back to the exact view.

Caveat: detection runs once, at boot. A session that expires mid-visit is not caught until the next load, and presents as panels quietly failing.

The status pill says Degraded

Cause: the daemon answered but GET /v1/console/summary returned no usable data. Almost always a scope problem; your identity lacks console:read.

Fix: check the token's scopes. Navigation still works; only the live panels are empty.

Controls I expected are missing

Check: the account badge at the bottom of the rail. It reads Operator or Customer.

Cause: the posture probe. Operator requires either auth_mode: off or a 200 from GET /v1/admin/version. Anything else, including a probe that fails, times out, or is blocked by a proxy, is customer, and every mutating control is hidden (render.js:62).

Fix: grant the admin scope. Note that a reverse proxy which strips or rejects that route will silently demote you.

A whole destination is missing from the rail

Check: System › Settings, the Runtime capabilities card.

Cause: the destination is capability-gated and the daemon's plan does not report it available. Link graph is the shipped example.

Fix: configure the upstream, for Link graph, CORECRUXD_CORECRUX_GRAPH_BASE_URL (http/console.rs:761). The validation is strict: a malformed or incomplete descriptor resolves to unavailable with a reason code, so read the reason (render.js:531).

A page says "disabled: set CORECRUXD_…"

Cause: a feature flag is off. This is the console reporting accurately, not failing.

Fix: the flag is named in the message. Defaults are tabulated in chapter 1 §1.6.

A write button refuses with an Article 14 message

Cause: no bound passport. Every gated write requires one (render.js:648).

Fix: Trust › Pending mints, the Bound approver panel. Choose a passport and click Bind. The binding is browser-local, stored under crux-console-bound-passport.

A button is visible but permanently disabled

Cause: it is one of the controls gated for a stated, grounded reason, not a milestone. The reason is in its tooltip.

ControlReason
Add repo, Set as planning repoNo project id and no real repository, no groundable request body (pages.js:1668)
Queue ingestThe only real route is a synchronous ingest needing a documents[] payload; shape mismatch
Install (Extensions)The route wants a full manifest object; the form supplies a URL. Installing from the signed catalogue by id is wired, in Studio › Library
Apply defaults to all tenantsNo bulk route; "all" would be an unbounded client loop
Run sweep nowNo HTTP route, an MCP dry-run tool plus a background timer
Export audit bundleThe real route is a read needing a session id and CORECRUXD_OBSERVE
Send (Raw JSON-RPC)It would dispatch an arbitrary MCP tools/call, an arbitrary-write surface. Permanent
Every Settings select and toggleread-only in the console, set on the daemon host (pages.js:792)

The Rings view shows plans I do not recognise

Check: the data-src attribute on the rings root. live means real; absent means snapshot.

Cause: the ring only swaps its embedded snapshot for your board when the feed returns at least 50 ExecPlan items with provenance timestamps in in_progress, complete or blocked (render.js:15119).

Fix: none needed; it is a deliberate fallback so the page is never empty. Below the threshold, read the work board instead.

The ExecPlan board is empty but plans exist

Cause: CRUX_EXECPLANS_ROOT is not set in the daemon's process environment. Without it the read-time projection has no files to read, and /v1/work returns only the kanban table (work_execplans.rs:58).

Fix: set it on the daemon, not in your shell, and restart.

Meters is empty

Two distinct empty states, and the page distinguishes them:

MessageCauseFix
cost lens off or unreachableCORECRUXD_FEATURE_COST_LENS offEnable it
/v1/cost/report: reachable, emptyFlag on, nothing postedRun corecruxctl session cost --post

The Activity log is empty

Cause: CORECRUXD_FEATURE_ACTIVITY_LOG off, a warning banner names it (render.js:12026).

Fix: enable it. The flag gates both GET /v1/activity and the live stream.

"Go live" shows "Live (reconnecting)"

Cause: the EventSource on /v1/events/stream dropped. Usually a proxy buffering or timing out server-sent events.

Fix: configure the proxy to pass SSE unbuffered. The backfill and Reload paths are unaffected.

Studio › Library is empty

Cause: the daemon caches the catalogue index; nothing fetches it automatically. There is deliberately no refresh button, because there is no fetch route.

Fix: corecruxctl studio sync (render.js:8691).

8.5 Data and identity

A tenant vanished

Check, in order:

  1. Memory › Memory, is hide system tenants on? Tenants prefixed __ are hidden by default (pages.js:218).
  2. Memory › Facts, turn reserved __\* on and search the tenant id.
  3. Memory › Facts, set as of to a time before it disappeared. If it is there, the facts were retired, not lost.
  4. Turn superseded on. A consolidation supersedes target facts; they stop resolving but remain in the store.
  5. Confirm you are on the same data directory. The default data_dir is relative, so two starts from different working directories give two divergent stores.

The most likely cause is (1) or (5). A tenant does not disappear from the store on its own.

The daemon reports a different identity and receipts stop verifying

Cause: passport.key is missing and a new one was generated silently.

Fix: there is none. It cannot be recovered. It also encrypts stored integration credentials through a derived subkey, so those are gone too. This is why chapter 9 exists.

A second daemon started against the same data

Cause: the LOCK flock is per resolved path. Two starts from different working directories against the default relative data_dir do not collide; they diverge.

Fix: always set an absolute CORECRUXD_DATA_DIR.

A repo scan is marked failed: "daemon restarted before scan completed"

Cause: expected restart recovery, not corruption.

Fix: re-run the scan.

8.6 Network and API

An MCP client cannot connect after the session handshake

Cause: channels.mcp from POST /session points at port 14800, where no /mcp route exists.

Fix: use port 14801. See chapter 16 B3.

A gRPC call returns unimplemented

Cause: every registered RPC does, in every build from this repository.

Fix: use HTTP. See chapter 16 B5.

POST /v1/admin/append returns a platform-upgrade response

Cause: it always does. The dataplane pool is hard-coded to None. It is not a subscription problem and no flag enables it.

Fix: use the fact and local-ingest surfaces.

A route you expected to be blocked is not

Cause: route-auth defaults to shadow mode; it logs route_auth_shadow_mismatch and continues.

Fix: CORECRUXD_ROUTE_AUTH=enforce. Expect console routes to start returning 403 when you do, and soak it first.

8.7 Logging

Logs are not JSON despite setting CORECRUX_LOG_FORMAT=json

Cause: no code reads that name. Every shipped manifest sets it anyway.

Fix: LOG_FORMAT=json.

Log level changes have no effect

Cause: RUST_LOG overrides CORECRUXD_LOG_LEVEL entirely when set.

Fix: unset RUST_LOG, or set the level there.

8.8 When you have fixed it

Two things, both cheap.

Do not retry blindly. Read the logs, journalctl -u corecruxd, or the container logs, before the next attempt. A retry against a full disk produces a second identical failure and no information.

Record it. A one-paragraph incident note with the symptom, the cause, the fixing change and the reproduction steps is the difference between an incident you solve once and one you solve every quarter.

Sources