From 9c75cfe08e1f80d31406e94ad0229adcf69f73d9 Mon Sep 17 00:00:00 2001 From: dchaudhari7177 <111210939+dchaudhari7177@users.noreply.github.com> Date: Wed, 19 Aug 2026 16:54:13 +0530 Subject: [PATCH 1/2] docs: add a status and doctor troubleshooting matrix status and doctor emit a lot of fields and no guidance on which combinations mean something is wrong. Add a matrix plus a worked section per state, each with the next safe action. Covers the five states the issue names -- plugin installed with no observed evidence, Shadow Mode active, incomplete hook coverage, enforcement not earned, and stale or drifted evidence -- and adds CODEX_NOT_FOUND, because that one makes every status number meaningless and should be checked first. Example output was generated by running both commands against a scratch data root rather than written by hand, so the field names and shapes match what diagnostics.py actually emits. Identifiers were then replaced with synthetic placeholders; no real repository hash appears. The distinctions worth having in writing, all of which read the wrong way round at first glance: - capability: Tool Enforcement appears on a fresh install. It is what the integration can do, not what is in force; authority.effective is that. - an empty effective_blockers does not mean enforcement is happening, only that nothing blocks the configured mode -- which is shadow. - doctor reports capability_label: Observe when it cannot confirm control, degrading to what it can prove. - when ledger.valid is false, status falls back to summarizing raw records, so counts stay populated while EVIDENCE_ROOT_UNVERIFIED blocks promotion. Populated counts are not verified evidence. Nothing implies enforcement where the runtime fails open: the doctor section states that effective_policy.effective false means actions proceed. Documentation only. Linked from docs/index.md under Operations, and links out to the Codex integration, the integration labels and the privacy docs. Closes #35 --- docs/index.md | 1 + docs/operations/troubleshooting.md | 168 +++++++++++++++++++++++++++++ 2 files changed, 169 insertions(+) create mode 100644 docs/operations/troubleshooting.md diff --git a/docs/index.md b/docs/index.md index 3cf6acf..3c3d6d8 100644 --- a/docs/index.md +++ b/docs/index.md @@ -34,6 +34,7 @@ MARGINAL documentation is organized by user intent instead of keeping every guid ## Operations - [Privacy](operations/privacy.md) +- [Troubleshooting status and doctor](operations/troubleshooting.md) - [Website operations](operations/website.md) - [Codex plugin submission](operations/codex-plugin-submission.md) - [Privacy notice](../PRIVACY.md) diff --git a/docs/operations/troubleshooting.md b/docs/operations/troubleshooting.md new file mode 100644 index 0000000..8cfefbe --- /dev/null +++ b/docs/operations/troubleshooting.md @@ -0,0 +1,168 @@ +# Troubleshooting status and doctor + +`marginal status` and `marginal doctor` answer two different questions. +`status` reports what MARGINAL has observed and what authority it currently +has. `doctor` reports whether the local Codex integration can observe anything +at all. + +Read `doctor` first. If Codex is not reachable, every number in `status` is +zero for a reason that has nothing to do with your evidence. + +Every identifier below is synthetic. Repository hashes are truncated to +`repo-alpha…` style placeholders; a real one is a 64-character hex digest. + +## The matrix + +| What you see | What it means | Next safe action | +| --- | --- | --- | +| `hook_state: not_observed`, `evidence_records: 0` | The plugin is installed but has never seen a session. | Run a Codex session in this repository. Nothing is wrong yet. | +| `mode: shadow`, `authority.effective: L0` | Shadow Mode. MARGINAL is recording and recommending only. | Nothing. This is the correct state until enforcement is earned. | +| `coverage_ratio` below `1.0`, `COVERAGE` in `next_promotion_blockers` | Some actions were not coverable by a hook. | Keep running sessions. Check `doctor` for `hooks_enabled: false`. | +| `authority.eligible: L0` with a non-empty `next_promotion_blockers` | Enforcement is not earned. | Clear the named blockers. Do not configure Enforce Mode to force it. | +| `stale_session_receipts` above `0`, or `ledger.valid: false` | Session receipts were left behind, or the governance ledger did not verify. | Read `ledger.error_codes` and `ledger.first_invalid_sequence`. | +| `doctor` → `blocking_reasons: ["CODEX_NOT_FOUND"]` | Codex CLI was not found. | Install Codex, then re-run `doctor`. `status` numbers are meaningless until this clears. | + +## Plugin installed, no observed evidence + +``` +$ marginal status +mode: shadow +capability: Tool Enforcement +repository_hash: repo-alpha000000000000000000000000000000000000000000000000000000 +hook_state: not_observed +hooks_observed: False +hooks_active: False +evidence_records: 0 +covered_actions: 0 +coverable_actions: 0 +coverage_ratio: 0.0 +``` + +`hook_state` has three values, and the difference matters: + +- `not_observed` — no hook event has ever arrived. +- `observed` — events arrived previously, no session is live now. +- `active` — at least one session is live (`active_hook_sessions` above zero). + +A fresh install sits at `not_observed`. That is not a fault. Run a Codex +session; if it stays `not_observed` afterwards, the hooks are not firing and +`doctor` is where to look. + +Note that `capability: Tool Enforcement` appears even here. It describes what +the Codex integration is *capable* of, not what is currently in force — that +is `authority.effective`. + +## Shadow Mode is active + +``` +$ marginal status +mode: shadow +authority: {"ceiling": "L3", "configured_mode": "shadow", "current": "L0", + "effective": "L0", "effective_blockers": [], "eligible": "L0"} +``` + +Shadow Mode records and recommends; it does not block. `effective: L0` is the +authority actually in force. + +`effective_blockers` being empty does **not** mean enforcement is happening. It +means nothing is *blocking* the configured mode from taking effect — and the +configured mode is `shadow`. Read `configured_mode` and `effective` together. + +## Incomplete hook coverage + +``` +$ marginal status +hook_state: active +active_hook_sessions: 1 +covered_actions: 34 +coverable_actions: 41 +coverage_ratio: 0.8292682926829268 +next_promotion_blockers: ["COVERAGE", "MINIMUM_REVIEWS"] +``` + +`coverage_ratio` is `covered_actions / coverable_actions`. Below `1.0`, some +actions Codex took were not seen by a hook. + +This is the concrete reason Codex is labelled Tool Enforcement rather than +Full Compute Enforcement: specialized and hosted tool paths can fall outside +local hook coverage. Partial coverage is expected, not necessarily a +misconfiguration. Treat a sudden drop as the signal, not a value below 1. + +## Enforcement is not earned + +``` +$ marginal status +authority: {"ceiling": "L3", "configured_mode": "shadow", "current": "L0", + "effective": "L0", "eligible": "L0"} +next_promotion_blockers: ["MINIMUM_ACTIONS", "MINIMUM_SESSIONS", "COVERAGE", + "MINIMUM_REVIEWS", "OUTCOME_UNOBSERVABLE", + "EVIDENCE_ROOT_UNVERIFIED"] +``` + +`eligible` is what the evidence would currently support; `ceiling` is the most +the configuration would ever allow. Promotion needs `eligible` to reach `L3`, +which needs `next_promotion_blockers` empty. + +Each blocker is a separate requirement — clearing one does not shorten the +others. `OUTCOME_UNOBSERVABLE` and `EVIDENCE_ROOT_UNVERIFIED` in particular are +not about volume: the first means outcomes could not be classified, the second +that the evidence root did not verify. + +There is no flag that skips this. That is the point of Earned Enforcement. + +## Stale or drifted evidence + +``` +$ marginal status +stale_session_receipts: 3 +ledger: {"error_codes": ["IO_ERROR"], "first_invalid_sequence": null, + "records": 0, "root_hash": null, "valid": false} +permissions: {"evidence": "not_created", "governance_ledger": "not_created"} +``` + +Two different problems share this shape. + +`stale_session_receipts` counts sessions that started and never cleanly +finished — usually a crashed or killed Codex process. They age out; a +persistent non-zero count means sessions are not terminating cleanly. + +`ledger.valid: false` is more serious. Read the two fields next to it: +`first_invalid_sequence` names the record where verification failed, and a +`null` there with `records: 0` and `IO_ERROR` means the ledger could not be +read at all — usually it does not exist yet, which is normal before the first +session. A non-null `first_invalid_sequence` means the chain broke at a +specific record, and the evidence after it is not trustworthy. + +When the ledger does not verify, `status` falls back to summarizing raw +records, so the counts stay populated while `EVIDENCE_ROOT_UNVERIFIED` blocks +promotion. Do not read populated counts as verified evidence. + +## Codex is not reachable + +``` +$ marginal doctor +available: False +version: +hooks_enabled: False +plugins_enabled: False +capability_level: observe +capability_label: Observe +blocking_reasons: ["CODEX_NOT_FOUND"] +``` + +`doctor` reports `capability_label: Observe` when it cannot confirm control, +even though the Codex integration is Tool Enforcement when working. The label +degrades to what can be proven, not what is intended. + +`effective_policy.effective: false` is the field that says enforcement is not +in force. When it is `false`, the runtime fails open: actions proceed. Nothing +in `status` should be read as blocking while that is the case. + +## Related + +- [Codex plugin](../integrations/codex.md) — what the integration observes + and controls +- [Integration overview](../integrations/overview.md#integration-labels) — + Observe, Tool Enforcement, Full Compute Enforcement +- [Privacy](privacy.md) — what the Decision Ledger stores and what + `ledger-export` shares From 8051fe27a96b8cd29acb78d8db41900e74893c80 Mon Sep 17 00:00:00 2001 From: dchaudhari7177 <111210939+dchaudhari7177@users.noreply.github.com> Date: Thu, 20 Aug 2026 11:45:07 +0530 Subject: [PATCH 2/2] docs: correct three claims the troubleshooting matrix overstated All three were me asserting more than the code supports. coverage_ratio. Both sides of covered_actions / coverable_actions are sums over decision records already in the evidence store, so an action that never reached a hook produces no record and lands in neither -- it is invisible to the ratio, not counted against it. "Actions Codex took were not seen by a hook" described the one thing the number cannot see. Reworded to what it does summarize, with the limit stated outright: 1.0 is equally consistent with complete coverage and with a hook that stopped firing, and doctor is the tool for that question. Also noted that the 0.0 on a fresh install is the zero-denominator fallback in diagnostics.py, not a measurement of zero coverage. stale_session_receipts. There is no TTL. _active_hook_sessions counts a receipt stale when _live_session_repository returns None -- unreadable file, symlink, oversized, non-loopback host, token under 16 bytes, or a loopback probe that does not come back ok inside 0.25s. "They age out" said the count resolves by waiting; it resolves by removing the file. Added the two consequences that follow: a slow session is counted stale for that run, and stale is not filtered by repository the way active_hook_sessions is. CODEX_NOT_FOUND. status reads evidence persisted on disk and keeps reporting it whether or not the Codex CLI is on PATH right now, so "status numbers are meaningless until this clears" was wrong. Split the two questions explicitly -- doctor asks whether new activity can be observed, status reports what was already recorded -- and named the failure mode that actually applies: reading an unchanging status as a quiet period rather than a broken integration. Same correction on the first matrix row: not_observed with zero evidence is the absence of evidence, and an uninstalled plugin prints exactly the same thing, so it cannot be described as "the plugin is installed". --- docs/operations/troubleshooting.md | 69 ++++++++++++++++++++++++++---- 1 file changed, 60 insertions(+), 9 deletions(-) diff --git a/docs/operations/troubleshooting.md b/docs/operations/troubleshooting.md index 8cfefbe..34e7ea6 100644 --- a/docs/operations/troubleshooting.md +++ b/docs/operations/troubleshooting.md @@ -15,12 +15,12 @@ Every identifier below is synthetic. Repository hashes are truncated to | What you see | What it means | Next safe action | | --- | --- | --- | -| `hook_state: not_observed`, `evidence_records: 0` | The plugin is installed but has never seen a session. | Run a Codex session in this repository. Nothing is wrong yet. | +| `hook_state: not_observed`, `evidence_records: 0` | No hook event has ever arrived here. Expected on a fresh install — but identical to the plugin not being installed at all. | Run a Codex session in this repository, then look again. If it does not move, run `doctor`. | | `mode: shadow`, `authority.effective: L0` | Shadow Mode. MARGINAL is recording and recommending only. | Nothing. This is the correct state until enforcement is earned. | -| `coverage_ratio` below `1.0`, `COVERAGE` in `next_promotion_blockers` | Some actions were not coverable by a hook. | Keep running sessions. Check `doctor` for `hooks_enabled: false`. | +| `coverage_ratio` below `1.0`, `COVERAGE` in `next_promotion_blockers` | Of the decisions MARGINAL recorded, some were not covered. Says nothing about actions that never reached a hook. | Keep running sessions. Check `doctor` for `hooks_enabled: false`. | | `authority.eligible: L0` with a non-empty `next_promotion_blockers` | Enforcement is not earned. | Clear the named blockers. Do not configure Enforce Mode to force it. | -| `stale_session_receipts` above `0`, or `ledger.valid: false` | Session receipts were left behind, or the governance ledger did not verify. | Read `ledger.error_codes` and `ledger.first_invalid_sequence`. | -| `doctor` → `blocking_reasons: ["CODEX_NOT_FOUND"]` | Codex CLI was not found. | Install Codex, then re-run `doctor`. `status` numbers are meaningless until this clears. | +| `stale_session_receipts` above `0`, or `ledger.valid: false` | Receipt files exist whose session did not answer, or the governance ledger did not verify. | Remove receipts you know are dead. For the ledger, read `ledger.error_codes` and `ledger.first_invalid_sequence`. | +| `doctor` → `blocking_reasons: ["CODEX_NOT_FOUND"]` | Codex CLI was not found *now*. No new activity can be observed until it is. | Install Codex, then re-run `doctor`. Evidence already on disk stays valid and `status` still reports it. | ## Plugin installed, no observed evidence @@ -48,6 +48,17 @@ A fresh install sits at `not_observed`. That is not a fault. Run a Codex session; if it stays `not_observed` afterwards, the hooks are not firing and `doctor` is where to look. +What this state does **not** tell you is whether the plugin is installed. +`not_observed` with `evidence_records: 0` is the absence of evidence, and a +machine where the plugin was never installed prints exactly the same thing. +`doctor` is what distinguishes them — it probes the integration rather than +reading what the integration has already recorded. + +`coverage_ratio: 0.0` here is the same kind of artefact. The ratio is +`covered_actions / coverable_actions`, and with a zero denominator it is +reported as `0.0` rather than left undefined. It means "nothing recorded", not +"nothing covered". + Note that `capability: Tool Enforcement` appears even here. It describes what the Codex integration is *capable* of, not what is currently in force — that is `authority.effective`. @@ -80,8 +91,22 @@ coverage_ratio: 0.8292682926829268 next_promotion_blockers: ["COVERAGE", "MINIMUM_REVIEWS"] ``` -`coverage_ratio` is `covered_actions / coverable_actions`. Below `1.0`, some -actions Codex took were not seen by a hook. +`coverage_ratio` is `covered_actions / coverable_actions`. Read what those two +numbers are counted over: both are sums across the decision records already in +the evidence store. Below `1.0` means *of the decisions MARGINAL recorded*, +some were marked coverable and not covered. + +It is worth being exact about the limit, because the intuitive reading is +wrong. An action that never reached a hook produces no decision record, so it +lands in neither the numerator nor the denominator — it is invisible to this +ratio rather than counted against it. `coverage_ratio` therefore summarizes the +decision evidence MARGINAL observed. It is not a measure of total Codex runtime +coverage, and it cannot tell you what it never saw. A ratio of `1.0` is +consistent with complete coverage and equally consistent with a hook that +stopped firing. + +Use it as a within-evidence quality signal, and use `doctor` for the separate +question of whether the integration can observe activity at all. This is the concrete reason Codex is labelled Tool Enforcement rather than Full Compute Enforcement: specialized and hosted tool paths can fall outside @@ -122,9 +147,23 @@ permissions: {"evidence": "not_created", "governance_ledger": "not_created"} Two different problems share this shape. -`stale_session_receipts` counts sessions that started and never cleanly -finished — usually a crashed or killed Codex process. They age out; a -persistent non-zero count means sessions are not terminating cleanly. +`stale_session_receipts` counts receipt files under `sessions/` whose session +did not answer when `status` asked it. Reachability is the whole test: the +receipt is read, and the session behind it is probed over loopback with a short +timeout. A receipt counts as stale if the file cannot be read, if it fails its +safety checks (a symlink, oversized, a non-loopback host, a token under 16 +bytes), or if the probe does not come back `ok`. + +There is **no time-based TTL**, so these do not age out on their own. A receipt +left by a crashed or killed Codex process stays counted until the file is +removed. A non-zero count means "these receipts point at nothing reachable +right now" — treat it as a prompt to clear dead receipts, not as something that +resolves by waiting. + +Two consequences worth knowing: a session that is merely slow to answer within +the probe timeout is counted stale for that run, and the stale count is not +filtered by repository — unreachable receipts from any repository on this +machine are included, while `active_hook_sessions` counts only this one. `ledger.valid: false` is more serious. Read the two fields next to it: `first_invalid_sequence` names the record where verification failed, and a @@ -158,6 +197,18 @@ degrades to what can be proven, not what is intended. in force. When it is `false`, the runtime fails open: actions proceed. Nothing in `status` should be read as blocking while that is the case. +`doctor` and `status` are answering different questions here, and this is the +state where the difference shows. `doctor` probes the integration now: can +MARGINAL observe new Codex activity? `CODEX_NOT_FOUND` answers no. `status` +reads evidence already persisted on disk for this repository, which is +unaffected — the counts, the ledger and the promotion blockers keep reporting +what was recorded before, and they remain valid. + +So do not read `CODEX_NOT_FOUND` as invalidating `status`. Read it as: the +existing record still stands, and nothing will be added to it until Codex is +reachable again. The stale reading to guard against is treating an unchanging +`status` as evidence of a quiet period rather than of a broken integration. + ## Related - [Codex plugin](../integrations/codex.md) — what the integration observes