fix: log dispatch-marker degradation once and quarantine stale bundles

[tbrownio]

Summary

What — The daemon's dispatch-marker scanner now logs dispatch_marker_ingest_degraded at most once per marker per daemon process (both reasons: invalid_sidecar and no_parent_turn), and an age-based sweep quarantines stale, already-handled dispatch bundles out of the scan path — moving every file that shares the marker basename, never deleting anything.

Why — Six stale marker bundles under ended sessions were re-logged on every ~5-second scan cycle, forever: ~100k junk journald lines/day on the sole production daemon host, drowning real signals (this obscured live debugging during the observability smoke test). The degraded log fired before the ingest dedup, and nothing ever removed bundles whose owning session would never run another turn.

How — D1: both emit sites are gated by a process-local per-marker memo (<sessionId>:<file>, reason-independent), and invalid_sidecar rescans are additionally suppressed by the durable agent_invocations.source_key lookup (new public EventLog.hasCodexInvocation), so suppression survives restarts. D2: inside the same scan loop, a bundle older than DISPATCH_QUARANTINE_AGE_MS (default 24h — far above any live consume window) that is already ingested or unresumable (no_parent_turn) is moved to DISPATCH_QUARANTINE_DIR (default <db-dir>/dispatch-quarantine/<sessionId>/, outside every scan path and outside the worktree so worktree cleanup can't destroy it). Siblings move first and .done moves last, so a partial failure self-heals on the next scan; destination collisions allocate a fresh suffixed name instead of overwriting. D3 (host env vars + Linear admin scope) is deliberately ops-only — no daemon code; see Deploy notes.

Done means: a degraded marker produces exactly one log line per daemon process; stale handled bundles leave .codex-dispatches/ for the quarantine directory with the event log's invocation record intact; young bundles are untouched; the operator has exact commands for the two host config gaps.

Visual overview

Before the scanner re-logs every stale marker on each 5-second cycle forever; after, gate 1 (per-process memo + durable SQLite source_key) limits each marker to one degraded line, and gate 2 moves handled bundles older than 24h — every basename sibling, .done last, never overwriting — to dispatch-quarantine/ beside the event DB, outside every scan path. (.excalidraw source in ./tmp/daemon-log-hygiene/refs/.)

User journeys

No user-facing surface — the "user" is the daemon operator. Operator journey: journalctl -u linear-agent-daemon is currently ~100k degraded-marker lines/day; after deploy, each stale marker contributes at most one line per daemon start, and within 24h the stale bundles move to <db-dir>/dispatch-quarantine/ where they remain inspectable. Reconcile safety nets switch on only after the operator applies the Deploy notes.

Verification

• AC1 (log-once, both reasons) — regression test logs each degraded dispatch marker once across rescans and reason transitions drives two ingestDispatches() cycles over fabricated invalid_sidecar, no_parent_turn, and reason-transition markers; exactly one dispatch_marker_ingest_degraded per marker. cd daemon && pnpm test: 364 passed | 1 skipped (final HEAD).
• AC2 (stale bundle fully quarantined) — test quarantines every stale ingested sibling while retaining its invocation: bundle incl. .sh sibling aged past threshold → every basename file moved to the quarantine dir, source dir empty, dispatch_marker_quarantined logged.
• AC3 (young bundle untouched) — test leaves a young ingested dispatch bundle in the live consume path: below-threshold bundle → all files remain, no quarantine dir created.
• AC4 (invocation record survives) — same AC2 test asserts log.invocations(owner) still returns the record with sourceKey: dispatch:<owner>:<basename>.done after the move.
• Failure path (background-job rubric blocker) — test logs a quarantine failure and completes the bundle on the next scan: a real filesystem obstruction forces the move to fail → dispatch_marker_quarantine_failed logged, bundle intact; obstruction removed → next scan completes the quarantine.
• AC5 / AC6 — not verified in this run: operator-manual post-deploy checks on the production host (red tier); see Manual tests + Deploy notes. AC6 is conditional on the Linear admin-scope deploy-gate.
• Gates — pnpm typecheck, pnpm build, bash -n over the seven ops scripts: all exit 0. Plan review: 3 Codex passes, all Must Fix resolved. Post-PR code review: 3 Codex passes ending Approve (0 Must Fix, 0 Should Fix open). Verification: independent backend-verifier passes at the verify stage and again as the final QA drive on HEAD af2841e.

Manual tests

Important (operator, production host, post-deploy):

• AC5 — left to human: production host (red tier) — populate PLANNER_APP_ACTOR_ID / IMPLEMENTER_APP_ACTOR_ID in /etc/linear-agent-daemon/env, restart, then journalctl -u linear-agent-daemon --since -2min | grep -c reconcile_sessions_skipped → 0
• AC6 — left to human: production host + Linear workspace admin (red tier) — after the admin-scope deploy-gate succeeds: journalctl -u linear-agent-daemon --since -2min | grep reconcile_webhook → success events only, zero reconcile_webhook_failed. If Linear rejects the grant (the runbook records a live 2026-07 rejection): file follow-up item webhook-reconcile-fallback and record AC6 blocked, not failed
• Post-deploy log hygiene — left to human: production host (red tier) — one hour of journald shows each stale marker at most once, and stale bundles appear under /var/lib/linear-agent-daemon/dispatch-quarantine/ within 24h

Areas not affected: webhook ingestion, turn execution, Linear activity posting, worktree lifecycle, operations CLI — no code paths outside scanDispatchMarkers(), config loading, and one new read-only EventLog accessor.

QA results

Automated QA executed all command-shaped items against final HEAD af2841e: full suite 364 passed | 1 skipped, including the seven dispatch-scan regressions (AC1–AC4, failure path, mid-loop race abort, memo prune); typecheck, build, and bash -n all exit 0. 0 of 3 Manual-test items were machine-runnable — all three are red-tier production-host operator checks, left to the human. Post-PR review found and fixed two issues on the PR: a quarantine/turn-open race (the open-turn predicate is now re-checked before every file move) and an unbounded log-once memo (now pruned). Proof in the QA evidence comment.

Deploy notes

Nothing blocks verification (AC1–AC4 are repo-local tests). Ordered for deploy on the production host linear-agent (GCE, project bloom-agents) — all items are operator-run; this pipeline touched no production resource:

1. ⛔ Deploy the daemon build — operator-managed per daemon/ops/runbook.md (daemonctl flow). No schema change: the new hasCodexInvocation is a read-only accessor; SQLite DDL untouched.
2. ⛔ AC5 — host env vars (red: production config):

   sudoedit /etc/linear-agent-daemon/env   # add:
   PLANNER_APP_ACTOR_ID=<planner app actor UUID>
   IMPLEMENTER_APP_ACTOR_ID=<implementer app actor UUID>
   sudo systemctl restart linear-agent-daemon
   journalctl -u linear-agent-daemon --since -2min | grep -c reconcile_sessions_skipped   # expect 0
   

3. ⛔ AC6 — Linear admin scope (red: Linear workspace admin action): grant admin to both apps (bloom-planner, bloom-implementer) per daemon/ops/runbook.md §"Register the two OAuth apps", then run the deploy-gate confirmation that client_credentials tokens honor admin for webhooks/updateWebhook. On rejection (the runbook documents Linear rejecting this live in 2026-07): file webhook-reconcile-fallback — "Decide and implement the webhook re-enable path when Linear rejects the admin scope on client_credentials tokens" — and record AC6 blocked.
4. Optional env (defaults are correct) — DISPATCH_QUARANTINE_AGE_MS (default 86400000) and DISPATCH_QUARANTINE_DIR (default <db-dir>/dispatch-quarantine) exist for tuning; no action needed.

Residual risks

• The quarantine directory grows without bound by design (move-never-delete); dead files only, operator-prunable.
• A no_parent_turn bundle older than 24h is quarantined even though a parent turn could theoretically appear later — locked decision (no event-log coupling); the bundle remains recoverable from quarantine by hand.
• Log-once state for no_parent_turn is per-process: a daemon restart re-logs each such marker once (bounded by restarts, and the 24h sweep removes the source).
• Accepted race bound: a turn opening mid-sweep can land at most one rename before the per-move open-turn re-check aborts the bundle; .done moves last so the remainder self-heals on a later scan, and every sweepable bundle's prompt/report already lives durably in SQLite.

Closes #115

[tbrownio]

QA evidence — automated pass on final HEAD af2841e

Backend-only change: no UI surface, so no browser drive or captures — all proof is quoted command output, re-runnable from daemon/.

AC1 — degraded marker logs exactly once (both reasons, incl. reason transition)

✓ logs each degraded dispatch marker once across rescans and reason transitions

AC2 — stale handled bundle fully quarantined (every basename sibling, incl. .sh)

✓ quarantines every stale ingested sibling while retaining its invocation — emits "event":"dispatch_marker_quarantined", source directory empty afterward

AC3 — young bundle untouched for the live consume path

✓ leaves a young ingested dispatch bundle in the live consume path

AC4 — event log retains the invocation record after quarantine

same AC2 test asserts sourceKey: dispatch:<owner>:<basename>.done still returned by log.invocations(owner) post-move

Failure path (background-job rubric blocker 2)

✓ logs a quarantine failure and completes the bundle on the next scan — real filesystem obstruction forces the move to fail (dispatch_marker_quarantine_failed, bundle intact), next scan completes the quarantine

Review-round regressions (found by the post-PR review lanes, fixed on this PR)

✓ leaves a stale bundle when its owner opens a turn during quarantine moves — per-move open-turn re-check aborts the sweep mid-bundle, later scan self-heals
✓ finishes a partial quarantine without overwriting archived siblings
✓ prunes a consumed marker from the degraded-log memo

Suite and gates

Test Files  26 passed | 1 skipped (27)
Tests       364 passed | 1 skipped (365)
pnpm typecheck → exit 0 · pnpm build → exit 0
bash -n ops/provision.sh ops/daemonctl ops/wait-for-daemon-health.sh \
  ops/claudex ops/claudex-fable ops/proxy-accounts.sh ops/codex-provider-gate.sh → exit 0

---

Passed automated: AC1, AC2, AC3, AC4 + rubric blockers (idempotent rescan, failure path, real-handler invocation) + typecheck/build/shell gates.

Remaining for the human (all red-tier production actions — exact commands in the PR body's Deploy notes):

• AC5 — populate both *_APP_ACTOR_ID values on the host, restart, confirm zero reconcile_sessions_skipped_missing_app_actor_id.
• AC6 — Linear admin-scope deploy-gate; on rejection file webhook-reconcile-fallback and record AC6 blocked.
• Post-deploy log-hygiene spot check (one line per stale marker; bundles under dispatch-quarantine/ within 24h).

[tbrownio]

---

type: wrap-up-report
item: daemon-log-hygiene
pr: #117

Wrap-Up Report — daemon-log-hygiene

What was built

The dispatch-marker scanner now emits dispatch_marker_ingest_degraded at most once per marker per daemon process (both reasons, reason-transition-proof, with invalid_sidecar rescans also suppressed durably via the SQLite source_key lookup), and an age-based sweep (default 24h, DISPATCH_QUARANTINE_AGE_MS) moves already-ingested or unresumable bundles — every basename sibling, siblings first and .done last, collision-safe, never deleting — to DISPATCH_QUARANTINE_DIR (default beside the event DB, outside every scan path). D3 (host *_APP_ACTOR_ID vars + Linear admin scope) stayed ops-only per the item's lock: no daemon code; the runbook gained an explicit post-provision checklist and the PR carries the exact operator commands.

Verification evidence

• AC1–AC4: seven dispatch-scan regression tests in daemon/test/sessions.test.ts (log-once incl. reason transition; full quarantine with .sh sibling; young-bundle non-interference; invocation retention; failure-path with induced move failure and next-scan self-heal; mid-loop open-turn race abort; memo prune). Full suite on final HEAD af2841e: 364 passed | 1 skipped; pnpm typecheck, pnpm build, bash -n over the seven ops scripts all exit 0 — independently proven twice by ephemeral backend-verifier dispatches (Step 3 and the QA drive).
• AC5/AC6: operator-manual post-deploy checks on the production host — red tier, never executed by this run; exact commands in the PR's Deploy notes. AC6 is conditional on the Linear admin-scope deploy-gate, which the runbook records as rejected live in 2026-07 — expect its blocked branch (file webhook-reconcile-fallback).

Review outcome

Plan loop: 3/3 passes, Must Fix 3 → 3 → 1 (last one a trivial command-cwd wording fix), all resolved, final confidence 9. Post-PR loop: 3/3 passes, ending Approve — Must Fix: 0, Should Fix: 0; the loop surfaced and fixed two real defects on the PR (quarantine racing a turn opened mid-sweep → per-move open-turn re-check; unbounded log-once memo → prune on quarantine and on marker disappearance). Verify stage caught one rubric gap (no induced failure-path test) — fixed and re-verified. QA drive: all command-shaped items executed and green; 0 of 3 Manual-test items machine-runnable (all red-tier production checks, annotated left to human in the PR body); 0 QA findings. No deferred Should Fix survivors — nothing to carry as inline comments.

Human action required

• ⛔ Blocks verification / QA (prerequisite): none — AC1–AC4 verified repo-locally.
• ⛔ You must do (deploy / external):
  1. Deploy the daemon build to linear-agent per daemon/ops/runbook.md (daemonctl flow).
  2. AC5 — sudoedit /etc/linear-agent-daemon/env, add PLANNER_APP_ACTOR_ID= and IMPLEMENTER_APP_ACTOR_ID=, sudo systemctl restart linear-agent-daemon, then journalctl -u linear-agent-daemon --since -2min | grep -c reconcile_sessions_skipped → 0.
  3. AC6 — grant admin to both Linear apps and run the runbook deploy-gate for webhooks/updateWebhook under client_credentials; on rejection (expected per the runbook's live 2026-07 observation) file webhook-reconcile-fallback and record AC6 blocked, not failed.
• ✅ Done for you (applied in-run): all code, tests, runbook checklist, PR body/diagram/QA evidence; no staging resources existed to touch — this pipeline touched no production resource.

Residual risks / follow-ups

• Quarantine directory grows unboundedly by design (move-never-delete); operator-prunable dead files.
• Accepted race bound: at most one rename can land after a turn opens mid-sweep (synchronous per-move re-check; .done-last self-heal; all sweepable data already durable in SQLite).
• A >24h no_parent_turn bundle is quarantined even if a parent turn could later appear (locked D2 decision; recoverable by hand).
• Follow-up item if the AC6 gate fails: webhook-reconcile-fallback — webhook re-enable path without Linear admin on client_credentials.

Dial record

zone: 1
lanes: single-codex
passes: {plan: 3/3, post_pr: 3/3}
findings: {plan: {pass1: {codex: 4, claude: 0}, later: {codex: 4, claude: 0}},   # Must+Should; MF only: 3 / 3+1
           post_pr: {pass1: {codex: 2, claude: 0}, later: {codex: 2, claude: 0}}} # pass2 re-raised both; pass3: 0
verifiers: {frontend: skipped, qa_pass: ran}   # no UI surface; QA = command-shaped set (zone-1 non-user-visible)
qa_findings: 0
wall_clock: 0:49          # transcript JSONL 2026-07-23T22:48:24Z -> 23:37Z
deviations: none
pr_size: {files_changed: 9, additions: 526, deletions: 20}
tokens:
  codex: {total: 801583, by_role: {code_researcher: 87894, plan_reviewer: 133997,
          implementer: 263615, backend_verifier: 136503, code_reviewer: 179574}}
  claude_subagents: 0     # no Claude sub-agents dispatched (single lane, no web research needed)
  overseer: 13748203      # all usage fields deduped by message.id; cache-read-dominated
                          # (output 95221, input 166, cache_creation 241396, cache_read 13411420)
  total: 14549786
spend_ratio: 26648.9      # total / 546 changed lines; overseer cache reads dominate the numerator
agents:
  - {role: code-researcher, model: gpt-5.6-sol, effort: low, dispatches: 1, wall_clock: unknown, tokens: 87894}
  - {role: plan-reviewer, model: gpt-5.6-sol, effort: low, dispatches: 3, wall_clock: unknown, tokens: 133997}
  - {role: implementer, model: gpt-5.6-sol, effort: medium, dispatches: 4, wall_clock: unknown, tokens: 263615}
  - {role: backend-verifier, model: gpt-5.6-sol, effort: low, dispatches: 3, wall_clock: unknown, tokens: 136503}
  - {role: code-reviewer, model: gpt-5.6-sol, effort: low, dispatches: 3, wall_clock: unknown, tokens: 179574}

Deltas vs plan

none — the final diff matches the plan's Files-changed table; the two post-PR fix rounds (race guard, memo prune) stayed within daemon/src/sessions.ts + daemon/test/sessions.test.ts and are recorded in the plan's Plan deltas.

[tbrownio]

Postmortem — daemon-log-hygiene (ops-only)

Run operations (always)

Wall-clock 0:49 (2026-07-23 22:48:24 → 23:37:26 UTC), agent-active ≈ 100%, human-idle 0:00 (0%) — the only genuine human message in the transcript is the initial /do invocation; the run chained Load → Plan → Implement → Verify → PR → Review+QA → Wrap-up in one uninterrupted turn. Zero stalls, zero nudges, no post-completion idle (wrap-up ran immediately after QA publication). No AskUserQuestion gates, no rate-limit hits; all sub-agent waits were productive (detached Codex dispatches polled in-turn).

Per-step timing (scripted from the session transcript; tokens from each dispatch's tokens used log line; est. cost is not computed — GPT-5.6-sol rates are not in model-prices.md, so unknown rather than a guess):

Step / dispatch	Start	End	Dur	Tokens	Note
Step 0 load + preflight	22:48:24	22:50:42	2m18s	—	incl. stale-marker pickup, ff to origin/main
code-researcher	22:50:42	22:52:42	2m00s	87,894	dossier; Overseer read code in parallel
plan writing (Overseer)	22:52:42	22:55:31	2m49s	—	plan.md authored
plan-reviewer p1	22:55:31	22:56:27	56s	46,263	3 MF + 1 SF
plan-reviewer p2	22:57:37	22:58:30	53s	42,633	3 MF (2 new)
plan-reviewer p3	22:59:24	23:00:18	54s	45,101	1 MF (trivial cwd wording)
implementer r1	23:00:58	23:07:25	6m27s	158,378	full slice, self-checked green
backend-verifier v1	23:07:55	23:09:28	1m33s	38,545	fail — rubric failure-path blocker
implementer r2	23:09:45	23:11:18	1m33s	31,550	failure-injection test
backend-verifier v2	23:11:34	23:12:56	1m22s	34,127	pass
Step 4 gate/commit/PR	23:13:12	23:14:44	1m32s	—	PR #117 opened
code-reviewer p1	23:14:44	23:18:53	4m09s	70,244	1 MF + 1 SF; fully overlapped with Overseer authoring/hosting the Visual-overview diagram
implementer r3	23:19:27	23:21:54	2m27s	39,439	race guard + memo clear
code-reviewer p2	23:22:12	23:23:48	1m36s	59,592	MF-1 persists (narrower)
implementer r4	23:24:59	23:27:15	2m16s	34,248	per-move gate + memo prune
code-reviewer p3	23:27:35	23:29:19	1m44s	49,738	Approve — 0 MF
QA backend-verifier	23:29:41	23:31:40	1m59s	63,831	all command-shaped items green
QA publication	23:31:40	23:35:36	3m56s	—	incl. ~3m30s self-inflicted loss (see below)
Step 6 wrap-up	23:35:36	23:37:26	1m50s	—	wrapup, bundle PUT, label, notify

Aggregates: Load 5% · Plan 21% · Implement+Verify 25% · PR 3% · Review+QA 42% · Wrap 4%. Summed overseer turnaround between dispatches ≈ 6 min total across 14 dispatches (fast pickup; the in-turn poll loops kept every gap under ~30s). Overseer main-loop usage: output 95,221 / cache-read 13.41M tokens across 89 messages.

The one defect worth recording: during QA publication the Overseer edited the PR body through an unquoted bash heredoc; the markdown's backticked strings executed as command substitution, running the full daemon test suite twice concurrently (two of those incidental runs flaked on timeouts, forcing a serial re-run to prove they were load-flakes) and aborting the intended edit. Cost: ~3.5 minutes + one redundant suite run. The fix that removes the biggest (only) stall-shaped loss: always use quoted heredocs (<<'EOF') when piping markdown/code through the shell.

What we asked for

Stop the dispatch-marker scanner re-logging the same stale markers every ~5s (~100k journald lines/day), quarantine stale handled bundles (move-never-delete, ~24h age), and capture the two host config gaps (APP_ACTOR_ID vars, Linear admin scope) as operator deploy actions — D3 ops-only, AC6 conditional.

Outcome vs intended

On-target — no outcome gap known at wrap-up. AC1–AC4 verified with regression tests plus rubric blockers; AC5/AC6 are post-deploy operator checks by design. The outcome half of this postmortem runs after the human's PR review.

Why the gap happened

Omitted — operational-only postmortem.

What to change so it doesn't recur

1. claude/skills/do/SKILL.md (Step 5, evidence/publication bullets) — add one line to the body-update instruction: "Shell-quoting rule for body edits: when piping a PR body or comment containing backticks through bash (heredocs, python3 - <<), always quote the delimiter (<<'EOF') — an unquoted heredoc executes the markdown's backticked strings as commands." Evidence: this run lost ~3.5 min and ran the test suite twice concurrently from exactly this mistake.
2. claude/skills/postmortem/references/model-prices.md (or wherever dispatch cost rates live) — add GPT-5.6-sol token rates so Codex dispatch costs stop being recorded as unknown; every /do run on this stack currently emits token counts with no cost column.

Both proposals are recorded for human review — nothing applied.

Dial record & right-sizing

zone: 1
lanes: single-codex
passes: {plan: 3/3, post_pr: 3/3}
findings: {plan: {pass1: {codex: 4, claude: 0}, later: {codex: 4, claude: 0}},
           post_pr: {pass1: {codex: 2, claude: 0}, later: {codex: 2, claude: 0}}}
verifiers: {frontend: skipped, qa_pass: ran}
qa_findings: 0
wall_clock: 0:49
deviations: none
pr_size: {files_changed: 9, additions: 526, deletions: 20}
tokens:
  codex: {total: 801583, by_role: {code_researcher: 87894, plan_reviewer: 133997,
          implementer: 263615, backend_verifier: 136503, code_reviewer: 179574}}
  claude_subagents: 0
  overseer: 13748203   # deduped by message.id; output 95221, cache_read 13411420
  total: 14549786
spend_ratio: 26648.9   # cache-read-dominated; Codex-only ratio: 1468.1
agents:
  - {role: code-researcher, model: gpt-5.6-sol, effort: low, dispatches: 1, wall_clock: "2:00", tokens: 87894}
  - {role: plan-reviewer, model: gpt-5.6-sol, effort: low, dispatches: 3, wall_clock: "2:43", tokens: 133997}
  - {role: implementer, model: gpt-5.6-sol, effort: medium, dispatches: 4, wall_clock: "12:43", tokens: 263615}
  - {role: backend-verifier, model: gpt-5.6-sol, effort: low, dispatches: 3, wall_clock: "4:54", tokens: 136503}
  - {role: code-reviewer, model: gpt-5.6-sol, effort: low, dispatches: 3, wall_clock: "7:29", tokens: 179574}

Right-sizing: right-sized. Every pass was trigger-driven and every fix round answered a concrete Must Fix; the two lowest-yield spends were plan pass 3 (45,101 tokens → one trivial wording finding, but it was the confirmation pass after a materially revised plan) and post-PR pass 3 (49,738 tokens → 0 findings, but it was the mandatory re-review that closed the loop). No dial change would have cut spend without giving up a required confirmation; the zone-1 single-Codex lane at cap 3/3 fit this item.

Acceptance

Still awaiting review (operations-only postmortem; the outcome half runs when the human returns from PR review).

System changes

Comment URLs: recorded after posting (see the postmortem comments on issue #115 and PR #117). Proposal verdicts: pending human review — this auto-run does not wait.