fix: dispatch-marker scan logs invalid_sidecar forever; reconcile disabled by two config gaps

[tbrownio]

---
type: bug-report
id: daemon-log-hygiene
status: ready
zone: 1
severity: medium
pr:
artifact_bundle: https://linear-agent.bloomapi.com/a/JVLw3W4XmKu88bmAkLSaiA/
---

Intent: Stop the daemon's dispatch-marker scanner from re-logging the same stale marker files every ~5s forever (~100k journald lines/day observed), and switch on the two reconcile safety nets that are silently inert on the production host.

Root causes are confirmed: the degraded log fires on every scan for both emit reasons (invalid_sidecar, no_parent_turn) before the ingest dedup, and nothing cleans up bundles whose owning session ended; the Linear apps were never granted the admin scope webhook reconcile needs; PLANNER_APP_ACTOR_ID / IMPLEMENTER_APP_ACTOR_ID were never set on the host.

Locked: D1 log-once per marker (both reasons) · D2 age-based quarantine sweep (~24h, move never delete) · D3 the env + admin-scope items are manual ops with a deploy-gate; AC6 conditional on Linear honoring admin for client_credentials tokens.

Verification: AC1–AC4 daemon regression tests (pnpm test); AC5–AC6 operator journalctl checks post-deploy. Full Justification from the Socratic gate is in the bundle's item.md.

Artifact bundle: https://linear-agent.bloomapi.com/a/JVLw3W4XmKu88bmAkLSaiA/

[tbrownio]

Postmortem — daemon-log-hygiene (ops-only)

Run operations (always)

Wall-clock 0:49 (2026-07-23 22:48:24 → 23:37:26 UTC), agent-active ≈ 100%, human-idle 0:00 (0%) — the only genuine human message in the transcript is the initial /do invocation; the run chained Load → Plan → Implement → Verify → PR → Review+QA → Wrap-up in one uninterrupted turn. Zero stalls, zero nudges, no post-completion idle (wrap-up ran immediately after QA publication). No AskUserQuestion gates, no rate-limit hits; all sub-agent waits were productive (detached Codex dispatches polled in-turn).

Per-step timing (scripted from the session transcript; tokens from each dispatch's tokens used log line; est. cost is not computed — GPT-5.6-sol rates are not in model-prices.md, so unknown rather than a guess):

Step / dispatch	Start	End	Dur	Tokens	Note
Step 0 load + preflight	22:48:24	22:50:42	2m18s	—	incl. stale-marker pickup, ff to origin/main
code-researcher	22:50:42	22:52:42	2m00s	87,894	dossier; Overseer read code in parallel
plan writing (Overseer)	22:52:42	22:55:31	2m49s	—	plan.md authored
plan-reviewer p1	22:55:31	22:56:27	56s	46,263	3 MF + 1 SF
plan-reviewer p2	22:57:37	22:58:30	53s	42,633	3 MF (2 new)
plan-reviewer p3	22:59:24	23:00:18	54s	45,101	1 MF (trivial cwd wording)
implementer r1	23:00:58	23:07:25	6m27s	158,378	full slice, self-checked green
backend-verifier v1	23:07:55	23:09:28	1m33s	38,545	fail — rubric failure-path blocker
implementer r2	23:09:45	23:11:18	1m33s	31,550	failure-injection test
backend-verifier v2	23:11:34	23:12:56	1m22s	34,127	pass
Step 4 gate/commit/PR	23:13:12	23:14:44	1m32s	—	PR #117 opened
code-reviewer p1	23:14:44	23:18:53	4m09s	70,244	1 MF + 1 SF; fully overlapped with Overseer authoring/hosting the Visual-overview diagram
implementer r3	23:19:27	23:21:54	2m27s	39,439	race guard + memo clear
code-reviewer p2	23:22:12	23:23:48	1m36s	59,592	MF-1 persists (narrower)
implementer r4	23:24:59	23:27:15	2m16s	34,248	per-move gate + memo prune
code-reviewer p3	23:27:35	23:29:19	1m44s	49,738	Approve — 0 MF
QA backend-verifier	23:29:41	23:31:40	1m59s	63,831	all command-shaped items green
QA publication	23:31:40	23:35:36	3m56s	—	incl. ~3m30s self-inflicted loss (see below)
Step 6 wrap-up	23:35:36	23:37:26	1m50s	—	wrapup, bundle PUT, label, notify

Aggregates: Load 5% · Plan 21% · Implement+Verify 25% · PR 3% · Review+QA 42% · Wrap 4%. Summed overseer turnaround between dispatches ≈ 6 min total across 14 dispatches (fast pickup; the in-turn poll loops kept every gap under ~30s). Overseer main-loop usage: output 95,221 / cache-read 13.41M tokens across 89 messages.

The one defect worth recording: during QA publication the Overseer edited the PR body through an unquoted bash heredoc; the markdown's backticked strings executed as command substitution, running the full daemon test suite twice concurrently (two of those incidental runs flaked on timeouts, forcing a serial re-run to prove they were load-flakes) and aborting the intended edit. Cost: ~3.5 minutes + one redundant suite run. The fix that removes the biggest (only) stall-shaped loss: always use quoted heredocs (<<'EOF') when piping markdown/code through the shell.

What we asked for

Stop the dispatch-marker scanner re-logging the same stale markers every ~5s (~100k journald lines/day), quarantine stale handled bundles (move-never-delete, ~24h age), and capture the two host config gaps (APP_ACTOR_ID vars, Linear admin scope) as operator deploy actions — D3 ops-only, AC6 conditional.

Outcome vs intended

On-target — no outcome gap known at wrap-up. AC1–AC4 verified with regression tests plus rubric blockers; AC5/AC6 are post-deploy operator checks by design. The outcome half of this postmortem runs after the human's PR review.

Why the gap happened

Omitted — operational-only postmortem.

What to change so it doesn't recur

1. claude/skills/do/SKILL.md (Step 5, evidence/publication bullets) — add one line to the body-update instruction: "Shell-quoting rule for body edits: when piping a PR body or comment containing backticks through bash (heredocs, python3 - <<), always quote the delimiter (<<'EOF') — an unquoted heredoc executes the markdown's backticked strings as commands." Evidence: this run lost ~3.5 min and ran the test suite twice concurrently from exactly this mistake.
2. claude/skills/postmortem/references/model-prices.md (or wherever dispatch cost rates live) — add GPT-5.6-sol token rates so Codex dispatch costs stop being recorded as unknown; every /do run on this stack currently emits token counts with no cost column.

Both proposals are recorded for human review — nothing applied.

Dial record & right-sizing

zone: 1
lanes: single-codex
passes: {plan: 3/3, post_pr: 3/3}
findings: {plan: {pass1: {codex: 4, claude: 0}, later: {codex: 4, claude: 0}},
           post_pr: {pass1: {codex: 2, claude: 0}, later: {codex: 2, claude: 0}}}
verifiers: {frontend: skipped, qa_pass: ran}
qa_findings: 0
wall_clock: 0:49
deviations: none
pr_size: {files_changed: 9, additions: 526, deletions: 20}
tokens:
  codex: {total: 801583, by_role: {code_researcher: 87894, plan_reviewer: 133997,
          implementer: 263615, backend_verifier: 136503, code_reviewer: 179574}}
  claude_subagents: 0
  overseer: 13748203   # deduped by message.id; output 95221, cache_read 13411420
  total: 14549786
spend_ratio: 26648.9   # cache-read-dominated; Codex-only ratio: 1468.1
agents:
  - {role: code-researcher, model: gpt-5.6-sol, effort: low, dispatches: 1, wall_clock: "2:00", tokens: 87894}
  - {role: plan-reviewer, model: gpt-5.6-sol, effort: low, dispatches: 3, wall_clock: "2:43", tokens: 133997}
  - {role: implementer, model: gpt-5.6-sol, effort: medium, dispatches: 4, wall_clock: "12:43", tokens: 263615}
  - {role: backend-verifier, model: gpt-5.6-sol, effort: low, dispatches: 3, wall_clock: "4:54", tokens: 136503}
  - {role: code-reviewer, model: gpt-5.6-sol, effort: low, dispatches: 3, wall_clock: "7:29", tokens: 179574}

Right-sizing: right-sized. Every pass was trigger-driven and every fix round answered a concrete Must Fix; the two lowest-yield spends were plan pass 3 (45,101 tokens → one trivial wording finding, but it was the confirmation pass after a materially revised plan) and post-PR pass 3 (49,738 tokens → 0 findings, but it was the mandatory re-review that closed the loop). No dial change would have cut spend without giving up a required confirmation; the zone-1 single-Codex lane at cap 3/3 fit this item.

Acceptance

Still awaiting review (operations-only postmortem; the outcome half runs when the human returns from PR review).

System changes

Comment URLs: recorded after posting (see the postmortem comments on issue #115 and PR #117). Proposal verdicts: pending human review — this auto-run does not wait.

[tbrownio]

Postmortem — daemon-log-hygiene (outcome follow-up)

Acceptance: merged as-is — zero rework commits after the awaiting-human-review label (merged 2026-07-24T17:41Z). Outcome: on-target — deployed the same day via daemonctl reload (cecbfbf → 023e928, health accepted); live journald shows the old process spamming invalid_sidecar every ~5s until 18:10:30 UTC and zero degraded lines since, via the durable source_key suppression. Quarantine correctly deferred: the two currently-stale bundles are under the 24h threshold (AC3) and will sweep ~03:46/07:10 UTC.

Post-deploy observation for a possible follow-up item: the two degraded bundles have off-convention basenames (…-449434-qa.done, backend-verifier-qa-…) — a dispatcher deviating from <role>-<epoch>-<pid>-<seq> is what invalidates their sidecars. AC5 (*_APP_ACTOR_ID) and AC6 (admin-scope gate) remain open operator items.