Skip to content

feat: event-driven RunPane orchestration#363

Open
parsakhaz wants to merge 4 commits into
mainfrom
issue-356-event-orchestration
Open

feat: event-driven RunPane orchestration#363
parsakhaz wants to merge 4 commits into
mainfrom
issue-356-event-orchestration

Conversation

@parsakhaz

@parsakhaz parsakhaz commented Jul 23, 2026

Copy link
Copy Markdown
Member

Summary

What. Gives RunPane an event-driven supervision surface: an orchestrator can block on one command until a panel becomes idle, needs input, gets blocked, or exits — across many panels at once — instead of polling panels screen and inferring transitions itself.

Why. A long-lived Pane Chat orchestrator had no way to sleep until something meaningful happened. It re-read unchanged terminal output on a timer, burning tool calls, context tokens and wall-clock while risking missed blockers between polls. panels wait --for ready made this worse: isCliReady latches permanently after the CLI's first response, so ready returned while the agent was still actively working.

How. Three layers, landed as three commits (not squashed):

Commit Content
9626f77 Semantic state model — orthogonal terminalReady / agentActivity / inputRequired / blocked, configurable byte-silence debounce, surfaced additively in existing responses
96e8bd6 Event log — bounded in-memory ring, global monotonic cursor, panels events / watch / await
2992bc1 Multi-target supervision + verified startup — await-any, panes watch --include-future-panels, panes status --changed-since, guarded agent startup

Everything is additive. panels wait is frozen and unchanged (D2); npm, Python and the generated contracts stay in parity (D7).

Scope compression. The epic's five phases were compressed at the owner's request: phases 1 and 2 shipped on their own plans, phases 3 and 4 were folded into a single plan and commit, and phase 5 was cut. See Not done below.

Refs #356this PR does not close the issue; the deferred items below remain open under it.

Visual overview

Visual overview: none — this change has no user-visible surface. It is a CLI/daemon contract change; nothing in the Pane UI renders differently.

flowchart LR
  subgraph before["Before — polling"]
    O1[Orchestrator] -->|"panels screen (loop)"| P1[Panel]
    P1 -->|full screen text| O1
    O1 -.->|"infers transitions,<br/>misses blockers between polls"| O1
  end
Loading
flowchart LR
  subgraph after["After — event-driven"]
    T[PTY bytes] --> S[Semantic state machine<br/>terminalReady · agentActivity<br/>inputRequired · blocked]
    S --> R[(Bounded event ring<br/>epoch:n cursor)]
    R -->|broadcast| W["panels watch / await<br/>await-any · panes watch"]
    R -->|replay --since| W
    R -->|pull| E["panels events<br/>panes status --changed-since"]
    W --> O2[Orchestrator]
    E --> O2
    O2 -.->|"heartbeat reconcile<br/>(dropped event resolves)"| R
  end
Loading

User journeys

J1 — Supervise one agent without polling. runpane panels await --panel <id> --event agent-idle --timeout-ms 600000 blocks silently and returns the moment the agent goes idle, needs input, gets blocked, or exits — naming which via matchedEvent. Exit 0 matched, 2 timed out.

J2 — Supervise a whole workstream. runpane panes watch --pane <id> --include-future-panels --jsonl streams one compact JSON record per semantic transition across every panel in the pane, including panels created after the watch started. Zero records while nothing changes.

J3 — Resume after a disconnect. Re-issue with --since <last cursor>. Retained events replay strictly after that cursor, in order, duplicates identifiable by identical ids. A cursor predating an app restart, or aged out of the ring, stops the command with a structured cursor_expired naming the earliest available cursor and the snapshot command to reconcile from — never a silent empty stream. Exit 3.

J4 — Create a pane and know the agent really started. runpane panes create ... --start-agent --wait-active returns ok: true only after the prompt was verifiably accepted and the agent was observed becoming active afterwards.

J5 — Hit an interstitial. (fork) An allowlisted reversible prompt (the Codex optional-update prompt) under --handle-known-interstitials safe is answered, listed in the response, and the original prompt submitted exactly once. Otherwise — auth, permissions, destructive confirmations, terms, payments, directory trust — the flow stops and emits input_required, regardless of the flag.

Verification

Re-run on current head (2992bc1), after rebase onto a551c1c (v2.4.34):

Check Result
pnpm typecheck pass — all 4 workspaces
pnpm lint pass — 0 errors, 165 warnings (all pre-existing no-console in frontend)
PATH=…node@22/bin:$PATH CI=true pnpm --filter main test pass — 46 files / 442 tests
node scripts/test-runpane-contract.js pass — generated files up to date, CLI contract checks passed
pnpm --filter runpane test (wrapper suite, new) pass — 15 tests

Nothing is failing.

Note on Node: the main suite requires Node 22. Under a newer ambient Node the prebuilt better-sqlite3-multiple-ciphers ABI mismatches and four unrelated DB tests fail — an environment artifact, not a code failure.

Epic ACs proven, by named test

Phase 1 — semantic state model (main/src/ipc/runpane.test.ts, main/src/services/terminalPanelManager.test.ts)

AC Named test
AC1 AC1 reports terminal readiness and active agent activity as distinct screen fields
AC2 AC2/AC3 transitions active to idle at configured debounce %dms, not N-1
AC3 same, parameterized over 1000ms and 45000ms
AC4 AC4 persists exited activity without an intermediate semantic idle and preserves the legacy idle edge

Phase 2 — event log, cursors, watch/await (main/src/services/runpaneEventLog.test.ts, packages/runpane/src/localControl.watch.test.ts)

AC Named test
AC1 AC1 remains blocked while ready and active, then resolves agent-idle on an idle transition
AC2 AC2/AC3 watch emits one compact complete JSONL record for one transition and none while unchanged
AC3 same test (zero-record half)
AC4 AC4 replays strictly after a cursor in order and preserves duplicate-identifying ids + AC4 merges an event broadcast during replay exactly once in numeric cursor order
AC5 AC5 returns cursor_expired with the earliest cursor when retention ages a cursor out
AC6 AC6 rejects foreign pre-restart, future, and malformed cursors after runtime re-bootstrap
AC7 AC7 heartbeat recovers suppressed %s delivery and marks reconciliation (state-backed and edge-only)
AC8 AC8 timeout returns current reconciled state and distinct exit 2

Phase 3 — multi-target supervision

AC Named test
AC1 phase3 AC1: await-any identifies the winning panelId and paneId
AC2 phase3 AC2: panes watch include-future-panels emits panel_created and later transitions
AC3 phase3 AC3: panes status changed-since returns changed panels and cursor for gap-free watch

Phase 4 — verified startup + safe interstitials

AC Named test
AC1 phase4 AC1: high-level start ok includes verifiedSubmitted true and active agentActivity
AC2 phase4 AC2: create returns submission_unverified with attempt count after bounded staged retries
AC3 phase4 AC3: safe allowlisted interstitial is handled before the original prompt is submitted exactly once
AC4 phase4 AC4: directory-trust interstitial with safe flag emits input_required and performs no PTY write

Not done

Deferred, nothing silently dropped. Issue #356 stays open for these.

Epic phase 5 — all four ACs unsatisfied

  • Phase 5 AC1panels events emit --type <ns>.<name> --data-file producing durable, namespaced, schema-versioned, trust-labelled milestones (D5). Not implemented; no milestone table, no emit command.
  • Phase 5 AC2 — warning severity separation: an mcp_authentication-style advisory must report blocking: false and emit no blocked event while the agent keeps producing output. Not implemented; warnings are not classified today.
  • Phase 5 AC3 — an unchanged warning must not re-emit. Not implemented (follows AC2).
  • Phase 5 AC4panels output --panel --since <output-cursor> --limit for incremental output. Not implemented.

A reviewed plan for all four exists in tmp/356/plan-3.md (Group C), including the resolved decision that panels output --since should read persisted session_outputs rows rather than scrollback (row ids are the only monotonic identity; scrollback is one synthetic record), and a two-namespace cursor design so durable milestone cursors are never compared against the in-memory lifecycle cursor.

Also cut

  • Idempotent panes create retry (a caller-supplied request key). The obvious design is unsound: creation is enqueued before a session exists, the queue unconditionally adds a job, the session UUID is generated inside creation, and sessions has no request-key column or uniqueness constraint — so two concurrent retries would both enqueue. A correct version needs an atomic durable reservation with its own migration. Not attempted rather than half-built. This is not an epic AC.

All ACs from phases 1–4 are satisfied (table above).

Manual tests

  • Start a real Codex agent and confirm panels await --event agent-idle returns when it finishes, not when it becomes ready.
  • Confirm the 60s idle debounce default feels right for Claude Code and Codex; tune agentIdleDebounceMs if not.
  • Drive a real directory-trust prompt with --handle-known-interstitials safe and confirm the session survives and input_required is emitted.
  • Run a long panes watch against a busy multi-panel workstream and check payload volume vs the polling baseline (the epic's dogfood metric).

QA results

No app-driving QA pass was run — this change has no UI surface and every AC is command-shaped, proven by the named tests above. The four manual items are left to the human.

Deploy notes

  • No schema changes. The one migration this epic contemplated (runpane_milestones) belongs to the deferred phase 5.
  • New config key, additive and optional: AppConfig.agentIdleDebounceMs, default 60000. Existing installs get the default; no migration.
  • No new runtime dependencies. vitest added as a devDependency of packages/runpane so the wrapper suite can run.
  • CI: the new pnpm --filter runpane test suite is not wired into .github/workflows/quality.yml — a one-line follow-up, kept out of an already-large diff.
  • Nothing production-facing; no secrets, no one-time scripts, no backfills.

Residual risks

  • The interstitial gate is defense-in-depth, not type-enforced. All create-time automatic input flows through a single CreateInputGuard chokepoint, and the terminal manager additionally refuses to deliver create-owned input from its delayed CLI-ready callback. A future author could still bypass it by calling writeToTerminal directly from a new create path. Four review passes each found a bypass in a different create path before this was centralized — treat new create-time write paths as security-relevant.
  • Byte-silence idle detection reads an agent thinking silently past the debounce as a false idle. Mitigated by the configurable debounce and heartbeat reconciliation, and validated empirically (both flagship TUIs emit continuously while working) — but it is a heuristic, not a semantic signal.
  • Two attention truths coexist: renderer-facing activityStatus on its untouched 30s timer, orchestrator-facing agentActivity on the configurable one. Accepted per D4; migrating renderer notifications onto the canonical layer is a named follow-up.
  • The ring is in-memory and dies with the app — by design (D3); a pre-restart cursor yields cursor_expired pointing at snapshot reconciliation.

@parsakhaz parsakhaz added the awaiting-human-review Run complete; awaiting human PR review label Jul 23, 2026
@parsakhaz

Copy link
Copy Markdown
Member Author

Wrap-up — Epic 356, event-driven RunPane orchestration

PR: #363 · Branch: issue-356-event-orchestration · Size: 40 files, +5855/-132

What shipped

Three commits on a rebase onto v2.4.34:

  • 487257b — semantic panel state model (epic phase 1)
  • 5b4a93f — event ring, cursors, panels events/watch/await (epic phase 2)
  • a64a384 — multi-target supervision + verified startup (epic phases 3–4, compressed)

Scope change mid-run

The epic owner collapsed phases 2–5 into one remaining phase partway through, with a
stated priority order and permission to land a working subset. Phase 2 was already in
flight and completed on its own plan; phases 3–5 were replanned as one compressed plan
(plan-3.md) with a single review pass. Phase 5 was reached last and cut, per that
ordering — recorded under "Not done" in the PR body with a plan for finishing it.

Dial record

Dial Value
zone 1 (epic — full machinery, no escalation)
review lanes single (Codex)
plan passes phase 1: 3/3 · phase 2: 3/3 · compressed phase: 1/1 (owner-compressed)
post-implementation review passes phase 1: 2 · final diff: 4 (all security-driven)
QA pass command-shaped only; no UI surface
frontend verifier not run — no user-visible surface

Findings yield

Plan review found 20 Must Fixes on phase 2's plan alone and 4 on the compressed plan —
including two that were architecturally wrong (a module-scoped event log that would have
survived daemon re-bootstrap, and a hasNewOutput definition that was not computable from
the fields it exposed).

Code review found six D6 interstitial bypasses across four passes, each in a different
create-time write path. Gating call sites one at a time did not converge; the fix was to
centralize behind a single CreateInputGuard chokepoint plus an ownership marker the
terminal manager refuses to deliver. This is the run's most important outcome: without it,
a no-wait Codex create landing on a directory-trust prompt would have blindly typed into
it — the exact failure that destroyed a real session during the epic's research.

Two AC tests written by the implementer were caught asserting around the behavior they
claimed to prove (one fabricated both initialInputSentAt and an already-active snapshot;
another used staged/interstitial strings that the old buggy implementation would also have
passed). Both were rewritten.

Token spend

Role Dispatches Tokens
code-researcher 3 ~447k
plan-reviewer 7 ~683k
implementer 7 ~3.35M
code-reviewer 5 ~510k
backend-verifier 2 ~157k
Codex total 24 ~5.15M
Claude main loop unknown (not exposed this session)

Spend ratio: ~880 tokens per changed line, against a 5855-line diff.

Notes for the reviewer

  • The interstitial gate is defense-in-depth, not type-enforced. New create-time write paths
    are security-relevant; see Residual risks in the PR.
  • pnpm --filter runpane test is new and not yet wired into CI — one-line follow-up.
  • Phase 5's plan (milestones, warning severity, panels output --since) is written and
    reviewed in tmp/356/plan-3.md Group C, including the resolved decision that
    --since should read persisted session_outputs rows rather than scrollback.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2992bc1a80

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread main/src/services/runpaneBlockerDetection.ts Outdated
Comment thread packages/runpane/src/localControl.ts
Comment thread main/src/services/runpaneInterstitials.ts
@parsakhaz

Copy link
Copy Markdown
Member Author

Postmortem — 356 (ops-only)

Postmortem — 356 (ops-only)

Run operations (always)

Wall-clock span: 2026-07-22 20:45 → 2026-07-23 07:46 UTC = 661 min (11.0 h), 561 transcript events.
Human-idle: ~0 min (0.0%). Only three human messages arrived all run, each answered within seconds. Post-completion idle: 0 — the run ended on its own output.

On the waiting axis this run was effectively fully autonomous. The interesting failures are elsewhere: one autonomy leak that happened to cost nothing, and one hung dispatch that cost 25 minutes.

Per-step timing

Step / dispatch Start Dur Tokens Note
Preflight + load (gh, deps, item, artifacts) 20:45 ~8m branch reset to origin/main; 3 artifacts harvested
code-researcher · phase 1 13:48 6m 124,474 full-lane dossier
plan-1 authoring (overseer) 13:55 ~13m
plan-reviewer · p1 pass 1 14:18 8m 98,461 6 Must Fix
plan-reviewer · p1 pass 2 14:26 8m 98,197 3 Must Fix
plan-reviewer · p1 pass 3 14:33 7m 89,236 1 Must Fix
implementer · phase 1 14:18 10m 191,059
backend-verifier · phase 1 14:30 3m 78,555 all 4 ACs pass
code-reviewer · p1 pass 1 14:34 6m 131,473 2 Must Fix
implementer · p1 fix 1 14:39 7m +153,919
code-reviewer · p1 pass 2 14:47 6m 129,150 0 Must Fix — loop exits
commit 9626f77 (phase 1) 14:51
code-researcher · phase 2 14:52 9m 179,786 ran parallel with commit
plan-2 authoring (overseer) 15:00 ~18m
plan-reviewer · p2 pass 1 15:00 25m unknown ⚠️ HUNG — killed manually, output lost
plan-reviewer · p2 pass 1b 15:20 7m 76,940 9 Must Fix
plan-reviewer · p2 pass 2 15:27 7m 116,351 8 Must Fix
plan-reviewer · p2 pass 3 15:34 7m 129,760 3 Must Fix
implementer · phase 2 15:41 26m 415,009 largest single dispatch
commit 96e8bd6 (phase 2) 16:29
code-researcher · phases 3-5 16:11 9m 142,423 overlapped phase-2 implementer
plan-3 authoring (overseer) 16:20 ~15m compressed plan, owner scope change
plan-reviewer · p3 pass 1 (only) 16:49 6m 74,110 4 Must Fix
implementer · phase 3+4 16:53 15m 281,560 Group C cut
implementer · p3 AC-tests 17:10 7m +154,360 exposed 2 real bugs
backend-verifier · final 17:18 3m 71,323 1 fail (order-dependent)
code-reviewer · final pass 1 17:18 4m 181,733 4 Must Fix, 2 security
implementer · fix round 2 17:24 6m +86,861
code-reviewer · fixscope 1 23:56 4m 120,674 2 Must Fix, 2 security
implementer · fix round 3 00:03 6m +230,630
code-reviewer · fixscope 2 00:12 3m 118,357 3 Must Fix, 2 security
implementer · fix round 4 (centralize) 00:17 17m +174,278 CreateInputGuard
code-reviewer · fixscope 3 00:35 5m 161,889 1 Must Fix, 1 security
implementer · fix round 5 00:44 6m +84,212
Build gate, rebase, push, PR 00:50 ~12m PR #363

Aggregates. Phase 1 ≈ 62 min (9%). Phase 2 ≈ 97 min (15%), plus 25 min lost to the hung dispatch. Phases 3+4 including five security fix rounds ≈ 200 min (30%). Overseer authoring/turnaround between dispatches ≈ 165 min (25%) — the single largest non-dispatch cost, dominated by writing and revising three plans by hand.

gantt
  title Epic 356 — dispatch timeline (parallelism vs serialization)
  dateFormat HH:mm
  axisFormat %H:%M
  section Phase 1
  research            :13:48, 6m
  plan review x3      :14:18, 22m
  implement           :14:18, 10m
  verify + review x2  :14:30, 23m
  section Phase 2
  research            :14:52, 9m
  plan review HUNG    :crit, 15:00, 25m
  plan review x3      :15:20, 21m
  implement           :15:41, 26m
  section Phase 3+4
  research (overlap)  :16:11, 9m
  plan review x1      :16:49, 6m
  implement           :16:53, 15m
  AC tests            :17:10, 7m
  section Security fix loop
  review 1 + fix      :17:18, 12m
  review 2 + fix      :23:56, 13m
  review 3 + fix      :00:12, 22m
  review 4 + fix      :00:35, 15m
Loading

Stalls (autonomy leaks), ranked

  1. One turn-end needing a "continue" nudge, at ~06:35 local, immediately before the scoped security re-review. The overseer ended its turn with work remaining and did not schedule a self-wakeup. Cost was ~0 min only because the human happened to be present; overnight this would have stalled indefinitely. This is the single most important operational finding — the /do skill's own rule ("ending a turn with work remaining requires a ScheduleWakeup") was violated by its own overseer.
  2. Hung Codex dispatch, ~25 min lost. The plan-reviewer phase-2 pass-1 dispatch ran past its 900 s watchdog and had to be killed by hand; its output was lost and the pass re-run from scratch. Root cause below.

Blocker inventory

  • AskUserQuestion gates: 0. No red-tier gate was hit; no production surface, no schema change, no secrets.
  • Rate-limit hits: 0.
  • Legitimate background waits: 24 Codex dispatches, ~3.5 h of productive dispatch time — not leaks.
  • Mid-run human input: 2 owner scope messages (base-branch correction; phase compression). Both were genuine input answered inline, not stalls.
  • Tooling failures: 1 (the hung dispatch); 1 timeout binary absent on macOS, worked around inline.

The single change that would have removed the biggest stall

Fix the watchdog in the codex skill. Its documented launcher is
perl -e 'alarm shift; exec @ARGV' 900 codex exec … — but exec replaces the Perl process with Node, and the pending SIGALRM never fires against it, so the watchdog is silently inert. Every hung dispatch in the fleet runs unbounded until a human notices. Replacing it with a background-kill watchdog (which this run switched to mid-flight, with no further hangs across 14 subsequent dispatches) removes the failure class entirely.

What we asked for

Epic #356: an event-driven supervision surface for RunPane so a long-lived orchestrator can block until something meaningful happens instead of polling — five phases, zone: 1, one PR.

Outcome vs intended

Deferred to the human's PR review. Phases 1–4 shipped with all fifteen of their ACs proven by named tests; phase 5 (four ACs) was cut under an owner-directed compression with an explicit priority order that anticipated exactly this. The compression was the owner's call mid-run, so "fell short" is not the right frame — but whether the shipping subset is the right subset is theirs to judge at #363.

Why the gap happened

Omitted — operational-only postmortem. The outcome half runs after the human reviews #363.

What to change so it doesn't recur

1. Fix the Codex watchdog (highest value — silent failure across every run)

dcouple/orchestraclaude/skills/codex/SKILL.md, step 2 launcher.

-#!/usr/bin/env bash
-perl -e 'alarm shift; exec @ARGV or die "exec failed: $!"' <cap> \
-  codex exec -m gpt-5.6-sol -c model_reasoning_effort="<effort>" --yolo \
-  [--ephemeral] --skip-git-repo-check -C <repo root> \
-  -o <owner dir>/<name>.md "$(cat <owner dir>/<name>.prompt)" </dev/null
-status=$?
+#!/usr/bin/env bash
+# NOTE: `perl -e 'alarm N; exec ...'` does NOT work — exec replaces Perl with
+# Node and the pending SIGALRM never fires, leaving the dispatch unbounded.
+codex exec -m gpt-5.6-sol -c model_reasoning_effort="<effort>" --yolo \
+  [--ephemeral] --skip-git-repo-check -C <repo root> \
+  -o <owner dir>/<name>.md "$(cat <owner dir>/<name>.prompt)" </dev/null &
+cpid=$!
+( sleep <cap>; kill -9 $cpid 2>/dev/null ) &
+wpid=$!
+wait $cpid; status=$?
+kill $wpid 2>/dev/null

2. Make the self-wakeup rule mechanical, not advisory

dcouple/orchestraclaude/skills/do/SKILL.md, Autonomy & safety.

 - **A phase or step boundary is not a turn boundary.** …
+  **Mechanical check before ending any turn:** if a dispatch is outstanding, or
+  any plan/phase checkbox is unticked, the turn MUST end with a `ScheduleWakeup`
+  call in the same message. Ending a turn with neither the run complete nor a
+  wakeup scheduled is a pipeline defect, not a judgment call — an overseer that
+  "expects to continue immediately" is exactly the case that stalls overnight.

3. Give the code-reviewer an explicit "enumerate the call sites" instruction for safety-critical invariants

dcouple/orchestraclaude/agents/code-reviewer.md.

Four consecutive review passes each found a D6 bypass in a different create-time write path. Each reviewer verified the paths it was pointed at and stopped; none asked "what is the complete set of paths that can reach a PTY write?". The fix only converged when the overseer ordered centralization.

+## Invariant-shaped findings
+
+When a finding is an invariant that must hold on EVERY path ("never write
+without classifying", "never emit twice", "always hold the lock"), do not stop
+at the path you were shown. Enumerate the complete set of call sites that can
+reach the guarded operation, list them in the report, and state for each whether
+it is guarded. If the set cannot be enumerated from the code, say so — that
+unenumerability IS the finding, and the fix is a chokepoint, not another patch.

4. Have the implementer's charter forbid tests that fabricate the state under test

dcouple/orchestrareferences/agents/implementer/instructions.md.

Three separate tests in this run asserted around their own behaviour: one fabricated both initialInputSentAt and an already-active snapshot to "prove" verified startup; another chose staged/interstitial strings that the buggy global-replacement implementation would also have passed. Each was caught by review, but only after shipping through a verifier.

+## Tests must be able to fail
+
+A test that fabricates the state it claims to verify proves nothing. Before
+writing an assertion, ask: would this test FAIL against the pre-fix code? If it
+would pass either way, it is not a regression test. Specifically: do not
+hand-construct the snapshot/flag that the code under test is supposed to derive;
+drive the real callback and let the code produce it. When a fix is about
+ordering, the test must exercise the ordering.

5. Record that plan-review yield stays high through pass 3 on large phases

dcouple/orchestrareferences/zones.md, The record.

Phase 2's plan needed all three passes and yielded 9 / 8 / 3 Must Fixes — no pass was wasted. This is evidence against trimming plan-review caps at zone 1 for large phases, and worth recording before some future tuning pass lowers them.

Dial record & right-sizing

Dial Value
zone 1 (epic — full machinery, no escalation)
review lanes single (Codex)
plan passes phase 1: 3/3 · phase 2: 3/3 (+1 hung) · phase 3: 1/1 (owner-compressed)
plan Must Fixes phase 1: 6/3/1 · phase 2: 9/8/3 · phase 3: 4
post-impl review passes phase 1: 2 · final diff: 4
post-impl Must Fixes phase 1: 2 → 0 · final: 4 → 2 → 3 → 1 (all security)
verifiers backend ×2; frontend-verifier not run (no UI surface)
QA pass command-shaped only
wall-clock 661 min (11.0 h), ~0% human-idle
pr_size 40 files, +5855 / −132
tokens — Codex ≈ 3.89 M (implementer 1.77 M · code-reviewer 843 k · plan-reviewer 683 k · code-researcher 447 k · backend-verifier 150 k) + 1 hung dispatch unknown
tokens — main loop output 237 k · cache read 51.6 M · cache write 840 k · input 4 k
spend_ratio ≈ 690 tokens per changed line

Agents roster: code-researcher (gpt-5.6-sol/low, ×3), plan-reviewer (medium, ×7), implementer (medium, ×9 across 3 sessions), code-reviewer (medium ×5, high ×1), backend-verifier (low, ×2). 24 dispatches total.

Judgment: review effort was right-sized, arguably underdone. Every single review pass — 7 plan, 6 code — returned Must Fixes; not one was pure spend. The final security loop is the proof: passes returning 4, 2, 3 and 1 Must Fixes, each a genuine bypass in a path the previous pass had not examined. Had the cap been enforced at 3, the run would have shipped the pass-4 bypass, in which a create timing out before CLI-ready would blindly type into a directory-trust prompt. The dial that would have changed this outcome is the post-PR loop cap — and the right lesson is not "raise the cap" but the code-reviewer charter change above (proposal 3): the cap was never the binding constraint, the reviewers' path-at-a-time framing was.

Acceptance

Still awaiting human review — PR #363 is open and labelled awaiting-human-review.

System changes

Comment URLs recorded below on publish. Approval verdicts pending — this postmortem was auto-run at wrap-up and does not wait for them.

@parsakhaz

Copy link
Copy Markdown
Member Author

The "Not done" items in the description are tracked in #364, so they survive #356 closing when this merges.

Status note for reviewers: this branch is now 2 commits behind main (#362 and release v2.4.35 landed during review) and there is real overlap in main/src/types/config.ts, frontend/src/types/config.ts, main/src/services/configManager.ts and packages/runpane/package.json — both changes add config keys. A rebase with conflict resolution is needed before merge, and the three unresolved P1 threads are being addressed first.

Phase 1 of the event-driven RunPane orchestration epic (#356).

Adds a canonical, agent-agnostic panel state layer in the main process and
surfaces it additively in the existing `panels screen` / `panels wait` JSON
responses, so an orchestrator can finally distinguish "the CLI has answered
once" (terminalReady) from "the agent is working right now" (agentActivity).

- New state fields: terminalReady, agentActivity
  (unknown|starting|active|idle|exited), inputRequired, blocked, hasNewOutput,
  outputGeneration, lastMeaningfulEventAt.
- Idle detection is a configurable byte-silence debounce
  (agentIdleDebounceMs, default 60s) running parallel to — not replacing —
  the existing fixed 30s activityStatus timer, so human notification timing
  is unchanged.
- Exit state is persisted onto the panel's customState so agentActivity
  reports "exited" after the live terminal record is torn down; all six
  terminal-map removal paths clear the semantic timer, and stale PTY
  callbacks are rejected by record identity.
- Blocker detection moves into the screen result over a fixed 80-line window,
  independent of the caller's --limit; `panels wait` reuses it and its
  matching semantics are unchanged.
- Contract schemas extended and regenerated; npm, shared, and Python
  artifacts stay in parity.

Refs #356
Phase 2 of the event-driven RunPane orchestration epic (#356).

Turns phase 1's semantic state into an observable event stream, so an
orchestrator can block until something meaningful happens instead of polling.

- Bounded in-memory event ring, daemon-host scoped via PaneRuntime, with a
  global monotonic `<epoch>:<n>` cursor that doubles as the event id. A
  re-bootstrap mints a fresh epoch, so a pre-restart cursor yields a
  structured `cursor_expired` naming the earliest available cursor and a
  snapshot reconciliation command — never a silent empty stream (D3).
- Semantic transitions emitted for panel_created, terminal_ready,
  prompt_staged, prompt_submitted, agent_active, agent_idle, input_required,
  blocked, unblocked, panel_exited and panel_archived. Blocker edges are
  observed continuously via a trailing-throttled scan reached by both visible
  and hidden terminals.
- Three new commands: `panels events` (pull replay, keeping polling
  first-class per D8), `panels watch --jsonl --since` (long-lived JSONL
  subscription), and `panels await --event` (exit-on-first-match). One
  retained socket buffers live frames across every replay, including
  heartbeats, so replay and live never race.
- Heartbeat reconciliation on a strictly periodic timer replays the ring from
  the processed cursor, so a dropped event resolves the wait instead of
  stalling to timeout (D9). Exit codes: 0 matched, 2 timed out, 3 cursor
  expired, 1 transport.
- `panels wait` is untouched and stays frozen (D2); everything is additive,
  with npm, Python and generated contracts in parity (D7).

Refs #356
Compressed final phase of the event-driven RunPane orchestration epic (#356),
folding the epic's original phases 3 and 4.

Multi-target supervision:
- `panels await-any` across repeated --panel/--pane targets, identifying the
  winning panelId and paneId.
- `panes watch --include-future-panels` extends the watched set when a panel is
  created inside the pane, without restarting the watch.
- `panes status --changed-since` returns only panels with transitions after the
  cursor, plus a snapshot cursor taken after the state read so a subsequent
  watch observes no gap (D3 snapshot-then-subscribe, D8 pull surface).
- Both wrappers open the retained stream before resolving pane targets, so
  events emitted during resolution are not lost.

Verified startup and safe interstitials:
- `panes create --start-agent --wait-active` reports ok:true only after
  verifiedSubmitted and an agent_active transition causally observed after the
  prompt was accepted — a rendering interstitial no longer counts as "active".
- All create-time automatic input flows through a single CreateInputGuard
  chokepoint that classifies the current screen deny-list-first immediately
  before every write and submit sequence. Consequential interstitials — auth,
  permissions, destructive confirmations, terms, payments and directory trust —
  are never auto-answered and emit input_required instead (D6). Only an
  explicit allowlist of reversible responses is handled, under caller opt-in,
  and the original prompt is then submitted exactly once.
- Readiness failure keeps the delivery premark and marks input as create-owned,
  so the terminal manager's delayed CLI-ready callback cannot deliver it
  unguarded after the create returns.

Repeatable target flags are declared in the canonical contract and mirrored in
both wrappers with parity coverage. Everything is additive; `panels wait` stays
frozen (D2).

Old phase 5 — milestones, warning severity separation, and `panels output
--since` — is not implemented; see the PR body.

Refs #356
…on startup

Three P1 findings from review on #363.

Consequential-prompt patterns required actual prompt structure. `permission`,
`delete`, `payment` and `billing` previously matched as bare keywords anywhere
on the recent screen, so ordinary agent output — "deleting obsolete files",
"reviewing permission handling" — tripped CreateInputGuard, emitted
input_required, and marked pane creation unsuccessful when no interstitial
existed. A consequential keyword now blocks only when it sits in an actual
interactive prompt: a trailing question, a [y/n] affordance, a confirm/press
directive, or a selection menu. Self-contained prompts (directory trust,
authentication) still match on their own, so the gate stays fail-closed.

The ongoing blocker scanner now reuses the interstitial classifier instead of
maintaining a second, weaker matcher. It previously recognised only the Codex
update prompt and the literal phrase "press enter to continue", so selection
menus, [y/n] prompts, directory trust, authentication and permission decisions
produced no blocked/input_required event after startup — leaving agent_idle
indistinguishable from "waiting for input", the exact confusion this epic
exists to remove. The codex-update blocker kind and its suggested command are
preserved.

`panes create --from-json` now carries the startup guarantees. --start-agent,
--wait-active and --handle-known-interstitials were applied only on the
non-JSON path, so a --from-json create reported success with no submission or
activity verification. Fixed in both the npm and Python wrappers with a named
test each, keeping wrapper parity.

Tests cover both directions per keyword family: a genuine auth, permission,
destructive, payment or directory-trust prompt still blocks, and prose merely
mentioning the keyword does not.

Refs #356
@parsakhaz
parsakhaz force-pushed the issue-356-event-orchestration branch from 2992bc1 to 2f67493 Compare July 23, 2026 08:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

awaiting-human-review Run complete; awaiting human PR review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant