Repository navigation
Add a shared timeline assembler for structured agent chats (not wired yet) - #25064
Conversation
…al timeline folder Pure moves so a shared timeline assembler can use them: Codex's message ordinal counter becomes ProviderTurnMessageOrdinals and Claude's turn-row revision becomes the provider-neutral agent-journal turn-row revision. Only names and import paths change.
…found again after a restart - A sink transition is admitted whole or not at all; its steps run back to back at their turn in the journal's write queue, and each resolver reads the fold with every earlier write landed. A resolver may also say where the row belongs (turn scope, provider reference), and the writer always hears how the transition landed. A resolved lifecycle batch chooses its settlement mutations from the fold at execution. - New optional row field providerItemRef: the provider's own reference for the item a row is, written only where the row's identity cannot spell it (Codex keys messages by their place in the turn and renumbers its item ids on resume). Set by the creating write, kept by revisions, indexed by the journal fold, never read by clients. A downgrade test shows an older host and client render such rows unchanged. - Provider timeline identity schemes (shared legacy arm, Codex) and the join index that resolves a provider item to its row from memory or the fold: ordinals and request incarnations are read back from the rows, so a restart or an evicted entry finds the original row instead of placing a new one.
1ded8f5 to
f7eb1cb
Compare
… joins read from a replaced epoch A fresh join index continued a turn's messages at the first free place, so a journal holding only a later ordinal (an imported or removed earlier row) had its sequence back-filled. The place is now one past the highest ordinal any row or echoed send holds there, read through a pure scheme reader. The join caches also drop what they read when the journal's epoch is replaced.
8c1655a to
1c0fb11
Compare
…t its turn in the journal Adapters translate their provider's dialect into a small grammar (turns, items, streamed text, requests, context facts, session end/reset); one shared assembler turns it into the journal rows every structured lane writes. Each event is planned as one sink transition. Which row a write lands on, whether a replay writes anything, and every change to what the assembler knows (its ledger) are decided by the transition's resolvers at the event's turn in the journal's write queue, against the fold as it stands then. A forecast (the ledger plus admitted events still queued) only answers apply() at once. So a refused event allocates nothing, a write the journal rejects leaves no trace in memory, and a restart or evicted cache finds the same rows again. Text and full snapshots of one provider item share one row and one lifecycle; reset always flushes text and settles the old session from the journal; the open-work budget is derived from what is actually open. Codex migration contracts compare against the existing Codex translator, including a restart mid-stream and a repeat that outlives the join cache.
…nd background work A third review found two blockers with the earlier rounds' cause, a remembered interpretation trusted after the journal moved on: - A reused request id was judged by its earlier prompt's settled turn before asking which turn the new one lands in, so a real approval in a later turn was dropped. The target turn now decides: the old turn again is a replay; a different live turn opens the next prompt beside it. - A text stream checked its row's turn only on its first write, and a turn's end released streams by the turn planning expected. Every write now checks the row, a turn's end stops the streams whose rows are in it, and turn status reads the journal first, so another writer's Stop wins. Also: a message boundary drawn by an event the journal held as a replay no longer splits an anonymous message; a send naming a turn not yet open waits for that turn; the budget charges a stream's thread and turn strings and the turn caches are byte-bounded; the open turn ends when the journal shows it settled; a turn's opener is read from the journal's row. Background work is now Orca's existing background-task row instead of a tool call flagged `outlivesTurn` (a flag remembered only in memory, so a restart failed the task). A turn's end never settles that row, so it survives restarts; session end leaves one in flight unverifiable. Three tests that opened a background tool call with `outlivesTurn` now open a background-task row and keep their original expectations about which turn the row stays in.
1c0fb11 to
a9dbaaa
Compare
… streams A row kept for the anonymous stream that may continue it, and the marker that a stopped stream's queued writes write nothing, lived in the live-stream map and were never removed when no stream followed. They now live in their own bounded maps, so the live map holds open streams only.
Keeps both multi-row writes: the transition's per-row write (each built after the one before it folds) and main's all-or-nothing queued-rejection write. Main's ledger receipt rides the shared single-row write. A resolved lifecycle batch cannot carry rejectsQueued, so it is never silently dropped.
…b2025/acp-c1-timeline-assembler
…imeline-assembler
…b2025/acp-c1-timeline-assembler
…timeline-transitions
…m the transition PR Nothing in production reaches the state they defended (an assembler that lost its memory while its child keeps streaming the same turn), and the stored Codex id was positional. The legacy identity scheme moves to the assembler PR with its first caller; the Codex scheme and any persisted reference wait for Codex to move onto the assembler. The Codex ordinal counter goes back to codex/, since no neutral code imports it.
A settlement too large for one row now commits all its rows or none, through the journal's existing all-or-nothing write, instead of a row-by-row writer. Every row is built before any commits, so a settlement naming one item twice is refused before anything is written.
…d-forget Nothing reads which steps of a transition wrote. A failed step fails the sink, leaving the steps before it written; the header says so, and tests cover it plus a settlement whose second row fails inside the transaction. writeAgentJournalTurnRow returns nothing again, as on main.
A person's Stop settled the open turn's row without a word to the assembler. The assembler then forgot the turn: its running tools and pending prompts were never settled, the provider's own end and withdrawal were dropped, and the turn's text streams stayed counted against the open budget for the life of the process. Text the provider kept streaming afterwards could land as a message outside the stopped turn. - The open turn the journal shows settled ends first, as one transition, through the same settlement the provider's turn.end plans; its streams stop and their keys drop later text until that turn's end or the next turn opens. - turn.end and request.withdrawn are admitted for a turn or request the journal holds; their settlement writes nothing for rows already settled. - The budget's re-check frees streams whose turn settled. - A settled tool keeps its terminal body against any differing write. - Session-end settlement of lost background work uses the journal's own lostLiveWorkJournalBody instead of a copy. - The rig's window elapses before every read, and restart swaps and disposes the old assembler.
When another writer settles the open turn (a person's Stop), the assembler now only stops that turn's text and cancels its pending prompts. Running tool calls stay the agent's: a progress update or completion it reports after the Stop lands as reported, and whatever is still running settles at the agent's turn end for that turn, the next turn's open, or the session's end. An agent's end for an earlier turn while a newer one is open no longer clears the open turn's activity line or ends its anonymous reply. An unnamed end right after a Stop ends the stopped turn instead of being dropped. The test rig's restart no longer writes the dead assembler's window text, matching dispose.
…imeline-assembler
The type-aware lint requires an exhaustive switch with no default case. Also retitle a Stop test to say what it asserts.
…timeline-transitions
…imeline-assembler
…ansitions' into brennanb2025/acp-c1-timeline-assembler
Review summaryThe problem this PR addresses. Each agent Orca runs as a structured chat (Claude, Codex) has its own translator that turns what the agent does into chat-history rows, and each re-implements the same bookkeeping: open a turn, end it once, record how it ended, close out half-finished tool calls and approval prompts. New agents over the Agent Client Protocol (Grok first) would need a third copy. This PR adds one shared timeline assembler that owns that bookkeeping; an agent's adapter only reports events. User-facing change: none yet. Nothing calls the assembler (the Grok wiring in #25225 will). Main's own restart cleanup now shares two helper functions with it, with identical behaviour (main's suites pass). What changed during review. The assembler was rebuilt. The earlier version kept two copies of its state (a forecast and a ledger rebuilt from the journal), a provider-id join index and replay recognition, to survive the assembler losing its memory while the agent process keeps streaming. Review showed no production path reaches that state, so the rebuild uses:
Fixed during review (each with a test that fails without the fix):
Deferred.
Verified.
Not verified. No live agent, no Windows or Linux run, no headless or SSH runtime. Nothing is wired yet. |
…assembler The common pattern discards the history a provider replays while loading a session, so the assembler has no use for an input.history event.
Review follow-up to the adoption removal: drop the comment naming the provider's saved message, and make requestedAt required since every pending input comes from input.accepted.
- journal-store.ts imports: main #24576 dropped journalStoreLoadedFields; this branch types appendResolvedItem off JournalItemAppender, so neither it nor JournalResolvedItem is imported. - event-sink-queue: keeps this branch's journalItems; journalStopDecidesTurn takes main #24864's two arguments (openedBy plumbing removed on main).
…ansitions' into brennanb2025/acp-c1-timeline-assembler
|
Removed saved-history adoption ( What was removed: the Why: the common pattern discards the history an agent replays when it loads a saved session (keeping only context usage), so the assembler has no use for it. Nothing else changed. Verified: the 13 assembler test files plus #25141's transition test (14 files, 114 tests) and main's settlement suites that share code with this PR (8 files, 84 tests) pass; one gated node typecheck found only this machine's stale-install |
|
Merged current main after #25141 landed; head |
|
CI at 72873e3: every PR-owned check passes (static analysis + typecheck, mobile, packaging, 4 of 5 unit shards). The three red checks are main's own failures, identical on current main and not touched by this PR: (1) cross-version |
ELI5
The problem. In a structured chat, Orca turns everything an agent does into rows in the chat's saved history (the "journal"): its reply text, each tool call, approval prompts, and a "turn" row that records when the agent started and stopped working on one request. Claude and Codex each have their own converter, and each re-implements the same bookkeeping: open a turn, end it exactly once with the agent's verdict, and close out tool calls and prompts left half-finished when a turn ends or the agent process dies. The next agents (Grok, and others that speak the Agent Client Protocol, "ACP") would need a third copy, and every bug in that bookkeeping would need fixing three times.
This PR adds one shared piece, a timeline assembler, that owns the bookkeeping. An agent's adapter only reports what happened ("a turn started", "this tool call finished", "this text streamed"), and the assembler writes the rows. Nothing uses it yet, so there is no user-visible change.
Why it was rewritten. The earlier version of this PR kept a second copy of its own state (a "forecast" beside a "ledger") plus recovery logic that rebuilt what it knew from the journal and recognised replayed history, so that it could carry on if it lost its memory while the agent kept running. Production never reaches that situation: one assembler lives exactly as long as one agent process. When the process dies, Orca's existing dead-generation settlement (the "sweep" that closes out whatever a dead process left running) runs before the next process's events are written, and the next process gets a new assembler. The assembler never sees replayed history: when an agent reloads a saved session, the adapter drops what it replays (keeping only what it says about context usage), as the common pattern does. So the second copy and the recovery logic are gone, and with them the classes of bugs that three review rounds kept finding in them.
What Changed
User-facing change: none. The assembler is not wired into any agent or into the runtime. If an agent later opens more work at once than the assembler's budget (below), the user sees what a failed chat-history write shows today: the session ends, the turn shows as interrupted and the open tool calls as failed. No new message, banner or notice is added.
The mechanism. Everything lives in
src/main/native-chat/agent-session-timeline/and imports nothing from Electron, so it can run in the headless runtime.One state, changed only when an event is accepted (
provider-timeline-state.ts,provider-timeline-assembler.ts). Each provider event becomes one queued journal operation (Admit one provider event's journal writes as one queued operation, decided when it runs #25141's transition). What the assembler knows (the open turn, sends waiting for a turn, running tool calls and pending prompts, items it closed, the counter for ids it makes up) changes only when the queue accepts the event. The queue is first-in first-out, so the order events are accepted in is the order this assembler's rows are written in. An event the queue refuses changes nothing, takes no id, and is simply re-applied.Every decision that depends on the journal is made when the write runs. A resumed agent's events can be accepted before the chat's journal is attached, and the sweep for the previous process lands in between, so the assembler cannot decide from the journal at acceptance time. Each write reads the row it needs by its id when it runs:
turn-settled) until the agent ends that turn (by naming it, or with an unnamed end right after the Stop) or the next turn starts, so it can never land as a separate message outside the stopped turn;Facts other writers own (a person's Stop, a client's answer, the sweep) are read from the journal by key and never copied into memory.
A turn another writer ended stops its text and prompts; its tool calls wait for the agent. Any writer that settles a turn row while the agent still runs (for example a future Stop that marks the turn interrupted itself) is a writer the assembler must cope with. None exists on main today: ACP's Stop, in the Grok branch, sends a cancel to the agent and waits for the agent's own end of the turn. This is the contract for such a writer. Before handling the next event, the assembler sees the settled row and, in one queued operation: stops that turn's text streams (they stop counting against the budget), cancels its prompts that are still pending (the agent cannot use an answer to a cancelled turn), and leaves the turn row as that writer left it. It does not settle the turn's running tool calls. Agents may still finish a tool call between the cancel and their end of the turn, so a progress update or a "completed" the agent reports after the Stop lands as reported. Whatever is still running settles (as failed) at the agent's own
turn.endfor that turn, or when the next turn opens, or when the session ends. The agent's lateturn.endand prompt withdrawal are accepted, not dropped; only an end for a turn that neither this process opened nor the journal holds is dropped as unknown. An agent's end for an earlier turn, arriving while a newer turn is open, settles only that earlier turn: the open turn's activity line and its reply text are left alone.History of this paragraph: in the first revision the Stop left tool calls showing as running and prompts answerable, and each stopped reply kept a budget slot, so after about 128 stopped streams the session failed. The second revision settled the tool calls as failed at the Stop, which made a tool the agent completed just after the Stop show as failed, depending only on which arrived first. This revision follows the common pattern: tool calls end at the agent's own turn boundary.
Rows are found by spelling their ids, with no lookup cache (
provider-timeline-rows.ts,provider-timeline-identity.ts). Every row id is spelled from the agent's own ids (its turn id, tool-call id, message id, thread) in Orca's existinglegacyid format, so finding a row is spelling its id and reading it. Prompts are spelled with the process's generation (request:g:<generation>:<id>), because ACP's request ids restart at 0 with every new agent process. The spelling module moved here from Admit one provider event's journal writes as one queued operation, decided when it runs #25141 together with its first user.Text and full snapshots of one agent item share one row (
provider-timeline-text-streams.ts,-text-events.ts). A named stream is the item with that id on its agent thread; an anonymous stream becomes a new message each time it restarts. Every other event is an ordering point: text owed ahead of it is written first.text.closeanditem.closeboth settle the item, so a late delta cannot overwrite it.Bounded memory (
provider-timeline-budget.ts). One budget (128 entries, 1 MiB) covers running tool calls, pending prompts and open text streams. Before refusing, it checks each entry against the journal, so a prompt a client answered, or a text stream whose turn has ended, frees its room. Past the budget the event is refused asfailed, which ends the session the way a failed journal write does.One shared "how do you close this out" function (
journal-terminal-settlement.ts).terminalAgentJournalBody(a running tool call becomes failed, a pending prompt becomes cancelled) is now used by both the assembler's turn/session settlement and main's dead-generation settlement, which previously had its own private copy. Main's behaviour is unchanged; its tests pass as they were.Background work stays Orca's existing background-task row: a turn's end leaves it alone, its own updates settle it, and the session's end marks one still in flight
unverifiable(Orca lost sight of it), never "exited". The session's end useslostLiveWorkJournalBody, now exported from main'sjournal-subagent-liveness.ts, which is the same function the journal already applies to rows a dead host left behind when it reopens. The assembler's own copy is deleted. Its results now match main's exactly: no settled time is stamped, and a subagent still shown as working is markedunverifiabletoo.Test rig restarts the way production does.
rig.restart()now disposes the old process's assembler (text still in its batching window is lost, as in production), runs the dead-generation sweep, and makes a new assembler in a new generation the rig'sassembler(it is now async).rig.assemble()gives another assembler of the same generation for tests that need their own sink. The rig's text-batching window elapses before every read of the journal, so all text streamed so far is written. Before, only the first piece of text in a window was written, which could hide a bug.Who owns which part of the lifecycle
For the stacked branches. API kept:
createProviderTimelineAssemblerwithapply/flush/dispose/openTurnId,ProviderTimelineEvent,ProviderTimelineSink,providerTimelineSink,spellProviderTimelineKey,providerTimelineKeyPart, and the rig's exports. Changed: noschemeoption (thelegacyspelling is the only one);session.resetremoved;input.acceptedhas nojoin.item; prompt row ids include the generation; drop reasonsturn-replayed/item-replayed/request-replayedare gone (turn-settledreports an open of a turn the journal holds finished, or text for a stream its stopped turn ended);turn.endandrequest.withdrawnfor a turn or prompt the journal holds are accepted instead of dropped; an unnamedturn.endright after a Stop ends the stopped turn; a Stop leaves the turn's running tool calls to the agent's end;rig.restart()is async and replacesrig.assembler.dispose()drops text still in the batching window, so a lane appliessession.endedfirst.Why
Three review rounds of the earlier version each found new places where the assembler's remembered view and the journal disagreed (after a restart, after a cache dropped an entry, before the journal was attached, behind a replay). Each fix added a guard beside the second copy. The second copy existed only to survive losing memory while the agent kept running, which the process lifecycle already rules out: an assembler and its agent process start and end together, and the sweep closes a dead process's work before the next one's events land. Removing the situation removes the bug class, instead of guarding each case.
Alternatives considered:
Differences from the common pattern
Linked Issue
N/A (maintainer). Part of the structured native chat for other agents stack; builds on #25141, now in main.
Visual Proof
N/A: no UI or behaviour change. The module is not wired into anything.
Testing
What I verified
Current main refresh (
72873e3ca9edee25c84d6101a6c7bbc8e2515e7b): retargeted to main after Admit one provider event's journal writes as one queued operation, decided when it runs #25141 landed; all 5043 own added/removed patch lines are identical over the new base, with only 33 C1b files in the diff. All 27 explicit assembler/transition/shared-settlement suites (271 tests) and scoped format/lint pass; main's submission-position test and lockfile are preserved exactly. Fresh CI is pending; main's parser-dependency test failure is tracked separately.This head (
a59276fc709, which also merges Admit one provider event's journal writes as one queued operation, decided when it runs #25141's latest branch with currentmain) removes saved-history adoption (theinput.historyevent, its decision and the three "adopted session" tests), because the common pattern discards the history an agent replays on load. With it gone, a waiting user message is only ever Orca's send, so itsrequestedAtis now required. Nothing else changed. A fresh reviewer checked the removal: no behaviour outside adoption changed, and nothing insrc/still names the removed event. One gated node typecheck (on Translate ACP traffic into shared timeline events #25090's head, which contains this one) found only the 5 stale-installstream-jsonerrors described below. Main's settlement suites that share code with this PR (dead-generation settlement,journal-subagent-liveness, crash and stale-turn suites: 8 files, 84 tests) pass after that merge. CI on this head is green, including "static analysis and typecheck" and all five unit-test shards (two jobs that never got a runner were re-run). 14 assembler-area test files, 114 tests, all pass (every assembler test file, the id-spelling test, and Admit one provider event's journal writes as one queued operation, decided when it runs #25141's transition test).Main's suites that share code with this PR, 18 files, 213 tests, all pass: dead-generation settlement, crash during start, crash at turn end, stale-turn open and verdict, a send opening a stale turn, the four Stop-event suites, unexpected exit,
journal-subagent-liveness, journal store, write queue, event sink, and the three text-batching (coalescer) suites.Tests kept from the earlier version where the scenario can happen in production: a prompt id reused in a later turn opens a new prompt after a withdrawn or answered one; a refused event takes nothing; the budget refuses past 128 entries or 1 MiB (including a 1 MiB thread id); background-task rows; a send naming a turn that has not opened yet. Tests for the removed recovery and replay behaviour were deleted.
Tests for the rewrite: a prompt id the previous process used opens a new row after a restart; a resumed process's events accepted before the journal attaches are decided against the journal the sweep left; the id spelling.
Tests for a turn another writer settled (
provider-timeline-assembler-stop.test.ts): the stopped turn's pending prompt is cancelled at once, even when the next event writes nothing, while its running tool call stays running; the agent's later end fails that tool, and the result is the same when the agent's end lands before the Stop; a progress update and a "completed" the agent reports after the Stop land as reported, the same as when the completion lands before the Stop; an unnamed end right after the Stop ends the stopped turn (a second one is dropped); the next turn's open settles a stopped turn the agent never ended; a stopped turn's running tool calls hold budget until the agent's end; 140 turns each stopped and then ended by the agent never fill the budget; a stream in a stopped turn frees its budget slot; the rest of a stopped anonymous stream, of a named stream whose first write found the turn stopped, and of a stream accepted before the journal attached is dropped, not written as a separate message; the same stream starts a new message once the stopped turn ends or the next turn opens. An earlier turn's late end (provider-timeline-assembler-earlier-turn-end.test.ts) leaves the open turn's activity line, its reply as one message, and its running tool call alone. Also: a finished tool call keeps its final body against a later update, including the sweep's "failed"; and every piece of text in one batching window is written.Each guard fails a test when removed. For the rewrite I removed each of these in turn: the per-write turn check on streamed text, "turn row only where none is", "no running work in a finished turn", "finished tool never set running", the request-number probe, the generation in prompt ids, committing id counters only on acceptance, "no prompt in a finished turn", "withdraw cancels only a pending prompt". This round removed each new guard and re-ran: ending a turn another writer settled (3 tests fail), accepting an agent's end for a turn the journal holds (3), accepting a withdrawal for a prompt the journal holds (2), the budget freeing streams whose turn ended (1), the stopped-stream marker (3), leaving that marker when a write finds the turn ended (1), and "a finished tool keeps its final body" (2). The latest round removed each change and re-ran: a Stop settling tool calls again (3 tests fail), a Stop dropping the stopped turn's tool calls from the budget (1), the next turn's open ignoring a stopped turn (1), refusing a running tool's update in a stopped turn (1), clearing the activity line on any turn's end (1), ending the open turn's anonymous reply on any turn's end (1), and an unnamed end naming only the open turn (2).
oxlintandoxfmtpass on every changed file; nomax-linesexception.Typecheck and CI on the previous head (
ad8ebbd1f31): merges currentmain(d17351401d7) and Admit one provider event's journal writes as one queued operation, decided when it runs #25141's branch, which now contains the samemain. One local node typecheck on it found only 5Cannot find module 'stream-json/…'errors insrc/main/ai-vault/session-document-stream.tsandsrc/shared/json-token-reader*.ts, files this PR does not touch, from this machine's out-of-date dependency install. After the merge, the 14 test files above plus 11 of main's settlement suites (dead-generation settlement,journal-subagent-liveness, journal crash boundary, the four Stop-event suites, Claude Stop turn end, crash at turn end, crash during start, unexpected exit) pass: 25 files, 235 tests. On the previous head CI failed only "Verify localization coverage", on two stringsmainadded in fix: close worktree dialog before slow script checks #25472, so it never reached its typecheck. On this head CI's "static analysis and typecheck" job passes, including localization coverage, the type-aware lint andpnpm run typecheck.What I did not verify
Size: about 2,380 production lines in the assembler folder (about 1,890 without comments and blank lines), down from about 3,240 in the earlier version; the transition primitive itself is in #25141.
AI Disclosure
Review
Agent skill upstream boundary
docs/reference/agent-skill-sharing-upstream-boundary.mdand copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.Notes
legacyid format; prompt ids add the process generation, which only rows written by this not-yet-wired module use. Main's behaviour is unchanged, apart from sharing two existing functions (terminalAgentJournalBody,lostLiveWorkJournalBody) that main already applied.qoder-history-search-downgradetest fails on unrelated PRs too; it is not caused by this branch.Checklist
N/Awith reasonpnpm lint,pnpm typecheck,pnpm test, andpnpm buildpass (or CI will cover; local preferred). Locally:oxlint/oxfmton changed files, one node typecheck (stale-install errors only), and the test files listed above; CI covers the rest.Current-head CI result
Head
72873e3ca9edee25c84d6101a6c7bbc8e2515e7b. Current-head CI has finished: static analysis/typecheck, mobile, packaging and the other unit shards passed. The only test failures are main’s unchanged release-checkout parser-dependency assertion (#25731) and file-explorer cache-scan assertion (expected 2001 visits, got 4002); final verification failed from those two results. The cache failure reproduces locally and its sidebar/store code matches merged main byte for byte. No branch-owned failure remains; no local typecheck was run.