Repository navigation
feat(native-chat): one structured-chat journal database per host, owned by one process - #23613
Merged
Merged
Conversation
…ed by one process Every structured chat on a state directory now lives in one SQLite file, agent-session-journal.db, opened once by the process holding agent-session-journal.owner: an empty SQLite file whose held BEGIN EXCLUSIVE is a kernel byte-range lock, refused while another process holds it and released when the holder dies. - Stores own no connection: the per-chat handle, its close contract and the close-retry registry are gone; closing a conversation drains its writes, and the one connection closes last at teardown. - The owner lock is taken at runtime start, before orca-runtime.json is written; a process that does not own the chats is not published and refuses every structured request with journalUnavailable and words that say what to do. It retries the lock with backoff and runs the full install once it holds it. - A journal that will not open fails the host install: every chat says "Unable to load this chat." (journalCorrupt), and nothing is renamed, deleted or rebuilt. A newer build's database is refused and left byte-identical. - An append is one INSERT. The listing status is a column, written after the rows it describes and keyed by (epoch, sequence). - A per-chat journal from an earlier build is copied in verbatim (epoch UUID and every sequence) on that chat's first open, and its directory is retired only after the copy commits. - auto_vacuum = INCREMENTAL, with freed pages handed back in bounded steps after every delete.
…ogether Each chat's live epoch owns a block of row ids, block * 2^32 + seq, so a chat's rows share leaf pages with nobody else's, a replay is one range scan, and replacing or rewinding a chat deletes one contiguous range. Measured on the largest real chat (61 MB) beside 19 interleaved peers: 39 ms and 7.5 MB of WAL, against 214 ms and 102 MB for a (session_id, epoch, seq) key. - Ids are computed in Number arithmetic, never bitwise. A sequence is refused outside [1, 2^32) and a block at 2^21, which keeps every id below 2^53. - A replace, roll or import allocates a fresh block, moves the chat's pointer, and deletes the old block in the same transaction, so no orphan block exists. - The listing status write moves into its own writer beside the column.
… Orca wrote it after a downgrade A per-chat journal.db that reappears after its chat was copied in is the newer history: an older build, run after a downgrade, attached the chat and wrote it. - journal_imports records the (epoch, tip) each chat was copied from, in the copy's own transaction. A file already copied is never copied again, across any number of restarts after a failed rename; a file that differs always is. - Newest writer wins, per chat, with a row saying the chat was continued in an older version of Orca. When both builds wrote past the recorded tip under one epoch, the copy takes a fresh epoch, so readers reset instead of skipping rows. - Each copied directory retires to its own .imported-<epoch8>-<ms> name, so a second downgrade and re-upgrade never collides with the first.
Attach-flow and reconcile-attach fixtures stop passing a journal database those inputs do not take, and host and restore fixtures pass the one they now require instead of the removed journal root.
… chat journal A process refused the chat journal, because another Orca owns it or because its own journal will not open, failed startup restoration: the window booted in degraded no-save mode and a paired phone could not list any tabs. Startup restoration now treats the refusal structured requests are getting as having no structured host; terminals, tabs and saving go on, structured requests are still refused by the gate, and the install is retried on the next one. Any other install error fails startup as before.
…s are refused Session history (listing and preparing a resume) and a terminal typing a resume command only check whether a structured chat owns a provider session. In a process refused the chat journal they failed outright. They now take the refusal chats are getting as having no structured host, the same treatment startup restoration gets, through one shared helper; any other install failure still fails them. Chat requests keep the gate's refusal.
…ts schema A crash between creating a chat's journal.db and creating its tables left an empty or schema-less file. Each chat used to open that file as an empty chat; the importer instead refused the open as "try again" forever. A file with no journal_sessions table is now read as never written, the same as one with no rows. A file that is not a database, or whose read fails, is still refused.
… restore Opening a chat's journal is synchronous SQLite now that no per-chat directory is created first, so the restore of every visible chat ran as one main-thread task. Each chat now waits for a macrotask before it opens.
The one-time copy of an earlier build's per-chat journal ran as one transaction, which blocked the main thread for 650 ms on the largest chat. Rows now copy 512 at a time, each batch its own transaction, yielding to the event loop between batches. The rows go into a block journal_import_blocks reserves, which no reader follows and no other chat is allocated; the last batch publishes the chat's pointer, repair marker and import marker together and releases the reservation. A copy that stops midway leaves only that block, which the next open clears and copies again. Two opens of one chat import one after the other.
A lock file that is not a database, or cannot be opened, made the claim throw before any refusal was recorded, so startup restoration failed on every launch. The claim now sits in the same try as the database open and records the same typed refusal.
…ning pass A reclaim pass ended as soon as the freelist stopped shrinking between steps, so a second delete that freed more than one step's worth mid-pass ended it early and left those pages on the freelist. A pass now ends only when a step itself frees nothing, or the freelist is empty.
…o update A chat opened read-only because a newer Orca wrote rows to it answered a send with the generic write failure. It now refuses the way a database a newer Orca wrote does, with the same reason and words.
…hats A process whose chats another Orca owns, or whose chat journal will not open, has no structured host. Its session-tabs inventory still answered, with no chat rows, and the renderer read that as "every chat was closed": it removed the restored chat tabs and the next session save persisted their placement away. The inventory now says `agentSessionsUnverifiable` when the last tab restore ran with chats on disk but no host to list them. The flag is set and cleared at the per-client projection point beside the client-hosted page hold, and the restore is memoised only once a host answered, so a later lock takeover or journal open republishes the chats and clears it. The renderer keeps agent-session tabs, and keeps cancellation tombstones, against an inventory that does not affirm its chat set.
…ss's way past it A process refused because another Orca owns the profile's chats sent the generic `journalUnavailable` reason, so current desktop and phone surfaces said "couldn't open this chat's history right now. Try again." — a step that never helps while the other Orca runs. The refusal now names its own reason, `journalOwnedElsewhere`, with the refused process's kind (dev desktop, packaged, orcad) as a fact. Each kind gets its own step: quit the other Orca, or give this one its own profile (ORCA_DEV_USER_DATA_PATH) or data folder (ORCA_USER_DATA). The sentences are added to the shared notice copy, the desktop catalogs in all six locales, and the boot catalog. A client that predates the reason reads it as none and keeps the code's words; an unknown kind reads as the packaged app's step. The `message` released clients print is unchanged. Which requests refuse does not change.
…o ask A refused startup kept its hostless result, so after the owner quit this process never installed a host, never reconciled restart leases, and kept telling clients it could not list its chats until a desktop chat request. Taking the lock now reruns startup restoration once and pushes the chats.
The lock frees as its holder exits, but a refused process only learns that on its next retry, and the 30 s cap left a second Orca refusing chats for up to half a minute after the owner quit. One retry is an open and BEGIN EXCLUSIVE on an empty file.
…ublish chats A list that landed on the refused startup after the lock was taken finished after the takeover had already checked, so nothing republished. The takeover now installs first, waits for any restore in flight, and restores only then; the restore that clears "cannot tell" pushes the frames itself, so a list that heals the inventory first reaches subscribers too.
Quitting cancelled a refused claim's retry only at its end, so a retry firing during the stop's awaits took the lock and installed a host the stop never tore down, and the lock was then released under an open journal. The stop now cancels the retry first, keeping the refusal, and repeats its teardown while an install that began during it (a takeover already under way) is pending, so no journal connection outlives the lock.
A refusal the host throws reaches the client as an RPC error whose message is the bare code; its reason and facts ride only in the error's data, which no client read. The chat pane's status line therefore printed agent_session_journal_unreadable, a send took the bare "not sent" path, and other writes said the outcome was unconfirmed. One shared reader, agentSessionThrownRefusal, now reads the refusal from the error data. A failed history read shows the refusal's read-history words, a send keeps the refusal behind its Retry exactly as a returned refusal does, and the other writes (desktop and phone) name the refusal instead of doubting the outcome. The phone's read failure goes through the same reader.
Every chat request retries a journal open that failed, which is intended, but each retry also logged the failure with its full stack: a junk database file logged the same "file is not a database" error 189 times in a minute. The open now logs a failure only when its code and message differ from the last one logged, and forgets it once an open succeeds. The retry is unchanged. The open moves to its own module beside the runtime, which had no room left.
…ies it on first use Startup restore opened every restored chat, and that open copied the chat's per-chat file into the host database, so the first boot after an upgrade paid the whole one-time copy before the chat list appeared. A restore open now reads a chat that is still in its per-chat file straight from that file, read-only, with the importer's own reader, and closes the file before moving on. That read drives the listing, the status row and the restart offer, as it did when every chat had its own file. The copy becomes owed work on the chat's write queue: it runs before the chat's first write, and a reader that reaches the chat awaits it. A chat the host already holds, or that was copied before, still opens through the import and its reimport rules, and so does a file whose read needs a repair written.
…anned Each teardown pass clears the registered host before it awaits an install in flight, and that install registers its host when it finishes. The pass then tore the host down but left it registered, so a request after the stop was served by a host whose journal was closed. The stop now clears the slot once its passes are done.
synchronous = FULL fsyncs each commit, but macOS fsync leaves the drive cache unflushed, so FULL alone does not survive a power loss there. With checkpoint_fullfsync, each checkpoint uses F_FULLFSYNC; elsewhere it is a no-op. The comment that said FULL alone was enough is corrected.
An imported chat's per-chat file was kept under an `.imported-*` name, which
doubled the disk its history takes. The copy now reads back from the host
database before it is published: its items, submissions, epoch and tip must
match the file's. Only then does one transaction publish the chat with its
import marker, and the file and its WAL files are deleted, the directory too
when nothing else is in it (a pre-SQLite transcript there is kept).
A copy that does not match is never published: the file stays, the chat is
refused as unreadable ("Unable to load this chat."), and the mismatch is logged
once. A file left behind by a failed delete or a crash matches the marker, so
the next open deletes it rather than copying it again; a file an older build
wrote after a downgrade still differs, and is still copied again.
The check that a copied chat reads back as its per-chat file folded both whole, each in one synchronous task: over half a second on the largest chat. Both reads now go a batch at a time between turns of the event loop, like the copy itself, and count rows as well, so a copy that lost a row with no item in it is caught too.
![pullfrog[bot]](https://avatars.githubusercontent.com/in/1768019?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
ℹ️ No new code issues in this delta — the two commits are green on
tc:node,oxlint, and their focused tests (87 passing). One informational note on the description; no inline findings.
Reviewed changes
Re-reviewed the two commits pushed after 94182a8, which open a newer-Orca journal read-only and remove the saved listing-status machinery.
- A newer-build journal opens read-only instead of being refused.
openJournalDatabasenow returnsOpenJournalDatabase { db, readOnly }and, whenuser_versionis above this build's, returns areadonly: trueconnection;JournalHostDatabaseexposes that asreadOnly, andrestoreJournalStorediverts it to a newrestoreFromNewerDatabasethat folds each chat from the database (or its not-yet-copied per-chat file), adopts it read-only, and refuses every write asjournal_read_only, so clients print the "Chats were saved by a newer Orca" message. - Saved listing status removed. The
status_json/status_seqcolumns,journal-session-status.ts, the sharedstructured-agent-session-saved-status.ts,JournalListingStatusWriter/saveListingStatus, and the status feed's call to it are all deleted;journal_sessionsis back to(session_id, workspace_id, epoch)and the schema stays at version 3. - Underlying cause plumbed through the refusal.
AgentSessionJournalErrornow acceptsErrorOptions, and the read-only restore attaches the read failure ascauserather than flattening it.
ℹ️ The description still records the refused-newer-schema behavior this delta reverses
The body's "Differences from the common pattern" and "For a decision before merge" both state that a database stamped by a newer build is refused and left byte-identical; the new commits instead open it read-only and show its chats. Together with the still-stale owner-lock, block-key, listing-status and reclaim paragraphs, the description now documents several behaviors the code no longer has, and it is the only record of these decisions.
Technical details
# The PR description now contradicts the newer-schema and listing-status changes
## Affected sites
- PR body "Differences from the common pattern": "A database stamped by a newer build ... is refused and the file left byte-identical" — now opens read-only and serves its chats (`journal-store-restore.ts:82`).
- PR body "For a decision before merge": the rollback-rebuild item and "block-keyed rows" describe superseded layouts.
- PR body "Scope"/"What Changed": still list block-keyed rows, the listing-status column and bounded free-page reclaim, removed by `94182a8`/`f0c46576`.
## Required outcome
- Rewrite the newer-schema decision to say the database opens read-only with its history and refuses writes, and drop the block-key/listing-status/owner-lock text.DeepSeek Flash (free via Pullfrog for OSS) ｜ 𝕏
Brings in #23059 (turn facts from the turn record, /compact as a sent message) and #23026 (Stop from the moment a message is sent), among 23 commits. Conflicts keep main's behaviour on the one shared journal database: per-chat handle and close tests stay deleted, main's new host tests open the shared database, and appended rows name their turn scope.
Contributor
There was a problem hiding this comment.
✅ No new issues found — the merge is clean and verified.
Reviewed changes
Re-reviewed the merge commit pushed after f0c46576, which merges origin/main (d4530a8a) into the branch and reconciles main's turn-scope attribution with the shared journal.
- Merged
origin/mainand adopted turn-scope attribution. Every journal write now states aturnScope; the branch's restore writes repair and file-format notices asAGENT_JOURNAL_THREAD_SCOPE, and epoch replacement derives one from position when the source states none. - Migrated main's host tests to the shared database dependency. The four host tests main added after the merge base now construct the host with
journalDatabase: openTestJournalHostDatabase(root)instead ofjournalRoot, clearing the merge-reftypecheckfailure raised in the previous review. - Pulled in main's cancel and command-turn paths.
performCancelnow lives instructured-agent-session-turns-cancel.tsand is re-exported fromstructured-agent-session-turns.ts;handOverSubmissionroutes a command body tohandOverStructuredAgentSessionCommand. - Verified green.
pnpm tc:nodeandpnpm tc:webexit 0; 33 focused vitest files / 319 tests pass over the journal and the migrated host tests; no conflict markers remain.
DeepSeek Flash (free via Pullfrog for OSS) ｜ 𝕏
…ot to try again
When a newer Orca wrote the chat journal, this build opens it read-only. A send or a Stop was
refused with the reason `journalUnavailable`, so today's desktop and phone clients chose the
words for an open that can clear: "Orca couldn't open this chat's history right now. Try again."
Retrying never cleared it; only updating Orca does.
The refusal now names its own reason, `journalWrittenByNewerOrca`, whose words are "Chats were
saved by a newer Orca. Update Orca to keep using them." A read refused the same way names it
too. An older client does not know the reason, drops it, and falls back to the code's words
("Orca couldn't read this chat's saved history."), and released clients still print the message.
…usable, not as retryable A chat journal database stamped with schema 1 or 2 was written only by unreleased development builds of this change. Opening it threw a plain error, which every chat reported as "Orca couldn't open this chat's history right now. Try again." Retrying never cleared it. It now throws a named error that is classified as unusable, so every chat says "Unable to load this chat." The one log line names the file, says an unreleased development build wrote it, and says to move it aside. Nothing migrates or renames it.
…e comments The chat-only owner lock is gone, so only a chat journal that will not open leaves a runtime unable to list its chats.
…eft behind A per-chat file left without its WAL is set aside, not copied again; nothing runs an incremental vacuum yet, so the auto_vacuum mode is kept for a later pass; and the idle sweep drops a chat's in-memory fold, since a chat holds no journal connection.
Each is used only inside its own module now; the teardown's export served a deleted test.
…eck every journal table The test named 'migrates an older user_version forward' covers only a version-0 file that already has its tables; versions 1 and 2 are refused. The table test now also checks journal_imports and journal_set_aside.
…used a window The running dev instance may be a background launch or a server, which show no window. The line now says only that this launch passed its request to that instance.
… connection it opened The install opened the chat journal database and closed it only if the record store then failed to open. A later failure, such as the model catalog wiring or the host constructor, left the connection open, and the next install opened a second one in the same process. Every failure after the open now closes it.
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
+58
to
61
| await ensureStructuredAgentSessionHostUnlessRefused(ensureHost) | ||
| const host = getStructuredAgentSessionHost() | ||
| if (!host) { | ||
| return |
There was a problem hiding this comment.
When this process cannot open the chat journal, the new refusal bypass lets a terminal run a target-less or matching claude --resume command without checking who owns the structured session. If another Orca already has that provider conversation active, the terminal can start a second provider writer against it. The resume needs an ownership verdict before it can be treated as safe.
Knowledge Base Used:
Contributor
There was a problem hiding this comment.
ℹ️ No new issues in this delta — eight commits of targeted fixes, comment/export cleanup, and tests; no actionable findings.
Reviewed changes
Re-reviewed the eight commits pushed after a6a514fa, which split the newer-Orca journal into its own refusal reason, treat an unreleased-schema journal as unusable, close a leaked journal connection on a failed install, and tidy comments and exports.
- A journal a newer Orca wrote gets its own refusal reason.
journalWrittenByNewerOrcareplaced the retryablejournalUnavailablefor a database stamped by a newer Orca or a chat holding newer rows, so a send or Stop now says to update Orca instead of to try again; the new copy lands in all six catalogs and both client surfaces. - A journal only an unreleased build wrote reads as unusable.
openJournalDatabasethrowsJournalUnreleasedSchemaErrorfor versions 1–2 andclassifyJournalOpenFailuremaps it tojournalCorrupt, so those chats say "Unable to load this chat." while the once-logged line names the file to move aside. - A failed install closes the journal connection it opened.
install()now wraps everything past the journal open, so a throw from the record store, environment resolvers or model-catalog wiring closes the one connection before rethrowing. - Comments and exports corrected. Wording left over from the removed owner lock and per-chat handle is gone; eight names nothing imports are no longer exported; the dev single-instance exit line no longer claims it focused a window.
DeepSeek Flash (free via Pullfrog for OSS) ｜ 𝕏
Brings in the host-owned queue for mid-turn messages (#23726) and the /clear and Codex rewind unlock (#23524), expressed on the one host journal database: - Queue transactions run through runJournalTransaction on the shared connection, and every appended row keeps the draft hooks inside that same transaction. - The open-time draft repair writes nothing for a chat with no draft, so a chat whose first-use copy is owed stays uncopied until its first real write. - A newer Orca's journal takes no new draft, and Send-now, Delete and Resume on it refuse with the same update words a send gets. - Main's queue tests open the shared database; a table-healing case restarts the host around its edit, since the table is ensured when the database opens.
Contributor
There was a problem hiding this comment.
Important
Merging main's host-owned queued-message feature (#23726) into this branch leaves the migration from a per-chat journal.db lossy: that feature stores a chat's queued drafts and queue pauses in the same per-chat file the lazy importer copies and then deletes, and the copy takes only the journal rows. If any shipped or beta build carries #23726, a user's unsent queued messages are silently dropped on the first open after this upgrade.
Reviewed changes
Re-reviewed the merge of origin/main (29c49aec31) into the branch, which pulls main's host-owned queued-message feature and reconciles it with the one-database-per-host journal.
- Queued-message writes routed through the shared connection.
journal-queued-messages.tswrites throughJournalHostDatabase.transaction, andjournal-row-writer.tsruns main's draft-consume hook plus the SAVEPOINT-wrapped standing bookkeeping inside that same transaction, with the host database injected into the store instead of an owned per-chat connection (journal-store.ts). - Draft tables created at every writable open.
openJournalDatabasecallsensureQueuedMessagesTableaftercreateJournalSchema, deliberately without auser_versionbump so a downgraded build stays writable (journal-database.ts:55). - Newer-Orca refusal extended to the queue.
refusingNewerOrcaJournalwraps Send-now, Delete and Resume,maybeQueueStructuredAgentSessionSendshort-circuits a read-only journal to the immediate path, and a newstructured-agent-session-queued-newer-orca.test.tscovers the read-only queue. - Base merges. The other
origin/maincommits (unfinished/clearand Codex rewind recovery, browser viewport/crash fixes) are byte-identical to main and outside this PR's own diff.
⚠️ Queued drafts and queue pauses are dropped when a legacy per-chat journal is imported
A build containing #23726 stored a chat's queued drafts in queued_messages and its queue pause in queued_message_pauses, inside that chat's per-chat journal.db. The lazy importer copies only journal_rows and journal_repairs from that file, verifies, then deletes it — so the drafts and pause are discarded without an error, a notice, or a reset. Because the draft tables add no user_version bump, the file still passes the schema gate (LEGACY_JOURNAL_SCHEMA_VERSION = 2) and is retired.
Technical details
# Legacy queued drafts are not copied before the per-chat file is deleted
## Affected sites
- `src/main/native-chat/agent-session-journal/journal-per-session-import.ts:187-228` — `copyLegacyJournal` inserts only `journal_rows` (`INSERT_ROW`, `:58`) and `journal_repairs` (`UPSERT_REPAIR`, `:60`); no `queued_messages` / `queued_message_pauses` read or insert exists in the importer.
- `src/main/native-chat/agent-session-journal/journal-per-session-source.ts:18-25` — the legacy reader's SQL selects only `journal_sessions`, `journal_rows` and `journal_repairs`.
- `src/main/native-chat/agent-session-journal/journal-per-session-source.ts:14,35-37` — a schema-2 file (the shape #23726 left: `queued-message-schema.ts:15-22` documents no version bump) passes the gate.
- `src/main/native-chat/agent-session-journal/journal-per-session-import.ts:141` — `retireLegacyJournal(...)` removes `journal.db`, `-wal`, `-shm` and the directory after a successful copy, so the only record of the drafts is gone.
- No other mechanism recovers them: `journal-queued-messages.ts:91-107` reads drafts only from the shared database, and a waiting draft has no `journal_rows` row to re-derive from.
- The renderer drops its own outbox entry once the host answers `queued` (`src/shared/structured-agent-session-send-disposition.ts`), so the host draft table was the sole record.
## Required outcome
- If a build containing #23726 can have shipped, migrate a legacy file's `queued_messages` and `queued_message_pauses` into the shared database as part of the same publish, or refuse/set the file aside instead of deleting it. Add a test that seeds a legacy per-chat file with a queued draft and asserts it survives the import; no such test exists today.
## Open questions for the human
- Has a build containing #23726 (per-chat draft store) reached users? That decides whether this is field data loss or a nightly-only gap, and whether the migration is required before merge.DeepSeek Flash (free via Pullfrog for OSS) ｜ 𝕏
| caller, | ||
| envelope, | ||
| plan, | ||
| { ...plan, run: (ctx) => refusingNewerOrcaJournal(plan.run(ctx)) }, |
Contributor
There was a problem hiding this comment.
refusingNewerOrcaJournal covers Send-now, Delete and Resume, but Stop withdraws queued submissions through mutateStructuredAgentSession (structured-agent-session-host-mutations.ts:186), not mutateQueued. On a read-only (newer-Orca) journal that write throws AgentSessionJournalError('journal_read_only'), which is neither an AgentSessionRefusalError nor in the RPC allowlists (src/main/runtime/rpc/errors.ts), so it surfaces as runtime_error/unconfirmed instead of the "update Orca" words. This is reachable when the read-only journal still holds a queued submission and a client or paired surface offers Stop.
Technical details
# Stop's queue withdrawal is not classified for a read-only journal
## Affected sites
- `src/main/native-chat/agent-session-wire/structured-agent-session-host-mutations.ts:183-222` — Stop's plan runs through `mutateStructuredAgentSession` and calls `ctx.journal.rejectQueuedSubmissions(...)` at `:186`.
- `src/main/native-chat/agent-session-journal/journal-pending-submission-recovery.ts:67-86` — `rejectJournalQueuedSubmissions` writes via `resolveDispatch` for each queued submission, with no read-only gate.
- `src/main/native-chat/agent-session-journal/journal-row-writer.ts:40` / `journal-write-guards.ts:23-30` — the write throws `AgentSessionJournalError('journal_read_only')`.
- `src/main/runtime/rpc/errors.ts:166-256` — `journal_read_only` is not an `AgentSessionRefusalError` and is absent from the passthrough sets, so it maps to `runtime_error`.
## Required outcome
- When a write meets a read-only journal on the Stop path, return the same `journalWrittenByNewerOrca` refusal the send path returns (`structured-agent-session-turns.ts:150-156`), or guard the call site with `ctx.journal.isReadOnly`. Worth a test mirroring `structured-agent-session-queued-newer-orca.test.ts` that also exercises `stop()`.
Contributor
Author
Status: merged (head 8c0bd34)The problemEvery structured chat kept its history in its own small SQLite file with its own connection. That per-chat handle lifecycle caused a run of bugs: handles leaking and blocking renames on Windows (#18652), quit closing files while cleanup still wrote (#18974), and What changes for you
What the review changed
Deferred (P3, in the review ledger)
Known costs
Verified
Not verified
|
brennanb2025
added a commit
that referenced
this pull request
Sep 30, 2026
Main (#23613) moved every chat's journal into one database per host and removed the per-chat test opener; the restart test opens its journals through the host-database test support instead.
Merged
9 tasks done
8 of 9 tasks
This was referenced Sep 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
Every structured chat kept its history in its own small SQLite file,
agent-session-journal/<workspace>/<session>/journal.db, with its own connection that opened, closed and retried closing on its own schedule. That per-chat handle lifecycle produced a run of bugs: handles leaked and blocked renames on Windows (#18652), quit closed files while cleanup still wrote to them (#18974), and removing a chat directory failed withENOTEMPTYwhile handles were open (#18824).Nothing also stopped two Orca processes on one profile from writing the same chat. Dev builds skip the single-instance lock, so two
pnpm devruns on the default profile both opened every chat. Opening a chat settles its handed-over sends and running turn without a lease (structured-agent-session-conversation-open.ts), and each process numbers a chat's rows from its own memory (journal-row-writer.ts). Measured on main, the second process marked the first one's live sendunknown, and the first process's next row failed withUNIQUE constraint failed.What changes for users
agent-session-journal.db, opened once by the running Orca. Nothing a user does changes; the per-chat handle bugs above can no longer happen.ORCA_DEV_USER_DATA_PATH, and exits. Running two dev builds side by side needsORCA_DEV_USER_DATA_PATHset for the second one.Scope
agent-session-journal.db, one connection, owned by the process that holds the existing single-instance lock.shouldSkipSingleInstanceLock). Serve mode, the macOS diagnostic bypass and e2e launches behave as before.(session_id, epoch, seq)in a rowid table, the same declaration main's per-chat files use.checkpoint_fullfsync = ONbesidesynchronous = FULL, so checkpoints reach the disk on macOS.journal-close-retry.ts,journal-store-close.ts, the per-store open/close contract, recovery close-on-throw, teardown's close fan-out).How it works
agent-session-journal.dbonce at install. A chat's store holds no connection; closing a conversation drains its admitted writes, and the one connection closes last at teardown, kept until the close actually succeeds. Every transaction goes through one helper that includes COMMIT: a failed commit is rolled back with its original error kept, and if the rollback fails too, chats answer a typed try-again refusal instead of the connection being stuck.(session_id, epoch, seq). Replay scans a chat's epoch in order; a replace, rollover or rewind deletes that epoch's range in the same transaction that publishes the new one.Why this design
The shared file removes the per-chat handle lifecycle and every bug that lived in it, instead of adding more guards around it: there is one connection, opened once and closed once. One process per profile, enforced at launch by the lock Orca already uses for packaged builds, removes the two-writer corruption at its source for every store, not just chats. Copying old chats lazily, on first use, keeps the first launch after upgrading no slower than main; the check against the original before deleting it means an upgrade can never lose history silently.
Alternatives considered and dropped during review: a chat-only lock with a refused second process that could take the chats over while running (it guarded only dev builds, and the in-process takeover produced races that three review rounds kept finding); rows keyed in contiguous blocks per chat (a measured rewind win, but no observed failure, and neither reference does it); an after-every-delete free-page pass (neither reference does it; reclaiming belongs with the retention work); saving each chat's list status in this PR (nothing reads it yet).
Differences from the common pattern
Each difference is labelled intended (safe to keep, with why) or temporary (what removes it). One process per profile and one database per data directory are not listed: both match.
Known costs and limits
Measurement gate
G1: census
ENOTEMPTYremoving a chat directory with handles openG1: scenarios (fault injection, same host and store API on both builds)
Measured on the first reviewed head of this PR; the storage paths these exercise (commit, crash, open, corruption) are unchanged by the later redesign except two processes, which the process lock now stops before either opens the journal.
G2: performance, macOS
Real host, record store and journal on a read-only copy of a real profile (1,090 per-chat journals, 1.83 GB); only the provider process is faked. This PR at
94182a8a21b(the current row layout) against maina68d62911e, median of 3 (rewind: median of 5). The disk was 98-99% full during these runs, so single runs varied widely; the ranges are in the review notes.G2: performance, Windows
Measured on an earlier head of this PR (
67f5e35adec), which already copied chats lazily but still used the block row layout. A re-measurement on the current layout is running and will replace this table. Windows low-spec machine, the fixed 247-chat seed, the real app, against maina68d62911eon the same machine, median of 2.Linked Issue
N/A (maintainer PR).
Visual Proof
Live Electron QA of dev builds on a separate Mac with real Claude chats, from fresh profiles seeded by main, at
5ca53728160.First boot after upgrading: a chat main wrote opens with its history intact, and a new message is answered:
A damaged journal file: the app starts normally, tabs are kept, and each chat says "Unable to load this chat.":
A journal written by a newer Orca: history stays visible, and a send says to update Orca:
Testing
Live Electron QA ran on a separate Mac with real Claude chats, main against this PR, from fresh profiles seeded by main: first boot after upgrading (all chats listed; a chat's per-chat file untouched until it is used, then copied exactly and deleted; a closed chat copied only when reopened; a send answered); a second dev launch on the same profile (exits with code 3 and its one line, the first window stays up, the journal files unchanged); a damaged journal (the app starts with no error toast, tabs kept, "Unable to load this chat.", the terminal works, the file unchanged, one log line); a newer-schema journal (history visible, sends say to update Orca, the file unchanged); quit mid-reply and relaunch; an idle-stopped chat, then a send; and boot to first paint, where six paired runs showed no measurable difference from main. All passed.
Earlier heads were also tested with real Codex chats merged with #23552: import from main, a 16-turn rewind (correct after relaunch), quit mid-turn and idle stop.
New tests, each ablated (the core line removed, and the test turns red for that reason):
checkpoint_fullfsyncread back on;(session_id, epoch);Existing reducer, identity, cursor and row-schema suites keep their assertions.
tc:nodeandtc:webpass.AI Disclosure
Built with Claude.
Review
Author: @BrennanKB5
Agent skill upstream boundary
docs/reference/agent-skill-sharing-upstream-boundary.mdand copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.Notes
SSH and remote hosts: orcad owns its own data root, takes its own existing startup lock there, and holds its own journal file; nothing crosses hosts. Folder workspaces: the journal is keyed by session and workspace id, never by a path, and QA covered chats in a folder workspace. Mixed versions: one new refusal reason (
journalWrittenByNewerOrca) and one optional field on the chat tab list (agentSessionsUnverifiable). An older client ignores the field and shows its generic "couldn't read this chat's saved history" words for the new reason; it never rejects the reply.Checklist
N/Awith reasonpnpm lint,pnpm typecheck,pnpm test, andpnpm buildpass (or CI will cover; local preferred)