Implementation: record, replay and fork, end to end - #2
Merged
yi-here merged 67 commits intoAug 29, 2026
Merged
Conversation
The CLI is most of this product's surface and most of its output ends up somewhere that is not a terminal, so the rules are encoded as tests: colour only on a real TTY, NO_COLOR honoured, no progress animation under --ci, one greppable `level event key=value` line per fact, and tables with aligned columns rather than box drawing so output survives a pipe into awk. Two things worth calling out. Errors carry what happened, what it means and the next command to run — an error missing the third is a bug report we receive instead. And output goes through the same secret filter as the write path: terminals scroll into screenshots, so a key printed once is a key leaked forever. Argument parsing is hand-rolled because a debugger you install to diagnose a broken environment should not drag in a dependency tree of its own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
Implements spec §4. Harnesses are not deterministic — generated ids, context compaction firing at a different point — so a replayed request frequently is not byte-identical to the recorded one, and matching degrades through four rungs: canonical hash, then same-position-small-difference, then same-trailing-message-different-prefix, then no match. Normalization decides what "the same request" means, and the choices are the substance here: tool order is sorted because declaration order does not change an answer, per-invocation metadata is dropped, tool_use ids are excluded from identity while names and inputs are kept, and sampling parameters stay because they genuinely change the result. The invariant the tests pin down is that anything below rung 1 must carry a divergence. There is an explicit test whose failure message says so: a debugger that quietly guesses is worse than no debugger, because you will believe it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
Exact, fork and compare are not three subsystems. They are this one server with a cursor: the position in the recorded stream where it stops serving from disk and starts serving from the network. Three properties the tests pin down. Replay makes no network call at all — the test injects a fetch that flips a flag and asserts the flag stays false, because blocking egress at the socket is different from discouraging it. An unmatched request returns 409 naming the mismatch and the --loose flag rather than inventing a reply, since a fabricated response makes every downstream conclusion worthless. And auth headers are stripped before anything is recorded or forwarded. Wire formats live behind a Dialect interface rather than being hard-coded, so a third party can support a new model API without touching the interception core. Test resolution now aliases workspace packages to source, so `npx vitest` needs no prior build — a mandatory build step is how a five-minute dev loop quietly becomes twenty. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
The fake agent behaves like the real targets in the only respects OrcaReplay depends on: it reads ANTHROPIC_BASE_URL, drives a multi-turn tool loop, and resends the whole conversation each turn. That last part is the mechanism the whole project rests on — it is what lets the proxy observe tool results without hooking the harness — so the fixture exercises it directly. The fake model is deterministic on purpose. An exact-replay test cannot distinguish "replay worked" from "the model happened to agree" unless the recording is reproducible to begin with. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
Cashes in the observation the project rests on. A tool result never appears in the response that requested it — the harness runs the tool and hands the result back on the next request — so each tool_use is held open and closed when a later request carries the matching tool_result. That is how a proxy seeing only model traffic reconstructs the whole tool loop without touching the agent. Writing the test first caught a bug that would have shipped: because the conversation is resent in full every turn, a result already turned into an event was being re-emitted on every subsequent request, accumulating one duplicate per turn per tool call over a long run. Fixed with a closed-id set, which also let the mid-conversation case stay correct — a result for a call we never saw is recorded rather than dropped, since recording can begin partway through a session. Tool calls whose result never arrives are reported by unresolved() rather than silently disappearing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
The working system: record an agent, replay it offline, fork it onto another model. 595 tests, built test-first throughout. Packages - core: append-only trace writer, reader, content-addressed blobs, checkpoint derivation, and a redactor every write path goes through - fs-capture: shadow git index snapshotting the workspace per turn - providers: Anthropic and OpenAI wire translation both ways, plus pricing - proxy: record, replay and hybrid cursors over the matching ladder - adapters: claude-code, codex, opencode and a generic escape hatch - mcp-shim: byte-faithful JSON-RPC tee over stdio - viewer: one self-contained HTML file, no network, no dependencies - cli: the commands that wire it together Bugs the tests caught before they shipped - A hostile `.gitignore` with `!.env` re-included secrets, because git's info/exclude is lower precedence than the workspace's own ignore file. Now enforced with pathspecs no ignore file can override — and `GIT_LITERAL_PATHSPECS` in the environment silently disarmed those too, so the git env is scrubbed. - Shannon entropy alone flags ordinary identifiers: `getUserAuthenticationTokenFromRequestHeaders` scores 4.08 bits/char. Since traces are mostly source code, the rule as specified corrupted the payloads the trace exists to preserve. A mixed-alphabet guard fixes it with essentially no loss of recall; the spec now says so. - OpenAI's `prompt_tokens` includes cached tokens and Anthropic's `input_tokens` does not, so every cached OpenAI turn was double-billed. - Recorded bodies were being parsed as if already canonical, which made every replayed request look like a major divergence instead of an exact match. - Fork restored from a fresh shadow store rather than the source run's, so the tree object was missing — the silent-wrong-state failure checkpoints exist to prevent. - Embedded git repositories were stored as gitlinks whose contents never entered the shadow store, so a fork would have debugged a workspace quietly missing a subtree. Detected and refused rather than half-restored. Verified against the real Claude Code binary: it authenticates with its own `authorization: Bearer` header and ignores an injected key, so the proxy now forwards auth upstream while still never writing it to a trace. Dropping that header would have broken subscription users only — the worst kind of bug to ship. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
Closes three gaps between what the docs promised and what the code did.
`orca scrub` now exists. SECURITY.md and the bug-report template both told
people to run it before sharing a trace, which made its absence a real
problem rather than a missing nice-to-have. It rewrites events.jsonl and every
text blob, re-runs the standard detectors over what the write path missed, and
refreshes the integrity digest — a scrubbed trace that failed its own
integrity check would be unusable, and people would stop scrubbing.
Binary blobs are left byte-identical. The first version of that guard was
wrong in the most embarrassing direction: `text.includes(' ')` skipped
anything containing a space, which is most text. The replacement rejects a
file whose UTF-8 round trip loses bytes, and a mutation check confirms the
test actually fails without it.
MCP capture is wired into record behind `--mcp-config <path>`. Opt-in rather
than discovered, because the config lives somewhere different for every
harness and guessing wrong means rewriting a file the user did not ask us to
touch. HTTP and SSE servers are reported as skipped, never silently dropped.
The README status table now says shell capture via PATH shim is not
implemented, rather than "partial". It also records that a Codex CLI on a
ChatGPT subscription talks to its own backend, so base-URL redirection does
not capture that auth mode. Overstating coverage is how a debugger loses the
trust it needs.
607 tests.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
Three real failures, found by running CI's own commands from a clean install rather than trusting a warm node_modules. `npm ci` could not install at all. The CLI package was renamed to `orcareplay` after the last lockfile regeneration, so package.json and package-lock.json were out of sync — every CI job would have died before running a single test. Trace writes were fired in parallel from the proxy's callbacks. Each one snapshots the workspace, so two overlapping `git add` calls collided on index.lock and failed intermittently. Serializing also keeps `seq` in the order exchanges actually happened, which the format depends on. A failing task no longer stalls the queue: one bad write costs one event, not the rest of the run. Snapshotting a live workspace races the agent still writing to it, which git reports as "short read while indexing". Transient collisions are now retried with a short backoff, and a snapshot that still fails is recorded as a note rather than thrown — losing a snapshot costs one checkpoint, whereas aborting costs the user the run they were trying to record, which is the thing that is hard to reproduce. Permanent failures (no git, not a repository) are not retried, since waiting on them only delays the agent. The e2e suite now passes 10 runs out of 10; before these fixes it failed roughly one in three. 615 tests, verified against `npm ci` from an empty node_modules. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
list, show, checkpoints and export were the only user-facing commands with no direct test — they were exercised incidentally by the end-to-end suite but nothing pinned their behaviour. The assertions worth having are the ones about what the commands promise rather than what they print: `export` states what is about to leave the machine before it writes anything, the exported file carries no external reference at all (the single-file constraint is the growth mechanic, so it is asserted rather than assumed), `checkpoints` explains --no-fs when a run has none, `list` tells you how to record one when there are none, and a run selector that is not shaped like a run id is rejected rather than resolved into a filesystem walk. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
Contributor
Author
CI is red for a reason outside this PR — needs a repo setting, not a code changeAll four checks ( Evidence that it is not this diff:
The likely cause is GitHub Actions not being enabled or entitled for this repository yet (new repo, or org runner/billing policy). I cannot change that from here — it needs someone with repo or org settings access to check Settings → Actions → General, and that Actions has runner minutes available. What I verified insteadThe exact command sequence CI runs, executed locally from a clean $ rm -rf node_modules && npm ci # 67 packages
$ npm run fmt:check # All matched files use Prettier code style!
$ npx tsc --build # clean
$ npx vitest run # 627 passed (627)
$ node scripts/conformance.mjs # 27 events checked, 0 failure(s)
$ node scripts/check-neutrality.mjs # 0 failure(s)One real CI bug this did surface, now fixedRunning The same clean-install pass also surfaced two genuine races, both fixed in that commit: trace writes were firing in parallel and colliding on git's I am not re-running the checks again — three runs across three commits is enough to establish the pattern, and a fourth will fail the same way until the repository setting changes. Generated by Claude Code |
5 tasks
Running the command for real, rather than trusting the unit tests, exposed that only the first model ever worked. `compare` was passing the caller's selector through to each fork. With the usual "last", that selector is re-resolved per branch — so after the first fork it pointed at the child just created, which has no checkpoints, and every later model failed with an error that had nothing to do with the model. The verdict table then read as though two of three models had failed the task. That is worse than an outright crash: the output looked like a result. Fixed by resolving the run once and forking each branch from that concrete id. The test pins the property the command exists for — every branch must share a parent and a fork point, because the comparison only means anything if the model is the sole variable. Also: a genuinely cheap model was rendering as $0.0000. Cost precision now scales to the value, since "cheaper" is the entire reason that column is there. An unknown model still shows a dash rather than a number. 632 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
Two things the product promised and did not do. `orca record` prints "orca replay <run> --ui" as the suggested next command, and that flag was silently ignored. It now serves the viewer when the replay finishes — on the child run after a fork, since the child is the one carrying the outcome you asked the question about. `compare` had no --verify, so its VERDICT column meant only "the agent exited 0". That is not the question anyone asks of a comparison table, and a verdict that looks like an answer while measuring something else is the same failure mode as a confidently wrong cost. The command now runs a verify command inside each fork's worktree and takes its exit code as the verdict, falling back to the agent's exit code when none is given. The worktree test earns its keep: it greps for content only the agent writes, so it fails if the verify command runs in the original workspace rather than the fork. That required the fixture to stop writing to an env-pinned path and use its own working directory, which is what a real agent does anyway. 636 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
Motion in a debugger is functional or it is noise — the reader is mid-incident. So there is exactly one animation idea here, and it is the product: this is a replay tool, so the viewer can now replay. Space or the transport button steps the selection through the timeline, held on each row for as long as the recording says the gap actually was. That timing is the whole point. A uniform tick would be simpler and would throw away the only thing playback adds over pressing j: the feel of where the run stalled and where it span in a tight retry loop. Gaps are compressed on a square-root curve so small ones stay distinguishable while a ten-minute wait registers without being endured, clamped to 60ms–1s. Speeds are 1x, 4x, 16x, real time first. Everything else supports that. A playhead slides to the selected row, giving playback a physical position. A progress rail shows how far through the run you are. The detail pane enters from below going forward and from above going back, so you know which way you moved when something else is doing the moving. Any manual navigation — key, click, filter — takes the wheel back immediately. All of it is transform and opacity only, so nothing reflows on a large trace, and all of it is disabled under prefers-reduced-motion, where playback still works and simply steps instead of gliding. The attention pulse started as a box-shadow ring and the design-system test rejected it: this system uses hairlines and weight, never depth. It is now a composited overlay whose opacity animates, which is both in-system and cheaper. Two bugs the tests caught: restarting playback from the end selected the first row and then immediately stepped past it, so you never saw it; and the client assumed a browser `window` that its own test harness does not provide, which would have broken navigation rather than just motion in any embedding context. 656 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
Closes the one item from the original brief that was outright absent. A PATH shim in front of sh and bash records what the protocol layer structurally cannot: the exit code, the real wall duration, and which stream each byte came out of. The model only ever sees the harness's rendering of stdout, one turn late and often truncated. Three design decisions, each from a failure the tests forced: Resolution excludes the shim directory by *resolved real path*, not string compare. Our directory is first on PATH, so a shim that fails to exclude itself re-executes itself forever — inside the user's agent run, which is the worst place to find out. PATH routinely spells the same directory several ways, and a string compare misses those. The shim parameters travel as argv, not environment. An agent is entitled to sanitise the environment of commands it runs, and an env-carried shim would have kept working while silently capturing nothing — precisely the failure this layer exists to prevent. Byte counting requires piping, which strips the TTY, so it only happens when there is no TTY to strip. A harness that checks isatty must not change behaviour because we were watching. Every capture-side failure is swallowed: an unwritable frames file, a missing runner, a spawn error. Losing a frame costs a line in the trace; throwing costs the run the user was actually trying to record. The byte-fidelity test compares shimmed output against unshimmed rather than a hand-written expectation — the first version asserted the author's understanding of printf and failed while the shim was correct. 25 new tests. README and architecture docs now say this works, because it does. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
A model-versus-model result grounded in a real task is the most shareable thing this tool produces, and it was only reachable as terminal output. `--share` writes it as one self-contained SVG that renders anywhere with nothing installed. The card states what it measured — the fork point and the verify command — next to the verdicts. A cost column with no stated verdict command is a number people quote out of context, and this project already refuses to print a confidently wrong cost. Model names come out of a trace, which is someone else's machine, so they are escaped. The failing row reads by form as well as by word: solid inverse, per the design system, not a colour. Two test corrections worth noting. The "no external reference" assertion originally rejected any URL-shaped string and so rejected the SVG namespace declaration, which is an identifier and never a request; it now asserts the property that matters. And the first render shipped with its column headers drawn underneath the first row's background — a geometry bug no assertion covered, now pinned by one comparing header and row positions. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
Three how-to pages titled the way people search — "my agent broke something", "why did my agent delete that file", "would a different model have got this right" — rather than by feature name. Someone hitting this problem is not looking for "fork replay"; they are looking for their symptom. Each page ends in the commands that answer it. Twelve good-first-issues with the file to start in and a done-when condition, ordered by how much each actually helps: an adapter brings a whole agent's userbase, a provider unlocks a model nobody could fork onto, a redaction rule protects everyone, an analyzer proves the format is useful to people who did not write it. The adapter section asks contributors to answer one question before writing code — does the agent respect a base-URL environment variable — because that single fact decides whether the work is an afternoon or a week, and the answer is worth filing even with no code attached. The redaction section asks for false-positive samples as explicitly as it asks for new rules. The entropy detector already needed a mixed-alphabet guard because a long camelCase identifier scores 4.08 bits/char, and traces are mostly source code. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
The launch gates in the design say the demo is the first thing in the README, because it is the only element that converts and prose above it is a tax on that. This is it. An animated SVG rather than a GIF: no recorder, no terminal emulator, no encoder in the toolchain, five kilobytes, crisp at any zoom, and diffable in review. It also obeys the rule everything else here obeys — self-contained, no external reference, renders with nothing installed — and it honours prefers-reduced-motion by showing the finished session instead of animating. Lines appear on the timings of the session rather than a uniform tick, which is the same argument the viewer's playback makes: the shape of the gaps is the information. Regenerate with `node scripts/render-demo.mjs > docs/demo.svg`. The first render silently lost the compare table's column alignment, because SVG collapses runs of whitespace unless told not to — and that alignment is the one thing the table exists to show. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
Two templates for the contribution paths that most need a low-friction on-ramp. The provider template states the price-returns-null rule up front, because a contributor who guesses a cost produces a number that lands in a comparison table someone decides from. The redaction template opens by telling people not to paste the secret, and gives false positives their own checkbox with the reasoning attached: traces are mostly source code, so a rule that over-fires corrupts exactly the payload the trace exists to preserve. Reports of over-redaction are as valuable as reports of misses, and nobody files those unless invited. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
Closes the remaining gaps from the audit against the original brief. `orca gc` reclaims space, and its most important behaviour is what it refuses to delete: a run that is the parent of a surviving fork. Forking is the product's whole point, and silently orphaning a fork's provenance would be data loss. Chained forks needed a fixed-point rescue loop rather than one pass — a single pass fails the whole-chain test. `orca doctor` answers "will this actually work here" before someone spends an hour finding out. This tool injects environment variables into other people's processes and shells out to git; when that does not work, the user currently has no way to see why. Its git checks go through the same hardened spawn path recording uses, not an approximation of it. Adapter contract tests close the project's own top-listed risk, which until now had no guard at all. Every registered adapter is checked automatically, so a new one cannot forget to opt in. It immediately found a live bug: the opencode adapter invented an ANTHROPIC_API_KEY when only OPENAI_API_KEY was present, and OpenCode picks its provider from the credentials it can see — so recording would have silently changed which model the agent called. Fixed. The Python SDK is read-only by design. The TypeScript writer stays the one implementation of the write path, because two writers means two redaction implementations, which is how a secret leaks. It exists because everyone who turns traces into eval datasets works in pandas, and because a second implementation is the only real proof the format is not just a TypeScript library with a spec attached. 783 TypeScript tests, 162 Python tests, verified from a clean npm ci. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
The spec gives the manifest parent_run, fork_point and fork_model, and all three were dead. Fork provenance existed only as an attribute on the `fork` event, so anything reading a manifest — the Python SDK, a third-party tool, `orca gc` deciding whether a run may be deleted — saw an orphan. `orca gc` is the sharp end of that: it refuses to delete a run that is the parent of a surviving fork, and it only avoided deleting every parent ever forked because it reads the event stream as well. That is a workaround for a gap the format did not intend. Found by the gc agent while implementing the parent-run protection, which is a good argument for building the consumer of a format field before assuming the field works. Provenance keys are spread conditionally rather than set to undefined: the schema forbids unknown keys, and a plain recording should carry no provenance at all rather than three nulls every reader has to special-case. Also aligns the `orca scrub` help line to the same column as the other ten. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
Four defects found by running the tool as a new user would, against a real Claude Code, rather than against fixtures. 1. Replay matched nothing on any real recording. The writer spills a payload over the inline limit as `JSON.stringify(payload)`, and every wire body is a string — so a spilled body is stored as a quoted, escaped JSON string literal while the same body under the limit is stored as itself. Replay read the blob bytes back without undoing that, canonicalized an escaped copy (`model: ''`, `messages: []`), and dropped every request to rung 4. Invisible in tests because every fixture body sat under 4 KB; fatal in practice because a real harness sends a system prompt and tool catalogue that spill on turn one. Against a real recording this moves `matched=0 unmatched=12` (agent hung) to `matched=2/3 unmatched=0`, replaying the recorded reply with the network off. 2. The halt was a 409, which sits in the Anthropic and OpenAI SDKs' default retry set. The harness re-sent the unmatched request until its budget ran out and then stalled, so the operator saw a hang rather than a reason. Now 400, in the provider's own error shape, with the message clients print. 3. An unmatched request was counted and never explained: `unmatched: 12` with no reason. Added `onUnmatched`, surfaced as a `replay.unmatched` warning as it happens, and a halted replay no longer reports exit 0. 4. Rung 2 accepted any difference inside a tolerance with a 64-character floor — on a short request, half the body. Swapping the user's question for an unrelated one landed inside it and was served the recorded answer labelled `minor`. Rung 2 now requires an identical trailing message: it is for drift around the question, not the question itself. Also: `orca record claude` failed with "unknown adapter", though the README headline, the demo and every doc use it — adapters now carry aliases, with the canonical id still the one written to the manifest. And the install section promised `npx orcareplay`, which does not resolve because v0 is unpublished; it now documents the from-source path that works, including that `npm install -g .` at the workspace root installs nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
Three smaller findings from the same pass over the built CLI. `orca compare --models a,b,c` leaves the parent plus one run per model in the directory, and `orca list` showed them as four unrelated ids — the output of the headline command was unreadable the moment you came back to it. `listRuns` now reads the provenance already in every manifest and the table carries a FROM column; `orca show` names the parent, the checkpoint and the model it forked onto. `orca show` printed cost with four fixed decimals, so a genuinely cheap run read as `$0.0000` — free rather than small. It now uses the same scaling formatter as the compare table. CI built and tested the TypeScript packages but never ran the Python SDK, which reads the same normative format; a format change could break it with CI green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
Exact replay reproduced the model loop and nothing else, which is not enough for any harness that touches files. Two facts pull against each other, and the previous behaviour lost to both. A harness reads files into the conversation, so the bytes on disk end up inside the recorded request, and the run then edits those same files — replaying in the directory you recorded in re-reads what the recording changed and halts at rung 4, correctly but uselessly. And a harness writes *absolute* paths into its tool calls, so replaying a copy of that directory somewhere else makes the agent read outside its working directory, where it gets a permission refusal in place of the file and diverges just as badly. Measured on a real 42-event Claude Code run that edits a file: a scratch copy halted at the first tool result; the same trace restored at its own path replayed all six exchanges with nothing unmatched and exit 0. So replay now restores the recorded state over the working tree, at the path the run was recorded in. That is defensible only because it is reversible — the current tree is snapshotted into a scratch store first and put back in a `finally`, so a replay is observationally a no-op on your checkout, and the snapshot id is printed before anything is touched in case the process is killed in between. `--worktree` keeps the scratch copy, `--in-place` restores nothing, and a replay invoked from a directory the run was not recorded in never restores. A halt now names the recording's directory and the replay's alongside the distance. "distance 205343" is true and unactionable; the two paths are what tell you whether the harness was reading somewhere it was never given. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
The timeline rendered each row's `meta` and dropped its `detail`, so the DETAIL column carried token counts for a model response and was empty for everything else. What that hid was exactly what the capture layers exist to record: a shell command's exit code — named in the shell shim's own docstring as the thing the model never sees — showed as an empty cell beside its duration, tool results lost their success or failure, and a tool call and its result rendered as two identical rows, which made a real 42-event trace look like it had been printed twice. Both halves now render. The same trace goes from 7 TOOL Bash 9 TOOL Bash to the command that ran and the result it returned, and shell frames finally show `exit 0 · 9ms` rather than `9ms`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
The job I added two commits ago would have failed for a real reason the moment Actions started working. Its comment claimed there was nothing to install because the SDK is standard-library only — true of the SDK, and irrelevant: `actions/setup-python` hands you a clean interpreter from the tool cache, and pytest is not on it. Reproduced in a fresh venv: $ python -m pytest python/ -q /tmp/.../bin/python: No module named pytest It installs `./python[dev]` rather than bare pytest, since the package already declares that extra and installing the package also checks that `pip install` works at all — which is how anyone will actually get it. Verified in a clean venv: install exit 0, 180 passed. Also points the package's Homepage and Specification URLs at this repository. They pointed at github.com/orcareplay/orcareplay, which does not exist, and package metadata is the one place a broken link is permanent. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
Three defects found by opening the viewer in a browser and looking at it,
which no test here had ever done.
**Events past the fold were unreachable.** `.list` and `.pane-col` are grid
items, and a grid item defaults to `min-height: auto` — so they grew to their
content instead of to their track, and the `overflow: auto` on `.rows` was
handed an unbounded height. Measured on a 42-event trace in a 1440x900
window: `.rows` reported `scrollHeight === clientHeight` (nothing to scroll),
grew to 1099px inside an 869px column, and painted the overflow into
`body { overflow: hidden }` where it was clipped. The last row sat at y=1228
with no way to scroll to it. Everything from event 27 on simply did not
exist for the user, and the taller the trace the more of it vanished.
Now `minmax(0, 1fr)` on the grid row and `min-height: 0` on both items.
Verified in Chromium: `canScroll: true`, and event 41 lands exactly on the
footer edge.
**Filesystem events rendered blank.** The renderer read `added`, `removed`
and `files`; `orca record` writes `insertions`, `deletions` and `changes`.
Every FILE row showed an empty detail and never said whether the file was
added, modified or deleted — the one thing it exists to say. It survived
because the tests invented their own attribute shape and the renderer agreed
with the invention, so the new cases use the writer's real payload. A row now
reads `auth.ts modified +1 −1`.
**The footer still advertised `npx orcareplay`**, which does not resolve.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
Adds the OrcaCode Review action, which reviews each pull request with an LLM, posts findings as inline comments, and fails the `review` check when it finds something serious. Two choices are worth recording, because neither is the shipped default. `block-on: "P0,P1,P2"` blocks the merge on every severity rather than the default `P0,P1`. Advisory findings will turn the check red; that is the intent, not an accident. Note that P3 exists in the engine's rubric and in the console's display settings but not in this gate — `block-on` accepts P0, P1, and P2 only. `settings: "false"` skips the dashboard fetch entirely, so this repository reads none of the workspace defaults and no console change can reach it. The cost is that the settings which exist only in the dashboard — model choice, exhaustive mode, quiet mode, and the custom rubric — are unavailable here. Of those, only model choice was actually in use, and the trade buys something worth more: what reviews this repository is decided by a file under review, not by an account setting shared with every other repository on the key. The `if:` condition and the `on:` block are the action's own. They encode two things that are easy to break by tidying: `pull_request_target` is what lets a fork's pull request be reviewed at all, and the author-association check on `issue_comment` is what stops any drive-by commenter from spending the key's wallet with `/orcacode-review`. Because `pull_request_target` reads the workflow from the base branch, this pull request will not review itself. The first review happens on whatever opens after this merges. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three gaps from the audit, all in the live path.
**The proxy buffered.** `goLive` did `await upstreamRes.text()` before writing
a byte, so a model response that arrives over seconds reached the agent all at
once, after the model had finished — every turn of an interactive session
appeared to hang for its full duration, and the agent's own progressive
rendering had nothing progressive left to render. `docs/architecture.md`
claimed the opposite ("tees a canonical copy while streaming through").
It now tees: chunks go straight to the client, the recorded copy is assembled
on the way past, and a full socket applies backpressure rather than buffering
on the agent's behalf. The test is built so buffering cannot pass it by luck —
the stub upstream withholds its second chunk until the *client* has received
the first, so a buffering proxy deadlocks, and a race turns that into a
legible failure instead of a hung suite.
**Failed exchanges were dropped.** Recording was guarded by
`if (upstreamRes.ok)`, so a run that died on rate limits produced a trace with
no evidence of why, and replay then came up short exactly the exchanges that
explain the failure you opened the trace to understand. Every exchange is
recorded now, whatever its status.
**`--no-color` was inert.** It is in `orca --help`, and nothing read it:
`Output` computed colour from isTTY/ci/NO_COLOR alone. The `bool('color', true)`
fallback is what distinguishes an explicit `--no-color` from an unset flag —
`bool('color')` alone returns false for both, which would have disabled colour
for everybody.
Also documents four flags that were load-bearing and invisible: `--from` on
compare, `--port`, and `--upstream-anthropic` / `--upstream-openai`, which are
the only way to point a run at a gateway.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
The hero was a hand-written console block that no code produces — invented run ids, an invented "3 capture layers active" line, an invented "68/68 matched exact". For a tool whose whole pitch is that it shows you what actually happened, that was the worst possible thing to ship. It is now a GIF of one real session: a Claude Code run recorded, replayed with the network off, then forked at checkpoint 4 onto two models and graded by `npx tsc --noEmit`. `docs/media/transcript.txt` is the raw capture, so every line in the animation can be traced back to output a command actually produced, and `scripts/render-demo.mjs` regenerates the GIF from it. Adds two sections that were missing entirely — the viewer and compare — each with real media: the timeline filtering a 42-event run down to its tool loop, and a verdict card from two models forked off the same checkpoint. A GIF rather than an animated SVG because GitHub, X and HN all render a GIF the same way; `docs/demo.svg` (also fabricated — it advertised an `mcp=on` flag record does not emit) is deleted rather than fixed. The three packages the generator needs are deliberately not in package.json: they redraw README art and nobody running `npm ci` to work on OrcaReplay should pay for them. Note for whoever comes next — the ffmpeg bundled with Playwright cannot do this job at all, having neither a PNG decoder nor a GIF muxer, so the encoder is pure JavaScript. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
Two writer/renderer disagreements, both found while correcting the docs that described the behaviour they were supposed to have. **`causes` was empty on every tool result ever recorded.** The deriver built the `tool.result` event and dropped the pending call from its map in the same loop — but the recorder resolves the call's seq *after* derive returns, because the seq only exists once the event has been written. By then there was nothing left to answer with, so the lookup returned undefined and the field was quietly omitted. No error, no warning: just a flat event list where the format promises a chain you can walk. The entry now outlives the result that answered it, and `#closed` — which already existed to stop a resent result being re-emitted — becomes the authority on whether a call was answered, so `unresolved()` stops counting closed calls as orphans. Verified on a real Claude Code recording: `tool.result` seq 9 now carries `causes: [7]`, which is its `Bash` call. **A failed tool call rendered as `ok`.** The recorder writes `is_error` from the provider's own tool_result flag; the viewer checked `error` and `ok`, neither of which it ever sets. So a failure showed the word "ok" in a normal tone. Blank would have been bad — asserting that a failure succeeded is worse, because the timeline is precisely where someone goes to find the failure. Same family as the fs.change mismatch fixed earlier, and the reason both survived is that the tests invented their own attribute shape and the renderer agreed with the invention. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
`orca doctor` answers "will recording actually work here", and it checked node, git, the workspace, disk, a port and which agents are installed — but not two of the four capture layers it exists to vouch for. Both fail in the same quiet way, which is exactly why they belong here. They are shims: a PATH entry in front of sh/bash, and a rewritten MCP config launching a JSON-RPC tee. When either cannot start, nothing visible goes wrong — the agent runs, the run succeeds, and the trace is simply missing a layer, which you find out when you go looking for an exit code that was never recorded. `doctor` exists to move that discovery forward in time. The shell check installs a shim into a temp directory, runs a command through it, and confirms a frame came back — not that the scripts were written, which is a different and weaker claim. Verified by deleting the compiled runner: it reports `warn` naming the missing file and tells you to build or pass --no-shell, rather than failing later inside somebody's agent. `mcp.ts` already had `shimIsRunnable()` with the docstring "Exposed for the doctor command". Nothing had ever imported it. Both warn rather than fail: a missing capture layer degrades the trace, and refusing to record at all would cost the user the run they came for. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
Missed in the previous commit; CI gates on prettier. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
…ject Three things, one of which is a bug fixed before it could bite. **`orca setup`** — the shortest path from "I want to compare four models" to a working `orca compare`. Doing that by hand meant knowing `--upstream-anthropic` and `--upstream-openai` exist, that one gateway serves both wire formats, and where the key goes; all real, none discoverable. It asks instead, stores a gateway, a key and a default model list at `~/.config/orca/config.json` (0600 inside a 0700 directory), and — the part that earns the command — asks the gateway what it actually serves, so a wrong URL or a dead key is an answer now rather than a 401 in the middle of a comparison. `orca models` lists what is available with prices where they are known and a dash where they are not, because inventing a number for an unknown model is how a comparison table ends up quoting a cost that was never real. The key never reaches a trace by construction rather than by a rule: it is attached to the outbound request only, while what gets recorded is built from the incoming request with auth stripped. It is also withheld entirely when a flag sends that traffic somewhere other than the gateway that issued it — pointing `--upstream-anthropic` at the vendor while a gateway is configured must not hand that vendor a credential the gateway issued. `auth=stored` rather than `key=stored` in the output, because the terminal guard redacts any field named `key` — correctly, and it would have hidden the one fact that line exists to convey. **Exact replay writes its own trace** (spec §4: every inexact match is an event in the trace, which held on the fork path and nowhere else). A new run with `parent_run` set and no invented `fork_point`, holding the divergences and the unmatched requests. The served exchanges are deliberately *not* copied: each was read out of the parent and handed back byte for byte, so duplicating them doubles the store for content identical by construction. An unmatched request is the converse — something the agent asked that the recording never contained, so it exists nowhere else and is recorded. The rule: the trace holds what replaying discovered, and points at the parent for what replaying merely repeated. `--no-trace` opts out. **And the guard in `orca gc` was wrong.** It decided a directory was a reclaimable fork worktree from `parent_run` plus a `$TMPDIR/orca-*` path shape. Replay traces have `parent_run` and the *user's own cwd* — so a project living at a path of that shape would have been deleted by `orca gc --older-than 7d`. The commit that added it claimed the fork check was "the load-bearing half of the guard"; this feature quietly invalidated that. It now keys on `fork_point`, which is exactly the field an exact replay lacks, restoring what the comment already claimed. Cost: a `--worktree` exact replay's scratch directory is no longer reclaimed, which leaks a tmpdir the OS cleans up — against deleting someone's work. 971 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
`parent_run` used to imply a fork, and since exact replay started writing its own run it does not: a replay trace has a parent and deliberately no `fork_point`. `orca show` rendered that as "forked from run_x at checkpoint undefined" and `orca list` as `run_x@?`, both of which read as a corrupt trace rather than as the different thing they describe. Keyed on `fork_point`, which is the field that actually distinguishes them: "replay of run_x" in show, and the bare parent id in list. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
The remaining capture gap: OrcaReplay redirects a base-URL environment variable, which captures nothing from a harness that ignores one — a Codex CLI signed in with a ChatGPT subscription being the concrete case. SECURITY.md said this was not implemented. It is now, opt-in, per run. `node:crypto` can generate keys, sign, hash and *parse* certificates, but it cannot mint one, and there is no API for it. Every MITM proxy in the ecosystem reaches for node-forge or selfsigned here. Instead there is a ~200-line encoder-only DER writer covering exactly the ASN.1 that RFC 5280 needs for a certificate we construct ourselves. It never parses anything, so the usual ASN.1 attack surface does not exist, and verification is left to OpenSSL where it belongs. No dependency added; the lockfile is unchanged. The security properties are the reason this feature is contentious, so each one has a test that fails without it: - CONNECT is refused entirely unless interception was asked for. - A host outside the allowlist is tunnelled without being decrypted, and a bare wildcard is rejected — it is a request to decrypt everything. - The default list is model API hosts and nothing else, and it deliberately leaves the OpenAI sign-in host alone, because that flow carries the credential itself. - The CA key sits at 0600 inside a 0700 directory and is deleted when the run ends; the certificate expires within a day, so a leaked key is worthless tomorrow. - The child trusts the CA through a bundle that *adds* it to the system roots rather than replacing them, and never through a system trust store. - An origin whose certificate cannot be verified is refused rather than downgraded. - Intercepted model traffic is recorded as a model exchange rather than as opaque net traffic, and forwarded as it arrives rather than buffered. 1001 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
Review pass one, reading the interception code rather than its test names. The property the whole feature stands on had no test. The CA key exists so orca can impersonate a host to the child; a copy of it in a trace someone attaches to an issue is a signing key handed to whoever reads the issue. Nothing in the capture path has any reason to touch it, which is exactly why it is worth asserting — that is the kind of invariant a later refactor breaks with no error to notice. Writing it turned up a better guarantee than the one I was checking for. `RunCa` does not expose the key at all: it is a private field, and the file at `keyPath` is the only place it exists, so the capture path cannot reach it even by mistake. The test now reads that file and defends the one remaining route — something copying its contents into a record — checking both the PEM and its base64 body with the armour stripped, which is the shape it would take embedded in JSON. Mutation-verified: adding the key to the emitted exchange fails the test; removing it passes. Also confirmed by reading, not by test names: `rejectUnauthorized` is left at Node's default of true so an unverifiable origin is refused rather than downgraded, SNI is set for named hosts and omitted for literal addresses, extra roots are concatenated with the system store rather than replacing it, and the wildcard matcher requires the leading dot — so `*.chatgpt.com` cannot match `evilchatgpt.com`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
The previous commit is red, and that is my fault rather than a defect: I ran `git add -A` while the interception work was still being written, so it swept in a half-finished `record.ts` and a CLI test file whose three cases had no implementation behind them yet. The message described only the CA-key test I had added, which is not what the commit contained. Fixing forward rather than rewriting pushed history. This completes it. `orca record --tls-intercept` stands up the run CA, points the child at the proxy with `HTTPS_PROXY`, and hands it a trust bundle through its own environment — never a system trust store. Off unless asked for, which is the absence of a CONNECT listener rather than a branch inside one. 1012 tests, 58 files, build and format clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
Removing the key was the teardown path's job, which means it did not happen on
any path that never reaches teardown: an agent that fails to launch, a throw
mid-recording, a Ctrl-C. A private key surviving a crashed run is not an
acceptable failure mode for the thing whose entire justification is that the
key is ephemeral.
A `process.once('exit')` backstop covers those, using the sync fs API because
exit handlers cannot await — best effort by construction, since if it fails
there is nothing left to do. The listener is removed on the ordinary path
first: leaving one behind would keep the closure, and a reference to the run,
alive for the life of the process, and orca records more than one run per
process.
Verified against a real run earlier: after `orca record claude --tls-intercept`
finishes, the `tls/` directory is gone and grepping the whole run directory for
"PRIVATE KEY" returns nothing.
1015 tests.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
…was safe **The gateway key was sent to whatever host a flag pointed at.** The guard was `.some()` over the dialect origins, so redirecting one dialect left the other still defaulting to the gateway, the check passed, and the key was returned — and `upstreamHeaders` are attached to every live call, there being no per-dialect header channel. Verified against the built CLI: with a gateway configured, `--upstream-anthropic https://api.anthropic.com` produced headers carrying `sk-gateway-key`, which then went to Anthropic on every `/v1/messages`. This is the exact failure the comment claimed to prevent and the README promises against, and I told the user it was handled. The test that was supposed to cover it redirected *both* dialects, which is the only arrangement that worked. The rule is unanimity now: attach the key when every origin we might reach is the gateway, otherwise not at all. Withholding costs an unauthenticated request and a clear 401; the other way hands a third party a credential and says nothing. **Boolean flags ate the following positional.** The parser had no notion of a value-less flag, so any flag followed by a non-flag token consumed it and `bool()` then returned its fallback — the flag silently off *and* the positional gone. `orca replay --worktree last` is documented as "never touches your files" and instead restored the recorded tree over the working directory. `orca record --tls-intercept codex` turned interception off and lost the agent name, so record auto-detected and could capture a different harness. A declared list rather than a heuristic, because guessing from the next token gets `orca compare --models last` wrong. **`last` resolved to the replay trace.** Exact replay writes its own run, so the newest run in a directory is usually a report about the one before it, and every command defaults to `last`. `orca scrub last --match my-hostname` — the line in the README — scrubbed the empty trace, found nothing, and printed "nothing matched — the trace is unchanged" while the secret sat in the recording beside it. `last` now skips replay traces; a fork stays eligible, naming a trace explicitly still works, and a directory holding only traces still resolves. Verified: the same command now removes 63 matches across 8 files. Every one of these is a case where a test existed and passed while the behaviour was wrong, because the test happened to use the one argument order or the one flag arrangement that worked. 1024 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
Found by smoke-testing the built CLI rather than by reasoning:
`orca record --tls-intercept -- <missing-binary>` threw before any teardown
ran and left `ca.key` sitting on disk. The likeliest failure of all — the
agent is not installed — happened before the code that deletes the key.
`recordCommand` is now a thin wrapper that owns the key's lifetime, so no
failure path inside can skip it, backed by the `process.once('exit')` handler
already in `RunCa` and a 24-hour certificate expiry as the last resort. Three
independent guarantees for one property, which is proportionate for a signing
key that exists so orca can impersonate hosts to a process.
The test drives the real failure: record against a binary that does not exist,
then assert the key is gone.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
Two failures, one cause: `recordCommand` had no `try/finally`, so a throw from the child launch skipped both `proxy.close()` and `writer.close()`. The visible one is a hang. A listening server keeps Node's event loop alive, so `orca record generic-openai -- /nonexistent` printed its error and then never exited — verified against the built CLI and killed at 12 seconds. Mistyping an agent name is close to the first thing a new user does. It now exits immediately with "spawn ENOENT / is it installed and on your PATH?". The quiet one is an unsealed trace. Everything the agent did is on disk, but with no `ended_at`, `counts` or `integrity`, so `verifyIntegrity` reports the run as *tampered* rather than as unfinished — and the same shape loses a complete recording to a single failed write. It is sealed on the error path now, with `run.end` carrying the error and no exit code, since the code is unknown rather than zero. Sealing rather than discarding, because a run that died is often the one you most want to read. Also: SECURITY.md said "TLS interception: not implemented", which stopped being true a few commits ago. Replaced with what the feature actually does — opt-in per run, ephemeral CA at 0600 in a 0700 directory deleted on every exit path, never a trust store, allowlist-only with sign-in origins deliberately excluded, origin verification never disabled, and the two things it refuses loudly rather than silently (HTTP/2 and websocket upgrades inside an intercepted session). And the conformance coverage line reported `net.request`/`net.response` as unexercised while the recorder was emitting them, which is the opposite of the honesty that report exists to provide. Its synthetic run now writes a pair. 1025 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
An unsealed run reported as tampered. `verifyIntegrity()` returned `ok: false` for a run whose recorder was killed and for a run whose events file had been edited under it, and every caller that reports to a person read that as the second. A recorder that dies leaves the run you most want to open, and telling its owner the trace "changed since the run ended" is an accusation about a file nothing touched. `IntegrityResult` now carries `state`, and the Python example stops printing "MISMATCH expected got <hash>". `orca scrub` could destroy the evidence it was cleaning. It wrote `events.jsonl` with a plain `writeFile`, which truncates before it writes: a ^C in that window left a truncated events file and a manifest digest describing the whole one. Nothing is written now until every file has been scrubbed and checked, and each lands by rename. It also had no `--dry-run` — the one command in the tool whose effect cannot be undone, and the only one you could not look at first. It fires the standard detectors alongside your literal, so what it takes out is genuinely not knowable in advance. Exact replay leaked a copy of your working tree per invocation. The pre-replay safety snapshot is a shadow git store holding the whole workspace, left in $TMPDIR forever, in a directory `orca gc` deliberately will not sweep. Replay owns it, so replay cleans it up — after the restore succeeds, never before, because on the failure path it holds the only copy of your uncommitted work. `--tls-intercept` was accepted by every command and honoured by one. `orca replay --model`, `orca fork` and `orca compare` all launch a real agent, and all three took the flag and threw it away — so a harness talking to its own backend over TLS went uncaptured while the operator believed otherwise, and the absence of `net.*` events read as "the agent made no other calls". The setup, the child trust wiring and the audit note now live in one module both paths use. A fork that mints a CA now disposes it on every exit path, and no longer hangs when the agent will not launch — the same two failures `orca record` had. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
Writing an end-to-end test for the MCP half of 763c844 found the layer had never functioned at all. `resolveShimEntry()` asked for `@orcareplay/mcp-shim/dist/cli.js` and fell back to `@orcareplay/mcp-shim` when that threw. It always threw: the package declares an `exports` map, and an exports map blocks every subpath it does not list. So the fallback ran every time and resolved to `dist/index.js` — the library. The damage is worse than a missing capture layer. Every stdio MCP server in the agent's config is rewritten to launch that path, and the library exports and exits, so `--mcp-config` silently *broke the agent's MCP servers* while recording nothing. `orca doctor` reported the shim runnable throughout, because it asked only whether a process exited cleanly and the wrong one exits 0. Behind that sat the same class of bug again: the shim writes `{name, dir}` and the recorder read `{server, direction}`. `JSON.parse` casts to whatever the call site claims, so nothing disagreed — every frame would have arrived with an undefined server, and `direction === 'in'` false for all of them, recording every request as a response. The record type now lives with the shim that writes it and the recorder imports it. Also in this pass: - The three copies of the secret-header set had drifted. `api-key` (Azure) and `x-goog-api-key` (Google) were known to the TLS interceptor alone, so the same credential was stripped on the intercepted path and written on the recorded one. One list in core now, with the vendor names pinned by name — a test that only iterates the list gets smaller when a name is deleted from it, it never goes red. - `MatchRung` and `DivergenceLevel` were restated in the proxy rather than derived from the schema's own constants; `MATCH_RUNGS` had no reference at all and could have been changed without a test noticing. - Two tests that could not fail. `expect(mono_us).toBeGreaterThanOrEqual(0)` is true of every possible number including the drain reading it was meant to rule out, and the shell equivalent was true whether the timestamp was real or not; both now assert a gap the wrong implementation cannot produce. - `orca doctor`'s own tests ran against the repository root, where the `.orca writable` check writes a probe file — a test that mutates the checkout. - The code-review action ran from a floating `@v1` tag in a `pull_request_target` job holding a secret; pinned to the commit it resolved to. - Docs that had become false: network capture is no longer "not implemented", `--tls-intercept` had no prose at all, exact replay's second run was undocumented, `orca ls` is not a command, and the neutrality claim said CI enforces something it cannot until a vendor plugin exists. - The `Provider` layer is a published extension point with no in-tree consumer, and both its docstring and `docs/plugins.md` claimed the live path went through it. It does not, deliberately: the proxy forwards the agent's own bytes, and routing that through a `Provider` would lose fidelity for nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
Running the built CLI end to end turned up a hole in the core loop:
orca record ...
orca replay last --from 4 --model X
orca compare last --models a,b
→ this run has no checkpoints, so there is nothing to fork from
A fork ran a live agent in a worktree and recorded only the conversation. A
checkpoint is *derived* from a filesystem snapshot (spec §3), so a fork had no
checkpoints at all — it could not be forked, and `last` resolving to it is what
turned that into a failure of the very next command. The tool is pitched on
iterative exploration; branching had a depth limit of one.
Forks now capture the filesystem the way a recording does: an initial snapshot
of the state they branched at, then one per turn, degrading to a warning rather
than aborting the run when the shadow index will not start. The snapshot-and-
append logic is one function both paths call, rather than a block in `record`
that `replay` never had.
Verified against the built CLI: `compare` immediately after a fork now forks the
fork and prints a real table for both models.
Also: `orca scrub --dry-run` printed `would_change=2` and then listed three
files. `files=N` has always meant files whose *contents* were scrubbed, and
`redactions.json` only gains a line saying how many removals happened — so the
ledger is now named on a line of its own instead of silently padding the list.
The conformance report now exercises `mcp.request`, `mcp.response`, `fork` and
`divergence`, leaving `route.decision` as the one declared type nothing emits.
The MCP pair could not have been added before this branch, because the layer did
not work — which is the coverage line doing its job.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
… agent Building a demo surfaced it: replaying a working MCP recording exited 2 with `matched=0`, and the agent printed "MCP_CONFIG_PATH is required". Replay and fork launch the same agent the recording did, so they had the same blind spot record had before `--mcp-config` existed — the harness either talks to servers orca cannot see, or does not start at all. The recording knew which config to use and had nowhere to write it down: `manifest.argv` holds the agent's own arguments and `--mcp-config` is orca's. So record now leaves an `mcp_instrumented` note carrying the source path, the same way a TLS run leaves one carrying the CA digest, and replay and fork read it back. `--mcp-config` on the replay still wins. Deliberately not reused: the rewritten config in the parent run directory. Its servers already point at the parent's frames file, so instrumenting from it would append this replay's traffic to the recording it is replaying. A source config that has since moved is said out loud rather than silently dropping the layer — an absent `mcp.*` would otherwise read as "the agent made no MCP calls" instead of "orca was not looking". The env wiring and the frame drain now live in one place both paths call, and the fork tracks when each turn began so a drained frame is attributed to the turn it happened during rather than to whichever turn was last. Verified against the built CLI: the same recording that exited 2 now replays `matched=2 total=2 divergences=0 unmatched=0 exit=0`, with the MCP pair in the replay trace and in the fork trace. Mutation-verified three ways. One test premise was wrong before it was right: deleting the config from the workspace does not remove it, because exact replay restores the recorded tree over the working directory and brings it back. The test now puts the config outside the workspace, where the restore cannot undo the deletion. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
`orca setup` asked for a gateway URL and offered nothing, which is a poor question: the people who need a gateway are the ones who do not have one. It now suggests OrcaRouter — visible in the prompt, one Enter to accept, overtyped or `--gateway`-ed away in a keystroke — and `orca setup --key <k>` works non-interactively without naming a URL, which used to fail with "no gateway to configure". The line it does NOT cross, and there is a test named after it: a default is not a redirect. With no gateway configured, `orca record` still proxies the agent's own calls to whatever provider it was already talking to, on the agent's own key. Rerouting an unconfigured recording would post someone's source code to a third party as a side effect of pressing record, on a key that would not authenticate there anyway. Facts checked against primary sources rather than assumed, since this hardcodes a vendor endpoint: - The origin is `https://api.orcarouter.ai`, deliberately without the `/v1` an OpenAI SDK is configured with — that SDK appends only `/chat/completions`, while orca appends the whole dialect path and probes `/v1/models`. A `/v1` here would produce `/v1/v1/chat/completions`. Confirmed against the maintainers' own published action. - Model ids there are namespaced (`anthropic/claude-sonnet-4.6`). Verified both readers already cope: dialect selection matches `(?:.*\/)?claude[-.]`, and `resolveModelId` strips the namespace before pricing. No code change needed — checked rather than assumed. - The key-shape hint is printed by the command, so the README transcript is real output rather than an aspiration. One thing the check turned up: `sk-orca-` keys were redacted, but under a rule named `openai_key`. `sk-` is not an OpenAI prefix, it is the convention half the industry copied — Anthropic's `sk-ant-` was mislabelled the same way. The rule name reaches the trace, `redactions.json` and the placeholder itself, so it now reads `sk_api_key`, with OrcaRouter and Anthropic cases in the table. The redaction policy version moves to 2 with it, because the placeholder text is part of what a trace records. The README says all of this where someone will read it, and the neutrality section now separates the two claims it was quietly conflating: a default you can see and overtype is not the same thing as a privileged API, and the vendor still gets none of the latter. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
Making OrcaRouter the default left every printed suggestion wrong for it. `orca setup` finished with `--models claude-opus-5,gpt-5.2` and `orca compare` errored with `--models claude-opus-5,glm-5.3-flash` — bare ids, against a gateway that namespaces by provider (`anthropic/claude-opus-5`). A copyable line that fails against the gateway orca just configured is worse than no line. `orca setup` now builds that line from what the probe actually returned, and says `orca models` when the gateway could not be reached rather than inventing two ids for a server nobody could talk to. `orca compare` points at `orca models` for the same reason: model ids are gateway-specific and this command cannot know them without asking. Also: `mcpSourceFrom` and `usedMcp` were exported with no consumer outside the module and no direct test — the same dead-surface smell this branch has been removing elsewhere. They carry the flag-over-recording precedence that decides whether a replay captures MCP at all, so they get unit tests rather than being quietly hidden. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
…he example trace **`route.decision` was declared and emitted by nothing.** Spec §2 defines it as "a gateway chose a model", and on a fork orca *is* the gateway: it substitutes the model, picks the wire format that serves it, and picks the origin. It was making all three choices silently, so a verdict table reading `claude-opus-5 vs gpt-5.2` said nothing about where either one went, and a fork that fell back to the recorded provider's origin looked identical to one that did not. It now records model, target dialect, the dialect the request arrived in, origin and reason — and stays silent on an ordinary recording, which chooses nothing. That last property is only observable at the proxy, so it is tested there; the CLI tests could not have caught an over-eager emit. **The `Provider` layer is gone** — `http.ts`, `anthropic.ts`, `openai.ts`, `registry.ts` and their two test files, about 400 lines of production code and 570 of tests. All seventeen exports had zero consumers outside the package, checked one by one. It survived this long on a docstring claiming the live path went through it; it never did, and it never could without losing fidelity, because the proxy forwards the agent's own bytes rather than re-serialising a request it already holds verbatim. The `Provider` interface stays published in `@orcareplay/plugin-api`: an interface is cheap to publish, a wrong implementation is not. **The shipped example trace recorded a `checkpoint` event.** Spec §3 says a checkpoint is derived at read time and never recorded, so the one fixture whose job is to show a third-party implementer what the format looks like was teaching them to emit the event the format forbids — and a Python test asserted the fixture carried one, which made a passing suite out of it. The event is gone, seq renumbered dense with `causes` remapped, digest and counts refreshed, and that test now asserts the opposite. Goldens were recomputed from both the Python and TypeScript implementations independently and cross-checked, so the cross-language guarantee still holds rather than being re-pinned to whatever one of them printed. Two tests were pinned to fixture arithmetic rather than to behaviour — a hardcoded `snapped back from 20`, and an example that found its snap target with `seq + 3`. Both broke for a reason unrelated to what they test. They now find a non-checkpoint seq instead of guessing one. The conformance report separates a type that is unexercised by design from one that is unexercised by omission, because reporting `checkpoint` beside real gaps is what invites someone to "fix" it by emitting it. Every other declared type is now exercised. README: a hook, badges, a three-command quickstart above the fold, and a table against observability tools — the install instructions used to sit 140 lines below the first screen. Plus a help-wanted section that names the file to start in, since the format is v0 and the decisions are still cheap to change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
…rning `npm ci` on a fresh clone printed **5 vulnerabilities, 1 critical** — the first thing a newcomer sees, and enough to make some of them close the tab. All five were the vitest/vite dev toolchain (`npm audit --omit=dev` was already clean), but "it's only dev dependencies" is a thing a maintainer knows and a visitor does not. vitest 2.1 -> 4.1 and esbuild 0.24 -> 0.28: **0 vulnerabilities**, and the suite passes unchanged across the major bump, viewer bundling included. OrcaRouter now has the top of the README: a linked mark above the badge row and a line saying what it is and why `orca setup` points there. The mark in `docs/orcarouter.svg` is a PLACEHOLDER wordmark, not the real brand asset — this environment blocks orcarouter.ai outright (403 at the CONNECT tunnel), so the logo could not be fetched and no URL for it could be verified. Guessing a path would have put a broken image at the very top of the page. Drop the real file at that path and the README needs no edit. Also, from watching the viewer render the `route.decision` events added an hour ago: the row read `claude-haiku-4-5 claude-haiku-4-5 is served by…`. The viewer renders `model` as the label and `reason` as the detail, so a reason opening with the model name spends the row saying the same thing twice. Reworded, and pinned so it cannot drift back. Verified end to end against the *globally installed* binary rather than dist/: clean clone, npm ci, npm run build, npm install -g ./packages/cli, then doctor (10/10 ok, both shims), record, list, show, checkpoints, replay (matched 2/2), fork, compare (two models, real costs), export, scrub --dry-run and gc --dry-run — ten steps, all exit 0. Seven error paths return exit 1 with an actionable next step. The exported HTML has zero external references, zero console errors, and renders 13 rows including the new ROUTE ones. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
…need Following the vitest 2 -> 4 bump: vitest pulls a vite that requires `^20.19.0 || >=22.12.0`, and the monorepo root still declared `>=20.0.0`. A contributor on node 20.0-20.18 would have had npm stay silent and the mismatch surface later as something stranger than a version error. The root now declares the range its toolchain actually needs. `packages/cli` deliberately does not move: the published CLI really does run on node 20.0, and tightening it would turn a dev-tooling constraint into a user-facing one. Nobody running `orca` is affected, and CONTRIBUTING says why the two differ. Found by checking the CI workflow against the bump rather than waiting for CI — which cannot tell me, having executed zero steps on ten consecutive head SHAs. The same pass confirmed `npx vitest run --reporter=verbose` still works on vitest 4 and that its engines cover both legs of the node 20/22 matrix. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
Answering "where does it save the data" turned up that `.orca/` lands in `git status` as an untracked directory, one `git add -A` away from being committed and pushed — in the working tree of the project it just recorded. A trace is the conversation the model saw, which is your source, plus shell output, a snapshot of the whole workspace and an environment allowlist. SECURITY.md already says to treat one as roughly as sensitive as a shell history plus a heap dump; nothing stopped it being pushed to a public repo. `.orca/.gitignore` containing `*` is git's own idiom for a directory that excludes itself: no edit to the user's own `.gitignore`, nothing to remember, and it works in a repo orca has never seen before. Written once at creation and never rewritten, so anyone who deliberately wants their traces tracked can empty it and orca will not put it back — there is a test for that, because a tool that undoes a user's decision on the next run is worse than one that never made it. The three places that created the runs directory by hand now go through one `ensureRunsDir`, so record, exact replay and fork cannot drift on this. Verified against the global binary in a fresh repo: `git status --porcelain` is empty after a recording. Also documents where runs are kept, since that was the actual question: the per-project layout, what each file in a run directory is, and the four commands for finding an old session. `orca list` reads the directories directly, so a trace someone sends you works by dropping it in — nothing indexes it, and there is no database to corrupt. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
Someone landing on this repo had to read four paragraphs to learn what the thing does. Two diagrams now carry it, both Mermaid so GitHub renders them inline and they stay editable rather than becoming a binary nobody can update: - **Capture.** The agent on the left, orca's four layers boxed so the boundary is visible, one trace on the right. The earlier draft put the mechanism on the agent's own edge, which read as orca writing the working tree; the agent does the verb, orca's layer is the node. - **The cursor.** Turns 1-4 from disk, turns 5+ from the network, the cursor between them — which is the whole of exact/fork/compare in one picture, with a table underneath saying where each command puts it. Themed `neutral` rather than Mermaid's default lavender: the project's own viewer is monochrome, and a purple diagram in a monochrome README reads as imported. Both render clean in GitHub's light and dark backgrounds — checked by rendering them in a real browser on both, not assumed. The gateway node names OrcaRouter, since that is what `orca setup` points at. Also adds a walkthrough of an actual bug hunt: show, replay, compare. Every line of it is **copied from a real run**, not composed — the README two screens above says "nothing here is mocked up", and a plausible-looking transcript underneath that would have made the claim false. The first draft of this section was composed, so it was thrown away and the scenario re-run for real. It reads better anyway: the recorded run's exit code is 0 while seq 12 shows the check exiting 1, which is a sharper illustration than anything invented. One self-inflicted mess worth recording: a slice-based string replace duplicated a diagram 26 times and grew the file to 1521 lines. Caught by validating the rendered output rather than trusting the edit. Reverted and redone with exact-match replaces asserted to hit exactly once. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
…r nothing Two of the three things blocking a launch were fixable here. Both are now done and verified rather than assumed. **Every internal dependency was `"*"`.** That does not fail loudly, which is the problem: `*` resolves to whatever is latest on the registry, so a 0.1.0 CLI would silently install a 0.9.0 core — and on a first publish it resolves to nothing at all, because nothing exists yet. All eight packages now pin their siblings to an exact version. Pinning costs ordering: the CLI cannot be published before the core it names. `scripts/publish-order.mjs` topologically sorts the workspaces so that order is computed from the manifests rather than written down somewhere that goes stale, and it exits 1 on a cycle, on a version disagreement, or on any internal dependency that is not an exact version. The release workflow gates on it, and four tests cover it — a release only runs on a tag, which is the worst moment to discover the order was wrong. Proved the publish would work rather than trusting `npm pack`: packed all eleven packages, installed the CLI from tarballs into a clean project, and ran it. Both shims report ok out of the packaged `dist/` — the exact case that was silently broken for the whole of v0's development — and record, replay (matched 2/2), fork and export all work from that install. `RELEASING.md` keeps that verification as the last step, because "npm accepted it" is not the same as "a fresh install runs". **The `review` check was red on every PR for want of a secret.** A check that can only fail teaches people to ignore red checks, which is the opposite of what a review gate is for. The job now skips when `ORCAROUTER_API_KEY` is absent, which also stops it failing on fork PRs, where secrets are unavailable by design. Also adds the OrcaRouter links — all models, OrcaCode Review, X, Hugging Face — at the top and in a footer, and corrects the install section: the packages are built and verified for npm but nothing is published, so it says that instead of promising npx today. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
Recorded Claude Code doing an actual task for the first time — a failing money-splitting test in a scratch repo — and the replay of it matched 1 of 13 requests before halting. Two bugs, both invisible to a suite that only ever replays fixtures: The distance metric took the longest common prefix and suffix of the two serialized bodies. Claude Code carries a session-scoped id in its system prompt *and* another in a tool description; with two edits that far apart, every one of the 200 KB between them counted as changed. A sixteen-character drift scored 217,568, so rung 2 could not fire and every request fell to rung 3 or 4. Distance is now summed over the leaves that actually differ, which bounds each difference to the field containing it. Redaction made rung 1 unreachable and nobody noticed, because no fixture has a secret in it. The recorded side holds `<secret:kind:hash8>`; the incoming side holds the value itself, and the digest is salted per run so it could not be reproduced even if it were. The matcher now redacts the incoming request with the same policy and compares the kind rather than the digest — and reports that fold as a minor divergence, since it is an approximation. Two smaller things the same recording found: the ask may now drift by 2% of its own size, capped at 512 characters, because a bundled skill quoted its own 32-character content-addressed cache path in a tool result and that alone halted the run; and `replay.done` said `matched=1 total=13` for a replay that reused all thirteen, which reads as a failure. It now says `reused=13/13 exact=1`. Spec §4 gains both rules, since a reimplementation would hit the same wall. That run now replays offline end to end: reused=13/13, unmatched=0, exit=0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
Japanese, Simplified Chinese, Korean, German, French, Spanish and Arabic, in docs/i18n/, with a language row at the top of each one and of the English README. Full translations, not summaries: every section, table row and status line is there, and a check asserts each file has the same heading, table, code fence, image and diagram counts as the English original. Terminal output stays verbatim, because it is output — what an English reader sees is what a Japanese reader will see on their own machine, and translating it would be inventing a program that does not exist. Prose inside command comments is translated, since that is writing rather than output. Diagram labels are translated in place; a check confirms every mermaid block is structurally identical to the English one that is known to render, differing only inside its quoted labels. Headings differ per language, so the badge links that pointed at `#install` would have died in translation. Each file carries an explicit `<a id="install">` instead. All 200 relative links across the eight files are verified to resolve — the depth changes from the repository root to docs/i18n/, so every path had to. Arabic is wrapped in a single `dir="rtl"` block, which GFM renders as markdown because of the blank lines around it, and the logo floats right rather than left. Also refreshes the `replay.done` line quoted in the walkthrough, which this session's matcher work reshaped, and formats two files prettier was unhappy with. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
Forking a real Claude Code recording died on the first live request: messages.1.content.2.tool_use.id: String should match pattern '^[a-zA-Z0-9_-]+$' A `tool_use` id is 25 characters of mixed-case base62, so the entropy sweep took it for a secret. Replay hands recorded bytes straight back to the agent, the agent echoes the id on its next turn, and the API refuses it. `orca compare` — the feature this is pitched on — could not have worked against any real recording; no fixture has an id in it to notice. Protocol identifiers (`id`, `tool_use_id`, `tool_call_id`) are now exempt from the entropy *guess*, not from the pattern rules: a credential parked under a key called `id` is still redacted by shape, and a test holds that line. The first version of this shipped broken and passed its own test. A response body is stored as a string inside the event's JSON, so the scanner sees \"id\":\"toolu_…\", and a pattern written with bare quotes matched nothing where it mattered. The tests now use the stored form. Then the fork got one turn further and hit the same class of bug again: messages.1.content.0: Invalid `signature` in `thinking` block A thinking signature is base64, must round-trip verbatim, and authenticates text the trace already holds in full — so keeping it costs no secrecy that was not already spent. Two matcher changes from the same recording. Leaf distance was measured by common prefix and suffix, which counted the 1,289 characters lying between the first and last of four `duration_ms` floats; it is now measured per line, and counts the four. And a request whose only difference is inside tool output is now served from the recording as a `major` divergence instead of halting the replay: orca does not intercept tool execution, so a replayed agent really re-runs `npm test`, which really reprints its own timings. A changed question, or a check that went from failing to passing, still refuses. That run now replays offline end to end: reused=7/7, unmatched=0, exit=0, and a fork of it reaches the same tree the recording did. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
The project had never been pointed at a real agent, and the first Claude Code run recorded through it broke four things at once — a distance metric that scored a sixteen-character drift as 217,568, redaction that made an exact match structurally unreachable, redaction that broke every fork by eating tool ids and thinking signatures, and the plain fact that a replayed agent re-runs its own tools and they reprint their own timings. All four are fixed and all four are now written down, because "we tested it against a real agent and here is what it taught us" is the part a reader actually wants, and the last bullet is a permanent property of not patching the harness rather than a bug that went away. The numbers quoted are that run's: reused=7/7 exact=2 divergences=5 unmatched=0 exit=0 for the replay, and a fork of it reaching the same tree the recording did after solving the bug on its own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
yi-here
pushed a commit
that referenced
this pull request
Aug 29, 2026
PR #2 was merged into orca/format-and-project rather than into main, so main carried the trace format and the project scaffolding while the working system — the CLI, the proxy, the viewer, the adapters, the Python SDK and the translated READMEs — sat one branch away. Nothing was lost: main's tree was identical to the merge base, so it had contributed a merge commit and no content of its own. This brings the two together. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C2yYKwaBo9qTzYVVHyCcrY
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
The working system behind the three commands. Stacked on #1 — review that first.
That animation is one real session, captured with
NO_COLOR=1;docs/media/transcript.txtis the raw output, so every line in it can be traced back to a command that actually produced it.Why
Model APIs are stateless, so an agent resends the whole conversation every turn — including the previous turn's tool results. A proxy in front of the model therefore sees the entire loop, which is why nothing here patches the agent: it gets a couple of environment variables and is otherwise untouched.
Exact, fork and compare are not three subsystems. They are one proxy with a cursor: the position in the recorded stream where it stops serving from disk and starts serving from the network.
Packages:
core(append-only writer, reader, content-addressed blobs, checkpoint derivation, redactor) ·fs-capture(shadow git index) ·providers(Anthropic + OpenAI translation both ways, pricing) ·proxy(matching ladder, three cursors, opt-in TLS interception) ·adapters(claude-code, codex, opencode, generic) ·shell-shim·mcp-shim·viewer(one self-contained HTML file) ·cli· a read-only Python SDK.Tests
1067 TypeScript tests across 60 files, plus 180 Python. Written test-first throughout, including an end-to-end suite that drives a real child process, a real proxy and a real trace on disk.
Verified against the globally installed binary rather than
dist/: clean clone →npm ci→npm run build→npm install -g ./packages/cli, thendoctor(10/10, both shims),record,list,show,checkpoints,replay,fork,compare,export,scrub --dry-run,gc --dry-run— ten steps, all exit 0. Seven error paths return exit 1 with an actionable next step. The exported HTML has zero external references and zero console errors.Bugs the tests caught before they shipped
.gitignorewith!.envre-included secrets. Git'sinfo/excludeis lower precedence than the workspace's own ignore file, so the specified exclusion mechanism was defeatable. Now enforced with pathspecs no ignore file can override — andGIT_LITERAL_PATHSPECSin the ambient environment silently disarmed those too, so the git environment is scrubbed.getUserAuthenticationTokenFromRequestHeadersscores 4.08 bits/char. Since agent traces are mostly source code, the rule as specified corrupted exactly the payloads the trace exists to preserve. A mixed-alphabet guard fixes it with essentially no loss of recall; the spec is updated to match.prompt_tokensincludes cached tokens and Anthropic'sinput_tokensdoes not, so every cached OpenAI turn was double-billed.Bugs only running it as a user could find
The suite was green at 851 tests while several of these were live. Each was found by using the built binary against a real agent, or by opening the viewer and looking at it.
JSON.stringify(payload), and every wire body is a string — so a spilled body was stored as a quoted, escaped JSON string literal while the same body under the limit was stored as itself. Invisible in tests because every fixture sat under 4 KB; fatal in practice because a real harness spills on turn one.--model gpt-5.2on an Anthropic-recorded run sent an Anthropic body to api.anthropic.com.min-height: auto, so every event past the fold was unreachable.causeswas empty on every tool result ever recorded. A flat event list where the format promises a chain.ok. The recorder writesis_error; the viewer checkederrorandok, neither of which it sets.orca gcwould have deleted a user's project directory. The worktree guard keyed onparent_runplus a$TMPDIR/orca-*path shape; replay traces then acquiredparent_runwith the user's own cwd. Caught by the feature that introduced the hazard, and re-keyed onfork_point.Two invariants have tests whose failure messages state them outright: replay makes no network call at all, and anything below rung 1 must carry a divergence.
Since the first review round
orca setup/orca models. Stores a gateway, key and default model list at~/.config/orca/config.json(0600 in a 0700 directory), and asks the gateway what it actually serves so a wrong URL is an answer now rather than a 401 mid-comparison.parent_runand no inventedfork_point, holding divergences and unmatched requests.lastskips replay traces, so the commands that default to it still mean the recording.0600, expires within a day, is trusted only through the child's own environment, and is deleted at exit including on a crash. A mutation-verified test proves the key cannot reach a recording.Since the second review round
Every item found by running the built CLI as a user would, or by writing the test that had been missing. All mutation-verified.
orca doctorsaid it did.resolveShimEntry()asked for a subpath the package'sexportsmap blocks, so the fallback ran every time and resolved to the library. Every stdio MCP server was rewritten to launch a module that exports and exits —--mcp-configsilently broke the agent's MCP servers as well as recording nothing. Behind it, the shim writes{name, dir}and the recorder read{server, direction}.--tls-interceptwas accepted by every command and honoured by one.replay --model,forkandcompareall launch a live agent and all three threw the flag away.orca compare laststraight after a fork failed with "this run has no checkpoints". Branching had a depth limit of one.orca scrubcould destroy the evidence it was cleaning (plainwriteFiletruncates first), and had no--dry-run— the one command whose effect cannot be undone was the only one you could not look at first.ok: falsefor a killed recorder and an edited file.$TMPDIRper invocation, in a directorygcdeliberately will not sweep.api-key(Azure) andx-goog-api-key(Google) were known to the TLS interceptor alone.expect(mono_us).toBeGreaterThanOrEqual(0).Since the third review round
--mcp-configexisted. The recording knew which config to use and had nowhere to write it down (manifest.argvholds the agent's arguments;--mcp-configis orca's), so record now leaves anmcp_instrumentednote and replay reads it back.route.decisionwas declared and emitted by nothing. Spec §2 defines it as "a gateway chose a model", and on a fork orca is the gateway — it substitutes the model, picks the wire format and picks the origin, and was doing all three silently.Providerlayer is deleted — ~400 lines of production code and 570 of tests, all seventeen exports checked individually for consumers outside the package (zero). The interface stays published in@orcareplay/plugin-api.checkpointevent, which spec §3 forbids. Removed, seq renumbered dense withcausesremapped, goldens recomputed from the Python and TypeScript implementations independently and cross-checked.--gatewayoverrides. A default, not a redirect: with nothing configured,orca recordstill proxies the agent's own calls to whatever provider it was already talking to.npm ciprinted 5 vulnerabilities, 1 critical — all dev toolchain. vitest 2.1 → 4.1 and esbuild 0.24 → 0.28: 0 vulnerabilities, suite unchanged.Since the fourth review round — pointed at a real agent for the first time
An earlier version of this description said the system had been "verified repeatedly against the real Claude Code binary". That was true of
recordand of the CLI surface, and not true ofreplayorfork: what had been exercised end to end was a fake agent against fixtures. Recording an actual Claude Code session fixing an actual bug — a money-splitting function whose parts did not sum to the total — and then replaying it broke four things at once. Each is a class of bug no fixture can contain, and each is fixed and mutation-verified here.duration_msfloats scattered through fifty lines of test output scored 1,289 instead of the real ~40.<secret:kind:hash8>; the incoming side holds the value, and the digest is salted per run by design so it could not be reproduced even in principle. The matcher now redacts the incoming request with the same policy and compares the kind of secret rather than the digest — reported as aminordivergence, since folding is an approximation. Spec §4 gains both rules, because a reimplementation hits the same wall.tool_useid is 25 characters of mixed-case base62, so the entropy sweep took it; replay hands recorded bytes back to the agent, the agent echoes the id on its next turn, and the API answers400 messages.1.content.2.tool_use.id: String should match pattern '^[a-zA-Z0-9_-]+$'. Anthropicthinkingblocks fail the same way on theirsignature. Protocol values that must round-trip are now exempt from the entropy guess — never from the pattern rules, so a credential parked under a key calledidis still redacted by shape, with a test holding that line. Worth flagging for review: the first version of this fix passed its own tests and was still broken, because a response body is stored as a string inside the event JSON and the scanner sees\"id\":\"toolu_…\"; the tests now use the stored form.npm testagain and it really reprints its own durations. A request whose differences are all inside tool output is now served from the recording as amajordivergence instead of halting the replay. Deliberately narrow: a changed question, or a check that went from failing to passing (is_erroris not blanked), still refuses at rung 4.replay.donesaidmatched=1 total=13for a replay that reused all thirteen, because rung 1 is unreachable whenever anything was redacted. It now reportsreused=7/7 exact=2.That recording now replays offline end to end —
reused=7/7 exact=2 divergences=5 unmatched=0 exit=0, every approximation named in the trace — and a fork of it from a checkpoint reaches the same tree the recording did, having solved the bug on its own. The README says all of this, including the last bullet, which is a permanent property rather than a bug that went away.Also since the fourth round
docs/i18n/, with a language row on each. Full translations, not summaries: a check asserts each file has the same heading, table, code-fence, image and diagram counts as the English original, that every Mermaid block is structurally identical to the English one known to render, and that all 200 relative links across the eight files resolve. Terminal output stays verbatim because it is output. Arabic is wrapped in a singledir="rtl"block.scripts/publish-order.mjscomputes the topological publish order and fails on a cycle, a non-exact internal dependency, or a version disagreement;.github/workflows/release.ymlis tag-gated and gates on it. All eleven packages were packed and installed from tarballs into a clean project to prove it.Known gaps
docs/orcarouter.svgis a placeholder wordmark, not the real brand asset. The sandbox this was built in blocksorcarouter.ai(403 at the CONNECT tunnel), so the logo could not be fetched. Drop the real file at that path; the README needs no edit.api.orcarouter.aiwas never probed live, for the same reason.orca setupdegrades correctly (config saved, clear warning) if the probe path is wrong.checkpointis the one declared event type nothing emits, deliberately: spec §3 derives it at read time.CI
Every check on this PR is red for an environmental reason, not a code one. Across ~50 runs on five branches — including #1, whose diff is Markdown and JSON Schema, and #4, which is someone else's — every job fails 2–5 seconds after queueing with
created_at == started_at, zero steps executed and 404 logs. It has now reproduced identically on fifteen consecutive head SHAs. Nothing runs. It needs Settings → Actions → General and org Actions billing.One check count did change on purpose: the OrcaCode Review job now skips instead of failing when
ORCAROUTER_API_KEYis absent, taking the red checks from six to five.Because CI has never executed, everything above was validated locally:
fmt:check,tsc --build,vitest run(1067 tests across 60 files),scripts/conformance.mjs,scripts/check-neutrality.mjs,scripts/publish-order.mjs,pytest python/(180), andnpm audit(0 vulnerabilities).Checklist
npm run checkpasses locally