Tool Replay

Tool Replay captures, replays, forks, diffs, and promotes synchronous and sequential asynchronous AI effects into deterministic offline regression tests.

Supports explicit instrumentation of OpenAI and AsyncOpenAI Responses API calls with zero provider transport during replay, fork, verification, and generated regression tests.

Current capabilities

• M5: explicit, non-mutating instrumentation of synchronous OpenAI.responses.create calls through instrument(...).
• M5.1: lossless preservation of JSON-portable top-level OpenAI Response fields, while interpreted output and message-content unions remain closed and fail explicitly for unsupported discriminators.
• M6A: native sequential async def effects with strict async replay, fork, promotion, and case verification.
• M6B: explicit instrumentation of sequential, non-streaming AsyncOpenAI.responses.create calls through the same shared OpenAI codec.

Current limitations

OpenAI support is Responses API only and requires openai==2.20.0. Async effects must be sequential: overlapping or nested instrumented async effects are not supported, and there is no concurrency or asyncio.gather(...) guarantee. There is no streaming, tools or function calls, file or binary input, Chat Completions, Azure OpenAI, another provider adapter, or retry tracing.

OpenAI setup and offline examples

Install the pinned optional OpenAI SDK:

uv sync --extra openai

Both maintained examples are offline by default and require no API key:

uv run python examples/openai_responses_approval.py --offline-demo
uv run python examples/openai_async_responses_approval.py

The architectural contracts are in rfcs/. Read all four RFCs before changing runtime, matching, or storage behavior.

Development

Python 3.11 or newer and uv are required.

uv sync
uv run ruff format --check .
uv run ruff check .
uv run pyright
uv run pytest

OpenAI Build Week 2026

Tool Replay was created during OpenAI Build Week 2026 for the Developer Tools track.

The project's first commit was created on July 15, 2026, within the official submission period.

GPT-5.6 Sol was used for product planning, architectural research, milestone design, implementation prompts, and technical supervision. GPT-5.6 Sol and Terra were used for milestone-scoped implementation, testing, and review.

Tool Replay is an independent open-source project and is not affiliated with or endorsed by OpenAI.

License

Tool Replay is available under the MIT License.

Failure replay (M1)

Recorded TimeoutError failures are persisted and reproduced during strict replay without executing the live effect body. M1 is synchronous and supports portable JSON values; allowlisted built-in exceptions are reconstructed, custom exceptions become ReplayedEffectError, and tracebacks are not captured.

Strict fork with output patching (M2)

• record: run live effect bodies and persist their inputs/outputs.
• replay: substitute recorded outputs/failures for the exact recorded identity; never run live effect bodies.
• fork: like replay, but against a new branch trace, with selected recorded outputs replaced by patched values before your application code runs; still never runs live effect bodies.

import toolreplay

patch = toolreplay.replace_output(
    effect="crm.customer.get",
    inputs={"customer_id": 42},
    occurrence=0,
    value={"id": 42, "active": False},
)

with toolreplay.fork("source.trp", "branch.trp", patches=[patch]):
    decision = decide(42)  # observes the patched customer, not the recorded one

Every patch is resolved against the source trace's exact recorded (namespace, name, input_fingerprint, occurrence) identity before any application code runs. After the with block exits normally, Tool Replay verifies every configured patch was applied and every source operation was observed; an application exception always propagates unmodified instead of being masked by that check. See examples/fork_replace_output.py.

M2 limitations: synchronous only, strict tail only (tail="strict"), output replacement only (no replace_input/replace_error/drop/force_live), exact effect identity is required for both replay and patch targeting, only portable JSON values are supported, and there is no live tail, no structural diff/first-divergence analysis, and no provider adapters.

Trace diff (M3)

record:
  capture what happened

replay:
  reproduce what happened

fork:
  replace selected recorded outputs

diff:
  show the first observable difference between traces

toolreplay.diff(source_trace, target_trace) is a read-only structural comparison that answers one narrow question: what is the first observable behavioral difference between two recorded executions? It does not claim root cause, causal proof, internal program-state divergence, semantic equivalence, or intent.

import toolreplay

report = toolreplay.diff("source.trp", "branch.trp")
if report.first_divergence is not None:
    print(report.first_divergence)
print(report.to_dict())  # JSON-compatible, deterministic

Operations are aligned in two conservative phases: exact alignment on the full (namespace, name, input_fingerprint, occurrence) identity, then, for whatever remains unmatched, structural alignment on (namespace, name, occurrence) -- but only when exactly one candidate exists on each side and the pairing does not violate ordering established by exact-alignment anchors. Ambiguous or otherwise unsafe pairings are left unmatched rather than guessed; a trace containing duplicate complete identities fails explicitly instead of aligning arbitrarily. Detected difference types include input_changed, output_changed, status_changed, error_changed, operation_missing, operation_added, and order_changed. When a target operation's persisted v3 branch lineage (origin="patched", patch_id, source_logical_operation_id) proves it was produced by a replace_output patch, the report exposes a narrow attribution -- never a claim that the patch caused downstream differences.

Command line:

uv run toolreplay diff source.trp branch.trp
uv run toolreplay diff source.trp branch.trp --format json

Text output is deterministic, unambiguous, and suitable for snapshots or CI logs; --format json prints only JSON (with format_version) to stdout. Exit code is 0 whether or not differences were found, and non-zero only for invalid arguments or a comparison failure (there is no --fail-on-difference in this milestone). See examples/diff_traces.py.

M3 limitations: synchronous, local trace model only; only captured operations are compared (no pure internal-state inspection); no root-cause analysis or causal propagation; structural alignment is conservative and ambiguous operations are left unresolved rather than guessed; no semantic or embedding-based text comparison; no provider-specific interpretation; and no web viewer.

Promote a trace or fork into a regression test (M4)

record → replay → fork → diff → promote

toolreplay.promote(trace, output, *, name, runner, expected) turns an independently-replayable fixed trace (typically a branch produced by fork()) into a portable, self-contained pytest case:

import toolreplay

case = toolreplay.promote(
    "branch.trp",
    "cases/crm_customer_inactive_rejects",
    name="crm_customer_inactive_rejects",
    runner="myapp.cases:crm_customer_inactive_rejects",  # module:callable
    expected={"decision": "rejected"},
)
print(case.directory, case.trace_sha256)

The runner is a synchronous, no-argument, importable callable (a plain module-level function, or module.path:Class.method) that exercises your application code; by the time it runs, promote()/verify_case() has already opened a strict-replay session against the case's trace, so any @effect-decorated function the runner calls is transparently substituted -- the runner itself never opens a session.

promote() stages the case in a sibling temporary directory, verifies it end to end (integrity, runner resolution, strict replay, completeness, and the behavioral expectation) via verify_case(), and only then publishes it atomically to output; any failure leaves no partial output directory behind. The destination must not already exist -- there is no overwrite/force/merge.

A promoted case directory contains exactly three assets:

• case.json -- a deterministic, canonical-JSON manifest (format, format_version=1, name, trace (content-addressed path, digest, size_bytes), runner, replay.mode="strict", expectation, and scope). No timestamps, random ids, machine paths, source/branch trace paths, patch data, diff reports, git hashes, or environment dumps.

• trace-sha256-<64 lowercase hex>.trp -- a real byte copy (never a link) of the trace, named after its own SHA-256 digest.

• test_case.py -- a thin, fully static generated pytest module:

  from pathlib import Path
  
  import toolreplay
  
  
  def test_toolreplay_case() -> None:
      toolreplay.verify_case(Path(__file__).resolve().parent)

toolreplay.verify_case(case_directory) re-runs a promoted case and is what the generated test calls. It validates the manifest and trace integrity -- including the trace's SHA-256 digest and byte size -- before it ever imports or executes the runner's module; only after integrity passes does it resolve and import the runner, strictly replay it (running zero live effect bodies), confirm every recorded operation was consumed, and finally compare the runner's return value against the case's expected value. It opens the trace read-only and never writes or modifies any case asset.

verify_case() raises one of four case-specific errors, each meaning something different:

• CaseIntegrityError -- the manifest or trace asset itself is invalid (missing/malformed/wrong-version manifest, bad digest, an absolute/escaping/symlinked trace path, a digest or size mismatch, or a malformed/missing/async/argument-requiring runner target). Always raised before any application code is imported or executed.
• ReplayContractError -- strict replay of the runner failed: a recorded effect could not be found or was ambiguous, or the case trace contained a recorded operation the runner never consumed. Wraps the original ReplayMiss/AmbiguousReplayMatch as __cause__ when there is one.
• BehavioralRegressionError -- strict replay completed successfully, every required operation was consumed, and the runner's effect bodies ran zero times, but the runner's return value differs from the case's expected value. Its very existence proves replay already succeeded. Carries expected, actual, and deterministic JSON-path differences.
• ApplicationExecutionError -- the runner raised an exception that is not itself a replay-contract failure, or returned a value that is not portable JSON. Preserves the original exception as __cause__. KeyboardInterrupt/SystemExit are never wrapped.

Command line:

uv run toolreplay promote branch.trp \
  --output cases/crm_customer_inactive_rejects \
  --name crm_customer_inactive_rejects \
  --runner myapp.cases:crm_customer_inactive_rejects \
  --expect-json '{"decision": "rejected"}'
# or: --expect-file expected.json

On success it prints the case directory, name, trace path, trace SHA-256, generated pytest path, and a confirmation line; exit code 0. It exits non-zero for invalid arguments, invalid/missing expectation JSON, an existing destination, or any promotion failure, with the error on stderr and no partial case directory left behind. See examples/promote_regression_case.py.

M4 limitations: the runner must be a synchronous, no-argument, importable callable (no async, no required arguments); the only supported expectation is exact canonical-JSON equality of the return value (return_equals) -- no expected-exception assertions, matchers, or DSL; a case only covers the instrumented effects captured in its trace -- it makes no claim about uninstrumented side effects, external system state, non-deterministic control flow, databases, subprocesses, or concurrency/timing; there is no sandboxing, code signing, packaging, pytest plugin/collector, or recursive toolreplay verify <dir> discovery command.

Synchronous OpenAI Responses adapter (M5/M5.1)

M5 supports exactly openai==2.20.0: synchronous, non-streaming responses.create only. The core package does not import OpenAI; importing or using the adapter without this optional dependency raises OpenAIAdapterDependencyError.

from openai import OpenAI
from toolreplay.adapters.openai import instrument

client = instrument(OpenAI())
response = client.responses.create(
    model="...",
    input="...",
    reasoning={"effort": "minimal"},
    max_output_tokens=512,
)

instrument() is an explicit, non-mutating wrapper. Use client.original_client as the escape hatch to the unwrapped client. stream may be omitted or explicitly False. The four supported semantic request fields are model, a UTF-8 string input, reasoning={"effort": "minimal"|"low"|"medium"|"high"}, and a positive max_output_tokens.

Successful calls and supported provider errors are recorded as portable envelopes. Strict replay, strict fork, promoted-case verification, and generated regression tests reconstruct the exact public Response from those envelopes with zero provider transport. M5.1 preserves every JSON-portable top-level field from the SDK's public serialization without silently dropping data. The interpreted semantic unions remain closed: output supports only reasoning and message items, and message content is limited to output_text and refusal; unsupported discriminators fail explicitly. The application input intentionally sent to the model is recorded; credentials, authorization and custom headers, transport internals, and raw HTTP objects are excluded. The prompt_cache_retention: "in_memory" -> "in-memory" compatibility shim is specific to SDK 2.20.0.

The maintained synchronous command is listed in OpenAI setup and offline examples above; its --offline-demo workflow requires no API key.

It demonstrates recording approve, strict replay with zero provider calls, a strict fork that replaces the complete portable response envelope with manual_review, and a first observable divergence of openai.responses.create. Promotion is covered by deterministic tests rather than the default demo. Its API is:

toolreplay.promote(
    branch,
    "tests/cases/openai-approval",
    name="openai-approval",
    runner="your_module:run_case",
    expected="manual_review",
)

To make one live supported call, configure credentials locally and set a model; credentials are never printed. This credential-gated path is not part of default CI and does not claim deterministic provider wording:

$env:OPENAI_MODEL="..."
uv run python examples/openai_responses_approval.py --live-record

The shared current OpenAI and async limitations are listed prominently above. M5's invocation shell is synchronous; M6B adds the asynchronous shell without duplicating the request, response, error, redaction, or compatibility codec.

Local incident replay demo

The maintained visual demo presents an intentionally buggy refund decision: a captured approve model result produces approved, even though refunds above $1,000 require manual review. Its default workflow is fully offline and uses the actual Tool Replay record, strict replay, fork, diff, promotion, and case verification APIs--not JavaScript simulations or a provider call.

After installing the OpenAI extra as shown above, run:

uv run python -m examples.incident_replay_demo.server

Open http://127.0.0.1:8765. Select Replay incident, Apply correction, Compare executions, then Promote to test. Each server run owns a temporary local scenario; use Reset demo (or restart the server) to reset it. Replay, fork, branch replay, and case verification use an exploding local transport and display zero provider calls. The fork is a controlled alternate execution that verifies the desired behavior before creating an offline pytest case; its First observable divergence is not a claim of automatic root cause discovery.

This presentation demo has the same M5 limitations: it is synchronous, non-streaming Responses-only, supports no tools/function calls, Azure, files, other providers, or partial patches. It requires no API key and never exposes credentials, trace paths, HTTP objects, or transport configuration.

AsyncOpenAI Responses adapter (M6B)

instrument(AsyncOpenAI(...)) explicitly wraps only non-streaming responses.create using the same closed request and portable response/error contract as the synchronous adapter. It accepts the exact request surface described above, including stream omitted or explicitly False. Strict async replay and fork do not create or await a provider coroutine and make zero provider transports. Use promote_async() and verify_case_async() for self-contained offline async cases; their generated synchronous pytest owns one asyncio.run() loop.

The maintained asynchronous command is listed in OpenAI setup and offline examples above; its default workflow requires no API key. The optional credential-gated live capture uses AsyncOpenAI with max_retries=0 and a counting httpx.AsyncClient; it is never run by default and does not print credentials. The shared current limitations are listed prominently above.