Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion COMPATIBILITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@ Names are C# versions of the JavaScript API: `agent.act` is `ActAsync`, `screen.
- An entry that a verified replay finished, or that already holds the same flow, is not rewritten, except to write in the step fields an older entry lacks.
- As upstream, the screen is read by one settle policy per action, for live acts and replays. After a tap, a key press, a select, a check, a navigation, or a back, the next settled look waits up to 2 s for the screen to leave the shape the action was resolved against, then reads until two looks 75 ms apart agree, for at most 1 s. After a fill it waits for the change only; after a scroll, up to 500 ms; after a secret fill, for no change. Node references, focus, and clock-like values do not count as a change, and after an action an empty document does not count as a settled screen. Upstream also counts a screenshot as the shape; the port counts the scroll position instead, so a scroll that moves no node still ends the change wait. An act's start, the screen after each action, each page of a `scroll_to` with a text, and the recording's end screen are held still; a replay settles the look before each action as the previous action's policy asks, takes a held-still look between the repeats of a scroll, and reads the end state raw.
- A replay waits up to `ReplayTimeout` (15 s; a .NET-only fixture property, not a config key) for each recorded target and for the recorded end state. Upstream waits 15 s for the end route and sizes the anchor wait from the recording.
- `FileStepCache.Write` is upstream's `FileCacheStore.write` (`cache/store.ts`): it writes the entry through `AtomicWrite.WriteFileAtomic`, a sibling temporary file moved over the entry file, so a crashed or concurrent writer leaves no torn entry, and two writers on one key leave the last write, as upstream's store documents. Upstream's store awaits `writeFileAtomic`; `IStepCache.Write` is synchronous, so the port uses the synchronous twin (upstream's `writeFileAtomicSync`, which its CLI and telemetry stores use). Windows-only difference: a POSIX rename replaces the target in one step, but Windows refuses a move over a file that another move is replacing (`ERROR_ACCESS_DENIED`) or that a reader holds open (`ERROR_SHARING_VIOLATION`). Upstream's `rename` does not retry, so there the losing writer throws and the recording is lost. The port retries such a move up to ten times with a growing pause (about 90 ms in all) before the error is the writer's, so the write lands and the step reports `Saved` where it would have reported `NotWritten`. A denial that lasts, such as a read-only target, reports the same error after the same wait; any other error reports at once; on other systems the retry never runs. .NET-only names: `AtomicWrite.MoveAttempts`, `AccessDenied`, `SharingViolation`, and `RetriesMove` (the Windows errors and the retry), `Utf8` (the encoding both twins write with), and `TemporaryPath` (upstream's `temporaryPath`).
- `cache.mode` `read-only` replays but never writes or evicts, including the `end-mismatch` eviction. `off` neither replays nor records. Under `cache.strict`, a stale recording fails the act with `REPLAY_STALE` and stays in place; a strict run never writes or evicts, and its retries replay. A `no-entry` miss fails too when the cache directory holds a recording of the same step under another key. Entries record the step as the test, the engine, the instruction, the params digest, the repeat index, and the agent; the port stores the engine where upstream stores the target id, and the instruction text where upstream stores its digest. Only an entry that records the whole step counts, and, outside `cache.strict`, a verified whole replay in `read-write` mode writes the missing fields into an older entry. A custom `IStepCache` is not checked. The message names the runner or the agent's context, because the port key has no app identity.

## Visual comparison
Expand Down Expand Up @@ -553,7 +554,7 @@ Upstream writes `report.json`, a `report-1` document of the whole run (`report/b
- **When and where.** Upstream builds the document once, when its runner ends the run, and writes it to `<output>/report.json`. A test framework runs the port's attempts as sessions with no run around them, so each session adds its attempt to the process's run (`ReportFile.WriteAsync`, .NET-only) and the whole document is written again, atomically, to `report.json` beside the results directory (`.e2e/report.json` by default, as `ai-trace.json` is). The file is complete after every attempt. `run.finishedAt` and `run.status` are those of the attempts so far, not final, and `run.exitCode` is the worst exit code of the results, as upstream's `verdictExitCodes` derives it: a failed or timed-out result exits 1, or 2, 3, or 4 when the category of an attempt error is `configuration`, `infrastructure`, or `internal` (an agent code's category comes from `AgentError.CategoryByCode`, upstream's `CATEGORY_BY_CODE`, so a blocked `AUTH_CREDENTIAL_UNAVAILABLE` exits 2), and an interruption exits 130, which outweighs the rest. `run.status` follows the code (`passed`, `failed`, `interrupted`, else `error`), and `blocked` is derived from the results afterwards. A result keeps its wire form until it gains an attempt, so a large suite does not serialize every result again after each attempt. A result id is the test's file and title, so a test that runs twice in one process (another target, a repeat) is one result that its last run replaces, as `RunReport` already treats it. Test processes that share an output directory overwrite one another's file, and a file from an earlier run is replaced by the first attempt. It is written whatever the `trace` mode, for passing tests too. A write that fails (disk, permission) prints one line to stderr and does not fail the test.
- **A closed session's report** (`feat(runner): expose the closed hosted session report`, `b750125`). `E2ESession.Report()` is upstream's `session.report()`: once the session is disposed and its trace page, when the mode keeps one, is written, it returns the attempt's report-1 document as a single-result run (`ReportFile.SingleAsync`, .NET-only; upstream's `writeHostTrace` returns the `Report` it builds). It returns null before that and when the page cannot be written. The return type is the public `E2E.Reports.Report`, upstream's `Report` (an alias of `Report1Document`) with its nested types in C# casing and the names of `report/build.ts` and `run/steps.ts` (`ReportRun`, `ReportTarget`, `ReportResult`, `ReportAttempt`, `ReportStep`, `ReportError`, `ArtifactRecord`, `StepEvent`, `StepMetrics`, `StepModelInfo`, `StepCacheRecord`, `FailureEvidence`, `SkipInfo`, `VcsInfo`, and the rest). They live in the `E2E.Reports` namespace because the port's own internal records for the same parts keep upstream's names in `E2E` (`ReportRunStatus` and `ReportExplore` are shared and stay in `E2E`). The typed report is read from the document that `report.json` holds, so the two tell one attempt alike; `ReportSerializer` is .NET-only and the only place that reads it; the library never writes the typed form. Differences: the typed report has no `serialGroups` and no `carried` (the port has none), no `extensions` and no `stack`; enum members stand for upstream's string unions (`ResultStatus.TimedOut` for `timed-out`); durations stay `DurationMs` as upstream, but `ReportExplore` keeps its `TimeSpan` members; a session that keeps no trace (the mode keeps none for the attempt's verdict) still returns its report, where upstream's hosted session always writes a trace; a trace page that cannot be written is reported on stderr and the session releases without a failure, so upstream's `close()` failure for it has no port. `E2ESession.AttemptId` (internal) is upstream's `attemptId`. Tests: `OpenHostReport/OpenHostReportTests` has both `it` of `open-host-report.test.ts`; the first is partial and renamed `Returns_independent_closed_reports_that_remain_stable_across_a_second_session` (no `gadget.dropFile()` fixture, and the soft failure is `CloseSoftFailures`' result rather than an entry of `close()`'s failures), the second is partial and renamed `Returns_null_when_the_trace_write_fails_and_the_session_still_releases` (no `close()` failure check). .NET-only: `Tells_the_same_result_as_report_json_for_the_same_session` checks that `Report()` and `report.json` tell one attempt alike, which upstream's hosted session (no `report.json`) has no counterpart for. `TypedReport/TypedReportTests` (.NET-only) reads the valid report-1 fixture into a `Report` and writes it back, and checks that the document is the same bar the serial groups, `carried`, `extensions`, and `stack`. The document's `run.id` is the session's own, so it is not compared. `ReportFile.DetectVcs` (.NET-only) asks version control once per process and project root for both `report.json` and `Report()`; a failed lookup is asked again, and `Asks_for_the_commit_once_per_project_root_however_many_sessions_ask_and_asks_again_after_a_failure` covers both.
- `schema/report-v1.schema.json` is upstream's schema, unchanged. `schema/fixtures/report-v1.valid.json` and `report-v1.invalid.json` are upstream's fixtures. The E2E package packs `schema/*.schema.json` under `schema/` (not the fixtures), as upstream's `files` does. Upstream's other three schemas (`agent-judgment-v1`, `agent-judgment-v2`, `session-v1`) are not ported: their features are not.
- `WriteReport` (`Report/WriteReport.cs`, internal) is upstream's `report/write.ts`: `WriteTextReportAsync`, `WriteJsonReportAsync` (two-space indent, a trailing newline), and `ToPosixPath`. `AtomicWrite.WriteFileAtomicAsync` is upstream's `writeFileAtomic` (`internal/atomic-write.ts`): a sibling temporary file with random bytes in its name, moved over the target. The trace page and the failure screen file are written through `WriteTextReportAsync`, as upstream's `traces.ts` does (it replaces the port's own `TracePages.WriteTextAsync`). The writer serializes with the port's JSON defaults (camel case, null omitted), so a caller redacts the text and shapes the document. Difference: a string with a lone UTF-16 surrogate makes `WriteJsonReportAsync` throw, where `JSON.stringify` writes an escape; the builder that cuts text to the schema's limits must not split a surrogate pair. Not ported: the synchronous twin, the file `mode` option, and `fileReporter`, since the port has no reporters.
- `WriteReport` (`Report/WriteReport.cs`, internal) is upstream's `report/write.ts`: `WriteTextReportAsync`, `WriteJsonReportAsync` (two-space indent, a trailing newline), and `ToPosixPath`. `AtomicWrite.WriteFileAtomicAsync` is upstream's `writeFileAtomic` (`internal/atomic-write.ts`): a sibling temporary file with random bytes in its name, moved over the target; `AtomicWrite.WriteFileAtomic` is the synchronous twin (`writeFileAtomicSync`), which `FileStepCache.Write` uses because `IStepCache.Write` is synchronous. On Windows both retry a move the file system refused while another move was replacing the target or a reader held it open (see [Replay cache](#replay-cache)). The trace page and the failure screen file are written through `WriteTextReportAsync`, as upstream's `traces.ts` does (it replaces the port's own `TracePages.WriteTextAsync`). The writer serializes with the port's JSON defaults (camel case, null omitted), so a caller redacts the text and shapes the document. Difference: a string with a lone UTF-16 surrogate makes `WriteJsonReportAsync` throw, where `JSON.stringify` writes an escape; the builder that cuts text to the schema's limits must not split a surrogate pair. Not ported: the file `mode` option, and `fileReporter`, since the port has no reporters.
- The schema is checked in tests only. The tests use JsonSchema.Net 8.0.5 (MIT; version 9 and later use another license), in the test helper project `tests/E2E.Tests.Schema`. The shipped package validates nothing and has no new dependency. `ReportSchema.AssertValidReport` is upstream's `assertValidReport`, with `format` asserted as Ajv does.
- Run-level blocks (`Report/BuildReport.cs`, upstream's `report/build.ts`): `DescribeTarget`, `ProjectSource`, `ComputeSummary`, `DeriveRunStatus`, `ComputeUsage`, `SerializeTarget`, and `Build`, which assembles `Report1Document` with `ReportRun`'s new members (`Status`, `ExitCode`, `StartedAt`, `FinishedAt`, `Project`, `Environment`, `Vcs`, `Errors`, `Summary`, `Limits`, `Usage`). `ReportRun.Targets` are `ReportTarget`s now, not names. `ReportResult` gained `DeclarationIndex`, `Tags`, and `Selected`; `ReportArtifact` gained `Size`; `StepMetrics` gained `ObservationBytes` (the port does not measure it, so `usage.maxObservationBytes` and each agent step's `metrics.observationBytes` read 0). `AttemptReport.Build` fills each kept file's `Size`, so `usage.artifactBytes` adds them up. The engine version of a per-attempt target is `IEngine.Version`, which the session carries in `AttemptScope.EngineVersion`; a blank version reads `unversioned`. `Git.RunAsync` removes the variables that point git at a repository (`GIT_DIR`, `GIT_WORK_TREE`, and the like) before it starts git, so a hook or wrapper cannot move the repository it reads; upstream passes the parent's environment through. A target's `baseOrigin` is the scheme, host, and non-default port of the app URL, without credentials, and only for http and https, as `new URL(...).origin` is for a web URL. Differences: the port has no serial groups, no `--last-failed` rerun (`carried`), and no `failOnSkippedFailure`, so `DeriveRunStatus` reads results only. `DescribeTarget` takes an `IEngine`, which has no name, so the engine name is its platform; its flag capabilities are spelled in camel case (`tapModifiers`); `stateCapability` is false, since the port has no state hooks. A target's environment is always `test` (no `app.environment` key), the project is `unknown` with a zero digest (no project id or config digest, as upstream reports without a config), `limits` are the defaults (the config rejects `maxObservationBytes` and `maxInputTokens`), and `environment.runtime` reads `.NET <version>` where upstream's reads `node <version>`.
- .NET-only names in this part: `ReportEngine`, `ReportProject`, `ReportEnvironment` (upstream writes these as inline object shapes), `ReportRunStatus` (upstream's status string union), `BuildReportOptions.TargetProvenance` is upstream's own, `Git.RunAsync`'s `budget` parameter (for tests), and `DescribeTarget(platform, version, capabilities)` (the session knows its engine by those three, not as an `IEngine`).
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -280,7 +280,7 @@ From a clone of this repo, run the tool with `dotnet run --project src/E2E.Cli -

An agent entry also takes `judge` (the model id for `assert`, `waitFor`, and `extract`), `system`, `context`, `maxSteps`, `maxModelCalls`, `judgmentTimeout`, and `providerOptions`. `agents` can name more agents than `default`; a call picks one with its `Agent` option, such as `new ActOptions { Agent = "careful" }`. Project tools (`AgentTools` on the fixture) are code, so they are not config keys; see topic `agent`. So is a custom executor (`AgentExecutor` on the fixture, `AgentOptions.Executor`), which replaces the built-in agent for `act` and `assert` with a `ToolLoop` of your own tools or your own `IStepExecutor`; see topic `agent`.

`cache.mode` is `off`, `read-only`, or `read-write`. Unset, it is `read-write` locally and `read-only` when `CI` is set. `cache.strict` fails a recording that no longer matches with `REPLAY_STALE` instead of running the step live. Under `cache.strict`, retries replay too, and the run never writes the cache. `cache.dir` resolves against the config file's directory. `E2E_CACHE_DIR` replaces it, so checkouts and worktrees of one project can share a replay cache without editing the config; the mode stays as configured.
`cache.mode` is `off`, `read-only`, or `read-write`. Unset, it is `read-write` locally and `read-only` when `CI` is set. `cache.strict` fails a recording that no longer matches with `REPLAY_STALE` instead of running the step live. Under `cache.strict`, retries replay too, and the run never writes the cache. `cache.dir` resolves against the config file's directory. `E2E_CACHE_DIR` replaces it, so checkouts and worktrees of one project can share a replay cache without editing the config; the mode stays as configured. Parallel runs can share the directory: each write replaces its entry file atomically, and two writers on one key leave the last write. On Windows, a write the file system refuses while another write replaces the same entry, or while a reader holds it open, is retried for a moment before the step reports it as not written.

A cache entry belongs to one test, instruction, set of params, and agent. Each of these changes causes a miss: renaming the test or its class, changing the instruction or an ordinary param, running with another configured agent, changing the agent's `context` or a test's `[AgentContext]` (a secret in it enters the key as its name only), or switching to another engine. Changing the model doesn't cause a miss.

Expand Down
6 changes: 5 additions & 1 deletion skills/e2e-dotnet/references/running.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,11 @@ Entries live under `cache.dir` (`.e2e/cache/` by default), one file per key,
relative to the config file. `E2E_CACHE_DIR` replaces `cache.dir`, so worktrees can share one directory;
it resolves from the config file's directory, a leading `~` or `~/` is the
home directory, a blank value is ignored, and the mode stays as configured.
There are no `cache` commands; delete the
Parallel runs can share the directory: each write replaces its entry file
atomically, and two runs that write one key leave the last write. On Windows,
a write that the file system refuses while another write replaces the same
entry, or while a reader holds it open, is retried for a moment before it
counts as not written. There are no `cache` commands; delete the
directory to clear it. The modes are `read-write`, `read-only`, and `off`.
Topic `agent` explains what is recorded. Secret values never enter cache
entries: a secret an act received in `Params` that shows in a control's name,
Expand Down
25 changes: 1 addition & 24 deletions src/E2E/Cache/StepCache.cs
Original file line number Diff line number Diff line change
Expand Up @@ -243,30 +243,7 @@ public void Write(string key, CacheEntry entry)
{
ArgumentNullException.ThrowIfNull(entry);
Directory.CreateDirectory(_directory);
var path = PathFor(key);
var temp = path + "." + Environment.ProcessId + "." + Guid.NewGuid().ToString("N") + ".tmp";
try
{
File.WriteAllText(temp, JsonSerializer.Serialize(entry, JsonDefaults.Options));
File.Move(temp, path, overwrite: true);
}
catch
{
try
{
File.Delete(temp);
}
catch (IOException)
{
// The write error is the one to report.
}
catch (UnauthorizedAccessException)
{
// The write error is the one to report.
}

throw;
}
AtomicWrite.WriteFileAtomic(PathFor(key), JsonSerializer.Serialize(entry, JsonDefaults.Options));
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
}

public void Delete(string key)
Expand Down
Loading
Loading