From 106eebc56aabd3fc9072dcfbf437d1a6272953e2 Mon Sep 17 00:00:00 2001 From: Dario Kondratiuk Date: Mon, 26 Oct 2026 16:17:44 -0300 Subject: [PATCH 1/5] Config: maxObservationBytes and maxInputTokens with observation cutting (closes #646) Port upstream agent/observation-budget.ts: the agent keys, the byte budget around what the request already costs, the cut of an over-budget screen in acts and judgments, the judgment pre-flight, and the run limits. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01DH1b4gwq2kmMK94pdZRQZm --- COMPATIBILITY.md | 12 +- README.md | 2 +- .../api/E2E.Engine.BrowserDownload.html | 13 ++ docs-site/public/api/E2E.Engine.Dialog.html | 13 ++ .../public/api/E2E.Engine.DialogHandler.html | 13 ++ .../public/api/E2E.Engine.DialogType.html | 13 ++ .../api/E2E.Engine.EngineFixtureContext.html | 13 ++ .../api/E2E.Engine.IEngineFixtureHost.html | 13 ++ .../docs/api/E2E.Engine/DocumentElement.md | 48 ++++---- .../docs/api/E2E.Engine/DocumentPage.md | 22 ++-- .../docs/api/E2E.Engine/DocumentWorld.md | 4 +- .../docs/api/E2E.Engine/EngineFinishInfo.md | 12 +- .../docs/api/E2E.Engine/EngineInitInfo.md | 12 +- .../docs/api/E2E.Engine/EnginePrepareInfo.md | 14 +-- .../api/E2E.Engine/EnginePrepareResult.md | 6 +- .../docs/api/E2E.Engine/EngineScreenshot.md | 10 +- .../docs/api/E2E.Engine/EngineStartOptions.md | 22 ++-- .../content/docs/api/E2E.Engine/IEngine.md | 2 +- .../docs/api/E2E.Engine/IEngineSession.md | 2 +- .../content/docs/api/E2E.Engine/WebEngine.md | 2 +- .../docs/api/E2E.OAuth/CopilotProvider.md | 2 +- .../src/content/docs/api/E2E/ActOptions.md | 14 +-- .../src/content/docs/api/E2E/ActResult.md | 10 +- .../src/content/docs/api/E2E/ActionEvent.md | 12 +- .../src/content/docs/api/E2E/ActionHook.md | 2 +- .../src/content/docs/api/E2E/ActionOptions.md | 4 +- .../src/content/docs/api/E2E/ActionSource.md | 6 +- .../src/content/docs/api/E2E/AgentConfig.md | 68 ++++++++--- .../src/content/docs/api/E2E/AgentOptions.md | 41 ++++++- .../src/content/docs/api/E2E/AppConfig.md | 12 +- .../src/content/docs/api/E2E/AssertOptions.md | 10 +- .../src/content/docs/api/E2E/CacheConfig.md | 8 +- .../src/content/docs/api/E2E/CacheInfo.md | 18 +-- .../src/content/docs/api/E2E/CacheMode.md | 8 +- .../src/content/docs/api/E2E/E2EConfig.md | 42 +++---- .../src/content/docs/api/E2E/E2EDefaults.md | 33 +++++ .../src/content/docs/api/E2E/E2EFixture.md | 84 +++++++++---- .../src/content/docs/api/E2E/E2ESession.md | 8 +- .../content/docs/api/E2E/E2ESessionOptions.md | 114 ++++++++++++------ .../content/docs/api/E2E/ExtractOptions.md | 8 +- docs-site/src/content/docs/api/E2E/Locator.md | 2 +- .../content/docs/api/E2E/LocatorExpression.md | 46 +++---- .../docs/api/E2E/LocatorWaitForOptions.md | 6 +- .../docs/api/E2E/PressSequentiallyOptions.md | 4 +- .../docs/api/E2E/ScrollUntilVisibleOptions.md | 8 +- .../src/content/docs/api/E2E/TargetConfig.md | 12 +- .../content/docs/api/E2E/WaitForOptions.md | 12 +- .../src/content/docs/api/E2E/WaitForState.md | 10 +- skills/e2e-dotnet/references/agent.md | 20 ++- skills/e2e-dotnet/references/debugging.md | 2 +- src/E2E/Agent/Act.cs | 16 ++- src/E2E/Agent/ActContext.cs | 5 +- src/E2E/Agent/ActTools.cs | 2 +- src/E2E/Agent/Agent.cs | 63 ++++++++-- src/E2E/Agent/AgentObservation.cs | 4 - src/E2E/Agent/AgentOptions.cs | 59 ++++++++- src/E2E/Agent/Model.cs | 21 ++++ src/E2E/Agent/ObservationBudget.cs | 53 ++++++++ src/E2E/Agent/ObservationFeed.cs | 74 +++++++++++- src/E2E/Agent/SnapshotText.cs | 71 ++++++++--- src/E2E/Config/E2EConfig.cs | 47 +++++++- src/E2E/Config/E2EDefaults.cs | 9 ++ src/E2E/Explore/Executor.cs | 4 + src/E2E/Explore/Explore.cs | 4 + src/E2E/Locator/Screen.cs | 2 +- src/E2E/Report/BuildReport.cs | 5 +- src/E2E/Report/ReportFile.cs | 6 + src/E2E/Report/ReportRecords.cs | 16 ++- src/E2E/Run/AttemptReport.cs | 2 +- src/E2E/Run/E2EFixture.cs | 8 ++ src/E2E/Run/E2ESession.cs | 14 ++- src/E2E/Run/FailureEvidence.cs | 2 +- src/E2E/Run/StepScreens.cs | 3 - .../AgentConfig/ACustomExecutorTests.cs | 2 + .../AgentConfig/AgentConfigDefaultsTests.cs | 20 ++- tests/E2E.Tests/AgentConfig/RunLimitsTests.cs | 49 ++++++++ .../AgentConfig/TheAgentsEntryShapeTests.cs | 3 +- .../ObservationByteBudgetTests.cs | 29 +++++ .../PrepareObservationTests.cs | 30 ++++- .../ObservationBudgetWiringTests.cs | 87 +++++++++++++ .../ObservationByteBudgetTests.cs | 59 +++++++++ tests/E2E.Tests/ConfigTests.cs | 9 +- .../ReportStatus/ReportStatusBuilders.cs | 4 +- .../E2E.Tests/ReportStatus/RunLimitsTests.cs | 22 +++- 84 files changed, 1295 insertions(+), 383 deletions(-) create mode 100644 docs-site/public/api/E2E.Engine.BrowserDownload.html create mode 100644 docs-site/public/api/E2E.Engine.Dialog.html create mode 100644 docs-site/public/api/E2E.Engine.DialogHandler.html create mode 100644 docs-site/public/api/E2E.Engine.DialogType.html create mode 100644 docs-site/public/api/E2E.Engine.EngineFixtureContext.html create mode 100644 docs-site/public/api/E2E.Engine.IEngineFixtureHost.html create mode 100644 src/E2E/Agent/ObservationBudget.cs create mode 100644 tests/E2E.Tests/AgentConfig/RunLimitsTests.cs create mode 100644 tests/E2E.Tests/AgentObservation/ObservationByteBudgetTests.cs create mode 100644 tests/E2E.Tests/AgentObservationBudget/ObservationBudgetWiringTests.cs create mode 100644 tests/E2E.Tests/AgentObservationBudget/ObservationByteBudgetTests.cs diff --git a/COMPATIBILITY.md b/COMPATIBILITY.md index 1c69ccf4..7daf83cd 100644 --- a/COMPATIBILITY.md +++ b/COMPATIBILITY.md @@ -128,7 +128,7 @@ Names are C# versions of the JavaScript API: `agent.act` is `ActAsync`, `screen. - The full locator action set (`dragTo`, `swipe`, `hover`, `setInputFiles`, `secondaryTap`, `longPress`, pointer points). `secondaryTap` and `longPress` have no counterpart in the document engine - The screen-level `screen.tapAt` and `screen.swipe` - Per-action options other than `timeout` (and `delay` for `pressSequentially`): click `modifiers` on a locator (the agent's `tap` and `double_tap` hold them, above), tap `position`, and the `{ label, value, index }` form of `selectOption` -- Diff-only observations. Each model turn receives a full text snapshot +- Diff-only observations. Each model turn receives a full text snapshot, cut at the observation byte budget (see [Agents](#agents)) - Telemetry - `STEP_NOT_AWAITED`, the error that fails a test body, hook, or fixture teardown that returns before its `expect.poll` finished, and cancels that poll. The step recorder has `AbandonRunning` and records a step the test did not await as `STEP_NOT_AWAITED`, but no fixture calls it yet, and `expect.poll` is not a step. Nothing cancels a poll the test does not await, so it keeps reading until its timeout or the `CancellationToken` passed to it. The compiler warns (CS4014) only for a bare, un-awaited `Task` call in an `async` method - Browser fixture members `locator` and `frameLocator`, and the `expect(browser)` matcher `toHaveClass` (it needs `browser.locator`). They wait for the engine's `locate` to take a `frame` and a `selector` expression kind (#680); the web engine locates in the engine now (`LocateAsync`), over its tree @@ -494,9 +494,7 @@ Not ported: - `AssertAsync` on an agent with an executor goes to it as a step of kind `Assert`, with the agent's budgets and `JudgmentTimeout`. An assert is never cached, and a passing one verifies the acts before it. `Vision` is refused with `UNSUPPORTED_CAPABILITY`, since the executor decides what its model sees, and `Screenshot` keeps its evidence after the verdict, bounded by the action timeout: a capture that outlives it is dropped and the verdict stands. `WaitForAsync` and `ExtractAsync` keep the judge, which is the executor's `Judge` or `Model`. `ExecutorCache.Off` never replays or records, and takes no look before the step. A step with the cache on that cannot read its first screen (no app opened) still runs, and records nothing - `TypeSecretAsync` fills a secret the step's params declared, and refuses with `POLICY_DENIED` any other name, a disabled or non-editable field, and a field whose purpose does not fit a password secret. A secret the attempt does not know is `AUTH_CREDENTIAL_UNAVAILABLE`. The built-in agent's `fill_secret` goes through the same check, so its refusals read as the dispatcher's messages and not as `Secret fill refused: ...`. `ExecutorStep.Params` is JSON: a `Secret` is its name and, when set, its purpose, and every string and key that holds a secret value the attempt knows is redacted. `ExecutorAttempt` has the session's `TestTitle` as `TestId`, a fresh id per session, and `Index` as the attempt number minus one; `ExecutorStep.Index` counts the agent steps of the attempt, where upstream's counts every step. `AttachTranscript` and `AttachScreenshotAsync` write under the attempt's results directory (`transcript-.txt`, `screenshots/-