From 758c60f232833933f68be63b6c15be59e2e835a3 Mon Sep 17 00:00:00 2001 From: Petr Glaser <12586960+BleedingDev@users.noreply.github.com> Date: Sun, 20 Sep 2026 16:44:09 +0200 Subject: [PATCH 1/2] docs: design run-centered agent workflows and staged learning Make resumable runs, explicit evidence, and safe next actions the shared contract across CLI, MCP, and executor adapters. Refine the planned Playwright/Midscene fallback with deterministic routing, dispatch uncertainty, cumulative budgets, scoped verification, and direct extraction. Add a vision, system design, agent contract, reviewed-recipe policy, and gated implementation plan. Link proposals from the current guides without advertising unimplemented commands or changing installed skills. Extend documentation-link validation to AGENTS.md and VISION.md. No runtime implementation, model configuration, dependency versions, or workflow permissions changed. --- AGENTS.md | 2 + README.md | 6 ++ VISION.md | 50 +++++++++++++++ docs/agent-contract.md | 124 ++++++++++++++++++++++++++++++++++++ docs/design.md | 121 +++++++++++++++++++++++++++++++++++ docs/fallback.md | 2 + docs/implementation-plan.md | 119 ++++++++++++++++++++++++++++++++++ docs/learning.md | 65 +++++++++++++++++++ docs/testing.md | 2 + test/docs.test.mjs | 2 +- 10 files changed, 492 insertions(+), 1 deletion(-) create mode 100644 VISION.md create mode 100644 docs/agent-contract.md create mode 100644 docs/design.md create mode 100644 docs/implementation-plan.md create mode 100644 docs/learning.md diff --git a/AGENTS.md b/AGENTS.md index 1e5dda8..59b0abc 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -2,6 +2,8 @@ Read `package.json` for commands and `README.md` for supported behavior. Run `npm run check` before committing. Browser integration tests are documented in the README. +For orchestration or protocol changes, read [the system design](docs/design.md) and [agent contract](docs/agent-contract.md). For reusable site knowledge, read [learning and reuse](docs/learning.md). Follow [the implementation plan](docs/implementation-plan.md) for delivery gates. These describe proposed behavior; advertise it in operational guides and skills only after the corresponding tests pass. [VISION.md](VISION.md) indexes the design. + CLI and MCP call the same Effect v4 service. Keep the Jev action loop upstream; `worker/bridge.py` owns only session handoff, assertions, and the process protocol. Pin upgrades as one reviewed change with both lock files updated. Browser selection must fail closed. Never add automatic personal-profile discovery, implicit cloud fallback, or retries of uncertain production mutations. Keep stdout valid JSON or MCP; redact credentials and avoid raw model error dumps. diff --git a/README.md b/README.md index 7ff6946..f6032ae 100644 --- a/README.md +++ b/README.md @@ -116,6 +116,12 @@ Jev's MVP lacks support for some frames, shadow roots, canvas, uploads, popup ta The name is not a benchmark claim. CI exercises real Chrome with scripted model decisions. Live-model reliability, production authentication, and performance on your applications remain to be measured. Patchright and visual-agent frameworks are not included. +## Design and next release + +The proposed design centers work on a resumable run with checkpoints, evidence, and explicit uncertainty. It adds scoped verification, requested data extraction, native image delivery, bounded Midscene fallback, and reviewed recipe reuse. These capabilities are not implemented yet. + +Start with [VISION.md](VISION.md), then follow the task-specific design links there. The [implementation plan](docs/implementation-plan.md) lists delivery gates and evaluations. Current instructions and examples above remain the source of truth for using this release. + ## Develop ```sh diff --git a/VISION.md b/VISION.md new file mode 100644 index 0000000..ca8b28f --- /dev/null +++ b/VISION.md @@ -0,0 +1,50 @@ +# An agent should inherit the work, not reconstruct it + +Status: proposed direction. The [README](README.md) documents the shipped CLI. The [implementation plan](docs/implementation-plan.md) separates that baseline from the work below. + +fastest-e2e should let an agent answer four questions after any interruption: What was requested? What actually happened? What remains uncertain? What can I safely do next? + +The unit of work is a run, not an engine call or a browser tab. A run keeps the task, browser identity, checkpoints, evidence, remaining budget, and authorized scope together. Changing engines or coding agents should not require reconstructing them from a conversation. + +```mermaid +flowchart LR + Intent["Task and expected outcome"] --> Run["Run with checkpoints"] + Run --> Control["Cheapest suitable execution"] + Control --> Evidence["Fresh observations and checks"] + Evidence --> Run + Run --> Result["Answer or test report"] + Result -. "reviewed, scoped reuse" .-> Recipe["Recipe"] + Recipe -. "fresh preconditions" .-> Control +``` + +## What the agent should experience + +A read request returns the requested value with its source, not just a tab ID. A test request returns evidence against the chosen requirements, not a model's completion claim. An unsupported widget offers a continuation on the same run. An uncertain submission offers inspection, not another submission. + +Routine steps stay inside the executor. The coding agent spends its reasoning on interpreting requirements, selecting useful checks, and diagnosing exceptions. It receives enough context to make those decisions and fetches more only when needed. + +A second attempt should cost less because it can reuse validated navigation and targeting knowledge. It must still observe current state and verify the requested result. Learned knowledge never grants permission, supplies today's answer from yesterday's page, or relaxes a test expectation. + +## Optimize for verified work + +First preserve the intended account, authorized changes, and test meaning. Then maximize correctly completed tasks and correctly identified defects. Reduce total time, model calls, output tokens, and repeated exploration within those constraints. + +A fast false pass is a failure. A cheap blocked result is also a failure when a supported, authorized continuation was available. Evaluate both. + +## Keep the system small + +Reuse Jev's policy, Browser Harness's connection, Playwright's targeting, and Midscene's visual loop. The repository owns their shared run contract and evidence. Do not build another browser driver, general workflow language, vector database, or agent framework. + +Keep two operational skills and one setup skill. Skills explain when to act and when to inspect evidence. The executable describes installed capabilities. The run carries task-specific state. Reviewed recipes carry reusable site knowledge. None of these should become a growing global prompt. + +## Read by task + +| Task | Source of truth | +| --- | --- | +| Use the current release | [README](README.md) and command help | +| Change run ownership, execution, or verdicts | [System design](docs/design.md) | +| Change what agents send or receive | [Agent contract](docs/agent-contract.md) | +| Add reusable knowledge or execution recipes | [Learning and reuse](docs/learning.md) | +| Implement or evaluate the next release | [Implementation plan](docs/implementation-plan.md) | + +The design documents describe a target, not additional working commands. Update the operational guides and skills only when the matching behavior has passed its acceptance checks. diff --git a/docs/agent-contract.md b/docs/agent-contract.md new file mode 100644 index 0000000..70b4198 --- /dev/null +++ b/docs/agent-contract.md @@ -0,0 +1,124 @@ +# Agent interaction contract + +Status: proposed, not current command syntax. [System design](design.md) owns run and recovery semantics. This document owns what an agent supplies, receives, and needs to remember. Delivery is tracked in [the implementation plan](implementation-plan.md). + +## Begin with the outcome + +For browser use, provide the URL, bounded goal, stopping condition, and optional requested data. For testing, also provide requirement IDs and observable expectations. Setup supplies the profile and permitted model providers once. Task-specific account, deployment, mutation scope, and cleanup expectations remain explicit. + +Accept a goal without forcing a test file, and a scenario without forcing an elaborate workflow definition. Resolve defaults locally and return their effective values. Detect input, configuration, unsupported-schema, and missing-capability errors before taking browser actions or spending model calls where possible. + +Treat the PR reference as context supplied by the coding agent. It reads repository changes and available deployment evidence through its own tools. The browser runtime does not need a GitHub integration to validate a UI. + +## Small vocabulary, progressive detail + +These are proposed extensions to existing commands. Exact flags and schemas must be finalized with contract tests before appearing in installed skills. + +| Question | CLI operation | MCP equivalent | +| --- | --- | --- | +| Can this installation do the task? | `doctor` | `browser_doctor` | +| Perform a bounded task | `run` | `browser_run` | +| Test these expectations | `test` | `browser_test` | +| What happened, or what is visible now? | `inspect --run ID --view ...` | `browser_inspect` | +| Continue remaining work | `resume --run ID` | `browser_resume` | +| Show this owned page | `screenshot --run ID` | `browser_screenshot` | +| Check saved expectations again, without replay | `verify --run ID` | `browser_verify` | +| Release task tabs | `close --run ID` | `browser_close` | + +Retain the existing target-based inspection and close commands for compatibility. A run resolves targets internally. When a run owns several tabs, return labeled target choices and require disambiguation where necessary. Never make the agent repeatedly copy a CDP endpoint or infer a target ID from a URL. + +`run` and `test` should accept structured input through files/stdin as well as common CLI arguments. Reserve stdout for final JSON; send opt-in progress to stderr or supported MCP progress notifications. No banner, provider log, or Base64 image dump belongs in text output. + +Do not add a general `execute anything` tool to the default MCP tool list. Keep trusted script execution opt-in. Offer narrow operations and whole-task execution before requiring code authoring. + +## Every response should orient the next decision + +Return a compact summary first. Include the run ID, record revision, execution state, verification state, requested answer, relevant evidence, remaining budget, and legal next actions. Summaries are derived from structured receipts and checks; they do not require an additional summarizer-model call. + +This shortened example illustrates an uncertain submission, not an implemented response schema: + +```json +{ + "schemaVersion": 2, + "runId": "r_settings_01", + "revision": 7, + "status": "blocked", + "execution": { + "state": "blocked", + "reasonCode": "mutation_uncertain", + "lastCheckpoint": "display_name_entered" + }, + "verification": { "status": "not_run", "pending": ["saved_after_reload"] }, + "mutation": { "state": "uncertain", "actionId": "a_save_04" }, + "summary": "Save was dispatched; its outcome is unknown. The task tab is retained.", + "budget": { "actionsRemaining": 12, "modelCost": { "kind": "unknown" } }, + "evidenceRefs": ["e_before_save"], + "nextActions": [ + { "operation": "inspect", "arguments": { "runId": "r_settings_01", "view": "page" } } + ] +} +``` + +Next actions are typed suggestions computed from state, capabilities, and policy. They are not website-authored instructions or additional authorization. Validate all preconditions again when invoked. Return enough context with an error to use the suggested remedy without searching a stack trace. + +## Observation is a budgeted query + +Extend `inspect` with summary, page, checks, and history views. Summary reads the durable record and makes no browser or model calls. Page reads live state. Checks and history retrieve recorded evidence, with explicit capture times and a clear distinction from fresh inspection. + +Default page output contains the current URL, title, task-relevant visible controls/text, frame boundaries, pending dialog, and supported locator hints. Preserve accessible labels and exact observed values when they matter. Report empty, unavailable, stale, and truncated as different conditions. + +Provide a small default output limit, explicit truncation, and stable cursors bound to an observation or event sequence. Prefer a focused frame/region query to repeatedly dumping the whole page. Changes since an earlier observation must identify their base; a new agent can always request a full focused view. Never truncate a failed check or a required extraction field silently. + +Expose which observation strategies are available and which are not. A missing element in an unsupported frame is unknown, not an observed absence. An inspection may reveal account identity through normal UI; a profile directory name does not prove the active tenant or role. + +### Requested answers, not tab IDs alone + +Allow an optional `extract` description and supported structured-output schema on browser tasks. Return requested values with observation references and the method used: direct DOM read or model interpretation. Type validation is not factual verification. Failed or incomplete extraction returns a field-level error or partial result, not invented values. + +Prefer direct control/text extraction when a scoped locator is known. Use a visual/model query only when necessary and charge it to the run. A task asking for a display name should normally finish in one public call with that value and evidence, rather than requiring `run`, an unrelated inspection, and a separate interpretation step. + +### Images + +MCP screenshots return an actual image content block plus structured metadata. CLI returns a local artifact path and the same metadata. Resource references need a supported retrieval route; a path alone is not an image for a remote MCP client. Provide a bounded direct-content route when that client cannot read local files. + +Metadata includes observation ID, browser/target identity, timestamp, viewport size in CSS pixels, encoded image size, crop origin, and scale. Coordinate operations are bound to that observation, not unexplained screen coordinates. Prefer fresh locator-based control where possible. + +Keep viewport images the default. Full-page captures, video, and verbose traces are explicit requests or configured failure diagnostics. Midscene's own image-reading ability must not depend on the outer coding agent's model supporting vision. A client without image viewing still receives a useful textual summary and an honest visual-verification limitation. + +## Result interpretation + +Preserve the legacy top-level `done`, `passed`, `failed`, and `blocked` values and exit codes. Introduce versioned structured details additively; reject incompatible requests rather than silently ignoring important fields. + +| Situation | Meaning | +| --- | --- | +| Executor reports completion without checks | `done`, unverified | +| Required execution completes and all declared checks pass | `passed`, with evidence type per check | +| A declared check mismatches after completed execution | `failed`; product attribution still requires diagnosis | +| Execution, extraction, required scope, or verification cannot finish | `blocked`, with partial evidence preserved | +| Case never executed or target change not deployed | Untested coverage in the report, not a fabricated runtime pass | + +Distinguish goal outcome from cleanup outcome. A successful feature check with failed cleanup reports both and identifies leftover production data. Preserve evidence collected before cleanup; checking the cleaned-up state does not replace it. + +Expected blockers have stable codes such as `auth_required`, `unsupported_scope`, `ambiguous_target`, `mutation_uncertain`, `budget_exhausted`, and `session_changed`. Redacted diagnostics carry an evidence reference where useful. Malformed MCP requests use protocol errors; task failures remain interpretable tool results. Transport errors must not cause automatic mutation retries. + +## Survive context loss and concurrent agents + +A replacement coding agent should need only the run ID and configured home. `inspect` supplies the original task, checkpoint, observed changes, unresolved uncertainty, pending checks, and safe continuation options. It does not replay a conversational transcript. + +Resume accepts an expected run revision and refuses stale competing requests. The runtime also revalidates live preconditions. Two agents resuming one run cannot dispatch in parallel. Cancellation leaves a checkpoint or an explicit uncertain boundary, not a success-looking empty response. + +Long work may later need start/status/cancel delivery to survive client request timeouts. Keep it out of the initial API unless client tests demonstrate that need. A transport disconnect must never silently create unbounded detached work. + +## Setup and skills + +`doctor` should return a capability/readiness matrix by engine and operation, including disabled capabilities and remedies. Separate installed, configured, key-present, provider-tested, and browser-connected. Do not call every provider on every task. Live credential validation is explicit and its resource use visible. + +Operational skills retain their current commands until this contract ships. Then `browser-use` teaches run/inspect/resume decisions, and `browser-test` teaches requirement-to-evidence mapping and failure interpretation. Setup teaches capability readiness and client registration. Recipes and troubleshooting are disclosed only when relevant; none belong in always-loaded skill descriptions. + +Test CLI/MCP behavior against the same fixtures. Test actual skill discovery and image delivery per supported client. Do not claim one universal installation path or confuse generic shell compatibility with validated Codex, Claude Code, OpenCode, Pi, or OMP integration. + +## Design references + +[Anthropic's tool-design guidance](https://www.anthropic.com/engineering/writing-tools-for-agents) motivates task-shaped operations, meaningful results, bounded output, and evaluation with agents. Our concrete response contracts above are proposals to test, not guarantees derived from that guidance. + +The [MCP tools specification](https://modelcontextprotocol.io/specification/2025-06-18/server/tools) defines structured content, images, and errors. Validate the selected Effect transport against those representations. Follow [writing-for-agents](https://github.com/mattpocock/skills/blob/main/skills/productivity/writing-for-agents/SKILL.md) for conditional references and single-source documentation. diff --git a/docs/design.md b/docs/design.md new file mode 100644 index 0000000..724edbb --- /dev/null +++ b/docs/design.md @@ -0,0 +1,121 @@ +# System design + +Status: proposed. Baseline reviewed at `b3b1500`. Read [the vision](../VISION.md) for the objective, [the agent contract](agent-contract.md) for interaction details, and [the implementation plan](implementation-plan.md) for delivery gates. + +## Current baseline + +The [runtime](../src/runtime.ts) runs a pinned Python worker per request. The [host](../src/host.ts) binds it to the configured Chrome generation and locks one configured home. The [worker](../worker/bridge.py) registers task tabs and checks the top-level document. Runs return a tab ID and status, not a durable execution record or extracted answer. Browser Harness fallback is trusted script execution. There is no autonomous Midscene integration or native MCP screenshot result yet. + +Those are useful building blocks. They leave progress, recovery decisions, and evidence interpretation in the outer agent's conversation. The target below moves durable facts into the runtime without moving general product reasoning there. + +## Ownership follows the task + +```text +Task specification What outcome, scope, and evidence are required? + Run What has happened and what remains? + Checkpoint What can be continued from this observed state? + Scoped operations Observe, act, extract, and check on owned targets. + Executor adapter Jev, Playwright recipe, or Midscene. + Session lease Exact profile, Chrome generation, and task tabs. +``` + +Evidence refers back to a run, checkpoint, target, and requirement. Recipes are reusable procedures selected by the run, not owners of the browser or definitions of success. + +| Owner | Owns | Does not own | +| --- | --- | --- | +| Coding agent | Requirement interpretation, case selection, deployment evidence, diagnosis | Low-level control between every routine click | +| Effect runtime | Validated task specification, run state, budgets, session lease, routing, result assembly | A replacement browser engine or model reasoning trace | +| Executor adapter | Execution through its upstream library, progress receipts, cancellation | Permission changes, success criteria, another profile | +| Verifier | Fresh checks with explicit scope and evidence type | Actions that repair the application to satisfy a check | +| Recipe store | Reviewed, parameterized knowledge and applicability | Credentials, permissions, authoritative current page state | + +Use Effect Schema for shared contracts and Context services for these responsibilities. Keep CLI and MCP as transport adapters to the same handlers. Keep a single package until an independently deployable component earns a split. + +## A run outlives an invocation + +Create a run ID and persist its validated intent before the first browser action. Keep the current browser-generation ID, owned targets, expected outcomes, requested extraction, attempts, checkpoints, mutation receipts, budget usage, and evidence references under that run. + +Use a local append-only event file and atomically replaced summary. Start with files, not a workflow service or database. Record observable decisions and action receipts, not hidden model reasoning. Events need sequence numbers and input/schema versions so summaries can be rebuilt and incompatible records rejected explicitly. + +A paused or interrupted run remains inspectable after the CLI exits or the coding agent loses its context. It is resumable only if the original live browser state can be validated. Browser restart preserves the record, not a promise of restoring in-memory tab state. + +Journal before an action can be dispatched and record its completion afterward. If a process dies between those writes, mark the action uncertain. Observation may resolve uncertainty; replay cannot. Request deduplication prevents duplicate runtime dispatch for an identical request ID. It cannot promise exactly-once side effects on a website. + +A request ID is scoped to the configured home and canonical input digest. Reusing it with different input is a conflict. A duplicate completed request returns its recorded result; a duplicate active request returns the run handle, not another execution. + +## Checkpoints and exclusive control + +A checkpoint identifies the reached UI state, completed subgoal, unchanged remaining goal, evidence, target lineage, and budget consumed. Prefer semantic boundaries such as an opened dialog or a confirmed save. Do not require the outer agent to author a click-by-click plan. + +Hold one exclusive lease per canonical profile and Chrome generation during an execution segment, including an engine handoff. The existing home lock alone does not protect the same profile configured under two homes. Detect and reject that collision. Releasing an invocation's lease does not transfer ownership of its tabs. + +Between invocations, a person may change a tab. Resume validates browser identity, target ownership, checkpoint preconditions, and new observations. A stored revision is a journal version, not proof that the live DOM is unchanged. Re-resolve locators and reject stale image coordinates after navigation, scrolling, layout changes, or an invalid screenshot reference. + +Jev relinquishes control before another adapter acts. A popup may join the run only through verified opener lineage and allowed navigation context. Never choose a page by array order or URL alone. Detaching an adapter must leave Chrome and unrelated tabs open. + +## Choose the cheapest capable executor + +Selection uses validated task requirements, current capabilities, live state, and approved recipe applicability. The policy is conditional, not a mandatory sequence of paid trials. + +```text +if a reviewed UI recipe matches and its preconditions hold: + run the recipe and verify fresh evidence +else if a known interaction is directly supported by scoped Playwright operations: + use those operations without a visual model +else if the requested goal is supported by Jev: + run Jev +else if vision is configured, allowed, and suitable: + run Midscene on the same owned target +else: + return the precise missing capability and safe next actions +``` + +Ordinary iframe or open-shadow-root targeting does not itself require vision. Use Midscene for rendered controls whose useful identity is visual, or when a bounded semantic workflow needs its visual planner. Never infer capability support solely from the presence of an iframe or canvas somewhere on the page. + +`auto`, `jev`, and `vision` remain the proposed public policies. `auto` may skip a predictably unsuitable engine and may use a validated recipe. An explicit engine selection does not silently switch engines. Record the selected executor and reason. Load Playwright/Midscene only for operations that need them. + +The adapter must expose enough supported hooks to meter steps and report dispatch uncertainty. If that cannot be demonstrated for a pinned upstream version, limit it to bounded calls and disable automatic continuation across ambiguous boundaries. Do not invent reliable classification by parsing a model's prose. + +### Recovery rules + +| Observation | Response | +| --- | --- | +| Unsupported interaction, no unresolved mutation, suitable authorized engine available | Continue the remaining subgoal on the same run within the remaining budget | +| Lost response after a potentially mutating action | Mark uncertain; inspect before offering continuation | +| Assertion mismatch after completed execution | Preserve failure; diagnose without another agent trying to make it pass | +| Missing provider, expired login, unsupported scope, or exhausted budget | Return a typed blocker and a concrete remedy | +| User cancellation or lost browser identity | Stop dispatch; never escalate to another engine | + +A model's low confidence can justify more observation. It is not calibrated proof that replay or escalation is safe. Automatic recovery needs an observed capability failure and satisfied preconditions. + +## Verification is separate from execution + +Execution completion, requirement verification, and mutation certainty are separate fields. Finishing a flow is not evidence that it worked. An assertion mismatch is an observation, not automatically proof of a product defect. + +Freeze requirement IDs and expected results before the actions they evaluate. Checks refer to checkpoints and explicit frame/locator scopes. Evidence records the expected/actual pair, capture time, URL, target, observation ID, and method. Record deployment identity as verified, user-supplied, or unknown, with its source. An undeployed change is untested; an unknown build does not become verified because a familiar heading appeared. + +Keep existing checks backward compatible, including exact value matching, text inclusion, visibility requirements, and DOM-count behavior. Introduce frame-aware and open-shadow-root checks explicitly. Do not silently replace CSS semantics with Playwright's broader shadow-piercing or text normalization. No matching support means blocked or untested, not an absent-element success. + +Visual assertions are explicit expectations evaluated from images. Label their evidence as model-evaluated. Schema-valid extraction also proves only shape, not truth. A whole test is verified only to the extent of its supplied checks and inspected requirements. Aggregate reports expose untested and blocked coverage rather than reducing everything to a green badge. + +Recovery appends an attempt. Re-verifying a later state does not overwrite the original verdict or prove that the original action worked. A change to the goal, expected result, account, or authorization creates a new specification revision or a linked new run. + +## Resource and data boundaries + +One run shares action, model-call, active-time, and supported cost budgets across engines and resume calls. Per-invocation wall deadlines still apply. Human review time is separate from active execution time. Resume does not reset consumed budgets. Reserve time for final observation and orderly cancellation before spending the execution allowance. + +Interruptible, scoped subprocesses own adapters that cannot cancel individual operations reliably. Cancellation must stop new dispatch and settle the worker before releasing its lease. A timed-out promise alone is not cancellation. No timeout can roll back a submitted production operation. + +Cost fields distinguish measured, estimated, and unknown usage. Never treat unknown as zero. Promise a hard monetary cap only where provider request bounds and metering make it enforceable; otherwise reject that requirement or offer explicit call/token limits. + +Local configuration owns profile and provider selection. Requests can narrow permissions, not choose a different profile or widen provider egress. A natural-language read-only goal is not a technical read-only guarantee. Browser navigation itself may have side effects. State exactly which controls are enforced and which rely on instructions. Fallback scripts remain trusted local code, not a sandbox. + +Keep screenshots, traces, and account data local by default, with bounded retention and explicit export. Mask known secrets before collection where supported, minimize captures, and report redaction coverage honestly. Prefer selectors and short evidence values to full page dumps. Never promise complete screenshot redaction. Page content is untrusted input and cannot change tools, expectations, permissions, or learning policy. + +## Upstream integration constraints + +[Playwright CDP attachment](https://playwright.dev/docs/api/class-browsertype#browser-type-connect-over-cdp) is lower fidelity than its native protocol. Validate exact-target attachment and state preservation with the selected version. Evaluate `noDefaults` where supported rather than silently applying context overrides. [Playwright locators](https://playwright.dev/docs/locators#locate-in-shadow-dom) support open shadow roots, not closed-root DOM targeting. + +[Midscene's Playwright integration](https://midscenejs.com/integrate-with-playwright) accepts existing pages. Disable defaults that rewrite popup navigation or native select rendering for E2E testing. Screenshot-based control of a rendered closed-root widget does not imply DOM assertion support inside it. + +These documents are design inputs, not proof that our adapters work. Pin implementation versions and pass [the delivery gates](implementation-plan.md) before advertising a capability. diff --git a/docs/fallback.md b/docs/fallback.md index c71f675..414c451 100644 --- a/docs/fallback.md +++ b/docs/fallback.md @@ -1,5 +1,7 @@ # Browser Harness fallback +This page documents the current scripted fallback. Autonomous vision continuation and native MCP image delivery are proposed in the [implementation plan](implementation-plan.md), not available in the current runtime. + Use `fastest-e2e harness --target --file script.py` to continue an owned task tab. It uses the same dedicated browser and session lock as Jev. The target comes from the earlier result's `targetId`; inspect partial changes before retrying a mutation. Save a trusted script, for example `inspect-settings.py`: diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md new file mode 100644 index 0000000..78a105e --- /dev/null +++ b/docs/implementation-plan.md @@ -0,0 +1,119 @@ +# Implementation plan + +Status: proposed. No stages below are implemented by this documentation change. Baseline is `b3b1500`. This plan supersedes the earlier chat-only fallback sequence while retaining Jev, same-profile CDP, Playwright, and Midscene as the intended integrations. + +[System design](design.md) owns behavior. [Agent contract](agent-contract.md) owns requests and responses. [Learning and reuse](learning.md) owns recipe policy. This file owns sequencing, acceptance gates, and evaluation. Keep installed skills and examples on the current CLI until their matching gate passes. + +## What changes from the earlier plan + +| Earlier emphasis | Revised decision | +| --- | --- | +| Add a second autonomous engine | Establish a durable run and evidence contract before engine handoff | +| Use vision for unsupported DOM | Use scoped deterministic operations for supported frames/open roots; reserve vision for visual work | +| Return status and target | Return the requested answer, observed progress, uncertainty, evidence, and legal next actions | +| Resume a tab | Resume a validated checkpoint with shared budgets and immutable failed attempts | +| Measure runtime | Measure time and total resources to correct, verified completion, including recovery | +| Solve each task afresh | Add reviewed, scoped recipes only after fresh verification and safety gates exist | + +```mermaid +flowchart TD + G0["G0: attachment and upstream-hook proofs"] --> G1["G1: durable run and response contract"] + G1 --> G2["G2: scoped observation, extraction, verification, images"] + G2 --> G3["G3: bounded Midscene handoff"] + G3 --> G4["G4: CLI/MCP and real-agent evaluations"] + G4 --> G5["G5: measured recipe reuse"] +``` + +These are successive usable increments. G1 and G2 should improve Jev-only use without requiring a vision provider. G5 is not a prerequisite for shipping the fallback repair. + +## G0. Prove integration boundaries + +- [ ] Pin compatible Playwright/Midscene versions and keep Effect/core adapter versions matched. Verify APIs against installed source, not examples on an unpinned branch. +- [ ] Attach to the registered target in existing dedicated Chrome. Test duplicate URLs, unsaved input, storage, focus/media settings, nested frames, and two independent profiles. +- [ ] Demonstrate disconnection/cancellation without closing shared Chrome. Check same-profile/two-home contention and exact owned-popup lineage. +- [ ] Identify supported pre-dispatch, completion, and cancellation hooks in Jev and Midscene. Document what can and cannot be observed without rewriting their loops. + +Exit gate: repeatable, secret-free integration tests show correct target selection and preserved state. A missing receipt or cancellation hook blocks automatic handoff at that boundary. It does not justify speculative recovery or an unreviewed engine rewrite. Deliver the proof and its limitations before committing to automatic mode. + +## G1. Make runs durable and resumable + +- [ ] Add the versioned run record, event journal, semantic checkpoints, request deduplication, and summary projection through Effect services. +- [ ] Persist intent before browser actions and dispatch/completion receipts around supported action boundaries. Preserve unknown outcomes on interruption. +- [ ] Use profile/generation-scoped leases with explicit release and stale-owner recovery rules. Reject stale expected revisions and competing resume requests. +- [ ] Extend result details and inspection while preserving current input formats, statuses, target-based commands, and exit codes. +- [ ] Implement resume of validated live state, cumulative budgets, and local record inspection after browser disconnect. New browser generations require explicit new execution, not false continuation. + +Exit gate: kill the caller after an action dispatch. A fresh process can explain the last checkpoint and uncertainty from the run ID alone. Repeating the same request cannot dispatch twice. Same-ID/different-input and concurrent resume requests are rejected. No test passes from an empty response or lost worker. + +## G2. Make observation and verification usable + +- [ ] Add focused live inspection, labeled frame/locator scopes, exact evidence references, and bounded/paged history. Keep recorded observations distinguishable from live state. +- [ ] Add optional structured extraction to browser tasks. Direct reads precede model interpretation where suitable. Incomplete or schema-invalid extraction stays explicit. +- [ ] Add scoped Playwright checks for open roots and nested/cross-origin frames. Preserve legacy selector and matching semantics. Detect unsupported scopes instead of treating them as absence. +- [ ] Add `verify` on saved expectations and checkpoints. Preserve original attempts, attribution uncertainty, and separate cleanup results. +- [ ] Add screenshot artifacts with identity, capture time, coordinate scale/crop metadata, bounded retention, CLI paths, and native MCP image content. +- [ ] Add typed stop reasons and capability/readiness descriptions with actionable remedies. Zero provider calls for ordinary setup inspection. + +Exit gate: a read task returns its requested value and evidence in one public execution call. Positive and intentionally broken pages produce correct verdicts inside nested frames and open roots. A closed-root DOM check is unsupported, never an accidental pass. Test strict selector ambiguity, hidden matches, stale observations, missing evidence, extraction errors, unsupported clients, and screenshot size/scale limits. + +## G3. Add bounded autonomous visual continuation + +- [ ] Connect Midscene to the run's validated page and remaining subgoal. Share ownership, deadlines, mutation receipts, and budgets with Jev. +- [ ] Implement `auto`, `jev`, and `vision` policies plus the recovery table in the system design. Normal visual use must not require arbitrary local script execution. +- [ ] Keep explicit vision configuration and provider/data-egress consent. Disable UI-altering defaults for popup navigation and native-select rendering. +- [ ] Add explicit visual assertions with image evidence and model-evaluated labeling. Keep them separate from deterministic checks. +- [ ] Make cancellation stop adapter dispatch before another executor or caller can acquire the lease. Keep screenshots off successful Jev-only paths unless requested. + +Exit gate: Jev reaches an unsupported visual control, Midscene continues the same run without replaying prior work, and fresh checks establish the result. A timed-out save, failed assertion, missing provider, user cancellation, or browser mismatch does not trigger automatic retry. Controlled tests prove routing and transport; opt-in live tests separately establish visual-model behavior. + +## G4. Evaluate with agents, then ship the guides + +- [ ] Run identical scenario and failure fixtures through CLI and MCP. Include structured errors, native images, cancellation, and a client that cannot read server-local files. +- [ ] Give fresh coding agents only the installed skills and a task. Record unnecessary tool calls, missing information, incorrect actions, repeated setup, and manual interventions. +- [ ] Test context loss and replacement-agent takeover from a run ID. Test full-state and incremental observation paths, not only a prepared happy-path prompt. +- [ ] Verify skill discovery and relevant tool/image capabilities separately in each supported client. Publish tested, partial, and untested status for Codex, Claude Code, OpenCode, Pi, and OMP. +- [ ] Update the README, setup/fallback/testing references, and existing three skills only for shipped behavior. Generate schemas/help from shared contracts; keep troubleshooting and recipes out of global skill descriptions. + +Exit gate: the documented fresh-install, read, test, inspect, resume, and fallback examples work. A text-only outer agent can delegate a visual task without pretending it saw the screenshot. Test reports identify the deployment evidence and do not claim unexecuted requirements passed. Limit the release's compatibility claims to combinations actually exercised. + +## G5. Accumulate useful procedures + +- [ ] Add explicit local candidate creation, manifest review, promotion, versioning, and quarantine using existing scenario/script formats. +- [ ] Select only reviewed recipes with fresh preconditions and current task authorization. Never use saved answers or historical behavior as the current test oracle. +- [ ] Test wrong-origin/account/role rejection, changed UI, stale coordinates, ambiguous selectors, recipe drift, imported hostile instructions, and sanitized export. +- [ ] Compare repeated tasks against the no-reuse baseline, including recipe creation, review, and maintenance costs. Retain only measured improvements. + +Exit gate: a scoped repeated task needs fewer decisions or model calls while still using fresh evidence. A stale or incompatible recipe is rejected before unsafe action. Disabling reuse leaves the ordinary engine paths functional. No automatic rewrite of skills or export of private traces occurs. + +## Evaluation protocol + +Use a small local test application to validate the runner's browser mechanics. Those controlled fixtures are for this repository's tests; they do not introduce an API-setup requirement for real user applications. Production evaluation uses only approved UI interactions, permitted accounts, and explicitly agreed cleanup. + +Keep suites distinct: + +| Suite | Proves | Does not prove | +| --- | --- | --- | +| Contracts and controlled adapter decisions | State transitions, routing, budgets, deterministic evidence and transport | Hosted model reasoning quality | +| Real Chrome integration | Exact-target attachment, input, scopes, screenshots, lifecycle | Every browser/version/site combination | +| Opt-in live-model tasks | End-to-end behavior of configured models on chosen cases | Universal reliability or production suitability | +| Approved production journeys | Target application's actual account/session/UI behavior | Correctness on unrelated accounts or deployments | + +Evaluate basic reading, forms and persistence, same/cross-origin nested frames, open/closed-root widgets, canvas, new tabs, expired auth, ambiguous controls, navigation during observation, and long-running pages. Include intentionally broken product behavior and uncertain submissions, not only successful tasks. + +Compare baseline Jev, the new Jev-only path, auto routing, explicit vision, and later recipe reuse. Use matched cases and equivalent fresh starting state. Repeat trials, disclose sample size, distinguish cold and warm starts, and report variability. Do not rerun production writes solely to fill a benchmark without permission. + +Record verified completion, false passes, false defect claims, blocked coverage, wrong-account/tab actions, duplicate mutations, recovery success, cleanup outcome, total elapsed time, outer-agent calls/tokens, executor calls/tokens, screenshot bytes, metered/estimated/unknown spend, and human interventions. Include attachment, verification, and unsuccessful attempts in totals. + +Correctness and scope violations fail the gate regardless of latency. For the regression fixtures, require no false passes, wrong-target actions, or duplicate uncertain submissions. For performance, record the baseline before selecting an overhead allowance; no fabricated speed target or universal ranking. A successful Jev-only run must make no vision calls. Summary generation and read-only capability discovery must make no model calls. + +## Migration and implementation shape + +Extend shared contracts and the existing runtime rather than creating separate CLI/MCP implementations. Separate orchestration, records, observation/evidence, and adapters behind small Effect services. Reuse upstream execution and locator implementations. Introduce a new source module only when its tests and responsibility warrant it; do not prebuild a monorepo or generic capability compiler. + +Keep old scenarios runnable. Add versioned fields and commands with strict validation and clear errors. Do not silently broaden old selectors, discard old results, or make a newly configured provider the default for existing installations. New automatic recovery is opt-in until its gates pass. Record schema and adapter versions with runs and recipes. + +The initial repair does not include desktop/native automation, a general workflow DSL, vector search, autonomous permission grants, automatic production-data cleanup, or a new browser profile. Native input, popup, and browser state remain part of the behavior under test. + +## Review decisions + +The review should settle the run-centered contract, the deterministic-before-vision rule, strict uncertainty handling, and staged recipe reuse. Exact default output/time budgets and client support claims remain empirical choices for the gates above. Schema/code implementation and live production evaluation are separate changes, not implied by approving these documents. diff --git a/docs/learning.md b/docs/learning.md new file mode 100644 index 0000000..429dd92 --- /dev/null +++ b/docs/learning.md @@ -0,0 +1,65 @@ +# Learning and reuse + +Status: proposed. This document defines how runs can make later work cheaper without making their evidence or permissions less trustworthy. It extends the [system design](design.md); delivery starts only after the core [run and verification gates](implementation-plan.md) pass. + +## Store different things separately + +| Record | Lifetime and purpose | Never treated as | +| --- | --- | --- | +| Observation/evidence | Run-local record of what was seen at a stated time | Current page state on a later run | +| Site hint | Scoped, expiring navigation or targeting advice with provenance | An executable instruction or permission | +| Recipe | Reviewed, versioned procedure with inputs, preconditions, checkpoints, and postconditions | A saved answer or a weakened test oracle | + +Start with explicit recipes and a small exact-match index. Add hints only if repeated tasks show a measurable benefit. No embedding service, global memory agent, or automatic skill rewriting is required. + +## Learn the route, verify the destination + +```mermaid +flowchart LR + Run["Verified run"] --> Candidate["Candidate recipe"] + Candidate --> Review["Review code and data scope"] + Review --> Validate["Fresh trials and negative cases"] + Validate --> Eligible["Eligible scoped recipe"] + Eligible --> Preconditions["Check applicability now"] + Preconditions --> Execute["UI execution and fresh verification"] + Execute --> History["Record success or failure"] + History -. "drift or mismatch" .-> Disabled["Quarantined recipe"] +``` + +Candidate generation and promotion are explicit maintenance operations. They are not extra model calls on every successful task. Recording minimal receipts during execution is sufficient to propose a candidate later. No background learner silently mines account data. + +Prefer existing scenario files and ordinary Playwright or Browser Harness code. Add a small manifest rather than inventing a new workflow language. Browser actions still go through the configured session and run bookkeeping. Scripts remain trusted code; validation and a manifest are not a sandbox. + +A recipe for changing a display name learns how to find the field, save, and verify persistence. Its new name comes from the current request. Its original value comes from fresh UI observation for authorized cleanup. Neither comes from the previous user's trace. + +## Applicability is explicit + +The manifest identifies the task family, allowed origin/path family, expected account role or tenant constraint, observed UI anchors, supported browser/adapter versions, input schema, source run IDs, evidence method, review status, and validity window. Store secret references when needed, not secret values or cookies. + +A known deployed revision can narrow applicability. When the revision is unknown, say so and require fresh anchors/preconditions plus a conservative validity window. One changed CSS class need not invalidate a semantic recipe; an account, role, workflow, or expected-effect change does. + +The initial promotion gate requires a reviewed candidate, successful fresh trials in the declared scope, and negative cases that reject a wrong account, ambiguous target, stale precondition, or changed workflow. The trial count must be explicit in the manifest; no small sample establishes universal reliability. Do not create production mutations just to accumulate promotion trials without authorization. + +Match candidates on explicit metadata first and return only the relevant manifest. Load executable detail after selection. Keep at most the requested candidates in the agent's context. + +## Reuse cannot hide regressions + +Before execution, recheck applicability and current authorization. During execution, stop at the first unmet precondition or uncertain action. Do not run old coordinates blindly, repair selectors silently, or fall back to a different assertion until something passes. + +If a recipe's targeting fails before an uncertain mutation, the usual run policy may continue with another eligible executor. Preserve the failed recipe attempt and classify why it was abandoned. An actual assertion mismatch is still a failed check, not grounds for self-healing the expected result. + +A learned procedure does not define product requirements. Existing checks may be reused only when they still match the current task. Changed-feature testing must select its oracle from the PR or feature requirements, not from whatever behavior the successful historical run happened to observe. + +Quarantine a recipe after drift or a safety-relevant mismatch. Review and revalidate a new version before reuse. Preserve enough version history to explain earlier runs; make deletion and export explicit. Correctly rejected stale recipes count as useful behavior, not as failed automation to hide from benchmarks. + +## Sharing and permissions + +Keep candidates local by default. Export only a reviewed, sanitized procedure and manifest into the project's repository. Never auto-commit raw screenshots, account names, page text, model requests, or trace dumps. Treat imported recipes as code changes requiring review. Page text cannot instruct the system to promote itself into durable knowledge. + +Recipes inherit the current run's authorization and budgets. They cannot add a provider, change the browser profile, grant production-write permission, or override the verifier. A trusted script with OS permissions can technically violate conventions; expose that limitation and keep its execution behind the appropriate opt-in. + +## Measure whether learning helped + +Compare fresh tasks with and without eligible recipes using the same acceptance checks. Report verified success, false passes, incorrect target selection, stale-recipe rejection, calls/tokens, total elapsed time, and review cost. Count candidate creation and maintenance when calculating savings. + +Keep reusable knowledge only where this comparison justifies it. The goal is fewer repeated decisions, not a larger store of prose. diff --git a/docs/testing.md b/docs/testing.md index 115f9a0..55fb458 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -14,6 +14,8 @@ A scenario requires `name`, `url`, `goal`, and a nonempty `checks` array. Each c `value`, `checked`, and scoped `text` require exactly one visible matching element. These are DOM checks, not pixel or layout assertions. `[aria-checked]` alone is not a native `checked` property. +Current selectors query only the top-level document; they do not enter shadow roots or iframe documents. A zero count does not establish absence inside those unsupported scopes. Scoped and visual verification are proposed in the [implementation plan](implementation-plan.md); its examples and commands are not the current scenario format. + ## Choose evidence before acting ```text diff --git a/test/docs.test.mjs b/test/docs.test.mjs index a7870dd..9972805 100644 --- a/test/docs.test.mjs +++ b/test/docs.test.mjs @@ -39,7 +39,7 @@ test("skill names, descriptions, and local reference pointers are valid", async }); test("local documentation links resolve", async () => { - const docs = ["README.md", ...(await readdir(path.join(root, "docs"))).filter(name => name.endsWith(".md")).map(name => `docs/${name}`)]; + const docs = ["README.md", "AGENTS.md", "VISION.md", ...(await readdir(path.join(root, "docs"))).filter(name => name.endsWith(".md")).map(name => `docs/${name}`)]; for (const file of docs) { const source = await read(file); for (const match of source.matchAll(/\[[^\]]*\]\(([^)\s]+)\)/g)) { From d4dae0fb586b43c2c7eeebad25a8d1697798df39 Mon Sep 17 00:00:00 2001 From: Petr Glaser <12586960+BleedingDev@users.noreply.github.com> Date: Sun, 20 Sep 2026 19:23:24 +0200 Subject: [PATCH 2/2] docs: handle lost browser state and consolidate the design Distinguish validated live continuation from reconstruction, restart, and unrecoverable loss. Cover half-filled forms, policy-limited recovery inputs, same-tab reloads, unknown autosave/submission effects, and preserving crash failures in tests. Replace five proposal documents with one design and implementation checklist. Repair guide links and documentation-link validation. Runtime, operational skills, dependencies, and SHA-pinned Actions remain unchanged. --- AGENTS.md | 2 +- README.md | 6 +- VISION.md | 50 -------------- docs/agent-contract.md | 124 ---------------------------------- docs/design.md | 131 ++++++++++++------------------------ docs/fallback.md | 2 +- docs/implementation-plan.md | 119 -------------------------------- docs/learning.md | 65 ------------------ docs/testing.md | 2 +- test/docs.test.mjs | 2 +- 10 files changed, 48 insertions(+), 455 deletions(-) delete mode 100644 VISION.md delete mode 100644 docs/agent-contract.md delete mode 100644 docs/implementation-plan.md delete mode 100644 docs/learning.md diff --git a/AGENTS.md b/AGENTS.md index 59b0abc..0235171 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -2,7 +2,7 @@ Read `package.json` for commands and `README.md` for supported behavior. Run `npm run check` before committing. Browser integration tests are documented in the README. -For orchestration or protocol changes, read [the system design](docs/design.md) and [agent contract](docs/agent-contract.md). For reusable site knowledge, read [learning and reuse](docs/learning.md). Follow [the implementation plan](docs/implementation-plan.md) for delivery gates. These describe proposed behavior; advertise it in operational guides and skills only after the corresponding tests pass. [VISION.md](VISION.md) indexes the design. +For planned runtime changes, read [the design and checklist](docs/design.md). Keep proposals out of operational skills until implemented. A saved run is not saved browser state; distinguish validated continuation from reconstruction, restart, and unrecoverable loss. Keep design decisions in that one document, not parallel process documents. CLI and MCP call the same Effect v4 service. Keep the Jev action loop upstream; `worker/bridge.py` owns only session handoff, assertions, and the process protocol. Pin upgrades as one reviewed change with both lock files updated. diff --git a/README.md b/README.md index f6032ae..fe51f89 100644 --- a/README.md +++ b/README.md @@ -116,11 +116,9 @@ Jev's MVP lacks support for some frames, shadow roots, canvas, uploads, popup ta The name is not a benchmark claim. CI exercises real Chrome with scripted model decisions. Live-model reliability, production authentication, and performance on your applications remain to be measured. Patchright and visual-agent frameworks are not included. -## Design and next release +## Planned work -The proposed design centers work on a resumable run with checkpoints, evidence, and explicit uncertainty. It adds scoped verification, requested data extraction, native image delivery, bounded Midscene fallback, and reviewed recipe reuse. These capabilities are not implemented yet. - -Start with [VISION.md](VISION.md), then follow the task-specific design links there. The [implementation plan](docs/implementation-plan.md) lists delivery gates and evaluations. Current instructions and examples above remain the source of truth for using this release. +The [design and implementation checklist](docs/design.md) covers agent ergonomics, scoped/visual fallback, and recovery when live browser state is lost. These are proposals, not current capabilities. A saved run must never imply that a crashed page or half-filled form can be restored. ## Develop diff --git a/VISION.md b/VISION.md deleted file mode 100644 index ca8b28f..0000000 --- a/VISION.md +++ /dev/null @@ -1,50 +0,0 @@ -# An agent should inherit the work, not reconstruct it - -Status: proposed direction. The [README](README.md) documents the shipped CLI. The [implementation plan](docs/implementation-plan.md) separates that baseline from the work below. - -fastest-e2e should let an agent answer four questions after any interruption: What was requested? What actually happened? What remains uncertain? What can I safely do next? - -The unit of work is a run, not an engine call or a browser tab. A run keeps the task, browser identity, checkpoints, evidence, remaining budget, and authorized scope together. Changing engines or coding agents should not require reconstructing them from a conversation. - -```mermaid -flowchart LR - Intent["Task and expected outcome"] --> Run["Run with checkpoints"] - Run --> Control["Cheapest suitable execution"] - Control --> Evidence["Fresh observations and checks"] - Evidence --> Run - Run --> Result["Answer or test report"] - Result -. "reviewed, scoped reuse" .-> Recipe["Recipe"] - Recipe -. "fresh preconditions" .-> Control -``` - -## What the agent should experience - -A read request returns the requested value with its source, not just a tab ID. A test request returns evidence against the chosen requirements, not a model's completion claim. An unsupported widget offers a continuation on the same run. An uncertain submission offers inspection, not another submission. - -Routine steps stay inside the executor. The coding agent spends its reasoning on interpreting requirements, selecting useful checks, and diagnosing exceptions. It receives enough context to make those decisions and fetches more only when needed. - -A second attempt should cost less because it can reuse validated navigation and targeting knowledge. It must still observe current state and verify the requested result. Learned knowledge never grants permission, supplies today's answer from yesterday's page, or relaxes a test expectation. - -## Optimize for verified work - -First preserve the intended account, authorized changes, and test meaning. Then maximize correctly completed tasks and correctly identified defects. Reduce total time, model calls, output tokens, and repeated exploration within those constraints. - -A fast false pass is a failure. A cheap blocked result is also a failure when a supported, authorized continuation was available. Evaluate both. - -## Keep the system small - -Reuse Jev's policy, Browser Harness's connection, Playwright's targeting, and Midscene's visual loop. The repository owns their shared run contract and evidence. Do not build another browser driver, general workflow language, vector database, or agent framework. - -Keep two operational skills and one setup skill. Skills explain when to act and when to inspect evidence. The executable describes installed capabilities. The run carries task-specific state. Reviewed recipes carry reusable site knowledge. None of these should become a growing global prompt. - -## Read by task - -| Task | Source of truth | -| --- | --- | -| Use the current release | [README](README.md) and command help | -| Change run ownership, execution, or verdicts | [System design](docs/design.md) | -| Change what agents send or receive | [Agent contract](docs/agent-contract.md) | -| Add reusable knowledge or execution recipes | [Learning and reuse](docs/learning.md) | -| Implement or evaluate the next release | [Implementation plan](docs/implementation-plan.md) | - -The design documents describe a target, not additional working commands. Update the operational guides and skills only when the matching behavior has passed its acceptance checks. diff --git a/docs/agent-contract.md b/docs/agent-contract.md deleted file mode 100644 index 70b4198..0000000 --- a/docs/agent-contract.md +++ /dev/null @@ -1,124 +0,0 @@ -# Agent interaction contract - -Status: proposed, not current command syntax. [System design](design.md) owns run and recovery semantics. This document owns what an agent supplies, receives, and needs to remember. Delivery is tracked in [the implementation plan](implementation-plan.md). - -## Begin with the outcome - -For browser use, provide the URL, bounded goal, stopping condition, and optional requested data. For testing, also provide requirement IDs and observable expectations. Setup supplies the profile and permitted model providers once. Task-specific account, deployment, mutation scope, and cleanup expectations remain explicit. - -Accept a goal without forcing a test file, and a scenario without forcing an elaborate workflow definition. Resolve defaults locally and return their effective values. Detect input, configuration, unsupported-schema, and missing-capability errors before taking browser actions or spending model calls where possible. - -Treat the PR reference as context supplied by the coding agent. It reads repository changes and available deployment evidence through its own tools. The browser runtime does not need a GitHub integration to validate a UI. - -## Small vocabulary, progressive detail - -These are proposed extensions to existing commands. Exact flags and schemas must be finalized with contract tests before appearing in installed skills. - -| Question | CLI operation | MCP equivalent | -| --- | --- | --- | -| Can this installation do the task? | `doctor` | `browser_doctor` | -| Perform a bounded task | `run` | `browser_run` | -| Test these expectations | `test` | `browser_test` | -| What happened, or what is visible now? | `inspect --run ID --view ...` | `browser_inspect` | -| Continue remaining work | `resume --run ID` | `browser_resume` | -| Show this owned page | `screenshot --run ID` | `browser_screenshot` | -| Check saved expectations again, without replay | `verify --run ID` | `browser_verify` | -| Release task tabs | `close --run ID` | `browser_close` | - -Retain the existing target-based inspection and close commands for compatibility. A run resolves targets internally. When a run owns several tabs, return labeled target choices and require disambiguation where necessary. Never make the agent repeatedly copy a CDP endpoint or infer a target ID from a URL. - -`run` and `test` should accept structured input through files/stdin as well as common CLI arguments. Reserve stdout for final JSON; send opt-in progress to stderr or supported MCP progress notifications. No banner, provider log, or Base64 image dump belongs in text output. - -Do not add a general `execute anything` tool to the default MCP tool list. Keep trusted script execution opt-in. Offer narrow operations and whole-task execution before requiring code authoring. - -## Every response should orient the next decision - -Return a compact summary first. Include the run ID, record revision, execution state, verification state, requested answer, relevant evidence, remaining budget, and legal next actions. Summaries are derived from structured receipts and checks; they do not require an additional summarizer-model call. - -This shortened example illustrates an uncertain submission, not an implemented response schema: - -```json -{ - "schemaVersion": 2, - "runId": "r_settings_01", - "revision": 7, - "status": "blocked", - "execution": { - "state": "blocked", - "reasonCode": "mutation_uncertain", - "lastCheckpoint": "display_name_entered" - }, - "verification": { "status": "not_run", "pending": ["saved_after_reload"] }, - "mutation": { "state": "uncertain", "actionId": "a_save_04" }, - "summary": "Save was dispatched; its outcome is unknown. The task tab is retained.", - "budget": { "actionsRemaining": 12, "modelCost": { "kind": "unknown" } }, - "evidenceRefs": ["e_before_save"], - "nextActions": [ - { "operation": "inspect", "arguments": { "runId": "r_settings_01", "view": "page" } } - ] -} -``` - -Next actions are typed suggestions computed from state, capabilities, and policy. They are not website-authored instructions or additional authorization. Validate all preconditions again when invoked. Return enough context with an error to use the suggested remedy without searching a stack trace. - -## Observation is a budgeted query - -Extend `inspect` with summary, page, checks, and history views. Summary reads the durable record and makes no browser or model calls. Page reads live state. Checks and history retrieve recorded evidence, with explicit capture times and a clear distinction from fresh inspection. - -Default page output contains the current URL, title, task-relevant visible controls/text, frame boundaries, pending dialog, and supported locator hints. Preserve accessible labels and exact observed values when they matter. Report empty, unavailable, stale, and truncated as different conditions. - -Provide a small default output limit, explicit truncation, and stable cursors bound to an observation or event sequence. Prefer a focused frame/region query to repeatedly dumping the whole page. Changes since an earlier observation must identify their base; a new agent can always request a full focused view. Never truncate a failed check or a required extraction field silently. - -Expose which observation strategies are available and which are not. A missing element in an unsupported frame is unknown, not an observed absence. An inspection may reveal account identity through normal UI; a profile directory name does not prove the active tenant or role. - -### Requested answers, not tab IDs alone - -Allow an optional `extract` description and supported structured-output schema on browser tasks. Return requested values with observation references and the method used: direct DOM read or model interpretation. Type validation is not factual verification. Failed or incomplete extraction returns a field-level error or partial result, not invented values. - -Prefer direct control/text extraction when a scoped locator is known. Use a visual/model query only when necessary and charge it to the run. A task asking for a display name should normally finish in one public call with that value and evidence, rather than requiring `run`, an unrelated inspection, and a separate interpretation step. - -### Images - -MCP screenshots return an actual image content block plus structured metadata. CLI returns a local artifact path and the same metadata. Resource references need a supported retrieval route; a path alone is not an image for a remote MCP client. Provide a bounded direct-content route when that client cannot read local files. - -Metadata includes observation ID, browser/target identity, timestamp, viewport size in CSS pixels, encoded image size, crop origin, and scale. Coordinate operations are bound to that observation, not unexplained screen coordinates. Prefer fresh locator-based control where possible. - -Keep viewport images the default. Full-page captures, video, and verbose traces are explicit requests or configured failure diagnostics. Midscene's own image-reading ability must not depend on the outer coding agent's model supporting vision. A client without image viewing still receives a useful textual summary and an honest visual-verification limitation. - -## Result interpretation - -Preserve the legacy top-level `done`, `passed`, `failed`, and `blocked` values and exit codes. Introduce versioned structured details additively; reject incompatible requests rather than silently ignoring important fields. - -| Situation | Meaning | -| --- | --- | -| Executor reports completion without checks | `done`, unverified | -| Required execution completes and all declared checks pass | `passed`, with evidence type per check | -| A declared check mismatches after completed execution | `failed`; product attribution still requires diagnosis | -| Execution, extraction, required scope, or verification cannot finish | `blocked`, with partial evidence preserved | -| Case never executed or target change not deployed | Untested coverage in the report, not a fabricated runtime pass | - -Distinguish goal outcome from cleanup outcome. A successful feature check with failed cleanup reports both and identifies leftover production data. Preserve evidence collected before cleanup; checking the cleaned-up state does not replace it. - -Expected blockers have stable codes such as `auth_required`, `unsupported_scope`, `ambiguous_target`, `mutation_uncertain`, `budget_exhausted`, and `session_changed`. Redacted diagnostics carry an evidence reference where useful. Malformed MCP requests use protocol errors; task failures remain interpretable tool results. Transport errors must not cause automatic mutation retries. - -## Survive context loss and concurrent agents - -A replacement coding agent should need only the run ID and configured home. `inspect` supplies the original task, checkpoint, observed changes, unresolved uncertainty, pending checks, and safe continuation options. It does not replay a conversational transcript. - -Resume accepts an expected run revision and refuses stale competing requests. The runtime also revalidates live preconditions. Two agents resuming one run cannot dispatch in parallel. Cancellation leaves a checkpoint or an explicit uncertain boundary, not a success-looking empty response. - -Long work may later need start/status/cancel delivery to survive client request timeouts. Keep it out of the initial API unless client tests demonstrate that need. A transport disconnect must never silently create unbounded detached work. - -## Setup and skills - -`doctor` should return a capability/readiness matrix by engine and operation, including disabled capabilities and remedies. Separate installed, configured, key-present, provider-tested, and browser-connected. Do not call every provider on every task. Live credential validation is explicit and its resource use visible. - -Operational skills retain their current commands until this contract ships. Then `browser-use` teaches run/inspect/resume decisions, and `browser-test` teaches requirement-to-evidence mapping and failure interpretation. Setup teaches capability readiness and client registration. Recipes and troubleshooting are disclosed only when relevant; none belong in always-loaded skill descriptions. - -Test CLI/MCP behavior against the same fixtures. Test actual skill discovery and image delivery per supported client. Do not claim one universal installation path or confuse generic shell compatibility with validated Codex, Claude Code, OpenCode, Pi, or OMP integration. - -## Design references - -[Anthropic's tool-design guidance](https://www.anthropic.com/engineering/writing-tools-for-agents) motivates task-shaped operations, meaningful results, bounded output, and evaluation with agents. Our concrete response contracts above are proposals to test, not guarantees derived from that guidance. - -The [MCP tools specification](https://modelcontextprotocol.io/specification/2025-06-18/server/tools) defines structured content, images, and errors. Validate the selected Effect transport against those representations. Follow [writing-for-agents](https://github.com/mattpocock/skills/blob/main/skills/productivity/writing-for-agents/SKILL.md) for conditional references and single-source documentation. diff --git a/docs/design.md b/docs/design.md index 724edbb..b9bd386 100644 --- a/docs/design.md +++ b/docs/design.md @@ -1,121 +1,74 @@ -# System design +# Agent workflow design -Status: proposed. Baseline reviewed at `b3b1500`. Read [the vision](../VISION.md) for the objective, [the agent contract](agent-contract.md) for interaction details, and [the implementation plan](implementation-plan.md) for delivery gates. +Proposed, not shipped. [README](../README.md) documents the current CLI. This document replaces the separate vision, agent-contract, learning, and implementation-plan documents. -## Current baseline +## One record of the work, not a browser snapshot -The [runtime](../src/runtime.ts) runs a pinned Python worker per request. The [host](../src/host.ts) binds it to the configured Chrome generation and locks one configured home. The [worker](../worker/bridge.py) registers task tabs and checks the top-level document. Runs return a tab ID and status, not a durable execution record or extracted answer. Browser Harness fallback is trusted script execution. There is no autonomous Midscene integration or native MCP screenshot result yet. - -Those are useful building blocks. They leave progress, recovery decisions, and evidence interpretation in the outer agent's conversation. The target below moves durable facts into the runtime without moving general product reasoning there. - -## Ownership follows the task +A run records intent, progress, evidence, uncertainty, and next actions across engine switches and agent context loss. **Persisting a run does not persist the application's live state.** ```text -Task specification What outcome, scope, and evidence are required? - Run What has happened and what remains? - Checkpoint What can be continued from this observed state? - Scoped operations Observe, act, extract, and check on owned targets. - Executor adapter Jev, Playwright recipe, or Midscene. - Session lease Exact profile, Chrome generation, and task tabs. +Task + expectations + Run: attempts, evidence, recovery status, remaining budget + Executor: Jev / scoped Playwright / Midscene + Owned Chrome profile, tab, and live document ``` -Evidence refers back to a run, checkpoint, target, and requirement. Recipes are reusable procedures selected by the run, not owners of the browser or definitions of success. +CLI and MCP share Effect services and upstream engines. A local journal records permitted intent and dispatch/completion receipts. Flush before dispatch; incomplete receipts mean uncertainty, not permission to retry. Deduplication and profile-scoped leases prevent competing dispatch, not exactly-once website effects. -| Owner | Owns | Does not own | -| --- | --- | --- | -| Coding agent | Requirement interpretation, case selection, deployment evidence, diagnosis | Low-level control between every routine click | -| Effect runtime | Validated task specification, run state, budgets, session lease, routing, result assembly | A replacement browser engine or model reasoning trace | -| Executor adapter | Execution through its upstream library, progress receipts, cancellation | Permission changes, success criteria, another profile | -| Verifier | Fresh checks with explicit scope and evidence type | Actions that repair the application to satisfy a check | -| Recipe store | Reviewed, parameterized knowledge and applicability | Credentials, permissions, authoritative current page state | +## Recovery after interruption -Use Effect Schema for shared contracts and Context services for these responsibilities. Keep CLI and MCP as transport adapters to the same handlers. Keep a single package until an independently deployable component earns a split. +Stop dispatch. Distinguish client disconnection from page/browser crash. Revalidate account, browser, owned target, document generation, and task-specific live state. An unchanged URL or tab ID is insufficient: the same tab can reload and lose its form. Unproven continuity is unknown. -## A run outlives an invocation +| Recovery | Required evidence and behavior | +| --- | --- | +| **Resume** | The relevant live state survived and there is no unresolved mutation. Continue remaining work without reload or replay. | +| **Reconstruct** | State was lost, but a validated UI route and permitted inputs or an application draft are available. Rebuild only the missing state, then verify it. This is a new attempt, not restoration of the old document. | +| **Restart** | Reconstruction is unavailable, but the workflow can safely start again. Start a linked attempt from the beginning within existing authorization and remaining budget. | +| **Blocked** | Inputs are unavailable or must not be retained, reconstruction is unsafe, or an earlier effect is uncertain. State exactly what was lost and what needs user input or reconciliation. | -Create a run ID and persist its validated intent before the first browser action. Keep the current browser-generation ID, owned targets, expected outcomes, requested extraction, attempts, checkpoints, mutation receipts, budget usage, and evidence references under that run. +Unknown effects take precedence over reconstruction/restart. A crash during Save, navigation, typing with autosave, or a file upload may leave a server-side effect even when the UI vanished. Reconcile through available application UI before repeating it; an empty form or missing toast is not proof nothing happened. If evidence is inconclusive, stop. Never use a new engine as an implicit retry. -Use a local append-only event file and atomically replaced summary. Start with files, not a workflow service or database. Record observable decisions and action receipts, not hidden model reasoning. Events need sequence numbers and input/schema versions so summaries can be rebuilt and incompatible records rejected explicitly. +A fresh document invalidates handles, coordinates, and live checkpoints. Bind replacement tabs explicitly to the configured profile; never discover another browser. Stop old adapter control before recovery, including same-profile/two-home contention. -A paused or interrupted run remains inspectable after the CLI exits or the coding agent loses its context. It is resumable only if the original live browser state can be validated. Browser restart preserves the record, not a promise of restoring in-memory tab state. +### Half-filled forms -Journal before an action can be dispatched and record its completion afterward. If a process dies between those writes, mark the action uncertain. Observation may resolve uncertainty; replay cannot. Request deduplication prevents duplicate runtime dispatch for an identical request ID. It cannot promise exactly-once side effects on a website. +Before an action, retain only policy-permitted input intent or a secure reference to it; distinguish intended values from values actually observed in the page. If recovery is enabled, explicitly allowlist the non-secret fields needed to reconstruct the task. Prefer user-supplied inputs and application drafts over copying the page. Verify draft persistence through the UI rather than assuming autosave worked. -A request ID is scoped to the configured home and canonical input digest. Reusing it with different input is a conflict. A duplicate completed request returns its recorded result; a duplicate active request returns the run handle, not another execution. +Capture allowed recovery data before failure, with its source and last observation. A crash cannot supply a final snapshot, and edits after the last capture may be lost. Keep recovery payloads outside transcripts and git, with restricted local access, expiry, and deletion. No blanket DOM, storage, form, or screenshot dumps. Do not retain passwords, one-time codes, authentication material, or file contents as recovery payloads; use existing credential mechanisms or request local re-entry/reselection. Inputs forbidden from storage remain unavailable after a crash. -## Checkpoints and exclusive control +Example: reopen a wizard and re-enter retained field values through normal UI controls, re-resolving targets and verifying each rebuilt step. Do this only if repeated input/navigation is known safe; ordinary fields may autosave or trigger other effects. Do not recreate hidden JavaScript state or inject values into application stores. Files, opaque editor state, expired tokens, and unrecorded edits may require fresh user input or a restart. There is no generic lossless browser restore. -A checkpoint identifies the reached UI state, completed subgoal, unchanged remaining goal, evidence, target lineage, and budget consumed. Prefer semantic boundaries such as an opened dialog or a confirmed save. Do not require the outer agent to author a click-by-click plan. +Report the recovery choice, evidence, lost/available inputs without their values, unresolved effects, and next action. Share budgets across attempts and bound retries; stop repeated crashes. -Hold one exclusive lease per canonical profile and Chrome generation during an execution segment, including an engine handoff. The existing home lock alone does not protect the same profile configured under two homes. Detect and reject that collision. Releasing an invocation's lease does not transfer ownership of its tabs. +### Keep test failures visible -Between invocations, a person may change a tab. Resume validates browser identity, target ownership, checkpoint preconditions, and new observations. A stored revision is a journal version, not proof that the live DOM is unchanged. Re-resolve locators and reject stale image coordinates after navigation, scrolling, layout changes, or an invalid screenshot reference. +Preserve interrupted/failed attempts and unknown crash causes. A later pass is a recovered/retried pass, not uninterrupted success. When draft persistence or crash recovery is under test, refilling the form bypasses the requirement: retain the failure and test reconstruction separately. -Jev relinquishes control before another adapter acts. A popup may join the run only through verified opener lineage and allowed navigation context. Never choose a page by array order or URL alone. Detaching an adapter must leave Chrome and unrelated tabs open. +## Agent control and execution -## Choose the cheapest capable executor +Return requested data with evidence, progress, uncertainty, budget, and typed next actions. Summary inspection needs no model calls. Separate focused live inspection from bounded recorded history; label stale, missing, truncated, and model-interpreted data. Extraction schema validity is not factual verification. -Selection uses validated task requirements, current capabilities, live state, and approved recipe applicability. The policy is conditional, not a mandatory sequence of paid trials. +Extend existing operations with run inspection/recovery, screenshots, and checking saved expectations. Keep exact schemas in code and preserve existing scenarios/exit codes. Separate execution, verification, recovery, and cleanup outcomes. Readiness distinguishes installed/configured from actually tested without routine provider calls. -```text -if a reviewed UI recipe matches and its preconditions hold: - run the recipe and verify fresh evidence -else if a known interaction is directly supported by scoped Playwright operations: - use those operations without a visual model -else if the requested goal is supported by Jev: - run Jev -else if vision is configured, allowed, and suitable: - run Midscene on the same owned target -else: - return the precise missing capability and safe next actions -``` +Use reviewed UI recipes when applicable, scoped deterministic operations for known interactions, Jev for suitable autonomous work, and Midscene for visual tasks. Preserve explicit `auto`/`jev`/`vision` selection. Lazy-load fallback; successful Jev paths need no screenshots or vision calls unless requested. -Ordinary iframe or open-shadow-root targeting does not itself require vision. Use Midscene for rendered controls whose useful identity is visual, or when a bounded semantic workflow needs its visual planner. Never infer capability support solely from the presence of an iframe or canvas somewhere on the page. +Prove exact-target CDP handoff without state changes. Normal fallback must not require arbitrary scripts; vision requires explicit provider/data-sharing configuration. Disable popup/navigation and native-select rewriting during tests. Open-root/frame checks must preserve legacy matching semantics; unsupported scope is unknown. Closed-root visual control does not imply DOM assertions. Freeze expectations and label visual judgments; failed checks never trigger attempts to make them pass. -`auto`, `jev`, and `vision` remain the proposed public policies. `auto` may skip a predictably unsuitable engine and may use a validated recipe. An explicit engine selection does not silently switch engines. Record the selected executor and reason. Load Playwright/Midscene only for operations that need them. +MCP returns actual image content, CLI a local path, both with capture time, target/document, dimensions, crop, and CSS scaling. Do not assume remote clients can read server-local paths. Midscene must work without outer-agent vision. -The adapter must expose enough supported hooks to meter steps and report dispatch uncertainty. If that cannot be demonstrated for a pinned upstream version, limit it to bounded calls and disable automatic continuation across ambiguous boundaries. Do not invent reliable classification by parsing a model's prose. +Settle cancelled workers before releasing control. Budgets span engines/attempts; unknown cost is not zero. Read-only goals are guidance, not enforced isolation. Website content cannot change authorization. Keep evidence local/minimal; do not promise complete redaction. -### Recovery rules +## Implement and prove -| Observation | Response | +| Increment | Required tests | | --- | --- | -| Unsupported interaction, no unresolved mutation, suitable authorized engine available | Continue the remaining subgoal on the same run within the remaining budget | -| Lost response after a potentially mutating action | Mark uncertain; inspect before offering continuation | -| Assertion mismatch after completed execution | Preserve failure; diagnose without another agent trying to make it pass | -| Missing provider, expired login, unsupported scope, or exhausted budget | Return a typed blocker and a concrete remedy | -| User cancellation or lost browser identity | Stop dispatch; never escalate to another engine | - -A model's low confidence can justify more observation. It is not calibrated proof that replay or escalation is safe. Automatic recovery needs an observed capability failure and satisfied preconditions. - -## Verification is separate from execution - -Execution completion, requirement verification, and mutation certainty are separate fields. Finishing a flow is not evidence that it worked. An assertion mismatch is an observation, not automatically proof of a product defect. - -Freeze requirement IDs and expected results before the actions they evaluate. Checks refer to checkpoints and explicit frame/locator scopes. Evidence records the expected/actual pair, capture time, URL, target, observation ID, and method. Record deployment identity as verified, user-supplied, or unknown, with its source. An undeployed change is untested; an unknown build does not become verified because a familiar heading appeared. - -Keep existing checks backward compatible, including exact value matching, text inclusion, visibility requirements, and DOM-count behavior. Introduce frame-aware and open-shadow-root checks explicitly. Do not silently replace CSS semantics with Playwright's broader shadow-piercing or text normalization. No matching support means blocked or untested, not an absent-element success. - -Visual assertions are explicit expectations evaluated from images. Label their evidence as model-evaluated. Schema-valid extraction also proves only shape, not truth. A whole test is verified only to the extent of its supplied checks and inspected requirements. Aggregate reports expose untested and blocked coverage rather than reducing everything to a green badge. - -Recovery appends an attempt. Re-verifying a later state does not overwrite the original verdict or prove that the original action worked. A change to the goal, expected result, account, or authorization creates a new specification revision or a linked new run. - -## Resource and data boundaries - -One run shares action, model-call, active-time, and supported cost budgets across engines and resume calls. Per-invocation wall deadlines still apply. Human review time is separate from active execution time. Resume does not reset consumed budgets. Reserve time for final observation and orderly cancellation before spending the execution allowance. - -Interruptible, scoped subprocesses own adapters that cannot cancel individual operations reliably. Cancellation must stop new dispatch and settle the worker before releasing its lease. A timed-out promise alone is not cancellation. No timeout can roll back a submitted production operation. - -Cost fields distinguish measured, estimated, and unknown usage. Never treat unknown as zero. Promise a hard monetary cap only where provider request bounds and metering make it enforceable; otherwise reject that requirement or offer explicit call/token limits. - -Local configuration owns profile and provider selection. Requests can narrow permissions, not choose a different profile or widen provider egress. A natural-language read-only goal is not a technical read-only guarantee. Browser navigation itself may have side effects. State exactly which controls are enforced and which rely on instructions. Fallback scripts remain trusted local code, not a sandbox. - -Keep screenshots, traces, and account data local by default, with bounded retention and explicit export. Mask known secrets before collection where supported, minimize captures, and report redaction coverage honestly. Prefer selectors and short evidence values to full page dumps. Never promise complete screenshot redaction. Page content is untrusted input and cannot change tools, expectations, permissions, or learning policy. - -## Upstream integration constraints +| Run and recovery records | Kill the caller, disconnect CDP, crash the renderer, reload the same tab, and restart Chrome. Test known, unsaved, stale, forbidden-to-store, and missing inputs; expired auth; safe reconstruction and explicit inability to recover. | +| Safe action boundaries | Crash before dispatch, after dispatch but before receipt, and after a server effect. Include autosave and a submitted Save with a lost response. No duplicate uncertain writes, wrong-profile actions, or concurrent resume; budgets never reset. | +| Observation and fallback | Scoped frame/shadow assertions, partial extraction, actual MCP images, stale coordinates, and same-tab visual handoff. Recreated forms must have fresh evidence. Preserve crashes and original verdicts, including a deliberately broken persistence test. | +| Agent usability and reuse | A fresh agent needs only the run ID to diagnose the interruption. Compare total verified success, false passes, time, calls, and bytes. Test real clients; distinguish controlled model decisions from opt-in live-model trials. | -[Playwright CDP attachment](https://playwright.dev/docs/api/class-browsertype#browser-type-connect-over-cdp) is lower fidelity than its native protocol. Validate exact-target attachment and state preservation with the selected version. Evaluate `noDefaults` where supported rather than silently applying context overrides. [Playwright locators](https://playwright.dev/docs/locators#locate-in-shadow-dom) support open shadow roots, not closed-root DOM targeting. +Recovery remains explicit until tests establish safe automatic cases. Missing upstream hooks cannot produce reliable action receipts. CI uses controlled pages; production validation remains authorized and UI-only. -[Midscene's Playwright integration](https://midscenejs.com/integrate-with-playwright) accepts existing pages. Disable defaults that rewrite popup navigation or native select rendering for E2E testing. Screenshot-based control of a rendered closed-root widget does not imply DOM assertion support inside it. +Later, reuse reviewed procedures only when fresh account/origin, permission, and UI preconditions hold; quarantine drift. Never reuse old answers or permissions. Existing scripts and a small manifest suffice; measure savings before adding learning machinery. -These documents are design inputs, not proof that our adapters work. Pin implementation versions and pass [the delivery gates](implementation-plan.md) before advertising a capability. +Keep the three skills; update them only as behavior ships. This file owns design/checklist, while code, tests, and operational guides own usage. No parallel process documents. diff --git a/docs/fallback.md b/docs/fallback.md index 414c451..9a8cf53 100644 --- a/docs/fallback.md +++ b/docs/fallback.md @@ -1,6 +1,6 @@ # Browser Harness fallback -This page documents the current scripted fallback. Autonomous vision continuation and native MCP image delivery are proposed in the [implementation plan](implementation-plan.md), not available in the current runtime. +This page documents the current scripted fallback. Autonomous vision continuation and native MCP image delivery are proposed in the [design and checklist](design.md), not available in the current runtime. Use `fastest-e2e harness --target --file script.py` to continue an owned task tab. It uses the same dedicated browser and session lock as Jev. The target comes from the earlier result's `targetId`; inspect partial changes before retrying a mutation. diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md deleted file mode 100644 index 78a105e..0000000 --- a/docs/implementation-plan.md +++ /dev/null @@ -1,119 +0,0 @@ -# Implementation plan - -Status: proposed. No stages below are implemented by this documentation change. Baseline is `b3b1500`. This plan supersedes the earlier chat-only fallback sequence while retaining Jev, same-profile CDP, Playwright, and Midscene as the intended integrations. - -[System design](design.md) owns behavior. [Agent contract](agent-contract.md) owns requests and responses. [Learning and reuse](learning.md) owns recipe policy. This file owns sequencing, acceptance gates, and evaluation. Keep installed skills and examples on the current CLI until their matching gate passes. - -## What changes from the earlier plan - -| Earlier emphasis | Revised decision | -| --- | --- | -| Add a second autonomous engine | Establish a durable run and evidence contract before engine handoff | -| Use vision for unsupported DOM | Use scoped deterministic operations for supported frames/open roots; reserve vision for visual work | -| Return status and target | Return the requested answer, observed progress, uncertainty, evidence, and legal next actions | -| Resume a tab | Resume a validated checkpoint with shared budgets and immutable failed attempts | -| Measure runtime | Measure time and total resources to correct, verified completion, including recovery | -| Solve each task afresh | Add reviewed, scoped recipes only after fresh verification and safety gates exist | - -```mermaid -flowchart TD - G0["G0: attachment and upstream-hook proofs"] --> G1["G1: durable run and response contract"] - G1 --> G2["G2: scoped observation, extraction, verification, images"] - G2 --> G3["G3: bounded Midscene handoff"] - G3 --> G4["G4: CLI/MCP and real-agent evaluations"] - G4 --> G5["G5: measured recipe reuse"] -``` - -These are successive usable increments. G1 and G2 should improve Jev-only use without requiring a vision provider. G5 is not a prerequisite for shipping the fallback repair. - -## G0. Prove integration boundaries - -- [ ] Pin compatible Playwright/Midscene versions and keep Effect/core adapter versions matched. Verify APIs against installed source, not examples on an unpinned branch. -- [ ] Attach to the registered target in existing dedicated Chrome. Test duplicate URLs, unsaved input, storage, focus/media settings, nested frames, and two independent profiles. -- [ ] Demonstrate disconnection/cancellation without closing shared Chrome. Check same-profile/two-home contention and exact owned-popup lineage. -- [ ] Identify supported pre-dispatch, completion, and cancellation hooks in Jev and Midscene. Document what can and cannot be observed without rewriting their loops. - -Exit gate: repeatable, secret-free integration tests show correct target selection and preserved state. A missing receipt or cancellation hook blocks automatic handoff at that boundary. It does not justify speculative recovery or an unreviewed engine rewrite. Deliver the proof and its limitations before committing to automatic mode. - -## G1. Make runs durable and resumable - -- [ ] Add the versioned run record, event journal, semantic checkpoints, request deduplication, and summary projection through Effect services. -- [ ] Persist intent before browser actions and dispatch/completion receipts around supported action boundaries. Preserve unknown outcomes on interruption. -- [ ] Use profile/generation-scoped leases with explicit release and stale-owner recovery rules. Reject stale expected revisions and competing resume requests. -- [ ] Extend result details and inspection while preserving current input formats, statuses, target-based commands, and exit codes. -- [ ] Implement resume of validated live state, cumulative budgets, and local record inspection after browser disconnect. New browser generations require explicit new execution, not false continuation. - -Exit gate: kill the caller after an action dispatch. A fresh process can explain the last checkpoint and uncertainty from the run ID alone. Repeating the same request cannot dispatch twice. Same-ID/different-input and concurrent resume requests are rejected. No test passes from an empty response or lost worker. - -## G2. Make observation and verification usable - -- [ ] Add focused live inspection, labeled frame/locator scopes, exact evidence references, and bounded/paged history. Keep recorded observations distinguishable from live state. -- [ ] Add optional structured extraction to browser tasks. Direct reads precede model interpretation where suitable. Incomplete or schema-invalid extraction stays explicit. -- [ ] Add scoped Playwright checks for open roots and nested/cross-origin frames. Preserve legacy selector and matching semantics. Detect unsupported scopes instead of treating them as absence. -- [ ] Add `verify` on saved expectations and checkpoints. Preserve original attempts, attribution uncertainty, and separate cleanup results. -- [ ] Add screenshot artifacts with identity, capture time, coordinate scale/crop metadata, bounded retention, CLI paths, and native MCP image content. -- [ ] Add typed stop reasons and capability/readiness descriptions with actionable remedies. Zero provider calls for ordinary setup inspection. - -Exit gate: a read task returns its requested value and evidence in one public execution call. Positive and intentionally broken pages produce correct verdicts inside nested frames and open roots. A closed-root DOM check is unsupported, never an accidental pass. Test strict selector ambiguity, hidden matches, stale observations, missing evidence, extraction errors, unsupported clients, and screenshot size/scale limits. - -## G3. Add bounded autonomous visual continuation - -- [ ] Connect Midscene to the run's validated page and remaining subgoal. Share ownership, deadlines, mutation receipts, and budgets with Jev. -- [ ] Implement `auto`, `jev`, and `vision` policies plus the recovery table in the system design. Normal visual use must not require arbitrary local script execution. -- [ ] Keep explicit vision configuration and provider/data-egress consent. Disable UI-altering defaults for popup navigation and native-select rendering. -- [ ] Add explicit visual assertions with image evidence and model-evaluated labeling. Keep them separate from deterministic checks. -- [ ] Make cancellation stop adapter dispatch before another executor or caller can acquire the lease. Keep screenshots off successful Jev-only paths unless requested. - -Exit gate: Jev reaches an unsupported visual control, Midscene continues the same run without replaying prior work, and fresh checks establish the result. A timed-out save, failed assertion, missing provider, user cancellation, or browser mismatch does not trigger automatic retry. Controlled tests prove routing and transport; opt-in live tests separately establish visual-model behavior. - -## G4. Evaluate with agents, then ship the guides - -- [ ] Run identical scenario and failure fixtures through CLI and MCP. Include structured errors, native images, cancellation, and a client that cannot read server-local files. -- [ ] Give fresh coding agents only the installed skills and a task. Record unnecessary tool calls, missing information, incorrect actions, repeated setup, and manual interventions. -- [ ] Test context loss and replacement-agent takeover from a run ID. Test full-state and incremental observation paths, not only a prepared happy-path prompt. -- [ ] Verify skill discovery and relevant tool/image capabilities separately in each supported client. Publish tested, partial, and untested status for Codex, Claude Code, OpenCode, Pi, and OMP. -- [ ] Update the README, setup/fallback/testing references, and existing three skills only for shipped behavior. Generate schemas/help from shared contracts; keep troubleshooting and recipes out of global skill descriptions. - -Exit gate: the documented fresh-install, read, test, inspect, resume, and fallback examples work. A text-only outer agent can delegate a visual task without pretending it saw the screenshot. Test reports identify the deployment evidence and do not claim unexecuted requirements passed. Limit the release's compatibility claims to combinations actually exercised. - -## G5. Accumulate useful procedures - -- [ ] Add explicit local candidate creation, manifest review, promotion, versioning, and quarantine using existing scenario/script formats. -- [ ] Select only reviewed recipes with fresh preconditions and current task authorization. Never use saved answers or historical behavior as the current test oracle. -- [ ] Test wrong-origin/account/role rejection, changed UI, stale coordinates, ambiguous selectors, recipe drift, imported hostile instructions, and sanitized export. -- [ ] Compare repeated tasks against the no-reuse baseline, including recipe creation, review, and maintenance costs. Retain only measured improvements. - -Exit gate: a scoped repeated task needs fewer decisions or model calls while still using fresh evidence. A stale or incompatible recipe is rejected before unsafe action. Disabling reuse leaves the ordinary engine paths functional. No automatic rewrite of skills or export of private traces occurs. - -## Evaluation protocol - -Use a small local test application to validate the runner's browser mechanics. Those controlled fixtures are for this repository's tests; they do not introduce an API-setup requirement for real user applications. Production evaluation uses only approved UI interactions, permitted accounts, and explicitly agreed cleanup. - -Keep suites distinct: - -| Suite | Proves | Does not prove | -| --- | --- | --- | -| Contracts and controlled adapter decisions | State transitions, routing, budgets, deterministic evidence and transport | Hosted model reasoning quality | -| Real Chrome integration | Exact-target attachment, input, scopes, screenshots, lifecycle | Every browser/version/site combination | -| Opt-in live-model tasks | End-to-end behavior of configured models on chosen cases | Universal reliability or production suitability | -| Approved production journeys | Target application's actual account/session/UI behavior | Correctness on unrelated accounts or deployments | - -Evaluate basic reading, forms and persistence, same/cross-origin nested frames, open/closed-root widgets, canvas, new tabs, expired auth, ambiguous controls, navigation during observation, and long-running pages. Include intentionally broken product behavior and uncertain submissions, not only successful tasks. - -Compare baseline Jev, the new Jev-only path, auto routing, explicit vision, and later recipe reuse. Use matched cases and equivalent fresh starting state. Repeat trials, disclose sample size, distinguish cold and warm starts, and report variability. Do not rerun production writes solely to fill a benchmark without permission. - -Record verified completion, false passes, false defect claims, blocked coverage, wrong-account/tab actions, duplicate mutations, recovery success, cleanup outcome, total elapsed time, outer-agent calls/tokens, executor calls/tokens, screenshot bytes, metered/estimated/unknown spend, and human interventions. Include attachment, verification, and unsuccessful attempts in totals. - -Correctness and scope violations fail the gate regardless of latency. For the regression fixtures, require no false passes, wrong-target actions, or duplicate uncertain submissions. For performance, record the baseline before selecting an overhead allowance; no fabricated speed target or universal ranking. A successful Jev-only run must make no vision calls. Summary generation and read-only capability discovery must make no model calls. - -## Migration and implementation shape - -Extend shared contracts and the existing runtime rather than creating separate CLI/MCP implementations. Separate orchestration, records, observation/evidence, and adapters behind small Effect services. Reuse upstream execution and locator implementations. Introduce a new source module only when its tests and responsibility warrant it; do not prebuild a monorepo or generic capability compiler. - -Keep old scenarios runnable. Add versioned fields and commands with strict validation and clear errors. Do not silently broaden old selectors, discard old results, or make a newly configured provider the default for existing installations. New automatic recovery is opt-in until its gates pass. Record schema and adapter versions with runs and recipes. - -The initial repair does not include desktop/native automation, a general workflow DSL, vector search, autonomous permission grants, automatic production-data cleanup, or a new browser profile. Native input, popup, and browser state remain part of the behavior under test. - -## Review decisions - -The review should settle the run-centered contract, the deterministic-before-vision rule, strict uncertainty handling, and staged recipe reuse. Exact default output/time budgets and client support claims remain empirical choices for the gates above. Schema/code implementation and live production evaluation are separate changes, not implied by approving these documents. diff --git a/docs/learning.md b/docs/learning.md deleted file mode 100644 index 429dd92..0000000 --- a/docs/learning.md +++ /dev/null @@ -1,65 +0,0 @@ -# Learning and reuse - -Status: proposed. This document defines how runs can make later work cheaper without making their evidence or permissions less trustworthy. It extends the [system design](design.md); delivery starts only after the core [run and verification gates](implementation-plan.md) pass. - -## Store different things separately - -| Record | Lifetime and purpose | Never treated as | -| --- | --- | --- | -| Observation/evidence | Run-local record of what was seen at a stated time | Current page state on a later run | -| Site hint | Scoped, expiring navigation or targeting advice with provenance | An executable instruction or permission | -| Recipe | Reviewed, versioned procedure with inputs, preconditions, checkpoints, and postconditions | A saved answer or a weakened test oracle | - -Start with explicit recipes and a small exact-match index. Add hints only if repeated tasks show a measurable benefit. No embedding service, global memory agent, or automatic skill rewriting is required. - -## Learn the route, verify the destination - -```mermaid -flowchart LR - Run["Verified run"] --> Candidate["Candidate recipe"] - Candidate --> Review["Review code and data scope"] - Review --> Validate["Fresh trials and negative cases"] - Validate --> Eligible["Eligible scoped recipe"] - Eligible --> Preconditions["Check applicability now"] - Preconditions --> Execute["UI execution and fresh verification"] - Execute --> History["Record success or failure"] - History -. "drift or mismatch" .-> Disabled["Quarantined recipe"] -``` - -Candidate generation and promotion are explicit maintenance operations. They are not extra model calls on every successful task. Recording minimal receipts during execution is sufficient to propose a candidate later. No background learner silently mines account data. - -Prefer existing scenario files and ordinary Playwright or Browser Harness code. Add a small manifest rather than inventing a new workflow language. Browser actions still go through the configured session and run bookkeeping. Scripts remain trusted code; validation and a manifest are not a sandbox. - -A recipe for changing a display name learns how to find the field, save, and verify persistence. Its new name comes from the current request. Its original value comes from fresh UI observation for authorized cleanup. Neither comes from the previous user's trace. - -## Applicability is explicit - -The manifest identifies the task family, allowed origin/path family, expected account role or tenant constraint, observed UI anchors, supported browser/adapter versions, input schema, source run IDs, evidence method, review status, and validity window. Store secret references when needed, not secret values or cookies. - -A known deployed revision can narrow applicability. When the revision is unknown, say so and require fresh anchors/preconditions plus a conservative validity window. One changed CSS class need not invalidate a semantic recipe; an account, role, workflow, or expected-effect change does. - -The initial promotion gate requires a reviewed candidate, successful fresh trials in the declared scope, and negative cases that reject a wrong account, ambiguous target, stale precondition, or changed workflow. The trial count must be explicit in the manifest; no small sample establishes universal reliability. Do not create production mutations just to accumulate promotion trials without authorization. - -Match candidates on explicit metadata first and return only the relevant manifest. Load executable detail after selection. Keep at most the requested candidates in the agent's context. - -## Reuse cannot hide regressions - -Before execution, recheck applicability and current authorization. During execution, stop at the first unmet precondition or uncertain action. Do not run old coordinates blindly, repair selectors silently, or fall back to a different assertion until something passes. - -If a recipe's targeting fails before an uncertain mutation, the usual run policy may continue with another eligible executor. Preserve the failed recipe attempt and classify why it was abandoned. An actual assertion mismatch is still a failed check, not grounds for self-healing the expected result. - -A learned procedure does not define product requirements. Existing checks may be reused only when they still match the current task. Changed-feature testing must select its oracle from the PR or feature requirements, not from whatever behavior the successful historical run happened to observe. - -Quarantine a recipe after drift or a safety-relevant mismatch. Review and revalidate a new version before reuse. Preserve enough version history to explain earlier runs; make deletion and export explicit. Correctly rejected stale recipes count as useful behavior, not as failed automation to hide from benchmarks. - -## Sharing and permissions - -Keep candidates local by default. Export only a reviewed, sanitized procedure and manifest into the project's repository. Never auto-commit raw screenshots, account names, page text, model requests, or trace dumps. Treat imported recipes as code changes requiring review. Page text cannot instruct the system to promote itself into durable knowledge. - -Recipes inherit the current run's authorization and budgets. They cannot add a provider, change the browser profile, grant production-write permission, or override the verifier. A trusted script with OS permissions can technically violate conventions; expose that limitation and keep its execution behind the appropriate opt-in. - -## Measure whether learning helped - -Compare fresh tasks with and without eligible recipes using the same acceptance checks. Report verified success, false passes, incorrect target selection, stale-recipe rejection, calls/tokens, total elapsed time, and review cost. Count candidate creation and maintenance when calculating savings. - -Keep reusable knowledge only where this comparison justifies it. The goal is fewer repeated decisions, not a larger store of prose. diff --git a/docs/testing.md b/docs/testing.md index 55fb458..089bcf5 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -14,7 +14,7 @@ A scenario requires `name`, `url`, `goal`, and a nonempty `checks` array. Each c `value`, `checked`, and scoped `text` require exactly one visible matching element. These are DOM checks, not pixel or layout assertions. `[aria-checked]` alone is not a native `checked` property. -Current selectors query only the top-level document; they do not enter shadow roots or iframe documents. A zero count does not establish absence inside those unsupported scopes. Scoped and visual verification are proposed in the [implementation plan](implementation-plan.md); its examples and commands are not the current scenario format. +Current selectors query only the top-level document; they do not enter shadow roots or iframe documents. A zero count does not establish absence inside those unsupported scopes. Scoped and visual verification are proposed in the [design and checklist](design.md), not part of the current scenario format. ## Choose evidence before acting diff --git a/test/docs.test.mjs b/test/docs.test.mjs index 9972805..9e6380b 100644 --- a/test/docs.test.mjs +++ b/test/docs.test.mjs @@ -39,7 +39,7 @@ test("skill names, descriptions, and local reference pointers are valid", async }); test("local documentation links resolve", async () => { - const docs = ["README.md", "AGENTS.md", "VISION.md", ...(await readdir(path.join(root, "docs"))).filter(name => name.endsWith(".md")).map(name => `docs/${name}`)]; + const docs = ["README.md", "AGENTS.md", ...(await readdir(path.join(root, "docs"))).filter(name => name.endsWith(".md")).map(name => `docs/${name}`)]; for (const file of docs) { const source = await read(file); for (const match of source.matchAll(/\[[^\]]*\]\(([^)\s]+)\)/g)) {