Bind approvals to reviewed evidence, sandbox validation network, fix flaky test

.env.example

@@ -22,7 +22,11 @@ FOREMAN_WORKSPACE_BRIDGE_URL=
 FOREMAN_WORKSPACE_ALLOWED_SCOPE=
 # JSON argv list; commands are run by Foreman in a disposable validation
 # workspace. A workspace workflow requires a source repo, bridge URL, allowed
-# scope, and at least one configured command together.
+# scope, and at least one configured command together. Commands run without
+# network access unless they set "network": true (true or false only), for
+# example [{"name":"Install","command":"pnpm","args":["install","--frozen-lockfile"],"network":true}].
+# Without the field, a pnpm/npm/yarn install (install, i, ci, add, or bare yarn) gets the
+# network and every other command does not.
 FOREMAN_VALIDATION_COMMANDS=[]
 FOREMAN_VALIDATION_TIMEOUT_MS=120000
 FOREMAN_VALIDATION_MAX_OUTPUT_BYTES=1048576

README.md

@@ -13,7 +13,7 @@ Human ↔ Planner  →  Orchestrator  →  Worker  →  checks  →  Reviewer  
 3. **Worker.** The Worker edits an isolated workspace seeded from the pinned Git base. Foreman fetches the complete snapshot, byte-compares it against the pinned commit, enforces allowed scope, and computes the exact diff.
 4. **Checks.** Foreman runs every configured validation command in a disposable workspace. An empty result (no file changes) fails the result gate even if commands pass.
 5. **Reviewer.** The Reviewer reads the verified diff and check results in read-only mode. Its recommendation is advisory; it cannot modify the result. A `request_changes` or `reject` verdict triggers a bounded Orchestrator → Worker → Reviewer correction while budget remains.
-6. **Your decision.** The "Ready for your review" panel shows a checklist, the checks pipeline (local Foreman checks and remote GitHub CI), and an Approve / Reject pair. Approval records an immutable decision bound to the evidence. It does not change Git.
+6. **Your decision.** The "Ready for your review" panel shows a checklist, the checks pipeline (local Foreman checks and remote GitHub CI), and an Approve / Reject pair. Approval records an immutable decision bound to the evidence digest you reviewed. It does not change Git.
 7. **Promote → GitHub.** Approved results can be promoted to a local commit, pushed, opened as a PR, and merged, each as a separate confirmed action.
 
 ## Quick start
@@ -70,13 +70,17 @@ The checks pipeline in the review panel shows local Foreman results alongside re
 The Worker's files are untrusted, and validation runs them (`pnpm install` lifecycle scripts, test files). Foreman therefore runs every validation command inside a [bubblewrap](https://github.com/containers/bubblewrap) (`bwrap`) sandbox by default. On Linux, install it with `apt install bubblewrap` or `dnf install bubblewrap`. If `bwrap` is missing or cannot create a sandbox, validation fails with an error and does not run on the host.
 
 - **Hidden from the command:** your home directory (`/home`, `/root`, `$HOME`), Foreman's data directory, the source repository checkout, `/tmp`, `/var/tmp`, `/run` (Docker and D-Bus sockets, the user runtime directory), `/mnt`, `/media` and `/srv` are replaced by empty temporary directories. The rest of the host is mounted read-only.
-- **Provided:** the workspace, read-write at `/tmp/workspace`; a fresh `HOME`; and an environment of only `PATH`, `LANG`, `LC_ALL`, `HOME`, `TMPDIR` and the cache variables below. The command has its own PID, IPC and UTS namespaces and no capabilities. A timeout or output overflow kills its whole process tree.
+- **Provided:** the workspace, read-write at `/tmp/workspace`; a fresh `HOME`; and an environment of only `PATH`, `LANG`, `LC_ALL`, `HOME`, `TMPDIR` and the cache variables below. The command has its own PID, IPC, UTS and network namespaces and no capabilities. A timeout or output overflow kills its whole process tree.
 - **Toolchains under a hidden directory** (nvm, pnpm in `~/.local`) are mounted back read-only: `PATH` entries, the install prefix of the command and of `node` (when it is at least two levels below the home directory), and the paths in `FOREMAN_VALIDATION_SANDBOX_RO_PATHS`. Version managers that keep state beside their shims (Volta `~/.volta`, asdf `~/.asdf`, rustup `~/.rustup`, Homebrew on Linux) need their directory listed there.
 - **Package-manager cache:** `<data dir>/validation-cache` (owner-only) is mounted read-write at `/tmp/foreman-cache`, with `XDG_CACHE_HOME`, `XDG_DATA_HOME`, `npm_config_cache`, `npm_config_store_dir` and `pnpm_config_store_dir` (pnpm 10 and 11), `YARN_CACHE_FOLDER` and `COREPACK_HOME` pointing into it, so installs do not download everything each time. It is shared by every validation and project. pnpm and npm verify package integrity against the lockfile, but a command that ran hostile code could still leave bad data behind, for example a swapped pnpm binary under `XDG_DATA_HOME`. Delete the directory if you suspect that.
-- **Accepted risk, network access:** the sandbox keeps the host network because installs need it. A validation command can reach the internet, any service listening on localhost (including Foreman's own API) and the host's abstract-namespace Unix sockets, though it does not receive your proxy or credential variables. Do not treat the sandbox as a network boundary.
+- **Network policy: none, except commands that explicitly need it.** Every validation command runs in an empty network namespace (`bwrap --unshare-net`). It has no interface except a working loopback, so tests that bind and connect to `127.0.0.1` inside the sandbox still work, but it cannot reach Foreman's own API, the per-project bridge or any other service on the host's loopback, the host's abstract-namespace Unix sockets, the cloud instance metadata endpoint, the local network or the internet, and DNS lookups fail. A hostile test therefore cannot drive Foreman's API for another run or fetch instance credentials. If bubblewrap cannot create the network namespace on your host (some containers refuse), the sandbox probe fails and validation reports the sandbox as unavailable; it never falls back to the host network.
+- **The `network` flag.** Every validation command takes an optional boolean `network`: in `FOREMAN_VALIDATION_COMMANDS` (`{"name","command","args","cwd?","network?"}`), in a saved workspace setup, in the task-start command overrides, and as the per-command **Network** checkbox in the open-repository dialog and the task's advanced options. Only `true` keeps the host network for that command; only the JSON values `true` and `false` are accepted, anything else is rejected. Each check records what it got as `network` (with `sandbox: none` every check is recorded as `network: true`, since nothing isolates it), and the checks pipeline shows a **Network** badge on checks that ran with network access.
+- **Which commands get the network by default.** The open-repository dialog starts each command from the repository inspector's suggestion: the dependency install (`pnpm install --frozen-lockfile`, `npm ci`) has `network: true`, and so do `cargo test` and `go test ./...` because they download dependencies on first run. Every other suggestion (`pnpm run test`, `pnpm run typecheck`, `python -m pytest`, ...) is offline; the install step puts `node_modules` in the shared validation workspace, so the checks after it need no network. A command that fetches tooling implicitly (for example pnpm downloading the version pinned in `packageManager`) fails offline and needs the checkbox.
+- **Older setups and configs.** A command with no `network` field, in a saved workspace setup, in `FOREMAN_VALIDATION_COMMANDS` or in a task-start override, is treated as `network: true` if it is a recognised package-manager install and `network: false` otherwise. Recognised means the executable is exactly `pnpm`, `npm` or `yarn` with first argument `install`, `i`, `ci` or `add`, or bare `yarn` with no arguments. An explicit `network` always wins, so `{"command":"pnpm","args":["install","--offline"],"network":false}` stays offline. Runs stored before the flag existed are not rewritten (their approval is bound to a digest of the stored command list); a stored command without the field gets the same default when it runs, and the check records the effective value.
+- **Residual risk, network-enabled commands.** A command with `network: true` shares the host network namespace, including the host's loopback services and the metadata endpoint. Package lifecycle scripts (`preinstall`, `postinstall`, `prepare`) run during a network-enabled install, so a hostile dependency or Worker-edited manifest gets the network for the length of that install. Grant `network` only to the install step, and consider `--ignore-scripts` if your dependencies allow it. The offline default protects every other command, including the tests.
 - Run Foreman as an unprivileged user. The sandbox hides paths but cannot stop a root process from reading root-only files that remain visible, such as `/etc/shadow`.
 
-`FOREMAN_VALIDATION_SANDBOX=none` is an explicit, unsafe opt-out for hosts without bubblewrap (for example macOS). Worker-authored code then runs directly on your machine with your credentials. Foreman prints a warning at startup, marks each check with the mode it ran in, and reports `validationSandbox: {mode, available}` from `GET /api/status`. Foreman's own tests run validation through the sandbox, so `pnpm test` needs `bwrap` as well.
+`FOREMAN_VALIDATION_SANDBOX=none` is an explicit, unsafe opt-out for hosts without bubblewrap (for example macOS). Worker-authored code then runs directly on your machine with your credentials and the host network, and the `network` flag has no effect. Foreman prints a warning at startup, marks each check with the mode it ran in, and reports `validationSandbox: {mode, available}` from `GET /api/status`. Foreman's own tests run validation through the sandbox, so `pnpm test` needs `bwrap` as well.
 
 ## Reviewing and approving results
 
@@ -87,7 +91,7 @@ When a run reaches `awaiting_approval`, the **Ready for your review** panel appe
 - **Approve result** / **Reject** buttons.
 - A stepper: Review → Approve → Promote → Push → PR → Merge.
 
-**Approve result** records an immutable decision bound to the evidence digest. It does not change Git.
+**Approve result** records an immutable decision bound to the evidence digest you reviewed: the panel loads the digest for the evidence it displays (`GET /api/runs/:id/decision`) and sends it with Approve or Reject. If the evidence changed after that (a validation retry, a correction cycle, another tab), Foreman refuses the decision with a message asking you to review the current result. It does not change Git.
 
 **Promote approved result** creates a local commit rooted at the pinned base and a result branch. Your checkout is unchanged.
 
@@ -140,7 +144,7 @@ The standard local flow requires no environment variables. The following setting
 | `FOREMAN_WORKSPACE_SOURCE_REPO` | — | Local Git repository for snapshot verification. Required with the manual bridge workflow. |
 | `FOREMAN_WORKSPACE_BRIDGE_URL` | — | Loopback URL for an external workspace bridge. |
 | `FOREMAN_WORKSPACE_ALLOWED_SCOPE` | — | Comma-separated exact paths or directory prefixes ending in `/` allowed in Worker results. |
-| `FOREMAN_VALIDATION_COMMANDS` | — | JSON array of `{"name","command","args","cwd?"}` entries run in the disposable validation workspace. |
+| `FOREMAN_VALIDATION_COMMANDS` | — | JSON array of `{"name","command","args","cwd?","network?"}` entries run in the disposable validation workspace. `network` is `true` or `false`; validation runs offline unless it is `true`, and a package-manager install without the field defaults to `true`. See [Validation sandbox](#validation-sandbox). |
 | `FOREMAN_VALIDATION_TIMEOUT_MS` | `120000` | Per-command time limit (max 600,000 ms). |
 | `FOREMAN_VALIDATION_MAX_OUTPUT_BYTES` | `1048576` | Per-command output capture bound (max 16 MiB). |
 | `FOREMAN_VALIDATION_SANDBOX` | `bwrap` | `bwrap` runs every validation command in a bubblewrap sandbox and fails if it is unavailable. `none` runs them directly on the host with your credentials, and is unsafe. See [Validation sandbox](#validation-sandbox). |

config.schema.json

@@ -17,7 +17,7 @@
     "FOREMAN_WORKSPACE_SOURCE_REPO": { "type": "string", "description": "Local Git source repository used for pinned Worker snapshot verification" },
     "FOREMAN_WORKSPACE_BRIDGE_URL": { "type": "string", "format": "uri", "description": "Loopback URL for the external workspace bridge" },
     "FOREMAN_WORKSPACE_ALLOWED_SCOPE": { "type": "string", "description": "Comma-separated exact paths or directory prefixes ending in / allowed for Worker changes" },
-    "FOREMAN_VALIDATION_COMMANDS": { "type": "string", "description": "JSON array of validation command objects with explicit command and args" },
+    "FOREMAN_VALIDATION_COMMANDS": { "type": "string", "description": "JSON array of validation command objects {name, command, args, cwd?, network?} with explicit command and args. network is true or false (no other value is accepted): validation commands run without network access unless it is true. When omitted, a pnpm, npm or yarn install (first argument install, i, ci or add, or bare yarn) defaults to true and every other command to false" },
     "FOREMAN_VALIDATION_TIMEOUT_MS": { "type": "integer", "minimum": 1, "maximum": 600000, "default": 120000 },
     "FOREMAN_VALIDATION_MAX_OUTPUT_BYTES": { "type": "integer", "minimum": 1, "maximum": 16777216, "default": 1048576 },
     "FOREMAN_VALIDATION_SANDBOX": { "type": "string", "enum": ["bwrap", "none"], "default": "bwrap", "description": "bwrap runs every validation command inside a bubblewrap sandbox and fails closed when it is unavailable; none runs them unsandboxed on the host (unsafe opt-out)" },

docs/architecture.md

@@ -22,4 +22,4 @@ When the Orchestrator returns an empty or soft-denied response (e.g. AGY invokin
 
 ## Approval and Git promotion
 
-Human approval records an immutable decision bound to the evidence digest and pinned input base. Its `evidenceCommit` names the pinned base; it is not the result commit. Promotion is a separate explicit operation. Before creating a result commit, the controller rechecks the stored snapshot bytes and modes, allowed scope, pinned base, validation record, and Reviewer binding. It builds the tree in an isolated worktree, commits with the pinned base as parent, then compares the complete result tree to the approved snapshot. Only that verified result commit and tree are reported as applied. A failed or interrupted attempt remains distinct from the human decision and can be retried idempotently; conflicting branch state is rejected.
+Human approval records an immutable decision bound to the evidence digest and pinned input base. The digest is the one the operator reviewed: `GET /api/runs/:id/decision` returns it with the Worker response, validation, and Reviewer recommendation IDs it covers (only at a decision checkpoint), and the approve/reject request must carry it as `evidenceDigest`. `approveRun` recomputes the digest inside the same store mutation and refuses a missing or malformed value (400) or a stale one (409), so an approval or rejection never applies to evidence the operator did not see. Its `evidenceCommit` names the pinned base; it is not the result commit. Promotion is a separate explicit operation. Before creating a result commit, the controller rechecks the stored snapshot bytes and modes, allowed scope, pinned base, validation record, and Reviewer binding. It builds the tree in an isolated worktree, commits with the pinned base as parent, then compares the complete result tree to the approved snapshot. Only that verified result commit and tree are reported as applied. A failed or interrupted attempt remains distinct from the human decision and can be retried idempotently; conflicting branch state is rejected.