diff --git a/.github/workflows/live.yml b/.github/workflows/live.yml new file mode 100644 index 0000000..f8d7038 --- /dev/null +++ b/.github/workflows/live.yml @@ -0,0 +1,72 @@ +# This workflow implements build target 8 of docs/design/stories/293.md: the +# live rung. Task #324, anchoring NEVER-293.10. +# +# NEVER-293.10 (anti-claim): the live rung is NEVER a required CI check, and +# it NEVER sends PRIVATE-class content to a cloud model. +# +# WHY A SEPARATE FILE, ON PURPOSE, RATHER THAN A JOB IN ci.yml: +# +# A job added to ci.yml runs on every `pull_request` — that file's `on:` +# block (ci.yml:30-36) triggers on `pull_request`, bare, with no +# `branches:` filter. A live job must NEVER run on a PR. ci.yml +# deliberately carries no `paths:` filter and no `branches:` filter +# anywhere (NEVER-9.8, ci.yml:16-22), because a filtered workflow leaves +# its required checks stuck at "pending" forever — so there is no way to +# carve out a non-PR-triggering job inside that file without reintroducing +# the exact filter NEVER-9.8 forbids. +# +# A separate `workflow_dispatch`-only workflow is the only shape that +# CANNOT fire on a PR. That is precisely what makes it safe from ever +# being accidentally promoted to required: a required context that never +# reports on a PR blocks every PR permanently (issue #204's rule, restated +# in ci.yml's comment above the `seam` job) — so a context that can never +# report on a PR is structurally disqualified from ever being made +# required, not merely policy-disqualified. +# +# A third, independent reason: `pull_request` from a fork receives no +# secrets. A required live job on a fork PR could never authenticate to a +# cloud model even if someone tried to wire it up. This is Problem 7a of +# docs/design/stories/293.md — three independently sufficient reasons a +# required live job is structurally illegal, not merely inadvisable. +# +# WHAT THIS JOB ACTUALLY DOES — AND WHAT IT DOES NOT DO: +# +# This job does NOT run the live rung end to end. A real conformance run +# is MODEL-DRIVEN by a human operator (see scripts/live-rung.ts's own +# header): the operator's agent host reads a SKILL.md and performs it +# against a real forge, issuing one `bun run live-rung exec` per tool +# call. `scripts/live-rung.ts` is only the RECORDING SEAM that run is +# driven through, not the run itself, and no CI job can stand in for a +# human operator driving a model. +# +# What this job DOES do is prove the seam still works, safely and without +# touching any forge: it runs `bun run live-rung --help` (prints usage, +# exits 0, touches no filesystem or network) and `bun test +# test/live-rung.test.ts` (the seam's own unit suite, entirely against +# fixtures and the fake port). It creates no session, resolves no real +# commit, spawns no real `gh`/`git` beyond what the test suite itself +# sandboxes, and reaches no cloud model. + +name: Live rung seam check + +on: + workflow_dispatch: + +env: + BUN_VERSION: "1.4.0" + NODE_VERSION: "20" + +jobs: + live: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: oven-sh/setup-bun@v2 + with: + bun-version: ${{ env.BUN_VERSION }} + - uses: actions/setup-node@v4 + with: + node-version: ${{ env.NODE_VERSION }} + - run: bun install + - run: bun run live-rung --help + - run: bun test test/live-rung.test.ts diff --git a/docs/evidence/324-20260915T131526Z.md b/docs/evidence/324-20260915T131526Z.md new file mode 100644 index 0000000..64065a2 --- /dev/null +++ b/docs/evidence/324-20260915T131526Z.md @@ -0,0 +1,272 @@ +--- +issue: 324 +story: 293 +repo: dev-pmallapp/iAI +branch: task/324-live-rung-never-required +commit: 603b02d21bc422fe6ea4b36b90b3de630a85af0e +run: 2026-09-15T13:15:26Z +verdict: PASS +--- + +# Evidence: #324 — the live rung, never required + +Build target 8 of `docs/design/stories/293.md`. Cases **13, 14, 15 and 19**. + +`commit:` pins the **parent** of the commit carrying this artifact. The live +conformance run itself was performed on **2026-09-12 at commit `1527dfb`**, +which is what the attestation below states. + +**This artifact is deliberately self-contained.** Its subject — the fixture +repository `dev-pmallapp/iai-live-rung-fixture` — was created to be destroyed, +and has been deleted. Every figure and every argv below is reproduced here in +full, because there is nothing left to go and look at. An artifact whose +evidence lives only in a deleted repository is an assertion, not a record. + +## Chain + +``` +bun run build exit 0 +bun test 1156 tests / 1142 pass / 0 fail / 14 skip / 4218 expect() +bun run lint 0 violations across 126 files +bun run typecheck exit 0 +bun run skill-lint 4 SKILL.md + 12 contract files, 0 errors +bun run claim-lint 124 files, 0 errors, 1 warning +bun run skill-harness exit 0 +bun run skill-harness --inject exit 1 <- this mode's success condition +scripts/verify-required-checks.sh PASS, SIX contexts +scripts/verify-workflow-hygiene.sh PASS +``` + +The 1 claim-lint warning is `docs/design/stories/9.md`, which predates the +metadata convention (ruling G1b of #41). **It must stay a warning.** + +## Case results + +| Case | Claim | Verdict | +|---|---|---| +| 13 | `NEVER-293.10` | **PASS** — `live` absent from all four hardcoded lists; count read, not hardcoded | +| 14 | `NEVER-293.10` | **PASS** — 0 `if:` keys on required jobs, count asserted non-zero first | +| 15 | `NEVER-293.10` | **PASS** — over all 4 classes, denominator read from `DATA_CLASSES` | +| 19 | `CLAIM-293.9` | **PASS with a qualification** — attested below; token cost is an upper bound, see #337 | + +## The live conformance run + +A model — `amd-anthropic/Claude-Opus-5` on host **opencode**, which is routing +category **`plan`** per `references/model-routing.md:32` — read +`skills/goal-create/SKILL.md` and performed it twice against a real forge. +Every command was routed through `scripts/live-rung.ts`'s `exec`, the only +recorded route to the real port. + +**This is a different population from the seam rung's.** The seam rung runs a +*transcription* against a *fake*; gate ruling G2 turns on that distinction. +This run is the model reading the markdown. + +| | run 1 | run 2 | +|---|---|---| +| calls | 4 | 3 | +| reads | 3 | 3 | +| forge mutations | **1** | **0** | + +Run 2 issued **no mutating command at all**, which is exactly what +`skills/goal-create/SKILL.md:91-93` promises. + +### The result worth keeping + +**The live run and the transcription agree exactly.** `SCENARIO_ROSTER`'s +`goal-create/absent-milestone` scores run 1 at `mutations=1 reads=3` and run 2 +at `mutations=0 reads=3`. The live model produced the same six numbers. + +That is evidence for `CLAIM-293.11` the transcription audit could not produce. +`docs/audits/293-transcription-audit.md` compared **prose to code**; this +compares **code to reality**. Neither subsumes the other, and the audit's +verdict is not upgraded by this: n=1 over a non-deterministic trajectory is an +existence proof, not a property. `CLAIM-41.8` stays `model-judged`, and gate +ruling G2 still forbids reading either result as licence to change that. + +### The conformance artifact, verbatim + +```json +{ + "schemaVersion": 1, + "session": { + "repo": "dev-pmallapp/iai-live-rung-fixture", + "resolvedCommit": "1527dfbedb274760617dbfec7a5d3b281e8674e7", + "startedAt": "2026-09-12T15:28:29.924Z", + "host": "opencode" + }, + "tokenCost": 305200, + "runs": { + "1": { "totalCalls": 4, "reads": 3, "mutations": 1, + "kinds": { "forge-read": 2, "forge-mutation": 1, "git-read": 1, "git-write": 0, "unclassified": 0 } }, + "2": { "totalCalls": 3, "reads": 3, "mutations": 0, + "kinds": { "forge-read": 2, "forge-mutation": 0, "git-read": 1, "git-write": 0, "unclassified": 0 } } + }, + "overall": { "totalCalls": 7, "reads": 6, "mutations": 1 }, + "nonZeroExits": [] +} +``` + +## #330 is paid, and the precedent inverted + +**No `gh` argv this repository builds had ever been executed.** Six tasks +deferred it. This run executed **thirteen**, each built by the real exported +builder in `packages/core/src/gh/`, each routed through the recording seam: + +| Family | argv executed | Result | +|---|---|---| +| milestones | `milestoneList`, `milestoneCreate`, `milestoneUpdate` | 3/3 accepted | +| labels | `labelList`, `labelCreate` | 2/2 accepted | +| issues | `issueList`, `issueCreate`, `issueEditBody`, `issueClose` | 4/4 accepted | +| comments | `commentList`, `commentCreate`, `commentEdit` | 3/3 accepted | +| issues | `gh issue edit --milestone` | 1/1 accepted | + +**13 executed, 13 accepted, 0 rejected.** `nonZeroExits: []`. + +The S2.1 precedent was sixty passing tests and **7 of 9 real verbs rejected on +first contact**. That did not repeat. The argv layer is vindicated — for these +thirteen. `pr` (12 exports), `sub-issues` (14), and the rest of each family +remain unexecuted, so **#330 is paid in part, not in full.** + +### The exit code was not taken as the answer + +#320's M9/M9b pair measured that a double which refuses and a double which +accepts return the **same verdict** and differ only in **state**. So every +mutation was verified by reading the forge afterwards: + +| Mutation | State observed after | +|---|---| +| `milestoneUpdate` | milestone 2's description read `\| probe \| AMENDED by milestoneUpdate \|` | +| `issueEditBody` | issue 1's body read `edited by issueEditBody` | +| `issueClose` | issue 1 state `CLOSED` | +| `commentEdit` | the comment body read `edited by commentEdit` | +| `issue edit --milestone` | issue 1's milestone read `probe-milestone-330` | +| `labelCreate` | label `probe-330` present on issue 1 | + +### Both of the fake's unmodelled shapes work against a real forge + +`docs/audits/293-transcription-audit.md` recorded two authorised mutations that +are dead in the seam roster: `goal-create`'s milestone `PATCH`, and +`story-create`'s `--milestone` reassign on `gh issue edit`. Both were executed +here and **both succeeded**. + +So the dead branches are dead **because of the fake**, precisely as the audit +claimed — not because the transcriptions are wrong. That closes the record the +audit could only open. + +## Case 15, and the finding underneath it + +The egress gate runs **first** in `live-rung init`, before anything touches the +filesystem, and demonstrates its converse on the same path: if a `USER/`-derived +payload is not refused, `init` fails. A gate that cannot be shown refusing is an +unexercised branch. Observed on the live path: + +``` +egress gate ADMITTED the declared-synthetic goals fixture (class INTERNAL, destination cloud) +egress gate correctly REFUSED a USER/-derived payload (class PRIVATE) +``` + +**The finding, now #336.** `packages/core/src/classify/classify.ts` imports only +`./levels` and `./recognisers` — it never imports `./path`, so `classifyPath`'s +`USER/` rule is consulted by nothing. Measured: + +| payload | `classify()` | +|---|---| +| `{ source_path: "USER/HEALTH/labs.yaml", … }` | `PRIVATE` | +| `{ zzz_unrelated: "x" }` — no `USER/` anywhere | `PRIVATE` | +| `{ work_state: "USER/HEALTH/labs.yaml" }` | **`INTERNAL`** | +| `classifyPath("USER/HEALTH/labs.yaml")` | `PRIVATE` | + +Row 3 is the hole. Row 2 is why it went unnoticed: a test using a `USER/`-ish +payload with unrecognised keys passes **whether or not the rule is wired in** — +it cannot distinguish the reason for its own answer. + +**Case 15 still passes honestly.** Its wording is *"`USER/`-derived content is +`PRIVATE` **by fail-safe default**"* — the unrecognised-key rule, which is +exactly what fires. The test now says so in its own name, a control proves the +discrimination, and a tripwire asserts `classify.ts` does not import `./path`, +so wiring it in forces this record to be revisited deliberately. + +**`NEVER-293.10` holds:** the gap is in **classification**, not in the guard. +The guard blocks everything it classifies `PRIVATE`. + +**And the gate is not what protects this command.** A real goals document under +`work_state` would clear it exactly as the synthetic fixture did. What stops a +run against the real source is narrower and duller: the payload is built from +`renderGoalsFixture` with no flag, argument or code path that can redirect it — +asserted by test. Two protections guarding two different things; conflating them +would leave a hole nobody is watching. + +## Why the live rung can never become required + +`.github/workflows/live.yml` is a **separate file** with `workflow_dispatch` as +its only trigger. The separation is the mechanism, not tidiness: + +- a job in `ci.yml` runs on every `pull_request`, and a live job must not; +- `ci.yml` deliberately carries no `paths:` and no `branches:` filter, because + NEVER-9.8 says a filtered workflow leaves required checks pending forever — so + there is no way to add a non-PR-triggering job to that file; +- a `workflow_dispatch`-only workflow cannot fire on a PR, which is also why it + can never be accidentally promoted: a context that never reports would block + every PR forever if required; +- fork PRs receive no secrets — Problem 7a's third independent reason. + +The job proves the seam (`live-rung --help` and its test file) and never opens a +session against a real repository. + +`test/ci-workflow.test.ts` asserts the count **before and after** by reading it, +never by hardcoding `6` — so **#334** taking it to seven will not require editing +this test to keep it true. The partition now spans **both** workflow files: every +job in every workflow must be required, or named not-required with a reason. + +## Attestation — `CLAIM-293.9`, case 19 + +`references/verification.md:37` defines `human-attested` as *"a person states it +is so"*, failing when *"nobody is named, or the attestation is inferred from +silence"*. + +> **I, `dev-pmallapp`, state** that the live conformance run of `goal-create` was +> performed on **2026-09-12**, at commit **`1527dfb`**, on host **opencode**, at +> routing category **`plan`** (`amd-anthropic/Claude-Opus-5`, per +> `references/model-routing.md:32`), with the artifact reproduced in full in this +> document. +> +> The run was a fair exercise: the goals source was #319's declared-`synthetic` +> fixture, no expected answer was placed in the prompt, and the mutation counts +> were produced by the recording seam rather than transcribed by hand. + +**Token cost: 305.2k tokens / \$36.14 — an opencode SESSION TOTAL for +2026-09-12, not the isolated cost of the `goal-create` verb.** + +`docs/milestones/M2.md:240-241` asks for the measured cost of *Tier 0 plus one +Tier-1 verb*. This figure is an **upper bound** on that: the same session also +built the live rung, the transcription audit, the CI workflow and the mutation +rounds. It is recorded as an upper bound and **must not be quoted as the verb's +cost**. + +> **`M2.md:240-241` SHOULD NOT BE TICKED ON THIS NUMBER.** Raised as **#337**, +> with the cheap fix named: one dedicated session that runs a single Tier-1 verb +> end to end, so the session total *is* the verb's cost. `--token-cost` already +> accepts the figure. Closing a context-budget criterion on a bound two orders of +> magnitude above the thing being bounded is the *"stronger-sounding class than +> the check supports"* failure `references/verification.md:43-46` names. + +## The fixture repository is gone, by design + +`dev-pmallapp/iai-live-rung-fixture` was created for this run and **deleted by +the operator via the web interface afterwards** — verified absent at the time of +writing: `gh repo view` returns *"Could not resolve to a Repository"*. + +The token in use carries `repo` but **not `delete_repo`**, so the deletion could +not be automated. Recorded because the next live rung will hit the same wall. + +## What this does NOT establish + +- **`CLAIM-293.11` is not upgraded.** n=1 over a non-deterministic trajectory is + an existence proof, not a property. `CLAIM-41.8` and `CLAIM-47.4` stay + `model-judged`; gate ruling G2 forbids otherwise. +- **#330 is paid in part.** Thirteen argvs met a real forge. `pr` (12 exports) + and `sub-issues` (14) did not. The bill is smaller, not settled. +- **M2's token-cost criterion is not closed** — #337. +- **The `USER/` classification gap is open** — #336. No caller exploits it today. +- **One skill was run live, not four.** `goal-create` only. +- **The live rung is not a required check, and must never become one.** diff --git a/package.json b/package.json index b8480eb..3a903c7 100644 --- a/package.json +++ b/package.json @@ -11,7 +11,8 @@ "typecheck": "tsc -b tsconfig.json --force", "skill-lint": "bun scripts/skill-lint.ts", "claim-lint": "bun scripts/claim-lint.ts", - "skill-harness": "bun scripts/skill-harness.ts" + "skill-harness": "bun scripts/skill-harness.ts", + "live-rung": "bun scripts/live-rung.ts" }, "devDependencies": { "typescript": "^7.0.2", diff --git a/packages/core/src/classify/index.ts b/packages/core/src/classify/index.ts index 3e0ee6e..6a5017d 100644 --- a/packages/core/src/classify/index.ts +++ b/packages/core/src/classify/index.ts @@ -1,4 +1,5 @@ export type { DataClass } from "./levels"; +export { DATA_CLASSES } from "./levels"; export { classify } from "./classify"; export { classifyPath } from "./path"; export { classifyKeyName, classifySecretShape, normalizeKey } from "./recognisers"; diff --git a/packages/core/src/classify/levels.ts b/packages/core/src/classify/levels.ts index ca03764..7eaf666 100644 --- a/packages/core/src/classify/levels.ts +++ b/packages/core/src/classify/levels.ts @@ -4,12 +4,26 @@ // result (docs/design/stories/15.md, CLAIM-15.1). export type DataClass = "PUBLIC" | "INTERNAL" | "PRIVATE" | "SECRET"; -const RANK: Record = { - PUBLIC: 0, - INTERNAL: 1, - PRIVATE: 2, - SECRET: 3, -}; +// THE single source of truth for which classes exist, in rank order, +// least-sensitive first (Task #324, case 15 of docs/test-plans/293-plan.md, +// anchoring NEVER-293.10). `RANK` below is DERIVED from this array rather +// than restating the four members, so a fifth class added to `DataClass` +// cannot silently go missing from one of the two without also going missing +// from the other — the two would simply fail to compile or fail +// packages/core/test/classify.test.ts's agreement assertions instead. +// +// Frozen so a caller holding a reference cannot mutate the shared array out +// from under every other reader of this module. +export const DATA_CLASSES: readonly DataClass[] = Object.freeze([ + "PUBLIC", + "INTERNAL", + "PRIVATE", + "SECRET", +]); + +const RANK: Record = Object.freeze( + Object.fromEntries(DATA_CLASSES.map((dataClass, index) => [dataClass, index])) as Record, +); export function rankOf(dataClass: DataClass): number { return RANK[dataClass]; diff --git a/packages/core/test/levels.test.ts b/packages/core/test/levels.test.ts new file mode 100644 index 0000000..0796cd1 --- /dev/null +++ b/packages/core/test/levels.test.ts @@ -0,0 +1,90 @@ +// Tests for Task #324, case 15 (P0, NEVER-293.10) step 1: the class count +// must be readable FROM THE CLASSIFIER, not hardcoded by a caller. +// `DATA_CLASSES` (packages/core/src/classify/levels.ts) is the single +// exported source of truth for which classes exist; `RANK` is derived from +// it rather than restating the four members, so the two structures cannot +// drift apart -- these tests prove that agreement rather than assuming it. +import { describe, expect, test } from "bun:test"; +import { DATA_CLASSES, type DataClass } from "../src/classify/index"; +import { maxClass, rankOf } from "../src/classify/levels"; + +describe("DATA_CLASSES (P0, NEVER-293.10): the class count is readable from the classifier", () => { + test("0. denominator non-zero first", () => { + expect(DATA_CLASSES.length).toBeGreaterThan(0); + }); + + // Asserted as a literal 4 in the TEST, per the task's own instruction -- + // production code must never hardcode this number, but a test proving the + // corpus is exactly the four documented classes is the whole point. + test("1. there are exactly 4 classes today", () => { + expect(DATA_CLASSES.length).toBe(4); + }); + + test("2. every member of DATA_CLASSES has a rank (rankOf never throws, never returns undefined)", () => { + for (const dataClass of DATA_CLASSES) { + expect(() => rankOf(dataClass)).not.toThrow(); + expect(rankOf(dataClass)).toBeDefined(); + expect(typeof rankOf(dataClass)).toBe("number"); + } + }); + + test("3. ranks are assigned in DATA_CLASSES's own order, least-sensitive first", () => { + for (let i = 0; i < DATA_CLASSES.length; i += 1) { + expect(rankOf(DATA_CLASSES[i] as DataClass)).toBe(i); + } + }); + + test("4. every rank is unique -- no two classes share a rank", () => { + const ranks = DATA_CLASSES.map((dataClass) => rankOf(dataClass)); + expect(new Set(ranks).size).toBe(DATA_CLASSES.length); + }); + + test("5. the reverse direction: every DataClass the type union admits is present in DATA_CLASSES", () => { + // Runtime check: a class removed from DATA_CLASSES (but still a member + // of the DataClass union) must fail here directly -- this is the + // assertion the mutation table's case (b) is aimed at. + const required: readonly DataClass[] = ["PUBLIC", "INTERNAL", "PRIVATE", "SECRET"]; + for (const dataClass of required) { + expect(DATA_CLASSES.includes(dataClass)).toBe(true); + } + }); + + test("5b. compile-time backstop: exhaustive switch over DataClass, mirrored against DATA_CLASSES", () => { + // If `DataClass` ever grows a member absent from this switch, the + // `never` assignment in the default arm fails to compile -- a second, + // independent guarantee that DATA_CLASSES cannot silently fall behind + // the type it is supposed to enumerate. + function assertKnown(dataClass: DataClass): true { + switch (dataClass) { + case "PUBLIC": + case "INTERNAL": + case "PRIVATE": + case "SECRET": + return true; + default: { + const exhaustive: never = dataClass; + throw new Error(`unreachable: ${String(exhaustive)}`); + } + } + } + for (const dataClass of DATA_CLASSES) { + expect(assertKnown(dataClass)).toBe(true); + } + }); + + test("6. the array is frozen -- a caller cannot mutate the shared export", () => { + expect(Object.isFrozen(DATA_CLASSES)).toBe(true); + expect(() => { + (DATA_CLASSES as unknown as DataClass[]).push("PUBLIC"); + }).toThrow(); + }); + + test("7. maxClass agrees with rank order over every ordered pair", () => { + for (const a of DATA_CLASSES) { + for (const b of DATA_CLASSES) { + const expected = rankOf(a) >= rankOf(b) ? a : b; + expect(maxClass(a, b)).toBe(expected); + } + } + }); +}); diff --git a/packages/harness/src/index.ts b/packages/harness/src/index.ts index f21a51a..f7f3052 100644 --- a/packages/harness/src/index.ts +++ b/packages/harness/src/index.ts @@ -91,3 +91,6 @@ export type { ScenarioId } from "./scenario-roster"; export { INJECTION_ROSTER } from "./injection-roster"; export type { InjectionScenarioId } from "./injection-roster"; + +export { admitToLiveModel, inspectLiveEgress, LiveEgressRefusal } from "./live-egress"; +export type { LiveEgressAdmitted, LiveEgressInspection } from "./live-egress"; diff --git a/packages/harness/src/live-egress.ts b/packages/harness/src/live-egress.ts new file mode 100644 index 0000000..b49106b --- /dev/null +++ b/packages/harness/src/live-egress.ts @@ -0,0 +1,106 @@ +// The live rung's egress gate (Task #324, case 15 of +// docs/test-plans/293-plan.md, anchoring NEVER-293.10). +// +// THIS IS A SECURITY REQUIREMENT, NOT A TEST-DESIGN DETAIL. Decision (raised +// but not gated) at docs/design/stories/293.md:636-639 says so in terms: +// "The live rung is an egress question, not a test-design detail." A `USER/` +// -derived payload is `PRIVATE` by fail-safe default +// (packages/core/src/classify/path.ts's `USER/` rule, CLAIM-15.2) and +// reaches no cloud model UNDER ANY CONSENT — not granted, not withheld, not +// omitted, not malformed. That is Decision 2 of docs/design/stories/243.md, +// enforced in packages/core/src/guards/egress.ts and proved inert by +// NEVER-243.6 (packages/core/test/egress.test.ts, case 13 especially): the +// `consent` parameter accepted below is PROVABLY INERT for the PRIVATE/cloud +// cell this module exists to guard, on purpose, forever, unless a later +// Story explicitly revisits that Decision. Passing `{ granted: true }` here +// buys nothing for a PRIVATE payload against a cloud destination. +// +// PURE: no fs, no net, no process, no Bun globals, no node builtin import — +// same standing decision as packages/core/src/classify and +// packages/core/src/guards (CLAIM-15.6). This module only calls the two +// pure predicates it wraps. +// +// THE GATE THE LIVE RUNNER MUST CLEAR. `admitToLiveModel` is the single +// call a live runner makes before any payload reaches a cloud model. It +// returns a value ONLY when the payload is admitted; a blocked payload +// THROWS a typed `LiveEgressRefusal` instead of returning a boolean the +// caller could examine and ignore. There is no code path through this +// module that hands back "false" for a caller to skip past — the runner +// either receives an admitted result, or unwinds through the refusal, same +// as any other exception. `inspectLiveEgress` is provided alongside it for +// callers (tests, diagnostics) that want the class-and-decision pair +// without the throw. +import { + CONSENT_WITHHELD, + checkEgress, + classify, + type DataClass, + type Destination, + type EgressConsent, + type EgressDecision, +} from "iai-core"; + +export interface LiveEgressInspection { + readonly dataClass: DataClass; + readonly decision: EgressDecision; +} + +export interface LiveEgressAdmitted { + readonly dataClass: DataClass; + readonly decision: EgressDecision & { readonly action: "allow" }; +} + +/** Classify `payload`, check it against `destination`, and return BOTH the + * class and the decision — never merely one or the other. Never throws + * (`classify` and `checkEgress` are both total functions that fail closed + * internally); this is the inspection primitive `admitToLiveModel` below is + * built on, exposed separately for callers that need to observe a block + * without catching an exception. */ +export function inspectLiveEgress( + payload: unknown, + destination: Destination, + consent: EgressConsent = CONSENT_WITHHELD, +): LiveEgressInspection { + const dataClass = classify(payload); + const decision = checkEgress(payload, destination, consent); + return { dataClass, decision }; +} + +/** Thrown by `admitToLiveModel` when the egress gate blocks. Carries the + * class and the decision that produced the refusal, so a caller that does + * catch it can still report why, without having to re-run the check. */ +export class LiveEgressRefusal extends Error { + readonly dataClass: DataClass; + readonly decision: EgressDecision & { readonly action: "block" }; + + constructor(dataClass: DataClass, decision: EgressDecision & { readonly action: "block" }) { + super(decision.message); + this.name = "LiveEgressRefusal"; + this.dataClass = dataClass; + this.decision = decision; + } +} + +/** THE GATE. A live runner calls this immediately before handing `payload` + * to a cloud model at `destination`. On a block, this THROWS + * `LiveEgressRefusal` — it never returns a falsy value the caller could + * check and route around. On an allow, it returns the admitted result; + * there is nothing further to check because there is no other way to reach + * this return. + * + * `consent` defaults to `CONSENT_WITHHELD`, matching `checkEgress`'s own + * fail-closed default (NEVER-15.8): an omitted consent is never treated as + * more permissive than an explicit withheld one. Passing a granted consent + * changes nothing for a PRIVATE payload against a cloud destination — see + * the module header. */ +export function admitToLiveModel( + payload: unknown, + destination: Destination, + consent: EgressConsent = CONSENT_WITHHELD, +): LiveEgressAdmitted { + const { dataClass, decision } = inspectLiveEgress(payload, destination, consent); + if (decision.action === "block") { + throw new LiveEgressRefusal(dataClass, decision); + } + return { dataClass, decision: decision as EgressDecision & { readonly action: "allow" } }; +} diff --git a/packages/harness/test/live-egress.test.ts b/packages/harness/test/live-egress.test.ts new file mode 100644 index 0000000..5cd52ef --- /dev/null +++ b/packages/harness/test/live-egress.test.ts @@ -0,0 +1,229 @@ +// Tests for Task #324, case 15 of docs/test-plans/293-plan.md, anchoring +// NEVER-293.10: "The live runner refuses a PRIVATE-class payload." +// +// Corpus: synthetic. A `USER/`-derived payload cannot be committed to a +// public tree (that is the whole point of `USER/` being gitignored, +// packages/core/src/classify/path.ts), so the PRIVATE fixture here is a +// synthetic stand-in shaped like one, never a real one. +import { readFileSync } from "node:fs"; +import { describe, expect, test } from "bun:test"; +import { + classify, + classifyPath, + CONSENT_WITHHELD, + DATA_CLASSES, + type DataClass, + type Destination, + type EgressConsent, +} from "iai-core"; +import { admitToLiveModel, inspectLiveEgress, LiveEgressRefusal } from "../src/index"; + +const CLOUD: Destination = { vendor: "anthropic", locality: "cloud" }; +const CONSENT_GRANTED: EgressConsent = { granted: true }; + +// A payload per class, over the same corpus shape egress.test.ts uses, so +// this file's fixtures classify exactly as documented rather than by +// coincidence. +const PAYLOAD_BY_CLASS: Record = { + PUBLIC: { ticker: "AAPL", broker: "Schwab", doi: "10.1000/xyz" }, + INTERNAL: { design_doc: "docs/design/09-security.md", issue_title: "egress guard" }, + PRIVATE: { ldl: 130, account_balance: 1200 }, + SECRET: { api_key: "sk-abcdefgh12345678" }, +}; + +// A `USER/`-derived payload: PRIVATE by the fail-safe path rule +// (packages/core/src/classify/path.ts), never a real private-repo path. +const USER_DERIVED_PAYLOAD = { source_path: "USER/HEALTH/labs.yaml", contents: "synthetic-only, not a real record" }; + +describe("inspectLiveEgress (P0, NEVER-293.10): all 4 classes, denominator read from the classifier", () => { + // DENOMINATOR NON-ZERO FIRST. + test("0. DATA_CLASSES is non-empty, and the corpus below covers every member", () => { + expect(DATA_CLASSES.length).toBeGreaterThan(0); + expect(Object.keys(PAYLOAD_BY_CLASS).length).toBe(DATA_CLASSES.length); + }); + + // Documented matrix, per packages/core/src/guards/egress.ts's header + // comment: PUBLIC/INTERNAL allow to cloud regardless of consent; PRIVATE + // and SECRET block to cloud regardless of consent. + const EXPECTED_CLOUD_ACTION: Record = { + PUBLIC: "allow", + INTERNAL: "allow", + PRIVATE: "block", + SECRET: "block", + }; + + for (const dataClass of DATA_CLASSES) { + test(`${dataClass}: classifies to itself, and cloud decision matches the documented matrix`, () => { + const payload = PAYLOAD_BY_CLASS[dataClass]; + const inspection = inspectLiveEgress(payload, CLOUD, CONSENT_WITHHELD); + expect(inspection.dataClass).toBe(dataClass); + expect(inspection.decision.action).toBe(EXPECTED_CLOUD_ACTION[dataClass]); + }); + } + + test("all 4 classes were exercised, not a silently short loop", () => { + let count = 0; + for (const dataClass of DATA_CLASSES) { + count += 1; + expect(EXPECTED_CLOUD_ACTION[dataClass]).toBeDefined(); + } + expect(count).toBe(DATA_CLASSES.length); + expect(count).toBe(4); + }); +}); + +describe("admitToLiveModel (P0, NEVER-293.10): the live gate allows PUBLIC to cloud", () => { + test("a PUBLIC payload is admitted, not refused (the block is not vacuous)", () => { + const admitted = admitToLiveModel(PAYLOAD_BY_CLASS.PUBLIC, CLOUD, CONSENT_WITHHELD); + expect(admitted.dataClass).toBe("PUBLIC"); + expect(admitted.decision.action).toBe("allow"); + }); +}); + +// Issue #336. `classify()` (packages/core/src/classify/classify.ts) imports +// only `./levels` and `./recognisers` -- it never imports `./path`, so +// `classifyPath` and its `USER/` root rule (packages/core/src/classify/ +// path.ts) are NOT consulted by `classify()`. The test below this comment +// used to be named "classify resolves USER/-derived content to PRIVATE", +// which read as if the `USER/` rule were the reason. It is not: the +// `USER_DERIVED_PAYLOAD` object has keys `source_path` and `contents`, +// neither of which `classifyKeyName` recognises, so it lands on PRIVATE by +// the unrecognised-key fail-safe default (the same default that resolves an +// empty object, `null`, or any other unrecognised shape to PRIVATE) -- +// exactly as if the string "USER/HEALTH/labs.yaml" were replaced by "x". The +// control payload below makes that swap explicit. +// +// This does NOT invalidate case 15 of docs/test-plans/293-plan.md ("The live +// runner refuses a PRIVATE-class payload"). That case's own wording is +// "`USER/`-derived content is `PRIVATE` by fail-safe default", which is +// precisely the unrecognised-key rule demonstrated here -- case 15 passes +// honestly and continues to do so. +// +// NEVER-293.10 ("The live runner refuses a PRIVATE-class payload") also +// still holds: the guard in packages/core/src/guards/egress.ts blocks +// everything it is handed that classifies PRIVATE, and it is handed a +// PRIVATE class here -- just not for the reason its class name might +// suggest. The gap #336 identifies is in CLASSIFICATION (`classify()` not +// consulting `classifyPath`), not in the GUARD, which does exactly what it +// is documented to do with whatever class it receives. +// +// No caller is known to exploit this today, and the live rung itself stays +// protected: its fixture (`USER_DERIVED_PAYLOAD` above) is a declared +// synthetic literal with no redirection path from a real `USER/`-rooted +// source, so nothing here routes real private-repo content anywhere. +describe("admitToLiveModel (P0, NEVER-293.10): a USER/-derived payload classifies PRIVATE by the unrecognised-key fail-safe default, NOT the USER/ path rule (issue #336)", () => { + test("classify resolves USER/-derived content to PRIVATE by the unrecognised-key fail-safe default, not the USER/ path rule", () => { + // The payload this whole file's live-gate tests exercise: PRIVATE. + const inspection = inspectLiveEgress(USER_DERIVED_PAYLOAD, CLOUD, CONSENT_WITHHELD); + expect(inspection.dataClass).toBe("PRIVATE"); + + // THE CONTROL: a payload with an unrecognised key and no "USER/" + // anywhere in it classifies to the exact same answer. If the `USER/` + // rule were what produced PRIVATE above, this payload -- which never + // mentions `USER/` -- would have no reason to land on PRIVATE too. + const NO_USER_ANYWHERE_PAYLOAD = { zzz_unrelated: "x" }; + expect(classify(NO_USER_ANYWHERE_PAYLOAD)).toBe(classify(USER_DERIVED_PAYLOAD)); + expect(classify(NO_USER_ANYWHERE_PAYLOAD)).toBe("PRIVATE"); + + // The `USER/` rule EXISTS: classifyPath (packages/core/src/classify/ + // path.ts) resolves a `USER/`-rooted path to PRIVATE on its own. + expect(classifyPath("USER/HEALTH/labs.yaml")).toBe("PRIVATE"); + + // ...but `classify()` never reaches it: the identical string, placed at + // a key `classifyKeyName` does not recognise, resolves INTERNAL -- + // content shaped exactly like the PRIVATE fixture above would clear a + // cloud destination under the documented matrix (PUBLIC/INTERNAL allow, + // PRIVATE/SECRET block) if it arrived through `classify()` this way. + expect(classify({ work_state: "USER/HEALTH/labs.yaml" })).toBe("INTERNAL"); + }); + + // TRIPWIRE for #336. If someone later wires `classifyPath` into + // `classify()` (e.g. adds `import { classifyPath } from "./path";` to + // classify.ts), this assertion goes red -- forcing a deliberate revisit of + // this test file and the case-15 record in docs/test-plans/293-plan.md, + // rather than silently changing what this file's green has meant. + test("tripwire: classify.ts does not import ./path (issue #336) -- update this file and 293-plan.md case 15 deliberately if it ever does", () => { + const classifySource = readFileSync( + new URL("../../core/src/classify/classify.ts", import.meta.url), + "utf8", + ); + expect(classifySource.length).toBeGreaterThan(0); + expect(classifySource).not.toMatch(/from\s+["']\.\/path["']/); + }); +}); + +describe("admitToLiveModel (P0, NEVER-293.10): USER/-derived PRIVATE content reaches no cloud model under any consent", () => { + test("consent withheld: the gate throws LiveEgressRefusal, never returns", () => { + expect(() => admitToLiveModel(USER_DERIVED_PAYLOAD, CLOUD, CONSENT_WITHHELD)).toThrow(LiveEgressRefusal); + }); + + test("consent GRANTED: the gate STILL throws LiveEgressRefusal — consent does not unlock it", () => { + expect(() => admitToLiveModel(USER_DERIVED_PAYLOAD, CLOUD, CONSENT_GRANTED)).toThrow(LiveEgressRefusal); + }); + + test("consent omitted entirely: the gate still throws (fail-closed default)", () => { + expect(() => admitToLiveModel(USER_DERIVED_PAYLOAD, CLOUD)).toThrow(LiveEgressRefusal); + }); + + test("the refusal names the class and the decision, over both consent states", () => { + for (const consent of [CONSENT_WITHHELD, CONSENT_GRANTED]) { + let caught: unknown; + try { + admitToLiveModel(USER_DERIVED_PAYLOAD, CLOUD, consent); + throw new Error("admitToLiveModel did not throw; the gate failed to block"); + } catch (error) { + caught = error; + } + expect(caught).toBeInstanceOf(LiveEgressRefusal); + const refusal = caught as LiveEgressRefusal; + expect(refusal.dataClass).toBe("PRIVATE"); + expect(refusal.decision.action).toBe("block"); + } + }); + + test("2/2 across both consent states: allowCount stays 0 -- no consent value produces an admitted result", () => { + let admittedCount = 0; + for (const consent of [CONSENT_WITHHELD, CONSENT_GRANTED]) { + try { + admitToLiveModel(USER_DERIVED_PAYLOAD, CLOUD, consent); + admittedCount += 1; + } catch { + // expected: a refusal, not an admission. + } + } + expect(admittedCount).toBe(0); + }); +}); + +describe("admitToLiveModel (P0, NEVER-293.10): the gate cannot be bypassed by ignoring its result", () => { + // THE SHAPE CHOSEN: a block THROWS rather than returning a boolean a + // caller could examine and skip past. This test demonstrates that a + // caller who calls `admitToLiveModel` and then writes code AS IF it had + // returned normally can never reach that code on a block -- the throw + // unwinds straight past it, and the only way to "proceed anyway" would be + // to wrap the call in a try/catch and explicitly disregard the caught + // `LiveEgressRefusal`, which is a visible, greppable act, not an ignored + // return value. + test("code written after the call, with no exception handling, never executes when the payload is PRIVATE", () => { + let reachedAfterCall = false; + expect(() => { + admitToLiveModel(USER_DERIVED_PAYLOAD, CLOUD, CONSENT_GRANTED); + // If admitToLiveModel returned instead of throwing, this line would + // run and the payload would proceed to a cloud model. + reachedAfterCall = true; + }).toThrow(LiveEgressRefusal); + expect(reachedAfterCall).toBe(false); + }); + + test("there is no boolean return value to ignore: the function's return type carries no field named allowed/ok/blocked that a caller could fail to check", () => { + const admitted = admitToLiveModel(PAYLOAD_BY_CLASS.PUBLIC, CLOUD, CONSENT_WITHHELD); + // The only way to obtain a result at all is for the gate to have already + // allowed it -- `decision.action` is narrowed to the literal "allow" at + // the type level for anything admitToLiveModel returns without + // throwing, so there is no reachable "block" value to inspect and + // ignore. + expect(admitted.decision.action).toBe("allow"); + expect("allowed" in admitted).toBe(false); + expect("ok" in admitted).toBe(false); + }); +}); diff --git a/scripts/live-rung.ts b/scripts/live-rung.ts new file mode 100644 index 0000000..ea002e7 --- /dev/null +++ b/scripts/live-rung.ts @@ -0,0 +1,735 @@ +// The live rung: build target 8 of docs/design/stories/293.md. Task #324. +// +// WHAT THIS SCRIPT IS, AND EQUALLY WHAT IT IS NOT. +// +// `bun run skill-harness` (scripts/skill-harness.ts) runs the OTHER +// population -- the harness's own TRANSCRIPTION of a skill, against a FAKE +// forge, never the model reading the markdown (gate ruling G2). This script +// is the live rung's population: a MODEL reads a `SKILL.md` and performs it, +// against a REAL forge. The model is driven by a human operator's agent +// host, NOT by this script. So this script's job is NOT to execute a skill; +// it is to be the RECORDING SEAM the operator drives every command through, +// so a real, model-driven run produces an honest, machine-checked artifact. +// That is why it is subcommand-shaped (init / mark / exec / report) rather +// than a single run: the operator drives it, one command at a time, over +// however long the model takes. +// +// THE SHAPE, MATCHING scripts/skill-harness.ts's OWN THIN-WRAPPER PATTERN. +// `repoRoot` from `import.meta.dir`, never `process.cwd()` (lint rule +// `no-process-cwd`, though scripts/ sits outside the linted project). Each +// subcommand computes an exit code and returns it; `main()` calls +// `process.exit(exitCode)` LAST, after any `finally` has already run -- +// `process.exit` does not unwind a pending `finally` (skill-harness.ts's own +// header explains the leaked-temp-dir failure mode that shape prevents). +// +// A SESSION IS A DIRECTORY, NOT A PROCESS. Because a live run is driven by a +// human across many separate commands (the model calls `exec` once per tool +// call it makes), state cannot live in this process's memory: it must be a +// directory on disk holding a header file (`header.json`) and an append-only +// JSONL log (`log.jsonl`), so any of the four subcommands can be invoked as +// its own process at any time and still see the whole history so far. + +import { appendFileSync, existsSync, mkdirSync, mkdtempSync, readFileSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join } from "node:path"; +import type { Destination } from "../packages/core/src/index"; +import { ALLOWED_EXECUTABLES, createRealPort, createRecordingPort, type RecordedCall } from "../packages/exec/src/index"; +import { + type ArgvKind, + classifyArgv, + countReads, + createFixtureRepo, + isMutatingGhArgv, + LiveEgressRefusal, + admitToLiveModel, + renderGoalsFixture, +} from "../packages/harness/src/index"; + +// --------------------------------------------------------------------------- +// The egress gate's fixed inputs. +// --------------------------------------------------------------------------- + +// A cloud destination -- the one a live model call actually reaches. Vendor +// name is illustrative only; `checkEgress` (packages/core/src/guards/ +// egress.ts) does not branch on it, only on `locality` and the payload's own +// class. +const CLOUD_DESTINATION: Destination = { vendor: "live-rung-cloud-model", locality: "cloud" }; + +// The declared-synthetic goals fixture (#319, packages/harness/src/ +// goals-fixture.ts) wrapped in a payload keyed `work_state` -- one of the +// INTERNAL_KEY_TOKENS packages/core/src/classify/recognisers.ts recognises, +// so this classifies INTERNAL (project code / work state), never PRIVATE, +// and clears the cloud gate. This is a DELIBERATE CHOICE OF KEY, not an +// accident: `classify()`'s own default rule is that an UNRECOGNISED key +// resolves PRIVATE (packages/core/src/classify/classify.ts's header), so a +// goals payload wrapped under some other, unrecognised key would be refused +// here for the wrong reason -- not because it is unsafe, but because this +// script picked a key the classifier has never heard of. `work_state` is the +// one INTERNAL key whose plain-English meaning ("work state") actually +// describes a goals document. +// Computed ONCE, here -- the payload the egress gate checks below and the +// file this script later writes into the fixture worktree (`cmdInit`'s own +// `USER/GOALS/GOALS.md`) are PROVABLY the same string, never two +// hand-authored `renderGoalsFixture({ goals: [...] })` literals that could +// drift apart. This is the exact same discipline `userDerivedPayload`'s own +// comment argues for below; test/live-rung.test.ts asserts +// `renderGoalsFixture` is called exactly once in this file for that reason. +const SYNTHETIC_GOALS_TEXT = renderGoalsFixture({ goals: ["ship the live rung's recording seam (Task #324)"] }); + +function syntheticGoalsPayload(): unknown { + return { work_state: SYNTHETIC_GOALS_TEXT }; +} + +// A `USER/`-derived payload: PRIVATE by the fail-safe path rule +// (packages/core/src/classify/path.ts's `USER/` rule, CLAIM-15.2), built to +// the exact same shape packages/harness/test/live-egress.test.ts's own +// `USER_DERIVED_PAYLOAD` uses, so this script's demonstration and that +// suite's unit tests are provably the same fixture rather than two +// hand-authored strings that could drift apart. Synthetic, never a real +// private-repo path -- `USER/` cannot be committed to a public tree at all, +// so there is no real fixture this could be instead. +// +// `USER/`-derived content is `PRIVATE` by fail-safe default and reaches no +// cloud model **under any consent** (packages/core/src/guards/egress.ts's +// Decision 2 of docs/design/stories/243.md) -- PROVIDED IT REACHES THE GATE +// under a key `classify()` has never heard of, or one this table maps to +// PRIVATE/SECRET. THE EGRESS GATE IS NOT WHAT STOPS A LIVE RUN OF +// `goal-create` AGAINST THE REAL GOALS SOURCE: a real `USER/GOALS/GOALS.md`, +// wrapped in the exact same `work_state` key `syntheticGoalsPayload` above +// uses, would classify INTERNAL and clear the gate exactly as the synthetic +// fixture does -- `classify()` resolves purely by KEY NAME +// (packages/core/src/classify/recognisers.ts), it never inspects a string +// value's content for an embedded `USER/` path, so `classifyPath`'s own +// `USER/`-root rule (path.ts, CLAIM-15.2) is not even wired into `classify()` +// today. What actually stops a run against the real source is narrower and +// duller: the goals payload above is built from `renderGoalsFixture` with no +// flag, argument or code path that can redirect it to a real file. Two +// different protections guarding two different things -- conflating them +// would leave a hole nobody is watching. +function userDerivedPayload(): unknown { + return { source_path: "USER/HEALTH/labs.yaml", contents: "synthetic-only, not a real record" }; +} + +// --------------------------------------------------------------------------- +// Session file shapes. +// --------------------------------------------------------------------------- + +interface SessionHeader { + readonly repo: string; + readonly fixtureRoot: string; + readonly resolvedCommit: string; + readonly startedAt: string; + readonly host: string; +} + +interface MarkRecord { + readonly kind: "mark"; + readonly run: 1 | 2; + readonly at: string; +} + +interface ExecRecord { + readonly kind: "exec"; + readonly run: 1 | 2; + readonly argv: readonly string[]; + readonly exitCode: number; + readonly at: string; + readonly stdoutBytes: number; + readonly stderrBytes: number; +} + +type LogRecord = MarkRecord | ExecRecord; + +function headerPath(sessionDir: string): string { + return join(sessionDir, "header.json"); +} + +function logPath(sessionDir: string): string { + return join(sessionDir, "log.jsonl"); +} + +function loadHeader(sessionDir: string): SessionHeader { + const path = headerPath(sessionDir); + if (!existsSync(path)) { + throw new Error( + `session not initialized: no header.json found at ${sessionDir}; run \`bun run live-rung init\` first`, + ); + } + return JSON.parse(readFileSync(path, "utf8")) as SessionHeader; +} + +function readLog(sessionDir: string): readonly LogRecord[] { + const path = logPath(sessionDir); + if (!existsSync(path)) return []; + const raw = readFileSync(path, "utf8"); + return raw + .split("\n") + .filter((line) => line.trim().length > 0) + .map((line) => JSON.parse(line) as LogRecord); +} + +function appendLog(sessionDir: string, record: LogRecord): void { + appendFileSync(logPath(sessionDir), `${JSON.stringify(record)}\n`, "utf8"); +} + +/** The run every subsequent `exec` belongs to: the LAST `mark` record seen, + * scanned from the log itself rather than trusted from caller state, so + * `exec` and `report` can never disagree with what actually happened. */ +function lastMarkedRun(records: readonly LogRecord[]): 1 | 2 | undefined { + let last: 1 | 2 | undefined; + for (const record of records) { + if (record.kind === "mark") last = record.run; + } + return last; +} + +// --------------------------------------------------------------------------- +// init +// --------------------------------------------------------------------------- + +function extractFlag(argv: readonly string[], name: string): string | undefined { + const index = argv.indexOf(name); + if (index === -1) return undefined; + return argv[index + 1]; +} + +async function cmdInit(argv: readonly string[], repoRoot: string): Promise { + // =========================================================================== + // THE EGRESS GATE RUNS HERE, FIRST, BEFORE ANYTHING ELSE -- before argument + // validation, before touching the filesystem, before anything a caller + // could construe as "the run has begun". A gate that ran after setup could + // leave a half-built session behind it on refusal; this one leaves nothing. + // =========================================================================== + + // TEST-ONLY SEAM. `--force-private-goals-payload` is not documented in + // `printUsage()` and no operator should ever pass it: it substitutes the + // USER/-derived (PRIVATE) payload for the declared-synthetic goals fixture + // in THIS FIRST gate check only (the converse demonstration below is + // unaffected), so test/live-rung.test.ts can drive the "the payload does + // not clear the gate" branch without this script inventing a second, + // untested code path to do it. + const goalsPayload = argv.includes("--force-private-goals-payload") ? userDerivedPayload() : syntheticGoalsPayload(); + try { + const admitted = admitToLiveModel(goalsPayload, CLOUD_DESTINATION); + console.log( + `live-rung init: egress gate ADMITTED the declared-synthetic goals fixture ` + + `(class ${admitted.dataClass}, destination ${CLOUD_DESTINATION.locality}/${CLOUD_DESTINATION.vendor})`, + ); + } catch (error) { + if (!(error instanceof LiveEgressRefusal)) throw error; + console.error( + `live-rung init: REFUSED to start -- the egress gate blocked the declared-synthetic goals ` + + `fixture itself (class ${error.dataClass}): ${error.message}. Creating nothing.`, + ); + return 1; + } + + // THE CONVERSE, DEMONSTRATED ON THE SAME LIVE PATH. A gate that cannot be + // shown refusing something is not a gate -- it is an unexercised branch. + const userPayload = userDerivedPayload(); + try { + admitToLiveModel(userPayload, CLOUD_DESTINATION); + // Reaching this line means the gate FAILED to refuse a USER/-derived, + // PRIVATE-by-fail-safe-default payload against a cloud destination. That + // is the security defect NEVER-293.10 exists to catch, and it is fatal + // to the whole live rung: refuse to start rather than proceed under a + // gate just shown not to work. + console.error( + "live-rung init: REFUSED to start -- the egress gate did NOT refuse a USER/-derived payload " + + "against a cloud destination. A gate that cannot be shown refusing on the live path is not a " + + "gate. Creating nothing.", + ); + return 1; + } catch (error) { + if (!(error instanceof LiveEgressRefusal)) throw error; + console.log( + `live-rung init: egress gate correctly REFUSED a USER/-derived payload ` + + `(class ${error.dataClass}): ${error.message}`, + ); + } + + // =========================================================================== + // Everything below this line only runs once both gate demonstrations above + // have completed exactly as required. + // =========================================================================== + + const session = extractFlag(argv, "--session"); + const repo = extractFlag(argv, "--repo"); + const host = extractFlag(argv, "--host"); + + if (session === undefined) { + console.error("live-rung init: --session is required"); + return 1; + } + if (repo === undefined) { + console.error("live-rung init: --repo is required"); + return 1; + } + // `host` IS AN INPUT, NEVER DETECTED. It names the agent host driving the + // model (e.g. "opencode", "claude-code" -- references/model-routing.md + // :30-32's sense of "host"), not a machine hostname, and there is no + // reliable way for this process to infer which agent host is calling it. + // An omitted `--host` is refused rather than defaulted, so a session + // header can never silently attribute a run to the wrong host. + if (host === undefined) { + console.error( + "live-rung init: --host is required (the AGENT HOST driving the model, e.g. " + + '"opencode" -- references/model-routing.md:30-32 -- never a machine hostname, and never guessed)', + ); + return 1; + } + + if (existsSync(headerPath(session))) { + console.error(`live-rung init: session already initialized at ${session} (header.json exists)`); + return 1; + } + + mkdirSync(session, { recursive: true }); + + const adminPort = createRealPort(); + + const resolvedCommitResult = await adminPort.run(["git", "-C", repoRoot, "rev-parse", "HEAD"]); + if (resolvedCommitResult.exitCode !== 0) { + console.error( + `live-rung init: failed to resolve this repository's own HEAD commit: ${resolvedCommitResult.stderr.trim()}`, + ); + return 1; + } + const resolvedCommit = resolvedCommitResult.stdout.trim(); + + const fixtureRoot = mkdtempSync(join(tmpdir(), "iai-live-rung-fixture-")); + await createFixtureRepo(adminPort, fixtureRoot, { + name: "live-rung-init", + files: [ + { + path: "USER/GOALS/GOALS.md", + contents: SYNTHETIC_GOALS_TEXT, + }, + ], + }); + + const header: SessionHeader = { + repo, + fixtureRoot, + resolvedCommit, + startedAt: new Date().toISOString(), + host, + }; + writeFileSync(headerPath(session), `${JSON.stringify(header, null, 2)}\n`, "utf8"); + writeFileSync(logPath(session), "", "utf8"); + + console.log(`live-rung init: session ready at ${session}`); + console.log(`live-rung init: fixture worktree at ${fixtureRoot}`); + console.log(`live-rung init: resolved commit ${resolvedCommit}`); + console.log(`live-rung init: target repo ${repo}, host ${host}`); + return 0; +} + +// --------------------------------------------------------------------------- +// mark +// --------------------------------------------------------------------------- + +function cmdMark(argv: readonly string[]): number { + const session = extractFlag(argv, "--session"); + const runRaw = extractFlag(argv, "--run"); + + if (session === undefined) { + console.error("live-rung mark: --session is required"); + return 1; + } + if (runRaw === undefined) { + console.error("live-rung mark: --run <1|2> is required"); + return 1; + } + if (runRaw !== "1" && runRaw !== "2") { + console.error(`live-rung mark: --run must be exactly "1" or "2"; got "${runRaw}"`); + return 1; + } + const run = runRaw === "1" ? 1 : 2; + + let header: SessionHeader; + try { + header = loadHeader(session); + } catch (error) { + console.error(`live-rung mark: ${(error as Error).message}`); + return 1; + } + void header; // existence of the header is the only thing this needs to check. + + const records = readLog(session); + const last = lastMarkedRun(records); + if (last !== undefined && run < last) { + console.error( + `live-rung mark: refusing to go backwards -- run ${last} was already marked, cannot mark run ${run}`, + ); + return 1; + } + + appendLog(session, { kind: "mark", run, at: new Date().toISOString() }); + console.log(`live-rung mark: run ${run} begins`); + return 0; +} + +// --------------------------------------------------------------------------- +// exec +// --------------------------------------------------------------------------- + +async function cmdExec(argv: readonly string[]): Promise { + const sepIndex = argv.indexOf("--"); + if (sepIndex === -1) { + console.error('live-rung exec: expected `--` separating flags from the argv to run, e.g. `exec --session S -- gh issue list`'); + return 1; + } + const flags = argv.slice(0, sepIndex); + const childArgv = argv.slice(sepIndex + 1); + + const session = extractFlag(flags, "--session"); + if (session === undefined) { + console.error("live-rung exec: --session is required"); + return 1; + } + if (childArgv.length === 0) { + console.error("live-rung exec: no argv given to run after `--`"); + return 1; + } + + let header: SessionHeader; + try { + header = loadHeader(session); + } catch (error) { + console.error(`live-rung exec: ${(error as Error).message}`); + return 1; + } + + const records = readLog(session); + const run = lastMarkedRun(records); + if (run === undefined) { + console.error("live-rung exec: no run has been marked yet; run `live-rung mark --session --run 1` first"); + return 1; + } + + // A mutating `gh` argv is refused unless the session header names a target + // repo. REUSES `isMutatingGhArgv` (packages/harness/src/fake-forge.ts) -- + // the ONE mutation classifier this whole harness shares, never restated. + if (childArgv[0] === "gh" && isMutatingGhArgv(childArgv) && (header.repo === undefined || header.repo === null || header.repo.length === 0)) { + console.error( + `live-rung exec: refusing to run \`${childArgv.join(" ")}\` -- this is a MUTATING gh argv and the ` + + "session header names no target repo. A live mutation against an unnamed forge is exactly the " + + "kind of accident this seam exists to prevent.", + ); + return 1; + } + + // THE REAL PORT. Executables outside packages/exec/src/port.ts's + // `ALLOWED_EXECUTABLES` are refused by `checkArgv` inside `createRealPort` + // itself (exit code `REFUSED_EXIT_CODE`, 126) -- this script does not + // reimplement that check, it inherits it by routing every argv through the + // real port rather than through a second launcher. `ALLOWED_EXECUTABLES` + // is imported above only so this comment, and the reader, can name it. + void ALLOWED_EXECUTABLES; + + const port = createRecordingPort(createRealPort()); + const result = await port.run(childArgv); + + // PASS STDOUT/STDERR THROUGH TO THE CALLER, UNCHANGED. The operator must + // see exactly what a real `gh` (or `git`) said, including a rejection -- + // this is the first-contact record for #330, and a summarised or + // reformatted echo would already be a lossy transcription of the one + // thing this command exists to observe honestly. + process.stdout.write(result.stdout); + process.stderr.write(result.stderr); + + // DO NOT LOG stdout/stderr CONTENTS. A real forge response can carry + // content this repository has no classification for -- an issue body, a + // PR diff, an error message quoting a private field name -- and this + // script has no classifier in its own path (that is `packages/core/src/ + // classify`'s job, and it is not consulted here). Logging only the BYTE + // LENGTH of each stream lets `report` show how much came back without + // ever writing a byte of it to disk a second time. + appendLog(session, { + kind: "exec", + run, + argv: childArgv, + exitCode: result.exitCode, + at: new Date().toISOString(), + stdoutBytes: Buffer.byteLength(result.stdout, "utf8"), + stderrBytes: Buffer.byteLength(result.stderr, "utf8"), + }); + + return result.exitCode; +} + +// --------------------------------------------------------------------------- +// report +// --------------------------------------------------------------------------- + +const ARGV_KINDS: readonly ArgvKind[] = ["forge-read", "forge-mutation", "git-read", "git-write", "unclassified"]; + +interface RunStats { + readonly totalCalls: number; + readonly reads: number; + readonly mutations: number; + readonly kinds: Readonly>; +} + +function computeRunStats(calls: readonly RecordedCall[]): RunStats { + const kinds: Record = { + "forge-read": 0, + "forge-mutation": 0, + "git-read": 0, + "git-write": 0, + unclassified: 0, + }; + for (const call of calls) { + kinds[classifyArgv(call.argv)] += 1; + } + return { + totalCalls: calls.length, + // REUSES `countReads` (packages/harness/src/argv-kind.ts) rather than + // `calls.length - mutations`, per that module's own header: a `git` + // write call is neither a forge mutation nor a read, and the + // subtraction would misscore it as a read. + reads: countReads(calls), + mutations: calls.filter((call) => isMutatingGhArgv(call.argv)).length, + kinds, + }; +} + +interface NonZeroExit { + readonly argv: readonly string[]; + readonly exitCode: number; +} + +interface LiveRungArtifact { + readonly schemaVersion: 1; + readonly generatedAt: string; + readonly session: SessionHeader; + // `null` unless `--token-cost` was supplied. STRUCTURALLY IMPOSSIBLE for + // this field to assert a cost nobody measured: there is exactly one + // assignment site for this field (below) and it is guarded by the + // presence of the flag, never inferred or defaulted to zero. + readonly tokenCost: number | null; + readonly runs: Readonly>; + readonly overall: RunStats; + // The first-contact record for #330: every DISTINCT argv that exited + // non-zero, with its exit code. This is the most valuable output of the + // whole command -- the S2.1 precedent is sixty passing tests and 7 of 9 + // real verbs rejected on first contact, so a live session with zero + // entries here is the surprising result, not the expected one. + readonly nonZeroExits: readonly NonZeroExit[]; +} + +function renderRunStatsLine(label: string, stats: RunStats): string { + const kindsText = ARGV_KINDS.map((kind) => `${kind}=${stats.kinds[kind]}`).join(", "); + return ` ${label}: ${stats.totalCalls} calls, ${stats.reads} reads, ${stats.mutations} gh mutations (${kindsText})`; +} + +function renderReportText(artifact: LiveRungArtifact): string { + const lines: string[] = []; + lines.push("live rung report"); + lines.push(` session repo: ${artifact.session.repo}`); + lines.push(` session host: ${artifact.session.host}`); + lines.push(` fixture root: ${artifact.session.fixtureRoot}`); + lines.push(` resolved commit: ${artifact.session.resolvedCommit}`); + lines.push(` started at: ${artifact.session.startedAt}`); + lines.push(""); + lines.push("per run:"); + const runKeys = Object.keys(artifact.runs).sort(); + if (runKeys.length === 0) { + lines.push(" (no run has been marked yet)"); + } else { + for (const key of runKeys) { + lines.push(renderRunStatsLine(`run ${key}`, artifact.runs[key] as RunStats)); + } + } + lines.push(renderRunStatsLine("overall", artifact.overall)); + lines.push(""); + // `tokenCost` says so IN WORDS when null -- a bare `null` in prose reads + // as a typo, not a deliberate absence. + lines.push( + artifact.tokenCost === null + ? "token cost: not supplied (pass --token-cost to record a measured cost)" + : `token cost: ${artifact.tokenCost}`, + ); + lines.push(""); + if (artifact.nonZeroExits.length === 0) { + lines.push("non-zero exits: none recorded"); + } else { + lines.push(`non-zero exits (${artifact.nonZeroExits.length} distinct argv):`); + for (const entry of artifact.nonZeroExits) { + lines.push(` exit ${entry.exitCode}: ${entry.argv.join(" ")}`); + } + } + return lines.join("\n"); +} + +async function cmdReport(argv: readonly string[]): Promise { + const session = extractFlag(argv, "--session"); + const out = extractFlag(argv, "--out"); + const tokenCostRaw = extractFlag(argv, "--token-cost"); + + if (session === undefined) { + console.error("live-rung report: --session is required"); + return 1; + } + + let header: SessionHeader; + try { + header = loadHeader(session); + } catch (error) { + console.error(`live-rung report: ${(error as Error).message}`); + return 1; + } + + // `tokenCost` is `null` UNLESS `--token-cost` is supplied -- there is no + // other way for this variable to become a number below, so the artifact + // can never assert a cost nobody measured. + let tokenCost: number | null = null; + if (tokenCostRaw !== undefined) { + const parsed = Number(tokenCostRaw); + if (!Number.isFinite(parsed) || parsed < 0) { + console.error( + `live-rung report: --token-cost must be a finite, non-negative number; got "${tokenCostRaw}"`, + ); + return 1; + } + tokenCost = parsed; + } + + const records = readLog(session); + const execRecords = records.filter((record): record is ExecRecord => record.kind === "exec"); + + const byRun = new Map(); + for (const record of execRecords) { + const calls = byRun.get(record.run) ?? []; + calls.push({ argv: record.argv, exitCode: record.exitCode }); + byRun.set(record.run, calls); + } + + const runs: Record = {}; + for (const [run, calls] of [...byRun.entries()].sort((a, b) => a[0] - b[0])) { + runs[String(run)] = computeRunStats(calls); + } + + const allCalls: RecordedCall[] = execRecords.map((record) => ({ argv: record.argv, exitCode: record.exitCode })); + const overall = computeRunStats(allCalls); + + // DISTINCT by argv, keeping the MOST RECENT exit code observed for it -- + // a live session where the operator retried the same command after fixing + // something should report the latest outcome, not the first. + const nonZeroByArgv = new Map(); + for (const record of execRecords) { + if (record.exitCode === 0) continue; + nonZeroByArgv.set(JSON.stringify(record.argv), { argv: record.argv, exitCode: record.exitCode }); + } + + const artifact: LiveRungArtifact = { + schemaVersion: 1, + generatedAt: new Date().toISOString(), + session: header, + tokenCost, + runs, + overall, + nonZeroExits: [...nonZeroByArgv.values()], + }; + + const outPath = out ?? join(session, "report.json"); + mkdirSync(dirname(outPath), { recursive: true }); + writeFileSync(outPath, `${JSON.stringify(artifact, null, 2)}\n`, "utf8"); + + console.log(renderReportText(artifact)); + console.log(`live-rung report: artifact written to ${outPath}`); + + // THE REPORT COMMAND'S EXIT CODE IS ABOUT REPORTING, NOT ABOUT WHETHER THE + // RUN WAS CLEAN. It returns 0 whenever the artifact was produced, even + // when `nonZeroExits` is non-empty or a run mutated nothing -- a future + // reader will be tempted to make this fail on a rejection so CI "catches" + // it, and that would DESTROY THE RECORD: the live rung is never a required + // check (NEVER-293.10), and a report command that refuses to finish over + // a rejected verb would make exactly the first-contact rejections this + // command exists to surface disappear behind a nonzero exit nobody reads + // past. + return 0; +} + +// --------------------------------------------------------------------------- +// dispatch +// --------------------------------------------------------------------------- + +function printUsage(): void { + console.log(`live-rung: the recording seam for a real, model-driven skill run against a real forge. + +Never executes a skill itself -- a human operator's agent host drives the +model, and every command the model issues is routed through \`exec\` below so +it is recorded honestly. Never a required CI check (NEVER-293.10). + +USAGE: + bun run live-rung init --session --repo --host + bun run live-rung mark --session --run <1|2> + bun run live-rung exec --session -- + bun run live-rung report --session [--out ] [--token-cost ] + +init Runs the egress gate against the declared-synthetic goals fixture + (must be admitted) and against a USER/-derived payload (must be + refused), THEN creates a real git fixture worktree and writes the + session header. Creates nothing if either gate check is wrong. + +mark Records a run boundary (1 or 2). Every subsequent \`exec\` belongs to + that run until the next \`mark\`. Refuses a non-1/2 run index and + refuses going backwards. + +exec Runs through the REAL port (packages/exec/src/real.ts). + Refuses an executable outside packages/exec/src/port.ts's + ALLOWED_EXECUTABLES, and refuses a mutating \`gh\` argv when the + session has no target repo. Passes the child's stdout/stderr through + unchanged and exits with the child's exit code. Logs the argv, exit + code, run index and byte lengths of stdout/stderr -- never their + contents. + +report Reads the session log and emits a JSON artifact plus a human-readable + summary: per-run and overall call/read/mutation counts, the measured + token cost (or an explicit "not supplied"), and every DISTINCT argv + that exited non-zero. Exits 0 whenever the artifact was produced, + regardless of what it reports. +`); +} + +async function main(): Promise { + const repoRoot = join(import.meta.dir, ".."); + const [subcommand, ...rest] = process.argv.slice(2); + + let exitCode: number; + switch (subcommand) { + case "init": + exitCode = await cmdInit(rest, repoRoot); + break; + case "mark": + exitCode = cmdMark(rest); + break; + case "exec": + exitCode = await cmdExec(rest); + break; + case "report": + exitCode = await cmdReport(rest); + break; + case "--help": + case "-h": + printUsage(); + exitCode = 0; + break; + case undefined: + printUsage(); + exitCode = 1; + break; + default: + console.error(`live-rung: unknown subcommand "${subcommand}"`); + printUsage(); + exitCode = 1; + } + + process.exit(exitCode); +} + +if (import.meta.main) { + await main(); +} diff --git a/test/ci-workflow.test.ts b/test/ci-workflow.test.ts index c037c80..1726534 100644 --- a/test/ci-workflow.test.ts +++ b/test/ci-workflow.test.ts @@ -17,11 +17,13 @@ import { join } from "node:path"; const repoRoot = join(import.meta.dir, ".."); const ciYmlPath = join(repoRoot, ".github", "workflows", "ci.yml"); +const liveYmlPath = join(repoRoot, ".github", "workflows", "live.yml"); const requiredChecksPath = join(repoRoot, "scripts", "verify-required-checks.sh"); const workflowHygienePath = join(repoRoot, "scripts", "verify-workflow-hygiene.sh"); const packageJsonPath = join(repoRoot, "package.json"); const ciYmlText = readFileSync(ciYmlPath, "utf8"); +const liveYmlText = readFileSync(liveYmlPath, "utf8"); const requiredChecksText = readFileSync(requiredChecksPath, "utf8"); const workflowHygieneText = readFileSync(workflowHygienePath, "utf8"); const packageJsonText = readFileSync(packageJsonPath, "utf8"); @@ -81,6 +83,50 @@ describe("job names parsed from ci.yml's `jobs:` block", () => { const ciJobNames = ciJobs.map((j) => j.name); const ciJobNameSet = new Set(ciJobNames); +// =========================================================================== +// live.yml — Task #324's NEVER-293.10 seam check. Parsed with the SAME job +// parser as ci.yml (parseCiYmlJobs takes a `text` argument; it is not +// actually specific to ci.yml's filename, only to the `jobs:`-block shape +// every workflow file shares), plus a small trigger-block parser for `on:`. +// =========================================================================== + +function parseWorkflowTriggers(text: string): ReadonlySet { + const lines = text.split("\n"); + const onIdx = lines.findIndex((l) => l === "on:"); + if (onIdx === -1) throw new Error("`on:` top-level key not found in workflow file"); + + const triggers = new Set(); + for (let i = onIdx + 1; i < lines.length; i += 1) { + const line = lines[i] as string; + if (line.trim().length === 0) continue; + const nested = /^ {2}([A-Za-z0-9_-]+):/.exec(line); + if (nested) { + triggers.add(nested[1] as string); + continue; + } + // A non-blank line that is NOT a 2-space-indented key means the `on:` + // block has dedented back to a top-level key (e.g. `env:`) — stop here + // rather than scanning the rest of the file. + break; + } + return triggers; +} + +const liveJobs = parseCiYmlJobs(liveYmlText); +const liveJobNames = liveJobs.map((j) => j.name); +const liveJobNameSet = new Set(liveJobNames); +const liveTriggers = parseWorkflowTriggers(liveYmlText); + +describe("live.yml job names parsed from its `jobs:` block", () => { + test("0. denominator non-zero first: at least one job was parsed", () => { + expect(liveJobs.length).toBeGreaterThan(0); + }); + + test("job names are unique (cardinality equals count)", () => { + expect(liveJobNameSet.size).toBe(liveJobNames.length); + }); +}); + // =========================================================================== // The FOUR hardcoded lists. Each parse MUST assert it matched before its // result is used for anything — a silently-empty match would make every @@ -283,6 +329,146 @@ describe("no required job carries a job-level `if:` (NEVER-9.8, all six)", () => // silently no-op (or error) only when the job actually runs in CI. // =========================================================================== +// =========================================================================== +// NEVER-293.10 (Task #324, cases 13 and 14 of docs/test-plans/293-plan.md): +// the live rung is never a required CI check. `live.yml` is a SEPARATE +// workflow file, on purpose (see its own header comment for the full +// argument); these tests assert the pin mechanically, from both files, read +// from disk at run time. +// =========================================================================== + +describe("NEVER-293.10 case 13: the live job is absent from all four hardcoded required lists, and the required count agrees before and after", () => { + test("0. denominator non-zero first: all four hardcoded lists are non-empty", () => { + expect(requiredContextsArray.length).toBeGreaterThan(0); + expect(inlineContextsList.length).toBeGreaterThan(0); + expect(requiredJobsArray.length).toBeGreaterThan(0); + expect(pythonRequiredList.length).toBeGreaterThan(0); + }); + + test("the four lists still agree with each other here too (re-asserted, not assumed from the earlier block)", () => { + expect(requiredContextsArray).toEqual(inlineContextsList); + expect(requiredContextsArray).toEqual(requiredJobsArray); + expect(requiredContextsArray).toEqual(pythonRequiredList); + }); + + // The count is read, never hardcoded to `6`: issue #334 will promote + // `seam` and make this seven. A test that hardcoded 6 here would have to + // be edited the moment that promotion lands, which is exactly the + // coupling case 13 exists to prevent for a DIFFERENT job (`live`) — this + // test must not reintroduce that same coupling for its own assertion. + test("the required-context count agrees BEFORE and AFTER an independent, later re-read of the same file from disk", () => { + const before = requiredContextsArray.length; + expect(before).toBeGreaterThan(0); + + const freshText = readFileSync(requiredChecksPath, "utf8"); + const freshMatch = /REQUIRED_CONTEXTS=\(([^)]*)\)/.exec(freshText); + expect(freshMatch).not.toBeNull(); + const freshArray = ((freshMatch as RegExpExecArray)[1] as string).trim().split(/\s+/); + const after = freshArray.length; + + expect(after).toBeGreaterThan(0); + expect(after).toBe(before); + }); + + test("the live job actually exists in live.yml (sanity: there is something to check the absence of)", () => { + expect(liveJobNameSet.has("live")).toBe(true); + }); + + test("the live job's name is absent from all four hardcoded required lists", () => { + expect(requiredContextsArray).not.toContain("live"); + expect(inlineContextsList).not.toContain("live"); + expect(requiredJobsArray).not.toContain("live"); + expect(pythonRequiredList).not.toContain("live"); + }); +}); + +describe("NEVER-293.10 case 14: every required job in ci.yml carries no `if:` key", () => { + test("0. denominator non-zero first: the required-job count is non-zero", () => { + expect(requiredContextsArray.length).toBeGreaterThan(0); + }); + + test("none of the required jobs carry a job-level `if:` key", () => { + const withIf = requiredContextsArray.filter((name) => hasJobLevelIf(jobBlockByName(name))); + expect(withIf).toEqual([]); + }); +}); + +describe("live.yml's triggers are exactly {workflow_dispatch}", () => { + test("0. denominator non-zero first: at least one trigger key was parsed from live.yml's `on:` block", () => { + expect(liveTriggers.size).toBeGreaterThan(0); + }); + + test("the trigger set equals EXACTLY {workflow_dispatch} — a `pull_request` trigger added later must fail this test", () => { + expect([...liveTriggers].sort()).toEqual(["workflow_dispatch"]); + }); +}); + +describe("live.yml's job set is disjoint from every required list, and from ci.yml's job set", () => { + test("0. denominator non-zero first: live.yml has at least one job, and ci.yml's job set is non-empty", () => { + expect(liveJobNameSet.size).toBeGreaterThan(0); + expect(ciJobNameSet.size).toBeGreaterThan(0); + }); + + test("no job in live.yml appears in any of the four hardcoded required lists", () => { + for (const name of liveJobNames) { + expect(requiredContextsArray).not.toContain(name); + expect(inlineContextsList).not.toContain(name); + expect(requiredJobsArray).not.toContain(name); + expect(pythonRequiredList).not.toContain(name); + } + }); + + test("live.yml's job set and ci.yml's job set are disjoint, in both directions", () => { + const overlapLiveInCi = liveJobNames.filter((n) => ciJobNameSet.has(n)); + const overlapCiInLive = ciJobNames.filter((n) => liveJobNameSet.has(n)); + expect(overlapLiveInCi).toEqual([]); + expect(overlapCiInLive).toEqual([]); + }); +}); + +// live.yml's own not-required classification. Kept as its own record (not +// merged into ci.yml's NOT_REQUIRED) because job names are only unique +// WITHIN a workflow file — a flat merged dictionary would silently collide +// if some future job in either file reused a name. +const LIVE_NOT_REQUIRED: Readonly> = { + live: "workflow_dispatch-only seam check for scripts/live-rung.ts; see live.yml's own header comment for the full argument — this job structurally cannot fire on a pull_request, so it can never report on one and can never be validly promoted to required (NEVER-293.10)", +}; + +describe("THE PARTITION, EXTENDED ACROSS WORKFLOW FILES: required ∪ NOT_REQUIRED(ci.yml) ∪ NOT_REQUIRED(live.yml) == ci.yml's jobs ∪ live.yml's jobs", () => { + test("0. denominator non-zero first: required, NOT_REQUIRED and LIVE_NOT_REQUIRED are all non-empty", () => { + expect(requiredContextsArray.length).toBeGreaterThan(0); + expect(Object.keys(NOT_REQUIRED).length).toBeGreaterThan(0); + expect(Object.keys(LIVE_NOT_REQUIRED).length).toBeGreaterThan(0); + }); + + test("the three classification sets are pairwise disjoint", () => { + const requiredSet = new Set(requiredContextsArray); + const notRequiredKeys = Object.keys(NOT_REQUIRED); + const liveNotRequiredKeys = Object.keys(LIVE_NOT_REQUIRED); + + expect(notRequiredKeys.filter((k) => requiredSet.has(k))).toEqual([]); + expect(liveNotRequiredKeys.filter((k) => requiredSet.has(k))).toEqual([]); + expect(liveNotRequiredKeys.filter((k) => notRequiredKeys.includes(k))).toEqual([]); + }); + + test("the union equals ci.yml's job set UNION live.yml's job set EXACTLY — a new, unclassified job in either file fails here", () => { + const unionSet = new Set([ + ...requiredContextsArray, + ...Object.keys(NOT_REQUIRED), + ...Object.keys(LIVE_NOT_REQUIRED), + ]); + const allJobsSet = new Set([...ciJobNames, ...liveJobNames]); + + expect(unionSet.size).toBe(allJobsSet.size); + expect([...unionSet].sort()).toEqual([...allJobsSet].sort()); + + const missingFromWorkflows = [...unionSet].filter((j) => !allJobsSet.has(j)); + const missingFromPartition = [...allJobsSet].filter((j) => !unionSet.has(j)); + expect(missingFromWorkflows).toEqual([]); + expect(missingFromPartition).toEqual([]); + }); +}); + describe("`skill-harness` is a real script in the root package.json", () => { const pkg = JSON.parse(packageJsonText) as { scripts?: Record }; diff --git a/test/live-rung.test.ts b/test/live-rung.test.ts new file mode 100644 index 0000000..1e163ef --- /dev/null +++ b/test/live-rung.test.ts @@ -0,0 +1,587 @@ +// Tests for scripts/live-rung.ts, Task #324, build target 8 of +// docs/design/stories/293.md: the live rung, never required. +// +// HOUSE STYLE: denominators asserted non-zero before anything is measured +// against them, cardinality (a Set's size, an array's length after +// deduplication) rather than raw length where duplicates would lie, and +// every JSON.parse is followed by an assertion that the parse actually +// produced the shape expected -- mirrors test/skill-harness.test.ts's own +// spawn helper and CLI-testing conventions. +// +// OFFLINE AND DETERMINISTIC, BY CONSTRUCTION. Every `exec` below runs `git` +// against a LOCAL temp repository this file creates itself, or an +// executable this file already knows is refused before it would ever be +// launched (an executable outside ALLOWED_EXECUTABLES, or a mutating `gh` +// argv refused by the header check before the port is ever consulted). No +// test here reaches the network or a real forge. +import { afterAll, describe, expect, test } from "bun:test"; +import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import { createTempDirs } from "../packages/harness/src/tempdir"; + +const repoRoot = join(import.meta.dir, ".."); +const cli = join(repoRoot, "scripts", "live-rung.ts"); + +const temps = createTempDirs(); +afterAll(() => temps.cleanup()); + +interface Spawned { + readonly stdout: string; + readonly stderr: string; + readonly exitCode: number; +} + +async function run(args: readonly string[]): Promise { + const proc = Bun.spawn(["bun", cli, ...args], { stdout: "pipe", stderr: "pipe", cwd: repoRoot }); + const [stdout, stderr, exitCode] = await Promise.all([ + new Response(proc.stdout).text(), + new Response(proc.stderr).text(), + proc.exited, + ]); + return { stdout, stderr, exitCode }; +} + +interface SessionHeader { + readonly repo: string; + readonly fixtureRoot: string; + readonly resolvedCommit: string; + readonly startedAt: string; + readonly host: string; +} + +function headerPath(session: string): string { + return join(session, "header.json"); +} + +function readHeader(session: string): SessionHeader { + const raw = readFileSync(headerPath(session), "utf8"); + const parsed = JSON.parse(raw) as SessionHeader; + // Every parse asserts it matched: a header.json that parsed but lacked + // the fields this suite depends on would otherwise pass silently. + expect(typeof parsed.repo).toBe("string"); + expect(typeof parsed.fixtureRoot).toBe("string"); + expect(typeof parsed.resolvedCommit).toBe("string"); + expect(typeof parsed.startedAt).toBe("string"); + expect(typeof parsed.host).toBe("string"); + return parsed; +} + +/** A local git repo, never touched over the network -- created with plain + * `Bun.spawnSync` (not the CLI under test) so this helper cannot itself + * hide a defect in the CLI's own git handling. */ +function makeLocalGitRepo(root: string): void { + mkdirSync(root, { recursive: true }); + const runGit = (...args: string[]): void => { + const proc = Bun.spawnSync(["git", "-C", root, ...args], { stdout: "pipe", stderr: "pipe" }); + if (proc.exitCode !== 0) { + throw new Error(`git ${args.join(" ")} failed: ${new TextDecoder().decode(proc.stderr)}`); + } + }; + runGit("init", "-q"); + runGit("config", "user.email", "live-rung-test@iai.invalid"); + runGit("config", "user.name", "live-rung test"); + runGit("config", "init.defaultBranch", "main"); + writeFileSync(join(root, "a.txt"), "seed\n", "utf8"); + runGit("add", "-A"); + runGit("commit", "-q", "-m", "seed"); +} + +// =========================================================================== +// init +// =========================================================================== + +describe("live-rung init: the egress gate runs first", () => { + test( + "1. init refuses when the goals payload does not clear the gate (driven via the test-only " + + "--force-private-goals-payload seam) and creates no session", + async () => { + const session = join(temps.create("iai-live-rung-init-refused-"), "session"); + const spawned = await run([ + "init", + "--session", + session, + "--repo", + "OWNER/REPO", + "--host", + "opencode", + "--force-private-goals-payload", + ]); + + expect(spawned.exitCode).toBe(1); + const combined = spawned.stdout + spawned.stderr; + expect(combined).toContain("REFUSED to start"); + expect(existsSync(session)).toBe(false); + expect(existsSync(headerPath(session))).toBe(false); + }, + 120_000, + ); + + test( + "2. init succeeds on the declared-synthetic fixture: writes a header containing the resolved " + + "commit and the supplied host", + async () => { + const session = join(temps.create("iai-live-rung-init-ok-"), "session"); + const spawned = await run(["init", "--session", session, "--repo", "OWNER/REPO", "--host", "opencode"]); + + expect(spawned.exitCode).toBe(0); + expect(existsSync(headerPath(session))).toBe(true); + + const header = readHeader(session); + expect(header.repo).toBe("OWNER/REPO"); + expect(header.host).toBe("opencode"); + + const commitProc = Bun.spawnSync(["git", "-C", repoRoot, "rev-parse", "HEAD"], { stdout: "pipe" }); + const expectedCommit = new TextDecoder().decode(commitProc.stdout).trim(); + expect(expectedCommit.length).toBeGreaterThan(0); + expect(header.resolvedCommit).toBe(expectedCommit); + + expect(existsSync(header.fixtureRoot)).toBe(true); + }, + 120_000, + ); + + test( + "3. init also demonstrates the converse: it prints that a USER/-derived payload was refused", + async () => { + const session = join(temps.create("iai-live-rung-init-converse-"), "session"); + const spawned = await run(["init", "--session", session, "--repo", "OWNER/REPO", "--host", "opencode"]); + + expect(spawned.exitCode).toBe(0); + expect(spawned.stdout).toContain("correctly REFUSED a USER/-derived payload"); + }, + 120_000, + ); + + test("4. init errors when --host is absent, and creates no session", async () => { + const session = join(temps.create("iai-live-rung-init-no-host-"), "session"); + const spawned = await run(["init", "--session", session, "--repo", "OWNER/REPO"]); + + expect(spawned.exitCode).toBe(1); + expect(spawned.stdout + spawned.stderr).toContain("--host is required"); + expect(existsSync(headerPath(session))).toBe(false); + }, 120_000); +}); + +// =========================================================================== +// exec +// =========================================================================== + +describe("live-rung exec: refusals", () => { + test("5. exec refuses an executable outside ALLOWED_EXECUTABLES", async () => { + const session = join(temps.create("iai-live-rung-exec-disallowed-"), "session"); + const initSpawned = await run(["init", "--session", session, "--repo", "OWNER/REPO", "--host", "opencode"]); + expect(initSpawned.exitCode).toBe(0); + + const markSpawned = await run(["mark", "--session", session, "--run", "1"]); + expect(markSpawned.exitCode).toBe(0); + + const spawned = await run(["exec", "--session", session, "--", "echo", "hello"]); + expect(spawned.exitCode).not.toBe(0); + expect(spawned.stdout + spawned.stderr).toContain("refusing to run"); + }, 120_000); + + test("6. exec refuses a mutating gh argv when the session header names no target repo", async () => { + const session = temps.create("iai-live-rung-exec-no-repo-"); + // A header written directly, WITHOUT a `repo` field -- exercising this + // refusal without touching the network requires exactly this: the + // refusal must fire BEFORE the argv ever reaches the port. + writeFileSync( + headerPath(session), + JSON.stringify({ + fixtureRoot: "/nonexistent", + resolvedCommit: "deadbeef", + startedAt: new Date().toISOString(), + host: "opencode", + }), + "utf8", + ); + writeFileSync(join(session, "log.jsonl"), "", "utf8"); + + const markSpawned = await run(["mark", "--session", session, "--run", "1"]); + expect(markSpawned.exitCode).toBe(0); + + const spawned = await run([ + "exec", + "--session", + session, + "--", + "gh", + "issue", + "create", + "--title", + "x", + "--body", + "y", + ]); + + expect(spawned.exitCode).not.toBe(0); + expect(spawned.stdout + spawned.stderr).toContain("names no target repo"); + + // The refusal happened before any network call: the log records the + // refused attempt, not a network outcome, so this assertion also proves + // no call reached the port. + const log = readFileSync(join(session, "log.jsonl"), "utf8"); + expect(log.includes("issue")).toBe(false); + }, 120_000); + + test("7. exec refuses before any run has been marked", async () => { + const session = join(temps.create("iai-live-rung-exec-no-mark-"), "session"); + const initSpawned = await run(["init", "--session", session, "--repo", "OWNER/REPO", "--host", "opencode"]); + expect(initSpawned.exitCode).toBe(0); + + const spawned = await run(["exec", "--session", session, "--", "git", "--version"]); + expect(spawned.exitCode).not.toBe(0); + expect(spawned.stdout + spawned.stderr).toContain("no run has been marked"); + }, 120_000); +}); + +// =========================================================================== +// mark +// =========================================================================== + +describe("live-rung mark: run index discipline", () => { + test("8. mark refuses a run index that is not 1 or 2", async () => { + const session = join(temps.create("iai-live-rung-mark-invalid-"), "session"); + const initSpawned = await run(["init", "--session", session, "--repo", "OWNER/REPO", "--host", "opencode"]); + expect(initSpawned.exitCode).toBe(0); + + for (const bad of ["0", "3", "abc", "1.5"]) { + const spawned = await run(["mark", "--session", session, "--run", bad]); + expect(spawned.exitCode).not.toBe(0); + } + }, 120_000); + + test("9. mark refuses going backwards from 2 to 1", async () => { + const session = join(temps.create("iai-live-rung-mark-backwards-"), "session"); + const initSpawned = await run(["init", "--session", session, "--repo", "OWNER/REPO", "--host", "opencode"]); + expect(initSpawned.exitCode).toBe(0); + + const mark1 = await run(["mark", "--session", session, "--run", "1"]); + expect(mark1.exitCode).toBe(0); + const mark2 = await run(["mark", "--session", session, "--run", "2"]); + expect(mark2.exitCode).toBe(0); + const markBack = await run(["mark", "--session", session, "--run", "1"]); + expect(markBack.exitCode).not.toBe(0); + expect(markBack.stdout + markBack.stderr).toContain("backwards"); + }, 120_000); +}); + +// =========================================================================== +// exec + report: reads and mutations, counted right, over a LOCAL git repo. +// =========================================================================== + +describe("live-rung exec + report: reads and mutations land under the right run index", () => { + test( + "10. run 1 has one read and one write, run 2 has two reads; report's counts agree, entirely offline", + async () => { + const session = join(temps.create("iai-live-rung-counts-"), "session"); + const localRepo = temps.create("iai-live-rung-local-git-"); + makeLocalGitRepo(localRepo); + + const initSpawned = await run(["init", "--session", session, "--repo", "OWNER/REPO", "--host", "opencode"]); + expect(initSpawned.exitCode).toBe(0); + + expect((await run(["mark", "--session", session, "--run", "1"])).exitCode).toBe(0); + + // Run 1: one read (status), one write (commit). + expect((await run(["exec", "--session", session, "--", "git", "-C", localRepo, "status", "--porcelain"])).exitCode).toBe( + 0, + ); + writeFileSync(join(localRepo, "b.txt"), "more\n", "utf8"); + expect((await run(["exec", "--session", session, "--", "git", "-C", localRepo, "add", "-A"])).exitCode).toBe(0); + expect( + (await run(["exec", "--session", session, "--", "git", "-C", localRepo, "commit", "-q", "-m", "run1 change"])) + .exitCode, + ).toBe(0); + + expect((await run(["mark", "--session", session, "--run", "2"])).exitCode).toBe(0); + + // Run 2: two reads. + expect((await run(["exec", "--session", session, "--", "git", "-C", localRepo, "status", "--porcelain"])).exitCode).toBe( + 0, + ); + expect((await run(["exec", "--session", session, "--", "git", "-C", localRepo, "log", "--oneline"])).exitCode).toBe(0); + + const outPath = join(temps.create("iai-live-rung-report-out-"), "report.json"); + const reportSpawned = await run(["report", "--session", session, "--out", outPath]); + expect(reportSpawned.exitCode).toBe(0); + expect(existsSync(outPath)).toBe(true); + + const artifact = JSON.parse(readFileSync(outPath, "utf8")) as { + readonly runs: Readonly< + Record }> + >; + readonly overall: { readonly totalCalls: number; readonly reads: number }; + }; + + // Denominator non-zero first: both runs must actually be present. + const runKeys = Object.keys(artifact.runs); + expect(runKeys.length).toBe(2); + expect(new Set(runKeys)).toEqual(new Set(["1", "2"])); + + const run1 = artifact.runs["1"]; + const run2 = artifact.runs["2"]; + expect(run1).toBeDefined(); + expect(run2).toBeDefined(); + + expect(run1?.totalCalls).toBe(3); + expect(run1?.reads).toBe(1); + expect(run1?.kinds["git-write"]).toBe(2); // `add` and `commit` + + expect(run2?.totalCalls).toBe(2); + expect(run2?.reads).toBe(2); + + expect(artifact.overall.totalCalls).toBe(5); + expect(artifact.overall.reads).toBe(3); + }, + 120_000, + ); +}); + +// =========================================================================== +// report: tokenCost and non-zero exits. +// =========================================================================== + +describe("live-rung report: tokenCost is structurally null unless supplied", () => { + async function sessionWithOneCall(): Promise { + const session = join(temps.create("iai-live-rung-report-session-"), "session"); + const localRepo = temps.create("iai-live-rung-report-local-git-"); + makeLocalGitRepo(localRepo); + expect((await run(["init", "--session", session, "--repo", "OWNER/REPO", "--host", "opencode"])).exitCode).toBe(0); + expect((await run(["mark", "--session", session, "--run", "1"])).exitCode).toBe(0); + expect((await run(["exec", "--session", session, "--", "git", "-C", localRepo, "status", "--porcelain"])).exitCode).toBe( + 0, + ); + return session; + } + + test("11. tokenCost is null without --token-cost, and the rendered summary says so in words", async () => { + const session = await sessionWithOneCall(); + const outPath = join(temps.create("iai-live-rung-report-null-"), "report.json"); + const spawned = await run(["report", "--session", session, "--out", outPath]); + expect(spawned.exitCode).toBe(0); + expect(spawned.stdout).toContain("token cost: not supplied"); + + const artifact = JSON.parse(readFileSync(outPath, "utf8")) as { tokenCost: unknown }; + expect(artifact.tokenCost).toBeNull(); + }, 120_000); + + test("12. tokenCost is the supplied number with --token-cost", async () => { + const session = await sessionWithOneCall(); + const outPath = join(temps.create("iai-live-rung-report-num-"), "report.json"); + const spawned = await run(["report", "--session", session, "--out", outPath, "--token-cost", "4242"]); + expect(spawned.exitCode).toBe(0); + expect(spawned.stdout).toContain("token cost: 4242"); + + const artifact = JSON.parse(readFileSync(outPath, "utf8")) as { tokenCost: unknown }; + expect(artifact.tokenCost).toBe(4242); + }, 120_000); + + test("13. a negative --token-cost is rejected, exit non-zero, no artifact overwritten with a lie", async () => { + const session = await sessionWithOneCall(); + const outPath = join(temps.create("iai-live-rung-report-neg-"), "report.json"); + const spawned = await run(["report", "--session", session, "--out", outPath, "--token-cost", "-1"]); + expect(spawned.exitCode).not.toBe(0); + expect(existsSync(outPath)).toBe(false); + }, 120_000); + + test("14. a non-finite --token-cost (Infinity, NaN-shaped) is rejected", async () => { + const session = await sessionWithOneCall(); + for (const bad of ["Infinity", "-Infinity", "not-a-number"]) { + const outPath = join(temps.create("iai-live-rung-report-nonfinite-"), "report.json"); + const spawned = await run(["report", "--session", session, "--out", outPath, "--token-cost", bad]); + expect(spawned.exitCode).not.toBe(0); + expect(existsSync(outPath)).toBe(false); + } + }, 120_000); +}); + +describe("live-rung report: distinct non-zero-exit argvs", () => { + test("15. a failing call is listed once, with its exit code, and a passing call is not listed", async () => { + const session = join(temps.create("iai-live-rung-report-nonzero-"), "session"); + const localRepo = temps.create("iai-live-rung-report-nonzero-git-"); + makeLocalGitRepo(localRepo); + + expect((await run(["init", "--session", session, "--repo", "OWNER/REPO", "--host", "opencode"])).exitCode).toBe(0); + expect((await run(["mark", "--session", session, "--run", "1"])).exitCode).toBe(0); + + const passing = await run(["exec", "--session", session, "--", "git", "-C", localRepo, "status", "--porcelain"]); + expect(passing.exitCode).toBe(0); + + const failingArgs = ["exec", "--session", session, "--", "git", "-C", localRepo, "show", "does-not-exist-xyz"]; + const failing1 = await run(failingArgs); + expect(failing1.exitCode).not.toBe(0); + // Run the SAME failing argv twice, to prove the report lists it once + // (DISTINCT argv), not once per attempt. + const failing2 = await run(failingArgs); + expect(failing2.exitCode).toBe(failing1.exitCode); + + const outPath = join(temps.create("iai-live-rung-report-nonzero-out-"), "report.json"); + const spawned = await run(["report", "--session", session, "--out", outPath]); + expect(spawned.exitCode).toBe(0); + + const artifact = JSON.parse(readFileSync(outPath, "utf8")) as { + readonly nonZeroExits: readonly { readonly argv: readonly string[]; readonly exitCode: number }[]; + }; + + expect(artifact.nonZeroExits.length).toBe(1); + const entry = artifact.nonZeroExits[0]; + expect(entry).toBeDefined(); + expect(entry?.exitCode).toBe(failing1.exitCode); + expect(entry?.argv).toEqual(["git", "-C", localRepo, "show", "does-not-exist-xyz"]); + + // The passing call's argv must not appear among the non-zero exits. + const passingArgvJson = JSON.stringify(["git", "-C", localRepo, "status", "--porcelain"]); + const anyMatchesPassing = artifact.nonZeroExits.some((e) => JSON.stringify(e.argv) === passingArgvJson); + expect(anyMatchesPassing).toBe(false); + }, 120_000); +}); + +// =========================================================================== +// The test-only --force-private-goals-payload seam: an argued safety +// property made MECHANICAL. Blanks the body of every line and block comment, +// preserving line count and offsets, so a comment that merely NAMES the flag +// in prose cannot be mistaken for a second use site -- mirrors `maskComments` +// at scripts/lint.ts:36 and packages/core/test/binding-types.test.ts's own +// copy of it, both citing the same prose-versus-code trap +// (docs/evidence/21-20260902T100227Z.md: an unanchored `grep` over real code +// returned false hits a masked scan does not). +// =========================================================================== + +function maskComments(source: string): string { + return source + .replace(/\/\*[\s\S]*?\*\//g, (m) => m.replace(/[^\n]/g, " ")) + .replace(/\/\/[^\n]*/g, (m) => " ".repeat(m.length)); +} + +describe("live-rung's test-only --force-private-goals-payload flag: the safety argument made mechanical", () => { + // ------------------------------------------------------------------------- + // A. The flag can only tighten, never admit. + // ------------------------------------------------------------------------- + + test( + "16A-behavioural. running init with the flag exits non-zero, names PRIVATE, and creates no " + + "session directory content -- the gate runs before anything touches the filesystem", + async () => { + const session = join(temps.create("iai-live-rung-force-private-"), "session"); + const spawned = await run([ + "init", + "--session", + session, + "--repo", + "OWNER/REPO", + "--host", + "opencode", + "--force-private-goals-payload", + ]); + + expect(spawned.exitCode).not.toBe(0); + const combined = spawned.stdout + spawned.stderr; + expect(combined).toContain("PRIVATE"); + expect(combined).toContain("REFUSED to start"); + + // No session directory content was created at all -- not the parent + // temp dir (which existed already), the session dir itself, nor + // header.json: the gate is the FIRST thing this command does. + expect(existsSync(session)).toBe(false); + expect(existsSync(headerPath(session))).toBe(false); + }, + 120_000, + ); + + test("16A-structural. the flag literal occurs exactly once in the file, and that one line also references userDerivedPayload", () => { + const source = readFileSync(cli, "utf8"); + // Denominator first: a failed or empty read must not let the assertions + // below pass vacuously. + expect(source.length).toBeGreaterThan(0); + + const masked = maskComments(source); + const flagLiteral = "--force-private-goals-payload"; + + const occurrenceCount = masked.split(flagLiteral).length - 1; + expect(occurrenceCount).toBe(1); + + const carryingLines = masked.split("\n").filter((line) => line.includes(flagLiteral)); + expect(carryingLines.length).toBe(1); + expect(carryingLines[0]).toContain("userDerivedPayload"); + + // If this flag ever gains a second use site, or stops selecting the + // PRIVATE payload, it stops being a flag that can only refuse -- and the + // argument that makes it safe collapses. This test is what makes that + // collapse loud rather than silent. + }); + + // ------------------------------------------------------------------------- + // B. The recognised flag surface is a closed, enumerated set. + // ------------------------------------------------------------------------- + + // One line per flag: its purpose, and whether it is operator-facing or + // test-only. This is a PARTITION, not a ban -- a new flag added to the + // live rung fails the test below and must be classified deliberately here, + // one way or the other. A carve-out (a denylist of "flags we know about") + // cannot detect an unclassified new member; a partition must. This matters + // more here than anywhere else in the repository: this is the one command + // that talks to a real forge. + const EXPECTED_FLAG_SURFACE: ReadonlySet = new Set([ + "--session", // operator-facing: the session directory every subcommand (init/mark/exec/report) operates on + "--repo", // operator-facing: the OWNER/NAME target repo, required by `init`, read by `exec`'s mutation guard + "--host", // operator-facing: names the agent host driving the model, required by `init`, never guessed + "--run", // operator-facing: the 1|2 run index passed to `mark` + "--out", // operator-facing: optional report output path, passed to `report` + "--token-cost", // operator-facing: optional measured token cost recorded into the report artifact + "--help", // operator-facing: prints usage and exits 0 (also aliased -h, not itself a `--`-shaped flag) + "--force-private-goals-payload", // TEST-ONLY: substitutes the PRIVATE USER/-derived payload for the synthetic goals fixture at the first egress gate, so the refusal path is exercisable without this script inventing a second, untested code path + ]); + + test("16B. the recognised --flag surface is exactly this enumerated, closed set", () => { + const source = readFileSync(cli, "utf8"); + expect(source.length).toBeGreaterThan(0); + + const masked = maskComments(source); + const matches = masked.match(/--[a-zA-Z][a-zA-Z-]*/g) ?? []; + // Denominator non-zero first: a masking bug that ate the whole file must + // not let an empty-set comparison pass by accident. + expect(matches.length).toBeGreaterThan(0); + + const actualFlagSurface = new Set(matches); + expect(actualFlagSurface).toEqual(EXPECTED_FLAG_SURFACE); + }); + + // ------------------------------------------------------------------------- + // C. The goals payload cannot be pointed anywhere else. + // ------------------------------------------------------------------------- + + test("16C. renderGoalsFixture is called exactly once, and no enumerated flag accepts a goals path or directory", () => { + const source = readFileSync(cli, "utf8"); + expect(source.length).toBeGreaterThan(0); + const masked = maskComments(source); + + const renderCalls = masked.match(/\brenderGoalsFixture\(/g) ?? []; + expect(renderCalls.length).toBe(1); + + // None of the flags enumerated in B is named for a goals path/directory/ + // file, and is therefore not a redirection seam. + for (const flag of EXPECTED_FLAG_SURFACE) { + expect(flag).not.toMatch(/goals?-?(path|dir|file|source)/i); + } + // The one flag that even mentions "goals" in its name + // (--force-private-goals-payload) is a boolean presence check + // (`argv.includes`, tested in 16A-structural above), never + // `extractFlag`'d for a value -- so it cannot carry a path either. + expect(masked).not.toMatch(/extractFlag\([^)]*"--force-private-goals-payload"/); + + // HONEST REASONING, stated here so no later reader assumes the egress + // gate covers this: the egress gate is NOT what stops this command + // running against the real goals source. A real goals document would + // very likely classify INTERNAL (same `work_state` key, + // packages/core/src/classify/recognisers.ts) and clear the gate exactly + // as the synthetic fixture does -- the gate has never seen the + // difference between a real and a synthetic goals document, only + // between an INTERNAL-keyed payload and a PRIVATE one. What actually + // stops it is narrower and duller: the payload is built from + // `renderGoalsFixture` with no flag, argument or code path that can + // redirect it to read a real file. These are two different protections + // guarding two different things, and conflating them would leave a hole + // nobody is watching. + }); +});