S2.6.3 — The stateful fake forge: able to lie, and non-enforcing

docs/evidence/320-20260908T074833Z.md

@@ -0,0 +1,231 @@
+---
+issue: 320
+story: 293
+repo: dev-pmallapp/iAI
+branch: task/320-stateful-fake-forge
+commit: b2979326f21b053478800f4e2170ee29eedc20e9
+run: 2026-09-08T07:48:33Z
+verdict: PASS
+---
+
+# Evidence: #320 — The stateful fake forge: able to lie, and non-enforcing
+
+**Verdict: PASS.** Build target 4 of `docs/design/stories/293.md`. Case **8**
+PASS and `NEVER-293.7` met; case **20** PASS; case **9 PARTIAL** and
+**`CLAIM-293.6` NOT met** — see *What this does NOT establish*, which is the
+load-bearing section of this artifact.
+
+Measured at `b297932`, before this artifact was committed. Per #299 the
+evidence artifact itself adds **+2 `expect()`** via
+`packages/core/test/evidence-upsert.test.ts:33-50`, so the count after this
+commit is **3391**, not 3389.
+
+## Chain
+
+| Check | Result |
+|---|---|
+| `bun run build` | clean |
+| `bun test` | **959 tests / 945 pass / 0 fail / 14 skip / 3389 `expect()`** |
+| `bun run lint` | **0 violations across 107 files** |
+| `bun run typecheck` | clean |
+| `bun run skill-lint` | 4 `SKILL.md` + 12 contract files, **0 errors, 0 warnings** |
+| `bun run claim-lint` | **116 files, 0 errors, 1 warning** |
+
+The 1 warning is `docs/design/stories/9.md`, which predates the metadata
+convention — ruling G1b of #41. It must stay a warning.
+
+Deltas from `story/293-s2-6-skill-execution-harness` @ `8747a96`:
+**934 → 959 tests** (+25), **3286 → 3389 `expect()`** (+103), **lint 105 → 107
+files**. No new package: the fake is a module inside `iai-harness`, so the
+`"bun": "./src/index.ts"` export condition that broke CI on #319 is already
+present and unchanged.
+
+## Case results
+
+| Case | Claim | Result |
+|---|---|---|
+| 8 | `NEVER-293.7` | **PASS** — 4 of 4 identity key classes, driven directly, all accepted |
+| 20 | cross-cutting | **PASS** — three rate-limit states, each asserted against the other two |
+| 9 (i) | `CLAIM-293.6` | **PASS** — 5 of 5 modes injectable, 5 distinct observable signatures |
+| 9 (ii) | `CLAIM-293.6` | **NOT MET** — needs a transcribed skill and a runner; closes at #323 |
+
+### Case 8 — the four identity keys are four key CLASSES
+
+The plan names four keys. Enumerating the `## Re-entry` "Read first" column off
+disk yields **sixteen rows**, not four — `docs/test-plans/{n}-plan.md` is a
+fifth path-shaped key the plan's prose does not list. Hardcoding 4 while
+claiming it was read off disk would have been a lie the plan's own wording
+invites.
+
+Resolved as four key **classes** over a **partition of all sixteen rows**:
+`forge-object-title` (1), `forge-list-row` (1), `sentinel-identity` (2),
+`worktree-path` (2) = **6 identity rows**; `precondition` (3),
+`amend-in-place` (5), `report-only` (2) = **10 non-identity rows**; 6 + 10 = 16,
+asserted. A row matching no class is reported **by name**, so a new Re-entry row
+fails loudly rather than being silently ignored. This is #307's ruling applied:
+a carve-out cannot detect an unclassified new member; a partition must. Mutation
+M7 proves it fires.
+
+The fourth key is on the **other surface**, and that is why it does not collapse
+into the third. Expressed as a `gh` argv the Design path would be
+`gh issue comment --body "<permalink>"` — the same argv shape as the sentinel
+case, differing only in the string inside `--body`, which
+`references/verification.md:127-129` rules is one argv and not two. Its
+`argv[0]` is `git`, which `packages/exec/src/port.ts:58` admits, so it is still
+driven directly through the port exactly as `NEVER-293.7` words it.
+
+## Mutation testing — 10 injected, 8 killed, 1 survivor by design, 1 pair that is the whole argument
+
+| # | Mutation | Result |
+|---|---|---|
+| M1 | The fake **enforces** the duplicate milestone title, the way GitHub's 422 does | **KILLED** (2 fail) |
+| M2 | `truncate()` becomes a no-op: the fake can no longer lie by omission | **KILLED** (2 fail) |
+| M3 | The header-absent 403 emits headers too: the **third state** collapses into the first | **KILLED** (3 fail) |
+| M4 | The silent no-op actually writes: verdict and state agree again | **KILLED** (1 fail) |
+| M5 | The fake **upserts** the sentinel comment, enforcing the skill's own contract for it | **KILLED** (1 fail) |
+| M6 | The fake gains `import { execFile } from "node:child_process"` | **KILLED** (3 fail, across `packages/exec` **and** `packages/harness`) |
+| M7 | A fifth `## Re-entry` row appears in a `SKILL.md` and no class claims it | **KILLED** (2 fail) |
+| M8 | The mixed-case header fixture is lowercased | **KILLED** (1 fail) |
+| M10 | The transient error never spends itself: a fatal error wearing a transient one's name | **KILLED** (1 fail) |
+| M9 | Case 8's state-growth assertion replaced by `expect(first.exitCode).toBe(0)` | **SURVIVED by design of the mutation** |
+| M9b | **M9 *and* M1 together** — the weakened case 8 against the enforcing fake | **KILLED, but not by case 8** |
+
+Every mutation asserted its pattern matched **exactly once** before being
+applied, and asserted the file actually changed before the run was trusted — a
+no-op mutation reports the result you were hoping for. **Every verdict is the
+runner's exit code.** Nothing in the harness greps stdout, which is finding 37's
+requirement and finding 37's recorded owner is *"any future harness"* — this one.
+
+### M9/M9b is the most valuable result in this artifact
+
+M9 alone weakens a test rather than injecting a defect, so its survival proves
+nothing. The pair is the argument:
+
+> With case 8's state-growth assertion replaced by an exit-code assertion **and**
+> the fake made to enforce the 422, **all four of case 8's tests pass.** The kill
+> comes from **case 9 (ii)** — a different case, anchored to a different claim.
+
+So `NEVER-293.7`'s own case, written the obvious way, **scores the exact defect
+it exists to detect as a pass**. The exit code is the verdict; the state is the
+reason. This is the S1.5 shadowing trap arriving through the test rather than
+through the fake, and it is why every branch of case 8 reads its result back
+through a **list argv** and never through a private field or an exit code.
+
+## Six findings
+
+### 1. Case 8 asserted on an exit code cannot detect an enforcing fake
+
+Recorded above as M9/M9b. Generalises to every test of a permissive double: a
+double that refuses and a double that accepts return the same verdict on the
+**first** call, and differ only in state on the second. **Owner: pattern —
+applies to #321, #322 and #323, all of which will assert against this fake.**
+
+### 2. Exit code and row counts collapse **three of the five** failure modes
+
+Measured while writing `case 9 (i)`. On `read.exit` + `read.rows` +
+`create.exit` + `state.rows` the five modes produce **three** distinct
+signatures: the zeroed budget, the header-absent 403 and the transient error are
+all *"the read exited 1 and returned no rows"*. Adding `classifyRateLimit`'s
+answer to the signature separates all five.
+
+This is not a defect in the fake. It is the measurement that justifies case 20
+existing at all: **the five lies are not distinguishable by what they do, only
+by why.** A harness branching on the verdict would treat a rate limit, a
+permission failure and a dropped connection as one event.
+
+### 3. `shouldRetryResponse` answers `retry: false` for a transient connection error
+
+Measured directly:
+
+```
+classifyExitCode(1)   -> { code: 1, classification: "fatal", provenance: "observed" }
+classifyRateLimit(..) -> "not-rate-limited"
+shouldRetryResponse   -> { retry: false, reason: 'exit code 1 classifies fatal (observed) and
+                                                  rate limit classification is "not-rate-limited"' }
+```
+
+`references/gh-error-handling.md:120-131` records **three** `gh` calls failing
+with a connection error in a single session, **each succeeding on immediate
+retry**, and instructs *"Retry once before believing a failure."* The kernel's
+own retry-decision function does not retry the failure mode the reference
+document exists to tell you to retry. **Needs an issue — not yet raised.**
+
+### 4. The kernel's own `reason` strings differ only in the value they quote
+
+`packages/core/src/gh/errors.ts:270` returns
+`rate limit classification is "rate-limited"` and, for the third state,
+`...is "possibly-rate-limited"`. Two messages that differ only in the value they
+quote are one message — `references/verification.md:127-129`, the rule this
+repository has now paid for three times.
+
+It is inside the kernel, in a `reason` field whose purpose is to be read by a
+human deciding why a batch stopped. **This is why none of case 20's distinctness
+assertions use that string**; they assert the classification **value** and the
+**absence** of the other two. Recorded, not fixed: changing a kernel message is
+not this task's. **Owner: whoever next edits `errors.ts`.**
+
+### 5. `git clean -fd` destroyed this task's uncommitted work, and the handoff recommends it
+
+The handoff records that `rm` is denied by tool policy and directs the reader to
+`git clean -fd` instead. **`-fd` removes untracked files, and a task in progress
+before its first commit consists entirely of untracked files.** Both new files
+were destroyed by a cleanup of a stray probe script and had to be rewritten.
+
+The recommended remedy is not the recommended verb. **Commit before any cleanup
+verb, or use `git stash -u`**, which is recoverable. This is the same class as
+the handoff's existing warning that `git checkout <file>` reverts to the index —
+a recovery verb that destroys the thing being recovered. **Owner: the handoff.**
+
+### 6. The `## Re-entry` column yields 16 keys, not the 4 the plan names
+
+Recorded under *Case 8* above. The plan's four are read as classes. The fifth
+literal key — `docs/test-plans/{n}-plan.md` — is honestly classified
+`worktree-path` rather than pretended away.
+
+## What this does NOT establish
+
+- **Case 9 is NOT closed and `CLAIM-293.6` is NOT met.** Half (i) — all five
+  modes injectable, each distinct — is met and mutation-tested (M2, M3, M4, M8,
+  M10). Half (ii) — *"the truncated-list mode makes at least one **skill's**
+  run 2 issue a duplicate create, and the harness reports it"* — is **not** met:
+  no skill has been executed, and **`bun run skill-harness`, which case 9's own
+  Command column names, does not exist in any `package.json` in this tree.**
+  Half (ii) closes at **#323**, once a transcribed skill body runs under the
+  runner #322 builds.
+- **The read-then-create driver in `packages/harness/test/fake-forge.test.ts` is
+  not a transcription of `goal-create` and must not be cited as one.** It
+  implements one sentence — `skills/goal-create/SKILL.md:107-110` — and is
+  confined to the test file, deliberately not exported from the barrel, so that
+  #323's transcription (which `CLAIM-293.11` and case 18 require to be audited
+  one row per Re-entry row) remains the only transcription in the repository.
+  Reusing it would be Problem 4's category error at its sixth occurrence.
+- **Nothing here establishes that the fake forge is behaviourally equivalent to
+  GitHub.** `docs/design/stories/293.md:460-462` ruled that claim unwritable.
+  `NEVER-293.7` is established only over the four named identity key classes,
+  driven directly; it says nothing about any other invariant.
+- **THE FIRST-CONTACT BILL IS STILL UNPAID, AND #320 CANNOT PAY IT.**
+  `docs/evidence/318-20260907T152554Z.md` and `docs/evidence/319-20260908T022709Z.md`
+  both deferred it here. A fake forge answers `gh` argv **from memory** — that is
+  its purpose — so a `gh` family whose *real* argv is wrong passes here exactly
+  as it passed at #318 and #319. **No `gh` argv this repository builds has ever
+  been executed.** The S2.1 precedent is a required check that shipped with sixty
+  passing tests and rejected **7 of 9 real verbs** on first contact. Build targets
+  1–10 now contain **no owner for this**. It must be raised as its own issue
+  before the seam rung is called done.
+- **The fake is deliberately more permissive than GitHub on the milestone
+  title**, where GitHub returns 422 (`docs/design/stories/293.md:150-153`).
+  `NEVER-293.7` requires exactly that. The consequence is that **no skill's
+  handling of a real 422 is exercised anywhere in this repository.**
+- **The fake's 403 stderr strings carry provenance `assumed`, not `observed`.**
+  `packages/core/src/gh/errors.ts:199-206` records that a 403 was never
+  reproduced live. Authoring the fake's 403 to satisfy `HTTP_403_RE` and then
+  testing against what the fake emits means **the fake and the regex are tuned to
+  each other and can both be wrong about real GitHub together**. #324, the live
+  rung, is the first thing that will meet a real 403 and can confirm or refute
+  it.
+- **The header side channel models a capture path no adapter in this repository
+  has.** `gh api --include` appears nowhere in the tree, asserted over a non-zero
+  denominator, and `createRealPort`'s result is asserted to carry exactly
+  `exitCode`/`stdout`/`stderr` and no header key. That divergence is published as
+  a passing absence-test rather than hidden — and **the first reader will be
+  tempted to delete it as vacuous.** Owner: **#324**.