Repository navigation
S2.6.3 — The stateful fake forge: able to lie, and non-enforcing #320
Description
Parent: #293
Build target 4. Anchors: CLAIM-293.6, NEVER-293.7. Cases: 9, 8.
Blocked by: #318
Gate ruling G4: the fake must be able to lie, AND must not enforce
Two separate requirements, both ruled YES.
It must be able to lie
4 of 4 skill bodies name rate-limited-read-as-absence as their damaging failure mode. story-create says it loudest: a rate-limited read treated as "no Stories exist" is "the single most damaging failure mode of this skill", because absence is what every create step keys on.
A fixture forge that always answers correctly cannot exercise the failure the skills were written to survive — and a check that cannot meaningfully fail is what .github/workflows/ci.yml:137-146 names as its own disqualifier for smoke-install.
Five injectable modes, per references/gh-error-handling.md:
- rate limit with a zeroed budget header
- 403 with no header — the third state, which is the one most likely to be forgotten
- a truncated unpaginated list
- a silent no-op mutation (
gh pr readyhas done this for real) - a transient connection error
It must NOT enforce
This is the subtler half and it fails by passing. GitHub's milestone endpoint returns 422 on a duplicate title, and goal-create uses the milestone title as its identity key. If the fake enforces that, run 2 makes no mutation because the fake refused it — not because the skill declined to try.
A guard shadowed by another guard, one layer out: a skill shadowed by its own test double. The S1.5 remedy generalises exactly — record the attempted mutating argv, never only the successful one, and make the fake permissive exactly where GitHub is permissive.
Done when
- All 5 failure modes injectable; the truncated-list mode makes at least one skill's run 2 issue a duplicate create, and the harness reports it (case 9) — this is the case that proves the harness can fail
- Driven directly with a duplicate-creating argv for each of the 4 identity keys, the fake accepts (case 8). Key list enumerated from the four bodies, count asserted
- The 4-of-4 rate-limit-hazard count is read off disk and asserted, not hardcoded
- Assert the reason, not the verdict — messages pairwise distinct (case 20)
Refs #293
Activity
#318: record the verification evidence
dev-pmallapp commentedon Sep 8, 2026
iai-evidence
Verdict: PASS, with one case explicitly PARTIAL. Artifact:
docs/evidence/320-20260908T074833Z.md, pinned at8bebb3276397215617c9d426663ac918472a84b7. PR #329.Build target 4 of
docs/design/stories/293.md. Case 8 PASS (NEVER-293.7met), case 20 PASS, case 9 PARTIAL —CLAIM-293.6is NOT met.Chain at
b297932959 tests / 945 pass / 0 fail / 14 skip / 3389
expect()· lint 0 across 107 files · typecheck clean · skill-lint 4 + 12, 0 errors · claim-lint 116 files, 0 errors, 1 warning (the expecteddocs/design/stories/9.mdone).Mutation testing — 10 injected, 8 killed
truncate()becomes a no-op — the fake can no longer lienode:child_processpackages/execandpackages/harness## Re-entryrow no class claimsM9/M9b is the whole argument for asserting state, not verdict
expect(first.exitCode).toBe(0)Under the weakened assertion the kill comes from case 9 (ii) — a different case, anchored to a different claim. So
NEVER-293.7's own case, written the obvious way, scores the exact defect it exists to detect as a pass.Case 9 is PARTIAL, deliberately
Half (i) — all five modes injectable, five distinct signatures — is met. Half (ii) — "the truncated-list mode makes at least one skill's run 2 issue a duplicate create" — is not: no skill has been executed, and case 9's own Command column names
bun run skill-harness, which exists in nopackage.jsonin this tree. It closes at #323. The six-line read-then-create driver in the test file is not a transcription ofgoal-create, is not exported from the barrel, and must not be cited as one.Findings raised, none fixed here
shouldRetryResponseanswersretry: falsefor a transient connection error — the one failurereferences/gh-error-handling.md:120-131records three times in a session and tells you to retry. Needs an issue.packages/core/src/gh/errors.ts:270's two rate-limit reasons differ only in the value they quote — the shapereferences/verification.md:127-129forbids, inside the kernel. It is why none of case 20's distinctness assertions use that string.git clean -fddestroyed this task's uncommitted work, and the handoff recommends it as the safe alternative torm.-fdremoves untracked files, which is what a task before its first commit entirely consists of.## Re-entrycolumn yields 16 keys, not the 4 the plan names. Resolved as four key classes over a partition of all 16 rows, with unmapped rows reported by name.What this does NOT establish
The first-contact bill is still unpaid and this task cannot pay it. #318 and #319 both deferred it here. A fake forge answers
ghargv from memory, so aghfamily whose real argv is wrong passes here exactly as it passed against golden argv. Noghargv this repository builds has ever been executed. Build targets 1–10 now contain no owner for it — it needs its own issue before the seam rung is called done.The fake's 403 strings carry provenance
assumed, matchingerrors.ts:199-206; the fake andHTTP_403_REare tuned to each other and can both be wrong about real GitHub together. #324 is the first thing that will meet a real 403.dev-pmallapp commentedon Sep 8, 2026
Closed on merge. PR #329 merged into
story/293-s2-6-skill-execution-harnessate1c24219752f0480231798077f971d5cd3829d81, merged by a human — iAI never merges (CONTRIBUTING.md:416).Verdict stands at PASS for cases 8 and 20. Case 9 remains PARTIAL and
CLAIM-293.6is NOT met — half (ii) needs a transcribed skill and a runner, and closes at #323, which must not assume it. Evidence:docs/evidence/320-20260908T074833Z.md.Three findings carry forward to #321: assert state and never the verdict; exit code and row counts collapse three of the five lies; and
shouldRetryResponseanswersretry: falsefor a transient connection error, now #331. The unpaid first-contact bill is #330.