diff --git a/CHANGELOG.md b/CHANGELOG.md index 6d3cbc2..26c84dd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,14 @@ All notable changes to PlanAnvil are documented here. ## [Unreleased] +### Qualification closure — 2026-09-07 + +- full self-hosted run #27 (`34140846679`) reproduced all C01-C16 on executed source `a9cdcdc1e0cad70e88b60869e75e4166046dd306`, Codex CLI 0.153.4, `gpt-5.6-sol`, Debian 13; +- finite C08 now proves the intended missing-checkpoint stop and completed repaired recovery without timeout; C09/C10 regressions passed; +- preserve the exact full archive and provenance, update current capability evidence, and retain historical #25 unchanged; +- C13 retains the documented project-scoped, known-error-gated non-ephemeral fallback; no new transport claim or product/runtime/security change; +- qualification is recorded; signed production publication remains a separate action. + ### Added - add a controlled `workflow_dispatch`-only Codex qualification workflow for the trusted `plananvil-codex` self-hosted runner; @@ -59,14 +67,14 @@ All notable changes to PlanAnvil are documented here. - preserve complete baseline 2.3 live evidence from full run #25, `34060321283`, tested at `d0384f76bc4150d33bb8f51ef5981f3243b3cfb3` with Codex CLI 0.153.4, `gpt-5.6-sol`, Debian 13; all C01–C16 were reproduced; - retain the exact source-bound archive, hashes and original limitations; C13 passed via the permitted project-native non-ephemeral fallback; -- replace the old C08 repaired-path workload with a finite pressure/finish scenario and strict termination checks; the old positive timeout remains in historical evidence and the replacement still requires live confirmation; +- replace the old C08 repaired-path workload with a finite pressure/finish scenario and strict termination checks; the old positive timeout remains in historical evidence; full run #27 subsequently confirmed the replacement live; - add C08-only qualification and real-CLI loopback conformance without changing C09/C10/C13 runtime behavior, product payload or live runner security; - reset newly materialized template indices to unexecuted package results instead of inheriting historical success labels; - validate qualification archive integrity and qualified product identity, and require committed finite C08 completion evidence before production publication. ### Release status -0.2.0 remains a release candidate. Full baseline qualification is recorded; production publication awaits the finite C08 live follow-up, strict release validation and a verified signed annotated tag. No tag or release has been published by the qualification-closure change. +0.2.0 is qualified for the recorded configuration. Full run #27 (`34140846679`) confirmed C01-C16 and finite C08 stop/repair on source `a9cdcdc1e0cad70e88b60869e75e4166046dd306`; its complete source-bound evidence is committed and #25 remains unchanged. Production publication remains a separate verified signed annotated-tag action after strict release validation. No tag or release has been published by this evidence import. ## [0.1.0] - 2026-07-12 diff --git a/README.md b/README.md index 85652d8..e915e78 100644 --- a/README.md +++ b/README.md @@ -10,9 +10,9 @@ It **generates and validates a plan but never executes it**. Product implementat The deterministic generator core, schemas, templates, tests, optional planning agents, defense-in-depth hooks, repository installer/upgrader/uninstaller, release tooling, and a deterministic C01-C16 live-qualification template archive/materializer are implemented. -**Full baseline 2.3 qualification passed:** run [#25](https://github.com/KeyffMS/PlanAnvil/actions/runs/34060321283) reproduced C01–C16 on source commit `d0384f76bc4150d33bb8f51ef5981f3243b3cfb3`, with Codex CLI `0.153.4`, model `gpt-5.6-sol`, and Debian 13. C04 is informational; the other fifteen capabilities gate qualification. The complete sanitized evidence is committed under `capabilities/` and preserved unchanged under `qualifications/34060321283/`. +**Full baseline 2.3 qualification and finite C08 closure passed:** run [#27](https://github.com/KeyffMS/PlanAnvil/actions/runs/34140846679) reproduced C01–C16 on source commit `a9cdcdc1e0cad70e88b60869e75e4166046dd306`, with Codex CLI `0.153.4`, model `gpt-5.6-sol`, and Debian 13. C04 is informational; the other fifteen capabilities gate qualification. The exact sanitized packages are committed under `capabilities/`; the original archive, summary and provenance are preserved under `qualifications/34140846679/`. Historical run #25 remains unchanged. -**Release status remains candidate.** The old C08 positive trial proved its narrow unblock assertion but later timed out. Its finite replacement must finish in a new live `mode=c08` run and its evidence must be committed before production publication. C13 was reproduced through the explicit known-error-gated non-ephemeral fallback, not by proving ephemeral spawning works. +**Qualification is complete for that tested configuration; signed publication is a separate step.** C08 now proves both the expected missing-checkpoint stop and a completed repaired recovery, without timeout. C09 and both C10 probes also completed without timeout. C13 was reproduced through the explicit known-error-gated, project-scoped non-ephemeral fallback; ephemeral spawning is not claimed as working. This evidence import does not create a production tag or GitHub Release. See [current qualification and release status](https://github.com/KeyffMS/PlanAnvil/blob/main/docs/QUALIFICATION_STATUS.md) for the precise evidence boundary and next step. Deterministic CI and loopback CLI conformance are regression checks, not substitutes for live model evidence. Production publication also retains protected-main, clean-tree and verified signed annotated-tag requirements. @@ -104,6 +104,8 @@ python -m unittest discover -s .agents/skills/plan-anvil/tests -v python -m unittest discover -s tests -v python -m compileall -q .agents/skills/plan-anvil .codex/hooks tools tests python tools/release_check.py --candidate +# On a clean checkout containing the reviewed live evidence: +python tools/release_check.py ``` ## Release and live qualification diff --git a/capabilities/C01/README.md b/capabilities/C01/README.md index 2b0acc2..e31338a 100644 --- a/capabilities/C01/README.md +++ b/capabilities/C01/README.md @@ -20,11 +20,11 @@ Use `fixture/`, `config/README.md`, `prompt.txt`, and `run-command.txt`. Replace ## Live qualification -- Date: `2026-09-06` +- Date: `2026-09-07` - Codex: `codex-cli 0.153.4` - Model: `gpt-5.6-sol` - OS: `Debian GNU/Linux 13 (trixie)` - Permission mode: `approval=never; sandbox=per-trial; model-tool network disabled` - Project trust: `trusted via CLI override for disposable fixture repositories` -- Source commit: `d0384f76bc4150d33bb8f51ef5981f3243b3cfb3` +- Source commit: `a9cdcdc1e0cad70e88b60869e75e4166046dd306` - Result: `REPRODUCED` diff --git a/capabilities/C01/actual.sanitized.json b/capabilities/C01/actual.sanitized.json index 5eabc40..645a33f 100644 --- a/capabilities/C01/actual.sanitized.json +++ b/capabilities/C01/actual.sanitized.json @@ -2,17 +2,17 @@ "capability_id": "C01", "environment": { "codex_version": "codex-cli 0.153.4", - "fixture_commit": "4df35777e3afd6f5b6a9aecfe22793e65bba83ea", + "fixture_commit": "00f746c24feb4ecc943c59cbe8006db2a01e9cd3", "model": "gpt-5.6-sol", "os": "Debian GNU/Linux 13 (trixie)", "permission_mode": "approval=never; sandbox=per-trial; model-tool network disabled", "project_trust": "trusted via CLI override for disposable fixture repositories", - "source_commit": "d0384f76bc4150d33bb8f51ef5981f3243b3cfb3" + "source_commit": "a9cdcdc1e0cad70e88b60869e75e4166046dd306" }, "observations": [ - "The explicit activation trial resolved the repository fixture skill and produced the activation proof `FIXTURE_SKILL_ACTIVE`.", - "The nested-working-directory trial verified repository identity, discovered the same fixture skill, and produced `FIXTURE_SKILL_ACTIVE`.", - "Both trials completed with PASS outcomes, no blocker, and no recorded error events." + "The explicit-activation trial resolved the fixture skill from the repository skill directory and recorded activation proof `FIXTURE_SKILL_ACTIVE`.", + "The nested-working-directory trial verified repository identity from its nested start location, discovered and invoked the fixture skill, and recorded the same activation proof.", + "The first trial did not exercise nested discovery, but the completed second trial directly exercised and passed that assertion." ], "result": "REPRODUCED", "schema_version": "1.0", @@ -20,9 +20,14 @@ { "assertions": [ { - "evidence": "Explicit invocation resolved `fixture-capability` from `.agents/skills/fixture-capability/SKILL.md`; activation proof: FIXTURE_SKILL_ACTIVE.", - "name": "Repository skill resolution and explicit activation", + "evidence": "`fixture-capability` was resolved from `.agents/skills/fixture-capability/SKILL.md` and explicitly activated with proof `FIXTURE_SKILL_ACTIVE`.", + "name": "The fixture skill appears in repository skill discovery.", "status": "PASS" + }, + { + "evidence": "Not exercised by this root-level explicit-activation trial.", + "name": "Discovery works when Codex starts from a nested working directory inside the same repository.", + "status": "BLOCKED" } ], "blocker": null, @@ -48,7 +53,7 @@ "branch": "master", "cached_paths": [], "commit_count": 2, - "head": "4df35777e3afd6f5b6a9aecfe22793e65bba83ea", + "head": "00f746c24feb4ecc943c59cbe8006db2a01e9cd3", "index_tree": "715ef280db9d5ab81405996e12aa2be15877d964", "refs": [ "refs/heads/master" @@ -60,7 +65,7 @@ "branch": "master", "cached_paths": [], "commit_count": 2, - "head": "4df35777e3afd6f5b6a9aecfe22793e65bba83ea", + "head": "00f746c24feb4ecc943c59cbe8006db2a01e9cd3", "index_tree": "715ef280db9d5ab81405996e12aa2be15877d964", "refs": [ "refs/heads/master" @@ -79,12 +84,12 @@ { "assertions": [ { - "evidence": "Explicit invocation produced activation proof FIXTURE_SKILL_ACTIVE.", + "evidence": "$fixture-capability was discovered and invoked; activation proof: FIXTURE_SKILL_ACTIVE.", "name": "The fixture skill appears in repository skill discovery.", "status": "PASS" }, { - "evidence": "Repository identity was verified from the nested working directory, and the repository skill was then discovered and activated.", + "evidence": "Git repository identity was verified from the nested starting directory without reporting its absolute path.", "name": "Discovery works when Codex starts from a nested working directory inside the same repository.", "status": "PASS" } @@ -92,19 +97,19 @@ "blocker": null, "capability_id": "C01", "event_summary": { - "completed_command_items": 3, + "completed_command_items": 2, "completed_file_change_items": 0, "error_events": 0, "event_types": { - "item.completed": 8, - "item.started": 3, + "item.completed": 7, + "item.started": 2, "thread.started": 1, "turn.completed": 1, "turn.started": 1 }, "item_types": { "agent_message": 3, - "command_execution": 6, + "command_execution": 4, "error": 2 } }, @@ -112,7 +117,7 @@ "branch": "master", "cached_paths": [], "commit_count": 2, - "head": "4df35777e3afd6f5b6a9aecfe22793e65bba83ea", + "head": "00f746c24feb4ecc943c59cbe8006db2a01e9cd3", "index_tree": "715ef280db9d5ab81405996e12aa2be15877d964", "refs": [ "refs/heads/master" @@ -124,7 +129,7 @@ "branch": "master", "cached_paths": [], "commit_count": 2, - "head": "4df35777e3afd6f5b6a9aecfe22793e65bba83ea", + "head": "00f746c24feb4ecc943c59cbe8006db2a01e9cd3", "index_tree": "715ef280db9d5ab81405996e12aa2be15877d964", "refs": [ "refs/heads/master" diff --git a/capabilities/C01/evaluation.json b/capabilities/C01/evaluation.json index d6d15b0..c03d710 100644 --- a/capabilities/C01/evaluation.json +++ b/capabilities/C01/evaluation.json @@ -3,5 +3,5 @@ "expected_met": true, "result": "REPRODUCED", "schema_version": "1.0", - "summary": "Sanitized trial evidence demonstrates repository skill discovery both explicitly and from a nested working directory." + "summary": "Sanitized trial evidence demonstrates both repository skill discovery and discovery from a nested working directory." } diff --git a/capabilities/C01/hashes.json b/capabilities/C01/hashes.json index d866554..161e656 100644 --- a/capabilities/C01/hashes.json +++ b/capabilities/C01/hashes.json @@ -1,10 +1,10 @@ { "algorithm": "sha256", "files": { - "README.md": "422e8b7912cf13e186c621d5eeae747575e5389e5b135b157596ee4201819565", - "actual.sanitized.json": "9042d7e35a10682bff746c4313193baf4d1947dfd814853ca03f78555825aba9", + "README.md": "f0379083ccec22707ec999a82f6cda7971024e970b17f2276d74b478340bde81", + "actual.sanitized.json": "9efbd720ac6e947272ef1b998ad3b51c36b1b3ab3cd652ffab368a456bc443ba", "config/README.md": "2b6beca33571ad108adcdbac9becc82a7fe658cf672305f01f58f4ea54565b4e", - "evaluation.json": "12aeecb91deeaef38f633d67f60451a07ba176539a5430cfc16c82c8bafa7ee5", + "evaluation.json": "46bf08a3331e054e3dccd8a2822e6526994e85b7de2bfd4f04e61a7a2c1cd62e", "expected.json": "c41329fbf6892a65b85cfe1166e86d2a3b59b684a9a81601974fca0c54a0cee7", "fixture/.agents/skills/fixture-capability/SKILL.md": "0c4c7985ed4a06a2fa45d0d6a1887fec0f5a59892ae1d0d88317ac1c82a7a4ff", "fixture/README.md": "a3ac65c07ebf67442b31cf3bc1d28ac5c849bd08fee9cba93b0a31339e7602d8", diff --git a/capabilities/C02/README.md b/capabilities/C02/README.md index d6440c2..7026eb8 100644 --- a/capabilities/C02/README.md +++ b/capabilities/C02/README.md @@ -20,11 +20,11 @@ Use `fixture/`, `config/README.md`, `prompt.txt`, and `run-command.txt`. Replace ## Live qualification -- Date: `2026-09-06` +- Date: `2026-09-07` - Codex: `codex-cli 0.153.4` - Model: `gpt-5.6-sol` - OS: `Debian GNU/Linux 13 (trixie)` - Permission mode: `approval=never; sandbox=per-trial; model-tool network disabled` - Project trust: `trusted via CLI override for disposable fixture repositories` -- Source commit: `d0384f76bc4150d33bb8f51ef5981f3243b3cfb3` +- Source commit: `a9cdcdc1e0cad70e88b60869e75e4166046dd306` - Result: `REPRODUCED` diff --git a/capabilities/C02/actual.sanitized.json b/capabilities/C02/actual.sanitized.json index 50b31b4..417d13c 100644 --- a/capabilities/C02/actual.sanitized.json +++ b/capabilities/C02/actual.sanitized.json @@ -2,17 +2,17 @@ "capability_id": "C02", "environment": { "codex_version": "codex-cli 0.153.4", - "fixture_commit": "b006122b504e9bdf38fa9d0a21c35ab866fba584", + "fixture_commit": "ac01932c34c0e42bb7a84fe986452f011ffae50f", "model": "gpt-5.6-sol", "os": "Debian GNU/Linux 13 (trixie)", "permission_mode": "approval=never; sandbox=per-trial; model-tool network disabled", "project_trust": "trusted via CLI override for disposable fixture repositories", - "source_commit": "d0384f76bc4150d33bb8f51ef5981f3243b3cfb3" + "source_commit": "a9cdcdc1e0cad70e88b60869e75e4166046dd306" }, "observations": [ - "Implicit-activation trial completed and passed: a semantically matching prompt without the skill name produced no skill-specific activation token.", - "Explicit-activation trial completed and passed: the explicit $fixture-capability prompt supplied the expected FIXTURE_SKILL_ACTIVE token.", - "Both material assertions in the expected specification were directly exercised; neither trial reported a blocker or runtime error." + "The completed implicit-invocation trial used a semantically matching request without the skill name and observed no skill-specific activation token.", + "The completed explicit-invocation trial used $fixture-capability and observed the required literal proof token FIXTURE_SKILL_ACTIVE.", + "Both expected assertions are individually marked PASS; both trials report PASS with no blocker or error events." ], "result": "REPRODUCED", "schema_version": "1.0", @@ -20,7 +20,7 @@ { "assertions": [ { - "evidence": "The semantic match did not place any skill-specific activation token in the active instructions.", + "evidence": "The semantically matching request produced no skill-specific activation token in the active instructions.", "name": "A semantically matching prompt without the skill name does not activate the fixture skill.", "status": "PASS" } @@ -48,7 +48,7 @@ "branch": "master", "cached_paths": [], "commit_count": 2, - "head": "b006122b504e9bdf38fa9d0a21c35ab866fba584", + "head": "ac01932c34c0e42bb7a84fe986452f011ffae50f", "index_tree": "6455a6dfb42207be3ad780cca165fa8f60e78d57", "refs": [ "refs/heads/master" @@ -60,7 +60,7 @@ "branch": "master", "cached_paths": [], "commit_count": 2, - "head": "b006122b504e9bdf38fa9d0a21c35ab866fba584", + "head": "ac01932c34c0e42bb7a84fe986452f011ffae50f", "index_tree": "6455a6dfb42207be3ad780cca165fa8f60e78d57", "refs": [ "refs/heads/master" @@ -84,7 +84,7 @@ { "assertions": [ { - "evidence": "Activated skill context required and supplied `FIXTURE_SKILL_ACTIVE`.", + "evidence": "Activated skill context required and supplied the literal proof token: FIXTURE_SKILL_ACTIVE", "name": "An explicit $fixture-capability prompt activates the fixture skill.", "status": "PASS" } @@ -112,7 +112,7 @@ "branch": "master", "cached_paths": [], "commit_count": 2, - "head": "b006122b504e9bdf38fa9d0a21c35ab866fba584", + "head": "ac01932c34c0e42bb7a84fe986452f011ffae50f", "index_tree": "6455a6dfb42207be3ad780cca165fa8f60e78d57", "refs": [ "refs/heads/master" @@ -124,7 +124,7 @@ "branch": "master", "cached_paths": [], "commit_count": 2, - "head": "b006122b504e9bdf38fa9d0a21c35ab866fba584", + "head": "ac01932c34c0e42bb7a84fe986452f011ffae50f", "index_tree": "6455a6dfb42207be3ad780cca165fa8f60e78d57", "refs": [ "refs/heads/master" @@ -133,7 +133,7 @@ "worktree_count": 1 }, "observations": [ - "FIXTURE_SKILL_ACTIVE" + "Explicit invocation activated fixture-capability; the skill file was not directly inspected and no files were modified." ], "outcome": "PASS", "outer_skill_policy": { diff --git a/capabilities/C02/evaluation.json b/capabilities/C02/evaluation.json index b4afe4c..80bd8a4 100644 --- a/capabilities/C02/evaluation.json +++ b/capabilities/C02/evaluation.json @@ -3,5 +3,5 @@ "expected_met": true, "result": "REPRODUCED", "schema_version": "1.0", - "summary": "Sanitized trial evidence demonstrates that the fixture skill remains inactive for an unnamed semantic match and activates when explicitly named." + "summary": "Sanitized trial evidence demonstrates both explicit-only activation assertions." } diff --git a/capabilities/C02/hashes.json b/capabilities/C02/hashes.json index f84eacf..a92369a 100644 --- a/capabilities/C02/hashes.json +++ b/capabilities/C02/hashes.json @@ -1,10 +1,10 @@ { "algorithm": "sha256", "files": { - "README.md": "d48f15a8b53baa0aa7ee54ee3c1b56ae44747159f9579963461180277f052027", - "actual.sanitized.json": "84a176fed4880e6a59bb18f46d9c24962b6f371ad6de8d9458da75deaa488923", + "README.md": "0eb2ce77906499f392ee20894bba2939ac41e4b968f25d16caf06dd22af4eeea", + "actual.sanitized.json": "66acfd0d0ce95591dd9beddd138d4d9066e597b7a47c5f9f3e9888c2b5a3bcc7", "config/README.md": "22255f099cfe99dea718801c9f78b2230d229a560ba02f1f048e35f9ca765780", - "evaluation.json": "a1bb7d9496bd9daba9a07f296b1ce42acc133a91d36c9f8c7aa81fb4c329f548", + "evaluation.json": "e004a439026664fa6fa715c15603e012ca343f3685091ceb3b7381da9b87c4d7", "expected.json": "fe22f0a70480c8c1b932e67af607ca5e3f539512162dd22b903001a3b215d0b8", "fixture/.agents/skills/fixture-capability/SKILL.md": "35e41a201965e0a9c83ce9dbcf151275a7ad5718edd63c21976e0def24204cd5", "fixture/.agents/skills/fixture-capability/agents/openai.yaml": "5f4b205f412e2e168092f152801b7dc72bc00e7716147305ebc5fa6bfabe0087", diff --git a/capabilities/C03/README.md b/capabilities/C03/README.md index cec8ed0..cabe506 100644 --- a/capabilities/C03/README.md +++ b/capabilities/C03/README.md @@ -20,11 +20,11 @@ Use `fixture/`, `config/README.md`, `prompt.txt`, and `run-command.txt`. Replace ## Live qualification -- Date: `2026-09-06` +- Date: `2026-09-07` - Codex: `codex-cli 0.153.4` - Model: `gpt-5.6-sol` - OS: `Debian GNU/Linux 13 (trixie)` - Permission mode: `approval=never; sandbox=per-trial; model-tool network disabled` - Project trust: `trusted via CLI override for disposable fixture repositories` -- Source commit: `d0384f76bc4150d33bb8f51ef5981f3243b3cfb3` +- Source commit: `a9cdcdc1e0cad70e88b60869e75e4166046dd306` - Result: `REPRODUCED` diff --git a/capabilities/C03/actual.sanitized.json b/capabilities/C03/actual.sanitized.json index b418364..7324d44 100644 --- a/capabilities/C03/actual.sanitized.json +++ b/capabilities/C03/actual.sanitized.json @@ -2,12 +2,12 @@ "capability_id": "C03", "environment": { "codex_version": "codex-cli 0.153.4", - "fixture_commit": "368b6459d9c801e9ab70d931c2af313814051424", + "fixture_commit": "e2f97dd2aacb420decf67085a8a27675fcde8bc6", "model": "gpt-5.6-sol", "os": "Debian GNU/Linux 13 (trixie)", "permission_mode": "approval=never; sandbox=per-trial; model-tool network disabled", "project_trust": "trusted via CLI override for disposable fixture repositories", - "source_commit": "d0384f76bc4150d33bb8f51ef5981f3243b3cfb3" + "source_commit": "a9cdcdc1e0cad70e88b60869e75e4166046dd306" }, "observations": [ "bootstrap_git_probe_completed=true", @@ -77,21 +77,20 @@ "model_payload": { "assertions": [ { - "evidence": "Exact command exited 0; `git status --porcelain=v1` produced no output.", - "name": "Auxiliary C03 source is reachable and clean", + "evidence": "Exit code 0; `git status --porcelain=v1` produced no output.", + "name": "Auxiliary source is clean", "status": "PASS" }, { - "evidence": "`git rev-parse HEAD` returned `368b6459d9c801e9ab70d931c2af313814051424`.", - "name": "HEAD is resolvable", + "evidence": "`git rev-parse HEAD` returned `e2f97dd2aacb420decf67085a8a27675fcde8bc6`.", + "name": "Auxiliary source is reachable", "status": "PASS" } ], "blocker": null, "capability_id": "C03", "observations": [ - "Working tree status is clean.", - "HEAD: 368b6459d9c801e9ab70d931c2af313814051424" + "The exact requested command completed successfully without modifying files." ], "outcome": "PASS", "trial": "explicit_plananvil_topology" diff --git a/capabilities/C03/hashes.json b/capabilities/C03/hashes.json index a82cee2..03504aa 100644 --- a/capabilities/C03/hashes.json +++ b/capabilities/C03/hashes.json @@ -1,8 +1,8 @@ { "algorithm": "sha256", "files": { - "README.md": "084f89a9262a3a3b9c0396f015dae1660de6350d5d0909785b92400e0d042a6f", - "actual.sanitized.json": "fed985f84bfc26a716d5a6741f9066613f3cc4d177a4e3a2b2276595114af98f", + "README.md": "4c476f680f984930da4d3ea499d4ebf5a45c3ab2facfba3a4e93ab273f549fe7", + "actual.sanitized.json": "60e42e4668a8ef002b445449940e3d6c3085f3b3aaf259c5bba88ab237bbe8e3", "config/README.md": "4ddf277c13326c7e01250b638a72c2adbaf1742f2df2526a048d75c5bc5cd815", "evaluation.json": "e24e3f4ca1d6846aa737ab82fe44e56975ac2536416873ff0ae7dd8c42ab9baa", "expected.json": "ae01636f9854bfe67f02bc2dcb2aa817dc6e92b29c93e98a2486cf4f6dc379d0", diff --git a/capabilities/C04/README.md b/capabilities/C04/README.md index 7be117d..2c01ebc 100644 --- a/capabilities/C04/README.md +++ b/capabilities/C04/README.md @@ -20,11 +20,11 @@ Use `fixture/`, `config/README.md`, `prompt.txt`, and `run-command.txt`. Replace ## Live qualification -- Date: `2026-09-06` +- Date: `2026-09-07` - Codex: `codex-cli 0.153.4` - Model: `gpt-5.6-sol` - OS: `Debian GNU/Linux 13 (trixie)` - Permission mode: `approval=never; sandbox=per-trial; model-tool network disabled` - Project trust: `trusted via CLI override for disposable fixture repositories` -- Source commit: `d0384f76bc4150d33bb8f51ef5981f3243b3cfb3` +- Source commit: `a9cdcdc1e0cad70e88b60869e75e4166046dd306` - Result: `REPRODUCED` diff --git a/capabilities/C04/actual.sanitized.json b/capabilities/C04/actual.sanitized.json index e53d86c..86e6ba6 100644 --- a/capabilities/C04/actual.sanitized.json +++ b/capabilities/C04/actual.sanitized.json @@ -2,18 +2,17 @@ "capability_id": "C04", "environment": { "codex_version": "codex-cli 0.153.4", - "fixture_commit": "4630aec248648a255faaf8cf10caa1ee8ea162d2", + "fixture_commit": "265b882eea3430d63025a1d0c70635400606c686", "model": "gpt-5.6-sol", "os": "Debian GNU/Linux 13 (trixie)", "permission_mode": "approval=never; sandbox=per-trial; model-tool network disabled", "project_trust": "trusted via CLI override for disposable fixture repositories", - "source_commit": "d0384f76bc4150d33bb8f51ef5981f3243b3cfb3" + "source_commit": "a9cdcdc1e0cad70e88b60869e75e4166046dd306" }, "observations": [ - "The child agent successfully started a nested child agent, and the nested child completed.", - "The trial reports no repository, Git, specification, or PlanAnvil source changes.", - "Git state was unchanged before and after the trial, with no cached or working-tree changes.", - "Both expected assertions passed and the completed trial reported no blocker." + "A child agent successfully started a nested child agent, which completed.", + "The trial recorded the nested-agent result as informational only.", + "Git state was unchanged, with no file, configuration, or release-gate changes observed." ], "result": "REPRODUCED", "schema_version": "1.0", @@ -21,12 +20,12 @@ { "assertions": [ { - "evidence": "A child agent successfully started one nested child agent, which completed.", + "evidence": "A child agent successfully started one nested child agent, and the nested child completed.", "name": "Record whether a child agent can start another agent in the tested runtime/configuration.", "status": "PASS" }, { - "evidence": "Observation only; no repository, Git, specification, or PlanAnvil source changes were made.", + "evidence": "The observation was informational only; no PlanAnvil files, configuration, or release-gate behavior were changed.", "name": "Do not change PlanAnvil flat-topology safety based on this observation alone.", "status": "PASS" } @@ -55,8 +54,8 @@ "branch": "master", "cached_paths": [], "commit_count": 2, - "head": "4630aec248648a255faaf8cf10caa1ee8ea162d2", - "index_tree": "14cb369c4d1705946abdf80c6a0b35e7f621ca70", + "head": "265b882eea3430d63025a1d0c70635400606c686", + "index_tree": "ed6d0dcc1639cb6097a6d38a6ba7ed1bd4b30320", "refs": [ "refs/heads/master" ], @@ -67,8 +66,8 @@ "branch": "master", "cached_paths": [], "commit_count": 2, - "head": "4630aec248648a255faaf8cf10caa1ee8ea162d2", - "index_tree": "14cb369c4d1705946abdf80c6a0b35e7f621ca70", + "head": "265b882eea3430d63025a1d0c70635400606c686", + "index_tree": "ed6d0dcc1639cb6097a6d38a6ba7ed1bd4b30320", "refs": [ "refs/heads/master" ], @@ -76,14 +75,14 @@ "worktree_count": 1 }, "observations": [ - "Nested subagent spawn: allowed.", - "Nested child completion: completed.", + "Nested subagent start: accepted.", + "Nested subagent completion: observed.", "PlanAnvil flat-topology safety: unchanged." ], "outcome": "PASS", "sandbox": "read-only", - "trial": "informational-nested-subagent-observation", - "trial_name": "informational-nested-subagent-observation" + "trial": "observe_nested_subagent_behavior", + "trial_name": "observe_nested_subagent_behavior" }, { "evaluator_event_summary": { diff --git a/capabilities/C04/evaluation.json b/capabilities/C04/evaluation.json index 0eb3ab6..c50ee38 100644 --- a/capabilities/C04/evaluation.json +++ b/capabilities/C04/evaluation.json @@ -3,5 +3,5 @@ "expected_met": true, "result": "REPRODUCED", "schema_version": "1.0", - "summary": "The sanitized trial evidence demonstrates nested child-agent startup and completion while preserving PlanAnvil flat-topology safety unchanged." + "summary": "The completed trial demonstrates nested-agent startup and completion while preserving PlanAnvil flat-topology safety unchanged." } diff --git a/capabilities/C04/hashes.json b/capabilities/C04/hashes.json index 94b0401..b85affb 100644 --- a/capabilities/C04/hashes.json +++ b/capabilities/C04/hashes.json @@ -1,10 +1,10 @@ { "algorithm": "sha256", "files": { - "README.md": "057477bfe912e3da03670d6e743127ee85c9b37e6bb463512b8f3a8720630118", - "actual.sanitized.json": "a38184bac31ae43392862502b94c48399d292e1bd21aa784c76132749932b528", + "README.md": "ec7eb1cb9d8008fee5d6f174b84df64e489979f2c4107c0eb19e8edbe4590089", + "actual.sanitized.json": "e7a5c53e7a7255e6d67cf6ac0b5235ebc58ed98501e9c1a9dd7f2ffd476deb11", "config/README.md": "bdc289615d6db57a98f7bc2781c858f946334e5a5dbb4cb75bb84e9f470550be", - "evaluation.json": "6b180f007786f37a7c66a27f5f99812be0c75e83fd272a87ed5e43ee4f184566", + "evaluation.json": "ff53ed6b64288180e9c6f1bf2aee696d602616ead7c86e54654f649926272972", "expected.json": "6117c318bfa411e142263f9de73c2b3319edca4d15a8df667e4755137c24aab5", "fixture/README.md": "2eeb7e16493350395a0d5cd37677f36de8720634c98974048a8979ba38a383c2", "prompt.txt": "df898b6d1d46fef013ddb8e3e409388a228ff24ea668b760d51491f1652e8e7d", diff --git a/capabilities/C05/README.md b/capabilities/C05/README.md index 685937c..d1fc087 100644 --- a/capabilities/C05/README.md +++ b/capabilities/C05/README.md @@ -20,11 +20,11 @@ Use `fixture/`, `config/README.md`, `prompt.txt`, and `run-command.txt`. Replace ## Live qualification -- Date: `2026-09-06` +- Date: `2026-09-07` - Codex: `codex-cli 0.153.4` - Model: `gpt-5.6-sol` - OS: `Debian GNU/Linux 13 (trixie)` - Permission mode: `approval=never; sandbox=per-trial; model-tool network disabled` - Project trust: `trusted via CLI override for disposable fixture repositories` -- Source commit: `d0384f76bc4150d33bb8f51ef5981f3243b3cfb3` +- Source commit: `a9cdcdc1e0cad70e88b60869e75e4166046dd306` - Result: `REPRODUCED` diff --git a/capabilities/C05/actual.sanitized.json b/capabilities/C05/actual.sanitized.json index 7e1501e..d32243f 100644 --- a/capabilities/C05/actual.sanitized.json +++ b/capabilities/C05/actual.sanitized.json @@ -2,19 +2,19 @@ "capability_id": "C05", "environment": { "codex_version": "codex-cli 0.153.4", - "fixture_commit": "5e166a7bcd80c2ba804b9471d9f95360e26ed82a", + "fixture_commit": "0a27a0eb7875e00fbff44bda8439ba15d5fad15c", "model": "gpt-5.6-sol", "os": "Debian GNU/Linux 13 (trixie)", "permission_mode": "approval=never; sandbox=per-trial; model-tool network disabled", "project_trust": "trusted via CLI override for disposable fixture repositories", - "source_commit": "d0384f76bc4150d33bb8f51ef5981f3243b3cfb3" + "source_commit": "a9cdcdc1e0cad70e88b60869e75e4166046dd306" }, "observations": [ - "A fresh explicit bundle passed digest verification, named only one safe relative payload path, and the payload hash matched.", - "Changing the payload while leaving the bundle valid produced a file-hash mismatch and a STALE rejection.", - "Changing the bundle without updating its sidecar caused digest verification to fail before its file list was processed.", - "Removing the bundle caused digest verification to fail before any payload list was trusted or read.", - "An escaped candidate failed repository containment validation and was rejected without being opened." + "Fresh handoff accepted only the explicitly named file after both bundle-digest and file-hash verification.", + "A modified target file was detected as stale and rejected with a nonzero exit.", + "A changed bundle failed digest verification and its file list was not trusted.", + "A missing bundle failed verification and no target file was processed.", + "An escaped path was rejected at the repository boundary before outside content was opened." ], "result": "REPRODUCED", "schema_version": "1.0", @@ -22,32 +22,32 @@ { "assertions": [ { - "evidence": "review-bundle.sha256 matched review-bundle.json: 70ea11387dfb83e737f773b5162995fffc23568c65a6bc22aac6a6fa7df25169.", - "name": "Reviewer receives only the explicit immutable bundle.", + "evidence": "review-bundle.sha256 validated review-bundle.json: OK", + "name": "bundle_digest_verified", "status": "PASS" }, { - "evidence": "The verified bundle named only PLAN.md; its safe relative path was accepted and its SHA-256 matched 8890f7b98b2d70b87ab0b855104125faee2646938d2f7979a420f0c1343aed6b. Verification gates reject digest, path, existence, or file-hash mismatches.", - "name": "A stale, missing, changed, or escaped bundle is rejected.", + "evidence": "Only PLAN.md was named; its SHA-256 matched 8890f7b98b2d70b87ab0b855104125faee2646938d2f7979a420f0c1343aed6b", + "name": "explicit_file_hashes_verified", "status": "PASS" } ], "blocker": null, "capability_id": "C05", "event_summary": { - "completed_command_items": 4, + "completed_command_items": 3, "completed_file_change_items": 0, "error_events": 0, "event_types": { - "item.completed": 8, - "item.started": 4, + "item.completed": 7, + "item.started": 3, "thread.started": 1, "turn.completed": 1, "turn.started": 1 }, "item_types": { "agent_message": 2, - "command_execution": 8, + "command_execution": 6, "error": 2 } }, @@ -55,7 +55,7 @@ "branch": "master", "cached_paths": [], "commit_count": 2, - "head": "5e166a7bcd80c2ba804b9471d9f95360e26ed82a", + "head": "0a27a0eb7875e00fbff44bda8439ba15d5fad15c", "index_tree": "922fdcc70a8d66ce979be583581289abd6073113", "refs": [ "refs/heads/master" @@ -67,7 +67,7 @@ "branch": "master", "cached_paths": [], "commit_count": 2, - "head": "5e166a7bcd80c2ba804b9471d9f95360e26ed82a", + "head": "0a27a0eb7875e00fbff44bda8439ba15d5fad15c", "index_tree": "922fdcc70a8d66ce979be583581289abd6073113", "refs": [ "refs/heads/master" @@ -76,8 +76,6 @@ "worktree_count": 1 }, "observations": [ - "Verified the bundle digest before reading its manifest.", - "Read only the explicitly named fixture file after validating its recorded hash.", "Accepted the immutable explicit handoff." ], "outcome": "PASS", @@ -88,12 +86,12 @@ { "assertions": [ { - "evidence": "Processed only explicit review-bundle.json; its recorded digest verified OK.", + "evidence": "Processed explicit review-bundle.json; its recorded bundle digest still matched.", "name": "Reviewer receives only the explicit immutable bundle.", "status": "PASS" }, { - "evidence": "After only PLAN.md changed, recorded hash 8890f7b9\u2026 differed from actual dad1b8fe\u2026; handoff was rejected as STALE.", + "evidence": "Only PLAN.md changed. Its actual hash differed from the bundle-recorded hash; processing returned HANDOFF=STALE and rejected it.", "name": "A stale, missing, changed, or escaped bundle is rejected.", "status": "PASS" } @@ -101,19 +99,19 @@ "blocker": null, "capability_id": "C05", "event_summary": { - "completed_command_items": 3, + "completed_command_items": 8, "completed_file_change_items": 1, "error_events": 0, "event_types": { - "item.completed": 9, - "item.started": 4, + "item.completed": 14, + "item.started": 9, "thread.started": 1, "turn.completed": 1, "turn.started": 1 }, "item_types": { "agent_message": 3, - "command_execution": 6, + "command_execution": 16, "error": 2, "file_change": 2 } @@ -122,7 +120,7 @@ "branch": "master", "cached_paths": [], "commit_count": 2, - "head": "5e166a7bcd80c2ba804b9471d9f95360e26ed82a", + "head": "0a27a0eb7875e00fbff44bda8439ba15d5fad15c", "index_tree": "922fdcc70a8d66ce979be583581289abd6073113", "refs": [ "refs/heads/master" @@ -136,7 +134,7 @@ "branch": "master", "cached_paths": [], "commit_count": 2, - "head": "5e166a7bcd80c2ba804b9471d9f95360e26ed82a", + "head": "0a27a0eb7875e00fbff44bda8439ba15d5fad15c", "index_tree": "922fdcc70a8d66ce979be583581289abd6073113", "refs": [ "refs/heads/master" @@ -145,9 +143,9 @@ "worktree_count": 1 }, "observations": [ - "Only PLAN.md was modified.", - "Bundle digest remained valid.", - "Recorded PLAN.md hash mismatch was detected and classified STALE." + "Bundle integrity: MATCH.", + "Stale target detected: PLAN.md.", + "Process exit: 3 (rejected)." ], "outcome": "PASS", "sandbox": "workspace-write", @@ -157,12 +155,12 @@ { "assertions": [ { - "evidence": "Explicit handoff verification used review-bundle.sha256 for review-bundle.json and stopped at digest validation.", + "evidence": "Explicit handoff used review-bundle.json with review-bundle.sha256; digest validation ran before file-list processing.", "name": "Reviewer receives only the explicit immutable bundle.", "status": "PASS" }, { - "evidence": "After review-bundle.json was modified without changing its sidecar, sha256sum reported FAILED and exited 1; the bundle file list was not processed.", + "evidence": "After only review-bundle.json was changed, handoff returned exit status 65 with BUNDLE_DIGEST_MISMATCH and file_list_trusted=false.", "name": "A stale, missing, changed, or escaped bundle is rejected.", "status": "PASS" } @@ -170,19 +168,19 @@ "blocker": null, "capability_id": "C05", "event_summary": { - "completed_command_items": 4, + "completed_command_items": 3, "completed_file_change_items": 1, "error_events": 0, "event_types": { - "item.completed": 10, - "item.started": 5, + "item.completed": 9, + "item.started": 4, "thread.started": 1, "turn.completed": 1, "turn.started": 1 }, "item_types": { "agent_message": 3, - "command_execution": 8, + "command_execution": 6, "error": 2, "file_change": 2 } @@ -191,7 +189,7 @@ "branch": "master", "cached_paths": [], "commit_count": 2, - "head": "5e166a7bcd80c2ba804b9471d9f95360e26ed82a", + "head": "0a27a0eb7875e00fbff44bda8439ba15d5fad15c", "index_tree": "922fdcc70a8d66ce979be583581289abd6073113", "refs": [ "refs/heads/master" @@ -205,7 +203,7 @@ "branch": "master", "cached_paths": [], "commit_count": 2, - "head": "5e166a7bcd80c2ba804b9471d9f95360e26ed82a", + "head": "0a27a0eb7875e00fbff44bda8439ba15d5fad15c", "index_tree": "922fdcc70a8d66ce979be583581289abd6073113", "refs": [ "refs/heads/master" @@ -214,8 +212,10 @@ "worktree_count": 1 }, "observations": [ - "Only review-bundle.json was modified; review-bundle.sha256 remained unchanged.", - "Digest verification rejected the changed bundle before its file list was trusted." + "changed_paths=review-bundle.json", + "handoff=REJECTED", + "reason=BUNDLE_DIGEST_MISMATCH", + "file_list_trusted=false" ], "outcome": "PASS", "sandbox": "workspace-write", @@ -225,12 +225,12 @@ { "assertions": [ { - "evidence": "Handoff attempted only via the recorded explicit path review-bundle.json; payload files were not processed.", + "evidence": "Explicit handoff verification used review-bundle.sha256 and failed opening review-bundle.json; no target file was processed.", "name": "Reviewer receives only the explicit immutable bundle.", "status": "PASS" }, { - "evidence": "After deletion, digest verification reported review-bundle.json missing and exited with status 1, so the handoff was rejected.", + "evidence": "After review-bundle.json was removed, sha256sum --check exited 1 with \u201cNo such file or directory\u201d and \u201cFAILED open or read\u201d; the handoff was rejected.", "name": "A stale, missing, changed, or escaped bundle is rejected.", "status": "PASS" } @@ -238,19 +238,19 @@ "blocker": null, "capability_id": "C05", "event_summary": { - "completed_command_items": 7, + "completed_command_items": 6, "completed_file_change_items": 1, "error_events": 0, "event_types": { - "item.completed": 13, - "item.started": 8, + "item.completed": 12, + "item.started": 7, "thread.started": 1, "turn.completed": 1, "turn.started": 1 }, "item_types": { "agent_message": 3, - "command_execution": 14, + "command_execution": 12, "error": 2, "file_change": 2 } @@ -259,7 +259,7 @@ "branch": "master", "cached_paths": [], "commit_count": 2, - "head": "5e166a7bcd80c2ba804b9471d9f95360e26ed82a", + "head": "0a27a0eb7875e00fbff44bda8439ba15d5fad15c", "index_tree": "922fdcc70a8d66ce979be583581289abd6073113", "refs": [ "refs/heads/master" @@ -273,7 +273,7 @@ "branch": "master", "cached_paths": [], "commit_count": 2, - "head": "5e166a7bcd80c2ba804b9471d9f95360e26ed82a", + "head": "0a27a0eb7875e00fbff44bda8439ba15d5fad15c", "index_tree": "922fdcc70a8d66ce979be583581289abd6073113", "refs": [ "refs/heads/master" @@ -282,9 +282,8 @@ "worktree_count": 1 }, "observations": [ - "review-bundle.json was removed.", - "Explicit digest verification failed because the bundle could not be opened.", - "The handoff was rejected before trusting or reading its file list." + "review-bundle.json was removed inside the disposable fixture.", + "The explicit immutable handoff check failed because the required bundle was missing." ], "outcome": "PASS", "sandbox": "workspace-write", @@ -294,32 +293,32 @@ { "assertions": [ { - "evidence": "The explicit candidate was canonicalized against the repository root before use; containment evaluated false and the decision was `reject_escaped`.", - "name": "Reviewer receives only the explicit immutable bundle.", + "evidence": "Canonical boundary comparison returned REJECT_ESCAPED.", + "name": "escaped_bundle_path_rejected", "status": "PASS" }, { - "evidence": "The escaped candidate was rejected without opening or trusting its contents (`candidate_opened: false`).", - "name": "A stale, missing, changed, or escaped bundle is rejected.", + "evidence": "Candidate was rejected before any content read; candidate_opened=no.", + "name": "outside_file_not_opened_or_trusted", "status": "PASS" } ], "blocker": null, "capability_id": "C05", "event_summary": { - "completed_command_items": 4, + "completed_command_items": 3, "completed_file_change_items": 0, "error_events": 0, "event_types": { - "item.completed": 9, - "item.started": 4, + "item.completed": 8, + "item.started": 3, "thread.started": 1, "turn.completed": 1, "turn.started": 1 }, "item_types": { "agent_message": 3, - "command_execution": 8, + "command_execution": 6, "error": 2 } }, @@ -327,7 +326,7 @@ "branch": "master", "cached_paths": [], "commit_count": 2, - "head": "5e166a7bcd80c2ba804b9471d9f95360e26ed82a", + "head": "0a27a0eb7875e00fbff44bda8439ba15d5fad15c", "index_tree": "922fdcc70a8d66ce979be583581289abd6073113", "refs": [ "refs/heads/master" @@ -339,7 +338,7 @@ "branch": "master", "cached_paths": [], "commit_count": 2, - "head": "5e166a7bcd80c2ba804b9471d9f95360e26ed82a", + "head": "0a27a0eb7875e00fbff44bda8439ba15d5fad15c", "index_tree": "922fdcc70a8d66ce979be583581289abd6073113", "refs": [ "refs/heads/master" @@ -348,7 +347,8 @@ "worktree_count": 1 }, "observations": [ - "Boundary check completed successfully: `inside_repository: false`, `decision: reject_escaped`." + "Candidate canonicalized outside the current repository boundary.", + "No outside-file contents were accessed." ], "outcome": "PASS", "sandbox": "read-only", diff --git a/capabilities/C05/evaluation.json b/capabilities/C05/evaluation.json index d7b49bd..4ac8643 100644 --- a/capabilities/C05/evaluation.json +++ b/capabilities/C05/evaluation.json @@ -3,5 +3,5 @@ "expected_met": true, "result": "REPRODUCED", "schema_version": "1.0", - "summary": "The completed trials demonstrate acceptance of only the verified explicit immutable bundle and rejection of stale, missing, changed, and escaped handoffs." + "summary": "Sanitized trials demonstrate acceptance of the explicit immutable bundle and rejection of stale, missing, changed, and escaped cases." } diff --git a/capabilities/C05/hashes.json b/capabilities/C05/hashes.json index 5c86ce8..e65ec3e 100644 --- a/capabilities/C05/hashes.json +++ b/capabilities/C05/hashes.json @@ -1,10 +1,10 @@ { "algorithm": "sha256", "files": { - "README.md": "39ed93c7ad03d9e80f3fe172de358e8e659ac1114c1708c44a3e4d791b3d9210", - "actual.sanitized.json": "e0caaf122ffdc02c3273cebb75f5a260d89ddeda1764998205b77e8d403dc071", + "README.md": "125c189f80963148349ee1f00542497f60146f290f91d6a67a52702d6ee15a21", + "actual.sanitized.json": "e00a02467e9d5a67861dd0b58384c74c05beb09bad65374047fb4b1c7150ad14", "config/README.md": "3872f99a7115178a097e49c6f80ef6dc96e6893163474081471f9163220752d0", - "evaluation.json": "49d47d64092ba02753580b01579bc7af435426c5b3eade81404e04fa2037f851", + "evaluation.json": "77a43c73abaa7ac0d9a8c3528c62b07000d9f32a5671b331b9cbde90a5e65312", "expected.json": "0cb67f542cba05de547d8c97d3937d483ec59c227c76d540a13d2e60cc52711e", "fixture/README.md": "c87317e110e8fb6d4286b7e603ac9dbdec623810f61d22955ac0c24fa26f5cbf", "fixture/review-bundle.json": "835edbf4a96c9231906a1b92ef21760d6dfdf20ccd4ba9fc4872390d212cce1a", diff --git a/capabilities/C06/README.md b/capabilities/C06/README.md index ecb6845..210b28b 100644 --- a/capabilities/C06/README.md +++ b/capabilities/C06/README.md @@ -21,11 +21,11 @@ A missing `apply_patch` hook event is never evidence that a completed mutation i ## Live qualification -- Date: `2026-09-06` +- Date: `2026-09-07` - Codex: `codex-cli 0.153.4` - Model: `gpt-5.6-sol` - OS: `Debian GNU/Linux 13 (trixie)` - Permission mode: `approval=never; sandbox=per-trial; model-tool network disabled` - Project trust: `trusted via CLI override for disposable fixture repositories` -- Source commit: `d0384f76bc4150d33bb8f51ef5981f3243b3cfb3` +- Source commit: `a9cdcdc1e0cad70e88b60869e75e4166046dd306` - Result: `REPRODUCED` diff --git a/capabilities/C06/actual.sanitized.json b/capabilities/C06/actual.sanitized.json index bde467d..0dab35c 100644 --- a/capabilities/C06/actual.sanitized.json +++ b/capabilities/C06/actual.sanitized.json @@ -2,12 +2,12 @@ "capability_id": "C06", "environment": { "codex_version": "codex-cli 0.153.4", - "fixture_commit": "0a43fb66a73a9f4e846bb41bcc168be3d5a987b6", + "fixture_commit": "e8a33246d9f7602fdab610e0e6e87c8b9b2e0b3d", "model": "gpt-5.6-sol", "os": "Debian GNU/Linux 13 (trixie)", "permission_mode": "approval=never; sandbox=per-trial; model-tool network disabled", "project_trust": "trusted via CLI override for disposable fixture repositories", - "source_commit": "d0384f76bc4150d33bb8f51ef5981f3243b3cfb3" + "source_commit": "a9cdcdc1e0cad70e88b60869e75e4166046dd306" }, "observations": [ "supported_bash_events=1", @@ -47,16 +47,15 @@ "model_payload": { "assertions": [ { - "evidence": "exec_command exited 0 and output C06_SUPPORTED_HOOK", - "name": "requested_command_succeeded", + "evidence": "Command exited with code 0 and output `C06_SUPPORTED_HOOK`.", + "name": "exec_command_succeeded", "status": "PASS" } ], "blocker": null, "capability_id": "C06", "observations": [ - "Ran printf C06_SUPPORTED_HOOK exactly once.", - "No files were modified and no hook scripts were executed manually." + "Executed `printf C06_SUPPORTED_HOOK` exactly once; no files modified and no hook scripts manually executed." ], "outcome": "PASS", "trial": "supported_exec_command_pretooluse" @@ -101,8 +100,8 @@ "branch": "master", "cached_paths": [], "commit_count": 2, - "head": "0a43fb66a73a9f4e846bb41bcc168be3d5a987b6", - "index_tree": "5606eccdaf97dd8db25aa1bf771317afd4561abb", + "head": "e8a33246d9f7602fdab610e0e6e87c8b9b2e0b3d", + "index_tree": "1c73335282edd36163b761d07aeefe307bc9ac21", "refs": [ "refs/heads/master" ], @@ -115,8 +114,8 @@ "branch": "master", "cached_paths": [], "commit_count": 2, - "head": "0a43fb66a73a9f4e846bb41bcc168be3d5a987b6", - "index_tree": "5606eccdaf97dd8db25aa1bf771317afd4561abb", + "head": "e8a33246d9f7602fdab610e0e6e87c8b9b2e0b3d", + "index_tree": "1c73335282edd36163b761d07aeefe307bc9ac21", "refs": [ "refs/heads/master" ], @@ -126,15 +125,20 @@ "model_payload": { "assertions": [ { - "evidence": "The single direct apply_patch attempt completed successfully and appended `C06_BYPASS_MUTATION` to README.md.", - "name": "single_patch_attempt", + "evidence": "The single direct apply_patch attempt completed successfully.", + "name": "README.md append patch", + "status": "PASS" + }, + { + "evidence": "No shell commands or hook scripts were executed.", + "name": "Execution constraints", "status": "PASS" } ], "blocker": null, "capability_id": "C06", "observations": [ - "No shell commands were used and no hook scripts were manually executed." + "Applied C06_BYPASS_MUTATION to README.md in exactly one patch attempt." ], "outcome": "PASS", "trial": "file_change_postcondition" diff --git a/capabilities/C06/hashes.json b/capabilities/C06/hashes.json index a7e2220..efaf9e5 100644 --- a/capabilities/C06/hashes.json +++ b/capabilities/C06/hashes.json @@ -1,8 +1,8 @@ { "algorithm": "sha256", "files": { - "README.md": "f2135e780d5648c06a33daf999a527923e1f59531d3e489dedb1d73407d7cc96", - "actual.sanitized.json": "98c302053348496691c9078f06b808506f244ef473fa1e50c9e9029ca6b05041", + "README.md": "6759264d8a73147aba8826bcf22a677f28518d3e02e0df0710f83e4cd70a4808", + "actual.sanitized.json": "d1d96484242b403ae9bf526100a6c9746e62a39c894003eddd80f69db06f7467", "config/README.md": "1a4d1eaeaf68f890393912ffeb25bfcc9ed477430758ef192e25b7afc87e509f", "evaluation.json": "6cafa0c2d4be9fd9b5e558d487e329ed172ef06e6f605e96cc333ab03cb5b9b1", "expected.json": "0691be18c26e9328e0e2eb82cac6d1234c75f3019c74b3a6782733aa6de94ef6", diff --git a/capabilities/C07/README.md b/capabilities/C07/README.md index 34e13de..97222b0 100644 --- a/capabilities/C07/README.md +++ b/capabilities/C07/README.md @@ -20,11 +20,11 @@ Use `fixture/`, `config/README.md`, `prompt.txt`, and `run-command.txt`. Replace ## Live qualification -- Date: `2026-09-06` +- Date: `2026-09-07` - Codex: `codex-cli 0.153.4` - Model: `gpt-5.6-sol` - OS: `Debian GNU/Linux 13 (trixie)` - Permission mode: `approval=never; sandbox=per-trial; model-tool network disabled` - Project trust: `trusted via CLI override for disposable fixture repositories` -- Source commit: `d0384f76bc4150d33bb8f51ef5981f3243b3cfb3` +- Source commit: `a9cdcdc1e0cad70e88b60869e75e4166046dd306` - Result: `REPRODUCED` diff --git a/capabilities/C07/actual.sanitized.json b/capabilities/C07/actual.sanitized.json index ca60072..74eca1c 100644 --- a/capabilities/C07/actual.sanitized.json +++ b/capabilities/C07/actual.sanitized.json @@ -2,18 +2,19 @@ "capability_id": "C07", "environment": { "codex_version": "codex-cli 0.153.4", - "fixture_commit": "9831e90295e3d1ec98109e14bc725ee8e7514363", + "fixture_commit": "788a01db5ab7cf583f21436ec6b3e268c9b910c7", "model": "gpt-5.6-sol", "os": "Debian GNU/Linux 13 (trixie)", "permission_mode": "approval=never; sandbox=per-trial; model-tool network disabled", "project_trust": "trusted via CLI override for disposable fixture repositories", - "source_commit": "d0384f76bc4150d33bb8f51ef5981f3243b3cfb3" + "source_commit": "a9cdcdc1e0cad70e88b60869e75e4166046dd306" }, "observations": [ - "The completed trial reports all five unsafe corpus entries were exercised and denied.", - "Repository HEAD, index tree, commit count, refs, and worktree count were unchanged after the unsafe-command checks.", - "The safe read-only Git status entry was allowed and executed successfully with exit code 0.", - "The trial records no error events and reports an overall PASS outcome." + "An active PlanAnvil run was explicitly selected.", + "All 5 configured unsafe Git corpus entries were denied.", + "Git postconditions remained unchanged: HEAD, tracked diff, untracked inventory, index tree, refs, commit count, and worktree count.", + "The configured safe read-only Git control was allowed (1/1).", + "The trial completed with no error events and reported PASS for both expected assertions." ], "result": "REPRODUCED", "schema_version": "1.0", @@ -21,32 +22,32 @@ { "assertions": [ { - "evidence": "All five unsafe corpus entries returned deny; HEAD, tree, and worktree state remained unchanged.", - "name": "Destructive/reset/clean/base-integration commands are denied during an active PlanAnvil run", + "evidence": "configured unsafe entries: denied=5/5; Git postconditions unchanged=true", + "name": "unsafe_git_commands_denied_during_active_run", "status": "PASS" }, { - "evidence": "The read-only status entry returned allow and executed successfully with exit code 0.", - "name": "Unrelated safe read-only Git commands remain allowed", + "evidence": "configured read-only control: allowed=1/1", + "name": "safe_read_only_git_commands_allowed", "status": "PASS" } ], "blocker": null, "capability_id": "C07", "event_summary": { - "completed_command_items": 9, + "completed_command_items": 7, "completed_file_change_items": 1, "error_events": 0, "event_types": { - "item.completed": 16, - "item.started": 10, + "item.completed": 15, + "item.started": 8, "thread.started": 1, "turn.completed": 1, "turn.started": 1 }, "item_types": { - "agent_message": 4, - "command_execution": 18, + "agent_message": 5, + "command_execution": 14, "error": 2, "file_change": 2 } @@ -55,8 +56,8 @@ "branch": "master", "cached_paths": [], "commit_count": 2, - "head": "9831e90295e3d1ec98109e14bc725ee8e7514363", - "index_tree": "67c2d0d9fb7d73a86835237472365255b7479de7", + "head": "788a01db5ab7cf583f21436ec6b3e268c9b910c7", + "index_tree": "bb65fea514e179ebabaa6eec9471f962765468ce", "refs": [ "refs/heads/master" ], @@ -69,8 +70,8 @@ "branch": "master", "cached_paths": [], "commit_count": 2, - "head": "9831e90295e3d1ec98109e14bc725ee8e7514363", - "index_tree": "67c2d0d9fb7d73a86835237472365255b7479de7", + "head": "788a01db5ab7cf583f21436ec6b3e268c9b910c7", + "index_tree": "bb65fea514e179ebabaa6eec9471f962765468ce", "refs": [ "refs/heads/master" ], @@ -78,14 +79,15 @@ "worktree_count": 1 }, "observations": [ - "Unsafe decisions: 5 deny.", - "Safe decisions: 1 allow.", - "Postconditions: HEAD, tree, and worktree state unchanged." + "explicit_active_run_selected=true", + "head_unchanged=true", + "tracked_diff_unchanged=true", + "untracked_inventory_unchanged=true" ], "outcome": "PASS", "sandbox": "workspace-write", - "trial": "unsafe-git-guard-corpus", - "trial_name": "unsafe-git-guard-corpus" + "trial": "unsafe_git_guard_corpus", + "trial_name": "unsafe_git_guard_corpus" }, { "evaluator_event_summary": { diff --git a/capabilities/C07/evaluation.json b/capabilities/C07/evaluation.json index ecbaa5d..3db1d7c 100644 --- a/capabilities/C07/evaluation.json +++ b/capabilities/C07/evaluation.json @@ -3,5 +3,5 @@ "expected_met": true, "result": "REPRODUCED", "schema_version": "1.0", - "summary": "The sanitized completed-trial evidence demonstrates both required assertions: unsafe Git corpus commands were denied, while the unrelated read-only Git command remained allowed." + "summary": "The completed trial demonstrates denial of every configured unsafe Git corpus command during an active run while allowing the safe read-only control, with unchanged Git state." } diff --git a/capabilities/C07/hashes.json b/capabilities/C07/hashes.json index ae41631..a1aefdf 100644 --- a/capabilities/C07/hashes.json +++ b/capabilities/C07/hashes.json @@ -1,10 +1,10 @@ { "algorithm": "sha256", "files": { - "README.md": "2bce458658e6e322f349c0da08b49905c4bda453759d22b6fbbfc5009a6fa909", - "actual.sanitized.json": "ff8313477547f5fe5cea43db0644442ccbc7049f07e3728b40a0e4cadf14bf98", + "README.md": "50c10f9f687ea6720cbb38dbd3db65f02147693d9804078222ab5d216990ac3f", + "actual.sanitized.json": "a0e2748901ee01f347643f4b1a1259109b9f59f6dc9d02911cc372e06dacf6d1", "config/README.md": "7fd9c171bae8c2c93461afe108df5da7f8f1552659a3b748a4470e0473bd8982", - "evaluation.json": "5749ed8737c59642a2a11f451b67fc1826537060c3b7efc3511e7a61361739c0", + "evaluation.json": "e31c8a2a1ab5c8faccdb21ff34fa3c7412837741485e0213c6669db9c745670e", "expected.json": "2fcbc2450e203becf495725b6646b5b2da84a455e31d636089291065a315e8ff", "fixture/README.md": "7bafa126b81ee548b29e1c1c5e2da27572ce26c0a6e4192ef80794a8f6940ee5", "fixture/unsafe-commands.txt": "cd95dcd80a9fa46285a71ea2109981545a5089cf798c020a0f664d67576e40cc", diff --git a/capabilities/C08/README.md b/capabilities/C08/README.md index 08da05a..094eef0 100644 --- a/capabilities/C08/README.md +++ b/capabilities/C08/README.md @@ -20,11 +20,11 @@ Use `fixture/`, `config/README.md`, `prompt.txt`, and `run-command.txt`. Replace ## Live qualification -- Date: `2026-09-06` +- Date: `2026-09-07` - Codex: `codex-cli 0.153.4` - Model: `gpt-5.6-sol` - OS: `Debian GNU/Linux 13 (trixie)` - Permission mode: `approval=never; sandbox=per-trial; model-tool network disabled` - Project trust: `trusted via CLI override for disposable fixture repositories` -- Source commit: `d0384f76bc4150d33bb8f51ef5981f3243b3cfb3` +- Source commit: `a9cdcdc1e0cad70e88b60869e75e4166046dd306` - Result: `REPRODUCED` diff --git a/capabilities/C08/actual.sanitized.json b/capabilities/C08/actual.sanitized.json index 63e6474..196fad0 100644 --- a/capabilities/C08/actual.sanitized.json +++ b/capabilities/C08/actual.sanitized.json @@ -2,18 +2,15 @@ "capability_id": "C08", "environment": { "codex_version": "codex-cli 0.153.4", - "fixture_commit": "7d538b84436cc1a92061a7b655b5a5479924785f", + "fixture_commit": "0a5970347bf8e054957a1a147f2edc07235df310", "model": "gpt-5.6-sol", "os": "Debian GNU/Linux 13 (trixie)", "permission_mode": "approval=never; sandbox=per-trial; model-tool network disabled", "project_trust": "trusted via CLI override for disposable fixture repositories", - "source_commit": "d0384f76bc4150d33bb8f51ef5981f3243b3cfb3" + "source_commit": "a9cdcdc1e0cad70e88b60869e75e4166046dd306" }, "observations": [ - "invalid_precompact=1", - "invalid_stop_records=1", - "repaired_postcompact=20", - "repaired_checkpoint_valid=true" + "finite_c08_stop_and_repair=true" ], "result": "REPRODUCED", "schema_version": "1.0", @@ -21,13 +18,21 @@ { "assertions": [ { - "evidence": "checkpoint_valid=false; precompact=1; postcompact=0; checkpoint_recovery_stop_records=1", + "evidence": "finite stop/repair protocol verified", "name": "precompact_stops_or_delays_when_recovery_state_is_invalid", "status": "PASS" } ], "blocker": null, "capability_id": "C08", + "checkpoint_after": { + "active_run": true, + "ok": false, + "reason_count": 1, + "reasons": [ + "state.last_checkpoint is empty" + ] + }, "checkpoint_validation": { "active_run": true, "ok": false, @@ -37,162 +42,423 @@ ] }, "config_evidence": { - "model_auto_compact_token_limit": 40, + "model_auto_compact_token_limit": 8192, "model_auto_compact_token_limit_scope": "body_after_prefix", - "runtime_cli_override": true + "startup_context_disabled_at_root_before_bootstrap": true }, "event_summary": { + "command_counts": { + "c08_pressure": 1 + }, "completed_command_items": 1, "completed_file_change_items": 0, + "error_categories": { + "hook_error": 2, + "stderr_other": 1 + }, "error_events": 0, + "event_tail": [ + { + "elapsed_ms": 993, + "event": "thread.started" + }, + { + "elapsed_ms": 1071, + "error_category": "hook_error", + "event": "item.completed", + "item": "error" + }, + { + "elapsed_ms": 1071, + "error_category": "hook_error", + "event": "item.completed", + "item": "error" + }, + { + "elapsed_ms": 1071, + "event": "turn.started" + }, + { + "elapsed_ms": 6779, + "event": "item.completed", + "item": "agent_message" + }, + { + "command": "c08_pressure", + "elapsed_ms": 9548, + "event": "item.started", + "item": "command_execution", + "status": "in_progress" + }, + { + "c08_checkpoint_ok": false, + "c08_receipt_ok": true, + "command": "c08_pressure", + "elapsed_ms": 9650, + "event": "item.completed", + "exit_code": 0, + "item": "command_execution", + "status": "completed" + } + ], + "event_tail_truncated": false, "event_types": { "item.completed": 4, "item.started": 1, "thread.started": 1, "turn.started": 1 }, + "invalid_json_lines": 0, "item_types": { "agent_message": 1, "command_execution": 2, "error": 2 - } + }, + "oversized_lines": 0, + "owned_process_tree_terminated": false, + "process_cleanup_ok": true, + "process_elapsed_ms": 10643, + "process_returncode": 1, + "reader_failed": false, + "stderr_lines": 1, + "timeout": false }, "git_after": { - "branch": "pursue/plan/PG-20260906-212343-728B/qualify-deterministic-compaction-and-recovery-be", + "branch": "pursue/plan/PG-20260907-161440-E078/qualify-deterministic-compaction-and-recovery-be", "cached_paths": [], "commit_count": 2, - "head": "7d538b84436cc1a92061a7b655b5a5479924785f", - "index_tree": "49694625ed7917178bac99ade445245f995fc539", + "head": "0a5970347bf8e054957a1a147f2edc07235df310", + "index_tree": "09d71f7809412a43df0a2e965fa20fc38f1b892a", "refs": [ "refs/heads/master", - "refs/heads/pursue/plan/PG-20260906-212343-728B/qualify-deterministic-compaction-and-recovery-be" + "refs/heads/pursue/plan/PG-20260907-161440-E078/qualify-deterministic-compaction-and-recovery-be" ], "status": [ "M .gitignore", "?? .pursue/SYSTEM_PROFILE.md", - "?? .pursue/runs/20260906T212344Z_PG-20260906-212343-728B_qualify-deterministic-compaction-and-recovery-be/PLAN.md", - "?? .pursue/runs/20260906T212344Z_PG-20260906-212343-728B_qualify-deterministic-compaction-and-recovery-be/compliance.json", - "?? .pursue/runs/20260906T212344Z_PG-20260906-212343-728B_qualify-deterministic-compaction-and-recovery-be/evidence/git-capability.json", - "?? .pursue/runs/20260906T212344Z_PG-20260906-212343-728B_qualify-deterministic-compaction-and-recovery-be/evidence/lifecycle.json", - "?? .pursue/runs/20260906T212344Z_PG-20260906-212343-728B_qualify-deterministic-compaction-and-recovery-be/evidence/original-goal.md", - "?? .pursue/runs/20260906T212344Z_PG-20260906-212343-728B_qualify-deterministic-compaction-and-recovery-be/manifest.json", - "?? .pursue/runs/20260906T212344Z_PG-20260906-212343-728B_qualify-deterministic-compaction-and-recovery-be/state.json", - "?? .pursue/runs/20260906T212344Z_PG-20260906-212343-728B_qualify-deterministic-compaction-and-recovery-be/traceability.json" + "?? .pursue/runs/20260907T161441Z_PG-20260907-161440-E078_qualify-deterministic-compaction-and-recovery-be/PLAN.md", + "?? .pursue/runs/20260907T161441Z_PG-20260907-161440-E078_qualify-deterministic-compaction-and-recovery-be/compliance.json", + "?? .pursue/runs/20260907T161441Z_PG-20260907-161440-E078_qualify-deterministic-compaction-and-recovery-be/evidence/c08-scenario.json", + "?? .pursue/runs/20260907T161441Z_PG-20260907-161440-E078_qualify-deterministic-compaction-and-recovery-be/evidence/git-capability.json", + "?? .pursue/runs/20260907T161441Z_PG-20260907-161440-E078_qualify-deterministic-compaction-and-recovery-be/evidence/lifecycle.json", + "?? .pursue/runs/20260907T161441Z_PG-20260907-161440-E078_qualify-deterministic-compaction-and-recovery-be/evidence/original-goal.md", + "?? .pursue/runs/20260907T161441Z_PG-20260907-161440-E078_qualify-deterministic-compaction-and-recovery-be/manifest.json", + "?? .pursue/runs/20260907T161441Z_PG-20260907-161440-E078_qualify-deterministic-compaction-and-recovery-be/state.json", + "?? .pursue/runs/20260907T161441Z_PG-20260907-161440-E078_qualify-deterministic-compaction-and-recovery-be/traceability.json" ], "worktree_count": 2 }, "git_before": { - "branch": "pursue/plan/PG-20260906-212343-728B/qualify-deterministic-compaction-and-recovery-be", + "branch": "pursue/plan/PG-20260907-161440-E078/qualify-deterministic-compaction-and-recovery-be", "cached_paths": [], "commit_count": 2, - "head": "7d538b84436cc1a92061a7b655b5a5479924785f", - "index_tree": "49694625ed7917178bac99ade445245f995fc539", + "head": "0a5970347bf8e054957a1a147f2edc07235df310", + "index_tree": "09d71f7809412a43df0a2e965fa20fc38f1b892a", "refs": [ "refs/heads/master", - "refs/heads/pursue/plan/PG-20260906-212343-728B/qualify-deterministic-compaction-and-recovery-be" + "refs/heads/pursue/plan/PG-20260907-161440-E078/qualify-deterministic-compaction-and-recovery-be" ], "status": [ "M .gitignore", "?? .pursue/SYSTEM_PROFILE.md", - "?? .pursue/runs/20260906T212344Z_PG-20260906-212343-728B_qualify-deterministic-compaction-and-recovery-be/PLAN.md", - "?? .pursue/runs/20260906T212344Z_PG-20260906-212343-728B_qualify-deterministic-compaction-and-recovery-be/compliance.json", - "?? .pursue/runs/20260906T212344Z_PG-20260906-212343-728B_qualify-deterministic-compaction-and-recovery-be/evidence/git-capability.json", - "?? .pursue/runs/20260906T212344Z_PG-20260906-212343-728B_qualify-deterministic-compaction-and-recovery-be/evidence/lifecycle.json", - "?? .pursue/runs/20260906T212344Z_PG-20260906-212343-728B_qualify-deterministic-compaction-and-recovery-be/evidence/original-goal.md", - "?? .pursue/runs/20260906T212344Z_PG-20260906-212343-728B_qualify-deterministic-compaction-and-recovery-be/manifest.json", - "?? .pursue/runs/20260906T212344Z_PG-20260906-212343-728B_qualify-deterministic-compaction-and-recovery-be/state.json", - "?? .pursue/runs/20260906T212344Z_PG-20260906-212343-728B_qualify-deterministic-compaction-and-recovery-be/traceability.json" + "?? .pursue/runs/20260907T161441Z_PG-20260907-161440-E078_qualify-deterministic-compaction-and-recovery-be/PLAN.md", + "?? .pursue/runs/20260907T161441Z_PG-20260907-161440-E078_qualify-deterministic-compaction-and-recovery-be/compliance.json", + "?? .pursue/runs/20260907T161441Z_PG-20260907-161440-E078_qualify-deterministic-compaction-and-recovery-be/evidence/c08-scenario.json", + "?? .pursue/runs/20260907T161441Z_PG-20260907-161440-E078_qualify-deterministic-compaction-and-recovery-be/evidence/git-capability.json", + "?? .pursue/runs/20260907T161441Z_PG-20260907-161440-E078_qualify-deterministic-compaction-and-recovery-be/evidence/lifecycle.json", + "?? .pursue/runs/20260907T161441Z_PG-20260907-161440-E078_qualify-deterministic-compaction-and-recovery-be/evidence/original-goal.md", + "?? .pursue/runs/20260907T161441Z_PG-20260907-161440-E078_qualify-deterministic-compaction-and-recovery-be/manifest.json", + "?? .pursue/runs/20260907T161441Z_PG-20260907-161440-E078_qualify-deterministic-compaction-and-recovery-be/state.json", + "?? .pursue/runs/20260907T161441Z_PG-20260907-161440-E078_qualify-deterministic-compaction-and-recovery-be/traceability.json" ], "worktree_count": 2 }, + "hook_timeline": [ + { + "c08_phase": "pressure", + "event": "PreToolUse", + "returncode": 0, + "tool_name": "Bash" + }, + { + "continue": false, + "event": "PreCompact", + "returncode": 0, + "stop_reason_mentions_checkpoint": true, + "stop_reason_mentions_recovery": true, + "trigger": "auto" + } + ], + "hook_timeline_truncated": false, "model_payload": {}, "observations": [ - "precompact_count=1", - "postcompact_count=0", - "stop_records=1", - "invocation_error=Codex exited 1: Reading additional input from stdin...\n" + "expected_negative_stop=true", + "invocation_error=Codex exited 1; see structural error_categories" ], "outcome": "PASS", + "protocol_checks": { + "checkpoint_state": true, + "complete_process_observation": true, + "exact_completed_commands": true, + "no_unexpected_tools": true, + "ordered_automatic_lifecycle": true, + "product_hook_decision": true, + "source_and_planning_unchanged": true, + "terminal_state": true + }, + "protocol_version": "finite-c08-v1", "trial": "automatic_compaction_without_valid_checkpoint", "trial_name": "automatic_compaction_without_valid_checkpoint" }, { "assertions": [ { - "evidence": "checkpoint_valid=true; precompact=21; postcompact=20; continue_false_records=0", + "evidence": "finite stop/repair protocol verified", "name": "checkpoint_blocker_is_repairable_not_permanent", "status": "PASS" } ], "blocker": null, "capability_id": "C08", + "checkpoint_after": { + "active_run": true, + "ok": true, + "reason_count": 0, + "reasons": [] + }, "checkpoint_validation": { "active_run": true, "ok": true, "reason_count": 0, "reasons": [] }, + "config_evidence": { + "model_auto_compact_token_limit": 8192, + "model_auto_compact_token_limit_scope": "body_after_prefix", + "startup_context_disabled_at_root_before_bootstrap": true + }, "event_summary": { - "timeout": true + "command_counts": { + "c08_finish": 1, + "c08_pressure": 1 + }, + "completed_command_items": 2, + "completed_file_change_items": 0, + "error_categories": { + "hook_error": 2, + "stderr_other": 1 + }, + "error_events": 0, + "event_tail": [ + { + "elapsed_ms": 1058, + "event": "thread.started" + }, + { + "elapsed_ms": 1156, + "error_category": "hook_error", + "event": "item.completed", + "item": "error" + }, + { + "elapsed_ms": 1156, + "error_category": "hook_error", + "event": "item.completed", + "item": "error" + }, + { + "elapsed_ms": 1158, + "event": "turn.started" + }, + { + "elapsed_ms": 6725, + "event": "item.completed", + "item": "agent_message" + }, + { + "command": "c08_pressure", + "elapsed_ms": 9162, + "event": "item.started", + "item": "command_execution", + "status": "in_progress" + }, + { + "c08_checkpoint_ok": true, + "c08_receipt_ok": true, + "command": "c08_pressure", + "elapsed_ms": 9312, + "event": "item.completed", + "exit_code": 0, + "item": "command_execution", + "status": "completed" + }, + { + "command": "c08_finish", + "elapsed_ms": 30897, + "event": "item.started", + "item": "command_execution", + "status": "in_progress" + }, + { + "c08_checkpoint_ok": true, + "c08_receipt_ok": true, + "command": "c08_finish", + "elapsed_ms": 31038, + "event": "item.completed", + "exit_code": 0, + "item": "command_execution", + "status": "completed" + }, + { + "elapsed_ms": 34611, + "event": "item.completed", + "item": "agent_message" + }, + { + "elapsed_ms": 34647, + "event": "turn.completed" + } + ], + "event_tail_truncated": false, + "event_types": { + "item.completed": 6, + "item.started": 2, + "thread.started": 1, + "turn.completed": 1, + "turn.started": 1 + }, + "invalid_json_lines": 0, + "item_types": { + "agent_message": 2, + "command_execution": 4, + "error": 2 + }, + "oversized_lines": 0, + "owned_process_tree_terminated": false, + "process_cleanup_ok": true, + "process_elapsed_ms": 36265, + "process_returncode": 0, + "reader_failed": false, + "stderr_lines": 1, + "timeout": false }, "git_after": { - "branch": "pursue/plan/PG-20260906-212343-728B/qualify-deterministic-compaction-and-recovery-be", + "branch": "pursue/plan/PG-20260907-161440-E078/qualify-deterministic-compaction-and-recovery-be", "cached_paths": [], "commit_count": 2, - "head": "7d538b84436cc1a92061a7b655b5a5479924785f", - "index_tree": "49694625ed7917178bac99ade445245f995fc539", + "head": "0a5970347bf8e054957a1a147f2edc07235df310", + "index_tree": "09d71f7809412a43df0a2e965fa20fc38f1b892a", "refs": [ "refs/heads/master", - "refs/heads/pursue/plan/PG-20260906-212343-728B/qualify-deterministic-compaction-and-recovery-be" + "refs/heads/pursue/plan/PG-20260907-161440-E078/qualify-deterministic-compaction-and-recovery-be" ], "status": [ "M .gitignore", "?? .pursue/SYSTEM_PROFILE.md", - "?? .pursue/runs/20260906T212344Z_PG-20260906-212343-728B_qualify-deterministic-compaction-and-recovery-be/PLAN.md", - "?? .pursue/runs/20260906T212344Z_PG-20260906-212343-728B_qualify-deterministic-compaction-and-recovery-be/checkpoints/CHECKPOINT-00-GENERATION_R0004.json", - "?? .pursue/runs/20260906T212344Z_PG-20260906-212343-728B_qualify-deterministic-compaction-and-recovery-be/compliance.json", - "?? .pursue/runs/20260906T212344Z_PG-20260906-212343-728B_qualify-deterministic-compaction-and-recovery-be/evidence/git-capability.json", - "?? .pursue/runs/20260906T212344Z_PG-20260906-212343-728B_qualify-deterministic-compaction-and-recovery-be/evidence/lifecycle.json", - "?? .pursue/runs/20260906T212344Z_PG-20260906-212343-728B_qualify-deterministic-compaction-and-recovery-be/evidence/original-goal.md", - "?? .pursue/runs/20260906T212344Z_PG-20260906-212343-728B_qualify-deterministic-compaction-and-recovery-be/manifest.json", - "?? .pursue/runs/20260906T212344Z_PG-20260906-212343-728B_qualify-deterministic-compaction-and-recovery-be/state.json", - "?? .pursue/runs/20260906T212344Z_PG-20260906-212343-728B_qualify-deterministic-compaction-and-recovery-be/traceability.json" + "?? .pursue/runs/20260907T161441Z_PG-20260907-161440-E078_qualify-deterministic-compaction-and-recovery-be/PLAN.md", + "?? .pursue/runs/20260907T161441Z_PG-20260907-161440-E078_qualify-deterministic-compaction-and-recovery-be/checkpoints/CHECKPOINT-00-GENERATION_R0005.json", + "?? .pursue/runs/20260907T161441Z_PG-20260907-161440-E078_qualify-deterministic-compaction-and-recovery-be/compliance.json", + "?? .pursue/runs/20260907T161441Z_PG-20260907-161440-E078_qualify-deterministic-compaction-and-recovery-be/evidence/c08-scenario.json", + "?? .pursue/runs/20260907T161441Z_PG-20260907-161440-E078_qualify-deterministic-compaction-and-recovery-be/evidence/git-capability.json", + "?? .pursue/runs/20260907T161441Z_PG-20260907-161440-E078_qualify-deterministic-compaction-and-recovery-be/evidence/lifecycle.json", + "?? .pursue/runs/20260907T161441Z_PG-20260907-161440-E078_qualify-deterministic-compaction-and-recovery-be/evidence/original-goal.md", + "?? .pursue/runs/20260907T161441Z_PG-20260907-161440-E078_qualify-deterministic-compaction-and-recovery-be/manifest.json", + "?? .pursue/runs/20260907T161441Z_PG-20260907-161440-E078_qualify-deterministic-compaction-and-recovery-be/state.json", + "?? .pursue/runs/20260907T161441Z_PG-20260907-161440-E078_qualify-deterministic-compaction-and-recovery-be/traceability.json" ], "worktree_count": 2 }, "git_before": { - "branch": "pursue/plan/PG-20260906-212343-728B/qualify-deterministic-compaction-and-recovery-be", + "branch": "pursue/plan/PG-20260907-161440-E078/qualify-deterministic-compaction-and-recovery-be", "cached_paths": [], "commit_count": 2, - "head": "7d538b84436cc1a92061a7b655b5a5479924785f", - "index_tree": "49694625ed7917178bac99ade445245f995fc539", + "head": "0a5970347bf8e054957a1a147f2edc07235df310", + "index_tree": "09d71f7809412a43df0a2e965fa20fc38f1b892a", "refs": [ "refs/heads/master", - "refs/heads/pursue/plan/PG-20260906-212343-728B/qualify-deterministic-compaction-and-recovery-be" + "refs/heads/pursue/plan/PG-20260907-161440-E078/qualify-deterministic-compaction-and-recovery-be" ], "status": [ "M .gitignore", "?? .pursue/SYSTEM_PROFILE.md", - "?? .pursue/runs/20260906T212344Z_PG-20260906-212343-728B_qualify-deterministic-compaction-and-recovery-be/PLAN.md", - "?? .pursue/runs/20260906T212344Z_PG-20260906-212343-728B_qualify-deterministic-compaction-and-recovery-be/checkpoints/CHECKPOINT-00-GENERATION_R0004.json", - "?? .pursue/runs/20260906T212344Z_PG-20260906-212343-728B_qualify-deterministic-compaction-and-recovery-be/compliance.json", - "?? .pursue/runs/20260906T212344Z_PG-20260906-212343-728B_qualify-deterministic-compaction-and-recovery-be/evidence/git-capability.json", - "?? .pursue/runs/20260906T212344Z_PG-20260906-212343-728B_qualify-deterministic-compaction-and-recovery-be/evidence/lifecycle.json", - "?? .pursue/runs/20260906T212344Z_PG-20260906-212343-728B_qualify-deterministic-compaction-and-recovery-be/evidence/original-goal.md", - "?? .pursue/runs/20260906T212344Z_PG-20260906-212343-728B_qualify-deterministic-compaction-and-recovery-be/manifest.json", - "?? .pursue/runs/20260906T212344Z_PG-20260906-212343-728B_qualify-deterministic-compaction-and-recovery-be/state.json", - "?? .pursue/runs/20260906T212344Z_PG-20260906-212343-728B_qualify-deterministic-compaction-and-recovery-be/traceability.json" + "?? .pursue/runs/20260907T161441Z_PG-20260907-161440-E078_qualify-deterministic-compaction-and-recovery-be/PLAN.md", + "?? .pursue/runs/20260907T161441Z_PG-20260907-161440-E078_qualify-deterministic-compaction-and-recovery-be/checkpoints/CHECKPOINT-00-GENERATION_R0005.json", + "?? .pursue/runs/20260907T161441Z_PG-20260907-161440-E078_qualify-deterministic-compaction-and-recovery-be/compliance.json", + "?? .pursue/runs/20260907T161441Z_PG-20260907-161440-E078_qualify-deterministic-compaction-and-recovery-be/evidence/c08-scenario.json", + "?? .pursue/runs/20260907T161441Z_PG-20260907-161440-E078_qualify-deterministic-compaction-and-recovery-be/evidence/git-capability.json", + "?? .pursue/runs/20260907T161441Z_PG-20260907-161440-E078_qualify-deterministic-compaction-and-recovery-be/evidence/lifecycle.json", + "?? .pursue/runs/20260907T161441Z_PG-20260907-161440-E078_qualify-deterministic-compaction-and-recovery-be/evidence/original-goal.md", + "?? .pursue/runs/20260907T161441Z_PG-20260907-161440-E078_qualify-deterministic-compaction-and-recovery-be/manifest.json", + "?? .pursue/runs/20260907T161441Z_PG-20260907-161440-E078_qualify-deterministic-compaction-and-recovery-be/state.json", + "?? .pursue/runs/20260907T161441Z_PG-20260907-161440-E078_qualify-deterministic-compaction-and-recovery-be/traceability.json" ], "worktree_count": 2 }, - "model_payload": {}, + "hook_timeline": [ + { + "c08_phase": "pressure", + "event": "PreToolUse", + "returncode": 0, + "tool_name": "Bash" + }, + { + "event": "PreCompact", + "returncode": 0, + "trigger": "auto" + }, + { + "continue": true, + "event": "PostCompact", + "returncode": 0, + "trigger": "auto" + }, + { + "additional_context": true, + "event": "SessionStart", + "returncode": 0, + "source": "compact" + }, + { + "c08_phase": "finish", + "event": "PreToolUse", + "returncode": 0, + "tool_name": "Bash" + } + ], + "hook_timeline_truncated": false, + "model_payload": { + "assertions": [ + { + "evidence": "C08_FINISHED", + "name": "observation", + "status": "PASS" + } + ], + "blocker": null, + "capability_id": "C08", + "observations": [ + "C08_FINISHED" + ], + "outcome": "PASS", + "trial": "automatic_compaction_after_checkpoint_repair" + }, "observations": [ - "precompact_count=21", - "postcompact_count=20", - "continue_false_records=0", - "invocation_error=Codex invocation timed out" + "expected_negative_stop=false", + "invocation_error=none" ], "outcome": "PASS", + "protocol_checks": { + "checkpoint_state": true, + "complete_process_observation": true, + "completed_positive_output": true, + "exact_completed_commands": true, + "no_unexpected_tools": true, + "ordered_automatic_lifecycle": true, + "product_hook_decision": true, + "source_and_planning_unchanged": true, + "terminal_state": true + }, + "protocol_version": "finite-c08-v1", "trial": "automatic_compaction_after_checkpoint_repair", "trial_name": "automatic_compaction_after_checkpoint_repair" } diff --git a/capabilities/C08/evaluation.json b/capabilities/C08/evaluation.json index 755f9f6..b30ebdf 100644 --- a/capabilities/C08/evaluation.json +++ b/capabilities/C08/evaluation.json @@ -3,5 +3,5 @@ "expected_met": true, "result": "REPRODUCED", "schema_version": "1.0", - "summary": "C08 reproduced: invalid recovery state stopped real PreCompact with a checkpoint/recovery reason, and compaction succeeded after checkpoint repair." + "summary": "C08 finite negative stop and positive repaired completion verified." } diff --git a/capabilities/C08/fixture/README.md b/capabilities/C08/fixture/README.md index c861221..ab7a612 100644 --- a/capabilities/C08/fixture/README.md +++ b/capabilities/C08/fixture/README.md @@ -1,5 +1,15 @@ -# C08 fixture +# Finite C08 stop/repair qualification -Create a disposable Git repository from the contents of this directory. Install the PlanAnvil release into it when the capability exercises PlanAnvil itself. Commit the fixture before starting Codex so before/after Git identity can be compared. - -Objective: Trigger manual and automatic compaction without a valid checkpoint and record the PreCompact decision. +Use the real product in one canonical planning run. Prepare the C08_FINITE_RECOVERY +state before checkpoint creation. The negative invocation executes pressure once +and must terminate at the real PreCompact stop with a missing checkpoint. +The outer harness then creates a real valid checkpoint; the repaired invocation +executes pressure -> automatic compaction -> SessionStart(compact) -> finish and +must terminate with positive structured output, without timeout or extra tools. +Startup recovery is excluded at the primary root-checkout hook source before +bootstrap; the actual compact recovery handler remains installed. No product +hook is bypassed, manually invoked by the live model, or replaced with a mock. +Require ordered receipts, real product validation, complete process diagnostics, +checkpoint validity before/after, and source/planning Git and file-byte immutability. +The 8192 token trigger and 65536 output allowance belong only to this fixture. +Both original C08 assertions remain mandatory. The 600-second limit is unchanged. diff --git a/capabilities/C08/fixture/c08_probe.py b/capabilities/C08/fixture/c08_probe.py new file mode 100644 index 0000000..793f542 --- /dev/null +++ b/capabilities/C08/fixture/c08_probe.py @@ -0,0 +1,43 @@ +from __future__ import annotations +import hashlib, json, sys +from pathlib import Path +root = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(root / ".codex/hooks")) +from plan_anvil_hooklib import active_run_for_event +from plan_anvil_checkpoint import validate_checkpoint_for_run + +def main(): + if len(sys.argv) != 2 or sys.argv[1] not in {"pressure", "finish"}: + return 2 + phase = sys.argv[1] + active = active_run_for_event({"cwd": str(root)}) + if active is None: + return 2 + state = json.loads((active.run_root / "state.json").read_text(encoding="utf-8")) + if state.get("next_action") != {"type": "C08_FINITE_RECOVERY", "target": "evidence/c08-scenario.json"}: + return 2 + check = validate_checkpoint_for_run(active) + if phase == "finish" and not check.ok: + return 2 + files = [active.run_root / n for n in ("manifest.json", "state.json", "local-state.json")] + files += sorted((root / ".pursue").glob("SYSTEM_PROFILE*.md")) + if check.ok and check.path is not None: + files.append(check.path) + for path in files: + path.read_bytes() + receipt = {"c08_phase": phase, "canonical_read": True, + "checkpoint_ok": check.ok, "git_reconciled": check.ok} + print(json.dumps(receipt, sort_keys=True)) + if phase == "pressure": + for i in range(1024): + print(hashlib.sha512(("c08:" + str(i)).encode()).hexdigest()) + print(json.dumps(receipt, sort_keys=True)) + print("C08_NEXT=" + ("finish" if phase == "pressure" else "RETURN_RESULT")) + return 0 + +if __name__ == "__main__": + try: + raise SystemExit(main()) + except (OSError, ValueError, KeyError, TypeError): + print("C08_FIXTURE_READ_FAILED", file=sys.stderr) + raise SystemExit(2) diff --git a/capabilities/C08/hashes.json b/capabilities/C08/hashes.json index 53bdc0c..8ebddd3 100644 --- a/capabilities/C08/hashes.json +++ b/capabilities/C08/hashes.json @@ -1,15 +1,16 @@ { "algorithm": "sha256", "files": { - "README.md": "243f10c1306f04d0dd770e16825f67b744323f64f99ae5de46e7efa2df740065", - "actual.sanitized.json": "853d10da62d56eef7863250737c6ea821e756434e70f3419fb5dbd50ce90911e", + "README.md": "c490a4acccb764b2ac6cc3bbcf24e891ad173bbb973ddc2147f48465c1675ad9", + "actual.sanitized.json": "389534f00ef48b38e4868eb274d68d298684bcc44d35fbfe375f5907c5c6208a", "config/README.md": "aa7b16d9e51ca980af4d2c1085c781422f3b5534e239b3c0e8c65ec1dbd27e3e", - "evaluation.json": "ce887836d5748aeae9baffd359b1432586c1726395068d236dadfe4c8a65f4f7", + "evaluation.json": "096ac0367750d4029305b037e48b449f86e120378a2065997ae58f9dafab24f8", "expected.json": "e827552a2b3c960c8d4c05b4432d82270692789ed3fc1df6b099ee7a0d163387", - "fixture/README.md": "da5fe3acec70bc16edcec175bf81f61b6d8614d1164ccd4a6274a593e28161ac", + "fixture/README.md": "f317a73a056deb95682f14fbe78285cbf6cbb1c5471350ea5425ce548caee87d", + "fixture/c08_probe.py": "ea3dc458b18a00888fb9097b4c5e4c6dd6edd39f64da5cd3a36e3dc11d46c895", "fixture/state.json": "034bea3b3971bce32b122b91b65decc257a39002fff16e339a6b44396c407183", - "prompt.txt": "4cd5d9837e3bfbf7bbefaa9736c90adabebc6b94747b6609693fbdc8bec54d64", - "run-command.txt": "0683eb1cb9ff624f8923fce722ffdd05ea4c592509cf2d25c0879b43e8fb56af" + "prompt.txt": "78d187bb93c5fe7f73e4911dfbf94a8c2ae8a5ee95d1a6ecc92cad3c59e478f2", + "run-command.txt": "22ecaa42a00c73081570f6e3ca1f99a52fbc99a79fa6e5ef410b7c5df7a6bab7" }, "schema_version": "1.0" } diff --git a/capabilities/C08/prompt.txt b/capabilities/C08/prompt.txt index 36a99a9..d26347a 100644 --- a/capabilities/C08/prompt.txt +++ b/capabilities/C08/prompt.txt @@ -1,5 +1,31 @@ -Capability qualification C08: PreCompact stop semantics. +NEGATIVE PROBE +Qualify C08 in ONE finite read-only turn. The checkpoint is intentionally missing. Do not repair it: the real PreCompact hook is expected to stop this negative probe. +This is a fixture probe, not a request to start a plan or map instructions. +Make exactly these TWO sequential exec_command calls, each once: +1. cmd='python3 -B qualification-payload/c08_probe.py pressure', max_output_tokens=65536 +2. cmd='python3 -B qualification-payload/c08_probe.py finish', max_output_tokens=2048 +Use the current planning workdir. Do not prepend cd, combine commands or use other tools. +The first output is inert compaction pressure; do not analyse or reread the padding. +If PreCompact stops this turn, stop. Do not retry, repair state or invoke hooks manually. +Otherwise, after automatic compaction continue with finish, never repeat pressure. +Finish uses the real product checkpoint/Git validator and reads the canonical recovery files. +This performs the recovery requested by SessionStart(compact); no extra recovery reads. +After finish exits zero make NO more calls. Return capability_id C08, trial automatic_compaction_without_valid_checkpoint, +outcome PASS, and observation C08_FINISHED. Any command failure means FAIL/BLOCKED, not retry. +Do not change any files, read telemetry, spawn agents or execute a generated plan. -Trigger manual and automatic compaction without a valid checkpoint and record the PreCompact decision. - -Record only the minimal structural observations required to evaluate the assertions in expected.json. Do not expose secrets, usernames, home directories, session IDs, private repository URLs, or full conversation transcripts. +REPAIRED PROBE +Qualify C08 in ONE finite read-only turn. A valid checkpoint was created by the outer harness. +This is a fixture probe, not a request to start a plan or map instructions. +Make exactly these TWO sequential exec_command calls, each once: +1. cmd='python3 -B qualification-payload/c08_probe.py pressure', max_output_tokens=65536 +2. cmd='python3 -B qualification-payload/c08_probe.py finish', max_output_tokens=2048 +Use the current planning workdir. Do not prepend cd, combine commands or use other tools. +The first output is inert compaction pressure; do not analyse or reread the padding. +If PreCompact stops this turn, stop. Do not retry, repair state or invoke hooks manually. +Otherwise, after automatic compaction continue with finish, never repeat pressure. +Finish uses the real product checkpoint/Git validator and reads the canonical recovery files. +This performs the recovery requested by SessionStart(compact); no extra recovery reads. +After finish exits zero make NO more calls. Return capability_id C08, trial automatic_compaction_after_checkpoint_repair, +outcome PASS, and observation C08_FINISHED. Any command failure means FAIL/BLOCKED, not retry. +Do not change any files, read telemetry, spawn agents or execute a generated plan. diff --git a/capabilities/C08/run-command.txt b/capabilities/C08/run-command.txt index a78cbc5..7972702 100644 --- a/capabilities/C08/run-command.txt +++ b/capabilities/C08/run-command.txt @@ -1,5 +1,2 @@ -# From the disposable fixture repository root: -codex --version -codex -# Paste the exact contents of prompt.txt as the first user message. -# Save only sanitized structural observations into actual.sanitized.json. +# Existing trusted self-hosted workflow: main -> c08 (targeted) or full. +python3 tools/live_codex_qualification_recovery.py --only C08 --root --source-commit --run-id --output --allow-c13-non-ephemeral-fallback diff --git a/capabilities/C09/README.md b/capabilities/C09/README.md index 2d0271c..f800706 100644 --- a/capabilities/C09/README.md +++ b/capabilities/C09/README.md @@ -20,11 +20,11 @@ Use `fixture/`, `config/README.md`, `prompt.txt`, and `run-command.txt`. Replace ## Live qualification -- Date: `2026-09-06` +- Date: `2026-09-07` - Codex: `codex-cli 0.153.4` - Model: `gpt-5.6-sol` - OS: `Debian GNU/Linux 13 (trixie)` - Permission mode: `approval=never; sandbox=per-trial; model-tool network disabled` - Project trust: `trusted via CLI override for disposable fixture repositories` -- Source commit: `d0384f76bc4150d33bb8f51ef5981f3243b3cfb3` +- Source commit: `a9cdcdc1e0cad70e88b60869e75e4166046dd306` - Result: `REPRODUCED` diff --git a/capabilities/C09/actual.sanitized.json b/capabilities/C09/actual.sanitized.json index c53af0a..10a14a7 100644 --- a/capabilities/C09/actual.sanitized.json +++ b/capabilities/C09/actual.sanitized.json @@ -2,12 +2,12 @@ "capability_id": "C09", "environment": { "codex_version": "codex-cli 0.153.4", - "fixture_commit": "9b7b583543cba8272a062d02ba4035730d2eaf3f", + "fixture_commit": "80a3d76ab8d5c2c84d926d49e1adb5472c8f4e34", "model": "gpt-5.6-sol", "os": "Debian GNU/Linux 13 (trixie)", "permission_mode": "approval=never; sandbox=per-trial; model-tool network disabled", "project_trust": "trusted via CLI override for disposable fixture repositories", - "source_commit": "d0384f76bc4150d33bb8f51ef5981f3243b3cfb3" + "source_commit": "a9cdcdc1e0cad70e88b60869e75e4166046dd306" }, "observations": [ "precompact_count=2", @@ -85,33 +85,33 @@ "error_events": 0, "event_tail": [ { - "elapsed_ms": 1029, + "elapsed_ms": 1026, "event": "thread.started" }, { - "elapsed_ms": 1107, + "elapsed_ms": 1078, "error_category": "hook_error", "event": "item.completed", "item": "error" }, { - "elapsed_ms": 1107, + "elapsed_ms": 1078, "error_category": "hook_error", "event": "item.completed", "item": "error" }, { - "elapsed_ms": 1107, + "elapsed_ms": 1078, "event": "turn.started" }, { - "elapsed_ms": 6392, + "elapsed_ms": 7254, "event": "item.completed", "item": "agent_message" }, { "command": "c09_first", - "elapsed_ms": 8437, + "elapsed_ms": 9921, "event": "item.started", "item": "command_execution", "status": "in_progress" @@ -119,7 +119,7 @@ { "c09_receipt_ok": true, "command": "c09_first", - "elapsed_ms": 8536, + "elapsed_ms": 10054, "event": "item.completed", "exit_code": 0, "item": "command_execution", @@ -127,7 +127,7 @@ }, { "command": "c09_second", - "elapsed_ms": 26509, + "elapsed_ms": 31751, "event": "item.started", "item": "command_execution", "status": "in_progress" @@ -135,7 +135,7 @@ { "c09_receipt_ok": true, "command": "c09_second", - "elapsed_ms": 26594, + "elapsed_ms": 31955, "event": "item.completed", "exit_code": 0, "item": "command_execution", @@ -143,7 +143,7 @@ }, { "command": "c09_finish", - "elapsed_ms": 46392, + "elapsed_ms": 55836, "event": "item.started", "item": "command_execution", "status": "in_progress" @@ -151,19 +151,19 @@ { "c09_receipt_ok": true, "command": "c09_finish", - "elapsed_ms": 46509, + "elapsed_ms": 55959, "event": "item.completed", "exit_code": 0, "item": "command_execution", "status": "completed" }, { - "elapsed_ms": 49858, + "elapsed_ms": 59236, "event": "item.completed", "item": "agent_message" }, { - "elapsed_ms": 49877, + "elapsed_ms": 59273, "event": "turn.completed" } ], @@ -184,61 +184,61 @@ "oversized_lines": 0, "owned_process_tree_terminated": false, "process_cleanup_ok": true, - "process_elapsed_ms": 51544, + "process_elapsed_ms": 60679, "process_returncode": 0, "reader_failed": false, "stderr_lines": 1, "timeout": false }, "git_after": { - "branch": "pursue/plan/PG-20260906-213357-3700/qualify-deterministic-compaction-and-recovery-be", + "branch": "pursue/plan/PG-20260907-161530-2B84/qualify-deterministic-compaction-and-recovery-be", "cached_paths": [], "commit_count": 2, - "head": "9b7b583543cba8272a062d02ba4035730d2eaf3f", - "index_tree": "a42e29caa8e41a4708954aa135a439fce6e941c6", + "head": "80a3d76ab8d5c2c84d926d49e1adb5472c8f4e34", + "index_tree": "c9ba440c99d1fb398572df7a696164bc9c2d78af", "refs": [ "refs/heads/master", - "refs/heads/pursue/plan/PG-20260906-213357-3700/qualify-deterministic-compaction-and-recovery-be" + "refs/heads/pursue/plan/PG-20260907-161530-2B84/qualify-deterministic-compaction-and-recovery-be" ], "status": [ "M .gitignore", "?? .pursue/SYSTEM_PROFILE.md", - "?? .pursue/runs/20260906T213358Z_PG-20260906-213357-3700_qualify-deterministic-compaction-and-recovery-be/PLAN.md", - "?? .pursue/runs/20260906T213358Z_PG-20260906-213357-3700_qualify-deterministic-compaction-and-recovery-be/checkpoints/CHECKPOINT-00-GENERATION_R0005.json", - "?? .pursue/runs/20260906T213358Z_PG-20260906-213357-3700_qualify-deterministic-compaction-and-recovery-be/compliance.json", - "?? .pursue/runs/20260906T213358Z_PG-20260906-213357-3700_qualify-deterministic-compaction-and-recovery-be/evidence/c09-scenario.json", - "?? .pursue/runs/20260906T213358Z_PG-20260906-213357-3700_qualify-deterministic-compaction-and-recovery-be/evidence/git-capability.json", - "?? .pursue/runs/20260906T213358Z_PG-20260906-213357-3700_qualify-deterministic-compaction-and-recovery-be/evidence/lifecycle.json", - "?? .pursue/runs/20260906T213358Z_PG-20260906-213357-3700_qualify-deterministic-compaction-and-recovery-be/evidence/original-goal.md", - "?? .pursue/runs/20260906T213358Z_PG-20260906-213357-3700_qualify-deterministic-compaction-and-recovery-be/manifest.json", - "?? .pursue/runs/20260906T213358Z_PG-20260906-213357-3700_qualify-deterministic-compaction-and-recovery-be/state.json", - "?? .pursue/runs/20260906T213358Z_PG-20260906-213357-3700_qualify-deterministic-compaction-and-recovery-be/traceability.json" + "?? .pursue/runs/20260907T161530Z_PG-20260907-161530-2B84_qualify-deterministic-compaction-and-recovery-be/PLAN.md", + "?? .pursue/runs/20260907T161530Z_PG-20260907-161530-2B84_qualify-deterministic-compaction-and-recovery-be/checkpoints/CHECKPOINT-00-GENERATION_R0005.json", + "?? .pursue/runs/20260907T161530Z_PG-20260907-161530-2B84_qualify-deterministic-compaction-and-recovery-be/compliance.json", + "?? .pursue/runs/20260907T161530Z_PG-20260907-161530-2B84_qualify-deterministic-compaction-and-recovery-be/evidence/c09-scenario.json", + "?? .pursue/runs/20260907T161530Z_PG-20260907-161530-2B84_qualify-deterministic-compaction-and-recovery-be/evidence/git-capability.json", + "?? .pursue/runs/20260907T161530Z_PG-20260907-161530-2B84_qualify-deterministic-compaction-and-recovery-be/evidence/lifecycle.json", + "?? .pursue/runs/20260907T161530Z_PG-20260907-161530-2B84_qualify-deterministic-compaction-and-recovery-be/evidence/original-goal.md", + "?? .pursue/runs/20260907T161530Z_PG-20260907-161530-2B84_qualify-deterministic-compaction-and-recovery-be/manifest.json", + "?? .pursue/runs/20260907T161530Z_PG-20260907-161530-2B84_qualify-deterministic-compaction-and-recovery-be/state.json", + "?? .pursue/runs/20260907T161530Z_PG-20260907-161530-2B84_qualify-deterministic-compaction-and-recovery-be/traceability.json" ], "worktree_count": 2 }, "git_before": { - "branch": "pursue/plan/PG-20260906-213357-3700/qualify-deterministic-compaction-and-recovery-be", + "branch": "pursue/plan/PG-20260907-161530-2B84/qualify-deterministic-compaction-and-recovery-be", "cached_paths": [], "commit_count": 2, - "head": "9b7b583543cba8272a062d02ba4035730d2eaf3f", - "index_tree": "a42e29caa8e41a4708954aa135a439fce6e941c6", + "head": "80a3d76ab8d5c2c84d926d49e1adb5472c8f4e34", + "index_tree": "c9ba440c99d1fb398572df7a696164bc9c2d78af", "refs": [ "refs/heads/master", - "refs/heads/pursue/plan/PG-20260906-213357-3700/qualify-deterministic-compaction-and-recovery-be" + "refs/heads/pursue/plan/PG-20260907-161530-2B84/qualify-deterministic-compaction-and-recovery-be" ], "status": [ "M .gitignore", "?? .pursue/SYSTEM_PROFILE.md", - "?? .pursue/runs/20260906T213358Z_PG-20260906-213357-3700_qualify-deterministic-compaction-and-recovery-be/PLAN.md", - "?? .pursue/runs/20260906T213358Z_PG-20260906-213357-3700_qualify-deterministic-compaction-and-recovery-be/checkpoints/CHECKPOINT-00-GENERATION_R0005.json", - "?? .pursue/runs/20260906T213358Z_PG-20260906-213357-3700_qualify-deterministic-compaction-and-recovery-be/compliance.json", - "?? .pursue/runs/20260906T213358Z_PG-20260906-213357-3700_qualify-deterministic-compaction-and-recovery-be/evidence/c09-scenario.json", - "?? .pursue/runs/20260906T213358Z_PG-20260906-213357-3700_qualify-deterministic-compaction-and-recovery-be/evidence/git-capability.json", - "?? .pursue/runs/20260906T213358Z_PG-20260906-213357-3700_qualify-deterministic-compaction-and-recovery-be/evidence/lifecycle.json", - "?? .pursue/runs/20260906T213358Z_PG-20260906-213357-3700_qualify-deterministic-compaction-and-recovery-be/evidence/original-goal.md", - "?? .pursue/runs/20260906T213358Z_PG-20260906-213357-3700_qualify-deterministic-compaction-and-recovery-be/manifest.json", - "?? .pursue/runs/20260906T213358Z_PG-20260906-213357-3700_qualify-deterministic-compaction-and-recovery-be/state.json", - "?? .pursue/runs/20260906T213358Z_PG-20260906-213357-3700_qualify-deterministic-compaction-and-recovery-be/traceability.json" + "?? .pursue/runs/20260907T161530Z_PG-20260907-161530-2B84_qualify-deterministic-compaction-and-recovery-be/PLAN.md", + "?? .pursue/runs/20260907T161530Z_PG-20260907-161530-2B84_qualify-deterministic-compaction-and-recovery-be/checkpoints/CHECKPOINT-00-GENERATION_R0005.json", + "?? .pursue/runs/20260907T161530Z_PG-20260907-161530-2B84_qualify-deterministic-compaction-and-recovery-be/compliance.json", + "?? .pursue/runs/20260907T161530Z_PG-20260907-161530-2B84_qualify-deterministic-compaction-and-recovery-be/evidence/c09-scenario.json", + "?? .pursue/runs/20260907T161530Z_PG-20260907-161530-2B84_qualify-deterministic-compaction-and-recovery-be/evidence/git-capability.json", + "?? .pursue/runs/20260907T161530Z_PG-20260907-161530-2B84_qualify-deterministic-compaction-and-recovery-be/evidence/lifecycle.json", + "?? .pursue/runs/20260907T161530Z_PG-20260907-161530-2B84_qualify-deterministic-compaction-and-recovery-be/evidence/original-goal.md", + "?? .pursue/runs/20260907T161530Z_PG-20260907-161530-2B84_qualify-deterministic-compaction-and-recovery-be/manifest.json", + "?? .pursue/runs/20260907T161530Z_PG-20260907-161530-2B84_qualify-deterministic-compaction-and-recovery-be/state.json", + "?? .pursue/runs/20260907T161530Z_PG-20260907-161530-2B84_qualify-deterministic-compaction-and-recovery-be/traceability.json" ], "worktree_count": 2 }, @@ -304,7 +304,7 @@ "assertions": [ { "evidence": "C09_FINISHED", - "name": "finite_recovery_completed", + "name": "finite_recovery", "status": "PASS" } ], diff --git a/capabilities/C09/hashes.json b/capabilities/C09/hashes.json index 49246e8..f1ca637 100644 --- a/capabilities/C09/hashes.json +++ b/capabilities/C09/hashes.json @@ -1,8 +1,8 @@ { "algorithm": "sha256", "files": { - "README.md": "d36e36088f85f40b914b6a588f6a03c3e0d77aca14a6434b18a9f8a8431f1e57", - "actual.sanitized.json": "9ec69c08079d97294592bec475f38e660526886277179e81667eba1b8fe537d8", + "README.md": "edc6e51181d44340b94e59dd1afd9bed78a604f92583180dc7ef5ac5b34f364a", + "actual.sanitized.json": "855456434611853465eca32cc5c4cf8a3c890a91c7f8a8914c3b0e45e0beeb47", "config/README.md": "9dfed81bb95428672754c1e3b74e95a2a712915d8d7bc3807663a276708b1d27", "evaluation.json": "a19a7aa2000dbd526c4bafd8f0fdca27dcf67fc24199c8607abae0949358fbe4", "expected.json": "5f004af3a4ac22453c495e36de23986a5d4c48292173b7604c72d27b5acb3c0a", diff --git a/capabilities/C10/README.md b/capabilities/C10/README.md index e1ecd31..1447f7d 100644 --- a/capabilities/C10/README.md +++ b/capabilities/C10/README.md @@ -12,11 +12,11 @@ The two acceptance assertions remain unchanged. Exact opaque echo, actual lifecy ## Live qualification -- Date: `2026-09-06` +- Date: `2026-09-07` - Codex: `codex-cli 0.153.4` - Model: `gpt-5.6-sol` - OS: `Debian GNU/Linux 13 (trixie)` - Permission mode: `approval=never; sandbox=per-trial; model-tool network disabled` - Project trust: `trusted via CLI override for disposable fixture repositories` -- Source commit: `d0384f76bc4150d33bb8f51ef5981f3243b3cfb3` +- Source commit: `a9cdcdc1e0cad70e88b60869e75e4166046dd306` - Result: `REPRODUCED` diff --git a/capabilities/C10/actual.sanitized.json b/capabilities/C10/actual.sanitized.json index 9747f61..978d494 100644 --- a/capabilities/C10/actual.sanitized.json +++ b/capabilities/C10/actual.sanitized.json @@ -2,12 +2,12 @@ "capability_id": "C10", "environment": { "codex_version": "codex-cli 0.153.4", - "fixture_commit": "175d5963baeaebe5f90d15a09aeec64c3c7b9781", + "fixture_commit": "7106efd129b1976c24df5128309622cc5dd0252b", "model": "gpt-5.6-sol", "os": "Debian GNU/Linux 13 (trixie)", "permission_mode": "approval=never; sandbox=per-trial; model-tool network disabled", "project_trust": "trusted via CLI override for disposable fixture repositories", - "source_commit": "d0384f76bc4150d33bb8f51ef5981f3243b3cfb3" + "source_commit": "a9cdcdc1e0cad70e88b60869e75e4166046dd306" }, "observations": [ "fixture_prepared_by_outer_harness=true", @@ -67,32 +67,32 @@ "error_events": 0, "event_tail": [ { - "elapsed_ms": 944, + "elapsed_ms": 1093, "event": "thread.started" }, { - "elapsed_ms": 1004, + "elapsed_ms": 1114, "error_category": "hook_error", "event": "item.completed", "item": "error" }, { - "elapsed_ms": 1004, + "elapsed_ms": 1114, "error_category": "hook_error", "event": "item.completed", "item": "error" }, { - "elapsed_ms": 1004, + "elapsed_ms": 1114, "event": "turn.started" }, { - "elapsed_ms": 6597, + "elapsed_ms": 9384, "event": "item.completed", "item": "agent_message" }, { - "elapsed_ms": 6619, + "elapsed_ms": 9476, "event": "turn.completed" } ], @@ -111,7 +111,7 @@ "oversized_lines": 0, "owned_process_tree_terminated": false, "process_cleanup_ok": true, - "process_elapsed_ms": 7912, + "process_elapsed_ms": 11392, "process_returncode": 0, "raw_payload_checks": { "candidate_count": 1, @@ -131,7 +131,7 @@ "timeout": false }, "model_payload_summary": { - "assertion_count": 0, + "assertion_count": 1, "capability_id": "C10", "observation_count": 1, "outcome": "PASS", @@ -269,52 +269,52 @@ "error_events": 0, "event_tail": [ { - "elapsed_ms": 1037, + "elapsed_ms": 1153, "event": "thread.started" }, { - "elapsed_ms": 1063, + "elapsed_ms": 1240, "error_category": "hook_error", "event": "item.completed", "item": "error" }, { - "elapsed_ms": 1063, + "elapsed_ms": 1240, "error_category": "hook_error", "event": "item.completed", "item": "error" }, { - "elapsed_ms": 1063, + "elapsed_ms": 1240, "event": "turn.started" }, { - "elapsed_ms": 5260, + "elapsed_ms": 7615, "event": "item.completed", "item": "agent_message" }, { "command": "segment_01", - "elapsed_ms": 6931, + "elapsed_ms": 10201, "event": "item.started", "item": "command_execution", "status": "in_progress" }, { "command": "segment_01", - "elapsed_ms": 6931, + "elapsed_ms": 10201, "event": "item.completed", "exit_code": 0, "item": "command_execution", "status": "completed" }, { - "elapsed_ms": 26019, + "elapsed_ms": 28774, "event": "item.completed", "item": "agent_message" }, { - "elapsed_ms": 26069, + "elapsed_ms": 28801, "event": "turn.completed" } ], @@ -335,7 +335,7 @@ "oversized_lines": 0, "owned_process_tree_terminated": false, "process_cleanup_ok": true, - "process_elapsed_ms": 28098, + "process_elapsed_ms": 30248, "process_returncode": 0, "raw_payload_checks": { "candidate_count": 1, @@ -354,7 +354,7 @@ "stderr_lines": 1, "timeout": false }, - "fixture_commit": "bef010e03824541ea388d3ef0b8c5e066742f8a8", + "fixture_commit": "053d8cdb268717c5fb66fe3f0d6bbf5a29508907", "model_payload_summary": { "assertion_count": 1, "capability_id": "C10", diff --git a/capabilities/C10/hashes.json b/capabilities/C10/hashes.json index d4fd032..a51d39c 100644 --- a/capabilities/C10/hashes.json +++ b/capabilities/C10/hashes.json @@ -1,8 +1,8 @@ { "algorithm": "sha256", "files": { - "README.md": "4239fc647385f1f9e3cea56f32f87414f8cf3db1c1ffebb92bc73a1711e0001e", - "actual.sanitized.json": "66113f872fe3427b0c16b7d18b4c8470a00d92d34a8ac8a2b7b780c593fe2f29", + "README.md": "c29660a66e8131925d5fb66cb28d7d4664f0ab6955a2099ff406d3c283710027", + "actual.sanitized.json": "0f9fd0a1d368332acb678e08787cb920d750647e5442c25382ecdade1044fb93", "config/README.md": "c061722e978e44397d855eb96054d0c8e516b354e3b4ab9aa79b050b6b1d4f4d", "evaluation.json": "43718c162c617e4521223447c84ada14062b76fce2f07da83634237d2802e1a6", "expected.json": "cee68b6991d33cbec2695ff47d84c6b2f5bf5634672dfab7d00bcdddcc43e52a", diff --git a/capabilities/C11/README.md b/capabilities/C11/README.md index 052aefc..bc35fb6 100644 --- a/capabilities/C11/README.md +++ b/capabilities/C11/README.md @@ -20,11 +20,11 @@ Use `fixture/`, `config/README.md`, `prompt.txt`, and `run-command.txt`. Replace ## Live qualification -- Date: `2026-09-06` +- Date: `2026-09-07` - Codex: `codex-cli 0.153.4` - Model: `gpt-5.6-sol` - OS: `Debian GNU/Linux 13 (trixie)` - Permission mode: `approval=never; sandbox=per-trial; model-tool network disabled` - Project trust: `trusted via CLI override for disposable fixture repositories` -- Source commit: `d0384f76bc4150d33bb8f51ef5981f3243b3cfb3` +- Source commit: `a9cdcdc1e0cad70e88b60869e75e4166046dd306` - Result: `REPRODUCED` diff --git a/capabilities/C11/actual.sanitized.json b/capabilities/C11/actual.sanitized.json index d90ddfc..f079629 100644 --- a/capabilities/C11/actual.sanitized.json +++ b/capabilities/C11/actual.sanitized.json @@ -2,18 +2,17 @@ "capability_id": "C11", "environment": { "codex_version": "codex-cli 0.153.4", - "fixture_commit": "afc1904ca4a7f42db30f25a9d7a0eaff92cec032", + "fixture_commit": "bb75a1c6f6302baa844c0fb78eb8d40e15b5da72", "model": "gpt-5.6-sol", "os": "Debian GNU/Linux 13 (trixie)", "permission_mode": "approval=never; sandbox=per-trial; model-tool network disabled", "project_trust": "trusted via CLI override for disposable fixture repositories", - "source_commit": "d0384f76bc4150d33bb8f51ef5981f3243b3cfb3" + "source_commit": "a9cdcdc1e0cad70e88b60869e75e4166046dd306" }, "observations": [ - "The completed trial showed both root and nested instruction markers active from the nested working directory.", - "The nested override marker was active, the sibling nested instruction marker was absent, and the effective conflict value was nested-wins.", - "The recorded precedence matched the supplied documentation baseline: root-to-working-directory merge, one selected instruction file per directory, and closer instructions taking precedence.", - "The instruction map selected the expected root and nested override files in order, recorded full reads, and included hashes for both applicable files." + "Root and nested instruction markers were both active, demonstrating root-to-CWD scope.", + "The nested conflict value won, and the lower-priority nested instruction was ignored when an override existed; this matches the recorded baseline.", + "The instruction map selected both applicable files in root-to-nested order, recorded full reads, and included a hash for each selected file." ], "result": "REPRODUCED", "schema_version": "1.0", @@ -22,22 +21,22 @@ "assertions": [ { "evidence": "Present in active project instructions.", - "name": "ROOT_SCOPE_MARKER active", + "name": "ROOT_SCOPE_MARKER", "status": "PASS" }, { "evidence": "Present in active project instructions.", - "name": "NESTED_OVERRIDE_MARKER active", + "name": "NESTED_OVERRIDE_MARKER", "status": "PASS" }, { - "evidence": "Not present in active project instructions.", - "name": "NESTED_SHOULD_BE_IGNORED absent", + "evidence": "Absent from active project instructions.", + "name": "NESTED_SHOULD_BE_IGNORED", "status": "PASS" }, { - "evidence": "Active project instructions specify CONFLICT_RULE=nested-wins after the root value.", - "name": "Effective conflicting value is nested-wins", + "evidence": "Active instruction states CONFLICT_RULE=nested-wins.", + "name": "effective_conflicting_value", "status": "PASS" } ], @@ -77,8 +76,8 @@ "branch": "master", "cached_paths": [], "commit_count": 2, - "head": "afc1904ca4a7f42db30f25a9d7a0eaff92cec032", - "index_tree": "a735a982cc55a9a70bec535494481be40c76eb33", + "head": "bb75a1c6f6302baa844c0fb78eb8d40e15b5da72", + "index_tree": "84005085100d1eddfdee96a268b27d5d2a0a5da3", "refs": [ "refs/heads/master" ], @@ -89,8 +88,8 @@ "branch": "master", "cached_paths": [], "commit_count": 2, - "head": "afc1904ca4a7f42db30f25a9d7a0eaff92cec032", - "index_tree": "a735a982cc55a9a70bec535494481be40c76eb33", + "head": "bb75a1c6f6302baa844c0fb78eb8d40e15b5da72", + "index_tree": "84005085100d1eddfdee96a268b27d5d2a0a5da3", "refs": [ "refs/heads/master" ], @@ -98,7 +97,7 @@ "worktree_count": 1 }, "observations": [ - "Evaluated runtime-provided active project instructions from the nested working directory without opening AGENTS files or invoking tools." + "Both root and nested markers are active; the nested conflict value takes precedence." ], "outcome": "PASS", "outer_instruction_map": { diff --git a/capabilities/C11/evaluation.json b/capabilities/C11/evaluation.json index 88ff9fb..0896d38 100644 --- a/capabilities/C11/evaluation.json +++ b/capabilities/C11/evaluation.json @@ -3,5 +3,5 @@ "expected_met": true, "result": "REPRODUCED", "schema_version": "1.0", - "summary": "The sanitized completed trial demonstrates all three expected assertions for project instruction scope, nested precedence, and complete mapped-file reads/hashes." + "summary": "The completed trial directly demonstrated all expected scope, precedence, and complete instruction-file mapping assertions." } diff --git a/capabilities/C11/hashes.json b/capabilities/C11/hashes.json index 8aaa7bc..841294e 100644 --- a/capabilities/C11/hashes.json +++ b/capabilities/C11/hashes.json @@ -1,10 +1,10 @@ { "algorithm": "sha256", "files": { - "README.md": "6297da96c440fc7bfc0b930161de82793bbf4f49a1c0f1a1224f902ea4b1609f", - "actual.sanitized.json": "30b04cae34620d50760008cb9e978cdfb2dd4dffdd3c23a603a1606816cdb83e", + "README.md": "63c49485df61b26faa1aeb623dd871c2cd8f91735b65388cd2fbcf2ecc68429f", + "actual.sanitized.json": "785f9e6a4ad8c112319512a5d20cf21673e96f52108de519d92b1fd201099920", "config/README.md": "9821e82809e4b89e1b97d79a8ea39bc3a49a095966bfd61117fc7027d00689d5", - "evaluation.json": "4858183858f706177b1b88dffbf5f1cf6404c64bc7c4bf2d144989f502f5c8b7", + "evaluation.json": "c8823a78ac54cfe126c99f913a14445ebbb80ac6d094def944e9dab62010466e", "expected.json": "7860035e5e46e13d9f45eb67f3113544b00031665e039895fccc41ae281db8a8", "fixture/AGENTS.md": "b1008628626c749a1cedb6aaa45a2c341a8f3f94dcd61aad8ede952c8815fc90", "fixture/README.md": "a43c9ab0fe2054f1e38867560547eaf17591d0a49e4358139fcd243d8e57cbf0", diff --git a/capabilities/C12/README.md b/capabilities/C12/README.md index a6736e1..a045b23 100644 --- a/capabilities/C12/README.md +++ b/capabilities/C12/README.md @@ -20,11 +20,11 @@ Use `fixture/`, `config/README.md`, `prompt.txt`, and `run-command.txt`. Replace ## Live qualification -- Date: `2026-09-06` +- Date: `2026-09-07` - Codex: `codex-cli 0.153.4` - Model: `gpt-5.6-sol` - OS: `Debian GNU/Linux 13 (trixie)` - Permission mode: `approval=never; sandbox=per-trial; model-tool network disabled` - Project trust: `trusted via CLI override for disposable fixture repositories` -- Source commit: `d0384f76bc4150d33bb8f51ef5981f3243b3cfb3` +- Source commit: `a9cdcdc1e0cad70e88b60869e75e4166046dd306` - Result: `REPRODUCED` diff --git a/capabilities/C12/actual.sanitized.json b/capabilities/C12/actual.sanitized.json index ae930d5..e0c3ff3 100644 --- a/capabilities/C12/actual.sanitized.json +++ b/capabilities/C12/actual.sanitized.json @@ -2,12 +2,12 @@ "capability_id": "C12", "environment": { "codex_version": "codex-cli 0.153.4", - "fixture_commit": "76348a12b0b01541ea5cc3ff7a5fb8942fd98fca", + "fixture_commit": "943327f5c2dbb6f5b85866de65974c840107d205", "model": "gpt-5.6-sol", "os": "Debian GNU/Linux 13 (trixie)", "permission_mode": "approval=never; sandbox=per-trial; model-tool network disabled", "project_trust": "trusted via CLI override for disposable fixture repositories", - "source_commit": "d0384f76bc4150d33bb8f51ef5981f3243b3cfb3" + "source_commit": "a9cdcdc1e0cad70e88b60869e75e4166046dd306" }, "observations": [ "automatic_outcome=PASS", @@ -53,8 +53,8 @@ "branch": "master", "cached_paths": [], "commit_count": 2, - "head": "76348a12b0b01541ea5cc3ff7a5fb8942fd98fca", - "index_tree": "09e441d25e916ca414a56e81fb918d21b92d4cbf", + "head": "943327f5c2dbb6f5b85866de65974c840107d205", + "index_tree": "d2c9828045a94cccc8f95194232c838165af8cfb", "refs": [ "refs/heads/master" ], @@ -65,8 +65,8 @@ "branch": "master", "cached_paths": [], "commit_count": 2, - "head": "76348a12b0b01541ea5cc3ff7a5fb8942fd98fca", - "index_tree": "09e441d25e916ca414a56e81fb918d21b92d4cbf", + "head": "943327f5c2dbb6f5b85866de65974c840107d205", + "index_tree": "d2c9828045a94cccc8f95194232c838165af8cfb", "refs": [ "refs/heads/master" ], @@ -101,9 +101,9 @@ "execution": "outer deterministic PlanAnvil map_instructions", "observations": [ "fixture_bytes=4320", - "expected_sha256=sha256:e4915518d8e2e19cd371a38d72cd508976b2df05463513c3de4645a8cbb0b7ff", + "expected_sha256=sha256:79bb8900eb43c5cc8cee71fd0803553cadcdff2ab8d69b3ebd4da77f0cfea29c", "map_error=none", - "{\"automatic_byte_limit\": 1024, \"bytes\": 4320, \"full_read\": true, \"path\": \"AGENTS.md\", \"sha256\": \"sha256:e4915518d8e2e19cd371a38d72cd508976b2df05463513c3de4645a8cbb0b7ff\", \"truncation_risk\": true}" + "{\"automatic_byte_limit\": 1024, \"bytes\": 4320, \"full_read\": true, \"path\": \"AGENTS.md\", \"sha256\": \"sha256:79bb8900eb43c5cc8cee71fd0803553cadcdff2ab8d69b3ebd4da77f0cfea29c\", \"truncation_risk\": true}" ], "outcome": "PASS", "trial": "plananvil_explicit_full_file_hash", diff --git a/capabilities/C12/hashes.json b/capabilities/C12/hashes.json index 6001ce7..2047ca9 100644 --- a/capabilities/C12/hashes.json +++ b/capabilities/C12/hashes.json @@ -1,8 +1,8 @@ { "algorithm": "sha256", "files": { - "README.md": "170f980e84506d659dee2c722826937a60cad4332d96d4031588d9f76b059e04", - "actual.sanitized.json": "21106b6cbd5e62e0a5bcea6b525d28399542fd4e31a1f0b4fc892a62a2df8dce", + "README.md": "16560d4c1e59748514482f0213ff8528163a61b2d9871a803dda2b0bc6694276", + "actual.sanitized.json": "774f9c9ae5286f110408ddcafdfefeab2be5afc9939d170b515613b2aa152a8a", "config/README.md": "af665c6549f9f126f30a50b9b522363e9d51bc535afac28f0a5d1dd14db1ea62", "evaluation.json": "c8869d4db670db69f3b65b32cd83b551f2f95c963d3552defa7cef0c213a303e", "expected.json": "4c439c4a37de0e0cdf003b05d628017437914693c679f5c6ad7ed1485f92c811", diff --git a/capabilities/C13/README.md b/capabilities/C13/README.md index 23fadd8..a0bb50f 100644 --- a/capabilities/C13/README.md +++ b/capabilities/C13/README.md @@ -28,11 +28,11 @@ Record the exact Codex version, model slug, OS, permission mode, persisted proje ## Live qualification -- Date: `2026-09-06` +- Date: `2026-09-07` - Codex: `codex-cli 0.153.4` - Model: `gpt-5.6-sol` - OS: `Debian GNU/Linux 13 (trixie)` - Permission mode: `approval=never; sandbox=per-trial; model-tool network disabled` - Project trust: `trusted via CLI override for disposable fixture repositories` -- Source commit: `d0384f76bc4150d33bb8f51ef5981f3243b3cfb3` +- Source commit: `a9cdcdc1e0cad70e88b60869e75e4166046dd306` - Result: `REPRODUCED` diff --git a/capabilities/C13/actual.sanitized.json b/capabilities/C13/actual.sanitized.json index 903a62c..d558b6d 100644 --- a/capabilities/C13/actual.sanitized.json +++ b/capabilities/C13/actual.sanitized.json @@ -2,12 +2,12 @@ "capability_id": "C13", "environment": { "codex_version": "codex-cli 0.153.4", - "fixture_commit": "65db081b125fa69f483a4ddb285ed73f41f7c0b9", + "fixture_commit": "f144f9cfb6f67c4a1ba4e0bbf8b9a078c7b1401f", "model": "gpt-5.6-sol", "os": "Debian GNU/Linux 13 (trixie)", "permission_mode": "approval=never; sandbox=per-trial; model-tool network disabled", "project_trust": "trusted via CLI override for disposable fixture repositories", - "source_commit": "d0384f76bc4150d33bb8f51ef5981f3243b3cfb3" + "source_commit": "a9cdcdc1e0cad70e88b60869e75e4166046dd306" }, "observations": [ "ephemeral_outcome=BLOCKED", @@ -69,8 +69,8 @@ "branch": "master", "cached_paths": [], "commit_count": 2, - "head": "526eb1c88f9bc8988e0b97f45d5263bbf03c524e", - "index_tree": "28d00f3956074a6cd29d131d5fcfaabe904567a4", + "head": "3a2fc75cab0e2e0d553223747d26ebbfcac1e64c", + "index_tree": "8feeea53689941b36e1968780d5969d9c2be7b0f", "refs": [ "refs/heads/master" ], @@ -81,8 +81,8 @@ "branch": "master", "cached_paths": [], "commit_count": 2, - "head": "526eb1c88f9bc8988e0b97f45d5263bbf03c524e", - "index_tree": "28d00f3956074a6cd29d131d5fcfaabe904567a4", + "head": "3a2fc75cab0e2e0d553223747d26ebbfcac1e64c", + "index_tree": "8feeea53689941b36e1968780d5969d9c2be7b0f", "refs": [ "refs/heads/master" ], @@ -146,25 +146,25 @@ "completed_file_change_items": 0, "error_events": 0, "event_types": { - "item.completed": 6, + "item.completed": 5, "item.started": 1, "thread.started": 1, "turn.completed": 1, "turn.started": 1 }, "item_types": { - "agent_message": 3, + "agent_message": 2, "collab_tool_call": 2, "error": 2 } }, - "fallback_fixture_commit": "65db081b125fa69f483a4ddb285ed73f41f7c0b9", + "fallback_fixture_commit": "f144f9cfb6f67c4a1ba4e0bbf8b9a078c7b1401f", "git_after": { "branch": "master", "cached_paths": [], "commit_count": 2, - "head": "65db081b125fa69f483a4ddb285ed73f41f7c0b9", - "index_tree": "28d00f3956074a6cd29d131d5fcfaabe904567a4", + "head": "f144f9cfb6f67c4a1ba4e0bbf8b9a078c7b1401f", + "index_tree": "8feeea53689941b36e1968780d5969d9c2be7b0f", "refs": [ "refs/heads/master" ], @@ -175,8 +175,8 @@ "branch": "master", "cached_paths": [], "commit_count": 2, - "head": "65db081b125fa69f483a4ddb285ed73f41f7c0b9", - "index_tree": "28d00f3956074a6cd29d131d5fcfaabe904567a4", + "head": "f144f9cfb6f67c4a1ba4e0bbf8b9a078c7b1401f", + "index_tree": "8feeea53689941b36e1968780d5969d9c2be7b0f", "refs": [ "refs/heads/master" ], diff --git a/capabilities/C13/hashes.json b/capabilities/C13/hashes.json index 226bad8..fb11efb 100644 --- a/capabilities/C13/hashes.json +++ b/capabilities/C13/hashes.json @@ -1,8 +1,8 @@ { "algorithm": "sha256", "files": { - "README.md": "fc125afb7fcf1c6ac1c4223c7eeb0a1e9877e68d086782987a5c263a29bcfc7f", - "actual.sanitized.json": "ce48064b618f973f92ed2c9c4fadc4641ea188d29b0eaf9a760ac358c556af20", + "README.md": "510a3547f6d9e3d61a472be34c868cce210ceeeb3a10939370c5ac78f79130e0", + "actual.sanitized.json": "a1cdc6f9884c3727314a8a745bd898efe8fc70d01ac993bed049d3094d141590", "config/README.md": "9e90b31ef6002415db8caa4dfa544b1172a7f52ed04f27e9182c420a8bb6272c", "evaluation.json": "19febe046d2e4aee5a6764d4bb77bc520928fccc1e55fcab87b93aff036eb9b5", "expected.json": "3ac2f1bd7ac5fc1bdf9a4488192d3faa0a57559882c1dcfb589eaef37b198c3a", diff --git a/capabilities/C14/README.md b/capabilities/C14/README.md index 1729a88..d9b2ca2 100644 --- a/capabilities/C14/README.md +++ b/capabilities/C14/README.md @@ -20,11 +20,11 @@ Use `fixture/`, `config/README.md`, `prompt.txt`, and `run-command.txt`. Replace ## Live qualification -- Date: `2026-09-06` +- Date: `2026-09-07` - Codex: `codex-cli 0.153.4` - Model: `gpt-5.6-sol` - OS: `Debian GNU/Linux 13 (trixie)` - Permission mode: `approval=never; sandbox=per-trial; model-tool network disabled` - Project trust: `trusted via CLI override for disposable fixture repositories` -- Source commit: `d0384f76bc4150d33bb8f51ef5981f3243b3cfb3` +- Source commit: `a9cdcdc1e0cad70e88b60869e75e4166046dd306` - Result: `REPRODUCED` diff --git a/capabilities/C14/actual.sanitized.json b/capabilities/C14/actual.sanitized.json index 82a6035..1a639ff 100644 --- a/capabilities/C14/actual.sanitized.json +++ b/capabilities/C14/actual.sanitized.json @@ -2,18 +2,18 @@ "capability_id": "C14", "environment": { "codex_version": "codex-cli 0.153.4", - "fixture_commit": "88cb2bbb6323953441802f4ecb9ce59882f93371", + "fixture_commit": "d6f3d12094858637bebf0a8bca23c2f552faf139", "model": "gpt-5.6-sol", "os": "Debian GNU/Linux 13 (trixie)", "permission_mode": "approval=never; sandbox=per-trial; model-tool network disabled", "project_trust": "trusted via CLI override for disposable fixture repositories", - "source_commit": "d0384f76bc4150d33bb8f51ef5981f3243b3cfb3" + "source_commit": "a9cdcdc1e0cad70e88b60869e75e4166046dd306" }, "observations": [ - "The completed trial recorded the source branch and HEAD as identical before and after.", - "The source index tree, status, cached paths, and tracked source-file hash were unchanged.", - "A distinct linked planning worktree and branch were created; planning artifacts and changes were present there while the source remained clean and unchanged.", - "All three expected assertions received direct PASS evidence with no reported trial errors or blocker." + "Source branch remained main and HEAD was identical before and after.", + "Source index tree was identical before and after; status and cached paths remained empty.", + "A distinct planning branch and linked worktree were created, with planning changes present there and absent from the source worktree.", + "The completed trial reported all isolation checks passing with no blocker." ], "result": "REPRODUCED", "schema_version": "1.0", @@ -21,17 +21,17 @@ { "assertions": [ { - "evidence": "Pre/post comparisons: branch=true and HEAD=true.", + "evidence": "Source remained on main at HEAD d6f3d12094858637bebf0a8bca23c2f552faf139.", "name": "Source branch and HEAD are unchanged.", "status": "PASS" }, { - "evidence": "Pre/post comparisons: index tree=true, status hash=true, source.txt hash=true.", + "evidence": "Index tree remained cfdbf1b513a55619f2d834d40624867a8eea6c96; status remained clean; source.txt SHA-256 remained 496688b86291a24cc50c23fe4a5ef0874f5c78938d9ace7714c9ae30124fb60e.", "name": "Source index/status hashes are unchanged.", "status": "PASS" }, { - "evidence": "Controller returned ok=true/PROFILE_READY; distinct planning branch=true; requested linked worktree=true; branch match=true; plan and scaffold present=true; planning changes present=true while source remained unchanged.", + "evidence": "Controller returned PROFILE_READY. The registered linked worktree exists at ../worktrees/planning on distinct branch pursue/plan/PG-20260907-161916-D637/qualify-planning-isolation; .pursue scaffold and generated .gitignore exist there and not in source.", "name": "Planning changes exist only in the isolated planning worktree/branch.", "status": "PASS" } @@ -39,19 +39,19 @@ "blocker": null, "capability_id": "C14", "event_summary": { - "completed_command_items": 3, + "completed_command_items": 4, "completed_file_change_items": 0, "error_events": 0, "event_types": { - "item.completed": 9, - "item.started": 3, + "item.completed": 10, + "item.started": 4, "thread.started": 1, "turn.completed": 1, "turn.started": 1 }, "item_types": { "agent_message": 4, - "command_execution": 6, + "command_execution": 8, "error": 2 } }, @@ -59,11 +59,11 @@ "branch": "main", "cached_paths": [], "commit_count": 2, - "head": "88cb2bbb6323953441802f4ecb9ce59882f93371", - "index_tree": "e7827d168ec97a90dc89c96688c100ba8df6a2f6", + "head": "d6f3d12094858637bebf0a8bca23c2f552faf139", + "index_tree": "cfdbf1b513a55619f2d834d40624867a8eea6c96", "refs": [ "refs/heads/main", - "refs/heads/pursue/plan/PG-20260906-213658-6642/qualify-planning-isolation" + "refs/heads/pursue/plan/PG-20260907-161916-D637/qualify-planning-isolation" ], "status": [], "worktree_count": 2 @@ -72,8 +72,8 @@ "branch": "main", "cached_paths": [], "commit_count": 2, - "head": "88cb2bbb6323953441802f4ecb9ce59882f93371", - "index_tree": "e7827d168ec97a90dc89c96688c100ba8df6a2f6", + "head": "d6f3d12094858637bebf0a8bca23c2f552faf139", + "index_tree": "cfdbf1b513a55619f2d834d40624867a8eea6c96", "refs": [ "refs/heads/main" ], @@ -81,9 +81,9 @@ "worktree_count": 1 }, "observations": [ - "The real deterministic controller completed successfully.", - "The reported planning worktree exists at ../worktrees/planning on a distinct linked branch.", - "outer_source_head=88cb2bbb6323953441802f4ecb9ce59882f93371", + "The real deterministic controller exited 0 with ok=true and implementation_executed=false.", + "All post-run isolation checks passed.", + "outer_source_head=d6f3d12094858637bebf0a8bca23c2f552faf139", "outer_source_branch=main", "outer_planning_worktree_exists=true", "outer_planning_branch_distinct=true", diff --git a/capabilities/C14/evaluation.json b/capabilities/C14/evaluation.json index 198da33..21aecb9 100644 --- a/capabilities/C14/evaluation.json +++ b/capabilities/C14/evaluation.json @@ -3,5 +3,5 @@ "expected_met": true, "result": "REPRODUCED", "schema_version": "1.0", - "summary": "The sanitized trial evidence demonstrates source immutability and confines planning changes to the isolated planning worktree and branch." + "summary": "The sanitized completed trial demonstrates all expected source-immutability and planning-isolation assertions." } diff --git a/capabilities/C14/hashes.json b/capabilities/C14/hashes.json index fd4457f..b79f54a 100644 --- a/capabilities/C14/hashes.json +++ b/capabilities/C14/hashes.json @@ -1,10 +1,10 @@ { "algorithm": "sha256", "files": { - "README.md": "8ad19a5d6ed996ac1c0f7cc51463c6ed7e8211ba5cec006373712a0cf2cc83ca", - "actual.sanitized.json": "c270644753208475857a47399189af467b07dcea471c30f87bf94547018fd162", + "README.md": "607ca05a47eabcd545a8589e2ce3ed1cd33586eb879e61f034bb58a14fd66372", + "actual.sanitized.json": "f372ca2afda0bd0ab6ccc1abd036beed3130b3f9048af07619d80ccf1c31b193", "config/README.md": "f4a1b9caa06a34fe8423474de42b5c40e9d15c78832bdc77e185f444c0d8a38d", - "evaluation.json": "dd6d071e88070be068a258b66fab6573a05b623615b21dca3524f5a0857b8af9", + "evaluation.json": "6c25869ae8977b88ac6b339e48c25b14a6a2585f66d2d6036d45884493ef6e4a", "expected.json": "ab00a41be3f7cf3df797b26c822d262d1ca368686a492e57eb25cdf2b1bcd029", "fixture/README.md": "c14da481c5feea9cf2c6a959ff2e2da2acefb7b7d0b5441d683115efab0a7a4f", "fixture/source.txt": "46dee92c6d90b92f5b5be4e4bb8c2370038ba17f600a9a941428428d775727f5", diff --git a/capabilities/C15/README.md b/capabilities/C15/README.md index 70a9767..87a2c9c 100644 --- a/capabilities/C15/README.md +++ b/capabilities/C15/README.md @@ -20,11 +20,11 @@ Use `fixture/`, `config/README.md`, `prompt.txt`, and `run-command.txt`. Replace ## Live qualification -- Date: `2026-09-06` +- Date: `2026-09-07` - Codex: `codex-cli 0.153.4` - Model: `gpt-5.6-sol` - OS: `Debian GNU/Linux 13 (trixie)` - Permission mode: `approval=never; sandbox=per-trial; model-tool network disabled` - Project trust: `trusted via CLI override for disposable fixture repositories` -- Source commit: `d0384f76bc4150d33bb8f51ef5981f3243b3cfb3` +- Source commit: `a9cdcdc1e0cad70e88b60869e75e4166046dd306` - Result: `REPRODUCED` diff --git a/capabilities/C15/actual.sanitized.json b/capabilities/C15/actual.sanitized.json index c17fcf5..efd66d0 100644 --- a/capabilities/C15/actual.sanitized.json +++ b/capabilities/C15/actual.sanitized.json @@ -2,17 +2,18 @@ "capability_id": "C15", "environment": { "codex_version": "codex-cli 0.153.4", - "fixture_commit": "df27abcf1df0466dc7d35972a6a1a5377e28e144", + "fixture_commit": "8f19a7c1b7a00f8caaa27f4df789773ab52cd981", "model": "gpt-5.6-sol", "os": "Debian GNU/Linux 13 (trixie)", "permission_mode": "approval=never; sandbox=per-trial; model-tool network disabled", "project_trust": "trusted via CLI override for disposable fixture repositories", - "source_commit": "d0384f76bc4150d33bb8f51ef5981f3243b3cfb3" + "source_commit": "a9cdcdc1e0cad70e88b60869e75e4166046dd306" }, "observations": [ - "Completed trial produced FAIL readiness with one CRITICAL coverage/traceability finding and HIGH rollback/recovery omissions.", - "Post-comparison checks report both blind-review Markdown and sidecar hashes unchanged, with their binding still valid.", - "The sidecar identifies a reviewer role, and the review was performed by a fresh read-only sub-agent separate from the controller/planner role." + "Completed trial produced a FAIL verdict with two CRITICAL findings: missing rollback and an uncovered critical requirement.", + "Post-comparison checks confirmed that the blind-review markdown and sidecar hashes were unchanged.", + "The review sidecar identified an independent blind-reviewer role distinct from the controller/planner role.", + "The trial reported no blocker and an overall PASS outcome." ], "result": "REPRODUCED", "schema_version": "1.0", @@ -20,17 +21,17 @@ { "assertions": [ { - "evidence": "Fresh reviewer returned FAIL with one CRITICAL coverage/traceability finding and HIGH rollback/recovery omissions.", + "evidence": "Fresh blind reviewer returned FAIL with two CRITICAL findings: missing rollback and uncovered critical requirement.", "name": "Seeded critical defects are reported or deterministically block readiness.", "status": "PASS" }, { - "evidence": "Post-comparison checks confirmed both review artifact hashes unchanged; Markdown-to-sidecar binding also remained valid.", + "evidence": "Post-comparison checks confirmed both recorded review artifact hashes remained unchanged.", "name": "Blind review markdown/sidecar hashes do not change during comparison.", "status": "PASS" }, { - "evidence": "Review sidecar recorded author_role=plan-anvil-reviewer; review was performed by a fresh read-only sub-agent separate from the controller.", + "evidence": "Review sidecar records an independently named blind reviewer role distinct from the controller/planner role.", "name": "Reviewer identity is independent from the planner role.", "status": "PASS" } @@ -38,12 +39,12 @@ "blocker": null, "capability_id": "C15", "event_summary": { - "completed_command_items": 9, - "completed_file_change_items": 3, + "completed_command_items": 10, + "completed_file_change_items": 4, "error_events": 0, "event_types": { - "item.completed": 19, - "item.started": 13, + "item.completed": 21, + "item.started": 15, "thread.started": 1, "turn.completed": 1, "turn.started": 1 @@ -51,24 +52,26 @@ "item_types": { "agent_message": 4, "collab_tool_call": 2, - "command_execution": 18, + "command_execution": 20, "error": 2, - "file_change": 6 + "file_change": 8 } }, "git_after": { "branch": "master", "cached_paths": [], "commit_count": 2, - "head": "df27abcf1df0466dc7d35972a6a1a5377e28e144", - "index_tree": "f32840c1e4bdc04ae8cdb96860185b675fc0541a", + "head": "8f19a7c1b7a00f8caaa27f4df789773ab52cd981", + "index_tree": "9adb9b499e76efb78451552983cda34c72c48f85", "refs": [ "refs/heads/master" ], "status": [ - "?? reports/plan-review/blind-review.json", - "?? reports/plan-review/blind-review.md", - "?? reports/plan-review/comparison.json" + "?? blind-review.json", + "?? blind-review.md", + "?? comparison.json", + "?? review-bundle.json", + "?? review-bundle.sha256" ], "worktree_count": 1 }, @@ -76,8 +79,8 @@ "branch": "master", "cached_paths": [], "commit_count": 2, - "head": "df27abcf1df0466dc7d35972a6a1a5377e28e144", - "index_tree": "f32840c1e4bdc04ae8cdb96860185b675fc0541a", + "head": "8f19a7c1b7a00f8caaa27f4df789773ab52cd981", + "index_tree": "9adb9b499e76efb78451552983cda34c72c48f85", "refs": [ "refs/heads/master" ], @@ -85,17 +88,15 @@ "worktree_count": 1 }, "observations": [ - "Blind result: FAIL.", - "Comparison result: FAIL.", - "Critical findings: 1.", - "Markdown hash unchanged: true.", - "Sidecar hash unchanged: true.", - "Independent reviewer role: true." + "Bundle digest verification succeeded.", + "Immutable review recorded verdict FAIL with two critical findings.", + "Separate comparison confirmed unchanged markdown and sidecar hashes.", + "Original seeded fixture remained unchanged." ], "outcome": "PASS", "sandbox": "workspace-write", - "trial": "immutable-blind-review", - "trial_name": "immutable-blind-review" + "trial": "immutable_blind_review", + "trial_name": "immutable_blind_review" }, { "evaluator_event_summary": { diff --git a/capabilities/C15/evaluation.json b/capabilities/C15/evaluation.json index 6b7a5ad..35c42dd 100644 --- a/capabilities/C15/evaluation.json +++ b/capabilities/C15/evaluation.json @@ -3,5 +3,5 @@ "expected_met": true, "result": "REPRODUCED", "schema_version": "1.0", - "summary": "The completed sanitized trial demonstrates all three required assertions: seeded critical defects blocked readiness, review artifacts remained immutable during comparison, and reviewer identity was independent of planning." + "summary": "The completed sanitized trial demonstrates all three expected assertions." } diff --git a/capabilities/C15/hashes.json b/capabilities/C15/hashes.json index 48ce530..b619a12 100644 --- a/capabilities/C15/hashes.json +++ b/capabilities/C15/hashes.json @@ -1,10 +1,10 @@ { "algorithm": "sha256", "files": { - "README.md": "1f456bba4ca191befe4b6e307cab4b3a77db07084fe193e7485ac8aed238e13d", - "actual.sanitized.json": "8577c880ca8937ca34778d59a906da5d07a02bd4d5e313739b3f51e8465857bc", + "README.md": "12a19acf1b6c9cbb9404fa1d586aee9edec9bf32f7b36ca637ef1ff8c90bafeb", + "actual.sanitized.json": "03f2c625358d2d823f2c264979bfaaf93a52818eb5f7d6ec74baf4bb66bdbec4", "config/README.md": "bebf0302010890c9db73a1e52855c635497168dcf8bc8016186be0f8e6345018", - "evaluation.json": "f3271baa986f259a26c3e0dc15ea62466557c2af1c8472732710578906530e9a", + "evaluation.json": "81ae05c91c8feb3ff26dcd29bc51624b4f73057430b58db96223bdcf867b748f", "expected.json": "9f35c7c55cc79f84a19f3bce9e3d978fb7e86414330f5e21b55fe1dcaf7cad54", "fixture/README.md": "a1836296b9646450b5f231a0db055c11c19a6675fdc8beaa4c5f824fd2730d87", "fixture/seeded-plan.md": "958041a2986063da97941c6078cf9764fdf5183dfd29eb55bf1dc37d526135c5", diff --git a/capabilities/C16/README.md b/capabilities/C16/README.md index 8db4d4f..4ceb12a 100644 --- a/capabilities/C16/README.md +++ b/capabilities/C16/README.md @@ -20,11 +20,11 @@ Use `fixture/`, `config/README.md`, `prompt.txt`, and `run-command.txt`. Replace ## Live qualification -- Date: `2026-09-06` +- Date: `2026-09-07` - Codex: `codex-cli 0.153.4` - Model: `gpt-5.6-sol` - OS: `Debian GNU/Linux 13 (trixie)` - Permission mode: `approval=never; sandbox=per-trial; model-tool network disabled` - Project trust: `trusted via CLI override for disposable fixture repositories` -- Source commit: `d0384f76bc4150d33bb8f51ef5981f3243b3cfb3` +- Source commit: `a9cdcdc1e0cad70e88b60869e75e4166046dd306` - Result: `REPRODUCED` diff --git a/capabilities/C16/actual.sanitized.json b/capabilities/C16/actual.sanitized.json index 58ec0fe..7232c26 100644 --- a/capabilities/C16/actual.sanitized.json +++ b/capabilities/C16/actual.sanitized.json @@ -2,12 +2,12 @@ "capability_id": "C16", "environment": { "codex_version": "codex-cli 0.153.4", - "fixture_commit": "0e5fdbad2f62c221e4fe3c5c1b53a6e645eb28cd", + "fixture_commit": "c6d09bae00cf0a7deefabc1b6a3017e3bd05de3e", "model": "gpt-5.6-sol", "os": "Debian GNU/Linux 13 (trixie)", "permission_mode": "approval=never; sandbox=per-trial; model-tool network disabled", "project_trust": "trusted via CLI override for disposable fixture repositories", - "source_commit": "d0384f76bc4150d33bb8f51ef5981f3243b3cfb3" + "source_commit": "a9cdcdc1e0cad70e88b60869e75e4166046dd306" }, "observations": [ "outer_all_pass=true", @@ -20,22 +20,22 @@ { "assertions": [ { - "evidence": "Probe completed with ok=true, result=GIT_READY, and all reported steps succeeded.", + "evidence": "Live probe exited 0 with result=GIT_READY; every reported Git operation had ok=true.", "name": "Probe accurately reports each Git capability outcome.", "status": "PASS" }, { - "evidence": "Independent checks found no temporary refs, branches, worktrees, probe files, or entries under ../probes.", + "evidence": "No temporary probe refs, branches, worktrees, files, or entries under the probe parent remained.", "name": "Temporary refs/branches/worktrees/files are cleaned.", "status": "PASS" }, { - "evidence": "source_snapshot_changed=[]; source remained on main at the original HEAD with a clean status.", + "evidence": "source_snapshot_changed=[] and the source worktree remained clean on main at the original HEAD.", "name": "Source repository snapshot is preserved.", "status": "PASS" }, { - "evidence": "The real commit operation succeeded; no signing or repository-hook failure diagnostics occurred.", + "evidence": "The real probe completed without signing or repository-hook failures requiring classification.", "name": "Signing and repository-hook failures are classified from actual diagnostics.", "status": "PASS" } @@ -63,8 +63,8 @@ "branch": "main", "cached_paths": [], "commit_count": 2, - "head": "0e5fdbad2f62c221e4fe3c5c1b53a6e645eb28cd", - "index_tree": "d0f99c956fcb88e978566d60c763b86076e3226a", + "head": "c6d09bae00cf0a7deefabc1b6a3017e3bd05de3e", + "index_tree": "d6e50af2eacc26740687f9f8f7edc4c31501e6aa", "refs": [ "refs/heads/main" ], @@ -75,8 +75,8 @@ "branch": "main", "cached_paths": [], "commit_count": 2, - "head": "0e5fdbad2f62c221e4fe3c5c1b53a6e645eb28cd", - "index_tree": "d0f99c956fcb88e978566d60c763b86076e3226a", + "head": "c6d09bae00cf0a7deefabc1b6a3017e3bd05de3e", + "index_tree": "d6e50af2eacc26740687f9f8f7edc4c31501e6aa", "refs": [ "refs/heads/main" ], @@ -84,8 +84,9 @@ "worktree_count": 1 }, "observations": [ - "Command exited 0.", - "cleanup_errors=[] and no probe artifacts remained." + "cleanup_errors=[]", + "source_snapshot_changed=[]", + "No probe artifacts remained." ], "outcome": "PASS", "outer_diagnostic_basis": { @@ -104,37 +105,42 @@ { "assertions": [ { - "evidence": "result was GIT_SIGNING_BLOCKED with actual GPG signing-failure diagnostics", - "name": "Git signing failure classified", + "evidence": "Live probe result was GIT_SIGNING_BLOCKED from actual GPG signing diagnostics.", + "name": "Probe accurately reports each Git capability outcome.", "status": "PASS" }, { - "evidence": "source_snapshot_changed was empty", - "name": "Source snapshot preserved", + "evidence": "Temporary ref, probe branch, and probe path were confirmed absent.", + "name": "Temporary refs/branches/worktrees/files are cleaned.", "status": "PASS" }, { - "evidence": "cleanup_errors was empty; temporary ref and probe branch were absent; no matching worktree or probe artifacts remained", - "name": "Probe cleanup completed", + "evidence": "source_snapshot_changed was empty and source status remained clean.", + "name": "Source repository snapshot is preserved.", + "status": "PASS" + }, + { + "evidence": "Commit failure diagnostic reported GPG signing failure and was classified GIT_SIGNING_BLOCKED.", + "name": "Signing and repository-hook failures are classified from actual diagnostics.", "status": "PASS" } ], "blocker": null, "capability_id": "C16", "event_summary": { - "completed_command_items": 4, + "completed_command_items": 2, "completed_file_change_items": 0, "error_events": 0, "event_types": { - "item.completed": 10, - "item.started": 4, + "item.completed": 7, + "item.started": 2, "thread.started": 1, "turn.completed": 1, "turn.started": 1 }, "item_types": { - "agent_message": 4, - "command_execution": 8, + "agent_message": 3, + "command_execution": 4, "error": 2 } }, @@ -142,8 +148,8 @@ "branch": "main", "cached_paths": [], "commit_count": 2, - "head": "0e5fdbad2f62c221e4fe3c5c1b53a6e645eb28cd", - "index_tree": "d0f99c956fcb88e978566d60c763b86076e3226a", + "head": "c6d09bae00cf0a7deefabc1b6a3017e3bd05de3e", + "index_tree": "d6e50af2eacc26740687f9f8f7edc4c31501e6aa", "refs": [ "refs/heads/main" ], @@ -154,8 +160,8 @@ "branch": "main", "cached_paths": [], "commit_count": 2, - "head": "0e5fdbad2f62c221e4fe3c5c1b53a6e645eb28cd", - "index_tree": "d0f99c956fcb88e978566d60c763b86076e3226a", + "head": "c6d09bae00cf0a7deefabc1b6a3017e3bd05de3e", + "index_tree": "d6e50af2eacc26740687f9f8f7edc4c31501e6aa", "refs": [ "refs/heads/main" ], @@ -163,7 +169,8 @@ "worktree_count": 1 }, "observations": [ - "Probe exited 2 with plan_status BLOCKED_BY_RUNTIME_PREREQUISITE, the expected process behavior for GIT_SIGNING_BLOCKED." + "cleanup_errors was empty.", + "No probe artifacts remained." ], "outcome": "PASS", "outer_diagnostic_basis": { @@ -182,22 +189,22 @@ { "assertions": [ { - "evidence": "Live probe returned result GIT_HOOK_BLOCKED with diagnostic \"pre-commit hook failed: C16 fixture hook rejection\".", + "evidence": "Live probe result was GIT_HOOK_BLOCKED from actual pre-commit hook rejection.", "name": "Probe accurately reports each Git capability outcome.", "status": "PASS" }, { - "evidence": "cleanup_errors was empty; no temporary refs, branches, worktrees, or files remained.", + "evidence": "Temporary ref and probe branch were absent; no matching worktrees or probe artifacts remained.", "name": "Temporary refs/branches/worktrees/files are cleaned.", "status": "PASS" }, { - "evidence": "source_snapshot_changed was empty and the source worktree remained clean on main.", + "evidence": "source_snapshot_changed was empty.", "name": "Source repository snapshot is preserved.", "status": "PASS" }, { - "evidence": "The actual pre-commit hook rejection was classified as GIT_HOOK_BLOCKED.", + "evidence": "Diagnostic classified the pre-commit hook failure as GIT_HOOK_BLOCKED.", "name": "Signing and repository-hook failures are classified from actual diagnostics.", "status": "PASS" } @@ -209,14 +216,14 @@ "completed_file_change_items": 0, "error_events": 0, "event_types": { - "item.completed": 6, + "item.completed": 7, "item.started": 2, "thread.started": 1, "turn.completed": 1, "turn.started": 1 }, "item_types": { - "agent_message": 2, + "agent_message": 3, "command_execution": 4, "error": 2 } @@ -225,8 +232,8 @@ "branch": "main", "cached_paths": [], "commit_count": 2, - "head": "0e5fdbad2f62c221e4fe3c5c1b53a6e645eb28cd", - "index_tree": "d0f99c956fcb88e978566d60c763b86076e3226a", + "head": "c6d09bae00cf0a7deefabc1b6a3017e3bd05de3e", + "index_tree": "d6e50af2eacc26740687f9f8f7edc4c31501e6aa", "refs": [ "refs/heads/main" ], @@ -237,8 +244,8 @@ "branch": "main", "cached_paths": [], "commit_count": 2, - "head": "0e5fdbad2f62c221e4fe3c5c1b53a6e645eb28cd", - "index_tree": "d0f99c956fcb88e978566d60c763b86076e3226a", + "head": "c6d09bae00cf0a7deefabc1b6a3017e3bd05de3e", + "index_tree": "d6e50af2eacc26740687f9f8f7edc4c31501e6aa", "refs": [ "refs/heads/main" ], @@ -246,9 +253,8 @@ "worktree_count": 1 }, "observations": [ - "Probe result: GIT_HOOK_BLOCKED.", - "source_snapshot_changed: []; cleanup_errors: [].", - "Post-probe artifact check found only the original source worktree on main." + "cleanup_errors was empty.", + "No probe artifacts remained." ], "outcome": "PASS", "outer_diagnostic_basis": { diff --git a/capabilities/C16/hashes.json b/capabilities/C16/hashes.json index b6db89c..ac199fb 100644 --- a/capabilities/C16/hashes.json +++ b/capabilities/C16/hashes.json @@ -1,8 +1,8 @@ { "algorithm": "sha256", "files": { - "README.md": "74b49792d803d57e7fd23cc955041231d27812c5e6af8d51da76f6e17f5abc54", - "actual.sanitized.json": "1b2002580aa28fd816e56d47b80cb6afe397803f7101b6571f98b1330b4ccb8c", + "README.md": "2f3c562b54d23d2bc093cdbc041f7e5d092b31f08595acd3f677d35c0ba7c445", + "actual.sanitized.json": "75a018e6b0b4c9efb1ee88c34d203817e9d4e85727dff69e9ee50626eb68867f", "config/README.md": "d0666f09608c05e3f0e506d457cebf74e110f3cb7474464b05222992b6d65b34", "evaluation.json": "e3a8788c8fb42aaaa6ea36ab4c721691bdb7e1e1cdbfdb3361c063cb643f7c7e", "expected.json": "8006ed5e85c38ebd01d7786544557a6fc56a5fedb94a4da580f4d44ac7ba6262", diff --git a/capabilities/index.json b/capabilities/index.json index cd6b48d..eb138d5 100644 --- a/capabilities/index.json +++ b/capabilities/index.json @@ -131,13 +131,13 @@ } ], "evidence_package_state": "LIVE_QUALIFIED", - "generated_at": "2026-09-06", + "generated_at": "2026-09-07", "qualification_attempt": { - "date": "2026-09-06", - "github_actions_run": "34060321283", + "date": "2026-09-07", + "github_actions_run": "34140846679", "live_codex_result": "PASS", "required_not_reproduced": [], - "source_commit": "d0384f76bc4150d33bb8f51ef5981f3243b3cfb3" + "source_commit": "a9cdcdc1e0cad70e88b60869e75e4166046dd306" }, "release_gate": "Required capabilities must have committed sanitized REPRODUCED evidence before production readiness." } diff --git a/docs/CODEX_SANDBOX_RUNBOOK.md b/docs/CODEX_SANDBOX_RUNBOOK.md index 9334ddc..d7f7227 100644 --- a/docs/CODEX_SANDBOX_RUNBOOK.md +++ b/docs/CODEX_SANDBOX_RUNBOOK.md @@ -1,6 +1,6 @@ # Codex sandbox qualification runbook -Full run #25 is archived and passed baseline 2.3 on Codex CLI 0.153.4. The finite C08 stop/repair replacement is merged. Publication now requires a successful new full run and import of its complete evidence. `QUALIFICATION_STATUS.md` is the current status; this runbook describes both full requalification and targeted checks. +Full run #27 is reviewed and archived: baseline 2.3 C01-C16 and finite C08 stop/repair passed on Codex CLI 0.153.4. Its complete evidence is recorded in the repository; no new run is needed just to import it. Historical #25 remains unchanged. `QUALIFICATION_STATUS.md` records the tested configuration, limitations and publication boundary. This runbook describes future full requalification and targeted checks. ## Sandbox prerequisites @@ -27,9 +27,9 @@ If the probe fails with `setting up uid map: Operation not permitted`, fix the P The preferred qualification path is `.github/workflows/plananvil-codex-qualification.yml`. The workflow is intentionally `workflow_dispatch`-only, accepts execution only from `main`, uses Environment `plananvil-codex`, and targets `[self-hosted, linux, x64, plananvil, codex]`. -Use **mode=full** for the next publication-closing C01-C16 sequence. The production validator requires one complete source-bound full-run archive with the finite C08 result, not a combination of partial archives. `mode=c08` is available to diagnose only the finite stop/repair replacement. `mode=recovery` retains C09/C10/C13; `mode=c13` selects C13 only. `mode=smoke` verifies basic runtime/authentication. The precision/variant matrices are diagnostics, not release evidence. All targeted modes keep `release_gate_passed=false`, even when their selected checks pass. +Use **mode=full** when a new publication-closing C01-C16 qualification is needed. The production validator requires one complete source-bound full-run archive with the finite C08 result, not a combination of partial archives. `mode=c08` is available to diagnose only the finite stop/repair replacement. `mode=recovery` retains C09/C10/C13; `mode=c13` selects C13 only. `mode=smoke` verifies basic runtime/authentication. The precision/variant matrices are diagnostics, not release evidence. All targeted modes keep `release_gate_passed=false`, even when their selected checks pass. -Start a NEW **PlanAnvil Codex qualification -> Run workflow -> main -> full** from an account allowed by the local runner's initiating-actor policy. Automated run #26 (`34109662176`) was rejected by that policy before any Codex trial; do not rerun it or relax the allowlist. The repair is on main, but that rejected launch provides no model-backed result. +For requalification, start a NEW **PlanAnvil Codex qualification -> Run workflow -> main -> full** from an account allowed by the local runner's initiating-actor policy. Automated run #26 (`34109662176`) was rejected before any Codex trial. Owner-initiated #27 then passed; do not rerun the rejected launch or relax the allowlist. Both `full` and `c08` use the same active v7 C08 runtime with real live model inference. The outer harness deliberately starts without a checkpoint, observes the actual PreCompact stop and termination, creates/validates a real checkpoint, then requires one pressure -> compaction -> SessionStart(compact) -> finish sequence and a completed positive result. The disposable fixture excludes ordinary startup recovery only; product files and C10 startup coverage remain unchanged. @@ -117,7 +117,7 @@ C04 is informational/non-gating in baseline 2.3 but should still be observed if ## Final gate -Full capability qualification and publication closure are separate. Preserve the original #25 archive unchanged. For publication, review and commit a new successful full-run archive whose C08 has finite completion evidence, along with its exact C01-C16 packages and actual executed source SHA. Update `qualifications/index.json` to that full run as described in `RELEASE.md`. Never replace the full-run index with a partial-mode index or splice a targeted result into an older archive. +Full capability qualification and signed publication are separate. Current full run #27 supplies finite C08 completion and exact C01-C16 packages; `qualifications/index.json` identifies it. Preserve both #25 and #27 archives unchanged. Any future qualification must be imported as one complete source-bound full archive as described in `RELEASE.md`. Never replace the full-run index with a partial-mode index or splice a targeted result into an older archive. When the complete current full-run evidence is committed: diff --git a/docs/QUALIFICATION_STATUS.md b/docs/QUALIFICATION_STATUS.md index 4d04c04..624b758 100644 --- a/docs/QUALIFICATION_STATUS.md +++ b/docs/QUALIFICATION_STATUS.md @@ -1,81 +1,92 @@ # Qualification status -## Recorded live result - -Full run #25 (`34060321283`) passed baseline 2.3 C01-C16 on product source -`d0384f76bc4150d33bb8f51ef5981f3243b3cfb3`, Codex CLI 0.153.4, -`gpt-5.6-sol`, Debian 13. C04 is informational. The immutable original archive, -summary, checksums and caveats are in `qualifications/34060321283/`. - -## Implemented closure - -PR #34 merged the finite C08 repair and the reviewed evidence into main at -`ba3f56a644a385c8aa2e8e5c9c934999184cc282`. PR CI #118 and post-merge CI #119 -passed all eight jobs, including actual Codex CLI loopback conformance. These -are implementation/integration checks, not a new live-model result. - -The historical positive C08 trial timed out after meeting its narrow unblock -assertion. It remains unchanged in the archive. The finite replacement requires -the same canonical run to stop at PreCompact without a checkpoint, then, after -outer checkpoint repair, complete pressure -> compact recovery -> finish with -exit zero. Timeout or incomplete telemetry cannot pass. - -C13 passed in the explicitly allowed project-scoped non-ephemeral fallback. -Ephemeral spawning is not claimed as working. The product .agents/.codex payload -and the C09/C10/C13 runtime paths are unchanged by this closure. - -## Required operator action - -The automated full launch #26 (`34109662176`) used the repaired main commit, -but the self-hosted runner's job-start policy rejected its initiating actor: - -```text -PlanAnvil runner policy denied this job: initiating actor is not allowed -``` - -The hook exited 77 before runner preflight, fixture preparation or any Codex -trial. This run has no new capability evidence and is not a C08 test failure. -Do not change the runner allowlist, spoof the actor or relax the trusted-workflow -policy to make an automated launch pass. - -An operator whose account is allowed by the runner policy must start a NEW run: - -**Actions -> PlanAnvil Codex qualification -> Run workflow -> main -> full**. - -Use Run workflow, not Re-run of the denied automated run. GitHub Actions -orchestrates the job; all model-backed trials run on the existing authenticated -self-hosted runner, not on a GitHub-hosted machine. - -## Evidence required for publication - -The current production validator requires ONE complete, source-bound full-run -archive, exact matching current capability packages, unchanged qualified product -bytes, and finite C08 stop/repair proof in that same archive. Therefore the next -publication proof must be a successful new `full` run. The `c08` mode remains a -useful targeted diagnostic, but its partial archive alone cannot close the -production gate. Do not splice it into or relabel the immutable #25 archive. - -After success, review and verify the new sanitized archive, preserve it under -`qualifications//`, import its exact C01-C16 packages, and update -`qualifications/index.json` through a protected PR. Keep #25 unchanged as history. -Record the actually executed source SHA, not the later import commit. Run the -strict release check on a clean tree after import. No production tag or release -has been created by these preparation steps. - -## Verification layers - -Unit/process tests and pinned real-CLI loopback tests check implementation and -protocol behavior. Only the authenticated self-hosted qualification workflow -provides live-model evidence. Newly materialized templates always start BLOCKED; -archived evidence cannot silently qualify a fresh fixture or changed product. - -## Audit index +## Current reviewed full result + +**Live qualification is complete for the tested configuration.** Owner-initiated +full run #27 (`34140846679`) passed baseline 2.3 C01-C16 on executed source +`a9cdcdc1e0cad70e88b60869e75e4166046dd306`, Codex CLI 0.153.4, +model `gpt-5.6-sol`, Debian GNU/Linux 13. C04 is informational; fifteen +capabilities are required. The full run reports `release_gate_passed=true`. +All model-backed trials ran on the authenticated self-hosted runner. + +The original archive, exact summary and verified provenance are preserved in +`qualifications/34140846679/`. `qualifications/index.json.current_run` points to +that run. Current `capabilities/C01` through `C16` are exact archive copies, +not rewritten observations or a combination of partial runs. The executed source +SHA remains the original SHA, never the later evidence-import commit. + +## C08 closure and recovery regressions + +C08 completed its finite stop/repair protocol in the same canonical run: + +- Without a checkpoint: one completed pressure command, an actual automatic + PreCompact `continue=false`, expected process exit 1 after 10.643 seconds, + no PostCompact and no timeout. +- After the outer harness created a real valid checkpoint: pressure, one + automatic PreCompact/PostCompact cycle, SessionStart(source=compact), finish + reconciliation and positive structured completion; exit 0 after 36.265 seconds, + no timeout. All nine positive protocol checks passed. + +C09 completed all three reconciliations and two ordered automatic recovery +cycles with exit 0 in 60.679 seconds. Both independent C10 context-delivery +probes completed with exact recovery values and exit 0. Process cleanup and +checkpoint/source immutability checks passed for the recovery probes. + +There are no recorded `timeout=true` values in the current actual evidence. +This is not a guarantee of future model behavior or support for untested +Codex/model/operating-system combinations. + +## Retained compatibility limitation + +C13 passed using the explicitly allowed project-scoped non-ephemeral fallback +after the recognized ephemeral parent-thread failure. Agent and SubagentStart +hook remained project-scoped; child context, cleanup and authentication +invariants passed. Ephemeral spawning is not claimed as working. + +## Integrity and publication boundary + +The archive contains 166 manifest-listed files plus the manifest, including +three hidden fixture files. All SHA-256 checks, per-capability package validation +and source identities were verified before import. The qualified .agents/.codex +product bytes, runtime implementations, permission policy and release guards +are unchanged by this evidence/documentation update. + +On a clean checkout, run `python tools/validate_capabilities.py` and +`python tools/release_check.py` without `--candidate`. The latter requires the +complete full-run archive, exact packages, unchanged qualified product bytes +and finite C08 completion. Freshly materialized unexecuted templates still start +BLOCKED and cannot inherit the archived success. + +No additional live run is required merely to record these same results or +update documentation. Requalify when relevant product/runtime inputs change. +Production publication remains a separate owner-authorized verified signed +annotated tag and release workflow, as described in `RELEASE.md`. No production +tag or release is created by the evidence import. + +## Preserved history + +Run #25 (`34060321283`, source `d0384f76bc4150d33bb8f51ef5981f3243b3cfb3`) +passed the older narrow assertions but its positive C08 trial later timed out. +Its original archive, summary, provenance and caveat remain unchanged under +`qualifications/34060321283/`; it is not the current publication evidence. + +PR #34 merged the finite C08 repair. Automated run #26 (`34109662176`) was +rejected by the local initiating-actor policy before Codex. It provides no live +result. The owner-initiated #27 completed without changing or bypassing that +policy. PR #35 clarified the requirement for one complete full archive. + +## Verification layers and audit index + +Unit/process tests and real-CLI loopback conformance check implementation and +protocol behavior. Only the authenticated self-hosted run supplies live-model +evidence. The archived full result does not imply execution of every possible +application plan; PlanAnvil generates and validates plans, never implements them. - `CODEX_CAPABILITY_QUALIFICATION_2026-08-28.md`: historical prerequisite-limited attempt. - `CODEX_QUALIFICATION_EXECUTION_AUDIT_2026-09-05.md`: C13 argv and C10 worktree discovery. -- `CODEX_RECOVERY_DELIVERY_AUDIT_2026-09-05.md`: supported context delivery and diagnostics. -- `CODEX_C09_FINITE_RECOVERY_AUDIT_2026-09-06.md`: completed finite C09 repair. -- `CODEX_C08_CLOSURE_AUDIT_2026-09-07.md`: finite C08 and evidence closure. +- `CODEX_RECOVERY_DELIVERY_AUDIT_2026-09-05.md`: context delivery and diagnostics. +- `CODEX_C09_FINITE_RECOVERY_AUDIT_2026-09-06.md`: finite C09 repair. +- `CODEX_C08_CLOSURE_AUDIT_2026-09-07.md`: finite C08 implementation. -Older harness modules are retained because v7 still imports them. They are not -independent supported entry points. Do not delete imported layers as cosmetic cleanup. +Older harness modules remain active v7 dependencies, not separate supported +entry points. They must not be removed as cosmetic cleanup. diff --git a/docs/RELEASE.md b/docs/RELEASE.md index a750ae4..1de947a 100644 --- a/docs/RELEASE.md +++ b/docs/RELEASE.md @@ -25,7 +25,7 @@ python tools/build_release.py --output dist Before a production tag, execute `docs/CODEX_SANDBOX_RUNBOOK.md`. Required entries in `capabilities/index.json` must be changed from `BLOCKED` to `REPRODUCED` only after their complete sanitized evidence package is committed and `python tools/validate_capabilities.py` passes. -For the current C08 closure, use a NEW `mode=full` run on main, initiated by an account allowed by the self-hosted runner. The production validator requires a single complete full-run archive containing the finite C08 proof. A successful `c08` or `recovery` diagnostic is useful but its partial archive cannot replace that full run. See `docs/QUALIFICATION_STATUS.md` for the merged repair and the automated launch rejected before testing. +Current full run #27 (`34140846679`) passed and its exact reviewed evidence is committed, including finite C08 stop/repair completion. No further live run is required merely to record this result. The production validator still requires one complete full-run archive; a `c08` or `recovery` diagnostic cannot replace it. See `docs/QUALIFICATION_STATUS.md` for provenance, tested scope and the retained C13 fallback limitation. ## Publish @@ -54,16 +54,17 @@ The production `release_check.py` enforces clean-tree state in addition to versi Before production release, protect `main` as tracked in issue #6: PR-only changes, required CI, up-to-date branch, conversation resolution, and no force push/delete. -## Qualification closure (2026-09-07) +## Qualification closure (reviewed 2026-09-08) The baseline #25 evidence is preserved immutably. `qualifications/index.json` identifies the current reviewed full run; its source SHA always remains the actually executed commit, never the later evidence-import commit. `release_check.py` also validates the original archive digest/manifest, exact current capability packages, qualified .agents/.codex bytes, and finite C08 -negative-stop/positive-completion evidence. The old C08 timeout keeps production -blocked until the new complete full-run result is committed. Candidate checks -intentionally remain usable before that result exists. +negative-stop/positive-completion evidence. Full #27 now supplies that proof; +the old #25 timeout remains recorded as history. Candidate checks still do not +substitute for strict production validation. Run `python tools/release_check.py` +on the clean evidence-import commit before any signed publication. To import a successful new full result, preserve its exact archive, summary and verified provenance under `qualifications//`, import its exact C01-C16 @@ -78,7 +79,8 @@ seven required status checks, strict up-to-date branch, conversation resolution, no deletion or force push, no bypass actors. The protected distribution job now also requires real-CLI conformance to succeed. No policy is loosened for closure. -After the complete new full-run evidence is imported and CI is green, preparation -is complete; the signed annotated production tag remains a separate authorized -publication. No unsigned or lightweight tag may substitute for the required -verified signature. +The complete #27 evidence is imported. After strict validation and green CI on +the evidence-import commit, qualification preparation is complete; a signed +annotated production tag remains a separate authorized publication. This import +does not create a tag or release. No unsigned or lightweight tag may substitute +for the required verified signature. diff --git a/qualifications/34140846679/evidence-artifact.zip b/qualifications/34140846679/evidence-artifact.zip new file mode 100644 index 0000000..6296313 Binary files /dev/null and b/qualifications/34140846679/evidence-artifact.zip differ diff --git a/qualifications/34140846679/provenance.json b/qualifications/34140846679/provenance.json new file mode 100644 index 0000000..c038684 --- /dev/null +++ b/qualifications/34140846679/provenance.json @@ -0,0 +1,18 @@ +{ + "artifact_id": 10026707683, + "evidence_sha256": "83beb50d27198c2cc0ecd72bc6ca4b1ec2d6570bcb23cc0d56c12ec97408ec75", + "github_actions_run": "34140846679", + "integrity": { + "complete": true, + "file_count": 166, + "manifest_listed_hidden_files": 3, + "schema_version": "1.0" + }, + "limitations": [ + "C13 passed through the explicitly permitted project-scoped non-ephemeral fallback; ephemeral transport remains unsupported in this observed path." + ], + "outer_sha256": "9583393745fc8292de701c51064e5569f24092e978da7164c8f50070e7d35e73", + "qualification_mode": "full", + "schema_version": "1.0", + "source_commit": "a9cdcdc1e0cad70e88b60869e75e4166046dd306" +} diff --git a/qualifications/34140846679/qualification-summary.json b/qualifications/34140846679/qualification-summary.json new file mode 100644 index 0000000..52119ec --- /dev/null +++ b/qualifications/34140846679/qualification-summary.json @@ -0,0 +1,29 @@ +{ + "codex_version": "codex-cli 0.153.4", + "date": "2026-09-07", + "github_actions_run": "34140846679", + "model": "gpt-5.6-sol", + "os": "Debian GNU/Linux 13 (trixie)", + "release_gate_passed": true, + "required_not_reproduced": [], + "results": { + "C01": "REPRODUCED", + "C02": "REPRODUCED", + "C03": "REPRODUCED", + "C04": "REPRODUCED", + "C05": "REPRODUCED", + "C06": "REPRODUCED", + "C07": "REPRODUCED", + "C08": "REPRODUCED", + "C09": "REPRODUCED", + "C10": "REPRODUCED", + "C11": "REPRODUCED", + "C12": "REPRODUCED", + "C13": "REPRODUCED", + "C14": "REPRODUCED", + "C15": "REPRODUCED", + "C16": "REPRODUCED" + }, + "schema_version": "1.0", + "source_commit": "a9cdcdc1e0cad70e88b60869e75e4166046dd306" +} diff --git a/qualifications/index.json b/qualifications/index.json index 3a6c693..18ecd42 100644 --- a/qualifications/index.json +++ b/qualifications/index.json @@ -1,6 +1,6 @@ { "baseline_run": "34060321283", - "c08_closure": "PENDING_LIVE", - "current_run": "34060321283", + "c08_closure": "REPRODUCED", + "current_run": "34140846679", "schema_version": "1.0" } diff --git a/tests/test_qualification_closure.py b/tests/test_qualification_closure.py index cf27474..638335a 100644 --- a/tests/test_qualification_closure.py +++ b/tests/test_qualification_closure.py @@ -11,10 +11,39 @@ sys.path.insert(0, str(ROOT/'tools')) import prepare_capabilities as prepare import qualification_closure as closure -from qualification_artifact import build_archive +import release_check +from qualification_artifact import build_archive, verify_archive class ClosureTests(unittest.TestCase): + def test_reviewed_current_full_archive_and_summary_are_consistent(self): + # Verify archived evidence, not production eligibility of a development PR. + # Current product binding remains the responsibility of release_check.py. + index = json.loads((ROOT / 'qualifications/index.json').read_text()) + self.assertEqual(index['c08_closure'], 'REPRODUCED') + folder = ROOT / 'qualifications' / index['current_run'] + provenance = json.loads((folder / 'provenance.json').read_text()) + self.assertEqual(provenance['qualification_mode'], 'full') + verify_archive(folder / 'evidence-artifact.zip') + with zipfile.ZipFile(folder / 'evidence-artifact.zip') as z: + summary_bytes = z.read('qualification-summary.json') + self.assertEqual((folder / 'qualification-summary.json').read_bytes(), summary_bytes) + summary = json.loads(summary_bytes) + self.assertEqual(summary['github_actions_run'], index['current_run']) + self.assertEqual(summary['source_commit'], provenance['source_commit']) + self.assertTrue(summary['release_gate_passed']) + self.assertEqual(summary['required_not_reproduced'], []) + + def test_current_package_change_is_not_accepted_as_archived_evidence(self): + with tempfile.TemporaryDirectory() as tmp: + root = self.copy_repo(tmp) + path = root / 'capabilities/C09/actual.sanitized.json' + # Even a JSON-preserving edit is not the original source-bound record. + with path.open('ab') as stream: + stream.write(b'\n') + self.assertTrue(any('current capability differs from archived run' in error + for error in closure.closure_blockers(root))) + def copy_repo(self, tmp): target = Path(tmp)/'repo' shutil.copytree(ROOT, target, ignore=shutil.ignore_patterns('.git', '__pycache__')) @@ -40,6 +69,8 @@ def test_product_change_is_rejected_independently_of_git_status(self): root=self.copy_repo(tmp) (root/'.agents/skills/plan-anvil/SKILL.md').write_text('changed') self.assertIn('product bytes changed since recorded live qualification', closure.closure_blockers(root)) + # Ordinary candidate PRs stay usable before new main-only live qualification. + self.assertEqual(release_check.release_blockers(root, require_reproduced=False), []) def test_archive_digest_corruption_is_rejected(self): with tempfile.TemporaryDirectory() as tmp: