Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,14 @@ All notable changes to PlanAnvil are documented here.

## [Unreleased]

### Qualification closure — 2026-09-07

- full self-hosted run #27 (`34140846679`) reproduced all C01-C16 on executed source `a9cdcdc1e0cad70e88b60869e75e4166046dd306`, Codex CLI 0.153.4, `gpt-5.6-sol`, Debian 13;
- finite C08 now proves the intended missing-checkpoint stop and completed repaired recovery without timeout; C09/C10 regressions passed;
- preserve the exact full archive and provenance, update current capability evidence, and retain historical #25 unchanged;
- C13 retains the documented project-scoped, known-error-gated non-ephemeral fallback; no new transport claim or product/runtime/security change;
- qualification is recorded; signed production publication remains a separate action.

### Added

- add a controlled `workflow_dispatch`-only Codex qualification workflow for the trusted `plananvil-codex` self-hosted runner;
Expand Down Expand Up @@ -59,14 +67,14 @@ All notable changes to PlanAnvil are documented here.

- preserve complete baseline 2.3 live evidence from full run #25, `34060321283`, tested at `d0384f76bc4150d33bb8f51ef5981f3243b3cfb3` with Codex CLI 0.153.4, `gpt-5.6-sol`, Debian 13; all C01–C16 were reproduced;
- retain the exact source-bound archive, hashes and original limitations; C13 passed via the permitted project-native non-ephemeral fallback;
- replace the old C08 repaired-path workload with a finite pressure/finish scenario and strict termination checks; the old positive timeout remains in historical evidence and the replacement still requires live confirmation;
- replace the old C08 repaired-path workload with a finite pressure/finish scenario and strict termination checks; the old positive timeout remains in historical evidence; full run #27 subsequently confirmed the replacement live;
- add C08-only qualification and real-CLI loopback conformance without changing C09/C10/C13 runtime behavior, product payload or live runner security;
- reset newly materialized template indices to unexecuted package results instead of inheriting historical success labels;
- validate qualification archive integrity and qualified product identity, and require committed finite C08 completion evidence before production publication.

### Release status

0.2.0 remains a release candidate. Full baseline qualification is recorded; production publication awaits the finite C08 live follow-up, strict release validation and a verified signed annotated tag. No tag or release has been published by the qualification-closure change.
0.2.0 is qualified for the recorded configuration. Full run #27 (`34140846679`) confirmed C01-C16 and finite C08 stop/repair on source `a9cdcdc1e0cad70e88b60869e75e4166046dd306`; its complete source-bound evidence is committed and #25 remains unchanged. Production publication remains a separate verified signed annotated-tag action after strict release validation. No tag or release has been published by this evidence import.

## [0.1.0] - 2026-07-12

Expand Down
6 changes: 4 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,9 @@ It **generates and validates a plan but never executes it**. Product implementat

The deterministic generator core, schemas, templates, tests, optional planning agents, defense-in-depth hooks, repository installer/upgrader/uninstaller, release tooling, and a deterministic C01-C16 live-qualification template archive/materializer are implemented.

**Full baseline 2.3 qualification passed:** run [#25](https://github.com/KeyffMS/PlanAnvil/actions/runs/34060321283) reproduced C01–C16 on source commit `d0384f76bc4150d33bb8f51ef5981f3243b3cfb3`, with Codex CLI `0.153.4`, model `gpt-5.6-sol`, and Debian 13. C04 is informational; the other fifteen capabilities gate qualification. The complete sanitized evidence is committed under `capabilities/` and preserved unchanged under `qualifications/34060321283/`.
**Full baseline 2.3 qualification and finite C08 closure passed:** run [#27](https://github.com/KeyffMS/PlanAnvil/actions/runs/34140846679) reproduced C01–C16 on source commit `a9cdcdc1e0cad70e88b60869e75e4166046dd306`, with Codex CLI `0.153.4`, model `gpt-5.6-sol`, and Debian 13. C04 is informational; the other fifteen capabilities gate qualification. The exact sanitized packages are committed under `capabilities/`; the original archive, summary and provenance are preserved under `qualifications/34140846679/`. Historical run #25 remains unchanged.

**Release status remains candidate.** The old C08 positive trial proved its narrow unblock assertion but later timed out. Its finite replacement must finish in a new live `mode=c08` run and its evidence must be committed before production publication. C13 was reproduced through the explicit known-error-gated non-ephemeral fallback, not by proving ephemeral spawning works.
**Qualification is complete for that tested configuration; signed publication is a separate step.** C08 now proves both the expected missing-checkpoint stop and a completed repaired recovery, without timeout. C09 and both C10 probes also completed without timeout. C13 was reproduced through the explicit known-error-gated, project-scoped non-ephemeral fallback; ephemeral spawning is not claimed as working. This evidence import does not create a production tag or GitHub Release.

See [current qualification and release status](https://github.com/KeyffMS/PlanAnvil/blob/main/docs/QUALIFICATION_STATUS.md) for the precise evidence boundary and next step. Deterministic CI and loopback CLI conformance are regression checks, not substitutes for live model evidence. Production publication also retains protected-main, clean-tree and verified signed annotated-tag requirements.

Expand Down Expand Up @@ -104,6 +104,8 @@ python -m unittest discover -s .agents/skills/plan-anvil/tests -v
python -m unittest discover -s tests -v
python -m compileall -q .agents/skills/plan-anvil .codex/hooks tools tests
python tools/release_check.py --candidate
# On a clean checkout containing the reviewed live evidence:
python tools/release_check.py
```

## Release and live qualification
Expand Down
4 changes: 2 additions & 2 deletions capabilities/C01/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,11 @@ Use `fixture/`, `config/README.md`, `prompt.txt`, and `run-command.txt`. Replace

## Live qualification

- Date: `2026-09-06`
- Date: `2026-09-07`
- Codex: `codex-cli 0.153.4`
- Model: `gpt-5.6-sol`
- OS: `Debian GNU/Linux 13 (trixie)`
- Permission mode: `approval=never; sandbox=per-trial; model-tool network disabled`
- Project trust: `trusted via CLI override for disposable fixture repositories`
- Source commit: `d0384f76bc4150d33bb8f51ef5981f3243b3cfb3`
- Source commit: `a9cdcdc1e0cad70e88b60869e75e4166046dd306`
- Result: `REPRODUCED`
39 changes: 22 additions & 17 deletions capabilities/C01/actual.sanitized.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,27 +2,32 @@
"capability_id": "C01",
"environment": {
"codex_version": "codex-cli 0.153.4",
"fixture_commit": "4df35777e3afd6f5b6a9aecfe22793e65bba83ea",
"fixture_commit": "00f746c24feb4ecc943c59cbe8006db2a01e9cd3",
"model": "gpt-5.6-sol",
"os": "Debian GNU/Linux 13 (trixie)",
"permission_mode": "approval=never; sandbox=per-trial; model-tool network disabled",
"project_trust": "trusted via CLI override for disposable fixture repositories",
"source_commit": "d0384f76bc4150d33bb8f51ef5981f3243b3cfb3"
"source_commit": "a9cdcdc1e0cad70e88b60869e75e4166046dd306"
},
"observations": [
"The explicit activation trial resolved the repository fixture skill and produced the activation proof `FIXTURE_SKILL_ACTIVE`.",
"The nested-working-directory trial verified repository identity, discovered the same fixture skill, and produced `FIXTURE_SKILL_ACTIVE`.",
"Both trials completed with PASS outcomes, no blocker, and no recorded error events."
"The explicit-activation trial resolved the fixture skill from the repository skill directory and recorded activation proof `FIXTURE_SKILL_ACTIVE`.",
"The nested-working-directory trial verified repository identity from its nested start location, discovered and invoked the fixture skill, and recorded the same activation proof.",
"The first trial did not exercise nested discovery, but the completed second trial directly exercised and passed that assertion."
],
"result": "REPRODUCED",
"schema_version": "1.0",
"trials": [
{
"assertions": [
{
"evidence": "Explicit invocation resolved `fixture-capability` from `.agents/skills/fixture-capability/SKILL.md`; activation proof: FIXTURE_SKILL_ACTIVE.",
"name": "Repository skill resolution and explicit activation",
"evidence": "`fixture-capability` was resolved from `.agents/skills/fixture-capability/SKILL.md` and explicitly activated with proof `FIXTURE_SKILL_ACTIVE`.",
"name": "The fixture skill appears in repository skill discovery.",
"status": "PASS"
},
{
"evidence": "Not exercised by this root-level explicit-activation trial.",
"name": "Discovery works when Codex starts from a nested working directory inside the same repository.",
"status": "BLOCKED"
}
],
"blocker": null,
Expand All @@ -48,7 +53,7 @@
"branch": "master",
"cached_paths": [],
"commit_count": 2,
"head": "4df35777e3afd6f5b6a9aecfe22793e65bba83ea",
"head": "00f746c24feb4ecc943c59cbe8006db2a01e9cd3",
"index_tree": "715ef280db9d5ab81405996e12aa2be15877d964",
"refs": [
"refs/heads/master"
Expand All @@ -60,7 +65,7 @@
"branch": "master",
"cached_paths": [],
"commit_count": 2,
"head": "4df35777e3afd6f5b6a9aecfe22793e65bba83ea",
"head": "00f746c24feb4ecc943c59cbe8006db2a01e9cd3",
"index_tree": "715ef280db9d5ab81405996e12aa2be15877d964",
"refs": [
"refs/heads/master"
Expand All @@ -79,40 +84,40 @@
{
"assertions": [
{
"evidence": "Explicit invocation produced activation proof FIXTURE_SKILL_ACTIVE.",
"evidence": "$fixture-capability was discovered and invoked; activation proof: FIXTURE_SKILL_ACTIVE.",
"name": "The fixture skill appears in repository skill discovery.",
"status": "PASS"
},
{
"evidence": "Repository identity was verified from the nested working directory, and the repository skill was then discovered and activated.",
"evidence": "Git repository identity was verified from the nested starting directory without reporting its absolute path.",
"name": "Discovery works when Codex starts from a nested working directory inside the same repository.",
"status": "PASS"
}
],
"blocker": null,
"capability_id": "C01",
"event_summary": {
"completed_command_items": 3,
"completed_command_items": 2,
"completed_file_change_items": 0,
"error_events": 0,
"event_types": {
"item.completed": 8,
"item.started": 3,
"item.completed": 7,
"item.started": 2,
"thread.started": 1,
"turn.completed": 1,
"turn.started": 1
},
"item_types": {
"agent_message": 3,
"command_execution": 6,
"command_execution": 4,
"error": 2
}
},
"git_after": {
"branch": "master",
"cached_paths": [],
"commit_count": 2,
"head": "4df35777e3afd6f5b6a9aecfe22793e65bba83ea",
"head": "00f746c24feb4ecc943c59cbe8006db2a01e9cd3",
"index_tree": "715ef280db9d5ab81405996e12aa2be15877d964",
"refs": [
"refs/heads/master"
Expand All @@ -124,7 +129,7 @@
"branch": "master",
"cached_paths": [],
"commit_count": 2,
"head": "4df35777e3afd6f5b6a9aecfe22793e65bba83ea",
"head": "00f746c24feb4ecc943c59cbe8006db2a01e9cd3",
"index_tree": "715ef280db9d5ab81405996e12aa2be15877d964",
"refs": [
"refs/heads/master"
Expand Down
2 changes: 1 addition & 1 deletion capabilities/C01/evaluation.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,5 +3,5 @@
"expected_met": true,
"result": "REPRODUCED",
"schema_version": "1.0",
"summary": "Sanitized trial evidence demonstrates repository skill discovery both explicitly and from a nested working directory."
"summary": "Sanitized trial evidence demonstrates both repository skill discovery and discovery from a nested working directory."
}
6 changes: 3 additions & 3 deletions capabilities/C01/hashes.json
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
{
"algorithm": "sha256",
"files": {
"README.md": "422e8b7912cf13e186c621d5eeae747575e5389e5b135b157596ee4201819565",
"actual.sanitized.json": "9042d7e35a10682bff746c4313193baf4d1947dfd814853ca03f78555825aba9",
"README.md": "f0379083ccec22707ec999a82f6cda7971024e970b17f2276d74b478340bde81",
"actual.sanitized.json": "9efbd720ac6e947272ef1b998ad3b51c36b1b3ab3cd652ffab368a456bc443ba",
"config/README.md": "2b6beca33571ad108adcdbac9becc82a7fe658cf672305f01f58f4ea54565b4e",
"evaluation.json": "12aeecb91deeaef38f633d67f60451a07ba176539a5430cfc16c82c8bafa7ee5",
"evaluation.json": "46bf08a3331e054e3dccd8a2822e6526994e85b7de2bfd4f04e61a7a2c1cd62e",
"expected.json": "c41329fbf6892a65b85cfe1166e86d2a3b59b684a9a81601974fca0c54a0cee7",
"fixture/.agents/skills/fixture-capability/SKILL.md": "0c4c7985ed4a06a2fa45d0d6a1887fec0f5a59892ae1d0d88317ac1c82a7a4ff",
"fixture/README.md": "a3ac65c07ebf67442b31cf3bc1d28ac5c849bd08fee9cba93b0a31339e7602d8",
Expand Down
4 changes: 2 additions & 2 deletions capabilities/C02/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,11 @@ Use `fixture/`, `config/README.md`, `prompt.txt`, and `run-command.txt`. Replace

## Live qualification

- Date: `2026-09-06`
- Date: `2026-09-07`
- Codex: `codex-cli 0.153.4`
- Model: `gpt-5.6-sol`
- OS: `Debian GNU/Linux 13 (trixie)`
- Permission mode: `approval=never; sandbox=per-trial; model-tool network disabled`
- Project trust: `trusted via CLI override for disposable fixture repositories`
- Source commit: `d0384f76bc4150d33bb8f51ef5981f3243b3cfb3`
- Source commit: `a9cdcdc1e0cad70e88b60869e75e4166046dd306`
- Result: `REPRODUCED`
24 changes: 12 additions & 12 deletions capabilities/C02/actual.sanitized.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,25 +2,25 @@
"capability_id": "C02",
"environment": {
"codex_version": "codex-cli 0.153.4",
"fixture_commit": "b006122b504e9bdf38fa9d0a21c35ab866fba584",
"fixture_commit": "ac01932c34c0e42bb7a84fe986452f011ffae50f",
"model": "gpt-5.6-sol",
"os": "Debian GNU/Linux 13 (trixie)",
"permission_mode": "approval=never; sandbox=per-trial; model-tool network disabled",
"project_trust": "trusted via CLI override for disposable fixture repositories",
"source_commit": "d0384f76bc4150d33bb8f51ef5981f3243b3cfb3"
"source_commit": "a9cdcdc1e0cad70e88b60869e75e4166046dd306"
},
"observations": [
"Implicit-activation trial completed and passed: a semantically matching prompt without the skill name produced no skill-specific activation token.",
"Explicit-activation trial completed and passed: the explicit $fixture-capability prompt supplied the expected FIXTURE_SKILL_ACTIVE token.",
"Both material assertions in the expected specification were directly exercised; neither trial reported a blocker or runtime error."
"The completed implicit-invocation trial used a semantically matching request without the skill name and observed no skill-specific activation token.",
"The completed explicit-invocation trial used $fixture-capability and observed the required literal proof token FIXTURE_SKILL_ACTIVE.",
"Both expected assertions are individually marked PASS; both trials report PASS with no blocker or error events."
],
"result": "REPRODUCED",
"schema_version": "1.0",
"trials": [
{
"assertions": [
{
"evidence": "The semantic match did not place any skill-specific activation token in the active instructions.",
"evidence": "The semantically matching request produced no skill-specific activation token in the active instructions.",
"name": "A semantically matching prompt without the skill name does not activate the fixture skill.",
"status": "PASS"
}
Expand Down Expand Up @@ -48,7 +48,7 @@
"branch": "master",
"cached_paths": [],
"commit_count": 2,
"head": "b006122b504e9bdf38fa9d0a21c35ab866fba584",
"head": "ac01932c34c0e42bb7a84fe986452f011ffae50f",
"index_tree": "6455a6dfb42207be3ad780cca165fa8f60e78d57",
"refs": [
"refs/heads/master"
Expand All @@ -60,7 +60,7 @@
"branch": "master",
"cached_paths": [],
"commit_count": 2,
"head": "b006122b504e9bdf38fa9d0a21c35ab866fba584",
"head": "ac01932c34c0e42bb7a84fe986452f011ffae50f",
"index_tree": "6455a6dfb42207be3ad780cca165fa8f60e78d57",
"refs": [
"refs/heads/master"
Expand All @@ -84,7 +84,7 @@
{
"assertions": [
{
"evidence": "Activated skill context required and supplied `FIXTURE_SKILL_ACTIVE`.",
"evidence": "Activated skill context required and supplied the literal proof token: FIXTURE_SKILL_ACTIVE",
"name": "An explicit $fixture-capability prompt activates the fixture skill.",
"status": "PASS"
}
Expand Down Expand Up @@ -112,7 +112,7 @@
"branch": "master",
"cached_paths": [],
"commit_count": 2,
"head": "b006122b504e9bdf38fa9d0a21c35ab866fba584",
"head": "ac01932c34c0e42bb7a84fe986452f011ffae50f",
"index_tree": "6455a6dfb42207be3ad780cca165fa8f60e78d57",
"refs": [
"refs/heads/master"
Expand All @@ -124,7 +124,7 @@
"branch": "master",
"cached_paths": [],
"commit_count": 2,
"head": "b006122b504e9bdf38fa9d0a21c35ab866fba584",
"head": "ac01932c34c0e42bb7a84fe986452f011ffae50f",
"index_tree": "6455a6dfb42207be3ad780cca165fa8f60e78d57",
"refs": [
"refs/heads/master"
Expand All @@ -133,7 +133,7 @@
"worktree_count": 1
},
"observations": [
"FIXTURE_SKILL_ACTIVE"
"Explicit invocation activated fixture-capability; the skill file was not directly inspected and no files were modified."
],
"outcome": "PASS",
"outer_skill_policy": {
Expand Down
2 changes: 1 addition & 1 deletion capabilities/C02/evaluation.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,5 +3,5 @@
"expected_met": true,
"result": "REPRODUCED",
"schema_version": "1.0",
"summary": "Sanitized trial evidence demonstrates that the fixture skill remains inactive for an unnamed semantic match and activates when explicitly named."
"summary": "Sanitized trial evidence demonstrates both explicit-only activation assertions."
}
6 changes: 3 additions & 3 deletions capabilities/C02/hashes.json
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
{
"algorithm": "sha256",
"files": {
"README.md": "d48f15a8b53baa0aa7ee54ee3c1b56ae44747159f9579963461180277f052027",
"actual.sanitized.json": "84a176fed4880e6a59bb18f46d9c24962b6f371ad6de8d9458da75deaa488923",
"README.md": "0eb2ce77906499f392ee20894bba2939ac41e4b968f25d16caf06dd22af4eeea",
"actual.sanitized.json": "66acfd0d0ce95591dd9beddd138d4d9066e597b7a47c5f9f3e9888c2b5a3bcc7",
"config/README.md": "22255f099cfe99dea718801c9f78b2230d229a560ba02f1f048e35f9ca765780",
"evaluation.json": "a1bb7d9496bd9daba9a07f296b1ce42acc133a91d36c9f8c7aa81fb4c329f548",
"evaluation.json": "e004a439026664fa6fa715c15603e012ca343f3685091ceb3b7381da9b87c4d7",
"expected.json": "fe22f0a70480c8c1b932e67af607ca5e3f539512162dd22b903001a3b215d0b8",
"fixture/.agents/skills/fixture-capability/SKILL.md": "35e41a201965e0a9c83ce9dbcf151275a7ad5718edd63c21976e0def24204cd5",
"fixture/.agents/skills/fixture-capability/agents/openai.yaml": "5f4b205f412e2e168092f152801b7dc72bc00e7716147305ebc5fa6bfabe0087",
Expand Down
4 changes: 2 additions & 2 deletions capabilities/C03/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,11 @@ Use `fixture/`, `config/README.md`, `prompt.txt`, and `run-command.txt`. Replace

## Live qualification

- Date: `2026-09-06`
- Date: `2026-09-07`
- Codex: `codex-cli 0.153.4`
- Model: `gpt-5.6-sol`
- OS: `Debian GNU/Linux 13 (trixie)`
- Permission mode: `approval=never; sandbox=per-trial; model-tool network disabled`
- Project trust: `trusted via CLI override for disposable fixture repositories`
- Source commit: `d0384f76bc4150d33bb8f51ef5981f3243b3cfb3`
- Source commit: `a9cdcdc1e0cad70e88b60869e75e4166046dd306`
- Result: `REPRODUCED`
Loading