Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions .github/workflows/plananvil-codex-qualification.yml
Original file line number Diff line number Diff line change
Expand Up @@ -136,12 +136,15 @@ jobs:
run: |
set -euo pipefail
cd "${QUALIFICATION_REPO}"
qualification_args=()
# Baseline 2.3 permits the narrowly gated C13 fallback in both full and
# C13-only modes. The harness still tries ephemeral first and ignores
# this permission unless the recognized parent-thread failure occurs.
qualification_args=(--allow-c13-non-ephemeral-fallback)
if [ "${{ inputs.mode }}" = "c13" ]; then
qualification_args+=(--only C13 --allow-c13-non-ephemeral-fallback)
qualification_args+=(--only C13)
fi
set +e
python3 tools/live_codex_qualification_harness_v5.py \
python3 tools/live_codex_qualification_harness_v6.py \
--root "${QUALIFICATION_REPO}" \
--source-commit "${GITHUB_SHA}" \
--run-id "${GITHUB_RUN_ID}" \
Expand Down
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ All notable changes to PlanAnvil are documented here.
- make C12 a deterministic runtime byte-budget probe with redundant `project_doc_max_bytes` enforcement, secret head/tail markers, zero-tool automatic-loading evidence, and outer PlanAnvil full-file hash verification;
- make C06, C08, and C09 deterministic live probes using the real PlanAnvil PreToolUse/PreCompact/PostCompact hooks, explicit postcondition evidence, low-limit `body_after_prefix` auto-compaction triggers, checkpoint repair, repeated compaction, and post-second-compaction continuation checks;
- add a diagnostic-only C13 transport mode that keeps `full` strict-ephemeral, retries non-ephemeral only for the recognized parent-thread registration failure, uses an isolated disposable `CODEX_HOME`, requires cleanup/auth-metadata invariants, and verifies real SubagentStart semantics with an outer-generated context proof before any baseline 2.3 decision;
- promote C13 qualification to baseline 2.3: full qualification remains ephemeral-first but may use a known-error-gated non-ephemeral retry with a synthetic home-scoped `fixture_agent` in disposable `CODEX_HOME`, while the real `SubagentStart` hook remains project-scoped and cleanup/auth invariants remain fail-closed;
- require production releases to use a GitHub-verified signed annotated tag whose target is reachable from `main`;
- fail the production release gate closed when the release worktree is dirty or Git cleanliness cannot be verified;
- document the controlled self-hosted Codex qualification path and keep the previous sandbox procedure as a manual fallback.
Expand Down
8 changes: 5 additions & 3 deletions capabilities/C13/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,11 @@
- Source: `DOCUMENTED`
- Release-gating: `yes`
- Current result: `BLOCKED`
- Qualification attempt: `2026-08-28`
- Qualification attempt: `2026-09-02`
- Documentation check: `PASS` against current Codex hooks documentation.
- Deterministic support: hook code compiles and the full suite passed in run #24.
- Live blocker: no authenticated Codex runtime is available to spawn a subagent and capture the event.
- Baseline: `2.3`.
- Qualification transport: ephemeral-first; only the recognized parent-thread registration failure permits a controlled non-ephemeral retry with a synthetic home-scoped `fixture_agent` in a disposable `CODEX_HOME`, while the real `SubagentStart` hook remains project-scoped.
- Latest diagnostic: run #8 confirmed the ephemeral parent-thread blocker and separately showed that a project-scoped synthetic agent can fail before `SubagentStart`; neither observation counts as semantic reproduction.
- Live blocker: a new full baseline-2.3 run must reach the real `SubagentStart` boundary and verify `additionalContext`, `continue=false`, child context echo, repository immutability, session cleanup, and auth-metadata invariants.

Do not change the result to `REPRODUCED` until the complete sanitized live package exists.
15 changes: 7 additions & 8 deletions capabilities/index.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"baseline_version": "2.2",
"baseline_version": "2.3",
"capabilities": [
{"evidence_directory":"capabilities/C01","expected_behavior":"Repository skills are discovered from .agents/skills","id":"C01","required":true,"result":"BLOCKED","source":"DOCUMENTED"},
{"evidence_directory":"capabilities/C02","expected_behavior":"allow_implicit_invocation false disables implicit invocation while explicit invocation remains available","id":"C02","required":true,"result":"BLOCKED","source":"DOCUMENTED"},
Expand All @@ -19,14 +19,13 @@
{"evidence_directory":"capabilities/C16","expected_behavior":"Git probe reports refs, branches, worktrees, index, commits, and cleanup","id":"C16","required":true,"result":"BLOCKED","source":"CONTRACT_DEFINED"}
],
"evidence_package_state": "TEMPLATE_ARCHIVE_READY",
"generated_at": "2026-08-28",
"generated_at": "2026-09-02",
"qualification_attempt": {
"date": "2026-08-28",
"deterministic_ci_commit": "0e1091fc4168671215ebcd89c92db702ea33ccdd",
"deterministic_ci_run": 26,
"deterministic_ci_result": "PASS",
"live_codex_result": "BLOCKED",
"blocker": "No authenticated Codex runtime or codex executable is available in the qualification execution environment."
"date": "2026-09-02",
"diagnostic_run": 8,
"diagnostic_source_commit": "ebc8cb119434df746b64ffaba3201d0d0ca4c6cd",
"live_codex_result": "PARTIAL",
"blocker": "C13 run #8 separated the ephemeral parent-thread registration blocker and project-scoped synthetic-agent spawn limitation from the SubagentStart semantic assertion; baseline 2.3 prepares the controlled full qualification transport but does not mark C13 REPRODUCED."
},
"release_gate": "Required capabilities must have committed sanitized REPRODUCED evidence before production readiness."
}
34 changes: 27 additions & 7 deletions docs/CODEX_CAPABILITY_BASELINE.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# PlanAnvil — Codex Capability Baseline

> **Baseline version:** 2.2
> **Review date:** 2026-08-28
> **Baseline version:** 2.3
> **Review date:** 2026-09-02
> **Purpose:** define current expected Codex behavior and reproducible release tests.
> **Authority:** current official OpenAI documentation has precedence.

Expand Down Expand Up @@ -66,16 +66,16 @@ Do not commit session transcripts, credentials, private paths, unrelated Git dat
| C10 | `PostCompact` and `SessionStart` can provide recovery context | DOCUMENTED | BLOCKED | Inject only a recovery pointer |
| C11 | Project instructions follow documented directory scope and precedence | DOCUMENTED | BLOCKED | Explicitly map affected instructions |
| C12 | `project_doc_max_bytes` can truncate automatic instruction loading | DOCUMENTED | BLOCKED | Read, size and hash complete files explicitly |
| C13 | `SubagentStart` can add context but `continue: false` does not stop subagent startup | DOCUMENTED | BLOCKED | Context and audit only |
| C13 | `SubagentStart` can add context but `continue: false` does not stop subagent startup | DOCUMENTED | BLOCKED | Context/audit only; qualify ephemeral-first with a controlled home-scoped fallback when the recognized ephemeral parent-thread blocker occurs |
| C14 | Planning isolation preserves the source branch, SHA, index and files | CONTRACT_DEFINED | BLOCKED | Planning worktree isolation is mandatory |
| C15 | Blind review is immutable and detects seeded contract defects | CONTRACT_DEFINED | BLOCKED | Hash review before separate comparison |
| C16 | The Git probe accurately reports refs, branches, worktrees, index, commits and cleanup | CONTRACT_DEFINED | BLOCKED | No artifact generation before required Git capabilities pass |

## 4. Release gate

C01, C02, C03 and C05 through C16 must be `REPRODUCED` before production readiness. C04 is informational for PlanAnvil 2.2 because generated execution deliberately forbids nested descendants.
C01, C02, C03 and C05 through C16 must be `REPRODUCED` before production readiness. C04 is informational for PlanAnvil 2.3 because generated execution deliberately forbids nested descendants.

The 2026-08-28 qualification attempt is `BLOCKED` for live Codex reproduction because the available execution environment does not expose an authenticated Codex runtime or `codex` executable. Deterministic contract tests remain useful supporting evidence but cannot substitute for the required live packages.
Baseline 2.3 does not mark C13 reproduced from the 2026-09-02 transport diagnostic. That run separated two runtime limitations from the semantic assertion: `codex exec --ephemeral` reproduced the known parent-thread registration failure before `SubagentStart`, while a non-ephemeral attempt progressed past that failure but a project-scoped synthetic custom agent still did not reach `SubagentStart`. The next release-gating run must still obtain real semantic evidence.

## 5. Test requirements

Expand All @@ -87,6 +87,24 @@ Verify nested-directory discovery, explicit activation, disabled implicit activa

Use current documented agent configuration (`agents.enabled` and `agents.max_concurrent_threads_per_session`). Record the event tree for required reviewer/profiler dispatch. Do not rely on undocumented nesting-depth configuration. Separately assert that generated execution contracts require a flat direct-child topology.

### C13 SubagentStart qualification transport

The semantic assertion under test is the documented `SubagentStart` behavior, not `codex exec --ephemeral` persistence and not project-scoped custom-agent discovery.

C13 therefore uses this fail-closed transport contract:

1. start with a fresh real `codex exec --ephemeral` trial using an aligned project-scoped synthetic agent (`fixture_agent.toml`, declared name `fixture_agent`) and a real project-scoped `SubagentStart` hook;
2. if that trial reaches `SubagentStart`, evaluate the semantics directly and do not use a fallback;
3. permit a non-ephemeral retry only when the ephemeral attempt matches the recognized `collab spawn failed: no thread with id` parent-thread registration failure before `SubagentStart`;
4. for that retry, create a separate disposable repository containing the project-scoped hook/config but no project-scoped custom agent;
5. materialize the synthetic `fixture_agent` only under a private disposable `CODEX_HOME/agents/fixture_agent.toml`, retaining the real project-scoped `SubagentStart` hook as the semantic boundary under test;
6. keep approval `never`, C13 sandbox `read-only`, model-tool network disabled and project trust limited to the disposable fixture;
7. bridge file-backed authentication only through a temporary symlink, never read or copy the credential file, isolate SQLite/log state, disable message-history persistence, then remove the complete disposable `CODEX_HOME` and verify auth metadata is unchanged;
8. require exactly one real `SubagentStart`, `additionalContext` from that hook, `continue=false` from the same hook, and a child echo of an opaque proof that was not present in the root-agent prompt;
9. classify missing transport/discovery evidence as `BLOCKED`, and classify contradictory behavior after the real `SubagentStart` boundary is reached as `FAILED`.

The fallback is a qualification transport exception only. It does not make home-scoped custom agents a PlanAnvil product requirement and it does not weaken sandbox, approval, trust, network or evidence-sanitization boundaries.

### File handoff

Verify explicit review/profile brief paths, expected hashes and rejection of missing, stale or escaped paths.
Expand Down Expand Up @@ -115,15 +133,17 @@ Use seeded defects including missing rollback, uncovered requirements, risks wit

Record separate outcomes for ordinary file writes, temporary refs, branches, linked worktrees, index updates, commits, signing, repository hooks and cleanup under every supported permission mode.

## 6. Current official documentation checked on 2026-08-28
## 6. Current official documentation checked on 2026-09-02

- Skills: `https://developers.openai.com/codex/skills`
- Configuration reference: `https://developers.openai.com/codex/config-reference`
- Subagents: `https://developers.openai.com/codex/subagents`
- Hooks: `https://developers.openai.com/codex/hooks`
- AGENTS.md: `https://developers.openai.com/codex/guides/agents-md`

Notable drift from baseline 2.1: current subagent documentation exposes `agents.enabled` and `agents.max_concurrent_threads_per_session`; it does not document `agents.max_depth`. `agents.max_threads` is retained only as a legacy concurrency alias. PlanAnvil therefore enforces flat topology in its generated execution contract instead of relying on a runtime depth setting.
Baseline 2.3 keeps the semantic capability matrix from 2.2 and changes only the C13 qualification transport contract. The change is motivated by controlled live observations plus upstream runtime reports for ephemeral parent-thread registration and project-scoped custom-agent spawning; those reports are diagnostic evidence, not normative sources for the expected `SubagentStart` semantics.

The earlier 2.2 subagent decision remains: current subagent documentation exposes `agents.enabled` and `agents.max_concurrent_threads_per_session`; it does not document `agents.max_depth`. PlanAnvil therefore enforces flat topology in its generated execution contract instead of relying on a runtime depth setting.

## 7. Architecture rule

Expand Down
Loading