Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/plananvil-codex-qualification.yml
Original file line number Diff line number Diff line change
Expand Up @@ -326,7 +326,7 @@ jobs:
qualification_args+=(--only C13)
fi
set +e
python3 tools/live_codex_qualification_harness_v6.py \
python3 tools/live_codex_qualification_harness_v7.py \
--root "${QUALIFICATION_REPO}" \
--source-commit "${GITHUB_SHA}" \
--run-id "${GITHUB_RUN_ID}" \
Expand Down
20 changes: 10 additions & 10 deletions docs/CODEX_CAPABILITY_BASELINE.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ Do not commit session transcripts, credentials, private paths, unrelated Git dat
| C10 | `PostCompact` and `SessionStart` can provide recovery context | DOCUMENTED | BLOCKED | Inject only a recovery pointer |
| C11 | Project instructions follow documented directory scope and precedence | DOCUMENTED | BLOCKED | Explicitly map affected instructions |
| C12 | `project_doc_max_bytes` can truncate automatic instruction loading | DOCUMENTED | BLOCKED | Read, size and hash complete files explicitly |
| C13 | `SubagentStart` can add context but `continue: false` does not stop subagent startup | DOCUMENTED | BLOCKED | Context/audit only; qualify ephemeral-first with a controlled home-scoped fallback when the recognized ephemeral parent-thread blocker occurs |
| C13 | `SubagentStart` can add context but `continue: false` does not stop subagent startup | DOCUMENTED | BLOCKED | Context/audit only; qualify ephemeral-first with a controlled project-scoped non-ephemeral fallback when the recognized ephemeral parent-thread blocker occurs |
| C14 | Planning isolation preserves the source branch, SHA, index and files | CONTRACT_DEFINED | BLOCKED | Planning worktree isolation is mandatory |
| C15 | Blind review is immutable and detects seeded contract defects | CONTRACT_DEFINED | BLOCKED | Hash review before separate comparison |
| C16 | The Git probe accurately reports refs, branches, worktrees, index, commits and cleanup | CONTRACT_DEFINED | BLOCKED | No artifact generation before required Git capabilities pass |
Expand All @@ -75,7 +75,7 @@ Do not commit session transcripts, credentials, private paths, unrelated Git dat

C01, C02, C03 and C05 through C16 must be `REPRODUCED` before production readiness. C04 is informational for PlanAnvil 2.3 because generated execution deliberately forbids nested descendants.

Baseline 2.3 does not mark C13 reproduced from the 2026-09-02 transport diagnostic. That run separated two runtime limitations from the semantic assertion: `codex exec --ephemeral` reproduced the known parent-thread registration failure before `SubagentStart`, while a non-ephemeral attempt progressed past that failure but a project-scoped synthetic custom agent still did not reach `SubagentStart`. The next release-gating run must still obtain real semantic evidence.
Baseline 2.3 does not infer C13 reproduction from transport diagnostics. Controlled Codex 0.152 runs established that `codex exec --ephemeral` can hit a parent-thread registration failure even when project roles are otherwise valid, while non-ephemeral project-agent execution can progress normally. The release-gating fallback must therefore keep both the synthetic role and the `SubagentStart` hook project-scoped and use an isolated user home only for trust, authentication bridging, and disposable persistence.

## 5. Test requirements

Expand All @@ -89,21 +89,21 @@ Use current documented agent configuration (`agents.enabled` and `agents.max_con

### C13 SubagentStart qualification transport

The semantic assertion under test is the documented `SubagentStart` behavior, not `codex exec --ephemeral` persistence and not project-scoped custom-agent discovery.
The semantic assertion under test is the documented `SubagentStart` behavior, not `codex exec --ephemeral` persistence.

C13 therefore uses this fail-closed transport contract:

1. start with a fresh real `codex exec --ephemeral` trial using an aligned project-scoped synthetic agent (`fixture_agent.toml`, declared name `fixture_agent`) and a real project-scoped `SubagentStart` hook;
2. if that trial reaches `SubagentStart`, evaluate the semantics directly and do not use a fallback;
3. permit a non-ephemeral retry only when the ephemeral attempt matches the recognized `collab spawn failed: no thread with id` parent-thread registration failure before `SubagentStart`;
4. for that retry, create a separate disposable repository containing the project-scoped hook/config but no project-scoped custom agent;
5. materialize the synthetic `fixture_agent` only under a private disposable `CODEX_HOME/agents/fixture_agent.toml`, retaining the real project-scoped `SubagentStart` hook as the semantic boundary under test;
2. if that trial reaches `SubagentStart` without the recognized transport failure, evaluate the semantics directly and do not use a fallback;
3. permit a non-ephemeral retry only when the ephemeral attempt matches the recognized `collab spawn failed: no thread with id` parent-thread registration failure;
4. for that retry, create a separate disposable repository that keeps the synthetic `fixture_agent` project-scoped, explicitly declares `[agents.fixture_agent]`, and keeps the real `SubagentStart` hook project-scoped;
5. use a private disposable `CODEX_HOME` only to persist the fixture trust decision, bridge file-backed authentication, and isolate non-ephemeral session/SQLite/log state; do not move the agent or hook into the home layer;
6. keep approval `never`, C13 sandbox `read-only`, model-tool network disabled and project trust limited to the disposable fixture;
7. bridge file-backed authentication only through a temporary symlink, never read or copy the credential file, isolate SQLite/log state, disable message-history persistence, then remove the complete disposable `CODEX_HOME` and verify auth metadata is unchanged;
8. require exactly one real `SubagentStart`, `additionalContext` from that hook, `continue=false` from the same hook, and a child echo of an opaque proof that was not present in the root-agent prompt;
8. require exactly one real project-scoped `SubagentStart`, `additionalContext` from that hook, `continue=false` from the same hook, and a child echo of an opaque proof that was not present in the root-agent prompt;
9. classify missing transport/discovery evidence as `BLOCKED`, and classify contradictory behavior after the real `SubagentStart` boundary is reached as `FAILED`.

The fallback is a qualification transport exception only. It does not make home-scoped custom agents a PlanAnvil product requirement and it does not weaken sandbox, approval, trust, network or evidence-sanitization boundaries.
The fallback is a qualification transport exception only. It does not change PlanAnvil's project-scoped agent/hook product contract and it does not weaken sandbox, approval, trust, network or evidence-sanitization boundaries.

### File handoff

Expand Down Expand Up @@ -141,7 +141,7 @@ Record separate outcomes for ordinary file writes, temporary refs, branches, lin
- Hooks: `https://developers.openai.com/codex/hooks`
- AGENTS.md: `https://developers.openai.com/codex/guides/agents-md`

Baseline 2.3 keeps the semantic capability matrix from 2.2 and changes only the C13 qualification transport contract. The change is motivated by controlled live observations plus upstream runtime reports for ephemeral parent-thread registration and project-scoped custom-agent spawning; those reports are diagnostic evidence, not normative sources for the expected `SubagentStart` semantics.
Baseline 2.3 keeps the semantic capability matrix from 2.2 and changes only the C13 qualification transport contract. The transport remains ephemeral-first, but a recognized Codex 0.152 parent-thread registration failure may be retried non-ephemerally without changing the project-scoped role/hook semantics under test.

The earlier 2.2 subagent decision remains: current subagent documentation exposes `agents.enabled` and `agents.max_concurrent_threads_per_session`; it does not document `agents.max_depth`. PlanAnvil therefore enforces flat topology in its generated execution contract instead of relying on a runtime depth setting.

Expand Down
18 changes: 10 additions & 8 deletions docs/CODEX_SANDBOX_RUNBOOK.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,17 +29,19 @@ The preferred qualification path is `.github/workflows/plananvil-codex-qualifica

Use `mode=full` for the release-gating C01-C16 sequence. `mode=c13` remains available as a shorter C13-only probe, but it is not a substitute for the full release gate.

The controlled runner must provide `plananvil-qualification-workspace`. The workflow creates a disposable workspace with that helper, fetches only the exact dispatched `main` SHA, materializes the C01-C16 evidence templates, and runs `tools/live_codex_qualification_harness_v6.py`. Model `gpt-5.6-sol` is pinned, approval policy remains `never`, model-tool network access is disabled, and `workspace-write` is granted only to disposable fixture roots when a trial requires it. Vetted project hooks may bypass only the interactive hook-trust prompt; approval and filesystem sandboxing remain enabled.
The controlled runner must provide `plananvil-qualification-workspace`. The workflow creates a disposable workspace with that helper, fetches only the exact dispatched `main` SHA, materializes the C01-C16 evidence templates, and runs `tools/live_codex_qualification_harness_v7.py`. Model `gpt-5.6-sol` is pinned, approval policy remains `never`, model-tool network access is disabled, and `workspace-write` is granted only to disposable fixture roots when a trial requires it. Vetted project hooks may bypass only the interactive hook-trust prompt; approval and filesystem sandboxing remain enabled.

All normal agent tasks remain ephemeral. Baseline 2.3 introduces exactly one transport exception for C13: the harness may retry C13 non-ephemerally only when the first real ephemeral attempt fails before `SubagentStart` with the recognized `collab spawn failed: no thread with id` parent-thread registration error. Any other ephemeral blocker remains `BLOCKED` and does not activate the exception.
For C08/C09, Codex 0.152 project trust remains a persisted user-config setting, but long full runs must use the runner's real `CODEX_HOME` so the CLI can refresh live authentication normally. The harness temporarily appends only the disposable fixture trust entry to the runner's `config.toml`, removes the invalid CLI trust path, and restores `config.toml` byte-for-byte after the capability. Authentication/session files are not copied or replaced by this trust bridge.

All normal agent tasks remain ephemeral. Baseline 2.3 introduces exactly one transport exception for C13: the harness may retry C13 non-ephemerally only when the first real ephemeral attempt matches the recognized `collab spawn failed: no thread with id` parent-thread registration error. Any other ephemeral blocker remains `BLOCKED` and does not activate the exception.

### C13 baseline 2.3 transport

C13 tests the documented `SubagentStart` semantics, not persistence of `codex exec --ephemeral` and not project-scoped custom-agent discovery.
C13 tests the documented `SubagentStart` semantics, not persistence of `codex exec --ephemeral`.

The first C13 attempt is still a real project-scoped configuration. The synthetic agent file is `.codex/agents/fixture_agent.toml`, its declared name is `fixture_agent`, the prompt requests `fixture_agent`, and the project-scoped `SubagentStart` matcher targets the same name. The hook injects an outer-generated opaque context proof and intentionally returns `continue=false`.
The first C13 attempt is a real project-scoped configuration. The project-scoped synthetic agent file is `.codex/agents/fixture_agent.toml`, `[agents.fixture_agent]` points to that file, the prompt requests `fixture_agent`, and the project-scoped `SubagentStart` matcher targets the same name. The hook injects an outer-generated opaque context proof and intentionally returns `continue=false`.

If and only if that ephemeral attempt hits the recognized parent-thread registration failure before the hook, the harness creates a second disposable Git repository. That repository contains the project-scoped C13 hook/config but deliberately contains **no** `.codex/agents` custom agent. The synthetic `fixture_agent` is instead materialized under a private disposable `CODEX_HOME/agents/fixture_agent.toml`. This separates the `SubagentStart` semantic assertion from the independently observed project-scoped custom-agent discovery/spawn limitation while keeping the hook under test project-scoped.
If and only if that ephemeral attempt hits the recognized parent-thread registration failure, the harness creates a second disposable Git repository and retries non-ephemerally. The fallback remains product-aligned: the project-scoped synthetic agent is explicitly declared as `[agents.fixture_agent]`, and the real `SubagentStart` hook remains project-scoped. The disposable `CODEX_HOME` is used only for the fixture trust decision, file-backed authentication bridge, and isolated non-ephemeral persistence.

The non-ephemeral retry preserves all security boundaries:

Expand All @@ -55,9 +57,9 @@ The non-ephemeral retry preserves all security boundaries:
- the authenticated source `auth.json` metadata must remain unchanged;
- evidence retains only structural counts/booleans and never session/thread IDs or the opaque proof value.

C13 is `REPRODUCED` only when exactly one real project-scoped `SubagentStart` hook event occurs, that hook returns both `additionalContext` and `continue=false`, and the real child returns the unseen injected proof. If the semantic boundary is reached but the child lacks the context or startup is stopped by `continue=false`, the result is `FAILED`. If the semantic boundary is not reached or cleanup/auth isolation cannot be proved, the result is `BLOCKED`.
C13 is `REPRODUCED` only when exactly one real project-scoped `SubagentStart` hook event occurs, that hook returns both `additionalContext` and `continue=false`, and the real project-scoped child returns the unseen injected proof. If the semantic boundary is reached but the child lacks the context or startup is stopped by `continue=false`, the result is `FAILED`. If the semantic boundary is not reached or cleanup/auth isolation cannot be proved, the result is `BLOCKED`.

The run #8 diagnostic on Codex 0.152.0 established the reason for this baseline change: ephemeral execution reproduced the parent-thread registration blocker, while a non-ephemeral retry progressed further but the project-scoped synthetic agent still failed before `SubagentStart`. Baseline 2.3 changes the qualification transport only; it does not count that diagnostic as C13 semantic reproduction.
The transport correction is deliberately narrow. It does not move PlanAnvil roles or hooks into user configuration, does not treat a home-scoped synthetic role as product-equivalent, and does not weaken sandbox, approval, trust, network, source-immutability, or evidence-sanitization requirements.

## Evidence and sanitization

Expand All @@ -82,7 +84,7 @@ python tools/validate_capabilities.py
The release-gating controller invocation must include the baseline 2.3 C13 transport permission:

```text
python tools/live_codex_qualification_harness_v6.py \
python tools/live_codex_qualification_harness_v7.py \
--source-commit <FULL_MAIN_SHA> \
--run-id <RUN_ID> \
--output <SANITIZED_ARTIFACT_DIR> \
Expand Down
8 changes: 6 additions & 2 deletions tests/test_live_codex_qualification_harness_v4.py
Original file line number Diff line number Diff line change
Expand Up @@ -55,16 +55,20 @@ def test_safety_boundary_is_not_weakened(self) -> None:
self.assertIn('sandbox="read-only"', self.source)
self.assertIn('sandbox="workspace-write"', self.source)

def test_v4_is_chained_under_current_v6_wrapper(self) -> None:
def test_v4_is_chained_under_current_v7_wrapper(self) -> None:
v5 = (ROOT / "tools" / "live_codex_qualification_harness_v5.py").read_text(
encoding="utf-8"
)
v6 = (ROOT / "tools" / "live_codex_qualification_harness_v6.py").read_text(
encoding="utf-8"
)
v7 = (ROOT / "tools" / "live_codex_qualification_harness_v7.py").read_text(
encoding="utf-8"
)
self.assertIn("import live_codex_qualification_harness_v4 as prior", v5)
self.assertIn("import live_codex_qualification_harness_v5 as prior", v6)
self.assertIn("python3 tools/live_codex_qualification_harness_v6.py", self.workflow)
self.assertIn("import live_codex_qualification_harness_v6 as v6", v7)
self.assertIn("python3 tools/live_codex_qualification_harness_v7.py", self.workflow)


if __name__ == "__main__":
Expand Down
7 changes: 4 additions & 3 deletions tests/test_live_codex_qualification_harness_v5.py
Original file line number Diff line number Diff line change
Expand Up @@ -72,20 +72,21 @@ def test_safety_boundary_is_not_weakened(self) -> None:
self.assertIn("completed_file_change_items", self.source)
self.assertIn("repository_unchanged", self.source)

def test_current_workflow_uses_v6_and_keeps_c13_short_mode(self) -> None:
def test_current_workflow_uses_v7_and_keeps_c13_short_mode(self) -> None:
self.assertIn("- c13", self.workflow)
self.assertIn("inputs.mode == 'c13'", self.workflow)
self.assertIn("python3 tools/live_codex_qualification_harness_v6.py", self.workflow)
self.assertIn("python3 tools/live_codex_qualification_harness_v7.py", self.workflow)
self.assertIn("--only C13", self.workflow)
self.assertIn("--allow-c13-non-ephemeral-fallback", self.workflow)
self.assertIn("inputs.mode == 'full'", self.workflow)

def test_runbook_documents_baseline23_full_transport(self) -> None:
self.assertIn("mode=c13", self.runbook)
self.assertIn("baseline 2.3", self.runbook.lower())
self.assertIn("CODEX_HOME/agents/fixture_agent.toml", self.runbook)
self.assertIn("project-scoped synthetic agent", self.runbook)
self.assertIn("project-scoped", self.runbook)
self.assertIn("mode=full", self.runbook)
self.assertIn("live_codex_qualification_harness_v7.py", self.runbook)


if __name__ == "__main__":
Expand Down
Loading