Repair live qualification after regression audit

docs/CODEX_QUALIFICATION_REGRESSION_AUDIT_2026-09-03.md

@@ -0,0 +1,108 @@
+# Codex qualification regression audit — 2026-09-03
+
+## Scope
+
+This audit was completed before another qualification or CI run. It reviews the live-qualification controller chain, the full sanitized evidence from qualification run #9 (`33732852918`, source commit `7bf68e3fa8d6caff8ed8af1bf98f6be30d4670cd`), and the Codex 0.152.0 runtime contracts relevant to C03, C06, C08, C09, C13, and C16.
+
+The purpose is to remove harness-induced failure modes before spending another self-hosted Codex run.
+
+## Run #9 control result
+
+Run #9 completed the controller and uploaded a valid evidence artifact. It produced 10/16 `REPRODUCED` results. The six non-reproduced capabilities were C03, C06, C08, C09, C13, and C16.
+
+A capability is treated as a harness defect only when the evidence and controller establish that the test itself prevented or obscured the behavior being measured. Contradictory runtime behavior is not relabeled as a harness defect.
+
+## Findings
+
+### R1 — read-only hook telemetry can break the hook under test (C08, C09, C13)
+
+Severity: critical.
+
+The v4/v5 proxies execute the real hook, then write recorder JSONL under `.pursue/` inside the fixture repository, and only after that write forward the real hook stdout/stderr and return code to Codex. C08, C09, and C13 execute these paths with `sandbox=read-only`. A failed recorder write can therefore terminate the proxy before the real hook result reaches Codex.
+
+Repair: telemetry moves to a disposable external sidecar, recorder I/O is fail-open, and the real hook stdout/stderr/return code is always forwarded. Missing telemetry may block qualification but may never change hook semantics.
+
+### R2 — C08/C09 auto-compaction must be independent of recorder success
+
+Severity: high.
+
+Run #9 showed zero PreCompact/PostCompact records. Because R1 can suppress the real hook response, that absence does not distinguish “no compaction” from “compaction attempted but proxy failed”.
+
+Repair: fix R1 first, retain redundant project + CLI auto-compaction configuration, use deterministic low `body_after_prefix` thresholds, and require real sidecar PreCompact/PostCompact records. C09 still requires two complete compaction cycles and a real tool call after the second PostCompact.
+
+### R3 — C13 transport/discovery fallback works; remaining result is contaminated by R1
+
+Severity: high.
+
+Run #9 established that the baseline-2.3 non-ephemeral fallback creates a session and starts the home-scoped `fixture_agent`: the child returned `C13_CONTEXT_MISSING`. Parent-thread persistence and home-scoped agent discovery are therefore no longer the immediate blocker. The same trial recorded zero SubagentStart proxy events, but that proxy uses the R1 recorder pattern.
+
+Repair: retain ephemeral-first, the known-error-only fallback, the home-scoped fallback agent, and the project-scoped hook; move telemetry to the external fail-open sidecar; require exactly one real SubagentStart record, `continue=false`, `additionalContext`, child echo of the opaque outer proof, repository immutability, cleanup and auth-metadata invariants.
+
+If the repaired probe still starts the child without a SubagentStart record/context proof, that becomes runtime evidence rather than a recorder artifact.
+
+### R4 — C03 uses an implicit workspace Git repository whose `.git` remains protected
+
+Severity: high.
+
+C03 reached the real PlanAnvil bootstrap, but the mandatory reversible Git probe could not create a temporary ref. `workspace-write` does not make the implicit workspace `.git` metadata writable.
+
+Repair: use a command-driver repository and place the synthetic PlanAnvil source repository in an explicit auxiliary writable root. Run the real outer `plan_anvil.py start`, assert source branch/head/index/file preservation and cleanup, verify current agents configuration and flat direct-child execution contract, and calculate C03 deterministically in the outer harness rather than with a model evaluator.
+
+### R5 — C16 has successful real trials but the model evaluator discards diagnostic basis
+
+Severity: critical false-blocker.
+
+Run #9 produced PASS for `GIT_READY`, `GIT_SIGNING_BLOCKED`, and `GIT_HOOK_BLOCKED`, with empty snapshot changes and cleanup errors. The capability was nevertheless marked `BLOCKED` because the evaluator requested diagnostic basis that the sanitized trial payload omitted.
+
+The product probe already returns bounded per-step diagnostics in `steps[].detail` and the fixtures use deterministic markers.
+
+Repair: execute/parse the real probe in the outer harness, reduce controlled diagnostics to booleans such as `signing_diagnostic_observed` and `hook_diagnostic_observed` before sanitization, and calculate C16 deterministically. Raw stderr, private paths and session data are not retained.
+
+### R6 — C06 must not be converted to green without an isolated runtime repro
+
+Severity: high.
+
+Run #9 completed a real direct `apply_patch` file change while the PlanAnvil PreToolUse recorder observed zero `apply_patch` events. This trial used `workspace-write`, so R1 does not explain the result. Codex 0.152.0 defines a PreToolUse payload for direct `apply_patch`, and PlanAnvil's matcher includes `apply_patch`.
+
+Repair: retain the integrated PlanAnvil trial and add a second minimal repository containing only one project-scoped `PreToolUse` hook matching `^apply_patch$`. If the minimal hook fires but PlanAnvil does not, C06 remains `BLOCKED` as an integration defect. If even the minimal current-runtime hook does not fire after a successful direct patch, C06 is `FAILED`. Only both hook observations plus mandatory postcondition detection may produce `REPRODUCED`.
+
+### R7 — model planner/evaluator dependence remains in release-gating paths
+
+Severity: high.
+
+C03 still used the generic model-driven fixture planner/evaluator and C16 delegated its final decision to the evaluator. This creates avoidable non-determinism.
+
+Repair: C03 and C16 join C06/C08/C09/C13 as outer-deterministic release-gate decisions. Model output may supply live runtime observations but is not the sole arbiter when the outer harness can directly verify the assertion.
+
+### R8 — wrapper-chain growth increases regression risk
+
+Severity: medium.
+
+The controller already chains historical overrides through v1-v6. Adding v7 would make ownership harder to audit.
+
+Repair: keep `live_codex_qualification_harness_v6.py` as the workflow entrypoint. It explicitly owns C03, C06, C08, C09, C13, and C16 through one regression helper and delegates all other capabilities to the already-proven chain.
+
+### R9 — previously reproduced capabilities
+
+Run #9 evidence for C01, C02, C04, C05, C07, C10, C11, C12, C14, and C15 was reviewed for the same controller-induced false-green patterns. No release-gate assertion was found to depend on the failing recorder pattern.
+
+C10 contains a blocked PostCompact observation, but its required expected assertions are pointer/context recovery and reconstruction from canonical files/Git; the successful SessionStart trial directly demonstrates those assertions. No C10 change is included in this repair set.
+
+## Repair invariants
+
+1. qualification remains `workflow_dispatch` only from `main`;
+2. self-hosted runner labels and Environment remain unchanged;
+3. `approval=never` remains unchanged;
+4. model-tool network access remains disabled;
+5. no `danger-full-access`, approval/sandbox bypass, privileged container, or `SYS_ADMIN` is introduced;
+6. project-hook trust bypass may only bypass the interactive hook-trust prompt;
+7. every Git-mutating fixture is disposable and explicitly scoped;
+8. evidence stores structural booleans/counts/hashes, not transcripts, credentials, private paths, or session/thread identifiers;
+9. all six repaired capabilities use deterministic outer release-gate decisions;
+10. no full/self-hosted qualification is run until hosted regression tests for this repair set are green.
+
+## Exit criteria before the next full run
+
+The repair PR may be merged only if hosted CI verifies the consolidated v6 ownership/delegation contract, fail-open external hook telemetry, deterministic C03 auxiliary Git-root setup, C06 isolated PreToolUse comparison, deterministic C16 diagnostic basis, valid materialized C01-C16 packages, absence of forbidden sandbox/container flags, and the existing cross-platform core/distribution checks.
+
+Only after merge and green post-merge CI should a new `mode=full` qualification be dispatched.

tests/test_live_codex_qualification_harness_v6.py

@@ -6,6 +6,7 @@
 
 ROOT = Path(__file__).resolve().parents[1]
 MODULE_PATH = ROOT / "tools" / "live_codex_qualification_harness_v6.py"
+REGRESSION_PATH = ROOT / "tools" / "live_codex_qualification_regression.py"
 WORKFLOW_PATH = ROOT / ".github" / "workflows" / "plananvil-codex-qualification.yml"
 BASELINE_PATH = ROOT / "docs" / "CODEX_CAPABILITY_BASELINE.md"
 RUNBOOK_PATH = ROOT / "docs" / "CODEX_SANDBOX_RUNBOOK.md"
@@ -15,14 +16,19 @@ class LiveCodexHarnessV6Tests(unittest.TestCase):
     @classmethod
     def setUpClass(cls) -> None:
         cls.source = MODULE_PATH.read_text(encoding="utf-8")
+        cls.regression = REGRESSION_PATH.read_text(encoding="utf-8")
         cls.workflow = WORKFLOW_PATH.read_text(encoding="utf-8")
         cls.baseline = BASELINE_PATH.read_text(encoding="utf-8")
         cls.runbook = RUNBOOK_PATH.read_text(encoding="utf-8")
 
-    def test_v6_overrides_only_c13_and_inherits_v5(self) -> None:
-        self.assertIn('TARGET_CAPABILITIES = {"C13"}', self.source)
-        self.assertIn("import live_codex_qualification_harness_v5 as prior", self.source)
+    def test_v6_consolidates_exact_regression_audit_targets(self) -> None:
+        self.assertIn("import live_codex_qualification_regression as regression", self.source)
+        self.assertIn("TARGET_CAPABILITIES = regression.TARGET_CAPABILITIES", self.source)
+        self.assertIn('TARGET_CAPABILITIES = {"C03", "C06", "C08", "C09", "C13", "C16"}', self.regression)
         self.assertIn("_ORIGINAL_CAPABILITY_RUNTIME = prior.capability_runtime", self.source)
+        for capability_id in ("C03", "C06", "C08", "C09", "C13"):
+            self.assertIn(f'if capability_id == "{capability_id}"', self.source)
+        self.assertIn("return regression.run_c16(**common)", self.source)
 
     def test_agent_identity_is_aligned(self) -> None:
         self.assertIn('HOME_AGENT_NAME = "fixture_agent"', self.source)
@@ -41,6 +47,7 @@ def test_fallback_separates_agent_discovery_from_project_hook(self) -> None:
         self.assertIn('trial_n["agent_fixture_scope"] = "disposable_CODEX_HOME"', self.source)
         self.assertIn('trial_n["project_agent_present"] = False', self.source)
         self.assertIn('trial_n["project_scoped_subagent_start_hook"] = True', self.source)
+        self.assertIn("regression.run_c13(_c13_runtime", self.source)
 
     def test_fallback_is_still_known_error_gated(self) -> None:
         self.assertIn("known_e and ALLOW_NON_EPHEMERAL_FALLBACK", self.source)
@@ -54,13 +61,13 @@ def test_non_ephemeral_cleanup_and_auth_invariants_remain_required(self) -> None
         self.assertIn("auth_metadata_unchanged", self.source)
         self.assertIn("home_scoped_fixture_agent_materialized", self.source)
 
-    def test_full_workflow_enables_baseline23_fallback(self) -> None:
+    def test_full_workflow_stays_on_v6_and_enables_baseline23_fallback(self) -> None:
         self.assertIn("python3 tools/live_codex_qualification_harness_v6.py", self.workflow)
         self.assertIn("qualification_args=(--allow-c13-non-ephemeral-fallback)", self.workflow)
         self.assertIn("--only C13", self.workflow)
         self.assertIn("inputs.mode == 'full'", self.workflow)
 
-    def test_baseline_and_runbook_are_23(self) -> None:
+    def test_baseline_and_runbook_remain_23(self) -> None:
         self.assertIn("Baseline version:** 2.3", self.baseline)
         self.assertIn("ephemeral-first", self.baseline)
         self.assertIn("home-scoped", self.baseline)
@@ -69,12 +76,13 @@ def test_baseline_and_runbook_are_23(self) -> None:
         self.assertIn("project-scoped", self.runbook)
 
     def test_safety_boundary_is_not_weakened(self) -> None:
-        self.assertNotIn("--dangerously-bypass-approvals-and-sandbox", self.source)
-        self.assertNotIn("danger-full-access", self.source)
-        self.assertNotIn("--privileged", self.source)
-        self.assertNotIn("SYS_ADMIN", self.source)
+        combined = self.source + "\n" + self.regression
+        self.assertNotIn("--dangerously-bypass-approvals-and-sandbox", combined)
+        self.assertNotIn("danger-full-access", combined)
+        self.assertNotIn("--privileged", combined)
+        self.assertNotIn("SYS_ADMIN", combined)
         self.assertIn('sandbox_mode = "read-only"', self.source)
-        self.assertIn("base.git_snapshot", self.source)
+        self.assertIn("base.git_snapshot", combined)
 
 
 if __name__ == "__main__":

tests/test_live_codex_qualification_regression.py

@@ -0,0 +1,95 @@
+from __future__ import annotations
+
+from pathlib import Path
+import unittest
+
+
+ROOT = Path(__file__).resolve().parents[1]
+SOURCE_PATH = ROOT / "tools" / "live_codex_qualification_regression.py"
+AUDIT_PATH = ROOT / "docs" / "CODEX_QUALIFICATION_REGRESSION_AUDIT_2026-09-03.md"
+
+
+class QualificationRegressionAuditTests(unittest.TestCase):
+    @classmethod
+    def setUpClass(cls) -> None:
+        cls.source = SOURCE_PATH.read_text(encoding="utf-8")
+        cls.audit = AUDIT_PATH.read_text(encoding="utf-8")
+
+    def test_exact_audited_capability_set(self) -> None:
+        self.assertIn('TARGET_CAPABILITIES = {"C03", "C06", "C08", "C09", "C13", "C16"}', self.source)
+
+    def test_hook_telemetry_is_external_and_fail_open(self) -> None:
+        self.assertIn('HOOK_LOG_ENV = "PLANANVIL_QUAL_HOOK_LOG"', self.source)
+        self.assertIn('os.environ["TMPDIR"] = str(path.parent.resolve())', self.source)
+        self.assertIn('os.environ[HOOK_LOG_ENV] = str(path.resolve())', self.source)
+        self.assertIn("except Exception:\n    pass\n# Recorder failure is never allowed to change the hook result.", self.source)
+        recorder_index = self.source.index("except Exception:\n    pass\n# Recorder failure is never allowed to change the hook result.")
+        stdout_index = self.source.index("sys.stdout.write(completed.stdout)")
+        self.assertLess(recorder_index, stdout_index)
+        self.assertNotIn('qualification-hook-events.jsonl"\nlog.parent.mkdir', self.source)
+
+    def test_compaction_repairs_use_low_redundant_triggers(self) -> None:
+        self.assertIn("v4.C08_COMPACT_LIMIT = 40", self.source)
+        self.assertIn("v4.C09_COMPACT_LIMIT = 200", self.source)
+        self.assertIn("expanded = [name for name in names for _ in range(2)]", self.source)
+        self.assertIn("return v4._c08_runtime(**kwargs)", self.source)
+        self.assertIn("return v4._c09_runtime(**kwargs)", self.source)
+
+    def test_c03_uses_explicit_auxiliary_git_source_and_outer_decision(self) -> None:
+        self.assertIn('source = worktrees / "source"', self.source)
+        self.assertIn('planning = worktrees / "planning"', self.source)
+        self.assertIn('".agents/skills/plan-anvil/scripts/plan_anvil.py", "start"', self.source)
+        self.assertIn("source_core_unchanged", self.source)
+        self.assertIn("execution_contract_findings", self.source)
+        self.assertIn("flat\\s+direct[- ]child\\s+topology", self.source)
+        c03 = self.source[self.source.index("def run_c03("):self.source.index("# C06")]
+        self.assertNotIn("evaluator", c03.lower())
+
+    def test_c06_has_integrated_and_minimal_apply_patch_repros(self) -> None:
+        self.assertIn('"matcher": "^apply_patch$"', self.source)
+        self.assertIn("minimal_apply_patch_pretooluse", self.source)
+        self.assertIn("supported_apply_patch", self.source)
+        self.assertIn("The isolated PreToolUse hook fired, but the installed PlanAnvil PreToolUse hook did not.", self.source)
+        self.assertIn("Direct apply_patch completed but the isolated current-runtime PreToolUse hook did not fire.", self.source)
+        self.assertIn("outer_non_intercepted_postcondition", self.source)
+
+    def test_c13_reuses_baseline23_transport_with_repaired_telemetry(self) -> None:
+        self.assertIn("def run_c13(current_runtime", self.source)
+        self.assertIn("_patched_v5_c13", self.source)
+        self.assertIn("v5._c13_hook_proxy_source = _fail_open_proxy_source", self.source)
+        self.assertIn("v5._run_c13_codex = run_c13", self.source)
+
+    def test_c16_uses_outer_diagnostic_basis_without_raw_diagnostics(self) -> None:
+        self.assertIn("def _outer_probe", self.source)
+        self.assertIn("signing_diagnostic_observed", self.source)
+        self.assertIn("hook_diagnostic_observed", self.source)
+        self.assertIn('"raw_diagnostics_retained=false"', self.source)
+        c16 = self.source[self.source.index("def run_c16("):]
+        self.assertNotIn("evaluator", c16.lower())
+
+    def test_safety_invariants(self) -> None:
+        for forbidden in (
+            "--dangerously-bypass-approvals-and-sandbox",
+            "danger-full-access",
+            "--privileged",
+            "SYS_ADMIN",
+        ):
+            self.assertNotIn(forbidden, self.source)
+        self.assertIn('sandbox="workspace-write"', self.source)
+        self.assertIn("_patched_v4", self.source)
+        self.assertIn("_patched_v5_c13", self.source)
+
+    def test_audit_records_root_causes_and_exit_gate(self) -> None:
+        for marker in (
+            "R1 — read-only hook telemetry",
+            "R4 — C03",
+            "R5 — C16",
+            "R6 — C06",
+            "R8 — wrapper-chain growth",
+            "no full/self-hosted qualification is run until hosted regression tests",
+        ):
+            self.assertIn(marker, self.audit)
+
+
+if __name__ == "__main__":
+    unittest.main()