auto cli-todo canonical run blocked at safe-default closure (R1 evidence)

[shaun0927]

Summary

R1 live canonical run (OUROBOROS_RUN_CANONICAL=1 pytest tests/canonical/ -k cli-todo on 265aedb4) terminated with phase=blocked at safe-default synthesis. ooo auto cannot complete the cli-todo canonical scenario, which fails SSOT #1157 success condition #1 (all four canonical goals reach status=complete). Two coupled defects produce a single symptom:

• B1 — Logic: safe-default fallback cannot close the interview when the backend accepts the synthesis answer but does not flag the resulting turn as seed_ready/completed. Defaults are then rolled back and the auto pipeline exits blocked.
• B2 — Envelope: the resulting blocker is emitted with last_error_code=None, violating the canonical 8-code mapping contract introduced by feat(auto): canonical stop_reason_code for interview-layer blockers #1151.

This issue tracks the behavioural fix. #1211 (open) is the observability complement for the same decision point and should land alongside, not instead of, the work proposed here. Bug A (the test-harness is_ok() / unwrap_err() mis-call that previously masked this evidence behind a TypeError) is fixed in #1218.

Reproduction

# Once #1218 merges; current main HEAD = 265aedb4 cannot reproduce without it
OUROBOROS_RUN_CANONICAL=1 uv run pytest tests/canonical/ -k cli-todo -v
# ~350s wall, ~$1 LLM cost. Expect phase=blocked terminal.

Local evidence preserved in .ooo-observability/R1-cli-todo-20260525-1739.log on the reporter's worktree (auto_session_id=auto_3f44b20d63b7, interview_id=interview_169770e8f45c48cf).

R1 evidence — interview ambiguity trajectory (13 rounds, never crossed the 0.2 gate)

round	overall	goal	constraints	success_criteria	ready?
3	0.298	0.78	0.62	0.68	False
4	0.304	0.78	0.62	0.66	False
5	0.540	0.55	0.35	0.45	False
6	0.401	0.74	0.52	0.49	False
7	0.464	0.68	0.46	0.42	False
8	0.361	0.72	0.55	0.62	False
9	0.421	0.72	0.42	0.55	False
10	0.353	0.74	0.55	0.62	False
11	0.397	0.72	0.55	0.50	False
12	0.373	0.72	0.58	0.55	False

Score oscillates between 0.298 and 0.540 — never converges below the 0.2 readiness gate. After round 12 the auto driver enters the safe-default fallback at 08:45:41:

auto.interview.safe_default.entered
  ambiguity_score=0.373  backend_done=False  ledger_done=False
  open_gaps=['runtime_context']  max_rounds=12

R1 evidence — terminal state (recovered from log; JSON dump was not retained on disk)

• phase=blocked
• blocker: safe-default synthesis did not close the persisted interview: backend_done=False, ledger defaults rolled back
• last_error_code=None
• seed_origin=none
• runtime_probe_evidence=[]

Suspect code

B1 — Logic: safe-default cannot close when backend silently no-ops synthesis

src/ouroboros/auto/interview_driver.py:440-454

state.interview_session_id = synthesis_turn.session_id
state.pending_question = synthesis_turn.question
if not (synthesis_turn.seed_ready or synthesis_turn.completed):
    _revert_safe_default_entries(ledger, finalization.defaulted_sections)
    blocker = (
        "safe-default synthesis did not close the persisted interview: "
        "backend_done=False, ledger defaults rolled back"
    )
    state.ledger = ledger.to_dict()
    state.mark_blocked(blocker, tool_name="interview.safe_default_synthesis")
    record_authoring_backend(state)
    self._save(state)
    return AutoInterviewResult(
        "blocked", state.interview_session_id, ledger, self.max_rounds, blocker
    )

When the Socratic backend accepts the synthesis answer but never flags seed_ready/completed on the resulting turn (the cli-todo runtime_context gap reproduces this reliably), #1167's policy rolls every default back and exits blocked. The backend appears to treat the driver-injected synthesis as just another user response, not a terminator.

B2 — Envelope: last_error_code never set for this blocker

src/ouroboros/auto/state.py:626-636

def mark_blocked(
    self,
    message: str,
    *,
    tool_name: str | None = None,
    error_code: str | None = None,
) -> None:
    self.last_tool_name = tool_name
    self.last_error_code = error_code
    self.transition(AutoPhase.BLOCKED, message, error=message)

Both safe-default failure sites in interview_driver.py (lines 434 and 449) call mark_blocked(blocker, tool_name="interview.safe_default_synthesis") without passing error_code=, so last_error_code defaults to None. The terminal envelope carries the rich blocker text but no canonical code — breaking the #1151 8-code mapping contract.

Sub-tasks

• B1 (logic) — src/ouroboros/auto/interview_driver.py:440-454. Decide closure policy when the backend ack is content-only: either extend the safe-default contract with a third closure mode (alongside mutual_agreement and ledger_only) that accepts "backend echoed, ledger satisfied" as a close, or fail forward into a deterministic ledger_only close instead of reverting defaults. Document the chosen policy on feat(auto): safe-default closure mode + partial-unsafe blocker code (PR-B2) #1167.
• B2 (envelope) — Add INTERVIEW_SAFE_DEFAULT_SYNTHESIS_NONCLOSURE (or equivalent — must be drawn from the feat(auto): canonical stop_reason_code for interview-layer blockers #1151 alphabet) and pass it as error_code= at both mark_blocked call sites in interview_driver.py:434, 449. Add a regression test under tests/auto/ asserting that any safe-default blocker emits a non-None last_error_code from the documented alphabet.

Test-harness coordination

• fix(canonical): call Result.is_ok/is_err as properties #1218 (open) fixes the test harness that previously masked this evidence behind a TypeError. Reproduction above assumes fix(canonical): call Result.is_ok/is_err as properties #1218 has merged.
• fix: prove canonical live harness opt-in #1214 (open) adds a _Ok test stub with method-shape is_ok() / unwrap_err() that mirrors the broken API. When fix(canonical): call Result.is_ok/is_err as properties #1218 lands, that stub must be aligned with the real Result property API; otherwise it silently re-introduces shape drift. See coordination note on fix(canonical): call Result.is_ok/is_err as properties #1218.

Prior art / related work

• instrument(auto): emit structured-log events at safe-default decision points #1138 (merged) — safe-default decision-point observability baseline.
• feat(auto): safe-default closure mode + partial-unsafe blocker code (PR-B2) #1167 / PR-B2 (merged) — added safe-default closure mode + partial-unsafe blocker code; the rollback policy at the heart of B1.
• feat(auto): canonical stop_reason_code for interview-layer blockers #1151 (merged) — canonical 8-code stop_reason_code for interview-layer blockers; B2 fills a gap in that alphabet.
• fix: log safe-default synthesis nonclosure #1211 (open) — emits structured auto.interview.safe_default_synthesis_nonclosure event. Observability-only by design; complementary to this issue, not a substitute.
• fix(auto): close safe-defaultable interview gaps at max rounds #1122 / fix(auto): let safe-default synthesis close interviews #1129 — earlier safe-default work referenced by instrument(auto): emit structured-log events at safe-default decision points #1138.

Cross-refs

• SSOT: Meta SSOT: ooo auto Vision — Autonomous Completion Engine #1157
• Harness unblock: fix(canonical): call Result.is_ok/is_err as properties #1218
• Logic baseline: feat(auto): safe-default closure mode + partial-unsafe blocker code (PR-B2) #1167, fix(auto): close safe-defaultable interview gaps at max rounds #1122, fix(auto): let safe-default synthesis close interviews #1129
• Envelope baseline: feat(auto): canonical stop_reason_code for interview-layer blockers #1151
• Observability complement (open): fix: log safe-default synthesis nonclosure #1211
• Observability base (merged): instrument(auto): emit structured-log events at safe-default decision points #1138
• Harness stub coordination: fix: prove canonical live harness opt-in #1214

Constraints (per evidence-driven minimal-substrate policy)

• No second live R1 run until B1 lands — same evidence, $1 wasted.
• No new substrates or abstractions; both sub-tasks are edits to existing modules.
• No direct push to main.

[shaun0927]

R4 deferred-probe coverage (evidence for envelope-side gap)

Generated on 265aedb4 via:

uv run python -c "from ouroboros.auto.task_classes import TaskClass; from ouroboros.orchestrator.runtime_evidence import probes_for_task_class, DeferredProbe, HeadlessRunProbe; [print(f'{tc.value:<22} real={sum(1 for p in probes_for_task_class(tc) if isinstance(p, HeadlessRunProbe))} deferred={sum(1 for p in probes_for_task_class(tc) if isinstance(p, DeferredProbe))} kinds={[p.kind for p in probes_for_task_class(tc)]}') for tc in TaskClass]"

Output (preserved in .ooo-observability/R4-probe-coverage.txt):

library                real=0 deferred=2 kinds=['import_smoke', 'unit_tests']
cli                    real=1 deferred=1 kinds=['headless_run', 'stdout_golden']
web_service            real=1 deferred=1 kinds=['headless_run', 'api_smoke']
webhook                real=0 deferred=2 kinds=['api_smoke', 'side_effect_probe']
data_pipeline          real=1 deferred=1 kinds=['headless_run', 'output_fixture_diff']
game_2d                real=1 deferred=1 kinds=['sim_trace', 'headless_run']
refactor_in_place      real=0 deferred=1 kinds=['test_suite_parity']

TaskClasses with zero real probes (3/7)

These classes currently produce runtime_probe_evidence=[] even on a successful run — which is exactly the R1 cli-todo symptom on the envelope side. cli-todo (class cli) does have one real probe but R1 still emitted empty evidence because the pipeline blocked before any probe could fire; the three classes below would emit empty evidence even on the happy path.

TaskClass	Current deferred kinds	Recommended next real probe
library	import_smoke, unit_tests	Promote import_smoke to real: pip install -e . && python -c "import {pkg}" exit code is the minimum verification that the artifact is even loadable; cheap, hermetic, no fixtures needed.
webhook	api_smoke, side_effect_probe	Promote api_smoke to real: POST a synthetic event to the registered handler and assert 2xx; pair with an in-process mock sink so the existing deferred side_effect_probe can stay deferred until a real sidecar exists.
refactor_in_place	test_suite_parity	Promote test_suite_parity to real: run the project's existing test suite headlessly and assert exit 0. This is the only meaningful "did the refactor break anything" signal — leaving it deferred makes every refactor terminal pass evidence-free.

Why this is attached here, not filed separately

B2 promises a non-None last_error_code for the safe-default nonclosure case. The same envelope contract is silently hollow on the success path for three task classes — any future Bug B-style failure on those classes would terminate with both last_error_code=None and runtime_probe_evidence=[], leaving the postmortem with nothing to grep. Promoting one real probe per zero-coverage class is the symmetric fix on the success side of the envelope.

Suggested SSOT priority: after B1/B2 land, sequence is library:import_smoke (smallest), then refactor_in_place:test_suite_parity (highest leverage), then webhook:api_smoke (needs a sample fixture).

[Q00]

Opened PR #1220 for the bounded B1/B2 fix.

Scope:

• safe-default synthesis ack now closes as safe_default when the transcript write succeeded and the ledger is structurally complete;
• transcript-sync failure remains fail-closed and now carries a typed stop-reason code;
• focused auto interview regressions and ruff checks passed locally.

Posted by agentos-roadmap-warden — bot. Reply with /warden ignore to suppress further comments on this thread.

[ouroboros-agent]

There's already an open PR addressing this: #1220. Review is in progress.

ouroboros-agent[bot]