feat(auto): runtime-probe envelope + advisory probe_runner (L3-2)

[shaun0927]

Summary

L3-2 slice of #1176 — surfaces runtime acceptance evidence (L3-1 substrate, #1181) onto AutoPipelineResult via an optional probe_runner callback on AutoPipeline, and treats configured runtime-probe failures as PRODUCT_COMPLETE blockers.

Adds the minimal envelope + completion-gate plumbing:

• AutoPipelineResult.runtime_probe_evidence: tuple[RuntimeEvidence, ...] = () — frozen evidence tuple; empty when no runner is wired (default/backwards compatible).
• AutoPipeline.probe_runner optional async callback. The caller decides which probes to invoke (often via probes_for_task_class with the active task class from L1-d / future binding work).
• AutoPipeline._complete_runtime_probe_blocker runs the callback once per run() before COMPLETE is returned on PRODUCT_COMPLETE paths, caches evidence, and surfaces it via every _result() call in the same run.
• Runtime evidence is no longer advisory-only when a runner is configured: explicit probe passed=False and runner infrastructure exceptions downgrade the terminal to BLOCKED instead of allowing a false PRODUCT_COMPLETE.
• Both Ralph-success direct completion and EVALUATE-pass completion paths invoke the same runtime-probe completion gate.

This remains minimal-substrate: no new EventStore event family, projection, CLI flag, probe kind, or command-source policy is introduced here.

What lands

• src/ouroboros/auto/pipeline.py: runtime_probe_evidence field + typed probe_runner callback + completion-gate helper + cache + _result() plumbing.
• tests/unit/auto/test_pipeline_runtime_probe_envelope.py: integration tests for empty-by-default, successful evidence surface, runner exception blocking, explicit probe failure blocking, and EVALUATE-pass completion coverage.

What is NOT in this PR

• New probe kinds (sim_trace, render_hash, api_smoke) or real probe implementations beyond the L3-1 headless_run substrate.
• EventStore/projection/audit-event persistence for runtime probes.
• Automatic command-source selection / task-class binding inside AutoPipeline; this slice consumes a wired runner rather than owning binding policy.
• L0 canonical scenario fixture consumption of runtime evidence.

Test plan

• uv run pytest tests/unit/auto/test_pipeline_runtime_probe_envelope.py -q → 5 passed.
• uv run pytest tests/unit/auto/test_pipeline_runtime_probe_envelope.py tests/unit/orchestrator/test_runtime_evidence.py -q → 22 passed.
• uv run pytest tests/unit/auto -q → 910 passed.
• uv run ruff check src/ouroboros/auto/pipeline.py tests/unit/auto/test_pipeline_runtime_probe_envelope.py → clean.
• uv run mypy src/ouroboros/auto/pipeline.py → clean.
• GitHub checks on head a3df25d: Ruff, MyPy, Python 3.12/3.13/3.14, Bridge TypeScript, enforce-envelope, enforce-boundary → pass.

Refs #1157, #1176, #1181 (L3-1 substrate), #1173 (L1-a catalog).

[ouroboros-agent]

Review — ouroboros-agent[bot]

Verdict: APPROVE

Reviewing commit da60637 for PR #1190

Review record: 74dbfb8d-b347-4da4-9997-4e3f13ee69ba

Blocking Findings

No in-scope blocking findings remained after policy filtering.

Non-blocking Suggestions

None.

Design Notes

Not assessed. The review inputs could not be read due the execution sandbox failure.

Policy Notes

• Omitted 1 finding(s) that referenced files outside the current PR changed-files scope.

Recovery Notes

First recoverable review artifact generated from codex analysis log.

---

Reviewed by ouroboros-agent[bot] via Codex deep analysis

[ouroboros-agent]

Review — ouroboros-agent[bot]

Verdict: APPROVE

Reviewing commit b9a65aa for PR #1190

Review record: 2e2bda19-a9c5-433f-9013-f7cb0eae3768

Blocking Findings

No in-scope blocking findings remained after policy filtering.

Non-blocking Suggestions

None.

Design Notes

Unable to complete the review: every local file-read command failed before execution because the sandbox could not create its bwrap namespace. I could not inspect /tmp/pr_diff_1190.patch, the comments, or the changed files, so I cannot give a valid architectural assessment.

Recovery Notes

First recoverable review artifact generated from codex analysis log.

---

Reviewed by ouroboros-agent[bot] via Codex deep analysis

[shaun0927]

Merge-readiness rationale for PR #1190

I re-reviewed this PR against #961 and the #1157/#1176 AgentOS SSOT direction, with special attention to the minimal-substrate audit and the risk of turning L3 runtime acceptance into another over-engineered surface.

What this PR does

PR #1190 is the L3-2 envelope/completion-gate slice for runtime acceptance evidence:

• adds AutoPipelineResult.runtime_probe_evidence as an additive result-envelope field;
• adds an optional typed AutoPipeline.probe_runner callback;
• invokes the runner once per run() before PRODUCT_COMPLETE is returned on both direct Ralph completion and EVALUATE-pass completion paths;
• caches and surfaces returned RuntimeEvidence on the result envelope;
• blocks PRODUCT_COMPLETE when a configured probe explicitly fails or the runner itself fails.

Why this is aligned with the AgentOS SSOT

The key #961/#1157 lesson is: add substrate only when evidence demands it. The corrected PR follows that rule:

• It does not add a new EventStore event family, projection, migration, CLI flag, probe kind, command-source policy, or plugin/runtime surface.
• It reuses the L3-1 RuntimeEvidence substrate and only consumes an injected runner.
• It keeps task-class binding and command selection outside AutoPipeline, which avoids making this envelope slice own policy it cannot yet prove.
• It now satisfies the Meta SSOT slice: L3 — Runtime acceptance substrate (minimal, headless_run only) #1176 L3-2 intent that runtime evidence affects completion, instead of being an advisory-only annotation.

What improved during this review

The original head exposed runtime evidence but swallowed runner failures and allowed probe output to remain advisory-only. That was too weak for #1176 L3-2 because PRODUCT_COMPLETE could still be returned even when a configured runtime probe failed.

Commit b9a65aa corrected that by making runtime evidence a completion-grade input:

• runner exception → BLOCKED with tool_name="probe_runner";
• RuntimeEvidence(passed=False) → BLOCKED with the failing probe summary;
• successful evidence remains surfaced on runtime_probe_evidence;
• EVALUATE-pass completion now goes through the same probe gate.

An empty commit a3df25d was added only to retrigger automation; it has no functional changes.

Verification

Local verification:

• uv run ruff check src/ouroboros/auto/pipeline.py tests/unit/auto/test_pipeline_runtime_probe_envelope.py — pass
• uv run mypy src/ouroboros/auto/pipeline.py — pass
• uv run pytest tests/unit/auto/test_pipeline_runtime_probe_envelope.py tests/unit/orchestrator/test_runtime_evidence.py -q — 22 passed
• uv run pytest tests/unit/auto -q — 910 passed

GitHub verification on the current head is green for Ruff, MyPy, Python 3.12/3.13/3.14 tests, Bridge TypeScript, enforce-envelope, and enforce-boundary.

Ouroboros-agent has an APPROVE review on the substantive fix commit with no blocking or non-blocking findings. Its design-note section reports a sandbox file-read failure, not an architectural objection; the independent SSOT review above covers the architectural/design assessment.

Final recommendation

This PR is mergeable. It is a narrow, additive L3-2 slice that now enforces runtime-probe evidence at the PRODUCT_COMPLETE boundary without introducing new AgentOS substrate or broadening the roadmap scope.

[shaun0927]

PR Review Summary

Verdict

Approve

Scope Reviewed

• PR intent: land Meta SSOT slice: L3 — Runtime acceptance substrate (minimal, headless_run only) #1176 L3-2 by exposing runtime acceptance evidence on the ooo auto result envelope and making configured runtime probes participate in PRODUCT_COMPLETE acceptance.
• Main changed areas: src/ouroboros/auto/pipeline.py (AutoPipelineResult.runtime_probe_evidence, optional probe_runner, cached evidence, completion blocker helper, direct Ralph and EVALUATE completion paths) and tests/unit/auto/test_pipeline_runtime_probe_envelope.py.
• Tests reviewed: new runtime-probe envelope integration tests, existing L3-1 runtime evidence tests, all tests/unit/auto.
• Checks considered: local Ruff, MyPy, targeted pytest, full auto unit pytest; GitHub checks for Ruff, MyPy, Python 3.12/3.13/3.14, Bridge TypeScript, enforce-envelope, enforce-boundary are green.

Blocking Issues

None.

Warnings

None.

Mutation-Test Thinking

• Likely mutants that should be killed:
  • Removing _last_probe_evidence = () at run start should leak evidence across reused AutoPipeline runs; the cache behavior is directly represented by the single-run invocation assertions and would be a good future two-run regression if this surface grows.
  • Changing if not item.passed to if item.passed in _complete_runtime_probe_blocker would block successful evidence and is killed by test_envelope_carries_probe_evidence_when_runner_wired / test_evaluate_complete_path_invokes_probe_runner.
  • Removing the runner exception branch would let infrastructure failures escape or false-complete; killed by test_runner_exception_blocks_product_complete.
  • Removing the EVALUATE-pass hook would silently skip probes on QA-graded completion; killed by test_evaluate_complete_path_invokes_probe_runner.
• Mutants current tests may not catch:
  • A two-run cache leak on the same AutoPipeline instance is indirectly guarded by the explicit reset in code but not directly asserted in this new file.
• Additional tests recommended: optional follow-up only if the runner becomes production-wired by default; add a reused-pipeline two-run test with different evidence tuples.

Complexity / CRAP-style Risk

• High-risk functions/modules: AutoPipeline is already a high-traffic orchestration class, but this patch adds one narrow helper and two call sites rather than another state machine.
• Complexity increase: low. The new branch is linear: no runner => no-op; runner exception => BLOCKED; returned evidence with failures => BLOCKED; otherwise complete.
• Test coverage concern: acceptable. The tests cover default no-op, successful evidence, explicit failure, runner exception, and both direct Ralph completion and EVALUATE completion paths.
• Refactoring recommendation: none for this PR. Pulling probe binding/command selection into this helper would be over-engineering and should stay out of scope.

Test Quality Assessment (6/7)

• Strong tests:
  • Backward compatibility: no probe_runner keeps runtime_probe_evidence == () and completion succeeds.
  • Success path: configured runner returns RuntimeEvidence, surfaces on the result envelope, and runs once.
  • Failure path: runner exception and passed=False evidence both block PRODUCT_COMPLETE with tool_name="probe_runner".
  • Coverage of both completion paths: direct Ralph complete and EVALUATE-pass complete.
• Weak tests:
  • The tests use local stubs and do not exercise a real HeadlessRunProbe subprocess through AutoPipeline; that is acceptable because real command binding is explicitly out of scope for this slice.
• Missing edge cases:
  • Reusing one AutoPipeline object across two run() calls with different probe output could be asserted in a future regression.
• Mocking concerns: low. The stubs model the existing dependency-injection seam (probe_runner) rather than mocking internals.

Security / Operational Risk

None.

Operational notes:

• This PR does not execute shell commands by itself; it consumes an injected runner.
• It avoids logging secrets or adding persistence surfaces.
• Runner failures now become resumable/visible BLOCKED outcomes instead of false PRODUCT_COMPLETE success.

Looks Good

• The corrected behavior aligns with Meta SSOT slice: L3 — Runtime acceptance substrate (minimal, headless_run only) #1176 L3-2: runtime evidence is a completion-grade input, not merely an advisory envelope.
• The implementation preserves Meta SSOT: AgentOS roadmap sequencing (#920–#960) #961/Meta SSOT: ooo auto Vision — Autonomous Completion Engine #1157 minimal-substrate discipline: no new EventStore events, projections, CLI/API flags, command-source policy, or probe kinds.
• The optional runner keeps backwards compatibility for existing AutoPipeline callers.
• The PR body now accurately reflects the post-fix behavior and verification.

Final Recommendation

Approve. This PR is merge-ready: no blocking findings, no meaningful warnings, green CI, local verification passed, and the design stays narrowly within the AgentOS SSOT direction while fixing the original advisory-only gap.

[ouroboros-agent]

Review — ouroboros-agent[bot]

Verdict: APPROVE

Reviewing commit a3df25d for PR #1190

Review record: e793d48f-4d6c-4d4f-aa3d-8b72e8719f44

Blocking Findings

No in-scope blocking findings remained after policy filtering.

Non-blocking Suggestions

None.

Design Notes

Unable to complete the scope-aware review: every attempt to read the provided snapshot files failed before execution because the sandbox runner cannot create its namespace (bwrap: No permissions to create a new namespace). I did not run any git commands.

Recovery Notes

First recoverable review artifact generated from codex analysis log.

---

Reviewed by ouroboros-agent[bot] via Codex deep analysis

[shaun0927]

Fresh independent merge-readiness rationale for PR #1190

I re-reviewed this PR independently against #961, #1157, and the L3 runtime-acceptance SSOT in #1176. I ignored the bot review for the design assessment and used the issue SSOT, local diff, tests, and current CI state as evidence.

SSOT alignment

PR #1190 is aligned with the accepted AgentOS direction:

• Meta SSOT slice: L3 — Runtime acceptance substrate (minimal, headless_run only) #1176 defines L3 v1 as minimal runtime acceptance evidence: reuse the L3-1 RuntimeEvidence / HeadlessRunProbe substrate, surface evidence on AutoPipelineResult, and avoid new EventStore/projection/control-plane surfaces.
• This PR adds an optional probe_runner callback and an additive runtime_probe_evidence result-envelope field.
• Configured runtime evidence now participates in PRODUCT_COMPLETE acceptance: runner infrastructure failures and explicit passed=False evidence block completion instead of producing a false complete result.

Over-engineering review

I do not see over-engineering in the final shape. The PR does not add probe kinds, EventStore events, projections, CLI flags, task-class binding policy, or command-source selection. It keeps AutoPipeline as a consumer of an injected runner, which is the right boundary for this L3-2 envelope/completion-gate slice.

Fresh verification

Local checks from a clean dedicated worktree at head a3df25df:

• uv run pytest tests/unit/auto/test_pipeline_runtime_probe_envelope.py tests/unit/orchestrator/test_runtime_evidence.py -q → 22 passed
• uv run pytest tests/unit/auto -q → 910 passed
• targeted ruff check → passed
• targeted ruff format --check → passed
• targeted mypy src/ouroboros/auto/pipeline.py → passed

Current GitHub checks on the PR head are also green: Ruff, MyPy, Python 3.12/3.13/3.14 tests, Bridge TypeScript, enforce-envelope, and enforce-boundary.

Final recommendation

Merge is appropriate. PR #1190 is a narrow L3-2 runtime-evidence envelope and completion-gate slice. It fixes the advisory-only gap while preserving the SSOT's minimal-substrate boundary.

[shaun0927]

PR Review Summary

Verdict

Approve

Scope Reviewed

• PR intent: surface runtime acceptance evidence on AutoPipelineResult and make configured runtime probes participate in PRODUCT_COMPLETE acceptance.
• Main changed areas: AutoPipelineResult.runtime_probe_evidence, optional AutoPipeline.probe_runner, probe evidence cache/reset, completion blocker helper, direct Ralph completion and EVALUATE-pass completion paths, runtime probe envelope tests.
• Tests reviewed: runtime probe envelope tests, L3-1 runtime evidence tests, full tests/unit/auto.
• Checks considered: fresh local pytest/ruff/mypy plus current green GitHub checks.

Blocking Issues

None.

Warnings

None.

Mutation-Test Thinking

• Likely mutants that should be killed:
  • Removing the run-entry _last_probe_evidence = () reset and leaking evidence between reused pipeline runs.
  • Inverting the not item.passed failure check.
  • Swallowing runner exceptions and returning PRODUCT_COMPLETE.
  • Removing the EVALUATE-pass probe gate.
  • Omitting runtime_probe_evidence from _result().
• Mutants current tests may not catch:
  • A two-run same-AutoPipeline cache-leak scenario is indirectly guarded by code reset but could be asserted explicitly if this surface grows.
• Additional tests recommended:
  • Optional future regression: reuse one pipeline instance across two runs with different probe evidence once production runner binding is added.

Complexity / CRAP-style Risk

• High-risk functions/modules: AutoPipeline is central, but this patch adds a small linear helper and two completion call sites.
• Complexity increase: low; no new state machine or persistence layer.
• Test coverage concern: acceptable for this slice.
• Refactoring recommendation: none. Pulling task-class binding or command selection into AutoPipeline would be premature and out of scope.

Test Quality Assessment (6/7)

• Strong tests: empty-by-default compatibility, successful evidence surface, explicit probe failure blocking, runner exception blocking, direct Ralph and EVALUATE completion paths.
• Weak tests: real subprocess HeadlessRunProbe is not invoked through AutoPipeline, which is acceptable because binding/command selection is explicitly outside this PR.
• Missing edge cases: reused pipeline two-run cache-leak regression, optional follow-up only.
• Mocking concerns: low; tests use the intended dependency-injection seam.

Security / Operational Risk

None. The PR does not execute commands itself; it only consumes an injected runner. It avoids new persistence/secrets/logging surfaces and makes runner failure visible as BLOCKED instead of false success.

Looks Good

• Corrects the advisory-only gap so configured runtime probes are completion-grade evidence.
• Preserves Meta SSOT slice: L3 — Runtime acceptance substrate (minimal, headless_run only) #1176's minimal-substrate rule: no new EventStore event family, projection, CLI/API flag, probe kind, or binding policy.
• Keeps backwards compatibility when no runner is wired.
• Local and GitHub verification are green.

Final Recommendation

Approve. PR #1190 is merge-ready and implements the L3-2 runtime evidence envelope/completion gate at the right narrow boundary.

[ouroboros-agent]

Review — ouroboros-agent[bot]

Verdict: APPROVE

Reviewing commit a3df25d for PR #1190

Review record: ac7fddb3-6575-42aa-8c34-bac0464bf562

Blocking Findings

No in-scope blocking findings remained after policy filtering.

Non-blocking Suggestions

None.

Design Notes

Unable to assess architecture or implementation because the local command runner cannot access the provided snapshot.

Policy Notes

• Omitted 1 finding(s) that referenced files outside the current PR changed-files scope.

Recovery Notes

First recoverable review artifact generated from codex analysis log.

---

Reviewed by ouroboros-agent[bot] via Codex deep analysis

[ouroboros-agent]

Review — ouroboros-agent[bot]

Verdict: REQUEST_CHANGES

PR #1190
Branch: feat/pP8 | 2 files, +451/-0 | CI: Bridge TypeScript pass 11s https://github.com/Q00/ouroboros/actions/runs/26319808762/job/77486486827
Scope: architecture-level
HEAD checked: 35be0e97a3aacb144172c990f00081907576c1e0

What Improved

• Added a typed runtime_probe_evidence envelope on AutoPipelineResult.
• Added completion-gate logic that can downgrade configured probe failures and runner exceptions from COMPLETE to BLOCKED.
• Added direct AutoPipeline tests for empty default evidence, passing evidence, runner exception blocking, failed probe blocking, and EVALUATE-pass completion.

Issue #N/A Requirements

Requirement	Status
Surface runtime acceptance evidence on AutoPipelineResult	Partially met: present only for the in-process result where _last_probe_evidence is populated.
Treat configured runtime-probe failures as PRODUCT_COMPLETE blockers	Partially met: direct AutoPipeline injection blocks, but public CLI/MCP entrypoints do not configure a runner.
Cache evidence within a run	Mostly met for non-empty evidence; not sufficient for durable replay.
Preserve consumer contract across persistence/resume	Not met: completed-session replay loses evidence.
Meaningful tests for newly added logic	Partially met: direct unit coverage exists, but persistence and public entrypoint boundaries are untested.

Prior Findings Status

Prior Finding	Status
Prior review context	MODIFIED — No prior review text was provided in this audit prompt; concerns above are newly raised against current HEAD.

Blockers

#	File:Line	Severity	Confidence	Finding
1	src/ouroboros/auto/pipeline.py:337	High	95%	Runtime probe evidence is not durable, so replay/resume drops the audit evidence from completed sessions. run() clears _last_probe_evidence at every entry, a resumed COMPLETE session returns immediately, and _result() only reads the in-memory cache rather than persisted state. That means the same auto_session_id can initially return runtime evidence, then later return runtime_probe_evidence=() after --resume/MCP replay, breaking the persistence and consumer contract for acceptance evidence.
2	src/ouroboros/mcp/tools/auto_handler.py:485	High	90%	The production MCP ouroboros_auto composition constructs AutoPipeline without any probe_runner, and the CLI path does the same at src/ouroboros/cli/commands/auto.py:510. As a result, normal complete_product runs through the public entrypoints never invoke runtime probes and cannot be blocked by runtime-probe failure; only tests or custom embedders that manually instantiate AutoPipeline(probe_runner=...) get the new completion gate.

Follow-ups

#	File:Line	Priority	Confidence	Suggestion
—	—	—	—	None.

Test Coverage

• Covered: direct AutoPipeline paths for no runner, successful evidence, runner exception, failed evidence, and EVALUATE-pass completion.
• Missing: persisted replay/resume coverage proving a completed session still surfaces the same runtime evidence after reload.
• Missing: CLI/MCP composition coverage proving public complete_product entrypoints actually wire a probe runner and expose/block on its evidence.

Design / Roadmap Gate

Affected-boundary review fails. The change touches a completion-grade input, so the relevant boundaries are not just _complete_runtime_probe_blocker; they include persisted AutoPipelineState, resume/replay, and public CLI/MCP composition. Current HEAD keeps runtime evidence in an instance-local cache only and does not wire the runner through the production entrypoints, so the acceptance evidence is neither durable nor actually enforced for normal complete_product consumers.

Merge Recommendation

Post-merge corrective PR required: persist validated runtime evidence on AutoPipelineState, replay it through _result(), add resume tests, and wire/test a production probe runner path for CLI/MCP complete-product sessions before relying on this as runtime acceptance gating.

Review-Metadata:
verdict: REQUEST_CHANGES
github_event: COMMENT
review_kind: post_merge_audit
merge_eligible: false
head_sha: 35be0e9
source_read_ok: true
diff_read_ok: true
blocking_count: 0