Repository navigation
test(plugins): list marker-bearing user messages when a codex turn lacks its prompt - #2941
Conversation
…cks its prompt Finding 2019: three kubernetes QA cells failed the Calendar step with promptSeen false while the tool call, result and terminal answer were all mirrored. The failure printed only the turn object, so it could not tell a late or missing prompt mirror from a second, unmirrored copy of the prompt that moves the evidence window past the mirrored one. The evidence script now returns the newest ten marker-bearing user messages (seq, whether inside the window, mirror identity and origin, idempotency key, timestamp; no content). The three Codex turn checks (tool call, denied call, disabled tool) include them in their messages. The window start is unchanged. A stubbed-transcript test proves the unmirrored-copy shape reproduces the observed failure and that the new listing names both rows.
|
🦞👀 Pull request received. I will update this pull request when review starts. ClawSweeper review completeClawSweeper finished reviewing this revision. The review result is being finalized. |
clawsweeper
Bot
added
clawsweeper:intake:bug
|
Codex review: needs maintainer review before merge. What this changesThis PR adds marker-bearing user-message metadata to Codex transcript assertion failures and tests the resulting diagnostics. Example: The calendar-marker-2019 prompt mirror is at sequence 0, followed by an unmirrored duplicate at sequence 1.
Review scores
ProductKind: Test only · Worth it: Yes · Fix scope: Complete Merge readiness✅ Ready for maintainer review Keep this PR open: it adds useful, focused QA diagnostics without weakening transcript assertions, and no actionable defect was found. Priority: P3 Before mergeNone. FindingsNone. Tests
Agent review detailsHow this fits togetherThe real plugin QA helper reads an Agent's SQLite transcript through execGateway, correlates native Codex turn mirrors, and produces assertions and failure diagnostics. flowchart LR
A[Calendar QA scenario] --> B[Native plugin assertions]
B --> C[Gateway transcript reader]
C --> D[SQLite transcript]
D --> E[Turn evidence and marker metadata]
E --> F[Assertion result or diagnostic]
Technical reviewBest possible solution: Retain the bounded metadata diagnostics and existing turn-isolation assertions so a subsequent QA failure can distinguish an excluded prompt mirror from an absent one. Do we have a high-confidence way to reproduce the issue? Source inspection establishes that a later marker-bearing user message excludes an earlier prompt mirror; the added SQLite scenario exercises that path, but no target code was run during this review. Is this the best way to solve the issue? The diagnostic-only change is appropriately narrow: it reuses the existing reader and assertion owners while preserving the isolation rule needed to reject stale turn evidence. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning medium; reviewed against 7afa66cb5532. Provenance checked
TestingProof path: in-process harness. SecurityNone. EvidenceWhat I checked:
Likely related people:
Review metrics
LabelsLabel changes:
Label justifications:
Rating scale6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior. WorkflowClawSweeper edits this one comment on every review. Comment Reviewed October 19, 2026, 9:55 AM ET / 13:55 UTC. |
1 participant
Problem
Finding 2019: three kubernetes QA cells (#2916 run 39217526356, #2918 run 39218164613, #2905 run 39206960045) failed the Calendar step's Codex mirror evidence with
promptSeen: false. The tool call, the result and the terminal answer were all mirrored. The failure printed only the turn object.The evidence window starts at the last marker-bearing user message. So the failure can't tell a missing or late
:promptmirror (runtime) from a second, unmirrored copy of the prompt that moves the window past the mirrored one (test).Change
Test-only diagnostics. The window start is unchanged.
markerUserMessages, the newest ten user messages carrying the marker:seq, whether each is inside the window, mirror identity and origin, idempotency key, timestamp. No content.Verification
tests/integration/plugin-driver-real-preflight.test.mjs. It runs the real evidence script against local SQLite transcripts through a stubbedexecGateway:codexMirrorTurnVerified).promptSeen: falsewith call, result and answer seen). The message lists both rows, the mirror outside the window.tests/conformance/. It stays in the preflight file, whichchecks-baseline-2already runs.Risks
Low. Test-only. The evidence output grows by at most ten small records.
For the owner: the next sighting will show which case it is.
:promptrow at all would point at the runtime's prompt mirror (product).