Skip to content

Retain bounded Curated review failures for diagnosis #148

Open

Description

@R055LE

Observed after #144

Read-only production acceptance on November 12 checked all five scheduled runs since the context-limit fix at 58c03ad5b94ef99252c14285b3acc6c348df3522:

Local check date Outcome Model calls Settled cost
November 8 evidence review rejected the revision: supported 4 $0.115918
November 9 evidence review rejected the revision: supported 4 $0.121854
November 10 posted 4 $0.117986
November 11 revision left a rejected item unchanged 3 $0.070122
November 12 posted 4 $0.114854

No context/JSON errors recurred in these checks. All 19 attempts settled; total $0.540734. No pending deliveries or unsettled attempts. Both successful posts used the semantic revision path. The three rejected workflows are not deliberate quiet/no-news decisions.

Diagnostic gap

stories.parse_review() reports the gate name supported, but the scheduled outcome does not retain which reason/change/public-claim item failed. require_rejected_items_repaired() detects unchanged rejected text and citations. Failed drafts and their structured reviewer verdicts are discarded; only sanitized status and the rejected input fingerprint remain. Accepted decisions are persisted only after stories.decide() succeeds.

Consequently the actual failed drafts cannot be inspected now. The logs establish rejection but cannot show whether it was a justified evidence/citation problem, ineffective revision, or reviewer false rejection. Ordinary supported flags are Boolean-only; there is no hidden explanatory rationale to recover.

Scope and acceptance

  • Retain a bounded, versioned diagnostic record for editorial failures, linked to the input version and workflow. Distinguish stage and failure gate.
  • Preserve the rejected item positions/fields, text and exact cited spans, the corresponding revised items, and both structured verdicts when available. Include only the bounded evidence/render projection needed to judge the specific failure. Missing initial/final stages remain explicitly absent.
  • Keep records private and operator-readable with explicit count/size/age limits. Never copy credentials, hidden reasoning, unrestricted provider payloads or full prompts to logs, public issues or ops messages. Treat retained model/source strings as untrusted data.
  • Make sanitized status identify the failed field/stage or diagnostic reference without leaking the retained content.
  • Add offline regressions for post-revision support rejection, unchanged rejected items, absent final review, retention bounds and safe diagnostics. Preserve accepted-output behavior, exact-citation checks, completeness review, one-revision/four-call limit and the existing fail-closed result.
  • Inspect captured examples before changing prompts, reviewer thresholds or models. A fresh model run is not a reconstruction of a discarded response.

This issue records the next diagnostic improvement. It does not authorize paid replay, relaxed review, production configuration changes or implementation in this status-review session. Related: #115, #110, #143, #97, #98.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions