Skip to content

Explain research failures when a model response reaches its output limit #692

Closed

Description

@magnus919

A live durable-research verification completed successfully on one deployment, but another admission failed during drafting with outcome_unknown. Gateway accounting showed a successful 138.458-second response consuming exactly the configured 8,192 completion-token ceiling. The failed admission was retained and was not replayed. A separately admitted narrower question subsequently completed with all three report artifacts.

The exact finish reason was not retained, so token exhaustion is a strong indication rather than a confirmed diagnosis. The service currently sanitizes transport rejection details, making a truncated response difficult to distinguish from an unknown network outcome.

Improve this without silently increasing budgets or retrying paid calls:

  • Retain safe, bounded failure diagnostics such as HTTP status, finish reason and token counts, excluding response text, credentials and provider endpoints.
  • Distinguish a known incomplete response from a dispatch whose outcome is unknown, while preserving the operation ledger and recovery rules.
  • Add deterministic tests for output-limit responses and unchanged no-replay behavior.
  • Evaluate any prompt or model-setting change separately before claiming improved completion reliability.

The shared five-minute deadline in PR #691 is deployed and verified. This follow-up concerns output-limit reliability and diagnostics, not another timer extension.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions