Skip to content

Account for prompt caching in story evaluations - #133

Merged
R055LE merged 12 commits into
mainfrom
issue-132-cache-accounting
Oct 30, 2026
Merged

R055LE merged 12 commits into
mainfrom
issue-132-cache-accounting

Conversation

@R055LE

@R055LE R055LE commented Oct 29, 2026 •

Copy link
Copy Markdown
Owner

The evaluation harness bypassed response caching while leaving provider prompt caching uncontrolled. Its reservations used base input prices, and its artifacts dropped the metadata needed to explain higher reported charges.

This change gives the existing profiles explicit prompt-cache policies, serializes the documented OpenAI bypass, and reserves possible cache-write pricing. It retains allowlisted generation IDs, cache counters and cost components. Missing details remain unknown; reported activity under a requested-off policy stops the run after preserving available accounting. Invalid core token totals no longer erase otherwise valid cache evidence.

Stacked on #120. Fixes #132. Production request profiles and daily reservations remain separate work in #116 and #117.

Validation:

  • Reproduced the reservation regression before implementation.
  • SDK wire tests cover both selected roles; budget regressions prove rejection before provider I/O.
  • Independent review passed, including the invalid-token-total correction.
  • Full offline suite: 672 passed, 87.60% coverage. Ruff and git diff --check passed.
  • Network-free planning succeeds with the corrected reservations.
  • CI passed on Python 3.12 and 3.14, including the dependency audit.

Existing story prompts, reviewer policy and historical ledgers are unchanged. No paid calls were made. Offline tests establish request serialization; live acceptance of the new provider control remains unverified.

@R055LE
R055LE marked this pull request as ready for review October 30, 2026 15:43
@R055LE
R055LE enabled auto-merge (squash) October 30, 2026 15:44
@R055LE
R055LE merged commit 3ce13da into main Oct 30, 2026
2 checks passed
@R055LE
R055LE deleted the issue-132-cache-accounting branch October 30, 2026 15:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Distinguish prompt caching from response caching in evaluation accounting

1 participant