Skip to content

fix(routing): close ccmax cache cost feedback loop - #20

Open
SakuraPuare wants to merge 1 commit into
mirainya:mainfrom
ShuYingJiYu:fix/ccmax-cache-cost-loop
Open

SakuraPuare wants to merge 1 commit into
mirainya:mainfrom
ShuYingJiYu:fix/ccmax-cache-cost-loop

Conversation

@SakuraPuare

Copy link
Copy Markdown
Collaborator

Summary

Long-running sessions kept re-attributing cache cost back into itself, so upstreams that failed early or occasionally still showed up as cheap. This PR closes that loop with three focused changes:

  1. Rolling cache creation counts and token volumes — cache reads and writes can overlap within one response; pricing now tracks both concurrently instead of collapsing to a single write-per-miss figure.
  2. Persist per-attempt actuals — rejected route decisions and the real per-attempt cost are stored, and low-sample failures get a small penalty so unavailable or repeatedly failing upstreams remain explainable in the UI.
  3. Focused test coverage — new tests for routing / store / audit paths (new file: internal/server/routing_audit_test.go).

Files touched (12)

  • internal/forward/forward.go
  • internal/routing/cachestate.go
  • internal/routing/cost.go
  • internal/routing/cost_test.go
  • internal/routing/selector.go
  • internal/routing/selector_test.go
  • internal/routing/types.go
  • internal/scheduler/intelligent.go
  • internal/server/routing_audit.go
  • internal/server/routing_audit_test.go (new)
  • internal/store/routing_observations.go
  • internal/store/routing_persistence_test.go

Validation

Rebased on top of current main (post-#19). On our CI runner:

go vet ./...     # clean
go test ./...    # all packages ok

Fork context: this PR carries the routing/cost fix on top of the InputInflation three-part accounting already in main; the cachedPortion/writeTokens decomposition preserves that accounting and adds the rolling create-count and observedWriteTokens override on top.

Use rolling cache creation counts and token volumes when pricing long-running sessions, because cache reads and writes can overlap in one response.
Persist rejected route decisions and actual per-attempt cost, and penalize low-sample failures so unavailable or repeatedly failing upstreams remain explainable.
Add focused store, routing, and audit coverage.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant