fix(routing): close ccmax cache cost feedback loop - #20
Open
SakuraPuare wants to merge 1 commit into
Open
SakuraPuare wants to merge 1 commit into
SakuraPuare wants to merge 1 commit into
Conversation
Use rolling cache creation counts and token volumes when pricing long-running sessions, because cache reads and writes can overlap in one response. Persist rejected route decisions and actual per-attempt cost, and penalize low-sample failures so unavailable or repeatedly failing upstreams remain explainable. Add focused store, routing, and audit coverage.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Long-running sessions kept re-attributing cache cost back into itself, so upstreams that failed early or occasionally still showed up as cheap. This PR closes that loop with three focused changes:
internal/server/routing_audit_test.go).Files touched (12)
internal/forward/forward.gointernal/routing/cachestate.gointernal/routing/cost.gointernal/routing/cost_test.gointernal/routing/selector.gointernal/routing/selector_test.gointernal/routing/types.gointernal/scheduler/intelligent.gointernal/server/routing_audit.gointernal/server/routing_audit_test.go(new)internal/store/routing_observations.gointernal/store/routing_persistence_test.goValidation
Rebased on top of current
main(post-#19). On our CI runner:Fork context: this PR carries the routing/cost fix on top of the InputInflation three-part accounting already in
main; thecachedPortion/writeTokensdecomposition preserves that accounting and adds the rolling create-count and observedWriteTokens override on top.