Skip to content

fix: avoid charging request-time quota hit - #2816

Open
PatilHrushikesh wants to merge 5 commits into
theagentrouter:mainfrom
PatilHrushikesh:fix/quota-request-time-hit-addend
Open

PatilHrushikesh wants to merge 5 commits into
theagentrouter:mainfrom
PatilHrushikesh:fix/quota-request-time-hit-addend

Conversation

@PatilHrushikesh

@PatilHrushikesh PatilHrushikesh commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Description

Fixes token quota accounting for request-time rate-limit checks.

Previously, QuotaPolicy request-time rate-limit entries omitted hits_addend. Envoy Rate Limit Service treats an omitted value as an increment of 1, so a completed request with total_tokens = 28 charged 1 at request time and 28 again when the response completed, resulting in 29 tokens deducted.

This change:

  • Sets an explicit request-time hits_addend of 0 in buildSimpleModelEntries.
  • Sets the same explicit zero addend for bucket-rule and default-bucket entries in buildBucketRuleLimitEntries.
  • Preserves the stream-done addend sourced from dynamic quota_cost metadata, which charges the actual token usage.
  • Adds and updates tests covering simple models, bucket rules, default buckets, client selectors, and stream-done entries.

Validation

  • go test -count=1 ./internal/extensionserver
  • go test -count=1 ./internal/controller ./internal/extensionserver
  • Linter checks passed.
  • Rebuilt and deployed controller and extproc images to the k1 cluster.
  • Live gateway validation confirmed total_tokens = 28, request-time hits_addend = 0, stream-done hits_addend = 28, and an RLS cache increment of exactly 28.

Special notes for reviewers

BackendTrafficPolicy is not changed by this PR. Its token-cost configuration already supports an explicit request cost of 0 and a response cost from AI Gateway dynamic metadata. Ordinary request-count BackendTrafficPolicy limits continue to use their intended request increment.

Signed-off-by: Hrushikesh Patil <hrushi2900@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@netlify

netlify Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for theagentrouter canceled.

Name Link
🔨 Latest commit 1b2e32b
🔍 Latest deploy log https://app.netlify.com/projects/theagentrouter/deploys/6acb28942debd40008ca6631

@PatilHrushikesh
PatilHrushikesh marked this pull request as ready for review October 8, 2026 10:10
@PatilHrushikesh
PatilHrushikesh requested a review from a team as a code owner October 8, 2026 10:10
@codecov

codecov Bot commented Oct 8, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@missBerg missBerg added bug Something isn't working area/quota Rate limiting, quota, token/cost accounting and attribution labels Oct 9, 2026

@AyushSawant18588 AyushSawant18588 left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes LGTM. The test failure seems due to a transient issue can it be checked and rerun?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/quota Rate limiting, quota, token/cost accounting and attribution bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants