feat(auto-routing): add model-switch telemetry to decision logging#4623
Merged
Conversation
Contributor
Code Review SummaryStatus: No Issues Found | Recommendation: Merge Executive SummaryReviewed the additive model-switch telemetry (commit 1, previously covered), the shared constraint-predicate refactor and file-modality fix (commit 2), and the mobile touch-target tweak (commit 3); both prior review findings are resolved in the current code and no new issues were found. Files Reviewed (10 files)
Previous Review Summary (commit 2d94747)Current summary above is authoritative. Previous snapshots are kept for context only. Previous review (commit 2d94747)Status: No Issues Found | Recommendation: Merge Executive SummaryReviewed the additive model-switch telemetry fields ( Files Reviewed (7 files)
Reviewed by claude-sonnet-5 · Input: 50 · Output: 10.2K · Cached: 1.4M Review guidance: REVIEW.md from base branch |
St0rmz1
reviewed
Jul 20, 2026
St0rmz1
reviewed
Jul 20, 2026
St0rmz1
approved these changes
Jul 20, 2026
The auto_routing_decision event could not measure real model switching: sticky:false fires even when nothing changed, and ~1% success sampling decimates within-session switch sequences. - incumbentModel, switched, switchReason (threshold/cost/capability, derived inside computeDecision), and routeChanged (previous route key now stored alongside the sticky model) on every decision log line - switches bypass the success sample rate; sampled:false marks bypassed lines so downstream rate math only scales sampled rows - contextTokens from constraints.promptTokensEstimate when present
…ty detection Extract modelSatisfiesConstraints/satisfiesRequiredModalities/contextProvablyTooSmall into decision-engine so the coding-plan short-circuit and benchmark routing no longer duplicate the policy. Drop the bare `file`-key modality fallback that misclassified structured tool output as a document request; typed input_file/document cases still cover the real provider shapes.
Add hitSlop={2} so the 40pt affordance meets the 44pt guideline, matching the
mic/composer control convention.
iscekic
force-pushed
the
feat/auto-routing-decision-telemetry
branch
from
July 20, 2026 23:47
2d94747 to
46e1560
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #4611; retarget to
mainonce it merges.Problem
We cannot measure real model switching in kilo-auto/efficient today: the
auto_routing_decisionlog event has no incumbent field (sticky: falsefires even when the model did not change), and successful decisions are sampled at ~1%, which decimates within-session switch sequences. This telemetry gates planned policy improvements (threshold hardening, near-miss hysteresis, route-flip debounce).Changes
incumbentModel— the sticky model considered for this decision (null when none).switched— the incumbent existed and the decided model differs from it.switchReason— returned bycomputeDecisionon the decision (additive optional contract field):threshold(incumbent denied, off the route, or below the accuracy bar),cost(the mode's switch condition triggered),capability(ejected by feat(auto-routing): modality and context-size aware efficient routing #4611's modality/context filters), null when there was no incumbent or it was kept.routeChanged— this decision's route key vs the incumbent's; the sticky cache entry now stores the route key alongside the model. Null when unknowable (no incumbent, or a pre-routeKey cache entry — old entries still parse).switched: truealways log;sampled: falsemarks bypassed lines (also retries/failures) so downstream rate math only scales sampled rows.contextTokens—constraints.promptTokensEstimatewhen the gateway sent it (available since feat(auto-routing): modality and context-size aware efficient routing #4611), null otherwise; no new plumbing.Notes / assumptions
best_accuracymode the accuracy-gap switch condition is reported ascost— it is the same "eligible incumbent, but the switch pays off" branch as the cost factor.Verification
services/auto-routing: vitest 138 passed (new coverage: switchReason threshold/cost/capability, routeChanged incl. legacy sticky entries, sampling bypass on switch, drop of routine unsampled decisions), typecheck, lint clean.packages/auto-routing-contracts: vitest 83 passed, typecheck, lint clean.