Shunt submission hand-off — RouterArena (sub_10) - #205
Open
KookaS wants to merge 3 commits into
Open
Conversation
Author
|
/evaluate |
…ion reproducible + CI-green - router module: SHUNT_PIN constant + seed-dir-first resolution (router_inference/router/shunt_seed/ preferred, env/artifact root fallback), typed _PRIOR (fixes mypy Collection[str] index errors on 3.9-3.12), no unused _import_shunt assignment (ruff F841); import registered in __all__ (ruff F401). - shunt_bench_prior: _embed_many callable narrowing (mypy 'None' not callable), drop zip(strict=) for py3.9 mypy, SPDX header, no codespell hits. - bundle router_inference/router/shunt_seed/ (sample + embeddings verbatim, results filtered to the menu models) + REPRODUCE.md + .codespellrc so a clean checkout at the pin regenerates byte-identical predictions. - predictions files untouched.
…nch_prior) Prediction choices are now all deepseek/deepseek-v4-flash, produced by Shunt's real session_cascade engine at the pinned commit; answers reused from the recorded cache (no re-inference).
Author
|
/evaluate |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Shunt (https://github.com/KookaS/shunt, @KookaS, open-source) routes every
prompt through its real
session_cascaderouter engine (pinned Shunt commit7ee66af): each prompt is passed toRouterEngine.decide, then mapped to thechosen arena model. RouterArena's stateless single-turn / no-verification regime
provides no verified-outcome channel (no repo to re-test), so Shunt's
verified-failure escalation ladder never fires and the router deterministically
stays on its cheapest live model — deepseek/deepseek-v4-flash (809/809
sub_10, 420/420 robustness).
This is an honest deployability probe: it exercises Shunt's shipped routing
machinery end-to-end against RouterArena's harness, and its score should be read
as deepseek-v4-flash's standalone accuracy/cost under that harness — not as
evidence of Shunt's routing value, which requires the verified-outcome channel
RouterArena does not provide. No RouterArena data was used to train or tune
anything. Reproducibility instructions are in
REPRODUCE.md.