Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions REPRODUCE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# Shunt submission — RouterArena (sub_10) — how to reproduce

Router **shunt** (@KookaS, https://github.com/KookaS/shunt, open-source) is a thin
adapter over Shunt's real router engine: every prompt is routed through
`RouterEngine.decide` under Shunt's shipped `session_cascade` strategy over
Shunt's live pool (`deepseek-v4-flash`, `deepseek-v4-pro`). RouterArena's
stateless single-turn / no-verification regime provides no verified-outcome
channel, so Shunt's verified-failure escalation ladder never fires and the
router deterministically picks its cheapest live model
(`deepseek/deepseek-v4-flash`).

## Pin

The router code was generated against Shunt at this commit (Shunt is not on
PyPI — install from git):

```
SHUNT_PIN = 7ee66af5ddcbd78886a58e5336b21ffc5291da56
pip install "git+https://github.com/KookaS/shunt@7ee66af5ddcbd78886a58e5336b21ffc5291da56"
```

or point `SHUNT_SRC` at a Shunt `src/` checkout of that commit.

## Regenerate the routing decisions

From this repository root (a Shunt install or `SHUNT_SRC` must be available):

```
uv run python router_inference/generate_prediction_file.py shunt sub_10 --no-optimality
uv run python router_inference/generate_prediction_file.py shunt robustness --no-optimality
```

The recorded answers in `predictions/shunt.json` are the models' own outputs
(provider generations are not byte-reproducible); the router CHOICES are
deterministic. The two prediction files are required by RouterArena's workflow
(exactly one base file + one `-robustness` file per PR).

## Honest framing

This is a model baseline produced by Shunt's real `session_cascade` machinery:
without a verification channel the router stays on its cheapest live model.
Training/tuning on RouterArena data was not used — the engine routes from Shunt's
shipped configuration only.
17 changes: 17 additions & 0 deletions router_inference/config/shunt.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{
"pipeline_params": {
"router_name": "shunt",
"router_cls_name": "ShuntSessionCascade",
"models": [
"deepseek/deepseek-v4-flash",
"deepseek/deepseek-v4-pro"
],
"default_model": "deepseek/deepseek-v4-flash"
},
"description": "Shunt (https://github.com/KookaS/shunt, open-source) running its shipped `session_cascade` strategy over its live pool (deepseek/deepseek-v4-flash, deepseek/deepseek-v4-pro) via the real RouterEngine. RouterArena's stateless single-turn regime gives Shunt no verified-outcome channel (no repo to re-test, no confirmation step), so the verified-failure escalation ladder never fires and the router stays on its cheapest live model (deepseek/deepseek-v4-flash) — the measured QA outcome that the cheap rung dominates. Provider routing uses the native DeepSeek API for the menu models.",
"affiliation": "@KookaS",
"github": "https://github.com/KookaS/shunt",
"homepage": "https://github.com/KookaS/shunt",
"contact": "https://github.com/KookaS/shunt/issues",
"open_source": true
}
Loading