Add a rollouts viewer preset with model, outcome and stage columns - #707
Add a rollouts viewer preset with model, outcome and stage columns#707v-positronic wants to merge 1 commit into
Conversation
A manual rollout round records the endpoint that served each episode, the operator's verdict, and the progress rung she marks as the arm reaches it. The `rollouts` preset derives those into columns and groups the round by endpoint, so a reader sees which model produced which outcome. Ticket: Positronic-Robotics/internal#1063 #open Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f7a983fc9b
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| stage=rollout_stage_label, | ||
| stage_rank=rollout_stage_rank, |
There was a problem hiding this comment.
Pair the stage label with its rank for sorting
When an operator clicks the Stage header, the frontend sorts the visible stage cell itself, so these plain labels are ordered alphabetically; the separately derived stage_rank is neither returned as a table column nor associated with stage. Return the stage as the supported [rank, label] cell value (or otherwise expose the rank as its sort value) so the ladder ordering is actually used.
Useful? React with 👍 / 👎.
| model=rollout_model, | ||
| outcome=rollout_outcome, | ||
| stage=rollout_stage_label, | ||
| stage_rank=rollout_stage_rank, | ||
| items=rollout_items, |
There was a problem hiding this comment.
Replace shared rollout field literals with constants
Rule hardcoded-keys violated:
The derived fields model, outcome, stage, stage_rank, and items are written in this Derive and then addressed again by bare string literals in the episode table and grouping code; define each shared name once and use those constants at every producer and consumer.
AGENTS.md reference: AGENTS.md:L7-L8
Useful? React with 👍 / 👎.
A manual rollout round records three things the existing presets do not show: the endpoint that served each episode (
inference.policy.label), the operator's verdict (eval.outcome), and the progress rung she marks as the arm reaches it (theprogress.statesignal).positronic.cfg.server rolloutsderives them and lists a round with them:| # | Duration | Task | Model | Outcome | Stage | Items | Started |
Outcomeis a badge — Success green, Fail red, Safety amber, the rest neutral; an unscored episode readsUnscored.Stageis the highest rung reached, in the operator's words, withstage_rankcarried alongside so the column sorts by the ladder rather than alphabetically.A
modelsgroup table leaderboards the round by endpoint: episodes, successes, success rate, and how many reached the target, filterable by task.ProgressStageis the ladder: itsvalueis the code a rollout records, itslabelis what the table shows, and its declaration order is the ranking. The codes mirror the platform repo'srollouts_contract.progress.Stage, which positronic does not depend on, so they are spelled once here beside the code that reads them — as are the rollout statics the platform console writes.__main__now dispatches{'finetune': ..., 'rollouts': ...}. Nothing invokedpython -m positronic.cfg.serverwithout a sub-command — no unit, no script — so the added level breaks no caller;positronic-server, the entry point everything uses, is untouched.The dataset composes
ds.local_allrather thands.local: a round is a directory of per-run datasets, and the operator's scores live in the edit logs beside them, so the whole tree has to be loaded foreval.outcometo be present at all.Verified
positronic/server/tests positronic/cfg/tests)./api/episodesreturns['__index__', '__duration__', 'task', 'model', 'outcome', 'stage', 'items', 'started'], every row carrying a real endpoint label, verdict and rung.Ticket: Positronic-Robotics/internal#1063