Skip to content

Add a rollouts viewer preset with model, outcome and stage columns - #707

Open
v-positronic wants to merge 1 commit into
mainfrom
rollouts-viewer-table
Open

Add a rollouts viewer preset with model, outcome and stage columns#707
v-positronic wants to merge 1 commit into
mainfrom
rollouts-viewer-table

Conversation

@v-positronic

Copy link
Copy Markdown
Collaborator

A manual rollout round records three things the existing presets do not show: the endpoint that served each episode (inference.policy.label), the operator's verdict (eval.outcome), and the progress rung she marks as the arm reaches it (the progress.state signal).

positronic.cfg.server rollouts derives them and lists a round with them:

| # | Duration | Task | Model | Outcome | Stage | Items | Started |

Outcome is a badge — Success green, Fail red, Safety amber, the rest neutral; an unscored episode reads Unscored. Stage is the highest rung reached, in the operator's words, with stage_rank carried alongside so the column sorts by the ladder rather than alphabetically.

A models group table leaderboards the round by endpoint: episodes, successes, success rate, and how many reached the target, filterable by task.

ProgressStage is the ladder: its value is the code a rollout records, its label is what the table shows, and its declaration order is the ranking. The codes mirror the platform repo's rollouts_contract.progress.Stage, which positronic does not depend on, so they are spelled once here beside the code that reads them — as are the rollout statics the platform console writes.

__main__ now dispatches {'finetune': ..., 'rollouts': ...}. Nothing invoked python -m positronic.cfg.server without a sub-command — no unit, no script — so the added level breaks no caller; positronic-server, the entry point everything uses, is untouched.

The dataset composes ds.local_all rather than ds.local: a round is a directory of per-run datasets, and the operator's scores live in the edit logs beside them, so the whole tree has to be loaded for eval.outcome to be present at all.

Verified

  • 52 tests pass (positronic/server/tests positronic/cfg/tests).
  • Against a local copy of the 3 September round (54 episodes), /api/episodes returns ['__index__', '__duration__', 'task', 'model', 'outcome', 'stage', 'items', 'started'], every row carrying a real endpoint label, verdict and rung.
  • Serving the two-day round (90 episodes): three endpoints at 30 episodes each, outcomes 24 Success / 63 Ran out of time / 2 Fail / 1 Safety, and 26 episodes reaching the target.

Ticket: Positronic-Robotics/internal#1063

A manual rollout round records the endpoint that served each episode, the
operator's verdict, and the progress rung she marks as the arm reaches it.
The `rollouts` preset derives those into columns and groups the round by
endpoint, so a reader sees which model produced which outcome.

Ticket: Positronic-Robotics/internal#1063 #open
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f7a983fc9b

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread positronic/cfg/server.py
Comment on lines +224 to +225
stage=rollout_stage_label,
stage_rank=rollout_stage_rank,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Pair the stage label with its rank for sorting

When an operator clicks the Stage header, the frontend sorts the visible stage cell itself, so these plain labels are ordered alphabetically; the separately derived stage_rank is neither returned as a table column nor associated with stage. Return the stage as the supported [rank, label] cell value (or otherwise expose the rank as its sort value) so the ladder ordering is actually used.

Useful? React with 👍 / 👎.

Comment thread positronic/cfg/server.py
Comment on lines +222 to +226
model=rollout_model,
outcome=rollout_outcome,
stage=rollout_stage_label,
stage_rank=rollout_stage_rank,
items=rollout_items,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Replace shared rollout field literals with constants

Rule hardcoded-keys violated:
The derived fields model, outcome, stage, stage_rank, and items are written in this Derive and then addressed again by bare string literals in the episode table and grouping code; define each shared name once and use those constants at every producer and consumer.

AGENTS.md reference: AGENTS.md:L7-L8

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant