Make a Runway blind-eval round a one-line CLI

[vertix]

Summary

Runway ships batches of checkpoints to Modal every few days and asks for blind rollouts on them. This
branch is what makes that round a one-line CLI, and it holds the presets naming the endpoints currently
under eval.

Endpoints are identified by the operator's name, not the checkpoint they serve.
server.checkpoint_path is identical across two deployments of one checkpoint, so SampledPolicy._get_keys
refused such a set outright ("must be distinguishable by …") and the analysis table would have merged their
episodes into one row. RemotePolicy takes a label, production keys on the name each endpoint already
carries in endpoints, and analysis.model() prefers it. The checkpoint stays recorded for provenance.

production can pass headers. It had no way to, so endpoints behind Modal's proxy were unreachable
through it — only the single-endpoint .remote config could carry credentials. One set of credentials
fronts every endpoint, so the headers reach all of them.

Warmup reads every sub-policy's meta at once. Reading a remote sub-policy's meta opens a session and
blocks until the backend has loaded its model, and the reads were sequential, so warmup cost the sum of
the cold starts (measured 270–602 s each on these endpoints). Now it costs the slowest one.

The episode counter is seeded before warmup samples against it. Warmup opens a session, which already
draws a sub-policy, so the draw has to see what the output directory already recorded — otherwise a resumed
run restarts the balance from zero.

Two rolling presets, one per Runway owner — runway_anton (curie) and runway_ziyi (gyros). They ship a
new batch every few days, so their preset gets repointed rather than a new preset added per batch, and the
operator's command never changes.

Two smaller things: the sampled sub-policy is logged by SampledPolicy for every sampler rather than only
inside BalancedSampler, and a rejected handshake names the URL it was opening — the whole diagnosis when
several endpoints come up at once.

The first commit is unrelated plumbing: the type-check hook builds .venv-typecheck so the ratchet sees a
fixed dependency set, but never points basedpyright at it, so imports resolve against the working .venv and
whichever extras were last synced decide the result. On this machine that was 18 errors on a clean tree; with
the interpreter pinned, 0. It needs no baseline change — main already carries those entries. Happy to split
it out if you'd rather.

Test plan

• pytest: 839 passed, 4 skipped.
• Test: two mock endpoints reporting the same checkpoint_path stay distinguishable by label, and an
  unlabelled endpoint carries no label field.
• Test: three stub sub-policies whose first meta read blocks on a threading.Barrier(3) — it can only pass
  if the reads overlap. Verified it fails (5.6 s barrier timeout) against the previous sequential comprehension.
• Config resolution checked with positronic-inference phail --policy=.runway_ziyi --policy.headers=… --help.
• Run against the live endpoints on the rig: eval rounds on 30 Jul, 3 Aug (Runway's gyros and curie
  batches), reports delivered. runway_ziyi's current three endpoints are not yet run — that is tomorrow's
  round; their hostnames were probed (401 behind the proxy, vs 404 for a name that does not exist).

Note

Conflicts with main after #505's move of raw metadata keys into positronic.keys — mechanical, both
conflicts are string literals this branch also touches (analysis.model, EpisodeCounter.seed_from). Not
rebased yet because the rig runs its eval from this branch in the morning.

[v-positronic]

Closing — half of this is now platform's, and the half that is not travels with the sampler.

The presets do not belong here. runway_anton / runway_ziyi name a customer's endpoints inside the public library, and the rollouts MCP already owns that end: it passes the whole label→endpoint map as --policy.endpoints, records it as the un-blinding map a scorecard joins on afterwards, and probes each endpoint before it launches. A preset naming the round currently under evaluation is a second, staler copy of that in a repo customers read.

The mechanics are real and are not lost. Two things here have no equivalent anywhere: RemotePolicy(label=…) with SampledPolicy keyed on label rather than server.checkpoint_path — without it two deployments of one checkpoint report identical metadata, collapse to a single sampling key, and the balance is wrong for exactly the comparison Runway asks for — and reading sub-policy meta concurrently, since each read can cost a model load. Both are recorded on Positronic-Robotics/internal#387, which moves SampledPolicy and its samplers out of positronic; they land with it rather than here.

Reopen or re-cut against that ticket if the labelling is wanted before the move.