A single bad endpoint kills the whole sampled eval run

[vertix]

What happens

One unresponsive endpoint in a sampled blind eval kills the entire run, and the in-progress episode with it.

InferenceClient.infer gives up after infer_timeout (180 s) and raises TimeoutError. Nothing between
there and the top catches it: it propagates out of harness.py:329 (actions = self._session(...)) through
World.run, every background process is torn down, the open episode is discarded, and the operator is back
at a shell prompt. Restarting costs another sync of the output directory plus a full warmup — around 15
minutes on 6 Aug — before the next episode can run.

This happened twice in a row on 6 Aug with gyros-nm-n60-50k (16:07 and 16:19, both dying ~184 s after the
endpoint was sampled). Two operator hours were spent on two crashes and no episodes. The other two endpoints
in the set were healthy the whole time and could have carried the round.

Why warmup doesn't catch it

SampledPolicy._get_keys (policy/base.py:264) reads every sub-policy's meta in parallel, and for a
RemotePolicy that opens a throwaway session and handshakes (policy/remote.py:113). So warmup does force
each server to answer ready.

But warmup never sends an observation. An endpoint that reports ready before its weights are usable
passes warmup and then stalls on the first real inference — which is exactly what both crashes looked like.
The throwaway session is also closed immediately after the handshake, so any per-connection warm state the
server built is dropped before the first episode opens its own session.

Scope

This is not specific to one endpoint. Every endpoint-side fault we have hit this month has been fatal to the
whole run:

• 6 Aug — nm_n60_50k accepts the session, never answers the first inference. Run dies at 180 s. Twice.
• 3 Aug — a curie endpoint sent a 2525 KiB observation into Modal's 2 MiB per-message cap; the connection
  closed mid-episode and the run died.
• 5 Aug — a subtler variant: episode 40 ran its full 145 s with no commands recorded at all (2.7 rad of
  joint travel against 50–82 for a real episode). Nothing failed loudly, so the operator scored it a
  legitimate zero and it counted against the endpoint that drew it.

So the class is: the harness treats any endpoint fault as fatal, except when it doesn't notice at all.

What "progress" should mean

Open for discussion — the shape matters more than the details:

1. Quarantine rather than crash. On a timeout or connection error, drop that endpoint from the sampled
   set for the rest of the run and carry on with the others. A round with two of three endpoints is worth far
   more than no round.
2. Retry once with a fresh session before quarantining, since a half-open socket is not the same as a
   dead backend.
3. Warm with a real inference. Send one observation per endpoint during warmup and exclude any endpoint
   that fails or exceeds a budget. Problems then surface before the operator starts, not 40 minutes in.
4. Mark the episode as a rig fault, not a policy result. The in-flight episode should be recorded as
   discarded, and an episode with no commands (the 5 Aug case) should never reach the scoring UI as a normal
   zero.
5. infer_timeout should be reachable from the CLI. production passes headers to each RemotePolicy
   but not the timeout, so today adjusting it needs a code change.

Two constraints any design has to respect:

• The eval is blind. The operator must be told that an endpoint dropped out — otherwise they cannot
  reason about a round that suddenly has two arms — but must not be told which one.
• BalancedSampler counts per key. A key set that shrinks mid-run needs defined behaviour, or the
  remaining endpoints' balance silently changes meaning.

Why it is worth doing now

The Runway rounds are becoming routine — four this week, new endpoints every few days, and the endpoints are
deployed by a team we don't control and cannot debug. Endpoints that are half-broken on arrival are the
normal case, not the exception, and each one currently costs an operator session.
Due: 2026-08-11

[v-positronic]

2026-08-07 — a fourth instance, and the first where the fault is a bad COMMAND rather than a bad connection.

The operator pressed Start on blind batch batch_20260807-103142. The sampled policy's first action reached the Franka driver, which refused the command shape:

NotImplementedError: Unsupported command {'type': 'cartesian_pos', 'pose': array([12 floats])}
  at positronic/drivers/roboarm/franka.py:336

pimm/world.py then logged Stopping background processes... and terminated all six: both SLCamera.run, Robot.run, Robotiq2F.run, DsWriterAgent.run and EvalUI.run. Zero episodes recorded — the operator's console vanished from her screen mid-session and the whole round was lost, not one episode. The arm re-parked cleanly (brakes locked, FCI off, control released, no errors), so it was not dangerous; it was total.

Why it belongs here rather than in a new ticket: the propagation path is the same one this ticket describes, one layer lower. franka.py's match ends case _: raise NotImplementedError, and a raise inside a pimm control system's run propagates through world.py:443 _bg_wrapper and takes the world down. So the class holds beyond timeouts and connection errors — any raise from any component during an episode is fatal to the batch, and quarantine-on-timeout (option 1 above) would not have caught this one.

The trigger itself is a separate, already-known defect and should NOT be folded in here: the served gyros endpoints return robot_command as a raw wire dict, and nothing decodes it into a typed command before the driver sees it. Fixing that removes this instance; it does not change the blast radius.

The design question this instance sharpens, stated rather than answered. For a run whose recorder and arm are shared across policies, "fail this episode" needs a definition before code:

• (a) The episode. Recorded and scored as a failure against the policy that drew it, or discarded as a rig fault and not counted? Option 4 above says discarded; a command the policy actually emitted is arguably the policy's own failure, which argues the other way. The two answers give different denominators in the round's numbers.
• (b) The sampler. After a fault, does the endpoint leave the sampled set for the rest of the run (option 1), or stay in it and get retried on the next draw? A shrinking key set changes what BalancedSampler's per-key counts mean mid-run — the constraint this ticket already names.
• (c) The operator. What is she told, given the round is blind? "One arm dropped out" is necessary for her to reason about the rest of the session; naming which endpoint is not permitted.

These are contract decisions for the owner of the harness contract, not something to settle in the fix.

Sibling symptom, deliberately not merged: Positronic-Robotics/internal#258 (two ZED cameras opened in the same instant race in the SDK, killing a launch at startup — three launches lost on 2026-08-07). Different fault, different layer, same "any component failure kills the run" property. Also related: Positronic-Robotics/internal#115 (a cartesian_reflex mid-run takes the world down through the same path).

Filed against this ticket rather than a new one because the matter — a per-policy fault must not have the whole batch as its blast radius — is the one already written here. Labelled kind:work, ball:agent in the same turn; this repo carries no project: labels, so it surfaces in the rollouts list through its cross-links rather than a label.

[v-positronic]

2026-08-07, a SECOND trigger on the same rig, and this one is not an endpoint fault at all.

Blind batch batch_20260807-110748 ran two scored episodes, then died at the reset between episodes:

franka.py:238  raise RuntimeError(f'homing failed: {goal.reason or goal.status}')
RuntimeError: homing failed: libfranka: Move command aborted: motion aborted by reflex!
["cartesian_reflex", "joint_motion_generator_acceleration_discontinuity"]

The raise propagates through pimm/world.py:443 _bg_wrapper and all six background processes stop — cameras, arm, gripper, recorder and the operator's UI — so the console vanished from her screen and the remaining 28 episodes of the round were lost.

The log shows repeated cartesian_reflex trips during the episodes themselves (several alone, one as ["joint_reflex", "cartesian_reflex"]) before the homing trip. Which of those were under the policy and which under the reset has NOT been correlated against the episode windows, so no cause is claimed here (rollouts/AGENTS.md).

Why it matters for this ticket's end state: a reflex trip during homing is an ORDINARY physical event on a rig doing contact-rich work — the arm meets the tote, the table, a spoon. Quarantining an endpoint does not address it, and neither does any timeout: the fault is the rig's own reset, and the run it kills is the operator's whole slot. Whatever "fail this episode rather than the run" comes to mean, a failed homing has to be inside it.

Data was not lost: both episodes uploaded complete (both camera videos, frame indices, robot_state.{q,dq,ee_pose}, robot_command.pose, target_grip, meta.json, static.json) and both carry the operator's scores in edits.jsonl. The arm re-parked cleanly on teardown.

Index of the day's residue: Positronic-Robotics/internal#263.

[v-positronic]

2026-08-07 — re-verified against positronic main. The core claim stands: no recovery path exists on
any limb. Three corrections and one confirmation to the description above.

CONFIRMED: infer_timeout is 180 s — positronic/offboard/client.py:19 DEFAULT_INFER_TIMEOUT = 180.0, applied at client.py:76, raising at :82-84. The only guard in the whole run loop is
harness.py:534-537, which catches pimm.NoValueException and nothing else; run() at :499-507
catches BaseException only to seal a telemetry span and re-raises. _end_episode
(harness.py:332-366) is called from :521 with no guard at all, so a fault in the post-episode
homing propagates identically — that limb is structural, not hypothetical.

CORRECTION 1 — item 5 is half wrong. infer_timeout IS reachable from the CLI on the
single-endpoint path: .remote is cfn.Config(RemotePolicy, url=...) (cfg/policy.py:62), a Config
on the class, so --policy.infer_timeout= reaches RemotePolicy.__init__ today, for the same reason
--policy.headers works there. It is unreachable only on production (cfg/policy.py:72-95), which
is a function config. So the item is "make it reachable on the batch preset", not "make it reachable".

CORRECTION 2 — warmup does not read meta in parallel. SampledPolicy._get_keys
(policy/base.py:259-268) is a sequential generator expression over p.meta. The rest of the warmup
analysis holds: remote.py:125-132 opens a throwaway session, reads metadata and closes it in a
finally, and no observation is ever sent — which is why a ready endpoint still stalls on the first
real inference.

CORRECTION 3 — "restarting costs a sync plus a warmup" understates what survives, and the mechanism
matters for item 4. There is no per-episode upload at all: positronic/cli/eval/run.py:240 registers
the output dir with pos3.sync(output_dir, sync_on_error=True) at interval=300, and the crash
safety comes from _internal_main being wrapped @pos3.with_mirror() (positronic/inference.py:96),
whose finally flushes on the error path. Two consequences: the recording_dir upload is registered
WITHOUT sync_on_error (remote.py:185), so traces are not flushed on a crash even when they are
being written; and a SIGKILL/SIGTERM unwinds nothing, so up to 5 minutes of finished episodes are
lost (internal#262).

FIELD EVIDENCE, 2026-08-07: nine of ten batch launches on the lab rig ended without completing, two
of them on a RuntimeError and one on a bare Traceback — 2 episodes recorded in 2h20 of rig time.
This is the ticket's cost, measured (internal#238).

Next concrete check: whether quarantine (option 1) can be made to satisfy the BalancedSampler
constraint without changing what a per-key count means mid-round.

[v-positronic]

2026-08-07 — SECOND instance today, measured, and the first one where it cost a partner's episodes.

batch_20260807-135609, endpoint gyros-p497-fm167k-us, 13 episodes recorded before the run died:

RuntimeError: homing failed: libfranka: Move command aborted: motion aborted by reflex! ["cartesian_reflex"]
  raise RuntimeError(f'homing failed: {goal.reason or goal.status}')

So this is the end-of-episode homing case this ticket already names, not the served-command case: a reflex trip during the RESET killed the whole run rather than the episode. The arm parked correctly at exit (parked_at_exit: True), and the 13 recorded episodes are safe in s3://runway/inference/070826/ — the loss is the remainder of the round, which then needs a fresh launch and a fresh operator sitting.

The KeyError: '/psm_…' lines after it are the benign multiprocessing resource_tracker teardown, not a second fault.

That makes today: one round ended by a served policy's unreadable command, one round ended by a homing reflex trip. Both are single faults ending a whole multi-episode round. On a contact-rich task where the arm reaches into a tote, a reflex trip during reset is an ordinary physical event, so the cost of this is proportional to how long a round is — and the rounds are getting longer.

Next concrete check: whether a failed homing can be retried or the episode abandoned without ending the run, so a round continues from the next episode.

[michaelmanly]

@vertix

Given how many different exceptions are taking down the same outer run loop, I’d probably prove the recovery contract off-rig before burning another operator session.

Could the smallest harness path be run against mocked endpoint/robot components?

If so, I’d test three injected failures first, inference timeout, invalid command, and reset exception , and assert that only the current episode dies, the batch continues, and the logs/artifacts survive. That should tell you whether the containment design works before putting it back on the real rig.