Untrusted path reports BrokenPipeError instead of the real error when the host throws into a worker

[JadenFiotto-Kaufman]

An out-of-order access — a common user mistake — is diagnosed correctly on the trusted path and reported as an opaque IPC failure on the untrusted path. Untrusted is the default for real users, so this is the error most people will actually hit.

Reproduction

with model.trace("the " * 64, remote=True):
    lg = model.lm_head.output.save()              # runs LAST
    h0 = model.transformer.h[0].output[0].save()  # runs FIRST -> already passed

path	what the user sees
trusted	OutOfOrderError: 'model.transformer.h.0.output.i0' was requested but the model already ran past it
untrusted	BrokenPipeError: [Errno 32] Broken pipe

The same block in the correct order completes on both paths. Fully reproducible.

Mechanism

From the model actor's log for the untrusted request, the runner does raise the right error:

File "/w/outoforder.py", line 19, in out_of_order
  h0 = m.transformer.h[0].output[0].save()
  value = Mediator.value(location)
  return cls.event(Event.VALUE, location)
  return worker.parent.switch(Pending(event, location, iteration, *rest))
nnsight.intervention.interleaver.OutOfOrderError: 'model.transformer.h.0.output.i0' was requested but the model already ran past it

and then, on the host side:

return driver.pump(connection)
  self.interleave(connection, fn_name, parks, args, kwargs)
  connection.send(("DONE", result))
  self.send_raw(encode(value))
  send_frame(self.sock, data)
  sock.sendall(struct.pack(">Q", len(data)) + data)
BrokenPipeError: [Errno 32] Broken pipe

The host detects the dangling worker and THROWs OutOfOrderError into the runner. The runner raises it and terminates. The host then writes ("DONE", result) to a socket whose peer is already gone, and the resulting BrokenPipeError is what gets reported — the thrown error is never collected.

The misdiagnosis reaches telemetry as well. Two requests, same block, seconds apart:

error_type=OutOfOrderError   ... email=loadtest-trusted@ndif.us     exec_ms=22.12
error_type=BrokenPipeError   ... email=loadtest-untrusted@ndif.us   exec_ms=292.04

so error-rate breakdowns attribute a user error to infrastructure.

Suggested

After THROWing into a worker, the host should treat a subsequent write failure as "the runner died from the error I just threw" and report that error rather than the pipe error — or collect the runner's terminal state before sending DONE. Any error the host itself throws in is already known to it, so it does not need to survive the round trip.

Observed on the Test deployment, ndif 0.8 @ be38d78, nnsight 0.8 @ 3bed88cc.

[JadenFiotto-Kaufman]

Scoping note: this is specific to the THROW path, not a general weakness of the sandbox.

Ran the same A/B across five other user mistakes. Every one reports identically on both paths:

fault	trusted	untrusted
shape mismatch on assignment	RuntimeError: The expanded size of the tensor (768) must match...	same
attribute that does not exist	AttributeError: 'Envoy' object (nor its module) has attribute...	same
index far out of bounds	IndexError: index 999999 is out of bounds...	same
allocate ~160 GB on the GPU	torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 149.01 GiB	same
divide by zero	COMPLETED (inf, as expected)	same

So ordinary exceptions raised inside the block propagate correctly over the runner socket. It is only the case where the host throws into the worker — a dangling mediator, i.e. OutOfOrderError — where the runner dies before the host's next write and the pipe error masks the real one.

Worth noting the CUDA OOM case was handled cleanly on both paths and the actor survived it: the replica stayed RUNNING and a normal trace completed immediately afterwards with the expected result.