NDIF support: probe, findings, and upstream fix

[elliottower]

Can the model panel run on NDIF instead of rented GPUs? Short answer: not today, and the reason is worth recording.

scripts/ndif_feasibility/probe_ndif.py runs eight probes against the live service, each declared as data so the emitted JSON carries its own repo_id, input string, hook expression and expected outcome. Results in results/ndif_feasibility/.

What it found

• NDIF provisions models absent from its published catalogue on request, with no special access — so the catalogue is not the constraint
• LanguageModel._remoteable_model_key serializes only {repo_id, revision}, so the server rebuilds every model as AutoModelForCausalLM and no masked LM can be provisioned. Every RNA foundation model is a masked LM.
• past that, every encoder model fails during graph execution with RemoteException: name 'hooked_output' is not defined. Envoy._handle_overloaded_mount synthesizes a class named <cls>.Preserved per instance — a dot in __name__, never bound in a module, so it cannot be pickled. It fires for any module defining .output, which is every BERT- and ESM-family layer. Decoder-only models never take that path, which is why they work and encoders do not.

The control that isolates it: BERT reaches the server through the unmodified path (it registers as BertLMHeadModel) and fails identically, so the failure is the encoder module tree, not the workaround used to get masked LMs provisioned.

Verification

nnsight_repro/test_envoy_overloaded_mount.py reproduces it offline on two tiny public models in ~14 s, no cluster and no API key. modal_verify_patch.py installs one checkout on Linux and runs identical checks on main and on a patched branch, main being the negative control.

• nnsight's own tests/test_tiny.py passes on both
• RNABERT and SpliceBERT envoy classes go from unpicklable to picklable
• per-position embedding shift under a single-nucleotide substitution is identical to four decimals on both branches, so nothing about the computation changes

Upstream

Both fixes are submitted as ndif-team/nnsight#700.

Consequence for the panel

None in the short term — the panel stays on rented GPUs, which is where the modal_*.py scripts already run it. Even with remote execution fixed, multimolecule has to be in the serving image for nine of the models. This is recorded so the question does not get re-litigated from scratch.