Rewrite skills for nnsight 0.8 and expand to 18 skills - #5
Merged
Merged
Conversation
The existing skills targeted pre-0.7 nnsight and would mislead an agent on 0.8: `.value` after save, `.output[0]` on blocks that now return tensors, `LanguageModel`, `with tracer.all():`, `model.generator.output`. All of it is rewritten against 0.8, and a test suite executes every example so it stays that way. Foundation skills: - nnsight (renamed from nnsight-basics): SKILL.md plus a references/ tree covering the execution model, access and modification, batching, generation, gradients, caching and scan, control flow, source tracing, per-architecture module paths, and full API tables. Adds scripts/inspect_model.py, which reports module paths, block-child execution order, and tensor-vs-tuple outputs without downloading weights, and scripts/check_env.py. - nnsight-debugging: triage table, error catalogue reproduced against 0.8, and a pre-0.8 porting guide. - nnsight-remote (renamed from remote): sessions, non-blocking and async jobs, transfer reduction, setup and troubleshooting. New technique skills: ablation, attention-analysis, circuit-discovery, probing, sae-and-dictionary-learning, model-editing-and-lora, interp-experiment-design, nnterp, vllm, diffusion-and-multimodal. Testing: tests/ extracts every fenced python block from every skill and runs it against real models, with directives for skip / setup / remote / gpu / slow / expect-error. 221 blocks execute against gpt2, SmolLM2, pythia and llava; remote blocks run against a local NDIF when NDIF_HOST is set. tests/test_structure.py enforces frontmatter, Codex symlinks, manifests, link resolution, and bans pre-0.8 API from runnable examples. CI runs the CPU-safe subset. Several findings came out of running the examples rather than writing them: attribution patching anti-correlates with real patching for whole-layer interventions (r = -0.357) and recovers to +0.999 for single positions; the IOI circuit is sufficient but not complete, and a random head set already holds 65% of the metric; a sentiment probe scores 100% at layer 0; a toy SAE reports 0.999 explained variance with 2041/2048 features dead. Each is documented with the measurement rather than as a general caution. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The nnterp skill documented `enable_attention_probs=True` as broken on GPT-2. The cause was a transformers 4.x -> 5.x change, not nnsight 0.8: GPT-2's `eager_attention_forward` moved from `module.attn_dropout(attn_weights)` to `nn.functional.dropout(...)`, so the nnsight source operation is now named `nn_functional_dropout_0` rather than `module_attn_dropout_0`. Fixed upstream in nnterp by trying both spellings. Replaces the known-incompatibility note with working, executed examples of reading and assigning attention probabilities, plus guidance for diagnosing the same class of failure on other architectures. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three things that would have bitten a fresh clone: - CI installed `nnsight>=0.8`, which does not exist on PyPI (latest release is 0.7.0), so the workflow would have failed on its first run. Install the 0.8 branch from git until it ships. - CI did not constrain transformers, and these skills require 5.x. In 4.57 a GPT-2 block returns `(hidden_states,)` rather than a plain tensor and its attention dropout is `module.attn_dropout(...)` rather than `nn.functional.dropout(...)`, so the `.output` and `.source` examples are wrong there. Pinned in CI and stated in the README, CLAUDE.md, and the access-and-modify reference. - `make test` sets NDIF_HOST by default, so remote blocks failed rather than skipped for anyone without a local deployment. The host is now probed once and treated as absent when unreachable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Answers the question the skills repo could not: does any of this actually help
an agent use nnsight, and what does it cost? The independent variable is which
resources the agent gets; the dependent variables are whether it solves the
task and the tokens, dollars, time and turns spent getting there.
Resource conditions cross {nothing, skills, nnsight docs, website tutorials,
nnsight source} with a delivery mode. Agentic mode is the honest test: skills
load natively through Claude Code's --plugin-dir, docs/tutorials/source are
reachable through Read/Grep/Glob, and the agent must navigate. Static mode
pastes the material into the system prompt instead, which separates "the
content is right" from "the routing works". Tools are read-only, so the
measurement is documentation-driven code generation rather than
iterate-until-green.
80 tasks: 33 code (ported from nnsight's tests/agent-evals to 0.8), 32 MCQs,
and 15 debugging tasks where the agent gets broken code plus the real symptom.
The debug set is drawn from failure modes reproduced against 0.8, and several
are silent — no exception, just the wrong answer.
Porting audited the old suite: 10 of its 32 MCQs had answers that are wrong on
0.8 (MissedProviderError is gone, tuple item-assignment now raises, the
END/EXCEPTION mediator events were removed). Those were rewritten with the
superseded answer kept as a distractor, since it is what an agent working from
stale material picks.
Every code and debug task carries a reference solution, and `evalkit.audit`
runs all 48 through the real runner with no LLM calls — that is what catches an
unsatisfiable verifier or a dependency upgrade breaking a canonical pattern.
It already caught two of my own tasks (an embedding-transfer prompt one token
too short).
Reporting gives pass rate with Wilson intervals, tokens and dollars per solve,
latency, a failure taxonomy, and which files each condition's agents actually
opened. Runs are appended as JSONL so a sweep is resumable and cost-capped.
A 12-run smoke sweep on Sonnet: none 25%, docs 75%, skills 100%, at $0.015 /
$0.142 / $0.110 per solve. Four tasks, so the intervals are wide — the full
grid is 3,360 runs and roughly $320.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The claude-code backend authenticates with the local Claude Code login, so a subscription is never billed per token. The CLI reports total_cost_usd anyway — it is the API-equivalent price of the tokens, not a charge — and presenting it as "$0.88 spent" was misleading. Tokens are now the primary budget signal: --max-tokens caps a sweep, --dry-run leads with a token estimate, and every dollar figure is labelled $-equiv. The more serious fix is error classification. On a subscription the realistic way a long sweep dies is hitting a usage window, and previously every cell after that point would have been recorded as a failed task — a wall of zeros indistinguishable from the resource suddenly not working. Account-level errors (usage limit, 429, expired login) are now classified as `limit`, the sweep stops, and the offending cell is left unrecorded so --resume retries it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two harness defects found by running 224 cells, both biased against the resource conditions: - The CLI sometimes completes successfully with an empty `result` while the answer sat in an assistant text block. Fall back to the streamed text. - Separately, it intermittently produces no text anywhere — tool calls, a stop, nothing else. That is a flake, not a wrong answer, so retry once before recording; a persistent empty answer is now an agent error rather than a silent task failure. Between them these cost 5 of 224 runs, every one in a resource condition and none in the baseline, which would have understated exactly what the testbed exists to measure. First sweep (32 MCQs x 7 conditions, Sonnet, 11.2M tokens, 23 min): none 88%, tutorials 94%, source/skills/everything 97%, docs/docs+tutorials 100%. Every interval overlaps. The finding is that MCQs do not discriminate — a capable model clears them from parametric knowledge alone — so the code and debug tasks are where the conditions have to be compared. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Caught in review, not by the harness. All three were ported from the 0.7-era suite and kept answers that the 0.8 source contradicts: - eproperty: the decorated stub body is NOT dead. `__get__` calls it as the preprocess — `value = Mediator.value(location)` then `value = self._preprocess(obj, value)` — and its return value is what the user reads. The old "never executed, only donates __name__/__doc__" description is now the distractor. - PYMOUNT mounts `save` alone (`mount(save, "save")`), not `.save()` and `.stop()`. - edit(inplace=True) stores a Mediator on `envoy._edits`, cleared by clear_edits(); `_default_mediators` no longer exists. The eproperty question is the instructive one: every condition picked the wrong answer, so it scored 100% everywhere while actually rewarding stale knowledge. A question whose right answer is what old documentation says inflates every condition equally and measures nothing. Results for all three are dropped and re-run. Verifying an MCQ against source is now as load-bearing as running a reference solution, and unlike reference solutions nothing automates it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
19/21 correct after the fix, versus 21/21 "correct" when the keyed answers were the stale ones. The two misses are both `tutorials`, which is the condition least likely to describe descriptor internals — a plausible failure rather than a uniform one, which is what a working question looks like. Final MCQ sweep, 224 runs on Sonnet: none 88%, tutorials 88%, source 97%, skills 97%, everything 97%, docs 100%, docs+tutorials 100%. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Measured after the first sweep: the keyed answer was the longest choice in 29/32 questions and letter B in 29/32. Either heuristic alone scores 91%, which is essentially where the no-resources baseline landed (88%). The MCQ half was measuring string length and letter position. All 32 rewritten with choices of comparable length and detail, and the answer position rotated (A=5 B=10 C=8 D=9). The keyed answer is now the longest in 7/32 (22%, below chance) and no question exceeds a 1.6x internal length spread. `evalkit.audit --mcq-bias` enforces all three so it cannot regress. Every answer was re-verified. A fourth factual error turned up while checking: the scan question was keyed to an unsaved local raising, but at module scope it survives — the UnboundLocalError only happens inside a function, which the question now specifies. Confirmed by running both scopes. Reference solutions still 48/48. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
336 runs, 48 tasks x 7 conditions, Sonnet, 26M tokens, 111 minutes.
pass 95% CI tok/solve
none 44% 31-58% 1,499
source 77% 63-87% 170,305
tutorials 92% 80-97% 116,080
docs 94% 83-98% 88,286
skills 94% 83-98% 77,101
docs+tutorials 85% 73-93% 91,462
everything 88% 75-94% 79,363
Unlike the MCQ half, this separates: the baseline interval (31-58%) does not
overlap docs, skills or tutorials. Skills and docs tie on accuracy; skills gets
there on 13% fewer tokens and fewer turns.
The failure taxonomy explains the whole result. The dominant failure is the
pre-transformers-5 `.output[0]` idiom — the agent unwraps a tuple that is now a
tensor, then indexes a 2-D result three ways. Share of runs emitting it:
none 48%, source 38%, tutorials 31%, docs+tutorials 23%, docs 21%,
skills 10%, everything 10%
and IndexErrors: none 7, source 9, docs+tutorials 4, tutorials 3, docs 2,
skills 0, everything 0.
Two results worth not overselling: combining resources scored *lower* than the
best single one in both cases (docs+tutorials 85% vs docs 94%; everything 88%
vs skills 94%), with overlapping intervals — consistent in direction across
both pairs, but n=48 and one repeat cannot establish it. And reading the source
is the worst-performing resource as well as the most expensive, which is a
statement about what source code is for, not about its correctness.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Opus grid reported every code and debug task failing — 0/33 even with skills, where Sonnet had scored 94% on the same tasks. It was not the model and not the harness: the nnsight checkout that provides the editable install had been switched from 0.8 to a feature branch that does not export TransformersModel, so every task died on `from nnsight import TransformersModel` before reaching the agent's code. MCQs kept passing, because they execute nothing — which is exactly what made the failure look like a model collapse. run.py now imports TransformersModel and prints the resolved nnsight path before the first cell, and exits with the branch hint if that fails. One second of checking against an hour of runs recorded as task failures. The 64 valid Opus MCQ records are kept; the 101 code/debug records — all of them the same ImportError — are quarantined under results/archive/ rather than deleted, since they are the evidence for this fix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
560 runs, 80 tasks x 7 conditions, 36.4M tokens, 185 minutes. code+debug (n=48) mcq (n=32) none 58% 78% source 94% 100% tutorials 96% 88% docs 94% 100% docs+tutorials 92% 100% skills 98% 84% everything 98% 100% On code and debug — where the agent consults its resources 95-99% of the time regardless of condition — skills is the strongest resource, and every resource is far above the 58% baseline. The MCQ column is confounded, and the confound is in my harness. Consultation rate on MCQs by condition: docs 97%, tutorials 97%, docs+tutorials 94%, source 88%, everything 69%, skills 53%. The file-based conditions get a system prompt saying material is on disk and to use Read/Grep; the skills condition gets "skills are installed — use them". The first reads as an instruction, the second as an offer, and invoking a Skill is a heavier action than a Read. So half the skills MCQ runs answered from parametric memory and scored like the baseline: consulted 94% vs did-not 75%. That is a property of my prompt, not of skills. The MCQ half of this grid should not be quoted per-condition until the prompts are matched and it is re-run; the code+debug half is unaffected because consultation there is near-universal. Also worth recording: three MCQs the skills genuinely cannot answer because the material does not cover them — eproperty semantics and the mediator event protocol have zero mentions across all 18 skills, and PYMOUNT appears only as one row of a config table. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…d two at-scale recipes Written from what actually broke while running the eight remote tutorial notebooks against a live NDIF; every added block is executed by the suite. Corrections. "Variables from outside the session are unavailable inside" is backwards — every name the block reads is captured from the enclosing scope and pickled into the payload, which is how a steering vector or a label tensor reaches the server at all. The real constraints are that they must be picklable, they arrive on the CPU, and edits stay server-side unless saved. An outer container does come back if you save it from inside the block. "Imports are whitelisted" named a list with no counterpart in either codebase, and omitted transformers/accelerate/peft/diffusers, which are assumed present. Restated as what the source supports: it has to be installed on the server, `_SERVER_MODULES` is what `remote="local"` simulates, and `nnsight.compare()` diffs the rest. New rules: take device and dtype off an activation at run time (an undispatched model reports `meta`, and the server may shard across cards), and a shipped class needs `super(MyClass, self)` because it is recompiled outside any class body. New sections in sessions-and-jobs: loading the dataset on the server rather than pickling a memory-mapped Dataset into the request, and running an optimizer loop inside one session. Both follow the same principle the file already teaches — upload the minimum, download the minimum. Also fixes an unrelated pre-existing failure in the nnsight skill: the attention-pattern example unpacked a tuple from `.save()` inside the block, so neither name was bound to the saved object and neither came back. Confirmed to fail identically on nnsight 78e871ba, before this branch's changes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
From six subagent runs against the repos alone.
`model.tokenizer.batch_decode(model.tokenizer(x).input_ids)` returns ONE string
on transformers 5.x — input_ids is a flat list of ints, so batch_decode reads it
as a single sequence. It appears in attention-analysis, activation-patching,
attribution-patching, logit-lens and nnsight/references/gradients.md, and it is
the first line anyone writes for this kind of analysis. It usually fails
silently: the "where a head looks" loops just print one garbage row. Replaced
with a per-id decode everywhere.
model-editing-and-lora advertises training an adapter through a frozen model,
which is exactly what an agent selects for "train a LoRA", and its recipe cannot
work remotely — both ways it fails are silent. A client-side optimizer over a
shipped module leaves the client's .grad None forever and reprints the same loss
to the last decimal; `.to(model.device)` on an undispatched model is `.to("meta")`,
which discards the weights while the job still returns COMPLETED. Both are now
called out there, with the diagnostic (print a parameter norm), and the skill
links nnsight-remote, which it didn't.
nnsight-remote's review table gains those two as rows, and its remote="local"
paragraph is cut back to what that mode actually establishes — that the block
survives the round trip.
attention-analysis gains a correct head-ablation section: cut before c_proj,
because slicing attn.output is 21x off, and establish a null distribution first —
at a middle layer the top-attending head is often indistinguishable from one with
~0.0006 attention. ablation cross-links per-head-attention.
135 tests pass, 228 blocks executed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
For a model too big for one GPU, sharded across several with transformers TP under torchrun. Covers the two rules SPMD puts on intervention code (no rank-dependent control flow; seed before sampling, which is a correctness requirement rather than hygiene), which values are actually sharded, and a diagnosis table for the symptoms -- an activation at 1/N width, a run that hangs with no error, generated text that differs per rank. The vllm row's "tensor parallelism" claim is narrowed to match: with two multi-GPU paths an agent needs to tell them apart, so each skill now ends by saying when to prefer the other. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…sion say Two claims went stale with nnsight's tensor-parallel fixes. `moe_tp_experts` was listed among the unsupported expert-parallel styles. It needs no gather at all -- its forward all-reduces, so both sides arrive whole -- and refusing it cost Mixtral, DeepSeek-V3, Qwen3-MoE and around twenty-five other shipped configs. Most mixture-of-experts checkpoints work; the page now says which styles genuinely don't. Adds the transformers >= 5.15 requirement, which is not optional and fails in a way worth naming: below it a tied LM head is gathered but never sharded, so logits come back tp_size times too wide with a correct argmax inside the first copy. Also fixes the diagnostic for a fragment-width activation -- the flag moved from `model.interleaver.enabled` to `model.interleaver.fragments.enabled` when the gather became a collaborator shared with vLLM. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sibling to `tensor-parallel`: the other answer to "this model does not fit". Covers the names that go in the dtype slot (nf4/int4/4bit, fp4, int8/8bit, fp8), what a trace sees, the compute dtype and why int8 differs, and the two things that surprise people -- a 4-bit `.weight` is a packed uint8 blob, and the memory saved is well short of what bytes-per-weight predicts because embeddings and the LM head stay 16-bit. All four code blocks execute (`test: gpu`) against gpt2 on a real card, so the Linear4bit class name, the activation dtype, the packed weight shape and the compute-dtype override are checked rather than asserted in prose. The accuracy and memory table is from Llama-3.2-1B, measured. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Was written as untested. Verified since on Llama-3.3-70B-Instruct at nf4 across 4 A100s: transformers shards the packed weights and the gather still returns gate_proj.output at its full 28672 rather than one rank's 7168. 43.3 GB across four cards against ~141 GB for bfloat16. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tracer.result is served on the vLLM path now — the finished RequestOutput, with per-step values still coming from model.logits/model.samples — so the skill no longer tells the model it parks the worker forever. generate is not a plain alias for trace either: in a `with` block it traces, without one it just runs and hands back the outputs. Adds the registration section: one install instead of one per prompt, values arriving on output.saves, the prefix-caching requirement, and the async with / aclear form. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Follows nnsight's rename. Adds `clear_edits`, the `serve=url` form, the `tracer.result.saves` name-collision note, and the barrier limitation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… ones Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Found by agents following these skills verbatim during an interpretability sweep; each was believed and acted on. - `nnsight/references/generation.md` presented bounding the loop as the fix for trailing code being dropped after `tracer.all()`. A bounded `iter[:N]` drops it too whenever the model stops early, and `max_new_tokens` is only ever an upper bound -- so for a reasoning model, which never runs to the cap, bounding can never be the fix. The separate empty invoke always works. Also documents that `iter` step 0 is the *prefill*: an intervention inside the loop lands on every prompt position at once on step 0 and on one token afterwards ([(1,10,768), (1,1,768), (1,1,768)] for a 10-token prompt), which is not stated anywhere else, and drops a stale `chat.model.layers[10].output[0]` that the same skill's own `inspect_model.py` contradicts. - `model-steering/SKILL.md` said a vector inside a `tracer.iter` loop is "re-added at each generated token". It is re-added on every forward *pass*, and pass 0 is the prefill -- use `iter[1:]` to steer only the generated tokens. - `probing/SKILL.md` and `sae-and-dictionary-learning/SKILL.md` collect activations without `torch.no_grad()`. A trace runs with autograd on, so saved activations arrive with a live `grad_fn` pinning the forward graph: 3.6x peak memory here, and these are exactly the recipes people scale to a corpus. `.detach()` on the saved tensor does not help -- the graph is already built. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ler wording vllm: the engine runs eagerly unless taps=[...] keeps CUDA-graph replay for the declared locations (syntax, non-tap refusal, in-place edits, clone rule, measured numbers); tensor parallelism gathers via fragments with the fused-projection rank-order and DCP notes; chunked prefill is off by default and a chunked traced prompt is refused; no VLLM_ALLOW_INSECURE_SERIALIZATION needed; the spawn claim dropped; PP and speculative decoding listed as unsupported. tensor-parallel: .source values are handed over as-is (no runtime warning); moe_tp_experts is a partial at the handoff that nnsight reduces; tests live in tests/tp/. nnsight/execution-model and generation, debugging/error-catalogue: the per-module controller rather than forward hooks; present-tense phrasing; the three vLLM engine refusals catalogued. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rallel refs
Every block used to be test: skip. They now run under the suite on a GPU
(SmolLM2-135M-Instruct, one engine per file; Qwen2.5-0.5B for tp=2) and were
verified on vLLM 0.27.1 / nnsight 0.8: 10 executed, 4 compiled-only (serving
needs a server; MoE and hybrid name larger checkpoints).
Rewritten around what breaks first, from an evaluation where seven agents
worked from the docs alone: the flat [tokens, hidden] rows, the (hidden,
residual) layer-output pair and its sum, clone-what-you-keep on the eager
engine, where tensors live; tracer.result as the last read; step 0 is the
prefill and upstream writes go at the top of an iter body; slots merge and
append does not; passing values between invokes with two traces (patching
demo); n>1 container layout; model.edit() sweeps with the measured cost of a
model reference inside the block; graph taps with the hybrid-trunk pin and
tp=8 numbers; nnsight-serve's forwarded flags, boolean form, /health and
no OpenAI routes; the logit lens via logits_processor; gate.output as a
(logits, bias) pair. Depth moved to references/{graph-taps,serving,
parallel-and-architectures}.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
model.edit(name=...) and the edits= request argument on trace/invoke/plain generate and over serve; executed block verified on the 0.8 branch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Also: trailing code that needs tracer.result goes in a separate invoke, not after the with block, where the result is unreadable.
…stead An over-running tracer.iter loop warns and is cut short (blocks now assert the short length, the unbound tail, and the warning text instead of expecting OutOfOrderError); the batched-write row rule is stated as an unchecked constraint, and the vLLM wrong-rows refusal demo is replaced by the plain warning that a wrong-height write can take the engine down; the error catalogue and triage tables route the symptoms (short result, late shape error) instead of the removed messages. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ustom dict plan loads unchecked
…monstrates the refusal
Cold review of PR #7 found three rewritten sections that still print tables without asserting what the prose claims, plus two garbled sentences: - model-steering function vectors (finding 2, MAJOR): both blocks now collect their decodes and assert the counts and named rows — the vector hits exactly {Portugal, Austria, Thailand}, exactly {Poland, Greece, Sweden} echo the country name, the random control moves exactly 3/7 outputs, the few-shot control fails exactly {Portugal, Brazil} (' Rome' / ' Buenos'), and the Portugal coincidence (an FV hit on a few-shot failure) is asserted directly. - activation-patching layer x position map (finding 3): the subject assert now splits early (L0-L4, max crosses zero) from late (L8-L10, none does), and the suffix band and ' of' column are pinned flat at baseline within 0.15 through L6. - attribution-patching validate (finding 4): r_last > r_subject becomes r_last > 0.99 and |r_subject| < 0.5, pinning the +0.999 / +0.168 headline instead of a bare ordering. - error-catalogue (finding 6): "no warning and no warning" -> "no warning and no error". - vllm SKILL (finding 7): the self-contradicting run-on about engine-side warnings vs build failures rewritten and rewrapped; no behavior claims changed. All strengthened blocks re-run green locally (model-steering 7/7, activation-patching 9/9, attribution-patching 5/5 blocks ran); test_structure 119 passed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A fully-cut path fails loudly; a bypassed cut silently drops that module's contribution from every upstream gradient — measured 45% relative error on a bit-identical forward. From the cold review of the website PR.
nnterp does only the label read now: no load-time cross-check, no fp16 refusal (NaN is back to being the user's to check — validation passes vacuously on it), and check_renaming=False disables the accessor. The module's returned weights stay documented as a label-free second reading to sanity-check against.
0.8 skills audit: every block executes and asserts what it claims
nnsight 0.8 now refuses tracing keypoint-matching (its unit input is a pair of images, which the list convention would split) with pointers at model.pipe([image_a, image_b]) and a model.image_processor-built encoding. Block gate re-run: 17/17 blocks pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
nnsight 0.8 inverted which side of the collision moves. On a module whose own child is named `output`, the child is now `.E_output` and `.output` keeps its usual meaning; `.nns_output` is gone. Where the old advice inverted along with the rule, it is replaced rather than renamed. access-and-modify said to check `print(model)` for a `.output` that looks like a module, which can no longer happen, so it points at the `E_output/output` label the repr prints instead. inspect_model.py built its note from the names it detected but printed a fixed `.nns_output / .nns_input`; it now names the attributes it actually found. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…nd missing The stress sweep of nnsight 0.8 caught the plugin stating four things that are not true and leaving out five that cost agents real time. Wrong, and inherited from the docs: `generate` is not greedy by default — it uses the checkpoint's `generation_config`, and Qwen3-8B and Llama-3.2-1B-Instruct both sample from it; a vLLM decoder layer's `output[1]` is the residual after this layer's attention, not the stream entering it; a VLM sweep *can* be batched over the chat route, so it is one forward pass and not twelve; the PEFT example's `adapted.model.layers[16].output[0]` is wrong twice over; and the in-place SAE attach form writes the attachment's result into its own input, so backward through it raises. Wrong, and the skills' own: causal tracing does not need 7B parameters — Llama-3.2-1B gives a textbook two-site trace on the skill's own prompt, so the requirement is a confidently-known fact and enough depth, not a size; FLUX.2 reads Qwen3 hidden states 9/18/27, so the `[-2]` recipe is a no-op there; Qwen3-VL's adapter is not "(none)" and zeroing only the merger leaves the answer unchanged; whisper runs its encoder twice; base `NNsight` has no `.scan()`; `-tp 2` prints two "Ignoring unknown argument" lines, and `n > 1` over serve silently returns one sequence's saves. Missing: `envoys=` and `eproperty` appeared nowhere, so the path reference gains a worked per-head accessor; the path table had no MoE, SSM or hybrid rows and stated the tensor-vs-tuple rule as if it generalized; nothing said wrapping mutates the module for the rest of the process; `.skip()` was absent from the vLLM skill although it can take the engine down; and `skip` advancing the run past the module was unstated. `scripts/inspect_model.py` gains `--task` and `--trust-remote-code` (a repo with no `pipeline_tag` died on task inference, a remote-code one on an interactive prompt), prints the task beside the class it built, descends into MoE blocks — naming the router and experts, and the first non-dense layer — and stops diagnosing every scan failure as data-dependent control flow. The batching sections state the rule from nnsight#722: a leading dim that is a whole multiple of the batch size is scoped, and a write to a layout that cannot be scoped warns. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
skills: correct the claims the sweep disproved, and cover what it found missing
The marketplace is named ndif-team, so `/plugin install nnsight@skills` never resolved — the first command a new user runs. A structure test now pins the install line to the manifests so it cannot drift again. Codex reads `.agents/skills`; only `.codex/skills` was here, so add the second symlink tree and check both. Also say that skills load from their description rather than by name, add a verify step to each install block, and reduce the transformers 4.x explanation to the requirement (it is already in CLAUDE.md). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rule 3 already tells an agent to clone before saving on vLLM. The env var is the engine-wide form, so it belongs there rather than in a section of its own: what it does, that it is read in the worker and so must be set before `VLLM(...)`, and that it costs in-place edits — with the assignment form to use under it. No new code block: the vLLM skill's blocks run on hakone against a real engine, and this is a process-level setting that would need its own engine to demonstrate. Shipped in nnsight by #662 (merged as cb5a1ac5 on 0.8).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The existing skills targeted pre-0.7 nnsight and would actively mislead an agent on 0.8 —
.valueaftersave,.output[0]on blocks that now return plain tensors,LanguageModel,with tracer.all():,model.generator.output. This rewrites all of them against 0.8 and adds a test suite that executes every example, so they can't drift back.Skills
Foundation
nnsight(renamed fromnnsight-basics) — SKILL.md plus areferences/tree: execution model, access and modification, batching, generation, gradients, caching and scan, control flow, source tracing, per-architecture module paths, and full API tables. Two runnable scripts:inspect_model.pyreports module paths, block-child execution order (registration order is not execution order), and tensor-vs-tuple outputs without downloading weights — a 27B model in ~8s;check_env.pyreports versions, GPUs, NDIF key/host, deployed models, and the local-vs-NDIF package diff.nnsight-debugging— triage table, an error catalogue reproduced against 0.8 (exact messages), and a pre-0.8 porting guide.nnsight-remote(renamed fromremote) — sessions, non-blocking and async jobs, transfer reduction, setup and troubleshooting.New technique skills —
ablation,attention-analysis,circuit-discovery,probing,sae-and-dictionary-learning,model-editing-and-lora,interp-experiment-design,nnterp,vllm,diffusion-and-multimodal.Rewritten —
logit-lens,activation-patching(absorbs DAS),attribution-patching,causal-tracing,model-steering(absorbs function vectors).Testing
tests/extracts every fencedpythonblock from every skill and executes it against real models. Blocks in a file share a namespace and run in document order; directives control execution:221 blocks execute, 0 failures (gpt2, SmolLM2-135M-Instruct, pythia-70m, llava-interleave-0.5b). Remote blocks run against a local NDIF when
NDIF_HOSTis set and skip cleanly otherwise.tests/test_structure.py(105 checks) enforces frontmatter, Codex symlinks, manifests, link resolution, and bans pre-0.8 API from runnable examples. CI runs the CPU-safe subset.Two constraints the harness surfaced, both documented in
CLAUDE.md: nnsight needs the trace block's source on disk (so blocks run viarunpy, notexec), and a shared namespace can mask a false claim about an unassigned variable.Findings from running the examples
Several results came out of execution rather than authoring, and changed what the skills say:
Known issues worth a look
nnterp'senable_attention_probs=Truefails on GPT-2 + transformers 5.15 (expectsmodule_attn_dropout_0, the op is nownn_functional_dropout_0). Documented with a workaround.nnsight.__version__reports0.7.1.dev41+g...on the 0.8 branch, while these skills say "0.8" throughout — worth a tag before release.vllmexamples are compile-checked only (no vLLM in the test environment), and diffusion examples likewise — the available tiny SD pipeline has a mismatched CLIP config.🤖 Generated with Claude Code