0.8 skills audit: every block executes and asserts what it claims - #7
Merged
Merged
Conversation
The three ways an agent gets stuck with `.source` were all missing, and all three surface as the same `OutOfOrderError` about the model having run past a location — a message that points away from the real cause: - An operation on a branch this config never takes. GPT-2's attention lists 50 operations and runs 28; `transpose_0` is the dead cross-attention key transpose and `transpose_2` the live one. New section, with an `expect-error` block. - The first `.source` on a module landing after something else was read. The "all layers in one pass" example only worked because its requests happened to come in layer order; it now warms each module outside the trace. - A drill asked for after the operation's own `.output`. Also: `F.softmax` and friends are not drillable (a function that names itself has no cell to bind), an assignment's `.output` need not be a tensor, and the labels are anchored to transformers 5.15. Every block now asserts its result rather than printing it — probability rows sum to 1 and scores do not, patterns differ across layers, key length grows across generation steps. 11 blocks, all passing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A whole-value replacement has to keep the rows the block owns; a donor activation captured at a different prompt length is how a patching sweep gets a short one. The new block runs on SmolLM2 and asserts the message. `.scan()` says what it raises rather than sitting in a list of things that are unavailable. Errors from the worker come home as a `RuntimeError` carrying the original type, so match the message; warnings and engine-build failures do not come home at all, which is where to look for a bad `taps=` entry. Under graphs, an un-cloned list of captures is not uniformly wrong: every decode entry holds the last step's values, but the prefill entry is its own buffer, which is why such a list looks partly right. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…default - references/access-and-modify.md said calling an envoy runs its forward with "no hooks". A forward_hook registered on the module demonstrably fires twice: once for the real forward, once for the ad-hoc call. hook= names whether nnsight watches the call. - The same file gains the rule that a replacement has to descend from the value it replaces if gradients are wanted, and the object-id error you get otherwise. - .inputs is assignable and is the only way to edit past the first argument. - api-reference.md had tracer.cache(non_blocking=True); the signature is False. - tracer.result is served after the forward, so it goes after model.output. - NameError alongside UnboundLocalError for an unsaved name at module scope. - The SKILL.md orientation block and the collection example now assert their results instead of printing them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every row of the porting table was executed again, from an imported module under untouched warning filters — the case that made the old table wrong. Deprecations are `nnsight.NNsightDeprecationWarning` now, a `FutureWarning`, so the caveat that a warning only shows in `__main__` is gone rather than softened: it holds nowhere. Two rows that read "silent" have messages, `model.generator.output` and the `with tracer.iter[...]:` block form, and three message texts changed. The table is split by what the reader sees — raises immediately, runs and warns, runs and says nothing — because that is what they have when they arrive. The gradient row said the old `retain_grad()` idiom was broken; it runs and gives the identical gradient, so it is not the first thing to change. `model.generator.output` and `tracer.result` are the same tensor, prompt ids included. The table implied otherwise. Catalogue additions, all reproduced: the two capture-time refusals (a body on the `with` line, a body whose first statement is a `try`), a nested `model.trace`, `return` in a body, a script on stdin, `backward()` under `torch.no_grad()`, the batched row-count refusal and how it arrives through vLLM, vLLM's `scan`, and the barrier's call form — it is `b()`, not `b.wait()`. SKILL.md: the empty-saved-list row is remote-only and now says so, `.save()` marks by object identity rather than by variable name, and the triage table carries the shapes that stop a block before it runs. Every block asserts what it claims rather than printing it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…g subjects The skill was the weakest of the four TP surfaces and the only version-wrong one. - The floor is transformers 5.16, not 5.15, in the frontmatter description and the section heading. The reason given was the 5.14/5.15 tied-LM-head defect; the actual floor is that 5.16 moved the plan onto the model. - `tp_plan="auto"` does not raise `TypeError`. It is accepted, shards correctly, and skips nnsight's degree check, so gpt2 under torchrun loads whole on every rank in silence. The diagnosing row is now that symptom. - `grouped_gemm` and `ep_router` are supported. Both were exercised end to end on Qwen1.5-MoE-A2.7B at ep=2. Only `megamoe_router`, `megamoe_experts`, `moe_identity_expert` and `mla_kv_a_proj` are refused, and a plan containing one produces `UnshardableCheckpoint`, not `UnsupportedParallelStyle`. - The headline edit example raised at tp=2 and tp=4. Clone, edit, assign back, now stated as rule 3 alongside the failing form. Added: the clone rule; `gather`/`shard` with the head-ablation example; parameters as DTensors with the per-rank reduction trap and the `aten.fill_` in-place limitation; `max_tp_size` as a pre-flight, which is the skill's first executed block; the conditional `embed_tokens.output` constraint; the three launch failures; and bfloat16 versus float32 drift. Verified on hakone, 2 and 4 A100s, transformers 5.16.1 / torch 2.9.1. tests/test_structure.py 119 passed; the skill's blocks ran=1 compiled-only=12.
SKILL.md's headline generation rule said to bound the loop; the rule is that a loop must not ask for a step the run does not make, with min_new_tokens= as what makes a bound hold. references/generation.md — rewritten around that rule: the bounded over-run raises, the open loop warns and drops its trailing statements, the empty-invoke form, the empty-body hang, and the one-step-late write. Blocks assert their claims rather than printing them. references/control-flow.md — the pass-through example now feeds a block its own input (the previous one zeroed the MLP and called it a pass-through); the replacement contract covers dtype and device and says the failure arrives as a bare torch error; the stop section adds that the run's result is unreachable afterwards and that saving it first runs the whole pass; a trace body cannot start with try:. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The sweep pattern taught a nnsight.save([]) appended from every invoke and read back by position. That is only safe because every invoke in the example appends at the same module; vary the module and the list comes back in model-reached order with no error. Switched to a dict keyed by the loop variable, with the failing case shown and asserted. Added the padding section: every invoke is padded to the batch's longest input, so [:, -1] is stable and any absolute index is not, and pad positions carry activations whose norm can exceed the real tokens' — both asserted. Rewrote the sharing and barrier sections around what actually decides whether a cross-invoke name is readable: where the reading worker has parked. A barrier is for the consumer that cannot park past the producer, which includes every consumer whose first statement is a write. Added what an over-counted barrier does, since it surfaces as a NameError nowhere near the barrier. Three rows added to the constraints table: a batched write must keep its rows, tracer.stop() halts the shared forward rather than one invoke, and direct input does not combine with invokes. All 12 blocks run and assert. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…t run The eager column of the taps table was tokens per second, which is not a property of the card: an eager engine spends a Python round trip per module call on the driver, so it follows the host's spare CPU (8B plain generation measured at 86 tok/s on a quiet box and 54 on a busy one, where a tapped engine gives 89 on both). It is published as a share of vanilla now, with the measurement that it is vLLM's cost and not nnsight's — plain `vllm.LLM(..., enforce_eager=True)` and `VLLM(...)` generating with no trace measure the same four times over. `VLLM_WORKER_MULTIPROC_METHOD=spawn` is not something you set: vLLM sets it once CUDA is initialized, which dispatching does. What it costs you is the `__main__` guard, on one GPU as much as on eight. `nnsight-serve` gains the two limits that are silent — short flags dropped, and no CLI spelling for `taps=` — and the reference gains an async section: saved names are not pushed into your frame, the stream is consumed once and a second await answers `None`, and one engine belongs to one event loop. Pipeline parallelism now names the error it fails with. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The CROSS_INVOKER row told a porting reader to reach for `tracer.barrier(n)` "when writing across the same module". The module is not the variable, and the real rule — a barrier is for a consumer that cannot park past the producer's location before reading — is homed in `docs/usage/invoke-and-batching.md`. Point there; invokes share values without help. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…neration steps caching-and-scan.md: replace the invented "only modules reached after the call" rule with the ValueError that actually fires; add the per-invoke padding hazard with measured norms and the empty-invoke escape hatch; detach=False as the differentiability switch; a measured cost table showing a cache and a save loop are the same speed (19.0 vs 18.8 ms) and unfiltered is 20x. model-editing-and-lora: the persistent-edit demo printed the same token as the baseline, so it demonstrated nothing - now it changes the prediction and asserts it. Add the two boundaries an edit does not cross. The adapter-in-an-edit example ran at prefill only; put it under tracer.iter[:] and count the steps with a cache (1 of 5 without, 5 of 5 with). Note PEFT's base_model.model re-rooting and the .to_local() requirement for weight edits under tensor parallelism.
Every diffusion block in this skill was `test: skip` behind "verify against your
own pipeline", and three of them were wrong.
The "diffusion lens" section described per-timestep latent decoding. The
published technique (Toker et al., ACL 2024) probes the *text encoder*: read an
intermediate layer, push it into the encoder's final layer norm, render. That
section is now the real thing, executed on stabilityai/sd-turbo, and asserts what
it claims -- the image from layer 21 is closer to the full-encoder image than the
one from layer 0 (mean |diff| 0.0926 vs 0.2296).
It also decoded the wrong tensor. `unet.conv_out.output` is the predicted noise,
not the latent. Across six steps, correlation with the finished image:
latent (unet.inputs[0][0]) 0.362 -> 0.838
conv_out 0.333 -> 0.219
Decoding intermediate latents is now its own section under its own name, with the
`.clone()` the reused latent buffer requires and the guidance-doubling note.
The DINOv3 path `vision.encoder.layer[6]` does not exist; it is
`vision.model.layer[6]`, `[1, 201, 384]`. The block ran for the first time and
caught it.
The image-ablation demo compared `argmax` on a token that does not move, so it
printed 'The' against 'The' and read as a broken technique. Top-5 shows the
distribution moving (This 0.324 -> 0.026) and generation shows the answer
changing ('The color of the image is not specified in the caption.' vs 'The color
is this.').
`model.tokenizer` is populated on every VLM tested; it is `None` for tasks with
no text side, which is the real rule. `scan` names the error it raises
(`GuardOnDataDependentSymNode`) and which families it hits.
New: the traced-run one-step default, the scheduler's denoiser call count (PNDM
makes N+1), the open-`iter[:]` hang, in-loop ordering, the four measured VLM
module layouts, the processor-builds-the-ids rule, and chunked tasks -- one row
per chunk in one forward, the two dict-input tasks, and mask-generation's
refusal.
All 17 blocks execute; the six diffusion ones are `test: gpu`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The catalogue only had the loud half. A `tracer.barrier(n)` whose `n` is larger than the blocks that call it raises at run end without ever releasing; one whose `n` is smaller releases as soon as `n` arrive, so the waiting blocks resume before the producer bound its value and the reader gets a `NameError` on a variable with no visible connection to the barrier. Nothing in the output says "barrier" at all. So the `NameError` row carries that as its second cause, and the barrier row says which way of getting the count wrong is the safe one. Mechanism lives in `docs/usage/barrier.md`; both rows point there rather than restating it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…in-place error The attribution-patching skill's flagship validation step was a divide-by-zero. It patched a whole layer's residual output, which makes everything downstream a deterministic function of a clean state, so all twelve layers returned exactly the clean metric (2.4247, std 1.17e-05). The reported r = -0.357 was the correlation of the attribution scores with float32 rounding error, and "top-3 by real patch: [3, 2, 10]" was topk on a twelve-way tie. The conclusion drawn from it — that attribution and real patching disagree — was a false lesson from a real-looking number. Replace it with a sweep over position slices, which is non-degenerate: four subject-token positions give r = +0.168 (real-effect spread 1.5652) and the last position alone gives r = +0.999 (spread 1.4942). Both were already the neighbouring table rows and both reproduce. The sweep now asserts that the real effects have variance, so the degenerate case cannot silently return, and it is keyed by layer rather than appended so cross-invoke ordering cannot scramble it. `references/gradients.md` claimed `module.output[:] = new_tensor` raises `RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation`. It does not: replacement and in-place give identical loss and identical gradient (5.251406192779541 both ways), and the file's own training loop eleven lines earlier uses the supposedly-broken form. The real footgun is position, not form — a write to a tensor the forward has already passed is a silent no-op on a forward-only trace and an autograd error the moment you add a backward. Both are now shown, the second under `expect-error`. Also adds where gradients are unavailable (inside `model.generate()`, on a frozen model, off the metric's path), the `retain_grad()` alternative, and asserts on the blocks that state a number. All blocks execute: attribution-patching ran=5, nnsight/references/gradients.md ran=12. test_structure 119 passed.
logit-lens - The Llama-family row sent readers to model.model.layers[i].output[0]. A block returns a plain tensor, so that indexes the batch, and the example printed <|endoftext|> for every layer while the harness went green. The row, the example and the "returns a tuple" sentence are gone; the example now asserts its decoded tokens. - Pythia unembeds through model.lm_head. model.embed_out does not exist. - New "Check the wiring" section: lm_head(final_norm(layers[-1].output)) is bit-identical to model.output.logits on gpt2, SmolLM2, Qwen2.5-0.5B and pythia-70m, and off by 51.0 on gemma-2, which softcaps. Executed on gpt2 and pythia, with the softcap reapplication shown as a runnable no-op on gpt2. - Tuned lens: keep the frozen norm and head, learn a translator. Check the norm before concluding the lens does not transfer - a missing ln_f gives ' the' at p=1.0000 on every layer, the same symptom. attention-analysis - attn.output[1] is the probability matrix under eager on every family tested; .source is for the raw pre-cast, pre-dropout softmax. - Anchors: row sums to 1 and an exactly-zero upper triangle, asserted. - Induction detector gains a BOS token and a non-repeating control, and asserts it recovers ARENA's five heads with controls under 0.01. Notes the IOI list disagreement instead of averaging it away. - Sink metric excludes q=0 (A[0,0] is 1 by construction; 0.642 vs 0.615 over 144 heads on a 14-token prompt). - torch.no_grad() on every capture; head_dim off the module, not hidden // n_heads; GQA slicing stated positively; gemma-2 attention softcapping and nnterp's attention_probabilities added. - The two ablation routes now assert they agree. probing - The two controls the skill named but never ran are code: a random direction of the same norm as the floor for the causal claim (probe +5.315, negated -2.950, eight random directions between -0.02 and +1.85 against a +1.129 baseline). - New concept-erasure section: LEACE fitted per site, applied inside the forward pass, with the retrained-probe anchor, a rank-matched random control, and the sequential refit that is needed because a baseline-fitted eraser is the wrong projection downstream (0.462 at the site, 0.974 one block later; sequential holds every block at chance). - no_grad memory stated as the measured pair for this capture (279 MiB against 1293 MiB above the weights) rather than a ratio whose accounting is unstated. Every pasted number in all three files reproduces on nnsight 0.8 / transformers 5.15.0.dev0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…sert every block The attention-probabilities section described an accessor that failed at construction on every family dispatching through ALL_ATTENTION_FUNCTIONS; it now documents the accessor that drills for the interface call, with the ordering rule it imposes (probabilities live inside the attention forward, so they come before attentions_output), the query-head convention that makes probs[:, h] correct on a grouped-query model, the fp16 refusal, and the layer-0 scope of load-time validation. The stated recovery path was wrong twice over -- print_source() on a model that construction never returned, and an operation name where an AttnProbFunction is wanted -- and is replaced by two blocks that run. The plain-nnsight comparison taught `model.model.layers[5].output[0]`, which on transformers 5 indexes the sequence dimension of a block that returns a tensor and hands back (seq, hidden) with the batch dimension gone: the wrong-but- plausible value the page claims nnterp eliminates. The tuple story is stated as measured -- modern decoder blocks return tensors, Bloom and GPT-J blocks return tuples, attention modules return (output, weights) everywhere. Every block now asserts what it claims: shapes against hidden_size/vocab_size, next_token_probs against softmax(logits[:, -1]), rows of the attention pattern summing to 1 with a bfloat16-sized tolerance, the causal mask, and that each write moves the logits. The compose example compared two different prompts under a claim of equivalence and printed False; it compares the same invoke. Dropped: the LanguageModel deprecation-warning note, which describes a warning a subclass never fires, and the transformers-4.57-versus-5 history. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Measured: automatic-speech-recognition populates feature_extractor and tokenizer, audio-classification only feature_extractor. Dropped the scan traceback's function name, which the fake-tensor logger trims and I could not re-confirm. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The skill's headline table promised `fp8` raises on hardware without FP8 support. It does not: below compute capability 8.9 transformers logs a warning, sets `dequantize` and loads bfloat16 with the quantizer still attached. The banner said "verified on one A100" -- an A100 is 8.0, so that row was never covered by it. The banner now names the card the rest of this was measured on, and says plainly that no card available here can verify `fp8` positively. "`int8` is nearly free in accuracy" was half the story. It is at least 2.9x bfloat16's forward, and its float16 compute dtype makes a gradient over a large-magnitude loss entirely NaN -- which silently ends attribution patching on an int8 model. Both are now executed blocks. The accuracy table reported a hidden-state norm, which moves 3% at `nf4` while the argmax under it changes on one token in eight. Replaced with KL and top-1 agreement. "A GPU. bitsandbytes cannot quantize on CPU" is false on 0.50: `nf4` loads and runs there. The `meta` device is the real restriction. New in "What it will not do": MoE experts are never quantized (~3% saved), `dtype="int3"` loads float32, and `load_in_4bit=`/`load_in_8bit=` are transformers 4 spellings whose failure names nothing the caller wrote. The Diagnosing table gains a row for each, and the two rows describing `bytes_per_element` errors are labelled as the server-side sizing path they come from. Also: the `attn.shape` comment said `(1, 11, 2304)` for a 10-token GPT-2 prompt, and "as if unquantized" for a float32 checkpoint whose activations quantizing halves. Every block now asserts. `references/modules-and-architectures.md` listed `torch_dtype=` and no quantization at all, so an nnsight-skill agent hitting an OOM had no pointer here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`max_tokens` is a cap, not a promise, and a loop the run cannot supply is now an error naming what it asked for and what the run made. The remedies are `ignore_eos=True` or `min_tokens=N` (the error text names `min_new_tokens=`, which this path refuses), or `tracer.all()` with the trailing statements after the `with` block. An open loop still warns, from the EngineCore subprocess where `warnings.catch_warnings` cannot see it, which is the only behaviour left that differs from the local path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A bound that stops short of the run's last step completes clean while writing the wrong steps, so the catalogue and the skill both say that a loop body's ordering bug is not something an exception can be relied on to report — check the writes against a per-step baseline. The over-run message is re-quoted from source: vLLM spells the minimum `min_tokens=`/`ignore_eos=True`, no minimum survives a stop string, and trailing code belongs in a separate `tracer.invoke()` rather than after the `with`, where `tracer.result` cannot be read. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
model-steering - The alpha=0.25 row of the coefficient sweep never ran. Steering makes GPT-2 emit EOS after 11 of 12 forward passes, the bounded `tracer.iter[:12]` loop never completed, `ids = tracer.result.save()` was skipped, and the sweep re-printed the previous alpha's generation. "Nothing happens at 0.25" was a stale variable. `min_new_tokens=12` fixes it; alpha=0.25 gives 'released in the UK', not 'released in Japan'. The sweep now asserts that no two rows are identical. - Added a repeated-bigram column as the collapse signal, and replaced the next-token-entropy fluency proxy, which is non-monotonic (5.43 at alpha 0, 5.95 at alpha 2.0 where the output is a loop), with p(unsteered top token), which falls monotonically 0.088 -> 0.0004 and is asserted to. - Function vectors: extended to seven countries with a matched-norm random control. Three hits; the control changes three of seven outputs. Added the few-shot control as a runnable block: five of seven with demonstrations, and `Portugal`, the row the published example showed, is one of the two failures. - The opposite-direction check is now run on the skill's own vector, which fails it. - `tracer.iter[1:]` -> `tracer.iter[1:12]`, with the reason. sae-and-dictionary-learning - L0 is compared against `d_model`, with the rotated-identity null as a runnable block: explained variance 1.000, L0 768 of 768, and on 8,192 held-out GPT-2 activations also 0 dead, cosine 1.000, dCE -0.000, loss recovered 1.0000. Every reported metric except L0 comes back perfect on data it never saw. - Metrics are computed on a held-out split. Added the memorised-dictionary result (8% dead on the fitting set, 69% held out) as the reason. - Collection masks pad positions: 29 of 112 vectors were pad, and a GPT-2 pad residual carries 8.3x the norm of a real one. - Max-activating examples mask before ranking. 327 of 1,678 live features have their unmasked argmax on a pad token, where the left-padding correction goes negative and a `max(0, position)` guard prints token 0 as the answer. - Feature steering was a silent null (base and steered both ' released'). It now picks the feature by max activation and sweeps, and says outright that the dictionary is too degenerate for the tokens to mean anything. - "104 activation vectors" -> the 83 the block actually collects. - The unconditioned "3.6x peak memory" is now the measured pair (1133 MiB against 242 MiB, 64 sequences) with the note that the multiple is workload dependent. - Added the observe-without-splicing edit, which leaves the model's answer unchanged while making `sae.encoder.output` readable. Every block in both files runs and reproduces its published output. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Merge the two dose-response bullets, and name both few-shot failures (Brazil answers with Argentina's capital) so the five-of-seven count reads. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The rule is stated once at collection; feature analysis points back to it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…dence causal-tracing: move the ROME protocol to Qwen3-8B, where the classic two-site structure reproduces from single-state restoration (subject site, layers 6-15, recovery 1.16; last-token site, layers 24-35, recovery 1.03), averaged over ten noise seeds. Add noise calibration, since 3-sigma of the embedding matrix leaves one seed in ten at P=0.93 on this model. Replace the windowed-residual table, which is a single-layer restore relabelled at one position, with severed MLP and attention traces that split the two sites. Record the GPT-2-small measurement that motivates the move: the same protocol there tops out at 3.4% recovery at the subject with no peak. circuit-discovery: scale faithfulness against the reachable floor. Mean-ablating all 144 heads holds the metric at +0.785, so the old denominator inflated every score by 66 points and the ten worst heads scored 85%. On the new scale the circuit is 96%, the random control -2%, and a deliberately wrong circuit of ten layer-0-2 heads is -0.1%, asserted. Prune against the floor (three heads, 93%) and read completeness as 41% of achievable damage. activation-patching: delete the layer-11 "sanity check", which passes for a 1000x-noise donor. Replace with a self-patch that must not move the metric and a matched-norm random donor that swings it +3.6. Reframe the layer sweep as reachability, print the layer x position map that actually localizes, seed the DAS direction, and add the arbitrary-target null to the DAS controls. interp-experiment-design: a constant added to a residual stream is removed by the next LayerNorm (+10000 moves log P by <1e-3), so extreme-value checks need a real tensor. Add the empty-circuit floor to the controls table. ablation: state why the ragged mean-ablation corpus is safe (left padding plus [:, -1, :]) and what breaks it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every invoke's activations are padded to the batch's longest prompt, so an absolute position slice means something different depending on what else is in the batch. Every sweep in the activation-patching skill batches one prompt pair at one length, which is what makes slice(1, 5) and the appended result order safe there; say so where a reader would otherwise generalise it. Also name the empty-circuit floor in circuit-discovery's description, since that is now what the skill is for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Also: trailing code that needs tracer.result goes in a separate invoke, not after the with block, where the result is unreadable.
…stead An over-running tracer.iter loop warns and is cut short (blocks now assert the short length, the unbound tail, and the warning text instead of expecting OutOfOrderError); the batched-write row rule is stated as an unchecked constraint, and the vLLM wrong-rows refusal demo is replaced by the plain warning that a wrong-height write can take the engine down; the error catalogue and triage tables route the symptoms (short result, late shape error) instead of the removed messages. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ustom dict plan loads unchecked
…monstrates the refusal
Cold review of PR #7 found three rewritten sections that still print tables without asserting what the prose claims, plus two garbled sentences: - model-steering function vectors (finding 2, MAJOR): both blocks now collect their decodes and assert the counts and named rows — the vector hits exactly {Portugal, Austria, Thailand}, exactly {Poland, Greece, Sweden} echo the country name, the random control moves exactly 3/7 outputs, the few-shot control fails exactly {Portugal, Brazil} (' Rome' / ' Buenos'), and the Portugal coincidence (an FV hit on a few-shot failure) is asserted directly. - activation-patching layer x position map (finding 3): the subject assert now splits early (L0-L4, max crosses zero) from late (L8-L10, none does), and the suffix band and ' of' column are pinned flat at baseline within 0.15 through L6. - attribution-patching validate (finding 4): r_last > r_subject becomes r_last > 0.99 and |r_subject| < 0.5, pinning the +0.999 / +0.168 headline instead of a bare ordering. - error-catalogue (finding 6): "no warning and no warning" -> "no warning and no error". - vllm SKILL (finding 7): the self-contradicting run-on about engine-side warnings vs build failures rewritten and rewrapped; no behavior claims changed. All strengthened blocks re-run green locally (model-steering 7/7, activation-patching 9/9, attribution-patching 5/5 blocks ran); test_structure 119 passed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A fully-cut path fails loudly; a bypassed cut silently drops that module's contribution from every upstream gradient — measured 45% relative error on a bit-identical forward. From the cold review of the website PR.
nnterp does only the label read now: no load-time cross-check, no fp16 refusal (NaN is back to being the user's to check — validation passes vacuously on it), and check_renaming=False disables the accessor. The module's returned weights stay documented as a label-free second reading to sanity-check against.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The skills half of the 0.8 release audit. The core finding of the whole pass surfaced here: every fenced block was already executed by the harness, but almost none asserted — so a block that ran to completion with a wrong answer stayed green forever. That is how the logit-lens skill printed
<|endoftext|>for all thirty layers with a passing test. Every block this PR touches now asserts the thing it claims, as does every block a skill's description advertises as its headline result.Results that changed when measured
attention_probabilitieson 5 of 7 families; with the drilling fix upstream, all 12 blocks run and assert real probability matrices.tp_plan="auto"bypass, per-rank weight reductions, the one-step diffusion trace, and the vLLM row-count rule are all corrected against measurement; the TP and diffusion skills went from mostly-skipped to fully executed.CI
The workflow now installs nnterp (its skill's blocks could never run on the runner) from the
nnsight-0.8branch, switching to PyPI once the coordinated release ships.Gates:
test_structure.py119 passed;test_skills.pyfully green — 260 blocks on CPU/A6000, 12 vLLM blocks on an A100.🤖 Generated with Claude Code