docs: correct iteration, backward, skip, edit-attach and generate claims

[JadenFiotto-Kaufman]

Documentation only — no code changes. Nine findings from the 0.8 agent stress sweep, each one a page promising something the implementation does not do. Every claim below was re-measured in this worktree unless noted.

What was wrong, and what changed

tracer.iter off-by-one (item 9). Inside a loop starting above step 0 the step pin relaxes after the body's first request, so a read that goes backwards in model order silently binds to the next occurrence instead of raising OutOfOrderError as the same body does in a plain trace or under iter[0:...]. Measured on gpt2: baseline h[0] norms [54.691, 61.350, 53.869, 54.052], and iter[1:3] reading h[6] then h[0] returns [53.869, 54.052] — labels 1 and 2 holding steps 2 and 3. When the skew runs off the end of the run it surfaces as the "never reached" warning, which reads as "the loop asked for a step the run did not make" and points at min_new_tokens=; a descending step list (iter[[3, 1]]) is cut short the same way although the run made step 1. New section in docs/gotchas/iteration.md, both snippets run.

docs/usage/iter-all-next.md's "Whatever iteration was before the loop is restored on exit, so loops can nest" is the same mechanism read backwards: what is restored is a relaxed pointer, so a read after a nested inner loop binds to the step the inner loop left the model on. Measured: an inner iter[[3]] inside an outer iter[1:2] makes the outer read return step 3's value. Sentence corrected and linked to the gotcha.

Gradient checkpointing (item 10). One gotcha line in docs/usage/backward-and-grad.md: torch.utils.checkpoint runs the segment twice, the block served the first pass, and the recompute — the one autograd differentiates — runs unmodified, so the forward output is right and the parameter gradients inside the segment are wrong with nothing to signal it. Turn checkpointing off while tracing.

Nested backward blocks (item 17). One gotcha line: a backward block inside another dies with ValueError: cyclic parent chain from inside autograd. Re-ran the repro here — sibling blocks work (create_graph=True, retain_graph=True on the first), both the model case and the plain-tensor case raise.

.skip() on vLLM (item 15). New section in docs/models/vllm.md plus a Limitations bullet: a skip has to tile every row of the step, and a step's rows hold other traces', other tenants', and decodes whose block already finished, so with more than one request in flight the skip cannot cover it; the ValueError is raised outside the per-request deferral, so it takes the engine down (EngineDeadError, every client's in-flight requests lost) rather than the request. Whether a script survives is the scheduler's choice. docs/usage/skip.md gains the same limitation and the reason its pass-through idiom kills a vLLM engine: a decoder layer is called forward(positions, hidden_states, residual), so .input is the positions vector — read args, kwargs = layer.inputs instead. Not re-run here (no vLLM engine to spare on a shared card); taken from the sweep's six logged reproductions and the source.

Per-head attribution (item 23.1). attribution-patching.md told you to reshape attn.output[0], which is the output of c_proj, where the hidden dimension no longer decomposes per head — its own sibling page refutes it in a warning box. Replaced with the projection input (attn.c_proj.input / o_proj.input) or .source.attention_interface_1.output[0]. Verified on gpt2 with the page's own prompt pair: the two correct routes agree bitwise (max abs diff 0.0), and the old route disagrees in sign for 70 of 144 heads and names a different top head, (9, 8) against (6, 9).

In-place module attach (item 23.2). layer.output[:] = layer.adapter(acts, hook=True) writes the attachment's output into the tensor that is its own input, so the backward the pattern exists for raises RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation. Confirmed by running repro_4_inplace_backward.py: in-place fails, the replacement form (layer.output = ...) backprops. Fixed at edit.md:94/:103 and sae-and-auxiliary-modules.md:358, and at the same idiom in cache.md:225 and gotchas/integrations.md:97 — the latter re-run here on the non-inplace model.edit() path it documents: the attachment's internals stay observable and adapter.inner.weight.grad arrives.

"Greedy by default" (item 23.3). False: generate uses the checkpoint's generation_config, and gemma-3-1b-it, Llama-3.1-8B-Instruct and Qwen3-4B-Instruct-2507 all ship do_sample=True in it. Ran gemma-3-1b-it: two default generates differ, do_sample=False makes them identical. Fixed in generate.md (front matter, intro, the section, the comparison table), CLAUDE.md, and the three pages that repeat the claim: reference/api-quick-reference.md, usage/pipe.md's comparison table and reference/version-history.md.

cache(include_inputs=True) (item 23.5). Documented as behaviour, not promised as a fix: the device= move reaches tensors and containers, not tensors inside arbitrary objects, so a decoder block's past_key_values keeps the model's live DynamicCache on the compute device. Re-measured on gpt2, batch 32 x ~80 tokens, 8 traces held: 8 MiB of GPU growth without it, 1320 MiB with it, about a fifth more time per trace; the memory returns when the caches are released. The gotcha bullet claiming the cache moves tensors to CPU is qualified.

Batching row rule (follow-on to #722). docs/concepts/batching-and-invokers.md still stated the old exact-match rule in three places (the intro, the narrow/widen bullets, the gotcha). Rewritten against src/nnsight/intervention/batching.py on fix/batching: a tensor is scoped when its leading dim is total or a whole multiple of it (k = shape[0] // total, narrowed at [start*k, size*k)), anything else is served whole, a replacement for such a value is dropped with a warning and an in-place edit to one is caught by torch's version counter at the next narrow/widen. The page's acknowledgement that a shape coincidence can mislead now reads as the accepted cost of the modulo rule. Verified by running that branch's own TestFlattenedRows and TestUnscopedWrites (6 passed) from a git archive of fix/batching in a scratch directory — no other worktree touched, no stash.

Tested

No code changed, so no test run applies. Every snippet I wrote or corrected was executed against this worktree's src (PYTHONPATH=<worktree>/src) on gpt2 or gemma-3-1b-it, except the vLLM ones — I did not build an engine on the shared GPU, and those rest on the sweep's reproductions plus the source.

Deliberately left alone

• The declined code fixes behind items 9, 10, 15 and 23.5 — the pages describe the behaviour as it is.
• Both defects survive only in the two pages routed to other agents: "greedy by default" at docs/models/transformers-model.md:115 and :347, and the in-place attach idiom at docs/concepts/envoy.md:157. Every other occurrence is fixed here.
• docs/models/vllm.md:99's "output[1] the residual stream entering it" is measurably the wrong residual per the sweep, but it belongs to skill item 24 and I left the wording untouched.
• Item 23.4 (docs/concepts/envoy.md) is another agent's.

🤖 Generated with Claude Code