[xsuler]

What does this PR do?

Ordinary one-token decode currently reads request IDs and lengths back to the host and recreates block/slot tensors each iteration. Keep scheduler-owned logical lengths and persistent device tables, updating mappings on admission, compaction or KV-page growth. Skip the empty cache-invalidation reduction for adapters without cache invalidation.

EOS uses a bounded pinned flag transfer and an explicit CUDA event. CPU length bookkeeping runs before waiting, and the event guards buffer reuse. Shared CPU cancellation flags are checked using cached CPU request IDs. Completion delivery, stop/length handling, recurrent-slot reset and graph scratch-row handling retain their existing semantics. Speculative decode and NPU retain their legacy metadata path.

Related issue

Fixes #633

Type of change

• ⚡ Performance improvement

How was it tested?

Baseline: latest main 0de39d1b8e507110b6b622d373f71d06729004b5 (includes #635). Compare its isolated inference module against this change with the same resident model, KV cache and graph buckets. The main matrix scripts token choice to keep workloads identical; model forward and TP-selected-token logprob computation remain real. It bypasses greedy argmax, so production greedy measurements are reported separately. GPU: H20-3e, BF16, PyTorch 2.8.0a0+34c6371d24.nv25.08, CUDA 13. Prompt lengths 254–257, 32 output tokens; batch sizes 1/8/64. Graph buckets include 1/2/4/8/64; compilation disabled. Each case warms baseline and candidate once, then alternates ordering for 3 measurements per variant (5 for the seeded 4-layer Qwen3 model); tables show medians from rank 0.

• Same-policy fixed-token logprob abs diff mean = max = 0 in all 36 eager/graph matrix cases (Qwen3 TP1, Ling TP1 and TP4), plus CPU cancellation and live refill cases. Outputs and finish reasons match. The harness asserts atol=rtol=1e-5; observed differences are exactly zero.
• Real greedy Ling TP4, eager and graph, batches 1/8, 16 tokens: outputs, finish reasons and selected logprobs match exactly.
• CPU regression covers block growth, persistent storage, compaction/refill, slot reset/reuse, EOS, length limits, cancellation, and shrinking batches across graph buckets 8→4→2→1. Full suite: 1694 passed, 173 skipped, 12 subtests passed, using PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 ARENO_CHECK_TOKEN_IDS=1 python -m pytest -q tests/ examples/agentic/pi/ -k cpu. Local macOS C++ compilation required replacing a stale SDK path in the compiler invocation; repository sources were unchanged for this environment fix.
• pre-commit run --all-files --hook-stage manual passed.

Measured CUDA Graph performance

steady ignores EOS and reaches the fixed length limit. tail_refill runs twice the active capacity with output lengths 4/8/16/32, EOS, refill, compaction and a shrinking tail. TPOT is decode-window wall time / actual decode iterations; the window includes interleaved refill and final delivery. host wall is time outside model/sample enqueue, including completion waits and delivery, not pure CPU compute. Throughput includes prefill and delivery. All arrows are main → candidate; total speedup is main wall time / candidate wall time.

Model / TP	Batch	Scenario	TPOT ms	tok/s	Host wall ms	Total speedup
Qwen3 tiny / TP1	1	steady	0.621 → 0.414	1376.7 → 1935.8	8.4 → 2.9	1.406×
Qwen3 tiny / TP1	1	tail_refill	1.052 → 0.930	832.2 → 916.6	4.3 → 2.9	1.101×
Qwen3 tiny / TP1	8	steady	0.658 → 0.411	10265.9 → 14844.2	10.1 → 3.6	1.446×
Qwen3 tiny / TP1	8	tail_refill	1.184 → 1.044	4058.4 → 4554.6	20.5 → 14.4	1.122×
Qwen3 tiny / TP1	64	steady	0.810 → 0.476	46287.4 → 60424.0	28.6 → 19.7	1.305×
Qwen3 tiny / TP1	64	tail_refill	1.901 → 1.811	17970.0 → 18729.8	51.5 → 47.2	1.042×
Ling / TP1	1	steady	6.248 → 5.851	120.5 → 126.1	184.7 → 147.8	1.046×
Ling / TP1	1	tail_refill	13.458 → 13.151	58.3 → 59.0	68.5 → 67.3	1.012×
Ling / TP1	8	steady	8.148 → 7.745	616.9 → 633.5	271.5 → 235.8	1.027×
Ling / TP1	8	tail_refill	16.202 → 16.239	264.2 → 262.9	399.8 → 394.1	0.995×
Ling / TP1	64	steady	14.959 → 14.461	1284.3 → 1303.4	772.5 → 701.7	1.015×
Ling / TP1	64	tail_refill	43.274 → 42.801	620.0 → 623.5	1211.6 → 1203.5	1.006×
Ling / TP4	1	steady	6.820 → 6.406	108.7 → 113.7	186.8 → 50.7	1.046×
Ling / TP4	1	tail_refill	14.649 → 14.626	53.0 → 53.4	65.1 → 62.8	1.008×
Ling / TP4	8	steady	8.802 → 8.386	616.6 → 633.7	259.3 → 87.2	1.028×
Ling / TP4	8	tail_refill	17.609 → 17.563	252.9 → 247.8	369.1 → 364.1	0.980×
Ling / TP4	64	steady	10.014 → 9.543	1852.3 → 1951.8	377.0 → 180.6	1.054×
Ling / TP4	64	tail_refill	36.771 → 35.180	776.2 → 805.2	599.0 → 587.6	1.037×

Real greedy sampling, Ling TP4, CUDA Graph, 32 output tokens, batches 1/8/64, 3 repeats per variant (the normal sharded argmax is enabled):

Batch	TPOT ms	tok/s	Total speedup
1	7.535 → 7.102	102.8 → 107.7	1.047×
8	9.592 → 9.150	581.6 → 610.7	1.050×
64	10.768 → 10.091	1802.6 → 1889.8	1.048×

These greedy runs also have exactly matching outputs, finish reasons and logprobs (mean/max abs diff = 0).

Additional Ling TP4 batch-8 scenarios, 3 repeats per variant:

Mode	Scenario	TPOT ms	Total speedup
eager	cancel_refill	59.719 → 62.506	0.958×
eager	continuous_refill	61.411 → 63.053	0.975×
graph	cancel_refill	18.372 → 17.327	1.043×
graph	continuous_refill	17.567 → 17.435	1.007×

The improvement is workload-dependent: Ling graph steady decode improves, while the TP4 batch-8 tail has a 0.980× total speedup. Cancellation/live-refill eager cases have 0.958×/0.975× speedup; these regressions are included above. Eager matrix speedups range 0.972×–1.131×. Peak allocated GPU memory on rank 0, graph batch-64 steady (all graph buckets preallocated): Qwen3 tiny / TP1: 344.730 → 344.731 MiB (Δ +1024 bytes); Ling / TP1: 51784.167 → 51784.168 MiB (Δ +1024 bytes); Ling / TP4: 13507.204 → 13507.205 MiB (Δ +1024 bytes).

Bounded Nsight evidence

Nsight Systems 2025.4.1, Qwen3 TP1, graph batch 8, 64 output tokens; capture starts after both variants' warmup/parity checks. In the interior metadata window of a 63-decode-iteration steady trace:

• D2H transfers: 372 → 0; H2D transfers: 124 → 2 (KV-page updates).
• cudaStreamSynchronize: 496 → 2 in that window.
• GPU idle-gap total: 53.284 → 34.161 ms (35.9% reduction); maximum gap: 0.233 → 0.215 ms. GPU window extends through final completion/output work.
• EOS is explicitly not synchronization-free: a separate candidate tail/refill trace has 78 decode iterations and 77 completion event waits in the interior window. Its interior metadata window contains 172 D2H transfers, including completion flags, admissions, membership changes and delivery. Flag transfer is bounded by the active capacity; pinned storage is reused only after the event. Legacy cancellation/compaction/delivery operations still synchronize when membership changes.

Reproduction (with an editable install of the candidate checkout and a local model):

git show 0de39d1:areno/engine/inference.py > /tmp/baseline-inference.py
NVIDIA_TF32_OVERRIDE=0 OMP_NUM_THREADS=4 torchrun --standalone --nproc-per-node=4 \
  scripts/validation/decode_metadata.py --model /path/to/Ling-3.0-tiny \
  --baseline /tmp/baseline-inference.py --output /tmp/ling-tp4.json
# TP1: use python instead of torchrun; seeded dense adapter: --tiny instead of --model.
# Greedy parity: --greedy --batches 1 8 --tokens 16 --scenarios steady.
# Live controls: --batches 8 --scenarios cancel_refill continuous_refill.
nsys profile --trace=cuda,nvtx --sample=none --cpuctxsw=none \
  --cuda-graph-trace=node --capture-range=cudaProfilerApi --capture-range-end=stop \
  --force-overwrite=true -o /tmp/candidate-trace \
  python scripts/validation/decode_metadata.py --tiny \
  --baseline /tmp/baseline-inference.py --output /tmp/profile.json \
  --batches 8 --tokens 64 --modes graph --scenarios steady --profile candidate
# Repeat with --profile baseline, export both reports to SQLite, then:
python scripts/validation/decode_metadata_trace.py /tmp/baseline-trace.sqlite \
  /tmp/candidate-trace.sqlite --output /tmp/trace-summary.json

Checklist

• The PR title summarizes the contribution.
• Linked the related issue in the description.
• Existing CPU tests pass.
• New behavior is covered by tests.
• Described test commands, numerical tolerances, hardware, performance and remaining synchronization.
• Public API / CLI remains backward-compatible.