diff --git a/README.md b/README.md
index 17502f55..b0ffc7bc 100644
--- a/README.md
+++ b/README.md
@@ -143,7 +143,7 @@ a fixed scale at startup. See the
| Checkpoint | Implementation | Documented validation scope | Guide |
| --- | --- | --- | --- |
-| Ouro-1.4B | Native Ouro | Existing runtime and benchmark coverage under their recorded configurations | [Runtime](docs/runtime.md), [accuracy](docs/accuracy.md) |
+| Ouro-1.4B | Native Ouro | Ascend BF16 full prefill with eager Python execution, paged KV and optional fixed-depth async scheduling. Arbitrary chunked prefill remains unresolved. | [Runtime](docs/runtime.md), [Ascend recipe](docs/recipes/ouro_ascend.md#6-fixed-depth-asynchronous-engine-plan) |
| Ouro-2.6B | Shared Ouro | Shares the Ouro implementation; not separately qualified, and 1.4B quality/performance results are not inherited | [Scope notes](docs/recipes/ouro_thinking.md#6-evidence-and-limitations) |
| Ouro-1.4B-Thinking | Reuses Ouro | Bounded BF16, fixed-four-loop synchronous basic integration | [Shared recipe](docs/recipes/ouro_thinking.md) |
| Ouro-2.6B-Thinking | Reuses Ouro | Bounded BF16, fixed-four-loop synchronous basic integration | [Shared recipe](docs/recipes/ouro_thinking.md) |
diff --git a/docs/recipes/ouro_ascend.md b/docs/recipes/ouro_ascend.md
new file mode 100644
index 00000000..82c3887a
--- /dev/null
+++ b/docs/recipes/ouro_ascend.md
@@ -0,0 +1,138 @@
+# Ouro on Ascend
+
+Run Ouro-1.4B on one Ascend NPU with BF16, greedy decoding and four recurrent
+loops. The Python runtime uses Torch's `scaled_dot_product_attention` through
+the `ascend_eager` backend, matching HF's SDPA entry point. Torch-NPU selects
+the device kernel according to the input shape, mask and hardware.
+The example enables the optional asynchronous engine plan.
+
+## Prepare the environment and model
+
+Install the project dependencies in a compatible Torch, Torch-NPU and CANN
+environment. Use the checkpoint's config and byte-level tokenizer.
+
+```bash
+source /path/to/cann/set_env.sh
+source /path/to/npu-environment/bin/activate
+python -m pip install --no-deps --no-build-isolation -e .
+export MODEL=/path/to/prepared/Ouro-1.4B
+export ASCEND_RT_VISIBLE_DEVICES=0 # Replace 0 with your assigned NPU.
+```
+
+## Serve fixed-depth requests
+
+This example sizes KV for 32 resident requests with 128 input and 128 output
+tokens each. Set `--num-blocks` explicitly and adjust it for the expected
+sequence lengths, resident request count and available device memory. For
+whole-cohort prefill, the chunk size must fit each prompt and the batched token
+budget must fit the sum of prompt lengths in the cohort.
+
+```bash
+TASK_QUEUE_ENABLE=2 OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 \
+python -m vllm_rlt.entrypoints.serve \
+ --model "$MODEL" --tokenizer "$MODEL" --served-model-name ouro \
+ --device npu --dtype bfloat16 --attention-backend ascend_eager \
+ --mode refill --exit-mode ouro_delayed \
+ --async-scheduling --single-stream --no-uva \
+ --block-size 16 --num-blocks 2064 --max-num-seqs 32 \
+ --max-num-batched-tokens 8192 --prefill-chunk-size 1024 \
+ --max-prefill-batches-before-decode 32 --host 127.0.0.1 --port 8123
+```
+
+For synchronous eager execution, omit `--async-scheduling`.
+
+Wait for `/health` to return HTTP 200, then send a completion. Add
+`"stream": true` for server-sent events.
+
+```bash
+curl --fail http://127.0.0.1:8123/health
+curl --fail http://127.0.0.1:8123/v1/completions \
+ -H 'Content-Type: application/json' \
+ -d '{"model":"ouro","prompt":"The capital of France is","max_tokens":8,"temperature":0,"min_loops":4,"max_loops":4,"exit_threshold":1}'
+```
+
+
+## Fixed-depth asynchronous engine plan
+
+- **Scheduling.** Stream events track completion and protect buffer and request
+ lifetimes. Decode PRELUDE and each recurrent loop are separate submissions,
+ so the scheduler can select a new batch at each boundary.
+- **Reuse and batching.** Ascend uses the shared packed-prefill metadata by
+ default with `ascend_eager`. Use `--no-packed-prefill` to select the per-query
+ layout. Greedy sampling and compatible prefill are batched. Full prefill
+ uses causal SDPA.
+ Cached chunks and ragged decode pass boolean masks to preserve each row's
+ visible KV range.
+- **Paged KV.** Each loop writes to the page pool through PyTorch indexed
+ assignment using its own block table.
+ The Ascend backend gathers the visible KV rows for Torch SDPA.
+
+The example uses private, fully reserved `last_exited` KV. Keep graphs,
+static buffers, padding, prefix caching, incremental KV allocation, preemption,
+speculation and prefill/decode disaggregation disabled in this configuration.
+Greedy parity remains unqualified for arbitrary chunked or mixed-length inputs
+and adaptive exit.
+
+
+## Performance and correctness
+
+A higher prefill quota lets more requests reach decode together and can delay
+decode for existing requests. The scheduler default is 1.
+
+### Full unified W1-W7 validation
+
+This historical run used the C++ host executor and explicit eager attention
+on both sides of the comparison. For the SDPA configuration above, compare
+against HF SDPA with the same weights, input IDs, output lengths and concurrency.
+Check the actual prefill and decode kernels on both sides with a short profile.
+Measure throughput, TTFT and TPOT with the profiler disabled. Separate
+synchronous and asynchronous vllm-rlt runs to measure the effect of scheduling.
+
+Runtime `7b4537f` used the same budget 8192, chunk 1024 and prefill quota 32 for
+all seven [unified execution workloads](../unified_execution.md#protocol-and-comparison-rules).
+The comparison ran Ouro-1.4B BF16 with four fixed loops and greedy decoding on
+one 64 GiB Ascend 910B3 per run, with CANN 9, Torch/Torch-NPU 2.10 and
+Transformers 4.54.1. The reference used official HF eager at revision
+`574fa66cb8bf5abdc979642d01cf2b79b16bfab1` with a DynamicCache adapter.
+
+All 42 runs completed with 1,422 successful requests and zero runtime failures.
+Each case used three unprofiled repetitions per arm on the same card and CPU
+affinity. Each repetition admitted closed-loop traffic for 60 seconds and then
+fully drained. Throughput includes drain and excludes startup, compilation and
+warmup. Timing uses the in-process engine actor. W1–W5 ignore EOS. W6/W7 use
+natural EOS with a 512-token cap and a fixed ShareGPT selection.
+
+| Workload | Input / output / clients | Residents per arm | HF eager tok/s | Engine tok/s | vs HF | Complete output matches |
+| --- | --- | ---: | ---: | ---: | ---: | ---: |
+| W1 | 128 / 128 / 1 | 1 | 6.334 [5.284–6.496] | 9.464 [9.403–10.394] | +49.42% | 15/15 |
+| W2 | 128 / 128 / 32 | 32 | 201.314 [200.099–204.437] | 305.496 [293.376–316.662] | +51.75% | 480/480 |
+| W3 | 4096 / 128 / 1 | 1 | 6.254 [6.206–6.268] | 7.569 [6.110–9.215] | +21.03% | 12/12 |
+| W4 | 4096 / 128 / 32 | 4 | 21.211 [21.012–21.580] | 31.312 [31.135–31.405] | +47.62% | 120/132 |
+| W5 | 1024 / 512 / 8 | 8 | 50.885 [50.234–51.362] | 79.791 [76.843–80.666] | +56.80% | 48/48 |
+| W6 | ShareGPT / ≤512 / 8 | 8 | 26.343 [26.060–26.385] | 37.992 [37.906–38.879] | +44.22% | 18/27 |
+| W7 | ShareGPT / ≤512 / 32 | 8 | 36.341 [35.576–36.931] | 36.194 [35.808–36.679] | -0.41% | 45/99 |
+
+Rates are medians [min–max]. Resident limits reflect single-device memory
+capacity and apply equally to both arms. Complete output checks compare token IDs
+and finish reasons against timed or supplemental HF references.
+
+**W1, W2, W3 and W5 passed the covered-input output checks.** W4, W6 and W7
+still differ from HF, so their throughput figures are diagnostic. W1–W5 used
+owned KV and native dispatch throughout timing. W6/W7 used paged KV and Python
+execution. W7 throughput ranges overlap, and its peak allocated memory was
+30.06 GiB for the engine versus 19.54 GiB for HF. At W2, mean TPOT fell from
+157.04 to 102.73 ms while TTFT p95 rose from 315.98 to 616.33 ms.
+
+## Run regression checks
+
+With an assigned NPU visible and `MODEL` set as above:
+
+```bash
+VLLM_RLT_OURO_CHECKPOINT="$MODEL" python -m pytest -q --run-npu \
+ tests/test_ascend_eager.py tests/test_ascend_kv.py \
+ tests/test_ascend_decode_scheduling.py \
+ tests/test_ascend_checkpoint.py tests/test_ascend_serving.py
+```
+
+For broader evaluation, see [accuracy](../accuracy.md) and
+[serving benchmarks](../serving.md).
diff --git a/pyproject.toml b/pyproject.toml
index fa46c0c8..067040f1 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -43,7 +43,10 @@ license-files = ["LICENSE", "NOTICE"]
[tool.pytest.ini_options]
testpaths = ["tests"]
pythonpath = ["."]
-markers = ["gpu: requires an explicitly selected CUDA device and --run-gpu"]
+markers = [
+ "gpu: requires an explicitly selected CUDA device and --run-gpu",
+ "npu: requires an assigned Ascend device and torch-npu/CANN",
+]
[tool.ruff]
line-length = 100
diff --git a/tests/conftest.py b/tests/conftest.py
index f65e8a42..6981e499 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -8,15 +8,21 @@ def pytest_addoption(parser):
parser.addoption(
"--run-gpu", action="store_true", help="Run CUDA tests on explicitly selected devices"
)
+ parser.addoption("--run-npu", action="store_true", help="Run Ascend tests on an assigned NPU")
def pytest_collection_modifyitems(config, items):
enabled = config.getoption("--run-gpu")
if enabled and not os.environ.get("CUDA_VISIBLE_DEVICES"):
raise pytest.UsageError("--run-gpu requires explicit CUDA_VISIBLE_DEVICES")
+ npu_enabled = config.getoption("--run-npu")
+ if npu_enabled and not os.environ.get("ASCEND_RT_VISIBLE_DEVICES"):
+ raise pytest.UsageError("--run-npu requires assigned ASCEND_RT_VISIBLE_DEVICES")
for item in items:
if "gpu" in item.keywords and not enabled:
item.add_marker(pytest.mark.skip(reason="set CUDA_VISIBLE_DEVICES and use --run-gpu"))
+ if "npu" in item.keywords and not npu_enabled:
+ item.add_marker(pytest.mark.skip(reason="use --run-npu on an assigned NPU"))
@pytest.fixture(autouse=True, scope="session")
diff --git a/tests/test_ascend.py b/tests/test_ascend.py
new file mode 100644
index 00000000..fc4209d7
--- /dev/null
+++ b/tests/test_ascend.py
@@ -0,0 +1,170 @@
+"""Opt-in synchronous BF16 Ouro checks using explicit Torch attention on Ascend."""
+
+import pytest
+import torch
+
+from tests.helpers import tiny_ouro_config
+from vllm_rlt import CacheConfig, SamplingParams, SchedulerConfig
+from vllm_rlt.core.kv_cache_manager import KVCacheManager
+from vllm_rlt.device import resolve_device
+from vllm_rlt.engine.stage_engine import StageEngine
+from vllm_rlt.kernels.paged_attention import torch_paged_attention
+from vllm_rlt.models import OuroForCausalLM
+
+pytestmark = pytest.mark.npu
+
+
+@pytest.fixture
+def npu_device():
+ resolve_device("npu")
+ torch.npu.set_device(0)
+ yield torch.device("npu:0")
+ torch.npu.synchronize()
+ torch.npu.empty_cache()
+
+
+def assert_bf16_close(actual, expected):
+ a, b = actual.detach().float().cpu(), expected.detach().float().cpu()
+ assert torch.isfinite(a).all() and torch.isfinite(b).all()
+ assert (a - b).norm() / b.norm().clamp_min(1e-8) <= 0.02
+ torch.testing.assert_close(a, b, atol=0.02, rtol=0.02)
+
+
+@pytest.mark.parametrize("kv_heads", [1, 2, 4])
+def test_torch_attention_on_npu_matches_cpu_prefix_oracle(npu_device, kv_heads):
+ torch.manual_seed(83)
+ keys = torch.randn(8, 2, 16, kv_heads, 64, dtype=torch.bfloat16)
+ values = torch.randn_like(keys)
+ q = torch.randn(64, 4, 64, dtype=torch.bfloat16)
+ lengths = torch.tensor([0, 1, 15, 16, 17, 31, 32, 3] * 8, dtype=torch.int32)
+ tables = torch.tensor([[5, 2], [1, 7]] * 32, dtype=torch.int32)
+ expected = torch_paged_attention(q, keys[:, 1], values[:, 1], tables, lengths)
+ actual = torch_paged_attention(
+ q.to(npu_device),
+ keys.to(npu_device)[:, 1],
+ values.to(npu_device)[:, 1],
+ tables.to(npu_device),
+ lengths.to(npu_device),
+ )
+ assert_bf16_close(actual, expected)
+ assert torch.count_nonzero(actual[0]).item() == 0
+
+
+def make_npu_model(device):
+ torch.manual_seed(41)
+ config = tiny_ouro_config(hidden_size=256, intermediate_size=512, head_dim=64)
+ return OuroForCausalLM(config).to(device=device, dtype=torch.bfloat16).eval()
+
+
+@torch.inference_mode()
+def recurrent_states(model, chunks, *, tokens=None, incremental=False):
+ c = model.config
+ cache = KVCacheManager(
+ c.num_hidden_layers,
+ c.num_key_value_heads,
+ c.head_dim,
+ 32,
+ 16,
+ 4,
+ device=next(model.parameters()).device,
+ dtype=torch.bfloat16,
+ backend="torch",
+ incremental_allocation=incremental,
+ )
+ tokens = tokens if tokens is not None else ([3, 7, 2, 11, 5, 17, 8] * 3)[:21]
+ assert cache.allocate("a", len(tokens), initial_tokens=16 if incremental else None)
+ states, kv, start = {}, {}, 0
+ try:
+ for size in chunks:
+ assert cache.ensure_capacity("a", start + size)
+ positions = list(range(start, start + size))
+ hidden = model.prelude(torch.tensor(tokens[start : start + size], device=cache.device))
+ for depth in range(4):
+ hidden, _ = model.recurrent(hidden, ["a"] * size, [depth] * size, positions, cache)
+ states[(start, depth)] = (hidden.clone(), model.coda(hidden).clone())
+ start += size
+ assert start == len(tokens)
+ for depth in range(4):
+ for layer in range(c.num_hidden_layers):
+ assert all(cache.token_written("a", p, depth) for p in range(len(tokens)))
+ kv[(layer, depth)] = tuple(
+ t.clone() for t in cache.read(layer, "a", depth, len(tokens))
+ )
+ assert len({tuple(cache.get_block_table("a", d)) for d in range(4)}) == 4
+ return states, kv
+ finally:
+ cache.free("a")
+ assert cache.num_used_blocks == 0
+
+
+def test_npu_chunked_prefill_and_incremental_kv(npu_device):
+ model = make_npu_model(npu_device)
+ expected, expected_kv = recurrent_states(model, [19, 1, 1])
+ incremental, incremental_kv = recurrent_states(model, [19, 1, 1], incremental=True)
+ for key in expected:
+ for a, b in zip(incremental[key], expected[key], strict=True):
+ torch.testing.assert_close(a, b, atol=0, rtol=0)
+ for key in expected_kv:
+ for a, b in zip(incremental_kv[key], expected_kv[key], strict=True):
+ torch.testing.assert_close(a, b, atol=0, rtol=0)
+ chunked, chunked_kv = recurrent_states(model, [5, 14, 1, 1], incremental=True)
+ for key in ((19, d) for d in range(4)):
+ for a, b in zip(chunked[key], expected[key], strict=True):
+ assert_bf16_close(a, b)
+ for key in expected_kv:
+ for a, b in zip(chunked_kv[key], expected_kv[key], strict=True):
+ assert_bf16_close(a, b)
+
+
+def test_npu_cancel_reuse_and_multiple_requests(npu_device):
+ engine = StageEngine(
+ make_npu_model(npu_device),
+ attention_backend="torch",
+ cache_config=CacheConfig(num_blocks=32, block_size=16, incremental_allocation=True),
+ scheduler_config=SchedulerConfig(max_num_seqs=2, max_num_batched_tokens=32),
+ )
+ try:
+ params = SamplingParams(max_tokens=3, temperature=0, ignore_eos=True)
+ engine.add_request("cancel", [3, 7, 2] * 6, params)
+ engine.add_request("keep", [5, 9, 11], params)
+ engine.step()
+ engine.abort_request("cancel")
+ engine.add_request("cancel", [13, 14, 15], params)
+ finished = {}
+ for _ in range(200):
+ for output in engine.step():
+ if output.finished:
+ finished[output.request_id] = output
+ if not engine.has_unfinished_requests():
+ break
+ assert set(finished) == {"keep", "cancel"}
+ assert all(
+ o.exit_depths == [4] * 3 and o.finish_reason == "length" for o in finished.values()
+ )
+ assert engine.cache_manager.num_used_blocks == 0
+ finally:
+ engine.close()
+
+
+@pytest.mark.parametrize("ignore_eos", [False, True])
+def test_npu_eos_and_length_handling(npu_device, ignore_eos):
+ model = make_npu_model(npu_device)
+ model.lm_head.weight.zero_() # Argmax 0 is the tiny model's actual EOS token.
+ engine = StageEngine(model, attention_backend="torch", cache_config=CacheConfig(num_blocks=16))
+ try:
+ engine.add_request(
+ "eos", [3, 7, 2], SamplingParams(max_tokens=3, temperature=0, ignore_eos=ignore_eos)
+ )
+ output = None
+ for _ in range(100):
+ for item in engine.step():
+ if item.finished:
+ output = item
+ if not engine.has_unfinished_requests():
+ break
+ assert output is not None
+ assert output.token_ids == [0] * (3 if ignore_eos else 1)
+ assert output.finish_reason == ("length" if ignore_eos else "stop")
+ assert engine.cache_manager.num_used_blocks == 0
+ finally:
+ engine.close()
diff --git a/tests/test_ascend_checkpoint.py b/tests/test_ascend_checkpoint.py
new file mode 100644
index 00000000..c26e8fe8
--- /dev/null
+++ b/tests/test_ascend_checkpoint.py
@@ -0,0 +1,100 @@
+"""Opt-in local-checkpoint KV and lifecycle regressions on Ascend."""
+
+import os
+from pathlib import Path
+
+import pytest
+import torch
+
+from tests.test_ascend import assert_bf16_close, recurrent_states
+from vllm_rlt import CacheConfig, SamplingParams, SchedulerConfig
+from vllm_rlt.device import resolve_device
+from vllm_rlt.engine.stage_engine import StageEngine
+from vllm_rlt.models import OuroForCausalLM, load_tokenizer
+
+pytestmark = pytest.mark.npu
+
+
+@pytest.fixture(scope="module")
+def checkpoint():
+ source = os.environ.get("VLLM_RLT_OURO_CHECKPOINT")
+ if not source:
+ pytest.skip("set VLLM_RLT_OURO_CHECKPOINT to a prepared local Ouro checkpoint")
+ if not Path(source).is_dir():
+ pytest.fail("VLLM_RLT_OURO_CHECKPOINT must be a prepared local directory")
+ resolve_device("npu")
+ torch.npu.set_device(0)
+ model = OuroForCausalLM.from_pretrained(source, device="npu", dtype=torch.bfloat16)
+ assert all(p.dtype == torch.bfloat16 for p in model.parameters())
+ yield model, load_tokenizer(source)
+ torch.npu.synchronize()
+
+
+@torch.inference_mode()
+def test_checkpoint_incremental_pages_preserve_hidden_logits_and_kv(checkpoint):
+ model, tokenizer = checkpoint
+ tokens = tokenizer.encode(
+ "The capital of France is Paris. The capital of Japan is Tokyo. "
+ "A language model predicts the next token in a sequence.",
+ add_special_tokens=False,
+ )[:21]
+ assert len(tokens) == 21
+ expected, expected_kv = recurrent_states(model, [19, 1, 1], tokens=tokens)
+ actual, actual_kv = recurrent_states(model, [19, 1, 1], tokens=tokens, incremental=True)
+ pairs = {}
+ for key in expected:
+ for name, a, b in zip(("hidden", "logits"), actual[key], expected[key], strict=True):
+ pairs[f"{key}.{name}"] = (a, b)
+ for key in expected_kv:
+ for name, a, b in zip(("key", "value"), actual_kv[key], expected_kv[key], strict=True):
+ pairs[f"{key}.{name}"] = (a, b)
+ for a, b in pairs.values():
+ assert_bf16_close(a, b)
+ torch.testing.assert_close(a, b, atol=0, rtol=0)
+
+
+@pytest.mark.parametrize("backend", ["torch", "ascend_eager"])
+@torch.inference_mode()
+def test_checkpoint_greedy_replay_and_request_cleanup(checkpoint, backend):
+ model, tokenizer = checkpoint
+ engine = StageEngine(
+ model,
+ attention_backend=backend,
+ cache_config=CacheConfig(num_blocks=64),
+ scheduler_config=SchedulerConfig(max_num_seqs=2, max_num_batched_tokens=32),
+ )
+ prompts = ["The capital of France is", "One plus one equals"]
+ loops = model.config.total_ut_steps
+ params = SamplingParams(
+ max_tokens=4, temperature=0, min_loops=loops, max_loops=loops, ignore_eos=True
+ )
+ expected = None
+ try:
+ # Repeat the same packed workload and reuse IDs after all pages are freed.
+ # Expected IDs come from this checkpoint, not a particular weight file.
+ for _ in range(2):
+ finished = {}
+ for index, prompt in enumerate(prompts):
+ engine.add_request(
+ str(index), tokenizer.encode(prompt, add_special_tokens=False), params
+ )
+ for _ in range(200):
+ for output in engine.step():
+ if output.finished:
+ assert len(output.token_ids) == params.max_tokens
+ assert all(
+ 0 <= token < model.config.vocab_size for token in output.token_ids
+ )
+ assert output.exit_depths == [loops] * params.max_tokens
+ assert output.finish_reason == "length"
+ finished[output.request_id] = list(output.token_ids)
+ if not engine.has_unfinished_requests():
+ break
+ assert set(finished) == {str(index) for index in range(len(prompts))}
+ assert engine.cache_manager.num_used_blocks == 0
+ if expected is None:
+ expected = finished
+ else:
+ assert finished == expected
+ finally:
+ engine.close()
diff --git a/tests/test_ascend_compatibility.py b/tests/test_ascend_compatibility.py
new file mode 100644
index 00000000..9f264189
--- /dev/null
+++ b/tests/test_ascend_compatibility.py
@@ -0,0 +1,89 @@
+"""Ascend opt-ins preserve the existing CPU/CUDA/ROCm API and defaults."""
+
+import builtins
+from dataclasses import fields
+
+import pytest
+import torch
+
+from tests.helpers import tiny_ouro_model
+from vllm_rlt import CacheConfig, SchedulerConfig
+from vllm_rlt.attention.backend import BackendCapabilities
+from vllm_rlt.device import stream_module
+from vllm_rlt.engine.stage_engine import StageEngine
+from vllm_rlt.layers.rotary_embedding import RotaryEmbedding
+from vllm_rlt.worker.executor import ModelRuntime
+from vllm_rlt.worker.model_runner import ModelRunner
+from vllm_rlt.worker.sampler import Sampler
+
+
+def test_backend_capability_positional_arguments_keep_their_meaning():
+ names = tuple(f.name for f in fields(BackendCapabilities))
+ assert names == (
+ "packed_prefill",
+ "cuda_graphs",
+ "async_scheduling",
+ "packed_matches_decode",
+ "fp8_kv",
+ "fp8_kv_static",
+ "device_context_lengths",
+ "sliding_window",
+ "packed_sliding_window",
+ )
+ caps = BackendCapabilities(False, False, False, False, False, True, False, True)
+ assert caps.sliding_window and caps.fp8_kv_static
+
+
+@pytest.mark.parametrize("hip", [None, "6.4"])
+def test_cuda_and_rocm_stream_selection_never_loads_npu(monkeypatch, hip):
+ monkeypatch.setattr(torch.version, "hip", hip)
+ original = builtins.__import__
+
+ def guarded(name, *args, **kwargs):
+ if name == "torch_npu" or name.startswith("vllm_rlt.worker.ascend_model_runner"):
+ pytest.fail("CUDA/ROCm selected an Ascend implementation")
+ return original(name, *args, **kwargs)
+
+ monkeypatch.setattr(builtins, "__import__", guarded)
+ assert stream_module(torch.device("cuda:0")) is torch.cuda
+
+
+@pytest.mark.parametrize("hip", [None, "6.4"])
+def test_cuda_rope_keeps_device_math_without_host_migration(monkeypatch, hip):
+ monkeypatch.setattr(torch.version, "hip", hip)
+ rotary = RotaryEmbedding(128)
+ arange = torch.arange
+ seen = []
+
+ def recording(*args, **kwargs):
+ seen.append(kwargs["device"])
+ # CPU execution of the same arithmetic; no CUDA/ROCm hardware claim.
+ return arange(*args, **dict(kwargs, device="cpu"))
+
+ monkeypatch.setattr(torch, "arange", recording)
+ actual = rotary.frequencies(torch.device("cuda:0"))
+ assert seen == [torch.device("cuda:0")]
+ expected = 1.0 / (10000.0 ** (arange(0, 128, 2, dtype=torch.float32) / 128))
+ torch.testing.assert_close(actual, expected, atol=0, rtol=0)
+
+
+def test_default_engine_keeps_upstream_runtime_and_sampler(monkeypatch):
+ original = builtins.__import__
+
+ def guarded(name, *args, **kwargs):
+ if name == "torch_npu" or name.startswith("vllm_rlt.worker.ascend_model_runner"):
+ pytest.fail("default engine loaded Ascend worker code")
+ return original(name, *args, **kwargs)
+
+ monkeypatch.setattr(builtins, "__import__", guarded)
+ engine = StageEngine(
+ tiny_ouro_model(),
+ cache_config=CacheConfig(num_blocks=128),
+ scheduler_config=SchedulerConfig(max_num_seqs=2, max_num_batched_tokens=128),
+ )
+ try:
+ assert type(engine.model_runner) is ModelRunner
+ assert type(engine.model_runner.runtime) is ModelRuntime
+ assert type(engine.model_runner.sampler) is Sampler
+ finally:
+ engine.close()
diff --git a/tests/test_ascend_decode_scheduling.py b/tests/test_ascend_decode_scheduling.py
new file mode 100644
index 00000000..c66aea06
--- /dev/null
+++ b/tests/test_ascend_decode_scheduling.py
@@ -0,0 +1,239 @@
+"""Ascend decode scheduling, refill and asynchronous result delivery."""
+
+import pytest
+import torch
+
+from tests.helpers import make_cpu_cache, tiny_ouro_config
+from tests.test_ascend import make_npu_model
+from tests.test_npu_async import drain
+from vllm_rlt import CacheConfig, ExecutionConfig, ExitConfig, SamplingParams, SchedulerConfig
+from vllm_rlt.core.scheduler import ScheduledRequest, Scheduler, SchedulerOutput
+from vllm_rlt.device import resolve_device
+from vllm_rlt.engine.stage_engine import StageEngine
+from vllm_rlt.request import Request, RequestPhase
+
+
+def test_decode_prelude_leaves_recurrent_work_to_scheduler():
+ scheduler = Scheduler(SchedulerConfig(), None, total_ut_steps=4, exit_mode="ouro_delayed")
+ request = Request("a", [1], SamplingParams(min_loops=4, max_loops=4))
+ request.phase = RequestPhase.PRELUDE
+ scheduler.requests["a"] = request
+ item = ScheduledRequest(request, phase=RequestPhase.PRELUDE)
+ assert scheduler._planned_loop_count([item]) == 0
+ batch = SchedulerOutput(RequestPhase.PRELUDE, [item], 0)
+ update = scheduler.update_after_submission(batch)
+ assert request.loops_done == 0
+ assert request.phase == RequestPhase.RECURRENT
+ assert update.exited == []
+ request.phase = RequestPhase.PRELUDE
+ scheduler.requests.pop("a")
+ scheduler.update_after_submission(batch)
+ assert request.phase == RequestPhase.PRELUDE # A stale submission cannot advance a request.
+
+
+@pytest.mark.parametrize("late_arrival", [False, True])
+def test_decode_preserves_refill_opportunity(late_arrival):
+ scheduler = Scheduler(
+ SchedulerConfig(max_num_seqs=2, max_num_batched_tokens=3, prefill_chunk_size=3),
+ make_cpu_cache(tiny_ouro_config(), blocks=128),
+ total_ut_steps=4,
+ exit_mode="ouro_delayed",
+ )
+ params = SamplingParams(max_tokens=16, min_loops=4, max_loops=4, ignore_eos=True)
+ first = Request("a", [1, 2, 3], params)
+ second = Request("b", [4, 5, 6], params)
+ scheduler.add_request(first)
+ if not late_arrival:
+ scheduler.add_request(second)
+ batch = scheduler.schedule()
+ assert batch.phase == RequestPhase.PREFILL
+ assert [item.request for item in batch.requests] == [first]
+ scheduler.update_after_submission(batch)
+ batch = scheduler.schedule()
+ assert batch.phase == RequestPhase.CODA
+ scheduler.update_after_submission(batch)
+ if late_arrival:
+ scheduler.add_request(second)
+ assert second.phase == (RequestPhase.WAITING if late_arrival else RequestPhase.PREFILL)
+
+ # Every prompt consumes a complete token budget. Refill must still get a
+ # turn before the first generation ends.
+ for _ in range(8):
+ batch = scheduler.schedule()
+ assert batch is not None
+ scheduler.update_after_submission(batch)
+ if any(item.request is second for item in batch.requests):
+ assert batch.phase == RequestPhase.PREFILL
+ assert first.num_scheduled_outputs < params.max_tokens
+ break
+ else:
+ pytest.fail("decode boundaries starved a pending prompt")
+ targets = {request.request_id: request.num_scheduled_outputs + 1 for request in (first, second)}
+ for _ in range(12):
+ batch = scheduler.schedule()
+ assert batch is not None
+ scheduler.update_after_submission(batch)
+ if all(
+ request.num_scheduled_outputs >= targets[request.request_id]
+ for request in (first, second)
+ ):
+ break
+ else:
+ pytest.fail("refilled decode starved the existing recurrent request")
+
+
+def engine_for(model, multi_stream=False, token_budget=128):
+ return StageEngine(
+ model,
+ attention_backend="ascend_eager",
+ cache_config=CacheConfig(num_blocks=128, block_size=16),
+ scheduler_config=SchedulerConfig(
+ max_num_seqs=3, max_num_batched_tokens=token_budget, prefill_chunk_size=128
+ ),
+ execution_config=ExecutionConfig(
+ async_scheduling=True,
+ multi_stream=multi_stream,
+ use_uva=False,
+ ),
+ exit_config=ExitConfig("ouro_delayed"),
+ )
+
+
+@pytest.mark.parametrize("waiting", [False, True])
+def test_decode_prelude_keeps_recurrent_due(waiting):
+ scheduler = Scheduler(
+ SchedulerConfig(max_num_seqs=2, max_prefill_batches_before_decode=4),
+ make_cpu_cache(tiny_ouro_config(), blocks=128),
+ total_ut_steps=4,
+ exit_mode="ouro_delayed",
+ )
+ params = SamplingParams(max_tokens=8, min_loops=4, max_loops=4, ignore_eos=True)
+ scheduler.add_request(Request("a", [1, 2, 3], params))
+ for expected in (RequestPhase.PREFILL, RequestPhase.CODA):
+ batch = scheduler.schedule()
+ assert batch.phase == expected
+ scheduler.update_after_submission(batch)
+ if waiting:
+ scheduler.add_request(Request("b", [4, 5, 6], params))
+ for _ in range(scheduler.config.max_prefill_batches_before_decode):
+ scheduler.policy.record_batch(RequestPhase.PREFILL)
+ assert scheduler.policy.decode_due
+ batch = scheduler.schedule()
+ assert batch.phase == RequestPhase.PRELUDE
+ assert batch.loop_count == 0
+ assert scheduler.policy.decode_due
+
+
+@pytest.mark.npu
+@torch.inference_mode()
+@pytest.mark.parametrize("multi_stream", [False, True])
+def test_ascend_decode_refills_during_generation(multi_stream):
+ resolve_device("npu")
+ torch.npu.set_device(0)
+ model = make_npu_model(torch.device("npu:0"))
+ params = SamplingParams(max_tokens=8, min_loops=4, max_loops=4, ignore_eos=True)
+ engine = engine_for(model, multi_stream, token_budget=3)
+ try:
+ for name, prompt in [("a", [3, 7, 11]), ("b", [13, 17, 19])]:
+ engine.add_request(name, prompt, params)
+ actual, first_finished, second_started = {}, False, False
+ first_tokens_at_second_start = None
+ first_tokens, first_advanced = 0, False
+ for _ in range(128):
+ if not engine.has_unfinished_requests():
+ break
+ for out in engine.step():
+ if out.request_id == "b" and not second_started:
+ assert not first_finished
+ second_started = True
+ first_tokens_at_second_start = first_tokens
+ if out.request_id == "a":
+ first_tokens = len(out.token_ids)
+ if second_started and "b" not in actual:
+ first_advanced |= first_tokens > first_tokens_at_second_start
+ if out.finished:
+ actual[out.request_id] = (
+ list(out.token_ids),
+ list(out.exit_depths),
+ out.finish_reason,
+ )
+ if out.request_id == "a":
+ first_finished = True
+ else:
+ pytest.fail("refill did not drain in 128 steps")
+ assert second_started and first_advanced
+ assert set(actual) == {"a", "b"}
+ assert engine.cache_manager.num_used_blocks == 0
+ finally:
+ engine.close()
+
+
+@pytest.mark.npu
+@torch.inference_mode()
+@pytest.mark.parametrize("multi_stream", [False, True])
+def test_ascend_decode_preserves_cancel_reuse_and_eos(multi_stream):
+ resolve_device("npu")
+ torch.npu.set_device(0)
+ model = make_npu_model(torch.device("npu:0"))
+ params = SamplingParams(max_tokens=4, min_loops=4, max_loops=4, ignore_eos=True)
+ engine = engine_for(model, multi_stream)
+ try:
+ for i, prompt in enumerate([[3, 7, 11], [13, 17, 19]]):
+ engine.add_request(str(i), prompt, params)
+ expected = drain(engine)
+ assert engine.cache_manager.num_used_blocks == 0
+ engine.add_request("0", [2, 5, 9], params)
+ engine.add_request("1", [13, 17, 19], params)
+ engine.step()
+ engine.step()
+ engine.step()
+ engine.abort_request("0")
+ engine.add_request("0", [3, 7, 11], params)
+ assert drain(engine) == expected
+ eos = model.config.eos_token_id
+ engine.model_runner.sampler.sample_batch = lambda logits, options, generators: (
+ logits.new_full((len(options),), eos, dtype=torch.long)
+ )
+ engine.add_request(
+ "eos", [3, 7, 11], SamplingParams(max_tokens=4, min_loops=4, max_loops=4)
+ )
+ assert drain(engine)["eos"] == ([eos], [4], "stop")
+ finally:
+ engine.close()
+
+
+@pytest.mark.npu
+@torch.inference_mode()
+def test_ascend_prelude_uses_device_sample_before_coda_delivery():
+ resolve_device("npu")
+ torch.npu.set_device(0)
+ model = make_npu_model(torch.device("npu:0"))
+ engine = engine_for(model)
+
+ class HeldEvent:
+ def query(self):
+ return False
+
+ def synchronize(self):
+ pytest.fail("decode plan waited for host token delivery")
+
+ try:
+ engine.add_request(
+ "a", [3, 7, 11], SamplingParams(max_tokens=3, min_loops=4, max_loops=4, ignore_eos=True)
+ )
+ engine.step()
+ engine.step()
+ ticket = engine._pending_coda[0]
+ actual_event = ticket.event
+ ticket.event = HeldEvent()
+ try:
+ engine.step()
+ assert engine.last_schedule.phase == RequestPhase.PRELUDE
+ assert engine.last_schedule.loop_count == 0
+ request = engine.scheduler.requests["a"]
+ assert request.loops_done == 0 and request.generated_token_ids == []
+ finally:
+ ticket.event = actual_event
+ assert drain(engine)["a"][1] == [4, 4, 4]
+ finally:
+ engine.close()
diff --git a/tests/test_ascend_eager.py b/tests/test_ascend_eager.py
new file mode 100644
index 00000000..f985a6d0
--- /dev/null
+++ b/tests/test_ascend_eager.py
@@ -0,0 +1,307 @@
+"""Causal/GQA semantics for Ascend SDPA over paged KV."""
+
+import pytest
+import torch
+
+from vllm_rlt.kernels.ascend_attention import AscendAttention, _flat_kv_rows, _prefill_geometry
+
+
+@pytest.mark.parametrize("kv_heads", [1, 2, 4])
+def test_eager_groups_match_independent_causal_queries(kv_heads):
+ torch.manual_seed(730)
+ keys = torch.randn(8, 16, kv_heads, 64, dtype=torch.bfloat16)
+ values = torch.randn_like(keys)
+ q = torch.randn(7, 4, 64, dtype=torch.bfloat16)
+ lengths = torch.tensor([0, 1, 15, 16, 17, 3, 31], dtype=torch.int32)
+ tables = torch.tensor([[4, 1], [4, 1], [4, 1], [4, 1], [4, 1], [7, 2], [7, 2]])
+ actual = AscendAttention()(q, keys, values, tables, lengths)
+ expected = torch.zeros_like(q)
+ for row, length in enumerate(lengths.tolist()):
+ if not length:
+ continue
+ pos = torch.arange(length)
+ k = keys[tables[row, pos // 16], pos % 16].repeat_interleave(4 // kv_heads, dim=1)
+ v = values[tables[row, pos // 16], pos % 16].repeat_interleave(4 // kv_heads, dim=1)
+ scores = torch.einsum("hd,thd->ht", q[row], k) * (64**-0.5)
+ probs = scores.softmax(-1, dtype=torch.float32).to(q.dtype)
+ expected[row] = torch.einsum("ht,thd->hd", probs, v)
+ torch.testing.assert_close(actual, expected, atol=0.02, rtol=0.02)
+ assert torch.isfinite(actual).all()
+ assert torch.count_nonzero(actual[0]) == 0
+
+
+@pytest.mark.npu
+@torch.inference_mode()
+def test_pooled_kv_reads_match_layer_views_across_pages_and_depths():
+ from vllm_rlt.core.kv_cache_manager import KVCacheManager
+ from vllm_rlt.device import resolve_device
+
+ device = resolve_device("npu:0")
+ torch.npu.set_device(device)
+ torch.manual_seed(99)
+ cache = KVCacheManager(
+ 3, 2, 64, 16, 16, 2, device=device, dtype=torch.bfloat16, backend="ascend_eager"
+ )
+ assert cache.allocate("a", 17)
+ assert cache.allocate("b", 3)
+ batch = cache._prepare_batch(["a"] * 17 + ["b"] * 3, [0] * 17 + [1] * 3, [*range(17), 0, 1, 2])
+ for layer in range(3):
+ k = torch.randn(20, 2, 64, device=device, dtype=torch.bfloat16)
+ v = torch.randn_like(k)
+ cache._write_prepared(layer, batch, k, v)
+
+ q = torch.randn(20, 4, 64, device=device, dtype=torch.bfloat16)
+ for layer in range(3):
+ expected = cache.attention(
+ q,
+ cache.key_cache[:, layer],
+ cache.value_cache[:, layer],
+ batch.block_tables,
+ batch.context_lengths,
+ )
+ actual = cache._attend_prepared(layer, batch, q)
+ torch.testing.assert_close(actual, expected, atol=0, rtol=0)
+
+
+@pytest.mark.npu
+@pytest.mark.parametrize("chunk_size", [4096, 128])
+@torch.inference_mode()
+def test_engine_matches_the_same_manual_prefill_chunks(chunk_size):
+ from tests.helpers import tiny_ouro_config
+ from vllm_rlt import CacheConfig, SamplingParams, SchedulerConfig
+ from vllm_rlt.core.kv_cache_manager import KVCacheManager
+ from vllm_rlt.device import resolve_device
+ from vllm_rlt.engine.stage_engine import StageEngine
+ from vllm_rlt.models import OuroForCausalLM
+
+ device = resolve_device("npu:0")
+ torch.npu.set_device(device)
+ torch.manual_seed(41)
+ model = OuroForCausalLM(
+ tiny_ouro_config(
+ hidden_size=256, intermediate_size=512, head_dim=64, max_position_embeddings=256
+ )
+ ).to(device=device, dtype=torch.bfloat16)
+ c = model.config
+ tokens = ([3, 7, 2, 11, 5, 17, 8] * 22)[:149]
+ cache = KVCacheManager(
+ c.num_hidden_layers,
+ c.num_key_value_heads,
+ c.head_dim,
+ 128,
+ 16,
+ 4,
+ device=device,
+ dtype=torch.bfloat16,
+ backend="ascend_eager",
+ )
+ assert cache.allocate("manual", len(tokens) + 3)
+ expected, manual_states = [], []
+ handle = model.model.norm.register_forward_hook(
+ lambda _m, _i, output: manual_states.append(output.detach().cpu().clone())
+ )
+ try:
+ for start in range(0, len(tokens), chunk_size):
+ end = min(start + chunk_size, len(tokens))
+ hidden = model.prelude(torch.tensor(tokens[start:end], device=device))
+ for depth in range(4):
+ hidden, _ = model.recurrent(
+ hidden,
+ ["manual"] * (end - start),
+ [depth] * (end - start),
+ list(range(start, end)),
+ cache,
+ compute_gate=False,
+ )
+ for step in range(3):
+ expected.append(int(model.coda(hidden[-1:])[0].argmax()))
+ if step < 2:
+ pos = len(tokens) + step
+ hidden = model.prelude(torch.tensor([expected[-1]], device=device))
+ for depth in range(4):
+ hidden, _ = model.recurrent(
+ hidden, ["manual"], [depth], [pos], cache, compute_gate=False
+ )
+ finally:
+ handle.remove()
+ cache.free("manual")
+ actual_states = []
+ handle = model.model.norm.register_forward_hook(
+ lambda _m, _i, output: actual_states.append(output.detach().cpu().clone())
+ )
+ engine = StageEngine(
+ model,
+ attention_backend="ascend_eager",
+ cache_config=CacheConfig(num_blocks=128),
+ scheduler_config=SchedulerConfig(
+ max_num_seqs=1, max_num_batched_tokens=4096, prefill_chunk_size=chunk_size
+ ),
+ )
+ try:
+ assert engine.execution_config.packed_prefill
+ engine.add_request(
+ "engine",
+ tokens,
+ SamplingParams(max_tokens=3, temperature=0, min_loops=4, max_loops=4, ignore_eos=True),
+ )
+ result = None
+ while engine.has_unfinished_requests():
+ for out in engine.step():
+ if out.finished:
+ result = out
+ assert result is not None
+ assert result.token_ids == expected
+ assert result.exit_depths == [4] * 3
+ assert len(actual_states) == len(manual_states)
+ for actual, reference in zip(actual_states, manual_states, strict=True):
+ torch.testing.assert_close(actual, reference, atol=0, rtol=0)
+ assert engine.cache_manager.num_used_blocks == 0
+ finally:
+ handle.remove()
+ engine.close()
+
+
+@pytest.mark.parametrize("device_name", ["cpu", pytest.param("npu", marks=pytest.mark.npu)])
+@torch.inference_mode()
+def test_cached_continuation_passes_bool_mask_to_sdpa(device_name, monkeypatch):
+ from vllm_rlt.device import resolve_device
+
+ device = resolve_device(device_name)
+ if device_name == "npu":
+ torch.npu.set_device(0)
+ keys = torch.zeros(1, 16, 2, 64, dtype=torch.bfloat16, device=device)
+ values = torch.zeros_like(keys)
+ for pos, value in enumerate((1, 3, 8)):
+ values[0, pos].fill_(value)
+ query = torch.ones(2, 2, 64, dtype=torch.bfloat16, device=device)
+ table = torch.tensor([[0], [0]], dtype=torch.int32, device=device)
+ lengths = torch.tensor([2, 3], dtype=torch.int32, device=device)
+ sdpa = torch.nn.functional.scaled_dot_product_attention
+ calls = []
+
+ def check_sdpa(q, k, v, *, attn_mask, is_causal):
+ calls.append(attn_mask)
+ assert not is_causal
+ assert q.shape == (1, 2, 2, 64) and k.shape == v.shape == (1, 2, 3, 64)
+ assert attn_mask.dtype == torch.bool
+ assert attn_mask.tolist() == [[True, True, False], [True, True, True]]
+ return sdpa(q, k, v, attn_mask=attn_mask, is_causal=is_causal)
+
+ monkeypatch.setattr(torch.nn.functional, "scaled_dot_product_attention", check_sdpa)
+ actual = AscendAttention()(query, keys, values, table, lengths)
+ # Equal scores average the visible values. The first query sees the cached
+ # prefix and itself, while only the second query can see the last value.
+ expected = torch.tensor([2, 4], device=device, dtype=query.dtype)[:, None, None]
+ torch.testing.assert_close(actual, expected.expand_as(actual), atol=0.016, rtol=0)
+ assert len(calls) == 1
+
+
+class _PreparedBatch:
+ cu_seqlens_q = None
+
+ """Only the host metadata the two batched attention paths consume."""
+
+ def __init__(self, rows, tables, write_blocks, write_offsets):
+ self.rows = rows
+ self.block_tables = tables
+ self.write_blocks = write_blocks
+ self.write_offsets = write_offsets
+
+
+def _batched_fixture(kind):
+ layers, blocks, block_size, kv_heads, head_dim = 3, 24, 4, 2, 8
+ head_groups = 2
+ dtype = torch.bfloat16 # the Ascend eager contract is BF16 end to end
+ torch.manual_seed(913)
+ pool_k = torch.randn(blocks, layers, block_size, kv_heads, head_dim, dtype=dtype)
+ pool_v = torch.randn_like(pool_k)
+ keys, values = pool_k[:, 1], pool_v[:, 1]
+
+ first, second, third, fourth = (object() for _ in range(4))
+ if kind == "prefill_grid":
+ rows = [(a, 0, pos) for a in (first, second) for pos in range(4)]
+ elif kind == "prefill_single":
+ rows = [(first, 0, pos) for pos in range(4)]
+ elif kind == "ragged_decode":
+ rows = [(a, 0, 5) for a in (first, second, third, fourth)]
+ elif kind == "ragged_padded_tables":
+ # Mixed natural lengths: short rows have -1 in the high page slots,
+ # exactly as plan_attention_metadata pads a shorter block table.
+ rows = [(first, 0, 9), (second, 0, 1), (third, 0, 5), (fourth, 0, 2)]
+ elif kind == "single_decode":
+ rows = [(first, 0, 5)]
+ else: # irregular: a prefill group with a missing position
+ rows = [(first, 0, 0), (first, 0, 2), (first, 0, 3)]
+
+ page_tables = {
+ id(first): [2, 5, 7, 9],
+ id(second): [3, 6, 8, 10],
+ id(third): [4, 11, 12, 13],
+ id(fourth): [14, 15, 16, 17],
+ }
+ if kind == "ragged_padded_tables":
+ page_tables = {
+ id(first): [2, 5, 7, 9],
+ id(second): [3, -1, -1, -1],
+ id(third): [4, 11, -1, -1],
+ id(fourth): [14, -1, -1, -1],
+ }
+ tables, write_blocks, write_offsets = [], [], []
+ for allocation, _depth, position in rows:
+ table = page_tables[id(allocation)]
+ tables.append(table)
+ write_blocks.append(table[position // block_size])
+ write_offsets.append(position % block_size)
+ batch = _PreparedBatch(
+ rows,
+ torch.tensor(tables, dtype=torch.int32),
+ torch.tensor(write_blocks, dtype=torch.int64),
+ torch.tensor(write_offsets, dtype=torch.int64),
+ )
+ q = torch.randn(len(rows), kv_heads * head_groups, head_dim, dtype=dtype)
+ return q, keys, values, batch
+
+
+@pytest.mark.parametrize(
+ "kind,expected_prefill,expected_ragged",
+ [
+ ("prefill_grid", 1, 0),
+ ("prefill_single", 1, 0),
+ ("ragged_decode", 0, 1),
+ ("ragged_padded_tables", 0, 1),
+ ("single_decode", 0, 0),
+ ("irregular", 0, 0),
+ ],
+)
+@torch.inference_mode()
+def test_batched_attention_paths_match_per_group_sdpa(kind, expected_prefill, expected_ragged):
+ """Keep query/result ordering and masks consistent when batching SDPA calls."""
+ q, keys, values, batch = _batched_fixture(kind)
+ kernel = AscendAttention()
+ groups = kernel._plan(
+ [(id(allocation), depth) for allocation, depth, _ in batch.rows],
+ [position + 1 for _, _, position in batch.rows],
+ q.device,
+ )
+ expected = kernel._forward(
+ q,
+ keys,
+ values,
+ batch.block_tables,
+ groups,
+ flat_rows=_flat_kv_rows(keys, values),
+ )
+ actual = kernel.attend_metadata(q, keys, values, batch)
+ torch.testing.assert_close(actual, expected, atol=0, rtol=0)
+ assert kernel.batched_prefill_calls == expected_prefill
+ assert kernel.ragged_calls == expected_ragged
+
+
+def test_prefill_geometry_accepts_one_request_but_rejects_irregular_grids():
+ first, second = object(), object()
+ assert _prefill_geometry([(first, 0, pos) for pos in range(4)]) == (1, 4)
+ assert _prefill_geometry(
+ [(first, 0, pos) for pos in range(4)] + [(second, 0, pos) for pos in range(4)]
+ ) == (2, 4)
+ assert _prefill_geometry([(first, 0, 0), (first, 0, 2)]) is None
+ assert _prefill_geometry([(first, 0, 1)]) is None
diff --git a/tests/test_ascend_kv.py b/tests/test_ascend_kv.py
new file mode 100644
index 00000000..7d53d071
--- /dev/null
+++ b/tests/test_ascend_kv.py
@@ -0,0 +1,54 @@
+"""Paged KV writes and descriptor lifetime on an assigned Ascend device."""
+
+import pytest
+import torch
+
+from vllm_rlt.core.kv_cache_manager import KVCacheManager
+from vllm_rlt.device import resolve_device
+
+pytestmark = pytest.mark.npu
+
+
+@pytest.mark.parametrize("rows", [1, 3])
+@pytest.mark.parametrize("backend", ["torch", "ascend_eager"])
+@torch.inference_mode()
+def test_paged_kv_writes_preserve_inputs_layers_and_reused_ids(rows, backend):
+ device = resolve_device("npu:0")
+ torch.npu.set_device(device)
+ cache = KVCacheManager(
+ 2, 2, 64, 32, 16, 4, device=device, dtype=torch.bfloat16, backend=backend
+ )
+ cache.key_cache.zero_()
+ cache.value_cache.zero_()
+ assert cache.allocate("a", 17)
+ assert cache.allocate("b", 3)
+ ids, depths, positions = ["a", "b", "a"][:rows], [0, 2, 3][:rows], [15, 0, 16][:rows]
+ batch = cache._prepare_batch(ids, depths, positions)
+ torch.manual_seed(99)
+ k = torch.randn(rows, 2, 64, device=device, dtype=torch.bfloat16)
+ v = torch.randn_like(k)
+ saved_k, saved_v = k.clone(), v.clone()
+ expected_k, expected_v = cache.key_cache.clone(), cache.value_cache.clone()
+ for layer in range(2):
+ expected_k[batch.write_blocks, layer, batch.write_offsets] = k
+ expected_v[batch.write_blocks, layer, batch.write_offsets] = v
+ cache._write_prepared(layer, batch, k, v)
+ torch.testing.assert_close(cache.key_cache, expected_k, atol=0, rtol=0)
+ torch.testing.assert_close(cache.value_cache, expected_v, atol=0, rtol=0)
+ torch.testing.assert_close(k, saved_k, atol=0, rtol=0)
+ torch.testing.assert_close(v, saved_v, atol=0, rtol=0)
+ cache.free("a")
+ assert cache.allocate("a", 17)
+ with pytest.raises((ValueError, RuntimeError)):
+ cache._write_prepared(0, batch, k, v)
+ new_batch = cache._prepare_batch(["a"], [1], [0])
+ expected_k, expected_v = cache.key_cache.clone(), cache.value_cache.clone()
+ expected_k[new_batch.write_blocks, 1, new_batch.write_offsets] = k[:1]
+ expected_v[new_batch.write_blocks, 1, new_batch.write_offsets] = v[:1]
+ cache._write_prepared(1, new_batch, k[:1], v[:1])
+ torch.testing.assert_close(cache.key_cache, expected_k, atol=0, rtol=0)
+ torch.testing.assert_close(cache.value_cache, expected_v, atol=0, rtol=0)
+ cache.free("a")
+ cache.free("b")
+ assert cache.num_used_blocks == 0
+ torch.npu.synchronize()
diff --git a/tests/test_ascend_metadata.py b/tests/test_ascend_metadata.py
new file mode 100644
index 00000000..5d8ec1c3
--- /dev/null
+++ b/tests/test_ascend_metadata.py
@@ -0,0 +1,214 @@
+"""Metadata reuse must preserve sparse coverage, ownership and eager values."""
+
+import gc
+import weakref
+
+import pytest
+import torch
+
+from vllm_rlt.core.kv_cache_manager import KVCacheManager, _sequence_coverage, _WrittenPositions
+from vllm_rlt.kernels.ascend_attention import AscendAttention, _flat_kv_rows
+
+
+@pytest.mark.parametrize(
+ "ranges",
+ [
+ [(0, 1024), (1024, 4096)],
+ [(5, 9), (0, 3), (3, 5)],
+ [(9, 12), (4, 7), (0, 6), (6, 10)],
+ [(2, 5), (2, 7), (0, 3), (1, 4), (9, 9)],
+ ],
+)
+def test_written_ranges_preserve_sparse_and_overlapping_coverage(ranges):
+ reference, actual = _WrittenPositions(), _WrittenPositions()
+ for start, end in ranges:
+ for position in range(start, end):
+ reference.add(position)
+ actual.add_range(start, end)
+ assert (actual.prefix, actual.pending) == (reference.prefix, reference.pending)
+
+
+def test_sequence_coverage_keeps_holes_and_plane_identity():
+ first, second = object(), object()
+ rows = [(first, 0, p) for p in (2, 0, 1, 5)] + [(first, 1, 0), (second, 0, 0)]
+ plans = _sequence_coverage(rows, lambda depth: depth)
+ assert [(p.allocation, p.plane, p.ranges) for p in plans] == [
+ (first, 0, ((0, 3), (5, 6))),
+ (first, 1, ((0, 1),)),
+ (second, 0, ((0, 1),)),
+ ]
+
+
+def _cache():
+ from vllm_rlt.device import resolve_device
+
+ device = resolve_device("npu:0")
+ torch.npu.set_device(device)
+ cache = KVCacheManager(
+ 2, 2, 64, 64, 16, 4, device=device, dtype=torch.bfloat16, backend="ascend_eager"
+ )
+ cache.key_cache.fill_(float("nan"))
+ cache.value_cache.fill_(float("nan"))
+ lengths = [3, 7, 17]
+ for index, length in enumerate(lengths):
+ rid = str(index)
+ assert cache.allocate(rid, length)
+ for depth in (0, 1):
+ batch = cache._prepare_batch([rid] * length, [depth] * length, list(range(length)))
+ for layer in range(2):
+ k = torch.randn(length, 2, 64, device=device, dtype=torch.bfloat16)
+ v = torch.randn_like(k)
+ cache._write_prepared(layer, batch, k, v)
+ return cache, lengths
+
+
+def _reference(kernel, q, keys, values, batch):
+ groups = kernel._plan(
+ [(id(a), depth) for a, depth, _ in batch.rows],
+ [p + 1 for _, _, p in batch.rows],
+ q.device,
+ )
+ return kernel._forward(
+ q, keys, values, batch.block_tables, groups, flat_rows=_flat_kv_rows(keys, values)
+ )
+
+
+@pytest.mark.npu
+@pytest.mark.parametrize("padded", [False, True])
+@torch.inference_mode()
+def test_unwritten_nan_padding_is_never_consumed(padded):
+ cache, lengths = _cache()
+ batch = cache._prepare_batch(
+ ["0", "1", "2"],
+ [0] * 3,
+ [n - 1 for n in lengths],
+ for_write=False,
+ padded_rows=4 if padded else None,
+ )
+ # Noncontiguous Q must retain the original contiguous attention layout.
+ q = torch.randn(4 if padded else 3, 4, 128, device=cache.device, dtype=cache.dtype)[:, :, ::2]
+ for layer in range(2):
+ keys, values = cache.key_cache[:, layer], cache.value_cache[:, layer]
+ expected = _reference(AscendAttention(), q, keys, values, batch)
+ actual = cache._attend_prepared(layer, batch, q)
+ assert torch.isfinite(actual).all()
+ torch.testing.assert_close(actual, expected, rtol=0, atol=0)
+ if padded:
+ assert torch.count_nonzero(actual[3]) == 0
+ for rid in ("0", "1", "2"):
+ cache.free(rid)
+
+
+@pytest.mark.npu
+@torch.inference_mode()
+def test_layout_reuse_does_not_reuse_addresses_or_outputs():
+ cache, lengths = _cache()
+ kernel = cache.attention.kernel
+ q = torch.randn(3, 4, 64, device=cache.device, dtype=cache.dtype)
+ saved = None
+ for depth in (0, 1, 0):
+ batch = cache._prepare_batch(
+ ["0", "1", "2"], [depth] * 3, [n - 1 for n in lengths], for_write=False
+ )
+ expected = _reference(
+ AscendAttention(), q, cache.key_cache[:, 0], cache.value_cache[:, 0], batch
+ )
+ actual = cache._attend_prepared(0, batch, q)
+ torch.testing.assert_close(actual, expected, rtol=0, atol=0)
+ if saved is None:
+ saved, snapshot = actual, actual.clone()
+ assert kernel.layout_reuses >= 2
+ torch.testing.assert_close(saved, snapshot, rtol=0, atol=0)
+ reference = weakref.ref(batch)
+ del batch
+ gc.collect()
+ assert reference() is None
+ for rid in ("0", "1", "2"):
+ cache.free(rid)
+
+
+@pytest.mark.npu
+@torch.inference_mode()
+def test_cached_validation_keeps_layer_holes_truncation_and_reused_id_checks():
+ cache, _ = _cache()
+ batch = cache._prepare_batch(["0", "0"], [2, 2], [0, 2])
+ k = torch.randn(2, 2, 64, device=cache.device, dtype=cache.dtype)
+ q = torch.randn(2, 4, 64, device=cache.device, dtype=cache.dtype)
+ cache._write_prepared(0, batch, k, k)
+ with pytest.raises(RuntimeError, match="uninitialized KV"):
+ cache._attend_prepared(0, batch, q)
+ middle = cache._prepare_batch(["0"], [2], [1])
+ cache._write_prepared(0, middle, k[:1], k[:1])
+ assert torch.isfinite(cache._attend_prepared(0, batch, q)).all()
+ with pytest.raises(RuntimeError, match="uninitialized KV"):
+ cache._attend_prepared(1, batch, q)
+ cache.truncate_suffix("0", 0)
+ with pytest.raises(RuntimeError, match="uninitialized KV"):
+ cache._attend_prepared(0, batch, q)
+ cache.free("0")
+ assert cache.allocate("0", 3)
+ before = cache.key_cache.clone()
+ with pytest.raises(RuntimeError, match="stale prepared KV"):
+ cache._write_prepared(1, batch, k, k)
+ torch.testing.assert_close(cache.key_cache, before, rtol=0, atol=0, equal_nan=True)
+ for rid in ("0", "1", "2"):
+ cache.free(rid)
+
+
+@pytest.mark.npu
+@torch.inference_mode()
+def test_layout_eviction_and_cross_stream_use_keep_returned_outputs():
+ cache, _ = _cache()
+ kernel = cache.attention.kernel
+ q = torch.randn(3, 4, 64, device=cache.device, dtype=cache.dtype)
+ producer, consumer = torch.npu.Stream(), torch.npu.Stream()
+ producer.wait_stream(torch.npu.current_stream())
+ batch = cache._prepare_batch(["0", "1", "2"], [0] * 3, [2, 6, 16], for_write=False)
+ # The caller owns readiness of KV and descriptor staging; cached layouts
+ # are private to each stream and are never overwritten for a later batch.
+ torch.npu.synchronize()
+ with torch.npu.stream(producer):
+ held = cache._attend_prepared(0, batch, q)
+ snapshot = held.clone()
+ done = torch.npu.Event()
+ done.record()
+ consumer.wait_event(done)
+ with torch.npu.stream(consumer):
+ second = cache._attend_prepared(0, batch, q)
+ for position in range(2, 14):
+ newer = cache._prepare_batch(
+ ["0", "1", "2"], [0] * 3, [2, 6, position], for_write=False
+ )
+ cache._attend_prepared(0, newer, q)
+ torch.npu.synchronize()
+ torch.testing.assert_close(second, snapshot, rtol=0, atol=0)
+ torch.testing.assert_close(held, snapshot, rtol=0, atol=0)
+ assert len(kernel._layouts) <= 8
+ assert kernel._layout_bytes <= kernel._layout_limit
+ for rid in ("0", "1", "2"):
+ cache.free(rid)
+
+
+@pytest.mark.parametrize("kind", ["ragged_decode", "irregular"])
+@torch.inference_mode()
+def test_failed_plan_preparation_invalidates_the_previous_descriptor(kind, monkeypatch):
+ from copy import copy
+
+ from tests.test_ascend_eager import _batched_fixture
+
+ q, keys, values, first = _batched_fixture(kind)
+ kernel = AscendAttention()
+ expected = kernel.attend_metadata(q, keys, values, first).clone()
+ second = copy(first)
+ target = "_prepare_decode" if kind == "ragged_decode" else "_plan"
+ original = getattr(kernel, target)
+
+ def failed(*args, **kwargs):
+ raise RuntimeError("injected plan preparation failure")
+
+ monkeypatch.setattr(kernel, target, failed)
+ with pytest.raises(RuntimeError, match="injected plan preparation"):
+ kernel.attend_metadata(q, keys, values, second)
+ monkeypatch.setattr(kernel, target, original)
+ actual = kernel.attend_metadata(q, keys, values, first)
+ torch.testing.assert_close(actual, expected, rtol=0, atol=0)
diff --git a/tests/test_ascend_packed_prefill.py b/tests/test_ascend_packed_prefill.py
new file mode 100644
index 00000000..12ddcc5e
--- /dev/null
+++ b/tests/test_ascend_packed_prefill.py
@@ -0,0 +1,56 @@
+"""Ascend attention consumes the common packed KV descriptor."""
+
+import pytest
+import torch
+
+from vllm_rlt.attention.backend import AttentionBackend, BackendCapabilities
+from vllm_rlt.core.kv_cache_manager import KVCacheManager
+from vllm_rlt.device import resolve_device
+from vllm_rlt.kernels.ascend_attention import AscendAttention
+
+
+@pytest.mark.parametrize("device_name", ["cpu", pytest.param("npu:0", marks=pytest.mark.npu)])
+@torch.inference_mode()
+def test_packed_prefill_mixes_cached_chunks_and_single_queries(device_name):
+ device = resolve_device(device_name)
+ if device.type == "npu":
+ torch.npu.set_device(device)
+ attention = AttentionBackend(
+ AscendAttention(),
+ BackendCapabilities(packed_prefill=True),
+ {"backend": "ascend_eager", "selected_backend": "ascend_eager"},
+ )
+ cache = KVCacheManager(
+ 1, 2, 64, 16, 16, 2, device=device, dtype=torch.bfloat16, attention=attention
+ )
+ for request in ("a", "b", "c"):
+ assert cache.allocate(request, 3)
+
+ def values(items):
+ return (
+ torch.tensor(items, device=device, dtype=cache.dtype)
+ .view(-1, 1, 1)
+ .expand(-1, cache.num_kv_heads, cache.head_dim)
+ .contiguous()
+ )
+
+ history = values([1, 10, 20, 100])
+ cache.write(
+ 0, ["a", "b", "b", "c"], [0, 1, 1, 0], [0, 0, 1, 0], torch.zeros_like(history), history
+ )
+ ids, depths, positions = ["a", "a", "b", "c"], [0, 0, 1, 0], [1, 2, 2, 1]
+ packed = cache._prepare_batch(ids, depths, positions, packed_prefill=True)
+ assert packed.cu_seqlens_q.tolist() == [0, 2, 3, 4]
+ assert packed.context_lengths.tolist() == [3, 3, 2]
+ assert packed.block_tables.shape[0] == 3
+
+ new = values([3, 8, 30, 120])
+ query = torch.zeros_like(new)
+ cache._write_prepared(0, packed, torch.zeros_like(new), new)
+ actual = cache._attend_prepared(0, packed, query)
+ expected = values([2, 4, 20, 110])
+ torch.testing.assert_close(actual, expected, atol=0, rtol=0)
+
+ per_query = cache._prepare_batch(ids, depths, positions, for_write=False)
+ reference = cache._attend_prepared(0, per_query, query)
+ torch.testing.assert_close(actual, reference, atol=0, rtol=0)
diff --git a/tests/test_ascend_serving.py b/tests/test_ascend_serving.py
new file mode 100644
index 00000000..f1edbd09
--- /dev/null
+++ b/tests/test_ascend_serving.py
@@ -0,0 +1,143 @@
+"""Real socket + spawned NPU worker smoke, opt-in with pinned local weights."""
+
+import asyncio
+import json
+import os
+import signal
+import socket
+import subprocess
+import sys
+import time
+from pathlib import Path
+
+import pytest
+
+pytestmark = pytest.mark.npu
+
+
+@pytest.mark.parametrize(
+ ("backend", "asynchronous"),
+ [("torch", False), ("ascend_eager", False), ("ascend_eager", True)],
+ ids=["torch-sync", "ascend-sync", "ascend-async"],
+)
+def test_checkpoint_http_completions_stream_and_disconnect(tmp_path, backend, asynchronous):
+ import aiohttp
+
+ source = os.environ.get("VLLM_RLT_OURO_CHECKPOINT")
+ if not source:
+ pytest.skip("set VLLM_RLT_OURO_CHECKPOINT to the pinned local Ouro-1.4B checkpoint")
+ mode = backend + ("-async" if asynchronous else "-sync")
+ folder = Path(os.environ.get("VLLM_RLT_ASCEND_RESULTS", tmp_path)) / mode
+ folder.mkdir(parents=True, exist_ok=True)
+ with socket.socket() as listener:
+ listener.bind(("127.0.0.1", 0))
+ port = listener.getsockname()[1]
+ command = [
+ sys.executable,
+ "-m",
+ "vllm_rlt.entrypoints.serve",
+ "--model",
+ source,
+ "--served-model-name",
+ "ouro-ascend",
+ "--device",
+ "npu",
+ "--dtype",
+ "bfloat16",
+ "--attention-backend",
+ backend,
+ "--num-blocks",
+ "64",
+ "--max-num-seqs",
+ "2",
+ "--max-num-batched-tokens",
+ "32",
+ "--port",
+ str(port),
+ ]
+ if asynchronous:
+ command.extend(["--async-scheduling", "--exit-mode", "ouro_delayed", "--single-stream"])
+ with (folder / "service.log").open("w") as log:
+ process = subprocess.Popen(
+ command, stdout=log, stderr=subprocess.STDOUT, start_new_session=True
+ )
+
+ async def exercise():
+ url = f"http://127.0.0.1:{port}"
+ async with aiohttp.ClientSession(timeout=aiohttp.ClientTimeout(total=60)) as client:
+ deadline = time.monotonic() + 120
+ while True:
+ assert process.poll() is None, (folder / "service.log").read_text()
+ try:
+ async with client.get(url + "/health") as response:
+ if response.status == 200:
+ break
+ except aiohttp.ClientConnectionError:
+ pass
+ assert time.monotonic() < deadline, "NPU service startup timed out"
+ await asyncio.sleep(0.25)
+
+ def body(prompt, **extra):
+ return dict(
+ model="ouro-ascend",
+ prompt=prompt,
+ max_tokens=4,
+ temperature=0,
+ min_loops=4,
+ max_loops=4,
+ exit_threshold=1,
+ ignore_eos=True,
+ **extra,
+ )
+
+ async def complete(prompt):
+ async with client.post(url + "/v1/completions", json=body(prompt)) as response:
+ result = await response.json()
+ assert response.status == 200, result
+ assert result["usage"]["completion_tokens"] == 4
+ return result
+
+ prompts = ["The capital of France is", "One plus one equals"]
+ normal = await asyncio.gather(*(complete(prompt) for prompt in prompts))
+ events = []
+ async with client.post(
+ url + "/v1/completions",
+ json=body(prompts[0], stream=True, stream_options={"include_usage": True}),
+ ) as response:
+ assert response.status == 200
+ async for line in response.content:
+ if line.startswith(b"data: "):
+ data = line[6:].strip().decode()
+ events.append(data if data == "[DONE]" else json.loads(data))
+ assert events[-1] == "[DONE]"
+ text = "".join(
+ choice["text"] for event in events[:-1] for choice in event.get("choices", [])
+ )
+ assert text == normal[0]["choices"][0]["text"]
+ assert any(
+ isinstance(event, dict)
+ and (event.get("usage") or {}).get("completion_tokens") == 4
+ for event in events
+ )
+ # Close a real streaming connection early, then verify the worker
+ # remains usable. Direct engine tests check page reclamation.
+ async with client.post(
+ url + "/v1/completions", json=body(prompts[1], stream=True)
+ ) as response:
+ assert response.status == 200
+ await response.content.readline()
+ after_disconnect = await complete(prompts[1])
+ assert after_disconnect["choices"][0]["text"] == normal[1]["choices"][0]["text"]
+ return dict(normal=normal, stream=events, after_disconnect=after_disconnect)
+
+ try:
+ result = asyncio.run(exercise())
+ (folder / "service-results.json").write_text(json.dumps(result, indent=2))
+ finally:
+ process.terminate()
+ try:
+ process.wait(timeout=30)
+ except subprocess.TimeoutExpired:
+ os.killpg(process.pid, signal.SIGKILL)
+ process.wait(timeout=10)
+ assert process.returncode == 0, (folder / "service.log").read_text()
diff --git a/tests/test_attention_module.py b/tests/test_attention_module.py
index 5c4de6dd..823cf322 100644
--- a/tests/test_attention_module.py
+++ b/tests/test_attention_module.py
@@ -155,3 +155,73 @@ def stage(values, dtype):
assert len({batch.cu_seqlens_q.data_ptr() for batch in batches}) == 1
assert batches[0].cu_seqlens_q.tolist() == [0, 2, 3]
assert batches[0].max_seqlen_q == 2
+
+
+def test_prepared_attention_keeps_the_legacy_kernel_signature():
+ cache = KVCacheManager(1, 1, 4, 4, 2, 1)
+ assert cache.allocate("a", 3)
+ batch = cache._prepare_batch(["a"] * 3, [0] * 3, [0, 1, 2])
+ keys = torch.zeros(3, 1, 4)
+ values = torch.arange(12, dtype=torch.float32).reshape(3, 1, 4)
+ query = torch.zeros(3, 2, 4)
+ cache._write_prepared(0, batch, keys, values)
+ kernel = cache.attention.kernel
+ expected = kernel(
+ query,
+ cache.key_cache[:, 0],
+ cache.value_cache[:, 0],
+ batch.block_tables,
+ batch.context_lengths,
+ )
+ seen = []
+
+ def legacy(*args):
+ seen.append(args)
+ return kernel(*args)
+
+ cache.attention.kernel = legacy
+ actual = cache._attend_prepared(0, batch, query)
+ torch.testing.assert_close(actual, expected, atol=0, rtol=0)
+ assert len(seen) == 1 and len(seen[0]) == 5
+ assert seen[0][0] is query
+ assert seen[0][3] is batch.block_tables
+ assert seen[0][4] is batch.context_lengths
+ cache.free("a")
+
+
+def test_metadata_attention_borrows_only_a_live_validated_descriptor():
+ cache = KVCacheManager(1, 1, 4, 4, 2, 1)
+ assert cache.allocate("a", 3)
+ batch = cache._prepare_batch(["a"] * 3, [0] * 3, [0, 1, 2])
+ keys = torch.zeros(3, 1, 4)
+ values = torch.arange(12, dtype=torch.float32).reshape(3, 1, 4)
+ query = torch.zeros(3, 2, 4)
+ cache._write_prepared(0, batch, keys, values)
+ legacy = cache.attention.kernel
+ expected = legacy(
+ query,
+ cache.key_cache[:, 0],
+ cache.value_cache[:, 0],
+ batch.block_tables,
+ batch.context_lengths,
+ )
+ seen = []
+
+ class MetadataKernel:
+ def __call__(self, *args):
+ pytest.fail("metadata-aware kernels must not read device metadata back")
+
+ def attend_metadata(self, q, k, v, descriptor):
+ seen.append(descriptor)
+ return legacy(q, k, v, descriptor.block_tables, descriptor.context_lengths)
+
+ cache.attention.kernel = MetadataKernel()
+ actual = cache._attend_prepared(0, batch, query)
+ torch.testing.assert_close(actual, expected, atol=0, rtol=0)
+ assert len(seen) == 1 and seen[0] is batch
+ cache.free("a")
+ assert cache.allocate("a", 3)
+ with pytest.raises((ValueError, RuntimeError)):
+ cache._attend_prepared(0, batch, query)
+ assert len(seen) == 1
+ cache.free("a")
diff --git a/tests/test_cdb_runtime.py b/tests/test_cdb_runtime.py
index b6e224f6..740572f0 100644
--- a/tests/test_cdb_runtime.py
+++ b/tests/test_cdb_runtime.py
@@ -650,7 +650,15 @@ def test_cuda_dynamic_async_respects_buffers_and_matches_sync(layout, multi_stre
expected = LLM(base, **common).generate(prompts, params)
execution = ExecutionConfig(async_scheduling=True, multi_stream=multi_stream)
actual = LLM(base, execution_config=execution, **common)
- assert actual.engine.execution_config == execution
+ # Automatic fields resolve at engine construction; other switches must survive.
+ assert (
+ replace(
+ actual.engine.execution_config,
+ cuda_graph_max_batch_size=execution.cuda_graph_max_batch_size,
+ packed_prefill=execution.packed_prefill,
+ )
+ == execution
+ )
assert actual.engine.model_runner.requestspaces == {}
assert actual.engine.model_runner.states is actual.engine.model_runner.async_state.hidden
result = actual.generate(prompts, params)
diff --git a/tests/test_device.py b/tests/test_device.py
new file mode 100644
index 00000000..c33f8282
--- /dev/null
+++ b/tests/test_device.py
@@ -0,0 +1,83 @@
+"""Device bootstrap and fail-fast NPU scope without a device dependency."""
+
+import importlib
+from types import SimpleNamespace
+
+import pytest
+import torch
+
+from tests.helpers import tiny_ouro_config
+from vllm_rlt import CacheConfig, ExecutionConfig, ExitConfig, SchedulerConfig
+from vllm_rlt.core.memory import plan_cache
+from vllm_rlt.device import resolve_device, stream_module
+from vllm_rlt.engine.stage_engine import StageEngine
+from vllm_rlt.profiling import ProfileConfig
+
+
+def test_cpu_device_resolution_does_not_import_optional_backend(monkeypatch):
+ monkeypatch.setattr(
+ importlib, "import_module", lambda name: pytest.fail(f"unexpected import: {name}")
+ )
+ assert resolve_device("cpu") == torch.device("cpu")
+ assert resolve_device("cuda:2") == torch.device("cuda:2")
+ assert stream_module("cpu") is None
+ assert stream_module("cuda:2") is torch.cuda
+
+
+def npu_model_stub(**overrides):
+ parameter = SimpleNamespace(
+ device=SimpleNamespace(type="npu"), dtype=torch.bfloat16, element_size=lambda: 2
+ )
+ return SimpleNamespace(
+ parameters=lambda: iter([parameter]), config=tiny_ouro_config(**overrides)
+ )
+
+
+@pytest.mark.parametrize(
+ "options",
+ [
+ dict(execution_config=ExecutionConfig(cuda_graphs=True)),
+ dict(cache_config=CacheConfig(num_blocks=32, enable_prefix_caching=True)),
+ dict(scheduler_config=SchedulerConfig(enable_preemption=True)),
+ dict(exit_config=ExitConfig(mode="random_lookahead")),
+ ],
+)
+def test_npu_unsupported_execution_rejected_before_allocation(options):
+ with pytest.raises(ValueError, match="NPU Ouro requires local execution"):
+ StageEngine(npu_model_stub(), **options)
+
+
+@pytest.mark.parametrize("backend", ["torch", "ascend_eager"])
+def test_npu_requires_bf16_kv_before_allocation(backend):
+ with pytest.raises(ValueError, match="NPU Ouro requires BF16 KV cache"):
+ StageEngine(
+ npu_model_stub(),
+ cache_config=CacheConfig(kv_cache_dtype="fp8_e4m3"),
+ attention_backend=backend,
+ )
+
+
+def test_npu_does_not_use_cpu_default_memory_budget():
+ model = npu_model_stub()
+ with pytest.raises(ValueError, match="explicit num_blocks"):
+ plan_cache(model, CacheConfig(), SchedulerConfig(), ExecutionConfig(), "torch")
+ blocks, info = plan_cache(
+ model, CacheConfig(num_blocks=17), SchedulerConfig(), ExecutionConfig(), "torch"
+ )
+ assert blocks == 17 and info["source"] == "blocks"
+
+
+@pytest.mark.parametrize("backend", ["auto", "triton", "flash_attn_4"])
+def test_npu_requires_explicit_torch_before_allocation(backend):
+ with pytest.raises(ValueError, match="explicit attention_backend='torch'"):
+ StageEngine(npu_model_stub(), attention_backend=backend)
+
+
+def test_builtin_profiler_does_not_mislabel_npu_as_cpu_only(tmp_path):
+ with pytest.raises(ValueError, match="torch_npu.profiler"):
+ ProfileConfig(str(tmp_path)).resolve_activities("npu:0")
+
+
+def test_npu_async_requires_explicit_delayed_exit_policy():
+ with pytest.raises(ValueError, match="async scheduling requires ouro_delayed"):
+ StageEngine(npu_model_stub(), execution_config=ExecutionConfig(async_scheduling=True))
diff --git a/tests/test_fast_execution_plan.py b/tests/test_fast_execution_plan.py
new file mode 100644
index 00000000..85bd0528
--- /dev/null
+++ b/tests/test_fast_execution_plan.py
@@ -0,0 +1,103 @@
+"""Ownership and failure boundaries of Ascend recurrent execution."""
+
+import pytest
+import torch
+
+from tests.test_ascend import make_npu_model
+from vllm_rlt.core.kv_cache_manager import KVCacheManager
+from vllm_rlt.device import resolve_device
+from vllm_rlt.request import Operation
+from vllm_rlt.worker.executor import ForwardBatch, ModelRuntime
+
+pytestmark = pytest.mark.npu
+
+
+def cache_for(model):
+ c = model.config
+ cache = KVCacheManager(
+ c.num_hidden_layers,
+ c.num_key_value_heads,
+ c.head_dim,
+ 32,
+ 16,
+ 4,
+ device="npu:0",
+ dtype=torch.bfloat16,
+ backend="ascend_eager",
+ )
+ assert cache.allocate("a", 8)
+ return cache
+
+
+def metadata(cache, start, count):
+ return cache._prepare_batches(
+ ["a"] * count,
+ [[depth] * count for depth in range(4)],
+ list(range(start, start + count)),
+ )
+
+
+@torch.inference_mode()
+def test_failed_traversal_keeps_partial_layer_coverage_visible():
+ resolve_device("npu")
+ torch.npu.set_device(0)
+ model = make_npu_model(torch.device("npu:0"))
+ cache = cache_for(model)
+ runtime = ModelRuntime(model, cache)
+ batch = metadata(cache, 0, 3)[0]
+
+ def fail(*args):
+ raise RuntimeError("injected layer failure")
+
+ handle = model.model.layers[1].register_forward_pre_hook(fail)
+ try:
+ hidden = model.prelude(torch.tensor([3, 7, 11], device="npu:0"))
+ with pytest.raises(RuntimeError, match="injected layer failure"):
+ runtime.forward(ForwardBatch(Operation.RECURRENT, hidden=hidden, metadata=(batch,)))
+ finally:
+ handle.remove()
+ with pytest.raises(RuntimeError, match="every layer"):
+ cache.finalize_token("a", 1, 0)
+ q = torch.zeros(
+ 3,
+ model.config.num_attention_heads,
+ model.config.head_dim,
+ device="npu:0",
+ dtype=torch.bfloat16,
+ )
+ assert torch.isfinite(cache._attend_prepared(0, batch, q)).all()
+ with pytest.raises(RuntimeError, match="uninitialized KV history"):
+ cache._attend_prepared(1, batch, q)
+ # Rerun all layers for the same page slots after the interrupted write.
+ actual = runtime.forward(
+ ForwardBatch(Operation.RECURRENT, hidden=hidden, metadata=(batch,))
+ ).hidden
+ reference = cache_for(model)
+ expected = model.recurrent_prepared(hidden, metadata(reference, 0, 3)[0], reference)[0]
+ torch.testing.assert_close(actual, expected, atol=0, rtol=0)
+ cache.free("a")
+ reference.free("a")
+
+
+@torch.inference_mode()
+def test_paged_coverage_after_truncation_and_restore():
+ resolve_device("npu")
+ torch.npu.set_device(0)
+ model = make_npu_model(torch.device("npu:0"))
+ cache = cache_for(model)
+ runtime = ModelRuntime(model, cache)
+ hidden = model.prelude(torch.tensor([3, 7, 11], device="npu:0"))
+ for batch in metadata(cache, 0, 3):
+ runtime.forward(ForwardBatch(Operation.RECURRENT, hidden=hidden, metadata=(batch,)))
+ cache.finalize_token("a", 2, 3)
+ snapshot = cache.snapshot("a")
+ cache.truncate_suffix("a", 1)
+ with pytest.raises(RuntimeError, match="every layer"):
+ cache.finalize_token("a", 2, 3)
+ cache.truncate_suffix("a", 0)
+ cache.restore("a", snapshot)
+ cache.finalize_token("a", 2, 3)
+ cache.truncate_suffix("a", 1)
+ with pytest.raises(RuntimeError, match="every layer"):
+ cache.finalize_token("a", 2, 3)
+ cache.free("a")
diff --git a/tests/test_npu_async.py b/tests/test_npu_async.py
new file mode 100644
index 00000000..52ba27de
--- /dev/null
+++ b/tests/test_npu_async.py
@@ -0,0 +1,238 @@
+"""NPU event ownership and device-token lookahead preserve eager execution."""
+
+import pytest
+import torch
+
+from tests.test_ascend import make_npu_model
+from vllm_rlt import CacheConfig, ExecutionConfig, ExitConfig, SamplingParams, SchedulerConfig
+from vllm_rlt.core.scheduler import ScheduledRequest, Scheduler, SchedulerOutput
+from vllm_rlt.device import resolve_device
+from vllm_rlt.engine.stage_engine import StageEngine
+from vllm_rlt.request import Request, RequestPhase
+
+
+def test_fixed_depth_decode_keeps_each_loop_schedulable():
+ scheduler = Scheduler(SchedulerConfig(), None, total_ut_steps=4, exit_mode="ouro_delayed")
+ request = Request("a", [1], SamplingParams(min_loops=1, max_loops=4))
+ request.phase = RequestPhase.RECURRENT
+ scheduler.requests["a"] = request
+ for depth in range(4):
+ assert request.phase == RequestPhase.RECURRENT
+ item = ScheduledRequest(request, depth=depth, phase=RequestPhase.RECURRENT)
+ assert scheduler._planned_loop_count([item]) == 1
+ scheduler.update_after_submission(SchedulerOutput(RequestPhase.RECURRENT, [item], 1))
+ assert request.loops_done == depth + 1
+ assert request.phase == RequestPhase.CODA
+
+
+def make_engine(model, asynchronous, multi_stream):
+ return StageEngine(
+ model,
+ attention_backend="ascend_eager",
+ cache_config=CacheConfig(num_blocks=128, block_size=16, incremental_allocation=True),
+ scheduler_config=SchedulerConfig(
+ max_num_seqs=3, max_num_batched_tokens=128, prefill_chunk_size=128
+ ),
+ execution_config=ExecutionConfig(
+ async_scheduling=asynchronous,
+ multi_stream=multi_stream,
+ static_buffers=False,
+ pad_to_power_of_two=False,
+ use_uva=False,
+ cuda_graphs=False,
+ ),
+ exit_config=ExitConfig("ouro_delayed" if asynchronous else "ouro"),
+ )
+
+
+def drain(engine):
+ got = {}
+ for _ in range(300):
+ for output in engine.step():
+ if output.finished:
+ got[output.request_id] = (
+ output.token_ids,
+ output.exit_depths,
+ output.finish_reason,
+ )
+ if not engine.has_unfinished_requests():
+ assert engine.cache_manager.num_used_blocks == 0
+ return got
+ pytest.fail("NPU async engine did not drain")
+
+
+@pytest.mark.npu
+@pytest.mark.parametrize("multi_stream", [False, True])
+@torch.inference_mode()
+def test_async_npu_matches_sync_without_device_barriers(monkeypatch, multi_stream):
+ resolve_device("npu")
+ torch.npu.set_device(0)
+ model = make_npu_model(torch.device("npu:0"))
+ prompts = [[3, 5, 7] * 7, [2, 11, 17], [13, 19] * 9]
+ params = SamplingParams(max_tokens=4, min_loops=4, max_loops=4, ignore_eos=True)
+ reference = None
+ for asynchronous in (False, True):
+ engine = make_engine(model, asynchronous, multi_stream)
+ try:
+ for repeat in range(2):
+ for i, prompt in enumerate(prompts):
+ engine.add_request(str(i), prompt, params)
+ with monkeypatch.context() as patch:
+
+ def forbidden(*args, **kwargs):
+ pytest.fail("submission called device-wide synchronize")
+
+ patch.setattr(torch.npu, "synchronize", forbidden)
+ actual = drain(engine)
+ assert all(depths == [4] * 4 for _, depths, _ in actual.values())
+ if reference is None:
+ reference = actual
+ else:
+ assert actual == reference
+ finally:
+ engine.close()
+
+
+@pytest.mark.npu
+@torch.inference_mode()
+def test_npu_delayed_delivery_consumes_device_token_before_host_result():
+ resolve_device("npu")
+ torch.npu.set_device(0)
+ model = make_npu_model(torch.device("npu:0"))
+ engine = make_engine(model, True, False)
+ params = SamplingParams(max_tokens=3, min_loops=4, max_loops=4, ignore_eos=True)
+
+ class HeldEvent:
+ def query(self):
+ return False
+
+ def synchronize(self):
+ pytest.fail("host delivery was required before the next device input")
+
+ try:
+ engine.add_request("a", [3, 7, 11], params)
+ engine.step()
+ engine.step()
+ ticket = engine._pending_coda[0]
+ real_event = ticket.event
+ ticket.event = HeldEvent()
+ try:
+ request = engine.scheduler.requests["a"]
+ assert request.generated_token_ids == []
+ engine.step()
+ assert engine.last_schedule.phase == RequestPhase.PRELUDE
+ engine.step()
+ assert engine.last_schedule.phase == RequestPhase.RECURRENT
+ assert request.loops_done == 4
+ assert request.generated_token_ids == []
+ assert ticket.device_values.device.type == "npu"
+ finally:
+ ticket.event = real_event
+ assert drain(engine)["a"][1] == [4, 4, 4]
+ finally:
+ engine.close()
+
+
+@pytest.mark.npu
+@torch.inference_mode()
+def test_npu_cancel_reuse_rejects_late_coda_and_eos_stops():
+ resolve_device("npu")
+ torch.npu.set_device(0)
+ model = make_npu_model(torch.device("npu:0"))
+ params = SamplingParams(max_tokens=3, min_loops=4, max_loops=4, ignore_eos=True)
+ expected_engine = make_engine(model, False, False)
+ expected_engine.add_request("cancel", [13, 17, 19], params)
+ expected_engine.add_request("keep", [3, 7, 11], params)
+ expected = drain(expected_engine)
+ expected_engine.close()
+ engine = make_engine(model, True, True)
+ try:
+ engine.add_request("cancel", [2, 5] * 9, params)
+ engine.add_request("keep", [3, 7, 11], params)
+ engine.step()
+ engine.step()
+ engine.abort_request("cancel")
+ engine.add_request("cancel", [13, 17, 19], params)
+ assert drain(engine) == expected
+ eos = model.config.eos_token_id
+ engine.model_runner.sampler.sample_batch = lambda logits, options, generators: (
+ logits.new_full((len(options),), eos, dtype=torch.long)
+ )
+ engine.add_request("eos", [3, 7], SamplingParams(max_tokens=4, min_loops=4, max_loops=4))
+ assert drain(engine)["eos"] == ([eos], [4], "stop")
+ finally:
+ engine.close()
+
+
+@pytest.mark.npu
+def test_npu_batched_greedy_preserves_ties_and_padding():
+ from vllm_rlt.worker.ascend_model_runner import AscendSampler
+
+ device = resolve_device("npu:0")
+ logits = torch.zeros((3, 49152), dtype=torch.bfloat16)
+ logits[0, 1] = 3
+ logits[0, 40000] = 3
+ logits[1, 0] = 4
+ logits[1, -1] = 4
+ logits[2, 0] = 9
+ logits = logits.to(device)
+ params = (SamplingParams(), SamplingParams())
+ actual = AscendSampler().sample_batch(logits, params, (None, None))
+ expected = torch.stack([row.argmax() for row in logits[:2]])
+ assert actual.cpu().tolist() == [1, 0]
+ assert torch.equal(actual, expected)
+
+
+@pytest.mark.npu
+@torch.inference_mode()
+def test_npu_hidden_gather_keeps_cross_stream_sources_alive():
+ from types import SimpleNamespace
+
+ from vllm_rlt.worker.model_runner import ModelRunner
+
+ device = resolve_device("npu:0")
+ torch.npu.set_device(device)
+ producer = torch.npu.Stream(device=device)
+ consumer = torch.npu.Stream(device=device)
+ blocker = torch.npu.Stream(device=device)
+ with torch.npu.stream(producer):
+ source = torch.full((64,), 7.0, dtype=torch.bfloat16, device=device)
+ producer.synchronize()
+ with torch.npu.stream(consumer):
+ warm = torch.stack([source])
+ consumer.synchronize()
+ del warm
+ pointer = source.data_ptr()
+ request = SimpleNamespace(request_id="a", hidden_state=source)
+ batch = SimpleNamespace(spans=[SimpleNamespace(request=request)])
+ matrix = torch.ones((16384, 16384), dtype=torch.bfloat16, device=device)
+ product = torch.empty_like(matrix)
+ blocker.wait_stream(torch.npu.current_stream(device))
+ with torch.npu.stream(blocker):
+ for _ in range(4):
+ torch.mm(matrix, matrix, out=product)
+ gate = torch.npu.Event()
+ gate.record(blocker)
+ consumer.wait_event(gate)
+ runner = SimpleNamespace(events={})
+ try:
+ with torch.npu.stream(consumer):
+ ModelRunner._wait_for_requests(runner, batch, consumer)
+ actual = ModelRunner._gather(runner, [request], None, 1)
+ request.hidden_state = None
+ del source
+ poisoned = []
+ with torch.npu.stream(producer):
+ for _ in range(32):
+ value = torch.empty((64,), dtype=torch.bfloat16, device=device)
+ value.fill_(99.0)
+ poisoned.append(value)
+ assert not gate.query(), "consumer delay expired before the source reuse check"
+ assert all(value.data_ptr() != pointer for value in poisoned)
+ producer.synchronize()
+ consumer.synchronize()
+ assert actual.cpu().tolist() == [[7.0] * 64]
+ finally:
+ producer.synchronize()
+ consumer.synchronize()
+ blocker.synchronize()
diff --git a/tests/test_npu_batch_decode.py b/tests/test_npu_batch_decode.py
new file mode 100644
index 00000000..6c5dac9e
--- /dev/null
+++ b/tests/test_npu_batch_decode.py
@@ -0,0 +1,236 @@
+"""Decode scheduling and KV completion events preserve request lifetimes."""
+
+from contextlib import nullcontext
+
+import pytest
+import torch
+
+from tests.helpers import make_cpu_cache, tiny_ouro_model
+from vllm_rlt import ExecutionConfig, SamplingParams, SchedulerConfig
+from vllm_rlt.request import Request
+from vllm_rlt.worker.ascend_model_runner import AscendModelRunner
+from vllm_rlt.worker.model_runner import ModelRunner
+
+
+def runner_cpu():
+ model = tiny_ouro_model()
+ return ModelRunner(
+ model,
+ make_cpu_cache(model.config),
+ execution_config=ExecutionConfig(
+ async_scheduling=True,
+ multi_stream=False,
+ static_buffers=False,
+ pad_to_power_of_two=False,
+ use_uva=False,
+ ),
+ scheduler_config=SchedulerConfig(max_num_seqs=3, max_num_batched_tokens=8),
+ )
+
+
+def requests():
+ return [Request(x, [1], SamplingParams(max_tokens=3)) for x in ["a", "b"]]
+
+
+def test_cpu_keeps_copy_semantics():
+ runner = runner_cpu()
+ reqs = requests()
+ hidden = torch.arange(64.0).reshape(2, 32)
+ runner._save_batch(reqs, hidden, None)
+ expected = reqs[0].hidden_state.clone()
+ hidden.fill_(99)
+ assert torch.equal(reqs[0].hidden_state, expected)
+
+
+class Event:
+ def __init__(self):
+ self.recorded = []
+
+ def record(self, stream):
+ self.recorded.append(stream)
+
+
+class Stream:
+ def __init__(self):
+ self.waited = []
+ self.synchronized = 0
+
+ def wait_event(self, event):
+ self.waited.append(event)
+
+ def synchronize(self):
+ self.synchronized += 1
+
+
+class DeviceOps:
+ def __init__(self):
+ self.created = []
+
+ def stream(self, stream):
+ return nullcontext()
+
+ def Event(self):
+ e = Event()
+ self.created.append(e)
+ return e
+
+
+def write_depth(runner, reqs, depth, *, layers=None):
+ cache = runner.cache_manager
+ for r in reqs:
+ assert cache.allocate(r.request_id, 4)
+ r.loops_done = depth + 1
+ c = runner.model.config
+ data = torch.arange(
+ len(reqs) * c.num_key_value_heads * c.head_dim, dtype=torch.float32
+ ).reshape(len(reqs), c.num_key_value_heads, c.head_dim)
+ for layer in range(cache.num_layers) if layers is None else layers:
+ cache.write(
+ layer,
+ [r.request_id for r in reqs],
+ [depth] * len(reqs),
+ [0] * len(reqs),
+ data,
+ data + 1,
+ )
+ return data
+
+
+def fake_events(runner, reqs):
+ runner.boundary_stream = Stream()
+ runner.device_ops = DeviceOps()
+ source = Event()
+ runner.events = {r.request_id: source for r in reqs}
+ return source
+
+
+def test_full_depth_finalize_reuses_event_and_still_validates():
+ runner = runner_cpu()
+ reqs = requests()
+ write_depth(runner, reqs, 3)
+ source = fake_events(runner, reqs)
+ AscendModelRunner._finalize_npu_batch(runner, reqs)
+ assert all(runner.events[r.request_id] is source for r in reqs)
+ assert not runner.device_ops.created and not runner.boundary_stream.waited
+ bad = runner_cpu()
+ rs = requests()
+ write_depth(bad, rs, 3, layers=[0])
+ fake_events(bad, rs)
+ with pytest.raises(RuntimeError, match="every layer"):
+ AscendModelRunner._finalize_npu_batch(bad, rs)
+ assert not bad.device_ops.created
+
+
+def test_early_exit_finalize_copies_all_planes_then_shares_event():
+ runner = runner_cpu()
+ reqs = requests()
+ data = write_depth(runner, reqs, 0)
+ source = fake_events(runner, reqs)
+ AscendModelRunner._finalize_npu_batch(runner, reqs)
+ assert runner.boundary_stream.waited == [source]
+ assert len(runner.device_ops.created) == 1
+ done = runner.device_ops.created[0]
+ assert done.recorded == [runner.boundary_stream]
+ assert all(runner.events[r.request_id] is done for r in reqs)
+ for i, r in enumerate(reqs):
+ for depth in range(1, 4):
+ for layer in range(runner.cache_manager.num_layers):
+ k, v = runner.cache_manager.read(layer, r.request_id, depth, 1)
+ assert torch.equal(k[0], data[i]) and torch.equal(v[0], data[i] + 1)
+
+
+def test_partial_finalize_failure_drains_copies_before_cleanup():
+ runner = runner_cpu()
+ reqs = requests()
+ write_depth(runner, reqs, 0)
+ fake_events(runner, reqs)
+ # Keep allocation live but make the second row lack a required layer.
+ runner.cache_manager.truncate_suffix("b", 0)
+ with pytest.raises(RuntimeError, match="every layer"):
+ AscendModelRunner._finalize_npu_batch(runner, reqs)
+ assert runner.boundary_stream.synchronized == 1
+ assert not runner.device_ops.created
+
+
+@pytest.mark.npu
+@pytest.mark.parametrize("multi_stream", [False, True])
+@torch.inference_mode()
+def test_npu_async_decode_matches_sync_when_reusing_request_ids(multi_stream):
+ from tests.test_ascend import make_npu_model
+ from tests.test_npu_async import drain, make_engine
+ from vllm_rlt.device import resolve_device
+
+ device = resolve_device("npu:0")
+ torch.npu.set_device(device)
+ model = make_npu_model(device)
+ baseline = make_engine(model, False, multi_stream)
+ params = SamplingParams(max_tokens=4, min_loops=4, max_loops=4, ignore_eos=True)
+ prompts = [[3, 7, 11] * 7, [2, 5, 13]]
+ for i, p in enumerate(prompts):
+ baseline.add_request(str(i), p, params)
+ expected = drain(baseline)
+ baseline.close()
+ engine = make_engine(model, True, multi_stream)
+ try:
+ for repeat in range(2):
+ for i, p in enumerate(prompts):
+ engine.add_request(str(i), p, params)
+ assert drain(engine) == expected
+ finally:
+ engine.close()
+
+
+@pytest.mark.npu
+@pytest.mark.parametrize("multi_stream", [False, True])
+@torch.inference_mode()
+def test_npu_partial_depth_finalize_matches_per_request_path(multi_stream):
+ # Serving still requires four loops. Test generic propagation directly.
+ from tests.test_ascend import make_npu_model
+ from tests.test_npu_async import make_engine
+ from vllm_rlt.device import resolve_device
+
+ device = resolve_device("npu:0")
+ torch.npu.set_device(device)
+ model = make_npu_model(device)
+ results = []
+ for optimized in [False, True]:
+ engine = make_engine(model, True, multi_stream)
+ runner, cache = engine.model_runner, engine.cache_manager
+ reqs = [Request(str(i), [1], SamplingParams()) for i in range(3)]
+ try:
+ for r in reqs:
+ assert cache.allocate(r.request_id, 4)
+ with torch.npu.stream(runner.core_stream):
+ for i, depth in enumerate([1, 1, 3]):
+ reqs[i].loops_done = depth + 1
+ shape = (1, model.config.num_key_value_heads, model.config.head_dim)
+ k = torch.full(shape, float(i + 2), device=device, dtype=torch.bfloat16)
+ for layer in range(cache.num_layers):
+ cache.write(layer, [str(i)], [depth], [0], k, k + 1)
+ source = torch.npu.Event()
+ source.record(runner.core_stream)
+ runner.events = {r.request_id: source for r in reqs}
+ if optimized:
+ runner.finalize_many(reqs)
+ assert runner.events["0"] is runner.events["1"]
+ assert runner.events["2"] is source
+ else:
+ for r in reqs:
+ runner.finalize(r)
+ runner.boundary_stream.synchronize()
+ actual = {}
+ for i, first in enumerate([1, 1, 3]):
+ for depth in range(first, 4):
+ for layer in range(cache.num_layers):
+ actual[(i, depth, layer)] = tuple(
+ t.cpu() for t in cache.read(layer, str(i), depth, 1)
+ )
+ results.append(actual)
+ finally:
+ for r in reqs:
+ runner.release(r.request_id)
+ cache.free(r.request_id)
+ engine.close()
+ assert results[0].keys() == results[1].keys()
+ for key in results[0]:
+ assert all(torch.equal(a, b) for a, b in zip(results[0][key], results[1][key]))
diff --git a/vllm_rlt/attention/backend.py b/vllm_rlt/attention/backend.py
index 6a1bf8bb..c5ebdbfa 100644
--- a/vllm_rlt/attention/backend.py
+++ b/vllm_rlt/attention/backend.py
@@ -47,6 +47,24 @@ class AttentionBackend:
def __call__(self, *args, **kwargs):
return self.kernel(*args, **kwargs)
+ def attend_metadata(self, q, keys, values, batch, **kwargs):
+ """Use the prepared batch for packed or per-query attention."""
+ method = getattr(self.kernel, "attend_metadata", None)
+ if method is not None:
+ return method(q, keys, values, batch, **kwargs)
+ if batch.cu_seqlens_q is not None:
+ return self.prefill(
+ q,
+ keys,
+ values,
+ batch.block_tables,
+ batch.context_lengths,
+ batch.cu_seqlens_q,
+ batch.max_seqlen_q,
+ **kwargs,
+ )
+ return self(q, keys, values, batch.block_tables, batch.context_lengths, **kwargs)
+
def prefill(self, *args, **kwargs):
return self.kernel.prefill(*args, **kwargs)
@@ -99,12 +117,23 @@ def backend_capabilities(implementation) -> BackendCapabilities:
return implementation.capabilities
-def validate_backend_name(backend: str) -> None:
- if backend not in {"auto", "torch", "triton", *FLASH_BACKENDS}:
+def validate_backend_name(backend: str, *, device_type: str | None = None) -> None:
+ if device_type == "npu" and backend not in ("torch", "ascend_eager"):
+ raise ValueError(
+ "NPU execution requires explicit attention_backend='torch' or 'ascend_eager'"
+ )
+ if backend not in {"auto", "torch", "triton", "ascend_eager", *FLASH_BACKENDS}:
raise ValueError("unknown attention backend")
def _create_explicit(backend, device, dtype, head_dim, block_size, *, packed, packed_window_shape):
+ if backend == "ascend_eager":
+ if device.type != "npu" or dtype != torch.bfloat16:
+ raise ValueError("ascend_eager requires an NPU and BF16")
+ from vllm_rlt.kernels.ascend_attention import AscendAttention
+
+ capabilities = BackendCapabilities(packed_prefill=packed, async_scheduling=True)
+ return AttentionBackend(AscendAttention(), capabilities, {"backend": backend})
if backend == "triton":
if device.type != "cuda":
raise ValueError("the Triton attention backend requires a CUDA or ROCm device")
@@ -169,7 +198,7 @@ def create_backend(
Engine callers that can calibrate may satisfy an FP8 KV request with a
fixed-scale backend. Cache-only callers must provide a resolved scale.
"""
- validate_backend_name(backend)
+ validate_backend_name(backend, device_type=device.type)
candidates = _AUTO_PRIORITY if backend == "auto" else (backend,)
skipped = []
if backend == "auto" and device.type != "cuda":
diff --git a/vllm_rlt/core/kv_cache_manager.py b/vllm_rlt/core/kv_cache_manager.py
index 04a9c768..c6187d77 100644
--- a/vllm_rlt/core/kv_cache_manager.py
+++ b/vllm_rlt/core/kv_cache_manager.py
@@ -12,6 +12,7 @@
import hashlib
import struct
+import weakref
from collections import OrderedDict
from collections.abc import Callable, Sequence
from copy import deepcopy
@@ -27,6 +28,7 @@
create_backend,
validate_backend_name,
)
+from vllm_rlt.device import resolve_device
from vllm_rlt.request import ExecutionBatch
from vllm_rlt.worker.buffers import stage_metadata
from vllm_rlt.worker.metadata import DeviceKVMetadata, KVMetadata, MetadataBuilder
@@ -81,6 +83,20 @@ def add(self, position: int) -> None:
self.pending.remove(self.prefix)
self.prefix += 1
+ def add_range(self, start: int, end: int) -> None:
+ """Record [start, end), preserving sparse/out-of-order coverage."""
+ if start >= end or end <= self.prefix:
+ return
+ if start > self.prefix:
+ self.pending.update(range(start, end))
+ return
+ self.prefix = end
+ if self.pending:
+ self.pending = {position for position in self.pending if position >= end}
+ while self.prefix in self.pending:
+ self.pending.remove(self.prefix)
+ self.prefix += 1
+
def __contains__(self, position: int) -> bool:
return position < self.prefix or position in self.pending
@@ -94,6 +110,40 @@ class _Allocation:
release_requested: bool = False
+@dataclass(frozen=True)
+class _SequenceCoverage:
+ allocation: _Allocation
+ plane: int
+ ranges: tuple[tuple[int, int], ...]
+
+
+@dataclass(frozen=True)
+class _PreparedCoverage:
+ allocation_epoch: int
+ sequences: Sequence[_SequenceCoverage]
+
+
+def _sequence_coverage(rows, plane):
+ """Collapse query rows once, leaving written state owned by the allocation."""
+ grouped = {}
+ for allocation, depth, position in rows:
+ key = (id(allocation), plane(depth))
+ if key not in grouped:
+ grouped[key] = (allocation, [])
+ grouped[key][1].append(position)
+ sequences = []
+ for (_, storage_plane), (allocation, positions) in grouped.items():
+ ranges = []
+ for position in sorted(positions):
+ if ranges and position <= ranges[-1][1]:
+ start, end = ranges[-1]
+ ranges[-1] = (start, max(end, position + 1))
+ else:
+ ranges.append((position, position + 1))
+ sequences.append(_SequenceCoverage(allocation, storage_plane, tuple(ranges)))
+ return tuple(sequences)
+
+
@dataclass(frozen=True)
class KVSnapshot:
"""What preemption keeps for one request: its pages and its written state.
@@ -187,7 +237,7 @@ def __init__(
):
raise ValueError("recurrent_layers must be valid cache layer indices")
self.storage_depths = max_loops if layout == "last_exited" else 1
- self.device = torch.device(device)
+ self.device = resolve_device(device)
self.dtype = dtype
self.attention = attention or create_backend(
backend,
@@ -230,6 +280,14 @@ def __init__(
self.device = self.key_cache.device
self._free_blocks = list(reversed(range(num_blocks)))
self._allocations: dict[str, _Allocation] = {}
+ self._allocation_epoch = 0
+ # Only the Ascend host-metadata path changes. Weak keys must not keep
+ # request descriptors or their staged tensor storage alive.
+ self._prepared_coverage = (
+ weakref.WeakKeyDictionary()
+ if self.device.type == "npu" and self.backend == "ascend_eager"
+ else None
+ )
self.enable_prefix_caching = enable_prefix_caching
self.incremental_allocation = incremental_allocation
# Full allocation already reserves each request's maximum KV capacity.
@@ -397,6 +455,7 @@ def allocate(self, request_id: str, max_tokens: int, *, initial_tokens=None, pre
for _ in range(self.storage_depths)
],
)
+ self._allocation_epoch += 1
self.prefix_queries += 1
self.prefix_hits += length
return True
@@ -416,6 +475,7 @@ def ensure_capacity(self, request_id, tokens):
t + tuple(blocks[d * extra : (d + 1) * extra])
for d, t in enumerate(allocation.block_tables)
)
+ self._allocation_epoch += 1
return True
def truncate_suffix(self, request_id: str, frontier: int) -> None:
@@ -445,6 +505,7 @@ def free(self, request_id: str) -> None:
return
allocation = self._allocations.pop(request_id, None)
if allocation is not None:
+ self._allocation_epoch += 1
self._drop_refs(b for table in allocation.block_tables for b in reversed(table))
def pin_transfer(self, request_id: str, transfer_id: str):
@@ -730,12 +791,26 @@ def _stage(self, values, dtype):
host = torch.tensor(values, dtype=dtype, pin_memory=self.device.type == "cuda")
return host.to(self.device, non_blocking=True)
- def _require_live_batch(self, batch: KVMetadata | DeviceKVMetadata) -> None:
+ def _require_live_batch(self, batch: KVMetadata | DeviceKVMetadata):
if isinstance(batch, KVMetadata) and batch.owner is not self:
raise ValueError("prepared KV batch belongs to a different cache manager")
+ plan = None
+ reusable = self._prepared_coverage is not None and isinstance(batch, KVMetadata)
+ epoch = self._allocation_epoch
+ if reusable:
+ plan = self._prepared_coverage.get(batch)
+ if plan is not None and plan.allocation_epoch == epoch:
+ return plan
for request_id, allocation in batch.allocations:
if self._allocations.get(request_id) is not allocation:
raise RuntimeError(f"stale prepared KV batch for request {request_id!r}")
+ if reusable:
+ sequences = (
+ _sequence_coverage(batch.rows, self._plane) if plan is None else plan.sequences
+ )
+ plan = _PreparedCoverage(epoch, sequences)
+ self._prepared_coverage[batch] = plan
+ return plan
def _validate_tensor(self, tensor, batch_size, name, *, query=False):
if tensor.ndim != 3 or tensor.shape[0] != batch_size or tensor.shape[-1] != self.head_dim:
@@ -771,10 +846,14 @@ def write(
@torch.no_grad()
def _write_prepared(
- self, layer: int, batch: KVMetadata | DeviceKVMetadata, k: torch.Tensor, v: torch.Tensor
+ self,
+ layer: int,
+ batch: KVMetadata | DeviceKVMetadata,
+ k: torch.Tensor,
+ v: torch.Tensor,
) -> None:
self._validate_layer(layer)
- self._require_live_batch(batch)
+ coverage = self._require_live_batch(batch)
if isinstance(batch, DeviceKVMetadata):
self._validate_tensor(k, len(batch.position_ids), "k")
self._validate_tensor(v, len(batch.position_ids), "v")
@@ -794,12 +873,20 @@ def _write_prepared(
raise ValueError("a read-only prepared KV batch cannot be written")
self._validate_tensor(k, len(batch.position_ids), "k")
self._validate_tensor(v, len(batch.position_ids), "v")
- if not batch.rows:
- return
rows = len(batch.rows)
+ if not rows:
+ return
self._store_layer(layer, batch.write_blocks, batch.write_offsets, k[:rows], v[:rows])
- for allocation, depth, position in batch.rows:
- allocation.written[self._plane(depth)][layer].add(position)
+ if coverage is None:
+ for allocation, depth, position in batch.rows:
+ allocation.written[self._plane(depth)][layer].add(position)
+ else:
+ # Advance only this layer, only after BOTH device copies enqueue.
+ # Keep holes and partial-layer writes observable to later attention.
+ for sequence in coverage.sequences:
+ written = sequence.allocation.written[sequence.plane][layer]
+ for start, end in sequence.ranges:
+ written.add_range(start, end)
def _store_layer(self, layer, blocks, offsets, k, v, *, mask=None, active=None) -> None:
"""Write one layer's K/V rows, quantizing them for an FP8 pool.
@@ -889,7 +976,7 @@ def _attend_prepared(
window: int | None = None,
) -> torch.Tensor:
self._validate_layer(layer)
- self._require_live_batch(batch)
+ coverage = self._require_live_batch(batch)
self._validate_tensor(q, len(batch.position_ids), "q", query=True)
if window is not None:
if batch.cu_seqlens_q is None:
@@ -913,27 +1000,21 @@ def _attend_prepared(
)
if not batch.rows:
return torch.empty_like(q)
- for allocation, depth, position in batch.rows:
- self._require_prefix(allocation, layer, depth, position + 1)
- if batch.cu_seqlens_q is not None:
- return self.attention.prefill(
- q,
- self.key_cache[:, layer],
- self.value_cache[:, layer],
- batch.block_tables,
- batch.context_lengths,
- batch.cu_seqlens_q,
- batch.max_seqlen_q,
- **self._layer_scales(layer),
- **({} if window is None else {"window": window}),
- )
- return self.attention(
+ if coverage is None:
+ for allocation, depth, position in batch.rows:
+ self._require_prefix(allocation, layer, depth, position + 1)
+ else:
+ for sequence in coverage.sequences:
+ self._require_prefix(
+ sequence.allocation, layer, sequence.plane, sequence.ranges[-1][1]
+ )
+ return self.attention.attend_metadata(
q,
self.key_cache[:, layer],
self.value_cache[:, layer],
- batch.block_tables,
- batch.context_lengths,
+ batch,
**self._layer_scales(layer),
+ **({} if window is None else {"window": window}),
)
@torch.no_grad()
diff --git a/vllm_rlt/core/memory.py b/vllm_rlt/core/memory.py
index 9b87ca2c..1c52c16c 100644
--- a/vllm_rlt/core/memory.py
+++ b/vllm_rlt/core/memory.py
@@ -154,6 +154,8 @@ def plan_cache(
if cache.kv_cache_memory_bytes is not None:
blocks = budget_blocks(cache.kv_cache_memory_bytes, per_block)
return blocks, {"source": "bytes", "bytes_per_block": per_block}
+ if device.type == "npu":
+ raise ValueError("NPU requires explicit num_blocks or kv_cache_memory_bytes")
if device.type != "cuda":
return 256, {"source": "cpu_default", "bytes_per_block": per_block}
torch.cuda.synchronize(device)
diff --git a/vllm_rlt/device.py b/vllm_rlt/device.py
new file mode 100644
index 00000000..d33b47a8
--- /dev/null
+++ b/vllm_rlt/device.py
@@ -0,0 +1,24 @@
+"""Register optional device types only when explicitly requested."""
+
+import importlib
+
+import torch
+
+
+def resolve_device(device) -> torch.device:
+ if isinstance(device, str) and device.split(":", 1)[0] == "npu":
+ # torch.device cannot parse private-use names until torch-npu registers
+ # them. Do this in each spawned worker, without importing it on CPU/CUDA.
+ importlib.import_module("torch_npu")
+ return torch.device(device)
+
+
+def stream_module(device):
+ """Resolve stream/event operations from the owning tensor device."""
+ device = resolve_device(device)
+ if device.type == "cuda":
+ return torch.cuda
+ if device.type == "npu":
+ importlib.import_module("torch_npu")
+ return torch.npu
+ return None
diff --git a/vllm_rlt/engine/packed_prefill.py b/vllm_rlt/engine/packed_prefill.py
index 82eec246..dfb5a486 100644
--- a/vllm_rlt/engine/packed_prefill.py
+++ b/vllm_rlt/engine/packed_prefill.py
@@ -34,8 +34,9 @@ def packed_prefill_eligible(
"with ordinary prefill, without speculation or preemption"
)
return False
- if device_type != "cuda" or layout != "last_exited":
+ if device_type not in ("cuda", "npu") or layout != "last_exited":
if requested is True:
- raise ValueError("packed_prefill requires CUDA packed prefill with last_exited KV")
+ raise ValueError("packed_prefill requires CUDA or Ascend with last_exited KV")
return False
- return requested is True or not staged_prefill
+ # Ascend prepares both ordinary and staged packed metadata on the host.
+ return requested is True or device_type == "npu" or not staged_prefill
diff --git a/vllm_rlt/engine/stage_engine.py b/vllm_rlt/engine/stage_engine.py
index ec40c771..3e3866af 100644
--- a/vllm_rlt/engine/stage_engine.py
+++ b/vllm_rlt/engine/stage_engine.py
@@ -2,6 +2,8 @@
from dataclasses import dataclass, replace
from typing import TYPE_CHECKING, cast
+import torch
+
from vllm_rlt.attention import BackendCapabilities, create_backend
from vllm_rlt.config import (
CacheConfig,
@@ -73,6 +75,27 @@ def __init__(
self.execution_config = resolve_cuda_graph_rows(
execution_config or ExecutionConfig(), scheduler_config, speculative_config
)
+ if parameter.device.type == "npu":
+ if config.to_dict()["model_type"] != "ouro" or config.total_ut_steps != 4:
+ raise ValueError("NPU currently supports Ouro with four loops")
+ if parameter.dtype != torch.bfloat16:
+ raise ValueError("NPU Ouro requires BF16 weights")
+ if cache_config.kv_cache_dtype != "auto":
+ raise ValueError("NPU Ouro requires BF16 KV cache")
+ if (
+ self.execution_config.cuda_graphs
+ or (self.execution_config.async_scheduling and self.execution_config.static_buffers)
+ or speculative_config is not None
+ or self.stage_config.role != "prefill_decode"
+ or scheduler_config.enable_preemption
+ or cache_config.enable_prefix_caching
+ or cache_config.layout != "last_exited"
+ or self.exit_config.mode not in ("ouro", "ouro_delayed")
+ ):
+ raise ValueError(
+ "NPU Ouro requires local execution, ouro/ouro_delayed exit mode and "
+ "last_exited KV without speculation, graphs, preemption or prefix caching"
+ )
staged_prefill = requires_staged_prefill(
scheduler_config, produces_tokens=self.stage_config.produces_tokens
)
@@ -211,7 +234,8 @@ def __init__(
packed_sliding_window=require_packed and sliding_window,
cuda_graphs=self.execution_config.cuda_graphs,
async_scheduling=(
- parameter.device.type == "cuda" and self.execution_config.async_scheduling
+ parameter.device.type in ("cuda", "npu")
+ and self.execution_config.async_scheduling
),
device_context_lengths=self.async_speculative,
fp8_kv=cache_config.kv_cache_dtype == "fp8_e4m3",
@@ -277,7 +301,12 @@ def __init__(
# speculative paths must advance the same per-request slot, and only the
# engine knows when a request terminates.
self.rng = RngRegistry()
- self.model_runner = ModelRunner(
+ runner_type = ModelRunner
+ if parameter.device.type == "npu":
+ from vllm_rlt.worker.ascend_model_runner import AscendModelRunner
+
+ runner_type = AscendModelRunner
+ self.model_runner = runner_type(
model,
self.cache_manager,
exit_config=self.exit_config,
@@ -389,6 +418,10 @@ def _add_request(self, request_id, prompt_token_ids, sampling_params, *, trace_i
if any(type(t) is not int or not 0 <= t < config.vocab_size for t in prompt_token_ids):
raise ValueError("prompt token IDs must be integers within the model vocabulary")
max_loops = params.max_loops or config.total_ut_steps
+ if next(self.model.parameters()).device.type == "npu" and (
+ max_loops != 4 or params.exit_threshold != 1.0 or params.temperature != 0
+ ):
+ raise ValueError("NPU Ouro requests require four loops, exit_threshold=1 and greedy")
if max_loops > config.total_ut_steps or params.min_loops > max_loops:
raise ValueError("requested loop bounds exceed the model's supported depth")
if getattr(self.model, "fixed_loop_count", False) and (
@@ -939,8 +972,7 @@ def _step_async(self):
request_events=self.model_runner.events,
signals=signals,
)
- for row in update.next_prelude:
- batch.requests[row].request.input_token_tensor = ticket.device_values[row]
+ self.model_runner.bind_sampled_tokens(batch, ticket.device_values, update.next_prelude)
for row in update.retain_signal:
self._pending_exit_signals[batch.requests[row].request.request_id] = SignalHandle(
ticket, row
diff --git a/vllm_rlt/engine/worker.py b/vllm_rlt/engine/worker.py
index 9a2b2805..80e98985 100644
--- a/vllm_rlt/engine/worker.py
+++ b/vllm_rlt/engine/worker.py
@@ -13,6 +13,7 @@
import torch
+from vllm_rlt.device import resolve_device
from vllm_rlt.engine.connector import TransferConnector
from vllm_rlt.engine.stage_config import StageConfig
from vllm_rlt.engine.stage_engine import StageEngine
@@ -33,12 +34,16 @@ def __init__(
):
if type(max_control_messages) is not int or max_control_messages < 1:
raise ValueError("max_control_messages must be a positive integer")
- device = torch.device(f"cuda:{device}" if isinstance(device, int) else device)
+ device = resolve_device(f"cuda:{device}" if isinstance(device, int) else device)
torch.set_num_threads(1)
if device.type == "cuda":
if device.index is None:
device = torch.device("cuda", torch.cuda.current_device())
torch.cuda.set_device(device)
+ elif device.type == "npu":
+ if device.index is None:
+ device = torch.device("npu", torch.npu.current_device())
+ torch.npu.set_device(device)
torch.manual_seed(options["seed"])
self.channel = channel
self.max_control_messages = max_control_messages
diff --git a/vllm_rlt/entrypoints/cli.py b/vllm_rlt/entrypoints/cli.py
index a9b8025c..f05313c1 100644
--- a/vllm_rlt/entrypoints/cli.py
+++ b/vllm_rlt/entrypoints/cli.py
@@ -7,6 +7,7 @@
import torch
from vllm_rlt import LLM, SamplingParams, SchedulerConfig
+from vllm_rlt.attention import validate_backend_name
from vllm_rlt.entrypoints.runtime_args import (
add_runtime_args,
profile_config_from_args,
@@ -67,6 +68,7 @@ def main():
"--attention-backend",
type=str.lower,
choices=[
+ "ascend_eager",
"auto",
"torch",
"triton",
@@ -100,6 +102,10 @@ def main():
parser.error("--toy uses built-in token ID prompts; omit --prompt")
if not args.toy and not args.prompt:
parser.error("provide --prompt or use --toy for a CPU smoke test")
+ try:
+ validate_backend_name(args.attention_backend, device_type=args.device.split(":", 1)[0])
+ except ValueError as exc:
+ parser.error(str(exc))
profiling = profile_config_from_args(args)
if profiling is not None:
profiling.resolve_activities(args.device)
diff --git a/vllm_rlt/entrypoints/serve.py b/vllm_rlt/entrypoints/serve.py
index dc5eef6c..c7fc3c84 100644
--- a/vllm_rlt/entrypoints/serve.py
+++ b/vllm_rlt/entrypoints/serve.py
@@ -7,6 +7,7 @@
import torch
+from vllm_rlt.attention import validate_backend_name
from vllm_rlt.config import SchedulerConfig
from vllm_rlt.engine.llm_engine import StageClient
from vllm_rlt.entrypoints.runtime_args import (
@@ -82,12 +83,13 @@ def main():
parser.add_argument("--tokenizer")
parser.add_argument("--tokenizer-revision")
parser.add_argument("--served-model-name")
- parser.add_argument("--device", choices=["cpu", "cuda"], default="cuda")
+ parser.add_argument("--device", choices=["cpu", "cuda", "npu"], default="cuda")
parser.add_argument("--dtype", choices=["bfloat16", "float32"], default="bfloat16")
parser.add_argument(
"--attention-backend",
type=str.lower,
choices=[
+ "ascend_eager",
"auto",
"torch",
"triton",
@@ -117,6 +119,10 @@ def main():
limits = ServingLimits(**{name: getattr(args, name) for name in defaults})
if args.device == "cpu" and args.attention_backend not in ("auto", "torch"):
parser.error("CPU execution requires --attention-backend torch or auto")
+ try:
+ validate_backend_name(args.attention_backend, device_type=args.device)
+ except ValueError as exc:
+ parser.error(str(exc))
torch.set_num_threads(args.cpu_threads)
logging.basicConfig(level=logging.INFO)
app = create_app(
diff --git a/vllm_rlt/kernels/ascend_attention.py b/vllm_rlt/kernels/ascend_attention.py
new file mode 100644
index 00000000..12940520
--- /dev/null
+++ b/vllm_rlt/kernels/ascend_attention.py
@@ -0,0 +1,332 @@
+# SPDX-License-Identifier: Apache-2.0
+"""Use Torch SDPA on Ascend with the runtime's paged KV cache."""
+
+import weakref
+from collections import OrderedDict
+
+import torch
+from torch.nn import functional as F
+
+
+def _attend(query, key, value, mask=None, *, is_causal=False):
+ # Match HF's BNSD inputs. A bool mask lets Torch-NPU dispatch fused SDPA
+ # for cached chunks and variable-length decode on supported hardware.
+ groups = query.shape[1] // key.shape[1]
+ if groups != 1:
+ key = key.repeat_interleave(groups, dim=1)
+ value = value.repeat_interleave(groups, dim=1)
+ return F.scaled_dot_product_attention(
+ query.contiguous(),
+ key.contiguous(),
+ value.contiguous(),
+ attn_mask=mask,
+ is_causal=is_causal,
+ )
+
+
+def _flat_kv_rows(keys, values):
+ blocks, block_size, heads, head_dim = keys.shape
+ block_stride, *row_strides = keys.stride()
+ row_width = heads * head_dim
+ if (
+ not blocks
+ or not row_width
+ or values.shape != keys.shape
+ or values.stride() != keys.stride()
+ or tuple(row_strides) != (row_width, head_dim, 1)
+ or block_stride < block_size * row_width
+ or block_stride % row_width
+ ):
+ return None
+ rows_per_block = block_stride // row_width
+ # The view starts at this layer's storage offset. Include the gaps between
+ # its pages, then gather only rows belonging to the selected layer.
+ shape = ((blocks - 1) * rows_per_block + block_size, heads, head_dim)
+ return (
+ keys.as_strided(shape, row_strides),
+ values.as_strided(shape, row_strides),
+ rows_per_block,
+ )
+
+
+def _prefill_geometry(rows):
+ """Detect a regular grid of equal-length, complete prompt traversals."""
+ if not rows or rows[0][2] != 0:
+ return None
+ width = rows[-1][2] + 1
+ size, remainder = divmod(len(rows), width)
+ if size < 1 or width < 2 or remainder:
+ return None
+ groups = [(id(alloc), depth) for alloc, depth, _ in rows[::width]]
+ if len(set(groups)) != size or any(
+ ((id(alloc), depth), pos) != (groups[i // width], i % width)
+ for i, (alloc, depth, pos) in enumerate(rows)
+ ):
+ return None
+ return size, width
+
+
+class AscendAttention:
+ """Batch compatible queries; borrow only validated host metadata."""
+
+ def __init__(self):
+ self._metadata = None
+ self._groups = None
+ self._geometry = None
+ self._write_slots = None
+ self._ragged_plan = None
+ self._signature = None
+ self._direct_decode = False
+ self._decode_plan = None
+ # Read-only layouts may be shared across recurrent depths. Addresses
+ # stay local to each descriptor, and tensors never cross stream keys.
+ self._layouts = OrderedDict()
+ self._layout_bytes = 0
+ self._layout_limit = 64 * 1024 * 1024
+ self.layout_reuses = 0
+ self.batched_prefill_calls = 0
+ self.ragged_calls = 0
+ self.fallback_calls = 0
+ self.continuation_calls = 0
+
+ @staticmethod
+ def _stream_key(device):
+ if device.type == "npu":
+ # Raw npu_stream resolves an ACL handle and drains the task queue.
+ # Cache identity only needs the logical, device-scoped stream ID.
+ return torch.npu.current_stream(device).stream_id
+ if device.type == "cuda":
+ return torch.cuda.current_stream(device).cuda_stream
+ return None
+
+ def _layout(self, key, build):
+ cached = self._layouts.get(key)
+ if cached is not None:
+ self._layouts.move_to_end(key)
+ self.layout_reuses += 1
+ return cached[0]
+ tensors = build()
+ size = sum(t.numel() * t.element_size() for t in tensors if t is not None)
+ if size <= self._layout_limit:
+ while self._layouts and self._layout_bytes + size > self._layout_limit:
+ _, (_, old_size) = self._layouts.popitem(last=False)
+ self._layout_bytes -= old_size
+ # Bound Python entries too: many tiny short-context plans are common.
+ if len(self._layouts) == 8:
+ _, (_, old_size) = self._layouts.popitem(last=False)
+ self._layout_bytes -= old_size
+ self._layouts[key] = (tensors, size)
+ self._layout_bytes += size
+ return tensors
+
+ def _prepare_decode(self, q, keys, batch, flat_rows, stream):
+ lengths = tuple(position + 1 for _, _, position in batch.rows)
+ maximum = max(lengths)
+ block_size = keys.shape[-3]
+
+ def build():
+ positions = torch.arange(maximum, device=q.device)
+ valid = None
+ if min(lengths) != maximum:
+ valid = positions[None, :] < torch.tensor(lengths, device=q.device)[:, None]
+ return positions // block_size, positions % block_size, valid
+
+ pages, offsets, valid = self._layout(
+ ("decode", q.device, stream, block_size, lengths), build
+ )
+ rows = len(lengths)
+ tables = batch.block_tables
+ if len(tables) != rows:
+ tables = tables[:rows]
+ blocks = tables[:, pages].to(torch.int64)
+ flat_keys, _, rows_per_block = flat_rows
+ slots = (blocks * rows_per_block + offsets).clamp(0, flat_keys.shape[0] - 1)
+ if valid is not None:
+ # Avoid reading unwritten bytes in masked columns. Position zero is
+ # initialized whenever the cache's prefix check admits this row.
+ slots = torch.where(valid, slots, slots[:, :1])
+ mask = valid[:, None, None, :] if valid is not None else None
+ self._decode_plan = (slots.reshape(-1), mask, rows, maximum)
+
+ def _decode(self, q, keys, values, flat_rows):
+ slots, mask, rows, maximum = self._decode_plan
+ flat_keys, flat_values, _ = flat_rows
+ kv_heads, head_dim = keys.shape[-2:]
+ key = flat_keys.index_select(0, slots).reshape(rows, maximum, kv_heads, head_dim)
+ value = flat_values.index_select(0, slots).reshape(rows, maximum, kv_heads, head_dim)
+ # Real rows form an identity prefix. Avoid gathering Q and scattering
+ # the result on every layer.
+ query = q if len(q) == rows else q[:rows]
+ query = query.reshape(rows, q.shape[1], 1, head_dim)
+ output = _attend(query, key.transpose(1, 2), value.transpose(1, 2), mask).squeeze(2)
+ if len(q) == rows:
+ return output
+ result = torch.zeros_like(q)
+ result[:rows].copy_(output)
+ return result
+
+ def _plan(self, sequence_keys, lengths, device, *, packed=False):
+ grouped = {}
+ for row, (key, length) in enumerate(zip(sequence_keys, lengths, strict=True)):
+ if length:
+ grouped.setdefault(key, []).append(row)
+ planned = []
+ for sequence, rows in enumerate(grouped.values()):
+ maximum = max(lengths[row] for row in rows)
+ indices = torch.tensor(rows, dtype=torch.long, device=device)
+ positions = torch.arange(maximum, device=device)
+ group_lengths = [lengths[row] for row in rows]
+ initial = group_lengths == list(range(1, maximum + 1))
+ row_lengths = torch.tensor(group_lengths, device=device)
+ mask = (
+ positions[None, :] < row_lengths[:, None] if not initial and len(rows) > 1 else None
+ )
+ # Packed table rows follow sequence order. Query indices still
+ # address the flattened Q and output tensors.
+ table_row = sequence if packed else rows[0]
+ planned.append((table_row, indices, positions, mask, initial, row_lengths))
+ return planned
+
+ def _forward(self, q, keys, values, tables, groups, *, flat_rows=None, result=None):
+ if result is None:
+ result = torch.zeros_like(q)
+ block_size = keys.shape[-3]
+ for table_row, indices, positions, mask, initial, _lengths in groups:
+ blocks = tables[table_row, positions // block_size]
+ offsets = positions % block_size
+ if flat_rows is None:
+ k, v = keys[blocks, offsets], values[blocks, offsets]
+ else:
+ flat_keys, flat_values, rows_per_block = flat_rows
+ slots = blocks.to(torch.int64) * rows_per_block + offsets
+ slots = slots.clamp(0, flat_keys.shape[0] - 1)
+ k, v = flat_keys.index_select(0, slots), flat_values.index_select(0, slots)
+ query = q.index_select(0, indices).transpose(0, 1).unsqueeze(0)
+ key = k.transpose(0, 1).unsqueeze(0)
+ value = v.transpose(0, 1).unsqueeze(0)
+ output = (
+ _attend(query, key, value, mask, is_causal=initial and indices.numel() > 1)
+ .squeeze(0)
+ .transpose(0, 1)
+ )
+ result.index_copy_(0, indices, output)
+ return result
+
+ def _batched_prefill(self, q, keys, values, batch, flat_rows, geometry):
+ size, width = geometry
+ flat_keys, flat_values, rows_per_block = flat_rows
+ kv_heads, head_dim = keys.shape[-2:]
+ if self._write_slots is None:
+ self._write_slots = (
+ batch.write_blocks.to(torch.int64) * rows_per_block
+ + batch.write_offsets.to(torch.int64)
+ ).clamp(0, flat_keys.shape[0] - 1)
+ slots = self._write_slots
+ key = (
+ flat_keys.index_select(0, slots)
+ .reshape(size, width, kv_heads, head_dim)
+ .transpose(1, 2)
+ )
+ value = (
+ flat_values.index_select(0, slots)
+ .reshape(size, width, kv_heads, head_dim)
+ .transpose(1, 2)
+ )
+ query = q.reshape(size, width, q.shape[1], head_dim).transpose(1, 2)
+ return _attend(query, key, value, is_causal=True).transpose(1, 2).reshape_as(q)
+
+ def _ragged_decode(self, q, keys, values, tables, groups, flat_rows, result):
+ """One attention call for independent single-row decode groups."""
+ flat_keys, flat_values, rows_per_block = flat_rows
+ block_size, kv_heads, head_dim = keys.shape[-3:]
+ if self._ragged_plan is None:
+ indices = torch.cat([group[1] for group in groups])
+ lengths = torch.cat([group[5] for group in groups])
+ maximum = max(group[2].numel() for group in groups)
+ positions = torch.arange(maximum, device=q.device)
+ pages = positions // block_size
+ offsets = positions % block_size
+ table_rows = torch.tensor([group[0] for group in groups], device=q.device)
+ row_tables = tables.index_select(0, table_rows)
+ blocks = row_tables[:, pages].to(torch.int64)
+ slots = (blocks * rows_per_block + offsets.to(torch.int64)).clamp(
+ 0, flat_keys.shape[0] - 1
+ )
+ rows = indices.numel()
+ valid = positions.unsqueeze(0) < lengths.unsqueeze(1)
+ slots = torch.where(valid, slots, slots[:, :1])
+ self._ragged_plan = (indices, maximum, valid[:, None, None, :], slots, rows)
+ indices, maximum, mask, slots, rows = self._ragged_plan
+ key = flat_keys.index_select(0, slots.reshape(-1)).reshape(
+ rows, maximum, kv_heads, head_dim
+ )
+ value = flat_values.index_select(0, slots.reshape(-1)).reshape(
+ rows, maximum, kv_heads, head_dim
+ )
+ query = q.index_select(0, indices).reshape(rows, q.shape[1], 1, head_dim)
+ output = _attend(query, key.transpose(1, 2), value.transpose(1, 2), mask).squeeze(2)
+ result.index_copy_(0, indices, output)
+ return result
+
+ def __call__(self, q, keys, values, tables, lengths):
+ groups = self._plan([tuple(row) for row in tables.tolist()], lengths.tolist(), q.device)
+ return self._forward(q, keys, values, tables, groups)
+
+ def attend_metadata(self, q, keys, values, batch):
+ packed = batch.cu_seqlens_q is not None
+ flat_rows = _flat_kv_rows(keys, values)
+ stream = self._stream_key(q.device)
+ signature = (q.device, q.dtype, q.shape, keys.shape, keys.stride(), values.stride(), stream)
+ if self._metadata is None or self._metadata() is not batch or self._signature != signature:
+ # Failed preparation must not leave the previous descriptor key
+ # pointing at partially replaced groups/addresses.
+ self._metadata = None
+ self._signature = None
+ self._geometry = _prefill_geometry(batch.rows) if len(q) == len(batch.rows) else None
+ self._write_slots = None
+ self._ragged_plan = None
+ self._decode_plan = None
+ self._direct_decode = (
+ flat_rows is not None
+ and len(batch.rows) >= 2
+ and all(position > 0 for _, _, position in batch.rows)
+ and len({(id(a), depth) for a, depth, _ in batch.rows}) == len(batch.rows)
+ )
+ self._groups = None
+ if self._direct_decode:
+ self._prepare_decode(q, keys, batch, flat_rows, stream)
+ elif flat_rows is None or self._geometry is None:
+ self._groups = self._plan(
+ [(id(allocation), depth) for allocation, depth, _ in batch.rows],
+ [position + 1 for _, _, position in batch.rows],
+ q.device,
+ packed=packed,
+ )
+ self._metadata = weakref.ref(batch)
+ self._signature = signature
+ if self._direct_decode:
+ self.ragged_calls += 1
+ return self._decode(q, keys, values, flat_rows)
+ if flat_rows is None:
+ self.fallback_calls += 1
+ return self._forward(q, keys, values, batch.block_tables, self._groups)
+ if self._geometry is not None:
+ self.batched_prefill_calls += 1
+ return self._batched_prefill(q, keys, values, batch, flat_rows, self._geometry)
+ ragged, remaining = [], []
+ for group in self._groups:
+ if not group[4] and group[1].numel() == 1:
+ ragged.append(group)
+ else:
+ remaining.append(group)
+ # A single row gains nothing from batching but pays extra plan construction.
+ if len(ragged) < 2:
+ remaining.extend(ragged)
+ ragged = []
+ result = self._forward(q, keys, values, batch.block_tables, remaining, flat_rows=flat_rows)
+ if ragged:
+ self.ragged_calls += 1
+ result = self._ragged_decode(
+ q, keys, values, batch.block_tables, ragged, flat_rows, result
+ )
+ return result
diff --git a/vllm_rlt/layers/rotary_embedding.py b/vllm_rlt/layers/rotary_embedding.py
index 297e98d6..84b52096 100644
--- a/vllm_rlt/layers/rotary_embedding.py
+++ b/vllm_rlt/layers/rotary_embedding.py
@@ -37,7 +37,10 @@ def __init__(self, head_dim: int, rope_theta: float = 10000.0) -> None:
self.register_buffer("inv_freq", self.frequencies(), persistent=False)
def frequencies(self, device: torch.device | str | None = None) -> torch.Tensor:
- return 1.0 / (
+ target = device
+ if device is not None and torch.device(device).type == "npu":
+ device = "cpu" # HF creates NPU rotary constants on the host.
+ frequencies = 1.0 / (
self.rope_theta
** (
torch.arange(0, self.head_dim, 2, dtype=torch.float32, device=device)
@@ -45,6 +48,12 @@ def frequencies(self, device: torch.device | str | None = None) -> torch.Tensor:
)
)
+ return (
+ frequencies.to(target)
+ if device == "cpu" and target is not None and torch.device(target).type == "npu"
+ else frequencies
+ )
+
def _apply(self, fn, recurse=True):
super()._apply(fn, recurse=recurse)
# Module.to(dtype) otherwise rounds these constants before forward's
diff --git a/vllm_rlt/models/ouro.py b/vllm_rlt/models/ouro.py
index 31b37f9c..5041fdf3 100644
--- a/vllm_rlt/models/ouro.py
+++ b/vllm_rlt/models/ouro.py
@@ -19,6 +19,7 @@
import torch
from torch import nn
+from vllm_rlt.device import resolve_device
from vllm_rlt.kernels import silu_and_mul
from vllm_rlt.layers import (
RMSNorm,
@@ -309,6 +310,7 @@ def from_pretrained(
"""
from safetensors import safe_open
+ device = resolve_device(device)
if dtype not in (torch.float32, torch.float16, torch.bfloat16):
raise ValueError("dtype must be float32, float16, or bfloat16")
folder = Path(path_or_repo).expanduser()
diff --git a/vllm_rlt/profiling.py b/vllm_rlt/profiling.py
index 41245be8..757bbafd 100644
--- a/vllm_rlt/profiling.py
+++ b/vllm_rlt/profiling.py
@@ -51,6 +51,8 @@ def __post_init__(self):
object.__setattr__(self, "activities", values)
def resolve_activities(self, device):
+ if str(device).startswith("npu"):
+ raise ValueError("NPU device profiling requires torch_npu.profiler")
names = self.activities or (("cpu", "cuda") if str(device).startswith("cuda") else ("cpu",))
values = [getattr(torch.profiler.ProfilerActivity, name.upper()) for name in names]
if not set(values) <= torch.profiler.supported_activities():
diff --git a/vllm_rlt/worker/ascend_model_runner.py b/vllm_rlt/worker/ascend_model_runner.py
new file mode 100644
index 00000000..efd1c74d
--- /dev/null
+++ b/vllm_rlt/worker/ascend_model_runner.py
@@ -0,0 +1,109 @@
+"""Ascend execution with device-resident state, batched sampling and KV finalization."""
+
+import torch
+
+from vllm_rlt.worker.model_runner import ModelRunner
+from vllm_rlt.worker.sampler import Sampler
+
+
+class AscendSampler(Sampler):
+ """Keep batched greedy dispatch out of the CUDA/ROCm sampling path."""
+
+ def sample_batch(self, logits, params, generators):
+ if getattr(self.sample, "__func__", None) is Sampler.sample and all(
+ item.temperature == 0 for item in params
+ ):
+ return logits[: len(params)].argmax(dim=-1)
+ return torch.stack(
+ [
+ self.sample(row, item, generator)
+ for row, item, generator in zip(logits, params, generators)
+ ]
+ )
+
+
+class AscendModelRunner(ModelRunner):
+ def __init__(self, *args, **kwargs):
+ super().__init__(*args, **kwargs)
+ if self.device.type != "npu":
+ raise ValueError("AscendModelRunner requires an NPU")
+ self.sampler = AscendSampler()
+
+ def _prepare_tokens(self, prepared, size):
+ """Materialize snapshotted token sources on the current compute stream."""
+ sources = prepared.token_sources
+ workspace = prepared.workspace
+ stream = (
+ self.device_ops.current_stream(self.device) if self.device_ops is not None else None
+ )
+ if prepared.routing is not None:
+ tokens = prepared.routing.gather(tokens=True)
+ elif all(isinstance(source, torch.Tensor) for source in sources):
+ tokens = torch.stack(sources)
+ elif all(isinstance(source, tuple) and len(source) == 1 for source in sources):
+ ids = [source[0] for source in sources]
+ tokens = (
+ workspace.tokens(ids, size)
+ if workspace is not None
+ else torch.tensor(ids, device=self.device, dtype=torch.long)
+ )
+ else:
+ tokens = torch.cat(
+ [
+ source.reshape(-1)
+ if isinstance(source, torch.Tensor)
+ else torch.tensor(source, device=self.device, dtype=torch.long)
+ for source in sources
+ ]
+ )
+ if prepared.routing is None and size > len(tokens):
+ tokens = torch.cat((tokens, tokens.new_zeros(size - len(tokens))))
+ for span, source in zip(prepared.batch.spans, sources):
+ if isinstance(source, torch.Tensor) and source.device.type in ("cuda", "npu"):
+ source.record_stream(stream)
+ if span.request.input_token_tensor is source:
+ span.request.input_token_tensor = None
+ return tokens
+
+ def _finalize_npu_batch(self, requests):
+ cache = self.cache_manager
+ copies = []
+ for request in requests:
+ depth = request.loops_done - 1
+ if cache.layout == "shared" or depth + 1 == cache.max_loops:
+ # This still verifies every required layer. With no destination
+ # there is no device work: the existing core event is sufficient.
+ cache.finalize_token(request.request_id, request.position, depth)
+ else:
+ copies.append(request)
+ if not copies:
+ return
+ stream = self.boundary_stream
+ with self.device_ops.stream(stream):
+ try:
+ dependencies = {
+ self.events[request.request_id]
+ for request in copies
+ if request.request_id in self.events
+ }
+ for event in dependencies:
+ stream.wait_event(event)
+ for request in copies:
+ cache.finalize_token(
+ request.request_id, request.position, request.loops_done - 1
+ )
+ done = self.device_ops.Event()
+ done.record(stream)
+ except Exception:
+ # A later request can fail after earlier copies were enqueued.
+ # Finish those readers before cleanup may recycle their pages.
+ stream.synchronize()
+ raise
+ for request in copies:
+ self.events[request.request_id] = done
+
+ def finalize_many(self, requests):
+ if self.execution_config.async_scheduling and self.cache_manager.backend == "ascend_eager":
+ self._finalize_npu_batch(requests)
+ else:
+ super().finalize_many(requests)
diff --git a/vllm_rlt/worker/model_runner.py b/vllm_rlt/worker/model_runner.py
index 70c251ae..d7312d92 100644
--- a/vllm_rlt/worker/model_runner.py
+++ b/vllm_rlt/worker/model_runner.py
@@ -9,6 +9,7 @@
from vllm_rlt.config import ExecutionConfig, ExitConfig, SchedulerConfig
from vllm_rlt.core.scheduler import SchedulerOutput
+from vllm_rlt.device import stream_module
from vllm_rlt.request import ExecutionBatch, Operation, Request
from vllm_rlt.worker.buffers import (
PackedMetadataBuffer,
@@ -167,6 +168,7 @@ def __init__(
self.runtime = ModelRuntime(self.model, cache_manager)
parameter = next(model.parameters())
self.device = parameter.device
+ self.device_ops = stream_module(self.device)
self.sampler = Sampler()
# Sampling RNG state lives with the engine, which shares this registry
# between the CODA runner and the speculative runner.
@@ -218,20 +220,20 @@ def __init__(
self.lookahead_head = head.to(device=self.device, dtype=parameter.dtype).eval()
self.lookahead_head.requires_grad_(False)
self.core_stream = self.boundary_stream = self.copy_stream = self.readback_stream = None
- if self.device.type == "cuda":
- self.core_stream = torch.cuda.Stream(device=self.device)
+ if self.device_ops is not None:
+ self.core_stream = self.device_ops.Stream(device=self.device)
self.boundary_stream = (
- torch.cuda.Stream(device=self.device)
+ self.device_ops.Stream(device=self.device)
if self.execution_config.multi_stream
else self.core_stream
)
self.copy_stream = (
- torch.cuda.Stream(device=self.device)
+ self.device_ops.Stream(device=self.device)
if self.execution_config.multi_stream
else self.core_stream
)
self.readback_stream = (
- torch.cuda.Stream(device=self.device)
+ self.device_ops.Stream(device=self.device)
if self.execution_config.multi_stream
else self.core_stream
)
@@ -241,9 +243,9 @@ def __init__(
self.copy_stream,
self.readback_stream,
):
- stream.wait_stream(torch.cuda.current_stream(self.device))
+ stream.wait_stream(self.device_ops.current_stream(self.device))
self.readback_slots = []
- if self.device.type == "cuda":
+ if self.device_ops is not None:
# cudaHostAlloc during submission can serialize streams. Allocate all
# readback storage before work starts, retaining leases until consumed.
for dtype in (torch.float32, torch.int64):
@@ -344,7 +346,7 @@ def __init__(
# Buffers allocated after stream creation must be visible there too.
for stream in (self.core_stream, self.boundary_stream):
if stream is not None:
- stream.wait_stream(torch.cuda.current_stream(self.device))
+ stream.wait_stream(self.device_ops.current_stream(self.device))
self.async_state = None
if use_async_state:
@@ -364,7 +366,7 @@ def __init__(
self.state_slots = self.async_state.slots
self.free_state_slots = self.async_state.free
for stream in (self.core_stream, self.boundary_stream, self.copy_stream):
- stream.wait_stream(torch.cuda.current_stream(self.device))
+ stream.wait_stream(self.device_ops.current_stream(self.device))
def _size(self, count):
size = count
@@ -612,6 +614,11 @@ def _save_batch(self, requests, hidden, routing):
for request in requests:
request.hidden_state = self.states[self.state_slots[request.request_id]]
+ def bind_sampled_tokens(self, batch, values, rows):
+ """Bind the scheduler-selected sample rows as the next prelude inputs."""
+ for row in rows:
+ batch.requests[row].request.input_token_tensor = values[row]
+
def _gather(self, requests, workspace, size):
if workspace is None:
return torch.stack([r.hidden_state for r in requests])
@@ -720,7 +727,9 @@ def _prepare_shared_loop_inputs(self, batch):
def _uses_packed_loop_inputs(self, batch):
capabilities = self.cache_manager.attention_capabilities
return (
- batch.operation == Operation.PRELUDE
+ # PackedMetadataBuffer expands write addresses with a CUDA kernel.
+ self.device.type == "cuda"
+ and batch.operation == Operation.PRELUDE
and batch.loop_count
and self.cache_manager.layout != "shared"
and capabilities.packed_prefill
@@ -974,10 +983,10 @@ def prepare_on_gpu(self, prepared):
),
)
if prepared.routing is not None:
- with torch.cuda.stream(self.copy_stream):
+ with self.device_ops.stream(self.copy_stream):
kv = prepared.routing.transfer(prepared.kv)
if self.device.type == "cuda":
- torch.cuda.current_stream(self.device).wait_event(prepared.routing.ready_event)
+ self.device_ops.current_stream(self.device).wait_event(prepared.routing.ready_event)
prepared = replace(prepared, kv=kv)
elif prepared.kv is not None and prepared.workspace is not None:
prepared = replace(prepared, kv=prepared.workspace.upload_metadata(prepared.kv))
@@ -1139,14 +1148,21 @@ def _execute_batch(self, batch: ExecutionBatch, prepared):
)
values = selected.float().sigmoid()
elif forward.compute_logits:
- values = torch.stack(
- [
- self.sampler.sample(row, params, generator)
- for row, params, generator in zip(
- result.logits, prepared.sampling_params, prepared.sampling_generators
- )
- ]
- )
+ if self.device.type == "npu":
+ values = self.sampler.sample_batch(
+ result.logits, prepared.sampling_params, prepared.sampling_generators
+ )
+ else:
+ values = torch.stack(
+ [
+ self.sampler.sample(row, params, generator)
+ for row, params, generator in zip(
+ result.logits,
+ prepared.sampling_params,
+ prepared.sampling_generators,
+ )
+ ]
+ )
if prepared.routing is not None:
prepared.routing.scatter(values, tokens=True)
return values
@@ -1168,9 +1184,9 @@ def _readback_slot(self, result):
raise RuntimeError("readback buffers exhausted; retire completed submissions")
def _synchronize_submission(self, stream, prepared):
- if self.device.type != "cuda":
+ if self.device_ops is None:
return
- (stream or torch.cuda.current_stream(self.device)).synchronize()
+ (stream or self.device_ops.current_stream(self.device)).synchronize()
if self.readback_stream is not None and self.readback_stream is not stream:
self.readback_stream.synchronize()
if prepared is not None and prepared.routing is not None:
@@ -1208,14 +1224,14 @@ def submit(self, scheduled_batch: SchedulerOutput | None = None, *, batch: Execu
self.split_prefill_submission_waits += 1
prepared = self.prepare_on_cpu(batch)
source_stream = (
- torch.cuda.current_stream(self.device) if self.device.type == "cuda" else None
+ self.device_ops.current_stream(self.device) if self.device_ops is not None else None
)
if prepared.routing is not None and prepared.routing.imports:
self.copy_stream.wait_stream(source_stream)
# Imported hidden/token tensors may have been produced by core,
# not by the caller's stream. The copy itself must wait for them.
self._wait_for_requests(batch, self.copy_stream)
- with torch.cuda.stream(stream) if stream is not None else nullcontext():
+ with self.device_ops.stream(stream) if stream is not None else nullcontext():
if stream is not None:
if self.async_state is None and source_stream != stream:
stream.wait_stream(source_stream)
@@ -1249,21 +1265,21 @@ def submit(self, scheduled_batch: SchedulerOutput | None = None, *, batch: Execu
result = None
device_values = result
event = compute_done = None
- if self.device.type == "cuda":
- compute_stream = torch.cuda.current_stream(self.device)
- compute_done = torch.cuda.Event()
+ if self.device_ops is not None:
+ compute_stream = self.device_ops.current_stream(self.device)
+ compute_done = self.device_ops.Event()
compute_done.record(compute_stream)
event = compute_done
if result is not None:
slot = self._readback_slot(result)
readback_stream = self.readback_stream or compute_stream
- with torch.cuda.stream(readback_stream):
+ with self.device_ops.stream(readback_stream):
if readback_stream != compute_stream:
readback_stream.wait_event(compute_done)
result.record_stream(readback_stream)
host = slot.storage[: result.numel()].view(result.shape)
host.copy_(result, non_blocking=True)
- event = torch.cuda.Event()
+ event = self.device_ops.Event()
event.record(readback_stream)
slot.event = event
result = host
@@ -1294,7 +1310,7 @@ def submit(self, scheduled_batch: SchedulerOutput | None = None, *, batch: Execu
and prepared.routing is not None
and not routing_finish_attempted
):
- with torch.cuda.stream(stream) if stream is not None else nullcontext():
+ with self.device_ops.stream(stream) if stream is not None else nullcontext():
routing_finish_attempted = True
prepared.routing.record_done()
finally:
@@ -1303,7 +1319,7 @@ def submit(self, scheduled_batch: SchedulerOutput | None = None, *, batch: Execu
raise
finally:
if prepared is not None and prepared.workspace is not None:
- with torch.cuda.stream(stream) if stream is not None else nullcontext():
+ with self.device_ops.stream(stream) if stream is not None else nullcontext():
prepared.workspace.record_done()
def finalize_many(self, requests):
@@ -1383,7 +1399,7 @@ def finalize_many(self, requests):
def finalize(self, request):
# The final core event must precede copies and coda on the boundary stream.
stream = self.boundary_stream
- with torch.cuda.stream(stream) if stream is not None else nullcontext():
+ with self.device_ops.stream(stream) if stream is not None else nullcontext():
event = self.events.get(request.request_id)
if stream is not None and event is not None:
stream.wait_event(event)
@@ -1391,7 +1407,7 @@ def finalize(self, request):
request.request_id, request.position, request.loops_done - 1
)
if stream is not None:
- event = torch.cuda.Event()
+ event = self.device_ops.Event()
event.record(stream)
self.events[request.request_id] = event
@@ -1427,6 +1443,9 @@ def release(self, request_id):
self.rng.drop(request_id)
def synchronize(self):
+ if self.device.type == "npu":
+ torch.npu.synchronize(self.device)
+ return
for stream in (
self.core_stream,
self.boundary_stream,
diff --git a/vllm_rlt/worker/tensors.py b/vllm_rlt/worker/tensors.py
index 503dee5c..241f4ec1 100644
--- a/vllm_rlt/worker/tensors.py
+++ b/vllm_rlt/worker/tensors.py
@@ -90,7 +90,7 @@ def copy_tensor(source, target, *, secondary=None, use_uva=True):
def pinned_tensor(shape, dtype, device):
"""Allocate host staging; CPU execution needs no pinned allocation."""
return torch.empty(
- shape, dtype=dtype, device="cpu", pin_memory=torch.device(device).type == "cuda"
+ shape, dtype=dtype, device="cpu", pin_memory=torch.device(device).type in ("cuda", "npu")
)