diff --git a/README.md b/README.md index 17502f55..b0ffc7bc 100644 --- a/README.md +++ b/README.md @@ -143,7 +143,7 @@ a fixed scale at startup. See the | Checkpoint | Implementation | Documented validation scope | Guide | | --- | --- | --- | --- | -| Ouro-1.4B | Native Ouro | Existing runtime and benchmark coverage under their recorded configurations | [Runtime](docs/runtime.md), [accuracy](docs/accuracy.md) | +| Ouro-1.4B | Native Ouro | Ascend BF16 full prefill with eager Python execution, paged KV and optional fixed-depth async scheduling. Arbitrary chunked prefill remains unresolved. | [Runtime](docs/runtime.md), [Ascend recipe](docs/recipes/ouro_ascend.md#6-fixed-depth-asynchronous-engine-plan) | | Ouro-2.6B | Shared Ouro | Shares the Ouro implementation; not separately qualified, and 1.4B quality/performance results are not inherited | [Scope notes](docs/recipes/ouro_thinking.md#6-evidence-and-limitations) | | Ouro-1.4B-Thinking | Reuses Ouro | Bounded BF16, fixed-four-loop synchronous basic integration | [Shared recipe](docs/recipes/ouro_thinking.md) | | Ouro-2.6B-Thinking | Reuses Ouro | Bounded BF16, fixed-four-loop synchronous basic integration | [Shared recipe](docs/recipes/ouro_thinking.md) | diff --git a/docs/recipes/ouro_ascend.md b/docs/recipes/ouro_ascend.md new file mode 100644 index 00000000..82c3887a --- /dev/null +++ b/docs/recipes/ouro_ascend.md @@ -0,0 +1,138 @@ +# Ouro on Ascend + +Run Ouro-1.4B on one Ascend NPU with BF16, greedy decoding and four recurrent +loops. The Python runtime uses Torch's `scaled_dot_product_attention` through +the `ascend_eager` backend, matching HF's SDPA entry point. Torch-NPU selects +the device kernel according to the input shape, mask and hardware. +The example enables the optional asynchronous engine plan. + +## Prepare the environment and model + +Install the project dependencies in a compatible Torch, Torch-NPU and CANN +environment. Use the checkpoint's config and byte-level tokenizer. + +```bash +source /path/to/cann/set_env.sh +source /path/to/npu-environment/bin/activate +python -m pip install --no-deps --no-build-isolation -e . +export MODEL=/path/to/prepared/Ouro-1.4B +export ASCEND_RT_VISIBLE_DEVICES=0 # Replace 0 with your assigned NPU. +``` + +## Serve fixed-depth requests + +This example sizes KV for 32 resident requests with 128 input and 128 output +tokens each. Set `--num-blocks` explicitly and adjust it for the expected +sequence lengths, resident request count and available device memory. For +whole-cohort prefill, the chunk size must fit each prompt and the batched token +budget must fit the sum of prompt lengths in the cohort. + +```bash +TASK_QUEUE_ENABLE=2 OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 \ +python -m vllm_rlt.entrypoints.serve \ + --model "$MODEL" --tokenizer "$MODEL" --served-model-name ouro \ + --device npu --dtype bfloat16 --attention-backend ascend_eager \ + --mode refill --exit-mode ouro_delayed \ + --async-scheduling --single-stream --no-uva \ + --block-size 16 --num-blocks 2064 --max-num-seqs 32 \ + --max-num-batched-tokens 8192 --prefill-chunk-size 1024 \ + --max-prefill-batches-before-decode 32 --host 127.0.0.1 --port 8123 +``` + +For synchronous eager execution, omit `--async-scheduling`. + +Wait for `/health` to return HTTP 200, then send a completion. Add +`"stream": true` for server-sent events. + +```bash +curl --fail http://127.0.0.1:8123/health +curl --fail http://127.0.0.1:8123/v1/completions \ + -H 'Content-Type: application/json' \ + -d '{"model":"ouro","prompt":"The capital of France is","max_tokens":8,"temperature":0,"min_loops":4,"max_loops":4,"exit_threshold":1}' +``` + + +## Fixed-depth asynchronous engine plan + +- **Scheduling.** Stream events track completion and protect buffer and request + lifetimes. Decode PRELUDE and each recurrent loop are separate submissions, + so the scheduler can select a new batch at each boundary. +- **Reuse and batching.** Ascend uses the shared packed-prefill metadata by + default with `ascend_eager`. Use `--no-packed-prefill` to select the per-query + layout. Greedy sampling and compatible prefill are batched. Full prefill + uses causal SDPA. + Cached chunks and ragged decode pass boolean masks to preserve each row's + visible KV range. +- **Paged KV.** Each loop writes to the page pool through PyTorch indexed + assignment using its own block table. + The Ascend backend gathers the visible KV rows for Torch SDPA. + +The example uses private, fully reserved `last_exited` KV. Keep graphs, +static buffers, padding, prefix caching, incremental KV allocation, preemption, +speculation and prefill/decode disaggregation disabled in this configuration. +Greedy parity remains unqualified for arbitrary chunked or mixed-length inputs +and adaptive exit. + + +## Performance and correctness + +A higher prefill quota lets more requests reach decode together and can delay +decode for existing requests. The scheduler default is 1. + +### Full unified W1-W7 validation + +This historical run used the C++ host executor and explicit eager attention +on both sides of the comparison. For the SDPA configuration above, compare +against HF SDPA with the same weights, input IDs, output lengths and concurrency. +Check the actual prefill and decode kernels on both sides with a short profile. +Measure throughput, TTFT and TPOT with the profiler disabled. Separate +synchronous and asynchronous vllm-rlt runs to measure the effect of scheduling. + +Runtime `7b4537f` used the same budget 8192, chunk 1024 and prefill quota 32 for +all seven [unified execution workloads](../unified_execution.md#protocol-and-comparison-rules). +The comparison ran Ouro-1.4B BF16 with four fixed loops and greedy decoding on +one 64 GiB Ascend 910B3 per run, with CANN 9, Torch/Torch-NPU 2.10 and +Transformers 4.54.1. The reference used official HF eager at revision +`574fa66cb8bf5abdc979642d01cf2b79b16bfab1` with a DynamicCache adapter. + +All 42 runs completed with 1,422 successful requests and zero runtime failures. +Each case used three unprofiled repetitions per arm on the same card and CPU +affinity. Each repetition admitted closed-loop traffic for 60 seconds and then +fully drained. Throughput includes drain and excludes startup, compilation and +warmup. Timing uses the in-process engine actor. W1–W5 ignore EOS. W6/W7 use +natural EOS with a 512-token cap and a fixed ShareGPT selection. + +| Workload | Input / output / clients | Residents per arm | HF eager tok/s | Engine tok/s | vs HF | Complete output matches | +| --- | --- | ---: | ---: | ---: | ---: | ---: | +| W1 | 128 / 128 / 1 | 1 | 6.334 [5.284–6.496] | 9.464 [9.403–10.394] | +49.42% | 15/15 | +| W2 | 128 / 128 / 32 | 32 | 201.314 [200.099–204.437] | 305.496 [293.376–316.662] | +51.75% | 480/480 | +| W3 | 4096 / 128 / 1 | 1 | 6.254 [6.206–6.268] | 7.569 [6.110–9.215] | +21.03% | 12/12 | +| W4 | 4096 / 128 / 32 | 4 | 21.211 [21.012–21.580] | 31.312 [31.135–31.405] | +47.62% | 120/132 | +| W5 | 1024 / 512 / 8 | 8 | 50.885 [50.234–51.362] | 79.791 [76.843–80.666] | +56.80% | 48/48 | +| W6 | ShareGPT / ≤512 / 8 | 8 | 26.343 [26.060–26.385] | 37.992 [37.906–38.879] | +44.22% | 18/27 | +| W7 | ShareGPT / ≤512 / 32 | 8 | 36.341 [35.576–36.931] | 36.194 [35.808–36.679] | -0.41% | 45/99 | + +Rates are medians [min–max]. Resident limits reflect single-device memory +capacity and apply equally to both arms. Complete output checks compare token IDs +and finish reasons against timed or supplemental HF references. + +**W1, W2, W3 and W5 passed the covered-input output checks.** W4, W6 and W7 +still differ from HF, so their throughput figures are diagnostic. W1–W5 used +owned KV and native dispatch throughout timing. W6/W7 used paged KV and Python +execution. W7 throughput ranges overlap, and its peak allocated memory was +30.06 GiB for the engine versus 19.54 GiB for HF. At W2, mean TPOT fell from +157.04 to 102.73 ms while TTFT p95 rose from 315.98 to 616.33 ms. + +## Run regression checks + +With an assigned NPU visible and `MODEL` set as above: + +```bash +VLLM_RLT_OURO_CHECKPOINT="$MODEL" python -m pytest -q --run-npu \ + tests/test_ascend_eager.py tests/test_ascend_kv.py \ + tests/test_ascend_decode_scheduling.py \ + tests/test_ascend_checkpoint.py tests/test_ascend_serving.py +``` + +For broader evaluation, see [accuracy](../accuracy.md) and +[serving benchmarks](../serving.md). diff --git a/pyproject.toml b/pyproject.toml index fa46c0c8..067040f1 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -43,7 +43,10 @@ license-files = ["LICENSE", "NOTICE"] [tool.pytest.ini_options] testpaths = ["tests"] pythonpath = ["."] -markers = ["gpu: requires an explicitly selected CUDA device and --run-gpu"] +markers = [ + "gpu: requires an explicitly selected CUDA device and --run-gpu", + "npu: requires an assigned Ascend device and torch-npu/CANN", +] [tool.ruff] line-length = 100 diff --git a/tests/conftest.py b/tests/conftest.py index f65e8a42..6981e499 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -8,15 +8,21 @@ def pytest_addoption(parser): parser.addoption( "--run-gpu", action="store_true", help="Run CUDA tests on explicitly selected devices" ) + parser.addoption("--run-npu", action="store_true", help="Run Ascend tests on an assigned NPU") def pytest_collection_modifyitems(config, items): enabled = config.getoption("--run-gpu") if enabled and not os.environ.get("CUDA_VISIBLE_DEVICES"): raise pytest.UsageError("--run-gpu requires explicit CUDA_VISIBLE_DEVICES") + npu_enabled = config.getoption("--run-npu") + if npu_enabled and not os.environ.get("ASCEND_RT_VISIBLE_DEVICES"): + raise pytest.UsageError("--run-npu requires assigned ASCEND_RT_VISIBLE_DEVICES") for item in items: if "gpu" in item.keywords and not enabled: item.add_marker(pytest.mark.skip(reason="set CUDA_VISIBLE_DEVICES and use --run-gpu")) + if "npu" in item.keywords and not npu_enabled: + item.add_marker(pytest.mark.skip(reason="use --run-npu on an assigned NPU")) @pytest.fixture(autouse=True, scope="session") diff --git a/tests/test_ascend.py b/tests/test_ascend.py new file mode 100644 index 00000000..fc4209d7 --- /dev/null +++ b/tests/test_ascend.py @@ -0,0 +1,170 @@ +"""Opt-in synchronous BF16 Ouro checks using explicit Torch attention on Ascend.""" + +import pytest +import torch + +from tests.helpers import tiny_ouro_config +from vllm_rlt import CacheConfig, SamplingParams, SchedulerConfig +from vllm_rlt.core.kv_cache_manager import KVCacheManager +from vllm_rlt.device import resolve_device +from vllm_rlt.engine.stage_engine import StageEngine +from vllm_rlt.kernels.paged_attention import torch_paged_attention +from vllm_rlt.models import OuroForCausalLM + +pytestmark = pytest.mark.npu + + +@pytest.fixture +def npu_device(): + resolve_device("npu") + torch.npu.set_device(0) + yield torch.device("npu:0") + torch.npu.synchronize() + torch.npu.empty_cache() + + +def assert_bf16_close(actual, expected): + a, b = actual.detach().float().cpu(), expected.detach().float().cpu() + assert torch.isfinite(a).all() and torch.isfinite(b).all() + assert (a - b).norm() / b.norm().clamp_min(1e-8) <= 0.02 + torch.testing.assert_close(a, b, atol=0.02, rtol=0.02) + + +@pytest.mark.parametrize("kv_heads", [1, 2, 4]) +def test_torch_attention_on_npu_matches_cpu_prefix_oracle(npu_device, kv_heads): + torch.manual_seed(83) + keys = torch.randn(8, 2, 16, kv_heads, 64, dtype=torch.bfloat16) + values = torch.randn_like(keys) + q = torch.randn(64, 4, 64, dtype=torch.bfloat16) + lengths = torch.tensor([0, 1, 15, 16, 17, 31, 32, 3] * 8, dtype=torch.int32) + tables = torch.tensor([[5, 2], [1, 7]] * 32, dtype=torch.int32) + expected = torch_paged_attention(q, keys[:, 1], values[:, 1], tables, lengths) + actual = torch_paged_attention( + q.to(npu_device), + keys.to(npu_device)[:, 1], + values.to(npu_device)[:, 1], + tables.to(npu_device), + lengths.to(npu_device), + ) + assert_bf16_close(actual, expected) + assert torch.count_nonzero(actual[0]).item() == 0 + + +def make_npu_model(device): + torch.manual_seed(41) + config = tiny_ouro_config(hidden_size=256, intermediate_size=512, head_dim=64) + return OuroForCausalLM(config).to(device=device, dtype=torch.bfloat16).eval() + + +@torch.inference_mode() +def recurrent_states(model, chunks, *, tokens=None, incremental=False): + c = model.config + cache = KVCacheManager( + c.num_hidden_layers, + c.num_key_value_heads, + c.head_dim, + 32, + 16, + 4, + device=next(model.parameters()).device, + dtype=torch.bfloat16, + backend="torch", + incremental_allocation=incremental, + ) + tokens = tokens if tokens is not None else ([3, 7, 2, 11, 5, 17, 8] * 3)[:21] + assert cache.allocate("a", len(tokens), initial_tokens=16 if incremental else None) + states, kv, start = {}, {}, 0 + try: + for size in chunks: + assert cache.ensure_capacity("a", start + size) + positions = list(range(start, start + size)) + hidden = model.prelude(torch.tensor(tokens[start : start + size], device=cache.device)) + for depth in range(4): + hidden, _ = model.recurrent(hidden, ["a"] * size, [depth] * size, positions, cache) + states[(start, depth)] = (hidden.clone(), model.coda(hidden).clone()) + start += size + assert start == len(tokens) + for depth in range(4): + for layer in range(c.num_hidden_layers): + assert all(cache.token_written("a", p, depth) for p in range(len(tokens))) + kv[(layer, depth)] = tuple( + t.clone() for t in cache.read(layer, "a", depth, len(tokens)) + ) + assert len({tuple(cache.get_block_table("a", d)) for d in range(4)}) == 4 + return states, kv + finally: + cache.free("a") + assert cache.num_used_blocks == 0 + + +def test_npu_chunked_prefill_and_incremental_kv(npu_device): + model = make_npu_model(npu_device) + expected, expected_kv = recurrent_states(model, [19, 1, 1]) + incremental, incremental_kv = recurrent_states(model, [19, 1, 1], incremental=True) + for key in expected: + for a, b in zip(incremental[key], expected[key], strict=True): + torch.testing.assert_close(a, b, atol=0, rtol=0) + for key in expected_kv: + for a, b in zip(incremental_kv[key], expected_kv[key], strict=True): + torch.testing.assert_close(a, b, atol=0, rtol=0) + chunked, chunked_kv = recurrent_states(model, [5, 14, 1, 1], incremental=True) + for key in ((19, d) for d in range(4)): + for a, b in zip(chunked[key], expected[key], strict=True): + assert_bf16_close(a, b) + for key in expected_kv: + for a, b in zip(chunked_kv[key], expected_kv[key], strict=True): + assert_bf16_close(a, b) + + +def test_npu_cancel_reuse_and_multiple_requests(npu_device): + engine = StageEngine( + make_npu_model(npu_device), + attention_backend="torch", + cache_config=CacheConfig(num_blocks=32, block_size=16, incremental_allocation=True), + scheduler_config=SchedulerConfig(max_num_seqs=2, max_num_batched_tokens=32), + ) + try: + params = SamplingParams(max_tokens=3, temperature=0, ignore_eos=True) + engine.add_request("cancel", [3, 7, 2] * 6, params) + engine.add_request("keep", [5, 9, 11], params) + engine.step() + engine.abort_request("cancel") + engine.add_request("cancel", [13, 14, 15], params) + finished = {} + for _ in range(200): + for output in engine.step(): + if output.finished: + finished[output.request_id] = output + if not engine.has_unfinished_requests(): + break + assert set(finished) == {"keep", "cancel"} + assert all( + o.exit_depths == [4] * 3 and o.finish_reason == "length" for o in finished.values() + ) + assert engine.cache_manager.num_used_blocks == 0 + finally: + engine.close() + + +@pytest.mark.parametrize("ignore_eos", [False, True]) +def test_npu_eos_and_length_handling(npu_device, ignore_eos): + model = make_npu_model(npu_device) + model.lm_head.weight.zero_() # Argmax 0 is the tiny model's actual EOS token. + engine = StageEngine(model, attention_backend="torch", cache_config=CacheConfig(num_blocks=16)) + try: + engine.add_request( + "eos", [3, 7, 2], SamplingParams(max_tokens=3, temperature=0, ignore_eos=ignore_eos) + ) + output = None + for _ in range(100): + for item in engine.step(): + if item.finished: + output = item + if not engine.has_unfinished_requests(): + break + assert output is not None + assert output.token_ids == [0] * (3 if ignore_eos else 1) + assert output.finish_reason == ("length" if ignore_eos else "stop") + assert engine.cache_manager.num_used_blocks == 0 + finally: + engine.close() diff --git a/tests/test_ascend_checkpoint.py b/tests/test_ascend_checkpoint.py new file mode 100644 index 00000000..c26e8fe8 --- /dev/null +++ b/tests/test_ascend_checkpoint.py @@ -0,0 +1,100 @@ +"""Opt-in local-checkpoint KV and lifecycle regressions on Ascend.""" + +import os +from pathlib import Path + +import pytest +import torch + +from tests.test_ascend import assert_bf16_close, recurrent_states +from vllm_rlt import CacheConfig, SamplingParams, SchedulerConfig +from vllm_rlt.device import resolve_device +from vllm_rlt.engine.stage_engine import StageEngine +from vllm_rlt.models import OuroForCausalLM, load_tokenizer + +pytestmark = pytest.mark.npu + + +@pytest.fixture(scope="module") +def checkpoint(): + source = os.environ.get("VLLM_RLT_OURO_CHECKPOINT") + if not source: + pytest.skip("set VLLM_RLT_OURO_CHECKPOINT to a prepared local Ouro checkpoint") + if not Path(source).is_dir(): + pytest.fail("VLLM_RLT_OURO_CHECKPOINT must be a prepared local directory") + resolve_device("npu") + torch.npu.set_device(0) + model = OuroForCausalLM.from_pretrained(source, device="npu", dtype=torch.bfloat16) + assert all(p.dtype == torch.bfloat16 for p in model.parameters()) + yield model, load_tokenizer(source) + torch.npu.synchronize() + + +@torch.inference_mode() +def test_checkpoint_incremental_pages_preserve_hidden_logits_and_kv(checkpoint): + model, tokenizer = checkpoint + tokens = tokenizer.encode( + "The capital of France is Paris. The capital of Japan is Tokyo. " + "A language model predicts the next token in a sequence.", + add_special_tokens=False, + )[:21] + assert len(tokens) == 21 + expected, expected_kv = recurrent_states(model, [19, 1, 1], tokens=tokens) + actual, actual_kv = recurrent_states(model, [19, 1, 1], tokens=tokens, incremental=True) + pairs = {} + for key in expected: + for name, a, b in zip(("hidden", "logits"), actual[key], expected[key], strict=True): + pairs[f"{key}.{name}"] = (a, b) + for key in expected_kv: + for name, a, b in zip(("key", "value"), actual_kv[key], expected_kv[key], strict=True): + pairs[f"{key}.{name}"] = (a, b) + for a, b in pairs.values(): + assert_bf16_close(a, b) + torch.testing.assert_close(a, b, atol=0, rtol=0) + + +@pytest.mark.parametrize("backend", ["torch", "ascend_eager"]) +@torch.inference_mode() +def test_checkpoint_greedy_replay_and_request_cleanup(checkpoint, backend): + model, tokenizer = checkpoint + engine = StageEngine( + model, + attention_backend=backend, + cache_config=CacheConfig(num_blocks=64), + scheduler_config=SchedulerConfig(max_num_seqs=2, max_num_batched_tokens=32), + ) + prompts = ["The capital of France is", "One plus one equals"] + loops = model.config.total_ut_steps + params = SamplingParams( + max_tokens=4, temperature=0, min_loops=loops, max_loops=loops, ignore_eos=True + ) + expected = None + try: + # Repeat the same packed workload and reuse IDs after all pages are freed. + # Expected IDs come from this checkpoint, not a particular weight file. + for _ in range(2): + finished = {} + for index, prompt in enumerate(prompts): + engine.add_request( + str(index), tokenizer.encode(prompt, add_special_tokens=False), params + ) + for _ in range(200): + for output in engine.step(): + if output.finished: + assert len(output.token_ids) == params.max_tokens + assert all( + 0 <= token < model.config.vocab_size for token in output.token_ids + ) + assert output.exit_depths == [loops] * params.max_tokens + assert output.finish_reason == "length" + finished[output.request_id] = list(output.token_ids) + if not engine.has_unfinished_requests(): + break + assert set(finished) == {str(index) for index in range(len(prompts))} + assert engine.cache_manager.num_used_blocks == 0 + if expected is None: + expected = finished + else: + assert finished == expected + finally: + engine.close() diff --git a/tests/test_ascend_compatibility.py b/tests/test_ascend_compatibility.py new file mode 100644 index 00000000..9f264189 --- /dev/null +++ b/tests/test_ascend_compatibility.py @@ -0,0 +1,89 @@ +"""Ascend opt-ins preserve the existing CPU/CUDA/ROCm API and defaults.""" + +import builtins +from dataclasses import fields + +import pytest +import torch + +from tests.helpers import tiny_ouro_model +from vllm_rlt import CacheConfig, SchedulerConfig +from vllm_rlt.attention.backend import BackendCapabilities +from vllm_rlt.device import stream_module +from vllm_rlt.engine.stage_engine import StageEngine +from vllm_rlt.layers.rotary_embedding import RotaryEmbedding +from vllm_rlt.worker.executor import ModelRuntime +from vllm_rlt.worker.model_runner import ModelRunner +from vllm_rlt.worker.sampler import Sampler + + +def test_backend_capability_positional_arguments_keep_their_meaning(): + names = tuple(f.name for f in fields(BackendCapabilities)) + assert names == ( + "packed_prefill", + "cuda_graphs", + "async_scheduling", + "packed_matches_decode", + "fp8_kv", + "fp8_kv_static", + "device_context_lengths", + "sliding_window", + "packed_sliding_window", + ) + caps = BackendCapabilities(False, False, False, False, False, True, False, True) + assert caps.sliding_window and caps.fp8_kv_static + + +@pytest.mark.parametrize("hip", [None, "6.4"]) +def test_cuda_and_rocm_stream_selection_never_loads_npu(monkeypatch, hip): + monkeypatch.setattr(torch.version, "hip", hip) + original = builtins.__import__ + + def guarded(name, *args, **kwargs): + if name == "torch_npu" or name.startswith("vllm_rlt.worker.ascend_model_runner"): + pytest.fail("CUDA/ROCm selected an Ascend implementation") + return original(name, *args, **kwargs) + + monkeypatch.setattr(builtins, "__import__", guarded) + assert stream_module(torch.device("cuda:0")) is torch.cuda + + +@pytest.mark.parametrize("hip", [None, "6.4"]) +def test_cuda_rope_keeps_device_math_without_host_migration(monkeypatch, hip): + monkeypatch.setattr(torch.version, "hip", hip) + rotary = RotaryEmbedding(128) + arange = torch.arange + seen = [] + + def recording(*args, **kwargs): + seen.append(kwargs["device"]) + # CPU execution of the same arithmetic; no CUDA/ROCm hardware claim. + return arange(*args, **dict(kwargs, device="cpu")) + + monkeypatch.setattr(torch, "arange", recording) + actual = rotary.frequencies(torch.device("cuda:0")) + assert seen == [torch.device("cuda:0")] + expected = 1.0 / (10000.0 ** (arange(0, 128, 2, dtype=torch.float32) / 128)) + torch.testing.assert_close(actual, expected, atol=0, rtol=0) + + +def test_default_engine_keeps_upstream_runtime_and_sampler(monkeypatch): + original = builtins.__import__ + + def guarded(name, *args, **kwargs): + if name == "torch_npu" or name.startswith("vllm_rlt.worker.ascend_model_runner"): + pytest.fail("default engine loaded Ascend worker code") + return original(name, *args, **kwargs) + + monkeypatch.setattr(builtins, "__import__", guarded) + engine = StageEngine( + tiny_ouro_model(), + cache_config=CacheConfig(num_blocks=128), + scheduler_config=SchedulerConfig(max_num_seqs=2, max_num_batched_tokens=128), + ) + try: + assert type(engine.model_runner) is ModelRunner + assert type(engine.model_runner.runtime) is ModelRuntime + assert type(engine.model_runner.sampler) is Sampler + finally: + engine.close() diff --git a/tests/test_ascend_decode_scheduling.py b/tests/test_ascend_decode_scheduling.py new file mode 100644 index 00000000..c66aea06 --- /dev/null +++ b/tests/test_ascend_decode_scheduling.py @@ -0,0 +1,239 @@ +"""Ascend decode scheduling, refill and asynchronous result delivery.""" + +import pytest +import torch + +from tests.helpers import make_cpu_cache, tiny_ouro_config +from tests.test_ascend import make_npu_model +from tests.test_npu_async import drain +from vllm_rlt import CacheConfig, ExecutionConfig, ExitConfig, SamplingParams, SchedulerConfig +from vllm_rlt.core.scheduler import ScheduledRequest, Scheduler, SchedulerOutput +from vllm_rlt.device import resolve_device +from vllm_rlt.engine.stage_engine import StageEngine +from vllm_rlt.request import Request, RequestPhase + + +def test_decode_prelude_leaves_recurrent_work_to_scheduler(): + scheduler = Scheduler(SchedulerConfig(), None, total_ut_steps=4, exit_mode="ouro_delayed") + request = Request("a", [1], SamplingParams(min_loops=4, max_loops=4)) + request.phase = RequestPhase.PRELUDE + scheduler.requests["a"] = request + item = ScheduledRequest(request, phase=RequestPhase.PRELUDE) + assert scheduler._planned_loop_count([item]) == 0 + batch = SchedulerOutput(RequestPhase.PRELUDE, [item], 0) + update = scheduler.update_after_submission(batch) + assert request.loops_done == 0 + assert request.phase == RequestPhase.RECURRENT + assert update.exited == [] + request.phase = RequestPhase.PRELUDE + scheduler.requests.pop("a") + scheduler.update_after_submission(batch) + assert request.phase == RequestPhase.PRELUDE # A stale submission cannot advance a request. + + +@pytest.mark.parametrize("late_arrival", [False, True]) +def test_decode_preserves_refill_opportunity(late_arrival): + scheduler = Scheduler( + SchedulerConfig(max_num_seqs=2, max_num_batched_tokens=3, prefill_chunk_size=3), + make_cpu_cache(tiny_ouro_config(), blocks=128), + total_ut_steps=4, + exit_mode="ouro_delayed", + ) + params = SamplingParams(max_tokens=16, min_loops=4, max_loops=4, ignore_eos=True) + first = Request("a", [1, 2, 3], params) + second = Request("b", [4, 5, 6], params) + scheduler.add_request(first) + if not late_arrival: + scheduler.add_request(second) + batch = scheduler.schedule() + assert batch.phase == RequestPhase.PREFILL + assert [item.request for item in batch.requests] == [first] + scheduler.update_after_submission(batch) + batch = scheduler.schedule() + assert batch.phase == RequestPhase.CODA + scheduler.update_after_submission(batch) + if late_arrival: + scheduler.add_request(second) + assert second.phase == (RequestPhase.WAITING if late_arrival else RequestPhase.PREFILL) + + # Every prompt consumes a complete token budget. Refill must still get a + # turn before the first generation ends. + for _ in range(8): + batch = scheduler.schedule() + assert batch is not None + scheduler.update_after_submission(batch) + if any(item.request is second for item in batch.requests): + assert batch.phase == RequestPhase.PREFILL + assert first.num_scheduled_outputs < params.max_tokens + break + else: + pytest.fail("decode boundaries starved a pending prompt") + targets = {request.request_id: request.num_scheduled_outputs + 1 for request in (first, second)} + for _ in range(12): + batch = scheduler.schedule() + assert batch is not None + scheduler.update_after_submission(batch) + if all( + request.num_scheduled_outputs >= targets[request.request_id] + for request in (first, second) + ): + break + else: + pytest.fail("refilled decode starved the existing recurrent request") + + +def engine_for(model, multi_stream=False, token_budget=128): + return StageEngine( + model, + attention_backend="ascend_eager", + cache_config=CacheConfig(num_blocks=128, block_size=16), + scheduler_config=SchedulerConfig( + max_num_seqs=3, max_num_batched_tokens=token_budget, prefill_chunk_size=128 + ), + execution_config=ExecutionConfig( + async_scheduling=True, + multi_stream=multi_stream, + use_uva=False, + ), + exit_config=ExitConfig("ouro_delayed"), + ) + + +@pytest.mark.parametrize("waiting", [False, True]) +def test_decode_prelude_keeps_recurrent_due(waiting): + scheduler = Scheduler( + SchedulerConfig(max_num_seqs=2, max_prefill_batches_before_decode=4), + make_cpu_cache(tiny_ouro_config(), blocks=128), + total_ut_steps=4, + exit_mode="ouro_delayed", + ) + params = SamplingParams(max_tokens=8, min_loops=4, max_loops=4, ignore_eos=True) + scheduler.add_request(Request("a", [1, 2, 3], params)) + for expected in (RequestPhase.PREFILL, RequestPhase.CODA): + batch = scheduler.schedule() + assert batch.phase == expected + scheduler.update_after_submission(batch) + if waiting: + scheduler.add_request(Request("b", [4, 5, 6], params)) + for _ in range(scheduler.config.max_prefill_batches_before_decode): + scheduler.policy.record_batch(RequestPhase.PREFILL) + assert scheduler.policy.decode_due + batch = scheduler.schedule() + assert batch.phase == RequestPhase.PRELUDE + assert batch.loop_count == 0 + assert scheduler.policy.decode_due + + +@pytest.mark.npu +@torch.inference_mode() +@pytest.mark.parametrize("multi_stream", [False, True]) +def test_ascend_decode_refills_during_generation(multi_stream): + resolve_device("npu") + torch.npu.set_device(0) + model = make_npu_model(torch.device("npu:0")) + params = SamplingParams(max_tokens=8, min_loops=4, max_loops=4, ignore_eos=True) + engine = engine_for(model, multi_stream, token_budget=3) + try: + for name, prompt in [("a", [3, 7, 11]), ("b", [13, 17, 19])]: + engine.add_request(name, prompt, params) + actual, first_finished, second_started = {}, False, False + first_tokens_at_second_start = None + first_tokens, first_advanced = 0, False + for _ in range(128): + if not engine.has_unfinished_requests(): + break + for out in engine.step(): + if out.request_id == "b" and not second_started: + assert not first_finished + second_started = True + first_tokens_at_second_start = first_tokens + if out.request_id == "a": + first_tokens = len(out.token_ids) + if second_started and "b" not in actual: + first_advanced |= first_tokens > first_tokens_at_second_start + if out.finished: + actual[out.request_id] = ( + list(out.token_ids), + list(out.exit_depths), + out.finish_reason, + ) + if out.request_id == "a": + first_finished = True + else: + pytest.fail("refill did not drain in 128 steps") + assert second_started and first_advanced + assert set(actual) == {"a", "b"} + assert engine.cache_manager.num_used_blocks == 0 + finally: + engine.close() + + +@pytest.mark.npu +@torch.inference_mode() +@pytest.mark.parametrize("multi_stream", [False, True]) +def test_ascend_decode_preserves_cancel_reuse_and_eos(multi_stream): + resolve_device("npu") + torch.npu.set_device(0) + model = make_npu_model(torch.device("npu:0")) + params = SamplingParams(max_tokens=4, min_loops=4, max_loops=4, ignore_eos=True) + engine = engine_for(model, multi_stream) + try: + for i, prompt in enumerate([[3, 7, 11], [13, 17, 19]]): + engine.add_request(str(i), prompt, params) + expected = drain(engine) + assert engine.cache_manager.num_used_blocks == 0 + engine.add_request("0", [2, 5, 9], params) + engine.add_request("1", [13, 17, 19], params) + engine.step() + engine.step() + engine.step() + engine.abort_request("0") + engine.add_request("0", [3, 7, 11], params) + assert drain(engine) == expected + eos = model.config.eos_token_id + engine.model_runner.sampler.sample_batch = lambda logits, options, generators: ( + logits.new_full((len(options),), eos, dtype=torch.long) + ) + engine.add_request( + "eos", [3, 7, 11], SamplingParams(max_tokens=4, min_loops=4, max_loops=4) + ) + assert drain(engine)["eos"] == ([eos], [4], "stop") + finally: + engine.close() + + +@pytest.mark.npu +@torch.inference_mode() +def test_ascend_prelude_uses_device_sample_before_coda_delivery(): + resolve_device("npu") + torch.npu.set_device(0) + model = make_npu_model(torch.device("npu:0")) + engine = engine_for(model) + + class HeldEvent: + def query(self): + return False + + def synchronize(self): + pytest.fail("decode plan waited for host token delivery") + + try: + engine.add_request( + "a", [3, 7, 11], SamplingParams(max_tokens=3, min_loops=4, max_loops=4, ignore_eos=True) + ) + engine.step() + engine.step() + ticket = engine._pending_coda[0] + actual_event = ticket.event + ticket.event = HeldEvent() + try: + engine.step() + assert engine.last_schedule.phase == RequestPhase.PRELUDE + assert engine.last_schedule.loop_count == 0 + request = engine.scheduler.requests["a"] + assert request.loops_done == 0 and request.generated_token_ids == [] + finally: + ticket.event = actual_event + assert drain(engine)["a"][1] == [4, 4, 4] + finally: + engine.close() diff --git a/tests/test_ascend_eager.py b/tests/test_ascend_eager.py new file mode 100644 index 00000000..f985a6d0 --- /dev/null +++ b/tests/test_ascend_eager.py @@ -0,0 +1,307 @@ +"""Causal/GQA semantics for Ascend SDPA over paged KV.""" + +import pytest +import torch + +from vllm_rlt.kernels.ascend_attention import AscendAttention, _flat_kv_rows, _prefill_geometry + + +@pytest.mark.parametrize("kv_heads", [1, 2, 4]) +def test_eager_groups_match_independent_causal_queries(kv_heads): + torch.manual_seed(730) + keys = torch.randn(8, 16, kv_heads, 64, dtype=torch.bfloat16) + values = torch.randn_like(keys) + q = torch.randn(7, 4, 64, dtype=torch.bfloat16) + lengths = torch.tensor([0, 1, 15, 16, 17, 3, 31], dtype=torch.int32) + tables = torch.tensor([[4, 1], [4, 1], [4, 1], [4, 1], [4, 1], [7, 2], [7, 2]]) + actual = AscendAttention()(q, keys, values, tables, lengths) + expected = torch.zeros_like(q) + for row, length in enumerate(lengths.tolist()): + if not length: + continue + pos = torch.arange(length) + k = keys[tables[row, pos // 16], pos % 16].repeat_interleave(4 // kv_heads, dim=1) + v = values[tables[row, pos // 16], pos % 16].repeat_interleave(4 // kv_heads, dim=1) + scores = torch.einsum("hd,thd->ht", q[row], k) * (64**-0.5) + probs = scores.softmax(-1, dtype=torch.float32).to(q.dtype) + expected[row] = torch.einsum("ht,thd->hd", probs, v) + torch.testing.assert_close(actual, expected, atol=0.02, rtol=0.02) + assert torch.isfinite(actual).all() + assert torch.count_nonzero(actual[0]) == 0 + + +@pytest.mark.npu +@torch.inference_mode() +def test_pooled_kv_reads_match_layer_views_across_pages_and_depths(): + from vllm_rlt.core.kv_cache_manager import KVCacheManager + from vllm_rlt.device import resolve_device + + device = resolve_device("npu:0") + torch.npu.set_device(device) + torch.manual_seed(99) + cache = KVCacheManager( + 3, 2, 64, 16, 16, 2, device=device, dtype=torch.bfloat16, backend="ascend_eager" + ) + assert cache.allocate("a", 17) + assert cache.allocate("b", 3) + batch = cache._prepare_batch(["a"] * 17 + ["b"] * 3, [0] * 17 + [1] * 3, [*range(17), 0, 1, 2]) + for layer in range(3): + k = torch.randn(20, 2, 64, device=device, dtype=torch.bfloat16) + v = torch.randn_like(k) + cache._write_prepared(layer, batch, k, v) + + q = torch.randn(20, 4, 64, device=device, dtype=torch.bfloat16) + for layer in range(3): + expected = cache.attention( + q, + cache.key_cache[:, layer], + cache.value_cache[:, layer], + batch.block_tables, + batch.context_lengths, + ) + actual = cache._attend_prepared(layer, batch, q) + torch.testing.assert_close(actual, expected, atol=0, rtol=0) + + +@pytest.mark.npu +@pytest.mark.parametrize("chunk_size", [4096, 128]) +@torch.inference_mode() +def test_engine_matches_the_same_manual_prefill_chunks(chunk_size): + from tests.helpers import tiny_ouro_config + from vllm_rlt import CacheConfig, SamplingParams, SchedulerConfig + from vllm_rlt.core.kv_cache_manager import KVCacheManager + from vllm_rlt.device import resolve_device + from vllm_rlt.engine.stage_engine import StageEngine + from vllm_rlt.models import OuroForCausalLM + + device = resolve_device("npu:0") + torch.npu.set_device(device) + torch.manual_seed(41) + model = OuroForCausalLM( + tiny_ouro_config( + hidden_size=256, intermediate_size=512, head_dim=64, max_position_embeddings=256 + ) + ).to(device=device, dtype=torch.bfloat16) + c = model.config + tokens = ([3, 7, 2, 11, 5, 17, 8] * 22)[:149] + cache = KVCacheManager( + c.num_hidden_layers, + c.num_key_value_heads, + c.head_dim, + 128, + 16, + 4, + device=device, + dtype=torch.bfloat16, + backend="ascend_eager", + ) + assert cache.allocate("manual", len(tokens) + 3) + expected, manual_states = [], [] + handle = model.model.norm.register_forward_hook( + lambda _m, _i, output: manual_states.append(output.detach().cpu().clone()) + ) + try: + for start in range(0, len(tokens), chunk_size): + end = min(start + chunk_size, len(tokens)) + hidden = model.prelude(torch.tensor(tokens[start:end], device=device)) + for depth in range(4): + hidden, _ = model.recurrent( + hidden, + ["manual"] * (end - start), + [depth] * (end - start), + list(range(start, end)), + cache, + compute_gate=False, + ) + for step in range(3): + expected.append(int(model.coda(hidden[-1:])[0].argmax())) + if step < 2: + pos = len(tokens) + step + hidden = model.prelude(torch.tensor([expected[-1]], device=device)) + for depth in range(4): + hidden, _ = model.recurrent( + hidden, ["manual"], [depth], [pos], cache, compute_gate=False + ) + finally: + handle.remove() + cache.free("manual") + actual_states = [] + handle = model.model.norm.register_forward_hook( + lambda _m, _i, output: actual_states.append(output.detach().cpu().clone()) + ) + engine = StageEngine( + model, + attention_backend="ascend_eager", + cache_config=CacheConfig(num_blocks=128), + scheduler_config=SchedulerConfig( + max_num_seqs=1, max_num_batched_tokens=4096, prefill_chunk_size=chunk_size + ), + ) + try: + assert engine.execution_config.packed_prefill + engine.add_request( + "engine", + tokens, + SamplingParams(max_tokens=3, temperature=0, min_loops=4, max_loops=4, ignore_eos=True), + ) + result = None + while engine.has_unfinished_requests(): + for out in engine.step(): + if out.finished: + result = out + assert result is not None + assert result.token_ids == expected + assert result.exit_depths == [4] * 3 + assert len(actual_states) == len(manual_states) + for actual, reference in zip(actual_states, manual_states, strict=True): + torch.testing.assert_close(actual, reference, atol=0, rtol=0) + assert engine.cache_manager.num_used_blocks == 0 + finally: + handle.remove() + engine.close() + + +@pytest.mark.parametrize("device_name", ["cpu", pytest.param("npu", marks=pytest.mark.npu)]) +@torch.inference_mode() +def test_cached_continuation_passes_bool_mask_to_sdpa(device_name, monkeypatch): + from vllm_rlt.device import resolve_device + + device = resolve_device(device_name) + if device_name == "npu": + torch.npu.set_device(0) + keys = torch.zeros(1, 16, 2, 64, dtype=torch.bfloat16, device=device) + values = torch.zeros_like(keys) + for pos, value in enumerate((1, 3, 8)): + values[0, pos].fill_(value) + query = torch.ones(2, 2, 64, dtype=torch.bfloat16, device=device) + table = torch.tensor([[0], [0]], dtype=torch.int32, device=device) + lengths = torch.tensor([2, 3], dtype=torch.int32, device=device) + sdpa = torch.nn.functional.scaled_dot_product_attention + calls = [] + + def check_sdpa(q, k, v, *, attn_mask, is_causal): + calls.append(attn_mask) + assert not is_causal + assert q.shape == (1, 2, 2, 64) and k.shape == v.shape == (1, 2, 3, 64) + assert attn_mask.dtype == torch.bool + assert attn_mask.tolist() == [[True, True, False], [True, True, True]] + return sdpa(q, k, v, attn_mask=attn_mask, is_causal=is_causal) + + monkeypatch.setattr(torch.nn.functional, "scaled_dot_product_attention", check_sdpa) + actual = AscendAttention()(query, keys, values, table, lengths) + # Equal scores average the visible values. The first query sees the cached + # prefix and itself, while only the second query can see the last value. + expected = torch.tensor([2, 4], device=device, dtype=query.dtype)[:, None, None] + torch.testing.assert_close(actual, expected.expand_as(actual), atol=0.016, rtol=0) + assert len(calls) == 1 + + +class _PreparedBatch: + cu_seqlens_q = None + + """Only the host metadata the two batched attention paths consume.""" + + def __init__(self, rows, tables, write_blocks, write_offsets): + self.rows = rows + self.block_tables = tables + self.write_blocks = write_blocks + self.write_offsets = write_offsets + + +def _batched_fixture(kind): + layers, blocks, block_size, kv_heads, head_dim = 3, 24, 4, 2, 8 + head_groups = 2 + dtype = torch.bfloat16 # the Ascend eager contract is BF16 end to end + torch.manual_seed(913) + pool_k = torch.randn(blocks, layers, block_size, kv_heads, head_dim, dtype=dtype) + pool_v = torch.randn_like(pool_k) + keys, values = pool_k[:, 1], pool_v[:, 1] + + first, second, third, fourth = (object() for _ in range(4)) + if kind == "prefill_grid": + rows = [(a, 0, pos) for a in (first, second) for pos in range(4)] + elif kind == "prefill_single": + rows = [(first, 0, pos) for pos in range(4)] + elif kind == "ragged_decode": + rows = [(a, 0, 5) for a in (first, second, third, fourth)] + elif kind == "ragged_padded_tables": + # Mixed natural lengths: short rows have -1 in the high page slots, + # exactly as plan_attention_metadata pads a shorter block table. + rows = [(first, 0, 9), (second, 0, 1), (third, 0, 5), (fourth, 0, 2)] + elif kind == "single_decode": + rows = [(first, 0, 5)] + else: # irregular: a prefill group with a missing position + rows = [(first, 0, 0), (first, 0, 2), (first, 0, 3)] + + page_tables = { + id(first): [2, 5, 7, 9], + id(second): [3, 6, 8, 10], + id(third): [4, 11, 12, 13], + id(fourth): [14, 15, 16, 17], + } + if kind == "ragged_padded_tables": + page_tables = { + id(first): [2, 5, 7, 9], + id(second): [3, -1, -1, -1], + id(third): [4, 11, -1, -1], + id(fourth): [14, -1, -1, -1], + } + tables, write_blocks, write_offsets = [], [], [] + for allocation, _depth, position in rows: + table = page_tables[id(allocation)] + tables.append(table) + write_blocks.append(table[position // block_size]) + write_offsets.append(position % block_size) + batch = _PreparedBatch( + rows, + torch.tensor(tables, dtype=torch.int32), + torch.tensor(write_blocks, dtype=torch.int64), + torch.tensor(write_offsets, dtype=torch.int64), + ) + q = torch.randn(len(rows), kv_heads * head_groups, head_dim, dtype=dtype) + return q, keys, values, batch + + +@pytest.mark.parametrize( + "kind,expected_prefill,expected_ragged", + [ + ("prefill_grid", 1, 0), + ("prefill_single", 1, 0), + ("ragged_decode", 0, 1), + ("ragged_padded_tables", 0, 1), + ("single_decode", 0, 0), + ("irregular", 0, 0), + ], +) +@torch.inference_mode() +def test_batched_attention_paths_match_per_group_sdpa(kind, expected_prefill, expected_ragged): + """Keep query/result ordering and masks consistent when batching SDPA calls.""" + q, keys, values, batch = _batched_fixture(kind) + kernel = AscendAttention() + groups = kernel._plan( + [(id(allocation), depth) for allocation, depth, _ in batch.rows], + [position + 1 for _, _, position in batch.rows], + q.device, + ) + expected = kernel._forward( + q, + keys, + values, + batch.block_tables, + groups, + flat_rows=_flat_kv_rows(keys, values), + ) + actual = kernel.attend_metadata(q, keys, values, batch) + torch.testing.assert_close(actual, expected, atol=0, rtol=0) + assert kernel.batched_prefill_calls == expected_prefill + assert kernel.ragged_calls == expected_ragged + + +def test_prefill_geometry_accepts_one_request_but_rejects_irregular_grids(): + first, second = object(), object() + assert _prefill_geometry([(first, 0, pos) for pos in range(4)]) == (1, 4) + assert _prefill_geometry( + [(first, 0, pos) for pos in range(4)] + [(second, 0, pos) for pos in range(4)] + ) == (2, 4) + assert _prefill_geometry([(first, 0, 0), (first, 0, 2)]) is None + assert _prefill_geometry([(first, 0, 1)]) is None diff --git a/tests/test_ascend_kv.py b/tests/test_ascend_kv.py new file mode 100644 index 00000000..7d53d071 --- /dev/null +++ b/tests/test_ascend_kv.py @@ -0,0 +1,54 @@ +"""Paged KV writes and descriptor lifetime on an assigned Ascend device.""" + +import pytest +import torch + +from vllm_rlt.core.kv_cache_manager import KVCacheManager +from vllm_rlt.device import resolve_device + +pytestmark = pytest.mark.npu + + +@pytest.mark.parametrize("rows", [1, 3]) +@pytest.mark.parametrize("backend", ["torch", "ascend_eager"]) +@torch.inference_mode() +def test_paged_kv_writes_preserve_inputs_layers_and_reused_ids(rows, backend): + device = resolve_device("npu:0") + torch.npu.set_device(device) + cache = KVCacheManager( + 2, 2, 64, 32, 16, 4, device=device, dtype=torch.bfloat16, backend=backend + ) + cache.key_cache.zero_() + cache.value_cache.zero_() + assert cache.allocate("a", 17) + assert cache.allocate("b", 3) + ids, depths, positions = ["a", "b", "a"][:rows], [0, 2, 3][:rows], [15, 0, 16][:rows] + batch = cache._prepare_batch(ids, depths, positions) + torch.manual_seed(99) + k = torch.randn(rows, 2, 64, device=device, dtype=torch.bfloat16) + v = torch.randn_like(k) + saved_k, saved_v = k.clone(), v.clone() + expected_k, expected_v = cache.key_cache.clone(), cache.value_cache.clone() + for layer in range(2): + expected_k[batch.write_blocks, layer, batch.write_offsets] = k + expected_v[batch.write_blocks, layer, batch.write_offsets] = v + cache._write_prepared(layer, batch, k, v) + torch.testing.assert_close(cache.key_cache, expected_k, atol=0, rtol=0) + torch.testing.assert_close(cache.value_cache, expected_v, atol=0, rtol=0) + torch.testing.assert_close(k, saved_k, atol=0, rtol=0) + torch.testing.assert_close(v, saved_v, atol=0, rtol=0) + cache.free("a") + assert cache.allocate("a", 17) + with pytest.raises((ValueError, RuntimeError)): + cache._write_prepared(0, batch, k, v) + new_batch = cache._prepare_batch(["a"], [1], [0]) + expected_k, expected_v = cache.key_cache.clone(), cache.value_cache.clone() + expected_k[new_batch.write_blocks, 1, new_batch.write_offsets] = k[:1] + expected_v[new_batch.write_blocks, 1, new_batch.write_offsets] = v[:1] + cache._write_prepared(1, new_batch, k[:1], v[:1]) + torch.testing.assert_close(cache.key_cache, expected_k, atol=0, rtol=0) + torch.testing.assert_close(cache.value_cache, expected_v, atol=0, rtol=0) + cache.free("a") + cache.free("b") + assert cache.num_used_blocks == 0 + torch.npu.synchronize() diff --git a/tests/test_ascend_metadata.py b/tests/test_ascend_metadata.py new file mode 100644 index 00000000..5d8ec1c3 --- /dev/null +++ b/tests/test_ascend_metadata.py @@ -0,0 +1,214 @@ +"""Metadata reuse must preserve sparse coverage, ownership and eager values.""" + +import gc +import weakref + +import pytest +import torch + +from vllm_rlt.core.kv_cache_manager import KVCacheManager, _sequence_coverage, _WrittenPositions +from vllm_rlt.kernels.ascend_attention import AscendAttention, _flat_kv_rows + + +@pytest.mark.parametrize( + "ranges", + [ + [(0, 1024), (1024, 4096)], + [(5, 9), (0, 3), (3, 5)], + [(9, 12), (4, 7), (0, 6), (6, 10)], + [(2, 5), (2, 7), (0, 3), (1, 4), (9, 9)], + ], +) +def test_written_ranges_preserve_sparse_and_overlapping_coverage(ranges): + reference, actual = _WrittenPositions(), _WrittenPositions() + for start, end in ranges: + for position in range(start, end): + reference.add(position) + actual.add_range(start, end) + assert (actual.prefix, actual.pending) == (reference.prefix, reference.pending) + + +def test_sequence_coverage_keeps_holes_and_plane_identity(): + first, second = object(), object() + rows = [(first, 0, p) for p in (2, 0, 1, 5)] + [(first, 1, 0), (second, 0, 0)] + plans = _sequence_coverage(rows, lambda depth: depth) + assert [(p.allocation, p.plane, p.ranges) for p in plans] == [ + (first, 0, ((0, 3), (5, 6))), + (first, 1, ((0, 1),)), + (second, 0, ((0, 1),)), + ] + + +def _cache(): + from vllm_rlt.device import resolve_device + + device = resolve_device("npu:0") + torch.npu.set_device(device) + cache = KVCacheManager( + 2, 2, 64, 64, 16, 4, device=device, dtype=torch.bfloat16, backend="ascend_eager" + ) + cache.key_cache.fill_(float("nan")) + cache.value_cache.fill_(float("nan")) + lengths = [3, 7, 17] + for index, length in enumerate(lengths): + rid = str(index) + assert cache.allocate(rid, length) + for depth in (0, 1): + batch = cache._prepare_batch([rid] * length, [depth] * length, list(range(length))) + for layer in range(2): + k = torch.randn(length, 2, 64, device=device, dtype=torch.bfloat16) + v = torch.randn_like(k) + cache._write_prepared(layer, batch, k, v) + return cache, lengths + + +def _reference(kernel, q, keys, values, batch): + groups = kernel._plan( + [(id(a), depth) for a, depth, _ in batch.rows], + [p + 1 for _, _, p in batch.rows], + q.device, + ) + return kernel._forward( + q, keys, values, batch.block_tables, groups, flat_rows=_flat_kv_rows(keys, values) + ) + + +@pytest.mark.npu +@pytest.mark.parametrize("padded", [False, True]) +@torch.inference_mode() +def test_unwritten_nan_padding_is_never_consumed(padded): + cache, lengths = _cache() + batch = cache._prepare_batch( + ["0", "1", "2"], + [0] * 3, + [n - 1 for n in lengths], + for_write=False, + padded_rows=4 if padded else None, + ) + # Noncontiguous Q must retain the original contiguous attention layout. + q = torch.randn(4 if padded else 3, 4, 128, device=cache.device, dtype=cache.dtype)[:, :, ::2] + for layer in range(2): + keys, values = cache.key_cache[:, layer], cache.value_cache[:, layer] + expected = _reference(AscendAttention(), q, keys, values, batch) + actual = cache._attend_prepared(layer, batch, q) + assert torch.isfinite(actual).all() + torch.testing.assert_close(actual, expected, rtol=0, atol=0) + if padded: + assert torch.count_nonzero(actual[3]) == 0 + for rid in ("0", "1", "2"): + cache.free(rid) + + +@pytest.mark.npu +@torch.inference_mode() +def test_layout_reuse_does_not_reuse_addresses_or_outputs(): + cache, lengths = _cache() + kernel = cache.attention.kernel + q = torch.randn(3, 4, 64, device=cache.device, dtype=cache.dtype) + saved = None + for depth in (0, 1, 0): + batch = cache._prepare_batch( + ["0", "1", "2"], [depth] * 3, [n - 1 for n in lengths], for_write=False + ) + expected = _reference( + AscendAttention(), q, cache.key_cache[:, 0], cache.value_cache[:, 0], batch + ) + actual = cache._attend_prepared(0, batch, q) + torch.testing.assert_close(actual, expected, rtol=0, atol=0) + if saved is None: + saved, snapshot = actual, actual.clone() + assert kernel.layout_reuses >= 2 + torch.testing.assert_close(saved, snapshot, rtol=0, atol=0) + reference = weakref.ref(batch) + del batch + gc.collect() + assert reference() is None + for rid in ("0", "1", "2"): + cache.free(rid) + + +@pytest.mark.npu +@torch.inference_mode() +def test_cached_validation_keeps_layer_holes_truncation_and_reused_id_checks(): + cache, _ = _cache() + batch = cache._prepare_batch(["0", "0"], [2, 2], [0, 2]) + k = torch.randn(2, 2, 64, device=cache.device, dtype=cache.dtype) + q = torch.randn(2, 4, 64, device=cache.device, dtype=cache.dtype) + cache._write_prepared(0, batch, k, k) + with pytest.raises(RuntimeError, match="uninitialized KV"): + cache._attend_prepared(0, batch, q) + middle = cache._prepare_batch(["0"], [2], [1]) + cache._write_prepared(0, middle, k[:1], k[:1]) + assert torch.isfinite(cache._attend_prepared(0, batch, q)).all() + with pytest.raises(RuntimeError, match="uninitialized KV"): + cache._attend_prepared(1, batch, q) + cache.truncate_suffix("0", 0) + with pytest.raises(RuntimeError, match="uninitialized KV"): + cache._attend_prepared(0, batch, q) + cache.free("0") + assert cache.allocate("0", 3) + before = cache.key_cache.clone() + with pytest.raises(RuntimeError, match="stale prepared KV"): + cache._write_prepared(1, batch, k, k) + torch.testing.assert_close(cache.key_cache, before, rtol=0, atol=0, equal_nan=True) + for rid in ("0", "1", "2"): + cache.free(rid) + + +@pytest.mark.npu +@torch.inference_mode() +def test_layout_eviction_and_cross_stream_use_keep_returned_outputs(): + cache, _ = _cache() + kernel = cache.attention.kernel + q = torch.randn(3, 4, 64, device=cache.device, dtype=cache.dtype) + producer, consumer = torch.npu.Stream(), torch.npu.Stream() + producer.wait_stream(torch.npu.current_stream()) + batch = cache._prepare_batch(["0", "1", "2"], [0] * 3, [2, 6, 16], for_write=False) + # The caller owns readiness of KV and descriptor staging; cached layouts + # are private to each stream and are never overwritten for a later batch. + torch.npu.synchronize() + with torch.npu.stream(producer): + held = cache._attend_prepared(0, batch, q) + snapshot = held.clone() + done = torch.npu.Event() + done.record() + consumer.wait_event(done) + with torch.npu.stream(consumer): + second = cache._attend_prepared(0, batch, q) + for position in range(2, 14): + newer = cache._prepare_batch( + ["0", "1", "2"], [0] * 3, [2, 6, position], for_write=False + ) + cache._attend_prepared(0, newer, q) + torch.npu.synchronize() + torch.testing.assert_close(second, snapshot, rtol=0, atol=0) + torch.testing.assert_close(held, snapshot, rtol=0, atol=0) + assert len(kernel._layouts) <= 8 + assert kernel._layout_bytes <= kernel._layout_limit + for rid in ("0", "1", "2"): + cache.free(rid) + + +@pytest.mark.parametrize("kind", ["ragged_decode", "irregular"]) +@torch.inference_mode() +def test_failed_plan_preparation_invalidates_the_previous_descriptor(kind, monkeypatch): + from copy import copy + + from tests.test_ascend_eager import _batched_fixture + + q, keys, values, first = _batched_fixture(kind) + kernel = AscendAttention() + expected = kernel.attend_metadata(q, keys, values, first).clone() + second = copy(first) + target = "_prepare_decode" if kind == "ragged_decode" else "_plan" + original = getattr(kernel, target) + + def failed(*args, **kwargs): + raise RuntimeError("injected plan preparation failure") + + monkeypatch.setattr(kernel, target, failed) + with pytest.raises(RuntimeError, match="injected plan preparation"): + kernel.attend_metadata(q, keys, values, second) + monkeypatch.setattr(kernel, target, original) + actual = kernel.attend_metadata(q, keys, values, first) + torch.testing.assert_close(actual, expected, rtol=0, atol=0) diff --git a/tests/test_ascend_packed_prefill.py b/tests/test_ascend_packed_prefill.py new file mode 100644 index 00000000..12ddcc5e --- /dev/null +++ b/tests/test_ascend_packed_prefill.py @@ -0,0 +1,56 @@ +"""Ascend attention consumes the common packed KV descriptor.""" + +import pytest +import torch + +from vllm_rlt.attention.backend import AttentionBackend, BackendCapabilities +from vllm_rlt.core.kv_cache_manager import KVCacheManager +from vllm_rlt.device import resolve_device +from vllm_rlt.kernels.ascend_attention import AscendAttention + + +@pytest.mark.parametrize("device_name", ["cpu", pytest.param("npu:0", marks=pytest.mark.npu)]) +@torch.inference_mode() +def test_packed_prefill_mixes_cached_chunks_and_single_queries(device_name): + device = resolve_device(device_name) + if device.type == "npu": + torch.npu.set_device(device) + attention = AttentionBackend( + AscendAttention(), + BackendCapabilities(packed_prefill=True), + {"backend": "ascend_eager", "selected_backend": "ascend_eager"}, + ) + cache = KVCacheManager( + 1, 2, 64, 16, 16, 2, device=device, dtype=torch.bfloat16, attention=attention + ) + for request in ("a", "b", "c"): + assert cache.allocate(request, 3) + + def values(items): + return ( + torch.tensor(items, device=device, dtype=cache.dtype) + .view(-1, 1, 1) + .expand(-1, cache.num_kv_heads, cache.head_dim) + .contiguous() + ) + + history = values([1, 10, 20, 100]) + cache.write( + 0, ["a", "b", "b", "c"], [0, 1, 1, 0], [0, 0, 1, 0], torch.zeros_like(history), history + ) + ids, depths, positions = ["a", "a", "b", "c"], [0, 0, 1, 0], [1, 2, 2, 1] + packed = cache._prepare_batch(ids, depths, positions, packed_prefill=True) + assert packed.cu_seqlens_q.tolist() == [0, 2, 3, 4] + assert packed.context_lengths.tolist() == [3, 3, 2] + assert packed.block_tables.shape[0] == 3 + + new = values([3, 8, 30, 120]) + query = torch.zeros_like(new) + cache._write_prepared(0, packed, torch.zeros_like(new), new) + actual = cache._attend_prepared(0, packed, query) + expected = values([2, 4, 20, 110]) + torch.testing.assert_close(actual, expected, atol=0, rtol=0) + + per_query = cache._prepare_batch(ids, depths, positions, for_write=False) + reference = cache._attend_prepared(0, per_query, query) + torch.testing.assert_close(actual, reference, atol=0, rtol=0) diff --git a/tests/test_ascend_serving.py b/tests/test_ascend_serving.py new file mode 100644 index 00000000..f1edbd09 --- /dev/null +++ b/tests/test_ascend_serving.py @@ -0,0 +1,143 @@ +"""Real socket + spawned NPU worker smoke, opt-in with pinned local weights.""" + +import asyncio +import json +import os +import signal +import socket +import subprocess +import sys +import time +from pathlib import Path + +import pytest + +pytestmark = pytest.mark.npu + + +@pytest.mark.parametrize( + ("backend", "asynchronous"), + [("torch", False), ("ascend_eager", False), ("ascend_eager", True)], + ids=["torch-sync", "ascend-sync", "ascend-async"], +) +def test_checkpoint_http_completions_stream_and_disconnect(tmp_path, backend, asynchronous): + import aiohttp + + source = os.environ.get("VLLM_RLT_OURO_CHECKPOINT") + if not source: + pytest.skip("set VLLM_RLT_OURO_CHECKPOINT to the pinned local Ouro-1.4B checkpoint") + mode = backend + ("-async" if asynchronous else "-sync") + folder = Path(os.environ.get("VLLM_RLT_ASCEND_RESULTS", tmp_path)) / mode + folder.mkdir(parents=True, exist_ok=True) + with socket.socket() as listener: + listener.bind(("127.0.0.1", 0)) + port = listener.getsockname()[1] + command = [ + sys.executable, + "-m", + "vllm_rlt.entrypoints.serve", + "--model", + source, + "--served-model-name", + "ouro-ascend", + "--device", + "npu", + "--dtype", + "bfloat16", + "--attention-backend", + backend, + "--num-blocks", + "64", + "--max-num-seqs", + "2", + "--max-num-batched-tokens", + "32", + "--port", + str(port), + ] + if asynchronous: + command.extend(["--async-scheduling", "--exit-mode", "ouro_delayed", "--single-stream"]) + with (folder / "service.log").open("w") as log: + process = subprocess.Popen( + command, stdout=log, stderr=subprocess.STDOUT, start_new_session=True + ) + + async def exercise(): + url = f"http://127.0.0.1:{port}" + async with aiohttp.ClientSession(timeout=aiohttp.ClientTimeout(total=60)) as client: + deadline = time.monotonic() + 120 + while True: + assert process.poll() is None, (folder / "service.log").read_text() + try: + async with client.get(url + "/health") as response: + if response.status == 200: + break + except aiohttp.ClientConnectionError: + pass + assert time.monotonic() < deadline, "NPU service startup timed out" + await asyncio.sleep(0.25) + + def body(prompt, **extra): + return dict( + model="ouro-ascend", + prompt=prompt, + max_tokens=4, + temperature=0, + min_loops=4, + max_loops=4, + exit_threshold=1, + ignore_eos=True, + **extra, + ) + + async def complete(prompt): + async with client.post(url + "/v1/completions", json=body(prompt)) as response: + result = await response.json() + assert response.status == 200, result + assert result["usage"]["completion_tokens"] == 4 + return result + + prompts = ["The capital of France is", "One plus one equals"] + normal = await asyncio.gather(*(complete(prompt) for prompt in prompts)) + events = [] + async with client.post( + url + "/v1/completions", + json=body(prompts[0], stream=True, stream_options={"include_usage": True}), + ) as response: + assert response.status == 200 + async for line in response.content: + if line.startswith(b"data: "): + data = line[6:].strip().decode() + events.append(data if data == "[DONE]" else json.loads(data)) + assert events[-1] == "[DONE]" + text = "".join( + choice["text"] for event in events[:-1] for choice in event.get("choices", []) + ) + assert text == normal[0]["choices"][0]["text"] + assert any( + isinstance(event, dict) + and (event.get("usage") or {}).get("completion_tokens") == 4 + for event in events + ) + # Close a real streaming connection early, then verify the worker + # remains usable. Direct engine tests check page reclamation. + async with client.post( + url + "/v1/completions", json=body(prompts[1], stream=True) + ) as response: + assert response.status == 200 + await response.content.readline() + after_disconnect = await complete(prompts[1]) + assert after_disconnect["choices"][0]["text"] == normal[1]["choices"][0]["text"] + return dict(normal=normal, stream=events, after_disconnect=after_disconnect) + + try: + result = asyncio.run(exercise()) + (folder / "service-results.json").write_text(json.dumps(result, indent=2)) + finally: + process.terminate() + try: + process.wait(timeout=30) + except subprocess.TimeoutExpired: + os.killpg(process.pid, signal.SIGKILL) + process.wait(timeout=10) + assert process.returncode == 0, (folder / "service.log").read_text() diff --git a/tests/test_attention_module.py b/tests/test_attention_module.py index 5c4de6dd..823cf322 100644 --- a/tests/test_attention_module.py +++ b/tests/test_attention_module.py @@ -155,3 +155,73 @@ def stage(values, dtype): assert len({batch.cu_seqlens_q.data_ptr() for batch in batches}) == 1 assert batches[0].cu_seqlens_q.tolist() == [0, 2, 3] assert batches[0].max_seqlen_q == 2 + + +def test_prepared_attention_keeps_the_legacy_kernel_signature(): + cache = KVCacheManager(1, 1, 4, 4, 2, 1) + assert cache.allocate("a", 3) + batch = cache._prepare_batch(["a"] * 3, [0] * 3, [0, 1, 2]) + keys = torch.zeros(3, 1, 4) + values = torch.arange(12, dtype=torch.float32).reshape(3, 1, 4) + query = torch.zeros(3, 2, 4) + cache._write_prepared(0, batch, keys, values) + kernel = cache.attention.kernel + expected = kernel( + query, + cache.key_cache[:, 0], + cache.value_cache[:, 0], + batch.block_tables, + batch.context_lengths, + ) + seen = [] + + def legacy(*args): + seen.append(args) + return kernel(*args) + + cache.attention.kernel = legacy + actual = cache._attend_prepared(0, batch, query) + torch.testing.assert_close(actual, expected, atol=0, rtol=0) + assert len(seen) == 1 and len(seen[0]) == 5 + assert seen[0][0] is query + assert seen[0][3] is batch.block_tables + assert seen[0][4] is batch.context_lengths + cache.free("a") + + +def test_metadata_attention_borrows_only_a_live_validated_descriptor(): + cache = KVCacheManager(1, 1, 4, 4, 2, 1) + assert cache.allocate("a", 3) + batch = cache._prepare_batch(["a"] * 3, [0] * 3, [0, 1, 2]) + keys = torch.zeros(3, 1, 4) + values = torch.arange(12, dtype=torch.float32).reshape(3, 1, 4) + query = torch.zeros(3, 2, 4) + cache._write_prepared(0, batch, keys, values) + legacy = cache.attention.kernel + expected = legacy( + query, + cache.key_cache[:, 0], + cache.value_cache[:, 0], + batch.block_tables, + batch.context_lengths, + ) + seen = [] + + class MetadataKernel: + def __call__(self, *args): + pytest.fail("metadata-aware kernels must not read device metadata back") + + def attend_metadata(self, q, k, v, descriptor): + seen.append(descriptor) + return legacy(q, k, v, descriptor.block_tables, descriptor.context_lengths) + + cache.attention.kernel = MetadataKernel() + actual = cache._attend_prepared(0, batch, query) + torch.testing.assert_close(actual, expected, atol=0, rtol=0) + assert len(seen) == 1 and seen[0] is batch + cache.free("a") + assert cache.allocate("a", 3) + with pytest.raises((ValueError, RuntimeError)): + cache._attend_prepared(0, batch, query) + assert len(seen) == 1 + cache.free("a") diff --git a/tests/test_cdb_runtime.py b/tests/test_cdb_runtime.py index b6e224f6..740572f0 100644 --- a/tests/test_cdb_runtime.py +++ b/tests/test_cdb_runtime.py @@ -650,7 +650,15 @@ def test_cuda_dynamic_async_respects_buffers_and_matches_sync(layout, multi_stre expected = LLM(base, **common).generate(prompts, params) execution = ExecutionConfig(async_scheduling=True, multi_stream=multi_stream) actual = LLM(base, execution_config=execution, **common) - assert actual.engine.execution_config == execution + # Automatic fields resolve at engine construction; other switches must survive. + assert ( + replace( + actual.engine.execution_config, + cuda_graph_max_batch_size=execution.cuda_graph_max_batch_size, + packed_prefill=execution.packed_prefill, + ) + == execution + ) assert actual.engine.model_runner.requestspaces == {} assert actual.engine.model_runner.states is actual.engine.model_runner.async_state.hidden result = actual.generate(prompts, params) diff --git a/tests/test_device.py b/tests/test_device.py new file mode 100644 index 00000000..c33f8282 --- /dev/null +++ b/tests/test_device.py @@ -0,0 +1,83 @@ +"""Device bootstrap and fail-fast NPU scope without a device dependency.""" + +import importlib +from types import SimpleNamespace + +import pytest +import torch + +from tests.helpers import tiny_ouro_config +from vllm_rlt import CacheConfig, ExecutionConfig, ExitConfig, SchedulerConfig +from vllm_rlt.core.memory import plan_cache +from vllm_rlt.device import resolve_device, stream_module +from vllm_rlt.engine.stage_engine import StageEngine +from vllm_rlt.profiling import ProfileConfig + + +def test_cpu_device_resolution_does_not_import_optional_backend(monkeypatch): + monkeypatch.setattr( + importlib, "import_module", lambda name: pytest.fail(f"unexpected import: {name}") + ) + assert resolve_device("cpu") == torch.device("cpu") + assert resolve_device("cuda:2") == torch.device("cuda:2") + assert stream_module("cpu") is None + assert stream_module("cuda:2") is torch.cuda + + +def npu_model_stub(**overrides): + parameter = SimpleNamespace( + device=SimpleNamespace(type="npu"), dtype=torch.bfloat16, element_size=lambda: 2 + ) + return SimpleNamespace( + parameters=lambda: iter([parameter]), config=tiny_ouro_config(**overrides) + ) + + +@pytest.mark.parametrize( + "options", + [ + dict(execution_config=ExecutionConfig(cuda_graphs=True)), + dict(cache_config=CacheConfig(num_blocks=32, enable_prefix_caching=True)), + dict(scheduler_config=SchedulerConfig(enable_preemption=True)), + dict(exit_config=ExitConfig(mode="random_lookahead")), + ], +) +def test_npu_unsupported_execution_rejected_before_allocation(options): + with pytest.raises(ValueError, match="NPU Ouro requires local execution"): + StageEngine(npu_model_stub(), **options) + + +@pytest.mark.parametrize("backend", ["torch", "ascend_eager"]) +def test_npu_requires_bf16_kv_before_allocation(backend): + with pytest.raises(ValueError, match="NPU Ouro requires BF16 KV cache"): + StageEngine( + npu_model_stub(), + cache_config=CacheConfig(kv_cache_dtype="fp8_e4m3"), + attention_backend=backend, + ) + + +def test_npu_does_not_use_cpu_default_memory_budget(): + model = npu_model_stub() + with pytest.raises(ValueError, match="explicit num_blocks"): + plan_cache(model, CacheConfig(), SchedulerConfig(), ExecutionConfig(), "torch") + blocks, info = plan_cache( + model, CacheConfig(num_blocks=17), SchedulerConfig(), ExecutionConfig(), "torch" + ) + assert blocks == 17 and info["source"] == "blocks" + + +@pytest.mark.parametrize("backend", ["auto", "triton", "flash_attn_4"]) +def test_npu_requires_explicit_torch_before_allocation(backend): + with pytest.raises(ValueError, match="explicit attention_backend='torch'"): + StageEngine(npu_model_stub(), attention_backend=backend) + + +def test_builtin_profiler_does_not_mislabel_npu_as_cpu_only(tmp_path): + with pytest.raises(ValueError, match="torch_npu.profiler"): + ProfileConfig(str(tmp_path)).resolve_activities("npu:0") + + +def test_npu_async_requires_explicit_delayed_exit_policy(): + with pytest.raises(ValueError, match="async scheduling requires ouro_delayed"): + StageEngine(npu_model_stub(), execution_config=ExecutionConfig(async_scheduling=True)) diff --git a/tests/test_fast_execution_plan.py b/tests/test_fast_execution_plan.py new file mode 100644 index 00000000..85bd0528 --- /dev/null +++ b/tests/test_fast_execution_plan.py @@ -0,0 +1,103 @@ +"""Ownership and failure boundaries of Ascend recurrent execution.""" + +import pytest +import torch + +from tests.test_ascend import make_npu_model +from vllm_rlt.core.kv_cache_manager import KVCacheManager +from vllm_rlt.device import resolve_device +from vllm_rlt.request import Operation +from vllm_rlt.worker.executor import ForwardBatch, ModelRuntime + +pytestmark = pytest.mark.npu + + +def cache_for(model): + c = model.config + cache = KVCacheManager( + c.num_hidden_layers, + c.num_key_value_heads, + c.head_dim, + 32, + 16, + 4, + device="npu:0", + dtype=torch.bfloat16, + backend="ascend_eager", + ) + assert cache.allocate("a", 8) + return cache + + +def metadata(cache, start, count): + return cache._prepare_batches( + ["a"] * count, + [[depth] * count for depth in range(4)], + list(range(start, start + count)), + ) + + +@torch.inference_mode() +def test_failed_traversal_keeps_partial_layer_coverage_visible(): + resolve_device("npu") + torch.npu.set_device(0) + model = make_npu_model(torch.device("npu:0")) + cache = cache_for(model) + runtime = ModelRuntime(model, cache) + batch = metadata(cache, 0, 3)[0] + + def fail(*args): + raise RuntimeError("injected layer failure") + + handle = model.model.layers[1].register_forward_pre_hook(fail) + try: + hidden = model.prelude(torch.tensor([3, 7, 11], device="npu:0")) + with pytest.raises(RuntimeError, match="injected layer failure"): + runtime.forward(ForwardBatch(Operation.RECURRENT, hidden=hidden, metadata=(batch,))) + finally: + handle.remove() + with pytest.raises(RuntimeError, match="every layer"): + cache.finalize_token("a", 1, 0) + q = torch.zeros( + 3, + model.config.num_attention_heads, + model.config.head_dim, + device="npu:0", + dtype=torch.bfloat16, + ) + assert torch.isfinite(cache._attend_prepared(0, batch, q)).all() + with pytest.raises(RuntimeError, match="uninitialized KV history"): + cache._attend_prepared(1, batch, q) + # Rerun all layers for the same page slots after the interrupted write. + actual = runtime.forward( + ForwardBatch(Operation.RECURRENT, hidden=hidden, metadata=(batch,)) + ).hidden + reference = cache_for(model) + expected = model.recurrent_prepared(hidden, metadata(reference, 0, 3)[0], reference)[0] + torch.testing.assert_close(actual, expected, atol=0, rtol=0) + cache.free("a") + reference.free("a") + + +@torch.inference_mode() +def test_paged_coverage_after_truncation_and_restore(): + resolve_device("npu") + torch.npu.set_device(0) + model = make_npu_model(torch.device("npu:0")) + cache = cache_for(model) + runtime = ModelRuntime(model, cache) + hidden = model.prelude(torch.tensor([3, 7, 11], device="npu:0")) + for batch in metadata(cache, 0, 3): + runtime.forward(ForwardBatch(Operation.RECURRENT, hidden=hidden, metadata=(batch,))) + cache.finalize_token("a", 2, 3) + snapshot = cache.snapshot("a") + cache.truncate_suffix("a", 1) + with pytest.raises(RuntimeError, match="every layer"): + cache.finalize_token("a", 2, 3) + cache.truncate_suffix("a", 0) + cache.restore("a", snapshot) + cache.finalize_token("a", 2, 3) + cache.truncate_suffix("a", 1) + with pytest.raises(RuntimeError, match="every layer"): + cache.finalize_token("a", 2, 3) + cache.free("a") diff --git a/tests/test_npu_async.py b/tests/test_npu_async.py new file mode 100644 index 00000000..52ba27de --- /dev/null +++ b/tests/test_npu_async.py @@ -0,0 +1,238 @@ +"""NPU event ownership and device-token lookahead preserve eager execution.""" + +import pytest +import torch + +from tests.test_ascend import make_npu_model +from vllm_rlt import CacheConfig, ExecutionConfig, ExitConfig, SamplingParams, SchedulerConfig +from vllm_rlt.core.scheduler import ScheduledRequest, Scheduler, SchedulerOutput +from vllm_rlt.device import resolve_device +from vllm_rlt.engine.stage_engine import StageEngine +from vllm_rlt.request import Request, RequestPhase + + +def test_fixed_depth_decode_keeps_each_loop_schedulable(): + scheduler = Scheduler(SchedulerConfig(), None, total_ut_steps=4, exit_mode="ouro_delayed") + request = Request("a", [1], SamplingParams(min_loops=1, max_loops=4)) + request.phase = RequestPhase.RECURRENT + scheduler.requests["a"] = request + for depth in range(4): + assert request.phase == RequestPhase.RECURRENT + item = ScheduledRequest(request, depth=depth, phase=RequestPhase.RECURRENT) + assert scheduler._planned_loop_count([item]) == 1 + scheduler.update_after_submission(SchedulerOutput(RequestPhase.RECURRENT, [item], 1)) + assert request.loops_done == depth + 1 + assert request.phase == RequestPhase.CODA + + +def make_engine(model, asynchronous, multi_stream): + return StageEngine( + model, + attention_backend="ascend_eager", + cache_config=CacheConfig(num_blocks=128, block_size=16, incremental_allocation=True), + scheduler_config=SchedulerConfig( + max_num_seqs=3, max_num_batched_tokens=128, prefill_chunk_size=128 + ), + execution_config=ExecutionConfig( + async_scheduling=asynchronous, + multi_stream=multi_stream, + static_buffers=False, + pad_to_power_of_two=False, + use_uva=False, + cuda_graphs=False, + ), + exit_config=ExitConfig("ouro_delayed" if asynchronous else "ouro"), + ) + + +def drain(engine): + got = {} + for _ in range(300): + for output in engine.step(): + if output.finished: + got[output.request_id] = ( + output.token_ids, + output.exit_depths, + output.finish_reason, + ) + if not engine.has_unfinished_requests(): + assert engine.cache_manager.num_used_blocks == 0 + return got + pytest.fail("NPU async engine did not drain") + + +@pytest.mark.npu +@pytest.mark.parametrize("multi_stream", [False, True]) +@torch.inference_mode() +def test_async_npu_matches_sync_without_device_barriers(monkeypatch, multi_stream): + resolve_device("npu") + torch.npu.set_device(0) + model = make_npu_model(torch.device("npu:0")) + prompts = [[3, 5, 7] * 7, [2, 11, 17], [13, 19] * 9] + params = SamplingParams(max_tokens=4, min_loops=4, max_loops=4, ignore_eos=True) + reference = None + for asynchronous in (False, True): + engine = make_engine(model, asynchronous, multi_stream) + try: + for repeat in range(2): + for i, prompt in enumerate(prompts): + engine.add_request(str(i), prompt, params) + with monkeypatch.context() as patch: + + def forbidden(*args, **kwargs): + pytest.fail("submission called device-wide synchronize") + + patch.setattr(torch.npu, "synchronize", forbidden) + actual = drain(engine) + assert all(depths == [4] * 4 for _, depths, _ in actual.values()) + if reference is None: + reference = actual + else: + assert actual == reference + finally: + engine.close() + + +@pytest.mark.npu +@torch.inference_mode() +def test_npu_delayed_delivery_consumes_device_token_before_host_result(): + resolve_device("npu") + torch.npu.set_device(0) + model = make_npu_model(torch.device("npu:0")) + engine = make_engine(model, True, False) + params = SamplingParams(max_tokens=3, min_loops=4, max_loops=4, ignore_eos=True) + + class HeldEvent: + def query(self): + return False + + def synchronize(self): + pytest.fail("host delivery was required before the next device input") + + try: + engine.add_request("a", [3, 7, 11], params) + engine.step() + engine.step() + ticket = engine._pending_coda[0] + real_event = ticket.event + ticket.event = HeldEvent() + try: + request = engine.scheduler.requests["a"] + assert request.generated_token_ids == [] + engine.step() + assert engine.last_schedule.phase == RequestPhase.PRELUDE + engine.step() + assert engine.last_schedule.phase == RequestPhase.RECURRENT + assert request.loops_done == 4 + assert request.generated_token_ids == [] + assert ticket.device_values.device.type == "npu" + finally: + ticket.event = real_event + assert drain(engine)["a"][1] == [4, 4, 4] + finally: + engine.close() + + +@pytest.mark.npu +@torch.inference_mode() +def test_npu_cancel_reuse_rejects_late_coda_and_eos_stops(): + resolve_device("npu") + torch.npu.set_device(0) + model = make_npu_model(torch.device("npu:0")) + params = SamplingParams(max_tokens=3, min_loops=4, max_loops=4, ignore_eos=True) + expected_engine = make_engine(model, False, False) + expected_engine.add_request("cancel", [13, 17, 19], params) + expected_engine.add_request("keep", [3, 7, 11], params) + expected = drain(expected_engine) + expected_engine.close() + engine = make_engine(model, True, True) + try: + engine.add_request("cancel", [2, 5] * 9, params) + engine.add_request("keep", [3, 7, 11], params) + engine.step() + engine.step() + engine.abort_request("cancel") + engine.add_request("cancel", [13, 17, 19], params) + assert drain(engine) == expected + eos = model.config.eos_token_id + engine.model_runner.sampler.sample_batch = lambda logits, options, generators: ( + logits.new_full((len(options),), eos, dtype=torch.long) + ) + engine.add_request("eos", [3, 7], SamplingParams(max_tokens=4, min_loops=4, max_loops=4)) + assert drain(engine)["eos"] == ([eos], [4], "stop") + finally: + engine.close() + + +@pytest.mark.npu +def test_npu_batched_greedy_preserves_ties_and_padding(): + from vllm_rlt.worker.ascend_model_runner import AscendSampler + + device = resolve_device("npu:0") + logits = torch.zeros((3, 49152), dtype=torch.bfloat16) + logits[0, 1] = 3 + logits[0, 40000] = 3 + logits[1, 0] = 4 + logits[1, -1] = 4 + logits[2, 0] = 9 + logits = logits.to(device) + params = (SamplingParams(), SamplingParams()) + actual = AscendSampler().sample_batch(logits, params, (None, None)) + expected = torch.stack([row.argmax() for row in logits[:2]]) + assert actual.cpu().tolist() == [1, 0] + assert torch.equal(actual, expected) + + +@pytest.mark.npu +@torch.inference_mode() +def test_npu_hidden_gather_keeps_cross_stream_sources_alive(): + from types import SimpleNamespace + + from vllm_rlt.worker.model_runner import ModelRunner + + device = resolve_device("npu:0") + torch.npu.set_device(device) + producer = torch.npu.Stream(device=device) + consumer = torch.npu.Stream(device=device) + blocker = torch.npu.Stream(device=device) + with torch.npu.stream(producer): + source = torch.full((64,), 7.0, dtype=torch.bfloat16, device=device) + producer.synchronize() + with torch.npu.stream(consumer): + warm = torch.stack([source]) + consumer.synchronize() + del warm + pointer = source.data_ptr() + request = SimpleNamespace(request_id="a", hidden_state=source) + batch = SimpleNamespace(spans=[SimpleNamespace(request=request)]) + matrix = torch.ones((16384, 16384), dtype=torch.bfloat16, device=device) + product = torch.empty_like(matrix) + blocker.wait_stream(torch.npu.current_stream(device)) + with torch.npu.stream(blocker): + for _ in range(4): + torch.mm(matrix, matrix, out=product) + gate = torch.npu.Event() + gate.record(blocker) + consumer.wait_event(gate) + runner = SimpleNamespace(events={}) + try: + with torch.npu.stream(consumer): + ModelRunner._wait_for_requests(runner, batch, consumer) + actual = ModelRunner._gather(runner, [request], None, 1) + request.hidden_state = None + del source + poisoned = [] + with torch.npu.stream(producer): + for _ in range(32): + value = torch.empty((64,), dtype=torch.bfloat16, device=device) + value.fill_(99.0) + poisoned.append(value) + assert not gate.query(), "consumer delay expired before the source reuse check" + assert all(value.data_ptr() != pointer for value in poisoned) + producer.synchronize() + consumer.synchronize() + assert actual.cpu().tolist() == [[7.0] * 64] + finally: + producer.synchronize() + consumer.synchronize() + blocker.synchronize() diff --git a/tests/test_npu_batch_decode.py b/tests/test_npu_batch_decode.py new file mode 100644 index 00000000..6c5dac9e --- /dev/null +++ b/tests/test_npu_batch_decode.py @@ -0,0 +1,236 @@ +"""Decode scheduling and KV completion events preserve request lifetimes.""" + +from contextlib import nullcontext + +import pytest +import torch + +from tests.helpers import make_cpu_cache, tiny_ouro_model +from vllm_rlt import ExecutionConfig, SamplingParams, SchedulerConfig +from vllm_rlt.request import Request +from vllm_rlt.worker.ascend_model_runner import AscendModelRunner +from vllm_rlt.worker.model_runner import ModelRunner + + +def runner_cpu(): + model = tiny_ouro_model() + return ModelRunner( + model, + make_cpu_cache(model.config), + execution_config=ExecutionConfig( + async_scheduling=True, + multi_stream=False, + static_buffers=False, + pad_to_power_of_two=False, + use_uva=False, + ), + scheduler_config=SchedulerConfig(max_num_seqs=3, max_num_batched_tokens=8), + ) + + +def requests(): + return [Request(x, [1], SamplingParams(max_tokens=3)) for x in ["a", "b"]] + + +def test_cpu_keeps_copy_semantics(): + runner = runner_cpu() + reqs = requests() + hidden = torch.arange(64.0).reshape(2, 32) + runner._save_batch(reqs, hidden, None) + expected = reqs[0].hidden_state.clone() + hidden.fill_(99) + assert torch.equal(reqs[0].hidden_state, expected) + + +class Event: + def __init__(self): + self.recorded = [] + + def record(self, stream): + self.recorded.append(stream) + + +class Stream: + def __init__(self): + self.waited = [] + self.synchronized = 0 + + def wait_event(self, event): + self.waited.append(event) + + def synchronize(self): + self.synchronized += 1 + + +class DeviceOps: + def __init__(self): + self.created = [] + + def stream(self, stream): + return nullcontext() + + def Event(self): + e = Event() + self.created.append(e) + return e + + +def write_depth(runner, reqs, depth, *, layers=None): + cache = runner.cache_manager + for r in reqs: + assert cache.allocate(r.request_id, 4) + r.loops_done = depth + 1 + c = runner.model.config + data = torch.arange( + len(reqs) * c.num_key_value_heads * c.head_dim, dtype=torch.float32 + ).reshape(len(reqs), c.num_key_value_heads, c.head_dim) + for layer in range(cache.num_layers) if layers is None else layers: + cache.write( + layer, + [r.request_id for r in reqs], + [depth] * len(reqs), + [0] * len(reqs), + data, + data + 1, + ) + return data + + +def fake_events(runner, reqs): + runner.boundary_stream = Stream() + runner.device_ops = DeviceOps() + source = Event() + runner.events = {r.request_id: source for r in reqs} + return source + + +def test_full_depth_finalize_reuses_event_and_still_validates(): + runner = runner_cpu() + reqs = requests() + write_depth(runner, reqs, 3) + source = fake_events(runner, reqs) + AscendModelRunner._finalize_npu_batch(runner, reqs) + assert all(runner.events[r.request_id] is source for r in reqs) + assert not runner.device_ops.created and not runner.boundary_stream.waited + bad = runner_cpu() + rs = requests() + write_depth(bad, rs, 3, layers=[0]) + fake_events(bad, rs) + with pytest.raises(RuntimeError, match="every layer"): + AscendModelRunner._finalize_npu_batch(bad, rs) + assert not bad.device_ops.created + + +def test_early_exit_finalize_copies_all_planes_then_shares_event(): + runner = runner_cpu() + reqs = requests() + data = write_depth(runner, reqs, 0) + source = fake_events(runner, reqs) + AscendModelRunner._finalize_npu_batch(runner, reqs) + assert runner.boundary_stream.waited == [source] + assert len(runner.device_ops.created) == 1 + done = runner.device_ops.created[0] + assert done.recorded == [runner.boundary_stream] + assert all(runner.events[r.request_id] is done for r in reqs) + for i, r in enumerate(reqs): + for depth in range(1, 4): + for layer in range(runner.cache_manager.num_layers): + k, v = runner.cache_manager.read(layer, r.request_id, depth, 1) + assert torch.equal(k[0], data[i]) and torch.equal(v[0], data[i] + 1) + + +def test_partial_finalize_failure_drains_copies_before_cleanup(): + runner = runner_cpu() + reqs = requests() + write_depth(runner, reqs, 0) + fake_events(runner, reqs) + # Keep allocation live but make the second row lack a required layer. + runner.cache_manager.truncate_suffix("b", 0) + with pytest.raises(RuntimeError, match="every layer"): + AscendModelRunner._finalize_npu_batch(runner, reqs) + assert runner.boundary_stream.synchronized == 1 + assert not runner.device_ops.created + + +@pytest.mark.npu +@pytest.mark.parametrize("multi_stream", [False, True]) +@torch.inference_mode() +def test_npu_async_decode_matches_sync_when_reusing_request_ids(multi_stream): + from tests.test_ascend import make_npu_model + from tests.test_npu_async import drain, make_engine + from vllm_rlt.device import resolve_device + + device = resolve_device("npu:0") + torch.npu.set_device(device) + model = make_npu_model(device) + baseline = make_engine(model, False, multi_stream) + params = SamplingParams(max_tokens=4, min_loops=4, max_loops=4, ignore_eos=True) + prompts = [[3, 7, 11] * 7, [2, 5, 13]] + for i, p in enumerate(prompts): + baseline.add_request(str(i), p, params) + expected = drain(baseline) + baseline.close() + engine = make_engine(model, True, multi_stream) + try: + for repeat in range(2): + for i, p in enumerate(prompts): + engine.add_request(str(i), p, params) + assert drain(engine) == expected + finally: + engine.close() + + +@pytest.mark.npu +@pytest.mark.parametrize("multi_stream", [False, True]) +@torch.inference_mode() +def test_npu_partial_depth_finalize_matches_per_request_path(multi_stream): + # Serving still requires four loops. Test generic propagation directly. + from tests.test_ascend import make_npu_model + from tests.test_npu_async import make_engine + from vllm_rlt.device import resolve_device + + device = resolve_device("npu:0") + torch.npu.set_device(device) + model = make_npu_model(device) + results = [] + for optimized in [False, True]: + engine = make_engine(model, True, multi_stream) + runner, cache = engine.model_runner, engine.cache_manager + reqs = [Request(str(i), [1], SamplingParams()) for i in range(3)] + try: + for r in reqs: + assert cache.allocate(r.request_id, 4) + with torch.npu.stream(runner.core_stream): + for i, depth in enumerate([1, 1, 3]): + reqs[i].loops_done = depth + 1 + shape = (1, model.config.num_key_value_heads, model.config.head_dim) + k = torch.full(shape, float(i + 2), device=device, dtype=torch.bfloat16) + for layer in range(cache.num_layers): + cache.write(layer, [str(i)], [depth], [0], k, k + 1) + source = torch.npu.Event() + source.record(runner.core_stream) + runner.events = {r.request_id: source for r in reqs} + if optimized: + runner.finalize_many(reqs) + assert runner.events["0"] is runner.events["1"] + assert runner.events["2"] is source + else: + for r in reqs: + runner.finalize(r) + runner.boundary_stream.synchronize() + actual = {} + for i, first in enumerate([1, 1, 3]): + for depth in range(first, 4): + for layer in range(cache.num_layers): + actual[(i, depth, layer)] = tuple( + t.cpu() for t in cache.read(layer, str(i), depth, 1) + ) + results.append(actual) + finally: + for r in reqs: + runner.release(r.request_id) + cache.free(r.request_id) + engine.close() + assert results[0].keys() == results[1].keys() + for key in results[0]: + assert all(torch.equal(a, b) for a, b in zip(results[0][key], results[1][key])) diff --git a/vllm_rlt/attention/backend.py b/vllm_rlt/attention/backend.py index 6a1bf8bb..c5ebdbfa 100644 --- a/vllm_rlt/attention/backend.py +++ b/vllm_rlt/attention/backend.py @@ -47,6 +47,24 @@ class AttentionBackend: def __call__(self, *args, **kwargs): return self.kernel(*args, **kwargs) + def attend_metadata(self, q, keys, values, batch, **kwargs): + """Use the prepared batch for packed or per-query attention.""" + method = getattr(self.kernel, "attend_metadata", None) + if method is not None: + return method(q, keys, values, batch, **kwargs) + if batch.cu_seqlens_q is not None: + return self.prefill( + q, + keys, + values, + batch.block_tables, + batch.context_lengths, + batch.cu_seqlens_q, + batch.max_seqlen_q, + **kwargs, + ) + return self(q, keys, values, batch.block_tables, batch.context_lengths, **kwargs) + def prefill(self, *args, **kwargs): return self.kernel.prefill(*args, **kwargs) @@ -99,12 +117,23 @@ def backend_capabilities(implementation) -> BackendCapabilities: return implementation.capabilities -def validate_backend_name(backend: str) -> None: - if backend not in {"auto", "torch", "triton", *FLASH_BACKENDS}: +def validate_backend_name(backend: str, *, device_type: str | None = None) -> None: + if device_type == "npu" and backend not in ("torch", "ascend_eager"): + raise ValueError( + "NPU execution requires explicit attention_backend='torch' or 'ascend_eager'" + ) + if backend not in {"auto", "torch", "triton", "ascend_eager", *FLASH_BACKENDS}: raise ValueError("unknown attention backend") def _create_explicit(backend, device, dtype, head_dim, block_size, *, packed, packed_window_shape): + if backend == "ascend_eager": + if device.type != "npu" or dtype != torch.bfloat16: + raise ValueError("ascend_eager requires an NPU and BF16") + from vllm_rlt.kernels.ascend_attention import AscendAttention + + capabilities = BackendCapabilities(packed_prefill=packed, async_scheduling=True) + return AttentionBackend(AscendAttention(), capabilities, {"backend": backend}) if backend == "triton": if device.type != "cuda": raise ValueError("the Triton attention backend requires a CUDA or ROCm device") @@ -169,7 +198,7 @@ def create_backend( Engine callers that can calibrate may satisfy an FP8 KV request with a fixed-scale backend. Cache-only callers must provide a resolved scale. """ - validate_backend_name(backend) + validate_backend_name(backend, device_type=device.type) candidates = _AUTO_PRIORITY if backend == "auto" else (backend,) skipped = [] if backend == "auto" and device.type != "cuda": diff --git a/vllm_rlt/core/kv_cache_manager.py b/vllm_rlt/core/kv_cache_manager.py index 04a9c768..c6187d77 100644 --- a/vllm_rlt/core/kv_cache_manager.py +++ b/vllm_rlt/core/kv_cache_manager.py @@ -12,6 +12,7 @@ import hashlib import struct +import weakref from collections import OrderedDict from collections.abc import Callable, Sequence from copy import deepcopy @@ -27,6 +28,7 @@ create_backend, validate_backend_name, ) +from vllm_rlt.device import resolve_device from vllm_rlt.request import ExecutionBatch from vllm_rlt.worker.buffers import stage_metadata from vllm_rlt.worker.metadata import DeviceKVMetadata, KVMetadata, MetadataBuilder @@ -81,6 +83,20 @@ def add(self, position: int) -> None: self.pending.remove(self.prefix) self.prefix += 1 + def add_range(self, start: int, end: int) -> None: + """Record [start, end), preserving sparse/out-of-order coverage.""" + if start >= end or end <= self.prefix: + return + if start > self.prefix: + self.pending.update(range(start, end)) + return + self.prefix = end + if self.pending: + self.pending = {position for position in self.pending if position >= end} + while self.prefix in self.pending: + self.pending.remove(self.prefix) + self.prefix += 1 + def __contains__(self, position: int) -> bool: return position < self.prefix or position in self.pending @@ -94,6 +110,40 @@ class _Allocation: release_requested: bool = False +@dataclass(frozen=True) +class _SequenceCoverage: + allocation: _Allocation + plane: int + ranges: tuple[tuple[int, int], ...] + + +@dataclass(frozen=True) +class _PreparedCoverage: + allocation_epoch: int + sequences: Sequence[_SequenceCoverage] + + +def _sequence_coverage(rows, plane): + """Collapse query rows once, leaving written state owned by the allocation.""" + grouped = {} + for allocation, depth, position in rows: + key = (id(allocation), plane(depth)) + if key not in grouped: + grouped[key] = (allocation, []) + grouped[key][1].append(position) + sequences = [] + for (_, storage_plane), (allocation, positions) in grouped.items(): + ranges = [] + for position in sorted(positions): + if ranges and position <= ranges[-1][1]: + start, end = ranges[-1] + ranges[-1] = (start, max(end, position + 1)) + else: + ranges.append((position, position + 1)) + sequences.append(_SequenceCoverage(allocation, storage_plane, tuple(ranges))) + return tuple(sequences) + + @dataclass(frozen=True) class KVSnapshot: """What preemption keeps for one request: its pages and its written state. @@ -187,7 +237,7 @@ def __init__( ): raise ValueError("recurrent_layers must be valid cache layer indices") self.storage_depths = max_loops if layout == "last_exited" else 1 - self.device = torch.device(device) + self.device = resolve_device(device) self.dtype = dtype self.attention = attention or create_backend( backend, @@ -230,6 +280,14 @@ def __init__( self.device = self.key_cache.device self._free_blocks = list(reversed(range(num_blocks))) self._allocations: dict[str, _Allocation] = {} + self._allocation_epoch = 0 + # Only the Ascend host-metadata path changes. Weak keys must not keep + # request descriptors or their staged tensor storage alive. + self._prepared_coverage = ( + weakref.WeakKeyDictionary() + if self.device.type == "npu" and self.backend == "ascend_eager" + else None + ) self.enable_prefix_caching = enable_prefix_caching self.incremental_allocation = incremental_allocation # Full allocation already reserves each request's maximum KV capacity. @@ -397,6 +455,7 @@ def allocate(self, request_id: str, max_tokens: int, *, initial_tokens=None, pre for _ in range(self.storage_depths) ], ) + self._allocation_epoch += 1 self.prefix_queries += 1 self.prefix_hits += length return True @@ -416,6 +475,7 @@ def ensure_capacity(self, request_id, tokens): t + tuple(blocks[d * extra : (d + 1) * extra]) for d, t in enumerate(allocation.block_tables) ) + self._allocation_epoch += 1 return True def truncate_suffix(self, request_id: str, frontier: int) -> None: @@ -445,6 +505,7 @@ def free(self, request_id: str) -> None: return allocation = self._allocations.pop(request_id, None) if allocation is not None: + self._allocation_epoch += 1 self._drop_refs(b for table in allocation.block_tables for b in reversed(table)) def pin_transfer(self, request_id: str, transfer_id: str): @@ -730,12 +791,26 @@ def _stage(self, values, dtype): host = torch.tensor(values, dtype=dtype, pin_memory=self.device.type == "cuda") return host.to(self.device, non_blocking=True) - def _require_live_batch(self, batch: KVMetadata | DeviceKVMetadata) -> None: + def _require_live_batch(self, batch: KVMetadata | DeviceKVMetadata): if isinstance(batch, KVMetadata) and batch.owner is not self: raise ValueError("prepared KV batch belongs to a different cache manager") + plan = None + reusable = self._prepared_coverage is not None and isinstance(batch, KVMetadata) + epoch = self._allocation_epoch + if reusable: + plan = self._prepared_coverage.get(batch) + if plan is not None and plan.allocation_epoch == epoch: + return plan for request_id, allocation in batch.allocations: if self._allocations.get(request_id) is not allocation: raise RuntimeError(f"stale prepared KV batch for request {request_id!r}") + if reusable: + sequences = ( + _sequence_coverage(batch.rows, self._plane) if plan is None else plan.sequences + ) + plan = _PreparedCoverage(epoch, sequences) + self._prepared_coverage[batch] = plan + return plan def _validate_tensor(self, tensor, batch_size, name, *, query=False): if tensor.ndim != 3 or tensor.shape[0] != batch_size or tensor.shape[-1] != self.head_dim: @@ -771,10 +846,14 @@ def write( @torch.no_grad() def _write_prepared( - self, layer: int, batch: KVMetadata | DeviceKVMetadata, k: torch.Tensor, v: torch.Tensor + self, + layer: int, + batch: KVMetadata | DeviceKVMetadata, + k: torch.Tensor, + v: torch.Tensor, ) -> None: self._validate_layer(layer) - self._require_live_batch(batch) + coverage = self._require_live_batch(batch) if isinstance(batch, DeviceKVMetadata): self._validate_tensor(k, len(batch.position_ids), "k") self._validate_tensor(v, len(batch.position_ids), "v") @@ -794,12 +873,20 @@ def _write_prepared( raise ValueError("a read-only prepared KV batch cannot be written") self._validate_tensor(k, len(batch.position_ids), "k") self._validate_tensor(v, len(batch.position_ids), "v") - if not batch.rows: - return rows = len(batch.rows) + if not rows: + return self._store_layer(layer, batch.write_blocks, batch.write_offsets, k[:rows], v[:rows]) - for allocation, depth, position in batch.rows: - allocation.written[self._plane(depth)][layer].add(position) + if coverage is None: + for allocation, depth, position in batch.rows: + allocation.written[self._plane(depth)][layer].add(position) + else: + # Advance only this layer, only after BOTH device copies enqueue. + # Keep holes and partial-layer writes observable to later attention. + for sequence in coverage.sequences: + written = sequence.allocation.written[sequence.plane][layer] + for start, end in sequence.ranges: + written.add_range(start, end) def _store_layer(self, layer, blocks, offsets, k, v, *, mask=None, active=None) -> None: """Write one layer's K/V rows, quantizing them for an FP8 pool. @@ -889,7 +976,7 @@ def _attend_prepared( window: int | None = None, ) -> torch.Tensor: self._validate_layer(layer) - self._require_live_batch(batch) + coverage = self._require_live_batch(batch) self._validate_tensor(q, len(batch.position_ids), "q", query=True) if window is not None: if batch.cu_seqlens_q is None: @@ -913,27 +1000,21 @@ def _attend_prepared( ) if not batch.rows: return torch.empty_like(q) - for allocation, depth, position in batch.rows: - self._require_prefix(allocation, layer, depth, position + 1) - if batch.cu_seqlens_q is not None: - return self.attention.prefill( - q, - self.key_cache[:, layer], - self.value_cache[:, layer], - batch.block_tables, - batch.context_lengths, - batch.cu_seqlens_q, - batch.max_seqlen_q, - **self._layer_scales(layer), - **({} if window is None else {"window": window}), - ) - return self.attention( + if coverage is None: + for allocation, depth, position in batch.rows: + self._require_prefix(allocation, layer, depth, position + 1) + else: + for sequence in coverage.sequences: + self._require_prefix( + sequence.allocation, layer, sequence.plane, sequence.ranges[-1][1] + ) + return self.attention.attend_metadata( q, self.key_cache[:, layer], self.value_cache[:, layer], - batch.block_tables, - batch.context_lengths, + batch, **self._layer_scales(layer), + **({} if window is None else {"window": window}), ) @torch.no_grad() diff --git a/vllm_rlt/core/memory.py b/vllm_rlt/core/memory.py index 9b87ca2c..1c52c16c 100644 --- a/vllm_rlt/core/memory.py +++ b/vllm_rlt/core/memory.py @@ -154,6 +154,8 @@ def plan_cache( if cache.kv_cache_memory_bytes is not None: blocks = budget_blocks(cache.kv_cache_memory_bytes, per_block) return blocks, {"source": "bytes", "bytes_per_block": per_block} + if device.type == "npu": + raise ValueError("NPU requires explicit num_blocks or kv_cache_memory_bytes") if device.type != "cuda": return 256, {"source": "cpu_default", "bytes_per_block": per_block} torch.cuda.synchronize(device) diff --git a/vllm_rlt/device.py b/vllm_rlt/device.py new file mode 100644 index 00000000..d33b47a8 --- /dev/null +++ b/vllm_rlt/device.py @@ -0,0 +1,24 @@ +"""Register optional device types only when explicitly requested.""" + +import importlib + +import torch + + +def resolve_device(device) -> torch.device: + if isinstance(device, str) and device.split(":", 1)[0] == "npu": + # torch.device cannot parse private-use names until torch-npu registers + # them. Do this in each spawned worker, without importing it on CPU/CUDA. + importlib.import_module("torch_npu") + return torch.device(device) + + +def stream_module(device): + """Resolve stream/event operations from the owning tensor device.""" + device = resolve_device(device) + if device.type == "cuda": + return torch.cuda + if device.type == "npu": + importlib.import_module("torch_npu") + return torch.npu + return None diff --git a/vllm_rlt/engine/packed_prefill.py b/vllm_rlt/engine/packed_prefill.py index 82eec246..dfb5a486 100644 --- a/vllm_rlt/engine/packed_prefill.py +++ b/vllm_rlt/engine/packed_prefill.py @@ -34,8 +34,9 @@ def packed_prefill_eligible( "with ordinary prefill, without speculation or preemption" ) return False - if device_type != "cuda" or layout != "last_exited": + if device_type not in ("cuda", "npu") or layout != "last_exited": if requested is True: - raise ValueError("packed_prefill requires CUDA packed prefill with last_exited KV") + raise ValueError("packed_prefill requires CUDA or Ascend with last_exited KV") return False - return requested is True or not staged_prefill + # Ascend prepares both ordinary and staged packed metadata on the host. + return requested is True or device_type == "npu" or not staged_prefill diff --git a/vllm_rlt/engine/stage_engine.py b/vllm_rlt/engine/stage_engine.py index ec40c771..3e3866af 100644 --- a/vllm_rlt/engine/stage_engine.py +++ b/vllm_rlt/engine/stage_engine.py @@ -2,6 +2,8 @@ from dataclasses import dataclass, replace from typing import TYPE_CHECKING, cast +import torch + from vllm_rlt.attention import BackendCapabilities, create_backend from vllm_rlt.config import ( CacheConfig, @@ -73,6 +75,27 @@ def __init__( self.execution_config = resolve_cuda_graph_rows( execution_config or ExecutionConfig(), scheduler_config, speculative_config ) + if parameter.device.type == "npu": + if config.to_dict()["model_type"] != "ouro" or config.total_ut_steps != 4: + raise ValueError("NPU currently supports Ouro with four loops") + if parameter.dtype != torch.bfloat16: + raise ValueError("NPU Ouro requires BF16 weights") + if cache_config.kv_cache_dtype != "auto": + raise ValueError("NPU Ouro requires BF16 KV cache") + if ( + self.execution_config.cuda_graphs + or (self.execution_config.async_scheduling and self.execution_config.static_buffers) + or speculative_config is not None + or self.stage_config.role != "prefill_decode" + or scheduler_config.enable_preemption + or cache_config.enable_prefix_caching + or cache_config.layout != "last_exited" + or self.exit_config.mode not in ("ouro", "ouro_delayed") + ): + raise ValueError( + "NPU Ouro requires local execution, ouro/ouro_delayed exit mode and " + "last_exited KV without speculation, graphs, preemption or prefix caching" + ) staged_prefill = requires_staged_prefill( scheduler_config, produces_tokens=self.stage_config.produces_tokens ) @@ -211,7 +234,8 @@ def __init__( packed_sliding_window=require_packed and sliding_window, cuda_graphs=self.execution_config.cuda_graphs, async_scheduling=( - parameter.device.type == "cuda" and self.execution_config.async_scheduling + parameter.device.type in ("cuda", "npu") + and self.execution_config.async_scheduling ), device_context_lengths=self.async_speculative, fp8_kv=cache_config.kv_cache_dtype == "fp8_e4m3", @@ -277,7 +301,12 @@ def __init__( # speculative paths must advance the same per-request slot, and only the # engine knows when a request terminates. self.rng = RngRegistry() - self.model_runner = ModelRunner( + runner_type = ModelRunner + if parameter.device.type == "npu": + from vllm_rlt.worker.ascend_model_runner import AscendModelRunner + + runner_type = AscendModelRunner + self.model_runner = runner_type( model, self.cache_manager, exit_config=self.exit_config, @@ -389,6 +418,10 @@ def _add_request(self, request_id, prompt_token_ids, sampling_params, *, trace_i if any(type(t) is not int or not 0 <= t < config.vocab_size for t in prompt_token_ids): raise ValueError("prompt token IDs must be integers within the model vocabulary") max_loops = params.max_loops or config.total_ut_steps + if next(self.model.parameters()).device.type == "npu" and ( + max_loops != 4 or params.exit_threshold != 1.0 or params.temperature != 0 + ): + raise ValueError("NPU Ouro requests require four loops, exit_threshold=1 and greedy") if max_loops > config.total_ut_steps or params.min_loops > max_loops: raise ValueError("requested loop bounds exceed the model's supported depth") if getattr(self.model, "fixed_loop_count", False) and ( @@ -939,8 +972,7 @@ def _step_async(self): request_events=self.model_runner.events, signals=signals, ) - for row in update.next_prelude: - batch.requests[row].request.input_token_tensor = ticket.device_values[row] + self.model_runner.bind_sampled_tokens(batch, ticket.device_values, update.next_prelude) for row in update.retain_signal: self._pending_exit_signals[batch.requests[row].request.request_id] = SignalHandle( ticket, row diff --git a/vllm_rlt/engine/worker.py b/vllm_rlt/engine/worker.py index 9a2b2805..80e98985 100644 --- a/vllm_rlt/engine/worker.py +++ b/vllm_rlt/engine/worker.py @@ -13,6 +13,7 @@ import torch +from vllm_rlt.device import resolve_device from vllm_rlt.engine.connector import TransferConnector from vllm_rlt.engine.stage_config import StageConfig from vllm_rlt.engine.stage_engine import StageEngine @@ -33,12 +34,16 @@ def __init__( ): if type(max_control_messages) is not int or max_control_messages < 1: raise ValueError("max_control_messages must be a positive integer") - device = torch.device(f"cuda:{device}" if isinstance(device, int) else device) + device = resolve_device(f"cuda:{device}" if isinstance(device, int) else device) torch.set_num_threads(1) if device.type == "cuda": if device.index is None: device = torch.device("cuda", torch.cuda.current_device()) torch.cuda.set_device(device) + elif device.type == "npu": + if device.index is None: + device = torch.device("npu", torch.npu.current_device()) + torch.npu.set_device(device) torch.manual_seed(options["seed"]) self.channel = channel self.max_control_messages = max_control_messages diff --git a/vllm_rlt/entrypoints/cli.py b/vllm_rlt/entrypoints/cli.py index a9b8025c..f05313c1 100644 --- a/vllm_rlt/entrypoints/cli.py +++ b/vllm_rlt/entrypoints/cli.py @@ -7,6 +7,7 @@ import torch from vllm_rlt import LLM, SamplingParams, SchedulerConfig +from vllm_rlt.attention import validate_backend_name from vllm_rlt.entrypoints.runtime_args import ( add_runtime_args, profile_config_from_args, @@ -67,6 +68,7 @@ def main(): "--attention-backend", type=str.lower, choices=[ + "ascend_eager", "auto", "torch", "triton", @@ -100,6 +102,10 @@ def main(): parser.error("--toy uses built-in token ID prompts; omit --prompt") if not args.toy and not args.prompt: parser.error("provide --prompt or use --toy for a CPU smoke test") + try: + validate_backend_name(args.attention_backend, device_type=args.device.split(":", 1)[0]) + except ValueError as exc: + parser.error(str(exc)) profiling = profile_config_from_args(args) if profiling is not None: profiling.resolve_activities(args.device) diff --git a/vllm_rlt/entrypoints/serve.py b/vllm_rlt/entrypoints/serve.py index dc5eef6c..c7fc3c84 100644 --- a/vllm_rlt/entrypoints/serve.py +++ b/vllm_rlt/entrypoints/serve.py @@ -7,6 +7,7 @@ import torch +from vllm_rlt.attention import validate_backend_name from vllm_rlt.config import SchedulerConfig from vllm_rlt.engine.llm_engine import StageClient from vllm_rlt.entrypoints.runtime_args import ( @@ -82,12 +83,13 @@ def main(): parser.add_argument("--tokenizer") parser.add_argument("--tokenizer-revision") parser.add_argument("--served-model-name") - parser.add_argument("--device", choices=["cpu", "cuda"], default="cuda") + parser.add_argument("--device", choices=["cpu", "cuda", "npu"], default="cuda") parser.add_argument("--dtype", choices=["bfloat16", "float32"], default="bfloat16") parser.add_argument( "--attention-backend", type=str.lower, choices=[ + "ascend_eager", "auto", "torch", "triton", @@ -117,6 +119,10 @@ def main(): limits = ServingLimits(**{name: getattr(args, name) for name in defaults}) if args.device == "cpu" and args.attention_backend not in ("auto", "torch"): parser.error("CPU execution requires --attention-backend torch or auto") + try: + validate_backend_name(args.attention_backend, device_type=args.device) + except ValueError as exc: + parser.error(str(exc)) torch.set_num_threads(args.cpu_threads) logging.basicConfig(level=logging.INFO) app = create_app( diff --git a/vllm_rlt/kernels/ascend_attention.py b/vllm_rlt/kernels/ascend_attention.py new file mode 100644 index 00000000..12940520 --- /dev/null +++ b/vllm_rlt/kernels/ascend_attention.py @@ -0,0 +1,332 @@ +# SPDX-License-Identifier: Apache-2.0 +"""Use Torch SDPA on Ascend with the runtime's paged KV cache.""" + +import weakref +from collections import OrderedDict + +import torch +from torch.nn import functional as F + + +def _attend(query, key, value, mask=None, *, is_causal=False): + # Match HF's BNSD inputs. A bool mask lets Torch-NPU dispatch fused SDPA + # for cached chunks and variable-length decode on supported hardware. + groups = query.shape[1] // key.shape[1] + if groups != 1: + key = key.repeat_interleave(groups, dim=1) + value = value.repeat_interleave(groups, dim=1) + return F.scaled_dot_product_attention( + query.contiguous(), + key.contiguous(), + value.contiguous(), + attn_mask=mask, + is_causal=is_causal, + ) + + +def _flat_kv_rows(keys, values): + blocks, block_size, heads, head_dim = keys.shape + block_stride, *row_strides = keys.stride() + row_width = heads * head_dim + if ( + not blocks + or not row_width + or values.shape != keys.shape + or values.stride() != keys.stride() + or tuple(row_strides) != (row_width, head_dim, 1) + or block_stride < block_size * row_width + or block_stride % row_width + ): + return None + rows_per_block = block_stride // row_width + # The view starts at this layer's storage offset. Include the gaps between + # its pages, then gather only rows belonging to the selected layer. + shape = ((blocks - 1) * rows_per_block + block_size, heads, head_dim) + return ( + keys.as_strided(shape, row_strides), + values.as_strided(shape, row_strides), + rows_per_block, + ) + + +def _prefill_geometry(rows): + """Detect a regular grid of equal-length, complete prompt traversals.""" + if not rows or rows[0][2] != 0: + return None + width = rows[-1][2] + 1 + size, remainder = divmod(len(rows), width) + if size < 1 or width < 2 or remainder: + return None + groups = [(id(alloc), depth) for alloc, depth, _ in rows[::width]] + if len(set(groups)) != size or any( + ((id(alloc), depth), pos) != (groups[i // width], i % width) + for i, (alloc, depth, pos) in enumerate(rows) + ): + return None + return size, width + + +class AscendAttention: + """Batch compatible queries; borrow only validated host metadata.""" + + def __init__(self): + self._metadata = None + self._groups = None + self._geometry = None + self._write_slots = None + self._ragged_plan = None + self._signature = None + self._direct_decode = False + self._decode_plan = None + # Read-only layouts may be shared across recurrent depths. Addresses + # stay local to each descriptor, and tensors never cross stream keys. + self._layouts = OrderedDict() + self._layout_bytes = 0 + self._layout_limit = 64 * 1024 * 1024 + self.layout_reuses = 0 + self.batched_prefill_calls = 0 + self.ragged_calls = 0 + self.fallback_calls = 0 + self.continuation_calls = 0 + + @staticmethod + def _stream_key(device): + if device.type == "npu": + # Raw npu_stream resolves an ACL handle and drains the task queue. + # Cache identity only needs the logical, device-scoped stream ID. + return torch.npu.current_stream(device).stream_id + if device.type == "cuda": + return torch.cuda.current_stream(device).cuda_stream + return None + + def _layout(self, key, build): + cached = self._layouts.get(key) + if cached is not None: + self._layouts.move_to_end(key) + self.layout_reuses += 1 + return cached[0] + tensors = build() + size = sum(t.numel() * t.element_size() for t in tensors if t is not None) + if size <= self._layout_limit: + while self._layouts and self._layout_bytes + size > self._layout_limit: + _, (_, old_size) = self._layouts.popitem(last=False) + self._layout_bytes -= old_size + # Bound Python entries too: many tiny short-context plans are common. + if len(self._layouts) == 8: + _, (_, old_size) = self._layouts.popitem(last=False) + self._layout_bytes -= old_size + self._layouts[key] = (tensors, size) + self._layout_bytes += size + return tensors + + def _prepare_decode(self, q, keys, batch, flat_rows, stream): + lengths = tuple(position + 1 for _, _, position in batch.rows) + maximum = max(lengths) + block_size = keys.shape[-3] + + def build(): + positions = torch.arange(maximum, device=q.device) + valid = None + if min(lengths) != maximum: + valid = positions[None, :] < torch.tensor(lengths, device=q.device)[:, None] + return positions // block_size, positions % block_size, valid + + pages, offsets, valid = self._layout( + ("decode", q.device, stream, block_size, lengths), build + ) + rows = len(lengths) + tables = batch.block_tables + if len(tables) != rows: + tables = tables[:rows] + blocks = tables[:, pages].to(torch.int64) + flat_keys, _, rows_per_block = flat_rows + slots = (blocks * rows_per_block + offsets).clamp(0, flat_keys.shape[0] - 1) + if valid is not None: + # Avoid reading unwritten bytes in masked columns. Position zero is + # initialized whenever the cache's prefix check admits this row. + slots = torch.where(valid, slots, slots[:, :1]) + mask = valid[:, None, None, :] if valid is not None else None + self._decode_plan = (slots.reshape(-1), mask, rows, maximum) + + def _decode(self, q, keys, values, flat_rows): + slots, mask, rows, maximum = self._decode_plan + flat_keys, flat_values, _ = flat_rows + kv_heads, head_dim = keys.shape[-2:] + key = flat_keys.index_select(0, slots).reshape(rows, maximum, kv_heads, head_dim) + value = flat_values.index_select(0, slots).reshape(rows, maximum, kv_heads, head_dim) + # Real rows form an identity prefix. Avoid gathering Q and scattering + # the result on every layer. + query = q if len(q) == rows else q[:rows] + query = query.reshape(rows, q.shape[1], 1, head_dim) + output = _attend(query, key.transpose(1, 2), value.transpose(1, 2), mask).squeeze(2) + if len(q) == rows: + return output + result = torch.zeros_like(q) + result[:rows].copy_(output) + return result + + def _plan(self, sequence_keys, lengths, device, *, packed=False): + grouped = {} + for row, (key, length) in enumerate(zip(sequence_keys, lengths, strict=True)): + if length: + grouped.setdefault(key, []).append(row) + planned = [] + for sequence, rows in enumerate(grouped.values()): + maximum = max(lengths[row] for row in rows) + indices = torch.tensor(rows, dtype=torch.long, device=device) + positions = torch.arange(maximum, device=device) + group_lengths = [lengths[row] for row in rows] + initial = group_lengths == list(range(1, maximum + 1)) + row_lengths = torch.tensor(group_lengths, device=device) + mask = ( + positions[None, :] < row_lengths[:, None] if not initial and len(rows) > 1 else None + ) + # Packed table rows follow sequence order. Query indices still + # address the flattened Q and output tensors. + table_row = sequence if packed else rows[0] + planned.append((table_row, indices, positions, mask, initial, row_lengths)) + return planned + + def _forward(self, q, keys, values, tables, groups, *, flat_rows=None, result=None): + if result is None: + result = torch.zeros_like(q) + block_size = keys.shape[-3] + for table_row, indices, positions, mask, initial, _lengths in groups: + blocks = tables[table_row, positions // block_size] + offsets = positions % block_size + if flat_rows is None: + k, v = keys[blocks, offsets], values[blocks, offsets] + else: + flat_keys, flat_values, rows_per_block = flat_rows + slots = blocks.to(torch.int64) * rows_per_block + offsets + slots = slots.clamp(0, flat_keys.shape[0] - 1) + k, v = flat_keys.index_select(0, slots), flat_values.index_select(0, slots) + query = q.index_select(0, indices).transpose(0, 1).unsqueeze(0) + key = k.transpose(0, 1).unsqueeze(0) + value = v.transpose(0, 1).unsqueeze(0) + output = ( + _attend(query, key, value, mask, is_causal=initial and indices.numel() > 1) + .squeeze(0) + .transpose(0, 1) + ) + result.index_copy_(0, indices, output) + return result + + def _batched_prefill(self, q, keys, values, batch, flat_rows, geometry): + size, width = geometry + flat_keys, flat_values, rows_per_block = flat_rows + kv_heads, head_dim = keys.shape[-2:] + if self._write_slots is None: + self._write_slots = ( + batch.write_blocks.to(torch.int64) * rows_per_block + + batch.write_offsets.to(torch.int64) + ).clamp(0, flat_keys.shape[0] - 1) + slots = self._write_slots + key = ( + flat_keys.index_select(0, slots) + .reshape(size, width, kv_heads, head_dim) + .transpose(1, 2) + ) + value = ( + flat_values.index_select(0, slots) + .reshape(size, width, kv_heads, head_dim) + .transpose(1, 2) + ) + query = q.reshape(size, width, q.shape[1], head_dim).transpose(1, 2) + return _attend(query, key, value, is_causal=True).transpose(1, 2).reshape_as(q) + + def _ragged_decode(self, q, keys, values, tables, groups, flat_rows, result): + """One attention call for independent single-row decode groups.""" + flat_keys, flat_values, rows_per_block = flat_rows + block_size, kv_heads, head_dim = keys.shape[-3:] + if self._ragged_plan is None: + indices = torch.cat([group[1] for group in groups]) + lengths = torch.cat([group[5] for group in groups]) + maximum = max(group[2].numel() for group in groups) + positions = torch.arange(maximum, device=q.device) + pages = positions // block_size + offsets = positions % block_size + table_rows = torch.tensor([group[0] for group in groups], device=q.device) + row_tables = tables.index_select(0, table_rows) + blocks = row_tables[:, pages].to(torch.int64) + slots = (blocks * rows_per_block + offsets.to(torch.int64)).clamp( + 0, flat_keys.shape[0] - 1 + ) + rows = indices.numel() + valid = positions.unsqueeze(0) < lengths.unsqueeze(1) + slots = torch.where(valid, slots, slots[:, :1]) + self._ragged_plan = (indices, maximum, valid[:, None, None, :], slots, rows) + indices, maximum, mask, slots, rows = self._ragged_plan + key = flat_keys.index_select(0, slots.reshape(-1)).reshape( + rows, maximum, kv_heads, head_dim + ) + value = flat_values.index_select(0, slots.reshape(-1)).reshape( + rows, maximum, kv_heads, head_dim + ) + query = q.index_select(0, indices).reshape(rows, q.shape[1], 1, head_dim) + output = _attend(query, key.transpose(1, 2), value.transpose(1, 2), mask).squeeze(2) + result.index_copy_(0, indices, output) + return result + + def __call__(self, q, keys, values, tables, lengths): + groups = self._plan([tuple(row) for row in tables.tolist()], lengths.tolist(), q.device) + return self._forward(q, keys, values, tables, groups) + + def attend_metadata(self, q, keys, values, batch): + packed = batch.cu_seqlens_q is not None + flat_rows = _flat_kv_rows(keys, values) + stream = self._stream_key(q.device) + signature = (q.device, q.dtype, q.shape, keys.shape, keys.stride(), values.stride(), stream) + if self._metadata is None or self._metadata() is not batch or self._signature != signature: + # Failed preparation must not leave the previous descriptor key + # pointing at partially replaced groups/addresses. + self._metadata = None + self._signature = None + self._geometry = _prefill_geometry(batch.rows) if len(q) == len(batch.rows) else None + self._write_slots = None + self._ragged_plan = None + self._decode_plan = None + self._direct_decode = ( + flat_rows is not None + and len(batch.rows) >= 2 + and all(position > 0 for _, _, position in batch.rows) + and len({(id(a), depth) for a, depth, _ in batch.rows}) == len(batch.rows) + ) + self._groups = None + if self._direct_decode: + self._prepare_decode(q, keys, batch, flat_rows, stream) + elif flat_rows is None or self._geometry is None: + self._groups = self._plan( + [(id(allocation), depth) for allocation, depth, _ in batch.rows], + [position + 1 for _, _, position in batch.rows], + q.device, + packed=packed, + ) + self._metadata = weakref.ref(batch) + self._signature = signature + if self._direct_decode: + self.ragged_calls += 1 + return self._decode(q, keys, values, flat_rows) + if flat_rows is None: + self.fallback_calls += 1 + return self._forward(q, keys, values, batch.block_tables, self._groups) + if self._geometry is not None: + self.batched_prefill_calls += 1 + return self._batched_prefill(q, keys, values, batch, flat_rows, self._geometry) + ragged, remaining = [], [] + for group in self._groups: + if not group[4] and group[1].numel() == 1: + ragged.append(group) + else: + remaining.append(group) + # A single row gains nothing from batching but pays extra plan construction. + if len(ragged) < 2: + remaining.extend(ragged) + ragged = [] + result = self._forward(q, keys, values, batch.block_tables, remaining, flat_rows=flat_rows) + if ragged: + self.ragged_calls += 1 + result = self._ragged_decode( + q, keys, values, batch.block_tables, ragged, flat_rows, result + ) + return result diff --git a/vllm_rlt/layers/rotary_embedding.py b/vllm_rlt/layers/rotary_embedding.py index 297e98d6..84b52096 100644 --- a/vllm_rlt/layers/rotary_embedding.py +++ b/vllm_rlt/layers/rotary_embedding.py @@ -37,7 +37,10 @@ def __init__(self, head_dim: int, rope_theta: float = 10000.0) -> None: self.register_buffer("inv_freq", self.frequencies(), persistent=False) def frequencies(self, device: torch.device | str | None = None) -> torch.Tensor: - return 1.0 / ( + target = device + if device is not None and torch.device(device).type == "npu": + device = "cpu" # HF creates NPU rotary constants on the host. + frequencies = 1.0 / ( self.rope_theta ** ( torch.arange(0, self.head_dim, 2, dtype=torch.float32, device=device) @@ -45,6 +48,12 @@ def frequencies(self, device: torch.device | str | None = None) -> torch.Tensor: ) ) + return ( + frequencies.to(target) + if device == "cpu" and target is not None and torch.device(target).type == "npu" + else frequencies + ) + def _apply(self, fn, recurse=True): super()._apply(fn, recurse=recurse) # Module.to(dtype) otherwise rounds these constants before forward's diff --git a/vllm_rlt/models/ouro.py b/vllm_rlt/models/ouro.py index 31b37f9c..5041fdf3 100644 --- a/vllm_rlt/models/ouro.py +++ b/vllm_rlt/models/ouro.py @@ -19,6 +19,7 @@ import torch from torch import nn +from vllm_rlt.device import resolve_device from vllm_rlt.kernels import silu_and_mul from vllm_rlt.layers import ( RMSNorm, @@ -309,6 +310,7 @@ def from_pretrained( """ from safetensors import safe_open + device = resolve_device(device) if dtype not in (torch.float32, torch.float16, torch.bfloat16): raise ValueError("dtype must be float32, float16, or bfloat16") folder = Path(path_or_repo).expanduser() diff --git a/vllm_rlt/profiling.py b/vllm_rlt/profiling.py index 41245be8..757bbafd 100644 --- a/vllm_rlt/profiling.py +++ b/vllm_rlt/profiling.py @@ -51,6 +51,8 @@ def __post_init__(self): object.__setattr__(self, "activities", values) def resolve_activities(self, device): + if str(device).startswith("npu"): + raise ValueError("NPU device profiling requires torch_npu.profiler") names = self.activities or (("cpu", "cuda") if str(device).startswith("cuda") else ("cpu",)) values = [getattr(torch.profiler.ProfilerActivity, name.upper()) for name in names] if not set(values) <= torch.profiler.supported_activities(): diff --git a/vllm_rlt/worker/ascend_model_runner.py b/vllm_rlt/worker/ascend_model_runner.py new file mode 100644 index 00000000..efd1c74d --- /dev/null +++ b/vllm_rlt/worker/ascend_model_runner.py @@ -0,0 +1,109 @@ +"""Ascend execution with device-resident state, batched sampling and KV finalization.""" + +import torch + +from vllm_rlt.worker.model_runner import ModelRunner +from vllm_rlt.worker.sampler import Sampler + + +class AscendSampler(Sampler): + """Keep batched greedy dispatch out of the CUDA/ROCm sampling path.""" + + def sample_batch(self, logits, params, generators): + if getattr(self.sample, "__func__", None) is Sampler.sample and all( + item.temperature == 0 for item in params + ): + return logits[: len(params)].argmax(dim=-1) + return torch.stack( + [ + self.sample(row, item, generator) + for row, item, generator in zip(logits, params, generators) + ] + ) + + +class AscendModelRunner(ModelRunner): + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + if self.device.type != "npu": + raise ValueError("AscendModelRunner requires an NPU") + self.sampler = AscendSampler() + + def _prepare_tokens(self, prepared, size): + """Materialize snapshotted token sources on the current compute stream.""" + sources = prepared.token_sources + workspace = prepared.workspace + stream = ( + self.device_ops.current_stream(self.device) if self.device_ops is not None else None + ) + if prepared.routing is not None: + tokens = prepared.routing.gather(tokens=True) + elif all(isinstance(source, torch.Tensor) for source in sources): + tokens = torch.stack(sources) + elif all(isinstance(source, tuple) and len(source) == 1 for source in sources): + ids = [source[0] for source in sources] + tokens = ( + workspace.tokens(ids, size) + if workspace is not None + else torch.tensor(ids, device=self.device, dtype=torch.long) + ) + else: + tokens = torch.cat( + [ + source.reshape(-1) + if isinstance(source, torch.Tensor) + else torch.tensor(source, device=self.device, dtype=torch.long) + for source in sources + ] + ) + if prepared.routing is None and size > len(tokens): + tokens = torch.cat((tokens, tokens.new_zeros(size - len(tokens)))) + for span, source in zip(prepared.batch.spans, sources): + if isinstance(source, torch.Tensor) and source.device.type in ("cuda", "npu"): + source.record_stream(stream) + if span.request.input_token_tensor is source: + span.request.input_token_tensor = None + return tokens + + def _finalize_npu_batch(self, requests): + cache = self.cache_manager + copies = [] + for request in requests: + depth = request.loops_done - 1 + if cache.layout == "shared" or depth + 1 == cache.max_loops: + # This still verifies every required layer. With no destination + # there is no device work: the existing core event is sufficient. + cache.finalize_token(request.request_id, request.position, depth) + else: + copies.append(request) + if not copies: + return + stream = self.boundary_stream + with self.device_ops.stream(stream): + try: + dependencies = { + self.events[request.request_id] + for request in copies + if request.request_id in self.events + } + for event in dependencies: + stream.wait_event(event) + for request in copies: + cache.finalize_token( + request.request_id, request.position, request.loops_done - 1 + ) + done = self.device_ops.Event() + done.record(stream) + except Exception: + # A later request can fail after earlier copies were enqueued. + # Finish those readers before cleanup may recycle their pages. + stream.synchronize() + raise + for request in copies: + self.events[request.request_id] = done + + def finalize_many(self, requests): + if self.execution_config.async_scheduling and self.cache_manager.backend == "ascend_eager": + self._finalize_npu_batch(requests) + else: + super().finalize_many(requests) diff --git a/vllm_rlt/worker/model_runner.py b/vllm_rlt/worker/model_runner.py index 70c251ae..d7312d92 100644 --- a/vllm_rlt/worker/model_runner.py +++ b/vllm_rlt/worker/model_runner.py @@ -9,6 +9,7 @@ from vllm_rlt.config import ExecutionConfig, ExitConfig, SchedulerConfig from vllm_rlt.core.scheduler import SchedulerOutput +from vllm_rlt.device import stream_module from vllm_rlt.request import ExecutionBatch, Operation, Request from vllm_rlt.worker.buffers import ( PackedMetadataBuffer, @@ -167,6 +168,7 @@ def __init__( self.runtime = ModelRuntime(self.model, cache_manager) parameter = next(model.parameters()) self.device = parameter.device + self.device_ops = stream_module(self.device) self.sampler = Sampler() # Sampling RNG state lives with the engine, which shares this registry # between the CODA runner and the speculative runner. @@ -218,20 +220,20 @@ def __init__( self.lookahead_head = head.to(device=self.device, dtype=parameter.dtype).eval() self.lookahead_head.requires_grad_(False) self.core_stream = self.boundary_stream = self.copy_stream = self.readback_stream = None - if self.device.type == "cuda": - self.core_stream = torch.cuda.Stream(device=self.device) + if self.device_ops is not None: + self.core_stream = self.device_ops.Stream(device=self.device) self.boundary_stream = ( - torch.cuda.Stream(device=self.device) + self.device_ops.Stream(device=self.device) if self.execution_config.multi_stream else self.core_stream ) self.copy_stream = ( - torch.cuda.Stream(device=self.device) + self.device_ops.Stream(device=self.device) if self.execution_config.multi_stream else self.core_stream ) self.readback_stream = ( - torch.cuda.Stream(device=self.device) + self.device_ops.Stream(device=self.device) if self.execution_config.multi_stream else self.core_stream ) @@ -241,9 +243,9 @@ def __init__( self.copy_stream, self.readback_stream, ): - stream.wait_stream(torch.cuda.current_stream(self.device)) + stream.wait_stream(self.device_ops.current_stream(self.device)) self.readback_slots = [] - if self.device.type == "cuda": + if self.device_ops is not None: # cudaHostAlloc during submission can serialize streams. Allocate all # readback storage before work starts, retaining leases until consumed. for dtype in (torch.float32, torch.int64): @@ -344,7 +346,7 @@ def __init__( # Buffers allocated after stream creation must be visible there too. for stream in (self.core_stream, self.boundary_stream): if stream is not None: - stream.wait_stream(torch.cuda.current_stream(self.device)) + stream.wait_stream(self.device_ops.current_stream(self.device)) self.async_state = None if use_async_state: @@ -364,7 +366,7 @@ def __init__( self.state_slots = self.async_state.slots self.free_state_slots = self.async_state.free for stream in (self.core_stream, self.boundary_stream, self.copy_stream): - stream.wait_stream(torch.cuda.current_stream(self.device)) + stream.wait_stream(self.device_ops.current_stream(self.device)) def _size(self, count): size = count @@ -612,6 +614,11 @@ def _save_batch(self, requests, hidden, routing): for request in requests: request.hidden_state = self.states[self.state_slots[request.request_id]] + def bind_sampled_tokens(self, batch, values, rows): + """Bind the scheduler-selected sample rows as the next prelude inputs.""" + for row in rows: + batch.requests[row].request.input_token_tensor = values[row] + def _gather(self, requests, workspace, size): if workspace is None: return torch.stack([r.hidden_state for r in requests]) @@ -720,7 +727,9 @@ def _prepare_shared_loop_inputs(self, batch): def _uses_packed_loop_inputs(self, batch): capabilities = self.cache_manager.attention_capabilities return ( - batch.operation == Operation.PRELUDE + # PackedMetadataBuffer expands write addresses with a CUDA kernel. + self.device.type == "cuda" + and batch.operation == Operation.PRELUDE and batch.loop_count and self.cache_manager.layout != "shared" and capabilities.packed_prefill @@ -974,10 +983,10 @@ def prepare_on_gpu(self, prepared): ), ) if prepared.routing is not None: - with torch.cuda.stream(self.copy_stream): + with self.device_ops.stream(self.copy_stream): kv = prepared.routing.transfer(prepared.kv) if self.device.type == "cuda": - torch.cuda.current_stream(self.device).wait_event(prepared.routing.ready_event) + self.device_ops.current_stream(self.device).wait_event(prepared.routing.ready_event) prepared = replace(prepared, kv=kv) elif prepared.kv is not None and prepared.workspace is not None: prepared = replace(prepared, kv=prepared.workspace.upload_metadata(prepared.kv)) @@ -1139,14 +1148,21 @@ def _execute_batch(self, batch: ExecutionBatch, prepared): ) values = selected.float().sigmoid() elif forward.compute_logits: - values = torch.stack( - [ - self.sampler.sample(row, params, generator) - for row, params, generator in zip( - result.logits, prepared.sampling_params, prepared.sampling_generators - ) - ] - ) + if self.device.type == "npu": + values = self.sampler.sample_batch( + result.logits, prepared.sampling_params, prepared.sampling_generators + ) + else: + values = torch.stack( + [ + self.sampler.sample(row, params, generator) + for row, params, generator in zip( + result.logits, + prepared.sampling_params, + prepared.sampling_generators, + ) + ] + ) if prepared.routing is not None: prepared.routing.scatter(values, tokens=True) return values @@ -1168,9 +1184,9 @@ def _readback_slot(self, result): raise RuntimeError("readback buffers exhausted; retire completed submissions") def _synchronize_submission(self, stream, prepared): - if self.device.type != "cuda": + if self.device_ops is None: return - (stream or torch.cuda.current_stream(self.device)).synchronize() + (stream or self.device_ops.current_stream(self.device)).synchronize() if self.readback_stream is not None and self.readback_stream is not stream: self.readback_stream.synchronize() if prepared is not None and prepared.routing is not None: @@ -1208,14 +1224,14 @@ def submit(self, scheduled_batch: SchedulerOutput | None = None, *, batch: Execu self.split_prefill_submission_waits += 1 prepared = self.prepare_on_cpu(batch) source_stream = ( - torch.cuda.current_stream(self.device) if self.device.type == "cuda" else None + self.device_ops.current_stream(self.device) if self.device_ops is not None else None ) if prepared.routing is not None and prepared.routing.imports: self.copy_stream.wait_stream(source_stream) # Imported hidden/token tensors may have been produced by core, # not by the caller's stream. The copy itself must wait for them. self._wait_for_requests(batch, self.copy_stream) - with torch.cuda.stream(stream) if stream is not None else nullcontext(): + with self.device_ops.stream(stream) if stream is not None else nullcontext(): if stream is not None: if self.async_state is None and source_stream != stream: stream.wait_stream(source_stream) @@ -1249,21 +1265,21 @@ def submit(self, scheduled_batch: SchedulerOutput | None = None, *, batch: Execu result = None device_values = result event = compute_done = None - if self.device.type == "cuda": - compute_stream = torch.cuda.current_stream(self.device) - compute_done = torch.cuda.Event() + if self.device_ops is not None: + compute_stream = self.device_ops.current_stream(self.device) + compute_done = self.device_ops.Event() compute_done.record(compute_stream) event = compute_done if result is not None: slot = self._readback_slot(result) readback_stream = self.readback_stream or compute_stream - with torch.cuda.stream(readback_stream): + with self.device_ops.stream(readback_stream): if readback_stream != compute_stream: readback_stream.wait_event(compute_done) result.record_stream(readback_stream) host = slot.storage[: result.numel()].view(result.shape) host.copy_(result, non_blocking=True) - event = torch.cuda.Event() + event = self.device_ops.Event() event.record(readback_stream) slot.event = event result = host @@ -1294,7 +1310,7 @@ def submit(self, scheduled_batch: SchedulerOutput | None = None, *, batch: Execu and prepared.routing is not None and not routing_finish_attempted ): - with torch.cuda.stream(stream) if stream is not None else nullcontext(): + with self.device_ops.stream(stream) if stream is not None else nullcontext(): routing_finish_attempted = True prepared.routing.record_done() finally: @@ -1303,7 +1319,7 @@ def submit(self, scheduled_batch: SchedulerOutput | None = None, *, batch: Execu raise finally: if prepared is not None and prepared.workspace is not None: - with torch.cuda.stream(stream) if stream is not None else nullcontext(): + with self.device_ops.stream(stream) if stream is not None else nullcontext(): prepared.workspace.record_done() def finalize_many(self, requests): @@ -1383,7 +1399,7 @@ def finalize_many(self, requests): def finalize(self, request): # The final core event must precede copies and coda on the boundary stream. stream = self.boundary_stream - with torch.cuda.stream(stream) if stream is not None else nullcontext(): + with self.device_ops.stream(stream) if stream is not None else nullcontext(): event = self.events.get(request.request_id) if stream is not None and event is not None: stream.wait_event(event) @@ -1391,7 +1407,7 @@ def finalize(self, request): request.request_id, request.position, request.loops_done - 1 ) if stream is not None: - event = torch.cuda.Event() + event = self.device_ops.Event() event.record(stream) self.events[request.request_id] = event @@ -1427,6 +1443,9 @@ def release(self, request_id): self.rng.drop(request_id) def synchronize(self): + if self.device.type == "npu": + torch.npu.synchronize(self.device) + return for stream in ( self.core_stream, self.boundary_stream, diff --git a/vllm_rlt/worker/tensors.py b/vllm_rlt/worker/tensors.py index 503dee5c..241f4ec1 100644 --- a/vllm_rlt/worker/tensors.py +++ b/vllm_rlt/worker/tensors.py @@ -90,7 +90,7 @@ def copy_tensor(source, target, *, secondary=None, use_uva=True): def pinned_tensor(shape, dtype, device): """Allocate host staging; CPU execution needs no pinned allocation.""" return torch.empty( - shape, dtype=dtype, device="cpu", pin_memory=torch.device(device).type == "cuda" + shape, dtype=dtype, device="cpu", pin_memory=torch.device(device).type in ("cuda", "npu") )