Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
614ee6c
feat: bootstrap synchronous Ouro execution on Ascend NPU
Nov 2, 2026
dc798cb
feat: expose synchronous BF16 Ouro on Ascend through Torch attention
Nov 2, 2026
629e845
docs: record Ascend validation results and quality blockers
Nov 2, 2026
b571af6
perf: use native Ascend scatter for paged KV writes
Nov 2, 2026
65bf157
feat: add explicit BF16 eager attention on Ascend
Nov 2, 2026
87f9a21
test: guard Ascend BF16 attention rounding contract
Nov 2, 2026
9c6c1cc
fix: keep native KV writes opt-in with Ascend eager
Nov 2, 2026
227e966
docs: record Ascend BF16 contract and measured validation
Nov 2, 2026
3f0bb13
fix: remove Ascend SKU assumptions and clarify backend contracts
Nov 3, 2026
f9cfe1b
docs: distinguish portability checks from historical measurements
Nov 3, 2026
95f4999
test: reuse native checkpoint loading on Ascend
Nov 3, 2026
7b34b4d
docs: simplify the Ascend setup recipe
Nov 3, 2026
3e115df
perf: batch Ascend eager attention and build KV write indices on device
Nov 9, 2026
028f9c6
feat(ascend): add asynchronous eager Ouro execution
Nov 17, 2026
394f3b1
fix(ascend): preserve refill progress with coalesced preludes
Nov 17, 2026
7b4537f
fix(ascend): account for coalesced recurrent work
Nov 17, 2026
fa45eee
docs(ascend): report complete unified workload validation
Nov 17, 2026
6e5392d
docs(ascend): streamline the Ouro recipe
Nov 18, 2026
a3a4e2b
Merge upstream main into the Ascend Ouro contribution
Nov 18, 2026
1a8f080
refactor(ascend): use shared Ouro runtime and paged KV
bjf-frz Nov 18, 2026
c043b36
Merge main into Ascend Ouro contribution
bjf-frz Nov 18, 2026
756f7bf
refactor(ascend): reuse packed prefill and per-loop scheduling
bjf-frz Nov 19, 2026
c611c10
refactor(ascend): use the shared paged KV writer
bjf-frz Nov 19, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,7 @@ a fixed scale at startup. See the

| Checkpoint | Implementation | Documented validation scope | Guide |
| --- | --- | --- | --- |
| Ouro-1.4B | Native Ouro | Existing runtime and benchmark coverage under their recorded configurations | [Runtime](docs/runtime.md), [accuracy](docs/accuracy.md) |
| Ouro-1.4B | Native Ouro | Ascend BF16 full prefill with eager Python execution, paged KV and optional fixed-depth async scheduling. Arbitrary chunked prefill remains unresolved. | [Runtime](docs/runtime.md), [Ascend recipe](docs/recipes/ouro_ascend.md#6-fixed-depth-asynchronous-engine-plan) |
| Ouro-2.6B | Shared Ouro | Shares the Ouro implementation; not separately qualified, and 1.4B quality/performance results are not inherited | [Scope notes](docs/recipes/ouro_thinking.md#6-evidence-and-limitations) |
| Ouro-1.4B-Thinking | Reuses Ouro | Bounded BF16, fixed-four-loop synchronous basic integration | [Shared recipe](docs/recipes/ouro_thinking.md) |
| Ouro-2.6B-Thinking | Reuses Ouro | Bounded BF16, fixed-four-loop synchronous basic integration | [Shared recipe](docs/recipes/ouro_thinking.md) |
Expand Down
138 changes: 138 additions & 0 deletions docs/recipes/ouro_ascend.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,138 @@
# Ouro on Ascend

Run Ouro-1.4B on one Ascend NPU with BF16, greedy decoding and four recurrent
loops. The Python runtime uses Torch's `scaled_dot_product_attention` through
the `ascend_eager` backend, matching HF's SDPA entry point. Torch-NPU selects
the device kernel according to the input shape, mask and hardware.
The example enables the optional asynchronous engine plan.

## Prepare the environment and model

Install the project dependencies in a compatible Torch, Torch-NPU and CANN
environment. Use the checkpoint's config and byte-level tokenizer.

```bash
source /path/to/cann/set_env.sh
source /path/to/npu-environment/bin/activate
python -m pip install --no-deps --no-build-isolation -e .
export MODEL=/path/to/prepared/Ouro-1.4B
export ASCEND_RT_VISIBLE_DEVICES=0 # Replace 0 with your assigned NPU.
```

## Serve fixed-depth requests

This example sizes KV for 32 resident requests with 128 input and 128 output
tokens each. Set `--num-blocks` explicitly and adjust it for the expected
sequence lengths, resident request count and available device memory. For
whole-cohort prefill, the chunk size must fit each prompt and the batched token
budget must fit the sum of prompt lengths in the cohort.

```bash
TASK_QUEUE_ENABLE=2 OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 \
python -m vllm_rlt.entrypoints.serve \
--model "$MODEL" --tokenizer "$MODEL" --served-model-name ouro \
--device npu --dtype bfloat16 --attention-backend ascend_eager \
--mode refill --exit-mode ouro_delayed \
--async-scheduling --single-stream --no-uva \
--block-size 16 --num-blocks 2064 --max-num-seqs 32 \
--max-num-batched-tokens 8192 --prefill-chunk-size 1024 \
--max-prefill-batches-before-decode 32 --host 127.0.0.1 --port 8123
```

For synchronous eager execution, omit `--async-scheduling`.

Wait for `/health` to return HTTP 200, then send a completion. Add
`"stream": true` for server-sent events.

```bash
curl --fail http://127.0.0.1:8123/health
curl --fail http://127.0.0.1:8123/v1/completions \
-H 'Content-Type: application/json' \
-d '{"model":"ouro","prompt":"The capital of France is","max_tokens":8,"temperature":0,"min_loops":4,"max_loops":4,"exit_threshold":1}'
```

<a id="6-fixed-depth-asynchronous-engine-plan"></a>
## Fixed-depth asynchronous engine plan

- **Scheduling.** Stream events track completion and protect buffer and request
lifetimes. Decode PRELUDE and each recurrent loop are separate submissions,
so the scheduler can select a new batch at each boundary.
- **Reuse and batching.** Ascend uses the shared packed-prefill metadata by
default with `ascend_eager`. Use `--no-packed-prefill` to select the per-query
layout. Greedy sampling and compatible prefill are batched. Full prefill
uses causal SDPA.
Cached chunks and ragged decode pass boolean masks to preserve each row's
visible KV range.
- **Paged KV.** Each loop writes to the page pool through PyTorch indexed
assignment using its own block table.
The Ascend backend gathers the visible KV rows for Torch SDPA.

The example uses private, fully reserved `last_exited` KV. Keep graphs,
static buffers, padding, prefix caching, incremental KV allocation, preemption,
speculation and prefill/decode disaggregation disabled in this configuration.
Greedy parity remains unqualified for arbitrary chunked or mixed-length inputs
and adaptive exit.

<a id="8-profile-guided-high-batch-scheduling"></a>
## Performance and correctness

A higher prefill quota lets more requests reach decode together and can delay
decode for existing requests. The scheduler default is 1.

### Full unified W1-W7 validation

This historical run used the C++ host executor and explicit eager attention
on both sides of the comparison. For the SDPA configuration above, compare
against HF SDPA with the same weights, input IDs, output lengths and concurrency.
Check the actual prefill and decode kernels on both sides with a short profile.
Measure throughput, TTFT and TPOT with the profiler disabled. Separate
synchronous and asynchronous vllm-rlt runs to measure the effect of scheduling.

Runtime `7b4537f` used the same budget 8192, chunk 1024 and prefill quota 32 for
all seven [unified execution workloads](../unified_execution.md#protocol-and-comparison-rules).
The comparison ran Ouro-1.4B BF16 with four fixed loops and greedy decoding on
one 64 GiB Ascend 910B3 per run, with CANN 9, Torch/Torch-NPU 2.10 and
Transformers 4.54.1. The reference used official HF eager at revision
`574fa66cb8bf5abdc979642d01cf2b79b16bfab1` with a DynamicCache adapter.

All 42 runs completed with 1,422 successful requests and zero runtime failures.
Each case used three unprofiled repetitions per arm on the same card and CPU
affinity. Each repetition admitted closed-loop traffic for 60 seconds and then
fully drained. Throughput includes drain and excludes startup, compilation and
warmup. Timing uses the in-process engine actor. W1–W5 ignore EOS. W6/W7 use
natural EOS with a 512-token cap and a fixed ShareGPT selection.

| Workload | Input / output / clients | Residents per arm | HF eager tok/s | Engine tok/s | vs HF | Complete output matches |
| --- | --- | ---: | ---: | ---: | ---: | ---: |
| W1 | 128 / 128 / 1 | 1 | 6.334 [5.284–6.496] | 9.464 [9.403–10.394] | +49.42% | 15/15 |
| W2 | 128 / 128 / 32 | 32 | 201.314 [200.099–204.437] | 305.496 [293.376–316.662] | +51.75% | 480/480 |
| W3 | 4096 / 128 / 1 | 1 | 6.254 [6.206–6.268] | 7.569 [6.110–9.215] | +21.03% | 12/12 |
| W4 | 4096 / 128 / 32 | 4 | 21.211 [21.012–21.580] | 31.312 [31.135–31.405] | +47.62% | 120/132 |
| W5 | 1024 / 512 / 8 | 8 | 50.885 [50.234–51.362] | 79.791 [76.843–80.666] | +56.80% | 48/48 |
| W6 | ShareGPT / ≤512 / 8 | 8 | 26.343 [26.060–26.385] | 37.992 [37.906–38.879] | +44.22% | 18/27 |
| W7 | ShareGPT / ≤512 / 32 | 8 | 36.341 [35.576–36.931] | 36.194 [35.808–36.679] | -0.41% | 45/99 |

Rates are medians [min–max]. Resident limits reflect single-device memory
capacity and apply equally to both arms. Complete output checks compare token IDs
and finish reasons against timed or supplemental HF references.

**W1, W2, W3 and W5 passed the covered-input output checks.** W4, W6 and W7
still differ from HF, so their throughput figures are diagnostic. W1–W5 used
owned KV and native dispatch throughout timing. W6/W7 used paged KV and Python
execution. W7 throughput ranges overlap, and its peak allocated memory was
30.06 GiB for the engine versus 19.54 GiB for HF. At W2, mean TPOT fell from
157.04 to 102.73 ms while TTFT p95 rose from 315.98 to 616.33 ms.

## Run regression checks

With an assigned NPU visible and `MODEL` set as above:

```bash
VLLM_RLT_OURO_CHECKPOINT="$MODEL" python -m pytest -q --run-npu \
tests/test_ascend_eager.py tests/test_ascend_kv.py \
tests/test_ascend_decode_scheduling.py \
tests/test_ascend_checkpoint.py tests/test_ascend_serving.py
```

For broader evaluation, see [accuracy](../accuracy.md) and
[serving benchmarks](../serving.md).
5 changes: 4 additions & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,10 @@ license-files = ["LICENSE", "NOTICE"]
[tool.pytest.ini_options]
testpaths = ["tests"]
pythonpath = ["."]
markers = ["gpu: requires an explicitly selected CUDA device and --run-gpu"]
markers = [
"gpu: requires an explicitly selected CUDA device and --run-gpu",
"npu: requires an assigned Ascend device and torch-npu/CANN",
]

[tool.ruff]
line-length = 100
Expand Down
6 changes: 6 additions & 0 deletions tests/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,15 +8,21 @@ def pytest_addoption(parser):
parser.addoption(
"--run-gpu", action="store_true", help="Run CUDA tests on explicitly selected devices"
)
parser.addoption("--run-npu", action="store_true", help="Run Ascend tests on an assigned NPU")


def pytest_collection_modifyitems(config, items):
enabled = config.getoption("--run-gpu")
if enabled and not os.environ.get("CUDA_VISIBLE_DEVICES"):
raise pytest.UsageError("--run-gpu requires explicit CUDA_VISIBLE_DEVICES")
npu_enabled = config.getoption("--run-npu")
if npu_enabled and not os.environ.get("ASCEND_RT_VISIBLE_DEVICES"):
raise pytest.UsageError("--run-npu requires assigned ASCEND_RT_VISIBLE_DEVICES")
for item in items:
if "gpu" in item.keywords and not enabled:
item.add_marker(pytest.mark.skip(reason="set CUDA_VISIBLE_DEVICES and use --run-gpu"))
if "npu" in item.keywords and not npu_enabled:
item.add_marker(pytest.mark.skip(reason="use --run-npu on an assigned NPU"))


@pytest.fixture(autouse=True, scope="session")
Expand Down
170 changes: 170 additions & 0 deletions tests/test_ascend.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,170 @@
"""Opt-in synchronous BF16 Ouro checks using explicit Torch attention on Ascend."""

import pytest
import torch

from tests.helpers import tiny_ouro_config
from vllm_rlt import CacheConfig, SamplingParams, SchedulerConfig
from vllm_rlt.core.kv_cache_manager import KVCacheManager
from vllm_rlt.device import resolve_device
from vllm_rlt.engine.stage_engine import StageEngine
from vllm_rlt.kernels.paged_attention import torch_paged_attention
from vllm_rlt.models import OuroForCausalLM

pytestmark = pytest.mark.npu


@pytest.fixture
def npu_device():
resolve_device("npu")
torch.npu.set_device(0)
yield torch.device("npu:0")
torch.npu.synchronize()
torch.npu.empty_cache()


def assert_bf16_close(actual, expected):
a, b = actual.detach().float().cpu(), expected.detach().float().cpu()
assert torch.isfinite(a).all() and torch.isfinite(b).all()
assert (a - b).norm() / b.norm().clamp_min(1e-8) <= 0.02
torch.testing.assert_close(a, b, atol=0.02, rtol=0.02)


@pytest.mark.parametrize("kv_heads", [1, 2, 4])
def test_torch_attention_on_npu_matches_cpu_prefix_oracle(npu_device, kv_heads):
torch.manual_seed(83)
keys = torch.randn(8, 2, 16, kv_heads, 64, dtype=torch.bfloat16)
values = torch.randn_like(keys)
q = torch.randn(64, 4, 64, dtype=torch.bfloat16)
lengths = torch.tensor([0, 1, 15, 16, 17, 31, 32, 3] * 8, dtype=torch.int32)
tables = torch.tensor([[5, 2], [1, 7]] * 32, dtype=torch.int32)
expected = torch_paged_attention(q, keys[:, 1], values[:, 1], tables, lengths)
actual = torch_paged_attention(
q.to(npu_device),
keys.to(npu_device)[:, 1],
values.to(npu_device)[:, 1],
tables.to(npu_device),
lengths.to(npu_device),
)
assert_bf16_close(actual, expected)
assert torch.count_nonzero(actual[0]).item() == 0


def make_npu_model(device):
torch.manual_seed(41)
config = tiny_ouro_config(hidden_size=256, intermediate_size=512, head_dim=64)
return OuroForCausalLM(config).to(device=device, dtype=torch.bfloat16).eval()


@torch.inference_mode()
def recurrent_states(model, chunks, *, tokens=None, incremental=False):
c = model.config
cache = KVCacheManager(
c.num_hidden_layers,
c.num_key_value_heads,
c.head_dim,
32,
16,
4,
device=next(model.parameters()).device,
dtype=torch.bfloat16,
backend="torch",
incremental_allocation=incremental,
)
tokens = tokens if tokens is not None else ([3, 7, 2, 11, 5, 17, 8] * 3)[:21]
assert cache.allocate("a", len(tokens), initial_tokens=16 if incremental else None)
states, kv, start = {}, {}, 0
try:
for size in chunks:
assert cache.ensure_capacity("a", start + size)
positions = list(range(start, start + size))
hidden = model.prelude(torch.tensor(tokens[start : start + size], device=cache.device))
for depth in range(4):
hidden, _ = model.recurrent(hidden, ["a"] * size, [depth] * size, positions, cache)
states[(start, depth)] = (hidden.clone(), model.coda(hidden).clone())
start += size
assert start == len(tokens)
for depth in range(4):
for layer in range(c.num_hidden_layers):
assert all(cache.token_written("a", p, depth) for p in range(len(tokens)))
kv[(layer, depth)] = tuple(
t.clone() for t in cache.read(layer, "a", depth, len(tokens))
)
assert len({tuple(cache.get_block_table("a", d)) for d in range(4)}) == 4
return states, kv
finally:
cache.free("a")
assert cache.num_used_blocks == 0


def test_npu_chunked_prefill_and_incremental_kv(npu_device):
model = make_npu_model(npu_device)
expected, expected_kv = recurrent_states(model, [19, 1, 1])
incremental, incremental_kv = recurrent_states(model, [19, 1, 1], incremental=True)
for key in expected:
for a, b in zip(incremental[key], expected[key], strict=True):
torch.testing.assert_close(a, b, atol=0, rtol=0)
for key in expected_kv:
for a, b in zip(incremental_kv[key], expected_kv[key], strict=True):
torch.testing.assert_close(a, b, atol=0, rtol=0)
chunked, chunked_kv = recurrent_states(model, [5, 14, 1, 1], incremental=True)
for key in ((19, d) for d in range(4)):
for a, b in zip(chunked[key], expected[key], strict=True):
assert_bf16_close(a, b)
for key in expected_kv:
for a, b in zip(chunked_kv[key], expected_kv[key], strict=True):
assert_bf16_close(a, b)


def test_npu_cancel_reuse_and_multiple_requests(npu_device):
engine = StageEngine(
make_npu_model(npu_device),
attention_backend="torch",
cache_config=CacheConfig(num_blocks=32, block_size=16, incremental_allocation=True),
scheduler_config=SchedulerConfig(max_num_seqs=2, max_num_batched_tokens=32),
)
try:
params = SamplingParams(max_tokens=3, temperature=0, ignore_eos=True)
engine.add_request("cancel", [3, 7, 2] * 6, params)
engine.add_request("keep", [5, 9, 11], params)
engine.step()
engine.abort_request("cancel")
engine.add_request("cancel", [13, 14, 15], params)
finished = {}
for _ in range(200):
for output in engine.step():
if output.finished:
finished[output.request_id] = output
if not engine.has_unfinished_requests():
break
assert set(finished) == {"keep", "cancel"}
assert all(
o.exit_depths == [4] * 3 and o.finish_reason == "length" for o in finished.values()
)
assert engine.cache_manager.num_used_blocks == 0
finally:
engine.close()


@pytest.mark.parametrize("ignore_eos", [False, True])
def test_npu_eos_and_length_handling(npu_device, ignore_eos):
model = make_npu_model(npu_device)
model.lm_head.weight.zero_() # Argmax 0 is the tiny model's actual EOS token.
engine = StageEngine(model, attention_backend="torch", cache_config=CacheConfig(num_blocks=16))
try:
engine.add_request(
"eos", [3, 7, 2], SamplingParams(max_tokens=3, temperature=0, ignore_eos=ignore_eos)
)
output = None
for _ in range(100):
for item in engine.step():
if item.finished:
output = item
if not engine.has_unfinished_requests():
break
assert output is not None
assert output.token_ids == [0] * (3 if ignore_eos else 1)
assert output.finish_reason == ("length" if ignore_eos else "stop")
assert engine.cache_manager.num_used_blocks == 0
finally:
engine.close()
Loading