diff --git a/README.md b/README.md index 59f0663a..17502f55 100644 --- a/README.md +++ b/README.md @@ -133,6 +133,11 @@ Packed prefill uses `ExecutionConfig.packed_prefill` and the `--packed-prefill` / `--no-packed-prefill` flags. Input transfer is controlled independently by `use_uva`; see the [selection rules](docs/runtime.md#packed-prefill). +FP8 KV defaults to per-vector scales on Torch/Triton. Experimental native FA4 FP8, +including Dory sliding attention, uses `flash-attn-4==4.0.0b39` and calibrates +a fixed scale at startup. See the +[hardware and calibration requirements](docs/runtime.md#experimental-fa4-static-scale-adapter). + ## Model Support diff --git a/docs/runtime.md b/docs/runtime.md index 910d3451..571ecb59 100644 --- a/docs/runtime.md +++ b/docs/runtime.md @@ -325,6 +325,80 @@ zero watermark in non-incremental mode, see [scheduler admission](scheduler.md#a Pool sizing establishes physical capacity; scheduler admission decides how much of that capacity can safely be promised to new work. +## FP8 KV cache + +`CacheConfig(kv_cache_dtype="fp8_e4m3")` (CLI `--kv-cache-dtype fp8_e4m3`) stores K and V as +float8 e4m3 codes with one float32 scale per (block, layer, token, KV head) vector +(`scale = amax / 448`). Values are quantized when written and dequantized by the attention +backend when read, so a block costs `head_dim + 4` bytes per vector instead of +`head_dim * element_size`: 51.6% of BF16 at `head_dim=128`, and automatic sizing or +`--kv-cache-memory-bytes` admits about 1.94x as many tokens. The default `"auto"` stores the +model dtype and leaves every existing path unchanged. + +- Supported: Torch (reference) and Triton attention; synchronous and asynchronous scheduling; + CUDA graphs (quantized writes are captured); LAST_EXITED finalize copies, including the + asynchronous finalize kernel, copy codes and scales; preemption snapshots carry the scales; + `KVCacheManager.read` returns dequantized values. +- Triton's split-KV decode and packed paged-prefill kernels both apply the scales on load + (prefill dequantizes to the query dtype before `tl.dot`). Device-masked writes (speculative + verification, bucketed CUDA graphs) use the same fused writer with the row mask applied. +- Triton writes with one fused quantize-and-store kernel per layer. The Torch backend quantizes + with Torch ops and is the reference; both writers produce the same codes and scales. +- With Triton FP8 KV, PD transfer sends K/V scales with the corresponding cache + rows. P and D must use the same KV dtype. +- Each layer writes the current token's K/V before attending, so attention reads the current + token in FP8 as well. Accuracy should be checked per model and workload; adaptive exits can + change where a token exits because the gate sees slightly different attention. + +### Experimental FA4 static-scale adapter + +Use `--kv-cache-dtype fp8_e4m3 --attention-backend flash_attn_4` +(Python: `CacheConfig(kv_cache_dtype="fp8_e4m3")` with `attention_backend="flash_attn_4"`). +Backend `auto` also selects this path when FA4 meets the execution requirements. The prototype +targets FA4 4.0.0b39 on SM10x with BF16 models and head dimensions 64, 128 or 256. +Install the pinned dependency with `pip install -e '.[flash-attn4]'`. Both BF16 +and FP8 use FA4's public API; FP8 passes `TensorwiseScales`, without a private +forward entry point or temporary BF16 KV staging. +At startup the engine runs a short BF16 prefill with deterministic vocabulary +IDs and one decode input when context length permits. It measures the maximum +absolute Q/K/V value across all layers and loops, adds 10% headroom, divides by 448, +and rounds up to a power of two. The result stays fixed for the lifetime of the +KV pool and is recorded in the startup log. Values outside `±448 * scale` +saturate. FA4 quantizes Q with this scale and receives `[batch, kv_head]` +Q/K/V descales in both decode and packed prefill. This changes compute precision +in addition to KV storage precision. Validate model accuracy on representative +requests because the startup probe only samples a short synthetic sequence. + +The existing per-vector scale pools and finalize/snapshot paths are retained +for a small reviewable prototype; byte accounting still includes them. Snapshots +with different scale policies cannot be restored. Torch/Triton can read the +fixed-scale cache for reference checks. FA4 packed-vs-decode bitwise matching is +not advertised for FP8. In PD, workers calibrate from the same startup inputs +and verify that their fixed scale matches before exchanging KV pages. + +Dory uses the same adapter: sliding attention passes `(window - 1, 0)` to FA4, +because the runtime window includes the current token. Full-attention layers +keep their causal mask. Packed prompt chunks retain bottom-right alignment +against an existing prefix. Both `per_loop` and `last_loop` KV modes, using +their required layouts, support asynchronous scheduling and CUDA Graphs in +the input, recurrent and output stages. Dory's existing rejection of mixed +scheduling and boundary-KV prefix caching remains in effect. + +Validation on B300 with FA4 4.0.0b39 covers 69 GPU cases across kernel, +end-to-end and regression checks: head dimensions 64/128/256, packed sliding +windows with existing prefixes, snapshot/finalize paths, and Dory 2/4-loop +eager versus async single/multi-stream Graph execution without fallback. +The real Dory 4B checkpoint also completed a 1030-token prompt across its +1024-token window and matched FP8 eager output under asynchronous Graphs. + +An unprofiled eager comparison on that checkpoint (2 loops, 128 input tokens, +16 output tokens, one request, three warmed repetitions) measured median +output throughput of 20.45 tok/s with BF16 and 18.21 tok/s with FP8. Allocated +KV pool storage fell from 576 MiB to 292.5 MiB at equal block counts. This +small case does not demonstrate a speedup or establish model accuracy. Its +fixed scale of 16 came from one separate calibration prompt and is not a +recommended default; validate calibration and task quality for deployment. + ## Replay and validation commands Trace mode expects `depths_by_request`, with one depth per output: full prefill depth first, then each decode input's exit depth. Admission validates coverage and loop bounds. Request IDs must match the trace. The public engine replay fixes depth decisions; token generation still follows the configured sampler. diff --git a/pyproject.toml b/pyproject.toml index 40652f8c..fa46c0c8 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -25,7 +25,7 @@ text = [] pd = ["nixl==1.4.1"] serve = [] flash-attn = ["flash-attn>=2.8.3"] -flash-attn4 = ["flash-attn-4==4.0.0b30"] +flash-attn4 = ["flash-attn-4==4.0.0b39"] triton = [] dev = ["pytest>=8", "ruff>=0.9"] diff --git a/tests/test_backend_selection.py b/tests/test_backend_selection.py index 3778ae71..cc90cf90 100644 --- a/tests/test_backend_selection.py +++ b/tests/test_backend_selection.py @@ -17,7 +17,13 @@ def packages(monkeypatch): from vllm_rlt.kernels import flash_attention - installed = {"flash_attn", "flash_attn_3.flash_attn_interface", "flash_attn.cute", "triton"} + installed = { + "flash_attn", + "flash_attn_3.flash_attn_interface", + "flash_attn.cute", + "TensorwiseScales", + "triton", + } original_import = flash_attention.importlib.import_module def import_module(name): @@ -28,14 +34,19 @@ def import_module(name): if name in ("flash_attn", "flash_attn_3.flash_attn_interface", "flash_attn.cute"): if name not in installed: raise ImportError(f"{name} is not installed") - return SimpleNamespace( + result = SimpleNamespace( flash_attn_with_kvcache=lambda *args, **kwargs: None, flash_attn_varlen_func=lambda *args, **kwargs: None, ) + if name == "flash_attn.cute" and "TensorwiseScales" in installed: + result.TensorwiseScales = SimpleNamespace + return result return original_import(name) monkeypatch.setattr(flash_attention.importlib, "import_module", import_module) - monkeypatch.setattr(flash_attention, "version", lambda name: "test") + monkeypatch.setattr( + flash_attention, "version", lambda name: "4.0.0b39" if name == "flash-attn-4" else "test" + ) monkeypatch.setattr(torch.version, "hip", None) return installed @@ -53,6 +64,94 @@ def test_cpu_auto_reports_torch_and_runs_reference_attention(caplog): torch.testing.assert_close(selected(q, k, v, table, lengths), explicit(q, k, v, table, lengths)) +@pytest.mark.parametrize( + "sm,dtype,head_dim,supported", + [ + ((10, 3), torch.bfloat16, 128, True), + ((9, 0), torch.bfloat16, 128, False), + ((10, 3), torch.float16, 128, False), + ((10, 3), torch.bfloat16, 256, True), + ], +) +def test_fa4_static_fp8_selection(packages, monkeypatch, sm, dtype, head_dim, supported): + monkeypatch.setattr(torch.cuda, "get_device_capability", lambda device: sm) + args = ("flash_attn_4", torch.device("cuda"), dtype, head_dim, 16) + requirement = BackendCapabilities(fp8_kv_static=True) + if not supported: + with pytest.raises(ValueError, match="fp8_kv_static"): + create_backend(*args, required_capabilities=requirement) + return + selected = create_backend(*args, required_capabilities=requirement) + assert not selected.capabilities.packed_matches_decode + with pytest.raises(ValueError, match="fp8_kv"): + create_backend(*args, required_capabilities=BackendCapabilities(fp8_kv=True)) + selected = create_backend("auto", *args[1:], required_capabilities=requirement) + assert selected.info["selected_backend"] == "flash_attn_4" + + +def test_fa4_static_fp8_requires_public_scale_api(packages, monkeypatch): + monkeypatch.setattr(torch.cuda, "get_device_capability", lambda device: (10, 3)) + packages.remove("TensorwiseScales") + args = ("flash_attn_4", torch.device("cuda"), torch.bfloat16, 128, 16) + assert create_backend(*args).info["selected_backend"] == "flash_attn_4" + with pytest.raises(ValueError, match="fp8_kv_static"): + create_backend(*args, required_capabilities=BackendCapabilities(fp8_kv_static=True)) + + +@pytest.mark.parametrize( + "name,sm,block_size", + [("flash_attn_2", (8, 9), 256), ("flash_attn_3", (9, 0), 16), ("flash_attn_4", (10, 3), 16)], +) +def test_flash_rejects_per_vector_fp8_without_staging(packages, monkeypatch, name, sm, block_size): + monkeypatch.setattr(torch.cuda, "get_device_capability", lambda device: sm) + with pytest.raises(ValueError, match="fp8_kv"): + create_backend( + name, + torch.device("cuda"), + torch.bfloat16, + 128, + block_size, + required_capabilities=BackendCapabilities(fp8_kv=True), + ) + + +@pytest.mark.parametrize("calibrate", [False, True]) +def test_auto_static_fp8_keeps_graph_support_and_falls_back_if_scale_api_missing( + packages, monkeypatch, calibrate +): + monkeypatch.setattr(torch.cuda, "get_device_capability", lambda device: (10, 3)) + args = ("auto", torch.device("cuda"), torch.bfloat16, 128, 16) + required = BackendCapabilities(fp8_kv=calibrate, fp8_kv_static=not calibrate, cuda_graphs=True) + options = dict(required_capabilities=required, allow_fp8_calibration=calibrate) + selected = create_backend(*args, **options) + assert selected.info["selected_backend"] == "flash_attn_4" + assert selected.capabilities.fp8_kv_static and selected.capabilities.cuda_graphs + assert not selected.capabilities.packed_matches_decode + + packages.remove("TensorwiseScales") + selected = create_backend(*args, **options) + assert selected.info["selected_backend"] == "triton" + assert selected.capabilities.fp8_kv_static and selected.capabilities.cuda_graphs + + +def test_dory_requirements_select_native_fa4_and_reject_old_hd256(packages, monkeypatch): + monkeypatch.setattr(torch.cuda, "get_device_capability", lambda device: (10, 3)) + args = (torch.device("cuda"), torch.bfloat16, 256, 16) + required = BackendCapabilities( + sliding_window=True, fp8_kv_static=True, cuda_graphs=True, async_scheduling=True + ) + selected = create_backend("auto", *args, required_capabilities=required) + assert selected.info["selected_backend"] == "flash_attn_4" + assert selected.capabilities.sliding_window + packages.remove("TensorwiseScales") + with pytest.raises(ValueError, match="sliding_window"): + create_backend("flash_attn_4", *args, required_capabilities=required) + assert ( + create_backend("auto", *args, required_capabilities=required).info["selected_backend"] + == "triton" + ) + + @pytest.mark.parametrize( "sm,dtype,head_dim,block_size,missing,expected", [ @@ -270,7 +369,15 @@ class ProfileReached(Exception): pass def plan_cache( - model, cache, scheduler, execution, selected, *, speculative=None, staged_prefill=None + model, + cache, + scheduler, + execution, + selected, + *, + speculative=None, + staged_prefill=None, + fp8_scale=None, ): assert selected == "triton" # FA2 cannot use the configured 16-token pages. assert staged_prefill is False diff --git a/tests/test_cuda_graph.py b/tests/test_cuda_graph.py index 2870b539..a0f08c51 100644 --- a/tests/test_cuda_graph.py +++ b/tests/test_cuda_graph.py @@ -455,7 +455,8 @@ def test_bucketed_kv_graph_combines_active_rows_and_pipeline_write_mask(): keys = torch.full((4, 1, 2, 1, 4), -1.0, device="cuda") values = torch.full_like(keys, -2.0) - cache = SimpleNamespace(key_cache=keys, value_cache=values) + # A BF16 pool has no FP8 scales, as KVCacheManager reports them. + cache = SimpleNamespace(key_cache=keys, value_cache=values, key_scale=None, value_scale=None) batch = SimpleNamespace( active_rows=torch.zeros((), dtype=torch.int32, device="cuda"), write_mask=torch.ones(5, dtype=torch.bool, device="cuda"), diff --git a/tests/test_dory_gpu.py b/tests/test_dory_gpu.py index 1038a0ba..3a04240f 100644 --- a/tests/test_dory_gpu.py +++ b/tests/test_dory_gpu.py @@ -449,9 +449,10 @@ def compile_tile(*args, window, **kwargs): assert probe(index, 256, 16, PackedWindowShape(16, 4, 36, 1024)) is None +@pytest.mark.parametrize("fp8_kv", [False, True]) @pytest.mark.parametrize(("window", "expected_windows"), [(None, [0]), (128, [0, 128])]) def test_prefill_tile_probe_compiles_only_the_default_and_caches_result( - monkeypatch, window, expected_windows + monkeypatch, window, expected_windows, fp8_kv ): from types import SimpleNamespace @@ -470,12 +471,17 @@ def test_prefill_tile_probe_compiles_only_the_default_and_caches_result( def compile_tile(*args, **kwargs): assert (kwargs["BLOCK_M"], kwargs["BLOCK_N"]) == (32, 32) assert kwargs["COMPENSATE"] + assert kwargs["HAS_SCALE"] is fp8_kv + assert args[1].dtype == (torch.float8_e4m3fn if fp8_kv else torch.bfloat16) + if fp8_kv: + assert args[7].dtype == args[8].dtype == torch.float32 + assert args[7].shape == args[1].shape[:-1] assert all(arg.device.type == "meta" for arg in args if isinstance(arg, torch.Tensor)) windows.append(kwargs["WINDOW"]) return SimpleNamespace(metadata=SimpleNamespace(shared=limit)) monkeypatch.setattr(attention._paged_prefill_kernel, "warmup", compile_tile) - args = (device, torch.bfloat16, 64, 16, PackedWindowShape(4, 2, 2, window)) + args = (device, torch.bfloat16, 64, 16, PackedWindowShape(4, 2, 2, window, fp8_kv)) attention._select_packed_window_tile.cache_clear() try: diff --git a/tests/test_fa4_fp8.py b/tests/test_fa4_fp8.py new file mode 100644 index 00000000..18c347b2 --- /dev/null +++ b/tests/test_fa4_fp8.py @@ -0,0 +1,504 @@ +"""Static-scale FA4 prototype: CPU interface checks and GPU numerics.""" + +import argparse +from types import SimpleNamespace + +import pytest +import torch + +from vllm_rlt import CacheConfig +from vllm_rlt.attention import create_backend +from vllm_rlt.core.kv_cache_manager import KVCacheManager, quantize_fp8 +from vllm_rlt.entrypoints.runtime_args import add_runtime_args, runtime_configs +from vllm_rlt.kernels.paged_attention import torch_paged_attention, torch_paged_attention_windowed + + +@pytest.mark.parametrize("scale", [0, -1, float("nan"), float("inf"), 1e-50, 1e40, True]) +def test_invalid_static_scale_is_rejected(scale): + with pytest.raises(ValueError, match="fp8_scale"): + KVCacheManager(1, 1, 64, 8, 16, 1, kv_cache_dtype="fp8_e4m3", fp8_scale=scale) + + +def test_static_scale_requires_fp8(): + with pytest.raises(ValueError, match="requires"): + KVCacheManager(1, 1, 64, 8, 16, 1, fp8_scale=0.02) + + +def test_fp8_cli_leaves_scale_selection_to_engine(): + parser = argparse.ArgumentParser() + parser.add_argument("--num-blocks", type=int) + parser.add_argument("--block-size", type=int, default=16) + add_runtime_args(parser) + args = parser.parse_args(["--kv-cache-dtype", "fp8_e4m3"]) + configs = runtime_configs(args) + assert configs["cache_config"].kv_cache_dtype == "fp8_e4m3" + with pytest.raises(SystemExit): + parser.parse_args(["--kv-cache-dtype", "fp8_e4m3", "--fp8-scale", "0.02"]) + + +def test_static_quantization_saturates_and_does_not_rescale_old_tokens(): + cache = KVCacheManager(1, 1, 64, 8, 16, 1, kv_cache_dtype="fp8_e4m3", fp8_scale=0.02) + cache.allocate("a", 2) + for pos, value in enumerate([0.25, 1000.0]): + x = torch.full((1, 1, 64), value) + cache.write(0, ["a"], [0], [pos], x, -x) + k, v = cache.read(0, "a", 0) + torch.testing.assert_close(k[0], torch.full_like(k[0], 0.24)) + torch.testing.assert_close(k[1], torch.full_like(k[1], 448 * 0.02)) + torch.testing.assert_close(v, -k) + + +@pytest.mark.parametrize("packed", [False, True]) +def test_cache_to_fa4_fp8_contract_on_cpu(monkeypatch, packed): + """Execute the real adapter/cache; substitute only the CUDA FA implementation.""" + from vllm_rlt.kernels import flash_attention + + calls = [] + + def fake_fa(q, k, v, **kwargs): + assert q.dtype == k.dtype == v.dtype == torch.float8_e4m3fn + batch = kwargs["page_table"].shape[0] + for name in ("q", "k", "v"): + scale = getattr(kwargs["scales"], name) + assert scale.shape == (batch, 2) and scale.dtype == torch.float32 + torch.testing.assert_close(scale, torch.full_like(scale, 0.02)) + assert kwargs["causal"] and kwargs["num_splits"] == 1 + calls.append((q.clone(), kwargs)) + return torch.zeros(q.shape, dtype=torch.bfloat16), None + + original_import = flash_attention.importlib.import_module + + def public_fa(q, k, v, **kwargs): + if q.dtype == torch.float8_e4m3fn: + return fake_fa(q, k, v, **kwargs) + assert q.dtype == k.dtype == v.dtype == torch.bfloat16 + assert "scales" not in kwargs + return torch.zeros_like(q), None + + def import_module(name): + if name == "flash_attn.cute": + return SimpleNamespace( + flash_attn_varlen_func=public_fa, TensorwiseScales=SimpleNamespace + ) + return original_import(name) + + monkeypatch.setattr(flash_attention.importlib, "import_module", import_module) + monkeypatch.setattr(flash_attention, "version", lambda name: "4.0.0b39") + monkeypatch.setattr(torch.version, "hip", None) + monkeypatch.setattr(torch.cuda, "get_device_capability", lambda device: (10, 3)) + from vllm_rlt.attention import BackendCapabilities + + attention = create_backend( + "flash_attn_4", + torch.device("cuda"), + torch.bfloat16, + 64, + 16, + required_capabilities=BackendCapabilities(fp8_kv_static=True), + ) + cache = KVCacheManager( + 2, + 2, + 64, + 16, + 16, + 2, + dtype=torch.bfloat16, + attention=attention, + kv_cache_dtype="fp8_e4m3", + fp8_scale=0.02, + ) + cache.allocate("a", 3) + cache.allocate("b", 3) + batch = cache._prepare_batch(["a", "a", "b"], [0, 0, 1], [0, 1, 0], packed_prefill=packed) + q = torch.linspace(-10, 10, 3 * 4 * 64).reshape(3, 4, 64).bfloat16() + kv = q[:, :2] + cache._write_prepared(1, batch, kv, -kv) + output = cache._attend_prepared(1, batch, q) + assert output.shape == q.shape and output.dtype == q.dtype + q_codes, kwargs = calls.pop() + expected, _ = quantize_fp8(q, 0.02) + torch.testing.assert_close(q_codes.reshape_as(q).float(), expected.float(), rtol=0, atol=0) + if packed: + assert kwargs["cu_seqlens_q"].tolist() == [0, 2, 3] + assert kwargs["seqused_k"].tolist() == [2, 1] + else: + assert kwargs["seqused_k"].tolist() == [1, 2, 1] + with pytest.raises(ValueError, match="fixed scalar"): + attention( + q, + cache.key_cache[:, 1], + cache.value_cache[:, 1], + batch.block_tables, + batch.context_lengths, + key_scale=cache.key_scale[:, 1], + value_scale=cache.value_scale[:, 1], + ) + # Both BF16 and FP8 use the public API; only FP8 supplies scales. + flat = cache._prepare_batch(["a", "a", "b"], [0, 0, 1], [0, 1, 0]) + attention( + q, + cache.key_cache[:, 1].bfloat16(), + cache.value_cache[:, 1].bfloat16(), + flat.block_tables, + flat.context_lengths, + ) + + +@pytest.mark.gpu +@pytest.mark.parametrize("packed", [False, True]) +@pytest.mark.parametrize("head_dim", [64, 128, 256]) +def test_fa4_fp8_matches_dequantized_paged_reference(packed, head_dim, record_property): + if torch.cuda.get_device_capability()[0] != 10: + pytest.skip("FA4 FP8 requires SM10x") + torch.manual_seed(91) + cache = KVCacheManager( + 2, + 2, + head_dim, + 32, + 16, + 2, + "cuda", + torch.bfloat16, + "flash_attn_4", + kv_cache_dtype="fp8_e4m3", + fp8_scale=0.02, + ) + cache.allocate("a", 32) + cache.allocate("b", 32) + ids = ["a"] * 19 + ["b"] * 7 + depths = [0] * 19 + [1] * 7 + positions = list(range(19)) + list(range(7)) + batch = cache._prepare_batch(ids, depths, positions, packed_prefill=packed) + flat = cache._prepare_batch(ids, depths, positions) + q = torch.randn(26, 4, head_dim, device="cuda", dtype=torch.bfloat16) + k = torch.randn(26, 2, head_dim, device="cuda", dtype=torch.bfloat16) + cache._write_prepared(1, batch, k, -k) + actual = cache._attend_prepared(1, batch, q) + q_codes, q_scale = quantize_fp8(q, 0.02) + expected = torch_paged_attention( + q_codes.float() * q_scale.unsqueeze(-1), + cache.key_cache[:, 1], + cache.value_cache[:, 1], + flat.block_tables, + flat.context_lengths, + cache.key_scale[:, 1], + cache.value_scale[:, 1], + ) + # Compare to quantized inputs; this is a kernel check, not model quality. + torch.testing.assert_close(actual.float(), expected.float(), atol=0.04, rtol=0.05) + error = actual.float() - expected.float() + record_property("max_abs_error", error.abs().max().item()) + record_property("rms_error", error.square().mean().sqrt().item()) + + +@pytest.mark.gpu +def test_fa4_static_fp8_prefix_snapshot_and_mixed_depth_finalize(): + if torch.cuda.get_device_capability()[0] != 10: + pytest.skip("FA4 FP8 requires SM10x") + torch.manual_seed(163) + cache = KVCacheManager( + 2, + 2, + 128, + 32, + 16, + 3, + "cuda", + torch.bfloat16, + "flash_attn_4", + enable_prefix_caching=True, + kv_cache_dtype="fp8_e4m3", + fp8_scale=0.02, + ) + tokens = list(range(18)) + assert cache.allocate("source", 32) + for depth in range(3): + for layer in range(2): + values = torch.randn(17, 2, 128, device="cuda", dtype=torch.bfloat16) + cache.write(layer, ["source"] * 17, [depth] * 17, list(range(17)), values, -values) + for depth in (0, 1): + for layer in range(2): + values = torch.randn(1, 2, 128, device="cuda", dtype=torch.bfloat16) + cache.write(layer, ["source"], [depth], [17], values, -values) + cache.finalize_token("source", 17, 1) + for layer in range(2): + for source, copied in zip(cache.read(layer, "source", 1), cache.read(layer, "source", 2)): + torch.testing.assert_close(source[-1], copied[-1], rtol=0, atol=0) + cache.publish_prefix("source", tokens, len(tokens)) + prefix = cache.lookup_prefix(tokens) + assert len(prefix) == 1 # Only the full 16-token block can be shared. + assert cache.allocate("shared", 32, prefix=prefix) + for depth in range(3): + for layer in range(2): + for original, shared in zip( + cache.read(layer, "source", depth, 16), cache.read(layer, "shared", depth) + ): + torch.testing.assert_close(shared, original, rtol=0, atol=0) + q = torch.randn(3, 4, 128, device="cuda", dtype=torch.bfloat16) + before = cache.attend(1, ["source"] * 3, [0, 1, 2], [17] * 3, q) + snapshot = cache.snapshot("source") + cache.free("source") + assert cache.allocate("restored", 32) + cache.restore("restored", snapshot) + after = cache.attend(1, ["restored"] * 3, [0, 1, 2], [17] * 3, q) + torch.testing.assert_close(after, before, rtol=0, atol=0) + cache.free("shared") + cache.free("restored") + + +@pytest.mark.gpu +@pytest.mark.parametrize("multi_stream", [False, True]) +def test_fa4_static_fp8_async_graphs_match_eager(multi_stream): + from tests.helpers import tiny_ouro_config + from vllm_rlt import ExecutionConfig, ExitConfig, SamplingParams, SchedulerConfig + from vllm_rlt.engine.llm_engine import LLMEngine + from vllm_rlt.models import OuroForCausalLM + + if torch.cuda.get_device_capability()[0] != 10: + pytest.skip("FA4 FP8 requires SM10x") + torch.manual_seed(321) + model = OuroForCausalLM(tiny_ouro_config(head_dim=64)).to("cuda", torch.bfloat16) + traces = {str(i): [4, 1, 3, 2, 4, 2, 1][: 5 + i] for i in range(3)} + + def run(execution): + engine = LLMEngine( + model, + cache_config=CacheConfig( + num_blocks=None if execution.cuda_graphs else 64, kv_cache_dtype="fp8_e4m3" + ), + scheduler_config=SchedulerConfig( + max_num_seqs=3, max_num_batched_tokens=8, prefill_chunk_size=4 + ), + execution_config=execution, + exit_config=ExitConfig("trace", depths_by_request=traces), + attention_backend="flash_attn_4", + ) + try: + scale = engine.cache_manager.fp8_scale + assert scale is not None and scale > 0 + for i in range(3): + engine.add_request( + str(i), + [2 + i, 3, 4] * (i + 1), + SamplingParams(max_tokens=5 + i, min_loops=1, ignore_eos=True), + ) + done = {} + while engine.has_unfinished_requests(): + for output in engine.step(): + if output.finished: + done[output.request_id] = (output.token_ids, list(output.exit_depths)) + assert len(done) == 3 + assert engine.cache_manager.num_used_blocks == 0 + if execution.cuda_graphs: + graphs = engine.model_runner.graphs + assert graphs.captures > 0 + assert graphs.replays > graphs.captures + assert graphs.fallbacks == 0 + return scale, done + finally: + engine.close() + + reference = run(ExecutionConfig()) + candidate = run( + ExecutionConfig(async_scheduling=True, multi_stream=multi_stream, cuda_graphs=True) + ) + assert candidate == reference + + +@pytest.mark.gpu +@pytest.mark.parametrize("head_dim", [64, 128, 256]) +@pytest.mark.parametrize("fixed_scale", [None, 0.02]) +def test_fa4_windowed_attention_matches_quantized_reference(head_dim, fixed_scale): + if torch.cuda.get_device_capability()[0] != 10: + pytest.skip("FA4 FP8 requires SM10x") + torch.manual_seed(419) + attention = create_backend("flash_attn_4", torch.device("cuda"), torch.bfloat16, head_dim, 16) + q = torch.randn(4, 4, head_dim, device="cuda", dtype=torch.bfloat16) + # Layer slices preserve the real pool strides, including for head_dim=256. + k, v = [ + torch.randn(12, 3, 16, 2, head_dim, device="cuda", dtype=q.dtype)[:, 1] for _ in range(2) + ] + tables = torch.tensor( + [[7, 2, 9], [3, 0, 0], [1, 8, 6], [0, 0, 0]], device="cuda", dtype=torch.int32 + ) + lengths = torch.tensor([35, 1, 47, 0], device="cuda", dtype=torch.int32) + scales = {} + reference_q = q + key_scale = value_scale = None + if fixed_scale is not None: + q_codes, q_scale = quantize_fp8(q, fixed_scale) + reference_q = q_codes.float() * q_scale.unsqueeze(-1) + # Keep non-contiguous layer strides for the FP8 pool too. + k, v = [ + x.to(torch.float8_e4m3fn) + for x in ( + torch.randn(12, 3, 16, 2, head_dim, device="cuda") / fixed_scale, + torch.randn(12, 3, 16, 2, head_dim, device="cuda") / fixed_scale, + ) + ] + k, v = k[:, 1], v[:, 1] + key_scale = value_scale = torch.full(k.shape[:-1], fixed_scale, device="cuda") + scalar = torch.tensor(fixed_scale, device="cuda") + scales = dict(key_scale=scalar, value_scale=scalar) + for window in (1, 4, 16, 17, 1024): + actual = attention.windowed(q, k, v, tables, lengths, window, **scales) + expected = torch_paged_attention_windowed( + reference_q, k, v, tables, lengths, window, key_scale, value_scale + ) + torch.testing.assert_close(actual.float(), expected.float(), atol=0.04, rtol=0.05) + assert torch.count_nonzero(actual[-1]) == 0 + + +@pytest.mark.gpu +@pytest.mark.parametrize("kv_mode,loops", [("per_loop", 2), ("last_loop", 4)]) +@pytest.mark.parametrize("mode", ["async", "multi", "graph", "async_graph", "multi_graph"]) +def test_dory_fa4_fp8_execution_modes_match_eager(kv_mode, loops, mode): + from tests.test_dory import tiny_dory_config + from vllm_rlt import ExecutionConfig, SamplingParams, SchedulerConfig + from vllm_rlt.engine.llm_engine import LLMEngine + from vllm_rlt.models import DoryForCausalLM + + if torch.cuda.get_device_capability()[0] != 10: + pytest.skip("FA4 FP8 requires SM10x") + torch.manual_seed(42) + model = DoryForCausalLM( + tiny_dory_config(head_dim=256, n_recurrent_loops=loops, recurrent_kv_cache_mode=kv_mode) + ).to("cuda", torch.bfloat16) + prompts = {"a": [4, 7, 3], "b": [9, 2, 6, 5], "c": [8, 3, 3, 11, 5, 2, 7, 13, 6]} + + def run(execution): + engine = LLMEngine( + model, + attention_backend="auto", + cache_config=CacheConfig( + num_blocks=128, + block_size=16, + layout=model.kv_cache_layout, + kv_cache_dtype="fp8_e4m3", + ), + scheduler_config=SchedulerConfig(max_num_seqs=3, max_num_batched_tokens=16), + execution_config=execution, + ) + try: + assert engine.cache_manager.backend == "flash_attn_4" + assert engine.execution_config.packed_prefill == ( + kv_mode == "per_loop" + and not execution.async_scheduling + and not execution.cuda_graphs + ) + params = SamplingParams(max_tokens=8, min_loops=loops, max_loops=loops, ignore_eos=True) + finished = {} + for rid, prompt in prompts.items(): + engine.add_request(rid, prompt, params) + for _ in range(300): + if not engine.has_unfinished_requests(): + break + for out in engine.step(): + if out.finished: + finished[out.request_id] = (out.token_ids, list(out.exit_depths)) + assert len(finished) == len(prompts) + assert engine.cache_manager.num_used_blocks == 0 + if execution.cuda_graphs: + runner = engine.model_runner + for graphs in (runner.prelude_graphs, runner.graphs, runner.coda_graphs): + assert graphs.captures > 0 and graphs.replays > 0 + assert graphs.fallbacks == 0 + return finished + finally: + engine.close() + + expected = run(ExecutionConfig()) + actual = run( + ExecutionConfig( + async_scheduling=mode != "graph", + multi_stream=mode.startswith("multi"), + cuda_graphs="graph" in mode, + ) + ) + assert actual == expected + + +def test_dory_fp8_preserves_mixed_scheduler_rejection(): + from tests.test_dory import tiny_dory_config + from vllm_rlt import SchedulerConfig + from vllm_rlt.engine.llm_engine import LLMEngine + from vllm_rlt.models import DoryForCausalLM + + model = DoryForCausalLM(tiny_dory_config(head_dim=256)) + with pytest.raises(ValueError, match="boundary-KV models do not support staged prefill"): + LLMEngine( + model, + attention_backend="flash_attn_4", + cache_config=CacheConfig(kv_cache_dtype="fp8_e4m3"), + scheduler_config=SchedulerConfig(mode="mixed"), + ) + + +@pytest.mark.gpu +@pytest.mark.parametrize("head_dim", [64, 128, 256]) +def test_fa4_fp8_packed_window_with_existing_prefix_matches_per_row_reference(head_dim): + from vllm_rlt.worker.cuda_graph import _DeviceCache + + if torch.cuda.get_device_capability()[0] != 10: + pytest.skip("FA4 FP8 requires SM10x") + torch.manual_seed(819) + cache = KVCacheManager( + 2, + 2, + head_dim, + 32, + 16, + 2, + "cuda", + torch.bfloat16, + "flash_attn_4", + kv_cache_dtype="fp8_e4m3", + fp8_scale=0.02, + ) + for rid in ("a", "b"): + assert cache.allocate(rid, 32) + ids, depths, positions = ( + ["a"] * 19 + ["b"] * 7, + [0] * 19 + [1] * 7, + list(range(19)) + list(range(7)), + ) + k, v = [torch.randn(26, 2, head_dim, device="cuda", dtype=torch.bfloat16) for _ in range(2)] + cache.write(1, ids, depths, positions, k, v) + ids, depths, positions = ( + ["a"] * 4 + ["b"] * 3, + [0] * 4 + [1] * 3, + list(range(15, 19)) + list(range(4, 7)), + ) + packed = cache._prepare_batch(ids, depths, positions, packed_prefill=True) + flat = cache._prepare_batch(ids, depths, positions) + q = torch.randn(7, 4, head_dim, device="cuda", dtype=torch.bfloat16) + codes, scales = quantize_fp8(q, 0.02) + expected = torch_paged_attention_windowed( + codes.float() * scales.unsqueeze(-1), + cache.key_cache[:, 1], + cache.value_cache[:, 1], + flat.block_tables, + flat.context_lengths, + 4, + cache.key_scale[:, 1], + cache.value_scale[:, 1], + ) + # FA4 quantizes the softmax numerator to E4M3 before P@V. Its rounding + # error is bounded by 1/16 of each weighted |V| contribution, including + # outputs near zero where an output-relative tolerance is inappropriate. + weighted_abs_v = torch_paged_attention_windowed( + codes.float() * scales.unsqueeze(-1), + cache.key_cache[:, 1], + cache.value_cache[:, 1].float().abs(), + flat.block_tables, + flat.context_lengths, + 4, + cache.key_scale[:, 1], + cache.value_scale[:, 1], + ) + for target in (cache, _DeviceCache(cache)): + actual = target._attend_prepared(1, packed, q, window=4) + # Include BF16 output rounding and small softmax reduction error. + assert torch.all((actual.float() - expected.float()).abs() <= 0.07 * weighted_abs_v + 0.005) diff --git a/tests/test_fp8_kv_cache.py b/tests/test_fp8_kv_cache.py new file mode 100644 index 00000000..373a99ce --- /dev/null +++ b/tests/test_fp8_kv_cache.py @@ -0,0 +1,444 @@ +"""FP8 e4m3 KV pool: per-vector quantization, attention, LAST_EXITED copies and preemption.""" + +import pytest +import torch + +from tests.helpers import tiny_ouro_config +from vllm_rlt import CacheConfig, ExecutionConfig, ExitConfig, SamplingParams, SchedulerConfig +from vllm_rlt.attention import AttentionBackend, BackendCapabilities +from vllm_rlt.core.kv_cache_manager import ( + FP8_E4M3_MAX, + KVCacheManager, + dequantize_fp8, + kv_bytes_per_block, + quantize_fp8, +) +from vllm_rlt.engine.llm_engine import LLMEngine +from vllm_rlt.kernels.paged_attention import torch_paged_attention +from vllm_rlt.models import OuroForCausalLM + + +def make_cache(kv_cache_dtype="fp8_e4m3", **overrides): + values = dict( + num_layers=2, + num_kv_heads=2, + head_dim=8, + num_blocks=64, + block_size=4, + max_loops=4, + kv_cache_dtype=kv_cache_dtype, + ) + values.update(overrides) + return KVCacheManager(**values) + + +def fill(cache, rid, length, depths=range(4), seed=0): + """Write random K/V for positions [0, length) at every depth and layer.""" + generator = torch.Generator().manual_seed(seed) + for depth in depths: + for layer in range(cache.num_layers): + shape = (length, cache.num_kv_heads, cache.head_dim) + k = torch.randn(shape, generator=generator) * (1 + layer) + v = torch.randn(shape, generator=generator) + cache.write(layer, [rid] * length, [depth] * length, list(range(length)), k, v) + + +def test_quantize_round_trip_is_bounded_per_vector(): + x = torch.randn(64, 4, 32) * torch.logspace(-3, 3, 64).view(64, 1, 1) + codes, scale = quantize_fp8(x) + assert codes.dtype == torch.float8_e4m3fn and scale.shape == x.shape[:-1] + error = (dequantize_fp8(codes, scale, torch.float32) - x).abs() + # e4m3 keeps 3 mantissa bits: rounding error is at most 1/16 of the vector's amax. + assert (error <= x.abs().amax(-1, keepdim=True) / 16 + 1e-30).all() + zeros = quantize_fp8(torch.zeros(2, 8)) + assert torch.isfinite(zeros[1]).all() and not dequantize_fp8(*zeros, torch.float32).any() + + +def test_storage_layout_and_byte_accounting(): + cache = make_cache() + assert cache.key_cache.dtype == torch.float8_e4m3fn + assert cache.key_scale.shape == cache.key_cache.shape[:-1] + assert cache.bytes_per_block == kv_bytes_per_block(2, 4, 2, 8, 1, "fp8_e4m3") + assert cache.bytes_per_block == 2 * 2 * 4 * 2 * (8 + 4) + plain = make_cache("auto", dtype=torch.bfloat16) + assert plain.key_scale is None and plain.bytes_per_block == 2 * 2 * 4 * 2 * 8 * 2 + with pytest.raises(ValueError, match="kv_cache_dtype"): + make_cache("int4") + + +@pytest.mark.parametrize("fixed_scale", [None, 0.02]) +def test_attention_equals_attention_over_dequantized_values(fixed_scale): + cache = make_cache(fp8_scale=fixed_scale) + reference = make_cache("auto") + assert cache.allocate("r", 11) and reference.allocate("r", 11) + fill(cache, "r", 11) + for layer in range(cache.num_layers): + for depth in range(4): + k, v = cache.read(layer, "r", depth) + reference.write(layer, ["r"] * 11, [depth] * 11, list(range(11)), k, v) + q = torch.randn(3, 2, 8) + rows = (["r"] * 3, [0, 2, 3], [10, 4, 7]) + for layer in range(cache.num_layers): + got = cache.attend(layer, *rows, q) + want = reference.attend(layer, *rows, q) + torch.testing.assert_close(got, want, rtol=1e-5, atol=1e-6) + + +def test_windowed_attention_reads_fp8_scales(): + cache = make_cache() + reference = make_cache("auto") + assert cache.allocate("r", 11) and reference.allocate("r", 11) + fill(cache, "r", 11) + for layer in range(cache.num_layers): + for depth in range(4): + k, v = cache.read(layer, "r", depth) + reference.write(layer, ["r"] * 11, [depth] * 11, list(range(11)), k, v) + q = torch.randn(3, 2, 8) + rows = (["r"] * 3, [0, 2, 3], [10, 4, 7]) + for layer in range(cache.num_layers): + got = cache.attend(layer, *rows, q, window=3) + want = reference.attend(layer, *rows, q, window=3) + torch.testing.assert_close(got, want, rtol=1e-5, atol=1e-6) + assert not torch.allclose(got, cache.attend(layer, *rows, q)) + + +def test_graph_and_triton_window_paths_forward_fp8_scales(monkeypatch): + import vllm_rlt.kernels.paged_attention as paged + from vllm_rlt.worker.cuda_graph import _DeviceCache + + cache = make_cache() + assert cache.allocate("r", 11) + fill(cache, "r", 11) + q = torch.randn(3, 2, 8) + rows = (["r"] * 3, [0, 2, 3], [10, 4, 7]) + batch = cache._prepare_batch(*rows, for_write=False) + for layer in range(cache.num_layers): + # Graph replay's window branch reads the same scales as eager attention. + got = _DeviceCache(cache)._attend_prepared(layer, batch, q, window=3) + torch.testing.assert_close(got, cache.attend(layer, *rows, q, window=3)) + + calls = [] + monkeypatch.setattr(paged, "triton_paged_attention", lambda *a, **k: calls.append((a, k))) + scale = cache.key_scale[:, 0] + paged.TritonPagedAttention().windowed( + q, None, None, None, None, 3, key_scale=scale, value_scale=scale + ) + ((args, kwargs),) = calls + assert args[5] is scale and args[6] is scale and kwargs == {"window": 3} + + +def test_torch_kernel_scales_multiply_each_token_vector(): + codes, scale = quantize_fp8(torch.randn(5, 4, 2, 8)) # [block, token, head, dim] + tables = torch.tensor([[3, 1, 4]], dtype=torch.int32) + lengths = torch.tensor([10], dtype=torch.int32) + q = torch.randn(1, 2, 8) + got = torch_paged_attention(q, codes, codes, tables, lengths, scale, scale) + dense = dequantize_fp8(codes, scale, torch.float32) + want = torch_paged_attention(q, dense, dense, tables, lengths) + torch.testing.assert_close(got, want, rtol=1e-5, atol=1e-6) + + +@pytest.mark.parametrize("fixed_scale", [None, 0.02]) +def test_finalize_copies_codes_and_scales_to_deeper_planes(fixed_scale): + cache = make_cache(fp8_scale=fixed_scale) + assert cache.allocate("r", 6) + fill(cache, "r", 5) + fill(cache, "r", 6, depths=[0], seed=1) # position 5 exists only at depth 0 + cache.finalize_token("r", 5, 0) + for layer in range(cache.num_layers): + exited = [t[5] for t in cache.read(layer, "r", 0)] + for depth in (1, 2, 3): + deeper = [t[5] for t in cache.read(layer, "r", depth)] + assert all(torch.equal(a, b) for a, b in zip(exited, deeper)) + + +@pytest.mark.parametrize("fixed_scale", [None, 0.02]) +def test_snapshot_restore_is_bitwise_and_checks_dtype(fixed_scale): + cache = make_cache(fp8_scale=fixed_scale) + assert cache.allocate("r", 7) + fill(cache, "r", 7) + before = [cache.read(layer, "r", depth) for layer in range(2) for depth in range(4)] + snapshot = cache.snapshot("r") + assert snapshot.keys.dtype == torch.float8_e4m3fn and snapshot.key_scales is not None + cache.free("r") + assert cache.allocate("r", 7) + cache.restore("r", snapshot) + after = [cache.read(layer, "r", depth) for layer in range(2) for depth in range(4)] + assert all(torch.equal(a, b) for x, y in zip(before, after) for a, b in zip(x, y)) + plain = make_cache("auto") + assert plain.allocate("r", 7) + with pytest.raises(ValueError, match="dtype"): + plain.restore("r", snapshot) + different_scale = make_cache(fp8_scale=0.03) + assert different_scale.allocate("r", 7) + with pytest.raises(ValueError, match="scale policy"): + different_scale.restore("r", snapshot) + + +def test_backends_without_fp8_support_are_rejected(): + info = {"backend": "flash", "selected_backend": "flash"} + backend = AttentionBackend(torch_paged_attention, BackendCapabilities(), info) + with pytest.raises(ValueError, match="FP8"): + make_cache(attention=backend) + + +def run_engine(model, kv_cache_dtype, prompts, params, exit_config): + engine = LLMEngine( + model, + cache_config=CacheConfig(128, 4, kv_cache_dtype=kv_cache_dtype), + exit_config=exit_config, + ) + assert (engine.cache_manager.key_scale is not None) == (kv_cache_dtype == "fp8_e4m3") + for i, prompt in enumerate(prompts): + engine.add_request(str(i), prompt, params) + done = {} + while engine.has_unfinished_requests(): + for output in engine.step(): + if output.finished: + done[output.request_id] = output + assert engine.cache_manager.num_used_blocks == 0 + return done + + +def tiny_model(): + torch.manual_seed(5) + model = OuroForCausalLM(tiny_ouro_config()) + with torch.no_grad(): + model.lm_head.weight.mul_(8) # decisive logits: FP8 KV error stays below the margins + return model + + +def test_fixed_depth_engine_tokens_match_unquantized_kv(): + model = tiny_model() + prompts = [[2, 3, 4, 5], [7, 8], [9, 10, 11]] + params = SamplingParams(max_tokens=12, ignore_eos=True) + plain = run_engine(model, "auto", prompts, params, ExitConfig("ouro")) + fp8 = run_engine(model, "fp8_e4m3", prompts, params, ExitConfig("ouro")) + assert {k: o.token_ids for k, o in fp8.items()} == {k: o.token_ids for k, o in plain.items()} + + +def test_early_exit_trace_replays_through_fp8_finalize_copies(): + # A fixed exit trace isolates the LAST_EXITED copy path from gate sensitivity. + traces = {"0": [4, 1, 3, 2, 4, 1, 2, 3], "1": [4, 2, 2, 1, 1, 4, 3, 2]} + params = SamplingParams(max_tokens=8, ignore_eos=True, min_loops=1) + done = run_engine( + tiny_model(), + "fp8_e4m3", + [[2, 3, 4, 5], [7, 8]], + params, + ExitConfig("trace", depths_by_request=traces), + ) + assert {k: list(o.exit_depths) for k, o in done.items()} == traces + + +@pytest.mark.gpu +def test_triton_fp8_attention_matches_torch_reference(): + from vllm_rlt.kernels.triton_attention import paged_attention + + torch.manual_seed(3) + codes, scale = quantize_fp8(torch.randn(40, 16, 4, 128, device="cuda")) + tables = torch.randint(0, 40, (6, 9), device="cuda", dtype=torch.int32) + lengths = torch.tensor([1, 17, 50, 100, 144, 3], device="cuda", dtype=torch.int32) + q = torch.randn(6, 8, 128, device="cuda", dtype=torch.bfloat16) + got = paged_attention(q, codes, codes, tables, lengths, scale, scale) + want = torch_paged_attention(q, codes, codes, tables, lengths, scale, scale) + torch.testing.assert_close(got.float(), want.float(), rtol=2e-2, atol=2e-2) + + +@pytest.mark.gpu +def test_triton_fp8_windowed_attention_matches_torch_reference(): + from vllm_rlt.kernels.paged_attention import torch_paged_attention_windowed + from vllm_rlt.kernels.triton_attention import paged_attention + + torch.manual_seed(5) + codes, scale = quantize_fp8(torch.randn(40, 16, 4, 128, device="cuda")) + tables = torch.randint(0, 40, (6, 9), device="cuda", dtype=torch.int32) + lengths = torch.tensor([1, 17, 50, 100, 144, 3], device="cuda", dtype=torch.int32) + q = torch.randn(6, 8, 128, device="cuda", dtype=torch.bfloat16) + got = paged_attention(q, codes, codes, tables, lengths, scale, scale, window=24) + want = torch_paged_attention_windowed(q, codes, codes, tables, lengths, 24, scale, scale) + torch.testing.assert_close(got.float(), want.float(), rtol=2e-2, atol=2e-2) + + +@pytest.mark.gpu +@pytest.mark.parametrize("mode", ["async", "multi"]) +@pytest.mark.parametrize("graphs", [False, True]) +def test_fp8_async_and_graph_paths_match_sync_eager(mode, graphs): + # Covers the async finalize kernel (scale copies) and quantized writes inside graphs. + torch.manual_seed(321) + model = OuroForCausalLM(tiny_ouro_config(head_dim=64)).to("cuda", torch.bfloat16) + traces = {str(i): [4, 1, 3, 2, 4, 2, 1, 3, 2][: 5 + i] for i in range(3)} + + def run(execution): + engine = LLMEngine( + model, + cache_config=CacheConfig(num_blocks=64, block_size=16, kv_cache_dtype="fp8_e4m3"), + scheduler_config=SchedulerConfig( + max_num_seqs=3, max_num_batched_tokens=8, prefill_chunk_size=4 + ), + execution_config=execution, + exit_config=ExitConfig("trace", depths_by_request=traces), + attention_backend="triton", + ) + for i in range(3): + engine.add_request( + str(i), + [2 + i, 3, 4] * (i + 1), + SamplingParams(max_tokens=5 + i, min_loops=1, ignore_eos=True), + ) + done = {} + while engine.has_unfinished_requests(): + for output in engine.step(): + if output.finished: + done[output.request_id] = (output.token_ids, list(output.exit_depths)) + assert engine.cache_manager.num_used_blocks == 0 + return done + + reference = run(ExecutionConfig()) + candidate = run( + ExecutionConfig(async_scheduling=True, multi_stream=mode == "multi", cuda_graphs=graphs) + ) + assert candidate == reference + + +@pytest.mark.gpu +@pytest.mark.parametrize("fixed_scale", [None, 0.02]) +def test_triton_fp8_store_matches_quantize_fp8_bitwise(fixed_scale): + from vllm_rlt.kernels.triton_attention import store_fp8 + + torch.manual_seed(2) + pages, layers, block, kv_heads, head_dim, rows, layer = 12, 3, 8, 4, 128, 40, 1 + shape = (pages, layers, block, kv_heads, head_dim) + codes = [torch.zeros(shape, device="cuda", dtype=torch.float8_e4m3fn) for _ in range(2)] + scales = [torch.ones(shape[:-1], device="cuda") for _ in range(2)] + slots = torch.randperm(pages * block, device="cuda")[:rows] + blocks, offsets = slots // block, slots % block + magnitudes = torch.logspace(-30, 4, rows, device="cuda").view(rows, 1, 1) + k = (torch.randn(rows, kv_heads, head_dim, device="cuda") * magnitudes).to(torch.bfloat16) + k[3] = 0 # A zero vector keeps a finite scale and zero codes. + # V rows strided across heads, as a slice of a wider projection would be. + v = torch.randn(rows, kv_heads, 2 * head_dim, device="cuda", dtype=torch.bfloat16) + v = v[..., :head_dim] + store_fp8( + k, + v, + *(t[:, layer] for t in codes), + *(t[:, layer] for t in scales), + blocks, + offsets, + FP8_E4M3_MAX, + fixed_scale=fixed_scale, + ) + untouched = torch.ones(shape[:-1], dtype=torch.bool, device="cuda") + untouched[blocks, layer, offsets] = False + for x, code, scale in zip((k, v), codes, scales): + want_codes, want_scale = quantize_fp8(x, fixed_scale) + assert torch.equal(code[blocks, layer, offsets].float(), want_codes.float()) + assert torch.equal(scale[blocks, layer, offsets], want_scale) + assert not code.float()[untouched].any() and bool((scale[untouched] == 1).all()) + + +@pytest.mark.gpu +@pytest.mark.parametrize("fixed_scale", [None, 0.02]) +@pytest.mark.parametrize("active", [0, 7, 10]) +def test_triton_fp8_store_skips_masked_and_inactive_rows(fixed_scale, active): + # The device-masked writes used by speculative verification and bucketed CUDA graphs. + from vllm_rlt.kernels.triton_attention import store_fp8 + + torch.manual_seed(4) + pages, block, kv_heads, head_dim, rows = 6, 8, 2, 64, 10 + shape = (pages, block, kv_heads, head_dim) + codes = [torch.zeros(shape, device="cuda", dtype=torch.float8_e4m3fn) for _ in range(2)] + scales = [torch.ones(shape[:-1], device="cuda") for _ in range(2)] + slots = torch.randperm(pages * block, device="cuda")[:rows] + blocks, offsets = slots // block, slots % block + k = torch.randn(rows, kv_heads, head_dim, device="cuda", dtype=torch.bfloat16) + v = torch.randn(rows, kv_heads, head_dim, device="cuda", dtype=torch.bfloat16) + mask = torch.tensor([1, 0, 1, 1, 0, 1, 1, 1, 1, 1], device="cuda", dtype=torch.bool) + count = torch.tensor(active, device="cuda", dtype=torch.int32) + store_fp8( + k, + v, + *codes, + *scales, + blocks, + offsets, + FP8_E4M3_MAX, + mask=mask, + active=count, + fixed_scale=fixed_scale, + ) + written = mask.clone() + written[active:] = False + for x, code, scale in zip((k, v), codes, scales): + want_codes, want_scale = quantize_fp8(x, fixed_scale) + assert torch.equal(code[blocks, offsets][written].float(), want_codes[written].float()) + assert torch.equal(scale[blocks, offsets][written], want_scale[written]) + assert not code[blocks, offsets][~written].float().any() + assert bool((scale[blocks, offsets][~written] == 1).all()) + + +@pytest.mark.gpu +@pytest.mark.parametrize("dtype", [torch.bfloat16, torch.float32]) +def test_triton_fp8_prefill_matches_dequantized_prefill(dtype): + from vllm_rlt.kernels.triton_attention import paged_prefill_attention + + torch.manual_seed(5) + # The FP32 dense reference kernel needs ~112 KB of shared memory at head + # dim 128, above the ~99 KB of sm_86/sm_89/sm_120; 64 fits everywhere. + head_dim = 128 if dtype is torch.bfloat16 else 64 + pages, block, kv_heads, heads = 40, 16, 2, 8 + codes, scale = quantize_fp8(torch.randn(pages, block, kv_heads, head_dim, device="cuda")) + # Block 0 is never mapped: its NaN codes (e4m3fn 0x7F) must not reach any output. + codes[0].view(torch.uint8).fill_(0x7F) + dense = dequantize_fp8(codes, scale, dtype) + tables = torch.randint(1, pages, (3, 12), device="cuda", dtype=torch.int32) + lengths = torch.tensor([70, 5, 150], device="cuda", dtype=torch.int32) + query_lengths = torch.tensor([70, 3, 64], device="cuda", dtype=torch.int32) + cu = torch.zeros(4, device="cuda", dtype=torch.int32) + cu[1:] = query_lengths.cumsum(0) + q = torch.randn(int(cu[-1]), heads, head_dim, device="cuda", dtype=dtype) + args = (tables, lengths, cu, int(query_lengths.max())) + got = paged_prefill_attention(q, codes, codes, *args, scale, scale) + want = paged_prefill_attention(q, dense, dense, *args) + assert torch.isfinite(got.float()).all() + torch.testing.assert_close(got.float(), want.float(), rtol=1e-2, atol=1e-2) + + +def run_cuda_engine(model, backend, execution, traces, prompts, kv_cache_dtype="fp8_e4m3"): + engine = LLMEngine( + model, + cache_config=CacheConfig(num_blocks=64, block_size=16, kv_cache_dtype=kv_cache_dtype), + scheduler_config=SchedulerConfig( + max_num_seqs=3, max_num_batched_tokens=64, prefill_chunk_size=32 + ), + execution_config=execution, + exit_config=ExitConfig("trace", depths_by_request=traces), + attention_backend=backend, + ) + for i, prompt in enumerate(prompts): + params = SamplingParams(max_tokens=len(traces[str(i)]), min_loops=1, ignore_eos=True) + engine.add_request(str(i), prompt, params) + done = {} + while engine.has_unfinished_requests(): + for output in engine.step(): + if output.finished: + done[output.request_id] = (output.token_ids, list(output.exit_depths)) + assert engine.cache_manager.num_used_blocks == 0 + return done + + +@pytest.mark.gpu +@pytest.mark.parametrize("graphs", [False, True]) +def test_triton_fp8_engine_matches_torch_fp8_engine(graphs): + # Fused quantized writes and Triton reads, through chunked prefill and decode, against + # the Torch backend's reference writes and reads; FP32 isolates the KV path. + torch.manual_seed(5) + model = OuroForCausalLM(tiny_ouro_config(head_dim=64)).to("cuda") + with torch.no_grad(): + model.lm_head.weight.mul_(8) + prompts = [[2 + (7 * i + n) % 60 for i in range(n)] for n in (5, 41, 100)] + traces = {str(i): [4, 1, 3, 2, 4, 2, 1, 3, 2, 4, 3, 1][: 8 + 2 * i] for i in range(3)} + reference = run_cuda_engine(model, "torch", ExecutionConfig(), traces, prompts) + execution = ExecutionConfig(async_scheduling=graphs, cuda_graphs=graphs) + assert run_cuda_engine(model, "triton", execution, traces, prompts) == reference diff --git a/tests/test_pd.py b/tests/test_pd.py index bb1554c5..df21cb60 100644 --- a/tests/test_pd.py +++ b/tests/test_pd.py @@ -19,7 +19,7 @@ from vllm_rlt.engine.llm_engine import LLMEngine from vllm_rlt.models import OuroForCausalLM from vllm_rlt.pd.config import PDConfig -from vllm_rlt.pd.transport import kv_segments, partition_segments +from vllm_rlt.pd.transport import NixlConnector, kv_segments, partition_segments from vllm_rlt.profiling import Profiler @@ -53,35 +53,26 @@ def test_receive_publishes_all_depths_only_after_commit(): c.mark_imported_prefix("a", 7) -def test_segment_ranges_cover_only_valid_tokens_and_all_layers_depths(): - c = KVCacheManager(3, 2, 8, 24, 4, 4) +@pytest.mark.parametrize("kv_dtype,scale", [("auto", None), ("fp8_e4m3", None), ("fp8_e4m3", 0.5)]) +def test_segment_ranges_cover_only_valid_tokens_and_all_layers_depths(monkeypatch, kv_dtype, scale): + fake_nixl(monkeypatch) + c = KVCacheManager(3, 2, 8, 24, 4, 4, kv_cache_dtype=kv_dtype, fp8_scale=scale) c.allocate("a", 9) - k = c.key_cache - info = dict( - block_size=4, - num_blocks=24, - num_layers=3, - device=0, - block_bytes=k.stride(0) * 4, - layer_bytes=k.stride(1) * 4, - token_bytes=k.stride(2) * 4, - key_ptr=0, - value_ptr=10**9, - ) + connector = NixlConnector("stage", c, torch.zeros(2, 16), PDConfig()) + tensors = (c.key_cache, c.value_cache) + if kv_dtype == "fp8_e4m3" and scale is None: + tensors += (c.key_scale, c.value_scale) + assert {t.data_ptr() for t in tensors} <= {t.data_ptr() for t in connector.storage} + assert len(connector.storage) == len(tensors) + 1 # includes hidden state tables = c.plane_block_tables("a") expected = set() for table in tables: for token in range(2, 9): for layer in range(3): - for base in (0, 10**9): - pos = ( - base - + table[token // 4] * info["block_bytes"] - + layer * info["layer_bytes"] - + (token % 4) * info["token_bytes"] - ) - expected.update(range(pos, pos + info["token_bytes"])) - segments = list(kv_segments(info, tables, 2, 9)) + for tensor in tensors: + row = tensor[table[token // 4], layer, token % 4] + expected.update(range(row.data_ptr(), row.data_ptr() + row.nbytes)) + segments = list(kv_segments(connector.info, tables, 2, 9)) actual = set() for address, size, _ in segments: actual.update(range(address, address + size)) @@ -96,6 +87,13 @@ def test_segment_ranges_cover_only_valid_tokens_and_all_layers_depths(): assert covered == expected +def test_pd_rejects_mismatched_kv_dtypes_before_launching_workers(): + from vllm_rlt.pd.engine import PDEngine + + with pytest.raises(ValueError, match="same KV cache dtype"): + PDEngine(tiny_ouro_config(), prefill_cache_config=CacheConfig(kv_cache_dtype="fp8_e4m3")) + + def test_partition_merges_segments_contiguous_on_both_sides(): # Three adjacent 100-byte blocks on both pools collapse into one range; a target # that is not adjacent at the same boundary keeps its own descriptor. @@ -353,13 +351,22 @@ def create_pd( speculative=False, asynchronous=True, prefix_caching=False, + backend="triton", + kv_dtype="auto", ): pytest.importorskip("nixl") from vllm_rlt.pd.engine import PDEngine if torch.cuda.device_count() < (4 if multi else 2): pytest.skip("requires 2/4 visible GPUs") + if backend == "flash_attn_4": + pytest.importorskip("flash_attn.cute") + if any(torch.cuda.get_device_capability(i)[0] != 10 for i in (0, 1)): + pytest.skip("FA4 FP8 requires SM10x on both workers") config = tiny_ouro_config(head_dim=64) + cache = CacheConfig( + 128, 4, layout, enable_prefix_caching=prefix_caching, kv_cache_dtype=kv_dtype + ) return PDEngine( config, pd_config=PDConfig( @@ -370,8 +377,8 @@ def create_pd( request_timeout=90, startup_timeout=90, ), - prefill_cache_config=CacheConfig(128, 4, layout, enable_prefix_caching=prefix_caching), - decode_cache_config=CacheConfig(128, 4, layout, enable_prefix_caching=prefix_caching), + prefill_cache_config=cache, + decode_cache_config=cache, prefill_scheduler_config=SchedulerConfig( mode=mode, max_num_seqs=2, @@ -390,28 +397,35 @@ def create_pd( pad_to_power_of_two=graph, cuda_graphs=graph, ), - attention_backend="triton", + attention_backend=backend, seed=123, ) @pytest.mark.gpu @pytest.mark.parametrize( - "graph,layout", [(False, "last_exited"), (True, "last_exited"), (False, "shared")] + "graph,layout,backend,kv_dtype", + [ + (False, "last_exited", "triton", "auto"), + (True, "last_exited", "triton", "auto"), + (False, "shared", "triton", "auto"), + (False, "last_exited", "triton", "fp8_e4m3"), + (True, "last_exited", "flash_attn_4", "fp8_e4m3"), + ], ) -def test_pd_generation_refill_cancel_and_reuse_matches_local(graph, layout): +def test_pd_generation_refill_cancel_and_reuse_matches_local(graph, layout, backend, kv_dtype): params = SamplingParams(max_tokens=4, min_loops=1, ignore_eos=True) - with create_pd(graph=graph, layout=layout) as e: + with create_pd(graph=graph, layout=layout, backend=backend, kv_dtype=kv_dtype) as e: torch.manual_seed(123) model = OuroForCausalLM(tiny_ouro_config(head_dim=64)).to("cuda:0", torch.bfloat16) reference = LLMEngine( model, - cache_config=CacheConfig(128, 4, layout), + cache_config=CacheConfig(128, 4, layout, kv_cache_dtype=kv_dtype), scheduler_config=SchedulerConfig( max_num_seqs=2, max_num_batched_tokens=5, prefill_chunk_size=3 ), exit_config=ExitConfig("trace", depths_by_request={"t": [4, 2, 3, 1] * 16}), - attention_backend="triton", + attention_backend=backend, ) prompts = [[1, 2, 3, 4, 5, 6, 7], [7, 6, 5, 4, 3], [2] * 9] for i, prompt in enumerate(prompts): @@ -825,8 +839,6 @@ def agent_config(**kwargs): def test_ucx_backend_is_created_with_error_handling_none(monkeypatch): - from vllm_rlt.pd.transport import NixlConnector - created = fake_nixl(monkeypatch) # No hidden switch: an obsolete environment variable cannot change the mode. monkeypatch.setenv("VLLM_RLT_NIXL_UCX_ERROR_HANDLING", "peer") @@ -840,10 +852,17 @@ def test_ucx_backend_is_created_with_error_handling_none(monkeypatch): def test_non_ucx_backend_is_not_created_with_ucx_parameters(monkeypatch): - from vllm_rlt.pd.transport import NixlConnector - created = fake_nixl(monkeypatch) cache = KVCacheManager(3, 2, 8, 24, 4, 4) NixlConnector("stage", cache, torch.zeros(2, 16), PDConfig(backend="GDS")) assert created["agent_config"] == {"backends": ["GDS"]} assert "backend" not in created + + +def test_pd_handshake_rejects_different_fixed_scales(monkeypatch): + fake_nixl(monkeypatch) + cache = KVCacheManager(3, 2, 8, 24, 4, 4, kv_cache_dtype="fp8_e4m3", fp8_scale=0.5) + connector = NixlConnector("decode", cache, torch.zeros(2, 16), PDConfig()) + assert torch.all(cache.key_scale == 0.5) and torch.all(cache.value_scale == 0.5) + with pytest.raises(ValueError, match="same calibrated FP8 scale"): + connector.connect(dict(connector.info, fp8_scale=1.0)) diff --git a/vllm_rlt/attention/backend.py b/vllm_rlt/attention/backend.py index fcc58278..6a1bf8bb 100644 --- a/vllm_rlt/attention/backend.py +++ b/vllm_rlt/attention/backend.py @@ -25,6 +25,10 @@ class BackendCapabilities: # Packed rows reproduce single-query decode exactly; speculative verification # relies on this to match serial decoding token for token. packed_matches_decode: bool = False + # Reads an FP8 e4m3 KV pool with per-vector float32 scales. + fp8_kv: bool = False + # Reads FP8 KV quantized using one fixed scalar for the cache lifetime. + fp8_kv_static: bool = False # Consume GPU-owned context lengths without a host read between submissions. device_context_lengths: bool = False # Mask keys outside a per-layer sliding window (not storage reclamation). @@ -60,13 +64,26 @@ def generation(self): return getattr(self.kernel, "generation", None) -_TORCH_CAPABILITIES = BackendCapabilities(sliding_window=True) +_TORCH_CAPABILITIES = BackendCapabilities(fp8_kv=True, fp8_kv_static=True, sliding_window=True) _TRITON_CAPABILITIES = BackendCapabilities( - cuda_graphs=True, async_scheduling=True, device_context_lengths=True, sliding_window=True + cuda_graphs=True, + async_scheduling=True, + device_context_lengths=True, + fp8_kv=True, + fp8_kv_static=True, + sliding_window=True, ) _FLASH_CAPABILITIES = { - 2: BackendCapabilities(cuda_graphs=True, async_scheduling=True, device_context_lengths=True), - 3: BackendCapabilities(cuda_graphs=True, async_scheduling=True, device_context_lengths=True), + 2: BackendCapabilities( + cuda_graphs=True, + async_scheduling=True, + device_context_lengths=True, + ), + 3: BackendCapabilities( + cuda_graphs=True, + async_scheduling=True, + device_context_lengths=True, + ), 4: BackendCapabilities( packed_prefill=True, cuda_graphs=True, @@ -118,7 +135,13 @@ def _create_explicit(backend, device, dtype, head_dim, block_size, *, packed, pa ) if backend in FLASH_BACKENDS: kernel = FlashPagedAttention(device, dtype, head_dim, block_size, backend) - return AttentionBackend(kernel, _FLASH_CAPABILITIES[kernel.generation], dict(kernel.info)) + capabilities = replace( + _FLASH_CAPABILITIES[kernel.generation], + fp8_kv_static=kernel.supports_fp8_static, + sliding_window=kernel.supports_sliding_window, + packed_sliding_window=kernel.supports_sliding_window, + ) + return AttentionBackend(kernel, capabilities, dict(kernel.info)) return AttentionBackend(TorchPagedAttention(), _TORCH_CAPABILITIES, {"backend": "torch"}) @@ -132,6 +155,7 @@ def create_backend( required_capabilities: BackendCapabilities = BackendCapabilities(), prefer_packed_prefill: bool = False, packed_window_shape: PackedWindowShape | None = None, + allow_fp8_calibration: bool = False, ) -> AttentionBackend: """Explicit choices fail without fallback; auto tries FA4/3/2, Triton, then Torch. @@ -142,6 +166,8 @@ def create_backend( Preferred packed prefill enables it where available without filtering or reordering candidates; required packed prefill still rejects unsupported ones. A packed window shape opts into tile selection and keeps row attention available. + Engine callers that can calibrate may satisfy an FP8 KV request with a + fixed-scale backend. Cache-only callers must provide a resolved scale. """ validate_backend_name(backend) candidates = _AUTO_PRIORITY if backend == "auto" else (backend,) @@ -160,6 +186,20 @@ def create_backend( packed=required_capabilities.packed_prefill or prefer_packed_prefill, packed_window_shape=packed_window_shape, ) + capabilities = implementation.capabilities + requirements = required_capabilities + if ( + allow_fp8_calibration + and requirements.fp8_kv + and not capabilities.fp8_kv + and capabilities.fp8_kv_static + ): + requirements = replace(requirements, fp8_kv=False, fp8_kv_static=True) + if requirements.fp8_kv_static and candidate in FLASH_BACKENDS: + # Native FP8 packed-vs-decode bitwise equality is not established. + implementation.capabilities = replace( + implementation.capabilities, packed_matches_decode=False + ) missing = [ name for name in ( @@ -167,11 +207,12 @@ def create_backend( "cuda_graphs", "async_scheduling", "device_context_lengths", + "fp8_kv", + "fp8_kv_static", "sliding_window", "packed_sliding_window", ) - if getattr(required_capabilities, name) - and not getattr(implementation.capabilities, name) + if getattr(requirements, name) and not getattr(implementation.capabilities, name) ] if missing: raise ValueError(f"{candidate} lacks required capabilities: {', '.join(missing)}") diff --git a/vllm_rlt/config.py b/vllm_rlt/config.py index 1619ba80..7213adb0 100644 --- a/vllm_rlt/config.py +++ b/vllm_rlt/config.py @@ -20,6 +20,9 @@ class CacheConfig: incremental_allocation: bool = False # Admission headroom for incremental allocation; otherwise effective watermark is zero. watermark_ratio: float = 0.0 + # "auto" stores KV in the model dtype; "fp8_e4m3" stores float8 e4m3 codes with + # scales chosen by the engine for the selected attention backend. + kv_cache_dtype: str = "auto" def __post_init__(self): for name in ("enable_prefix_caching", "incremental_allocation"): @@ -34,6 +37,8 @@ def __post_init__(self): _positive("block_size", self.block_size) if self.layout not in ("last_exited", "shared"): raise ValueError("layout must be 'last_exited' or 'shared'") + if self.kv_cache_dtype not in ("auto", "fp8_e4m3"): + raise ValueError("kv_cache_dtype must be 'auto' or 'fp8_e4m3'") if ( not math.isfinite(self.gpu_memory_utilization) or not 0 < self.gpu_memory_utilization <= 1 diff --git a/vllm_rlt/core/kv_cache_manager.py b/vllm_rlt/core/kv_cache_manager.py index ec59c9cc..04a9c768 100644 --- a/vllm_rlt/core/kv_cache_manager.py +++ b/vllm_rlt/core/kv_cache_manager.py @@ -4,6 +4,10 @@ SHARED overwrites a single plane and needs no exit copies. Optional incremental allocation separates logical capacity from physical pages. Immutable full-depth prompt blocks can be shared through the prefix cache. + +With ``kv_cache_dtype="fp8_e4m3"`` the pool stores K/V as float8 e4m3 with one +float32 scale per (block, layer, token, KV head) vector: values are quantized on +write and dequantized by the attention backends on read. """ import hashlib @@ -27,6 +31,37 @@ from vllm_rlt.worker.buffers import stage_metadata from vllm_rlt.worker.metadata import DeviceKVMetadata, KVMetadata, MetadataBuilder +KV_CACHE_DTYPES = ("auto", "fp8_e4m3") +FP8_E4M3_MAX = 448.0 + + +def kv_bytes_per_block( + num_layers, block_size, num_kv_heads, head_dim, element_size, kv_cache_dtype +): + """Physical bytes of one block (K and V for all layers), including FP8 scales.""" + if kv_cache_dtype not in KV_CACHE_DTYPES: + raise ValueError(f"kv_cache_dtype must be one of {KV_CACHE_DTYPES}") + vector = head_dim * element_size if kv_cache_dtype == "auto" else head_dim + 4 + return 2 * num_layers * block_size * num_kv_heads * vector + + +def quantize_fp8(x: torch.Tensor, fixed_scale=None) -> tuple[torch.Tensor, torch.Tensor]: + """Symmetric e4m3 with dynamic per-vector scales or an explicit fixed scale.""" + x = x.float() + # A reciprocal multiply, as CUDA division by a scalar does anyway: CPU, CUDA and + # the fused Triton writer then produce the same scale bits. + scale = ( + (x.abs().amax(-1) * (1.0 / FP8_E4M3_MAX)).clamp_min(torch.finfo(torch.float32).tiny) + if fixed_scale is None + else torch.full(x.shape[:-1], fixed_scale, dtype=torch.float32, device=x.device) + ) + codes = (x / scale.unsqueeze(-1)).clamp(-FP8_E4M3_MAX, FP8_E4M3_MAX) + return codes.to(torch.float8_e4m3fn), scale + + +def dequantize_fp8(codes: torch.Tensor, scale: torch.Tensor, dtype: torch.dtype) -> torch.Tensor: + return (codes.float() * scale.unsqueeze(-1)).to(dtype) + @dataclass class _WrittenPositions: @@ -75,6 +110,10 @@ class KVSnapshot: written: list[list[_WrittenPositions]] keys: torch.Tensor values: torch.Tensor + # FP8 pools only: per-vector scales, laid out like keys/values minus head_dim. + key_scales: torch.Tensor | None = None + value_scales: torch.Tensor | None = None + fp8_scale: float | None = None class KVCacheManager: @@ -103,6 +142,8 @@ def __init__( recurrent_layers: Sequence[int] | None = None, *, attention: AttentionBackend | None = None, + kv_cache_dtype: str = "auto", + fp8_scale: float | None = None, ): for name, value in ( ("num_layers", num_layers), @@ -120,6 +161,19 @@ def __init__( raise ValueError("KV dtype must be float32, float16, or bfloat16") if layout not in {"last_exited", "shared"}: raise ValueError("unsupported KV layout") + if kv_cache_dtype not in KV_CACHE_DTYPES: + raise ValueError(f"kv_cache_dtype must be one of {KV_CACHE_DTYPES}") + self.kv_cache_dtype = kv_cache_dtype + if fp8_scale is not None: + if kv_cache_dtype != "fp8_e4m3": + raise ValueError("fp8_scale requires kv_cache_dtype='fp8_e4m3'") + # Keep the value finite and normal after conversion to device float32. + if ( + type(fp8_scale) not in (int, float) + or not 2**-126 <= fp8_scale <= (2 - 2**-23) * 2**127 + ): + raise ValueError("fp8_scale must be a positive, finite, normal float32 value") + self.fp8_scale = fp8_scale self.layout = layout self.recurrent_layers = ( tuple(range(num_layers)) if recurrent_layers is None else tuple(recurrent_layers) @@ -136,13 +190,42 @@ def __init__( self.device = torch.device(device) self.dtype = dtype self.attention = attention or create_backend( - backend, self.device, dtype, head_dim, block_size + backend, + self.device, + dtype, + head_dim, + block_size, + required_capabilities=BackendCapabilities( + fp8_kv=kv_cache_dtype == "fp8_e4m3" and fp8_scale is None, + fp8_kv_static=fp8_scale is not None, + ), ) self.attention_info = self.attention.info self.backend = self.attention_info["selected_backend"] + capability = "fp8_kv" if fp8_scale is None else "fp8_kv_static" + if kv_cache_dtype != "auto" and not getattr(self.attention.capabilities, capability): + raise ValueError( + f"the {self.attention_info['backend']} backend does not read FP8 KV " + f"with {'per-vector' if fp8_scale is None else 'fixed'} scales" + ) shape = (num_blocks, num_layers, block_size, num_kv_heads, head_dim) - self.key_cache = torch.empty(shape, device=self.device, dtype=dtype) + storage = torch.float8_e4m3fn if kv_cache_dtype == "fp8_e4m3" else dtype + self.key_cache = torch.empty(shape, device=self.device, dtype=storage) self.value_cache = torch.empty_like(self.key_cache) + # One scale per stored vector; None for an unquantized pool. + self.key_scale = self.value_scale = None + if kv_cache_dtype == "fp8_e4m3": + # Imported fixed-scale pages use the same local scale as locally written pages. + scale = 1.0 if fp8_scale is None else fp8_scale + self.key_scale = torch.full(shape[:-1], scale, device=self.device, dtype=torch.float32) + self.value_scale = torch.full_like(self.key_scale, scale) + # Keep the existing per-vector storage/lifecycle for this prototype. FA + # borrows the immutable scalar; Torch/Triton continue reading the scale pool. + self._fixed_scale = ( + torch.tensor(fp8_scale, device=self.device, dtype=torch.float32) + if fp8_scale is not None + else None + ) # Resolve an implicit CUDA index to the actual storage device once. self.device = self.key_cache.device self._free_blocks = list(reversed(range(num_blocks))) @@ -174,13 +257,13 @@ def num_used_blocks(self) -> int: @property def bytes_per_block(self) -> int: - return ( - 2 - * self.num_layers - * self.block_size - * self.num_kv_heads - * self.head_dim - * self.key_cache.element_size() + return kv_bytes_per_block( + self.num_layers, + self.block_size, + self.num_kv_heads, + self.head_dim, + self.key_cache.element_size(), + self.kv_cache_dtype, ) def required_blocks(self, max_tokens: int) -> int: @@ -489,17 +572,24 @@ def snapshot(self, request_id: str) -> KVSnapshot: if allocation.transfer_leases: raise RuntimeError("cannot snapshot KV during a transfer") blocks = [b for table in allocation.block_tables for b in table] - keys = torch.empty((len(blocks), *self.key_cache.shape[1:]), dtype=self.dtype) + keys = torch.empty((len(blocks), *self.key_cache.shape[1:]), dtype=self.key_cache.dtype) values = torch.empty_like(keys) for row, block in enumerate(blocks): keys[row].copy_(self.key_cache[block]) values[row].copy_(self.value_cache[block]) + key_scales = value_scales = None + if self.key_scale is not None: + key_scales = self.key_scale[blocks].cpu() + value_scales = self.value_scale[blocks].cpu() return KVSnapshot( max_tokens=allocation.max_tokens, pages=len(allocation.block_tables[0]), written=deepcopy(allocation.written), keys=keys, values=values, + key_scales=key_scales, + value_scales=value_scales, + fp8_scale=self.fp8_scale, ) def restore(self, request_id: str, snapshot: KVSnapshot) -> None: @@ -516,10 +606,17 @@ def restore(self, request_id: str, snapshot: KVSnapshot) -> None: raise ValueError("snapshot page count does not match the allocation") if any(w.prefix or w.pending for plane in allocation.written for w in plane): raise RuntimeError("cannot restore into an allocation that already holds KV") + if (self.key_scale is None) != (snapshot.key_scales is None): + raise ValueError("snapshot KV dtype does not match the pool") + if self.fp8_scale != snapshot.fp8_scale: + raise ValueError("snapshot FP8 scale policy does not match the pool") blocks = [b for table in allocation.block_tables for b in table] for row, block in enumerate(blocks): self.key_cache[block].copy_(snapshot.keys[row]) self.value_cache[block].copy_(snapshot.values[row]) + if self.key_scale is not None: + self.key_scale[block].copy_(snapshot.key_scales[row]) + self.value_scale[block].copy_(snapshot.value_scales[row]) allocation.written = deepcopy(snapshot.written) def _plane(self, depth: int) -> int: @@ -681,7 +778,11 @@ def _write_prepared( if isinstance(batch, DeviceKVMetadata): self._validate_tensor(k, len(batch.position_ids), "k") self._validate_tensor(v, len(batch.position_ids), "v") - if batch.write_mask is not None: + if self.key_scale is not None: + self._store_layer( + layer, batch.write_blocks, batch.write_offsets, k, v, mask=batch.write_mask + ) + elif batch.write_mask is not None: from vllm_rlt.kernels.routing import masked_kv_store masked_kv_store(self.key_cache[:, layer], self.value_cache[:, layer], k, v, batch) @@ -695,11 +796,64 @@ def _write_prepared( self._validate_tensor(v, len(batch.position_ids), "v") if not batch.rows: return - self.key_cache[batch.write_blocks, layer, batch.write_offsets] = k[: len(batch.rows)] - self.value_cache[batch.write_blocks, layer, batch.write_offsets] = v[: len(batch.rows)] + rows = len(batch.rows) + self._store_layer(layer, batch.write_blocks, batch.write_offsets, k[:rows], v[:rows]) for allocation, depth, position in batch.rows: allocation.written[self._plane(depth)][layer].add(position) + def _store_layer(self, layer, blocks, offsets, k, v, *, mask=None, active=None) -> None: + """Write one layer's K/V rows, quantizing them for an FP8 pool. + + FP8 only: ``mask`` (bool [rows]) and ``active`` (int32 row count) skip + rows on the device, like the unquantized masked and graph writers. + """ + if self.key_scale is None: + if mask is not None or active is not None: + raise ValueError("row masks are handled by the unquantized device writers") + self.key_cache[blocks, layer, offsets] = k + self.value_cache[blocks, layer, offsets] = v + return + if ( + self.backend == "triton" + or mask is not None + or active is not None + or (self.fp8_scale is not None and self.device.type == "cuda") + ): + # One fused launch instead of about 20 small kernels per layer write. + # Masked writes always take it, as the unquantized masked writers are + # Triton too and boolean indexing would read the mask on the host. + from vllm_rlt.kernels.triton_attention import store_fp8 + + store_fp8( + k, + v, + self.key_cache[:, layer], + self.value_cache[:, layer], + self.key_scale[:, layer], + self.value_scale[:, layer], + blocks, + offsets, + FP8_E4M3_MAX, + mask=mask, + active=active, + fixed_scale=self.fp8_scale, + ) + return + k_codes, k_scale = quantize_fp8(k, self.fp8_scale) + v_codes, v_scale = quantize_fp8(v, self.fp8_scale) + self.key_cache[blocks, layer, offsets] = k_codes + self.value_cache[blocks, layer, offsets] = v_codes + self.key_scale[blocks, layer, offsets] = k_scale + self.value_scale[blocks, layer, offsets] = v_scale + + def _layer_scales(self, layer) -> dict: + """Extra attention arguments for one layer: its scales, or nothing.""" + if self.key_scale is None: + return {} + if self._fixed_scale is not None and not self.attention.capabilities.fp8_kv: + return dict(key_scale=self._fixed_scale, value_scale=self._fixed_scale) + return dict(key_scale=self.key_scale[:, layer], value_scale=self.value_scale[:, layer]) + def _require_prefix(self, allocation, layer, depth, length): written = allocation.written[self._plane(depth)][layer] if length > written.prefix: @@ -755,6 +909,7 @@ def _attend_prepared( self.value_cache[:, layer], batch.block_tables, batch.context_lengths, + **self._layer_scales(layer), ) if not batch.rows: return torch.empty_like(q) @@ -769,6 +924,7 @@ def _attend_prepared( batch.context_lengths, batch.cu_seqlens_q, batch.max_seqlen_q, + **self._layer_scales(layer), **({} if window is None else {"window": window}), ) return self.attention( @@ -777,6 +933,7 @@ def _attend_prepared( self.value_cache[:, layer], batch.block_tables, batch.context_lengths, + **self._layer_scales(layer), ) @torch.no_grad() @@ -794,8 +951,6 @@ def _attend_windowed( raise NotImplementedError( f"the {self.backend} attention backend does not implement sliding-window masking" ) - if batch.cu_seqlens_q is not None: - raise NotImplementedError("sliding-window attention needs one query row per token") # Device-owned positions are ordered by the runner's stream; the host # frontier lags them, as for unwindowed attention. if not isinstance(batch, DeviceKVMetadata): @@ -810,6 +965,7 @@ def _attend_windowed( batch.block_tables, batch.context_lengths, window, + **self._layer_scales(layer), ) @torch.no_grad() @@ -826,18 +982,16 @@ def finalize_token(self, request_id: str, position: int, exit_depth: int) -> Non ] # Basic-index copies avoid a blocking host->device index tensor on the # boundary stream. That transfer would serialize final core and routing. + tensors = [self.key_cache, self.value_cache] + if self.key_scale is not None: + tensors += [self.key_scale, self.value_scale] for destination in destinations: - if len(self.recurrent_layers) == self.num_layers: - self.key_cache[destination, :, offset].copy_(self.key_cache[source, :, offset]) - self.value_cache[destination, :, offset].copy_(self.value_cache[source, :, offset]) - else: - for layer in self.recurrent_layers: - self.key_cache[destination, layer, offset].copy_( - self.key_cache[source, layer, offset] - ) - self.value_cache[destination, layer, offset].copy_( - self.value_cache[source, layer, offset] - ) + for tensor in tensors: + if len(self.recurrent_layers) == self.num_layers: + tensor[destination, :, offset].copy_(tensor[source, :, offset]) + else: + for layer in self.recurrent_layers: + tensor[destination, layer, offset].copy_(tensor[source, layer, offset]) self._record_finalized(allocation, position, exit_depth) @torch.no_grad() @@ -861,4 +1015,11 @@ def read(self, layer: int, request_id: str, depth: int, length: int | None = Non ) blocks = table[positions // self.block_size] offsets = positions % self.block_size - return self.key_cache[blocks, layer, offsets], self.value_cache[blocks, layer, offsets] + keys = self.key_cache[blocks, layer, offsets] + values = self.value_cache[blocks, layer, offsets] + if self.key_scale is None: + return keys, values + return ( + dequantize_fp8(keys, self.key_scale[blocks, layer, offsets], self.dtype), + dequantize_fp8(values, self.value_scale[blocks, layer, offsets], self.dtype), + ) diff --git a/vllm_rlt/core/memory.py b/vllm_rlt/core/memory.py index 1f7ed1e0..9b87ca2c 100644 --- a/vllm_rlt/core/memory.py +++ b/vllm_rlt/core/memory.py @@ -5,7 +5,7 @@ import torch from vllm_rlt.config import cuda_graph_rows -from vllm_rlt.core.kv_cache_manager import KVCacheManager +from vllm_rlt.core.kv_cache_manager import FP8_E4M3_MAX, KVCacheManager, kv_bytes_per_block from vllm_rlt.worker.buffers import execution_buffer_bytes from vllm_rlt.worker.cuda_graph import speculative_capture_sizes @@ -50,22 +50,104 @@ def graph_memory_reserve(model, scheduler, execution, speculative=None): return pools * execution.cuda_graph_memory_reserve_bytes + retained_logits +def _probe_forward(model, cache, tokens, ids, positions): + count = len(ids) + boundary = None + if getattr(model, "requires_boundary_kv", False): + boundary = cache._prepare_batch(ids, [0] * count, positions) + hidden = model.prelude_prepared(tokens, boundary, cache) + else: + hidden = model.prelude(tokens) + for depth in range(model.config.total_ut_steps): + hidden, _ = model.recurrent(hidden, ids, [depth] * count, positions, cache) + if boundary is not None: + return model.coda_prepared(hidden, boundary, cache) + return model.coda(hidden) + + +class _CalibrationCache(KVCacheManager): + """Temporary model-dtype cache that records Q/K/V ranges at the kernel boundary.""" + + def __init__(self, **kwargs): + super().__init__(**kwargs) + self.maxima = [] + + def _store_layer(self, layer, blocks, offsets, k, v, **kwargs): + self.maxima.extend((k.abs().amax(), v.abs().amax())) + super()._store_layer(layer, blocks, offsets, k, v, **kwargs) + + def _attend_prepared(self, layer, batch, q, **kwargs): + self.maxima.append(q.abs().amax()) + return super()._attend_prepared(layer, batch, q, **kwargs) + + +@torch.inference_mode() +def calibrate_fp8_scale(model, cache, attention) -> float: + """Run bounded BF16 prefill and one decode input before allocating the FP8 pool.""" + parameter = next(model.parameters()) + config = model.config + # P and D use different scheduler limits but must measure the same input. + length = min(128, max(1, config.max_position_embeddings - 1)) + capacity = min(length + 1, config.max_position_embeddings) + num_blocks = math.ceil(capacity / cache.block_size) + if cache.layout == "last_exited": + num_blocks *= config.total_ut_steps + probe = _CalibrationCache( + num_layers=getattr(config, "num_kv_layers", config.num_hidden_layers), + num_kv_heads=config.num_key_value_heads, + head_dim=config.head_dim, + max_loops=config.total_ut_steps, + num_blocks=num_blocks, + block_size=cache.block_size, + layout=cache.layout, + device=parameter.device, + dtype=parameter.dtype, + attention=attention, + recurrent_layers=getattr(model, "recurrent_kv_layers", None), + ) + request = "fp8-calibration" + probe.allocate(request, capacity) + # Sample vocabulary IDs without touching the engine's or the caller's RNG. + generator = torch.Generator(device="cpu").manual_seed(0) + tokens = torch.randint(config.vocab_size, (length,), generator=generator, device="cpu") + logits = _probe_forward( + model, probe, tokens.to(parameter.device), [request] * length, list(range(length)) + ) + if capacity > length: + _probe_forward(model, probe, logits[-1:].argmax(dim=-1), [request], [length]) + amax = torch.stack(probe.maxima).amax().item() + if not math.isfinite(amax): + raise ValueError("FP8 startup calibration encountered non-finite Q/K/V values") + # Leave headroom for inputs outside this brief startup sample. Serve with a + # single immutable scale so existing KV pages keep their numerical meaning. + scale = max(amax * 1.1 / FP8_E4M3_MAX, torch.finfo(torch.float32).tiny) + return 2.0 ** math.ceil(math.log2(scale)) + + @torch.inference_mode() def plan_cache( - model, cache, scheduler, execution, backend, *, speculative=None, staged_prefill=None + model, + cache, + scheduler, + execution, + backend, + *, + speculative=None, + staged_prefill=None, + fp8_scale=None, ): parameter = next(model.parameters()) config, device = model.config, parameter.device # Models whose physical stack mixes KV-owning and KV-free sublayers (Dory) # expose the compact count; everything else keeps the layer count. kv_layers = getattr(config, "num_kv_layers", config.num_hidden_layers) - per_block = ( - 2 - * kv_layers - * cache.block_size - * config.num_key_value_heads - * config.head_dim - * parameter.element_size() + per_block = kv_bytes_per_block( + kv_layers, + cache.block_size, + config.num_key_value_heads, + config.head_dim, + parameter.element_size(), + cache.kv_cache_dtype, ) if cache.num_blocks is not None: return cache.num_blocks, {"source": "blocks", "bytes_per_block": per_block} @@ -99,12 +181,15 @@ def plan_cache( head_dim=config.head_dim, max_loops=config.total_ut_steps, num_blocks=sum(math.ceil(length / cache.block_size) for length in lengths) - * config.total_ut_steps, + * (config.total_ut_steps if cache.layout == "last_exited" else 1), block_size=cache.block_size, + layout=cache.layout, device=device, dtype=parameter.dtype, backend=backend, recurrent_layers=getattr(model, "recurrent_kv_layers", None), + kv_cache_dtype=cache.kv_cache_dtype, + fp8_scale=fp8_scale, ) ids, positions = [], [] for index, length in enumerate(lengths): @@ -113,21 +198,10 @@ def plan_cache( ids.extend([rid] * length) positions.extend(range(length)) tokens = torch.zeros(count, dtype=torch.long, device=device) - if getattr(model, "requires_boundary_kv", False): - boundary = probe._prepare_batch(ids, [0] * count, positions) - hidden = model.prelude_prepared(tokens, boundary, probe) - else: - hidden = model.prelude(tokens) - for depth in range(config.total_ut_steps): - hidden, _ = model.recurrent(hidden, ids, [depth] * count, positions, probe) - logits = ( - model.coda_prepared(hidden, boundary, probe) - if getattr(model, "requires_boundary_kv", False) - else model.coda(hidden) - ) + logits = _probe_forward(model, probe, tokens, ids, positions) torch.cuda.synchronize(device) peak = max(0, torch.cuda.max_memory_allocated(device) - allocated) - del hidden, logits, probe + del logits, probe # Profiling includes a conservative temporary KV allocation. Also budget for # long-context reference attention (Triton's online softmax has bounded scratch). context_scratch = 0 diff --git a/vllm_rlt/engine/stage_engine.py b/vllm_rlt/engine/stage_engine.py index ce700908..ec40c771 100644 --- a/vllm_rlt/engine/stage_engine.py +++ b/vllm_rlt/engine/stage_engine.py @@ -11,7 +11,7 @@ resolve_cuda_graph_rows, ) from vllm_rlt.core.kv_cache_manager import KVCacheManager -from vllm_rlt.core.memory import plan_cache +from vllm_rlt.core.memory import calibrate_fp8_scale, plan_cache from vllm_rlt.core.scheduler import Scheduler from vllm_rlt.core.scheduling_policy import requires_staged_prefill from vllm_rlt.engine.connector import StageConnector @@ -191,7 +191,11 @@ def __init__( window = getattr(config, "sliding_window", None) packed_window_shape = ( PackedWindowShape( - config.num_attention_heads, config.num_key_value_heads, kv_layers, window + config.num_attention_heads, + config.num_key_value_heads, + kv_layers, + window, + fp8_kv=cache_config.kv_cache_dtype == "fp8_e4m3", ) if prefer_packed and sliding_window else None @@ -210,11 +214,19 @@ def __init__( parameter.device.type == "cuda" and self.execution_config.async_scheduling ), device_context_lengths=self.async_speculative, + fp8_kv=cache_config.kv_cache_dtype == "fp8_e4m3", sliding_window=sliding_window, ), prefer_packed_prefill=prefer_packed and not sliding_window, packed_window_shape=packed_window_shape, + allow_fp8_calibration=True, ) + fp8_scale = None + if cache_config.kv_cache_dtype == "fp8_e4m3" and not attention.capabilities.fp8_kv: + fp8_scale = calibrate_fp8_scale(model, cache_config, attention) + logger.info( + "Calibrated FP8 scale %g for %s", fp8_scale, attention.info["selected_backend"] + ) if self.execution_config.packed_prefill is None: supported = attention.capabilities.packed_prefill and ( not sliding_window or attention.capabilities.packed_sliding_window @@ -230,6 +242,7 @@ def __init__( attention.info["selected_backend"], speculative=speculative_config, staged_prefill=staged_prefill, + fp8_scale=fp8_scale, ) self.cache_manager = KVCacheManager( num_layers=kv_layers, @@ -247,6 +260,8 @@ def __init__( incremental_allocation=cache_config.incremental_allocation, watermark_ratio=cache_config.watermark_ratio, recurrent_layers=getattr(model, "recurrent_kv_layers", None), + kv_cache_dtype=cache_config.kv_cache_dtype, + fp8_scale=fp8_scale, ) self.scheduler = Scheduler( scheduler_config, diff --git a/vllm_rlt/entrypoints/runtime_args.py b/vllm_rlt/entrypoints/runtime_args.py index 49d2b13f..679a198c 100644 --- a/vllm_rlt/entrypoints/runtime_args.py +++ b/vllm_rlt/entrypoints/runtime_args.py @@ -45,6 +45,12 @@ def add_runtime_args(parser): help="Copy metadata to the GPU instead of reading mapped host memory", ) parser.add_argument("--kv-layout", choices=["last_exited", "shared"], default="last_exited") + parser.add_argument( + "--kv-cache-dtype", + choices=["auto", "fp8_e4m3"], + default="auto", + help="FP8 KV: per-vector scales on Torch/Triton, startup calibration on FA4", + ) parser.add_argument("--gpu-memory-utilization", type=float, default=0.9) parser.add_argument("--kv-cache-memory-bytes", type=int) parser.add_argument("--memory-reserve-bytes", type=int, default=256 * 1024 * 1024) @@ -115,6 +121,7 @@ def runtime_configs(args): num_blocks=args.num_blocks, block_size=args.block_size, layout=args.kv_layout, + kv_cache_dtype=args.kv_cache_dtype, gpu_memory_utilization=args.gpu_memory_utilization, kv_cache_memory_bytes=args.kv_cache_memory_bytes, memory_reserve_bytes=args.memory_reserve_bytes, diff --git a/vllm_rlt/kernels/flash_attention.py b/vllm_rlt/kernels/flash_attention.py index 8555464b..4d167091 100644 --- a/vllm_rlt/kernels/flash_attention.py +++ b/vllm_rlt/kernels/flash_attention.py @@ -20,7 +20,7 @@ def select_version(capability, backend): if major == 12 and backend in ("flash_attn", "flash_attn_4"): raise ValueError( f"Paged FlashAttention-4 is not supported on SM{major}{minor} by " - "flash-attn-4==4.0.0b30; use --attention-backend triton until a " + "the pinned flash-attn-4 release; use --attention-backend triton until a " "compatible paged implementation is available." ) automatic = 4 if major == 10 else 3 if major == 9 else 2 @@ -39,7 +39,8 @@ def __init__(self, device, dtype, head_dim, block_size, backend="flash_attn"): raise ValueError("flash_attn requires float16 or bfloat16") if head_dim > 256 or head_dim % 8: raise ValueError("flash_attn requires head_dim divisible by 8 and <= 256") - self.generation = select_version(torch.cuda.get_device_capability(device), backend) + capability = torch.cuda.get_device_capability(device) + self.generation = select_version(capability, backend) if self.generation == 2 and block_size % 256: raise ValueError("FlashAttention-2 paged KV requires --block-size a multiple of 256") module, package, function = { @@ -48,13 +49,27 @@ def __init__(self, device, dtype, head_dim, block_size, backend="flash_attn"): 4: ("flash_attn.cute", "flash-attn-4", "flash_attn_varlen_func"), }[self.generation] try: - self.kernel = getattr(importlib.import_module(module), function) + implementation = importlib.import_module(module) + self.kernel = getattr(implementation, function) package_version = version(package) except (ImportError, AttributeError) as error: raise ImportError( f"{backend} selected FA{self.generation}; install the official {package} package. " "No backend fallback was performed." ) from error + # FA4's public scale API also identifies the release family with generic + # hd256 paged kernels. Older releases used a restricted hd256 kernel. + self.scale_type = getattr(implementation, "TensorwiseScales", None) + self.supports_fp8_static = ( + self.generation == 4 + and self.scale_type is not None + and capability[0] == 10 + and dtype == torch.bfloat16 + and head_dim in (64, 128, 256) + ) + self.supports_sliding_window = self.generation == 4 and ( + head_dim != 256 or self.scale_type is not None + ) self.info = dict( backend="flash_attn", generation=self.generation, @@ -63,6 +78,37 @@ def __init__(self, device, dtype, head_dim, block_size, backend="flash_attn"): ) logging.getLogger(__name__).warning("Attention implementation: %s", self.info) + def _fp8_inputs(self, q, key_cache, value_cache, key_scale, value_scale, batch_size): + """FA4 consumes one descale per sequence/KV head, never per token. + + This prototype accepts immutable scalar scales only. Q uses the K scale, + independent of batch composition and prefill chunk boundaries. + """ + if key_cache.dtype != torch.float8_e4m3fn: + if key_scale is not None or value_scale is not None: + raise ValueError("FA descales require an FP8 KV cache") + return q, {} + if not self.supports_fp8_static: + raise ValueError( + "static FP8 requires FA4 with TensorwiseScales on SM10x, BF16, " + "head_dim 64, 128 or 256; install the flash-attn4 extra" + ) + if value_cache.dtype != key_cache.dtype: + raise ValueError("FA FP8 K/V must have the same dtype") + for scale in (key_scale, value_scale): + if ( + scale is None + or scale.ndim != 0 + or scale.dtype != torch.float32 + or scale.device != q.device + ): + raise ValueError("FA4 FP8 requires fixed scalar float32 scales on the query device") + shape = (batch_size, key_cache.shape[-2]) + q_codes = (q.float() / key_scale).clamp(-448.0, 448.0).to(torch.float8_e4m3fn) + k_descale = key_scale.expand(shape).contiguous() + v_descale = value_scale.expand(shape).contiguous() + return q_codes, dict(scales=self.scale_type(q=k_descale, k=k_descale, v=v_descale)) + def prefill( self, q, @@ -72,6 +118,10 @@ def prefill( context_lengths, cu_seqlens_q, max_seqlen_q, + *, + key_scale=None, + value_scale=None, + window=None, ): """Attend packed chunks to their prefixes with bottom-right causal alignment. @@ -81,6 +131,14 @@ def prefill( """ if self.generation != 4: raise ValueError("packed paged prefill requires FlashAttention-4") + if window is not None and not self.supports_sliding_window: + raise NotImplementedError( + "this FlashAttention implementation lacks sliding-window support" + ) + output_dtype = q.dtype + q, descales = self._fp8_inputs( + q, key_cache, value_cache, key_scale, value_scale, block_tables.shape[0] + ) out = self.kernel( q, key_cache, @@ -92,14 +150,36 @@ def prefill( max_seqlen_k=block_tables.shape[1] * key_cache.shape[1], causal=True, num_splits=1, + **({"window_size": (window - 1, 0)} if window is not None else {}), + **descales, ) - return out[0] if isinstance(out, tuple) else out + out = out[0] if isinstance(out, tuple) else out + return out.to(output_dtype) - def __call__(self, q, key_cache, value_cache, block_tables, context_lengths): + def __call__( + self, + q, + key_cache, + value_cache, + block_tables, + context_lengths, + *, + key_scale=None, + value_scale=None, + window=None, + ): + if window is not None and not self.supports_sliding_window: + raise NotImplementedError( + "this FlashAttention implementation lacks sliding-window support" + ) if q.shape[0] == 0: return torch.empty_like(q) # The visible prefix already includes exactly the current causal position. # Do not append K/V here: the cache manager has written it at this depth. + output_dtype = q.dtype + q, descales = self._fp8_inputs( + q, key_cache, value_cache, key_scale, value_scale, block_tables.shape[0] + ) query = q.unsqueeze(1) if self.generation == 4: out = self.kernel( @@ -114,6 +194,8 @@ def __call__(self, q, key_cache, value_cache, block_tables, context_lengths): # Auto splitting depends on batch size and changes BF16 reductions. # Keep the partition fixed across scheduler batch compositions. num_splits=1, + **({"window_size": (window - 1, 0)} if window is not None else {}), + **descales, ) else: page_argument = "block_table" if self.generation == 2 else "page_table" @@ -127,4 +209,28 @@ def __call__(self, q, key_cache, value_cache, block_tables, context_lengths): ) if isinstance(out, tuple): out = out[0] # FA4 beta returns (output, optional LSE). - return out.squeeze(1) + return out.squeeze(1).to(output_dtype) + + def windowed( + self, + q, + key_cache, + value_cache, + block_tables, + context_lengths, + window, + *, + key_scale=None, + value_scale=None, + ): + # The runtime window includes the current token; FA counts preceding keys. + return self( + q, + key_cache, + value_cache, + block_tables, + context_lengths, + key_scale=key_scale, + value_scale=value_scale, + window=window, + ) diff --git a/vllm_rlt/kernels/paged_attention.py b/vllm_rlt/kernels/paged_attention.py index d92d8279..cf2b3db6 100644 --- a/vllm_rlt/kernels/paged_attention.py +++ b/vllm_rlt/kernels/paged_attention.py @@ -16,8 +16,13 @@ def torch_paged_attention( value_cache: torch.Tensor, block_tables: torch.Tensor, context_lengths: torch.Tensor, + key_scale: torch.Tensor | None = None, + value_scale: torch.Tensor | None = None, ) -> torch.Tensor: - """Reference attention over [physical block, token, KV head, dimension].""" + """Reference attention over [physical block, token, KV head, dimension]. + + FP8 pools pass per-vector scales shaped [physical block, token, KV head]. + """ output = torch.empty_like(q) block_size = key_cache.shape[1] head_groups = q.shape[1] // key_cache.shape[2] @@ -29,8 +34,13 @@ def torch_paged_attention( token_positions = torch.arange(length, device=q.device) blocks = block_tables[row, token_positions // block_size] offsets = token_positions % block_size - keys = key_cache[blocks, offsets].repeat_interleave(head_groups, dim=1) - values = value_cache[blocks, offsets].repeat_interleave(head_groups, dim=1) + keys = key_cache[blocks, offsets].float() + values = value_cache[blocks, offsets].float() + if key_scale is not None: + keys = keys * key_scale[blocks, offsets].unsqueeze(-1) + values = values * value_scale[blocks, offsets].unsqueeze(-1) + keys = keys.repeat_interleave(head_groups, dim=1) + values = values.repeat_interleave(head_groups, dim=1) scores = torch.einsum("hd,thd->ht", q[row].float(), keys.float()) * scale probabilities = torch.softmax(scores, dim=-1) output[row] = torch.einsum("ht,thd->hd", probabilities, values.float()).to(q.dtype) @@ -44,12 +54,15 @@ def torch_paged_attention_windowed( block_tables: torch.Tensor, context_lengths: torch.Tensor, window: int, + key_scale: torch.Tensor | None = None, + value_scale: torch.Tensor | None = None, ) -> torch.Tensor: """Reference sliding-window attention over the same layout. Row ``row`` attends only its last ``window`` keys, including the query's own position, matching per-layer sliding-window semantics. History outside the - window stays stored: this is masking, not reclamation. + window stays stored: this is masking, not reclamation. FP8 pools pass + per-vector scales as in ``torch_paged_attention``. """ if isinstance(window, bool) or not isinstance(window, int) or window <= 0: raise ValueError("window must be a positive integer") @@ -64,8 +77,13 @@ def torch_paged_attention_windowed( token_positions = torch.arange(max(0, length - window), length, device=q.device) blocks = block_tables[row, token_positions // block_size] offsets = token_positions % block_size - keys = key_cache[blocks, offsets].repeat_interleave(head_groups, dim=1) - values = value_cache[blocks, offsets].repeat_interleave(head_groups, dim=1) + keys = key_cache[blocks, offsets].float() + values = value_cache[blocks, offsets].float() + if key_scale is not None: + keys = keys * key_scale[blocks, offsets].unsqueeze(-1) + values = values * value_scale[blocks, offsets].unsqueeze(-1) + keys = keys.repeat_interleave(head_groups, dim=1) + values = values.repeat_interleave(head_groups, dim=1) scores = torch.einsum("hd,thd->ht", q[row].float(), keys.float()) * scale probabilities = torch.softmax(scores, dim=-1) output[row] = torch.einsum("ht,thd->hd", probabilities, values.float()).to(q.dtype) @@ -75,12 +93,40 @@ def torch_paged_attention_windowed( class TorchPagedAttention: """Reference single-query attention, with and without sliding windows.""" - def __call__(self, q, key_cache, value_cache, block_tables, context_lengths): - return torch_paged_attention(q, key_cache, value_cache, block_tables, context_lengths) + def __call__( + self, + q, + key_cache, + value_cache, + block_tables, + context_lengths, + key_scale=None, + value_scale=None, + ): + return torch_paged_attention( + q, key_cache, value_cache, block_tables, context_lengths, key_scale, value_scale + ) - def windowed(self, q, key_cache, value_cache, block_tables, context_lengths, window): + def windowed( + self, + q, + key_cache, + value_cache, + block_tables, + context_lengths, + window, + key_scale=None, + value_scale=None, + ): return torch_paged_attention_windowed( - q, key_cache, value_cache, block_tables, context_lengths, window + q, + key_cache, + value_cache, + block_tables, + context_lengths, + window, + key_scale, + value_scale, ) @@ -90,6 +136,8 @@ def triton_paged_attention( value_cache: torch.Tensor, block_tables: torch.Tensor, context_lengths: torch.Tensor, + key_scale: torch.Tensor | None = None, + value_scale: torch.Tensor | None = None, window: int = 0, ) -> torch.Tensor: """Dispatch explicitly to Triton, with no silent reference fallback.""" @@ -100,7 +148,16 @@ def triton_paged_attention( # Keep Triton optional and avoid loading its runtime for CPU-only callers. from vllm_rlt.kernels.triton_attention import paged_attention - return paged_attention(q, key_cache, value_cache, block_tables, context_lengths, window=window) + return paged_attention( + q, + key_cache, + value_cache, + block_tables, + context_lengths, + key_scale, + value_scale, + window=window, + ) @dataclass(frozen=True) @@ -109,6 +166,7 @@ class PackedWindowShape: kv_heads: int kv_layers: int window: int | None + fp8_kv: bool = False def triton_paged_prefill( @@ -119,6 +177,8 @@ def triton_paged_prefill( context_lengths, cu_seqlens_q, max_seqlen_q, + key_scale=None, + value_scale=None, *, window=0, compensate=False, @@ -134,6 +194,8 @@ def triton_paged_prefill( context_lengths, cu_seqlens_q, max_seqlen_q, + key_scale, + value_scale, window=window, compensate=compensate, ) @@ -145,12 +207,40 @@ class TritonPagedAttention: def __init__(self, *, compensate=False): self.compensate = compensate - def __call__(self, q, key_cache, value_cache, block_tables, context_lengths): - return triton_paged_attention(q, key_cache, value_cache, block_tables, context_lengths) + def __call__( + self, + q, + key_cache, + value_cache, + block_tables, + context_lengths, + key_scale=None, + value_scale=None, + ): + return triton_paged_attention( + q, key_cache, value_cache, block_tables, context_lengths, key_scale, value_scale + ) - def windowed(self, q, key_cache, value_cache, block_tables, context_lengths, window): + def windowed( + self, + q, + key_cache, + value_cache, + block_tables, + context_lengths, + window, + key_scale=None, + value_scale=None, + ): return triton_paged_attention( - q, key_cache, value_cache, block_tables, context_lengths, window=window + q, + key_cache, + value_cache, + block_tables, + context_lengths, + key_scale, + value_scale, + window=window, ) def prefill(self, *args, **kwargs): diff --git a/vllm_rlt/kernels/routing.py b/vllm_rlt/kernels/routing.py index 71a77a1b..d80ee7e2 100644 --- a/vllm_rlt/kernels/routing.py +++ b/vllm_rlt/kernels/routing.py @@ -171,8 +171,10 @@ def finalize_kernel( offset = (x // CHANNELS) * LAYER_STRIDE + (pos % PAGE) * TOKEN_STRIDE + x % CHANNELS src = source.to(tl.int64) * BLOCK_STRIDE + offset dst = dest.to(tl.int64) * BLOCK_STRIDE + offset - k = tl.load(K + src, x < LAYERS * CHANNELS, 0) - v = tl.load(V + src, x < LAYERS * CHANNELS, 0) + # No fill value: masked lanes are never stored, and an integer fill cannot be + # cast to an FP8 pool's element type. + k = tl.load(K + src, x < LAYERS * CHANNELS) + v = tl.load(V + src, x < LAYERS * CHANNELS) tl.store(K + dst, k, x < LAYERS * CHANNELS) tl.store(V + dst, v, x < LAYERS * CHANNELS) diff --git a/vllm_rlt/kernels/triton_attention.py b/vllm_rlt/kernels/triton_attention.py index ae9a8d90..413b15dc 100644 --- a/vllm_rlt/kernels/triton_attention.py +++ b/vllm_rlt/kernels/triton_attention.py @@ -6,6 +6,9 @@ merges the parts. A row's partition depends only on its own length, so its result does not depend on the batch it runs in. Without splitting, one long row serialized its whole context in one program and set the decode step time. + +FP8 e4m3 pools are read with one float32 scale per (block, token, KV head) +vector; both kernels dequantize on load. """ from functools import lru_cache, partial @@ -33,6 +36,8 @@ def _paged_attention_kernel( PART_M, PART_L, PART_O, + KS, + VS, q_batch_stride: tl.constexpr, q_head_stride: tl.constexpr, q_dim_stride: tl.constexpr, @@ -45,6 +50,10 @@ def _paged_attention_kernel( v_head_stride: tl.constexpr, v_dim_stride: tl.constexpr, table_stride: tl.constexpr, + s_block_stride: tl.constexpr, + s_token_stride: tl.constexpr, + s_head_stride: tl.constexpr, + HAS_SCALE: tl.constexpr, HEAD_DIM: tl.constexpr, HEAD_GROUPS: tl.constexpr, NUM_HEADS: tl.constexpr, @@ -88,29 +97,40 @@ def _paged_attention_kernel( # Physical IDs fit in int32, but ID * block_stride can exceed 2**31 # elements in a large KV pool. Promote BEFORE multiplying, for K and V. offsets = positions % PAGE_SIZE - keys = tl.load( + kv_mask = valid_tokens[:, None] & (dims[None, :] < HEAD_DIM) + key_pointers = ( K + blocks[:, None] * k_block_stride + offsets[:, None] * k_token_stride + kv_head * k_head_stride - + dims[None, :] * k_dim_stride, - mask=valid_tokens[:, None] & (dims[None, :] < HEAD_DIM), - other=0, - ).to(tl.float32) + + dims[None, :] * k_dim_stride + ) + if HAS_SCALE: + # An integer fill value cannot be cast to fp8: zero masked lanes after widening. + scale_offsets = ( + blocks * s_block_stride + offsets * s_token_stride + kv_head * s_head_stride + ) + keys = tl.where(kv_mask, tl.load(key_pointers, mask=kv_mask).to(tl.float32), 0.0) + keys = keys * tl.load(KS + scale_offsets, mask=valid_tokens, other=0.0)[:, None] + else: + keys = tl.load(key_pointers, mask=kv_mask, other=0).to(tl.float32) scores = tl.sum(keys * query[None, :], axis=1) * SCALE scores = tl.where(valid_tokens, scores, -float("inf")) next_maximum = tl.maximum(maximum, tl.max(scores, axis=0)) correction = tl.exp(maximum - next_maximum) weights = tl.exp(scores - next_maximum) - values = tl.load( + value_pointers = ( V + blocks[:, None] * v_block_stride + offsets[:, None] * v_token_stride + kv_head * v_head_stride - + dims[None, :] * v_dim_stride, - mask=valid_tokens[:, None] & (dims[None, :] < HEAD_DIM), - other=0, - ).to(tl.float32) + + dims[None, :] * v_dim_stride + ) + if HAS_SCALE: + values = tl.where(kv_mask, tl.load(value_pointers, mask=kv_mask).to(tl.float32), 0.0) + values = values * tl.load(VS + scale_offsets, mask=valid_tokens, other=0.0)[:, None] + else: + values = tl.load(value_pointers, mask=kv_mask, other=0).to(tl.float32) accumulator = accumulator * correction + tl.sum(weights[:, None] * values, axis=0) normalizer = normalizer * correction + tl.sum(weights, axis=0) maximum = next_maximum @@ -153,7 +173,16 @@ def _paged_attention_merge_kernel( ) -def paged_attention(q, key_cache, value_cache, block_tables, context_lengths, window=0): +def paged_attention( + q, + key_cache, + value_cache, + block_tables, + context_lengths, + key_scale=None, + value_scale=None, + window=0, +): """Launch split-KV parts over [row, head, split], then merge per [row, head]. ``window`` masks each row to its last ``window`` keys (0 keeps the full @@ -164,6 +193,11 @@ def paged_attention(q, key_cache, value_cache, block_tables, context_lengths, wi output = torch.empty_like(q) if q.shape[0] == 0: return output + has_scale = key_scale is not None + # Unused pointer arguments when the pool is not quantized. + key_scale = key_scale if has_scale else context_lengths + value_scale = value_scale if has_scale else context_lengths + scale_strides = key_scale.stride() if has_scale else (0, 0, 0) rows, heads = q.shape[0], q.shape[1] block_d = triton.next_power_of_2(q.shape[-1]) part_m = torch.empty((rows, heads, NUM_SPLITS), device=q.device, dtype=torch.float32) @@ -178,10 +212,14 @@ def paged_attention(q, key_cache, value_cache, block_tables, context_lengths, wi part_m, part_l, part_o, + key_scale, + value_scale, *q.stride(), *key_cache.stride(), *value_cache.stride(), block_tables.stride(0), + *scale_strides, + HAS_SCALE=has_scale, HEAD_DIM=q.shape[-1], HEAD_GROUPS=heads // key_cache.shape[2], NUM_HEADS=heads, @@ -217,6 +255,8 @@ def _paged_prefill_kernel( LENGTHS, CU_Q, OUT, + KS, + VS, q_token_stride, q_head_stride, k_block_stride, @@ -228,6 +268,10 @@ def _paged_prefill_kernel( table_stride, out_token_stride, out_head_stride, + s_block_stride, + s_token_stride, + s_head_stride, + HAS_SCALE: tl.constexpr, HEAD_DIM: tl.constexpr, HEAD_GROUPS: tl.constexpr, PAGE_SIZE: tl.constexpr, @@ -277,15 +321,25 @@ def _paged_prefill_kernel( TABLES + seq * table_stride + keys_pos // PAGE_SIZE, mask=key_valid, other=0 ).to(tl.int64) offsets = keys_pos % PAGE_SIZE - keys = tl.load( + kv_mask = key_valid[:, None] & dim_valid[None, :] + key_pointers = ( K + blocks[:, None] * k_block_stride + offsets[:, None] * k_token_stride + kv_head * k_head_stride - + dims[None, :], - mask=key_valid[:, None] & dim_valid[None, :], - other=0.0, + + dims[None, :] ) + if HAS_SCALE: + # Dequantize to the query dtype for tl.dot. Masked lanes are zeroed + # after widening: an unfilled fp8 lane may hold a NaN code. + scale_offsets = ( + blocks * s_block_stride + offsets * s_token_stride + kv_head * s_head_stride + ) + keys = tl.where(kv_mask, tl.load(key_pointers, mask=kv_mask).to(tl.float32), 0.0) + keys = keys * tl.load(KS + scale_offsets, mask=key_valid, other=0.0)[:, None] + keys = keys.to(Q.dtype.element_ty) + else: + keys = tl.load(key_pointers, mask=kv_mask, other=0.0) scores = tl.dot(query, tl.trans(keys), input_precision=PRECISION) * SCALE visible = key_valid[None, :] & (keys_pos[None, :] <= positions[:, None]) if WINDOW: @@ -296,15 +350,19 @@ def _paged_prefill_kernel( safe = tl.where(next_maximum == -float("inf"), 0.0, next_maximum) correction = tl.exp(maximum - safe) weights = tl.exp(scores - safe[:, None]) - values = tl.load( + value_pointers = ( V + blocks[:, None] * v_block_stride + offsets[:, None] * v_token_stride + kv_head * v_head_stride - + dims[None, :], - mask=key_valid[:, None] & dim_valid[None, :], - other=0.0, + + dims[None, :] ) + if HAS_SCALE: + values = tl.where(kv_mask, tl.load(value_pointers, mask=kv_mask).to(tl.float32), 0.0) + values = values * tl.load(VS + scale_offsets, mask=key_valid, other=0.0)[:, None] + values = values.to(Q.dtype.element_ty) + else: + values = tl.load(value_pointers, mask=kv_mask, other=0.0) rounded = weights.to(values.dtype) partial = tl.dot(rounded, values, input_precision=PRECISION) if COMPENSATE: @@ -333,6 +391,8 @@ def paged_prefill_attention( context_lengths, cu_seqlens_q, max_seqlen_q, + key_scale=None, + value_scale=None, *, window=0, compensate=False, @@ -343,6 +403,7 @@ def paged_prefill_attention( cu_seqlens_q[s]:cu_seqlens_q[s + 1] and keys [0, context_lengths[s]) through block_tables[s]. Query i of a sequence sits at position context_length - query_length + i and attends keys up to that position. + FP8 pools pass per-vector scales [physical block, token, KV head]. """ output = torch.empty_like(q) sequences = block_tables.shape[0] @@ -364,6 +425,8 @@ def paged_prefill_attention( cu_seqlens_q, output, max_seqlen_q, + key_scale=key_scale, + value_scale=value_scale, window=window, compensate=compensate, ) @@ -382,6 +445,8 @@ def _launch_paged_prefill( *, window, compensate, + key_scale=None, + value_scale=None, compile_only=False, ): block_m, block_n = _PACKED_WINDOW_TILE if compensate else (64, 64) @@ -392,6 +457,11 @@ def _launch_paged_prefill( if compile_only else _paged_prefill_kernel[grid] ) + has_scale = key_scale is not None + # Unused pointer arguments when the pool is not quantized. + key_scale = key_scale if has_scale else context_lengths + value_scale = value_scale if has_scale else context_lengths + scale_strides = key_scale.stride() if has_scale else (0, 0, 0) return launch( q, key_cache, @@ -400,6 +470,8 @@ def _launch_paged_prefill( context_lengths, cu_seqlens_q, output, + key_scale, + value_scale, q.stride(0), q.stride(1), key_cache.stride(0), @@ -411,6 +483,8 @@ def _launch_paged_prefill( block_tables.stride(0), output.stride(0), output.stride(1), + *scale_strides, + HAS_SCALE=has_scale, HEAD_DIM=q.shape[-1], HEAD_GROUPS=q.shape[1] // key_cache.shape[2], PAGE_SIZE=key_cache.shape[1], @@ -428,6 +502,127 @@ def _launch_paged_prefill( ) +@triton.jit +def _fp8_store_kernel( + K, + V, + K_CODES, + V_CODES, + K_SCALES, + V_SCALES, + BLOCKS, + OFFSETS, + ACTIVE, + MASK, + k_row_stride, + k_head_stride, + v_row_stride, + v_head_stride, + c_block_stride, + c_token_stride, + c_head_stride, + s_block_stride, + s_token_stride, + s_head_stride, + HEAD_DIM: tl.constexpr, + BLOCK_D: tl.constexpr, + FP8_MAX: tl.constexpr, + INV_FP8_MAX: tl.constexpr, + TINY: tl.constexpr, + FIXED_SCALE: tl.constexpr, +): + row = tl.program_id(0) + head = tl.program_id(1) + # Optional device-side row skipping: rows at or past *ACTIVE (bucketed graph + # padding) and rows whose MASK entry is false write neither codes nor scales. + enabled = row >= 0 + if ACTIVE is not None: + enabled = enabled & (row < tl.load(ACTIVE)) + if MASK is not None: + enabled = enabled & tl.load(MASK + row) + if enabled: + dims = tl.arange(0, BLOCK_D) + valid = dims < HEAD_DIM + block = tl.load(BLOCKS + row).to(tl.int64) + offset = tl.load(OFFSETS + row).to(tl.int64) + codes_at = block * c_block_stride + offset * c_token_stride + head * c_head_stride + dims + scale_at = block * s_block_stride + offset * s_token_stride + head * s_head_stride + # The same float32 operations as kv_cache_manager.quantize_fp8, in the same order. + k = tl.load(K + row * k_row_stride + head * k_head_stride + dims, mask=valid, other=0.0) + k = k.to(tl.float32) + if FIXED_SCALE is None: + k_scale = tl.maximum(tl.max(tl.abs(k), axis=0) * INV_FP8_MAX, TINY) + else: + k_scale = tl.full((), FIXED_SCALE, tl.float32) + k_codes = tl.minimum(tl.maximum(tl.div_rn(k, k_scale), -FP8_MAX), FP8_MAX) + tl.store(K_CODES + codes_at, k_codes.to(K_CODES.dtype.element_ty), mask=valid) + tl.store(K_SCALES + scale_at, k_scale) + v = tl.load(V + row * v_row_stride + head * v_head_stride + dims, mask=valid, other=0.0) + v = v.to(tl.float32) + if FIXED_SCALE is None: + v_scale = tl.maximum(tl.max(tl.abs(v), axis=0) * INV_FP8_MAX, TINY) + else: + v_scale = tl.full((), FIXED_SCALE, tl.float32) + v_codes = tl.minimum(tl.maximum(tl.div_rn(v, v_scale), -FP8_MAX), FP8_MAX) + tl.store(V_CODES + codes_at, v_codes.to(V_CODES.dtype.element_ty), mask=valid) + tl.store(V_SCALES + scale_at, v_scale) + + +def store_fp8( + k, + v, + key_codes, + value_codes, + key_scale, + value_scale, + blocks, + offsets, + fp8_max, + *, + mask=None, + active=None, + fixed_scale=None, +): + """Quantize rows k, v [row, KV head, dim] and scatter codes and scales in one launch. + + Row i goes to (blocks[i], offsets[i]) of one layer's code pools + [physical block, token, KV head, dim] and scale pools [physical block, token, KV head]. + Optional ``mask`` (bool [row]) and ``active`` (int32 scalar row count) skip rows + on the device, as the unquantized masked and graph KV writers do. + """ + if k.shape[0] == 0: + return + k = k if k.stride(-1) == 1 else k.contiguous() + v = v if v.stride(-1) == 1 else v.contiguous() + if key_codes.stride() != value_codes.stride() or key_scale.stride() != value_scale.stride(): + raise ValueError("FP8 key and value pools must share a layout") + _fp8_store_kernel[(k.shape[0], k.shape[1])]( + k, + v, + key_codes, + value_codes, + key_scale, + value_scale, + blocks, + offsets, + active, + mask, + k.stride(0), + k.stride(1), + v.stride(0), + v.stride(1), + *key_codes.stride()[:3], + *key_scale.stride(), + HEAD_DIM=k.shape[-1], + BLOCK_D=triton.next_power_of_2(k.shape[-1]), + FP8_MAX=fp8_max, + INV_FP8_MAX=1.0 / fp8_max, + TINY=torch.finfo(torch.float32).tiny, + FIXED_SCALE=fixed_scale, + num_warps=1, + ) + + def select_packed_window_tile(device, dtype, head_dim, block_size, shape): if torch.version.hip or dtype != torch.bfloat16: return None @@ -443,7 +638,10 @@ def select_packed_window_tile(device, dtype, head_dim, block_size, shape): def _select_packed_window_tile(index, head_dim, block_size, shape): # Warmup only needs tensor metadata. Preserve the KV pool's interleaved layer strides. q = torch.empty((256, shape.query_heads, head_dim), device="meta", dtype=torch.bfloat16) - kv = q.new_empty((1, shape.kv_layers, block_size, shape.kv_heads, head_dim))[:, 0] + kv_shape = (1, shape.kv_layers, block_size, shape.kv_heads) + kv_dtype = torch.float8_e4m3fn if shape.fp8_kv else q.dtype + kv = q.new_empty((*kv_shape, head_dim), dtype=kv_dtype)[:, 0] + scale = q.new_empty(kv_shape, dtype=torch.float32)[:, 0] if shape.fp8_kv else None table = q.new_empty((1, 16), dtype=torch.int32) lengths = q.new_empty(1, dtype=torch.int32) offsets = q.new_empty(2, dtype=torch.int32) @@ -463,6 +661,8 @@ def _select_packed_window_tile(index, head_dim, block_size, shape): len(q), window=window, compensate=True, + key_scale=scale, + value_scale=scale, compile_only=True, ) if compiled.metadata.shared > limit: diff --git a/vllm_rlt/pd/engine.py b/vllm_rlt/pd/engine.py index cebf8015..26f7e51d 100644 --- a/vllm_rlt/pd/engine.py +++ b/vllm_rlt/pd/engine.py @@ -96,6 +96,8 @@ def __init__( d_cache = decode_cache_config or CacheConfig() if (p_cache.layout, p_cache.block_size) != (d_cache.layout, d_cache.block_size): raise ValueError("P and D must use the same KV layout and block size") + if p_cache.kv_cache_dtype != d_cache.kv_cache_dtype: + raise ValueError("P and D must use the same KV cache dtype") if attention_backend not in ("triton", "flash_attn", "flash_attn_4"): raise ValueError("PD supports CUDA Triton or paged FA4 attention") self.scheduling_policy = p_scheduler.policy diff --git a/vllm_rlt/pd/transport.py b/vllm_rlt/pd/transport.py index 092bb47f..f1b2a3d5 100644 --- a/vllm_rlt/pd/transport.py +++ b/vllm_rlt/pd/transport.py @@ -15,6 +15,17 @@ class Transfer: submitted: float +def _kv_region(key, value): + element = key.element_size() + return dict( + key_ptr=key.data_ptr(), + value_ptr=value.data_ptr(), + block_bytes=key.stride(0) * element, + layer_bytes=key.stride(1) * element, + token_bytes=key.stride(2) * element, + ) + + def kv_segments(info, tables, start, end): """Map valid token ranges to contiguous byte segments, preserving every depth. @@ -24,6 +35,8 @@ def kv_segments(info, tables, start, end): page = info["block_size"] if not 0 <= start < end: raise ValueError("invalid KV range") + scale_region = info.get("scale_region") + regions = (info,) if scale_region is None else (info, scale_region) for table in tables: if end > len(table) * page: raise ValueError("KV range exceeds allocation") @@ -33,17 +46,21 @@ def kv_segments(info, tables, start, end): raise ValueError("physical block outside registered pool") lo = max(start - logical * page, 0) hi = min(end - logical * page, page) - for base in (info["key_ptr"], info["value_ptr"]): - address = base + block * info["block_bytes"] - if lo == 0 and hi == page: - yield (address, info["block_bytes"], info["device"]) - else: - for layer in range(info["num_layers"]): - yield ( - address + layer * info["layer_bytes"] + lo * info["token_bytes"], - (hi - lo) * info["token_bytes"], - info["device"], - ) + for region in regions: + block_bytes = region["block_bytes"] + layer_bytes = region["layer_bytes"] + token_bytes = region["token_bytes"] + for base in (region["key_ptr"], region["value_ptr"]): + address = base + block * block_bytes + if lo == 0 and hi == page: + yield (address, block_bytes, info["device"]) + else: + for layer in range(info["num_layers"]): + yield ( + address + layer * layer_bytes + lo * token_bytes, + (hi - lo) * token_bytes, + info["device"], + ) def coalesce_segments(source, target): @@ -112,14 +129,18 @@ def __init__(self, name, cache, hidden, config): else: self.agent = nixl_agent(name, nixl_agent_config(backends=[config.backend])) self.storage = (cache.key_cache, cache.value_cache, hidden) + # Dynamic scales travel with their KV rows. Fixed scales are checked + # during the handshake and already initialized on the receiving side. + scale_region = None + if cache.key_scale is not None and cache.fp8_scale is None: + self.storage += (cache.key_scale, cache.value_scale) + scale_region = _kv_region(cache.key_scale, cache.value_scale) self.registrations = [self.agent.register_memory(t) for t in self.storage] k = cache.key_cache - element = k.element_size() self.info = dict( + **_kv_region(k, cache.value_cache), agent=name, metadata=self.agent.get_agent_metadata(), - key_ptr=k.data_ptr(), - value_ptr=cache.value_cache.data_ptr(), hidden_ptr=hidden.data_ptr(), hidden_bytes=hidden.stride(0) * hidden.element_size(), device=k.device.index, @@ -127,9 +148,9 @@ def __init__(self, name, cache, hidden, config): block_size=cache.block_size, num_layers=cache.num_layers, depths=cache.storage_depths, - block_bytes=k.stride(0) * element, - layer_bytes=k.stride(1) * element, - token_bytes=k.stride(2) * element, + kv_dtype=str(k.dtype), + fp8_scale=cache.fp8_scale, + scale_region=scale_region, ) self.peers = {} self.pending = deque() @@ -139,15 +160,23 @@ def __init__(self, name, cache, hidden, config): self.transfer_seconds = 0.0 def connect(self, peer): + strides = ("block_bytes", "layer_bytes", "token_bytes") + for key in ("block_size", "num_layers", "depths", "hidden_bytes", "kv_dtype", *strides): + if peer[key] != self.info[key]: + raise ValueError(f"PD KV geometry mismatch: {key}") + if peer["fp8_scale"] != self.info["fp8_scale"]: + raise ValueError("P and D must use the same calibrated FP8 scale") + local_scales, peer_scales = self.info["scale_region"], peer["scale_region"] + if (local_scales is None) != (peer_scales is None) or ( + local_scales is not None and any(local_scales[k] != peer_scales[k] for k in strides) + ): + raise ValueError("PD KV scale geometry mismatch") name = self.agent.add_remote_agent(peer["metadata"]) name = name.decode() if isinstance(name, bytes) else name if name != peer["agent"]: raise ValueError( f"NIXL agent metadata identity mismatch: {name!r} != {peer['agent']!r}" ) - for key in ("block_size", "num_layers", "depths", "block_bytes", "hidden_bytes"): - if peer[key] != self.info[key]: - raise ValueError(f"PD KV geometry mismatch: {key}") self.peers[name] = peer def submit(self, transfer_id, sequence, peer, local, remote, size): diff --git a/vllm_rlt/worker/async_state.py b/vllm_rlt/worker/async_state.py index be0d423a..b69fd822 100644 --- a/vllm_rlt/worker/async_state.py +++ b/vllm_rlt/worker/async_state.py @@ -132,7 +132,7 @@ def __init__(self, cache, config, scheduler, rows, *, use_uva=True): self.planes = config.total_ut_steps if cache.layout == "last_exited" else 1 self.hidden = torch.empty( (scheduler.max_num_seqs, config.hidden_size), - dtype=cache.key_cache.dtype, + dtype=cache.dtype, device=cache.device, ) self.tokens = torch.empty( diff --git a/vllm_rlt/worker/cuda_graph.py b/vllm_rlt/worker/cuda_graph.py index c0f0ceac..d3cd05c0 100644 --- a/vllm_rlt/worker/cuda_graph.py +++ b/vllm_rlt/worker/cuda_graph.py @@ -107,7 +107,18 @@ def __init__(self, cache): def _write_prepared(self, layer, batch, k, v): cache = self.cache - if batch.active_rows is not None or ( + if cache.key_scale is not None: + # FP8 pools: one fused quantizing writer that honors the same row masks. + cache._store_layer( + layer, + batch.write_blocks, + batch.write_offsets, + k, + v, + mask=batch.write_mask, + active=batch.active_rows, + ) + elif batch.active_rows is not None or ( batch.write_mask is None and cache.backend == "triton" ): from vllm_rlt.kernels.graph_kv import write_graph_kv @@ -123,10 +134,9 @@ def _write_prepared(self, layer, batch, k, v): def _attend_prepared(self, layer, batch, q, *, window=None): cache = self.cache - if window is not None: + scales = cache._layer_scales(layer) + if window is not None and batch.cu_seqlens_q is None: # The window is a kernel constant; lengths and tables stay graph inputs. - if batch.cu_seqlens_q is not None: - raise NotImplementedError("sliding-window attention needs one query row per token") return cache.attention.windowed( q, cache.key_cache[:, layer], @@ -134,6 +144,7 @@ def _attend_prepared(self, layer, batch, q, *, window=None): batch.block_tables, batch.context_lengths, window, + **scales, ) if batch.cu_seqlens_q is not None: return cache.attention.prefill( @@ -144,6 +155,8 @@ def _attend_prepared(self, layer, batch, q, *, window=None): batch.context_lengths, batch.cu_seqlens_q, batch.max_seqlen_q, + **({"window": window} if window is not None else {}), + **scales, ) return cache.attention( q, @@ -151,6 +164,7 @@ def _attend_prepared(self, layer, batch, q, *, window=None): cache.value_cache[:, layer], batch.block_tables, batch.context_lengths, + **scales, ) diff --git a/vllm_rlt/worker/model_runner.py b/vllm_rlt/worker/model_runner.py index 3766c5b2..70c251ae 100644 --- a/vllm_rlt/worker/model_runner.py +++ b/vllm_rlt/worker/model_runner.py @@ -1354,6 +1354,27 @@ def finalize_many(self, requests): *cache.key_cache.stride()[:3], 256, ) + if cache.key_scale is not None: + # FP8 pools: the same copy for the per-vector scales. + finalize_kernel[ + ( + len(copies), + cache.max_loops, + math.ceil(cache.num_layers * cache.num_kv_heads / 256), + ) + ]( + routing_buffers.descriptor, + self.async_state.tables, + cache.key_scale, + cache.value_scale, + self.async_state.width, + cache.max_loops, + cache.block_size, + cache.num_layers, + cache.num_kv_heads, + *cache.key_scale.stride()[:3], + 256, + ) routing_buffers.record_done() for request in copies: self.events[request.request_id] = routing_buffers.done