Skip to content

Commit a8adbd2

Browse files
K2.A.1 fix: cast KakeyaLattice round-tripped K/V back to resident dtype
_round_trip_resident_through_compressor crashed on CUDA bf16: index_copy_(): self and source expected to have the same dtype, but got (self) BFloat16 and (source) Float KakeyaLattice's compress/decompress runs its quantize/dequantize math in fp32 for fidelity, so the round-tripped K/V come back fp32 while the resident K/V cache is the model compute dtype (bf16 on CUDA). index_copy_ requires matching dtype+device. Cast K/V_round_tripped back to the destination's dtype+device before writing. (Didn't surface on Mac where the fp32 path matched.) Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
1 parent 17a7791 commit a8adbd2

1 file changed

Lines changed: 11 additions & 2 deletions

File tree

inference_engine/v04/dlm_restored_verifier.py

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -201,8 +201,17 @@ def _round_trip_resident_through_compressor(
201201
# resident positions get the round-tripped values.
202202
K_out = K.clone()
203203
V_out = V.clone()
204-
K_out.index_copy_(-2, pos_tensor, K_round_tripped)
205-
V_out.index_copy_(-2, pos_tensor, V_round_tripped)
204+
# The compressor round-trip may upcast to fp32 (KakeyaLattice's
205+
# quantize/dequantize math runs in fp32 for numerical fidelity),
206+
# whereas the resident K/V cache is the model's compute dtype
207+
# (bf16 on CUDA). index_copy_ requires matching dtype (and device),
208+
# so cast the round-tripped tensors back before writing them in.
209+
K_out.index_copy_(
210+
-2, pos_tensor, K_round_tripped.to(device=K_out.device, dtype=K_out.dtype),
211+
)
212+
V_out.index_copy_(
213+
-2, pos_tensor, V_round_tripped.to(device=V_out.device, dtype=V_out.dtype),
214+
)
206215
return K_out, V_out
207216

208217

0 commit comments

Comments
 (0)