Commit a8adbd2
K2.A.1 fix: cast KakeyaLattice round-tripped K/V back to resident dtype
_round_trip_resident_through_compressor crashed on CUDA bf16:
index_copy_(): self and source expected to have the same dtype,
but got (self) BFloat16 and (source) Float
KakeyaLattice's compress/decompress runs its quantize/dequantize math in
fp32 for fidelity, so the round-tripped K/V come back fp32 while the
resident K/V cache is the model compute dtype (bf16 on CUDA). index_copy_
requires matching dtype+device. Cast K/V_round_tripped back to the
destination's dtype+device before writing. (Didn't surface on Mac where
the fp32 path matched.)
Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>1 parent 17a7791 commit a8adbd2
1 file changed
Lines changed: 11 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
201 | 201 | | |
202 | 202 | | |
203 | 203 | | |
204 | | - | |
205 | | - | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
206 | 215 | | |
207 | 216 | | |
208 | 217 | | |
| |||
0 commit comments