bug(cpu/matmul): int overflow in Q8_0/F16 GEMV weight row offsets for >2 GB tensors

[unsafePtr]

Problem

Two GEMV weight-offset expressions in src/DotLLM.Cpu/Kernels/MatMul.cs compute a row
offset with int arithmetic. For a single tensor above ~2 GB the product wraps, producing a
negative/incorrect pointer offset — an out-of-bounds read, not a clean failure.

1. ComputeRows — weightsQ8 + row * rowBytes (byte stride, most exposed)

int rowBytes = blockCount * Q8_0BlockBytes;   // ~1.06 bytes/weight
for (; row + 3 < m; row += 4)
    VecDotQ8_0Vnni_4Rows(weightsQ8 + row * rowBytes, ...);

Repeated across all four dispatch tiers (AVX-512+VNNI / AVX-512 / AVX2 / scalar), roughly
lines 141–192. Max value is m * rowBytes = total bytes of the Q8_0 tensor, so it wraps
past 2^31 bytes = 2 GB. A 256k-vocab x 8192-hidden Q8_0 LM head is ~2.23 GB — over the line.
Because the stride is in bytes rather than elements, this is the closest of the two to being hit.

Only the GEMV path is exposed. Called from ComputeGemmTiled, row < tileRows <= 256
(ComputeTileM clamps to 256) and the base pointer is already pre-offset via
(long)mStart * q8RowBytes, so the tiled path is safe. GemvQ8_0 passes the full m.

2. GemvF16 — weightsHalf + row * k (lines ~1567, ~1580)

for (int row = 0; row < m; row++)
    var srcRow = new ReadOnlySpan<Half>(weightsHalf + row * k, k);

Max is m * k elements, wrapping past 2^31 elements (~4.3 GB of F16).

Note the sibling GemmF16 already widens the identical m * k expression
(weightsHalf + (long)mStart * k), as do GemmF32 (a + (long)mStart * k) and
ComputeGemmTiled (weightsQ8 + (long)mStart * q8RowBytes). Same bound, inconsistently
handled — the Gemm paths have the cast, the Gemv paths don't.

Explicitly not in scope

Activation-buffer offsets (b + t * k, c + t * m + mStart, inputQ8 + t * q8RowBytes) are
self-limiting and should be left alone: each indexes a buffer whose total size is the same
product, so wrapping would require an >8 GB logits/input allocation (or >2 GB of quantized
activations, i.e. n > ~493k tokens in one batch). Not reachable in any realistic configuration.

Acceptance criteria

• Widen the two weight-offset expressions to long, matching the existing
  (long)mStart * ... convention already used in the Gemm paths.
• No change to activation-offset arithmetic (see above).
• Confirm no measurable regression: these are loop-invariant or cheap index computations,
  but ComputeRows is the Q8_0 decode hot path, so a before/after GemvQ8_0 benchmark
  should accompany the change.

Priority

Low. Requires a single tensor >2 GB (Q8_0) or >4.3 GB (F16) to trigger. Filing so it is
recorded rather than rediscovered.

[unsafePtr]

Clarification: the managed-array size limit does not gate this

Recording this because "surely .NET cannot have arrays that big" is the obvious first objection. It does not apply, for two independent reasons.

The weights are never managed arrays. GgufFile memory-maps the file and hands out a raw base pointer:

mmf = MemoryMappedFile.CreateFromFile(filePath, FileMode.Open, null, 0, MemoryMappedFileAccess.Read);
accessor = mmf.CreateViewAccessor(0, 0, MemoryMappedFileAccess.Read);
accessor.SafeMemoryMappedViewHandle.AcquirePointer(ref basePointer);

TransformerWeights stores them as nint (QWeight, KWeight, VWeight, OWeight, GateWeight, UpWeight, DownWeight, OutputWeight); only the small norm weights are float[]. A >2 GB mapped view is legal on 64-bit and byte* arithmetic is 64-bit, so the sole 32-bit component is the int multiply itself.

And on modern .NET there is no 2 GB array cap to begin with. Per runtime configuration options for garbage collection, the default is that the GC supports arrays greater than 2 GB — equivalent to DOTNET_gcAllowVeryLargeObjects=1 — and the switch may become obsolete in a future release. The opt-in, default-off behaviour belongs to the .NET Framework <gcAllowVeryLargeObjects> app.config element, not to .NET Core/5+. Element count stays int-bounded via Array.MaxLength (.NET 6+). Nothing in .NET 11 alters any of this.

Concrete reachable case for site 1 (Q8_0)

Llama 3.1 405B — hidden 16384, vocab 128256, giving Q8_0 rowBytes = (16384 / 32) * 34 = 17,408:

m * rowBytes = 128,256 * 17,408 = 2,232,680,448  >  int.MaxValue (2,147,483,647)

Overflows by roughly 85 MB. The first affected row is 123,362, so the last 4,894 rows — 3.8% of the vocab — receive a wrapped negative offset. That is an out-of-bounds read on a shipping model.

Correction to site 2 (F16): less reachable than filed

GemvF16's row * k counts elements rather than bytes, so on that same model:

m * k = 128,256 * 16,384 = 2,101,346,304  <  int.MaxValue   (~2.2% under)

Q8_0 tips over only because its stride is ~1.0625 bytes per weight where F16's is 1 element per weight. The F16 site therefore requires a tensor larger than any current model, and the issue description presents the two as comparable, which is wrong.

Net effect on priority: site 1 is more urgent than filed, since a shipping model reaches it; site 2 is more theoretical than filed. Both fixes remain one-line widenings.