v1.15.0 — stream_store family completion

CHANGELOG.md

@@ -1,5 +1,77 @@
 # Changelog
 
+## v1.15.0 — 2026-05-20 — Non-temporal store family completion
+
+### Added
+
+- **Scalar `stream_store` overloads** — `i16/u16/i32/u32/i64/u64` value
+  types now accepted in addition to vectors. Lowers via the same
+  `!nontemporal` metadata path used for the vector form. On x86, `movnti`
+  is emitted for i32/u32/i64/u64 (i16/u16 fall through to plain `mov`).
+  On aarch64, `stnp` is synthesized only when LLVM can self-pair (i64
+  splits to a `w`-pair; i32/i16 fall through to plain `str`/`strh` — see
+  the target-lowering table in the reference for the full matrix).
+  Closes the scalar-write surface gap blocking Olorin's `q4k_repack.ea`
+  kernel.
+- **`fence_nt()` intrinsic** — zero-argument store-store memory barrier
+  for intra-kernel ordering of preceding `stream_store` operations.
+  Lowers to `sfence` on x86 (via `@llvm.x86.sse.sfence`), `dmb ishst` on
+  aarch64 (via `@llvm.aarch64.dmb`). Completes the `prefetch_nta` +
+  `stream_store` + `fence_nt` non-temporal memory-hint family.
+
+### Fixed
+
+- **Vector `stream_store` alignment** — pre-existing bug where the
+  `set_alignment` call used the element type's natural alignment (e.g.
+  4 bytes for `f32x4`) rather than the full vector width. LLVM was
+  silently decomposing 128/256-bit NT stores to a sequence of scalar
+  (non-temporal-hinted) stores because the alignment metadata didn't
+  authorize vector-width stores. Caught by the new objdump assertions;
+  alignment now set to `element_size * lane_count`, so x86 emits
+  `movntps`/`vmovntps`/`movntdq`/`vmovntdq` directly. Behavior change:
+  kernels passing vector-aligned buffers see the intended fast path;
+  kernels passing misaligned buffers now SIGSEGV per the documented
+  alignment contract
+  (previously they got slow scalarized stores).
+
+### Changed
+
+- **`stream_store` reference documentation** upgraded to `prefetch_nta`
+  parity. Adds target-specific lowering table covering all vector and
+  scalar widths, explicit alignment contract (general protection fault
+  on x86 misalignment), explicit ordering contract, and a "When NOT to
+  use" section calling out working-buffer anti-patterns (softmax
+  accumulators, FWHT scratch) to prevent adoption regressions.
+
+### Test hardening
+
+- aarch64-gated runtime tests for scalar `stream_store` and `fence_nt`,
+  verified on Pi 5 (Cortex-A76, LLVM 18.1.8) against expected mnemonics.
+- objdump-level assertions verifying `movnti` / `movntps` / `vmovntps` /
+  `vmovntdq` / `sfence` actually emitted on x86 (not just present in IR
+  metadata) — caught the vector alignment bug fixed above.
+- aarch64 objdump-level assertions for `dmb ishst` (`fence_nt`), the
+  i64-to-`stnp` w-pair split (scalar i64), and the q-to-`stnp` d-pair
+  split (vector 128-bit) — pin the actually-observed LLVM 18 behavior
+  so a future LLVM upgrade changing aarch64 NT-store synthesis lands
+  loudly rather than silently.
+- Alignment-failure crash test pinning the alignment contract via
+  deliberate SIGSEGV from a 1-byte-misaligned `f32x8` `stream_store`.
+
+### Out of scope (deferred)
+
+- In-language `parallel_for` keyword / binding-layer parallel dispatch —
+  deferred indefinitely. Audit of the highest-performance Eä consumers
+  (Olorin and Cougar) showed both already ship custom SpinBarrier-based
+  thread pools strictly more capable than any generic primitive Eä
+  could provide. See ROADMAP for amended entry.
+- Sub-byte bit-packing intrinsics (`load_packed_iN_to_*`) — deferred
+  indefinitely. Audit showed every shipped quantized-weight consumer
+  either fuses unpack into compute (Cougar BitNet), has already-clean
+  2-op unpacks (eakv Q4_1), or uses format-specific mixed-width
+  layouts that no generic intrinsic could capture (Olorin GGML Q4_K /
+  Q3_K).
+
 ## v1.14.0 — 2026-05-19 — f32 transcendental family complete + Olorin-driven SIMD primitives
 
 Closes the v1.11.0-era "Future API consistency" list. The f32 transcendental approximation family is feature-complete: `tanh_approx_f32`, `log_approx_f32`, `sin_approx_f32`, and `cos_approx_f32` join `exp_poly_f32` (v1.11.0), all sharing the same f32-vector-only contract and ~3e-6 absolute error budget. `u16x32` + `lo256_u16x32` / `hi256_u16x32` close the i16/u16 lane-extractor symmetry deferred in v1.12.0 PR #10. `wmul_u64(u32x4, u32x4) -> u64x4` ships as a fused alternative to the v1.12.0 `wmul_u64_lo` / `wmul_u64_hi` pair. Olorin-driven SIMD primitives: `permute_runtime` (AVX2 runtime data permute), `prefetch_write` / `prefetch_nta` (write-intent + non-temporal cache hints), and two-source `shuffle(a, b, [indices])`. Doc-side: the types reference now lists every lexer-accepted vector type — closing a multi-release gap that included the AVX-512BW byte/word types — and the cookbook tanh-GELU recipe was rewritten to use the new `tanh_approx_f32` instead of the catastrophic-cancellation-prone `(exp_poly_f32(2x) - 1) / (exp_poly_f32(2x) + 1)` identity.

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "ea-compiler"
-version = "1.14.0"
+version = "1.15.0"
 edition = "2024"
 description = "Eä — compute kernel compiler"
 authors = ["Peter Lukka <peter.lukka@gmail.com>"]

ROADMAP.md

@@ -2,7 +2,60 @@
 
 Forward-looking notes. Ordered by leverage, not by effort.
 
-## Shipped in v1.14.0 (UNRELEASED)
+## Shipped in v1.15.0
+
+### Scalar `stream_store` overloads
+
+`stream_store(*mut i16, offset, i16)`, `stream_store(*mut u16, offset, u16)`,
+`stream_store(*mut i32, offset, i32)`, `stream_store(*mut u32, offset, u32)`,
+`stream_store(*mut i64, offset, i64)`, `stream_store(*mut u64, offset, u64)`.
+Same `!nontemporal` metadata path as the existing vector form.
+
+Cross-target lowering (Pi 5-verified, LLVM 18.1.8): x86 emits `movnti` for
+i32/u32/i64/u64 (i16/u16 fall through to plain `mov`); aarch64 emits `stnp`
+only when LLVM can self-pair the value — i64/u64 splits to a `w`-pair via
+`lsr` and gets `stnp w, w`, while i32/u32 and i16/u16 lower to plain
+`str`/`strh` with the NT hint silently dropped. **For aarch64 NT semantics,
+prefer 64-bit-or-wider element widths.** See the target-lowering table in
+`docs/src/reference/intrinsics.md` for the full matrix and the
+why-no-scalar-stnp explanation.
+
+Concrete consumer pulling: Olorin's `q4k_repack.ea` (pure-streaming block-
+layout repack blocked on scalar surface; pairs 16-byte fields, so i64
+writes get the aarch64 win).
+
+### `fence_nt()` intrinsic
+
+Zero-argument store-store memory barrier. Lowers to `sfence` on x86 (via
+`@llvm.x86.sse.sfence`) and `dmb ishst` on aarch64 (via `@llvm.aarch64.dmb`
+with operand 10). Completes the `prefetch_nta` + `stream_store` + `fence_nt`
+non-temporal memory-hint family. Most callers will not need it — host-side
+sync primitives provide cross-thread release semantics — but for the rare
+intra-kernel ordering case it's the documented expression. Note: does NOT
+provide store-to-load ordering; a full barrier (`mfence`/`dmb sy`) is needed
+for write-then-read-back patterns.
+
+### Vector `stream_store` alignment fix
+
+Pre-existing bug surfaced by the new objdump test discipline: the
+`set_alignment` call on the store instruction passed element-width alignment
+rather than vector-width. LLVM 18 took this conservatively and decomposed
+NT vector stores into a sequence of scalar (non-temporal-hinted) stores,
+defeating the entire point of the intrinsic. Fix: set alignment to
+`element_size * lane_count`. Behavior change visible to callers — fast path
+for aligned buffers, SIGSEGV per the documented contract for misaligned.
+Caught and pinned by the new alignment-failure crash test.
+
+### `stream_store` documentation upgrade
+
+Reference docs upgraded to `prefetch_nta` parity: target-specific lowering
+table, alignment contract, ordering contract, and "When NOT to use" anti-
+pattern guidance. The last item is most important — prevents adoption
+regressions in working-buffer kernels (softmax accumulators, FWHT scratch)
+where blanket-substituting `store → stream_store` would degrade by forcing
+DRAM round-trips on cache-resident data.
+
+## Shipped in v1.14.0
 
 ### Runtime SIMD permute
 
@@ -74,17 +127,33 @@ Today the language spec is spread across `docs/src/reference/*.md` (types, intri
 
 ## Future additions
 
-### Multi-core / `parallel_for` primitive
-
-Eä is single-thread SIMD today; concurrency comes from outer-loop threading in the Rust / Python / Go caller. A `parallel_for(range, body)` primitive that spawns SIMD work across cores would change the per-call performance model from "kernel uses one core" to "kernel uses the machine." Pi 5 has 4 A76 cores; Zen 4 has 16+. With single-thread SIMD perf increasingly memory-bound (chacha20 hits 3.6 GB/s on Zen 4 = DRAM ceiling, not compute), multi-core is the unused dimension where the next 2–10× lives.
-
-Open design questions:
-- **Thread-pool model.** Static partition is simple but loses to imbalanced workloads; work-stealing (rayon-style) is robust but adds runtime dependency. Eä's "no implicit runtime" stance suggests caller-supplied pool injection.
-- **C ABI interaction.** Does the kernel signature change? `(thread_id, num_threads)` extra args, or hidden global?
-- **Determinism.** Reductions become non-associative under parallel execution; `reduce_add` semantics across threads need a documented contract.
-- **NUMA / cache discipline.** First-touch placement matters on Zen 4 and on multi-socket. Probably out-of-scope for v1.x, but the API shape shouldn't preclude it.
-
-Likely a v1.15+ initiative — too large for v1.14.0, but worth scoping early so the smaller carry-overs don't constrain the design space.
+### Multi-core / `parallel_for` primitive — deferred indefinitely (v1.15 audit)
+
+Eä remains single-thread SIMD. Multi-core orchestration is the host's
+responsibility. The v1.15 brainstorm evaluated two candidate forms — an
+in-language `parallel_for` keyword and a binding-layer `parallel: true`
+metadata flag generating per-language wrappers — and dropped both after a
+consumer audit:
+
+- **Olorin** ships `src/inference/threadpool.rs`, a custom SpinBarrier-based
+  pool with `run_graph<F: Fn(usize, usize, &SpinBarrier, &AtomicI32) + Send
+  + Sync>`. Designed for ggml-style inference graphs with cross-kernel
+  barriers and atomic-int dynamic work counters. Strictly more capable than
+  anything generic Eä could provide; a binding-layer wrapper cannot express
+  cross-kernel barriers.
+- **Cougar** invokes Eä kernels (`q4k_4row_dot` — 4 rows per call) inside
+  `pool.run(n_threads, |tid, _| { ... })`, looping many times per thread
+  with per-thread accumulator state across calls. A "wrap in N threads,
+  one call each" binding wrapper would replace this with strictly worse
+  work distribution.
+- **eakv** has no parallelism today but the workaround for the dequant case
+  is 3 lines of `concurrent.futures.ThreadPoolExecutor` in eakv's own code
+  — not enough pull to justify permanent surface area in eacompute.
+
+Criteria for revisiting: a new consumer profile that custom Olorin/Cougar
+pools don't already serve — most likely a Python-first consumer with
+non-trivial host-side parallelism cost, or a consumer with simpler graph
+structure than ggml-style inference. None has surfaced.
 
 ### Autoresearch ↔ perf-regression feedback
 

docs/src/guide/common-intrinsics.md

@@ -38,15 +38,16 @@ These are equivalent to plain `load` but make the element type visible in the so
 
 ## stream_store
 
-Non-temporal store that bypasses the CPU cache. Use for write-only output buffers where you will not read the data back soon:
+Non-temporal store — bypasses cache, used for write-only output. Vector
+or scalar value (v1.15.0 added scalar i16/u16/i32/u32/i64/u64). See
+[reference](../reference/intrinsics.md#stream_store) for the full
+alignment contract, ordering contract, and anti-patterns.
 
 ```
-let result: f32x8 = a .* b
-stream_store(out, i, result)
+stream_store(out, i, result)         // vector
+stream_store(out, i, scalar_value)   // scalar (v1.15.0+)
 ```
 
-This avoids polluting the cache with output data, leaving more cache space for inputs. Only beneficial for large output arrays that will not be re-read immediately.
-
 ## fma
 
 Fused multiply-add: computes `a * b + c` in a single instruction with a single rounding (more accurate than separate multiply and add):
@@ -172,7 +173,8 @@ The `rem` parameter specifies how many elements (starting from lane 0) are valid
 | `splat(s)` | scalar | vector | Broadcast to all lanes |
 | `load(ptr, i)` | pointer, offset | vector | Load vector from memory |
 | `store(ptr, i, v)` | pointer, offset, vector | void | Write vector to memory |
-| `stream_store(ptr, i, v)` | pointer, offset, vector | void | Non-temporal write |
+| `stream_store(ptr, i, v)` | pointer, offset, vector or scalar | void | Non-temporal write |
+| `fence_nt()` | none | void | Store-store barrier for stream_store ordering |
 | `fma(a, b, c)` | 3 values | same type | `a * b + c` fused |
 | `reduce_add(v)` | vector | scalar | Sum all lanes |
 | `reduce_max(v)` | vector | scalar | Max across lanes |

docs/src/reference/intrinsics.md

@@ -62,12 +62,133 @@ store(out, i, result);
 
 ### stream_store
 
-Non-temporal store that bypasses the CPU cache. Use for write-only output that will not be read back soon.
+Non-temporal store that bypasses the CPU cache. Use for write-only output
+the kernel will not read back soon. Pairs with `prefetch_nta` (the read-side
+non-temporal hint) and `fence_nt` (the ordering primitive).
 
 ```
-stream_store(out, i, result);
+stream_store(out, i, v)              // vector form — v is f32xN, i32xN, etc.
+stream_store(out, i, scalar_value)   // scalar form (v1.15.0) — i16/u16/i32/u32/i64/u64
 ```
 
+**Target lowering** (verified on LLVM 18.1.8, x86_64 Zen 4 and aarch64 Cortex-A76):
+
+| Width / form | x86 | aarch64 |
+|---|---|---|
+| Vector 128-bit (f32x4, i32x4, ...) | `movntps` / `movntdq` | `stnp d, d, [x]` (LLVM splits the 128-bit q-register into a d-pair to use the only available aarch64 NT store) |
+| Vector 256-bit (AVX2) | `vmovntps` / `vmovntdq` | n/a (256-bit not on NEON) |
+| Vector 512-bit (AVX-512) | `vmovntps` / `vmovntdq` zmm | n/a |
+| Scalar i64 / u64 | `movnti` (SSE2, 64-bit mode) | `stnp w, w, [x]` (LLVM splits the i64 into a w-pair to use `stnp`; emits an `lsr` for the high half) |
+| Scalar i32 / u32 | `movnti` (SSE2) | plain `str` — NT hint silently dropped |
+| Scalar i16 / u16 | regular `mov` — NT hint silently dropped | plain `strh` — NT hint silently dropped |
+
+**aarch64 has no scalar non-temporal store instruction.** The only NT store on
+aarch64 is `stnp` (Store Non-temporal Pair), which requires two operands. LLVM
+18 honors `!nontemporal` only when it can synthesize an `stnp`:
+
+- **64-bit scalars (i64/u64)** self-pair to a `w` register pair — NT hint preserved.
+- **128-bit vectors** self-pair to a `d` register pair — NT hint preserved.
+- **32-bit and 16-bit scalars** have no pair-friendly form — LLVM emits plain
+  `str` / `strh` and the NT hint is dropped silently.
+
+For aarch64 NT semantics, **prefer 64-bit or wider element widths.** The i32/u32
+and i16/u16 scalar overloads still type-check and run, but provide no cache-
+bypass benefit on aarch64; the same is true of i16/u16 on x86 (no `movnti16`
+exists). They ship for cross-platform shape symmetry — a single Eä kernel using
+`stream_store` compiles and runs on both targets without per-width branching.
+
+Note also that LLVM 18 does **not** fuse two adjacent `stream_store(*mut i32, ...)`
+calls into a single `stnp` pair on aarch64 — they lower to a regular `stp` with
+the NT hint dropped. If you need NT-paired stores on aarch64, write the data as
+i64 (two 32-bit values packed) or as a vector type.
+
+**Alignment contract:**
+
+Vector `stream_store` requires the destination pointer plus byte offset to
+be aligned to the vector's natural size (16 bytes for 128-bit, 32 bytes for
+256-bit, 64 bytes for 512-bit). Misaligned NT vector stores raise a general
+protection fault on x86. Scalar `stream_store` requires natural alignment to
+the scalar size on x86 (4-byte for i32/u32, 8-byte for i64/u64). Callers
+must provide aligned buffers; Eä does not insert runtime alignment checks.
+
+**Ordering contract:**
+
+NT stores are weakly ordered on x86 (write-combining memory order). Other
+cores or subsequent reads in the same thread may observe them out of
+program order. For cross-thread visibility, the typical pathway is through
+a host-side synchronization primitive after the kernel returns
+(`pthread_join`, `rayon::scope`, `WaitGroup.Wait`) — these provide release
+semantics that flush WC buffers. For intra-kernel ordering (writing then
+reading the same memory in the same kernel call), use `fence_nt()`
+explicitly. Eä does not insert an implicit fence at kernel return.
+
+**When NOT to use:**
+
+Do not use `stream_store` for working buffers the same kernel reads back.
+The non-temporal hint asks the cache to *not* keep the line; if the kernel
+reads the data soon afterward, the read goes to DRAM and is slower than a
+regular `store` followed by a cache hit. Working-buffer examples that
+should use plain `store`:
+
+- Softmax accumulators (e.g. `scores_buf` in attention kernels)
+- FWHT scratch arrays (e.g. `scratch` in JL-projection kernels)
+- Per-iteration partial sums or running statistics
+
+`stream_store` is appropriate when the destination is a final output passed
+to the next kernel call, a memory region the current kernel never re-reads,
+or a buffer that will not be touched again until a downstream consumer
+pulls it from DRAM later.
+
+### fence_nt
+
+Store-store memory barrier providing intra-kernel ordering of preceding
+`stream_store` operations. Zero arguments, returns void.
+
+```
+fence_nt()
+```
+
+**Target lowering:**
+
+| Target | Instruction |
+|---|---|
+| x86 | `sfence` (via `@llvm.x86.sse.sfence`) |
+| aarch64 | `dmb ishst` (via `@llvm.aarch64.dmb` with operand `10`) |
+
+These are the narrowest available barriers for store-only ordering —
+explicit target intrinsics rather than the IR-level `fence release`, which
+would lower to `mfence` on x86 and `dmb ish` on aarch64 (both heavier than
+needed for NT-store ordering).
+
+**Semantics:**
+
+`fence_nt()` orders `stream_store` writes relative to each other and
+relative to subsequent regular stores. It does *not* order stores relative
+to subsequent loads — for a write-then-read-back pattern in the same kernel,
+a full barrier (`mfence` on x86, `dmb sy` on aarch64) is needed instead.
+Eä does not currently expose a full-barrier intrinsic.
+
+**When to use:**
+
+Use `fence_nt()` when the same kernel writes via `stream_store` to multiple
+non-overlapping regions in a defined order and a later kernel (or downstream
+reader) relies on observing those writes in the same order. This is
+uncommon — most callers don't need it, because cross-thread visibility
+comes from the host's sync primitive (`pthread_join`, `rayon::scope`,
+`WaitGroup.Wait`) which already provides release semantics that flush
+write-combining buffers between threads.
+
+**When NOT to use:**
+
+- Between successive `stream_store` calls to different addresses if no
+  store-ordering requirement exists — NT stores to the same address
+  complete in program order regardless of fences.
+- At the end of a kernel as "insurance" — the caller's sync primitive
+  handles cross-thread fencing more efficiently after the kernel returns.
+- For write-then-read-back patterns — `fence_nt()` does not provide store-
+  to-load ordering. Use a regular `store` for the working data, or do not
+  read NT-written data back in the same kernel.
+
 ### load_masked
 
 Masked vector load. Lanes where the mask is false are not loaded.

src/codegen/mod.rs

@@ -19,6 +19,8 @@ mod simd_dotprod;
 #[cfg(feature = "llvm")]
 mod simd_exp_poly;
 #[cfg(feature = "llvm")]
+mod simd_fence;
+#[cfg(feature = "llvm")]
 mod simd_fp16;
 #[cfg(feature = "llvm")]
 mod simd_lane;