v1.15.0 — stream_store family completion - #38
Merged
Conversation
Adds check_stream_store separate from check_store. Allows i16/u16/i32/u32/i64/u64 scalar values in addition to vectors, prerequisite for q4k_repack-style scalar streaming writes that cannot use the vector form. Codegen still scalarizes through .into_vector_value() — next commit completes the scalar codegen path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
compile_stream_store now branches on scalar vs. vector value type. Scalar path stores a single i16/u16/i32/u32/i64/u64 with the same !nontemporal metadata as the vector form. Lowers to movnti on x86 SSE2 for i32/i64; i16/u16 fall through to regular mov on x86 since no dedicated NT scalar instruction exists at that width. Closes the q4k_repack scalar-write surface gap identified during v1.15 brainstorm consumer audit (Olorin v2.0.3). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New zero-arg intrinsic providing intra-kernel store ordering after stream_store writes. Lowers to: - x86: @llvm.x86.sse.sfence() -> sfence - aarch64: @llvm.aarch64.dmb(i32 10) -> dmb ishst Explicit target intrinsics rather than IR fence release, which would lower to mfence (x86) / dmb ish (aarch64) — heavier than needed for store-only ordering. Cross-thread visibility comes from host-side sync primitives (pthread_join, rayon::scope, WaitGroup.Wait) which already provide release semantics. fence_nt() is for the rare intra-kernel case where the same kernel writes via stream_store and reads the same memory back before returning. Completes the prefetch_nta + stream_store + fence_nt non-temporal memory-hint family. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Documents what LLVM 18 emits on aarch64 for !nontemporal scalar stores and fence_nt via runtime + IR assertions. Tests are target_arch-gated; require Pi 5 (per feedback_pi5_access_constraint) for full run before release-cut. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds assert_intrinsic_in_disassembly checks for movnti (scalar i32/i64), movntps / movntdq (vector 128-bit), vmovntps / vmovntdq (vector 256-bit), and sfence (fence_nt). Per feedback_llvm_intrinsic_linktest, IR-level !nontemporal does not guarantee the emitted assembly uses an NT mnemonic — LLVM can drop the hint silently. Also adds a deliberate alignment-failure crash test pinning the documented alignment contract: vmovntps with a 1-byte-misaligned pointer must raise SIGSEGV. Uses child-process pattern so the parent test runner survives. Fixes vector stream_store codegen: alignment was incorrectly set to element size (4 bytes for f32), causing LLVM to emit scalar movntsd instead of vector movntps/vmovntps. Now computed as element_size * element_count to signal vector-width intent to LLVM's backend. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
stream_store reference brought to prefetch_nta parity:
- Target-specific lowering table covering vector + scalar widths
- Explicit alignment contract (GPF on x86 misalignment)
- Explicit ordering contract (weak ordering, caller-side fence)
- "When NOT to use" section listing working-buffer anti-patterns
(softmax accumulators, FWHT scratch) to prevent regressions
during consumer adoption
New fence_nt reference section covering target lowering (sfence /
dmb ishst), intended-use (intra-kernel ordering only), and
anti-patterns (cross-thread fencing is the caller's sync primitive's
job; store-to-load ordering requires a full barrier).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bumps version to 1.15.0. CHANGELOG documents the additions (scalar stream_store overloads, fence_nt), the fixed pre-existing vector alignment bug, the docs upgrade, and the test hardening. ROADMAP amends the Multi-core / parallel_for entry with the v1.15 audit finding (Olorin + Cougar already ship custom SpinBarrier-based pools strictly more capable than any generic primitive eacompute could provide). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
VectorType::get_size() already returns u32; the explicit cast triggers clippy::unnecessary_cast. Caught by the v1.15.0 release-cut clippy gate. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Remove stale "Task 2 lands later" comment in stream_store scalar typeck test; Task 2 has shipped on this branch. - Fix misleading "Read back after fence to verify visibility" comment in test_fence_nt_runtime_x86 — the kernel writes + fences; the C host reads back, not the kernel. - Replace "scalar movntsd sequences" with the more accurate "sequence of scalar (non-temporal-hinted) stores" in CHANGELOG and ROADMAP — movntsd is SSE4a/AMD-specific, the actual LLVM fallback for under- aligned NT stores is not guaranteed to be that specific mnemonic. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pi 5 (Cortex-A76, LLVM 18.1.8) capture revealed the original v1.15.0
aarch64 lowering claims were wrong for 4 of the 6 stream_store rows.
Root cause: aarch64 has no scalar non-temporal store instruction. The
only NT store is stnp (Store Non-temporal Pair), which requires two
operands. LLVM 18 honors !nontemporal only when it can synthesize stnp:
- i64 self-pairs to a w-pair via lsr — NT preserved
- 128-bit vector q-register self-pairs to a d-pair — NT preserved
- i32 / i16 scalars have no pair-friendly form — NT hint dropped,
plain str / strh emitted
Adjacent stream_store(*mut i32, ...) calls fuse to regular stp on
aarch64, not stnp — LLVM does not synthesize NT pairs from sequential
non-NT-fusable stores.
Doc and metadata changes:
- docs/src/reference/intrinsics.md: target-lowering table corrected;
added explanatory paragraph about aarch64-no-scalar-NT semantics
and the "prefer 64-bit-or-wider on aarch64" guidance.
- CHANGELOG.md: scalar overloads entry now lists the actual per-target
lowering matrix instead of x86-only mnemonics.
- ROADMAP.md: shipped-in-v1.15.0 entry expanded with the aarch64
semantics summary and pointer to the reference for the full matrix.
Test additions (aarch64-gated, pinning observed LLVM 18 behavior):
- test_stream_store_scalar_i64_emits_stnp_aarch64
- test_stream_store_f32x4_emits_stnp_aarch64
- test_stream_store_scalar_i32_emits_plain_str_aarch64 (PIN: hint dropped)
- test_stream_store_scalar_i16_emits_plain_strh_aarch64 (PIN: hint dropped)
- test_fence_nt_emits_dmb_ishst_aarch64
Mirrors the x86 objdump discipline so a future LLVM upgrade that
changes any of these mnemonics will fail loudly instead of silently
drifting away from the reference docs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
stream_storeoverloads (i16/u16/i32/u32/i64/u64) — closes the surface gap blocking Olorin'sq4k_repack.eakernel. Lowers tomovntion x86 SSE2 for i32/i64.fence_nt()intrinsic — zero-arg store-store barrier (sfenceon x86,dmb ishston aarch64). Completes theprefetch_nta+stream_store+fence_ntnon-temporal memory-hint family.stream_storealignment fix (pre-existing bug surfaced by new objdump tests) — alignment was set to element width, causing LLVM to silently decompose 128/256-bit NT stores to scalar non-temporal-hinted sequences. Now set toelement_size * lane_count. Behavior change: vector-aligned buffers get the fast path; misaligned buffers now SIGSEGV per the documented contract.stream_storereference docs upgraded toprefetch_ntaparity — target-lowering table, alignment contract, ordering contract, and "When NOT to use" anti-pattern guidance (softmax accumulators, FWHT scratch).Test Plan
-D warningsfeedback_pi5_access_constraint): check out branch onpeter@10.46.0.27, runcargo test --features llvm --release --test stream_store_tests --test fence_nt_tests— the 3 aarch64-gated tests must pass on real silicon.fence_nt()lowers todmb ishstand scalarstream_store(*mut i64, ..., v)lowers tostr/stnp. If mnemonics differ from docs, amend before tagging.v1.15.0after Pi 5 verification passes.