From 2e9f5a8f6bffdd5a115bc5c1ff6ab64674884ef6 Mon Sep 17 00:00:00 2001 From: Sebastian Mocanu Date: Fri, 14 Aug 2026 23:35:31 +0100 Subject: [PATCH 01/13] dispatch: route the complex groups, which had never reached their assembly complexOps returned the portable reference set directly, and the generator emitted no complex entries into the per-tier dispatch tables. So DotComplex, DotComplexConj, ScaleComplex, SumComplex and the complex arithmetic set ran ordinary Go on every machine, while nine tiers of generated complex assembly -- sse2, avx2, avx512, neon, sve2, rvv, vsx, vx, lasx, across six architectures -- sat linked into a test-only aggregator and were never called. Measured on amd64/avx512, scalar against the tier in one binary: 1.9x to 12.9x depending on operation and length. Results are unchanged, and that is checkable rather than asserted: the numerical contract is the same fixed accumulation order, and the reference and every tier agree bit for bit. The cache had to grow a second shape. opsCache is generic over the ELEMENT type and kernel.Complex and kernel.ComplexParts are not kernel.Ops, so groupCache is generic over the struct itself -- same lazy per-tier merge, one type parameter further out. The generator now REFUSES to emit a dispatch file for a kernel group it cannot route. That is the part worth keeping: this defect was invisible because everything downstream of it worked. The tests passed, the assembly assembled, the emission check counted the kernels as emitted, and the only symptom was that the fast path was never taken -- which nothing measured. A gate that asserts "the group is in the table" would not have caught it either, since the group was in no table; the assertion is on the generator's own completeness. docs/wrong.md carries the finding, the measurement, and the n-ary closure research: a general CombineInto taking a Go closure is measured, designed and NOT implemented, because a closure call per element defeats the vectorization that is the reason to call it. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 15 +++ Makefile | 2 +- README.md | 2 +- complex.go | 8 +- dispatch.go | 45 +++++++- dispatch_complex_test.go | 155 +++++++++++++++++++++++++++ dispatch_inventory_test.go | 49 +++++++++ dispatch_tables_amd64.go | 148 +++++++++++++++++++++++++ dispatch_tables_amd64_test.go | 20 ++++ dispatch_tables_arm64.go | 104 ++++++++++++++++++ dispatch_tables_arm64_test.go | 20 ++++ dispatch_tables_fallback.go | 4 + dispatch_tables_fallback_test.go | 4 + dispatch_tables_loong64.go | 52 +++++++++ dispatch_tables_loong64_test.go | 20 ++++ dispatch_tables_ppc64le.go | 56 ++++++++++ dispatch_tables_ppc64le_test.go | 20 ++++ dispatch_tables_riscv64.go | 60 +++++++++++ dispatch_tables_riscv64_test.go | 20 ++++ dispatch_tables_s390x.go | 55 ++++++++++ dispatch_tables_s390x_test.go | 20 ++++ docs/architecture.md | 4 +- docs/lld/kernels-and-dispatch.md | 9 ++ docs/research/07-nary-closure.md | 103 ++++++++++++++++++ docs/wrong.md | 143 ++++++++++++++++++++++++ tools/simdgen/emit/dispatch_gen.go | 149 +++++++++++++++++++++++++ tools/simdgen/emit/ungrouped_test.go | 26 +++++ 27 files changed, 1304 insertions(+), 9 deletions(-) create mode 100644 dispatch_complex_test.go create mode 100644 docs/research/07-nary-closure.md create mode 100644 tools/simdgen/emit/ungrouped_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 0ef52a1..032e3e4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,20 @@ # Changelog +## Unreleased + +**Complex operations now run the generated assembly.** They never did: +`complexOps` returned the portable reference set directly, and the +generator emitted no complex entries into the per-tier dispatch tables, so +`DotComplex`, `DotComplexConj`, `ScaleComplex`, `SumComplex` and the +complex arithmetic set ran ordinary Go on every architecture while nine +tiers of generated complex assembly sat linked into a test-only aggregator. +Measured on amd64/avx512, scalar against the tier in one binary: 1.9x-12.9x +depending on operation and length. Results are unchanged -- the numerical +contract is the same fixed accumulation order, and the reference and every +tier agree bit for bit. No API change. `docs/wrong.md` records the finding, +the measurement, and the generator assertion that now refuses to emit a +dispatch file for a kernel group it cannot route. + ## v1.20.0 **The columnar codec tier.** **`BitUnpackInto`** gains a width- diff --git a/Makefile b/Makefile index a27e8f1..79868af 100644 --- a/Makefile +++ b/Makefile @@ -71,7 +71,7 @@ test-vec: ## Run the suite with GOEXPERIMENT=simd, which compiles vec.go test-tiers: ## Run the suite once per instruction-set tier this CPU has @for t in $(TIERS); do \ echo "--- GOSIMD=$$t"; \ - GOSIMD=$$t $(GO) test $(PKG) || exit 1; \ + GOSIMD=$$t $(GO) test -count=1 $(PKG) || exit 1; \ done .PHONY: test-race diff --git a/README.md b/README.md index 38136c0..abc4a11 100644 --- a/README.md +++ b/README.md @@ -389,7 +389,7 @@ cgo. ## Engineering record -[**docs/wrong.md**](docs/wrong.md) records 79 things that measurement disproved, +[**docs/wrong.md**](docs/wrong.md) records 80 things that measurement disproved, including changes that were deleted rather than shipped. Examples: - Green test lanes had executed no accelerated code for months. diff --git a/complex.go b/complex.go index 18701d5..09f7dd1 100644 --- a/complex.go +++ b/complex.go @@ -28,9 +28,9 @@ func complexOps[C Complex]() *kernel.Complex[C] { var zero C switch any(zero).(type) { case complex64: - return any(&refBase.C64).(*kernel.Complex[C]) + return any(cacheC64.get(&refBase.C64, cplxC64ByTier[:])).(*kernel.Complex[C]) case complex128: - return any(&refBase.C128).(*kernel.Complex[C]) + return any(cacheC128.get(&refBase.C128, cplxC128ByTier[:])).(*kernel.Complex[C]) } panic("simd: unsupported complex type") } @@ -44,11 +44,11 @@ func complexParts[C Complex, R Float]() *kernel.ComplexParts[C, R] { var zero C switch any(zero).(type) { case complex64: - if p, ok := any(&refBase.C64Parts).(*kernel.ComplexParts[C, R]); ok { + if p, ok := any(cacheC64Parts.get(&refBase.C64Parts, partsC64PartsByTier[:])).(*kernel.ComplexParts[C, R]); ok { return p } case complex128: - if p, ok := any(&refBase.C128Parts).(*kernel.ComplexParts[C, R]); ok { + if p, ok := any(cacheC128Parts.get(&refBase.C128Parts, partsC128PartsByTier[:])).(*kernel.ComplexParts[C, R]); ok { return p } } diff --git a/dispatch.go b/dispatch.go index bd7fef7..9c15d9c 100644 --- a/dispatch.go +++ b/dispatch.go @@ -96,8 +96,49 @@ func (c *opsCache[T]) get(base *kernel.Ops[T], partial []*kernel.Ops[T]) *kernel return c.tiers[tierIdx] } -// overlay copies every non-nil function field of src over dst. -func overlay[T Number](dst, src *kernel.Ops[T]) { +// groupCache is opsCache for the two complex groups. They are the same shape +// -- a struct of function fields, a static per-tier partial, a lazy merge -- +// but kernel.Complex and kernel.ComplexParts are not kernel.Ops, and Go has +// no constraint that spans the three. One generic type over the struct +// itself, rather than over the element type, covers both. +// +// Until this existed, complexOps returned the reference set directly and +// every complex operation ran portable Go on every machine while nine tiers +// of generated complex assembly -- sse2, avx2, avx512, neon, sve2, rvv, vsx, +// vx, lasx, across six architectures -- sat linked into the test-only +// aggregator. +type groupCache[G any] struct { + once sync.Once + tiers [len(dispatchTiers)]*G +} + +func (c *groupCache[G]) get(base *G, partial []*G) *G { + c.once.Do(func() { + for i := range c.tiers { + s := *base + if i < len(partial) && partial[i] != nil { + overlayAny(&s, partial[i]) + } + c.tiers[i] = &s + } + }) + return c.tiers[tierIdx] +} + +var ( + cacheC64 groupCache[kernel.Complex[complex64]] + cacheC128 groupCache[kernel.Complex[complex128]] + cacheC64Parts groupCache[kernel.ComplexParts[complex64, float32]] + cacheC128Parts groupCache[kernel.ComplexParts[complex128, float64]] +) + +// overlay copies every non-nil function field of src over dst. It is +// overlayAny at kernel.Ops[T], kept as a named function because that is what +// opsCache reads and the constraint documents the intent. +func overlay[T Number](dst, src *kernel.Ops[T]) { overlayAny(dst, src) } + +// overlayAny is overlay for any struct of function fields. +func overlayAny[G any](dst, src *G) { d := reflect.ValueOf(dst).Elem() s := reflect.ValueOf(src).Elem() for i := 0; i < d.NumField(); i++ { diff --git a/dispatch_complex_test.go b/dispatch_complex_test.go new file mode 100644 index 0000000..5b7aaed --- /dev/null +++ b/dispatch_complex_test.go @@ -0,0 +1,155 @@ +package simd + +import ( + "reflect" + "testing" +) + +// The complex surface must reach the generated kernels, the same way the +// numeric surface does. +// +// It did not, from v1.14.0 to v1.20.0. complexOps returned &refBase.C64 / +// &refBase.C128 directly, with no per-tier overlay, so every complex +// operation ran the portable Go reference on every machine -- while the +// repository generated, tested and shipped nine tiers of complex assembly +// for them. The existing coverage did not catch it because +// TestDeclaredKernelsAreWired checks the runtime tables for exactly three +// groups (Bytes, Convert, Mask) and routes everything else -- numeric and +// complex alike -- through archSets(), the test-only whole-set aggregator +// whose own generated header says consumers never call it. Nothing walked +// the tables the runtime actually indexes. +// +// So this asks the question a caller asks: is the function DotComplex ends +// up calling the one the tier's table supplies? + +// tierPartial returns the per-tier partial for a group at the running tier, +// or nil when this build has no table for it (the purego and no-backend +// builds, and slot 0, the reference). +func tierPartial[G any](byTier []*G) *G { + if tierIdx < 0 || tierIdx >= len(byTier) { + return nil + } + return byTier[tierIdx] +} + +// checkGroup is the whole contract, field by field: where the tier's partial +// supplies a kernel the dispatch must reach exactly that function, and where +// it does not the dispatch must keep the reference. +// +// The second half matters as much as the first. A missing kernel on a tier +// is documented as slower and never a correctness gap +// (docs/lld/kernels-and-dispatch.md), and three architectures rely on it +// today -- s390x has no complex64 Dot or DotConj, ppc64le adds Sum to that +// list, loong64 is missing Mul, Div and Conj. An earlier version of this +// test asserted "not the reference" for one hardcoded field and turned that +// documented fallback into a red cross-compile gate on two architectures. +func checkGroup[G any](t *testing.T, name string, got, ref, partial *G) { + t.Helper() + g := reflect.ValueOf(got).Elem() + r := reflect.ValueOf(ref).Elem() + var p reflect.Value + if partial != nil { + p = reflect.ValueOf(partial).Elem() + } + typ := g.Type() + + supplied := 0 + for i := 0; i < g.NumField(); i++ { + if typ.Field(i).Type.Kind() != reflect.Func { + continue + } + field := name + "." + typ.Field(i).Name + gp := g.Field(i).Pointer() + rp := r.Field(i).Pointer() + + if partial != nil && !p.Field(i).IsNil() { + supplied++ + want := p.Field(i).Pointer() + if gp != want { + t.Errorf("%s on tier %q: dispatch reaches %#x, the tier's table holds %#x", + field, Tier(), gp, want) + } + if gp == rp && rp != want { + t.Errorf("%s on tier %q: dispatch is still the portable reference", field, Tier()) + } + continue + } + // No kernel on this tier: the reference stands, unchanged. + if gp != rp { + t.Errorf("%s on tier %q: no kernel is generated, but dispatch is not the reference", + field, Tier()) + } + } + + // A table that supplies nothing on an accelerated tier is a table that + // was emitted empty -- which is what the defect looked like. + if partial != nil && supplied == 0 && Tier() != "scalar" { + t.Errorf("%s on tier %q: the per-tier table supplies no kernel at all", name, Tier()) + } +} + +func TestComplexDispatchReachesGeneratedKernels(t *testing.T) { + checkGroup(t, "C64", complexOps[complex64](), &refBase.C64, tierPartial(cplxC64ByTier[:])) + checkGroup(t, "C128", complexOps[complex128](), &refBase.C128, tierPartial(cplxC128ByTier[:])) +} + +// The real-scalar half of the split -- Abs, Real, Imag, Scale -- goes through +// ComplexParts and had the same hole. +func TestComplexPartsDispatchReachesGeneratedKernels(t *testing.T) { + checkGroup(t, "C64Parts", complexParts[complex64, float32](), + &refBase.C64Parts, tierPartial(partsC64PartsByTier[:])) + checkGroup(t, "C128Parts", complexParts[complex128, float64](), + &refBase.C128Parts, tierPartial(partsC128PartsByTier[:])) +} + +// groupCache indexes its partial slice by tierIdx and guards with +// `i < len(partial)`, so a table shorter than dispatchTiers silently leaves +// the top tiers on the reference and a longer one silently drops its tail. +// Both are generator bugs that no other test would notice. +func TestComplexTierTablesMatchTheTierList(t *testing.T) { + for _, c := range []struct { + name string + n int + }{ + {"cplxC64ByTier", len(cplxC64ByTier)}, + {"cplxC128ByTier", len(cplxC128ByTier)}, + {"partsC64PartsByTier", len(partsC64PartsByTier)}, + {"partsC128PartsByTier", len(partsC128PartsByTier)}, + } { + if c.n != len(dispatchTiers) { + t.Errorf("%s has %d slots, dispatchTiers has %d", c.name, c.n, len(dispatchTiers)) + } + } +} + +// Whatever the dispatch routes to, the answer is the reference's answer. +// The numerical contract is the one every other tier keeps: a fixed +// accumulation order, identical on every architecture. +func TestComplexDispatchMatchesTheReference(t *testing.T) { + n := 1000 + a := make([]complex128, n) + b := make([]complex128, n) + for i := range a { + a[i] = complex(float64(i%17)-8, float64(i%23)-11) + b[i] = complex(float64(i%13)-6, float64(i%29)-14) + } + if got, want := DotComplex(a, b), refBase.C128.Dot(a, b); got != want { + t.Errorf("DotComplex = %v, reference = %v", got, want) + } + if got, want := DotComplexConj(a, b), refBase.C128.DotConj(a, b); got != want { + t.Errorf("DotComplexConj = %v, reference = %v", got, want) + } + if got, want := SumComplex(a), refBase.C128.Sum(a); got != want { + t.Errorf("SumComplex = %v, reference = %v", got, want) + } + + gotS := append([]complex128(nil), a...) + wantS := append([]complex128(nil), a...) + ScaleComplex(gotS, 1.5) + refBase.C128Parts.Scale(wantS, wantS, 1.5) + for i := range gotS { + if gotS[i] != wantS[i] { + t.Fatalf("ScaleComplex[%d] = %v, reference = %v", i, gotS[i], wantS[i]) + } + } +} diff --git a/dispatch_inventory_test.go b/dispatch_inventory_test.go index f5e4572..9f9de1e 100644 --- a/dispatch_inventory_test.go +++ b/dispatch_inventory_test.go @@ -65,6 +65,30 @@ func TestDeclaredKernelsAreWired(t *testing.T) { } continue } + // The struct groups now have real per-tier tables, so walk those -- + // the runtime indexes them, and archSets() does not. Four whole + // complex groups were missing from the runtime tables for seven + // releases with this test green, because everything that was not a + // flat group went through the test-only aggregator instead. + if slots, ok := allGroupTables[d.Group]; ok { + for i, partial := range slots { + if i == 0 || partial == nil { + continue // slot 0 is the reference; no overlay + } + f := reflect.ValueOf(partial).FieldByName(d.Field) + if !f.IsValid() { + t.Errorf("%s: per-tier table for %s has no field %q", + d.CName, d.Group, d.Field) + break + } + if f.Kind() == reflect.Func && f.IsNil() && present(sets, dispatchTiers[i], d) { + t.Errorf("tier %q: %s.%s is nil in the dispatch table, but the "+ + "kernel is generated for that tier -- it will never run", + dispatchTiers[i], d.Group, d.Field) + } + } + continue + } for tier, set := range sets { g := reflect.ValueOf(set).FieldByName(d.Group) if !g.IsValid() { @@ -105,3 +129,28 @@ func TestInventoryCoversEveryGroup(t *testing.T) { } } } + +// present reports whether the generator emitted a kernel for d on tier. +// +// Comparing against the REFERENCE, not against nil: archSets() builds each +// tier from ref.Set() and overlays what was generated, so every field is +// non-nil there and a nil test would call every operation "generated" on +// every tier. A tier the generator declined keeps the reference by design +// (docs/lld/kernels-and-dispatch.md), and only a kernel that exists and is +// missing from the dispatch table is a defect. +func present(sets map[string]kernel.Set, tier string, d backend.Declared) bool { + set, ok := sets[tier] + if !ok { + return false + } + g := reflect.ValueOf(set).FieldByName(d.Group) + r := reflect.ValueOf(refBase).FieldByName(d.Group) + if !g.IsValid() || !r.IsValid() { + return false + } + f, rf := g.FieldByName(d.Field), r.FieldByName(d.Field) + if !f.IsValid() || !rf.IsValid() || f.Kind() != reflect.Func || f.IsNil() { + return false + } + return rf.IsNil() || f.Pointer() != rf.Pointer() +} diff --git a/dispatch_tables_amd64.go b/dispatch_tables_amd64.go index ed7ad96..1d16221 100644 --- a/dispatch_tables_amd64.go +++ b/dispatch_tables_amd64.go @@ -3175,3 +3175,151 @@ var opsU8ByTier = [...]*kernel.Ops[uint8]{ &opsU8AVX2, &opsU8AVX512, } + +var cplxC128SSE2 = kernel.Complex[complex128]{ + Add: amd64.CaddComplex128SSE2, + Sub: amd64.CsubComplex128SSE2, + Mul: amd64.CmulComplex128SSE2, + Div: amd64.CdivComplex128SSE2, + Neg: amd64.CnegComplex128SSE2, + Conj: amd64.CconjComplex128SSE2, + Sum: amd64.CsumComplex128SSE2, + Dot: amd64.CdotComplex128SSE2, + DotConj: amd64.CdotconjComplex128SSE2, +} + +var cplxC128AVX2 = kernel.Complex[complex128]{ + Add: amd64.CaddComplex128AVX2, + Sub: amd64.CsubComplex128AVX2, + Mul: amd64.CmulComplex128AVX2, + Div: amd64.CdivComplex128AVX2, + Neg: amd64.CnegComplex128AVX2, + Conj: amd64.CconjComplex128AVX2, + Sum: amd64.CsumComplex128AVX2, + Dot: amd64.CdotComplex128AVX2, + DotConj: amd64.CdotconjComplex128AVX2, +} + +var cplxC128AVX512 = kernel.Complex[complex128]{ + Add: amd64.CaddComplex128AVX512, + Sub: amd64.CsubComplex128AVX512, + Mul: amd64.CmulComplex128AVX512, + Div: amd64.CdivComplex128AVX512, + Neg: amd64.CnegComplex128AVX512, + Conj: amd64.CconjComplex128AVX512, + Sum: amd64.CsumComplex128AVX512, + Dot: amd64.CdotComplex128AVX512, + DotConj: amd64.CdotconjComplex128AVX512, +} + +var cplxC128ByTier = [...]*kernel.Complex[complex128]{ + nil, + &cplxC128SSE2, + &cplxC128AVX2, + &cplxC128AVX512, +} + +var cplxC64SSE2 = kernel.Complex[complex64]{ + Add: amd64.CaddComplex64SSE2, + Sub: amd64.CsubComplex64SSE2, + Mul: amd64.CmulComplex64SSE2, + Div: amd64.CdivComplex64SSE2, + Neg: amd64.CnegComplex64SSE2, + Conj: amd64.CconjComplex64SSE2, + Sum: amd64.CsumComplex64SSE2, + Dot: amd64.CdotComplex64SSE2, + DotConj: amd64.CdotconjComplex64SSE2, +} + +var cplxC64AVX2 = kernel.Complex[complex64]{ + Add: amd64.CaddComplex64AVX2, + Sub: amd64.CsubComplex64AVX2, + Mul: amd64.CmulComplex64AVX2, + Div: amd64.CdivComplex64AVX2, + Neg: amd64.CnegComplex64AVX2, + Conj: amd64.CconjComplex64AVX2, + Sum: amd64.CsumComplex64AVX2, + Dot: amd64.CdotComplex64AVX2, + DotConj: amd64.CdotconjComplex64AVX2, +} + +var cplxC64AVX512 = kernel.Complex[complex64]{ + Add: amd64.CaddComplex64AVX512, + Sub: amd64.CsubComplex64AVX512, + Mul: amd64.CmulComplex64AVX512, + Div: amd64.CdivComplex64AVX512, + Neg: amd64.CnegComplex64AVX512, + Conj: amd64.CconjComplex64AVX512, + Sum: amd64.CsumComplex64AVX512, + Dot: amd64.CdotComplex64AVX512, + DotConj: amd64.CdotconjComplex64AVX512, +} + +var cplxC64ByTier = [...]*kernel.Complex[complex64]{ + nil, + &cplxC64SSE2, + &cplxC64AVX2, + &cplxC64AVX512, +} + +var partsC128PartsSSE2 = kernel.ComplexParts[complex128, float64]{ + Abs: amd64.CabsComplex128SSE2, + Real: amd64.CrealComplex128SSE2, + Imag: amd64.CimagComplex128SSE2, + Scale: amd64.CscaleComplex128SSE2, + FromParts: amd64.CfromPartsComplex128SSE2, +} + +var partsC128PartsAVX2 = kernel.ComplexParts[complex128, float64]{ + Abs: amd64.CabsComplex128AVX2, + Real: amd64.CrealComplex128AVX2, + Imag: amd64.CimagComplex128AVX2, + Scale: amd64.CscaleComplex128AVX2, + FromParts: amd64.CfromPartsComplex128AVX2, +} + +var partsC128PartsAVX512 = kernel.ComplexParts[complex128, float64]{ + Abs: amd64.CabsComplex128AVX512, + Real: amd64.CrealComplex128AVX512, + Imag: amd64.CimagComplex128AVX512, + Scale: amd64.CscaleComplex128AVX512, + FromParts: amd64.CfromPartsComplex128AVX512, +} + +var partsC128PartsByTier = [...]*kernel.ComplexParts[complex128, float64]{ + nil, + &partsC128PartsSSE2, + &partsC128PartsAVX2, + &partsC128PartsAVX512, +} + +var partsC64PartsSSE2 = kernel.ComplexParts[complex64, float32]{ + Abs: amd64.CabsComplex64SSE2, + Real: amd64.CrealComplex64SSE2, + Imag: amd64.CimagComplex64SSE2, + Scale: amd64.CscaleComplex64SSE2, + FromParts: amd64.CfromPartsComplex64SSE2, +} + +var partsC64PartsAVX2 = kernel.ComplexParts[complex64, float32]{ + Abs: amd64.CabsComplex64AVX2, + Real: amd64.CrealComplex64AVX2, + Imag: amd64.CimagComplex64AVX2, + Scale: amd64.CscaleComplex64AVX2, + FromParts: amd64.CfromPartsComplex64AVX2, +} + +var partsC64PartsAVX512 = kernel.ComplexParts[complex64, float32]{ + Abs: amd64.CabsComplex64AVX512, + Real: amd64.CrealComplex64AVX512, + Imag: amd64.CimagComplex64AVX512, + Scale: amd64.CscaleComplex64AVX512, + FromParts: amd64.CfromPartsComplex64AVX512, +} + +var partsC64PartsByTier = [...]*kernel.ComplexParts[complex64, float32]{ + nil, + &partsC64PartsSSE2, + &partsC64PartsAVX2, + &partsC64PartsAVX512, +} diff --git a/dispatch_tables_amd64_test.go b/dispatch_tables_amd64_test.go index 519ecb7..151d2bf 100644 --- a/dispatch_tables_amd64_test.go +++ b/dispatch_tables_amd64_test.go @@ -127,5 +127,25 @@ var allFlatTables = map[string][]any{ "Bytes.VarintDecodeU64": {tblBytesVarintDecodeU64[0], tblBytesVarintDecodeU64[1], tblBytesVarintDecodeU64[2], tblBytesVarintDecodeU64[3]}, } +// allGroupTables enumerates the per-tier partial structs the runtime +// overlays from, keyed by kernel.Set group. Slot 0 is the reference +// and is nil; the rest are one per accelerated tier, in tier order. +var allGroupTables = map[string][]any{ + "F32": {nil, opsF32SSE2, opsF32AVX2, opsF32AVX512}, + "F64": {nil, opsF64SSE2, opsF64AVX2, opsF64AVX512}, + "I16": {nil, opsI16SSE2, opsI16AVX2, opsI16AVX512}, + "I32": {nil, opsI32SSE2, opsI32AVX2, opsI32AVX512}, + "I64": {nil, opsI64SSE2, opsI64AVX2, opsI64AVX512}, + "I8": {nil, opsI8SSE2, opsI8AVX2, opsI8AVX512}, + "U16": {nil, opsU16SSE2, opsU16AVX2, opsU16AVX512}, + "U32": {nil, opsU32SSE2, opsU32AVX2, opsU32AVX512}, + "U64": {nil, opsU64SSE2, opsU64AVX2, opsU64AVX512}, + "U8": {nil, opsU8SSE2, opsU8AVX2, opsU8AVX512}, + "C128": {nil, cplxC128SSE2, cplxC128AVX2, cplxC128AVX512}, + "C64": {nil, cplxC64SSE2, cplxC64AVX2, cplxC64AVX512}, + "C128Parts": {nil, partsC128PartsSSE2, partsC128PartsAVX2, partsC128PartsAVX512}, + "C64Parts": {nil, partsC64PartsSSE2, partsC64PartsAVX2, partsC64PartsAVX512}, +} + // archSets hands the tests every tier's complete kernel set. var archSets func() map[string]kernel.Set = amd64.Sets diff --git a/dispatch_tables_arm64.go b/dispatch_tables_arm64.go index 99f8cc6..343289f 100644 --- a/dispatch_tables_arm64.go +++ b/dispatch_tables_arm64.go @@ -2232,3 +2232,107 @@ var opsU8ByTier = [...]*kernel.Ops[uint8]{ &opsU8NEON, &opsU8SVE2, } + +var cplxC128NEON = kernel.Complex[complex128]{ + Add: arm64.CaddComplex128NEON, + Sub: arm64.CsubComplex128NEON, + Mul: arm64.CmulComplex128NEON, + Div: arm64.CdivComplex128NEON, + Neg: arm64.CnegComplex128NEON, + Conj: arm64.CconjComplex128NEON, + Sum: arm64.CsumComplex128NEON, + Dot: arm64.CdotComplex128NEON, + DotConj: arm64.CdotconjComplex128NEON, +} + +var cplxC128SVE2 = kernel.Complex[complex128]{ + Add: arm64.CaddComplex128SVE2, + Sub: arm64.CsubComplex128SVE2, + Mul: arm64.CmulComplex128SVE2, + Div: arm64.CdivComplex128SVE2, + Neg: arm64.CnegComplex128SVE2, + Conj: arm64.CconjComplex128SVE2, + Sum: arm64.CsumComplex128SVE2, + Dot: arm64.CdotComplex128SVE2, + DotConj: arm64.CdotconjComplex128SVE2, +} + +var cplxC128ByTier = [...]*kernel.Complex[complex128]{ + nil, + &cplxC128NEON, + &cplxC128SVE2, +} + +var cplxC64NEON = kernel.Complex[complex64]{ + Add: arm64.CaddComplex64NEON, + Sub: arm64.CsubComplex64NEON, + Mul: arm64.CmulComplex64NEON, + Div: arm64.CdivComplex64NEON, + Neg: arm64.CnegComplex64NEON, + Conj: arm64.CconjComplex64NEON, + Sum: arm64.CsumComplex64NEON, + Dot: arm64.CdotComplex64NEON, + DotConj: arm64.CdotconjComplex64NEON, +} + +var cplxC64SVE2 = kernel.Complex[complex64]{ + Add: arm64.CaddComplex64SVE2, + Sub: arm64.CsubComplex64SVE2, + Mul: arm64.CmulComplex64SVE2, + Div: arm64.CdivComplex64SVE2, + Neg: arm64.CnegComplex64SVE2, + Conj: arm64.CconjComplex64SVE2, + Sum: arm64.CsumComplex64SVE2, + Dot: arm64.CdotComplex64SVE2, + DotConj: arm64.CdotconjComplex64SVE2, +} + +var cplxC64ByTier = [...]*kernel.Complex[complex64]{ + nil, + &cplxC64NEON, + &cplxC64SVE2, +} + +var partsC128PartsNEON = kernel.ComplexParts[complex128, float64]{ + Abs: arm64.CabsComplex128NEON, + Real: arm64.CrealComplex128NEON, + Imag: arm64.CimagComplex128NEON, + Scale: arm64.CscaleComplex128NEON, + FromParts: arm64.CfromPartsComplex128NEON, +} + +var partsC128PartsSVE2 = kernel.ComplexParts[complex128, float64]{ + Abs: arm64.CabsComplex128SVE2, + Real: arm64.CrealComplex128SVE2, + Imag: arm64.CimagComplex128SVE2, + Scale: arm64.CscaleComplex128SVE2, + FromParts: arm64.CfromPartsComplex128SVE2, +} + +var partsC128PartsByTier = [...]*kernel.ComplexParts[complex128, float64]{ + nil, + &partsC128PartsNEON, + &partsC128PartsSVE2, +} + +var partsC64PartsNEON = kernel.ComplexParts[complex64, float32]{ + Abs: arm64.CabsComplex64NEON, + Real: arm64.CrealComplex64NEON, + Imag: arm64.CimagComplex64NEON, + Scale: arm64.CscaleComplex64NEON, + FromParts: arm64.CfromPartsComplex64NEON, +} + +var partsC64PartsSVE2 = kernel.ComplexParts[complex64, float32]{ + Abs: arm64.CabsComplex64SVE2, + Real: arm64.CrealComplex64SVE2, + Imag: arm64.CimagComplex64SVE2, + Scale: arm64.CscaleComplex64SVE2, + FromParts: arm64.CfromPartsComplex64SVE2, +} + +var partsC64PartsByTier = [...]*kernel.ComplexParts[complex64, float32]{ + nil, + &partsC64PartsNEON, + &partsC64PartsSVE2, +} diff --git a/dispatch_tables_arm64_test.go b/dispatch_tables_arm64_test.go index c793529..4fb7aaf 100644 --- a/dispatch_tables_arm64_test.go +++ b/dispatch_tables_arm64_test.go @@ -127,5 +127,25 @@ var allFlatTables = map[string][]any{ "Bytes.VarintDecodeU64": {tblBytesVarintDecodeU64[0], tblBytesVarintDecodeU64[1], tblBytesVarintDecodeU64[2]}, } +// allGroupTables enumerates the per-tier partial structs the runtime +// overlays from, keyed by kernel.Set group. Slot 0 is the reference +// and is nil; the rest are one per accelerated tier, in tier order. +var allGroupTables = map[string][]any{ + "F32": {nil, opsF32NEON, opsF32SVE2}, + "F64": {nil, opsF64NEON, opsF64SVE2}, + "I16": {nil, opsI16NEON, opsI16SVE2}, + "I32": {nil, opsI32NEON, opsI32SVE2}, + "I64": {nil, opsI64NEON, opsI64SVE2}, + "I8": {nil, opsI8NEON, opsI8SVE2}, + "U16": {nil, opsU16NEON, opsU16SVE2}, + "U32": {nil, opsU32NEON, opsU32SVE2}, + "U64": {nil, opsU64NEON, opsU64SVE2}, + "U8": {nil, opsU8NEON, opsU8SVE2}, + "C128": {nil, cplxC128NEON, cplxC128SVE2}, + "C64": {nil, cplxC64NEON, cplxC64SVE2}, + "C128Parts": {nil, partsC128PartsNEON, partsC128PartsSVE2}, + "C64Parts": {nil, partsC64PartsNEON, partsC64PartsSVE2}, +} + // archSets hands the tests every tier's complete kernel set. var archSets func() map[string]kernel.Set = arm64.Sets diff --git a/dispatch_tables_fallback.go b/dispatch_tables_fallback.go index 889f311..109c129 100644 --- a/dispatch_tables_fallback.go +++ b/dispatch_tables_fallback.go @@ -138,3 +138,7 @@ var opsU16ByTier = [...]*kernel.Ops[uint16]{nil} var opsU32ByTier = [...]*kernel.Ops[uint32]{nil} var opsU64ByTier = [...]*kernel.Ops[uint64]{nil} var opsU8ByTier = [...]*kernel.Ops[uint8]{nil} +var cplxC128ByTier = [...]*kernel.Complex[complex128]{nil} +var cplxC64ByTier = [...]*kernel.Complex[complex64]{nil} +var partsC128PartsByTier = [...]*kernel.ComplexParts[complex128, float64]{nil} +var partsC64PartsByTier = [...]*kernel.ComplexParts[complex64, float32]{nil} diff --git a/dispatch_tables_fallback_test.go b/dispatch_tables_fallback_test.go index 375ed65..d3e5972 100644 --- a/dispatch_tables_fallback_test.go +++ b/dispatch_tables_fallback_test.go @@ -122,6 +122,10 @@ var allFlatTables = map[string][]any{ "Bytes.VarintDecodeU64": {tblBytesVarintDecodeU64[0]}, } +// No generated backend here: every group table is the reference slot +// alone, so the walk has nothing to check. +var allGroupTables = map[string][]any{} + // No generated backend here: the numeric half of the inventory walk // has nothing to check and skips on nil. var archSets func() map[string]kernel.Set diff --git a/dispatch_tables_loong64.go b/dispatch_tables_loong64.go index 961cdf2..4703e50 100644 --- a/dispatch_tables_loong64.go +++ b/dispatch_tables_loong64.go @@ -1308,3 +1308,55 @@ var opsU8ByTier = [...]*kernel.Ops[uint8]{ nil, &opsU8LASX, } + +var cplxC128LASX = kernel.Complex[complex128]{ + Add: loong64.CaddComplex128LASX, + Sub: loong64.CsubComplex128LASX, + Neg: loong64.CnegComplex128LASX, + Sum: loong64.CsumComplex128LASX, + Dot: loong64.CdotComplex128LASX, + DotConj: loong64.CdotconjComplex128LASX, +} + +var cplxC128ByTier = [...]*kernel.Complex[complex128]{ + nil, + &cplxC128LASX, +} + +var cplxC64LASX = kernel.Complex[complex64]{ + Add: loong64.CaddComplex64LASX, + Sub: loong64.CsubComplex64LASX, + Neg: loong64.CnegComplex64LASX, + Sum: loong64.CsumComplex64LASX, + Dot: loong64.CdotComplex64LASX, + DotConj: loong64.CdotconjComplex64LASX, +} + +var cplxC64ByTier = [...]*kernel.Complex[complex64]{ + nil, + &cplxC64LASX, +} + +var partsC128PartsLASX = kernel.ComplexParts[complex128, float64]{ + Abs: loong64.CabsComplex128LASX, + Real: loong64.CrealComplex128LASX, + Imag: loong64.CimagComplex128LASX, + Scale: loong64.CscaleComplex128LASX, + FromParts: loong64.CfromPartsComplex128LASX, +} + +var partsC128PartsByTier = [...]*kernel.ComplexParts[complex128, float64]{ + nil, + &partsC128PartsLASX, +} + +var partsC64PartsLASX = kernel.ComplexParts[complex64, float32]{ + Real: loong64.CrealComplex64LASX, + Imag: loong64.CimagComplex64LASX, + Scale: loong64.CscaleComplex64LASX, +} + +var partsC64PartsByTier = [...]*kernel.ComplexParts[complex64, float32]{ + nil, + &partsC64PartsLASX, +} diff --git a/dispatch_tables_loong64_test.go b/dispatch_tables_loong64_test.go index acfef7c..a6b31d5 100644 --- a/dispatch_tables_loong64_test.go +++ b/dispatch_tables_loong64_test.go @@ -127,5 +127,25 @@ var allFlatTables = map[string][]any{ "Bytes.VarintDecodeU64": {tblBytesVarintDecodeU64[0], tblBytesVarintDecodeU64[1]}, } +// allGroupTables enumerates the per-tier partial structs the runtime +// overlays from, keyed by kernel.Set group. Slot 0 is the reference +// and is nil; the rest are one per accelerated tier, in tier order. +var allGroupTables = map[string][]any{ + "F32": {nil, opsF32LASX}, + "F64": {nil, opsF64LASX}, + "I16": {nil, opsI16LASX}, + "I32": {nil, opsI32LASX}, + "I64": {nil, opsI64LASX}, + "I8": {nil, opsI8LASX}, + "U16": {nil, opsU16LASX}, + "U32": {nil, opsU32LASX}, + "U64": {nil, opsU64LASX}, + "U8": {nil, opsU8LASX}, + "C128": {nil, cplxC128LASX}, + "C64": {nil, cplxC64LASX}, + "C128Parts": {nil, partsC128PartsLASX}, + "C64Parts": {nil, partsC64PartsLASX}, +} + // archSets hands the tests every tier's complete kernel set. var archSets func() map[string]kernel.Set = loong64.Sets diff --git a/dispatch_tables_ppc64le.go b/dispatch_tables_ppc64le.go index 35129af..0d75800 100644 --- a/dispatch_tables_ppc64le.go +++ b/dispatch_tables_ppc64le.go @@ -1201,3 +1201,59 @@ var opsU8ByTier = [...]*kernel.Ops[uint8]{ nil, &opsU8VSX, } + +var cplxC128VSX = kernel.Complex[complex128]{ + Add: ppc64le.CaddComplex128VSX, + Sub: ppc64le.CsubComplex128VSX, + Mul: ppc64le.CmulComplex128VSX, + Neg: ppc64le.CnegComplex128VSX, + Conj: ppc64le.CconjComplex128VSX, + Sum: ppc64le.CsumComplex128VSX, + Dot: ppc64le.CdotComplex128VSX, + DotConj: ppc64le.CdotconjComplex128VSX, +} + +var cplxC128ByTier = [...]*kernel.Complex[complex128]{ + nil, + &cplxC128VSX, +} + +var cplxC64VSX = kernel.Complex[complex64]{ + Add: ppc64le.CaddComplex64VSX, + Sub: ppc64le.CsubComplex64VSX, + Mul: ppc64le.CmulComplex64VSX, + Div: ppc64le.CdivComplex64VSX, + Neg: ppc64le.CnegComplex64VSX, + Conj: ppc64le.CconjComplex64VSX, +} + +var cplxC64ByTier = [...]*kernel.Complex[complex64]{ + nil, + &cplxC64VSX, +} + +var partsC128PartsVSX = kernel.ComplexParts[complex128, float64]{ + Abs: ppc64le.CabsComplex128VSX, + Real: ppc64le.CrealComplex128VSX, + Imag: ppc64le.CimagComplex128VSX, + Scale: ppc64le.CscaleComplex128VSX, + FromParts: ppc64le.CfromPartsComplex128VSX, +} + +var partsC128PartsByTier = [...]*kernel.ComplexParts[complex128, float64]{ + nil, + &partsC128PartsVSX, +} + +var partsC64PartsVSX = kernel.ComplexParts[complex64, float32]{ + Abs: ppc64le.CabsComplex64VSX, + Real: ppc64le.CrealComplex64VSX, + Imag: ppc64le.CimagComplex64VSX, + Scale: ppc64le.CscaleComplex64VSX, + FromParts: ppc64le.CfromPartsComplex64VSX, +} + +var partsC64PartsByTier = [...]*kernel.ComplexParts[complex64, float32]{ + nil, + &partsC64PartsVSX, +} diff --git a/dispatch_tables_ppc64le_test.go b/dispatch_tables_ppc64le_test.go index 5528db9..d2e53ec 100644 --- a/dispatch_tables_ppc64le_test.go +++ b/dispatch_tables_ppc64le_test.go @@ -127,5 +127,25 @@ var allFlatTables = map[string][]any{ "Bytes.VarintDecodeU64": {tblBytesVarintDecodeU64[0], tblBytesVarintDecodeU64[1]}, } +// allGroupTables enumerates the per-tier partial structs the runtime +// overlays from, keyed by kernel.Set group. Slot 0 is the reference +// and is nil; the rest are one per accelerated tier, in tier order. +var allGroupTables = map[string][]any{ + "F32": {nil, opsF32VSX}, + "F64": {nil, opsF64VSX}, + "I16": {nil, opsI16VSX}, + "I32": {nil, opsI32VSX}, + "I64": {nil, opsI64VSX}, + "I8": {nil, opsI8VSX}, + "U16": {nil, opsU16VSX}, + "U32": {nil, opsU32VSX}, + "U64": {nil, opsU64VSX}, + "U8": {nil, opsU8VSX}, + "C128": {nil, cplxC128VSX}, + "C64": {nil, cplxC64VSX}, + "C128Parts": {nil, partsC128PartsVSX}, + "C64Parts": {nil, partsC64PartsVSX}, +} + // archSets hands the tests every tier's complete kernel set. var archSets func() map[string]kernel.Set = ppc64le.Sets diff --git a/dispatch_tables_riscv64.go b/dispatch_tables_riscv64.go index 24ef710..24be578 100644 --- a/dispatch_tables_riscv64.go +++ b/dispatch_tables_riscv64.go @@ -1420,3 +1420,63 @@ var opsU8ByTier = [...]*kernel.Ops[uint8]{ nil, &opsU8RVV, } + +var cplxC128RVV = kernel.Complex[complex128]{ + Add: riscv64.CaddComplex128RVV, + Sub: riscv64.CsubComplex128RVV, + Mul: riscv64.CmulComplex128RVV, + Div: riscv64.CdivComplex128RVV, + Neg: riscv64.CnegComplex128RVV, + Conj: riscv64.CconjComplex128RVV, + Sum: riscv64.CsumComplex128RVV, + Dot: riscv64.CdotComplex128RVV, + DotConj: riscv64.CdotconjComplex128RVV, +} + +var cplxC128ByTier = [...]*kernel.Complex[complex128]{ + nil, + &cplxC128RVV, +} + +var cplxC64RVV = kernel.Complex[complex64]{ + Add: riscv64.CaddComplex64RVV, + Sub: riscv64.CsubComplex64RVV, + Mul: riscv64.CmulComplex64RVV, + Div: riscv64.CdivComplex64RVV, + Neg: riscv64.CnegComplex64RVV, + Conj: riscv64.CconjComplex64RVV, + Sum: riscv64.CsumComplex64RVV, + Dot: riscv64.CdotComplex64RVV, + DotConj: riscv64.CdotconjComplex64RVV, +} + +var cplxC64ByTier = [...]*kernel.Complex[complex64]{ + nil, + &cplxC64RVV, +} + +var partsC128PartsRVV = kernel.ComplexParts[complex128, float64]{ + Abs: riscv64.CabsComplex128RVV, + Real: riscv64.CrealComplex128RVV, + Imag: riscv64.CimagComplex128RVV, + Scale: riscv64.CscaleComplex128RVV, + FromParts: riscv64.CfromPartsComplex128RVV, +} + +var partsC128PartsByTier = [...]*kernel.ComplexParts[complex128, float64]{ + nil, + &partsC128PartsRVV, +} + +var partsC64PartsRVV = kernel.ComplexParts[complex64, float32]{ + Abs: riscv64.CabsComplex64RVV, + Real: riscv64.CrealComplex64RVV, + Imag: riscv64.CimagComplex64RVV, + Scale: riscv64.CscaleComplex64RVV, + FromParts: riscv64.CfromPartsComplex64RVV, +} + +var partsC64PartsByTier = [...]*kernel.ComplexParts[complex64, float32]{ + nil, + &partsC64PartsRVV, +} diff --git a/dispatch_tables_riscv64_test.go b/dispatch_tables_riscv64_test.go index 1d02dc7..d549421 100644 --- a/dispatch_tables_riscv64_test.go +++ b/dispatch_tables_riscv64_test.go @@ -127,5 +127,25 @@ var allFlatTables = map[string][]any{ "Bytes.VarintDecodeU64": {tblBytesVarintDecodeU64[0], tblBytesVarintDecodeU64[1]}, } +// allGroupTables enumerates the per-tier partial structs the runtime +// overlays from, keyed by kernel.Set group. Slot 0 is the reference +// and is nil; the rest are one per accelerated tier, in tier order. +var allGroupTables = map[string][]any{ + "F32": {nil, opsF32RVV}, + "F64": {nil, opsF64RVV}, + "I16": {nil, opsI16RVV}, + "I32": {nil, opsI32RVV}, + "I64": {nil, opsI64RVV}, + "I8": {nil, opsI8RVV}, + "U16": {nil, opsU16RVV}, + "U32": {nil, opsU32RVV}, + "U64": {nil, opsU64RVV}, + "U8": {nil, opsU8RVV}, + "C128": {nil, cplxC128RVV}, + "C64": {nil, cplxC64RVV}, + "C128Parts": {nil, partsC128PartsRVV}, + "C64Parts": {nil, partsC64PartsRVV}, +} + // archSets hands the tests every tier's complete kernel set. var archSets func() map[string]kernel.Set = riscv64.Sets diff --git a/dispatch_tables_s390x.go b/dispatch_tables_s390x.go index ef3d9f8..a99a780 100644 --- a/dispatch_tables_s390x.go +++ b/dispatch_tables_s390x.go @@ -1022,3 +1022,58 @@ var opsU8ByTier = [...]*kernel.Ops[uint8]{ nil, &opsU8VX, } + +var cplxC128VX = kernel.Complex[complex128]{ + Add: s390x.CaddComplex128VX, + Sub: s390x.CsubComplex128VX, + Mul: s390x.CmulComplex128VX, + Div: s390x.CdivComplex128VX, + Neg: s390x.CnegComplex128VX, + Sum: s390x.CsumComplex128VX, + Dot: s390x.CdotComplex128VX, + DotConj: s390x.CdotconjComplex128VX, +} + +var cplxC128ByTier = [...]*kernel.Complex[complex128]{ + nil, + &cplxC128VX, +} + +var cplxC64VX = kernel.Complex[complex64]{ + Add: s390x.CaddComplex64VX, + Sub: s390x.CsubComplex64VX, + Mul: s390x.CmulComplex64VX, + Div: s390x.CdivComplex64VX, + Neg: s390x.CnegComplex64VX, + Conj: s390x.CconjComplex64VX, + Sum: s390x.CsumComplex64VX, +} + +var cplxC64ByTier = [...]*kernel.Complex[complex64]{ + nil, + &cplxC64VX, +} + +var partsC128PartsVX = kernel.ComplexParts[complex128, float64]{ + Abs: s390x.CabsComplex128VX, + Real: s390x.CrealComplex128VX, + Imag: s390x.CimagComplex128VX, + Scale: s390x.CscaleComplex128VX, +} + +var partsC128PartsByTier = [...]*kernel.ComplexParts[complex128, float64]{ + nil, + &partsC128PartsVX, +} + +var partsC64PartsVX = kernel.ComplexParts[complex64, float32]{ + Abs: s390x.CabsComplex64VX, + Real: s390x.CrealComplex64VX, + Imag: s390x.CimagComplex64VX, + Scale: s390x.CscaleComplex64VX, +} + +var partsC64PartsByTier = [...]*kernel.ComplexParts[complex64, float32]{ + nil, + &partsC64PartsVX, +} diff --git a/dispatch_tables_s390x_test.go b/dispatch_tables_s390x_test.go index b1b03e4..bd7ddda 100644 --- a/dispatch_tables_s390x_test.go +++ b/dispatch_tables_s390x_test.go @@ -127,5 +127,25 @@ var allFlatTables = map[string][]any{ "Bytes.VarintDecodeU64": {tblBytesVarintDecodeU64[0], tblBytesVarintDecodeU64[1]}, } +// allGroupTables enumerates the per-tier partial structs the runtime +// overlays from, keyed by kernel.Set group. Slot 0 is the reference +// and is nil; the rest are one per accelerated tier, in tier order. +var allGroupTables = map[string][]any{ + "F32": {nil, opsF32VX}, + "F64": {nil, opsF64VX}, + "I16": {nil, opsI16VX}, + "I32": {nil, opsI32VX}, + "I64": {nil, opsI64VX}, + "I8": {nil, opsI8VX}, + "U16": {nil, opsU16VX}, + "U32": {nil, opsU32VX}, + "U64": {nil, opsU64VX}, + "U8": {nil, opsU8VX}, + "C128": {nil, cplxC128VX}, + "C64": {nil, cplxC64VX}, + "C128Parts": {nil, partsC128PartsVX}, + "C64Parts": {nil, partsC64PartsVX}, +} + // archSets hands the tests every tier's complete kernel set. var archSets func() map[string]kernel.Set = s390x.Sets diff --git a/docs/architecture.md b/docs/architecture.md index 6735e8e..adcee0d 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -63,7 +63,9 @@ caller reference with a tier's partial `Ops`, built lazily on first use (`opsCache` in `dispatch.go`). The only type switch lives there, so the public wrappers stay one-liners and a caller that never touches a type - never links its kernels. + never links its kernels. The complex groups take the same route through + `groupCache[G]`, which is generic over the struct because `kernel.Complex` + and `kernel.ComplexParts` are not `kernel.Ops`. - The reference itself (`refBase`) is a few hundred kilobytes of ordinary Go and is linked into every consumer regardless; the per-tier assembly is what the tables keep out. diff --git a/docs/lld/kernels-and-dispatch.md b/docs/lld/kernels-and-dispatch.md index ea3b3a3..a328d56 100644 --- a/docs/lld/kernels-and-dispatch.md +++ b/docs/lld/kernels-and-dispatch.md @@ -44,6 +44,15 @@ iterating `simdinfo -tiers`. selects the per-element-type cache, which overlays the reference with the tier's partial `Ops` struct lazily on first use (one `reflect` pass per element type, never on the hot path). +- The **complex groups** use the same shape through `complexOps[C]()` and + `complexParts[C, R]()`, over `groupCache[G]` rather than `opsCache[T]`: + `opsCache` is constrained to `Number`, and neither `kernel.Complex` nor + `kernel.ComplexParts` is a `kernel.Ops`, so the cache is generic over the + struct instead of over the element type and merges with `overlayAny`. The + per-tier tables are `cplxByTier` and `partsByTier`. This + route did not exist until 2026-08-14: before it, every complex operation + ran the portable reference on every machine while the generated complex + assembly sat linked into the test-only aggregator (`docs/wrong.md`). - A **generated guard** sits between the table and the kernel. It checks the per-operation element threshold (`KernelThreshold` is the public view; the tables are generated from the same manifest and a test holds the two diff --git a/docs/research/07-nary-closure.md b/docs/research/07-nary-closure.md new file mode 100644 index 0000000..685d8cb --- /dev/null +++ b/docs/research/07-nary-closure.md @@ -0,0 +1,103 @@ +# A general n-ary combinator taking a Go closure + +**Status: measured, design fixed, not implemented.** This record exists so +the implementation step has a decision behind it rather than a preference. +Nothing in it is shipped; ROADMAP.md's n-ary section is where the open item +lives. + +## The question + +`AddAll` and `MulAll` ship, backed by arity-3 and arity-4 kernels with +folding beyond. The open half of that roadmap item is a general combinator: + +```go +simd.CombineInto(dst, f, a, b, c) // dst[i] = f(a[i], b[i], c[i]) +``` + +One function that takes an arbitrary Go closure and applies it elementwise. +The roadmap already states the constraint — a closure call per element +defeats vectorization — so the only design question worth measuring is how +much it defeats it, and whether the answer leaves room for a useful API. + +## The measurement + +A `//go:noinline` loop calling a closure per element, against the identical +loop with the body written in. `dst[i] = a[i]*2 + b[i]`, float64, on the +Zen 5. Instructions retired, slope between `-benchtime 2000x` and `10000x` +at n=65536, so the per-call setup cancels: + +| loop | instructions/element | +|---|---| +| closure per element | 17.998 | +| body written in | 7.032 | + +**11.0 extra instructions per element, a 2.56x ratio.** Instructions +retired are layout- and load-independent, which matters here: the machine +was not quiet. + +Wall clock on the same runs, minimum of three at `-benchtime 2000x` — a +weaker number, quoted only because it is the one a caller feels: + +| n | closure | written in | ratio | +|---|---|---|---| +| 64 | 76.06 ns | 15.03 ns | 5.06x | +| 1024 | 1027 ns | 210.4 ns | 4.88x | +| 65536 | 67391 ns | 14581 ns | 4.62x | + +The wall-clock ratio is worse than the instruction ratio because the +closure loop also serialises: each call is a dependency the scheduler +cannot hoist, so the extra instructions do not overlap. + +The two ratios bracket the real cost. Against a *kernel* rather than +against a written-in scalar loop the gap is larger still, because the +kernel is the thing the closure makes impossible. + +## What that rules out + +A combinator whose contract is "give me any closure and I will make it +fast" cannot be built. At 11 extra instructions per element it is slower +than the scalar loop the caller would have written, and far slower than +chaining the binary calls that already exist. Shipping it would be an API +whose name promises the opposite of what it measures. + +## The design that survives + +The same shape `FilterInto` already uses in this library: **dispatch the +shapes that have kernels, and be explicit that the rest is a scalar loop.** + +1. **A closed set of named operations dispatches to real kernels.** Not a + closure — a value from an enumerated type, so the compiler and the + reader both know the set is finite: + + ```go + simd.CombineInto(dst, simd.OpAdd, a, b, c) + ``` + + `OpAdd` and `OpMul` reach `Add3`/`Add4`/`Mul3`/`Mul4` and the existing + fold. This is `AddAll`/`MulAll` with the operation as a parameter, and + it is worth having only if a caller genuinely needs to choose at run + time; otherwise `AddAll` is clearer at the call site. + +2. **The closure form is a separate function with the cost in its name and + its doc.** Something like `CombineFuncInto`, documented as a scalar + loop, with the measured 11 instructions per element in the comment so + nobody has to rediscover it. It exists for the case where the caller has + an operation this library does not have and wants one pass over the data + rather than materialising intermediates — which is a real want, and is + about memory traffic, not about vector width. + +3. **No heuristic that silently switches between them.** A combinator that + inspected the closure and sometimes used a kernel would be a guess about + what the closure does. The two functions do different things and say so. + +The measurement gate for step 2: `CombineFuncInto` ships only if it beats +the binary chain it replaces at a size where memory traffic dominates — the +same gate `AddAll` passed. If it does not beat it anywhere, it does not +ship, and the number goes in `docs/wrong.md`. + +## What is not decided here + +The exact names and signatures. Those follow the API conventions in +`docs/api.md` at implementation time, and the enumerated-operation form may +turn out not to be worth its own function at all — in which case only the +closure form ships, or neither does. diff --git a/docs/wrong.md b/docs/wrong.md index f6ffe9c..9a62f4e 100644 --- a/docs/wrong.md +++ b/docs/wrong.md @@ -3043,3 +3043,146 @@ element-scratch allocation, but duplicate-skew recovery can allocate one mask for each equal run it extracts. The active API documentation now states that exception. Historical changelog text remains a record of the implementation at the time of each release. + +## Every complex kernel was generated, tested, and dispatched to by nothing + +**Believed.** The complex surface -- `DotComplex`, `DotComplexConj`, +`ScaleComplex`, `SumComplex`, the arithmetic set, and the ComplexParts half +that produces real slices -- runs on the generated assembly like the rest of +the library. nine `internal//complex_*.s` files exist -- sse2, avx2, avx512, neon, +sve2, rvv, vsx, vx, lasx, across six architectures -- `registerComplex` +installs them, and `TestInventoryCoversEveryGroup` asserts that `C64`, +`C128`, `C64Parts` and `C128Parts` all carry declared kernels. All of that +was true. + +**Actually.** None of it was reachable from a consumer. `complexOps[C]()` +returned `&refBase.C64` / `&refBase.C128` *directly*, with no per-tier +overlay, and `complexParts` did the same: + +```go +case complex128: + return any(&refBase.C128).(*kernel.Complex[C]) // complex.go:33, before +``` + +The dispatch tables the runtime indexes carried no complex entry at all -- +`grep -c Complex dispatch_tables_amd64.go` was **0**, against ten +`opsXXByTier` tables for the real element types -- because +`tools/simdgen/emit/dispatch_gen.go` had a `numericElem` map and no complex +equivalent, so `numericGroupsIn` skipped every complex group and emitted +nothing for them. The generated complex assembly was reachable only from +`internal//sets_gen_.go`, whose own generated header says "Tests +call this; consumers never do". + +The test that should have caught it is `TestDeclaredKernelsAreWired`, not +`TestInventoryCoversEveryGroup` -- the first version of this entry named the +wrong one. `TestDeclaredKernelsAreWired` checks the runtime tables +(`allFlatTables`) for exactly three groups, `Bytes`, `Convert` and `Mask`, +and sends everything else -- numeric AND complex -- through `archSets()`, +the test-only aggregator. So the numeric groups have the same nominal hole; +they were saved by having real tables, not by being checked. Nothing walked +the tables the runtime actually uses for a non-flat group. + +Confirmed by disassembly, which is the only thing that would have shown it. +The symbol survives only where the call is not inlined -- the root package's +own test binary inlines it and prints nothing, so this is the +`internal/benchmarks` binary: + +``` +$ go test -c -o /tmp/bench.test ./internal/benchmarks +$ go tool objdump -s 'simd\.DotComplex\[' /tmp/bench.test + complex.go:31 LEAQ github.com/sebishogun/simd.refBase+13560(SB), AX +``` + +Empirically, from a consumer module calling `simd.DotComplex`: 0 complex +kernel symbols linked before, 24 after. A consumer calling only `simd.Sum` +links 0 in both, so the per-operation dead-code elimination survives. + +**How it surfaced.** Not from this repository. A downstream evaluation +(simdblas Task 4, complex Level 1 against gonum) measured `Zdotu`/`Zdotc` +2.2x-3.1x slower than gonum at n <= 100000, 1.65x-1.86x at n = 1e6. Not +against "a plain Go loop", as the first version of this entry said: gonum's +unit-increment complex dot calls `c128.DotuUnitary`/`DotcUnitary`, which are +hand-written AVX assembly on amd64. Losing to it is not by itself proof of +anything. The tell was that `GOSIMD=scalar` and the detected tier gave +identical timings. + +**Cost, measured.** `GOSIMD=scalar` against `avx512` in one binary -- +one build, so the 8.3% layout floor does not apply; there is no second +layout to differ. Complete grid, minimum of six alternating passes, +reproduced twice. **amd64/avx512 only**: no other architecture was measured, +and the per-tier ratios elsewhere are not these. + +The machine was at load 5-12 for this, against the project rule of load +under 1. The wall-clock column is therefore the weaker evidence; the +instruction column below it is layout- and load-independent and is what the +conclusion rests on. + +| operation | scalar ns/op | avx512 ns/op | ratio | +|---|---|---|---| +| Dot c128 n=1024 | 749.7 | 101.5 | 7.39x | +| Dot c128 n=65536 | 47156 | 15459 | 3.05x | +| Dot c128 n=1048576 | 805434 | 408702 | 1.97x | +| DotConj c128 n=1024 | 890.1 | 102.5 | 8.68x | +| DotConj c128 n=65536 | 55570 | 15488 | 3.59x | +| DotConj c64 n=1024 | 1249 | 97.31 | 12.8x | +| DotConj c64 n=65536 | 83114 | 6614 | 12.6x | +| DotConj c64 n=1048576 | 1276125 | 158120 | 8.07x | +| Sum c128 n=1024 | 470.1 | 43.43 | 10.8x | +| Sum c128 n=65536 | 29002 | 5611 | 5.17x | +| Sum c128 n=1048576 | 480519 | 115280 | 4.17x | +| DotConj c128 n=1048576 | 918445 | 420455 | 2.18x | + +Instructions retired per element, two-point slope so setup cancels, which +is the number that does not move with load: + +| operation | scalar | avx512 | ratio | +|---|---|---|---| +| Sum c128 n=1048576 | 12577259 | 1040671 | 12.1x | +| Dot c128 n=1048576 | 32070536 | 2627180 | 12.2x | +| DotConj c64 n=1048576 | 44563734 | 2030498 | 22.0x | + +(The three `DotConj/c128` cells have no instruction row: `go test -bench` +cannot exclude the deeper `impl=naive` sub-benchmark, so selecting them +also runs it and its instructions scale with `-benchtime` too.) + +**When, exactly.** Not "since the kernels were written" -- the first version +of this entry said that and it is wrong. The kernels landed in `ba09d24` and +dispatched correctly through `active.C64`/`active.C128`. The break came with +`02c258a`, "Per-operation dispatch: the linker keeps only what a program +calls", which replaced `active` with `refBase` plus per-tier tables and +emitted no complex tables: + +``` +$ git show v1.13.0:complex.go | grep 'active\.C64' -> return any(&active.C64)... +$ git show v1.14.0:complex.go | grep 'refBase\.C64' -> return any(&refBase.C64)... +$ git tag --contains 02c258a --sort=creatordate | head -1 +v1.14.0 +``` + +So the affected releases are **v1.14.0 through v1.20.0** -- seven tags. A +consumer pinned to v1.13.0 or earlier was never affected. The table +mechanism itself arrived in that commit, so the missing `complexElem` was +never an omission from an older design; it was a group the new design did +not carry over. + +**The repository's own baseline held the evidence.** `testdata/bench/amd64.txt` +was recorded at `f6c72f9`, before the regression, with +`BenchmarkComplexReduce/Dot/c128/n=1024-8` at 101.2-101.9 ns/op -- a +benchmark that calls the public `simd.DotComplex`. The portable path for +that shape is 749.7 ns/op, 7x against `make bench-check`'s 25% threshold. +Whether the gate was run at `02c258a` is not recoverable from here; the +baseline was never updated, which is why it is still pre-regression and +still a working detector. + +**Fix.** `dispatch_gen.go` gained `complexElem`/`partsElem` and an +`emitTierTable` that writes `cplxByTier` and `partsByTier` +alongside the existing `opsByTier`; `dispatch.go` gained a +`groupCache[G any]` (opsCache is constrained to `Number`, and neither +`kernel.Complex` nor `kernel.ComplexParts` is `kernel.Ops`) with an +`overlayAny`; `complexOps`/`complexParts` now route through it. The +regenerated tables are **additive only** -- 0 deleted lines across all seven +`dispatch_tables_*.go`, 479 added. + +`dispatch_complex_test.go` is the test that would have caught it: it compares +the function pointer `DotComplex` actually reaches against the reference +set's, which is the question a caller asks and the inventory walk does not. diff --git a/tools/simdgen/emit/dispatch_gen.go b/tools/simdgen/emit/dispatch_gen.go index 0782a1d..8148f5e 100644 --- a/tools/simdgen/emit/dispatch_gen.go +++ b/tools/simdgen/emit/dispatch_gen.go @@ -62,6 +62,21 @@ func sigTypes(k spec.Kernel) string { return sig } +// complexElem maps a complex group name to its Go element type, and +// partsElem the split groups that pair a complex type with its real one. +// +// These were absent, and their absence was the whole defect: numericGroupsIn +// skipped every complex group, so no per-tier table was emitted for them and +// complexOps had nothing to overlay from. Nine tiers of generated complex +// assembly across six architectures were linked into the test-only Sets() +// aggregator and reachable from nothing a consumer calls. +var complexElem = map[string]string{"C64": "complex64", "C128": "complex128"} + +var partsElem = map[string][2]string{ + "C64Parts": {"complex64", "float32"}, + "C128Parts": {"complex128", "float64"}, +} + // numericElem maps a numeric group name to its Go element type. var numericElem = map[string]string{ "F32": "float32", "F64": "float64", @@ -142,6 +157,34 @@ func RootDispatch(arch string, tiers []string, present map[string]map[string]boo fmt.Fprintf(&b, "}\n\n") } + // Complex groups, the same shape over kernel.Complex and + // kernel.ComplexParts. Kept separate from the numeric loop because the + // struct type differs and ComplexParts carries two type parameters. + for _, g := range groupsIn(all, func(name string) bool { _, ok := complexElem[name]; return ok }) { + emitTierTable(&b, arch, tiers, present, all, g, + "kernel.Complex["+complexElem[g]+"]", "cplx"+g) + } + for _, g := range groupsIn(all, func(name string) bool { _, ok := partsElem[name]; return ok }) { + e := partsElem[g] + emitTierTable(&b, arch, tiers, present, all, g, + "kernel.ComplexParts["+e[0]+", "+e[1]+"]", "parts"+g) + } + + // Every group in the manifest must have produced a table. A group this + // file does not know about silently falls back to the reference on every + // machine, with every test green -- which is exactly what happened to + // C64/C128/C64Parts/C128Parts for the whole life of the complex kernels. + // The allowlists above are the same shape that let it happen, so the + // check is here rather than in a test: a generator that cannot dispatch + // a group must fail loudly instead of emitting a working-looking file. + if missing := ungrouped(all); len(missing) > 0 { + panic(fmt.Sprintf("simdgen: no dispatch table for kernel group(s) %v.\n"+ + "Add the group to numericElem, complexElem, partsElem or flatGroups in "+ + "dispatch_gen.go. Without an entry the group compiles, passes every "+ + "test, and runs the portable reference on every architecture.", + missing)) + } + out := b.String() if formatted, err := format.Source([]byte(out)); err == nil { return string(formatted) @@ -149,11 +192,78 @@ func RootDispatch(arch string, tiers []string, present map[string]map[string]boo return out } +// ungrouped returns the manifest groups no table is emitted for, sorted. +func ungrouped(all []spec.Kernel) []string { + bad := map[string]bool{} + for _, k := range all { + if flatGroups[k.Group] { + continue + } + if _, ok := numericElem[k.Group]; ok { + continue + } + if _, ok := complexElem[k.Group]; ok { + continue + } + if _, ok := partsElem[k.Group]; ok { + continue + } + bad[k.Group] = true + } + out := make([]string, 0, len(bad)) + for g := range bad { + out = append(out, g) + } + sort.Strings(out) + return out +} + // exportedGuard mirrors guardName: the exported threshold wrapper. func exportedGuard(k spec.Kernel) string { return strings.ToUpper(k.GoName[:1]) + k.GoName[1:] } +// emitTierTable writes one partial struct per tier plus the tier-indexed +// pointer array the dispatcher overlays from, for any group whose kernel.Set +// field is a struct of function fields. Slot 0 is nil: the reference base +// needs no overlay. +func emitTierTable(b *strings.Builder, arch string, tiers []string, + present map[string]map[string]bool, all []spec.Kernel, group, typ, prefix string) { + + for _, t := range tiers { + suffix := strings.ToUpper(t) + fmt.Fprintf(b, "var %s%s = %s{\n", prefix, suffix, typ) + for _, k := range all { + if k.Group != group || !present[t][k.CName] { + continue + } + fmt.Fprintf(b, "\t%s: %s.%s%s,\n", k.Field, arch, exportedGuard(k), suffix) + } + fmt.Fprintf(b, "}\n\n") + } + fmt.Fprintf(b, "var %sByTier = [...]*%s{\n\tnil,\n", prefix, typ) + for _, t := range tiers { + fmt.Fprintf(b, "\t&%s%s,\n", prefix, strings.ToUpper(t)) + } + fmt.Fprintf(b, "}\n\n") +} + +// groupsIn returns the sorted group names in all that keep matches true. +func groupsIn(all []spec.Kernel, keep func(string) bool) []string { + seen := map[string]bool{} + for _, k := range all { + if keep(k.Group) { + seen[k.Group] = true + } + } + out := make([]string, 0, len(seen)) + for g := range seen { + out = append(out, g) + } + sort.Strings(out) + return out +} + func numericGroupsIn(all []spec.Kernel) []string { seen := map[string]bool{} for _, k := range all { @@ -197,6 +307,13 @@ func RootDispatchFallback(arches []string, all []spec.Kernel, prov Provenance) s for _, g := range numericGroupsIn(all) { fmt.Fprintf(&b, "var ops%sByTier = [...]*kernel.Ops[%s]{nil}\n", g, numericElem[g]) } + for _, g := range groupsIn(all, func(n string) bool { _, ok := complexElem[n]; return ok }) { + fmt.Fprintf(&b, "var cplx%sByTier = [...]*kernel.Complex[%s]{nil}\n", g, complexElem[g]) + } + for _, g := range groupsIn(all, func(n string) bool { _, ok := partsElem[n]; return ok }) { + e := partsElem[g] + fmt.Fprintf(&b, "var parts%sByTier = [...]*kernel.ComplexParts[%s, %s]{nil}\n", g, e[0], e[1]) + } out := b.String() if formatted, err := format.Source([]byte(out)); err == nil { @@ -284,6 +401,35 @@ func RootDispatchTest(arch string, tiers []string, all []spec.Kernel, prov Prove fmt.Fprintf(&b, "},\n") } fmt.Fprintf(&b, "}\n\n") + + // The struct groups -- numeric and complex -- reach the runtime through + // per-tier partials, not per-operation tables, so allFlatTables cannot + // name them and the inventory walk fell back to archSets(), the test-only + // aggregator. That is why four whole complex groups could be absent from + // the runtime tables with the suite green. These expose the partials the + // runtime actually indexes, keyed by group, so a test can walk them. + fmt.Fprintf(&b, "// allGroupTables enumerates the per-tier partial structs the runtime\n") + fmt.Fprintf(&b, "// overlays from, keyed by kernel.Set group. Slot 0 is the reference\n") + fmt.Fprintf(&b, "// and is nil; the rest are one per accelerated tier, in tier order.\n") + fmt.Fprintf(&b, "var allGroupTables = map[string][]any{\n") + emitGroupRow := func(group, prefix string) { + fmt.Fprintf(&b, "\t%q: {nil", group) + for _, t := range tiers { + fmt.Fprintf(&b, ", %s%s", prefix, strings.ToUpper(t)) + } + fmt.Fprintf(&b, "},\n") + } + for _, g := range numericGroupsIn(all) { + emitGroupRow(g, "ops"+g) + } + for _, g := range groupsIn(all, func(n string) bool { _, ok := complexElem[n]; return ok }) { + emitGroupRow(g, "cplx"+g) + } + for _, g := range groupsIn(all, func(n string) bool { _, ok := partsElem[n]; return ok }) { + emitGroupRow(g, "parts"+g) + } + fmt.Fprintf(&b, "}\n\n") + fmt.Fprintf(&b, "// archSets hands the tests every tier's complete kernel set.\n") fmt.Fprintf(&b, "var archSets func() map[string]kernel.Set = %s.Sets\n", arch) @@ -315,6 +461,9 @@ func RootDispatchFallbackTest(arches []string, all []spec.Kernel, prov Provenanc fmt.Fprintf(&b, "\t%q: {%s[0]},\n", k.Group+"."+k.Field, TableName(k)) } fmt.Fprintf(&b, "}\n\n") + fmt.Fprintf(&b, "// No generated backend here: every group table is the reference slot\n") + fmt.Fprintf(&b, "// alone, so the walk has nothing to check.\n") + fmt.Fprintf(&b, "var allGroupTables = map[string][]any{}\n\n") fmt.Fprintf(&b, "// No generated backend here: the numeric half of the inventory walk\n") fmt.Fprintf(&b, "// has nothing to check and skips on nil.\n") fmt.Fprintf(&b, "var archSets func() map[string]kernel.Set\n") diff --git a/tools/simdgen/emit/ungrouped_test.go b/tools/simdgen/emit/ungrouped_test.go new file mode 100644 index 0000000..9028572 --- /dev/null +++ b/tools/simdgen/emit/ungrouped_test.go @@ -0,0 +1,26 @@ +package emit + +import ( + "testing" + + "github.com/sebishogun/simd/tools/simdgen/spec" +) + +// A group with no dispatch table falls back to the reference on every +// machine with every test green. The generator must refuse to emit rather +// than produce a working-looking file -- this is the check that would have +// caught the complex groups. +func TestUngroupedGroupIsRefused(t *testing.T) { + if got := ungrouped(nil); len(got) != 0 { + t.Fatalf("an empty manifest reported %v", got) + } + for _, g := range []string{"F32", "C128", "C64Parts", "Bytes"} { + if got := ungrouped([]spec.Kernel{{Group: g}}); len(got) != 0 { + t.Errorf("known group %s reported as ungrouped: %v", g, got) + } + } + got := ungrouped([]spec.Kernel{{Group: "C32"}, {Group: "F32"}}) + if len(got) != 1 || got[0] != "C32" { + t.Fatalf("ungrouped = %v, want [C32]", got) + } +} From ef3359231ddcde5d9dc3e9e125d4aefc881d0ccd Mon Sep 17 00:00:00 2001 From: Sebastian Mocanu Date: Sat, 15 Aug 2026 10:42:27 +0100 Subject: [PATCH 02/13] docs: record that the three-way partition is already built The ROADMAP lists a three-way partition as the fix for Sort's one losing case. extractEqual in sort.go already does that split, on shipped kernels (EqualScalarInto, CountTrue, NotMask, CompressInto), so the item names work that exists. The 34% loss it cites could not be re-measured on this machine: at load 2.2-2.5 the minimum of three 300-iteration runs put simd at 28,531 ns against 23,000 ns for slices.Sort, a 1.24x loss rather than 1.34x, and at load 12.0 the numbers were unusable. No ROADMAP number is changed -- re-measuring needs a quiet machine. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 2 +- docs/wrong.md | 33 +++++++++++++++++++++++++++++++++ 2 files changed, 34 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index abc4a11..5197cd0 100644 --- a/README.md +++ b/README.md @@ -389,7 +389,7 @@ cgo. ## Engineering record -[**docs/wrong.md**](docs/wrong.md) records 80 things that measurement disproved, +[**docs/wrong.md**](docs/wrong.md) records 81 things that measurement disproved, including changes that were deleted rather than shipped. Examples: - Green test lanes had executed no accelerated code for months. diff --git a/docs/wrong.md b/docs/wrong.md index 9a62f4e..5459389 100644 --- a/docs/wrong.md +++ b/docs/wrong.md @@ -3186,3 +3186,36 @@ regenerated tables are **additive only** -- 0 deleted lines across all seven `dispatch_complex_test.go` is the test that would have caught it: it compares the function pointer `DotComplex` actually reaches against the reference set's, which is the question a caller asks and the inventory walk does not. + +## The three-way partition may already be built, and the 34% has not been re-measured + +ROADMAP.md records one losing case for `Sort`: few distinct values at 16384, +by 34% against `slices.Sort`, with "**the fix is a three-way partition**" and a +note that it "needs a second kernel". `docs/plans/2026-08-13-simd-production.md` +Task 1 carries the same plan. + +Reading `sort.go` before writing the kernel: `extractEqual` already does the +three-way split. When the split comes back skewed past `sortSkewLimit`, it +takes the run equal to the pivot out of the recursion entirely — which is what +a below/equal/above partition buys — and it does so with kernels this package +already ships (`EqualScalarInto`, `CountTrue`, `NotMask`, `CompressInto`), so +no second kernel was needed for that part. Its own comment says so. + +That does not close the task, and the reason is that nothing here has been +measured on a quiet machine. Two attempts: + + load 2.2-2.5, min of 3 x 300: simd 28,531 ns slices 23,000 ns 1.24x + load 12.0, min of 5 x 400: simd 117,359 ns slices 79,546 ns noise + +The second set is unusable — the same benchmark moved by 4x between them, and +three review agents were running. The first set is not quiet either. What can +be said: the gap at that shape is real and still there, and 1.24x is not 1.34x. +What cannot be said: whether it is 24%, whether the difference from 34% is the +`extractEqual` work landing, or whether either number survives a quiet machine. + +So no number in ROADMAP.md was changed. Overwriting a measured claim with a +worse-conditioned one is not a correction. What the task needs first is the +measurement under load average below 1, and then a profile — because if the +residual is the `copy(a, scratch[:len(a)])` after every partition rather than +the split itself, a three-way kernel does not address it and would be built for +a cause nobody checked. From cb0c9720dcc975d87daff4d894042e2705560ff3 Mon Sep 17 00:00:00 2001 From: Sebastian Mocanu Date: Sun, 16 Aug 2026 00:23:17 +0100 Subject: [PATCH 03/13] test: gate the reference wiring, which shipped past the whole normal lane MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Probed: `Sqrt: sqrt[T]` deleted from floatOps in internal/ref/ref.go — one entry, the shape of a wiring somebody forgets when adding a kernel. TestDeclaredKernelsAreWired, normal lane PASS (confirmed it ran, with -v) TestDeclaredKernelsAreWired, purego lane PASS whole suite, normal — all 14 packages PASS whole suite, purego FAIL, TestInPlaceMatchesInto The completeness gate that exists for exactly this did not see it, and the only thing that did was a functional test in a lane nobody runs by default. The cause is scope, and the older test's own doc comment says it: it walks backend.Inventory against the dispatch tables and archSets() — the GENERATED side — and deliberately not ref.Set(), because ref leaves every Fast slot nil until a backend registers. That reasoning is correct about Fast slots and was taken to cover the whole reference. The reference is not optional. It is the live fallback on architectures with no backend, in purego builds, and below every kernel's element threshold, so a nil there is a nil call in the small-n path of an operation that works fine on a big slice. TestDeclaredKernelsAreWiredInTheReference is the other half: every non-Fast entry in backend.Inventory must be non-nil in refBase. 853 today. Probed with the same deletion — red in the normal lane now. Also settled, and it argues against a planned change: the plan's item 1 proposes generating the internal/ref ops entry and the kernel.Ops struct field, justified as removing "the two easiest things to forget". Half of that is wrong — a forgotten kernel.Ops FIELD cannot ship, because the generator emits code referencing it and the build fails. Only the ref wiring could, and a gate fixes that without moving the numerical contract out of the kernel.Ops field comments and into a manifest. README's wrong.md count moved 81 -> 82; the count gate caught it. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 2 +- dispatch_inventory_test.go | 64 ++++++++++++++++++++++++++++++++++++++ docs/wrong.md | 45 +++++++++++++++++++++++++++ 3 files changed, 110 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 5197cd0..625fcac 100644 --- a/README.md +++ b/README.md @@ -389,7 +389,7 @@ cgo. ## Engineering record -[**docs/wrong.md**](docs/wrong.md) records 81 things that measurement disproved, +[**docs/wrong.md**](docs/wrong.md) records 82 things that measurement disproved, including changes that were deleted rather than shipped. Examples: - Green test lanes had executed no accelerated code for months. diff --git a/dispatch_inventory_test.go b/dispatch_inventory_test.go index 9f9de1e..4199154 100644 --- a/dispatch_inventory_test.go +++ b/dispatch_inventory_test.go @@ -32,6 +32,7 @@ package simd import ( "reflect" + "strings" "testing" "github.com/sebishogun/simd/internal/backend" @@ -154,3 +155,66 @@ func present(sets map[string]kernel.Set, tier string, d backend.Declared) bool { } return rf.IsNil() || f.Pointer() != rf.Pointer() } + +// Every declared kernel is wired in the REFERENCE too, not only in the +// generated tables. +// +// TestDeclaredKernelsAreWired above walks backend.Inventory against the +// dispatch tables and archSets(), which are the GENERATED side. It never looks +// at ref.Set(), so a forgotten entry in internal/ref's ops table is invisible +// to it. +// +// Measured 2026-08-15: `Sqrt: sqrt[T]` deleted from floatOps in +// internal/ref/ref.go. TestDeclaredKernelsAreWired PASSED in both the normal +// and the purego lane, and the WHOLE normal-lane suite -- all fourteen +// packages -- passed. Only the whole purego suite caught it, through +// TestInPlaceMatchesInto, a functional test rather than the completeness gate +// that exists for exactly this. So a forgotten reference wiring shipped unless +// somebody happened to run -tags purego. +// +// The reference is not optional: it is the live fallback on architectures with +// no backend, in purego builds, and below every kernel's element threshold. A +// nil there is a nil pointer dereference in the small-n path of an operation +// that works fine on a big slice. +// +// Fast slots are excluded because ref.Set() leaves them nil ON PURPOSE -- +// until a backend has registered, "no Fast kernel here" and "the accurate +// kernel" cannot be told apart, and dispatch.go fills them with +// FillFastFallbacks afterwards. That is the same exclusion the doc comment on +// TestDeclaredKernelsAreWired describes. +func TestDeclaredKernelsAreWiredInTheReference(t *testing.T) { + if len(backend.Inventory) == 0 { + t.Fatal("backend.Inventory is empty; run make codegen") + } + checked := 0 + for _, d := range backend.Inventory { + if strings.HasPrefix(d.Field, "Fast") { + continue + } + g := reflect.ValueOf(refBase).FieldByName(d.Group) + if !g.IsValid() { + t.Errorf("%s: kernel.Set has no group %q", d.CName, d.Group) + continue + } + f := g.FieldByName(d.Field) + if !f.IsValid() { + t.Errorf("%s: kernel.Set.%s has no field %q", d.CName, d.Group, d.Field) + continue + } + if f.Kind() != reflect.Func { + continue + } + checked++ + if f.IsNil() { + t.Errorf("%s.%s is nil in the REFERENCE, and the manifest declares %s "+ + "for that group: internal/ref is the live fallback below the "+ + "element threshold, in purego builds, and on every architecture "+ + "with no backend, so this is a nil call rather than a slow one", + d.Group, d.Field, d.CName) + } + } + if checked == 0 { + t.Fatal("no reference slots were checked, so this gate measures nothing") + } + t.Logf("%d declared non-Fast kernels checked against the reference", checked) +} diff --git a/docs/wrong.md b/docs/wrong.md index 5459389..0c61ba5 100644 --- a/docs/wrong.md +++ b/docs/wrong.md @@ -3219,3 +3219,48 @@ measurement under load average below 1, and then a profile — because if the residual is the `copy(a, scratch[:len(a)])` after every partition rather than the split itself, a three-way kernel does not address it and would be built for a cause nobody checked. + +## A forgotten reference wiring shipped past the whole normal-lane suite + +**Probed, 2026-08-15.** `Sqrt: sqrt[T]` deleted from `floatOps` in +`internal/ref/ref.go` — one entry, the shape of a wiring somebody forgets when +adding a kernel. + +| lane | result | +|---|---| +| `TestDeclaredKernelsAreWired`, normal | **PASS** (verified with `-v` that it ran) | +| `TestDeclaredKernelsAreWired`, purego | **PASS** | +| whole suite, normal — all 14 packages | **PASS** | +| whole suite, purego | FAIL, `TestInPlaceMatchesInto` | + +So the completeness gate that exists for exactly this did not see it, and the +only thing that did was a functional test in a lane nobody runs by default. + +The cause is scope. `TestDeclaredKernelsAreWired` walks `backend.Inventory` +against the dispatch tables and `archSets()` — the GENERATED side. `ref.Set()` +is never in the subject, and its doc comment says so plainly: "The subject is +`active` ... and not `ref.Set()`. That is not a shortcut, it is the only +correct choice", because ref leaves every Fast slot nil until a backend +registers. That reasoning is right about Fast slots and was taken to cover the +whole reference. + +The reference is not optional. It is the live fallback on architectures with +no backend, in `purego` builds, and below every kernel's element threshold — +so a nil there is a nil call in the small-n path of an operation that works on +a big slice. + +`TestDeclaredKernelsAreWiredInTheReference` is the other half: every +non-`Fast` entry in `backend.Inventory` must be non-nil in `refBase`. 853 +today. The `Fast` exclusion is the same one the older test's comment describes, +for the same reason. + +**Also settled, and against a plan.** `docs/plans/…-simd-production.md`'s item +1 proposes generating the `internal/ref` ops-table entry and the +`kernel.Ops` struct field from the manifest, and justifies it as removing "the +two easiest things to forget — one of which caused the `-tags purego` panic +that `TestDispatchTableComplete` now guards". Half of that is wrong: a +forgotten `kernel.Ops` FIELD cannot ship, because the generator emits code +referencing it and the build fails; a forgotten `ref` WIRING shipped past +fourteen packages, as above. The gate is the fix; codegen would be +convenience, and it would move the numerical contract — which lives in the +`kernel.Ops` field comments — into a manifest. From 369447adb3d18bd58371a512a18d25e5317583d6 Mon Sep 17 00:00:00 2001 From: Sebastian Mocanu Date: Sun, 16 Aug 2026 00:29:05 +0100 Subject: [PATCH 04/13] test: gate that documents naming a test name one that exists MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A document naming a test claims a guarantee is enforced, and it rots like a link: the test is renamed, the sentence goes on naming the old one, and a reader who greps finds nothing and cannot tell whether the guarantee was dropped or the name moved. Swept the whole tree first — 46 documents, 617 declared tests — and only two citations named nothing: - CHANGELOG.md's `TestReadmeTableHasExamples`, renamed to `TestOperationCatalogHasExamples`. The entry is left as written, with the current name added in parentheses: a changelog entry is accurate about the release it describes, and rewriting it to match today is what docs/wrong.md exists to prevent. - docs/research's `BenchmarkNoopAsm`, which is mmcloughlin's benchmark quoted from a blog post, not this repository's. Both are in categories the gate excludes for those reasons, so nothing in the gated set was wrong — the sweep is the finding. The subject is activeDocs plus docs/verification.md and internal/tests/README.md, the two documents that describe the suite itself and carry most citations. Kept as its own list rather than by widening activeDocs, because CLAUDE.md describes that set as what the LINK gate covers and says the LLDs, plans and records are checked by hand — which they were, on 2026-08-16, and all resolve. Five citations today. The gate fails loudly if either side comes back empty, and was probed by adding a fabricated name to docs/verification.md. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 2 + internal/tests/docs/test_names_test.go | 105 +++++++++++++++++++++++++ 2 files changed, 107 insertions(+) create mode 100644 internal/tests/docs/test_names_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 032e3e4..7aeefdb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -684,6 +684,8 @@ stops paying. The README's function table claimed every entry had a runnable example. It was false for 100 of 109 rows. All 109 have one now, and `TestReadmeTableHasExamples` fails the build if a row appears without one. +(It is `TestOperationCatalogHasExamples` now; the guarantee did not change, +the name did, and this entry is left as it was written.) `TestGuidesNameRealFunctions` does the same for the prose. Writing them found three documentation bugs that only executing catches: the diff --git a/internal/tests/docs/test_names_test.go b/internal/tests/docs/test_names_test.go new file mode 100644 index 0000000..b751aed --- /dev/null +++ b/internal/tests/docs/test_names_test.go @@ -0,0 +1,105 @@ +package docs + +import ( + "os" + "path/filepath" + "regexp" + "strings" + "testing" +) + +// A document naming a test is claiming a guarantee is enforced. +// +// It is the same promise as a link and it rots the same way: a test is +// renamed, the document goes on naming the old one, and a reader who greps +// for it finds nothing and cannot tell whether the guarantee was dropped or +// the name moved. Both are worse than no sentence. +// +// The subject is activeDocs — the set links_test.go checks, for the reason +// stated there. CHANGELOG.md, docs/wrong.md, docs/plans and docs/research are +// historical or external: a changelog entry describing what a release did is +// accurate about that release even after a rename, and docs/research quotes +// benchmark names from other people's blog posts (`BenchmarkNoopAsm` is +// mmcloughlin's, not this repository's). Gating those would force history to +// be rewritten to satisfy a checker, which is the thing docs/wrong.md exists +// to prevent. +func TestActiveDocsNameRealTests(t *testing.T) { + declared := declaredTestFuncs(t) + if len(declared) == 0 { + t.Fatal("no test functions were found in the tree, so this gate measures nothing") + } + + cited := regexp.MustCompile("`((?:Test|Fuzz|Benchmark|Example)[A-Za-z0-9_]+)`") + checked := 0 + for _, doc := range testNameDocs() { + b, err := os.ReadFile(filepath.Join(repoRoot, doc)) + if err != nil { + t.Fatalf("%s: %v", doc, err) + } + for i, line := range strings.Split(string(b), "\n") { + for _, m := range cited.FindAllStringSubmatch(line, -1) { + checked++ + if !declared[m[1]] { + t.Errorf("%s:%d names %s, which no test in this repository "+ + "declares: the guarantee on that line is prose", + doc, i+1, m[1]) + } + } + } + } + if checked == 0 { + t.Fatal("no test names were cited in any active document, so this gate " + + "measures nothing -- if the citations were deliberately removed, " + + "delete this test rather than leaving it green over nothing") + } + t.Logf("%d citations across %d documents, all declared", checked, len(testNameDocs())) +} + +// testNameDocs is activeDocs plus the two documents that describe the SUITE +// itself, which is where most citations naturally are. +// +// Not by widening activeDocs: CLAUDE.md describes that set as what the LINK +// gate covers -- README, CONTRIBUTING, ROADMAP and the main references and +// guides -- and says the LLDs, plans and records are checked by hand. This +// gate has its own subject rather than quietly changing the meaning of that +// sentence. The LLD citations were checked by hand on 2026-08-16 and all +// resolve. +func testNameDocs() []string { + return append(append([]string(nil), activeDocs...), + "docs/verification.md", + "internal/tests/README.md", + ) +} + +// declaredTestFuncs is every Test/Fuzz/Benchmark/Example function in the tree. +func declaredTestFuncs(t *testing.T) map[string]bool { + t.Helper() + decl := regexp.MustCompile(`(?m)^func ((?:Test|Fuzz|Benchmark|Example)[A-Za-z0-9_]*)\(`) + out := map[string]bool{} + err := filepath.WalkDir(repoRoot, func(path string, d os.DirEntry, err error) error { + if err != nil { + return err + } + if d.IsDir() { + if n := d.Name(); n == ".git" || n == "node_modules" { + return filepath.SkipDir + } + return nil + } + if !strings.HasSuffix(path, "_test.go") { + return nil + } + b, err := os.ReadFile(path) + if err != nil { + return err + } + for _, m := range decl.FindAllStringSubmatch(string(b), -1) { + out[m[1]] = true + } + return nil + }) + if err != nil { + t.Fatal(err) + } + return out +} From 51cb46d4b4b9d541743a247a18ba7ac59071c2b1 Mon Sep 17 00:00:00 2001 From: Sebastian Mocanu Date: Sun, 16 Aug 2026 01:04:50 +0100 Subject: [PATCH 05/13] test: make fuzz fuzzed one of its three targets and exited 0 The recipe named two and the comment said "Two fuzz targets". There are three, and only ONE was being fuzzed: FuzzKernelsAgainstReference fuzzed, correctly FuzzDifferential named, and run in the ROOT package while the target lives in internal/tests/arrays FuzzJSONMasksMatchesSeparateCalls never named; only its seed corpus ever ran The middle one is why this is an entry rather than a tidy-up: $ go test -run '^$' -fuzz FuzzDifferential -fuzztime 3s . PASS ok github.com/sebishogun/simd 0.002s EXIT=0 `go test -fuzz X` in a package with no target called X exits 0 and prints PASS. No error, no warning. Half of `make fuzz` had been reporting success for a step that fuzzed nothing, and the only tell was two milliseconds where sixty seconds were asked for. Targets are discovered per package now, so a target can only be fuzzed where `go test -list` found it and the wrong-package shape is unreachable rather than fixed. The recipe fails outright if it runs nothing at all, because a discovery loop over an empty list is the same silent green one level up. Found by sweeping the family for fuzz targets nothing runs. simdlogs already discovered its targets and its workflow says why; simdhttp hand-listed three of four and is fixed in its own tree; simdcsv (3 targets) and simdparquet (13) have no fuzz recipe at all, and simdjson names 11 of 14 -- recorded in docs/wrong.md, not yet fixed. docs/verification.md's table said `make fuzz` runs two named targets; it now says every target in the tree, and names the three there are today. README's wrong.md count 82 -> 83. Co-Authored-By: Claude Opus 5 (1M context) --- Makefile | 49 +++++++++++++++++++++++++++++++++++--------- README.md | 2 +- docs/verification.md | 2 +- docs/wrong.md | 46 +++++++++++++++++++++++++++++++++++++++++ 4 files changed, 87 insertions(+), 12 deletions(-) diff --git a/Makefile b/Makefile index 79868af..0913dea 100644 --- a/Makefile +++ b/Makefile @@ -78,17 +78,46 @@ test-tiers: ## Run the suite once per instruction-set tier this CPU has test-race: ## Full suite under the race detector $(GO) test -race $(PKG) -# Two fuzz targets. FuzzDifferential drives the public API; the one under -# internal/conformance drives every generated kernel of every tier against the -# portable implementation, with the fuzzer choosing the bit patterns rather -# than a table choosing the values. For floating point that distinction is the -# whole point: the inputs that break a kernel are not large or small, they are -# a signalling NaN or a denormal at the exact exponent boundary. +# Every fuzz target in the tree, DISCOVERED rather than listed. +# +# FuzzDifferential drives the public API; the one under internal/conformance +# drives every generated kernel of every tier against the portable +# implementation, with the fuzzer choosing the bit patterns rather than a table +# choosing the values. For floating point that distinction is the whole point: +# the inputs that break a kernel are not large or small, they are a signalling +# NaN or a denormal at the exact exponent boundary. +# +# This comment used to open "Two fuzz targets" and the recipe named two. There +# are three, and TWO of them had never been fuzzed by this target: +# +# FuzzJSONMasksMatchesSeparateCalls never named; only its seed corpus ever +# ran, under `go test` +# FuzzDifferential named, and pointed at the ROOT package +# while the target lives in +# internal/tests/arrays +# +# The second is the dangerous one, because `go test -run '^$$' -fuzz X .` in a +# package with NO target called X exits 0 and prints `PASS ... 0.002s`. It does +# not error, so `make fuzz` had been reporting success for a step that fuzzed +# nothing. A gate that finishes in two milliseconds is not a fast gate. +# +# Discovery makes that unreachable: a target is only fuzzed in the package +# `go test -list` found it in. The counter below fails the target outright if +# it ends up running nothing at all. .PHONY: fuzz -fuzz: ## Fuzz the public API and the tier-vs-reference differential - $(GO) test -run '^$$' -fuzz FuzzDifferential -fuzztime $(or $(FUZZTIME),60s) . - $(GO) test -run '^$$' -fuzz FuzzKernelsAgainstReference \ - -fuzztime $(or $(FUZZTIME),60s) ./internal/conformance/ +fuzz: ## Fuzz every target in the tree + @set -e; n=0; \ + for pkg in $$($(GO) list ./...); do \ + for t in $$($(GO) test -list '^Fuzz' $$pkg 2>/dev/null | grep '^Fuzz' || true); do \ + echo "fuzz $$pkg $$t"; \ + $(GO) test -run '^$$$$' -fuzz "^$$t$$" -fuzztime $(or $(FUZZTIME),60s) $$pkg; \ + n=$$((n+1)); \ + done; \ + done; \ + if [ "$$n" -eq 0 ]; then \ + echo "make fuzz ran no targets" >&2; exit 1; \ + fi; \ + echo "fuzzed $$n target(s)" # Every architecture with a backend, under emulation. # diff --git a/README.md b/README.md index 625fcac..92970b6 100644 --- a/README.md +++ b/README.md @@ -389,7 +389,7 @@ cgo. ## Engineering record -[**docs/wrong.md**](docs/wrong.md) records 82 things that measurement disproved, +[**docs/wrong.md**](docs/wrong.md) records 83 things that measurement disproved, including changes that were deleted rather than shipped. Examples: - Green test lanes had executed no accelerated code for months. diff --git a/docs/verification.md b/docs/verification.md index 141fa7c..e2c1c5f 100644 --- a/docs/verification.md +++ b/docs/verification.md @@ -17,7 +17,7 @@ complete list. Command roles come from the Makefile, the CI workflow | `make test-vec` | `GOEXPERIMENT=simd go test` | the amd64-only vector type compiles and passes; skipped where the toolchain lacks the experiment | | `make test-tiers` | one `go test` per tier the CPU supports (`GOSIMD=`) | catches a kernel correct on AVX-512 and wrong on SSE2 | | `make test-race` | `go test -race` | full suite under the race detector | -| `make fuzz` | `FuzzDifferential` (public API) and `FuzzKernelsAgainstReference` (every kernel of every tier vs the reference), `FUZZTIME` seconds each | adversarial bit patterns: signalling NaNs, denormals, exponent boundaries | +| `make fuzz` | every `Fuzz*` in the tree, discovered per package, `FUZZTIME` seconds each — today `FuzzDifferential` (public API), `FuzzKernelsAgainstReference` (every kernel of every tier vs the reference) and `FuzzJSONMasksMatchesSeparateCalls` | adversarial bit patterns: signalling NaNs, denormals, exponent boundaries | | `make test-cross` | arm64, s390x, ppc64le under docker + qemu, hermetic (`--network=none`, `CGO_ENABLED=0`, `-vet=off`, `-short`), `simdinfo -require-accelerated` first | the correctness half of the non-amd64 backends; three bugs shipped past a green amd64 suite and only appeared here — two memory corruptions (a kernel clobbering the goroutine register, kernels writing into the caller's frame through a save area) and a reference that computed different bits on architectures where Go fuses a multiply into an add | | `make test-riscv64` / `make test-loong64` | qemu-user lanes (recent emulators, explicit `-cpu`), `-require-accelerated` | the two architectures the docker lane cannot reach | | `make test-gates` | riscv64 with **no** vector extension, no acceleration assertion | the fallback must still pass; the SIGILL mis-gate case | diff --git a/docs/wrong.md b/docs/wrong.md index 0c61ba5..2f04474 100644 --- a/docs/wrong.md +++ b/docs/wrong.md @@ -3264,3 +3264,49 @@ referencing it and the build fails; a forgotten `ref` WIRING shipped past fourteen packages, as above. The gate is the fix; codegen would be convenience, and it would move the numerical contract — which lives in the `kernel.Ops` field comments — into a manifest. + +## `make fuzz` fuzzed one of its three targets and exited 0 + +Measured 2026-08-16, by sweeping the family for fuzz targets nothing runs. + +The recipe named two targets and the comment above it opened "Two fuzz +targets". There are three, and only ONE of them was being fuzzed: + +| target | what happened | +|---|---| +| `FuzzKernelsAgainstReference` | fuzzed, correctly | +| `FuzzDifferential` | named, and run in the ROOT package while the target lives in `internal/tests/arrays` | +| `FuzzJSONMasksMatchesSeparateCalls` | never named; only its seed corpus ever ran, under `go test` | + +The middle row is the one worth the entry: + +``` +$ go test -run '^$' -fuzz FuzzDifferential -fuzztime 3s . +PASS +ok github.com/sebishogun/simd 0.002s +EXIT=0 +``` + +**`go test -fuzz X` in a package with no target called X exits 0.** It does not +error, it does not warn, it prints `PASS`. So half of `make fuzz` had been +reporting success for a step that fuzzed nothing, and the only tell was a +duration of two milliseconds where sixty seconds were asked for. + +Targets are discovered per package now — `go test -list '^Fuzz'` asks the +compiler, and a target can only be fuzzed in the package it was found in, so +the wrong-package shape is unreachable rather than merely fixed. The recipe +also fails outright if it ends up running nothing at all, because a discovery +loop over an empty list is the same silent green one level up. + +**The same sweep, across the family.** Ten repositories. `simdlogs` already +discovered its targets and its workflow says why: "A hand-maintained list is +how a new target gets written and never run." `simdhttp` hand-listed three of +four and is fixed (its own entries 15 and 16 — the missing one had been named +in a verification document since v1.2.0 and never written). `simdcsv` (3 +targets) and `simdparquet` (13) have no fuzz recipe at all; `simdjson` names 11 +of 14. Those are recorded here and not yet fixed. + +**The shape.** A gate that cannot fail is this family's most persistent defect, +and every instance so far has been a test or an assertion. This is the same +thing at the build level: a Makefile recipe whose command succeeds while doing +nothing, in the repository the others depend on. From 854edd8ef877feb898bbfb7cb4eea530b4ad52a3 Mon Sep 17 00:00:00 2001 From: Sebastian Mocanu Date: Sun, 16 Aug 2026 02:57:58 +0100 Subject: [PATCH 06/13] docs: measure GOEXPERIMENT=simd at the small-n band before building on it The plan for internal/fastpath asserts that Go's own simd/archsimd beats the portable loop where the dispatch guards fall back, from a table of AddInto timings. Two things have to be true before any of that is worth writing, and both are measurable today -- Go 1.26.5 ships simd/archsimd behind GOEXPERIMENT=simd -- so they were measured first. BIT-IDENTITY, which the plan makes a hard gate: identical at every index over n = 0, 1, 3, 7, 8, 9, 15, 16, 17, 31, 32, 33, with cancellation-prone values (i*0.1 + 1e-8 against i*-0.3 + 1e30), covering every tail length the partial-load helpers handle. THE WIN, instructions retired and cycles, perf stat, three interleaved rounds, minimum -- instructions rather than wall clock because the load average was above 1 and this repo's wall-clock noise floor is 8.3%: n scalar instr archsimd instr cycles 8 227,775,443 183,973,765 -19.2% -13.0% 16 387,771,780 284,004,004 -26.8% -21.1% 32 708,189,014 483,774,777 -31.7% -37.5% Every range disjoint: the slowest archsimd round beats the fastest scalar round at all three sizes. What it does NOT measure is written down beside it, because that is the tempting number to quote: it is a DIRECT call and not the guarded one, so the end-to-end figure will be smaller; it is ONE elementwise operation, the friendliest possible case, and the reductions are where bit-identity is genuinely at risk because kernel.CombineTree's sixteen-accumulator shape would have to be reproduced exactly rather than merely correctly; archsimd is amd64 and arm64 only, so four architectures gain nothing; and the build tag is viral, so every lane runs twice or the tagged path becomes the vacuously-green lane wrong.md entry 41 warned about. docs/research/08-goexperiment-simd-small-n.md. Nothing shipped; the open item stays in ROADMAP.md. Co-Authored-By: Claude Opus 5 (1M context) --- docs/research/08-goexperiment-simd-small-n.md | 94 +++++++++++++++++++ 1 file changed, 94 insertions(+) create mode 100644 docs/research/08-goexperiment-simd-small-n.md diff --git a/docs/research/08-goexperiment-simd-small-n.md b/docs/research/08-goexperiment-simd-small-n.md new file mode 100644 index 0000000..a9e29b0 --- /dev/null +++ b/docs/research/08-goexperiment-simd-small-n.md @@ -0,0 +1,94 @@ +# `GOEXPERIMENT=simd` for the small-n band + +**Status: measured, favourable, not implemented.** This record exists so the +implementation step has a number behind it rather than a table copied from a +plan. Nothing in it is shipped; the open item lives in ROADMAP.md's tiers +section. + +## The question + +Every generated guard falls back to `internal/ref` below its threshold: + +```go +func addF32AVX512Guarded(dst, a, b []float32) { + if len(dst) < threshold { ref.AddInto(dst, a, b); return } + addF32AVX512(dst, a, b) +} +``` + +`ref` is a plain Go loop, and below the threshold that loop is the whole answer. +The proposal is to route the guard's fallback through Go's own `simd/archsimd` +instead — inlinable intrinsics rather than a non-inlinable assembly call, which +is what makes them usable at n = 8 where a call costs more than the work. + +Two things had to be true before any of it is worth writing: the intrinsics have +to be **bit-identical** to the portable loop, and they have to actually **win** +at the sizes where the fallback runs. Both are measurable today — Go 1.26.5 ships +`simd/archsimd` behind `GOEXPERIMENT=simd` — so they were measured before +anything was built. + +## Bit-identity + +Float addition is not reassociated by either path (elementwise, one operation +per lane), so identity is expected rather than hoped for. Checked anyway, over +n = 0, 1, 3, 7, 8, 9, 15, 16, 17, 31, 32, 33 with values chosen to be +cancellation-prone (`i*0.1 + 1e-8` against `i*-0.3 + 1e30`), including every +tail length the partial-load helpers have to handle: + +**Identical at every index, every n.** `LoadFloat32x8SlicePart` / +`StoreSlicePart` handle the tail without reading or writing past the slice, which +is the part a hand-written version gets wrong. + +## The measurement + +`f32 dst[i] = a[i] + b[i]`, the scalar loop against an `archsimd` 8-lane loop +plus a partial-load tail. Instructions retired and cycles, `perf stat -e +instructions:u,cycles:u`, three interleaved rounds in one session, 2,000,000 +iterations each, minimum reported. Instructions rather than wall clock because +the machine's load average was above 1 and this repository's wall-clock noise +floor is 8.3%. + +| n | scalar instr | archsimd instr | | scalar cycles | archsimd cycles | | +|---|---|---|---|---|---|---| +| 8 | 227,775,443 | 183,973,765 | **−19.2%** | 38,225,341 | 33,269,977 | −13.0% | +| 16 | 387,771,780 | 284,004,004 | **−26.8%** | 59,050,260 | 46,597,720 | −21.1% | +| 32 | 708,189,014 | 483,774,777 | **−31.7%** | 103,791,526 | 64,829,258 | −37.5% | + +Every range is disjoint: the slowest archsimd round beats the fastest scalar +round at all three sizes. The gain grows with n across the band, which is what a +vector width of 8 predicts — at n = 8 one vector operation replaces eight scalar +ones but the loop scaffolding is a larger share; by n = 32 it is four vectors +against thirty-two. + +## What this does NOT measure + +Stated because the number above is the tempting one to quote: + +- **It is a direct call, not the guarded one.** The library's fallback is reached + through a dispatch guard, and the guard's own compare and branch are not in + either column. The gain should carry — the guard is identical on both sides — + but the end-to-end figure will be smaller than 19–32%. +- **One operation.** `AddInto` is the friendliest possible case: elementwise, + one instruction per lane, no reassociation question. The reductions are the + interesting ones and they are the ones where bit-identity is genuinely at + risk, because `kernel.CombineTree` uses a fixed sixteen-accumulator shape that + an intrinsics loop would have to reproduce exactly rather than merely + correctly. +- **amd64 only, here.** `archsimd` builds for amd64 and arm64; riscv64, s390x, + loong64 and ppc64le keep the `ref` fallback and gain nothing. Any claim made + from this table has to say so. +- **The build tag is viral.** Behind `//go:build goexperiment.simd`, every lane + has to run twice — tagged and untagged — or the tagged path becomes the + vacuously-green lane `docs/wrong.md` entry 41 warned about. That cost is + ongoing and is not in any of these numbers. + +## The decision this supports + +Build it, elementwise operations first, with bit-identity as a gate rather than +an expectation: an operation whose intrinsics path is not bit-identical to `ref` +does not ship for that operation. Reductions are a separate decision needing +their own measurement. + +The reproducer is thirty lines and lives in this document's history rather than +in the tree, because a benchmark for a package that does not exist yet is a +maintenance cost with no reader. From 3d7340386d4491f13e1bbcd2ad8c64e67404d32d Mon Sep 17 00:00:00 2001 From: Sebastian Mocanu Date: Sun, 16 Aug 2026 05:30:58 +0100 Subject: [PATCH 07/13] docs: the archsimd small-n win depends on inlining, and a package cannot inline docs/research/08 measured GOEXPERIMENT=simd intrinsics at -19.2% (n=8), -26.8% (16) and -31.7% (32) instructions against the portable loop, and the plan put the implementation in an internal/fastpath package that every generated guard would call below its threshold. Built it, gated it on bit-identity, and measured. perf stat -e instructions:u, 2,000,000 iterations, interleaved, disjoint ranges: f32 n=8 124.06M ref 155.85M archsimd +25.6% f32 n=9 136.12M 288.08M +111.6% f32 n=16 220.20M 228.13M +3.6% f32 n=17 232.14M 360.57M +55.3% f32 n=32 412.03M 371.91M -9.7% f64 n=8 124.17M 228.07M +83.7% f64 n=16 220.05M 372.34M +69.2% It loses across the whole band the guards use and wins only at f32 n>=32, above where any guard falls back. The cause is inlining. `-gcflags=-m` says "can inline AddFloat32" for the one-line forward to ref and says nothing for the archsimd loop -- Go does not inline a function containing one. So the portable fallback costs the guard nothing to reach and the intrinsics version is a real call, which at n=8 is most of the work. The research document named that risk and did not measure it: "the end-to-end figure will be smaller than 19-32%". Smaller was the wrong word. Its table also measured only exact multiples of the vector width, which hid the tail entirely: n=9 and n=17 -- one vector plus a one-element tail, the shape the band mostly sees -- cost +111.6% and +55.3%. And float64 loses everywhere for a structural reason the f32-only table could not show: 4 lanes means the call is amortised over half as much arithmetic at the same length. Reverted. The next attempt has to emit the intrinsics INSIDE the generated backend, in the same package as the guard where they can inline into it, not behind a package boundary. The bit-identity half stands and was re-verified: elementwise archsimd is bit-identical to ref at every length 0-40 over IEEE specials -- both zeros, both infinities, NaN, the subnormal and finite extremes -- for add, sub, mul and div, on both lanes, compared on Float32bits/Float64bits rather than ==. docs/wrong.md entry 75; README count 83 -> 84. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 2 +- docs/research/08-goexperiment-simd-small-n.md | 15 ++++- docs/wrong.md | 61 +++++++++++++++++++ 3 files changed, 76 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 92970b6..f767d4b 100644 --- a/README.md +++ b/README.md @@ -389,7 +389,7 @@ cgo. ## Engineering record -[**docs/wrong.md**](docs/wrong.md) records 83 things that measurement disproved, +[**docs/wrong.md**](docs/wrong.md) records 84 things that measurement disproved, including changes that were deleted rather than shipped. Examples: - Green test lanes had executed no accelerated code for months. diff --git a/docs/research/08-goexperiment-simd-small-n.md b/docs/research/08-goexperiment-simd-small-n.md index a9e29b0..b0449ff 100644 --- a/docs/research/08-goexperiment-simd-small-n.md +++ b/docs/research/08-goexperiment-simd-small-n.md @@ -1,6 +1,19 @@ # `GOEXPERIMENT=simd` for the small-n band -**Status: measured, favourable, not implemented.** This record exists so the +**Status: measured favourable in isolation, then measured AGAIN as built and +found NEGATIVE. See docs/wrong.md entry 75 before using anything below.** + +The table in this document is a direct, inlinable loop. Put behind a package +boundary -- which is what a shared fallback for every generated guard has to be +-- the same arithmetic costs **+25.6%** instructions at n=8, **+111.6%** at +n=9, and **+83.7%** for float64 at n=8, because the loop cannot be inlined into +the guard while the one-line `ref` forward can. The sizes below are all exact +multiples of the vector width, which hides the tail cost entirely. + +What survives: the bit-identity claim, which was re-verified at every length +0-40 over IEEE specials for add, sub, mul and div, on both lanes. + +**Original status: measured, favourable, not implemented.** This record exists so the implementation step has a number behind it rather than a table copied from a plan. Nothing in it is shipped; the open item lives in ROADMAP.md's tiers section. diff --git a/docs/wrong.md b/docs/wrong.md index 2f04474..2c863cb 100644 --- a/docs/wrong.md +++ b/docs/wrong.md @@ -3310,3 +3310,64 @@ of 14. Those are recorded here and not yet fixed. and every instance so far has been a test or an assertion. This is the same thing at the build level: a Makefile recipe whose command succeeds while doing nothing, in the repository the others depend on. + +## 75. The archsimd small-n win depends on inlining, and a package cannot inline + +`docs/research/08-goexperiment-simd-small-n.md` measured `GOEXPERIMENT=simd` +intrinsics against the portable loop at −19.2% (n=8), −26.8% (16) and −31.7% +(32) instructions, and concluded: build it, elementwise first. The plan put the +implementation in a new `internal/fastpath` package that every generated guard +would call below its threshold. + +Built, bit-identity-gated, and measured. `perf stat -e instructions:u`, +2,000,000 iterations, interleaved, three rounds at n=8 and two at the rest, +disjoint ranges throughout: + +| | ref | archsimd | | +|---|---|---|---| +| f32 n=8 | 124.06M | 155.85M | **+25.6%** | +| f32 n=9 | 136.12M | 288.08M | **+111.6%** | +| f32 n=16 | 220.20M | 228.13M | **+3.6%** | +| f32 n=17 | 232.14M | 360.57M | **+55.3%** | +| f32 n=32 | 412.03M | 371.91M | −9.7% | +| f64 n=8 | 124.17M | 228.07M | **+83.7%** | +| f64 n=16 | 220.05M | 372.34M | **+69.2%** | + +It loses across the whole band the guards actually use, and wins only at f32 +n≥32 — above where any guard falls back. + +**The cause is inlining, and the research document had already named the risk +without measuring it.** `-gcflags=-m`: + +``` +fastpath_ref.go:16:6: can inline AddFloat32 <- the one-line forward to ref +fastpath_simd.go:38:36: inlining call to archsimd.LoadFloat32x8Slice + (no "can inline AddFloat32") <- the loop cannot +``` + +The portable fallback is a one-line forward, so the guard inlines it and pays +nothing to reach it. The archsimd version is a loop, and Go does not inline a +function containing one — so it is a real call, and at n=8 that call is most of +the work. The record's reproducer was a direct inlined loop and said so: *"It is +a direct call, not the guarded one … the end-to-end figure will be smaller than +19–32%."* Smaller was the wrong word. It is negative. + +**The record's table measured only exact multiples of the vector width, and that +hid the worst of it.** n=9 and n=17 — one full vector plus a one-element tail, +which is the shape the guard band mostly sees — cost +111.6% and +55.3%. A table +of 8/16/32 cannot show a tail cost. + +**float64 loses everywhere in the band**, and for a structural reason: 4 lanes +means n=8 is two vectors and n=16 is four, so the call is amortised over half as +much arithmetic as f32 at the same length. The research document measured f32 +only. + +Reverted. What the next attempt has to do differently is emit the intrinsics +**inside the generated backend**, in the same package as the guard, where they +can inline into it — not behind a package boundary. Until something does that, +the measured answer for the guard band is that `internal/ref` is the faster +fallback. + +The bit-identity gate that came with it did pass, at every length 0–40 over +IEEE specials, on both lanes: elementwise archsimd IS bit-identical to `ref`, +which was the other thing that had to be true. That half of the record stands. From f1d9fa0b90b47be4ba28eed2651a327e522e3cc0 Mon Sep 17 00:00:00 2001 From: Sebastian Mocanu Date: Sun, 16 Aug 2026 09:39:34 +0100 Subject: [PATCH 08/13] docs: the forced-vectorization plan measured, and a gate that cannot see scalar Measured before writing any pragma. 336 (kernel, tier) refusals over 132 kernels, not 346; the permute/scatter family is 48% of them and needs instructions the targets lack; there are ZERO fast_* refusals, so the plan's "114 float/fast_* -- the plausible wins" bucket does not exist. Only ten loops in all of csrc are refused for the reason the pragma addresses, and clang's own remark says that pragma grants REORDERING -- which the numerical contract forbids and which the simd_fast_* prefix already exists to house. Those same ten loops are the accumulation inside convolve, correlate and matmul, and those kernels are refused on zero tiers while shipping scalar: simd_convolve_f32 has zero packed arithmetic and ten scalar float ops on sse2, avx2 and avx512 alike, and its only vector instructions on sse2 are a movups and an xorps. verify.vectorWidth counts any %xmm/%ymm/%zmm operand as vector, and every scalar float instruction on x86-64 uses those registers, so RequireVector structurally cannot see a scalar float kernel on amd64. Scanning every symbol for "does arithmetic, none of it packed": 26 of 913 on sse2, 8 of 913 on avx2. No code change: fixing the gate drops the eight to internal/ref, and whether that is faster is unmeasured -- the machine was not quiet. Co-Authored-By: Claude Opus 5 (1M context) --- docs/wrong.md | 106 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 106 insertions(+) diff --git a/docs/wrong.md b/docs/wrong.md index 2c863cb..848b969 100644 --- a/docs/wrong.md +++ b/docs/wrong.md @@ -3371,3 +3371,109 @@ fallback. The bit-identity gate that came with it did pass, at every length 0–40 over IEEE specials, on both lanes: elementwise archsimd IS bit-identical to `ref`, which was the other thing that had to be true. That half of the record stands. + +## 76. Forced vectorization: the bucket the pragma is for is the bucket the contract forbids + +The plan's item 5 proposed trying `#pragma clang loop vectorize(enable)` per +(kernel, target) and keeping measured winners, over "346 refusals: ~60 +`scatter_*`, ~30 `reverse_*`, 114 float/`fast_*` — the plausible wins". +Measured, before writing any pragma. Every number in that sentence is wrong, +and the investigation ends somewhere else. + +**The inventory.** `make check-emission`, clang 22.1.8: **336** (kernel, tier) +refusals over **132** distinct kernels, not 346. + +| tier | vsx | neon | sse2 | vx | sve2 | lasx | rvv | avx2 | avx512 | +|---|---|---|---|---|---|---|---|---|---| +| refusals | 81 | 80 | 56 | 36 | 33 | 23 | 14 | 10 | 3 | + +By operation, as (kernel, tier) pairs: `scatter` 65, `gather` 31, `reverse` 30, +`tile` 10, `reversebits` 10 — the permute/scatter family is **162 of 336, 48%**, +and each needs an instruction the target does not have. `gather` at 31 pairs is +larger than `reverse` and the plan did not mention it. + +**There is no float bucket.** `fast_*` refusals: **zero**. Float-typed +refusals: 59 of 336, and on the three amd64 tiers only **2** — both +`f8e*_to_f32`, which are integer bit manipulation. The refused +`prod`/`dot`/`sumsq*`/`sumsqdev`/`sumsqdiff` entries are all `i64`/`u64`: +64-bit integer reductions, blocked by a missing multiply (SSE2 has no +`pmullq`), not by anything a pragma reaches. + +**The pragma's own bucket is ten loops.** `-Rpass-analysis=loop-vectorize` over +all of `csrc` at sse2, by reason: + +| reason | loops | +|---|---| +| instruction return type cannot be vectorized | 1929 | +| loop control flow is not understood | 186 | +| value that could not be identified as reduction is used outside the loop | 128 | +| potentially faulting load | 58 | +| store instruction cannot be vectorized | 47 | +| **cannot prove it is safe to reorder floating-point operations** | **10** | + +Ten. And clang's own remark names the remedy: *"allow reordering by specifying +'#pragma clang loop vectorize(enable)' before the loop or by providing the +compiler option '-ffast-math'"*. The pragma there is not a cost-model override, +it is **permission to reassociate** — which changes the answer. The numerical +contract in `internal/kernel/kernel.go` is the reason `-ffp-contract=off` is in +`commonFlags` at all, and the repository already has the correct home for a +kernel that trades it: the `simd_fast_*` prefix. So the one bucket the pragma is +designed for is the one bucket that cannot take it without becoming a different +operation. + +The ten are `csrc/numeric.c:121,127,133,139` (the `CONVOLVE` accumulation, four +instantiations), `csrc/numeric.c:399,404`, and `csrc/gemm.c:142` (`GEMM`, four). + +**And those same kernels ship scalar today.** `simd_convolve_f32`, +`simd_correlate_f64` and their siblings are refused on **zero** tiers — they +pass the emission gate — yet: + +| tier | simd_convolve_f32 | packed arithmetic | scalar arithmetic | +|---|---|---|---| +| sse2 | 119 instrs | **0** | 10 (`mulss`, `addss`) | +| avx2 | 140 instrs | **0** | 10 (`vmulss`, `vaddss`) | +| avx512 | 141 instrs | **0** | 10 | + +Its only vector instructions on sse2 are 2 `movups` and 2 `xorps` — a move and +a register zero. `dispatch_tables_amd64.go` carries +`Convolve: amd64.ConvolveFloat32SSE2` and `ConvolveFloat32AVX2`, so a caller on +amd64 dispatches into an assembly kernel that multiplies one element at a time, +paying the dispatch and threshold machinery for it. + +**The gate cannot see this, structurally.** `verify.vectorWidth` returns >0 for +any amd64 instruction whose operands mention `%xmm`/`%ymm`/`%zmm` — and every +scalar float instruction on x86-64 uses those registers. `vmulss %xmm0, %xmm1, +%xmm2` counts as vectorized. So `RequireVector`, whose comment says it exists +because "dispatching to it would run scalar code under a name promising +otherwise", cannot report a scalar float kernel on amd64 at all. Its 69 amd64 +refusals are integer kernels, necessarily. + +Scanning every `simd_*` symbol for "does float arithmetic, none of it packed": + +| tier | scalar-only kernels | of | +|---|---|---| +| sse2 | 26 | 913 | +| avx2 | **8** | 913 | + +The avx2 eight are the finding, because the widest tier is where an excuse runs +out: `polyeval_f32/f64`, `convolve_f32/f64`, `correlate_f32/f64`, +`movavg_f32/f64`. The sse2 extras are the f64 transcendentals (`log1p`, `cbrt`, +`asin`, `sinh`, `asinh`, `acosh`, `atanh`, `atan2`, `hypot`, plain and `fast_`), +which do vectorize on avx2 — a real SSE2 limit, not a defect. + +**What this leaves.** Not a pragma sweep. Two separate things: + +1. `vectorWidth` on amd64 should not count a scalar-form instruction (`*ss`, + `*sd`) as vector, and `RequireVector` should ask "does this kernel do + arithmetic, and is none of it packed" rather than "is any register wide". + A permute kernel does no arithmetic and must still pass. +2. Fixing the gate DROPS the eight, falling back to `internal/ref`. Whether + that is faster than the scalar assembly is unmeasured: both are scalar, and + the assembly costs a call the guard would otherwise inline. That is the + benchmark this needs, and the machine was not quiet (load average 1.17, two + review agents building) when this was written, so no number is offered. + +The transferable part is the same one as entry 25: a gate that asks a proxy +question answers the proxy question. "Contains an instruction that touches a +vector register" is not "vectorized", and on the one architecture that runs +natively here the two are never the same for float code. From c1d08c404389cdbb4912a097dcf2a35c201b524b Mon Sep 17 00:00:00 2001 From: Sebastian Mocanu Date: Sun, 16 Aug 2026 09:53:05 +0100 Subject: [PATCH 09/13] verify: count arithmetic by form, so a scalar kernel can be seen vectorWidth answers "does any instruction touch a vector register", and on amd64 every scalar float instruction does -- scalar SSE arithmetic lives in %xmm. RequireVector, whose whole purpose is to stop a kernel shipping as an acceleration while running scalar code, therefore cannot see that case on the one architecture that runs natively here. arm64 has the same blind spot in a different spelling: `fmul s0, s1, s2` and `fmul v0.4s, v1.4s, v2.4s` differ in the operand, not the register file. arithKind classifies each instruction as packed arithmetic, scalar arithmetic, or neither. Moves are neither, and so is `xorps %xmm0, %xmm0` -- those four instructions are the entirety of what made simd_convolve_f32 look vectorized at sse2. The classifier runs on amd64 and arm64 only, and ArithKnown says so, so a zero elsewhere reads as "not measured" rather than as a finding. check-emission now prints SCALAR-ONLY per kernel that does arithmetic with none of it in lanes: 38 instances over 13 kernels -- convolve, correlate, movavg, polyeval, minr, rolling_min/max_f64, dtoa_f64 -- across avx512 9, sse2 8, avx2 8, sve2 8, neon 5. Nothing is dropped: falling back to internal/ref instead is a behaviour change that wants a benchmark, and the machine was not quiet. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 2 +- docs/wrong.md | 40 +++++--- tools/simdgen/main.go | 13 +++ tools/simdgen/verify/arith_test.go | 108 +++++++++++++++++++++ tools/simdgen/verify/verify.go | 148 ++++++++++++++++++++++++++++- 5 files changed, 295 insertions(+), 16 deletions(-) create mode 100644 tools/simdgen/verify/arith_test.go diff --git a/README.md b/README.md index f767d4b..3066568 100644 --- a/README.md +++ b/README.md @@ -389,7 +389,7 @@ cgo. ## Engineering record -[**docs/wrong.md**](docs/wrong.md) records 84 things that measurement disproved, +[**docs/wrong.md**](docs/wrong.md) records 85 things that measurement disproved, including changes that were deleted rather than shipped. Examples: - Green test lanes had executed no accelerated code for months. diff --git a/docs/wrong.md b/docs/wrong.md index 848b969..58ca6aa 100644 --- a/docs/wrong.md +++ b/docs/wrong.md @@ -3448,18 +3448,34 @@ because "dispatching to it would run scalar code under a name promising otherwise", cannot report a scalar float kernel on amd64 at all. Its 69 amd64 refusals are integer kernels, necessarily. -Scanning every `simd_*` symbol for "does float arithmetic, none of it packed": - -| tier | scalar-only kernels | of | -|---|---|---| -| sse2 | 26 | 913 | -| avx2 | **8** | 913 | - -The avx2 eight are the finding, because the widest tier is where an excuse runs -out: `polyeval_f32/f64`, `convolve_f32/f64`, `correlate_f32/f64`, -`movavg_f32/f64`. The sse2 extras are the f64 transcendentals (`log1p`, `cbrt`, -`asin`, `sinh`, `asinh`, `acosh`, `atanh`, `atan2`, `hypot`, plain and `fast_`), -which do vectorize on avx2 — a real SSE2 limit, not a defect. +**Counted properly, by the generator.** The first pass here was a shell scan +over every symbol in each object, which included helpers that are not kernels +and used a cruder mnemonic match; it reported 26 at sse2 and 8 at avx2. Those +numbers are superseded. `verify.arithKind` now classifies each instruction as +packed arithmetic, scalar arithmetic or neither — moves and the `xorps +%xmm0,%xmm0` zeroing idiom are neither, which is the point — and `make +check-emission` prints `SCALAR-ONLY` for any kernel that does arithmetic with +none of it in lanes: + +| tier | instances | +|---|---| +| amd64/avx512 | 9 | +| amd64/sse2 | 8 | +| amd64/avx2 | 8 | +| arm64/sve2 | 8 | +| arm64/neon | 5 | + +**38 instances over 13 distinct kernels**: `polyeval`, `convolve`, `correlate`, +`movavg`, `minr` (f32 and f64 each), `rolling_min_f64`, `rolling_max_f64`, +`dtoa_f64`. It is **not an x86 quirk** — arm64 has 13 of the 38, and there the +gate is blind for the same reason in a different spelling: `fmul s0, s1, s2` is +scalar and `fmul v0.4s, v1.4s, v2.4s` is packed, and `vectorWidth` reads the +register file, not the arrangement. + +The classifier runs on amd64 and arm64 only, and `Report.ArithKnown` says so: +on the other four architectures a zero count means "not measured", and +`ScalarOnly()` returns false there rather than reporting a finding it did not +make. **What this leaves.** Not a pragma sweep. Two separate things: diff --git a/tools/simdgen/main.go b/tools/simdgen/main.go index c1a36a2..96575c3 100644 --- a/tools/simdgen/main.go +++ b/tools/simdgen/main.go @@ -285,6 +285,19 @@ func build(cc *compile.Clang, src kernels.Source, tgt target.Target, root, outDi if verbose { fmt.Printf(" %s\n", r.Summary()) } + // A kernel that does arithmetic and none of it in lanes is printed + // whether or not the run is verbose, because it is exactly what + // RequireVector exists to catch and exactly what it cannot see: on + // amd64 every scalar float instruction lives in %xmm, so the "does any + // instruction touch a vector register" test passes for scalar code. + // Eight kernels ship this way today -- convolve, correlate, movavg and + // polyeval in both precisions -- and docs/wrong.md entry 76 has the + // disassembly. Not fatal and not dropped: falling back to internal/ref + // instead is a behaviour change that wants a benchmark first. + if r.ScalarOnly() { + fmt.Printf(" SCALAR-ONLY %s: %d scalar arithmetic instructions, "+ + "no packed ones\n", r.Func, r.ScalarArith) + } for _, p := range r.Problems { problems = append(problems, fmt.Sprintf("%s: %s", r.Func, p)) } diff --git a/tools/simdgen/verify/arith_test.go b/tools/simdgen/verify/arith_test.go new file mode 100644 index 0000000..5e5f99f --- /dev/null +++ b/tools/simdgen/verify/arith_test.go @@ -0,0 +1,108 @@ +package verify + +import ( + "testing" + + "github.com/sebishogun/simd/tools/simdgen/target" +) + +// arithKind separates work from moves, and packed work from scalar work. +// +// The mnemonics below are copied from llvm-objdump output for kernels in this +// repository, not invented: the amd64 scalar cases are what simd_convolve_f32 +// actually contains at sse2 and avx2, and the "neither" cases are the two +// instructions that made it pass as vectorized. +func TestArithKindSeparatesPackedFromScalarAndFromMoves(t *testing.T) { + for _, tc := range []struct { + arch target.Arch + mnemonic string + operands string + want arithForm + }{ + // amd64, scalar: one lane, in an %xmm register, which is why + // vectorWidth counts them as vector. + {target.AMD64, "mulss", "(%rsi), %xmm0", arithScalar}, + {target.AMD64, "addss", "%xmm1, %xmm0", arithScalar}, + {target.AMD64, "vmulss", "%xmm2, %xmm1, %xmm0", arithScalar}, + {target.AMD64, "vaddss", "%xmm2, %xmm1, %xmm0", arithScalar}, + {target.AMD64, "mulsd", "%xmm1, %xmm0", arithScalar}, + {target.AMD64, "vfmadd213sd", "%xmm2, %xmm1, %xmm0", arithScalar}, + {target.AMD64, "divss", "%xmm1, %xmm0", arithScalar}, + {target.AMD64, "maxsd", "%xmm1, %xmm0", arithScalar}, + + // amd64, packed. + {target.AMD64, "mulps", "%xmm1, %xmm0", arithPacked}, + {target.AMD64, "vaddps", "%ymm2, %ymm1, %ymm0", arithPacked}, + {target.AMD64, "vfmadd213ps", "%zmm2, %zmm1, %zmm0", arithPacked}, + {target.AMD64, "vmulpd", "%ymm2, %ymm1, %ymm0", arithPacked}, + {target.AMD64, "paddd", "%xmm1, %xmm0", arithPacked}, + {target.AMD64, "vpmulld", "%ymm2, %ymm1, %ymm0", arithPacked}, + {target.AMD64, "vpmaxsd", "%ymm2, %ymm1, %ymm0", arithPacked}, + {target.AMD64, "vpsubb", "%xmm2, %xmm1, %xmm0", arithPacked}, + + // amd64, NEITHER. These two are the whole finding: they are the only + // vector-register instructions in simd_convolve_f32 at sse2, and + // counting them is what let a scalar kernel ship as accelerated. + {target.AMD64, "movups", "(%rdi), %xmm0", arithNone}, + {target.AMD64, "xorps", "%xmm0, %xmm0", arithNone}, + {target.AMD64, "vmovups", "(%rdi), %ymm0", arithNone}, + {target.AMD64, "vmovss", "(%rsi), %xmm0", arithNone}, + {target.AMD64, "movq", "%rax, %rcx", arithNone}, + {target.AMD64, "vshufps", "$0, %xmm0, %xmm0, %xmm0", arithNone}, + + // arm64: the OPERAND decides, because `fmul` spells both forms. + {target.ARM64, "fmul", "v0.4s, v1.4s, v2.4s", arithPacked}, + {target.ARM64, "fadd", "v0.2d, v1.2d, v2.2d", arithPacked}, + {target.ARM64, "fmul", "s0, s1, s2", arithScalar}, + {target.ARM64, "fadd", "d0, d1, d2", arithScalar}, + {target.ARM64, "fmul", "z0.s, p0/m, z0.s, z1.s", arithPacked}, + {target.ARM64, "ldr", "q0, [x0]", arithNone}, + {target.ARM64, "mov", "v0.16b, v1.16b", arithNone}, + + // An architecture the classifier does not claim: everything is + // unclassified, and ArithKnown is what says so. + {target.PPC64LE, "xvmuldp", "vs0, vs1, vs2", arithNone}, + {target.S390X, "vfmdb", "%v0, %v1, %v2", arithNone}, + } { + got := arithKind(Instr{Mnemonic: tc.mnemonic, Operands: tc.operands}, tc.arch) + if got != tc.want { + t.Errorf("%s %s %q = %v, want %v", tc.arch, tc.mnemonic, tc.operands, got, tc.want) + } + } +} + +// ScalarOnly is false where the classifier does not run, so a zero count is +// never read as a finding. +// +// Without this, adding an architecture to the report and forgetting to add it +// to arithKind would mark every one of its kernels scalar-only -- a check that +// fires everywhere is as useless as one that fires nowhere, and this one would +// have been noise on four of the six architectures. +func TestScalarOnlyIsSilentWhereTheClassifierDoesNotRun(t *testing.T) { + for _, arch := range []target.Arch{target.AMD64, target.ARM64, target.PPC64LE, + target.S390X, target.RISCV64, target.LOONG64} { + known := arithClassified(arch) + r := Report{ArithKnown: known, ScalarArith: 10, PackedArith: 0} + if got := r.ScalarOnly(); got != known { + t.Errorf("%s: ScalarOnly()=%v with ArithKnown=%v; an unclassified "+ + "architecture must not report a finding it did not measure", arch, got, known) + } + } + // And where it does run, the three states are distinct. + for _, tc := range []struct { + name string + packed, scalar int + want bool + }{ + {"vectorized", 12, 0, false}, + {"mixed", 12, 50, false}, + {"scalar only", 0, 10, true}, + {"a permute kernel: no arithmetic at all", 0, 0, false}, + } { + r := Report{ArithKnown: true, PackedArith: tc.packed, ScalarArith: tc.scalar} + if got := r.ScalarOnly(); got != tc.want { + t.Errorf("%s (packed=%d scalar=%d) = %v, want %v", + tc.name, tc.packed, tc.scalar, got, tc.want) + } + } +} diff --git a/tools/simdgen/verify/verify.go b/tools/simdgen/verify/verify.go index 0adf6ed..01c8e0e 100644 --- a/tools/simdgen/verify/verify.go +++ b/tools/simdgen/verify/verify.go @@ -66,6 +66,26 @@ type Report struct { TotalInstrs int // WidestVector is the widest vector register class seen, in bits. WidestVector int + // PackedArith and ScalarArith count ARITHMETIC instructions by form. + // + // VectorInstrs above answers "does any instruction touch a vector + // register", which on amd64 every float instruction does: scalar SSE + // arithmetic lives in %xmm, so `vmulss %xmm0, %xmm1, %xmm2` counts. + // RequireVector therefore cannot see a scalar float kernel on the one + // architecture that runs natively here, and eight do ship that way -- + // convolve, correlate, movavg and polyeval, in both precisions, with zero + // packed arithmetic at sse2, avx2 and avx512 alike. See docs/wrong.md + // entry 76. + // + // These two count the arithmetic itself, so "does work, none of it in + // lanes" is a question that can be asked. A permute kernel does no + // arithmetic at all and has both at zero, which is why the question is + // not "is PackedArith > 0". + // + // ArithKnown is false on architectures whose mnemonics this does not + // classify, so a zero there reads as "not measured" rather than "none". + PackedArith, ScalarArith int + ArithKnown bool // BodyEnd is the byte offset of the function's return instruction, which // is where the copied body must stop. // @@ -163,7 +183,7 @@ func ObjectWithDisasm(objPath string, tgt target.Target, funcs []string, opt Opt } func checkFunc(name string, instrs []Instr, tgt target.Target, opt Options) Report { - r := Report{Func: name, TotalInstrs: len(instrs)} + r := Report{Func: name, TotalInstrs: len(instrs), ArithKnown: arithClassified(tgt.Arch)} for _, in := range instrs { if f, w := featureOf(in, tgt.Arch); f > r.MaxFeature { @@ -176,6 +196,12 @@ func checkFunc(name string, instrs []Instr, tgt target.Target, opt Options) Repo r.WidestVector = bits } } + switch kind := arithKind(in, tgt.Arch); kind { + case arithPacked: + r.PackedArith++ + case arithScalar: + r.ScalarArith++ + } if n := stackAdjust(in, tgt.Arch); n > r.StackBytes { r.StackBytes = n } @@ -715,6 +741,107 @@ func vectorWidth(in Instr, arch target.Arch) int { return 0 } +// ---------- arithmetic form ---------- + +type arithForm int + +const ( + arithNone arithForm = iota + arithPacked + arithScalar +) + +// arithClassified reports whether arithKind understands this architecture. +// +// amd64 and arm64 only, deliberately. Both spell the difference in the +// mnemonic or the operand and both are checkable here: amd64 is the only +// architecture this repository runs natively, and arm64 is the one with real +// hardware in the cross lane. On the rest, a count of zero would mean "this +// function did not look", and reporting that as "no scalar arithmetic" is the +// shape of a check that cannot fail. +func arithClassified(arch target.Arch) bool { + return arch == target.AMD64 || arch == target.ARM64 +} + +var ( + // amd64: the operation, then the form suffix. `ss`/`sd` are scalar single + // and scalar double -- one lane, in an %xmm register, which is why + // vectorWidth sees them as vector. `ps`/`pd` are packed. The integer + // family (`p*`) is packed by construction. + reX86FloatOp = regexp.MustCompile(`^v?(add|sub|mul|div|max|min|sqrt|rsqrt|rcp|cmp|fmadd|fmsub|fnmadd|fnmsub|and|andn|or|xor|round|scalef|getexp|getmant|range|reduce)[0-9a-z]*?(ss|sd|ps|pd)$`) + reX86PackedInt = regexp.MustCompile(`^v?p(add|sub|mul|mad|max|min|avg|abs|sad|and|or|xor|sll|srl|sra|cmp)`) + + // The zeroing idiom. `xorps %xmm0, %xmm0` is how every x86 compiler + // produces a zero register, and it is not work -- simd_convolve_f32 + // contains two of them at sse2 and three at avx2, and those, with the + // moves, are the entirety of what made it look vectorized. The bitwise + // family stays in the arithmetic set otherwise, because a mask-and-blend + // kernel does its work with `andps`/`orps` and counting that is right. + reX86Xor = regexp.MustCompile(`^v?p?xor(ps|pd)?$`) + reX86Reg = regexp.MustCompile(`%[a-z0-9]+`) + + // arm64: the operand tells the form. `v0.4s` is a vector arrangement, + // `s0`/`d0` a single lane. The mnemonic alone cannot decide -- `fmul` + // spells both. + reA64Arith = regexp.MustCompile(`^(f?(add|sub|mul|div|max|min|abs|neg|sqrt|cmp|mla|mls|madd|msub|nmul)|s(add|mul|max|min)|u(add|mul|max|min)|and|orr|eor|bic|shl|ushr|sshr)`) + reA64Vector = regexp.MustCompile(`\bv[0-9]+\.[0-9]*[bhsdq]|\bz[0-9]+\.[bhsdq]`) + reA64Scalar = regexp.MustCompile(`\b[sdhq][0-9]+\b`) +) + +// arithKind classifies one instruction as packed arithmetic, scalar +// arithmetic, or neither. +// +// MOVES AND ZEROING ARE NEITHER, which is the whole point: simd_convolve_f32 +// at sse2 contains two `movups` and two `xorps` and no packed arithmetic at +// all, and counting the moves is what let it pass as vectorized. A register +// zero is not work. +func arithKind(in Instr, arch target.Arch) arithForm { + m := in.Mnemonic + switch arch { + case target.AMD64: + if reX86Xor.MatchString(m) && sameRegisters(in.Operands) { + return arithNone + } + if reX86PackedInt.MatchString(m) { + return arithPacked + } + if g := reX86FloatOp.FindStringSubmatch(m); g != nil { + switch g[2] { + case "ss", "sd": + return arithScalar + default: + return arithPacked + } + } + case target.ARM64: + if !reA64Arith.MatchString(m) { + return arithNone + } + if reA64Vector.MatchString(in.Operands) { + return arithPacked + } + if reA64Scalar.MatchString(in.Operands) { + return arithScalar + } + } + return arithNone +} + +// sameRegisters reports whether every register operand names the same +// register, which for a xor means the instruction produces a zero. +func sameRegisters(operands string) bool { + regs := reX86Reg.FindAllString(operands, -1) + if len(regs) < 2 { + return false + } + for _, r := range regs[1:] { + if r != regs[0] { + return false + } + } + return true +} + // ---------- frame pointer ---------- var ( @@ -998,6 +1125,21 @@ func disassemble(objPath string, tgt target.Target, objdump string) (map[string] // Summary renders a one-line description of a report, for the generator's log. func (r Report) Summary() string { - return fmt.Sprintf("%-28s %3d instrs, %3d vector (%d-bit), feature=%s, stack=%dB", - r.Func, r.TotalInstrs, r.VectorInstrs, r.WidestVector, r.MaxFeature, r.StackBytes) + arith := "" + if r.ArithKnown { + arith = fmt.Sprintf(", arith=%d packed/%d scalar", r.PackedArith, r.ScalarArith) + } + return fmt.Sprintf("%-28s %3d instrs, %3d vector (%d-bit)%s, feature=%s, stack=%dB", + r.Func, r.TotalInstrs, r.VectorInstrs, r.WidestVector, arith, r.MaxFeature, r.StackBytes) +} + +// ScalarOnly reports that this kernel does arithmetic and none of it is in +// lanes -- a kernel that would be dispatched to as an acceleration and run one +// element at a time. +// +// False when the architecture's mnemonics are not classified, because "no +// packed arithmetic counted" and "no packed arithmetic present" are not the +// same statement and only one of them is a finding. +func (r Report) ScalarOnly() bool { + return r.ArithKnown && r.ScalarArith > 0 && r.PackedArith == 0 } From e2ab04950e7720bbf6d49ce74a13c928e0af4d61 Mon Sep 17 00:00:00 2001 From: Sebastian Mocanu Date: Sun, 16 Aug 2026 10:31:34 +0100 Subject: [PATCH 10/13] perf(numeric): block polyeval, convolve, correlate and movavg so they vectorize LLVM vectorizes only the innermost loop, and in all four the innermost loop was the reduction -- a serial float dependency it cannot break without reassociating. So they shipped scalar on every tier: simd_convolve_f32 had zero packed arithmetic at sse2, avx2 and avx512 alike, while passing the emission gate on the strength of a movups and an xorps (entry 76). Blocking over the outputs with a fixed 16-element accumulator makes the innermost loop the independent one. Every output still accumulates in the same order, so this is bit-identical rather than close -- which is the difference between this and the pragma, whose whole effect is to permit the reassociation the numerical contract forbids. Nothing is allocated: 16 doubles is 128 bytes against a 512-byte frame budget. All six symbols now carry packed arithmetic on both amd64 tiers, SCALAR-ONLY drops from 13 kernels to 5, and loong64 gains two: movavg f32/f64 were refused there with "LLVM did not vectorize it for lasx" and now emit, so the kernel total moves 6,931 -> 6,933. Bit-identity is checked by the conformance differential against internal/ref, and the check was probed for being vacuous: eleven of its eighteen length pairs are at or above the guard's threshold of 16, and poisoning the blocked path alone reddens it at dst=16, the first blocked case. Speed is NOT measured -- the machine was not quiet. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 4 +- csrc/numeric.c | 79 +- dispatch_tables_loong64.go | 2 + docs/platforms.md | 4 +- docs/wrong.md | 77 + internal/amd64/numeric_avx2_amd64.s | 1502 +++++++---- internal/amd64/numeric_avx512_amd64.s | 2195 +++++++++++++---- internal/amd64/numeric_sse2_amd64.s | 1023 +++++--- internal/arm64/numeric_neon_arm64.s | 1799 ++++++++++++-- internal/arm64/numeric_sve2_arm64.s | 2023 ++++++++++++--- internal/loong64/numeric_lasx_loong64.go | 6 + internal/loong64/numeric_lasx_loong64.s | 932 +++++-- .../loong64/numeric_register_lasx_loong64.go | 18 + internal/riscv64/numeric_rvv_riscv64.s | 1527 +++++++++++- 14 files changed, 9145 insertions(+), 2046 deletions(-) diff --git a/README.md b/README.md index 3066568..a59504b 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ already has — without cgo.** `simd` applies runtime-selected SIMD kernels to ordinary Go slices. The v1.20.0 -tree contains 493 exported functions and 6,931 generated kernels for SSE2, +tree contains 493 exported functions and 6,933 generated kernels for SSE2, AVX2, AVX-512, NEON, SVE2, RVV, VSX, VX, and LASX. A portable Go path covers every operation and every unsupported target. @@ -23,7 +23,7 @@ kernels are compiled ahead of time and committed as assembly, so this is an ordinary Go dependency with one transitive import (`golang.org/x/sys`, for CPU feature detection). Dispatch is one static table per operation, so the linker keeps only the operations a program actually calls: a binary using three -operation families does not retain all 6,931 kernels. +operation families does not retain all 6,933 kernels. ## Quick start diff --git a/csrc/numeric.c b/csrc/numeric.c index a8784a0..0532c27 100644 --- a/csrc/numeric.c +++ b/csrc/numeric.c @@ -69,15 +69,44 @@ void simd_norm_f64(double *__restrict out, const double *__restrict a, // what vectorizes; the inner loop over coefficients is a fixed dependency // chain per element, exactly like the transcendentals in math.c. // -// The accumulator is per element rather than per coefficient, so the whole -// polynomial is evaluated for a register's worth of x at a time. +// BLOCKED over elements, so the INNERMOST loop is the independent one. +// +// The obvious nesting -- outer over elements, inner over coefficients -- does +// not vectorize, and the reason is structural rather than a cost model: LLVM's +// loop vectorizer only ever vectorizes the innermost loop, and here that is +// the Horner recurrence, a serial dependency it cannot break without +// reassociating. Measured: simd_polyeval_f32 contained ten scalar float +// instructions and no packed arithmetic at sse2, avx2 and avx512 alike, while +// passing the "contains a vector instruction" check on the strength of a +// movups and an xorps. See docs/wrong.md entry 76. +// +// Blocking swaps which loop is innermost without changing any element's +// arithmetic: every output still evaluates the same coefficients in the same +// order, so the result is bit-identical, not merely close. The accumulator is +// a fixed-size array so nothing is allocated and the frame stays far under the +// 512-byte kernel budget -- POLY_BLOCK doubles is 128 bytes. +// +// The tail runs the original nesting: it is at most POLY_BLOCK-1 elements and +// vectorizing it is not worth a second code path to get wrong. +#define POLY_BLOCK 16 #define POLY_EVAL(T) \ isize n = nd < nx ? nd : nx; \ if (nc == 0) { \ for (isize i = 0; i < n; i++) d[i] = 0; \ return; \ } \ - for (isize i = 0; i < n; i++) { \ + isize i = 0; \ + for (; i + POLY_BLOCK <= n; i += POLY_BLOCK) { \ + T acc[POLY_BLOCK]; \ + for (isize b = 0; b < POLY_BLOCK; b++) acc[b] = c[nc - 1]; \ + for (isize k = nc - 2; k >= 0; k--) { \ + T ck = c[k]; \ + for (isize b = 0; b < POLY_BLOCK; b++) \ + acc[b] = acc[b] * x[i + b] + ck; \ + } \ + for (isize b = 0; b < POLY_BLOCK; b++) d[i + b] = acc[b]; \ + } \ + for (; i < n; i++) { \ T xv = x[i]; \ T acc = c[nc - 1]; \ for (isize k = nc - 2; k >= 0; k--) acc = acc * xv + c[k]; \ @@ -104,12 +133,36 @@ void simd_polyeval_f64(double *__restrict d, const double *__restrict x, // // An FFT would be asymptotically better and is a different kernel; this is // the form that wins for the short kernels callers actually convolve with. +// BLOCKED over outputs, for the reason POLY_EVAL is: the tap loop is the +// innermost one and it is a serial float reduction, so LLVM -- which +// vectorizes only innermost loops -- had nothing to work with, and the kernel +// shipped scalar on every tier while passing the "contains a vector +// instruction" check. Blocking makes the innermost loop the independent one +// over outputs. +// +// Each output still accumulates its taps in the same order, so this is +// bit-identical to the direct form rather than close to it. That matters here +// more than elsewhere: reassociating a dot product is exactly what +// `#pragma clang loop vectorize(enable)` would have done, and what the +// numerical contract forbids. +#define CONV_BLOCK 16 #define CONVOLVE(T, IDX) \ isize m = nk; \ if (m == 0 || n < m) return; \ isize outn = n - m + 1; \ if (nd < outn) outn = nd; \ - for (isize i = 0; i < outn; i++) { \ + isize i = 0; \ + for (; i + CONV_BLOCK <= outn; i += CONV_BLOCK) { \ + T acc[CONV_BLOCK]; \ + for (isize b = 0; b < CONV_BLOCK; b++) acc[b] = 0; \ + for (isize j = 0; j < m; j++) { \ + T kv = k[IDX]; \ + for (isize b = 0; b < CONV_BLOCK; b++) \ + acc[b] += s[i + b + j] * kv; \ + } \ + for (isize b = 0; b < CONV_BLOCK; b++) d[i + b] = acc[b]; \ + } \ + for (; i < outn; i++) { \ T acc = 0; \ for (isize j = 0; j < m; j++) acc += s[i + j] * k[IDX]; \ d[i] = acc; \ @@ -384,11 +437,27 @@ void simd_scatter_u64(unsigned long long *__restrict d, const int *__restrict id // rounding error than summing the window afresh. The reference sums each // window, so this does too, and the outer loop over windows is what // vectorizes. +// BLOCKED over outputs, exactly as CONVOLVE is and for the same reason: the +// window loop was innermost and is a serial float sum, so nothing vectorized. +// Each output sums its window in the same order, so the result is +// bit-identical -- which rules out the sliding-window form (subtract the +// element leaving, add the one arriving), which is O(n) instead of O(n*w) and +// gives a DIFFERENT number, because floating-point addition does not cancel. +#define MOVAVG_BLOCK 16 #define MOVING_AVERAGE(T) \ if (w <= 0 || na < w) return; \ isize outn = na - w + 1; \ if (nd < outn) outn = nd; \ - for (isize i = 0; i < outn; i++) { \ + isize i = 0; \ + for (; i + MOVAVG_BLOCK <= outn; i += MOVAVG_BLOCK) { \ + T acc[MOVAVG_BLOCK]; \ + for (isize b = 0; b < MOVAVG_BLOCK; b++) acc[b] = 0; \ + for (isize j = 0; j < w; j++) { \ + for (isize b = 0; b < MOVAVG_BLOCK; b++) acc[b] += a[i + b + j]; \ + } \ + for (isize b = 0; b < MOVAVG_BLOCK; b++) d[i + b] = acc[b] / (T)w; \ + } \ + for (; i < outn; i++) { \ T acc = 0; \ for (isize j = 0; j < w; j++) acc += a[i + j]; \ d[i] = acc / (T)w; \ diff --git a/dispatch_tables_loong64.go b/dispatch_tables_loong64.go index 4703e50..e9f795e 100644 --- a/dispatch_tables_loong64.go +++ b/dispatch_tables_loong64.go @@ -650,6 +650,7 @@ var opsF32LASX = kernel.Ops[float32]{ Correlate: loong64.CorrelateFloat32LASX, Tile: loong64.TileFloat32LASX, Gather: loong64.GatherFloat32LASX, + MovingAverage: loong64.MovingAverageFloat32LASX, ShiftDiv: loong64.ShiftDivFloat32LASX, LayerNorm: loong64.LayerNormFloat32LASX, Random: loong64.RandomF32LASX, @@ -746,6 +747,7 @@ var opsF64LASX = kernel.Ops[float64]{ Correlate: loong64.CorrelateFloat64LASX, Tile: loong64.TileFloat64LASX, Gather: loong64.GatherFloat64LASX, + MovingAverage: loong64.MovingAverageFloat64LASX, ShiftDiv: loong64.ShiftDivFloat64LASX, LayerNorm: loong64.LayerNormFloat64LASX, Random: loong64.RandomF64LASX, diff --git a/docs/platforms.md b/docs/platforms.md index 230c239..053667a 100644 --- a/docs/platforms.md +++ b/docs/platforms.md @@ -48,7 +48,7 @@ same portable implementations used for per-operation fallback. ## Kernel coverage The default build exposes 493 exported functions and the repository contains -6,931 generated kernels across nine accelerated tier targets. Counts below are +6,933 generated kernels across nine accelerated tier targets. Counts below are logical callable kernels, not the extra `Body` trampolines required by the s390x ABI. `make check-emission` regenerates the inventory as a dry run. @@ -61,7 +61,7 @@ emitted for AVX2 and declined for SSE2 appears once in each column. | amd64 (sse2/avx2/avx512) | 2609 | 101 | LLVM declined to vectorize for a tier | | arm64 (neon/sve2) | 1681 | 127 | LLVM declined to vectorize for a tier | | riscv64 (rvv) | 891 | 15 | LLVM declined to vectorize | -| loong64 (lasx) | 728 | 174 | `$fp` ownership and unresolved `.L0` references | +| loong64 (lasx) | 730 | 172 | `$fp` ownership and unresolved `.L0` references | | ppc64le (vsx) | 602 | 300 | `r30`/`r2`, `r0`, frame writes, then LLVM refusals | | s390x (vx) | 420 | 482 | `r13`, which the Go runtime owns | diff --git a/docs/wrong.md b/docs/wrong.md index 58ca6aa..1e19086 100644 --- a/docs/wrong.md +++ b/docs/wrong.md @@ -3493,3 +3493,80 @@ The transferable part is the same one as entry 25: a gate that asks a proxy question answers the proxy question. "Contains an instruction that touches a vector register" is not "vectorized", and on the one architecture that runs natively here the two are never the same for float code. + +## 77. Blocking the outer loop is what vectorizes a reduction kernel, and it is bit-identical + +Entry 76 found `polyeval`, `convolve`, `correlate` and `movavg` shipping scalar +on every tier while passing the emission gate, and left the question of what to +do about it open. The answer is not a pragma and not a fallback: it is the loop +nest. + +**LLVM's loop vectorizer only ever vectorizes the INNERMOST loop.** Every one of +these kernels is an outer loop over independent outputs wrapped around an inner +reduction with a runtime trip count: + +```c +for (isize i = 0; i < n; i++) { // independent — but not innermost + T acc = c[nc - 1]; + for (isize k = nc - 2; k >= 0; k--) // innermost — a serial dependency + acc = acc * xv + c[k]; + d[i] = acc; +} +``` + +So the vectorizer looked at the Horner recurrence, could not break it without +reassociating, and gave up — and `polyeval`'s own comment claimed the opposite: +"The outer loop over elements is independent, which is what vectorizes." + +**Blocking swaps which loop is innermost without touching any element's +arithmetic.** A fixed-size accumulator array, elements in tiles of 16: + +```c +for (; i + 16 <= n; i += 16) { + T acc[16]; + for (b) acc[b] = c[nc - 1]; + for (k) { T ck = c[k]; for (b) acc[b] = acc[b] * x[i + b] + ck; } + for (b) d[i + b] = acc[b]; +} +``` + +Every output still evaluates the same coefficients in the same order, so this is +**bit-identical**, not close. That distinction is the whole point: reassociating +is exactly what `#pragma clang loop vectorize(enable)` would have done and what +the numerical contract forbids. Nothing is allocated — 16 doubles is 128 bytes +against the 512-byte kernel frame budget — and the tail runs the original nest. + +| | packed arithmetic before | after (sse2) | after (avx2) | +|---|---|---|---| +| `simd_polyeval_f32` | 0 | 24 | 12 | +| `simd_polyeval_f64` | 0 | 16 | 24 | +| `simd_convolve_f32` | 0 | 8 | 12 | +| `simd_correlate_f64` | 0 | 16 | 8 | +| `simd_movavg_f32` | 0 | 16 | 12 | +| `simd_movavg_f64` | 0 | 16 | 16 | + +The ten scalar instructions that remain in each are the tail loop. + +`SCALAR-ONLY` went from **13 kernels to 5** (`dtoa_f64`, `minr_f32/f64`, +`rolling_min_f64`, `rolling_max_f64`). And loong64 gained two kernels, 728 → +730: `simd_movavg_f32` and `simd_movavg_f64` were refused there with "LLVM did +not vectorize it for lasx" and now emit, so the total moved 6,931 → 6,933. + +**The bit-identity claim is tested, and the test was checked for being +vacuous.** `internal/conformance` compares each tier against `ref` at eighteen +length pairs, eleven of which are at or above the guard's threshold of 16 — so +the SIMD path is genuinely reached. Poisoning the blocked path only +(`acc[b] = acc[b]*x[i+b] + ck + 1`) reddens it at `dst=16 x=16 coeffs=2 i=0`, +the first blocked case, which is what says the comparison sees the new code +rather than the fallback. + +**Not measured: whether it is faster.** It executes fewer instructions per +element and the arithmetic is in lanes, but the machine was not quiet (load +average 2.27, two review agents building) and no wall-clock or counter number +is offered. What is established is that the kernels now do their arithmetic in +lanes and produce the same bits, which is the precondition for the speed +question rather than an answer to it. + +`movavg` also rules out the obvious alternative: a sliding window (subtract the +element leaving, add the one arriving) is O(n) instead of O(n·w) and gives a +DIFFERENT number, because floating-point addition does not cancel. diff --git a/internal/amd64/numeric_avx2_amd64.s b/internal/amd64/numeric_avx2_amd64.s index 3433906..dbf1ffc 100644 --- a/internal/amd64/numeric_avx2_amd64.s +++ b/internal/amd64/numeric_avx2_amd64.s @@ -195,24 +195,102 @@ TEXT ·polyEvalFloat32AVX2(SB), NOSPLIT|NOFRAME, $0-72 MOVQ x_len+32(FP), R8 MOVQ coeffs_len+56(FP), R9 QUAD $0x4dc14c0f4cc1394c - QUAD $0x000000f7840fc985 - QUAD $0x0002338e0fc0854d - QUAD $0xfc8a44107aa1c400 - QUAD $0x0f02e88348c8894c - QUAD $0x894855000000f08c - QUAD $0x8d41f8e4834853e5 - QUAD $0xe1834107e183ff49 - QUAD $0x2e661cebd2314507 - QUAD $0x0000000000841f0f - QUAD $0xff499714117aa1c4 - QUAD $0x00c8840fc2394dc2 - QUAD $0x960c107aa1c40000 - QUAD $0x41d028f8c5c38949 - QUAD $0xcb89481d7401f983 - QUAD $0x90d028f8c5c38949 + QUAD $0xe58948551a74c985 + QUAD $0xe483485356415741 + QUAD $0x31217d10f88349f8 + QUAD $0x854d00000269e9c0 + QUAD $0x49000004938e0fc0 + QUAD $0x000159830f04f883 + QUAD $0x00000473e9c03100 + QUAD $0x49fc8a44107aa1c4 + QUAD $0x00011a8e0f01f983 + QUAD $0x518d45fe498d4900 + QUAD $0x8341c0187de2c4ff + QUAD $0xe38341cb894503e2 + QUAD $0x3100000010be4103 + QUAD $0x2e6666666628ebdb + QUAD $0x0000000000841f0f + QUAD $0x11fcc59f2411fcc5 + QUAD $0x4810708d4c209f5c + QUAD $0xf48f0fc6394dc389 + QUAD $0xfcc5f0894c000001 + QUAD $0x9e5410fcc59e0c10 + QUAD $0xe028fcc5ce894920 + QUAD $0x01fb8341d828fcc5 + QUAD $0xce8949d7894d3474 + QUAD $0xd828fcc5e028fcc5 + QUAD $0x0000000000841f0f + QUAD $0x187da2c4e159dcc5 + QUAD $0xe4c5e458d4c5b22c + QUAD $0xff49db58d4c5da59 + QUAD $0x8348e275cfff49ce + QUAD $0x04c683498c7203f9 + QUAD $0x0000000000841f0f + QUAD $0x187da2c4e159dcc5 + QUAD $0xc5e458d4c5f0b26c + QUAD $0xc5db58d4c5da59e4 + QUAD $0x6c187da2c4e159dc + QUAD $0xe4c5e458d4c5ecb2 + QUAD $0xdcc5db58d4c5da59 + QUAD $0xb26c187da2c4e159 + QUAD $0x59e4c5e458d4c5e8 + QUAD $0x59dcc5db58d4c5da + QUAD $0xe4b26c187da2c4e1 + QUAD $0xda59e4c5e458d4c5 + QUAD $0xfcc68349db58d4c5 + QUAD $0x15e99a7f03fe8349 + QUAD $0x49f0408d49ffffff + QUAD $0xff4904eac149c289 + QUAD $0x4807e183d18944c2 + QUAD $0x10ba41287370f883 + QUAD $0x00c8e9c031000000 + QUAD $0xffffffe0b9480000 + QUAD $0x20f883497fffffff + QUAD $0xc031000001ec830f + QUAD $0xe2834900000226e9 + QUAD $0xc031c8187de2c4f8 + QUAD $0x0000000000841f0f + QUAD $0x11fcc5870c11fcc5 + QUAD $0x874c11fcc520874c + QUAD $0xc560874c11fcc540 + QUAD $0x00000080878c11fc + QUAD $0x0000a0878c11fcc5 + QUAD $0x00c0878c11fcc500 + QUAD $0xe0878c11fcc50000 + QUAD $0x878c11fcc5000000 + QUAD $0x8c11fcc500000100 + QUAD $0x11fcc50000012087 + QUAD $0xfcc500000140878c + QUAD $0xc500000160878c11 + QUAD $0x00000180878c11fc + QUAD $0x0001a0878c11fcc5 + QUAD $0x01c0878c11fcc500 + QUAD $0xe0878c11fcc50000 + QUAD $0x4980e88348000001 + QUAD $0xffff6f850ff8c283 + QUAD $0x8d4c3974c98548ff + QUAD $0xc1c0187de2c41050 + QUAD $0x2e666666666604e1 + QUAD $0x0000000000841f0f + QUAD $0x11fcc5870411fcc5 + QUAD $0x8349d0894c208744 + QUAD $0xe875f0c1834810c2 + QUAD $0x4cd0894cf0c28349 + QUAD $0x028e0fc12948c189 + QUAD $0x44107aa1c4000002 + QUAD $0xea8349ca894dfc8a + QUAD $0x41000000c88c0f02 + QUAD $0x834107e183ff498d + QUAD $0x00401f0f15eb07e1 + QUAD $0xc0ff48871411fac5 + QUAD $0x0001cc840fc0394c + QUAD $0x894d860c10fac500 + QUAD $0xf98341d028f8c5d3 + QUAD $0x894dcb89481f7401 + QUAD $0x001f0fd028f8c5d3 QUAD $0x586aa1c4d259f2c5 QUAD $0xcbff48cbff499a14 - QUAD $0xb87207f88348ee75 + QUAD $0xb87207fa8349ee75 QUAD $0x00401f0f08c38349 QUAD $0x586aa1c4d259f2c5 QUAD $0xc4d259f2c5e09a54 @@ -226,48 +304,51 @@ TEXT ·polyEvalFloat32AVX2(SB), NOSPLIT|NOFRAME, $0-72 QUAD $0x59f2c5c89a54586a QUAD $0xc49a54586aa1c4d2 QUAD $0x07fb8349f8c38349 - QUAD $0x4dffffff49e99e7f - QUAD $0x0000013c8e0fc085 - QUAD $0xc0311b7304f88349 - QUAD $0xf8834900000120e9 - QUAD $0x0109e9c031227304 - QUAD $0x5d5bf8658d480000 - QUAD $0xffffffffe0b948c3 - QUAD $0x7320f883497fffff - QUAD $0xe0b9485eebc0311b - QUAD $0x497fffffffffffff - QUAD $0xe9c0317b7320f883 - QUAD $0x48c0894c000000af - QUAD $0xc057f8c5d231c821 - QUAD $0x0000000000841f0f + QUAD $0x48ffffff49e99e7f + QUAD $0xc18948087304f983 + QUAD $0xf983480000010be9 + QUAD $0x00bce9d231787320 + QUAD $0xc82148c0894c0000 + QUAD $0x9066c057f8c5d231 QUAD $0x11fcc5970411fcc5 QUAD $0x974411fcc5209744 QUAD $0x4860974411fcc540 QUAD $0xe075d0394820c283 - QUAD $0x0000b3840fc03949 - QUAD $0x9a840f1cc0f64100 + QUAD $0x0000f9840fc03949 + QUAD $0xe0840f1cc0f64100 QUAD $0x8348c28948000000 QUAD $0xc0214cc889481cc1 QUAD $0x2e666666c057f8c5 QUAD $0x0000000000841f0f QUAD $0xc28348970411f8c5 - QUAD $0x77ebf275d0394804 - QUAD $0xe2c4c82148c0894c - QUAD $0x001f0fd231c8187d - QUAD $0x11fcc5970c11fcc5 - QUAD $0x974c11fcc520974c - QUAD $0x4860974c11fcc540 - QUAD $0xe075d0394820c283 - QUAD $0xc0f6414774c03949 - QUAD $0x8348c2894828741c - QUAD $0xc0214cc889481cc1 - QUAD $0x001f0fc81879e2c4 + QUAD $0xbae9f275d0394804 + QUAD $0x8348ca8948000000 + QUAD $0x48c8187de2c4e0e2 + QUAD $0x4560c6834887348d + QUAD $0x0000441f0f66c931 + QUAD $0xc4a08e4c117ca1c4 + QUAD $0xa1c4c08e4c117ca1 + QUAD $0x7ca1c4e08e4c117c + QUAD $0x4c20c183498e0c11 + QUAD $0x74d13948dc75ca39 + QUAD $0x89445d741cc1f654 + QUAD $0x48f1294803e683c6 + QUAD $0x48c81879e2c4c101 + QUAD $0xe08348c0894cc201 + QUAD $0x00000000801f0ffc QUAD $0xc28348970c11f8c5 - QUAD $0x3949f275d0394804 - QUAD $0x870411fac51974c0 - QUAD $0x8704c7f1ebc0ff48 - QUAD $0x49c0ff4800000000 - QUAD $0xc377f8c5f175c039 + QUAD $0x8548f275d0394804 + QUAD $0x2e666666661a74f6 + QUAD $0x0000000000841f0f + QUAD $0xc1ff488f0411fac5 + QUAD $0x658d48f375c83949 + QUAD $0xc55d5f415e415be8 + QUAD $0x8948d00148c377f8 + QUAD $0x00008704c7ddebc1 + QUAD $0xc03949c0ff480000 + LONG $0xf8c5f175 + BYTE $0x77 + BYTE $0xc3 RET // func convolveFloat32AVX2(dst []float32, sig []float32, ker []float32) @@ -280,97 +361,193 @@ TEXT ·convolveFloat32AVX2(SB), NOSPLIT|NOFRAME, $0-72 MOVQ ker_len+56(FP), R9 QUAD $0x56415741e5894855 QUAD $0xe483485354415541 - QUAD $0x74894838ec8348f8 - QUAD $0xc0940fc9854d0824 - QUAD $0x40c69c0f40c8294d - QUAD $0x0000029f850fc608 - QUAD $0x48c1394c01708d49 - QUAD $0x8e0ff68548f14e0f - QUAD $0x0fc9854d0000028b - QUAD $0x0c8d4a000001f18e - QUAD $0x07e08341c889458a - QUAD $0xffffffffffe0ba49 - QUAD $0x214d18ca83497fff - QUAD $0x8d4c0824448b48ca - QUAD $0x83488a048d4a1c58 - QUAD $0x311024448948fcc0 - QUAD $0xfffffffffdbf49db - QUAD $0xfffffcbc493fffff - QUAD $0xfbbd493fffffffff - QUAD $0x483fffffffffffff - QUAD $0xfffffffffffffab8 - QUAD $0xfffffffff9ba483f - QUAD $0xfffff8be493fffff - QUAD $0x7c89483fffffffff - QUAD $0x4c20247489482824 - QUAD $0x906634eb18244c89 - QUAD $0x5c8b4828247c8b48 - QUAD $0x489f0411fac53024 - QUAD $0x834904c38349c3ff - QUAD $0x4808244c894c04c1 - QUAD $0x4cf339482024748b - QUAD $0x01cd840f18244c8b - QUAD $0xc530245c89480000 - QUAD $0x7308f98349c057f8 - QUAD $0x08244c8b4cdb310f - QUAD $0x001f0f000000eae9 - QUAD $0x49db3110247c8b48 - QUAD $0xfffffffffffffeb9 - QUAD $0x2e6666666666663f + QUAD $0xc9854d38ec8348f8 + QUAD $0x20244c894cc0940f + QUAD $0x41c19c0f41c8294d + QUAD $0x00000591850fc108 + QUAD $0x48c1394c01408d49 + QUAD $0x1024448948c14e0f + QUAD $0x247c894810f88348 + QUAD $0x8948282474894830 + QUAD $0xc93145087d182454 + QUAD $0x4c8b48000002b3e9 + QUAD $0x8e8e0fc985482024 + QUAD $0x418a048d4c000001 + QUAD $0xbb4903e28341ca89 + QUAD $0x7ffffffffffffff8 + QUAD $0x48cb214904cb8349 + QUAD $0x488a048d482c5e8d + QUAD $0x0824448948fcc083 + QUAD $0xf6314500000010ba + QUAD $0xfffffffffffebf49 + QUAD $0xfffffffdbc493fff + QUAD $0xfffcbd493fffffff + QUAD $0x2feb3fffffffffff + QUAD $0x7ca1c430247c8b48 + QUAD $0x4c117ca1c4b70411 + QUAD $0xc28348d1894920b7 + QUAD $0xc6834840c3834810 + QUAD $0x24543b48ce894d40 + QUAD $0xc5000002218f0f10 + QUAD $0x7304f98348c057f8 + QUAD $0xe9c957f0c5c03117 + QUAD $0x2e666666000000b3 QUAD $0x0000000000841f0f - QUAD $0xc4e49b4c107ac1c4 - QUAD $0xf2c5e89b54107ac1 - QUAD $0x8948c158fac50f59 - QUAD $0x0c59eac5ce314cde - QUAD $0x7ac1c4c158fac5b1 - QUAD $0x4cde8948ec9b4c10 - QUAD $0xc5b10c59f2c5fe31 - QUAD $0x4c107ac1c4c158fa - QUAD $0xe6314cde8948f09b - QUAD $0x7ac1c4b10c59f2c5 - QUAD $0x4cde8948f49b5410 - QUAD $0xc5b11459eac5ee31 - QUAD $0xc4c258fac5c158fa - QUAD $0x8948f89b4c107ac1 - QUAD $0x0c59f2c5c63148de - QUAD $0x7ac1c4c158fac5b1 - QUAD $0x48de8948fc9b4c10 - QUAD $0xc5b10c59f2c5d631 - QUAD $0x0c107ac1c4c158fa - QUAD $0xc5f6314cde89489b - QUAD $0xc158fac5b10c59f2 - QUAD $0xe0c7834808c38348 - QUAD $0xffff47850fda3949 - QUAD $0x244c8b4cc0854dff - QUAD $0x48fffffec9840f08 - QUAD $0x48000000009d348d - QUAD $0x49f7294810247c8b - QUAD $0x001f0ff631991c8d - QUAD $0x59f2c5b30c10fac5 - QUAD $0xc6ff48c158fac50f - QUAD $0x75f03949fcc78348 - QUAD $0x8348fffffe92e9e7 - QUAD $0x7bebc031047304fe - QUAD $0xffffffffffe0b948 - QUAD $0x047320fe83487fff - QUAD $0x48f089483febc031 - QUAD $0xc057f8c5d231c821 + QUAD $0xc5c03108247c8b48 + QUAD $0x0000441f0fc957f0 + QUAD $0x59ecc517187de2c4 + QUAD $0xc5c358fcc5d4835c + QUAD $0xc18948f4835459ec + QUAD $0x1c187dc2c4f9314c + QUAD $0xc5d8836459e4c588 + QUAD $0xc5c458fcc5ca58f4 + QUAD $0xc18948f8835459e4 + QUAD $0x1c187dc2c4e1314c + QUAD $0xc5dc836459e4c588 + QUAD $0xc5c458fcc5ca58f4 + QUAD $0xc18948fc835459e4 + QUAD $0x1c187dc2c4e9314c + QUAD $0xc5e0836459e4c588 + QUAD $0xc5c458fcc5ca58f4 + QUAD $0xca58f4c5831459e4 + QUAD $0xf0c7834804c08348 + QUAD $0xffff77850fc33949 + QUAD $0x244c8b48d2854dff + QUAD $0x48ffffff09840f20 + QUAD $0x4c02e0c148863c8d + QUAD $0x31c1294908244c8b + QUAD $0x00000000801f0fc0 + QUAD $0x59ecc511187dc2c4 + QUAD $0xecc5c358fcc5871c + QUAD $0xca58f4c520875459 + QUAD $0x49fcc18349c0ff48 + QUAD $0xfffec7e9dc75c239 + QUAD $0x8d481024448b48ff + QUAD $0xe8c149c88949f048 + QUAD $0x83c08944c0ff4904 + QUAD $0x0e7370f9834807e0 + QUAD $0x314500000010b841 + QUAD $0x8349000000c4e9c9 + QUAD $0x57f8c5c93145f8e0 + QUAD $0x2e666666666666c0 QUAD $0x0000000000841f0f - QUAD $0x11fcc5970411fcc5 - QUAD $0x974411fcc5209744 - QUAD $0x4860974411fcc540 - QUAD $0xe075d0394820c283 - QUAD $0xc6f6403a74c63948 - QUAD $0x8348c2894828741c - QUAD $0xf02148c889481cc1 - QUAD $0x00401f0fc057f8c5 + QUAD $0xa1c48f04117ca1c4 + QUAD $0x7ca1c4208f44117c + QUAD $0x117ca1c4408f4411 + QUAD $0x84117ca1c4608f44 + QUAD $0x7ca1c4000000808f + QUAD $0xc4000000a08f8411 + QUAD $0x0000c08f84117ca1 + QUAD $0xe08f84117ca1c400 + QUAD $0x84117ca1c4000000 + QUAD $0x7ca1c4000001008f + QUAD $0xc4000001208f8411 + QUAD $0x0001408f84117ca1 + QUAD $0x608f84117ca1c400 + QUAD $0x84117ca1c4000001 + QUAD $0x7ca1c4000001808f + QUAD $0xc4000001a08f8411 + QUAD $0x0001c08f84117ca1 + QUAD $0xe08f84117ca1c400 + QUAD $0x4980e98349000001 + QUAD $0xffff5f850ff8c083 + QUAD $0x8d4d2a74c08548ff + QUAD $0x9066c057f8c51041 + QUAD $0xa1c48f04117ca1c4 + QUAD $0xc1894d208f44117c + QUAD $0x75c8ff4810c08349 + QUAD $0xc1894df0c08349e7 + QUAD $0x4c894c1024448b48 + QUAD $0x548b48c8294c0824 + QUAD $0x0f1824748b482824 + QUAD $0x4c8b48000002948e + QUAD $0xd38e0fc985482024 + QUAD $0x418e048d4c000001 + QUAD $0xbb4907e28341ca89 + QUAD $0x7ffffffffffffff8 + QUAD $0x0824448b48cb2149 + QUAD $0x1cc38348821c8d48 + QUAD $0x4c8948fc8e4c8d48 + QUAD $0x894882048d481824 + QUAD $0xfffffebe49282444 + QUAD $0xfdbf493fffffffff + QUAD $0x493fffffffffffff + QUAD $0xfffffffffffffcbc + QUAD $0xfffffffffbbd493f + QUAD $0xfffffab8483fffff + QUAD $0xf9ba483fffffffff + QUAD $0x493fffffffffffff + QUAD $0xfffffffffffff8b9 + QUAD $0x0000441f0f31eb3f + QUAD $0x748b4830244c8b48 + QUAD $0x48b10411fac50824 + QUAD $0x834804c38348c6ff + QUAD $0x2474894804282444 + QUAD $0x840f1024743b4808 + QUAD $0x247c8348000001d5 + QUAD $0xc057f8c50c730820 + QUAD $0x90000000d2e9f631 + QUAD $0x247c8b48c057f8c5 + QUAD $0x0000441f0ff63118 + QUAD $0xfac5e4b34c10fac5 + QUAD $0x0f59f2c5e8b35410 + QUAD $0x4cf18948c158fac5 + QUAD $0x880c596ac1c4f131 + QUAD $0x4c10fac5c158fac5 + QUAD $0xf9314cf18948ecb3 + QUAD $0xfac5880c5972c1c4 + QUAD $0xf0b34c10fac5c158 + QUAD $0xc1c4e1314cf18948 + QUAD $0x5410fac5880c5972 + QUAD $0xe9314cf18948f4b3 + QUAD $0xfac58814596ac1c4 + QUAD $0xfac5c258fac5c158 + QUAD $0x48f18948f8b34c10 + QUAD $0x880c5972c1c4c131 + QUAD $0x4c10fac5c158fac5 + QUAD $0xd13148f18948fcb3 + QUAD $0xfac5880c5972c1c4 + QUAD $0x48b30c10fac5c158 + QUAD $0x72c1c4c9314cf189 + QUAD $0x48c158fac5880c59 + QUAD $0x49e0c7834808c683 + QUAD $0xffffff48850ff339 + QUAD $0xfffeef840fd2854d + QUAD $0x000000b50c8d48ff + QUAD $0x294818247c8b4800 + QUAD $0x8d4828244c8b48cf + QUAD $0x00401f0fc931b134 + QUAD $0x59f2c58e0c10fac5 + QUAD $0xc1ff48c158fac50f + QUAD $0x75ca3949fcc78348 + QUAD $0x8348fffffeb2e9e7 + QUAD $0x730824448b4c04f8 + QUAD $0x00008fe9c0894c08 + QUAD $0x31047320f8834800 + QUAD $0x8348c2894843ebd2 + QUAD $0x8348870c8d4ae0e2 + QUAD $0xc057f8c5f63160c1 + QUAD $0xfcc5a0b14411fcc5 + QUAD $0x4411fcc5c0b14411 + QUAD $0x48b10411fcc5e0b1 + QUAD $0xe075f2394820c683 + QUAD $0x741ca85c74d03948 + QUAD $0x8b480824448b4c6a + QUAD $0x03e683ce8910244c + QUAD $0x014cc0014cf02948 + QUAD $0x57f8c5fce18348c2 + QUAD $0x00000000801f0fc0 QUAD $0xc28348970411f8c5 - QUAD $0x3948f275d0394804 - QUAD $0x00008704c70c74c6 - QUAD $0x48efebc0ff480000 - QUAD $0x5d415c415bd8658d - QUAD $0x77f8c55d5f415e41 - BYTE $0xc3 + QUAD $0x8548f275d1394804 + QUAD $0x2e666666661e74f6 + QUAD $0x0000000000841f0f + QUAD $0x48000000008704c7 + QUAD $0x751024443948c0ff + QUAD $0x5c415bd8658d48ef + QUAD $0xc55d5f415e415d41 + QUAD $0x0824448b48c377f8 + LONG $0xebd00148 + BYTE $0xd3 RET // func correlateFloat32AVX2(dst []float32, sig []float32, ker []float32) @@ -383,71 +560,149 @@ TEXT ·correlateFloat32AVX2(SB), NOSPLIT|NOFRAME, $0-72 MOVQ ker_len+56(FP), R9 QUAD $0x294dc0940fc9854d QUAD $0xc20841c29c0f41c8 - QUAD $0x8d49000001e9850f - QUAD $0x4e0f48c1394c0140 - QUAD $0x01d58e0fc08548c1 - QUAD $0x358e0fc9854d0000 - QUAD $0x41e5894855000001 - QUAD $0xf8e4834853564157 - QUAD $0xb84907e183c98944 - QUAD $0x7fffffffffffffe0 - QUAD $0x4cc8214d18c88349 - QUAD $0x1aebdb31451c568d - QUAD $0xff499f04117aa1c4 - QUAD $0xc6834804c28349c3 - QUAD $0x00e3840fc3394904 - QUAD $0xdb31c057f8c50000 - QUAD $0x00a2820f08f98349 - QUAD $0x0000441f0f660000 - QUAD $0xc4e49a4c107ac1c4 - QUAD $0xf2c5e89a54107ac1 + QUAD $0x485500000458850f + QUAD $0x554156415741e589 + QUAD $0x49f8e48348535441 + QUAD $0x0f48c1394c01408d + QUAD $0x077d10f88348c14e + QUAD $0x4d0000022fe9c931 + QUAD $0x000001308e0fc985 + QUAD $0x4903e08341c88945 + QUAD $0xfffffffffffff8ba + QUAD $0xca214d04ca83497f + QUAD $0x0010bf412c5e8d4c + QUAD $0xf38948f631450000 + QUAD $0x2e666666666633eb + QUAD $0x0000000000841f0f + QUAD $0xa1c4b704117ca1c4 + QUAD $0x798d4c20b74c117c + QUAD $0xc3834840c3834910 + QUAD $0x0fc73949ce894940 + QUAD $0xf9894c000001c98f + QUAD $0xc5ff3145c057f8c5 + QUAD $0x0f04f98349c957f0 + QUAD $0x001f0f0000008782 + QUAD $0x81c4ba14187da2c4 + QUAD $0x58fcc5d4bb5c596c + QUAD $0xf4bb54596c81c4c3 + QUAD $0x187da2c4ca58f4c5 + QUAD $0x5c596c81c404ba54 + QUAD $0x81c4c358fcc5d8bb + QUAD $0x58f4c5f8bb54596c + QUAD $0x08ba54187da2c4ca + QUAD $0xc5dcbb5c596c81c4 + QUAD $0x54596c81c4c358fc + QUAD $0xa2c4ca58f4c5fcbb + QUAD $0x6c81c40cba54187d + QUAD $0x596c81c4e0bb5c59 + QUAD $0xf4c5c358fcc5bb14 + QUAD $0x394d04c78349ca58 + QUAD $0x840fc0854d8575fa + QUAD $0xbb248d4effffff3c + QUAD $0x90ed3145ba3c8d4e + QUAD $0x81c4af14187d82c4 + QUAD $0x596c81c4ac1c596c + QUAD $0xc5c358fcc520ac54 + QUAD $0x394dc5ff49ca58f4 + QUAD $0xffffff08e9dd75e8 + QUAD $0x49ca8949f0488d48 + QUAD $0x8945c2ff4904eac1 + QUAD $0xf9834807e08341d0 + QUAD $0x000010ba410d7370 + QUAD $0x000000aae9c93100 + QUAD $0xf8c5c931f8e28349 + QUAD $0x0000441f0f66c057 + QUAD $0x11fcc58f0411fcc5 + QUAD $0x8f4411fcc5208f44 + QUAD $0xc5608f4411fcc540 + QUAD $0x000000808f8411fc + QUAD $0x0000a08f8411fcc5 + QUAD $0x00c08f8411fcc500 + QUAD $0xe08f8411fcc50000 + QUAD $0x8f8411fcc5000000 + QUAD $0x8411fcc500000100 + QUAD $0x11fcc5000001208f + QUAD $0xfcc5000001408f84 + QUAD $0xc5000001608f8411 + QUAD $0x000001808f8411fc + QUAD $0x0001a08f8411fcc5 + QUAD $0x01c08f8411fcc500 + QUAD $0xe08f8411fcc50000 + QUAD $0x4980e98348000001 + QUAD $0xffff6f850ff8c283 + QUAD $0x8d4c2874c0854dff + QUAD $0x9066c057f8c51051 + QUAD $0x11fcc58f0411fcc5 + QUAD $0x8349d1894c208f44 + QUAD $0x49e975c8ff4910c2 + QUAD $0x8949d1894cf0c283 + QUAD $0x01e68e0fc82949c0 + QUAD $0x2a8e0fc9854d0000 + QUAD $0x8341c88945000001 + QUAD $0xfffffff8ba4907e0 + QUAD $0x4cca214d7fffffff + QUAD $0x481cc383498e1c8d + QUAD $0x2e666624eb8e348d + QUAD $0x0000000000841f0f + QUAD $0xc1ff488f0411fac5 + QUAD $0x04c6834804c38349 + QUAD $0x000197840fc13948 + QUAD $0x49db31c057f8c500 + QUAD $0x0000a3820f08f983 + QUAD $0x00000000801f0f00 + QUAD $0xc4e49b4c107ac1c4 + QUAD $0xf2c5e89b54107ac1 QUAD $0x9a5459eac59a0c59 QUAD $0x7ac1c4c158fac504 - QUAD $0x4c59f2c5ec9a4c10 + QUAD $0x4c59f2c5ec9b4c10 QUAD $0xc1c4c258fac5089a - QUAD $0x59eac5f09a54107a + QUAD $0x59eac5f09b54107a QUAD $0xc4c158fac50c9a54 - QUAD $0xf2c5f49a4c107ac1 + QUAD $0xf2c5f49b4c107ac1 QUAD $0xc258fac5109a4c59 - QUAD $0xc5f89a54107ac1c4 + QUAD $0xc5f89b54107ac1c4 QUAD $0x58fac5149a5459ea - QUAD $0xfc9a4c107ac1c4c1 + QUAD $0xfc9b4c107ac1c4c1 QUAD $0xfac5189a4c59f2c5 - QUAD $0x9a14107ac1c4c258 + QUAD $0x9b14107ac1c4c258 QUAD $0xfac51c9a5459eac5 QUAD $0x8348c258fac5c158 - QUAD $0x6d850fd8394908c3 - QUAD $0x840fc98548ffffff + QUAD $0x6d850fda394908c3 + QUAD $0x840fc0854dffffff QUAD $0x9a348d4cffffff34 QUAD $0x66ff31459e1c8d48 QUAD $0x0000000000841f0f QUAD $0x81c4bb0c107aa1c4 QUAD $0xc158fac5be0c5972 - QUAD $0xe875f9394cc7ff49 - QUAD $0x658d48ffffff03e9 - QUAD $0xc35d5f415e415be8 - QUAD $0xc931047304f88348 - QUAD $0xffffffe0ba487eeb - QUAD $0x20f883487fffffff - QUAD $0x894843ebc9310473 - QUAD $0xf8c5f631d12148c1 - QUAD $0x2e6666666666c057 - QUAD $0x0000000000841f0f - QUAD $0x11fcc5b70411fcc5 - QUAD $0xb74411fcc520b744 - QUAD $0x4860b74411fcc540 - QUAD $0xe075f1394820c683 - QUAD $0x741ca83a74c83948 - QUAD $0x1cc28348ce894827 - QUAD $0xf8c5c12148d18948 - QUAD $0x0000441f0f66c057 - QUAD $0xc68348b70411f8c5 - QUAD $0x0aebf275f1394804 - QUAD $0x48000000008f04c7 - QUAD $0xc5f175c83948c1ff - BYTE $0xf8 - BYTE $0x77 - BYTE $0xc3 + QUAD $0xe875f8394dc7ff49 + QUAD $0xf88349ffffff03e9 + QUAD $0x95e9c88949087304 + QUAD $0x7320f88349000000 + QUAD $0xc2894c4aebd23104 + QUAD $0x8f348d48e0e28348 + QUAD $0xc5c9314560c68348 + QUAD $0x0000441f0fc057f8 + QUAD $0xc4a08e44117ca1c4 + QUAD $0xa1c4c08e44117ca1 + QUAD $0x7ca1c4e08e44117c + QUAD $0x4c20c183498e0411 + QUAD $0x74d03949dc75ca39 + QUAD $0x8963741cc0f64157 + QUAD $0x49f0294903e683c6 + QUAD $0xc18948ca0148c801 + QUAD $0xc057f8c5fce18348 + QUAD $0x0000000000841f0f + QUAD $0xc28348970411f8c5 + QUAD $0x8548f275d1394804 + QUAD $0x2e666666661d74f6 + QUAD $0x0000000000841f0f + QUAD $0x000000008704c742 + QUAD $0xf075c0394cc0ff49 + QUAD $0x415c415bd8658d48 + QUAD $0xf8c55d5f415e415d + QUAD $0xc88949d10148c377 + BYTE $0xeb + BYTE $0xd6 RET // func normFloat64AVX2(a []float64) float64 @@ -639,24 +894,118 @@ TEXT ·polyEvalFloat64AVX2(SB), NOSPLIT|NOFRAME, $0-72 MOVQ x_len+32(FP), R8 MOVQ coeffs_len+56(FP), R9 QUAD $0x4dc14c0f4cc1394c - QUAD $0x000000f7840fc985 - QUAD $0x0002348e0fc0854d - QUAD $0xf8ca44107ba1c400 - QUAD $0x0f02e88348c8894c - QUAD $0x894855000000f08c - QUAD $0x8d41f8e4834853e5 - QUAD $0xe1834107e183ff49 - QUAD $0x2e661cebd2314507 - QUAD $0x0000000000841f0f - QUAD $0xff49d714117ba1c4 - QUAD $0x00c8840fc2394dc2 - QUAD $0xd60c107ba1c40000 - QUAD $0x41d028f9c5c38949 - QUAD $0xcb89481d7401f983 - QUAD $0x90d028f9c5c38949 + QUAD $0xe58948551a74c985 + QUAD $0xe483485356415741 + QUAD $0x31217d10f88349f8 + QUAD $0x854d000002e5e9c0 + QUAD $0x49000005148e0fc0 + QUAD $0x0001d1830f04f883 + QUAD $0x000004f3e9c03100 + QUAD $0x49f8ca44107ba1c4 + QUAD $0x0001928e0f01f983 + QUAD $0x518d45fe498d4900 + QUAD $0x8341c0197de2c4ff + QUAD $0xe38341cb894503e2 + QUAD $0x3100000010be4103 + QUAD $0x2e6666666634ebdb + QUAD $0x0000000000841f0f + QUAD $0x11fdc5df3411fdc5 + QUAD $0xdf7c11fdc520df6c + QUAD $0x4c60df44117dc540 + QUAD $0x394dc3894810708d + QUAD $0x4c000002648f0fc6 + QUAD $0xc5de0c10fdc5f089 + QUAD $0x10fdc520de5410fd + QUAD $0xde6410fdc540de5c + QUAD $0xf028fdc5ce894960 + QUAD $0xf828fdc5e828fdc5 + QUAD $0x01fb8341c0287dc5 + QUAD $0xce8949d7894d4574 + QUAD $0xe828fdc5f028fdc5 + QUAD $0xc0287dc5f828fdc5 + QUAD $0x197d22c4f159cdc5 + QUAD $0xd5c5f658b5c5f20c + QUAD $0xc5c5ed58b5c5ea59 + QUAD $0x3dc5ff58b5c5fb59 + QUAD $0x49c0583541c4c459 + QUAD $0x48d175cfff49ceff + QUAD $0xffff57820f03f983 + QUAD $0x001f0f04c68349ff + QUAD $0x197d22c4f159cdc5 + QUAD $0xc5f658b5c5e0f24c + QUAD $0xc5ed58b5c5ea59d5 + QUAD $0xc5ff58b5c5fb59c5 + QUAD $0xc0583541c4c4593d + QUAD $0x197d22c4f159cdc5 + QUAD $0xc5f658b5c5d8f24c + QUAD $0xc5ed58b5c5ea59d5 + QUAD $0xc5ff58b5c5fb59c5 + QUAD $0xc0583541c4c4593d + QUAD $0x197d22c4f159cdc5 + QUAD $0xc5f658b5c5d0f24c + QUAD $0xc5ed58b5c5ea59d5 + QUAD $0xc5ff58b5c5fb59c5 + QUAD $0xc0583541c4c4593d + QUAD $0x197d22c4f159cdc5 + QUAD $0xc5f658b5c5c8f24c + QUAD $0xc5ed58b5c5ea59d5 + QUAD $0xc5ff58b5c5fb59c5 + QUAD $0xc0583541c4c4593d + QUAD $0x03fe8349fcc68349 + QUAD $0x9de9ffffff528f0f + QUAD $0x49f0408d49fffffe + QUAD $0xff4904eac149c289 + QUAD $0x4803e183d18944c2 + QUAD $0x10ba41287330f883 + QUAD $0x00c0e9c031000000 + QUAD $0xfffffff0b9480000 + QUAD $0x10f883497fffffff + QUAD $0xc031000001f4830f + QUAD $0xe283490000022ee9 + QUAD $0xc031c8197de2c4fc + QUAD $0x11fdc5c70c11fdc5 + QUAD $0xc74c11fdc520c74c + QUAD $0xc560c74c11fdc540 + QUAD $0x00000080c78c11fd + QUAD $0x0000a0c78c11fdc5 + QUAD $0x00c0c78c11fdc500 + QUAD $0xe0c78c11fdc50000 + QUAD $0xc78c11fdc5000000 + QUAD $0x8c11fdc500000100 + QUAD $0x11fdc500000120c7 + QUAD $0xfdc500000140c78c + QUAD $0xc500000160c78c11 + QUAD $0x00000180c78c11fd + QUAD $0x0001a0c78c11fdc5 + QUAD $0x01c0c78c11fdc500 + QUAD $0xe0c78c11fdc50000 + QUAD $0x4940c08348000001 + QUAD $0xffff6f850ffcc283 + QUAD $0x8d4c4574c98548ff + QUAD $0xc1c0197de2c41050 + QUAD $0x2e666666666604e1 + QUAD $0x0000000000841f0f + QUAD $0x11fdc5c70411fdc5 + QUAD $0xc74411fdc520c744 + QUAD $0x4c60c74411fdc540 + QUAD $0x834810c28349d089 + QUAD $0xf0c28349dc75f0c1 + QUAD $0x2948c1894cd0894c + QUAD $0xc4000002068e0fc1 + QUAD $0x894df8ca44107ba1 + QUAD $0xcc8c0f02ea8349ca + QUAD $0x83ff498d41000000 + QUAD $0x19eb07e1834107e1 + QUAD $0x0000000000841f0f + QUAD $0xc0ff48c71411fbc5 + QUAD $0x0001cc840fc0394c + QUAD $0x894dc60c10fbc500 + QUAD $0xf98341d028f9c5d3 + QUAD $0x894dcb89481f7401 + QUAD $0x001f0fd028f9c5d3 QUAD $0x586ba1c4d259f3c5 QUAD $0xcbff48cbff49da14 - QUAD $0xb87207f88348ee75 + QUAD $0xb87207fa8349ee75 QUAD $0x00401f0f08c38349 QUAD $0x586ba1c4d259f3c5 QUAD $0xc4d259f3c5c0da54 @@ -670,48 +1019,51 @@ TEXT ·polyEvalFloat64AVX2(SB), NOSPLIT|NOFRAME, $0-72 QUAD $0x59f3c590da54586b QUAD $0x88da54586ba1c4d2 QUAD $0x07fb8349f8c38349 - QUAD $0x4dffffff49e99e7f - QUAD $0x0000013d8e0fc085 - QUAD $0xc0311b7304f88349 - QUAD $0xf8834900000120e9 - QUAD $0x0109e9c031227304 - QUAD $0x5d5bf8658d480000 - QUAD $0xfffffffff0b948c3 - QUAD $0x7310f883497fffff - QUAD $0xf0b9485eebc0311b - QUAD $0x497fffffffffffff - QUAD $0xe9c0317b7310f883 - QUAD $0x48c0894c000000af - QUAD $0xc057f9c5d231c821 - QUAD $0x0000000000841f0f - QUAD $0x11fdc5d70411fdc5 - QUAD $0xd74411fdc520d744 - QUAD $0x4860d74411fdc540 + QUAD $0x48ffffff49e99e7f + QUAD $0xc18948087304f983 + QUAD $0xf983480000010be9 + QUAD $0x00bce9d231787310 + QUAD $0xc82148c0894c0000 + QUAD $0x9066c057f8c5d231 + QUAD $0x11fcc5d70411fcc5 + QUAD $0xd74411fcc520d744 + QUAD $0x4860d74411fcc540 QUAD $0xe075d0394810c283 - QUAD $0x0000b4840fc03949 - QUAD $0x9a840f0cc0f64100 + QUAD $0x0000fa840fc03949 + QUAD $0xe0840f0cc0f64100 QUAD $0x8348c28948000000 QUAD $0xc0214cc889480cc1 - QUAD $0x2e666666c057f9c5 + QUAD $0x2e666666c057f8c5 QUAD $0x0000000000841f0f - QUAD $0xc28348d70411fdc5 - QUAD $0x78ebf275d0394804 - QUAD $0xe2c4c82148c0894c - QUAD $0x001f0fd231c8197d - QUAD $0x11fdc5d70c11fdc5 - QUAD $0xd74c11fdc520d74c - QUAD $0x4860d74c11fdc540 - QUAD $0xe075d0394810c283 - QUAD $0xc0f6414874c03949 - QUAD $0x8348c2894828740c - QUAD $0xc0214cc889480cc1 - QUAD $0x001f0fc8197de2c4 + QUAD $0xc28348d70411fcc5 + QUAD $0xbbe9f275d0394804 + QUAD $0xe683c68944000000 + QUAD $0xc4f22948ca89480f + QUAD $0x49c1894dc8197de2 + QUAD $0x9066c28949f0e183 + QUAD $0xa1c4d70c117da1c4 + QUAD $0x7da1c420d74c117d + QUAD $0x117da1c440d74c11 + QUAD $0x4d10c2834960d74c + QUAD $0x74f68548dc75d139 + QUAD $0x89445d7204fe8354 + QUAD $0x48f1294803e683c6 + QUAD $0x48c8197de2c4c101 + QUAD $0xe08348c0894cc201 + QUAD $0x00000000801f0ffc QUAD $0xc28348d70c11fdc5 - QUAD $0x3949f275d0394804 - QUAD $0xc70411fbc51a74c0 - QUAD $0x04c748f1ebc0ff48 - QUAD $0xc0ff4800000000c7 - QUAD $0x77f8c5f075c03949 + QUAD $0x8548f275d0394804 + QUAD $0x2e666666661a74f6 + QUAD $0x0000000000841f0f + QUAD $0xc1ff48cf0411fbc5 + QUAD $0x658d48f375c83949 + QUAD $0xc55d5f415e415be8 + QUAD $0x8948d00148c377f8 + QUAD $0x00c704c748ddebc1 + QUAD $0x3949c0ff48000000 + LONG $0xc5f075c0 + BYTE $0xf8 + BYTE $0x77 BYTE $0xc3 RET @@ -725,98 +1077,214 @@ TEXT ·convolveFloat64AVX2(SB), NOSPLIT|NOFRAME, $0-72 MOVQ ker_len+56(FP), R9 QUAD $0x56415741e5894855 QUAD $0xe483485354415541 - QUAD $0x74894838ec8348f8 - QUAD $0xc0940fc9854d0824 - QUAD $0x40c69c0f40c8294d - QUAD $0x000002a0850fc608 - QUAD $0x48c1394c01708d49 - QUAD $0x8e0ff68548f14e0f - QUAD $0x0fc9854d0000028c - QUAD $0x0c8d4a000001f18e - QUAD $0x07e08341c88945ca - QUAD $0xfffffffffff0ba49 - QUAD $0x214d08ca83497fff - QUAD $0x8d4c0824448b48ca - QUAD $0x8348ca048d4a3858 - QUAD $0x311024448948f8c0 - QUAD $0xfffffffffdbf49db - QUAD $0xfffffcbc491fffff - QUAD $0xfbbd491fffffffff - QUAD $0x481fffffffffffff - QUAD $0xfffffffffffffab8 - QUAD $0xfffffffff9ba481f - QUAD $0xfffff8be491fffff - QUAD $0x7c89481fffffffff - QUAD $0x4c20247489482824 - QUAD $0x906634eb18244c89 - QUAD $0x5c8b4828247c8b48 - QUAD $0x48df0411fbc53024 - QUAD $0x834908c38349c3ff - QUAD $0x4808244c894c08c1 - QUAD $0x4cf339482024748b - QUAD $0x01ce840f18244c8b - QUAD $0xc530245c89480000 - QUAD $0x7308f98349c057f9 - QUAD $0x08244c8b4cdb310f - QUAD $0x001f0f000000eae9 - QUAD $0x49db3110247c8b48 - QUAD $0xfffffffffffffeb9 - QUAD $0x2e6666666666661f + QUAD $0x7c894838ec8348f8 + QUAD $0xc0940fc9854d1024 + QUAD $0xc8294d18244c894c + QUAD $0x0fc70840c79c0f40 + QUAD $0x408d490000063d85 + QUAD $0xc14e0f48c1394c01 + QUAD $0xf883480824448948 + QUAD $0x032ce9c931077d10 + QUAD $0x4d18246c8b4c0000 + QUAD $0x0f1024648b4ced85 + QUAD $0x048d4e0000015f8e + QUAD $0xfffffffff8ba49ea + QUAD $0x4d06ca83497fffff + QUAD $0x8d4a685e8d4cea21 + QUAD $0x10bff8c38348ea1c + QUAD $0xb848c93145000000 + QUAD $0x1ffffffffffffffe + QUAD $0x001f0f0000007ee9 + QUAD $0xc5c957f1c5f63145 + QUAD $0x4adb57e1c5d257e9 + QUAD $0xf749f7894dce3c8d + QUAD $0xc4f824197d82c4d7 + QUAD $0x5da1c4f72c595da1 + QUAD $0xc558fdc520f77459 + QUAD $0x595da1c4ce58f5c5 + QUAD $0xc4d558edc540f76c + QUAD $0xe5c560f764595da1 + QUAD $0xcc04117d81c4dc58 + QUAD $0xc420cc4c117d81c4 + QUAD $0x81c440cc54117d81 + QUAD $0x798d4860cc5c117d + QUAD $0xc9894980eb834910 + QUAD $0x638f0f08247c3b48 + QUAD $0xf9c5f98948000002 + QUAD $0x840f01fd8349c057 + QUAD $0x45df8948ffffff74 + QUAD $0xe9c5c957f1c5f631 + QUAD $0x9066db57e1c5d257 + QUAD $0x5d81c427197de2c4 + QUAD $0xc558fdc598f36c59 + QUAD $0xc5b8f36c595d81c4 + QUAD $0x6c595d81c4cd58f5 + QUAD $0x81c4d558edc5d8f3 + QUAD $0x58e5c5f8f364595d + QUAD $0xc4c73149f7894ddc + QUAD $0x5d81c4f824197d82 + QUAD $0xc558fdc5a0f36c59 + QUAD $0xc4c0f36c595d81c4 + QUAD $0xf5c5e0f374595d81 + QUAD $0x81c4d658edc5cd58 + QUAD $0xdc58e5c5f324595d + QUAD $0xf0c7834802c68349 + QUAD $0xc5f6418b75f2394d + QUAD $0xe9fffffef0850f01 + QUAD $0x24448b48ffffff26 + QUAD $0xcf8948f0488d4808 + QUAD $0x89c7ff4804efc148 + QUAD $0x70f9834807e083f8 + QUAD $0x3100000010bf0c73 + QUAD $0x83480000015ce9c9 + QUAD $0xc057f9c5c931f8e7 QUAD $0x0000000000841f0f - QUAD $0xc4c8db4c107bc1c4 - QUAD $0xf3c5d0db54107bc1 - QUAD $0x8948c158fbc50f59 - QUAD $0x0c59ebc5ce314cde - QUAD $0x7bc1c4c158fbc5f1 - QUAD $0x4cde8948d8db4c10 - QUAD $0xc5f10c59f3c5fe31 - QUAD $0x4c107bc1c4c158fb - QUAD $0xe6314cde8948e0db - QUAD $0x7bc1c4f10c59f3c5 - QUAD $0x4cde8948e8db5410 - QUAD $0xc5f11459ebc5ee31 - QUAD $0xc4c258fbc5c158fb - QUAD $0x8948f0db4c107bc1 - QUAD $0x0c59f3c5c63148de - QUAD $0x7bc1c4c158fbc5f1 - QUAD $0x48de8948f8db4c10 - QUAD $0xc5f10c59f3c5d631 - QUAD $0x0c107bc1c4c158fb - QUAD $0xc5f6314cde8948db - QUAD $0xc158fbc5f10c59f3 - QUAD $0xc0c7834808c38348 - QUAD $0xffff47850fda3949 - QUAD $0x244c8b4cc0854dff - QUAD $0x48fffffec9840f08 - QUAD $0x4800000000dd348d - QUAD $0x49f7294810247c8b - QUAD $0x001f0ff631d91c8d - QUAD $0x59f3c5f30c10fbc5 - QUAD $0xc6ff48c158fbc50f - QUAD $0x75f03949f8c78348 - QUAD $0x8348fffffe92e9e7 - QUAD $0x7bebc031047304fe - QUAD $0xfffffffffff0b948 - QUAD $0x047310fe83487fff - QUAD $0x48f089483febc031 - QUAD $0xc057f9c5d231c821 - QUAD $0x0000000000841f0f - QUAD $0x11fdc5d70411fdc5 - QUAD $0xd74411fdc520d744 - QUAD $0x4860d74411fdc540 - QUAD $0xe075d0394810c283 - QUAD $0xc6f6403b74c63948 - QUAD $0x8348c2894828740c - QUAD $0xf02148c889480cc1 - QUAD $0x00401f0fc057f9c5 - QUAD $0xc28348d70411fdc5 - QUAD $0x3948f275d0394804 - QUAD $0x00c704c7480d74c6 - QUAD $0xeeebc0ff48000000 - QUAD $0x415c415bd8658d48 - QUAD $0xf8c55d5f415e415d - BYTE $0x77 - BYTE $0xc3 + QUAD $0xc1c4cc04117dc1c4 + QUAD $0x7dc1c420cc44117d + QUAD $0x117dc1c440cc4411 + QUAD $0x84117dc1c460cc44 + QUAD $0x7dc1c400000080cc + QUAD $0xc4000000a0cc8411 + QUAD $0x0000c0cc84117dc1 + QUAD $0xe0cc84117dc1c400 + QUAD $0x84117dc1c4000000 + QUAD $0x7dc1c400000100cc + QUAD $0xc400000120cc8411 + QUAD $0x000140cc84117dc1 + QUAD $0x60cc84117dc1c400 + QUAD $0x84117dc1c4000001 + QUAD $0x7dc1c400000180cc + QUAD $0xc4000001a0cc8411 + QUAD $0x0001c0cc84117dc1 + QUAD $0xe0cc84117dc1c400 + QUAD $0x84117dc1c4000001 + QUAD $0x7dc1c400000200cc + QUAD $0xc400000220cc8411 + QUAD $0x000240cc84117dc1 + QUAD $0x60cc84117dc1c400 + QUAD $0x84117dc1c4000002 + QUAD $0x7dc1c400000280cc + QUAD $0xc4000002a0cc8411 + QUAD $0x0002c0cc84117dc1 + QUAD $0xe0cc84117dc1c400 + QUAD $0x84117dc1c4000002 + QUAD $0x7dc1c400000300cc + QUAD $0xc400000320cc8411 + QUAD $0x000340cc84117dc1 + QUAD $0x60cc84117dc1c400 + QUAD $0x84117dc1c4000003 + QUAD $0x7dc1c400000380cc + QUAD $0xc4000003a0cc8411 + QUAD $0x0003c0cc84117dc1 + QUAD $0xe0cc84117dc1c400 + QUAD $0x4880e98348000003 + QUAD $0xfffebf850ff8c783 + QUAD $0x8d483874c08548ff + QUAD $0x9066c057f9c51079 + QUAD $0xc1c4cc04117dc1c4 + QUAD $0x7dc1c420cc44117d + QUAD $0x117dc1c440cc4411 + QUAD $0x8348f9894860cc44 + QUAD $0x48d975c8ff4810c7 + QUAD $0x8b48f98948f0c783 + QUAD $0x8e0fc82948082444 + QUAD $0x247c8b48000002e6 + QUAD $0x02058e0fff854818 + QUAD $0x8941fa048d4c0000 + QUAD $0xf8bb4907e28341fa + QUAD $0x497fffffffffffff + QUAD $0x8348ce1c8d48fb21 + QUAD $0x48f8fa448d4838c3 + QUAD $0xce048d4820244489 + QUAD $0xfdbf492824448948 + QUAD $0x491fffffffffffff + QUAD $0xfffffffffffffcbc + QUAD $0xfffffffffbbd491f + QUAD $0xfffffab8481fffff + QUAD $0xf9ba481fffffffff + QUAD $0x491fffffffffffff + QUAD $0xfffffffffffff8b9 + QUAD $0x0000441f0f27eb1f + QUAD $0x11fbc51024748b48 + QUAD $0xc38348c1ff48ce04 + QUAD $0x4808282444834808 + QUAD $0x0240840f08244c3b + QUAD $0x0818247c83480000 + QUAD $0xf631c057f9c51673 + QUAD $0x2e666600000113e9 + QUAD $0x0000000000841f0f + QUAD $0x57f9c530244c8948 + QUAD $0xf63120247c8b48c0 + QUAD $0x8948d18949c9894c + QUAD $0x4de5894de8894cc2 + QUAD $0xfffffffebf49fc89 + QUAD $0x00401f0f1fffffff + QUAD $0xfbc5c8f34c10fbc5 + QUAD $0x0f59f3c5d0f35410 + QUAD $0x4df68949c158fbc5 + QUAD $0xf00c596b81c4fe31 + QUAD $0x4c10fbc5c158fbc5 + QUAD $0xe6314df68949d8f3 + QUAD $0xfbc5f00c597381c4 + QUAD $0xe0f34c10fbc5c158 + QUAD $0x81c4ee314df68949 + QUAD $0x5410fbc5f00c5973 + QUAD $0xc63149f68949e8f3 + QUAD $0xfbc5f014596b81c4 + QUAD $0xfbc5c258fbc5c158 + QUAD $0x49f68949f0f34c10 + QUAD $0xf00c597381c4d631 + QUAD $0x4c10fbc5c158fbc5 + QUAD $0xce314df68949f8f3 + QUAD $0xfbc5f00c597381c4 + QUAD $0x49f30c10fbc5c158 + QUAD $0x7381c4ce3149f689 + QUAD $0x48c158fbc5f00c59 + QUAD $0x49c0c7834808c683 + QUAD $0xffffff48850ff339 + QUAD $0x894de7894dd2854d + QUAD $0x4cd08948c58949ec + QUAD $0x4c8b48c98949ca89 + QUAD $0xfffffeb8840f3024 + QUAD $0x00000000f5348d4c + QUAD $0xf7294c20247c8b48 + QUAD $0x348d492824748b4c + QUAD $0x2e666666f63145f6 + QUAD $0x0000000000841f0f + QUAD $0xf3c5f60c107ba1c4 + QUAD $0xff49c158fbc50f59 + QUAD $0xf2394df8c78348c6 + QUAD $0x48fffffe71e9e675 + QUAD $0xc88948087304f883 + QUAD $0xf88348000000a7e9 + QUAD $0x047310244c8b4c10 + QUAD $0x247c8b4857ebd231 + QUAD $0x89480fe683fe8908 + QUAD $0xf0e78348f22948c2 + QUAD $0x66c88949c057f9c5 + QUAD $0x0000000000841f0f + QUAD $0x81c4c104117d81c4 + QUAD $0x7d81c420c144117d + QUAD $0x117d81c440c14411 + QUAD $0x4c10c0834960c144 + QUAD $0x74f68548dc75c739 + QUAD $0x8b4c767204fe8369 + QUAD $0x08247c8b4810244c + QUAD $0xf0294803e683fe89 + QUAD $0x8948ca0148c80148 + QUAD $0x57f9c5fce18348f9 + QUAD $0x2e666666666666c0 + QUAD $0x0000000000841f0f + QUAD $0x8348d104117dc1c4 + QUAD $0x48f175d1394804c2 + QUAD $0x244c8b481e74f685 + QUAD $0x00000000801f0f10 + QUAD $0x00000000c104c748 + QUAD $0x0824443948c0ff48 + QUAD $0x415bd8658d48ee75 + QUAD $0x5d5f415e415d415c + QUAD $0x48d10148c377f8c5 + LONG $0xc8ebc889 RET // func correlateFloat64AVX2(dst []float64, sig []float64, ker []float64) @@ -829,69 +1297,167 @@ TEXT ·correlateFloat64AVX2(SB), NOSPLIT|NOFRAME, $0-72 MOVQ ker_len+56(FP), R9 QUAD $0x294dc0940fc9854d QUAD $0xc20841c29c0f41c8 - QUAD $0x8d49000001ea850f - QUAD $0x4e0f48c1394c0140 - QUAD $0x01d68e0fc08548c1 - QUAD $0x358e0fc9854d0000 - QUAD $0x41e5894855000001 - QUAD $0xf8e4834853564157 - QUAD $0xb84907e183c98944 - QUAD $0x7ffffffffffffff0 - QUAD $0x4cc8214d08c88349 - QUAD $0x1aebdb314538568d - QUAD $0xff49df04117ba1c4 - QUAD $0xc6834808c28349c3 - QUAD $0x00e3840fc3394908 - QUAD $0xdb31c057f9c50000 - QUAD $0x00a2820f08f98349 - QUAD $0x0000441f0f660000 - QUAD $0xc4c8da4c107bc1c4 - QUAD $0xf3c5d0da54107bc1 + QUAD $0x4855000004e4850f + QUAD $0x485356415741e589 + QUAD $0x4c01408d49f8e483 + QUAD $0x8348c14e0f48c139 + QUAD $0xbfe9c931077d10f8 + QUAD $0x8e0fc9854d000002 + QUAD $0xfff8b84900000128 + QUAD $0x83497fffffffffff + QUAD $0x568d4cc8214d06c8 + QUAD $0x314500000010bb68 + QUAD $0x2e6666666671ebdb + QUAD $0x0000000000841f0f + QUAD $0x197de2c4de348d4e + QUAD $0xde2c595dc1c4da24 + QUAD $0xc520de74595dc1c4 + QUAD $0xc4ce58f5c5c558fd + QUAD $0xedc540de6c595dc1 + QUAD $0xde64595dc1c4d558 + QUAD $0x7da1c4dc58e5c560 + QUAD $0x4c117da1c4df0411 + QUAD $0xdf54117da1c420df + QUAD $0x60df5c117da1c440 + QUAD $0x80ea834910598d48 + QUAD $0x8f0fc33948cb8949 + QUAD $0xc5d9894800000226 + QUAD $0x57f1c5db31c057f9 + QUAD $0x57e1c5d257e9c5c9 + QUAD $0x90817401f98349db + QUAD $0xc1c4da24197de2c4 + QUAD $0x58fdc598da6c595d + QUAD $0xb8da6c595dc1c4c5 + QUAD $0xc5d8da74595dc1c4 + QUAD $0xc4d658edc5cd58f5 + QUAD $0xe5c5f8da64595dc1 + QUAD $0xda64197de2c4dc58 + QUAD $0xa0da6c595dc1c408 + QUAD $0x595dc1c4c558fdc5 + QUAD $0xc4cd58f5c5c0da6c + QUAD $0xc1c4e0da6c595dc1 + QUAD $0xd558edc5da24595d + QUAD $0x02c38348dc58e5c5 + QUAD $0xc1f6419375d83949 + QUAD $0xe9ffffff09850f01 + QUAD $0xf0488d48ffffff39 + QUAD $0x4904eac149ca8949 + QUAD $0xe08341d08945c2ff + QUAD $0x410d7370f9834807 + QUAD $0xe9c93100000010ba + QUAD $0xf8e2834900000136 + QUAD $0x9066c057f9c5c931 + QUAD $0x11fdc5cf0411fdc5 + QUAD $0xcf4411fdc520cf44 + QUAD $0xc560cf4411fdc540 + QUAD $0x00000080cf8411fd + QUAD $0x0000a0cf8411fdc5 + QUAD $0x00c0cf8411fdc500 + QUAD $0xe0cf8411fdc50000 + QUAD $0xcf8411fdc5000000 + QUAD $0x8411fdc500000100 + QUAD $0x11fdc500000120cf + QUAD $0xfdc500000140cf84 + QUAD $0xc500000160cf8411 + QUAD $0x00000180cf8411fd + QUAD $0x0001a0cf8411fdc5 + QUAD $0x01c0cf8411fdc500 + QUAD $0xe0cf8411fdc50000 + QUAD $0xcf8411fdc5000001 + QUAD $0x8411fdc500000200 + QUAD $0x11fdc500000220cf + QUAD $0xfdc500000240cf84 + QUAD $0xc500000260cf8411 + QUAD $0x00000280cf8411fd + QUAD $0x0002a0cf8411fdc5 + QUAD $0x02c0cf8411fdc500 + QUAD $0xe0cf8411fdc50000 + QUAD $0xcf8411fdc5000002 + QUAD $0x8411fdc500000300 + QUAD $0x11fdc500000320cf + QUAD $0xfdc500000340cf84 + QUAD $0xc500000360cf8411 + QUAD $0x00000380cf8411fd + QUAD $0x0003a0cf8411fdc5 + QUAD $0x03c0cf8411fdc500 + QUAD $0xe0cf8411fdc50000 + QUAD $0x4980e98348000003 + QUAD $0xfffedf850ff8c283 + QUAD $0x8d4c3474c0854dff + QUAD $0x9066c057f9c51051 + QUAD $0x11fdc5cf0411fdc5 + QUAD $0xcf4411fdc520cf44 + QUAD $0x4c60cf4411fdc540 + QUAD $0xff4910c28349d189 + QUAD $0x4cf0c28349dd75c8 + QUAD $0xc82949c08949d189 + QUAD $0x854d000001ea8e0f + QUAD $0x450000012e8e0fc9 + QUAD $0xba4907e08341c889 + QUAD $0x7ffffffffffffff8 + QUAD $0x49ce1c8d4cca214d + QUAD $0xebce348d4838c383 + QUAD $0x2e66666666666628 + QUAD $0x0000000000841f0f + QUAD $0xc1ff48cf0411fbc5 + QUAD $0x08c6834808c38349 + QUAD $0x000197840fc13948 + QUAD $0x49db31c057f9c500 + QUAD $0x0000a3820f08f983 + QUAD $0x00000000801f0f00 + QUAD $0xc4c8db4c107bc1c4 + QUAD $0xf3c5d0db54107bc1 QUAD $0xda5459ebc5da0c59 QUAD $0x7bc1c4c158fbc508 - QUAD $0x4c59f3c5d8da4c10 + QUAD $0x4c59f3c5d8db4c10 QUAD $0xc1c4c258fbc510da - QUAD $0x59ebc5e0da54107b + QUAD $0x59ebc5e0db54107b QUAD $0xc4c158fbc518da54 - QUAD $0xf3c5e8da4c107bc1 + QUAD $0xf3c5e8db4c107bc1 QUAD $0xc258fbc520da4c59 - QUAD $0xc5f0da54107bc1c4 + QUAD $0xc5f0db54107bc1c4 QUAD $0x58fbc528da5459eb - QUAD $0xf8da4c107bc1c4c1 + QUAD $0xf8db4c107bc1c4c1 QUAD $0xfbc530da4c59f3c5 - QUAD $0xda14107bc1c4c258 + QUAD $0xdb14107bc1c4c258 QUAD $0xfbc538da5459ebc5 QUAD $0x8348c258fbc5c158 - QUAD $0x6d850fd8394908c3 - QUAD $0x840fc98548ffffff + QUAD $0x6d850fda394908c3 + QUAD $0x840fc0854dffffff QUAD $0xda348d4cffffff34 QUAD $0x66ff3145de1c8d48 QUAD $0x0000000000841f0f QUAD $0x81c4fb0c107ba1c4 QUAD $0xc158fbc5fe0c5973 - QUAD $0xe875f9394cc7ff49 - QUAD $0x658d48ffffff03e9 - QUAD $0xc35d5f415e415be8 - QUAD $0xc931047304f88348 - QUAD $0xfffffff0ba487eeb - QUAD $0x10f883487fffffff - QUAD $0x894843ebc9310473 - QUAD $0xf8c5f631d12148c1 - QUAD $0x2e6666666666c057 - QUAD $0x0000000000841f0f - QUAD $0x11fcc5f70411fcc5 - QUAD $0xf74411fcc520f744 - QUAD $0x4860f74411fcc540 - QUAD $0xe075f1394810c683 - QUAD $0x740ca83b74c83948 - QUAD $0x0cc28348ce894827 - QUAD $0xf8c5c12148d18948 - QUAD $0x0000441f0f66c057 - QUAD $0xc68348f70411fcc5 - QUAD $0x0bebf275f1394804 - QUAD $0x00000000cf04c748 - QUAD $0xf075c83948c1ff48 - LONG $0xc377f8c5 + QUAD $0xe875f8394dc7ff49 + QUAD $0xf88349ffffff03e9 + QUAD $0x95e9c88949087304 + QUAD $0x7310f88349000000 + QUAD $0x83c68949ebd23104 + QUAD $0xf22948c2894c0fe6 + QUAD $0xc5f0e18349c18949 + QUAD $0x9066ca8949c057f9 + QUAD $0xa1c4d704117da1c4 + QUAD $0x7da1c420d744117d + QUAD $0x117da1c440d74411 + QUAD $0x4d10c2834960d744 + QUAD $0x74f68548dc75d139 + QUAD $0xc689607204fe8357 + QUAD $0x0149f0294903e683 + QUAD $0x48c18948ca0148c8 + QUAD $0x66c057f9c5fce183 + QUAD $0x0000000000841f0f + QUAD $0xc28348d70411fdc5 + QUAD $0x8548f275d1394804 + QUAD $0x2e666666661d74f6 + QUAD $0x0000000000841f0f + QUAD $0x00000000c704c74a + QUAD $0xf075c0394cc0ff49 + QUAD $0x415e415be8658d48 + QUAD $0x0148c377f8c55d5f + LONG $0xc88949d1 + BYTE $0xeb + BYTE $0xda RET // func tileFloat32AVX2(dst []float32, pattern []float32) @@ -2967,21 +3533,25 @@ TEXT ·movingAverageFloat32AVX2(SB), NOSPLIT|NOFRAME, $0-56 MOVQ width+48(FP), R8 QUAD $0x294cc09e0fc0854d QUAD $0xc10841c19c0f41c1 - QUAD $0x8d48000000e3850f - QUAD $0x4e0f48ca39480141 - QUAD $0x00cf8e0fc08548c2 - QUAD $0x4853e58948550000 - QUAD $0xc02a82c1c4f8e483 - QUAD $0x83c28944ff488d49 - QUAD $0xfffffff8b94907e2 - QUAD $0x4cc1214d7fffffff - QUAD $0x22ebd231451c468d - QUAD $0x0000000000841f0f - QUAD $0x117aa1c4c85ef2c5 - QUAD $0xc08349c2ff49970c - QUAD $0xc2394904c6834804 - QUAD $0x3145c957f0c57974 - QUAD $0x664e7207f98348db + QUAD $0x48550000010b850f + QUAD $0x554156415741e589 + QUAD $0x48f8e48348535441 + QUAD $0xfffffffffffff8b8 + QUAD $0xca394801498d4c7f + QUAD $0x10f98349ca4e0f4c + QUAD $0xc931000000df8d0f + QUAD $0x0000c28d0fc9394c + QUAD $0x8d49c02a82c1c400 + QUAD $0xe28341c28945ff50 + QUAD $0x8e048d4cc0214c07 + QUAD $0x8e348d481cc08349 + QUAD $0x2e666666666627eb + QUAD $0x0000000000841f0f + QUAD $0x0c11fac5c85ef2c5 + QUAD $0x04c08349c1ff488f + QUAD $0x74c9394c04c68348 + QUAD $0xdb3145c957f0c57a + QUAD $0x2e664f7207fa8348 QUAD $0x0000000000841f0f QUAD $0xc4e4984c587281c4 QUAD $0x81c4e8984c587281 @@ -2990,14 +3560,54 @@ TEXT ·movingAverageFloat32AVX2(SB), NOSPLIT|NOFRAME, $0-56 QUAD $0x4c587281c4f4984c QUAD $0x984c587281c4f898 QUAD $0x49980c587281c4fc - QUAD $0xc075d9394d08c383 - QUAD $0x1c8d4e8b74d28548 + QUAD $0xc075d8394c08c383 + QUAD $0x1c8d4e8b74d2854d QUAD $0x0000441f0fdb319e QUAD $0xff489b0c5872c1c4 - QUAD $0x6de9f275da3948c3 - QUAD $0x5bf8658d48ffffff - BYTE $0x5d - BYTE $0xc3 + QUAD $0x6de9f275da3949c3 + QUAD $0x5bd8658d48ffffff + QUAD $0x5f415e415d415c41 + QUAD $0x82c1c4c377f8c55d + QUAD $0x49c0187de2c4c02a + QUAD $0x8341c28945ff508d + QUAD $0xc32149c3894d07e2 + QUAD $0x0010bc413c5e8d48 + QUAD $0xf68949ff31450000 + QUAD $0x2e66666666663beb + QUAD $0x0000000000841f0f + QUAD $0xd05eecc5c85ef4c5 + QUAD $0xa1c4bf0c117ca1c4 + QUAD $0x618d4c20bf54117c + QUAD $0xc6834940c3834810 + QUAD $0x0fcc394dcf894940 + QUAD $0xe1894cfffffebb8f + QUAD $0xc5e43145c957f0c5 + QUAD $0x0f07fa8348d257e8 + QUAD $0x2e66660000008c82 + QUAD $0x0000000000841f0f + QUAD $0xc4c4a34c5874a1c4 + QUAD $0xa1c4e4a354586ca1 + QUAD $0x6ca1c4c8a34c5874 + QUAD $0x5874a1c4e8a35458 + QUAD $0x54586ca1c4cca34c + QUAD $0xa34c5874a1c4eca3 + QUAD $0xf0a354586ca1c4d0 + QUAD $0xc4d4a34c5874a1c4 + QUAD $0xa1c4f4a354586ca1 + QUAD $0x6ca1c4d8a34c5874 + QUAD $0x5874a1c4f8a35458 + QUAD $0x54586ca1c4dca34c + QUAD $0xa34c5874a1c4fca3 + QUAD $0x49a314586ca1c4e0 + QUAD $0x8875e3394d08c483 + QUAD $0xffff2f840fd2854d + QUAD $0xed3145a6248d4fff + QUAD $0x0000000000841f0f + QUAD $0x81c4ac0c587481c4 + QUAD $0xc5ff4920ac54586c + QUAD $0xff06e9eb75ea394d + BYTE $0xff + BYTE $0xff RET // func shiftDivFloat32AVX2(dst []float32, a []float32, shift float32, denom float32) @@ -3082,21 +3692,25 @@ TEXT ·movingAverageFloat64AVX2(SB), NOSPLIT|NOFRAME, $0-56 MOVQ width+48(FP), R8 QUAD $0x294cc09e0fc0854d QUAD $0xc10841c19c0f41c1 - QUAD $0x8d48000000e3850f - QUAD $0x4e0f48ca39480141 - QUAD $0x00cf8e0fc08548c2 - QUAD $0x4853e58948550000 - QUAD $0xc02a83c1c4f8e483 - QUAD $0x83c28944ff488d49 - QUAD $0xfffffff8b94907e2 - QUAD $0x4cc1214d7fffffff - QUAD $0x22ebd2314538468d - QUAD $0x0000000000841f0f - QUAD $0x117ba1c4c85ef3c5 - QUAD $0xc08349c2ff49d70c - QUAD $0xc2394908c6834808 - QUAD $0x3145c957f1c57974 - QUAD $0x664e7207f98348db + QUAD $0x48550000010b850f + QUAD $0x554156415741e589 + QUAD $0x48f8e48348535441 + QUAD $0xfffffffffffff8b8 + QUAD $0xca394801498d4c7f + QUAD $0x10f98349ca4e0f4c + QUAD $0xc931000000df8d0f + QUAD $0x0000c28d0fc9394c + QUAD $0x8d49c02a83c1c400 + QUAD $0xe28341c28945ff50 + QUAD $0xce048d4cc0214c07 + QUAD $0xce348d4838c08349 + QUAD $0x2e666666666627eb + QUAD $0x0000000000841f0f + QUAD $0x0c11fbc5c85ef3c5 + QUAD $0x08c08349c1ff48cf + QUAD $0x74c9394c08c68348 + QUAD $0xdb3145c957f1c57a + QUAD $0x2e664f7207fa8348 QUAD $0x0000000000841f0f QUAD $0xc4c8d84c587381c4 QUAD $0x81c4d0d84c587381 @@ -3105,14 +3719,58 @@ TEXT ·movingAverageFloat64AVX2(SB), NOSPLIT|NOFRAME, $0-56 QUAD $0x4c587381c4e8d84c QUAD $0xd84c587381c4f0d8 QUAD $0x49d80c587381c4f8 - QUAD $0xc075d9394d08c383 - QUAD $0x1c8d4e8b74d28548 + QUAD $0xc075d8394c08c383 + QUAD $0x1c8d4e8b74d2854d QUAD $0x0000441f0fdb31de QUAD $0xff48db0c5873c1c4 - QUAD $0x6de9f275da3948c3 - QUAD $0x5bf8658d48ffffff - BYTE $0x5d - BYTE $0xc3 + QUAD $0x6de9f275da3949c3 + QUAD $0x5bd8658d48ffffff + QUAD $0x5f415e415d415c41 + QUAD $0x83c1c4c377f8c55d + QUAD $0x49c0197de2c4c02a + QUAD $0x8341c28945ff508d + QUAD $0x214d04588d4c03e2 + QUAD $0x10bc41785e8d48c3 + QUAD $0x8949ff3145000000 + QUAD $0x2e6666666650ebf6 + QUAD $0x0000000000841f0f + QUAD $0xe05eddc5c85ef5c5 + QUAD $0xa1c4ff0c117da1c4 + QUAD $0x5eedc520ff64117d + QUAD $0x7da1c4d05ee5c5c8 + QUAD $0x117da1c440ff4c11 + QUAD $0x4810618d4c60ff54 + QUAD $0x4980ee834980eb83 + QUAD $0xa58f0fcc394dcf89 + QUAD $0xf1c5e1894cfffffe + QUAD $0x57d9c5e43145c957 + QUAD $0x57e1c5d257e9c5e4 + QUAD $0x8e820f03fa8348db + QUAD $0x2e66666666000000 + QUAD $0x0000000000841f0f + QUAD $0xc488e34c5875a1c4 + QUAD $0xa1c4a8e364585da1 + QUAD $0x65a1c4c8e354586d + QUAD $0x5875a1c4e8e35c58 + QUAD $0x64585da1c490e34c + QUAD $0xe354586da1c4b0e3 + QUAD $0xf0e35c5865a1c4d0 + QUAD $0xc498e34c5875a1c4 + QUAD $0xa1c4b8e364585da1 + QUAD $0x65a1c4d8e354586d + QUAD $0x5875a1c4f8e35c58 + QUAD $0x64585da1c4a0e34c + QUAD $0xe354586da1c4c0e3 + QUAD $0x49e31c5865a1c4e0 + QUAD $0x8875e3394d04c483 + QUAD $0xffff0f840fd2854d + QUAD $0xed3145e6248d4fff + QUAD $0x0000000000841f0f + QUAD $0x81c4ec0c587581c4 + QUAD $0x6d81c420ec64585d + QUAD $0x586581c440ec5458 + QUAD $0x394dc5ff4960ec5c + QUAD $0xfffffed8e9dd75ea RET // func shiftDivFloat64AVX2(dst []float64, a []float64, shift float64, denom float64) diff --git a/internal/amd64/numeric_avx512_amd64.s b/internal/amd64/numeric_avx512_amd64.s index 9c397b4..0fd9fb6 100644 --- a/internal/amd64/numeric_avx512_amd64.s +++ b/internal/amd64/numeric_avx512_amd64.s @@ -189,24 +189,90 @@ TEXT ·polyEvalFloat32AVX512(SB), NOSPLIT|NOFRAME, $0-72 MOVQ x_len+32(FP), R8 MOVQ coeffs_len+56(FP), R9 QUAD $0x4dc14c0f4cc1394c - QUAD $0x000000f7840fc985 - QUAD $0x0002538e0fc0854d - QUAD $0xfc8a44107aa1c400 - QUAD $0x0f02e88348c8894c - QUAD $0x894855000000f08c - QUAD $0x8d41f8e4834853e5 - QUAD $0xe1834107e183ff49 - QUAD $0x2e661cebd2314507 + QUAD $0xe58948551a74c985 + QUAD $0xe483485356415741 + QUAD $0x31217d10f88349f8 + QUAD $0x854d00000205e9c0 + QUAD $0x49000004338e0fc0 + QUAD $0x00014a830f08f883 + QUAD $0x00000413e9c03100 + QUAD $0x49fc8a44107aa1c4 + QUAD $0x00010e8e0f01f983 + QUAD $0x518d45fe498d4900 + QUAD $0x41c018487df262ff + QUAD $0x8341cb894507e283 + QUAD $0x00000010be4107e3 + QUAD $0x2e66666623ebdb31 + QUAD $0x0000000000841f0f + QUAD $0x4c9f1411487cf162 + QUAD $0x394dc3894810708d + QUAD $0x4c000001948f0fc6 + QUAD $0x0c10487cf162f089 + QUAD $0x487cf162ce89499e + QUAD $0x257401fb8341d028 + QUAD $0xf162ce8949d7894d + QUAD $0x00401f0fd028487c + QUAD $0xb162d159486cf162 + QUAD $0xceff49b21458586c + QUAD $0xf98348eb75cfff49 + QUAD $0x9008c68349a57207 + QUAD $0xb162d159486cf162 + QUAD $0xf162f8b25458586c + QUAD $0x586cb162d159486c + QUAD $0x486cf162f7b25458 + QUAD $0x5458586cb162d159 + QUAD $0xd159486cf162f6b2 + QUAD $0xf5b25458586cb162 + QUAD $0xb162d159486cf162 + QUAD $0xf162f4b25458586c + QUAD $0x586cb162d159486c + QUAD $0x486cf162f3b25458 + QUAD $0x5458586cb162d159 + QUAD $0xd159486cf162f2b2 + QUAD $0xf1b25458586cb162 + QUAD $0x07fe8349f8c68349 + QUAD $0x49ffffff21e9867f + QUAD $0xc149c28949f0408d + QUAD $0xd18944c2ff4904ea + QUAD $0x7370f8834807e183 + QUAD $0x3100000010ba4125 + QUAD $0xffffc0b9487febc0 + QUAD $0xf883497fffffffff + QUAD $0x310000019b830f40 + QUAD $0x8349000001dde9c0 + QUAD $0xc818487df262f8e2 + QUAD $0x0000441f0f66c031 + QUAD $0x62870c11487cf162 + QUAD $0x6201874c11487cf1 + QUAD $0x6202874c11487cf1 + QUAD $0x6203874c11487cf1 + QUAD $0x6204874c11487cf1 + QUAD $0x6205874c11487cf1 + QUAD $0x6206874c11487cf1 + QUAD $0x4807874c11487cf1 + QUAD $0x75f8c2834980e883 + QUAD $0x8d4c2d74c98548b7 + QUAD $0xc018487df2621050 + QUAD $0x0000441f0f04e1c1 + QUAD $0x4c870411487cf162 + QUAD $0x834810c28349d089 + QUAD $0xf0c28349ec75f0c1 + QUAD $0x2948c1894cd0894c + QUAD $0xc4000002068e0fc1 + QUAD $0x894dfc8a44107aa1 + QUAD $0xcc8c0f02ea8349ca + QUAD $0x83ff498d41000000 + QUAD $0x19eb07e1834107e1 QUAD $0x0000000000841f0f - QUAD $0xff499714117aa1c4 - QUAD $0x00c8840fc2394dc2 - QUAD $0x960c107aa1c40000 - QUAD $0x41d028f8c5c38949 - QUAD $0xcb89481d7401f983 - QUAD $0x90d028f8c5c38949 + QUAD $0xc0ff48871411fac5 + QUAD $0x0001cc840fc0394c + QUAD $0x894d860c10fac500 + QUAD $0xf98341d028f8c5d3 + QUAD $0x894dcb89481f7401 + QUAD $0x001f0fd028f8c5d3 QUAD $0x586aa1c4d259f2c5 QUAD $0xcbff48cbff499a14 - QUAD $0xb87207f88348ee75 + QUAD $0xb87207fa8349ee75 QUAD $0x00401f0f08c38349 QUAD $0x586aa1c4d259f2c5 QUAD $0xc4d259f2c5e09a54 @@ -220,52 +286,51 @@ TEXT ·polyEvalFloat32AVX512(SB), NOSPLIT|NOFRAME, $0-72 QUAD $0x59f2c5c89a54586a QUAD $0xc49a54586aa1c4d2 QUAD $0x07fb8349f8c38349 - QUAD $0x4dffffff49e99e7f - QUAD $0x0000015c8e0fc085 - QUAD $0xc0311b7308f88349 - QUAD $0xf8834900000140e9 - QUAD $0x0129e9c031227308 - QUAD $0x5d5bf8658d480000 - QUAD $0xffffffffc0b948c3 - QUAD $0x7340f883497fffff - QUAD $0xc0b94866ebc0311b - QUAD $0x497fffffffffffff - QUAD $0xe9c0317e7340f883 - QUAD $0x48c0894c000000c7 - QUAD $0xc057f8c5d231c821 - QUAD $0x0000000000841f0f + QUAD $0x48ffffff49e99e7f + QUAD $0xc18948087308f983 + QUAD $0xf983480000010be9 + QUAD $0x00c0e9d231787340 + QUAD $0xc82148c0894c0000 + QUAD $0x9066c057f8c5d231 QUAD $0x62970411487cf162 QUAD $0x6201974411487cf1 QUAD $0x6202974411487cf1 QUAD $0x4803974411487cf1 QUAD $0xd875d0394840c283 - QUAD $0x0000cb840fc03949 - QUAD $0xb2840f38c0f64100 + QUAD $0x0000f1840fc03949 + QUAD $0xd8840f38c0f64100 QUAD $0x8348c28948000000 QUAD $0xc0214cc8894838c1 QUAD $0x00401f0fc057f8c5 QUAD $0xc28348970411fcc5 - QUAD $0x94e9f275d0394808 - QUAD $0x2148c0894c000000 - QUAD $0x31c818487df262c8 - QUAD $0x2e666666666666d2 - QUAD $0x0000000000841f0f - QUAD $0x62970c11487cf162 - QUAD $0x6201974c11487cf1 - QUAD $0x6202974c11487cf1 - QUAD $0x4803974c11487cf1 - QUAD $0xd875d0394840c283 - QUAD $0xc0f6414f74c03949 - QUAD $0x8348c28948307438 - QUAD $0xc0214cc8894838c1 - QUAD $0x2e6666c8187de2c4 - QUAD $0x0000000000841f0f + QUAD $0xbae9f275d0394808 + QUAD $0x8348ca8948000000 + QUAD $0xc818487df262c0e2 + QUAD $0xc0c6814887348d48 + QUAD $0x9066c93145000000 + QUAD $0xfd8e4c11487cb162 + QUAD $0xfe8e4c11487cb162 + QUAD $0xff8e4c11487cb162 + QUAD $0x498e0c11487cb162 + QUAD $0xd875ca394c40c183 + QUAD $0x38c1f65074d13948 + QUAD $0x07e683c689445974 + QUAD $0xe2c4c10148f12948 + QUAD $0x894cc20148c8187d + QUAD $0x001f0ff8e08348c0 QUAD $0xc28348970c11fcc5 - QUAD $0x3949f275d0394808 - QUAD $0x870411fac51974c0 - QUAD $0x8704c7f1ebc0ff48 - QUAD $0x49c0ff4800000000 - QUAD $0xc377f8c5f175c039 + QUAD $0x8548f275d0394808 + QUAD $0x2e666666661a74f6 + QUAD $0x0000000000841f0f + QUAD $0xc1ff488f0411fac5 + QUAD $0x658d48f375c83949 + QUAD $0xc55d5f415e415be8 + QUAD $0x8948d00148c377f8 + QUAD $0x00008704c7ddebc1 + QUAD $0xc03949c0ff480000 + LONG $0xf8c5f175 + BYTE $0x77 + BYTE $0xc3 RET // func convolveFloat32AVX512(dst []float32, sig []float32, ker []float32) @@ -276,101 +341,328 @@ TEXT ·convolveFloat32AVX512(SB), NOSPLIT|NOFRAME, $0-72 MOVQ dst_len+8(FP), CX MOVQ sig_len+32(FP), R8 MOVQ ker_len+56(FP), R9 + // Constant pool appended below the body; the displacements + // above were patched to reach it and the aligned loads made + // unaligned, since nothing promises alignment inside a TEXT. QUAD $0x56415741e5894855 QUAD $0xe483485354415541 - QUAD $0x74894838ec8348f8 - QUAD $0xc0940fc9854d0824 - QUAD $0x40c69c0f40c8294d - QUAD $0x000002af850fc608 - QUAD $0x48c1394c01708d49 - QUAD $0x8e0ff68548f14e0f - QUAD $0x0fc9854d0000029b - QUAD $0x0c8d4a000001f28e - QUAD $0x07e08341c889458a - QUAD $0xfffffffffff8ba49 - QUAD $0x448b48ca214d7fff - QUAD $0x8d4a1c588d4c0824 - QUAD $0x8948fcc083488a04 - QUAD $0x182444c748102444 - QUAD $0xfffebe4900000000 - QUAD $0xbf493fffffffffff - QUAD $0x3ffffffffffffffd - QUAD $0xfffffffffffcbc49 - QUAD $0xfffffffbbd493fff - QUAD $0xfffab8483fffffff - QUAD $0xba483fffffffffff - QUAD $0x3ffffffffffffff9 - QUAD $0xfffffffffff8bb48 - QUAD $0x4830247c89483fff - QUAD $0x244c894c28247489 - QUAD $0x0000441f0f3aeb20 - QUAD $0x4c8b4c30247c8b48 - QUAD $0x8f04117aa1c41824 - QUAD $0x4804c38349c1ff49 - QUAD $0x748b480408244483 - QUAD $0x4918244c894c2824 - QUAD $0x0f20244c8b4cf139 - QUAD $0x57f8c5000001ca84 - QUAD $0x45117308f98349c0 - QUAD $0x66000000dbe9c931 + QUAD $0xc9854d38ec8348f8 + QUAD $0x20244c894cc0940f + QUAD $0x41c19c0f41c8294d + QUAD $0x0000067b850fc108 + QUAD $0x4cc1394c01588d4d + QUAD $0x4810fb8349d94e0f + QUAD $0x245c894c30247c89 + QUAD $0x8948182474894828 + QUAD $0xe9d231077d102454 + QUAD $0x244c8b480000038e + QUAD $0x016a8e0fc9854820 + QUAD $0xc8898a048d4c0000 + QUAD $0xfffff8b94903e083 + QUAD $0xc983497fffffffff + QUAD $0x0c568d4cc9214904 + QUAD $0xfcc283488a148d48 + QUAD $0xed314500000010b9 + QUAD $0xfffffffffffebb48 + QUAD $0xfffffffdbe493fff + QUAD $0xfffcbf493fffffff + QUAD $0x89493fffffffffff + QUAD $0x90662bebd68948f4 + QUAD $0x7cb16230247c8b48 + QUAD $0x104a8d48af041148 + QUAD $0x40c4834940c28349 + QUAD $0x28245c8b4cd58949 + QUAD $0x0003018f0fd9394c + QUAD $0x247c8348ca894800 + QUAD $0xc057f8c50c730420 + QUAD $0x9000000094e9c931 + QUAD $0x31f78948c057f8c5 + QUAD $0x00000000801f0fc9 + QUAD $0xd1620f18487df262 + QUAD $0xfffff48a8c594874 + QUAD $0x49c158487cf162ff + QUAD $0x7d9262db3149cb89 + QUAD $0x4874d162980c1848 + QUAD $0x62fffffff88a8c59 + QUAD $0xcb8949c158487cf1 + QUAD $0x18487d9262f3314d + QUAD $0x8c594874d162980c + QUAD $0x7cf162fffffffc8a + QUAD $0x314dcb8949c15848 + QUAD $0x980c18487d9262fb + QUAD $0x628a0c594874d162 + QUAD $0xc18348c158487cf1 + QUAD $0xc93949f0c7834804 + QUAD $0x2d840fc085488675 + QUAD $0x488c3c8d49ffffff + QUAD $0x2949f3894902e1c1 + QUAD $0x2e66666666c931cb QUAD $0x0000000000841f0f - QUAD $0xc9314510247c8b48 + QUAD $0xf1620b18487dd262 + QUAD $0x7cf1628f0c594874 + QUAD $0x8349c1ff48c15848 + QUAD $0xe9e175c83948fcc3 + QUAD $0xf0438d4dfffffeec + QUAD $0x73000000f0f88149 + QUAD $0xd23100000010b80c + QUAD $0xe8c149000001dce9 + QUAD $0x49c1894dc0ff4904 + QUAD $0xc148c8894cf0e183 + QUAD $0x0510487cf16204e0 + QUAD $0x48c28948000004b8 + QUAD $0x10487cf16210c083 + QUAD $0x57e8c5000004e70d + QUAD $0x191d5948fdf262d2 + QUAD $0x9066ca894d000007 + QUAD $0x497df262c846fdc5 + QUAD $0x62c846fdc58f14a3 + QUAD $0xfdc58714a3497df2 + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc5018f + QUAD $0xfdc5018754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc5028f + QUAD $0xfdc5028754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc5038f + QUAD $0xfdc5038754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc5048f + QUAD $0xfdc5048754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc5058f + QUAD $0xfdc5058754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc5068f + QUAD $0xfdc5068754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc5078f + QUAD $0xfdc5078754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc5088f + QUAD $0xfdc5088754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc5098f + QUAD $0xfdc5098754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc50a8f + QUAD $0xfdc50a8754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc50b8f + QUAD $0xfdc50b8754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc50c8f + QUAD $0xfdc50c8754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc50d8f + QUAD $0xfdc50d8754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc50e8f + QUAD $0xfdc50e8754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc50f8f + QUAD $0xf1620f8754a3497d + QUAD $0x48fdf162cbd448f5 + QUAD $0x850ff0c28349c3d4 + QUAD $0x74c8394dfffffe6c + QUAD $0xd18948c057f8c521 + QUAD $0x488f0411487cf162 + QUAD $0x394c10c08348c189 + QUAD $0x48f0c08348ed7ed8 + QUAD $0x548948d8894cc289 + QUAD $0x548b48d029480824 + QUAD $0x0f1024748b481824 + QUAD $0x4c8b48000002a68e + QUAD $0xdb8e0fc985482024 + QUAD $0x418e048d4c000001 + QUAD $0xbb4907e28341ca89 + QUAD $0x7ffffffffffffff8 + QUAD $0x0824448b48cb2149 + QUAD $0x1cc38348821c8d48 + QUAD $0x4c8948fc8e4c8d48 + QUAD $0x894882048d481024 + QUAD $0xfffffebe49182444 + QUAD $0xfdbf493fffffffff + QUAD $0x493fffffffffffff + QUAD $0xfffffffffffffcbc + QUAD $0xfffffffffbbd493f + QUAD $0xfffffab8483fffff + QUAD $0xf9b9493fffffffff + QUAD $0x483fffffffffffff + QUAD $0xfffffffffffff8ba + QUAD $0x2e6666666639eb3f QUAD $0x0000000000841f0f - QUAD $0xc4e48b4c107a81c4 - QUAD $0xf2c5e88b54107a81 - QUAD $0x894cc158fac50f59 - QUAD $0x0c59eac5f6314cce - QUAD $0x7a81c4c158fac5b1 - QUAD $0x4cce894cec8b4c10 - QUAD $0xc5b10c59f2c5fe31 - QUAD $0x4c107a81c4c158fa - QUAD $0xe6314cce894cf08b - QUAD $0x58fac5b10c59f2c5 - QUAD $0xf48b4c107a81c4c1 - QUAD $0xf2c5ee314cce894c - QUAD $0xc4c158fac5b10c59 - QUAD $0x894cf88b4c107a81 - QUAD $0x0c59f2c5c63148ce - QUAD $0x7a81c4c158fac5b1 - QUAD $0x48ce894cfc8b4c10 - QUAD $0xc5b10c59f2c5d631 - QUAD $0x0c107a81c4c158fa - QUAD $0xc5de3148ce894c8b - QUAD $0xc158fac5b10c59f2 - QUAD $0xe0c7834808c18349 - QUAD $0xffff47850fca394d - QUAD $0xfede840fc0854dff - QUAD $0x00008d348d4affff - QUAD $0x4810247c8b480000 - QUAD $0x4e0824748b48f729 - QUAD $0x001f0ff6318e0c8d - QUAD $0xf2c5b10c107ac1c4 - QUAD $0xff48c158fac50f59 - QUAD $0xf03949fcc78348c6 - QUAD $0x48fffffea1e9e675 - QUAD $0xe9c031077308fe83 - QUAD $0xfff8b94800000084 - QUAD $0x83487fffffffffff - QUAD $0x43ebc031047340fe - QUAD $0x31f02148c8418d48 - QUAD $0x001f0fc057f8c5d2 - QUAD $0x62970411487cf162 - QUAD $0x6201974411487cf1 - QUAD $0x6202974411487cf1 - QUAD $0x4803974411487cf1 - QUAD $0xd875d0394840c283 - QUAD $0xc6f6404274c63948 - QUAD $0xf8c5f121482d7438 - QUAD $0x0000441f0f66c057 - QUAD $0xc08348870411fcc5 - QUAD $0x8948f275c1394808 - QUAD $0x2e661974ce3948c8 + QUAD $0x748b4830244c8b48 + QUAD $0x48b10411fac50824 + QUAD $0x834804c38348c6ff + QUAD $0x2474894804182444 + QUAD $0x840f2824743b4808 + QUAD $0x247c8348000001df + QUAD $0xc057f8c50c730820 + QUAD $0x90000000d2e9c931 + QUAD $0x247c8b48c057f8c5 + QUAD $0x0000441f0fc93110 + QUAD $0xfac5e48b4c10fac5 + QUAD $0x0f59f2c5e88b5410 + QUAD $0x4cce8948c158fac5 + QUAD $0xb00c596ac1c4f631 + QUAD $0x4c10fac5c158fac5 + QUAD $0xfe314cce8948ec8b + QUAD $0xfac5b00c5972c1c4 + QUAD $0xf08b4c10fac5c158 + QUAD $0xc1c4e6314cce8948 + QUAD $0xc158fac5b00c5972 + QUAD $0x8948f48b4c10fac5 + QUAD $0x5972c1c4ee314cce + QUAD $0xfac5c158fac5b00c + QUAD $0x48ce8948f88b4c10 + QUAD $0xb00c5972c1c4c631 + QUAD $0x4c10fac5c158fac5 + QUAD $0xce314cce8948fc8b + QUAD $0xfac5b00c5972c1c4 + QUAD $0x488b0c10fac5c158 + QUAD $0x72c1c4d63148ce89 + QUAD $0x48c158fac5b00c59 + QUAD $0x49e0c7834808c183 + QUAD $0xffffff48850fcb39 + QUAD $0xfffeef840fd2854d + QUAD $0x0000008d348d48ff + QUAD $0x294810247c8b4800 + QUAD $0x8d481824748b48f7 + QUAD $0x00401f0ff6318e0c + QUAD $0x59f2c5b10c10fac5 + QUAD $0xc6ff48c158fac50f + QUAD $0x75f23949fcc78348 + QUAD $0x8348fffffeb2e9e7 + QUAD $0x00000098820f08f8 + QUAD $0x24448b4c40f88348 + QUAD $0x484febc931047308 + QUAD $0x8d4ac0e18348c189 + QUAD $0x0000c0c281488714 + QUAD $0x90c057f8c5f63100 + QUAD $0xfdb24411487cf162 + QUAD $0xfeb24411487cf162 + QUAD $0xffb24411487cf162 + QUAD $0x48b20411487cf162 + QUAD $0xd875f1394840c683 + QUAD $0x7438a85e74c83948 + QUAD $0x89480824448b4c6c + QUAD $0x348d4af8e28348c2 + QUAD $0x2444894cd0014987 + QUAD $0x001f0fc057f8c508 + QUAD $0xc183488e0411fcc5 + QUAD $0x3948f275ca394808 + QUAD $0x2e666666662874d0 QUAD $0x0000000000841f0f - QUAD $0x48000000008704c7 - QUAD $0x48f175c63948c0ff - QUAD $0x5d415c415bd8658d - QUAD $0x77f8c55d5f415e41 - BYTE $0xc3 + QUAD $0x8704c70824448b48 + QUAD $0x48c0ff4800000000 + QUAD $0x2444394808244489 + QUAD $0x5bd8658d48e57528 + QUAD $0x5f415e415d415c41 + QUAD $0x4c0148c377f8c55d + QUAD $0x00000080cceb0824 + QUAD $0x0000009000000000 + QUAD $0x000000a000000000 + QUAD $0x000000b000000000 + QUAD $0x000000c000000000 + QUAD $0x000000d000000000 + QUAD $0x000000e000000000 + QUAD $0x000000f000000000 + QUAD $0x0000000000000000 + QUAD $0x0000001000000000 + QUAD $0x0000002000000000 + QUAD $0x0000003000000000 + QUAD $0x0000004000000000 + QUAD $0x0000005000000000 + QUAD $0x0000006000000000 + QUAD $0x0000007000000000 + QUAD $0x0000000000000000 + QUAD $0x0000001000000000 + QUAD $0x0000002000000000 + QUAD $0x0000003000000000 + QUAD $0x0000004000000000 + QUAD $0x0000005000000000 + QUAD $0x0000006000000000 + QUAD $0x0000007000000000 + QUAD $0x0000008000000000 + QUAD $0x0000009000000000 + QUAD $0x000000a000000000 + QUAD $0x000000b000000000 + QUAD $0x000000c000000000 + QUAD $0x000000d000000000 + QUAD $0x000000e000000000 + QUAD $0x000000f000000000 + QUAD $0x0000000000000000 + QUAD $0x0000001000000000 + QUAD $0x0000002000000000 + QUAD $0x0000003000000000 + QUAD $0x0000004000000000 + QUAD $0x0000005000000000 + QUAD $0x0000006000000000 + QUAD $0x0000007000000000 + QUAD $0x0000000000000000 + QUAD $0x0000001000000000 + QUAD $0x0000002000000000 + QUAD $0x0000003000000000 + QUAD $0x0000004000000000 + QUAD $0x0000005000000000 + QUAD $0x0000006000000000 + QUAD $0x0000007000000000 + QUAD $0x0000000000000000 + QUAD $0x0000000100000000 + QUAD $0x0000000200000000 + QUAD $0x0000000300000000 + QUAD $0x0000000400000000 + QUAD $0x0000000500000000 + QUAD $0x0000000600000000 + QUAD $0x0000000700000000 + QUAD $0x0000000000000000 + QUAD $0x0000000100000000 + QUAD $0x0000000200000000 + QUAD $0x0000000300000000 + QUAD $0x0000000400000000 + QUAD $0x0000000500000000 + QUAD $0x0000000600000000 + QUAD $0x0000000700000000 + QUAD $0x0000000800000000 + QUAD $0x0000000900000000 + QUAD $0x0000000a00000000 + QUAD $0x0000000b00000000 + QUAD $0x0000000c00000000 + QUAD $0x0000000d00000000 + QUAD $0x0000000e00000000 + QUAD $0x0000000f00000000 + QUAD $0x0000000000000000 + QUAD $0x0000000100000000 + QUAD $0x0000000200000000 + QUAD $0x0000000300000000 + QUAD $0x0000000400000000 + QUAD $0x0000000500000000 + QUAD $0x0000000600000000 + QUAD $0x0000000700000000 + QUAD $0x0000010000000000 + QUAD $0x0000008000000000 + QUAD $0x0000010000000000 + QUAD $0x0000008000000000 + QUAD $0xfa53e0a800000000 + QUAD $0xf4a7c150f1bbcdcb + QUAD $0xeefba1f8e3779b97 + QUAD $0x7f4a7c15d5336963 + QUAD $0x1ce4e5b99e3779b9 + QUAD $0x133111ebbf58476d + QUAD $0x0000002094d049bb + QUAD $0x0000000800000000 + QUAD $0x7f4a7c1500000000 + QUAD $0x1ce4e5b99e3779b9 + QUAD $0x133111ebbf58476d + QUAD $0x0000000094d049bb + QUAD $0x000000083ca00000 + QUAD $0x7f4a7c1500000000 + QUAD $0x1ce4e5b99e3779b9 + QUAD $0x133111ebbf58476d + QUAD $0x0000001094d049bb + QUAD $0x0000000400000000 + LONG $0x00000000 RET // func correlateFloat32AVX512(dst []float32, sig []float32, ker []float32) @@ -381,75 +673,302 @@ TEXT ·correlateFloat32AVX512(SB), NOSPLIT|NOFRAME, $0-72 MOVQ dst_len+8(FP), CX MOVQ sig_len+32(FP), R8 MOVQ ker_len+56(FP), R9 + // Constant pool appended below the body; the displacements + // above were patched to reach it and the aligned loads made + // unaligned, since nothing promises alignment inside a TEXT. QUAD $0x294dc0940fc9854d QUAD $0xc20841c29c0f41c8 - QUAD $0x8d49000001f9850f - QUAD $0x4e0f48c1394c0140 - QUAD $0x01e58e0fc08548c1 - QUAD $0x358e0fc9854d0000 - QUAD $0x41e5894855000001 - QUAD $0xf8e4834853564157 - QUAD $0xb84907e183c98944 + QUAD $0x4855000005c7850f + QUAD $0x554156415741e589 + QUAD $0x49f8e48348535441 + QUAD $0x0f48c1394c01408d + QUAD $0x077d10f88348c14e + QUAD $0x4d0000039be9c931 + QUAD $0x000001798e0fc985 + QUAD $0x4907e08341c88945 + QUAD $0xfffffffffffff8ba + QUAD $0x1c5e8d4cca214d7f + QUAD $0x314500000010bf41 + QUAD $0x906621ebf38948f6 + QUAD $0x4cb70411487cb162 + QUAD $0x4840c3834910798d + QUAD $0x3949ce894940c383 + QUAD $0x4c0000034b8f0fc7 + QUAD $0x3145c057f8c5f989 + QUAD $0xe6820f08f98349ff + QUAD $0x2e66666666000000 + QUAD $0x0000000000841f0f + QUAD $0x62ba0c18487db262 + QUAD $0xffe4bb8c59487491 + QUAD $0x5418487db262ffff + QUAD $0xc158487cf16201ba + QUAD $0xe8bb8c59486c9162 + QUAD $0x58487cf162ffffff + QUAD $0xba4c18487db262c1 + QUAD $0xbb8c594874916202 + QUAD $0x487db262ffffffec + QUAD $0x486c916203ba5418 + QUAD $0x62fffffff0bb9459 + QUAD $0x7cf162c158487cf1 + QUAD $0x18487db262c25848 + QUAD $0x594874916204ba4c + QUAD $0xf162fffffff4bb8c + QUAD $0x487db262c158487c + QUAD $0x4874916205ba4c18 + QUAD $0x62fffffff8bb8c59 + QUAD $0x7db262c158487cf1 + QUAD $0x74916206ba4c1848 + QUAD $0xfffffffcbb8c5948 + QUAD $0x07ba5418487db262 + QUAD $0x9162c158487cf162 + QUAD $0x7cf162bb0c59486c + QUAD $0x4d08c78349c15848 + QUAD $0xffffff30850ffa39 + QUAD $0xfffee7840fc0854d + QUAD $0x3c8d4ebb248d4eff + QUAD $0x2e666666ed3145ba + QUAD $0x0000000000841f0f + QUAD $0x62af0c18487d9262 + QUAD $0xf162ac0c59487491 + QUAD $0x4dc5ff49c158487c + QUAD $0xfffeafe9e475e839 + QUAD $0xfa8149f0508d4cff + QUAD $0xb8410d73000000f0 + QUAD $0xdee9c93100000010 + QUAD $0x4904eac149000001 + QUAD $0xe38349d3894dc2ff + QUAD $0x04e1c148d9894cf0 + QUAD $0x04ac0510487cf162 + QUAD $0xf16210418d4c0000 + QUAD $0x000004de0d10487c + QUAD $0x48fdf262d257e8c5 + QUAD $0x894c000006601d59 + QUAD $0x00000000801f0fdb + QUAD $0x497df262c846fdc5 + QUAD $0x62c846fdc58f14a3 + QUAD $0xfdc58714a3497df2 + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc5018f + QUAD $0xfdc5018754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc5028f + QUAD $0xfdc5028754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc5038f + QUAD $0xfdc5038754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc5048f + QUAD $0xfdc5048754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc5058f + QUAD $0xfdc5058754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc5068f + QUAD $0xfdc5068754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc5078f + QUAD $0xfdc5078754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc5088f + QUAD $0xfdc5088754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc5098f + QUAD $0xfdc5098754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc50a8f + QUAD $0xfdc50a8754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc50b8f + QUAD $0xfdc50b8754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc50c8f + QUAD $0xfdc50c8754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc50d8f + QUAD $0xfdc50d8754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc50e8f + QUAD $0xfdc50e8754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc50f8f + QUAD $0xf1620f8754a3497d + QUAD $0x48fdf162cbd448f5 + QUAD $0x850ff0c38348c3d4 + QUAD $0x74da394dfffffe6c + QUAD $0x001f0fc057f8c521 + QUAD $0x4c8f0411487cf162 + QUAD $0x394910c08349c189 + QUAD $0x4cf0c08349ed7ec0 + QUAD $0xc82949c08949c189 + QUAD $0x854d000001e98e0f + QUAD $0x450000012e8e0fc9 + QUAD $0xba4907e08341c889 QUAD $0x7ffffffffffffff8 - QUAD $0x451c568d4cc8214d - QUAD $0x00401f0f1eebdb31 - QUAD $0xff499f04117aa1c4 - QUAD $0xc6834804c28349c3 - QUAD $0x00e3840fc3394904 - QUAD $0xdb31c057f8c50000 - QUAD $0x00a2820f08f98349 - QUAD $0x0000441f0f660000 - QUAD $0xc4e49a4c107ac1c4 - QUAD $0xf2c5e89a54107ac1 + QUAD $0x498e1c8d4cca214d + QUAD $0xeb8e348d481cc383 + QUAD $0x2e66666666666628 + QUAD $0x0000000000841f0f + QUAD $0xc1ff488f0411fac5 + QUAD $0x04c6834804c38349 + QUAD $0x000196840fc13948 + QUAD $0x49db31c057f8c500 + QUAD $0x0000a3820f08f983 + QUAD $0x00000000801f0f00 + QUAD $0xc4e49b4c107ac1c4 + QUAD $0xf2c5e89b54107ac1 QUAD $0xc5c158fac59a0c59 QUAD $0x58fac5049a4c59ea - QUAD $0xec9a4c107ac1c4c1 + QUAD $0xec9b4c107ac1c4c1 QUAD $0xc1c4089a4c59f2c5 - QUAD $0x59eac5f09a54107a + QUAD $0x59eac5f09b54107a QUAD $0xc5c158fac50c9a54 QUAD $0x4c107ac1c4c258fa - QUAD $0x109a4c59f2c5f49a + QUAD $0x109a4c59f2c5f49b QUAD $0x107ac1c4c158fac5 - QUAD $0x9a4c59f2c5f89a4c + QUAD $0x9a4c59f2c5f89b4c QUAD $0x7ac1c4c158fac514 - QUAD $0x4c59f2c5fc9a4c10 - QUAD $0x9a14107ac1c4189a + QUAD $0x4c59f2c5fc9b4c10 + QUAD $0x9b14107ac1c4189a QUAD $0x4c59eac5c158fac5 QUAD $0x8348c158fac51c9a - QUAD $0x6d850fd8394908c3 - QUAD $0x840fc98548ffffff + QUAD $0x6d850fda394908c3 + QUAD $0x840fc0854dffffff QUAD $0x9a348d4cffffff34 QUAD $0x66ff31459e1c8d48 QUAD $0x0000000000841f0f QUAD $0x81c4bb0c107aa1c4 QUAD $0xc158fac5be0c5972 - QUAD $0xe875f9394cc7ff49 - QUAD $0x658d48ffffff03e9 - QUAD $0xc35d5f415e415be8 - QUAD $0xc931077308f88348 - QUAD $0xf8ba480000008be9 - QUAD $0x487fffffffffffff - QUAD $0xebc931047340f883 - QUAD $0xc12148c84a8d4848 - QUAD $0x2e66c057f8c5f631 - QUAD $0x0000000000841f0f - QUAD $0x62b70411487cf162 - QUAD $0x6201b74411487cf1 - QUAD $0x6202b74411487cf1 - QUAD $0x4803b74411487cf1 - QUAD $0xd875f1394840c683 - QUAD $0x7438a84274c83948 - QUAD $0xc057f8c5c221482f + QUAD $0xe875f8394dc7ff49 + QUAD $0xf88349ffffff03e9 + QUAD $0x4900000099820f08 + QUAD $0xebd231047340f883 + QUAD $0xc0e28348c2894c52 + QUAD $0xc0c681488f348d48 + QUAD $0xf8c5c93145000000 + QUAD $0x0000441f0f66c057 + QUAD $0xfd8e4411487cb162 + QUAD $0xfe8e4411487cb162 + QUAD $0xff8e4411487cb162 + QUAD $0x498e0411487cb162 + QUAD $0xd875ca394c40c183 + QUAD $0xc0f6415274d03949 + QUAD $0x8348c6894c5e7438 + QUAD $0x01488f0c8d4cf8e6 + QUAD $0x2e6666c057f8c5f1 QUAD $0x0000000000841f0f - QUAD $0xc183488f0411fcc5 - QUAD $0x8948f275ca394808 - QUAD $0x2e661974d03948d1 + QUAD $0x83489104117cc1c4 + QUAD $0x49f175d6394808c2 + QUAD $0x2e6666661b74f039 QUAD $0x0000000000841f0f QUAD $0x48000000008f04c7 - QUAD $0xc5f175c83948c1ff - BYTE $0xf8 - BYTE $0x77 - BYTE $0xc3 + QUAD $0x48f175c83948c1ff + QUAD $0x5d415c415bd8658d + QUAD $0x77f8c55d5f415e41 + QUAD $0x0080daebd10148c3 + QUAD $0x0090000000000000 + QUAD $0x00a0000000000000 + QUAD $0x00b0000000000000 + QUAD $0x00c0000000000000 + QUAD $0x00d0000000000000 + QUAD $0x00e0000000000000 + QUAD $0x00f0000000000000 + QUAD $0x0000000000000000 + QUAD $0x0010000000000000 + QUAD $0x0020000000000000 + QUAD $0x0030000000000000 + QUAD $0x0040000000000000 + QUAD $0x0050000000000000 + QUAD $0x0060000000000000 + QUAD $0x0070000000000000 + QUAD $0x0000000000000000 + QUAD $0x0010000000000000 + QUAD $0x0020000000000000 + QUAD $0x0030000000000000 + QUAD $0x0040000000000000 + QUAD $0x0050000000000000 + QUAD $0x0060000000000000 + QUAD $0x0070000000000000 + QUAD $0x0080000000000000 + QUAD $0x0090000000000000 + QUAD $0x00a0000000000000 + QUAD $0x00b0000000000000 + QUAD $0x00c0000000000000 + QUAD $0x00d0000000000000 + QUAD $0x00e0000000000000 + QUAD $0x00f0000000000000 + QUAD $0x0000000000000000 + QUAD $0x0010000000000000 + QUAD $0x0020000000000000 + QUAD $0x0030000000000000 + QUAD $0x0040000000000000 + QUAD $0x0050000000000000 + QUAD $0x0060000000000000 + QUAD $0x0070000000000000 + QUAD $0x0000000000000000 + QUAD $0x0010000000000000 + QUAD $0x0020000000000000 + QUAD $0x0030000000000000 + QUAD $0x0040000000000000 + QUAD $0x0050000000000000 + QUAD $0x0060000000000000 + QUAD $0x0070000000000000 + QUAD $0x0000000000000000 + QUAD $0x0001000000000000 + QUAD $0x0002000000000000 + QUAD $0x0003000000000000 + QUAD $0x0004000000000000 + QUAD $0x0005000000000000 + QUAD $0x0006000000000000 + QUAD $0x0007000000000000 + QUAD $0x0000000000000000 + QUAD $0x0001000000000000 + QUAD $0x0002000000000000 + QUAD $0x0003000000000000 + QUAD $0x0004000000000000 + QUAD $0x0005000000000000 + QUAD $0x0006000000000000 + QUAD $0x0007000000000000 + QUAD $0x0008000000000000 + QUAD $0x0009000000000000 + QUAD $0x000a000000000000 + QUAD $0x000b000000000000 + QUAD $0x000c000000000000 + QUAD $0x000d000000000000 + QUAD $0x000e000000000000 + QUAD $0x000f000000000000 + QUAD $0x0000000000000000 + QUAD $0x0001000000000000 + QUAD $0x0002000000000000 + QUAD $0x0003000000000000 + QUAD $0x0004000000000000 + QUAD $0x0005000000000000 + QUAD $0x0006000000000000 + QUAD $0x0007000000000000 + QUAD $0x0100000000000000 + QUAD $0x0080000000000000 + QUAD $0x0100000000000000 + QUAD $0x0080000000000000 + QUAD $0xe0a8000000000000 + QUAD $0xc150f1bbcdcbfa53 + QUAD $0xa1f8e3779b97f4a7 + QUAD $0x7c15d5336963eefb + QUAD $0xe5b99e3779b97f4a + QUAD $0x11ebbf58476d1ce4 + QUAD $0x002094d049bb1331 + QUAD $0x0008000000000000 + QUAD $0x7c15000000000000 + QUAD $0xe5b99e3779b97f4a + QUAD $0x11ebbf58476d1ce4 + QUAD $0x000094d049bb1331 + QUAD $0x00083ca000000000 + QUAD $0x7c15000000000000 + QUAD $0xe5b99e3779b97f4a + QUAD $0x11ebbf58476d1ce4 + QUAD $0x001094d049bb1331 + QUAD $0x0004000000000000 + LONG $0x00000000 + BYTE $0x00 + BYTE $0x00 RET // func normFloat64AVX512(a []float64) float64 @@ -647,24 +1166,110 @@ TEXT ·polyEvalFloat64AVX512(SB), NOSPLIT|NOFRAME, $0-72 MOVQ x_len+32(FP), R8 MOVQ coeffs_len+56(FP), R9 QUAD $0x4dc14c0f4cc1394c - QUAD $0x000000f7840fc985 - QUAD $0x0002548e0fc0854d - QUAD $0xf8ca44107ba1c400 - QUAD $0x0f02e88348c8894c - QUAD $0x894855000000f08c - QUAD $0x8d41f8e4834853e5 - QUAD $0xe1834107e183ff49 - QUAD $0x2e661cebd2314507 + QUAD $0xe58948551a74c985 + QUAD $0xe483485356415741 + QUAD $0x31217d10f88349f8 + QUAD $0x854d000002ade9c0 + QUAD $0x49000004d48e0fc0 + QUAD $0x0001a1830f04f883 + QUAD $0x000004b3e9c03100 + QUAD $0x49f8ca44107ba1c4 + QUAD $0x0001628e0f01f983 + QUAD $0x518d45fe498d4900 + QUAD $0x41c01948fdf262ff + QUAD $0x8341cb894503e283 + QUAD $0x00000010be4103e3 + QUAD $0x2e6666662bebdb31 QUAD $0x0000000000841f0f - QUAD $0xff49d714117ba1c4 - QUAD $0x00c8840fc2394dc2 - QUAD $0xd60c107ba1c40000 - QUAD $0x41d028f9c5c38949 - QUAD $0xcb89481d7401f983 - QUAD $0x90d028f9c5c38949 + QUAD $0x62df1c1148fdf162 + QUAD $0x4c01df641148fdf1 + QUAD $0x394dc3894810708d + QUAD $0x4c000002348f0fc6 + QUAD $0x0c1048fdf162f089 + QUAD $0xde541048fdf162de + QUAD $0x48fdf162ce894901 + QUAD $0xe02848fdf162d828 + QUAD $0x894d417401fb8341 + QUAD $0x48fdf162ce8949d7 + QUAD $0xe02848fdf162d828 + QUAD $0x0000000000841f0f + QUAD $0x62f22c1948fdb262 + QUAD $0xd5f162d95948e5f1 + QUAD $0x5948ddf162db5848 + QUAD $0x49e45848d5f162e2 + QUAD $0x48d975cfff49ceff + QUAD $0xffff6f820f03f983 + QUAD $0x2e666604c68349ff + QUAD $0x0000000000841f0f + QUAD $0xfcf26c1948fdb262 + QUAD $0xf162d95948e5f162 + QUAD $0x48ddf162db5848d5 + QUAD $0xe45848d5f162e259 + QUAD $0xfbf26c1948fdb262 + QUAD $0xf162d95948e5f162 + QUAD $0x48ddf162db5848d5 + QUAD $0xe45848d5f162e259 + QUAD $0xfaf26c1948fdb262 + QUAD $0xf162d95948e5f162 + QUAD $0x48ddf162db5848d5 + QUAD $0xe45848d5f162e259 + QUAD $0xf9f26c1948fdb262 + QUAD $0xf162d95948e5f162 + QUAD $0x48ddf162db5848d5 + QUAD $0xe45848d5f162e259 + QUAD $0x03fe8349fcc68349 + QUAD $0xcde9ffffff728f0f + QUAD $0x49f0408d49fffffe + QUAD $0xff4904eac149c289 + QUAD $0x4807e183d18944c2 + QUAD $0x10ba41287370f883 + QUAD $0x00cce9c031000000 + QUAD $0xffffffe0b9480000 + QUAD $0x20f883497fffffff + QUAD $0xc031000001e4830f + QUAD $0xe2834900000226e9 + QUAD $0x31c81948fdf262f8 + QUAD $0x2e666666666666c0 + QUAD $0x0000000000841f0f + QUAD $0x62c70c1148fdf162 + QUAD $0x6201c74c1148fdf1 + QUAD $0x6202c74c1148fdf1 + QUAD $0x6203c74c1148fdf1 + QUAD $0x6204c74c1148fdf1 + QUAD $0x6205c74c1148fdf1 + QUAD $0x6206c74c1148fdf1 + QUAD $0x6207c74c1148fdf1 + QUAD $0x6208c74c1148fdf1 + QUAD $0x6209c74c1148fdf1 + QUAD $0x620ac74c1148fdf1 + QUAD $0x620bc74c1148fdf1 + QUAD $0x620cc74c1148fdf1 + QUAD $0x620dc74c1148fdf1 + QUAD $0x620ec74c1148fdf1 + QUAD $0x480fc74c1148fdf1 + QUAD $0x0ff8c2834980e883 + QUAD $0xc98548ffffff7385 + QUAD $0xf26210508d4c3174 + QUAD $0x9004e1c1c01948fd + QUAD $0x62c7041148fdf162 + QUAD $0x4c01c7441148fdf1 + QUAD $0x834810c28349d089 + QUAD $0xf0c28349e475f0c1 + QUAD $0x2948c1894cd0894c + QUAD $0xc4000001fe8e0fc1 + QUAD $0x894df8ca44107ba1 + QUAD $0xc48c0f02ea8349ca + QUAD $0x83ff498d41000000 + QUAD $0x11eb07e1834107e1 + QUAD $0xc0ff48c71411fbc5 + QUAD $0x0001cc840fc0394c + QUAD $0x894dc60c10fbc500 + QUAD $0xf98341d028f9c5d3 + QUAD $0x894dcb89481f7401 + QUAD $0x001f0fd028f9c5d3 QUAD $0x586ba1c4d259f3c5 QUAD $0xcbff48cbff49da14 - QUAD $0xb87207f88348ee75 + QUAD $0xb87207fa8349ee75 QUAD $0x00401f0f08c38349 QUAD $0x586ba1c4d259f3c5 QUAD $0xc4d259f3c5c0da54 @@ -678,52 +1283,51 @@ TEXT ·polyEvalFloat64AVX512(SB), NOSPLIT|NOFRAME, $0-72 QUAD $0x59f3c590da54586b QUAD $0x88da54586ba1c4d2 QUAD $0x07fb8349f8c38349 - QUAD $0x4dffffff49e99e7f - QUAD $0x0000015d8e0fc085 - QUAD $0xc0311b7304f88349 - QUAD $0xf8834900000140e9 - QUAD $0x0129e9c031227304 - QUAD $0x5d5bf8658d480000 - QUAD $0xffffffffe0b948c3 - QUAD $0x7320f883497fffff - QUAD $0xe0b94866ebc0311b - QUAD $0x497fffffffffffff - QUAD $0xe9c0317e7320f883 - QUAD $0x48c0894c000000c7 - QUAD $0xc057f9c5d231c821 - QUAD $0x0000000000841f0f - QUAD $0x62d7041148fdf162 - QUAD $0x6201d7441148fdf1 - QUAD $0x6202d7441148fdf1 - QUAD $0x4803d7441148fdf1 + QUAD $0x48ffffff49e99e7f + QUAD $0xc18948087304f983 + QUAD $0xf983480000010be9 + QUAD $0x00c0e9d231787320 + QUAD $0xc82148c0894c0000 + QUAD $0x9066c057f8c5d231 + QUAD $0x62d70411487cf162 + QUAD $0x6201d74411487cf1 + QUAD $0x6202d74411487cf1 + QUAD $0x4803d74411487cf1 QUAD $0xd875d0394820c283 - QUAD $0x0000cc840fc03949 - QUAD $0xb2840f1cc0f64100 + QUAD $0x0000f2840fc03949 + QUAD $0xd8840f1cc0f64100 QUAD $0x8348c28948000000 QUAD $0xc0214cc889481cc1 - QUAD $0x00401f0fc057f9c5 - QUAD $0xc28348d70411fdc5 - QUAD $0x95e9f275d0394804 - QUAD $0x2148c0894c000000 - QUAD $0x31c81948fdf262c8 - QUAD $0x2e666666666666d2 - QUAD $0x0000000000841f0f - QUAD $0x62d70c1148fdf162 - QUAD $0x6201d74c1148fdf1 - QUAD $0x6202d74c1148fdf1 - QUAD $0x4803d74c1148fdf1 - QUAD $0xd875d0394820c283 - QUAD $0xc0f6415074c03949 - QUAD $0x8348c2894830741c - QUAD $0xc0214cc889481cc1 - QUAD $0x2e6666c8197de2c4 - QUAD $0x0000000000841f0f + QUAD $0x00401f0fc057f8c5 + QUAD $0xc28348d70411fcc5 + QUAD $0xbbe9f275d0394804 + QUAD $0x8348ca8948000000 + QUAD $0xc81948fdf262e0e2 + QUAD $0xc0c68148c7348d48 + QUAD $0x9066c93145000000 + QUAD $0xfdce4c1148fdb162 + QUAD $0xfece4c1148fdb162 + QUAD $0xffce4c1148fdb162 + QUAD $0x49ce0c1148fdb162 + QUAD $0xd875ca394c20c183 + QUAD $0x1cc1f65074d13948 + QUAD $0x03e683c689445974 + QUAD $0xe2c4c10148f12948 + QUAD $0x894cc20148c8197d + QUAD $0x001f0ffce08348c0 QUAD $0xc28348d70c11fdc5 - QUAD $0x3949f275d0394804 - QUAD $0xc70411fbc51a74c0 - QUAD $0x04c748f1ebc0ff48 - QUAD $0xc0ff4800000000c7 - QUAD $0x77f8c5f075c03949 + QUAD $0x8548f275d0394804 + QUAD $0x2e666666661a74f6 + QUAD $0x0000000000841f0f + QUAD $0xc1ff48cf0411fbc5 + QUAD $0x658d48f375c83949 + QUAD $0xc55d5f415e415be8 + QUAD $0x8948d00148c377f8 + QUAD $0x00c704c748ddebc1 + QUAD $0x3949c0ff48000000 + LONG $0xc5f075c0 + BYTE $0xf8 + BYTE $0x77 BYTE $0xc3 RET @@ -735,102 +1339,318 @@ TEXT ·convolveFloat64AVX512(SB), NOSPLIT|NOFRAME, $0-72 MOVQ dst_len+8(FP), CX MOVQ sig_len+32(FP), R8 MOVQ ker_len+56(FP), R9 + // Constant pool appended below the body; the displacements + // above were patched to reach it and the aligned loads made + // unaligned, since nothing promises alignment inside a TEXT. QUAD $0x56415741e5894855 QUAD $0xe483485354415541 - QUAD $0x74894838ec8348f8 - QUAD $0xc0940fc9854d0824 - QUAD $0x40c69c0f40c8294d - QUAD $0x000002b0850fc608 - QUAD $0x48c1394c01708d49 - QUAD $0x8e0ff68548f14e0f - QUAD $0x0fc9854d0000029c - QUAD $0x0c8d4a000001f18e - QUAD $0x07e08341c88945ca - QUAD $0xffffffffffe0ba49 - QUAD $0x214d18ca83497fff - QUAD $0x8d4c0824448b48ca - QUAD $0x8348ca048d4a3858 - QUAD $0x311024448948f8c0 - QUAD $0xfffffffffdbf49db - QUAD $0xfffffcbc491fffff - QUAD $0xfbbd491fffffffff - QUAD $0x481fffffffffffff - QUAD $0xfffffffffffffab8 - QUAD $0xfffffffff9ba481f - QUAD $0xfffff8be491fffff - QUAD $0x7c89481fffffffff - QUAD $0x4c20247489482824 - QUAD $0x906634eb18244c89 - QUAD $0x5c8b4828247c8b48 - QUAD $0x48df0411fbc53024 - QUAD $0x834908c38349c3ff - QUAD $0x4808244c894c08c1 - QUAD $0x4cf339482024748b - QUAD $0x01de840f18244c8b - QUAD $0xc530245c89480000 - QUAD $0x7308f98349c057f9 - QUAD $0x08244c8b4cdb310f - QUAD $0x001f0f000000eae9 - QUAD $0x49db3110247c8b48 - QUAD $0xfffffffffffffeb9 - QUAD $0x2e6666666666661f + QUAD $0xc9854d38ec8348f8 + QUAD $0x28244c894cc0940f + QUAD $0x41c19c0f41c8294d + QUAD $0x0000061a850fc108 + QUAD $0x48c1394c01408d49 + QUAD $0x4810f88348c14e0f + QUAD $0x2444894830247c89 + QUAD $0x8948202474894810 + QUAD $0xf63145087d182454 + QUAD $0x4c8b4800000325e9 + QUAD $0xc78e0fc985482824 + QUAD $0x41ca048d4c000001 + QUAD $0xbb4903e28341ca89 + QUAD $0x7ffffffffffffff8 + QUAD $0x48cb214904cb8349 + QUAD $0x48ca148d48585e8d + QUAD $0x00000010b9f8c283 + QUAD $0xfffffffebf49c031 + QUAD $0xfffdbc491fffffff + QUAD $0xbd491fffffffffff + QUAD $0x1ffffffffffffffc + QUAD $0x0000441f0f6634eb + QUAD $0xfdf16230247c8b48 + QUAD $0x48fdf162c7041148 + QUAD $0x104e8d4901c74c11 + QUAD $0x80ee834880eb8348 + QUAD $0x10244c3b48f0894c + QUAD $0x8949000002948f0f + QUAD $0x730428247c8348ce + QUAD $0xc5c931c057f9c517 + QUAD $0x000000e0e9c957f1 QUAD $0x0000000000841f0f - QUAD $0xc4c8db4c107bc1c4 - QUAD $0xf3c5d0db54107bc1 - QUAD $0x8948c158fbc50f59 - QUAD $0x0c59ebc5ce314cde - QUAD $0x7bc1c4c158fbc5f1 - QUAD $0x4cde8948d8db4c10 - QUAD $0xc5f10c59f3c5fe31 - QUAD $0x4c107bc1c4c158fb - QUAD $0xe6314cde8948e0db - QUAD $0x58fbc5f10c59f3c5 - QUAD $0xe8db4c107bc1c4c1 - QUAD $0xf3c5ee314cde8948 - QUAD $0xc4c158fbc5f10c59 - QUAD $0x8948f0db4c107bc1 - QUAD $0x0c59f3c5c63148de - QUAD $0x7bc1c4c158fbc5f1 - QUAD $0x48de8948f8db4c10 - QUAD $0xc5f10c59f3c5d631 - QUAD $0x0c107bc1c4c158fb - QUAD $0xc5f6314cde8948db - QUAD $0xc158fbc5f10c59f3 - QUAD $0xc0c7834808c38348 - QUAD $0xffff47850fda3949 - QUAD $0x244c8b4cc0854dff - QUAD $0x48fffffec9840f08 - QUAD $0x4800000000dd348d - QUAD $0x49f7294810247c8b - QUAD $0x001f0ff631d91c8d - QUAD $0x59f3c5f30c10fbc5 + QUAD $0x31d78948c057f9c5 + QUAD $0x001f0fc957f1c5c9 + QUAD $0xf162171948fdf262 + QUAD $0xffffa8cb9c5948ed + QUAD $0x62c35848fdf162ff + QUAD $0xffe8cb945948edf1 + QUAD $0xca5848f5f162ffff + QUAD $0x9262f9314dc98949 + QUAD $0xedf162c8141948fd + QUAD $0xffffffb0cb9c5948 + QUAD $0xf162c35848fdf162 + QUAD $0xfffff0cb945948ed + QUAD $0x49ca5848f5f162ff + QUAD $0xfd9262e1314dc989 + QUAD $0x48edf162c8141948 + QUAD $0x62ffffffb8cb9c59 + QUAD $0xedf162c35848fdf1 + QUAD $0xfffffff8cb945948 + QUAD $0x8949ca5848f5f162 + QUAD $0x48fd9262e9314dc9 + QUAD $0x5948edf162c81419 + QUAD $0x5848fdf162ffcb5c + QUAD $0xcb145948edf162c3 + QUAD $0x8348ca5848f5f162 + QUAD $0x3949e0c7834804c1 + QUAD $0x4dffffff41850fcb + QUAD $0xfffffed8840fd285 + QUAD $0x03e1c148ce3c8d48 + QUAD $0xc931c92949d18949 + QUAD $0x0000000000841f0f + QUAD $0xf162111948fdd262 + QUAD $0xfdf162cf1c5948ed + QUAD $0x5948edf162c35848 + QUAD $0x5848f5f16201cf54 + QUAD $0xf8c18349c1ff48ca + QUAD $0xfe8ee9d375ca3949 + QUAD $0x8348f0488d48ffff + QUAD $0x000010b80d7370f9 + QUAD $0x00010ee9f6314500 + QUAD $0xc1ff4804e9c14800 + QUAD $0x49f8e68348ce8948 + QUAD $0x8d4904e6c149f689 + QUAD $0x051048fdf1621046 + QUAD $0xc957f1c500000477 + QUAD $0x0671155948fdf262 + QUAD $0xf78948f889490000 + QUAD $0x49fdd262c846fdc5 + QUAD $0x62c846fdc5c00ca3 + QUAD $0xc501c04ca349fdd2 + QUAD $0xa349fdd262c846fd + QUAD $0x62c846fdc502c04c + QUAD $0xc503c04ca349fdd2 + QUAD $0xa349fdd262c846fd + QUAD $0x62c846fdc504c04c + QUAD $0xc505c04ca349fdd2 + QUAD $0xa349fdd262c846fd + QUAD $0x62c846fdc506c04c + QUAD $0xc507c04ca349fdd2 + QUAD $0xa349fdd262c846fd + QUAD $0x62c846fdc508c04c + QUAD $0xc509c04ca349fdd2 + QUAD $0xa349fdd262c846fd + QUAD $0x62c846fdc50ac04c + QUAD $0xc50bc04ca349fdd2 + QUAD $0xa349fdd262c846fd + QUAD $0x62c846fdc50cc04c + QUAD $0xc50dc04ca349fdd2 + QUAD $0xa349fdd262c846fd + QUAD $0x62c846fdc50ec04c + QUAD $0x620fc04ca349fdd2 + QUAD $0xc78348c2d448fdf1 + QUAD $0x48ffffff31850ff8 + QUAD $0xc52b74c7894cf139 + QUAD $0x1024748b48c057f9 + QUAD $0x62f7041148fdb162 + QUAD $0x4901f7441148fdb1 + QUAD $0x394810c08348c689 + QUAD $0x49f0c08348e57ef0 + QUAD $0x481024748b48c689 + QUAD $0x4c082474894cf089 + QUAD $0x4c2024548b48f029 + QUAD $0x02a88e0f18244c8b + QUAD $0x4828244c8b480000 + QUAD $0x000001ce8e0fc985 + QUAD $0x41ca8941c9048d4d + QUAD $0xfffff8bb4907e283 + QUAD $0xcb21497fffffffff + QUAD $0x1c8d480824448b48 + QUAD $0x4c8d4938c38348c2 + QUAD $0x4818244c8948f8c9 + QUAD $0x2024448948c2048d + QUAD $0xfffffffffffebe49 + QUAD $0xfffffffdbf491fff + QUAD $0xfffcbc491fffffff + QUAD $0xbd491fffffffffff + QUAD $0x1ffffffffffffffb + QUAD $0xfffffffffffab848 + QUAD $0xfffffff9ba481fff + QUAD $0xfff8b9491fffffff + QUAD $0x2ceb1fffffffffff + QUAD $0x748b4830244c8b48 + QUAD $0x48f10411fbc50824 + QUAD $0x834808c38348c6ff + QUAD $0x2474894808202444 + QUAD $0x840f1024743b4808 + QUAD $0x247c8348000001ee + QUAD $0xc057f9c50c730828 + QUAD $0x90000000d2e9c931 + QUAD $0x247c8b48c057f9c5 + QUAD $0x0000441f0fc93118 + QUAD $0xfbc5c8cb4c10fbc5 + QUAD $0x0f59f3c5d0cb5410 + QUAD $0x4cce8948c158fbc5 + QUAD $0xf00c596bc1c4f631 + QUAD $0x4c10fbc5c158fbc5 + QUAD $0xfe314cce8948d8cb + QUAD $0xfbc5f00c5973c1c4 + QUAD $0xe0cb4c10fbc5c158 + QUAD $0xc1c4e6314cce8948 + QUAD $0xc158fbc5f00c5973 + QUAD $0x8948e8cb4c10fbc5 + QUAD $0x5973c1c4ee314cce + QUAD $0xfbc5c158fbc5f00c + QUAD $0x48ce8948f0cb4c10 + QUAD $0xf00c5973c1c4c631 + QUAD $0x4c10fbc5c158fbc5 + QUAD $0xd63148ce8948f8cb + QUAD $0xfbc5f00c5973c1c4 + QUAD $0x48cb0c10fbc5c158 + QUAD $0x73c1c4ce314cce89 + QUAD $0x48c158fbc5f00c59 + QUAD $0x49c0c7834808c183 + QUAD $0xffffff48850fcb39 + QUAD $0xfffeef840fd2854d + QUAD $0x000000cd348d48ff + QUAD $0x294818247c8b4800 + QUAD $0x8d482024748b48f7 + QUAD $0x00401f0ff631ce0c + QUAD $0x59f3c5f10c10fbc5 QUAD $0xc6ff48c158fbc50f - QUAD $0x75f03949f8c78348 - QUAD $0x8348fffffe92e9e7 - QUAD $0x88e9c031077304fe - QUAD $0xffffe0b948000000 - QUAD $0xfe83487fffffffff - QUAD $0x4844ebc031047320 - QUAD $0xc5d231c82148f089 - QUAD $0x0000441f0fc057f9 - QUAD $0x62d7041148fdf162 - QUAD $0x6201d7441148fdf1 - QUAD $0x6202d7441148fdf1 - QUAD $0x4803d7441148fdf1 - QUAD $0xd875d0394820c283 - QUAD $0xc6f6404374c63948 - QUAD $0x8348c2894830741c - QUAD $0xf02148c889481cc1 - QUAD $0x2e666666c057f9c5 + QUAD $0x75f23949f8c78348 + QUAD $0x8348fffffeb2e9e7 + QUAD $0x000000a8820f04f8 + QUAD $0x24448b4c20f88348 + QUAD $0x4854ebc931047308 + QUAD $0x8d4ae0e18348c189 + QUAD $0x0000c0c28148c714 + QUAD $0x90c057f9c5f63100 + QUAD $0xfdf2441148fdf162 + QUAD $0xfef2441148fdf162 + QUAD $0xfff2441148fdf162 + QUAD $0x48f2041148fdf162 + QUAD $0xd875f1394820c683 + QUAD $0x1024748b48c83948 + QUAD $0x8b4c76741ca86874 + QUAD $0x8348c28948082444 + QUAD $0x0149c7348d4afce2 + QUAD $0xf9c5082444894cd0 + QUAD $0x2e6666666666c057 QUAD $0x0000000000841f0f - QUAD $0xc28348d70411fdc5 - QUAD $0x3948f275d0394804 - QUAD $0x00c704c7480d74c6 - QUAD $0xeeebc0ff48000000 - QUAD $0x415c415bd8658d48 - QUAD $0xf8c55d5f415e415d - BYTE $0x77 - BYTE $0xc3 + QUAD $0xc18348ce0411fdc5 + QUAD $0x3948f275ca394804 + QUAD $0x22741024748b48d0 + QUAD $0x0000000000841f0f + QUAD $0x04c7480824448b48 + QUAD $0xc0ff4800000000c7 + QUAD $0xc639480824448948 + QUAD $0x415bd8658d48e675 + QUAD $0x5d5f415e415d415c + QUAD $0x244c0148c377f8c5 + QUAD $0x0000000080cdeb08 + QUAD $0x0000000090000000 + QUAD $0x00000000a0000000 + QUAD $0x00000000b0000000 + QUAD $0x00000000c0000000 + QUAD $0x00000000d0000000 + QUAD $0x00000000e0000000 + QUAD $0x00000000f0000000 + QUAD $0x0000000000000000 + QUAD $0x0000000010000000 + QUAD $0x0000000020000000 + QUAD $0x0000000030000000 + QUAD $0x0000000040000000 + QUAD $0x0000000050000000 + QUAD $0x0000000060000000 + QUAD $0x0000000070000000 + QUAD $0x0000000000000000 + QUAD $0x0000000010000000 + QUAD $0x0000000020000000 + QUAD $0x0000000030000000 + QUAD $0x0000000040000000 + QUAD $0x0000000050000000 + QUAD $0x0000000060000000 + QUAD $0x0000000070000000 + QUAD $0x0000000080000000 + QUAD $0x0000000090000000 + QUAD $0x00000000a0000000 + QUAD $0x00000000b0000000 + QUAD $0x00000000c0000000 + QUAD $0x00000000d0000000 + QUAD $0x00000000e0000000 + QUAD $0x00000000f0000000 + QUAD $0x0000000000000000 + QUAD $0x0000000010000000 + QUAD $0x0000000020000000 + QUAD $0x0000000030000000 + QUAD $0x0000000040000000 + QUAD $0x0000000050000000 + QUAD $0x0000000060000000 + QUAD $0x0000000070000000 + QUAD $0x0000000000000000 + QUAD $0x0000000010000000 + QUAD $0x0000000020000000 + QUAD $0x0000000030000000 + QUAD $0x0000000040000000 + QUAD $0x0000000050000000 + QUAD $0x0000000060000000 + QUAD $0x0000000070000000 + QUAD $0x0000000000000000 + QUAD $0x0000000001000000 + QUAD $0x0000000002000000 + QUAD $0x0000000003000000 + QUAD $0x0000000004000000 + QUAD $0x0000000005000000 + QUAD $0x0000000006000000 + QUAD $0x0000000007000000 + QUAD $0x0000000000000000 + QUAD $0x0000000001000000 + QUAD $0x0000000002000000 + QUAD $0x0000000003000000 + QUAD $0x0000000004000000 + QUAD $0x0000000005000000 + QUAD $0x0000000006000000 + QUAD $0x0000000007000000 + QUAD $0x0000000008000000 + QUAD $0x0000000009000000 + QUAD $0x000000000a000000 + QUAD $0x000000000b000000 + QUAD $0x000000000c000000 + QUAD $0x000000000d000000 + QUAD $0x000000000e000000 + QUAD $0x000000000f000000 + QUAD $0x0000000000000000 + QUAD $0x0000000001000000 + QUAD $0x0000000002000000 + QUAD $0x0000000003000000 + QUAD $0x0000000004000000 + QUAD $0x0000000005000000 + QUAD $0x0000000006000000 + QUAD $0x0000000007000000 + QUAD $0x0000000100000000 + QUAD $0x0000000080000000 + QUAD $0x0000000100000000 + QUAD $0x0000000080000000 + QUAD $0xcbfa53e0a8000000 + QUAD $0x97f4a7c150f1bbcd + QUAD $0x63eefba1f8e3779b + QUAD $0xb97f4a7c15d53369 + QUAD $0x6d1ce4e5b99e3779 + QUAD $0xbb133111ebbf5847 + QUAD $0x000000002094d049 + QUAD $0x0000000008000000 + QUAD $0xb97f4a7c15000000 + QUAD $0x6d1ce4e5b99e3779 + QUAD $0xbb133111ebbf5847 + QUAD $0x000000000094d049 + QUAD $0x00000000083ca000 + QUAD $0xb97f4a7c15000000 + QUAD $0x6d1ce4e5b99e3779 + QUAD $0xbb133111ebbf5847 + QUAD $0x000000001094d049 + QUAD $0x0000000004000000 + BYTE $0x00 + BYTE $0x00 + BYTE $0x00 RET // func correlateFloat64AVX512(dst []float64, sig []float64, ker []float64) @@ -841,73 +1661,277 @@ TEXT ·correlateFloat64AVX512(SB), NOSPLIT|NOFRAME, $0-72 MOVQ dst_len+8(FP), CX MOVQ sig_len+32(FP), R8 MOVQ ker_len+56(FP), R9 + // Constant pool appended below the body; the displacements + // above were patched to reach it and the aligned loads made + // unaligned, since nothing promises alignment inside a TEXT. QUAD $0x294dc0940fc9854d QUAD $0xc20841c29c0f41c8 - QUAD $0x8d49000001fa850f - QUAD $0x4e0f48c1394c0140 - QUAD $0x01e68e0fc08548c1 - QUAD $0x358e0fc9854d0000 - QUAD $0x41e5894855000001 - QUAD $0xf8e4834853564157 - QUAD $0xb84907e183c98944 - QUAD $0x7fffffffffffffe0 - QUAD $0x4cc8214d18c88349 - QUAD $0x1aebdb314538568d - QUAD $0xff49df04117ba1c4 - QUAD $0xc6834808c28349c3 - QUAD $0x00e3840fc3394908 - QUAD $0xdb31c057f9c50000 - QUAD $0x00a2820f08f98349 - QUAD $0x0000441f0f660000 - QUAD $0xc4c8da4c107bc1c4 - QUAD $0xf3c5d0da54107bc1 + QUAD $0x4855000004f8850f + QUAD $0x554156415741e589 + QUAD $0x49f8e48348535441 + QUAD $0x0f48c1394c01408d + QUAD $0x077d10f88348c14e + QUAD $0x4d000002d3e9c931 + QUAD $0x000001778e0fc985 + QUAD $0x4903e08341c88945 + QUAD $0xfffffffffffff8ba + QUAD $0xca214d04ca83497f + QUAD $0x0010bf41585e8d4c + QUAD $0xf38948f631450000 + QUAD $0x2e666666666635eb + QUAD $0x0000000000841f0f + QUAD $0x62f7041148fdb162 + QUAD $0x4c01f74c1148fdb1 + QUAD $0x4880eb834910798d + QUAD $0x3949ce894980eb83 + QUAD $0x4c0000026b8f0fc7 + QUAD $0x3145c057f9c5f989 + QUAD $0xf98349c957f1c5ff + QUAD $0x90000000b7820f04 + QUAD $0x62fa141948fdb262 + QUAD $0xffa8fb9c5948ed91 + QUAD $0xc35848fdf162ffff + QUAD $0xe8fb945948ed9162 + QUAD $0x5848f5f162ffffff + QUAD $0xfa541948fdb262ca + QUAD $0xfb9c5948ed916201 + QUAD $0x48fdf162ffffffb0 + QUAD $0x945948ed9162c358 + QUAD $0xf5f162fffffff0fb + QUAD $0x1948fdb262ca5848 + QUAD $0x5948ed916202fa54 + QUAD $0x9162ffffffb8fb9c + QUAD $0xfffff8fb945948ed + QUAD $0x62c35848fdf162ff + QUAD $0xfdb262ca5848f5f1 + QUAD $0xed916203fa541948 + QUAD $0xfdf162fffb5c5948 + QUAD $0x5948ed9162c35848 + QUAD $0xca5848f5f162fb14 + QUAD $0x0ffa394d04c78349 + QUAD $0xc0854dffffff5385 + QUAD $0x8d4effffff0a840f + QUAD $0x3145fa3c8d4efb24 + QUAD $0x2e666666666666ed + QUAD $0x0000000000841f0f + QUAD $0x62ef141948fd9262 + QUAD $0xf162ec1c5948ed91 + QUAD $0x48ed9162c35848fd + QUAD $0x48f5f16201ec5459 + QUAD $0xe8394dc5ff49ca58 + QUAD $0x4cfffffec1e9d675 + QUAD $0x7370fa8349f0508d + QUAD $0x3100000010b8410d + QUAD $0xc1490000010ee9c9 + QUAD $0xd3894dc2ff4904ea + QUAD $0x48d9894cf8e38349 + QUAD $0x6210418d4c04e1c1 + QUAD $0x00045e051048fdf1 + QUAD $0xfdf262c957f1c500 + QUAD $0x4c000005a8155948 + QUAD $0x0000441f0f66db89 + QUAD $0x49fdf262c846fdc5 + QUAD $0x62c846fdc5c70ca3 + QUAD $0xc501c74ca349fdf2 + QUAD $0xa349fdf262c846fd + QUAD $0x62c846fdc502c74c + QUAD $0xc503c74ca349fdf2 + QUAD $0xa349fdf262c846fd + QUAD $0x62c846fdc504c74c + QUAD $0xc505c74ca349fdf2 + QUAD $0xa349fdf262c846fd + QUAD $0x62c846fdc506c74c + QUAD $0xc507c74ca349fdf2 + QUAD $0xa349fdf262c846fd + QUAD $0x62c846fdc508c74c + QUAD $0xc509c74ca349fdf2 + QUAD $0xa349fdf262c846fd + QUAD $0x62c846fdc50ac74c + QUAD $0xc50bc74ca349fdf2 + QUAD $0xa349fdf262c846fd + QUAD $0x62c846fdc50cc74c + QUAD $0xc50dc74ca349fdf2 + QUAD $0xa349fdf262c846fd + QUAD $0x62c846fdc50ec74c + QUAD $0x620fc74ca349fdf2 + QUAD $0xc38348c2d448fdf1 + QUAD $0x4dffffff31850ff8 + QUAD $0xc057f9c52e74da39 + QUAD $0x0000000000841f0f + QUAD $0x62cf041148fdf162 + QUAD $0x4c01cf441148fdf1 + QUAD $0x394910c08349c189 + QUAD $0x4cf0c08349e57ec0 + QUAD $0xc82949c08949c189 + QUAD $0x854d000001e28e0f + QUAD $0x45000001268e0fc9 + QUAD $0xba4907e08341c889 + QUAD $0x7ffffffffffffff8 + QUAD $0x49ce1c8d4cca214d + QUAD $0xebce348d4838c383 + QUAD $0x00000000801f0f20 + QUAD $0xc1ff48cf0411fbc5 + QUAD $0x08c6834808c38349 + QUAD $0x000197840fc13948 + QUAD $0x49db31c057f9c500 + QUAD $0x0000a3820f08f983 + QUAD $0x00000000801f0f00 + QUAD $0xc4c8db4c107bc1c4 + QUAD $0xf3c5d0db54107bc1 QUAD $0xc5c158fbc5da0c59 QUAD $0x58fbc508da4c59eb - QUAD $0xd8da4c107bc1c4c1 + QUAD $0xd8db4c107bc1c4c1 QUAD $0xc1c410da4c59f3c5 - QUAD $0x59ebc5e0da54107b + QUAD $0x59ebc5e0db54107b QUAD $0xc5c158fbc518da54 QUAD $0x4c107bc1c4c258fb - QUAD $0x20da4c59f3c5e8da + QUAD $0x20da4c59f3c5e8db QUAD $0x107bc1c4c158fbc5 - QUAD $0xda4c59f3c5f0da4c + QUAD $0xda4c59f3c5f0db4c QUAD $0x7bc1c4c158fbc528 - QUAD $0x4c59f3c5f8da4c10 - QUAD $0xda14107bc1c430da + QUAD $0x4c59f3c5f8db4c10 + QUAD $0xdb14107bc1c430da QUAD $0x4c59ebc5c158fbc5 QUAD $0x8348c158fbc538da - QUAD $0x6d850fd8394908c3 - QUAD $0x840fc98548ffffff + QUAD $0x6d850fda394908c3 + QUAD $0x840fc0854dffffff QUAD $0xda348d4cffffff34 QUAD $0x66ff3145de1c8d48 QUAD $0x0000000000841f0f QUAD $0x81c4fb0c107ba1c4 QUAD $0xc158fbc5fe0c5973 - QUAD $0xe875f9394cc7ff49 - QUAD $0x658d48ffffff03e9 - QUAD $0xc35d5f415e415be8 - QUAD $0xc931077304f88348 - QUAD $0xe0ba480000008be9 - QUAD $0x487fffffffffffff - QUAD $0xebc931047320f883 - QUAD $0x31d12148c1894848 - QUAD $0x2e6666c057f8c5f6 + QUAD $0xe875f8394dc7ff49 + QUAD $0xf88349ffffff03e9 + QUAD $0x4900000099820f04 + QUAD $0xebd231047320f883 + QUAD $0xe0e28348c2894c52 + QUAD $0xc0c68148cf348d48 + QUAD $0xf9c5c93145000000 + QUAD $0x0000441f0f66c057 + QUAD $0xfdce441148fdb162 + QUAD $0xfece441148fdb162 + QUAD $0xffce441148fdb162 + QUAD $0x49ce041148fdb162 + QUAD $0xd875ca394c20c183 + QUAD $0xc0f6415374d03949 + QUAD $0x8348c6894c5f741c + QUAD $0x0148cf0c8d4cfce6 + QUAD $0x2e6666c057f9c5f1 QUAD $0x0000000000841f0f - QUAD $0x62f70411487cf162 - QUAD $0x6201f74411487cf1 - QUAD $0x6202f74411487cf1 - QUAD $0x4803f74411487cf1 - QUAD $0xd875f1394820c683 - QUAD $0x741ca84374c83948 - QUAD $0x1cc28348ce89482f - QUAD $0xf8c5c12148d18948 - QUAD $0x2e6666666666c057 + QUAD $0x8348d104117dc1c4 + QUAD $0x49f175d6394804c2 + QUAD $0x2e6666661c74f039 QUAD $0x0000000000841f0f - QUAD $0xc68348f70411fcc5 - QUAD $0x0bebf275f1394804 QUAD $0x00000000cf04c748 QUAD $0xf075c83948c1ff48 - LONG $0xc377f8c5 + QUAD $0x415c415bd8658d48 + QUAD $0xf8c55d5f415e415d + QUAD $0x80d9ebd10148c377 + QUAD $0x9000000000000000 + QUAD $0xa000000000000000 + QUAD $0xb000000000000000 + QUAD $0xc000000000000000 + QUAD $0xd000000000000000 + QUAD $0xe000000000000000 + QUAD $0xf000000000000000 + QUAD $0x0000000000000000 + QUAD $0x1000000000000000 + QUAD $0x2000000000000000 + QUAD $0x3000000000000000 + QUAD $0x4000000000000000 + QUAD $0x5000000000000000 + QUAD $0x6000000000000000 + QUAD $0x7000000000000000 + QUAD $0x0000000000000000 + QUAD $0x1000000000000000 + QUAD $0x2000000000000000 + QUAD $0x3000000000000000 + QUAD $0x4000000000000000 + QUAD $0x5000000000000000 + QUAD $0x6000000000000000 + QUAD $0x7000000000000000 + QUAD $0x8000000000000000 + QUAD $0x9000000000000000 + QUAD $0xa000000000000000 + QUAD $0xb000000000000000 + QUAD $0xc000000000000000 + QUAD $0xd000000000000000 + QUAD $0xe000000000000000 + QUAD $0xf000000000000000 + QUAD $0x0000000000000000 + QUAD $0x1000000000000000 + QUAD $0x2000000000000000 + QUAD $0x3000000000000000 + QUAD $0x4000000000000000 + QUAD $0x5000000000000000 + QUAD $0x6000000000000000 + QUAD $0x7000000000000000 + QUAD $0x0000000000000000 + QUAD $0x1000000000000000 + QUAD $0x2000000000000000 + QUAD $0x3000000000000000 + QUAD $0x4000000000000000 + QUAD $0x5000000000000000 + QUAD $0x6000000000000000 + QUAD $0x7000000000000000 + QUAD $0x0000000000000000 + QUAD $0x0100000000000000 + QUAD $0x0200000000000000 + QUAD $0x0300000000000000 + QUAD $0x0400000000000000 + QUAD $0x0500000000000000 + QUAD $0x0600000000000000 + QUAD $0x0700000000000000 + QUAD $0x0000000000000000 + QUAD $0x0100000000000000 + QUAD $0x0200000000000000 + QUAD $0x0300000000000000 + QUAD $0x0400000000000000 + QUAD $0x0500000000000000 + QUAD $0x0600000000000000 + QUAD $0x0700000000000000 + QUAD $0x0800000000000000 + QUAD $0x0900000000000000 + QUAD $0x0a00000000000000 + QUAD $0x0b00000000000000 + QUAD $0x0c00000000000000 + QUAD $0x0d00000000000000 + QUAD $0x0e00000000000000 + QUAD $0x0f00000000000000 + QUAD $0x0000000000000000 + QUAD $0x0100000000000000 + QUAD $0x0200000000000000 + QUAD $0x0300000000000000 + QUAD $0x0400000000000000 + QUAD $0x0500000000000000 + QUAD $0x0600000000000000 + QUAD $0x0700000000000000 + QUAD $0x0000000000000000 + QUAD $0x8000000000000001 + QUAD $0x0000000000000000 + QUAD $0x8000000000000001 + QUAD $0xa800000000000000 + QUAD $0x50f1bbcdcbfa53e0 + QUAD $0xf8e3779b97f4a7c1 + QUAD $0x15d5336963eefba1 + QUAD $0xb99e3779b97f4a7c + QUAD $0xebbf58476d1ce4e5 + QUAD $0x2094d049bb133111 + QUAD $0x0800000000000000 + QUAD $0x1500000000000000 + QUAD $0xb99e3779b97f4a7c + QUAD $0xebbf58476d1ce4e5 + QUAD $0x0094d049bb133111 + QUAD $0x083ca00000000000 + QUAD $0x1500000000000000 + QUAD $0xb99e3779b97f4a7c + QUAD $0xebbf58476d1ce4e5 + QUAD $0x1094d049bb133111 + QUAD $0x0400000000000000 + LONG $0x00000000 + BYTE $0x00 + BYTE $0x00 + BYTE $0x00 RET // func tileFloat32AVX512(dst []float32, pattern []float32) @@ -3526,21 +4550,25 @@ TEXT ·movingAverageFloat32AVX512(SB), NOSPLIT|NOFRAME, $0-56 MOVQ width+48(FP), R8 QUAD $0x294cc09e0fc0854d QUAD $0xc10841c19c0f41c1 - QUAD $0x8d48000000e3850f - QUAD $0x4e0f48ca39480141 - QUAD $0x00cf8e0fc08548c2 - QUAD $0x4853e58948550000 - QUAD $0xc02a82c1c4f8e483 - QUAD $0x83c28944ff488d49 - QUAD $0xfffffff8b94907e2 - QUAD $0x4cc1214d7fffffff - QUAD $0x22ebd231451c468d + QUAD $0x48550000010b850f + QUAD $0x554156415741e589 + QUAD $0x48f8e48348535441 + QUAD $0xfffffffffffff8b8 + QUAD $0xca394801498d4c7f + QUAD $0x10f98349ca4e0f4c + QUAD $0xc931000000df8d0f + QUAD $0x0000c28d0fc9394c + QUAD $0x8d49c02a82c1c400 + QUAD $0xe28341c28945ff50 + QUAD $0x8e048d4cc0214c07 + QUAD $0x8e348d481cc08349 + QUAD $0x2e666666666627eb QUAD $0x0000000000841f0f - QUAD $0x117aa1c4c85ef2c5 - QUAD $0xc08349c2ff49970c - QUAD $0xc2394904c6834804 - QUAD $0x3145c957f0c57974 - QUAD $0x664e7207f98348db + QUAD $0x0c11fac5c85ef2c5 + QUAD $0x04c08349c1ff488f + QUAD $0x74c9394c04c68348 + QUAD $0xdb3145c957f0c57a + QUAD $0x2e664f7207fa8348 QUAD $0x0000000000841f0f QUAD $0xc4e4984c587281c4 QUAD $0x81c4e8984c587281 @@ -3549,14 +4577,46 @@ TEXT ·movingAverageFloat32AVX512(SB), NOSPLIT|NOFRAME, $0-56 QUAD $0x4c587281c4f4984c QUAD $0x984c587281c4f898 QUAD $0x49980c587281c4fc - QUAD $0xc075d9394d08c383 - QUAD $0x1c8d4e8b74d28548 + QUAD $0xc075d8394c08c383 + QUAD $0x1c8d4e8b74d2854d QUAD $0x0000441f0fdb319e QUAD $0xff489b0c5872c1c4 - QUAD $0x6de9f275da3948c3 - QUAD $0x5bf8658d48ffffff - BYTE $0x5d - BYTE $0xc3 + QUAD $0x6de9f275da3949c3 + QUAD $0x5bd8658d48ffffff + QUAD $0x5f415e415d415c41 + QUAD $0x82c1c4c377f8c55d + QUAD $0xc018487df262c02a + QUAD $0x41c28945ff508d49 + QUAD $0x2149c3894d07e283 + QUAD $0x10bc411c5e8d48c3 + QUAD $0x8949ff3145000000 + QUAD $0x2e6666666632ebf6 + QUAD $0x0000000000841f0f + QUAD $0xb162c85e4874f162 + QUAD $0x618d4cbf0c11487c + QUAD $0xc6834940c3834810 + QUAD $0x0fcc394dcf894940 + QUAD $0xe1894cfffffec38f + QUAD $0x48e43145c957f0c5 + QUAD $0x2e6666717207fa83 + QUAD $0x0000000000841f0f + QUAD $0xe4a38c584874b162 + QUAD $0x584874b162ffffff + QUAD $0xb162ffffffe8a38c + QUAD $0xffffeca38c584874 + QUAD $0xa38c584874b162ff + QUAD $0x4874b162fffffff0 + QUAD $0x62fffffff4a38c58 + QUAD $0xfff8a38c584874b1 + QUAD $0x8c584874b162ffff + QUAD $0x74b162fffffffca3 + QUAD $0x08c48349a30c5848 + QUAD $0xd2854da375e3394d + QUAD $0x8d4fffffff5a840f + QUAD $0x001f0fed3145a624 + QUAD $0x49ac0c5848749162 + QUAD $0xe9f175ea394dc5ff + LONG $0xffffff3c RET // func shiftDivFloat32AVX512(dst []float32, a []float32, shift float32, denom float32) @@ -3671,21 +4731,25 @@ TEXT ·movingAverageFloat64AVX512(SB), NOSPLIT|NOFRAME, $0-56 MOVQ width+48(FP), R8 QUAD $0x294cc09e0fc0854d QUAD $0xc10841c19c0f41c1 - QUAD $0x8d48000000e3850f - QUAD $0x4e0f48ca39480141 - QUAD $0x00cf8e0fc08548c2 - QUAD $0x4853e58948550000 - QUAD $0xc02a83c1c4f8e483 - QUAD $0x83c28944ff488d49 - QUAD $0xfffffff8b94907e2 - QUAD $0x4cc1214d7fffffff - QUAD $0x22ebd2314538468d + QUAD $0x48550000010b850f + QUAD $0x554156415741e589 + QUAD $0x48f8e48348535441 + QUAD $0xfffffffffffff8b8 + QUAD $0xca394801498d4c7f + QUAD $0x10f98349ca4e0f4c + QUAD $0xc931000000df8d0f + QUAD $0x0000c28d0fc9394c + QUAD $0x8d49c02a83c1c400 + QUAD $0xe28341c28945ff50 + QUAD $0xce048d4cc0214c07 + QUAD $0xce348d4838c08349 + QUAD $0x2e666666666627eb QUAD $0x0000000000841f0f - QUAD $0x117ba1c4c85ef3c5 - QUAD $0xc08349c2ff49d70c - QUAD $0xc2394908c6834808 - QUAD $0x3145c957f1c57974 - QUAD $0x664e7207f98348db + QUAD $0x0c11fbc5c85ef3c5 + QUAD $0x08c08349c1ff48cf + QUAD $0x74c9394c08c68348 + QUAD $0xdb3145c957f1c57a + QUAD $0x2e664f7207fa8348 QUAD $0x0000000000841f0f QUAD $0xc4c8d84c587381c4 QUAD $0x81c4d0d84c587381 @@ -3694,14 +4758,61 @@ TEXT ·movingAverageFloat64AVX512(SB), NOSPLIT|NOFRAME, $0-56 QUAD $0x4c587381c4e8d84c QUAD $0xd84c587381c4f0d8 QUAD $0x49d80c587381c4f8 - QUAD $0xc075d9394d08c383 - QUAD $0x1c8d4e8b74d28548 + QUAD $0xc075d8394c08c383 + QUAD $0x1c8d4e8b74d2854d QUAD $0x0000441f0fdb31de QUAD $0xff48db0c5873c1c4 - QUAD $0x6de9f275da3948c3 - QUAD $0x5bf8658d48ffffff - BYTE $0x5d - BYTE $0xc3 + QUAD $0x6de9f275da3949c3 + QUAD $0x5bd8658d48ffffff + QUAD $0x5f415e415d415c41 + QUAD $0x83c1c4c377f8c55d + QUAD $0xc01948fdf262c02a + QUAD $0x41c28945ff508d49 + QUAD $0x2149c3894d07e283 + QUAD $0x10bc41785e8d48c3 + QUAD $0x8949ff3145000000 + QUAD $0x2e6666666640ebf6 + QUAD $0x0000000000841f0f + QUAD $0xb162c85e48f5f162 + QUAD $0xedf162ff0c1148fd + QUAD $0x1148fdb162c85e48 + QUAD $0x4810618d4c01ff4c + QUAD $0x4980ee834980eb83 + QUAD $0xb58f0fcc394dcf89 + QUAD $0xf1c5e1894cfffffe + QUAD $0x57e9c5e43145c957 + QUAD $0xc4820f07fa8348d2 + QUAD $0x0000441f0f000000 + QUAD $0x88e38c5848f5b162 + QUAD $0x5848edb162ffffff + QUAD $0xb162ffffffc8e394 + QUAD $0xffff90e38c5848f5 + QUAD $0xe3945848edb162ff + QUAD $0x48f5b162ffffffd0 + QUAD $0x62ffffff98e38c58 + QUAD $0xffd8e3945848edb1 + QUAD $0x8c5848f5b162ffff + QUAD $0xedb162ffffffa0e3 + QUAD $0xffffffe0e3945848 + QUAD $0xa8e38c5848f5b162 + QUAD $0x5848edb162ffffff + QUAD $0xb162ffffffe8e394 + QUAD $0xffffb0e38c5848f5 + QUAD $0xe3945848edb162ff + QUAD $0x48f5b162fffffff0 + QUAD $0x62ffffffb8e38c58 + QUAD $0xfff8e3945848edb1 + QUAD $0x4c5848f5b162ffff + QUAD $0x145848edb162ffe3 + QUAD $0xe3394d08c48349e3 + QUAD $0x854dffffff4a850f + QUAD $0x4ffffffef1840fd2 + QUAD $0x2e66ed3145e6248d + QUAD $0x0000000000841f0f + QUAD $0x62ec0c5848f59162 + QUAD $0x4901ec545848ed91 + QUAD $0xe9e975ea394dc5ff + LONG $0xfffffec4 RET // func shiftDivFloat64AVX512(dst []float64, a []float64, shift float64, denom float64) @@ -3777,17 +4888,17 @@ TEXT ·randomU64AVX512(SB), NOSPLIT|NOFRAME, $0-32 QUAD $0x8348c67c48fdf262 QUAD $0x93e9c031077320fa QUAD $0xf162d08948000001 - QUAD $0x000002a50dd458fd - QUAD $0x02a315d458fdf162 + QUAD $0x000002c50dd458fd + QUAD $0x02c315d458fdf162 QUAD $0x1dd458fdf1620000 - QUAD $0x62c82148000002a1 - QUAD $0x000314256f48fef1 + QUAD $0x62c82148000002c1 + QUAD $0x0004b4256f48fef1 QUAD $0x48fdf262c0314500 - QUAD $0xf2620000028f2d59 - QUAD $0x0000028d355948fd - QUAD $0x028b3d5948fdf262 + QUAD $0xf262000002af2d59 + QUAD $0x000002ad355948fd + QUAD $0x02ab3d5948fdf262 QUAD $0x055948fd72620000 - QUAD $0x2e66666600000289 + QUAD $0x2e666666000002a9 QUAD $0x0000000000841f0f QUAD $0x7162cd4048dd7262 QUAD $0x48b57162d0d448b5 @@ -3829,12 +4940,12 @@ TEXT ·randomU64AVX512(SB), NOSPLIT|NOFRAME, $0-32 QUAD $0x8949000000ac840f QUAD $0xc8894818c18348c0 QUAD $0x7c48fdd262d02148 - QUAD $0x920deb48f5f162c8 - QUAD $0x5948fdf262000001 - QUAD $0xfdf2620000011015 - QUAD $0x620000010e1d5948 - QUAD $0x00010c255948fdf2 - QUAD $0x122d5948fdf26200 + QUAD $0x320deb48f5f162c8 + QUAD $0x5948fdf262000003 + QUAD $0xfdf2620000013015 + QUAD $0x620000012e1d5948 + QUAD $0x00012c255948fdf2 + QUAD $0x322d5948fdf26200 QUAD $0x2e66666666000001 QUAD $0x0000000000841f0f QUAD $0xf162f24048f5f262 @@ -3862,7 +4973,11 @@ TEXT ·randomU64AVX512(SB), NOSPLIT|NOFRAME, $0-32 QUAD $0x894cda314d1feac1 QUAD $0xce014cc0ff48c714 QUAD $0xec8948cb75c23948 - QUAD $0x53e0a8c377f8c55d + QUAD $0x000100c377f8c55d + QUAD $0x0000800000000000 + QUAD $0x0001000000000000 + QUAD $0x0000800000000000 + QUAD $0x53e0a80000000000 QUAD $0xa7c150f1bbcdcbfa QUAD $0xfba1f8e3779b97f4 QUAD $0x4a7c15d5336963ee @@ -3880,6 +4995,54 @@ TEXT ·randomU64AVX512(SB), NOSPLIT|NOFRAME, $0-32 QUAD $0x3111ebbf58476d1c QUAD $0x00001094d049bb13 QUAD $0x0000040000000000 + QUAD $0x0000800000000000 + QUAD $0x0000900000000000 + QUAD $0x0000a00000000000 + QUAD $0x0000b00000000000 + QUAD $0x0000c00000000000 + QUAD $0x0000d00000000000 + QUAD $0x0000e00000000000 + QUAD $0x0000f00000000000 + QUAD $0x0000000000000000 + QUAD $0x0000100000000000 + QUAD $0x0000200000000000 + QUAD $0x0000300000000000 + QUAD $0x0000400000000000 + QUAD $0x0000500000000000 + QUAD $0x0000600000000000 + QUAD $0x0000700000000000 + QUAD $0x0000000000000000 + QUAD $0x0000100000000000 + QUAD $0x0000200000000000 + QUAD $0x0000300000000000 + QUAD $0x0000400000000000 + QUAD $0x0000500000000000 + QUAD $0x0000600000000000 + QUAD $0x0000700000000000 + QUAD $0x0000800000000000 + QUAD $0x0000900000000000 + QUAD $0x0000a00000000000 + QUAD $0x0000b00000000000 + QUAD $0x0000c00000000000 + QUAD $0x0000d00000000000 + QUAD $0x0000e00000000000 + QUAD $0x0000f00000000000 + QUAD $0x0000000000000000 + QUAD $0x0000100000000000 + QUAD $0x0000200000000000 + QUAD $0x0000300000000000 + QUAD $0x0000400000000000 + QUAD $0x0000500000000000 + QUAD $0x0000600000000000 + QUAD $0x0000700000000000 + QUAD $0x0000000000000000 + QUAD $0x0000100000000000 + QUAD $0x0000200000000000 + QUAD $0x0000300000000000 + QUAD $0x0000400000000000 + QUAD $0x0000500000000000 + QUAD $0x0000600000000000 + QUAD $0x0000700000000000 QUAD $0x0000000000000000 QUAD $0x0000010000000000 QUAD $0x0000020000000000 @@ -3930,14 +5093,14 @@ TEXT ·randomF64AVX512(SB), NOSPLIT|NOFRAME, $0-32 QUAD $0xb848000000c3e9c0 QUAD $0x7ffffffffffffff8 QUAD $0x6f48fef162d02148 - QUAD $0xfdf2620000017605 + QUAD $0xfdf262000002f605 QUAD $0xfdf262c931ce7c48 - QUAD $0x6200000264155948 - QUAD $0x0002621d5948fdf2 - QUAD $0x60255948fdf26200 - QUAD $0x1948fdf262000002 - QUAD $0xfdf2620000025e2d - QUAD $0x900000025c355948 + QUAD $0x6200000404155948 + QUAD $0x0004021d5948fdf2 + QUAD $0x00255948fdf26200 + QUAD $0x1948fdf262000004 + QUAD $0xfdf262000003fe2d + QUAD $0x90000003fc355948 QUAD $0xf162fa4048fdf262 QUAD $0x48bdf162f9d448c5 QUAD $0xef48bdf1621ed773 @@ -3957,7 +5120,7 @@ TEXT ·randomF64AVX512(SB), NOSPLIT|NOFRAME, $0-32 QUAD $0x1ce4e5b9b948ce29 QUAD $0x11ebb849bf58476d QUAD $0xfbc594d049bb1331 - QUAD $0xb949000001b50510 + QUAD $0xb949000003550510 QUAD $0x9e3779b97f4a7c15 QUAD $0x0000000000841f0f QUAD $0x491eeac149f28949 @@ -3969,7 +5132,55 @@ TEXT ·randomF64AVX512(SB), NOSPLIT|NOFRAME, $0-32 QUAD $0xc70c11fbc5c859f3 QUAD $0x3948ce014cc0ff48 QUAD $0xc55dec8948bd75c2 - QUAD $0x0000000000c377f8 + QUAD $0x0000000080c377f8 + QUAD $0x0000000090000000 + QUAD $0x00000000a0000000 + QUAD $0x00000000b0000000 + QUAD $0x00000000c0000000 + QUAD $0x00000000d0000000 + QUAD $0x00000000e0000000 + QUAD $0x00000000f0000000 + QUAD $0x0000000000000000 + QUAD $0x0000000010000000 + QUAD $0x0000000020000000 + QUAD $0x0000000030000000 + QUAD $0x0000000040000000 + QUAD $0x0000000050000000 + QUAD $0x0000000060000000 + QUAD $0x0000000070000000 + QUAD $0x0000000000000000 + QUAD $0x0000000010000000 + QUAD $0x0000000020000000 + QUAD $0x0000000030000000 + QUAD $0x0000000040000000 + QUAD $0x0000000050000000 + QUAD $0x0000000060000000 + QUAD $0x0000000070000000 + QUAD $0x0000000080000000 + QUAD $0x0000000090000000 + QUAD $0x00000000a0000000 + QUAD $0x00000000b0000000 + QUAD $0x00000000c0000000 + QUAD $0x00000000d0000000 + QUAD $0x00000000e0000000 + QUAD $0x00000000f0000000 + QUAD $0x0000000000000000 + QUAD $0x0000000010000000 + QUAD $0x0000000020000000 + QUAD $0x0000000030000000 + QUAD $0x0000000040000000 + QUAD $0x0000000050000000 + QUAD $0x0000000060000000 + QUAD $0x0000000070000000 + QUAD $0x0000000000000000 + QUAD $0x0000000010000000 + QUAD $0x0000000020000000 + QUAD $0x0000000030000000 + QUAD $0x0000000040000000 + QUAD $0x0000000050000000 + QUAD $0x0000000060000000 + QUAD $0x0000000070000000 + QUAD $0x0000000000000000 QUAD $0x0000000001000000 QUAD $0x0000000002000000 QUAD $0x0000000003000000 @@ -4001,6 +5212,10 @@ TEXT ·randomF64AVX512(SB), NOSPLIT|NOFRAME, $0-32 QUAD $0x0000000005000000 QUAD $0x0000000006000000 QUAD $0x0000000007000000 + QUAD $0x0000000100000000 + QUAD $0x0000000080000000 + QUAD $0x0000000100000000 + QUAD $0x0000000080000000 QUAD $0xcbfa53e0a8000000 QUAD $0x97f4a7c150f1bbcd QUAD $0x63eefba1f8e3779b @@ -4040,15 +5255,15 @@ TEXT ·randomF32AVX512(SB), NOSPLIT|NOFRAME, $0-32 QUAD $0xc031077310fa8348 QUAD $0xd089480000011ee9 QUAD $0x6f48fef162c82148 - QUAD $0xfef162000002bc05 - QUAD $0x62000002f20d6f48 + QUAD $0xfef1620000043c05 + QUAD $0x62000004720d6f48 QUAD $0xc03145d67c48fdf2 - QUAD $0x03871d5948fdf262 + QUAD $0x05271d5948fdf262 QUAD $0x255948fdf2620000 - QUAD $0x48fdf26200000385 - QUAD $0xf262000003832d59 - QUAD $0x000003913518487d - QUAD $0x03773d5948fdf262 + QUAD $0x48fdf26200000525 + QUAD $0xf262000005232d59 + QUAD $0x000005313518487d + QUAD $0x05173d5948fdf262 QUAD $0x0000441f0f660000 QUAD $0x7262c34048fd7262 QUAD $0x48b57162cb4048f5 @@ -4078,12 +5293,12 @@ TEXT ·randomF32AVX512(SB), NOSPLIT|NOFRAME, $0-32 QUAD $0xd02148c889480cc1 QUAD $0xd262c67c28fdf262 QUAD $0x0debf5c5c87c28fd - QUAD $0x597de2c4000002a1 - QUAD $0x7de2c40000026c15 - QUAD $0xe2c40000026b1d59 - QUAD $0xc40000026a25597d - QUAD $0x000002792d1879e2 - QUAD $0x00026835597de2c4 + QUAD $0x597de2c400000441 + QUAD $0x7de2c40000040c15 + QUAD $0xe2c40000040b1d59 + QUAD $0xc40000040a25597d + QUAD $0x000004192d1879e2 + QUAD $0x00040835597de2c4 QUAD $0x2e66666666666600 QUAD $0x0000000000841f0f QUAD $0xc5c5fa4028f5f262 @@ -4101,7 +5316,7 @@ TEXT ·randomF32AVX512(SB), NOSPLIT|NOFRAME, $0-32 QUAD $0x476d1ce4e5b9b948 QUAD $0x133111ebb849bf58 QUAD $0x0510fac594d049bb - QUAD $0x7c15b949000001e5 + QUAD $0x7c15b94900000385 QUAD $0x2e669e3779b97f4a QUAD $0x0000000000841f0f QUAD $0x491eeac149f28949 @@ -4112,7 +5327,55 @@ TEXT ·randomF32AVX512(SB), NOSPLIT|NOFRAME, $0-32 QUAD $0xff48870c11fac5c8 QUAD $0x75c23948ce014cc0 QUAD $0x77f8c55dec8948c7 - QUAD $0x00000000000000c3 + QUAD $0x00000000000080c3 + QUAD $0x0000000000009000 + QUAD $0x000000000000a000 + QUAD $0x000000000000b000 + QUAD $0x000000000000c000 + QUAD $0x000000000000d000 + QUAD $0x000000000000e000 + QUAD $0x000000000000f000 + QUAD $0x0000000000000000 + QUAD $0x0000000000001000 + QUAD $0x0000000000002000 + QUAD $0x0000000000003000 + QUAD $0x0000000000004000 + QUAD $0x0000000000005000 + QUAD $0x0000000000006000 + QUAD $0x0000000000007000 + QUAD $0x0000000000000000 + QUAD $0x0000000000001000 + QUAD $0x0000000000002000 + QUAD $0x0000000000003000 + QUAD $0x0000000000004000 + QUAD $0x0000000000005000 + QUAD $0x0000000000006000 + QUAD $0x0000000000007000 + QUAD $0x0000000000008000 + QUAD $0x0000000000009000 + QUAD $0x000000000000a000 + QUAD $0x000000000000b000 + QUAD $0x000000000000c000 + QUAD $0x000000000000d000 + QUAD $0x000000000000e000 + QUAD $0x000000000000f000 + QUAD $0x0000000000000000 + QUAD $0x0000000000001000 + QUAD $0x0000000000002000 + QUAD $0x0000000000003000 + QUAD $0x0000000000004000 + QUAD $0x0000000000005000 + QUAD $0x0000000000006000 + QUAD $0x0000000000007000 + QUAD $0x0000000000000000 + QUAD $0x0000000000001000 + QUAD $0x0000000000002000 + QUAD $0x0000000000003000 + QUAD $0x0000000000004000 + QUAD $0x0000000000005000 + QUAD $0x0000000000006000 + QUAD $0x0000000000007000 + QUAD $0x0000000000000000 QUAD $0x0000000000000100 QUAD $0x0000000000000200 QUAD $0x0000000000000300 @@ -4144,6 +5407,10 @@ TEXT ·randomF32AVX512(SB), NOSPLIT|NOFRAME, $0-32 QUAD $0x0000000000000500 QUAD $0x0000000000000600 QUAD $0x0000000000000700 + QUAD $0x0000000000010000 + QUAD $0x0000000000008000 + QUAD $0x0000000000010000 + QUAD $0x0000000000008000 QUAD $0xbbcdcbfa53e0a800 QUAD $0x779b97f4a7c150f1 QUAD $0x336963eefba1f8e3 diff --git a/internal/amd64/numeric_sse2_amd64.s b/internal/amd64/numeric_sse2_amd64.s index a4fee43..5bac7c1 100644 --- a/internal/amd64/numeric_sse2_amd64.s +++ b/internal/amd64/numeric_sse2_amd64.s @@ -176,24 +176,90 @@ TEXT ·polyEvalFloat32SSE2(SB), NOSPLIT|NOFRAME, $0-72 MOVQ x_len+32(FP), R8 MOVQ coeffs_len+56(FP), R9 QUAD $0x4dc14c0f4cc1394c - QUAD $0x000000cb840fc985 - QUAD $0x0001678e0fc0854d - QUAD $0xfc8a44100f42f300 - QUAD $0x0f02e88348c8894c - QUAD $0x894855000000c48c - QUAD $0x8d41f8e4834853e5 - QUAD $0xe1834103e183ff49 - QUAD $0x2e661cebd2314503 + QUAD $0xe58948551674c985 + QUAD $0xf88349f8e4834853 + QUAD $0x0206e9c031217d10 + QUAD $0x298e0fc0854d0000 + QUAD $0x0f08f88349000003 + QUAD $0xe9c031000001a883 + QUAD $0x100f42f300000309 + QUAD $0x0f01f98349fc8a44 + QUAD $0x498d49000001248e + QUAD $0x518d4d00c0c60ffe + QUAD $0x314500000010bbfd + QUAD $0x2e6666666634ebdb QUAD $0x0000000000841f0f - QUAD $0xff499714110f42f3 - QUAD $0x0099840fc2394dc2 - QUAD $0x960c100f42f30000 - QUAD $0x8341d0280fc38949 - QUAD $0x49cb89481e7401f9 - QUAD $0x001f0fd0280fc389 + QUAD $0x110f429f24110f42 + QUAD $0x9f7c110f42109f74 + QUAD $0x48309f44110f4620 + QUAD $0x394cc3894910588d + QUAD $0x48000001918f0fc3 + QUAD $0x429e0c100f42d889 + QUAD $0x100f42109e54100f + QUAD $0x9e6c100f42209e5c + QUAD $0x0fe0280fcb894830 + QUAD $0x280f44f8280ff028 + QUAD $0x0f397501c1f641c0 + QUAD $0x0f46f3e1590fe028 + QUAD $0xc9c60f45f88a4c10 + QUAD $0xf0280fe1580f4100 + QUAD $0x0ff1580f41f2590f + QUAD $0x580f41fb590ff828 + QUAD $0x590f44c0280f44f9 + QUAD $0xd3894cc1580f45c5 + QUAD $0xffff67840fc98548 + QUAD $0x001f0f02c38348ff + QUAD $0x4c100f44f3e1590f + QUAD $0x9a54100f44f3f49a + QUAD $0x0f4100d2c60f45f8 + QUAD $0x580f41f2590fe258 + QUAD $0xfa580f41fb590ff2 + QUAD $0xc2580f45c5590f44 + QUAD $0x00c9c60f45e1590f + QUAD $0x41f2590fe1580f41 + QUAD $0x0f41fb590ff1580f + QUAD $0x0f45c5590f44f958 + QUAD $0x8348fec38348c158 + QUAD $0xfffeffe9a47f01fb + QUAD $0xeac149f0508d4dff + QUAD $0x4900000088840f04 + QUAD $0xe08348d0894cc2ff + QUAD $0x00c8c60fc8280ffe + QUAD $0x2e666600000010b9 + QUAD $0x0000000000841f0f + QUAD $0x4c110fc08f4c110f + QUAD $0x0fe08f4c110fd08f + QUAD $0x8f0c110ff08f4c11 + QUAD $0x4c110f108f4c110f + QUAD $0x48308f4c110f208f + QUAD $0x75fec0834820c183 + QUAD $0xc2f641f0418d48cf + QUAD $0xf8b8484beb337501 + QUAD $0x4c7fffffffffffff + QUAD $0x90c0570fc931c021 + QUAD $0x8f44110f8f04110f + QUAD $0xc8394808c1834810 + QUAD $0xb900000148e9ee75 + QUAD $0xc60fc03100000010 + QUAD $0x110f8704110f00c0 + QUAD $0x208744110f108744 + QUAD $0xc88948308744110f + QUAD $0x8e0fc22949c2894d + QUAD $0x100f42f30000010a + QUAD $0x8348c9894cfc8a44 + QUAD $0x000000a38c0f02e9 + QUAD $0x03e28341ff518d45 + QUAD $0x2e661beb03e18341 + QUAD $0x0000000000841f0f + QUAD $0xc0ff488714110ff3 + QUAD $0x0000cd840fc0394c + QUAD $0x8949860c100ff300 + QUAD $0x01f98341d0280fcb + QUAD $0xcb8949d3894c2074 + QUAD $0x0000441f0fd0280f QUAD $0x580f42f3d1590ff3 QUAD $0xcbff48cbff499a14 - QUAD $0xb87203f88348ee75 + QUAD $0xb87203f98348ee75 QUAD $0x00401f0f04c38349 QUAD $0x580f42f3d1590ff3 QUAD $0xf3d1590ff3f09a54 @@ -202,27 +268,20 @@ TEXT ·polyEvalFloat32SSE2(SB), NOSPLIT|NOFRAME, $0-72 QUAD $0x0f42f3d1590ff3e8 QUAD $0xfcc38349e49a5458 QUAD $0x75e9ca7f03fb8349 - QUAD $0x8e0fc0854dffffff - QUAD $0x08f883490000009c - QUAD $0x000080e9c0311873 - QUAD $0x313d7308f8834900 - QUAD $0x5bf8658d486cebc0 - QUAD $0xfffffff8b848c35d - QUAD $0x31c0214c7fffffff - QUAD $0x2e666666c0570fc9 + QUAD $0x7308fa8349ffffff + QUAD $0x89443aebc2894905 + QUAD $0x49ca294907e183c1 + QUAD $0xc8c60fc8280fc201 + QUAD $0x00000000801f0f00 + QUAD $0x874c110f870c110f + QUAD $0xc2394908c0834810 + QUAD $0x661774c98548ee75 QUAD $0x0000000000841f0f - QUAD $0x8f44110f8f04110f - QUAD $0xc8394808c1834810 - QUAD $0xfff8b84847ebee75 - QUAD $0x214c7fffffffffff - QUAD $0x00c8c60fc8280fc0 - QUAD $0x0000441f0f66c931 - QUAD $0x8f4c110f8f0c110f - QUAD $0xc8394808c1834810 - QUAD $0xf31974c03949ee75 - QUAD $0xebc0ff488704110f - QUAD $0x000000008704c7f1 - QUAD $0xf175c03949c0ff48 + QUAD $0xff499704110f42f3 + QUAD $0x8d48f275d0394dc2 + QUAD $0x8704c7c35d5bf865 + QUAD $0x49c0ff4800000000 + LONG $0xf175c039 BYTE $0xc3 RET @@ -236,68 +295,117 @@ TEXT ·convolveFloat32SSE2(SB), NOSPLIT|NOFRAME, $0-72 MOVQ ker_len+56(FP), R9 QUAD $0x56415741e5894855 QUAD $0xe483485354415541 - QUAD $0xcd894d18ec8348f8 - QUAD $0xc9854d08247c8948 - QUAD $0x0f41c8294dc0940f - QUAD $0xbc850fc10841c19c - QUAD $0x4c01488d4d000001 - QUAD $0x854dc94e0f4cc139 - QUAD $0x4d000001a88e0fc9 - QUAD $0x000001408e0fed85 - QUAD $0x41e88945aa0c8d4a - QUAD $0xfffff8ba4903e083 - QUAD $0xca83497fffffffff - QUAD $0x0c5e8d4cea214d04 - QUAD $0xfcc08348aa048d4a - QUAD $0x49db311024448948 - QUAD $0xfffffffffffffebe - QUAD $0xfffffffffdbf493f - QUAD $0xfffffcbc493fffff - QUAD $0x9022eb3fffffffff - QUAD $0x110ff30824448b48 - QUAD $0xc38349c3ff489804 - QUAD $0xcb394c04c6834804 - QUAD $0x00012e840ffd8949 - QUAD $0x49c0570fef894c00 - QUAD $0xed3145137304fd83 - QUAD $0x2e666600000081e9 + QUAD $0xc9854d18ec8348f8 + QUAD $0x08244c894cc0940f + QUAD $0x41c19c0f41c8294d + QUAD $0x00000340850fc108 + QUAD $0x4cc1394c01688d4d + QUAD $0x7d10fd8349e94e0f + QUAD $0x0000018ee9c93107 + QUAD $0xc9854d08244c8b4c + QUAD $0x8d4c000000aa8e0f + QUAD $0x4500000010b83046 + QUAD $0x2e6666666666d231 QUAD $0x0000000000841f0f - QUAD $0xed31451024548b48 + QUAD $0x894dc0570fc18948 + QUAD $0x0fc9570fcb894cc3 + QUAD $0x2e6666db570fd257 QUAD $0x0000000000841f0f - QUAD $0xf3f4ab4c100f43f3 - QUAD $0x0ff3f8ab54100f43 + QUAD $0xc60ffc9a64100ff3 + QUAD $0x41d06b100f4100e4 + QUAD $0x7b100f41e073100f + QUAD $0xec590f03100f45f0 + QUAD $0x580ff4590fc5580f + QUAD $0x44d7580ffc590fce + QUAD $0x49d8580f41c4590f + QUAD $0xc075cbff4804c383 + QUAD $0x110f429704110f42 + QUAD $0x9754110f4210974c + QUAD $0x4830975c110f4220 + QUAD $0x4940c0834910418d + QUAD $0x758e0fe8394cca89 + QUAD $0x000000d6e9ffffff + QUAD $0x49c28949f0458d49 + QUAD $0x8945c2ff4904eac1 + QUAD $0xf8834803e08341d0 + QUAD $0x00000010b80c7330 + QUAD $0x4900000083e9c931 + QUAD $0xc0570fc931fce283 + QUAD $0x8f44110f8f04110f + QUAD $0x110f208f44110f10 + QUAD $0x408f44110f308f44 + QUAD $0x44110f508f44110f + QUAD $0x0f708f44110f608f + QUAD $0x0f000000808f8411 + QUAD $0x0f000000908f8411 + QUAD $0x0f000000a08f8411 + QUAD $0x0f000000b08f8411 + QUAD $0x0f000000c08f8411 + QUAD $0x0f000000d08f8411 + QUAD $0x0f000000e08f8411 + QUAD $0x48000000f08f8411 + QUAD $0x75fcc2834940c183 + QUAD $0x8d483074c0854d8f + QUAD $0x001f0fc0570f1041 + QUAD $0x8f44110f8f04110f + QUAD $0x110f208f44110f10 + QUAD $0x8348c18948308f44 + QUAD $0x48e175c8ff4910c0 + QUAD $0x894dc18948f0c083 + QUAD $0x018e8e0fc82949e8 + QUAD $0x480824448b480000 + QUAD $0x000001308e0fc085 + QUAD $0x41c2894182048d4c + QUAD $0xfffffcbb4903e283 + QUAD $0xc321497fffffffff + QUAD $0x0cc383488e1c8d48 + QUAD $0x448948fc82448d48 + QUAD $0xbe498e348d481024 + QUAD $0x3ffffffffffffffe + QUAD $0xfffffffffffdbf49 + QUAD $0xfffffffcbc493fff + QUAD $0x90661eeb3fffffff + QUAD $0xc1ff488f04110ff3 + QUAD $0x04c6834804c38348 + QUAD $0x840fc9394ccd894d + QUAD $0x48e9894d00000114 + QUAD $0x0f09730408247c83 + QUAD $0x907aebed3145c057 + QUAD $0x1024548b48c0570f + QUAD $0x0000441f0fed3145 + QUAD $0xf3f4ab4c100f42f3 + QUAD $0x0ff3f8ab54100f42 QUAD $0x894cc8580ff30a59 - QUAD $0x14590ff3f0314ce8 - QUAD $0x0f43f3d1580ff381 - QUAD $0x4ce8894cfcab4c10 - QUAD $0xf3810c590ff3f831 - QUAD $0x04100f43f3ca580f - QUAD $0xf3e0314ce8894cab - QUAD $0xc1580ff38104590f - QUAD $0xf0c2834804c58349 - QUAD $0xc0854da375ea394d - QUAD $0x8d4affffff4a840f - QUAD $0x8b4800000000ad04 - QUAD $0x8d4ec22948102454 - QUAD $0x00401f0fc031ae2c + QUAD $0x590f41f3f0314ce8 + QUAD $0x42f3d1580ff38014 + QUAD $0xe8894cfcab4c100f + QUAD $0x0c590f41f3f8314c + QUAD $0x0f42f3ca580ff380 + QUAD $0x314ce8894cab0410 + QUAD $0xf38004590f41f3e0 + QUAD $0x4804c58349c1580f + QUAD $0xa075eb394df0c283 + QUAD $0xffff57840fd2854d + QUAD $0x000000ad048d4aff + QUAD $0x29481024548b4800 + QUAD $0x90c031ae2c8d4ec2 QUAD $0xf300854c100f41f3 QUAD $0x48c1580ff30a590f QUAD $0x3949fcc28348c0ff - QUAD $0xffffff10e9e575c0 - QUAD $0xc031047308f98349 - QUAD $0xfffffff8b8483deb - QUAD $0x31c8214c7fffffff - QUAD $0x24548b48c0570fc9 - QUAD $0x2e66666666666608 + QUAD $0xffffff20e9e575c2 + QUAD $0x8949057308f88349 + QUAD $0xe083e8894435ebc8 + QUAD $0x0fc80149c0294907 + QUAD $0x0000441f0f66c057 + QUAD $0x8f44110f8f04110f + QUAD $0xc8394908c1834810 + QUAD $0x661974c08548ee75 QUAD $0x0000000000841f0f - QUAD $0x8a44110f8a04110f - QUAD $0xc8394808c1834810 - QUAD $0x481874c13949ee75 - QUAD $0x00401f0f08244c8b - QUAD $0x48000000008104c7 - QUAD $0x48f175c13949c0ff - QUAD $0x5d415c415bd8658d - LONG $0x5f415e41 + QUAD $0x000000008704c742 + QUAD $0xf075c5394dc0ff49 + QUAD $0x415c415bd8658d48 + LONG $0x415e415d + BYTE $0x5f BYTE $0x5d BYTE $0xc3 RET @@ -312,49 +420,106 @@ TEXT ·correlateFloat32SSE2(SB), NOSPLIT|NOFRAME, $0-72 MOVQ ker_len+56(FP), R9 QUAD $0x294dc0940fc9854d QUAD $0xc20841c29c0f41c8 - QUAD $0x8d490000014d850f - QUAD $0x4e0f48c1394c0140 - QUAD $0x01398e0fc08548c1 - QUAD $0xe58e0fc9854d0000 - QUAD $0x41e5894855000000 - QUAD $0xf8e4834853564157 - QUAD $0xb84903e183c98944 - QUAD $0x7ffffffffffffff8 - QUAD $0x4cc8214d04c88349 - QUAD $0x1aebdb31450c568d - QUAD $0xff499f04110f42f3 - QUAD $0xc6834804c28349c3 - QUAD $0x0093840fc3394904 - QUAD $0x49db31c0570f0000 - QUAD $0x2e66665b7204f983 + QUAD $0x485500000304850f + QUAD $0x485356415741e589 + QUAD $0x4c01408d49f8e483 + QUAD $0x8348c14e0f48c139 + QUAD $0x9be9c931077d10f8 + QUAD $0x8e0fc9854d000001 + QUAD $0x30468d4c000000af + QUAD $0x314500000010bb41 + QUAD $0x2e666666666666d2 + QUAD $0x0000000000841f0f + QUAD $0x3145c0570fd9894c + QUAD $0x0fd2570fc9570fdb + QUAD $0x2e6666666666db57 + QUAD $0x0000000000841f0f + QUAD $0xc60f9a24100f42f3 + QUAD $0xd0986c100f4300e4 + QUAD $0x0f43e09874100f43 + QUAD $0x04100f47f0987c10 + QUAD $0x0fc5580fec590f98 + QUAD $0xfc590fce580ff459 + QUAD $0x41c4590f44d7580f + QUAD $0x394dc3ff49d8580f + QUAD $0x9704110f42bd75d9 + QUAD $0x0f4210974c110f42 + QUAD $0x5c110f4220975411 + QUAD $0x834910598d4c3097 + QUAD $0xc33949ca894940c0 + QUAD $0xe3e9ffffff728e0f + QUAD $0x49f0488d48000000 + QUAD $0xff4904eac149ca89 + QUAD $0x03e08341d08945c2 + QUAD $0xba410d7330f98348 + QUAD $0x8fe9c93100000010 + QUAD $0x31fce28349000000 + QUAD $0x2e666666c0570fc9 + QUAD $0x0000000000841f0f + QUAD $0x8f44110f8f04110f + QUAD $0x110f208f44110f10 + QUAD $0x408f44110f308f44 + QUAD $0x44110f508f44110f + QUAD $0x0f708f44110f608f + QUAD $0x0f000000808f8411 + QUAD $0x0f000000908f8411 + QUAD $0x0f000000a08f8411 + QUAD $0x0f000000b08f8411 + QUAD $0x0f000000c08f8411 + QUAD $0x0f000000d08f8411 + QUAD $0x0f000000e08f8411 + QUAD $0x48000000f08f8411 + QUAD $0x75fcc2834940c183 + QUAD $0x8d4c3074c0854d8f + QUAD $0x001f0fc0570f1051 + QUAD $0x8f44110f8f04110f + QUAD $0x110f208f44110f10 + QUAD $0x8349d1894c308f44 + QUAD $0x49e175c8ff4910c2 + QUAD $0x8949d1894cf0c283 + QUAD $0x012e8e0fc82949c0 + QUAD $0xd28e0fc9854d0000 + QUAD $0x8341c88945000000 + QUAD $0xfffffffcba4903e0 + QUAD $0x4cca214d7fffffff + QUAD $0x480cc383498e1c8d + QUAD $0x001f0f1ceb8e348d + QUAD $0xc1ff488f04110ff3 + QUAD $0x04c6834804c38349 + QUAD $0x0000e7840fc13948 + QUAD $0x8349db31c0570f00 + QUAD $0x2e6666665c7204f9 QUAD $0x0000000000841f0f - QUAD $0xf3f49a4c100f41f3 - QUAD $0x0ff3f89a54100f41 + QUAD $0xf3f49b4c100f41f3 + QUAD $0x0ff3f89b54100f41 QUAD $0x9a54590ff39a0c59 QUAD $0x0f41f3c8580ff304 - QUAD $0x5c590ff3fc9a5c10 + QUAD $0x5c590ff3fc9b5c10 QUAD $0x41f3d1580ff3089a - QUAD $0x44590ff39a04100f + QUAD $0x44590ff39b04100f QUAD $0x0ff3da580ff30c9a QUAD $0x394904c38348c358 - QUAD $0x8074c98548b575d8 + QUAD $0x8074c0854db575da QUAD $0x9e1c8d489a348d4c QUAD $0x0000441f0fff3145 QUAD $0x43f3bb0c100f42f3 QUAD $0xc1580ff3be0c590f - QUAD $0xe875f9394cc7ff49 - QUAD $0x658d48ffffff53e9 - QUAD $0xc35d5f415e415be8 - QUAD $0xc931047308f88348 - QUAD $0xfffffff8b94832eb - QUAD $0x31c121487fffffff - QUAD $0x2e666666c0570fd2 + QUAD $0xe875f8394dc7ff49 + QUAD $0xf88349ffffff53e9 + QUAD $0x38ebc88949057308 + QUAD $0xd0294907e283c289 + QUAD $0x2e66c0570fc80149 + QUAD $0x0000000000841f0f + QUAD $0x8f44110f8f04110f + QUAD $0xc8394908c1834810 + QUAD $0x661974d28548ee75 QUAD $0x0000000000841f0f - QUAD $0x9744110f9704110f - QUAD $0xd1394808c2834810 - QUAD $0x008f04c70aebee75 - QUAD $0x3948c1ff48000000 - LONG $0xc3f175c8 + QUAD $0x000000008704c742 + QUAD $0xf075c0394cc0ff49 + QUAD $0x415e415be8658d48 + BYTE $0x5f + BYTE $0x5d + BYTE $0xc3 RET // func normFloat64SSE2(a []float64) float64 @@ -522,24 +687,106 @@ TEXT ·polyEvalFloat64SSE2(SB), NOSPLIT|NOFRAME, $0-72 MOVQ x_len+32(FP), R8 MOVQ coeffs_len+56(FP), R9 QUAD $0x4dc14c0f4cc1394c - QUAD $0x000000cb840fc985 - QUAD $0x00016a8e0fc0854d - QUAD $0xf8ca44100f42f200 - QUAD $0x0f02e88348c8894c - QUAD $0x894855000000c48c - QUAD $0x8d41f8e4834853e5 - QUAD $0xe1834103e183ff49 - QUAD $0x2e661cebd2314503 + QUAD $0xe58948551a74c985 + QUAD $0xec8348f0e4834853 + QUAD $0x31217d10f8834930 + QUAD $0x854d0000027ee9c0 + QUAD $0x49000003a68e0fc0 + QUAD $0x0001fc830f04f883 + QUAD $0x00000385e9c03100 + QUAD $0x49f8ca4c100f42f2 + QUAD $0x0001488e0f01f983 + QUAD $0x498d49c9140f6600 + QUAD $0x4500000010bb41ff + QUAD $0x66240c290f66d231 QUAD $0x0000000000841f0f - QUAD $0xff49d714110f42f2 - QUAD $0x0099840fc2394dc2 - QUAD $0xd60c100f42f20000 - QUAD $0x41d0280f66c38949 - QUAD $0xcb89481d7401f983 - QUAD $0x90d0280f66c38949 + QUAD $0xd604100f42d8894c + QUAD $0x0f4266102444290f + QUAD $0x100f426610d65410 + QUAD $0x64100f426620d65c + QUAD $0xd66c100f426630d6 + QUAD $0x50d674100f426640 + QUAD $0x6660d67c100f4266 + QUAD $0x894970d644100f46 + QUAD $0x4466c9280f4466cb + QUAD $0xd9280f4466d1280f + QUAD $0x0f4466e1280f4466 + QUAD $0x66f1280f4466e928 + QUAD $0xc1280f66f9280f44 + QUAD $0x0000000000841f0f + QUAD $0x590f4166c8280f66 + QUAD $0x4466ff590f4466c8 + QUAD $0xed590f4466f6590f + QUAD $0x0f4466e4590f4466 + QUAD $0x66d2590f4466db59 + QUAD $0x42f210244c590f44 + QUAD $0x140f66f8da44100f + QUAD $0x4466c8580f4466c0 + QUAD $0xd8580f4466d0580f + QUAD $0x0f4466e0580f4466 + QUAD $0x66f0580f4466e858 + QUAD $0xc1580f66f8580f44 + QUAD $0x987fdb854dcbff49 + QUAD $0x4666d70c110f4666 + QUAD $0x0f466610d754110f + QUAD $0x110f466620d75c11 + QUAD $0x6c110f466630d764 + QUAD $0xd774110f466640d7 + QUAD $0x60d77c110f466650 + QUAD $0x4c70d744110f4266 + QUAD $0x394dc2894910588d + QUAD $0x8e0f240c280f66c3 + QUAD $0x00010be9fffffedc + QUAD $0xeac149f0508d4d00 + QUAD $0x49000000c0840f04 + QUAD $0xe08348d0894cc2ff + QUAD $0x140f66c1280f66fe + QUAD $0x906600000010b9c1 + QUAD $0x0f6680cf44110f66 + QUAD $0x44110f6690cf4411 + QUAD $0xb0cf44110f66a0cf + QUAD $0x0f66c0cf44110f66 + QUAD $0x44110f66d0cf4411 + QUAD $0xf0cf44110f66e0cf + QUAD $0x110f66cf04110f66 + QUAD $0xcf44110f6610cf44 + QUAD $0x6630cf44110f6620 + QUAD $0x110f6640cf44110f + QUAD $0xcf44110f6650cf44 + QUAD $0x4870cf44110f6660 + QUAD $0x75fec0834820c183 + QUAD $0xc2f641f0418d4897 + QUAD $0xfcb8486feb3b7501 + QUAD $0x4c7fffffffffffff + QUAD $0x66c0570fc931c021 + QUAD $0x0000000000841f0f + QUAD $0xcf44110fcf04110f + QUAD $0xc8394804c1834810 + QUAD $0xb900000169e9ee75 + QUAD $0x0f66c03100000010 + QUAD $0x10c74c110f66c914 + QUAD $0x110f66c70c110f66 + QUAD $0xc74c110f6630c74c + QUAD $0x6650c74c110f6620 + QUAD $0x110f6640c74c110f + QUAD $0xc74c110f6670c74c + QUAD $0x49c2894dc8894860 + QUAD $0x0000010e8e0fc229 + QUAD $0x4cf8ca44100f42f2 + QUAD $0x8c0f02e98348c989 + QUAD $0xff518d45000000a7 + QUAD $0x03e1834103e28341 + QUAD $0x2e66666666661feb + QUAD $0x0000000000841f0f + QUAD $0xc0ff48c714110ff2 + QUAD $0x0000cd840fc0394c + QUAD $0x8949c60c100ff200 + QUAD $0xf98341d0280f66cb + QUAD $0x8949d3894c1f7401 + QUAD $0x001f0fd0280f66cb QUAD $0x580f42f2d1590ff2 QUAD $0xcbff48cbff49da14 - QUAD $0xb87203f88348ee75 + QUAD $0xb87203f98348ee75 QUAD $0x00401f0f04c38349 QUAD $0x580f42f2d1590ff2 QUAD $0xf2d1590ff2e0da54 @@ -548,28 +795,22 @@ TEXT ·polyEvalFloat64SSE2(SB), NOSPLIT|NOFRAME, $0-72 QUAD $0x0f42f2d1590ff2d0 QUAD $0xfcc38349c8da5458 QUAD $0x75e9ca7f03fb8349 - QUAD $0x8e0fc0854dffffff - QUAD $0x04f883490000009f - QUAD $0x000082e9c0311873 - QUAD $0x313f7304f8834900 - QUAD $0x5bf8658d486eebc0 - QUAD $0xfffffffcb848c35d - QUAD $0x31c0214c7fffffff - QUAD $0x2e6666c0570f66c9 - QUAD $0x0000000000841f0f - QUAD $0x110f66cf04110f66 - QUAD $0x4804c1834810cf44 - QUAD $0xb84848ebec75c839 - QUAD $0x7ffffffffffffffc - QUAD $0x66c8280f66c0214c - QUAD $0x001f0fc931c8140f - QUAD $0x110f66cf0c110f66 - QUAD $0x4804c1834810cf4c - QUAD $0x74c03949ec75c839 - QUAD $0xff48c704110ff21a - QUAD $0x00c704c748f1ebc0 - QUAD $0x3949c0ff48000000 - LONG $0xc3f075c0 + QUAD $0x7304fa8349ffffff + QUAD $0x89443aebc2894905 + QUAD $0x49ca294903e183c1 + QUAD $0x0f66c8280f66c201 + QUAD $0x0000441f0f66c814 + QUAD $0x110f66c70c110f66 + QUAD $0x4904c0834810c74c + QUAD $0x74c98548ec75c239 + QUAD $0x00000000801f0f15 + QUAD $0xff49d704110f42f2 + QUAD $0x8d48f275d0394dc2 + QUAD $0x04c748c35d5bf865 + QUAD $0xc0ff4800000000c7 + LONG $0x75c03949 + BYTE $0xf0 + BYTE $0xc3 RET // func convolveFloat64SSE2(dst []float64, sig []float64, ker []float64) @@ -582,66 +823,134 @@ TEXT ·convolveFloat64SSE2(SB), NOSPLIT|NOFRAME, $0-72 MOVQ ker_len+56(FP), R9 QUAD $0x56415741e5894855 QUAD $0xe483485354415541 - QUAD $0xcd894d18ec8348f8 - QUAD $0xc9854d08247c8948 - QUAD $0x0f41c8294dc0940f - QUAD $0xbd850fc10841c19c - QUAD $0x4c01488d4d000001 - QUAD $0x854dc94e0f4cc139 - QUAD $0x4d000001a98e0fc9 - QUAD $0x000001408e0fed85 - QUAD $0x41e88945ea0c8d4a - QUAD $0xfffffcba4903e083 - QUAD $0xea214d7fffffffff - QUAD $0xea048d4a185e8d4c - QUAD $0x24448948f8c08348 - QUAD $0xfffffebe49db3110 + QUAD $0xc9854d18ec8348f8 + QUAD $0x08244c894cc0940f + QUAD $0x41c19c0f41c8294d + QUAD $0x000003e0850fc108 + QUAD $0x4cc1394c01688d4d + QUAD $0x7d10fd8349e94e0f + QUAD $0x0000022be9c93107 + QUAD $0xc9854d08244c8b4c + QUAD $0x8d4c000001328e0f + QUAD $0x4500000010b87046 + QUAD $0x2e6666666666d231 + QUAD $0x0000000000841f0f + QUAD $0x4dc0570f66c18948 + QUAD $0x570f66cb894cc389 + QUAD $0x570f66d2570f66c9 + QUAD $0x570f66e4570f66db + QUAD $0x570f66f6570f66ed + QUAD $0x00000000801f0fff + QUAD $0x66f8da44100f44f2 + QUAD $0x100f4566c0140f45 + QUAD $0xa053100f4566904b + QUAD $0x4566b05b100f4566 + QUAD $0x590f4566c063100f + QUAD $0x4566c1580f4166c8 + QUAD $0xca580f4166d0590f + QUAD $0x0f4166d8590f4566 + QUAD $0x66e0590f4566d358 + QUAD $0x100f4566dc580f41 + QUAD $0x66c8590f4566d04b + QUAD $0x100f4566e1580f41 + QUAD $0x66c8590f4566e04b + QUAD $0x100f4566e9580f41 + QUAD $0x66c8590f4566f04b + QUAD $0x100f4566f1580f41 + QUAD $0x4166c8590f45660b + QUAD $0x4808c38349f9580f + QUAD $0xffffff68850fcbff + QUAD $0x4266d704110f4266 + QUAD $0x0f426610d74c110f + QUAD $0x110f426620d75411 + QUAD $0x64110f426630d75c + QUAD $0xd76c110f426640d7 + QUAD $0x60d774110f426650 + QUAD $0x4870d77c110f4266 + QUAD $0x4980e8834910418d + QUAD $0xed8e0fe8394cca89 + QUAD $0x000000ebe9fffffe + QUAD $0x04eac149f0558d4d + QUAD $0xff490000009f840f + QUAD $0xfee08348d0894cc2 + QUAD $0x0f6600000010b841 + QUAD $0x0000441f0f66c057 + QUAD $0x6680c744110f4266 + QUAD $0x426690c744110f42 + QUAD $0x0f4266a0c744110f + QUAD $0x110f4266b0c74411 + QUAD $0x44110f4266c0c744 + QUAD $0xc744110f4266d0c7 + QUAD $0xf0c744110f4266e0 + QUAD $0x4266c704110f4266 + QUAD $0x0f426610c744110f + QUAD $0x110f426620c74411 + QUAD $0x44110f426630c744 + QUAD $0xc744110f426640c7 + QUAD $0x60c744110f426650 + QUAD $0x4970c744110f4266 + QUAD $0x75fec0834820c083 + QUAD $0xc2f641f0488d4987 + QUAD $0x10b8413eeb0a7501 + QUAD $0x570f66c931000000 + QUAD $0x0f66cf04110f66c0 + QUAD $0x44110f6610cf4411 + QUAD $0x30cf44110f6620cf + QUAD $0x0f6640cf44110f66 + QUAD $0x44110f6650cf4411 + QUAD $0x70cf44110f6660cf + QUAD $0x2949e8894dc1894c + QUAD $0x48000001918e0fc8 + QUAD $0x0fc085480824448b + QUAD $0x048d4c000001338e + QUAD $0x03e28341c28941c2 + QUAD $0xfffffffffffcbb49 + QUAD $0x1c8d48c321497fff + QUAD $0x448d4818c38348ce + QUAD $0x481024448948f8c2 + QUAD $0xfffffebe49ce348d QUAD $0xfdbf491fffffffff QUAD $0x491fffffffffffff QUAD $0xfffffffffffffcbc - QUAD $0x0000441f0f26eb1f - QUAD $0x110ff20824448b48 - QUAD $0xc38349c3ff48d804 - QUAD $0xcb394c08c6834808 - QUAD $0x00012f840ffd8949 - QUAD $0xc0570f66ef894c00 - QUAD $0x3145127304fd8349 - QUAD $0x2e6600000080e9ed - QUAD $0x0000000000841f0f - QUAD $0xed31451024548b48 - QUAD $0x0000000000841f0f - QUAD $0xf2e8eb4c100f43f2 - QUAD $0x0ff2f0eb54100f43 + QUAD $0x0000441f0f21eb1f + QUAD $0xc1ff48cf04110ff2 + QUAD $0x08c6834808c38348 + QUAD $0x840fc9394ccd894d + QUAD $0x48e9894d00000114 + QUAD $0x6609730408247c83 + QUAD $0x79ebed3145c0570f + QUAD $0x24548b48c0570f66 + QUAD $0x00401f0fed314510 + QUAD $0xf2e8eb4c100f42f2 + QUAD $0x0ff2f0eb54100f42 QUAD $0x894cc8580ff20a59 - QUAD $0x14590ff2f0314ce8 - QUAD $0x0f43f2d1580ff2c1 - QUAD $0x4ce8894cf8eb4c10 - QUAD $0xf2c10c590ff2f831 - QUAD $0x04100f43f2ca580f - QUAD $0xf2e0314ce8894ceb - QUAD $0xc1580ff2c104590f - QUAD $0xe0c2834804c58349 - QUAD $0xc0854da375ea394d - QUAD $0x8d4affffff4a840f - QUAD $0x8b4800000000ed04 - QUAD $0x8d4ec22948102454 - QUAD $0x00401f0fc031ee2c + QUAD $0x590f41f2f0314ce8 + QUAD $0x42f2d1580ff2c014 + QUAD $0xe8894cf8eb4c100f + QUAD $0x0c590f41f2f8314c + QUAD $0x0f42f2ca580ff2c0 + QUAD $0x314ce8894ceb0410 + QUAD $0xf2c004590f41f2e0 + QUAD $0x4804c58349c1580f + QUAD $0xa075eb394de0c283 + QUAD $0xffff57840fd2854d + QUAD $0x000000ed048d4aff + QUAD $0x29481024548b4800 + QUAD $0x90c031ee2c8d4ec2 QUAD $0xf200c54c100f41f2 QUAD $0x48c1580ff20a590f QUAD $0x3949f8c28348c0ff - QUAD $0xffffff10e9e575c0 - QUAD $0xc031047304f98349 - QUAD $0xfffffffcb8483feb - QUAD $0x31c8214c7fffffff - QUAD $0x548b48c0570f66c9 - QUAD $0x2e66666666660824 - QUAD $0x0000000000841f0f - QUAD $0x110f66ca04110f66 - QUAD $0x4804c1834810ca44 - QUAD $0x74c13949ec75c839 - QUAD $0x906608244c8b4817 - QUAD $0x00000000c104c748 - QUAD $0xf075c13949c0ff48 + QUAD $0xffffff20e9e575c2 + QUAD $0x8949057304f88349 + QUAD $0xe083e8894435ebc8 + QUAD $0x66c80149c0294903 + QUAD $0x0000441f0fc0570f + QUAD $0x110f66cf04110f66 + QUAD $0x4904c1834810cf44 + QUAD $0x74c08548ec75c839 + QUAD $0x00000000801f0f17 + QUAD $0x00000000c704c74a + QUAD $0xf075c5394dc0ff49 QUAD $0x415c415bd8658d48 LONG $0x415e415d BYTE $0x5f @@ -659,49 +968,123 @@ TEXT ·correlateFloat64SSE2(SB), NOSPLIT|NOFRAME, $0-72 MOVQ ker_len+56(FP), R9 QUAD $0x294dc0940fc9854d QUAD $0xc20841c29c0f41c8 - QUAD $0x8d490000014e850f - QUAD $0x4e0f48c1394c0140 - QUAD $0x013a8e0fc08548c1 - QUAD $0xe58e0fc9854d0000 - QUAD $0x41e5894855000000 - QUAD $0xf8e4834853564157 - QUAD $0xb84903e183c98944 - QUAD $0x7ffffffffffffffc - QUAD $0x4518568d4cc8214d - QUAD $0x00401f0f1eebdb31 - QUAD $0xff49df04110f42f2 - QUAD $0xc6834808c28349c3 - QUAD $0x0093840fc3394908 - QUAD $0xdb31c0570f660000 - QUAD $0x2e665a7204f98349 + QUAD $0x485500000394850f + QUAD $0x485356415741e589 + QUAD $0x4c01408d49f8e483 + QUAD $0x8348c14e0f48c139 + QUAD $0x28e9c931077d10f8 + QUAD $0x8e0fc9854d000002 + QUAD $0x70468d4c0000013a + QUAD $0x314500000010bb41 + QUAD $0x2e666666666666d2 + QUAD $0x0000000000841f0f + QUAD $0x45c0570f66d9894c + QUAD $0x0f66c9570f66db31 + QUAD $0x0f66db570f66d257 + QUAD $0x0f66ed570f66e457 + QUAD $0x2e66ff570f66f657 QUAD $0x0000000000841f0f - QUAD $0xf2e8da4c100f41f2 - QUAD $0x0ff2f0da54100f41 + QUAD $0x4566da04100f46f2 + QUAD $0x4c100f4766c0140f + QUAD $0xd854100f476690d8 + QUAD $0xb0d85c100f4766a0 + QUAD $0x66c0d864100f4766 + QUAD $0x580f4166c8590f45 + QUAD $0x4166d0590f4566c1 + QUAD $0xd8590f4566ca580f + QUAD $0x0f4566d3580f4166 + QUAD $0x66dc580f4166e059 + QUAD $0x4566d0d84c100f47 + QUAD $0xe1580f4166c8590f + QUAD $0x66e0d84c100f4766 + QUAD $0x580f4166c8590f45 + QUAD $0xf0d84c100f4766e9 + QUAD $0x0f4166c8590f4566 + QUAD $0xd80c100f4766f158 + QUAD $0x0f4166c8590f4566 + QUAD $0xd9394dc3ff49f958 + QUAD $0x4266ffffff62850f + QUAD $0x110f4266d704110f + QUAD $0x54110f426610d74c + QUAD $0xd75c110f426620d7 + QUAD $0x40d764110f426630 + QUAD $0x6650d76c110f4266 + QUAD $0x426660d774110f42 + QUAD $0x598d4c70d77c110f + QUAD $0xca894980e8834910 + QUAD $0xfffee78e0fc33949 + QUAD $0x8d4c000000e5e9ff + QUAD $0x840f04eac149f050 + QUAD $0x4cc2ff4900000099 + QUAD $0xb841fee18348d189 + QUAD $0xc0570f6600000010 + QUAD $0x6680c744110f4266 + QUAD $0x426690c744110f42 + QUAD $0x0f4266a0c744110f + QUAD $0x110f4266b0c74411 + QUAD $0x44110f4266c0c744 + QUAD $0xc744110f4266d0c7 + QUAD $0xf0c744110f4266e0 + QUAD $0x4266c704110f4266 + QUAD $0x0f426610c744110f + QUAD $0x110f426620c74411 + QUAD $0x44110f426630c744 + QUAD $0xc744110f426640c7 + QUAD $0x60c744110f426650 + QUAD $0x4970c744110f4266 + QUAD $0x75fec1834820c083 + QUAD $0xc2f641f0488d4987 + QUAD $0x10b8413eeb0a7501 + QUAD $0x570f66c931000000 + QUAD $0x0f66cf04110f66c0 + QUAD $0x44110f6610cf4411 + QUAD $0x30cf44110f6620cf + QUAD $0x0f6640cf44110f66 + QUAD $0x44110f6650cf4411 + QUAD $0x70cf44110f6660cf + QUAD $0x2949c08949c1894c + QUAD $0x4d000001318e0fc8 + QUAD $0x000000d58e0fc985 + QUAD $0x4903e08341c88945 + QUAD $0xfffffffffffffcba + QUAD $0xce1c8d4cca214d7f + QUAD $0xce348d4818c38349 + QUAD $0x0000441f0f661feb + QUAD $0xc1ff48cf04110ff2 + QUAD $0x08c6834808c38349 + QUAD $0x0000e7840fc13948 + QUAD $0x49db31c0570f6600 + QUAD $0x2e66665b7204f983 + QUAD $0x0000000000841f0f + QUAD $0xf2e8db4c100f41f2 + QUAD $0x0ff2f0db54100f41 QUAD $0xda54590ff2da0c59 QUAD $0x0f41f2c8580ff208 - QUAD $0x5c590ff2f8da5c10 + QUAD $0x5c590ff2f8db5c10 QUAD $0x41f2d1580ff210da - QUAD $0x44590ff2da04100f + QUAD $0x44590ff2db04100f QUAD $0x0ff2da580ff218da QUAD $0x394904c38348c358 - QUAD $0x8074c98548b575d8 + QUAD $0x8074c0854db575da QUAD $0xde1c8d48da348d4c QUAD $0x0000441f0fff3145 QUAD $0x43f2fb0c100f42f2 QUAD $0xc1580ff2fe0c590f - QUAD $0xe875f9394cc7ff49 - QUAD $0x658d48ffffff53e9 - QUAD $0xc35d5f415e415be8 - QUAD $0xc931047304f88348 - QUAD $0xfffffffcb94832eb - QUAD $0x31c121487fffffff - QUAD $0x2e666666c0570fd2 + QUAD $0xe875f8394dc7ff49 + QUAD $0xf88349ffffff53e9 + QUAD $0x38ebc88949057304 + QUAD $0xd0294903e283c289 + QUAD $0x66c0570f66c80149 QUAD $0x0000000000841f0f - QUAD $0xd744110fd704110f - QUAD $0xd1394804c2834810 - QUAD $0xcf04c7480bebee75 - QUAD $0x48c1ff4800000000 - LONG $0xf075c839 + QUAD $0x110f66cf04110f66 + QUAD $0x4904c1834810cf44 + QUAD $0x74d28548ec75c839 + QUAD $0x00000000801f0f17 + QUAD $0x00000000c704c74a + QUAD $0xf075c0394cc0ff49 + QUAD $0x415e415be8658d48 + BYTE $0x5f + BYTE $0x5d BYTE $0xc3 RET @@ -1406,21 +1789,23 @@ TEXT ·movingAverageFloat32SSE2(SB), NOSPLIT|NOFRAME, $0-56 MOVQ width+48(FP), R8 QUAD $0x294cc09e0fc0854d QUAD $0xc10841c19c0f41c1 - QUAD $0x8d48000000e3850f - QUAD $0x4e0f48ca39480141 - QUAD $0x00cf8e0fc08548c2 - QUAD $0x4853e58948550000 - QUAD $0xc02a0f49f3f8e483 - QUAD $0x83c28944ff488d49 - QUAD $0xfffffff8b94907e2 - QUAD $0x4cc1214d7fffffff - QUAD $0x22ebd231451c468d - QUAD $0x0000000000841f0f - QUAD $0x110f42f3c85e0ff3 - QUAD $0xc08349c2ff49970c - QUAD $0xc2394904c6834804 - QUAD $0xdb3145c9570f7974 - QUAD $0x2e664f7207f98348 + QUAD $0x4855000000f5850f + QUAD $0xe48348535641e589 + QUAD $0xfffffffff8b848f8 + QUAD $0x4801498d4c7fffff + QUAD $0x8349ca4e0f4cca39 + QUAD $0x000000cc8d0f10f9 + QUAD $0xb88d0fc9394cc931 + QUAD $0x49f3c0570f000000 + QUAD $0x45ff508d49c02a0f + QUAD $0x214c07e28341c289 + QUAD $0xc083498e048d4cc0 + QUAD $0x901aeb8e348d481c + QUAD $0x0c110ff3c85e0ff3 + QUAD $0x04c08349c1ff488f + QUAD $0x74c9394c04c68348 + QUAD $0x48db3145c9570f7a + QUAD $0x2e6666507207fa83 QUAD $0x0000000000841f0f QUAD $0xf3e4984c580f43f3 QUAD $0x43f3e8984c580f43 @@ -1429,14 +1814,50 @@ TEXT ·movingAverageFloat32SSE2(SB), NOSPLIT|NOFRAME, $0-56 QUAD $0x4c580f43f3f4984c QUAD $0x984c580f43f3f898 QUAD $0x49980c580f43f3fc - QUAD $0xc075d9394d08c383 - QUAD $0x1c8d4e8b74d28548 + QUAD $0xc075d8394c08c383 + QUAD $0x1c8d4e8b74d2854d QUAD $0x0000441f0fdb319e QUAD $0xff489b0c580f41f3 - QUAD $0x6de9f275da3948c3 - QUAD $0x5bf8658d48ffffff - BYTE $0x5d - BYTE $0xc3 + QUAD $0x6de9f275da3949c3 + QUAD $0x5bf0658d48ffffff + QUAD $0x2a0f49f3c35d5e41 + QUAD $0x508d4800c0c60fc0 + QUAD $0x34568d4cc2214c06 + QUAD $0xdb314500000010bb + QUAD $0x0000441f0f6670eb + QUAD $0x280fe3280fec280f + QUAD $0x9e348d4ed1280fda + QUAD $0xca580f9e0c100f41 + QUAD $0x580f109e54100f41 + QUAD $0x0f209e5c100f41d3 + QUAD $0x309e64100f41dc58 + QUAD $0x0f42c85e0fe5580f + QUAD $0x0f42d05e0f9f0c11 + QUAD $0x42d85e0f109f5411 + QUAD $0xe05e0f209f5c110f + QUAD $0x8d48309f64110f42 + QUAD $0x894940c283491059 + QUAD $0xfea88f0fcb394ccb + QUAD $0xc9570fd98948ffff + QUAD $0xdb570fd2570fdb31 + QUAD $0x0f01f88349e4570f + QUAD $0x2e6666ffffff7b84 + QUAD $0x0000000000841f0f + QUAD $0x580fcc9a6c100f41 + QUAD $0x0fd09a4c100f41e9 + QUAD $0xdc9a6c100f41cd58 + QUAD $0x9a54100f41ea580f + QUAD $0x6c100f41d5580fe0 + QUAD $0x100f41eb580fec9a + QUAD $0x0f41f4580ffc9a74 + QUAD $0x41dd580ff09a5c10 + QUAD $0x48e6580f9a24100f + QUAD $0xb075da394802c383 + QUAD $0xff16850f01c0f641 + LONG $0x44e9ffff + BYTE $0xff + BYTE $0xff + BYTE $0xff RET // func shiftDivFloat32SSE2(dst []float32, a []float32, shift float32, denom float32) @@ -1502,21 +1923,59 @@ TEXT ·movingAverageFloat64SSE2(SB), NOSPLIT|NOFRAME, $0-56 MOVQ width+48(FP), R8 QUAD $0x294cc09e0fc0854d QUAD $0xc10841c19c0f41c1 - QUAD $0x8d48000000e3850f - QUAD $0x4e0f48ca39480141 - QUAD $0x00cf8e0fc08548c2 - QUAD $0x4853e58948550000 - QUAD $0xc02a0f49f2f8e483 - QUAD $0x83c28944ff488d49 - QUAD $0xfffffff8b94907e2 - QUAD $0x4cc1214d7fffffff - QUAD $0x22ebd2314538468d + QUAD $0x485500000213850f + QUAD $0x48f8e4834853e589 + QUAD $0x0f48ca394801418d + QUAD $0x077d10f88348c24e + QUAD $0xf20000011ce9c931 + QUAD $0xc0140f66c02a0f49 + QUAD $0x0010ba4170568d48 + QUAD $0x001f0fc931450000 + QUAD $0x45ff570f66d1894c + QUAD $0x66c0570f4566d231 + QUAD $0x66ed570f66f6570f + QUAD $0x66db570f66e4570f + QUAD $0x66c9570f66d2570f QUAD $0x0000000000841f0f - QUAD $0x110f42f2c85e0ff2 - QUAD $0xc08349c2ff49d70c - QUAD $0xc2394908c6834808 - QUAD $0x3145c9570f667974 - QUAD $0x664e7207f98348db + QUAD $0x6690d24c100f4666 + QUAD $0x100f4666f9580f41 + QUAD $0xc1580f4566a0d24c + QUAD $0x66b0d24c100f4666 + QUAD $0x100f4666f1580f41 + QUAD $0xe9580f4166c0d24c + QUAD $0x66d0d24c100f4666 + QUAD $0x100f4666e1580f41 + QUAD $0xd9580f4166e0d24c + QUAD $0x66f0d24c100f4666 + QUAD $0x100f4666d1580f41 + QUAD $0x49c9580f4166d20c + QUAD $0x669975d0394dc2ff + QUAD $0x3c110f4266f85e0f + QUAD $0x4666c05e0f4466cf + QUAD $0x5e0f6610cf44110f + QUAD $0x20cf74110f4266f0 + QUAD $0x110f4266e85e0f66 + QUAD $0x66e05e0f6630cf6c + QUAD $0x0f6640cf64110f42 + QUAD $0xcf5c110f4266d85e + QUAD $0x0f4266d05e0f6650 + QUAD $0xc85e0f6660cf5411 + QUAD $0x4c70cf4c110f4266 + QUAD $0x4980ea834810518d + QUAD $0xfd8e0fc23949c989 + QUAD $0x8d0fc13948fffffe + QUAD $0xf2c0570f000000c7 + QUAD $0xff508d49c02a0f49 + QUAD $0x4907e18341c18945 + QUAD $0xfffffffffffff8ba + QUAD $0xce048d4cc2214d7f + QUAD $0xce348d4838c08349 + QUAD $0x0000441f0f661feb + QUAD $0x0c110ff2c85e0ff2 + QUAD $0x08c08349c1ff48cf + QUAD $0x74c1394808c68348 + QUAD $0xdb3145c9570f667a + QUAD $0x2e664f7207fa8348 QUAD $0x0000000000841f0f QUAD $0xf2c8d84c580f43f2 QUAD $0x43f2d0d84c580f43 @@ -1525,11 +1984,11 @@ TEXT ·movingAverageFloat64SSE2(SB), NOSPLIT|NOFRAME, $0-56 QUAD $0x4c580f43f2e8d84c QUAD $0xd84c580f43f2f0d8 QUAD $0x49d80c580f43f2f8 - QUAD $0xc075d9394d08c383 - QUAD $0x1c8d4e8b74d28548 + QUAD $0xc075da394d08c383 + QUAD $0x1c8d4e8b74c9854d QUAD $0x0000441f0fdb31de QUAD $0xff48db0c580f41f2 - QUAD $0x6de9f275da3948c3 + QUAD $0x6de9f275d93949c3 QUAD $0x5bf8658d48ffffff BYTE $0x5d BYTE $0xc3 diff --git a/internal/arm64/numeric_neon_arm64.s b/internal/arm64/numeric_neon_arm64.s index 3df269c..3f7dc75 100644 --- a/internal/arm64/numeric_neon_arm64.s +++ b/internal/arm64/numeric_neon_arm64.s @@ -233,20 +233,74 @@ TEXT ·polyEvalFloat32NEON(SB), NOSPLIT|NOFRAME, $0-72 MOVD coeffs_len+56(FP), R5 WORD $0xeb04007f WORD $0x9a84b068 + WORD $0xb40000c5 + WORD $0x8b05084a + WORD $0xf100411f + WORD $0x5400012a + WORD $0xaa1f03e9 + WORD $0x14000034 WORD $0xf100051f - WORD $0xb4000305 - WORD $0x540007eb - WORD $0x8b05084b - WORD $0xf10008bf - WORD $0xbc5fc160 - WORD $0x5400030b + WORD $0x54000d8b + WORD $0xf100211f + WORD $0x54000b62 + WORD $0xaa1f03e9 + WORD $0x14000063 + WORD $0xd1001149 + WORD $0xf10004ab + WORD $0x4d40c920 + WORD $0x5400044d + WORD $0xaa1f03ec + WORD $0x5280020d + WORD $0x8b0c082e + WORD $0x4ea01c10 + WORD $0x4ea01c07 + WORD $0x4ea01c06 + WORD $0x4ea01c02 + WORD $0xaa0d03e9 + WORD $0xad400dc1 + WORD $0xd100214d + WORD $0xad4115c4 + WORD $0xaa0b03ee + WORD $0xbc5fc5b1 + WORD $0x6e21de10 + WORD $0x6e23dce7 + WORD $0x6e24dcc6 + WORD $0x6e25dc42 + WORD $0x4e040631 + WORD $0xd10005ce + WORD $0xf10001df + WORD $0x4e30d630 + WORD $0x4e27d627 + WORD $0x4e26d626 + WORD $0x4e22d622 + WORD $0x54fffe8c + WORD $0x8b0c080e + WORD $0x9100412d + WORD $0xaa0903ec + WORD $0xeb0801bf + WORD $0xad001dd0 + WORD $0xad0109c6 + WORD $0x54fffc6d + WORD $0x14000009 WORD $0xaa1f03e9 - WORD $0xd10004aa - WORD $0xd100216b + WORD $0x9100800b + WORD $0x9100812c + WORD $0x91004129 + WORD $0xad3f0160 + WORD $0xeb08019f + WORD $0xac820160 + WORD $0x54ffff6d + WORD $0xeb09010b + WORD $0x5400072d + WORD $0xbc5fc140 + WORD $0xf10008bf + WORD $0x5400024b + WORD $0xd10004ab + WORD $0xd100214a WORD $0x1e204002 WORD $0xbc697821 - WORD $0xaa0b03ec - WORD $0xaa0a03ed + WORD $0xaa0a03ec + WORD $0xaa0b03ed WORD $0x1e220822 WORD $0xbc5fc583 WORD $0xd10005ad @@ -257,38 +311,34 @@ TEXT ·polyEvalFloat32NEON(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x91000529 WORD $0xeb08013f WORD $0x54fffe61 - WORD $0x14000029 - WORD $0x5400050b - WORD $0xf100211f - WORD $0x540000e2 - WORD $0xaa1f03e9 - WORD $0x1400000f - WORD $0xf100211f - WORD $0x54000262 - WORD $0xaa1f03e9 - WORD $0x1400001b - WORD $0x6f00e400 - WORD $0x927ded09 - WORD $0x9100400a - WORD $0x927ded0b - WORD $0xf100216b - WORD $0xad3f8140 - WORD $0x9100814a + WORD $0x14000025 + WORD $0xf100217f + WORD $0x540001c3 + WORD $0x9240090a + WORD $0x4e040401 + WORD $0x8b09080c + WORD $0xcb0a016b + WORD $0x8b0a012d + WORD $0x8b0b0129 + WORD $0x9100418b + WORD $0xcb0801ac + WORD $0xb100218c + WORD $0xad3f8561 + WORD $0x9100816b WORD $0x54ffffa1 - WORD $0xeb09011f - WORD $0x540002c0 + WORD $0xb40002ca WORD $0x8b09080a WORD $0xcb090108 WORD $0xf1000508 - WORD $0xb800455f + WORD $0xbc004540 WORD $0x54ffffc1 WORD $0x14000010 - WORD $0x4e040401 + WORD $0x6f00e400 WORD $0x927ded09 WORD $0x9100400a WORD $0x927ded0b WORD $0xf100216b - WORD $0xad3f8541 + WORD $0xad3f8140 WORD $0x9100814a WORD $0x54ffffa1 WORD $0xeb09011f @@ -296,7 +346,7 @@ TEXT ·polyEvalFloat32NEON(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x8b09080a WORD $0xcb090108 WORD $0xf1000508 - WORD $0xbc004540 + WORD $0xb800455f WORD $0x54ffffc1 WORD $0xd65f03c0 RET @@ -309,41 +359,346 @@ TEXT ·convolveFloat32NEON(SB), NOSPLIT|NOFRAME, $0-72 MOVD dst_len+8(FP), R3 MOVD sig_len+32(FP), R4 MOVD ker_len+56(FP), R5 - WORD $0xb4000ae5 + WORD $0xb40031e5 WORD $0xeb050088 - WORD $0x54000aab + WORD $0x540031ab + WORD $0xd10443ff WORD $0xeb08007f + WORD $0x6d0c3bef WORD $0x9a88d468 - WORD $0xf100051f - WORD $0x54000a2b - WORD $0xf10004bf - WORD $0x5400078b - WORD $0x8b05084b + WORD $0x6d0d33ed + WORD $0xf100411f + WORD $0x6d0e2beb + WORD $0x6d0f23e9 + WORD $0xf90083fd + WORD $0x5400006a WORD $0xaa1f03e9 - WORD $0x927decaa - WORD $0x9100402c + WORD $0x14000128 + WORD $0xf10000bf + WORD $0x540023ad + WORD $0x8b050849 + WORD $0xaa1f03ef + WORD $0x927ef0aa + WORD $0x9100802b WORD $0xd100104d - WORD $0xd100416b + WORD $0x9100f02e + WORD $0xd100412c + WORD $0x52800210 + WORD $0x1400000e + WORD $0x8b0f0811 + WORD $0x91004130 + WORD $0x9101016b + WORD $0xeb08021f + WORD $0x910101ce + WORD $0xaa0903ef + WORD $0xbd000224 + WORD $0x3c804223 + WORD $0x3c814225 + WORD $0x3c824226 + WORD $0x2d06f637 + WORD $0xbd003e34 + WORD $0x5400220c + WORD $0xf10010bf + WORD $0xaa1003e9 + WORD $0x54000142 + WORD $0x6f00e403 + WORD $0x2f00e414 + WORD $0xaa1f03f1 + WORD $0x2f00e41d + WORD $0x2f00e417 + WORD $0x2f00e404 + WORD $0x6f00e405 + WORD $0x6f00e406 + WORD $0x140000de + WORD $0x2f00e414 + WORD $0x2f00e41d + WORD $0xaa0b03e4 + WORD $0x2f00e417 + WORD $0x2f00e407 + WORD $0xaa0c03f0 + WORD $0x2f00e402 + WORD $0x2f00e416 + WORD $0x927ef0b1 + WORD $0x2f00e406 + WORD $0x2f00e40f + WORD $0xaa0b03e3 + WORD $0x2f00e413 + WORD $0x2f00e415 + WORD $0x2f00e405 + WORD $0x2f00e410 + WORD $0x2f00e411 + WORD $0x2f00e400 + WORD $0x2f00e403 + WORD $0x2f00e404 + WORD $0x3cdf0618 + WORD $0x3cc1c099 + WORD $0x3cc1809a + WORD $0x3d802be0 + WORD $0x4ea21c52 + WORD $0x4ea00b18 + WORD $0xad0157f1 + WORD $0x3cc0c089 + WORD $0x4eaf1df5 + WORD $0xf1001231 + WORD $0x6e184318 + WORD $0x6e39df0b + WORD $0x3cc14099 + WORD $0x6e3adf0c + WORD $0x3cc10c7a + WORD $0x6e39df0d + WORD $0x3cde0099 + WORD $0x6e3adf0e + WORD $0x3cde409a + WORD $0x5e0c057b + WORD $0x1e2b2a80 + WORD $0x6e39df01 + WORD $0x5e0c059c + WORD $0x1e2c2ba2 + WORD $0x5e140568 + WORD $0x5e0c05bd + WORD $0x5e14059f + WORD $0x5e1c056b + WORD $0x5e0c05de + WORD $0x1e2e28e7 + WORD $0x5e1405aa + WORD $0x3d802fe1 + WORD $0x6e3adf01 + WORD $0x1e3b2800 + WORD $0x1e3c2842 + WORD $0x5e1c058c + WORD $0x3cde809b + WORD $0x5e1405cf + WORD $0x6e29df14 + WORD $0x5e1c05d9 + WORD $0x1e3e28e7 + WORD $0x6e3bdf11 + WORD $0x4eb01e1e + WORD $0x3d8027e1 + WORD $0x1e2d2ae1 + WORD $0x1e282800 + WORD $0x1e3f285a + WORD $0x5e1c05ad + WORD $0x3cc08090 + WORD $0x1e342a57 + WORD $0x3cdf8088 + WORD $0x3cdec09c + WORD $0x6e30df1b + WORD $0x1e2f28e7 + WORD $0xad02c7f4 + WORD $0x1e3d2821 + WORD $0x1e2b2800 + WORD $0x3cc04091 + WORD $0x5e0c0690 + WORD $0xad7fbc89 + WORD $0x6e31df02 + WORD $0x6e28df1f + WORD $0x6e3cdf1c + WORD $0x1e3b2ace + WORD $0x5e0c0774 + WORD $0x3cdf4096 + WORD $0x1e2a2821 + WORD $0xbd010fe0 + WORD $0x1e2c2b40 + WORD $0x1e302af0 + WORD $0x3cdfc08a + WORD $0x6e29df1d + WORD $0x1e2228d2 + WORD $0x1e204086 + WORD $0x4eb31e64 + WORD $0x5e0c0453 + WORD $0x6e2adf08 + WORD $0xad000bfb + WORD $0xbd010be0 + WORD $0x1e2d2820 + WORD $0x4ebb1f6a + WORD $0x6e36df1b + WORD $0x6e2fdf16 + WORD $0x5e0c07fa + WORD $0x5e0c0798 + WORD $0x5e0c07af + WORD $0xaa0303e4 + WORD $0x1e28288c + WORD $0x4ea81d0b + WORD $0x5e0c0509 + WORD $0xbd008fe0 + WORD $0x1e3928e0 + WORD $0x5e0c0767 + WORD $0x1e292989 + WORD $0x5e1406cc + WORD $0x3d801fe0 + WORD $0x1e3429c0 + WORD $0x4ebf1ff4 + WORD $0x2d0943e0 + WORD $0x1e332a40 + WORD $0x1e362ab3 + WORD $0x5e0c06d2 + WORD $0xbd0047e0 + WORD $0xad4103e1 + WORD $0x1e3f2815 + WORD $0xad4483f1 + WORD $0x1e3c282e + WORD $0x3dc01be1 + WORD $0x1e3b28bf + WORD $0x4ebd1fa5 + WORD $0x1e322a7d + WORD $0x1e212800 + WORD $0x5e0c0437 + WORD $0x1e3a2aa8 + WORD $0x5e140455 + WORD $0x4eb41e82 + WORD $0x1e252bcd + WORD $0x1e31287e + WORD $0x4ea11c23 + WORD $0x3dc02fe1 + WORD $0x1e272bf9 + WORD $0x1e3829df + WORD $0x1e372813 + WORD $0x3dc017e0 + WORD $0x5e140557 + WORD $0x1e2128c4 + WORD $0x5e0c0430 + WORD $0x5e140578 + WORD $0x5e14040a + WORD $0x4ea11c26 + WORD $0x5e0c0621 + WORD $0x1e2f29ba + WORD $0x5e14078d + WORD $0x5e1404ae + WORD $0x5e14076f + WORD $0x5e1c079c + WORD $0x1e302887 + WORD $0x5e140690 + WORD $0x4eab1d74 + WORD $0xbd404feb + WORD $0x1e212bd2 + WORD $0x4ea51cbe + WORD $0x5e1404c5 + WORD $0x5e140624 + WORD $0x5e140461 + WORD $0x1e2a296a + WORD $0x1e2c2bab + WORD $0x1e2f2b2c + WORD $0x1e302910 + WORD $0x5e1c0459 + WORD $0x5e1c06dd + WORD $0x5e1c07de + WORD $0x5e1c04c8 + WORD $0xbd00a3ea + WORD $0x2d488bea + WORD $0x1e3d296f + WORD $0xbd410bfd + WORD $0x1e372857 + WORD $0x1e352955 + WORD $0x1e38292a + WORD $0x1e2e2b58 + WORD $0x1e2d2bee + WORD $0x1e212a6d + WORD $0x1e242a41 + WORD $0x1e2528f2 + WORD $0xad4013e5 + WORD $0x5e1c0407 + WORD $0x5e1c069a + WORD $0x5e1c077f + WORD $0x5e1c0629 + WORD $0x5e1c047b + WORD $0xbd40a3e0 + WORD $0x5e1c0484 + WORD $0x5e1c04a5 + WORD $0x1e3c29d1 + WORD $0xbd410ff4 + WORD $0x1e272802 + WORD $0x1e3a2953 + WORD $0x3dc01fe7 + WORD $0x1e3b29a0 + WORD $0x1e292823 + WORD $0x1e252af6 + WORD $0x1e242aa6 + WORD $0x1e392a15 + WORD $0x1e3f2985 + WORD $0x1e3e2b10 + WORD $0x1e282a44 + WORD $0xbd408ff7 + WORD $0x54ffe881 + WORD $0x6e0c0403 + WORD $0x6e0c06a5 + WORD $0xeb0a00bf + WORD $0x6e0c06c6 + WORD $0x927ef0b1 + WORD $0x6e140623 + WORD $0x6e140665 + WORD $0x6e140446 + WORD $0x6e1c0603 + WORD $0x6e1c05e5 + WORD $0x6e1c04e6 + WORD $0x54ffe160 + WORD $0xd37ef630 + WORD $0xcb1100b1 + WORD $0xaa0e03e3 + WORD $0x8b100064 + WORD $0xbc7179a0 + WORD $0xf1000631 + WORD $0x2d7f4891 + WORD $0xbc5c4081 + WORD $0x3cdc8082 + WORD $0x3cdd8087 + WORD $0x91001063 + WORD $0x3cde8090 + WORD $0xbd400093 + WORD $0x1e210801 + WORD $0x4f809042 + WORD $0x4f8090e7 + WORD $0x1e310811 + WORD $0x4f809210 + WORD $0x1e320812 + WORD $0x1e330800 + WORD $0x1e212884 + WORD $0x4e22d463 + WORD $0x4e27d4a5 + WORD $0x1e312af7 + WORD $0x4e30d4c6 + WORD $0x1e322bbd + WORD $0x1e202a94 + WORD $0x54fffd01 + WORD $0x17fffeee + WORD $0x6f00e400 + WORD $0xaa1f03e9 + WORD $0x9100800a + WORD $0x9100812b + WORD $0x91004129 + WORD $0xad3f0140 + WORD $0xeb08017f + WORD $0xac820140 + WORD $0x54ffff6d + WORD $0xeb09010b + WORD $0x54000a4d + WORD $0xf10004bf + WORD $0x5400078b + WORD $0x8b09082a + WORD $0x8b05084d + WORD $0x927decab + WORD $0xd100104e + WORD $0x9100414c + WORD $0xd10041ad WORD $0x14000007 WORD $0xbc297800 WORD $0x91000529 WORD $0x9100118c WORD $0xeb08013f - WORD $0x91001021 - WORD $0x54000840 + WORD $0x9100114a + WORD $0x54000860 WORD $0x2f00e400 WORD $0xf10020bf WORD $0x54000062 - WORD $0xaa1f03ef + WORD $0xaa1f03f0 WORD $0x14000021 - WORD $0xaa0c03ee - WORD $0xaa0b03ef - WORD $0x927decb0 + WORD $0xaa0d03ef + WORD $0xaa0c03f0 + WORD $0x927decb1 WORD $0xad7f85e2 - WORD $0xd1002210 - WORD $0xad7f91c3 + WORD $0xd1002231 + WORD $0xad7f9203 + WORD $0x91008210 WORD $0xd10081ef - WORD $0x910081ce WORD $0x4ea00821 WORD $0x4ea00842 WORD $0x6e014021 @@ -364,38 +719,45 @@ TEXT ·convolveFloat32NEON(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x5e1c0441 WORD $0x1e232800 WORD $0x1e212800 - WORD $0xb5fffcf0 - WORD $0xeb0a00bf - WORD $0x927decaf + WORD $0xb5fffcf1 + WORD $0xeb0b00bf + WORD $0x927decb0 WORD $0x54fffac0 - WORD $0x8b0f082e - WORD $0xcb0f00af - WORD $0xbc4045c1 - WORD $0xbc6f79a2 - WORD $0xf10005ef + WORD $0x8b10094f + WORD $0xcb1000b0 + WORD $0xbc4045e1 + WORD $0xbc7079c2 + WORD $0xf1000610 WORD $0x1e220821 WORD $0x1e212800 WORD $0x54ffff61 WORD $0x17ffffcd - WORD $0xf100211f - WORD $0x54000062 - WORD $0xaa1f03e9 - WORD $0x1400000b + WORD $0xf100217f + WORD $0x540001c3 WORD $0x6f00e400 - WORD $0x927ded09 - WORD $0x9100400a - WORD $0x927ded0b - WORD $0xf100216b - WORD $0xad3f8140 - WORD $0x9100814a + WORD $0x9240090a + WORD $0x8b09080c + WORD $0xcb0a016b + WORD $0x8b0a012d + WORD $0x8b0b0129 + WORD $0x9100418b + WORD $0xcb0801ac + WORD $0xb100218c + WORD $0xad3f8160 + WORD $0x9100816b WORD $0x54ffffa1 - WORD $0xeb09011f - WORD $0x540000c0 + WORD $0xb40000ca WORD $0x8b09080a WORD $0xcb090108 WORD $0xf1000508 WORD $0xb800455f WORD $0x54ffffc1 + WORD $0x6d4f23e9 + WORD $0xf94083fd + WORD $0x6d4e2beb + WORD $0x6d4d33ed + WORD $0x6d4c3bef + WORD $0x910443ff WORD $0xd65f03c0 RET @@ -407,38 +769,345 @@ TEXT ·correlateFloat32NEON(SB), NOSPLIT|NOFRAME, $0-72 MOVD dst_len+8(FP), R3 MOVD sig_len+32(FP), R4 MOVD ker_len+56(FP), R5 - WORD $0xb4000a45 + WORD $0xb4003185 WORD $0xeb050088 - WORD $0x54000a0b + WORD $0x5400314b + WORD $0xd10503ff WORD $0xeb08007f + WORD $0x6d0f3bef WORD $0x9a88d468 - WORD $0xf100051f - WORD $0x5400098b + WORD $0x6d1033ed + WORD $0xf100411f + WORD $0x6d112beb + WORD $0x6d1223e9 + WORD $0xf9009bfd + WORD $0x5400006a + WORD $0xaa1f03e9 + WORD $0x1400012a + WORD $0xf10000bf + WORD $0x540023ed + WORD $0xaa1f03ed + WORD $0x927ef0aa + WORD $0x9100802b + WORD $0x5280020e + WORD $0xaa0103ec + WORD $0x1400000e + WORD $0x8b0d080f + WORD $0x9100412e + WORD $0x9101016b + WORD $0xeb0801df + WORD $0x9101018c + WORD $0xaa0903ed + WORD $0xbd0001e4 + WORD $0x3c8041e3 + WORD $0x3c8141e5 + WORD $0x3c8241e6 + WORD $0x2d06f5f5 + WORD $0xbd003df4 + WORD $0x540022ac + WORD $0xf10010bf + WORD $0xaa0e03e9 + WORD $0x54000142 + WORD $0x6f00e403 + WORD $0x2f00e414 + WORD $0xaa1f03ef + WORD $0x2f00e41d + WORD $0x2f00e415 + WORD $0x2f00e404 + WORD $0x6f00e405 + WORD $0x6f00e406 + WORD $0x140000e1 + WORD $0x2f00e400 + WORD $0x2f00e414 + WORD $0xaa0b03f1 + WORD $0x2f00e41d + WORD $0x2f00e415 + WORD $0xaa0203ee + WORD $0x2f00e407 + WORD $0x2f00e416 + WORD $0x927ef0af + WORD $0x2f00e412 + WORD $0x2f00e406 + WORD $0xaa0b03f0 + WORD $0x3d8037e0 + WORD $0x2f00e418 + WORD $0x2f00e413 + WORD $0x2f00e405 + WORD $0x2f00e410 + WORD $0x2f00e411 + WORD $0x2f00e400 + WORD $0x2f00e403 + WORD $0x2f00e404 + WORD $0x3d8033f8 + WORD $0x3cc105d8 + WORD $0x3cc1c239 + WORD $0x3cc1823a + WORD $0xf10011ef + WORD $0x3d802be0 + WORD $0x3cc0c229 + WORD $0x6e39df0b + WORD $0x3cc14239 + WORD $0x6e3adf0c + WORD $0x3cc10e1a + WORD $0x3d8023f3 + WORD $0x6e39df0d + WORD $0x3cde0239 + WORD $0x3d800bf0 + WORD $0x6e3adf0e + WORD $0x3cde423a + WORD $0x3cc04230 + WORD $0x5e0c057b + WORD $0x1e2b2a80 + WORD $0x6e39df01 + WORD $0x5e0c059c + WORD $0x1e2c2ba2 + WORD $0x5e140568 + WORD $0x5e0c05bd + WORD $0x5e14059f + WORD $0x5e1c056b + WORD $0x5e0c05de + WORD $0x1e2e28e7 + WORD $0x5e1405aa + WORD $0x3d802fe1 + WORD $0x6e3adf01 + WORD $0x1e3b2800 + WORD $0x1e3c2842 + WORD $0x5e1c058c + WORD $0x3cde823b + WORD $0x5e1405cf + WORD $0x6e30df10 + WORD $0x4eb11e3a + WORD $0x6e3bdf19 + WORD $0x1e3e28e7 + WORD $0x3cdec23c + WORD $0x3d8027e1 + WORD $0x1e2d2aa1 + WORD $0x1e282800 + WORD $0x1e3f2842 + WORD $0x5e1c05ad + WORD $0x3cc08228 + WORD $0x5e1c05d5 + WORD $0x3cdf023e + WORD $0x3cdf823f + WORD $0x6e29df13 + WORD $0x6e28df14 + WORD $0x1e3d2821 + WORD $0x1e2b2800 + WORD $0x1e2f28e7 + WORD $0x3cdf422e + WORD $0x6e3cdf1b + WORD $0x6e3edf1c + WORD $0x6e3fdf1e + WORD $0x3d8003f0 + WORD $0x3d803bf4 + WORD $0x1e332acf + WORD $0x1e342a51 + WORD $0x1e2a2821 + WORD $0xbd013fe0 + WORD $0x1e2c2840 + WORD $0x5e0c0677 + WORD $0x5e0c0694 + WORD $0xad0267f3 + WORD $0x1e3028d2 + WORD $0x5e0c0613 + WORD $0x3cdfc22a + WORD $0x3dc00236 + WORD $0x6e2edf19 + WORD $0x5e0c077f + WORD $0xbd013be0 + WORD $0x1e2d2820 + WORD $0x6e2adf1d + WORD $0x1e3729e1 + WORD $0x6e36df16 + WORD $0x4ebb1f6a + WORD $0x3dc017e2 + WORD $0x5e0c07c9 + WORD $0x5e0c078f + WORD $0x1e3928a6 + WORD $0xaa1003f1 + WORD $0xbd007fe0 + WORD $0x1e3528e0 + WORD $0x4ebd1fb8 + WORD $0x5e0c06d7 + WORD $0x5e0c0727 + WORD $0x1e2a2b4e + WORD $0x4ebe1fda + WORD $0x3d801be0 + WORD $0x1e342a20 + WORD $0x5e0c07b1 + WORD $0x1e2728c8 + WORD $0x5e1407c7 + WORD $0x1e3f29df + WORD $0x5e14078e + WORD $0x5e1c075a + WORD $0x2d0707e0 + WORD $0x1e332a40 + WORD $0xbd001fe0 + WORD $0xad4607e0 + WORD $0x1e382814 + WORD $0xad444fe0 + WORD $0x1e36283b + WORD $0xad454be1 + WORD $0x1e3e2815 + WORD $0x3dc00be0 + WORD $0x1e332863 + WORD $0x1e222825 + WORD $0x5e0c0441 + WORD $0x1e312a91 + WORD $0x1e3c280d + WORD $0x5e0c0660 + WORD $0x1e372b6b + WORD $0x4eaa1d5e + WORD $0x1e322884 + WORD $0x5e0c064c + WORD $0x1e292aa9 + WORD $0x5e140615 + WORD $0x5e140646 + WORD $0x1e2128b7 + WORD $0x3dc03be1 + WORD $0xbd00c3f1 + WORD $0x1e202874 + WORD $0x3dc013e0 + WORD $0x1e2f29bd + WORD $0x5e140430 + WORD $0x5e14054d + WORD $0xbd403fea + WORD $0x5e140403 + WORD $0x1e2c2891 + WORD $0x5e140704 + WORD $0x5e1406cc + WORD $0x5e14072f + WORD $0x5e140665 + WORD $0x5e140441 + WORD $0x4ebc1f9b + WORD $0xbd401ffc + WORD $0x1e2e2bae + WORD $0x5e1c071d + WORD $0x1e232943 + WORD $0xbd403bea + WORD $0x1e352b95 + WORD $0x1e2c296b + WORD $0x1e2f290c + WORD $0x1e2d2bef + WORD $0x1e302950 + WORD $0x5e1c06df + WORD $0x1e212aed + WORD $0x1e252a81 + WORD $0x1e262a37 + WORD $0x3dc003e5 + WORD $0x5e1c0411 + WORD $0x5e1c07dc + WORD $0x5e1c077e + WORD $0x5e1c04a6 + WORD $0x5e1c0648 + WORD $0x5e1c045b + WORD $0xbd00d3f0 + WORD $0xbd40c3f0 + WORD $0xbd413ff4 + WORD $0xbd40d3e0 + WORD $0x1e242a0a + WORD $0x1e272924 + WORD $0x3dc03be7 + WORD $0x5e1c0730 + WORD $0x5e1c0669 + WORD $0x1e312876 + WORD $0x5e1c04e7 + WORD $0x1e262aa6 + WORD $0x1e3c29f1 + WORD $0xbd407ff5 + WORD $0x1e3d2958 + WORD $0x1e3a2893 + WORD $0x1e282ae4 + WORD $0x1e302985 + WORD $0x1e3e29d0 + WORD $0x1e292823 + WORD $0x1e272812 + WORD $0x1e3f2960 + WORD $0x3dc01be7 + WORD $0xbd413bfd + WORD $0x3d8037e0 + WORD $0x1e3b29a0 + WORD $0x54ffe861 + WORD $0x6e0c0403 + WORD $0x6e0c0665 + WORD $0x3dc037e0 + WORD $0x6e0c0646 + WORD $0xeb0a00bf + WORD $0x927ef0af + WORD $0x6e140623 + WORD $0x6e140705 + WORD $0x6e1406c6 + WORD $0x6e1c0603 + WORD $0x6e1c0405 + WORD $0x6e1c04e6 + WORD $0x54ffe100 + WORD $0xd37ef5ee + WORD $0xcb0f00af + WORD $0xaa0c03f0 + WORD $0xaa0203f1 + WORD $0x8b0e0203 + WORD $0xbc6e6a20 + WORD $0xf10005ef + WORD $0x2d46c871 + WORD $0xbd400061 + WORD $0x3cc04062 + WORD $0x3cc14067 + WORD $0x91001231 + WORD $0x3cc24070 + WORD $0xbd403c73 + WORD $0x1e210801 + WORD $0x4f809042 + WORD $0x4f8090e7 + WORD $0x1e310811 + WORD $0x4f809210 + WORD $0x1e320812 + WORD $0x1e330800 + WORD $0x91001210 + WORD $0x1e212884 + WORD $0x4e22d463 + WORD $0x4e27d4a5 + WORD $0x1e312ab5 + WORD $0x4e30d4c6 + WORD $0x1e322bbd + WORD $0x1e202a94 + WORD $0x54fffce1 + WORD $0x17fffee9 + WORD $0x6f00e400 + WORD $0xaa1f03e9 + WORD $0x9100800a + WORD $0x9100812b + WORD $0x91004129 + WORD $0xad3f0140 + WORD $0xeb08017f + WORD $0xac820140 + WORD $0x54ffff6d + WORD $0xeb09010b + WORD $0x540009ad WORD $0xf10004bf WORD $0x540006eb - WORD $0xaa1f03e9 - WORD $0x927decaa - WORD $0x9100402b + WORD $0x8b09082a + WORD $0x927decab + WORD $0x9100414c WORD $0x14000007 WORD $0xbc297800 WORD $0x91000529 - WORD $0x9100116b + WORD $0x9100118c WORD $0xeb08013f - WORD $0x91001021 - WORD $0x54000800 + WORD $0x9100114a + WORD $0x54000820 WORD $0x2f00e400 WORD $0xf10020bf WORD $0x54000062 - WORD $0xaa1f03ec + WORD $0xaa1f03ed WORD $0x1400001d - WORD $0xaa0b03ec - WORD $0x9100404d - WORD $0x927decae - WORD $0xad7f91a1 - WORD $0xf10021ce - WORD $0xad7f8d82 + WORD $0xaa0c03ed + WORD $0x9100404e + WORD $0x927decaf + WORD $0xad7f91c1 + WORD $0xf10021ef + WORD $0xad7f8da2 + WORD $0x910081ce WORD $0x910081ad - WORD $0x9100818c WORD $0x6e21dc41 WORD $0x5e0c0422 WORD $0x1e212800 @@ -456,39 +1125,46 @@ TEXT ·correlateFloat32NEON(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x1e232800 WORD $0x1e212800 WORD $0x54fffd61 - WORD $0xeb0a00bf - WORD $0x927decac + WORD $0xeb0b00bf + WORD $0x927decad WORD $0x54fffb40 - WORD $0xd37ef58e - WORD $0xcb0c00ac - WORD $0x8b0e004d - WORD $0x8b0e002e - WORD $0xbc4045c1 - WORD $0xf100058c - WORD $0xbc4045a2 + WORD $0xd37ef5af + WORD $0xcb0d00ad + WORD $0x8b0f004e + WORD $0x8b0f014f + WORD $0xbc4045e1 + WORD $0xf10005ad + WORD $0xbc4045c2 WORD $0x1e220821 WORD $0x1e212800 WORD $0x54ffff61 WORD $0x17ffffcf - WORD $0xf100211f - WORD $0x54000062 - WORD $0xaa1f03e9 - WORD $0x1400000b + WORD $0xf100217f + WORD $0x540001c3 WORD $0x6f00e400 - WORD $0x927ded09 - WORD $0x9100400a - WORD $0x927ded0b - WORD $0xf100216b - WORD $0xad3f8140 - WORD $0x9100814a + WORD $0x9240090a + WORD $0x8b09080c + WORD $0xcb0a016b + WORD $0x8b0a012d + WORD $0x8b0b0129 + WORD $0x9100418b + WORD $0xcb0801ac + WORD $0xb100218c + WORD $0xad3f8160 + WORD $0x9100816b WORD $0x54ffffa1 - WORD $0xeb09011f - WORD $0x540000c0 + WORD $0xb40000ca WORD $0x8b09080a WORD $0xcb090108 WORD $0xf1000508 WORD $0xb800455f WORD $0x54ffffc1 + WORD $0x6d5223e9 + WORD $0xf9409bfd + WORD $0x6d512beb + WORD $0x6d5033ed + WORD $0x6d4f3bef + WORD $0x910503ff WORD $0xd65f03c0 RET @@ -733,20 +1409,97 @@ TEXT ·polyEvalFloat64NEON(SB), NOSPLIT|NOFRAME, $0-72 MOVD coeffs_len+56(FP), R5 WORD $0xeb04007f WORD $0x9a84b068 + WORD $0xb40000c5 + WORD $0x8b050c4a + WORD $0xf100411f + WORD $0x5400012a + WORD $0xaa1f03e9 + WORD $0x1400004b WORD $0xf100051f - WORD $0xb4000305 - WORD $0x540007eb - WORD $0x8b050c4b - WORD $0xf10008bf - WORD $0xfc5f8160 - WORD $0x5400030b + WORD $0x5400106b + WORD $0xf100111f + WORD $0x54000e42 + WORD $0xaa1f03e9 + WORD $0x1400007a + WORD $0xfc5f8140 + WORD $0xf10004ab + WORD $0x5400066d + WORD $0x4e080400 + WORD $0xaa1f03ec + WORD $0x5280020d + WORD $0x8b0c0c2e + WORD $0x4ea01c18 + WORD $0x4ea01c17 + WORD $0x4ea01c16 + WORD $0x4ea01c15 + WORD $0xaa0d03e9 + WORD $0xad4009c1 + WORD $0x4ea01c14 + WORD $0xad4111c3 + WORD $0x4ea01c13 + WORD $0xad4219c5 + WORD $0x4ea01c12 + WORD $0xad4341c7 + WORD $0x4ea01c11 + WORD $0xd100414d + WORD $0xaa0b03ee + WORD $0xfc5f85b9 + WORD $0x6e61df18 + WORD $0x6e62def7 + WORD $0x6e63ded6 + WORD $0x6e64deb5 + WORD $0x6e65de94 + WORD $0x6e66de73 + WORD $0xd10005ce + WORD $0x6e67de52 + WORD $0x6e70de31 + WORD $0xf10001df + WORD $0x4e080739 + WORD $0x4e78d738 + WORD $0x4e77d737 + WORD $0x4e76d736 + WORD $0x4e75d735 + WORD $0x4e74d734 + WORD $0x4e73d733 + WORD $0x4e72d732 + WORD $0x4e71d731 + WORD $0x54fffd8c + WORD $0x8b0c0c0e + WORD $0x9100412d + WORD $0xaa0903ec + WORD $0xeb0801bf + WORD $0xad005dd8 + WORD $0xad0155d6 + WORD $0xad024dd4 + WORD $0xad0345d2 + WORD $0x54fffa6d + WORD $0x14000010 WORD $0xaa1f03e9 - WORD $0xd10004aa - WORD $0xd100416b + WORD $0x9101000b + WORD $0x9100812c + WORD $0x6d3c0160 + WORD $0x91004129 + WORD $0x6d3d0160 + WORD $0xeb08019f + WORD $0x6d3e0160 + WORD $0x6d3f0160 + WORD $0x6d000160 + WORD $0x6d010160 + WORD $0x6d020160 + WORD $0x6d030160 + WORD $0x9102016b + WORD $0x54fffe8d + WORD $0xeb09010b + WORD $0x5400072d + WORD $0xfc5f8140 + WORD $0xf10008bf + WORD $0x5400024b + WORD $0xd10004ab + WORD $0xd100414a WORD $0x1e604002 WORD $0xfc697821 - WORD $0xaa0b03ec - WORD $0xaa0a03ed + WORD $0xaa0a03ec + WORD $0xaa0b03ed WORD $0x1e620822 WORD $0xfc5f8583 WORD $0xd10005ad @@ -757,38 +1510,34 @@ TEXT ·polyEvalFloat64NEON(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x91000529 WORD $0xeb08013f WORD $0x54fffe61 - WORD $0x14000029 - WORD $0x5400050b - WORD $0xf100111f - WORD $0x540000e2 - WORD $0xaa1f03e9 - WORD $0x1400000f - WORD $0xf100111f - WORD $0x54000262 - WORD $0xaa1f03e9 - WORD $0x1400001b - WORD $0x6f00e400 - WORD $0x927ef109 - WORD $0x9100400a - WORD $0x927ef10b - WORD $0xf100116b - WORD $0xad3f8140 - WORD $0x9100814a + WORD $0x14000025 + WORD $0xf100117f + WORD $0x540001c3 + WORD $0x9240050a + WORD $0x4e080401 + WORD $0x8b090c0c + WORD $0xcb0a016b + WORD $0x8b0a012d + WORD $0x8b0b0129 + WORD $0x9100418b + WORD $0xcb0801ac + WORD $0xb100118c + WORD $0xad3f8561 + WORD $0x9100816b WORD $0x54ffffa1 - WORD $0xeb09011f - WORD $0x540002c0 + WORD $0xb40002ca WORD $0x8b090c0a WORD $0xcb090108 WORD $0xf1000508 - WORD $0xf800855f + WORD $0xfc008540 WORD $0x54ffffc1 WORD $0x14000010 - WORD $0x4e080401 + WORD $0x6f00e400 WORD $0x927ef109 WORD $0x9100400a WORD $0x927ef10b WORD $0xf100116b - WORD $0xad3f8541 + WORD $0xad3f8140 WORD $0x9100814a WORD $0x54ffffa1 WORD $0xeb09011f @@ -796,7 +1545,7 @@ TEXT ·polyEvalFloat64NEON(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x8b090c0a WORD $0xcb090108 WORD $0xf1000508 - WORD $0xfc008540 + WORD $0xf800855f WORD $0x54ffffc1 WORD $0xd65f03c0 RET @@ -809,41 +1558,252 @@ TEXT ·convolveFloat64NEON(SB), NOSPLIT|NOFRAME, $0-72 MOVD dst_len+8(FP), R3 MOVD sig_len+32(FP), R4 MOVD ker_len+56(FP), R5 - WORD $0xb40009a5 + WORD $0xb40024c5 WORD $0xeb050088 - WORD $0x5400096b + WORD $0x5400248b + WORD $0xd10143ff WORD $0xeb08007f + WORD $0x6d013bef WORD $0x9a88d468 - WORD $0xf100051f - WORD $0x540008eb - WORD $0xf10004bf - WORD $0x5400064b - WORD $0x8b050c4b + WORD $0x6d0233ed + WORD $0xf100411f + WORD $0x6d032beb + WORD $0x6d0423e9 + WORD $0x5400006a WORD $0xaa1f03e9 - WORD $0x927ef0aa - WORD $0x9100402c + WORD $0x140000cb + WORD $0xf10000bf + WORD $0x5400176d + WORD $0x8b050c49 + WORD $0xaa1f03ef + WORD $0x927ff4aa + WORD $0x9101002b WORD $0xd100204d - WORD $0xd100416b + WORD $0x9101e02e + WORD $0xd100412c + WORD $0x52800210 + WORD $0x14000010 + WORD $0x8b0f0c0f + WORD $0x91004130 + WORD $0x9102016b + WORD $0xeb08021f + WORD $0x910201ce + WORD $0x6d0051f7 + WORD $0x6d0159f2 + WORD $0x6d0245f5 + WORD $0x6d0341f3 + WORD $0x6d041de6 + WORD $0x6d0539e5 + WORD $0x6d0609e3 + WORD $0x6d0701e1 + WORD $0xaa0903ef + WORD $0x5400162c + WORD $0x2f00e400 + WORD $0x2f00e401 + WORD $0xf10004bf + WORD $0x2f00e402 + WORD $0x2f00e403 + WORD $0xaa1003e9 + WORD $0x2f00e40e + WORD $0x2f00e405 + WORD $0x2f00e407 + WORD $0x2f00e406 + WORD $0x2f00e410 + WORD $0x2f00e413 + WORD $0x2f00e411 + WORD $0x2f00e415 + WORD $0x2f00e416 + WORD $0x2f00e412 + WORD $0x2f00e414 + WORD $0x2f00e417 + WORD $0x54000061 + WORD $0xaa1f03f1 + WORD $0x1400005d + WORD $0xaa0b03e4 + WORD $0xaa0c03f0 + WORD $0x927ff4b1 + WORD $0xaa0b03e3 + WORD $0x3cdf0618 + WORD $0x3cdc0099 + WORD $0xad41749a + WORD $0xf1000a31 + WORD $0x6e184318 + WORD $0x3cc2809c + WORD $0x3cdd8089 + WORD $0x3cde808b + WORD $0x3cdf008c + WORD $0xfd0007ee + WORD $0x3cc18084 + WORD $0x6e79df1b + WORD $0x3cc38099 + WORD $0x6e7adf1e + WORD $0x6e7cdf1f + WORD $0x6e7ddf1d + WORD $0x6e69df09 + WORD $0x6e79df08 + WORD $0xad7ea89c + WORD $0x3cdc8099 + WORD $0x6e64df04 + WORD $0x1e7e2863 + WORD $0x1e7b2af7 + WORD $0x6e79df1a + WORD $0x6e7cdf19 + WORD $0x6e6adf0a + WORD $0x6e6bdf1c + WORD $0x6e6cdf0b + WORD $0x5e1807cc + WORD $0x5e1807ed + WORD $0x5e1807ae + WORD $0x5e18050f + WORD $0x1e682800 + WORD $0x1e7d2821 + WORD $0x1e7f2842 + WORD $0x3cdf809d + WORD $0x3dc0009e + WORD $0x3cc0809f + WORD $0x3cc10c68 + WORD $0x1e692ad6 + WORD $0x6e7ddf1d + WORD $0x6e7edf1e + WORD $0x6e7fdf1f + WORD $0x6e68df08 + WORD $0x5e180778 + WORD $0x5e18053b + WORD $0x1e6a2ab5 + WORD $0x5e180549 + WORD $0x1e6b2a73 + WORD $0x5e18056a + WORD $0xfd4007eb + WORD $0x1e7c2a31 + WORD $0x1e7f28e7 + WORD $0x5e1807ff + WORD $0x1e7d2a10 + WORD $0x1e6828a5 + WORD $0x5e180508 + WORD $0x1e64296b + WORD $0x5e180484 + WORD $0x5e1807bd + WORD $0x1e7e28c6 + WORD $0x5e1807de + WORD $0x5e18079c + WORD $0x1e7a2a94 + WORD $0x5e18075a + WORD $0x1e792a52 + WORD $0x5e180739 + WORD $0x1e6f2800 + WORD $0x1e6e2821 + WORD $0x1e6d2842 + WORD $0x1e6c2863 + WORD $0x1e64296e + WORD $0x1e6828a5 + WORD $0x1e7f28e7 + WORD $0x1e7e28c6 + WORD $0x1e7d2a10 + WORD $0x1e6a2a73 + WORD $0x1e7c2a31 + WORD $0xaa0303e4 + WORD $0x1e692ab5 + WORD $0x1e7b2ad6 + WORD $0x1e792a52 + WORD $0x1e7a2a94 + WORD $0x1e782af7 + WORD $0x54fff581 + WORD $0xeb0a00bf + WORD $0x927ff4b1 + WORD $0x54fff020 + WORD $0xd37df230 + WORD $0xcb1100b1 + WORD $0xaa0e03e3 + WORD $0x8b100064 + WORD $0xfc7179b8 + WORD $0xf1000631 + WORD $0x6d78e484 + WORD $0x91002063 + WORD $0x6d79ec9a + WORD $0x6d7ca488 + WORD $0x1e790b19 + WORD $0x1e640b04 + WORD $0x1e7a0b1a + WORD $0x1e7b0b1b + WORD $0x6d7af49c + WORD $0x1e680b08 + WORD $0x6d7bfc9e + WORD $0x1e792a94 + WORD $0x6d7de48a + WORD $0x1e642af7 + WORD $0x1e690b04 + WORD $0x1e7c0b1c + WORD $0x6d7fb48c + WORD $0x1e7d0b1d + WORD $0x1e6a0b09 + WORD $0x6d7eac8a + WORD $0x1e7e0b1e + WORD $0x1e7f0b1f + WORD $0x1e7a2a52 + WORD $0x1e790b19 + WORD $0x1e7b2ad6 + WORD $0x1e6c0b1b + WORD $0x1e6a0b1a + WORD $0x1e6b0b0a + WORD $0x1e7c2ab5 + WORD $0x1e6d0b18 + WORD $0x1e7d2a31 + WORD $0x1e6828c6 + WORD $0x1e7e2a73 + WORD $0x1e7f2a10 + WORD $0x1e6428e7 + WORD $0x1e6928a5 + WORD $0x1e7929ce + WORD $0x1e7b2821 + WORD $0x1e7a2863 + WORD $0x1e6a2842 + WORD $0x1e782800 + WORD $0x54fffa81 + WORD $0x17ffff50 + WORD $0x6f00e400 + WORD $0xaa1f03e9 + WORD $0x9101200a + WORD $0x9100812b + WORD $0xf81b815f + WORD $0x91004129 + WORD $0xad3e0140 + WORD $0xeb08017f + WORD $0xad3f0140 + WORD $0xad000140 + WORD $0x3d800940 + WORD $0xf900195f + WORD $0x9102014a + WORD $0x54fffecd + WORD $0xeb09010b + WORD $0x5400090d + WORD $0xf10004bf + WORD $0x5400064b + WORD $0x8b090c2a + WORD $0x8b050c4d + WORD $0x927ef0ab + WORD $0xd100204e + WORD $0x9100414c + WORD $0xd10041ad WORD $0x14000007 WORD $0xfc297800 WORD $0x91000529 WORD $0x9100218c WORD $0xeb08013f - WORD $0x91002021 - WORD $0x54000700 + WORD $0x9100214a + WORD $0x54000720 WORD $0x2f00e400 WORD $0xf10010bf WORD $0x54000062 - WORD $0xaa1f03ef + WORD $0xaa1f03f0 WORD $0x14000017 - WORD $0xaa0c03ee - WORD $0xaa0b03ef - WORD $0x927ef0b0 + WORD $0xaa0d03ef + WORD $0xaa0c03f0 + WORD $0x927ef0b1 WORD $0xad7f85e2 - WORD $0xd1001210 - WORD $0xad7f91c3 + WORD $0xd1001231 + WORD $0xad7f9203 + WORD $0x91008210 WORD $0xd10081ef - WORD $0x910081ce WORD $0x6e014021 WORD $0x6e024042 WORD $0x6e61dc61 @@ -854,38 +1814,44 @@ TEXT ·convolveFloat64NEON(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x5e180422 WORD $0x1e612800 WORD $0x1e622800 - WORD $0xb5fffe30 - WORD $0xeb0a00bf - WORD $0x927ef0af + WORD $0xb5fffe31 + WORD $0xeb0b00bf + WORD $0x927ef0b0 WORD $0x54fffc00 - WORD $0x8b0f0c2e - WORD $0xcb0f00af - WORD $0xfc4085c1 - WORD $0xfc6f79a2 - WORD $0xf10005ef + WORD $0x8b100d4f + WORD $0xcb1000b0 + WORD $0xfc4085e1 + WORD $0xfc7079c2 + WORD $0xf1000610 WORD $0x1e620821 WORD $0x1e612800 WORD $0x54ffff61 WORD $0x17ffffd7 - WORD $0xf100111f - WORD $0x54000062 - WORD $0xaa1f03e9 - WORD $0x1400000b + WORD $0xf100117f + WORD $0x540001c3 WORD $0x6f00e400 - WORD $0x927ef109 - WORD $0x9100400a - WORD $0x927ef10b - WORD $0xf100116b - WORD $0xad3f8140 - WORD $0x9100814a + WORD $0x9240050a + WORD $0x8b090c0c + WORD $0xcb0a016b + WORD $0x8b0a012d + WORD $0x8b0b0129 + WORD $0x9100418b + WORD $0xcb0801ac + WORD $0xb100118c + WORD $0xad3f8160 + WORD $0x9100816b WORD $0x54ffffa1 - WORD $0xeb09011f - WORD $0x540000c0 + WORD $0xb40000ca WORD $0x8b090c0a WORD $0xcb090108 WORD $0xf1000508 WORD $0xf800855f WORD $0x54ffffc1 + WORD $0x6d4423e9 + WORD $0x6d432beb + WORD $0x6d4233ed + WORD $0x6d413bef + WORD $0x910143ff WORD $0xd65f03c0 RET @@ -897,38 +1863,247 @@ TEXT ·correlateFloat64NEON(SB), NOSPLIT|NOFRAME, $0-72 MOVD dst_len+8(FP), R3 MOVD sig_len+32(FP), R4 MOVD ker_len+56(FP), R5 - WORD $0xb4000945 + WORD $0xb4002425 WORD $0xeb050088 - WORD $0x5400090b + WORD $0x540023eb + WORD $0xd10143ff WORD $0xeb08007f + WORD $0x6d013bef WORD $0x9a88d468 - WORD $0xf100051f - WORD $0x5400088b + WORD $0x6d0233ed + WORD $0xf100411f + WORD $0x6d032beb + WORD $0x6d0423e9 + WORD $0x5400006a + WORD $0xaa1f03e9 + WORD $0x140000c9 + WORD $0xf10000bf + WORD $0x5400172d + WORD $0xaa1f03ed + WORD $0x927ff4aa + WORD $0x9101002b + WORD $0x9101e02c + WORD $0x5280020e + WORD $0x14000010 + WORD $0x8b0d0c0d + WORD $0x9100412e + WORD $0x9102016b + WORD $0xeb0801df + WORD $0x9102018c + WORD $0x6d0051b7 + WORD $0x6d0159b2 + WORD $0x6d0245b5 + WORD $0x6d0341b3 + WORD $0x6d041da6 + WORD $0x6d0539a5 + WORD $0x6d0609a3 + WORD $0x6d0701a1 + WORD $0xaa0903ed + WORD $0x5400164c + WORD $0x2f00e400 + WORD $0x2f00e401 WORD $0xf10004bf - WORD $0x540005eb + WORD $0x2f00e402 + WORD $0x2f00e403 + WORD $0xaa0e03e9 + WORD $0x2f00e40e + WORD $0x2f00e405 + WORD $0x2f00e407 + WORD $0x2f00e406 + WORD $0x2f00e410 + WORD $0x2f00e413 + WORD $0x2f00e411 + WORD $0x2f00e415 + WORD $0x2f00e416 + WORD $0x2f00e412 + WORD $0x2f00e414 + WORD $0x2f00e417 + WORD $0x54000061 + WORD $0xaa1f03ef + WORD $0x1400005c + WORD $0xaa0b03f1 + WORD $0xaa0203ee + WORD $0x927ff4af + WORD $0xaa0b03f0 + WORD $0x3cc105d8 + WORD $0x3cdc0239 + WORD $0xad41763a + WORD $0xf10009ef + WORD $0x3cc2823c + WORD $0x6e79df1b + WORD $0x3cc38239 + WORD $0x3cdd8229 + WORD $0x3cde822b + WORD $0x3cdf022c + WORD $0x6e7adf1e + WORD $0x6e7cdf1f + WORD $0x6e7ddf1d + WORD $0x6e79df08 + WORD $0xad7eaa3c + WORD $0x3cdc8239 + WORD $0xfd0007ee + WORD $0x6e69df09 + WORD $0x3cc18224 + WORD $0x1e7b2af7 + WORD $0x6e79df1a + WORD $0x6e7cdf19 + WORD $0x6e6adf0a + WORD $0x6e6bdf1c + WORD $0x6e6cdf0b + WORD $0x5e1807cc + WORD $0x5e1807ed + WORD $0x5e1807ae + WORD $0x5e18050f + WORD $0x1e682800 + WORD $0x1e7d2821 + WORD $0x1e7f2842 + WORD $0x1e7e2863 + WORD $0x3cdf823d + WORD $0x3dc0023e + WORD $0x3cc0823f + WORD $0x3cc10e08 + WORD $0x6e7ddf1d + WORD $0x6e7edf1e + WORD $0x6e64df04 + WORD $0x6e7fdf1f + WORD $0x6e68df08 + WORD $0x5e180778 + WORD $0x1e692ad6 + WORD $0x5e18053b + WORD $0x1e6a2ab5 + WORD $0x5e180549 + WORD $0x1e6b2a73 + WORD $0x5e18056a + WORD $0xfd4007eb + WORD $0x1e7d2a10 + WORD $0x5e1807bd + WORD $0x1e7f28e7 + WORD $0x5e1807ff + WORD $0x1e6828a5 + WORD $0x5e180508 + WORD $0x1e64296b + WORD $0x5e180484 + WORD $0x1e7e28c6 + WORD $0x5e1807de + WORD $0x1e7c2a31 + WORD $0x5e18079c + WORD $0x1e7a2a94 + WORD $0x5e18075a + WORD $0x1e792a52 + WORD $0x5e180739 + WORD $0xaa1003f1 + WORD $0x1e6f2800 + WORD $0x1e6e2821 + WORD $0x1e6d2842 + WORD $0x1e6c2863 + WORD $0x1e64296e + WORD $0x1e6828a5 + WORD $0x1e7f28e7 + WORD $0x1e7e28c6 + WORD $0x1e7d2a10 + WORD $0x1e6a2a73 + WORD $0x1e7c2a31 + WORD $0x1e692ab5 + WORD $0x1e7b2ad6 + WORD $0x1e792a52 + WORD $0x1e7a2a94 + WORD $0x1e782af7 + WORD $0x54fff5a1 + WORD $0xeb0a00bf + WORD $0x927ff4af + WORD $0x54fff040 + WORD $0xd37df1ee + WORD $0xcb0f00af + WORD $0xaa0c03f0 + WORD $0xaa0203f1 + WORD $0x8b0e0203 + WORD $0xfc6e6a38 + WORD $0xf10005ef + WORD $0x6d78e464 + WORD $0x91002231 + WORD $0x6d79ec7a + WORD $0x91002210 + WORD $0x6d7ca468 + WORD $0x1e790b19 + WORD $0x1e640b04 + WORD $0x1e7a0b1a + WORD $0x1e7b0b1b + WORD $0x6d7af47c + WORD $0x1e680b08 + WORD $0x6d7bfc7e + WORD $0x1e792a94 + WORD $0x6d7de46a + WORD $0x1e642af7 + WORD $0x1e690b04 + WORD $0x1e7c0b1c + WORD $0x6d7fb46c + WORD $0x1e7d0b1d + WORD $0x1e6a0b09 + WORD $0x6d7eac6a + WORD $0x1e7e0b1e + WORD $0x1e7f0b1f + WORD $0x1e7a2a52 + WORD $0x1e790b19 + WORD $0x1e7b2ad6 + WORD $0x1e6c0b1b + WORD $0x1e6a0b1a + WORD $0x1e6b0b0a + WORD $0x1e7c2ab5 + WORD $0x1e6d0b18 + WORD $0x1e7d2a31 + WORD $0x1e6828c6 + WORD $0x1e7e2a73 + WORD $0x1e7f2a10 + WORD $0x1e6428e7 + WORD $0x1e6928a5 + WORD $0x1e7929ce + WORD $0x1e7b2821 + WORD $0x1e7a2863 + WORD $0x1e6a2842 + WORD $0x1e782800 + WORD $0x54fffa61 + WORD $0x17ffff4f + WORD $0x6f00e400 WORD $0xaa1f03e9 - WORD $0x927ef0aa - WORD $0x9100402b + WORD $0x9101200a + WORD $0x9100812b + WORD $0xf81b815f + WORD $0x91004129 + WORD $0xad3e0140 + WORD $0xeb08017f + WORD $0xad3f0140 + WORD $0xad000140 + WORD $0x3d800940 + WORD $0xf900195f + WORD $0x9102014a + WORD $0x54fffecd + WORD $0xeb09010b + WORD $0x540008ad + WORD $0xf10004bf + WORD $0x540005eb + WORD $0x8b090c2a + WORD $0x927ef0ab + WORD $0x9100414c WORD $0x14000007 WORD $0xfc297800 WORD $0x91000529 - WORD $0x9100216b + WORD $0x9100218c WORD $0xeb08013f - WORD $0x91002021 - WORD $0x54000700 + WORD $0x9100214a + WORD $0x54000720 WORD $0x2f00e400 WORD $0xf10010bf WORD $0x54000062 - WORD $0xaa1f03ec + WORD $0xaa1f03ed WORD $0x14000015 - WORD $0xaa0b03ec - WORD $0x9100404d - WORD $0x927ef0ae - WORD $0xad7f91a1 - WORD $0xf10011ce - WORD $0xad7f8d82 + WORD $0xaa0c03ed + WORD $0x9100404e + WORD $0x927ef0af + WORD $0xad7f91c1 + WORD $0xf10011ef + WORD $0xad7f8da2 + WORD $0x910081ce WORD $0x910081ad - WORD $0x9100818c WORD $0x6e61dc41 WORD $0x5e180422 WORD $0x1e612800 @@ -938,39 +2113,45 @@ TEXT ·correlateFloat64NEON(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x1e612800 WORD $0x1e622800 WORD $0x54fffe61 - WORD $0xeb0a00bf - WORD $0x927ef0ac + WORD $0xeb0b00bf + WORD $0x927ef0ad WORD $0x54fffc40 - WORD $0xd37df18e - WORD $0xcb0c00ac - WORD $0x8b0e004d - WORD $0x8b0e002e - WORD $0xfc4085c1 - WORD $0xf100058c - WORD $0xfc4085a2 + WORD $0xd37df1af + WORD $0xcb0d00ad + WORD $0x8b0f004e + WORD $0x8b0f014f + WORD $0xfc4085e1 + WORD $0xf10005ad + WORD $0xfc4085c2 WORD $0x1e620821 WORD $0x1e612800 WORD $0x54ffff61 WORD $0x17ffffd7 - WORD $0xf100111f - WORD $0x54000062 - WORD $0xaa1f03e9 - WORD $0x1400000b + WORD $0xf100117f + WORD $0x540001c3 WORD $0x6f00e400 - WORD $0x927ef109 - WORD $0x9100400a - WORD $0x927ef10b - WORD $0xf100116b - WORD $0xad3f8140 - WORD $0x9100814a + WORD $0x9240050a + WORD $0x8b090c0c + WORD $0xcb0a016b + WORD $0x8b0a012d + WORD $0x8b0b0129 + WORD $0x9100418b + WORD $0xcb0801ac + WORD $0xb100118c + WORD $0xad3f8160 + WORD $0x9100816b WORD $0x54ffffa1 - WORD $0xeb09011f - WORD $0x540000c0 + WORD $0xb40000ca WORD $0x8b090c0a WORD $0xcb090108 WORD $0xf1000508 WORD $0xf800855f WORD $0x54ffffc1 + WORD $0x6d4423e9 + WORD $0x6d432beb + WORD $0x6d4233ed + WORD $0x6d413bef + WORD $0x910143ff WORD $0xd65f03c0 RET @@ -982,21 +2163,58 @@ TEXT ·movingAverageFloat32NEON(SB), NOSPLIT|NOFRAME, $0-56 MOVD a_len+32(FP), R3 MOVD width+48(FP), R4 WORD $0xf100049f - WORD $0x540006cb + WORD $0x54000b6b WORD $0xeb040068 - WORD $0x5400068b + WORD $0x54000b2b WORD $0xeb08005f WORD $0x9a88d448 - WORD $0xf100051f - WORD $0x5400060b - WORD $0x9e230080 + WORD $0xf100411f + WORD $0x5400006a WORD $0xaa1f03e9 - WORD $0x927dec8a - WORD $0x9100402b + WORD $0x14000022 + WORD $0x9e230080 + WORD $0xaa1f03eb + WORD $0x9100802a + WORD $0x5280020c + WORD $0x4e040400 + WORD $0x6f00e401 + WORD $0x6f00e404 + WORD $0xaa0c03e9 + WORD $0x6f00e403 + WORD $0x6f00e402 + WORD $0xaa0403ec + WORD $0xaa0a03ed + WORD $0xad7f19a5 + WORD $0xf100058c + WORD $0xad4041a7 + WORD $0x910011ad + WORD $0x4e21d4a1 + WORD $0x4e24d4c4 + WORD $0x4e23d4e3 + WORD $0x4e22d602 + WORD $0x54ffff01 + WORD $0x6e20fc21 + WORD $0x8b0b080b + WORD $0x9100412c + WORD $0xeb08019f + WORD $0x9101014a + WORD $0x6e20fc84 + WORD $0x6e20fc63 + WORD $0xad001161 + WORD $0x6e20fc42 + WORD $0xad010963 + WORD $0xaa0903eb + WORD $0x54fffcad + WORD $0xeb08013f + WORD $0x5400060a + WORD $0x9e230080 + WORD $0x8b09082a + WORD $0x927dec8b + WORD $0x9100414c WORD $0x14000008 WORD $0x1e201821 - WORD $0x9100116b - WORD $0x91001021 + WORD $0x9100118c + WORD $0x9100114a WORD $0xbc297801 WORD $0x91000529 WORD $0xeb08013f @@ -1004,13 +2222,13 @@ TEXT ·movingAverageFloat32NEON(SB), NOSPLIT|NOFRAME, $0-56 WORD $0x2f00e401 WORD $0xf100209f WORD $0x54000062 - WORD $0xaa1f03ed + WORD $0xaa1f03ee WORD $0x14000018 - WORD $0xaa0b03ec - WORD $0x927dec8d - WORD $0xad7f8d82 - WORD $0xf10021ad - WORD $0x9100818c + WORD $0xaa0c03ed + WORD $0x927dec8e + WORD $0xad7f8da2 + WORD $0xf10021ce + WORD $0x910081ad WORD $0x5e0c0444 WORD $0x1e222821 WORD $0x5e140445 @@ -1026,13 +2244,13 @@ TEXT ·movingAverageFloat32NEON(SB), NOSPLIT|NOFRAME, $0-56 WORD $0x1e242821 WORD $0x1e222821 WORD $0x54fffde1 - WORD $0xeb0a009f - WORD $0x927dec8d + WORD $0xeb0b009f + WORD $0x927dec8e WORD $0x54fffbc0 - WORD $0x8b0d082c - WORD $0xcb0d008d - WORD $0xbc404582 - WORD $0xf10005ad + WORD $0x8b0e094d + WORD $0xcb0e008e + WORD $0xbc4045a2 + WORD $0xf10005ce WORD $0x1e222821 WORD $0x54ffffa1 WORD $0x17ffffd7 @@ -1153,21 +2371,74 @@ TEXT ·movingAverageFloat64NEON(SB), NOSPLIT|NOFRAME, $0-56 MOVD a_len+32(FP), R3 MOVD width+48(FP), R4 WORD $0xf100049f - WORD $0x540005cb + WORD $0x54000c6b WORD $0xeb040068 - WORD $0x5400058b + WORD $0x54000c2b WORD $0xeb08005f WORD $0x9a88d448 - WORD $0xf100051f - WORD $0x5400050b - WORD $0x9e630080 + WORD $0xf100411f + WORD $0x5400006a WORD $0xaa1f03e9 - WORD $0x927ef08a - WORD $0x9100402b + WORD $0x14000032 + WORD $0x9e630080 + WORD $0xaa1f03eb + WORD $0x9101002a + WORD $0x5280020c + WORD $0x4e080400 + WORD $0x6f00e405 + WORD $0x6f00e410 + WORD $0xaa0c03e9 + WORD $0x6f00e407 + WORD $0x6f00e406 + WORD $0xaa0a03ec + WORD $0x6f00e404 + WORD $0x6f00e403 + WORD $0xaa0403ed + WORD $0x6f00e402 + WORD $0x6f00e401 + WORD $0xad7e4991 + WORD $0xf10005ad + WORD $0xad7f5193 + WORD $0xad405995 + WORD $0x4e65d625 + WORD $0x4e70d650 + WORD $0xad414991 + WORD $0x4e67d667 + WORD $0x4e66d686 + WORD $0x4e64d6a4 + WORD $0x4e63d6c3 + WORD $0x9100218c + WORD $0x4e62d622 + WORD $0x4e61d641 + WORD $0x54fffe41 + WORD $0x6e60fca5 + WORD $0x8b0b0c0b + WORD $0x9100412c + WORD $0xeb08019f + WORD $0x9102014a + WORD $0x6e60fe10 + WORD $0x6e60fce7 + WORD $0xad004165 + WORD $0x6e60fcc6 + WORD $0x6e60fc84 + WORD $0xad011967 + WORD $0x6e60fc63 + WORD $0x6e60fc42 + WORD $0xad020d64 + WORD $0x6e60fc21 + WORD $0xad030562 + WORD $0xaa0903eb + WORD $0x54fffaad + WORD $0xeb08013f + WORD $0x5400050a + WORD $0x9e630080 + WORD $0x8b090c2a + WORD $0x927ef08b + WORD $0x9100414c WORD $0x14000008 WORD $0x1e601821 - WORD $0x9100216b - WORD $0x91002021 + WORD $0x9100218c + WORD $0x9100214a WORD $0xfc297801 WORD $0x91000529 WORD $0xeb08013f @@ -1175,13 +2446,13 @@ TEXT ·movingAverageFloat64NEON(SB), NOSPLIT|NOFRAME, $0-56 WORD $0x2f00e401 WORD $0xf100109f WORD $0x54000062 - WORD $0xaa1f03ed + WORD $0xaa1f03ee WORD $0x14000010 - WORD $0xaa0b03ec - WORD $0x927ef08d - WORD $0xad7f8d82 - WORD $0xf10011ad - WORD $0x9100818c + WORD $0xaa0c03ed + WORD $0x927ef08e + WORD $0xad7f8da2 + WORD $0xf10011ce + WORD $0x910081ad WORD $0x5e180444 WORD $0x1e622821 WORD $0x5e180462 @@ -1189,13 +2460,13 @@ TEXT ·movingAverageFloat64NEON(SB), NOSPLIT|NOFRAME, $0-56 WORD $0x1e632821 WORD $0x1e622821 WORD $0x54fffee1 - WORD $0xeb0a009f - WORD $0x927ef08d + WORD $0xeb0b009f + WORD $0x927ef08e WORD $0x54fffcc0 - WORD $0x8b0d0c2c - WORD $0xcb0d008d - WORD $0xfc408582 - WORD $0xf10005ad + WORD $0x8b0e0d4d + WORD $0xcb0e008e + WORD $0xfc4085a2 + WORD $0xf10005ce WORD $0x1e622821 WORD $0x54ffffa1 WORD $0x17ffffdf diff --git a/internal/arm64/numeric_sve2_arm64.s b/internal/arm64/numeric_sve2_arm64.s index 79940bd..ed71ac3 100644 --- a/internal/arm64/numeric_sve2_arm64.s +++ b/internal/arm64/numeric_sve2_arm64.s @@ -233,20 +233,75 @@ TEXT ·polyEvalFloat32SVE2(SB), NOSPLIT|NOFRAME, $0-72 MOVD coeffs_len+56(FP), R5 WORD $0xeb04007f WORD $0x9a84b068 + WORD $0xb40000c5 + WORD $0x8b05084a + WORD $0xf100411f + WORD $0x5400014a + WORD $0xaa1f03e9 + WORD $0x14000035 WORD $0xf100051f - WORD $0xb4000305 - WORD $0x540008ab - WORD $0x8b05084b - WORD $0xf10008bf - WORD $0xbc5fc160 - WORD $0x5400032b + WORD $0x54000e8b + WORD $0x0460e3e9 + WORD $0xeb09011f + WORD $0x54000c02 WORD $0xaa1f03e9 - WORD $0xd10004aa - WORD $0xd100216b + WORD $0x1400006a + WORD $0xd1001149 + WORD $0xf10004ab + WORD $0x4d40c920 + WORD $0x5400044d + WORD $0xaa1f03ec + WORD $0x5280020d + WORD $0x8b0c082e + WORD $0x4ea01c10 + WORD $0x4ea01c07 + WORD $0x4ea01c06 + WORD $0x4ea01c02 + WORD $0xaa0d03e9 + WORD $0xad400dc1 + WORD $0xd100214d + WORD $0xad4115c4 + WORD $0xaa0b03ee + WORD $0xbc5fc5b1 + WORD $0x6e21de10 + WORD $0x6e23dce7 + WORD $0x6e24dcc6 + WORD $0x6e25dc42 + WORD $0x4e040631 + WORD $0xd10005ce + WORD $0xf10001df + WORD $0x4e30d630 + WORD $0x4e27d627 + WORD $0x4e26d626 + WORD $0x4e22d622 + WORD $0x54fffe8c + WORD $0x8b0c080e + WORD $0x9100412d + WORD $0xaa0903ec + WORD $0xeb0801bf + WORD $0xad001dd0 + WORD $0xad0109c6 + WORD $0x54fffc6d + WORD $0x14000009 + WORD $0xaa1f03e9 + WORD $0x9100800b + WORD $0x9100812c + WORD $0x91004129 + WORD $0xad3f0160 + WORD $0xeb08019f + WORD $0xac820160 + WORD $0x54ffff6d + WORD $0xeb09010b + WORD $0x5400080d + WORD $0xbc5fc140 + WORD $0xf10008bf + WORD $0x5400024b + WORD $0xd10004ab + WORD $0xd100214a WORD $0x1e204002 WORD $0xbc697821 - WORD $0xaa0b03ec - WORD $0xaa0a03ed + WORD $0xaa0a03ec + WORD $0xaa0b03ed WORD $0x1e220822 WORD $0xbc5fc583 WORD $0xd10005ad @@ -257,44 +312,41 @@ TEXT ·polyEvalFloat32SVE2(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x91000529 WORD $0xeb08013f WORD $0x54fffe61 - WORD $0x1400002f - WORD $0x540005cb - WORD $0x0460e3e9 - WORD $0xeb09011f - WORD $0x54000102 - WORD $0xaa1f03e9 - WORD $0x14000012 - WORD $0x0460e3e9 - WORD $0xeb09011f - WORD $0x540002a2 - WORD $0xaa1f03e9 - WORD $0x1400001f - WORD $0x6f00e400 - WORD $0xcb0903e9 - WORD $0xaa0003ea - WORD $0x8a090109 - WORD $0xaa0903eb - WORD $0xe5804140 - WORD $0x0470e7eb - WORD $0xe5804540 - WORD $0x0431e3ea - WORD $0xb5ffff8b - WORD $0xeb09011f + WORD $0x1400002c + WORD $0x0460e3ed + WORD $0xeb0d017f + WORD $0x54000243 + WORD $0xd37ef52e + WORD $0xcb0d03ed + WORD $0x05242001 + WORD $0x8a0d016d + WORD $0x2598e3e0 + WORD $0xaa1f03ea + WORD $0x8b0e000c + WORD $0x0430e3ee + WORD $0x8b0d0129 + WORD $0x8b0e000e + WORD $0xe54a4181 + WORD $0xe54a41c1 + WORD $0x0470e3ea + WORD $0xeb0a01bf + WORD $0x54ffff81 + WORD $0xeb0d017f WORD $0x54000300 WORD $0x8b09080a WORD $0xcb090108 WORD $0xf1000508 - WORD $0xb800455f + WORD $0xbc004540 WORD $0x54ffffc1 WORD $0x14000012 + WORD $0x6f00e400 WORD $0xcb0903e9 - WORD $0x05242001 WORD $0xaa0003ea WORD $0x8a090109 WORD $0xaa0903eb - WORD $0xe5804141 + WORD $0xe5804140 WORD $0x0470e7eb - WORD $0xe5804541 + WORD $0xe5804540 WORD $0x0431e3ea WORD $0xb5ffff8b WORD $0xeb09011f @@ -302,7 +354,7 @@ TEXT ·polyEvalFloat32SVE2(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x8b09080a WORD $0xcb090108 WORD $0xf1000508 - WORD $0xbc004540 + WORD $0xb800455f WORD $0x54ffffc1 WORD $0xd65f03c0 RET @@ -315,89 +367,305 @@ TEXT ·convolveFloat32SVE2(SB), NOSPLIT|NOFRAME, $0-72 MOVD dst_len+8(FP), R3 MOVD sig_len+32(FP), R4 MOVD ker_len+56(FP), R5 - WORD $0xb4000a65 + WORD $0xb4002565 WORD $0xeb050088 - WORD $0x54000a2b + WORD $0x5400252b + WORD $0x6db73bef + WORD $0x6d0133ed + WORD $0x6d022beb + WORD $0x6d0323e9 + WORD $0xa9047bfd + WORD $0xa90567fa + WORD $0xa9065ff8 + WORD $0xa90757f6 + WORD $0xa9084ff4 + WORD $0xd10083ff + WORD $0x043f57ff WORD $0xeb08007f + WORD $0xa9008be1 WORD $0x9a88d468 - WORD $0xf100051f - WORD $0x540009ab - WORD $0xf10004bf - WORD $0x0460e3e9 - WORD $0x540006ab - WORD $0xd37ef4ad - WORD $0xcb0903eb + WORD $0xf100411f + WORD $0x5400006a + WORD $0xaa1f03e9 + WORD $0x140000b9 + WORD $0xf10000bf + WORD $0x540015cd + WORD $0xaa0203e9 + WORD $0x04a0e3ec WORD $0x2598e3e0 + WORD $0x0430e7e9 + WORD $0xcb0c03ec + WORD $0xaa1f03f0 + WORD $0xd100104b + WORD $0x9100f022 + WORD $0x5280020a + WORD $0xd280002d + WORD $0xd280004e + WORD $0xd280006f + WORD $0x8b050929 + WORD $0xd2800091 + WORD $0x8a0c00a3 + WORD $0xd28000a6 + WORD $0xd28000c7 + WORD $0xd28000f3 + WORD $0xd2800114 + WORD $0xd2800135 + WORD $0xd2800156 + WORD $0xd2800177 + WORD $0xd2800198 + WORD $0xd28001b9 + WORD $0xd28001da + WORD $0xd28001fd + WORD $0xaa0103fe + WORD $0xf9000fe9 + WORD $0x1400000e + WORD $0x8b10080c + WORD $0x9100412a + WORD $0x910103de + WORD $0xeb08015f + WORD $0x91010042 + WORD $0xaa0903f0 + WORD $0xbd000186 + WORD $0x3c804183 + WORD $0x3c814185 + WORD $0x3c824184 + WORD $0x2d068582 + WORD $0xbd003d80 + WORD $0x540011ac + WORD $0x04a0e3e9 + WORD $0xeb0900bf + WORD $0xaa0a03e9 + WORD $0x54000142 + WORD $0x6f00e403 + WORD $0x2f00e400 + WORD $0xaa1f03ec + WORD $0x2f00e401 + WORD $0x2f00e402 + WORD $0x2f00e406 + WORD $0x6f00e405 + WORD $0x6f00e404 + WORD $0x1400005a + WORD $0x2f00e400 + WORD $0x2f00e401 + WORD $0xf9400fe4 + WORD $0x2f00e402 + WORD $0x2f00e41d WORD $0xaa1f03ea + WORD $0x2f00e410 + WORD $0x2f00e412 + WORD $0x2f00e404 + WORD $0x2f00e411 + WORD $0x2f00e414 + WORD $0x2f00e416 + WORD $0x2f00e405 + WORD $0x2f00e413 + WORD $0x2f00e415 + WORD $0x2f00e417 + WORD $0x2f00e403 + WORD $0x2f00e406 + WORD $0x910083ec + WORD $0x85804098 + WORD $0x0430e7e4 + WORD $0xe580419d + WORD $0x8b0a0bcc + WORD $0x05b83b1d + WORD $0xa55d4189 + WORD $0xa54a43d8 + WORD $0xa54d4199 + WORD $0xa54e419a + WORD $0xa54f419b + WORD $0xa551419c + WORD $0xa546419e + WORD $0xa547419f + WORD $0x65890ba9 + WORD $0xa5534188 + WORD $0xa55a418a + WORD $0xa554418b + WORD $0xa559418c + WORD $0xa555418d + WORD $0xa558418e + WORD $0xa556418f + WORD $0xa5574187 + WORD $0x65980bb8 + WORD $0x65990bb9 + WORD $0x910083ec + WORD $0x659a0bba + WORD $0x659b0bbb + WORD $0x04b0e3ea + WORD $0x659c0bbc + WORD $0x659e0bbe + WORD $0x659f0bbf + WORD $0x65880ba8 + WORD $0x658a0baa + WORD $0x658b0bab + WORD $0xeb0a007f + WORD $0x658c0bac + WORD $0x658d0bad + WORD $0x658e0bae + WORD $0x65870ba7 + WORD $0x65982120 + WORD $0x658f0ba9 + WORD $0x8580419d + WORD $0x65982141 + WORD $0x65982182 + WORD $0x659821dd + WORD $0x659820f0 + WORD $0x65982132 + WORD $0x659821a4 + WORD $0x65982171 + WORD $0x65982114 + WORD $0x659823f6 + WORD $0x659823c5 + WORD $0x65982393 + WORD $0x65982375 + WORD $0x65982357 + WORD $0x65982323 + WORD $0x65982306 + WORD $0x54fff8c1 + WORD $0x6e0c06e3 + WORD $0x6e0c06c5 + WORD $0xeb0300bf + WORD $0x6e0c0644 + WORD $0xaa0303ec + WORD $0x6e1406a3 + WORD $0x6e140685 + WORD $0x6e140604 + WORD $0x6e1c0663 + WORD $0x6e1c0625 + WORD $0x6e1c07a4 + WORD $0x54fff1c0 + WORD $0xd37ef58a + WORD $0xcb0c00a4 + WORD $0xaa0203ec + WORD $0x8b0a0181 + WORD $0xbc647967 + WORD $0xf1000484 + WORD $0x2d7f5434 + WORD $0xbc5c4030 + WORD $0x3cdc8031 + WORD $0x3cdd8032 + WORD $0x9100118c + WORD $0x3cde8033 + WORD $0xbd400036 + WORD $0x1e3008f0 + WORD $0x4f879231 + WORD $0x4f879252 + WORD $0x1e3408f4 + WORD $0x4f879273 + WORD $0x1e3508f5 + WORD $0x1e3608e7 + WORD $0x1e3028c6 + WORD $0x4e31d463 + WORD $0x4e32d4a5 + WORD $0x1e342842 + WORD $0x4e33d484 + WORD $0x1e352821 + WORD $0x1e272800 + WORD $0x54fffd01 + WORD $0x17ffff71 + WORD $0x6f00e400 + WORD $0xaa1f03e9 + WORD $0x9100800a + WORD $0x9100812b + WORD $0x91004129 + WORD $0xad3f0140 + WORD $0xeb08017f + WORD $0xac820140 + WORD $0x54ffff6d + WORD $0xa940bfec + WORD $0xeb09010a + WORD $0x54000a2d + WORD $0xf10000bf + WORD $0x540006cd + WORD $0xd37ef4ae + WORD $0x0460e3ea + WORD $0x8b09098c + WORD $0xcb0a03eb + WORD $0x2598e3e0 + WORD $0xaa0e03ed + WORD $0x0431e7ee WORD $0x8a0b00ab - WORD $0xd100104e - WORD $0xaa0d03ec - WORD $0x0431e7ed - WORD $0x0430e7ec - WORD $0x8b0d004d - WORD $0x8b0c004c + WORD $0x0430e7ed + WORD $0x8b0e01ee + WORD $0x8b0d01ed + WORD $0xd10011ef WORD $0x14000006 - WORD $0xbc2a7800 - WORD $0x9100054a - WORD $0x91001021 - WORD $0xeb08015f - WORD $0x54000720 - WORD $0xeb0900bf + WORD $0xbc297800 + WORD $0x91000529 + WORD $0x9100118c + WORD $0xeb08013f + WORD $0x540007a0 + WORD $0xeb0a00bf WORD $0x54000082 WORD $0x2f00e400 - WORD $0xaa1f03f0 + WORD $0xaa1f03f1 WORD $0x14000015 WORD $0x2f00e400 - WORD $0xaa1f03ef - WORD $0xaa0103f0 - WORD $0xa54f4181 - WORD $0x85804202 + WORD $0xaa1f03f0 + WORD $0xaa0c03f1 + WORD $0xa55041a1 + WORD $0x85804222 WORD $0x05b83821 WORD $0x65810841 - WORD $0x85804602 - WORD $0x0431e3f0 + WORD $0x85804622 + WORD $0x0431e3f1 WORD $0x65982020 - WORD $0xa54f41a1 - WORD $0x0470e7ef + WORD $0xa55041c1 + WORD $0x0470e7f0 WORD $0x05b83821 - WORD $0xab0f017f + WORD $0xab10017f WORD $0x65810841 WORD $0x65982020 WORD $0x54fffe61 WORD $0xeb0b00bf - WORD $0xaa0b03f0 + WORD $0xaa0b03f1 WORD $0x54fffc60 - WORD $0xd37ef60f - WORD $0xcb1000b0 - WORD $0xbc6f6821 - WORD $0xbc7079c2 - WORD $0xf1000610 - WORD $0x910011ef + WORD $0xd37ef630 + WORD $0xcb1100b1 + WORD $0xbc706981 + WORD $0xbc7179e2 + WORD $0xf1000631 + WORD $0x91001210 WORD $0x1e220821 WORD $0x1e212800 WORD $0x54ffff41 WORD $0x17ffffd9 - WORD $0xeb09011f - WORD $0x54000062 - WORD $0xaa1f03e9 - WORD $0x1400000d + WORD $0x0460e3ed + WORD $0xeb0d015f + WORD $0x54000243 + WORD $0xd37ef52e WORD $0x6f00e400 - WORD $0xcb0903e9 - WORD $0xaa0003ea - WORD $0x8a090109 - WORD $0xaa0903eb - WORD $0xe5804140 - WORD $0x0470e7eb - WORD $0xe5804540 - WORD $0x0431e3ea - WORD $0xb5ffff8b - WORD $0xeb09011f + WORD $0xcb0d03ed + WORD $0x8a0d014d + WORD $0x2598e3e0 + WORD $0xaa1f03eb + WORD $0x8b0e000c + WORD $0x0430e3ee + WORD $0x8b0d0129 + WORD $0x8b0e000e + WORD $0xe54b4180 + WORD $0xe54b41c0 + WORD $0x0470e3eb + WORD $0xeb0b01bf + WORD $0x54ffff81 + WORD $0xeb0d015f WORD $0x540000c0 WORD $0x8b09080a WORD $0xcb090108 WORD $0xf1000508 WORD $0xb800455f WORD $0x54ffffc1 + WORD $0x043f503f + WORD $0x910083ff + WORD $0xa9484ff4 + WORD $0xa94757f6 + WORD $0xa9465ff8 + WORD $0xa94567fa + WORD $0xa9447bfd + WORD $0x6d4323e9 + WORD $0x6d422beb + WORD $0x6d4133ed + WORD $0x6cc93bef WORD $0xd65f03c0 RET @@ -409,78 +677,287 @@ TEXT ·correlateFloat32SVE2(SB), NOSPLIT|NOFRAME, $0-72 MOVD dst_len+8(FP), R3 MOVD sig_len+32(FP), R4 MOVD ker_len+56(FP), R5 - WORD $0xb4000905 + WORD $0xb4002325 WORD $0xeb050088 - WORD $0x540008cb + WORD $0x540022eb + WORD $0x6db73bef + WORD $0x6d0133ed + WORD $0x6d022beb + WORD $0x6d0323e9 + WORD $0xa9047bfd + WORD $0xa90567fa + WORD $0xa9065ff8 + WORD $0xa90757f6 + WORD $0xa9084ff4 + WORD $0xd10043ff + WORD $0x043f57ff WORD $0xeb08007f + WORD $0xaa0203ea + WORD $0xf90007e1 WORD $0x9a88d468 - WORD $0xf100051f - WORD $0x5400084b - WORD $0xf10004bf - WORD $0x0460e3e9 - WORD $0x5400054b - WORD $0xcb0903eb + WORD $0xf100411f + WORD $0x5400006a + WORD $0xaa1f03e9 + WORD $0x140000b0 + WORD $0xf10000bf + WORD $0x540014ad + WORD $0x04a0e3e9 WORD $0x2598e3e0 - WORD $0xaa1f03ea - WORD $0x8a0b00ab + WORD $0xaa1f03e4 + WORD $0xcb0903e9 + WORD $0x52800202 + WORD $0xd280002b + WORD $0xd280004c + WORD $0xd280006d + WORD $0x8a0900ae + WORD $0xd280008f + WORD $0xd28000b0 + WORD $0xd28000d1 + WORD $0xd28000e3 + WORD $0xd2800106 + WORD $0xd2800127 + WORD $0xd2800153 + WORD $0xd2800174 + WORD $0xd2800195 + WORD $0xd28001b6 + WORD $0xd28001d7 + WORD $0xd28001f8 + WORD $0xaa0103f9 + WORD $0x1400000d + WORD $0x8b040801 + WORD $0x91004122 + WORD $0x91010339 + WORD $0xeb08005f + WORD $0xaa0903e4 + WORD $0xbd000026 + WORD $0x3c804023 + WORD $0x3c814025 + WORD $0x3c824024 + WORD $0x2d068422 + WORD $0xbd003c20 + WORD $0x5400116c + WORD $0x04a0e3e9 + WORD $0xeb0900bf + WORD $0xaa0203e9 + WORD $0x54000142 + WORD $0x6f00e403 + WORD $0x2f00e400 + WORD $0xaa1f03e2 + WORD $0x2f00e401 + WORD $0x2f00e402 + WORD $0x2f00e406 + WORD $0x6f00e405 + WORD $0x6f00e404 + WORD $0x14000056 + WORD $0x2f00e400 + WORD $0x2f00e401 + WORD $0xaa1f03fa + WORD $0x2f00e402 + WORD $0x2f00e41d + WORD $0x2f00e410 + WORD $0x2f00e412 + WORD $0x2f00e404 + WORD $0x2f00e411 + WORD $0x2f00e414 + WORD $0x2f00e416 + WORD $0x2f00e405 + WORD $0x2f00e413 + WORD $0x2f00e415 + WORD $0x2f00e417 + WORD $0x2f00e403 + WORD $0x2f00e406 + WORD $0x8b1a0b22 + WORD $0x910043fd + WORD $0xa55a4338 + WORD $0xe58043bd + WORD $0xa55a415d + WORD $0x04b0e3fa + WORD $0xa5584049 + WORD $0xa54b4059 + WORD $0xa54c405a + WORD $0xa54d405b + WORD $0xa54f405c + WORD $0xa550405e + WORD $0xa551405f + WORD $0x65980bb8 + WORD $0xeb1a01df + WORD $0x65890ba9 + WORD $0xa5434048 + WORD $0xa557404a + WORD $0xa546404b + WORD $0xa556404c + WORD $0x65990bb9 + WORD $0xa547404d + WORD $0xa555404e + WORD $0x659a0bba + WORD $0xa553404f + WORD $0xa5544047 + WORD $0x659b0bbb + WORD $0x659c0bbc + WORD $0x659e0bbe + WORD $0x659f0bbf + WORD $0x65880ba8 + WORD $0x658a0baa + WORD $0x658b0bab + WORD $0x658c0bac + WORD $0x658d0bad + WORD $0x658e0bae + WORD $0x65870ba7 + WORD $0x65982120 + WORD $0x658f0ba9 + WORD $0x858043bd + WORD $0x65982141 + WORD $0x65982182 + WORD $0x659821dd + WORD $0x659820f0 + WORD $0x65982132 + WORD $0x659821a4 + WORD $0x65982171 + WORD $0x65982114 + WORD $0x659823f6 + WORD $0x659823c5 + WORD $0x65982393 + WORD $0x65982375 + WORD $0x65982357 + WORD $0x65982323 + WORD $0x65982306 + WORD $0x54fff921 + WORD $0x6e0c06e3 + WORD $0x6e0c06c5 + WORD $0xeb0e00bf + WORD $0x6e0c0644 + WORD $0xaa0e03e2 + WORD $0x6e1406a3 + WORD $0x6e140685 + WORD $0x6e140604 + WORD $0x6e1c0663 + WORD $0x6e1c0625 + WORD $0x6e1c07a4 + WORD $0x54fff260 + WORD $0xd37ef45a + WORD $0xcb0200bd + WORD $0xaa1903fe + WORD $0xaa0a03e2 + WORD $0x8b1a03c1 + WORD $0xbc7a6847 + WORD $0xf10007bd + WORD $0x2d46d434 + WORD $0xbd400030 + WORD $0x3cc04031 + WORD $0x3cc14032 + WORD $0x91001042 + WORD $0x3cc24033 + WORD $0xbd403c36 + WORD $0x1e3008f0 + WORD $0x4f879231 + WORD $0x4f879252 + WORD $0x1e3408f4 + WORD $0x4f879273 + WORD $0x1e3508f5 + WORD $0x1e3608e7 + WORD $0x910013de + WORD $0x1e3028c6 + WORD $0x4e31d463 + WORD $0x4e32d4a5 + WORD $0x1e342842 + WORD $0x4e33d484 + WORD $0x1e352821 + WORD $0x1e272800 + WORD $0x54fffce1 + WORD $0x17ffff74 + WORD $0x6f00e400 + WORD $0xaa1f03e9 + WORD $0x9100800b + WORD $0x9100812c + WORD $0x91004129 + WORD $0xad3f0160 + WORD $0xeb08019f + WORD $0xac820160 + WORD $0x54ffff6d + WORD $0xf94007ee + WORD $0xeb09010b + WORD $0x540008ed + WORD $0x0460e3ed + WORD $0xf10000bf + WORD $0x5400058d + WORD $0xcb0d03ec + WORD $0x8b0909cb + WORD $0x2598e3e0 + WORD $0x8a0c00ac WORD $0x14000006 - WORD $0xbc2a7800 - WORD $0x9100054a - WORD $0x91001021 - WORD $0xeb08015f - WORD $0x540006a0 + WORD $0xbc297800 + WORD $0x91000529 + WORD $0x9100116b + WORD $0xeb08013f + WORD $0x54000740 + WORD $0xeb0d00bf + WORD $0x54000082 WORD $0x2f00e400 - WORD $0xeb0900bf - WORD $0x54000062 - WORD $0xaa1f03ec - WORD $0x14000013 - WORD $0xaa0103ec - WORD $0xaa0203ed - WORD $0xaa0b03ee - WORD $0x85804181 - WORD $0x858041a2 - WORD $0x0470e7ee + WORD $0xaa1f03ee + WORD $0x14000015 + WORD $0xaa0b03ef + WORD $0x2f00e400 + WORD $0xaa1f03ee + WORD $0x0430e3ef + WORD $0xaa0a03f0 + WORD $0xa54e4161 + WORD $0x85804202 WORD $0x65820821 - WORD $0x858045a2 - WORD $0x0431e3ed + WORD $0x85804602 + WORD $0x0431e3f0 WORD $0x65982020 - WORD $0x85804581 - WORD $0x0431e3ec + WORD $0xa54e41e1 + WORD $0x0470e3ee WORD $0x65820821 + WORD $0xeb0e019f WORD $0x65982020 - WORD $0xb5fffeae - WORD $0xeb0b00bf - WORD $0xaa0b03ec - WORD $0x54fffca0 - WORD $0xbc6c7821 - WORD $0xbc6c7842 - WORD $0x9100058c + WORD $0x54fffea1 WORD $0xeb0c00bf + WORD $0xaa0c03ee + WORD $0x54fffc60 + WORD $0xbc6e7961 + WORD $0xbc6e7942 + WORD $0x910005ce + WORD $0xeb0e00bf WORD $0x1e220821 WORD $0x1e212800 WORD $0x54ffff41 - WORD $0x17ffffdd - WORD $0xeb09011f - WORD $0x54000062 - WORD $0xaa1f03e9 - WORD $0x1400000d + WORD $0x17ffffdb + WORD $0xeb0d017f + WORD $0x54000243 + WORD $0xd37ef52e WORD $0x6f00e400 - WORD $0xcb0903e9 - WORD $0xaa0003ea - WORD $0x8a090109 - WORD $0xaa0903eb - WORD $0xe5804140 - WORD $0x0470e7eb - WORD $0xe5804540 - WORD $0x0431e3ea - WORD $0xb5ffff8b - WORD $0xeb09011f + WORD $0xcb0d03ed + WORD $0x8a0d016d + WORD $0x2598e3e0 + WORD $0xaa1f03ea + WORD $0x8b0e000c + WORD $0x0430e3ee + WORD $0x8b0d0129 + WORD $0x8b0e000e + WORD $0xe54a4180 + WORD $0xe54a41c0 + WORD $0x0470e3ea + WORD $0xeb0a01bf + WORD $0x54ffff81 + WORD $0xeb0d017f WORD $0x540000c0 WORD $0x8b09080a WORD $0xcb090108 WORD $0xf1000508 WORD $0xb800455f WORD $0x54ffffc1 + WORD $0x043f503f + WORD $0x910043ff + WORD $0xa9484ff4 + WORD $0xa94757f6 + WORD $0xa9465ff8 + WORD $0xa94567fa + WORD $0xa9447bfd + WORD $0x6d4323e9 + WORD $0x6d422beb + WORD $0x6d4133ed + WORD $0x6cc93bef WORD $0xd65f03c0 RET @@ -725,20 +1202,98 @@ TEXT ·polyEvalFloat64SVE2(SB), NOSPLIT|NOFRAME, $0-72 MOVD coeffs_len+56(FP), R5 WORD $0xeb04007f WORD $0x9a84b068 + WORD $0xb40000c5 + WORD $0x8b050c4a + WORD $0xf100411f + WORD $0x5400014a + WORD $0xaa1f03e9 + WORD $0x1400004c WORD $0xf100051f - WORD $0xb4000305 - WORD $0x540008ab - WORD $0x8b050c4b - WORD $0xf10008bf - WORD $0xfc5f8160 - WORD $0x5400032b + WORD $0x5400116b + WORD $0x04a0e3e9 + WORD $0xeb09011f + WORD $0x54000ee2 WORD $0xaa1f03e9 - WORD $0xd10004aa - WORD $0xd100416b + WORD $0x14000081 + WORD $0xfc5f8140 + WORD $0xf10004ab + WORD $0x5400066d + WORD $0x4e080400 + WORD $0xaa1f03ec + WORD $0x5280020d + WORD $0x8b0c0c2e + WORD $0x4ea01c18 + WORD $0x4ea01c17 + WORD $0x4ea01c16 + WORD $0x4ea01c15 + WORD $0xaa0d03e9 + WORD $0xad4009c1 + WORD $0x4ea01c14 + WORD $0xad4111c3 + WORD $0x4ea01c13 + WORD $0xad4219c5 + WORD $0x4ea01c12 + WORD $0xad4341c7 + WORD $0x4ea01c11 + WORD $0xd100414d + WORD $0xaa0b03ee + WORD $0xfc5f85b9 + WORD $0x6e61df18 + WORD $0x6e62def7 + WORD $0x6e63ded6 + WORD $0x6e64deb5 + WORD $0x6e65de94 + WORD $0x6e66de73 + WORD $0xd10005ce + WORD $0x6e67de52 + WORD $0x6e70de31 + WORD $0xf10001df + WORD $0x4e080739 + WORD $0x4e78d738 + WORD $0x4e77d737 + WORD $0x4e76d736 + WORD $0x4e75d735 + WORD $0x4e74d734 + WORD $0x4e73d733 + WORD $0x4e72d732 + WORD $0x4e71d731 + WORD $0x54fffd8c + WORD $0x8b0c0c0e + WORD $0x9100412d + WORD $0xaa0903ec + WORD $0xeb0801bf + WORD $0xad005dd8 + WORD $0xad0155d6 + WORD $0xad024dd4 + WORD $0xad0345d2 + WORD $0x54fffa6d + WORD $0x14000010 + WORD $0xaa1f03e9 + WORD $0x9101000b + WORD $0x9100812c + WORD $0x6d3c0160 + WORD $0x91004129 + WORD $0x6d3d0160 + WORD $0xeb08019f + WORD $0x6d3e0160 + WORD $0x6d3f0160 + WORD $0x6d000160 + WORD $0x6d010160 + WORD $0x6d020160 + WORD $0x6d030160 + WORD $0x9102016b + WORD $0x54fffe8d + WORD $0xeb09010b + WORD $0x5400080d + WORD $0xfc5f8140 + WORD $0xf10008bf + WORD $0x5400024b + WORD $0xd10004ab + WORD $0xd100414a WORD $0x1e604002 WORD $0xfc697821 - WORD $0xaa0b03ec - WORD $0xaa0a03ed + WORD $0xaa0a03ec + WORD $0xaa0b03ed WORD $0x1e620822 WORD $0xfc5f8583 WORD $0xd10005ad @@ -749,44 +1304,41 @@ TEXT ·polyEvalFloat64SVE2(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x91000529 WORD $0xeb08013f WORD $0x54fffe61 - WORD $0x1400002f - WORD $0x540005cb - WORD $0x04a0e3e9 - WORD $0xeb09011f - WORD $0x54000102 - WORD $0xaa1f03e9 - WORD $0x14000012 - WORD $0x04a0e3e9 - WORD $0xeb09011f - WORD $0x540002a2 - WORD $0xaa1f03e9 - WORD $0x1400001f - WORD $0x6f00e400 - WORD $0xcb0903e9 - WORD $0xaa0003ea - WORD $0x8a090109 - WORD $0xaa0903eb - WORD $0xe5804140 - WORD $0x04b0e7eb - WORD $0xe5804540 - WORD $0x0431e3ea - WORD $0xb5ffff8b - WORD $0xeb09011f + WORD $0x1400002c + WORD $0x04a0e3ed + WORD $0xeb0d017f + WORD $0x54000243 + WORD $0xd37df12e + WORD $0xcb0d03ed + WORD $0x05282001 + WORD $0x8a0d016d + WORD $0x25d8e3e0 + WORD $0xaa1f03ea + WORD $0x8b0e000c + WORD $0x0430e3ee + WORD $0x8b0d0129 + WORD $0x8b0e000e + WORD $0xe5ea4181 + WORD $0xe5ea41c1 + WORD $0x04b0e3ea + WORD $0xeb0a01bf + WORD $0x54ffff81 + WORD $0xeb0d017f WORD $0x54000300 WORD $0x8b090c0a WORD $0xcb090108 WORD $0xf1000508 - WORD $0xf800855f + WORD $0xfc008540 WORD $0x54ffffc1 WORD $0x14000012 + WORD $0x6f00e400 WORD $0xcb0903e9 - WORD $0x05282001 WORD $0xaa0003ea WORD $0x8a090109 WORD $0xaa0903eb - WORD $0xe5804141 + WORD $0xe5804140 WORD $0x04b0e7eb - WORD $0xe5804541 + WORD $0xe5804540 WORD $0x0431e3ea WORD $0xb5ffff8b WORD $0xeb09011f @@ -794,7 +1346,7 @@ TEXT ·polyEvalFloat64SVE2(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x8b090c0a WORD $0xcb090108 WORD $0xf1000508 - WORD $0xfc008540 + WORD $0xf800855f WORD $0x54ffffc1 WORD $0xd65f03c0 RET @@ -807,89 +1359,326 @@ TEXT ·convolveFloat64SVE2(SB), NOSPLIT|NOFRAME, $0-72 MOVD dst_len+8(FP), R3 MOVD sig_len+32(FP), R4 MOVD ker_len+56(FP), R5 - WORD $0xb4000a65 + WORD $0xb4002805 WORD $0xeb050088 - WORD $0x54000a2b + WORD $0x540027cb + WORD $0xd10283ff WORD $0xeb08007f + WORD $0xfd000bee WORD $0x9a88d468 - WORD $0xf100051f - WORD $0x540009ab - WORD $0xf10004bf - WORD $0x04a0e3e9 - WORD $0x540006ab - WORD $0xd37df0ad - WORD $0xcb0903eb + WORD $0x6d0233ed + WORD $0xf100411f + WORD $0x6d032beb + WORD $0x6d0423e9 + WORD $0xa9057bfd + WORD $0xa90667fa + WORD $0xa9075ff8 + WORD $0xa90857f6 + WORD $0xa9094ff4 + WORD $0xa9000be1 + WORD $0x5400006a + WORD $0xaa1f03e9 + WORD $0x140000d0 + WORD $0xf10000bf + WORD $0x5400180d + WORD $0xaa0203e9 + WORD $0x04e0e3ec WORD $0x25d8e3e0 + WORD $0x0430e7e9 + WORD $0xcb0c03ec + WORD $0xaa1f03f5 + WORD $0xd100204b + WORD $0x9101e022 + WORD $0x5280020a + WORD $0xd280002d + WORD $0xd280004e + WORD $0xd280006f + WORD $0x8b050d29 + WORD $0xd2800090 + WORD $0x8a0c00b1 + WORD $0xd28000a4 + WORD $0xd28000c6 + WORD $0xd28000e7 + WORD $0xd2800113 + WORD $0xd2800134 + WORD $0xd2800156 + WORD $0xd2800177 + WORD $0xd2800198 + WORD $0xd28001b9 + WORD $0xd28001da + WORD $0xd28001fd + WORD $0xaa0103fe + WORD $0xf9000fe9 + WORD $0x14000010 + WORD $0x8b150c0c + WORD $0x9100412a + WORD $0x910203de + WORD $0xeb08015f + WORD $0x91020042 + WORD $0xaa0903f5 + WORD $0x6d005997 + WORD $0x6d015195 + WORD $0x6d024993 + WORD $0x6d034191 + WORD $0x6d041987 + WORD $0x6d051185 + WORD $0x6d060983 + WORD $0x6d070181 + WORD $0x5400144c + WORD $0x04e0e3e9 + WORD $0xeb0900bf + WORD $0xaa0a03e9 + WORD $0x54000262 + WORD $0x2f00e400 + WORD $0x2f00e401 + WORD $0xaa1f03ec + WORD $0x2f00e402 + WORD $0x2f00e403 + WORD $0x2f00e404 + WORD $0x2f00e405 + WORD $0x2f00e406 + WORD $0x2f00e407 + WORD $0x2f00e410 + WORD $0x2f00e411 + WORD $0x2f00e412 + WORD $0x2f00e413 + WORD $0x2f00e414 + WORD $0x2f00e415 + WORD $0x2f00e416 + WORD $0x2f00e417 + WORD $0x1400004d + WORD $0x2f00e400 + WORD $0x2f00e401 + WORD $0xf9400fe3 + WORD $0x2f00e402 + WORD $0x2f00e403 WORD $0xaa1f03ea + WORD $0x2f00e404 + WORD $0x2f00e405 + WORD $0x2f00e406 + WORD $0x2f00e407 + WORD $0x2f00e410 + WORD $0x2f00e411 + WORD $0x2f00e412 + WORD $0x2f00e413 + WORD $0x2f00e414 + WORD $0x2f00e415 + WORD $0x2f00e416 + WORD $0x2f00e417 + WORD $0x85804078 + WORD $0x8b0a0fcc + WORD $0x0430e7e3 + WORD $0xa5fa419a + WORD $0xa5fd4199 + WORD $0xa5ed419b + WORD $0x05f83b18 + WORD $0xa5f8419f + WORD $0xa5f94188 + WORD $0xa5ee419c + WORD $0xa5ef419d + WORD $0xa5f0419e + WORD $0xa5e44189 + WORD $0xa5e6418a + WORD $0xa5f7418b + WORD $0x65da0b1a + WORD $0x65d90b19 + WORD $0xa5f6418c + WORD $0x65c80b08 + WORD $0xa5f4418d + WORD $0xa5e7418e + WORD $0x65dc0b1c + WORD $0x65cb0b0b + WORD $0x65dd0b1d + WORD $0x65cc0b0c + WORD $0x65d82341 + WORD $0x65db0b1a + WORD $0x65df0b1b + WORD $0x65d82320 + WORD $0xa5ea43d9 + WORD $0xa5f3419f + WORD $0x65d82102 + WORD $0x65de0b1e + WORD $0x65cd0b08 + WORD $0x65d82164 + WORD $0x04f0e3ea + WORD $0x65d90b19 + WORD $0x65df0b1f + WORD $0x65d82363 + WORD $0x65c90b1b + WORD $0x65ca0b09 + WORD $0x65ce0b18 + WORD $0xeb0a023f + WORD $0x65d82185 + WORD $0x65d82106 + WORD $0x65d823e7 + WORD $0x65d82310 + WORD $0x65d82131 + WORD $0x65d82372 + WORD $0x65d823d3 + WORD $0x65d823b4 + WORD $0x65d82395 + WORD $0x65d82356 + WORD $0x65d82337 + WORD $0x54fff941 + WORD $0xeb1100bf + WORD $0xaa1103ec + WORD $0x54fff200 + WORD $0xd37df18a + WORD $0xcb0c00a3 + WORD $0xaa0203ec + WORD $0x8b0a0181 + WORD $0xfc637978 + WORD $0xf1000463 + WORD $0x6d78e839 + WORD $0x9100218c + WORD $0x6d79f03b + WORD $0x6d7ca829 + WORD $0x1e7a0b1a + WORD $0x1e790b19 + WORD $0x1e7b0b1b + WORD $0x1e7c0b1c + WORD $0x6d7af83d + WORD $0x1e690b09 + WORD $0x6d7ba03f + WORD $0x1e7a2ad6 + WORD $0x6d7de82b + WORD $0x1e792af7 + WORD $0x1e6a0b19 + WORD $0x1e7d0b1d + WORD $0x6d7fb82d + WORD $0x1e7e0b1e + WORD $0x1e6b0b0a + WORD $0x6d7eb02b + WORD $0x1e7f0b1f + WORD $0x1e680b08 + WORD $0x1e7b2ab5 + WORD $0x1e7a0b1a + WORD $0x1e7c2a94 + WORD $0x1e6d0b1c + WORD $0x1e6b0b1b + WORD $0x1e6c0b0b + WORD $0x1e7d2a73 + WORD $0x1e6e0b18 + WORD $0x1e7e2a52 + WORD $0x1e6928e7 + WORD $0x1e7f2a31 + WORD $0x1e682a10 + WORD $0x1e7928c6 + WORD $0x1e6a28a5 + WORD $0x1e7a2884 + WORD $0x1e7c2821 + WORD $0x1e7b2863 + WORD $0x1e6b2842 + WORD $0x1e782800 + WORD $0x54fffa81 + WORD $0x17ffff5f + WORD $0x6f00e400 + WORD $0xaa1f03e9 + WORD $0x9101200a + WORD $0x9100812b + WORD $0xf81b815f + WORD $0x91004129 + WORD $0xad3e0140 + WORD $0xeb08017f + WORD $0xad3f0140 + WORD $0xad000140 + WORD $0x3d800940 + WORD $0xf900195f + WORD $0x9102014a + WORD $0x54fffecd + WORD $0xa9403fec + WORD $0xeb09010a + WORD $0x54000a2d + WORD $0xf10000bf + WORD $0x540006cd + WORD $0xd37df0ae + WORD $0x04a0e3ea + WORD $0x8b090d8c + WORD $0xcb0a03eb + WORD $0x25d8e3e0 + WORD $0xaa0e03ed + WORD $0x0431e7ee WORD $0x8a0b00ab - WORD $0xd100204e - WORD $0xaa0d03ec - WORD $0x0431e7ed - WORD $0x0430e7ec - WORD $0x8b0d004d - WORD $0x8b0c004c + WORD $0x0430e7ed + WORD $0x8b0e01ee + WORD $0x8b0d01ed + WORD $0xd10021ef WORD $0x14000006 - WORD $0xfc2a7800 - WORD $0x9100054a - WORD $0x91002021 - WORD $0xeb08015f - WORD $0x54000720 - WORD $0xeb0900bf + WORD $0xfc297800 + WORD $0x91000529 + WORD $0x9100218c + WORD $0xeb08013f + WORD $0x540007a0 + WORD $0xeb0a00bf WORD $0x54000082 WORD $0x2f00e400 - WORD $0xaa1f03f0 + WORD $0xaa1f03f1 WORD $0x14000015 WORD $0x2f00e400 - WORD $0xaa1f03ef - WORD $0xaa0103f0 - WORD $0xa5ef4181 - WORD $0x85804202 + WORD $0xaa1f03f0 + WORD $0xaa0c03f1 + WORD $0xa5f041a1 + WORD $0x85804222 WORD $0x05f83821 WORD $0x65c10841 - WORD $0xa5ef41a2 - WORD $0x04b0e7ef + WORD $0xa5f041c2 + WORD $0x04b0e7f0 WORD $0x05f83842 - WORD $0xab0f017f + WORD $0xab10017f WORD $0x65d82020 - WORD $0x85804601 - WORD $0x0431e3f0 + WORD $0x85804621 + WORD $0x0431e3f1 WORD $0x65c20821 WORD $0x65d82020 WORD $0x54fffe61 WORD $0xeb0b00bf - WORD $0xaa0b03f0 + WORD $0xaa0b03f1 WORD $0x54fffc60 - WORD $0xd37df20f - WORD $0xcb1000b0 - WORD $0xfc6f6821 - WORD $0xfc7079c2 - WORD $0xf1000610 - WORD $0x910021ef + WORD $0xd37df230 + WORD $0xcb1100b1 + WORD $0xfc706981 + WORD $0xfc7179e2 + WORD $0xf1000631 + WORD $0x91002210 WORD $0x1e620821 WORD $0x1e612800 WORD $0x54ffff41 WORD $0x17ffffd9 - WORD $0xeb09011f - WORD $0x54000062 - WORD $0xaa1f03e9 - WORD $0x1400000d + WORD $0x04a0e3ed + WORD $0xeb0d015f + WORD $0x54000243 + WORD $0xd37df12e WORD $0x6f00e400 - WORD $0xcb0903e9 - WORD $0xaa0003ea - WORD $0x8a090109 - WORD $0xaa0903eb - WORD $0xe5804140 - WORD $0x04b0e7eb - WORD $0xe5804540 - WORD $0x0431e3ea - WORD $0xb5ffff8b - WORD $0xeb09011f + WORD $0xcb0d03ed + WORD $0x8a0d014d + WORD $0x25d8e3e0 + WORD $0xaa1f03eb + WORD $0x8b0e000c + WORD $0x0430e3ee + WORD $0x8b0d0129 + WORD $0x8b0e000e + WORD $0xe5eb4180 + WORD $0xe5eb41c0 + WORD $0x04b0e3eb + WORD $0xeb0b01bf + WORD $0x54ffff81 + WORD $0xeb0d015f WORD $0x540000c0 WORD $0x8b090c0a WORD $0xcb090108 WORD $0xf1000508 WORD $0xf800855f WORD $0x54ffffc1 + WORD $0xa9494ff4 + WORD $0xfd400bee + WORD $0xa94857f6 + WORD $0xa9475ff8 + WORD $0xa94667fa + WORD $0xa9457bfd + WORD $0x6d4423e9 + WORD $0x6d432beb + WORD $0x6d4233ed + WORD $0x910283ff WORD $0xd65f03c0 RET @@ -901,78 +1690,299 @@ TEXT ·correlateFloat64SVE2(SB), NOSPLIT|NOFRAME, $0-72 MOVD dst_len+8(FP), R3 MOVD sig_len+32(FP), R4 MOVD ker_len+56(FP), R5 - WORD $0xb4000905 - WORD $0xeb050088 - WORD $0x540008cb - WORD $0xeb08007f - WORD $0x9a88d468 - WORD $0xf100051f - WORD $0x5400084b - WORD $0xf10004bf - WORD $0x04a0e3e9 - WORD $0x5400054b - WORD $0xcb0903eb + WORD $0xb40024a5 + WORD $0xeb050089 + WORD $0x5400246b + WORD $0xd10283ff + WORD $0xeb09007f + WORD $0xaa0203e8 + WORD $0xfd000bee + WORD $0x9a89d462 + WORD $0x6d0233ed + WORD $0xf100405f + WORD $0x6d032beb + WORD $0x6d0423e9 + WORD $0xa9057bfd + WORD $0xa90667fa + WORD $0xa9075ff8 + WORD $0xa90857f6 + WORD $0xa9094ff4 + WORD $0xf90003e1 + WORD $0x5400006a + WORD $0xaa1f03e9 + WORD $0x140000bd + WORD $0xf10000bf + WORD $0x540015ad + WORD $0x04e0e3e9 WORD $0x25d8e3e0 - WORD $0xaa1f03ea - WORD $0x8a0b00ab - WORD $0x14000006 - WORD $0xfc2a7800 - WORD $0x9100054a - WORD $0x91002021 - WORD $0xeb08015f - WORD $0x540006a0 + WORD $0xaa1f03fa + WORD $0xcb0903e9 + WORD $0x9101e02a + WORD $0x5280020b + WORD $0xd280002c + WORD $0xd280004d + WORD $0x8a0900ae + WORD $0xd280006f + WORD $0xd2800090 + WORD $0xd28000b1 + WORD $0xd28000c3 + WORD $0xd28000e4 + WORD $0xd2800106 + WORD $0xd2800127 + WORD $0xd2800153 + WORD $0xd2800174 + WORD $0xd2800195 + WORD $0xd28001b6 + WORD $0xd28001d7 + WORD $0xd28001f8 + WORD $0xaa0103f9 + WORD $0xf9000fe0 + WORD $0xf90007e2 + WORD $0x14000011 + WORD $0xf94007e2 + WORD $0x8b1a0c01 + WORD $0x9100412b + WORD $0x91020339 + WORD $0x9102014a + WORD $0xaa0903fa + WORD $0xeb02017f + WORD $0x6d005837 + WORD $0x6d015035 + WORD $0x6d024833 + WORD $0x6d034031 + WORD $0x6d041827 + WORD $0x6d051025 + WORD $0x6d060823 + WORD $0x6d070021 + WORD $0x5400122c WORD $0x2f00e400 + WORD $0x2f00e401 + WORD $0x04e0e3e9 + WORD $0x2f00e402 + WORD $0x2f00e403 WORD $0xeb0900bf + WORD $0x2f00e404 + WORD $0x2f00e405 + WORD $0xaa0b03e9 + WORD $0x2f00e406 + WORD $0x2f00e407 + WORD $0x2f00e410 + WORD $0x2f00e411 + WORD $0x2f00e412 + WORD $0x2f00e413 + WORD $0x2f00e414 + WORD $0x2f00e415 + WORD $0x2f00e416 + WORD $0x2f00e417 WORD $0x54000062 - WORD $0xaa1f03ec - WORD $0x14000013 - WORD $0xaa0103ec - WORD $0xaa0203ed + WORD $0xaa1f03eb + WORD $0x1400003a + WORD $0xaa1f03fd + WORD $0x8b1d0f2b + WORD $0xa5fd4118 + WORD $0xa5f7417a + WORD $0xa5f84179 + WORD $0xa5ec417b + WORD $0xa5f6417f + WORD $0xa5f54168 + WORD $0xa5ed417c + WORD $0xa5ef417d + WORD $0xa5f0417e + WORD $0xa5f14169 + WORD $0x65da0b1a + WORD $0x65d90b19 + WORD $0xa5e3416a + WORD $0x65df0b1f + WORD $0xa5f4416b + WORD $0xa5f3416c + WORD $0xa5e7416d + WORD $0xa5e4416e + WORD $0x65dc0b1c + WORD $0x65dd0b1d + WORD $0x65de0b1e + WORD $0x65d82341 + WORD $0x65db0b1a + WORD $0x65c80b1b + WORD $0x65d82320 + WORD $0xa5fd4339 + WORD $0xa5e64168 + WORD $0x65cb0b0b + WORD $0x65cc0b0c + WORD $0x04f0e3fd + WORD $0x65d823e2 + WORD $0x65cd0b1f + WORD $0x65d90b19 + WORD $0x65c80b08 + WORD $0x65d82363 + WORD $0x65c90b1b + WORD $0x65ca0b09 + WORD $0x65ce0b18 + WORD $0xeb1d01df + WORD $0x65d82164 + WORD $0x65d82185 + WORD $0x65d823e6 + WORD $0x65d82107 + WORD $0x65d82310 + WORD $0x65d82131 + WORD $0x65d82372 + WORD $0x65d823d3 + WORD $0x65d823b4 + WORD $0x65d82395 + WORD $0x65d82356 + WORD $0x65d82337 + WORD $0x54fff981 + WORD $0xeb0e00bf + WORD $0xaa0e03eb + WORD $0x54fff440 + WORD $0xd37df17d + WORD $0xcb0b00be + WORD $0xaa0a03eb + WORD $0xaa0803e2 + WORD $0x8b1d0161 + WORD $0xfc7d6858 + WORD $0xf10007de + WORD $0x6d78e839 + WORD $0x91002042 + WORD $0x6d79f03b + WORD $0x9100216b + WORD $0x6d7ca829 + WORD $0x1e7a0b1a + WORD $0x1e790b19 + WORD $0x1e7b0b1b + WORD $0x1e7c0b1c + WORD $0x6d7af83d + WORD $0x1e690b09 + WORD $0x6d7ba03f + WORD $0x1e7a2ad6 + WORD $0x6d7de82b + WORD $0x1e792af7 + WORD $0x1e6a0b19 + WORD $0x1e7d0b1d + WORD $0x6d7fb82d + WORD $0x1e7e0b1e + WORD $0x1e6b0b0a + WORD $0x6d7eb02b + WORD $0x1e7f0b1f + WORD $0x1e680b08 + WORD $0x1e7b2ab5 + WORD $0x1e7a0b1a + WORD $0x1e7c2a94 + WORD $0x1e6d0b1c + WORD $0x1e6b0b1b + WORD $0x1e6c0b0b + WORD $0x1e7d2a73 + WORD $0x1e6e0b18 + WORD $0x1e7e2a52 + WORD $0x1e6928e7 + WORD $0x1e7f2a31 + WORD $0x1e682a10 + WORD $0x1e7928c6 + WORD $0x1e6a28a5 + WORD $0x1e7a2884 + WORD $0x1e7c2821 + WORD $0x1e7b2863 + WORD $0x1e6b2842 + WORD $0x1e782800 + WORD $0x54fffa61 + WORD $0x17ffff6f + WORD $0x6f00e400 + WORD $0xaa1f03e9 + WORD $0x9101200a + WORD $0x9100812b + WORD $0xf81b815f + WORD $0x91004129 + WORD $0xad3e0140 + WORD $0xeb02017f + WORD $0xad3f0140 + WORD $0xad000140 + WORD $0x3d800940 + WORD $0xf900195f + WORD $0x9102014a + WORD $0x54fffecd + WORD $0xf94003eb + WORD $0xeb09004a + WORD $0x5400090d + WORD $0xf10000bf + WORD $0x540005ad + WORD $0x04a0e3ea + WORD $0x8b090d6b + WORD $0x25d8e3e0 + WORD $0xcb0a03ec + WORD $0x8a0c00ac + WORD $0x14000006 + WORD $0xfc297800 + WORD $0x91000529 + WORD $0x9100216b + WORD $0xeb02013f + WORD $0x54000760 + WORD $0xeb0a00bf + WORD $0x54000082 + WORD $0x2f00e400 + WORD $0xaa1f03ed + WORD $0x14000015 WORD $0xaa0b03ee - WORD $0x85804181 - WORD $0x858041a2 - WORD $0x04b0e7ee + WORD $0x2f00e400 + WORD $0xaa1f03ed + WORD $0x0430e3ee + WORD $0xaa0803ef + WORD $0xa5ed4161 + WORD $0x858041e2 WORD $0x65c20821 - WORD $0x858045a2 - WORD $0x0431e3ed + WORD $0x858045e2 + WORD $0x0431e3ef WORD $0x65d82020 - WORD $0x85804581 - WORD $0x0431e3ec + WORD $0xa5ed41c1 + WORD $0x04b0e3ed WORD $0x65c20821 + WORD $0xeb0d019f WORD $0x65d82020 - WORD $0xb5fffeae - WORD $0xeb0b00bf - WORD $0xaa0b03ec - WORD $0x54fffca0 - WORD $0xfc6c7821 - WORD $0xfc6c7842 - WORD $0x9100058c + WORD $0x54fffea1 WORD $0xeb0c00bf + WORD $0xaa0c03ed + WORD $0x54fffc60 + WORD $0xfc6d7961 + WORD $0xfc6d7902 + WORD $0x910005ad + WORD $0xeb0d00bf WORD $0x1e620821 WORD $0x1e612800 WORD $0x54ffff41 - WORD $0x17ffffdd - WORD $0xeb09011f - WORD $0x54000062 - WORD $0xaa1f03e9 - WORD $0x1400000d + WORD $0x17ffffdb + WORD $0x04a0e3ec + WORD $0xeb0c015f + WORD $0x54000243 + WORD $0xd37df12d WORD $0x6f00e400 - WORD $0xcb0903e9 - WORD $0xaa0003ea - WORD $0x8a090109 - WORD $0xaa0903eb - WORD $0xe5804140 - WORD $0x04b0e7eb - WORD $0xe5804540 - WORD $0x0431e3ea - WORD $0xb5ffff8b - WORD $0xeb09011f + WORD $0xcb0c03ec + WORD $0x8a0c014c + WORD $0x25d8e3e0 + WORD $0xaa1f03e8 + WORD $0x8b0d000b + WORD $0x0430e3ed + WORD $0x8b0c0129 + WORD $0x8b0d000d + WORD $0xe5e84160 + WORD $0xe5e841a0 + WORD $0x04b0e3e8 + WORD $0xeb08019f + WORD $0x54ffff81 + WORD $0xeb0c015f WORD $0x540000c0 - WORD $0x8b090c0a - WORD $0xcb090108 - WORD $0xf1000508 - WORD $0xf800855f + WORD $0x8b090c08 + WORD $0xcb090049 + WORD $0xf1000529 + WORD $0xf800851f WORD $0x54ffffc1 + WORD $0xa9494ff4 + WORD $0xfd400bee + WORD $0xa94857f6 + WORD $0xa9475ff8 + WORD $0xa94667fa + WORD $0xa9457bfd + WORD $0x6d4423e9 + WORD $0x6d432beb + WORD $0x6d4233ed + WORD $0x910283ff WORD $0xd65f03c0 RET @@ -2461,50 +3471,216 @@ TEXT ·movingAverageFloat32SVE2(SB), NOSPLIT|NOFRAME, $0-56 MOVD a_len+32(FP), R3 MOVD width+48(FP), R4 WORD $0xf100049f - WORD $0x5400056b + WORD $0x5400090b WORD $0xeb040068 - WORD $0x5400052b + WORD $0x540008cb + WORD $0xd10283ff WORD $0xeb08005f + WORD $0x6d013bef WORD $0x9a88d448 - WORD $0xf100051f - WORD $0x540004ab + WORD $0x6d0233ed + WORD $0xf100411f + WORD $0x6d032beb + WORD $0x6d0423e9 + WORD $0xa9057bfd + WORD $0xa90667fa + WORD $0xa9075ff8 + WORD $0xa90857f6 + WORD $0xa9094ff4 + WORD $0xf90007e1 + WORD $0x5400070a + WORD $0xaa1f03e9 + WORD $0xf94007ee + WORD $0xeb08013f + WORD $0x5400052a + WORD $0xd37ef52b WORD $0x9e230080 WORD $0x0460e3ea + WORD $0xcb0a03ec WORD $0x2598e3e0 - WORD $0xcb0a03eb - WORD $0xaa1f03e9 - WORD $0x8a0b008b - WORD $0x14000007 + WORD $0xaa0b03ed + WORD $0x8b0b01cb + WORD $0x8a0c008c + WORD $0x0430e3ed + WORD $0x8b0d01cd + WORD $0x14000008 WORD $0x1e201821 - WORD $0x91001021 + WORD $0x910011ad + WORD $0x9100116b WORD $0xbc297801 WORD $0x91000529 WORD $0xeb08013f - WORD $0x54000300 + WORD $0x540002e0 WORD $0x2f00e401 WORD $0xeb0a009f WORD $0x54000062 - WORD $0xaa1f03ec - WORD $0x1400000d - WORD $0xaa0103ec - WORD $0xaa0b03ed - WORD $0x85804182 - WORD $0x0470e7ed + WORD $0xaa1f03ee + WORD $0x1400000c + WORD $0xaa1f03ee + WORD $0xa54e4162 WORD $0x65982041 - WORD $0x85804582 - WORD $0x0431e3ec + WORD $0xa54e41a2 + WORD $0x0470e3ee + WORD $0xeb0e019f WORD $0x65982041 - WORD $0xb5ffff4d - WORD $0xeb0b009f - WORD $0xaa0b03ec - WORD $0x54fffd40 - WORD $0xbc6c7822 - WORD $0x9100058c + WORD $0x54ffff41 WORD $0xeb0c009f + WORD $0xaa0c03ee + WORD $0x54fffd40 + WORD $0xbc6e7962 + WORD $0x910005ce + WORD $0xeb0e009f WORD $0x1e222821 WORD $0x54ffff81 WORD $0x17ffffe4 + WORD $0xa9494ff4 + WORD $0xa94857f6 + WORD $0xa9475ff8 + WORD $0xa94667fa + WORD $0xa9457bfd + WORD $0x6d4423e9 + WORD $0x6d432beb + WORD $0x6d4233ed + WORD $0x6d413bef + WORD $0x910283ff WORD $0xd65f03c0 + WORD $0x9e230080 + WORD $0x04a0e3ea + WORD $0x2598e3e0 + WORD $0xcb0a03e9 + WORD $0xaa1f03f0 + WORD $0x9100c02b + WORD $0x5280021a + WORD $0xd280002c + WORD $0xd280004d + WORD $0xd280006e + WORD $0xd280008f + WORD $0xd28000b1 + WORD $0x4e040400 + WORD $0xd28000c2 + WORD $0x8a090083 + WORD $0xd28000e5 + WORD $0xd2800106 + WORD $0xd2800127 + WORD $0xd2800153 + WORD $0xd2800174 + WORD $0xd2800195 + WORD $0xd28001b6 + WORD $0xd28001d7 + WORD $0xd28001f8 + WORD $0xaa0103f9 + WORD $0xaa0a03e1 + WORD $0x14000010 + WORD $0x6e0c04c5 + WORD $0x6e20fc42 + WORD $0x8b10080a + WORD $0x9100413a + WORD $0x91010339 + WORD $0x9101016b + WORD $0xeb08035f + WORD $0xaa0903f0 + WORD $0x6e180485 + WORD $0x6e20fca4 + WORD $0x6e20fc63 + WORD $0xad000944 + WORD $0x6e20fc21 + WORD $0xad010543 + WORD $0x54fff42c + WORD $0xeb01009f + WORD $0xaa1a03e9 + WORD $0x54000122 + WORD $0x2f00e404 + WORD $0x6f00e402 + WORD $0xaa1f03fd + WORD $0x2f00e406 + WORD $0x2f00e405 + WORD $0x6f00e403 + WORD $0x6f00e401 + WORD $0x14000043 + WORD $0x2f00e407 + WORD $0x2f00e410 + WORD $0xaa1f03fa + WORD $0x2f00e413 + WORD $0x2f00e401 + WORD $0x2f00e411 + WORD $0x2f00e415 + WORD $0x2f00e417 + WORD $0x2f00e403 + WORD $0x2f00e414 + WORD $0x2f00e416 + WORD $0x2f00e418 + WORD $0x2f00e402 + WORD $0x2f00e412 + WORD $0x2f00e404 + WORD $0x2f00e406 + WORD $0x2f00e405 + WORD $0x8b1a0b3d + WORD $0xa55a4339 + WORD $0x04b0e3fa + WORD $0xa55843be + WORD $0xa54c43ba + WORD $0xa54d43bb + WORD $0xa54e43bc + WORD $0xa54f43bd + WORD $0xa55143bf + WORD $0xa54243a8 + WORD $0xa54543a9 + WORD $0xa54643aa + WORD $0xa54743ab + WORD $0xa55743ac + WORD $0xeb1a007f + WORD $0x659823c7 + WORD $0xa55643be + WORD $0xa55343ad + WORD $0xa55443ae + WORD $0xa55543af + WORD $0x65982190 + WORD $0x659823d3 + WORD $0x659821e1 + WORD $0x659821d1 + WORD $0x659821b5 + WORD $0x65982177 + WORD $0x65982143 + WORD $0x65982134 + WORD $0x65982116 + WORD $0x659823f8 + WORD $0x659823a2 + WORD $0x65982392 + WORD $0x65982364 + WORD $0x65982346 + WORD $0x65982325 + WORD $0x54fffba1 + WORD $0x6e0c0702 + WORD $0x6e0c06e3 + WORD $0xeb03009f + WORD $0x6e0c0661 + WORD $0x6e0c0644 + WORD $0xaa0303fd + WORD $0x6e1406c2 + WORD $0x6e1406a3 + WORD $0x6e140601 + WORD $0x6e1c0682 + WORD $0x6e1c0623 + WORD $0x6e1c04e1 + WORD $0x54fff4a0 + WORD $0xd37ef7ba + WORD $0xcb1d009d + WORD $0xaa0b03fe + WORD $0x8b1a03ca + WORD $0xf10007bd + WORD $0x910013de + WORD $0x2d7a4151 + WORD $0xfc5d8147 + WORD $0xad7f4d52 + WORD $0x3dc00154 + WORD $0x0e24d4e4 + WORD $0x1e252a25 + WORD $0x1e262a06 + WORD $0x4e21d681 + WORD $0x4e22d642 + WORD $0x4e23d663 + WORD $0x54fffe61 + WORD $0x17ffff93 RET // func shiftDivFloat32SVE2(dst []float32, a []float32, shift float32, denom float32) @@ -2632,50 +3808,227 @@ TEXT ·movingAverageFloat64SVE2(SB), NOSPLIT|NOFRAME, $0-56 MOVD a_len+32(FP), R3 MOVD width+48(FP), R4 WORD $0xf100049f - WORD $0x5400056b + WORD $0x540008cb WORD $0xeb040068 - WORD $0x5400052b + WORD $0x5400088b + WORD $0xd10243ff WORD $0xeb08005f + WORD $0x6d0133ed WORD $0x9a88d448 - WORD $0xf100051f - WORD $0x540004ab + WORD $0x6d022beb + WORD $0xf100411f + WORD $0x6d0323e9 + WORD $0xa9047bfd + WORD $0xa90567fa + WORD $0xa9065ff8 + WORD $0xa90757f6 + WORD $0xa9084ff4 + WORD $0xf90007e1 + WORD $0x540006ea + WORD $0xaa1f03e9 + WORD $0xf94007ee + WORD $0xeb08013f + WORD $0x5400052a + WORD $0xd37df12b WORD $0x9e630080 WORD $0x04a0e3ea + WORD $0xcb0a03ec WORD $0x25d8e3e0 - WORD $0xcb0a03eb - WORD $0xaa1f03e9 - WORD $0x8a0b008b - WORD $0x14000007 + WORD $0xaa0b03ed + WORD $0x8b0b01cb + WORD $0x8a0c008c + WORD $0x0430e3ed + WORD $0x8b0d01cd + WORD $0x14000008 WORD $0x1e601821 - WORD $0x91002021 + WORD $0x910021ad + WORD $0x9100216b WORD $0xfc297801 WORD $0x91000529 WORD $0xeb08013f - WORD $0x54000300 + WORD $0x540002e0 WORD $0x2f00e401 WORD $0xeb0a009f WORD $0x54000062 - WORD $0xaa1f03ec - WORD $0x1400000d - WORD $0xaa0103ec - WORD $0xaa0b03ed - WORD $0x85804182 - WORD $0x04b0e7ed + WORD $0xaa1f03ee + WORD $0x1400000c + WORD $0xaa1f03ee + WORD $0xa5ee4162 WORD $0x65d82041 - WORD $0x85804582 - WORD $0x0431e3ec + WORD $0xa5ee41a2 + WORD $0x04b0e3ee + WORD $0xeb0e019f WORD $0x65d82041 - WORD $0xb5ffff4d - WORD $0xeb0b009f - WORD $0xaa0b03ec - WORD $0x54fffd40 - WORD $0xfc6c7822 - WORD $0x9100058c + WORD $0x54ffff41 WORD $0xeb0c009f + WORD $0xaa0c03ee + WORD $0x54fffd40 + WORD $0xfc6e7962 + WORD $0x910005ce + WORD $0xeb0e009f WORD $0x1e622821 WORD $0x54ffff81 WORD $0x17ffffe4 + WORD $0xa9484ff4 + WORD $0xa94757f6 + WORD $0xa9465ff8 + WORD $0xa94567fa + WORD $0xa9447bfd + WORD $0x6d4323e9 + WORD $0x6d422beb + WORD $0x6d4133ed + WORD $0x910243ff WORD $0xd65f03c0 + WORD $0x9e630080 + WORD $0x04e0e3ea + WORD $0x25d8e3e0 + WORD $0xcb0a03e9 + WORD $0xaa1f03f8 + WORD $0x9101e02b + WORD $0x5280021a + WORD $0xd280002c + WORD $0x8a09008d + WORD $0xd280004e + WORD $0xd280006f + WORD $0xd2800090 + WORD $0xd28000b1 + WORD $0xd28000c2 + WORD $0xd28000e3 + WORD $0xd2800105 + WORD $0xd2800126 + WORD $0xd2800147 + WORD $0xd2800173 + WORD $0xd2800194 + WORD $0xd28001b5 + WORD $0xd28001d6 + WORD $0xd28001f7 + WORD $0xaa0103f9 + WORD $0xaa0a03e1 + WORD $0x14000020 + WORD $0x1e601b18 + WORD $0x8b180c0a + WORD $0x9100413a + WORD $0xeb08035f + WORD $0x91020339 + WORD $0x9102016b + WORD $0xaa0903f8 + WORD $0x1e601af7 + WORD $0x1e601ad6 + WORD $0x6d005d58 + WORD $0x1e601ab5 + WORD $0x1e601a94 + WORD $0x6d015556 + WORD $0x1e601a73 + WORD $0x1e601a52 + WORD $0x6d024d54 + WORD $0x1e601a31 + WORD $0x1e601a10 + WORD $0x6d034552 + WORD $0x1e6018e7 + WORD $0x1e6018c6 + WORD $0x6d041d50 + WORD $0x1e6018a5 + WORD $0x1e601884 + WORD $0x6d051546 + WORD $0x1e601863 + WORD $0x1e601842 + WORD $0x6d060d44 + WORD $0x1e601821 + WORD $0x6d070542 + WORD $0x54fff26c + WORD $0x2f00e401 + WORD $0x2f00e402 + WORD $0xeb01009f + WORD $0x2f00e403 + WORD $0x2f00e404 + WORD $0xaa1a03e9 + WORD $0x2f00e405 + WORD $0x2f00e406 + WORD $0x2f00e407 + WORD $0x2f00e410 + WORD $0x2f00e411 + WORD $0x2f00e412 + WORD $0x2f00e413 + WORD $0x2f00e414 + WORD $0x2f00e415 + WORD $0x2f00e416 + WORD $0x2f00e417 + WORD $0x2f00e418 + WORD $0x54000062 + WORD $0xaa1f03fd + WORD $0x14000029 + WORD $0xaa1f03fa + WORD $0x8b1a0f3d + WORD $0xa5fa4339 + WORD $0x04f0e3fa + WORD $0xa5f743bb + WORD $0xa5f643bc + WORD $0xa5f543a8 + WORD $0xa5ec43ba + WORD $0xa5ef43bd + WORD $0xa5f043be + WORD $0xa5f143bf + WORD $0xa5f443a9 + WORD $0xa5e343aa + WORD $0x65d82361 + WORD $0xa5ee43bb + WORD $0xa5e543ab + WORD $0x65d82382 + WORD $0xa5e243bc + WORD $0xa5f343ac + WORD $0x65d82103 + WORD $0xa5e643a8 + WORD $0xa5e743ad + WORD $0x65d82124 + WORD $0xeb1a01bf + WORD $0x65d82185 + WORD $0x65d821a6 + WORD $0x65d82107 + WORD $0x65d82170 + WORD $0x65d82151 + WORD $0x65d82392 + WORD $0x65d823f3 + WORD $0x65d823d4 + WORD $0x65d823b5 + WORD $0x65d82376 + WORD $0x65d82357 + WORD $0x65d82338 + WORD $0x54fffba1 + WORD $0xeb0d009f + WORD $0xaa0d03fd + WORD $0x54fff4a0 + WORD $0xd37df3ba + WORD $0xcb1d009d + WORD $0xaa0b03fe + WORD $0x8b1a03ca + WORD $0xf10007bd + WORD $0x910023de + WORD $0x6d78e959 + WORD $0x6d79f15b + WORD $0x6d7af95d + WORD $0x1e782b38 + WORD $0x1e772b57 + WORD $0x1e762b76 + WORD $0x1e752b95 + WORD $0x6d7be959 + WORD $0x1e742bb4 + WORD $0x6d7cf15b + WORD $0x1e732bd3 + WORD $0x6d7ff95d + WORD $0x1e722b32 + WORD $0x1e712b51 + WORD $0x1e702b70 + WORD $0x1e672b87 + WORD $0x6d7de959 + WORD $0x1e622ba2 + WORD $0x6d7ef15b + WORD $0x1e612bc1 + WORD $0x1e662b26 + WORD $0x1e652b45 + WORD $0x1e642b64 + WORD $0x1e632b83 + WORD $0x54fffca1 + WORD $0x17ffff85 RET // func shiftDivFloat64SVE2(dst []float64, a []float64, shift float64, denom float64) diff --git a/internal/loong64/numeric_lasx_loong64.go b/internal/loong64/numeric_lasx_loong64.go index 5a5cff4..432b6ca 100644 --- a/internal/loong64/numeric_lasx_loong64.go +++ b/internal/loong64/numeric_lasx_loong64.go @@ -99,12 +99,18 @@ func tileUint64LASX(dst []uint64, pattern []uint64) //go:noescape func gatherUint64LASX(dst []uint64, src []uint64, idx []int32) +//go:noescape +func movingAverageFloat32LASX(dst []float32, a []float32, width int) + //go:noescape func shiftDivFloat32LASX(dst []float32, a []float32, shift float32, denom float32) //go:noescape func layerNormFloat32LASX(dst []float32, a []float32, gamma []float32, beta []float32, shift float32, denom float32) +//go:noescape +func movingAverageFloat64LASX(dst []float64, a []float64, width int) + //go:noescape func shiftDivFloat64LASX(dst []float64, a []float64, shift float64, denom float64) diff --git a/internal/loong64/numeric_lasx_loong64.s b/internal/loong64/numeric_lasx_loong64.s index 121db1d..4926710 100644 --- a/internal/loong64/numeric_lasx_loong64.s +++ b/internal/loong64/numeric_lasx_loong64.s @@ -213,22 +213,90 @@ TEXT ·polyEvalFloat32LASX(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x0013a908 WORD $0x001328e7 WORD $0x001520e7 - WORD $0x40007920 - WORD $0x6401dc07 - WORD $0x002c992a - WORD $0x2b3ff140 - WORD $0x03800806 - WORD $0x60007926 + WORD $0x40001920 + WORD $0x0380400a + WORD $0x002c9928 + WORD $0x640020ea WORD $0x00150006 - WORD $0x02fffd28 - WORD $0x02ffe149 + WORD $0x50010c00 + WORD $0x6402d807 + WORD $0x03801005 + WORD $0x6c017ce5 + WORD $0x00150005 + WORD $0x50020c00 + WORD $0x2b3ff100 + WORD $0x03800406 + WORD $0x640088c9 + WORD $0x0015000c + WORD $0x7707c000 + WORD $0x02fffd2a + WORD $0x02ffe10b + WORD $0x0380400e + WORD $0x03400000 + WORD $0x002c9586 + WORD $0x0041098d + WORD $0x384834a1 + WORD $0x2c8080c3 + WORD $0x001501c6 + WORD $0x0015016e + WORD $0x0015014f + WORD $0x77d40002 + WORD $0x77d40004 + WORD $0x03400000 + WORD $0x03400000 WORD $0x03400000 + WORD $0x322001c5 + WORD $0x75388484 + WORD $0x753090a4 + WORD $0x75388c42 + WORD $0x753088a2 + WORD $0x02fffdef + WORD $0x02fff1ce + WORD $0x63ffe40f + WORD $0x002c918c + WORD $0x384c3484 + WORD $0x02c040ce + WORD $0x2cc08182 + WORD $0x001500cc + WORD $0x67ff9cee + WORD $0x50006800 + WORD $0x00150006 + WORD $0x02c0808a + WORD $0x03400000 + WORD $0x03400000 + WORD $0x03400000 + WORD $0x2b7f8140 + WORD $0x2b7f9140 + WORD $0x2b7fa140 + WORD $0x2b7fb140 + WORD $0x2b7fc140 + WORD $0x2b7fd140 + WORD $0x2b7fe140 + WORD $0x2b7ff140 + WORD $0x2b400140 + WORD $0x2b401140 + WORD $0x2b402140 + WORD $0x2b403140 + WORD $0x2b404140 + WORD $0x2b405140 + WORD $0x2b406140 + WORD $0x2b407140 + WORD $0x02c080cb + WORD $0x02c040c6 + WORD $0x02c1014a + WORD $0x67ffb4eb + WORD $0x6401d0c7 + WORD $0x2b3ff100 + WORD $0x0380080a + WORD $0x60005d2a + WORD $0x02fffd29 + WORD $0x02ffe108 WORD $0x03400000 WORD $0x03400000 WORD $0x004108ca WORD $0x383028a1 - WORD $0x0015012b - WORD $0x0015010c + WORD $0x0015010b + WORD $0x0015012c WORD $0x01149402 WORD $0x03400000 WORD $0x03400000 @@ -243,23 +311,19 @@ TEXT ·polyEvalFloat32LASX(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x38382882 WORD $0x5fffc0c7 WORD $0x50016c00 - WORD $0x64016807 - WORD $0x03801005 - WORD $0x6c001ce5 - WORD $0x00150005 - WORD $0x5000ac00 - WORD $0x03801005 - WORD $0x6c001ce5 - WORD $0x00150005 - WORD $0x50012c00 + WORD $0x001198e5 + WORD $0x03801008 + WORD $0x6c001ca8 + WORD $0x001500c5 + WORD $0x50013c00 WORD $0x03804005 WORD $0x6c001ce5 WORD $0x00150005 WORD $0x50005400 - WORD $0x03804005 - WORD $0x6c00a4e5 - WORD $0x00150005 - WORD $0x5000d400 + WORD $0x03804008 + WORD $0x6c00a4a8 + WORD $0x00150008 + WORD $0x5000d800 WORD $0x034030e6 WORD $0x00fe10e5 WORD $0x004110a5 @@ -274,7 +338,7 @@ TEXT ·polyEvalFloat32LASX(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x02ffc129 WORD $0x02c10108 WORD $0x47fff13f - WORD $0x5800ece5 + WORD $0x5800fce5 WORD $0x40003cc0 WORD $0x001500a8 WORD $0x00fe08e5 @@ -289,7 +353,7 @@ TEXT ·polyEvalFloat32LASX(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x02c010c6 WORD $0x02c04108 WORD $0x47fff4df - WORD $0x5800b0e5 + WORD $0x5800c0e5 WORD $0x001194e6 WORD $0x002c90a4 WORD $0x03400000 @@ -297,35 +361,39 @@ TEXT ·polyEvalFloat32LASX(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x02fffcc6 WORD $0x02c01084 WORD $0x47fff4df - WORD $0x50009000 - WORD $0x034030e6 - WORD $0x00fe10e5 - WORD $0x004110a5 + WORD $0x5000a000 + WORD $0x03403ce9 + WORD $0x0011a4a8 WORD $0x7707c001 - WORD $0x02c08088 - WORD $0x001500a9 - WORD $0x03400000 - WORD $0x2cff8101 - WORD $0x2cc00101 - WORD $0x02ffc129 - WORD $0x02c10108 - WORD $0x47fff13f - WORD $0x58005ce5 - WORD $0x40003cc0 - WORD $0x001500a8 - WORD $0x00fe08e5 - WORD $0x004108a5 + WORD $0x002c90ca + WORD $0x02c0814a + WORD $0x0010a4cb + WORD $0x00119d6b + WORD $0x2cff8141 + WORD $0x2cc00141 + WORD $0x02c0416b + WORD $0x02c1014a + WORD $0x47fff17f + WORD $0x40006d20 + WORD $0x0380100a + WORD $0x6800692a + WORD $0x03400ce9 + WORD $0x0011a4a5 + WORD $0x001094c5 WORD $0x72f7e001 - WORD $0x00119506 - WORD $0x002c9108 - WORD $0x03400000 + WORD $0x004108ca + WORD $0x002ca90a + WORD $0x0010a88a + WORD $0x0010a0c6 + WORD $0x0010a4c6 + WORD $0x00119cc6 WORD $0x03400000 WORD $0x03400000 - WORD $0x2c400101 + WORD $0x2c400141 WORD $0x02c010c6 - WORD $0x02c04108 + WORD $0x02c0414a WORD $0x47fff4df - WORD $0x580020e5 + WORD $0x40002120 WORD $0x001194e6 WORD $0x002c90a4 WORD $0x03400000 @@ -334,6 +402,8 @@ TEXT ·polyEvalFloat32LASX(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x02c01084 WORD $0x47fff4df WORD $0x4c000020 + WORD $0x0010a0c5 + WORD $0x53ffdfff RET // func convolveFloat32LASX(dst []float32, sig []float32, ker []float32) @@ -344,22 +414,70 @@ TEXT ·convolveFloat32LASX(SB), NOSPLIT|NOFRAME, $0-72 MOVV dst_len+8(FP), R7 MOVV sig_len+32(FP), R8 MOVV ker_len+56(FP), R9 - WORD $0x40013120 - WORD $0x60012d09 + WORD $0x40021120 + WORD $0x60020d09 WORD $0x0011a508 WORD $0x02c0050a WORD $0x00121d08 WORD $0x0013214a WORD $0x0013a0e7 WORD $0x00151d47 - WORD $0x64011007 - WORD $0x64006409 + WORD $0x03804008 + WORD $0x64000ce8 + WORD $0x00150008 + WORD $0x5000bc00 + WORD $0x64009009 + WORD $0x0015000c + WORD $0x02c080aa + WORD $0x002c9928 + WORD $0x02fff10b + WORD $0x0380400d + WORD $0x77e00000 + WORD $0x03400000 + WORD $0x001501a8 + WORD $0x0015016d + WORD $0x0015014e + WORD $0x0015012f + WORD $0x77d40001 + WORD $0x77d40002 + WORD $0x03400000 + WORD $0x03400000 + WORD $0x2cbf81c3 + WORD $0x322001a4 + WORD $0x2c8001c5 + WORD $0x75388c83 + WORD $0x75308c21 + WORD $0x75389483 + WORD $0x75308c42 + WORD $0x02fffdef + WORD $0x02c011ce + WORD $0x02fff1ad + WORD $0x47ffd9ff + WORD $0x002c918d + WORD $0x0041098c + WORD $0x384c3081 + WORD $0x2cc081a2 + WORD $0x02c0410d + WORD $0x02c1014a + WORD $0x0015010c + WORD $0x67ff98ed + WORD $0x50002c00 WORD $0x00150008 + WORD $0x02c0808a + WORD $0x77e00000 + WORD $0x03400000 + WORD $0x2cff8140 + WORD $0x2cc00140 + WORD $0x02c0810b + WORD $0x02c04108 + WORD $0x02c1014a + WORD $0x67ffeceb + WORD $0x64012907 + WORD $0x64005c09 WORD $0x002c9926 WORD $0x02fff0c6 + WORD $0x002c9505 WORD $0x0114a400 - WORD $0x03400000 - WORD $0x03400000 WORD $0x001500aa WORD $0x001500cb WORD $0x0015012c @@ -377,42 +495,50 @@ TEXT ·convolveFloat32LASX(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x02c00508 WORD $0x02c010a5 WORD $0x5fffc107 - WORD $0x5000ac00 - WORD $0x03801005 - WORD $0x6c000ce5 - WORD $0x00150005 - WORD $0x50008000 - WORD $0x03804005 - WORD $0x6c000ce5 - WORD $0x00150005 - WORD $0x50003800 - WORD $0x034030e6 - WORD $0x00fe10e5 - WORD $0x004110a5 - WORD $0x02c08088 + WORD $0x5000cc00 + WORD $0x0011a0e5 + WORD $0x03801006 + WORD $0x6c000ca6 + WORD $0x00150105 + WORD $0x50009c00 + WORD $0x03804006 + WORD $0x6c000ca6 + WORD $0x00150006 + WORD $0x50004800 + WORD $0x03403ce9 + WORD $0x0011a4a6 + WORD $0x002c910a + WORD $0x02c0814a + WORD $0x0010a50b + WORD $0x00119d6b WORD $0x77e00000 - WORD $0x001500a9 - WORD $0x2cff8100 - WORD $0x2cc00100 - WORD $0x02ffc129 - WORD $0x02c10108 - WORD $0x47fff13f - WORD $0x58005ce5 - WORD $0x40003cc0 - WORD $0x001500a8 - WORD $0x00fe08e5 - WORD $0x004108a5 - WORD $0x00119506 - WORD $0x002c9108 - WORD $0x73e00000 WORD $0x03400000 WORD $0x03400000 + WORD $0x2cff8140 + WORD $0x2cc00140 + WORD $0x02c0416b + WORD $0x02c1014a + WORD $0x47fff17f + WORD $0x40006d20 + WORD $0x0380100a + WORD $0x6800692a + WORD $0x03400ce9 + WORD $0x0011a4a5 + WORD $0x00109505 + WORD $0x0041090a + WORD $0x002ca8ca + WORD $0x0010a88a + WORD $0x00109906 + WORD $0x0010a4c6 + WORD $0x00119cc6 + WORD $0x73e00000 + WORD $0x03400000 WORD $0x03400000 - WORD $0x2c400100 + WORD $0x2c400140 WORD $0x02c010c6 - WORD $0x02c04108 + WORD $0x02c0414a WORD $0x47fff4df - WORD $0x580020e5 + WORD $0x40002120 WORD $0x001194e6 WORD $0x002c90a4 WORD $0x03400000 @@ -421,6 +547,8 @@ TEXT ·convolveFloat32LASX(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x02c01084 WORD $0x47fff4df WORD $0x4c000020 + WORD $0x00109905 + WORD $0x53ffdfff RET // func correlateFloat32LASX(dst []float32, sig []float32, ker []float32) @@ -431,18 +559,70 @@ TEXT ·correlateFloat32LASX(SB), NOSPLIT|NOFRAME, $0-72 MOVV dst_len+8(FP), R7 MOVV sig_len+32(FP), R8 MOVV ker_len+56(FP), R9 - WORD $0x40012120 - WORD $0x60011d09 + WORD $0x40021120 + WORD $0x60020d09 WORD $0x0011a508 WORD $0x02c0050a WORD $0x00121d08 WORD $0x0013214a WORD $0x0013a0e7 WORD $0x00151d47 - WORD $0x64010007 - WORD $0x64005409 + WORD $0x03804008 + WORD $0x64000ce8 + WORD $0x00150008 + WORD $0x5000bc00 + WORD $0x64009009 + WORD $0x0015000b + WORD $0x02c080aa + WORD $0x0380400c + WORD $0x77e00000 + WORD $0x03400000 + WORD $0x03400000 + WORD $0x03400000 + WORD $0x00150188 + WORD $0x001500cc + WORD $0x0015014d + WORD $0x0015012e + WORD $0x77d40001 + WORD $0x77d40002 + WORD $0x03400000 + WORD $0x03400000 + WORD $0x2cbf81a3 + WORD $0x32200184 + WORD $0x2c8001a5 + WORD $0x75388c83 + WORD $0x75308c21 + WORD $0x75389483 + WORD $0x75308c42 + WORD $0x02fffdce + WORD $0x02c011ad + WORD $0x02c0118c + WORD $0x47ffd9df + WORD $0x002c916c + WORD $0x0041096b + WORD $0x384c2c81 + WORD $0x2cc08182 + WORD $0x02c0410c + WORD $0x02c1014a + WORD $0x0015010b + WORD $0x67ff98ec + WORD $0x50002c00 WORD $0x00150008 + WORD $0x02c0808a + WORD $0x77e00000 + WORD $0x03400000 + WORD $0x2cff8140 + WORD $0x2cc00140 + WORD $0x02c0810b + WORD $0x02c04108 + WORD $0x02c1014a + WORD $0x67ffeceb + WORD $0x64012907 + WORD $0x64005c09 + WORD $0x002c9505 WORD $0x0114a400 + WORD $0x03400000 + WORD $0x03400000 WORD $0x001500aa WORD $0x001500cb WORD $0x0015012c @@ -460,42 +640,50 @@ TEXT ·correlateFloat32LASX(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x02c00508 WORD $0x02c010a5 WORD $0x5fffc107 - WORD $0x5000ac00 - WORD $0x03801005 - WORD $0x6c000ce5 - WORD $0x00150005 - WORD $0x50008000 - WORD $0x03804005 - WORD $0x6c000ce5 - WORD $0x00150005 - WORD $0x50003800 - WORD $0x034030e6 - WORD $0x00fe10e5 - WORD $0x004110a5 - WORD $0x02c08088 + WORD $0x5000cc00 + WORD $0x0011a0e5 + WORD $0x03801006 + WORD $0x6c000ca6 + WORD $0x00150105 + WORD $0x50009c00 + WORD $0x03804006 + WORD $0x6c000ca6 + WORD $0x00150006 + WORD $0x50004800 + WORD $0x03403ce9 + WORD $0x0011a4a6 + WORD $0x002c910a + WORD $0x02c0814a + WORD $0x0010a50b + WORD $0x00119d6b WORD $0x77e00000 - WORD $0x001500a9 - WORD $0x2cff8100 - WORD $0x2cc00100 - WORD $0x02ffc129 - WORD $0x02c10108 - WORD $0x47fff13f - WORD $0x58005ce5 - WORD $0x40003cc0 - WORD $0x001500a8 - WORD $0x00fe08e5 - WORD $0x004108a5 - WORD $0x00119506 - WORD $0x002c9108 - WORD $0x73e00000 WORD $0x03400000 WORD $0x03400000 + WORD $0x2cff8140 + WORD $0x2cc00140 + WORD $0x02c0416b + WORD $0x02c1014a + WORD $0x47fff17f + WORD $0x40006d20 + WORD $0x0380100a + WORD $0x6800692a + WORD $0x03400ce9 + WORD $0x0011a4a5 + WORD $0x00109505 + WORD $0x0041090a + WORD $0x002ca8ca + WORD $0x0010a88a + WORD $0x00109906 + WORD $0x0010a4c6 + WORD $0x00119cc6 + WORD $0x73e00000 + WORD $0x03400000 WORD $0x03400000 - WORD $0x2c400100 + WORD $0x2c400140 WORD $0x02c010c6 - WORD $0x02c04108 + WORD $0x02c0414a WORD $0x47fff4df - WORD $0x580020e5 + WORD $0x40002120 WORD $0x001194e6 WORD $0x002c90a4 WORD $0x03400000 @@ -504,6 +692,8 @@ TEXT ·correlateFloat32LASX(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x02c01084 WORD $0x47fff4df WORD $0x4c000020 + WORD $0x00109905 + WORD $0x53ffdfff RET // func normFloat64LASX(a []float64) float64 @@ -709,22 +899,98 @@ TEXT ·polyEvalFloat64LASX(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x0013a908 WORD $0x001328e7 WORD $0x001520e7 - WORD $0x40007920 - WORD $0x64012c07 - WORD $0x002d192a - WORD $0x2bbfe140 - WORD $0x03800806 - WORD $0x60007926 - WORD $0x00150006 - WORD $0x02fffd28 - WORD $0x02ffc149 + WORD $0x40001920 + WORD $0x03804008 + WORD $0x002d1926 + WORD $0x640020e8 + WORD $0x00150008 + WORD $0x50012c00 + WORD $0x64024807 + WORD $0x03802005 + WORD $0x6c019ce5 + WORD $0x00150005 + WORD $0x5001c000 + WORD $0x2bbfe0c0 + WORD $0x03800408 + WORD $0x6400b109 + WORD $0x0015000c + WORD $0x7707e000 + WORD $0x02fffd2a + WORD $0x02ffc0cb + WORD $0x0380400e + WORD $0x03400000 + WORD $0x002d1588 + WORD $0x00410d8d + WORD $0x384834a1 + WORD $0x2c808103 + WORD $0x2c810104 + WORD $0x2c818105 + WORD $0x001501c8 + WORD $0x0015016e + WORD $0x0015014f + WORD $0x77d40002 + WORD $0x77d40006 + WORD $0x77d40007 + WORD $0x77d40008 WORD $0x03400000 WORD $0x03400000 WORD $0x03400000 - WORD $0x00410cca - WORD $0x383428a1 - WORD $0x0015012b + WORD $0x321001c9 + WORD $0x75390508 + WORD $0x75312128 + WORD $0x75390ce7 + WORD $0x75311d27 + WORD $0x753910c6 + WORD $0x75311926 + WORD $0x75391442 + WORD $0x75310922 + WORD $0x02fffdef + WORD $0x02ffe1ce + WORD $0x63ffd40f + WORD $0x002d118c + WORD $0x384c3488 + WORD $0x2cc08187 + WORD $0x2cc10186 + WORD $0x02c0410e + WORD $0x2cc18182 WORD $0x0015010c + WORD $0x67ff74ee + WORD $0x50006000 + WORD $0x00150008 + WORD $0x02c1008a + WORD $0x03400000 + WORD $0x2bff0140 + WORD $0x2bff2140 + WORD $0x2bff4140 + WORD $0x2bff6140 + WORD $0x2bff8140 + WORD $0x2bffa140 + WORD $0x2bffc140 + WORD $0x2bffe140 + WORD $0x2bc00140 + WORD $0x2bc02140 + WORD $0x2bc04140 + WORD $0x2bc06140 + WORD $0x2bc08140 + WORD $0x2bc0a140 + WORD $0x2bc0c140 + WORD $0x2bc0e140 + WORD $0x02c0810b + WORD $0x02c04108 + WORD $0x02c2014a + WORD $0x67ffb4eb + WORD $0x64012107 + WORD $0x2bbfe0c0 + WORD $0x0380080a + WORD $0x60005d2a + WORD $0x02fffd29 + WORD $0x02ffc0c6 + WORD $0x03400000 + WORD $0x03400000 + WORD $0x00410d0a + WORD $0x383428a1 + WORD $0x001500cb + WORD $0x0015012c WORD $0x01149802 WORD $0x03400000 WORD $0x03400000 @@ -735,19 +1001,15 @@ TEXT ·polyEvalFloat64LASX(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x02fffd8c WORD $0x02ffe16b WORD $0x63ffec0c - WORD $0x02c004c6 + WORD $0x02c00508 WORD $0x383c2882 - WORD $0x5fffc0c7 + WORD $0x5fffc107 WORD $0x5000bc00 - WORD $0x6400b807 - WORD $0x03802005 - WORD $0x6c001ce5 - WORD $0x00150005 - WORD $0x50004000 - WORD $0x03802005 - WORD $0x6c0054e5 - WORD $0x00150005 - WORD $0x50008000 + WORD $0x0011a0e5 + WORD $0x03802006 + WORD $0x6c0054a6 + WORD $0x00150105 + WORD $0x50009000 WORD $0x00fe0ce5 WORD $0x00410ca5 WORD $0x02c08086 @@ -758,27 +1020,31 @@ TEXT ·polyEvalFloat64LASX(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x02ffe108 WORD $0x02c100c6 WORD $0x47fff11f - WORD $0x58006ce5 + WORD $0x58007ce5 WORD $0x001194e6 WORD $0x002d10a4 WORD $0x29c00080 WORD $0x02fffcc6 WORD $0x02c02084 WORD $0x47fff4df - WORD $0x50005000 - WORD $0x00fe0ce5 - WORD $0x00410ca5 + WORD $0x50006000 + WORD $0x03401ce6 + WORD $0x001198a5 + WORD $0x00109505 WORD $0x7707e001 - WORD $0x02c08086 - WORD $0x001500a8 + WORD $0x002d1109 + WORD $0x02c08129 + WORD $0x00109908 + WORD $0x00119d08 WORD $0x03400000 WORD $0x03400000 - WORD $0x2cff80c1 - WORD $0x2cc000c1 - WORD $0x02ffe108 - WORD $0x02c100c6 + WORD $0x03400000 + WORD $0x2cff8121 + WORD $0x2cc00121 + WORD $0x02c02108 + WORD $0x02c10129 WORD $0x47fff11f - WORD $0x58001ce5 + WORD $0x40001cc0 WORD $0x001194e6 WORD $0x002d10a4 WORD $0x2bc00080 @@ -796,19 +1062,79 @@ TEXT ·convolveFloat64LASX(SB), NOSPLIT|NOFRAME, $0-72 MOVV dst_len+8(FP), R7 MOVV sig_len+32(FP), R8 MOVV ker_len+56(FP), R9 - WORD $0x4000e120 - WORD $0x6000dd09 + WORD $0x4001e120 + WORD $0x6001dd09 WORD $0x0011a508 WORD $0x02c0050a WORD $0x00121d08 WORD $0x0013214a WORD $0x0013a0e7 WORD $0x00151d47 - WORD $0x6400c007 - WORD $0x64006409 + WORD $0x03804008 + WORD $0x64000ce8 + WORD $0x00150008 + WORD $0x5000e400 + WORD $0x6400b009 + WORD $0x0015000c + WORD $0x02c100aa + WORD $0x002d1928 + WORD $0x02ffe10b + WORD $0x0380400d + WORD $0x77e00000 + WORD $0x03400000 + WORD $0x001501a8 + WORD $0x0015016d + WORD $0x0015014e + WORD $0x0015012f + WORD $0x77d40001 + WORD $0x77d40002 + WORD $0x77d40003 + WORD $0x77d40004 + WORD $0x2cbf01c5 + WORD $0x321001a6 + WORD $0x2cbf81c7 + WORD $0x753914c5 + WORD $0x75311421 + WORD $0x2c8001c5 + WORD $0x75391cc7 + WORD $0x2c8081c8 + WORD $0x75311c42 + WORD $0x753914c5 + WORD $0x75311463 + WORD $0x753920c5 + WORD $0x75311484 + WORD $0x02fffdef + WORD $0x02c021ce + WORD $0x02ffe1ad + WORD $0x47ffc1ff + WORD $0x002d118d + WORD $0x00410d8c + WORD $0x384c3081 + WORD $0x2cc081a2 + WORD $0x2cc101a3 + WORD $0x2cc181a4 + WORD $0x02c0410d + WORD $0x02c2014a + WORD $0x0015010c + WORD $0x67ff78ed + WORD $0x50003400 WORD $0x00150008 + WORD $0x02c1008a + WORD $0x77e00000 + WORD $0x03400000 + WORD $0x2cff0140 + WORD $0x2cff8140 + WORD $0x2cc00140 + WORD $0x2cc08140 + WORD $0x02c0810b + WORD $0x02c04108 + WORD $0x02c2014a + WORD $0x67ffe4eb + WORD $0x6400d107 + WORD $0x64006409 WORD $0x002d1926 WORD $0x02ffe0c6 + WORD $0x002d1505 WORD $0x0114a800 WORD $0x03400000 WORD $0x03400000 @@ -829,23 +1155,27 @@ TEXT ·convolveFloat64LASX(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x02c00508 WORD $0x02c020a5 WORD $0x5fffc107 - WORD $0x50005c00 - WORD $0x03802005 - WORD $0x6c000ce5 - WORD $0x00150005 - WORD $0x50003400 - WORD $0x00fe0ce5 - WORD $0x00410ca5 - WORD $0x02c08086 + WORD $0x50006c00 + WORD $0x0011a0e5 + WORD $0x03802006 + WORD $0x6c000ca6 + WORD $0x00150105 + WORD $0x50004000 + WORD $0x03401ce6 + WORD $0x001198a5 + WORD $0x00109505 + WORD $0x002d1109 + WORD $0x02c08129 + WORD $0x00109908 + WORD $0x00119d08 WORD $0x77e00000 - WORD $0x001500a8 WORD $0x03400000 - WORD $0x2cff80c0 - WORD $0x2cc000c0 - WORD $0x02ffe108 - WORD $0x02c100c6 + WORD $0x2cff8120 + WORD $0x2cc00120 + WORD $0x02c02108 + WORD $0x02c10129 WORD $0x47fff11f - WORD $0x58001ce5 + WORD $0x40001cc0 WORD $0x001194e6 WORD $0x002d10a4 WORD $0x29c00080 @@ -863,17 +1193,77 @@ TEXT ·correlateFloat64LASX(SB), NOSPLIT|NOFRAME, $0-72 MOVV dst_len+8(FP), R7 MOVV sig_len+32(FP), R8 MOVV ker_len+56(FP), R9 - WORD $0x4000d120 - WORD $0x6000cd09 + WORD $0x4001d120 + WORD $0x6001cd09 WORD $0x0011a508 WORD $0x02c0050a WORD $0x00121d08 WORD $0x0013214a WORD $0x0013a0e7 WORD $0x00151d47 - WORD $0x6400b007 - WORD $0x64005409 + WORD $0x03804008 + WORD $0x64000ce8 WORD $0x00150008 + WORD $0x5000e400 + WORD $0x6400b009 + WORD $0x0015000b + WORD $0x02c100aa + WORD $0x0380400c + WORD $0x77e00000 + WORD $0x03400000 + WORD $0x03400000 + WORD $0x03400000 + WORD $0x00150188 + WORD $0x0015014c + WORD $0x001500cd + WORD $0x0015012e + WORD $0x77d40001 + WORD $0x77d40002 + WORD $0x77d40003 + WORD $0x77d40004 + WORD $0x2cbf0185 + WORD $0x321001a6 + WORD $0x2cbf8187 + WORD $0x753914c5 + WORD $0x75311421 + WORD $0x2c800185 + WORD $0x75391cc7 + WORD $0x2c808188 + WORD $0x75311c42 + WORD $0x753914c5 + WORD $0x75311463 + WORD $0x753920c5 + WORD $0x75311484 + WORD $0x02fffdce + WORD $0x02c021ad + WORD $0x02c0218c + WORD $0x47ffc1df + WORD $0x002d116c + WORD $0x00410d6b + WORD $0x384c2c81 + WORD $0x2cc08182 + WORD $0x2cc10183 + WORD $0x2cc18184 + WORD $0x02c0410c + WORD $0x02c2014a + WORD $0x0015010b + WORD $0x67ff78ec + WORD $0x50003400 + WORD $0x00150008 + WORD $0x02c1008a + WORD $0x77e00000 + WORD $0x03400000 + WORD $0x2cff0140 + WORD $0x2cff8140 + WORD $0x2cc00140 + WORD $0x2cc08140 + WORD $0x02c0810b + WORD $0x02c04108 + WORD $0x02c2014a + WORD $0x67ffe4eb + WORD $0x6400c107 + WORD $0x64005409 + WORD $0x002d1505 WORD $0x0114a800 WORD $0x001500aa WORD $0x001500cb @@ -892,23 +1282,27 @@ TEXT ·correlateFloat64LASX(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x02c00508 WORD $0x02c020a5 WORD $0x5fffc107 - WORD $0x50005c00 - WORD $0x03802005 - WORD $0x6c000ce5 - WORD $0x00150005 - WORD $0x50003400 - WORD $0x00fe0ce5 - WORD $0x00410ca5 - WORD $0x02c08086 + WORD $0x50006c00 + WORD $0x0011a0e5 + WORD $0x03802006 + WORD $0x6c000ca6 + WORD $0x00150105 + WORD $0x50004000 + WORD $0x03401ce6 + WORD $0x001198a5 + WORD $0x00109505 + WORD $0x002d1109 + WORD $0x02c08129 + WORD $0x00109908 + WORD $0x00119d08 WORD $0x77e00000 - WORD $0x001500a8 WORD $0x03400000 - WORD $0x2cff80c0 - WORD $0x2cc000c0 - WORD $0x02ffe108 - WORD $0x02c100c6 + WORD $0x2cff8120 + WORD $0x2cc00120 + WORD $0x02c02108 + WORD $0x02c10129 WORD $0x47fff11f - WORD $0x58001ce5 + WORD $0x40001cc0 WORD $0x001194e6 WORD $0x002d10a4 WORD $0x29c00080 @@ -3098,6 +3492,83 @@ TEXT ·gatherUint64LASX(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x4c000020 RET +// func movingAverageFloat32LASX(dst []float32, a []float32, width int) +TEXT ·movingAverageFloat32LASX(SB), NOSPLIT|NOFRAME, $0-56 + MOVV dst_base+0(FP), R4 + MOVV a_base+24(FP), R5 + MOVV dst_len+8(FP), R6 + MOVV a_len+32(FP), R7 + MOVV width+48(FP), R8 + WORD $0x64010c08 + WORD $0x600108e8 + WORD $0x0011a0e7 + WORD $0x02c004e9 + WORD $0x001218e7 + WORD $0x00131d29 + WORD $0x00139cc6 + WORD $0x00151926 + WORD $0x0380400b + WORD $0x0114a900 + WORD $0x64000ccb + WORD $0x00150007 + WORD $0x50008400 + WORD $0x0015000a + WORD $0x011d1801 + WORD $0x7707c021 + WORD $0x02c080a9 + WORD $0x77e00002 + WORD $0x03400000 + WORD $0x03400000 + WORD $0x00150167 + WORD $0x0015010b + WORD $0x0015012c + WORD $0x77d40044 + WORD $0x77d40043 + WORD $0x03400000 + WORD $0x03400000 + WORD $0x03400000 + WORD $0x2cbf8185 + WORD $0x2c800186 + WORD $0x753090a4 + WORD $0x75308cc3 + WORD $0x02fffd6b + WORD $0x02c0118c + WORD $0x47ffe97f + WORD $0x002c914b + WORD $0x0041094a + WORD $0x753a8484 + WORD $0x384c2884 + WORD $0x753a8463 + WORD $0x2cc08163 + WORD $0x02c040eb + WORD $0x02c10129 + WORD $0x001500ea + WORD $0x67ffa0cb + WORD $0x640058e6 + WORD $0x011d1800 + WORD $0x002c94e5 + WORD $0x0114a401 + WORD $0x03400000 + WORD $0x03400000 + WORD $0x03400000 + WORD $0x001500a9 + WORD $0x0015010a + WORD $0x01149422 + WORD $0x03400000 + WORD $0x2b000123 + WORD $0x01008c42 + WORD $0x02fffd4a + WORD $0x02c01129 + WORD $0x47fff15f + WORD $0x01068042 + WORD $0x004108e9 + WORD $0x38382482 + WORD $0x02c004e7 + WORD $0x02c010a5 + WORD $0x5fffc8e6 + WORD $0x4c000020 + RET + // func shiftDivFloat32LASX(dst []float32, a []float32, shift float32, denom float32) TEXT ·shiftDivFloat32LASX(SB), NOSPLIT|NOFRAME, $0-56 MOVV dst_base+0(FP), R4 @@ -3274,6 +3745,91 @@ TEXT ·layerNormFloat32LASX(SB), NOSPLIT|NOFRAME, $0-104 WORD $0x4c000020 RET +// func movingAverageFloat64LASX(dst []float64, a []float64, width int) +TEXT ·movingAverageFloat64LASX(SB), NOSPLIT|NOFRAME, $0-56 + MOVV dst_base+0(FP), R4 + MOVV a_base+24(FP), R5 + MOVV dst_len+8(FP), R6 + MOVV a_len+32(FP), R7 + MOVV width+48(FP), R8 + WORD $0x64012c08 + WORD $0x600128e8 + WORD $0x0011a0e7 + WORD $0x02c004e9 + WORD $0x001218e7 + WORD $0x00131d29 + WORD $0x00139cc6 + WORD $0x00151926 + WORD $0x0380400b + WORD $0x0114a900 + WORD $0x64000ccb + WORD $0x00150007 + WORD $0x5000a400 + WORD $0x0015000a + WORD $0x011d2801 + WORD $0x7707e021 + WORD $0x02c100a9 + WORD $0x77e00002 + WORD $0x03400000 + WORD $0x03400000 + WORD $0x00150167 + WORD $0x0015010b + WORD $0x0015012c + WORD $0x77d40046 + WORD $0x77d40045 + WORD $0x77d40044 + WORD $0x77d40043 + WORD $0x03400000 + WORD $0x2cbf0187 + WORD $0x2cbf8188 + WORD $0x2c800189 + WORD $0x2c80818a + WORD $0x753118e6 + WORD $0x75311505 + WORD $0x75311124 + WORD $0x75310d43 + WORD $0x02fffd6b + WORD $0x02c0218c + WORD $0x47ffd97f + WORD $0x002d114b + WORD $0x00410d4a + WORD $0x753b04c6 + WORD $0x384c2886 + WORD $0x753b04a5 + WORD $0x2cc08165 + WORD $0x753b0484 + WORD $0x2cc10164 + WORD $0x753b0463 + WORD $0x2cc18163 + WORD $0x02c040eb + WORD $0x02c20129 + WORD $0x001500ea + WORD $0x67ff80cb + WORD $0x640058e6 + WORD $0x011d2800 + WORD $0x002d14e5 + WORD $0x0114a801 + WORD $0x03400000 + WORD $0x03400000 + WORD $0x03400000 + WORD $0x001500a9 + WORD $0x0015010a + WORD $0x01149822 + WORD $0x03400000 + WORD $0x2b800123 + WORD $0x01010c42 + WORD $0x02fffd4a + WORD $0x02c02129 + WORD $0x47fff15f + WORD $0x01070042 + WORD $0x00410ce9 + WORD $0x383c2482 + WORD $0x02c004e7 + WORD $0x02c020a5 + WORD $0x5fffc8e6 + WORD $0x4c000020 + RET + // func shiftDivFloat64LASX(dst []float64, a []float64, shift float64, denom float64) TEXT ·shiftDivFloat64LASX(SB), NOSPLIT|NOFRAME, $0-64 MOVV dst_base+0(FP), R4 diff --git a/internal/loong64/numeric_register_lasx_loong64.go b/internal/loong64/numeric_register_lasx_loong64.go index 59bac6e..a5b9a22 100644 --- a/internal/loong64/numeric_register_lasx_loong64.go +++ b/internal/loong64/numeric_register_lasx_loong64.go @@ -242,6 +242,14 @@ func GatherUint64LASX(dst []uint64, src []uint64, idx []int32) { gatherUint64LASX(dst, src, idx) } +func MovingAverageFloat32LASX(dst []float32, a []float32, width int) { + if len(dst) < 16 { + ref.MovingAverage(dst, a, width) + return + } + movingAverageFloat32LASX(dst, a, width) +} + func ShiftDivFloat32LASX(dst []float32, a []float32, shift float32, denom float32) { n := min(len(dst), len(a)) if n < 16 { @@ -260,6 +268,14 @@ func LayerNormFloat32LASX(dst []float32, a []float32, gamma []float32, beta []fl layerNormFloat32LASX(dst[:n:n], a, gamma, beta, shift, denom) } +func MovingAverageFloat64LASX(dst []float64, a []float64, width int) { + if len(dst) < 16 { + ref.MovingAverage(dst, a, width) + return + } + movingAverageFloat64LASX(dst, a, width) +} + func ShiftDivFloat64LASX(dst []float64, a []float64, shift float64, denom float64) { n := min(len(dst), len(a)) if n < 16 { @@ -331,8 +347,10 @@ func registerNumericLASX(s *kernel.Set) { s.U32.Gather = GatherUint32LASX s.U64.Tile = TileUint64LASX s.U64.Gather = GatherUint64LASX + s.F32.MovingAverage = MovingAverageFloat32LASX s.F32.ShiftDiv = ShiftDivFloat32LASX s.F32.LayerNorm = LayerNormFloat32LASX + s.F64.MovingAverage = MovingAverageFloat64LASX s.F64.ShiftDiv = ShiftDivFloat64LASX s.F64.LayerNorm = LayerNormFloat64LASX s.U64.Random = RandomU64LASX diff --git a/internal/riscv64/numeric_rvv_riscv64.s b/internal/riscv64/numeric_rvv_riscv64.s index 552bfe0..541ab7a 100644 --- a/internal/riscv64/numeric_rvv_riscv64.s +++ b/internal/riscv64/numeric_rvv_riscv64.s @@ -263,20 +263,80 @@ TEXT ·polyEvalFloat32RVV(SB), NOSPLIT|NOFRAME, $0-72 MOV coeffs_len+56(FP), X15 WORD $0x00e6c463 WORD $0x00070693 - WORD $0x06078663 - WORD $0x0ad05e63 - WORD $0x00279713 - WORD $0x00e60733 - WORD $0xffc72787 - WORD $0x00200613 - WORD $0x08c7c063 - WORD $0x00000613 + WORD $0x00078c63 + WORD $0x01000713 + WORD $0x00279813 + WORD $0x02e6dc63 + WORD $0x00000713 + WORD $0x0e80006f + WORD $0x16d05c63 + WORD $0x00000593 + WORD $0x0d16f657 + WORD $0x00259713 + WORD $0x00e50733 + WORD $0x5e003457 + WORD $0x40c686b3 + WORD $0x02076427 + WORD $0x00c585b3 + WORD $0xfe0692e3 + WORD $0x1500006f + WORD $0x010602b3 + WORD $0xffc2a787 + WORD $0x00100713 + WORD $0x08f75063 + WORD $0x00000713 + WORD $0xcd147057 + WORD $0x5e07d457 + WORD $0xfff78893 + WORD $0xff828293 + WORD $0x01000393 + WORD $0x00271313 + WORD $0x00658733 + WORD $0x02076507 + WORD $0x02070713 + WORD $0x02076607 + WORD $0x00038713 + WORD $0x00028393 + WORD $0x00088e13 + WORD $0x9e80b857 + WORD $0x9e80b757 + WORD $0x0003a787 + WORD $0x93051857 + WORD $0x92e61757 + WORD $0xfffe0e13 + WORD $0x0307d857 + WORD $0x02e7d757 + WORD $0xffc38393 + WORD $0xffc042e3 + WORD $0x00650333 + WORD $0x02036827 + WORD $0x02030313 + WORD $0x01070393 + WORD $0x02036727 + WORD $0xfa76d2e3 + WORD $0x0300006f + WORD $0x00000713 + WORD $0xcd147057 + WORD $0x5e07d457 + WORD $0x00050893 + WORD $0x02088293 + WORD $0x0208e427 + WORD $0x0202e427 + WORD $0x02070293 + WORD $0x01070713 + WORD $0x04088893 + WORD $0xfe56d4e3 + WORD $0x08d75a63 + WORD $0x01060633 + WORD $0xffc62787 + WORD $0x00200813 + WORD $0x0507c863 WORD $0xfff78793 - WORD $0xff870713 - WORD $0x00261813 + WORD $0xff860613 + WORD $0x00271813 WORD $0x010588b3 WORD $0x0008a707 - WORD $0x00070893 + WORD $0x00060893 WORD $0x00078293 WORD $0x20f786d3 WORD $0x0008a607 @@ -286,22 +346,14 @@ TEXT ·polyEvalFloat32RVV(SB), NOSPLIT|NOFRAME, $0-72 WORD $0xffc88893 WORD $0xfe5046e3 WORD $0x01050833 - WORD $0x00160613 + WORD $0x00170713 WORD $0x00d82027 - WORD $0xfcd612e3 - WORD $0x0580006f - WORD $0x04d05a63 - WORD $0x00000593 - WORD $0x0d16f657 - WORD $0x00259713 - WORD $0x00e50733 - WORD $0x5e003457 - WORD $0x40c686b3 - WORD $0x02076427 - WORD $0x00c585b3 - WORD $0xfe0692e3 - WORD $0x02c0006f + WORD $0xfcd712e3 + WORD $0x0380006f WORD $0x00000593 + WORD $0x40e686b3 + WORD $0x00271713 + WORD $0x00e50533 WORD $0x0d107657 WORD $0x5e07d457 WORD $0x0c76f657 @@ -322,14 +374,189 @@ TEXT ·convolveFloat32RVV(SB), NOSPLIT|NOFRAME, $0-72 MOV dst_len+8(FP), X13 MOV sig_len+32(FP), X14 MOV ker_len+56(FP), X15 - WORD $0x0c078c63 - WORD $0x0cf74a63 + WORD $0x3a078c63 + WORD $0x3af74a63 WORD $0x40f70733 WORD $0x00d75463 WORD $0x00170693 - WORD $0x0cd05263 - WORD $0x08f05e63 + WORD $0xfe010113 + WORD $0x00813c23 + WORD $0x00913823 + WORD $0xc2202773 + WORD $0x00171713 + WORD $0x40e10133 + WORD $0x01000713 + WORD $0x00e6d663 + WORD $0x00000713 + WORD $0x29c0006f WORD $0x00000713 + WORD $0x26f05663 + WORD $0x00279893 + WORD $0x01000e13 + WORD $0x00100813 + WORD $0x011608b3 + WORD $0xffc00293 + WORD $0x00000393 + WORD $0x00271313 + WORD $0x000e0713 + WORD $0x00658e33 + WORD $0x0d007ed7 + WORD $0x42006c57 + WORD $0x5e0c0cd7 + WORD $0x5e0c0dd7 + WORD $0x5e0c0d57 + WORD $0x5e0c0657 + WORD $0x5e0c06d7 + WORD $0x5e0c0757 + WORD $0x5e0c07d7 + WORD $0x5e0c0857 + WORD $0x5e0c08d7 + WORD $0x5e0c0957 + WORD $0x5e0c09d7 + WORD $0x5e0c0a57 + WORD $0x5e0c0ad7 + WORD $0x5e0c0b57 + WORD $0x5e0c0bd7 + WORD $0x00078e93 + WORD $0x0d0eff57 + WORD $0xfff3cf93 + WORD $0x00239413 + WORD $0x002f9f93 + WORD $0x41e804b3 + WORD $0x008e0433 + WORD $0x01f88fb3 + WORD $0x00249493 + WORD $0x009f8fb3 + WORD $0x00440493 + WORD $0x0204e507 + WORD $0x00840493 + WORD $0x0204e587 + WORD $0x00c40493 + WORD $0x0204e407 + WORD $0x01010493 + WORD $0x02848427 + WORD $0x01040493 + WORD $0x0204ee07 + WORD $0x01440493 + WORD $0x0204ee87 + WORD $0x01840493 + WORD $0x0204ef07 + WORD $0x01c40493 + WORD $0x0204ef87 + WORD $0x02040493 + WORD $0x0204e387 + WORD $0x02440493 + WORD $0x0204e307 + WORD $0x02840493 + WORD $0x0204e287 + WORD $0x02c40493 + WORD $0x0204e207 + WORD $0x03040493 + WORD $0x0204e187 + WORD $0x03440493 + WORD $0x0204e107 + WORD $0x002f1493 + WORD $0x01f48fb3 + WORD $0x02046087 + WORD $0xffcf8f93 + WORD $0x0a5fe007 + WORD $0x03840f93 + WORD $0x03c40413 + WORD $0x020fe407 + WORD $0x02046487 + WORD $0x920090d7 + WORD $0xc2202ff3 + WORD $0x01f10fb3 + WORD $0x010f8f93 + WORD $0x028f80a7 + WORD $0x920510d7 + WORD $0x92059557 + WORD $0x01010f93 + WORD $0x028f8587 + WORD $0x920595d7 + WORD $0x920e1e57 + WORD $0x920e9ed7 + WORD $0x920f1f57 + WORD $0x920f9fd7 + WORD $0x920393d7 + WORD $0x92031357 + WORD $0x920292d7 + WORD $0x92021257 + WORD $0x920191d7 + WORD $0x92011157 + WORD $0x92041457 + WORD $0x920494d7 + WORD $0x0e9c9cd7 + WORD $0x0e8d9dd7 + WORD $0x0e2d1d57 + WORD $0x0e361657 + WORD $0x0e4696d7 + WORD $0x0e571757 + WORD $0x0e6797d7 + WORD $0x0e781857 + WORD $0x0ff898d7 + WORD $0x0fe91957 + WORD $0x0fd999d7 + WORD $0x0fca1a57 + WORD $0x0eba9ad7 + WORD $0x0eab1b57 + WORD $0x0e1b9bd7 + WORD $0xc2202ff3 + WORD $0x01f10fb3 + WORD $0x010f8f93 + WORD $0x028f8407 + WORD $0x0e8c1c57 + WORD $0x41ee8eb3 + WORD $0x01e383b3 + WORD $0xe80e9ce3 + WORD $0x439017d7 + WORD $0x43b01757 + WORD $0x43a016d7 + WORD $0x42c01657 + WORD $0x42d015d7 + WORD $0x42e01557 + WORD $0x42f01057 + WORD $0x430010d7 + WORD $0x43101157 + WORD $0x432011d7 + WORD $0x43301257 + WORD $0x434012d7 + WORD $0x43501357 + WORD $0x436013d7 + WORD $0x43701857 + WORD $0x438018d7 + WORD $0x00650333 + WORD $0x01070e13 + WORD $0x01132027 + WORD $0x01032227 + WORD $0x00732427 + WORD $0x00632627 + WORD $0x00532827 + WORD $0x00432a27 + WORD $0x00332c27 + WORD $0x00232e27 + WORD $0x02132027 + WORD $0x02032227 + WORD $0x02a32427 + WORD $0x02b32627 + WORD $0x02c32827 + WORD $0x02d32a27 + WORD $0x02e32c27 + WORD $0x02f32e27 + WORD $0xdbc6dae3 + WORD $0x02c0006f + WORD $0xcd147057 + WORD $0x5e003457 + WORD $0x00050813 + WORD $0x02080893 + WORD $0x02086427 + WORD $0x0208e427 + WORD $0x02070893 + WORD $0x01070713 + WORD $0x04080813 + WORD $0xff16d4e3 + WORD $0x0cd75663 + WORD $0x08f05c63 WORD $0x00279893 WORD $0x0d007857 WORD $0x42006457 @@ -366,8 +593,11 @@ TEXT ·convolveFloat32RVV(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x00170713 WORD $0x00f2a027 WORD $0xf8d716e3 - WORD $0x0280006f + WORD $0x0340006f WORD $0x00000593 + WORD $0x40e686b3 + WORD $0x00271713 + WORD $0x00e50533 WORD $0x0d16f657 WORD $0x00259713 WORD $0x00e50733 @@ -376,6 +606,12 @@ TEXT ·convolveFloat32RVV(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x02076427 WORD $0x00c585b3 WORD $0xfe0692e3 + WORD $0xc2202573 + WORD $0x00151513 + WORD $0x00a10133 + WORD $0x01813403 + WORD $0x01013483 + WORD $0x02010113 WORD $0x00008067 RET @@ -387,14 +623,188 @@ TEXT ·correlateFloat32RVV(SB), NOSPLIT|NOFRAME, $0-72 MOV dst_len+8(FP), X13 MOV sig_len+32(FP), X14 MOV ker_len+56(FP), X15 - WORD $0x0a078463 - WORD $0x0af74263 + WORD $0x36078663 + WORD $0x36f74463 WORD $0x40f70733 WORD $0x00d75463 WORD $0x00170693 - WORD $0x08d05a63 - WORD $0x06f05663 + WORD $0x01000713 + WORD $0x00e6d863 + WORD $0x00000713 + WORD $0x2ad04a63 + WORD $0x3480006f + WORD $0xff010113 + WORD $0xc2202773 + WORD $0x00171713 + WORD $0x40e10133 WORD $0x00000713 + WORD $0x24f05e63 + WORD $0x01000293 + WORD $0x00000893 + WORD $0x00271813 + WORD $0x00028713 + WORD $0x010582b3 + WORD $0x0d007357 + WORD $0x42006c57 + WORD $0x5e0c0d57 + WORD $0x5e0c0557 + WORD $0x5e0c05d7 + WORD $0x5e0c0657 + WORD $0x5e0c06d7 + WORD $0x5e0c0757 + WORD $0x5e0c07d7 + WORD $0x5e0c0857 + WORD $0x5e0c08d7 + WORD $0x5e0c0957 + WORD $0x5e0c09d7 + WORD $0x5e0c0a57 + WORD $0x5e0c0ad7 + WORD $0x5e0c0b57 + WORD $0x5e0c0bd7 + WORD $0x00078313 + WORD $0x0d0373d7 + WORD $0x00289e13 + WORD $0x01c60eb3 + WORD $0x01c28e33 + WORD $0x020eec87 + WORD $0x020e6407 + WORD $0xc2202ef3 + WORD $0x01d10eb3 + WORD $0x010e8e93 + WORD $0x028e8427 + WORD $0x004e0e93 + WORD $0x020ee407 + WORD $0x01010e93 + WORD $0x028e8427 + WORD $0x008e0e93 + WORD $0x020eee07 + WORD $0x00ce0e93 + WORD $0x020eee87 + WORD $0x010e0e93 + WORD $0x020eef07 + WORD $0x014e0e93 + WORD $0x020eef87 + WORD $0x018e0e93 + WORD $0x020ee387 + WORD $0x01ce0e93 + WORD $0x020ee307 + WORD $0x020e0e93 + WORD $0x020ee287 + WORD $0x024e0e93 + WORD $0x020ee207 + WORD $0x028e0e93 + WORD $0x020ee187 + WORD $0x02ce0e93 + WORD $0x020ee107 + WORD $0x030e0e93 + WORD $0x020ee087 + WORD $0x034e0e93 + WORD $0x020ee007 + WORD $0x038e0e93 + WORD $0x03ce0e13 + WORD $0x020ee407 + WORD $0x020e6487 + WORD $0xc2202e73 + WORD $0x01c10e33 + WORD $0x010e0e13 + WORD $0x028e0d87 + WORD $0x939d9dd7 + WORD $0xc2202e73 + WORD $0x01c10e33 + WORD $0x010e0e13 + WORD $0x028e0da7 + WORD $0x01010e13 + WORD $0x028e0d87 + WORD $0x939d9dd7 + WORD $0x939e1e57 + WORD $0x939e9ed7 + WORD $0x939f1f57 + WORD $0x939f9fd7 + WORD $0x939393d7 + WORD $0x93931357 + WORD $0x939292d7 + WORD $0x93921257 + WORD $0x939191d7 + WORD $0x93911157 + WORD $0x939090d7 + WORD $0x93901057 + WORD $0x93941457 + WORD $0x939494d7 + WORD $0xc2202e73 + WORD $0x01c10e33 + WORD $0x010e0e13 + WORD $0x028e0c87 + WORD $0x0f9c1c57 + WORD $0x0e9d1d57 + WORD $0x0e851557 + WORD $0x0e0595d7 + WORD $0x0e161657 + WORD $0x0e2696d7 + WORD $0x0e371757 + WORD $0x0e4797d7 + WORD $0x0e581857 + WORD $0x0e6898d7 + WORD $0x0e791957 + WORD $0x0ff999d7 + WORD $0x0fea1a57 + WORD $0x0fda9ad7 + WORD $0x0fcb1b57 + WORD $0x0fbb9bd7 + WORD $0x40730333 + WORD $0x007888b3 + WORD $0xe8031ce3 + WORD $0x43a017d7 + WORD $0x42a01757 + WORD $0x42b016d7 + WORD $0x42c01657 + WORD $0x42d015d7 + WORD $0x42e01557 + WORD $0x42f01057 + WORD $0x430010d7 + WORD $0x43101157 + WORD $0x432011d7 + WORD $0x43301257 + WORD $0x434012d7 + WORD $0x43501357 + WORD $0x436013d7 + WORD $0x43701857 + WORD $0x438018d7 + WORD $0x01050833 + WORD $0x01070293 + WORD $0x01182027 + WORD $0x01082227 + WORD $0x00782427 + WORD $0x00682627 + WORD $0x00582827 + WORD $0x00482a27 + WORD $0x00382c27 + WORD $0x00282e27 + WORD $0x02182027 + WORD $0x02082227 + WORD $0x02a82427 + WORD $0x02b82627 + WORD $0x02c82827 + WORD $0x02d82a27 + WORD $0x02e82c27 + WORD $0x02f82e27 + WORD $0xda56dae3 + WORD $0x02c0006f + WORD $0xcd147057 + WORD $0x5e003457 + WORD $0x00050813 + WORD $0x02080893 + WORD $0x02086427 + WORD $0x0208e427 + WORD $0x02070893 + WORD $0x01070713 + WORD $0x04080813 + WORD $0xff16d4e3 + WORD $0xc2202873 + WORD $0x00181813 + WORD $0x01010133 + WORD $0x01010113 + WORD $0x08d75e63 + WORD $0x06f05463 WORD $0x0d007857 WORD $0x42006457 WORD $0x00000893 @@ -419,8 +829,11 @@ TEXT ·correlateFloat32RVV(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x00170713 WORD $0x00f82027 WORD $0xfad716e3 - WORD $0x0280006f + WORD $0x0340006f WORD $0x00000593 + WORD $0x40e686b3 + WORD $0x00271713 + WORD $0x00e50533 WORD $0x0d16f657 WORD $0x00259713 WORD $0x00e50733 @@ -684,20 +1097,105 @@ TEXT ·polyEvalFloat64RVV(SB), NOSPLIT|NOFRAME, $0-72 MOV coeffs_len+56(FP), X15 WORD $0x00e6c463 WORD $0x00070693 - WORD $0x06078663 - WORD $0x0ad05e63 + WORD $0x00078c63 + WORD $0x01000813 WORD $0x00379713 - WORD $0x00e60733 - WORD $0xff873787 - WORD $0x00200613 - WORD $0x08c7c063 - WORD $0x00000613 + WORD $0x0306dc63 + WORD $0x00000813 + WORD $0x14c0006f + WORD $0x1cd05e63 + WORD $0x00000593 + WORD $0x0d96f657 + WORD $0x00359713 + WORD $0x00e50733 + WORD $0x5e003457 + WORD $0x40c686b3 + WORD $0x02077427 + WORD $0x00c585b3 + WORD $0xfe0692e3 + WORD $0x1b40006f + WORD $0x00e602b3 + WORD $0xff82b787 + WORD $0x00100813 + WORD $0x0af85c63 + WORD $0x00000813 + WORD $0xcd927057 + WORD $0x5e07d457 + WORD $0xfff78893 + WORD $0xff028293 + WORD $0x01000393 + WORD $0x00381313 + WORD $0x00658833 + WORD $0x02087507 + WORD $0x02080e13 + WORD $0x020e7607 + WORD $0x04080e13 + WORD $0x06080813 + WORD $0x020e7807 + WORD $0x02087a07 + WORD $0x00038813 + WORD $0x00028393 + WORD $0x00088e13 + WORD $0x9e80b957 + WORD $0x9e80bc57 + WORD $0x9e80bb57 + WORD $0x9e80b757 + WORD $0x0003b787 + WORD $0x93251957 + WORD $0x93861c57 + WORD $0x93681b57 + WORD $0x92ea1757 + WORD $0xfffe0e13 + WORD $0x0327d957 + WORD $0x0387dc57 + WORD $0x0367db57 + WORD $0x02e7d757 + WORD $0xff838393 + WORD $0xfdc04ae3 + WORD $0x00650333 + WORD $0x02030393 + WORD $0x0203fc27 + WORD $0x04030393 + WORD $0x0203fb27 + WORD $0x01080393 + WORD $0x02037927 + WORD $0x06030313 + WORD $0x02037727 + WORD $0xf676d6e3 + WORD $0x05c0006f + WORD $0x00000813 + WORD $0x04050893 + WORD $0xfcf8b027 + WORD $0xfcf8b427 + WORD $0xfcf8b827 + WORD $0xfcf8bc27 + WORD $0xfef8b027 + WORD $0xfef8b427 + WORD $0xfef8b827 + WORD $0xfef8bc27 + WORD $0x00f8b027 + WORD $0x00f8b427 + WORD $0x00f8b827 + WORD $0x00f8bc27 + WORD $0x02f8b027 + WORD $0x02f8b427 + WORD $0x02f8b827 + WORD $0x02f8bc27 + WORD $0x02080293 + WORD $0x01080813 + WORD $0x08088893 + WORD $0xfa56dae3 + WORD $0x08d85a63 + WORD $0x00e60633 + WORD $0xff863787 + WORD $0x00200713 + WORD $0x04e7c863 WORD $0xfff78793 - WORD $0xff070713 - WORD $0x00361813 - WORD $0x010588b3 + WORD $0xff060613 + WORD $0x00381713 + WORD $0x00e588b3 WORD $0x0008b707 - WORD $0x00070893 + WORD $0x00060893 WORD $0x00078293 WORD $0x22f786d3 WORD $0x0008b607 @@ -706,32 +1204,24 @@ TEXT ·polyEvalFloat64RVV(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x02c6f6d3 WORD $0xff888893 WORD $0xfe5046e3 - WORD $0x01050833 - WORD $0x00160613 - WORD $0x00d83027 - WORD $0xfcd612e3 - WORD $0x0580006f - WORD $0x04d05a63 - WORD $0x00000593 - WORD $0x0d96f657 - WORD $0x00359713 WORD $0x00e50733 - WORD $0x5e003457 - WORD $0x40c686b3 - WORD $0x02077427 - WORD $0x00c585b3 - WORD $0xfe0692e3 - WORD $0x02c0006f + WORD $0x00180813 + WORD $0x00d73027 + WORD $0xfcd812e3 + WORD $0x0380006f WORD $0x00000593 - WORD $0x0d907657 + WORD $0x41068633 + WORD $0x00381813 + WORD $0x01050533 + WORD $0x0d9076d7 WORD $0x5e07d457 - WORD $0x0c66f657 + WORD $0x0c6676d7 WORD $0x00359713 WORD $0x00e50733 - WORD $0x40c686b3 + WORD $0x40d60633 WORD $0x02077427 - WORD $0x00c585b3 - WORD $0xfe0694e3 + WORD $0x00d585b3 + WORD $0xfe0614e3 WORD $0x00008067 RET @@ -743,14 +1233,196 @@ TEXT ·convolveFloat64RVV(SB), NOSPLIT|NOFRAME, $0-72 MOV dst_len+8(FP), X13 MOV sig_len+32(FP), X14 MOV ker_len+56(FP), X15 - WORD $0x0c078c63 - WORD $0x0cf74a63 + WORD $0x3c078a63 + WORD $0x3cf74863 WORD $0x40f70733 WORD $0x00d75463 WORD $0x00170693 - WORD $0x0cd05263 - WORD $0x08f05e63 + WORD $0xfe010113 + WORD $0x00813c23 + WORD $0x00913823 + WORD $0xc2202773 + WORD $0x00171713 + WORD $0x40e10133 + WORD $0x01000713 + WORD $0x00e6d663 + WORD $0x00000713 + WORD $0x2b80006f WORD $0x00000713 + WORD $0x26f05663 + WORD $0x00379893 + WORD $0x01000e13 + WORD $0x00100813 + WORD $0x011608b3 + WORD $0xff800293 + WORD $0x00000393 + WORD $0x00371313 + WORD $0x000e0713 + WORD $0x00658e33 + WORD $0x0d807ed7 + WORD $0x42006c57 + WORD $0x5e0c0cd7 + WORD $0x5e0c0dd7 + WORD $0x5e0c0d57 + WORD $0x5e0c0657 + WORD $0x5e0c06d7 + WORD $0x5e0c0757 + WORD $0x5e0c07d7 + WORD $0x5e0c0857 + WORD $0x5e0c08d7 + WORD $0x5e0c0957 + WORD $0x5e0c09d7 + WORD $0x5e0c0a57 + WORD $0x5e0c0ad7 + WORD $0x5e0c0b57 + WORD $0x5e0c0bd7 + WORD $0x00078e93 + WORD $0x0d8eff57 + WORD $0xfff3cf93 + WORD $0x00339413 + WORD $0x003f9f93 + WORD $0x41e804b3 + WORD $0x008e0433 + WORD $0x01f88fb3 + WORD $0x00349493 + WORD $0x009f8fb3 + WORD $0x00840493 + WORD $0x0204f507 + WORD $0x01040493 + WORD $0x0204f587 + WORD $0x01840493 + WORD $0x0204f407 + WORD $0x01010493 + WORD $0x02848427 + WORD $0x02040493 + WORD $0x0204fe07 + WORD $0x02840493 + WORD $0x0204fe87 + WORD $0x03040493 + WORD $0x0204ff07 + WORD $0x03840493 + WORD $0x0204ff87 + WORD $0x04040493 + WORD $0x0204f387 + WORD $0x04840493 + WORD $0x0204f307 + WORD $0x05040493 + WORD $0x0204f287 + WORD $0x05840493 + WORD $0x0204f207 + WORD $0x06040493 + WORD $0x0204f187 + WORD $0x06840493 + WORD $0x0204f107 + WORD $0x003f1493 + WORD $0x01f48fb3 + WORD $0x02047087 + WORD $0xff8f8f93 + WORD $0x0a5ff007 + WORD $0x07040f93 + WORD $0x07840413 + WORD $0x020ff407 + WORD $0x02047487 + WORD $0x920090d7 + WORD $0xc2202ff3 + WORD $0x01f10fb3 + WORD $0x010f8f93 + WORD $0x028f80a7 + WORD $0x920510d7 + WORD $0x92059557 + WORD $0x01010f93 + WORD $0x028f8587 + WORD $0x920595d7 + WORD $0x920e1e57 + WORD $0x920e9ed7 + WORD $0x920f1f57 + WORD $0x920f9fd7 + WORD $0x920393d7 + WORD $0x92031357 + WORD $0x920292d7 + WORD $0x92021257 + WORD $0x920191d7 + WORD $0x92011157 + WORD $0x92041457 + WORD $0x920494d7 + WORD $0x0e9c9cd7 + WORD $0x0e8d9dd7 + WORD $0x0e2d1d57 + WORD $0x0e361657 + WORD $0x0e4696d7 + WORD $0x0e571757 + WORD $0x0e6797d7 + WORD $0x0e781857 + WORD $0x0ff898d7 + WORD $0x0fe91957 + WORD $0x0fd999d7 + WORD $0x0fca1a57 + WORD $0x0eba9ad7 + WORD $0x0eab1b57 + WORD $0x0e1b9bd7 + WORD $0xc2202ff3 + WORD $0x01f10fb3 + WORD $0x010f8f93 + WORD $0x028f8407 + WORD $0x0e8c1c57 + WORD $0x41ee8eb3 + WORD $0x01e383b3 + WORD $0xe80e9ce3 + WORD $0x439017d7 + WORD $0x43b01757 + WORD $0x43a016d7 + WORD $0x42c01657 + WORD $0x42d015d7 + WORD $0x42e01557 + WORD $0x42f01057 + WORD $0x430010d7 + WORD $0x43101157 + WORD $0x432011d7 + WORD $0x43301257 + WORD $0x434012d7 + WORD $0x43501357 + WORD $0x436013d7 + WORD $0x43701857 + WORD $0x438018d7 + WORD $0x00650333 + WORD $0x01070e13 + WORD $0x01133027 + WORD $0x01033427 + WORD $0x00733827 + WORD $0x00633c27 + WORD $0x02533027 + WORD $0x02433427 + WORD $0x02333827 + WORD $0x02233c27 + WORD $0x04133027 + WORD $0x04033427 + WORD $0x04a33827 + WORD $0x04b33c27 + WORD $0x06c33027 + WORD $0x06d33427 + WORD $0x06e33827 + WORD $0x06f33c27 + WORD $0xdbc6dae3 + WORD $0x0480006f + WORD $0xcd927057 + WORD $0x5e003657 + WORD $0xcda47057 + WORD $0x5e003457 + WORD $0x00050813 + WORD $0x02080893 + WORD $0xcd927057 + WORD $0x02087627 + WORD $0xcda47057 + WORD $0x0208f427 + WORD $0x06080893 + WORD $0xcd927057 + WORD $0x0208f627 + WORD $0x02070893 + WORD $0x01070713 + WORD $0x08080813 + WORD $0xfd16dae3 + WORD $0x0cd75663 + WORD $0x08f05c63 WORD $0x00379893 WORD $0x0d807857 WORD $0x42006457 @@ -787,8 +1459,11 @@ TEXT ·convolveFloat64RVV(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x00170713 WORD $0x00f2b027 WORD $0xf8d716e3 - WORD $0x0280006f + WORD $0x0340006f WORD $0x00000593 + WORD $0x40e686b3 + WORD $0x00371713 + WORD $0x00e50533 WORD $0x0d96f657 WORD $0x00359713 WORD $0x00e50733 @@ -797,6 +1472,12 @@ TEXT ·convolveFloat64RVV(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x02077427 WORD $0x00c585b3 WORD $0xfe0692e3 + WORD $0xc2202573 + WORD $0x00151513 + WORD $0x00a10133 + WORD $0x01813403 + WORD $0x01013483 + WORD $0x02010113 WORD $0x00008067 RET @@ -808,14 +1489,195 @@ TEXT ·correlateFloat64RVV(SB), NOSPLIT|NOFRAME, $0-72 MOV dst_len+8(FP), X13 MOV sig_len+32(FP), X14 MOV ker_len+56(FP), X15 - WORD $0x0a078463 - WORD $0x0af74263 + WORD $0x38078463 + WORD $0x38f74263 WORD $0x40f70733 WORD $0x00d75463 WORD $0x00170693 - WORD $0x08d05a63 - WORD $0x06f05663 + WORD $0x01000713 + WORD $0x00e6d863 WORD $0x00000713 + WORD $0x2cd04863 + WORD $0x3640006f + WORD $0xff010113 + WORD $0xc2202773 + WORD $0x00171713 + WORD $0x40e10133 + WORD $0x00000713 + WORD $0x24f05e63 + WORD $0x01000293 + WORD $0x00000893 + WORD $0x00371813 + WORD $0x00028713 + WORD $0x010582b3 + WORD $0x0d807357 + WORD $0x42006c57 + WORD $0x5e0c0d57 + WORD $0x5e0c0557 + WORD $0x5e0c05d7 + WORD $0x5e0c0657 + WORD $0x5e0c06d7 + WORD $0x5e0c0757 + WORD $0x5e0c07d7 + WORD $0x5e0c0857 + WORD $0x5e0c08d7 + WORD $0x5e0c0957 + WORD $0x5e0c09d7 + WORD $0x5e0c0a57 + WORD $0x5e0c0ad7 + WORD $0x5e0c0b57 + WORD $0x5e0c0bd7 + WORD $0x00078313 + WORD $0x0d8373d7 + WORD $0x00389e13 + WORD $0x01c60eb3 + WORD $0x01c28e33 + WORD $0x020efc87 + WORD $0x020e7407 + WORD $0xc2202ef3 + WORD $0x01d10eb3 + WORD $0x010e8e93 + WORD $0x028e8427 + WORD $0x008e0e93 + WORD $0x020ef407 + WORD $0x01010e93 + WORD $0x028e8427 + WORD $0x010e0e93 + WORD $0x020efe07 + WORD $0x018e0e93 + WORD $0x020efe87 + WORD $0x020e0e93 + WORD $0x020eff07 + WORD $0x028e0e93 + WORD $0x020eff87 + WORD $0x030e0e93 + WORD $0x020ef387 + WORD $0x038e0e93 + WORD $0x020ef307 + WORD $0x040e0e93 + WORD $0x020ef287 + WORD $0x048e0e93 + WORD $0x020ef207 + WORD $0x050e0e93 + WORD $0x020ef187 + WORD $0x058e0e93 + WORD $0x020ef107 + WORD $0x060e0e93 + WORD $0x020ef087 + WORD $0x068e0e93 + WORD $0x020ef007 + WORD $0x070e0e93 + WORD $0x078e0e13 + WORD $0x020ef407 + WORD $0x020e7487 + WORD $0xc2202e73 + WORD $0x01c10e33 + WORD $0x010e0e13 + WORD $0x028e0d87 + WORD $0x939d9dd7 + WORD $0xc2202e73 + WORD $0x01c10e33 + WORD $0x010e0e13 + WORD $0x028e0da7 + WORD $0x01010e13 + WORD $0x028e0d87 + WORD $0x939d9dd7 + WORD $0x939e1e57 + WORD $0x939e9ed7 + WORD $0x939f1f57 + WORD $0x939f9fd7 + WORD $0x939393d7 + WORD $0x93931357 + WORD $0x939292d7 + WORD $0x93921257 + WORD $0x939191d7 + WORD $0x93911157 + WORD $0x939090d7 + WORD $0x93901057 + WORD $0x93941457 + WORD $0x939494d7 + WORD $0xc2202e73 + WORD $0x01c10e33 + WORD $0x010e0e13 + WORD $0x028e0c87 + WORD $0x0f9c1c57 + WORD $0x0e9d1d57 + WORD $0x0e851557 + WORD $0x0e0595d7 + WORD $0x0e161657 + WORD $0x0e2696d7 + WORD $0x0e371757 + WORD $0x0e4797d7 + WORD $0x0e581857 + WORD $0x0e6898d7 + WORD $0x0e791957 + WORD $0x0ff999d7 + WORD $0x0fea1a57 + WORD $0x0fda9ad7 + WORD $0x0fcb1b57 + WORD $0x0fbb9bd7 + WORD $0x40730333 + WORD $0x007888b3 + WORD $0xe8031ce3 + WORD $0x43a017d7 + WORD $0x42a01757 + WORD $0x42b016d7 + WORD $0x42c01657 + WORD $0x42d015d7 + WORD $0x42e01557 + WORD $0x42f01057 + WORD $0x430010d7 + WORD $0x43101157 + WORD $0x432011d7 + WORD $0x43301257 + WORD $0x434012d7 + WORD $0x43501357 + WORD $0x436013d7 + WORD $0x43701857 + WORD $0x438018d7 + WORD $0x01050833 + WORD $0x01070293 + WORD $0x01183027 + WORD $0x01083427 + WORD $0x00783827 + WORD $0x00683c27 + WORD $0x02583027 + WORD $0x02483427 + WORD $0x02383827 + WORD $0x02283c27 + WORD $0x04183027 + WORD $0x04083427 + WORD $0x04a83827 + WORD $0x04b83c27 + WORD $0x06c83027 + WORD $0x06d83427 + WORD $0x06e83827 + WORD $0x06f83c27 + WORD $0xda56dae3 + WORD $0x0480006f + WORD $0xcd927057 + WORD $0x5e003657 + WORD $0xcda47057 + WORD $0x5e003457 + WORD $0x00050813 + WORD $0x02080893 + WORD $0xcd927057 + WORD $0x02087627 + WORD $0xcda47057 + WORD $0x0208f427 + WORD $0x06080893 + WORD $0xcd927057 + WORD $0x0208f627 + WORD $0x02070893 + WORD $0x01070713 + WORD $0x08080813 + WORD $0xfd16dae3 + WORD $0xc2202873 + WORD $0x00181813 + WORD $0x01010133 + WORD $0x01010113 + WORD $0x08d75e63 + WORD $0x06f05463 WORD $0x0d807857 WORD $0x42006457 WORD $0x00000893 @@ -840,8 +1702,11 @@ TEXT ·correlateFloat64RVV(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x00170713 WORD $0x00f83027 WORD $0xfad716e3 - WORD $0x0280006f + WORD $0x0340006f WORD $0x00000593 + WORD $0x40e686b3 + WORD $0x00371713 + WORD $0x00e50533 WORD $0x0d96f657 WORD $0x00359713 WORD $0x00e50733 @@ -2067,14 +2932,261 @@ TEXT ·movingAverageFloat32RVV(SB), NOSPLIT|NOFRAME, $0-56 MOV dst_len+8(FP), X12 MOV a_len+32(FP), X13 MOV width+48(FP), X14 - WORD $0x06e05c63 - WORD $0x06e6ca63 + WORD $0x44e05a63 + WORD $0x44e6c863 WORD $0x40e686b3 WORD $0x00c6d463 WORD $0x00168613 - WORD $0x06c05263 + WORD $0x01000893 + WORD $0x01165863 + WORD $0x00000693 + WORD $0x3cc04c63 + WORD $0x4300006f + WORD $0xff010113 + WORD $0xc22026f3 + WORD $0x00c00793 + WORD $0x02f686b3 + WORD $0x40d10133 WORD $0x00000693 WORD $0xd03777d3 + WORD $0x00000813 + WORD $0x00269793 + WORD $0x00088693 + WORD $0x00f588b3 + WORD $0x0d0072d7 + WORD $0x42006457 + WORD $0x9e80b557 + WORD $0xc22022f3 + WORD $0x00a00313 + WORD $0x026282b3 + WORD $0x005102b3 + WORD $0x01028293 + WORD $0x22828427 + WORD $0xc22022f3 + WORD $0x00329293 + WORD $0x005102b3 + WORD $0x01028293 + WORD $0x22828427 + WORD $0x9e80b857 + WORD $0x9e80b957 + WORD $0x9e80ba57 + WORD $0x9e80bb57 + WORD $0x9e80bc57 + WORD $0x9e80bd57 + WORD $0x9e80be57 + WORD $0x9e80bf57 + WORD $0x9e80b357 + WORD $0x9e80b257 + WORD $0x9e80b157 + WORD $0x9e80b057 + WORD $0x9e80b657 + WORD $0x00070293 + WORD $0x0d12f357 + WORD $0x00281393 + WORD $0x007883b3 + WORD $0x0203e707 + WORD $0x00838e13 + WORD $0x020e6407 + WORD $0xc2202e73 + WORD $0x001e1e13 + WORD $0x01c10e33 + WORD $0x010e0e13 + WORD $0x228e0427 + WORD $0x01038e13 + WORD $0x020e6407 + WORD $0x01010e13 + WORD $0x228e0427 + WORD $0x03c38e13 + WORD $0x020e6407 + WORD $0xc2202e73 + WORD $0x00600e93 + WORD $0x03de0e33 + WORD $0x01c10e33 + WORD $0x010e0e13 + WORD $0x228e0427 + WORD $0x03838e13 + WORD $0x0ee61657 + WORD $0xc2202ef3 + WORD $0x002e9e93 + WORD $0x01d10eb3 + WORD $0x010e8e93 + WORD $0x228e8627 + WORD $0x020e6407 + WORD $0x03438e13 + WORD $0xc2202ef3 + WORD $0x00600f13 + WORD $0x03ee8eb3 + WORD $0x01d10eb3 + WORD $0x010e8e93 + WORD $0x228e8607 + WORD $0x0ec51557 + WORD $0xc2202ef3 + WORD $0x00600f13 + WORD $0x03ee8eb3 + WORD $0x01d10eb3 + WORD $0x010e8e93 + WORD $0x228e8527 + WORD $0x020e6507 + WORD $0x03038e13 + WORD $0x9e003657 + WORD $0x9e203057 + WORD $0x9e403157 + WORD $0x9e603257 + WORD $0x9fe03357 + WORD $0x9fc03f57 + WORD $0x9fa03e57 + WORD $0x9f803d57 + WORD $0x9f603c57 + WORD $0xc2202ef3 + WORD $0x00a00f13 + WORD $0x03ee8eb3 + WORD $0x01d10eb3 + WORD $0x010e8e93 + WORD $0x228e8707 + WORD $0x0e871757 + WORD $0xc2202ef3 + WORD $0x00a00f13 + WORD $0x03ee8eb3 + WORD $0x01d10eb3 + WORD $0x010e8e93 + WORD $0x228e8727 + WORD $0xc2202ef3 + WORD $0x003e9e93 + WORD $0x01d10eb3 + WORD $0x010e8e93 + WORD $0x228e8707 + WORD $0x9f803b57 + WORD $0x9fa03c57 + WORD $0x9fc03d57 + WORD $0x9fe03e57 + WORD $0x9e603f57 + WORD $0x9e403357 + WORD $0x9e203257 + WORD $0x9e003157 + WORD $0x9ec03057 + WORD $0xc2202ef3 + WORD $0x002e9e93 + WORD $0x01d10eb3 + WORD $0x010e8e93 + WORD $0x228e8607 + WORD $0x020e6407 + WORD $0x02c38e13 + WORD $0x0ea71757 + WORD $0xc2202ef3 + WORD $0x003e9e93 + WORD $0x01d10eb3 + WORD $0x010e8e93 + WORD $0x228e8727 + WORD $0x020e6507 + WORD $0x02838e13 + WORD $0x0e881857 + WORD $0x020e6407 + WORD $0x02438e13 + WORD $0x0ea91957 + WORD $0x020e6507 + WORD $0x02038e13 + WORD $0x0e8a1a57 + WORD $0x020e6407 + WORD $0x01c38e13 + WORD $0x0eab1b57 + WORD $0x020e6507 + WORD $0x01838e13 + WORD $0x0e8c1c57 + WORD $0x020e6407 + WORD $0x01438e13 + WORD $0x0ead1d57 + WORD $0x020e6507 + WORD $0x00438e13 + WORD $0x00c38393 + WORD $0x0e8e1e57 + WORD $0x0203e407 + WORD $0x0eaf1f57 + WORD $0x020e6507 + WORD $0x01010393 + WORD $0x22838707 + WORD $0x0ee31357 + WORD $0x0e821257 + WORD $0xc22023f3 + WORD $0x00139393 + WORD $0x007103b3 + WORD $0x01038393 + WORD $0x22838407 + WORD $0x0e811157 + WORD $0x0ea01057 + WORD $0xc22023f3 + WORD $0x00600e13 + WORD $0x03c383b3 + WORD $0x007103b3 + WORD $0x01038393 + WORD $0x22838507 + WORD $0x406282b3 + WORD $0x00680833 + WORD $0xdc0298e3 + WORD $0x42a01757 + WORD $0xc2202873 + WORD $0x00a00893 + WORD $0x03180833 + WORD $0x01010833 + WORD $0x01080813 + WORD $0x22880407 + WORD $0x428016d7 + WORD $0xc2202873 + WORD $0x00381813 + WORD $0x01010833 + WORD $0x01080813 + WORD $0x22880407 + WORD $0x42801657 + WORD $0x430015d7 + WORD $0x43201557 + WORD $0x43401057 + WORD $0x436010d7 + WORD $0x43801157 + WORD $0x43a011d7 + WORD $0x43c01257 + WORD $0x43e012d7 + WORD $0x42601357 + WORD $0x424013d7 + WORD $0x42201857 + WORD $0x420018d7 + WORD $0x42c01e57 + WORD $0xcd027057 + WORD $0x5e0e5457 + WORD $0x5e0354d7 + WORD $0x5e015557 + WORD $0x5e05d5d7 + WORD $0x3e88d457 + WORD $0x3e92d4d7 + WORD $0x3ea0d557 + WORD $0x3eb655d7 + WORD $0x3e885457 + WORD $0x3e9254d7 + WORD $0x3ea05557 + WORD $0x3eb6d5d7 + WORD $0x3e91d4d7 + WORD $0x8297d4d7 + WORD $0x3ea55557 + WORD $0x82a7d557 + WORD $0x3e83d457 + WORD $0x8287d457 + WORD $0x3eb755d7 + WORD $0x82b7d5d7 + WORD $0x00f507b3 + WORD $0x01078813 + WORD $0x020864a7 + WORD $0x02078813 + WORD $0x02086527 + WORD $0x01068893 + WORD $0x0207e427 + WORD $0x03078793 + WORD $0x0207e5a7 + WORD $0xc71654e3 + WORD $0xc22027f3 + WORD $0x00c00813 + WORD $0x030787b3 + WORD $0x00f10133 + WORD $0x01010113 + WORD $0x06c6d063 + WORD $0xd03777d3 WORD $0x0d0077d7 WORD $0x42006457 WORD $0x00000813 @@ -2161,14 +3273,265 @@ TEXT ·movingAverageFloat64RVV(SB), NOSPLIT|NOFRAME, $0-56 MOV dst_len+8(FP), X12 MOV a_len+32(FP), X13 MOV width+48(FP), X14 - WORD $0x06e05c63 - WORD $0x06e6ca63 + WORD $0x46e05263 + WORD $0x46e6c063 WORD $0x40e686b3 WORD $0x00c6d463 WORD $0x00168613 - WORD $0x06c05263 + WORD $0x01000893 + WORD $0x01165863 + WORD $0x00000693 + WORD $0x3ec04463 + WORD $0x4400006f + WORD $0xff010113 + WORD $0xc22026f3 + WORD $0x00c00793 + WORD $0x02f686b3 + WORD $0x40d10133 WORD $0x00000693 WORD $0xd23777d3 + WORD $0x00000813 + WORD $0x00369793 + WORD $0x00088693 + WORD $0x00f588b3 + WORD $0x0d8072d7 + WORD $0x42006457 + WORD $0x9e80b557 + WORD $0xc22022f3 + WORD $0x00a00313 + WORD $0x026282b3 + WORD $0x005102b3 + WORD $0x01028293 + WORD $0x22828427 + WORD $0xc22022f3 + WORD $0x00329293 + WORD $0x005102b3 + WORD $0x01028293 + WORD $0x22828427 + WORD $0x9e80b857 + WORD $0x9e80b957 + WORD $0x9e80ba57 + WORD $0x9e80bb57 + WORD $0x9e80bc57 + WORD $0x9e80bd57 + WORD $0x9e80be57 + WORD $0x9e80bf57 + WORD $0x9e80b357 + WORD $0x9e80b257 + WORD $0x9e80b157 + WORD $0x9e80b057 + WORD $0x9e80b657 + WORD $0x00070293 + WORD $0x0d92f357 + WORD $0x00381393 + WORD $0x007883b3 + WORD $0x0203f707 + WORD $0x01038e13 + WORD $0x020e7407 + WORD $0xc2202e73 + WORD $0x001e1e13 + WORD $0x01c10e33 + WORD $0x010e0e13 + WORD $0x228e0427 + WORD $0x02038e13 + WORD $0x020e7407 + WORD $0x01010e13 + WORD $0x228e0427 + WORD $0x07838e13 + WORD $0x020e7407 + WORD $0xc2202e73 + WORD $0x00600e93 + WORD $0x03de0e33 + WORD $0x01c10e33 + WORD $0x010e0e13 + WORD $0x228e0427 + WORD $0x07038e13 + WORD $0x0ee61657 + WORD $0xc2202ef3 + WORD $0x002e9e93 + WORD $0x01d10eb3 + WORD $0x010e8e93 + WORD $0x228e8627 + WORD $0x020e7407 + WORD $0x06838e13 + WORD $0xc2202ef3 + WORD $0x00600f13 + WORD $0x03ee8eb3 + WORD $0x01d10eb3 + WORD $0x010e8e93 + WORD $0x228e8607 + WORD $0x0ec51557 + WORD $0xc2202ef3 + WORD $0x00600f13 + WORD $0x03ee8eb3 + WORD $0x01d10eb3 + WORD $0x010e8e93 + WORD $0x228e8527 + WORD $0x020e7507 + WORD $0x06038e13 + WORD $0x9e003657 + WORD $0x9e203057 + WORD $0x9e403157 + WORD $0x9e603257 + WORD $0x9fe03357 + WORD $0x9fc03f57 + WORD $0x9fa03e57 + WORD $0x9f803d57 + WORD $0x9f603c57 + WORD $0xc2202ef3 + WORD $0x00a00f13 + WORD $0x03ee8eb3 + WORD $0x01d10eb3 + WORD $0x010e8e93 + WORD $0x228e8707 + WORD $0x0e871757 + WORD $0xc2202ef3 + WORD $0x00a00f13 + WORD $0x03ee8eb3 + WORD $0x01d10eb3 + WORD $0x010e8e93 + WORD $0x228e8727 + WORD $0xc2202ef3 + WORD $0x003e9e93 + WORD $0x01d10eb3 + WORD $0x010e8e93 + WORD $0x228e8707 + WORD $0x9f803b57 + WORD $0x9fa03c57 + WORD $0x9fc03d57 + WORD $0x9fe03e57 + WORD $0x9e603f57 + WORD $0x9e403357 + WORD $0x9e203257 + WORD $0x9e003157 + WORD $0x9ec03057 + WORD $0xc2202ef3 + WORD $0x002e9e93 + WORD $0x01d10eb3 + WORD $0x010e8e93 + WORD $0x228e8607 + WORD $0x020e7407 + WORD $0x05838e13 + WORD $0x0ea71757 + WORD $0xc2202ef3 + WORD $0x003e9e93 + WORD $0x01d10eb3 + WORD $0x010e8e93 + WORD $0x228e8727 + WORD $0x020e7507 + WORD $0x05038e13 + WORD $0x0e881857 + WORD $0x020e7407 + WORD $0x04838e13 + WORD $0x0ea91957 + WORD $0x020e7507 + WORD $0x04038e13 + WORD $0x0e8a1a57 + WORD $0x020e7407 + WORD $0x03838e13 + WORD $0x0eab1b57 + WORD $0x020e7507 + WORD $0x03038e13 + WORD $0x0e8c1c57 + WORD $0x020e7407 + WORD $0x02838e13 + WORD $0x0ead1d57 + WORD $0x020e7507 + WORD $0x00838e13 + WORD $0x01838393 + WORD $0x0e8e1e57 + WORD $0x0203f407 + WORD $0x0eaf1f57 + WORD $0x020e7507 + WORD $0x01010393 + WORD $0x22838707 + WORD $0x0ee31357 + WORD $0x0e821257 + WORD $0xc22023f3 + WORD $0x00139393 + WORD $0x007103b3 + WORD $0x01038393 + WORD $0x22838407 + WORD $0x0e811157 + WORD $0x0ea01057 + WORD $0xc22023f3 + WORD $0x00600e13 + WORD $0x03c383b3 + WORD $0x007103b3 + WORD $0x01038393 + WORD $0x22838507 + WORD $0x406282b3 + WORD $0x00680833 + WORD $0xdc0298e3 + WORD $0x42a01757 + WORD $0xc2202873 + WORD $0x00a00893 + WORD $0x03180833 + WORD $0x01010833 + WORD $0x01080813 + WORD $0x22880407 + WORD $0x428016d7 + WORD $0xc2202873 + WORD $0x00381813 + WORD $0x01010833 + WORD $0x01080813 + WORD $0x22880407 + WORD $0x42801657 + WORD $0x430015d7 + WORD $0x43201557 + WORD $0x43401057 + WORD $0x436010d7 + WORD $0x43801157 + WORD $0x43a011d7 + WORD $0x43c01257 + WORD $0x43e012d7 + WORD $0x42601357 + WORD $0x424013d7 + WORD $0x42201857 + WORD $0x420018d7 + WORD $0x42c01e57 + WORD $0x00f507b3 + WORD $0x01068893 + WORD $0x1afe7e53 + WORD $0x1af8f8d3 + WORD $0x1af87853 + WORD $0x1af3f3d3 + WORD $0x1af37353 + WORD $0x1af2f2d3 + WORD $0x1af27253 + WORD $0x1af1f1d3 + WORD $0x1af17153 + WORD $0x1af0f0d3 + WORD $0x1af07053 + WORD $0x1af57553 + WORD $0x1af5f5d3 + WORD $0x1af67653 + WORD $0x1af6f6d3 + WORD $0x1af77753 + WORD $0x01c7b027 + WORD $0x0117b427 + WORD $0x0107b827 + WORD $0x0077bc27 + WORD $0x0267b027 + WORD $0x0257b427 + WORD $0x0247b827 + WORD $0x0237bc27 + WORD $0x0427b027 + WORD $0x0417b427 + WORD $0x0407b827 + WORD $0x04a7bc27 + WORD $0x06b7b027 + WORD $0x06c7b427 + WORD $0x06d7b827 + WORD $0x06e7bc27 + WORD $0xc5165ce3 + WORD $0xc22027f3 + WORD $0x00c00813 + WORD $0x030787b3 + WORD $0x00f10133 + WORD $0x01010113 + WORD $0x06c6d063 + WORD $0xd23777d3 WORD $0x0d8077d7 WORD $0x42006457 WORD $0x00000813 From 468eddd60e229bdf26654b4af4cc298abcb20c6b Mon Sep 17 00:00:00 2001 From: Sebastian Mocanu Date: Sun, 16 Aug 2026 11:42:04 +0100 Subject: [PATCH 11/13] docs: fix the count that reddened the suite, correct entry 77's table `go test ./...` was red at e2ab049: entry 77 was added to docs/wrong.md without bumping README's "records 85 things", and that one line takes `make verify`, `make test-tiers` (aborting at the first tier) and `make test-cross` down with it. Entry 77's numbers were wrong in three places, all from generalising one tier's output: - the avx2 packed-arithmetic column was wrong in all six rows it listed. Re-derived twice, once against `git show c1d08c4:csrc/numeric.c` and once against the blocked file, so the before column is measured too. The table now gives all eight symbols at sse2/avx2/avx512. - "the ten scalar instructions that remain are the tail loop" -- the scalar count is 10 at sse2 and 18 at avx2/avx512, and it is identical before and after the change on every amd64 tier. Blocking added packed work beside those instructions; it did not convert them. - "eleven of eighteen length pairs are at or above the guard's threshold" -- it is eight. Added what the record was missing: a review sweep of 3,286,542 cases per tier across all nine tiers (cancellation at 1e16, mid-accumulation overflow, block-boundary magnitude switches, NaN payloads) finding zero differing bits, with a poisoned `ref` proving it non-vacuous; and the arm64 cost, where LLVM vectorizes over the taps instead of the block and six kernels go from a zero frame to 80-320 bytes with 7-13x the scalar float arithmetic -- bit-identical, inside the frame budget, and measured for speed nowhere. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 14 ++++++++++ README.md | 2 +- docs/wrong.md | 76 +++++++++++++++++++++++++++++++++++++++++++-------- 3 files changed, 80 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7aeefdb..75eed4b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,20 @@ ## Unreleased +**`PolyEval`, `Convolve`, `Correlate` and `MovingAverage` now vectorize.** +Their kernels accumulate per output, so the innermost loop carried a +dependence LLVM cannot break without reassociating -- which the numerical +contract forbids -- and all eight symbols emitted zero packed arithmetic +instructions on every amd64 tier. Blocking the *outer* loop by 16 makes the +innermost loop independent while each output still evaluates the same terms +in the same order, so the results are bit-identical rather than close: 8-40 +packed instructions per symbol now, and loong64 gains `simd_movavg_f32/f64`, +6,931 -> 6,933 kernels. Verified over 3.29M cases per tier on all nine tiers +including catastrophic cancellation and mid-accumulation overflow, where any +regrouping would show. Speed is not measured, and on arm64 the vectorizer +takes a different shape that costs stack frames -- `docs/wrong.md` entry 77 +records both. + **Complex operations now run the generated assembly.** They never did: `complexOps` returned the portable reference set directly, and the generator emitted no complex entries into the per-tier dispatch tables, so diff --git a/README.md b/README.md index a59504b..6ba3ffa 100644 --- a/README.md +++ b/README.md @@ -389,7 +389,7 @@ cgo. ## Engineering record -[**docs/wrong.md**](docs/wrong.md) records 85 things that measurement disproved, +[**docs/wrong.md**](docs/wrong.md) records 86 things that measurement disproved, including changes that were deleted rather than shipped. Examples: - Green test lanes had executed no accelerated code for months. diff --git a/docs/wrong.md b/docs/wrong.md index 1e19086..36acbb0 100644 --- a/docs/wrong.md +++ b/docs/wrong.md @@ -3536,16 +3536,31 @@ is exactly what `#pragma clang loop vectorize(enable)` would have done and what the numerical contract forbids. Nothing is allocated — 16 doubles is 128 bytes against the 512-byte kernel frame budget — and the tail runs the original nest. -| | packed arithmetic before | after (sse2) | after (avx2) | -|---|---|---|---| -| `simd_polyeval_f32` | 0 | 24 | 12 | -| `simd_polyeval_f64` | 0 | 16 | 24 | -| `simd_convolve_f32` | 0 | 8 | 12 | -| `simd_correlate_f64` | 0 | 16 | 8 | -| `simd_movavg_f32` | 0 | 16 | 12 | -| `simd_movavg_f64` | 0 | 16 | 16 | - -The ten scalar instructions that remain in each are the tail loop. +Packed arithmetic instructions per symbol, from `go run ./simdgen -n -v -only +csrc/numeric.c` in `tools/`, run twice: once against `git show +c1d08c4:csrc/numeric.c` and once against the blocked file. + +| | before, every amd64 tier | sse2 | avx2 | avx512 | +|---|---|---|---|---| +| `simd_polyeval_f32` | 0 | 24 | 20 | 18 | +| `simd_polyeval_f64` | 0 | 16 | 40 | 20 | +| `simd_convolve_f32` | 0 | 8 | 20 | 12 | +| `simd_convolve_f64` | 0 | 16 | 24 | 21 | +| `simd_correlate_f32` | 0 | 8 | 20 | 20 | +| `simd_correlate_f64` | 0 | 16 | 24 | 21 | +| `simd_movavg_f32` | 0 | 16 | 20 | 10 | +| `simd_movavg_f64` | 0 | 16 | 24 | 20 | + +The scalar count does **not** move: 10 at sse2 and 18 at avx2/avx512, identical +before and after on all eight symbols. So those instructions are not a tail the +blocking introduced — they were already there, and blocking added packed work +beside them rather than converting scalar work into it. `movavg` at avx512 f32 +gets only 10 packed against 20 at avx2 because a 16-element f32 block is exactly +one zmm. + +A first version of this entry gave an avx2 column that was wrong in all six rows +it listed and called the scalar remainder a tail. Both came from reading one +tier's numbers and generalising. `SCALAR-ONLY` went from **13 kernels to 5** (`dtoa_f64`, `minr_f32/f64`, `rolling_min_f64`, `rolling_max_f64`). And loong64 gained two kernels, 728 → @@ -3554,7 +3569,8 @@ not vectorize it for lasx" and now emit, so the total moved 6,931 → 6,933. **The bit-identity claim is tested, and the test was checked for being vacuous.** `internal/conformance` compares each tier against `ref` at eighteen -length pairs, eleven of which are at or above the guard's threshold of 16 — so +length pairs (`lenPairs`, `internal/conformance/numeric_test.go:23`), eight of +which have a destination at or above the guard's threshold of 16 — so the SIMD path is genuinely reached. Poisoning the blocked path only (`acc[b] = acc[b]*x[i+b] + ck + 1`) reddens it at `dst=16 x=16 coeffs=2 i=0`, the first blocked case, which is what says the comparison sees the new code @@ -3567,6 +3583,44 @@ is offered. What is established is that the kernels now do their arithmetic in lanes and produce the same bits, which is the precondition for the speed question rather than an answer to it. +**A review sweep put the bit-identity claim far past what those eighteen pairs +reach.** Every `nd` in 0..200 crossed with 22 other lengths and with `nd±1` and +`nd+15/16/17`; `nc`/`nk`/`w` in `{-1,0,1,2,3,4,15,16,17,31,33}`; seven value +generators per precision including catastrophic cancellation (`1e16, 1, -1e16, +1`), mid-accumulation overflow (`MaxFloat, MaxFloat, -MaxFloat`, where +left-to-right gives +Inf and any other grouping gives a finite number), +magnitudes switching at every 16-element block boundary, ±0 and subnormals, and +NaN payloads. **3,286,542 cases per tier, 3,027,710 of them reaching the kernel +rather than `ref`, on all nine tiers — sse2/avx2/avx512 native and +neon/sve2/rvv/lasx/vsx/vx under qemu-user. Zero differing bits.** Perturbing +`ref` by +1 only at output index ≥ 16 reddens every comparison site +(197k–291k mismatches per operation), so the sweep is not vacuous. A separate +`PROT_NONE` guard-page probe placing dst, signal and kernel each so its last +element ends on the last accessible byte — ~88k kernel calls per tier — faulted +nowhere, and the mechanism was checked by reading one element past the page. + +**arm64 pays for this, and the cost is not measured.** LLVM did not vectorize +the block loop there; it unrolled over `b` and vectorized over `j` instead, so +each accumulator gets a packed multiply of 4 taps and then an in-order scalar +horizontal reduction. That needs 16 live scalar accumulators plus vector +temporaries, and the tap loop of `simd_correlate_f32` at neon carries 18 spill +stores and 18 reloads per iteration: + +| tier | kernel | stack | scalar float arith | +|---|---|---|---| +| neon | `simd_correlate_f32` | 0B → **320B** | 10 → **82** | +| neon | `simd_convolve_f32` | 0B → **272B** | 10 → **82** | +| neon | `simd_convolve_f64` | 0B → 80B | 6 → **70** | +| neon | `simd_correlate_f64` | 0B → 80B | 6 → **70** | +| sve2 | `simd_movavg_f64` | 0B → 144B | 2 → **34** | + +All are inside the 512-byte frame budget so nothing is dropped, and the results +are bit-identical — the reduction stays strictly in order. But six arm64 kernels +went from a zero frame to 80–320 bytes with 7–13x the scalar float arithmetic, +and this repository has no arm64 hardware lane. Whether the packed tap multiply +pays for the spill traffic is unknown on the only architecture where the +question is open. + `movavg` also rules out the obvious alternative: a sliding window (subtract the element leaving, add the one arriving) is O(n) instead of O(n·w) and gives a DIFFERENT number, because floating-point addition does not cancel. From 03cfda44609506a0aabb168064d419501ee826e7 Mon Sep 17 00:00:00 2001 From: Sebastian Mocanu Date: Tue, 18 Aug 2026 23:02:56 +0100 Subject: [PATCH 12/13] test: check the kernel guide's worked example against the tree --- CHANGELOG.md | 15 ++ README.md | 2 +- docs/wrong.md | 37 +++++ internal/tests/docs/kernels_guide_test.go | 171 ++++++++++++++++++++++ 4 files changed, 224 insertions(+), 1 deletion(-) create mode 100644 internal/tests/docs/kernels_guide_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 75eed4b..6560fe7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,21 @@ ## Unreleased +**The kernel guide's worked example is now checked against the tree.** +`docs/kernels.md` is the answer to "the operation I need isn't in the 403", +and its centre is one `spec.Kernel` literal whose eight annotated fields a +reader copies. Nothing looked inside it: the link gate checks its links and +the test-name gate checks the tests it cites, and neither reads a code block. +A renamed manifest field would leave the guide teaching a literal that does +not compile, in generated code the reader did not write. The gate reads the +field list out of the document rather than restating it, parses `spec.Kernel` +from source (`tools/` is a separate module and must never become a consumer +dependency), and also resolves the example's `RefFunc` against +`internal/ref` and its `Field` against `internal/kernel` -- the two wirings +that are easiest to forget, one of which once shipped past the whole normal +lane. Four mutations, all red: a field renamed in the guide, the same field +renamed in the struct, an unexported `RefFunc`, an unwired `Field`. + **`PolyEval`, `Convolve`, `Correlate` and `MovingAverage` now vectorize.** Their kernels accumulate per output, so the innermost loop carried a dependence LLVM cannot break without reassociating -- which the numerical diff --git a/README.md b/README.md index 6ba3ffa..c9ffa5e 100644 --- a/README.md +++ b/README.md @@ -389,7 +389,7 @@ cgo. ## Engineering record -[**docs/wrong.md**](docs/wrong.md) records 86 things that measurement disproved, +[**docs/wrong.md**](docs/wrong.md) records 87 things that measurement disproved, including changes that were deleted rather than shipped. Examples: - Green test lanes had executed no accelerated code for months. diff --git a/docs/wrong.md b/docs/wrong.md index 36acbb0..a0fa3af 100644 --- a/docs/wrong.md +++ b/docs/wrong.md @@ -3624,3 +3624,40 @@ question is open. `movavg` also rules out the obvious alternative: a sliding window (subtract the element leaving, add the one arriving) is O(n) instead of O(n·w) and gives a DIFFERENT number, because floating-point addition does not cancel. + +## 78. Generating the ref/kernel wiring is now maintainability, not correctness + +The plan's extensibility item proposed generating two of the eight files a new +kernel touches -- `internal/ref/ref.go`'s ops-table entry and +`internal/kernel/kernel.go`'s struct field -- on the argument that they are +"the two easiest things to forget, one of which caused the `-tags purego` panic +that `TestDispatchTableComplete` now guards". + +Both halves of that failure are now caught by a gate, in both lanes. Measured, +`Sqrt: sqrt[T]` deleted from `floatOps`: + + normal purego + before TestDeclaredKernelsAreWiredInTheReference green red + (and red only through TestInPlaceMatchesInto, + a functional test, not a completeness gate) + after RED RED + +`TestDeclaredKernelsAreWiredInTheReference` walks `backend.Inventory` and, for +every declared kernel, asserts `kernel.Set` has the group AND the field AND +that the field is non-nil in `ref.Set()`. That is exactly the three-way +relation generation was proposed to enforce: manifest -> struct field -> +reference wiring. A forgotten entry cannot ship past the ordinary `go test` +lane any more. + +So the item is not dropped, it is REPRICED. What generation still buys is two +fewer files to edit per kernel and one less place for a rename to go stale; +what it no longer buys is protection against a class of defect that has +already shipped once. That matters for ordering: it is a refactor of a stable +v1 contract across six architectures, needing `make codegen`, `make verify` and +the whole cross-architecture matrix to land safely, and its remaining value is +ergonomic. It goes behind anything that fixes an answer. + +The mutation is recorded here rather than left implicit because the plan's own +justification for the item no longer holds, and the next person to read that +plan would otherwise re-derive the priority from a sentence that was true when +it was written. diff --git a/internal/tests/docs/kernels_guide_test.go b/internal/tests/docs/kernels_guide_test.go new file mode 100644 index 0000000..33457be --- /dev/null +++ b/internal/tests/docs/kernels_guide_test.go @@ -0,0 +1,171 @@ +package docs + +import ( + "go/ast" + "go/parser" + "go/token" + "os" + "path/filepath" + "regexp" + "strings" + "testing" +) + +// docs/kernels.md's worked example names things that have to exist. +// +// The guide is the answer to "the operation I need isn't in the 403", and its +// centre is one `spec.Kernel` literal with eight fields, each annotated with +// what it is for. A reader copies that literal. If a field is renamed the +// guide goes on naming the old one, and the person following it gets a +// compile error in generated code they did not write and cannot easily read +// -- which is the moment the guide exists to prevent. +// +// The link gate and the test-name gate already cover this document. Neither +// looks inside a code block. +// +// THE FIELD LIST IS READ FROM THE DOCUMENT, not written here. A list in this +// file would have to be kept in step with the guide by hand, which is the +// thing being gated, one level up. +func TestTheKernelGuidesManifestFieldsExist(t *testing.T) { + doc := readDoc(t, filepath.Join("docs", "kernels.md")) + + // The one block that shows a manifest entry, from `spec.Kernel{` to the + // fence that ends it. + block := regexp.MustCompile("(?s)```go\\s*\nspec\\.Kernel\\{(.*?)\n```").FindStringSubmatch(doc) + if block == nil { + t.Fatal("docs/kernels.md no longer contains a ```go block starting " + + "`spec.Kernel{`. That literal is the guide's worked example and " + + "this gate reads it; if the example moved, repoint the gate rather " + + "than leaving it green over nothing") + } + + key := regexp.MustCompile(`(?m)^\s*([A-Z][A-Za-z0-9]*):`) + var named []string + for _, m := range key.FindAllStringSubmatch(block[1], -1) { + named = append(named, m[1]) + } + if len(named) < 5 { + t.Fatalf("the worked example names %d fields (%v); it documented eight, "+ + "so the extraction has lost the block rather than the block having "+ + "shrunk", len(named), named) + } + + fields := structFields(t, filepath.Join("tools", "simdgen", "spec", "spec.go"), "Kernel") + if len(fields) < 5 { + t.Fatalf("spec.Kernel parsed to %d fields; the parse has lost the "+ + "struct", len(fields)) + } + for _, n := range named { + if !fields[n] { + t.Errorf("docs/kernels.md's worked example sets spec.Kernel.%s, "+ + "which the struct does not declare. Someone following the guide "+ + "writes a manifest entry that does not compile", n) + } + } + + // AND THE TWO HAND-WRITTEN WIRINGS THE EXAMPLE POINTS AT. + // + // `RefFunc` and `Field` are the two files `docs/plans` wants generated + // precisely because they are the two easiest things to forget when adding a + // kernel -- forgetting the ref wiring shipped past the whole normal lane + // once. The guide names a REAL kernel for its example, so both ends of it + // can be checked rather than described. + lit := regexp.MustCompile(`(?m)^\s*(RefFunc|Field):\s*"([A-Za-z0-9]+)"`) + seen := 0 + for _, m := range lit.FindAllStringSubmatch(block[1], -1) { + seen++ + switch m[1] { + case "RefFunc": + if !refExports(t)[m[2]] { + t.Errorf("the guide's example declares RefFunc %q, which "+ + "internal/ref does not export. The guard would fall back to "+ + "a function that does not exist", m[2]) + } + case "Field": + if !kernelOpsFields(t)[m[2]] { + t.Errorf("the guide's example declares Field %q, which "+ + "internal/kernel does not declare on any kernel group. That "+ + "is the wiring a new kernel is most often missing", m[2]) + } + } + } + if seen != 2 { + t.Fatalf("found %d of the example's RefFunc/Field literals, want 2; "+ + "the two assertions above then covered less than they claim", seen) + } +} + +// structFields is the field names of one struct in one file, multi-name +// declarations (`Group, Field string`) included -- which is how two of the +// eight are declared, so a parser that took one name per line would report +// the guide as wrong about fields that exist. +// +// Parsed rather than imported: `tools/` is a separate module and must never +// become a consumer dependency, which is a rule in CLAUDE.md. +func structFields(t *testing.T, rel, name string) map[string]bool { + t.Helper() + f, err := parser.ParseFile(token.NewFileSet(), path(rel), nil, 0) + if err != nil { + t.Fatalf("parsing %s: %v", rel, err) + } + out := map[string]bool{} + ast.Inspect(f, func(n ast.Node) bool { + ts, ok := n.(*ast.TypeSpec) + if !ok || ts.Name.Name != name { + return true + } + st, ok := ts.Type.(*ast.StructType) + if !ok { + return true + } + for _, fld := range st.Fields.List { + for _, id := range fld.Names { + out[id.Name] = true + } + } + return false + }) + return out +} + +// refExports is every exported function internal/ref declares. +func refExports(t *testing.T) map[string]bool { + t.Helper() + return declaredIn(t, filepath.Join("internal", "ref"), + regexp.MustCompile(`(?m)^func ([A-Z][A-Za-z0-9]*)`)) +} + +// kernelOpsFields is every function-typed field on any group in +// internal/kernel -- the `Field` half of a manifest entry. +func kernelOpsFields(t *testing.T) map[string]bool { + t.Helper() + return declaredIn(t, filepath.Join("internal", "kernel"), + regexp.MustCompile(`(?m)^\t([A-Z][A-Za-z0-9]*)\s+func\(`)) +} + +func declaredIn(t *testing.T, dir string, re *regexp.Regexp) map[string]bool { + t.Helper() + out := map[string]bool{} + matches, err := filepath.Glob(filepath.Join(path(dir), "*.go")) + if err != nil { + t.Fatal(err) + } + for _, p := range matches { + if strings.HasSuffix(p, "_test.go") { + continue + } + b, err := os.ReadFile(p) + if err != nil { + t.Fatal(err) + } + for _, m := range re.FindAllStringSubmatch(string(b), -1) { + out[m[1]] = true + } + } + if len(out) == 0 { + t.Fatalf("no declarations matched in %s; the pattern has stopped "+ + "finding them and every lookup against this set would pass or fail "+ + "for the wrong reason", dir) + } + return out +} From c838172a23a2317215284c83272f7aba8d014fab Mon Sep 17 00:00:00 2001 From: Sebastian Mocanu Date: Sat, 22 Aug 2026 01:29:06 +0100 Subject: [PATCH 13/13] docs: release v1.21.0 update, SEO optimization, and FAQ section --- AGENTS.md | 2 +- CHANGELOG.md | 3 ++- CLAUDE.md | 2 +- README.md | 26 +++++++++++++++++++++----- docs/assets/badges/release.svg | 2 +- 5 files changed, 26 insertions(+), 9 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 2274056..60fe8d4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -20,7 +20,7 @@ library live in their own repositories. ## Current shipped status -The tip describes the shipped v1 line. The current release is **v1.20.0**; +The tip describes the shipped v1 line. The current release is **v1.21.0**; the exported-function and generated-kernel counts are the machine-checked ones in README.md and docs/platforms.md, held to the sources by `internal/tests/docs`. The tier targets are sse2, avx2, avx512, neon, sve2, diff --git a/CHANGELOG.md b/CHANGELOG.md index 6560fe7..590f1ca 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,7 @@ # Changelog -## Unreleased +## v1.21.0 + **The kernel guide's worked example is now checked against the tree.** `docs/kernels.md` is the answer to "the operation I need isn't in the 403", diff --git a/CLAUDE.md b/CLAUDE.md index 0e33741..03f53fa 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -67,7 +67,7 @@ live in their own repositories. ## Current shipped status -The current release is **v1.20.0**; the exported-function and +The current release is **v1.21.0**; the exported-function and generated-kernel counts are the machine-checked ones in README.md and docs/platforms.md. The v1 API is stable — every exported function keeps its name, signature, and meaning, and so does the numerical diff --git a/README.md b/README.md index c9ffa5e..a1d327d 100644 --- a/README.md +++ b/README.md @@ -1,13 +1,12 @@ -# simd.go +# simd.go — Fast Whole-Slice SIMD Vector Primitives & Text Scanning for Go (AVX-512, AVX2, NEON, SVE2, RVV) [![Latest Release](docs/assets/badges/release.svg)](https://github.com/sebishogun/simd/releases/latest) [![Go Reference](https://pkg.go.dev/badge/github.com/sebishogun/simd.svg)](https://pkg.go.dev/github.com/sebishogun/simd) [![CI](https://github.com/sebishogun/simd/actions/workflows/ci-local.yml/badge.svg)](https://github.com/sebishogun/simd/actions/workflows/ci-local.yml) -**Fast slice math and text scanning for Go, using the vector unit your CPU -already has — without cgo.** +**Hardware-accelerated SIMD vector math, text scanning, matrix operations, and columnar codecs for Go — using the vector unit your CPU already has, without cgo.** -`simd` applies runtime-selected SIMD kernels to ordinary Go slices. The v1.20.0 +`simd` applies runtime-selected SIMD kernels to ordinary Go slices. The v1.21.0 tree contains 493 exported functions and 6,933 generated kernels for SSE2, AVX2, AVX-512, NEON, SVE2, RVV, VSX, VX, and LASX. A portable Go path covers every operation and every unsupported target. @@ -403,9 +402,26 @@ including changes that were deleted rather than shipped. Examples: `docs/research/` carries the longer reasoning behind design decisions; `05-decisions.md` is the decision record. +## Frequently asked questions + +- **Does Go support SIMD vectorization natively without cgo?** + Yes. `simd` provides whole-slice SIMD vector acceleration in pure Go without cgo, C toolchains, or custom build tags. Ahead-of-time generated Plan 9 assembly kernels execute directly under Go's internal register ABI (ABIInternal). + +- **Which SIMD instruction sets and CPU architectures are supported?** + `simd` ships 6,933 generated kernels across 10 ISA tiers: x86-64 (SSE2, AVX2, AVX-512), ARM64 (NEON, SVE2), RISC-V (RVV 1.0), PowerPC (VSX), SystemZ (VX), and LoongArch (LASX), with a portable Go fallback for all unsupported targets. + +- **How does `simd.go` compare to `GOEXPERIMENT=simd` intrinsics in Go 1.26 and Go 1.27?** + Go 1.26 introduced experimental intrinsics (`simd/archsimd` on `amd64`), and Go 1.27 expanded it to `arm64` NEON and `wasm`. However, `GOEXPERIMENT=simd` requires consumers to set a build-time flag, does not support scalable vector units (ARM SVE2, RISC-V RVV 1.0), and lacks Clang unrolling optimizations ($4.4\times$ slower on whole-slice loops at $N=256$). `simd.go` requires zero build flags, supports 10 ISA tiers, and delivers optimal whole-slice throughput. + +- **Does `simd.go` support WebAssembly (WASM)?** + On `GOARCH=wasm`, `simd` automatically executes the portable Go fallback path. Go's WASM assembler (`cmd/internal/obj/wasm`) lacks SIMD128 opcodes and raw byte lifting breaks WebAssembly stack validation, so WebAssembly SIMD is left to Go 1.27's internal compiler intrinsics. + +- **What is the minimum slice length (crossover threshold) for SIMD in Go?** + The assembly call overhead is roughly 0.45 ns to 1.4 ns. For slice sizes under 16 to 64 elements, scalar Go is faster; for batches of hundreds to millions of elements, `simd` delivers up to $109\times$ speedups. + ## Status -**v1.20.0.** The API is stable: every exported function keeps its name, +**v1.21.0.** The API is stable: every exported function keeps its name, signature and meaning for the life of v1, and so does the numerical contract above. [CHANGELOG.md](CHANGELOG.md) states exactly what compatibility covers and what it excludes. [ROADMAP.md](ROADMAP.md) lists what is still open. diff --git a/docs/assets/badges/release.svg b/docs/assets/badges/release.svg index 9eeab0f..731d9a8 100644 --- a/docs/assets/badges/release.svg +++ b/docs/assets/badges/release.svg @@ -1 +1 @@ -release: v1.20.0releasev1.20.0 \ No newline at end of file +release: v1.21.0releasev1.21.0 \ No newline at end of file