diff --git a/AGENTS.md b/AGENTS.md index 2274056..60fe8d4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -20,7 +20,7 @@ library live in their own repositories. ## Current shipped status -The tip describes the shipped v1 line. The current release is **v1.20.0**; +The tip describes the shipped v1 line. The current release is **v1.21.0**; the exported-function and generated-kernel counts are the machine-checked ones in README.md and docs/platforms.md, held to the sources by `internal/tests/docs`. The tier targets are sse2, avx2, avx512, neon, sve2, diff --git a/CHANGELOG.md b/CHANGELOG.md index 0ef52a1..590f1ca 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,50 @@ # Changelog +## v1.21.0 + + +**The kernel guide's worked example is now checked against the tree.** +`docs/kernels.md` is the answer to "the operation I need isn't in the 403", +and its centre is one `spec.Kernel` literal whose eight annotated fields a +reader copies. Nothing looked inside it: the link gate checks its links and +the test-name gate checks the tests it cites, and neither reads a code block. +A renamed manifest field would leave the guide teaching a literal that does +not compile, in generated code the reader did not write. The gate reads the +field list out of the document rather than restating it, parses `spec.Kernel` +from source (`tools/` is a separate module and must never become a consumer +dependency), and also resolves the example's `RefFunc` against +`internal/ref` and its `Field` against `internal/kernel` -- the two wirings +that are easiest to forget, one of which once shipped past the whole normal +lane. Four mutations, all red: a field renamed in the guide, the same field +renamed in the struct, an unexported `RefFunc`, an unwired `Field`. + +**`PolyEval`, `Convolve`, `Correlate` and `MovingAverage` now vectorize.** +Their kernels accumulate per output, so the innermost loop carried a +dependence LLVM cannot break without reassociating -- which the numerical +contract forbids -- and all eight symbols emitted zero packed arithmetic +instructions on every amd64 tier. Blocking the *outer* loop by 16 makes the +innermost loop independent while each output still evaluates the same terms +in the same order, so the results are bit-identical rather than close: 8-40 +packed instructions per symbol now, and loong64 gains `simd_movavg_f32/f64`, +6,931 -> 6,933 kernels. Verified over 3.29M cases per tier on all nine tiers +including catastrophic cancellation and mid-accumulation overflow, where any +regrouping would show. Speed is not measured, and on arm64 the vectorizer +takes a different shape that costs stack frames -- `docs/wrong.md` entry 77 +records both. + +**Complex operations now run the generated assembly.** They never did: +`complexOps` returned the portable reference set directly, and the +generator emitted no complex entries into the per-tier dispatch tables, so +`DotComplex`, `DotComplexConj`, `ScaleComplex`, `SumComplex` and the +complex arithmetic set ran ordinary Go on every architecture while nine +tiers of generated complex assembly sat linked into a test-only aggregator. +Measured on amd64/avx512, scalar against the tier in one binary: 1.9x-12.9x +depending on operation and length. Results are unchanged -- the numerical +contract is the same fixed accumulation order, and the reference and every +tier agree bit for bit. No API change. `docs/wrong.md` records the finding, +the measurement, and the generator assertion that now refuses to emit a +dispatch file for a kernel group it cannot route. + ## v1.20.0 **The columnar codec tier.** **`BitUnpackInto`** gains a width- @@ -669,6 +714,8 @@ stops paying. The README's function table claimed every entry had a runnable example. It was false for 100 of 109 rows. All 109 have one now, and `TestReadmeTableHasExamples` fails the build if a row appears without one. +(It is `TestOperationCatalogHasExamples` now; the guarantee did not change, +the name did, and this entry is left as it was written.) `TestGuidesNameRealFunctions` does the same for the prose. Writing them found three documentation bugs that only executing catches: the diff --git a/CLAUDE.md b/CLAUDE.md index 0e33741..03f53fa 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -67,7 +67,7 @@ live in their own repositories. ## Current shipped status -The current release is **v1.20.0**; the exported-function and +The current release is **v1.21.0**; the exported-function and generated-kernel counts are the machine-checked ones in README.md and docs/platforms.md. The v1 API is stable — every exported function keeps its name, signature, and meaning, and so does the numerical diff --git a/Makefile b/Makefile index a27e8f1..0913dea 100644 --- a/Makefile +++ b/Makefile @@ -71,24 +71,53 @@ test-vec: ## Run the suite with GOEXPERIMENT=simd, which compiles vec.go test-tiers: ## Run the suite once per instruction-set tier this CPU has @for t in $(TIERS); do \ echo "--- GOSIMD=$$t"; \ - GOSIMD=$$t $(GO) test $(PKG) || exit 1; \ + GOSIMD=$$t $(GO) test -count=1 $(PKG) || exit 1; \ done .PHONY: test-race test-race: ## Full suite under the race detector $(GO) test -race $(PKG) -# Two fuzz targets. FuzzDifferential drives the public API; the one under -# internal/conformance drives every generated kernel of every tier against the -# portable implementation, with the fuzzer choosing the bit patterns rather -# than a table choosing the values. For floating point that distinction is the -# whole point: the inputs that break a kernel are not large or small, they are -# a signalling NaN or a denormal at the exact exponent boundary. +# Every fuzz target in the tree, DISCOVERED rather than listed. +# +# FuzzDifferential drives the public API; the one under internal/conformance +# drives every generated kernel of every tier against the portable +# implementation, with the fuzzer choosing the bit patterns rather than a table +# choosing the values. For floating point that distinction is the whole point: +# the inputs that break a kernel are not large or small, they are a signalling +# NaN or a denormal at the exact exponent boundary. +# +# This comment used to open "Two fuzz targets" and the recipe named two. There +# are three, and TWO of them had never been fuzzed by this target: +# +# FuzzJSONMasksMatchesSeparateCalls never named; only its seed corpus ever +# ran, under `go test` +# FuzzDifferential named, and pointed at the ROOT package +# while the target lives in +# internal/tests/arrays +# +# The second is the dangerous one, because `go test -run '^$$' -fuzz X .` in a +# package with NO target called X exits 0 and prints `PASS ... 0.002s`. It does +# not error, so `make fuzz` had been reporting success for a step that fuzzed +# nothing. A gate that finishes in two milliseconds is not a fast gate. +# +# Discovery makes that unreachable: a target is only fuzzed in the package +# `go test -list` found it in. The counter below fails the target outright if +# it ends up running nothing at all. .PHONY: fuzz -fuzz: ## Fuzz the public API and the tier-vs-reference differential - $(GO) test -run '^$$' -fuzz FuzzDifferential -fuzztime $(or $(FUZZTIME),60s) . - $(GO) test -run '^$$' -fuzz FuzzKernelsAgainstReference \ - -fuzztime $(or $(FUZZTIME),60s) ./internal/conformance/ +fuzz: ## Fuzz every target in the tree + @set -e; n=0; \ + for pkg in $$($(GO) list ./...); do \ + for t in $$($(GO) test -list '^Fuzz' $$pkg 2>/dev/null | grep '^Fuzz' || true); do \ + echo "fuzz $$pkg $$t"; \ + $(GO) test -run '^$$$$' -fuzz "^$$t$$" -fuzztime $(or $(FUZZTIME),60s) $$pkg; \ + n=$$((n+1)); \ + done; \ + done; \ + if [ "$$n" -eq 0 ]; then \ + echo "make fuzz ran no targets" >&2; exit 1; \ + fi; \ + echo "fuzzed $$n target(s)" # Every architecture with a backend, under emulation. # diff --git a/README.md b/README.md index 38136c0..a1d327d 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,13 @@ -# simd.go +# simd.go — Fast Whole-Slice SIMD Vector Primitives & Text Scanning for Go (AVX-512, AVX2, NEON, SVE2, RVV) [![Latest Release](docs/assets/badges/release.svg)](https://github.com/sebishogun/simd/releases/latest) [![Go Reference](https://pkg.go.dev/badge/github.com/sebishogun/simd.svg)](https://pkg.go.dev/github.com/sebishogun/simd) [![CI](https://github.com/sebishogun/simd/actions/workflows/ci-local.yml/badge.svg)](https://github.com/sebishogun/simd/actions/workflows/ci-local.yml) -**Fast slice math and text scanning for Go, using the vector unit your CPU -already has — without cgo.** +**Hardware-accelerated SIMD vector math, text scanning, matrix operations, and columnar codecs for Go — using the vector unit your CPU already has, without cgo.** -`simd` applies runtime-selected SIMD kernels to ordinary Go slices. The v1.20.0 -tree contains 493 exported functions and 6,931 generated kernels for SSE2, +`simd` applies runtime-selected SIMD kernels to ordinary Go slices. The v1.21.0 +tree contains 493 exported functions and 6,933 generated kernels for SSE2, AVX2, AVX-512, NEON, SVE2, RVV, VSX, VX, and LASX. A portable Go path covers every operation and every unsupported target. @@ -23,7 +22,7 @@ kernels are compiled ahead of time and committed as assembly, so this is an ordinary Go dependency with one transitive import (`golang.org/x/sys`, for CPU feature detection). Dispatch is one static table per operation, so the linker keeps only the operations a program actually calls: a binary using three -operation families does not retain all 6,931 kernels. +operation families does not retain all 6,933 kernels. ## Quick start @@ -389,7 +388,7 @@ cgo. ## Engineering record -[**docs/wrong.md**](docs/wrong.md) records 79 things that measurement disproved, +[**docs/wrong.md**](docs/wrong.md) records 87 things that measurement disproved, including changes that were deleted rather than shipped. Examples: - Green test lanes had executed no accelerated code for months. @@ -403,9 +402,26 @@ including changes that were deleted rather than shipped. Examples: `docs/research/` carries the longer reasoning behind design decisions; `05-decisions.md` is the decision record. +## Frequently asked questions + +- **Does Go support SIMD vectorization natively without cgo?** + Yes. `simd` provides whole-slice SIMD vector acceleration in pure Go without cgo, C toolchains, or custom build tags. Ahead-of-time generated Plan 9 assembly kernels execute directly under Go's internal register ABI (ABIInternal). + +- **Which SIMD instruction sets and CPU architectures are supported?** + `simd` ships 6,933 generated kernels across 10 ISA tiers: x86-64 (SSE2, AVX2, AVX-512), ARM64 (NEON, SVE2), RISC-V (RVV 1.0), PowerPC (VSX), SystemZ (VX), and LoongArch (LASX), with a portable Go fallback for all unsupported targets. + +- **How does `simd.go` compare to `GOEXPERIMENT=simd` intrinsics in Go 1.26 and Go 1.27?** + Go 1.26 introduced experimental intrinsics (`simd/archsimd` on `amd64`), and Go 1.27 expanded it to `arm64` NEON and `wasm`. However, `GOEXPERIMENT=simd` requires consumers to set a build-time flag, does not support scalable vector units (ARM SVE2, RISC-V RVV 1.0), and lacks Clang unrolling optimizations ($4.4\times$ slower on whole-slice loops at $N=256$). `simd.go` requires zero build flags, supports 10 ISA tiers, and delivers optimal whole-slice throughput. + +- **Does `simd.go` support WebAssembly (WASM)?** + On `GOARCH=wasm`, `simd` automatically executes the portable Go fallback path. Go's WASM assembler (`cmd/internal/obj/wasm`) lacks SIMD128 opcodes and raw byte lifting breaks WebAssembly stack validation, so WebAssembly SIMD is left to Go 1.27's internal compiler intrinsics. + +- **What is the minimum slice length (crossover threshold) for SIMD in Go?** + The assembly call overhead is roughly 0.45 ns to 1.4 ns. For slice sizes under 16 to 64 elements, scalar Go is faster; for batches of hundreds to millions of elements, `simd` delivers up to $109\times$ speedups. + ## Status -**v1.20.0.** The API is stable: every exported function keeps its name, +**v1.21.0.** The API is stable: every exported function keeps its name, signature and meaning for the life of v1, and so does the numerical contract above. [CHANGELOG.md](CHANGELOG.md) states exactly what compatibility covers and what it excludes. [ROADMAP.md](ROADMAP.md) lists what is still open. diff --git a/complex.go b/complex.go index 18701d5..09f7dd1 100644 --- a/complex.go +++ b/complex.go @@ -28,9 +28,9 @@ func complexOps[C Complex]() *kernel.Complex[C] { var zero C switch any(zero).(type) { case complex64: - return any(&refBase.C64).(*kernel.Complex[C]) + return any(cacheC64.get(&refBase.C64, cplxC64ByTier[:])).(*kernel.Complex[C]) case complex128: - return any(&refBase.C128).(*kernel.Complex[C]) + return any(cacheC128.get(&refBase.C128, cplxC128ByTier[:])).(*kernel.Complex[C]) } panic("simd: unsupported complex type") } @@ -44,11 +44,11 @@ func complexParts[C Complex, R Float]() *kernel.ComplexParts[C, R] { var zero C switch any(zero).(type) { case complex64: - if p, ok := any(&refBase.C64Parts).(*kernel.ComplexParts[C, R]); ok { + if p, ok := any(cacheC64Parts.get(&refBase.C64Parts, partsC64PartsByTier[:])).(*kernel.ComplexParts[C, R]); ok { return p } case complex128: - if p, ok := any(&refBase.C128Parts).(*kernel.ComplexParts[C, R]); ok { + if p, ok := any(cacheC128Parts.get(&refBase.C128Parts, partsC128PartsByTier[:])).(*kernel.ComplexParts[C, R]); ok { return p } } diff --git a/csrc/numeric.c b/csrc/numeric.c index a8784a0..0532c27 100644 --- a/csrc/numeric.c +++ b/csrc/numeric.c @@ -69,15 +69,44 @@ void simd_norm_f64(double *__restrict out, const double *__restrict a, // what vectorizes; the inner loop over coefficients is a fixed dependency // chain per element, exactly like the transcendentals in math.c. // -// The accumulator is per element rather than per coefficient, so the whole -// polynomial is evaluated for a register's worth of x at a time. +// BLOCKED over elements, so the INNERMOST loop is the independent one. +// +// The obvious nesting -- outer over elements, inner over coefficients -- does +// not vectorize, and the reason is structural rather than a cost model: LLVM's +// loop vectorizer only ever vectorizes the innermost loop, and here that is +// the Horner recurrence, a serial dependency it cannot break without +// reassociating. Measured: simd_polyeval_f32 contained ten scalar float +// instructions and no packed arithmetic at sse2, avx2 and avx512 alike, while +// passing the "contains a vector instruction" check on the strength of a +// movups and an xorps. See docs/wrong.md entry 76. +// +// Blocking swaps which loop is innermost without changing any element's +// arithmetic: every output still evaluates the same coefficients in the same +// order, so the result is bit-identical, not merely close. The accumulator is +// a fixed-size array so nothing is allocated and the frame stays far under the +// 512-byte kernel budget -- POLY_BLOCK doubles is 128 bytes. +// +// The tail runs the original nesting: it is at most POLY_BLOCK-1 elements and +// vectorizing it is not worth a second code path to get wrong. +#define POLY_BLOCK 16 #define POLY_EVAL(T) \ isize n = nd < nx ? nd : nx; \ if (nc == 0) { \ for (isize i = 0; i < n; i++) d[i] = 0; \ return; \ } \ - for (isize i = 0; i < n; i++) { \ + isize i = 0; \ + for (; i + POLY_BLOCK <= n; i += POLY_BLOCK) { \ + T acc[POLY_BLOCK]; \ + for (isize b = 0; b < POLY_BLOCK; b++) acc[b] = c[nc - 1]; \ + for (isize k = nc - 2; k >= 0; k--) { \ + T ck = c[k]; \ + for (isize b = 0; b < POLY_BLOCK; b++) \ + acc[b] = acc[b] * x[i + b] + ck; \ + } \ + for (isize b = 0; b < POLY_BLOCK; b++) d[i + b] = acc[b]; \ + } \ + for (; i < n; i++) { \ T xv = x[i]; \ T acc = c[nc - 1]; \ for (isize k = nc - 2; k >= 0; k--) acc = acc * xv + c[k]; \ @@ -104,12 +133,36 @@ void simd_polyeval_f64(double *__restrict d, const double *__restrict x, // // An FFT would be asymptotically better and is a different kernel; this is // the form that wins for the short kernels callers actually convolve with. +// BLOCKED over outputs, for the reason POLY_EVAL is: the tap loop is the +// innermost one and it is a serial float reduction, so LLVM -- which +// vectorizes only innermost loops -- had nothing to work with, and the kernel +// shipped scalar on every tier while passing the "contains a vector +// instruction" check. Blocking makes the innermost loop the independent one +// over outputs. +// +// Each output still accumulates its taps in the same order, so this is +// bit-identical to the direct form rather than close to it. That matters here +// more than elsewhere: reassociating a dot product is exactly what +// `#pragma clang loop vectorize(enable)` would have done, and what the +// numerical contract forbids. +#define CONV_BLOCK 16 #define CONVOLVE(T, IDX) \ isize m = nk; \ if (m == 0 || n < m) return; \ isize outn = n - m + 1; \ if (nd < outn) outn = nd; \ - for (isize i = 0; i < outn; i++) { \ + isize i = 0; \ + for (; i + CONV_BLOCK <= outn; i += CONV_BLOCK) { \ + T acc[CONV_BLOCK]; \ + for (isize b = 0; b < CONV_BLOCK; b++) acc[b] = 0; \ + for (isize j = 0; j < m; j++) { \ + T kv = k[IDX]; \ + for (isize b = 0; b < CONV_BLOCK; b++) \ + acc[b] += s[i + b + j] * kv; \ + } \ + for (isize b = 0; b < CONV_BLOCK; b++) d[i + b] = acc[b]; \ + } \ + for (; i < outn; i++) { \ T acc = 0; \ for (isize j = 0; j < m; j++) acc += s[i + j] * k[IDX]; \ d[i] = acc; \ @@ -384,11 +437,27 @@ void simd_scatter_u64(unsigned long long *__restrict d, const int *__restrict id // rounding error than summing the window afresh. The reference sums each // window, so this does too, and the outer loop over windows is what // vectorizes. +// BLOCKED over outputs, exactly as CONVOLVE is and for the same reason: the +// window loop was innermost and is a serial float sum, so nothing vectorized. +// Each output sums its window in the same order, so the result is +// bit-identical -- which rules out the sliding-window form (subtract the +// element leaving, add the one arriving), which is O(n) instead of O(n*w) and +// gives a DIFFERENT number, because floating-point addition does not cancel. +#define MOVAVG_BLOCK 16 #define MOVING_AVERAGE(T) \ if (w <= 0 || na < w) return; \ isize outn = na - w + 1; \ if (nd < outn) outn = nd; \ - for (isize i = 0; i < outn; i++) { \ + isize i = 0; \ + for (; i + MOVAVG_BLOCK <= outn; i += MOVAVG_BLOCK) { \ + T acc[MOVAVG_BLOCK]; \ + for (isize b = 0; b < MOVAVG_BLOCK; b++) acc[b] = 0; \ + for (isize j = 0; j < w; j++) { \ + for (isize b = 0; b < MOVAVG_BLOCK; b++) acc[b] += a[i + b + j]; \ + } \ + for (isize b = 0; b < MOVAVG_BLOCK; b++) d[i + b] = acc[b] / (T)w; \ + } \ + for (; i < outn; i++) { \ T acc = 0; \ for (isize j = 0; j < w; j++) acc += a[i + j]; \ d[i] = acc / (T)w; \ diff --git a/dispatch.go b/dispatch.go index bd7fef7..9c15d9c 100644 --- a/dispatch.go +++ b/dispatch.go @@ -96,8 +96,49 @@ func (c *opsCache[T]) get(base *kernel.Ops[T], partial []*kernel.Ops[T]) *kernel return c.tiers[tierIdx] } -// overlay copies every non-nil function field of src over dst. -func overlay[T Number](dst, src *kernel.Ops[T]) { +// groupCache is opsCache for the two complex groups. They are the same shape +// -- a struct of function fields, a static per-tier partial, a lazy merge -- +// but kernel.Complex and kernel.ComplexParts are not kernel.Ops, and Go has +// no constraint that spans the three. One generic type over the struct +// itself, rather than over the element type, covers both. +// +// Until this existed, complexOps returned the reference set directly and +// every complex operation ran portable Go on every machine while nine tiers +// of generated complex assembly -- sse2, avx2, avx512, neon, sve2, rvv, vsx, +// vx, lasx, across six architectures -- sat linked into the test-only +// aggregator. +type groupCache[G any] struct { + once sync.Once + tiers [len(dispatchTiers)]*G +} + +func (c *groupCache[G]) get(base *G, partial []*G) *G { + c.once.Do(func() { + for i := range c.tiers { + s := *base + if i < len(partial) && partial[i] != nil { + overlayAny(&s, partial[i]) + } + c.tiers[i] = &s + } + }) + return c.tiers[tierIdx] +} + +var ( + cacheC64 groupCache[kernel.Complex[complex64]] + cacheC128 groupCache[kernel.Complex[complex128]] + cacheC64Parts groupCache[kernel.ComplexParts[complex64, float32]] + cacheC128Parts groupCache[kernel.ComplexParts[complex128, float64]] +) + +// overlay copies every non-nil function field of src over dst. It is +// overlayAny at kernel.Ops[T], kept as a named function because that is what +// opsCache reads and the constraint documents the intent. +func overlay[T Number](dst, src *kernel.Ops[T]) { overlayAny(dst, src) } + +// overlayAny is overlay for any struct of function fields. +func overlayAny[G any](dst, src *G) { d := reflect.ValueOf(dst).Elem() s := reflect.ValueOf(src).Elem() for i := 0; i < d.NumField(); i++ { diff --git a/dispatch_complex_test.go b/dispatch_complex_test.go new file mode 100644 index 0000000..5b7aaed --- /dev/null +++ b/dispatch_complex_test.go @@ -0,0 +1,155 @@ +package simd + +import ( + "reflect" + "testing" +) + +// The complex surface must reach the generated kernels, the same way the +// numeric surface does. +// +// It did not, from v1.14.0 to v1.20.0. complexOps returned &refBase.C64 / +// &refBase.C128 directly, with no per-tier overlay, so every complex +// operation ran the portable Go reference on every machine -- while the +// repository generated, tested and shipped nine tiers of complex assembly +// for them. The existing coverage did not catch it because +// TestDeclaredKernelsAreWired checks the runtime tables for exactly three +// groups (Bytes, Convert, Mask) and routes everything else -- numeric and +// complex alike -- through archSets(), the test-only whole-set aggregator +// whose own generated header says consumers never call it. Nothing walked +// the tables the runtime actually indexes. +// +// So this asks the question a caller asks: is the function DotComplex ends +// up calling the one the tier's table supplies? + +// tierPartial returns the per-tier partial for a group at the running tier, +// or nil when this build has no table for it (the purego and no-backend +// builds, and slot 0, the reference). +func tierPartial[G any](byTier []*G) *G { + if tierIdx < 0 || tierIdx >= len(byTier) { + return nil + } + return byTier[tierIdx] +} + +// checkGroup is the whole contract, field by field: where the tier's partial +// supplies a kernel the dispatch must reach exactly that function, and where +// it does not the dispatch must keep the reference. +// +// The second half matters as much as the first. A missing kernel on a tier +// is documented as slower and never a correctness gap +// (docs/lld/kernels-and-dispatch.md), and three architectures rely on it +// today -- s390x has no complex64 Dot or DotConj, ppc64le adds Sum to that +// list, loong64 is missing Mul, Div and Conj. An earlier version of this +// test asserted "not the reference" for one hardcoded field and turned that +// documented fallback into a red cross-compile gate on two architectures. +func checkGroup[G any](t *testing.T, name string, got, ref, partial *G) { + t.Helper() + g := reflect.ValueOf(got).Elem() + r := reflect.ValueOf(ref).Elem() + var p reflect.Value + if partial != nil { + p = reflect.ValueOf(partial).Elem() + } + typ := g.Type() + + supplied := 0 + for i := 0; i < g.NumField(); i++ { + if typ.Field(i).Type.Kind() != reflect.Func { + continue + } + field := name + "." + typ.Field(i).Name + gp := g.Field(i).Pointer() + rp := r.Field(i).Pointer() + + if partial != nil && !p.Field(i).IsNil() { + supplied++ + want := p.Field(i).Pointer() + if gp != want { + t.Errorf("%s on tier %q: dispatch reaches %#x, the tier's table holds %#x", + field, Tier(), gp, want) + } + if gp == rp && rp != want { + t.Errorf("%s on tier %q: dispatch is still the portable reference", field, Tier()) + } + continue + } + // No kernel on this tier: the reference stands, unchanged. + if gp != rp { + t.Errorf("%s on tier %q: no kernel is generated, but dispatch is not the reference", + field, Tier()) + } + } + + // A table that supplies nothing on an accelerated tier is a table that + // was emitted empty -- which is what the defect looked like. + if partial != nil && supplied == 0 && Tier() != "scalar" { + t.Errorf("%s on tier %q: the per-tier table supplies no kernel at all", name, Tier()) + } +} + +func TestComplexDispatchReachesGeneratedKernels(t *testing.T) { + checkGroup(t, "C64", complexOps[complex64](), &refBase.C64, tierPartial(cplxC64ByTier[:])) + checkGroup(t, "C128", complexOps[complex128](), &refBase.C128, tierPartial(cplxC128ByTier[:])) +} + +// The real-scalar half of the split -- Abs, Real, Imag, Scale -- goes through +// ComplexParts and had the same hole. +func TestComplexPartsDispatchReachesGeneratedKernels(t *testing.T) { + checkGroup(t, "C64Parts", complexParts[complex64, float32](), + &refBase.C64Parts, tierPartial(partsC64PartsByTier[:])) + checkGroup(t, "C128Parts", complexParts[complex128, float64](), + &refBase.C128Parts, tierPartial(partsC128PartsByTier[:])) +} + +// groupCache indexes its partial slice by tierIdx and guards with +// `i < len(partial)`, so a table shorter than dispatchTiers silently leaves +// the top tiers on the reference and a longer one silently drops its tail. +// Both are generator bugs that no other test would notice. +func TestComplexTierTablesMatchTheTierList(t *testing.T) { + for _, c := range []struct { + name string + n int + }{ + {"cplxC64ByTier", len(cplxC64ByTier)}, + {"cplxC128ByTier", len(cplxC128ByTier)}, + {"partsC64PartsByTier", len(partsC64PartsByTier)}, + {"partsC128PartsByTier", len(partsC128PartsByTier)}, + } { + if c.n != len(dispatchTiers) { + t.Errorf("%s has %d slots, dispatchTiers has %d", c.name, c.n, len(dispatchTiers)) + } + } +} + +// Whatever the dispatch routes to, the answer is the reference's answer. +// The numerical contract is the one every other tier keeps: a fixed +// accumulation order, identical on every architecture. +func TestComplexDispatchMatchesTheReference(t *testing.T) { + n := 1000 + a := make([]complex128, n) + b := make([]complex128, n) + for i := range a { + a[i] = complex(float64(i%17)-8, float64(i%23)-11) + b[i] = complex(float64(i%13)-6, float64(i%29)-14) + } + if got, want := DotComplex(a, b), refBase.C128.Dot(a, b); got != want { + t.Errorf("DotComplex = %v, reference = %v", got, want) + } + if got, want := DotComplexConj(a, b), refBase.C128.DotConj(a, b); got != want { + t.Errorf("DotComplexConj = %v, reference = %v", got, want) + } + if got, want := SumComplex(a), refBase.C128.Sum(a); got != want { + t.Errorf("SumComplex = %v, reference = %v", got, want) + } + + gotS := append([]complex128(nil), a...) + wantS := append([]complex128(nil), a...) + ScaleComplex(gotS, 1.5) + refBase.C128Parts.Scale(wantS, wantS, 1.5) + for i := range gotS { + if gotS[i] != wantS[i] { + t.Fatalf("ScaleComplex[%d] = %v, reference = %v", i, gotS[i], wantS[i]) + } + } +} diff --git a/dispatch_inventory_test.go b/dispatch_inventory_test.go index f5e4572..4199154 100644 --- a/dispatch_inventory_test.go +++ b/dispatch_inventory_test.go @@ -32,6 +32,7 @@ package simd import ( "reflect" + "strings" "testing" "github.com/sebishogun/simd/internal/backend" @@ -65,6 +66,30 @@ func TestDeclaredKernelsAreWired(t *testing.T) { } continue } + // The struct groups now have real per-tier tables, so walk those -- + // the runtime indexes them, and archSets() does not. Four whole + // complex groups were missing from the runtime tables for seven + // releases with this test green, because everything that was not a + // flat group went through the test-only aggregator instead. + if slots, ok := allGroupTables[d.Group]; ok { + for i, partial := range slots { + if i == 0 || partial == nil { + continue // slot 0 is the reference; no overlay + } + f := reflect.ValueOf(partial).FieldByName(d.Field) + if !f.IsValid() { + t.Errorf("%s: per-tier table for %s has no field %q", + d.CName, d.Group, d.Field) + break + } + if f.Kind() == reflect.Func && f.IsNil() && present(sets, dispatchTiers[i], d) { + t.Errorf("tier %q: %s.%s is nil in the dispatch table, but the "+ + "kernel is generated for that tier -- it will never run", + dispatchTiers[i], d.Group, d.Field) + } + } + continue + } for tier, set := range sets { g := reflect.ValueOf(set).FieldByName(d.Group) if !g.IsValid() { @@ -105,3 +130,91 @@ func TestInventoryCoversEveryGroup(t *testing.T) { } } } + +// present reports whether the generator emitted a kernel for d on tier. +// +// Comparing against the REFERENCE, not against nil: archSets() builds each +// tier from ref.Set() and overlays what was generated, so every field is +// non-nil there and a nil test would call every operation "generated" on +// every tier. A tier the generator declined keeps the reference by design +// (docs/lld/kernels-and-dispatch.md), and only a kernel that exists and is +// missing from the dispatch table is a defect. +func present(sets map[string]kernel.Set, tier string, d backend.Declared) bool { + set, ok := sets[tier] + if !ok { + return false + } + g := reflect.ValueOf(set).FieldByName(d.Group) + r := reflect.ValueOf(refBase).FieldByName(d.Group) + if !g.IsValid() || !r.IsValid() { + return false + } + f, rf := g.FieldByName(d.Field), r.FieldByName(d.Field) + if !f.IsValid() || !rf.IsValid() || f.Kind() != reflect.Func || f.IsNil() { + return false + } + return rf.IsNil() || f.Pointer() != rf.Pointer() +} + +// Every declared kernel is wired in the REFERENCE too, not only in the +// generated tables. +// +// TestDeclaredKernelsAreWired above walks backend.Inventory against the +// dispatch tables and archSets(), which are the GENERATED side. It never looks +// at ref.Set(), so a forgotten entry in internal/ref's ops table is invisible +// to it. +// +// Measured 2026-08-15: `Sqrt: sqrt[T]` deleted from floatOps in +// internal/ref/ref.go. TestDeclaredKernelsAreWired PASSED in both the normal +// and the purego lane, and the WHOLE normal-lane suite -- all fourteen +// packages -- passed. Only the whole purego suite caught it, through +// TestInPlaceMatchesInto, a functional test rather than the completeness gate +// that exists for exactly this. So a forgotten reference wiring shipped unless +// somebody happened to run -tags purego. +// +// The reference is not optional: it is the live fallback on architectures with +// no backend, in purego builds, and below every kernel's element threshold. A +// nil there is a nil pointer dereference in the small-n path of an operation +// that works fine on a big slice. +// +// Fast slots are excluded because ref.Set() leaves them nil ON PURPOSE -- +// until a backend has registered, "no Fast kernel here" and "the accurate +// kernel" cannot be told apart, and dispatch.go fills them with +// FillFastFallbacks afterwards. That is the same exclusion the doc comment on +// TestDeclaredKernelsAreWired describes. +func TestDeclaredKernelsAreWiredInTheReference(t *testing.T) { + if len(backend.Inventory) == 0 { + t.Fatal("backend.Inventory is empty; run make codegen") + } + checked := 0 + for _, d := range backend.Inventory { + if strings.HasPrefix(d.Field, "Fast") { + continue + } + g := reflect.ValueOf(refBase).FieldByName(d.Group) + if !g.IsValid() { + t.Errorf("%s: kernel.Set has no group %q", d.CName, d.Group) + continue + } + f := g.FieldByName(d.Field) + if !f.IsValid() { + t.Errorf("%s: kernel.Set.%s has no field %q", d.CName, d.Group, d.Field) + continue + } + if f.Kind() != reflect.Func { + continue + } + checked++ + if f.IsNil() { + t.Errorf("%s.%s is nil in the REFERENCE, and the manifest declares %s "+ + "for that group: internal/ref is the live fallback below the "+ + "element threshold, in purego builds, and on every architecture "+ + "with no backend, so this is a nil call rather than a slow one", + d.Group, d.Field, d.CName) + } + } + if checked == 0 { + t.Fatal("no reference slots were checked, so this gate measures nothing") + } + t.Logf("%d declared non-Fast kernels checked against the reference", checked) +} diff --git a/dispatch_tables_amd64.go b/dispatch_tables_amd64.go index ed7ad96..1d16221 100644 --- a/dispatch_tables_amd64.go +++ b/dispatch_tables_amd64.go @@ -3175,3 +3175,151 @@ var opsU8ByTier = [...]*kernel.Ops[uint8]{ &opsU8AVX2, &opsU8AVX512, } + +var cplxC128SSE2 = kernel.Complex[complex128]{ + Add: amd64.CaddComplex128SSE2, + Sub: amd64.CsubComplex128SSE2, + Mul: amd64.CmulComplex128SSE2, + Div: amd64.CdivComplex128SSE2, + Neg: amd64.CnegComplex128SSE2, + Conj: amd64.CconjComplex128SSE2, + Sum: amd64.CsumComplex128SSE2, + Dot: amd64.CdotComplex128SSE2, + DotConj: amd64.CdotconjComplex128SSE2, +} + +var cplxC128AVX2 = kernel.Complex[complex128]{ + Add: amd64.CaddComplex128AVX2, + Sub: amd64.CsubComplex128AVX2, + Mul: amd64.CmulComplex128AVX2, + Div: amd64.CdivComplex128AVX2, + Neg: amd64.CnegComplex128AVX2, + Conj: amd64.CconjComplex128AVX2, + Sum: amd64.CsumComplex128AVX2, + Dot: amd64.CdotComplex128AVX2, + DotConj: amd64.CdotconjComplex128AVX2, +} + +var cplxC128AVX512 = kernel.Complex[complex128]{ + Add: amd64.CaddComplex128AVX512, + Sub: amd64.CsubComplex128AVX512, + Mul: amd64.CmulComplex128AVX512, + Div: amd64.CdivComplex128AVX512, + Neg: amd64.CnegComplex128AVX512, + Conj: amd64.CconjComplex128AVX512, + Sum: amd64.CsumComplex128AVX512, + Dot: amd64.CdotComplex128AVX512, + DotConj: amd64.CdotconjComplex128AVX512, +} + +var cplxC128ByTier = [...]*kernel.Complex[complex128]{ + nil, + &cplxC128SSE2, + &cplxC128AVX2, + &cplxC128AVX512, +} + +var cplxC64SSE2 = kernel.Complex[complex64]{ + Add: amd64.CaddComplex64SSE2, + Sub: amd64.CsubComplex64SSE2, + Mul: amd64.CmulComplex64SSE2, + Div: amd64.CdivComplex64SSE2, + Neg: amd64.CnegComplex64SSE2, + Conj: amd64.CconjComplex64SSE2, + Sum: amd64.CsumComplex64SSE2, + Dot: amd64.CdotComplex64SSE2, + DotConj: amd64.CdotconjComplex64SSE2, +} + +var cplxC64AVX2 = kernel.Complex[complex64]{ + Add: amd64.CaddComplex64AVX2, + Sub: amd64.CsubComplex64AVX2, + Mul: amd64.CmulComplex64AVX2, + Div: amd64.CdivComplex64AVX2, + Neg: amd64.CnegComplex64AVX2, + Conj: amd64.CconjComplex64AVX2, + Sum: amd64.CsumComplex64AVX2, + Dot: amd64.CdotComplex64AVX2, + DotConj: amd64.CdotconjComplex64AVX2, +} + +var cplxC64AVX512 = kernel.Complex[complex64]{ + Add: amd64.CaddComplex64AVX512, + Sub: amd64.CsubComplex64AVX512, + Mul: amd64.CmulComplex64AVX512, + Div: amd64.CdivComplex64AVX512, + Neg: amd64.CnegComplex64AVX512, + Conj: amd64.CconjComplex64AVX512, + Sum: amd64.CsumComplex64AVX512, + Dot: amd64.CdotComplex64AVX512, + DotConj: amd64.CdotconjComplex64AVX512, +} + +var cplxC64ByTier = [...]*kernel.Complex[complex64]{ + nil, + &cplxC64SSE2, + &cplxC64AVX2, + &cplxC64AVX512, +} + +var partsC128PartsSSE2 = kernel.ComplexParts[complex128, float64]{ + Abs: amd64.CabsComplex128SSE2, + Real: amd64.CrealComplex128SSE2, + Imag: amd64.CimagComplex128SSE2, + Scale: amd64.CscaleComplex128SSE2, + FromParts: amd64.CfromPartsComplex128SSE2, +} + +var partsC128PartsAVX2 = kernel.ComplexParts[complex128, float64]{ + Abs: amd64.CabsComplex128AVX2, + Real: amd64.CrealComplex128AVX2, + Imag: amd64.CimagComplex128AVX2, + Scale: amd64.CscaleComplex128AVX2, + FromParts: amd64.CfromPartsComplex128AVX2, +} + +var partsC128PartsAVX512 = kernel.ComplexParts[complex128, float64]{ + Abs: amd64.CabsComplex128AVX512, + Real: amd64.CrealComplex128AVX512, + Imag: amd64.CimagComplex128AVX512, + Scale: amd64.CscaleComplex128AVX512, + FromParts: amd64.CfromPartsComplex128AVX512, +} + +var partsC128PartsByTier = [...]*kernel.ComplexParts[complex128, float64]{ + nil, + &partsC128PartsSSE2, + &partsC128PartsAVX2, + &partsC128PartsAVX512, +} + +var partsC64PartsSSE2 = kernel.ComplexParts[complex64, float32]{ + Abs: amd64.CabsComplex64SSE2, + Real: amd64.CrealComplex64SSE2, + Imag: amd64.CimagComplex64SSE2, + Scale: amd64.CscaleComplex64SSE2, + FromParts: amd64.CfromPartsComplex64SSE2, +} + +var partsC64PartsAVX2 = kernel.ComplexParts[complex64, float32]{ + Abs: amd64.CabsComplex64AVX2, + Real: amd64.CrealComplex64AVX2, + Imag: amd64.CimagComplex64AVX2, + Scale: amd64.CscaleComplex64AVX2, + FromParts: amd64.CfromPartsComplex64AVX2, +} + +var partsC64PartsAVX512 = kernel.ComplexParts[complex64, float32]{ + Abs: amd64.CabsComplex64AVX512, + Real: amd64.CrealComplex64AVX512, + Imag: amd64.CimagComplex64AVX512, + Scale: amd64.CscaleComplex64AVX512, + FromParts: amd64.CfromPartsComplex64AVX512, +} + +var partsC64PartsByTier = [...]*kernel.ComplexParts[complex64, float32]{ + nil, + &partsC64PartsSSE2, + &partsC64PartsAVX2, + &partsC64PartsAVX512, +} diff --git a/dispatch_tables_amd64_test.go b/dispatch_tables_amd64_test.go index 519ecb7..151d2bf 100644 --- a/dispatch_tables_amd64_test.go +++ b/dispatch_tables_amd64_test.go @@ -127,5 +127,25 @@ var allFlatTables = map[string][]any{ "Bytes.VarintDecodeU64": {tblBytesVarintDecodeU64[0], tblBytesVarintDecodeU64[1], tblBytesVarintDecodeU64[2], tblBytesVarintDecodeU64[3]}, } +// allGroupTables enumerates the per-tier partial structs the runtime +// overlays from, keyed by kernel.Set group. Slot 0 is the reference +// and is nil; the rest are one per accelerated tier, in tier order. +var allGroupTables = map[string][]any{ + "F32": {nil, opsF32SSE2, opsF32AVX2, opsF32AVX512}, + "F64": {nil, opsF64SSE2, opsF64AVX2, opsF64AVX512}, + "I16": {nil, opsI16SSE2, opsI16AVX2, opsI16AVX512}, + "I32": {nil, opsI32SSE2, opsI32AVX2, opsI32AVX512}, + "I64": {nil, opsI64SSE2, opsI64AVX2, opsI64AVX512}, + "I8": {nil, opsI8SSE2, opsI8AVX2, opsI8AVX512}, + "U16": {nil, opsU16SSE2, opsU16AVX2, opsU16AVX512}, + "U32": {nil, opsU32SSE2, opsU32AVX2, opsU32AVX512}, + "U64": {nil, opsU64SSE2, opsU64AVX2, opsU64AVX512}, + "U8": {nil, opsU8SSE2, opsU8AVX2, opsU8AVX512}, + "C128": {nil, cplxC128SSE2, cplxC128AVX2, cplxC128AVX512}, + "C64": {nil, cplxC64SSE2, cplxC64AVX2, cplxC64AVX512}, + "C128Parts": {nil, partsC128PartsSSE2, partsC128PartsAVX2, partsC128PartsAVX512}, + "C64Parts": {nil, partsC64PartsSSE2, partsC64PartsAVX2, partsC64PartsAVX512}, +} + // archSets hands the tests every tier's complete kernel set. var archSets func() map[string]kernel.Set = amd64.Sets diff --git a/dispatch_tables_arm64.go b/dispatch_tables_arm64.go index 99f8cc6..343289f 100644 --- a/dispatch_tables_arm64.go +++ b/dispatch_tables_arm64.go @@ -2232,3 +2232,107 @@ var opsU8ByTier = [...]*kernel.Ops[uint8]{ &opsU8NEON, &opsU8SVE2, } + +var cplxC128NEON = kernel.Complex[complex128]{ + Add: arm64.CaddComplex128NEON, + Sub: arm64.CsubComplex128NEON, + Mul: arm64.CmulComplex128NEON, + Div: arm64.CdivComplex128NEON, + Neg: arm64.CnegComplex128NEON, + Conj: arm64.CconjComplex128NEON, + Sum: arm64.CsumComplex128NEON, + Dot: arm64.CdotComplex128NEON, + DotConj: arm64.CdotconjComplex128NEON, +} + +var cplxC128SVE2 = kernel.Complex[complex128]{ + Add: arm64.CaddComplex128SVE2, + Sub: arm64.CsubComplex128SVE2, + Mul: arm64.CmulComplex128SVE2, + Div: arm64.CdivComplex128SVE2, + Neg: arm64.CnegComplex128SVE2, + Conj: arm64.CconjComplex128SVE2, + Sum: arm64.CsumComplex128SVE2, + Dot: arm64.CdotComplex128SVE2, + DotConj: arm64.CdotconjComplex128SVE2, +} + +var cplxC128ByTier = [...]*kernel.Complex[complex128]{ + nil, + &cplxC128NEON, + &cplxC128SVE2, +} + +var cplxC64NEON = kernel.Complex[complex64]{ + Add: arm64.CaddComplex64NEON, + Sub: arm64.CsubComplex64NEON, + Mul: arm64.CmulComplex64NEON, + Div: arm64.CdivComplex64NEON, + Neg: arm64.CnegComplex64NEON, + Conj: arm64.CconjComplex64NEON, + Sum: arm64.CsumComplex64NEON, + Dot: arm64.CdotComplex64NEON, + DotConj: arm64.CdotconjComplex64NEON, +} + +var cplxC64SVE2 = kernel.Complex[complex64]{ + Add: arm64.CaddComplex64SVE2, + Sub: arm64.CsubComplex64SVE2, + Mul: arm64.CmulComplex64SVE2, + Div: arm64.CdivComplex64SVE2, + Neg: arm64.CnegComplex64SVE2, + Conj: arm64.CconjComplex64SVE2, + Sum: arm64.CsumComplex64SVE2, + Dot: arm64.CdotComplex64SVE2, + DotConj: arm64.CdotconjComplex64SVE2, +} + +var cplxC64ByTier = [...]*kernel.Complex[complex64]{ + nil, + &cplxC64NEON, + &cplxC64SVE2, +} + +var partsC128PartsNEON = kernel.ComplexParts[complex128, float64]{ + Abs: arm64.CabsComplex128NEON, + Real: arm64.CrealComplex128NEON, + Imag: arm64.CimagComplex128NEON, + Scale: arm64.CscaleComplex128NEON, + FromParts: arm64.CfromPartsComplex128NEON, +} + +var partsC128PartsSVE2 = kernel.ComplexParts[complex128, float64]{ + Abs: arm64.CabsComplex128SVE2, + Real: arm64.CrealComplex128SVE2, + Imag: arm64.CimagComplex128SVE2, + Scale: arm64.CscaleComplex128SVE2, + FromParts: arm64.CfromPartsComplex128SVE2, +} + +var partsC128PartsByTier = [...]*kernel.ComplexParts[complex128, float64]{ + nil, + &partsC128PartsNEON, + &partsC128PartsSVE2, +} + +var partsC64PartsNEON = kernel.ComplexParts[complex64, float32]{ + Abs: arm64.CabsComplex64NEON, + Real: arm64.CrealComplex64NEON, + Imag: arm64.CimagComplex64NEON, + Scale: arm64.CscaleComplex64NEON, + FromParts: arm64.CfromPartsComplex64NEON, +} + +var partsC64PartsSVE2 = kernel.ComplexParts[complex64, float32]{ + Abs: arm64.CabsComplex64SVE2, + Real: arm64.CrealComplex64SVE2, + Imag: arm64.CimagComplex64SVE2, + Scale: arm64.CscaleComplex64SVE2, + FromParts: arm64.CfromPartsComplex64SVE2, +} + +var partsC64PartsByTier = [...]*kernel.ComplexParts[complex64, float32]{ + nil, + &partsC64PartsNEON, + &partsC64PartsSVE2, +} diff --git a/dispatch_tables_arm64_test.go b/dispatch_tables_arm64_test.go index c793529..4fb7aaf 100644 --- a/dispatch_tables_arm64_test.go +++ b/dispatch_tables_arm64_test.go @@ -127,5 +127,25 @@ var allFlatTables = map[string][]any{ "Bytes.VarintDecodeU64": {tblBytesVarintDecodeU64[0], tblBytesVarintDecodeU64[1], tblBytesVarintDecodeU64[2]}, } +// allGroupTables enumerates the per-tier partial structs the runtime +// overlays from, keyed by kernel.Set group. Slot 0 is the reference +// and is nil; the rest are one per accelerated tier, in tier order. +var allGroupTables = map[string][]any{ + "F32": {nil, opsF32NEON, opsF32SVE2}, + "F64": {nil, opsF64NEON, opsF64SVE2}, + "I16": {nil, opsI16NEON, opsI16SVE2}, + "I32": {nil, opsI32NEON, opsI32SVE2}, + "I64": {nil, opsI64NEON, opsI64SVE2}, + "I8": {nil, opsI8NEON, opsI8SVE2}, + "U16": {nil, opsU16NEON, opsU16SVE2}, + "U32": {nil, opsU32NEON, opsU32SVE2}, + "U64": {nil, opsU64NEON, opsU64SVE2}, + "U8": {nil, opsU8NEON, opsU8SVE2}, + "C128": {nil, cplxC128NEON, cplxC128SVE2}, + "C64": {nil, cplxC64NEON, cplxC64SVE2}, + "C128Parts": {nil, partsC128PartsNEON, partsC128PartsSVE2}, + "C64Parts": {nil, partsC64PartsNEON, partsC64PartsSVE2}, +} + // archSets hands the tests every tier's complete kernel set. var archSets func() map[string]kernel.Set = arm64.Sets diff --git a/dispatch_tables_fallback.go b/dispatch_tables_fallback.go index 889f311..109c129 100644 --- a/dispatch_tables_fallback.go +++ b/dispatch_tables_fallback.go @@ -138,3 +138,7 @@ var opsU16ByTier = [...]*kernel.Ops[uint16]{nil} var opsU32ByTier = [...]*kernel.Ops[uint32]{nil} var opsU64ByTier = [...]*kernel.Ops[uint64]{nil} var opsU8ByTier = [...]*kernel.Ops[uint8]{nil} +var cplxC128ByTier = [...]*kernel.Complex[complex128]{nil} +var cplxC64ByTier = [...]*kernel.Complex[complex64]{nil} +var partsC128PartsByTier = [...]*kernel.ComplexParts[complex128, float64]{nil} +var partsC64PartsByTier = [...]*kernel.ComplexParts[complex64, float32]{nil} diff --git a/dispatch_tables_fallback_test.go b/dispatch_tables_fallback_test.go index 375ed65..d3e5972 100644 --- a/dispatch_tables_fallback_test.go +++ b/dispatch_tables_fallback_test.go @@ -122,6 +122,10 @@ var allFlatTables = map[string][]any{ "Bytes.VarintDecodeU64": {tblBytesVarintDecodeU64[0]}, } +// No generated backend here: every group table is the reference slot +// alone, so the walk has nothing to check. +var allGroupTables = map[string][]any{} + // No generated backend here: the numeric half of the inventory walk // has nothing to check and skips on nil. var archSets func() map[string]kernel.Set diff --git a/dispatch_tables_loong64.go b/dispatch_tables_loong64.go index 961cdf2..e9f795e 100644 --- a/dispatch_tables_loong64.go +++ b/dispatch_tables_loong64.go @@ -650,6 +650,7 @@ var opsF32LASX = kernel.Ops[float32]{ Correlate: loong64.CorrelateFloat32LASX, Tile: loong64.TileFloat32LASX, Gather: loong64.GatherFloat32LASX, + MovingAverage: loong64.MovingAverageFloat32LASX, ShiftDiv: loong64.ShiftDivFloat32LASX, LayerNorm: loong64.LayerNormFloat32LASX, Random: loong64.RandomF32LASX, @@ -746,6 +747,7 @@ var opsF64LASX = kernel.Ops[float64]{ Correlate: loong64.CorrelateFloat64LASX, Tile: loong64.TileFloat64LASX, Gather: loong64.GatherFloat64LASX, + MovingAverage: loong64.MovingAverageFloat64LASX, ShiftDiv: loong64.ShiftDivFloat64LASX, LayerNorm: loong64.LayerNormFloat64LASX, Random: loong64.RandomF64LASX, @@ -1308,3 +1310,55 @@ var opsU8ByTier = [...]*kernel.Ops[uint8]{ nil, &opsU8LASX, } + +var cplxC128LASX = kernel.Complex[complex128]{ + Add: loong64.CaddComplex128LASX, + Sub: loong64.CsubComplex128LASX, + Neg: loong64.CnegComplex128LASX, + Sum: loong64.CsumComplex128LASX, + Dot: loong64.CdotComplex128LASX, + DotConj: loong64.CdotconjComplex128LASX, +} + +var cplxC128ByTier = [...]*kernel.Complex[complex128]{ + nil, + &cplxC128LASX, +} + +var cplxC64LASX = kernel.Complex[complex64]{ + Add: loong64.CaddComplex64LASX, + Sub: loong64.CsubComplex64LASX, + Neg: loong64.CnegComplex64LASX, + Sum: loong64.CsumComplex64LASX, + Dot: loong64.CdotComplex64LASX, + DotConj: loong64.CdotconjComplex64LASX, +} + +var cplxC64ByTier = [...]*kernel.Complex[complex64]{ + nil, + &cplxC64LASX, +} + +var partsC128PartsLASX = kernel.ComplexParts[complex128, float64]{ + Abs: loong64.CabsComplex128LASX, + Real: loong64.CrealComplex128LASX, + Imag: loong64.CimagComplex128LASX, + Scale: loong64.CscaleComplex128LASX, + FromParts: loong64.CfromPartsComplex128LASX, +} + +var partsC128PartsByTier = [...]*kernel.ComplexParts[complex128, float64]{ + nil, + &partsC128PartsLASX, +} + +var partsC64PartsLASX = kernel.ComplexParts[complex64, float32]{ + Real: loong64.CrealComplex64LASX, + Imag: loong64.CimagComplex64LASX, + Scale: loong64.CscaleComplex64LASX, +} + +var partsC64PartsByTier = [...]*kernel.ComplexParts[complex64, float32]{ + nil, + &partsC64PartsLASX, +} diff --git a/dispatch_tables_loong64_test.go b/dispatch_tables_loong64_test.go index acfef7c..a6b31d5 100644 --- a/dispatch_tables_loong64_test.go +++ b/dispatch_tables_loong64_test.go @@ -127,5 +127,25 @@ var allFlatTables = map[string][]any{ "Bytes.VarintDecodeU64": {tblBytesVarintDecodeU64[0], tblBytesVarintDecodeU64[1]}, } +// allGroupTables enumerates the per-tier partial structs the runtime +// overlays from, keyed by kernel.Set group. Slot 0 is the reference +// and is nil; the rest are one per accelerated tier, in tier order. +var allGroupTables = map[string][]any{ + "F32": {nil, opsF32LASX}, + "F64": {nil, opsF64LASX}, + "I16": {nil, opsI16LASX}, + "I32": {nil, opsI32LASX}, + "I64": {nil, opsI64LASX}, + "I8": {nil, opsI8LASX}, + "U16": {nil, opsU16LASX}, + "U32": {nil, opsU32LASX}, + "U64": {nil, opsU64LASX}, + "U8": {nil, opsU8LASX}, + "C128": {nil, cplxC128LASX}, + "C64": {nil, cplxC64LASX}, + "C128Parts": {nil, partsC128PartsLASX}, + "C64Parts": {nil, partsC64PartsLASX}, +} + // archSets hands the tests every tier's complete kernel set. var archSets func() map[string]kernel.Set = loong64.Sets diff --git a/dispatch_tables_ppc64le.go b/dispatch_tables_ppc64le.go index 35129af..0d75800 100644 --- a/dispatch_tables_ppc64le.go +++ b/dispatch_tables_ppc64le.go @@ -1201,3 +1201,59 @@ var opsU8ByTier = [...]*kernel.Ops[uint8]{ nil, &opsU8VSX, } + +var cplxC128VSX = kernel.Complex[complex128]{ + Add: ppc64le.CaddComplex128VSX, + Sub: ppc64le.CsubComplex128VSX, + Mul: ppc64le.CmulComplex128VSX, + Neg: ppc64le.CnegComplex128VSX, + Conj: ppc64le.CconjComplex128VSX, + Sum: ppc64le.CsumComplex128VSX, + Dot: ppc64le.CdotComplex128VSX, + DotConj: ppc64le.CdotconjComplex128VSX, +} + +var cplxC128ByTier = [...]*kernel.Complex[complex128]{ + nil, + &cplxC128VSX, +} + +var cplxC64VSX = kernel.Complex[complex64]{ + Add: ppc64le.CaddComplex64VSX, + Sub: ppc64le.CsubComplex64VSX, + Mul: ppc64le.CmulComplex64VSX, + Div: ppc64le.CdivComplex64VSX, + Neg: ppc64le.CnegComplex64VSX, + Conj: ppc64le.CconjComplex64VSX, +} + +var cplxC64ByTier = [...]*kernel.Complex[complex64]{ + nil, + &cplxC64VSX, +} + +var partsC128PartsVSX = kernel.ComplexParts[complex128, float64]{ + Abs: ppc64le.CabsComplex128VSX, + Real: ppc64le.CrealComplex128VSX, + Imag: ppc64le.CimagComplex128VSX, + Scale: ppc64le.CscaleComplex128VSX, + FromParts: ppc64le.CfromPartsComplex128VSX, +} + +var partsC128PartsByTier = [...]*kernel.ComplexParts[complex128, float64]{ + nil, + &partsC128PartsVSX, +} + +var partsC64PartsVSX = kernel.ComplexParts[complex64, float32]{ + Abs: ppc64le.CabsComplex64VSX, + Real: ppc64le.CrealComplex64VSX, + Imag: ppc64le.CimagComplex64VSX, + Scale: ppc64le.CscaleComplex64VSX, + FromParts: ppc64le.CfromPartsComplex64VSX, +} + +var partsC64PartsByTier = [...]*kernel.ComplexParts[complex64, float32]{ + nil, + &partsC64PartsVSX, +} diff --git a/dispatch_tables_ppc64le_test.go b/dispatch_tables_ppc64le_test.go index 5528db9..d2e53ec 100644 --- a/dispatch_tables_ppc64le_test.go +++ b/dispatch_tables_ppc64le_test.go @@ -127,5 +127,25 @@ var allFlatTables = map[string][]any{ "Bytes.VarintDecodeU64": {tblBytesVarintDecodeU64[0], tblBytesVarintDecodeU64[1]}, } +// allGroupTables enumerates the per-tier partial structs the runtime +// overlays from, keyed by kernel.Set group. Slot 0 is the reference +// and is nil; the rest are one per accelerated tier, in tier order. +var allGroupTables = map[string][]any{ + "F32": {nil, opsF32VSX}, + "F64": {nil, opsF64VSX}, + "I16": {nil, opsI16VSX}, + "I32": {nil, opsI32VSX}, + "I64": {nil, opsI64VSX}, + "I8": {nil, opsI8VSX}, + "U16": {nil, opsU16VSX}, + "U32": {nil, opsU32VSX}, + "U64": {nil, opsU64VSX}, + "U8": {nil, opsU8VSX}, + "C128": {nil, cplxC128VSX}, + "C64": {nil, cplxC64VSX}, + "C128Parts": {nil, partsC128PartsVSX}, + "C64Parts": {nil, partsC64PartsVSX}, +} + // archSets hands the tests every tier's complete kernel set. var archSets func() map[string]kernel.Set = ppc64le.Sets diff --git a/dispatch_tables_riscv64.go b/dispatch_tables_riscv64.go index 24ef710..24be578 100644 --- a/dispatch_tables_riscv64.go +++ b/dispatch_tables_riscv64.go @@ -1420,3 +1420,63 @@ var opsU8ByTier = [...]*kernel.Ops[uint8]{ nil, &opsU8RVV, } + +var cplxC128RVV = kernel.Complex[complex128]{ + Add: riscv64.CaddComplex128RVV, + Sub: riscv64.CsubComplex128RVV, + Mul: riscv64.CmulComplex128RVV, + Div: riscv64.CdivComplex128RVV, + Neg: riscv64.CnegComplex128RVV, + Conj: riscv64.CconjComplex128RVV, + Sum: riscv64.CsumComplex128RVV, + Dot: riscv64.CdotComplex128RVV, + DotConj: riscv64.CdotconjComplex128RVV, +} + +var cplxC128ByTier = [...]*kernel.Complex[complex128]{ + nil, + &cplxC128RVV, +} + +var cplxC64RVV = kernel.Complex[complex64]{ + Add: riscv64.CaddComplex64RVV, + Sub: riscv64.CsubComplex64RVV, + Mul: riscv64.CmulComplex64RVV, + Div: riscv64.CdivComplex64RVV, + Neg: riscv64.CnegComplex64RVV, + Conj: riscv64.CconjComplex64RVV, + Sum: riscv64.CsumComplex64RVV, + Dot: riscv64.CdotComplex64RVV, + DotConj: riscv64.CdotconjComplex64RVV, +} + +var cplxC64ByTier = [...]*kernel.Complex[complex64]{ + nil, + &cplxC64RVV, +} + +var partsC128PartsRVV = kernel.ComplexParts[complex128, float64]{ + Abs: riscv64.CabsComplex128RVV, + Real: riscv64.CrealComplex128RVV, + Imag: riscv64.CimagComplex128RVV, + Scale: riscv64.CscaleComplex128RVV, + FromParts: riscv64.CfromPartsComplex128RVV, +} + +var partsC128PartsByTier = [...]*kernel.ComplexParts[complex128, float64]{ + nil, + &partsC128PartsRVV, +} + +var partsC64PartsRVV = kernel.ComplexParts[complex64, float32]{ + Abs: riscv64.CabsComplex64RVV, + Real: riscv64.CrealComplex64RVV, + Imag: riscv64.CimagComplex64RVV, + Scale: riscv64.CscaleComplex64RVV, + FromParts: riscv64.CfromPartsComplex64RVV, +} + +var partsC64PartsByTier = [...]*kernel.ComplexParts[complex64, float32]{ + nil, + &partsC64PartsRVV, +} diff --git a/dispatch_tables_riscv64_test.go b/dispatch_tables_riscv64_test.go index 1d02dc7..d549421 100644 --- a/dispatch_tables_riscv64_test.go +++ b/dispatch_tables_riscv64_test.go @@ -127,5 +127,25 @@ var allFlatTables = map[string][]any{ "Bytes.VarintDecodeU64": {tblBytesVarintDecodeU64[0], tblBytesVarintDecodeU64[1]}, } +// allGroupTables enumerates the per-tier partial structs the runtime +// overlays from, keyed by kernel.Set group. Slot 0 is the reference +// and is nil; the rest are one per accelerated tier, in tier order. +var allGroupTables = map[string][]any{ + "F32": {nil, opsF32RVV}, + "F64": {nil, opsF64RVV}, + "I16": {nil, opsI16RVV}, + "I32": {nil, opsI32RVV}, + "I64": {nil, opsI64RVV}, + "I8": {nil, opsI8RVV}, + "U16": {nil, opsU16RVV}, + "U32": {nil, opsU32RVV}, + "U64": {nil, opsU64RVV}, + "U8": {nil, opsU8RVV}, + "C128": {nil, cplxC128RVV}, + "C64": {nil, cplxC64RVV}, + "C128Parts": {nil, partsC128PartsRVV}, + "C64Parts": {nil, partsC64PartsRVV}, +} + // archSets hands the tests every tier's complete kernel set. var archSets func() map[string]kernel.Set = riscv64.Sets diff --git a/dispatch_tables_s390x.go b/dispatch_tables_s390x.go index ef3d9f8..a99a780 100644 --- a/dispatch_tables_s390x.go +++ b/dispatch_tables_s390x.go @@ -1022,3 +1022,58 @@ var opsU8ByTier = [...]*kernel.Ops[uint8]{ nil, &opsU8VX, } + +var cplxC128VX = kernel.Complex[complex128]{ + Add: s390x.CaddComplex128VX, + Sub: s390x.CsubComplex128VX, + Mul: s390x.CmulComplex128VX, + Div: s390x.CdivComplex128VX, + Neg: s390x.CnegComplex128VX, + Sum: s390x.CsumComplex128VX, + Dot: s390x.CdotComplex128VX, + DotConj: s390x.CdotconjComplex128VX, +} + +var cplxC128ByTier = [...]*kernel.Complex[complex128]{ + nil, + &cplxC128VX, +} + +var cplxC64VX = kernel.Complex[complex64]{ + Add: s390x.CaddComplex64VX, + Sub: s390x.CsubComplex64VX, + Mul: s390x.CmulComplex64VX, + Div: s390x.CdivComplex64VX, + Neg: s390x.CnegComplex64VX, + Conj: s390x.CconjComplex64VX, + Sum: s390x.CsumComplex64VX, +} + +var cplxC64ByTier = [...]*kernel.Complex[complex64]{ + nil, + &cplxC64VX, +} + +var partsC128PartsVX = kernel.ComplexParts[complex128, float64]{ + Abs: s390x.CabsComplex128VX, + Real: s390x.CrealComplex128VX, + Imag: s390x.CimagComplex128VX, + Scale: s390x.CscaleComplex128VX, +} + +var partsC128PartsByTier = [...]*kernel.ComplexParts[complex128, float64]{ + nil, + &partsC128PartsVX, +} + +var partsC64PartsVX = kernel.ComplexParts[complex64, float32]{ + Abs: s390x.CabsComplex64VX, + Real: s390x.CrealComplex64VX, + Imag: s390x.CimagComplex64VX, + Scale: s390x.CscaleComplex64VX, +} + +var partsC64PartsByTier = [...]*kernel.ComplexParts[complex64, float32]{ + nil, + &partsC64PartsVX, +} diff --git a/dispatch_tables_s390x_test.go b/dispatch_tables_s390x_test.go index b1b03e4..bd7ddda 100644 --- a/dispatch_tables_s390x_test.go +++ b/dispatch_tables_s390x_test.go @@ -127,5 +127,25 @@ var allFlatTables = map[string][]any{ "Bytes.VarintDecodeU64": {tblBytesVarintDecodeU64[0], tblBytesVarintDecodeU64[1]}, } +// allGroupTables enumerates the per-tier partial structs the runtime +// overlays from, keyed by kernel.Set group. Slot 0 is the reference +// and is nil; the rest are one per accelerated tier, in tier order. +var allGroupTables = map[string][]any{ + "F32": {nil, opsF32VX}, + "F64": {nil, opsF64VX}, + "I16": {nil, opsI16VX}, + "I32": {nil, opsI32VX}, + "I64": {nil, opsI64VX}, + "I8": {nil, opsI8VX}, + "U16": {nil, opsU16VX}, + "U32": {nil, opsU32VX}, + "U64": {nil, opsU64VX}, + "U8": {nil, opsU8VX}, + "C128": {nil, cplxC128VX}, + "C64": {nil, cplxC64VX}, + "C128Parts": {nil, partsC128PartsVX}, + "C64Parts": {nil, partsC64PartsVX}, +} + // archSets hands the tests every tier's complete kernel set. var archSets func() map[string]kernel.Set = s390x.Sets diff --git a/docs/architecture.md b/docs/architecture.md index 6735e8e..adcee0d 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -63,7 +63,9 @@ caller reference with a tier's partial `Ops`, built lazily on first use (`opsCache` in `dispatch.go`). The only type switch lives there, so the public wrappers stay one-liners and a caller that never touches a type - never links its kernels. + never links its kernels. The complex groups take the same route through + `groupCache[G]`, which is generic over the struct because `kernel.Complex` + and `kernel.ComplexParts` are not `kernel.Ops`. - The reference itself (`refBase`) is a few hundred kilobytes of ordinary Go and is linked into every consumer regardless; the per-tier assembly is what the tables keep out. diff --git a/docs/assets/badges/release.svg b/docs/assets/badges/release.svg index 9eeab0f..731d9a8 100644 --- a/docs/assets/badges/release.svg +++ b/docs/assets/badges/release.svg @@ -1 +1 @@ -release: v1.20.0releasev1.20.0 \ No newline at end of file +release: v1.21.0releasev1.21.0 \ No newline at end of file diff --git a/docs/lld/kernels-and-dispatch.md b/docs/lld/kernels-and-dispatch.md index ea3b3a3..a328d56 100644 --- a/docs/lld/kernels-and-dispatch.md +++ b/docs/lld/kernels-and-dispatch.md @@ -44,6 +44,15 @@ iterating `simdinfo -tiers`. selects the per-element-type cache, which overlays the reference with the tier's partial `Ops` struct lazily on first use (one `reflect` pass per element type, never on the hot path). +- The **complex groups** use the same shape through `complexOps[C]()` and + `complexParts[C, R]()`, over `groupCache[G]` rather than `opsCache[T]`: + `opsCache` is constrained to `Number`, and neither `kernel.Complex` nor + `kernel.ComplexParts` is a `kernel.Ops`, so the cache is generic over the + struct instead of over the element type and merges with `overlayAny`. The + per-tier tables are `cplxByTier` and `partsByTier`. This + route did not exist until 2026-08-14: before it, every complex operation + ran the portable reference on every machine while the generated complex + assembly sat linked into the test-only aggregator (`docs/wrong.md`). - A **generated guard** sits between the table and the kernel. It checks the per-operation element threshold (`KernelThreshold` is the public view; the tables are generated from the same manifest and a test holds the two diff --git a/docs/platforms.md b/docs/platforms.md index 230c239..053667a 100644 --- a/docs/platforms.md +++ b/docs/platforms.md @@ -48,7 +48,7 @@ same portable implementations used for per-operation fallback. ## Kernel coverage The default build exposes 493 exported functions and the repository contains -6,931 generated kernels across nine accelerated tier targets. Counts below are +6,933 generated kernels across nine accelerated tier targets. Counts below are logical callable kernels, not the extra `Body` trampolines required by the s390x ABI. `make check-emission` regenerates the inventory as a dry run. @@ -61,7 +61,7 @@ emitted for AVX2 and declined for SSE2 appears once in each column. | amd64 (sse2/avx2/avx512) | 2609 | 101 | LLVM declined to vectorize for a tier | | arm64 (neon/sve2) | 1681 | 127 | LLVM declined to vectorize for a tier | | riscv64 (rvv) | 891 | 15 | LLVM declined to vectorize | -| loong64 (lasx) | 728 | 174 | `$fp` ownership and unresolved `.L0` references | +| loong64 (lasx) | 730 | 172 | `$fp` ownership and unresolved `.L0` references | | ppc64le (vsx) | 602 | 300 | `r30`/`r2`, `r0`, frame writes, then LLVM refusals | | s390x (vx) | 420 | 482 | `r13`, which the Go runtime owns | diff --git a/docs/research/07-nary-closure.md b/docs/research/07-nary-closure.md new file mode 100644 index 0000000..685d8cb --- /dev/null +++ b/docs/research/07-nary-closure.md @@ -0,0 +1,103 @@ +# A general n-ary combinator taking a Go closure + +**Status: measured, design fixed, not implemented.** This record exists so +the implementation step has a decision behind it rather than a preference. +Nothing in it is shipped; ROADMAP.md's n-ary section is where the open item +lives. + +## The question + +`AddAll` and `MulAll` ship, backed by arity-3 and arity-4 kernels with +folding beyond. The open half of that roadmap item is a general combinator: + +```go +simd.CombineInto(dst, f, a, b, c) // dst[i] = f(a[i], b[i], c[i]) +``` + +One function that takes an arbitrary Go closure and applies it elementwise. +The roadmap already states the constraint — a closure call per element +defeats vectorization — so the only design question worth measuring is how +much it defeats it, and whether the answer leaves room for a useful API. + +## The measurement + +A `//go:noinline` loop calling a closure per element, against the identical +loop with the body written in. `dst[i] = a[i]*2 + b[i]`, float64, on the +Zen 5. Instructions retired, slope between `-benchtime 2000x` and `10000x` +at n=65536, so the per-call setup cancels: + +| loop | instructions/element | +|---|---| +| closure per element | 17.998 | +| body written in | 7.032 | + +**11.0 extra instructions per element, a 2.56x ratio.** Instructions +retired are layout- and load-independent, which matters here: the machine +was not quiet. + +Wall clock on the same runs, minimum of three at `-benchtime 2000x` — a +weaker number, quoted only because it is the one a caller feels: + +| n | closure | written in | ratio | +|---|---|---|---| +| 64 | 76.06 ns | 15.03 ns | 5.06x | +| 1024 | 1027 ns | 210.4 ns | 4.88x | +| 65536 | 67391 ns | 14581 ns | 4.62x | + +The wall-clock ratio is worse than the instruction ratio because the +closure loop also serialises: each call is a dependency the scheduler +cannot hoist, so the extra instructions do not overlap. + +The two ratios bracket the real cost. Against a *kernel* rather than +against a written-in scalar loop the gap is larger still, because the +kernel is the thing the closure makes impossible. + +## What that rules out + +A combinator whose contract is "give me any closure and I will make it +fast" cannot be built. At 11 extra instructions per element it is slower +than the scalar loop the caller would have written, and far slower than +chaining the binary calls that already exist. Shipping it would be an API +whose name promises the opposite of what it measures. + +## The design that survives + +The same shape `FilterInto` already uses in this library: **dispatch the +shapes that have kernels, and be explicit that the rest is a scalar loop.** + +1. **A closed set of named operations dispatches to real kernels.** Not a + closure — a value from an enumerated type, so the compiler and the + reader both know the set is finite: + + ```go + simd.CombineInto(dst, simd.OpAdd, a, b, c) + ``` + + `OpAdd` and `OpMul` reach `Add3`/`Add4`/`Mul3`/`Mul4` and the existing + fold. This is `AddAll`/`MulAll` with the operation as a parameter, and + it is worth having only if a caller genuinely needs to choose at run + time; otherwise `AddAll` is clearer at the call site. + +2. **The closure form is a separate function with the cost in its name and + its doc.** Something like `CombineFuncInto`, documented as a scalar + loop, with the measured 11 instructions per element in the comment so + nobody has to rediscover it. It exists for the case where the caller has + an operation this library does not have and wants one pass over the data + rather than materialising intermediates — which is a real want, and is + about memory traffic, not about vector width. + +3. **No heuristic that silently switches between them.** A combinator that + inspected the closure and sometimes used a kernel would be a guess about + what the closure does. The two functions do different things and say so. + +The measurement gate for step 2: `CombineFuncInto` ships only if it beats +the binary chain it replaces at a size where memory traffic dominates — the +same gate `AddAll` passed. If it does not beat it anywhere, it does not +ship, and the number goes in `docs/wrong.md`. + +## What is not decided here + +The exact names and signatures. Those follow the API conventions in +`docs/api.md` at implementation time, and the enumerated-operation form may +turn out not to be worth its own function at all — in which case only the +closure form ships, or neither does. diff --git a/docs/research/08-goexperiment-simd-small-n.md b/docs/research/08-goexperiment-simd-small-n.md new file mode 100644 index 0000000..b0449ff --- /dev/null +++ b/docs/research/08-goexperiment-simd-small-n.md @@ -0,0 +1,107 @@ +# `GOEXPERIMENT=simd` for the small-n band + +**Status: measured favourable in isolation, then measured AGAIN as built and +found NEGATIVE. See docs/wrong.md entry 75 before using anything below.** + +The table in this document is a direct, inlinable loop. Put behind a package +boundary -- which is what a shared fallback for every generated guard has to be +-- the same arithmetic costs **+25.6%** instructions at n=8, **+111.6%** at +n=9, and **+83.7%** for float64 at n=8, because the loop cannot be inlined into +the guard while the one-line `ref` forward can. The sizes below are all exact +multiples of the vector width, which hides the tail cost entirely. + +What survives: the bit-identity claim, which was re-verified at every length +0-40 over IEEE specials for add, sub, mul and div, on both lanes. + +**Original status: measured, favourable, not implemented.** This record exists so the +implementation step has a number behind it rather than a table copied from a +plan. Nothing in it is shipped; the open item lives in ROADMAP.md's tiers +section. + +## The question + +Every generated guard falls back to `internal/ref` below its threshold: + +```go +func addF32AVX512Guarded(dst, a, b []float32) { + if len(dst) < threshold { ref.AddInto(dst, a, b); return } + addF32AVX512(dst, a, b) +} +``` + +`ref` is a plain Go loop, and below the threshold that loop is the whole answer. +The proposal is to route the guard's fallback through Go's own `simd/archsimd` +instead — inlinable intrinsics rather than a non-inlinable assembly call, which +is what makes them usable at n = 8 where a call costs more than the work. + +Two things had to be true before any of it is worth writing: the intrinsics have +to be **bit-identical** to the portable loop, and they have to actually **win** +at the sizes where the fallback runs. Both are measurable today — Go 1.26.5 ships +`simd/archsimd` behind `GOEXPERIMENT=simd` — so they were measured before +anything was built. + +## Bit-identity + +Float addition is not reassociated by either path (elementwise, one operation +per lane), so identity is expected rather than hoped for. Checked anyway, over +n = 0, 1, 3, 7, 8, 9, 15, 16, 17, 31, 32, 33 with values chosen to be +cancellation-prone (`i*0.1 + 1e-8` against `i*-0.3 + 1e30`), including every +tail length the partial-load helpers have to handle: + +**Identical at every index, every n.** `LoadFloat32x8SlicePart` / +`StoreSlicePart` handle the tail without reading or writing past the slice, which +is the part a hand-written version gets wrong. + +## The measurement + +`f32 dst[i] = a[i] + b[i]`, the scalar loop against an `archsimd` 8-lane loop +plus a partial-load tail. Instructions retired and cycles, `perf stat -e +instructions:u,cycles:u`, three interleaved rounds in one session, 2,000,000 +iterations each, minimum reported. Instructions rather than wall clock because +the machine's load average was above 1 and this repository's wall-clock noise +floor is 8.3%. + +| n | scalar instr | archsimd instr | | scalar cycles | archsimd cycles | | +|---|---|---|---|---|---|---| +| 8 | 227,775,443 | 183,973,765 | **−19.2%** | 38,225,341 | 33,269,977 | −13.0% | +| 16 | 387,771,780 | 284,004,004 | **−26.8%** | 59,050,260 | 46,597,720 | −21.1% | +| 32 | 708,189,014 | 483,774,777 | **−31.7%** | 103,791,526 | 64,829,258 | −37.5% | + +Every range is disjoint: the slowest archsimd round beats the fastest scalar +round at all three sizes. The gain grows with n across the band, which is what a +vector width of 8 predicts — at n = 8 one vector operation replaces eight scalar +ones but the loop scaffolding is a larger share; by n = 32 it is four vectors +against thirty-two. + +## What this does NOT measure + +Stated because the number above is the tempting one to quote: + +- **It is a direct call, not the guarded one.** The library's fallback is reached + through a dispatch guard, and the guard's own compare and branch are not in + either column. The gain should carry — the guard is identical on both sides — + but the end-to-end figure will be smaller than 19–32%. +- **One operation.** `AddInto` is the friendliest possible case: elementwise, + one instruction per lane, no reassociation question. The reductions are the + interesting ones and they are the ones where bit-identity is genuinely at + risk, because `kernel.CombineTree` uses a fixed sixteen-accumulator shape that + an intrinsics loop would have to reproduce exactly rather than merely + correctly. +- **amd64 only, here.** `archsimd` builds for amd64 and arm64; riscv64, s390x, + loong64 and ppc64le keep the `ref` fallback and gain nothing. Any claim made + from this table has to say so. +- **The build tag is viral.** Behind `//go:build goexperiment.simd`, every lane + has to run twice — tagged and untagged — or the tagged path becomes the + vacuously-green lane `docs/wrong.md` entry 41 warned about. That cost is + ongoing and is not in any of these numbers. + +## The decision this supports + +Build it, elementwise operations first, with bit-identity as a gate rather than +an expectation: an operation whose intrinsics path is not bit-identical to `ref` +does not ship for that operation. Reductions are a separate decision needing +their own measurement. + +The reproducer is thirty lines and lives in this document's history rather than +in the tree, because a benchmark for a package that does not exist yet is a +maintenance cost with no reader. diff --git a/docs/verification.md b/docs/verification.md index 141fa7c..e2c1c5f 100644 --- a/docs/verification.md +++ b/docs/verification.md @@ -17,7 +17,7 @@ complete list. Command roles come from the Makefile, the CI workflow | `make test-vec` | `GOEXPERIMENT=simd go test` | the amd64-only vector type compiles and passes; skipped where the toolchain lacks the experiment | | `make test-tiers` | one `go test` per tier the CPU supports (`GOSIMD=`) | catches a kernel correct on AVX-512 and wrong on SSE2 | | `make test-race` | `go test -race` | full suite under the race detector | -| `make fuzz` | `FuzzDifferential` (public API) and `FuzzKernelsAgainstReference` (every kernel of every tier vs the reference), `FUZZTIME` seconds each | adversarial bit patterns: signalling NaNs, denormals, exponent boundaries | +| `make fuzz` | every `Fuzz*` in the tree, discovered per package, `FUZZTIME` seconds each — today `FuzzDifferential` (public API), `FuzzKernelsAgainstReference` (every kernel of every tier vs the reference) and `FuzzJSONMasksMatchesSeparateCalls` | adversarial bit patterns: signalling NaNs, denormals, exponent boundaries | | `make test-cross` | arm64, s390x, ppc64le under docker + qemu, hermetic (`--network=none`, `CGO_ENABLED=0`, `-vet=off`, `-short`), `simdinfo -require-accelerated` first | the correctness half of the non-amd64 backends; three bugs shipped past a green amd64 suite and only appeared here — two memory corruptions (a kernel clobbering the goroutine register, kernels writing into the caller's frame through a save area) and a reference that computed different bits on architectures where Go fuses a multiply into an add | | `make test-riscv64` / `make test-loong64` | qemu-user lanes (recent emulators, explicit `-cpu`), `-require-accelerated` | the two architectures the docker lane cannot reach | | `make test-gates` | riscv64 with **no** vector extension, no acceleration assertion | the fallback must still pass; the SIGILL mis-gate case | diff --git a/docs/wrong.md b/docs/wrong.md index f6ffe9c..a0fa3af 100644 --- a/docs/wrong.md +++ b/docs/wrong.md @@ -3043,3 +3043,621 @@ element-scratch allocation, but duplicate-skew recovery can allocate one mask for each equal run it extracts. The active API documentation now states that exception. Historical changelog text remains a record of the implementation at the time of each release. + +## Every complex kernel was generated, tested, and dispatched to by nothing + +**Believed.** The complex surface -- `DotComplex`, `DotComplexConj`, +`ScaleComplex`, `SumComplex`, the arithmetic set, and the ComplexParts half +that produces real slices -- runs on the generated assembly like the rest of +the library. nine `internal//complex_*.s` files exist -- sse2, avx2, avx512, neon, +sve2, rvv, vsx, vx, lasx, across six architectures -- `registerComplex` +installs them, and `TestInventoryCoversEveryGroup` asserts that `C64`, +`C128`, `C64Parts` and `C128Parts` all carry declared kernels. All of that +was true. + +**Actually.** None of it was reachable from a consumer. `complexOps[C]()` +returned `&refBase.C64` / `&refBase.C128` *directly*, with no per-tier +overlay, and `complexParts` did the same: + +```go +case complex128: + return any(&refBase.C128).(*kernel.Complex[C]) // complex.go:33, before +``` + +The dispatch tables the runtime indexes carried no complex entry at all -- +`grep -c Complex dispatch_tables_amd64.go` was **0**, against ten +`opsXXByTier` tables for the real element types -- because +`tools/simdgen/emit/dispatch_gen.go` had a `numericElem` map and no complex +equivalent, so `numericGroupsIn` skipped every complex group and emitted +nothing for them. The generated complex assembly was reachable only from +`internal//sets_gen_.go`, whose own generated header says "Tests +call this; consumers never do". + +The test that should have caught it is `TestDeclaredKernelsAreWired`, not +`TestInventoryCoversEveryGroup` -- the first version of this entry named the +wrong one. `TestDeclaredKernelsAreWired` checks the runtime tables +(`allFlatTables`) for exactly three groups, `Bytes`, `Convert` and `Mask`, +and sends everything else -- numeric AND complex -- through `archSets()`, +the test-only aggregator. So the numeric groups have the same nominal hole; +they were saved by having real tables, not by being checked. Nothing walked +the tables the runtime actually uses for a non-flat group. + +Confirmed by disassembly, which is the only thing that would have shown it. +The symbol survives only where the call is not inlined -- the root package's +own test binary inlines it and prints nothing, so this is the +`internal/benchmarks` binary: + +``` +$ go test -c -o /tmp/bench.test ./internal/benchmarks +$ go tool objdump -s 'simd\.DotComplex\[' /tmp/bench.test + complex.go:31 LEAQ github.com/sebishogun/simd.refBase+13560(SB), AX +``` + +Empirically, from a consumer module calling `simd.DotComplex`: 0 complex +kernel symbols linked before, 24 after. A consumer calling only `simd.Sum` +links 0 in both, so the per-operation dead-code elimination survives. + +**How it surfaced.** Not from this repository. A downstream evaluation +(simdblas Task 4, complex Level 1 against gonum) measured `Zdotu`/`Zdotc` +2.2x-3.1x slower than gonum at n <= 100000, 1.65x-1.86x at n = 1e6. Not +against "a plain Go loop", as the first version of this entry said: gonum's +unit-increment complex dot calls `c128.DotuUnitary`/`DotcUnitary`, which are +hand-written AVX assembly on amd64. Losing to it is not by itself proof of +anything. The tell was that `GOSIMD=scalar` and the detected tier gave +identical timings. + +**Cost, measured.** `GOSIMD=scalar` against `avx512` in one binary -- +one build, so the 8.3% layout floor does not apply; there is no second +layout to differ. Complete grid, minimum of six alternating passes, +reproduced twice. **amd64/avx512 only**: no other architecture was measured, +and the per-tier ratios elsewhere are not these. + +The machine was at load 5-12 for this, against the project rule of load +under 1. The wall-clock column is therefore the weaker evidence; the +instruction column below it is layout- and load-independent and is what the +conclusion rests on. + +| operation | scalar ns/op | avx512 ns/op | ratio | +|---|---|---|---| +| Dot c128 n=1024 | 749.7 | 101.5 | 7.39x | +| Dot c128 n=65536 | 47156 | 15459 | 3.05x | +| Dot c128 n=1048576 | 805434 | 408702 | 1.97x | +| DotConj c128 n=1024 | 890.1 | 102.5 | 8.68x | +| DotConj c128 n=65536 | 55570 | 15488 | 3.59x | +| DotConj c64 n=1024 | 1249 | 97.31 | 12.8x | +| DotConj c64 n=65536 | 83114 | 6614 | 12.6x | +| DotConj c64 n=1048576 | 1276125 | 158120 | 8.07x | +| Sum c128 n=1024 | 470.1 | 43.43 | 10.8x | +| Sum c128 n=65536 | 29002 | 5611 | 5.17x | +| Sum c128 n=1048576 | 480519 | 115280 | 4.17x | +| DotConj c128 n=1048576 | 918445 | 420455 | 2.18x | + +Instructions retired per element, two-point slope so setup cancels, which +is the number that does not move with load: + +| operation | scalar | avx512 | ratio | +|---|---|---|---| +| Sum c128 n=1048576 | 12577259 | 1040671 | 12.1x | +| Dot c128 n=1048576 | 32070536 | 2627180 | 12.2x | +| DotConj c64 n=1048576 | 44563734 | 2030498 | 22.0x | + +(The three `DotConj/c128` cells have no instruction row: `go test -bench` +cannot exclude the deeper `impl=naive` sub-benchmark, so selecting them +also runs it and its instructions scale with `-benchtime` too.) + +**When, exactly.** Not "since the kernels were written" -- the first version +of this entry said that and it is wrong. The kernels landed in `ba09d24` and +dispatched correctly through `active.C64`/`active.C128`. The break came with +`02c258a`, "Per-operation dispatch: the linker keeps only what a program +calls", which replaced `active` with `refBase` plus per-tier tables and +emitted no complex tables: + +``` +$ git show v1.13.0:complex.go | grep 'active\.C64' -> return any(&active.C64)... +$ git show v1.14.0:complex.go | grep 'refBase\.C64' -> return any(&refBase.C64)... +$ git tag --contains 02c258a --sort=creatordate | head -1 +v1.14.0 +``` + +So the affected releases are **v1.14.0 through v1.20.0** -- seven tags. A +consumer pinned to v1.13.0 or earlier was never affected. The table +mechanism itself arrived in that commit, so the missing `complexElem` was +never an omission from an older design; it was a group the new design did +not carry over. + +**The repository's own baseline held the evidence.** `testdata/bench/amd64.txt` +was recorded at `f6c72f9`, before the regression, with +`BenchmarkComplexReduce/Dot/c128/n=1024-8` at 101.2-101.9 ns/op -- a +benchmark that calls the public `simd.DotComplex`. The portable path for +that shape is 749.7 ns/op, 7x against `make bench-check`'s 25% threshold. +Whether the gate was run at `02c258a` is not recoverable from here; the +baseline was never updated, which is why it is still pre-regression and +still a working detector. + +**Fix.** `dispatch_gen.go` gained `complexElem`/`partsElem` and an +`emitTierTable` that writes `cplxByTier` and `partsByTier` +alongside the existing `opsByTier`; `dispatch.go` gained a +`groupCache[G any]` (opsCache is constrained to `Number`, and neither +`kernel.Complex` nor `kernel.ComplexParts` is `kernel.Ops`) with an +`overlayAny`; `complexOps`/`complexParts` now route through it. The +regenerated tables are **additive only** -- 0 deleted lines across all seven +`dispatch_tables_*.go`, 479 added. + +`dispatch_complex_test.go` is the test that would have caught it: it compares +the function pointer `DotComplex` actually reaches against the reference +set's, which is the question a caller asks and the inventory walk does not. + +## The three-way partition may already be built, and the 34% has not been re-measured + +ROADMAP.md records one losing case for `Sort`: few distinct values at 16384, +by 34% against `slices.Sort`, with "**the fix is a three-way partition**" and a +note that it "needs a second kernel". `docs/plans/2026-08-13-simd-production.md` +Task 1 carries the same plan. + +Reading `sort.go` before writing the kernel: `extractEqual` already does the +three-way split. When the split comes back skewed past `sortSkewLimit`, it +takes the run equal to the pivot out of the recursion entirely — which is what +a below/equal/above partition buys — and it does so with kernels this package +already ships (`EqualScalarInto`, `CountTrue`, `NotMask`, `CompressInto`), so +no second kernel was needed for that part. Its own comment says so. + +That does not close the task, and the reason is that nothing here has been +measured on a quiet machine. Two attempts: + + load 2.2-2.5, min of 3 x 300: simd 28,531 ns slices 23,000 ns 1.24x + load 12.0, min of 5 x 400: simd 117,359 ns slices 79,546 ns noise + +The second set is unusable — the same benchmark moved by 4x between them, and +three review agents were running. The first set is not quiet either. What can +be said: the gap at that shape is real and still there, and 1.24x is not 1.34x. +What cannot be said: whether it is 24%, whether the difference from 34% is the +`extractEqual` work landing, or whether either number survives a quiet machine. + +So no number in ROADMAP.md was changed. Overwriting a measured claim with a +worse-conditioned one is not a correction. What the task needs first is the +measurement under load average below 1, and then a profile — because if the +residual is the `copy(a, scratch[:len(a)])` after every partition rather than +the split itself, a three-way kernel does not address it and would be built for +a cause nobody checked. + +## A forgotten reference wiring shipped past the whole normal-lane suite + +**Probed, 2026-08-15.** `Sqrt: sqrt[T]` deleted from `floatOps` in +`internal/ref/ref.go` — one entry, the shape of a wiring somebody forgets when +adding a kernel. + +| lane | result | +|---|---| +| `TestDeclaredKernelsAreWired`, normal | **PASS** (verified with `-v` that it ran) | +| `TestDeclaredKernelsAreWired`, purego | **PASS** | +| whole suite, normal — all 14 packages | **PASS** | +| whole suite, purego | FAIL, `TestInPlaceMatchesInto` | + +So the completeness gate that exists for exactly this did not see it, and the +only thing that did was a functional test in a lane nobody runs by default. + +The cause is scope. `TestDeclaredKernelsAreWired` walks `backend.Inventory` +against the dispatch tables and `archSets()` — the GENERATED side. `ref.Set()` +is never in the subject, and its doc comment says so plainly: "The subject is +`active` ... and not `ref.Set()`. That is not a shortcut, it is the only +correct choice", because ref leaves every Fast slot nil until a backend +registers. That reasoning is right about Fast slots and was taken to cover the +whole reference. + +The reference is not optional. It is the live fallback on architectures with +no backend, in `purego` builds, and below every kernel's element threshold — +so a nil there is a nil call in the small-n path of an operation that works on +a big slice. + +`TestDeclaredKernelsAreWiredInTheReference` is the other half: every +non-`Fast` entry in `backend.Inventory` must be non-nil in `refBase`. 853 +today. The `Fast` exclusion is the same one the older test's comment describes, +for the same reason. + +**Also settled, and against a plan.** `docs/plans/…-simd-production.md`'s item +1 proposes generating the `internal/ref` ops-table entry and the +`kernel.Ops` struct field from the manifest, and justifies it as removing "the +two easiest things to forget — one of which caused the `-tags purego` panic +that `TestDispatchTableComplete` now guards". Half of that is wrong: a +forgotten `kernel.Ops` FIELD cannot ship, because the generator emits code +referencing it and the build fails; a forgotten `ref` WIRING shipped past +fourteen packages, as above. The gate is the fix; codegen would be +convenience, and it would move the numerical contract — which lives in the +`kernel.Ops` field comments — into a manifest. + +## `make fuzz` fuzzed one of its three targets and exited 0 + +Measured 2026-08-16, by sweeping the family for fuzz targets nothing runs. + +The recipe named two targets and the comment above it opened "Two fuzz +targets". There are three, and only ONE of them was being fuzzed: + +| target | what happened | +|---|---| +| `FuzzKernelsAgainstReference` | fuzzed, correctly | +| `FuzzDifferential` | named, and run in the ROOT package while the target lives in `internal/tests/arrays` | +| `FuzzJSONMasksMatchesSeparateCalls` | never named; only its seed corpus ever ran, under `go test` | + +The middle row is the one worth the entry: + +``` +$ go test -run '^$' -fuzz FuzzDifferential -fuzztime 3s . +PASS +ok github.com/sebishogun/simd 0.002s +EXIT=0 +``` + +**`go test -fuzz X` in a package with no target called X exits 0.** It does not +error, it does not warn, it prints `PASS`. So half of `make fuzz` had been +reporting success for a step that fuzzed nothing, and the only tell was a +duration of two milliseconds where sixty seconds were asked for. + +Targets are discovered per package now — `go test -list '^Fuzz'` asks the +compiler, and a target can only be fuzzed in the package it was found in, so +the wrong-package shape is unreachable rather than merely fixed. The recipe +also fails outright if it ends up running nothing at all, because a discovery +loop over an empty list is the same silent green one level up. + +**The same sweep, across the family.** Ten repositories. `simdlogs` already +discovered its targets and its workflow says why: "A hand-maintained list is +how a new target gets written and never run." `simdhttp` hand-listed three of +four and is fixed (its own entries 15 and 16 — the missing one had been named +in a verification document since v1.2.0 and never written). `simdcsv` (3 +targets) and `simdparquet` (13) have no fuzz recipe at all; `simdjson` names 11 +of 14. Those are recorded here and not yet fixed. + +**The shape.** A gate that cannot fail is this family's most persistent defect, +and every instance so far has been a test or an assertion. This is the same +thing at the build level: a Makefile recipe whose command succeeds while doing +nothing, in the repository the others depend on. + +## 75. The archsimd small-n win depends on inlining, and a package cannot inline + +`docs/research/08-goexperiment-simd-small-n.md` measured `GOEXPERIMENT=simd` +intrinsics against the portable loop at −19.2% (n=8), −26.8% (16) and −31.7% +(32) instructions, and concluded: build it, elementwise first. The plan put the +implementation in a new `internal/fastpath` package that every generated guard +would call below its threshold. + +Built, bit-identity-gated, and measured. `perf stat -e instructions:u`, +2,000,000 iterations, interleaved, three rounds at n=8 and two at the rest, +disjoint ranges throughout: + +| | ref | archsimd | | +|---|---|---|---| +| f32 n=8 | 124.06M | 155.85M | **+25.6%** | +| f32 n=9 | 136.12M | 288.08M | **+111.6%** | +| f32 n=16 | 220.20M | 228.13M | **+3.6%** | +| f32 n=17 | 232.14M | 360.57M | **+55.3%** | +| f32 n=32 | 412.03M | 371.91M | −9.7% | +| f64 n=8 | 124.17M | 228.07M | **+83.7%** | +| f64 n=16 | 220.05M | 372.34M | **+69.2%** | + +It loses across the whole band the guards actually use, and wins only at f32 +n≥32 — above where any guard falls back. + +**The cause is inlining, and the research document had already named the risk +without measuring it.** `-gcflags=-m`: + +``` +fastpath_ref.go:16:6: can inline AddFloat32 <- the one-line forward to ref +fastpath_simd.go:38:36: inlining call to archsimd.LoadFloat32x8Slice + (no "can inline AddFloat32") <- the loop cannot +``` + +The portable fallback is a one-line forward, so the guard inlines it and pays +nothing to reach it. The archsimd version is a loop, and Go does not inline a +function containing one — so it is a real call, and at n=8 that call is most of +the work. The record's reproducer was a direct inlined loop and said so: *"It is +a direct call, not the guarded one … the end-to-end figure will be smaller than +19–32%."* Smaller was the wrong word. It is negative. + +**The record's table measured only exact multiples of the vector width, and that +hid the worst of it.** n=9 and n=17 — one full vector plus a one-element tail, +which is the shape the guard band mostly sees — cost +111.6% and +55.3%. A table +of 8/16/32 cannot show a tail cost. + +**float64 loses everywhere in the band**, and for a structural reason: 4 lanes +means n=8 is two vectors and n=16 is four, so the call is amortised over half as +much arithmetic as f32 at the same length. The research document measured f32 +only. + +Reverted. What the next attempt has to do differently is emit the intrinsics +**inside the generated backend**, in the same package as the guard, where they +can inline into it — not behind a package boundary. Until something does that, +the measured answer for the guard band is that `internal/ref` is the faster +fallback. + +The bit-identity gate that came with it did pass, at every length 0–40 over +IEEE specials, on both lanes: elementwise archsimd IS bit-identical to `ref`, +which was the other thing that had to be true. That half of the record stands. + +## 76. Forced vectorization: the bucket the pragma is for is the bucket the contract forbids + +The plan's item 5 proposed trying `#pragma clang loop vectorize(enable)` per +(kernel, target) and keeping measured winners, over "346 refusals: ~60 +`scatter_*`, ~30 `reverse_*`, 114 float/`fast_*` — the plausible wins". +Measured, before writing any pragma. Every number in that sentence is wrong, +and the investigation ends somewhere else. + +**The inventory.** `make check-emission`, clang 22.1.8: **336** (kernel, tier) +refusals over **132** distinct kernels, not 346. + +| tier | vsx | neon | sse2 | vx | sve2 | lasx | rvv | avx2 | avx512 | +|---|---|---|---|---|---|---|---|---|---| +| refusals | 81 | 80 | 56 | 36 | 33 | 23 | 14 | 10 | 3 | + +By operation, as (kernel, tier) pairs: `scatter` 65, `gather` 31, `reverse` 30, +`tile` 10, `reversebits` 10 — the permute/scatter family is **162 of 336, 48%**, +and each needs an instruction the target does not have. `gather` at 31 pairs is +larger than `reverse` and the plan did not mention it. + +**There is no float bucket.** `fast_*` refusals: **zero**. Float-typed +refusals: 59 of 336, and on the three amd64 tiers only **2** — both +`f8e*_to_f32`, which are integer bit manipulation. The refused +`prod`/`dot`/`sumsq*`/`sumsqdev`/`sumsqdiff` entries are all `i64`/`u64`: +64-bit integer reductions, blocked by a missing multiply (SSE2 has no +`pmullq`), not by anything a pragma reaches. + +**The pragma's own bucket is ten loops.** `-Rpass-analysis=loop-vectorize` over +all of `csrc` at sse2, by reason: + +| reason | loops | +|---|---| +| instruction return type cannot be vectorized | 1929 | +| loop control flow is not understood | 186 | +| value that could not be identified as reduction is used outside the loop | 128 | +| potentially faulting load | 58 | +| store instruction cannot be vectorized | 47 | +| **cannot prove it is safe to reorder floating-point operations** | **10** | + +Ten. And clang's own remark names the remedy: *"allow reordering by specifying +'#pragma clang loop vectorize(enable)' before the loop or by providing the +compiler option '-ffast-math'"*. The pragma there is not a cost-model override, +it is **permission to reassociate** — which changes the answer. The numerical +contract in `internal/kernel/kernel.go` is the reason `-ffp-contract=off` is in +`commonFlags` at all, and the repository already has the correct home for a +kernel that trades it: the `simd_fast_*` prefix. So the one bucket the pragma is +designed for is the one bucket that cannot take it without becoming a different +operation. + +The ten are `csrc/numeric.c:121,127,133,139` (the `CONVOLVE` accumulation, four +instantiations), `csrc/numeric.c:399,404`, and `csrc/gemm.c:142` (`GEMM`, four). + +**And those same kernels ship scalar today.** `simd_convolve_f32`, +`simd_correlate_f64` and their siblings are refused on **zero** tiers — they +pass the emission gate — yet: + +| tier | simd_convolve_f32 | packed arithmetic | scalar arithmetic | +|---|---|---|---| +| sse2 | 119 instrs | **0** | 10 (`mulss`, `addss`) | +| avx2 | 140 instrs | **0** | 10 (`vmulss`, `vaddss`) | +| avx512 | 141 instrs | **0** | 10 | + +Its only vector instructions on sse2 are 2 `movups` and 2 `xorps` — a move and +a register zero. `dispatch_tables_amd64.go` carries +`Convolve: amd64.ConvolveFloat32SSE2` and `ConvolveFloat32AVX2`, so a caller on +amd64 dispatches into an assembly kernel that multiplies one element at a time, +paying the dispatch and threshold machinery for it. + +**The gate cannot see this, structurally.** `verify.vectorWidth` returns >0 for +any amd64 instruction whose operands mention `%xmm`/`%ymm`/`%zmm` — and every +scalar float instruction on x86-64 uses those registers. `vmulss %xmm0, %xmm1, +%xmm2` counts as vectorized. So `RequireVector`, whose comment says it exists +because "dispatching to it would run scalar code under a name promising +otherwise", cannot report a scalar float kernel on amd64 at all. Its 69 amd64 +refusals are integer kernels, necessarily. + +**Counted properly, by the generator.** The first pass here was a shell scan +over every symbol in each object, which included helpers that are not kernels +and used a cruder mnemonic match; it reported 26 at sse2 and 8 at avx2. Those +numbers are superseded. `verify.arithKind` now classifies each instruction as +packed arithmetic, scalar arithmetic or neither — moves and the `xorps +%xmm0,%xmm0` zeroing idiom are neither, which is the point — and `make +check-emission` prints `SCALAR-ONLY` for any kernel that does arithmetic with +none of it in lanes: + +| tier | instances | +|---|---| +| amd64/avx512 | 9 | +| amd64/sse2 | 8 | +| amd64/avx2 | 8 | +| arm64/sve2 | 8 | +| arm64/neon | 5 | + +**38 instances over 13 distinct kernels**: `polyeval`, `convolve`, `correlate`, +`movavg`, `minr` (f32 and f64 each), `rolling_min_f64`, `rolling_max_f64`, +`dtoa_f64`. It is **not an x86 quirk** — arm64 has 13 of the 38, and there the +gate is blind for the same reason in a different spelling: `fmul s0, s1, s2` is +scalar and `fmul v0.4s, v1.4s, v2.4s` is packed, and `vectorWidth` reads the +register file, not the arrangement. + +The classifier runs on amd64 and arm64 only, and `Report.ArithKnown` says so: +on the other four architectures a zero count means "not measured", and +`ScalarOnly()` returns false there rather than reporting a finding it did not +make. + +**What this leaves.** Not a pragma sweep. Two separate things: + +1. `vectorWidth` on amd64 should not count a scalar-form instruction (`*ss`, + `*sd`) as vector, and `RequireVector` should ask "does this kernel do + arithmetic, and is none of it packed" rather than "is any register wide". + A permute kernel does no arithmetic and must still pass. +2. Fixing the gate DROPS the eight, falling back to `internal/ref`. Whether + that is faster than the scalar assembly is unmeasured: both are scalar, and + the assembly costs a call the guard would otherwise inline. That is the + benchmark this needs, and the machine was not quiet (load average 1.17, two + review agents building) when this was written, so no number is offered. + +The transferable part is the same one as entry 25: a gate that asks a proxy +question answers the proxy question. "Contains an instruction that touches a +vector register" is not "vectorized", and on the one architecture that runs +natively here the two are never the same for float code. + +## 77. Blocking the outer loop is what vectorizes a reduction kernel, and it is bit-identical + +Entry 76 found `polyeval`, `convolve`, `correlate` and `movavg` shipping scalar +on every tier while passing the emission gate, and left the question of what to +do about it open. The answer is not a pragma and not a fallback: it is the loop +nest. + +**LLVM's loop vectorizer only ever vectorizes the INNERMOST loop.** Every one of +these kernels is an outer loop over independent outputs wrapped around an inner +reduction with a runtime trip count: + +```c +for (isize i = 0; i < n; i++) { // independent — but not innermost + T acc = c[nc - 1]; + for (isize k = nc - 2; k >= 0; k--) // innermost — a serial dependency + acc = acc * xv + c[k]; + d[i] = acc; +} +``` + +So the vectorizer looked at the Horner recurrence, could not break it without +reassociating, and gave up — and `polyeval`'s own comment claimed the opposite: +"The outer loop over elements is independent, which is what vectorizes." + +**Blocking swaps which loop is innermost without touching any element's +arithmetic.** A fixed-size accumulator array, elements in tiles of 16: + +```c +for (; i + 16 <= n; i += 16) { + T acc[16]; + for (b) acc[b] = c[nc - 1]; + for (k) { T ck = c[k]; for (b) acc[b] = acc[b] * x[i + b] + ck; } + for (b) d[i + b] = acc[b]; +} +``` + +Every output still evaluates the same coefficients in the same order, so this is +**bit-identical**, not close. That distinction is the whole point: reassociating +is exactly what `#pragma clang loop vectorize(enable)` would have done and what +the numerical contract forbids. Nothing is allocated — 16 doubles is 128 bytes +against the 512-byte kernel frame budget — and the tail runs the original nest. + +Packed arithmetic instructions per symbol, from `go run ./simdgen -n -v -only +csrc/numeric.c` in `tools/`, run twice: once against `git show +c1d08c4:csrc/numeric.c` and once against the blocked file. + +| | before, every amd64 tier | sse2 | avx2 | avx512 | +|---|---|---|---|---| +| `simd_polyeval_f32` | 0 | 24 | 20 | 18 | +| `simd_polyeval_f64` | 0 | 16 | 40 | 20 | +| `simd_convolve_f32` | 0 | 8 | 20 | 12 | +| `simd_convolve_f64` | 0 | 16 | 24 | 21 | +| `simd_correlate_f32` | 0 | 8 | 20 | 20 | +| `simd_correlate_f64` | 0 | 16 | 24 | 21 | +| `simd_movavg_f32` | 0 | 16 | 20 | 10 | +| `simd_movavg_f64` | 0 | 16 | 24 | 20 | + +The scalar count does **not** move: 10 at sse2 and 18 at avx2/avx512, identical +before and after on all eight symbols. So those instructions are not a tail the +blocking introduced — they were already there, and blocking added packed work +beside them rather than converting scalar work into it. `movavg` at avx512 f32 +gets only 10 packed against 20 at avx2 because a 16-element f32 block is exactly +one zmm. + +A first version of this entry gave an avx2 column that was wrong in all six rows +it listed and called the scalar remainder a tail. Both came from reading one +tier's numbers and generalising. + +`SCALAR-ONLY` went from **13 kernels to 5** (`dtoa_f64`, `minr_f32/f64`, +`rolling_min_f64`, `rolling_max_f64`). And loong64 gained two kernels, 728 → +730: `simd_movavg_f32` and `simd_movavg_f64` were refused there with "LLVM did +not vectorize it for lasx" and now emit, so the total moved 6,931 → 6,933. + +**The bit-identity claim is tested, and the test was checked for being +vacuous.** `internal/conformance` compares each tier against `ref` at eighteen +length pairs (`lenPairs`, `internal/conformance/numeric_test.go:23`), eight of +which have a destination at or above the guard's threshold of 16 — so +the SIMD path is genuinely reached. Poisoning the blocked path only +(`acc[b] = acc[b]*x[i+b] + ck + 1`) reddens it at `dst=16 x=16 coeffs=2 i=0`, +the first blocked case, which is what says the comparison sees the new code +rather than the fallback. + +**Not measured: whether it is faster.** It executes fewer instructions per +element and the arithmetic is in lanes, but the machine was not quiet (load +average 2.27, two review agents building) and no wall-clock or counter number +is offered. What is established is that the kernels now do their arithmetic in +lanes and produce the same bits, which is the precondition for the speed +question rather than an answer to it. + +**A review sweep put the bit-identity claim far past what those eighteen pairs +reach.** Every `nd` in 0..200 crossed with 22 other lengths and with `nd±1` and +`nd+15/16/17`; `nc`/`nk`/`w` in `{-1,0,1,2,3,4,15,16,17,31,33}`; seven value +generators per precision including catastrophic cancellation (`1e16, 1, -1e16, +1`), mid-accumulation overflow (`MaxFloat, MaxFloat, -MaxFloat`, where +left-to-right gives +Inf and any other grouping gives a finite number), +magnitudes switching at every 16-element block boundary, ±0 and subnormals, and +NaN payloads. **3,286,542 cases per tier, 3,027,710 of them reaching the kernel +rather than `ref`, on all nine tiers — sse2/avx2/avx512 native and +neon/sve2/rvv/lasx/vsx/vx under qemu-user. Zero differing bits.** Perturbing +`ref` by +1 only at output index ≥ 16 reddens every comparison site +(197k–291k mismatches per operation), so the sweep is not vacuous. A separate +`PROT_NONE` guard-page probe placing dst, signal and kernel each so its last +element ends on the last accessible byte — ~88k kernel calls per tier — faulted +nowhere, and the mechanism was checked by reading one element past the page. + +**arm64 pays for this, and the cost is not measured.** LLVM did not vectorize +the block loop there; it unrolled over `b` and vectorized over `j` instead, so +each accumulator gets a packed multiply of 4 taps and then an in-order scalar +horizontal reduction. That needs 16 live scalar accumulators plus vector +temporaries, and the tap loop of `simd_correlate_f32` at neon carries 18 spill +stores and 18 reloads per iteration: + +| tier | kernel | stack | scalar float arith | +|---|---|---|---| +| neon | `simd_correlate_f32` | 0B → **320B** | 10 → **82** | +| neon | `simd_convolve_f32` | 0B → **272B** | 10 → **82** | +| neon | `simd_convolve_f64` | 0B → 80B | 6 → **70** | +| neon | `simd_correlate_f64` | 0B → 80B | 6 → **70** | +| sve2 | `simd_movavg_f64` | 0B → 144B | 2 → **34** | + +All are inside the 512-byte frame budget so nothing is dropped, and the results +are bit-identical — the reduction stays strictly in order. But six arm64 kernels +went from a zero frame to 80–320 bytes with 7–13x the scalar float arithmetic, +and this repository has no arm64 hardware lane. Whether the packed tap multiply +pays for the spill traffic is unknown on the only architecture where the +question is open. + +`movavg` also rules out the obvious alternative: a sliding window (subtract the +element leaving, add the one arriving) is O(n) instead of O(n·w) and gives a +DIFFERENT number, because floating-point addition does not cancel. + +## 78. Generating the ref/kernel wiring is now maintainability, not correctness + +The plan's extensibility item proposed generating two of the eight files a new +kernel touches -- `internal/ref/ref.go`'s ops-table entry and +`internal/kernel/kernel.go`'s struct field -- on the argument that they are +"the two easiest things to forget, one of which caused the `-tags purego` panic +that `TestDispatchTableComplete` now guards". + +Both halves of that failure are now caught by a gate, in both lanes. Measured, +`Sqrt: sqrt[T]` deleted from `floatOps`: + + normal purego + before TestDeclaredKernelsAreWiredInTheReference green red + (and red only through TestInPlaceMatchesInto, + a functional test, not a completeness gate) + after RED RED + +`TestDeclaredKernelsAreWiredInTheReference` walks `backend.Inventory` and, for +every declared kernel, asserts `kernel.Set` has the group AND the field AND +that the field is non-nil in `ref.Set()`. That is exactly the three-way +relation generation was proposed to enforce: manifest -> struct field -> +reference wiring. A forgotten entry cannot ship past the ordinary `go test` +lane any more. + +So the item is not dropped, it is REPRICED. What generation still buys is two +fewer files to edit per kernel and one less place for a rename to go stale; +what it no longer buys is protection against a class of defect that has +already shipped once. That matters for ordering: it is a refactor of a stable +v1 contract across six architectures, needing `make codegen`, `make verify` and +the whole cross-architecture matrix to land safely, and its remaining value is +ergonomic. It goes behind anything that fixes an answer. + +The mutation is recorded here rather than left implicit because the plan's own +justification for the item no longer holds, and the next person to read that +plan would otherwise re-derive the priority from a sentence that was true when +it was written. diff --git a/internal/amd64/numeric_avx2_amd64.s b/internal/amd64/numeric_avx2_amd64.s index 3433906..dbf1ffc 100644 --- a/internal/amd64/numeric_avx2_amd64.s +++ b/internal/amd64/numeric_avx2_amd64.s @@ -195,24 +195,102 @@ TEXT ·polyEvalFloat32AVX2(SB), NOSPLIT|NOFRAME, $0-72 MOVQ x_len+32(FP), R8 MOVQ coeffs_len+56(FP), R9 QUAD $0x4dc14c0f4cc1394c - QUAD $0x000000f7840fc985 - QUAD $0x0002338e0fc0854d - QUAD $0xfc8a44107aa1c400 - QUAD $0x0f02e88348c8894c - QUAD $0x894855000000f08c - QUAD $0x8d41f8e4834853e5 - QUAD $0xe1834107e183ff49 - QUAD $0x2e661cebd2314507 - QUAD $0x0000000000841f0f - QUAD $0xff499714117aa1c4 - QUAD $0x00c8840fc2394dc2 - QUAD $0x960c107aa1c40000 - QUAD $0x41d028f8c5c38949 - QUAD $0xcb89481d7401f983 - QUAD $0x90d028f8c5c38949 + QUAD $0xe58948551a74c985 + QUAD $0xe483485356415741 + QUAD $0x31217d10f88349f8 + QUAD $0x854d00000269e9c0 + QUAD $0x49000004938e0fc0 + QUAD $0x000159830f04f883 + QUAD $0x00000473e9c03100 + QUAD $0x49fc8a44107aa1c4 + QUAD $0x00011a8e0f01f983 + QUAD $0x518d45fe498d4900 + QUAD $0x8341c0187de2c4ff + QUAD $0xe38341cb894503e2 + QUAD $0x3100000010be4103 + QUAD $0x2e6666666628ebdb + QUAD $0x0000000000841f0f + QUAD $0x11fcc59f2411fcc5 + QUAD $0x4810708d4c209f5c + QUAD $0xf48f0fc6394dc389 + QUAD $0xfcc5f0894c000001 + QUAD $0x9e5410fcc59e0c10 + QUAD $0xe028fcc5ce894920 + QUAD $0x01fb8341d828fcc5 + QUAD $0xce8949d7894d3474 + QUAD $0xd828fcc5e028fcc5 + QUAD $0x0000000000841f0f + QUAD $0x187da2c4e159dcc5 + QUAD $0xe4c5e458d4c5b22c + QUAD $0xff49db58d4c5da59 + QUAD $0x8348e275cfff49ce + QUAD $0x04c683498c7203f9 + QUAD $0x0000000000841f0f + QUAD $0x187da2c4e159dcc5 + QUAD $0xc5e458d4c5f0b26c + QUAD $0xc5db58d4c5da59e4 + QUAD $0x6c187da2c4e159dc + QUAD $0xe4c5e458d4c5ecb2 + QUAD $0xdcc5db58d4c5da59 + QUAD $0xb26c187da2c4e159 + QUAD $0x59e4c5e458d4c5e8 + QUAD $0x59dcc5db58d4c5da + QUAD $0xe4b26c187da2c4e1 + QUAD $0xda59e4c5e458d4c5 + QUAD $0xfcc68349db58d4c5 + QUAD $0x15e99a7f03fe8349 + QUAD $0x49f0408d49ffffff + QUAD $0xff4904eac149c289 + QUAD $0x4807e183d18944c2 + QUAD $0x10ba41287370f883 + QUAD $0x00c8e9c031000000 + QUAD $0xffffffe0b9480000 + QUAD $0x20f883497fffffff + QUAD $0xc031000001ec830f + QUAD $0xe2834900000226e9 + QUAD $0xc031c8187de2c4f8 + QUAD $0x0000000000841f0f + QUAD $0x11fcc5870c11fcc5 + QUAD $0x874c11fcc520874c + QUAD $0xc560874c11fcc540 + QUAD $0x00000080878c11fc + QUAD $0x0000a0878c11fcc5 + QUAD $0x00c0878c11fcc500 + QUAD $0xe0878c11fcc50000 + QUAD $0x878c11fcc5000000 + QUAD $0x8c11fcc500000100 + QUAD $0x11fcc50000012087 + QUAD $0xfcc500000140878c + QUAD $0xc500000160878c11 + QUAD $0x00000180878c11fc + QUAD $0x0001a0878c11fcc5 + QUAD $0x01c0878c11fcc500 + QUAD $0xe0878c11fcc50000 + QUAD $0x4980e88348000001 + QUAD $0xffff6f850ff8c283 + QUAD $0x8d4c3974c98548ff + QUAD $0xc1c0187de2c41050 + QUAD $0x2e666666666604e1 + QUAD $0x0000000000841f0f + QUAD $0x11fcc5870411fcc5 + QUAD $0x8349d0894c208744 + QUAD $0xe875f0c1834810c2 + QUAD $0x4cd0894cf0c28349 + QUAD $0x028e0fc12948c189 + QUAD $0x44107aa1c4000002 + QUAD $0xea8349ca894dfc8a + QUAD $0x41000000c88c0f02 + QUAD $0x834107e183ff498d + QUAD $0x00401f0f15eb07e1 + QUAD $0xc0ff48871411fac5 + QUAD $0x0001cc840fc0394c + QUAD $0x894d860c10fac500 + QUAD $0xf98341d028f8c5d3 + QUAD $0x894dcb89481f7401 + QUAD $0x001f0fd028f8c5d3 QUAD $0x586aa1c4d259f2c5 QUAD $0xcbff48cbff499a14 - QUAD $0xb87207f88348ee75 + QUAD $0xb87207fa8349ee75 QUAD $0x00401f0f08c38349 QUAD $0x586aa1c4d259f2c5 QUAD $0xc4d259f2c5e09a54 @@ -226,48 +304,51 @@ TEXT ·polyEvalFloat32AVX2(SB), NOSPLIT|NOFRAME, $0-72 QUAD $0x59f2c5c89a54586a QUAD $0xc49a54586aa1c4d2 QUAD $0x07fb8349f8c38349 - QUAD $0x4dffffff49e99e7f - QUAD $0x0000013c8e0fc085 - QUAD $0xc0311b7304f88349 - QUAD $0xf8834900000120e9 - QUAD $0x0109e9c031227304 - QUAD $0x5d5bf8658d480000 - QUAD $0xffffffffe0b948c3 - QUAD $0x7320f883497fffff - QUAD $0xe0b9485eebc0311b - QUAD $0x497fffffffffffff - QUAD $0xe9c0317b7320f883 - QUAD $0x48c0894c000000af - QUAD $0xc057f8c5d231c821 - QUAD $0x0000000000841f0f + QUAD $0x48ffffff49e99e7f + QUAD $0xc18948087304f983 + QUAD $0xf983480000010be9 + QUAD $0x00bce9d231787320 + QUAD $0xc82148c0894c0000 + QUAD $0x9066c057f8c5d231 QUAD $0x11fcc5970411fcc5 QUAD $0x974411fcc5209744 QUAD $0x4860974411fcc540 QUAD $0xe075d0394820c283 - QUAD $0x0000b3840fc03949 - QUAD $0x9a840f1cc0f64100 + QUAD $0x0000f9840fc03949 + QUAD $0xe0840f1cc0f64100 QUAD $0x8348c28948000000 QUAD $0xc0214cc889481cc1 QUAD $0x2e666666c057f8c5 QUAD $0x0000000000841f0f QUAD $0xc28348970411f8c5 - QUAD $0x77ebf275d0394804 - QUAD $0xe2c4c82148c0894c - QUAD $0x001f0fd231c8187d - QUAD $0x11fcc5970c11fcc5 - QUAD $0x974c11fcc520974c - QUAD $0x4860974c11fcc540 - QUAD $0xe075d0394820c283 - QUAD $0xc0f6414774c03949 - QUAD $0x8348c2894828741c - QUAD $0xc0214cc889481cc1 - QUAD $0x001f0fc81879e2c4 + QUAD $0xbae9f275d0394804 + QUAD $0x8348ca8948000000 + QUAD $0x48c8187de2c4e0e2 + QUAD $0x4560c6834887348d + QUAD $0x0000441f0f66c931 + QUAD $0xc4a08e4c117ca1c4 + QUAD $0xa1c4c08e4c117ca1 + QUAD $0x7ca1c4e08e4c117c + QUAD $0x4c20c183498e0c11 + QUAD $0x74d13948dc75ca39 + QUAD $0x89445d741cc1f654 + QUAD $0x48f1294803e683c6 + QUAD $0x48c81879e2c4c101 + QUAD $0xe08348c0894cc201 + QUAD $0x00000000801f0ffc QUAD $0xc28348970c11f8c5 - QUAD $0x3949f275d0394804 - QUAD $0x870411fac51974c0 - QUAD $0x8704c7f1ebc0ff48 - QUAD $0x49c0ff4800000000 - QUAD $0xc377f8c5f175c039 + QUAD $0x8548f275d0394804 + QUAD $0x2e666666661a74f6 + QUAD $0x0000000000841f0f + QUAD $0xc1ff488f0411fac5 + QUAD $0x658d48f375c83949 + QUAD $0xc55d5f415e415be8 + QUAD $0x8948d00148c377f8 + QUAD $0x00008704c7ddebc1 + QUAD $0xc03949c0ff480000 + LONG $0xf8c5f175 + BYTE $0x77 + BYTE $0xc3 RET // func convolveFloat32AVX2(dst []float32, sig []float32, ker []float32) @@ -280,97 +361,193 @@ TEXT ·convolveFloat32AVX2(SB), NOSPLIT|NOFRAME, $0-72 MOVQ ker_len+56(FP), R9 QUAD $0x56415741e5894855 QUAD $0xe483485354415541 - QUAD $0x74894838ec8348f8 - QUAD $0xc0940fc9854d0824 - QUAD $0x40c69c0f40c8294d - QUAD $0x0000029f850fc608 - QUAD $0x48c1394c01708d49 - QUAD $0x8e0ff68548f14e0f - QUAD $0x0fc9854d0000028b - QUAD $0x0c8d4a000001f18e - QUAD $0x07e08341c889458a - QUAD $0xffffffffffe0ba49 - QUAD $0x214d18ca83497fff - QUAD $0x8d4c0824448b48ca - QUAD $0x83488a048d4a1c58 - QUAD $0x311024448948fcc0 - QUAD $0xfffffffffdbf49db - QUAD $0xfffffcbc493fffff - QUAD $0xfbbd493fffffffff - QUAD $0x483fffffffffffff - QUAD $0xfffffffffffffab8 - QUAD $0xfffffffff9ba483f - QUAD $0xfffff8be493fffff - QUAD $0x7c89483fffffffff - QUAD $0x4c20247489482824 - QUAD $0x906634eb18244c89 - QUAD $0x5c8b4828247c8b48 - QUAD $0x489f0411fac53024 - QUAD $0x834904c38349c3ff - QUAD $0x4808244c894c04c1 - QUAD $0x4cf339482024748b - QUAD $0x01cd840f18244c8b - QUAD $0xc530245c89480000 - QUAD $0x7308f98349c057f8 - QUAD $0x08244c8b4cdb310f - QUAD $0x001f0f000000eae9 - QUAD $0x49db3110247c8b48 - QUAD $0xfffffffffffffeb9 - QUAD $0x2e6666666666663f + QUAD $0xc9854d38ec8348f8 + QUAD $0x20244c894cc0940f + QUAD $0x41c19c0f41c8294d + QUAD $0x00000591850fc108 + QUAD $0x48c1394c01408d49 + QUAD $0x1024448948c14e0f + QUAD $0x247c894810f88348 + QUAD $0x8948282474894830 + QUAD $0xc93145087d182454 + QUAD $0x4c8b48000002b3e9 + QUAD $0x8e8e0fc985482024 + QUAD $0x418a048d4c000001 + QUAD $0xbb4903e28341ca89 + QUAD $0x7ffffffffffffff8 + QUAD $0x48cb214904cb8349 + QUAD $0x488a048d482c5e8d + QUAD $0x0824448948fcc083 + QUAD $0xf6314500000010ba + QUAD $0xfffffffffffebf49 + QUAD $0xfffffffdbc493fff + QUAD $0xfffcbd493fffffff + QUAD $0x2feb3fffffffffff + QUAD $0x7ca1c430247c8b48 + QUAD $0x4c117ca1c4b70411 + QUAD $0xc28348d1894920b7 + QUAD $0xc6834840c3834810 + QUAD $0x24543b48ce894d40 + QUAD $0xc5000002218f0f10 + QUAD $0x7304f98348c057f8 + QUAD $0xe9c957f0c5c03117 + QUAD $0x2e666666000000b3 QUAD $0x0000000000841f0f - QUAD $0xc4e49b4c107ac1c4 - QUAD $0xf2c5e89b54107ac1 - QUAD $0x8948c158fac50f59 - QUAD $0x0c59eac5ce314cde - QUAD $0x7ac1c4c158fac5b1 - QUAD $0x4cde8948ec9b4c10 - QUAD $0xc5b10c59f2c5fe31 - QUAD $0x4c107ac1c4c158fa - QUAD $0xe6314cde8948f09b - QUAD $0x7ac1c4b10c59f2c5 - QUAD $0x4cde8948f49b5410 - QUAD $0xc5b11459eac5ee31 - QUAD $0xc4c258fac5c158fa - QUAD $0x8948f89b4c107ac1 - QUAD $0x0c59f2c5c63148de - QUAD $0x7ac1c4c158fac5b1 - QUAD $0x48de8948fc9b4c10 - QUAD $0xc5b10c59f2c5d631 - QUAD $0x0c107ac1c4c158fa - QUAD $0xc5f6314cde89489b - QUAD $0xc158fac5b10c59f2 - QUAD $0xe0c7834808c38348 - QUAD $0xffff47850fda3949 - QUAD $0x244c8b4cc0854dff - QUAD $0x48fffffec9840f08 - QUAD $0x48000000009d348d - QUAD $0x49f7294810247c8b - QUAD $0x001f0ff631991c8d - QUAD $0x59f2c5b30c10fac5 - QUAD $0xc6ff48c158fac50f - QUAD $0x75f03949fcc78348 - QUAD $0x8348fffffe92e9e7 - QUAD $0x7bebc031047304fe - QUAD $0xffffffffffe0b948 - QUAD $0x047320fe83487fff - QUAD $0x48f089483febc031 - QUAD $0xc057f8c5d231c821 + QUAD $0xc5c03108247c8b48 + QUAD $0x0000441f0fc957f0 + QUAD $0x59ecc517187de2c4 + QUAD $0xc5c358fcc5d4835c + QUAD $0xc18948f4835459ec + QUAD $0x1c187dc2c4f9314c + QUAD $0xc5d8836459e4c588 + QUAD $0xc5c458fcc5ca58f4 + QUAD $0xc18948f8835459e4 + QUAD $0x1c187dc2c4e1314c + QUAD $0xc5dc836459e4c588 + QUAD $0xc5c458fcc5ca58f4 + QUAD $0xc18948fc835459e4 + QUAD $0x1c187dc2c4e9314c + QUAD $0xc5e0836459e4c588 + QUAD $0xc5c458fcc5ca58f4 + QUAD $0xca58f4c5831459e4 + QUAD $0xf0c7834804c08348 + QUAD $0xffff77850fc33949 + QUAD $0x244c8b48d2854dff + QUAD $0x48ffffff09840f20 + QUAD $0x4c02e0c148863c8d + QUAD $0x31c1294908244c8b + QUAD $0x00000000801f0fc0 + QUAD $0x59ecc511187dc2c4 + QUAD $0xecc5c358fcc5871c + QUAD $0xca58f4c520875459 + QUAD $0x49fcc18349c0ff48 + QUAD $0xfffec7e9dc75c239 + QUAD $0x8d481024448b48ff + QUAD $0xe8c149c88949f048 + QUAD $0x83c08944c0ff4904 + QUAD $0x0e7370f9834807e0 + QUAD $0x314500000010b841 + QUAD $0x8349000000c4e9c9 + QUAD $0x57f8c5c93145f8e0 + QUAD $0x2e666666666666c0 QUAD $0x0000000000841f0f - QUAD $0x11fcc5970411fcc5 - QUAD $0x974411fcc5209744 - QUAD $0x4860974411fcc540 - QUAD $0xe075d0394820c283 - QUAD $0xc6f6403a74c63948 - QUAD $0x8348c2894828741c - QUAD $0xf02148c889481cc1 - QUAD $0x00401f0fc057f8c5 + QUAD $0xa1c48f04117ca1c4 + QUAD $0x7ca1c4208f44117c + QUAD $0x117ca1c4408f4411 + QUAD $0x84117ca1c4608f44 + QUAD $0x7ca1c4000000808f + QUAD $0xc4000000a08f8411 + QUAD $0x0000c08f84117ca1 + QUAD $0xe08f84117ca1c400 + QUAD $0x84117ca1c4000000 + QUAD $0x7ca1c4000001008f + QUAD $0xc4000001208f8411 + QUAD $0x0001408f84117ca1 + QUAD $0x608f84117ca1c400 + QUAD $0x84117ca1c4000001 + QUAD $0x7ca1c4000001808f + QUAD $0xc4000001a08f8411 + QUAD $0x0001c08f84117ca1 + QUAD $0xe08f84117ca1c400 + QUAD $0x4980e98349000001 + QUAD $0xffff5f850ff8c083 + QUAD $0x8d4d2a74c08548ff + QUAD $0x9066c057f8c51041 + QUAD $0xa1c48f04117ca1c4 + QUAD $0xc1894d208f44117c + QUAD $0x75c8ff4810c08349 + QUAD $0xc1894df0c08349e7 + QUAD $0x4c894c1024448b48 + QUAD $0x548b48c8294c0824 + QUAD $0x0f1824748b482824 + QUAD $0x4c8b48000002948e + QUAD $0xd38e0fc985482024 + QUAD $0x418e048d4c000001 + QUAD $0xbb4907e28341ca89 + QUAD $0x7ffffffffffffff8 + QUAD $0x0824448b48cb2149 + QUAD $0x1cc38348821c8d48 + QUAD $0x4c8948fc8e4c8d48 + QUAD $0x894882048d481824 + QUAD $0xfffffebe49282444 + QUAD $0xfdbf493fffffffff + QUAD $0x493fffffffffffff + QUAD $0xfffffffffffffcbc + QUAD $0xfffffffffbbd493f + QUAD $0xfffffab8483fffff + QUAD $0xf9ba483fffffffff + QUAD $0x493fffffffffffff + QUAD $0xfffffffffffff8b9 + QUAD $0x0000441f0f31eb3f + QUAD $0x748b4830244c8b48 + QUAD $0x48b10411fac50824 + QUAD $0x834804c38348c6ff + QUAD $0x2474894804282444 + QUAD $0x840f1024743b4808 + QUAD $0x247c8348000001d5 + QUAD $0xc057f8c50c730820 + QUAD $0x90000000d2e9f631 + QUAD $0x247c8b48c057f8c5 + QUAD $0x0000441f0ff63118 + QUAD $0xfac5e4b34c10fac5 + QUAD $0x0f59f2c5e8b35410 + QUAD $0x4cf18948c158fac5 + QUAD $0x880c596ac1c4f131 + QUAD $0x4c10fac5c158fac5 + QUAD $0xf9314cf18948ecb3 + QUAD $0xfac5880c5972c1c4 + QUAD $0xf0b34c10fac5c158 + QUAD $0xc1c4e1314cf18948 + QUAD $0x5410fac5880c5972 + QUAD $0xe9314cf18948f4b3 + QUAD $0xfac58814596ac1c4 + QUAD $0xfac5c258fac5c158 + QUAD $0x48f18948f8b34c10 + QUAD $0x880c5972c1c4c131 + QUAD $0x4c10fac5c158fac5 + QUAD $0xd13148f18948fcb3 + QUAD $0xfac5880c5972c1c4 + QUAD $0x48b30c10fac5c158 + QUAD $0x72c1c4c9314cf189 + QUAD $0x48c158fac5880c59 + QUAD $0x49e0c7834808c683 + QUAD $0xffffff48850ff339 + QUAD $0xfffeef840fd2854d + QUAD $0x000000b50c8d48ff + QUAD $0x294818247c8b4800 + QUAD $0x8d4828244c8b48cf + QUAD $0x00401f0fc931b134 + QUAD $0x59f2c58e0c10fac5 + QUAD $0xc1ff48c158fac50f + QUAD $0x75ca3949fcc78348 + QUAD $0x8348fffffeb2e9e7 + QUAD $0x730824448b4c04f8 + QUAD $0x00008fe9c0894c08 + QUAD $0x31047320f8834800 + QUAD $0x8348c2894843ebd2 + QUAD $0x8348870c8d4ae0e2 + QUAD $0xc057f8c5f63160c1 + QUAD $0xfcc5a0b14411fcc5 + QUAD $0x4411fcc5c0b14411 + QUAD $0x48b10411fcc5e0b1 + QUAD $0xe075f2394820c683 + QUAD $0x741ca85c74d03948 + QUAD $0x8b480824448b4c6a + QUAD $0x03e683ce8910244c + QUAD $0x014cc0014cf02948 + QUAD $0x57f8c5fce18348c2 + QUAD $0x00000000801f0fc0 QUAD $0xc28348970411f8c5 - QUAD $0x3948f275d0394804 - QUAD $0x00008704c70c74c6 - QUAD $0x48efebc0ff480000 - QUAD $0x5d415c415bd8658d - QUAD $0x77f8c55d5f415e41 - BYTE $0xc3 + QUAD $0x8548f275d1394804 + QUAD $0x2e666666661e74f6 + QUAD $0x0000000000841f0f + QUAD $0x48000000008704c7 + QUAD $0x751024443948c0ff + QUAD $0x5c415bd8658d48ef + QUAD $0xc55d5f415e415d41 + QUAD $0x0824448b48c377f8 + LONG $0xebd00148 + BYTE $0xd3 RET // func correlateFloat32AVX2(dst []float32, sig []float32, ker []float32) @@ -383,71 +560,149 @@ TEXT ·correlateFloat32AVX2(SB), NOSPLIT|NOFRAME, $0-72 MOVQ ker_len+56(FP), R9 QUAD $0x294dc0940fc9854d QUAD $0xc20841c29c0f41c8 - QUAD $0x8d49000001e9850f - QUAD $0x4e0f48c1394c0140 - QUAD $0x01d58e0fc08548c1 - QUAD $0x358e0fc9854d0000 - QUAD $0x41e5894855000001 - QUAD $0xf8e4834853564157 - QUAD $0xb84907e183c98944 - QUAD $0x7fffffffffffffe0 - QUAD $0x4cc8214d18c88349 - QUAD $0x1aebdb31451c568d - QUAD $0xff499f04117aa1c4 - QUAD $0xc6834804c28349c3 - QUAD $0x00e3840fc3394904 - QUAD $0xdb31c057f8c50000 - QUAD $0x00a2820f08f98349 - QUAD $0x0000441f0f660000 - QUAD $0xc4e49a4c107ac1c4 - QUAD $0xf2c5e89a54107ac1 + QUAD $0x485500000458850f + QUAD $0x554156415741e589 + QUAD $0x49f8e48348535441 + QUAD $0x0f48c1394c01408d + QUAD $0x077d10f88348c14e + QUAD $0x4d0000022fe9c931 + QUAD $0x000001308e0fc985 + QUAD $0x4903e08341c88945 + QUAD $0xfffffffffffff8ba + QUAD $0xca214d04ca83497f + QUAD $0x0010bf412c5e8d4c + QUAD $0xf38948f631450000 + QUAD $0x2e666666666633eb + QUAD $0x0000000000841f0f + QUAD $0xa1c4b704117ca1c4 + QUAD $0x798d4c20b74c117c + QUAD $0xc3834840c3834910 + QUAD $0x0fc73949ce894940 + QUAD $0xf9894c000001c98f + QUAD $0xc5ff3145c057f8c5 + QUAD $0x0f04f98349c957f0 + QUAD $0x001f0f0000008782 + QUAD $0x81c4ba14187da2c4 + QUAD $0x58fcc5d4bb5c596c + QUAD $0xf4bb54596c81c4c3 + QUAD $0x187da2c4ca58f4c5 + QUAD $0x5c596c81c404ba54 + QUAD $0x81c4c358fcc5d8bb + QUAD $0x58f4c5f8bb54596c + QUAD $0x08ba54187da2c4ca + QUAD $0xc5dcbb5c596c81c4 + QUAD $0x54596c81c4c358fc + QUAD $0xa2c4ca58f4c5fcbb + QUAD $0x6c81c40cba54187d + QUAD $0x596c81c4e0bb5c59 + QUAD $0xf4c5c358fcc5bb14 + QUAD $0x394d04c78349ca58 + QUAD $0x840fc0854d8575fa + QUAD $0xbb248d4effffff3c + QUAD $0x90ed3145ba3c8d4e + QUAD $0x81c4af14187d82c4 + QUAD $0x596c81c4ac1c596c + QUAD $0xc5c358fcc520ac54 + QUAD $0x394dc5ff49ca58f4 + QUAD $0xffffff08e9dd75e8 + QUAD $0x49ca8949f0488d48 + QUAD $0x8945c2ff4904eac1 + QUAD $0xf9834807e08341d0 + QUAD $0x000010ba410d7370 + QUAD $0x000000aae9c93100 + QUAD $0xf8c5c931f8e28349 + QUAD $0x0000441f0f66c057 + QUAD $0x11fcc58f0411fcc5 + QUAD $0x8f4411fcc5208f44 + QUAD $0xc5608f4411fcc540 + QUAD $0x000000808f8411fc + QUAD $0x0000a08f8411fcc5 + QUAD $0x00c08f8411fcc500 + QUAD $0xe08f8411fcc50000 + QUAD $0x8f8411fcc5000000 + QUAD $0x8411fcc500000100 + QUAD $0x11fcc5000001208f + QUAD $0xfcc5000001408f84 + QUAD $0xc5000001608f8411 + QUAD $0x000001808f8411fc + QUAD $0x0001a08f8411fcc5 + QUAD $0x01c08f8411fcc500 + QUAD $0xe08f8411fcc50000 + QUAD $0x4980e98348000001 + QUAD $0xffff6f850ff8c283 + QUAD $0x8d4c2874c0854dff + QUAD $0x9066c057f8c51051 + QUAD $0x11fcc58f0411fcc5 + QUAD $0x8349d1894c208f44 + QUAD $0x49e975c8ff4910c2 + QUAD $0x8949d1894cf0c283 + QUAD $0x01e68e0fc82949c0 + QUAD $0x2a8e0fc9854d0000 + QUAD $0x8341c88945000001 + QUAD $0xfffffff8ba4907e0 + QUAD $0x4cca214d7fffffff + QUAD $0x481cc383498e1c8d + QUAD $0x2e666624eb8e348d + QUAD $0x0000000000841f0f + QUAD $0xc1ff488f0411fac5 + QUAD $0x04c6834804c38349 + QUAD $0x000197840fc13948 + QUAD $0x49db31c057f8c500 + QUAD $0x0000a3820f08f983 + QUAD $0x00000000801f0f00 + QUAD $0xc4e49b4c107ac1c4 + QUAD $0xf2c5e89b54107ac1 QUAD $0x9a5459eac59a0c59 QUAD $0x7ac1c4c158fac504 - QUAD $0x4c59f2c5ec9a4c10 + QUAD $0x4c59f2c5ec9b4c10 QUAD $0xc1c4c258fac5089a - QUAD $0x59eac5f09a54107a + QUAD $0x59eac5f09b54107a QUAD $0xc4c158fac50c9a54 - QUAD $0xf2c5f49a4c107ac1 + QUAD $0xf2c5f49b4c107ac1 QUAD $0xc258fac5109a4c59 - QUAD $0xc5f89a54107ac1c4 + QUAD $0xc5f89b54107ac1c4 QUAD $0x58fac5149a5459ea - QUAD $0xfc9a4c107ac1c4c1 + QUAD $0xfc9b4c107ac1c4c1 QUAD $0xfac5189a4c59f2c5 - QUAD $0x9a14107ac1c4c258 + QUAD $0x9b14107ac1c4c258 QUAD $0xfac51c9a5459eac5 QUAD $0x8348c258fac5c158 - QUAD $0x6d850fd8394908c3 - QUAD $0x840fc98548ffffff + QUAD $0x6d850fda394908c3 + QUAD $0x840fc0854dffffff QUAD $0x9a348d4cffffff34 QUAD $0x66ff31459e1c8d48 QUAD $0x0000000000841f0f QUAD $0x81c4bb0c107aa1c4 QUAD $0xc158fac5be0c5972 - QUAD $0xe875f9394cc7ff49 - QUAD $0x658d48ffffff03e9 - QUAD $0xc35d5f415e415be8 - QUAD $0xc931047304f88348 - QUAD $0xffffffe0ba487eeb - QUAD $0x20f883487fffffff - QUAD $0x894843ebc9310473 - QUAD $0xf8c5f631d12148c1 - QUAD $0x2e6666666666c057 - QUAD $0x0000000000841f0f - QUAD $0x11fcc5b70411fcc5 - QUAD $0xb74411fcc520b744 - QUAD $0x4860b74411fcc540 - QUAD $0xe075f1394820c683 - QUAD $0x741ca83a74c83948 - QUAD $0x1cc28348ce894827 - QUAD $0xf8c5c12148d18948 - QUAD $0x0000441f0f66c057 - QUAD $0xc68348b70411f8c5 - QUAD $0x0aebf275f1394804 - QUAD $0x48000000008f04c7 - QUAD $0xc5f175c83948c1ff - BYTE $0xf8 - BYTE $0x77 - BYTE $0xc3 + QUAD $0xe875f8394dc7ff49 + QUAD $0xf88349ffffff03e9 + QUAD $0x95e9c88949087304 + QUAD $0x7320f88349000000 + QUAD $0xc2894c4aebd23104 + QUAD $0x8f348d48e0e28348 + QUAD $0xc5c9314560c68348 + QUAD $0x0000441f0fc057f8 + QUAD $0xc4a08e44117ca1c4 + QUAD $0xa1c4c08e44117ca1 + QUAD $0x7ca1c4e08e44117c + QUAD $0x4c20c183498e0411 + QUAD $0x74d03949dc75ca39 + QUAD $0x8963741cc0f64157 + QUAD $0x49f0294903e683c6 + QUAD $0xc18948ca0148c801 + QUAD $0xc057f8c5fce18348 + QUAD $0x0000000000841f0f + QUAD $0xc28348970411f8c5 + QUAD $0x8548f275d1394804 + QUAD $0x2e666666661d74f6 + QUAD $0x0000000000841f0f + QUAD $0x000000008704c742 + QUAD $0xf075c0394cc0ff49 + QUAD $0x415c415bd8658d48 + QUAD $0xf8c55d5f415e415d + QUAD $0xc88949d10148c377 + BYTE $0xeb + BYTE $0xd6 RET // func normFloat64AVX2(a []float64) float64 @@ -639,24 +894,118 @@ TEXT ·polyEvalFloat64AVX2(SB), NOSPLIT|NOFRAME, $0-72 MOVQ x_len+32(FP), R8 MOVQ coeffs_len+56(FP), R9 QUAD $0x4dc14c0f4cc1394c - QUAD $0x000000f7840fc985 - QUAD $0x0002348e0fc0854d - QUAD $0xf8ca44107ba1c400 - QUAD $0x0f02e88348c8894c - QUAD $0x894855000000f08c - QUAD $0x8d41f8e4834853e5 - QUAD $0xe1834107e183ff49 - QUAD $0x2e661cebd2314507 - QUAD $0x0000000000841f0f - QUAD $0xff49d714117ba1c4 - QUAD $0x00c8840fc2394dc2 - QUAD $0xd60c107ba1c40000 - QUAD $0x41d028f9c5c38949 - QUAD $0xcb89481d7401f983 - QUAD $0x90d028f9c5c38949 + QUAD $0xe58948551a74c985 + QUAD $0xe483485356415741 + QUAD $0x31217d10f88349f8 + QUAD $0x854d000002e5e9c0 + QUAD $0x49000005148e0fc0 + QUAD $0x0001d1830f04f883 + QUAD $0x000004f3e9c03100 + QUAD $0x49f8ca44107ba1c4 + QUAD $0x0001928e0f01f983 + QUAD $0x518d45fe498d4900 + QUAD $0x8341c0197de2c4ff + QUAD $0xe38341cb894503e2 + QUAD $0x3100000010be4103 + QUAD $0x2e6666666634ebdb + QUAD $0x0000000000841f0f + QUAD $0x11fdc5df3411fdc5 + QUAD $0xdf7c11fdc520df6c + QUAD $0x4c60df44117dc540 + QUAD $0x394dc3894810708d + QUAD $0x4c000002648f0fc6 + QUAD $0xc5de0c10fdc5f089 + QUAD $0x10fdc520de5410fd + QUAD $0xde6410fdc540de5c + QUAD $0xf028fdc5ce894960 + QUAD $0xf828fdc5e828fdc5 + QUAD $0x01fb8341c0287dc5 + QUAD $0xce8949d7894d4574 + QUAD $0xe828fdc5f028fdc5 + QUAD $0xc0287dc5f828fdc5 + QUAD $0x197d22c4f159cdc5 + QUAD $0xd5c5f658b5c5f20c + QUAD $0xc5c5ed58b5c5ea59 + QUAD $0x3dc5ff58b5c5fb59 + QUAD $0x49c0583541c4c459 + QUAD $0x48d175cfff49ceff + QUAD $0xffff57820f03f983 + QUAD $0x001f0f04c68349ff + QUAD $0x197d22c4f159cdc5 + QUAD $0xc5f658b5c5e0f24c + QUAD $0xc5ed58b5c5ea59d5 + QUAD $0xc5ff58b5c5fb59c5 + QUAD $0xc0583541c4c4593d + QUAD $0x197d22c4f159cdc5 + QUAD $0xc5f658b5c5d8f24c + QUAD $0xc5ed58b5c5ea59d5 + QUAD $0xc5ff58b5c5fb59c5 + QUAD $0xc0583541c4c4593d + QUAD $0x197d22c4f159cdc5 + QUAD $0xc5f658b5c5d0f24c + QUAD $0xc5ed58b5c5ea59d5 + QUAD $0xc5ff58b5c5fb59c5 + QUAD $0xc0583541c4c4593d + QUAD $0x197d22c4f159cdc5 + QUAD $0xc5f658b5c5c8f24c + QUAD $0xc5ed58b5c5ea59d5 + QUAD $0xc5ff58b5c5fb59c5 + QUAD $0xc0583541c4c4593d + QUAD $0x03fe8349fcc68349 + QUAD $0x9de9ffffff528f0f + QUAD $0x49f0408d49fffffe + QUAD $0xff4904eac149c289 + QUAD $0x4803e183d18944c2 + QUAD $0x10ba41287330f883 + QUAD $0x00c0e9c031000000 + QUAD $0xfffffff0b9480000 + QUAD $0x10f883497fffffff + QUAD $0xc031000001f4830f + QUAD $0xe283490000022ee9 + QUAD $0xc031c8197de2c4fc + QUAD $0x11fdc5c70c11fdc5 + QUAD $0xc74c11fdc520c74c + QUAD $0xc560c74c11fdc540 + QUAD $0x00000080c78c11fd + QUAD $0x0000a0c78c11fdc5 + QUAD $0x00c0c78c11fdc500 + QUAD $0xe0c78c11fdc50000 + QUAD $0xc78c11fdc5000000 + QUAD $0x8c11fdc500000100 + QUAD $0x11fdc500000120c7 + QUAD $0xfdc500000140c78c + QUAD $0xc500000160c78c11 + QUAD $0x00000180c78c11fd + QUAD $0x0001a0c78c11fdc5 + QUAD $0x01c0c78c11fdc500 + QUAD $0xe0c78c11fdc50000 + QUAD $0x4940c08348000001 + QUAD $0xffff6f850ffcc283 + QUAD $0x8d4c4574c98548ff + QUAD $0xc1c0197de2c41050 + QUAD $0x2e666666666604e1 + QUAD $0x0000000000841f0f + QUAD $0x11fdc5c70411fdc5 + QUAD $0xc74411fdc520c744 + QUAD $0x4c60c74411fdc540 + QUAD $0x834810c28349d089 + QUAD $0xf0c28349dc75f0c1 + QUAD $0x2948c1894cd0894c + QUAD $0xc4000002068e0fc1 + QUAD $0x894df8ca44107ba1 + QUAD $0xcc8c0f02ea8349ca + QUAD $0x83ff498d41000000 + QUAD $0x19eb07e1834107e1 + QUAD $0x0000000000841f0f + QUAD $0xc0ff48c71411fbc5 + QUAD $0x0001cc840fc0394c + QUAD $0x894dc60c10fbc500 + QUAD $0xf98341d028f9c5d3 + QUAD $0x894dcb89481f7401 + QUAD $0x001f0fd028f9c5d3 QUAD $0x586ba1c4d259f3c5 QUAD $0xcbff48cbff49da14 - QUAD $0xb87207f88348ee75 + QUAD $0xb87207fa8349ee75 QUAD $0x00401f0f08c38349 QUAD $0x586ba1c4d259f3c5 QUAD $0xc4d259f3c5c0da54 @@ -670,48 +1019,51 @@ TEXT ·polyEvalFloat64AVX2(SB), NOSPLIT|NOFRAME, $0-72 QUAD $0x59f3c590da54586b QUAD $0x88da54586ba1c4d2 QUAD $0x07fb8349f8c38349 - QUAD $0x4dffffff49e99e7f - QUAD $0x0000013d8e0fc085 - QUAD $0xc0311b7304f88349 - QUAD $0xf8834900000120e9 - QUAD $0x0109e9c031227304 - QUAD $0x5d5bf8658d480000 - QUAD $0xfffffffff0b948c3 - QUAD $0x7310f883497fffff - QUAD $0xf0b9485eebc0311b - QUAD $0x497fffffffffffff - QUAD $0xe9c0317b7310f883 - QUAD $0x48c0894c000000af - QUAD $0xc057f9c5d231c821 - QUAD $0x0000000000841f0f - QUAD $0x11fdc5d70411fdc5 - QUAD $0xd74411fdc520d744 - QUAD $0x4860d74411fdc540 + QUAD $0x48ffffff49e99e7f + QUAD $0xc18948087304f983 + QUAD $0xf983480000010be9 + QUAD $0x00bce9d231787310 + QUAD $0xc82148c0894c0000 + QUAD $0x9066c057f8c5d231 + QUAD $0x11fcc5d70411fcc5 + QUAD $0xd74411fcc520d744 + QUAD $0x4860d74411fcc540 QUAD $0xe075d0394810c283 - QUAD $0x0000b4840fc03949 - QUAD $0x9a840f0cc0f64100 + QUAD $0x0000fa840fc03949 + QUAD $0xe0840f0cc0f64100 QUAD $0x8348c28948000000 QUAD $0xc0214cc889480cc1 - QUAD $0x2e666666c057f9c5 + QUAD $0x2e666666c057f8c5 QUAD $0x0000000000841f0f - QUAD $0xc28348d70411fdc5 - QUAD $0x78ebf275d0394804 - QUAD $0xe2c4c82148c0894c - QUAD $0x001f0fd231c8197d - QUAD $0x11fdc5d70c11fdc5 - QUAD $0xd74c11fdc520d74c - QUAD $0x4860d74c11fdc540 - QUAD $0xe075d0394810c283 - QUAD $0xc0f6414874c03949 - QUAD $0x8348c2894828740c - QUAD $0xc0214cc889480cc1 - QUAD $0x001f0fc8197de2c4 + QUAD $0xc28348d70411fcc5 + QUAD $0xbbe9f275d0394804 + QUAD $0xe683c68944000000 + QUAD $0xc4f22948ca89480f + QUAD $0x49c1894dc8197de2 + QUAD $0x9066c28949f0e183 + QUAD $0xa1c4d70c117da1c4 + QUAD $0x7da1c420d74c117d + QUAD $0x117da1c440d74c11 + QUAD $0x4d10c2834960d74c + QUAD $0x74f68548dc75d139 + QUAD $0x89445d7204fe8354 + QUAD $0x48f1294803e683c6 + QUAD $0x48c8197de2c4c101 + QUAD $0xe08348c0894cc201 + QUAD $0x00000000801f0ffc QUAD $0xc28348d70c11fdc5 - QUAD $0x3949f275d0394804 - QUAD $0xc70411fbc51a74c0 - QUAD $0x04c748f1ebc0ff48 - QUAD $0xc0ff4800000000c7 - QUAD $0x77f8c5f075c03949 + QUAD $0x8548f275d0394804 + QUAD $0x2e666666661a74f6 + QUAD $0x0000000000841f0f + QUAD $0xc1ff48cf0411fbc5 + QUAD $0x658d48f375c83949 + QUAD $0xc55d5f415e415be8 + QUAD $0x8948d00148c377f8 + QUAD $0x00c704c748ddebc1 + QUAD $0x3949c0ff48000000 + LONG $0xc5f075c0 + BYTE $0xf8 + BYTE $0x77 BYTE $0xc3 RET @@ -725,98 +1077,214 @@ TEXT ·convolveFloat64AVX2(SB), NOSPLIT|NOFRAME, $0-72 MOVQ ker_len+56(FP), R9 QUAD $0x56415741e5894855 QUAD $0xe483485354415541 - QUAD $0x74894838ec8348f8 - QUAD $0xc0940fc9854d0824 - QUAD $0x40c69c0f40c8294d - QUAD $0x000002a0850fc608 - QUAD $0x48c1394c01708d49 - QUAD $0x8e0ff68548f14e0f - QUAD $0x0fc9854d0000028c - QUAD $0x0c8d4a000001f18e - QUAD $0x07e08341c88945ca - QUAD $0xfffffffffff0ba49 - QUAD $0x214d08ca83497fff - QUAD $0x8d4c0824448b48ca - QUAD $0x8348ca048d4a3858 - QUAD $0x311024448948f8c0 - QUAD $0xfffffffffdbf49db - QUAD $0xfffffcbc491fffff - QUAD $0xfbbd491fffffffff - QUAD $0x481fffffffffffff - QUAD $0xfffffffffffffab8 - QUAD $0xfffffffff9ba481f - QUAD $0xfffff8be491fffff - QUAD $0x7c89481fffffffff - QUAD $0x4c20247489482824 - QUAD $0x906634eb18244c89 - QUAD $0x5c8b4828247c8b48 - QUAD $0x48df0411fbc53024 - QUAD $0x834908c38349c3ff - QUAD $0x4808244c894c08c1 - QUAD $0x4cf339482024748b - QUAD $0x01ce840f18244c8b - QUAD $0xc530245c89480000 - QUAD $0x7308f98349c057f9 - QUAD $0x08244c8b4cdb310f - QUAD $0x001f0f000000eae9 - QUAD $0x49db3110247c8b48 - QUAD $0xfffffffffffffeb9 - QUAD $0x2e6666666666661f + QUAD $0x7c894838ec8348f8 + QUAD $0xc0940fc9854d1024 + QUAD $0xc8294d18244c894c + QUAD $0x0fc70840c79c0f40 + QUAD $0x408d490000063d85 + QUAD $0xc14e0f48c1394c01 + QUAD $0xf883480824448948 + QUAD $0x032ce9c931077d10 + QUAD $0x4d18246c8b4c0000 + QUAD $0x0f1024648b4ced85 + QUAD $0x048d4e0000015f8e + QUAD $0xfffffffff8ba49ea + QUAD $0x4d06ca83497fffff + QUAD $0x8d4a685e8d4cea21 + QUAD $0x10bff8c38348ea1c + QUAD $0xb848c93145000000 + QUAD $0x1ffffffffffffffe + QUAD $0x001f0f0000007ee9 + QUAD $0xc5c957f1c5f63145 + QUAD $0x4adb57e1c5d257e9 + QUAD $0xf749f7894dce3c8d + QUAD $0xc4f824197d82c4d7 + QUAD $0x5da1c4f72c595da1 + QUAD $0xc558fdc520f77459 + QUAD $0x595da1c4ce58f5c5 + QUAD $0xc4d558edc540f76c + QUAD $0xe5c560f764595da1 + QUAD $0xcc04117d81c4dc58 + QUAD $0xc420cc4c117d81c4 + QUAD $0x81c440cc54117d81 + QUAD $0x798d4860cc5c117d + QUAD $0xc9894980eb834910 + QUAD $0x638f0f08247c3b48 + QUAD $0xf9c5f98948000002 + QUAD $0x840f01fd8349c057 + QUAD $0x45df8948ffffff74 + QUAD $0xe9c5c957f1c5f631 + QUAD $0x9066db57e1c5d257 + QUAD $0x5d81c427197de2c4 + QUAD $0xc558fdc598f36c59 + QUAD $0xc5b8f36c595d81c4 + QUAD $0x6c595d81c4cd58f5 + QUAD $0x81c4d558edc5d8f3 + QUAD $0x58e5c5f8f364595d + QUAD $0xc4c73149f7894ddc + QUAD $0x5d81c4f824197d82 + QUAD $0xc558fdc5a0f36c59 + QUAD $0xc4c0f36c595d81c4 + QUAD $0xf5c5e0f374595d81 + QUAD $0x81c4d658edc5cd58 + QUAD $0xdc58e5c5f324595d + QUAD $0xf0c7834802c68349 + QUAD $0xc5f6418b75f2394d + QUAD $0xe9fffffef0850f01 + QUAD $0x24448b48ffffff26 + QUAD $0xcf8948f0488d4808 + QUAD $0x89c7ff4804efc148 + QUAD $0x70f9834807e083f8 + QUAD $0x3100000010bf0c73 + QUAD $0x83480000015ce9c9 + QUAD $0xc057f9c5c931f8e7 QUAD $0x0000000000841f0f - QUAD $0xc4c8db4c107bc1c4 - QUAD $0xf3c5d0db54107bc1 - QUAD $0x8948c158fbc50f59 - QUAD $0x0c59ebc5ce314cde - QUAD $0x7bc1c4c158fbc5f1 - QUAD $0x4cde8948d8db4c10 - QUAD $0xc5f10c59f3c5fe31 - QUAD $0x4c107bc1c4c158fb - QUAD $0xe6314cde8948e0db - QUAD $0x7bc1c4f10c59f3c5 - QUAD $0x4cde8948e8db5410 - QUAD $0xc5f11459ebc5ee31 - QUAD $0xc4c258fbc5c158fb - QUAD $0x8948f0db4c107bc1 - QUAD $0x0c59f3c5c63148de - QUAD $0x7bc1c4c158fbc5f1 - QUAD $0x48de8948f8db4c10 - QUAD $0xc5f10c59f3c5d631 - QUAD $0x0c107bc1c4c158fb - QUAD $0xc5f6314cde8948db - QUAD $0xc158fbc5f10c59f3 - QUAD $0xc0c7834808c38348 - QUAD $0xffff47850fda3949 - QUAD $0x244c8b4cc0854dff - QUAD $0x48fffffec9840f08 - QUAD $0x4800000000dd348d - QUAD $0x49f7294810247c8b - QUAD $0x001f0ff631d91c8d - QUAD $0x59f3c5f30c10fbc5 - QUAD $0xc6ff48c158fbc50f - QUAD $0x75f03949f8c78348 - QUAD $0x8348fffffe92e9e7 - QUAD $0x7bebc031047304fe - QUAD $0xfffffffffff0b948 - QUAD $0x047310fe83487fff - QUAD $0x48f089483febc031 - QUAD $0xc057f9c5d231c821 - QUAD $0x0000000000841f0f - QUAD $0x11fdc5d70411fdc5 - QUAD $0xd74411fdc520d744 - QUAD $0x4860d74411fdc540 - QUAD $0xe075d0394810c283 - QUAD $0xc6f6403b74c63948 - QUAD $0x8348c2894828740c - QUAD $0xf02148c889480cc1 - QUAD $0x00401f0fc057f9c5 - QUAD $0xc28348d70411fdc5 - QUAD $0x3948f275d0394804 - QUAD $0x00c704c7480d74c6 - QUAD $0xeeebc0ff48000000 - QUAD $0x415c415bd8658d48 - QUAD $0xf8c55d5f415e415d - BYTE $0x77 - BYTE $0xc3 + QUAD $0xc1c4cc04117dc1c4 + QUAD $0x7dc1c420cc44117d + QUAD $0x117dc1c440cc4411 + QUAD $0x84117dc1c460cc44 + QUAD $0x7dc1c400000080cc + QUAD $0xc4000000a0cc8411 + QUAD $0x0000c0cc84117dc1 + QUAD $0xe0cc84117dc1c400 + QUAD $0x84117dc1c4000000 + QUAD $0x7dc1c400000100cc + QUAD $0xc400000120cc8411 + QUAD $0x000140cc84117dc1 + QUAD $0x60cc84117dc1c400 + QUAD $0x84117dc1c4000001 + QUAD $0x7dc1c400000180cc + QUAD $0xc4000001a0cc8411 + QUAD $0x0001c0cc84117dc1 + QUAD $0xe0cc84117dc1c400 + QUAD $0x84117dc1c4000001 + QUAD $0x7dc1c400000200cc + QUAD $0xc400000220cc8411 + QUAD $0x000240cc84117dc1 + QUAD $0x60cc84117dc1c400 + QUAD $0x84117dc1c4000002 + QUAD $0x7dc1c400000280cc + QUAD $0xc4000002a0cc8411 + QUAD $0x0002c0cc84117dc1 + QUAD $0xe0cc84117dc1c400 + QUAD $0x84117dc1c4000002 + QUAD $0x7dc1c400000300cc + QUAD $0xc400000320cc8411 + QUAD $0x000340cc84117dc1 + QUAD $0x60cc84117dc1c400 + QUAD $0x84117dc1c4000003 + QUAD $0x7dc1c400000380cc + QUAD $0xc4000003a0cc8411 + QUAD $0x0003c0cc84117dc1 + QUAD $0xe0cc84117dc1c400 + QUAD $0x4880e98348000003 + QUAD $0xfffebf850ff8c783 + QUAD $0x8d483874c08548ff + QUAD $0x9066c057f9c51079 + QUAD $0xc1c4cc04117dc1c4 + QUAD $0x7dc1c420cc44117d + QUAD $0x117dc1c440cc4411 + QUAD $0x8348f9894860cc44 + QUAD $0x48d975c8ff4810c7 + QUAD $0x8b48f98948f0c783 + QUAD $0x8e0fc82948082444 + QUAD $0x247c8b48000002e6 + QUAD $0x02058e0fff854818 + QUAD $0x8941fa048d4c0000 + QUAD $0xf8bb4907e28341fa + QUAD $0x497fffffffffffff + QUAD $0x8348ce1c8d48fb21 + QUAD $0x48f8fa448d4838c3 + QUAD $0xce048d4820244489 + QUAD $0xfdbf492824448948 + QUAD $0x491fffffffffffff + QUAD $0xfffffffffffffcbc + QUAD $0xfffffffffbbd491f + QUAD $0xfffffab8481fffff + QUAD $0xf9ba481fffffffff + QUAD $0x491fffffffffffff + QUAD $0xfffffffffffff8b9 + QUAD $0x0000441f0f27eb1f + QUAD $0x11fbc51024748b48 + QUAD $0xc38348c1ff48ce04 + QUAD $0x4808282444834808 + QUAD $0x0240840f08244c3b + QUAD $0x0818247c83480000 + QUAD $0xf631c057f9c51673 + QUAD $0x2e666600000113e9 + QUAD $0x0000000000841f0f + QUAD $0x57f9c530244c8948 + QUAD $0xf63120247c8b48c0 + QUAD $0x8948d18949c9894c + QUAD $0x4de5894de8894cc2 + QUAD $0xfffffffebf49fc89 + QUAD $0x00401f0f1fffffff + QUAD $0xfbc5c8f34c10fbc5 + QUAD $0x0f59f3c5d0f35410 + QUAD $0x4df68949c158fbc5 + QUAD $0xf00c596b81c4fe31 + QUAD $0x4c10fbc5c158fbc5 + QUAD $0xe6314df68949d8f3 + QUAD $0xfbc5f00c597381c4 + QUAD $0xe0f34c10fbc5c158 + QUAD $0x81c4ee314df68949 + QUAD $0x5410fbc5f00c5973 + QUAD $0xc63149f68949e8f3 + QUAD $0xfbc5f014596b81c4 + QUAD $0xfbc5c258fbc5c158 + QUAD $0x49f68949f0f34c10 + QUAD $0xf00c597381c4d631 + QUAD $0x4c10fbc5c158fbc5 + QUAD $0xce314df68949f8f3 + QUAD $0xfbc5f00c597381c4 + QUAD $0x49f30c10fbc5c158 + QUAD $0x7381c4ce3149f689 + QUAD $0x48c158fbc5f00c59 + QUAD $0x49c0c7834808c683 + QUAD $0xffffff48850ff339 + QUAD $0x894de7894dd2854d + QUAD $0x4cd08948c58949ec + QUAD $0x4c8b48c98949ca89 + QUAD $0xfffffeb8840f3024 + QUAD $0x00000000f5348d4c + QUAD $0xf7294c20247c8b48 + QUAD $0x348d492824748b4c + QUAD $0x2e666666f63145f6 + QUAD $0x0000000000841f0f + QUAD $0xf3c5f60c107ba1c4 + QUAD $0xff49c158fbc50f59 + QUAD $0xf2394df8c78348c6 + QUAD $0x48fffffe71e9e675 + QUAD $0xc88948087304f883 + QUAD $0xf88348000000a7e9 + QUAD $0x047310244c8b4c10 + QUAD $0x247c8b4857ebd231 + QUAD $0x89480fe683fe8908 + QUAD $0xf0e78348f22948c2 + QUAD $0x66c88949c057f9c5 + QUAD $0x0000000000841f0f + QUAD $0x81c4c104117d81c4 + QUAD $0x7d81c420c144117d + QUAD $0x117d81c440c14411 + QUAD $0x4c10c0834960c144 + QUAD $0x74f68548dc75c739 + QUAD $0x8b4c767204fe8369 + QUAD $0x08247c8b4810244c + QUAD $0xf0294803e683fe89 + QUAD $0x8948ca0148c80148 + QUAD $0x57f9c5fce18348f9 + QUAD $0x2e666666666666c0 + QUAD $0x0000000000841f0f + QUAD $0x8348d104117dc1c4 + QUAD $0x48f175d1394804c2 + QUAD $0x244c8b481e74f685 + QUAD $0x00000000801f0f10 + QUAD $0x00000000c104c748 + QUAD $0x0824443948c0ff48 + QUAD $0x415bd8658d48ee75 + QUAD $0x5d5f415e415d415c + QUAD $0x48d10148c377f8c5 + LONG $0xc8ebc889 RET // func correlateFloat64AVX2(dst []float64, sig []float64, ker []float64) @@ -829,69 +1297,167 @@ TEXT ·correlateFloat64AVX2(SB), NOSPLIT|NOFRAME, $0-72 MOVQ ker_len+56(FP), R9 QUAD $0x294dc0940fc9854d QUAD $0xc20841c29c0f41c8 - QUAD $0x8d49000001ea850f - QUAD $0x4e0f48c1394c0140 - QUAD $0x01d68e0fc08548c1 - QUAD $0x358e0fc9854d0000 - QUAD $0x41e5894855000001 - QUAD $0xf8e4834853564157 - QUAD $0xb84907e183c98944 - QUAD $0x7ffffffffffffff0 - QUAD $0x4cc8214d08c88349 - QUAD $0x1aebdb314538568d - QUAD $0xff49df04117ba1c4 - QUAD $0xc6834808c28349c3 - QUAD $0x00e3840fc3394908 - QUAD $0xdb31c057f9c50000 - QUAD $0x00a2820f08f98349 - QUAD $0x0000441f0f660000 - QUAD $0xc4c8da4c107bc1c4 - QUAD $0xf3c5d0da54107bc1 + QUAD $0x4855000004e4850f + QUAD $0x485356415741e589 + QUAD $0x4c01408d49f8e483 + QUAD $0x8348c14e0f48c139 + QUAD $0xbfe9c931077d10f8 + QUAD $0x8e0fc9854d000002 + QUAD $0xfff8b84900000128 + QUAD $0x83497fffffffffff + QUAD $0x568d4cc8214d06c8 + QUAD $0x314500000010bb68 + QUAD $0x2e6666666671ebdb + QUAD $0x0000000000841f0f + QUAD $0x197de2c4de348d4e + QUAD $0xde2c595dc1c4da24 + QUAD $0xc520de74595dc1c4 + QUAD $0xc4ce58f5c5c558fd + QUAD $0xedc540de6c595dc1 + QUAD $0xde64595dc1c4d558 + QUAD $0x7da1c4dc58e5c560 + QUAD $0x4c117da1c4df0411 + QUAD $0xdf54117da1c420df + QUAD $0x60df5c117da1c440 + QUAD $0x80ea834910598d48 + QUAD $0x8f0fc33948cb8949 + QUAD $0xc5d9894800000226 + QUAD $0x57f1c5db31c057f9 + QUAD $0x57e1c5d257e9c5c9 + QUAD $0x90817401f98349db + QUAD $0xc1c4da24197de2c4 + QUAD $0x58fdc598da6c595d + QUAD $0xb8da6c595dc1c4c5 + QUAD $0xc5d8da74595dc1c4 + QUAD $0xc4d658edc5cd58f5 + QUAD $0xe5c5f8da64595dc1 + QUAD $0xda64197de2c4dc58 + QUAD $0xa0da6c595dc1c408 + QUAD $0x595dc1c4c558fdc5 + QUAD $0xc4cd58f5c5c0da6c + QUAD $0xc1c4e0da6c595dc1 + QUAD $0xd558edc5da24595d + QUAD $0x02c38348dc58e5c5 + QUAD $0xc1f6419375d83949 + QUAD $0xe9ffffff09850f01 + QUAD $0xf0488d48ffffff39 + QUAD $0x4904eac149ca8949 + QUAD $0xe08341d08945c2ff + QUAD $0x410d7370f9834807 + QUAD $0xe9c93100000010ba + QUAD $0xf8e2834900000136 + QUAD $0x9066c057f9c5c931 + QUAD $0x11fdc5cf0411fdc5 + QUAD $0xcf4411fdc520cf44 + QUAD $0xc560cf4411fdc540 + QUAD $0x00000080cf8411fd + QUAD $0x0000a0cf8411fdc5 + QUAD $0x00c0cf8411fdc500 + QUAD $0xe0cf8411fdc50000 + QUAD $0xcf8411fdc5000000 + QUAD $0x8411fdc500000100 + QUAD $0x11fdc500000120cf + QUAD $0xfdc500000140cf84 + QUAD $0xc500000160cf8411 + QUAD $0x00000180cf8411fd + QUAD $0x0001a0cf8411fdc5 + QUAD $0x01c0cf8411fdc500 + QUAD $0xe0cf8411fdc50000 + QUAD $0xcf8411fdc5000001 + QUAD $0x8411fdc500000200 + QUAD $0x11fdc500000220cf + QUAD $0xfdc500000240cf84 + QUAD $0xc500000260cf8411 + QUAD $0x00000280cf8411fd + QUAD $0x0002a0cf8411fdc5 + QUAD $0x02c0cf8411fdc500 + QUAD $0xe0cf8411fdc50000 + QUAD $0xcf8411fdc5000002 + QUAD $0x8411fdc500000300 + QUAD $0x11fdc500000320cf + QUAD $0xfdc500000340cf84 + QUAD $0xc500000360cf8411 + QUAD $0x00000380cf8411fd + QUAD $0x0003a0cf8411fdc5 + QUAD $0x03c0cf8411fdc500 + QUAD $0xe0cf8411fdc50000 + QUAD $0x4980e98348000003 + QUAD $0xfffedf850ff8c283 + QUAD $0x8d4c3474c0854dff + QUAD $0x9066c057f9c51051 + QUAD $0x11fdc5cf0411fdc5 + QUAD $0xcf4411fdc520cf44 + QUAD $0x4c60cf4411fdc540 + QUAD $0xff4910c28349d189 + QUAD $0x4cf0c28349dd75c8 + QUAD $0xc82949c08949d189 + QUAD $0x854d000001ea8e0f + QUAD $0x450000012e8e0fc9 + QUAD $0xba4907e08341c889 + QUAD $0x7ffffffffffffff8 + QUAD $0x49ce1c8d4cca214d + QUAD $0xebce348d4838c383 + QUAD $0x2e66666666666628 + QUAD $0x0000000000841f0f + QUAD $0xc1ff48cf0411fbc5 + QUAD $0x08c6834808c38349 + QUAD $0x000197840fc13948 + QUAD $0x49db31c057f9c500 + QUAD $0x0000a3820f08f983 + QUAD $0x00000000801f0f00 + QUAD $0xc4c8db4c107bc1c4 + QUAD $0xf3c5d0db54107bc1 QUAD $0xda5459ebc5da0c59 QUAD $0x7bc1c4c158fbc508 - QUAD $0x4c59f3c5d8da4c10 + QUAD $0x4c59f3c5d8db4c10 QUAD $0xc1c4c258fbc510da - QUAD $0x59ebc5e0da54107b + QUAD $0x59ebc5e0db54107b QUAD $0xc4c158fbc518da54 - QUAD $0xf3c5e8da4c107bc1 + QUAD $0xf3c5e8db4c107bc1 QUAD $0xc258fbc520da4c59 - QUAD $0xc5f0da54107bc1c4 + QUAD $0xc5f0db54107bc1c4 QUAD $0x58fbc528da5459eb - QUAD $0xf8da4c107bc1c4c1 + QUAD $0xf8db4c107bc1c4c1 QUAD $0xfbc530da4c59f3c5 - QUAD $0xda14107bc1c4c258 + QUAD $0xdb14107bc1c4c258 QUAD $0xfbc538da5459ebc5 QUAD $0x8348c258fbc5c158 - QUAD $0x6d850fd8394908c3 - QUAD $0x840fc98548ffffff + QUAD $0x6d850fda394908c3 + QUAD $0x840fc0854dffffff QUAD $0xda348d4cffffff34 QUAD $0x66ff3145de1c8d48 QUAD $0x0000000000841f0f QUAD $0x81c4fb0c107ba1c4 QUAD $0xc158fbc5fe0c5973 - QUAD $0xe875f9394cc7ff49 - QUAD $0x658d48ffffff03e9 - QUAD $0xc35d5f415e415be8 - QUAD $0xc931047304f88348 - QUAD $0xfffffff0ba487eeb - QUAD $0x10f883487fffffff - QUAD $0x894843ebc9310473 - QUAD $0xf8c5f631d12148c1 - QUAD $0x2e6666666666c057 - QUAD $0x0000000000841f0f - QUAD $0x11fcc5f70411fcc5 - QUAD $0xf74411fcc520f744 - QUAD $0x4860f74411fcc540 - QUAD $0xe075f1394810c683 - QUAD $0x740ca83b74c83948 - QUAD $0x0cc28348ce894827 - QUAD $0xf8c5c12148d18948 - QUAD $0x0000441f0f66c057 - QUAD $0xc68348f70411fcc5 - QUAD $0x0bebf275f1394804 - QUAD $0x00000000cf04c748 - QUAD $0xf075c83948c1ff48 - LONG $0xc377f8c5 + QUAD $0xe875f8394dc7ff49 + QUAD $0xf88349ffffff03e9 + QUAD $0x95e9c88949087304 + QUAD $0x7310f88349000000 + QUAD $0x83c68949ebd23104 + QUAD $0xf22948c2894c0fe6 + QUAD $0xc5f0e18349c18949 + QUAD $0x9066ca8949c057f9 + QUAD $0xa1c4d704117da1c4 + QUAD $0x7da1c420d744117d + QUAD $0x117da1c440d74411 + QUAD $0x4d10c2834960d744 + QUAD $0x74f68548dc75d139 + QUAD $0xc689607204fe8357 + QUAD $0x0149f0294903e683 + QUAD $0x48c18948ca0148c8 + QUAD $0x66c057f9c5fce183 + QUAD $0x0000000000841f0f + QUAD $0xc28348d70411fdc5 + QUAD $0x8548f275d1394804 + QUAD $0x2e666666661d74f6 + QUAD $0x0000000000841f0f + QUAD $0x00000000c704c74a + QUAD $0xf075c0394cc0ff49 + QUAD $0x415e415be8658d48 + QUAD $0x0148c377f8c55d5f + LONG $0xc88949d1 + BYTE $0xeb + BYTE $0xda RET // func tileFloat32AVX2(dst []float32, pattern []float32) @@ -2967,21 +3533,25 @@ TEXT ·movingAverageFloat32AVX2(SB), NOSPLIT|NOFRAME, $0-56 MOVQ width+48(FP), R8 QUAD $0x294cc09e0fc0854d QUAD $0xc10841c19c0f41c1 - QUAD $0x8d48000000e3850f - QUAD $0x4e0f48ca39480141 - QUAD $0x00cf8e0fc08548c2 - QUAD $0x4853e58948550000 - QUAD $0xc02a82c1c4f8e483 - QUAD $0x83c28944ff488d49 - QUAD $0xfffffff8b94907e2 - QUAD $0x4cc1214d7fffffff - QUAD $0x22ebd231451c468d - QUAD $0x0000000000841f0f - QUAD $0x117aa1c4c85ef2c5 - QUAD $0xc08349c2ff49970c - QUAD $0xc2394904c6834804 - QUAD $0x3145c957f0c57974 - QUAD $0x664e7207f98348db + QUAD $0x48550000010b850f + QUAD $0x554156415741e589 + QUAD $0x48f8e48348535441 + QUAD $0xfffffffffffff8b8 + QUAD $0xca394801498d4c7f + QUAD $0x10f98349ca4e0f4c + QUAD $0xc931000000df8d0f + QUAD $0x0000c28d0fc9394c + QUAD $0x8d49c02a82c1c400 + QUAD $0xe28341c28945ff50 + QUAD $0x8e048d4cc0214c07 + QUAD $0x8e348d481cc08349 + QUAD $0x2e666666666627eb + QUAD $0x0000000000841f0f + QUAD $0x0c11fac5c85ef2c5 + QUAD $0x04c08349c1ff488f + QUAD $0x74c9394c04c68348 + QUAD $0xdb3145c957f0c57a + QUAD $0x2e664f7207fa8348 QUAD $0x0000000000841f0f QUAD $0xc4e4984c587281c4 QUAD $0x81c4e8984c587281 @@ -2990,14 +3560,54 @@ TEXT ·movingAverageFloat32AVX2(SB), NOSPLIT|NOFRAME, $0-56 QUAD $0x4c587281c4f4984c QUAD $0x984c587281c4f898 QUAD $0x49980c587281c4fc - QUAD $0xc075d9394d08c383 - QUAD $0x1c8d4e8b74d28548 + QUAD $0xc075d8394c08c383 + QUAD $0x1c8d4e8b74d2854d QUAD $0x0000441f0fdb319e QUAD $0xff489b0c5872c1c4 - QUAD $0x6de9f275da3948c3 - QUAD $0x5bf8658d48ffffff - BYTE $0x5d - BYTE $0xc3 + QUAD $0x6de9f275da3949c3 + QUAD $0x5bd8658d48ffffff + QUAD $0x5f415e415d415c41 + QUAD $0x82c1c4c377f8c55d + QUAD $0x49c0187de2c4c02a + QUAD $0x8341c28945ff508d + QUAD $0xc32149c3894d07e2 + QUAD $0x0010bc413c5e8d48 + QUAD $0xf68949ff31450000 + QUAD $0x2e66666666663beb + QUAD $0x0000000000841f0f + QUAD $0xd05eecc5c85ef4c5 + QUAD $0xa1c4bf0c117ca1c4 + QUAD $0x618d4c20bf54117c + QUAD $0xc6834940c3834810 + QUAD $0x0fcc394dcf894940 + QUAD $0xe1894cfffffebb8f + QUAD $0xc5e43145c957f0c5 + QUAD $0x0f07fa8348d257e8 + QUAD $0x2e66660000008c82 + QUAD $0x0000000000841f0f + QUAD $0xc4c4a34c5874a1c4 + QUAD $0xa1c4e4a354586ca1 + QUAD $0x6ca1c4c8a34c5874 + QUAD $0x5874a1c4e8a35458 + QUAD $0x54586ca1c4cca34c + QUAD $0xa34c5874a1c4eca3 + QUAD $0xf0a354586ca1c4d0 + QUAD $0xc4d4a34c5874a1c4 + QUAD $0xa1c4f4a354586ca1 + QUAD $0x6ca1c4d8a34c5874 + QUAD $0x5874a1c4f8a35458 + QUAD $0x54586ca1c4dca34c + QUAD $0xa34c5874a1c4fca3 + QUAD $0x49a314586ca1c4e0 + QUAD $0x8875e3394d08c483 + QUAD $0xffff2f840fd2854d + QUAD $0xed3145a6248d4fff + QUAD $0x0000000000841f0f + QUAD $0x81c4ac0c587481c4 + QUAD $0xc5ff4920ac54586c + QUAD $0xff06e9eb75ea394d + BYTE $0xff + BYTE $0xff RET // func shiftDivFloat32AVX2(dst []float32, a []float32, shift float32, denom float32) @@ -3082,21 +3692,25 @@ TEXT ·movingAverageFloat64AVX2(SB), NOSPLIT|NOFRAME, $0-56 MOVQ width+48(FP), R8 QUAD $0x294cc09e0fc0854d QUAD $0xc10841c19c0f41c1 - QUAD $0x8d48000000e3850f - QUAD $0x4e0f48ca39480141 - QUAD $0x00cf8e0fc08548c2 - QUAD $0x4853e58948550000 - QUAD $0xc02a83c1c4f8e483 - QUAD $0x83c28944ff488d49 - QUAD $0xfffffff8b94907e2 - QUAD $0x4cc1214d7fffffff - QUAD $0x22ebd2314538468d - QUAD $0x0000000000841f0f - QUAD $0x117ba1c4c85ef3c5 - QUAD $0xc08349c2ff49d70c - QUAD $0xc2394908c6834808 - QUAD $0x3145c957f1c57974 - QUAD $0x664e7207f98348db + QUAD $0x48550000010b850f + QUAD $0x554156415741e589 + QUAD $0x48f8e48348535441 + QUAD $0xfffffffffffff8b8 + QUAD $0xca394801498d4c7f + QUAD $0x10f98349ca4e0f4c + QUAD $0xc931000000df8d0f + QUAD $0x0000c28d0fc9394c + QUAD $0x8d49c02a83c1c400 + QUAD $0xe28341c28945ff50 + QUAD $0xce048d4cc0214c07 + QUAD $0xce348d4838c08349 + QUAD $0x2e666666666627eb + QUAD $0x0000000000841f0f + QUAD $0x0c11fbc5c85ef3c5 + QUAD $0x08c08349c1ff48cf + QUAD $0x74c9394c08c68348 + QUAD $0xdb3145c957f1c57a + QUAD $0x2e664f7207fa8348 QUAD $0x0000000000841f0f QUAD $0xc4c8d84c587381c4 QUAD $0x81c4d0d84c587381 @@ -3105,14 +3719,58 @@ TEXT ·movingAverageFloat64AVX2(SB), NOSPLIT|NOFRAME, $0-56 QUAD $0x4c587381c4e8d84c QUAD $0xd84c587381c4f0d8 QUAD $0x49d80c587381c4f8 - QUAD $0xc075d9394d08c383 - QUAD $0x1c8d4e8b74d28548 + QUAD $0xc075d8394c08c383 + QUAD $0x1c8d4e8b74d2854d QUAD $0x0000441f0fdb31de QUAD $0xff48db0c5873c1c4 - QUAD $0x6de9f275da3948c3 - QUAD $0x5bf8658d48ffffff - BYTE $0x5d - BYTE $0xc3 + QUAD $0x6de9f275da3949c3 + QUAD $0x5bd8658d48ffffff + QUAD $0x5f415e415d415c41 + QUAD $0x83c1c4c377f8c55d + QUAD $0x49c0197de2c4c02a + QUAD $0x8341c28945ff508d + QUAD $0x214d04588d4c03e2 + QUAD $0x10bc41785e8d48c3 + QUAD $0x8949ff3145000000 + QUAD $0x2e6666666650ebf6 + QUAD $0x0000000000841f0f + QUAD $0xe05eddc5c85ef5c5 + QUAD $0xa1c4ff0c117da1c4 + QUAD $0x5eedc520ff64117d + QUAD $0x7da1c4d05ee5c5c8 + QUAD $0x117da1c440ff4c11 + QUAD $0x4810618d4c60ff54 + QUAD $0x4980ee834980eb83 + QUAD $0xa58f0fcc394dcf89 + QUAD $0xf1c5e1894cfffffe + QUAD $0x57d9c5e43145c957 + QUAD $0x57e1c5d257e9c5e4 + QUAD $0x8e820f03fa8348db + QUAD $0x2e66666666000000 + QUAD $0x0000000000841f0f + QUAD $0xc488e34c5875a1c4 + QUAD $0xa1c4a8e364585da1 + QUAD $0x65a1c4c8e354586d + QUAD $0x5875a1c4e8e35c58 + QUAD $0x64585da1c490e34c + QUAD $0xe354586da1c4b0e3 + QUAD $0xf0e35c5865a1c4d0 + QUAD $0xc498e34c5875a1c4 + QUAD $0xa1c4b8e364585da1 + QUAD $0x65a1c4d8e354586d + QUAD $0x5875a1c4f8e35c58 + QUAD $0x64585da1c4a0e34c + QUAD $0xe354586da1c4c0e3 + QUAD $0x49e31c5865a1c4e0 + QUAD $0x8875e3394d04c483 + QUAD $0xffff0f840fd2854d + QUAD $0xed3145e6248d4fff + QUAD $0x0000000000841f0f + QUAD $0x81c4ec0c587581c4 + QUAD $0x6d81c420ec64585d + QUAD $0x586581c440ec5458 + QUAD $0x394dc5ff4960ec5c + QUAD $0xfffffed8e9dd75ea RET // func shiftDivFloat64AVX2(dst []float64, a []float64, shift float64, denom float64) diff --git a/internal/amd64/numeric_avx512_amd64.s b/internal/amd64/numeric_avx512_amd64.s index 9c397b4..0fd9fb6 100644 --- a/internal/amd64/numeric_avx512_amd64.s +++ b/internal/amd64/numeric_avx512_amd64.s @@ -189,24 +189,90 @@ TEXT ·polyEvalFloat32AVX512(SB), NOSPLIT|NOFRAME, $0-72 MOVQ x_len+32(FP), R8 MOVQ coeffs_len+56(FP), R9 QUAD $0x4dc14c0f4cc1394c - QUAD $0x000000f7840fc985 - QUAD $0x0002538e0fc0854d - QUAD $0xfc8a44107aa1c400 - QUAD $0x0f02e88348c8894c - QUAD $0x894855000000f08c - QUAD $0x8d41f8e4834853e5 - QUAD $0xe1834107e183ff49 - QUAD $0x2e661cebd2314507 + QUAD $0xe58948551a74c985 + QUAD $0xe483485356415741 + QUAD $0x31217d10f88349f8 + QUAD $0x854d00000205e9c0 + QUAD $0x49000004338e0fc0 + QUAD $0x00014a830f08f883 + QUAD $0x00000413e9c03100 + QUAD $0x49fc8a44107aa1c4 + QUAD $0x00010e8e0f01f983 + QUAD $0x518d45fe498d4900 + QUAD $0x41c018487df262ff + QUAD $0x8341cb894507e283 + QUAD $0x00000010be4107e3 + QUAD $0x2e66666623ebdb31 + QUAD $0x0000000000841f0f + QUAD $0x4c9f1411487cf162 + QUAD $0x394dc3894810708d + QUAD $0x4c000001948f0fc6 + QUAD $0x0c10487cf162f089 + QUAD $0x487cf162ce89499e + QUAD $0x257401fb8341d028 + QUAD $0xf162ce8949d7894d + QUAD $0x00401f0fd028487c + QUAD $0xb162d159486cf162 + QUAD $0xceff49b21458586c + QUAD $0xf98348eb75cfff49 + QUAD $0x9008c68349a57207 + QUAD $0xb162d159486cf162 + QUAD $0xf162f8b25458586c + QUAD $0x586cb162d159486c + QUAD $0x486cf162f7b25458 + QUAD $0x5458586cb162d159 + QUAD $0xd159486cf162f6b2 + QUAD $0xf5b25458586cb162 + QUAD $0xb162d159486cf162 + QUAD $0xf162f4b25458586c + QUAD $0x586cb162d159486c + QUAD $0x486cf162f3b25458 + QUAD $0x5458586cb162d159 + QUAD $0xd159486cf162f2b2 + QUAD $0xf1b25458586cb162 + QUAD $0x07fe8349f8c68349 + QUAD $0x49ffffff21e9867f + QUAD $0xc149c28949f0408d + QUAD $0xd18944c2ff4904ea + QUAD $0x7370f8834807e183 + QUAD $0x3100000010ba4125 + QUAD $0xffffc0b9487febc0 + QUAD $0xf883497fffffffff + QUAD $0x310000019b830f40 + QUAD $0x8349000001dde9c0 + QUAD $0xc818487df262f8e2 + QUAD $0x0000441f0f66c031 + QUAD $0x62870c11487cf162 + QUAD $0x6201874c11487cf1 + QUAD $0x6202874c11487cf1 + QUAD $0x6203874c11487cf1 + QUAD $0x6204874c11487cf1 + QUAD $0x6205874c11487cf1 + QUAD $0x6206874c11487cf1 + QUAD $0x4807874c11487cf1 + QUAD $0x75f8c2834980e883 + QUAD $0x8d4c2d74c98548b7 + QUAD $0xc018487df2621050 + QUAD $0x0000441f0f04e1c1 + QUAD $0x4c870411487cf162 + QUAD $0x834810c28349d089 + QUAD $0xf0c28349ec75f0c1 + QUAD $0x2948c1894cd0894c + QUAD $0xc4000002068e0fc1 + QUAD $0x894dfc8a44107aa1 + QUAD $0xcc8c0f02ea8349ca + QUAD $0x83ff498d41000000 + QUAD $0x19eb07e1834107e1 QUAD $0x0000000000841f0f - QUAD $0xff499714117aa1c4 - QUAD $0x00c8840fc2394dc2 - QUAD $0x960c107aa1c40000 - QUAD $0x41d028f8c5c38949 - QUAD $0xcb89481d7401f983 - QUAD $0x90d028f8c5c38949 + QUAD $0xc0ff48871411fac5 + QUAD $0x0001cc840fc0394c + QUAD $0x894d860c10fac500 + QUAD $0xf98341d028f8c5d3 + QUAD $0x894dcb89481f7401 + QUAD $0x001f0fd028f8c5d3 QUAD $0x586aa1c4d259f2c5 QUAD $0xcbff48cbff499a14 - QUAD $0xb87207f88348ee75 + QUAD $0xb87207fa8349ee75 QUAD $0x00401f0f08c38349 QUAD $0x586aa1c4d259f2c5 QUAD $0xc4d259f2c5e09a54 @@ -220,52 +286,51 @@ TEXT ·polyEvalFloat32AVX512(SB), NOSPLIT|NOFRAME, $0-72 QUAD $0x59f2c5c89a54586a QUAD $0xc49a54586aa1c4d2 QUAD $0x07fb8349f8c38349 - QUAD $0x4dffffff49e99e7f - QUAD $0x0000015c8e0fc085 - QUAD $0xc0311b7308f88349 - QUAD $0xf8834900000140e9 - QUAD $0x0129e9c031227308 - QUAD $0x5d5bf8658d480000 - QUAD $0xffffffffc0b948c3 - QUAD $0x7340f883497fffff - QUAD $0xc0b94866ebc0311b - QUAD $0x497fffffffffffff - QUAD $0xe9c0317e7340f883 - QUAD $0x48c0894c000000c7 - QUAD $0xc057f8c5d231c821 - QUAD $0x0000000000841f0f + QUAD $0x48ffffff49e99e7f + QUAD $0xc18948087308f983 + QUAD $0xf983480000010be9 + QUAD $0x00c0e9d231787340 + QUAD $0xc82148c0894c0000 + QUAD $0x9066c057f8c5d231 QUAD $0x62970411487cf162 QUAD $0x6201974411487cf1 QUAD $0x6202974411487cf1 QUAD $0x4803974411487cf1 QUAD $0xd875d0394840c283 - QUAD $0x0000cb840fc03949 - QUAD $0xb2840f38c0f64100 + QUAD $0x0000f1840fc03949 + QUAD $0xd8840f38c0f64100 QUAD $0x8348c28948000000 QUAD $0xc0214cc8894838c1 QUAD $0x00401f0fc057f8c5 QUAD $0xc28348970411fcc5 - QUAD $0x94e9f275d0394808 - QUAD $0x2148c0894c000000 - QUAD $0x31c818487df262c8 - QUAD $0x2e666666666666d2 - QUAD $0x0000000000841f0f - QUAD $0x62970c11487cf162 - QUAD $0x6201974c11487cf1 - QUAD $0x6202974c11487cf1 - QUAD $0x4803974c11487cf1 - QUAD $0xd875d0394840c283 - QUAD $0xc0f6414f74c03949 - QUAD $0x8348c28948307438 - QUAD $0xc0214cc8894838c1 - QUAD $0x2e6666c8187de2c4 - QUAD $0x0000000000841f0f + QUAD $0xbae9f275d0394808 + QUAD $0x8348ca8948000000 + QUAD $0xc818487df262c0e2 + QUAD $0xc0c6814887348d48 + QUAD $0x9066c93145000000 + QUAD $0xfd8e4c11487cb162 + QUAD $0xfe8e4c11487cb162 + QUAD $0xff8e4c11487cb162 + QUAD $0x498e0c11487cb162 + QUAD $0xd875ca394c40c183 + QUAD $0x38c1f65074d13948 + QUAD $0x07e683c689445974 + QUAD $0xe2c4c10148f12948 + QUAD $0x894cc20148c8187d + QUAD $0x001f0ff8e08348c0 QUAD $0xc28348970c11fcc5 - QUAD $0x3949f275d0394808 - QUAD $0x870411fac51974c0 - QUAD $0x8704c7f1ebc0ff48 - QUAD $0x49c0ff4800000000 - QUAD $0xc377f8c5f175c039 + QUAD $0x8548f275d0394808 + QUAD $0x2e666666661a74f6 + QUAD $0x0000000000841f0f + QUAD $0xc1ff488f0411fac5 + QUAD $0x658d48f375c83949 + QUAD $0xc55d5f415e415be8 + QUAD $0x8948d00148c377f8 + QUAD $0x00008704c7ddebc1 + QUAD $0xc03949c0ff480000 + LONG $0xf8c5f175 + BYTE $0x77 + BYTE $0xc3 RET // func convolveFloat32AVX512(dst []float32, sig []float32, ker []float32) @@ -276,101 +341,328 @@ TEXT ·convolveFloat32AVX512(SB), NOSPLIT|NOFRAME, $0-72 MOVQ dst_len+8(FP), CX MOVQ sig_len+32(FP), R8 MOVQ ker_len+56(FP), R9 + // Constant pool appended below the body; the displacements + // above were patched to reach it and the aligned loads made + // unaligned, since nothing promises alignment inside a TEXT. QUAD $0x56415741e5894855 QUAD $0xe483485354415541 - QUAD $0x74894838ec8348f8 - QUAD $0xc0940fc9854d0824 - QUAD $0x40c69c0f40c8294d - QUAD $0x000002af850fc608 - QUAD $0x48c1394c01708d49 - QUAD $0x8e0ff68548f14e0f - QUAD $0x0fc9854d0000029b - QUAD $0x0c8d4a000001f28e - QUAD $0x07e08341c889458a - QUAD $0xfffffffffff8ba49 - QUAD $0x448b48ca214d7fff - QUAD $0x8d4a1c588d4c0824 - QUAD $0x8948fcc083488a04 - QUAD $0x182444c748102444 - QUAD $0xfffebe4900000000 - QUAD $0xbf493fffffffffff - QUAD $0x3ffffffffffffffd - QUAD $0xfffffffffffcbc49 - QUAD $0xfffffffbbd493fff - QUAD $0xfffab8483fffffff - QUAD $0xba483fffffffffff - QUAD $0x3ffffffffffffff9 - QUAD $0xfffffffffff8bb48 - QUAD $0x4830247c89483fff - QUAD $0x244c894c28247489 - QUAD $0x0000441f0f3aeb20 - QUAD $0x4c8b4c30247c8b48 - QUAD $0x8f04117aa1c41824 - QUAD $0x4804c38349c1ff49 - QUAD $0x748b480408244483 - QUAD $0x4918244c894c2824 - QUAD $0x0f20244c8b4cf139 - QUAD $0x57f8c5000001ca84 - QUAD $0x45117308f98349c0 - QUAD $0x66000000dbe9c931 + QUAD $0xc9854d38ec8348f8 + QUAD $0x20244c894cc0940f + QUAD $0x41c19c0f41c8294d + QUAD $0x0000067b850fc108 + QUAD $0x4cc1394c01588d4d + QUAD $0x4810fb8349d94e0f + QUAD $0x245c894c30247c89 + QUAD $0x8948182474894828 + QUAD $0xe9d231077d102454 + QUAD $0x244c8b480000038e + QUAD $0x016a8e0fc9854820 + QUAD $0xc8898a048d4c0000 + QUAD $0xfffff8b94903e083 + QUAD $0xc983497fffffffff + QUAD $0x0c568d4cc9214904 + QUAD $0xfcc283488a148d48 + QUAD $0xed314500000010b9 + QUAD $0xfffffffffffebb48 + QUAD $0xfffffffdbe493fff + QUAD $0xfffcbf493fffffff + QUAD $0x89493fffffffffff + QUAD $0x90662bebd68948f4 + QUAD $0x7cb16230247c8b48 + QUAD $0x104a8d48af041148 + QUAD $0x40c4834940c28349 + QUAD $0x28245c8b4cd58949 + QUAD $0x0003018f0fd9394c + QUAD $0x247c8348ca894800 + QUAD $0xc057f8c50c730420 + QUAD $0x9000000094e9c931 + QUAD $0x31f78948c057f8c5 + QUAD $0x00000000801f0fc9 + QUAD $0xd1620f18487df262 + QUAD $0xfffff48a8c594874 + QUAD $0x49c158487cf162ff + QUAD $0x7d9262db3149cb89 + QUAD $0x4874d162980c1848 + QUAD $0x62fffffff88a8c59 + QUAD $0xcb8949c158487cf1 + QUAD $0x18487d9262f3314d + QUAD $0x8c594874d162980c + QUAD $0x7cf162fffffffc8a + QUAD $0x314dcb8949c15848 + QUAD $0x980c18487d9262fb + QUAD $0x628a0c594874d162 + QUAD $0xc18348c158487cf1 + QUAD $0xc93949f0c7834804 + QUAD $0x2d840fc085488675 + QUAD $0x488c3c8d49ffffff + QUAD $0x2949f3894902e1c1 + QUAD $0x2e66666666c931cb QUAD $0x0000000000841f0f - QUAD $0xc9314510247c8b48 + QUAD $0xf1620b18487dd262 + QUAD $0x7cf1628f0c594874 + QUAD $0x8349c1ff48c15848 + QUAD $0xe9e175c83948fcc3 + QUAD $0xf0438d4dfffffeec + QUAD $0x73000000f0f88149 + QUAD $0xd23100000010b80c + QUAD $0xe8c149000001dce9 + QUAD $0x49c1894dc0ff4904 + QUAD $0xc148c8894cf0e183 + QUAD $0x0510487cf16204e0 + QUAD $0x48c28948000004b8 + QUAD $0x10487cf16210c083 + QUAD $0x57e8c5000004e70d + QUAD $0x191d5948fdf262d2 + QUAD $0x9066ca894d000007 + QUAD $0x497df262c846fdc5 + QUAD $0x62c846fdc58f14a3 + QUAD $0xfdc58714a3497df2 + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc5018f + QUAD $0xfdc5018754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc5028f + QUAD $0xfdc5028754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc5038f + QUAD $0xfdc5038754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc5048f + QUAD $0xfdc5048754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc5058f + QUAD $0xfdc5058754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc5068f + QUAD $0xfdc5068754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc5078f + QUAD $0xfdc5078754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc5088f + QUAD $0xfdc5088754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc5098f + QUAD $0xfdc5098754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc50a8f + QUAD $0xfdc50a8754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc50b8f + QUAD $0xfdc50b8754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc50c8f + QUAD $0xfdc50c8754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc50d8f + QUAD $0xfdc50d8754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc50e8f + QUAD $0xfdc50e8754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc50f8f + QUAD $0xf1620f8754a3497d + QUAD $0x48fdf162cbd448f5 + QUAD $0x850ff0c28349c3d4 + QUAD $0x74c8394dfffffe6c + QUAD $0xd18948c057f8c521 + QUAD $0x488f0411487cf162 + QUAD $0x394c10c08348c189 + QUAD $0x48f0c08348ed7ed8 + QUAD $0x548948d8894cc289 + QUAD $0x548b48d029480824 + QUAD $0x0f1024748b481824 + QUAD $0x4c8b48000002a68e + QUAD $0xdb8e0fc985482024 + QUAD $0x418e048d4c000001 + QUAD $0xbb4907e28341ca89 + QUAD $0x7ffffffffffffff8 + QUAD $0x0824448b48cb2149 + QUAD $0x1cc38348821c8d48 + QUAD $0x4c8948fc8e4c8d48 + QUAD $0x894882048d481024 + QUAD $0xfffffebe49182444 + QUAD $0xfdbf493fffffffff + QUAD $0x493fffffffffffff + QUAD $0xfffffffffffffcbc + QUAD $0xfffffffffbbd493f + QUAD $0xfffffab8483fffff + QUAD $0xf9b9493fffffffff + QUAD $0x483fffffffffffff + QUAD $0xfffffffffffff8ba + QUAD $0x2e6666666639eb3f QUAD $0x0000000000841f0f - QUAD $0xc4e48b4c107a81c4 - QUAD $0xf2c5e88b54107a81 - QUAD $0x894cc158fac50f59 - QUAD $0x0c59eac5f6314cce - QUAD $0x7a81c4c158fac5b1 - QUAD $0x4cce894cec8b4c10 - QUAD $0xc5b10c59f2c5fe31 - QUAD $0x4c107a81c4c158fa - QUAD $0xe6314cce894cf08b - QUAD $0x58fac5b10c59f2c5 - QUAD $0xf48b4c107a81c4c1 - QUAD $0xf2c5ee314cce894c - QUAD $0xc4c158fac5b10c59 - QUAD $0x894cf88b4c107a81 - QUAD $0x0c59f2c5c63148ce - QUAD $0x7a81c4c158fac5b1 - QUAD $0x48ce894cfc8b4c10 - QUAD $0xc5b10c59f2c5d631 - QUAD $0x0c107a81c4c158fa - QUAD $0xc5de3148ce894c8b - QUAD $0xc158fac5b10c59f2 - QUAD $0xe0c7834808c18349 - QUAD $0xffff47850fca394d - QUAD $0xfede840fc0854dff - QUAD $0x00008d348d4affff - QUAD $0x4810247c8b480000 - QUAD $0x4e0824748b48f729 - QUAD $0x001f0ff6318e0c8d - QUAD $0xf2c5b10c107ac1c4 - QUAD $0xff48c158fac50f59 - QUAD $0xf03949fcc78348c6 - QUAD $0x48fffffea1e9e675 - QUAD $0xe9c031077308fe83 - QUAD $0xfff8b94800000084 - QUAD $0x83487fffffffffff - QUAD $0x43ebc031047340fe - QUAD $0x31f02148c8418d48 - QUAD $0x001f0fc057f8c5d2 - QUAD $0x62970411487cf162 - QUAD $0x6201974411487cf1 - QUAD $0x6202974411487cf1 - QUAD $0x4803974411487cf1 - QUAD $0xd875d0394840c283 - QUAD $0xc6f6404274c63948 - QUAD $0xf8c5f121482d7438 - QUAD $0x0000441f0f66c057 - QUAD $0xc08348870411fcc5 - QUAD $0x8948f275c1394808 - QUAD $0x2e661974ce3948c8 + QUAD $0x748b4830244c8b48 + QUAD $0x48b10411fac50824 + QUAD $0x834804c38348c6ff + QUAD $0x2474894804182444 + QUAD $0x840f2824743b4808 + QUAD $0x247c8348000001df + QUAD $0xc057f8c50c730820 + QUAD $0x90000000d2e9c931 + QUAD $0x247c8b48c057f8c5 + QUAD $0x0000441f0fc93110 + QUAD $0xfac5e48b4c10fac5 + QUAD $0x0f59f2c5e88b5410 + QUAD $0x4cce8948c158fac5 + QUAD $0xb00c596ac1c4f631 + QUAD $0x4c10fac5c158fac5 + QUAD $0xfe314cce8948ec8b + QUAD $0xfac5b00c5972c1c4 + QUAD $0xf08b4c10fac5c158 + QUAD $0xc1c4e6314cce8948 + QUAD $0xc158fac5b00c5972 + QUAD $0x8948f48b4c10fac5 + QUAD $0x5972c1c4ee314cce + QUAD $0xfac5c158fac5b00c + QUAD $0x48ce8948f88b4c10 + QUAD $0xb00c5972c1c4c631 + QUAD $0x4c10fac5c158fac5 + QUAD $0xce314cce8948fc8b + QUAD $0xfac5b00c5972c1c4 + QUAD $0x488b0c10fac5c158 + QUAD $0x72c1c4d63148ce89 + QUAD $0x48c158fac5b00c59 + QUAD $0x49e0c7834808c183 + QUAD $0xffffff48850fcb39 + QUAD $0xfffeef840fd2854d + QUAD $0x0000008d348d48ff + QUAD $0x294810247c8b4800 + QUAD $0x8d481824748b48f7 + QUAD $0x00401f0ff6318e0c + QUAD $0x59f2c5b10c10fac5 + QUAD $0xc6ff48c158fac50f + QUAD $0x75f23949fcc78348 + QUAD $0x8348fffffeb2e9e7 + QUAD $0x00000098820f08f8 + QUAD $0x24448b4c40f88348 + QUAD $0x484febc931047308 + QUAD $0x8d4ac0e18348c189 + QUAD $0x0000c0c281488714 + QUAD $0x90c057f8c5f63100 + QUAD $0xfdb24411487cf162 + QUAD $0xfeb24411487cf162 + QUAD $0xffb24411487cf162 + QUAD $0x48b20411487cf162 + QUAD $0xd875f1394840c683 + QUAD $0x7438a85e74c83948 + QUAD $0x89480824448b4c6c + QUAD $0x348d4af8e28348c2 + QUAD $0x2444894cd0014987 + QUAD $0x001f0fc057f8c508 + QUAD $0xc183488e0411fcc5 + QUAD $0x3948f275ca394808 + QUAD $0x2e666666662874d0 QUAD $0x0000000000841f0f - QUAD $0x48000000008704c7 - QUAD $0x48f175c63948c0ff - QUAD $0x5d415c415bd8658d - QUAD $0x77f8c55d5f415e41 - BYTE $0xc3 + QUAD $0x8704c70824448b48 + QUAD $0x48c0ff4800000000 + QUAD $0x2444394808244489 + QUAD $0x5bd8658d48e57528 + QUAD $0x5f415e415d415c41 + QUAD $0x4c0148c377f8c55d + QUAD $0x00000080cceb0824 + QUAD $0x0000009000000000 + QUAD $0x000000a000000000 + QUAD $0x000000b000000000 + QUAD $0x000000c000000000 + QUAD $0x000000d000000000 + QUAD $0x000000e000000000 + QUAD $0x000000f000000000 + QUAD $0x0000000000000000 + QUAD $0x0000001000000000 + QUAD $0x0000002000000000 + QUAD $0x0000003000000000 + QUAD $0x0000004000000000 + QUAD $0x0000005000000000 + QUAD $0x0000006000000000 + QUAD $0x0000007000000000 + QUAD $0x0000000000000000 + QUAD $0x0000001000000000 + QUAD $0x0000002000000000 + QUAD $0x0000003000000000 + QUAD $0x0000004000000000 + QUAD $0x0000005000000000 + QUAD $0x0000006000000000 + QUAD $0x0000007000000000 + QUAD $0x0000008000000000 + QUAD $0x0000009000000000 + QUAD $0x000000a000000000 + QUAD $0x000000b000000000 + QUAD $0x000000c000000000 + QUAD $0x000000d000000000 + QUAD $0x000000e000000000 + QUAD $0x000000f000000000 + QUAD $0x0000000000000000 + QUAD $0x0000001000000000 + QUAD $0x0000002000000000 + QUAD $0x0000003000000000 + QUAD $0x0000004000000000 + QUAD $0x0000005000000000 + QUAD $0x0000006000000000 + QUAD $0x0000007000000000 + QUAD $0x0000000000000000 + QUAD $0x0000001000000000 + QUAD $0x0000002000000000 + QUAD $0x0000003000000000 + QUAD $0x0000004000000000 + QUAD $0x0000005000000000 + QUAD $0x0000006000000000 + QUAD $0x0000007000000000 + QUAD $0x0000000000000000 + QUAD $0x0000000100000000 + QUAD $0x0000000200000000 + QUAD $0x0000000300000000 + QUAD $0x0000000400000000 + QUAD $0x0000000500000000 + QUAD $0x0000000600000000 + QUAD $0x0000000700000000 + QUAD $0x0000000000000000 + QUAD $0x0000000100000000 + QUAD $0x0000000200000000 + QUAD $0x0000000300000000 + QUAD $0x0000000400000000 + QUAD $0x0000000500000000 + QUAD $0x0000000600000000 + QUAD $0x0000000700000000 + QUAD $0x0000000800000000 + QUAD $0x0000000900000000 + QUAD $0x0000000a00000000 + QUAD $0x0000000b00000000 + QUAD $0x0000000c00000000 + QUAD $0x0000000d00000000 + QUAD $0x0000000e00000000 + QUAD $0x0000000f00000000 + QUAD $0x0000000000000000 + QUAD $0x0000000100000000 + QUAD $0x0000000200000000 + QUAD $0x0000000300000000 + QUAD $0x0000000400000000 + QUAD $0x0000000500000000 + QUAD $0x0000000600000000 + QUAD $0x0000000700000000 + QUAD $0x0000010000000000 + QUAD $0x0000008000000000 + QUAD $0x0000010000000000 + QUAD $0x0000008000000000 + QUAD $0xfa53e0a800000000 + QUAD $0xf4a7c150f1bbcdcb + QUAD $0xeefba1f8e3779b97 + QUAD $0x7f4a7c15d5336963 + QUAD $0x1ce4e5b99e3779b9 + QUAD $0x133111ebbf58476d + QUAD $0x0000002094d049bb + QUAD $0x0000000800000000 + QUAD $0x7f4a7c1500000000 + QUAD $0x1ce4e5b99e3779b9 + QUAD $0x133111ebbf58476d + QUAD $0x0000000094d049bb + QUAD $0x000000083ca00000 + QUAD $0x7f4a7c1500000000 + QUAD $0x1ce4e5b99e3779b9 + QUAD $0x133111ebbf58476d + QUAD $0x0000001094d049bb + QUAD $0x0000000400000000 + LONG $0x00000000 RET // func correlateFloat32AVX512(dst []float32, sig []float32, ker []float32) @@ -381,75 +673,302 @@ TEXT ·correlateFloat32AVX512(SB), NOSPLIT|NOFRAME, $0-72 MOVQ dst_len+8(FP), CX MOVQ sig_len+32(FP), R8 MOVQ ker_len+56(FP), R9 + // Constant pool appended below the body; the displacements + // above were patched to reach it and the aligned loads made + // unaligned, since nothing promises alignment inside a TEXT. QUAD $0x294dc0940fc9854d QUAD $0xc20841c29c0f41c8 - QUAD $0x8d49000001f9850f - QUAD $0x4e0f48c1394c0140 - QUAD $0x01e58e0fc08548c1 - QUAD $0x358e0fc9854d0000 - QUAD $0x41e5894855000001 - QUAD $0xf8e4834853564157 - QUAD $0xb84907e183c98944 + QUAD $0x4855000005c7850f + QUAD $0x554156415741e589 + QUAD $0x49f8e48348535441 + QUAD $0x0f48c1394c01408d + QUAD $0x077d10f88348c14e + QUAD $0x4d0000039be9c931 + QUAD $0x000001798e0fc985 + QUAD $0x4907e08341c88945 + QUAD $0xfffffffffffff8ba + QUAD $0x1c5e8d4cca214d7f + QUAD $0x314500000010bf41 + QUAD $0x906621ebf38948f6 + QUAD $0x4cb70411487cb162 + QUAD $0x4840c3834910798d + QUAD $0x3949ce894940c383 + QUAD $0x4c0000034b8f0fc7 + QUAD $0x3145c057f8c5f989 + QUAD $0xe6820f08f98349ff + QUAD $0x2e66666666000000 + QUAD $0x0000000000841f0f + QUAD $0x62ba0c18487db262 + QUAD $0xffe4bb8c59487491 + QUAD $0x5418487db262ffff + QUAD $0xc158487cf16201ba + QUAD $0xe8bb8c59486c9162 + QUAD $0x58487cf162ffffff + QUAD $0xba4c18487db262c1 + QUAD $0xbb8c594874916202 + QUAD $0x487db262ffffffec + QUAD $0x486c916203ba5418 + QUAD $0x62fffffff0bb9459 + QUAD $0x7cf162c158487cf1 + QUAD $0x18487db262c25848 + QUAD $0x594874916204ba4c + QUAD $0xf162fffffff4bb8c + QUAD $0x487db262c158487c + QUAD $0x4874916205ba4c18 + QUAD $0x62fffffff8bb8c59 + QUAD $0x7db262c158487cf1 + QUAD $0x74916206ba4c1848 + QUAD $0xfffffffcbb8c5948 + QUAD $0x07ba5418487db262 + QUAD $0x9162c158487cf162 + QUAD $0x7cf162bb0c59486c + QUAD $0x4d08c78349c15848 + QUAD $0xffffff30850ffa39 + QUAD $0xfffee7840fc0854d + QUAD $0x3c8d4ebb248d4eff + QUAD $0x2e666666ed3145ba + QUAD $0x0000000000841f0f + QUAD $0x62af0c18487d9262 + QUAD $0xf162ac0c59487491 + QUAD $0x4dc5ff49c158487c + QUAD $0xfffeafe9e475e839 + QUAD $0xfa8149f0508d4cff + QUAD $0xb8410d73000000f0 + QUAD $0xdee9c93100000010 + QUAD $0x4904eac149000001 + QUAD $0xe38349d3894dc2ff + QUAD $0x04e1c148d9894cf0 + QUAD $0x04ac0510487cf162 + QUAD $0xf16210418d4c0000 + QUAD $0x000004de0d10487c + QUAD $0x48fdf262d257e8c5 + QUAD $0x894c000006601d59 + QUAD $0x00000000801f0fdb + QUAD $0x497df262c846fdc5 + QUAD $0x62c846fdc58f14a3 + QUAD $0xfdc58714a3497df2 + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc5018f + QUAD $0xfdc5018754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc5028f + QUAD $0xfdc5028754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc5038f + QUAD $0xfdc5038754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc5048f + QUAD $0xfdc5048754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc5058f + QUAD $0xfdc5058754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc5068f + QUAD $0xfdc5068754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc5078f + QUAD $0xfdc5078754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc5088f + QUAD $0xfdc5088754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc5098f + QUAD $0xfdc5098754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc50a8f + QUAD $0xfdc50a8754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc50b8f + QUAD $0xfdc50b8754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc50c8f + QUAD $0xfdc50c8754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc50d8f + QUAD $0xfdc50d8754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc50e8f + QUAD $0xfdc50e8754a3497d + QUAD $0x54a3497df262c846 + QUAD $0xf262c846fdc50f8f + QUAD $0xf1620f8754a3497d + QUAD $0x48fdf162cbd448f5 + QUAD $0x850ff0c38348c3d4 + QUAD $0x74da394dfffffe6c + QUAD $0x001f0fc057f8c521 + QUAD $0x4c8f0411487cf162 + QUAD $0x394910c08349c189 + QUAD $0x4cf0c08349ed7ec0 + QUAD $0xc82949c08949c189 + QUAD $0x854d000001e98e0f + QUAD $0x450000012e8e0fc9 + QUAD $0xba4907e08341c889 QUAD $0x7ffffffffffffff8 - QUAD $0x451c568d4cc8214d - QUAD $0x00401f0f1eebdb31 - QUAD $0xff499f04117aa1c4 - QUAD $0xc6834804c28349c3 - QUAD $0x00e3840fc3394904 - QUAD $0xdb31c057f8c50000 - QUAD $0x00a2820f08f98349 - QUAD $0x0000441f0f660000 - QUAD $0xc4e49a4c107ac1c4 - QUAD $0xf2c5e89a54107ac1 + QUAD $0x498e1c8d4cca214d + QUAD $0xeb8e348d481cc383 + QUAD $0x2e66666666666628 + QUAD $0x0000000000841f0f + QUAD $0xc1ff488f0411fac5 + QUAD $0x04c6834804c38349 + QUAD $0x000196840fc13948 + QUAD $0x49db31c057f8c500 + QUAD $0x0000a3820f08f983 + QUAD $0x00000000801f0f00 + QUAD $0xc4e49b4c107ac1c4 + QUAD $0xf2c5e89b54107ac1 QUAD $0xc5c158fac59a0c59 QUAD $0x58fac5049a4c59ea - QUAD $0xec9a4c107ac1c4c1 + QUAD $0xec9b4c107ac1c4c1 QUAD $0xc1c4089a4c59f2c5 - QUAD $0x59eac5f09a54107a + QUAD $0x59eac5f09b54107a QUAD $0xc5c158fac50c9a54 QUAD $0x4c107ac1c4c258fa - QUAD $0x109a4c59f2c5f49a + QUAD $0x109a4c59f2c5f49b QUAD $0x107ac1c4c158fac5 - QUAD $0x9a4c59f2c5f89a4c + QUAD $0x9a4c59f2c5f89b4c QUAD $0x7ac1c4c158fac514 - QUAD $0x4c59f2c5fc9a4c10 - QUAD $0x9a14107ac1c4189a + QUAD $0x4c59f2c5fc9b4c10 + QUAD $0x9b14107ac1c4189a QUAD $0x4c59eac5c158fac5 QUAD $0x8348c158fac51c9a - QUAD $0x6d850fd8394908c3 - QUAD $0x840fc98548ffffff + QUAD $0x6d850fda394908c3 + QUAD $0x840fc0854dffffff QUAD $0x9a348d4cffffff34 QUAD $0x66ff31459e1c8d48 QUAD $0x0000000000841f0f QUAD $0x81c4bb0c107aa1c4 QUAD $0xc158fac5be0c5972 - QUAD $0xe875f9394cc7ff49 - QUAD $0x658d48ffffff03e9 - QUAD $0xc35d5f415e415be8 - QUAD $0xc931077308f88348 - QUAD $0xf8ba480000008be9 - QUAD $0x487fffffffffffff - QUAD $0xebc931047340f883 - QUAD $0xc12148c84a8d4848 - QUAD $0x2e66c057f8c5f631 - QUAD $0x0000000000841f0f - QUAD $0x62b70411487cf162 - QUAD $0x6201b74411487cf1 - QUAD $0x6202b74411487cf1 - QUAD $0x4803b74411487cf1 - QUAD $0xd875f1394840c683 - QUAD $0x7438a84274c83948 - QUAD $0xc057f8c5c221482f + QUAD $0xe875f8394dc7ff49 + QUAD $0xf88349ffffff03e9 + QUAD $0x4900000099820f08 + QUAD $0xebd231047340f883 + QUAD $0xc0e28348c2894c52 + QUAD $0xc0c681488f348d48 + QUAD $0xf8c5c93145000000 + QUAD $0x0000441f0f66c057 + QUAD $0xfd8e4411487cb162 + QUAD $0xfe8e4411487cb162 + QUAD $0xff8e4411487cb162 + QUAD $0x498e0411487cb162 + QUAD $0xd875ca394c40c183 + QUAD $0xc0f6415274d03949 + QUAD $0x8348c6894c5e7438 + QUAD $0x01488f0c8d4cf8e6 + QUAD $0x2e6666c057f8c5f1 QUAD $0x0000000000841f0f - QUAD $0xc183488f0411fcc5 - QUAD $0x8948f275ca394808 - QUAD $0x2e661974d03948d1 + QUAD $0x83489104117cc1c4 + QUAD $0x49f175d6394808c2 + QUAD $0x2e6666661b74f039 QUAD $0x0000000000841f0f QUAD $0x48000000008f04c7 - QUAD $0xc5f175c83948c1ff - BYTE $0xf8 - BYTE $0x77 - BYTE $0xc3 + QUAD $0x48f175c83948c1ff + QUAD $0x5d415c415bd8658d + QUAD $0x77f8c55d5f415e41 + QUAD $0x0080daebd10148c3 + QUAD $0x0090000000000000 + QUAD $0x00a0000000000000 + QUAD $0x00b0000000000000 + QUAD $0x00c0000000000000 + QUAD $0x00d0000000000000 + QUAD $0x00e0000000000000 + QUAD $0x00f0000000000000 + QUAD $0x0000000000000000 + QUAD $0x0010000000000000 + QUAD $0x0020000000000000 + QUAD $0x0030000000000000 + QUAD $0x0040000000000000 + QUAD $0x0050000000000000 + QUAD $0x0060000000000000 + QUAD $0x0070000000000000 + QUAD $0x0000000000000000 + QUAD $0x0010000000000000 + QUAD $0x0020000000000000 + QUAD $0x0030000000000000 + QUAD $0x0040000000000000 + QUAD $0x0050000000000000 + QUAD $0x0060000000000000 + QUAD $0x0070000000000000 + QUAD $0x0080000000000000 + QUAD $0x0090000000000000 + QUAD $0x00a0000000000000 + QUAD $0x00b0000000000000 + QUAD $0x00c0000000000000 + QUAD $0x00d0000000000000 + QUAD $0x00e0000000000000 + QUAD $0x00f0000000000000 + QUAD $0x0000000000000000 + QUAD $0x0010000000000000 + QUAD $0x0020000000000000 + QUAD $0x0030000000000000 + QUAD $0x0040000000000000 + QUAD $0x0050000000000000 + QUAD $0x0060000000000000 + QUAD $0x0070000000000000 + QUAD $0x0000000000000000 + QUAD $0x0010000000000000 + QUAD $0x0020000000000000 + QUAD $0x0030000000000000 + QUAD $0x0040000000000000 + QUAD $0x0050000000000000 + QUAD $0x0060000000000000 + QUAD $0x0070000000000000 + QUAD $0x0000000000000000 + QUAD $0x0001000000000000 + QUAD $0x0002000000000000 + QUAD $0x0003000000000000 + QUAD $0x0004000000000000 + QUAD $0x0005000000000000 + QUAD $0x0006000000000000 + QUAD $0x0007000000000000 + QUAD $0x0000000000000000 + QUAD $0x0001000000000000 + QUAD $0x0002000000000000 + QUAD $0x0003000000000000 + QUAD $0x0004000000000000 + QUAD $0x0005000000000000 + QUAD $0x0006000000000000 + QUAD $0x0007000000000000 + QUAD $0x0008000000000000 + QUAD $0x0009000000000000 + QUAD $0x000a000000000000 + QUAD $0x000b000000000000 + QUAD $0x000c000000000000 + QUAD $0x000d000000000000 + QUAD $0x000e000000000000 + QUAD $0x000f000000000000 + QUAD $0x0000000000000000 + QUAD $0x0001000000000000 + QUAD $0x0002000000000000 + QUAD $0x0003000000000000 + QUAD $0x0004000000000000 + QUAD $0x0005000000000000 + QUAD $0x0006000000000000 + QUAD $0x0007000000000000 + QUAD $0x0100000000000000 + QUAD $0x0080000000000000 + QUAD $0x0100000000000000 + QUAD $0x0080000000000000 + QUAD $0xe0a8000000000000 + QUAD $0xc150f1bbcdcbfa53 + QUAD $0xa1f8e3779b97f4a7 + QUAD $0x7c15d5336963eefb + QUAD $0xe5b99e3779b97f4a + QUAD $0x11ebbf58476d1ce4 + QUAD $0x002094d049bb1331 + QUAD $0x0008000000000000 + QUAD $0x7c15000000000000 + QUAD $0xe5b99e3779b97f4a + QUAD $0x11ebbf58476d1ce4 + QUAD $0x000094d049bb1331 + QUAD $0x00083ca000000000 + QUAD $0x7c15000000000000 + QUAD $0xe5b99e3779b97f4a + QUAD $0x11ebbf58476d1ce4 + QUAD $0x001094d049bb1331 + QUAD $0x0004000000000000 + LONG $0x00000000 + BYTE $0x00 + BYTE $0x00 RET // func normFloat64AVX512(a []float64) float64 @@ -647,24 +1166,110 @@ TEXT ·polyEvalFloat64AVX512(SB), NOSPLIT|NOFRAME, $0-72 MOVQ x_len+32(FP), R8 MOVQ coeffs_len+56(FP), R9 QUAD $0x4dc14c0f4cc1394c - QUAD $0x000000f7840fc985 - QUAD $0x0002548e0fc0854d - QUAD $0xf8ca44107ba1c400 - QUAD $0x0f02e88348c8894c - QUAD $0x894855000000f08c - QUAD $0x8d41f8e4834853e5 - QUAD $0xe1834107e183ff49 - QUAD $0x2e661cebd2314507 + QUAD $0xe58948551a74c985 + QUAD $0xe483485356415741 + QUAD $0x31217d10f88349f8 + QUAD $0x854d000002ade9c0 + QUAD $0x49000004d48e0fc0 + QUAD $0x0001a1830f04f883 + QUAD $0x000004b3e9c03100 + QUAD $0x49f8ca44107ba1c4 + QUAD $0x0001628e0f01f983 + QUAD $0x518d45fe498d4900 + QUAD $0x41c01948fdf262ff + QUAD $0x8341cb894503e283 + QUAD $0x00000010be4103e3 + QUAD $0x2e6666662bebdb31 QUAD $0x0000000000841f0f - QUAD $0xff49d714117ba1c4 - QUAD $0x00c8840fc2394dc2 - QUAD $0xd60c107ba1c40000 - QUAD $0x41d028f9c5c38949 - QUAD $0xcb89481d7401f983 - QUAD $0x90d028f9c5c38949 + QUAD $0x62df1c1148fdf162 + QUAD $0x4c01df641148fdf1 + QUAD $0x394dc3894810708d + QUAD $0x4c000002348f0fc6 + QUAD $0x0c1048fdf162f089 + QUAD $0xde541048fdf162de + QUAD $0x48fdf162ce894901 + QUAD $0xe02848fdf162d828 + QUAD $0x894d417401fb8341 + QUAD $0x48fdf162ce8949d7 + QUAD $0xe02848fdf162d828 + QUAD $0x0000000000841f0f + QUAD $0x62f22c1948fdb262 + QUAD $0xd5f162d95948e5f1 + QUAD $0x5948ddf162db5848 + QUAD $0x49e45848d5f162e2 + QUAD $0x48d975cfff49ceff + QUAD $0xffff6f820f03f983 + QUAD $0x2e666604c68349ff + QUAD $0x0000000000841f0f + QUAD $0xfcf26c1948fdb262 + QUAD $0xf162d95948e5f162 + QUAD $0x48ddf162db5848d5 + QUAD $0xe45848d5f162e259 + QUAD $0xfbf26c1948fdb262 + QUAD $0xf162d95948e5f162 + QUAD $0x48ddf162db5848d5 + QUAD $0xe45848d5f162e259 + QUAD $0xfaf26c1948fdb262 + QUAD $0xf162d95948e5f162 + QUAD $0x48ddf162db5848d5 + QUAD $0xe45848d5f162e259 + QUAD $0xf9f26c1948fdb262 + QUAD $0xf162d95948e5f162 + QUAD $0x48ddf162db5848d5 + QUAD $0xe45848d5f162e259 + QUAD $0x03fe8349fcc68349 + QUAD $0xcde9ffffff728f0f + QUAD $0x49f0408d49fffffe + QUAD $0xff4904eac149c289 + QUAD $0x4807e183d18944c2 + QUAD $0x10ba41287370f883 + QUAD $0x00cce9c031000000 + QUAD $0xffffffe0b9480000 + QUAD $0x20f883497fffffff + QUAD $0xc031000001e4830f + QUAD $0xe2834900000226e9 + QUAD $0x31c81948fdf262f8 + QUAD $0x2e666666666666c0 + QUAD $0x0000000000841f0f + QUAD $0x62c70c1148fdf162 + QUAD $0x6201c74c1148fdf1 + QUAD $0x6202c74c1148fdf1 + QUAD $0x6203c74c1148fdf1 + QUAD $0x6204c74c1148fdf1 + QUAD $0x6205c74c1148fdf1 + QUAD $0x6206c74c1148fdf1 + QUAD $0x6207c74c1148fdf1 + QUAD $0x6208c74c1148fdf1 + QUAD $0x6209c74c1148fdf1 + QUAD $0x620ac74c1148fdf1 + QUAD $0x620bc74c1148fdf1 + QUAD $0x620cc74c1148fdf1 + QUAD $0x620dc74c1148fdf1 + QUAD $0x620ec74c1148fdf1 + QUAD $0x480fc74c1148fdf1 + QUAD $0x0ff8c2834980e883 + QUAD $0xc98548ffffff7385 + QUAD $0xf26210508d4c3174 + QUAD $0x9004e1c1c01948fd + QUAD $0x62c7041148fdf162 + QUAD $0x4c01c7441148fdf1 + QUAD $0x834810c28349d089 + QUAD $0xf0c28349e475f0c1 + QUAD $0x2948c1894cd0894c + QUAD $0xc4000001fe8e0fc1 + QUAD $0x894df8ca44107ba1 + QUAD $0xc48c0f02ea8349ca + QUAD $0x83ff498d41000000 + QUAD $0x11eb07e1834107e1 + QUAD $0xc0ff48c71411fbc5 + QUAD $0x0001cc840fc0394c + QUAD $0x894dc60c10fbc500 + QUAD $0xf98341d028f9c5d3 + QUAD $0x894dcb89481f7401 + QUAD $0x001f0fd028f9c5d3 QUAD $0x586ba1c4d259f3c5 QUAD $0xcbff48cbff49da14 - QUAD $0xb87207f88348ee75 + QUAD $0xb87207fa8349ee75 QUAD $0x00401f0f08c38349 QUAD $0x586ba1c4d259f3c5 QUAD $0xc4d259f3c5c0da54 @@ -678,52 +1283,51 @@ TEXT ·polyEvalFloat64AVX512(SB), NOSPLIT|NOFRAME, $0-72 QUAD $0x59f3c590da54586b QUAD $0x88da54586ba1c4d2 QUAD $0x07fb8349f8c38349 - QUAD $0x4dffffff49e99e7f - QUAD $0x0000015d8e0fc085 - QUAD $0xc0311b7304f88349 - QUAD $0xf8834900000140e9 - QUAD $0x0129e9c031227304 - QUAD $0x5d5bf8658d480000 - QUAD $0xffffffffe0b948c3 - QUAD $0x7320f883497fffff - QUAD $0xe0b94866ebc0311b - QUAD $0x497fffffffffffff - QUAD $0xe9c0317e7320f883 - QUAD $0x48c0894c000000c7 - QUAD $0xc057f9c5d231c821 - QUAD $0x0000000000841f0f - QUAD $0x62d7041148fdf162 - QUAD $0x6201d7441148fdf1 - QUAD $0x6202d7441148fdf1 - QUAD $0x4803d7441148fdf1 + QUAD $0x48ffffff49e99e7f + QUAD $0xc18948087304f983 + QUAD $0xf983480000010be9 + QUAD $0x00c0e9d231787320 + QUAD $0xc82148c0894c0000 + QUAD $0x9066c057f8c5d231 + QUAD $0x62d70411487cf162 + QUAD $0x6201d74411487cf1 + QUAD $0x6202d74411487cf1 + QUAD $0x4803d74411487cf1 QUAD $0xd875d0394820c283 - QUAD $0x0000cc840fc03949 - QUAD $0xb2840f1cc0f64100 + QUAD $0x0000f2840fc03949 + QUAD $0xd8840f1cc0f64100 QUAD $0x8348c28948000000 QUAD $0xc0214cc889481cc1 - QUAD $0x00401f0fc057f9c5 - QUAD $0xc28348d70411fdc5 - QUAD $0x95e9f275d0394804 - QUAD $0x2148c0894c000000 - QUAD $0x31c81948fdf262c8 - QUAD $0x2e666666666666d2 - QUAD $0x0000000000841f0f - QUAD $0x62d70c1148fdf162 - QUAD $0x6201d74c1148fdf1 - QUAD $0x6202d74c1148fdf1 - QUAD $0x4803d74c1148fdf1 - QUAD $0xd875d0394820c283 - QUAD $0xc0f6415074c03949 - QUAD $0x8348c2894830741c - QUAD $0xc0214cc889481cc1 - QUAD $0x2e6666c8197de2c4 - QUAD $0x0000000000841f0f + QUAD $0x00401f0fc057f8c5 + QUAD $0xc28348d70411fcc5 + QUAD $0xbbe9f275d0394804 + QUAD $0x8348ca8948000000 + QUAD $0xc81948fdf262e0e2 + QUAD $0xc0c68148c7348d48 + QUAD $0x9066c93145000000 + QUAD $0xfdce4c1148fdb162 + QUAD $0xfece4c1148fdb162 + QUAD $0xffce4c1148fdb162 + QUAD $0x49ce0c1148fdb162 + QUAD $0xd875ca394c20c183 + QUAD $0x1cc1f65074d13948 + QUAD $0x03e683c689445974 + QUAD $0xe2c4c10148f12948 + QUAD $0x894cc20148c8197d + QUAD $0x001f0ffce08348c0 QUAD $0xc28348d70c11fdc5 - QUAD $0x3949f275d0394804 - QUAD $0xc70411fbc51a74c0 - QUAD $0x04c748f1ebc0ff48 - QUAD $0xc0ff4800000000c7 - QUAD $0x77f8c5f075c03949 + QUAD $0x8548f275d0394804 + QUAD $0x2e666666661a74f6 + QUAD $0x0000000000841f0f + QUAD $0xc1ff48cf0411fbc5 + QUAD $0x658d48f375c83949 + QUAD $0xc55d5f415e415be8 + QUAD $0x8948d00148c377f8 + QUAD $0x00c704c748ddebc1 + QUAD $0x3949c0ff48000000 + LONG $0xc5f075c0 + BYTE $0xf8 + BYTE $0x77 BYTE $0xc3 RET @@ -735,102 +1339,318 @@ TEXT ·convolveFloat64AVX512(SB), NOSPLIT|NOFRAME, $0-72 MOVQ dst_len+8(FP), CX MOVQ sig_len+32(FP), R8 MOVQ ker_len+56(FP), R9 + // Constant pool appended below the body; the displacements + // above were patched to reach it and the aligned loads made + // unaligned, since nothing promises alignment inside a TEXT. QUAD $0x56415741e5894855 QUAD $0xe483485354415541 - QUAD $0x74894838ec8348f8 - QUAD $0xc0940fc9854d0824 - QUAD $0x40c69c0f40c8294d - QUAD $0x000002b0850fc608 - QUAD $0x48c1394c01708d49 - QUAD $0x8e0ff68548f14e0f - QUAD $0x0fc9854d0000029c - QUAD $0x0c8d4a000001f18e - QUAD $0x07e08341c88945ca - QUAD $0xffffffffffe0ba49 - QUAD $0x214d18ca83497fff - QUAD $0x8d4c0824448b48ca - QUAD $0x8348ca048d4a3858 - QUAD $0x311024448948f8c0 - QUAD $0xfffffffffdbf49db - QUAD $0xfffffcbc491fffff - QUAD $0xfbbd491fffffffff - QUAD $0x481fffffffffffff - QUAD $0xfffffffffffffab8 - QUAD $0xfffffffff9ba481f - QUAD $0xfffff8be491fffff - QUAD $0x7c89481fffffffff - QUAD $0x4c20247489482824 - QUAD $0x906634eb18244c89 - QUAD $0x5c8b4828247c8b48 - QUAD $0x48df0411fbc53024 - QUAD $0x834908c38349c3ff - QUAD $0x4808244c894c08c1 - QUAD $0x4cf339482024748b - QUAD $0x01de840f18244c8b - QUAD $0xc530245c89480000 - QUAD $0x7308f98349c057f9 - QUAD $0x08244c8b4cdb310f - QUAD $0x001f0f000000eae9 - QUAD $0x49db3110247c8b48 - QUAD $0xfffffffffffffeb9 - QUAD $0x2e6666666666661f + QUAD $0xc9854d38ec8348f8 + QUAD $0x28244c894cc0940f + QUAD $0x41c19c0f41c8294d + QUAD $0x0000061a850fc108 + QUAD $0x48c1394c01408d49 + QUAD $0x4810f88348c14e0f + QUAD $0x2444894830247c89 + QUAD $0x8948202474894810 + QUAD $0xf63145087d182454 + QUAD $0x4c8b4800000325e9 + QUAD $0xc78e0fc985482824 + QUAD $0x41ca048d4c000001 + QUAD $0xbb4903e28341ca89 + QUAD $0x7ffffffffffffff8 + QUAD $0x48cb214904cb8349 + QUAD $0x48ca148d48585e8d + QUAD $0x00000010b9f8c283 + QUAD $0xfffffffebf49c031 + QUAD $0xfffdbc491fffffff + QUAD $0xbd491fffffffffff + QUAD $0x1ffffffffffffffc + QUAD $0x0000441f0f6634eb + QUAD $0xfdf16230247c8b48 + QUAD $0x48fdf162c7041148 + QUAD $0x104e8d4901c74c11 + QUAD $0x80ee834880eb8348 + QUAD $0x10244c3b48f0894c + QUAD $0x8949000002948f0f + QUAD $0x730428247c8348ce + QUAD $0xc5c931c057f9c517 + QUAD $0x000000e0e9c957f1 QUAD $0x0000000000841f0f - QUAD $0xc4c8db4c107bc1c4 - QUAD $0xf3c5d0db54107bc1 - QUAD $0x8948c158fbc50f59 - QUAD $0x0c59ebc5ce314cde - QUAD $0x7bc1c4c158fbc5f1 - QUAD $0x4cde8948d8db4c10 - QUAD $0xc5f10c59f3c5fe31 - QUAD $0x4c107bc1c4c158fb - QUAD $0xe6314cde8948e0db - QUAD $0x58fbc5f10c59f3c5 - QUAD $0xe8db4c107bc1c4c1 - QUAD $0xf3c5ee314cde8948 - QUAD $0xc4c158fbc5f10c59 - QUAD $0x8948f0db4c107bc1 - QUAD $0x0c59f3c5c63148de - QUAD $0x7bc1c4c158fbc5f1 - QUAD $0x48de8948f8db4c10 - QUAD $0xc5f10c59f3c5d631 - QUAD $0x0c107bc1c4c158fb - QUAD $0xc5f6314cde8948db - QUAD $0xc158fbc5f10c59f3 - QUAD $0xc0c7834808c38348 - QUAD $0xffff47850fda3949 - QUAD $0x244c8b4cc0854dff - QUAD $0x48fffffec9840f08 - QUAD $0x4800000000dd348d - QUAD $0x49f7294810247c8b - QUAD $0x001f0ff631d91c8d - QUAD $0x59f3c5f30c10fbc5 + QUAD $0x31d78948c057f9c5 + QUAD $0x001f0fc957f1c5c9 + QUAD $0xf162171948fdf262 + QUAD $0xffffa8cb9c5948ed + QUAD $0x62c35848fdf162ff + QUAD $0xffe8cb945948edf1 + QUAD $0xca5848f5f162ffff + QUAD $0x9262f9314dc98949 + QUAD $0xedf162c8141948fd + QUAD $0xffffffb0cb9c5948 + QUAD $0xf162c35848fdf162 + QUAD $0xfffff0cb945948ed + QUAD $0x49ca5848f5f162ff + QUAD $0xfd9262e1314dc989 + QUAD $0x48edf162c8141948 + QUAD $0x62ffffffb8cb9c59 + QUAD $0xedf162c35848fdf1 + QUAD $0xfffffff8cb945948 + QUAD $0x8949ca5848f5f162 + QUAD $0x48fd9262e9314dc9 + QUAD $0x5948edf162c81419 + QUAD $0x5848fdf162ffcb5c + QUAD $0xcb145948edf162c3 + QUAD $0x8348ca5848f5f162 + QUAD $0x3949e0c7834804c1 + QUAD $0x4dffffff41850fcb + QUAD $0xfffffed8840fd285 + QUAD $0x03e1c148ce3c8d48 + QUAD $0xc931c92949d18949 + QUAD $0x0000000000841f0f + QUAD $0xf162111948fdd262 + QUAD $0xfdf162cf1c5948ed + QUAD $0x5948edf162c35848 + QUAD $0x5848f5f16201cf54 + QUAD $0xf8c18349c1ff48ca + QUAD $0xfe8ee9d375ca3949 + QUAD $0x8348f0488d48ffff + QUAD $0x000010b80d7370f9 + QUAD $0x00010ee9f6314500 + QUAD $0xc1ff4804e9c14800 + QUAD $0x49f8e68348ce8948 + QUAD $0x8d4904e6c149f689 + QUAD $0x051048fdf1621046 + QUAD $0xc957f1c500000477 + QUAD $0x0671155948fdf262 + QUAD $0xf78948f889490000 + QUAD $0x49fdd262c846fdc5 + QUAD $0x62c846fdc5c00ca3 + QUAD $0xc501c04ca349fdd2 + QUAD $0xa349fdd262c846fd + QUAD $0x62c846fdc502c04c + QUAD $0xc503c04ca349fdd2 + QUAD $0xa349fdd262c846fd + QUAD $0x62c846fdc504c04c + QUAD $0xc505c04ca349fdd2 + QUAD $0xa349fdd262c846fd + QUAD $0x62c846fdc506c04c + QUAD $0xc507c04ca349fdd2 + QUAD $0xa349fdd262c846fd + QUAD $0x62c846fdc508c04c + QUAD $0xc509c04ca349fdd2 + QUAD $0xa349fdd262c846fd + QUAD $0x62c846fdc50ac04c + QUAD $0xc50bc04ca349fdd2 + QUAD $0xa349fdd262c846fd + QUAD $0x62c846fdc50cc04c + QUAD $0xc50dc04ca349fdd2 + QUAD $0xa349fdd262c846fd + QUAD $0x62c846fdc50ec04c + QUAD $0x620fc04ca349fdd2 + QUAD $0xc78348c2d448fdf1 + QUAD $0x48ffffff31850ff8 + QUAD $0xc52b74c7894cf139 + QUAD $0x1024748b48c057f9 + QUAD $0x62f7041148fdb162 + QUAD $0x4901f7441148fdb1 + QUAD $0x394810c08348c689 + QUAD $0x49f0c08348e57ef0 + QUAD $0x481024748b48c689 + QUAD $0x4c082474894cf089 + QUAD $0x4c2024548b48f029 + QUAD $0x02a88e0f18244c8b + QUAD $0x4828244c8b480000 + QUAD $0x000001ce8e0fc985 + QUAD $0x41ca8941c9048d4d + QUAD $0xfffff8bb4907e283 + QUAD $0xcb21497fffffffff + QUAD $0x1c8d480824448b48 + QUAD $0x4c8d4938c38348c2 + QUAD $0x4818244c8948f8c9 + QUAD $0x2024448948c2048d + QUAD $0xfffffffffffebe49 + QUAD $0xfffffffdbf491fff + QUAD $0xfffcbc491fffffff + QUAD $0xbd491fffffffffff + QUAD $0x1ffffffffffffffb + QUAD $0xfffffffffffab848 + QUAD $0xfffffff9ba481fff + QUAD $0xfff8b9491fffffff + QUAD $0x2ceb1fffffffffff + QUAD $0x748b4830244c8b48 + QUAD $0x48f10411fbc50824 + QUAD $0x834808c38348c6ff + QUAD $0x2474894808202444 + QUAD $0x840f1024743b4808 + QUAD $0x247c8348000001ee + QUAD $0xc057f9c50c730828 + QUAD $0x90000000d2e9c931 + QUAD $0x247c8b48c057f9c5 + QUAD $0x0000441f0fc93118 + QUAD $0xfbc5c8cb4c10fbc5 + QUAD $0x0f59f3c5d0cb5410 + QUAD $0x4cce8948c158fbc5 + QUAD $0xf00c596bc1c4f631 + QUAD $0x4c10fbc5c158fbc5 + QUAD $0xfe314cce8948d8cb + QUAD $0xfbc5f00c5973c1c4 + QUAD $0xe0cb4c10fbc5c158 + QUAD $0xc1c4e6314cce8948 + QUAD $0xc158fbc5f00c5973 + QUAD $0x8948e8cb4c10fbc5 + QUAD $0x5973c1c4ee314cce + QUAD $0xfbc5c158fbc5f00c + QUAD $0x48ce8948f0cb4c10 + QUAD $0xf00c5973c1c4c631 + QUAD $0x4c10fbc5c158fbc5 + QUAD $0xd63148ce8948f8cb + QUAD $0xfbc5f00c5973c1c4 + QUAD $0x48cb0c10fbc5c158 + QUAD $0x73c1c4ce314cce89 + QUAD $0x48c158fbc5f00c59 + QUAD $0x49c0c7834808c183 + QUAD $0xffffff48850fcb39 + QUAD $0xfffeef840fd2854d + QUAD $0x000000cd348d48ff + QUAD $0x294818247c8b4800 + QUAD $0x8d482024748b48f7 + QUAD $0x00401f0ff631ce0c + QUAD $0x59f3c5f10c10fbc5 QUAD $0xc6ff48c158fbc50f - QUAD $0x75f03949f8c78348 - QUAD $0x8348fffffe92e9e7 - QUAD $0x88e9c031077304fe - QUAD $0xffffe0b948000000 - QUAD $0xfe83487fffffffff - QUAD $0x4844ebc031047320 - QUAD $0xc5d231c82148f089 - QUAD $0x0000441f0fc057f9 - QUAD $0x62d7041148fdf162 - QUAD $0x6201d7441148fdf1 - QUAD $0x6202d7441148fdf1 - QUAD $0x4803d7441148fdf1 - QUAD $0xd875d0394820c283 - QUAD $0xc6f6404374c63948 - QUAD $0x8348c2894830741c - QUAD $0xf02148c889481cc1 - QUAD $0x2e666666c057f9c5 + QUAD $0x75f23949f8c78348 + QUAD $0x8348fffffeb2e9e7 + QUAD $0x000000a8820f04f8 + QUAD $0x24448b4c20f88348 + QUAD $0x4854ebc931047308 + QUAD $0x8d4ae0e18348c189 + QUAD $0x0000c0c28148c714 + QUAD $0x90c057f9c5f63100 + QUAD $0xfdf2441148fdf162 + QUAD $0xfef2441148fdf162 + QUAD $0xfff2441148fdf162 + QUAD $0x48f2041148fdf162 + QUAD $0xd875f1394820c683 + QUAD $0x1024748b48c83948 + QUAD $0x8b4c76741ca86874 + QUAD $0x8348c28948082444 + QUAD $0x0149c7348d4afce2 + QUAD $0xf9c5082444894cd0 + QUAD $0x2e6666666666c057 QUAD $0x0000000000841f0f - QUAD $0xc28348d70411fdc5 - QUAD $0x3948f275d0394804 - QUAD $0x00c704c7480d74c6 - QUAD $0xeeebc0ff48000000 - QUAD $0x415c415bd8658d48 - QUAD $0xf8c55d5f415e415d - BYTE $0x77 - BYTE $0xc3 + QUAD $0xc18348ce0411fdc5 + QUAD $0x3948f275ca394804 + QUAD $0x22741024748b48d0 + QUAD $0x0000000000841f0f + QUAD $0x04c7480824448b48 + QUAD $0xc0ff4800000000c7 + QUAD $0xc639480824448948 + QUAD $0x415bd8658d48e675 + QUAD $0x5d5f415e415d415c + QUAD $0x244c0148c377f8c5 + QUAD $0x0000000080cdeb08 + QUAD $0x0000000090000000 + QUAD $0x00000000a0000000 + QUAD $0x00000000b0000000 + QUAD $0x00000000c0000000 + QUAD $0x00000000d0000000 + QUAD $0x00000000e0000000 + QUAD $0x00000000f0000000 + QUAD $0x0000000000000000 + QUAD $0x0000000010000000 + QUAD $0x0000000020000000 + QUAD $0x0000000030000000 + QUAD $0x0000000040000000 + QUAD $0x0000000050000000 + QUAD $0x0000000060000000 + QUAD $0x0000000070000000 + QUAD $0x0000000000000000 + QUAD $0x0000000010000000 + QUAD $0x0000000020000000 + QUAD $0x0000000030000000 + QUAD $0x0000000040000000 + QUAD $0x0000000050000000 + QUAD $0x0000000060000000 + QUAD $0x0000000070000000 + QUAD $0x0000000080000000 + QUAD $0x0000000090000000 + QUAD $0x00000000a0000000 + QUAD $0x00000000b0000000 + QUAD $0x00000000c0000000 + QUAD $0x00000000d0000000 + QUAD $0x00000000e0000000 + QUAD $0x00000000f0000000 + QUAD $0x0000000000000000 + QUAD $0x0000000010000000 + QUAD $0x0000000020000000 + QUAD $0x0000000030000000 + QUAD $0x0000000040000000 + QUAD $0x0000000050000000 + QUAD $0x0000000060000000 + QUAD $0x0000000070000000 + QUAD $0x0000000000000000 + QUAD $0x0000000010000000 + QUAD $0x0000000020000000 + QUAD $0x0000000030000000 + QUAD $0x0000000040000000 + QUAD $0x0000000050000000 + QUAD $0x0000000060000000 + QUAD $0x0000000070000000 + QUAD $0x0000000000000000 + QUAD $0x0000000001000000 + QUAD $0x0000000002000000 + QUAD $0x0000000003000000 + QUAD $0x0000000004000000 + QUAD $0x0000000005000000 + QUAD $0x0000000006000000 + QUAD $0x0000000007000000 + QUAD $0x0000000000000000 + QUAD $0x0000000001000000 + QUAD $0x0000000002000000 + QUAD $0x0000000003000000 + QUAD $0x0000000004000000 + QUAD $0x0000000005000000 + QUAD $0x0000000006000000 + QUAD $0x0000000007000000 + QUAD $0x0000000008000000 + QUAD $0x0000000009000000 + QUAD $0x000000000a000000 + QUAD $0x000000000b000000 + QUAD $0x000000000c000000 + QUAD $0x000000000d000000 + QUAD $0x000000000e000000 + QUAD $0x000000000f000000 + QUAD $0x0000000000000000 + QUAD $0x0000000001000000 + QUAD $0x0000000002000000 + QUAD $0x0000000003000000 + QUAD $0x0000000004000000 + QUAD $0x0000000005000000 + QUAD $0x0000000006000000 + QUAD $0x0000000007000000 + QUAD $0x0000000100000000 + QUAD $0x0000000080000000 + QUAD $0x0000000100000000 + QUAD $0x0000000080000000 + QUAD $0xcbfa53e0a8000000 + QUAD $0x97f4a7c150f1bbcd + QUAD $0x63eefba1f8e3779b + QUAD $0xb97f4a7c15d53369 + QUAD $0x6d1ce4e5b99e3779 + QUAD $0xbb133111ebbf5847 + QUAD $0x000000002094d049 + QUAD $0x0000000008000000 + QUAD $0xb97f4a7c15000000 + QUAD $0x6d1ce4e5b99e3779 + QUAD $0xbb133111ebbf5847 + QUAD $0x000000000094d049 + QUAD $0x00000000083ca000 + QUAD $0xb97f4a7c15000000 + QUAD $0x6d1ce4e5b99e3779 + QUAD $0xbb133111ebbf5847 + QUAD $0x000000001094d049 + QUAD $0x0000000004000000 + BYTE $0x00 + BYTE $0x00 + BYTE $0x00 RET // func correlateFloat64AVX512(dst []float64, sig []float64, ker []float64) @@ -841,73 +1661,277 @@ TEXT ·correlateFloat64AVX512(SB), NOSPLIT|NOFRAME, $0-72 MOVQ dst_len+8(FP), CX MOVQ sig_len+32(FP), R8 MOVQ ker_len+56(FP), R9 + // Constant pool appended below the body; the displacements + // above were patched to reach it and the aligned loads made + // unaligned, since nothing promises alignment inside a TEXT. QUAD $0x294dc0940fc9854d QUAD $0xc20841c29c0f41c8 - QUAD $0x8d49000001fa850f - QUAD $0x4e0f48c1394c0140 - QUAD $0x01e68e0fc08548c1 - QUAD $0x358e0fc9854d0000 - QUAD $0x41e5894855000001 - QUAD $0xf8e4834853564157 - QUAD $0xb84907e183c98944 - QUAD $0x7fffffffffffffe0 - QUAD $0x4cc8214d18c88349 - QUAD $0x1aebdb314538568d - QUAD $0xff49df04117ba1c4 - QUAD $0xc6834808c28349c3 - QUAD $0x00e3840fc3394908 - QUAD $0xdb31c057f9c50000 - QUAD $0x00a2820f08f98349 - QUAD $0x0000441f0f660000 - QUAD $0xc4c8da4c107bc1c4 - QUAD $0xf3c5d0da54107bc1 + QUAD $0x4855000004f8850f + QUAD $0x554156415741e589 + QUAD $0x49f8e48348535441 + QUAD $0x0f48c1394c01408d + QUAD $0x077d10f88348c14e + QUAD $0x4d000002d3e9c931 + QUAD $0x000001778e0fc985 + QUAD $0x4903e08341c88945 + QUAD $0xfffffffffffff8ba + QUAD $0xca214d04ca83497f + QUAD $0x0010bf41585e8d4c + QUAD $0xf38948f631450000 + QUAD $0x2e666666666635eb + QUAD $0x0000000000841f0f + QUAD $0x62f7041148fdb162 + QUAD $0x4c01f74c1148fdb1 + QUAD $0x4880eb834910798d + QUAD $0x3949ce894980eb83 + QUAD $0x4c0000026b8f0fc7 + QUAD $0x3145c057f9c5f989 + QUAD $0xf98349c957f1c5ff + QUAD $0x90000000b7820f04 + QUAD $0x62fa141948fdb262 + QUAD $0xffa8fb9c5948ed91 + QUAD $0xc35848fdf162ffff + QUAD $0xe8fb945948ed9162 + QUAD $0x5848f5f162ffffff + QUAD $0xfa541948fdb262ca + QUAD $0xfb9c5948ed916201 + QUAD $0x48fdf162ffffffb0 + QUAD $0x945948ed9162c358 + QUAD $0xf5f162fffffff0fb + QUAD $0x1948fdb262ca5848 + QUAD $0x5948ed916202fa54 + QUAD $0x9162ffffffb8fb9c + QUAD $0xfffff8fb945948ed + QUAD $0x62c35848fdf162ff + QUAD $0xfdb262ca5848f5f1 + QUAD $0xed916203fa541948 + QUAD $0xfdf162fffb5c5948 + QUAD $0x5948ed9162c35848 + QUAD $0xca5848f5f162fb14 + QUAD $0x0ffa394d04c78349 + QUAD $0xc0854dffffff5385 + QUAD $0x8d4effffff0a840f + QUAD $0x3145fa3c8d4efb24 + QUAD $0x2e666666666666ed + QUAD $0x0000000000841f0f + QUAD $0x62ef141948fd9262 + QUAD $0xf162ec1c5948ed91 + QUAD $0x48ed9162c35848fd + QUAD $0x48f5f16201ec5459 + QUAD $0xe8394dc5ff49ca58 + QUAD $0x4cfffffec1e9d675 + QUAD $0x7370fa8349f0508d + QUAD $0x3100000010b8410d + QUAD $0xc1490000010ee9c9 + QUAD $0xd3894dc2ff4904ea + QUAD $0x48d9894cf8e38349 + QUAD $0x6210418d4c04e1c1 + QUAD $0x00045e051048fdf1 + QUAD $0xfdf262c957f1c500 + QUAD $0x4c000005a8155948 + QUAD $0x0000441f0f66db89 + QUAD $0x49fdf262c846fdc5 + QUAD $0x62c846fdc5c70ca3 + QUAD $0xc501c74ca349fdf2 + QUAD $0xa349fdf262c846fd + QUAD $0x62c846fdc502c74c + QUAD $0xc503c74ca349fdf2 + QUAD $0xa349fdf262c846fd + QUAD $0x62c846fdc504c74c + QUAD $0xc505c74ca349fdf2 + QUAD $0xa349fdf262c846fd + QUAD $0x62c846fdc506c74c + QUAD $0xc507c74ca349fdf2 + QUAD $0xa349fdf262c846fd + QUAD $0x62c846fdc508c74c + QUAD $0xc509c74ca349fdf2 + QUAD $0xa349fdf262c846fd + QUAD $0x62c846fdc50ac74c + QUAD $0xc50bc74ca349fdf2 + QUAD $0xa349fdf262c846fd + QUAD $0x62c846fdc50cc74c + QUAD $0xc50dc74ca349fdf2 + QUAD $0xa349fdf262c846fd + QUAD $0x62c846fdc50ec74c + QUAD $0x620fc74ca349fdf2 + QUAD $0xc38348c2d448fdf1 + QUAD $0x4dffffff31850ff8 + QUAD $0xc057f9c52e74da39 + QUAD $0x0000000000841f0f + QUAD $0x62cf041148fdf162 + QUAD $0x4c01cf441148fdf1 + QUAD $0x394910c08349c189 + QUAD $0x4cf0c08349e57ec0 + QUAD $0xc82949c08949c189 + QUAD $0x854d000001e28e0f + QUAD $0x45000001268e0fc9 + QUAD $0xba4907e08341c889 + QUAD $0x7ffffffffffffff8 + QUAD $0x49ce1c8d4cca214d + QUAD $0xebce348d4838c383 + QUAD $0x00000000801f0f20 + QUAD $0xc1ff48cf0411fbc5 + QUAD $0x08c6834808c38349 + QUAD $0x000197840fc13948 + QUAD $0x49db31c057f9c500 + QUAD $0x0000a3820f08f983 + QUAD $0x00000000801f0f00 + QUAD $0xc4c8db4c107bc1c4 + QUAD $0xf3c5d0db54107bc1 QUAD $0xc5c158fbc5da0c59 QUAD $0x58fbc508da4c59eb - QUAD $0xd8da4c107bc1c4c1 + QUAD $0xd8db4c107bc1c4c1 QUAD $0xc1c410da4c59f3c5 - QUAD $0x59ebc5e0da54107b + QUAD $0x59ebc5e0db54107b QUAD $0xc5c158fbc518da54 QUAD $0x4c107bc1c4c258fb - QUAD $0x20da4c59f3c5e8da + QUAD $0x20da4c59f3c5e8db QUAD $0x107bc1c4c158fbc5 - QUAD $0xda4c59f3c5f0da4c + QUAD $0xda4c59f3c5f0db4c QUAD $0x7bc1c4c158fbc528 - QUAD $0x4c59f3c5f8da4c10 - QUAD $0xda14107bc1c430da + QUAD $0x4c59f3c5f8db4c10 + QUAD $0xdb14107bc1c430da QUAD $0x4c59ebc5c158fbc5 QUAD $0x8348c158fbc538da - QUAD $0x6d850fd8394908c3 - QUAD $0x840fc98548ffffff + QUAD $0x6d850fda394908c3 + QUAD $0x840fc0854dffffff QUAD $0xda348d4cffffff34 QUAD $0x66ff3145de1c8d48 QUAD $0x0000000000841f0f QUAD $0x81c4fb0c107ba1c4 QUAD $0xc158fbc5fe0c5973 - QUAD $0xe875f9394cc7ff49 - QUAD $0x658d48ffffff03e9 - QUAD $0xc35d5f415e415be8 - QUAD $0xc931077304f88348 - QUAD $0xe0ba480000008be9 - QUAD $0x487fffffffffffff - QUAD $0xebc931047320f883 - QUAD $0x31d12148c1894848 - QUAD $0x2e6666c057f8c5f6 + QUAD $0xe875f8394dc7ff49 + QUAD $0xf88349ffffff03e9 + QUAD $0x4900000099820f04 + QUAD $0xebd231047320f883 + QUAD $0xe0e28348c2894c52 + QUAD $0xc0c68148cf348d48 + QUAD $0xf9c5c93145000000 + QUAD $0x0000441f0f66c057 + QUAD $0xfdce441148fdb162 + QUAD $0xfece441148fdb162 + QUAD $0xffce441148fdb162 + QUAD $0x49ce041148fdb162 + QUAD $0xd875ca394c20c183 + QUAD $0xc0f6415374d03949 + QUAD $0x8348c6894c5f741c + QUAD $0x0148cf0c8d4cfce6 + QUAD $0x2e6666c057f9c5f1 QUAD $0x0000000000841f0f - QUAD $0x62f70411487cf162 - QUAD $0x6201f74411487cf1 - QUAD $0x6202f74411487cf1 - QUAD $0x4803f74411487cf1 - QUAD $0xd875f1394820c683 - QUAD $0x741ca84374c83948 - QUAD $0x1cc28348ce89482f - QUAD $0xf8c5c12148d18948 - QUAD $0x2e6666666666c057 + QUAD $0x8348d104117dc1c4 + QUAD $0x49f175d6394804c2 + QUAD $0x2e6666661c74f039 QUAD $0x0000000000841f0f - QUAD $0xc68348f70411fcc5 - QUAD $0x0bebf275f1394804 QUAD $0x00000000cf04c748 QUAD $0xf075c83948c1ff48 - LONG $0xc377f8c5 + QUAD $0x415c415bd8658d48 + QUAD $0xf8c55d5f415e415d + QUAD $0x80d9ebd10148c377 + QUAD $0x9000000000000000 + QUAD $0xa000000000000000 + QUAD $0xb000000000000000 + QUAD $0xc000000000000000 + QUAD $0xd000000000000000 + QUAD $0xe000000000000000 + QUAD $0xf000000000000000 + QUAD $0x0000000000000000 + QUAD $0x1000000000000000 + QUAD $0x2000000000000000 + QUAD $0x3000000000000000 + QUAD $0x4000000000000000 + QUAD $0x5000000000000000 + QUAD $0x6000000000000000 + QUAD $0x7000000000000000 + QUAD $0x0000000000000000 + QUAD $0x1000000000000000 + QUAD $0x2000000000000000 + QUAD $0x3000000000000000 + QUAD $0x4000000000000000 + QUAD $0x5000000000000000 + QUAD $0x6000000000000000 + QUAD $0x7000000000000000 + QUAD $0x8000000000000000 + QUAD $0x9000000000000000 + QUAD $0xa000000000000000 + QUAD $0xb000000000000000 + QUAD $0xc000000000000000 + QUAD $0xd000000000000000 + QUAD $0xe000000000000000 + QUAD $0xf000000000000000 + QUAD $0x0000000000000000 + QUAD $0x1000000000000000 + QUAD $0x2000000000000000 + QUAD $0x3000000000000000 + QUAD $0x4000000000000000 + QUAD $0x5000000000000000 + QUAD $0x6000000000000000 + QUAD $0x7000000000000000 + QUAD $0x0000000000000000 + QUAD $0x1000000000000000 + QUAD $0x2000000000000000 + QUAD $0x3000000000000000 + QUAD $0x4000000000000000 + QUAD $0x5000000000000000 + QUAD $0x6000000000000000 + QUAD $0x7000000000000000 + QUAD $0x0000000000000000 + QUAD $0x0100000000000000 + QUAD $0x0200000000000000 + QUAD $0x0300000000000000 + QUAD $0x0400000000000000 + QUAD $0x0500000000000000 + QUAD $0x0600000000000000 + QUAD $0x0700000000000000 + QUAD $0x0000000000000000 + QUAD $0x0100000000000000 + QUAD $0x0200000000000000 + QUAD $0x0300000000000000 + QUAD $0x0400000000000000 + QUAD $0x0500000000000000 + QUAD $0x0600000000000000 + QUAD $0x0700000000000000 + QUAD $0x0800000000000000 + QUAD $0x0900000000000000 + QUAD $0x0a00000000000000 + QUAD $0x0b00000000000000 + QUAD $0x0c00000000000000 + QUAD $0x0d00000000000000 + QUAD $0x0e00000000000000 + QUAD $0x0f00000000000000 + QUAD $0x0000000000000000 + QUAD $0x0100000000000000 + QUAD $0x0200000000000000 + QUAD $0x0300000000000000 + QUAD $0x0400000000000000 + QUAD $0x0500000000000000 + QUAD $0x0600000000000000 + QUAD $0x0700000000000000 + QUAD $0x0000000000000000 + QUAD $0x8000000000000001 + QUAD $0x0000000000000000 + QUAD $0x8000000000000001 + QUAD $0xa800000000000000 + QUAD $0x50f1bbcdcbfa53e0 + QUAD $0xf8e3779b97f4a7c1 + QUAD $0x15d5336963eefba1 + QUAD $0xb99e3779b97f4a7c + QUAD $0xebbf58476d1ce4e5 + QUAD $0x2094d049bb133111 + QUAD $0x0800000000000000 + QUAD $0x1500000000000000 + QUAD $0xb99e3779b97f4a7c + QUAD $0xebbf58476d1ce4e5 + QUAD $0x0094d049bb133111 + QUAD $0x083ca00000000000 + QUAD $0x1500000000000000 + QUAD $0xb99e3779b97f4a7c + QUAD $0xebbf58476d1ce4e5 + QUAD $0x1094d049bb133111 + QUAD $0x0400000000000000 + LONG $0x00000000 + BYTE $0x00 + BYTE $0x00 + BYTE $0x00 RET // func tileFloat32AVX512(dst []float32, pattern []float32) @@ -3526,21 +4550,25 @@ TEXT ·movingAverageFloat32AVX512(SB), NOSPLIT|NOFRAME, $0-56 MOVQ width+48(FP), R8 QUAD $0x294cc09e0fc0854d QUAD $0xc10841c19c0f41c1 - QUAD $0x8d48000000e3850f - QUAD $0x4e0f48ca39480141 - QUAD $0x00cf8e0fc08548c2 - QUAD $0x4853e58948550000 - QUAD $0xc02a82c1c4f8e483 - QUAD $0x83c28944ff488d49 - QUAD $0xfffffff8b94907e2 - QUAD $0x4cc1214d7fffffff - QUAD $0x22ebd231451c468d + QUAD $0x48550000010b850f + QUAD $0x554156415741e589 + QUAD $0x48f8e48348535441 + QUAD $0xfffffffffffff8b8 + QUAD $0xca394801498d4c7f + QUAD $0x10f98349ca4e0f4c + QUAD $0xc931000000df8d0f + QUAD $0x0000c28d0fc9394c + QUAD $0x8d49c02a82c1c400 + QUAD $0xe28341c28945ff50 + QUAD $0x8e048d4cc0214c07 + QUAD $0x8e348d481cc08349 + QUAD $0x2e666666666627eb QUAD $0x0000000000841f0f - QUAD $0x117aa1c4c85ef2c5 - QUAD $0xc08349c2ff49970c - QUAD $0xc2394904c6834804 - QUAD $0x3145c957f0c57974 - QUAD $0x664e7207f98348db + QUAD $0x0c11fac5c85ef2c5 + QUAD $0x04c08349c1ff488f + QUAD $0x74c9394c04c68348 + QUAD $0xdb3145c957f0c57a + QUAD $0x2e664f7207fa8348 QUAD $0x0000000000841f0f QUAD $0xc4e4984c587281c4 QUAD $0x81c4e8984c587281 @@ -3549,14 +4577,46 @@ TEXT ·movingAverageFloat32AVX512(SB), NOSPLIT|NOFRAME, $0-56 QUAD $0x4c587281c4f4984c QUAD $0x984c587281c4f898 QUAD $0x49980c587281c4fc - QUAD $0xc075d9394d08c383 - QUAD $0x1c8d4e8b74d28548 + QUAD $0xc075d8394c08c383 + QUAD $0x1c8d4e8b74d2854d QUAD $0x0000441f0fdb319e QUAD $0xff489b0c5872c1c4 - QUAD $0x6de9f275da3948c3 - QUAD $0x5bf8658d48ffffff - BYTE $0x5d - BYTE $0xc3 + QUAD $0x6de9f275da3949c3 + QUAD $0x5bd8658d48ffffff + QUAD $0x5f415e415d415c41 + QUAD $0x82c1c4c377f8c55d + QUAD $0xc018487df262c02a + QUAD $0x41c28945ff508d49 + QUAD $0x2149c3894d07e283 + QUAD $0x10bc411c5e8d48c3 + QUAD $0x8949ff3145000000 + QUAD $0x2e6666666632ebf6 + QUAD $0x0000000000841f0f + QUAD $0xb162c85e4874f162 + QUAD $0x618d4cbf0c11487c + QUAD $0xc6834940c3834810 + QUAD $0x0fcc394dcf894940 + QUAD $0xe1894cfffffec38f + QUAD $0x48e43145c957f0c5 + QUAD $0x2e6666717207fa83 + QUAD $0x0000000000841f0f + QUAD $0xe4a38c584874b162 + QUAD $0x584874b162ffffff + QUAD $0xb162ffffffe8a38c + QUAD $0xffffeca38c584874 + QUAD $0xa38c584874b162ff + QUAD $0x4874b162fffffff0 + QUAD $0x62fffffff4a38c58 + QUAD $0xfff8a38c584874b1 + QUAD $0x8c584874b162ffff + QUAD $0x74b162fffffffca3 + QUAD $0x08c48349a30c5848 + QUAD $0xd2854da375e3394d + QUAD $0x8d4fffffff5a840f + QUAD $0x001f0fed3145a624 + QUAD $0x49ac0c5848749162 + QUAD $0xe9f175ea394dc5ff + LONG $0xffffff3c RET // func shiftDivFloat32AVX512(dst []float32, a []float32, shift float32, denom float32) @@ -3671,21 +4731,25 @@ TEXT ·movingAverageFloat64AVX512(SB), NOSPLIT|NOFRAME, $0-56 MOVQ width+48(FP), R8 QUAD $0x294cc09e0fc0854d QUAD $0xc10841c19c0f41c1 - QUAD $0x8d48000000e3850f - QUAD $0x4e0f48ca39480141 - QUAD $0x00cf8e0fc08548c2 - QUAD $0x4853e58948550000 - QUAD $0xc02a83c1c4f8e483 - QUAD $0x83c28944ff488d49 - QUAD $0xfffffff8b94907e2 - QUAD $0x4cc1214d7fffffff - QUAD $0x22ebd2314538468d + QUAD $0x48550000010b850f + QUAD $0x554156415741e589 + QUAD $0x48f8e48348535441 + QUAD $0xfffffffffffff8b8 + QUAD $0xca394801498d4c7f + QUAD $0x10f98349ca4e0f4c + QUAD $0xc931000000df8d0f + QUAD $0x0000c28d0fc9394c + QUAD $0x8d49c02a83c1c400 + QUAD $0xe28341c28945ff50 + QUAD $0xce048d4cc0214c07 + QUAD $0xce348d4838c08349 + QUAD $0x2e666666666627eb QUAD $0x0000000000841f0f - QUAD $0x117ba1c4c85ef3c5 - QUAD $0xc08349c2ff49d70c - QUAD $0xc2394908c6834808 - QUAD $0x3145c957f1c57974 - QUAD $0x664e7207f98348db + QUAD $0x0c11fbc5c85ef3c5 + QUAD $0x08c08349c1ff48cf + QUAD $0x74c9394c08c68348 + QUAD $0xdb3145c957f1c57a + QUAD $0x2e664f7207fa8348 QUAD $0x0000000000841f0f QUAD $0xc4c8d84c587381c4 QUAD $0x81c4d0d84c587381 @@ -3694,14 +4758,61 @@ TEXT ·movingAverageFloat64AVX512(SB), NOSPLIT|NOFRAME, $0-56 QUAD $0x4c587381c4e8d84c QUAD $0xd84c587381c4f0d8 QUAD $0x49d80c587381c4f8 - QUAD $0xc075d9394d08c383 - QUAD $0x1c8d4e8b74d28548 + QUAD $0xc075d8394c08c383 + QUAD $0x1c8d4e8b74d2854d QUAD $0x0000441f0fdb31de QUAD $0xff48db0c5873c1c4 - QUAD $0x6de9f275da3948c3 - QUAD $0x5bf8658d48ffffff - BYTE $0x5d - BYTE $0xc3 + QUAD $0x6de9f275da3949c3 + QUAD $0x5bd8658d48ffffff + QUAD $0x5f415e415d415c41 + QUAD $0x83c1c4c377f8c55d + QUAD $0xc01948fdf262c02a + QUAD $0x41c28945ff508d49 + QUAD $0x2149c3894d07e283 + QUAD $0x10bc41785e8d48c3 + QUAD $0x8949ff3145000000 + QUAD $0x2e6666666640ebf6 + QUAD $0x0000000000841f0f + QUAD $0xb162c85e48f5f162 + QUAD $0xedf162ff0c1148fd + QUAD $0x1148fdb162c85e48 + QUAD $0x4810618d4c01ff4c + QUAD $0x4980ee834980eb83 + QUAD $0xb58f0fcc394dcf89 + QUAD $0xf1c5e1894cfffffe + QUAD $0x57e9c5e43145c957 + QUAD $0xc4820f07fa8348d2 + QUAD $0x0000441f0f000000 + QUAD $0x88e38c5848f5b162 + QUAD $0x5848edb162ffffff + QUAD $0xb162ffffffc8e394 + QUAD $0xffff90e38c5848f5 + QUAD $0xe3945848edb162ff + QUAD $0x48f5b162ffffffd0 + QUAD $0x62ffffff98e38c58 + QUAD $0xffd8e3945848edb1 + QUAD $0x8c5848f5b162ffff + QUAD $0xedb162ffffffa0e3 + QUAD $0xffffffe0e3945848 + QUAD $0xa8e38c5848f5b162 + QUAD $0x5848edb162ffffff + QUAD $0xb162ffffffe8e394 + QUAD $0xffffb0e38c5848f5 + QUAD $0xe3945848edb162ff + QUAD $0x48f5b162fffffff0 + QUAD $0x62ffffffb8e38c58 + QUAD $0xfff8e3945848edb1 + QUAD $0x4c5848f5b162ffff + QUAD $0x145848edb162ffe3 + QUAD $0xe3394d08c48349e3 + QUAD $0x854dffffff4a850f + QUAD $0x4ffffffef1840fd2 + QUAD $0x2e66ed3145e6248d + QUAD $0x0000000000841f0f + QUAD $0x62ec0c5848f59162 + QUAD $0x4901ec545848ed91 + QUAD $0xe9e975ea394dc5ff + LONG $0xfffffec4 RET // func shiftDivFloat64AVX512(dst []float64, a []float64, shift float64, denom float64) @@ -3777,17 +4888,17 @@ TEXT ·randomU64AVX512(SB), NOSPLIT|NOFRAME, $0-32 QUAD $0x8348c67c48fdf262 QUAD $0x93e9c031077320fa QUAD $0xf162d08948000001 - QUAD $0x000002a50dd458fd - QUAD $0x02a315d458fdf162 + QUAD $0x000002c50dd458fd + QUAD $0x02c315d458fdf162 QUAD $0x1dd458fdf1620000 - QUAD $0x62c82148000002a1 - QUAD $0x000314256f48fef1 + QUAD $0x62c82148000002c1 + QUAD $0x0004b4256f48fef1 QUAD $0x48fdf262c0314500 - QUAD $0xf2620000028f2d59 - QUAD $0x0000028d355948fd - QUAD $0x028b3d5948fdf262 + QUAD $0xf262000002af2d59 + QUAD $0x000002ad355948fd + QUAD $0x02ab3d5948fdf262 QUAD $0x055948fd72620000 - QUAD $0x2e66666600000289 + QUAD $0x2e666666000002a9 QUAD $0x0000000000841f0f QUAD $0x7162cd4048dd7262 QUAD $0x48b57162d0d448b5 @@ -3829,12 +4940,12 @@ TEXT ·randomU64AVX512(SB), NOSPLIT|NOFRAME, $0-32 QUAD $0x8949000000ac840f QUAD $0xc8894818c18348c0 QUAD $0x7c48fdd262d02148 - QUAD $0x920deb48f5f162c8 - QUAD $0x5948fdf262000001 - QUAD $0xfdf2620000011015 - QUAD $0x620000010e1d5948 - QUAD $0x00010c255948fdf2 - QUAD $0x122d5948fdf26200 + QUAD $0x320deb48f5f162c8 + QUAD $0x5948fdf262000003 + QUAD $0xfdf2620000013015 + QUAD $0x620000012e1d5948 + QUAD $0x00012c255948fdf2 + QUAD $0x322d5948fdf26200 QUAD $0x2e66666666000001 QUAD $0x0000000000841f0f QUAD $0xf162f24048f5f262 @@ -3862,7 +4973,11 @@ TEXT ·randomU64AVX512(SB), NOSPLIT|NOFRAME, $0-32 QUAD $0x894cda314d1feac1 QUAD $0xce014cc0ff48c714 QUAD $0xec8948cb75c23948 - QUAD $0x53e0a8c377f8c55d + QUAD $0x000100c377f8c55d + QUAD $0x0000800000000000 + QUAD $0x0001000000000000 + QUAD $0x0000800000000000 + QUAD $0x53e0a80000000000 QUAD $0xa7c150f1bbcdcbfa QUAD $0xfba1f8e3779b97f4 QUAD $0x4a7c15d5336963ee @@ -3880,6 +4995,54 @@ TEXT ·randomU64AVX512(SB), NOSPLIT|NOFRAME, $0-32 QUAD $0x3111ebbf58476d1c QUAD $0x00001094d049bb13 QUAD $0x0000040000000000 + QUAD $0x0000800000000000 + QUAD $0x0000900000000000 + QUAD $0x0000a00000000000 + QUAD $0x0000b00000000000 + QUAD $0x0000c00000000000 + QUAD $0x0000d00000000000 + QUAD $0x0000e00000000000 + QUAD $0x0000f00000000000 + QUAD $0x0000000000000000 + QUAD $0x0000100000000000 + QUAD $0x0000200000000000 + QUAD $0x0000300000000000 + QUAD $0x0000400000000000 + QUAD $0x0000500000000000 + QUAD $0x0000600000000000 + QUAD $0x0000700000000000 + QUAD $0x0000000000000000 + QUAD $0x0000100000000000 + QUAD $0x0000200000000000 + QUAD $0x0000300000000000 + QUAD $0x0000400000000000 + QUAD $0x0000500000000000 + QUAD $0x0000600000000000 + QUAD $0x0000700000000000 + QUAD $0x0000800000000000 + QUAD $0x0000900000000000 + QUAD $0x0000a00000000000 + QUAD $0x0000b00000000000 + QUAD $0x0000c00000000000 + QUAD $0x0000d00000000000 + QUAD $0x0000e00000000000 + QUAD $0x0000f00000000000 + QUAD $0x0000000000000000 + QUAD $0x0000100000000000 + QUAD $0x0000200000000000 + QUAD $0x0000300000000000 + QUAD $0x0000400000000000 + QUAD $0x0000500000000000 + QUAD $0x0000600000000000 + QUAD $0x0000700000000000 + QUAD $0x0000000000000000 + QUAD $0x0000100000000000 + QUAD $0x0000200000000000 + QUAD $0x0000300000000000 + QUAD $0x0000400000000000 + QUAD $0x0000500000000000 + QUAD $0x0000600000000000 + QUAD $0x0000700000000000 QUAD $0x0000000000000000 QUAD $0x0000010000000000 QUAD $0x0000020000000000 @@ -3930,14 +5093,14 @@ TEXT ·randomF64AVX512(SB), NOSPLIT|NOFRAME, $0-32 QUAD $0xb848000000c3e9c0 QUAD $0x7ffffffffffffff8 QUAD $0x6f48fef162d02148 - QUAD $0xfdf2620000017605 + QUAD $0xfdf262000002f605 QUAD $0xfdf262c931ce7c48 - QUAD $0x6200000264155948 - QUAD $0x0002621d5948fdf2 - QUAD $0x60255948fdf26200 - QUAD $0x1948fdf262000002 - QUAD $0xfdf2620000025e2d - QUAD $0x900000025c355948 + QUAD $0x6200000404155948 + QUAD $0x0004021d5948fdf2 + QUAD $0x00255948fdf26200 + QUAD $0x1948fdf262000004 + QUAD $0xfdf262000003fe2d + QUAD $0x90000003fc355948 QUAD $0xf162fa4048fdf262 QUAD $0x48bdf162f9d448c5 QUAD $0xef48bdf1621ed773 @@ -3957,7 +5120,7 @@ TEXT ·randomF64AVX512(SB), NOSPLIT|NOFRAME, $0-32 QUAD $0x1ce4e5b9b948ce29 QUAD $0x11ebb849bf58476d QUAD $0xfbc594d049bb1331 - QUAD $0xb949000001b50510 + QUAD $0xb949000003550510 QUAD $0x9e3779b97f4a7c15 QUAD $0x0000000000841f0f QUAD $0x491eeac149f28949 @@ -3969,7 +5132,55 @@ TEXT ·randomF64AVX512(SB), NOSPLIT|NOFRAME, $0-32 QUAD $0xc70c11fbc5c859f3 QUAD $0x3948ce014cc0ff48 QUAD $0xc55dec8948bd75c2 - QUAD $0x0000000000c377f8 + QUAD $0x0000000080c377f8 + QUAD $0x0000000090000000 + QUAD $0x00000000a0000000 + QUAD $0x00000000b0000000 + QUAD $0x00000000c0000000 + QUAD $0x00000000d0000000 + QUAD $0x00000000e0000000 + QUAD $0x00000000f0000000 + QUAD $0x0000000000000000 + QUAD $0x0000000010000000 + QUAD $0x0000000020000000 + QUAD $0x0000000030000000 + QUAD $0x0000000040000000 + QUAD $0x0000000050000000 + QUAD $0x0000000060000000 + QUAD $0x0000000070000000 + QUAD $0x0000000000000000 + QUAD $0x0000000010000000 + QUAD $0x0000000020000000 + QUAD $0x0000000030000000 + QUAD $0x0000000040000000 + QUAD $0x0000000050000000 + QUAD $0x0000000060000000 + QUAD $0x0000000070000000 + QUAD $0x0000000080000000 + QUAD $0x0000000090000000 + QUAD $0x00000000a0000000 + QUAD $0x00000000b0000000 + QUAD $0x00000000c0000000 + QUAD $0x00000000d0000000 + QUAD $0x00000000e0000000 + QUAD $0x00000000f0000000 + QUAD $0x0000000000000000 + QUAD $0x0000000010000000 + QUAD $0x0000000020000000 + QUAD $0x0000000030000000 + QUAD $0x0000000040000000 + QUAD $0x0000000050000000 + QUAD $0x0000000060000000 + QUAD $0x0000000070000000 + QUAD $0x0000000000000000 + QUAD $0x0000000010000000 + QUAD $0x0000000020000000 + QUAD $0x0000000030000000 + QUAD $0x0000000040000000 + QUAD $0x0000000050000000 + QUAD $0x0000000060000000 + QUAD $0x0000000070000000 + QUAD $0x0000000000000000 QUAD $0x0000000001000000 QUAD $0x0000000002000000 QUAD $0x0000000003000000 @@ -4001,6 +5212,10 @@ TEXT ·randomF64AVX512(SB), NOSPLIT|NOFRAME, $0-32 QUAD $0x0000000005000000 QUAD $0x0000000006000000 QUAD $0x0000000007000000 + QUAD $0x0000000100000000 + QUAD $0x0000000080000000 + QUAD $0x0000000100000000 + QUAD $0x0000000080000000 QUAD $0xcbfa53e0a8000000 QUAD $0x97f4a7c150f1bbcd QUAD $0x63eefba1f8e3779b @@ -4040,15 +5255,15 @@ TEXT ·randomF32AVX512(SB), NOSPLIT|NOFRAME, $0-32 QUAD $0xc031077310fa8348 QUAD $0xd089480000011ee9 QUAD $0x6f48fef162c82148 - QUAD $0xfef162000002bc05 - QUAD $0x62000002f20d6f48 + QUAD $0xfef1620000043c05 + QUAD $0x62000004720d6f48 QUAD $0xc03145d67c48fdf2 - QUAD $0x03871d5948fdf262 + QUAD $0x05271d5948fdf262 QUAD $0x255948fdf2620000 - QUAD $0x48fdf26200000385 - QUAD $0xf262000003832d59 - QUAD $0x000003913518487d - QUAD $0x03773d5948fdf262 + QUAD $0x48fdf26200000525 + QUAD $0xf262000005232d59 + QUAD $0x000005313518487d + QUAD $0x05173d5948fdf262 QUAD $0x0000441f0f660000 QUAD $0x7262c34048fd7262 QUAD $0x48b57162cb4048f5 @@ -4078,12 +5293,12 @@ TEXT ·randomF32AVX512(SB), NOSPLIT|NOFRAME, $0-32 QUAD $0xd02148c889480cc1 QUAD $0xd262c67c28fdf262 QUAD $0x0debf5c5c87c28fd - QUAD $0x597de2c4000002a1 - QUAD $0x7de2c40000026c15 - QUAD $0xe2c40000026b1d59 - QUAD $0xc40000026a25597d - QUAD $0x000002792d1879e2 - QUAD $0x00026835597de2c4 + QUAD $0x597de2c400000441 + QUAD $0x7de2c40000040c15 + QUAD $0xe2c40000040b1d59 + QUAD $0xc40000040a25597d + QUAD $0x000004192d1879e2 + QUAD $0x00040835597de2c4 QUAD $0x2e66666666666600 QUAD $0x0000000000841f0f QUAD $0xc5c5fa4028f5f262 @@ -4101,7 +5316,7 @@ TEXT ·randomF32AVX512(SB), NOSPLIT|NOFRAME, $0-32 QUAD $0x476d1ce4e5b9b948 QUAD $0x133111ebb849bf58 QUAD $0x0510fac594d049bb - QUAD $0x7c15b949000001e5 + QUAD $0x7c15b94900000385 QUAD $0x2e669e3779b97f4a QUAD $0x0000000000841f0f QUAD $0x491eeac149f28949 @@ -4112,7 +5327,55 @@ TEXT ·randomF32AVX512(SB), NOSPLIT|NOFRAME, $0-32 QUAD $0xff48870c11fac5c8 QUAD $0x75c23948ce014cc0 QUAD $0x77f8c55dec8948c7 - QUAD $0x00000000000000c3 + QUAD $0x00000000000080c3 + QUAD $0x0000000000009000 + QUAD $0x000000000000a000 + QUAD $0x000000000000b000 + QUAD $0x000000000000c000 + QUAD $0x000000000000d000 + QUAD $0x000000000000e000 + QUAD $0x000000000000f000 + QUAD $0x0000000000000000 + QUAD $0x0000000000001000 + QUAD $0x0000000000002000 + QUAD $0x0000000000003000 + QUAD $0x0000000000004000 + QUAD $0x0000000000005000 + QUAD $0x0000000000006000 + QUAD $0x0000000000007000 + QUAD $0x0000000000000000 + QUAD $0x0000000000001000 + QUAD $0x0000000000002000 + QUAD $0x0000000000003000 + QUAD $0x0000000000004000 + QUAD $0x0000000000005000 + QUAD $0x0000000000006000 + QUAD $0x0000000000007000 + QUAD $0x0000000000008000 + QUAD $0x0000000000009000 + QUAD $0x000000000000a000 + QUAD $0x000000000000b000 + QUAD $0x000000000000c000 + QUAD $0x000000000000d000 + QUAD $0x000000000000e000 + QUAD $0x000000000000f000 + QUAD $0x0000000000000000 + QUAD $0x0000000000001000 + QUAD $0x0000000000002000 + QUAD $0x0000000000003000 + QUAD $0x0000000000004000 + QUAD $0x0000000000005000 + QUAD $0x0000000000006000 + QUAD $0x0000000000007000 + QUAD $0x0000000000000000 + QUAD $0x0000000000001000 + QUAD $0x0000000000002000 + QUAD $0x0000000000003000 + QUAD $0x0000000000004000 + QUAD $0x0000000000005000 + QUAD $0x0000000000006000 + QUAD $0x0000000000007000 + QUAD $0x0000000000000000 QUAD $0x0000000000000100 QUAD $0x0000000000000200 QUAD $0x0000000000000300 @@ -4144,6 +5407,10 @@ TEXT ·randomF32AVX512(SB), NOSPLIT|NOFRAME, $0-32 QUAD $0x0000000000000500 QUAD $0x0000000000000600 QUAD $0x0000000000000700 + QUAD $0x0000000000010000 + QUAD $0x0000000000008000 + QUAD $0x0000000000010000 + QUAD $0x0000000000008000 QUAD $0xbbcdcbfa53e0a800 QUAD $0x779b97f4a7c150f1 QUAD $0x336963eefba1f8e3 diff --git a/internal/amd64/numeric_sse2_amd64.s b/internal/amd64/numeric_sse2_amd64.s index a4fee43..5bac7c1 100644 --- a/internal/amd64/numeric_sse2_amd64.s +++ b/internal/amd64/numeric_sse2_amd64.s @@ -176,24 +176,90 @@ TEXT ·polyEvalFloat32SSE2(SB), NOSPLIT|NOFRAME, $0-72 MOVQ x_len+32(FP), R8 MOVQ coeffs_len+56(FP), R9 QUAD $0x4dc14c0f4cc1394c - QUAD $0x000000cb840fc985 - QUAD $0x0001678e0fc0854d - QUAD $0xfc8a44100f42f300 - QUAD $0x0f02e88348c8894c - QUAD $0x894855000000c48c - QUAD $0x8d41f8e4834853e5 - QUAD $0xe1834103e183ff49 - QUAD $0x2e661cebd2314503 + QUAD $0xe58948551674c985 + QUAD $0xf88349f8e4834853 + QUAD $0x0206e9c031217d10 + QUAD $0x298e0fc0854d0000 + QUAD $0x0f08f88349000003 + QUAD $0xe9c031000001a883 + QUAD $0x100f42f300000309 + QUAD $0x0f01f98349fc8a44 + QUAD $0x498d49000001248e + QUAD $0x518d4d00c0c60ffe + QUAD $0x314500000010bbfd + QUAD $0x2e6666666634ebdb QUAD $0x0000000000841f0f - QUAD $0xff499714110f42f3 - QUAD $0x0099840fc2394dc2 - QUAD $0x960c100f42f30000 - QUAD $0x8341d0280fc38949 - QUAD $0x49cb89481e7401f9 - QUAD $0x001f0fd0280fc389 + QUAD $0x110f429f24110f42 + QUAD $0x9f7c110f42109f74 + QUAD $0x48309f44110f4620 + QUAD $0x394cc3894910588d + QUAD $0x48000001918f0fc3 + QUAD $0x429e0c100f42d889 + QUAD $0x100f42109e54100f + QUAD $0x9e6c100f42209e5c + QUAD $0x0fe0280fcb894830 + QUAD $0x280f44f8280ff028 + QUAD $0x0f397501c1f641c0 + QUAD $0x0f46f3e1590fe028 + QUAD $0xc9c60f45f88a4c10 + QUAD $0xf0280fe1580f4100 + QUAD $0x0ff1580f41f2590f + QUAD $0x580f41fb590ff828 + QUAD $0x590f44c0280f44f9 + QUAD $0xd3894cc1580f45c5 + QUAD $0xffff67840fc98548 + QUAD $0x001f0f02c38348ff + QUAD $0x4c100f44f3e1590f + QUAD $0x9a54100f44f3f49a + QUAD $0x0f4100d2c60f45f8 + QUAD $0x580f41f2590fe258 + QUAD $0xfa580f41fb590ff2 + QUAD $0xc2580f45c5590f44 + QUAD $0x00c9c60f45e1590f + QUAD $0x41f2590fe1580f41 + QUAD $0x0f41fb590ff1580f + QUAD $0x0f45c5590f44f958 + QUAD $0x8348fec38348c158 + QUAD $0xfffeffe9a47f01fb + QUAD $0xeac149f0508d4dff + QUAD $0x4900000088840f04 + QUAD $0xe08348d0894cc2ff + QUAD $0x00c8c60fc8280ffe + QUAD $0x2e666600000010b9 + QUAD $0x0000000000841f0f + QUAD $0x4c110fc08f4c110f + QUAD $0x0fe08f4c110fd08f + QUAD $0x8f0c110ff08f4c11 + QUAD $0x4c110f108f4c110f + QUAD $0x48308f4c110f208f + QUAD $0x75fec0834820c183 + QUAD $0xc2f641f0418d48cf + QUAD $0xf8b8484beb337501 + QUAD $0x4c7fffffffffffff + QUAD $0x90c0570fc931c021 + QUAD $0x8f44110f8f04110f + QUAD $0xc8394808c1834810 + QUAD $0xb900000148e9ee75 + QUAD $0xc60fc03100000010 + QUAD $0x110f8704110f00c0 + QUAD $0x208744110f108744 + QUAD $0xc88948308744110f + QUAD $0x8e0fc22949c2894d + QUAD $0x100f42f30000010a + QUAD $0x8348c9894cfc8a44 + QUAD $0x000000a38c0f02e9 + QUAD $0x03e28341ff518d45 + QUAD $0x2e661beb03e18341 + QUAD $0x0000000000841f0f + QUAD $0xc0ff488714110ff3 + QUAD $0x0000cd840fc0394c + QUAD $0x8949860c100ff300 + QUAD $0x01f98341d0280fcb + QUAD $0xcb8949d3894c2074 + QUAD $0x0000441f0fd0280f QUAD $0x580f42f3d1590ff3 QUAD $0xcbff48cbff499a14 - QUAD $0xb87203f88348ee75 + QUAD $0xb87203f98348ee75 QUAD $0x00401f0f04c38349 QUAD $0x580f42f3d1590ff3 QUAD $0xf3d1590ff3f09a54 @@ -202,27 +268,20 @@ TEXT ·polyEvalFloat32SSE2(SB), NOSPLIT|NOFRAME, $0-72 QUAD $0x0f42f3d1590ff3e8 QUAD $0xfcc38349e49a5458 QUAD $0x75e9ca7f03fb8349 - QUAD $0x8e0fc0854dffffff - QUAD $0x08f883490000009c - QUAD $0x000080e9c0311873 - QUAD $0x313d7308f8834900 - QUAD $0x5bf8658d486cebc0 - QUAD $0xfffffff8b848c35d - QUAD $0x31c0214c7fffffff - QUAD $0x2e666666c0570fc9 + QUAD $0x7308fa8349ffffff + QUAD $0x89443aebc2894905 + QUAD $0x49ca294907e183c1 + QUAD $0xc8c60fc8280fc201 + QUAD $0x00000000801f0f00 + QUAD $0x874c110f870c110f + QUAD $0xc2394908c0834810 + QUAD $0x661774c98548ee75 QUAD $0x0000000000841f0f - QUAD $0x8f44110f8f04110f - QUAD $0xc8394808c1834810 - QUAD $0xfff8b84847ebee75 - QUAD $0x214c7fffffffffff - QUAD $0x00c8c60fc8280fc0 - QUAD $0x0000441f0f66c931 - QUAD $0x8f4c110f8f0c110f - QUAD $0xc8394808c1834810 - QUAD $0xf31974c03949ee75 - QUAD $0xebc0ff488704110f - QUAD $0x000000008704c7f1 - QUAD $0xf175c03949c0ff48 + QUAD $0xff499704110f42f3 + QUAD $0x8d48f275d0394dc2 + QUAD $0x8704c7c35d5bf865 + QUAD $0x49c0ff4800000000 + LONG $0xf175c039 BYTE $0xc3 RET @@ -236,68 +295,117 @@ TEXT ·convolveFloat32SSE2(SB), NOSPLIT|NOFRAME, $0-72 MOVQ ker_len+56(FP), R9 QUAD $0x56415741e5894855 QUAD $0xe483485354415541 - QUAD $0xcd894d18ec8348f8 - QUAD $0xc9854d08247c8948 - QUAD $0x0f41c8294dc0940f - QUAD $0xbc850fc10841c19c - QUAD $0x4c01488d4d000001 - QUAD $0x854dc94e0f4cc139 - QUAD $0x4d000001a88e0fc9 - QUAD $0x000001408e0fed85 - QUAD $0x41e88945aa0c8d4a - QUAD $0xfffff8ba4903e083 - QUAD $0xca83497fffffffff - QUAD $0x0c5e8d4cea214d04 - QUAD $0xfcc08348aa048d4a - QUAD $0x49db311024448948 - QUAD $0xfffffffffffffebe - QUAD $0xfffffffffdbf493f - QUAD $0xfffffcbc493fffff - QUAD $0x9022eb3fffffffff - QUAD $0x110ff30824448b48 - QUAD $0xc38349c3ff489804 - QUAD $0xcb394c04c6834804 - QUAD $0x00012e840ffd8949 - QUAD $0x49c0570fef894c00 - QUAD $0xed3145137304fd83 - QUAD $0x2e666600000081e9 + QUAD $0xc9854d18ec8348f8 + QUAD $0x08244c894cc0940f + QUAD $0x41c19c0f41c8294d + QUAD $0x00000340850fc108 + QUAD $0x4cc1394c01688d4d + QUAD $0x7d10fd8349e94e0f + QUAD $0x0000018ee9c93107 + QUAD $0xc9854d08244c8b4c + QUAD $0x8d4c000000aa8e0f + QUAD $0x4500000010b83046 + QUAD $0x2e6666666666d231 QUAD $0x0000000000841f0f - QUAD $0xed31451024548b48 + QUAD $0x894dc0570fc18948 + QUAD $0x0fc9570fcb894cc3 + QUAD $0x2e6666db570fd257 QUAD $0x0000000000841f0f - QUAD $0xf3f4ab4c100f43f3 - QUAD $0x0ff3f8ab54100f43 + QUAD $0xc60ffc9a64100ff3 + QUAD $0x41d06b100f4100e4 + QUAD $0x7b100f41e073100f + QUAD $0xec590f03100f45f0 + QUAD $0x580ff4590fc5580f + QUAD $0x44d7580ffc590fce + QUAD $0x49d8580f41c4590f + QUAD $0xc075cbff4804c383 + QUAD $0x110f429704110f42 + QUAD $0x9754110f4210974c + QUAD $0x4830975c110f4220 + QUAD $0x4940c0834910418d + QUAD $0x758e0fe8394cca89 + QUAD $0x000000d6e9ffffff + QUAD $0x49c28949f0458d49 + QUAD $0x8945c2ff4904eac1 + QUAD $0xf8834803e08341d0 + QUAD $0x00000010b80c7330 + QUAD $0x4900000083e9c931 + QUAD $0xc0570fc931fce283 + QUAD $0x8f44110f8f04110f + QUAD $0x110f208f44110f10 + QUAD $0x408f44110f308f44 + QUAD $0x44110f508f44110f + QUAD $0x0f708f44110f608f + QUAD $0x0f000000808f8411 + QUAD $0x0f000000908f8411 + QUAD $0x0f000000a08f8411 + QUAD $0x0f000000b08f8411 + QUAD $0x0f000000c08f8411 + QUAD $0x0f000000d08f8411 + QUAD $0x0f000000e08f8411 + QUAD $0x48000000f08f8411 + QUAD $0x75fcc2834940c183 + QUAD $0x8d483074c0854d8f + QUAD $0x001f0fc0570f1041 + QUAD $0x8f44110f8f04110f + QUAD $0x110f208f44110f10 + QUAD $0x8348c18948308f44 + QUAD $0x48e175c8ff4910c0 + QUAD $0x894dc18948f0c083 + QUAD $0x018e8e0fc82949e8 + QUAD $0x480824448b480000 + QUAD $0x000001308e0fc085 + QUAD $0x41c2894182048d4c + QUAD $0xfffffcbb4903e283 + QUAD $0xc321497fffffffff + QUAD $0x0cc383488e1c8d48 + QUAD $0x448948fc82448d48 + QUAD $0xbe498e348d481024 + QUAD $0x3ffffffffffffffe + QUAD $0xfffffffffffdbf49 + QUAD $0xfffffffcbc493fff + QUAD $0x90661eeb3fffffff + QUAD $0xc1ff488f04110ff3 + QUAD $0x04c6834804c38348 + QUAD $0x840fc9394ccd894d + QUAD $0x48e9894d00000114 + QUAD $0x0f09730408247c83 + QUAD $0x907aebed3145c057 + QUAD $0x1024548b48c0570f + QUAD $0x0000441f0fed3145 + QUAD $0xf3f4ab4c100f42f3 + QUAD $0x0ff3f8ab54100f42 QUAD $0x894cc8580ff30a59 - QUAD $0x14590ff3f0314ce8 - QUAD $0x0f43f3d1580ff381 - QUAD $0x4ce8894cfcab4c10 - QUAD $0xf3810c590ff3f831 - QUAD $0x04100f43f3ca580f - QUAD $0xf3e0314ce8894cab - QUAD $0xc1580ff38104590f - QUAD $0xf0c2834804c58349 - QUAD $0xc0854da375ea394d - QUAD $0x8d4affffff4a840f - QUAD $0x8b4800000000ad04 - QUAD $0x8d4ec22948102454 - QUAD $0x00401f0fc031ae2c + QUAD $0x590f41f3f0314ce8 + QUAD $0x42f3d1580ff38014 + QUAD $0xe8894cfcab4c100f + QUAD $0x0c590f41f3f8314c + QUAD $0x0f42f3ca580ff380 + QUAD $0x314ce8894cab0410 + QUAD $0xf38004590f41f3e0 + QUAD $0x4804c58349c1580f + QUAD $0xa075eb394df0c283 + QUAD $0xffff57840fd2854d + QUAD $0x000000ad048d4aff + QUAD $0x29481024548b4800 + QUAD $0x90c031ae2c8d4ec2 QUAD $0xf300854c100f41f3 QUAD $0x48c1580ff30a590f QUAD $0x3949fcc28348c0ff - QUAD $0xffffff10e9e575c0 - QUAD $0xc031047308f98349 - QUAD $0xfffffff8b8483deb - QUAD $0x31c8214c7fffffff - QUAD $0x24548b48c0570fc9 - QUAD $0x2e66666666666608 + QUAD $0xffffff20e9e575c2 + QUAD $0x8949057308f88349 + QUAD $0xe083e8894435ebc8 + QUAD $0x0fc80149c0294907 + QUAD $0x0000441f0f66c057 + QUAD $0x8f44110f8f04110f + QUAD $0xc8394908c1834810 + QUAD $0x661974c08548ee75 QUAD $0x0000000000841f0f - QUAD $0x8a44110f8a04110f - QUAD $0xc8394808c1834810 - QUAD $0x481874c13949ee75 - QUAD $0x00401f0f08244c8b - QUAD $0x48000000008104c7 - QUAD $0x48f175c13949c0ff - QUAD $0x5d415c415bd8658d - LONG $0x5f415e41 + QUAD $0x000000008704c742 + QUAD $0xf075c5394dc0ff49 + QUAD $0x415c415bd8658d48 + LONG $0x415e415d + BYTE $0x5f BYTE $0x5d BYTE $0xc3 RET @@ -312,49 +420,106 @@ TEXT ·correlateFloat32SSE2(SB), NOSPLIT|NOFRAME, $0-72 MOVQ ker_len+56(FP), R9 QUAD $0x294dc0940fc9854d QUAD $0xc20841c29c0f41c8 - QUAD $0x8d490000014d850f - QUAD $0x4e0f48c1394c0140 - QUAD $0x01398e0fc08548c1 - QUAD $0xe58e0fc9854d0000 - QUAD $0x41e5894855000000 - QUAD $0xf8e4834853564157 - QUAD $0xb84903e183c98944 - QUAD $0x7ffffffffffffff8 - QUAD $0x4cc8214d04c88349 - QUAD $0x1aebdb31450c568d - QUAD $0xff499f04110f42f3 - QUAD $0xc6834804c28349c3 - QUAD $0x0093840fc3394904 - QUAD $0x49db31c0570f0000 - QUAD $0x2e66665b7204f983 + QUAD $0x485500000304850f + QUAD $0x485356415741e589 + QUAD $0x4c01408d49f8e483 + QUAD $0x8348c14e0f48c139 + QUAD $0x9be9c931077d10f8 + QUAD $0x8e0fc9854d000001 + QUAD $0x30468d4c000000af + QUAD $0x314500000010bb41 + QUAD $0x2e666666666666d2 + QUAD $0x0000000000841f0f + QUAD $0x3145c0570fd9894c + QUAD $0x0fd2570fc9570fdb + QUAD $0x2e6666666666db57 + QUAD $0x0000000000841f0f + QUAD $0xc60f9a24100f42f3 + QUAD $0xd0986c100f4300e4 + QUAD $0x0f43e09874100f43 + QUAD $0x04100f47f0987c10 + QUAD $0x0fc5580fec590f98 + QUAD $0xfc590fce580ff459 + QUAD $0x41c4590f44d7580f + QUAD $0x394dc3ff49d8580f + QUAD $0x9704110f42bd75d9 + QUAD $0x0f4210974c110f42 + QUAD $0x5c110f4220975411 + QUAD $0x834910598d4c3097 + QUAD $0xc33949ca894940c0 + QUAD $0xe3e9ffffff728e0f + QUAD $0x49f0488d48000000 + QUAD $0xff4904eac149ca89 + QUAD $0x03e08341d08945c2 + QUAD $0xba410d7330f98348 + QUAD $0x8fe9c93100000010 + QUAD $0x31fce28349000000 + QUAD $0x2e666666c0570fc9 + QUAD $0x0000000000841f0f + QUAD $0x8f44110f8f04110f + QUAD $0x110f208f44110f10 + QUAD $0x408f44110f308f44 + QUAD $0x44110f508f44110f + QUAD $0x0f708f44110f608f + QUAD $0x0f000000808f8411 + QUAD $0x0f000000908f8411 + QUAD $0x0f000000a08f8411 + QUAD $0x0f000000b08f8411 + QUAD $0x0f000000c08f8411 + QUAD $0x0f000000d08f8411 + QUAD $0x0f000000e08f8411 + QUAD $0x48000000f08f8411 + QUAD $0x75fcc2834940c183 + QUAD $0x8d4c3074c0854d8f + QUAD $0x001f0fc0570f1051 + QUAD $0x8f44110f8f04110f + QUAD $0x110f208f44110f10 + QUAD $0x8349d1894c308f44 + QUAD $0x49e175c8ff4910c2 + QUAD $0x8949d1894cf0c283 + QUAD $0x012e8e0fc82949c0 + QUAD $0xd28e0fc9854d0000 + QUAD $0x8341c88945000000 + QUAD $0xfffffffcba4903e0 + QUAD $0x4cca214d7fffffff + QUAD $0x480cc383498e1c8d + QUAD $0x001f0f1ceb8e348d + QUAD $0xc1ff488f04110ff3 + QUAD $0x04c6834804c38349 + QUAD $0x0000e7840fc13948 + QUAD $0x8349db31c0570f00 + QUAD $0x2e6666665c7204f9 QUAD $0x0000000000841f0f - QUAD $0xf3f49a4c100f41f3 - QUAD $0x0ff3f89a54100f41 + QUAD $0xf3f49b4c100f41f3 + QUAD $0x0ff3f89b54100f41 QUAD $0x9a54590ff39a0c59 QUAD $0x0f41f3c8580ff304 - QUAD $0x5c590ff3fc9a5c10 + QUAD $0x5c590ff3fc9b5c10 QUAD $0x41f3d1580ff3089a - QUAD $0x44590ff39a04100f + QUAD $0x44590ff39b04100f QUAD $0x0ff3da580ff30c9a QUAD $0x394904c38348c358 - QUAD $0x8074c98548b575d8 + QUAD $0x8074c0854db575da QUAD $0x9e1c8d489a348d4c QUAD $0x0000441f0fff3145 QUAD $0x43f3bb0c100f42f3 QUAD $0xc1580ff3be0c590f - QUAD $0xe875f9394cc7ff49 - QUAD $0x658d48ffffff53e9 - QUAD $0xc35d5f415e415be8 - QUAD $0xc931047308f88348 - QUAD $0xfffffff8b94832eb - QUAD $0x31c121487fffffff - QUAD $0x2e666666c0570fd2 + QUAD $0xe875f8394dc7ff49 + QUAD $0xf88349ffffff53e9 + QUAD $0x38ebc88949057308 + QUAD $0xd0294907e283c289 + QUAD $0x2e66c0570fc80149 + QUAD $0x0000000000841f0f + QUAD $0x8f44110f8f04110f + QUAD $0xc8394908c1834810 + QUAD $0x661974d28548ee75 QUAD $0x0000000000841f0f - QUAD $0x9744110f9704110f - QUAD $0xd1394808c2834810 - QUAD $0x008f04c70aebee75 - QUAD $0x3948c1ff48000000 - LONG $0xc3f175c8 + QUAD $0x000000008704c742 + QUAD $0xf075c0394cc0ff49 + QUAD $0x415e415be8658d48 + BYTE $0x5f + BYTE $0x5d + BYTE $0xc3 RET // func normFloat64SSE2(a []float64) float64 @@ -522,24 +687,106 @@ TEXT ·polyEvalFloat64SSE2(SB), NOSPLIT|NOFRAME, $0-72 MOVQ x_len+32(FP), R8 MOVQ coeffs_len+56(FP), R9 QUAD $0x4dc14c0f4cc1394c - QUAD $0x000000cb840fc985 - QUAD $0x00016a8e0fc0854d - QUAD $0xf8ca44100f42f200 - QUAD $0x0f02e88348c8894c - QUAD $0x894855000000c48c - QUAD $0x8d41f8e4834853e5 - QUAD $0xe1834103e183ff49 - QUAD $0x2e661cebd2314503 + QUAD $0xe58948551a74c985 + QUAD $0xec8348f0e4834853 + QUAD $0x31217d10f8834930 + QUAD $0x854d0000027ee9c0 + QUAD $0x49000003a68e0fc0 + QUAD $0x0001fc830f04f883 + QUAD $0x00000385e9c03100 + QUAD $0x49f8ca4c100f42f2 + QUAD $0x0001488e0f01f983 + QUAD $0x498d49c9140f6600 + QUAD $0x4500000010bb41ff + QUAD $0x66240c290f66d231 QUAD $0x0000000000841f0f - QUAD $0xff49d714110f42f2 - QUAD $0x0099840fc2394dc2 - QUAD $0xd60c100f42f20000 - QUAD $0x41d0280f66c38949 - QUAD $0xcb89481d7401f983 - QUAD $0x90d0280f66c38949 + QUAD $0xd604100f42d8894c + QUAD $0x0f4266102444290f + QUAD $0x100f426610d65410 + QUAD $0x64100f426620d65c + QUAD $0xd66c100f426630d6 + QUAD $0x50d674100f426640 + QUAD $0x6660d67c100f4266 + QUAD $0x894970d644100f46 + QUAD $0x4466c9280f4466cb + QUAD $0xd9280f4466d1280f + QUAD $0x0f4466e1280f4466 + QUAD $0x66f1280f4466e928 + QUAD $0xc1280f66f9280f44 + QUAD $0x0000000000841f0f + QUAD $0x590f4166c8280f66 + QUAD $0x4466ff590f4466c8 + QUAD $0xed590f4466f6590f + QUAD $0x0f4466e4590f4466 + QUAD $0x66d2590f4466db59 + QUAD $0x42f210244c590f44 + QUAD $0x140f66f8da44100f + QUAD $0x4466c8580f4466c0 + QUAD $0xd8580f4466d0580f + QUAD $0x0f4466e0580f4466 + QUAD $0x66f0580f4466e858 + QUAD $0xc1580f66f8580f44 + QUAD $0x987fdb854dcbff49 + QUAD $0x4666d70c110f4666 + QUAD $0x0f466610d754110f + QUAD $0x110f466620d75c11 + QUAD $0x6c110f466630d764 + QUAD $0xd774110f466640d7 + QUAD $0x60d77c110f466650 + QUAD $0x4c70d744110f4266 + QUAD $0x394dc2894910588d + QUAD $0x8e0f240c280f66c3 + QUAD $0x00010be9fffffedc + QUAD $0xeac149f0508d4d00 + QUAD $0x49000000c0840f04 + QUAD $0xe08348d0894cc2ff + QUAD $0x140f66c1280f66fe + QUAD $0x906600000010b9c1 + QUAD $0x0f6680cf44110f66 + QUAD $0x44110f6690cf4411 + QUAD $0xb0cf44110f66a0cf + QUAD $0x0f66c0cf44110f66 + QUAD $0x44110f66d0cf4411 + QUAD $0xf0cf44110f66e0cf + QUAD $0x110f66cf04110f66 + QUAD $0xcf44110f6610cf44 + QUAD $0x6630cf44110f6620 + QUAD $0x110f6640cf44110f + QUAD $0xcf44110f6650cf44 + QUAD $0x4870cf44110f6660 + QUAD $0x75fec0834820c183 + QUAD $0xc2f641f0418d4897 + QUAD $0xfcb8486feb3b7501 + QUAD $0x4c7fffffffffffff + QUAD $0x66c0570fc931c021 + QUAD $0x0000000000841f0f + QUAD $0xcf44110fcf04110f + QUAD $0xc8394804c1834810 + QUAD $0xb900000169e9ee75 + QUAD $0x0f66c03100000010 + QUAD $0x10c74c110f66c914 + QUAD $0x110f66c70c110f66 + QUAD $0xc74c110f6630c74c + QUAD $0x6650c74c110f6620 + QUAD $0x110f6640c74c110f + QUAD $0xc74c110f6670c74c + QUAD $0x49c2894dc8894860 + QUAD $0x0000010e8e0fc229 + QUAD $0x4cf8ca44100f42f2 + QUAD $0x8c0f02e98348c989 + QUAD $0xff518d45000000a7 + QUAD $0x03e1834103e28341 + QUAD $0x2e66666666661feb + QUAD $0x0000000000841f0f + QUAD $0xc0ff48c714110ff2 + QUAD $0x0000cd840fc0394c + QUAD $0x8949c60c100ff200 + QUAD $0xf98341d0280f66cb + QUAD $0x8949d3894c1f7401 + QUAD $0x001f0fd0280f66cb QUAD $0x580f42f2d1590ff2 QUAD $0xcbff48cbff49da14 - QUAD $0xb87203f88348ee75 + QUAD $0xb87203f98348ee75 QUAD $0x00401f0f04c38349 QUAD $0x580f42f2d1590ff2 QUAD $0xf2d1590ff2e0da54 @@ -548,28 +795,22 @@ TEXT ·polyEvalFloat64SSE2(SB), NOSPLIT|NOFRAME, $0-72 QUAD $0x0f42f2d1590ff2d0 QUAD $0xfcc38349c8da5458 QUAD $0x75e9ca7f03fb8349 - QUAD $0x8e0fc0854dffffff - QUAD $0x04f883490000009f - QUAD $0x000082e9c0311873 - QUAD $0x313f7304f8834900 - QUAD $0x5bf8658d486eebc0 - QUAD $0xfffffffcb848c35d - QUAD $0x31c0214c7fffffff - QUAD $0x2e6666c0570f66c9 - QUAD $0x0000000000841f0f - QUAD $0x110f66cf04110f66 - QUAD $0x4804c1834810cf44 - QUAD $0xb84848ebec75c839 - QUAD $0x7ffffffffffffffc - QUAD $0x66c8280f66c0214c - QUAD $0x001f0fc931c8140f - QUAD $0x110f66cf0c110f66 - QUAD $0x4804c1834810cf4c - QUAD $0x74c03949ec75c839 - QUAD $0xff48c704110ff21a - QUAD $0x00c704c748f1ebc0 - QUAD $0x3949c0ff48000000 - LONG $0xc3f075c0 + QUAD $0x7304fa8349ffffff + QUAD $0x89443aebc2894905 + QUAD $0x49ca294903e183c1 + QUAD $0x0f66c8280f66c201 + QUAD $0x0000441f0f66c814 + QUAD $0x110f66c70c110f66 + QUAD $0x4904c0834810c74c + QUAD $0x74c98548ec75c239 + QUAD $0x00000000801f0f15 + QUAD $0xff49d704110f42f2 + QUAD $0x8d48f275d0394dc2 + QUAD $0x04c748c35d5bf865 + QUAD $0xc0ff4800000000c7 + LONG $0x75c03949 + BYTE $0xf0 + BYTE $0xc3 RET // func convolveFloat64SSE2(dst []float64, sig []float64, ker []float64) @@ -582,66 +823,134 @@ TEXT ·convolveFloat64SSE2(SB), NOSPLIT|NOFRAME, $0-72 MOVQ ker_len+56(FP), R9 QUAD $0x56415741e5894855 QUAD $0xe483485354415541 - QUAD $0xcd894d18ec8348f8 - QUAD $0xc9854d08247c8948 - QUAD $0x0f41c8294dc0940f - QUAD $0xbd850fc10841c19c - QUAD $0x4c01488d4d000001 - QUAD $0x854dc94e0f4cc139 - QUAD $0x4d000001a98e0fc9 - QUAD $0x000001408e0fed85 - QUAD $0x41e88945ea0c8d4a - QUAD $0xfffffcba4903e083 - QUAD $0xea214d7fffffffff - QUAD $0xea048d4a185e8d4c - QUAD $0x24448948f8c08348 - QUAD $0xfffffebe49db3110 + QUAD $0xc9854d18ec8348f8 + QUAD $0x08244c894cc0940f + QUAD $0x41c19c0f41c8294d + QUAD $0x000003e0850fc108 + QUAD $0x4cc1394c01688d4d + QUAD $0x7d10fd8349e94e0f + QUAD $0x0000022be9c93107 + QUAD $0xc9854d08244c8b4c + QUAD $0x8d4c000001328e0f + QUAD $0x4500000010b87046 + QUAD $0x2e6666666666d231 + QUAD $0x0000000000841f0f + QUAD $0x4dc0570f66c18948 + QUAD $0x570f66cb894cc389 + QUAD $0x570f66d2570f66c9 + QUAD $0x570f66e4570f66db + QUAD $0x570f66f6570f66ed + QUAD $0x00000000801f0fff + QUAD $0x66f8da44100f44f2 + QUAD $0x100f4566c0140f45 + QUAD $0xa053100f4566904b + QUAD $0x4566b05b100f4566 + QUAD $0x590f4566c063100f + QUAD $0x4566c1580f4166c8 + QUAD $0xca580f4166d0590f + QUAD $0x0f4166d8590f4566 + QUAD $0x66e0590f4566d358 + QUAD $0x100f4566dc580f41 + QUAD $0x66c8590f4566d04b + QUAD $0x100f4566e1580f41 + QUAD $0x66c8590f4566e04b + QUAD $0x100f4566e9580f41 + QUAD $0x66c8590f4566f04b + QUAD $0x100f4566f1580f41 + QUAD $0x4166c8590f45660b + QUAD $0x4808c38349f9580f + QUAD $0xffffff68850fcbff + QUAD $0x4266d704110f4266 + QUAD $0x0f426610d74c110f + QUAD $0x110f426620d75411 + QUAD $0x64110f426630d75c + QUAD $0xd76c110f426640d7 + QUAD $0x60d774110f426650 + QUAD $0x4870d77c110f4266 + QUAD $0x4980e8834910418d + QUAD $0xed8e0fe8394cca89 + QUAD $0x000000ebe9fffffe + QUAD $0x04eac149f0558d4d + QUAD $0xff490000009f840f + QUAD $0xfee08348d0894cc2 + QUAD $0x0f6600000010b841 + QUAD $0x0000441f0f66c057 + QUAD $0x6680c744110f4266 + QUAD $0x426690c744110f42 + QUAD $0x0f4266a0c744110f + QUAD $0x110f4266b0c74411 + QUAD $0x44110f4266c0c744 + QUAD $0xc744110f4266d0c7 + QUAD $0xf0c744110f4266e0 + QUAD $0x4266c704110f4266 + QUAD $0x0f426610c744110f + QUAD $0x110f426620c74411 + QUAD $0x44110f426630c744 + QUAD $0xc744110f426640c7 + QUAD $0x60c744110f426650 + QUAD $0x4970c744110f4266 + QUAD $0x75fec0834820c083 + QUAD $0xc2f641f0488d4987 + QUAD $0x10b8413eeb0a7501 + QUAD $0x570f66c931000000 + QUAD $0x0f66cf04110f66c0 + QUAD $0x44110f6610cf4411 + QUAD $0x30cf44110f6620cf + QUAD $0x0f6640cf44110f66 + QUAD $0x44110f6650cf4411 + QUAD $0x70cf44110f6660cf + QUAD $0x2949e8894dc1894c + QUAD $0x48000001918e0fc8 + QUAD $0x0fc085480824448b + QUAD $0x048d4c000001338e + QUAD $0x03e28341c28941c2 + QUAD $0xfffffffffffcbb49 + QUAD $0x1c8d48c321497fff + QUAD $0x448d4818c38348ce + QUAD $0x481024448948f8c2 + QUAD $0xfffffebe49ce348d QUAD $0xfdbf491fffffffff QUAD $0x491fffffffffffff QUAD $0xfffffffffffffcbc - QUAD $0x0000441f0f26eb1f - QUAD $0x110ff20824448b48 - QUAD $0xc38349c3ff48d804 - QUAD $0xcb394c08c6834808 - QUAD $0x00012f840ffd8949 - QUAD $0xc0570f66ef894c00 - QUAD $0x3145127304fd8349 - QUAD $0x2e6600000080e9ed - QUAD $0x0000000000841f0f - QUAD $0xed31451024548b48 - QUAD $0x0000000000841f0f - QUAD $0xf2e8eb4c100f43f2 - QUAD $0x0ff2f0eb54100f43 + QUAD $0x0000441f0f21eb1f + QUAD $0xc1ff48cf04110ff2 + QUAD $0x08c6834808c38348 + QUAD $0x840fc9394ccd894d + QUAD $0x48e9894d00000114 + QUAD $0x6609730408247c83 + QUAD $0x79ebed3145c0570f + QUAD $0x24548b48c0570f66 + QUAD $0x00401f0fed314510 + QUAD $0xf2e8eb4c100f42f2 + QUAD $0x0ff2f0eb54100f42 QUAD $0x894cc8580ff20a59 - QUAD $0x14590ff2f0314ce8 - QUAD $0x0f43f2d1580ff2c1 - QUAD $0x4ce8894cf8eb4c10 - QUAD $0xf2c10c590ff2f831 - QUAD $0x04100f43f2ca580f - QUAD $0xf2e0314ce8894ceb - QUAD $0xc1580ff2c104590f - QUAD $0xe0c2834804c58349 - QUAD $0xc0854da375ea394d - QUAD $0x8d4affffff4a840f - QUAD $0x8b4800000000ed04 - QUAD $0x8d4ec22948102454 - QUAD $0x00401f0fc031ee2c + QUAD $0x590f41f2f0314ce8 + QUAD $0x42f2d1580ff2c014 + QUAD $0xe8894cf8eb4c100f + QUAD $0x0c590f41f2f8314c + QUAD $0x0f42f2ca580ff2c0 + QUAD $0x314ce8894ceb0410 + QUAD $0xf2c004590f41f2e0 + QUAD $0x4804c58349c1580f + QUAD $0xa075eb394de0c283 + QUAD $0xffff57840fd2854d + QUAD $0x000000ed048d4aff + QUAD $0x29481024548b4800 + QUAD $0x90c031ee2c8d4ec2 QUAD $0xf200c54c100f41f2 QUAD $0x48c1580ff20a590f QUAD $0x3949f8c28348c0ff - QUAD $0xffffff10e9e575c0 - QUAD $0xc031047304f98349 - QUAD $0xfffffffcb8483feb - QUAD $0x31c8214c7fffffff - QUAD $0x548b48c0570f66c9 - QUAD $0x2e66666666660824 - QUAD $0x0000000000841f0f - QUAD $0x110f66ca04110f66 - QUAD $0x4804c1834810ca44 - QUAD $0x74c13949ec75c839 - QUAD $0x906608244c8b4817 - QUAD $0x00000000c104c748 - QUAD $0xf075c13949c0ff48 + QUAD $0xffffff20e9e575c2 + QUAD $0x8949057304f88349 + QUAD $0xe083e8894435ebc8 + QUAD $0x66c80149c0294903 + QUAD $0x0000441f0fc0570f + QUAD $0x110f66cf04110f66 + QUAD $0x4904c1834810cf44 + QUAD $0x74c08548ec75c839 + QUAD $0x00000000801f0f17 + QUAD $0x00000000c704c74a + QUAD $0xf075c5394dc0ff49 QUAD $0x415c415bd8658d48 LONG $0x415e415d BYTE $0x5f @@ -659,49 +968,123 @@ TEXT ·correlateFloat64SSE2(SB), NOSPLIT|NOFRAME, $0-72 MOVQ ker_len+56(FP), R9 QUAD $0x294dc0940fc9854d QUAD $0xc20841c29c0f41c8 - QUAD $0x8d490000014e850f - QUAD $0x4e0f48c1394c0140 - QUAD $0x013a8e0fc08548c1 - QUAD $0xe58e0fc9854d0000 - QUAD $0x41e5894855000000 - QUAD $0xf8e4834853564157 - QUAD $0xb84903e183c98944 - QUAD $0x7ffffffffffffffc - QUAD $0x4518568d4cc8214d - QUAD $0x00401f0f1eebdb31 - QUAD $0xff49df04110f42f2 - QUAD $0xc6834808c28349c3 - QUAD $0x0093840fc3394908 - QUAD $0xdb31c0570f660000 - QUAD $0x2e665a7204f98349 + QUAD $0x485500000394850f + QUAD $0x485356415741e589 + QUAD $0x4c01408d49f8e483 + QUAD $0x8348c14e0f48c139 + QUAD $0x28e9c931077d10f8 + QUAD $0x8e0fc9854d000002 + QUAD $0x70468d4c0000013a + QUAD $0x314500000010bb41 + QUAD $0x2e666666666666d2 + QUAD $0x0000000000841f0f + QUAD $0x45c0570f66d9894c + QUAD $0x0f66c9570f66db31 + QUAD $0x0f66db570f66d257 + QUAD $0x0f66ed570f66e457 + QUAD $0x2e66ff570f66f657 QUAD $0x0000000000841f0f - QUAD $0xf2e8da4c100f41f2 - QUAD $0x0ff2f0da54100f41 + QUAD $0x4566da04100f46f2 + QUAD $0x4c100f4766c0140f + QUAD $0xd854100f476690d8 + QUAD $0xb0d85c100f4766a0 + QUAD $0x66c0d864100f4766 + QUAD $0x580f4166c8590f45 + QUAD $0x4166d0590f4566c1 + QUAD $0xd8590f4566ca580f + QUAD $0x0f4566d3580f4166 + QUAD $0x66dc580f4166e059 + QUAD $0x4566d0d84c100f47 + QUAD $0xe1580f4166c8590f + QUAD $0x66e0d84c100f4766 + QUAD $0x580f4166c8590f45 + QUAD $0xf0d84c100f4766e9 + QUAD $0x0f4166c8590f4566 + QUAD $0xd80c100f4766f158 + QUAD $0x0f4166c8590f4566 + QUAD $0xd9394dc3ff49f958 + QUAD $0x4266ffffff62850f + QUAD $0x110f4266d704110f + QUAD $0x54110f426610d74c + QUAD $0xd75c110f426620d7 + QUAD $0x40d764110f426630 + QUAD $0x6650d76c110f4266 + QUAD $0x426660d774110f42 + QUAD $0x598d4c70d77c110f + QUAD $0xca894980e8834910 + QUAD $0xfffee78e0fc33949 + QUAD $0x8d4c000000e5e9ff + QUAD $0x840f04eac149f050 + QUAD $0x4cc2ff4900000099 + QUAD $0xb841fee18348d189 + QUAD $0xc0570f6600000010 + QUAD $0x6680c744110f4266 + QUAD $0x426690c744110f42 + QUAD $0x0f4266a0c744110f + QUAD $0x110f4266b0c74411 + QUAD $0x44110f4266c0c744 + QUAD $0xc744110f4266d0c7 + QUAD $0xf0c744110f4266e0 + QUAD $0x4266c704110f4266 + QUAD $0x0f426610c744110f + QUAD $0x110f426620c74411 + QUAD $0x44110f426630c744 + QUAD $0xc744110f426640c7 + QUAD $0x60c744110f426650 + QUAD $0x4970c744110f4266 + QUAD $0x75fec1834820c083 + QUAD $0xc2f641f0488d4987 + QUAD $0x10b8413eeb0a7501 + QUAD $0x570f66c931000000 + QUAD $0x0f66cf04110f66c0 + QUAD $0x44110f6610cf4411 + QUAD $0x30cf44110f6620cf + QUAD $0x0f6640cf44110f66 + QUAD $0x44110f6650cf4411 + QUAD $0x70cf44110f6660cf + QUAD $0x2949c08949c1894c + QUAD $0x4d000001318e0fc8 + QUAD $0x000000d58e0fc985 + QUAD $0x4903e08341c88945 + QUAD $0xfffffffffffffcba + QUAD $0xce1c8d4cca214d7f + QUAD $0xce348d4818c38349 + QUAD $0x0000441f0f661feb + QUAD $0xc1ff48cf04110ff2 + QUAD $0x08c6834808c38349 + QUAD $0x0000e7840fc13948 + QUAD $0x49db31c0570f6600 + QUAD $0x2e66665b7204f983 + QUAD $0x0000000000841f0f + QUAD $0xf2e8db4c100f41f2 + QUAD $0x0ff2f0db54100f41 QUAD $0xda54590ff2da0c59 QUAD $0x0f41f2c8580ff208 - QUAD $0x5c590ff2f8da5c10 + QUAD $0x5c590ff2f8db5c10 QUAD $0x41f2d1580ff210da - QUAD $0x44590ff2da04100f + QUAD $0x44590ff2db04100f QUAD $0x0ff2da580ff218da QUAD $0x394904c38348c358 - QUAD $0x8074c98548b575d8 + QUAD $0x8074c0854db575da QUAD $0xde1c8d48da348d4c QUAD $0x0000441f0fff3145 QUAD $0x43f2fb0c100f42f2 QUAD $0xc1580ff2fe0c590f - QUAD $0xe875f9394cc7ff49 - QUAD $0x658d48ffffff53e9 - QUAD $0xc35d5f415e415be8 - QUAD $0xc931047304f88348 - QUAD $0xfffffffcb94832eb - QUAD $0x31c121487fffffff - QUAD $0x2e666666c0570fd2 + QUAD $0xe875f8394dc7ff49 + QUAD $0xf88349ffffff53e9 + QUAD $0x38ebc88949057304 + QUAD $0xd0294903e283c289 + QUAD $0x66c0570f66c80149 QUAD $0x0000000000841f0f - QUAD $0xd744110fd704110f - QUAD $0xd1394804c2834810 - QUAD $0xcf04c7480bebee75 - QUAD $0x48c1ff4800000000 - LONG $0xf075c839 + QUAD $0x110f66cf04110f66 + QUAD $0x4904c1834810cf44 + QUAD $0x74d28548ec75c839 + QUAD $0x00000000801f0f17 + QUAD $0x00000000c704c74a + QUAD $0xf075c0394cc0ff49 + QUAD $0x415e415be8658d48 + BYTE $0x5f + BYTE $0x5d BYTE $0xc3 RET @@ -1406,21 +1789,23 @@ TEXT ·movingAverageFloat32SSE2(SB), NOSPLIT|NOFRAME, $0-56 MOVQ width+48(FP), R8 QUAD $0x294cc09e0fc0854d QUAD $0xc10841c19c0f41c1 - QUAD $0x8d48000000e3850f - QUAD $0x4e0f48ca39480141 - QUAD $0x00cf8e0fc08548c2 - QUAD $0x4853e58948550000 - QUAD $0xc02a0f49f3f8e483 - QUAD $0x83c28944ff488d49 - QUAD $0xfffffff8b94907e2 - QUAD $0x4cc1214d7fffffff - QUAD $0x22ebd231451c468d - QUAD $0x0000000000841f0f - QUAD $0x110f42f3c85e0ff3 - QUAD $0xc08349c2ff49970c - QUAD $0xc2394904c6834804 - QUAD $0xdb3145c9570f7974 - QUAD $0x2e664f7207f98348 + QUAD $0x4855000000f5850f + QUAD $0xe48348535641e589 + QUAD $0xfffffffff8b848f8 + QUAD $0x4801498d4c7fffff + QUAD $0x8349ca4e0f4cca39 + QUAD $0x000000cc8d0f10f9 + QUAD $0xb88d0fc9394cc931 + QUAD $0x49f3c0570f000000 + QUAD $0x45ff508d49c02a0f + QUAD $0x214c07e28341c289 + QUAD $0xc083498e048d4cc0 + QUAD $0x901aeb8e348d481c + QUAD $0x0c110ff3c85e0ff3 + QUAD $0x04c08349c1ff488f + QUAD $0x74c9394c04c68348 + QUAD $0x48db3145c9570f7a + QUAD $0x2e6666507207fa83 QUAD $0x0000000000841f0f QUAD $0xf3e4984c580f43f3 QUAD $0x43f3e8984c580f43 @@ -1429,14 +1814,50 @@ TEXT ·movingAverageFloat32SSE2(SB), NOSPLIT|NOFRAME, $0-56 QUAD $0x4c580f43f3f4984c QUAD $0x984c580f43f3f898 QUAD $0x49980c580f43f3fc - QUAD $0xc075d9394d08c383 - QUAD $0x1c8d4e8b74d28548 + QUAD $0xc075d8394c08c383 + QUAD $0x1c8d4e8b74d2854d QUAD $0x0000441f0fdb319e QUAD $0xff489b0c580f41f3 - QUAD $0x6de9f275da3948c3 - QUAD $0x5bf8658d48ffffff - BYTE $0x5d - BYTE $0xc3 + QUAD $0x6de9f275da3949c3 + QUAD $0x5bf0658d48ffffff + QUAD $0x2a0f49f3c35d5e41 + QUAD $0x508d4800c0c60fc0 + QUAD $0x34568d4cc2214c06 + QUAD $0xdb314500000010bb + QUAD $0x0000441f0f6670eb + QUAD $0x280fe3280fec280f + QUAD $0x9e348d4ed1280fda + QUAD $0xca580f9e0c100f41 + QUAD $0x580f109e54100f41 + QUAD $0x0f209e5c100f41d3 + QUAD $0x309e64100f41dc58 + QUAD $0x0f42c85e0fe5580f + QUAD $0x0f42d05e0f9f0c11 + QUAD $0x42d85e0f109f5411 + QUAD $0xe05e0f209f5c110f + QUAD $0x8d48309f64110f42 + QUAD $0x894940c283491059 + QUAD $0xfea88f0fcb394ccb + QUAD $0xc9570fd98948ffff + QUAD $0xdb570fd2570fdb31 + QUAD $0x0f01f88349e4570f + QUAD $0x2e6666ffffff7b84 + QUAD $0x0000000000841f0f + QUAD $0x580fcc9a6c100f41 + QUAD $0x0fd09a4c100f41e9 + QUAD $0xdc9a6c100f41cd58 + QUAD $0x9a54100f41ea580f + QUAD $0x6c100f41d5580fe0 + QUAD $0x100f41eb580fec9a + QUAD $0x0f41f4580ffc9a74 + QUAD $0x41dd580ff09a5c10 + QUAD $0x48e6580f9a24100f + QUAD $0xb075da394802c383 + QUAD $0xff16850f01c0f641 + LONG $0x44e9ffff + BYTE $0xff + BYTE $0xff + BYTE $0xff RET // func shiftDivFloat32SSE2(dst []float32, a []float32, shift float32, denom float32) @@ -1502,21 +1923,59 @@ TEXT ·movingAverageFloat64SSE2(SB), NOSPLIT|NOFRAME, $0-56 MOVQ width+48(FP), R8 QUAD $0x294cc09e0fc0854d QUAD $0xc10841c19c0f41c1 - QUAD $0x8d48000000e3850f - QUAD $0x4e0f48ca39480141 - QUAD $0x00cf8e0fc08548c2 - QUAD $0x4853e58948550000 - QUAD $0xc02a0f49f2f8e483 - QUAD $0x83c28944ff488d49 - QUAD $0xfffffff8b94907e2 - QUAD $0x4cc1214d7fffffff - QUAD $0x22ebd2314538468d + QUAD $0x485500000213850f + QUAD $0x48f8e4834853e589 + QUAD $0x0f48ca394801418d + QUAD $0x077d10f88348c24e + QUAD $0xf20000011ce9c931 + QUAD $0xc0140f66c02a0f49 + QUAD $0x0010ba4170568d48 + QUAD $0x001f0fc931450000 + QUAD $0x45ff570f66d1894c + QUAD $0x66c0570f4566d231 + QUAD $0x66ed570f66f6570f + QUAD $0x66db570f66e4570f + QUAD $0x66c9570f66d2570f QUAD $0x0000000000841f0f - QUAD $0x110f42f2c85e0ff2 - QUAD $0xc08349c2ff49d70c - QUAD $0xc2394908c6834808 - QUAD $0x3145c9570f667974 - QUAD $0x664e7207f98348db + QUAD $0x6690d24c100f4666 + QUAD $0x100f4666f9580f41 + QUAD $0xc1580f4566a0d24c + QUAD $0x66b0d24c100f4666 + QUAD $0x100f4666f1580f41 + QUAD $0xe9580f4166c0d24c + QUAD $0x66d0d24c100f4666 + QUAD $0x100f4666e1580f41 + QUAD $0xd9580f4166e0d24c + QUAD $0x66f0d24c100f4666 + QUAD $0x100f4666d1580f41 + QUAD $0x49c9580f4166d20c + QUAD $0x669975d0394dc2ff + QUAD $0x3c110f4266f85e0f + QUAD $0x4666c05e0f4466cf + QUAD $0x5e0f6610cf44110f + QUAD $0x20cf74110f4266f0 + QUAD $0x110f4266e85e0f66 + QUAD $0x66e05e0f6630cf6c + QUAD $0x0f6640cf64110f42 + QUAD $0xcf5c110f4266d85e + QUAD $0x0f4266d05e0f6650 + QUAD $0xc85e0f6660cf5411 + QUAD $0x4c70cf4c110f4266 + QUAD $0x4980ea834810518d + QUAD $0xfd8e0fc23949c989 + QUAD $0x8d0fc13948fffffe + QUAD $0xf2c0570f000000c7 + QUAD $0xff508d49c02a0f49 + QUAD $0x4907e18341c18945 + QUAD $0xfffffffffffff8ba + QUAD $0xce048d4cc2214d7f + QUAD $0xce348d4838c08349 + QUAD $0x0000441f0f661feb + QUAD $0x0c110ff2c85e0ff2 + QUAD $0x08c08349c1ff48cf + QUAD $0x74c1394808c68348 + QUAD $0xdb3145c9570f667a + QUAD $0x2e664f7207fa8348 QUAD $0x0000000000841f0f QUAD $0xf2c8d84c580f43f2 QUAD $0x43f2d0d84c580f43 @@ -1525,11 +1984,11 @@ TEXT ·movingAverageFloat64SSE2(SB), NOSPLIT|NOFRAME, $0-56 QUAD $0x4c580f43f2e8d84c QUAD $0xd84c580f43f2f0d8 QUAD $0x49d80c580f43f2f8 - QUAD $0xc075d9394d08c383 - QUAD $0x1c8d4e8b74d28548 + QUAD $0xc075da394d08c383 + QUAD $0x1c8d4e8b74c9854d QUAD $0x0000441f0fdb31de QUAD $0xff48db0c580f41f2 - QUAD $0x6de9f275da3948c3 + QUAD $0x6de9f275d93949c3 QUAD $0x5bf8658d48ffffff BYTE $0x5d BYTE $0xc3 diff --git a/internal/arm64/numeric_neon_arm64.s b/internal/arm64/numeric_neon_arm64.s index 3df269c..3f7dc75 100644 --- a/internal/arm64/numeric_neon_arm64.s +++ b/internal/arm64/numeric_neon_arm64.s @@ -233,20 +233,74 @@ TEXT ·polyEvalFloat32NEON(SB), NOSPLIT|NOFRAME, $0-72 MOVD coeffs_len+56(FP), R5 WORD $0xeb04007f WORD $0x9a84b068 + WORD $0xb40000c5 + WORD $0x8b05084a + WORD $0xf100411f + WORD $0x5400012a + WORD $0xaa1f03e9 + WORD $0x14000034 WORD $0xf100051f - WORD $0xb4000305 - WORD $0x540007eb - WORD $0x8b05084b - WORD $0xf10008bf - WORD $0xbc5fc160 - WORD $0x5400030b + WORD $0x54000d8b + WORD $0xf100211f + WORD $0x54000b62 + WORD $0xaa1f03e9 + WORD $0x14000063 + WORD $0xd1001149 + WORD $0xf10004ab + WORD $0x4d40c920 + WORD $0x5400044d + WORD $0xaa1f03ec + WORD $0x5280020d + WORD $0x8b0c082e + WORD $0x4ea01c10 + WORD $0x4ea01c07 + WORD $0x4ea01c06 + WORD $0x4ea01c02 + WORD $0xaa0d03e9 + WORD $0xad400dc1 + WORD $0xd100214d + WORD $0xad4115c4 + WORD $0xaa0b03ee + WORD $0xbc5fc5b1 + WORD $0x6e21de10 + WORD $0x6e23dce7 + WORD $0x6e24dcc6 + WORD $0x6e25dc42 + WORD $0x4e040631 + WORD $0xd10005ce + WORD $0xf10001df + WORD $0x4e30d630 + WORD $0x4e27d627 + WORD $0x4e26d626 + WORD $0x4e22d622 + WORD $0x54fffe8c + WORD $0x8b0c080e + WORD $0x9100412d + WORD $0xaa0903ec + WORD $0xeb0801bf + WORD $0xad001dd0 + WORD $0xad0109c6 + WORD $0x54fffc6d + WORD $0x14000009 WORD $0xaa1f03e9 - WORD $0xd10004aa - WORD $0xd100216b + WORD $0x9100800b + WORD $0x9100812c + WORD $0x91004129 + WORD $0xad3f0160 + WORD $0xeb08019f + WORD $0xac820160 + WORD $0x54ffff6d + WORD $0xeb09010b + WORD $0x5400072d + WORD $0xbc5fc140 + WORD $0xf10008bf + WORD $0x5400024b + WORD $0xd10004ab + WORD $0xd100214a WORD $0x1e204002 WORD $0xbc697821 - WORD $0xaa0b03ec - WORD $0xaa0a03ed + WORD $0xaa0a03ec + WORD $0xaa0b03ed WORD $0x1e220822 WORD $0xbc5fc583 WORD $0xd10005ad @@ -257,38 +311,34 @@ TEXT ·polyEvalFloat32NEON(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x91000529 WORD $0xeb08013f WORD $0x54fffe61 - WORD $0x14000029 - WORD $0x5400050b - WORD $0xf100211f - WORD $0x540000e2 - WORD $0xaa1f03e9 - WORD $0x1400000f - WORD $0xf100211f - WORD $0x54000262 - WORD $0xaa1f03e9 - WORD $0x1400001b - WORD $0x6f00e400 - WORD $0x927ded09 - WORD $0x9100400a - WORD $0x927ded0b - WORD $0xf100216b - WORD $0xad3f8140 - WORD $0x9100814a + WORD $0x14000025 + WORD $0xf100217f + WORD $0x540001c3 + WORD $0x9240090a + WORD $0x4e040401 + WORD $0x8b09080c + WORD $0xcb0a016b + WORD $0x8b0a012d + WORD $0x8b0b0129 + WORD $0x9100418b + WORD $0xcb0801ac + WORD $0xb100218c + WORD $0xad3f8561 + WORD $0x9100816b WORD $0x54ffffa1 - WORD $0xeb09011f - WORD $0x540002c0 + WORD $0xb40002ca WORD $0x8b09080a WORD $0xcb090108 WORD $0xf1000508 - WORD $0xb800455f + WORD $0xbc004540 WORD $0x54ffffc1 WORD $0x14000010 - WORD $0x4e040401 + WORD $0x6f00e400 WORD $0x927ded09 WORD $0x9100400a WORD $0x927ded0b WORD $0xf100216b - WORD $0xad3f8541 + WORD $0xad3f8140 WORD $0x9100814a WORD $0x54ffffa1 WORD $0xeb09011f @@ -296,7 +346,7 @@ TEXT ·polyEvalFloat32NEON(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x8b09080a WORD $0xcb090108 WORD $0xf1000508 - WORD $0xbc004540 + WORD $0xb800455f WORD $0x54ffffc1 WORD $0xd65f03c0 RET @@ -309,41 +359,346 @@ TEXT ·convolveFloat32NEON(SB), NOSPLIT|NOFRAME, $0-72 MOVD dst_len+8(FP), R3 MOVD sig_len+32(FP), R4 MOVD ker_len+56(FP), R5 - WORD $0xb4000ae5 + WORD $0xb40031e5 WORD $0xeb050088 - WORD $0x54000aab + WORD $0x540031ab + WORD $0xd10443ff WORD $0xeb08007f + WORD $0x6d0c3bef WORD $0x9a88d468 - WORD $0xf100051f - WORD $0x54000a2b - WORD $0xf10004bf - WORD $0x5400078b - WORD $0x8b05084b + WORD $0x6d0d33ed + WORD $0xf100411f + WORD $0x6d0e2beb + WORD $0x6d0f23e9 + WORD $0xf90083fd + WORD $0x5400006a WORD $0xaa1f03e9 - WORD $0x927decaa - WORD $0x9100402c + WORD $0x14000128 + WORD $0xf10000bf + WORD $0x540023ad + WORD $0x8b050849 + WORD $0xaa1f03ef + WORD $0x927ef0aa + WORD $0x9100802b WORD $0xd100104d - WORD $0xd100416b + WORD $0x9100f02e + WORD $0xd100412c + WORD $0x52800210 + WORD $0x1400000e + WORD $0x8b0f0811 + WORD $0x91004130 + WORD $0x9101016b + WORD $0xeb08021f + WORD $0x910101ce + WORD $0xaa0903ef + WORD $0xbd000224 + WORD $0x3c804223 + WORD $0x3c814225 + WORD $0x3c824226 + WORD $0x2d06f637 + WORD $0xbd003e34 + WORD $0x5400220c + WORD $0xf10010bf + WORD $0xaa1003e9 + WORD $0x54000142 + WORD $0x6f00e403 + WORD $0x2f00e414 + WORD $0xaa1f03f1 + WORD $0x2f00e41d + WORD $0x2f00e417 + WORD $0x2f00e404 + WORD $0x6f00e405 + WORD $0x6f00e406 + WORD $0x140000de + WORD $0x2f00e414 + WORD $0x2f00e41d + WORD $0xaa0b03e4 + WORD $0x2f00e417 + WORD $0x2f00e407 + WORD $0xaa0c03f0 + WORD $0x2f00e402 + WORD $0x2f00e416 + WORD $0x927ef0b1 + WORD $0x2f00e406 + WORD $0x2f00e40f + WORD $0xaa0b03e3 + WORD $0x2f00e413 + WORD $0x2f00e415 + WORD $0x2f00e405 + WORD $0x2f00e410 + WORD $0x2f00e411 + WORD $0x2f00e400 + WORD $0x2f00e403 + WORD $0x2f00e404 + WORD $0x3cdf0618 + WORD $0x3cc1c099 + WORD $0x3cc1809a + WORD $0x3d802be0 + WORD $0x4ea21c52 + WORD $0x4ea00b18 + WORD $0xad0157f1 + WORD $0x3cc0c089 + WORD $0x4eaf1df5 + WORD $0xf1001231 + WORD $0x6e184318 + WORD $0x6e39df0b + WORD $0x3cc14099 + WORD $0x6e3adf0c + WORD $0x3cc10c7a + WORD $0x6e39df0d + WORD $0x3cde0099 + WORD $0x6e3adf0e + WORD $0x3cde409a + WORD $0x5e0c057b + WORD $0x1e2b2a80 + WORD $0x6e39df01 + WORD $0x5e0c059c + WORD $0x1e2c2ba2 + WORD $0x5e140568 + WORD $0x5e0c05bd + WORD $0x5e14059f + WORD $0x5e1c056b + WORD $0x5e0c05de + WORD $0x1e2e28e7 + WORD $0x5e1405aa + WORD $0x3d802fe1 + WORD $0x6e3adf01 + WORD $0x1e3b2800 + WORD $0x1e3c2842 + WORD $0x5e1c058c + WORD $0x3cde809b + WORD $0x5e1405cf + WORD $0x6e29df14 + WORD $0x5e1c05d9 + WORD $0x1e3e28e7 + WORD $0x6e3bdf11 + WORD $0x4eb01e1e + WORD $0x3d8027e1 + WORD $0x1e2d2ae1 + WORD $0x1e282800 + WORD $0x1e3f285a + WORD $0x5e1c05ad + WORD $0x3cc08090 + WORD $0x1e342a57 + WORD $0x3cdf8088 + WORD $0x3cdec09c + WORD $0x6e30df1b + WORD $0x1e2f28e7 + WORD $0xad02c7f4 + WORD $0x1e3d2821 + WORD $0x1e2b2800 + WORD $0x3cc04091 + WORD $0x5e0c0690 + WORD $0xad7fbc89 + WORD $0x6e31df02 + WORD $0x6e28df1f + WORD $0x6e3cdf1c + WORD $0x1e3b2ace + WORD $0x5e0c0774 + WORD $0x3cdf4096 + WORD $0x1e2a2821 + WORD $0xbd010fe0 + WORD $0x1e2c2b40 + WORD $0x1e302af0 + WORD $0x3cdfc08a + WORD $0x6e29df1d + WORD $0x1e2228d2 + WORD $0x1e204086 + WORD $0x4eb31e64 + WORD $0x5e0c0453 + WORD $0x6e2adf08 + WORD $0xad000bfb + WORD $0xbd010be0 + WORD $0x1e2d2820 + WORD $0x4ebb1f6a + WORD $0x6e36df1b + WORD $0x6e2fdf16 + WORD $0x5e0c07fa + WORD $0x5e0c0798 + WORD $0x5e0c07af + WORD $0xaa0303e4 + WORD $0x1e28288c + WORD $0x4ea81d0b + WORD $0x5e0c0509 + WORD $0xbd008fe0 + WORD $0x1e3928e0 + WORD $0x5e0c0767 + WORD $0x1e292989 + WORD $0x5e1406cc + WORD $0x3d801fe0 + WORD $0x1e3429c0 + WORD $0x4ebf1ff4 + WORD $0x2d0943e0 + WORD $0x1e332a40 + WORD $0x1e362ab3 + WORD $0x5e0c06d2 + WORD $0xbd0047e0 + WORD $0xad4103e1 + WORD $0x1e3f2815 + WORD $0xad4483f1 + WORD $0x1e3c282e + WORD $0x3dc01be1 + WORD $0x1e3b28bf + WORD $0x4ebd1fa5 + WORD $0x1e322a7d + WORD $0x1e212800 + WORD $0x5e0c0437 + WORD $0x1e3a2aa8 + WORD $0x5e140455 + WORD $0x4eb41e82 + WORD $0x1e252bcd + WORD $0x1e31287e + WORD $0x4ea11c23 + WORD $0x3dc02fe1 + WORD $0x1e272bf9 + WORD $0x1e3829df + WORD $0x1e372813 + WORD $0x3dc017e0 + WORD $0x5e140557 + WORD $0x1e2128c4 + WORD $0x5e0c0430 + WORD $0x5e140578 + WORD $0x5e14040a + WORD $0x4ea11c26 + WORD $0x5e0c0621 + WORD $0x1e2f29ba + WORD $0x5e14078d + WORD $0x5e1404ae + WORD $0x5e14076f + WORD $0x5e1c079c + WORD $0x1e302887 + WORD $0x5e140690 + WORD $0x4eab1d74 + WORD $0xbd404feb + WORD $0x1e212bd2 + WORD $0x4ea51cbe + WORD $0x5e1404c5 + WORD $0x5e140624 + WORD $0x5e140461 + WORD $0x1e2a296a + WORD $0x1e2c2bab + WORD $0x1e2f2b2c + WORD $0x1e302910 + WORD $0x5e1c0459 + WORD $0x5e1c06dd + WORD $0x5e1c07de + WORD $0x5e1c04c8 + WORD $0xbd00a3ea + WORD $0x2d488bea + WORD $0x1e3d296f + WORD $0xbd410bfd + WORD $0x1e372857 + WORD $0x1e352955 + WORD $0x1e38292a + WORD $0x1e2e2b58 + WORD $0x1e2d2bee + WORD $0x1e212a6d + WORD $0x1e242a41 + WORD $0x1e2528f2 + WORD $0xad4013e5 + WORD $0x5e1c0407 + WORD $0x5e1c069a + WORD $0x5e1c077f + WORD $0x5e1c0629 + WORD $0x5e1c047b + WORD $0xbd40a3e0 + WORD $0x5e1c0484 + WORD $0x5e1c04a5 + WORD $0x1e3c29d1 + WORD $0xbd410ff4 + WORD $0x1e272802 + WORD $0x1e3a2953 + WORD $0x3dc01fe7 + WORD $0x1e3b29a0 + WORD $0x1e292823 + WORD $0x1e252af6 + WORD $0x1e242aa6 + WORD $0x1e392a15 + WORD $0x1e3f2985 + WORD $0x1e3e2b10 + WORD $0x1e282a44 + WORD $0xbd408ff7 + WORD $0x54ffe881 + WORD $0x6e0c0403 + WORD $0x6e0c06a5 + WORD $0xeb0a00bf + WORD $0x6e0c06c6 + WORD $0x927ef0b1 + WORD $0x6e140623 + WORD $0x6e140665 + WORD $0x6e140446 + WORD $0x6e1c0603 + WORD $0x6e1c05e5 + WORD $0x6e1c04e6 + WORD $0x54ffe160 + WORD $0xd37ef630 + WORD $0xcb1100b1 + WORD $0xaa0e03e3 + WORD $0x8b100064 + WORD $0xbc7179a0 + WORD $0xf1000631 + WORD $0x2d7f4891 + WORD $0xbc5c4081 + WORD $0x3cdc8082 + WORD $0x3cdd8087 + WORD $0x91001063 + WORD $0x3cde8090 + WORD $0xbd400093 + WORD $0x1e210801 + WORD $0x4f809042 + WORD $0x4f8090e7 + WORD $0x1e310811 + WORD $0x4f809210 + WORD $0x1e320812 + WORD $0x1e330800 + WORD $0x1e212884 + WORD $0x4e22d463 + WORD $0x4e27d4a5 + WORD $0x1e312af7 + WORD $0x4e30d4c6 + WORD $0x1e322bbd + WORD $0x1e202a94 + WORD $0x54fffd01 + WORD $0x17fffeee + WORD $0x6f00e400 + WORD $0xaa1f03e9 + WORD $0x9100800a + WORD $0x9100812b + WORD $0x91004129 + WORD $0xad3f0140 + WORD $0xeb08017f + WORD $0xac820140 + WORD $0x54ffff6d + WORD $0xeb09010b + WORD $0x54000a4d + WORD $0xf10004bf + WORD $0x5400078b + WORD $0x8b09082a + WORD $0x8b05084d + WORD $0x927decab + WORD $0xd100104e + WORD $0x9100414c + WORD $0xd10041ad WORD $0x14000007 WORD $0xbc297800 WORD $0x91000529 WORD $0x9100118c WORD $0xeb08013f - WORD $0x91001021 - WORD $0x54000840 + WORD $0x9100114a + WORD $0x54000860 WORD $0x2f00e400 WORD $0xf10020bf WORD $0x54000062 - WORD $0xaa1f03ef + WORD $0xaa1f03f0 WORD $0x14000021 - WORD $0xaa0c03ee - WORD $0xaa0b03ef - WORD $0x927decb0 + WORD $0xaa0d03ef + WORD $0xaa0c03f0 + WORD $0x927decb1 WORD $0xad7f85e2 - WORD $0xd1002210 - WORD $0xad7f91c3 + WORD $0xd1002231 + WORD $0xad7f9203 + WORD $0x91008210 WORD $0xd10081ef - WORD $0x910081ce WORD $0x4ea00821 WORD $0x4ea00842 WORD $0x6e014021 @@ -364,38 +719,45 @@ TEXT ·convolveFloat32NEON(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x5e1c0441 WORD $0x1e232800 WORD $0x1e212800 - WORD $0xb5fffcf0 - WORD $0xeb0a00bf - WORD $0x927decaf + WORD $0xb5fffcf1 + WORD $0xeb0b00bf + WORD $0x927decb0 WORD $0x54fffac0 - WORD $0x8b0f082e - WORD $0xcb0f00af - WORD $0xbc4045c1 - WORD $0xbc6f79a2 - WORD $0xf10005ef + WORD $0x8b10094f + WORD $0xcb1000b0 + WORD $0xbc4045e1 + WORD $0xbc7079c2 + WORD $0xf1000610 WORD $0x1e220821 WORD $0x1e212800 WORD $0x54ffff61 WORD $0x17ffffcd - WORD $0xf100211f - WORD $0x54000062 - WORD $0xaa1f03e9 - WORD $0x1400000b + WORD $0xf100217f + WORD $0x540001c3 WORD $0x6f00e400 - WORD $0x927ded09 - WORD $0x9100400a - WORD $0x927ded0b - WORD $0xf100216b - WORD $0xad3f8140 - WORD $0x9100814a + WORD $0x9240090a + WORD $0x8b09080c + WORD $0xcb0a016b + WORD $0x8b0a012d + WORD $0x8b0b0129 + WORD $0x9100418b + WORD $0xcb0801ac + WORD $0xb100218c + WORD $0xad3f8160 + WORD $0x9100816b WORD $0x54ffffa1 - WORD $0xeb09011f - WORD $0x540000c0 + WORD $0xb40000ca WORD $0x8b09080a WORD $0xcb090108 WORD $0xf1000508 WORD $0xb800455f WORD $0x54ffffc1 + WORD $0x6d4f23e9 + WORD $0xf94083fd + WORD $0x6d4e2beb + WORD $0x6d4d33ed + WORD $0x6d4c3bef + WORD $0x910443ff WORD $0xd65f03c0 RET @@ -407,38 +769,345 @@ TEXT ·correlateFloat32NEON(SB), NOSPLIT|NOFRAME, $0-72 MOVD dst_len+8(FP), R3 MOVD sig_len+32(FP), R4 MOVD ker_len+56(FP), R5 - WORD $0xb4000a45 + WORD $0xb4003185 WORD $0xeb050088 - WORD $0x54000a0b + WORD $0x5400314b + WORD $0xd10503ff WORD $0xeb08007f + WORD $0x6d0f3bef WORD $0x9a88d468 - WORD $0xf100051f - WORD $0x5400098b + WORD $0x6d1033ed + WORD $0xf100411f + WORD $0x6d112beb + WORD $0x6d1223e9 + WORD $0xf9009bfd + WORD $0x5400006a + WORD $0xaa1f03e9 + WORD $0x1400012a + WORD $0xf10000bf + WORD $0x540023ed + WORD $0xaa1f03ed + WORD $0x927ef0aa + WORD $0x9100802b + WORD $0x5280020e + WORD $0xaa0103ec + WORD $0x1400000e + WORD $0x8b0d080f + WORD $0x9100412e + WORD $0x9101016b + WORD $0xeb0801df + WORD $0x9101018c + WORD $0xaa0903ed + WORD $0xbd0001e4 + WORD $0x3c8041e3 + WORD $0x3c8141e5 + WORD $0x3c8241e6 + WORD $0x2d06f5f5 + WORD $0xbd003df4 + WORD $0x540022ac + WORD $0xf10010bf + WORD $0xaa0e03e9 + WORD $0x54000142 + WORD $0x6f00e403 + WORD $0x2f00e414 + WORD $0xaa1f03ef + WORD $0x2f00e41d + WORD $0x2f00e415 + WORD $0x2f00e404 + WORD $0x6f00e405 + WORD $0x6f00e406 + WORD $0x140000e1 + WORD $0x2f00e400 + WORD $0x2f00e414 + WORD $0xaa0b03f1 + WORD $0x2f00e41d + WORD $0x2f00e415 + WORD $0xaa0203ee + WORD $0x2f00e407 + WORD $0x2f00e416 + WORD $0x927ef0af + WORD $0x2f00e412 + WORD $0x2f00e406 + WORD $0xaa0b03f0 + WORD $0x3d8037e0 + WORD $0x2f00e418 + WORD $0x2f00e413 + WORD $0x2f00e405 + WORD $0x2f00e410 + WORD $0x2f00e411 + WORD $0x2f00e400 + WORD $0x2f00e403 + WORD $0x2f00e404 + WORD $0x3d8033f8 + WORD $0x3cc105d8 + WORD $0x3cc1c239 + WORD $0x3cc1823a + WORD $0xf10011ef + WORD $0x3d802be0 + WORD $0x3cc0c229 + WORD $0x6e39df0b + WORD $0x3cc14239 + WORD $0x6e3adf0c + WORD $0x3cc10e1a + WORD $0x3d8023f3 + WORD $0x6e39df0d + WORD $0x3cde0239 + WORD $0x3d800bf0 + WORD $0x6e3adf0e + WORD $0x3cde423a + WORD $0x3cc04230 + WORD $0x5e0c057b + WORD $0x1e2b2a80 + WORD $0x6e39df01 + WORD $0x5e0c059c + WORD $0x1e2c2ba2 + WORD $0x5e140568 + WORD $0x5e0c05bd + WORD $0x5e14059f + WORD $0x5e1c056b + WORD $0x5e0c05de + WORD $0x1e2e28e7 + WORD $0x5e1405aa + WORD $0x3d802fe1 + WORD $0x6e3adf01 + WORD $0x1e3b2800 + WORD $0x1e3c2842 + WORD $0x5e1c058c + WORD $0x3cde823b + WORD $0x5e1405cf + WORD $0x6e30df10 + WORD $0x4eb11e3a + WORD $0x6e3bdf19 + WORD $0x1e3e28e7 + WORD $0x3cdec23c + WORD $0x3d8027e1 + WORD $0x1e2d2aa1 + WORD $0x1e282800 + WORD $0x1e3f2842 + WORD $0x5e1c05ad + WORD $0x3cc08228 + WORD $0x5e1c05d5 + WORD $0x3cdf023e + WORD $0x3cdf823f + WORD $0x6e29df13 + WORD $0x6e28df14 + WORD $0x1e3d2821 + WORD $0x1e2b2800 + WORD $0x1e2f28e7 + WORD $0x3cdf422e + WORD $0x6e3cdf1b + WORD $0x6e3edf1c + WORD $0x6e3fdf1e + WORD $0x3d8003f0 + WORD $0x3d803bf4 + WORD $0x1e332acf + WORD $0x1e342a51 + WORD $0x1e2a2821 + WORD $0xbd013fe0 + WORD $0x1e2c2840 + WORD $0x5e0c0677 + WORD $0x5e0c0694 + WORD $0xad0267f3 + WORD $0x1e3028d2 + WORD $0x5e0c0613 + WORD $0x3cdfc22a + WORD $0x3dc00236 + WORD $0x6e2edf19 + WORD $0x5e0c077f + WORD $0xbd013be0 + WORD $0x1e2d2820 + WORD $0x6e2adf1d + WORD $0x1e3729e1 + WORD $0x6e36df16 + WORD $0x4ebb1f6a + WORD $0x3dc017e2 + WORD $0x5e0c07c9 + WORD $0x5e0c078f + WORD $0x1e3928a6 + WORD $0xaa1003f1 + WORD $0xbd007fe0 + WORD $0x1e3528e0 + WORD $0x4ebd1fb8 + WORD $0x5e0c06d7 + WORD $0x5e0c0727 + WORD $0x1e2a2b4e + WORD $0x4ebe1fda + WORD $0x3d801be0 + WORD $0x1e342a20 + WORD $0x5e0c07b1 + WORD $0x1e2728c8 + WORD $0x5e1407c7 + WORD $0x1e3f29df + WORD $0x5e14078e + WORD $0x5e1c075a + WORD $0x2d0707e0 + WORD $0x1e332a40 + WORD $0xbd001fe0 + WORD $0xad4607e0 + WORD $0x1e382814 + WORD $0xad444fe0 + WORD $0x1e36283b + WORD $0xad454be1 + WORD $0x1e3e2815 + WORD $0x3dc00be0 + WORD $0x1e332863 + WORD $0x1e222825 + WORD $0x5e0c0441 + WORD $0x1e312a91 + WORD $0x1e3c280d + WORD $0x5e0c0660 + WORD $0x1e372b6b + WORD $0x4eaa1d5e + WORD $0x1e322884 + WORD $0x5e0c064c + WORD $0x1e292aa9 + WORD $0x5e140615 + WORD $0x5e140646 + WORD $0x1e2128b7 + WORD $0x3dc03be1 + WORD $0xbd00c3f1 + WORD $0x1e202874 + WORD $0x3dc013e0 + WORD $0x1e2f29bd + WORD $0x5e140430 + WORD $0x5e14054d + WORD $0xbd403fea + WORD $0x5e140403 + WORD $0x1e2c2891 + WORD $0x5e140704 + WORD $0x5e1406cc + WORD $0x5e14072f + WORD $0x5e140665 + WORD $0x5e140441 + WORD $0x4ebc1f9b + WORD $0xbd401ffc + WORD $0x1e2e2bae + WORD $0x5e1c071d + WORD $0x1e232943 + WORD $0xbd403bea + WORD $0x1e352b95 + WORD $0x1e2c296b + WORD $0x1e2f290c + WORD $0x1e2d2bef + WORD $0x1e302950 + WORD $0x5e1c06df + WORD $0x1e212aed + WORD $0x1e252a81 + WORD $0x1e262a37 + WORD $0x3dc003e5 + WORD $0x5e1c0411 + WORD $0x5e1c07dc + WORD $0x5e1c077e + WORD $0x5e1c04a6 + WORD $0x5e1c0648 + WORD $0x5e1c045b + WORD $0xbd00d3f0 + WORD $0xbd40c3f0 + WORD $0xbd413ff4 + WORD $0xbd40d3e0 + WORD $0x1e242a0a + WORD $0x1e272924 + WORD $0x3dc03be7 + WORD $0x5e1c0730 + WORD $0x5e1c0669 + WORD $0x1e312876 + WORD $0x5e1c04e7 + WORD $0x1e262aa6 + WORD $0x1e3c29f1 + WORD $0xbd407ff5 + WORD $0x1e3d2958 + WORD $0x1e3a2893 + WORD $0x1e282ae4 + WORD $0x1e302985 + WORD $0x1e3e29d0 + WORD $0x1e292823 + WORD $0x1e272812 + WORD $0x1e3f2960 + WORD $0x3dc01be7 + WORD $0xbd413bfd + WORD $0x3d8037e0 + WORD $0x1e3b29a0 + WORD $0x54ffe861 + WORD $0x6e0c0403 + WORD $0x6e0c0665 + WORD $0x3dc037e0 + WORD $0x6e0c0646 + WORD $0xeb0a00bf + WORD $0x927ef0af + WORD $0x6e140623 + WORD $0x6e140705 + WORD $0x6e1406c6 + WORD $0x6e1c0603 + WORD $0x6e1c0405 + WORD $0x6e1c04e6 + WORD $0x54ffe100 + WORD $0xd37ef5ee + WORD $0xcb0f00af + WORD $0xaa0c03f0 + WORD $0xaa0203f1 + WORD $0x8b0e0203 + WORD $0xbc6e6a20 + WORD $0xf10005ef + WORD $0x2d46c871 + WORD $0xbd400061 + WORD $0x3cc04062 + WORD $0x3cc14067 + WORD $0x91001231 + WORD $0x3cc24070 + WORD $0xbd403c73 + WORD $0x1e210801 + WORD $0x4f809042 + WORD $0x4f8090e7 + WORD $0x1e310811 + WORD $0x4f809210 + WORD $0x1e320812 + WORD $0x1e330800 + WORD $0x91001210 + WORD $0x1e212884 + WORD $0x4e22d463 + WORD $0x4e27d4a5 + WORD $0x1e312ab5 + WORD $0x4e30d4c6 + WORD $0x1e322bbd + WORD $0x1e202a94 + WORD $0x54fffce1 + WORD $0x17fffee9 + WORD $0x6f00e400 + WORD $0xaa1f03e9 + WORD $0x9100800a + WORD $0x9100812b + WORD $0x91004129 + WORD $0xad3f0140 + WORD $0xeb08017f + WORD $0xac820140 + WORD $0x54ffff6d + WORD $0xeb09010b + WORD $0x540009ad WORD $0xf10004bf WORD $0x540006eb - WORD $0xaa1f03e9 - WORD $0x927decaa - WORD $0x9100402b + WORD $0x8b09082a + WORD $0x927decab + WORD $0x9100414c WORD $0x14000007 WORD $0xbc297800 WORD $0x91000529 - WORD $0x9100116b + WORD $0x9100118c WORD $0xeb08013f - WORD $0x91001021 - WORD $0x54000800 + WORD $0x9100114a + WORD $0x54000820 WORD $0x2f00e400 WORD $0xf10020bf WORD $0x54000062 - WORD $0xaa1f03ec + WORD $0xaa1f03ed WORD $0x1400001d - WORD $0xaa0b03ec - WORD $0x9100404d - WORD $0x927decae - WORD $0xad7f91a1 - WORD $0xf10021ce - WORD $0xad7f8d82 + WORD $0xaa0c03ed + WORD $0x9100404e + WORD $0x927decaf + WORD $0xad7f91c1 + WORD $0xf10021ef + WORD $0xad7f8da2 + WORD $0x910081ce WORD $0x910081ad - WORD $0x9100818c WORD $0x6e21dc41 WORD $0x5e0c0422 WORD $0x1e212800 @@ -456,39 +1125,46 @@ TEXT ·correlateFloat32NEON(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x1e232800 WORD $0x1e212800 WORD $0x54fffd61 - WORD $0xeb0a00bf - WORD $0x927decac + WORD $0xeb0b00bf + WORD $0x927decad WORD $0x54fffb40 - WORD $0xd37ef58e - WORD $0xcb0c00ac - WORD $0x8b0e004d - WORD $0x8b0e002e - WORD $0xbc4045c1 - WORD $0xf100058c - WORD $0xbc4045a2 + WORD $0xd37ef5af + WORD $0xcb0d00ad + WORD $0x8b0f004e + WORD $0x8b0f014f + WORD $0xbc4045e1 + WORD $0xf10005ad + WORD $0xbc4045c2 WORD $0x1e220821 WORD $0x1e212800 WORD $0x54ffff61 WORD $0x17ffffcf - WORD $0xf100211f - WORD $0x54000062 - WORD $0xaa1f03e9 - WORD $0x1400000b + WORD $0xf100217f + WORD $0x540001c3 WORD $0x6f00e400 - WORD $0x927ded09 - WORD $0x9100400a - WORD $0x927ded0b - WORD $0xf100216b - WORD $0xad3f8140 - WORD $0x9100814a + WORD $0x9240090a + WORD $0x8b09080c + WORD $0xcb0a016b + WORD $0x8b0a012d + WORD $0x8b0b0129 + WORD $0x9100418b + WORD $0xcb0801ac + WORD $0xb100218c + WORD $0xad3f8160 + WORD $0x9100816b WORD $0x54ffffa1 - WORD $0xeb09011f - WORD $0x540000c0 + WORD $0xb40000ca WORD $0x8b09080a WORD $0xcb090108 WORD $0xf1000508 WORD $0xb800455f WORD $0x54ffffc1 + WORD $0x6d5223e9 + WORD $0xf9409bfd + WORD $0x6d512beb + WORD $0x6d5033ed + WORD $0x6d4f3bef + WORD $0x910503ff WORD $0xd65f03c0 RET @@ -733,20 +1409,97 @@ TEXT ·polyEvalFloat64NEON(SB), NOSPLIT|NOFRAME, $0-72 MOVD coeffs_len+56(FP), R5 WORD $0xeb04007f WORD $0x9a84b068 + WORD $0xb40000c5 + WORD $0x8b050c4a + WORD $0xf100411f + WORD $0x5400012a + WORD $0xaa1f03e9 + WORD $0x1400004b WORD $0xf100051f - WORD $0xb4000305 - WORD $0x540007eb - WORD $0x8b050c4b - WORD $0xf10008bf - WORD $0xfc5f8160 - WORD $0x5400030b + WORD $0x5400106b + WORD $0xf100111f + WORD $0x54000e42 + WORD $0xaa1f03e9 + WORD $0x1400007a + WORD $0xfc5f8140 + WORD $0xf10004ab + WORD $0x5400066d + WORD $0x4e080400 + WORD $0xaa1f03ec + WORD $0x5280020d + WORD $0x8b0c0c2e + WORD $0x4ea01c18 + WORD $0x4ea01c17 + WORD $0x4ea01c16 + WORD $0x4ea01c15 + WORD $0xaa0d03e9 + WORD $0xad4009c1 + WORD $0x4ea01c14 + WORD $0xad4111c3 + WORD $0x4ea01c13 + WORD $0xad4219c5 + WORD $0x4ea01c12 + WORD $0xad4341c7 + WORD $0x4ea01c11 + WORD $0xd100414d + WORD $0xaa0b03ee + WORD $0xfc5f85b9 + WORD $0x6e61df18 + WORD $0x6e62def7 + WORD $0x6e63ded6 + WORD $0x6e64deb5 + WORD $0x6e65de94 + WORD $0x6e66de73 + WORD $0xd10005ce + WORD $0x6e67de52 + WORD $0x6e70de31 + WORD $0xf10001df + WORD $0x4e080739 + WORD $0x4e78d738 + WORD $0x4e77d737 + WORD $0x4e76d736 + WORD $0x4e75d735 + WORD $0x4e74d734 + WORD $0x4e73d733 + WORD $0x4e72d732 + WORD $0x4e71d731 + WORD $0x54fffd8c + WORD $0x8b0c0c0e + WORD $0x9100412d + WORD $0xaa0903ec + WORD $0xeb0801bf + WORD $0xad005dd8 + WORD $0xad0155d6 + WORD $0xad024dd4 + WORD $0xad0345d2 + WORD $0x54fffa6d + WORD $0x14000010 WORD $0xaa1f03e9 - WORD $0xd10004aa - WORD $0xd100416b + WORD $0x9101000b + WORD $0x9100812c + WORD $0x6d3c0160 + WORD $0x91004129 + WORD $0x6d3d0160 + WORD $0xeb08019f + WORD $0x6d3e0160 + WORD $0x6d3f0160 + WORD $0x6d000160 + WORD $0x6d010160 + WORD $0x6d020160 + WORD $0x6d030160 + WORD $0x9102016b + WORD $0x54fffe8d + WORD $0xeb09010b + WORD $0x5400072d + WORD $0xfc5f8140 + WORD $0xf10008bf + WORD $0x5400024b + WORD $0xd10004ab + WORD $0xd100414a WORD $0x1e604002 WORD $0xfc697821 - WORD $0xaa0b03ec - WORD $0xaa0a03ed + WORD $0xaa0a03ec + WORD $0xaa0b03ed WORD $0x1e620822 WORD $0xfc5f8583 WORD $0xd10005ad @@ -757,38 +1510,34 @@ TEXT ·polyEvalFloat64NEON(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x91000529 WORD $0xeb08013f WORD $0x54fffe61 - WORD $0x14000029 - WORD $0x5400050b - WORD $0xf100111f - WORD $0x540000e2 - WORD $0xaa1f03e9 - WORD $0x1400000f - WORD $0xf100111f - WORD $0x54000262 - WORD $0xaa1f03e9 - WORD $0x1400001b - WORD $0x6f00e400 - WORD $0x927ef109 - WORD $0x9100400a - WORD $0x927ef10b - WORD $0xf100116b - WORD $0xad3f8140 - WORD $0x9100814a + WORD $0x14000025 + WORD $0xf100117f + WORD $0x540001c3 + WORD $0x9240050a + WORD $0x4e080401 + WORD $0x8b090c0c + WORD $0xcb0a016b + WORD $0x8b0a012d + WORD $0x8b0b0129 + WORD $0x9100418b + WORD $0xcb0801ac + WORD $0xb100118c + WORD $0xad3f8561 + WORD $0x9100816b WORD $0x54ffffa1 - WORD $0xeb09011f - WORD $0x540002c0 + WORD $0xb40002ca WORD $0x8b090c0a WORD $0xcb090108 WORD $0xf1000508 - WORD $0xf800855f + WORD $0xfc008540 WORD $0x54ffffc1 WORD $0x14000010 - WORD $0x4e080401 + WORD $0x6f00e400 WORD $0x927ef109 WORD $0x9100400a WORD $0x927ef10b WORD $0xf100116b - WORD $0xad3f8541 + WORD $0xad3f8140 WORD $0x9100814a WORD $0x54ffffa1 WORD $0xeb09011f @@ -796,7 +1545,7 @@ TEXT ·polyEvalFloat64NEON(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x8b090c0a WORD $0xcb090108 WORD $0xf1000508 - WORD $0xfc008540 + WORD $0xf800855f WORD $0x54ffffc1 WORD $0xd65f03c0 RET @@ -809,41 +1558,252 @@ TEXT ·convolveFloat64NEON(SB), NOSPLIT|NOFRAME, $0-72 MOVD dst_len+8(FP), R3 MOVD sig_len+32(FP), R4 MOVD ker_len+56(FP), R5 - WORD $0xb40009a5 + WORD $0xb40024c5 WORD $0xeb050088 - WORD $0x5400096b + WORD $0x5400248b + WORD $0xd10143ff WORD $0xeb08007f + WORD $0x6d013bef WORD $0x9a88d468 - WORD $0xf100051f - WORD $0x540008eb - WORD $0xf10004bf - WORD $0x5400064b - WORD $0x8b050c4b + WORD $0x6d0233ed + WORD $0xf100411f + WORD $0x6d032beb + WORD $0x6d0423e9 + WORD $0x5400006a WORD $0xaa1f03e9 - WORD $0x927ef0aa - WORD $0x9100402c + WORD $0x140000cb + WORD $0xf10000bf + WORD $0x5400176d + WORD $0x8b050c49 + WORD $0xaa1f03ef + WORD $0x927ff4aa + WORD $0x9101002b WORD $0xd100204d - WORD $0xd100416b + WORD $0x9101e02e + WORD $0xd100412c + WORD $0x52800210 + WORD $0x14000010 + WORD $0x8b0f0c0f + WORD $0x91004130 + WORD $0x9102016b + WORD $0xeb08021f + WORD $0x910201ce + WORD $0x6d0051f7 + WORD $0x6d0159f2 + WORD $0x6d0245f5 + WORD $0x6d0341f3 + WORD $0x6d041de6 + WORD $0x6d0539e5 + WORD $0x6d0609e3 + WORD $0x6d0701e1 + WORD $0xaa0903ef + WORD $0x5400162c + WORD $0x2f00e400 + WORD $0x2f00e401 + WORD $0xf10004bf + WORD $0x2f00e402 + WORD $0x2f00e403 + WORD $0xaa1003e9 + WORD $0x2f00e40e + WORD $0x2f00e405 + WORD $0x2f00e407 + WORD $0x2f00e406 + WORD $0x2f00e410 + WORD $0x2f00e413 + WORD $0x2f00e411 + WORD $0x2f00e415 + WORD $0x2f00e416 + WORD $0x2f00e412 + WORD $0x2f00e414 + WORD $0x2f00e417 + WORD $0x54000061 + WORD $0xaa1f03f1 + WORD $0x1400005d + WORD $0xaa0b03e4 + WORD $0xaa0c03f0 + WORD $0x927ff4b1 + WORD $0xaa0b03e3 + WORD $0x3cdf0618 + WORD $0x3cdc0099 + WORD $0xad41749a + WORD $0xf1000a31 + WORD $0x6e184318 + WORD $0x3cc2809c + WORD $0x3cdd8089 + WORD $0x3cde808b + WORD $0x3cdf008c + WORD $0xfd0007ee + WORD $0x3cc18084 + WORD $0x6e79df1b + WORD $0x3cc38099 + WORD $0x6e7adf1e + WORD $0x6e7cdf1f + WORD $0x6e7ddf1d + WORD $0x6e69df09 + WORD $0x6e79df08 + WORD $0xad7ea89c + WORD $0x3cdc8099 + WORD $0x6e64df04 + WORD $0x1e7e2863 + WORD $0x1e7b2af7 + WORD $0x6e79df1a + WORD $0x6e7cdf19 + WORD $0x6e6adf0a + WORD $0x6e6bdf1c + WORD $0x6e6cdf0b + WORD $0x5e1807cc + WORD $0x5e1807ed + WORD $0x5e1807ae + WORD $0x5e18050f + WORD $0x1e682800 + WORD $0x1e7d2821 + WORD $0x1e7f2842 + WORD $0x3cdf809d + WORD $0x3dc0009e + WORD $0x3cc0809f + WORD $0x3cc10c68 + WORD $0x1e692ad6 + WORD $0x6e7ddf1d + WORD $0x6e7edf1e + WORD $0x6e7fdf1f + WORD $0x6e68df08 + WORD $0x5e180778 + WORD $0x5e18053b + WORD $0x1e6a2ab5 + WORD $0x5e180549 + WORD $0x1e6b2a73 + WORD $0x5e18056a + WORD $0xfd4007eb + WORD $0x1e7c2a31 + WORD $0x1e7f28e7 + WORD $0x5e1807ff + WORD $0x1e7d2a10 + WORD $0x1e6828a5 + WORD $0x5e180508 + WORD $0x1e64296b + WORD $0x5e180484 + WORD $0x5e1807bd + WORD $0x1e7e28c6 + WORD $0x5e1807de + WORD $0x5e18079c + WORD $0x1e7a2a94 + WORD $0x5e18075a + WORD $0x1e792a52 + WORD $0x5e180739 + WORD $0x1e6f2800 + WORD $0x1e6e2821 + WORD $0x1e6d2842 + WORD $0x1e6c2863 + WORD $0x1e64296e + WORD $0x1e6828a5 + WORD $0x1e7f28e7 + WORD $0x1e7e28c6 + WORD $0x1e7d2a10 + WORD $0x1e6a2a73 + WORD $0x1e7c2a31 + WORD $0xaa0303e4 + WORD $0x1e692ab5 + WORD $0x1e7b2ad6 + WORD $0x1e792a52 + WORD $0x1e7a2a94 + WORD $0x1e782af7 + WORD $0x54fff581 + WORD $0xeb0a00bf + WORD $0x927ff4b1 + WORD $0x54fff020 + WORD $0xd37df230 + WORD $0xcb1100b1 + WORD $0xaa0e03e3 + WORD $0x8b100064 + WORD $0xfc7179b8 + WORD $0xf1000631 + WORD $0x6d78e484 + WORD $0x91002063 + WORD $0x6d79ec9a + WORD $0x6d7ca488 + WORD $0x1e790b19 + WORD $0x1e640b04 + WORD $0x1e7a0b1a + WORD $0x1e7b0b1b + WORD $0x6d7af49c + WORD $0x1e680b08 + WORD $0x6d7bfc9e + WORD $0x1e792a94 + WORD $0x6d7de48a + WORD $0x1e642af7 + WORD $0x1e690b04 + WORD $0x1e7c0b1c + WORD $0x6d7fb48c + WORD $0x1e7d0b1d + WORD $0x1e6a0b09 + WORD $0x6d7eac8a + WORD $0x1e7e0b1e + WORD $0x1e7f0b1f + WORD $0x1e7a2a52 + WORD $0x1e790b19 + WORD $0x1e7b2ad6 + WORD $0x1e6c0b1b + WORD $0x1e6a0b1a + WORD $0x1e6b0b0a + WORD $0x1e7c2ab5 + WORD $0x1e6d0b18 + WORD $0x1e7d2a31 + WORD $0x1e6828c6 + WORD $0x1e7e2a73 + WORD $0x1e7f2a10 + WORD $0x1e6428e7 + WORD $0x1e6928a5 + WORD $0x1e7929ce + WORD $0x1e7b2821 + WORD $0x1e7a2863 + WORD $0x1e6a2842 + WORD $0x1e782800 + WORD $0x54fffa81 + WORD $0x17ffff50 + WORD $0x6f00e400 + WORD $0xaa1f03e9 + WORD $0x9101200a + WORD $0x9100812b + WORD $0xf81b815f + WORD $0x91004129 + WORD $0xad3e0140 + WORD $0xeb08017f + WORD $0xad3f0140 + WORD $0xad000140 + WORD $0x3d800940 + WORD $0xf900195f + WORD $0x9102014a + WORD $0x54fffecd + WORD $0xeb09010b + WORD $0x5400090d + WORD $0xf10004bf + WORD $0x5400064b + WORD $0x8b090c2a + WORD $0x8b050c4d + WORD $0x927ef0ab + WORD $0xd100204e + WORD $0x9100414c + WORD $0xd10041ad WORD $0x14000007 WORD $0xfc297800 WORD $0x91000529 WORD $0x9100218c WORD $0xeb08013f - WORD $0x91002021 - WORD $0x54000700 + WORD $0x9100214a + WORD $0x54000720 WORD $0x2f00e400 WORD $0xf10010bf WORD $0x54000062 - WORD $0xaa1f03ef + WORD $0xaa1f03f0 WORD $0x14000017 - WORD $0xaa0c03ee - WORD $0xaa0b03ef - WORD $0x927ef0b0 + WORD $0xaa0d03ef + WORD $0xaa0c03f0 + WORD $0x927ef0b1 WORD $0xad7f85e2 - WORD $0xd1001210 - WORD $0xad7f91c3 + WORD $0xd1001231 + WORD $0xad7f9203 + WORD $0x91008210 WORD $0xd10081ef - WORD $0x910081ce WORD $0x6e014021 WORD $0x6e024042 WORD $0x6e61dc61 @@ -854,38 +1814,44 @@ TEXT ·convolveFloat64NEON(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x5e180422 WORD $0x1e612800 WORD $0x1e622800 - WORD $0xb5fffe30 - WORD $0xeb0a00bf - WORD $0x927ef0af + WORD $0xb5fffe31 + WORD $0xeb0b00bf + WORD $0x927ef0b0 WORD $0x54fffc00 - WORD $0x8b0f0c2e - WORD $0xcb0f00af - WORD $0xfc4085c1 - WORD $0xfc6f79a2 - WORD $0xf10005ef + WORD $0x8b100d4f + WORD $0xcb1000b0 + WORD $0xfc4085e1 + WORD $0xfc7079c2 + WORD $0xf1000610 WORD $0x1e620821 WORD $0x1e612800 WORD $0x54ffff61 WORD $0x17ffffd7 - WORD $0xf100111f - WORD $0x54000062 - WORD $0xaa1f03e9 - WORD $0x1400000b + WORD $0xf100117f + WORD $0x540001c3 WORD $0x6f00e400 - WORD $0x927ef109 - WORD $0x9100400a - WORD $0x927ef10b - WORD $0xf100116b - WORD $0xad3f8140 - WORD $0x9100814a + WORD $0x9240050a + WORD $0x8b090c0c + WORD $0xcb0a016b + WORD $0x8b0a012d + WORD $0x8b0b0129 + WORD $0x9100418b + WORD $0xcb0801ac + WORD $0xb100118c + WORD $0xad3f8160 + WORD $0x9100816b WORD $0x54ffffa1 - WORD $0xeb09011f - WORD $0x540000c0 + WORD $0xb40000ca WORD $0x8b090c0a WORD $0xcb090108 WORD $0xf1000508 WORD $0xf800855f WORD $0x54ffffc1 + WORD $0x6d4423e9 + WORD $0x6d432beb + WORD $0x6d4233ed + WORD $0x6d413bef + WORD $0x910143ff WORD $0xd65f03c0 RET @@ -897,38 +1863,247 @@ TEXT ·correlateFloat64NEON(SB), NOSPLIT|NOFRAME, $0-72 MOVD dst_len+8(FP), R3 MOVD sig_len+32(FP), R4 MOVD ker_len+56(FP), R5 - WORD $0xb4000945 + WORD $0xb4002425 WORD $0xeb050088 - WORD $0x5400090b + WORD $0x540023eb + WORD $0xd10143ff WORD $0xeb08007f + WORD $0x6d013bef WORD $0x9a88d468 - WORD $0xf100051f - WORD $0x5400088b + WORD $0x6d0233ed + WORD $0xf100411f + WORD $0x6d032beb + WORD $0x6d0423e9 + WORD $0x5400006a + WORD $0xaa1f03e9 + WORD $0x140000c9 + WORD $0xf10000bf + WORD $0x5400172d + WORD $0xaa1f03ed + WORD $0x927ff4aa + WORD $0x9101002b + WORD $0x9101e02c + WORD $0x5280020e + WORD $0x14000010 + WORD $0x8b0d0c0d + WORD $0x9100412e + WORD $0x9102016b + WORD $0xeb0801df + WORD $0x9102018c + WORD $0x6d0051b7 + WORD $0x6d0159b2 + WORD $0x6d0245b5 + WORD $0x6d0341b3 + WORD $0x6d041da6 + WORD $0x6d0539a5 + WORD $0x6d0609a3 + WORD $0x6d0701a1 + WORD $0xaa0903ed + WORD $0x5400164c + WORD $0x2f00e400 + WORD $0x2f00e401 WORD $0xf10004bf - WORD $0x540005eb + WORD $0x2f00e402 + WORD $0x2f00e403 + WORD $0xaa0e03e9 + WORD $0x2f00e40e + WORD $0x2f00e405 + WORD $0x2f00e407 + WORD $0x2f00e406 + WORD $0x2f00e410 + WORD $0x2f00e413 + WORD $0x2f00e411 + WORD $0x2f00e415 + WORD $0x2f00e416 + WORD $0x2f00e412 + WORD $0x2f00e414 + WORD $0x2f00e417 + WORD $0x54000061 + WORD $0xaa1f03ef + WORD $0x1400005c + WORD $0xaa0b03f1 + WORD $0xaa0203ee + WORD $0x927ff4af + WORD $0xaa0b03f0 + WORD $0x3cc105d8 + WORD $0x3cdc0239 + WORD $0xad41763a + WORD $0xf10009ef + WORD $0x3cc2823c + WORD $0x6e79df1b + WORD $0x3cc38239 + WORD $0x3cdd8229 + WORD $0x3cde822b + WORD $0x3cdf022c + WORD $0x6e7adf1e + WORD $0x6e7cdf1f + WORD $0x6e7ddf1d + WORD $0x6e79df08 + WORD $0xad7eaa3c + WORD $0x3cdc8239 + WORD $0xfd0007ee + WORD $0x6e69df09 + WORD $0x3cc18224 + WORD $0x1e7b2af7 + WORD $0x6e79df1a + WORD $0x6e7cdf19 + WORD $0x6e6adf0a + WORD $0x6e6bdf1c + WORD $0x6e6cdf0b + WORD $0x5e1807cc + WORD $0x5e1807ed + WORD $0x5e1807ae + WORD $0x5e18050f + WORD $0x1e682800 + WORD $0x1e7d2821 + WORD $0x1e7f2842 + WORD $0x1e7e2863 + WORD $0x3cdf823d + WORD $0x3dc0023e + WORD $0x3cc0823f + WORD $0x3cc10e08 + WORD $0x6e7ddf1d + WORD $0x6e7edf1e + WORD $0x6e64df04 + WORD $0x6e7fdf1f + WORD $0x6e68df08 + WORD $0x5e180778 + WORD $0x1e692ad6 + WORD $0x5e18053b + WORD $0x1e6a2ab5 + WORD $0x5e180549 + WORD $0x1e6b2a73 + WORD $0x5e18056a + WORD $0xfd4007eb + WORD $0x1e7d2a10 + WORD $0x5e1807bd + WORD $0x1e7f28e7 + WORD $0x5e1807ff + WORD $0x1e6828a5 + WORD $0x5e180508 + WORD $0x1e64296b + WORD $0x5e180484 + WORD $0x1e7e28c6 + WORD $0x5e1807de + WORD $0x1e7c2a31 + WORD $0x5e18079c + WORD $0x1e7a2a94 + WORD $0x5e18075a + WORD $0x1e792a52 + WORD $0x5e180739 + WORD $0xaa1003f1 + WORD $0x1e6f2800 + WORD $0x1e6e2821 + WORD $0x1e6d2842 + WORD $0x1e6c2863 + WORD $0x1e64296e + WORD $0x1e6828a5 + WORD $0x1e7f28e7 + WORD $0x1e7e28c6 + WORD $0x1e7d2a10 + WORD $0x1e6a2a73 + WORD $0x1e7c2a31 + WORD $0x1e692ab5 + WORD $0x1e7b2ad6 + WORD $0x1e792a52 + WORD $0x1e7a2a94 + WORD $0x1e782af7 + WORD $0x54fff5a1 + WORD $0xeb0a00bf + WORD $0x927ff4af + WORD $0x54fff040 + WORD $0xd37df1ee + WORD $0xcb0f00af + WORD $0xaa0c03f0 + WORD $0xaa0203f1 + WORD $0x8b0e0203 + WORD $0xfc6e6a38 + WORD $0xf10005ef + WORD $0x6d78e464 + WORD $0x91002231 + WORD $0x6d79ec7a + WORD $0x91002210 + WORD $0x6d7ca468 + WORD $0x1e790b19 + WORD $0x1e640b04 + WORD $0x1e7a0b1a + WORD $0x1e7b0b1b + WORD $0x6d7af47c + WORD $0x1e680b08 + WORD $0x6d7bfc7e + WORD $0x1e792a94 + WORD $0x6d7de46a + WORD $0x1e642af7 + WORD $0x1e690b04 + WORD $0x1e7c0b1c + WORD $0x6d7fb46c + WORD $0x1e7d0b1d + WORD $0x1e6a0b09 + WORD $0x6d7eac6a + WORD $0x1e7e0b1e + WORD $0x1e7f0b1f + WORD $0x1e7a2a52 + WORD $0x1e790b19 + WORD $0x1e7b2ad6 + WORD $0x1e6c0b1b + WORD $0x1e6a0b1a + WORD $0x1e6b0b0a + WORD $0x1e7c2ab5 + WORD $0x1e6d0b18 + WORD $0x1e7d2a31 + WORD $0x1e6828c6 + WORD $0x1e7e2a73 + WORD $0x1e7f2a10 + WORD $0x1e6428e7 + WORD $0x1e6928a5 + WORD $0x1e7929ce + WORD $0x1e7b2821 + WORD $0x1e7a2863 + WORD $0x1e6a2842 + WORD $0x1e782800 + WORD $0x54fffa61 + WORD $0x17ffff4f + WORD $0x6f00e400 WORD $0xaa1f03e9 - WORD $0x927ef0aa - WORD $0x9100402b + WORD $0x9101200a + WORD $0x9100812b + WORD $0xf81b815f + WORD $0x91004129 + WORD $0xad3e0140 + WORD $0xeb08017f + WORD $0xad3f0140 + WORD $0xad000140 + WORD $0x3d800940 + WORD $0xf900195f + WORD $0x9102014a + WORD $0x54fffecd + WORD $0xeb09010b + WORD $0x540008ad + WORD $0xf10004bf + WORD $0x540005eb + WORD $0x8b090c2a + WORD $0x927ef0ab + WORD $0x9100414c WORD $0x14000007 WORD $0xfc297800 WORD $0x91000529 - WORD $0x9100216b + WORD $0x9100218c WORD $0xeb08013f - WORD $0x91002021 - WORD $0x54000700 + WORD $0x9100214a + WORD $0x54000720 WORD $0x2f00e400 WORD $0xf10010bf WORD $0x54000062 - WORD $0xaa1f03ec + WORD $0xaa1f03ed WORD $0x14000015 - WORD $0xaa0b03ec - WORD $0x9100404d - WORD $0x927ef0ae - WORD $0xad7f91a1 - WORD $0xf10011ce - WORD $0xad7f8d82 + WORD $0xaa0c03ed + WORD $0x9100404e + WORD $0x927ef0af + WORD $0xad7f91c1 + WORD $0xf10011ef + WORD $0xad7f8da2 + WORD $0x910081ce WORD $0x910081ad - WORD $0x9100818c WORD $0x6e61dc41 WORD $0x5e180422 WORD $0x1e612800 @@ -938,39 +2113,45 @@ TEXT ·correlateFloat64NEON(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x1e612800 WORD $0x1e622800 WORD $0x54fffe61 - WORD $0xeb0a00bf - WORD $0x927ef0ac + WORD $0xeb0b00bf + WORD $0x927ef0ad WORD $0x54fffc40 - WORD $0xd37df18e - WORD $0xcb0c00ac - WORD $0x8b0e004d - WORD $0x8b0e002e - WORD $0xfc4085c1 - WORD $0xf100058c - WORD $0xfc4085a2 + WORD $0xd37df1af + WORD $0xcb0d00ad + WORD $0x8b0f004e + WORD $0x8b0f014f + WORD $0xfc4085e1 + WORD $0xf10005ad + WORD $0xfc4085c2 WORD $0x1e620821 WORD $0x1e612800 WORD $0x54ffff61 WORD $0x17ffffd7 - WORD $0xf100111f - WORD $0x54000062 - WORD $0xaa1f03e9 - WORD $0x1400000b + WORD $0xf100117f + WORD $0x540001c3 WORD $0x6f00e400 - WORD $0x927ef109 - WORD $0x9100400a - WORD $0x927ef10b - WORD $0xf100116b - WORD $0xad3f8140 - WORD $0x9100814a + WORD $0x9240050a + WORD $0x8b090c0c + WORD $0xcb0a016b + WORD $0x8b0a012d + WORD $0x8b0b0129 + WORD $0x9100418b + WORD $0xcb0801ac + WORD $0xb100118c + WORD $0xad3f8160 + WORD $0x9100816b WORD $0x54ffffa1 - WORD $0xeb09011f - WORD $0x540000c0 + WORD $0xb40000ca WORD $0x8b090c0a WORD $0xcb090108 WORD $0xf1000508 WORD $0xf800855f WORD $0x54ffffc1 + WORD $0x6d4423e9 + WORD $0x6d432beb + WORD $0x6d4233ed + WORD $0x6d413bef + WORD $0x910143ff WORD $0xd65f03c0 RET @@ -982,21 +2163,58 @@ TEXT ·movingAverageFloat32NEON(SB), NOSPLIT|NOFRAME, $0-56 MOVD a_len+32(FP), R3 MOVD width+48(FP), R4 WORD $0xf100049f - WORD $0x540006cb + WORD $0x54000b6b WORD $0xeb040068 - WORD $0x5400068b + WORD $0x54000b2b WORD $0xeb08005f WORD $0x9a88d448 - WORD $0xf100051f - WORD $0x5400060b - WORD $0x9e230080 + WORD $0xf100411f + WORD $0x5400006a WORD $0xaa1f03e9 - WORD $0x927dec8a - WORD $0x9100402b + WORD $0x14000022 + WORD $0x9e230080 + WORD $0xaa1f03eb + WORD $0x9100802a + WORD $0x5280020c + WORD $0x4e040400 + WORD $0x6f00e401 + WORD $0x6f00e404 + WORD $0xaa0c03e9 + WORD $0x6f00e403 + WORD $0x6f00e402 + WORD $0xaa0403ec + WORD $0xaa0a03ed + WORD $0xad7f19a5 + WORD $0xf100058c + WORD $0xad4041a7 + WORD $0x910011ad + WORD $0x4e21d4a1 + WORD $0x4e24d4c4 + WORD $0x4e23d4e3 + WORD $0x4e22d602 + WORD $0x54ffff01 + WORD $0x6e20fc21 + WORD $0x8b0b080b + WORD $0x9100412c + WORD $0xeb08019f + WORD $0x9101014a + WORD $0x6e20fc84 + WORD $0x6e20fc63 + WORD $0xad001161 + WORD $0x6e20fc42 + WORD $0xad010963 + WORD $0xaa0903eb + WORD $0x54fffcad + WORD $0xeb08013f + WORD $0x5400060a + WORD $0x9e230080 + WORD $0x8b09082a + WORD $0x927dec8b + WORD $0x9100414c WORD $0x14000008 WORD $0x1e201821 - WORD $0x9100116b - WORD $0x91001021 + WORD $0x9100118c + WORD $0x9100114a WORD $0xbc297801 WORD $0x91000529 WORD $0xeb08013f @@ -1004,13 +2222,13 @@ TEXT ·movingAverageFloat32NEON(SB), NOSPLIT|NOFRAME, $0-56 WORD $0x2f00e401 WORD $0xf100209f WORD $0x54000062 - WORD $0xaa1f03ed + WORD $0xaa1f03ee WORD $0x14000018 - WORD $0xaa0b03ec - WORD $0x927dec8d - WORD $0xad7f8d82 - WORD $0xf10021ad - WORD $0x9100818c + WORD $0xaa0c03ed + WORD $0x927dec8e + WORD $0xad7f8da2 + WORD $0xf10021ce + WORD $0x910081ad WORD $0x5e0c0444 WORD $0x1e222821 WORD $0x5e140445 @@ -1026,13 +2244,13 @@ TEXT ·movingAverageFloat32NEON(SB), NOSPLIT|NOFRAME, $0-56 WORD $0x1e242821 WORD $0x1e222821 WORD $0x54fffde1 - WORD $0xeb0a009f - WORD $0x927dec8d + WORD $0xeb0b009f + WORD $0x927dec8e WORD $0x54fffbc0 - WORD $0x8b0d082c - WORD $0xcb0d008d - WORD $0xbc404582 - WORD $0xf10005ad + WORD $0x8b0e094d + WORD $0xcb0e008e + WORD $0xbc4045a2 + WORD $0xf10005ce WORD $0x1e222821 WORD $0x54ffffa1 WORD $0x17ffffd7 @@ -1153,21 +2371,74 @@ TEXT ·movingAverageFloat64NEON(SB), NOSPLIT|NOFRAME, $0-56 MOVD a_len+32(FP), R3 MOVD width+48(FP), R4 WORD $0xf100049f - WORD $0x540005cb + WORD $0x54000c6b WORD $0xeb040068 - WORD $0x5400058b + WORD $0x54000c2b WORD $0xeb08005f WORD $0x9a88d448 - WORD $0xf100051f - WORD $0x5400050b - WORD $0x9e630080 + WORD $0xf100411f + WORD $0x5400006a WORD $0xaa1f03e9 - WORD $0x927ef08a - WORD $0x9100402b + WORD $0x14000032 + WORD $0x9e630080 + WORD $0xaa1f03eb + WORD $0x9101002a + WORD $0x5280020c + WORD $0x4e080400 + WORD $0x6f00e405 + WORD $0x6f00e410 + WORD $0xaa0c03e9 + WORD $0x6f00e407 + WORD $0x6f00e406 + WORD $0xaa0a03ec + WORD $0x6f00e404 + WORD $0x6f00e403 + WORD $0xaa0403ed + WORD $0x6f00e402 + WORD $0x6f00e401 + WORD $0xad7e4991 + WORD $0xf10005ad + WORD $0xad7f5193 + WORD $0xad405995 + WORD $0x4e65d625 + WORD $0x4e70d650 + WORD $0xad414991 + WORD $0x4e67d667 + WORD $0x4e66d686 + WORD $0x4e64d6a4 + WORD $0x4e63d6c3 + WORD $0x9100218c + WORD $0x4e62d622 + WORD $0x4e61d641 + WORD $0x54fffe41 + WORD $0x6e60fca5 + WORD $0x8b0b0c0b + WORD $0x9100412c + WORD $0xeb08019f + WORD $0x9102014a + WORD $0x6e60fe10 + WORD $0x6e60fce7 + WORD $0xad004165 + WORD $0x6e60fcc6 + WORD $0x6e60fc84 + WORD $0xad011967 + WORD $0x6e60fc63 + WORD $0x6e60fc42 + WORD $0xad020d64 + WORD $0x6e60fc21 + WORD $0xad030562 + WORD $0xaa0903eb + WORD $0x54fffaad + WORD $0xeb08013f + WORD $0x5400050a + WORD $0x9e630080 + WORD $0x8b090c2a + WORD $0x927ef08b + WORD $0x9100414c WORD $0x14000008 WORD $0x1e601821 - WORD $0x9100216b - WORD $0x91002021 + WORD $0x9100218c + WORD $0x9100214a WORD $0xfc297801 WORD $0x91000529 WORD $0xeb08013f @@ -1175,13 +2446,13 @@ TEXT ·movingAverageFloat64NEON(SB), NOSPLIT|NOFRAME, $0-56 WORD $0x2f00e401 WORD $0xf100109f WORD $0x54000062 - WORD $0xaa1f03ed + WORD $0xaa1f03ee WORD $0x14000010 - WORD $0xaa0b03ec - WORD $0x927ef08d - WORD $0xad7f8d82 - WORD $0xf10011ad - WORD $0x9100818c + WORD $0xaa0c03ed + WORD $0x927ef08e + WORD $0xad7f8da2 + WORD $0xf10011ce + WORD $0x910081ad WORD $0x5e180444 WORD $0x1e622821 WORD $0x5e180462 @@ -1189,13 +2460,13 @@ TEXT ·movingAverageFloat64NEON(SB), NOSPLIT|NOFRAME, $0-56 WORD $0x1e632821 WORD $0x1e622821 WORD $0x54fffee1 - WORD $0xeb0a009f - WORD $0x927ef08d + WORD $0xeb0b009f + WORD $0x927ef08e WORD $0x54fffcc0 - WORD $0x8b0d0c2c - WORD $0xcb0d008d - WORD $0xfc408582 - WORD $0xf10005ad + WORD $0x8b0e0d4d + WORD $0xcb0e008e + WORD $0xfc4085a2 + WORD $0xf10005ce WORD $0x1e622821 WORD $0x54ffffa1 WORD $0x17ffffdf diff --git a/internal/arm64/numeric_sve2_arm64.s b/internal/arm64/numeric_sve2_arm64.s index 79940bd..ed71ac3 100644 --- a/internal/arm64/numeric_sve2_arm64.s +++ b/internal/arm64/numeric_sve2_arm64.s @@ -233,20 +233,75 @@ TEXT ·polyEvalFloat32SVE2(SB), NOSPLIT|NOFRAME, $0-72 MOVD coeffs_len+56(FP), R5 WORD $0xeb04007f WORD $0x9a84b068 + WORD $0xb40000c5 + WORD $0x8b05084a + WORD $0xf100411f + WORD $0x5400014a + WORD $0xaa1f03e9 + WORD $0x14000035 WORD $0xf100051f - WORD $0xb4000305 - WORD $0x540008ab - WORD $0x8b05084b - WORD $0xf10008bf - WORD $0xbc5fc160 - WORD $0x5400032b + WORD $0x54000e8b + WORD $0x0460e3e9 + WORD $0xeb09011f + WORD $0x54000c02 WORD $0xaa1f03e9 - WORD $0xd10004aa - WORD $0xd100216b + WORD $0x1400006a + WORD $0xd1001149 + WORD $0xf10004ab + WORD $0x4d40c920 + WORD $0x5400044d + WORD $0xaa1f03ec + WORD $0x5280020d + WORD $0x8b0c082e + WORD $0x4ea01c10 + WORD $0x4ea01c07 + WORD $0x4ea01c06 + WORD $0x4ea01c02 + WORD $0xaa0d03e9 + WORD $0xad400dc1 + WORD $0xd100214d + WORD $0xad4115c4 + WORD $0xaa0b03ee + WORD $0xbc5fc5b1 + WORD $0x6e21de10 + WORD $0x6e23dce7 + WORD $0x6e24dcc6 + WORD $0x6e25dc42 + WORD $0x4e040631 + WORD $0xd10005ce + WORD $0xf10001df + WORD $0x4e30d630 + WORD $0x4e27d627 + WORD $0x4e26d626 + WORD $0x4e22d622 + WORD $0x54fffe8c + WORD $0x8b0c080e + WORD $0x9100412d + WORD $0xaa0903ec + WORD $0xeb0801bf + WORD $0xad001dd0 + WORD $0xad0109c6 + WORD $0x54fffc6d + WORD $0x14000009 + WORD $0xaa1f03e9 + WORD $0x9100800b + WORD $0x9100812c + WORD $0x91004129 + WORD $0xad3f0160 + WORD $0xeb08019f + WORD $0xac820160 + WORD $0x54ffff6d + WORD $0xeb09010b + WORD $0x5400080d + WORD $0xbc5fc140 + WORD $0xf10008bf + WORD $0x5400024b + WORD $0xd10004ab + WORD $0xd100214a WORD $0x1e204002 WORD $0xbc697821 - WORD $0xaa0b03ec - WORD $0xaa0a03ed + WORD $0xaa0a03ec + WORD $0xaa0b03ed WORD $0x1e220822 WORD $0xbc5fc583 WORD $0xd10005ad @@ -257,44 +312,41 @@ TEXT ·polyEvalFloat32SVE2(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x91000529 WORD $0xeb08013f WORD $0x54fffe61 - WORD $0x1400002f - WORD $0x540005cb - WORD $0x0460e3e9 - WORD $0xeb09011f - WORD $0x54000102 - WORD $0xaa1f03e9 - WORD $0x14000012 - WORD $0x0460e3e9 - WORD $0xeb09011f - WORD $0x540002a2 - WORD $0xaa1f03e9 - WORD $0x1400001f - WORD $0x6f00e400 - WORD $0xcb0903e9 - WORD $0xaa0003ea - WORD $0x8a090109 - WORD $0xaa0903eb - WORD $0xe5804140 - WORD $0x0470e7eb - WORD $0xe5804540 - WORD $0x0431e3ea - WORD $0xb5ffff8b - WORD $0xeb09011f + WORD $0x1400002c + WORD $0x0460e3ed + WORD $0xeb0d017f + WORD $0x54000243 + WORD $0xd37ef52e + WORD $0xcb0d03ed + WORD $0x05242001 + WORD $0x8a0d016d + WORD $0x2598e3e0 + WORD $0xaa1f03ea + WORD $0x8b0e000c + WORD $0x0430e3ee + WORD $0x8b0d0129 + WORD $0x8b0e000e + WORD $0xe54a4181 + WORD $0xe54a41c1 + WORD $0x0470e3ea + WORD $0xeb0a01bf + WORD $0x54ffff81 + WORD $0xeb0d017f WORD $0x54000300 WORD $0x8b09080a WORD $0xcb090108 WORD $0xf1000508 - WORD $0xb800455f + WORD $0xbc004540 WORD $0x54ffffc1 WORD $0x14000012 + WORD $0x6f00e400 WORD $0xcb0903e9 - WORD $0x05242001 WORD $0xaa0003ea WORD $0x8a090109 WORD $0xaa0903eb - WORD $0xe5804141 + WORD $0xe5804140 WORD $0x0470e7eb - WORD $0xe5804541 + WORD $0xe5804540 WORD $0x0431e3ea WORD $0xb5ffff8b WORD $0xeb09011f @@ -302,7 +354,7 @@ TEXT ·polyEvalFloat32SVE2(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x8b09080a WORD $0xcb090108 WORD $0xf1000508 - WORD $0xbc004540 + WORD $0xb800455f WORD $0x54ffffc1 WORD $0xd65f03c0 RET @@ -315,89 +367,305 @@ TEXT ·convolveFloat32SVE2(SB), NOSPLIT|NOFRAME, $0-72 MOVD dst_len+8(FP), R3 MOVD sig_len+32(FP), R4 MOVD ker_len+56(FP), R5 - WORD $0xb4000a65 + WORD $0xb4002565 WORD $0xeb050088 - WORD $0x54000a2b + WORD $0x5400252b + WORD $0x6db73bef + WORD $0x6d0133ed + WORD $0x6d022beb + WORD $0x6d0323e9 + WORD $0xa9047bfd + WORD $0xa90567fa + WORD $0xa9065ff8 + WORD $0xa90757f6 + WORD $0xa9084ff4 + WORD $0xd10083ff + WORD $0x043f57ff WORD $0xeb08007f + WORD $0xa9008be1 WORD $0x9a88d468 - WORD $0xf100051f - WORD $0x540009ab - WORD $0xf10004bf - WORD $0x0460e3e9 - WORD $0x540006ab - WORD $0xd37ef4ad - WORD $0xcb0903eb + WORD $0xf100411f + WORD $0x5400006a + WORD $0xaa1f03e9 + WORD $0x140000b9 + WORD $0xf10000bf + WORD $0x540015cd + WORD $0xaa0203e9 + WORD $0x04a0e3ec WORD $0x2598e3e0 + WORD $0x0430e7e9 + WORD $0xcb0c03ec + WORD $0xaa1f03f0 + WORD $0xd100104b + WORD $0x9100f022 + WORD $0x5280020a + WORD $0xd280002d + WORD $0xd280004e + WORD $0xd280006f + WORD $0x8b050929 + WORD $0xd2800091 + WORD $0x8a0c00a3 + WORD $0xd28000a6 + WORD $0xd28000c7 + WORD $0xd28000f3 + WORD $0xd2800114 + WORD $0xd2800135 + WORD $0xd2800156 + WORD $0xd2800177 + WORD $0xd2800198 + WORD $0xd28001b9 + WORD $0xd28001da + WORD $0xd28001fd + WORD $0xaa0103fe + WORD $0xf9000fe9 + WORD $0x1400000e + WORD $0x8b10080c + WORD $0x9100412a + WORD $0x910103de + WORD $0xeb08015f + WORD $0x91010042 + WORD $0xaa0903f0 + WORD $0xbd000186 + WORD $0x3c804183 + WORD $0x3c814185 + WORD $0x3c824184 + WORD $0x2d068582 + WORD $0xbd003d80 + WORD $0x540011ac + WORD $0x04a0e3e9 + WORD $0xeb0900bf + WORD $0xaa0a03e9 + WORD $0x54000142 + WORD $0x6f00e403 + WORD $0x2f00e400 + WORD $0xaa1f03ec + WORD $0x2f00e401 + WORD $0x2f00e402 + WORD $0x2f00e406 + WORD $0x6f00e405 + WORD $0x6f00e404 + WORD $0x1400005a + WORD $0x2f00e400 + WORD $0x2f00e401 + WORD $0xf9400fe4 + WORD $0x2f00e402 + WORD $0x2f00e41d WORD $0xaa1f03ea + WORD $0x2f00e410 + WORD $0x2f00e412 + WORD $0x2f00e404 + WORD $0x2f00e411 + WORD $0x2f00e414 + WORD $0x2f00e416 + WORD $0x2f00e405 + WORD $0x2f00e413 + WORD $0x2f00e415 + WORD $0x2f00e417 + WORD $0x2f00e403 + WORD $0x2f00e406 + WORD $0x910083ec + WORD $0x85804098 + WORD $0x0430e7e4 + WORD $0xe580419d + WORD $0x8b0a0bcc + WORD $0x05b83b1d + WORD $0xa55d4189 + WORD $0xa54a43d8 + WORD $0xa54d4199 + WORD $0xa54e419a + WORD $0xa54f419b + WORD $0xa551419c + WORD $0xa546419e + WORD $0xa547419f + WORD $0x65890ba9 + WORD $0xa5534188 + WORD $0xa55a418a + WORD $0xa554418b + WORD $0xa559418c + WORD $0xa555418d + WORD $0xa558418e + WORD $0xa556418f + WORD $0xa5574187 + WORD $0x65980bb8 + WORD $0x65990bb9 + WORD $0x910083ec + WORD $0x659a0bba + WORD $0x659b0bbb + WORD $0x04b0e3ea + WORD $0x659c0bbc + WORD $0x659e0bbe + WORD $0x659f0bbf + WORD $0x65880ba8 + WORD $0x658a0baa + WORD $0x658b0bab + WORD $0xeb0a007f + WORD $0x658c0bac + WORD $0x658d0bad + WORD $0x658e0bae + WORD $0x65870ba7 + WORD $0x65982120 + WORD $0x658f0ba9 + WORD $0x8580419d + WORD $0x65982141 + WORD $0x65982182 + WORD $0x659821dd + WORD $0x659820f0 + WORD $0x65982132 + WORD $0x659821a4 + WORD $0x65982171 + WORD $0x65982114 + WORD $0x659823f6 + WORD $0x659823c5 + WORD $0x65982393 + WORD $0x65982375 + WORD $0x65982357 + WORD $0x65982323 + WORD $0x65982306 + WORD $0x54fff8c1 + WORD $0x6e0c06e3 + WORD $0x6e0c06c5 + WORD $0xeb0300bf + WORD $0x6e0c0644 + WORD $0xaa0303ec + WORD $0x6e1406a3 + WORD $0x6e140685 + WORD $0x6e140604 + WORD $0x6e1c0663 + WORD $0x6e1c0625 + WORD $0x6e1c07a4 + WORD $0x54fff1c0 + WORD $0xd37ef58a + WORD $0xcb0c00a4 + WORD $0xaa0203ec + WORD $0x8b0a0181 + WORD $0xbc647967 + WORD $0xf1000484 + WORD $0x2d7f5434 + WORD $0xbc5c4030 + WORD $0x3cdc8031 + WORD $0x3cdd8032 + WORD $0x9100118c + WORD $0x3cde8033 + WORD $0xbd400036 + WORD $0x1e3008f0 + WORD $0x4f879231 + WORD $0x4f879252 + WORD $0x1e3408f4 + WORD $0x4f879273 + WORD $0x1e3508f5 + WORD $0x1e3608e7 + WORD $0x1e3028c6 + WORD $0x4e31d463 + WORD $0x4e32d4a5 + WORD $0x1e342842 + WORD $0x4e33d484 + WORD $0x1e352821 + WORD $0x1e272800 + WORD $0x54fffd01 + WORD $0x17ffff71 + WORD $0x6f00e400 + WORD $0xaa1f03e9 + WORD $0x9100800a + WORD $0x9100812b + WORD $0x91004129 + WORD $0xad3f0140 + WORD $0xeb08017f + WORD $0xac820140 + WORD $0x54ffff6d + WORD $0xa940bfec + WORD $0xeb09010a + WORD $0x54000a2d + WORD $0xf10000bf + WORD $0x540006cd + WORD $0xd37ef4ae + WORD $0x0460e3ea + WORD $0x8b09098c + WORD $0xcb0a03eb + WORD $0x2598e3e0 + WORD $0xaa0e03ed + WORD $0x0431e7ee WORD $0x8a0b00ab - WORD $0xd100104e - WORD $0xaa0d03ec - WORD $0x0431e7ed - WORD $0x0430e7ec - WORD $0x8b0d004d - WORD $0x8b0c004c + WORD $0x0430e7ed + WORD $0x8b0e01ee + WORD $0x8b0d01ed + WORD $0xd10011ef WORD $0x14000006 - WORD $0xbc2a7800 - WORD $0x9100054a - WORD $0x91001021 - WORD $0xeb08015f - WORD $0x54000720 - WORD $0xeb0900bf + WORD $0xbc297800 + WORD $0x91000529 + WORD $0x9100118c + WORD $0xeb08013f + WORD $0x540007a0 + WORD $0xeb0a00bf WORD $0x54000082 WORD $0x2f00e400 - WORD $0xaa1f03f0 + WORD $0xaa1f03f1 WORD $0x14000015 WORD $0x2f00e400 - WORD $0xaa1f03ef - WORD $0xaa0103f0 - WORD $0xa54f4181 - WORD $0x85804202 + WORD $0xaa1f03f0 + WORD $0xaa0c03f1 + WORD $0xa55041a1 + WORD $0x85804222 WORD $0x05b83821 WORD $0x65810841 - WORD $0x85804602 - WORD $0x0431e3f0 + WORD $0x85804622 + WORD $0x0431e3f1 WORD $0x65982020 - WORD $0xa54f41a1 - WORD $0x0470e7ef + WORD $0xa55041c1 + WORD $0x0470e7f0 WORD $0x05b83821 - WORD $0xab0f017f + WORD $0xab10017f WORD $0x65810841 WORD $0x65982020 WORD $0x54fffe61 WORD $0xeb0b00bf - WORD $0xaa0b03f0 + WORD $0xaa0b03f1 WORD $0x54fffc60 - WORD $0xd37ef60f - WORD $0xcb1000b0 - WORD $0xbc6f6821 - WORD $0xbc7079c2 - WORD $0xf1000610 - WORD $0x910011ef + WORD $0xd37ef630 + WORD $0xcb1100b1 + WORD $0xbc706981 + WORD $0xbc7179e2 + WORD $0xf1000631 + WORD $0x91001210 WORD $0x1e220821 WORD $0x1e212800 WORD $0x54ffff41 WORD $0x17ffffd9 - WORD $0xeb09011f - WORD $0x54000062 - WORD $0xaa1f03e9 - WORD $0x1400000d + WORD $0x0460e3ed + WORD $0xeb0d015f + WORD $0x54000243 + WORD $0xd37ef52e WORD $0x6f00e400 - WORD $0xcb0903e9 - WORD $0xaa0003ea - WORD $0x8a090109 - WORD $0xaa0903eb - WORD $0xe5804140 - WORD $0x0470e7eb - WORD $0xe5804540 - WORD $0x0431e3ea - WORD $0xb5ffff8b - WORD $0xeb09011f + WORD $0xcb0d03ed + WORD $0x8a0d014d + WORD $0x2598e3e0 + WORD $0xaa1f03eb + WORD $0x8b0e000c + WORD $0x0430e3ee + WORD $0x8b0d0129 + WORD $0x8b0e000e + WORD $0xe54b4180 + WORD $0xe54b41c0 + WORD $0x0470e3eb + WORD $0xeb0b01bf + WORD $0x54ffff81 + WORD $0xeb0d015f WORD $0x540000c0 WORD $0x8b09080a WORD $0xcb090108 WORD $0xf1000508 WORD $0xb800455f WORD $0x54ffffc1 + WORD $0x043f503f + WORD $0x910083ff + WORD $0xa9484ff4 + WORD $0xa94757f6 + WORD $0xa9465ff8 + WORD $0xa94567fa + WORD $0xa9447bfd + WORD $0x6d4323e9 + WORD $0x6d422beb + WORD $0x6d4133ed + WORD $0x6cc93bef WORD $0xd65f03c0 RET @@ -409,78 +677,287 @@ TEXT ·correlateFloat32SVE2(SB), NOSPLIT|NOFRAME, $0-72 MOVD dst_len+8(FP), R3 MOVD sig_len+32(FP), R4 MOVD ker_len+56(FP), R5 - WORD $0xb4000905 + WORD $0xb4002325 WORD $0xeb050088 - WORD $0x540008cb + WORD $0x540022eb + WORD $0x6db73bef + WORD $0x6d0133ed + WORD $0x6d022beb + WORD $0x6d0323e9 + WORD $0xa9047bfd + WORD $0xa90567fa + WORD $0xa9065ff8 + WORD $0xa90757f6 + WORD $0xa9084ff4 + WORD $0xd10043ff + WORD $0x043f57ff WORD $0xeb08007f + WORD $0xaa0203ea + WORD $0xf90007e1 WORD $0x9a88d468 - WORD $0xf100051f - WORD $0x5400084b - WORD $0xf10004bf - WORD $0x0460e3e9 - WORD $0x5400054b - WORD $0xcb0903eb + WORD $0xf100411f + WORD $0x5400006a + WORD $0xaa1f03e9 + WORD $0x140000b0 + WORD $0xf10000bf + WORD $0x540014ad + WORD $0x04a0e3e9 WORD $0x2598e3e0 - WORD $0xaa1f03ea - WORD $0x8a0b00ab + WORD $0xaa1f03e4 + WORD $0xcb0903e9 + WORD $0x52800202 + WORD $0xd280002b + WORD $0xd280004c + WORD $0xd280006d + WORD $0x8a0900ae + WORD $0xd280008f + WORD $0xd28000b0 + WORD $0xd28000d1 + WORD $0xd28000e3 + WORD $0xd2800106 + WORD $0xd2800127 + WORD $0xd2800153 + WORD $0xd2800174 + WORD $0xd2800195 + WORD $0xd28001b6 + WORD $0xd28001d7 + WORD $0xd28001f8 + WORD $0xaa0103f9 + WORD $0x1400000d + WORD $0x8b040801 + WORD $0x91004122 + WORD $0x91010339 + WORD $0xeb08005f + WORD $0xaa0903e4 + WORD $0xbd000026 + WORD $0x3c804023 + WORD $0x3c814025 + WORD $0x3c824024 + WORD $0x2d068422 + WORD $0xbd003c20 + WORD $0x5400116c + WORD $0x04a0e3e9 + WORD $0xeb0900bf + WORD $0xaa0203e9 + WORD $0x54000142 + WORD $0x6f00e403 + WORD $0x2f00e400 + WORD $0xaa1f03e2 + WORD $0x2f00e401 + WORD $0x2f00e402 + WORD $0x2f00e406 + WORD $0x6f00e405 + WORD $0x6f00e404 + WORD $0x14000056 + WORD $0x2f00e400 + WORD $0x2f00e401 + WORD $0xaa1f03fa + WORD $0x2f00e402 + WORD $0x2f00e41d + WORD $0x2f00e410 + WORD $0x2f00e412 + WORD $0x2f00e404 + WORD $0x2f00e411 + WORD $0x2f00e414 + WORD $0x2f00e416 + WORD $0x2f00e405 + WORD $0x2f00e413 + WORD $0x2f00e415 + WORD $0x2f00e417 + WORD $0x2f00e403 + WORD $0x2f00e406 + WORD $0x8b1a0b22 + WORD $0x910043fd + WORD $0xa55a4338 + WORD $0xe58043bd + WORD $0xa55a415d + WORD $0x04b0e3fa + WORD $0xa5584049 + WORD $0xa54b4059 + WORD $0xa54c405a + WORD $0xa54d405b + WORD $0xa54f405c + WORD $0xa550405e + WORD $0xa551405f + WORD $0x65980bb8 + WORD $0xeb1a01df + WORD $0x65890ba9 + WORD $0xa5434048 + WORD $0xa557404a + WORD $0xa546404b + WORD $0xa556404c + WORD $0x65990bb9 + WORD $0xa547404d + WORD $0xa555404e + WORD $0x659a0bba + WORD $0xa553404f + WORD $0xa5544047 + WORD $0x659b0bbb + WORD $0x659c0bbc + WORD $0x659e0bbe + WORD $0x659f0bbf + WORD $0x65880ba8 + WORD $0x658a0baa + WORD $0x658b0bab + WORD $0x658c0bac + WORD $0x658d0bad + WORD $0x658e0bae + WORD $0x65870ba7 + WORD $0x65982120 + WORD $0x658f0ba9 + WORD $0x858043bd + WORD $0x65982141 + WORD $0x65982182 + WORD $0x659821dd + WORD $0x659820f0 + WORD $0x65982132 + WORD $0x659821a4 + WORD $0x65982171 + WORD $0x65982114 + WORD $0x659823f6 + WORD $0x659823c5 + WORD $0x65982393 + WORD $0x65982375 + WORD $0x65982357 + WORD $0x65982323 + WORD $0x65982306 + WORD $0x54fff921 + WORD $0x6e0c06e3 + WORD $0x6e0c06c5 + WORD $0xeb0e00bf + WORD $0x6e0c0644 + WORD $0xaa0e03e2 + WORD $0x6e1406a3 + WORD $0x6e140685 + WORD $0x6e140604 + WORD $0x6e1c0663 + WORD $0x6e1c0625 + WORD $0x6e1c07a4 + WORD $0x54fff260 + WORD $0xd37ef45a + WORD $0xcb0200bd + WORD $0xaa1903fe + WORD $0xaa0a03e2 + WORD $0x8b1a03c1 + WORD $0xbc7a6847 + WORD $0xf10007bd + WORD $0x2d46d434 + WORD $0xbd400030 + WORD $0x3cc04031 + WORD $0x3cc14032 + WORD $0x91001042 + WORD $0x3cc24033 + WORD $0xbd403c36 + WORD $0x1e3008f0 + WORD $0x4f879231 + WORD $0x4f879252 + WORD $0x1e3408f4 + WORD $0x4f879273 + WORD $0x1e3508f5 + WORD $0x1e3608e7 + WORD $0x910013de + WORD $0x1e3028c6 + WORD $0x4e31d463 + WORD $0x4e32d4a5 + WORD $0x1e342842 + WORD $0x4e33d484 + WORD $0x1e352821 + WORD $0x1e272800 + WORD $0x54fffce1 + WORD $0x17ffff74 + WORD $0x6f00e400 + WORD $0xaa1f03e9 + WORD $0x9100800b + WORD $0x9100812c + WORD $0x91004129 + WORD $0xad3f0160 + WORD $0xeb08019f + WORD $0xac820160 + WORD $0x54ffff6d + WORD $0xf94007ee + WORD $0xeb09010b + WORD $0x540008ed + WORD $0x0460e3ed + WORD $0xf10000bf + WORD $0x5400058d + WORD $0xcb0d03ec + WORD $0x8b0909cb + WORD $0x2598e3e0 + WORD $0x8a0c00ac WORD $0x14000006 - WORD $0xbc2a7800 - WORD $0x9100054a - WORD $0x91001021 - WORD $0xeb08015f - WORD $0x540006a0 + WORD $0xbc297800 + WORD $0x91000529 + WORD $0x9100116b + WORD $0xeb08013f + WORD $0x54000740 + WORD $0xeb0d00bf + WORD $0x54000082 WORD $0x2f00e400 - WORD $0xeb0900bf - WORD $0x54000062 - WORD $0xaa1f03ec - WORD $0x14000013 - WORD $0xaa0103ec - WORD $0xaa0203ed - WORD $0xaa0b03ee - WORD $0x85804181 - WORD $0x858041a2 - WORD $0x0470e7ee + WORD $0xaa1f03ee + WORD $0x14000015 + WORD $0xaa0b03ef + WORD $0x2f00e400 + WORD $0xaa1f03ee + WORD $0x0430e3ef + WORD $0xaa0a03f0 + WORD $0xa54e4161 + WORD $0x85804202 WORD $0x65820821 - WORD $0x858045a2 - WORD $0x0431e3ed + WORD $0x85804602 + WORD $0x0431e3f0 WORD $0x65982020 - WORD $0x85804581 - WORD $0x0431e3ec + WORD $0xa54e41e1 + WORD $0x0470e3ee WORD $0x65820821 + WORD $0xeb0e019f WORD $0x65982020 - WORD $0xb5fffeae - WORD $0xeb0b00bf - WORD $0xaa0b03ec - WORD $0x54fffca0 - WORD $0xbc6c7821 - WORD $0xbc6c7842 - WORD $0x9100058c + WORD $0x54fffea1 WORD $0xeb0c00bf + WORD $0xaa0c03ee + WORD $0x54fffc60 + WORD $0xbc6e7961 + WORD $0xbc6e7942 + WORD $0x910005ce + WORD $0xeb0e00bf WORD $0x1e220821 WORD $0x1e212800 WORD $0x54ffff41 - WORD $0x17ffffdd - WORD $0xeb09011f - WORD $0x54000062 - WORD $0xaa1f03e9 - WORD $0x1400000d + WORD $0x17ffffdb + WORD $0xeb0d017f + WORD $0x54000243 + WORD $0xd37ef52e WORD $0x6f00e400 - WORD $0xcb0903e9 - WORD $0xaa0003ea - WORD $0x8a090109 - WORD $0xaa0903eb - WORD $0xe5804140 - WORD $0x0470e7eb - WORD $0xe5804540 - WORD $0x0431e3ea - WORD $0xb5ffff8b - WORD $0xeb09011f + WORD $0xcb0d03ed + WORD $0x8a0d016d + WORD $0x2598e3e0 + WORD $0xaa1f03ea + WORD $0x8b0e000c + WORD $0x0430e3ee + WORD $0x8b0d0129 + WORD $0x8b0e000e + WORD $0xe54a4180 + WORD $0xe54a41c0 + WORD $0x0470e3ea + WORD $0xeb0a01bf + WORD $0x54ffff81 + WORD $0xeb0d017f WORD $0x540000c0 WORD $0x8b09080a WORD $0xcb090108 WORD $0xf1000508 WORD $0xb800455f WORD $0x54ffffc1 + WORD $0x043f503f + WORD $0x910043ff + WORD $0xa9484ff4 + WORD $0xa94757f6 + WORD $0xa9465ff8 + WORD $0xa94567fa + WORD $0xa9447bfd + WORD $0x6d4323e9 + WORD $0x6d422beb + WORD $0x6d4133ed + WORD $0x6cc93bef WORD $0xd65f03c0 RET @@ -725,20 +1202,98 @@ TEXT ·polyEvalFloat64SVE2(SB), NOSPLIT|NOFRAME, $0-72 MOVD coeffs_len+56(FP), R5 WORD $0xeb04007f WORD $0x9a84b068 + WORD $0xb40000c5 + WORD $0x8b050c4a + WORD $0xf100411f + WORD $0x5400014a + WORD $0xaa1f03e9 + WORD $0x1400004c WORD $0xf100051f - WORD $0xb4000305 - WORD $0x540008ab - WORD $0x8b050c4b - WORD $0xf10008bf - WORD $0xfc5f8160 - WORD $0x5400032b + WORD $0x5400116b + WORD $0x04a0e3e9 + WORD $0xeb09011f + WORD $0x54000ee2 WORD $0xaa1f03e9 - WORD $0xd10004aa - WORD $0xd100416b + WORD $0x14000081 + WORD $0xfc5f8140 + WORD $0xf10004ab + WORD $0x5400066d + WORD $0x4e080400 + WORD $0xaa1f03ec + WORD $0x5280020d + WORD $0x8b0c0c2e + WORD $0x4ea01c18 + WORD $0x4ea01c17 + WORD $0x4ea01c16 + WORD $0x4ea01c15 + WORD $0xaa0d03e9 + WORD $0xad4009c1 + WORD $0x4ea01c14 + WORD $0xad4111c3 + WORD $0x4ea01c13 + WORD $0xad4219c5 + WORD $0x4ea01c12 + WORD $0xad4341c7 + WORD $0x4ea01c11 + WORD $0xd100414d + WORD $0xaa0b03ee + WORD $0xfc5f85b9 + WORD $0x6e61df18 + WORD $0x6e62def7 + WORD $0x6e63ded6 + WORD $0x6e64deb5 + WORD $0x6e65de94 + WORD $0x6e66de73 + WORD $0xd10005ce + WORD $0x6e67de52 + WORD $0x6e70de31 + WORD $0xf10001df + WORD $0x4e080739 + WORD $0x4e78d738 + WORD $0x4e77d737 + WORD $0x4e76d736 + WORD $0x4e75d735 + WORD $0x4e74d734 + WORD $0x4e73d733 + WORD $0x4e72d732 + WORD $0x4e71d731 + WORD $0x54fffd8c + WORD $0x8b0c0c0e + WORD $0x9100412d + WORD $0xaa0903ec + WORD $0xeb0801bf + WORD $0xad005dd8 + WORD $0xad0155d6 + WORD $0xad024dd4 + WORD $0xad0345d2 + WORD $0x54fffa6d + WORD $0x14000010 + WORD $0xaa1f03e9 + WORD $0x9101000b + WORD $0x9100812c + WORD $0x6d3c0160 + WORD $0x91004129 + WORD $0x6d3d0160 + WORD $0xeb08019f + WORD $0x6d3e0160 + WORD $0x6d3f0160 + WORD $0x6d000160 + WORD $0x6d010160 + WORD $0x6d020160 + WORD $0x6d030160 + WORD $0x9102016b + WORD $0x54fffe8d + WORD $0xeb09010b + WORD $0x5400080d + WORD $0xfc5f8140 + WORD $0xf10008bf + WORD $0x5400024b + WORD $0xd10004ab + WORD $0xd100414a WORD $0x1e604002 WORD $0xfc697821 - WORD $0xaa0b03ec - WORD $0xaa0a03ed + WORD $0xaa0a03ec + WORD $0xaa0b03ed WORD $0x1e620822 WORD $0xfc5f8583 WORD $0xd10005ad @@ -749,44 +1304,41 @@ TEXT ·polyEvalFloat64SVE2(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x91000529 WORD $0xeb08013f WORD $0x54fffe61 - WORD $0x1400002f - WORD $0x540005cb - WORD $0x04a0e3e9 - WORD $0xeb09011f - WORD $0x54000102 - WORD $0xaa1f03e9 - WORD $0x14000012 - WORD $0x04a0e3e9 - WORD $0xeb09011f - WORD $0x540002a2 - WORD $0xaa1f03e9 - WORD $0x1400001f - WORD $0x6f00e400 - WORD $0xcb0903e9 - WORD $0xaa0003ea - WORD $0x8a090109 - WORD $0xaa0903eb - WORD $0xe5804140 - WORD $0x04b0e7eb - WORD $0xe5804540 - WORD $0x0431e3ea - WORD $0xb5ffff8b - WORD $0xeb09011f + WORD $0x1400002c + WORD $0x04a0e3ed + WORD $0xeb0d017f + WORD $0x54000243 + WORD $0xd37df12e + WORD $0xcb0d03ed + WORD $0x05282001 + WORD $0x8a0d016d + WORD $0x25d8e3e0 + WORD $0xaa1f03ea + WORD $0x8b0e000c + WORD $0x0430e3ee + WORD $0x8b0d0129 + WORD $0x8b0e000e + WORD $0xe5ea4181 + WORD $0xe5ea41c1 + WORD $0x04b0e3ea + WORD $0xeb0a01bf + WORD $0x54ffff81 + WORD $0xeb0d017f WORD $0x54000300 WORD $0x8b090c0a WORD $0xcb090108 WORD $0xf1000508 - WORD $0xf800855f + WORD $0xfc008540 WORD $0x54ffffc1 WORD $0x14000012 + WORD $0x6f00e400 WORD $0xcb0903e9 - WORD $0x05282001 WORD $0xaa0003ea WORD $0x8a090109 WORD $0xaa0903eb - WORD $0xe5804141 + WORD $0xe5804140 WORD $0x04b0e7eb - WORD $0xe5804541 + WORD $0xe5804540 WORD $0x0431e3ea WORD $0xb5ffff8b WORD $0xeb09011f @@ -794,7 +1346,7 @@ TEXT ·polyEvalFloat64SVE2(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x8b090c0a WORD $0xcb090108 WORD $0xf1000508 - WORD $0xfc008540 + WORD $0xf800855f WORD $0x54ffffc1 WORD $0xd65f03c0 RET @@ -807,89 +1359,326 @@ TEXT ·convolveFloat64SVE2(SB), NOSPLIT|NOFRAME, $0-72 MOVD dst_len+8(FP), R3 MOVD sig_len+32(FP), R4 MOVD ker_len+56(FP), R5 - WORD $0xb4000a65 + WORD $0xb4002805 WORD $0xeb050088 - WORD $0x54000a2b + WORD $0x540027cb + WORD $0xd10283ff WORD $0xeb08007f + WORD $0xfd000bee WORD $0x9a88d468 - WORD $0xf100051f - WORD $0x540009ab - WORD $0xf10004bf - WORD $0x04a0e3e9 - WORD $0x540006ab - WORD $0xd37df0ad - WORD $0xcb0903eb + WORD $0x6d0233ed + WORD $0xf100411f + WORD $0x6d032beb + WORD $0x6d0423e9 + WORD $0xa9057bfd + WORD $0xa90667fa + WORD $0xa9075ff8 + WORD $0xa90857f6 + WORD $0xa9094ff4 + WORD $0xa9000be1 + WORD $0x5400006a + WORD $0xaa1f03e9 + WORD $0x140000d0 + WORD $0xf10000bf + WORD $0x5400180d + WORD $0xaa0203e9 + WORD $0x04e0e3ec WORD $0x25d8e3e0 + WORD $0x0430e7e9 + WORD $0xcb0c03ec + WORD $0xaa1f03f5 + WORD $0xd100204b + WORD $0x9101e022 + WORD $0x5280020a + WORD $0xd280002d + WORD $0xd280004e + WORD $0xd280006f + WORD $0x8b050d29 + WORD $0xd2800090 + WORD $0x8a0c00b1 + WORD $0xd28000a4 + WORD $0xd28000c6 + WORD $0xd28000e7 + WORD $0xd2800113 + WORD $0xd2800134 + WORD $0xd2800156 + WORD $0xd2800177 + WORD $0xd2800198 + WORD $0xd28001b9 + WORD $0xd28001da + WORD $0xd28001fd + WORD $0xaa0103fe + WORD $0xf9000fe9 + WORD $0x14000010 + WORD $0x8b150c0c + WORD $0x9100412a + WORD $0x910203de + WORD $0xeb08015f + WORD $0x91020042 + WORD $0xaa0903f5 + WORD $0x6d005997 + WORD $0x6d015195 + WORD $0x6d024993 + WORD $0x6d034191 + WORD $0x6d041987 + WORD $0x6d051185 + WORD $0x6d060983 + WORD $0x6d070181 + WORD $0x5400144c + WORD $0x04e0e3e9 + WORD $0xeb0900bf + WORD $0xaa0a03e9 + WORD $0x54000262 + WORD $0x2f00e400 + WORD $0x2f00e401 + WORD $0xaa1f03ec + WORD $0x2f00e402 + WORD $0x2f00e403 + WORD $0x2f00e404 + WORD $0x2f00e405 + WORD $0x2f00e406 + WORD $0x2f00e407 + WORD $0x2f00e410 + WORD $0x2f00e411 + WORD $0x2f00e412 + WORD $0x2f00e413 + WORD $0x2f00e414 + WORD $0x2f00e415 + WORD $0x2f00e416 + WORD $0x2f00e417 + WORD $0x1400004d + WORD $0x2f00e400 + WORD $0x2f00e401 + WORD $0xf9400fe3 + WORD $0x2f00e402 + WORD $0x2f00e403 WORD $0xaa1f03ea + WORD $0x2f00e404 + WORD $0x2f00e405 + WORD $0x2f00e406 + WORD $0x2f00e407 + WORD $0x2f00e410 + WORD $0x2f00e411 + WORD $0x2f00e412 + WORD $0x2f00e413 + WORD $0x2f00e414 + WORD $0x2f00e415 + WORD $0x2f00e416 + WORD $0x2f00e417 + WORD $0x85804078 + WORD $0x8b0a0fcc + WORD $0x0430e7e3 + WORD $0xa5fa419a + WORD $0xa5fd4199 + WORD $0xa5ed419b + WORD $0x05f83b18 + WORD $0xa5f8419f + WORD $0xa5f94188 + WORD $0xa5ee419c + WORD $0xa5ef419d + WORD $0xa5f0419e + WORD $0xa5e44189 + WORD $0xa5e6418a + WORD $0xa5f7418b + WORD $0x65da0b1a + WORD $0x65d90b19 + WORD $0xa5f6418c + WORD $0x65c80b08 + WORD $0xa5f4418d + WORD $0xa5e7418e + WORD $0x65dc0b1c + WORD $0x65cb0b0b + WORD $0x65dd0b1d + WORD $0x65cc0b0c + WORD $0x65d82341 + WORD $0x65db0b1a + WORD $0x65df0b1b + WORD $0x65d82320 + WORD $0xa5ea43d9 + WORD $0xa5f3419f + WORD $0x65d82102 + WORD $0x65de0b1e + WORD $0x65cd0b08 + WORD $0x65d82164 + WORD $0x04f0e3ea + WORD $0x65d90b19 + WORD $0x65df0b1f + WORD $0x65d82363 + WORD $0x65c90b1b + WORD $0x65ca0b09 + WORD $0x65ce0b18 + WORD $0xeb0a023f + WORD $0x65d82185 + WORD $0x65d82106 + WORD $0x65d823e7 + WORD $0x65d82310 + WORD $0x65d82131 + WORD $0x65d82372 + WORD $0x65d823d3 + WORD $0x65d823b4 + WORD $0x65d82395 + WORD $0x65d82356 + WORD $0x65d82337 + WORD $0x54fff941 + WORD $0xeb1100bf + WORD $0xaa1103ec + WORD $0x54fff200 + WORD $0xd37df18a + WORD $0xcb0c00a3 + WORD $0xaa0203ec + WORD $0x8b0a0181 + WORD $0xfc637978 + WORD $0xf1000463 + WORD $0x6d78e839 + WORD $0x9100218c + WORD $0x6d79f03b + WORD $0x6d7ca829 + WORD $0x1e7a0b1a + WORD $0x1e790b19 + WORD $0x1e7b0b1b + WORD $0x1e7c0b1c + WORD $0x6d7af83d + WORD $0x1e690b09 + WORD $0x6d7ba03f + WORD $0x1e7a2ad6 + WORD $0x6d7de82b + WORD $0x1e792af7 + WORD $0x1e6a0b19 + WORD $0x1e7d0b1d + WORD $0x6d7fb82d + WORD $0x1e7e0b1e + WORD $0x1e6b0b0a + WORD $0x6d7eb02b + WORD $0x1e7f0b1f + WORD $0x1e680b08 + WORD $0x1e7b2ab5 + WORD $0x1e7a0b1a + WORD $0x1e7c2a94 + WORD $0x1e6d0b1c + WORD $0x1e6b0b1b + WORD $0x1e6c0b0b + WORD $0x1e7d2a73 + WORD $0x1e6e0b18 + WORD $0x1e7e2a52 + WORD $0x1e6928e7 + WORD $0x1e7f2a31 + WORD $0x1e682a10 + WORD $0x1e7928c6 + WORD $0x1e6a28a5 + WORD $0x1e7a2884 + WORD $0x1e7c2821 + WORD $0x1e7b2863 + WORD $0x1e6b2842 + WORD $0x1e782800 + WORD $0x54fffa81 + WORD $0x17ffff5f + WORD $0x6f00e400 + WORD $0xaa1f03e9 + WORD $0x9101200a + WORD $0x9100812b + WORD $0xf81b815f + WORD $0x91004129 + WORD $0xad3e0140 + WORD $0xeb08017f + WORD $0xad3f0140 + WORD $0xad000140 + WORD $0x3d800940 + WORD $0xf900195f + WORD $0x9102014a + WORD $0x54fffecd + WORD $0xa9403fec + WORD $0xeb09010a + WORD $0x54000a2d + WORD $0xf10000bf + WORD $0x540006cd + WORD $0xd37df0ae + WORD $0x04a0e3ea + WORD $0x8b090d8c + WORD $0xcb0a03eb + WORD $0x25d8e3e0 + WORD $0xaa0e03ed + WORD $0x0431e7ee WORD $0x8a0b00ab - WORD $0xd100204e - WORD $0xaa0d03ec - WORD $0x0431e7ed - WORD $0x0430e7ec - WORD $0x8b0d004d - WORD $0x8b0c004c + WORD $0x0430e7ed + WORD $0x8b0e01ee + WORD $0x8b0d01ed + WORD $0xd10021ef WORD $0x14000006 - WORD $0xfc2a7800 - WORD $0x9100054a - WORD $0x91002021 - WORD $0xeb08015f - WORD $0x54000720 - WORD $0xeb0900bf + WORD $0xfc297800 + WORD $0x91000529 + WORD $0x9100218c + WORD $0xeb08013f + WORD $0x540007a0 + WORD $0xeb0a00bf WORD $0x54000082 WORD $0x2f00e400 - WORD $0xaa1f03f0 + WORD $0xaa1f03f1 WORD $0x14000015 WORD $0x2f00e400 - WORD $0xaa1f03ef - WORD $0xaa0103f0 - WORD $0xa5ef4181 - WORD $0x85804202 + WORD $0xaa1f03f0 + WORD $0xaa0c03f1 + WORD $0xa5f041a1 + WORD $0x85804222 WORD $0x05f83821 WORD $0x65c10841 - WORD $0xa5ef41a2 - WORD $0x04b0e7ef + WORD $0xa5f041c2 + WORD $0x04b0e7f0 WORD $0x05f83842 - WORD $0xab0f017f + WORD $0xab10017f WORD $0x65d82020 - WORD $0x85804601 - WORD $0x0431e3f0 + WORD $0x85804621 + WORD $0x0431e3f1 WORD $0x65c20821 WORD $0x65d82020 WORD $0x54fffe61 WORD $0xeb0b00bf - WORD $0xaa0b03f0 + WORD $0xaa0b03f1 WORD $0x54fffc60 - WORD $0xd37df20f - WORD $0xcb1000b0 - WORD $0xfc6f6821 - WORD $0xfc7079c2 - WORD $0xf1000610 - WORD $0x910021ef + WORD $0xd37df230 + WORD $0xcb1100b1 + WORD $0xfc706981 + WORD $0xfc7179e2 + WORD $0xf1000631 + WORD $0x91002210 WORD $0x1e620821 WORD $0x1e612800 WORD $0x54ffff41 WORD $0x17ffffd9 - WORD $0xeb09011f - WORD $0x54000062 - WORD $0xaa1f03e9 - WORD $0x1400000d + WORD $0x04a0e3ed + WORD $0xeb0d015f + WORD $0x54000243 + WORD $0xd37df12e WORD $0x6f00e400 - WORD $0xcb0903e9 - WORD $0xaa0003ea - WORD $0x8a090109 - WORD $0xaa0903eb - WORD $0xe5804140 - WORD $0x04b0e7eb - WORD $0xe5804540 - WORD $0x0431e3ea - WORD $0xb5ffff8b - WORD $0xeb09011f + WORD $0xcb0d03ed + WORD $0x8a0d014d + WORD $0x25d8e3e0 + WORD $0xaa1f03eb + WORD $0x8b0e000c + WORD $0x0430e3ee + WORD $0x8b0d0129 + WORD $0x8b0e000e + WORD $0xe5eb4180 + WORD $0xe5eb41c0 + WORD $0x04b0e3eb + WORD $0xeb0b01bf + WORD $0x54ffff81 + WORD $0xeb0d015f WORD $0x540000c0 WORD $0x8b090c0a WORD $0xcb090108 WORD $0xf1000508 WORD $0xf800855f WORD $0x54ffffc1 + WORD $0xa9494ff4 + WORD $0xfd400bee + WORD $0xa94857f6 + WORD $0xa9475ff8 + WORD $0xa94667fa + WORD $0xa9457bfd + WORD $0x6d4423e9 + WORD $0x6d432beb + WORD $0x6d4233ed + WORD $0x910283ff WORD $0xd65f03c0 RET @@ -901,78 +1690,299 @@ TEXT ·correlateFloat64SVE2(SB), NOSPLIT|NOFRAME, $0-72 MOVD dst_len+8(FP), R3 MOVD sig_len+32(FP), R4 MOVD ker_len+56(FP), R5 - WORD $0xb4000905 - WORD $0xeb050088 - WORD $0x540008cb - WORD $0xeb08007f - WORD $0x9a88d468 - WORD $0xf100051f - WORD $0x5400084b - WORD $0xf10004bf - WORD $0x04a0e3e9 - WORD $0x5400054b - WORD $0xcb0903eb + WORD $0xb40024a5 + WORD $0xeb050089 + WORD $0x5400246b + WORD $0xd10283ff + WORD $0xeb09007f + WORD $0xaa0203e8 + WORD $0xfd000bee + WORD $0x9a89d462 + WORD $0x6d0233ed + WORD $0xf100405f + WORD $0x6d032beb + WORD $0x6d0423e9 + WORD $0xa9057bfd + WORD $0xa90667fa + WORD $0xa9075ff8 + WORD $0xa90857f6 + WORD $0xa9094ff4 + WORD $0xf90003e1 + WORD $0x5400006a + WORD $0xaa1f03e9 + WORD $0x140000bd + WORD $0xf10000bf + WORD $0x540015ad + WORD $0x04e0e3e9 WORD $0x25d8e3e0 - WORD $0xaa1f03ea - WORD $0x8a0b00ab - WORD $0x14000006 - WORD $0xfc2a7800 - WORD $0x9100054a - WORD $0x91002021 - WORD $0xeb08015f - WORD $0x540006a0 + WORD $0xaa1f03fa + WORD $0xcb0903e9 + WORD $0x9101e02a + WORD $0x5280020b + WORD $0xd280002c + WORD $0xd280004d + WORD $0x8a0900ae + WORD $0xd280006f + WORD $0xd2800090 + WORD $0xd28000b1 + WORD $0xd28000c3 + WORD $0xd28000e4 + WORD $0xd2800106 + WORD $0xd2800127 + WORD $0xd2800153 + WORD $0xd2800174 + WORD $0xd2800195 + WORD $0xd28001b6 + WORD $0xd28001d7 + WORD $0xd28001f8 + WORD $0xaa0103f9 + WORD $0xf9000fe0 + WORD $0xf90007e2 + WORD $0x14000011 + WORD $0xf94007e2 + WORD $0x8b1a0c01 + WORD $0x9100412b + WORD $0x91020339 + WORD $0x9102014a + WORD $0xaa0903fa + WORD $0xeb02017f + WORD $0x6d005837 + WORD $0x6d015035 + WORD $0x6d024833 + WORD $0x6d034031 + WORD $0x6d041827 + WORD $0x6d051025 + WORD $0x6d060823 + WORD $0x6d070021 + WORD $0x5400122c WORD $0x2f00e400 + WORD $0x2f00e401 + WORD $0x04e0e3e9 + WORD $0x2f00e402 + WORD $0x2f00e403 WORD $0xeb0900bf + WORD $0x2f00e404 + WORD $0x2f00e405 + WORD $0xaa0b03e9 + WORD $0x2f00e406 + WORD $0x2f00e407 + WORD $0x2f00e410 + WORD $0x2f00e411 + WORD $0x2f00e412 + WORD $0x2f00e413 + WORD $0x2f00e414 + WORD $0x2f00e415 + WORD $0x2f00e416 + WORD $0x2f00e417 WORD $0x54000062 - WORD $0xaa1f03ec - WORD $0x14000013 - WORD $0xaa0103ec - WORD $0xaa0203ed + WORD $0xaa1f03eb + WORD $0x1400003a + WORD $0xaa1f03fd + WORD $0x8b1d0f2b + WORD $0xa5fd4118 + WORD $0xa5f7417a + WORD $0xa5f84179 + WORD $0xa5ec417b + WORD $0xa5f6417f + WORD $0xa5f54168 + WORD $0xa5ed417c + WORD $0xa5ef417d + WORD $0xa5f0417e + WORD $0xa5f14169 + WORD $0x65da0b1a + WORD $0x65d90b19 + WORD $0xa5e3416a + WORD $0x65df0b1f + WORD $0xa5f4416b + WORD $0xa5f3416c + WORD $0xa5e7416d + WORD $0xa5e4416e + WORD $0x65dc0b1c + WORD $0x65dd0b1d + WORD $0x65de0b1e + WORD $0x65d82341 + WORD $0x65db0b1a + WORD $0x65c80b1b + WORD $0x65d82320 + WORD $0xa5fd4339 + WORD $0xa5e64168 + WORD $0x65cb0b0b + WORD $0x65cc0b0c + WORD $0x04f0e3fd + WORD $0x65d823e2 + WORD $0x65cd0b1f + WORD $0x65d90b19 + WORD $0x65c80b08 + WORD $0x65d82363 + WORD $0x65c90b1b + WORD $0x65ca0b09 + WORD $0x65ce0b18 + WORD $0xeb1d01df + WORD $0x65d82164 + WORD $0x65d82185 + WORD $0x65d823e6 + WORD $0x65d82107 + WORD $0x65d82310 + WORD $0x65d82131 + WORD $0x65d82372 + WORD $0x65d823d3 + WORD $0x65d823b4 + WORD $0x65d82395 + WORD $0x65d82356 + WORD $0x65d82337 + WORD $0x54fff981 + WORD $0xeb0e00bf + WORD $0xaa0e03eb + WORD $0x54fff440 + WORD $0xd37df17d + WORD $0xcb0b00be + WORD $0xaa0a03eb + WORD $0xaa0803e2 + WORD $0x8b1d0161 + WORD $0xfc7d6858 + WORD $0xf10007de + WORD $0x6d78e839 + WORD $0x91002042 + WORD $0x6d79f03b + WORD $0x9100216b + WORD $0x6d7ca829 + WORD $0x1e7a0b1a + WORD $0x1e790b19 + WORD $0x1e7b0b1b + WORD $0x1e7c0b1c + WORD $0x6d7af83d + WORD $0x1e690b09 + WORD $0x6d7ba03f + WORD $0x1e7a2ad6 + WORD $0x6d7de82b + WORD $0x1e792af7 + WORD $0x1e6a0b19 + WORD $0x1e7d0b1d + WORD $0x6d7fb82d + WORD $0x1e7e0b1e + WORD $0x1e6b0b0a + WORD $0x6d7eb02b + WORD $0x1e7f0b1f + WORD $0x1e680b08 + WORD $0x1e7b2ab5 + WORD $0x1e7a0b1a + WORD $0x1e7c2a94 + WORD $0x1e6d0b1c + WORD $0x1e6b0b1b + WORD $0x1e6c0b0b + WORD $0x1e7d2a73 + WORD $0x1e6e0b18 + WORD $0x1e7e2a52 + WORD $0x1e6928e7 + WORD $0x1e7f2a31 + WORD $0x1e682a10 + WORD $0x1e7928c6 + WORD $0x1e6a28a5 + WORD $0x1e7a2884 + WORD $0x1e7c2821 + WORD $0x1e7b2863 + WORD $0x1e6b2842 + WORD $0x1e782800 + WORD $0x54fffa61 + WORD $0x17ffff6f + WORD $0x6f00e400 + WORD $0xaa1f03e9 + WORD $0x9101200a + WORD $0x9100812b + WORD $0xf81b815f + WORD $0x91004129 + WORD $0xad3e0140 + WORD $0xeb02017f + WORD $0xad3f0140 + WORD $0xad000140 + WORD $0x3d800940 + WORD $0xf900195f + WORD $0x9102014a + WORD $0x54fffecd + WORD $0xf94003eb + WORD $0xeb09004a + WORD $0x5400090d + WORD $0xf10000bf + WORD $0x540005ad + WORD $0x04a0e3ea + WORD $0x8b090d6b + WORD $0x25d8e3e0 + WORD $0xcb0a03ec + WORD $0x8a0c00ac + WORD $0x14000006 + WORD $0xfc297800 + WORD $0x91000529 + WORD $0x9100216b + WORD $0xeb02013f + WORD $0x54000760 + WORD $0xeb0a00bf + WORD $0x54000082 + WORD $0x2f00e400 + WORD $0xaa1f03ed + WORD $0x14000015 WORD $0xaa0b03ee - WORD $0x85804181 - WORD $0x858041a2 - WORD $0x04b0e7ee + WORD $0x2f00e400 + WORD $0xaa1f03ed + WORD $0x0430e3ee + WORD $0xaa0803ef + WORD $0xa5ed4161 + WORD $0x858041e2 WORD $0x65c20821 - WORD $0x858045a2 - WORD $0x0431e3ed + WORD $0x858045e2 + WORD $0x0431e3ef WORD $0x65d82020 - WORD $0x85804581 - WORD $0x0431e3ec + WORD $0xa5ed41c1 + WORD $0x04b0e3ed WORD $0x65c20821 + WORD $0xeb0d019f WORD $0x65d82020 - WORD $0xb5fffeae - WORD $0xeb0b00bf - WORD $0xaa0b03ec - WORD $0x54fffca0 - WORD $0xfc6c7821 - WORD $0xfc6c7842 - WORD $0x9100058c + WORD $0x54fffea1 WORD $0xeb0c00bf + WORD $0xaa0c03ed + WORD $0x54fffc60 + WORD $0xfc6d7961 + WORD $0xfc6d7902 + WORD $0x910005ad + WORD $0xeb0d00bf WORD $0x1e620821 WORD $0x1e612800 WORD $0x54ffff41 - WORD $0x17ffffdd - WORD $0xeb09011f - WORD $0x54000062 - WORD $0xaa1f03e9 - WORD $0x1400000d + WORD $0x17ffffdb + WORD $0x04a0e3ec + WORD $0xeb0c015f + WORD $0x54000243 + WORD $0xd37df12d WORD $0x6f00e400 - WORD $0xcb0903e9 - WORD $0xaa0003ea - WORD $0x8a090109 - WORD $0xaa0903eb - WORD $0xe5804140 - WORD $0x04b0e7eb - WORD $0xe5804540 - WORD $0x0431e3ea - WORD $0xb5ffff8b - WORD $0xeb09011f + WORD $0xcb0c03ec + WORD $0x8a0c014c + WORD $0x25d8e3e0 + WORD $0xaa1f03e8 + WORD $0x8b0d000b + WORD $0x0430e3ed + WORD $0x8b0c0129 + WORD $0x8b0d000d + WORD $0xe5e84160 + WORD $0xe5e841a0 + WORD $0x04b0e3e8 + WORD $0xeb08019f + WORD $0x54ffff81 + WORD $0xeb0c015f WORD $0x540000c0 - WORD $0x8b090c0a - WORD $0xcb090108 - WORD $0xf1000508 - WORD $0xf800855f + WORD $0x8b090c08 + WORD $0xcb090049 + WORD $0xf1000529 + WORD $0xf800851f WORD $0x54ffffc1 + WORD $0xa9494ff4 + WORD $0xfd400bee + WORD $0xa94857f6 + WORD $0xa9475ff8 + WORD $0xa94667fa + WORD $0xa9457bfd + WORD $0x6d4423e9 + WORD $0x6d432beb + WORD $0x6d4233ed + WORD $0x910283ff WORD $0xd65f03c0 RET @@ -2461,50 +3471,216 @@ TEXT ·movingAverageFloat32SVE2(SB), NOSPLIT|NOFRAME, $0-56 MOVD a_len+32(FP), R3 MOVD width+48(FP), R4 WORD $0xf100049f - WORD $0x5400056b + WORD $0x5400090b WORD $0xeb040068 - WORD $0x5400052b + WORD $0x540008cb + WORD $0xd10283ff WORD $0xeb08005f + WORD $0x6d013bef WORD $0x9a88d448 - WORD $0xf100051f - WORD $0x540004ab + WORD $0x6d0233ed + WORD $0xf100411f + WORD $0x6d032beb + WORD $0x6d0423e9 + WORD $0xa9057bfd + WORD $0xa90667fa + WORD $0xa9075ff8 + WORD $0xa90857f6 + WORD $0xa9094ff4 + WORD $0xf90007e1 + WORD $0x5400070a + WORD $0xaa1f03e9 + WORD $0xf94007ee + WORD $0xeb08013f + WORD $0x5400052a + WORD $0xd37ef52b WORD $0x9e230080 WORD $0x0460e3ea + WORD $0xcb0a03ec WORD $0x2598e3e0 - WORD $0xcb0a03eb - WORD $0xaa1f03e9 - WORD $0x8a0b008b - WORD $0x14000007 + WORD $0xaa0b03ed + WORD $0x8b0b01cb + WORD $0x8a0c008c + WORD $0x0430e3ed + WORD $0x8b0d01cd + WORD $0x14000008 WORD $0x1e201821 - WORD $0x91001021 + WORD $0x910011ad + WORD $0x9100116b WORD $0xbc297801 WORD $0x91000529 WORD $0xeb08013f - WORD $0x54000300 + WORD $0x540002e0 WORD $0x2f00e401 WORD $0xeb0a009f WORD $0x54000062 - WORD $0xaa1f03ec - WORD $0x1400000d - WORD $0xaa0103ec - WORD $0xaa0b03ed - WORD $0x85804182 - WORD $0x0470e7ed + WORD $0xaa1f03ee + WORD $0x1400000c + WORD $0xaa1f03ee + WORD $0xa54e4162 WORD $0x65982041 - WORD $0x85804582 - WORD $0x0431e3ec + WORD $0xa54e41a2 + WORD $0x0470e3ee + WORD $0xeb0e019f WORD $0x65982041 - WORD $0xb5ffff4d - WORD $0xeb0b009f - WORD $0xaa0b03ec - WORD $0x54fffd40 - WORD $0xbc6c7822 - WORD $0x9100058c + WORD $0x54ffff41 WORD $0xeb0c009f + WORD $0xaa0c03ee + WORD $0x54fffd40 + WORD $0xbc6e7962 + WORD $0x910005ce + WORD $0xeb0e009f WORD $0x1e222821 WORD $0x54ffff81 WORD $0x17ffffe4 + WORD $0xa9494ff4 + WORD $0xa94857f6 + WORD $0xa9475ff8 + WORD $0xa94667fa + WORD $0xa9457bfd + WORD $0x6d4423e9 + WORD $0x6d432beb + WORD $0x6d4233ed + WORD $0x6d413bef + WORD $0x910283ff WORD $0xd65f03c0 + WORD $0x9e230080 + WORD $0x04a0e3ea + WORD $0x2598e3e0 + WORD $0xcb0a03e9 + WORD $0xaa1f03f0 + WORD $0x9100c02b + WORD $0x5280021a + WORD $0xd280002c + WORD $0xd280004d + WORD $0xd280006e + WORD $0xd280008f + WORD $0xd28000b1 + WORD $0x4e040400 + WORD $0xd28000c2 + WORD $0x8a090083 + WORD $0xd28000e5 + WORD $0xd2800106 + WORD $0xd2800127 + WORD $0xd2800153 + WORD $0xd2800174 + WORD $0xd2800195 + WORD $0xd28001b6 + WORD $0xd28001d7 + WORD $0xd28001f8 + WORD $0xaa0103f9 + WORD $0xaa0a03e1 + WORD $0x14000010 + WORD $0x6e0c04c5 + WORD $0x6e20fc42 + WORD $0x8b10080a + WORD $0x9100413a + WORD $0x91010339 + WORD $0x9101016b + WORD $0xeb08035f + WORD $0xaa0903f0 + WORD $0x6e180485 + WORD $0x6e20fca4 + WORD $0x6e20fc63 + WORD $0xad000944 + WORD $0x6e20fc21 + WORD $0xad010543 + WORD $0x54fff42c + WORD $0xeb01009f + WORD $0xaa1a03e9 + WORD $0x54000122 + WORD $0x2f00e404 + WORD $0x6f00e402 + WORD $0xaa1f03fd + WORD $0x2f00e406 + WORD $0x2f00e405 + WORD $0x6f00e403 + WORD $0x6f00e401 + WORD $0x14000043 + WORD $0x2f00e407 + WORD $0x2f00e410 + WORD $0xaa1f03fa + WORD $0x2f00e413 + WORD $0x2f00e401 + WORD $0x2f00e411 + WORD $0x2f00e415 + WORD $0x2f00e417 + WORD $0x2f00e403 + WORD $0x2f00e414 + WORD $0x2f00e416 + WORD $0x2f00e418 + WORD $0x2f00e402 + WORD $0x2f00e412 + WORD $0x2f00e404 + WORD $0x2f00e406 + WORD $0x2f00e405 + WORD $0x8b1a0b3d + WORD $0xa55a4339 + WORD $0x04b0e3fa + WORD $0xa55843be + WORD $0xa54c43ba + WORD $0xa54d43bb + WORD $0xa54e43bc + WORD $0xa54f43bd + WORD $0xa55143bf + WORD $0xa54243a8 + WORD $0xa54543a9 + WORD $0xa54643aa + WORD $0xa54743ab + WORD $0xa55743ac + WORD $0xeb1a007f + WORD $0x659823c7 + WORD $0xa55643be + WORD $0xa55343ad + WORD $0xa55443ae + WORD $0xa55543af + WORD $0x65982190 + WORD $0x659823d3 + WORD $0x659821e1 + WORD $0x659821d1 + WORD $0x659821b5 + WORD $0x65982177 + WORD $0x65982143 + WORD $0x65982134 + WORD $0x65982116 + WORD $0x659823f8 + WORD $0x659823a2 + WORD $0x65982392 + WORD $0x65982364 + WORD $0x65982346 + WORD $0x65982325 + WORD $0x54fffba1 + WORD $0x6e0c0702 + WORD $0x6e0c06e3 + WORD $0xeb03009f + WORD $0x6e0c0661 + WORD $0x6e0c0644 + WORD $0xaa0303fd + WORD $0x6e1406c2 + WORD $0x6e1406a3 + WORD $0x6e140601 + WORD $0x6e1c0682 + WORD $0x6e1c0623 + WORD $0x6e1c04e1 + WORD $0x54fff4a0 + WORD $0xd37ef7ba + WORD $0xcb1d009d + WORD $0xaa0b03fe + WORD $0x8b1a03ca + WORD $0xf10007bd + WORD $0x910013de + WORD $0x2d7a4151 + WORD $0xfc5d8147 + WORD $0xad7f4d52 + WORD $0x3dc00154 + WORD $0x0e24d4e4 + WORD $0x1e252a25 + WORD $0x1e262a06 + WORD $0x4e21d681 + WORD $0x4e22d642 + WORD $0x4e23d663 + WORD $0x54fffe61 + WORD $0x17ffff93 RET // func shiftDivFloat32SVE2(dst []float32, a []float32, shift float32, denom float32) @@ -2632,50 +3808,227 @@ TEXT ·movingAverageFloat64SVE2(SB), NOSPLIT|NOFRAME, $0-56 MOVD a_len+32(FP), R3 MOVD width+48(FP), R4 WORD $0xf100049f - WORD $0x5400056b + WORD $0x540008cb WORD $0xeb040068 - WORD $0x5400052b + WORD $0x5400088b + WORD $0xd10243ff WORD $0xeb08005f + WORD $0x6d0133ed WORD $0x9a88d448 - WORD $0xf100051f - WORD $0x540004ab + WORD $0x6d022beb + WORD $0xf100411f + WORD $0x6d0323e9 + WORD $0xa9047bfd + WORD $0xa90567fa + WORD $0xa9065ff8 + WORD $0xa90757f6 + WORD $0xa9084ff4 + WORD $0xf90007e1 + WORD $0x540006ea + WORD $0xaa1f03e9 + WORD $0xf94007ee + WORD $0xeb08013f + WORD $0x5400052a + WORD $0xd37df12b WORD $0x9e630080 WORD $0x04a0e3ea + WORD $0xcb0a03ec WORD $0x25d8e3e0 - WORD $0xcb0a03eb - WORD $0xaa1f03e9 - WORD $0x8a0b008b - WORD $0x14000007 + WORD $0xaa0b03ed + WORD $0x8b0b01cb + WORD $0x8a0c008c + WORD $0x0430e3ed + WORD $0x8b0d01cd + WORD $0x14000008 WORD $0x1e601821 - WORD $0x91002021 + WORD $0x910021ad + WORD $0x9100216b WORD $0xfc297801 WORD $0x91000529 WORD $0xeb08013f - WORD $0x54000300 + WORD $0x540002e0 WORD $0x2f00e401 WORD $0xeb0a009f WORD $0x54000062 - WORD $0xaa1f03ec - WORD $0x1400000d - WORD $0xaa0103ec - WORD $0xaa0b03ed - WORD $0x85804182 - WORD $0x04b0e7ed + WORD $0xaa1f03ee + WORD $0x1400000c + WORD $0xaa1f03ee + WORD $0xa5ee4162 WORD $0x65d82041 - WORD $0x85804582 - WORD $0x0431e3ec + WORD $0xa5ee41a2 + WORD $0x04b0e3ee + WORD $0xeb0e019f WORD $0x65d82041 - WORD $0xb5ffff4d - WORD $0xeb0b009f - WORD $0xaa0b03ec - WORD $0x54fffd40 - WORD $0xfc6c7822 - WORD $0x9100058c + WORD $0x54ffff41 WORD $0xeb0c009f + WORD $0xaa0c03ee + WORD $0x54fffd40 + WORD $0xfc6e7962 + WORD $0x910005ce + WORD $0xeb0e009f WORD $0x1e622821 WORD $0x54ffff81 WORD $0x17ffffe4 + WORD $0xa9484ff4 + WORD $0xa94757f6 + WORD $0xa9465ff8 + WORD $0xa94567fa + WORD $0xa9447bfd + WORD $0x6d4323e9 + WORD $0x6d422beb + WORD $0x6d4133ed + WORD $0x910243ff WORD $0xd65f03c0 + WORD $0x9e630080 + WORD $0x04e0e3ea + WORD $0x25d8e3e0 + WORD $0xcb0a03e9 + WORD $0xaa1f03f8 + WORD $0x9101e02b + WORD $0x5280021a + WORD $0xd280002c + WORD $0x8a09008d + WORD $0xd280004e + WORD $0xd280006f + WORD $0xd2800090 + WORD $0xd28000b1 + WORD $0xd28000c2 + WORD $0xd28000e3 + WORD $0xd2800105 + WORD $0xd2800126 + WORD $0xd2800147 + WORD $0xd2800173 + WORD $0xd2800194 + WORD $0xd28001b5 + WORD $0xd28001d6 + WORD $0xd28001f7 + WORD $0xaa0103f9 + WORD $0xaa0a03e1 + WORD $0x14000020 + WORD $0x1e601b18 + WORD $0x8b180c0a + WORD $0x9100413a + WORD $0xeb08035f + WORD $0x91020339 + WORD $0x9102016b + WORD $0xaa0903f8 + WORD $0x1e601af7 + WORD $0x1e601ad6 + WORD $0x6d005d58 + WORD $0x1e601ab5 + WORD $0x1e601a94 + WORD $0x6d015556 + WORD $0x1e601a73 + WORD $0x1e601a52 + WORD $0x6d024d54 + WORD $0x1e601a31 + WORD $0x1e601a10 + WORD $0x6d034552 + WORD $0x1e6018e7 + WORD $0x1e6018c6 + WORD $0x6d041d50 + WORD $0x1e6018a5 + WORD $0x1e601884 + WORD $0x6d051546 + WORD $0x1e601863 + WORD $0x1e601842 + WORD $0x6d060d44 + WORD $0x1e601821 + WORD $0x6d070542 + WORD $0x54fff26c + WORD $0x2f00e401 + WORD $0x2f00e402 + WORD $0xeb01009f + WORD $0x2f00e403 + WORD $0x2f00e404 + WORD $0xaa1a03e9 + WORD $0x2f00e405 + WORD $0x2f00e406 + WORD $0x2f00e407 + WORD $0x2f00e410 + WORD $0x2f00e411 + WORD $0x2f00e412 + WORD $0x2f00e413 + WORD $0x2f00e414 + WORD $0x2f00e415 + WORD $0x2f00e416 + WORD $0x2f00e417 + WORD $0x2f00e418 + WORD $0x54000062 + WORD $0xaa1f03fd + WORD $0x14000029 + WORD $0xaa1f03fa + WORD $0x8b1a0f3d + WORD $0xa5fa4339 + WORD $0x04f0e3fa + WORD $0xa5f743bb + WORD $0xa5f643bc + WORD $0xa5f543a8 + WORD $0xa5ec43ba + WORD $0xa5ef43bd + WORD $0xa5f043be + WORD $0xa5f143bf + WORD $0xa5f443a9 + WORD $0xa5e343aa + WORD $0x65d82361 + WORD $0xa5ee43bb + WORD $0xa5e543ab + WORD $0x65d82382 + WORD $0xa5e243bc + WORD $0xa5f343ac + WORD $0x65d82103 + WORD $0xa5e643a8 + WORD $0xa5e743ad + WORD $0x65d82124 + WORD $0xeb1a01bf + WORD $0x65d82185 + WORD $0x65d821a6 + WORD $0x65d82107 + WORD $0x65d82170 + WORD $0x65d82151 + WORD $0x65d82392 + WORD $0x65d823f3 + WORD $0x65d823d4 + WORD $0x65d823b5 + WORD $0x65d82376 + WORD $0x65d82357 + WORD $0x65d82338 + WORD $0x54fffba1 + WORD $0xeb0d009f + WORD $0xaa0d03fd + WORD $0x54fff4a0 + WORD $0xd37df3ba + WORD $0xcb1d009d + WORD $0xaa0b03fe + WORD $0x8b1a03ca + WORD $0xf10007bd + WORD $0x910023de + WORD $0x6d78e959 + WORD $0x6d79f15b + WORD $0x6d7af95d + WORD $0x1e782b38 + WORD $0x1e772b57 + WORD $0x1e762b76 + WORD $0x1e752b95 + WORD $0x6d7be959 + WORD $0x1e742bb4 + WORD $0x6d7cf15b + WORD $0x1e732bd3 + WORD $0x6d7ff95d + WORD $0x1e722b32 + WORD $0x1e712b51 + WORD $0x1e702b70 + WORD $0x1e672b87 + WORD $0x6d7de959 + WORD $0x1e622ba2 + WORD $0x6d7ef15b + WORD $0x1e612bc1 + WORD $0x1e662b26 + WORD $0x1e652b45 + WORD $0x1e642b64 + WORD $0x1e632b83 + WORD $0x54fffca1 + WORD $0x17ffff85 RET // func shiftDivFloat64SVE2(dst []float64, a []float64, shift float64, denom float64) diff --git a/internal/loong64/numeric_lasx_loong64.go b/internal/loong64/numeric_lasx_loong64.go index 5a5cff4..432b6ca 100644 --- a/internal/loong64/numeric_lasx_loong64.go +++ b/internal/loong64/numeric_lasx_loong64.go @@ -99,12 +99,18 @@ func tileUint64LASX(dst []uint64, pattern []uint64) //go:noescape func gatherUint64LASX(dst []uint64, src []uint64, idx []int32) +//go:noescape +func movingAverageFloat32LASX(dst []float32, a []float32, width int) + //go:noescape func shiftDivFloat32LASX(dst []float32, a []float32, shift float32, denom float32) //go:noescape func layerNormFloat32LASX(dst []float32, a []float32, gamma []float32, beta []float32, shift float32, denom float32) +//go:noescape +func movingAverageFloat64LASX(dst []float64, a []float64, width int) + //go:noescape func shiftDivFloat64LASX(dst []float64, a []float64, shift float64, denom float64) diff --git a/internal/loong64/numeric_lasx_loong64.s b/internal/loong64/numeric_lasx_loong64.s index 121db1d..4926710 100644 --- a/internal/loong64/numeric_lasx_loong64.s +++ b/internal/loong64/numeric_lasx_loong64.s @@ -213,22 +213,90 @@ TEXT ·polyEvalFloat32LASX(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x0013a908 WORD $0x001328e7 WORD $0x001520e7 - WORD $0x40007920 - WORD $0x6401dc07 - WORD $0x002c992a - WORD $0x2b3ff140 - WORD $0x03800806 - WORD $0x60007926 + WORD $0x40001920 + WORD $0x0380400a + WORD $0x002c9928 + WORD $0x640020ea WORD $0x00150006 - WORD $0x02fffd28 - WORD $0x02ffe149 + WORD $0x50010c00 + WORD $0x6402d807 + WORD $0x03801005 + WORD $0x6c017ce5 + WORD $0x00150005 + WORD $0x50020c00 + WORD $0x2b3ff100 + WORD $0x03800406 + WORD $0x640088c9 + WORD $0x0015000c + WORD $0x7707c000 + WORD $0x02fffd2a + WORD $0x02ffe10b + WORD $0x0380400e + WORD $0x03400000 + WORD $0x002c9586 + WORD $0x0041098d + WORD $0x384834a1 + WORD $0x2c8080c3 + WORD $0x001501c6 + WORD $0x0015016e + WORD $0x0015014f + WORD $0x77d40002 + WORD $0x77d40004 + WORD $0x03400000 + WORD $0x03400000 WORD $0x03400000 + WORD $0x322001c5 + WORD $0x75388484 + WORD $0x753090a4 + WORD $0x75388c42 + WORD $0x753088a2 + WORD $0x02fffdef + WORD $0x02fff1ce + WORD $0x63ffe40f + WORD $0x002c918c + WORD $0x384c3484 + WORD $0x02c040ce + WORD $0x2cc08182 + WORD $0x001500cc + WORD $0x67ff9cee + WORD $0x50006800 + WORD $0x00150006 + WORD $0x02c0808a + WORD $0x03400000 + WORD $0x03400000 + WORD $0x03400000 + WORD $0x2b7f8140 + WORD $0x2b7f9140 + WORD $0x2b7fa140 + WORD $0x2b7fb140 + WORD $0x2b7fc140 + WORD $0x2b7fd140 + WORD $0x2b7fe140 + WORD $0x2b7ff140 + WORD $0x2b400140 + WORD $0x2b401140 + WORD $0x2b402140 + WORD $0x2b403140 + WORD $0x2b404140 + WORD $0x2b405140 + WORD $0x2b406140 + WORD $0x2b407140 + WORD $0x02c080cb + WORD $0x02c040c6 + WORD $0x02c1014a + WORD $0x67ffb4eb + WORD $0x6401d0c7 + WORD $0x2b3ff100 + WORD $0x0380080a + WORD $0x60005d2a + WORD $0x02fffd29 + WORD $0x02ffe108 WORD $0x03400000 WORD $0x03400000 WORD $0x004108ca WORD $0x383028a1 - WORD $0x0015012b - WORD $0x0015010c + WORD $0x0015010b + WORD $0x0015012c WORD $0x01149402 WORD $0x03400000 WORD $0x03400000 @@ -243,23 +311,19 @@ TEXT ·polyEvalFloat32LASX(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x38382882 WORD $0x5fffc0c7 WORD $0x50016c00 - WORD $0x64016807 - WORD $0x03801005 - WORD $0x6c001ce5 - WORD $0x00150005 - WORD $0x5000ac00 - WORD $0x03801005 - WORD $0x6c001ce5 - WORD $0x00150005 - WORD $0x50012c00 + WORD $0x001198e5 + WORD $0x03801008 + WORD $0x6c001ca8 + WORD $0x001500c5 + WORD $0x50013c00 WORD $0x03804005 WORD $0x6c001ce5 WORD $0x00150005 WORD $0x50005400 - WORD $0x03804005 - WORD $0x6c00a4e5 - WORD $0x00150005 - WORD $0x5000d400 + WORD $0x03804008 + WORD $0x6c00a4a8 + WORD $0x00150008 + WORD $0x5000d800 WORD $0x034030e6 WORD $0x00fe10e5 WORD $0x004110a5 @@ -274,7 +338,7 @@ TEXT ·polyEvalFloat32LASX(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x02ffc129 WORD $0x02c10108 WORD $0x47fff13f - WORD $0x5800ece5 + WORD $0x5800fce5 WORD $0x40003cc0 WORD $0x001500a8 WORD $0x00fe08e5 @@ -289,7 +353,7 @@ TEXT ·polyEvalFloat32LASX(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x02c010c6 WORD $0x02c04108 WORD $0x47fff4df - WORD $0x5800b0e5 + WORD $0x5800c0e5 WORD $0x001194e6 WORD $0x002c90a4 WORD $0x03400000 @@ -297,35 +361,39 @@ TEXT ·polyEvalFloat32LASX(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x02fffcc6 WORD $0x02c01084 WORD $0x47fff4df - WORD $0x50009000 - WORD $0x034030e6 - WORD $0x00fe10e5 - WORD $0x004110a5 + WORD $0x5000a000 + WORD $0x03403ce9 + WORD $0x0011a4a8 WORD $0x7707c001 - WORD $0x02c08088 - WORD $0x001500a9 - WORD $0x03400000 - WORD $0x2cff8101 - WORD $0x2cc00101 - WORD $0x02ffc129 - WORD $0x02c10108 - WORD $0x47fff13f - WORD $0x58005ce5 - WORD $0x40003cc0 - WORD $0x001500a8 - WORD $0x00fe08e5 - WORD $0x004108a5 + WORD $0x002c90ca + WORD $0x02c0814a + WORD $0x0010a4cb + WORD $0x00119d6b + WORD $0x2cff8141 + WORD $0x2cc00141 + WORD $0x02c0416b + WORD $0x02c1014a + WORD $0x47fff17f + WORD $0x40006d20 + WORD $0x0380100a + WORD $0x6800692a + WORD $0x03400ce9 + WORD $0x0011a4a5 + WORD $0x001094c5 WORD $0x72f7e001 - WORD $0x00119506 - WORD $0x002c9108 - WORD $0x03400000 + WORD $0x004108ca + WORD $0x002ca90a + WORD $0x0010a88a + WORD $0x0010a0c6 + WORD $0x0010a4c6 + WORD $0x00119cc6 WORD $0x03400000 WORD $0x03400000 - WORD $0x2c400101 + WORD $0x2c400141 WORD $0x02c010c6 - WORD $0x02c04108 + WORD $0x02c0414a WORD $0x47fff4df - WORD $0x580020e5 + WORD $0x40002120 WORD $0x001194e6 WORD $0x002c90a4 WORD $0x03400000 @@ -334,6 +402,8 @@ TEXT ·polyEvalFloat32LASX(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x02c01084 WORD $0x47fff4df WORD $0x4c000020 + WORD $0x0010a0c5 + WORD $0x53ffdfff RET // func convolveFloat32LASX(dst []float32, sig []float32, ker []float32) @@ -344,22 +414,70 @@ TEXT ·convolveFloat32LASX(SB), NOSPLIT|NOFRAME, $0-72 MOVV dst_len+8(FP), R7 MOVV sig_len+32(FP), R8 MOVV ker_len+56(FP), R9 - WORD $0x40013120 - WORD $0x60012d09 + WORD $0x40021120 + WORD $0x60020d09 WORD $0x0011a508 WORD $0x02c0050a WORD $0x00121d08 WORD $0x0013214a WORD $0x0013a0e7 WORD $0x00151d47 - WORD $0x64011007 - WORD $0x64006409 + WORD $0x03804008 + WORD $0x64000ce8 + WORD $0x00150008 + WORD $0x5000bc00 + WORD $0x64009009 + WORD $0x0015000c + WORD $0x02c080aa + WORD $0x002c9928 + WORD $0x02fff10b + WORD $0x0380400d + WORD $0x77e00000 + WORD $0x03400000 + WORD $0x001501a8 + WORD $0x0015016d + WORD $0x0015014e + WORD $0x0015012f + WORD $0x77d40001 + WORD $0x77d40002 + WORD $0x03400000 + WORD $0x03400000 + WORD $0x2cbf81c3 + WORD $0x322001a4 + WORD $0x2c8001c5 + WORD $0x75388c83 + WORD $0x75308c21 + WORD $0x75389483 + WORD $0x75308c42 + WORD $0x02fffdef + WORD $0x02c011ce + WORD $0x02fff1ad + WORD $0x47ffd9ff + WORD $0x002c918d + WORD $0x0041098c + WORD $0x384c3081 + WORD $0x2cc081a2 + WORD $0x02c0410d + WORD $0x02c1014a + WORD $0x0015010c + WORD $0x67ff98ed + WORD $0x50002c00 WORD $0x00150008 + WORD $0x02c0808a + WORD $0x77e00000 + WORD $0x03400000 + WORD $0x2cff8140 + WORD $0x2cc00140 + WORD $0x02c0810b + WORD $0x02c04108 + WORD $0x02c1014a + WORD $0x67ffeceb + WORD $0x64012907 + WORD $0x64005c09 WORD $0x002c9926 WORD $0x02fff0c6 + WORD $0x002c9505 WORD $0x0114a400 - WORD $0x03400000 - WORD $0x03400000 WORD $0x001500aa WORD $0x001500cb WORD $0x0015012c @@ -377,42 +495,50 @@ TEXT ·convolveFloat32LASX(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x02c00508 WORD $0x02c010a5 WORD $0x5fffc107 - WORD $0x5000ac00 - WORD $0x03801005 - WORD $0x6c000ce5 - WORD $0x00150005 - WORD $0x50008000 - WORD $0x03804005 - WORD $0x6c000ce5 - WORD $0x00150005 - WORD $0x50003800 - WORD $0x034030e6 - WORD $0x00fe10e5 - WORD $0x004110a5 - WORD $0x02c08088 + WORD $0x5000cc00 + WORD $0x0011a0e5 + WORD $0x03801006 + WORD $0x6c000ca6 + WORD $0x00150105 + WORD $0x50009c00 + WORD $0x03804006 + WORD $0x6c000ca6 + WORD $0x00150006 + WORD $0x50004800 + WORD $0x03403ce9 + WORD $0x0011a4a6 + WORD $0x002c910a + WORD $0x02c0814a + WORD $0x0010a50b + WORD $0x00119d6b WORD $0x77e00000 - WORD $0x001500a9 - WORD $0x2cff8100 - WORD $0x2cc00100 - WORD $0x02ffc129 - WORD $0x02c10108 - WORD $0x47fff13f - WORD $0x58005ce5 - WORD $0x40003cc0 - WORD $0x001500a8 - WORD $0x00fe08e5 - WORD $0x004108a5 - WORD $0x00119506 - WORD $0x002c9108 - WORD $0x73e00000 WORD $0x03400000 WORD $0x03400000 + WORD $0x2cff8140 + WORD $0x2cc00140 + WORD $0x02c0416b + WORD $0x02c1014a + WORD $0x47fff17f + WORD $0x40006d20 + WORD $0x0380100a + WORD $0x6800692a + WORD $0x03400ce9 + WORD $0x0011a4a5 + WORD $0x00109505 + WORD $0x0041090a + WORD $0x002ca8ca + WORD $0x0010a88a + WORD $0x00109906 + WORD $0x0010a4c6 + WORD $0x00119cc6 + WORD $0x73e00000 + WORD $0x03400000 WORD $0x03400000 - WORD $0x2c400100 + WORD $0x2c400140 WORD $0x02c010c6 - WORD $0x02c04108 + WORD $0x02c0414a WORD $0x47fff4df - WORD $0x580020e5 + WORD $0x40002120 WORD $0x001194e6 WORD $0x002c90a4 WORD $0x03400000 @@ -421,6 +547,8 @@ TEXT ·convolveFloat32LASX(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x02c01084 WORD $0x47fff4df WORD $0x4c000020 + WORD $0x00109905 + WORD $0x53ffdfff RET // func correlateFloat32LASX(dst []float32, sig []float32, ker []float32) @@ -431,18 +559,70 @@ TEXT ·correlateFloat32LASX(SB), NOSPLIT|NOFRAME, $0-72 MOVV dst_len+8(FP), R7 MOVV sig_len+32(FP), R8 MOVV ker_len+56(FP), R9 - WORD $0x40012120 - WORD $0x60011d09 + WORD $0x40021120 + WORD $0x60020d09 WORD $0x0011a508 WORD $0x02c0050a WORD $0x00121d08 WORD $0x0013214a WORD $0x0013a0e7 WORD $0x00151d47 - WORD $0x64010007 - WORD $0x64005409 + WORD $0x03804008 + WORD $0x64000ce8 + WORD $0x00150008 + WORD $0x5000bc00 + WORD $0x64009009 + WORD $0x0015000b + WORD $0x02c080aa + WORD $0x0380400c + WORD $0x77e00000 + WORD $0x03400000 + WORD $0x03400000 + WORD $0x03400000 + WORD $0x00150188 + WORD $0x001500cc + WORD $0x0015014d + WORD $0x0015012e + WORD $0x77d40001 + WORD $0x77d40002 + WORD $0x03400000 + WORD $0x03400000 + WORD $0x2cbf81a3 + WORD $0x32200184 + WORD $0x2c8001a5 + WORD $0x75388c83 + WORD $0x75308c21 + WORD $0x75389483 + WORD $0x75308c42 + WORD $0x02fffdce + WORD $0x02c011ad + WORD $0x02c0118c + WORD $0x47ffd9df + WORD $0x002c916c + WORD $0x0041096b + WORD $0x384c2c81 + WORD $0x2cc08182 + WORD $0x02c0410c + WORD $0x02c1014a + WORD $0x0015010b + WORD $0x67ff98ec + WORD $0x50002c00 WORD $0x00150008 + WORD $0x02c0808a + WORD $0x77e00000 + WORD $0x03400000 + WORD $0x2cff8140 + WORD $0x2cc00140 + WORD $0x02c0810b + WORD $0x02c04108 + WORD $0x02c1014a + WORD $0x67ffeceb + WORD $0x64012907 + WORD $0x64005c09 + WORD $0x002c9505 WORD $0x0114a400 + WORD $0x03400000 + WORD $0x03400000 WORD $0x001500aa WORD $0x001500cb WORD $0x0015012c @@ -460,42 +640,50 @@ TEXT ·correlateFloat32LASX(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x02c00508 WORD $0x02c010a5 WORD $0x5fffc107 - WORD $0x5000ac00 - WORD $0x03801005 - WORD $0x6c000ce5 - WORD $0x00150005 - WORD $0x50008000 - WORD $0x03804005 - WORD $0x6c000ce5 - WORD $0x00150005 - WORD $0x50003800 - WORD $0x034030e6 - WORD $0x00fe10e5 - WORD $0x004110a5 - WORD $0x02c08088 + WORD $0x5000cc00 + WORD $0x0011a0e5 + WORD $0x03801006 + WORD $0x6c000ca6 + WORD $0x00150105 + WORD $0x50009c00 + WORD $0x03804006 + WORD $0x6c000ca6 + WORD $0x00150006 + WORD $0x50004800 + WORD $0x03403ce9 + WORD $0x0011a4a6 + WORD $0x002c910a + WORD $0x02c0814a + WORD $0x0010a50b + WORD $0x00119d6b WORD $0x77e00000 - WORD $0x001500a9 - WORD $0x2cff8100 - WORD $0x2cc00100 - WORD $0x02ffc129 - WORD $0x02c10108 - WORD $0x47fff13f - WORD $0x58005ce5 - WORD $0x40003cc0 - WORD $0x001500a8 - WORD $0x00fe08e5 - WORD $0x004108a5 - WORD $0x00119506 - WORD $0x002c9108 - WORD $0x73e00000 WORD $0x03400000 WORD $0x03400000 + WORD $0x2cff8140 + WORD $0x2cc00140 + WORD $0x02c0416b + WORD $0x02c1014a + WORD $0x47fff17f + WORD $0x40006d20 + WORD $0x0380100a + WORD $0x6800692a + WORD $0x03400ce9 + WORD $0x0011a4a5 + WORD $0x00109505 + WORD $0x0041090a + WORD $0x002ca8ca + WORD $0x0010a88a + WORD $0x00109906 + WORD $0x0010a4c6 + WORD $0x00119cc6 + WORD $0x73e00000 + WORD $0x03400000 WORD $0x03400000 - WORD $0x2c400100 + WORD $0x2c400140 WORD $0x02c010c6 - WORD $0x02c04108 + WORD $0x02c0414a WORD $0x47fff4df - WORD $0x580020e5 + WORD $0x40002120 WORD $0x001194e6 WORD $0x002c90a4 WORD $0x03400000 @@ -504,6 +692,8 @@ TEXT ·correlateFloat32LASX(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x02c01084 WORD $0x47fff4df WORD $0x4c000020 + WORD $0x00109905 + WORD $0x53ffdfff RET // func normFloat64LASX(a []float64) float64 @@ -709,22 +899,98 @@ TEXT ·polyEvalFloat64LASX(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x0013a908 WORD $0x001328e7 WORD $0x001520e7 - WORD $0x40007920 - WORD $0x64012c07 - WORD $0x002d192a - WORD $0x2bbfe140 - WORD $0x03800806 - WORD $0x60007926 - WORD $0x00150006 - WORD $0x02fffd28 - WORD $0x02ffc149 + WORD $0x40001920 + WORD $0x03804008 + WORD $0x002d1926 + WORD $0x640020e8 + WORD $0x00150008 + WORD $0x50012c00 + WORD $0x64024807 + WORD $0x03802005 + WORD $0x6c019ce5 + WORD $0x00150005 + WORD $0x5001c000 + WORD $0x2bbfe0c0 + WORD $0x03800408 + WORD $0x6400b109 + WORD $0x0015000c + WORD $0x7707e000 + WORD $0x02fffd2a + WORD $0x02ffc0cb + WORD $0x0380400e + WORD $0x03400000 + WORD $0x002d1588 + WORD $0x00410d8d + WORD $0x384834a1 + WORD $0x2c808103 + WORD $0x2c810104 + WORD $0x2c818105 + WORD $0x001501c8 + WORD $0x0015016e + WORD $0x0015014f + WORD $0x77d40002 + WORD $0x77d40006 + WORD $0x77d40007 + WORD $0x77d40008 WORD $0x03400000 WORD $0x03400000 WORD $0x03400000 - WORD $0x00410cca - WORD $0x383428a1 - WORD $0x0015012b + WORD $0x321001c9 + WORD $0x75390508 + WORD $0x75312128 + WORD $0x75390ce7 + WORD $0x75311d27 + WORD $0x753910c6 + WORD $0x75311926 + WORD $0x75391442 + WORD $0x75310922 + WORD $0x02fffdef + WORD $0x02ffe1ce + WORD $0x63ffd40f + WORD $0x002d118c + WORD $0x384c3488 + WORD $0x2cc08187 + WORD $0x2cc10186 + WORD $0x02c0410e + WORD $0x2cc18182 WORD $0x0015010c + WORD $0x67ff74ee + WORD $0x50006000 + WORD $0x00150008 + WORD $0x02c1008a + WORD $0x03400000 + WORD $0x2bff0140 + WORD $0x2bff2140 + WORD $0x2bff4140 + WORD $0x2bff6140 + WORD $0x2bff8140 + WORD $0x2bffa140 + WORD $0x2bffc140 + WORD $0x2bffe140 + WORD $0x2bc00140 + WORD $0x2bc02140 + WORD $0x2bc04140 + WORD $0x2bc06140 + WORD $0x2bc08140 + WORD $0x2bc0a140 + WORD $0x2bc0c140 + WORD $0x2bc0e140 + WORD $0x02c0810b + WORD $0x02c04108 + WORD $0x02c2014a + WORD $0x67ffb4eb + WORD $0x64012107 + WORD $0x2bbfe0c0 + WORD $0x0380080a + WORD $0x60005d2a + WORD $0x02fffd29 + WORD $0x02ffc0c6 + WORD $0x03400000 + WORD $0x03400000 + WORD $0x00410d0a + WORD $0x383428a1 + WORD $0x001500cb + WORD $0x0015012c WORD $0x01149802 WORD $0x03400000 WORD $0x03400000 @@ -735,19 +1001,15 @@ TEXT ·polyEvalFloat64LASX(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x02fffd8c WORD $0x02ffe16b WORD $0x63ffec0c - WORD $0x02c004c6 + WORD $0x02c00508 WORD $0x383c2882 - WORD $0x5fffc0c7 + WORD $0x5fffc107 WORD $0x5000bc00 - WORD $0x6400b807 - WORD $0x03802005 - WORD $0x6c001ce5 - WORD $0x00150005 - WORD $0x50004000 - WORD $0x03802005 - WORD $0x6c0054e5 - WORD $0x00150005 - WORD $0x50008000 + WORD $0x0011a0e5 + WORD $0x03802006 + WORD $0x6c0054a6 + WORD $0x00150105 + WORD $0x50009000 WORD $0x00fe0ce5 WORD $0x00410ca5 WORD $0x02c08086 @@ -758,27 +1020,31 @@ TEXT ·polyEvalFloat64LASX(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x02ffe108 WORD $0x02c100c6 WORD $0x47fff11f - WORD $0x58006ce5 + WORD $0x58007ce5 WORD $0x001194e6 WORD $0x002d10a4 WORD $0x29c00080 WORD $0x02fffcc6 WORD $0x02c02084 WORD $0x47fff4df - WORD $0x50005000 - WORD $0x00fe0ce5 - WORD $0x00410ca5 + WORD $0x50006000 + WORD $0x03401ce6 + WORD $0x001198a5 + WORD $0x00109505 WORD $0x7707e001 - WORD $0x02c08086 - WORD $0x001500a8 + WORD $0x002d1109 + WORD $0x02c08129 + WORD $0x00109908 + WORD $0x00119d08 WORD $0x03400000 WORD $0x03400000 - WORD $0x2cff80c1 - WORD $0x2cc000c1 - WORD $0x02ffe108 - WORD $0x02c100c6 + WORD $0x03400000 + WORD $0x2cff8121 + WORD $0x2cc00121 + WORD $0x02c02108 + WORD $0x02c10129 WORD $0x47fff11f - WORD $0x58001ce5 + WORD $0x40001cc0 WORD $0x001194e6 WORD $0x002d10a4 WORD $0x2bc00080 @@ -796,19 +1062,79 @@ TEXT ·convolveFloat64LASX(SB), NOSPLIT|NOFRAME, $0-72 MOVV dst_len+8(FP), R7 MOVV sig_len+32(FP), R8 MOVV ker_len+56(FP), R9 - WORD $0x4000e120 - WORD $0x6000dd09 + WORD $0x4001e120 + WORD $0x6001dd09 WORD $0x0011a508 WORD $0x02c0050a WORD $0x00121d08 WORD $0x0013214a WORD $0x0013a0e7 WORD $0x00151d47 - WORD $0x6400c007 - WORD $0x64006409 + WORD $0x03804008 + WORD $0x64000ce8 + WORD $0x00150008 + WORD $0x5000e400 + WORD $0x6400b009 + WORD $0x0015000c + WORD $0x02c100aa + WORD $0x002d1928 + WORD $0x02ffe10b + WORD $0x0380400d + WORD $0x77e00000 + WORD $0x03400000 + WORD $0x001501a8 + WORD $0x0015016d + WORD $0x0015014e + WORD $0x0015012f + WORD $0x77d40001 + WORD $0x77d40002 + WORD $0x77d40003 + WORD $0x77d40004 + WORD $0x2cbf01c5 + WORD $0x321001a6 + WORD $0x2cbf81c7 + WORD $0x753914c5 + WORD $0x75311421 + WORD $0x2c8001c5 + WORD $0x75391cc7 + WORD $0x2c8081c8 + WORD $0x75311c42 + WORD $0x753914c5 + WORD $0x75311463 + WORD $0x753920c5 + WORD $0x75311484 + WORD $0x02fffdef + WORD $0x02c021ce + WORD $0x02ffe1ad + WORD $0x47ffc1ff + WORD $0x002d118d + WORD $0x00410d8c + WORD $0x384c3081 + WORD $0x2cc081a2 + WORD $0x2cc101a3 + WORD $0x2cc181a4 + WORD $0x02c0410d + WORD $0x02c2014a + WORD $0x0015010c + WORD $0x67ff78ed + WORD $0x50003400 WORD $0x00150008 + WORD $0x02c1008a + WORD $0x77e00000 + WORD $0x03400000 + WORD $0x2cff0140 + WORD $0x2cff8140 + WORD $0x2cc00140 + WORD $0x2cc08140 + WORD $0x02c0810b + WORD $0x02c04108 + WORD $0x02c2014a + WORD $0x67ffe4eb + WORD $0x6400d107 + WORD $0x64006409 WORD $0x002d1926 WORD $0x02ffe0c6 + WORD $0x002d1505 WORD $0x0114a800 WORD $0x03400000 WORD $0x03400000 @@ -829,23 +1155,27 @@ TEXT ·convolveFloat64LASX(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x02c00508 WORD $0x02c020a5 WORD $0x5fffc107 - WORD $0x50005c00 - WORD $0x03802005 - WORD $0x6c000ce5 - WORD $0x00150005 - WORD $0x50003400 - WORD $0x00fe0ce5 - WORD $0x00410ca5 - WORD $0x02c08086 + WORD $0x50006c00 + WORD $0x0011a0e5 + WORD $0x03802006 + WORD $0x6c000ca6 + WORD $0x00150105 + WORD $0x50004000 + WORD $0x03401ce6 + WORD $0x001198a5 + WORD $0x00109505 + WORD $0x002d1109 + WORD $0x02c08129 + WORD $0x00109908 + WORD $0x00119d08 WORD $0x77e00000 - WORD $0x001500a8 WORD $0x03400000 - WORD $0x2cff80c0 - WORD $0x2cc000c0 - WORD $0x02ffe108 - WORD $0x02c100c6 + WORD $0x2cff8120 + WORD $0x2cc00120 + WORD $0x02c02108 + WORD $0x02c10129 WORD $0x47fff11f - WORD $0x58001ce5 + WORD $0x40001cc0 WORD $0x001194e6 WORD $0x002d10a4 WORD $0x29c00080 @@ -863,17 +1193,77 @@ TEXT ·correlateFloat64LASX(SB), NOSPLIT|NOFRAME, $0-72 MOVV dst_len+8(FP), R7 MOVV sig_len+32(FP), R8 MOVV ker_len+56(FP), R9 - WORD $0x4000d120 - WORD $0x6000cd09 + WORD $0x4001d120 + WORD $0x6001cd09 WORD $0x0011a508 WORD $0x02c0050a WORD $0x00121d08 WORD $0x0013214a WORD $0x0013a0e7 WORD $0x00151d47 - WORD $0x6400b007 - WORD $0x64005409 + WORD $0x03804008 + WORD $0x64000ce8 WORD $0x00150008 + WORD $0x5000e400 + WORD $0x6400b009 + WORD $0x0015000b + WORD $0x02c100aa + WORD $0x0380400c + WORD $0x77e00000 + WORD $0x03400000 + WORD $0x03400000 + WORD $0x03400000 + WORD $0x00150188 + WORD $0x0015014c + WORD $0x001500cd + WORD $0x0015012e + WORD $0x77d40001 + WORD $0x77d40002 + WORD $0x77d40003 + WORD $0x77d40004 + WORD $0x2cbf0185 + WORD $0x321001a6 + WORD $0x2cbf8187 + WORD $0x753914c5 + WORD $0x75311421 + WORD $0x2c800185 + WORD $0x75391cc7 + WORD $0x2c808188 + WORD $0x75311c42 + WORD $0x753914c5 + WORD $0x75311463 + WORD $0x753920c5 + WORD $0x75311484 + WORD $0x02fffdce + WORD $0x02c021ad + WORD $0x02c0218c + WORD $0x47ffc1df + WORD $0x002d116c + WORD $0x00410d6b + WORD $0x384c2c81 + WORD $0x2cc08182 + WORD $0x2cc10183 + WORD $0x2cc18184 + WORD $0x02c0410c + WORD $0x02c2014a + WORD $0x0015010b + WORD $0x67ff78ec + WORD $0x50003400 + WORD $0x00150008 + WORD $0x02c1008a + WORD $0x77e00000 + WORD $0x03400000 + WORD $0x2cff0140 + WORD $0x2cff8140 + WORD $0x2cc00140 + WORD $0x2cc08140 + WORD $0x02c0810b + WORD $0x02c04108 + WORD $0x02c2014a + WORD $0x67ffe4eb + WORD $0x6400c107 + WORD $0x64005409 + WORD $0x002d1505 WORD $0x0114a800 WORD $0x001500aa WORD $0x001500cb @@ -892,23 +1282,27 @@ TEXT ·correlateFloat64LASX(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x02c00508 WORD $0x02c020a5 WORD $0x5fffc107 - WORD $0x50005c00 - WORD $0x03802005 - WORD $0x6c000ce5 - WORD $0x00150005 - WORD $0x50003400 - WORD $0x00fe0ce5 - WORD $0x00410ca5 - WORD $0x02c08086 + WORD $0x50006c00 + WORD $0x0011a0e5 + WORD $0x03802006 + WORD $0x6c000ca6 + WORD $0x00150105 + WORD $0x50004000 + WORD $0x03401ce6 + WORD $0x001198a5 + WORD $0x00109505 + WORD $0x002d1109 + WORD $0x02c08129 + WORD $0x00109908 + WORD $0x00119d08 WORD $0x77e00000 - WORD $0x001500a8 WORD $0x03400000 - WORD $0x2cff80c0 - WORD $0x2cc000c0 - WORD $0x02ffe108 - WORD $0x02c100c6 + WORD $0x2cff8120 + WORD $0x2cc00120 + WORD $0x02c02108 + WORD $0x02c10129 WORD $0x47fff11f - WORD $0x58001ce5 + WORD $0x40001cc0 WORD $0x001194e6 WORD $0x002d10a4 WORD $0x29c00080 @@ -3098,6 +3492,83 @@ TEXT ·gatherUint64LASX(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x4c000020 RET +// func movingAverageFloat32LASX(dst []float32, a []float32, width int) +TEXT ·movingAverageFloat32LASX(SB), NOSPLIT|NOFRAME, $0-56 + MOVV dst_base+0(FP), R4 + MOVV a_base+24(FP), R5 + MOVV dst_len+8(FP), R6 + MOVV a_len+32(FP), R7 + MOVV width+48(FP), R8 + WORD $0x64010c08 + WORD $0x600108e8 + WORD $0x0011a0e7 + WORD $0x02c004e9 + WORD $0x001218e7 + WORD $0x00131d29 + WORD $0x00139cc6 + WORD $0x00151926 + WORD $0x0380400b + WORD $0x0114a900 + WORD $0x64000ccb + WORD $0x00150007 + WORD $0x50008400 + WORD $0x0015000a + WORD $0x011d1801 + WORD $0x7707c021 + WORD $0x02c080a9 + WORD $0x77e00002 + WORD $0x03400000 + WORD $0x03400000 + WORD $0x00150167 + WORD $0x0015010b + WORD $0x0015012c + WORD $0x77d40044 + WORD $0x77d40043 + WORD $0x03400000 + WORD $0x03400000 + WORD $0x03400000 + WORD $0x2cbf8185 + WORD $0x2c800186 + WORD $0x753090a4 + WORD $0x75308cc3 + WORD $0x02fffd6b + WORD $0x02c0118c + WORD $0x47ffe97f + WORD $0x002c914b + WORD $0x0041094a + WORD $0x753a8484 + WORD $0x384c2884 + WORD $0x753a8463 + WORD $0x2cc08163 + WORD $0x02c040eb + WORD $0x02c10129 + WORD $0x001500ea + WORD $0x67ffa0cb + WORD $0x640058e6 + WORD $0x011d1800 + WORD $0x002c94e5 + WORD $0x0114a401 + WORD $0x03400000 + WORD $0x03400000 + WORD $0x03400000 + WORD $0x001500a9 + WORD $0x0015010a + WORD $0x01149422 + WORD $0x03400000 + WORD $0x2b000123 + WORD $0x01008c42 + WORD $0x02fffd4a + WORD $0x02c01129 + WORD $0x47fff15f + WORD $0x01068042 + WORD $0x004108e9 + WORD $0x38382482 + WORD $0x02c004e7 + WORD $0x02c010a5 + WORD $0x5fffc8e6 + WORD $0x4c000020 + RET + // func shiftDivFloat32LASX(dst []float32, a []float32, shift float32, denom float32) TEXT ·shiftDivFloat32LASX(SB), NOSPLIT|NOFRAME, $0-56 MOVV dst_base+0(FP), R4 @@ -3274,6 +3745,91 @@ TEXT ·layerNormFloat32LASX(SB), NOSPLIT|NOFRAME, $0-104 WORD $0x4c000020 RET +// func movingAverageFloat64LASX(dst []float64, a []float64, width int) +TEXT ·movingAverageFloat64LASX(SB), NOSPLIT|NOFRAME, $0-56 + MOVV dst_base+0(FP), R4 + MOVV a_base+24(FP), R5 + MOVV dst_len+8(FP), R6 + MOVV a_len+32(FP), R7 + MOVV width+48(FP), R8 + WORD $0x64012c08 + WORD $0x600128e8 + WORD $0x0011a0e7 + WORD $0x02c004e9 + WORD $0x001218e7 + WORD $0x00131d29 + WORD $0x00139cc6 + WORD $0x00151926 + WORD $0x0380400b + WORD $0x0114a900 + WORD $0x64000ccb + WORD $0x00150007 + WORD $0x5000a400 + WORD $0x0015000a + WORD $0x011d2801 + WORD $0x7707e021 + WORD $0x02c100a9 + WORD $0x77e00002 + WORD $0x03400000 + WORD $0x03400000 + WORD $0x00150167 + WORD $0x0015010b + WORD $0x0015012c + WORD $0x77d40046 + WORD $0x77d40045 + WORD $0x77d40044 + WORD $0x77d40043 + WORD $0x03400000 + WORD $0x2cbf0187 + WORD $0x2cbf8188 + WORD $0x2c800189 + WORD $0x2c80818a + WORD $0x753118e6 + WORD $0x75311505 + WORD $0x75311124 + WORD $0x75310d43 + WORD $0x02fffd6b + WORD $0x02c0218c + WORD $0x47ffd97f + WORD $0x002d114b + WORD $0x00410d4a + WORD $0x753b04c6 + WORD $0x384c2886 + WORD $0x753b04a5 + WORD $0x2cc08165 + WORD $0x753b0484 + WORD $0x2cc10164 + WORD $0x753b0463 + WORD $0x2cc18163 + WORD $0x02c040eb + WORD $0x02c20129 + WORD $0x001500ea + WORD $0x67ff80cb + WORD $0x640058e6 + WORD $0x011d2800 + WORD $0x002d14e5 + WORD $0x0114a801 + WORD $0x03400000 + WORD $0x03400000 + WORD $0x03400000 + WORD $0x001500a9 + WORD $0x0015010a + WORD $0x01149822 + WORD $0x03400000 + WORD $0x2b800123 + WORD $0x01010c42 + WORD $0x02fffd4a + WORD $0x02c02129 + WORD $0x47fff15f + WORD $0x01070042 + WORD $0x00410ce9 + WORD $0x383c2482 + WORD $0x02c004e7 + WORD $0x02c020a5 + WORD $0x5fffc8e6 + WORD $0x4c000020 + RET + // func shiftDivFloat64LASX(dst []float64, a []float64, shift float64, denom float64) TEXT ·shiftDivFloat64LASX(SB), NOSPLIT|NOFRAME, $0-64 MOVV dst_base+0(FP), R4 diff --git a/internal/loong64/numeric_register_lasx_loong64.go b/internal/loong64/numeric_register_lasx_loong64.go index 59bac6e..a5b9a22 100644 --- a/internal/loong64/numeric_register_lasx_loong64.go +++ b/internal/loong64/numeric_register_lasx_loong64.go @@ -242,6 +242,14 @@ func GatherUint64LASX(dst []uint64, src []uint64, idx []int32) { gatherUint64LASX(dst, src, idx) } +func MovingAverageFloat32LASX(dst []float32, a []float32, width int) { + if len(dst) < 16 { + ref.MovingAverage(dst, a, width) + return + } + movingAverageFloat32LASX(dst, a, width) +} + func ShiftDivFloat32LASX(dst []float32, a []float32, shift float32, denom float32) { n := min(len(dst), len(a)) if n < 16 { @@ -260,6 +268,14 @@ func LayerNormFloat32LASX(dst []float32, a []float32, gamma []float32, beta []fl layerNormFloat32LASX(dst[:n:n], a, gamma, beta, shift, denom) } +func MovingAverageFloat64LASX(dst []float64, a []float64, width int) { + if len(dst) < 16 { + ref.MovingAverage(dst, a, width) + return + } + movingAverageFloat64LASX(dst, a, width) +} + func ShiftDivFloat64LASX(dst []float64, a []float64, shift float64, denom float64) { n := min(len(dst), len(a)) if n < 16 { @@ -331,8 +347,10 @@ func registerNumericLASX(s *kernel.Set) { s.U32.Gather = GatherUint32LASX s.U64.Tile = TileUint64LASX s.U64.Gather = GatherUint64LASX + s.F32.MovingAverage = MovingAverageFloat32LASX s.F32.ShiftDiv = ShiftDivFloat32LASX s.F32.LayerNorm = LayerNormFloat32LASX + s.F64.MovingAverage = MovingAverageFloat64LASX s.F64.ShiftDiv = ShiftDivFloat64LASX s.F64.LayerNorm = LayerNormFloat64LASX s.U64.Random = RandomU64LASX diff --git a/internal/riscv64/numeric_rvv_riscv64.s b/internal/riscv64/numeric_rvv_riscv64.s index 552bfe0..541ab7a 100644 --- a/internal/riscv64/numeric_rvv_riscv64.s +++ b/internal/riscv64/numeric_rvv_riscv64.s @@ -263,20 +263,80 @@ TEXT ·polyEvalFloat32RVV(SB), NOSPLIT|NOFRAME, $0-72 MOV coeffs_len+56(FP), X15 WORD $0x00e6c463 WORD $0x00070693 - WORD $0x06078663 - WORD $0x0ad05e63 - WORD $0x00279713 - WORD $0x00e60733 - WORD $0xffc72787 - WORD $0x00200613 - WORD $0x08c7c063 - WORD $0x00000613 + WORD $0x00078c63 + WORD $0x01000713 + WORD $0x00279813 + WORD $0x02e6dc63 + WORD $0x00000713 + WORD $0x0e80006f + WORD $0x16d05c63 + WORD $0x00000593 + WORD $0x0d16f657 + WORD $0x00259713 + WORD $0x00e50733 + WORD $0x5e003457 + WORD $0x40c686b3 + WORD $0x02076427 + WORD $0x00c585b3 + WORD $0xfe0692e3 + WORD $0x1500006f + WORD $0x010602b3 + WORD $0xffc2a787 + WORD $0x00100713 + WORD $0x08f75063 + WORD $0x00000713 + WORD $0xcd147057 + WORD $0x5e07d457 + WORD $0xfff78893 + WORD $0xff828293 + WORD $0x01000393 + WORD $0x00271313 + WORD $0x00658733 + WORD $0x02076507 + WORD $0x02070713 + WORD $0x02076607 + WORD $0x00038713 + WORD $0x00028393 + WORD $0x00088e13 + WORD $0x9e80b857 + WORD $0x9e80b757 + WORD $0x0003a787 + WORD $0x93051857 + WORD $0x92e61757 + WORD $0xfffe0e13 + WORD $0x0307d857 + WORD $0x02e7d757 + WORD $0xffc38393 + WORD $0xffc042e3 + WORD $0x00650333 + WORD $0x02036827 + WORD $0x02030313 + WORD $0x01070393 + WORD $0x02036727 + WORD $0xfa76d2e3 + WORD $0x0300006f + WORD $0x00000713 + WORD $0xcd147057 + WORD $0x5e07d457 + WORD $0x00050893 + WORD $0x02088293 + WORD $0x0208e427 + WORD $0x0202e427 + WORD $0x02070293 + WORD $0x01070713 + WORD $0x04088893 + WORD $0xfe56d4e3 + WORD $0x08d75a63 + WORD $0x01060633 + WORD $0xffc62787 + WORD $0x00200813 + WORD $0x0507c863 WORD $0xfff78793 - WORD $0xff870713 - WORD $0x00261813 + WORD $0xff860613 + WORD $0x00271813 WORD $0x010588b3 WORD $0x0008a707 - WORD $0x00070893 + WORD $0x00060893 WORD $0x00078293 WORD $0x20f786d3 WORD $0x0008a607 @@ -286,22 +346,14 @@ TEXT ·polyEvalFloat32RVV(SB), NOSPLIT|NOFRAME, $0-72 WORD $0xffc88893 WORD $0xfe5046e3 WORD $0x01050833 - WORD $0x00160613 + WORD $0x00170713 WORD $0x00d82027 - WORD $0xfcd612e3 - WORD $0x0580006f - WORD $0x04d05a63 - WORD $0x00000593 - WORD $0x0d16f657 - WORD $0x00259713 - WORD $0x00e50733 - WORD $0x5e003457 - WORD $0x40c686b3 - WORD $0x02076427 - WORD $0x00c585b3 - WORD $0xfe0692e3 - WORD $0x02c0006f + WORD $0xfcd712e3 + WORD $0x0380006f WORD $0x00000593 + WORD $0x40e686b3 + WORD $0x00271713 + WORD $0x00e50533 WORD $0x0d107657 WORD $0x5e07d457 WORD $0x0c76f657 @@ -322,14 +374,189 @@ TEXT ·convolveFloat32RVV(SB), NOSPLIT|NOFRAME, $0-72 MOV dst_len+8(FP), X13 MOV sig_len+32(FP), X14 MOV ker_len+56(FP), X15 - WORD $0x0c078c63 - WORD $0x0cf74a63 + WORD $0x3a078c63 + WORD $0x3af74a63 WORD $0x40f70733 WORD $0x00d75463 WORD $0x00170693 - WORD $0x0cd05263 - WORD $0x08f05e63 + WORD $0xfe010113 + WORD $0x00813c23 + WORD $0x00913823 + WORD $0xc2202773 + WORD $0x00171713 + WORD $0x40e10133 + WORD $0x01000713 + WORD $0x00e6d663 + WORD $0x00000713 + WORD $0x29c0006f WORD $0x00000713 + WORD $0x26f05663 + WORD $0x00279893 + WORD $0x01000e13 + WORD $0x00100813 + WORD $0x011608b3 + WORD $0xffc00293 + WORD $0x00000393 + WORD $0x00271313 + WORD $0x000e0713 + WORD $0x00658e33 + WORD $0x0d007ed7 + WORD $0x42006c57 + WORD $0x5e0c0cd7 + WORD $0x5e0c0dd7 + WORD $0x5e0c0d57 + WORD $0x5e0c0657 + WORD $0x5e0c06d7 + WORD $0x5e0c0757 + WORD $0x5e0c07d7 + WORD $0x5e0c0857 + WORD $0x5e0c08d7 + WORD $0x5e0c0957 + WORD $0x5e0c09d7 + WORD $0x5e0c0a57 + WORD $0x5e0c0ad7 + WORD $0x5e0c0b57 + WORD $0x5e0c0bd7 + WORD $0x00078e93 + WORD $0x0d0eff57 + WORD $0xfff3cf93 + WORD $0x00239413 + WORD $0x002f9f93 + WORD $0x41e804b3 + WORD $0x008e0433 + WORD $0x01f88fb3 + WORD $0x00249493 + WORD $0x009f8fb3 + WORD $0x00440493 + WORD $0x0204e507 + WORD $0x00840493 + WORD $0x0204e587 + WORD $0x00c40493 + WORD $0x0204e407 + WORD $0x01010493 + WORD $0x02848427 + WORD $0x01040493 + WORD $0x0204ee07 + WORD $0x01440493 + WORD $0x0204ee87 + WORD $0x01840493 + WORD $0x0204ef07 + WORD $0x01c40493 + WORD $0x0204ef87 + WORD $0x02040493 + WORD $0x0204e387 + WORD $0x02440493 + WORD $0x0204e307 + WORD $0x02840493 + WORD $0x0204e287 + WORD $0x02c40493 + WORD $0x0204e207 + WORD $0x03040493 + WORD $0x0204e187 + WORD $0x03440493 + WORD $0x0204e107 + WORD $0x002f1493 + WORD $0x01f48fb3 + WORD $0x02046087 + WORD $0xffcf8f93 + WORD $0x0a5fe007 + WORD $0x03840f93 + WORD $0x03c40413 + WORD $0x020fe407 + WORD $0x02046487 + WORD $0x920090d7 + WORD $0xc2202ff3 + WORD $0x01f10fb3 + WORD $0x010f8f93 + WORD $0x028f80a7 + WORD $0x920510d7 + WORD $0x92059557 + WORD $0x01010f93 + WORD $0x028f8587 + WORD $0x920595d7 + WORD $0x920e1e57 + WORD $0x920e9ed7 + WORD $0x920f1f57 + WORD $0x920f9fd7 + WORD $0x920393d7 + WORD $0x92031357 + WORD $0x920292d7 + WORD $0x92021257 + WORD $0x920191d7 + WORD $0x92011157 + WORD $0x92041457 + WORD $0x920494d7 + WORD $0x0e9c9cd7 + WORD $0x0e8d9dd7 + WORD $0x0e2d1d57 + WORD $0x0e361657 + WORD $0x0e4696d7 + WORD $0x0e571757 + WORD $0x0e6797d7 + WORD $0x0e781857 + WORD $0x0ff898d7 + WORD $0x0fe91957 + WORD $0x0fd999d7 + WORD $0x0fca1a57 + WORD $0x0eba9ad7 + WORD $0x0eab1b57 + WORD $0x0e1b9bd7 + WORD $0xc2202ff3 + WORD $0x01f10fb3 + WORD $0x010f8f93 + WORD $0x028f8407 + WORD $0x0e8c1c57 + WORD $0x41ee8eb3 + WORD $0x01e383b3 + WORD $0xe80e9ce3 + WORD $0x439017d7 + WORD $0x43b01757 + WORD $0x43a016d7 + WORD $0x42c01657 + WORD $0x42d015d7 + WORD $0x42e01557 + WORD $0x42f01057 + WORD $0x430010d7 + WORD $0x43101157 + WORD $0x432011d7 + WORD $0x43301257 + WORD $0x434012d7 + WORD $0x43501357 + WORD $0x436013d7 + WORD $0x43701857 + WORD $0x438018d7 + WORD $0x00650333 + WORD $0x01070e13 + WORD $0x01132027 + WORD $0x01032227 + WORD $0x00732427 + WORD $0x00632627 + WORD $0x00532827 + WORD $0x00432a27 + WORD $0x00332c27 + WORD $0x00232e27 + WORD $0x02132027 + WORD $0x02032227 + WORD $0x02a32427 + WORD $0x02b32627 + WORD $0x02c32827 + WORD $0x02d32a27 + WORD $0x02e32c27 + WORD $0x02f32e27 + WORD $0xdbc6dae3 + WORD $0x02c0006f + WORD $0xcd147057 + WORD $0x5e003457 + WORD $0x00050813 + WORD $0x02080893 + WORD $0x02086427 + WORD $0x0208e427 + WORD $0x02070893 + WORD $0x01070713 + WORD $0x04080813 + WORD $0xff16d4e3 + WORD $0x0cd75663 + WORD $0x08f05c63 WORD $0x00279893 WORD $0x0d007857 WORD $0x42006457 @@ -366,8 +593,11 @@ TEXT ·convolveFloat32RVV(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x00170713 WORD $0x00f2a027 WORD $0xf8d716e3 - WORD $0x0280006f + WORD $0x0340006f WORD $0x00000593 + WORD $0x40e686b3 + WORD $0x00271713 + WORD $0x00e50533 WORD $0x0d16f657 WORD $0x00259713 WORD $0x00e50733 @@ -376,6 +606,12 @@ TEXT ·convolveFloat32RVV(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x02076427 WORD $0x00c585b3 WORD $0xfe0692e3 + WORD $0xc2202573 + WORD $0x00151513 + WORD $0x00a10133 + WORD $0x01813403 + WORD $0x01013483 + WORD $0x02010113 WORD $0x00008067 RET @@ -387,14 +623,188 @@ TEXT ·correlateFloat32RVV(SB), NOSPLIT|NOFRAME, $0-72 MOV dst_len+8(FP), X13 MOV sig_len+32(FP), X14 MOV ker_len+56(FP), X15 - WORD $0x0a078463 - WORD $0x0af74263 + WORD $0x36078663 + WORD $0x36f74463 WORD $0x40f70733 WORD $0x00d75463 WORD $0x00170693 - WORD $0x08d05a63 - WORD $0x06f05663 + WORD $0x01000713 + WORD $0x00e6d863 + WORD $0x00000713 + WORD $0x2ad04a63 + WORD $0x3480006f + WORD $0xff010113 + WORD $0xc2202773 + WORD $0x00171713 + WORD $0x40e10133 WORD $0x00000713 + WORD $0x24f05e63 + WORD $0x01000293 + WORD $0x00000893 + WORD $0x00271813 + WORD $0x00028713 + WORD $0x010582b3 + WORD $0x0d007357 + WORD $0x42006c57 + WORD $0x5e0c0d57 + WORD $0x5e0c0557 + WORD $0x5e0c05d7 + WORD $0x5e0c0657 + WORD $0x5e0c06d7 + WORD $0x5e0c0757 + WORD $0x5e0c07d7 + WORD $0x5e0c0857 + WORD $0x5e0c08d7 + WORD $0x5e0c0957 + WORD $0x5e0c09d7 + WORD $0x5e0c0a57 + WORD $0x5e0c0ad7 + WORD $0x5e0c0b57 + WORD $0x5e0c0bd7 + WORD $0x00078313 + WORD $0x0d0373d7 + WORD $0x00289e13 + WORD $0x01c60eb3 + WORD $0x01c28e33 + WORD $0x020eec87 + WORD $0x020e6407 + WORD $0xc2202ef3 + WORD $0x01d10eb3 + WORD $0x010e8e93 + WORD $0x028e8427 + WORD $0x004e0e93 + WORD $0x020ee407 + WORD $0x01010e93 + WORD $0x028e8427 + WORD $0x008e0e93 + WORD $0x020eee07 + WORD $0x00ce0e93 + WORD $0x020eee87 + WORD $0x010e0e93 + WORD $0x020eef07 + WORD $0x014e0e93 + WORD $0x020eef87 + WORD $0x018e0e93 + WORD $0x020ee387 + WORD $0x01ce0e93 + WORD $0x020ee307 + WORD $0x020e0e93 + WORD $0x020ee287 + WORD $0x024e0e93 + WORD $0x020ee207 + WORD $0x028e0e93 + WORD $0x020ee187 + WORD $0x02ce0e93 + WORD $0x020ee107 + WORD $0x030e0e93 + WORD $0x020ee087 + WORD $0x034e0e93 + WORD $0x020ee007 + WORD $0x038e0e93 + WORD $0x03ce0e13 + WORD $0x020ee407 + WORD $0x020e6487 + WORD $0xc2202e73 + WORD $0x01c10e33 + WORD $0x010e0e13 + WORD $0x028e0d87 + WORD $0x939d9dd7 + WORD $0xc2202e73 + WORD $0x01c10e33 + WORD $0x010e0e13 + WORD $0x028e0da7 + WORD $0x01010e13 + WORD $0x028e0d87 + WORD $0x939d9dd7 + WORD $0x939e1e57 + WORD $0x939e9ed7 + WORD $0x939f1f57 + WORD $0x939f9fd7 + WORD $0x939393d7 + WORD $0x93931357 + WORD $0x939292d7 + WORD $0x93921257 + WORD $0x939191d7 + WORD $0x93911157 + WORD $0x939090d7 + WORD $0x93901057 + WORD $0x93941457 + WORD $0x939494d7 + WORD $0xc2202e73 + WORD $0x01c10e33 + WORD $0x010e0e13 + WORD $0x028e0c87 + WORD $0x0f9c1c57 + WORD $0x0e9d1d57 + WORD $0x0e851557 + WORD $0x0e0595d7 + WORD $0x0e161657 + WORD $0x0e2696d7 + WORD $0x0e371757 + WORD $0x0e4797d7 + WORD $0x0e581857 + WORD $0x0e6898d7 + WORD $0x0e791957 + WORD $0x0ff999d7 + WORD $0x0fea1a57 + WORD $0x0fda9ad7 + WORD $0x0fcb1b57 + WORD $0x0fbb9bd7 + WORD $0x40730333 + WORD $0x007888b3 + WORD $0xe8031ce3 + WORD $0x43a017d7 + WORD $0x42a01757 + WORD $0x42b016d7 + WORD $0x42c01657 + WORD $0x42d015d7 + WORD $0x42e01557 + WORD $0x42f01057 + WORD $0x430010d7 + WORD $0x43101157 + WORD $0x432011d7 + WORD $0x43301257 + WORD $0x434012d7 + WORD $0x43501357 + WORD $0x436013d7 + WORD $0x43701857 + WORD $0x438018d7 + WORD $0x01050833 + WORD $0x01070293 + WORD $0x01182027 + WORD $0x01082227 + WORD $0x00782427 + WORD $0x00682627 + WORD $0x00582827 + WORD $0x00482a27 + WORD $0x00382c27 + WORD $0x00282e27 + WORD $0x02182027 + WORD $0x02082227 + WORD $0x02a82427 + WORD $0x02b82627 + WORD $0x02c82827 + WORD $0x02d82a27 + WORD $0x02e82c27 + WORD $0x02f82e27 + WORD $0xda56dae3 + WORD $0x02c0006f + WORD $0xcd147057 + WORD $0x5e003457 + WORD $0x00050813 + WORD $0x02080893 + WORD $0x02086427 + WORD $0x0208e427 + WORD $0x02070893 + WORD $0x01070713 + WORD $0x04080813 + WORD $0xff16d4e3 + WORD $0xc2202873 + WORD $0x00181813 + WORD $0x01010133 + WORD $0x01010113 + WORD $0x08d75e63 + WORD $0x06f05463 WORD $0x0d007857 WORD $0x42006457 WORD $0x00000893 @@ -419,8 +829,11 @@ TEXT ·correlateFloat32RVV(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x00170713 WORD $0x00f82027 WORD $0xfad716e3 - WORD $0x0280006f + WORD $0x0340006f WORD $0x00000593 + WORD $0x40e686b3 + WORD $0x00271713 + WORD $0x00e50533 WORD $0x0d16f657 WORD $0x00259713 WORD $0x00e50733 @@ -684,20 +1097,105 @@ TEXT ·polyEvalFloat64RVV(SB), NOSPLIT|NOFRAME, $0-72 MOV coeffs_len+56(FP), X15 WORD $0x00e6c463 WORD $0x00070693 - WORD $0x06078663 - WORD $0x0ad05e63 + WORD $0x00078c63 + WORD $0x01000813 WORD $0x00379713 - WORD $0x00e60733 - WORD $0xff873787 - WORD $0x00200613 - WORD $0x08c7c063 - WORD $0x00000613 + WORD $0x0306dc63 + WORD $0x00000813 + WORD $0x14c0006f + WORD $0x1cd05e63 + WORD $0x00000593 + WORD $0x0d96f657 + WORD $0x00359713 + WORD $0x00e50733 + WORD $0x5e003457 + WORD $0x40c686b3 + WORD $0x02077427 + WORD $0x00c585b3 + WORD $0xfe0692e3 + WORD $0x1b40006f + WORD $0x00e602b3 + WORD $0xff82b787 + WORD $0x00100813 + WORD $0x0af85c63 + WORD $0x00000813 + WORD $0xcd927057 + WORD $0x5e07d457 + WORD $0xfff78893 + WORD $0xff028293 + WORD $0x01000393 + WORD $0x00381313 + WORD $0x00658833 + WORD $0x02087507 + WORD $0x02080e13 + WORD $0x020e7607 + WORD $0x04080e13 + WORD $0x06080813 + WORD $0x020e7807 + WORD $0x02087a07 + WORD $0x00038813 + WORD $0x00028393 + WORD $0x00088e13 + WORD $0x9e80b957 + WORD $0x9e80bc57 + WORD $0x9e80bb57 + WORD $0x9e80b757 + WORD $0x0003b787 + WORD $0x93251957 + WORD $0x93861c57 + WORD $0x93681b57 + WORD $0x92ea1757 + WORD $0xfffe0e13 + WORD $0x0327d957 + WORD $0x0387dc57 + WORD $0x0367db57 + WORD $0x02e7d757 + WORD $0xff838393 + WORD $0xfdc04ae3 + WORD $0x00650333 + WORD $0x02030393 + WORD $0x0203fc27 + WORD $0x04030393 + WORD $0x0203fb27 + WORD $0x01080393 + WORD $0x02037927 + WORD $0x06030313 + WORD $0x02037727 + WORD $0xf676d6e3 + WORD $0x05c0006f + WORD $0x00000813 + WORD $0x04050893 + WORD $0xfcf8b027 + WORD $0xfcf8b427 + WORD $0xfcf8b827 + WORD $0xfcf8bc27 + WORD $0xfef8b027 + WORD $0xfef8b427 + WORD $0xfef8b827 + WORD $0xfef8bc27 + WORD $0x00f8b027 + WORD $0x00f8b427 + WORD $0x00f8b827 + WORD $0x00f8bc27 + WORD $0x02f8b027 + WORD $0x02f8b427 + WORD $0x02f8b827 + WORD $0x02f8bc27 + WORD $0x02080293 + WORD $0x01080813 + WORD $0x08088893 + WORD $0xfa56dae3 + WORD $0x08d85a63 + WORD $0x00e60633 + WORD $0xff863787 + WORD $0x00200713 + WORD $0x04e7c863 WORD $0xfff78793 - WORD $0xff070713 - WORD $0x00361813 - WORD $0x010588b3 + WORD $0xff060613 + WORD $0x00381713 + WORD $0x00e588b3 WORD $0x0008b707 - WORD $0x00070893 + WORD $0x00060893 WORD $0x00078293 WORD $0x22f786d3 WORD $0x0008b607 @@ -706,32 +1204,24 @@ TEXT ·polyEvalFloat64RVV(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x02c6f6d3 WORD $0xff888893 WORD $0xfe5046e3 - WORD $0x01050833 - WORD $0x00160613 - WORD $0x00d83027 - WORD $0xfcd612e3 - WORD $0x0580006f - WORD $0x04d05a63 - WORD $0x00000593 - WORD $0x0d96f657 - WORD $0x00359713 WORD $0x00e50733 - WORD $0x5e003457 - WORD $0x40c686b3 - WORD $0x02077427 - WORD $0x00c585b3 - WORD $0xfe0692e3 - WORD $0x02c0006f + WORD $0x00180813 + WORD $0x00d73027 + WORD $0xfcd812e3 + WORD $0x0380006f WORD $0x00000593 - WORD $0x0d907657 + WORD $0x41068633 + WORD $0x00381813 + WORD $0x01050533 + WORD $0x0d9076d7 WORD $0x5e07d457 - WORD $0x0c66f657 + WORD $0x0c6676d7 WORD $0x00359713 WORD $0x00e50733 - WORD $0x40c686b3 + WORD $0x40d60633 WORD $0x02077427 - WORD $0x00c585b3 - WORD $0xfe0694e3 + WORD $0x00d585b3 + WORD $0xfe0614e3 WORD $0x00008067 RET @@ -743,14 +1233,196 @@ TEXT ·convolveFloat64RVV(SB), NOSPLIT|NOFRAME, $0-72 MOV dst_len+8(FP), X13 MOV sig_len+32(FP), X14 MOV ker_len+56(FP), X15 - WORD $0x0c078c63 - WORD $0x0cf74a63 + WORD $0x3c078a63 + WORD $0x3cf74863 WORD $0x40f70733 WORD $0x00d75463 WORD $0x00170693 - WORD $0x0cd05263 - WORD $0x08f05e63 + WORD $0xfe010113 + WORD $0x00813c23 + WORD $0x00913823 + WORD $0xc2202773 + WORD $0x00171713 + WORD $0x40e10133 + WORD $0x01000713 + WORD $0x00e6d663 + WORD $0x00000713 + WORD $0x2b80006f WORD $0x00000713 + WORD $0x26f05663 + WORD $0x00379893 + WORD $0x01000e13 + WORD $0x00100813 + WORD $0x011608b3 + WORD $0xff800293 + WORD $0x00000393 + WORD $0x00371313 + WORD $0x000e0713 + WORD $0x00658e33 + WORD $0x0d807ed7 + WORD $0x42006c57 + WORD $0x5e0c0cd7 + WORD $0x5e0c0dd7 + WORD $0x5e0c0d57 + WORD $0x5e0c0657 + WORD $0x5e0c06d7 + WORD $0x5e0c0757 + WORD $0x5e0c07d7 + WORD $0x5e0c0857 + WORD $0x5e0c08d7 + WORD $0x5e0c0957 + WORD $0x5e0c09d7 + WORD $0x5e0c0a57 + WORD $0x5e0c0ad7 + WORD $0x5e0c0b57 + WORD $0x5e0c0bd7 + WORD $0x00078e93 + WORD $0x0d8eff57 + WORD $0xfff3cf93 + WORD $0x00339413 + WORD $0x003f9f93 + WORD $0x41e804b3 + WORD $0x008e0433 + WORD $0x01f88fb3 + WORD $0x00349493 + WORD $0x009f8fb3 + WORD $0x00840493 + WORD $0x0204f507 + WORD $0x01040493 + WORD $0x0204f587 + WORD $0x01840493 + WORD $0x0204f407 + WORD $0x01010493 + WORD $0x02848427 + WORD $0x02040493 + WORD $0x0204fe07 + WORD $0x02840493 + WORD $0x0204fe87 + WORD $0x03040493 + WORD $0x0204ff07 + WORD $0x03840493 + WORD $0x0204ff87 + WORD $0x04040493 + WORD $0x0204f387 + WORD $0x04840493 + WORD $0x0204f307 + WORD $0x05040493 + WORD $0x0204f287 + WORD $0x05840493 + WORD $0x0204f207 + WORD $0x06040493 + WORD $0x0204f187 + WORD $0x06840493 + WORD $0x0204f107 + WORD $0x003f1493 + WORD $0x01f48fb3 + WORD $0x02047087 + WORD $0xff8f8f93 + WORD $0x0a5ff007 + WORD $0x07040f93 + WORD $0x07840413 + WORD $0x020ff407 + WORD $0x02047487 + WORD $0x920090d7 + WORD $0xc2202ff3 + WORD $0x01f10fb3 + WORD $0x010f8f93 + WORD $0x028f80a7 + WORD $0x920510d7 + WORD $0x92059557 + WORD $0x01010f93 + WORD $0x028f8587 + WORD $0x920595d7 + WORD $0x920e1e57 + WORD $0x920e9ed7 + WORD $0x920f1f57 + WORD $0x920f9fd7 + WORD $0x920393d7 + WORD $0x92031357 + WORD $0x920292d7 + WORD $0x92021257 + WORD $0x920191d7 + WORD $0x92011157 + WORD $0x92041457 + WORD $0x920494d7 + WORD $0x0e9c9cd7 + WORD $0x0e8d9dd7 + WORD $0x0e2d1d57 + WORD $0x0e361657 + WORD $0x0e4696d7 + WORD $0x0e571757 + WORD $0x0e6797d7 + WORD $0x0e781857 + WORD $0x0ff898d7 + WORD $0x0fe91957 + WORD $0x0fd999d7 + WORD $0x0fca1a57 + WORD $0x0eba9ad7 + WORD $0x0eab1b57 + WORD $0x0e1b9bd7 + WORD $0xc2202ff3 + WORD $0x01f10fb3 + WORD $0x010f8f93 + WORD $0x028f8407 + WORD $0x0e8c1c57 + WORD $0x41ee8eb3 + WORD $0x01e383b3 + WORD $0xe80e9ce3 + WORD $0x439017d7 + WORD $0x43b01757 + WORD $0x43a016d7 + WORD $0x42c01657 + WORD $0x42d015d7 + WORD $0x42e01557 + WORD $0x42f01057 + WORD $0x430010d7 + WORD $0x43101157 + WORD $0x432011d7 + WORD $0x43301257 + WORD $0x434012d7 + WORD $0x43501357 + WORD $0x436013d7 + WORD $0x43701857 + WORD $0x438018d7 + WORD $0x00650333 + WORD $0x01070e13 + WORD $0x01133027 + WORD $0x01033427 + WORD $0x00733827 + WORD $0x00633c27 + WORD $0x02533027 + WORD $0x02433427 + WORD $0x02333827 + WORD $0x02233c27 + WORD $0x04133027 + WORD $0x04033427 + WORD $0x04a33827 + WORD $0x04b33c27 + WORD $0x06c33027 + WORD $0x06d33427 + WORD $0x06e33827 + WORD $0x06f33c27 + WORD $0xdbc6dae3 + WORD $0x0480006f + WORD $0xcd927057 + WORD $0x5e003657 + WORD $0xcda47057 + WORD $0x5e003457 + WORD $0x00050813 + WORD $0x02080893 + WORD $0xcd927057 + WORD $0x02087627 + WORD $0xcda47057 + WORD $0x0208f427 + WORD $0x06080893 + WORD $0xcd927057 + WORD $0x0208f627 + WORD $0x02070893 + WORD $0x01070713 + WORD $0x08080813 + WORD $0xfd16dae3 + WORD $0x0cd75663 + WORD $0x08f05c63 WORD $0x00379893 WORD $0x0d807857 WORD $0x42006457 @@ -787,8 +1459,11 @@ TEXT ·convolveFloat64RVV(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x00170713 WORD $0x00f2b027 WORD $0xf8d716e3 - WORD $0x0280006f + WORD $0x0340006f WORD $0x00000593 + WORD $0x40e686b3 + WORD $0x00371713 + WORD $0x00e50533 WORD $0x0d96f657 WORD $0x00359713 WORD $0x00e50733 @@ -797,6 +1472,12 @@ TEXT ·convolveFloat64RVV(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x02077427 WORD $0x00c585b3 WORD $0xfe0692e3 + WORD $0xc2202573 + WORD $0x00151513 + WORD $0x00a10133 + WORD $0x01813403 + WORD $0x01013483 + WORD $0x02010113 WORD $0x00008067 RET @@ -808,14 +1489,195 @@ TEXT ·correlateFloat64RVV(SB), NOSPLIT|NOFRAME, $0-72 MOV dst_len+8(FP), X13 MOV sig_len+32(FP), X14 MOV ker_len+56(FP), X15 - WORD $0x0a078463 - WORD $0x0af74263 + WORD $0x38078463 + WORD $0x38f74263 WORD $0x40f70733 WORD $0x00d75463 WORD $0x00170693 - WORD $0x08d05a63 - WORD $0x06f05663 + WORD $0x01000713 + WORD $0x00e6d863 WORD $0x00000713 + WORD $0x2cd04863 + WORD $0x3640006f + WORD $0xff010113 + WORD $0xc2202773 + WORD $0x00171713 + WORD $0x40e10133 + WORD $0x00000713 + WORD $0x24f05e63 + WORD $0x01000293 + WORD $0x00000893 + WORD $0x00371813 + WORD $0x00028713 + WORD $0x010582b3 + WORD $0x0d807357 + WORD $0x42006c57 + WORD $0x5e0c0d57 + WORD $0x5e0c0557 + WORD $0x5e0c05d7 + WORD $0x5e0c0657 + WORD $0x5e0c06d7 + WORD $0x5e0c0757 + WORD $0x5e0c07d7 + WORD $0x5e0c0857 + WORD $0x5e0c08d7 + WORD $0x5e0c0957 + WORD $0x5e0c09d7 + WORD $0x5e0c0a57 + WORD $0x5e0c0ad7 + WORD $0x5e0c0b57 + WORD $0x5e0c0bd7 + WORD $0x00078313 + WORD $0x0d8373d7 + WORD $0x00389e13 + WORD $0x01c60eb3 + WORD $0x01c28e33 + WORD $0x020efc87 + WORD $0x020e7407 + WORD $0xc2202ef3 + WORD $0x01d10eb3 + WORD $0x010e8e93 + WORD $0x028e8427 + WORD $0x008e0e93 + WORD $0x020ef407 + WORD $0x01010e93 + WORD $0x028e8427 + WORD $0x010e0e93 + WORD $0x020efe07 + WORD $0x018e0e93 + WORD $0x020efe87 + WORD $0x020e0e93 + WORD $0x020eff07 + WORD $0x028e0e93 + WORD $0x020eff87 + WORD $0x030e0e93 + WORD $0x020ef387 + WORD $0x038e0e93 + WORD $0x020ef307 + WORD $0x040e0e93 + WORD $0x020ef287 + WORD $0x048e0e93 + WORD $0x020ef207 + WORD $0x050e0e93 + WORD $0x020ef187 + WORD $0x058e0e93 + WORD $0x020ef107 + WORD $0x060e0e93 + WORD $0x020ef087 + WORD $0x068e0e93 + WORD $0x020ef007 + WORD $0x070e0e93 + WORD $0x078e0e13 + WORD $0x020ef407 + WORD $0x020e7487 + WORD $0xc2202e73 + WORD $0x01c10e33 + WORD $0x010e0e13 + WORD $0x028e0d87 + WORD $0x939d9dd7 + WORD $0xc2202e73 + WORD $0x01c10e33 + WORD $0x010e0e13 + WORD $0x028e0da7 + WORD $0x01010e13 + WORD $0x028e0d87 + WORD $0x939d9dd7 + WORD $0x939e1e57 + WORD $0x939e9ed7 + WORD $0x939f1f57 + WORD $0x939f9fd7 + WORD $0x939393d7 + WORD $0x93931357 + WORD $0x939292d7 + WORD $0x93921257 + WORD $0x939191d7 + WORD $0x93911157 + WORD $0x939090d7 + WORD $0x93901057 + WORD $0x93941457 + WORD $0x939494d7 + WORD $0xc2202e73 + WORD $0x01c10e33 + WORD $0x010e0e13 + WORD $0x028e0c87 + WORD $0x0f9c1c57 + WORD $0x0e9d1d57 + WORD $0x0e851557 + WORD $0x0e0595d7 + WORD $0x0e161657 + WORD $0x0e2696d7 + WORD $0x0e371757 + WORD $0x0e4797d7 + WORD $0x0e581857 + WORD $0x0e6898d7 + WORD $0x0e791957 + WORD $0x0ff999d7 + WORD $0x0fea1a57 + WORD $0x0fda9ad7 + WORD $0x0fcb1b57 + WORD $0x0fbb9bd7 + WORD $0x40730333 + WORD $0x007888b3 + WORD $0xe8031ce3 + WORD $0x43a017d7 + WORD $0x42a01757 + WORD $0x42b016d7 + WORD $0x42c01657 + WORD $0x42d015d7 + WORD $0x42e01557 + WORD $0x42f01057 + WORD $0x430010d7 + WORD $0x43101157 + WORD $0x432011d7 + WORD $0x43301257 + WORD $0x434012d7 + WORD $0x43501357 + WORD $0x436013d7 + WORD $0x43701857 + WORD $0x438018d7 + WORD $0x01050833 + WORD $0x01070293 + WORD $0x01183027 + WORD $0x01083427 + WORD $0x00783827 + WORD $0x00683c27 + WORD $0x02583027 + WORD $0x02483427 + WORD $0x02383827 + WORD $0x02283c27 + WORD $0x04183027 + WORD $0x04083427 + WORD $0x04a83827 + WORD $0x04b83c27 + WORD $0x06c83027 + WORD $0x06d83427 + WORD $0x06e83827 + WORD $0x06f83c27 + WORD $0xda56dae3 + WORD $0x0480006f + WORD $0xcd927057 + WORD $0x5e003657 + WORD $0xcda47057 + WORD $0x5e003457 + WORD $0x00050813 + WORD $0x02080893 + WORD $0xcd927057 + WORD $0x02087627 + WORD $0xcda47057 + WORD $0x0208f427 + WORD $0x06080893 + WORD $0xcd927057 + WORD $0x0208f627 + WORD $0x02070893 + WORD $0x01070713 + WORD $0x08080813 + WORD $0xfd16dae3 + WORD $0xc2202873 + WORD $0x00181813 + WORD $0x01010133 + WORD $0x01010113 + WORD $0x08d75e63 + WORD $0x06f05463 WORD $0x0d807857 WORD $0x42006457 WORD $0x00000893 @@ -840,8 +1702,11 @@ TEXT ·correlateFloat64RVV(SB), NOSPLIT|NOFRAME, $0-72 WORD $0x00170713 WORD $0x00f83027 WORD $0xfad716e3 - WORD $0x0280006f + WORD $0x0340006f WORD $0x00000593 + WORD $0x40e686b3 + WORD $0x00371713 + WORD $0x00e50533 WORD $0x0d96f657 WORD $0x00359713 WORD $0x00e50733 @@ -2067,14 +2932,261 @@ TEXT ·movingAverageFloat32RVV(SB), NOSPLIT|NOFRAME, $0-56 MOV dst_len+8(FP), X12 MOV a_len+32(FP), X13 MOV width+48(FP), X14 - WORD $0x06e05c63 - WORD $0x06e6ca63 + WORD $0x44e05a63 + WORD $0x44e6c863 WORD $0x40e686b3 WORD $0x00c6d463 WORD $0x00168613 - WORD $0x06c05263 + WORD $0x01000893 + WORD $0x01165863 + WORD $0x00000693 + WORD $0x3cc04c63 + WORD $0x4300006f + WORD $0xff010113 + WORD $0xc22026f3 + WORD $0x00c00793 + WORD $0x02f686b3 + WORD $0x40d10133 WORD $0x00000693 WORD $0xd03777d3 + WORD $0x00000813 + WORD $0x00269793 + WORD $0x00088693 + WORD $0x00f588b3 + WORD $0x0d0072d7 + WORD $0x42006457 + WORD $0x9e80b557 + WORD $0xc22022f3 + WORD $0x00a00313 + WORD $0x026282b3 + WORD $0x005102b3 + WORD $0x01028293 + WORD $0x22828427 + WORD $0xc22022f3 + WORD $0x00329293 + WORD $0x005102b3 + WORD $0x01028293 + WORD $0x22828427 + WORD $0x9e80b857 + WORD $0x9e80b957 + WORD $0x9e80ba57 + WORD $0x9e80bb57 + WORD $0x9e80bc57 + WORD $0x9e80bd57 + WORD $0x9e80be57 + WORD $0x9e80bf57 + WORD $0x9e80b357 + WORD $0x9e80b257 + WORD $0x9e80b157 + WORD $0x9e80b057 + WORD $0x9e80b657 + WORD $0x00070293 + WORD $0x0d12f357 + WORD $0x00281393 + WORD $0x007883b3 + WORD $0x0203e707 + WORD $0x00838e13 + WORD $0x020e6407 + WORD $0xc2202e73 + WORD $0x001e1e13 + WORD $0x01c10e33 + WORD $0x010e0e13 + WORD $0x228e0427 + WORD $0x01038e13 + WORD $0x020e6407 + WORD $0x01010e13 + WORD $0x228e0427 + WORD $0x03c38e13 + WORD $0x020e6407 + WORD $0xc2202e73 + WORD $0x00600e93 + WORD $0x03de0e33 + WORD $0x01c10e33 + WORD $0x010e0e13 + WORD $0x228e0427 + WORD $0x03838e13 + WORD $0x0ee61657 + WORD $0xc2202ef3 + WORD $0x002e9e93 + WORD $0x01d10eb3 + WORD $0x010e8e93 + WORD $0x228e8627 + WORD $0x020e6407 + WORD $0x03438e13 + WORD $0xc2202ef3 + WORD $0x00600f13 + WORD $0x03ee8eb3 + WORD $0x01d10eb3 + WORD $0x010e8e93 + WORD $0x228e8607 + WORD $0x0ec51557 + WORD $0xc2202ef3 + WORD $0x00600f13 + WORD $0x03ee8eb3 + WORD $0x01d10eb3 + WORD $0x010e8e93 + WORD $0x228e8527 + WORD $0x020e6507 + WORD $0x03038e13 + WORD $0x9e003657 + WORD $0x9e203057 + WORD $0x9e403157 + WORD $0x9e603257 + WORD $0x9fe03357 + WORD $0x9fc03f57 + WORD $0x9fa03e57 + WORD $0x9f803d57 + WORD $0x9f603c57 + WORD $0xc2202ef3 + WORD $0x00a00f13 + WORD $0x03ee8eb3 + WORD $0x01d10eb3 + WORD $0x010e8e93 + WORD $0x228e8707 + WORD $0x0e871757 + WORD $0xc2202ef3 + WORD $0x00a00f13 + WORD $0x03ee8eb3 + WORD $0x01d10eb3 + WORD $0x010e8e93 + WORD $0x228e8727 + WORD $0xc2202ef3 + WORD $0x003e9e93 + WORD $0x01d10eb3 + WORD $0x010e8e93 + WORD $0x228e8707 + WORD $0x9f803b57 + WORD $0x9fa03c57 + WORD $0x9fc03d57 + WORD $0x9fe03e57 + WORD $0x9e603f57 + WORD $0x9e403357 + WORD $0x9e203257 + WORD $0x9e003157 + WORD $0x9ec03057 + WORD $0xc2202ef3 + WORD $0x002e9e93 + WORD $0x01d10eb3 + WORD $0x010e8e93 + WORD $0x228e8607 + WORD $0x020e6407 + WORD $0x02c38e13 + WORD $0x0ea71757 + WORD $0xc2202ef3 + WORD $0x003e9e93 + WORD $0x01d10eb3 + WORD $0x010e8e93 + WORD $0x228e8727 + WORD $0x020e6507 + WORD $0x02838e13 + WORD $0x0e881857 + WORD $0x020e6407 + WORD $0x02438e13 + WORD $0x0ea91957 + WORD $0x020e6507 + WORD $0x02038e13 + WORD $0x0e8a1a57 + WORD $0x020e6407 + WORD $0x01c38e13 + WORD $0x0eab1b57 + WORD $0x020e6507 + WORD $0x01838e13 + WORD $0x0e8c1c57 + WORD $0x020e6407 + WORD $0x01438e13 + WORD $0x0ead1d57 + WORD $0x020e6507 + WORD $0x00438e13 + WORD $0x00c38393 + WORD $0x0e8e1e57 + WORD $0x0203e407 + WORD $0x0eaf1f57 + WORD $0x020e6507 + WORD $0x01010393 + WORD $0x22838707 + WORD $0x0ee31357 + WORD $0x0e821257 + WORD $0xc22023f3 + WORD $0x00139393 + WORD $0x007103b3 + WORD $0x01038393 + WORD $0x22838407 + WORD $0x0e811157 + WORD $0x0ea01057 + WORD $0xc22023f3 + WORD $0x00600e13 + WORD $0x03c383b3 + WORD $0x007103b3 + WORD $0x01038393 + WORD $0x22838507 + WORD $0x406282b3 + WORD $0x00680833 + WORD $0xdc0298e3 + WORD $0x42a01757 + WORD $0xc2202873 + WORD $0x00a00893 + WORD $0x03180833 + WORD $0x01010833 + WORD $0x01080813 + WORD $0x22880407 + WORD $0x428016d7 + WORD $0xc2202873 + WORD $0x00381813 + WORD $0x01010833 + WORD $0x01080813 + WORD $0x22880407 + WORD $0x42801657 + WORD $0x430015d7 + WORD $0x43201557 + WORD $0x43401057 + WORD $0x436010d7 + WORD $0x43801157 + WORD $0x43a011d7 + WORD $0x43c01257 + WORD $0x43e012d7 + WORD $0x42601357 + WORD $0x424013d7 + WORD $0x42201857 + WORD $0x420018d7 + WORD $0x42c01e57 + WORD $0xcd027057 + WORD $0x5e0e5457 + WORD $0x5e0354d7 + WORD $0x5e015557 + WORD $0x5e05d5d7 + WORD $0x3e88d457 + WORD $0x3e92d4d7 + WORD $0x3ea0d557 + WORD $0x3eb655d7 + WORD $0x3e885457 + WORD $0x3e9254d7 + WORD $0x3ea05557 + WORD $0x3eb6d5d7 + WORD $0x3e91d4d7 + WORD $0x8297d4d7 + WORD $0x3ea55557 + WORD $0x82a7d557 + WORD $0x3e83d457 + WORD $0x8287d457 + WORD $0x3eb755d7 + WORD $0x82b7d5d7 + WORD $0x00f507b3 + WORD $0x01078813 + WORD $0x020864a7 + WORD $0x02078813 + WORD $0x02086527 + WORD $0x01068893 + WORD $0x0207e427 + WORD $0x03078793 + WORD $0x0207e5a7 + WORD $0xc71654e3 + WORD $0xc22027f3 + WORD $0x00c00813 + WORD $0x030787b3 + WORD $0x00f10133 + WORD $0x01010113 + WORD $0x06c6d063 + WORD $0xd03777d3 WORD $0x0d0077d7 WORD $0x42006457 WORD $0x00000813 @@ -2161,14 +3273,265 @@ TEXT ·movingAverageFloat64RVV(SB), NOSPLIT|NOFRAME, $0-56 MOV dst_len+8(FP), X12 MOV a_len+32(FP), X13 MOV width+48(FP), X14 - WORD $0x06e05c63 - WORD $0x06e6ca63 + WORD $0x46e05263 + WORD $0x46e6c063 WORD $0x40e686b3 WORD $0x00c6d463 WORD $0x00168613 - WORD $0x06c05263 + WORD $0x01000893 + WORD $0x01165863 + WORD $0x00000693 + WORD $0x3ec04463 + WORD $0x4400006f + WORD $0xff010113 + WORD $0xc22026f3 + WORD $0x00c00793 + WORD $0x02f686b3 + WORD $0x40d10133 WORD $0x00000693 WORD $0xd23777d3 + WORD $0x00000813 + WORD $0x00369793 + WORD $0x00088693 + WORD $0x00f588b3 + WORD $0x0d8072d7 + WORD $0x42006457 + WORD $0x9e80b557 + WORD $0xc22022f3 + WORD $0x00a00313 + WORD $0x026282b3 + WORD $0x005102b3 + WORD $0x01028293 + WORD $0x22828427 + WORD $0xc22022f3 + WORD $0x00329293 + WORD $0x005102b3 + WORD $0x01028293 + WORD $0x22828427 + WORD $0x9e80b857 + WORD $0x9e80b957 + WORD $0x9e80ba57 + WORD $0x9e80bb57 + WORD $0x9e80bc57 + WORD $0x9e80bd57 + WORD $0x9e80be57 + WORD $0x9e80bf57 + WORD $0x9e80b357 + WORD $0x9e80b257 + WORD $0x9e80b157 + WORD $0x9e80b057 + WORD $0x9e80b657 + WORD $0x00070293 + WORD $0x0d92f357 + WORD $0x00381393 + WORD $0x007883b3 + WORD $0x0203f707 + WORD $0x01038e13 + WORD $0x020e7407 + WORD $0xc2202e73 + WORD $0x001e1e13 + WORD $0x01c10e33 + WORD $0x010e0e13 + WORD $0x228e0427 + WORD $0x02038e13 + WORD $0x020e7407 + WORD $0x01010e13 + WORD $0x228e0427 + WORD $0x07838e13 + WORD $0x020e7407 + WORD $0xc2202e73 + WORD $0x00600e93 + WORD $0x03de0e33 + WORD $0x01c10e33 + WORD $0x010e0e13 + WORD $0x228e0427 + WORD $0x07038e13 + WORD $0x0ee61657 + WORD $0xc2202ef3 + WORD $0x002e9e93 + WORD $0x01d10eb3 + WORD $0x010e8e93 + WORD $0x228e8627 + WORD $0x020e7407 + WORD $0x06838e13 + WORD $0xc2202ef3 + WORD $0x00600f13 + WORD $0x03ee8eb3 + WORD $0x01d10eb3 + WORD $0x010e8e93 + WORD $0x228e8607 + WORD $0x0ec51557 + WORD $0xc2202ef3 + WORD $0x00600f13 + WORD $0x03ee8eb3 + WORD $0x01d10eb3 + WORD $0x010e8e93 + WORD $0x228e8527 + WORD $0x020e7507 + WORD $0x06038e13 + WORD $0x9e003657 + WORD $0x9e203057 + WORD $0x9e403157 + WORD $0x9e603257 + WORD $0x9fe03357 + WORD $0x9fc03f57 + WORD $0x9fa03e57 + WORD $0x9f803d57 + WORD $0x9f603c57 + WORD $0xc2202ef3 + WORD $0x00a00f13 + WORD $0x03ee8eb3 + WORD $0x01d10eb3 + WORD $0x010e8e93 + WORD $0x228e8707 + WORD $0x0e871757 + WORD $0xc2202ef3 + WORD $0x00a00f13 + WORD $0x03ee8eb3 + WORD $0x01d10eb3 + WORD $0x010e8e93 + WORD $0x228e8727 + WORD $0xc2202ef3 + WORD $0x003e9e93 + WORD $0x01d10eb3 + WORD $0x010e8e93 + WORD $0x228e8707 + WORD $0x9f803b57 + WORD $0x9fa03c57 + WORD $0x9fc03d57 + WORD $0x9fe03e57 + WORD $0x9e603f57 + WORD $0x9e403357 + WORD $0x9e203257 + WORD $0x9e003157 + WORD $0x9ec03057 + WORD $0xc2202ef3 + WORD $0x002e9e93 + WORD $0x01d10eb3 + WORD $0x010e8e93 + WORD $0x228e8607 + WORD $0x020e7407 + WORD $0x05838e13 + WORD $0x0ea71757 + WORD $0xc2202ef3 + WORD $0x003e9e93 + WORD $0x01d10eb3 + WORD $0x010e8e93 + WORD $0x228e8727 + WORD $0x020e7507 + WORD $0x05038e13 + WORD $0x0e881857 + WORD $0x020e7407 + WORD $0x04838e13 + WORD $0x0ea91957 + WORD $0x020e7507 + WORD $0x04038e13 + WORD $0x0e8a1a57 + WORD $0x020e7407 + WORD $0x03838e13 + WORD $0x0eab1b57 + WORD $0x020e7507 + WORD $0x03038e13 + WORD $0x0e8c1c57 + WORD $0x020e7407 + WORD $0x02838e13 + WORD $0x0ead1d57 + WORD $0x020e7507 + WORD $0x00838e13 + WORD $0x01838393 + WORD $0x0e8e1e57 + WORD $0x0203f407 + WORD $0x0eaf1f57 + WORD $0x020e7507 + WORD $0x01010393 + WORD $0x22838707 + WORD $0x0ee31357 + WORD $0x0e821257 + WORD $0xc22023f3 + WORD $0x00139393 + WORD $0x007103b3 + WORD $0x01038393 + WORD $0x22838407 + WORD $0x0e811157 + WORD $0x0ea01057 + WORD $0xc22023f3 + WORD $0x00600e13 + WORD $0x03c383b3 + WORD $0x007103b3 + WORD $0x01038393 + WORD $0x22838507 + WORD $0x406282b3 + WORD $0x00680833 + WORD $0xdc0298e3 + WORD $0x42a01757 + WORD $0xc2202873 + WORD $0x00a00893 + WORD $0x03180833 + WORD $0x01010833 + WORD $0x01080813 + WORD $0x22880407 + WORD $0x428016d7 + WORD $0xc2202873 + WORD $0x00381813 + WORD $0x01010833 + WORD $0x01080813 + WORD $0x22880407 + WORD $0x42801657 + WORD $0x430015d7 + WORD $0x43201557 + WORD $0x43401057 + WORD $0x436010d7 + WORD $0x43801157 + WORD $0x43a011d7 + WORD $0x43c01257 + WORD $0x43e012d7 + WORD $0x42601357 + WORD $0x424013d7 + WORD $0x42201857 + WORD $0x420018d7 + WORD $0x42c01e57 + WORD $0x00f507b3 + WORD $0x01068893 + WORD $0x1afe7e53 + WORD $0x1af8f8d3 + WORD $0x1af87853 + WORD $0x1af3f3d3 + WORD $0x1af37353 + WORD $0x1af2f2d3 + WORD $0x1af27253 + WORD $0x1af1f1d3 + WORD $0x1af17153 + WORD $0x1af0f0d3 + WORD $0x1af07053 + WORD $0x1af57553 + WORD $0x1af5f5d3 + WORD $0x1af67653 + WORD $0x1af6f6d3 + WORD $0x1af77753 + WORD $0x01c7b027 + WORD $0x0117b427 + WORD $0x0107b827 + WORD $0x0077bc27 + WORD $0x0267b027 + WORD $0x0257b427 + WORD $0x0247b827 + WORD $0x0237bc27 + WORD $0x0427b027 + WORD $0x0417b427 + WORD $0x0407b827 + WORD $0x04a7bc27 + WORD $0x06b7b027 + WORD $0x06c7b427 + WORD $0x06d7b827 + WORD $0x06e7bc27 + WORD $0xc5165ce3 + WORD $0xc22027f3 + WORD $0x00c00813 + WORD $0x030787b3 + WORD $0x00f10133 + WORD $0x01010113 + WORD $0x06c6d063 + WORD $0xd23777d3 WORD $0x0d8077d7 WORD $0x42006457 WORD $0x00000813 diff --git a/internal/tests/docs/kernels_guide_test.go b/internal/tests/docs/kernels_guide_test.go new file mode 100644 index 0000000..33457be --- /dev/null +++ b/internal/tests/docs/kernels_guide_test.go @@ -0,0 +1,171 @@ +package docs + +import ( + "go/ast" + "go/parser" + "go/token" + "os" + "path/filepath" + "regexp" + "strings" + "testing" +) + +// docs/kernels.md's worked example names things that have to exist. +// +// The guide is the answer to "the operation I need isn't in the 403", and its +// centre is one `spec.Kernel` literal with eight fields, each annotated with +// what it is for. A reader copies that literal. If a field is renamed the +// guide goes on naming the old one, and the person following it gets a +// compile error in generated code they did not write and cannot easily read +// -- which is the moment the guide exists to prevent. +// +// The link gate and the test-name gate already cover this document. Neither +// looks inside a code block. +// +// THE FIELD LIST IS READ FROM THE DOCUMENT, not written here. A list in this +// file would have to be kept in step with the guide by hand, which is the +// thing being gated, one level up. +func TestTheKernelGuidesManifestFieldsExist(t *testing.T) { + doc := readDoc(t, filepath.Join("docs", "kernels.md")) + + // The one block that shows a manifest entry, from `spec.Kernel{` to the + // fence that ends it. + block := regexp.MustCompile("(?s)```go\\s*\nspec\\.Kernel\\{(.*?)\n```").FindStringSubmatch(doc) + if block == nil { + t.Fatal("docs/kernels.md no longer contains a ```go block starting " + + "`spec.Kernel{`. That literal is the guide's worked example and " + + "this gate reads it; if the example moved, repoint the gate rather " + + "than leaving it green over nothing") + } + + key := regexp.MustCompile(`(?m)^\s*([A-Z][A-Za-z0-9]*):`) + var named []string + for _, m := range key.FindAllStringSubmatch(block[1], -1) { + named = append(named, m[1]) + } + if len(named) < 5 { + t.Fatalf("the worked example names %d fields (%v); it documented eight, "+ + "so the extraction has lost the block rather than the block having "+ + "shrunk", len(named), named) + } + + fields := structFields(t, filepath.Join("tools", "simdgen", "spec", "spec.go"), "Kernel") + if len(fields) < 5 { + t.Fatalf("spec.Kernel parsed to %d fields; the parse has lost the "+ + "struct", len(fields)) + } + for _, n := range named { + if !fields[n] { + t.Errorf("docs/kernels.md's worked example sets spec.Kernel.%s, "+ + "which the struct does not declare. Someone following the guide "+ + "writes a manifest entry that does not compile", n) + } + } + + // AND THE TWO HAND-WRITTEN WIRINGS THE EXAMPLE POINTS AT. + // + // `RefFunc` and `Field` are the two files `docs/plans` wants generated + // precisely because they are the two easiest things to forget when adding a + // kernel -- forgetting the ref wiring shipped past the whole normal lane + // once. The guide names a REAL kernel for its example, so both ends of it + // can be checked rather than described. + lit := regexp.MustCompile(`(?m)^\s*(RefFunc|Field):\s*"([A-Za-z0-9]+)"`) + seen := 0 + for _, m := range lit.FindAllStringSubmatch(block[1], -1) { + seen++ + switch m[1] { + case "RefFunc": + if !refExports(t)[m[2]] { + t.Errorf("the guide's example declares RefFunc %q, which "+ + "internal/ref does not export. The guard would fall back to "+ + "a function that does not exist", m[2]) + } + case "Field": + if !kernelOpsFields(t)[m[2]] { + t.Errorf("the guide's example declares Field %q, which "+ + "internal/kernel does not declare on any kernel group. That "+ + "is the wiring a new kernel is most often missing", m[2]) + } + } + } + if seen != 2 { + t.Fatalf("found %d of the example's RefFunc/Field literals, want 2; "+ + "the two assertions above then covered less than they claim", seen) + } +} + +// structFields is the field names of one struct in one file, multi-name +// declarations (`Group, Field string`) included -- which is how two of the +// eight are declared, so a parser that took one name per line would report +// the guide as wrong about fields that exist. +// +// Parsed rather than imported: `tools/` is a separate module and must never +// become a consumer dependency, which is a rule in CLAUDE.md. +func structFields(t *testing.T, rel, name string) map[string]bool { + t.Helper() + f, err := parser.ParseFile(token.NewFileSet(), path(rel), nil, 0) + if err != nil { + t.Fatalf("parsing %s: %v", rel, err) + } + out := map[string]bool{} + ast.Inspect(f, func(n ast.Node) bool { + ts, ok := n.(*ast.TypeSpec) + if !ok || ts.Name.Name != name { + return true + } + st, ok := ts.Type.(*ast.StructType) + if !ok { + return true + } + for _, fld := range st.Fields.List { + for _, id := range fld.Names { + out[id.Name] = true + } + } + return false + }) + return out +} + +// refExports is every exported function internal/ref declares. +func refExports(t *testing.T) map[string]bool { + t.Helper() + return declaredIn(t, filepath.Join("internal", "ref"), + regexp.MustCompile(`(?m)^func ([A-Z][A-Za-z0-9]*)`)) +} + +// kernelOpsFields is every function-typed field on any group in +// internal/kernel -- the `Field` half of a manifest entry. +func kernelOpsFields(t *testing.T) map[string]bool { + t.Helper() + return declaredIn(t, filepath.Join("internal", "kernel"), + regexp.MustCompile(`(?m)^\t([A-Z][A-Za-z0-9]*)\s+func\(`)) +} + +func declaredIn(t *testing.T, dir string, re *regexp.Regexp) map[string]bool { + t.Helper() + out := map[string]bool{} + matches, err := filepath.Glob(filepath.Join(path(dir), "*.go")) + if err != nil { + t.Fatal(err) + } + for _, p := range matches { + if strings.HasSuffix(p, "_test.go") { + continue + } + b, err := os.ReadFile(p) + if err != nil { + t.Fatal(err) + } + for _, m := range re.FindAllStringSubmatch(string(b), -1) { + out[m[1]] = true + } + } + if len(out) == 0 { + t.Fatalf("no declarations matched in %s; the pattern has stopped "+ + "finding them and every lookup against this set would pass or fail "+ + "for the wrong reason", dir) + } + return out +} diff --git a/internal/tests/docs/test_names_test.go b/internal/tests/docs/test_names_test.go new file mode 100644 index 0000000..b751aed --- /dev/null +++ b/internal/tests/docs/test_names_test.go @@ -0,0 +1,105 @@ +package docs + +import ( + "os" + "path/filepath" + "regexp" + "strings" + "testing" +) + +// A document naming a test is claiming a guarantee is enforced. +// +// It is the same promise as a link and it rots the same way: a test is +// renamed, the document goes on naming the old one, and a reader who greps +// for it finds nothing and cannot tell whether the guarantee was dropped or +// the name moved. Both are worse than no sentence. +// +// The subject is activeDocs — the set links_test.go checks, for the reason +// stated there. CHANGELOG.md, docs/wrong.md, docs/plans and docs/research are +// historical or external: a changelog entry describing what a release did is +// accurate about that release even after a rename, and docs/research quotes +// benchmark names from other people's blog posts (`BenchmarkNoopAsm` is +// mmcloughlin's, not this repository's). Gating those would force history to +// be rewritten to satisfy a checker, which is the thing docs/wrong.md exists +// to prevent. +func TestActiveDocsNameRealTests(t *testing.T) { + declared := declaredTestFuncs(t) + if len(declared) == 0 { + t.Fatal("no test functions were found in the tree, so this gate measures nothing") + } + + cited := regexp.MustCompile("`((?:Test|Fuzz|Benchmark|Example)[A-Za-z0-9_]+)`") + checked := 0 + for _, doc := range testNameDocs() { + b, err := os.ReadFile(filepath.Join(repoRoot, doc)) + if err != nil { + t.Fatalf("%s: %v", doc, err) + } + for i, line := range strings.Split(string(b), "\n") { + for _, m := range cited.FindAllStringSubmatch(line, -1) { + checked++ + if !declared[m[1]] { + t.Errorf("%s:%d names %s, which no test in this repository "+ + "declares: the guarantee on that line is prose", + doc, i+1, m[1]) + } + } + } + } + if checked == 0 { + t.Fatal("no test names were cited in any active document, so this gate " + + "measures nothing -- if the citations were deliberately removed, " + + "delete this test rather than leaving it green over nothing") + } + t.Logf("%d citations across %d documents, all declared", checked, len(testNameDocs())) +} + +// testNameDocs is activeDocs plus the two documents that describe the SUITE +// itself, which is where most citations naturally are. +// +// Not by widening activeDocs: CLAUDE.md describes that set as what the LINK +// gate covers -- README, CONTRIBUTING, ROADMAP and the main references and +// guides -- and says the LLDs, plans and records are checked by hand. This +// gate has its own subject rather than quietly changing the meaning of that +// sentence. The LLD citations were checked by hand on 2026-08-16 and all +// resolve. +func testNameDocs() []string { + return append(append([]string(nil), activeDocs...), + "docs/verification.md", + "internal/tests/README.md", + ) +} + +// declaredTestFuncs is every Test/Fuzz/Benchmark/Example function in the tree. +func declaredTestFuncs(t *testing.T) map[string]bool { + t.Helper() + decl := regexp.MustCompile(`(?m)^func ((?:Test|Fuzz|Benchmark|Example)[A-Za-z0-9_]*)\(`) + out := map[string]bool{} + err := filepath.WalkDir(repoRoot, func(path string, d os.DirEntry, err error) error { + if err != nil { + return err + } + if d.IsDir() { + if n := d.Name(); n == ".git" || n == "node_modules" { + return filepath.SkipDir + } + return nil + } + if !strings.HasSuffix(path, "_test.go") { + return nil + } + b, err := os.ReadFile(path) + if err != nil { + return err + } + for _, m := range decl.FindAllStringSubmatch(string(b), -1) { + out[m[1]] = true + } + return nil + }) + if err != nil { + t.Fatal(err) + } + return out +} diff --git a/tools/simdgen/emit/dispatch_gen.go b/tools/simdgen/emit/dispatch_gen.go index 0782a1d..8148f5e 100644 --- a/tools/simdgen/emit/dispatch_gen.go +++ b/tools/simdgen/emit/dispatch_gen.go @@ -62,6 +62,21 @@ func sigTypes(k spec.Kernel) string { return sig } +// complexElem maps a complex group name to its Go element type, and +// partsElem the split groups that pair a complex type with its real one. +// +// These were absent, and their absence was the whole defect: numericGroupsIn +// skipped every complex group, so no per-tier table was emitted for them and +// complexOps had nothing to overlay from. Nine tiers of generated complex +// assembly across six architectures were linked into the test-only Sets() +// aggregator and reachable from nothing a consumer calls. +var complexElem = map[string]string{"C64": "complex64", "C128": "complex128"} + +var partsElem = map[string][2]string{ + "C64Parts": {"complex64", "float32"}, + "C128Parts": {"complex128", "float64"}, +} + // numericElem maps a numeric group name to its Go element type. var numericElem = map[string]string{ "F32": "float32", "F64": "float64", @@ -142,6 +157,34 @@ func RootDispatch(arch string, tiers []string, present map[string]map[string]boo fmt.Fprintf(&b, "}\n\n") } + // Complex groups, the same shape over kernel.Complex and + // kernel.ComplexParts. Kept separate from the numeric loop because the + // struct type differs and ComplexParts carries two type parameters. + for _, g := range groupsIn(all, func(name string) bool { _, ok := complexElem[name]; return ok }) { + emitTierTable(&b, arch, tiers, present, all, g, + "kernel.Complex["+complexElem[g]+"]", "cplx"+g) + } + for _, g := range groupsIn(all, func(name string) bool { _, ok := partsElem[name]; return ok }) { + e := partsElem[g] + emitTierTable(&b, arch, tiers, present, all, g, + "kernel.ComplexParts["+e[0]+", "+e[1]+"]", "parts"+g) + } + + // Every group in the manifest must have produced a table. A group this + // file does not know about silently falls back to the reference on every + // machine, with every test green -- which is exactly what happened to + // C64/C128/C64Parts/C128Parts for the whole life of the complex kernels. + // The allowlists above are the same shape that let it happen, so the + // check is here rather than in a test: a generator that cannot dispatch + // a group must fail loudly instead of emitting a working-looking file. + if missing := ungrouped(all); len(missing) > 0 { + panic(fmt.Sprintf("simdgen: no dispatch table for kernel group(s) %v.\n"+ + "Add the group to numericElem, complexElem, partsElem or flatGroups in "+ + "dispatch_gen.go. Without an entry the group compiles, passes every "+ + "test, and runs the portable reference on every architecture.", + missing)) + } + out := b.String() if formatted, err := format.Source([]byte(out)); err == nil { return string(formatted) @@ -149,11 +192,78 @@ func RootDispatch(arch string, tiers []string, present map[string]map[string]boo return out } +// ungrouped returns the manifest groups no table is emitted for, sorted. +func ungrouped(all []spec.Kernel) []string { + bad := map[string]bool{} + for _, k := range all { + if flatGroups[k.Group] { + continue + } + if _, ok := numericElem[k.Group]; ok { + continue + } + if _, ok := complexElem[k.Group]; ok { + continue + } + if _, ok := partsElem[k.Group]; ok { + continue + } + bad[k.Group] = true + } + out := make([]string, 0, len(bad)) + for g := range bad { + out = append(out, g) + } + sort.Strings(out) + return out +} + // exportedGuard mirrors guardName: the exported threshold wrapper. func exportedGuard(k spec.Kernel) string { return strings.ToUpper(k.GoName[:1]) + k.GoName[1:] } +// emitTierTable writes one partial struct per tier plus the tier-indexed +// pointer array the dispatcher overlays from, for any group whose kernel.Set +// field is a struct of function fields. Slot 0 is nil: the reference base +// needs no overlay. +func emitTierTable(b *strings.Builder, arch string, tiers []string, + present map[string]map[string]bool, all []spec.Kernel, group, typ, prefix string) { + + for _, t := range tiers { + suffix := strings.ToUpper(t) + fmt.Fprintf(b, "var %s%s = %s{\n", prefix, suffix, typ) + for _, k := range all { + if k.Group != group || !present[t][k.CName] { + continue + } + fmt.Fprintf(b, "\t%s: %s.%s%s,\n", k.Field, arch, exportedGuard(k), suffix) + } + fmt.Fprintf(b, "}\n\n") + } + fmt.Fprintf(b, "var %sByTier = [...]*%s{\n\tnil,\n", prefix, typ) + for _, t := range tiers { + fmt.Fprintf(b, "\t&%s%s,\n", prefix, strings.ToUpper(t)) + } + fmt.Fprintf(b, "}\n\n") +} + +// groupsIn returns the sorted group names in all that keep matches true. +func groupsIn(all []spec.Kernel, keep func(string) bool) []string { + seen := map[string]bool{} + for _, k := range all { + if keep(k.Group) { + seen[k.Group] = true + } + } + out := make([]string, 0, len(seen)) + for g := range seen { + out = append(out, g) + } + sort.Strings(out) + return out +} + func numericGroupsIn(all []spec.Kernel) []string { seen := map[string]bool{} for _, k := range all { @@ -197,6 +307,13 @@ func RootDispatchFallback(arches []string, all []spec.Kernel, prov Provenance) s for _, g := range numericGroupsIn(all) { fmt.Fprintf(&b, "var ops%sByTier = [...]*kernel.Ops[%s]{nil}\n", g, numericElem[g]) } + for _, g := range groupsIn(all, func(n string) bool { _, ok := complexElem[n]; return ok }) { + fmt.Fprintf(&b, "var cplx%sByTier = [...]*kernel.Complex[%s]{nil}\n", g, complexElem[g]) + } + for _, g := range groupsIn(all, func(n string) bool { _, ok := partsElem[n]; return ok }) { + e := partsElem[g] + fmt.Fprintf(&b, "var parts%sByTier = [...]*kernel.ComplexParts[%s, %s]{nil}\n", g, e[0], e[1]) + } out := b.String() if formatted, err := format.Source([]byte(out)); err == nil { @@ -284,6 +401,35 @@ func RootDispatchTest(arch string, tiers []string, all []spec.Kernel, prov Prove fmt.Fprintf(&b, "},\n") } fmt.Fprintf(&b, "}\n\n") + + // The struct groups -- numeric and complex -- reach the runtime through + // per-tier partials, not per-operation tables, so allFlatTables cannot + // name them and the inventory walk fell back to archSets(), the test-only + // aggregator. That is why four whole complex groups could be absent from + // the runtime tables with the suite green. These expose the partials the + // runtime actually indexes, keyed by group, so a test can walk them. + fmt.Fprintf(&b, "// allGroupTables enumerates the per-tier partial structs the runtime\n") + fmt.Fprintf(&b, "// overlays from, keyed by kernel.Set group. Slot 0 is the reference\n") + fmt.Fprintf(&b, "// and is nil; the rest are one per accelerated tier, in tier order.\n") + fmt.Fprintf(&b, "var allGroupTables = map[string][]any{\n") + emitGroupRow := func(group, prefix string) { + fmt.Fprintf(&b, "\t%q: {nil", group) + for _, t := range tiers { + fmt.Fprintf(&b, ", %s%s", prefix, strings.ToUpper(t)) + } + fmt.Fprintf(&b, "},\n") + } + for _, g := range numericGroupsIn(all) { + emitGroupRow(g, "ops"+g) + } + for _, g := range groupsIn(all, func(n string) bool { _, ok := complexElem[n]; return ok }) { + emitGroupRow(g, "cplx"+g) + } + for _, g := range groupsIn(all, func(n string) bool { _, ok := partsElem[n]; return ok }) { + emitGroupRow(g, "parts"+g) + } + fmt.Fprintf(&b, "}\n\n") + fmt.Fprintf(&b, "// archSets hands the tests every tier's complete kernel set.\n") fmt.Fprintf(&b, "var archSets func() map[string]kernel.Set = %s.Sets\n", arch) @@ -315,6 +461,9 @@ func RootDispatchFallbackTest(arches []string, all []spec.Kernel, prov Provenanc fmt.Fprintf(&b, "\t%q: {%s[0]},\n", k.Group+"."+k.Field, TableName(k)) } fmt.Fprintf(&b, "}\n\n") + fmt.Fprintf(&b, "// No generated backend here: every group table is the reference slot\n") + fmt.Fprintf(&b, "// alone, so the walk has nothing to check.\n") + fmt.Fprintf(&b, "var allGroupTables = map[string][]any{}\n\n") fmt.Fprintf(&b, "// No generated backend here: the numeric half of the inventory walk\n") fmt.Fprintf(&b, "// has nothing to check and skips on nil.\n") fmt.Fprintf(&b, "var archSets func() map[string]kernel.Set\n") diff --git a/tools/simdgen/emit/ungrouped_test.go b/tools/simdgen/emit/ungrouped_test.go new file mode 100644 index 0000000..9028572 --- /dev/null +++ b/tools/simdgen/emit/ungrouped_test.go @@ -0,0 +1,26 @@ +package emit + +import ( + "testing" + + "github.com/sebishogun/simd/tools/simdgen/spec" +) + +// A group with no dispatch table falls back to the reference on every +// machine with every test green. The generator must refuse to emit rather +// than produce a working-looking file -- this is the check that would have +// caught the complex groups. +func TestUngroupedGroupIsRefused(t *testing.T) { + if got := ungrouped(nil); len(got) != 0 { + t.Fatalf("an empty manifest reported %v", got) + } + for _, g := range []string{"F32", "C128", "C64Parts", "Bytes"} { + if got := ungrouped([]spec.Kernel{{Group: g}}); len(got) != 0 { + t.Errorf("known group %s reported as ungrouped: %v", g, got) + } + } + got := ungrouped([]spec.Kernel{{Group: "C32"}, {Group: "F32"}}) + if len(got) != 1 || got[0] != "C32" { + t.Fatalf("ungrouped = %v, want [C32]", got) + } +} diff --git a/tools/simdgen/main.go b/tools/simdgen/main.go index c1a36a2..96575c3 100644 --- a/tools/simdgen/main.go +++ b/tools/simdgen/main.go @@ -285,6 +285,19 @@ func build(cc *compile.Clang, src kernels.Source, tgt target.Target, root, outDi if verbose { fmt.Printf(" %s\n", r.Summary()) } + // A kernel that does arithmetic and none of it in lanes is printed + // whether or not the run is verbose, because it is exactly what + // RequireVector exists to catch and exactly what it cannot see: on + // amd64 every scalar float instruction lives in %xmm, so the "does any + // instruction touch a vector register" test passes for scalar code. + // Eight kernels ship this way today -- convolve, correlate, movavg and + // polyeval in both precisions -- and docs/wrong.md entry 76 has the + // disassembly. Not fatal and not dropped: falling back to internal/ref + // instead is a behaviour change that wants a benchmark first. + if r.ScalarOnly() { + fmt.Printf(" SCALAR-ONLY %s: %d scalar arithmetic instructions, "+ + "no packed ones\n", r.Func, r.ScalarArith) + } for _, p := range r.Problems { problems = append(problems, fmt.Sprintf("%s: %s", r.Func, p)) } diff --git a/tools/simdgen/verify/arith_test.go b/tools/simdgen/verify/arith_test.go new file mode 100644 index 0000000..5e5f99f --- /dev/null +++ b/tools/simdgen/verify/arith_test.go @@ -0,0 +1,108 @@ +package verify + +import ( + "testing" + + "github.com/sebishogun/simd/tools/simdgen/target" +) + +// arithKind separates work from moves, and packed work from scalar work. +// +// The mnemonics below are copied from llvm-objdump output for kernels in this +// repository, not invented: the amd64 scalar cases are what simd_convolve_f32 +// actually contains at sse2 and avx2, and the "neither" cases are the two +// instructions that made it pass as vectorized. +func TestArithKindSeparatesPackedFromScalarAndFromMoves(t *testing.T) { + for _, tc := range []struct { + arch target.Arch + mnemonic string + operands string + want arithForm + }{ + // amd64, scalar: one lane, in an %xmm register, which is why + // vectorWidth counts them as vector. + {target.AMD64, "mulss", "(%rsi), %xmm0", arithScalar}, + {target.AMD64, "addss", "%xmm1, %xmm0", arithScalar}, + {target.AMD64, "vmulss", "%xmm2, %xmm1, %xmm0", arithScalar}, + {target.AMD64, "vaddss", "%xmm2, %xmm1, %xmm0", arithScalar}, + {target.AMD64, "mulsd", "%xmm1, %xmm0", arithScalar}, + {target.AMD64, "vfmadd213sd", "%xmm2, %xmm1, %xmm0", arithScalar}, + {target.AMD64, "divss", "%xmm1, %xmm0", arithScalar}, + {target.AMD64, "maxsd", "%xmm1, %xmm0", arithScalar}, + + // amd64, packed. + {target.AMD64, "mulps", "%xmm1, %xmm0", arithPacked}, + {target.AMD64, "vaddps", "%ymm2, %ymm1, %ymm0", arithPacked}, + {target.AMD64, "vfmadd213ps", "%zmm2, %zmm1, %zmm0", arithPacked}, + {target.AMD64, "vmulpd", "%ymm2, %ymm1, %ymm0", arithPacked}, + {target.AMD64, "paddd", "%xmm1, %xmm0", arithPacked}, + {target.AMD64, "vpmulld", "%ymm2, %ymm1, %ymm0", arithPacked}, + {target.AMD64, "vpmaxsd", "%ymm2, %ymm1, %ymm0", arithPacked}, + {target.AMD64, "vpsubb", "%xmm2, %xmm1, %xmm0", arithPacked}, + + // amd64, NEITHER. These two are the whole finding: they are the only + // vector-register instructions in simd_convolve_f32 at sse2, and + // counting them is what let a scalar kernel ship as accelerated. + {target.AMD64, "movups", "(%rdi), %xmm0", arithNone}, + {target.AMD64, "xorps", "%xmm0, %xmm0", arithNone}, + {target.AMD64, "vmovups", "(%rdi), %ymm0", arithNone}, + {target.AMD64, "vmovss", "(%rsi), %xmm0", arithNone}, + {target.AMD64, "movq", "%rax, %rcx", arithNone}, + {target.AMD64, "vshufps", "$0, %xmm0, %xmm0, %xmm0", arithNone}, + + // arm64: the OPERAND decides, because `fmul` spells both forms. + {target.ARM64, "fmul", "v0.4s, v1.4s, v2.4s", arithPacked}, + {target.ARM64, "fadd", "v0.2d, v1.2d, v2.2d", arithPacked}, + {target.ARM64, "fmul", "s0, s1, s2", arithScalar}, + {target.ARM64, "fadd", "d0, d1, d2", arithScalar}, + {target.ARM64, "fmul", "z0.s, p0/m, z0.s, z1.s", arithPacked}, + {target.ARM64, "ldr", "q0, [x0]", arithNone}, + {target.ARM64, "mov", "v0.16b, v1.16b", arithNone}, + + // An architecture the classifier does not claim: everything is + // unclassified, and ArithKnown is what says so. + {target.PPC64LE, "xvmuldp", "vs0, vs1, vs2", arithNone}, + {target.S390X, "vfmdb", "%v0, %v1, %v2", arithNone}, + } { + got := arithKind(Instr{Mnemonic: tc.mnemonic, Operands: tc.operands}, tc.arch) + if got != tc.want { + t.Errorf("%s %s %q = %v, want %v", tc.arch, tc.mnemonic, tc.operands, got, tc.want) + } + } +} + +// ScalarOnly is false where the classifier does not run, so a zero count is +// never read as a finding. +// +// Without this, adding an architecture to the report and forgetting to add it +// to arithKind would mark every one of its kernels scalar-only -- a check that +// fires everywhere is as useless as one that fires nowhere, and this one would +// have been noise on four of the six architectures. +func TestScalarOnlyIsSilentWhereTheClassifierDoesNotRun(t *testing.T) { + for _, arch := range []target.Arch{target.AMD64, target.ARM64, target.PPC64LE, + target.S390X, target.RISCV64, target.LOONG64} { + known := arithClassified(arch) + r := Report{ArithKnown: known, ScalarArith: 10, PackedArith: 0} + if got := r.ScalarOnly(); got != known { + t.Errorf("%s: ScalarOnly()=%v with ArithKnown=%v; an unclassified "+ + "architecture must not report a finding it did not measure", arch, got, known) + } + } + // And where it does run, the three states are distinct. + for _, tc := range []struct { + name string + packed, scalar int + want bool + }{ + {"vectorized", 12, 0, false}, + {"mixed", 12, 50, false}, + {"scalar only", 0, 10, true}, + {"a permute kernel: no arithmetic at all", 0, 0, false}, + } { + r := Report{ArithKnown: true, PackedArith: tc.packed, ScalarArith: tc.scalar} + if got := r.ScalarOnly(); got != tc.want { + t.Errorf("%s (packed=%d scalar=%d) = %v, want %v", + tc.name, tc.packed, tc.scalar, got, tc.want) + } + } +} diff --git a/tools/simdgen/verify/verify.go b/tools/simdgen/verify/verify.go index 0adf6ed..01c8e0e 100644 --- a/tools/simdgen/verify/verify.go +++ b/tools/simdgen/verify/verify.go @@ -66,6 +66,26 @@ type Report struct { TotalInstrs int // WidestVector is the widest vector register class seen, in bits. WidestVector int + // PackedArith and ScalarArith count ARITHMETIC instructions by form. + // + // VectorInstrs above answers "does any instruction touch a vector + // register", which on amd64 every float instruction does: scalar SSE + // arithmetic lives in %xmm, so `vmulss %xmm0, %xmm1, %xmm2` counts. + // RequireVector therefore cannot see a scalar float kernel on the one + // architecture that runs natively here, and eight do ship that way -- + // convolve, correlate, movavg and polyeval, in both precisions, with zero + // packed arithmetic at sse2, avx2 and avx512 alike. See docs/wrong.md + // entry 76. + // + // These two count the arithmetic itself, so "does work, none of it in + // lanes" is a question that can be asked. A permute kernel does no + // arithmetic at all and has both at zero, which is why the question is + // not "is PackedArith > 0". + // + // ArithKnown is false on architectures whose mnemonics this does not + // classify, so a zero there reads as "not measured" rather than "none". + PackedArith, ScalarArith int + ArithKnown bool // BodyEnd is the byte offset of the function's return instruction, which // is where the copied body must stop. // @@ -163,7 +183,7 @@ func ObjectWithDisasm(objPath string, tgt target.Target, funcs []string, opt Opt } func checkFunc(name string, instrs []Instr, tgt target.Target, opt Options) Report { - r := Report{Func: name, TotalInstrs: len(instrs)} + r := Report{Func: name, TotalInstrs: len(instrs), ArithKnown: arithClassified(tgt.Arch)} for _, in := range instrs { if f, w := featureOf(in, tgt.Arch); f > r.MaxFeature { @@ -176,6 +196,12 @@ func checkFunc(name string, instrs []Instr, tgt target.Target, opt Options) Repo r.WidestVector = bits } } + switch kind := arithKind(in, tgt.Arch); kind { + case arithPacked: + r.PackedArith++ + case arithScalar: + r.ScalarArith++ + } if n := stackAdjust(in, tgt.Arch); n > r.StackBytes { r.StackBytes = n } @@ -715,6 +741,107 @@ func vectorWidth(in Instr, arch target.Arch) int { return 0 } +// ---------- arithmetic form ---------- + +type arithForm int + +const ( + arithNone arithForm = iota + arithPacked + arithScalar +) + +// arithClassified reports whether arithKind understands this architecture. +// +// amd64 and arm64 only, deliberately. Both spell the difference in the +// mnemonic or the operand and both are checkable here: amd64 is the only +// architecture this repository runs natively, and arm64 is the one with real +// hardware in the cross lane. On the rest, a count of zero would mean "this +// function did not look", and reporting that as "no scalar arithmetic" is the +// shape of a check that cannot fail. +func arithClassified(arch target.Arch) bool { + return arch == target.AMD64 || arch == target.ARM64 +} + +var ( + // amd64: the operation, then the form suffix. `ss`/`sd` are scalar single + // and scalar double -- one lane, in an %xmm register, which is why + // vectorWidth sees them as vector. `ps`/`pd` are packed. The integer + // family (`p*`) is packed by construction. + reX86FloatOp = regexp.MustCompile(`^v?(add|sub|mul|div|max|min|sqrt|rsqrt|rcp|cmp|fmadd|fmsub|fnmadd|fnmsub|and|andn|or|xor|round|scalef|getexp|getmant|range|reduce)[0-9a-z]*?(ss|sd|ps|pd)$`) + reX86PackedInt = regexp.MustCompile(`^v?p(add|sub|mul|mad|max|min|avg|abs|sad|and|or|xor|sll|srl|sra|cmp)`) + + // The zeroing idiom. `xorps %xmm0, %xmm0` is how every x86 compiler + // produces a zero register, and it is not work -- simd_convolve_f32 + // contains two of them at sse2 and three at avx2, and those, with the + // moves, are the entirety of what made it look vectorized. The bitwise + // family stays in the arithmetic set otherwise, because a mask-and-blend + // kernel does its work with `andps`/`orps` and counting that is right. + reX86Xor = regexp.MustCompile(`^v?p?xor(ps|pd)?$`) + reX86Reg = regexp.MustCompile(`%[a-z0-9]+`) + + // arm64: the operand tells the form. `v0.4s` is a vector arrangement, + // `s0`/`d0` a single lane. The mnemonic alone cannot decide -- `fmul` + // spells both. + reA64Arith = regexp.MustCompile(`^(f?(add|sub|mul|div|max|min|abs|neg|sqrt|cmp|mla|mls|madd|msub|nmul)|s(add|mul|max|min)|u(add|mul|max|min)|and|orr|eor|bic|shl|ushr|sshr)`) + reA64Vector = regexp.MustCompile(`\bv[0-9]+\.[0-9]*[bhsdq]|\bz[0-9]+\.[bhsdq]`) + reA64Scalar = regexp.MustCompile(`\b[sdhq][0-9]+\b`) +) + +// arithKind classifies one instruction as packed arithmetic, scalar +// arithmetic, or neither. +// +// MOVES AND ZEROING ARE NEITHER, which is the whole point: simd_convolve_f32 +// at sse2 contains two `movups` and two `xorps` and no packed arithmetic at +// all, and counting the moves is what let it pass as vectorized. A register +// zero is not work. +func arithKind(in Instr, arch target.Arch) arithForm { + m := in.Mnemonic + switch arch { + case target.AMD64: + if reX86Xor.MatchString(m) && sameRegisters(in.Operands) { + return arithNone + } + if reX86PackedInt.MatchString(m) { + return arithPacked + } + if g := reX86FloatOp.FindStringSubmatch(m); g != nil { + switch g[2] { + case "ss", "sd": + return arithScalar + default: + return arithPacked + } + } + case target.ARM64: + if !reA64Arith.MatchString(m) { + return arithNone + } + if reA64Vector.MatchString(in.Operands) { + return arithPacked + } + if reA64Scalar.MatchString(in.Operands) { + return arithScalar + } + } + return arithNone +} + +// sameRegisters reports whether every register operand names the same +// register, which for a xor means the instruction produces a zero. +func sameRegisters(operands string) bool { + regs := reX86Reg.FindAllString(operands, -1) + if len(regs) < 2 { + return false + } + for _, r := range regs[1:] { + if r != regs[0] { + return false + } + } + return true +} + // ---------- frame pointer ---------- var ( @@ -998,6 +1125,21 @@ func disassemble(objPath string, tgt target.Target, objdump string) (map[string] // Summary renders a one-line description of a report, for the generator's log. func (r Report) Summary() string { - return fmt.Sprintf("%-28s %3d instrs, %3d vector (%d-bit), feature=%s, stack=%dB", - r.Func, r.TotalInstrs, r.VectorInstrs, r.WidestVector, r.MaxFeature, r.StackBytes) + arith := "" + if r.ArithKnown { + arith = fmt.Sprintf(", arith=%d packed/%d scalar", r.PackedArith, r.ScalarArith) + } + return fmt.Sprintf("%-28s %3d instrs, %3d vector (%d-bit)%s, feature=%s, stack=%dB", + r.Func, r.TotalInstrs, r.VectorInstrs, r.WidestVector, arith, r.MaxFeature, r.StackBytes) +} + +// ScalarOnly reports that this kernel does arithmetic and none of it is in +// lanes -- a kernel that would be dispatched to as an acceleration and run one +// element at a time. +// +// False when the architecture's mnemonics are not classified, because "no +// packed arithmetic counted" and "no packed arithmetic present" are not the +// same statement and only one of them is a finding. +func (r Report) ScalarOnly() bool { + return r.ArithKnown && r.ScalarArith > 0 && r.PackedArith == 0 }