Repository navigation
perf(zstd): x64's hot paths, at or past libzstd's assembly on Zen 4 - #55
Merged
Merged
Conversation
The decoder was tuned on Arm64, where its loops keep every value in one of 31 registers. On x64, with 15, they spilled, and some of Arm64's idioms cost more there. Measured on a Ryzen 9 7950X (Zen 4, Windows 11, Native AOT) against libzstd 1.5.7 built with clang, its assembly Huffman decoder and its BMI2 paths included: - Sequences run by a loop of x64's own (ExecuteSequencesFastX64): the bitstream's bounds proved once a batch of sequences, the repeat offsets and the count in the state rather than in registers it had not, tests as compares and branches rather than flags combined, and matches copied 32 bytes at a time when their offset allows it. The paired loop of two blocks, faster on Arm64, spilled on x64, where the fused one is 1.2 to 1.3x as fast: it is off there. - One Huffman stream decoded in locals (DecodeHuffmanSymbols), the reader's state out of memory. - Four Huffman streams: each stream's next entry read as soon as its container is shifted, as libzstd's assembly does, and the four outputs written a quarter apart from two cursors. Single symbols, 3.0 to 2.5 cycles a symbol (libzstd, 2.47); double symbols, 1.63 to 1.51 (1.38). - ReadNCount's Advance inlined: by reference, it kept the description's position, count and bits in memory. A table description reads 3.8x as fast. - OverlapCopy's shuffle as ShuffleNative, a bare vpshufb on x64. Over libzstd, from efc0ec6: silesia 0.82-0.89x to 1.02-1.12x, zstd's github records 0.90-1.01x to 1.08-1.14x, with their dictionary 0.94-0.97x to 1.12-1.20x. The Huffman loops, the single-stream decoder and ReadNCount are shared with Arm64; the rest is x64's alone.
…erence As the decoder, the match finders and the entropy coder were tuned on Arm64. On x64, where they spilled and Arm64's idioms cost more, the steps (same machine and reference as the last commit): - Match finders: conditions as compares and branches on x64, where Arm64 combines them with & into conditional compares and x64 made setcc chains (fast, double-fast, lazy, hash chains, optimal); the window as a clamp of a pointer; the rows' and the long-distance matcher's prefetches as prefetches, which .NET has on x64 only. - The double-fast search with an attached dictionary, of x64's own (CompressBlockAttachedX64): libzstd's form held some twenty values across calls and spilled them all, a 440-byte frame; the search now holds ten, the rest read as operands. github + dict L3 1.12x. - Hash chains: the insertion before the search's values, which were live across its loop and spilled it (L5 1.08x, L7 1.06x on github's records); the skipping test out of that loop. - Sequences stored as their lengths and offset on x64, their codes computed in one pass once the block is found (DefersCodes); the sequence encoder reads the three tables in place, without gathering them; LargestCount in a local and with vector maxima. - Entropy tables by reference: a table repeated, predefined or a dictionary's is that table, and a new one is built into a table the previous state does not use, where libzstd copies 2 to 5 KiB a block (small frames with a dictionary, 1.07 to 1.08x). - Huffman's sort with its totals and its symbols without count in registers (1.03 to 1.05x from L5 to L13), the histogram counted in two sets, the FSE writer of the tree's weights promoted to registers. Over libzstd, from efc0ec6: silesia 0.80-1.02x to 0.97-1.07x, zstd's github records 0.85-0.97x to 0.96-1.20x, with their dictionary 0.69-1.17x to 0.97-1.19x. The tables by reference, the hash chains' order, the sort, the histogram and the FSE writer are shared with Arm64; the rest is x64's alone. Every frame is still libzstd's, byte for byte.
Vorticity.Zstd now has paths of x64's own, which an x64 runner takes: its portable paths, Arm64's shape without its vectors, ran nowhere in CI. The pass with intrinsics disabled runs its suite whole, with zstd's test data written first.
…std's assembly - The native reference builds on Windows, from Git Bash, with a GNU-compatible clang such as llvm-mingw: zstd's assembly Huffman decoder and its BMI2 paths, which MSVC builds neither of, and a PDB. The tests and the benchmarks load the platform's library name. - The program-counter sampler runs on Windows, symbolized through DbgHelp: functions, lines, every instruction of one function (--pcfunc), and the callers of one (--pccaller). - --core N keeps the timed thread on one logical processor at a high priority; the corpus takes a 1 MiB sample of Silesia every 16 MiB (silesia-sample) and --directions; micro-benchmarks of whole frames and corpora (decode, ccorpus), of libzstd's Huffman decoders (nx1, nx2), and the clock measured. - zstd-ab-aot.sh compares two kept builds (--before, --after, --no-build) on a pinned core.
…ted harness The encodings column by column, the advice and CompressorBenchmarks, measured again on the Ryzen 9 7950X at a3dfc8a. The page had been measured before the harness held both sides to the same work (78cb0c4): its scans took over the mapping the last pass left, where every scan now maps its file anew, as on the arm64 page, and pays for its pages on Windows. The head says so. Zstd's columns read faster: a take of 1 000 rows 60.3 to 35.1 ms on jittered timestamps, 64.1 to 31.7 ms on integers in 0..999, both under the hint; their scans, which now count the mapping, stay near 80 ms. CompressorBenchmarks' zstd candidate takes 7.5 us where it took 26.8. Pco, newer than the last measure, is now Smallest's pick on several integer columns.
On Windows the path ceilings failed by 16 bytes on the four axes that scan the main file. Windows 11 24H2 reopens a file the session keeps mapped by its name, with no handle (78bd3fc), and without a handle the tail is read as a slice of the mapping rather than positionally into a pooled buffer (29e0b37): one owner where the open of a handle had none, 16 bytes over that open, 40 over Linux and macOS, which open every file from a handle. A footer past the first read costs less by name, the handle's open then allocating the larger buffer: the map file's axis reads 8 bytes under Linux. The allowance is now asked of each file after its warm-ups, of the state the measured runs found: 40 bytes where the session keeps it mapped and the name answers, 24 for a handle's open, 0 elsewhere. Every axis then reads on Windows the headroom it reads on Linux, but the map file's, 48 bytes looser.
…ence binary `--ratio-check` refused any reference binary but the one its references were calibrated against, `--out` included, so no machine but the M4 could write its page's in-process section: an x64 machine builds its own `vxbench`, whose ratios are its own. Under another binary, `--out` now measures every axis, writes the section and holds none to a ceiling; without `--out` the gate refuses as before.
…there The comparisons with Rust on the x64 page were withdrawn in 9e94f34, until an x64 machine measured them on the corrected harness. They are measured again on the Ryzen 9 7950X, against a `vxbench` built on it: - a table, process against process: faster on 28 of 28 rows on one core, 26 of 28 on all 32, Rust ahead on a scan and a wide filter of its own file at ten million rows (0.75x, 0.74x); - decoding, per encoding: 43 of 61 files, a median of 1.12x; - in one process: 19 of 19 axes, 1.10x to 13.0x; - taking rows: 57 of 61 files, a median of 1.12x; - writing: 59 of 60 files, a median of 3.52x; - `LanesBenchmarks`: Rust at four workers takes 2.1 times our four lanes. The corpus is the generator's current one, four Pco widths more than the arm64 page's, and those are where Vorticity trails most: 0.32x to 0.49x to decode, 0.05x to 0.43x to take. The page's head says the ratios are this machine's, held to no ceiling, and the pages that pointed at it no longer say its Rust sections are missing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Bring Vorticity.Zstd on x64 to where it stands on Arm64: at or past libzstd 1.5.7, measured against a libzstd that has, as it has everywhere a GNU-compatible compiler builds it, its assembly Huffman decoder and its BMI2 paths.
Why
The codec was tuned on Arm64, whose 31 registers hold every value of its loops. x64 has 15, and 8 kept across a call: many loops spilled, some of them all their values, and some of Arm64's idioms cost more there. Conditions combined with
&become conditional compares on Arm64 andsetccchains on x64. A prefetch emulated by a load is the only kind Arm64 has in .NET, where x64 has real ones. A paired loop of two blocks wins on Arm64 and spills on x64.At efc0ec6, on a Ryzen 9 7950X (Zen 4, Windows 11, Native AOT), Vorticity.Zstd ran 0.82 to 1.01x libzstd's speed in decompression and 0.69 to 1.17x in compression. Small frames with a dictionary were the worst: 0.69x at level 3.
The change
Eight commits:
perf(zstd): decompression's hot loops on x64.perf(zstd): compression's hot paths on x64, its entropy tables by reference.&on x64, real prefetches, and an attached-dictionary double-fast search of x64's own.ci: run Vorticity.Zstd's suite with intrinsics disabled too. An x64 runner now takes x64's paths. The portable ones, Arm64's shape without its vectors, would otherwise run nowhere in CI.bench: profile Vorticity.Zstd on Windows x64 and compare it with libzstd's assembly.docs(guide): the x64 page's own sections measured again, on the harness corrected in 78cb0c4. The page had been measured before that fix, and its head now says how scans map their file.test(scan): allow for Windows' reopen by name in the path ceilings. Since 78bd3fc and 29e0b37, Windows 11 24H2 reopens a kept file by its name, with no handle, and reads its tail as a slice of the mapping. That is 16 bytes over a handle's open, soPathAllocationTestswas red on Windows. The allowance is now taken from the state each axis's measured runs found. This also makesbench/gate.sh's ratchets green on Windows.bench: write another machine's in-process section under its own reference binary.--ratio-check --outrefused anyvxbenchbut the M4's, so no other machine could write its page's in-process section. Under another binary it now measures every axis, writes the section and holds none to a ceiling. Without--out, the gate refuses as before.docs(guide): the x64 page measured against Rust, on avxbenchbuilt on the Ryzen. These sections had been withdrawn in 9e94f34. Vorticity is faster on:Vorticity trails most on four Pco widths new to the generator's corpus: 0.32x to 0.49x to decode, 0.05x to 0.43x to take.
What it brings
Vorticity.Zstd.Perf --core 20 --corpus silesia,github,github-dict, Native AOT, each direction a median over 5 s of interleaved rounds, every output checked against libzstd's. Speed over libzstd's, before (efc0ec6) → after:At the other levels (5, 9, 13, 16), compression reads 0.96 to 1.19 and decompression 1.08 to 1.20.
Below libzstd:
These are within run-to-run spread or diffuse. In paired runs, github + dictionary reads 0.99 at 5 and 1.02 at 7.
On the x64 page, measured again, the zstd candidate of
CompressorBenchmarkstakes 7.5 µs where it took 26.8 at b5104f2. A take of 1,000 rows from a Zstd column takes 35 ms where it took 60. Both figures also count what changed between b5104f2 and this branch.Risks and checks
Risk: Arm64 is not measured yet. The following are shared with Arm64:
Everything else is x64's alone, behind
X86Base/Bmi2checks the JIT folds away. All of the shared changes only take work away, except the read-ahead, which reorders the four streams' lookups. It goes behind an x64 check if the M4 says it costs there.Risk: the ratio gate is calibrated on the M4. On this machine,
bench/gate.sh --ratio-checkrefuses the Windows build of the reference: its binary is7527e2b5d861, and the references name1be9345d4816. The gate, which guards Arm64, runs on the M4. The x64 page's ratios are measured here and held to no ceiling.Tried and dropped:
Before merging:
Vorticity.Zstd.Tests: 1,265, with intrinsics and withDOTNET_EnableHWIntrinsic=0; the dictionary fuzz also at 10x its iterationsVorticity.Conformance: 923Vorticity.Tests: 7,273 (10 skipped) on Windows,PathAllocationTestsincluded.CorePressureTests.AQueryItsLanesTablesOutgrowFinishesUnderItsBudget(50)failed in one of three full runs; it passed in the other two and 5 of 5 alone, and this branch does not touch itbench/gate.shon Windows: ratchets and--ffi-checkgreen--corpus silesia,github,github-dict), the Huffman micro-benchmarks (--micro x1,x2),bench/gate.shwith--ratio-checkvxdump