Skip to content

[AUTOMATED] Follow copy propagations with rule-pool waves too (option copywaves, default on; hkxc 0x1406198f0 437 s -> 343 s, same C) - #1470

Open
mahaloz wants to merge 4 commits into
mainfrom
perf/copywaves
Open

mahaloz wants to merge 4 commits into
mainfrom
perf/copywaves

Conversation

@mahaloz

@mahaloz mahaloz commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

In serde-hkx 0.7.3's hkxc.exe, the 194 KB function 0x1406198f0 spends two thirds of its time in rule pools, and most of their 6,215 passes over it and its jump-table sub-pipelines change only two or three ops:

$ KUNA_ACTION_PROF=prof.txt kuna decompile-all hkxc.exe --addr 0x1406198f0 --json --jobs 1 --max-fn-seconds 0 > out.json
$ grep oppool1 prof.txt
  131779.7 ms   39.77%       4150 calls  jumptable/oppool1
   82568.0 ms   24.92%       2065 calls  decompile/oppool1

Side by side on two pinned cores of a busy machine, the whole function takes 437.0 s user on main and 342.6 s with this PR, with identical JSON.

The stack pointer's join at the function's shared exit block (0x1406490f3) has 626 to 674 inputs, each the COPY that the last call on that path left. RulePropagateCopy propagates one input per application, as upstream does, and a rule applies at most once per op per pass. So each analysis spends one pass per input, and 1,810 of those passes swept every op to make one propagation plus the removal of the COPY the pass before had freed.

  • New option copywaves (default on), widening deadwaves. RulePropagateCopy now records the op it changed and the Varnodes it read before and after. A pass whose only changes were dead-op removals and such replacements is followed by a wave over their neighbourhood, and a replacement inside a wave widens the wave instead of ending it.
  • A wave also offers an op it reaches only for its opcode (an op a sub-variable or split flow fires on) just those far-walking rules, until one applies. The other rules test only what lies next to the op, which the previous pass did not touch.

Tests: tests/stages/kuna-copywaves.xml (same C, live IR and op count with the option off and on) and unit test copy_waves_propagate_the_join_as_sweeps_do_and_visit_only_its_neighbourhood (same end state; the 30 passes after the first offer the unrelated ops nothing, and their far rule exactly as before). No datatest or stage expectation moved.

68,488 corpus functions (decbench O0/O2/O2-noinline, sqlite3.dll, _decimal.pyd, CrashLogger.dll, lua, luajit, libsqlite3, cross-arch lua/sqlite builds, APoseFix.dll, hkxc.exe's 48 and libpython3.11's 68 largest functions; --jobs 4 --option protoorder off) print identical records; the two that differ, tar -O2-noinline 0x10dd0 and sq_ppc32_O2 0x1007df90, are --jobs flakes that differ the same way between two builds of main.

Speed, 7 binaries (gzip -O0, grep, kmod, dash -O2-noinline, bzip2, liblua, CrashLogger.dll) on pinned cores of a busy machine, n=5: sum of per-binary min user 142.74 → 142.53 s, sum of medians 170.81 → 176.38 s (CrashLogger's median carries it; the other six 37.37 → 37.80 s).

Part of #1034.

🤖 Generated with Claude Code

https://claude.ai/code/session_0179iMKMUBscgGJBW9XHH5hJ

@mahaloz mahaloz added the full-ci Run the full cargo workspace suite on this PR before merge (internal PRs skip it by default) label Nov 3, 2026
…ve ops a wave reaches for their opcode only the far-walking rules (option copywaves, default on)

RulePropagateCopy replaces one input per application and a rule applies
once per op per pass, so a join with hundreds of copied inputs takes
hundreds of passes. serde-hkx's hkxc.exe 0x1406198f0 has one: the stack
pointer at its shared exit reads the COPY each path's last call left
(626 to 674 inputs), and 1,810 of the pool's passes over the function and
its jump-table sub-pipelines each propagated one and swept every op.

With copywaves the rule records the op and the Varnodes it touched, so a
pass whose only changes were dead-op removals and such replacements is
followed by a deadwaves wave, and a replacement inside a wave widens it.
A wave also offers an op it reaches only for its opcode just the
far-walking rules, since the others test only what lies next to the op.
Same C on hkxc 0x1406198f0 and 0x1406491d0, CrashLogger 0x180005650 and
libpython3.11 0x3b72c0.

Part of #1034.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0179iMKMUBscgGJBW9XHH5hJ
…e propagated value's definition and readers

[AUTOMATED] RulePropagateCopy recorded the op it changed, the Varnodes that
op read before the change and its output, but not the COPY's input the op
reads afterwards. The wave that follows therefore left out that input's
definition and its other readers, whose one-hop view the propagation changed
(the definition's output gained a reader), so a rule testing a definition's
readers could wait for a later full sweep. Record the new input as well, as
chapter 03 and the option summary describe ("every Varnode it read before and
after"). Unit test: a propagation that leaves its COPY alive, then a rule on
the propagated value's definition that tests its readers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0179iMKMUBscgGJBW9XHH5hJ
@mahaloz

mahaloz commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

[AUTOMATED] Pushed 0909823 on top of fb11e25: a wave after a copy propagation now also offers the propagated value's definition and readers.

Repro (fails on fb11e25, passes on 0909823):

cd decompiler && cargo test -p kuna-decomp --lib -- infra::action::tests::copy_waves_offer_the_definition_of_the_propagated_value

note_wave_input_change ran before op_set_input and recorded the op, the inputs it read before the change and its output, but not the COPY input it reads afterwards (invn). When the COPY stays alive (it has other readers), the following wave therefore never offered invn's definition, whose output had just gained a reader, and the pool could converge with a rule on that definition still unfired (the test: a rule on an INT_ADD that tests its output's readers). Chapter 03 and the option summary already describe the wave as covering "every Varnode it read before and after"; the commit records invn as well.

Cost: rule tests on hkxc.exe 0x1406198f0 775,493,707 -> 775,508,535 (+0.002%), tar/grep/libsqlite3 +0.01%. Output is identical to fb11e25 on 50,709 functions (23 binaries: x86-64 ELF and PE, MIPS BE/LE, PPC32).

mahaloz and others added 2 commits November 3, 2026 07:20
…tches a rule a wave misses

[AUTOMATED] RuleIgnoreNan follows a FLOAT_NAN result through three levels of
boolean readers, so a copy propagation or removal two or three ops away can
enable it while the following wave does not offer the FLOAT_NAN op. Mark it
walks_far: a wave now offers it every FLOAT_NAN op, a rare opcode.

Chapter 03 now says that a pool run can end on a wave, and that the stackstall
repeat plus the full first pass of every pool run is what makes a rule the
wave missed fire one stackstall iteration later; it also names
kuna_globalstorekeep's refusal inside RulePropagateCopy as such a reach.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0179iMKMUBscgGJBW9XHH5hJ
[AUTOMATED] Resolve the stage corpus count (691 on main plus kuna-copywaves = 692) and
re-record docs/baseline-stages.json on the merged tree (3815 keys).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0179iMKMUBscgGJBW9XHH5hJ

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-ci Run the full cargo workspace suite on this PR before merge (internal PRs skip it by default)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant