smite-ir: add program minimizers and wire them into AFL++ custom trim

[erickcestari]

When AFL finds an interesting input it gets handed to us for trimming before going back into the corpus. Keeps things tidy and makes crashes much easier to read.

There's a small Minimizer trait, same shape as Mutator, one method on a unit struct, and two implementations:

DeadCodeEliminator walks instructions in reverse with a refcount and drops anything that's both unreferenced and removable (i.e. not a network I/O op). The reverse direction lets chains collapse in one pass: once we drop a consumer its producer's count falls to zero and it goes too.

CommonSubexpressionEliminator is keyed on (operation, canonicalized_inputs). Canonicalizing each instruction's inputs through the running remap before the lookup makes the merge transitive: two duplicate LoadAmounts collapse, then the DerivePoints consuming them collapse, and so on.

Both transforms are safe in IR semantics; the only ops we can't touch are SendMessage and RecvAcceptChannel, because they actually do network I/O. One Operation::has_side_effects predicate gates both passes (same set, same reason).

smite-ir-mutator wires this into AFL's custom-trim ABI. Because both minimizers are deterministic and run to completion in-process, the shim composes them into a single candidate during init_trim, serializes it once, and hands it to AFL as a single round-trip — no phase machine, no per-phase fallback, no iterative feedback loop. If the trim is a no-op (program already minimal) init_trim returns 0 and AFL skips trim entirely.

End-to-end coverage

A new smite-ir-e2e-test crate is a minimal AFL++ harness that decodes a postcard-encoded Program, validates it, and publishes coverage manually to __afl_area_ptr. A new #[ignore]d e2e test under smite-ir-mutator spawns afl-fuzz against this binary with our cdylib loaded as AFL_CUSTOM_MUTATOR_LIBRARY and asserts every hook we export is actually used in a real fuzzing run.

The bitmap has to be bit-identical across DCE/CSE-trimmed variants of the same program (so AFL's trim cksum accepts the shrunk candidate) yet vary under our mutators (so AFL queues new entries). Any compiler-inserted edge whose hit count tracks program.instructions.len() fails the first half: DCE/CSE move the count across AFL's hit-count buckets and the cksum mismatches. postcard::from_bytes and Program::validate both contain such loops.

So the harness is built with RUSTFLAGS=-Cllvm-args=-sanitizer-coverage-level=0 and publishes coverage manually: for each instruction reachable (via inputs) from a side-effect root (SendMessage, RecvAcceptChannel), mark a slot derived from a content hash of (operation, hashes of inputs). Because the hash folds input content (not indices), DCE renumbering doesn't change it; CSE merges duplicates whose hashes were already equal; OperationParamMutator shifts an operation's hash; InputSwapMutator rewires an edge and shifts the consumer's hash.

This also encodes a broader smite design principle: coverage is driven only by side-effecting work. Pure setup instructions that never feed a Send/Recv produce zero coverage and AFL never queues them. The fuzzing signal lines up with the minimizer's notion of "useful work" so trimming can't change coverage.

The e2e test asserts five signals, all scraped from AFL's own AFL_DEBUG=1 output so the cdylib stays instrumentation-free:

1. Found 'afl_custom_<name>' lines at startup for all six exported hooks.
2. Queue filenames carry smite-ir:<last_sequence> from afl_custom_describe, with both branches of mutate_stacked surfacing (fresh and one of op-param / input-swap).
3. [Custom Trimming] START confirms afl_custom_init_trim is invoked.
4. START: Max 1 confirms the DCE+CSE pipeline shrank at least one input. The seed corpus mixes a DCE-reducible program (dead LoadAmount appended) and a CSE-reducible one (duplicate LoadPrivateKey injected) so both minimizer paths can fire.
5. [Custom Trimming] SUCCESS confirms AFL persisted at least one trimmed candidate — the trimmed bytes' coverage cksum matched the original. Verifies DCE+CSE preserve coverage end-to-end, not just that we offered a smaller candidate.

A afl-e2e.yml workflow runs this test on PRs that touch the AFL-relevant crates. cargo-afl is pinned via CARGO_AFL_VERSION and cached on that key, so bumping the env var invalidates the cache and selects the new version.

[Chand-ra]

Maybe it's a good idea to draft this until #53 gets merged? That way we can avoid any accidental merges or merge conflicts in the meantime.

[erickcestari]

Maybe it's a good idea to draft this until #53 gets merged? That way we can avoid any accidental merges or merge conflicts in the meantime.

Sure, I also need to rebase it. It will probably only be merged after the first milestone is finished.

[morehouse]

I think the iterator pattern doesn't fit very will with the duplicate-load and nop minimizers -- both could be fully minimized in a single call, and we wouldn't ever expect the post_trim call to report failure for them. Both could also be easily optimized to linear algorithms.

Since the goal is to actually trim/reduce inputs, I think we should also consider deleting the nops we insert -- currently the nops continue to take up space in the input.

[erickcestari]

I think the iterator pattern doesn't fit very will with the duplicate-load and nop minimizers -- both could be fully minimized in a single call, and we wouldn't ever expect the post_trim call to report failure for them. Both could also be easily optimized to linear algorithms.

Since the goal is to actually trim/reduce inputs, I think we should also consider deleting the nops we insert -- currently the nops continue to take up space in the input.

I've refactored almost all of the code. I've ensured that all pipeline trimming occurs once at afl_custom_init_trim. I've also made the algorithms linear O(N). I've completely removed the NOPs and the Iterator pattern.

[morehouse]

Nice, this is much cleaner.

[morehouse]

It would be good to do an e2e test with this to make sure our custom trim actually runs and is useful.

[morehouse]

We may also need to update last_sequence so we can tell if trim is actually being used by AFL.

[morehouse]

I'm starting to second-guess this new e2e test. It's become quite contrived, almost to the point that it will pass by construction. The way we define the custom coverage metric almost guarantees AFL will consider trimming a success.

We already have good unit tests that cover a significant portion of what the e2e test covers, so it seems like the e2e test brings in a lot of machinery for the marginal improvement to coverage.

What I really want to know is whether the minimizers are useful against a real target. I propose that we drop the e2e test and run a short LDK or LND fuzzing campaign with AFL_DEBUG=1. If we see "[Custom Trimming] SUCCESS" lines in the debug output, that's a good indicator that the minimizers are useful today, and let's get this merged. If not, a few things could be going on -- either our current mutators/generators don't really generate programs that are minimizable (in which case let's retest once we have more diversity in that area) or the instability of our fuzzing targets prevents AFL from accepting the minimizations (in which case trimming is broken for us anyway).