Smitebot: Automated Fuzzing Campaign Manager

[Ashish-Kumar-Dash]

Problem

Running a smite campaign today requires significant manual coordination at every stage. A typical 8-core LND campaign involves:

• Constructing AFL++ command lines with correct -M/-S flags for main and secondary runners
• Managing tmux sessions and monitoring externally via afl-whatsup
• After stopping: walking each runner's queue/ directory and invoking afl-cmin -X manually
• Running coverage-report.sh with no baseline or diff against prior runs
• Finding crash files with no systematic path to reproduction, minimization, or deduplication

Several of these fail silently — a missing -X flag skips Nyx mode entirely, truncated covcounters.* files under high parallel load produce misleading coverage numbers, a missed queue/ subdirectory silently discards corpus inputs. There is also no institutional memory across campaigns- no reliable way to know whether a new campaign covered more ground than the last, or whether a new crash is genuinely novel.

Solution

Smitebot is a Rust CLI tool that automates the full fuzzing lifecycle — from multi-core AFL++ campaign launch and clean shutdown, to corpus minimization, crash triage with deduplication, and historical coverage diffs. The same 11-step manual workflow becomes three commands: smitebot doctor, smitebot start, smitebot stop.

The design is inspired by what syzbot provides for the Linux kernel — a state-aware orchestration layer that makes fuzzing campaigns reproducible, auditable, and accessible.

---

Architecture

Smitebot is added as a new crate to the existing workspace. It is an orchestration layer, not a rewrite — existing scripts (coverage-report.sh, symbolize-crash.sh, setup-nyx.sh) encode subtle, validated, target-specific behavior and are called as subprocesses. Smitebot adds state, coordination, and the missing pipeline pieces around them.

smitebot/src/main.rs — clap command dispatch
smitebot/src/config.rs — TOML schema and validation
smitebot/src/state.rs — campaign state machine + JSON persistence
smitebot/src/campaign.rs — start / stop / status orchestration
smitebot/src/process.rs — AFL++ process supervision, wraps ManagedProcess
smitebot/src/corpus.rs — merge and minimize
smitebot/src/triage.rs — crash discovery, reproduce, afl-tmin, dedup
smitebot/src/coverage.rs — coverage-report.sh invocation, snapshot, diff
smitebot/src/doctor.rs — prerequisite checks
smitebot/src/docker.rs — image build wrappers

New dependencies: clap, toml, serde_json. Reused from workspace: log, thiserror, serde, nix.

Builds directly on PR #32 — ManagedProcess::process_group(0) is the foundation for the stop command.

---

Core Concepts

Campaign State

This is the foundational piece absent from the current codebase. Before smitebot can reliably stop a campaign, diff a coverage report, or triage crashes in context, it needs persistent knowledge of what is running.

State is written to ~/.smitebot/runs/<campaign-id>/state.json. Key fields: campaign_id, config_hash (SHA-256 of config snapshot), state enum (created/starting/running/stopping/stopped/failed), per-runner pid/pgid, and artifact directory pointers. All writes are atomic (temp file + rename) — a killed smitebot never leaves corrupted state..

Commands

smitebot doctor
Prerequisite checks before any campaign: x86_64 arch, /dev/kvm accessible, Docker daemon reachable, afl-fuzz/afl-cmin/afl-tmin/afl-whatsup on PATH, AFL++ Nyx mode built, libnyx.so on LD_LIBRARY_PATH, VMware backdoor enabled, smite scripts present and executable. --json flag for CI use.

smitebot start / stop / status
TOML config consistent with aflr_cfg_smite_lnd_encrypted_bytes.toml. Each runner spawned via ManagedProcess::spawn() with process_group(0) — pgid recorded to state, cleanup reaches grandchildren. Start rolls back all runners if any fail within 3 seconds of launch.

Nyx campaign stop is process group termination only — SIGTERM → timed wait → SIGKILL via killpg. CLN graceful stop (lightning-cli stop via docker exec) is scoped exclusively to the coverage/replay workflow, where CLN runs as a real long-lived host process and must exit cleanly for profraw files to flush correctly. This matches the existing Drop impl in cln.rs.. Status reads fuzzer_stats from each runner's output directory (execs_done, execs_per_sec, corpus_count, saved_crashes, saved_hangs) and prints a combined summary.

smitebot corpus merge / minimize
merge walks $afl_out/*/queue/ via std::fs, collects all inputs except README.txt, copies with normalized names to a merged directory. minimize invokes afl-cmin -X -i <merged> -o <minimized> -- <sharedir>.

Kept as separate commands intentionally: merge is always fast and safe (pure collection, no binary required); minimize requires the target binary and Nyx sharedir and is run deliberately.

smitebot crashes triage

Pipeline per crash input:

1. Discover — walk $afl_out/*/crashes/ across all runners
2. Reproduce — docker run in local mode with SMITE_INPUT set, retry ×3, classify Reproducible/Flaky
3. Minimize — afl-tmin -X for reproducible crashes
4. Symbolize — target-specific: CLN/LDK via llvm-symbolizer; LND goroutine stacks already symbolic; Eclair JVM output already symbolic
5. Fingerprint — SHA-256 over target:scenario:signal:frame_0:frame_1:frame_2
6. Deduplicate — check against ~/.smitebot/crashes/known.json
7. Report — per-bug-group directory with raw crash, minimized input, symbolized log, metadata JSON

smitebot coverage
Wraps coverage-report.sh as a subprocess, preserving all existing target-specific behavior. Adds: timestamped snapshots after each run, diff against prior snapshot. Parsing uses machine-readable formats — coverage.txt (LND) — already produced by the existing pipeline via covdata merge; smitebot snapshots this file after each run and computes per-file deltas across snapshots directly.), llvm-cov export --format=text (CLN/LDK), jacoco.csv (Eclair). No HTML scraping.

Testing Strategy

Three layers:

• Unit tests — config parsing, state machine transitions, fingerprint hashing, corpus file collection. No external dependencies, runs in CI.
• Integration tests — lifecycle commands tested with sleep infinity runners spawned via process_group(0), matching the exact spawn path used for real fuzzers. Synthetic state.json files injected to test reconcile and rollback paths independently. Grandchild absence in /proc asserted after smitebot stop.
• Fixture tests — corpus and triage tests run against pre-populated queue/ and crashes/ trees. No live fuzzer required.

CLN graceful stop (docker exec lightning-cli stop) is tested manually against a live container and gated behind --integration so it does not block CI on hosts without Docker.

---

Implementation Plan

Milestones are structured as vertical slices — each delivers a working system for a narrow set of functionality. After Milestone 2 is complete, most subsequent milestones can be developed in parallel.

• Milestone 1: Foundation — Crate scaffolding, TOML config schema, smitebot doctor, CI pipeline (fmt, clippy, test)
• Milestone 2: Campaign Lifecycle — smitebot start (single + multi-core), stop (with CLN graceful stop), status with fuzzer_stats aggregation
• Milestone 3: Reliability — Stale-state reconcile, rollback on partial start, integration tests for lifecycle commands
• Milestone 4: Corpus Management — smitebot corpus merge and smitebot corpus minimize with before/after reporting
• Milestone 5: Crash Triage — Discovery, reproduction, afl-tmin minimization, symbolization (CLN + LND), fingerprinting, dedup store
• Milestone 6: Crash Triage (LDK + Eclair) — Symbolization for remaining targets, report bundle generation, triage integration tests
• Milestone 7: Coverage — smitebot coverage with snapshot storage and LND diff parsing
• Milestone 8: Polish — End-to-end demo across all four targets, operator documentation, UX cleanup
• Milestone 9+ (Optional): Daemon mode with scheduled campaigns and crash alerts; per-target coverage diff for CLN/LDK/Eclair formats

If interested in a more deeper overview of Smitebot, you can checkout my proposal here

I'm happy to discuss any part of the design before work starts. Would love to hear your thoughts and review my plan.

[morehouse]

Concept ACK.

I think the implementation approach is good as well. A few questions/comments/suggestions.

Builds directly on PR #32 — ManagedProcess::process_group(0) is the foundation for the stop command.

Maybe this will be enough to cleanly stop fuzzing campaigns, or maybe there will be other challenges. When fuzzing with Nyx mode, AFL++ spawns qemu instances that don't always stop when killing AFL++, and I've had to pkill them in the past. I'm not sure if those qemu instances share a process group with AFL++ or not.

State is written to ~/.smitebot/runs/<campaign-id>/state.json. Key fields: campaign_id, config_hash (SHA-256 of config snapshot), state enum (created/starting/running/stopping/stopped/failed), per-runner pid/pgid, and artifact directory pointers. All writes are atomic (temp file + rename) — a killed smitebot never leaves corrupted state..

Other useful data to consider saving:

• campaign ID
• start time
• stop time
• target
• scenario
• Docker image name and digest
• sharedir path
• smite git hash
• merged corpus dir
• minimized corpus dir

smitebot doctor Prerequisite checks before any campaign: x86_64 arch, /dev/kvm accessible, Docker daemon reachable, afl-fuzz/afl-cmin/afl-tmin/afl-whatsup on PATH, AFL++ Nyx mode built, libnyx.so on LD_LIBRARY_PATH, VMware backdoor enabled, smite scripts present and executable. --json flag for CI use.

What would the --json flag do, and how do you intend to use it in CI?

smitebot start / stop / status TOML config consistent with aflr_cfg_smite_lnd_encrypted_bytes.toml.

This file doesn't exist. It would be helpful to provide a sample config file to illustrate what the config would look like.

We should also consider what flags to run with AFL++ in multi-core campaigns. AFL_Runner has a default set of flags we could borrow ideas from.

Also note that the IR scenarios will need to set additional environment variables to use the smite-ir-mutator library.

Each runner spawned via ManagedProcess::spawn() with process_group(0) — pgid recorded to state, cleanup reaches grandchildren. Start rolls back all runners if any fail within 3 seconds of launch.

3 seconds probably isn't enough. It can take AFL++ several minutes to start up, especially when given a large seed corpus. We may need to check fuzzer_stats and/or pattern-match on the logs instead.

Also I noticed you omitted smitebot build for building the Docker images. Was your idea to incorporate the build into smitebot start automatically? I think that would make sense, but it would also be helpful to have a separate smitebot build command for manual rebuilds or debugging.

smitebot corpus merge / minimize merge walks $afl_out/*/queue/ via std::fs, collects all inputs except README.txt, copies with normalized names to a merged directory. minimize invokes afl-cmin -X -i <merged> -o <minimized> -- <sharedir>.

Kept as separate commands intentionally: merge is always fast and safe (pure collection, no binary required); minimize requires the target binary and Nyx sharedir and is run deliberately.

It may also be worth allowing smitebot minimize to take multiple corpus directories as args, in which case it both merges and minimizes.

Pipeline per crash input:

1. **Discover** — walk `$afl_out/*/crashes/` across all runners

2. **Reproduce** — `docker run` in local mode with `SMITE_INPUT` set, retry ×3, classify `Reproducible`/`Flaky`

I think it's also worth having a standalone smitebot reproduce command for debugging purposes. It is also probably worth having a "flaky repro" flag --retries N that will retry the input N times or until the failure reproduces.

3. **Minimize** — `afl-tmin -X` for reproducible crashes

I'm not sure if afl-tmin will work properly with IR scenarios. Maybe there's some way we can have it use a custom minimization strategy that is IR-aware (e.g., #54).

4. **Symbolize** — target-specific: CLN/LDK via `llvm-symbolizer`; LND goroutine stacks already symbolic; Eclair JVM output already symbolic

ASan stack traces in CLN will definitely need to be symbolized, but I think LDK should already be symbolized.

5. **Fingerprint** — SHA-256 over `target:scenario:signal:frame_0:frame_1:frame_2`

Fingerprinting crashes can be tricky -- oftentimes the same root cause will produce slightly different stack traces, and changing the target version or compiler version can also change the fingerprint. It would be good to look at what other crash-dedup systems do (e.g., OSS-Fuzz/ClusterFuzz) to deal with this.

6. **Deduplicate** — check against `~/.smitebot/crashes/known.json`

What exactly gets stored in known.json?

7. **Report** — per-bug-group directory with raw crash, minimized input, symbolized log, metadata JSON

smitebot coverage Wraps coverage-report.sh as a subprocess, preserving all existing target-specific behavior. Adds: timestamped snapshots after each run, diff against prior snapshot. Parsing uses machine-readable formats — coverage.txt (LND) — already produced by the existing pipeline via covdata merge; smitebot snapshots this file after each run and computes per-file deltas across snapshots directly.), llvm-cov export --format=text (CLN/LDK), jacoco.csv (Eclair). No HTML scraping.

Makes sense, though we probably do want to generate some kind of HTML report after all the deltas are computed.

CLN graceful stop (docker exec lightning-cli stop) is tested manually against a live container and gated behind --integration so it does not block CI on hosts without Docker.

I think this is probably unnecessary. CLN doesn't need any special handling in Nyx fuzzing mode, only in coverage mode. And if a user cancels a coverage run, they shouldn't expect a coverage report anyway.

[Ashish-Kumar-Dash]

Hi Matt, Thanks for the review, I got more perspective & things to look out for. Here are some clarifications from my side-

Maybe this will be enough to cleanly stop fuzzing campaigns, or maybe there will be other challenges. When fuzzing with Nyx mode, AFL++ spawns qemu instances that don't always stop when killing AFL++, and I've had to pkill them in the past. I'm not sure if those qemu instances share a process group with AFL++ or not.

Thanks for informing. During Week 3 smoke testing I'll verify whether QEMU inherits AFL++'s PGID by inspection. If it doesn't, stop will add a targeted cleanup pass- killpg followed by scan filtered on the QEMU binary name before transitioning state to stopped. I'll document the findings

What would the --json flag do, and how do you intend to use it in CI?

Right now, the doctor will print human readable output, The --json flag will make the output in structured JSON instead, like this:
{
"checks": [
{
"name": "kvm",
"passed": true
},
{
"name": "afl-fuzz",
"passed": false,
"reason": "not found on PATH"
}
],
"overall": false
}
CI can then read the JSON to see the missing prerequisites in job annotations rather than requiring a human to read terminal output

This file doesn't exist. It would be helpful to provide a sample config file to illustrate what the config would look like.

My bad, It was generated during my competency task submission and I didn't reference it. You can check it out here
Do let me know your thoughts on it.

3 seconds probably isn't enough. It can take AFL++ several minutes to start up, especially when given a large seed corpus. We may need to check fuzzer_stats and/or pattern-match on the logs instead.

I'll get back to you on this during implementation

Also I noticed you omitted smitebot build for building the Docker images. Was your idea to incorporate the build into smitebot start automatically? I think that would make sense, but it would also be helpful to have a separate smitebot build command for manual rebuilds or debugging

Adding it as a standalone command. auto_build = true in config handles the common path, the explicit command covers manual rebuilds, post-source change rebuilds, and CI pipelines that separate build and run stages.

What exactly gets stored in known.json?

Sample schema-
{
"bugs": [{
"fingerprint": "sha256:abc123",
"target": "cln",
"scenario": "encrypted_bytes",
"signal": "SIGABRT",
"frames": ["fn0", "fn1", "fn2"],
"first_seen": "2026-05-01T14:23:00Z",
"report_dir": "~/.smitebot/crashes/cln/abc123",
"smite_git_hash": "deadbeef",
"status": "open"
}]
}
On each triage run, new crash fingerprint is checked against this. Match → duplicate, logged, no new report dir. No match → new entry written, report dir created.

I think this is probably unnecessary. CLN doesn't need any special handling in Nyx fuzzing mode, only in coverage mode. And if a user cancels a coverage run, they shouldn't expect a coverage report anyway.

Yes, already clarified earlier through mail. I will keep it in mind

I'm not sure if afl-tmin will work properly with IR scenarios. Maybe there's some way we can have it use a custom minimization strategy that is IR-aware (e.g., #54).

I'll review PR #54 before Week 8. If IR-aware minimization is available there, smitebot dispatches to it based on a config flag or scenario name. If not, minimization is skipped for IR scenarios, the raw crash input is preserved, and the triage report notes why. The pipeline doesn't block on this.

I think it's also worth having a standalone smitebot reproduce command for debugging purposes. It is also probably worth having a "flaky repro" flag --retries N that will retry the input N times or until the failure reproduces.

Yes, we can add it through-
smitebot reproduce --input --retries N
Runs the scenario binary in local mode up to N times, reports hit count. Separate from triage's automated reproduction, this is for debugging and fix verification. I need to start writing down things :)