Developed by SPYDIRBYTE

A content discovery fuzzer that doesn't just work a static wordlist and call it done. It learns what "nothing here" looks like on the specific target it's pointed at, recurses into anything that looks like a real directory, and actually reads every page it finds for new leads, links, JS fetch calls, JSON API references, folding them into the queue as it goes. ffuf is fast but dumb. feroxbuster recurses but only ever tries the wordlist. This does both, plus the part neither of them does: following the trail a page actually leaves behind.

Run it against targets you own or have explicit written authorization to test.

---

What it does differently

Learns the target's soft-404 shape before fuzzing starts. A lot of sites don't return a real 404, they return 200 with a branded catch-all page for anything that doesn't exist, sometimes with a timestamp or request id baked in so every one looks slightly different. A handful of deliberately fake paths get probed first (different depths, different extensions, since some targets 404 differently for /fake.php than /fake/), and every real hit gets compared against those samples with fuzzy text similarity, not an exact match, so the small dynamic bits don't defeat the comparison.

Keeps learning while it runs. The startup baseline only catches the soft-404 flavor that shows up for obviously fake paths. Some targets have a second one that only appears under a specific subdirectory, or a rate-limit page that only kicks in after enough requests. Any response shape that keeps recurring during the run gets compared against a few stored samples from that same shape and suppressed once it's clearly noise, with a hard ceiling so even wildly varying dynamic content can't force it to keep reporting the same junk forever.

Recurses into real directories automatically, the same idea as feroxbuster, a hit that looks like a directory gets the wordlist run against it too, and again under whatever it finds there, up to a configurable depth.

Reads every page for new leads, this is the actual differentiator. Every response gets scanned for href/src/action attributes and any quoted path-looking string sitting in the raw body, catches JS files referencing other JS files and API routes, JSON responses referencing other endpoints, a link only ever mentioned in an HTML comment, plus real framework route definitions (Express, Flask, Django) rather than just generic quoted strings. Anything found this way goes into the queue as a proven lead, not a guess, tagged separately in the output so you can tell what the wordlist found versus what the target's own content led you to.

Mutates confirmed finds into plausible variants. A real /admin is a strong signal that /admin-backup, /admin.old, or /admin2 might exist too, exactly the kind of thing a developer leaves behind without meaning to. Every confirmed find gets a bounded, priority ordered set of suffix/prefix/case variants tried automatically instead of hoping they happened to already be in the wordlist.

Fingerprints the tech stack and reprioritizes around it. Reads the target's own headers and homepage for signs of what's actually running, WordPress, Drupal, Laravel, Next.js, and reorders the wordlist so relevant paths (wp-admin, wp-json) go first and extensionless words get tried with the matching extension (.php, .aspx, .jsp) instead of guessing blind across every stack at once.

Pulls seeds from robots.txt and sitemap.xml. A Disallow line exists specifically to tell crawlers what not to index, which makes it one of the highest signal wordlists available, hand written by whoever runs the site. A sitemap is a free list of every page they want indexed. Both get parsed and queued before the wordlist even starts.

Learns this specific target's naming convention, not just generic suffixes. The mutation engine above guesses the same handful of suffixes on any target. This watches everything confirmed real across the whole run and looks for a shared shape, numbered sequences (report1, report2 implies report3 is worth trying), zero-padded numbers, shared prefix clusters (three different finds starting with user_ implies more user_ guesses are worth it), and feeds whatever it infers back into the queue, specific to this target, not a fixed list anyone could have already tried.

Fuzzes for hidden parameters on what it finds, off by default since it multiplies requests per finding. A curated list of common parameter names (debug, admin, format, token) gets tried against every confirmed file-like endpoint, compared against a clean baseline request, a parameter that actually changes the response is a real lead, one that's ignored comes back identical and gets dropped.

Classifies and scores every finding, not just status and length. Response bodies get read for what they actually are, a login form, an admin panel, exposed API docs, a raw config file, and scored for how interesting the content itself is, hits on credential-shaped words, internal-use language, stack traces, private key headers. Two 200s with similar length can be worlds apart in what they're worth looking at first, this is what lets you sort by that instead of chronological order.

Adapts when it gets blocked, not just backs off. A blocked response (403, a WAF signature in the body) triggers an automatic retry with alternate path encodings, case randomization, percent encoding, path tricks that survive a naive, exact-match blocking rule. Off by default, opt in with --evasion, since it's a meaningfully more aggressive posture than passive rate-limit backoff.

Virtual host discovery, a second mode entirely, fuzzes the Host header against a single IP instead of the URL path, catches name-based virtual hosting, internal tools, staging environments, admin panels, sitting on the same IP as the public site but invisible to anyone who only ever requests it by its real hostname. Give it a bare host or IP with no root domain and it derives one from the target itself automatically.

Resolves the target and figures out the scheme before doing anything else. Give it a bare host, example.com, no http:// or https://, and it resolves the DNS first (shown up top, not buried in logs), then checks whether HTTPS, HTTP, or both actually respond, and picks the one to run against.

The wordlist is tiered and categorized, not just one flat file. Every built-in entry (194 of them) is tagged with a category, admin, auth, config, version control, backup, cloud/infra, per-CMS (WordPress, Drupal, Joomla, Magento), per-language (PHP, Node, Python, Java, Ruby), so it isn't just bigger than a basic list, it's structured. --tier quick runs only the ~40 highest real-world hit-rate paths for a fast pass, --tier common is the ~194 categorized default, and --tier full loads a generated list of roughly 11,000 entries, real default paths for platforms most lists skip (Grafana, GitLab, Jenkins, Nextcloud, Vault, Umbraco, Strapi, Ghost, and dozens more), every REST API resource noun crossed with common actions and versioned prefixes (api/v2/invoices/export), and systematic backup/config permutations (extensions, date stamps, zero-padded numbering) instead of a dozen hand typed guesses. -w and --merge-wordlist both accept a local file path or a URL, straight from GitHub, a raw SecLists link, anything reachable over plain HTTP GET, no need to download it first. Paste a regular GitHub file page link (the one you'd copy out of your browser) instead of the raw one and it gets auto-corrected, that's the single most common gotcha with this kind of thing. --merge-wordlist combines any external list, local or remote, with whichever tier you're using, deduped, so you're never choosing between the two. The categories feed straight into tech fingerprinting too, a detected WordPress site gets every WordPress tagged word prioritized, not five hardcoded guesses.

The full list is generated, not hand maintained, generate_full_wordlist.py builds wordlists/full.txt from the categorized base plus systematic platform/API/backup permutations, run it yourself any time to regenerate or extend it. The live feed reads like an actual investigation log. Beyond the finding cards themselves, the step log calls out what's happening as it happens, found admin, depth search complete for backup once every wordlist entry under a discovered directory has been tried, subdomain found: staging.example.com in vhost mode. And the JSON report always includes how many requests were made and how many unique URLs were tried, with a --full-report flag if you want the complete list of every single one, off by default since that list can get large fast on a deep scan.

Certificate transparency subdomain discovery. Every SSL certificate ever issued for a domain is public record. --ct-subdomains queries crt.sh before anything else starts, shows every hostname found alongside its actual DNS resolution, and in --vhost mode adds them as extra candidate names on top of the built-in list, real, certificate confirmed hostnames instead of only generic guesses.

Remembers the last scan of every target. Re-run against the same target and it tells you what's new since last time and what's no longer there, stored locally, nothing leaves your machine. Batch mode runs a whole list of targets through the same engine in one pass, one report per target plus a combined summary, and every target in the batch gets the same before/after comparison against its own history.

Identifies the WAF vendor and adapts to it, not just a generic bypass list. Cloudflare, Akamai, Sucuri, Imperva, and ModSecurity each have their own header, cookie, and body signatures. Once identified, the technique order in adaptive evasion reorders itself around what's actually known to work against that specific product instead of trying every trick in the same fixed order regardless of what's actually in front of the target.

Usage

The dashboard, live trail graph, fingerprint readout, findings feed:

pip install -r requirements.txt
python server.py

Open http://localhost:5051. Switch between path fuzzing and vhost discovery with the tabs at the top.

Or the CLI:

python cli.py https://example.com
python cli.py example.com                  # no scheme needed, auto-detects HTTPS/HTTP
python cli.py https://example.com -w wordlists/common.txt --depth 4
python cli.py https://example.com --rps 100 --concurrency 50
python cli.py https://example.com --no-follow      # wordlist + recursion only
python cli.py https://example.com --no-mutate       # skip variant guessing
python cli.py https://example.com --no-fingerprint   # skip tech detection/reordering
python cli.py https://example.com --no-seeds          # skip robots.txt/sitemap.xml
python cli.py https://example.com --no-patterns         # skip site-specific pattern inference
python cli.py https://example.com --no-classify          # skip content classification/scoring
python cli.py https://example.com --params                 # also fuzz hidden GET parameters, slower
python cli.py https://example.com --evasion                  # auto-retry blocked paths with alternate encodings
python cli.py https://example.com --output results.json
python cli.py https://example.com --output results.json --full-report   # include every URL tried

# wordlist tiers and merging
python cli.py https://example.com --tier quick        # ~40 highest hit-rate paths, fast pass
python cli.py https://example.com --tier common        # default, ~194 categorized entries
python cli.py https://example.com --tier full          # ~11,000 generated entries
python cli.py https://example.com --merge-wordlist ~/SecLists/Discovery/Web-Content/common.txt

# straight from GitHub, local download not required, works with either
# the raw link or a regular browser file page link (auto-corrected)
python cli.py https://example.com -w "https://raw.githubusercontent.com/danielmiessler/SecLists/master/Discovery/Web-Content/common.txt"
python cli.py https://example.com --merge-wordlist "https://github.com/danielmiessler/SecLists/blob/master/Discovery/Web-Content/common.txt"
python cli.py https://example.com --tier quick --merge-wordlist my-custom-list.txt   # combine both

# certificate transparency, shows real subdomains + their resolutions
python cli.py https://example.com --ct-subdomains

# batch mode, one target per line in a text file, writes one report per
# target plus a combined summary CSV, each target diffed against its
# own scan history same as a single run would be
python cli.py --targets-file targets.txt --output scans/report

# virtual host discovery, fuzzes the Host header against one IP
python cli.py 203.0.113.5 --vhost example.com
python cli.py mail.example.com --vhost      # no domain given, auto-derives "example.com"
python cli.py 203.0.113.5 --vhost example.com --ct-subdomains   # adds real cert-confirmed hostnames as candidates too

-w still works exactly as before for a single custom file or a URL, this overrides the tier entirely. Without -w, --tier picks the built-in set, and --merge-wordlist (repeatable) layers any additional file or URL on top of whichever base you're using, own list, SecLists, dirb, a link straight off GitHub, deduped automatically against everything else already queued. The dashboard has the same option under fuzz options, "custom wordlist", paste a URL or a local path there instead of picking a tier.

Every finding line is tagged with where it came from:

Tag	Meaning
ROOT	the target's own root page
WORD	matched a wordlist entry directly
RECUR	found by running the wordlist against a discovered directory
TRAIL	found by reading a page's content, not from the wordlist at all
MUTATE	a guessed variant of something else already confirmed real
PATTERN	inferred from this target's own naming convention (numbering, prefixes)
PARAM	found via hidden parameter discovery on a confirmed endpoint
ROBOTS / SITEMAP	pulled from robots.txt or sitemap.xml before fuzzing started
VHOST	a hidden virtual host found via Host header fuzzing

Structure

.
├── fetcher.py       # async HTTP layer, rate limiting, adaptive backoff, Host header override
├── baseline.py       # learns the target's soft-404 shape before fuzzing starts
├── dedupe.py           # runtime noise filtering, fuzzy similarity + hard ceiling
├── link_extract.py      # reads response bodies for new leads + framework route patterns
├── fingerprint.py         # tech-stack detection, reprioritizes the wordlist around it
├── mutate.py                # generates plausible generic variants of confirmed finds
├── pattern_learn.py            # infers this target's own naming convention across all finds
├── seeds.py                       # robots.txt / sitemap.xml parsing for extra seed paths
├── param_fuzz.py                     # hidden GET parameter discovery (off by default)
├── classify.py                          # content classification + interest scoring
├── waf_detect.py                           # WAF vendor fingerprinting + targeted alternate encodings
├── recon.py                                  # DNS resolution + HTTP/HTTPS auto-detection
├── cert_transparency.py                        # crt.sh subdomain discovery
├── history.py                                    # local scan history + diffing
├── vhost_fuzz.py                                   # Host header fuzzing against a single IP
├── engine.py                      # the queue-driven fuzzing loop, ties everything together
├── wordlist.py                     # tiered, categorized wordlist + merge support
├── wordlists/full.txt               # generated ~11,000 entry list (see generate_full_wordlist.py)
├── generate_full_wordlist.py          # builds wordlists/full.txt from categorized base + permutations
├── cli.py                            # terminal front end, both fuzz and vhost modes
├── server.py                           # local dashboard (Flask + SSE streaming)
├── static/                               # dashboard HTML/CSS/JS, live trail graph
└── requirements.txt

• Mutation only fires once per unique confirmed path segment, and only from wordlist/recursion/robots findings, not from every single trail-discovered link, that would multiply request volume fast for marginal extra coverage
• Tech-stack fingerprinting reads the homepage only, a site that mixes stacks across different sections (a PHP-based main site with a Node.js API subpath) will only get prioritized for whatever the homepage itself reveals
• vhost discovery's baseline is a single request to the default vhost, more sensitive to a site with genuinely dynamic homepage content (a rotating banner, live data) producing an occasional false negative than the path-fuzzing baseline is, which samples several requests
• Route pattern detection in link_extract.py covers Express, Flask, and Django's common syntax, not every framework or every way a route can be defined in each of those

Limits worth knowing

• The registrable-domain heuristic behind vhost auto-fill (sub.example.com -> example.com) is naive, it doesn't handle every public suffix edge case (.co.uk and similar), give --vhost an explicit domain if the target's TLD is one of those

• WAF vendor fingerprinting reads headers, cookies, and body text signatures, a target running a heavily customized or white-labeled WAF deployment might not match any known signature, adaptive evasion still runs the generic technique order in that case, it just won't get the vendor-specific ordering boost

• The quick tier's ordering (which ~40 words count as "highest hit rate") is a curated judgment call based on well known public disclosure patterns, not a statistically derived ranking, it's meant as a fast, sensible first pass, not a scientifically optimal one

• --tier full is genuinely large (~11,000 entries), and the directory-recursion behavior (any extensionless 200 gets the whole wordlist run against it again) compounds with list size, a full-tier scan with recursion left at its default depth can take considerably longer than the quick or common tiers, --depth 0 or --no-follow gives a much faster single pass if you just want the base coverage. The same applies to any large custom or GitHub-pulled wordlist, a 4,000+ entry SecLists file with recursion, following, mutation, and pattern learning all left on by default is going to run a long time, not a bug, just what a thorough pass with a big list actually costs

• Wordlist URLs are fetched once with a plain HTTP GET before the scan starts, no authentication support, a private/gated repo file won't work, only public URLs

• Certificate transparency lookup depends on a free third-party service (crt.sh), if it's slow or unreachable this step just finds nothing and the rest of the scan proceeds normally

• Scan history is stored locally at ~/.spy-trail/scan_history.json, it's just findings and timestamps, no credentials, but delete it if you'd rather not keep it around

• HTTP/HTTPS auto-detection makes one request to each scheme, a target that's slow or rate-limits on the very first request could get misdetected, pass the scheme explicitly (https://...) if you already know it

• Pattern-learning mutation needs at least two confirmed findings that share a real shape to infer anything, on a site with no numbered or prefixed content it simply won't generate candidates, which is correct behavior, not a gap

• Hidden parameter discovery only checks a curated list of common names, not a real brute force, and it multiplies request volume per finding, which is why it's off by default

• Content classification and interest scoring are pattern-matching against known signatures and keywords, not semantic understanding, a page that discusses "passwords" in a blog post about security best practices scores the same as one that actually exposes one, treat the score as a sort order for triage, not a verdict

• Adaptive evasion covers a handful of encoding tricks that defeat naive, exact-match blocking rules, it won't get past a properly configured commercial WAF, and it's off by default since it's a more aggressive posture than passive backoff alone

• The directory-recursion heuristic (any extensionless hit with a 200 status gets the wordlist run against it again) is intentionally aggressive, closer to feroxbuster's default behavior than a conservative one, on a target with a lot of extensionless real pages this means more requests, the noise filter is what keeps that from spiraling, not a request budget

• Fuzzy similarity in both the baseline and the runtime noise tracker is a real technique but not infallible, a soft-404 page that varies enough (a completely different random sentence each time, not just a swapped id) could slip through the similarity check, the hard ceiling in dedupe.py exists specifically to catch that case even when the similarity check alone wouldn't

• Content-based discovery only reads text/HTML/JSON responses, binary files (images, archives) aren't parsed for embedded paths

• --depth bounds both recursion into found directories and how far a chain of discovered links can lead, a link found on a page found via another link won't be followed forever

License

MIT