Expand engine (directory scan, NDJSON/CSV, severity filters), docs, tests, and CI

[cognis-digital]

Motivation

yararun is a dependency-free, pure-stdlib YARA-subset rule engine and triage
toolkit. Its own tagline is "run rules over a directory", yet the CLI could
only scan individual files, and machine consumers were limited to JSON/SARIF.
This PR closes that gap and levels the repo up across four areas — features,
tests/CI, docs, and internal structure — without removing a single existing
entry point, API, or output shape. Everything is additive and the core stays
standard-library only.

What changed

Features / depth

• Directory & recursive scanning. New module yararun/scanfs.py with
  iter_targets() expands a mix of files, directories, and - (stdin) into a
  de-duplicated, stably-ordered stream of file paths. It honours:
  • recursion control (-R/--recursive, --no-recursive),
  • fnmatch --include / --exclude globs (matched against basename and
    the path relative to the walked root; exclude wins),
  • a per-file --max-bytes ceiling so a recursive scan can't be derailed by a
    multi-GB artifact,
  • optional --follow-symlinks,
  • a WalkStats counter surfaced via --stats (files/dirs/skips to stderr).
    scan now accepts directories in addition to files/stdin.
• Two new output formats.
  • --format ndjson — one compact, key-sorted JSON object per file, one per
    line (streaming/log-pipeline friendly) via core.to_ndjson.
  • --format csv — RFC-4180 CSV, one row per matched string, with a summary
    row even for clean files, via core.to_csv (stable CSV_COLUMNS).
• Severity controls. --min-severity display filter backed by the pure,
  non-mutating ScanResult.filtered(), plus severity_rank() /
  severity_at_least() helpers in core.

Tests & CI

• New tests/test_expansion.py (29 tests) with real assertions on real
  behaviour: severity helpers and their medium fallback, filtered()
  narrowing + non-mutation + intel preservation, NDJSON/CSV exporter shape and
  RFC-4180 quoting, the full iter_targets matrix (recursion, include/exclude
  precedence, size cap, dedup, stdin passthrough, missing paths, stats), and the
  new CLI surface (directory scans, --no-recursive, include glob, ndjson/csv,
  --min-severity, --max-bytes, --stats, single-file back-compat).
• Full suite: 131 passing (102 pre-existing + 29 new), all offline.
• CI upgraded (.github/workflows/ci.yml): pytest matrix on Python
  3.10-3.13 with PYTHONUTF8=1, plus a ruff lint job — a byte-compile
  syntax gate and a blocking ruff check --select E9,F63,F7,F82 (real defects
  only: syntax errors, undefined names, broken f-strings, bad comparisons), with
  a full non-blocking ruff report. ruff>=0.6 added to the dev extra.

Docs

• README overhauled from the archive stub into a full guide: overview, why,
  feature list, install (incl. extras), quick start, per-subcommand usage
  tables, output-format table, rule-writing, architecture diagram, Python API,
  configuration + exit-code reference, and a FAQ. The huntkit pointer is kept.
• New docs/USAGE.md — complete CLI reference, the supported rule-language
  grammar, exit codes, and recipes.
• Refreshed docs/ARCHITECTURE.md (accurate module/data-flow map) and
  ROADMAP.md (principles, shipped/near/mid/long-term, non-goals).

Refactor

• New modules are fully type-annotated and docstring'd; json import added to
  core for the exporters; new public names exported from yararun/__init__.
• No behaviour change to the parser, evaluator, scanner, or existing renderers.
• Version bumped 2.0.0 -> 2.1.0 (core.TOOL_VERSION, pyproject, VERSION).

Test results

$ PYTHONUTF8=1 python -m pytest -q
131 passed

$ ruff check --select E9,F63,F7,F82 yararun tests
All checks passed!

$ pip install -e .  &&  yararun --version
yararun 2.1.0

New usage examples

# recursive tree scan, only scripts, cap 5 MB, high+ only, as CSV
yararun scan ./mail-dump --include '*.ps1' --include '*.vba' \
    --max-bytes 5000000 --min-severity high --format csv > droppers.csv

# top-level only, print a walk summary to stderr
yararun scan ./incoming --no-recursive --stats

# stream findings as NDJSON into a log pipeline
yararun --format ndjson scan ./corpus | your-log-shipper

# unchanged: single file still yields a JSON object, stdin still works
yararun --format json scan suspicious.bin
cat payload | yararun scan -

Backward compatibility

Additive only. No files deleted, no public API or CLI flag removed, no
output shape changed. A single explicit file target still yields a JSON object
(array only for multiple/expanded targets); -/stdin, --fail-on, and every
existing subcommand behave exactly as before. A missing top-level target still
exits 2; the core/CLI remain standard-library only.

Checklist

• Additive - no removals of files, APIs, or behaviour
• Full test suite green (131 passing) with PYTHONUTF8=1
• New functionality is documented and tested
• CI installs deps and runs tests + lint on push/PR
• pip install -e . works; console script runs
• Original naming; no competitor references introduced