Step 14d/23 (Phase 3): Signatures — Implement aggregator script (normalize + deduplicate + publish)

[Jordan231111]

Summary

Create the aggregator script that combines signatures from all fetcher scripts, normalizes, deduplicates, and outputs the final combined signature database with manifest.

Depends On: Step 14b (ClamAV fetcher), Step 14c (MalwareBazaar fetcher)
Parent Issue: #12 (Integrate Real-World Malware Signatures)

Context

Individual fetcher scripts produce per-source signature files. The aggregator combines them into a single database that the Rust application can consume.

Key Changes

1. Create/update `python_scripts/generate_real_signatures.py`:
   • Import output from all fetcher scripts (ClamAV, MalwareBazaar, PhishTank if approved)
   • Normalize all entries to unified schema
   • Deduplicate on `hash_type + hash + signature_type`
   • Always include EICAR test signature (SHA-256, public domain)
   • Generate `signatures.db` in the format expected by `DatabaseManager`
   • Generate `manifest.json`: version (YYYY.MM.DD), SHA-256 checksum, signature count, sources list
2. Add `--output-dir` flag for output directory
3. Print summary: per-source counts, total unique signatures, duplicates removed
4. Validate output: fail if combined DB has zero signatures

Acceptance Criteria

• Aggregator combines signatures from all approved sources
• Deduplication removes redundant entries across sources
• EICAR test signature is always included
• Output matches the format expected by the Rust `DatabaseManager`
• Manifest includes version, checksum, signature count, and sources
• Summary prints per-source counts and deduplication stats
• Validation: aggregator fails if output would be empty