Add CTF automation toolkit: auto-solvers, categorization, hints, and flag submission

[Copilot]

Transforms ShadowParse into a complete CTF automation toolkit by adding intelligent challenge analysis, automated solving techniques, and platform integrations.

New Modules

ctf_solvers.py - Auto-solver library

• ROT1-25 cipher testing with readability scoring
• Single/multi-byte XOR bruteforce with frequency analysis
• Substitution cipher solver using English frequency patterns
• CyberChef-style encoding chains (Base64→Gunzip→Hex, etc.)
• QR/barcode detection and decoding
• Shellcode pattern detection and binary disassembly (i386, amd64, arm, aarch64, mips)

ctf_categorizer.py - Challenge classification

• Weighted scoring across 6 categories: Crypto, Forensics, Stego, Web, Network, Reverse
• Primary/secondary category detection with confidence scores
• Indicator-based analysis (entropy, protocols, file types, traffic patterns)

hint_engine.py - Progressive hint generation

• Context-aware hints with 3 difficulty levels
• Category-specific tool recommendations
• Customizable via hints.json

ctf_submission.py - Platform integrations

• CTFd, HackTheBox, and generic webhook support
• Configurable auto-submit with confirmation prompts
• Submission logging and error handling

Integration

Extended ShadowEngine with opt-in CTF mode:

engine = ShadowEngine(
    pcap_path, 
    ctf_mode=True,
    auto_solve=True,
    config_path="ctf_config.yaml"
)

CLI additions:

python shadowparse.py -f capture.pcap --ctf-mode --auto-solve --hints

New outputs when CTF mode enabled:

• ctf_analysis.md - Category, confidence, hints, tools
• auto_solver_results.txt - Decode attempts with confidence scores
• suggested_tools.txt - Category-based recommendations
• flag_submissions.log - Submission attempts and results

Key Design Decisions

• Graceful degradation: CTF features optional, fails silently if dependencies missing
• Size/depth limits on auto-solvers to prevent resource exhaustion
• Hex payload validation before parsing to prevent ValueError crashes
• Architecture validation for disassembly (defaults to i386 for unknown)
• Consistent confidence score formatting (0-1 range, displayed as percentage)

Dependencies

Added optional dependencies (tool works without them):

pwntools>=4.11.0, qrcode>=7.4.2, pillow>=10.0.0, 
pyzbar>=0.1.9, python-magic>=0.4.27, requests>=2.31.0, pyyaml>=6.0.1

Backward compatible - existing usage patterns unchanged.

Original prompt

Objective

Enhance ShadowParse with CTF-specific features to make it a complete CTF automation toolkit. This includes integrating auto-solving capabilities, intelligent challenge categorization, a hint system, and optional flag submission to popular CTF platforms.

Features to Implement

1. Auto-Solver Library Integration

Create a new module ctf_solvers.py that integrates with popular CTF tools:

• CyberChef Integration:

  • Integrate common CyberChef recipes for automatic decoding chains
  • Add "magic" auto-detection similar to CyberChef's magic operation
  • Common recipes: From Base64 → Gunzip → From Hex chains

• pwntools Integration:

  • Add pwntools utilities for binary analysis if executables are extracted
  • Auto-detect shellcode in payloads
  • Provide disassembly of extracted binaries

• Common CTF Tricks:

  • ROT-all: Try all ROT variations automatically
  • XOR bruteforce: Single-byte XOR key search
  • Substitution cipher solver with frequency analysis
  • QR code detection and decoding from extracted images
  • Barcode detection and decoding

2. Challenge Categorization System

Add intelligent detection to categorize the PCAP into CTF challenge types:

• Cryptography: High entropy, encoded data, cipher patterns
• Forensics: File carving, hidden data, metadata analysis
• Steganography: Image files, audio files, LSB patterns
• Web Exploitation: HTTP traffic, SQL injection patterns, XSS
• Network Analysis: DNS tunneling, covert channels, protocol abuse
• Reverse Engineering: Binary extraction, shellcode, obfuscated code

Implement scoring system to determine primary and secondary categories.

3. Intelligent Hint System

Create a hint_engine.py that provides progressive hints based on findings:

Example hint logic:

- If high entropy data found but no decodes successful → "Consider trying XOR with common keys or look for repeating patterns"
- If Base64 detected but contains non-printable after decode → "Try treating the decoded data as binary (image, archive, executable)"
- If DNS queries to unusual domains → "Check for DNS tunneling or exfiltration"
- If multiple encoding layers detected → "Consider automation with CyberChef or recursive decoding"
- If image files extracted → "Run steganography tools (steghide, zsteg, stegsolve)"
- If flags found but look incomplete → "Check for multi-part flags across different protocols"

Hints should be:

• Progressive (basic → advanced)
• Context-aware based on what was already found
• Actionable with specific tool recommendations

4. Flag Submission Integration (Optional)

Add optional flag submission to popular CTF platforms:

• CTFd API Integration:

  • Allow users to configure CTFd instance URL and API token
  • Auto-submit flags when found
  • Report submission success/failure

• HackTheBox API Integration:

  • Support HTB machine flag submission
  • Handle both user.txt and root.txt flags

• Generic Webhook Support:

  • Allow custom webhook URLs for flag submission
  • Support custom headers and authentication

Configuration via config file (ctf_config.yaml):

ctf_platform:
  enabled: true
  platform: "ctfd"  # or "htb", "webhook"
  url: "https://ctf.example.com"
  api_token: "your-token-here"
  auto_submit: false  # require confirmation before submitting

5. Enhanced Reporting for CTF

Update the reporting system to include CTF-specific sections:

• Challenge Category: Display detected category with confidence scores
• Auto-Solver Results: Show results from automated solving attempts
• Hints Section: Progressive hints based on findings
• Tool Recommendations: Suggest next tools to run
• Flag Summary: Dedicated section with all flags and submission status
• Cheat Sheet: Quick reference for common CTF commands based on findings

6. New CLI Arguments

Add new command-line options:

python shadowparse.py -f evidence.pcap --ctf-mode --auto-solve --submit-flags --config ctf_config.yaml

Options:

• --ctf-mode: Enable all CTF-specific features
• --auto-solve: Run auto-solver tools
• --submit-flags: Auto-submit found flags (requires config)
• --hints: Show progressive hints
• --config: Path to CTF configuration file

7. Dependencies to Add

Update requirements.txt with:

pwntools>=4.11.0
qrcode>=7.4.2
pillow>=10.0.0
pyzbar>=0.1.9
python-magic>=0.4.27
requests>=2.31.0
pyyaml>=6.0.1

Implementation Notes

• All CTF features should be opt-in to avoid bloating basic usage
• Auto-solvers should run safely without executing arbitrary code
• Flag submission should require explicit user confirmation by default
• Hints should be stored in a separate JSON file for easy customization
• Challenge categorization should use weighted scoring across multiple indicators
• All external API calls should have proper error handling and timeouts
• Add comprehensive logging for auto-solver attempts

Output Enhanc...

This pull request was created from Copilot chat.

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.