Skip to content

developer_guide.md

Latest commit

 

History

History
218 lines (157 loc) · 9.77 KB

developer_guide.md

File metadata and controls

218 lines (157 loc) · 9.77 KB
Raw

PhishShield-Engine: Comprehensive Developer Guide

Overview

This document provides exhaustive documentation on the architecture, command-line interfaces, security subsystems, and maintenance procedures for the PhishShield-Engine platform.

Installation & Build

1. Local Environment Setup

# Initialize Virtual Environment
python -m venv venv
venv\Scripts\activate        # Windows
# source venv/bin/activate   # macOS/Linux

# Install All Dependencies
pip install -r requirements.txt

# Run Database Migrations
alembic upgrade head

2. Environment Configuration

Copy .env.example to .env and set your configuration variables:

# Generate a cryptographically strong secret for JWT signing
python -c "import secrets; print(secrets.token_hex(32))"

In .env:

JWT_SECRET=<generated_secret>
JWT_EXPIRY_HOURS=1
LOGIN_MAX_ATTEMPTS=5
LOGIN_LOCKOUT_MINUTES=15
RESET_TOKEN_EXPIRY_HOURS=1
ENV=prod

3. Model Initialization

You must initialize your models before the first run. For a full deep-dive into the training lifecycle, see the ML Training Guide.

# Recommendation: Quick-start on 50,000 samples (fast mode)
python scripts/train_pipeline.py --generate --fast

Command Reference

I. The Management CLI (manage.py)

Used for day-to-day operations and threat intelligence management.

Command Arguments Description
serve --port <val> Launches the cinematic security dashboard.
block <domain> [--reason <str>] Manually injects a domain into the threat intelligence blocklist.
metrics (None) Aggregates local threat DB statistics and model health.

Example Usage:

python cli/manage.py block evil-hacker.com --reason "Active credential harvesting detected"

II. The Training Pipeline (train_pipeline.py)

Handles the entire ML lifecycle from raw data to registered production models.

Flag Default Description
--dataset_path data/raw/emails.csv Path to the training dataset.
--vectorizer tfidf Algorithm: tfidf, bow, or tfidf_char.
--sample_size (None) Sub-sample N rows (prevents OOM on large datasets).
--fast (False) Shortcut: 50k samples + Skip Ensemble.
--tune (False) Triggers RandomizedSearchCV tuning.
--ensemble (False) Trains model ensembles (voting or stacking).
--ensemble_kind voting Choose voting (fast) or stacking (accurate).
--generate (False) Generates synthetic phishing samples.

III. Operational Scripts

  • python scripts/retrain_scheduler.py: Background daemon that monitors data/feedback.db. Triggers retraining once new feedback exceeds the threshold.
  • python scripts/chaos_monkey.py: Injects faults (ML model corruption, load spikes) to verify graceful degradation.
  • python scripts/backup.py backup: Creates a timestamped snapshot of models, threat DBs, and configs.

Authentication & Security Subsystems

1. Authentication Architecture (src/api/auth.py)

  • Password Hashing: Utilizes bcrypt with salt generation (gensalt()). Enforces password complexity:
    • Minimum 8 characters
    • At least 1 numeric digit
    • At least 1 special character (!@#$%^&*...)
  • Session & JWT Management: Issues short-lived JWTs (default 1 hour expiry) with sub, iat, exp, and jti claims. Secrets are managed via SecretsVault with environment-variable precedence.
  • Brute-Force Lockout: Tracks failed_login_attempts per user. Reaching 5 failures locks the account for 15 minutes (locked_until).
  • Timing Attack Mitigation: Employs constant-time string comparison (hmac.compare_digest) for API key verification (pse_ prefix) and executes dummy password hashes for non-existent users.
  • Password Reset Flow: request_password_reset issues a single-use token (secrets.token_urlsafe(32)). Only SHA-256 digests (password_reset_token_hash) and expirations (password_reset_expires) are stored in the database.
  • Insecure Direct Object Reference (IDOR) Defense: All user resource queries (GET /auth/me, GET /auth/logs, GET /auth/logs/{log_id}, GET /auth/feedback/{feedback_id}, DELETE /auth/feedback/{feedback_id}) filter strictly by resource.user_id == current_user.id. Accessing unowned resources returns 404 Not Found without disclosing resource existence.

2. Rate Limiting (SlowAPI)

Auth Endpoints (per IP):

  • /auth/login: 5 requests/min
  • /auth/register: 3 requests/min
  • /auth/password-reset-request: 3 requests/min
  • /auth/password-reset: 5 requests/min

Prediction & AI Endpoints (per IP):

  • POST /predict: 15 requests/min
  • POST /analyze-security: 15 requests/min
  • POST /predict/batch: 5 requests/min
  • POST /export-report: 10 requests/min
  • POST /feedback: 10 requests/min

Global default: 60 requests/min per IP

All rate limit violations are logged as RATE_LIMIT_EXCEEDED events to logs/security_audit.log.

3. Database Schema & Migrations (src/core/database.py, alembic/)

The users table includes security governance fields:

  • username, password_hash, api_key
  • email (optional, unique)
  • is_email_verified (boolean flag)
  • failed_login_attempts (integer counter)
  • locked_until (datetime)
  • password_reset_token_hash (string)
  • password_reset_expires (datetime)

Schema revisions are maintained via Alembic (alembic/versions/8eb220da558a8768_add_auth_security_fields.py).

4. Abuse Protection & Anti-Bot Middleware (src/api/app.py)

PhishShield enforces three bot-detection layers via anti_bot_middleware applied before every API handler:

  • Honeypot Header Trap: Requests containing X-Honeypot-Trap or X-Bot-Check headers are blocked (HTTP 403). Legitimate clients never send these headers.
  • User-Agent Enforcement: All API paths require a non-empty User-Agent header. Missing headers indicate scripted or raw socket clients (HTTP 403).
  • Scanner Blacklist: Requests from known exploit and scraping tools (sqlmap, nikto, nmap, masscan, bytespider, zgrab, libwww-perl, python-urllib, dirbuster, w3af, acunetix, nessus, openvas) are blocked (HTTP 403).
  • Honeypot Payload Fields: PredictRequest and RegisterRequest include a hidden bot_trap field. Any non-empty value triggers Pydantic validation failure (HTTP 422).
  • Batch Size Cap: POST /predict/batch rejects arrays exceeding 50 emails to prevent bulk data harvesting.

All blocked events are logged to logs/security_audit.log with event types BOT_ATTACK_BLOCKED, BOT_BLOCKED_EMPTY_UA, and BOT_HONEYPOT_TRIGGERED.

4. Forensic Intelligence Layer

The calculate_security_risk function (in src/security/risk_scoring.py) executes 10 independent forensic scans:

  • Obfuscation Scan: Detects zero-width icons/markers via src/security/obfuscation_detector.py.
  • Homograph Scan: Identifies "lookalike" Unicode domains (punycode detection).
  • Mixed-Script Detection: Identifying "lookalike" characters from Latin, Greek, and Cyrillic scripts.
  • Brand Protection: Fuzzy-matches against 15+ high-value target brands.
  • Header Forensics: SPF/DKIM/DMARC validation and Return-Path analysis.
  • Domain Intel: Local SQLite lookup of known malicious domains.
  • IP-Based URLs: Detecting raw IP addresses in message links.
  • Suspicious URLs: Heuristic analysis of URL entropy and path patterns.
  • Behavioral Threat: Statistical mapping of text-based threat indicators.
  • Cyrillic URL Spoofing: Detects Cyrillic characters embedded in URLs (weight: 50).

5. Governance-as-Code (config/config.yaml)

Control the engine's behavior without modifying code:

  • Weights: Adjust the risk contribution of each forensic flag (including cyrillic_url: 50).
  • Thresholds: Define "High Risk" and "Suspicious" cutoffs.
  • Compliance: Define auto-retention days (Default: 30 days).

Production Orchestration (Docker)

The platform is designed to run as a multi-service architecture using Docker Compose:

# Launch API, Threat DB, and Retraining Scheduler (with model training)
docker-compose up --build -d

# Or build without training (use pre-trained models from models/ directory)
docker build --target base -t phishshield-base .
docker-compose up -d
  • Service API: Exposes the port 8000.
  • Service Scheduler: Runs the automated retraining watcher.
  • Persistence: Volumes are mapped for data/ and logs/ to prevent loss during restarts.
  • Training Control: Set TRAIN_MODELS=false build arg to skip training during image build (use pre-trained models).

Testing & Quality Assurance

  • Input Validation & Sanitization Tests: python -m pytest tests/test_input_validation.py -v (29 tests)
  • Auth & Security Unit Tests: python -m pytest tests/test_auth_unit.py -v (21 tests)
  • Abuse & Bot Protection Tests: python -m pytest tests/test_abuse_protection.py -v (29 tests)
  • Secrets Audit Tests: python -m pytest tests/test_secrets_audit.py -v (16 tests)
  • Full Security Test Suite: python -m pytest tests/test_input_validation.py tests/test_secrets_audit.py tests/test_auth_unit.py tests/test_abuse_protection.py -v (95 / 95 tests passing)
  • Integration Tests: python scripts/chaos_monkey.py (Simulates failure scenarios)
  • Compliance Audit: View logs/compliance.log to audit data retention and forensic overrides.
  • Security Audit Log: View logs/security_audit.log for authentication events, rate limit violations, bot blocking, and client error anomalies.
  • Warning Suppression: Tests use tests/conftest.py to suppress httpx deprecation and LGBM feature name warnings.

Maintainer

VIPHACKER100 (Aryan Ahirwar)
Cybersecurity Researcher | AI Security Lead
Last Updated: 2026-09-02