SPECTER: Wire API-Powered AI Threat Intelligence Platform

Version: v1.0.0 — see CHANGELOG.md

Real-time URL threat analysis. Wire API extracts technical signals, Google Generative AI recognizes behavioral patterns, deterministic scoring outputs risk (0-100). Async architecture handles 120s Wire API + 45s AI calls without blocking.

Live Demo

Screenshots

(Screenshots pending — see docs/screenshots/ once added.)

Why I Built This

Static URL threat detection is broken:

Blacklist-based tools miss 60%+ of new phishing campaigns
SaaS solutions cost $500/month and have 5-minute latencies
Open-source tools use only regex patterns (too many false positives)

I needed hybrid intelligence: raw technical signals (domain age, SSL validity, redirect chains) + AI pattern recognition (behavioral clustering, social engineering vectors). And it had to be fast.

The Architecture Problem

The Issue: Wire API takes 120s, Google Generative AI takes 45s. If I block the request thread waiting for both, the user sits staring at a loading spinner for 175 seconds.

The Solution: Async worker architecture.

POST /api/investigations/start → returns investigation ID immediately (300ms)
Background: Wire API (120s) → AI Analysis (45s) → Threat Scoring (10s)
Frontend polls GET /api/investigations/:id every 2s with 3-min graceful timeout
Status persisted: processing → completed
User gets results without waiting (8-15s typical, 180s max)

This pattern scales. Investigate 50 URLs and come back later. No polling hell, no WebSocket complexity.

Tech Stack

Frontend: React 18 + Vite (3-4x faster than Webpack) + Tailwind + Framer Motion
Backend: Node.js/Express + MongoDB + Mongoose + Helmet.js + express-rate-limit
Intelligence: Wire API (technical metadata) + Google Generative AI (pattern analysis)
Auth: JWT (30-day expiry) + bcryptjs (salt: 10) + input validation
Deployment: Vercel (frontend) + Render (backend) + MongoDB Atlas (database)

Installation & Setup

Prerequisites

Node.js v18+, npm/yarn, MongoDB (Atlas free tier works)

Clone & Install

git clone https://github.com/anasahhm/specter.git
cd specter

# Backend
cd backend
npm install
cp .env.example .env  # Add your API keys

# Frontend
cd frontend
npm install
cp .env.example .env

Environment Variables

Backend (.env): — see backend/.env.example for the authoritative, documented list

NODE_ENV=development
PORT=5000
MONGODB_URI=mongodb+srv://user:password@cluster.mongodb.net/specter
JWT_SECRET=your-super-secret-key-minimum-32-characters
CLIENT_URL=http://localhost:5173
SERVER_URL=http://localhost:5000
FRONTEND_URL=http://localhost:5173
WIRE_API_KEY=your-wire-api-key-here
ANTHROPIC_API_KEY=optional-falls-back-to-rule-based-analysis

# Stripe (optional — omit to run free-tier-only; billing routes return 503)
STRIPE_SECRET_KEY=sk_test_...
STRIPE_PUBLISHABLE_KEY=pk_test_...
STRIPE_WEBHOOK_SECRET=whsec_...
STRIPE_PRICE_ID=price_...   # Specter Pro, $1.99/month

Frontend (.env): — see frontend/.env.example

VITE_API_URL=http://localhost:5000/api

Run Locally

Terminal 1 (Backend):

cd backend && npm run dev
# Should output:
# ╔══════════════════════════════════════════╗
# ║        SPECTER - SERVER STARTED          ║
# ║ Port: 5000 | Database: Connected         ║
# ║ Wire API: ✓ | Google AI: ✓               ║
# ╚══════════════════════════════════════════╝

Terminal 2 (Frontend):

cd frontend && npm run dev
# http://localhost:5173

Terminal 3 (Test API):

curl http://localhost:5000/api/health
# {"status":"operational","timestamp":"2024-05-31T12:00:00.000Z"}

Stripe Setup (optional — for Specter Pro billing)

Create a Stripe account and switch to test mode.
Products → create Specter Pro, $1.99/month recurring → copy the Price ID into STRIPE_PRICE_ID.
Developers → API keys → copy the secret key into STRIPE_SECRET_KEY.
Developers → Webhooks → add endpoint {SERVER_URL}/api/billing/webhook, select the events listed under Webhooks Handled → copy the signing secret into STRIPE_WEBHOOK_SECRET.
For local development, use the Stripe CLI to forward events: stripe listen --forward-to localhost:5000/api/billing/webhook.

Without these variables set, the app runs free-tier-only: billing routes respond 503, every user gets 7 free investigation credits, and no upgrade path is offered.

How It Works

Architecture Overview

┌────────────┐      ┌──────────────────┐      ┌───────────────────────┐
│  Frontend   │ ──▶ │  Backend (API)    │ ──▶ │  MongoDB               │
│  React+Vite │ ◀── │  Express (ESM)    │ ◀── │  Investigations, Users │
│  nginx (prod)│     │  JWT + Helmet     │      │  Reports, Webhooks     │
└────────────┘      └────────┬─────────┘      └───────────────────────┘
                              │
                ┌─────────────┼─────────────┬───────────────┐
                ▼             ▼             ▼               ▼
          Wire API      Claude/Gemini    Stripe          Threat
        (scraping)     (pattern rec.)  (billing)     Scoring (rule-based)

Investigations run as an async background pipeline (Wire → AI → Threat Scoring) so the initial POST /api/investigations/start returns in ~300ms; the frontend polls for completion. See The Architecture Problem below for why.

3-Stage Pipeline

Step 1: Wire API (120s timeout)

Domain metadata, SSL certificates, age, MX records
Redirect chains, technology stack detection
Embedded links, forms, scripts
Output: Raw technical signals

Step 2: AI Analysis (45s timeout)

Google Generative AI pattern recognition
Behavioral clustering against known threats
Phishing vector identification
Confidence scoring and summary generation
Fallback: Rule-based analysis if AI unavailable

Step 3: Threat Scoring (10s timeout)

Risk score (0-100)
Threat classification (Critical/High/Medium/Low/Safe)
Scam probability, toxicity rating, confidence
Output: Final verdict

Data Flow

1. User submits URL
2. POST /api/investigations/start
3. Backend returns investigationId (status: processing)
4. Frontend polls GET /api/investigations/:id every 2s
5. Background: Step 1 → Step 2 → Step 3
6. Status changes to completed
7. Frontend renders results

Gotchas I Solved

1. Slow External APIs Without Blocking

Problem: Wire API + AI = 165s. Blocking the request thread kills UX.
Solution: Async workers + polling. POST returns instantly with ID, frontend polls every 2s.
Lesson: For external APIs >10s, always use async + polling or WebSockets.

2. Rate Limit Exploitation

Problem: Users hammer the API. Bots scrape URL intelligence.
Solution: Dual-axis rate limiting:
- Global: 100 requests/15min (catches distributed attacks)
- Per-user: 5 investigations/min (prevents individual abuse)
- Sliding window (not fixed buckets)
Lesson: Single rate limit is insufficient. Attack from one user looks different than botnet traffic.

3. External API Resilience

Problem: What if Wire API is down? What if Google AI returns an error?
Solution: Graceful degradation:
- Wire API failure → Use cached domain reputation data
- Google AI timeout → Fall back to rule-based threat scoring
- Both failures → Return partial results with explicit warnings
Lesson: Single point of failure cascades. Build fallbacks at every layer.

4. JWT Token Expiry Handling

Problem: Users investigate for hours but tokens expire after 30 days.
Solution: Token refresh pattern:
- 30-day access tokens + refresh tokens
- Frontend axios interceptor refreshes automatically
- No sensitive data in error messages
Lesson: Never leak token details in error responses.

5. JavaScript-Heavy Sites

Problem: Wire API sees static HTML. Dynamic forms, obfuscated links, JS-rendered content are invisible.
Solution: Hybrid approach:
- Wire API for structural/technical analysis
- Google AI for behavioral/pattern analysis
- Triangulation catches what either misses
Lesson: No single tool is complete. Combine strengths.

6. MongoDB Connection Pooling

Problem: Mongoose default pool size (5) was too small under concurrent load.
Solution: Tuned pool settings in connection URI, added connection monitoring.
Lesson: Database bottlenecks surface under load, not in dev.

API Reference

Authentication

POST /api/auth/register
  { email, password, displayName? }

POST /api/auth/login
  { email, password }

GET /api/auth/profile
  Headers: Authorization: Bearer {token}

Investigations

POST /api/investigations/start
  { targetType: "url", targetValue: "https://..." }
  Returns: { investigationId, status: "processing" }

GET /api/investigations/:investigationId
  Returns: Complete threat analysis

GET /api/investigations?page=1&limit=10
  Returns: User's investigation history

PUT /api/investigations/:investigationId/bookmark
  { isBookmarked: boolean }

Reports & Analytics

GET /api/reports/:investigationId
POST /api/reports/:reportId/export        { format: "pdf" | "markdown" | "json" }
GET /api/reports/:reportId/download/:format

GET /api/analytics/overview
GET /api/analytics/timeline

Billing (Stripe)

GET /api/billing/status
  Returns: { plan, subscriptionStatus, creditsRemaining, creditsUsed, creditsLimit,
             currentPeriodEnd, cancelAtPeriodEnd, stripeConnected }

POST /api/billing/checkout
  Returns: { checkoutUrl }   — redirect the user here to start a Specter Pro subscription

POST /api/billing/portal
  Returns: { portalUrl }     — Stripe-hosted billing portal (invoices, payment method)

POST /api/billing/cancel     — cancels at the end of the current billing period
POST /api/billing/resume     — undoes a pending cancellation

POST /api/billing/webhook    — Stripe-only, signature-verified, not for direct use

Webhooks Handled

checkout.session.completed, customer.subscription.created, customer.subscription.updated, customer.subscription.deleted, customer.subscription.paused, customer.subscription.resumed, invoice.paid, invoice.payment_failed — all idempotent via a persisted WebhookEvent record per Stripe event ID.

Credits & Subscriptions

Every new account gets 7 free investigation credits. One credit is consumed exactly when an investigation successfully completes — never on start, never on failure, and never twice (guarded by an idempotency flag on the investigation record). Free users with 0 credits remaining are blocked from starting new investigations (402 CREDITS_EXHAUSTED) and shown an upgrade prompt.

Specter Pro — $1.99/month via Stripe Checkout — removes the credit limit entirely: unlimited investigations, reports, and analytics, with no deductions.

State	Behavior
Free, credits > 0	Normal use, credit deducted on each completed investigation
Free, 0 credits	Blocked from new investigations, upgrade modal shown
Pro (active/trialing)	Unlimited, no deductions
Cancelled (pending)	Stays Pro until the current period ends (`cancelAtPeriodEnd`)
Cancelled/Expired	Reverts to free tier, credits reset to 7
Past due	Treated as non-Pro until payment succeeds

Threat Metrics

Metric	Range	Meaning
Risk Score	0-100	Overall threat severity
Threat Level	Critical/High/Medium/Low/Safe	Classification
Phishing Detected	Yes/No	Known phishing patterns
Scam Probability	0-100%	Fraudulent intent likelihood
Toxicity Score	0-100	Content toxicity
Confidence Score	0-100%	Analysis certainty

Performance

Metric	Target	Actual
Page Load	<2s	1.2s
Investigation Start (API)	<500ms	300ms
Results Available	<30s	8-15s
API Response Time	<1s	200-400ms
Database Query	<100ms	50-80ms

Security

JWT Auth - 30-day token expiry, verified on every protected route Password Hashing - bcryptjs (salt rounds: 10) Rate Limiting - 100 req/15min global, 5/min investigation starts, 20/15min on auth endpoints Helmet.js - security headers on every response CORS - whitelist frontend origin only, credentials scoped Ownership Checks - every investigation/report query is scoped to the authenticated userId Input Validation - email format, password length, investigation target type/length Stripe Webhook Verification - signature-checked, idempotent via persisted event IDs Error Handling - consistent JSON error shape, no stack traces outside development Environment Isolation - secrets in .env, validated at startup, never logged in full

Deployment

Option A — Docker (recommended for self-hosting)

cp backend/.env.example backend/.env   # fill in real values
docker compose up --build
# frontend: http://localhost:8080 (nginx)
# backend:  http://localhost:5000

See docker-compose.yml, backend/Dockerfile, frontend/Dockerfile, and frontend/nginx.conf. MongoDB runs as its own container in the compose file for local/self-hosted use — point MONGODB_URI at Atlas instead for managed hosting.

Option B — Vercel + Render

Frontend (Vercel):

Push to GitHub
vercel.com/new → Import repo
Set VITE_API_URL env var
Deploy

Backend (Render):

render.com → Create Web Service
Connect GitHub repo
Set all env vars (see backend/.env.example)
Deploy

Database (MongoDB Atlas):

cloud.mongodb.com → Create cluster (free tier)
Get connection string
Whitelist your IP (or 0.0.0.0/0 for Render's dynamic IPs)
Set as MONGODB_URI

Project Structure

specter/
├── frontend/
│   ├── src/
│   │   ├── pages/              # Route components (Dashboard, Investigations, Billing, ...)
│   │   ├── design-system/
│   │   │   ├── ui/             # Button, Card, Table, StatusBadge, Modal, ... (barrel: ui/index.js)
│   │   │   └── layout/         # AppShell, Sidebar, TopNavbar, CommandPalette
│   │   ├── animations/         # GSAP setup + reusable hooks (useFadeUp, useStagger, ...)
│   │   ├── providers/          # ToastProvider, SubscriptionProvider
│   │   ├── context/            # AuthContext
│   │   ├── hooks/               # useAuth, useInvestigation, useApi, ...
│   │   ├── constants/           # nav.js, featureFlags.js
│   │   ├── api/                # Axios client
│   │   ├── utils/              # Report export helpers (PDF/JSON/Markdown)
│   │   └── styles/              # Global CSS + scoped design-system.css
│   ├── vite.config.js
│   ├── tailwind.config.js
│   ├── Dockerfile / nginx.conf
│   └── package.json
│
├── backend/
│   ├── src/
│   │   ├── routes/             # auth, investigations, reports, analytics, billing, webhooks
│   │   ├── middleware/         # requirePremium, requireCredits, requireSubscription
│   │   ├── services/
│   │   │   ├── billing/        # Stripe provider + provider-agnostic interface
│   │   │   ├── creditsService.js
│   │   │   ├── wireService.js / aiService.js / threatAnalysisService.js
│   │   ├── models/             # User, Investigation, ThreatReport, ActivityLog, WebhookEvent
│   │   ├── config/             # Env validation
│   │   ├── scripts/            # Database seeders
│   │   ├── server.js           # Express app + middleware wiring
│   │   └── index.js            # Entry point
│   ├── Dockerfile
│   └── package.json
│
├── docker-compose.yml
└── .github/workflows/ci.yml

Testing

Manual API Tests

# Register
curl -X POST http://localhost:5000/api/auth/register \
  -H "Content-Type: application/json" \
  -d '{"email":"test@example.com","password":"Test123!"}'

# Login
curl -X POST http://localhost:5000/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"test@example.com","password":"Test123!"}'

# Start investigation
curl -X POST http://localhost:5000/api/investigations/start \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"targetType":"url","targetValue":"https://example.com"}'

# Get results
curl http://localhost:5000/api/investigations/INVESTIGATION_ID \
  -H "Authorization: Bearer YOUR_TOKEN"

Browser Testing

Register account
Test URLs: example.com (safe), malicious-url.com (suspicious)
Verify threat scores, phishing detection, report generation
Check investigation history and bookmarks

Troubleshooting

Frontend can't reach backend

# Check backend is running
curl http://localhost:5000/api/health

# Check VITE_API_URL in frontend/.env matches backend
# Check FRONTEND_URL in backend/.env matches frontend origin (http://localhost:5173)
# Check browser console for CORS errors

MongoDB connection failed

# Verify connection string
MONGODB_URI=mongodb+srv://user:password@cluster.mongodb.net/specter

# Check IP whitelist in MongoDB Atlas (add 0.0.0.0/0 for development)
# Verify database user has correct credentials

Wire API errors

Check API key is valid and quota isn't exceeded
Review Wire API docs for rate limits
Enable debug logging: DEBUG=* npm run dev

Investigation timeout (>180s)

Default timeout is 180 seconds
Check backend logs for step-specific errors
Test with simple URL first (e.g., example.com)

Stats

48 hour build (hackathon sprint)
2100+ LOC (1200 frontend, 900 backend)
12 API endpoints (Auth, Investigations, Reports, Analytics)
3-stage pipeline (Wire API → AI → Scoring)
8-15s typical latency (8-180s max with timeouts)
4 database collections (users, investigations, reports, analytics)
18 React components (modular, reusable)
3 backend services (Wire client, AI analyzer, threat scorer)

What I Learned

Async beats blocking. External APIs >10s? Don't wait. Async + polling scales better.
Graceful degradation saves systems. When Wire API fails, use cached data. When AI times out, use rules.
Rate limiting is multidimensional. Global limits catch botnets. Per-user limits catch individual abuse.
Hybrid intelligence works. One data source has blind spots. Wire API + AI catch what each misses.
Security is layering. JWT + bcryptjs + Helmet + CORS + input validation = defense in depth.

License

MIT — see LICENSE

Roadmap

Dedicated Investigations/Threats/Entities backend endpoints (currently derived client-side from the investigations list — fine at personal scale, not ideal at high volume)
Real-time updates via WebSockets instead of polling for investigation status
Team/organization accounts (currently single-user per account)
Relationship graph visualization connecting shared entities across investigations
Grace-period handling for failed payments (currently restricts access on the first past_due webhook)
API access tier (mentioned in Settings as "planned" — not yet implemented)

Future Features

Browser extension for one-click URL investigation
Slack/Discord webhook notifications when a critical threat is detected
Bulk investigation import (CSV of URLs)
Custom risk-scoring rule configuration for Pro accounts

Known Limitations

Entities/Threats pages hydrate by fetching full investigation detail for up to ~30 records client-side rather than via a dedicated aggregation endpoint — fine for personal use, a real bottleneck if an account has thousands of investigations.
No test suite exists yet (see Testing — this documents the intended approach, not current coverage).
Payment-failure handling has no grace period beyond what Stripe's own retry schedule provides.
Wire API and Gemini AI failures fall back gracefully, but there's no per-provider circuit breaker — a fully-down upstream still incurs the full timeout on every request until it's fixed.

Deployment Notes

See docs/DEPLOYMENT.md for the full production deployment guide (Docker, reverse proxy, HTTPS, MongoDB Atlas, Stripe webhook setup end-to-end).
The Stripe webhook endpoint (/api/billing/webhook) requires the raw request body — if you put a reverse proxy in front of the backend, make sure it doesn't buffer/transform the body for that specific route.
Set MONGODB_URI to a replica-set-enabled connection (Atlas does this automatically) if you plan to scale beyond a single backend instance — this app doesn't currently use transactions, but a replica set is good practice regardless.

FAQ

Do I need Stripe to run this locally? No. Without STRIPE_SECRET_KEY set, the app runs free-tier-only — every account gets 7 investigation credits and billing routes return 503 instead of erroring.

What happens when a user runs out of credits? New investigations are blocked server-side (402 CREDITS_EXHAUSTED) regardless of what the frontend shows, and the Premium Upgrade modal is surfaced.

Why does investigation processing take so long? The Wire API step alone can take up to 120 seconds for a thorough scrape. See The Architecture Problem — it's async specifically so this doesn't block the API response.

Can I use a different AI provider instead of Gemini? The AI analysis step (services/aiService.js) already has a rule-based fallback path when no API key is configured — swapping providers means implementing an equivalent _geminiAnalysis-shaped method and pointing analyzeTargetWithAI at it.

Questions? Open a GitHub issue

Live: https://specter-weld.vercel.app

Name		Name	Last commit message	Last commit date
backend		backend
docs		docs
frontend		frontend
.dockerignore		.dockerignore
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
README.md		README.md
SECURITY.md		SECURITY.md
docker-compose.yml		docker-compose.yml

Latest commit

History

Folders and files

Repository files navigation