HTTP Bot Defense Lab

A synthetic, defensive environment for reasoning about automated abuse on an HTTP marketplace. The project covers more than classification: it includes traffic generation, request-time policy, longer-window behavioral correlation, replay evaluation, analyst operations, observability, and cost assumptions.

It never connects to or automates a real application. Every account, device, network, item, and transaction is generated locally.

Why this project exists

Bot defense is an operating system, not a single model. A useful design must answer several questions at once:

• What can be decided safely in the request path?
• Which evasive behaviors require longer observation or entity correlation?
• How do we avoid treating shared devices or networks as proof of abuse?
• What evidence will an analyst see before introducing user friction?
• How do thresholds change false positives and missed abuse?
• What happens when telemetry, state, or a detector is unavailable?
• What does a candidate policy change on the exact same event stream?
• Can review, appeals, and user-friction costs sustain the intervention?

This repository makes those decisions executable and reviewable under one explicit assumption: the defender can observe server-side HTTP and marketplace events.

POC assumptions

These are working hypotheses used to make the prototype concrete. They are not claims about any production platform.

Category	POC assumption	What must be validated in production
Product boundary	The abuse surface is a companion-style marketplace whose meaningful actions reach a server-controlled HTTP API. Gameplay input automation, console memory, and proprietary non-HTTP protocols are outside scope.	Inventory the actual routes, protocols, services, and ownership boundaries.
Identity	Requests can be associated with authenticated account and session identifiers.	Review authentication semantics, account sharing, session rotation, and privacy constraints.
Device signal	A server-issued pseudonymous device identifier may be available, but it can be unstable or spoofed and is never proof by itself.	Measure stability, collision, replacement, emulator, and spoof-resistance behavior.
Network signal	A coarse network identifier can be derived, but households, VPNs, universities, carrier NAT, and IPv6 make it weak evidence.	Establish lawful derivation and retention, then measure legitimate fanout patterns.
Observability	Server-side timestamps, normalized endpoint/method, response outcome, request duration, and minimal marketplace-action metadata are available. No screen, input-device, or client-process visibility is assumed.	Confirm event completeness, clock behavior, missingness, retries, and schema quality.
Online budget	Request-time scoring should be bounded and add less than 10 ms p95. Most detector failures should fail open and alert.	Set endpoint-specific latency, availability, and failure-mode requirements.
Enforcement	Responses can progress from observe to throttle, challenge, temporary restriction, and review. Permanent suspension requires stronger governance than this POC provides.	Align actions with product policy, accessibility, support, legal, appeals, and rollback procedures.
Error cost	A false positive that adds friction to a legitimate user is assumed to cost more than one missed suspicious request.	Quantify harm by endpoint, intervention, user segment, and abuse outcome.
Data and scale	Raw request metadata should have short retention; traffic volume, burst shape, review capacity, and infrastructure cost are unknown.	Measure the real workload and establish purpose-specific retention and aggregation.
Evaluation	Synthetic profiles and exact labels are sufficient to test mechanics and known hypotheses, but not effectiveness. Score 60 is a demonstration threshold, not a calibrated production threshold.	Evaluate reviewed outcomes with temporal/account holdouts, observed base rates, ambiguity, drift, and appeal data.
Shadow policy	policy-v1 controls the HTTP response; policy-v2 receives the same observation and is recorded only as a counterfactual. Candidate sensitivity and cross-network correlation are hypotheses, not approved policy.	Build a governed registry, assignment, holdout, exposure, rollback, and last-known-good process.
Adversarial testing	Ten bounded synthetic scenarios are enough to expose known failure modes such as jitter, cover traffic, identifier rotation, and short bursts.	Red-team observed abuse journeys continuously without turning production telemetry into an offensive automation asset.
Operations model	Daily accounts are divided into bot/legitimate populations. Recall and FPR are account-level rates; intervention, review referral, challenge completion, and appeals are separate stages.	Reconcile identities, windows, repeated flags, prioritization, queue age, and outcome maturity against actual systems.
Economics	Harm, user-friction, and analyst costs are user-entered illustrative units for scenario comparison.	Agree on a harm taxonomy and measure support, accessibility, retention, transaction, and reviewer costs.

The fuller assumptions register pairs each hypothesis with a validation step. The threat model defines the explicit in-scope and out-of-scope behaviors.

Demonstrated result

The deterministic seed-42 scenario produces 2,680 requests from 74 accounts:

Population	Accounts	Request-time detected	Layered detected
Casual and power users	50	0	0
Fixed-loop automation	10	10	10
Randomized long-running automation	5	0	5
Low-rate coordinated automation	9	0	9

At the default score threshold of 60, the request-time layer has 41.67% recall in this constructed population. Adding asynchronous behavioral analysis raises the acceptance result to TP=24, FP=0, TN=50, FN=0. Lowering the threshold to 30 flags three legitimate household accounts that share a device, exposing the false-positive tradeoff instead of hiding it.

These are deterministic acceptance-test results, not an estimate of production accuracy. Real calibration requires reviewed outcomes, temporal holdouts, base-rate-aware precision, shadow deployment, and appeal data.

Adversarial frontier and shadow policy

The v1.1 experiment runner sends each of ten deterministic populations through policy-v1 and the shadow-only policy-v2. It varies timing jitter, velocity, session duration, cover traffic, device/network rotation, coordinated cluster size, short bursts, and legitimate power-user intensity.

go run ./cmd/experiment

The output intentionally contains wins, misses, and policy disagreements. It reports threshold curves, detection delay, bot orders before detection, projected precision at 0.1%, 1%, and 5% prevalence, and estimated review cases per 100K accounts. Generated JSON, CSV, and Markdown live under artifacts/experiment/ and remain uncommitted.

policy-v2 is not presented as “the answer.” It is a concrete example of how to test a more sensitive policy without changing user-facing enforcement.

Cross-repository adaptive closure

The resettable scorer now runs under the separate Red Queen Lab's bounded black-box search. The first real run found a 94-action v1 evasion. A cadence mitigation caught that family but was rejected because it detected every synthetic power-trader and delegated-household seed. A narrower four-account target/side coordination hypothesis caught the exact replay, blocked 54 later market actions, and reduced the trace's synthetic realized profit from 2,819 to 1,200 with zero friction on the three declared hard-negative personas.

Fresh adaptive search still found a profitable three-account evasion, now preserved as the next minimized failure. This is deliberately reported as a moving frontier, not a solved detector. See adaptive detector closure for the method, results, and comparison limits.

Anti-Abuse Operations Lab

Detector accuracy is only one input to a viable program. A separate interactive simulator converts account volume, prevalence, recall/FPR, intervention coverage, challenge completion, review capacity, appeals, and unit economics into daily operating outcomes.

go run ./cmd/operations

Open http://127.0.0.1:8082. The summary prioritizes three KPIs: harm prevented, legitimate friction per 10K accounts, and review-capacity utilization. All inputs are visible, formulas are documented, and preset thresholds/costs are explicitly illustrative rather than Sony or PlayStation claims.

Architecture

flowchart LR
    S[Deterministic traffic simulator] -->|HTTP| A[Mock marketplace API]
    A --> P[Request-time policy]
    P -. same observation .-> SP[Shadow policy]
    P --> D{Decision}
    D -->|allow / observe| R[Response]
    D -->|throttle / challenge / deny| R
    P --> E[Append-only HTTP events]
    E --> B[Async behavioral analysis]
    B --> C[Account assessments]
    E --> Q[Replay evaluation]
    C --> Q
    C --> U[Truth-blind analyst console]
    E --> U
    T[Separate simulator truth] --> Q
    Q --> O[Operations scenario model]

Loading

The online path uses 64 account-keyed shards, bounded samples and account cardinality, idle-state expiry, duplicate idempotency, and explicit late-event normalization. The offline path adds long-session features and account/device/network correlation. Shared network fanout contributes no score on its own; synchronized behavior or other evidence must accompany it.

HTTP attempts keep unique request IDs. The engine and authorized loopback adapter accept explicit observation IDs for retry tests, but the public mock API does not trust a client-supplied idempotency key.

Run the demo

Requires Go 1.24 or later.

git clone https://github.com/beejmaxx/http-bot-defense-lab.git
cd http-bot-defense-lab
go run ./cmd/demo

The command prints both the request-time baseline and layered result, followed by a threshold sensitivity table. It writes generated JSONL beneath artifacts/demo/, which is excluded from version control.

Start the investigation console:

go run ./cmd/console

Open http://127.0.0.1:8081. The console can inspect HTTP evidence, record a reasoned reversible intervention, and review the action audit. It never reads the simulator truth file.

Other useful commands:

# Re-evaluate saved evidence at another threshold
go run ./cmd/replay -threshold 40

# Explore a configurable telemetry cost envelope
go run ./cmd/cost -rps 10000 -retention-days 7

# Stress both policy versions and generate experiment artifacts
go run ./cmd/experiment

# Explore harm, user-friction, review, and appeal scenarios
go run ./cmd/operations

# Run the standalone marketplace API and /metrics endpoint
go run ./cmd/api

# Serve a resettable loopback-only scorer for Red Queen Lab
go run ./cmd/scorer -policy v1
# Alternatives for documented experiments: -policy v3 or -policy v4

# Verify correctness and concurrency
go vet ./...
go test ./...
go test -race ./...

# Measure request-time scoring
go test -bench=. -benchmem ./internal/policy

Detection layers

Request-time features include bounded endpoint velocity, order velocity, timing periodicity, continuous operation, and identity completeness. Decisions carry stable reason codes and map to allow, observe, throttle, challenge, or deny.

Asynchronous features include order-action dominance, active duration, device fanout, and synchronized actions among accounts sharing infrastructure. This layer is intentionally truth-independent; only the evaluator joins its output to the separate labels.

Repository map

Path	Responsibility
cmd/demo	Run the complete deterministic acceptance scenario
cmd/api	Serve the standalone mock marketplace and metrics
cmd/console	Serve the analyst investigation experience
cmd/replay	Re-evaluate persisted evidence
cmd/cost	Model traffic, retention, storage, and policy CPU
cmd/experiment	Run adversarial scenarios and compare versioned policies
cmd/operations	Serve the interactive anti-abuse operations simulator
cmd/scorer	Serve the resettable loopback-only adversarial-test contract
internal/policy	Explainable request-time scoring and enforcement
internal/behavior	Longer-window features and entity correlation
internal/simulator	Reproducible human and automation profiles
internal/evaluation	Confusion matrix and detection-delay reporting
internal/experiment	Evasion matrix, threshold/base-rate projections, and reports
internal/operations	KPI model, presets, sensitivity table, and dashboard
internal/console	Truth-blind queue, evidence timeline, and audit UI

Engineering choices

• Go standard library first. The core has no third-party runtime dependencies and builds as small binaries.
• Rules before opaque models. The first objective is observable policy behavior, trustworthy evaluation, and useful reason codes.
• Labels stay separate. Profile names never appear in HTTP events, assessments, account IDs, or the analyst console.
• Progressive enforcement. The console supports reversible actions and intentionally omits permanent suspension.
• Unknown scale is parameterized. The cost command measures the local event representation and applies configurable workload assumptions.
• Infrastructure follows evidence. Kafka, Redis, Kubernetes, and model serving are deferred until traffic, consistency, recovery, and ownership requirements justify them.

Honest limitations

• All behavior is synthetic and hypothesis-driven.
• Marketplace state and analyst actions are in memory.
• JSONL is used for inspectable demo persistence, not storage efficiency.
• The bounded asynchronous JSONL path can drop newest events under pressure and exposes that loss in metrics; it is not a crash-durable transactional outbox.
• The local analyst console has no authentication or durable case store.
• Policies have not been calibrated against reviewed real-world outcomes.
• Shadow comparison is local; durable assignment, policy registry, exposure logging, multi-instance state routing, and production rollback are not implemented.
• Operations inputs and sensitivity curves are illustrative assumptions, not forecasts.

The production path would begin with telemetry discovery and shadow evaluation, then add durable delivery, governed policy versions, authenticated case management, accessibility review, staged interventions, and rollback criteria.

Documentation

• Case study
• Threat model
• Architecture
• Evaluation methodology
• Adaptive detector closure
• KPI framework and formulas
• 30/60/90-day program plan
• Assumptions register
• Privacy and responsible enforcement
• Operational model
• Five-minute walkthrough

Safety

This is a defensive simulation. Its traffic generator is coupled to the local mock contract and is not an automation tool for third-party services. Please report security issues using the process in SECURITY.md.