An AI agent that teaches itself to get better at a task with — zero human input.
It plays, diagnoses what went wrong, produces code-level proposals, and only keeps the ones that prove themselves in games. The task happens to be StarCraft II, but the loops are general. A live React dashboard streams every phase — every proposed fix, every rejection, every win-rate swing — while the agent runs unattended for hours. (Technically an iterative LLM-guided reinforcement learning model with a learned advisor.)
Two complementary learning loops:
- Advised loop: signal comes from playing an external static opponent (Blizzard's stock SC2 AI). Claude reads the bot-vs-stock games and proposes fixes. Improvement is measured against a fixed bar that never moves.
- Evolve loop: signal comes from self-play between our own bot versions. No external opponent. Improvement is measured against the bot's own ancestors — a moving target that gets stronger each generation.
Advised drives progress against a known benchmark; evolve keeps the bot improving past it. They catch each other's blind spots.
Self-improvement loop 6 fixes proposed, validated, committed |
Self-play arena Live evolve run — fitness pool, stack apply, regression |
Live capture Game, advisor decisions, and backend logs streaming together |
Monitoring Per-rule reward contribution over recent games |
50% → 83% win rate at SC2 difficulty 4 — reached via 6 code changes the advised loop wrote and validated itself.
v4 → v7 in one unattended 8-hour soak on headless Linux — three auto-promoted bot versions in one night (Splash readiness, defensive structures, stutter-step kiting), each validated head-to-head against its ancestor. Lineage has since extended to v10 under a parallelized 4-way evolve substrate.
Most self-improving ML systems need a human in the loop for reward design, tuning, or debugging. This one closes that loop:
- Claude reads telemetry like a reviewer — names the specific failure ("attacked before the economy caught up"), not just a scalar reward. Ranked fixes with reasoning.
- Proposals become real code. Both loops land real commits on master (
[advised-auto]or[evo-auto]). Every change must pass pytest / mypy / ruff before shipping. - Every change is validated before commit. Advised auto-reverts fixes that don't hold up over N games; evolve rolls back promotions that lose to their own parent. No silent regressions.
- You can watch it happen. The dashboard streams every phase — proposals, rejections, win-rate swings, generation outcomes — live while the agent runs unattended for hours.
The architecture is task-agnostic — SC2 is the test case because it gives fast, machine-readable outcomes.
Deep dives: Wiki · Active plan
┌──────────────────────────────────────────────────────┐
│ /improve-bot-advised │
│ autonomous learning loop (4+ hours) │
│ │
┌─────────┐ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
│ │ │ │ │ │ │ │ │ │
│ THE │◄─────────►│ PLAY │───►│ THINK │───►│ FIX │ │
│ TASK │ │ │ │ │ │ │ │ │
│ │ │ └─────────┘ └─────────┘ └────┬────┘ │
│ (SC2) │◄──┐ │ │ │
│ │ │ │ ┌─────────┐ ┌─────────┐ ┌────▼────┐ │
│ │ └──────►│ TRAIN │◄───│ COMMIT │◄───│ TEST │ │
│ │ │ │ │ │ │ │ │ │
└─────────┘ │ └─────────┘ └─────────┘ └─────────┘ │
│ │ │
│ └──────── loop back to PLAY ──────────────► │
│ (or stop if time's up / 3 fails) │
└──────────────────────────────────────────────────────┘
The loop runs unattended for hours. Each cycle: play N games, Claude reads the results and proposes ranked fixes, apply one, re-run N games to validate, commit if it held, retrain the neural policy, repeat. Quality gates (pytest/mypy/ruff), auto-rollback, a wall-clock budget, and a 3-fail cap keep it safe.
Read the full architecture: improve-bot-advised-architecture.md
The bot grows by ancestor chain — each step requires two head-to-head wins:
v0 → v1 → v2 → v3 → …
Each generation plays out as a tournament in the SC2 sandbox:
Fitness round Regression round
───────────── ────────────────
imp_1 ⚔ parent (g games)
imp_2 ⚔ parent (g games) winners
⋮ stacked ────► vN+1 ⚔ vN (g games)
imp_k ⚔ parent (g games) into vN+1 │
▼
majority wins?
┌──────┴──────┐
yes no
commit vN+1 git revert
(lineage (lineage
grows) unchanged)
A different kind of loop from the advised one — instead of validating one improvement at a time, the arena generates a pool of orthogonal candidates per generation and lets them compete. Claude proposes the pool; fitness winners get stacked into a new snapshot (with an import-check gate catching bad combinations); regression gates the promotion. Close-losers retry against the next parent; evicted imps get replaced. Every arrow in the lineage is a real auto-commit on master.
Read the mechanism: improve-bot-evolve SKILL.md · gate-reduction plan
autonomous learning loop StarCraft II game
───────────────────────── ─────────────────
PLAY → THINK → FIX → (THE TASK)
TEST → COMMIT → TRAIN
▲ ▲
│ │
observe the loop observe the game
│ │
┌────────┴────────┐ ┌────────┴────────┐
│ Advisor tab │ │ JSONL logs │
│ Evolution tab │ │ SC2 client │
│ state.json │ │ replays/ │
└─────────────────┘ └─────────────────┘
Two things are running: the loop and the task it's learning from. The dashboard has a tab tuned to each vantage point, plus a state file that's the single source of truth for which phase is executing right now. Stuck-loop detection, alert rules, and operator overrides all hang off these views.
For the advised loop that's the Advisor tab + data/advised_run_state.json; for evolve it's the Evolution tab + data/evolve_run_state.json (plus data/evolve_pool.json for the current pool and data/evolve_results.jsonl for per-phase outcomes). Same pattern, different loop.
Read the full monitoring guide: monitoring.md
Claude is good at thinking and slow at acting; real-time games need the reverse. A Claude CLI call takes seconds to respond, but the bot observes the game every ~0.5s and issues dozens of actions per second during fights.
So the tight loop runs on rule-based strategy + a PPO policy network (both decide in milliseconds). Claude is used only where latency doesn't matter: mid-game advice (fire-and-forget, rate-limited), post-hoc diagnosis between batches, and writing the fix itself.
Fast-and-dumb does the playing. Slow-and-smart does the learning.
- Original Motivation - Beat my friend at a video game using AI.
- The basic bot - A rule-based bot player that built units and hoped for the best.
- Teaching it to think - Added a neural network so it could learn.
- Learning to fight - Targeted lessons: keep the army together, deny expansions, hold the line.
- Bringing in a coach - Claude whispers strategy tips mid-match alongside human commands.
- The observation deck - A live dashboard showing every decision, reward, and training run.
- The robot that trains itself - A daemon that plays, scores, promotes winners, and rolls back duds overnight.
- The first soak test - Left it alone — 17 things broke. Fixed them all.
- Claude as personal trainer - Claude watches games, diagnoses problems, and writes the fixes itself.
- The breakthrough - Self-improvement took win rate from 0% to 75% at difficulty 3.
- Reality check - Harder opponents exposed that training scores lie — need honest exams.
- The big merge - Three plans became one: every future bot is a snapshot that must beat its ancestors.
- Memory matters - Added memory (LSTM) and a gentler training recipe — 19/20 wins at difficulty 3.
- The arena - Bot versions fight each other in sandboxed self-play. Elo ladder + promotion gate decides who ships.
- Overnight run, zero promotions - Three compound 60% filters made the math impossible, dropped the redundant composition check.
- The arena produces winners - 7 hours, v0 → v1 → v2 — the self-play loop ships its first two auto-promotions.
- Headless training substrate - Moved the heavy training off Windows onto a headless Linux runtime (massive speed increase)
- Three promotions in one night, on Linux - First end-to-end successful headless evolve. v4 → v5 → v6 → v7 in one 8-hour unattended soak
- Parallelizing the arena - Evolve now runs four candidates in parallel per generation (concurrency window + worker-slot recycling). Steps 1-7 plus iter-3 hardening shipped 2026-04-30; subsequent generations took the lineage to v10.
- Now - Self-play evolution is producing auto-promotions unattended on a parallel substrate that scales beyond a single Windows desktop. Claude proposes orthogonal improvements, the arena filters, master advances. The platform from the earlier phases now has a working growth engine on top of it — and it runs anywhere Linux + SC2 will run.
- Next - Phase O scripted Hydra v1 (themed expert sub-policies with a rule-based switcher), then multi-race support: Zerg first, then Terran. Each race gets its own bot lineage competing on the Elo ladder.
Stack
| Layer | Tool | Why |
|---|---|---|
| Language | Python 3.12 | SC2 libraries are Python-native |
| SC2 interface | burnysc2 v7.1.3 | Async BotAI, actively maintained |
| AI advisor | Claude CLI (OAuth or API key) | Strategic advice mid-game, async subprocess |
| Build orders | Spawning Tool API | Community build order database |
| Backend | FastAPI | WebSocket + REST for dashboard |
| Frontend | React + TypeScript + Vite | Live dashboard with game state streaming |
| Deep learning | PyTorch + Stable Baselines 3 | PPO policy network for strategic decisions |
| Training data | SQLite | Structured (s,a,r,s') transition storage |
| Testing (Python) | pytest | 1448 unit tests, SC2 integration markers |
| Testing (Frontend) | vitest + jsdom + @testing-library/react | 119 component / hook / lib tests |
| Linting | ruff + mypy | Strict type checking, consistent style |
Setup
- Windows 11 (primary dev). Linux / WSL2 also supported for the headless training substrate — see documentation/wiki/cloud-deployment.md and documentation/wiki/linux-dev-environment.md.
- StarCraft II installed at
C:\Program Files (x86)\StarCraft II\(Windows) or~/StarCraftII/(Linux/WSL). - Python 3.12+
- uv package manager
- Node.js 18+ (for React frontend)
- SC2 maps from Blizzard CDN (password:
iagreetotheeula) — do NOT use GitHub map files (Git LFS pointers)
-
Install Python dependencies:
cd Alpha4Gate uv sync -
Create
.envfrom template and fill in your keys:cp .env.example .env # Edit .env: set SC2PATH, SPAWNING_TOOL_API_KEY # Claude advisor auth: claude CLI must be on PATH with OAuth token or API key
-
Install frontend dependencies:
cd frontend && npm install && cd ..
# One-shot: start backend + frontend together (Git Bash)
bash scripts/start-dev.sh
# Or in two terminals:
# Terminal 1: backend
uv run python -m bots.v0.runner --serve
# Terminal 2: frontend dev server
cd frontend && npm run dev
# Opens http://localhost:3000 proxying to :8765Dashboard tabs
| Tab | Purpose |
|---|---|
| Advisor | Live advised-run status, loop controls, strategic hints, reward injection |
| Evolution | Live improve-bot-evolve run — fitness pool, stack apply, regression, generation outcomes |
| Improvements | Unified timeline of advised + evolve improvements with source filter (refresh-on-demand) |
| Processes | Live system process monitor, port status, state files, backend restart |
| Alerts | Severity-filtered alert list with ack/dismiss + unread badge in nav |
| Help | Renders documentation/wiki/operator-commands.md from disk |
In-app AlertToast lives at the App root and shows new alerts as they fire, regardless of which tab is active.
Usage
# Single game vs Easy AI
uv run python -m bots.v0.runner --map Simple64
# Options
--difficulty 3 # AI difficulty 1-10 (default: Easy)
--realtime # Watch in realtime
--batch 5 # Run 5 games, aggregate stats
--build-order 4gate # Select build order (default: 4gate)
--serve # Start dashboard API server only
--no-claude # Disable Claude advisorThe Claude advisor is optional. Pass --no-claude (or simply don't install the claude CLI) and the bot runs entirely on its rule-based strategy + optional PPO policy. Skip step 2's Claude-auth note during setup.
What still works without Claude:
| Feature | Notes |
|---|---|
| Rule-based play vs SC2 AI | uv run python -m bots.v0.runner --no-claude --difficulty 3 |
| Batch runs + stats aggregation | --batch 10 --no-claude |
| Neural (PPO) play and training | Rule-based or hybrid decision mode; training loop is Claude-free |
| Dashboard — Evolution, Improvements, Processes, Alerts, Help | All non-Advisor tabs render and update as normal |
| Daemon loop (auto-training, promotion, rollback) | Does not call Claude |
What needs Claude:
- Advisor tab and the
/improve-bot-advisedautonomous improvement loop - Mid-game strategic advice — the bot's
claude_advisor.pysubprocess (visible in JSONL game logs and surfaced on the Advisor tab when an advised run is active)
Quick start without Claude:
uv sync
cd frontend && npm install && cd ..
uv run python -m bots.v0.runner --map Simple64 --difficulty 3 --no-claude --realtime
# In another terminal, watch the dashboard:
bash scripts/start-dev.shTesting & project structure
uv run pytest # 1448 unit tests (no SC2 needed)
uv run pytest -m sc2 # SC2 integration tests (SC2 must be running)
uv run ruff check . # Lint
uv run mypy src bots --strict # Type check
cd frontend && npx tsc --noEmit # TypeScript checkAlpha4Gate/
├── bots/v0/ # 55 Python modules (the lineage seed)
│ ├── commands/ # Strategic command system (parser, interpreter, executor, queue)
│ ├── learning/ # PPO training, neural_engine inference, features, rewards, imitation, winprob_heuristic
│ ├── bot.py # Main BotAI subclass, game loop orchestration
│ ├── decision_engine.py # Strategic state machine (6 states)
│ ├── give_up.py # Win-prob-based resign trigger (Phase N)
│ ├── army_coherence.py # Staging, grouping, engagement/retreat
│ ├── claude_advisor.py # Async Claude CLI subprocess
│ ├── api.py # FastAPI server (REST + WebSocket)
│ └── ... # scouting, config, macro, micro, etc.
├── bots/v1..v10/ # Promoted snapshots — each a self-contained stack with its own data/
├── bots/current/ # Thin pointer package (MetaPathFinder → active version, v10 today)
├── src/orchestrator/ # Version registry, snapshots, self-play, Elo ladder, evolve
├── tests/ # 1448 unit tests across 88 files (+ SC2 integration markers)
├── frontend/ # React + TypeScript dashboard (Vite, 6 tabs)
├── scripts/ # Live test, training analysis, evolve runner, sandbox hook
├── documentation/wiki/ # Project wiki (start with index.md)
├── documentation/plans/ # Active plans (alpha4gate-master-plan.md)
├── documentation/archived/ # Completed plans
├── bots/<v>/data/ # Per-version state: training.db, checkpoints/, reward_rules.json
├── data/ # Cross-version state: ladder, evolve runs, advised state (gitignored)
├── logs/ # JSONL game logs (gitignored)
└── replays/ # SC2 replays (gitignored)
How the bot plays — layers & design decisions
Claude Advisor (async, non-blocking subprocess)
|
Neural Engine — PPO policy network (optional, hybrid or pure RL mode)
|
Strategy Layer — state machine: opening -> expand -> attack -> defend -> fortify -> late_game
|
Command System — parser -> interpreter -> executor (AI-Assisted / Human Only / Hybrid)
|
Tactics Layer — macro manager, scouting, production balance
|
Coherence — army staging, grouping, engagement/retreat decisions
|
Micro Layer — army movement, kiting, focus fire, ability usage
The bot follows a build order during the opening, then transitions to dynamic decision-making. Claude provides optional strategic advice via async subprocess calls (rate-limited to 1 per 30 game-seconds). The neural engine can override or supplement rule-based strategy decisions using a PPO-trained policy.
- Layered separation: Strategy, tactics, coherence, and micro are independent modules with clear interfaces, testable in isolation
- Async Claude advisor: Fire-and-forget subprocess call, bot never blocks waiting for AI advice
- Deep learning pipeline: PPO policy via Stable Baselines 3, SQLite transition storage, imitation learning bootstrap, gymnasium environment wrapping SC2 state. Hybrid mode combines neural + rule-based decisions
- Strategic command system: Three execution modes (AI-Assisted, Human Only, Hybrid), natural language parser with recipe library, command queue with priority and TTL
- Defensive fortification: FORTIFY strategic state triggered by threat assessment, BuildBacklog for deferred production retry, FortificationManager places cannons and shield batteries at expansion choke points
- Army coherence: Staging areas outside enemy range, group-based engagement with critical mass gate, retreat decisions based on relative army strength
- JSONL logging: Extended schema with
strategic_state,decision_queue, andclaude_advicefor decision debugging - Cross-game persistence: JSON files in
data/and SQLite for training transitions - Build order system: Named build orders importable from Spawning Tool API, sequenced by supply thresholds
Wiki: documentation/wiki/index.md — system diagram + page map Active plan: documentation/plans/alpha4gate-master-plan.md
Acknowledgments
Special thanks to my friends for their support:
- Drew Goya
- William Gathright
- Willie Williams
Also thanks to all the creators and companies who made the mountain of tools this is built on (and anyone else I forgot).