Unreal Agent Benchmark

English · 简体中文

A reproducible benchmark for testing whether an AI agent can build, reopen, package, and launch a playable Unreal Engine 5.8 game.

Official SEELE AI Unreal game creation · Benchmark tasks · Results · Contributing

Create an Unreal game with SEELE AI

This repository is a personal, public benchmark. If you want to create an Unreal game with SEELE AI instead of assembling the benchmark environment yourself, visit the official product page:

Try SEELE AI for Unreal game creation →

What this benchmark measures

Most AI game-generation demos stop at screenshots, code snippets, or editor activity. This benchmark tests the complete delivery path:

• a playable player-controlled game loop;
• observable objectives and win/fail states;
• compilation and PIE gameplay;
• save/reopen persistence;
• Win64 packaging and packaged launch;
• time, API cost, failures, retries, and human intervention.

Scores belong to the complete model + Agent + MCP + Skill configuration. They are not presented as naked-model scores.

Current status: the UE 5.8 Canary build and cumulative coexistence test passed, and epic-open-ecosystem-v0.1 is frozen. Formal scored results have not been published yet.

Comparison design

Each task starts from the same clean Unreal Engine 5.8.0 Third Person Blueprint project. Prompt bytes, starting project, model conditions, reasoning mode, time budget, cost budget, manual-edit policy, shared asset access, and rubric are frozen within each comparison.

Configuration	System under test	Published evidence
epic-open-ecosystem	Codex + accepted public UE 5.8 MCP/CLI ecosystem + public Unreal Skill suite + shared private asset access	Public toolchain and Skills; sanitised result evidence
seele-agent-system	SEELE Agent + its Unreal production system	Approved sanitised evidence only

Shared asset access and confidentiality

Both configurations receive identical access to a private asset retrieval capability provided by SeeleAI. Its implementation, internal metadata, and operational details are confidential and excluded from this repository.

The shared capability may supply source assets such as models, materials, textures, vegetation, buildings, props, characters, animation, audio, VFX, and UI media. It may not supply a completed benchmark level, implemented gameplay loop, task-specific answer pack, or complete game. Asset discovery, selection, import, adaptation, placement, and integration remain part of the timed run and therefore part of the systems comparison.

To experience SeeleAI's Unreal game-creation system, visit the official Unreal game creation page.

The first release contains one run per task/configuration cell—six runs in total. It is a transparent multi-task comparison, not a statistically stable leaderboard.

Task	Focus	Budget
coastal-village-explorer-v0.1	Exploration, terrain, interaction, and world readability	60 min / USD 15
dungeon-escape-v0.1	Level flow, objectives, hazards, UI, and restart	90 min / USD 20
arena-bot-v0.1	Player controls, AI navigation, combat, win/fail, and replay	120 min / USD 30

Canonical task prompts and rubrics are stored under benchmark/.

Evidence and scoring

Every rubric totals 100 points:

Dimension	Points
Core playable loop	50
Engineering validity	15
Packaging and delivery	10
Autonomous verification and recovery	10
Visual and interaction quality	10
Efficiency	5

A valid completion claim requires direct evidence for six gates:

1. compile;
2. PIE player path;
3. save/reopen;
4. package;
5. packaged launch;
6. packaged smoke test.

Failed and incomplete runs remain visible. A screenshot, an editor asset, or a successful package command alone does not prove a playable result.

Published evidence follows a strict allowlist: final screenshots and videos, gameplay acceptance results, compile/PIE/save-reopen/package conclusions, aggregate timing and asset-count statistics, and sanitised failure summaries. Raw Agent traces, complete terminal logs, Skill-loading records, asset-service responses, internal paths, endpoints, identifiers, download URLs, private source, and private-system hashes or versions are not published.

Frozen public UE 5.8 ecosystem

The public configuration was admitted through a disposable UE 5.8.0 Canary project. Every component had to pass its own compile/runtime check and then the cumulative coexistence gates: loopback endpoints, typed mutation/readback, audited Python, PIE start/input/stop, save/close/reopen, and persisted cleanup.

Component	Frozen revision	Role
Epic native MCP + AllToolsets	UE 5.8.0 CL 55116800	Primary editor-native MCP and Toolsets
VibeUE	87ec7e63fd3552d65ac6c2d793ee2b675061a2da	Native Toolset extensions and audited Python
db-lyon ue-mcp	ff3edabd86243f6cf9bb6bcab2d92e645241d962	Broad editor/build gateway
soft-ue-cli	319635a1ffa31f9ece21193f500265e5c1cfad78	PIE input, CLI, capture and recovery
unreal-api-mcp	21559e3932fc31db57cb383fd0f3172a18177934	UE 5.8 C++ API lookup
UnrealMotionGraphicsMCP	adab2ead1eda523cdeb358e89b5478488551904e	Specialized UMG/BlueCode fallback

Monolith and UnrealAgentPlayer are explicitly excluded from v0.1: Monolith did not pass the UnrealGame build and loopback-only safety gate; UnrealAgentPlayer did not compile under UE 5.8 UHT. The exact admission record, endpoints, routing rules, known issues, counts, and Python audit fields are frozen in epic-open-ecosystem-v0.1.json.

Public Unreal Skill suite

The epic-open-ecosystem configuration uses one vendored Epic Skill plus eighteen independently authored Unreal production Skills.

Area	Skills
Entry and context	unreal-game-builder, unreal-project-context, unreal-asset-management
Authoring and framework	unreal-blueprint-cpp-authoring, unreal-gameplay-framework
Player loop	unreal-player-controls, unreal-objectives-interactions
Level and world	unreal-level-design, unreal-terrain-pcg, unreal-world-composition
AI and combat	unreal-ai-navigation, unreal-combat-systems
Presentation	unreal-ui-umg, unreal-animation, unreal-materials-lighting, unreal-vfx-audio
Verification	unreal-performance-profiling, unreal-playtest-delivery

skills/epic-unreal-mcp vendors the approved files from EpicGames/unreal-engine-skills-for-claude-code-plugin, pinned to commit:

766fb42370d9e251f7524fffb12cfdbc5b11a426

skills/public-suite.json freezes all 19 runtime Skill directories and file hashes. THIRD_PARTY_NOTICES.md records the five audited public sources, revisions, licences, adapted concepts, and API exclusions.

Validate the suite with:

pwsh -NoProfile -File ./scripts/Test-PublicSkills.ps1

Quick start

Prerequisites

• Windows and PowerShell 7;
• Unreal Engine 5.8.0, changelist 55116800;
• a locally licensed Third Person Blueprint template;
• Epic's ModelContextProtocol and AllToolsets plugins;
• Visual Studio 2022 C++ tools, Git, Codex, Node.js, Python and uv;
• FFmpeg or OBS when recording evidence.

Do not commit API credentials, copied Unreal content, Fab assets, or private system material.

1. Check the environment

pwsh -NoProfile -File ./scripts/Test-BenchmarkEnvironment.ps1
pwsh -NoProfile -File ./scripts/Test-PublicSkills.ps1

2. Create the local baseline

pwsh -NoProfile -File ./scripts/New-BenchmarkProject.ps1

Licensed Unreal content is copied into the ignored .local/baseline/UnrealAgentBench58/ directory. It is never redistributed by this repository.

3. Create an isolated run

$runId = 'arena-bot-epic-open-ecosystem-r01'

pwsh -NoProfile -File ./scripts/New-BenchmarkRun.ps1 `
  -BenchmarkId arena-bot-v0.1 `
  -RunId $runId `
  -ConfigurationId epic-open-ecosystem `
  -AgentHarness codex `
  -Model provider/model-id `
  -Reasoning high `
  -SkillCommit <public-suite-commit>

Every run gets an independent project, exact prompt.md, evidence directory, and frozen run-manifest.json. Public-ecosystem runs also receive a frozen project-root AGENTS.md that makes the copied Skill suite discoverable, routes implementation work to the admitted public MCP/CLI layers, and requires implementation artifacts before completion evidence.

Install the project plugins already admitted by the UE 5.8 Canary into the fresh run:

pwsh -NoProfile -File ./scripts/Install-EpicOpenEcosystem.ps1 `
  -RunRoot ".local/runs/$runId"

4. Capture MCP inventory and verify the run

Start Epic MCP on 127.0.0.1:8000 and SoftUEBridge on 127.0.0.1:8080. The validator discovers db-lyon's per-project loopback port from Saved/UE_MCP_Bridge/port.json and the UmgMcp endpoint from its live instance record. Then run:

pwsh -NoProfile -File ./scripts/Test-EpicOpenEcosystem.ps1 `
  -ProjectFile ".local/runs/$runId/project/UnrealAgentBench58.uproject" `
  -EvidencePath ".local/runs/$runId/evidence/open-ecosystem.json"

pwsh -NoProfile -File ./scripts/Test-UnrealMcp.ps1 `
  -EvidencePath ".local/runs/$runId/evidence/mcp.json"

pwsh -NoProfile -File ./scripts/Test-AgentRunConfiguration.ps1 `
  -RunRoot ".local/runs/$runId"

Run the public Agent with the generated project as its working directory, then give it the generated prompt without task-specific implementation hints. The project-root AGENTS.md supplies configuration-level identity, Skill and tool routing equally for every task in that configuration. Keep the complete evaluation record only in the controlled local run workspace. Publish only the allowlisted, sanitised evidence described above.

Publish a result

Store results at:

results/<task-id>/<configuration-id>/<run-id>/result.json

Then validate them:

pwsh -NoProfile -File ./scripts/Test-RunResult.ps1 `
  -Path ./results/<task-id>/<configuration-id>/<run-id>/result.json

Run the complete repository validation before proposing a result or protocol change:

pwsh -NoProfile -File ./scripts/validate_repository.ps1

See CONTRIBUTING.md before submitting a task, evaluator, public Skill, or result change.

Repository layout

benchmark/     Frozen tasks, prompts, budgets, and rubrics
skills/        Epic Skill, independent public Skills, and provenance
schemas/       Machine-readable result contract
scripts/       Environment, bootstrap, run, and validation tools
tests/         PowerShell behavior and policy fixtures
results/       Published result contract and approved evidence
docs/plans/    Benchmark design and implementation plan
.local/        Generated projects and evidence; ignored by Git

Project and intellectual-property boundary

This is a personal benchmark repository, not an official SEELE AI product repository. It may contain original benchmark automation, public task contracts, public Skills, licences, schemas, and approved sanitised result evidence. It must not contain employer source code, company Skill names or text, private MCP schemas, private prompts or raw logs, proprietary projects, unpublished assets, internal paths or service details, private-system hashes or versions, Unreal template content, plugin binaries, unlicensed Fab content, or credentials.

SEELE Agent execution, company-result media, branding claims, and non-public system details require separate company approval.

No general redistribution licence is granted until the repository maintainer selects and adds one.