Türkçe Dokümantasyon · Install · How It Works · Supported Agents · Privacy
Benchmark AI coding agents (Claude Code, Codex, Gemini CLI, OpenRouter) directly on the issues, pull requests, and tests that shaped YOUR codebase — not a generic public leaderboard.
Generic coding benchmarks can tell you which agent performs well on a shared task set. RepoArena is SWE-bench for your own repository: it answers the question that matters for your team:
Which AI coding agent actually performs best on this repository?
RepoArena reconstructs real historical tasks at their pre-fix commits, gives every agent the same sanitized problem, and independently verifies each generated patch in Docker. Results come from executed tests. RepoArena never inserts demo scores or estimated costs.
| Generic benchmark | RepoArena |
|---|---|
| Measures performance on public benchmark tasks | Measures performance on your repository's history |
| Uses benchmark-wide language and framework distributions | Preserves your stack, conventions, and test suite |
| May expose known solutions in public history | Gives solvers a synthetic one-commit repository |
| Produces a general leaderboard | Produces a repository-specific, reproducible comparison |
RepoArena concentrates engineering effort on the difficult part: creating trustworthy tasks from software history without leaking the human solution.
- Repository-specific tasks from merged GitHub pull requests, linked issues, source changes, test changes, and CI evidence.
- Quality gates that reject documentation-only, formatting-only, dependency-only, generated, oversized, unclear, or unverifiable candidates.
- Baseline validation that requires the historical base to fail reproducibly and the human source change to pass the same hidden tests reproducibly.
- Leakage-resistant reconstruction using
git archiveand a fresh synthetic repository with no remote, tags, reflog, branches, or future Git objects. - Independent Docker verification with fresh workspaces, bounded resources, no credentials, and networking disabled for final tests.
- Fair agent comparison across immutable task definitions and identical verifier logic.
- Local SQLite history plus terminal and self-contained HTML reports.
- Extensible adapters for Codex CLI, Claude Code, Gemini CLI, OpenRouter, and OpenAI-compatible routers such as 9Router.
RepoArena is currently distributed from source while the project is in alpha.
- Python 3.12 or newer
- Git
- a reachable Linux Docker daemon
- uv or pipx
- GitHub CLI authentication,
GITHUB_TOKEN, or a public GitHub repository - your own credentials for every enabled agent or router; RepoArena never supplies shared keys
git clone https://github.com/tugrakaymakcioglu/RepoArena.git
cd RepoArena
uv tool install .Using pipx instead:
pipx install .macOS/Linux:
sh ./scripts/build-images.shWindows PowerShell:
powershell -ExecutionPolicy Bypass -File .\scripts\build-images.ps1The scripts build local proxy, Codex, Claude, Gemini, and OpenCode images. You can pin provider CLI versions for a repeatable build; see the installation guide.
cd /path/to/your-project
repoarena init
repoarena doctor
repoarena discover
repoarena benchmark --agent codex
repoarena benchmark --agent claude
repoarena benchmark --agent gemini
repoarena benchmark --agent openrouter
repoarena benchmark --agent router
repoarena reportUse repoarena benchmark --all to snapshot one task set and run every enabled agent sequentially.
This recording shows the real 0.1.1 command surface. It contains no simulated benchmark scores
and makes no provider calls.
- Discover — Read GitHub and Git history without changing the active checkout.
- Score — Keep only focused changes with strong task, source, test, CI, and reproducibility signals.
- Validate — Prove twice that the historical base fails and the human source change passes.
- Reconstruct — Export the base into a one-commit repository that contains no future history.
- Solve — Run the selected agent in a resource-bounded Docker environment.
- Verify — Apply only its patch to an independent base, restore hidden tests, and test with the network disabled.
- Report — Store exact results locally and compare agents only on identical completed task sets.
The solver receives an opaque task ID, a sanitized description, and detected languages. It does not receive the original repository URL, issue or PR number, historical commit IDs, human patch, hidden tests, verifier commands, or expected outputs.
| Area | Alpha support |
|---|---|
| Repository metadata | GitHub via authenticated gh, optional GITHUB_TOKEN, or public REST API |
| Agents | Codex CLI, Claude Code, Gemini CLI, OpenRouter, and OpenAI-compatible routers in locally built Docker images |
| Verification profiles | Python/pytest, Node test scripts, Go tests, or an explicit custom image |
| Persistence | Local SQLite with transactional schema migrations |
| Reports | Rich terminal summary and escaped, self-contained HTML |
| Provider tests | Opt-in only; normal CI makes no paid API calls |
Your repository and benchmark results stay on your machine.
RepoArena has no account, telemetry, analytics, hosted database, model gateway, or RepoArena cloud backend. It never ships with shared provider keys and never pays for user inference.
The coding agent you select may send task text or repository code to its own provider. RepoArena's provider-only network restricts solver egress, but provider traffic is still remote processing under that provider's terms. The verifier receives no agent credentials and final verification has no network access.
Read the full threat model before using private or untrusted repositories.
repoarena init creates .repoarena/config.toml and local runtime directories. The configuration
contains no secret values; credentials come from environment variables or explicitly selected
provider-native files.
[benchmark]
quality_threshold = 70
timeout_seconds = 900
baseline_repetitions = 2
[sandbox]
engine = "docker"
solver_network = "provider-only"
setup_network = "bridge"
[agents.codex]
enabled = true
[agents.claude]
enabled = true
[agents.gemini]
enabled = false
[agents.openrouter]
enabled = false
model = "provider/model-id"
[agents.router]
enabled = false
base_url = "http://host.docker.internal:20128/v1"
model = "router-model-id"Valid agent patches are retained under the ignored .repoarena/runs/ directory. The database,
caches, reports, and run artifacts are also ignored automatically. Do not place API keys in
config.toml.
- Installation and first benchmark
- Provider and router configuration
- Changelog
- Architecture
- Benchmark task format
- Threat model
- Repository publishing metadata
- Contributing
- Security policy
- Support
uv sync --locked --all-extras
uv run pytest -m "not provider"
uv run ruff check .
uv run ruff format --check .
uv run mypy src
uv buildThe deterministic fake-agent suite covers successful, wrong, no-change, malformed, flaky, subprocess-failure, leakage, sandbox, verifier, storage, and report paths. Docker-marked tests run a real provider-free verifier flow when Docker is available. Real-provider calls are never required for CI.
RepoArena is alpha software. The core local vertical slice, Codex, Claude, Gemini, and router adapters, Docker isolation, and local reports are implemented; compatibility across diverse repository build systems will continue to improve.
Current limitations:
- GitHub is the only remote metadata provider.
- High-quality discovery currently requires a merged change with separable source and test patches.
- Automatic verifier profiles cover Python, Node, and Go; unusual builds need explicit commands and a Docker image.
- Binary and submodule patches are rejected in V1.
- Provider images are built locally rather than published by RepoArena.
- OpenCode and provider CLIs are third-party runtimes; pin tested versions for repeatable benchmarks.
- A local router must be reachable from Docker through
host.docker.internal; Linux engines need host-gateway support. - Provider endpoint allowlists may need updates when vendor authentication endpoints change.
The roadmap includes broader extraction, deeper GitHub integration, optional shared benchmark history, and task-specific recommendations. There is no SaaS, hosted worker, account system, or RepoArena-operated model gateway. Router support connects to infrastructure selected and paid for by the user.
Issues and pull requests are welcome. Security-sensitive changes to discovery, historical reconstruction, patches, or sandboxing must include deterministic regression tests and a short leakage/threat analysis. Start with CONTRIBUTING.md.
Please use GitHub Security Advisories for private vulnerability reports rather than public issues.
RepoArena is licensed under Apache-2.0.