Reads an issue → maps the repository → reproduces the bug as a failing test → localizes the fault → writes a patch → proves it in a sandbox → opens a pull request.
▶ Full-quality screen recording (MP4). This is a real run, not a mock-up. The dashboard replays the agent's event stream at a readable pace; the on-screen timestamps are the agent's real times.
- The problem
- The solution
- Demo: what you are watching
- Architecture
- How each stage works
- Two engines: LLM agent and search-based repair
- Sandboxing and safety model
- Benchmark
- Quick start
- Project layout
- Engineering decisions
- Limitations and roadmap
Most engineering time is not spent writing new code. It goes into understanding an unfamiliar codebase well enough to make a small, correct change: reading the bug report, finding where the behaviour lives, reproducing the failure, fixing it without breaking anything else, and packaging it for review.
Most "AI coding" tools stop at suggesting code. That leaves the hard part with the human:
| Gap | Why it matters |
|---|---|
| No grounding in the repo | Generated code calls functions that don't exist or ignores local conventions. |
| No reproduction | Without a failing test you can't tell whether the patch fixes this bug. |
| No verification | "Looks right" patches regress other behaviour, and nobody runs the suite. |
| Unsafe execution | Running an untrusted repo's tests (and AI-generated code) on your machine with your credentials is a real security risk. |
| No audit trail | You can't see why the agent chose a fix, so you can't trust it. |
ForgeMind treats issue resolution as an engineering pipeline with a verification gate at every step, not as a single prompt.
- Ground: parse the repo into an AST symbol index plus a BM25 search corpus, so every decision refers to real code.
- Reproduce first: compile the issue's snippet into a regression test and confirm it fails before touching any source.
- Localize with evidence: combine coverage spectra (Ochiai SBFL), traceback frames and issue-text relevance into a ranked list of suspicious lines.
- Repair: either a Claude-powered tool-using agent, or an offline search-based program-repair engine (AST mutations ranked by suspiciousness × bug-pattern priors).
- Verify independently: every candidate runs in an isolated sandbox. A patch is accepted only if the reproduction passes and the full suite stays green. The agent can't edit existing tests to get there.
- Ship a reviewable PR: minimal diff, regression test included, root-cause explanation, localization evidence and a full JSONL trajectory.
The recording shows ForgeMind resolving examples/invoicely/ISSUE.md, a realistic bug in a small invoicing library:
#42: Bulk discount not applied when ordering exactly 10 units. The pricing page says "10 units or more earn 10% off", but customers who order exactly 10 units are charged full price.
| Step | What happens on screen |
|---|---|
| Understand | Extracts identifiers (bulk_discount_rate, apply_discount) and the reproduction snippet from the issue. |
| Index | 8 files, 30 symbols, BM25 corpus built from AST-aware chunks. |
| Reproduce | Baseline suite is green (14 passed), so the bug is not covered by tests. ForgeMind generates tests/test_forgemind_issue_42.py, which fails: assert 0.0 == 0.1. |
| Localize | A tracer injected into pytest records per-test line coverage; Ochiai + traceback + lexical scores rank pricing.py lines. |
| Repair | 72 candidate patches generated from AST mutation operators. The top-ranked one, > → >=, passes. |
| Validate | Full suite in the sandbox: 15 passed, 0 failed (14 original + the new regression test). |
| Pull request | Draft PR with root cause, verification results, localization table and a 1-line diff plus the test. |
The same run in the terminal:
$ forgemind solve examples/invoicely/ISSUE.md --repo examples/invoicely
ForgeMind v0.1.0 autonomous software engineer
--------------------------------------------------------------
Issue #42: Bulk discount not applied when ordering exactly 10 units
0.00s [1/7] Understanding issue
0.00s bug report; 1 code block(s); key terms: bulk_discount_rate, invoicely, pricing, apply_discount
0.00s [2/7] Indexing repository
0.01s 8 files, 30 symbols, 230 lines (3 test files)
0.01s [3/7] Reproducing bug
0.30s baseline suite: 14 passed
0.55s reproduced wrote tests/test_forgemind_issue_42.py
-> assert 0.0 == 0.1
0.55s [4/7] Localizing fault
0.82s 1. invoicely/pricing.py:22 ####### 0.38 return 0.0
0.82s 2. invoicely/pricing.py:19 ####### 0.35 for minimum, rate in BULK_TIERS:
0.82s 3. invoicely/pricing.py:20 ####### 0.35 if quantity > minimum:
0.82s [5/7] Generating patch
0.83s 72 candidate patches ranked (budget 200)
1.32s #1 PASS invoicely/pricing.py:20 change comparison to `>=` in `quantity > minimum`
1.33s [6/7] Validating in sandbox
1.58s full suite: 15 passed, 0 failed
- if quantity > minimum:
+ if quantity >= minimum:
1.58s [7/7] Preparing pull request
--------------------------------------------------------------
RESOLVED in 1.58s
flowchart LR
I["GitHub issue<br/>owner/repo#N · URL · .md"] --> U
subgraph Orchestrator["Orchestrator (event-sourced pipeline)"]
direction LR
U["1 · Understand<br/>identifiers · snippets<br/>traceback frames"] --> X["2 · Index<br/>AST symbols · call edges<br/>BM25 chunks"]
X --> R["3 · Reproduce<br/>issue → failing<br/>regression test"]
R --> L["4 · Localize<br/>Ochiai SBFL + traceback<br/>+ lexical fusion"]
L --> P{"5 · Repair"}
P -->|LLM available| A["Claude tool-use agent<br/>read · search · edit · test"]
P -->|offline| S["Search-based repair<br/>AST mutations × priors"]
A --> V
S --> V["6 · Validate<br/>targeted → full suite<br/>tamper check"]
V --> G["7 · Pull request<br/>branch · commit · draft PR"]
end
SB[("Sandbox<br/>copy-on-write workspace<br/>rlimits · timeout · no network<br/>scrubbed env")]
R -.runs in.-> SB
L -.runs in.-> SB
A -.runs in.-> SB
S -.runs in.-> SB
V -.runs in.-> SB
Orchestrator --> E["Event bus"]
E --> T["Terminal UI"]
E --> D["Live web dashboard (SSE)"]
E --> J["trajectory.jsonl<br/>result.json · PR.md · patch.diff"]
Every stage emits structured events. The terminal renderer, the live dashboard and the on-disk trajectory are all just subscribers, so a run can be watched live, replayed later (forgemind replay) or audited.
1. Issue understanding (forgemind/issue.py)
Loads issues from the GitHub REST API (owner/repo#123, issue URLs) or local markdown. It extracts fenced code blocks, inline-code identifiers (weighted above prose), referenced file paths, Python traceback frames and the issue type.
2. Repository index (forgemind/repo/)
- AST symbol table: modules, classes, functions and methods with exact line spans, docstrings and call edges. Used for
find_symbol, outlines and "which function is this line in". - AST-aware chunking: one chunk per top-level symbol, so search results are whole functions rather than arbitrary windows.
- BM25 retrieval with identifier-aware tokenization:
applyDiscount,apply_discountandapply discountall match. Zero dependencies.
3. Reproduction (forgemind/agent/reproduce.py)
A fix you can't observe is a guess. ForgeMind compiles issue snippets into a pytest file:
- assertion snippets are used as-is;
- doctest sessions (
>>> f(212)then100.0) becomeassert f(212) == 100.0; expr # expected: valuecomments become assertions.
The test must fail on the unpatched code, and fail for the right reason: a reproduction that dies with ImportError or SyntaxError is rejected. Existing failing tests are used too. The regression test ships in the PR so the bug can't silently come back.
4. Fault localization (forgemind/localize/)
A small pytest plugin (_fm_tracer.py) is injected into the sandboxed test run and records which source lines each test executes. Three signals are fused:
| Signal | Formula / idea | Weight |
|---|---|---|
| Spectrum (Ochiai) | ef / sqrt((ef + nf) · (ef + ep)): lines run by failing tests but rarely by passing ones |
0.60 |
| Traceback proximity | frames in failing tracebacks that point into source files, decaying ±3 lines | 0.25 |
| Lexical relevance | BM25 similarity between the issue text and the surrounding function | 0.15 |
5. Repair (two engines, see below)
6. Validation (forgemind/repair/search.py, forgemind/agent/orchestrator.py)
Staged gates keep the loop fast and the result trustworthy:
- Static: the patched file must parse.
- Targeted: only the reproducing tests run (sub-second).
- Full suite: everything must pass, so no regressions.
- Tamper check: the final diff may not modify any pre-existing test file.
7. Pull request (forgemind/pr/)
Writes PR.md, patch.diff, result.json and trajectory.jsonl to the run directory. With --commit it creates a forgemind/issue-N branch and commits. With --open-pr it pushes and opens a draft PR through the GitHub API, so a human always reviews before merging.
Claude tool-use agent (--strategy llm) |
Search-based repair (--strategy search) |
|
|---|---|---|
| Needs | ANTHROPIC_API_KEY |
nothing (fully offline, deterministic) |
| Model | claude-opus-5-5 (override with FORGEMIND_MODEL) |
none |
| Fixes | arbitrary multi-line changes, new logic | single-site bugs: wrong operator, off-by-one, swapped call, wrong variable, … |
| Loop | model plans → calls tools → edits → runs tests → submits | rank mutations → apply → targeted tests → full suite |
| Verification | independent re-run on submit; a rejection plus evidence goes back to the model (self-correction) |
every candidate passes the same three gates |
--strategy auto (the default) uses the LLM agent when credentials are present and the search engine otherwise.
LLM agent tools (forgemind/llm/tools.py): list_files, read_file, search_code (BM25), find_symbol (AST), edit_file (exact, unique string replacement for reviewable diffs), create_file, run_tests, get_suspicious_lines (SBFL), submit. The model gets the issue, a repository outline, the generated reproduction test and the top suspicious lines up front. It uses adaptive thinking with effort: high, prompt caching on the system prompt, and server-side refusal fallbacks.
Search engine mutation operators (forgemind/repair/mutators.py): comparison swap, off-by-one on constants, range/len bounds and indices, offset removal, arithmetic-operator swap, and/or swap, condition negation, builtin/method swap (min↔max, upper↔lower, …), duplicated-operand repair (lo - lo), wrong-variable replacement, argument swap and statement deletion. Each operator has a prior drawn from how common that bug class is in real one-line fixes, and the candidate rank is suspiciousness(line) × prior(operator). Edits are spliced into the original text using AST column offsets rather than ast.unparse, so comments and formatting survive and the diff is minimal.
ForgeMind runs code it didn't write (the repo's tests) and code it generated itself (patches). Both are treated as untrusted.
| Layer | Mechanism |
|---|---|
| Workspace isolation | Every run works on a private throwaway copy. The original checkout is written only by --commit, and only after verification. |
| Path confinement | All file tools resolve paths and refuse anything outside the sandbox root (../../etc/passwd → PermissionError). |
| Resource limits | setrlimit for CPU seconds, file size and open files; a wall-clock timeout kills the whole process group. |
| Network denial | sandbox-exec deny-network profile on macOS, unshare -rn on Linux, --network none with the Docker backend. |
| Secret scrubbing | *TOKEN*, *SECRET*, *API_KEY*, AWS_* and similar are removed from the environment; HOME points into the sandbox. |
| Test integrity | The LLM can't edit existing test files (read-only guardrail), and the final diff is checked for test tampering. |
| Human in the loop | PRs are opened as drafts. |
Use --backend docker for container-level isolation (--memory 2g --cpus 2 --pids-limit 256 --network none).
forgemind bench runs every task in benchmarks/. Each task has a buggy repo, an issue written the way a user would file it, and hidden tests the agent never sees. A patch is plausible if it passes the visible suite plus its own reproduction, and correct only if it also passes the hidden tests. Reporting both makes overfitting visible instead of hiding it.
Search strategy, offline, on a laptop:
| Task | Bug class | Plausible | Correct | Candidates | Fix found |
|---|---|---|---|---|---|
clamp_swap |
wrong builtin | ✅ | ✅ | 5 | call min instead of max |
median_wrong_var |
duplicated operand | ✅ | ✅ | 4 | lo - lo → hi - lo |
paginate_off_by_one |
slice off-by-one | ✅ | ✅ | 4 | items[start:end - 1] → items[start:end] |
password_policy |
boolean operator | ✅ | ✅ | 6 | or → and |
slugify_failing_test |
wrong method, no snippet (red CI only) | ✅ | ✅ | 3 | .upper() → .lower() |
temperature_operator |
arithmetic operator | ✅ | ✅ | 13 | (f + 32) → (f - 32) |
leap_year_logic |
missing logic (needs new code) | ❌ | ❌ | 13 | out of scope for single-site mutation; needs the LLM engine |
Resolved: 6/7 (86%) correct, 0 overfit, ~28s total.
Overfitting was measured during development and fixed. An early version "fixed" spread() by changing lo - lo to lo + lo, which passed the issue's example but failed the hidden tests. The duplicated-operand operator was added in response, and the benchmark now catches that class of mistake automatically.
git clone https://github.com/coolstack-coder/ForgeMind-AI-Autonomous-Software-Engineering-Agent.git
cd ForgeMind-AI-Autonomous-Software-Engineering-Agent
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]" # add ,llm for the Claude agent, ,demo to record videos# Resolve the demo issue offline, with the live dashboard at http://127.0.0.1:8765
forgemind solve examples/invoicely/ISSUE.md --repo examples/invoicely --dashboard
# Resolve a real GitHub issue with Claude, commit on a branch and open a draft PR
export ANTHROPIC_API_KEY=... GITHUB_TOKEN=...
forgemind solve owner/repo#123 --repo ./path/to/checkout --strategy llm --commit --open-pr
# Benchmark, replay, inspect
forgemind bench --suite benchmarks
forgemind replay .forgemind/runs/<run>/trajectory.jsonl
forgemind index --repo examples/invoicely
# Tests
pytest -q
# Re-record the demo video (needs Chrome or `playwright install chromium`)
python scripts/record_demo.pyEvery run leaves an auditable folder:
.forgemind/runs/20261025-000750-issue-42/
├── trajectory.jsonl # every event: phases, tests, candidates, tool calls, verdicts
├── patch.diff # the verified change, including the regression test
├── PR.md # pull-request description
└── result.json # machine-readable outcome
forgemind/
├── cli.py # `forgemind solve | bench | replay | index` + terminal renderer
├── issue.py # GitHub / markdown ingestion, signal extraction
├── models.py # Issue, TestReport, Suspect, Patch, Resolution
├── agent/
│ ├── orchestrator.py # the 7-phase pipeline and verification gates
│ ├── reproduce.py # issue → failing regression test
│ ├── llm_agent.py # Claude tool-use loop with independent verification
│ └── events.py # event bus → JSONL trajectory + live subscribers
├── repo/ # AST indexer, BM25 retrieval
├── localize/ # Ochiai SBFL fusion + in-sandbox coverage tracer plugin
├── repair/ # AST mutation operators + staged search
├── sandbox/ # isolated execution (process / docker backends)
├── testing/ # pytest runner + JUnit XML parsing
├── llm/ # provider abstraction (Anthropic, scripted) + tool surface
├── pr/ # git branch/commit, GitHub draft PR, PR writer
├── dashboard/ # SSE server + single-file live dashboard
└── bench/ # benchmark harness (plausible vs. correct)
benchmarks/ # 7 tasks with hidden tests
examples/invoicely/ # demo repository + ISSUE.md
tests/ # 22 tests: unit, sandbox, end-to-end, scripted-LLM agent loop
scripts/record_demo.py # reproducible screen recording of a real run
- Reproduce before repair. A patch is accepted only against a test that was observed failing. This one rule removes most "confidently wrong" fixes.
- Verification is independent of the actor. The LLM's
submitis a request, not a verdict. The orchestrator re-runs everything itself and feeds rejections back as evidence. - Small, verifiable tool surface.
edit_filereplaces an exact, unique string instead of rewriting whole files, which gives reviewable diffs, obvious failures and no silent clobbering. - Text-preserving patches. Mutations splice the original source using AST offsets; comments and formatting are untouched.
- Event-sourced runs. One event stream drives the CLI, the dashboard and the trajectory file. Debugging an agent is reading its log.
- Measure plausible vs. correct. Hidden tests quantify overfitting, the known failure mode of test-driven repair.
- Zero runtime dependencies beyond pytest. BM25, SBFL, the sandbox and the dashboard are written from scratch with the standard library. The Anthropic SDK is optional.
- Fully testable offline.
ScriptedProviderreplays canned model turns, so the LLM loop, its guardrails and its self-correction path are covered by deterministic tests.
- The search engine fixes single-site bugs. Multi-hunk changes and new logic (see
leap_year_logic) are the LLM engine's job. - Python and pytest only for now. The indexer, runner and tracer are the language-specific parts.
- SBFL quality depends on test coverage; with no failing test and no executable snippet, localization falls back to retrieval.
- Roadmap: multi-location patch search guided by the LLM; SWE-bench-Lite harness; JavaScript and TypeScript (tree-sitter + Jest); test-generation to strengthen weak suites before repair; GitHub App with webhook-triggered runs.
Built by Anmol Advani (@coolstack-coder) · co-authored with Claude · MIT License

