Skip to content

About

Autonomous AI software engineer: takes a GitHub issue, maps the repo, reproduces the bug as a failing test, localizes the fault (SBFL), patches it with Claude or search-based repair, verifies in a sandbox and opens a pull request.

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

ForgeMind AI

An autonomous software engineering agent: from GitHub issue to verified pull request

Reads an issue → maps the repository → reproduces the bug as a failing test → localizes the fault → writes a patch → proves it in a sandbox → opens a pull request.

CI Python Tests Benchmark LLM License

ForgeMind resolving a real issue end-to-end on the live dashboard

▶ Full-quality screen recording (MP4). This is a real run, not a mock-up. The dashboard replays the agent's event stream at a readable pace; the on-screen timestamps are the agent's real times.


Table of contents


The problem

Most engineering time is not spent writing new code. It goes into understanding an unfamiliar codebase well enough to make a small, correct change: reading the bug report, finding where the behaviour lives, reproducing the failure, fixing it without breaking anything else, and packaging it for review.

Most "AI coding" tools stop at suggesting code. That leaves the hard part with the human:

Gap Why it matters
No grounding in the repo Generated code calls functions that don't exist or ignores local conventions.
No reproduction Without a failing test you can't tell whether the patch fixes this bug.
No verification "Looks right" patches regress other behaviour, and nobody runs the suite.
Unsafe execution Running an untrusted repo's tests (and AI-generated code) on your machine with your credentials is a real security risk.
No audit trail You can't see why the agent chose a fix, so you can't trust it.

The solution

ForgeMind treats issue resolution as an engineering pipeline with a verification gate at every step, not as a single prompt.

  1. Ground: parse the repo into an AST symbol index plus a BM25 search corpus, so every decision refers to real code.
  2. Reproduce first: compile the issue's snippet into a regression test and confirm it fails before touching any source.
  3. Localize with evidence: combine coverage spectra (Ochiai SBFL), traceback frames and issue-text relevance into a ranked list of suspicious lines.
  4. Repair: either a Claude-powered tool-using agent, or an offline search-based program-repair engine (AST mutations ranked by suspiciousness × bug-pattern priors).
  5. Verify independently: every candidate runs in an isolated sandbox. A patch is accepted only if the reproduction passes and the full suite stays green. The agent can't edit existing tests to get there.
  6. Ship a reviewable PR: minimal diff, regression test included, root-cause explanation, localization evidence and a full JSONL trajectory.

Demo: what you are watching

The recording shows ForgeMind resolving examples/invoicely/ISSUE.md, a realistic bug in a small invoicing library:

#42: Bulk discount not applied when ordering exactly 10 units. The pricing page says "10 units or more earn 10% off", but customers who order exactly 10 units are charged full price.

Step What happens on screen
Understand Extracts identifiers (bulk_discount_rate, apply_discount) and the reproduction snippet from the issue.
Index 8 files, 30 symbols, BM25 corpus built from AST-aware chunks.
Reproduce Baseline suite is green (14 passed), so the bug is not covered by tests. ForgeMind generates tests/test_forgemind_issue_42.py, which fails: assert 0.0 == 0.1.
Localize A tracer injected into pytest records per-test line coverage; Ochiai + traceback + lexical scores rank pricing.py lines.
Repair 72 candidate patches generated from AST mutation operators. The top-ranked one, > → >=, passes.
Validate Full suite in the sandbox: 15 passed, 0 failed (14 original + the new regression test).
Pull request Draft PR with root cause, verification results, localization table and a 1-line diff plus the test.

Live dashboard

Generated pull request

The same run in the terminal:

$ forgemind solve examples/invoicely/ISSUE.md --repo examples/invoicely

  ForgeMind  v0.1.0  autonomous software engineer
  --------------------------------------------------------------
  Issue #42: Bulk discount not applied when ordering exactly 10 units

  0.00s  [1/7] Understanding issue
  0.00s      bug report; 1 code block(s); key terms: bulk_discount_rate, invoicely, pricing, apply_discount
  0.00s  [2/7] Indexing repository
  0.01s      8 files, 30 symbols, 230 lines (3 test files)
  0.01s  [3/7] Reproducing bug
  0.30s      baseline suite: 14 passed
  0.55s      reproduced wrote tests/test_forgemind_issue_42.py
             -> assert 0.0 == 0.1
  0.55s  [4/7] Localizing fault
  0.82s      1. invoicely/pricing.py:22   ####### 0.38  return 0.0
  0.82s      2. invoicely/pricing.py:19   ####### 0.35  for minimum, rate in BULK_TIERS:
  0.82s      3. invoicely/pricing.py:20   ####### 0.35  if quantity > minimum:
  0.82s  [5/7] Generating patch
  0.83s      72 candidate patches ranked (budget 200)
  1.32s      #1   PASS  invoicely/pricing.py:20  change comparison to `>=` in `quantity > minimum`
  1.33s  [6/7] Validating in sandbox
  1.58s      full suite: 15 passed, 0 failed
             -        if quantity > minimum:
             +        if quantity >= minimum:
  1.58s  [7/7] Preparing pull request
  --------------------------------------------------------------
  RESOLVED in 1.58s

Architecture

flowchart LR
    I["GitHub issue<br/>owner/repo#N · URL · .md"] --> U

    subgraph Orchestrator["Orchestrator (event-sourced pipeline)"]
        direction LR
        U["1 · Understand<br/>identifiers · snippets<br/>traceback frames"] --> X["2 · Index<br/>AST symbols · call edges<br/>BM25 chunks"]
        X --> R["3 · Reproduce<br/>issue → failing<br/>regression test"]
        R --> L["4 · Localize<br/>Ochiai SBFL + traceback<br/>+ lexical fusion"]
        L --> P{"5 · Repair"}
        P -->|LLM available| A["Claude tool-use agent<br/>read · search · edit · test"]
        P -->|offline| S["Search-based repair<br/>AST mutations × priors"]
        A --> V
        S --> V["6 · Validate<br/>targeted → full suite<br/>tamper check"]
        V --> G["7 · Pull request<br/>branch · commit · draft PR"]
    end

    SB[("Sandbox<br/>copy-on-write workspace<br/>rlimits · timeout · no network<br/>scrubbed env")]
    R -.runs in.-> SB
    L -.runs in.-> SB
    A -.runs in.-> SB
    S -.runs in.-> SB
    V -.runs in.-> SB

    Orchestrator --> E["Event bus"]
    E --> T["Terminal UI"]
    E --> D["Live web dashboard (SSE)"]
    E --> J["trajectory.jsonl<br/>result.json · PR.md · patch.diff"]
Loading

Every stage emits structured events. The terminal renderer, the live dashboard and the on-disk trajectory are all just subscribers, so a run can be watched live, replayed later (forgemind replay) or audited.


How each stage works

1. Issue understanding (forgemind/issue.py)

Loads issues from the GitHub REST API (owner/repo#123, issue URLs) or local markdown. It extracts fenced code blocks, inline-code identifiers (weighted above prose), referenced file paths, Python traceback frames and the issue type.

2. Repository index (forgemind/repo/)

  • AST symbol table: modules, classes, functions and methods with exact line spans, docstrings and call edges. Used for find_symbol, outlines and "which function is this line in".
  • AST-aware chunking: one chunk per top-level symbol, so search results are whole functions rather than arbitrary windows.
  • BM25 retrieval with identifier-aware tokenization: applyDiscount, apply_discount and apply discount all match. Zero dependencies.

A fix you can't observe is a guess. ForgeMind compiles issue snippets into a pytest file:

  • assertion snippets are used as-is;
  • doctest sessions (>>> f(212) then 100.0) become assert f(212) == 100.0;
  • expr # expected: value comments become assertions.

The test must fail on the unpatched code, and fail for the right reason: a reproduction that dies with ImportError or SyntaxError is rejected. Existing failing tests are used too. The regression test ships in the PR so the bug can't silently come back.

4. Fault localization (forgemind/localize/)

A small pytest plugin (_fm_tracer.py) is injected into the sandboxed test run and records which source lines each test executes. Three signals are fused:

Signal Formula / idea Weight
Spectrum (Ochiai) ef / sqrt((ef + nf) · (ef + ep)): lines run by failing tests but rarely by passing ones 0.60
Traceback proximity frames in failing tracebacks that point into source files, decaying ±3 lines 0.25
Lexical relevance BM25 similarity between the issue text and the surrounding function 0.15

5. Repair (two engines, see below)

Staged gates keep the loop fast and the result trustworthy:

  1. Static: the patched file must parse.
  2. Targeted: only the reproducing tests run (sub-second).
  3. Full suite: everything must pass, so no regressions.
  4. Tamper check: the final diff may not modify any pre-existing test file.

7. Pull request (forgemind/pr/)

Writes PR.md, patch.diff, result.json and trajectory.jsonl to the run directory. With --commit it creates a forgemind/issue-N branch and commits. With --open-pr it pushes and opens a draft PR through the GitHub API, so a human always reviews before merging.


Two engines: LLM agent and search-based repair

Claude tool-use agent (--strategy llm) Search-based repair (--strategy search)
Needs ANTHROPIC_API_KEY nothing (fully offline, deterministic)
Model claude-opus-5-5 (override with FORGEMIND_MODEL) none
Fixes arbitrary multi-line changes, new logic single-site bugs: wrong operator, off-by-one, swapped call, wrong variable, …
Loop model plans → calls tools → edits → runs tests → submits rank mutations → apply → targeted tests → full suite
Verification independent re-run on submit; a rejection plus evidence goes back to the model (self-correction) every candidate passes the same three gates

--strategy auto (the default) uses the LLM agent when credentials are present and the search engine otherwise.

LLM agent tools (forgemind/llm/tools.py): list_files, read_file, search_code (BM25), find_symbol (AST), edit_file (exact, unique string replacement for reviewable diffs), create_file, run_tests, get_suspicious_lines (SBFL), submit. The model gets the issue, a repository outline, the generated reproduction test and the top suspicious lines up front. It uses adaptive thinking with effort: high, prompt caching on the system prompt, and server-side refusal fallbacks.

Search engine mutation operators (forgemind/repair/mutators.py): comparison swap, off-by-one on constants, range/len bounds and indices, offset removal, arithmetic-operator swap, and/or swap, condition negation, builtin/method swap (min↔max, upper↔lower, …), duplicated-operand repair (lo - lo), wrong-variable replacement, argument swap and statement deletion. Each operator has a prior drawn from how common that bug class is in real one-line fixes, and the candidate rank is suspiciousness(line) × prior(operator). Edits are spliced into the original text using AST column offsets rather than ast.unparse, so comments and formatting survive and the diff is minimal.


Sandboxing and safety model

ForgeMind runs code it didn't write (the repo's tests) and code it generated itself (patches). Both are treated as untrusted.

Layer Mechanism
Workspace isolation Every run works on a private throwaway copy. The original checkout is written only by --commit, and only after verification.
Path confinement All file tools resolve paths and refuse anything outside the sandbox root (../../etc/passwd → PermissionError).
Resource limits setrlimit for CPU seconds, file size and open files; a wall-clock timeout kills the whole process group.
Network denial sandbox-exec deny-network profile on macOS, unshare -rn on Linux, --network none with the Docker backend.
Secret scrubbing *TOKEN*, *SECRET*, *API_KEY*, AWS_* and similar are removed from the environment; HOME points into the sandbox.
Test integrity The LLM can't edit existing test files (read-only guardrail), and the final diff is checked for test tampering.
Human in the loop PRs are opened as drafts.

Use --backend docker for container-level isolation (--memory 2g --cpus 2 --pids-limit 256 --network none).


Benchmark

forgemind bench runs every task in benchmarks/. Each task has a buggy repo, an issue written the way a user would file it, and hidden tests the agent never sees. A patch is plausible if it passes the visible suite plus its own reproduction, and correct only if it also passes the hidden tests. Reporting both makes overfitting visible instead of hiding it.

Search strategy, offline, on a laptop:

Task Bug class Plausible Correct Candidates Fix found
clamp_swap wrong builtin ✅ ✅ 5 call min instead of max
median_wrong_var duplicated operand ✅ ✅ 4 lo - lo → hi - lo
paginate_off_by_one slice off-by-one ✅ ✅ 4 items[start:end - 1] → items[start:end]
password_policy boolean operator ✅ ✅ 6 or → and
slugify_failing_test wrong method, no snippet (red CI only) ✅ ✅ 3 .upper() → .lower()
temperature_operator arithmetic operator ✅ ✅ 13 (f + 32) → (f - 32)
leap_year_logic missing logic (needs new code) ❌ ❌ 13 out of scope for single-site mutation; needs the LLM engine

Resolved: 6/7 (86%) correct, 0 overfit, ~28s total.

Overfitting was measured during development and fixed. An early version "fixed" spread() by changing lo - lo to lo + lo, which passed the issue's example but failed the hidden tests. The duplicated-operand operator was added in response, and the benchmark now catches that class of mistake automatically.


Quick start

git clone https://github.com/coolstack-coder/ForgeMind-AI-Autonomous-Software-Engineering-Agent.git
cd ForgeMind-AI-Autonomous-Software-Engineering-Agent
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"          # add ,llm for the Claude agent, ,demo to record videos
# Resolve the demo issue offline, with the live dashboard at http://127.0.0.1:8765
forgemind solve examples/invoicely/ISSUE.md --repo examples/invoicely --dashboard

# Resolve a real GitHub issue with Claude, commit on a branch and open a draft PR
export ANTHROPIC_API_KEY=...  GITHUB_TOKEN=...
forgemind solve owner/repo#123 --repo ./path/to/checkout --strategy llm --commit --open-pr

# Benchmark, replay, inspect
forgemind bench --suite benchmarks
forgemind replay .forgemind/runs/<run>/trajectory.jsonl
forgemind index --repo examples/invoicely

# Tests
pytest -q

# Re-record the demo video (needs Chrome or `playwright install chromium`)
python scripts/record_demo.py

Every run leaves an auditable folder:

.forgemind/runs/20261025-000750-issue-42/
├── trajectory.jsonl   # every event: phases, tests, candidates, tool calls, verdicts
├── patch.diff         # the verified change, including the regression test
├── PR.md              # pull-request description
└── result.json        # machine-readable outcome

Project layout

forgemind/
├── cli.py                 # `forgemind solve | bench | replay | index` + terminal renderer
├── issue.py               # GitHub / markdown ingestion, signal extraction
├── models.py              # Issue, TestReport, Suspect, Patch, Resolution
├── agent/
│   ├── orchestrator.py    # the 7-phase pipeline and verification gates
│   ├── reproduce.py       # issue → failing regression test
│   ├── llm_agent.py       # Claude tool-use loop with independent verification
│   └── events.py          # event bus → JSONL trajectory + live subscribers
├── repo/                  # AST indexer, BM25 retrieval
├── localize/              # Ochiai SBFL fusion + in-sandbox coverage tracer plugin
├── repair/                # AST mutation operators + staged search
├── sandbox/               # isolated execution (process / docker backends)
├── testing/               # pytest runner + JUnit XML parsing
├── llm/                   # provider abstraction (Anthropic, scripted) + tool surface
├── pr/                    # git branch/commit, GitHub draft PR, PR writer
├── dashboard/             # SSE server + single-file live dashboard
└── bench/                 # benchmark harness (plausible vs. correct)
benchmarks/                # 7 tasks with hidden tests
examples/invoicely/        # demo repository + ISSUE.md
tests/                     # 22 tests: unit, sandbox, end-to-end, scripted-LLM agent loop
scripts/record_demo.py     # reproducible screen recording of a real run

Engineering decisions

  • Reproduce before repair. A patch is accepted only against a test that was observed failing. This one rule removes most "confidently wrong" fixes.
  • Verification is independent of the actor. The LLM's submit is a request, not a verdict. The orchestrator re-runs everything itself and feeds rejections back as evidence.
  • Small, verifiable tool surface. edit_file replaces an exact, unique string instead of rewriting whole files, which gives reviewable diffs, obvious failures and no silent clobbering.
  • Text-preserving patches. Mutations splice the original source using AST offsets; comments and formatting are untouched.
  • Event-sourced runs. One event stream drives the CLI, the dashboard and the trajectory file. Debugging an agent is reading its log.
  • Measure plausible vs. correct. Hidden tests quantify overfitting, the known failure mode of test-driven repair.
  • Zero runtime dependencies beyond pytest. BM25, SBFL, the sandbox and the dashboard are written from scratch with the standard library. The Anthropic SDK is optional.
  • Fully testable offline. ScriptedProvider replays canned model turns, so the LLM loop, its guardrails and its self-correction path are covered by deterministic tests.

Limitations and roadmap

  • The search engine fixes single-site bugs. Multi-hunk changes and new logic (see leap_year_logic) are the LLM engine's job.
  • Python and pytest only for now. The indexer, runner and tracer are the language-specific parts.
  • SBFL quality depends on test coverage; with no failing test and no executable snippet, localization falls back to retrieval.
  • Roadmap: multi-location patch search guided by the LLM; SWE-bench-Lite harness; JavaScript and TypeScript (tree-sitter + Jest); test-generation to strengthen weak suites before repair; GitHub App with webhook-triggered runs.

Built by Anmol Advani (@coolstack-coder) · co-authored with Claude · MIT License

About

Autonomous AI software engineer: takes a GitHub issue, maps the repo, reproduces the bug as a failing test, localizes the fault (SBFL), patches it with Claude or search-based repair, verifies in a sandbox and opens a pull request.

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages