A curated list of papers and benchmarks on repository-level code localization and automated GitHub issue resolution.
Repository-level code localization and automated GitHub issue resolution have become core research topics at the intersection of software engineering and natural language processing. Since the release of SWE-bench in 2023, the area has grown rapidly, with reported issue resolution performance improving from 12.5% in early work to over 72% by 2025. This repository organizes representative papers and benchmarks in this area, with the goal of helping researchers quickly understand the landscape and get started efficiently.
Repository-level code localization studies how to identify the files, classes, functions, or exact lines that should be modified in a code repository given a GitHub issue description. More broadly, GitHub issue resolution extends this into an end-to-end process covering issue understanding, localization, patch generation, and test-based validation.
The main challenges come from the huge repository search space, the need to reason over long-range cross-file dependencies, and the semantic gap between informal issue descriptions and precise program elements.
- Authors: Zhaoling Chen et al. (Yale University, USC, Stanford, All Hands AI)
- Method: parses the repository into a directed heterogeneous graph with node types such as directory, file, class, and function, and relation types such as containment, import, invocation, and inheritance. It introduces
SearchEntity,TraverseGraph, andRetrieveEntityto support multi-hop reasoning over repository structure. - Evaluated on:
SWE-bench Lite,Loc-Bench
- Authors: Siru Ouyang et al. (Tencent AI Lab, UIUC)
- Method: builds a line-level repository graph where nodes correspond to code lines and distinguish definition and reference nodes. It uses
tree-sitterfor AST parsing and supportsk-hop ego-graph retrieval. The graph can be integrated into downstream systems such as Agentless and SWE-agent. - Evaluated on:
SWE-bench Lite,CrossCodeEval
Alibaba LingmaAgent: Improving Automated Issue Resolution via Comprehensive Repository Exploration (FSE 2025 Industry Track)
- Authors: Yingwei Ma et al. (Alibaba Tongyi Lab)
- Method: constructs a top-down repository knowledge graph and explores the repository with Monte Carlo Tree Search. It adopts an explore-and-exploit strategy and uses a summary agent to consolidate discovered repository knowledge.
- Evaluated on:
SWE-bench Lite, internal Alibaba Cloud dataset
- Authors: Zhonghao Jiang et al. (ZJU-CTAG)
- Method: an LLM-driven function-level issue localization method with a two-stage graph search strategy: broad file-level exploration via a dynamically constructed module call graph, followed by deep function-level analysis using a function call graph. It introduces a pruner agent to filter irrelevant directions and a reflective alignment mechanism to recover from formatting errors under long contexts.
- Evaluated on:
SWE-bench Lite,SWE-bench Verified
CodexGraph: Bridging Large Language Models and Code Repositories via Code Graph Databases (NAACL 2025)
- Authors: Xiangyan Liu et al. (National University of Singapore, Alibaba)
- Method: integrates an LLM agent with a code graph database where nodes are symbols and edges are program relations. It follows a write-then-translate strategy: the agent first generates a natural language query, and a translation module converts it into a graph query.
- Evaluated on:
SWE-bench,CrossCodeEval,EvoCodeBench
- Authors: Fraol Batole, David OBrien et al. (Tulane University, Iowa State University, etc.)
- Method: an agent-oriented issue localization approach with static-analysis-aware repository reasoning.
- Authors: Zhongming Yu et al. (UC San Diego, Intel)
- Method: an LLM agent framework for issue localization with prioritized action scheduling.
- Authors: John Yang et al. (Princeton University, Stanford University)
- Method: introduces the Agent-Computer Interface (
ACI) and provides tools such asfind_file,search_file, andsearch_dir, together with integrated file viewing, file editing, and lint checking. - Evaluated on:
SWE-bench,HumanEvalFix
- Authors: Yuntong Zhang et al. (National University of Singapore)
- Method: uses AST-based code search and provides tools such as
search_class,search_method, andsearch_code. It also supports spectrum-based fault localization and iterative context retrieval. - Evaluated on:
SWE-bench Lite
- Authors: Anvith Pabba et al.
- Method: a workflow-driven semantics-aware repair agent that combines issue semantics, code semantics, and execution semantics to generate complete patches. Its pipeline includes execution-semantic retrieval, issue abstraction, code isolation, and a two-stage repair review.
- Evaluated on:
SWE-bench Lite
- Authors: Islem Bouzenia et al. (University of Stuttgart, UC Davis)
- Method: constrains the repair process with a finite-state machine that governs bug understanding, information collection, repair, and validation.
- Evaluated on:
Defects4J
- Authors: Xingyao Wang et al. (CMU, Berkeley, and others)
- Method: a generalist agent platform based on the CodeAct framework, featuring an event-stream architecture, Docker sandboxing, a built-in browser, and extensible
AgentSkillstooling. - Evaluated on:
SWE-bench Verified
MASAI: Modular Architecture for Software-engineering AI Agents (NeurIPS 2024 Workshop on Open-World Agents)
- Authors: Daman Arora et al. (Microsoft Research)
- Method: adopts a 5-subagent architecture composed of a Test Template Generator, Issue Reproducer, Edit Localizer, Fixer, and Ranker. Its modular design allows different strategies such as
ReActandCoTto be used for different subtasks, reducing unnecessary long trajectories. - Evaluated on:
SWE-bench Lite
- Authors: Wei Tao et al. (Fudan University)
- Method: uses four collaborating agent roles—Manager, Repository Custodian, Developer, and QA Engineer—to simulate the GitHub workflow, with the Manager handling task decomposition and coordination and the QA Engineer conducting code review.
- Evaluated on:
SWE-bench
- Authors: Dong Chen et al. (Huawei)
- Method: combines a multi-agent framework with predefined task graphs. Its roles include Reproducer, Programmer, and Tester, and it uses fault localization to provide precise edit positions while casting task-graph planning as a decision problem.
- Evaluated on:
SWE-bench Lite
- Authors: Huy Nhat Phan et al. (FPT Software AI Center)
- Method: introduces four specialized agents—Planner, Navigator, Code Editor, and Executor—to simulate the human developer workflow, supports multiple languages including Python and Java, and integrates the Zoekt search engine.
- Evaluated on:
SWE-bench Lite,SWE-bench Verified,RepoExec,Defects4J
SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinement (ICLR 2025)
- Authors: Antonis Antoniades, Albert Örwall et al.
- Method: combines MCTS with self-improvement. It uses a three-agent architecture composed of SWE-Agent, a Value Agent, and a Discriminator Agent, and relies on a hybrid value function that mixes numerical estimation with qualitative judgment.
- Evaluated on:
SWE-bench
- Authors: Chunqiu Steven Xia et al. (UIUC)
- Award: ACM SIGSOFT Distinguished Paper Award
- Method: a three-stage non-agent pipeline:
Localization → Repair → Patch Validation. It performs hierarchical localization from file level to class/function level and then to the concrete edit location, combining prompt-based reasoning with embedding retrieval and reproduction tests. - Evaluated on:
SWE-bench Lite,SWE-bench Verified
- Authors: Jianming Chang et al. (Southeast University, Singapore Management University)
- Method: a hierarchical bug localization framework with three levels: file, function, and statement. It uses static analysis to provide customized context, employs three specialized LLMs for the three levels, and uses program slicing to support localization.
- Evaluated on:
SWE-bench Lite
SWE-Fixer: Training Open-Source LLMs for Effective and Efficient GitHub Issue Resolution (ACL Findings 2025)
- Authors: Chengxing Xie et al.
- Method: a simplified two-stage pipeline consisting of code retrieval and code editing. It uses coarse-to-fine retrieval with BM25 for initial retrieval and a learned model for reranking. The editing model is trained with chain-of-thought data, and each issue requires only two inference rounds.
- Evaluated on:
SWE-bench
- Authors: Yihao Qin et al. (Chinese Academy of Sciences)
- Method: a three-stage fault localization pipeline composed of fault understanding, code navigation, and fault confirmation. It uses test behavior tracking, document-guided search, and multi-round dialogue to iteratively inspect suspicious methods.
- Evaluated on:
Defects4J-V1.2.0,Defects4J-V2.0.0
PatchPilot: A Cost-Efficient Software Engineering Agent with Early Attempts on Formal Verification (ICML 2025)
- Method: a rule-driven five-stage workflow covering reproduction, localization, generation, validation, and refinement. It also explores the use of formal verification through
CrossHairandZ3for patch checking. - Evaluated on:
SWE-bench Lite,SWE-bench Verified
Lingma SWE-GPT: An Open Development-Process-Centric Language Model for Automated Software Improvement (ISSTA 2025)
- Authors: Yingwei Ma et al. (Alibaba Tongyi Lab)
- Award: ACM SIGSOFT Distinguished Paper Award / Best Paper
- Method: an open model family centered on development processes. It learns from real code submission activity and adopts a three-stage framework of repository understanding, fault localization, and patch generation, with rejection sampling used to construct high-quality training data.
- Evaluated on:
SWE-bench Verified
- Authors: Zexiong Ma et al. (Peking University, ByteDance)
- Method: formulates issue localization as repo deep search, where an LLM must make multi-step decisions across search and navigation tools. The method uses rejection-sampled supervised fine-tuning followed by tool-integrated reinforcement learning optimized directly for localization quality.
- Evaluated on:
SWE-bench Verified
- Method: issue localization via code ranking.
- Evaluated on:
SWE-bench Lite,LocBench
- Method: decouples context retrieval from repository-level code editing and studies retrieval effectiveness under different tools and reasoning depths. It compares BM25-style retrieval and structure-aware tools, and analyzes the effect of self-reflection and iterative tool use.
- Evaluated on:
SWE-bench Lite,LCA Code Editing
- Method: explores the setting of feeding the entire code repository directly into a long-context language model instead of relying on explicit repository navigation and retrieval.
- Method: an analysis paper that systematically studies failure modes of code agents on
SWE-benchand points out that localization errors are one of the key causes of patch failure.
BLAZE: Cross-Language and Cross-Project Bug Localization via Dynamic Chunking and Hard Example Learning (TSE 2025)
- Method: targets cross-language and cross-project bug localization with dynamic chunking for long files and hard example learning for better generalization.
- Evaluated on:
BeetleBox
- Method: a dense retriever for repository-level code editing that explicitly fuses code semantics, repository structure, and call graph dependencies. It also proposes a loss function specialized for repository-level retrieval.
- Evaluated on:
SWE-bench,Long Code Arena
- Method: provides a large-scale high-quality contrastive dataset designed for training code retrievers and rerankers, and shows gains on function localization for issue-related tasks.
- Evaluated on:
CodeSearchNet
- Method: introduces non-parametric repository memory built from commit history and related issues. It allows a localization system to retrieve prior fixes, historical issue information, and repository summaries accumulated over time.
- Evaluated on:
SWE-bench Verified,SWE-bench Live
- Method: builds an experience-enhanced framework that extracts reusable knowledge, including failure experience, from historical agent trajectories and stores them in a multi-dimensional experience bank for future tasks.
- Evaluated on:
SWE-bench Verified
The following retrieval models and retrieval-oriented papers are frequently used as building blocks in repository-level localization and issue resolution systems.
| Method / Model | Venue | Type | Notes |
|---|---|---|---|
BM25 |
Classical IR baseline | Sparse retrieval | A strong lexical baseline widely used for initial file-level or chunk-level retrieval. |
Text Embeddings by Weakly-Supervised Contrastive Pre-training (E5) |
arXiv 2022 | Text embedding | A general-purpose dense retrieval model often used for issue-to-code matching. |
| Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documents | arXiv 2023 | Long-context embedding | Useful when repository context is long and retrieval units are larger than standard chunks. |
| Code Representation Learning At Scale | ICLR 2024 | Code embedding | A code-oriented representation learning reference for retrieval and reranking. |
| CodeBERT: A Pre-Trained Model for Programming and Natural Languages | EMNLP 2020 | Code representation learning | A foundational pretrained model for joint natural language and code representations. |
| GraphCodeBERT: Pre-training Code Representations with Data Flow | ICLR 2021 | Code representation learning | Extends code representation learning with data-flow structure. |
| UniXcoder: Unified Cross-Modal Pre-training for Code Representation | ACL 2022 | Code representation learning | A unified pretrained model for code understanding, retrieval, and generation. |
- Authors: Carlos E. Jimenez, John Yang et al. (Princeton University)
- Scale and features: 2,294 software engineering tasks from 12 popular Python repositories. Each task includes a GitHub issue description and the corresponding pull request, together with a Docker-based reproducible evaluation environment and fail-to-pass tests.
- Datasets:
SWE-bench,SWE-bench Lite,SWE-bench Verified
- Authors: John Yang et al. (Stanford, Princeton, Meta)
- Scale and features: 617 task instances from 17 JavaScript libraries, focusing on visual and user-interface software domains with image and video content in issue descriptions.
- Dataset:
SWE-bench Multimodal
Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving (NeurIPS 2025 Datasets and Benchmarks Track)
- Authors: Daoguang Zan et al. (ByteDance Seed)
- Scale and features: 2,132 high-quality instances across 8 languages, curated by 68 expert annotators from 2,456 candidates. The benchmark also provides difficulty levels and a companion
Multi-SWE-RLcommunity for reinforcement learning data. - Dataset:
Multi-SWE-bench
- Authors: SWE-bench team
- Scale and features: 300 tasks from 42 repositories across 9 programming languages.
- Dataset:
SWE-bench Multilingual
- Authors: Lianghong Guo et al. (Sun Yat-sen University)
- Scale and features: 959 task instances from 15 repositories, covering multiple languages, multiple modalities, and 8 application domains.
- Dataset:
OmniGIRL
- Authors: Daoguang Zan et al.
- Scale and features: a Java version of SWE-bench with Docker evaluation and leaderboard support.
- Dataset:
SWE-bench-Java
- Authors: Scale AI (Xiang Deng, Jeff Da et al.)
- Scale and features: 1,865 tasks from 41 professional repositories, including public, commercial, and held-out subsets. It focuses on long-horizon multi-file tasks and is explicitly designed to reduce contamination.
- Dataset:
SWE-bench Pro
SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? (arXiv 2025)
- Authors: Samuel Miserendino et al. (OpenAI Preparedness)
- Scale and features: 1,488 real-world freelance software engineering tasks with a total value of $1,000,000, validated with end-to-end Playwright tests.
- Dataset:
SWE-Lancer
SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents (arXiv 2025)
- Scale and features: a continuously updated benchmark mined from active GitHub repositories, with a fixed evaluation scaffold and explicit contamination tracking.
- Dataset:
SWE-rebench
- Authors: John Yang et al. (Stanford, Princeton, Alibaba Qwen)
- Scale and features: 50k+ task instances from 128 GitHub repositories. It uses an automatic synthesis pipeline to create training tasks by breaking existing tests and provides 26k SWE-agent trajectories.
- Dataset:
SWE-smith - Note: combines real and synthetic data
- Authors: Jiayi Pan et al. (UC Berkeley, UIUC, Apple)
- Scale and features: 2,438 real Python task instances from 11 repositories, plus a
SWE-Gym Litesubset. It is an executable environment for training both software engineering agents and verifiers. - Dataset:
SWE-Gym
- Authors: Lei Zhang et al. (Alibaba Qwen, Chinese Academy of Sciences)
- Scale and features: 16,061 training instances and 2,020 test instances. It introduces a test-driven data synthesis framework based on runtime dependency graphs.
- Dataset:
SWE-Flow - Note: synthetic data, test-driven development, runtime dependency graph
R2E-Gym: Procedural Environment Generation and Hybrid Verifiers for Scaling Open-Weights SWE Agents (COLM 2025)
- Authors: Naman Jain et al. (UC Berkeley, Together AI)
- Scale and features: 8.7k+ tasks in a procedurally generated executable training environment.
- Dataset:
R2E-Gym - Note: synthetic data
- Scale and features: a continuously updated live benchmark that addresses the static and contamination-prone nature of the original SWE-bench by collecting fresh GitHub issue and pull request instances, each with an isolated Docker environment.
- Dataset:
SWE-bench-Live
GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging (arXiv 2025)
- Scale and features: 54 end-to-end software tasks across domains and modalities. It emphasizes full workflows from repository understanding to environment setup, iterative development, and final delivery, and introduces an economic metric called
alpha-value. - Dataset:
GitTaskBench
- Scale and features: 576 high-quality repository-level question answering instances requiring multi-file and long-range reasoning.
- Dataset:
SWE-QA
SWE-Synth: Synthesizing Verifiable Bug-Fix Data to Enable LLMs in Resolving Real-World Bugs (arXiv 2025)
- Scale and features: a synthetic framework for creating bug-fix data with buggy code, repaired code, verification tests, and trajectories.
- Dataset:
SWE-Synth
- Scale and features:
MULocBenchcollects 1,100 issues from 46 GitHub Python projects, with more diverse issue types, root causes, localization scopes, and file types than earlier localization benchmarks. - Dataset:
MULocBench
- Scale and features: reorganizes
SWE-bench Verifiedinto time-ordered sequences to evaluate continual learning, experience accumulation, and forgetting behavior. - Dataset:
SWE-bench-CL
SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents (arXiv 2025)
- Scale and features: 2,110 instances from 21 repositories in Java, JavaScript, TypeScript, and Python, covering bug fixing, feature addition, and refactoring, with AST-level structural statistics and retrieval evaluation.
- Dataset:
SWE-PolyBench
- Scale and features: focuses on testing and validating real-world bug-fix tasks across multiple repositories and task types.
- Dataset:
SWT-Bench
- Scale and features: 140 repository-level performance optimization tasks collected from real performance-improving pull requests.
- Dataset:
SWE-Perf
If you find a missing paper, benchmark, or metadata error, feel free to open an issue or submit a pull request.