Code and datasets for our paper on CatchAll: Repository-Aware Exception Handling with Knowledge-Guided LLMs.
We propose CatchAll, a novel LLM-based approach for repository-aware exception handling. CatchAll enhances the understanding of repository context by incorporating three complementary sources of knowledge: 1) exception-prone APIs, derived from an empirically constructed API-Exception mapping that captures common exception-throwing behaviors of specific APIs in real-world codebases; 2) execution context, including the contextual call trace surrounding the current code snippet; and 3) exemplar knowledge, distilled from reusable exception-handling patterns in historical code. The knowledge is encoded into structured prompts to guide the LLM in generating accurate and context-aware exception-handling code. To evaluate CatchAll, we construct two new benchmarks for repository-aware exception handling: a large-scale dataset RepoExEval and a small-scale executable dataset RepoExEval-Exec. Experimental results demonstrate that CatchAll significantly outperforms all existing baselines in both exception type prediction and handling code synthesis. Specifically, it achieves a CodeBLEU of 0.31 (vs. 0.27% for the best baseline), an intent accuracy of 60.1% (vs. 48.0%), and a Pass@1 of 29% (vs. 25%). These results demonstrate its effectiveness in improving exception handling in large-scale code repositories.
data/
├── intermediate/
│ └── api_mapping.json # Stores the raw API exception mappings
├── processed/
│ ├── exception_description_android.json # Stores exception descriptions related to Android
│ ├── exception_doc_android.json # Stores Android exception hierarchy
│ ├── exception_doc.json # Stores JAVA SE exception hierarchy
│ ├── exception_similarity_matrix.json # Stores the similarity matrix of exceptions
│ ├── processed_test_data_small.parquet # Stores processed test data (small scale)
│ └── processed_train_data_small.parquet # Stores processed training data (small scale)
├── scripts/ # Stores scripts for data processing, catch block generation and type prediction
│ ├── catch_block/ # Scripts related to catch block generation
│ │ ├── exception_graph_gen.py # Script for generating similarity matrix of exceptions
│ │ ├── few_shot_gen.py # Script for retrieving few-shot sample
│ │ └── prompts.py # Script for generating prompts
│ ├── data_process/ # Scripts related to data processing
│ │ ├── api_seq_gen_parquet.py # Script for generating API sequences (stored as Parquet files)
│ │ ├── ast_gen_for_dataset.py # Script for generating abstract syntax trees for the dataset
│ │ ├── crawl_android_class.py # Script for crawling Android exception hierarchy
│ │ ├── data_clean.py # Script for data cleaning
│ │ └── train_test_split_json.py # Script for splitting training and test datasets
│ └── type/ # Scripts related to type processing
│ ├── api_exception_mapping.py # Script for generating API exception mappings
│ └── api_mapping_test.py # Script for exception type prediction- Environmental Setup
conda create -n CatchAll
conda activate CatchAll
pip install -r requirements.txt- Run API-Exception mappings' generation
python3 ./scripts/type/api_exception_mapping.py --model_name $model_name$ --max_requests $max_requests$- Run exception type prediction evaluation
python3 ./scripts/type/api_mapping_test.py --model_name $model_name$ --max_requests $max_requests$- Run exception handling evaluation
python3 ./scripts/catch_block/few_shot_gen.py --model_name $model_name$ --max_requests $max_requests$
Answer to RQ1: CatchAll achieves superior performance over state-of-the-art baselines in exception handling across GPT-4o and DeepSeek-V3, surpassing the strongest baseline, RepoCoder, by 14.8% and 25.2% in CodeBLEU and intent prediction accuracy, respectively.
Answer to RQ2: CatchAll significantly improves exception type prediction accuracy, achieving 53.6% TypeAcc with an absolute improvement of 12 percentage points over the best-performing baseline.
Answer to RQ3: All knowledge components in CatchAll contribute to the performance. Removing API-Exception mapping results in the largest performance degradation.
Answer to RQ4: CatchAll exhibits suboptimal performance with fewer than 10 retrieved samples during pattern abstraction, achieving peak performance at 20 samples.