Predicting customer credit default using Ensemble Learning techniques to support credit risk assessment and data-driven lending decisions.
Credit default prediction is one of the most important applications of Machine Learning in the financial industry. Financial institutions use predictive models to estimate whether customers are likely to default on future payments, enabling better lending decisions, reducing financial losses, and improving risk management.
This project develops a Credit Default Prediction System using the Default of Credit Card Clients Dataset from the UCI Machine Learning Repository. Five Machine Learning algorithms were implemented and compared, including Logistic Regression, Decision Tree, Random Forest, Gradient Boosting, and XGBoost.
Special emphasis was given to Ensemble Learning, comparing Bagging and Boosting techniques to evaluate how combining multiple models improves predictive performance, robustness, and generalization.
Financial institutions evaluate thousands of credit applications every day. Approving customers who are likely to default may generate significant financial losses, while rejecting reliable customers can reduce business opportunities.
The objective of this project is to develop predictive models capable of identifying customers with a higher probability of default before credit approval, supporting better lending decisions and improving credit risk management.
The project uses the Default of Credit Card Clients Dataset, available from the UCI Machine Learning Repository.
The dataset contains information about:
- Customer Demographics
- Credit Limit
- Education Level
- Marital Status
- Age
- Historical Repayment Status
- Monthly Bill Statements
- Monthly Payment Amounts
- Credit Default Status
Dataset source:
https://archive.ics.uci.edu/dataset/350/default+of+credit+card+clients
The dataset contains information for 30,000 credit card clients described by 24 predictive variables and one binary target indicating whether the customer defaulted on the following month's payment.
Due to GitHub repository size limitations, the dataset is not included in this repository.
To run this project, download the dataset and place the Excel file inside the data/ folder:
data/
└── default_of_credit_card_clients.xlsx
- Python
- Pandas
- NumPy
- Matplotlib
- Seaborn
- Scikit-Learn
- XGBoost
- Stratified K-Fold Cross Validation
- Jupyter Notebook
- Data Loading
- Initial Dataset Analysis
- Exploratory Data Analysis (EDA)
- Data Cleaning
- Missing Value Analysis
- Feature Selection
- Train-Test Split
- Feature Scaling
- Cross Validation
- Logistic Regression Modeling
- Decision Tree Modeling
- Random Forest Modeling (Bagging)
- Gradient Boosting Modeling
- XGBoost Modeling
- Model Evaluation
- Confusion Matrix Analysis
- ROC Curve Analysis
- Precision-Recall Curve Analysis
- Cross Validation Comparison
- Model Performance Comparison
- Feature Importance Analysis
- Business Insights
- Project Limitations
- Future Improvements
- Conclusion
The dataset presents a moderate class imbalance, with approximately 22% of customers defaulting on their payment and 78% remaining financially compliant. This reinforces the importance of evaluating classification models using metrics beyond simple accuracy.
Most customers possess relatively low credit limits, while only a small percentage have very high limits. Customers with larger credit limits generally exhibited lower default rates, suggesting a relationship between financial capacity and credit risk.
Previous repayment behavior proved to be the strongest indicator of future default. Customers with recent payment delays showed substantially higher default probabilities than customers who consistently paid on time.
The correlation matrix reveals strong relationships among repayment history variables and monthly bill statement variables, highlighting the importance of historical financial behavior for predicting future credit default.
To evaluate the robustness and generalization capability of each model, Stratified K-Fold Cross Validation was performed using the ROC-AUC metric.
The results demonstrate that Ensemble Learning algorithms consistently outperformed the baseline models.
- XGBoost achieved the highest mean ROC-AUC (0.784), demonstrating the strongest generalization performance across validation folds.
- Gradient Boosting and Random Forest produced nearly identical results, confirming the effectiveness of Ensemble Learning techniques for credit default prediction.
- Decision Tree achieved intermediate performance, while Logistic Regression presented the lowest mean ROC-AUC among the evaluated models.
The consistently low standard deviation observed during Cross Validation indicates that the Ensemble Learning models are stable and generalize well to unseen data, increasing confidence in their practical application for credit risk assessment.
Feature importance analysis was performed using Random Forest, Gradient Boosting and XGBoost.
All three Ensemble Learning models consistently identified the same variables as the most relevant predictors.
The most important features were:
- PAY_0
- PAY_2
- PAY_3
- PAY_4
- PAY_5
- PAY_6
- LIMIT_BAL
These results demonstrate that historical repayment behavior is considerably more informative than demographic variables, such as age, gender, education and marital status. This consistency across different Ensemble Learning algorithms increases confidence that repayment history is the primary driver of future credit default.
The models were evaluated using multiple classification metrics, including Accuracy, Precision, Recall, F1-Score, and ROC-AUC. Since credit default prediction involves an imbalanced classification problem, relying on a single metric such as Accuracy could lead to misleading conclusions. Therefore, multiple evaluation metrics were considered to provide a comprehensive assessment of model performance.
The comparison highlights different trade-offs between the models.
- XGBoost achieved the highest ROC-AUC (0.778), demonstrating the strongest ability to distinguish between defaulting and non-defaulting customers.
- Gradient Boosting achieved nearly identical performance, confirming the effectiveness of boosting methods.
- Random Forest obtained the highest Recall (0.579) and F1-Score (0.544), making it particularly suitable when the primary objective is to identify as many high-risk customers as possible.
- Logistic Regression achieved the highest Precision but presented the lowest Recall, indicating that although its positive predictions are more reliable, it fails to identify many defaulting customers.
- Decision Tree provided competitive performance but was consistently outperformed by the ensemble methods.
Overall, Ensemble Learning algorithms demonstrated superior predictive performance compared to individual models, confirming their effectiveness for credit risk assessment.
The exploratory analysis and Machine Learning models revealed several important business insights.
- Approximately 22% of customers defaulted on their payment.
- Historical repayment behavior proved to be the strongest predictor of future default.
- Customers with recent payment delays exhibited substantially higher default probabilities.
- Customers with higher monthly payment amounts generally presented lower credit risk.
- Higher credit limits tended to be associated with lower default rates.
- Historical repayment variables consistently dominated feature importance across all Ensemble Learning models.
- XGBoost achieved the highest mean ROC-AUC during Cross Validation, demonstrating the strongest generalization capability among the evaluated models.
- Random Forest achieved the highest Recall and F1-Score, making it a strong alternative when the business objective is to identify the largest possible number of high-risk customers.
- Institutions focused on minimizing financial losses may prefer XGBoost due to its superior discrimination capability.
- Institutions prioritizing the identification of as many high-risk customers as possible may benefit from Random Forest because of its higher Recall.
These findings demonstrate how Ensemble Learning techniques can improve credit risk assessment, support more accurate lending decisions, and reduce potential financial losses.
Credit_Default_Ensemble_Learning/
│
├── data/
│ └── default_of_credit_card_clients.xlsx
│
├── images/
│ ├── default_distribution.png
│ ├── credit_limit_distribution.png
│ ├── default_rate_pay0.png
│ ├── correlation_heatmap.png
│ ├── cross_validation_roc_auc.png
│ ├── model_performance_comparison.png
│ └── xgboost_feature_importance.png
│
├── notebooks/
│ └── credit_default_ensemble_learning.ipynb
│
├── README.md
├── requirements.txt
└── .gitignore
Talita Niqian Lin Wu
LinkedIn:
https://www.linkedin.com/in/talita-niqian-lin-wu/
GitHub: