A machine learning project for predicting customer churn in a telecommunications company. The project covers the complete supervised learning workflow, including exploratory data analysis, feature engineering, model comparison, hyperparameter optimization, and model interpretation with a focus on minimizing false negatives. The dataset used in this project is the Telecom Churn dataset available on Kaggle: https://www.kaggle.com/datasets/barun2104/telecom-churn
The primary objective is to build a classification model capable of accurately identifying customers who are likely to churn before they leave the service.
The project aims to:
- perform exploratory analysis of customer behavior;
- engineer meaningful predictive features;
- compare several tree-based machine learning algorithms;
- optimize model hyperparameters using Optuna;
- evaluate models using metrics appropriate for imbalanced classification;
- investigate misclassified customers, particularly false negatives, to understand model limitations.
The dataset contains 3,333 customer records with 10 original features and one binary target variable (Churn).
| Feature | Description |
|---|---|
| AccountWeeks | Number of weeks the customer has been subscribed |
| ContractRenewal | Whether the contract was recently renewed |
| DataPlan | Whether the customer has a data plan |
| DataUsage | Monthly data usage (GB) |
| CustServCalls | Number of customer service calls |
| DayMins | Average daytime minutes |
| DayCalls | Average daytime calls |
| MonthlyCharge | Monthly bill |
| OverageFee | Maximum overage fee in the last year |
| RoamMins | Average roaming minutes |
Target:
- Churn (0 = retained, 1 = churned)
Several additional features were created to capture customer behavior more effectively.
| Feature | Description |
|---|---|
| AvgCallDuration | Average daytime call duration |
| ChargePerGB | Monthly charge divided by data usage |
| ChargePerMinute | Monthly charge divided by daytime minutes |
| CustServiceCallsPerWeek | Customer support calls normalized by account age |
| RoamPerWeek | Roaming minutes normalized by account age |
The pipeline consists of:
- Train/test split
- Feature scaling
- Baseline model training
- Hyperparameter optimization
- Final model evaluation
- Error analysis
The following ensemble algorithms were compared:
- Random Forest
- Gradient Boosting
- Histogram Gradient Boosting
- XGBoost
- LightGBM
- CatBoost
Hyperparameters were optimized with Optuna.
Characteristics of the optimization process:
- 50 optimization trials per model
- Bayesian optimization
- Automatic search over model-specific parameter spaces
- Best hyperparameters retained for final evaluation
Because customer churn is an imbalanced classification problem, multiple evaluation metrics were considered:
- F1-score
- Recall
- ROC-AUC
- Average Precision
- Confusion Matrix
Special attention was paid to false negatives, since failing to identify a customer who will churn is generally more costly than incorrectly flagging a customer who would stay.
Among all evaluated models, Gradient Boosting achieved the best overall balance between recall and precision after hyperparameter optimization.
Final performance:
| Metric | Score |
|---|---|
| Recall | 0.649 |
| F1-score | 0.724 |
| Average Precision | 0.582 |
| ROC-AUC | 0.812 |
These results indicate that the model successfully identifies a substantial proportion of customers at risk of churn while maintaining good overall classification performance.
Beyond standard model evaluation, additional analysis was performed on prediction errors.
Two complementary investigations were carried out:
Customers incorrectly predicted as retained were compared with correctly identified churners to understand which characteristics make churn difficult to detect.
A Random Forest classifier was trained on these subsets to estimate feature importance and identify attributes contributing to misclassification.
A second experiment attempted to distinguish false negatives from correctly classified non-churners.
This analysis showed that these groups are considerably harder to separate, suggesting that many false negatives exhibit behavioral patterns very similar to customers who genuinely remain with the company.
.
├── data/
│ └── telecom_churn.csv
├── models/
│ └── gb_83.pkl
├── results/
├── pipeline.ipynb
└── README.md
Possible extensions of this project include:
- SHAP-based global and local model interpretation;
- deployment as a REST API or interactive dashboard.