This project develops an end-to-end customer segmentation framework using the Online Retail II transactional dataset. Transaction-level data is cleaned, transformed into customer-level behavioral features, and segmented using K-Means clustering.
The analysis combines RFM analysis, behavioral feature engineering, correlation analysis, VIF-based feature redundancy analysis, log transformation, standardization, clustering, PCA visualization, and business interpretation.
The objective is to identify meaningful customer groups based on purchasing behavior and translate those groups into actionable business strategies.
Key questions addressed:
- Who are the most valuable customers?
- Which customers are highly engaged?
- Which valuable customers are becoming inactive?
- Which customers have low engagement and monetary value?
- How should marketing strategies differ across customer segments?
Dataset link: https://archive.ics.uci.edu/dataset/502/online+retail+ii
The project uses the Online Retail II dataset containing transactions from:
- 2009–2010
- 2010–2011
| Dataset | Records | Features |
|---|---|---|
| 2009–2010 | 525,461 | 8 |
| 2010–2011 | 541,910 | 8 |
| Combined | 1,067,371 | 8 |
| Feature | Description |
|---|---|
| Invoice | Invoice/transaction identifier |
| StockCode | Product identifier |
| Description | Product description |
| Quantity | Number of units purchased |
| InvoiceDate | Transaction date and time |
| Price | Unit price |
| Customer ID | Unique customer identifier |
| Country | Customer's country |
Online Retail II Dataset
↓
Data Integration
↓
Data Cleaning
↓
Exploratory Data Analysis
↓
Customer-Level Aggregation
↓
RFM Analysis
↓
Behavioral Feature Engineering
↓
Correlation Analysis
↓
VIF-Based Feature Redundancy Analysis
↓
Outlier & Skewness Analysis
↓
Log Transformation
↓
Standardization
↓
K-Means Clustering
↓
Elbow Method + Silhouette Analysis
↓
Final K = 5
↓
Cluster Profiling
↓
PCA Visualization
↓
Business Recommendations
The two yearly datasets were loaded and combined.
The following cleaning steps were performed:
- Removed 34,335 duplicate records
- Removed 243,007 records with missing Customer ID
- Removed 18,390 cancelled invoices
- Removed 70 records with invalid prices
- No records with Quantity ≤ 0 remained after cancellation filtering
After cleaning, 779,425 transaction records remained.
A new feature, TotalAmount, was created:
TotalAmount = Quantity * PriceThis represents the monetary value of each transaction line.
Key metrics after cleaning:
| Metric | Value |
|---|---|
| Total Revenue | 17,374,804.27 |
| Customers | 5,878 |
| Orders | 36,969 |
| Products | 4,631 |
| Countries | 41 |
The transaction-value distribution was strongly right-skewed, with most transactions having relatively low values and a smaller number of high-value transactions.
Extreme values were excluded from selected visualizations for readability but were not automatically removed from the modeling dataset.
The transaction-level data was aggregated by Customer ID.
This transformed:
779,425 transactions → 5,878 customers
The customer-level dataset became the basis for segmentation.
RFM analysis formed the core of the customer segmentation framework.
Measures how recently a customer made a purchase.
Recency = Reference Date - Last Purchase Date
Lower values indicate more recent activity.
Measures how frequently a customer purchases, calculated using the number of unique invoices/orders.
Measures total customer spending:
Monetary = Sum of TotalAmount
Higher values indicate higher customer value.
The reference date used was 10 December 2011, based on the final transaction date in the dataset.
The following features were created:
AvgOrderValue = Monetary / Frequency
Number of distinct products purchased by a customer.
Total number of units purchased.
Number of distinct calendar days on which a customer made purchases.
PurchaseDays was initially included but later removed due to severe redundancy with Frequency.
Important correlations included:
| Feature Pair | Correlation |
|---|---|
| Frequency – PurchaseDays | 0.97 |
| Monetary – TotalQuantity | 0.87 |
| Frequency – UniqueProducts | 0.69 |
| UniqueProducts – PurchaseDays | 0.71 |
The 0.97 correlation between Frequency and PurchaseDays indicated substantial redundancy.
VIF was used as an additional feature-redundancy diagnostic.
| Feature | VIF |
|---|---|
| Frequency | 24.63 |
| PurchaseDays | 23.16 |
| Monetary | 5.26 |
| TotalQuantity | 4.46 |
| UniqueProducts | 2.70 |
| AvgOrderValue | 1.20 |
| Recency | 1.08 |
PurchaseDays was removed because of its extremely high redundancy with Frequency.
| Feature | VIF |
|---|---|
| Monetary | 5.06 |
| TotalQuantity | 4.46 |
| Frequency | 3.36 |
| UniqueProducts | 2.52 |
| AvgOrderValue | 1.18 |
| Recency | 1.08 |
The final modeling features were:
Recency
Frequency
Monetary
TotalQuantity
UniqueProducts
AvgOrderValue
TotalQuantity was retained because it captures purchase volume, providing a distinct behavioral dimension from monetary spending.
Customer-level features were examined using boxplots and distribution plots.
The customer features were strongly skewed, particularly Frequency, Monetary, Total Quantity, and Unique Products.
Extreme observations were not automatically deleted because unusually high purchasing activity can represent legitimate high-value customers.
The log1p() transformation was applied to reduce skewness and limit the influence of extreme values:
X_log = np.log1p(X)Since K-Means is distance-based, the final features were standardized using StandardScaler.
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X_log)K-Means was selected as the primary clustering algorithm.
The number of clusters was evaluated for:
K = 2 to 10
The inertia curve showed a noticeable flattening around approximately K = 5–6.
The highest silhouette score occurred at K = 2, but two clusters would produce overly broad customer groups for the business objective.
K = 5 was selected as a practical compromise between clustering quality, cluster granularity, and business interpretability.
KMeans(
n_clusters=5,
random_state=42,
n_init=10
)0.2348
This indicates relatively weak geometric separation between the five clusters. However, the clusters exhibit meaningful behavioral differences that can be translated into actionable business strategies.
| Cluster | Customers | Segment |
|---|---|---|
| 1 | 1,014 | High-Value Loyal Customers |
| 3 | 1,163 | Recent Occasional Customers |
| 2 | 1,209 | At-Risk High-Value Customers |
| 4 | 1,634 | Dormant Low-Value Customers |
| 0 | 858 | Lost/Inactive Customers |
| Cluster | Recency | Frequency | Monetary | Total Quantity | Unique Products | Avg Order Value |
|---|---|---|---|---|---|---|
| 1 | 28.7 | 21.1 | 12,148.8 | 7,216.5 | 230.8 | 601.4 |
| 3 | 33.2 | 4.6 | 1,036.5 | 627.7 | 68.8 | 246.4 |
| 2 | 205.4 | 4.9 | 2,400.3 | 1,620.3 | 92.0 | 622.4 |
| 4 | 346.8 | 1.8 | 501.9 | 268.1 | 29.6 | 315.7 |
| 0 | 350.4 | 1.4 | 149.7 | 80.8 | 9.7 | 115.8 |
These customers are highly active, purchase frequently, and generate substantially higher revenue.
Strategy: loyalty programs, exclusive offers, personalized recommendations, early access, cross-selling, and upselling.
Objective: retain and reward.
These customers are recently active but purchase less frequently and spend less than the high-value segment.
Strategy: personalized recommendations, cross-selling, repeat-purchase incentives, bundles, and limited-time offers.
Objective: increase purchase frequency and customer value.
These customers have been inactive for a long period but have historically generated substantial revenue and have a high average order value.
Strategy: targeted win-back campaigns, personalized discounts, reminders, and recommendations based on previous purchases.
Objective: reactivate historically valuable customers.
These customers have been inactive for nearly a year and have relatively low historical engagement and value.
Strategy: low-cost email campaigns and broad reactivation offers.
Objective: reactivate without excessive marketing expenditure.
These customers have the lowest frequency, monetary value, and product diversity and have not purchased for a very long period.
Strategy: low-cost or selective reactivation; deprioritize if campaigns are unsuccessful.
Objective: minimize unnecessary marketing expenditure.
PCA was used to visualize the customer feature space after standardization.
The first two principal components explained approximately 83.73% of total variance.
| Component | Variance Explained |
|---|---|
| PC1 | 67.69% |
| PC2 | 16.04% |
| PC3 | 8.84% |
| PC4 | 5.64% |
| PC5 | 1.74% |
| PC6 | 0.05% |
PCA was used for visualization rather than replacing the original feature space used by K-Means.
The High-Value Loyal segment contains 1,014 customers but has an average monetary value of approximately 12,149, far above the other segments.
The At-Risk High-Value segment contains 1,209 customers with average monetary value of approximately 2,400. Their historical value makes them a priority for win-back campaigns.
Recent Occasional Customers can potentially be converted into higher-value customers by increasing purchase frequency and basket value.
Dormant Low-Value Customers form the largest segment with 1,634 customers, but their average monetary value is only approximately 502.
The At-Risk High-Value segment demonstrates that inactive customers can still be strategically important when their historical spending is high.
The clustering analysis transformed more than 779,000 cleaned transaction records into an actionable segmentation framework covering 5,878 customers.
Five customer profiles were identified:
- High-Value Loyal
- Recent Occasional
- At-Risk High-Value
- Dormant Low-Value
- Lost/Inactive
The segmentation enables the business to move from a one-size-fits-all marketing strategy toward targeted customer engagement.
The key business opportunity is to retain high-value loyal customers while prioritizing the reactivation of historically valuable customers who have become inactive.
Overall, the project demonstrates how unsupervised learning can identify customer behavior patterns and translate them into practical marketing and retention strategies.
- Python
- Pandas
- NumPy
- Matplotlib
- Seaborn
- Scikit-learn
- K-Means
- StandardScaler
- PCA
- Silhouette Score
- Statsmodels
- Variance Inflation Factor (VIF)
Customer-Segmentation/
│
├── customer_segmentation.ipynb
├── README.md
└── data/
└── online_retail_II.xlsx
pip install pandas numpy matplotlib seaborn scikit-learn statsmodels openpyxlPlace the Online Retail II Excel file in the data/ directory.
jupyter notebook customer_segmentation.ipynbRun the notebook cells sequentially.
- Transformed transaction-level data into customer-level behavioral data.
- Applied RFM analysis to measure customer engagement and value.
- Engineered additional behavioral features.
- Used correlation and VIF to identify redundant features.
- Removed
PurchaseDaysbecause of severe redundancy withFrequency. - Applied log transformation to reduce skewness.
- Standardized features before distance-based clustering.
- Evaluated K-Means across multiple values of K.
- Selected K = 5 based on a balance of clustering quality and business interpretability.
- Used PCA for visualization.
- Identified five actionable customer segments.
- Developed segment-specific business recommendations.
Soham Sarkar
MSc in Statistics, IIT Kanpur