Statistical analysis of an A/B test for Luminary Shop, an e-commerce retailer testing a redesigned checkout flow. The test ran for 30 days across 10,000 users split evenly between control (old checkout) and treatment (new checkout).
Data scope: 10,000 users | 30 days | Jan 2024
| Metric | Control | Treatment | Uplift | Significant |
|---|---|---|---|---|
| Conversion Rate | 12.35% | 15.07% | +22.0% | YES (p=0.0001) |
| Avg Revenue/User | $9.75 | $11.69 | +19.9% | YES (p=0.003) |
Verdict: Ship it. Both conversion rate and revenue per user show statistically significant improvement with p < 0.05.
- +22% conversion uplift — treatment group converted at 15.07% vs 12.35% control
- p-value = 0.0001 — result is highly significant, not due to chance
- 95% CIs do not overlap — [11.4%, 13.3%] vs [14.1%, 16.1%]
- Revenue uplift +19.9% — confirmed via t-test (p=0.003)
- Consistent across all devices — Desktop +23%, Mobile +20%, Tablet +29%
- Adequately powered — 4,924 users per group vs 2,512 required for 80% power
- Two-proportion z-test for conversion rate comparison
- Independent samples t-test for revenue comparison
- 95% confidence intervals via normal approximation
- Sample size / power analysis post-hoc
- Python (pandas, NumPy, SciPy, statsmodels, Matplotlib)
- Methods: z-test, t-test, confidence intervals, power analysis
ab-testing-case-study/
├── data/
│ └── ab_test_data.csv
├── notebooks/
│ ├── generate_data.py
│ └── analysis.py
└── reports/
├── conversion_comparison.png
├── daily_trend.png
└── daily_trend.csv