Microsoft Teams is testing a new onboarding experience for new users. The hypothesis is that the new onboarding flow increases user activation rate, defined as completing at least 3 key actions in the first 7 days. This analysis determines whether the observed improvement is statistically significant or due to random chance.
| Parameter | Value |
|---|---|
| Baseline activation rate | 35% |
| Minimum detectable effect | 5% |
| Significance level (alpha) | 0.05 |
| Statistical power | 80% |
| Required sample size per group | 1,471 |
| Total users in experiment | 2,942 |
| Metric | Control | Treatment |
|---|---|---|
| Users | 1,471 | 1,471 |
| Activated users | 519 | 593 |
| Activation rate | 35.28% | 40.31% |
| Absolute lift | — | +5.03% |
| Relative lift | — | +14.26% |
- Test used: Two-proportion Z-test
- Z-statistic: significant
- P-value: 0.0049
- Result: Statistically significant (p < 0.05)
- 95% Confidence Interval for lift: (1.53%, 8.53%)
We reject the null hypothesis. The new onboarding experience produces a statistically significant improvement in activation rate.
At 100,000 monthly new users:
- 5,030 additional activated users per month
- $603,671 estimated annual revenue impact
- Power analysis: Calculated required sample size before running the experiment to ensure reliable results
- The peeking problem: Showed how checking results before reaching required sample size inflates false positive rates
- Confidence intervals: Reported a range of plausible effects rather than just a point estimate
- Business translation: Connected statistical results to concrete revenue impact
Ship the new onboarding experience to 100% of new users. Monitor activation rate for 30 days post-launch to confirm the effect holds outside experiment conditions and watch for novelty effect decay.
- Python 3.9
- pandas, numpy
- scipy (statistical testing)
- statsmodels (power analysis)
- matplotlib, seaborn
ab-testing-experiment/
├── notebooks/
│ ├── 01_ab_test_analysis.ipynb # Full analysis
│ ├── ab_test_results.png # Results visualization
│ └── ab_test_peeking.png # Peeking problem visualization
├── .gitignore
├── README.md
└── requirements.txt
# Clone the repo
git clone https://github.com/philipatosam/ab-testing-experiment.git
cd ab-testing-experiment
# Create and activate virtual environment
python3 -m venv venv
source venv/bin/activate
# Install dependencies
pip install -r requirements.txt
# Launch Jupyter
jupyter notebook- Run a sequential testing approach to handle peeking properly using methods like alpha spending
- Segment results by user type, device, and region to find where the effect is strongest
- Test for heterogeneous treatment effects across user segments
- Monitor for novelty effect decay in post-launch analysis