📊 A/B Test Analysis for Webpage Performance

1. 🎯 Project Goal & Business Problem

In the world of digital products, making data-driven decisions is key to growth. This project simulates a real-world scenario where a company needs to decide whether to launch a new webpage design. The goal is to determine, with statistical rigor, if the new design (treatment group) performs significantly better than the old design (control group) in terms of user conversion rate.

The core business question is: Should we invest resources to launch the new webpage design? This project provides a definitive, data-backed answer by conducting a thorough A/B test analysis.

---

2. 📈 Dashboard & Final Recommendation

The results of the analysis are presented in a professional, easy-to-understand Power BI dashboard. The dashboard highlights the key performance indicators (KPIs), the statistical significance of the results, and a clear business recommendation.

---

3. 💻 Technical Stack

• Data Analysis: Python, Pandas, NumPy
• Statistical Analysis: Statsmodels
• Development Environment: Jupyter Notebook
• Data Visualization & BI: Microsoft Power BI

---

4. ⚙️ Methodology

The project follows a structured, end-to-end analytical process to ensure the reliability and validity of the results.

a. Hypothesis Formulation

Before looking at the data, we define our hypotheses to avoid bias. The significance level (alpha) is set at α = 0.05.

• Null Hypothesis (H₀): There is no statistically significant difference in the conversion rates between the old webpage (control) and the new webpage (treatment).
  • p = p₀
• Alternative Hypothesis (H₁): There is a statistically significant difference in the conversion rates between the two webpages.
  • p ≠ p₀

b. Data Cleaning & Preparation

Data integrity is crucial for any valid analysis. The raw data from ab_data.csv was meticulously cleaned in the AB_Test_Analysis.ipynb notebook:

1. Removing Duplicate Users: The dataset contained users who were part of the experiment multiple times. To ensure each user's decision is counted only once, these duplicates were identified and removed.
2. Correcting Mismatched Groups: There were instances where users in the control group were shown the new_page, and vice versa. These misaligned records were dropped to maintain the integrity of the experiment groups.

c. Statistical Testing

With the cleaned data, a two-proportion Z-test was performed. This is the appropriate statistical test for comparing the conversion rates (which are proportions) of two different groups.

• The test calculates a p-value, which represents the probability of observing the results (or more extreme results) if the null hypothesis were true.

---

5. 💡 Results & Business Recommendation

The analysis yielded the following key results:

• Control Group Conversion Rate: 12.02%
• Treatment Group Conversion Rate: 11.87%
• Calculated P-value: 0.2323

Interpretation & Conclusion

The calculated p-value of 0.23 is significantly higher than our significance level of 0.05. This means we fail to reject the Null Hypothesis.

In business terms, there is no statistically significant evidence to suggest that the new webpage design performs any differently than the old design. The small drop in conversion rate for the treatment group is likely due to random chance and not a true effect of the new design.

Final Recommendation

Do not launch the new webpage. The experiment shows that the new design does not provide a statistically significant improvement in user conversion. Investing company time and resources to launch this new page would not yield a positive return on investment.

---

6. 🛠️ How to Replicate the Project

To set up and run this project on your local machine, follow these steps:

1. Clone the Repository:

   git clone [https://github.com/your-username/your-repository-name.git](https://github.com/your-username/your-repository-name.git)
   cd your-repository-name

2. Set Up a Virtual Environment (Recommended):

   python -m venv venv
   source venv/bin/activate  # On Windows, use `venv\Scripts\activate`

3. Install Dependencies:

   pip install pandas numpy statsmodels jupyterlab

4. Download the Dataset:

   • Download the dataset from Kaggle: A/B Testing Dataset
   • Place the ab_data.csv file in the root directory of the project.

5. Run the Analysis Notebook:

   • Launch Jupyter Lab: jupyter lab
   • Open the AB_Test_Analysis.ipynb notebook and run all cells. This will perform the statistical analysis and generate the ab_test_summary_data.csv file.

6. View the Power BI Dashboard:

   • Open the .pbix file using Power BI Desktop.
   • If prompted about a broken data source, go to Transform data -> Data source settings. Select the source and click "Change Source...". Browse to the ab_test_summary_data.csv file you generated in the previous step.