Step 1: Plan Your Analysis
- What is the performance metric you’ll use to evaluate the results of your test?
We need to have at least an 18% increase in profit compared to the comparative period while compared to the control stores; otherwise known as incremental lift.
- What is the test period?
The test period was a 12 week period running from 2016-April-29 to 2016-July-21
- At what level (day, week, month, etc.) should the data be aggregated?
The data should be aggregated at the weekly level
Step 2: Clean Up Your Data
In this step, you should prepare the data for steps 3 and 4. You should aggregate the transaction data to the appropriate level and filter on the appropriate data ranges. You can assume that there is no missing, incomplete, duplicate, or dirty data. You’re ready to move on to the next step when you have weekly transaction data for all stores.
Step 3: Match Treatment and Control Units
In this step, you should create the trend and seasonality variables, and use them along with your other control variable(s) to match two control units to each treatment unit. Note: Calculate the number of transactions per store per week to calculate trend and seasonality.
Apart from trends and seasonality…
- What control variables should be considered? Note: Only consider variables in the RoundRoastersStore file.
Sq_Ft, AvgMonthSales & Region
- What is the correlation between each potential control variable and your performance metric?
From pearson correlation analysis, we see that AvgMonthSales is statistically significant to Sq_Ft with it’s p-value less than 0.05 but a low correlation. AvgMonthSales is statistically significant to Sum_Gross.Margin with a 0.78 correlation.
- What control variables will you use to match treatment and control stores?
AvgMonthSales
- Please fill out the table below with your treatment and control stores pairs:
| Treatment Store | Control Store 1 | Control Store 2 |
| 1664 | 7162 | 8112 |
| 1675 | 1580 | 1807 |
| 1696 | 1964 | 1863 |
| 1700 | 2014 | 1630 |
| 1712 | 8162 | 7434 |
| 2288 | 9081 | 2568 |
| 2293 | 12219 | 9524 |
| 2301 | 3102 | 9328 |
| 2322 | 2409 | 3235 |
| 2341 | 12536 | 2383 |
Step 4: Analysis and Writeup
Conduct your A/B analysis and create a short report outlining your results and recommendations. (250 words limit)
Answer these questions. Be sure to include visualizations from your analysis:
- What is your recommendation – Should the company roll out the updated menu to all stores?
Yes, my recommendation would be to roll out the change across all stores because making the change would increase gross margin by more than 18%.
- What is the lift from the new menu for West and Central regions (include statistical significance)?
The lift for the West region is 37.9% and the statistical level is 99.5%
The lift for the Central region is 43.5% and the statistical level is 99.6%
- What is the lift from the new menu overall?
The lift for the new menu overall is 40.7%