We are interested in changing the color of "Start Now" button on an Udecity-like website to see the effect of click-through probability, which is measured by
# of users clicked / # of users visited
Based on 1000 users visited, we found that 100 users clicked. This gives us 10% of the click-through probability. We are also interested in
- significant level (usually referred as $\alpha$) of 5% (0.05)
- practical significance level of 2%, i.e., minimum effect that we care about
- power/sensitivity of 80% (fairly standard)
First, we use an online calculator that has been introduced in the course: https://www.evanmiller.org/ab-testing/sample-size.html. The terminologies used by this calculator and the corresponding ones that we mentioned above are:
- base conversion rate: click-through probability. This is estimated click-through probability before making the change
- minimum detectable effect: practical significance level, and we care about absolute difference.
- statistical power: power/sensitivity
- significant level: significant level
And the result of the calculator is 3,623.
Here we use the same equation in Python to derive the same result mentioned above.
## Calculate required sample size
def calc_sample_size(alpha, power, p, pct_mde, absolute=True):
""" Based on https://www.evanmiller.org/ab-testing/sample-size.html
Args:
alpha (float): How often are you willing to accept a Type I error (false positive)?
power (float): How often do you want to correctly detect a true positive (1-beta)?
p (float): Base conversion rate
pct_mde (float): Minimum detectable effect, relative to base conversion rate.
"""
if absolute:
delta = pct_mde
else:
delta = p*pct_mde
t_alpha2 = norm.ppf(1.0-alpha/2)
t_beta = norm.ppf(power)
sd1 = np.sqrt(2 * p * (1.0 - p))
sd2 = np.sqrt(p * (1.0 - p) + (p + delta) * (1.0 - p - delta))
return int(np.ceil((t_alpha2 * sd1 + t_beta * sd2) * (t_alpha2 * sd1 + t_beta * sd2) / (delta * delta)))
print(calc_sample_size(alpha=0.05, power=0.8, p=0.1, pct_mde=0.02))
Output:
3623
Source: Stackoverflow