UI/Process Design For A/B Testing
For fixed horizon A/B tests, the sample size needs to be determined based on alpha, beta, effect size and variance prior to a test. Therefore, in the experiment UI setup page, we need to ask users to enter alpha, beta, effect size and variance. Then the system uses these values to compute the required sample size. Is this correct?
Alpha, beta and effect size(mde) can be defined clearly by users. How about variance? We often observe conversion rates actually change over time during a test. This means variance can change over time as well. Also it is hard for users to estimate variance correctly prior to a test even if variance is stationary. If inputting variance is not feasible, what is the alternative solution for this?
Another UI question: if we use the calculated sample size to terminate the test, I assume we don't need to ask users to enter an end date for a test during setup. In other words, no end date set up for a test and the test is concluded automatically whenever the required sample size is reached. Is this correct?