{"id":83,"date":"2016-07-25T07:15:43","date_gmt":"2016-07-25T07:15:43","guid":{"rendered":"http:\/\/www.aarondefazio.com\/tangentially\/?p=83"},"modified":"2016-07-25T07:16:19","modified_gmt":"2016-07-25T07:16:19","slug":"how-to-do-ab-testing-with-early-stopping-correctly","status":"publish","type":"post","link":"https:\/\/www.aarondefazio.com\/tangentially\/?p=83","title":{"rendered":"How to do A\/B testing with early stopping correctly"},"content":{"rendered":"

<\/p>\n

<\/p>\n

It’s amazing the amount of confusion on how to run a simple A\/B test seen on the internet. The crux of the matter is
\nthis: If you’re going to look at the results of your experiment as it runs, you can not just repeatedly apply a 5% significance level t-test.<\/em> Doing so WILL lead to false-positive rates way above 5%, usually on the order of
\n17-30%.\n<\/p>\n

<\/p>\n

Most discussions of A\/B testing do recognize this problem, however the solutions they suggest are simply wrong. I
\ndiscuss a few of these misguided approaches below. It turns out that easy to use, practical solutions have been worked out
\nby clinical statistician decades ago, in papers with many thousands of citations. They call the setting “Group Sequential
\ndesigns” [Bartroff et al.<\/a>, 2013<\/a>]. At the bottom of this post I give tables and references so you can use group sequential
\ndesigns in your own experiments.\n<\/p>\n

<\/a>What’s wrong with just testing as you go?<\/h3>\n
<\/p>\n
The key problem is this: Whenever you look at the data (whether you run a formal statistical test or just
\neyeball it) you are making a decision about the effectiveness of the intervention. If the results look positive
\nand you may decide to stop the experiment, particularly if it looks like the intervention is giving very bad
\nresults.\n<\/p>\n
<\/p>\n
Say you make a p=0.05 statistical test at each step. On the very first step you have a 5% chance of a false positive (i.e.
\nthe intervention had no effect but it looks like it did). On each subsequent test, you have a non-zero additional probability
\nof picking up a false positive. It’s obviously not an additional 5% each time as the test statistics are highly correlated,
\nbut it turns out that the false positives accumulate very rapidly. Looking at the results each day for a 10
\nday experiment with say 1000 data points per day will give you about an accumulated 17% false positive
\nrate (According to a simple numerical simulation) if you stop the experiment on the first p<0.05 result you
\nsee.\n<\/p>\n
\n

\n <\/a><\/p>\n
<\/p>\n
$\"PIC\"$
\n <\/p>\n
Figure 1: <\/span>100 simulations of 0-effect (the null hypothesis) with no early stopping. Notice that at any point in time,
\nroughly 5% of the paths are outside the boundaries (A 5% false positive rate is expected).<\/span><\/div>\n
\n <\/div>\n
\n
\n

\n <\/a><\/p>\n
<\/p>\n
$\"PIC\"$
\n <\/p>\n
Figure 2: <\/span>Paths terminated when they cross the boundary on the next step. 12 of 100 simulations are stopped,
\ngiving a very high 12% false positive rate!<\/span><\/div>\n
\n <\/div>\n
\n
<\/a>The error-spending technique<\/h4>\n
<\/p>\n
They key idea of “Group Sequential” testing is that under the usual assumption that our test statistic
\nis normally distributed, it is possible to compute the false-positive probabilities at each stage exactly
\nusing dynamic programming. Since we can compute these probabilities, we can also adjust the
\ntest’s false-positive chance at every step so that the total <\/span>false-positive rate is below the threshold
\n $α<\/mi><\/math> we want. This is\nknow as the error spending approach. Say we want to run the test daily for a 30 day experiment. We provide a 30 element sequence\nα<\/mi><\/mrow>i<\/mi><\/mrow><\/msub \n><\/math> of per step error chances,\nwhich sums to less than sayα<\/mi>=<\/mo>0<\/mn>.<\/mo>0<\/mn>5<\/mn><\/math>.\nUsing dynamic programming, we can then determine a sequence of thresholds, one per day, that we can use with the\nstandard z-score test. The big advantage of this approach is that is a natural extension of the usual frequentist testing\nparadigm.\n<\/p>\n<\/p>\nThe error spending approach has a lot of flexibility, as the error chances can be distributed arbitrarily between the days. \nSeveral standard allocations are commonly used in the research literature.\n<\/p>\n<\/p>\nOf course, libraries are available for computing these boundaries in several languages. Tables for the most common \nsettings are also available. I’ve duplicated a few of these tables below for ease of reference.\n<\/p>\n<\/p>\n\n<\/a>Recommended approach<\/h3>\n <\/p>\n \n \n Choose a α<\/mi><\/math>-spending\n function, from those discussed below. The Pocock approach is the simplest.\n <\/li>\n Fix the maximum length of your experiment, and the number of times you wish to run the statistical test. \n I.e. daily for a 30 day experiment.\n <\/li>\n Lookup on the tables below, or calculate using the ldbounds <\/span>or GroupSeq <\/span>R packages, the z<\/mi><\/math>-score\n thresholds for each of the tests you will run. For example, for the Pocock approach,α<\/mi>=<\/mo>0<\/mn>.<\/mo>0<\/mn>1<\/mn><\/math>,\n 10 tests max, usez<\/mi>=<\/mo>3<\/mn>.<\/mo>1<\/mn>1<\/mn>7<\/mn><\/math>\n for every test. For the O’brien-Fleming approach it will be the sequence{<\/mo>8<\/mn>,<\/mo>5<\/mn>.<\/mo>8<\/mn>2<\/mn>,<\/mo>4<\/mn>.<\/mo> Start the experiment.\n <\/li>\n For each test during the course of experiment, compute the z<\/mi><\/math>-score\n to the threshold, and optionally stop the test if the score exceeds the threshold given in the lookup table.<\/li>\n<\/ol>\n<\/p>\n\n<\/a>Real valued example<\/h3>\n <\/p>\n The simplest case of A\/B testing is when each observation for each user is a numerical value, such \nas a count of activity that day, the sales total, or some other usage statistic. In this case the \n z<\/mi><\/math>-score when\nwe haven<\/mi><\/mrow>a<\/mi><\/mrow><\/msub \n><\/math> data\npoints from groupa<\/mi><\/math>\nandn<\/mi><\/mrow>b<\/mi><\/mrow><\/msub \n><\/math> from group\nb<\/mi><\/math> with means and\nstandard deviationsμ<\/mi><\/math>\nandσ<\/mi><\/math>\nis:\n<\/p>\n\n z<\/mi>=<\/mo>$