Guidelines for stopping an A/B (or MVT) test early due to negative metrics
A test I've conducted recently reached significance within 24 hours showing a huge drop in conversion. As the site I work on is owned by a large company with high revenue streams, I needed to pause the test to prevent loss of income.
After doing the analysis it turned out that Split C contained a small bug which was enough to pull down all of the results. I started the test again after the fix... same issue happened. 24 hours passed and the same negative effect has been seen.
My task now is to put stoppage guidelines in place to help us decide if a test should be stopped part way through to prevent a detrimental effect to the website - is there any standard measures to use in this case e.g. if CR = -10% then stop the test. Or, is this purely circumstantial?
Any advice around this would be appreciated.