A vendor I can guarantee is using a one-tailed test: with our A/B Testing Calculator and Statistical Significance and Sample Size Calculators. Vendors using two-tailed tests according to the ConversionXL article (Jul 2015), include: Optimizely, VWO (Visual Website Optimizer), Adobe Target, Maxymiser, Convert, Monetate.
However, while preparing for my course on “ Statistics in A/B Testing” for the ConversionXL Institute, an article has come to my attention, which collected advise from prominent figures in the CRO industry who mostly recommended two-sided tests over one-sided tests for reasons so detached from reality and so contrary to all logic, that I was forced to recognize the issue as being much bigger than suspected. The fact that almost all known industry vendors are reported as using two-sided tests either by default, or as the only option made me further realize the issue warrants much more than a comment under the article itself if I am to have any chance of properly covering the topic.
The question of whether one should run A/B tests (a.k.a online controlled experiments) using one-tailed versus two-tailed tests of significance was something I didn’t even consider important, as I thought the answer (one-tailed) was so self-evident that no discussion was necessary.
0 kommentar(er)
