Oct 07 2014

The Dangers of Quick A/B Tests

Testing is a science, and scientists go to great lengths to obtain accurate results. They ensure accuracy by writing a solid hypothesis, creating clean environments, minimizing outliers, and waiting for enough data to collect. You may not be looking for a cure or an answer regarding the cosmos, but if you want accurate results, you have to take a similar scientific approach.

A Solid Hypothesis

Sure, tests that rely on your gut instinct save time because they require little to no time to come up with a solid hypothesis, but there is a good chance that such a hypothesis is too weak. As an eCommerce store owner, it’s worth the upfront time to use analytics and other forms of data to create a solid hypothesis. The data-driven approach starts at the beginning of the testing process and sees you through to the conclusion of a test.

Qualitative data obtained through user testing and user surveys is a great place to start. Two of our favorite tools for this are UserTesting.com and Qualaroo.

The quantitative data found in your analytics can help affirm possible friction points found in the qualitative data you’ve collected, or it can be completely useful on its own. What do your current analytics tell you about how users behave on your website? Is there a point in the shopping funnel that they are having more trouble with than others? Perhaps it’s smarter to concentrate on that weakness in the shopping funnel rather than doing a “quick test” on your homepage slider content.

A Clean Testing Environment

If you don’t take the time to properly QA your test, you risk creating bugs that cause your site to not function properly, lead to revenue loss, and/or skew results. For example, the variation you create may break a feature on certain browsers, causing visitors using those browsers to convert at a significantly lower rate than the original, not because the original is better, but because broken features decrease conversion rates. Thoroughly QA-ing your test will keep this from happening. If you’re trying to fly through a test quickly, it may be tempting to skip this important step. If you do skip QA, be wary of the results. They could be wrong and you would never know it.

Limiting Stray Variables

We’ve seen it happen. A test will start towards statistical significance at an average pace, and then all of the sudden, it’s there. This can be the result of an enormous, atypical sale or a marketing push that happens during a live test (e.g. Groupon). In the world of statistics, these are referred to as outliers, observations that appears to deviate extremely from other observations in the sample, which make statistical analyses difficult. Allowing your tests to run longer and see more traffic can help minimize the effects of outliers. If you want to learn more about how much time it should take to complete a test, check out this article from Optimizely, “How Long to Run a Test.”

Test Smart

In order for you to make the best data-driven decisions, your tests need to reach statistical significance. You will greatly increase your odds of reaching a statistical significance by collecting enough data (i.e. testing a proper number of visitors), ensuring that the data has not been skewed, and starting with a solid hypothesis. Ultimately, if mitigating risk and maximizing the chances of increasing revenue are your goals, running quick tests are not the best way to achieve them.

Matthew Rickerby

Head of Marketing

Matt began his career in ecommerce at Blue Acorn over five years ago. His areas of expertise include persona development, account-based marketing, and content marketing. He has co-written speaking sessions for Bronto Summit, DIG South, GIANT, and Revolve, and received multiple awards for videography, blogging, and copywriting.

One Comment

Rick Buczynski
Oct 07 2014

I like how you encourage devising a solid hypothesis by combining qualitative and quantitative data.

I think there is a deception in the idea of a “quick A/B test” because of how easily they can now be setup these days. But, as you pointed out, statistical significance is key — and that component of testing doesn’t make the process any “faster.”

There are a couple of supplemental articles worth checking out for readers looking to go more in depth on this topic:

Ultimate Guide to A/B Testing

And an interesting counter argument to some of the above article:

How Not to Run an A/B Test

Thanks for sharing Matt!


Leave a Reply

Your email address will not be published. Required fields are marked *