A/B testing

How a split test is supposed to work

Two versions of the same page go live at once. Visitors are sent to one or the other at random, and the split keeps running until enough people have seen each version that a real difference can be told apart from ordinary noise. Then you keep the winner.

Random assignment is the whole trick. Both versions face the same week, the same weather, the same holiday, the same competitor promotion. That is what makes the comparison fair, and it is what publishing a change and then eyeballing the graph afterward can never give you.

The discipline is one difference at a time. Say a quote form asks nine questions and you suspect that is too many, so version B asks four. If version B also carries a new headline and a different photo, a win tells you nothing about which change earned it.

Why a small local site usually cannot run one honestly

Here is the arithmetic nobody leads with. Telling a modest improvement apart from random variation takes thousands of visits per version. Not thousands in total. Thousands each.

Take a site with a few hundred visits a month across every page, as a worked example. Split that between two versions of one page and the test needs the better part of a year to conclude anything, by which point the season, your pricing, and the competition have all moved underneath it. We are Orlando-based and have worked with local service businesses nationwide since 2008, and on sites that size a split test is usually a way to feel scientific rather than a way to learn something.

A result declared after a few dozen visits is a coin flip with a chart on it. The software will still show a winner, because one percentage always resolves higher than the other.

So the method on a small site has to be different. Remove friction where you find it, change one thing at a time, write down what changed and on what date, then compare a long stretch against the same stretch a year earlier rather than this month against last. That log is what makes a later swing readable at all. Six months on, it is the only thing that can tell you whether the drop you are staring at started the week you rebuilt the quote form or had been there all along. It is how our conversion work runs on sites that will never have test volume.

Terms that sit close to this one

Conversion rate is the number a test is trying to move, and it is worth defining before anything goes live, because a test with no agreed outcome ends in an argument rather than a decision. A micro conversion is a smaller step along the way, useful when the real conversion is too rare to measure quickly, and misleading when a rise in the small number gets read as a rise in paid work.

A landing page is the usual thing tested, since it exists to produce one action. A heatmap sits at the other end of the same job: it observes and suggests, while a test decides.