A/B testing, also called split testing, compares a current version, the control, with a changed version, the variant. Visitors or recipients are assigned to one version at random, and after enough of them have seen it you compare a single agreed metric, such as sign ups, purchases or email clicks.
Because both groups see their version during the same period, outside factors like seasonality, news or a competitor sale affect them equally. That is what separates a test from a before and after comparison. You will find split tests in email tools, ad platforms (Google Ads experiments, Meta A/B tests), website testing software and many landing page builders.
Mechanics
How a split test reaches a verdict
You begin with a hypothesis: shortening the form will raise completed requests because visitors abandon at the phone number field. You pick one primary metric, decide how many visitors each version needs before you look at the result, and let the tool split traffic, usually half and half.
When the test ends, the question is whether the gap between versions is larger than random noise would produce. Testing tools report this as statistical significance or as a probability that the variant beats the control. Smaller expected differences and lower baseline rates require more traffic, which is why low volume sites often cannot test small wording changes in any reasonable time.
What moves the result is the size of the change. Rewriting the offer, the headline or the form length tends to shift behavior more than changing a button color, so early tests should target the parts of the page visitors actually read.
Example
A worked example on a signup page
Assume a software company tests a new headline on its trial signup page. Over three weeks, 4,000 visitors see the control and 4,000 see the variant. The control produces 120 trials, a conversion rate of 3.0 percent. The variant produces 152 trials, or 3.8 percent.
The relative lift is the difference divided by the control rate: 0.8 divided by 3.0, which is about 27 percent. Say the testing tool reports that a gap this size would appear by chance less than 5 percent of the time; the team treats the variant as the winner and rolls it out.
The effect on cost is direct. If paid traffic of 8,000 visits a month costs 16,000, the control would bring 240 trials at about 67 each, while the variant would bring 304 trials at about 53 each. Had the test stopped after the first 500 visitors per side, a gap of a handful of trials could easily have pointed the other way.
Use
When a split test is the right tool
- Use it when a page or email gets enough traffic to reach a clear answer within a few weeks, since tests that run for months pick up too many outside changes.
- Use it for decisions that matter to revenue, such as the offer, the price presentation, the form or the main headline, rather than cosmetic details.
- It misleads when the metric is too shallow, for example a subject line that wins on opens but loses on purchases further down the funnel.
- For a low traffic site, a clear rewrite based on customer interviews is often more useful than a test that never reaches a result.
Watch out
Common A/B testing mistakes
- Checking the dashboard every day and stopping the test the moment the variant pulls ahead, which crowns winners that are only random swings.
- Changing the headline, the image and the form in one variant, then having no idea which of the three changes caused the result.
- Running a test over a holiday week only, so the winner reflects unusual shoppers who will not be there next month.
- Measuring clicks on a button instead of the final action, such as completed purchases or qualified requests, which is what the business needs.
- Never writing down the hypothesis and the result, so the team repeats old tests and forgets what it already learned.