A/B testing, also called split testing, compares a current version, the control, with a changed version, the variant. The tool sends each visitor or recipient to one version at random. Once enough people have seen both, you compare the one metric you agreed on up front, such as sign ups, purchases or email clicks.
Because both groups see their version during the same period, outside factors like seasonality, news or a competitor sale affect them equally. That's what separates a real test from a before and after comparison. You'll find split tests in email tools, ad platforms (Google Ads experiments, Meta A/B tests), website testing software and many landing page builders.
Mechanics
How a split test reaches a verdict
You begin with a hypothesis: shortening the form will raise completed requests because visitors abandon at the phone number field. You pick one primary metric, decide how many visitors each version needs before you look at the result, and let the tool split traffic, usually half and half.
When the test ends, the question is whether the gap between versions is larger than random noise would produce. Testing tools report this as statistical significance or as a probability that the variant beats the control. The smaller the difference you expect and the lower your baseline rate, the more traffic you need. That's why a low volume site usually can't test a small wording change in any reasonable time.
The size of the change is what moves the result. Rewriting the offer, the headline or the form length tends to shift behavior more than changing a button color, so early tests should target the parts of the page visitors actually read.
Example
A worked example on a signup page
Assume a software company tests a new headline on its trial signup page. Over three weeks, 4,000 visitors see the control and 4,000 see the variant. The control produces 120 trials, a conversion rate of 3.0 percent. The variant produces 152 trials, or 3.8 percent.
The relative lift is the difference divided by the control rate: 0.8 divided by 3.0, which is about 27 percent. Say the testing tool reports that a gap this size would show up by chance less than 5 percent of the time. The team calls the variant the winner and rolls it out.
The effect on cost is direct. If paid traffic of 8,000 visits a month costs 16,000, the control would bring 240 trials at about 67 each, while the variant would bring 304 trials at about 53 each. Stop the same test after 500 visitors per side, and a gap of a handful of trials could just as easily have pointed the other way.
Use
When a split test is the right tool
- Use it when a page or email gets enough traffic to reach a clear answer within a few weeks, since tests that run for months pick up too many outside changes.
- Use it for decisions that matter to revenue, such as the offer, the price presentation, the form or the main headline, rather than cosmetic details.
- It misleads when the metric is too shallow, for example a subject line that wins on opens but loses on purchases further down the funnel.
- For a low traffic site, a clear rewrite based on customer interviews is often more useful than a test that never reaches a result.
Watch out
Common A/B testing mistakes
- Checking the dashboard every day and stopping the test the moment the variant pulls ahead, which crowns winners that are only random swings.
- Changing the headline, the image and the form in one variant, then having no idea which of the three changes caused the result.
- Running a test over a holiday week only, so the winner reflects unusual shoppers who will not be there next month.
- Measuring button clicks instead of the action the business actually needs, such as a completed purchase or a qualified request.
- Never writing down the hypothesis and the result, so the team repeats old tests and forgets what it already learned.
FAQ
Questions about A/B testing
How long should an A/B test run before I trust the result?
Decide the length before launch, based on your baseline rate and the smallest lift worth detecting. Many teams cover at least two full weeks so weekdays and weekends both appear. Stopping early because one side looks ahead is the usual way a false winner gets declared.
How much traffic do I need for A/B testing?
It depends on how often the goal happens and how large a change you hope to see. A page with a low conversion rate and a subtle tweak needs many visitors per version. If your sample calculator says the test would take months, test a bolder change or a higher traffic page instead.
What is the difference between A/B testing and multivariate testing?
An A/B test compares whole versions, usually one change against the original. A multivariate test mixes several elements at once, such as headline and image, to see which combination performs. Multivariate designs split visitors into many groups, so they suit sites with heavy traffic.
