A/B Testing

A/B testing is the closest thing marketing has to a truth serum — and most teams are still making decisions based on gut feel instead. I've watched a single headline test on a landing page lift conversions 37%. I've also watched teams run tests with 200 visitors and declare a winner. The tool is only as good as the discipline behind it.

What Is A/B Testing?

A/B testing (also called split testing) is a controlled experiment where you compare two versions of a marketing asset — Version A (the control) and Version B (the variant) — to determine which performs better against a specific metric. You randomly split your audience so each group sees only one version, then measure the difference in outcomes like conversion rate, click-through rate, revenue per visitor, or engagement.

The method comes from randomized controlled trials in clinical research. The logic is identical: isolate one variable, test it against a control, and let the data tell you what works. In marketing, you can test virtually anything — email subject lines, landing page layouts, pricing displays, CTA button colors, ad copy, checkout flows, even entire brand positioning angles.

What separates real A/B testing from just "trying stuff" is statistical rigor. You need a hypothesis, a sufficient sample size, a defined test duration, and a predetermined significance threshold (usually 95% confidence). Without these, you're just gambling with data.

The Framework

Real-World Examples

Common Mistakes

Calling tests too early. This is the cardinal sin. With a small sample, random variance looks like a real difference. A test that shows a "25% lift" after 500 visitors might show 0% after 5,000. Commit to a sample size before you start and don't touch the results until you hit it.

Testing too many variables at once. If you change the headline, image, CTA, and layout simultaneously, you can't know which change drove the result. Test one variable at a time (A/B test) or use multivariate testing if you have enough traffic to support it.

Ignoring practical significance. A test might be statistically significant (p < 0.05) but only show a 0.3% improvement. Is that worth the engineering effort to implement? Statistical significance and business significance are different things.

Not accounting for external factors. Running a test during Black Friday and comparing it to normal traffic will produce misleading results. Segment your analysis and watch for seasonal, day-of-week, and promotional period effects.

Testing low-impact elements. Button color tests are the meme of A/B testing for a reason. Test things that matter: value propositions, pricing structures, offer framing, page layouts, and positioning angles. Test big, not small.

How It Connects to Other Concepts

Conversion rate optimization is A/B testing's primary domain. Every conversion rate improvement project should be backed by test data, not opinions.

A/B testing helps determine optimal positioning by testing different value propositions and messaging angles against real audience behavior rather than focus group opinions.

Penetration pricing vs. price skimming decisions can be informed by price sensitivity tests — showing different price points to different segments and measuring price elasticity in real time.

ROMI improves directly when A/B testing eliminates underperforming creative and optimizes high-performing variants.

Frequently Asked Questions

How long should an A/B test run?

Until you hit your predetermined sample size with 95% statistical confidence. For most websites, this means 2-4 weeks minimum. Never run a test for less than one full business cycle (typically 7 days) to account for day-of-week effects.

How much traffic do I need for A/B testing?

Depends on your current conversion rate and the minimum detectable effect you care about. To detect a 10% relative improvement on a 5% conversion rate at 95% confidence, you need roughly 31,000 visitors per variation. Use a sample size calculator.

Can I run multiple A/B tests simultaneously?

Yes, if the tests are on different pages or different elements that don't interact. Running overlapping tests on the same page creates interaction effects that can invalidate both tests.

What if neither version wins?

That's a valid result. It means the variable you tested doesn't meaningfully affect the outcome. Document it and test something else. Inconclusive tests still produce knowledge.

Is A/B testing the same as multivariate testing?

No. A/B tests one variable with two versions. Multivariate testing examines multiple variables and their combinations simultaneously. Multivariate requires significantly more traffic but can uncover interaction effects between variables.

What's the minimum conversion rate needed for A/B testing?

There's no minimum, but lower conversion rates require larger sample sizes to detect meaningful differences. If your conversion rate is 0.5%, you may need 100,000+ visitors per variation to detect a 20% relative improvement.

How do I A/B test email campaigns?

Split your email list randomly. Test subject lines (open rate), preview text (open rate), CTA copy (click rate), send time (open rate), or content format (conversion rate). Most email platforms have built-in A/B testing features.

Does A/B testing work for B2B with low traffic?

It's harder but not impossible. Focus on high-volume touchpoints (email, ads), use longer test durations, and accept larger minimum detectable effects. For very low-traffic scenarios, consider qualitative user testing instead.

Sources & References

1. "A/B Testing Guide." Optimizely
2. Kohavi, Ron et al. Trustworthy Online Controlled Experiments. Cambridge University Press, 2020.
3. "Sample Size Calculator." Evan Miller
4. "The Obama Campaign's A/B Testing." Optimizely Blog
5. "Experimentation at Netflix." Netflix Tech Blog
6. "A/B Testing Statistics." Harvard Business Review

Written by Conan Pesci · April 4, 2026