GlossaryGrowth & analytics
What is an A/B test?
Also called: split test, A/B testing, split testing, multivariate test
Definition
An A/B test randomly splits visitors between two versions of a page or feature, A and B, and compares a chosen metric, such as sign-up rate, to see which version performs better beyond chance.
A/B test, explained
The mechanics are simple: decide what you're testing and which metric decides the winner, split traffic randomly, run until you've reached the sample size you planned, then compare. The hard parts are statistical and practical.
The biggest statistical trap is peeking. If you check results repeatedly and stop the moment one version looks significantly better, you'll declare many false winners. Evan Miller's well-known essay "How Not To Run an A/B Test" shows how badly repeated peeking distorts significance levels. Decide the sample size in advance, or use a testing method designed for continuous monitoring.
The practical trap is running tests without enough traffic. Detecting a modest improvement in a low conversion rate can require tens of thousands of visitors per version. Most early startups don't have that. At low traffic, test big, bold changes (a different headline and offer, not a button color), or skip testing and learn from user interviews and session recordings.
Tests on public pages also need to respect search engines. Google's guidance on website testing is specific: don't cloak, meaning don't show Googlebot a different version than users see; if variants live on different URLs, use rel="canonical" pointing to the original; use 302 (temporary) redirects rather than 301s for test redirects; and run experiments only as long as necessary, then remove the test setup.
Record every test: the hypothesis, the variants, the result and what you decided. The log becomes a guide to what your audience responds to.
Why it matters for founders
A/B tests replace opinions with evidence, but only when run correctly. Bad tests produce false winners that you then build on.
Example
A startup with 40,000 monthly visitors tests a new pricing page layout, planned for two weeks at a fixed sample size, with the variant on a separate URL canonicalized to the original. Sign-ups rise, and the result holds after launch.
Common mistakes
- Stopping a test the moment it looks significant.
- Testing tiny changes on too little traffic.
- Showing Googlebot a different version, which is cloaking.
Sources
- Evan Miller: How not to run an A/B test
- Google Search Central: Minimize A/B testing impact in Google Search
Checked
Related terms
- Conversion funnelA conversion funnel is the sequence of steps a person takes from first discovering your product to becoming a paying customer, such as visit, sign-up, activation and purchase, with the percentage who move from each step to the next.
- Landing pageA landing page is a web page designed for visitors arriving from a specific source, such as an ad, launch, email or search, with a single goal like signing up, starting a trial or joining a waitlist.
- Call to action (CTA)A call to action (CTA) is the prompt, usually a button or link, that tells a visitor what to do next, such as "Start free trial", "Launch your product" or "Join the waitlist".
- Canonical tagA canonical tag is a line in a page's HTML,
<link rel="canonical" href="…">, that tells search engines which URL is the main version when several URLs show the same or very similar content. - 301 vs 302 redirectA 301 redirect says a page has moved permanently, so search engines should treat the new URL as canonical. A 302 says the move is temporary, so they keep showing the original URL.