Split testing landing pages means sending comparable visitors to two or more versions of a page at the same time and keeping the one that sells more. For paid traffic, run the variants behind the one URL your ads already point to, so the ads never change and never go back into learning. It's the only setup I'd trust on paid traffic.
Decide the sample size before you start, run at least a full week, and judge the test on sales or revenue per visitor, not clicks. That is how ElasticFunnels split testing works: the variant is picked on the server before the page loads.
Using ElasticFunnels? Follow the step-by-step tutorial, with a clip for every click: A/B test a landing page in 7 steps.
A bad split test is worse than no test. No test leaves you guessing. A bad one leaves you confident and wrong, and you build the next three tests on top of it.
What split testing a landing page means
You have a page that works. You think a new headline, a shorter page or a different offer would work better. Someone else on the team is just as sure it won't. A split test settles it with traffic instead of opinion: half the visitors get the current page (the control), half get the new one (the variant), and after enough visitors you compare the results.
"Split test" and "A/B test" mean the same thing to most people. A test with three or more variants is sometimes called A/B/n. The rules below apply to all of them.
Why a new URL resets your ads
Most people test a new page by building it at a new URL and sending some traffic there. On paid traffic that's the expensive way to do it, and the bill never shows up in the test report.
To send traffic to a second URL you either edit the ad's destination or launch new ads. On Meta, both are significant edits: the ad set goes back into the learning phase, and it needs about 50 optimization events in a week to leave it again (Meta). During learning, delivery is less stable and costs usually move. So the variant gets judged on worse traffic than the control, and the test is biased before it starts.
Keep the URL and move the split behind it instead. The ad points at the same link it always did. When a visitor arrives, the server picks the variant and serves that page. Nothing about the ad changes, so nothing restarts. Server-side selection under one URL is how split tests work in ElasticFunnels. The ad account is the one part of the system I would never reset for a page test.
How to split test a landing page, step by step
- Write the hypothesis. One sentence: "A headline that names the problem will sell more than one that names the product." If you cannot say what you expect and why, the result will not teach you anything.
- Change one thing. The headline, the offer, the length or the proof. Change five things and a winner tells you nothing about which one mattered.
- Pick the metric before you start. Sales per visitor or revenue per visitor. Refunds count against it if you can see them.
- Work out the sample size (below) and write it down.
- Split the traffic behind the same URL, usually 50/50 between control and variant.
- Wait. At least one full week, and until each arm has its sample. Do not stop because one variant looks ahead on day two.
- Decide, then record it. Keep the winner, write down what you learned, and start the next test from the new control.
How many visitors a test needs
The smaller the difference you want to detect, the more visitors it takes. The sample size depends on four numbers: your current conversion rate, the smallest lift worth detecting (the minimum detectable effect), the confidence level (usually 95%) and the power (usually 80%). Our A/B test sample size calculator works it out from them. At 95% confidence and 80% power, with a 50/50 split:
| Current conversion | Lift you want to detect | Visitors per variant |
|---|---|---|
| 2% | to 3% (+50%) | 3,826 |
| 2% | to 2.5% (+25%) | 13,809 |
| 2% | to 2.4% (+20%) | 21,109 |
| 3.3% | to 4.05% (+23%) | 9,878 |
| 5% | to 6% (+20%) | 8,158 |
The table is arithmetic, not a benchmark. Say your page gets 1,000 visitors a week and you want to detect 2% becoming 2.4%. That's 21,109 visitors per variant, or about ten months at 50/50. Nobody waits ten months for a headline. So on a small page, test big changes (the offer, the length, a presell) and leave button colours to sites with traffic to burn.
Two more things change the number:
- An uneven split is slower. At 70/30 the smaller arm collects visitors at 3/5 the pace of a 50/50 arm, so reaching the same number per variant takes 1.67 times as long. For the same power, a 70/30 test needs about 1.2 times the total traffic of a 50/50 test.
- More variants need more visitors. When you run 3 or more variants, the significance level is split across the comparisons (a Bonferroni correction), so each variant needs more traffic.
A minimum sample setting, such as 1,000 visitors per variant, is not the same thing. In ElasticFunnels, Min Sample Size per Variant is the point below which a result is flagged "interpret with caution". Set it to the number the calculator gives you. The Academy tutorial walks through a real example: 3.30% against 4.05% on 5,493 and 2,764 sessions is not significant yet, and would need about 9,900 sessions per variant to confirm.
Judge on sales, not clicks
A variant can win on clicks to the checkout and lose on sales. A louder headline pulls more curious clicks; a clearer price scares some of them off before they click. Clicks are the metric that makes a designer happy and a finance person nervous. What matters is what came out of the cart.
In ElasticFunnels each order is recorded with the variant the buyer saw, so a split test is decided on sales and revenue per visitor, including upsells. The significance is computed on the server, with a correction when you run 3 or more variants, and a winner is not called until each arm has enough data. Do not recompute it in a spreadsheet halfway through; checking repeatedly and stopping at the first good-looking result inflates false winners (Evan Miller).
What to test first
This is the order I'd test in. Start where every visitor looks and where the money is decided, and work down from there. Button colours are not on the list.
| Test | Why it is worth it |
|---|---|
| Headline and hook | Every visitor reads it; every metric below it is multiplied by it |
| The offer: price, bundle, guarantee | Often moves revenue more than any design change |
| Page length | Cold traffic and warm traffic want different amounts of argument |
| Presell before the sales page | Tests a whole step, not a detail (see how to create a sales funnel) |
| The upsell page | Small traffic, but every buyer sees it (see what is an upsell) |
Split testing and SEO
Testing does not hurt search rankings if it follows Google's rules: Googlebot sees the same variants people do, the variant is never chosen by user agent, alternate URLs carry rel=canonical to the original, redirect-based tests use 302 rather than 301, and the test ends when you have an answer (Google Search Central). A same-URL test keeps it simple: there is only one URL to index.
Setting up a same-URL test in ElasticFunnels
The short version is below. The step-by-step Academy tutorial shows every click in the app.
- Duplicate the page and make your one change in the copy.
- Add a split test on the page: pick the variants and the traffic share.
- Leave the ads alone. The URL does not change.
- Read the result in the split test report, on sales and revenue per visitor, and promote the winner when the report calls it.
You can split test whole pages or single sections of a page. The details are on the split testing page.
The judgment calls the steps above don't cover.
- If the calculator says a test needs most of a year, don't run it on that page. Change something bigger, or test where the traffic is.
- Treat a big lift on a small sample with suspicion. Early winners tend to shrink once they get all the traffic.
- Keep a log of the losers. A clean loss is cheap research, and it stops the team from re-running the same idea next quarter.
- Test the offer before the design. A new price or guarantee can move revenue more than any layout change.
Plenty of tests lose, and that's fine. A clean loss still tells you something about your buyers. A dirty win tells you nothing, and it quietly costs you everything you build on top of it.
