AI A/B testing means using an AI model for the work around a split test: reading results, explaining them, proposing the next hypothesis and building the variant. It should not mean letting the model do its own statistics.
With ElasticFunnels, an agent such as Claude Code or Codex reads the test with ef stats split, which reports the significance result the app computed, and the funnel with ef stats. It proposes one next test, you approve it, and it builds it under the same URL.
I like AI agents for the work around a split test, and I keep them well away from the maths. A model asked to judge six sessions can talk itself into a winner, and it will explain that winner very convincingly. The app already computes significance the same way every time. So the agent reads, explains and builds, and a person decides what ships.
Who does what
Split the job so each part is done by the thing that is good at it. Get this table right and most of the rest is plumbing.
| Job | Who |
|---|---|
| Assign visitors to variants | The ElasticFunnels server |
| Count sessions, sales and revenue per variant | The server |
| Compute significance | The server: a fixed-horizon z-test, Bonferroni with 3+ variants, flagged below the minimum sample |
| Read the result and explain it | The agent |
| Look for the next idea in the breakdowns | The agent |
| Pick the winner and approve the next test | You |
Step 1: give the agent the why
The server knows the numbers of each test, not what it was for. Keep a split-tests.md in the project with the test id, the hypothesis, which variant is the control and what a win means. The Claude Code walkthrough writes the first entry. The ef-stats skill that ef init installs tells the agent to read that file before it reports on a test. It's the most boring file in the project, and the one I'd least want to lose, because a test result without its hypothesis is just a number.
Step 2: read the result
List the tests, then read one:
ef stats splits
ID NAME TYPE STATUS TARGET VIEWS
526 Sleep advertorial: 3 a.m. vs 30 nights split_test running Demo Sleep Advertorial 0
ef stats split 526 --metrics sessions,conversion_rate,revenue --range 30d
The output is a table per variant plus a result line. This one had only our test visits:
VARIANT SESSIONS CONVERSION_RATE REVENUE
A: 3 a.m. (control) 3 0.00% $0.00
B: 30 nights 3 0.00% $0.00
p-value — power —
No winner yet — not enough data to score this test.
There are three possible results, and the agent should report each one differently:
- A variant is significant and every arm has its sample. Report it with the p-value and the lift over the control, as a candidate for you to decide on.
- Not significant yet, but the p-value looks low. The test is trending, not conclusive: the sample floor isn't met yet. A low p-value on a small sample is what stopping too early looks like. Report how many more sessions each arm needs.
- Not enough data. Say so and keep the test running.
The second case is the one I'd teach an agent to be most careful with. A low p-value on a small sample reads like good news, and it's exactly the moment a tired person ships a false winner.
If a variant shows up as j:null or blank, the graph's nodes have no codes and the visitors were never really split. That test has no result; re-push it through the CLI, which adds the codes, and start again.
Step 3: read the funnel around the test
The next idea usually comes from the breakdowns. The agent can see which metrics your brand tracks with ef stats metrics and which dimensions it can group by with ef stats fields, then ask:
ef stats --range 30d ef stats by utm_source -m revenue,sessions,conversion_rate ef stats by device -m sessions,conversion_rate ef stats by page -m revenue,sessions --limit 10
ef stats by utm_source run on our demo brand (demo data).Four rules keep the agent's reading honest. The ef-stats skill teaches all four:
- State the range and the timezone. Days are counted in a timezone; pass
--tzor set it once withef config set analyticsTz America/New_York. - Unavailable is not zero. A metric your brand doesn't track is listed as unavailable.
- "Change" is against the previous period of the same length, not last year.
- Check the row count.
--limitcuts the table and says so, and the blankutm_sourcerow is direct or untagged traffic.
Rule two matters more than it looks. An agent that reads "unavailable" as zero will write you a confident paragraph about a metric that was never tracked.
Step 4: ask for one next test
Give the agent a clear brief and ask for a proposal, not a change:
Read split-tests.md and the result of split test 526 for the last 30 days, in America/New_York. Then look at conversion rate by device and by utm_source. Propose ONE next test: the hypothesis, the single change, the control, and what would count as a win. Don't build it yet, and don't call a winner: report the result and leave that to me.
A useful proposal is short and testable: one change, tied to something in the numbers, with a win condition you can check with the same command later. If the agent suggests five changes at once, ask for one. A test that changes everything can't tell you what worked. I'd rather get one testable idea a week than five clever ones on a Monday.
Step 5: build it, record it, ship the winner
Once you approve, the agent builds the variant and wires the test exactly as in the advertorial walkthrough: a new page, a Split Test node under the same URL, validate, push. Then it appends the new test to split-tests.md.
When each arm has reached the planned sample and a variant is significant, the decision is yours. Mark it in the app: open the page's events and use Mark as Winner on that arm (or run ef splits winner from the CLI). Every visitor then gets the winning page at the same URL, and the ads never change. In ElasticFunnels the winner is a button a person presses, and I think that's exactly where it belongs.
More on how to size and judge a test in split testing landing pages under one URL, and everything an agent can do with your funnel on the AI marketing agents page.
- I'd never paste raw numbers into a chat and ask a model "is B winning?". It will answer, and the answer will sound right.
- Don't give an agent the power to end a test. Reading and building are cheap to undo; shipping a false winner to all your traffic isn't.
- Push back on a proposal that arrives with five changes. Ask what in the numbers points to each one.
- If a result surprises you, check the date range and timezone before you check the page.
The agent is fast, patient and happy to read a breakdown table at midnight. Let it do the reading and the building. Keep the judgment for yourself, and you get more tests out of the same week without fooling yourself any faster.
