A/B Testing Your Forms: Sample Size, p-Values and When to Stop

What a form A/B test actually compares

In a form A/B test, each visitor who reaches the form is randomly shown one of two versions: A (your current form) or B (the changed one). Both run at the same time, so seasonality, ad campaigns and the day of the week affect them equally. At the end, you compare a single number for each version, usually the conversion rate: submissions divided by views.

That last part is what makes a form test a statistics problem. Each view either ends in a submission or it doesn’t, so each version’s result is a proportion. The question you’re really asking is whether the two proportions differ by more than chance alone would explain. The NIST/SEMATECH e-Handbook of Statistical Methods treats exactly this situation (in its case, comparing the share of defective parts from two processes), and the same tools apply to forms: a z-test for two proportions when samples are reasonably large, and Fisher’s exact test when they’re small.

Decide four things before you start

The most reliable tests are the ones planned before any data comes in. Write these four things down, even if only in a note, before you switch the test on.

1. One change. Change one thing between A and B: a field removed, a label reworded, a button made more specific. If B differs in five ways, a win tells you nothing about which change helped.
2. One primary metric. For most forms, that’s the conversion rate (submissions ÷ views). Secondary numbers, such as average input time, are useful context, but the decision should rest on the metric you chose in advance.
3. The smallest difference worth detecting. If your form converts at 10%, would a move to 10.5% change anything for your business? Probably not. A move to 12% might. This number, called the minimum detectable effect, drives everything else.
4. The sample size and the stopping point. Work out how many views each version needs (see below), estimate how long that takes, and commit to reading the result only once you get there.

Why a 50:50 split is the default

You can split traffic unevenly, for example 80:20 to limit exposure to a risky change. But an uneven split costs you time. Penn State’s online course notes for clinical trial design (STAT 509) show this in their worked examples: with the same goals, a 2:1 or 3:1 allocation needs a larger total sample than 1:1. The reason is intuitive: a comparison is only as precise as its smaller group, and an uneven split makes one group small.

Here is the same effect for a form. We used the normal-approximation sample-size formula from the STAT 509 notes to compare a 10% baseline against a 15% challenger, at a two-sided 5% significance level and 80% power (both explained below).

A 75:25 split needs about 45% more total traffic than 50:50 to reach the same certainty. Unless the change carries a real risk (say, removing a field your sales team relies on), keep the split even.

Two more rules about the split. First, assignment should be random, not alternating or based on time of day. Second, it should be sticky: a visitor who sees B on Monday and comes back on Wednesday should see B again. Otherwise the same person can be counted in both groups, and the two versions bleed into each other.

How many views do you need?

The sample size for comparing two conversion rates depends on four inputs. You choose three of them; the fourth comes from your own data.

For equal groups, the STAT 509 formula for the number of views per version is n = 2 × (z₁₋α/₂ + z₁₋β)² × p̄(1 − p̄) ÷ (p₁ − p₂)², where p₁ and p₂ are the two conversion rates and p̄ is their average. With α = 0.05 (two-sided) and 80% power, z₁₋α/₂ is 1.96 and z₁₋β is about 0.84. NIST’s handbook gives the same structure for a single proportion and works through an example; it also points out that a minimum sample size computed without a power term only gives a 50% chance of detecting the change, which is why the power term matters.

Plugging typical form numbers into that formula gives the views needed per version. These are our calculations, not benchmarks, and they assume a 50:50 split.

Two things stand out. Small improvements need a lot of traffic: proving that a 5% form became a 6% form takes more than 8,000 views on each version. And the required sample falls quickly as the effect grows, roughly with the square of the difference. Halve the difference you want to detect and you need about four times the views. The STAT 509 notes make the same point about confidence intervals: halving the width means quadrupling the sample.

What to do if your form doesn’t get much traffic

Many contact and quote forms see a few hundred views a month. At that volume, a test aimed at a 10% relative lift could take years. That doesn’t mean testing is pointless, but it changes what you should test.

1. Test bigger changes. Removing three fields, moving the form higher on the page, or replacing a ten-field form with a three-field one can produce large differences, and large differences need small samples.
2. Measure a step closer to the form. If submissions are rare, the share of viewers who start the form, or who complete the field you changed, accumulates faster. Treat it as a leading signal, then confirm with submissions over a longer period.
3. Accept a longer test. Running for six or eight weeks is fine as long as nothing else about the form or the traffic changes midway.
4. Don’t test what you can just fix. A broken validation rule, a field that rejects valid phone numbers, or a button hidden on mobile should simply be fixed. Testing is for choices, not for bugs.

Turning views into days

Divide the views needed per version by the daily views each version will receive. With 120 form views a day split 50:50, each version gets about 60. A test needing 687 views per version therefore takes about 12 days. Round up to whole weeks, so each version sees every weekday and weekend the same number of times; many sites get different visitors on Monday morning than on Saturday night. Two full weeks would be the plan here.

Reading a p-value in plain English

When the test ends, your tool reports a p-value for the difference between A and B. The American Statistical Association’s 2016 statement on p-values puts its first principle this way: “P-values can indicate how incompatible the data are with a specified statistical model.” In an A/B test, that model is “A and B convert at the same rate,” and the p-value is the probability of seeing a gap at least as large as yours if that model were true.

Put plainly: if the two forms really performed the same, how surprising would a gap this large (or larger) be? A p-value of 0.03 means a gap at least this big would turn up about 3% of the time from random variation alone. That’s fairly surprising, so you have some evidence that the forms differ. A p-value of 0.40 means gaps this size are common even when nothing is different, so you have no real evidence either way.

The same observed conversion rates can produce very different p-values depending on how much data sits behind them. We calculated these with a standard chi-square test on a 2×2 table (converted vs. not, for A vs. B):

In the first row, B looks almost twice as good, yet the p-value is nowhere near 0.05: with 100 views, a four-conversion gap is easily produced by chance. In the second and third rows the rates are identical, but only the larger sample gives clear evidence. A p-value is a statement about rates and sample size together, never about rates alone.

What a p-value doesn’t tell you

The ASA statement lists six principles. Three of them come up constantly in A/B testing.

The 0.05 cutoff is a convention, not a law of nature. It’s a reasonable default because it was chosen before the test, which keeps everyone honest. What it isn’t is a switch that turns “no effect” into “effect.” A result at p = 0.049 and one at p = 0.051 carry almost the same evidence.

Why you shouldn’t stop a test early

Here is the most common way form tests produce false winners. You start a test, check the dashboard every morning, and on day four B is ahead with p = 0.04. You stop the test and ship B. It feels rigorous. It isn’t.

A 5% significance level means a 5% false-alarm rate for one look at the data at the planned end. Each extra look is another chance for random noise to cross the line, and once it crosses, you stop, so the noise never gets the chance to settle back. This was described for clinical trials by Klim McPherson in a 1974 note in the New England Journal of Medicine. As the James Lind Library summarizes it, testing accumulating data ten times without adjustment raises the nominal 5% false-positive rate to nearly 20%. In other words, a team that peeks daily for ten days and stops at the first p < 0.05 is wrong about one time in five when the forms are actually identical.

Stopping early also exaggerates the size of real effects. The days when a lucky streak pushes B ahead are exactly the days when you’re most tempted to stop, so early stopping tends to lock in the high point rather than the true average.

Three honest ways to handle peeking

1. Fixed horizon (simplest). Compute the sample size in advance, look at the dashboard if you must, but make the decision only when each version reaches its planned number of views. This is the approach the sample-size tables above assume.
2. A stricter threshold for interim looks. If you need to check partway through, use a much smaller cutoff at each look. In McPherson’s ten-look scenario, a threshold of 1.07% at each look kept the overall false-positive rate at 5%. The cost is that you need a clearer difference to stop early.
3. Formal sequential designs. Clinical trials use group sequential methods that spread the error budget across a small number of planned looks. They work well, but they need to be set up before the test begins, with the number and timing of looks decided in advance, as McPherson also emphasized.

One legitimate reason to stop early has nothing to do with statistics: B is broken. If the new version has a bug, a missing required field or a failing submit button, stop, fix and restart from zero. Don’t carry the data over.

Testing more than two versions

It’s tempting to test A against B, C and D at once. Every extra version, though, adds another comparison, and each comparison carries its own 5% false-alarm rate. NIST’s handbook warns that repeating pairwise comparisons over and over doesn’t work in general, because the overall significance level is no longer what was specified for a single pair.

The arithmetic is simple enough to do by hand. If each comparison against A has a 5% chance of a false alarm, and you treat the comparisons as roughly independent, the chance that at least one of k challengers “wins” by luck alone is about 1 − 0.95ᵏ.

The Bonferroni method, one of the approaches NIST lists for multiple comparisons, is the simplest fix: divide your significance level by the number of comparisons and require each challenger to clear that stricter bar. It’s conservative, but easy to apply. There’s also a traffic cost: every extra version thins the traffic each one gets, so a four-way test takes far longer than a two-way test. For most small sites, two versions at a time is the practical limit.

Small samples: when the usual test isn’t reliable

The chi-square and z-tests for two proportions rely on a normal approximation that works when samples are reasonably large. The STAT 509 notes give a rule of thumb: the approximation is useful when the expected number in each cell of the 2×2 table (conversions and non-conversions for each version) is greater than 5. Otherwise, Fisher’s exact test is recommended, and NIST describes it as an excellent choice for small samples.

In practice, if one version has only a handful of conversions, any p-value you see is on shaky ground. With very low counts, the right move is usually to keep the test running rather than to switch to a different test and hope for a better number.

Other ways form tests go wrong

Making the decision

When the planned sample is reached, there are only a few possible outcomes. Deciding in advance what you’ll do in each case removes most of the temptation to bend the rules.

Whatever happens, record the hypothesis, the dates, the sample sizes, the conversion rates and the p-value. After a few tests, that log tells you more about your visitors than any single winner.

A/B testing forms in marketingLABO

marketingLABO’s form builder includes A/B testing for forms. When you start a test, your current form becomes variant A and a copy becomes variant B, and traffic is split 50:50. You edit B like any other form, so it’s easy to keep the change to one thing.

The p-value comes from a chi-square test on each variant’s conversions and non-conversions compared with A’s, the same kind of 2×2 test described above. The results screen reminds you to wait for at least 100 views per variant before deciding. Treat that as a floor, not a target: use the sample-size table in this guide to set the real number for your form, and if you add variants C or D, apply the stricter threshold from the multiple-comparisons section rather than relying on the star alone.

Forms are on every plan, with one published form on Free, three on Light and unlimited forms on Standard; see plans and pricing. To choose what to test in the first place, read our guide to form abandonment, which shows how to find the field where people give up. If submissions are too rare to test on, our guide to micro-conversions covers smaller signals you can track along the way.

FAQ

How long should a form A/B test run?

Until each version reaches the sample size you calculated, rounded up to whole weeks. A test that hits its numbers in four days should still run for at least one full week so every day of the week is represented.

Is 100 views per variant enough?

Only for very large differences. In our table, 100 views per version can’t reliably detect even a jump from 5% to 10%, which needs about 436. Use 100 as the point where you start paying attention, not where you decide.

Should I use a one-sided or two-sided test?

Two-sided, unless you genuinely don’t care whether B could be worse. Form changes can and do make things worse, and a two-sided test detects that. The sample sizes in this guide assume a two-sided test at 0.05.

What if the result is p = 0.06?

Don’t extend the test just to push it under 0.05; that’s another form of peeking. Report it as weak evidence. If the change is cheap and B is simpler, shipping it can be reasonable. If it’s costly, plan a larger follow-up test.

Can I test the form and the page around it at the same time?

Not in the same test. If the headline above the form and the form itself both change, you can’t tell which one made the difference. Test them one after the other.

Ready to test your own form? Sign up free and build a form with drop-off tracking, then run your first test once you know which field to change.

Sources

1. NIST/SEMATECH e-Handbook of Statistical Methods, 7.3.3: How can we determine whether two processes produce the same proportion of defectives?
2. NIST/SEMATECH e-Handbook, 7.2.4.2: Sample sizes required
3. NIST/SEMATECH e-Handbook, 7.4.7: How can we make multiple comparisons?
4. Penn State STAT 509, Lesson 8: Sample Size and Power
5. American Statistical Association: Statement on Statistical Significance and P-Values (2016)
6. James Lind Library: Long overdue recognition of Klim McPherson’s 1974 article on sequential analysis of trial data