A/B testing

Two versions of a page can look quite different after a week. One gets more sign-ups, and the other gets fewer. That gap can be real, or it can be chance. A little statistics helps you tell which.

What a test result can tell you

The US National Institute of Standards and Technology (NIST) describes a statistical test as a way of making decisions with numbers. You use it to ask whether there's enough evidence to reject a starting assumption, called the null hypothesis.

In an A/B test, the starting assumption is that neither version is better. A result that does not reject it has two readings. You may carry on as if the versions perform the same. Or, in NIST's words, "we may not yet have enough data".

Decide the sample size first

Statisticians are often asked how many measurements a sample needs. NIST's answer is that "there is no correct answer without additional information (or assumptions)". Some of those things have to be assumed in advance.

NIST lists three. The first is how much risk you accept of a false alarm, where chance makes one version look better. The second is how much risk you accept of missing a real difference. The third is how much the numbers vary from one day or visitor to the next.

Checking your own numbers

The websitepromotion.co.uk site runs a free A/B test significance calculator. You enter the visitors and conversions for each version. It works out a p-value, which shows how likely a gap that size would be if neither version were better. Its guide suggests you weigh the result against your sample size and how long the test ran.

A test the government ran

In April 2016 a guest post on the Government Digital Service blog described an A/B test. People who had just booked a driving test saw a request to join the NHS Organ Donor Register. The NHS wrote eight versions of that request. The test showed which version was most effective, though not why.

The trial was run by the Behavioural Insights Team with NHS Blood and Transplant, the Department of Health, the Government Digital Service and the DVLA. The partners published preliminary results on GOV.UK on 24 December 2013.

Cookies and consent

Many websites store small files called cookies on a visitor's device. The Information Commission (which took over from the ICO on 30 September 2026) says a site needs the visitor's consent for them, apart from five cases the law sets out. The ICO finalised new guidance on this in April 2026, after the Data (Use and Access) Act changed the rules.

One of those cases covers statistics a site gathers to improve itself. The Information Commission lists A/B testing as likely to fit it. To rely on it, the site uses the information only to improve the site and keeps it as totals that can't identify anyone. It also tells visitors clearly what it's doing, and gives them a simple, free way to object.

Design that can harm consumers

The Competition and Markets Authority (CMA) uses the term online choice architecture for the environment in which people act online. That includes how choices are presented and placed, and how screens are designed. One example the CMA gives is the order in which search results appear.

In April 2022 the CMA's Behavioural Hub published two papers on how digital design can harm competition and consumers. One of them covers 21 practices and the evidence about each.

Useful links