---
title: What Is Statistical Significance in A/B Testing? | WebManics
description: Statistical significance tells you whether an A/B test result is real or down to chance. What it means, what 95% confidence is and the mistakes to avoid.
---

[Skip to main content](https://webmanics.com/glossary/statistical-significance#main-content) [Skip to main menu](https://webmanics.com/glossary/statistical-significance#main-menu) [Skip to footer](https://webmanics.com/glossary/statistical-significance#footer-menu)

[![webmanics-logo-rgb](https://webmanics.com/hubfs/agency-website/branding/logos/webmanics-logo-rgb.svg)](https://webmanics.com/)

- [Services](https://webmanics.com/services)
- [Case Studies](https://webmanics.com/case-studies)
- [Why Us](https://webmanics.com/why-webmanics)
- [Resources](https://webmanics.com/blog)

- [Conversion-Optimised Website Build](https://webmanics.com/services/conversion-optimised-website-build)
- [AI Search Optimisation](https://webmanics.com/services/ai-search-optimisation)
- [Website Conversion Audit](https://webmanics.com/website-conversion-audit)
- [CRO Growth Retainer](https://webmanics.com/services/cro-growth-retainer)

[All services](https://webmanics.com/services)

- [Blog](https://webmanics.com/blog)
- [CRO Glossary](https://webmanics.com/glossary)
- [FAQs](https://webmanics.com/faqs)

[All resources](https://webmanics.com/blog)

[Request a consultation](https://webmanics.com/request-a-consultation)

[![webmanics-logo-rgb](https://webmanics.com/hubfs/agency-website/branding/logos/webmanics-logo-rgb.svg)](https://webmanics.com/)

- [Services](https://webmanics.com/services)

    - [Conversion-Optimised Website Build](https://webmanics.com/services/conversion-optimised-website-build)
    - [AI Search Optimisation](https://webmanics.com/services/ai-search-optimisation)
    - [Website Conversion Audit](https://webmanics.com/website-conversion-audit)
    - [CRO Growth Retainer](https://webmanics.com/services/cro-growth-retainer)
- [Case Studies](https://webmanics.com/case-studies)
- [Why Us](https://webmanics.com/why-webmanics)
- [Resources](https://webmanics.com/blog)

    - [Blog](https://webmanics.com/blog)
    - [CRO Glossary](https://webmanics.com/glossary)
    - [FAQs](https://webmanics.com/faqs)

[Request a consultation](https://webmanics.com/request-a-consultation)

1. [Home](https://webmanics.com/)
2. •
3. [Glossary](https://webmanics.com/glossary/)
4. •
5. Statistical significance

# Statistical significance

**Statistical significance** is a measure of how likely it is that a result, such as one version of a page getting more enquiries than another, reflects a real difference rather than random chance. In website testing, a result is usually called significant when there is at least 95% confidence that the difference is real.

- [Analytics and Testing](https://webmanics.com/glossary/tag/analytics-and-testing)

Updated 8 November 2026 • 3 min read

On this page

[All glossary terms A–Z](https://webmanics.com/glossary)

## Statistical significance in plain English

Flip a coin ten times and you might get seven heads. That does not mean the coin is biased. It is just chance. Website results work the same way. If version B of a page gets 12 enquiries and version A gets 9, that could be a better page, or it could be a lucky week. Statistical significance is how you tell the difference before you make a decision.

## What "95% confidence" means

Most testing tools report a confidence level, or a p-value. A 95% confidence level (a p-value of 0.05) means that if there were really no difference between the versions, you would see a result this strong less than 5% of the time. It is a high bar, which is the point: it stops you rolling out changes that only looked good by luck.

## Why it matters for CRO

In [A/B testing](https://webmanics.com/glossary/ab-testing), statistical significance is what separates a real improvement from noise. Without it, businesses often:

- roll out a "winning" change that does nothing, or even hurts enquiries
- abandon a good idea because it was behind after a few days
- argue about results instead of acting on them

## What affects significance

- **Sample size.** More visitors and more conversions give clearer answers. A handful of enquiries rarely proves anything.
- **Size of the difference.** Big improvements show up quickly. Small ones need far more data.
- **Your baseline [conversion rate](https://webmanics.com/glossary/conversion-rate).** Lower rates need more traffic to detect a change.
- **Test length.** Running for at least one or two full weeks evens out weekday and weekend differences.

## Common mistakes

1. **Peeking and stopping early.** Checking daily and stopping the moment a result looks significant inflates false positives. Decide the sample size or duration up front.
2. **Testing too many variations at once** on a site without the traffic to support them.
3. **Treating significance as size.** A result can be statistically real but too small to matter commercially.
4. **Measuring clicks instead of enquiries.** Significance on the wrong metric still leads to the wrong decision.

## What if you do not have enough traffic?

Many service businesses never get enough enquiries to reach significance quickly. That is fine. Make evidence-led changes based on [heatmaps](https://webmanics.com/glossary/heatmap), [user testing](https://webmanics.com/glossary/user-testing) and analytics, then compare results before and after over a sensible period, allowing for seasonality.

## Related terms

- [A/B testing](https://webmanics.com/glossary/ab-testing)
- [Conversion rate](https://webmanics.com/glossary/conversion-rate)
- [Conversion rate optimisation (CRO)](https://webmanics.com/glossary/conversion-rate-optimisation)

## How WebManics can help

We check whether your traffic can support A/B testing before recommending it, design tests that can reach a reliable answer, and report results in plain English. Where testing is not realistic, we use careful before-and-after measurement instead. See how this works in our [CRO Growth Retainer](https://webmanics.com/services/cro-growth-retainer), or start with a [free website conversion audit](https://webmanics.com/website-conversion-audit).

## Want more enquiries from your website?

Get a free website conversion audit. We’ll show you what’s stopping visitors from getting in touch, and what to fix first. No obligation.

[![Request a free audit](https://hubspot-no-cache-eu1-prod.s3.amazonaws.com/cta/default/143736829/interactive-87103837382.png)](https://webmanics.com/hs/cta/wi/redirect?encryptedPayload=AVxigLLppsF2XTbkicU%2FxjLWLRjIxVJbdVXOdJzkdH5ZZVzjmP8Nk5S0oLKGdIe%2BIwt870Rq%2Fy4n5mrByQAnrpWUkz0CKUGX4aUAZJ%2FbptIoJ6YM%2BlHN6UzPYuyO0ipBS831JGM54b2IcIFz85bKTPD2XDyAfCJxXikRQgNfx2f%2F3Yg7EzAEWFU7kyWWYfpzi8rtfqEjgHLgPpZ%2FYoXIPhEQ1dxYG9sDgVYO0kM5XJgJEVajamc%3D&webInteractiveContentId=87103837382&portalId=143736829)

## More definitions

[Browse all terms A–Z](https://webmanics.com/glossary)

### [Information architecture Information architecture (IA) is the way a website's content is organised, labelled and connected, so people can find wh... Read definition](https://webmanics.com/glossary/information-architecture)

### [UTM parameters UTM parameters (or UTM codes) are short tags added to the end of a link that tell analytics tools like Google Analytics ... Read definition](https://webmanics.com/glossary/utm-parameters)

### [Social proof Social proof is evidence that other people already trust, use or recommend you, such as reviews, star ratings, testimoni... Read definition](https://webmanics.com/glossary/social-proof)

## Clarity first.   Conversion next.

[Request a consultation](https://webmanics.com/request-a-consultation)

[![](https://webmanics.com/hubfs/archive/globals/webmanics-purple-icon.svg)](https://webmanics.com/)

- [Conversion-Optimised Website Build](https://webmanics.com/services/conversion-optimised-website-build)
- [AI Search Optimisation](https://webmanics.com/services/ai-search-optimisation)
- [CRO Growth Retainer](https://webmanics.com/services/cro-growth-retainer)
- [Website Conversion Audit](https://webmanics.com/website-conversion-audit)
- [Case Studies](https://webmanics.com/case-studies)
- [Why Us](https://webmanics.com/why-webmanics)
- [FAQs](https://webmanics.com/faqs)
- [Blog](https://webmanics.com/blog)

© 2026 All rights reserved, WebManics.

- [Privacy Policy](https://webmanics.com/privacy-policy)
- [Cookie Policy](https://webmanics.com/cookie-policy)

Back to top

```json
{
  "@context" : "https://schema.org",
  "@type" : "BreadcrumbList",
  "itemListElement" : [ {
    "@type" : "ListItem",
    "item" : "https://webmanics.com",
    "name" : "Home",
    "position" : 1
  }, {
    "@type" : "ListItem",
    "item" : "https://webmanics.com/glossary",
    "name" : "CRO Glossary",
    "position" : 2
  }, {
    "@type" : "ListItem",
    "item" : "https://webmanics.com/glossary/tag/analytics-and-testing",
    "name" : "Analytics and Testing",
    "position" : 3
  }, {
    "@type" : "ListItem",
    "item" : "https://webmanics.com/glossary/statistical-significance",
    "name" : "Statistical significance",
    "position" : 4
  } ]
}
```

```json
{
  "@context" : "https://schema.org",
  "@id" : "https://webmanics.com/glossary/statistical-significance#term",
  "@type" : "DefinedTerm",
  "description" : "Statistical significance is a measure of how likely it is that a result, such as one version of a page getting more enquiries than another, reflects a real difference rather than random chance. In website testing, a result is usually called significant when there is at least 95% confidence that the difference is real.",
  "inDefinedTermSet" : {
    "@type" : "DefinedTermSet",
    "name" : "Glossary",
    "url" : "https://webmanics.com/glossary"
  },
  "name" : "Statistical significance",
  "url" : "https://webmanics.com/glossary/statistical-significance"
}
```

```json
{
  "@context" : "https://schema.org",
  "@id" : "https://webmanics.com/#organization",
  "@type" : "Organization",
  "address" : {
    "@type" : "PostalAddress",
    "addressLocality" : "City",
    "addressRegion" : "State",
    "postalCode" : "Zip code"
  },
  "logo" : {
    "@type" : "ImageObject",
    "url" : "https://143736829.fs1.hubspotusercontent-eu1.net/hubfs/143736829/wm-webmanics/vectors/webmanics-logo-dark.svg"
  },
  "name" : "WebManics",
  "url" : "https://webmanics.com"
}
```

```json
{
  "@context" : "https://schema.org",
  "@type" : "Article",
  "author" : [ {
    "@type" : "Person",
    "name" : "WebManics",
    "url" : "https://webmanics.com/glossary/author/author"
  } ],
  "dateModified" : "2026-11-08T21:29:02+0000",
  "datePublished" : "2026-11-08T21:29:02+0000",
  "description" : "Statistical significance tells you whether an A/B test result is real or down to chance. What it means, what 95% confidence is and the mistakes to avoid.",
  "headline" : "What Is Statistical Significance in A/B Testing? | WebManics",
  "inLanguage" : "en",
  "mainEntityOfPage" : {
    "@id" : "https://webmanics.com/glossary/statistical-significance"
  },
  "publisher" : {
    "@type" : "Organization",
    "name" : "WebManics",
    "url" : "https://webmanics.com/glossary/statistical-significance"
  }
}
```

```json
{
  "@context" : "https://schema.org",
  "@type" : "BlogPosting",
  "author" : {
    "@type" : "Person",
    "name" : "WebManics",
    "url" : "https://webmanics.com/glossary/author/author"
  },
  "dateModified" : "2026-11-08T21:29:02.107Z",
  "datePublished" : "2026-11-08T21:29:02.000Z",
  "headline" : "What Is Statistical Significance in A/B Testing? | WebManics",
  "mainEntityOfPage" : {
    "@id" : "https://webmanics.com/glossary/statistical-significance",
    "@type" : "WebPage"
  },
  "publisher" : {
    "@type" : "Organization",
    "logo" : {
      "@type" : "ImageObject",
      "url" : "https://webmanics.com/hubfs/wm-webmanics/vectors/webmanics-logo-dark.svg"
    },
    "name" : "WebManics"
  }
}
```

```json
{
  "@context" : "https://schema.org",
  "@type" : "BreadcrumbList",
  "itemListElement" : [ {
    "@type" : "ListItem",
    "item" : "https:///",
    "name" : "Home",
    "position" : 1
  }, {
    "@type" : "ListItem",
    "item" : "https:///glossary/",
    "name" : "Glossary",
    "position" : 2
  }, {
    "@type" : "ListItem",
    "item" : "https:///glossary/statistical-significance/",
    "name" : "Statistical significance",
    "position" : 3
  } ]
}
```