A/B testing: what it is, how to do it right, and mistakes that ruin your tests

What is A/B testing and why it's the foundation of optimization

A/B testing is a controlled experimentation method in which two versions of an element (a web page, an email, an ad) are compared to determine which produces better results on a defined metric.

Version A (control) is the original. Version B (variant) includes a specific change. Traffic is randomly split between both versions and, after a data collection period, the analysis determines which generates higher conversion with statistical significance.

It's not opinion. It's not a design committee voting. It's applied science for business.

According to VWO data, companies that implement consistent A/B testing programs improve their conversion rate by 20% to 50% annually. Not with a single miracle test — but through the accumulation of dozens of tests that generate compounding learnings.

The problem: most companies do testing wrong. Tests without hypotheses, insufficient sample sizes, premature conclusions. A poorly executed test is worse than not testing at all, because it generates false confidence in wrong decisions.

A/B testing methodology step by step

Step 1: Research and analysis

Before testing, you need to know where the problems are. Use these sources:

Quantitative data: GA4 (funnel analysis, pages with highest drop-off, device, segment), heatmaps, scroll maps.
Qualitative data: session recordings (Hotjar, Clarity), on-site surveys, customer interviews, support analysis.
Benchmarking: is your conversion rate below the industry benchmark? In which segments?

Step 2: Hypothesis formulation

A testing hypothesis has this structure:

"If [we make this change] to [this element] for [this user segment], then [this metric] will improve because [this data-based reason]."

Example: "If we simplify the checkout form from 5 fields to 3 fields for mobile users, the checkout conversion rate will improve because heatmap data shows that 65% of mobile users abandon at field 4."

Characteristics of a good hypothesis:

Based on data (not intuition).
Specific (one change, one metric, one segment).
Falsifiable (can be true or false).
Actionable (if it wins, you can implement it).

Step 3: Experiment design

Define before executing:

Primary metric (primary KPI): the metric that determines whether the test wins or loses. Only one.
Secondary metrics (guardrail metrics): metrics that shouldn't worsen. Example: testing a more aggressive popup may improve leads but worsen bounce rate.
Required sample size: calculate with a sample size calculator (e.g., Evan Miller) before starting. You need to know how many visitors you need to detect the minimum detectable effect (MDE) with significance.
Estimated duration: minimum 2 full weeks (to cover weekly cycles), ideally 4 weeks.
User segment: all visitors? Mobile only? New traffic only?

Step 4: Implementation

Use a testing tool (VWO, AB Tasty, Optimizely — Google Optimize has been discontinued). The tool must:

Split traffic randomly.
Maintain consistency (a user always sees the same version).
Reliably measure the primary metric.
Integrate with your analytics tool.

Step 5: Results analysis

Don't look at results before reaching the calculated sample size. Golden rule: never stop a test early because "you can already see the trend."

For analysis:

Statistical significance: minimum 95% confidence (p-value < 0.05).
Observed effect: is the effect size relevant for the business? A statistically significant +0.1% may not be worth implementing.
Consistency across segments: is the result consistent on mobile and desktop? For new and returning traffic?

Step 6: Documentation and learning

Every test — win, lose, or inconclusive — generates a learning. Document:

Original hypothesis.
What was changed (with screenshots).
Result (with data).
Learning: what do we know now that we didn't know before?
Next step: do we iterate on this hypothesis? Move to another?

Types of tests beyond classic A/B

Test type	What it is	When to use it	Complexity
A/B test	Control vs. 1 variant	Specific changes to a single element	Low
A/B/n test	Control vs. multiple variants	Testing several versions of the same element	Medium
Multivariate test (MVT)	Combines changes across multiple elements simultaneously	High traffic, exploring interactions between elements	High
Split URL test	Redirects traffic to completely different URLs	Complete page redesigns	Medium
Personalization	Adapted experiences by segment (no control group)	When you already know what works for each segment	High

What to test: the highest-impact areas

Not all tests have the same potential. Prioritize the areas with the greatest conversion impact:

High impact

Value proposition (headline): changing the main message on a landing page can generate +20-50% improvements.
CTA (Call to Action): text, color, position, and size of the main button.
Forms: number of fields, order, real-time validation.
Pricing and offer: how you present the price, discounts, plans, free trial vs. freemium.
Social proof: type and position of testimonials, reviews, client logos.

Medium impact

Page layout: section order, content distribution.
Images and video: hero image, product video vs. static image.
Navigation: menu structure, sticky header, breadcrumbs.
Secondary copy: descriptions, benefits, FAQs.

Low impact (but cumulative)

Micro-copy: secondary button text, placeholders, error messages.
Colors and typography: aesthetic changes without functional change.
Iconography: decorative vs. functional.

For AI-powered test ideas, check out our article on A/B test ideas with artificial intelligence.

Mistakes that ruin your A/B tests

Mistake 1: Not calculating sample size before starting

If you need 10,000 visitors per variant to detect a 5% effect and your landing page gets 500 visits/week, you'll need 20 weeks per variant. If you stop the test after 2 weeks because "you can already see the result," you're reading statistical noise, not signal.

Mistake 2: Peeking (looking at results too early)

The "peeking problem" is the most dangerous mistake. Every time you look at intermediate results and decide to act, you inflate the false positive rate. A test designed for 95% confidence may have only 50% real confidence if you check every day and stop when "it looks good."

Solution: define the sample size before starting and don't look until you reach it. Or use Bayesian/sequential methods designed to allow continuous monitoring.

Mistake 3: Testing changes that are too small

Changing a button color from blue to green rarely generates a detectable effect. If your MDE (Minimum Detectable Effect) is 1% and you need 500,000 visitors to detect it, the test isn't practical.

Solution: test radical changes first (value proposition, page structure, offer). Once you find the right direction, iterate with finer changes.

Mistake 4: Not segmenting results

A test can be a winner on desktop and a loser on mobile. Or a winner for direct traffic and neutral for paid traffic. If you only look at the aggregate result, you miss crucial information.

Mistake 5: Declaring a winner without statistical significance

"Variant B has a +3% after 3 days — let's implement it." No. Without statistical significance, that +3% could be perfectly random. You need a p-value < 0.05 at minimum.

Mistake 6: Tests without a hypothesis

Testing for the sake of testing ("let's try changing this and see what happens") doesn't generate cumulative learnings. Without a hypothesis, you don't know why it won or lost, and you can't iterate intelligently.

How to prioritize tests: the ICE framework

With dozens of possible test ideas, you need a prioritization system. The ICE framework is simple and effective:

I (Impact): how much potential impact does this test have on the primary metric? (1-10)
C (Confidence): how confident are you that the change will generate an improvement, based on data? (1-10)
E (Ease): how easy is it to implement this test? (1-10)

Multiply the three values to get the ICE score. Prioritize tests with the highest score.

Example:

Test	Impact	Confidence	Ease	ICE Score
Simplify checkout from 5 to 3 fields	9	8	7	504
Add video to PDP	6	5	4	120
Change CTA button color	2	2	9	36
New hero with social proof	8	7	6	336

A/B testing tools in 2026

Enterprise tools:

VWO: comprehensive, with integrated heatmaps and advanced segmentation.
AB Tasty: strong in personalization and AI-driven testing.
Optimizely: the most robust for experimentation programs at scale.
Kameleoon: strong in full-stack and server-side testing.

Mid-market tools:

Convert: privacy-first, good value for money.
Mida: lightweight, no performance impact, growing fast.

Selection considerations:

Compatibility with your stack (GTM, GA4, CDP).
Impact on web performance (flicker, load time).
Segmentation capabilities.
Support for server-side testing.
Price vs. traffic volume.

A/B testing and CRO: building an experimentation program

An isolated test is an experiment. A testing program is a continuous learning system.

Companies with mature experimentation programs:

Run 2-4 simultaneous tests across different parts of the funnel.
Document every test in a centralized repository.
Share learnings across teams (marketing, product, UX, development).
Measure the cumulative impact of the program (not just test by test).
Iterate on winning tests to extract maximum potential.

The typical program goes through three maturity phases:

Phase 1 (0-6 months): reactive tests, based on intuition, 1-2 tests/month.
Phase 2 (6-18 months): data-driven tests, prioritization framework, 4-6 tests/month.
Phase 3 (18+ months): experimentation culture, full-stack tests, personalization based on test data, 10+ tests/month.

For more ideas on improving your conversion rate beyond testing, check out our guide on how to increase web conversion rate.

Qualitative data as a complement to A/B testing

Quantitative tests tell you what works. Qualitative data tells you why. Combine both:

Before the test: session recordings and surveys to form the hypothesis.
During the test: heatmaps segmented by variant to understand behavior.
After the test: interviews or surveys to interpret unexpected results.

Tools like eye tracking can reveal visual attention patterns that explain why one design converts better than another.

From testing to optimizing: the complete cycle

A/B testing is one tool within a broader CRO process:

Research (data + feedback) → identify problems.
Hypothesize → formulate evidence-based solutions.
Test → validate with A/B testing.
Implement → scale winners to production.
Iterate → return to step 1 with new learnings.

Testing isn't the end — it's the validation mechanism that turns hypotheses into real business improvements.

If you want to implement a professional A/B testing program or take your current program to the next level, at Boost we specialize in CRO. Start with a free diagnostic at Scan&Boost.

Adrià Vidal is the founder of Boost. +1,000 optimization actions, +47.8% average conversion uplift per client, +€7.8M in additional revenue generated.