Your Next A/B Test Might Be Measuring Robots

For years, CRO has operated on a premise almost no one questioned: that anyone visiting a website is a person. Conversion rates are calculated, A/B test traffic is distributed, and winning versions are decided based on this premise.

This premise no longer fully holds true. According to Imperva's annual automated traffic report, in 2024, bot-generated traffic surpassed human traffic for the first time in a decade, accounting for 53% of the total. In addition to traditional bots (crawlers, scrapers, monitoring tools), a new category has emerged in the last two years: AI agents that browse, compare, and even make purchases on behalf of a user.

For a CRO team, this raises two questions. The first is technical: how much of what I'm measuring are actual people? The second is strategic: if a portion of my visitors aren't human, who am I optimizing for?

From Crawling Bots to Purchasing Agents

Bots are not new. What's new is their behavior. A classic crawler would enter, read the HTML, and leave: it wouldn't click, wouldn't execute most scripts, and was easy to identify. Analytics tools learned to filter them out long ago. GA4, for instance, automatically excludes known bot traffic.

AI agents operate differently. From 2025, assistants like ChatGPT, through their agent mode, and AI-powered browsers like Perplexity's Comet, can open a website in a real browser, scroll, fill out forms, compare products, and add items to the cart. To your analytics tool, such a session can closely resemble that of a person in a hurry: they enter, navigate with a clear objective, and leave.

Some agents identify themselves, while others do not. And while the volume of purchases completed by agents is still small, the volume of visits they make to research, compare prices, or summarize products is growing much faster.

The First Problem: Your Data

An A/B test operates under a simple premise: traffic is randomly distributed between versions, and both receive the same type of visitor. If a portion of that traffic is automated, this assumption can break down in three ways.

Automated Traffic Dilutes Conversion

If 10% of a test's sessions are bots that never purchase, the conversion rate for both versions decreases. This doesn't necessarily change which version wins, but it does make the differences smaller and requires the test to run longer to reach a conclusion.

Automated Traffic Isn't Always Distributed Equally

A bot that always enters without cookies might receive a different version on each visit. One that reuses the same session might always remain on the same version. If one version ends up with more automated traffic than the other, the comparison ceases to be fair.

This is a known problem in experimentation called sample ratio mismatch: the actual distribution of users between versions does not match the expected one. Leading testing tools, including Mida, offer calculators to detect it.

An Agent Doesn't Behave Like a User

The color of a button or a better photo doesn't influence it. If it navigates identically across both versions, it adds noise. If it interacts differently with one of them, for example, because a structural change makes it easier to find the price, it can skew the result towards one version for reasons entirely unrelated to human behavior.

How to Validate Your A/B Tests Before Declaring a Winner

There is no perfect filter, but there are practices that significantly reduce the risk.

• Check traffic distribution in every test. If one version receives significantly more or fewer users than it should, something is interfering. Before interpreting the result, you need to understand why.

• Cross-reference your testing tool with analytics. In Boost's projects, this is standard practice: the result from the testing tool is validated with GA4 before making a decision. When two sources that filter traffic differently align, the conclusion is more robust. When they don't, the discrepancy usually indicates something.

• Look for non-human patterns. Sessions with near-zero duration but many page views, visits from data centers instead of residential networks, traffic spikes at illogical hours for your market, or identical user journeys repeated hundreds of times. No single indicator is conclusive on its own, but together they allow you to build segments of suspicious traffic and exclude them from analysis.

• Choose metrics that a bot cannot complete. Anyone can make a click. An approved payment, however, cannot. The closer the primary test metric is to an actual sale, the less it is affected by automated traffic.

The Second Problem: Who Are You Optimizing For?

Here, the question ceases to be technical. If a growing portion of purchases begins with an AI assistant, your website has two audiences with very different needs.

A person is persuaded by photos, brand tone, reviews, and a sense of trust. An agent, on the other hand, reads the page literally. It needs to unambiguously find the final price, availability, delivery times, shipping costs, and return conditions.

If this information is scattered in dropdowns, embedded in images, or expressed with ambiguous phrases, the agent may not find it, misinterpret it, or discard your product in favor of one that is easier to read.

This has practical consequences for any ecommerce business.

Structured Data Is No Longer Just for SEO

The product and offer markup (price, stock, ratings) that previously served to improve visibility on Google is also what allows an agent to understand your catalog without interpreting the design.

Clarity Outweighs Persuasion

A message like "fast shipping" persuades a person. An agent needs "delivery in 24-48 business hours." The latter also works better with people, as many tests have shown, but with agents, it ceases to be optional.

The Product Feed Becomes Strategic Again

The catalogs that already feed Google Shopping or marketplaces are the foundation for agent-based purchasing programs being launched by major assistants. An incomplete or outdated feed means an invisible product.

What We Still Don't Know

It would be easy to turn this into a list of predictions. We won't do that. Today, we don't know what portion of purchases will eventually go through agents, nor at what speed, nor if users will trust them for high-value purchases. There are also no established methodologies yet for testing "for agents."

What we do know is what any CRO program can already do: ensure that the data used for decision-making reflects real people, and review whether key information on their website is understandable without needing to interpret the design.

The first protects the decisions you make today. The second prepares you for a channel that is still small but can grow very quickly.

Where to Start

Review the last tests you've declared winners. Check if the distribution of users between versions was as expected and if the result aligns with your analytics data.

Next, choose your best-selling product page and ask yourself if someone who only read the text, without seeing images or design, could know how much it costs, when it arrives, and how to return it.

If you want us to review the data quality of your tests or how your website is read, start with a digital analytics audit or a free Scan&Boost diagnosis.