Split testing assumes the visitors in each variant are comparable. Automated traffic breaks that assumption in ways that do not average out—and a test contaminated by bots does not merely produce a noisier result, it can produce a confidently wrong one.
Why bots poison split tests
The intuition that invalid traffic distributes evenly across variants, and therefore cancels out, is wrong for three reasons.
Bots convert at zero. Every automated session inflates the denominator of the variant it lands in without ever contributing to the numerator. If bot traffic is unevenly distributed—and it usually is, because rotation and caching are not random—one variant is penalised.
They compress differences. Suppose the true conversion rates are 4% and 5%. Add 40% non-converting automated traffic to both and you measure 2.4% against 3.0%. The relative difference survives, but the absolute gap narrows and your test needs far more traffic to reach significance.
They are not uniformly distributed. Crawlers frequently hit whichever URL is cached, indexed, or first in a rotation. Scrapers revisit the same variant repeatedly. This introduces systematic bias, not noise.
Filter first, then split. If your rotation happens before filtering, bot traffic has already been assigned to variants and your sample is contaminated before the experiment starts.
Filtering before you split
The sequence should be: score the visitor, decide whether they are eligible for the experiment, then assign a variant. Traffic that fails the filter either never enters the test or is tagged and excluded at analysis time.
Tagging rather than discarding is usually better. Keeping the excluded sessions—flagged, out of the primary analysis—lets you verify later that exclusions were distributed evenly and that you have not accidentally filtered out a real segment.
Choosing a rotation model
URL rotation supports several distribution strategies, and they answer different questions:
- Even split. Equal shares to each destination. The default for a clean comparison.
- Weighted split. Deliberate imbalance—90/10 to expose a risky variant to a small share while gathering data. Useful, but reaching significance on the small arm takes proportionally longer.
- Sequential rotation. Visitors cycle through destinations in order. Simple, and vulnerable to systematic bias if arrival patterns correlate with the cycle.
- Sticky assignment. A returning visitor sees the variant they saw before. Essential for any test where the conversion happens on a later visit.
Sticky assignment is the one most often skipped and most often needed. Without it, a visitor who returns twice before purchasing may see three different pages, and your attribution becomes meaningless.
Sample size, honestly
Two failure patterns account for most bad test decisions.
Stopping early. Conversion rates fluctuate wildly at low volume. A variant that is 40% ahead after two hundred visitors is telling you almost nothing. Decide the required sample before you start and do not look for reasons to stop when the answer is one you like.
Ignoring cycles. Traffic composition varies by hour and by day. A test that runs Tuesday to Thursday has sampled a specific population. Run for whole weeks where you can.
Calculate the sample size you need from your baseline rate and the smallest difference worth acting on. If the answer exceeds the traffic you can realistically gather, that is important information: it means the test cannot answer the question, and running it anyway just produces a confident-sounding coin flip.
Common mistakes
- Testing too many variants. Every additional arm divides your traffic and multiplies your chance of a spurious winner.
- Changing multiple elements at once. A variant that differs in headline, image, and price tells you something changed, not what.
- Measuring clicks instead of revenue. A variant can win on click-through and lose on order value.
- Forgetting the filter changed. If you tighten filtering mid-test, the populations before and after are different. Restart the test.
- Not verifying the split. Check actual distribution against intended distribution early. Misconfigured rotation is common and silent.
Validating that the test itself is sound
Before trusting a result, confirm the experiment behaved:
- Did each variant receive approximately its intended share of eligible traffic?
- Was the excluded (filtered) proportion similar across variants? A large discrepancy suggests the filter is interacting with the variants themselves.
- Do the variants show similar traffic composition—geography, device mix, source? Divergence means the split is not random.
- Does the result hold when you segment by device and by country, or is it driven entirely by one segment?
Takeaways
- Bot traffic biases split tests systematically; it does not average out.
- Filter before assigning variants, and tag exclusions rather than discarding them.
- Use sticky assignment whenever conversion can happen on a later visit.
- Fix your sample size in advance and run for whole weeks.
- Validate the split itself before believing the winner.
See it on your own traffic
MaxGuard scores every visit against IP intelligence, device and behavioural signals, then blocks, routes, or allows it by your rules.
Request access