A sample ratio mismatch (SRM) occurs when the observed share of users in experiment groups differs materially from the allocation expected by the test design. It can indicate a problem in assignment, exposure logging, filtering, or analysis. It is a diagnostic warning, not a result about which variant won.
Compare observed and expected allocation
Suppose a test is designed to send half of eligible users to each variant. After enough assignments, a large imbalance may be unlikely to arise from ordinary random variation alone. An SRM check compares the observed counts with the planned allocation using an appropriate statistical test.
Run the check on the population that was actually randomized, not only on users who completed a later conversion. Filtering by exposure or behavior can create apparent imbalance even when assignment was correct. State the inclusion rule for the test.
Investigate the assignment path
Common causes include a broken randomization function, inconsistent bucketing across pages, one variant failing to load, eligibility rules applied unevenly, or a tracking event missing for a group. Identity changes across devices and bot filtering can also affect observed counts.
Check assignment, exposure, event delivery, and analysis as separate stages. Compare logs or a trusted assignment source with analytics events. Look for imbalance by device, browser, geography, and entry path to narrow down where it begins.
Do not interpret outcomes too early
If the sample is mismatched, conversion differences may reflect who was measured rather than what the variants caused. Pause or qualify the decision until the issue is understood. Removing records after seeing results can introduce additional bias, so define any exclusion rules before analysis.
Not every small difference is an SRM. Use a planned threshold and sufficient sample to avoid reacting to random noise, and record the test allocation and check method.
Prevent future mismatches
Validate the experiment before launch with test users in each arm, verify consistent event schemas, and monitor allocation during the test. Keep a clear source of truth for assignment and exposure so downstream reports do not reconstruct group membership inconsistently.
An SRM does not tell you which version is better. It tells you to investigate whether the comparison is trustworthy.
How did this article land?
Choose one reaction. You can change it anytime.