How to Run a Holdout Test in Statsig to Measure Cumulative Impact
Individual experiments tell you whether each change helped. They do not tell you whether everything you shipped this quarter, taken together, moved the business. Effects can overlap, novelty can fade, and small wins measured in isolation do not always add up. Holdouts answer the cumulative question.
A holdout keeps a small percentage of users out of a set of features and experiments, so you can compare them with everyone else over a longer period. Statsig supports holdouts as a first-class feature. This guide covers when to use one, how to configure it, how to read results, and how to avoid the costs of holding users back too long.
Measuring a change in Statsig? Write the experiment brief before you ship.
Describe the change and the number you want to move. Vantage turns it into a written brief you can share with eng and data before anything ships.
How this works
- We save what you typed. No AI runs on this page.
- You create a free account.
- Vantage builds an experiment brief from it inside your own workspace.
Why ask for an account? We’re a small team, and a full generation runs a multi-step AI pipeline that costs us real money. So we tie it to a free account instead of running it anonymously. You also get to keep the result and edit it.
Your text is only used to create your project, and it stays private to your account. If you don’t sign up, we delete it after 7 days. Privacy policy
Step-by-step guide
Decide what question the holdout answers
Holdouts are most useful for measuring the combined impact of a team's work over a quarter or half, or for checking whether a feature's effect persists after the novelty wears off. Write the question down: 'What is the combined effect of the growth team's Q4 launches on week-4 retention?'
- Write the question the holdout answers
- Choose the scope: a team's features, a product area, or a single long-running feature
- Pick the primary metric to compare
Choose a holdout size
The holdout group needs to be large enough to detect the effect you care about, but every user in it misses improvements. Small single-digit percentages are common. Talk to your data team about the minimum size that gives a useful read on your primary metric with your traffic.
Create the holdout in Statsig
In Statsig, create a holdout and set the percentage of users to hold out. Choose which feature gates and experiments the holdout applies to. Statsig can also apply a holdout to new gates and experiments created after it, which is useful for team-level holdouts that should cover future launches automatically.
- Create a new holdout
- Set the holdout percentage
- Select the gates and experiments it applies to
- Decide whether new gates and experiments should be included
Make sure features respect the holdout
Users in the holdout should get the default experience for every included gate and experiment. This works when features are controlled through Statsig gates and experiments. Features shipped without a gate will reach holdout users, which breaks the comparison. Make gating part of the launch checklist for teams covered by a holdout.
Read results over time
Statsig compares metrics for users in the holdout with users exposed to the features. Look at the primary metric and a few guardrails. Holdout effects often take weeks to become clear, especially for retention metrics. Resist checking daily and drawing early conclusions.
- Review the primary metric difference between holdout and exposed users
- Check guardrail metrics such as performance and support contacts
- Compare the cumulative effect to the sum of individual experiment results
- Document findings for planning
Decide when to end the holdout
Set an end date when you create the holdout, usually aligned to a planning cycle. When it ends, release all included features to the held-out users. Keeping users out of improvements indefinitely has a real cost to their experience.
Use the results in planning
If the cumulative effect is much smaller than the sum of individual wins, your experiments may be overstating impact or interacting with each other. If it is larger, your features may be compounding. Either finding should shape how you size bets in the next planning cycle.
Common mistakes
Ungated launches
Features shipped outside Statsig reach holdout users and contaminate the comparison. Gate everything covered by the holdout.
Holdouts that never end
Users in a permanent holdout miss improvements and can have a worse experience. Set an end date.
Reading results too early
Cumulative effects on retention take time to appear. Wait for enough data before making claims.
Tips
Align holdout periods to quarters so results feed directly into planning.
Exclude internal users and test accounts from holdouts.
Report holdout results alongside individual experiment wins in quarterly reviews.
Keep a list of features covered by each holdout for reference.
How Vantage helps
Holdouts measure whether a quarter's work added up. Vantage keeps the link between what you planned and what shipped: PRDs carry the success metrics, tickets trace back to requirements, and bi-directional sync with Linear shows which requirements actually reached users, which makes holdout results easier to interpret.