How to Set Up Product Experimentation in Statsig
Every product decision is a hypothesis — 'we believe changing the onboarding flow will increase activation by 15%.' Without experimentation, you ship the change, activation may or may not improve, and you never know whether the change caused the improvement or something else did. A/B testing gives you the causal evidence that turns opinions into knowledge.
Statsig is purpose-built for product experimentation: it handles feature gates, experiment assignment, metric tracking, and statistical analysis in one platform. This guide walks through the full setup — from instrumenting your codebase with the Statsig SDK to defining metrics, configuring experiments with proper controls, and reading results with the statistical rigor that prevents you from calling winners too early.
Step-by-step guide
Install and configure the Statsig SDK
Add the Statsig SDK to your application (client-side for UI experiments, server-side for backend experiments). Initialize it with your Statsig API key and a user object that includes a stable user ID and any targeting properties (plan type, signup date, country). The SDK handles assignment, event logging, and exposure tracking automatically once initialized.
- Install the SDK: npm install statsig-js for client or statsig-node for server
- Initialize with your API key on app startup
- Pass a user object with stable_id, user_id, and custom properties for targeting
Create feature gates for controlled rollouts
Before running experiments, set up feature gates — boolean flags that control who sees a feature. In the Statsig console, create a gate with targeting rules: percentage-based (roll out to 10% of users), property-based (only Enterprise plan users), or list-based (specific user IDs for internal testing). Feature gates let you test code in production without exposing it to everyone.
- Go to Feature Gates > Create New Gate
- Define targeting rules: start with 0% rollout for internal testing
- Wrap the feature in your code with 'if (statsig.checkGate(user, "new_onboarding"))'
- Gradually increase the rollout percentage as confidence grows
Define your experiment metrics in Statsig
Before creating an experiment, define the metrics it will measure. In Statsig's Metrics section, create your primary metric (the one you are trying to move, e.g., 'Onboarding Completion Rate'), a guardrail metric (something that should not get worse, e.g., 'Error Rate'), and a secondary metric (something you hope improves as a side effect, e.g., 'Time to First Action').
- Go to Metrics > Create Metric
- Define the primary metric as an event-based ratio or count
- Create guardrail metrics for metrics that must not degrade
- Instrument your application to log the relevant events via the Statsig SDK
Create and configure the experiment
In the Statsig console, create a new experiment. Define the control group (current experience) and one or more treatment groups (the variants you are testing). Set the traffic allocation — typically 50/50 for a two-variant test. Configure the experiment to use your defined metrics and set the statistical significance threshold (usually 95% confidence). Do not start the experiment yet.
- Go to Experiments > Create New Experiment
- Define control and treatment groups with clear descriptions
- Set traffic allocation (50/50 for most tests, or lower for risky changes)
- Assign your primary, guardrail, and secondary metrics
Implement the experiment in code
In your application code, use the Statsig SDK to check which variant the user is in: 'statsig.getExperiment(user, "onboarding_v2")'. The returned object includes the variant name and any parameters you configured (button text, layout, colors). Implement the control and treatment experiences based on the variant. The SDK handles consistent assignment — the same user always sees the same variant.
- Use getExperiment() to retrieve the user's assignment and parameters
- Implement both control and treatment experiences in your code
- Verify with internal test users that both variants render correctly
- Check the Statsig console to confirm exposure events are being logged
Launch, monitor, and call the experiment
Start the experiment in the Statsig console. Monitor the results page daily for the first week to catch implementation bugs (uneven traffic split, missing metric events). Statsig provides a real-time results dashboard with confidence intervals for each metric. Wait until the experiment reaches statistical significance on the primary metric before calling a winner — this typically takes one to four weeks depending on traffic volume.
- Click 'Start' on the experiment in the console
- Check the diagnostics tab daily for the first week: verify traffic split and exposure counts
- Do not call the experiment until the primary metric reaches your significance threshold (typically p < 0.05)
- When called, either ship the winning variant to 100% or roll back to control
Common mistakes
Peeking at results and calling winners too early
Checking experiment results daily and stopping as soon as one variant looks better is the single most common experimentation error. Statistical significance requires a minimum sample size. Calling a winner at 80% confidence means you have a 20% chance of being wrong. Wait for the predetermined significance threshold.
Running experiments without guardrail metrics
An experiment that improves activation by 10% but increases error rate by 50% is a net negative. Always include guardrail metrics — things that must not get worse. If a guardrail degrades significantly, stop the experiment regardless of the primary metric.
Testing too many variants at once
Each additional variant divides your traffic further and increases the time to reach significance. Start with two variants (control and one treatment). If you need to test four variants, you need four times the traffic to reach significance in the same timeframe — which most growth-stage products do not have.
Not documenting experiment learnings
An experiment that does not change any metric is still valuable — it tells you what does not work. Document every experiment result (positive, negative, and neutral) in a shared experiment log. Without this record, teams repeat failed experiments and lose the compounding value of experimentation.
Tips
Use Statsig's 'Layer' feature to run multiple experiments on the same surface without interaction effects — it ensures each user is in at most one experiment per surface area.
Set up a minimum detectable effect (MDE) calculation before launching: if you need a 2% lift to be meaningful but your sample size can only detect a 10% lift, do not run the experiment.
Create an experiment backlog with hypotheses, expected impact, and effort estimates — prioritize experiments by expected learning value, not just potential metric lift.
Use Statsig's 'Autotune' for optimization experiments where you want to automatically allocate more traffic to the winning variant over time.
How Vantage helps
Vantage helps PMs define experiment hypotheses during PRD generation. When a requirement involves user-facing changes, Vantage suggests experiment configurations — primary metrics, guardrails, and sample size estimates — so the engineering ticket includes experimentation setup alongside the feature implementation, preventing the common pattern of shipping without measuring.