How to Build an Experiment Results Dashboard in Statsig
Running experiments without a clear results dashboard is like conducting a clinical trial without a patient chart — you're generating data but can't interpret it effectively. A well-built experiment results dashboard gives your team real-time visibility into how experiments are performing, when they've reached statistical significance, and whether the results justify shipping the variant to all users.
Statsig combines feature flags and experimentation in one platform, making it straightforward to set up experiments and track results. This guide shows you how to build a dashboard that goes beyond showing p-values — one that contextualizes results, tracks guardrail metrics, and supports the decision-making process from hypothesis to ship decision.
Step-by-step guide
Define Your Experiment Metrics Framework
Before building any dashboard, establish which metrics matter for your experiments. Define three categories: Primary Metrics (the thing you're trying to improve — conversion rate, engagement, revenue per user), Secondary Metrics (related metrics that provide context — session duration, feature adoption, page views), and Guardrail Metrics (metrics that must not degrade — page load time, error rate, support ticket volume). Configure these in Statsig's Metrics Catalog so they're reusable across experiments.
- Create metric definitions in Statsig with clear descriptions and the events they're computed from
- Set expected directions for each metric — 'higher is better' for conversion, 'lower is better' for error rate
Configure Your First Experiment in Statsig
Create an experiment in Statsig's console by defining the hypothesis, selecting the feature flag to test, setting the allocation percentage (start with 50/50 for maximum statistical power), and assigning your primary, secondary, and guardrail metrics. Set the experiment duration based on your minimum detectable effect — Statsig's sample size calculator tells you how many users you need. Don't end the experiment early based on interim results unless a guardrail is clearly violated.
- Use Statsig's targeting rules to limit the experiment to the relevant user segment if it shouldn't run for all users
- Set a 'do not end before' date based on the sample size calculation to prevent premature decisions
Build the Results Overview Dashboard
Create a Statsig dashboard with three main sections. The top row shows experiment metadata: name, hypothesis, start date, current allocation, and days running. The middle section shows primary metric results with the current lift, confidence interval, and statistical significance indicator. The bottom section shows secondary and guardrail metrics in a compact table format. This layout lets someone understand the experiment's status in under 30 seconds.
- Use Statsig's built-in experiment results view as the core widget — it shows lift, p-value, and confidence intervals
- Add a 'Cumulative Exposures' chart showing how many users have been exposed to each variant over time
Add Metric Deep-Dive Views
For each primary metric, add a detailed view showing: the metric trend over time for control and treatment groups (to spot divergence patterns), the cumulative lift trend (to see if the effect is stable or volatile), and a segment breakdown showing results by key user properties (plan type, platform, geography). These views help you understand not just whether the experiment works, but for whom and how consistently.
- Add a time-series chart comparing the daily metric value for control vs. treatment to spot novelty effects that fade over time
- Create segment breakdowns by at least user tenure and plan type to check if the effect varies across populations
Monitor Guardrail Metrics
Create a dedicated guardrail section on your dashboard that surfaces any metric degradation immediately. Configure alerts in Statsig that fire if a guardrail metric degrades beyond a threshold (e.g., page load time increases by more than 200ms, error rate increases by more than 0.5%). Guardrail violations should trigger a review — not an automatic kill — because transient effects can create false signals in the first few days.
- Set up Slack notifications for guardrail alerts so the experiment owner is informed immediately
- Document the 'kill criteria' for each experiment: what level of guardrail degradation requires stopping the experiment
Create a Ship Decision Framework
Add a 'Decision' section to your dashboard that guides the ship/iterate/kill decision. Include a checklist: Has the experiment run for the minimum planned duration? Has it reached the target sample size? Is the primary metric lift statistically significant (p < 0.05)? Is the lift practically significant (not just statistically)? Are all guardrail metrics within acceptable bounds? Are segment-level results consistent? Document the decision and reasoning directly on the dashboard when the experiment concludes.
- Create a decision log that records the outcome of each experiment: shipped, iterated, or killed, with the reasoning
- Archive completed experiments so the team can reference past results when designing future experiments
Common mistakes
Peeking at Results and Stopping Early
Checking experiment results daily and stopping as soon as you see significance inflates your false positive rate. Statistical significance can fluctuate early in an experiment — what looks significant on day 3 may disappear by day 10. Set a minimum runtime based on your sample size calculation and stick to it unless a guardrail is clearly violated.
Ignoring Practical Significance
A 0.1% lift in conversion that's statistically significant with a p-value of 0.03 is technically a positive result, but it may not be worth the engineering maintenance cost of the feature. Always evaluate practical significance alongside statistical significance — is the lift large enough to matter for your business?
Not Segmenting Results
An experiment that shows a positive average effect might actually be hurting one segment while helping another. Always check results by key segments (new vs. returning users, mobile vs. desktop, plan tier). A positive overall result masking a negative impact on your highest-value segment is a costly mistake.
Running Too Many Simultaneous Experiments
Overlapping experiments can interact in unexpected ways, contaminating each other's results. Track which users are in multiple experiments simultaneously, and avoid running experiments that modify the same user flows concurrently. If you must overlap, document the interaction risk and monitor for unexpected effects.
Tips
Use Statsig's 'Always On' feature gates to ensure experiment variants remain stable for individual users across sessions — inconsistent experiences invalidate results.
Start every experiment with a written hypothesis in the format 'We believe [change] will cause [metric] to [direction] because [reason]' — this forces clarity about what you're testing and why.
Review your experiment's exposure and metric events before launch to ensure data is flowing correctly — catching instrumentation errors early prevents wasted experiment runtime.
Build a team-wide experiment review cadence (weekly or biweekly) where active experiments are discussed and decisions are made collectively.
How Vantage helps
Vantage helps PMs define experiment hypotheses and success criteria within their PRDs. When a feature includes an experimentation plan, Vantage ensures that the experiment metrics, guardrails, and decision criteria are documented alongside the feature requirements so the team knows how the feature will be evaluated from day one.