How is Vantage different from ChatGPT or Claude?

ChatGPT and Claude are great for one-off generation. Vantage is a full workspace. It connects your tools, tracks dependencies across projects, catches conflicts between products, and surfaces what needs your attention every morning. It also remembers every past decision, so a new PM can query across all projects and get cited answers in seconds.

Do I have to switch away from Notion or Linear?

No. You keep all your tools. Vantage connects to them and builds intelligence on top. When you're ready, you can push tickets to Linear or Jira. But the value starts before you create anything — just by connecting your tools and seeing what the intelligence layer surfaces.

What happens on day one?

You connect your tools — GitHub, Linear, Slack — 2 minutes each. Vantage indexes your existing data overnight. By the next morning, you see your first decision queue: what's stale, what conflicts, what changed. No manual setup. No templates to fill out.

What if I only manage one project?

Vantage works for single projects, but it shines when you manage 3–5 or more. Conflict detection, cross-project memory, and the decision queue are most valuable when you have multiple projects that need to stay in sync. If you manage one project, start with the free tier and see if the cited query engine alone saves you time.

All data encrypted in transit and at rest. We never train on your data — the AI providers we use are contractually barred from it. You can disconnect any integration or delete your account at any time. Pursuing SOC 2 Type II.

How does Vantage keep documents and tickets up to date?

When context shifts, you add the new information and Vantage rebuilds every connected deliverable. Documents, prototypes, and tickets all update. You review the changes, accept or reject each one, and approve. Nothing updates silently, and every change is traceable.

How do I get started?

Sign up free at app.vantageos.tech. Connect your tools, and generate your first grounded document within minutes. No credit card required. Upgrade when you need more projects or features.

How long should I run an A/B test?

Run the test until you reach the sample size calculated by the power analysis, typically 2-4 weeks for most B2B SaaS products. Never run for less than 1 full business week cycle to account for day-of-week effects. For metrics that take time to materialize (like retention), you may need 4-8 weeks. The answer is always 'as long as the statistics require,' not 'until we see a result we like.'

What statistical significance level should I use for A/B tests?

The standard is 95% significance (p-value < 0.05) with 80% statistical power. For high-risk changes (like pricing or checkout flow), use 99% significance to reduce false positive risk. For low-risk changes (like copy tweaks), 90% significance may be acceptable to reduce required sample size. Never go below 90% for any experiment you plan to make a decision on.

What is the minimum traffic needed to run A/B tests?

It depends on the effect size you want to detect. To detect a 5% improvement in a metric with a 10% baseline rate at 95% significance, you need roughly 15,000 users per group. To detect a 20% improvement, you need about 1,000 per group. If your traffic is too low for small effect sizes, focus on testing big, bold changes that would produce large effects rather than incremental tweaks.

How-To2026-09-0411 min read

How to Build an A/B Test Analysis in Statsig

A/B testing is how product teams replace opinion-based decisions with evidence. But running experiments without statistical rigor produces results that are worse than guessing because they carry false confidence. Statsig provides the infrastructure for statistically valid experiments: automated sample size calculation, sequential testing, and metric monitoring that accounts for multiple comparisons.

The hard part of A/B testing is not the tooling but the discipline. Designing a clean experiment, choosing the right metrics, running it long enough, and interpreting results honestly all require rigor that most teams skip in the rush to ship. This guide covers the full lifecycle from hypothesis to decision, with the statistical fundamentals you need to trust your results.

Step-by-step guide

Form a testable hypothesis

Before creating an experiment in Statsig, write a clear hypothesis in the format: 'We believe that [change] will cause [metric] to [improve/decrease] by [estimated magnitude] because [reasoning].' For example: 'We believe that reducing the signup form from 5 fields to 3 fields will increase signup completion rate by 15% because field fatigue data shows abandonment peaks at field 4.' The hypothesis should be specific enough to be proven wrong.

Write the hypothesis in 'We believe...will cause...because' format
Identify the primary metric the experiment will measure
Estimate the minimum detectable effect size

Create the experiment in Statsig

In Statsig, navigate to Experiments and click 'Create Experiment.' Name it descriptively and add the hypothesis in the description. Set up the control group (current experience) and one treatment group (the change). Resist the urge to test multiple variants simultaneously unless you have extremely high traffic. Multi-variant tests require proportionally more users and time to reach statistical significance.

Configure metrics and guardrails

Add your primary metric (the one your hypothesis predicts will change), 2-3 secondary metrics (related metrics you expect to move in the same direction), and 2-3 guardrail metrics (metrics that should not degrade, like page load time, error rate, or revenue). Statsig will monitor all metrics and alert you if guardrails are breached. The guardrails protect against shipping a change that improves one metric at the expense of something more important.

Set the primary metric tied to your hypothesis
Add secondary metrics for supporting evidence
Configure guardrail metrics with degradation thresholds

Calculate sample size and run duration

Use Statsig's power calculator to determine how long the experiment needs to run. Input your current metric baseline, the minimum effect size you want to detect (your hypothesis estimate), your daily traffic volume, and your desired statistical significance (typically 95%) and power (typically 80%). Statsig will calculate the required sample size per group and estimated days to reach it. Commit to this duration before launching. Do not peek and stop early.

Launch and monitor the experiment

Launch the experiment and resist the urge to check results daily. Early data is noisy and will show false signals in both directions. Instead, check guardrail metrics to ensure the experiment is not causing harm. Statsig's sequential testing accounts for multiple looks at the data, but the fewer times you check, the cleaner your interpretation will be. Set a calendar reminder for the expected end date.

Launch the experiment to the calculated percentage of users
Monitor guardrail metrics for unexpected degradation
Set a calendar reminder for the planned analysis date

Analyze results with statistical rigor

When the experiment reaches the planned duration, open Statsig's results page. Check three things: Is the primary metric statistically significant (p-value below 0.05)? Is the effect size practically meaningful (a 0.1% improvement might be statistically significant but not worth the code complexity)? Are guardrail metrics clean (no statistically significant degradation)? Document your interpretation including confidence intervals, not just the point estimate.

Make and document the decision

Based on the results, make one of three decisions: ship (primary metric improved, guardrails clean), iterate (results inconclusive or effect too small), or revert (guardrail degradation or negative primary result). Document the decision in Statsig's experiment notes with the rationale, results summary, and any follow-up experiments planned. This decision log becomes a knowledge base of what works and what does not for your product.

Common mistakes

Stopping the experiment early based on early results

Early results in A/B tests are unreliable. A test showing a 20% lift after 2 days might show no effect after 2 weeks once the novelty effect wears off and the sample size is adequate. Commit to the planned duration and only stop early if guardrail metrics show severe degradation.

Testing too many things at once

Changing the button color, copy, placement, and size simultaneously makes it impossible to attribute results to any single change. Test one variable at a time. If you need to test a holistic redesign, that is fine, but understand that the result tells you whether the package is better, not which element drove the improvement.

Ignoring the novelty effect

Users often engage more with any change simply because it is new. This inflates short-term metrics. Run experiments for at least 2 full weeks to let the novelty effect wear off. For subscription products, run even longer to capture the impact on retention, not just initial engagement.

Cherry-picking metrics after seeing results

If you measure 20 metrics, one will likely show a statistically significant result by chance alone. Define your primary metric before launching and base your decision on it. Secondary metrics provide context but should not override a non-significant primary metric.

Tips

Use Statsig's experiment layers to ensure users in one experiment are excluded from conflicting experiments, preventing interaction effects that invalidate both results.

Create a standardized experiment brief template that includes hypothesis, primary metric, sample size calculation, and planned duration. Require it before any experiment launches.

After shipping a winning variant, run a holdback test (keep 5% of users on the old experience for 4 weeks) to confirm the long-term impact matches the experiment results.

Build a team experiment review meeting where completed experiments are presented with results, interpretation, and learnings. This shared knowledge prevents teams from re-running experiments others have already done.

How Vantage helps

Vantage helps PMs plan experiments as part of their feature development process. When writing a PRD in Vantage, you can define the A/B test strategy as a requirement, and the generated tickets will include experiment setup, metric configuration, and analysis tasks alongside the feature implementation work.