How to Run an A/B Test in PostHog
PostHog combines product analytics, feature flags, and experiments in one tool, which makes it practical for small teams to run real A/B tests without stitching services together. An experiment in PostHog uses a feature flag to split users into variants and measures the effect on metrics you choose.
This guide covers writing a hypothesis, creating the experiment, implementing the variants in code, choosing primary and secondary metrics, checking exposure, and deciding when to ship. It assumes PostHog is already capturing events in your product.
Measuring a change in PostHog? Write the experiment brief before you ship.
Describe the change and the number you want to move. Vantage turns it into a written brief you can share with eng and data before anything ships.
How this works
- We save what you typed. No AI runs on this page.
- You create a free account.
- Vantage builds an experiment brief from it inside your own workspace.
Why ask for an account? We’re a small team, and a full generation runs a multi-step AI pipeline that costs us real money. So we tie it to a free account instead of running it anonymously. You also get to keep the result and edit it.
Your text is only used to create your project, and it stays private to your account. If you don’t sign up, we delete it after 7 days. Privacy policy
Step-by-step guide
Write a hypothesis and pick one primary metric
Start with a specific hypothesis: 'Showing a checklist on the empty dashboard will increase the share of new users who create their first project within 24 hours.' Choose a single primary metric that the change should move. Secondary metrics can watch for side effects, but the decision should rest on the primary metric.
- Write the hypothesis as change, expected effect, and audience
- Choose one primary metric tied to the hypothesis
- List secondary metrics to watch for unintended effects
Create the experiment
Open Experiments in PostHog and create a new experiment. Give it a name and a feature flag key; PostHog creates the flag that controls variant assignment. By default there is a control and a test variant, and you can add more variants if needed. Set which users are eligible using release conditions such as user properties or cohorts.
- Create a new experiment and name it after the hypothesis
- Set the feature flag key
- Configure variants and their split
- Define participant targeting if the test is for a specific audience
Implement the variants in code
In your application, check the flag value with the PostHog SDK and render the variant accordingly, for example by comparing the result of getFeatureFlag for your key with 'test'. Calling the flag records exposure, so only call it where the user would actually see the difference. Calling it on page load for a feature buried three screens deep dilutes the results.
Choose metrics and check sample size
Add the primary metric to the experiment, either as a funnel (for example from dashboard viewed to project created) or as a trend on an event. PostHog provides guidance on sample size and expected running time based on your settings and current traffic. If the estimated duration is months, consider a larger change, a broader audience, or a metric closer to the change.
- Add the primary metric as a funnel or trend
- Add secondary metrics
- Review PostHog's sample size and duration estimate
- Adjust the plan if the test would take too long to reach a result
Launch and verify exposure
Launch the experiment and check early that both variants are receiving users in the expected proportion. A large imbalance usually indicates an implementation problem, such as the flag being evaluated in different places for different users. Fix issues early rather than discovering them after two weeks.
Do not stop early on a promising result
Results fluctuate early in a test. Decide the running time up front, based on sample size and at least one full weekly cycle, and stick to it unless there is clear harm. Stopping the moment a variant looks good increases the chance of shipping a false positive.
Read results and ship the decision
When the test reaches its planned duration, review the primary metric and whether the result is statistically significant according to PostHog. Check secondary metrics for harm. Ship the winning variant by rolling out the flag, then remove the flag and dead code in a follow-up ticket. Write a short summary of what you learned, including null results.
- Review the primary metric result and significance
- Check secondary metrics for negative effects
- Roll out the winning variant
- Create a ticket to clean up the flag and losing code path
Common mistakes
Evaluating the flag too early
Exposure should happen where the user would see the change. Evaluating the flag globally counts users who never encountered the variant and weakens the result.
Too many primary metrics
With several primary metrics, one will look significant by chance. Pick one and treat others as guardrails.
Leaving flags in the code after the test
Experiment flags that outlive the experiment add complexity and risk. Schedule cleanup as part of shipping the decision.
Tips
Run tests for at least one full week to account for weekday and weekend differences.
Keep an experiment log with hypothesis, result, and decision for every test.
Use cohorts to exclude internal users from experiments.
Prefer bold changes for early tests; small tweaks need much larger samples to detect.
How Vantage helps
An experiment is only as good as the hypothesis behind it. In Vantage, PMs write the hypothesis and success metric into the PRD, generate tickets for the variant work and the flag cleanup together, and push them to Linear or Jira so the test and its follow-up are planned as one piece of work.