Experiment Design
Experiment design is the discipline of translating a hypothesis into an experimental protocol that will produce a readable result. It specifies the unit of randomization, the primary and guardrail metrics, the required sample size, the exposure duration, and the analysis plan before the experiment is launched.
- The experiment is designed to be readable. If a null result cannot be trusted, the design is wrong.
- Sample size is calculated against the minimum detectable effect that would be operationally meaningful, not against convenience.
- Instrumentation for the primary metric is verified end-to-end before allocation is opened.
- The analysis plan is pre-registered. Post-hoc metric shopping is not allowed.
Where does this stage earn its keep?
Underpowered tests, contaminated allocations, and metrics that were not instrumented for the surface being changed. The result is months of experiments that cannot be trusted and a leadership team that stops believing the data.
Terms this stage depends on.
The probability that an experiment will detect an effect of a given size if one truly exists. Standard practice: 80% or higher.
The smallest true lift the experiment is powered to detect at the chosen significance level and power.
The entity assigned to variant (user, session, account). Chosen so that spillover between variants is prevented.
What changes when this stage is done properly.
Illustrative comparison of experiment discipline.
| Metric | Gut-driven attribution | Fully-instrumented data pipeline |
|---|---|---|
| Sample size | "Let it run for two weeks." | Calculated from MDE, baseline rate, power, and traffic. |
| Randomization | Cookie-based, resets on logout. | User-based, persistent across sessions and devices. |
| Analysis plan | Written after the readout. | Pre-registered before launch. |
The operational shape of this stage.
- 01Compute required sample size against the pre-registered MDE, baseline, and power.
- 02Select the unit of randomization that avoids cross-variant contamination on the surface being changed.
- 03Verify primary and guardrail metric instrumentation with a shadow test before opening allocation.
- 04Pre-register the analysis plan, including stopping rules and interim look protocol.
Sizing an A/B test on a 10,000-lead-per-month funnel
Baseline conversion 8%. Minimum detectable effect 20% relative (8% to 9.6%). At 80% power and a 5% two-sided significance level, the test requires roughly 8,500 users per variant. On 10,000 leads per month split 50/50, that is a roughly six-week cycle. Any test scoped for a lift below 20% relative on this surface will not read out in a reasonable window and should be re-scoped or replaced.
Frequently asked about this stage.
How do you size an A/B test?
+
How do you size an A/B test?
+What is a minimum detectable effect and how do I choose one?
+
What is a minimum detectable effect and how do I choose one?
+See this stage run against your numbers.
A 30-minute Growth Audit. You leave with two or three specific findings, whether or not we ever work together.
Field notes on measurement, experimentation, and growth for health and SaaS. No fluff.
Occasional dispatches from real engagements — attribution, revenue engineering, digital-health growth. Sent when there is something worth reading, not on a drip schedule.