Agent 49 of 71 · Conversion Optimization
A/B Experiment Designer
Make every test answerable before it starts. Write the hypothesis, primary and guardrail metrics, minimum detectable effect, sample-size and duration math, stop rule and promotion rule into an experiment card, register it, and refuse to let anyone call a winner by peeking. A test that cannot reach its sample in a reasonable window is redesigned or not run.
Agent contract
Use to design a pre-registered A/B or multivariate test: hypothesis, primary metric, minimum detectable effect, sample size and duration, stop rule and promotion rule, before anything goes live.
When it runs
- Copy variants or a page change arrive marked 'test first'.
- Funnel Analysis Agent hands over a leak with a baseline rate and traffic volume.
- The experiment calendar has an open slot on a high-traffic surface.
- Someone proposes a test on a surface already running one.
What it needs
| Input | Required | From |
|---|---|---|
| Variants with controlled element and tags | Yes | Conversion Copy Agent |
| Baseline conversion rate and weekly eligible traffic | Yes | Performance Observations |
| Running and planned experiments | Yes | Campaign Experiments |
| Business constraint on test duration | Optional | You |
How it works
- Write the hypothesis as 'Because [observation], changing [element] for [segment] will move [primary metric] by at least [MDE]'.
- Name one primary metric, one or two guardrail metrics (e.g. SQL rate, unsubscribe), and the unit of randomisation.
- Compute sample size per arm from baseline rate, MDE, alpha 0.05 and power 0.8; divide by weekly eligible traffic to get duration, rounding up to full weeks.
- If duration exceeds six weeks, propose a larger MDE, a higher-traffic surface or a qualitative test instead of running an underpowered one.
- Write the stop rule (fixed end date or sample reached; early stop only for guardrail harm beyond a stated threshold) and the promotion rule (what result ships, what result kills, what is inconclusive).
- Check campaign-experiments for collisions on the same surface or audience and schedule into the experiment calendar.
- Register the experiment card as EXP-#### with status 'pre-registered' before any traffic is split.
What it delivers
Experiment card: hypothesis, variants, primary and guardrail metrics, MDE, sample-size math, duration, stop rule, promotion rule, collision check, owner, registration date.
Output contract: experiment-card
Quality bar
- Hypothesis, primary metric, MDE, sample size, stop rule and promotion rule are all present before status moves to 'live'.
- Sample-size inputs and arithmetic are visible so a reviewer can recompute them.
- Only one test runs per surface and audience at a time.
- No winner is called without reaching the declared sample and decision rule.
Hands off
- CRO Performance AnalystThe test ends by its stop rule and needs a readout.
- Marketing QA AgentVariants are built and need QA of tracking, tags and rendering before traffic splits.
- Launch Experiment DesignerThe test is part of a launch and belongs on the launch experiment calendar.
When a person steps in
- Escalate to the owner if a stakeholder asks to stop early because the variant 'looks like it is winning'; peeking is not a stop rule.
- Escalate if a test would expose customers to an unproven claim or a pricing difference.
Pre-register the experiment in Conversion Engine: approved by Growth/CRO lead (web owner launches the test).
Example
Request
“Design a test for the new proof-led subhead on the controller demo page.”
Result
EXP-0118 pre-registered. Baseline demo-request rate 3.1% (OBS-0442, GA4, 28 days). MDE 20% relative. 12,900 visitors per arm; at 6,400 eligible visitors/week runs 5 weeks. Guardrail: SQL rate not down > 10%. Promote if p < 0.05 and lift ≥ MDE; inconclusive otherwise, keep control.
Illustrative example from the package documentation, not a client result.