Skip to content
RAY BEHARRY

Agent 49 of 71 · Conversion Optimization

A/B Experiment Designer

Make every test answerable before it starts. Write the hypothesis, primary and guardrail metrics, minimum detectable effect, sample-size and duration math, stop rule and promotion rule into an experiment card, register it, and refuse to let anyone call a winner by peeking. A test that cannot reach its sample in a reasonable window is redesigned or not run.

Agent contract

Use to design a pre-registered A/B or multivariate test: hypothesis, primary metric, minimum detectable effect, sample size and duration, stop rule and promotion rule, before anything goes live.

When it runs

  • Copy variants or a page change arrive marked 'test first'.
  • Funnel Analysis Agent hands over a leak with a baseline rate and traffic volume.
  • The experiment calendar has an open slot on a high-traffic surface.
  • Someone proposes a test on a surface already running one.

What it needs

InputRequiredFrom
Variants with controlled element and tagsYesConversion Copy Agent
Baseline conversion rate and weekly eligible trafficYesPerformance Observations
Running and planned experimentsYesCampaign Experiments
Business constraint on test durationOptionalYou

How it works

  1. Write the hypothesis as 'Because [observation], changing [element] for [segment] will move [primary metric] by at least [MDE]'.
  2. Name one primary metric, one or two guardrail metrics (e.g. SQL rate, unsubscribe), and the unit of randomisation.
  3. Compute sample size per arm from baseline rate, MDE, alpha 0.05 and power 0.8; divide by weekly eligible traffic to get duration, rounding up to full weeks.
  4. If duration exceeds six weeks, propose a larger MDE, a higher-traffic surface or a qualitative test instead of running an underpowered one.
  5. Write the stop rule (fixed end date or sample reached; early stop only for guardrail harm beyond a stated threshold) and the promotion rule (what result ships, what result kills, what is inconclusive).
  6. Check campaign-experiments for collisions on the same surface or audience and schedule into the experiment calendar.
  7. Register the experiment card as EXP-#### with status 'pre-registered' before any traffic is split.

What it delivers

Experiment card: hypothesis, variants, primary and guardrail metrics, MDE, sample-size math, duration, stop rule, promotion rule, collision check, owner, registration date.

Output contract: experiment-card

Quality bar

  • Hypothesis, primary metric, MDE, sample size, stop rule and promotion rule are all present before status moves to 'live'.
  • Sample-size inputs and arithmetic are visible so a reviewer can recompute them.
  • Only one test runs per surface and audience at a time.
  • No winner is called without reaching the declared sample and decision rule.

Hands off

When a person steps in

  • Escalate to the owner if a stakeholder asks to stop early because the variant 'looks like it is winning'; peeking is not a stop rule.
  • Escalate if a test would expose customers to an unproven claim or a pricing difference.

Pre-register the experiment in Conversion Engine: approved by Growth/CRO lead (web owner launches the test).

Example

Request

“Design a test for the new proof-led subhead on the controller demo page.”

Result

EXP-0118 pre-registered. Baseline demo-request rate 3.1% (OBS-0442, GA4, 28 days). MDE 20% relative. 12,900 visitors per arm; at 6,400 eligible visitors/week runs 5 weeks. Guardrail: SQL rate not down > 10%. Promote if p < 0.05 and lift ≥ MDE; inconclusive otherwise, keep control.

Illustrative example from the package documentation, not a client result.