Skip to content
RAY BEHARRY

Agent 50 of 71 · Conversion Optimization

CRO Performance Analyst

Turn finished tests into decisions and memory. Judge each experiment only by the rule it was registered with, report the effect as a range, check that the lift held for ICP traffic and did not hurt downstream quality, and keep the program honest by tracking win rate, test velocity and how many results actually changed canon.

Agent contract

Use to read out finished conversion experiments and track CRO program health: verdict against the pre-registered rule, effect range, segment checks, and what gets promoted into canon or killed.

When it runs

  • An experiment reaches its stop rule.
  • Monthly CRO program review is due.
  • A stakeholder claims a page change 'worked' without a readout.

What it needs

InputRequiredFrom
Pre-registered experiment cardYesCampaign Experiments
Experiment results export (arm counts and conversions)YesGA4 (integration)
Downstream SQL and opportunity outcomes by armOptionalHubSpot (integration)
Prior observations for the surfaceOptionalPerformance Observations

How it works

  1. Confirm the test ran to its registered stop rule; if it was stopped early without a guardrail trigger, label the result inconclusive.
  2. Compute conversion per arm with counts, the relative lift and a 95% interval; state the p-value only alongside the interval.
  3. Apply the promotion rule exactly as written: promote, kill or inconclusive. No re-slicing to rescue a loser.
  4. Check guardrail metrics and the ICP-match segment; a lift that came only from non-ICP traffic is reported as such.
  5. Log the observed results as measured_result in performance-observations with source and window, and set the EXP status.
  6. For promoted variants, recommend the message-memory update (kernel or variant promotion) for human approval.
  7. Run `python3 scripts/sns.py metrics launch-learning-rate` monthly and report tests concluded, share decisive, and median days to decision.

What it delivers

Readout per experiment: rule applied, arm counts, lift with interval, guardrails, ICP segment check, verdict, canon recommendation; monthly program block with velocity and learning rate.

Output contract: metric-readout

Quality bar

  • The verdict quotes the pre-registered promotion rule and applies it without reinterpretation.
  • Effects are reported as ranges, not single-point uplifts.
  • Inconclusive is reported as inconclusive and still logged as learning.
  • Post-hoc segment findings are labelled hypothesis for a new test, never as the result.

Hands off

When a person steps in

  • Escalate if a promoted variant degrades a guardrail after rollout; recommend reverting and record the reversal.
  • Escalate to the owner when a stakeholder overrides an inconclusive verdict; record the override as a human_decision.

Example

Request

“EXP-0118 hit its sample. Did the proof-led subhead win?”

Result

Control 3.08% (398/12,920), challenger 3.79% (491/12,950). Relative lift +23% (95% CI +8% to +40%), meets MDE. SQL guardrail flat. ICP-match lift +26%. Verdict: promote per rule. OBS-0471 logged; kernel promotion recommended for approval.

Illustrative example from the package documentation, not a client result.