A/B Testing

You have a landing page that converts at 3%. You think a new headline might do better. You split incoming traffic 50/50 — half sees the old page ("A"), half sees the new ("B"). After enough data, you compare conversion rates.

If B wins, you ship B. If A wins (or there's no meaningful difference), you keep A and learn something.

In order of expected impact:

1. Headline — biggest single lever in most tests
2. Offer / pricing — especially the framing ("$49" vs "$49/mo" vs "$1.63/day")
3. Primary CTA copy — "Start free" vs "Get started" vs "Try it free"
4. Social proof — which testimonials to feature, how to display
5. Form fields — removing or reordering

What's NOT worth testing (except for bored conversion consultants):
• Button color (unless extreme contrast change)
• Exact font choice
• Word-for-word microcopy in low-visibility areas
• Minor image variations

To detect a 10% relative lift (e.g., 3% → 3.3%) with statistical significance, you need roughly 15,000–20,000 visitors per variant. For a 50% lift, you can get away with ~2,000 per variant.

Most small sites run tests with 300 visitors and declare a winner. That's not a test — that's noise. If you don't have traffic for proper tests, do user interviews and qualitative research instead. It's faster and more informative at low volume.

• Stopping tests early because one variant is "clearly winning" at 100 visitors. Wait for the statistical threshold.
• Testing too many things at once — if you change 4 elements between A and B, you won't know which one mattered.
• Ignoring segment differences — a test might be neutral overall but strongly positive for one source of traffic.
• Never shipping the winner — running tests is meaningless if you don't act on results.