A/B testing your popups

A/B testing lets you compare two versions of a popup against each other on real visitors, so you can see which copy, design, or layout actually performs better before you commit to it. SupaPop splits your traffic, tracks each version separately, and tells you how likely each version is to beat your original. When you find a winner, one click copies its changes into your live popup.

Experiments run on a single popup at a time. Each test has a control (your baseline) and one variant you want to try against it.

Before you start

You need a popup that is already built and running.
Decide what you want to measure. You pick one primary metric per test, and the winner math runs only on that metric.
Variants are render-only. They can change copy, sections, content, and layout, but they cannot change your routing rules or your Klaviyo destinations. Those settings always come from the live popup.

How to create an experiment

Open the popup you want to test in the editor.
In the editor toolbar (the row with Sections, Container, and Tracking), click A/B Tests. This opens the experiments panel. If you see “No experiments yet.” you have a clean slate.
Click Create experiment. The New experiment form appears.
Give it a Name (this is required), then choose a Primary metric. Your options are Opt-in rate, Conversion rate, Completion rate, Revenue per session, Revenue per impression, and Revenue per submission.
Click Create. Your experiment is created as a draft.

How to add and edit variants

In the draft experiment card, click Add variant. The first variant you add is automatically named “Control”, marked as your baseline (shown with a star), and set as the version your other variant is measured against.
Click Add variant again to create the version you want to test. The second variant is named “Variant A” (a third would be “Variant B”, and so on).
To change a variant, click Edit visually next to it. This opens the same full editor you use for the live popup, but your changes only affect that variant. Your live popup is untouched. Note: Edit visually and Preview appear on the variants you are testing, not on Control. Control is your untouched baseline, so there is nothing to edit on it.
Want to check your work? Click ↗ Preview to open that variant in a new tab.
Each variant carries a traffic weight, and all weights must sum to exactly 100 before you can start. Every variant is created at a weight of 50, and weights are not editable in the panel. That means a two-variant test (Control plus one variant) lands at 50/50 and starts cleanly. If you add a third or more variant, the weights sum past 100 and you cannot start, and there is no way to rebalance them in the app. If Start is blocked for this reason, delete the extra variants and keep the test to two.

How to start the test

Click Start in the draft card. To start, your experiment must pass every gate:

At least 2 variants.
Exactly one variant marked as control.
Every weight greater than zero.
All weights summing to exactly 100.
A primary metric set.

If any gate fails, you will see a specific message telling you what to fix, and the experiment stays a draft. Once it starts, the card shows “Running. Check the Analytics dashboard for live results.”

While a test runs, editing the live popup is locked. You will see a banner explaining that you can edit the experiment’s variants, but to change the live popup you must stop the test first.

How to read results

Open the Analytics dashboard and select the popup being tested. You will see an Experiment: {name} section. This section only appears while a test is running.

The block shows:

Days running against your minimum runtime, with a checkmark when met.
Min sample, the smallest variant’s visitor count against your minimum, with a checkmark when met.
Your primary metric.

The results table shows each variant with its visitor count, purchases, revenue, the primary metric, the vs control difference, P(beats control), and a 95% range. P(beats control) is the probability the difference is real given the data so far. It is an observational estimate, not a guarantee or a causal claim.

Below the headline table you also get side-by-side comparisons for things like the step funnel, time on each step, and drop-off hotspots, each measured against your control.

How to promote a winner

While the test runs (or after it concludes without a winner), find the Promote winner section in the A/B Tests panel.
Click Promote {name} for the version you want to keep.
A dialog warns that this copies the variant’s changes into your live popup and concludes the experiment. You can add an optional reason, which is useful if you are promoting early before the gates are met.
Confirm. The winning version’s changes become your live popup, and the experiment closes.

If you would rather just end the test without picking a winner, use Stop experiment. You can add a short note about what you learned. Visitors stop being assigned to variants and revert to your live popup as it was.

Tips

Stopping is not promoting. If you stop a test, your live popup stays exactly as it was. A winning variant’s changes are lost unless you explicitly promote it.
Read your numbers before you conclude. The Experiment section disappears from the dashboard once the test is no longer running. There is no after-the-fact results view, so capture the figures you care about while it is live.
Two variants is the supported setup. Control plus one variant defaults to 50/50, which is the only weight split the panel can start without manual rebalancing (which the app does not offer). If you added extra variants and Start is blocked, delete them down to two.
Keep your control. The first variant is your control automatically. Don’t delete it. Without a control, Start fails.
Routing and Klaviyo cannot be tested. Those settings are stripped from variants, and promoting a winner never changes them.
For revenue tests, run longer. Purchases can lag behind impressions because of order reporting delay. A longer runtime captures more of the purchase tail than the 7-day default.
Variants are a snapshot. A variant starts from the live popup at the time you first edit it. Later changes to the live popup do not flow into running variants.

FAQ

Can I run two tests on the same popup at once? No. Only one experiment can run per popup at a time. If one is already running and you try to start another, you will see a message telling you to refresh to view it.

Will a returning visitor see a different version each visit? No. Each visitor is locked to the same variant for the whole test, so their experience stays consistent.

What happens to a concluded test? It stays visible in the A/B Tests panel showing “Concluded {date}” and, if you promoted one, “Winner promoted.” You can use Duplicate as draft to start a fresh iteration from it.