Documentation

Requirements

Checkoutlab changes the payment step of the Shopify checkout and assigns visitors from your storefront, so it needs both of those to exist:

• The Online Store sales channel. Assignment happens in a theme app embed, which only exists in a Shopify theme. A headless storefront cannot be assigned by this app.
• Shopify Plus, only to hide the express row. That uses a placement Shopify offers on Plus. On a lower plan Shopify ignores the placement and the wallet disappears from the payment method list entirely, which is a different experiment than the one you intend. The app says so on the form. Hiding and reordering regular payment methods work on every plan.
• A plan, but only to go live. Building an experiment and previewing both arms are free; putting one live is what needs a plan, and it is also what makes results start arriving. See Pricing.

The app asks for four access scopes and nothing more:

• read_orders, to count checkouts and orders. This is protected customer data under Shopify's rules; the app requests the minimum access level and reads no customer fields at all.
• read_markets, for the market picker. Without it you would be typing market handles by hand.
• read_payment_customizations and write_payment_customizations, to create the app's own Shopify Function and switch it on and off.

There is no customer scope, no product scope, no theme scope, no storefront access token and no web pixel scope.

Installation

Installing from the Shopify App Store takes care of authorisation and the app's own setup, including creating the payment customization. Two things remain, and the Settings page links straight to both with what the app can see of them itself.

1. Switch on the theme app embed. Online Store → Themes → Customize → App embeds → Checkoutlab → enable, then save. App embeds are off by default after any app install. The app deliberately asks for no theme access, so this is the one step it cannot verify: if the embed is off, nobody is assigned an arm and the results stay empty.
2. Repeat that after a theme change. Duplicating a theme or publishing a new one gives you a theme where the embed is off again. A running experiment that suddenly drops to zero is almost always this.

The payment customization under Settings → Payments is created by the app and switched on and off by itself: on while something is live, off when nothing is. If it ever goes missing, Settings has a Check and repair button. Nothing needs to be pasted into your theme code, and no snippet is added to theme.liquid.

How an experiment works

Three parts, each doing one job:

• Assignment. The theme app embed works out which experiment or rule applies to this visitor, puts them in an arm, and writes it to a cart attribute called _ab, in the form express-nl:t|ideal-first:c. The arm comes from a deterministic hash, so the same visitor always lands in the same arm.
• The variant. In the checkout, the payment customization function reads that attribute and applies the operations for every entry it finds. It runs on Shopify's own infrastructure as part of the checkout, so there is no request to our servers while a shopper is paying.
• Measurement. Two order webhooks count the outcome. See what gets measured.

Alongside _ab, the embed writes _ab-market and _ab-device. Those never drive an experiment, because the function never reads them, and exist only so the results can be broken down. Shopify copies all three onto the order itself, so you can see them in the admin under Additional details on any order.

Consent is respected without breaking the experiment. A visitor ID in local storage is, under the GDPR, no different from a cookie, so the embed only writes one when Shopify's Customer Privacy API says analytics processing is allowed. Without consent it falls back to the cart token, which already exists as a functional necessity and is stable for as long as the cart does. The experiment keeps running and nothing is stored on the device.

Running an experiment

1. Pick a type. Hide express checkout, hide payment methods, or reorder payment methods. For the last two you pick from the methods that actually came through your checkout, with the number of orders next to each, rather than typing a name you think you see.
2. Pick the markets. Leaving markets empty claims every market. That is not a conflict (see precedence), but it does mean a more narrowly targeted experiment will take over wherever it overlaps.
3. Add conditions, if the cart should qualify. See conditions.
4. Pick the split. 50/50 unless you have a reason. A smaller variant share needs proportionally more traffic for the same verdict.
5. Preview both arms. The Preview button opens your storefront with the panel on this experiment, where you can force yourself into either arm and then hit Go to checkout. Do it before you start. It is the only way to see what the variant really does, and it costs no visitors. A draft or paused experiment can be previewed too, which is the point: a draft is the last moment at which what it does can still be changed.
6. Start it. Starting is a separate action from saving, on purpose: it is the step with consequences for real shoppers, and it shouldn't be something you take along by accident while fixing a name. This is also the moment your plan or your trial is checked.

Run an A/A test first if you can. Start an experiment with an empty variant and confirm the arms converge and the sample ratio check stays quiet. If that fails, you are measuring something broken, and better to learn it on an experiment with nothing at stake.

Several experiments at once

Experiments only meet when they touch the same surface of the checkout. There are two: the express checkout row, and the payment method list. Hiding methods and reordering methods share the second one; hiding the express row has the first to itself.

Within one surface, exactly one experiment or rule applies to a given visitor, and the order is fixed:

1. More specific market targeting wins. An experiment on nl beats one on nl, de, fr, which beats one on every market.
2. An experiment beats a rule at equal scope. You started the experiment on purpose and want the answer; the rule is what you already know.
3. Oldest first, then key, only so two equally ranked entries always resolve the same way.

Whatever loses simply does not apply to that visitor. It is not stopped, and it keeps running in the markets where it does win. The app never blocks you from creating an overlap; it shows you where each experiment actually applies instead, on the home page, in the list and on the experiment itself. Worth knowing: an experiment can keep its “running” badge while a more specific one has quietly taken over its entire market.

So hiding Klarna everywhere except Germany while also hiding the express row is two experiments side by side: different surfaces, no interference at all.

Conditions

On top of markets, an experiment or a rule can carry up to five conditions on the cart itself, combined with And or Or:

• Cart total. Is at least, or is at most, an amount. Measured after discounts.
• Cart currency. Is, or is not, a given currency.

Markets decide who is in an experiment; conditions decide whether their cart qualifies once they reach the checkout. The split is deliberate: markets are known in the theme at assignment time, while a cart total is only real at the checkout.

The payment function and the checkouts/create webhook evaluate the same conditions against the same checkout, so what a shopper was shown and what gets measured cannot drift apart. A cart that does not qualify is neither treated nor counted: it was identical in both arms, so it says nothing about the experiment. A fact that cannot be read counts as not matching, because leaving the checkout alone is the safe side of an unknown.

What gets measured

Checkoutlab has no web pixel. The denominator is the checkouts/create webhook and the numerator is orders/create, and both payloads carry the cart attributes. That is deliberate: a pixel misses every checkout the browser never reports (a closed tab, an ad blocker, Shop Pay on another origin), and that loss is not guaranteed to be arm-independent, which is exactly the kind of bias that quietly decides an A/B test for you. Webhooks are server to server and complete.

Conversion is orders divided by checkouts, per arm. The baseline therefore sits around 50% instead of around 2%, which is why these experiments reach a verdict on far less traffic than a session-based test of the same change.

One thing is deliberately not in the numbers: express buttons on the cart or product page. Those skip the checkout entirely, carry no cart attributes, and count in neither arm, rightly so, because those buyers never saw the variant. If you have them switched on, expect a gap between these numbers and Shopify Analytics.

Reading the results

• Outcome. A two-sided two-proportion z-test at α = 0.05, sized for 80% power. Below 100 checkouts in an arm the page says there is not enough data rather than showing a difference that a single order could flip.
• Lift and share. The difference between the arms, and the split that actually occurred.
• Sample ratio check. If the real split differs from the configured one at p < 0.001, the page tells you not to trust the comparison. An SRM means the assignment or the measurement is broken, however pretty the difference looks.
• Breakdowns per market and per device. A variant that wins on mobile and loses on desktop averages out to “no difference”.
• Counts cross currencies; money does not. Every order counts towards the conversion, which is currency-independent. Revenue is shown in the dominant currency only.

Segment breakdowns are not separate experiments. Slicing one test into ten segments and picking the one that reached significance finds a “winner” about four times in ten by chance alone. Use a breakdown to see whether an effect is concentrated somewhere, then run an experiment targeted at that market to confirm it. That is also the cheapest thing precedence buys you: the follow-up experiment on nl automatically takes over from the broad one while it runs.

Locking a winner in

A finished experiment can be redeemed. Redeeming the variant turns its configuration into a rule: the same type, the same payment methods, the same markets and conditions, applied to 100% of the visitors it targets. The experiment stops. Redeeming the control just stops the experiment: the variant lost, so there is nothing to apply.

A rule gets its own key (<experiment key>-locked), never the experiment's. So the finished experiment keeps its measurements readable, and a webhook that arrives late cannot land on the rule and quietly pollute a result you have already acted on.

Rules sit in the same precedence ladder as experiments, one rung lower at equal specificity. Starting a new experiment on the same surface and market therefore takes over from the rule for as long as that experiment runs, which is exactly what you want, because you are asking the question again. When it ends, the rule resumes by itself.

Redeeming is the one action that is never part of the trial: a rule keeps running after a trial ends, so it is the part worth paying for. See pricing.

Previewing on your storefront

Append ?checkoutlab-preview to any storefront URL to open a panel listing everything that is live on that storefront. Open one and you get its two arms with an Activate button to force yourself into either. ?checkoutlab-preview=<key> opens that experiment straight away, which is what the Preview button in the app links to.

• Drafts and paused experiments can be previewed too. They apply to that one browser only, take over their surface there (so a draft express experiment is not stacked on top of a running one), and are measured nowhere. A stopped experiment is published nowhere and has no Preview button.
• Once an arm is forced, the panel offers Go to checkout, because that is where the effect is visible. With an empty cart it says so rather than bouncing you off Shopify's checkout.
• The panel stays on while you walk around the storefront, cart included. It is not on the checkout itself, because it is a theme app embed and the checkout is not the theme, but the forced arm still applies there, because it was written onto the cart before you left.
• Exit preview closes the panel and drops every forced arm. ?checkoutlab=reset clears the stored assignment and any override.
• To check by hand: await (await fetch('/cart.js')).json() and look at attributes._ab.

Troubleshooting

The results stay at zero. In order of likelihood: the theme app embed is off (step 1 of the installation, and the most common cause by far); the experiment is still draft or paused rather than running; it targets a market no traffic came from; its conditions exclude nearly every cart; or a more specific experiment has taken over its markets, and the app shows you where it actually applies.

A running experiment suddenly drops to zero. Almost always a theme change. A duplicated or newly published theme has the app embed off again.

The checkout doesn't change in the preview. Outside the markets an experiment targets, the panel offers no arms at all. Which payment methods a checkout holds differs per market, and forcing an arm there would show you a checkout the experiment is never going to produce.

The express row is still there in the variant arm. Hiding the express row needs Shopify Plus. On a lower plan Shopify ignores the placement, and what you see instead is the wallet gone from the payment method list.

The sample ratio check is firing. Stop reading the conversion numbers and look at the assignment first: a theme running an older version of the embed, a second A/B tool also writing cart attributes, or bot traffic hitting one arm. An experiment with a firing SRM has no usable result.

The numbers differ from Shopify Analytics. Expected: express buttons on the cart and product page bypass the checkout and count in neither arm. See what gets measured.

What the app stores

Per checkout and per order: the arm, the market handle, the device class, the order total and currency, and two Shopify identifiers: a checkout token and an order ID. No name, email, address, IP or customer ID. The two identifiers exist to deduplicate at-least-once webhook delivery and to make an erasure request actionable; without them there would be no way to find a given person's rows at all.

Measurements older than the retention window are purged automatically. The default is 400 days and you can set it anywhere from 30 to 1095 days in Settings, which also has a Delete all data button. The full picture, including sub-processors and encryption, is on Security & data processing.

Uninstalling

Removing the app removes the theme app embed and the payment customization with it, so your checkout returns to normal for every visitor immediately, including anywhere a locked-in rule was applying. Shopify sends us an erasure request 48 hours later and everything belonging to your shop is deleted. If you want a clean end to an experiment, stop it in the app and export what you need before uninstalling.