A/B tests
Compare two search setups on real shoppers, and know when the result is real.
An A/B test shows two (or up to four) search setups to different shoppers at the same time, and tells you which one works better on a real measure, such as the click rate or the order rate. A/B tests are in the Pro and Enterprise plans.
What you can test
| Kind | Example |
|---|---|
| Ranking | Boost products in stock, or new products, more or less |
| Synonyms | Your synonym list against the list plus new synonyms |
| Rules | Search with or without a set of pinned or boosted products |
| AI rewrite | AI rewrite on long requests against no AI rewrite |
Each test kind may change only its own settings, so a ranking test never changes your synonyms.
Measures
Pick one main measure: Click rate, Order rate, Add-to-cart rate or No-result rate (lower is better). The other measures are shown too.
Run a test
- In the dashboard, open A/B tests and click New test.
- Pick the kind, the main measure, the share of traffic in the test (1-100%), and set up each variant. The first variant is the control (what you have today).
- Before you start, SeekGist shows how many shopper sessions each variant needs and about how many days that takes at your traffic.
- Click Start. Variants are locked from now on.
- Read the result on the same page. Click Stop when the page says there is a winner, or when the test reached its planned size.
Only one test can run at a time per store.
Who sees what
Each shopper session is put in one variant by a hash of the session id and the test. A shopper keeps the same variant for the whole test, and the traffic share never moves a session between variants. No shopper personal data is used: session ids are hashed.
When is the result real?
You can open the result page at any time; looking often does not cause false winners. SeekGist shows winner only when all of these are true:
- the difference passes a test that stays valid however often you look (an "always valid" sequential test, mSPRT), at the 5% level;
- each variant has at least 100 sessions;
- the test has run for at least 7 days (one full week of shoppers, so weekdays and weekends both count);
- the traffic split looks right. If the sessions per variant are far from the shares you set, something is wrong with the setup, and SeekGist hides the winner and asks you to check the split.
With 3 or 4 variants, the test is corrected for the extra comparisons (Holm). The page also shows the classic p-value, marked as not for decisions while the test runs.
What the page says
| Message | Meaning |
|---|---|
| Collecting data | It is too early to judge. |
| No clear difference yet | Keep the test running. |
| One variant is ahead, but wait a full week | The difference looks real, but 7 days have not passed. |
| The difference is real | You have a winner. Apply it, then stop the test. |
| No clear difference | The test reached its planned size; the variants are about the same. |
| Stopped early | The test was stopped before 7 days or before its planned size. |
| Check the traffic split | The sessions per variant do not match the shares you set. |
Webhook
experiment.finished is sent when a test stops, with its result.