A/B testing

A/B testing is a experimentation process in which two or more versions of a variable (page, page element, etc.) are shown to different segments of users. The process collects information on the performance of each variant. This helps brands decide which changes they want to move forward with.

In NewStore, you can test variants of Shopping App feature configurations using Firebase.

This article describes how you can set up A/B tests for the Shopping App. It also contains a list of the supported A/B test scenarios.

Prerequisites

To set up and run A/B tests in the Shopping App, the following are required:

Consumers using iOS core version 2.124.0 or higher.
The person setting up the test needs access to the Firebase Console account provided by NewStore during onboarding.

Setting up an A/B test

To set up an A/B test, perform the following steps:

In the Firebase Console, navigate to the A/B Testing option on the left, and select Create experiment - Remote Config.
Follow the steps to set up the experiment:
1. Type an Experiment name for your A/B test.
2. In the Targeting section, define the percentage of users that should be a part of this A/B test.
3. In the Goals section, select the Primary metric to track (e.g. Purchase Revenue, Retention, etc.). You can define additional metrics as well.
4. In the Variants section, define the baseline and the variants that you want to test. For both baseline and variants, add the following:
  1. In the Parameter field, add the key that corresponds to the feature you want to test. See the available tests and the corresponding keys later in this article.
  2. In the Value field, select which behavior you want to show in this variant. For example, you might want to test different layouts on the product list. In that case, add the productListLayout parameter to the baseline, and in the value field add which behavior you want to test in the Baseline group (inspirational or two). Then in the other variant(s), add the same parameter with another value.
  3. Under Adjust variant weights, add participation percentages for baseline and variants groups.
5. After reviewing the draft experiment, select Start experiment to activate the variants and start collecting data.
During and after the A/B test, draw conclusions from the results shown in the Experiment results section. When there is a clear winner or change required, this can be managed from Firebase directly.

Example

As an example, let's see an A/B test for showing the color swatches on the product list page.

To see if hiding color swatches on the product list page works better for an imaginary brand's users than showing the swatches, the brand decides to set up and run an experiment for 100% of the users:

The Variant A user group is 80% of the users and receives false value for showColorSwatchesOnProductList.
The Baseline user group is thus 20% and receives a value of true for showColorSwatchesOnProductList.

After running the test for some days, Firebase shows the following test results:

When analyzing the results of this example, for the tested users the baseline is a clear winner based on several metrics, but mainly due to the revenue per user, which is $55.05 compared to the $21.08 result of Variant A. According to the results, the brand can stop the experiment and make the baseline the default.

List of available A/B tests

The following table shows the keys you can use for A/B testing:

Key	Controls	Type	Possible values and their meaning
`productListLayout`	Changes the product list layout.	`String`	`"inspirational"`: inspirational layout `"two"` : two-column layout
`showApplePayOnPDV`	Shows the Apple Pay express checkout on the product detail view in the main app.	`Boolean`	`true`: enable `false`: disable
`showBottomPopoverSoldOutVariants`	Shows the unavailable variants in the configuration at the bottom.	`Boolean`	`true`: show unavailable options on the bottom `false`: sort options as they are sent from the e-commerce environment
`showColorSwatchesOnProductList`	Shows color swatches at the bottom of the gallery view on the product listing page. This test needs previous configuration. If you want to use this key, reach out to your NewStore contact to ensure that the necessary feature is enabled.	`Boolean`	`true`: show `false`: hide
`showExpandedInfoTextOnDetailView`	Changes the "More info" text on product detail view.	`Boolean`	`true`: "Expand details" `false`: "More details"
`showMediaGalleryOnProductList`	Makes the gallery views horizontally scrolling and enables scrolling them on the product listing page.	`Boolean`	`true`: enable `false`: disable