Skip to main content

A/B testing

A/B testing is a experimentation process in which two or more versions of a variable (page, page element, etc.) are shown to different segments of users. The process collects information on the performance of each variant. This helps brands decide which changes they want to move forward with.

In NewStore, you can test variants of Shopping App feature configurations using Firebase.

This article describes how you can set up A/B tests for the Shopping App. It also contains a list of the supported A/B test scenarios.


To set up and run A/B tests in the Shopping App, the following are required:

  • Consumers using iOS core version 2.124.0 or higher.
  • The person setting up the test needs access to the Firebase Console account provided by NewStore during onboarding.

Setting up an A/B test

To set up an A/B test, perform the following steps:

  1. In the Firebase Console, navigate to the A/B Testing option on the left, and select Create experiment - Remote Config.
  2. Follow the steps to set up the experiment:
    1. Type an Experiment name for your A/B test.
    2. In the Targeting section, define the percentage of users that should be a part of this A/B test.
    3. In the Goals section, select the Primary metric to track (e.g. Purchase Revenue, Retention, etc.). You can define additional metrics as well.
    4. In the Variants section, define the baseline and the variants that you want to test. For both baseline and variants, add the following:
      1. In the Parameter field, add the key that corresponds to the feature you want to test. See the available tests and the corresponding keys later in this article.
      2. In the Value field, select which behavior you want to show in this variant. For example, you might want to test different layouts on the product list. In that case, add the productListLayout parameter to the baseline, and in the value field add which behavior you want to test in the Baseline group (inspirational or two). Then in the other variant(s), add the same parameter with another value.
      3. Under Adjust variant weights, add participation percentages for baseline and variants groups.
    5. After reviewing the draft experiment, select Start experiment to activate the variants and start collecting data.
  3. During and after the A/B test, draw conclusions from the results shown in the Experiment results section. When there is a clear winner or change required, this can be managed from Firebase directly.


As an example, let's see an A/B test for showing the color swatches on the product list page.

To see if hiding color swatches on the product list page works better for an imaginary brand's users than showing the swatches, the brand decides to set up and run an experiment for 100% of the users:

  • The Variant A user group is 80% of the users and receives false value for showColorSwatchesOnProductList.
  • The Baseline user group is thus 20% and receives a value of true for showColorSwatchesOnProductList.

After running the test for some days, Firebase shows the following test results:

When analyzing the results of this example, for the tested users the baseline is a clear winner based on several metrics, but mainly due to the revenue per user, which is $55.05 compared to the $21.08 result of Variant A. According to the results, the brand can stop the experiment and make the baseline the default.

List of available A/B tests

The following table shows the keys you can use for A/B testing:

KeyControlsTypePossible values and their meaning
productListLayoutChanges the product list layout.String"inspirational": inspirational layout
"two" : two-column layout
showApplePayOnPDVShows the Apple Pay express checkout on the product detail view in the main app.Booleantrue: enable
false: disable
showBottomPopoverSoldOutVariantsShows the unavailable variants in the configuration at the bottom.Booleantrue: show unavailable options on the bottom
false: sort options as they are sent from the e-commerce environment
showColorSwatchesOnProductListShows color swatches at the bottom of the gallery view on the product listing page. This test needs previous configuration. If you want to use this key, reach out to your NewStore contact to ensure that the necessary feature is enabled.Booleantrue: show
false: hide
showExpandedInfoTextOnDetailViewChanges the "More info" text on product detail view.Booleantrue: "Expand details"
false: "More details"
showMediaGalleryOnProductListMakes the gallery views horizontally scrolling and enables scrolling them on the product listing page.Booleantrue: enable
false: disable

What do the keys change

The following images show which part of the user interface each key controls. The differences are marked on the side of the screenshots.