Step by Step Guide to A/B Testing Shopify Themes (Using Convert Experiences)

Dionysia Kontotasiou
By
September 27, 2021 ·
Step by Step Guide to A/B Testing Shopify Themes (Using Convert Experiences)

As a Shopify business owner, finding the right theme is tough. ​​A lot of time and energy goes into setting up a successful ecommerce store. And it all starts with spending hours picking out the perfect theme to give your online store the right look – which makes it even more frustrating when it underperforms.

So many different themes to choose from, but how do they all compare? Which one should you use for your business to delight visitors and maximize sales?

Well, what if there was a way to pick more than one theme? To make sure the design actually works and is getting you sales before committing to just one?

A/B testing is a great way to measure the effectiveness of your chosen Shopify theme(s). One way to do this is by picking two (or more) themes with different designs (for example, theme A and theme B), then testing their effectiveness against each other!

This tutorial will walk you through everything you need to know about A/B testing Shopify themes. For illustration purposes, I’ll be detailing this process using Convert’s Split URL option.

1. Select the Shopify Themes You Want to Test

First of all, you need to select two Shopify themes to test. (You’ll need to limit yourself to only two themes for the first test. If you’re considering more than two themes, you’ll need to run subsequent tests.)

The main theme will be the one you are showing live to the Shopify store visitors, while the other that you want to test against will be in Preview Mode (not published).

In the Shopify app, go find your live theme in Sales Channels > Online Store, then Themes:

shopify themes

Then, pick the second theme you want to test from the Theme Library:

shopify theme library

2. Install the Convert Tracking Code in Both Themes

Now that you’ve chosen your themes from the Shopify platform, you’ll need to install the tracking code. This is an essential part that will ensure visitor behavior is tracked properly and the test will give you the rightful winner!

Simply follow the instructions below to install the Convert Experiences tracking code in BOTH the Shopify themes that you want to split test.

Go to Sales Channels > Online Store > Themes > More Actions > Edit Code.

Shopify theme edit

Then, select the theme.liquid file and paste the Convert tracking code below just before the first “{% if %}” section, as shown in the screenshot. Make sure you replace the project ID number with your own Convert account-project ID.

<!-- begin Convert Experiences code-->
<script type="text/javascript">
var _conv_page_type = "{% if template == 'index' %}homepage{% else template == "collections" %}collection{% else template == "article" %}article{% else template == "product" %}product{% else template == "cart" %}cart{% else %}other{% endif %}";
 var _conv_category_id = "";
 var _conv_category_name = "{{ collection.title }}";
 var _conv_product_sku = "";
 var _conv_product_name = "{{ product.title }}";
 var _conv_product_price = "{{ product.compare_at_price_min | money }}";
 var _conv_customer_id = "";
 var _conv_custom_v1 = "{{ tag }}";
 var _conv_custom_v2 = "{{ order.shipping_address.province }}";
 var _conv_custom_v3 = "{{ cart.item_count }}";
 var _conv_custom_v4 = "{{ cart.total_price | money_without_currency }}";
</script>
<script type="text/javascript" src="//cdn-3.convertexperiments.com/js/<REPLACE THIS WITH PROJECT ID NUMBER>.js"></script>
<!-- end Convert Experiences code -->
shopify result tracking code

Lastly, don’t forget to save your changes!

Like I mentioned above, you’ll need to apply the same steps to the second theme you want to test. Remember, that is the theme you’ll keep in preview mode.

Start by going to Sales Channels > Online Store > Themes > Theme Library > (select the theme you want) > Edit Code.

shopify edit theme code

Then, simply repeat the process for the second theme: Select the theme.liquid file, edit the code using the script above, and save your changes.

It should look like this, depending on which theme you selected:

shopify convert tracking code

For more detailed instructions on how to install the Convert tracking code, follow the steps here.

3. Allow Revenue Tracking

Now for the important part: tracking revenue. How else will you know which Shopify theme worked better for your business?

There are three main ways to track revenue, which I cover in greater detail in this post.

In short, you can decide to either

  1. Install the Convert tracking code on the order status Shopify page to be able to track revenue/purchases through Google Analytics,
  2. Set up manual revenue tracking code, or
  3. Use webhooks.

You can use revenue tracking to know when your customers visit your checkout and how often they make a purchase. It’s also a great way to learn more about purchasing trends in your store, like average order value and total sales per product.

To set it up, go to Settings > Checkout and scroll down to the Additional Scripts section. In the Additional Scripts form field paste the following code:

<script type="text/javascript">
  if (typeof _conv_host == "undefined") {
  window["_conv_prevent_bodyhide"] = true;
  (function() {
    var _conv_track = document.createElement("script");
    _conv_track.src = "//cdn-3.convertexperiments.com/js/XXXXX-XXXXX.js"
    document.getElementsByTagName("head")[0].appendChild(_conv_track
</script>

Replace the text, “XXXXX-XXXXX”, with your Convert account-project ID found in your Project’s Settings.

4. Find the Shopify Preview Theme ID

Next, you’ll need to edit the code of the unpublished theme from within the Shopify website. Remember, you can do so by going to Sales Channels > Online Store > Themes > More Actions > Edit Code.

Shopify Preview Theme ID

Then, you can copy the theme ID from the admin URL, /admin/themes/xxxxxxxx.

You will need this ID so that you can append ?preview_theme_id=xxxxxxxx as a variation URL for the split URL test (more on this in the next section).

5. Create a Convert Split URL Test

Now, let’s move to the A/B testing tool, in this case, Convert Experiences, to set up the split URL test.

shopify Split URL Variations

In the Edit Split URL Variations section, insert this code in the Original URL field :

^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))?

Use the following as the Variation URL, where the preview_theme_id variable is what you discovered in the previous step (and don’t forget to replace ‘123456’ with your own Shopify preview theme ID):

$1$3$5?preview_theme_id=123456&$7

The resulting URL will be something like this:

https://www.store.com?preview_theme_id=123456&.

Then, enable Support Regular Expressions and Transfer Original URL variables to the variation URL. Your test variations should look like this (except the theme ID will be different):

enable options Split URL Variations

6. Specify Which Shopify Pages To Target

Once again, go to the Convert Experiences app. Make sure that the Site Area section includes all the pages in your Shopify store (including all those valuable product pages!).

It could look something like this:

Shopify Pages To Target
Pro-Tip

Make sure you exclude pages when the query string contains: preview_theme_id=123456.
This is very important so that when you activate the experiment, it doesn’t cause a loop. If you see a loop, you’ll know you’ve missed this step.

shopify exclude pages

7. Hide the Shopify Preview Bar

If we stop here and just activate the split URL experiment when the preview theme template is shown randomly to website visitors, an annoying popup appears specifying that it is a preview template. This might not be appealing to your visitors and might raise concerns about what other tests are running on your store.

shopify preview ba

To avoid causing unnecessary friction, add the CSS code below to Shopify to hide the preview banner. Please note that if you want to hide this banner from checkout pages as well, you’ll need a Shopify Plus account, as the code only applies to Shopify Plus clients.

In Shopify, go to Online Store > Preferences > Google Analytics and add the following code in the Additional Google Analytics JavaScript box:

function() {
var css_override = document.createElement("style");
document.getElementsByTagName("head")[0].appendChild(css_override);
})();

var css = 'iframe#preview-bar-iframe { display: none !important; }',
head = document.head || document.getElementsByTagName('head')[0],
style = document.createElement('style');
style.type = 'text/css';
if (style.styleSheet) {
style.styleSheet.cssText = css;
} else {
style.appendChild(document.createTextNode(css));
}
head.appendChild(style);

8. Launch The Split URL Experiment

You are now ready to activate your split URL experiment. To do so, you will need to go to your experiment page in Convert Experiences and set it to Active.

experiment page in Convert Experiences

Once the experiment is live, this is what happens when a visitor lands on one of your Shopify store pages:

split URL experience flow

8. Analyze Your Results and Pick a Winner!

Convert uses Frequentist Inference to determine when and by how much your Shopify themes are better than each other. As a rule of thumb, we recommend waiting two weeks before choosing the winning theme, and the tool itself generally doesn’t tell you a winner earlier than this (unless you have high traffic on your site).

This is also the main reason that our free trial runs for 2 weeks.

In the world of ecommerce, two weeks isn’t even that long. So sticking to at least two weeks is not a bad idea. After all, we are testing entire themes so it won’t hurt to be completely sure about the impact your test has.

Conclusion

So there you have it.

Whether you’re a new Shopify store owner looking to find the perfect theme for your business or an experienced ecommerce business owner who’s looking for a better-performing one, Shopify A/B testing is something that can help you improve user experience, optimize your conversion rates and sales, and make better business decisions.

We’ve outlined all of the steps above so don’t worry if the testing process is unfamiliar territory for you! And if you’ve liked how easy to use Convert Experiences is, click on the banner below to take a free trial so you can start working towards finding the right Shopify theme right away.

Free Trial
Free Trial
Originally published September 27, 2021 - Updated November 10, 2022

Mobile reading?

Scan this QR code and take this blog with you, wherever you go.

Authors
Dionysia Kontotasiou
Dionysia Kontotasiou

Convert's Head of Integration and Privacy, helping customers with technical queries.

Editors
Carmen Apostu
Carmen Apostu

In her role as Head of Content at Convert, Carmen is dedicated to delivering top-notch content that people can’t help but read through. Connect with Carmen on LinkedIn for any inquiries or requests.

Start Your 15-Day Free Trial Right Now.
No Credit Card Required

You can always change your preferences later.
You're Almost Done.
I manage a marketing team
I manage a tech team
I research and/or hypothesize experiments
I code & QA experiments
Convert is committed to protecting your privacy.

Important. Please Read.

  • Check your inbox for the password to Convert’s trial account.
  • Log in using the link provided in that email.

This sign up flow is built for maximum security. You’re worth it!