Aug 24 2017

Ecommerce Myth Busters: Platform Cutovers Are Risky and Terrifying

Every day, in pursuit of more online revenue and simpler processes, brands begin the labor-intensive process of a cut-over. In doing so, they not only invest time and money, they risk everything they’ve built. We’ve all heard post-cutover horror stories about reduced traffic, slumping conversions on formerly high-performing pages, and loyal shoppers who struggle with new customer journeys. These are the potential consequences of boldness, but the threats don’t have to loom over your head. A little scientific thinking goes a long way to mitigating risk.

While risk management is status quo–such as hiring an agency that is CMMI-certified and filled with PMPs–a few other tactics will further reduce risk. You can benchmark, conduct tests before, during, and after launch, run usability tests (and on your competitor’s site too!), and ensure development practices sound SEO. In this article, we walk through all of these tactics so that you can build them into your cutover plan.

The Role Benchmarking Plays

Tracking the success of a new design or ecommerce platform requires relatively modest effort. Compare what you did during the same period last year, but make sure to consider any promotions at the time. Accurate year-over-year comparisons are indispensable. Above all, you compete against yourself, which means benchmarking is important whenever a major event is on the calendar.

Benchmarking also helps you to measure and validate your return on investment. If sales and revenue are up that’s obviously a great start, but by how much? Which customers, which pages, and which features are driving those increases?

You can further improve your ROI with post-cutover testing, especially if you find weak areas that could be switched out. To get a full grasp of the situation, compare numbers to your benchmark and factor in the overall cost of the project.

What You Should Test

Ecommerce and testing go hand-in-hand, especially when a brand has matured and brings in a sizable amount of traffic on a weekly basis. Typically our industry thinks of A/B testing and experiments when we say optimization or ecommerce testing; however, there is a lot of other components of your site and business that are worth tracking and testing again.

Thanks to the magic of Google, there are two areas of focus that you should build into any new ecommerce design: site speed and mobile. Mobile practices are cut and dry. You need a mobile-friendly site, you can’t use interstitials that prevent a user from navigating your site, and you need a great UX to ensure they follow their customer journey to purchase, and you should optimize your checkout like it’s a hot-rod. Site speed, on the other hand, has a lot of contributing factors.

Site Speed Performance

There are several reasons why site speed plays a vital role for ecommerce brands, but the two biggest are organic rankings on Google and the patience of users. If your site is slow to load, it’s less likely to appear on the first page of search results. It’s as simple as that, and nobody wants to rely only on paid traffic.

Just as important as your organic reach, we now live in a world of instant gratification and high-speed internet. As a result, online buyers have little to no patience for slow-loading sites, and that means they will find an alternative means of buying what they need if your site takes too long. Unless you’re the only horse in town, site speed should be a high priority to test against, and ensure it’s one of your success factors. Speaking of, we dive into success factors next.

What to Benchmark: Homepage, key landing pages, cart and checkout flow, product pages

Performance Metrics

Key performance metrics (KPIs) are what your ecommerce and marketing teams run on. Without this data, you don’t have goals or signposts to help you identify weak areas in our strategy that you need to remedy. As such, benchmarking these metrics before and after a new design or implementation are as you’d guess, key. Each brand has their own goals and KPIs, but here are some of the more common ones.


Ecommerce sites get a variety of traffic, from paid to organic to referral. Paid traffic takes more effort to benchmark against since it is campaign and promotion-dependent, but organic traffic is less dependent on stray variables. You’ll want to track monthly organic traffic, unique users, pageviews, pages per session, average session durations, bounce rates, and percent of new sessions. All of this you can track in Google Analytics.

Traffic Sources

Beyond paid traffic, referral traffic plays a key role. Being able to track the success of each of your promotional or sales channels is not just important during a cutover, but in general. In Google Analytics you can easily benchmark your various traffic sources such as referral, social, organic, and direct.

Mobile Traffic

If your ecommerce site isn’t already generating more traffic from mobile than desktop, recognize that it’s now inevitable. Regardless of the demographic and geographic segment, mobile continues to beat desktop on average. Benchmarking old mobile visitors versus new ones is an obvious comparison. You’ll want to track KPIs like mobile conversion rate, general traffic, sales compared to desktop users, and where customers fall out of the shopping funnel.

Call to Action Performance

A few years ago, when the benefits of A/B testing started to spread like gospel, we heard countless tales about retailers who tripled, quadrupled, and quintupled their conversion rate by testing the color and shape of buttons. People like to believe in easy solutions, so they gulped these stories up. The trend helped grow the industry into what it is today.

While people inflated the effects of these minor changes, they do contribute to the bigger picture of call-to-action (CTA) performance. Small, iterative changes allow you to see what kind of CTAs work best for your users and customers. Is it buy now, add to cart, just a symbol? Color, size, placement, and text will all impact your CTA performance. Benchmarking this is good, but regularly testing minor changes is better. The same can be said for landing pages tied to paid promotions.

Call Tracking

If you offer sales by phone, then it’s highly likely that you list a specific phone number listed on your site. Thanks to the magic of Google Analytics, you can easily tag and track when mobile visitors click that number. Doing so allows you know how many of your incoming sales call or customer service request are coming in through your website, and on which page it originated.

The Basics

While the above areas to benchmark are unique to a brand’s targeted audience, there are always some basic data sets that are worth benchmarking as well. However, they are likely already a part of your plan. Store traffic, conversions, sales, average order value, sale frequency, sales/promotions, steps to purchase, and cart abandonment rate are all the norm.

Revenue Per Visitor

Do all metrics hold the same weight? Of course not! At Blue Acorn we feel that when it comes to testing, revenue per visitor is one of the more important metrics to track. Conversion rates are useful, but they don’t always indicate a clear path to increasing revenue. We also agree with Optimizely’s approach in that RPV should not be tracked down to the single session of a user, but the average revenue per visitor (may be more than one session). The reason: mobile sites continue to drive more traffic, but consumers still convert less than they do on desktops. While the trend is shifting, depending on the product and price point, multiple visits in the journey of a purchase make revenue per visitor a better metric.

Optimization Metrics

Performance metrics get the limelight, but what about the contributing factors that feed into them? Tracking conversion rate is practically an obligation, but so are the landing and exit pages. Just as your cart additions and abandonment rates are important, but so is benchmarking your customer journey.

For more of the basics, see what you should test for before, during, and after a new site launch or cutover.

Customer Journey

If you’re using Google Analytics, your customer journey is probably already mapped for conversions. If you’re not already doing that, stop reading and do that immediately. Without that kind of optimization data, you’re going to be losing out on an immense amount of data that can quickly improve conversions and sales. By benchmarking and tracking the old customer journey against your new one, you’ll also be able to identify if you reduced steps and if customers are following the path you set in place (they rarely do).

Top Landing Pages and Exit Pages

Top earning pages should work just as well if not better after you put a new design or platform into play. By tracking which pages are successful before a launch, you can ensure that any adjustments made in the future won’t impact your KPIs. It’s especially important to track for landing pages tied to paid ads. If you have common exit pages, showing where they began to improve will also be a great internal win.

High-Engagement Pages Not Related to Ecommerce

Pages such as your careers page, customer service section, and customer financing page are important to track. When cutting over to a new design, ensure that people can still find your high-engagement pages, even if they are not direct sales drivers.

Usability and User Testing

Not all data can be expressed in graphs and numerals.  Quantitative data is important in providing a full picture. By adding in user testing, you’ll add context to the data already at your disposal. If you’re not sure how customers would feel about a feature, you can watch and listen to users reacting to it on a competitor’s site.


Over the years, SEO has changed from a development and tactical process to something more consistent with organic content. While some of these dated approaches still work, Google now relies heavily on giving higher status to content that people want to consume. Think less bolded words, keyword stuffing, and backlinks in comment sections, and more high-quality blog posts, clear product pages with logical headings, and a clean code base.

SEO is also often ignored when switching to a new design or platform, which in turn can negatively impact organic traffic. To make certain that this doesn’t end up as a sore point, use a tool like MOZ to create a benchmark, track your keywords, and then document the organic flow of traffic coming in from key landing pages.

What to Benchmark: SERP, keyword rankings, organic traffic on critical landing pages

Caveats and Considerations

The biggest hurdles to accurate and worthwhile benchmarking and testing results are achieving statistical significance and attaining validity. Ultimately, you need enough traffic to support your tests, and even if you hit statistical significance, it’s still plausible that the test results won’t be validated once fully enacted on the live site. To achieve statistical significance, you need a combination of time and traffic. While we all wish results would solidify in a week, the length will depend on how much traffic flows through to the test.

With that said, even if you have a ton of traffic coming in, reaching statistical significance in a short window of time is not an indicator that you should stop a test and declare a winner. Tests need to bake for a while. If you shut off a test too soon, you may have what looks like a clear winner, but a week later the tides can just as easily change. Unfortunately testing and benchmarking appear like simple measures, but due to the intricacies involved are often complex. Fortunately, you know a team who can help you navigate these situations.

If you’re looking to redesign your ecommerce site, implement a new platform, or just want to bounce ideas off an optimization strategy team, feel free to reach out to us here.

Elliot Volkman

Digital Marketing Manager

Elliot is Blue Acorn's Digital Marketing Manager. He hold a master's degree in communication from Gonzaga, and has several awards for journalism and digital marketing. In his spare time he is a long-distance runner and triathlete.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.