(Author’s note: this blog employs a silly, hackneyed metaphor in an effort to hold your attention. Proceed with caution and please direct any hostile feedback to our biz dev guy (email@example.com). Trust me, he deserves it.)
There aren’t that many cultural touchstones that fall squarely into the “if you’ve never experienced this, we probably wouldn’t get along” category. Seminal computer game The Oregon Trail is one of them. As a grade schooler, I spent countless hours in Mr. Vermillion’s computer class pondering important decisions like which of my classmates would make the best companions (e.g. eventual victims) and how I should cross the Kansas River (Take the ferry like a punk? Ford the river? Caulk the wagon and float across?!).
Nowadays, dysentery and cholera are less abundant but risk management is still a part of my day to day. In fact, incepting and executing an eCommerce optimization program is a bit like “the trail.” In other words, while risk planning and mitigation are paramount, unforeseen obstacles will always arise. As an exercise, let’s imagine some specific CRO obstacles in terms of a virtual settler family’s journey from the Missouri River to the Willamette Valley.
Risk Planning During the Discovery Process
While we generally suggest you wear clothes and eat, oxen and ammunition probably won’t do you a world of good when creating an optimization program. But as anyone who has piloted a successful optimization program knows, having an established process for concepting, executing and analyzing experiments and personalization campaigns is absolutely vital. Without a proper discovery period, it’s difficult to generate well-reasoned, actionable hypotheses.
However, almost as essential (if a bit less exciting) as plotting out the opportunities is examining the potential risks involved.
Risk: Say you’re running an experiment to add a duplicate checkout button at the top of your cart. And even though you’ve told your client or development team that they should be in a production freeze on the cart while the test is running, they deploy something anyway. If there’s a conflict, that additional element you’re testing could go missing or, even worse, stick around firing an error. This could be problematic for your data and UX.
Mitigation: Everyone should be doing cross-browser and device QA for experiments. That’s a no-brainer. But even this won’t catch regressive issues down the road. Visual regression testing is an awesome addition to your toolset. You can build out scripts through a tool like Applitools Eyes to run through pages targeted by your experiments on a local server, clicking on certain elements and taking screenshots. The tool will compare those shots to the baseline you established and alert you if there are differences. Within the dashboard, you’ll be able to drill down to find and fix them before they affect test results or conversion.
Oregon Trail Equivalent: This is your classic snakebite scenario. Just when you thought you had accounted for every corner case, something will spring up and bite you in the butt. Or wherever. Doesn’t have to be the butt.
Funky Old Code
Risk: Let’s face it, chances are you’ll eventually run into a code base that’s less than ideal for conversion testing. One of the places where we often run into gnarly legacy code structure is the cart/checkout. Oftentimes, this means lots of on-site JS updating the DOM to destroy everything you write or changing the markup and breaking your styling.
Mitigation: The most obvious path here is to attempt to avoid poor markup. However, that’s not always possible. Sometimes the entire code structure is poor or the pages with the poorest structure are key to your optimization efforts. A few things you can do:
Account for all potential risks during discovery with a thorough code audit.
Prioritize test hypotheses based on both level of impact and potential for markup migraines.
Assign more difficult experiments to senior members of your development staff so they can investigate why native functions are breaking the code and remediate accordingly.
Remember through all this that optimization platforms should not be used to fix broken on-site code.
Oregon Trail Equivalent: Drowning. As in you decided to spurn the relative ease of a ferry to the product page in favor of a caulked wagon into the checkout and now you’re drowning in an abyss of JS errors (metaphors!).
Pivoting Client or Upper Management Goals
Risk: Stop me when this sounds familiar. You spend hours poring over analytics and user testing clips looking for the most pressing opportunities for site optimization. You present all of your hypotheses to your client or boss, get alignment on an initial flight of tests and get all of your design comps and requirements in order. The developers dive right in, coding up a storm of pure Variation JS magic. Then, someone from the executive team, fresh out of a board of directors meeting, blindsides you with a request to stop what you’re doing and refocus all efforts on squeezing as much revenue as possible out of a massive 40% blowout they’re doing next week. “Just spin up some banners and modals through the optimization platform (it’s basically a CMS, right?).”
Mitigation: There is none; things like this will always happen to you.
Just kidding. Sort of. Pretty much all three of your major risk types are in play in this scenario: budget, schedule and quality. Nothing can gut a well-oiled optimization program faster than the executive-level trump card. So you might as well have a plan in place to get ahead of this stuff. Things you can do include: asking for yearly or quarterly business goals ahead of time, keeping an up-to-date promotional calendar and (perhaps most important) making sure you understand all of the layers of approval in an organization and doing your best to get everyone’s buy-in on your program so they’re less likely to railroad it.
Oregon Trail Equivalent: Dysentery. I don’t know why, but this one is definitely dysentery.
Many project risks are near eventualities and should be planned for as such. But a whole host of others will catch you totally off guard. Just as sure as the weather will turn, you’ll uncover some bogus code structure or a CMO will swoop in to derail or inflate your efforts. But proper risk planning and effective mitigation can do wonders to avoid the untimely demise of your program.