When engineers build complex machinery like jet engines and race cars, they use digital simulations to cost effectively optimize their designs before building physical manifestations of their creations that can be tested in the real world.
(Richard Prince/GM Racing Photo)
A similar technique can be used to optimize applications of merchandising algorithms before testing their performance online with real users. This technique is the offline pilot.
What are offline pilots, and why are they so critical for maximizing ROI?
An offline pilot uses your business data to train and run simulations against an algorithm to estimate its ability to improve an aspect of your merchandising experience. Offline pilots optimize your applications of personalized merchandising by helping you more efficiently:
- Determine the best applications for your business
- Evaluate the best data to use to power your applications
- Optimize the design of the underlying algorithms/models
Simply put, if you’re not using offline pilots to validate the potential of new merchandising applications you’re not maximizing your return on investment by:
- Minimizing resources spent implementing inferior applications
- Deploying the winning applications more quickly
- Avoiding bad customer experiences
The offline part is key. By taking initial testing offline, many of the costs of the online application of the algorithm can be deferred until after the desired pilot results are achieved. This is because engineering work such as the automation of the data inputs that power the algorithm and integration into your customer experience aren’t necessary for a pilot .
Deferring these costs allows you to evaluate pilot versions of the applications more rapidly and improve the applications that make it to online testing. These can be completely different applications, algorithm changes, or changes to input data to improve application performance.
Let’s face it, you only have so many touch points with your users to get statistically relevant online tests of an algorithm (you are A/B or multivariate testing your algorithms, right?). You shouldn’t waste these precious cycles on inferior applications. You also don’t want to harm your customer experience and business performance online testing an algorithm that could have been ruled out during an offline pilot.
How do offline pilots work?
At the start of the pilot process you should consider a broad range of candidate merchandising applications. Consider parts of your merchandise experiences with inherent discovery challenges, such as larger collections of products, media, categories, brands, etc. Also consider where discovery that leads to the most consumption occurs, as improvement in those areas will have the largest impact.
Consider not only where you can improve the ranking of existing collections, but also where you can employ recommendations to pull the best items out of larger collections. Recommendations can be particularly valuable when the items of interest to a user are spread across multiple collections, such as products that are spread across many categories.
Next consider the data you have available to power algorithms. There are three classes of data that are potentially valuable:
- Information about the items you merchandise
- Information about your users
- User activities in your user experience
At a minimum you need user activity representing interaction with catalog items that support the business goal you want to improve, as that data is essential for model development. For example, if you are trying to improve revenue from product sales, you will need to supply order activity data linking a user, a product purchased, and the revenue generated.
Start with your easily accessible data. Only invest in additional data if the pilot determines it unlikely that the data you have is insufficient to generate the goal improvement you are seeking.
Once your data is assembled, a pilot model can be generated and the application evaluated. During the pilot each application’s ability to rank or select items in your merchandising experience is assessed, using your activity data as proof of what the customer actually desired. Each application’s selections or rankings can be compared against a baseline, such as current selections or rankings in your customer experience (if available) or, less optimally, against more naive approaches such as newest items.
For example, let’s say you are evaluating a model’s ability to reorder products within a category for an ecommerce business. Using recent order history, you can evaluate the model’s ability to improve the positioning of products purchased by each user in the categories in which it appears vs. its current ranking in the customer experience. If the model ranks the product(s) a customer purchased at a significantly higher position that its current ranking, you now have a candidate for online testing.
With an ability to eliminate implementation costs on subpar applications, maximize the performance of the applications tested online, and more rapidly develop the applications, offline testing should be an invaluable part of your merchandising toolkit.
Attune conducts free offline pilots in all of our customer engagements. To start your own offline pilot, contact Attune using the Get Started button below.