Window Seascape by Charles Plaisted. My evaluation framework can provide a clear view of solutions and requirements.
When I undertake to evaluate a technology, I start with customer requirements. I formalize and publish these requirements freely to all in what I call an evaluation framework. This helps me compare the capabilities of multiple solutions, objectively and consistently. When I evaluate a vendor solution, I start by identifying the solution’s capabilities against each of the framework’s criteria.
Requirements
I derive the requirements for recommendation services by looking at all of its customers. “Customers” for recommendation services cover several roles, including:
• End-consumers of recommendations
• Business people managing recommendations
• Technical staff
Their requirements are the basis for my 100+ evaluation criteria.
As I prepare my requirements and evaluation criteria, I take some shortcuts. If all solutions offer a capability, I will drop it from my criteria regardless of its importance in order to simplify the evaluation process. For example, it is unarguably critical that you be able to start and stop the recommendation service. All products offer this capability, so it’s not a feature cluttering up my evaluation matrix.
More problematical are the capabilities that are very important but can’t be observed or usefully measured. At the very top of this list is, do Vendor A’s algorithms deliver better recommendations than Vendor B’s, of your content to your customers? The most accurate way to make this evaluation is to test your short list solutions in your environment.
In my observation, the most important capabilities are not in the technology, but in the vendor’s expertise and guidance to help you strategize, deploy and then continually optimize your recommendations. The most important technology capability for most users is the automation of recommendation selection, control, tracking and optimization. I’m not advocating a black box – you sometimes need to be able to exert control on how recommendations are selected. But you can’t be involved in the decisions for every page, product, category and article on your site. Think automatic transmission: you don’t have to choose a gear unless you want to.
Evaluation Categories
I have grouped evaluation criteria into seven categories:
Solution positioning
• Vendor’s guidance and advice
• Recommendation structures
• Recommendation management
• Integration and ecosystem
• Operations
• Vendor’s development and maintenance
• Company and Product Viability
The Recommendations Evaluation Framework is provided in a free Patricia Seybold Group report here. You can use the framework to evaluate solutions for your business. The recommendation solution requirements are provided in a blank matrix you can edit and populate with comparative data, also free, available here.
On Target 