By Derek Slater, Content Director, Ready State
As marketers start evaluating machine-learning systems, it’s crucial to spot when that title is deserved—and when it’s just a buzzword
Machine Learning is suddenly among the hottest labels under the sun, and products and services for marketers are especially eager to sew on that patch. Machine learning, of course, refers to the ability of a system to learn from data, improving its accuracy or performance without needing any additional programming. Machine learning systems help optimize themselves.
Just back from two of the big annual marketing shows, Kristen Alexander, vice president of marketing at Certain Inc., says “nearly everyone is touting either predictive, machine-learning, or AI capabilities.”
But in the buying process, marketers need to beware of ‘machine washing’—slapping that machine learning label onto a product that doesn’t really deliver the goods.
If you invest in a system expecting machine learning, and don’t get it—what’s the big deal? For starters, you eat into your experimentation budget and under-deliver on results. Now instead of achieving a breakthrough or ongoing improvement, you are stuck with another marketing automation system you have to optimize by hand.
To help you spot machine washing and evaluate machine learning products, we interviewed a data scientist, a CIO, and a marketing VP. Here are their insights.
First, questions to ask yourself (and your colleagues)
Don’t walk into a vendor demo cold. Before you try to make sense of product and service offerings, you first need to know a few things about your business requirements (duh!).
Kirk Borne, Principal Data Scientist at Booz Allen Hamilton, suggests knowing the answer to these three questions:
1) How will machine learning help with the program or project at hand?
It’s tempting to starting playing around with machine learning. But our experts say there is plenty of real work to be done, and enough real machine-learning products out there. It makes more sense to start with a real business project instead of a hypothetical.
Like anyone, marketers “can fall in love with a technology,” says Alexander. However, she says “it’s critical to ensure that the technology you acquire solves an acute pain point that is going to materially contribute return on investment.”
So, what is your project that will benefit from machine learning: Are you building a recommendation engine? Trying to improve your advertising media mix? Segmenting customers, testing web designs, exploring ways to target personalized email messages? In each case, know the payoff that machine learning provides in helping to automate work and optimize results.
Different tasks will need different tools. (Sounds obvious, but we’ll circle back to this question.)
2) What data do you have to work with?
To evaluate machine-learning tools, you need to understand what you’re going to feed them.
Critically, an answer like “customer data” isn’t detailed enough. Borne lists a number of specific qualities you should know about your data, including:
- Data types
- Formats
- Accessibility
- Quality
- Features
- Completeness
- Currency
- Lineage
To know this information, you will naturally need to talk to your IT team. Hopefully that conversation is already underway at a strategy level. This point can’t be over-emphasized. As machine learning is, under the hood, “a data and technology capability, it’s important for the CMO to partner with the CIO and Chief Data Officer to develop a strategy...to drive business outcomes,” said Isaac Sacolick, CIO at a financial services company.
“They should collaborate on selecting business use cases, evaluating solutions, and overseeing proof of concepts that can determine what technologies to pursue.“
3) What type of model will help?
If, like so many of us marketers, you weren’t a statistician or computer scientist, you might need a quick refresher course on algorithms and models:
Every machine-learning product is based on algorithms. An algorithm is just a series of mathematical steps used to transform and analyze data. And every algorithm has a statistical ‘model’ built in, or a format for the results of its work. Depending on the algorithm, that model might look like a bell curve, or a grouping of data points, and so on. (For a more detailed walk-through, see our machine learning primer for marketers.)
Before you look at machine-learning products, Borne says you should understand what kind of algorithm is likely to help address the problem at hand.
Different types of models include:
- Prediction
- Trend detection
- Segmentation
- Detection
- Classification
- Link/association discovery
A product based on a segmentation algorithm may include machine-learning capability, but that doesn’t mean it’s going to build your recommendation engine. It produces the wrong kind of model for that task.
Then, questions to ask the vendors
After you’ve answered those first three questions for yourself, you’re equipped to have productive conversations with product and service providers. Here are three questions to ask them.
1. Is this a data-consuming, algorithm-driven learning system?
According to Borne, this simple yes-or-no question is the central one for detecting machine washing.
For example, if a product works with models built using the vendor’s data, but doesn’t ‘learn’ or improve as it consumes your data, Borne says it’s not really a machine-learning system. It’s just a packaged application.
“That app may have been built based on machine-learning algorithms at the vendor's shop, but that doesn't give you access to any of its ML capability,” he says.
If the answer to this question is truly ‘yes,’ then you can ask follow-up questions to understand more about how the system works, and whether it’s going to work for you.
2. What type of algorithm or algorithms does the system use?
In your preparation, you asked yourself what kind of model or models would be applicable to your work.
So you are ready to evaluate the vendor’s answer to this question, and see if their product is the right sort.
If they won’t tell you, in at least broad terms (e.g. “We combine some kinds of regression analysis with a classification system”), something’s fishy and warrants more careful examination.
In evaluating possible partnerships for our agency, we have found that as you gain exposure to multiple vendors, you’ll start to be able to guess the types of algorithms at work yourself when you watch the demos.
3. How does the system measure and document its improvement?
If the system is optimizing its own performance, you should be able to see that.
The results might get better incrementally, or machine learning might lead you to an insight that make a huge impact. Either way, documenting the improvement will allow you to identify next steps—and justify further investment.
What’s next?
These question sets will get you started on evaluating products, and avoid getting duped by spurious vendor claims.
Our interviews with these experts also carried over into implementation, putting machine learning to work. We’ll cover that in a future column.