In the seminal book “The Four Steps to the Epiphany,” Steve Blank introduces the concept of “customer development” ― get out of the building and interview customers. While this is not a new concept ― product people with user-centered design training have always done this ― this is a huge development in startup-land, where technology used to run amuck.
Challenges with sample size
There is one small problem with customer development. It relies on qualitative research techniques like detailed interviews and observation, which are time consuming and costly. Additionally, these techniques involve deep interactions with a few individuals, and you always run the risk of talking to the wrong people about the wrong problems.
How do you know whether you can trust your results? One way is to increase sample size - but given each interaction can take a couple of hours all-in, trying to get to 100 conversations quickly becomes daunting.
Later on in the product’s life cycle, prototypes will need to be tested. Again, qualitative research techniques like usability benchmarks and observation are the best way to start. Right away we run into the same sample size problem.
A two-step approach: In-person, then remote
There is a way forward that will start you off with the qualitative insights you need, and end up with a big enough pool of data to make it credible. Start with in-person sessions, then scale it up with remote sessions.
Here are two quick examples.
From interviews to an on-line survey for problem research
Start with 20 interviews to validate hypotheses
Develop a solid persona for these target customers
Come up with 5-10 questions you want a larger group to answer
Run an on-line survey with 500 people and chart the results
From in-person testing to crowdsourced testing for solution research
Start with 5 in-person product testing sessions
Fix glaring problems
Check your homework with another 5 in-person sessions
Now engage a crowdsourced testing service like User Testing, User Think, or run your own DIY testing using Amazon’s Mechanical Turk’s on-demand workforce.
Crowdsourced research platforms for product research
There is a reason why crowdsourced testing is so attractive. For a very reasonable cost (under $100 per test in some cases), these platforms connect teams with large panels of on-demand testers, solving the subject recruitment problem and saving time and money.
There is one thing that new product teams need to keep in mind. Crowdsourced testing platforms are designed for solution research. You have to have a product for them to test first. You still have to lead with problem research the old fashioned way. You have to do both - or fall into what Bill Aulet, Managing Director of the Martin Trust Center for MIT Entrepreneurship, calls “our dangerous obsession with the MVP”.
Doing without the two-way debrief
Another thing you get from doing in-person qualitative research, versus crowdsourcing the testing, is that the latter is one-way: The tester provides you with their feedback, but you don’t get to debrief them and learn more with the back and forth.
Let’s say you are testing your product with a user and she gets stuck. You can help her move past this task to complete the rest on the list. Later on, you can probe deep into what happened during the debrief. None of this happens in a crowdsourced test - if the user gets stuck, she stays stuck. Crowdsourced feedback is still valuable - just not as a first line of defense.
When to go face to face, and when to crowdsource
There is a time and place for every research technique. In-person sessions are the best way when the uncertainly is high. Crowdsourced research is best for testing well understood things. They both have a place in a robust, ongoing research program.
To recap: Lead with in-person sessions, and then transition to a scalable, faster and cheaper alternative. That is the path to the epiphany.
Elaine Chen is a startup veteran and product strategy and innovation consultant who has brought numerous hardware and software products to market. As Founder and Managing Director of ConceptSpring, she works with executives and leaders of innovative teams to help them set up and run new product innovation initiatives with the speed and agility of a startup. She is also a Senior Lecturer at the MIT Sloan School of Management. Follow her at @chenelaine.