Creating Breakthrough Innovations Through Design With Big and Small Data Analysis

Co-written by Finn Birger Lie


For decades, Business Intelligence (BI) and, more recently, Big Data Analytics (BDA) has been successfully applied to create incremental innovations. These incremental innovations have been applied in such diverse areas as customer satisfaction, human resources, process management and merger & acquisitions. However, to date, the use of data has failed to provide new insights for breakthrough innovation.

How might BDA, together with Small Data (SD) be integrated early enough in the new product development process to create real breakthrough innovation and thus provide a competitive advantage?

For any data analytics to be useful for breakthrough design, it is important to understand the design processes it is intended to support. The conventional design process applies a series of steps combining analysis and synthesis, prototyping, testing and learning to create unique and valuable insights.

More advanced design processes, such as Design Thinking, include an element of design research (up front user, competitor or inspirational research) or Design Science Research (up front or concurrent modeling of the challenges and performance as needed.) These methods enable design teams to move from a problem-solution focus to reframing challenges-opportunities.

In doing so, design teams gain a better understanding of the current and future market, and technologies allowing them to see how their organization can best leverage this knowledge. The design teams can then create a roadmap that includes the concurrent building of new capabilities that enable the design of future offerings.

Innovation is often ambiguous. The Market - Technology Risk Matrix provides a useful mapping of new ventures and offerings according to their position in the market (Recognized Needs, Clarifying Needs and Realizing Needs) and their technology level (Current Technology, Applied New Technology and Development of New Technology).

Different combinations of Big Data Analytics (BDA) and Small Data (SDA) analytics may prove more productive, depending on where design identifies insights within in the Market - Technology Risk Matrix.


Utilizing grounded research and vertical thinking; BDA can support incremental innovation, optimizing an opportunity by identifying and describing dependencies and devising decision-making rules. On the other hand, through lateral thinking, SDA can utilize a combination of hypothesis and grounded research to support breakthrough innovation. With experts bridging their knowledge though brainstorming, they can effectively reframe the opportunity and identify new connections between domains. This assists in identifying innovative opportunities, which are then verified though SD experimentation. Thus, together, Big and Small Data can define and optimize new opportunities.

The Design Index and its internal parameters grouped under Strategy, Context and Execution provide especially useful metrics in assessing design performance in the various areas of the Market - Technology Risk matrix.

In conventional incremental innovation (characterized by the Design Quality Criteria: Performance and Expression) parameters and connections are relatively fixed and optimization takes place. Big Data can be useful in uncovering correlations between customer experience, sales and marketing efforts, manufacturing and logistics and an appropriate Big Data technology for describing and prescribing is machine learning.

Addressing breakthrough innovation (characterized by the Design Quality Criteria Context, Social/human and Innovation,) Small Data analytics becomes more useful when used together with humans framing the opportunity and solving the challenge.

Breakthrough innovation is the most challenging of the two innovation types and calls for uncovering latent needs, or even creating needs and meaning. An appropriate approach would be design science research, applying simple statistics such as Regression Analysis, Comparison of Means, Clustering Analysis and Multi Variable Linear Regression Analysis. Using these methods on quick field experimentation of user characteristics, such as user behavior, one can devise descriptive, prescriptive and predictive performance along various metrics and from there, formulate rules for making decisions.

Until now, breakthrough innovation has been seen as more of an art than a science, made possible by individual ingenuity, "gut feelings" and mostly attributed to luck. However, by applying Big and Small Data Analytics in combination with Design Quality Criteria, design teams can now speed up exploration time and the creative process, increasing the likelihood of identifying the key insights on which breakthrough innovations are based.

Special thanks to Finn Birger Lie for researching and co-writing this article.