As companies have decentralized strategic and innovation activities, promoting agility, they have become increasingly hospitable to catalysts -- these mission-driven leaders who corral corporate resources that are outside their traditional span of control to address sprawling challenges. They form networks or coalitions within and outside the company and are motivated by the desire to solve big--often global--problems." - Scott D. Anthony, "The New Corporate Garage"
Innovation has evolved in business. Specifically, 1997 to present day is marked by a transition from technical innovation to innovations in business models. Companies like Starbucks, Apple and Amazon are the champions of this new era. The recent surge in VC and federally backed Big Data startups is reflective of this new period of innovation. Acquiring, organizing, verifying and repurposing data sets is far from a new or innovative business concept. Data is everywhere and attainable, but acquiring data sets and turning them into invaluable business assets is where the true innovation occurs. The Worldwide Web is teeming with information ripe for the picking. Big Data companies' ability to attain necessary data at a cost that allows for a profit is what determines their viability. Human workforces in the form of scientists, outsourced work teams and crowdsourced labors ultimately bridge the gap between VC backed nightmares and unlimited profitability. Without efficient and cost effective resources like outsourced labor forces, Big Data companies would never achieve innovation in business model and consequently worth to modern consumers that crave information.
It all began with punch cards...
In 1889, Herman Hollerith patented the Electric Tabulating Machine. This was a particularly important event as it was one of the first machines designed to process large amounts of data. Machine-readable punched cards spawned what would later become I.B.M. and symbolized the beginning of a new industry as Hollerith used them to complete the 1890 census (population 63 million). Fast forward, the United States' population has skyrocketed to more than 311,500,000 and Big Data is one of the hottest industries. Long before Steve Jobs and his gang of super nerds were able to create the Apple I companies were geeking out over ancient devices like this.
The Internet, Big Data's last frontier...
It's growing. Indescribable. Indestructible. Nothing can stop it! With over 400 million tweets per day, more than 1 billion photos added to Instagram and 995 million monthly active users on Facebook, data is growing at an alarming rate! Social networks like Facebook have created some of the largest databases of consumer information as they continuously collect, store and organize information created by their global user community. The Internet serves as the world's foremost reserve of data, but our current attempts and capabilities to effectively mine and analyze data is dwarfed by the size and scope of the Worldwide Web's Big Data possibilities.
Data Scientists are sexy...
Yes, you read that correctly. Data Scientist has recently become one of the hottest career fields. Companies like Target pay upwards of $150,000 for data analysts that design algorithms to predict consumer spending habits. In fact, Target's ability to accurately determine whether or not a female customer is pregnant led to a dramatic increase in their maternity market revenue and a few angry parents along the way: http://www.forbes.com/sites/kashmirhill/2012/02/16/how-target-figured-out-a-teen-girl-was-pregnant-before-her-father-did/. Data Scientists are in great demand right now and there is nothing temporary about this phenomenon.
Social Scientists are sexier...
Nobody likes a "one-upper." While Data Scientists toil over the spending habits of the American consumer, Social Scientists track and graph patterns on a much larger scale. Their objective is to reveal sociological laws of human behavior in hopes of predicting political crises, revolutions and other social & economic instabilities. Social Scientists utilize algorithms to analyze publicly accessible data from Web search queries, blog entries, Internet traffic flow, financial market indicators, traffic webcams and changes in Wikipedia entries. On a personal note, this is both interesting and terrifying to me. "Defense network computers. New... powerful... hooked into everything, trusted to run it all. They say it got smart, a new order of intelligence. Then it saw all people as a threat, not just the ones on the other side. Decided our fate in a microsecond: extermination." OK, so that was a quote from Terminator, but you understand what I'm hinting at (this blog post written from my Doomsday bunker).
Current human role in Big Data: analysis, input and classification...
Data & Social Scientists utilize Big Data to track consumer information and create complex algorithms to predict future behavior, but are you aware of the other roles humans play in taming Big Data? Innovative companies like Instagram, Facebook and Twitter have drastically changed the way people connect and use the Internet. With those changes, there are new troves of data created. Humans are not an important part of utilizing new data, they are single most important part of the process. For example, the CIA utilizes crowdsourced labor to analyze millions of images taken by satellite of areas under surveillance. Workers log-in online and scan the satellite imagery for evidence of human rights violations, terrorist activity, weapons of mass destruction and other pertinent information.
With billions of dollars recently invested in Big Data startups, the race to obtaining and optimizing unique data sets has presented new opportunities for outsourced workforces. Having the database is no longer enough. For example, rewording and repurposing product descriptions for SEO purposes has become a necessary process for e-commerce companies. Largely, people serve as the necessary cog that transitions data from a collection of information to an invaluable set of insights which will transform your business. For instance, many successful startups like Hoppit have been created in hopes of refining and reusing existing databases to provide new services to consumers. Hoppit utilizes a combination of complex algorithms and outsourced teams to provide users with a search engine that serves as an extension of Yelp. Hoppit's founder, Steven Dziedzic, started the company based on a perceived need to classify restaurants and bars by mood and ambiance. Complex algorithms scan Yelp reviews in search of keywords that reveal moods/ambiance, while outsourced teams comb the Web in search of relevant photos and information that algorithms are not able to find. The result is a new experience, consumer friendly and a complete reuse of an existing consumer information database (Yelp). As companies like Hoppit rapidly multiply in numbers, crowdsourced and outsourced workforces will become vital to the process of creating verified and optimized from existing imperfect and disorganized information sources. Despite tremendous advances in technology, the ability to think subjectively is a uniquely human attribute. Eat it Watson.
By the numbers...
A recent poll conducted of 600 executives revealed:
- 9 of 10 see Big Data as being the fourth factor of production (Land, Labour, Capital and Big Data)
- 26% of companies felt using Big Data has significantly improved their business (expected to grow to 41% within the next three years)
- 60% of companies expect to invest dramatically in Big Data within the next three years
- Between 5-6% productivity gains are seen at companies that utilize decision-making powered by Big Data
The future is here...
Regardless of technological advance, the evolution of Big Data will be directly related to human analysis at all levels. After all, the data we are most concerned with presently is that which is created by humans. The revolution is not coming, it's already arrived. A company's success is no longer dependent on their technical innovation. Rather, the most successful companies will be the ones that are able to utilize Big Data in a manner that is cost effective and derives the greatest value. He who haveth the best business model, haveth the market share. Remember this, it's absolutely impossible to utilize Big Data by solely employing in-office workers. Big Data has seen a similar revolution to that of manufacturing. Taxes, high wages and unions encourage American companies to seek alternative offshore manufacturing solutions. Similarly, employing an in-office employee to analyze thousands of images is financially impossible. Turning to outsourcing and crowdsourcing is not a good option, it's the only option. We only have a few hundred years left before machines will run the world. Or, so I was told by a deranged fellow in Times Square. Big Data can be obtained and used for a price. Employ the correct methods for taming the beast that is Big Data and market domination becomes a real possibility.