Silicon Valley is beginning to sound more like a scene torn from the pages of a fairy tale than the tech hub of the world.
Where else but Silicon Valley do "Unicorns" munch lazily among Redwood cities, venture capitalists spout fountains gold promising everlasting returns, and entrepreneurial princes try their hand at pulling cash from stone. A land where dreams become reality with each exit and acquisition. It has heroes and villains; princes and princesses. It is a wonderfully bizarre place.
Of course there would be no fairy tale without a suitable King. In Silicon Valley, its name is Big Data.
Big Data has revolutionized many, if not all, of today's major industries. Companies are using data science to glean insights from the large customer data 'lakes' they have collected over the years. Nearly every business decision made today is informed, in some way or another, by observation gathered through the analytics and sciences associated with Big Data.
These processes have brought about new efficiencies and made use of data formerly neglected. Yet as our use of this technology grows, so too does our dependence, begging the question: is our faith in Big Data systems misplaced? Is 'King' Big Data clouding our judgement?
Across the Silicon Valley kingdom, one does not have to look far to find industries that have been transformed by Big Data practices. One such example is an industry of great interest to me, finance -- and in particular, lending.
Financial institutions large and small have been employing data science -- often referred to as "credit analytics" -- to aid in loan underwriting for a long time, well before FICO became a household name.
In many ways, banks are the original data-driven sector. As far back as the 1840's, the first credit reporting firms pioneered elaborate financial networks to analyze the financial activity of American business owners. Over time, these data-driven metrics have been further refined in order to provide a clearer view into a borrower's creditworthiness -- a crystal ball fit for a King.
As with all technological advancements, however, there have been challenges.
With so much confidence placed in extracting meaningful insights from data sets, it has effectively narrowed the industry's scope. For example: when lending decisions are derived from a single consumer data pool, as they typically are today, borrowers are assessed in isolation allowing only select financial variables to be considered. Meanwhile, signals with long-proven significance -- such as the community, or "personal social network," surrounding a borrower -- go unobserved. Signals that could supplement existing data and create better Big Data.
As a result, this industry-wide focus on isolated individual datasets has left more than 40 million Americans without access to affordable loans simply because their numbers don't look good on paper.
It is a cautionary tale for all industries who use data science to aid in decision making. Allowing precedence to guide analysis or allocating resources to only dive deeper into a single data 'lake' can divert attention from other potentially useful data sources. By contrast, promoting experimentation and exploration into new data sources can lead to a more complete, productive use of Big Data practices.
For the financial industry, a more accurate lending decision is possible by incorporating a new set of inputs: social connections.
Once a primary factor in lending decisions, the industry's model for capturing this data has broken down in recent decades as the scale of banks has grown astronomically large. In moving towards the digitized world we live in today, lenders lost the knowledge and ability to access the informative social data surrounding borrowers.
Today, however, with unprecedented levels of digital connectedness exhibited by modern borrowers, it is now possible to rediscover the 'finger lakes' of data found within social connections. In the same way community banks of days passed factored social data into their decisions, technology can bring this wealth of data back into focus.
In the land of fairy tales and unicorns we mustn't be lead astray by Big Data. Yes, Big Data is a good thing, but it must be guided with everyone's best interests in mind. There must be careful consideration into what data is being used and when beneficial, the adoption and creation of new data sets.
As more and more industries grow to rely on Big Data in decision-making, we should carefully examine the sources of Big Data and whether they reflect the entirety of a given situation, the complete picture. Valuable insights go unrecognized simply because no one ever thought to consider different, better data.