Just 10 years ago, finance was a small-data discipline. The small-data approach was partly due to the actual lack of data. To most investors, exchanges offered only four prices per stock per day: Open, High, Low and Close, and all of those were reported the following day (on the T+1 basis). Even the largest market makers did not store intraday data beyond what was mandated by regulators. Commodity trading floors, for instance, had only 21 days of history on hand until approximately five years ago. Finance Ph.D. programs almost exclusively taught analysis of closing prices, mentioning intraday variations only in passing.
Today, real-time streaming data is widely available. The proliferation of data is significantly changing business models in financial firms, whether in market making or long-term portfolio management. Even long-only portfolio managers nowadays add screens of data-driven signals to their portfolio selection models in order to abstract volatility and noise and realize pure returns for their investors. On the other hand, portfolio managers ignoring or under-studying the multitude of available data are adding a considerable risk to their investment portfolios.
Just acquiring the data, however, does not guarantee success. What does one do with the information? The answer often lies in years of research developed for other purposes in areas of mathematics and computer science. While many techniques for doing so have been developed in other area of science, the trickle-down into financial applications has been rather slow.
Processing large sets of information in reasonable time is another non-trivial task. The data is voluminous and cumbersome to work with. Design and optimization of financial data processing techniques is an area of active academic and practitioner study, and much evolution is happening in the space.
Some of the progress will be showcased in this year's upcoming Big Data Finance conference, a third annual conference dedicated to the newest research on the aggregation, processing and utilization of financial data to take place on March 6, 2015, at NYU Courant.
Prof. Marco Avellaneda of NYU Courant, for example, will discuss his groundbreaking research on utilizing digital image processing techniques to significantly improve options pricing. Prof. Alfred Galichon of MIT will discuss a different technique for fast interpretation of financial information, relevant to all modern portfolio managers and execution traders alike.
A separate set of Big Data problems in finance are related to data governance. Not all Big Data models need to be developed in-house and the number of vendors growing Big Data finance applications has been increasing steadily over the last few years. With external Big Data feeds come data strategies, risks for utilizing data, and policies for mitigating the said risks. At the Big Data finance 2015, Ivana Ruffini of the Federal Reserve Bank of Chicago will discuss the key strategies the Bank has deployed to keep track of various data initiatives.
Finance is no longer a small data discipline. The ability to process silos of information on the fly separate winners and losers in today's financial markets. Being aware of the latest Big Data finance tools and techniques is a necessity for every prudent Financial Services professional.
Irene Aldridge is Managing Director, Able Alpha Trading, LTD., and AbleMarkets.com, and author of High-Frequency Trading: A Practical Guide to Algorithmic Strategies and Trading Systems (2nd edition, Wiley, 2013).