In the Spring of 2013, a lively discussion on LinkedIn went like this:
- “If someone says ‘Big Data’ one more time, I am going to throw up”, declared Head of Marketing at a prominent software firm
- “Agree, ‘Big Data’ is such an annoying buzzword”, chimed in Head of Research at a mid-tier broker-dealer
- “Uh, it’s such a fad”, stated a well-funded hedge fund manager.
And so it went: “Big Data” is annoying, fleeting, and, by implication, useless. Fast forward to today, even though Big Data is a much more established term, eyes still roll when the subject comes up. And the key issue is still the same: okay, we’ve got data, so what’s the point?
It turns out that the point of Big Data is not about having a ton of data, although data is a prerequisite. Big Data comprises a set of analytical tools that are geared toward fast meaningful processing large data sets. Meaningful is an important keyword here: Big Data analytics are used to derive meaning from data, not just to shuffle the data from one database to another.
Exactly what kind of techniques do Big Data tools comprise? First and foremost, Big Data techniques are geared toward analyzing a more manageable amount of data that still represent the entire data set. While working with a huge data set can be fun and result in precise inferences, it can also be a very slow and painful process. Big Data helps researchers probabilistically identify the aspects of the data that are most important, allowing researchers to discard some data while sacrificing little or no accuracy. Since most financial researchers are still trained to study low-dimensional data (say daily, or very structured intraday bars), Big Data Finance techniques deliver an immediate gratification in reducing the amounts of data required to manipulate.
A second application of Big Data is finding the most important factor influencing a particular set of behaviors. For instance, we can use Big Data to evaluate the key drivers of stock price behavior common to all stocks. That driver turns out to be the movement of the aggregate market, which can be proxied by the price of the S&P 500 ETF or beta of each stock (for more details, see recent research by Financial Mathematics Professor Marco Avellaneda of NYU Courant, http://www.math.nyu.edu/faculty/avellane/). Due to the ever-sophisticated algorithmic linkages among various financial instruments of all asset classes and ETFs, a slightest perturbation in one stock may indeed be felt across the entire market and distributed back to each individual stock! Likewise, Big Data is used to automatically “read” (process) streaming news and figure out what is wheat and what is chaff as it pertains to stocks on the fly.
A third application of Big Data is finding natural affinity groups of similar stocks or potential substitutes. The concept of similar stocks is a key driver of trading strategies such as statistical arbitrage. With Big Data Finance, statistical arbitrage delves in one layer deeper, letting mathematics identify the stocks that should be tracked and arbitraged. Finding stocks that are potential substitutes is also critical for portfolio managers and execution traders alike: in low-liquidity conditions, it may be critical to find a liquid substitute to execute a quick trade, and then rebalance the portfolio to its target composition at the trader’s leisure. (Yes, Professor Avellaneda, http://www.math.nyu.edu/faculty/avellane/, has written on this subject too.)
Big Data is used for pattern recognition of trading strategies, for instance, by AbleMarkets (http://www.AbleMarkets.com) to detect participation of high-frequency traders and institutional investors in the markets. Big Data is also used in the reconstruction of lost data (say, vanished en route to a trader’s desktop during an Internet outage), even catching errors in algorithms – can all be done and is done today with Big Data Finance analytics. The techniques are numerous, established, trusted and true. To the surprise of many, Big Data can be traced back to 1901, when a noted statistician Karl Pearson developed Principal Component Analysis (PCA), a technique widely used in today’s data analysis.
Are you using Big Data in your financial analysis? If not, it is about time to start. In today’s reality, summarized in “Real-Time Risk: What Investors Should Know About Fintech, High-Frequency Trading and Flash Crashes” (by Irene Aldridge and Steve Krawciw, Wiley 2017, https://www.amazon.com/Real-Time-Risk-Investors-FinTech-High-Frequency/dp/1119318963), one thing is given: either you consume data, or the data swallows you.
Where can one learn how to use Big Data Finance techniques? A great place to do so is the annual Big Data Finance conference (http://www.BigDataFinanceConference.com/BDF2017), which will take place on May 19, 2017, at New York University. In its fifth year, the conference gathers together Big Data academics, practitioners, and even regulators to exchange ideas and latest techniques to benefit all. As Gene Schupak, PhD, Head of Research at Suite LLC, a company with cutting edge analytics for fixed income pricing and a sponsor of Big Data Finance, says, “The precision offered by Big Data techniques is critical for success in today’s exceedingly automated finance environment. We see a higher and higher demand for multi-dimensional analytics, such as Suite LLC software, especially in fixed income, our area of expertise. The conference is one of the very few events we support because of the timeliness and the value of the forum the conference creates.”
Irene Aldridge is Managing Director, Head of Research at AbleMarkets, a Big Data for Capital Markets company, specializing in real-time and near-real time Software-as-a-Service improving execution, portfolio allocation and risk management. She is a co-author of “Real-Time Risk: What Investors Should Know About Fintech, High-Frequency Trading and Flash Crashes” (Wiley, 2017, https://www.amazon.com/Real-Time-Risk-Investors-FinTech-High-Frequency/dp/1119318963), and an author of “High-Frequency Trading: A Practical Guide to Algorithmic Strategies and Trading Systems” (Wiley, 2nd edition, 2013, https://www.amazon.com/High-Frequency-Trading-Practical-Algorithmic-Strategies/dp/1118343506).