The August 2003 blackout of the northeast United States was a system of systems failure, in which things didn't go as planned, alarms didn't trigger, and a lot of time of engineers in charge of the Ohio power company was wasted trying to figure out what was going on, when "three other lines sagged into trees and switched off, forcing other power lines to shoulder an extra burden," according to a Scientific American article published five years later.
What caused the power outage? In a sense, a cascade effect with kill switches not working.
For software, which is often multiple languages doing certain tasks, with linkages and interconnections, the same can be said. Software is often a system of systems.
So when Knight Capital, a hugely successful, game-changing technology company that paved the way for high speed trading got ready to implement new trading software to participate in NYSE's Retail Liquidity Program on a slow summer day, they thought they had taken every precaution deemed. But that wasn't the case.
When the trading bell rang at 9:30 am, it took less than a minute to realize something had gone terribly wrong. Worse, those in the Knight Capital data center in Jersey City, NJ, had no authority or the nerve to stop the runaway train that automated and multiplied trades at an exponential rate.
Far worse, the one man in charge who could have hit the program's kill switch was recovering from getting his knee operated the day before and was out of state. It was not an easy feat to reach Knight Capital CEO Thomas. M. Joyce that day.
If the company had to do it all over again, the implementation of the next version of their trading software, it would be resounding yes. Like most opportunity costs mixed with poor planning and bad judgment, the downside, whether in personal life or the business arena, is so catastrophic that the ability to salvage a reputation, a career or financial losses appears unreachable, as far off as a distant star in the next galaxy.
In other words, that person's next life. Just ask Jon Corzine, who is being sued for his overleveraged positions in European junk bonds after running MF Global off the risk cliff.
For Tom Joyce having a knee operation the same week of the new software rollout was a terrible decision. It was made worse by his absence. No one had the authority to step in and shut down the trading operation that ran amok. Even the president of the United States has backup help. But Knight Capital did not. Amazing.
That is made all the more remarkable as the once soaring Wall Street company had a brilliant team of 475 IT professionals with an annual budget of more than $100 million.
Steven Sadoff, Knight's Global Head of Operations, Services and Technology, had just been featured among Institutional Investors Tech 50.
Knight Capital's failure to do a "soft" launch with hard restrictions and decision-makers in place is one disaster for the record books.
Knightmare on Wall Street dust jacket (credit: Edgar Perez)
Knightmare on Wall Street
Author Edgar Perez's book is so rich in detail, so riveting, opening with a brief history of the firm and then following with the dramatic August 1, 2012, morning trading meltdown that it boggled this author's mind. With no access to Knight's management, he compiled all the facts to the trading Titanic, with its many parallel story lines, avenues of deep research, and subplots -- which savior turned suitor turned predator -- and self-published Knightmare on its one-year anniversary in 2013.
Edgar Perez's book takes the reader on a wild ride with great historical analysis and insights into what exactly went wrong. A forensic analysis showed how an old forgotten line of code, like an ex-girlfriend, showed up and turned itself on, which began instantly trading a cascade of orders. By 10:00 am that morning, the Knight Capital IT team was able to stop the program, but it continued finishing trades for the next fifteen minutes. Then they began to assess the damage.
From a platform launch with a morning cup of coffee, to the brink of bankruptcy in less than an hour, Knightmare is a book that is both a page-turner and modern historical drama with outsized egos, supersized packages, and bailouts from rivals who turn into sharks in a chum feeding frenzy.
Perez, a Columbia Business School MBA graduate and former McKinsey and IBM strategy consultant, deftly delivers a solid read on many levels, from the bailout to the suitors, from the moment of crisis to what happened to many of the former Knight Capital employees. Their reputations in the C-suite are dented, made worst by last October's Security Exchange Commission's (SEC) cease and desist order with a whopping $12 million fine.
From the SEC's introduction:
5."... as gatekeepers to the financial markets, appropriately control the risks associated with market access, so as not to jeopardize their own financial condition, that of other market participants, the integrity of trading on the securities markets, and the stability of the financial system."
6. "...Subsection (b) also requires a broker or dealer to preserve a copy of its supervisory procedures and a written description of its risk management controls as part of its books and records."
In other words, Knight Capital had no plan for worst-case scenarios in place, and as "gatekeepers" in the heavily regulated financial industry they failed to do their job.
Sitting down with Author Edgar Perez
At breakfast in a midtown restaurant, I discussed Edgar Perez's book, his second on speed trading and Wall Street, and asked him what he thought of the SEC's findings.
Edgar Perez on Bloomberg News (credit: Bloomberg News)
He agreed with all of the details and the hefty fine, and said, "Had the SEC settlement come out earlier in the summer, of course, I would have included it in the book."
Perhaps the SEC read his book, because many of its findings can be found in Knightmare on Wall Street. In fact, Perez had addressed the SEC's Market Abuse Unit in Washington soon after his first book, The Speed Traders, was published.
Perez, who is a native of Peru, is a personable, humble, and talented author. His English is breezy, succinct, uses an economy of words to get to the point, and is a pleasure to read. He knows how to hit the right beats to take this true story and give it the drama it deserves, because many lives were ruined because few precautions were taken.
A number of parties have expressed interest in acquiring the rights to this story. "With the success of Wall Street: Money Never Sleeps and The Wolf of Wall Street, no doubt there is an appetite for stories that depict the dramatic intersections of money and egos," says Perez.
His next book will be on the government shutdown last November.
I asked him when he wants that to come out, and he said: "Before the mid-term elections. So people can read it, be more informed on what happened, and when they vote make a more valued decision."
In other words, learning about the people and their personalities behind the Senate and the Congress, not to mention the White House, should make for an interesting read.
The working title for his new book is The Eleventh Hour and hopefully he will get people in the Beltway to talk -- because if there is anyone who can frame that political gamble by the fighting parties, it's Edgar Perez.
For those who enjoy reading on the very human failures of Wall Street and the intersection of money and technology, then Knightmare is the book for you.