“Not everything that counts can be counted, and not everything that can be counted counts.”
Albert Einstein was equally prescient in matters beyond just space-time when he uttered this quote with a humorous twist on the word ‘count’. Indeed, in the realm of high-frequency trading, not everything that counts can be feasibly counted in the ever shorter duration of tick-to-trade time available for transaction. And not everything that can be easily counted in the Big Data universe actually counts in practical financial trading. Einstein’s quote seems to hint at the existence of a third possibility where we count exactly what we need counted: not more, not less.
From an economic viewpoint, both HFT and Big Data are both capital resource intensive and therefore will most certainly lead to certain evolutionary dead ends as time goes on, where the few remaining survivors face diminishing returns on their massive capital investments in state-of-the-art technology infrastructure for speed or for processing and storage capacity. For example, despite the massive investments in infrastructure, speed traders are trading fewer and fewer shares, from a high of 3.25 billion shares a day in 2009 down to 1.6 billion shares a day in 2012, according to this Bloomberg article. In addition, they are also making less money on each trade. At the other end of the spectrum, Big Data driven trading firms are locked in an arm race to provision for the unprecedented volume of business data generated worldwide, estimated to double every 15 months, requiring over 10 petabytes of storage and 100 teraflops of computing power on tens of thousands of CPUs in privately-run data centers. But how does one avoid the fate of diminishing returns in the end game, i.e., evolutionary dead ends?
Outrun by a saber-tooth tiger and out-sized compared to a woolly mammoth, the caveman hit upon a new idea for survival...
The relevant question here is this: could there be a third alternative that can be profitably sustained alongside the fast-evolving ecology of capital-intensive HFT and Big Data? A wiser choice, we believe, is to consider a financial universe that is not characterized by massive capital investment in technology or data but instead distinguished by continual process innovation. How might this be possible? What is actually involved in making this trade-off?
Recall that the overarching goal here is to Identify an ecologically diverse niche along the HFT vs. Big Data continuum that can accommodate multiple survivors with differentiated survival strategies. Both HFTs and Big Data driven trading firms are competing in their respective ecological niches that are narrowing over time, where only the speediest or the few that can process the largest volume of data become the ultimate survivors. Extinctions of firms in the long run are expected to be common under such scenarios.
Seeking an evolutionary pathway forward, driven not by speed or size but by continual process innovation.
Ceteris paribus, i.e., all else being equal, a higher ratio of conversion from raw data to harvested knowledge is certainly advantageous and should pay off handsomely in the long run. We hold constant the factors of outsourceable technology infrastructure, non-proprietary data sources, and available capital. We focus instead on maximizing the adoption and integration of tools and techniques within the trading platform, i.e., elements that accrue to overall process innovation within the firm. In this way, we are making a long-term bet on efficiency and yields made possible by the accrual of tools and techniques, and thus avoid the fate of diminishing returns in the evolutionary end games. After all, sharper teeth and claws or bigger size hadn’t helped the saber-tooth tiger or the woolly mammoth become viable in the long run. It was the use of tools, an evolving brain, plus the fortuitous discovery of fire for making raw meat digestible that ultimately made a difference to the survival of early humans. But who would have guessed a million years ago that it would be homo sapiens who now inherit the Earth?
According to Yuval Noah Harari, the author of Sapiens, humanity came from humble and marginal beginnings. On the savannas of earliest memory, human beings were third-rate scavengers, devising their earliest tools so as to better crack open bones for marrow. As Harari points out, these tools are testaments to both our ingenuity and our crippling weakness. The early humans needed that marrow because by the time they got to the carcass, the bones were all that was left. The humans didn't have the strength to compete with the lions who hunted it, nor with the hyenas and jackals who arrived once the lions had their fill. Our forebears came for that which the other predators discarded. Humanity began in the garbage; we’re but descendants of sly monkeys who worked an angle!
But what an angle that was! From our weakness on the open plain – our slow, bipedal gait, our perfunctory teeth – we soon accrue an array of strategies and tricks over historic time that enable us now to ascend to the top of the food chain.
So what does the end game look like going forward? A more efficient and higher-yielding data-to-knowledge processing pipeline would certainly be a key feature of any successful “lean data” driven trading firm, of which there will be many in such an ecologically diverse niche. All else being equal, i.e., where technology, data, and capital are held constant, we should expect to see a new breed of such “lean data” trading firms that know how to apply optimizing heuristics to move in more agile ways than Big Data driven trading firms while capturing trades over a longer time horizon than HFT firms could see. Evolution is a slow process. Only time will tell if our bid to bring domain knowledge into the search and discovery process for market inefficiencies will yield the promised riches of quantitative finance.
“The market was like a coin with a small flaw that makes it slightly more likely to come up heads than tails (or tails than heads). Out of a hundred flips, it was likely to come up heads fifty-two times, rather than fifty. The key to success was discovering those hidden flaws, as many as possible. The law of large numbers that Thorp had used to beat the dealer and then earn a fortune on Wall Street dictated that such flaws, exploited in hundreds if not thousands of securities, could yield vast riches.”
References:
- Harari, Yuval Noah (2015). Sapiens: A Brief History of Humankind. Harper.
- Patterson, Scott (2010). The Quants: How a New Breed of Math Whizzes Conquered Wall Street and Nearly Destroyed It (First Edition). Crown Business.