The Paradox of Prediction: Why We Can't Stop Forecasting What We Can't Know

Prediction delivers a dopamine hit that few activities match. When you call a market move correctly or movie’s success on the first day of release, the pleasure is neurochemical reward from pattern-recognition machinery honed over millions of years.  This explains why sports betting thrives despite unfavourable odds, and why millions choose individual stock picking over index funds. 

Daniel Kahneman's research reveals why we remain confident despite poor track records. Our fast, intuitive thinking generates compelling narratives that feel true, while our slow analytical system fails to scrutinise them adequately. We construct coherent stories from limited information, mistaking narrative consistency for predictive accuracy. This overconfidence persists because we remember successes vividly while failures fade into background noise. 

 When Prediction Actually Works
Before dismissing all forecasting as futile, we must acknowledge genuine successes. The distinction lies in system complexity and time horizons. When meteorologists predicted Cyclone Phailin would strike Odisha in 2013, India launched its largest evacuation. 

The cyclone made landfall with devastating force, yet fewer people died compared to a 1999 cyclone that killed thousands in the same region. Japan's 2011 tsunami warning system provided only minutes of notice, but in Kamaishi, a coastal city in Japan, most of the schoolchildren survived despite 40-metre waves.  NASA's asteroid tracking demonstrates prediction at its most fundamental. We've mapped 90% of near-Earth objects exceeding one kilometre. If one targeted Earth, we'd  know decades ahead, as NASA demonstrated in 2022 by successfully altering asteroid Dimorphos's orbit using DART.

The pattern is clear: short time horizons, simple systems, physical rather than social dynamics, and probabilistic rather than specific forecasts work. Weather predictions work  in 24-48 hours horizon but fail beyond a week. Market psychology involves millions of  adaptive agents making billions of decisions. The typical response is equally good learnt  through painstaking assiduous research and building immune systems.  

Philip Tetlock's research identified rare "superforecasters" who predicted complex  outcomes better than chance. They thought probabilistically, updated beliefs  incrementally, sought contradicting information, and broke complex problems into  components. Even these superior forecasters showed advantages primarily in shorter  timeframes. Notably, the more famous the expert, the worse their predictions. In this  case, the adverse consequences of incorrect predictions with higher magnitude of error  would be severe.

Forecasting Failure
Financial markets are prediction's graveyard. Long-Term Capital Management, staffed by  Nobel laureates, collapsed in 1998. Lehman Brothers' risk models assigned the 2008  housing collapse a minuscule probability. This event devastated the global financial  system and exposed the Achilles' heel of the stress-testing models. 

Nassim Taleb's Black Swan events explain why financial models fail catastrophically.  These high-impact, low-frequency occurrences lie outside normal expectations, have    extreme consequences, and are rationalised in hindsight. Financial institutions build  models based on historical data, creating an illusion of understanding while remaining  blind to what they haven't observed.

Consider monthly US Non-Farm Payroll predictions. Consensus forecasts from dozens of  professional economists miss by hundreds of thousands of jobs. Even the Federal Reserve  projections are revised constantly. The Fed's 2021 inflation forecasts famously described  rising prices as "transitory", which was proved spectacularly wrong as inflation soared to  40-year highs.

Earnings guidance has become the cornerstone of corporate briefings. While essential for  communication and managing expectations, its short-term focus can undermine long- term value creation. Many times, despite strong numbers, markets behave surprisingly,  increasing scepticism about predictive power.

The two famous long-colour streaks of multiple reds or blacks in a row most often cited  are 26 blacks at Monte Carlo in 1913 and 32 reds in a US casino in 1943. But both are  essentially anecdotal and lack a clear primary, verifiable casino log. Fortunes were made  and lost during these events because the roulette wheel has no memory. While current  day’s gold or cryptos current rally is truly exceptional and historic in its scale, the rally  qualifies to be termed as unprecedented winning streak in recent history. Predictions  about where this rally leads are equally heady. 

Random Walk Reality
Stock prices follow what economists call a "random walk." Each price movement is  essentially independent of the last, influenced by an unknowable combination of  fundamentals, psychology, positioning, liquidity, macro events, sentiment and pure  chance. You cannot consistently identify which prices are wrong, in which direction, or  when they'll correct.  

Bull or Bear markets run far longer than rational analysis suggests. Stocks that fall 80%  can fall another 80% from there. The market has no memory, no sense of fairness, no  obligation to mean-revert on your preferred timeline.: It’s worth recollecting Keynes's  1923 remark "Markets can stay irrational longer than you can stay solvent." 

The greatest protection against unpredictability comes from preparing for multiple  futures simultaneously. In financial terms, one cannot avoid risk but must measure and  manage it. 

Warren Buffett avoids forecasting market swings, calling such attempts futile. Instead, he  invests in companies with lasting competitive advantages, solid finances, and strong  leadership—preparing for any outcome without predicting it.  

JPMorgan Chase maintained fortress balance sheets that looked inefficient during good  time, but when the 2008 crisis hit, they acquired failing competitors. Meanwhile, Lehman  Brothers had models for specific scenarios and when an "impossible" scenario came up,  they had no backup plan. 

Taleb's concept of antifragility extends beyond resilience. Antifragile systems benefit  from volatility. In investing, this means structuring portfolios with limited downside but  unlimited upside potential. The barbell strategy which combines extreme safety with  aggressive but limited risk-taking, acknowledges we cannot predict which tail event will  occur while positioning to benefit if positive surprises materialise. 

Practical Wisdom
Wisdom lies not in predicting better but in relating to predictions differently.  Superforecasters succeed through superior habits: keeping score rigorously, breaking  problems into components, considering alternatives, updating incrementally, and  recognizing the limits of knowledge. Think in probabilities rather than certainties. 

Predict but be prepared for nasty negative surprises.  Let prediction satisfy curiosity, but  let preparation protect your future. Watch the markets, make guesses, enjoy when you're  right—but build your portfolio assuming you'll be wrong. 

Kahneman reminds us that we are pattern-seeking creatures living in a partly random  world. We will continue seeing patterns that aren't there and maintaining confidence  despite evidence of limitations. The solution isn't to eliminate these tendencies—that's  neurologically impossible. The solution is to build systems and strategies that succeed  despite them. 

The only reliable prediction is this: you will be surprised. The greatest leaders built  strategies that succeeded despite not knowing what would happen next. They respected  uncertainty rather than claiming to eliminate it. In a world of unpredictability, the only  rational response is to prepare for disorderly adverse developments caused by large errors.  

BasisPoint Briefing — Explainers from seasoned practitioners for readers who want more than the basics.