Human beings have been curious species throughout the course of evolution. From the times unknown, we have been trying hard to get a sneak peek into the future. Let us travel couple of centuries back in time to understand the basic constructs of prediction. We have largely depended up on Heuristics, Logical reasoning and past trends to foresee the outcome(s) of any given event. The ‘wiser’ ones of the community could interpret the available evidence and connect the dots to predict the future. These individuals were known as Prophets, clairvoyants or seers who could see through the future and were revered as demigods. In practice, these individuals ideally followed the simple Regression framework, I.e., to Identify a prediction variable, understand its environment, identify all the other variables which can have a systemic impact on it and try to derive the future outcome based on the past trends. The success or failure of the prediction depended upon accurate and comprehensive identification of these variables. This elite wisdom is what distinguished these individuals from the common pool.
One can acknowledge, categorize and reduce uncertainty but cannot completely eliminate it. This gives rise to scope of error and these wise people have always learnt from previous inconsistent predictions made by themselves or their community. They were also aware of the fact that the Impact variables could surface from any environment and could also belong to completely different system altogether. Moreover, these were very dynamic by nature and moved in and out of the prediction logic or equation. This basic framework could be applied to any kind of prediction, let it be Predicting the future of an individual based on movement of the stars or predicting whether an incident would take place based on factual evidence.
Let us look at the possible thought process of a seasoned astrologer behind predicting the future and connect it back to the above framework. Science of astrology interprets future outcomes based on behavioral impacts that the movement of stars or planets would have on a particular individual. An Astrologer’s frame of reference and the bounds of environment are gathered from two legs of factual evidence. The first leg advocates the fact that human brain is a simple transceiver, that generates and decodes electromagnetic and electro chemical impulses and the way we act has a strong bearing on the anomalies observed in these signals. The second leg of evidence is derived from the factors that can plausibly manipulate these signals- “The Cosmic forces” being one of them. This could also be traced back to etymology of the word “Lunatic”- The root of this word is luna, which means ‘of the moon’. This brings us to define the environment this community looks at: “Individuals, their psyche, their actions (leading to future outcomes), cosmic movements and their impacts on the individual’s actions”. Determining behavior of an individual based on his current status, past actions bring in the elements of time series-auto regression methodologies and effect of external factors such as cosmic movements bring in cross sectional and panel data. Now that the prediction environment has been established, astrologers now try to draw a regression line mentally based on the conventional wisdom inherited from their community. This practice has evolved a lot since its inception and the methods of prediction have also changed gradually.
Similar analogies could be drawn with respect to other nonscientific prediction techniques like Palmistry, Tarot card reading etc. where each of these techniques has its own logical reasoning backing the methodology. Traditional business decisions or outcome estimations till recent times were no different and were based on Heuristics, patterns or guesstimate of the managers. The major advantage we have over our previous generation for Identification and measurement of these impact variables is advancements in the field of technology. Now we can do the same in a more systematic and scientific way by leveraging on statistical modeling techniques, vast data sets, machine learning and powerful processing capabilities of computers and take the prediction to unprecedented levels of accuracy.
Let us look at an example: To determine the probability of a ‘Tail’ outcome on flipping a fair coin. Empirical records conveniently calls out the probability as 0.5, i.e., assigning an equal chance to each binary outcome. What if we determine all the factors that affect the outcome? Given that the weight of the coin is constant, let us assume that a person flipping the coin has a sensor band on his index finger, which can map and determine “position of the thumb’, “coin placement”, “force of the flip”, “angular momentum of the flip” and “gravity at the place of flip” and notes the outcomes over several trials. We can constantly evolve the model by identifying the missed out variables like “direction and strength of the wind” and eliminating the ones with spurious correlation to increase the accuracy.
Every system has impact variables that are external as well as internal to its environment and some of these could be controlled and some can’t. Success of prediction lies in understanding the dynamic environment, learning from error and constantly improving the estimation model with a systemic shift in the frame of reference whenever necessary. Hence, in order to forecast outcomes for a business or an individual accurately, the modern day ‘Data prophets’ (popularly referred to as ‘Data Scientists’) can’t rely on static prediction models anymore and need to equip themselves with right kind of technology to know “when their stars move”.