A key characteristic of a good forecaster is what Tetlock calls an “outside view”, which refers to the ability to detach yourself from a given situation, and think on purely objective ground. In order to illustrate this, let us consider the following example. Suppose you see a happy couple who seem to love each other very much. Someone asks you, “What is the chance that the couple may get divorced?” How will you response to that?
Normally people would say, “They are a happy couple. They are never going to divorce.” This is what Tetlock calls an “inside view”, because you are putting yourself into the couple’s perspective, and make a judgement based merely on your subjective impression.
A good forecaster, on the other hand, would take an “outside view” that, in order to assess the probability of divorce, he will gather data about the couple, like their income, their personality test results, etc. to come up with an objective estimate based on past occurrence.
Another interesting point mentioned is the diversity of data. Suppose you ask three experts whether a couple will divorce in the next three years. Expert A says there is a 70% chance. Expert B says there is a 70% chance. Expert C also says there is a 70% chance. Should you believe that there is, on average, a 70% chance that the couple will divorce?
Well, it depends on how the experts came to the conclusion. If all three of the expert are looking at the same set of data and using similar methods for prediction, then it is likely that the crash is just 70%.
However, if they are using three completely different sets of data and methods, and they do not talk to each other before making the predictions, then it is a completely different story. Say, Expert A uses personality tests, Expert B uses financial analyses, and Expert C interviews their friends and families. If this is the case, then the chance of a divorce will be higher than 70%, because there is more evidence pointing to the same direction.
Lastly, the author also discussed the important question of whether a superforecaster could be simply lucky. Famous fund manager Bill Miller, who outperformed the S&P 500 for 15 straight years (1991 to 2005), devastated his fund in the 3 years afterwards. Therefore, just because someone has been making good calls in the past, it doesn’t mean that his calls would continue to be good. To put it in another way: if I manage to get 15 consecutive heads by tossing a coin, it does not mean I am good at coin-flipping, and certainly not that I am going to keep getting heads. The reason is simple: if there are a large enough number of people flipping coins, then there has to be one or two lucky fellows who happen to get 15 heads in a row by pure accident.
To this, Tetlock acknowledged that each historical event is unique and it cannot be re-trialled for testing purpose. If an event is predicted to have 90% chance to happen, but it does not happen, it does not mean the predictor is wrong, because there is still a 10% chance of not happening. Thus, there is no sure way to ascertain the correctness of any estimation. To be a good forecaster, one has to be “ego-resilient” (i.e. not afraid of making mistakes), because no one could be 100% correct all the time.
In all, this is an excellent treatise on the art of precise thinking in our chaotic world. Professionals who deal with uncertainties, like poker players and financial traders, will probably love it.
Reference: Tetlock, P. E., & Gardner, D. (2015). Superforecasting: The art and science of prediction. New York, NY: Crown Publishers.
No comments:
Post a Comment