Assume that you are considering investing $700,000 and that this is
a large sum for you. Experts give you the names of two reputable
investment houses of long standing. When you visit the first of these,
Benjamin Company, Inc., the manager recommends Bailout Bonds as an
excellent investment for you. He gives you good reasons for his
recommendation and reassures you that he is very confident in his
advice, which is based on the analysis of top experts. He urges you to
act swiftly to avoid missing out.
As a cautious investor, you visit the other investment house that
was recommended to you, Franklin Company, Inc., their manager advises
you that Bailout Bonds would be a poor investment and that you would be
better off holding on to your money than putting such a large amount
into such a speculative investment. Like Benjamin’s manager, Franklin’s
manager provides good reasons and she too is confident in her advice.
Given the conflicting advice, you decide to check the track record
of each of these investment houses. Fortunately, numerous academic
studies have been published on the accuracy of each firm’s investment
recommendations since 1930. Each firm retains the advice of some of the
best experts in the world. You find that on average, the Benjamin
experts have been right for 50% of their forecasts – and wrong on 50%.
Interestingly, the Franklin experts have exactly the same performance
What would you do? Our guess on what most small business people would do is to skip this investment opportunity.
The year 1930 is roughly the date of the first studies on the value
of expert forecasts that relate to complex and uncertain situations.
Contrary to popular opinion, the findings are consistent with our
Benjamin versus Franklin story. For example, Philip E. Tetlock’s 2005
book, Expert Political Judgment describes how he recruited 284 people
whose professions included “commenting or offering advice on political
and economic trends.” He asked them to forecast the outcomes for
various situations. By 2003, he had accumulated 82,361 forecasts.
The conclusions from the studies are that:
1) expert opinions are useless for forecasting related to complex and uncertain situations.
2) expertise does not help; College students do as well as seasoned experts at such forecasting.
3) experts’ statements about confidence have virtually no value.
Indeed, if you put a group of experts in a room and have them make
forecasts, their confidence goes up rapidly, but this has no
relationship to accuracy.
The country faces a similar problem. But our leaders in Washington
are not debating about their own investments. Instead, they are
thinking about how to spend other people’s money. They have provided no
scientific basis for their decision and they show no awareness of how
one should properly approach such a forecasting problem.
There are ways to study this problem, but they should not be done in a rush, and they should not be done in group meetings.
We don’t know what will happen, but we do know what procedures to
use to obtain scientific forecasts of the outcomes of various plans.
Researchers in the field have been trying to spread the word on
scientific (evidence-based) forecasting by making forecasting knowledge
easily and freely available to others at http://forecastingprinciples.com
The question for our leaders is whether they should invest $700
billion when, despite their confidence, they are completely ignorant of
the outcome of the investment plan.
Dr. J. Scott Armstrong. Professor, The Wharton School, University of Pennsylvania
Dr. Kesten C. Green. Business and Economic Forecasting Unit, Monash University.