A friend of mine in the football analytics world, Joe Landers, sent me a link to a VERY interesting article from the Harvard Business Review, What Research Tells Us About Making Accurate Predictions – by Walter Frick. It dispels the notion that “predicting the future is just too hard” and cited studies that even “experts struggled to perform better than dart-throwing chimps” and “consistently less accurate than simple statistical algorithms.”
Joe caught my attention when he shot out the following Tweet with the associated link to the article;
Frick went on to explain research performed by Philip Tetlock of the University of Pennsylvania, considered the most famous work done on prediction, used a series of questions to pundits and foreign affairs experts to see if they could accurately “predict” geopolitical events (thus the dart-throwing chimp analogy). What Tetlock uncovered was a particular style of thinking that appeared to lend itself to more accurate forecasting through a willingness to “consider multiple explanations and balance them together before making a prediction performed better than those who relied on a single big idea.”
So as I continued to read Frick’s article, I supplanted the notion of forecasting world events by scholastic superstars with the idea of building NFL rosters using Free Agency and the Draft through an NFL Front Office. After all, player selection in both those venues is really nothing more than predicting the production that a player will bring to your organization’s efforts (both on and off the field).
Back to Philip Tetlock. Tetlock and his colleagues began running a series of “Forecasting Tournaments” looking for the factors that make people better predictors. Their results covered some 150,000 forecasts by 743 participants predicting 199 world events. Tetlock and friends found that those participating in their tournaments were more likely to predict potential outcomes than by blind chance, and they were also able to prove that top predictors were able to “improve over time.”
So can your NFL Front Office team advance its own good fortune by following the factors that showed to mark improvement in predicting from the Tetlock tournaments? Can they increase their chances to sign more productive free agents and draft more talented college prospects by heeding these predictive points?
Tetlock’s sample was proven to be highly intelligent (not always a given in an NFL front office). Intelligence showed to be a big factor early in the prediction process of a new domain, but tended to disolve a bit as the group became settled.
Coaches or GM’s that come in with noteworthy football acumen can be helpful in the beginning and that intelligence certainly doesn’t hurt down the road, but owners shouldn’t buy into the notion that they’ve hired the smartest guy in the room and thus all player predictions will be flawless. How many recent hires come flying out the gates, only to crash a season or two thereafter?
Tetlock’s studies showed participants who scored high in “political knowledge” tended to make better predictions in his tournament settings.
Coaches and GM’s/Scouts that have a higher level of player evaluation ability are more likely to carry that over to more sound predicting, and thus better choices in the free agency/draft. That seems somewhat “self-evident” and yet I’ve seen many coaches/personnel men without this skill allowed to participate fully in the process.
PRACTICE & ACCURACY
Tetlock found those that continually worked at their craft by making more predictions and participating in more tournaments, actually did improve their accuracy.
The repetitive “dirty work” of player evaluation can be a drag on even the most seasoned coaches/scouts, but the great ones continue to do it. Many head coaches and top executives in NFL Front Offices want to have the FINAL say in the process, but aren’t willing to put in the work to reach the evaluation. They’d rather sort through the reports presented to them & use their rightful DOMAIN expertise to reach the outcome.
NEXT UP – Part 2