A friend of mine in the football analytics world, Joe Landers, sent me a link to a VERY interesting article from the Harvard Business Review, What Research Tells Us About Making Accurate Predictions – by Walter Frick.
Here’s Part 2 of The Football Educator’s look on how an NFL Front Office could learn to improve its own accuracy.
TEAMS over INDIVIDUALS
Researchers split up Tetlock’s participants randomly into individual forecast situations and group settings. Groups can be biased and suffer their own problems in coming to accurate assessments, so Tetlock ensured they were trained on proper internal collaboration. As you might imagine, groups were more accurate than individuals. And “super forecasters” put on the same teams after Year 1 of the study, they became even more accurate. Super-Teams began to lose their cohesion after time, as individuals became entrenched in their own beliefs, while teams of “super forecasters” began to agree more and more over time.
This is where the media and outside world just don’t comprehend the internal dynamics of an NFL War Room. Final Say, Last Word, and Ultimate Authority over the roster just doesn’t get it done even on the most successful clubs. Despite how they want to portray the Czar Head Coach or Uber GM, effective personnel decisions are made in conjunction with total TEAM input and using the expertise of both the coaches & personnel men, along with leveraging as much collectible data as possible . No one person can do it all.
Another seemingly “self-evident” factor in predicting outcomes. Tetlock’s study showed individual open-mindedness to be a positive contributor, and that level or degree was significant under different circumstances.
Ego and power tend to cloud “open-mindedness” in the halls of many NFL organizations. Those clubs with leadership that displays this trait are apt to come to more accurate decision making when choosing players. But believe me that is NOT always the case.
TRAINING IN PROBABILITY
Tetlock found those his research trained in “probabilistic reasoning”, meaning the use of data from the past to help predict the future, were more accurate than those that didn’t receive the same training.
Much of the personnel process is using past player precedent (measurables, character, production, etc…) to compare/contrast one particular prospect’s ability with success in the future. Those in the NFL Front Office that focus on and use the strength of measurable data & analytics vs. just their “gut feeling” are more apt to score successful roster additions.
Tetlock groups who worked longer in the process of predicting, usually were the most accurate. Most NFL Front Offices that I know are hard at work from now until the day of the draft.
This is one area that it’s hard to concede that NFL clubs don’t follow. Nobody wants to be outworked, and the NFL Draft keeps getting pushed farther and farther back.
Not to be confused with “open-mindedness” but Tetlock sees the two related. Forecasters were given the opportunity to revise predictions as new information was gathered. Those groups willing to engage in revision were more apt to outperform those that didn’t.
The Football Educator has expounded time and again that the personnel process is like a 1000 piece puzzle, and one that more likely is to be accurately put together with the most interlocking pieces. Many coaches and personnel men, as well as the media engaged in their own predictive measures, want to jump to conclusions after the interpretation of a single event; Unproductive Senior Season, Bowl Game Performance, All-Star Game 1 on 1, Combine 40-Yard Dash, Pro Day Measurables, Personal Interview, Character Flaw, Mock Drafts, etc… Use of ALL these factors is more likely to produce a more accurate forecast of the player’s potential performance.
Frick concludes his article with “certainty is the enemy of accurate prediction” but conveys “it’s possible to use a mix of practice and procedure to improve.” However Tetlock’s early findings only prove that people make worse predictions than algorithms, and that ultimately a mix of the two will lead to better and more accurate predicting.
This is true in the NFL Front Office as well. Arthur Blank’s hiring of Dan Quinn as head coach (Quinn’s first experience at that level after only 2 seasons as an NFL DC) and giving him “full control” over the Falcons’ roster will not in itself turnaround Atlanta’s two year skid. It will be how quickly Quinn figures out the effective TEAM oriented process of forecasting NFL talent that determines if Blank’s Falcons return to playoff contention.
That can be said for a number of other NFL Front Offices that just don’t seem to get it. You can predict the future if you set aside your need for “complete control, final authority, and ultimate say.” The use of strong analytics and integrating the thoughts of top notch evaluators (coaches/scouts) will go a long way in making more accurate roster assessments.