Age, Uncertainty, and Draft Decisions
Historically age has been important for draft picks. How do teams use that history to inform decisions about the future?
This wasn’t the plan.
With the NFL draft around the corner, I intended to build a simple model and write about my projections for this year’s crop of quarterbacks. But after fiddling with inputs for a few days, my results remained no better than simply using draft position.
Fortunately, not all was lost. In my attempts to improve performance, one variable continually showed signal: age at the time of the draft.1 And that’s a more interesting subject to talk about.
At this point, most, if not all, professional organizations use a model to help inform draft decisions. The type of model, and the degree to which it’s relied upon may vary, but they have something that aggregates information to predict how good players will be.
And if my experience is at all representative, the inclusion of age in those models elicits strong feelings.2 That’s because 1) it clearly adds predictive value above and beyond scouting reports and performance and 2) where that value comes from is misunderstood.
When most people hear that age is important for draft picks, they assume it means:
Younger players have more time remaining for physical development
Younger players have played less and therefore their skills will develop quicker with professional instruction
Younger players are more athletic with higher upsides
Despite being plausible, none of these reasons explain why age matters. Or maybe they do. Because we don’t know, with any real certainty, why age shows signal. All we know is that the combination of scouts and statistics alone under-project young players and over-project old players. Anything beyond that is conjecture.
That’s a hard truth to accept. Uncertainty is far less comforting than an understood narrative.
And if it stopped at narrative, it wouldn’t be a problem. Unfortunately, it rarely stops at the rationale; that rationale is then weaponized to dismiss age when the story doesn’t fit. Anyone who’s been in draft meetings can attest. The young player who is physically mature moves down the board, or the older player who only recently began playing their sport rises.
It could be correct to discount age in those cases, but it’s not based on evidence. The objective information says that predictions improve when they account for age.
That leads to the important question: how should teams incorporate age into their models?
Because, like most things, it’s not as simple as taking the output at face value.
Draft models are built on history. A good training set — something recent enough to be useful but with enough time to understand outcomes — will consist mostly of data that’s 10 or more years old.3
But things have changed dramatically, for any organization, over ten years. Many of the scouts who wrote the old reports have moved on to new teams. Player development has modernized with new technologies and added emphasis on training, nutrition, and recovery. And maybe most importantly, teams understand far better that age matters.
Since Rany Jazayerli’s groundbreaking article 15 years ago, scouts have slowly acknowledged that young players tend to perform better.4
If evaluators are aware that age matters, and that awareness influences their reports, including a model adjustment for age runs the risk of overcorrecting. Then again, there’s plenty of research to suggest that awareness of bias isn’t enough to overcome bias. Even if scouts are factoring in age more than they did previously, the signal may be as strong as ever.
Teams use models because they’re better than humans at combining and weighing information, which is really what decisions in the draft are all about. But there’s a reason the most well-known quote about models tells us that only some of them are useful. Models don’t (perfectly) reflect reality and they require a great deal of choice and understanding to use effectively.
You could convince me that nearly any magnitude of adjustment for age makes sense; you just won’t be able to prove it.
The inclusion of age boils down to belief. Based on everything — history, environment, people — what do we believe is appropriate? What bet are we comfortable making?
Fernando Mendoza, the presumptive top pick in next Thursday’s NFL draft, is 22.5 years old. That’s slightly younger than the average quarterback drafted this century. How much that matters is a question that every NFL team has to answer for themselves.
While not all hits, the group of “recent” QBs that were 21 at the time of the draft includes Aaron Rodgers, Matthew Stafford, Lamar Jackson, Sam Darnold, Michael Vick, and CJ Stroud.
Example of this in the public space: the way Keith Law writes about age in his draft recaps.
A baseball model would be built on something like 2008-2017.
Michael Lewis also wrote about the Rockets discovering age in The Undoing Project.
Your point is well-taken that after people are aware of their biases that they may adjust and therefore the signal the model was trained upon may not generalize.
Similarly, another potential data drift case is that in the age of NIL more football/basketball prospects are choosing to return to school when in previous years they would have entered the draft, resulting in older draft prospect classes.
But there are still ways to prevent or at least mitigate the issue:
- Train only on current grades / critical factors as opposed to future grades, all-in-one prospect grades, and rankings.
- Remove scouting and mock draft info entirely.
- Remove age from the model entirely (but make sure the other inputs to the model are appropriately age-adjusted).
Those models will be worse than models with more info baked in but they will still provide a useful perspective alongside the more comprehensive models, especially in potential data drift cases like the age conundrum.