Your point is well-taken that after people are aware of their biases that they may adjust and therefore the signal the model was trained upon may not generalize.
Similarly, another potential data drift case is that in the age of NIL more football/basketball prospects are choosing to return to school when in previous years they would have entered the draft, resulting in older draft prospect classes.
But there are still ways to prevent or at least mitigate the issue:
- Train only on current grades / critical factors as opposed to future grades, all-in-one prospect grades, and rankings.
- Remove scouting and mock draft info entirely.
- Remove age from the model entirely (but make sure the other inputs to the model are appropriately age-adjusted).
Those models will be worse than models with more info baked in but they will still provide a useful perspective alongside the more comprehensive models, especially in potential data drift cases like the age conundrum.
Great points and suggestions. Baseball has it both ways — more players are going to/staying in college because of NIL and there also seem to be more HS players reclassifying to become eligible after three years of HS. I could see reasons the age of that player wouldn’t map cleanly to players who played their whole HS careers.
My guess is age still has a meaningful effect, but it’s important to find ways to account for everything we’re talking about. That can be in the model itself or with adjustments after the fact.
Your point is well-taken that after people are aware of their biases that they may adjust and therefore the signal the model was trained upon may not generalize.
Similarly, another potential data drift case is that in the age of NIL more football/basketball prospects are choosing to return to school when in previous years they would have entered the draft, resulting in older draft prospect classes.
But there are still ways to prevent or at least mitigate the issue:
- Train only on current grades / critical factors as opposed to future grades, all-in-one prospect grades, and rankings.
- Remove scouting and mock draft info entirely.
- Remove age from the model entirely (but make sure the other inputs to the model are appropriately age-adjusted).
Those models will be worse than models with more info baked in but they will still provide a useful perspective alongside the more comprehensive models, especially in potential data drift cases like the age conundrum.
Great points and suggestions. Baseball has it both ways — more players are going to/staying in college because of NIL and there also seem to be more HS players reclassifying to become eligible after three years of HS. I could see reasons the age of that player wouldn’t map cleanly to players who played their whole HS careers.
My guess is age still has a meaningful effect, but it’s important to find ways to account for everything we’re talking about. That can be in the model itself or with adjustments after the fact.