The struggle to make the grade.
With the end of the term quickly approaching, now is the time to take a long, hard look at grades — NFL Draft grades! The draft starts on Saturday, and even if you do not wish to watch 36 straight hours of ESPN, you can still take part in the fun. More than any other professional sport, the impact of rookies is paramount in football. In baseball, picks take years to make it to the major leagues. In basketball, a handful of the rookies become difference makers for their teams, but most rookies slowly work their way into their team’s rotation. However, in football, most teams rely upon the draft to fill at least two starting spots. Partially because the results of the draft are so quickly seen on the field, fans have taken an especially large interest in the draft in recent years.
Each year, immediately after the draft, a variety of media sources publish their draft grades. Fans are eager to see “how their team did.” In reality, though, these grades only tell the reader if their team did a good job picking the players that were conventionally popular. If a team picked a player that their organization liked, but others thought was unworthy of the selection, that would be immediately perceived as a bad pick. Sometimes, these less popular picks turn out to be very successful players, as was the case with Edgerrin James, whom the Indianapolis Colts selected in 1999. The Colts were questioned for picking James over Heisman winner (and recreational drug enthusiast) Ricky Williams, but James was ultimately a much more successful NFL running back. Conversely, the team that selects the players most favorably rated by media sources before the draft will get a high mark. Most graders admit that it is impossible to judge a team’s class immediately after the draft … yet since these columns are popular, almost every website and magazine produce post draft grades anyway.
As expected, these grades are pretty useless at predicting how “good” each team’s draft class will be. Since it is impossible to know how the players will turn out, the immediate grades hold little value. Grading the draft after three full seasons gives a more accurate assessment of each team’s personnel department, since after three seasons, we are able to tell which players turned out to be stars and which turned out to be busts. Scouts Inc. at ESPN.com has done exactly that in a recent article on the 2005 draft. In the article, they create a composite “immediate post-draft grade” that includes grades from ESPN, SI, Yahoo!, etc. and then they contrast these grades with a current grade based on how the players have panned out. The Scouts Inc. grade is subjective, based on their current ratings of the players the team picked. Needless to say, there are some huge disparities.
GM A.J. Smith’s San Diego Chargers are the only team to earn an “A” today. They received a B+ immediately after the draft. His risky first round picks of Shawne Merriman and Luis Castillo turned out to be fantastic. Merriman played DE in college and some teams wondered if he could make the transition to LB. Castillo tested positive for steroids in college and there was concern that his performance was driven by steroids or he would return to the juice again in the NFL. Ironically, it was Merriman who tested positive for steroids in the NFL. Fortunately for San Diego, he did this while becoming one of the top LB’s in the game … a link? I’ll save that for another article.
Minnesota and Detroit receive F’s for their updated grades after originally scoring a B- and A, respectively. Both teams used top 10 selections on wide receivers from schools that go by the acronym “USC.” Minnesota selected Troy Williamson of University of South Carolina seventh and Detroit picked Mike Williams of University of Southern California tenth. Neither player remains with his original team. Mike Williams will probably never play in the NFL again. Minnesota drew great praise for selecting defensive end Erasmus James of University of Wisconsin with their second selection in the first round, but he has turned out to be a marginal player. With two first round picks and no quality starters, the Scouts Inc. graders decided the Vikings deserve an F.
As you could probably guess, the grades on draft day do not match up too well with the grades awarded today. To test the correlations, I converted the grades to the familiar GPA scale (A=4, A-=3.67, B+=3.33, etc.). There is a slightly positive correlation between high grades on draft day and high grades today. This means that those teams with higher grades on draft day do tend to get a higher retrospective grade. The slope of the line is .3. This tells us that for every additional full letter grade higher that a team received on draft day, they should be roughly 1/3 of a letter up in the retrospective grade. This would mean that we should expect a team that got an A in 2005 to be 1/3 of a grade higher today than a team that got a B. However, the linear regression only accounts for 3.1 percent (r2=.031) of the variation in grades. So although there is a slightly positive correlation, the draft day grades are almost useless at predicting the future grades. No surprise there; even the writers admit that these are shots in the dark.
More interestingly, the average grade awarded on draft day was a 2.85 (B/B-), while the average grade today was a 2.32 (C+). It looks like we have had some serious grade deflation. This may sound like blasphemy to Harvard students, but I would guess that if we were to run similar tests for each draft, this may be the norm. Immediately after the draft, there is so much excitement and every player picked in the early rounds seems like he will be a superstar. As time goes by and many of the players do not pan out, the teams look like they did a worse job in the draft. 2005 turned out to be a particularly weak class, but it is interesting that the grades on the whole were considerably higher than they should have been. Really, the draft should be graded strictly on a curve. The average immediate grade and retrospective grade should be the same. If there is a particularly strong class and a team comes away with average players, they should get a below-average grade. Teams do not control the size of the pie; they can only hope to grab the largest slice. In this particular class of 2005, just because the year as a whole was weak does not mean that only one team should get an A. Perhaps the media expects too much from the draft.
If NFL personnel departments are similarly biased, it could be evidence that they overvalue draft picks. If over a series of years, their draft grades are consistently higher immediately after the draft than a few years later, they are expecting too much from their selections. NFL teams assess how they do after the draft, but (to my knowledge) they do not put actual grades on their own draft or others. Still, if these biases are present, a shrewd GM could trade away the overvalued uncertainty of the draft picks for more proven veterans. This could create salary cap issues since one of the functions of the draft is to fill the roster with cheap talent. However, maybe the teams would realize that they systematically overvalue the value of early picks and they could trade them for later selections.
Cade Massey and Richard Thaler have researched this topic and proven that teams may not systematically overvalue the average talent level throughout the draft, but they do overestimate their abilities to differentiate between players, especially at the top of the draft. In their paper, “Overconfidence vs. Market Efficiency in the National Football League,” they recommend that teams trade down from high in the draft to get multiple selections later. Using empirical evidence, they show that the difference in talent between picks in the draft is smaller than the difference in their contracts and the value of the picks (since the high picks could be traded for multiple lower picks). For example, the 15th pick is on average only a little better than the 20th pick, but the higher pick is paid much more and the team could have traded that pick for multiple later picks.
So when you read those draft grades early next week, don’t get too excited if your team gets an A and don’t fret if your favorite squad gets a D. As for your grades on midterms and finals … those unfortunately are good predictors of your ultimate grade at the end of the year.
Daniel Adler (dadler@fas) ’10 got an A in statistics.The Harvard Sports Analysis Collective (HSAC) blogs at (hcs.harvard.edu/~hsac/Blog/).