Given that NFL Draft prospects undergo one of the most comprehensive vetting processes imaginable, it's a wonder that evaluations still miss so badly from time to time. Organizations, as a whole, take players about where they ought to be selected in the NFL Draft, but on occasion teams like the 2006 Lions do a worse job than a dart board. Unfortunately for Detroit, or any team, it can only know so much about a man. Forty times and broad jumps are important measures, but how can a team ever understand the intangibles of a great athlete -- his instinct, his ambition, his leadership?
The Player Assessment Tool sets out to decipher the heretofore indecipherable. Introduced at the 2013 NFL Scouting Combine, the PAT is a 50-minute test that assesses players' thinking skills, and whether they have certain traits desired by general managers. Anonymous scouting reports are often derided for describing players with invented phrases that sound more like voodoo incantations (see: "embiggen his waffle-tossers," "real strong knee puller," and other nonsense by Jon Bois and Spencer Hall). The PAT, if results bear out, may finally lend some credence to the gibberish.
"This is meant to be, after you've done everything you've already done, including maybe interview the player at the Combine, this is meant to give you that last bit of information," Cyrus Mehri says. "And it will hopefully confirm a lot of what you had from traditional scouting. If there's divergence then you can follow up."
Mehri is an attorney, and one of the co-authors of the report that led to the implementation of the Rooney Rule requiring NFL teams to interview minority candidates for head coaching and senior front office positions. He approached the NFL with the idea for a test that is socioeconomically unbiased, addressing a criticism often leveled at the Wonderlic, the NFL Combine's notorious aptitude test.
The Wonderlic has been used by the NFL since the 1970s as a supposed measure of a player's ability to think quickly and -- since the 1990s when a personality component was added to the test -- thrive in a team setting. However, the test's results have been found to be mostly bunk. A 2009 study of the test results of 762 prospects across three draft classes found no correlation between a player's score and his on-field performance at every position except two. For defensive backs and tight ends the correlation was negative, meaning that those that those who performed worse on the test became better players.
The Wonderlic isn't necessarily a bad test, it just doesn't have much to do with football. It also counts life insurance companies, cabinet manufacturers and dog breeders among its clients. The Wonderlic strives to be a catch-all test applicable across multiple industries. The problem is, the traits inherent in an All-Pro linebacker may not be shared by puppy salesmen.
When the NFL greenlit Mehri's idea, he approached Harold Goldstein, an associate professor of industrial and organizational psychology at Baruch College, and Kenneth Yusko, an associate professor of human resources at Marymount, to develop a test catered toward potential NFL players. Mehri has been soliciting Goldstein's help on law cases for more than 10 years, and knew that he had done extensive research on reducing socioeconomic bias in personnel selection.
Prior to the 2013 NFL Combine, the three men worked for approximately six months "solid" to prepare the PAT for the first time. Yusko was primarily responsible for coming up with the questions. Neither Goldstein nor Yusko was available to comment for this story, but Sheldon Zedeck, a professor of psychology at UC-Berkeley not affiliated with the PAT, was able to speak about the issues with standardized testing.
The first is inherent cultural bias. Zedeck, along with Marjorie Schultz, studied the LSAT and its ability to predict effectiveness as a lawyer. They uncovered that while LSAT scores broke down by racial lines, with minority groups scoring lower, a test designed to suss out characteristics deemed by lawyers to be most important in the profession -- creativity, negotiation, stress management, interviewing, etc. -- showed that racial groups possessed those traits in relatively equal measure.
Like the Wonderlic, the LSAT isn't necessarily a bad measure of intelligence, but the weight it is given leads to a second issue: It's not a complete measurement.
"It's not that they're not getting things right, they're not getting the full picture of effectiveness," Zedeck says. "You could be the smartest guy in the class -- someone who's going to score very high on the SAT, you're going to do very well in all of your courses -- but can you see the relationship between this and that? Maybe not.
"You can fully understand and read a decision by a judge that relates to some insurance issue. Are you able to see that you may be able to use that decision in a real estate issue? Or in a healthcare issue that you have to present? Can you do more than read and understand things?"
Mehri, Goldstein and Yusko are striving to make the PAT a holistic exam, measuring both cognitive (a player's ability to think fast) and non-cognitive (traits like ambition, adaptability, perseverance) ability as it pertains to football, and structured to reduce the benefit of prior knowledge that many believe accounts for cultural differences in test results -- like knowing what a cul-de-sac is.
As Zedeck puts it: "If I wanted to compare people selling air conditioners, it would not be smart to compare those who are selling them in Alaska, or those back East these days, to Florida."
An unbiased, football-specific test sounds great as an idea, but is mind-bendingly difficult in practice -- "Imagine you're seeing some shape you've never seen before, a certain configuration, and you see another configuration and another configuration, again, questions regarding that," Mehri says. "Well none of that you've ever seen before. But it shows you how your mind looks."
Even harder than conjuring shapes the world has never seen may be ensuring that the players have no way to prepare for or game the test. The first edition of the PAT was developed with the input of eight NFL general managers who identified core traits they'd like to see in an NFL player -- a great idea, until agents figure out what those traits are and coach their clients on how to answer questions so that they look like model prospects.
The PAT attempts to be game-proof in a number of ways, however. For starters, players are not given a numeric score, unlike on the Wonderlic, so technically there is no way to do poorly on it. Players are profiled in four ways:
- Football smarts -- how quickly they process information.
- Psychological attributes -- their works habits, ambition, conscientiousness, etc.
- Learning style -- is a player a visual or aural learner, for example.
- Motivational cues -- what inspires a player to work and achieve.
Only one of those bullets -- football smarts -- might give a coach or executive the basis to compare players directly. The rest simply help the team understand the player better.
The other way to game-proof the test is by cleverly constructing the questions themselves.
"I could coach someone and say 'whenever you get a question about delegation, never delegate' -- I'm making this up," Zedeck says. "But I'd have Option A and Option B, both of them about never delegating, but some other nuance that differentiates them. So if they know the better nuance and the worse nuance will determine how effective or if they're less effective leaders who delegate."
The next step for Mehri, Goldstein and Yusko will be accumulating enough data to say, definitively, that they have created a tool that will help NFL teams make more informed decisions. Early results from Year 1 and Year 2 validation studies show a correlation between how players perform on the PAT and on the field -- "stronger than we would have expected," according to Mehri. Clearing the low bar set by the Wonderlic doesn't appear to be a problem.
NFL organizations themselves appear to be happy with the PAT. Mehri couldn't say that every team in the NFL is making use of the results, "but a lot of them are." One of the most frequent requests has been for more detailed coaching summaries to better inform position coaches how they can get the most out of their athletes.
"But it's still early returns," Mehri warns. "One way or the other we'd like to take the information, keep analyzing it, see if there are ways to tweak the PAT and make it better. So we're excited about where it is right now."
The experiment is still far from being completed, but if the PAT takes hold, it could mean a little less silliness during the NFL Draft season for us all -- fewer draft day misses, hopefully toned down anonymous scouting reports, and perhaps just a touch more sanity to an unwieldy process.