clock menu more-arrow no yes

Filed under:

A new view of NFL free agency: Searching for surplus value among the offensive tackles

New, comments

In the NFL, it's difficult to statistically quantify a player's value. Jason Chilton introduces the Dollar Value Performance method of answering that timeless question about determining the value of a player's contributions on the field. First up are the offensive tackles. SB Nation's GIF Tournament V

If you buy something from an SB Nation link, Vox Media may earn a commission. See our ethics statement.

Kyle Terada-USA TODAY Sports

With the start of 2014 free agency looming, we decided to approach it with an interesting and deceptively simple proposition: exploring which positions have provided NFL teams with the most "bang for their buck" in free agency. Conversely, that also includes which positions have provided teams with the worst return on their investment.

Always be wary of those deceptively simple propositions.

The question of relative free agent performance would be simple to spitball, but coming up with a more substantive answer meant confronting the fact that there's currently no real way to quantify the value of an NFL player's performance relative to his contract.

That meant we needed to create one.

Quantifying value in the NFL

SB Nation NFL

Of the major U.S. sports, it's no secret that the NFL lags far behind its peers when it comes to a publicly available means to measure and quantify player performance. Major League Baseball has led the advanced stats revolution, and currently enjoys a wealth of in-depth analytical stats, with Wins Above Replacement (WAR) being perhaps the most widely used means of quantifying contributions to victory by players at all positions. The NBA is in the midst of its own advanced stats renaissance, and John Hollinger's Player Efficiency Rating (PER) is one of a number of measures that assign an aggregate value to a player's performance across multiple statistical categories.

The NFL is nowhere close to that level of precision, and it may not get there for a long time -- if ever. Even if we fully nail down how much a 14-yard completion on second-and-7 from the 30-yard line impacts the probability of scoring points on that drive, apportioning "credit" and "blame" to each of the 22 players on the field for their role in that play might always be a dicey proposition.

For the moment, we have to keep things more simple. To create an overall value metric for the NFL, we've taken the approach of assigning players a relative value within their position group based on common performance metrics for that position. We've sourced those metrics from the well-regarded game charting data put out by, focusing on a couple of key player metrics at each position. In the absence of the kind of data that would enable true win-impact measurement, we've quantified performance by dollar value with a single metric called Dollar Value Performance (DVP).*

The goal of DVP is to assign a dollar value to each player's performance relative to his position as a whole, factoring in both his level and volume of play. A player who plays an average number of starter's snaps at an average performance level would be worth the average salary for that position. He can create more value by playing at a higher level for more snaps, and his value decreases if he's playing fewer snaps or turning in a below-average performance.

To keep this piece at a remotely reasonable length and get to the specific question of free agency, I'll save detailed discussion of the calculation mechanics for another article (but I will be happy to answer any questions in the comments). However, it is worth taking a quick look at the five key assumptions that undergird the approach we've taken to DVP calculation.

#1 - The market knows how to set the relative value of positions

To be sure, this assumption may not hold up as more advanced statistical approaches get better at quantifying contributions to success in the NFL. For now, though, we've got a current salary structure that's the result of a lot of smart GMs, coaches and agents working to determine what players are worth in a salary cap environment. We're willing to roll with the idea that paying the top tackles $X million and the top safeties $Y million makes sense, for now. That means that the top-end values at each individual position are determined by the current top salaries at that position, so we use those salaries to define the top of the market at each position.

#2 - Value has both a quality and a volume component

A player only has value if he's playing, and the value he creates over X number of snaps is dependent on his level of play. Our approach to quantifying both components is to assign a per-snap dollar value to each incremental level of play (for example, percentage of passing snaps with pressure allowed for an offensive lineman), assigning the appropriate value to each player and multiplying it by his snaps played.

#3 - Replacement-level and below replacement-level play exists

The concept of replacement-level play -- the performance you'd get from a player signed "off the street" -- is, as the name indicates, a critical component of baseball's Wins Above Replacement (WAR) measure of a player's value. That formula also recognizes the existence of play that's below replacement level -- play that's so poor that it it actually destroys value relative to what an "off the street" player might give you. For MLB, the "replacement player" analogue is a Triple-A call-up; for the NFL, it's an undrafted or street free agent or practice squad activation. Baseball's WAR formula recognizes that a "replacement player" is still a relatively elite athlete and not some guy yanked out of the stands, and the "replacement level" value in the DVP calculation makes the same assumption.

#4 - Average for a position includes all players who took a snap

An average level of position performance and an average salary for each position are critical inputs to DVP. We set the average performance level for a given metric by weighting the performance of all players who played at least one snap at that position by the number of snaps that they played. The average salary took into account every player who played at least one snap -- pure backups and guys who missed the entire season due to injury weren't factored in.

#5 - The Pass Is Primary

Every position on the field impacts both the run and pass game to some degree. The farther from the ball you get, the more obvious it is that passing game performance is the primary determinant of a player's value. But even for the big uglies on both sides of the ball, passing game performance was weighted higher than run game contributions for the simple reason that passing game success is far more critical to winning in the modern NFL. Anyone who needs convincing on that score should take a look at this SI article on passer rating differential and Super Bowl winners.

The simplified version

The DVP is calculated by assigning a per-snap dollar value to a player's run plays and pass plays based on his level of play (better play earns a higher per-play value). An example of per-play value is pressure allowed for offensive tackles - a tackle who allows pressure on 4.9% of pass plays (which is the average allowed by the top 10 players a the position) gets a per-snap value in line with the average per-snap cap charge for those Top 10 players. A guy who allows pressure on 7.3% of pass plays (basically average for the tackle position) gets the per-snap value of the position as a whole. A guy who allows pressure on 11% of pass plays (essentially replacement-level) gets the per-snap value of a guy making the league minimum. Once a guy has a per-snap value for the run and pass components of his game, they get multiplied by the run and pass snaps he played to determine his total DVP number.

So when do we get to free agency, already? Now! Now we get to free agency!

Free agency: Searching for surplus value

The goal of any contract signing of any player by any team should be the creation of surplus value -- in other words, the player should create more value on the field than what he's paid in his contract. Discussions of surplus value are part and parcel of just about every free agent signing in MLB these days. The sport's stat wizards have quantified a dollar value for a Win Above Replacement, allowing surplus value to be defined as the WAR he generates in a season (or over the life of his contract) over and above his contract value. The creation of the DVP measure aims to enable the same kind of discussion for the NFL by putting a dollar value on a player's performance and comparing that to his cap charge in a given season.

When baseball players in their late 20s sign multi-year free agent contracts, it's frequently assumed that they'll generate surplus value over the first years of their deal while generating negative value later on as their performance declines along fairly well-charted age curves. In the NFL, you'd instinctively assume that phenomenon to be even more pronounced.

The realities of salary cap management have led to significant backloading on almost all multi-year deals, and the sheer brutality of the sport means that performance can often fall off far more steeply than it does for the Boys of Summer. Astute fans also instinctively assume that winning NFL teams utilize free agency to fill, at most, a couple of key needs while providing depth or bridge starters at other spots -- the salary cap tends to ruthlessly punish whole hog attempts to "buy a winner."

But how much, if any, surplus value gets created in free agency? Are some positions prone to "better buys" or fraught with higher-than-average risk? Let's find out -- and let's start with the big uglies on offense.

Offensive Tackles

With the exception of quarterback, few positions see quality players extended by their own clubs as frequently as offensive tackles, particularly the (perhaps overvalued) boys on the blind side. Elite 20-something performers hardly ever make it to free agency without a strange set of circumstances, so teams shelling out big free agent dollars may struggle to find players talented enough to justify their largesse.

To set some context for how the Dollar Value Performance concept looks when it's applied, here's a look at the three top offensive tackles in terms of 2013 DVP:

Team Player Total Snaps Run Snaps Run Rating / Snap Run DVP Pass Snaps Pressure Allowed / Snap Pass DVP Total DVP 2013 Cap Hit 2013 Surplus Value
WAS Trent Williams 1170 479 2 $2942930 691 5.1% $6940060 $9882990 $7980393 $1902597
DEN Orlando Franklin 1114 493 0.1 $1213192 621 3.9% $8438274 $9651466 $1188000 $8463466
CLE Joe Thomas 1149 398 0.6 $1365165 751 4.9% $8043618 $9408783 $11400000 -$1991217

We've chosen one key metric for run and pass game performance to define value for offensive linemen. The Run Rating/Snap applies a formula to Pro Football Focus' run grades for O-linemen to give it a meaningful scale on a per-snap basis. Pressure allowed per snap simply measures the percentage of pass protection snaps where a lineman allows his QB to be sacked, hit or hurried. The run DVP assigns a dollar value to the player's run game performance (determined by snaps and quality) while the pass DVP does the same for passing game performance. Those two figures are summed into a player's Total DVP, and that figure is simply compared against his 2013 cap charge to determine his surplus (or negative) value.

Right off the bat, there are a few interesting dynamics at play.

Trent Williams is widely recognized as one of the league's best left tackles, and a sparkling two-way performance amidst a crumbling offense in 2013 certainly justified that reputation. Williams created substantial value in the run game and in pass protection, and the fact that he was well paid but not at the absolute top of the positional salary scale allowed Washington to capture some excess value from his outstanding performance.

Orlando Franklin came in second in total DVP among tackles, and with a tidy $1.2 million cap charge he had no problem bagging the surplus value crown at the position. He's a great example of the potential value inherent in reasonable contracts, and he's also an example of the limitations in these kinds of metrics. It's almost certain that Trent Williams' pass-blocking performance was more truly valuable than Franklin's in 2013. Franklin got to face more power rushers on the right side and blocked for quick-release impresario Peyton Manning, whereas Williams went up against the elite on the left while protecting a wounded RGIII. Those differences don't show up in the Pressure Allowed stat. PFF makes an effort to capture those types of dynamics in its play ratings, but on the passing side we decided to keep the first iteration of DVP as clean and simple as possible with a single input variable for passing performance.

Joe Thomas is another of the league's best, and he turned in a quality pass protection showing in 2013 while just serving as a sort of above-average run blocker. That $11.4 million cap charge created a tremendously high hurdle, though, and even a player of Thomas' caliber will struggle to justify such a hefty portion of his team's financial resources.

To see how the other half (or, I guess, the middle third) lives, here's a look at three tackles whose DVPs were right around the position average:

Team Player Total Snaps Run Snaps Run Rating / Snap Run DVP Pass Snaps Pressure Allowed / Snap Pass DVP Total DVP 2013 Cap Hit 2013 Surplus Value
NE Sebastian Vollmer 516 220 -0.2 $413446 296 5.4% $2676711 $3090156 $3000000 $90156
IND Gosder Cherilus 1092 436 -0.4 $760792 656 7.2% $1993965 $2754756 $3900000 -$1145244
GB Don Barclay 979 433 -2.9 $58076 546 6.6% $2752223 $2810299 $480833 $2329466

Three more players, three more sets of insights into the DVP concept. The Pats' Sebastian Vollmer had his season cut short by injury, but still delivered a strong enough performance (particularly in pass protection) to generate surplus value on his reasonable $3 million cap charge. Colts' free agent acquisition Gosder Cherilus was an average pass blocker and below-average (but not atrocious) run blocker, but even at a non-bank-breaking $3.9 million cap charge on the first year of his deal, he had trouble breaking even. One of the guys blowing the salary curve was Packers rookie LT Don Barclay, whose poise in pass protection helped him create a ton of surplus value on his minuscule rookie salary despite some highly dubious efforts in the ground game.

And now, finally, let's start to look at historic free agent performance at the tackle position. We've elected to start with a look at the 2011-2013 free agent classes, since they were all signed after the advent of the league's new Collective Bargaining Agreement and play in an era of readily available performance and contract data.

For all of 2011's well-deserved acclaim as a historic free agent feeding frenzy, few quality OTs actually ended up changing hands. Here's a look at the guys who did change teams in 2011:

Signing Team Player 2011 Cap Hit 2011 DVP 2011 Surplus 2012 DVP 2013 DVP Total Surplus to Date
WAS Sean Locklear $810000 $463809 -$346191 N/A (No longer on team) N/A (No longer on team) -$346191
KC Jared Gaither $685000 $61372 -$623628 N/A (No longer on team) N/A (No longer on team) -$623628
PHI Ryan Harris $735000 N/A (Injured) -$735000 N/A (No longer on team) N/A (No longer on team) -$735000
MIA Marc Colombo $2000000 $1024410 -$975590 N/A (No longer on team) N/A (No longer on team) -$975590

Not a great start for generating surplus value! A combination of injuries and ineffectiveness kept this free agent crop from creating any surplus value despite relatively bargain-basement contracts -- in fact, none of these guys even ended up taking a snap for their 2011 teams in the following season. Here's a look at the performance that these guys did manage during their brief time on the field in 2011:

Player Run Snaps Run Rating / Snap Pass Snaps Pressure Allowed / Snap
Sean Locklear 87 -2.1 205 8.3%
Jared Gaither 19 -7.9 7 0%
Ryan Harris 0 N/A 0 N/A
Marc Colombo 494 -1.6 543 9.8%

Would things get better in 2012? Let's see:

Signing Team Player 2012 Cap Hit 2012 DVP 2012 Surplus 2013 Cap Hit 2013 DVP 2013 Surplus Remaining Dead Money Total Surplus to Date
NYG Sean Locklear $890000 $1148005 $258005 $- N/A (No longer on team) $- $- $258005
SEA Frank Omiyale $1225000 $737081 -$487919 $- N/A (No longer on team) $- $- -$487919
KC Eric Winston $2500000 $4859665 $2359665 $3000000 N/A (No longer on team) -$3000000 $- -$640335
PHI Demetress Bell $3250000 $26188 -$3223812 $- N/A (No longer on team) $- $- -$3223812
SD Jared Gaither $2625000 $993771 -$1631229 $2000000 N/A (No longer on team) -$2000000 -$4000000 -$7631229

In his triumphant return to free agency, Sean Locklear redeemed himself by generating a cool quarter mil in surplus value for the Giants. Jared Gaither's return to free agency ... went less well. He proved to be one of the most disastrous signings of 2012, failing to earn back even half of his first-year cap hit before getting cut and hamstringing the Chargers' future efforts via a smooth $6 million in dead money in 2013 and 2014. Eric Winston's season was something of a head-scratcher, as the Chiefs cooled on him despite an outwardly strong on-field performance and cut him despite the consequence of 3 million dead dollars the next season. Below are these gents' performance stats for 2012:

Player Run Snaps Run Rating / Snap Pass Snaps Pressure Allowed / Snap
Eric Winston 515 1.4 508 6.7%
Jared Gaither 119 6.6 130 9.2%
Demetress Bell 184 -2.7 278 11.9%
Frank Omiyale 73 0.1 46 4.3%
Sean Locklear 292 -1.2 364 8%

Some better showings, but these signings still resulted in a lot of red in the total surplus ledger. Is the 2013 class looking any better so far?

Signing Team Player 2013 Cap Hit 2013 DVP 2013 Surplus Remaining Dead Money Total Surplus to Date
SD King Dunlap $1775000 $5127540 $3352540 $- $3352540
STL Jake Long $4250000 $5663833 $1413833 $- $1413833
WAS Jeremy Trueblood $840000 $1203969 $363969 $- $363969
NE Will Svitek $950000 $1274643 $324643 $- $324643
NO Jason Smith $65000 N/A (Cut in Pre-Season) -$65000 $- -$65000
SD Max Starks $160000 N/A (Cut in Pre-Season) -$160000 $- -$160000
CLE Rashad Butler $555000 N/A (Cut in Pre-Season) -$555000 $- -$555000
ARI Eric Winston $2000000 $1124471 -$875529 $- -$875529
DEN Winston Justice $991176 $34946 -$956230 $- -$956230
CHI Jermon Bushrod $3015000 $1954533 -$1060467 $- -$1060467
IND Gosder Cherilus $3900000 $2754756 -$1145244 $- -$1145244
MIA Tyson Clabo $3500000 $1527310 -$1972690 $- -$1972690

Finally, some green! King Dunlap was part of the Career Resurrection Tour that was the 2013 Chargers' offense, and Jake Long proved a welcome addition in St. Louis despite being lost to injury late in the season. Jeremy Trueblood and Will Svitek justified modest salaries while playing less than starter snaps. The middle of the chart represented straight zeros in DVP as the players weren't able to make it through final cuts in the preseason. We learned that the Chiefs might have been onto something when they soured on Winston, as he struggled to a subpar performance in the desert. Fellow rent-a-right-tackle Tyson Clabo fared just as badly in South Beach, while Jermon Bushrod and Gosder Cherilus managed to upgrade their respective O-lines while still failing to justify mid-range salaries.

The performance numbers:

Run Snaps Run Rating / Snap Pass Snaps Pressure Allowed / Snap
King Dunlap 354 5.6 333 5.7%
Jake Long 403 4 469 6.8%
Jeremy Trueblood 232 -1 409 8.1%
Will Svitek 89 -7.8 148 5.4%
Jason Smith N/A (Cut in Pre-Season)
Max Starks N/A (Cut in Pre-Season)
Rashad Butler N/A (Cut in Pre-Season)
Eric Winston 474 -1.9 612 9.5%
Winston Justice 8 0 11 9.1%
Jermon Bushrod 440 -0.2 630 8.7%
Gosder Cherilus 436 -0.4 656 7.2%
Tyson Clabo 344 -1.7 618 8.4%

So What Does It All Mean?

It's pretty early to say just yet. In a sense, these 21 guys represent 21 individual data points, and by most measures that's a pretty scanty sample size for drawing substantive conclusions. Anecdotally, at least, this look at free agent tackles seems to confirm the idea that few elite guys ever hit the street (Jake Long's age and Miami's crappy cap situation in 2012 caused a divorce that might not have happened in other places) while highlighting the dicey prospects of making your money back as a team leaning on free agency.

In our next installment, we'll look at three years' worth of interior O-line signings to see if the picture gets any brighter.

*ProFootballFocus has their own method of deriving dollar value for a player's performance, known as the Jahnke Value Model. While we think this is a great model (and we put a lot of stock in PFF's game tracking and player performance data), we developed our own model to allow things like differential weighting of run and pass performance and assigning values to below-replacement-level performances.