A new view of NFL free agency: Searching for surplus value among the offensive tackles

With the start of 2014 free agency looming, we decided to approach it with an interesting and deceptively simple proposition: exploring which positions have provided NFL teams with the most "bang for their buck" in free agency. Conversely, that also includes which positions have provided teams with the worst return on their investment.

Always be wary of those deceptively simple propositions.

The question of relative free agent performance would be simple to spitball, but coming up with a more substantive answer meant confronting the fact that there's currently no real way to quantify the value of an NFL player's performance relative to his contract.

That meant we needed to create one.

Quantifying value in the NFL

Of the major U.S. sports, it's no secret that the NFL lags far behind its peers when it comes to a publicly available means to measure and quantify player performance. Major League Baseball has led the advanced stats revolution, and currently enjoys a wealth of in-depth analytical stats, with Wins Above Replacement (WAR) being perhaps the most widely used means of quantifying contributions to victory by players at all positions. The NBA is in the midst of its own advanced stats renaissance, and John Hollinger's Player Efficiency Rating (PER) is one of a number of measures that assign an aggregate value to a player's performance across multiple statistical categories.

The NFL is nowhere close to that level of precision, and it may not get there for a long time -- if ever. Even if we fully nail down how much a 14-yard completion on second-and-7 from the 30-yard line impacts the probability of scoring points on that drive, apportioning "credit" and "blame" to each of the 22 players on the field for their role in that play might always be a dicey proposition.

For the moment, we have to keep things more simple. To create an overall value metric for the NFL, we've taken the approach of assigning players a relative value within their position group based on common performance metrics for that position. We've sourced those metrics from the well-regarded game charting data put out by ProFootballFocus.com, focusing on a couple of key player metrics at each position. In the absence of the kind of data that would enable true win-impact measurement, we've quantified performance by dollar value with a single metric called Dollar Value Performance (DVP).*

The goal of DVP is to assign a dollar value to each player's performance relative to his position as a whole, factoring in both his level and volume of play. A player who plays an average number of starter's snaps at an average performance level would be worth the average salary for that position. He can create more value by playing at a higher level for more snaps, and his value decreases if he's playing fewer snaps or turning in a below-average performance.

To keep this piece at a remotely reasonable length and get to the specific question of free agency, I'll save detailed discussion of the calculation mechanics for another article (but I will be happy to answer any questions in the comments). However, it is worth taking a quick look at the five key assumptions that undergird the approach we've taken to DVP calculation.

#1 - The market knows how to set the relative value of positions

To be sure, this assumption may not hold up as more advanced statistical approaches get better at quantifying contributions to success in the NFL. For now, though, we've got a current salary structure that's the result of a lot of smart GMs, coaches and agents working to determine what players are worth in a salary cap environment. We're willing to roll with the idea that paying the top tackles $X million and the top safeties $Y million makes sense, for now. That means that the top-end values at each individual position are determined by the current top salaries at that position, so we use those salaries to define the top of the market at each position.

#2 - Value has both a quality and a volume component

A player only has value if he's playing, and the value he creates over X number of snaps is dependent on his level of play. Our approach to quantifying both components is to assign a per-snap dollar value to each incremental level of play (for example, percentage of passing snaps with pressure allowed for an offensive lineman), assigning the appropriate value to each player and multiplying it by his snaps played.

#3 - Replacement-level and below replacement-level play exists

The concept of replacement-level play -- the performance you'd get from a player signed "off the street" -- is, as the name indicates, a critical component of baseball's Wins Above Replacement (WAR) measure of a player's value. That formula also recognizes the existence of play that's below replacement level -- play that's so poor that it it actually destroys value relative to what an "off the street" player might give you. For MLB, the "replacement player" analogue is a Triple-A call-up; for the NFL, it's an undrafted or street free agent or practice squad activation. Baseball's WAR formula recognizes that a "replacement player" is still a relatively elite athlete and not some guy yanked out of the stands, and the "replacement level" value in the DVP calculation makes the same assumption.

#4 - Average for a position includes all players who took a snap

An average level of position performance and an average salary for each position are critical inputs to DVP. We set the average performance level for a given metric by weighting the performance of all players who played at least one snap at that position by the number of snaps that they played. The average salary took into account every player who played at least one snap -- pure backups and guys who missed the entire season due to injury weren't factored in.

#5 - The Pass Is Primary

Every position on the field impacts both the run and pass game to some degree. The farther from the ball you get, the more obvious it is that passing game performance is the primary determinant of a player's value. But even for the big uglies on both sides of the ball, passing game performance was weighted higher than run game contributions for the simple reason that passing game success is far more critical to winning in the modern NFL. Anyone who needs convincing on that score should take a look at this SI article on passer rating differential and Super Bowl winners.

The simplified version

The DVP is calculated by assigning a per-snap dollar value to a player's run plays and pass plays based on his level of play (better play earns a higher per-play value). An example of per-play value is pressure allowed for offensive tackles - a tackle who allows pressure on 4.9% of pass plays (which is the average allowed by the top 10 players a the position) gets a per-snap value in line with the average per-snap cap charge for those Top 10 players. A guy who allows pressure on 7.3% of pass plays (basically average for the tackle position) gets the per-snap value of the position as a whole. A guy who allows pressure on 11% of pass plays (essentially replacement-level) gets the per-snap value of a guy making the league minimum. Once a guy has a per-snap value for the run and pass components of his game, they get multiplied by the run and pass snaps he played to determine his total DVP number.

So when do we get to free agency, already? Now! Now we get to free agency!

Free agency: Searching for surplus value

The goal of any contract signing of any player by any team should be the creation of surplus value -- in other words, the player should create more value on the field than what he's paid in his contract. Discussions of surplus value are part and parcel of just about every free agent signing in MLB these days. The sport's stat wizards have quantified a dollar value for a Win Above Replacement, allowing surplus value to be defined as the WAR he generates in a season (or over the life of his contract) over and above his contract value. The creation of the DVP measure aims to enable the same kind of discussion for the NFL by putting a dollar value on a player's performance and comparing that to his cap charge in a given season.

When baseball players in their late 20s sign multi-year free agent contracts, it's frequently assumed that they'll generate surplus value over the first years of their deal while generating negative value later on as their performance declines along fairly well-charted age curves. In the NFL, you'd instinctively assume that phenomenon to be even more pronounced.

The realities of salary cap management have led to significant backloading on almost all multi-year deals, and the sheer brutality of the sport means that performance can often fall off far more steeply than it does for the Boys of Summer. Astute fans also instinctively assume that winning NFL teams utilize free agency to fill, at most, a couple of key needs while providing depth or bridge starters at other spots -- the salary cap tends to ruthlessly punish whole hog attempts to "buy a winner."

But how much, if any, surplus value gets created in free agency? Are some positions prone to "better buys" or fraught with higher-than-average risk? Let's find out -- and let's start with the big uglies on offense.

Offensive Tackles

With the exception of quarterback, few positions see quality players extended by their own clubs as frequently as offensive tackles, particularly the (perhaps overvalued) boys on the blind side. Elite 20-something performers hardly ever make it to free agency without a strange set of circumstances, so teams shelling out big free agent dollars may struggle to find players talented enough to justify their largesse.

To set some context for how the Dollar Value Performance concept looks when it's applied, here's a look at the three top offensive tackles in terms of 2013 DVP:

Team	Player	Total Snaps	Run Snaps	Run Rating / Snap	Run DVP	Pass Snaps	Pressure Allowed / Snap	Pass DVP	Total DVP	2013 Cap Hit	2013 Surplus Value
WAS	Trent Williams	1170	479	2	$2942930	691	5.1%	$6940060	$9882990	$7980393	$1902597
DEN	Orlando Franklin	1114	493	0.1	$1213192	621	3.9%	$8438274	$9651466	$1188000	$8463466
CLE	Joe Thomas	1149	398	0.6	$1365165	751	4.9%	$8043618	$9408783	$11400000	-$1991217

We've chosen one key metric for run and pass game performance to define value for offensive linemen. The Run Rating/Snap applies a formula to Pro Football Focus' run grades for O-linemen to give it a meaningful scale on a per-snap basis. Pressure allowed per snap simply measures the percentage of pass protection snaps where a lineman allows his QB to be sacked, hit or hurried. The run DVP assigns a dollar value to the player's run game performance (determined by snaps and quality) while the pass DVP does the same for passing game performance. Those two figures are summed into a player's Total DVP, and that figure is simply compared against his 2013 cap charge to determine his surplus (or negative) value.

Right off the bat, there are a few interesting dynamics at play.

Trent Williams is widely recognized as one of the league's best left tackles, and a sparkling two-way performance amidst a crumbling offense in 2013 certainly justified that reputation. Williams created substantial value in the run game and in pass protection, and the fact that he was well paid but not at the absolute top of the positional salary scale allowed Washington to capture some excess value from his outstanding performance.

Orlando Franklin came in second in total DVP among tackles, and with a tidy $1.2 million cap charge he had no problem bagging the surplus value crown at the position. He's a great example of the potential value inherent in reasonable contracts, and he's also an example of the limitations in these kinds of metrics. It's almost certain that Trent Williams' pass-blocking performance was more truly valuable than Franklin's in 2013. Franklin got to face more power rushers on the right side and blocked for quick-release impresario Peyton Manning, whereas Williams went up against the elite on the left while protecting a wounded RGIII. Those differences don't show up in the Pressure Allowed stat. PFF makes an effort to capture those types of dynamics in its play ratings, but on the passing side we decided to keep the first iteration of DVP as clean and simple as possible with a single input variable for passing performance.

Joe Thomas is another of the league's best, and he turned in a quality pass protection showing in 2013 while just serving as a sort of above-average run blocker. That $11.4 million cap charge created a tremendously high hurdle, though, and even a player of Thomas' caliber will struggle to justify such a hefty portion of his team's financial resources.

To see how the other half (or, I guess, the middle third) lives, here's a look at three tackles whose DVPs were right around the position average:

Team	Player	Total Snaps	Run Snaps	Run Rating / Snap	Run DVP	Pass Snaps	Pressure Allowed / Snap	Pass DVP	Total DVP	2013 Cap Hit	2013 Surplus Value
NE	Sebastian Vollmer	516	220	-0.2	$413446	296	5.4%	$2676711	$3090156	$3000000	$90156
IND	Gosder Cherilus	1092	436	-0.4	$760792	656	7.2%	$1993965	$2754756	$3900000	-$1145244
GB	Don Barclay	979	433	-2.9	$58076	546	6.6%	$2752223	$2810299	$480833	$2329466

Three more players, three more sets of insights into the DVP concept. The Pats' Sebastian Vollmer had his season cut short by injury, but still delivered a strong enough performance (particularly in pass protection) to generate surplus value on his reasonable $3 million cap charge. Colts' free agent acquisition Gosder Cherilus was an average pass blocker and below-average (but not atrocious) run blocker, but even at a non-bank-breaking $3.9 million cap charge on the first year of his deal, he had trouble breaking even. One of the guys blowing the salary curve was Packers rookie LT Don Barclay, whose poise in pass protection helped him create a ton of surplus value on his minuscule rookie salary despite some highly dubious efforts in the ground game.

And now, finally, let's start to look at historic free agent performance at the tackle position. We've elected to start with a look at the 2011-2013 free agent classes, since they were all signed after the advent of the league's new Collective Bargaining Agreement and play in an era of readily available performance and contract data.

For all of 2011's well-deserved acclaim as a historic free agent feeding frenzy, few quality OTs actually ended up changing hands. Here's a look at the guys who did change teams in 2011:

Signing Team	Player	2011 Cap Hit	2011 DVP	2011 Surplus	2012 DVP	2013 DVP	Total Surplus to Date
WAS	Sean Locklear	$810000	$463809	-$346191	N/A (No longer on team)	N/A (No longer on team)	-$346191
KC	Jared Gaither	$685000	$61372	-$623628	N/A (No longer on team)	N/A (No longer on team)	-$623628
PHI	Ryan Harris	$735000	N/A (Injured)	-$735000	N/A (No longer on team)	N/A (No longer on team)	-$735000
MIA	Marc Colombo	$2000000	$1024410	-$975590	N/A (No longer on team)	N/A (No longer on team)	-$975590

Not a great start for generating surplus value! A combination of injuries and ineffectiveness kept this free agent crop from creating any surplus value despite relatively bargain-basement contracts -- in fact, none of these guys even ended up taking a snap for their 2011 teams in the following season. Here's a look at the performance that these guys did manage during their brief time on the field in 2011:

Player	Run Snaps	Run Rating / Snap	Pass Snaps	Pressure Allowed / Snap
Sean Locklear	87	-2.1	205	8.3%
Jared Gaither	19	-7.9	7	0%
Ryan Harris	0	N/A	0	N/A
Marc Colombo	494	-1.6	543	9.8%

Would things get better in 2012? Let's see:

Signing Team	Player	2012 Cap Hit	2012 DVP	2012 Surplus	2013 Cap Hit	2013 DVP	2013 Surplus	Remaining Dead Money	Total Surplus to Date
NYG	Sean Locklear	$890000	$1148005	$258005	$-	N/A (No longer on team)	$-	$-	$258005
SEA	Frank Omiyale	$1225000	$737081	-$487919	$-	N/A (No longer on team)	$-	$-	-$487919
KC	Eric Winston	$2500000	$4859665	$2359665	$3000000	N/A (No longer on team)	-$3000000	$-	-$640335
PHI	Demetress Bell	$3250000	$26188	-$3223812	$-	N/A (No longer on team)	$-	$-	-$3223812
SD	Jared Gaither	$2625000	$993771	-$1631229	$2000000	N/A (No longer on team)	-$2000000	-$4000000	-$7631229

In his triumphant return to free agency, Sean Locklear redeemed himself by generating a cool quarter mil in surplus value for the Giants. Jared Gaither's return to free agency ... went less well. He proved to be one of the most disastrous signings of 2012, failing to earn back even half of his first-year cap hit before getting cut and hamstringing the Chargers' future efforts via a smooth $6 million in dead money in 2013 and 2014. Eric Winston's season was something of a head-scratcher, as the Chiefs cooled on him despite an outwardly strong on-field performance and cut him despite the consequence of 3 million dead dollars the next season. Below are these gents' performance stats for 2012:

Player	Run Snaps	Run Rating / Snap	Pass Snaps	Pressure Allowed / Snap
Eric Winston	515	1.4	508	6.7%
Jared Gaither	119	6.6	130	9.2%
Demetress Bell	184	-2.7	278	11.9%
Frank Omiyale	73	0.1	46	4.3%
Sean Locklear	292	-1.2	364	8%

Some better showings, but these signings still resulted in a lot of red in the total surplus ledger. Is the 2013 class looking any better so far?

Signing Team	Player	2013 Cap Hit	2013 DVP	2013 Surplus	Remaining Dead Money	Total Surplus to Date
SD	King Dunlap	$1775000	$5127540	$3352540	$-	$3352540
STL	Jake Long	$4250000	$5663833	$1413833	$-	$1413833
WAS	Jeremy Trueblood	$840000	$1203969	$363969	$-	$363969
NE	Will Svitek	$950000	$1274643	$324643	$-	$324643
NO	Jason Smith	$65000	N/A (Cut in Pre-Season)	-$65000	$-	-$65000
SD	Max Starks	$160000	N/A (Cut in Pre-Season)	-$160000	$-	-$160000
CLE	Rashad Butler	$555000	N/A (Cut in Pre-Season)	-$555000	$-	-$555000
ARI	Eric Winston	$2000000	$1124471	-$875529	$-	-$875529
DEN	Winston Justice	$991176	$34946	-$956230	$-	-$956230
CHI	Jermon Bushrod	$3015000	$1954533	-$1060467	$-	-$1060467
IND	Gosder Cherilus	$3900000	$2754756	-$1145244	$-	-$1145244
MIA	Tyson Clabo	$3500000	$1527310	-$1972690	$-	-$1972690

Finally, some green! King Dunlap was part of the Career Resurrection Tour that was the 2013 Chargers' offense, and Jake Long proved a welcome addition in St. Louis despite being lost to injury late in the season. Jeremy Trueblood and Will Svitek justified modest salaries while playing less than starter snaps. The middle of the chart represented straight zeros in DVP as the players weren't able to make it through final cuts in the preseason. We learned that the Chiefs might have been onto something when they soured on Winston, as he struggled to a subpar performance in the desert. Fellow rent-a-right-tackle Tyson Clabo fared just as badly in South Beach, while Jermon Bushrod and Gosder Cherilus managed to upgrade their respective O-lines while still failing to justify mid-range salaries.

The performance numbers:

	Run Snaps	Run Rating / Snap	Pass Snaps	Pressure Allowed / Snap
King Dunlap	354	5.6	333	5.7%
Jake Long	403	4	469	6.8%
Jeremy Trueblood	232	-1	409	8.1%
Will Svitek	89	-7.8	148	5.4%
Jason Smith	N/A (Cut in Pre-Season)
Max Starks	N/A (Cut in Pre-Season)
Rashad Butler	N/A (Cut in Pre-Season)
Eric Winston	474	-1.9	612	9.5%
Winston Justice	8	0	11	9.1%
Jermon Bushrod	440	-0.2	630	8.7%
Gosder Cherilus	436	-0.4	656	7.2%
Tyson Clabo	344	-1.7	618	8.4%

So What Does It All Mean?

It's pretty early to say just yet. In a sense, these 21 guys represent 21 individual data points, and by most measures that's a pretty scanty sample size for drawing substantive conclusions. Anecdotally, at least, this look at free agent tackles seems to confirm the idea that few elite guys ever hit the street (Jake Long's age and Miami's crappy cap situation in 2012 caused a divorce that might not have happened in other places) while highlighting the dicey prospects of making your money back as a team leaning on free agency.

In our next installment, we'll look at three years' worth of interior O-line signings to see if the picture gets any brighter.

*ProFootballFocus has their own method of deriving dollar value for a player's performance, known as the Jahnke Value Model. While we think this is a great model (and we put a lot of stock in PFF's game tracking and player performance data), we developed our own model to allow things like differential weighting of run and pass performance and assigning values to below-replacement-level performances.