Review: Quantifying the importance of NHL shot differential

How important is shot differential really? Several analysts have looked at this several different ways and all came to similar answers.

By Eric T. Jan 28, 2014, 3:26pm EST

The Caps nervously celebrate upon being told their five-year PDO of 101.2 leads the league.

When I tell people I do hockey stats work, the response is often something like "Wow, neat. So what do the stats say about hockey?" My answer is typically something like this:

Goals are what really matter, so we'd like to evaluate players on how often they score goals or set up goals or prevent goals. But there are always random fluctuations -- players or teams can get hot for a little while and look better than they really are.

The issue is that shooting percentages fluctuate quite a bit. And with only a couple of goals per game, it turns out that it takes a lot longer than people naturally expect for those random fluctuations to even out.

Those shooting percentage fluctuations have a big impact on outcomes. Over tens of games, a team's results will be heavily driven by whether they happened to shoot for a higher percentage than their opponents did. But in today's NHL, nobody's been able to sustain a big edge in shooting percentage in the long run.

Because there are more than ten times as many shots as goals, the randomness in shot differential evens out a lot faster. And since the long-run differences in shooting percentages turn out to be modest, shot differential ends up being a big component of success.

If you want to predict future outcomes and only get to look at one stat, teams' current shot differential will help you a lot more than their current record or goal differential. Even over a month or two, teams' records can be quite deceiving, and a player's goal differential (plus/minus) for a whole year doesn't necessarily tell you much about his performance.

Before we go farther, I'd like to point out that I never said there's no such thing as shot quality. A lot of people come away with that impression, no matter how hard I try to clarify the distinction between "doesn't exist" and "isn't as important as people imagine".

But I've left an interesting question unanswered in this explanation. How important is shot quality -- is it almost as important as shot differential, almost completely inconsequential, or something in between? Let's take a quick review of numbers people have come up with.

Surveying the literature

Tom Awad used data from the NHL's play by play to estimate shot quality. His metric included shot location, rebounds, score effects, and a couple of other factors. He calculated how much of a team's goal differential could be explained by their shot differential and how much by their ability to control shot locations.

His conclusion was that "as an explanatory variable, it [shot quality] is 4 to 5 times less important than shot differential."

A couple of months ago, Phil Birnbaum took a completely different approach. Instead of trying to estimate the quality of each shot, he just looked at how well a team's shot differential or shooting percentage one year predicted their results in the following year.

I noted in a comment that the spread in team shot differential talent appeared to be three times as large as the spread in shot quality talent.

Gabe Desjardins has also looked into this by yet another method, using the observed results and probability calculations to estimate the skill component. I don't think he compiled his results in a single post, but we can tease it out from comments he made in a few places:

Relying on Tom's estimate of the spread of goaltending talent, Gabe said that goaltending is 5 percent of winning. JLikens came up with a nearly identical estimate for the spread of shooting talent, so we might infer that shooting is another 5 percent of winning, so the combined impact of shooting and goaltending is about 10 percent.

In that same article, Gabe said that Corsi and luck combine for 3/4 of winning, and he has shown that luck alone is about 38 percent. So we can infer that he estimates Corsi to be about 37 percent of winning -- about 3-4 times as important as shooting and goaltending combined.

It seems that no matter how you approach the problem, everything converges on something in this vicinity. Shot quality isn't insignificant, but it's clearly subordinate.

PDO and baseline expectations

Remember that in these studies, we were looking at shot quality over a period of 82 games or more. In the short run, shot quality varies much more than that, and can be a huge driver of results. But those short-term fluctuations don't have much predictive value; teams just don't seem to sustain a big edge in shot quality in today's NHL.

One common measure for this is PDO. PDO is the sum of a team's shooting percentage and save percentage; if a team shoots for a higher percentage than their opponents do, their PDO will be above 100.

In the long run, a team with good goaltending and/or good shooters will tend to have an above-average PDO. People have occasionally worked to calculate how much above average a team's expected PDO might be, based on their goalies' and/or shooters' career numbers.

Those numbers probably show a bit too large of a spread on expected performance because they didn't account for the role of random chance in the goalies' and/or shooters' career numbers, but the principle is reasonable and leads us to the conclusion that some teams really should be expected to have a PDO of 99 or 101 rather than 100.

Still, we often use PDO as a measure of luck in the short term. It's true that we don't necessarily expect everyone to be right at 100 in the long run, but when we use PDO to say that a team has gotten lucky, it's generally because they're way above 100.

When we're talking about Toronto at 104 or Colorado at 106, we can safely project significant regression without worrying about whether the team will be 99.6 or 100.7 in the long run. In our last 82-game season, there were some hilarious arguments about whether it was smart to bet on teams with high PDO over the first 20 games regressing.

Here's how that played out:

There is essentially zero correlation there; whether a team is at 104 or 96 through the first 20 games should have almost zero impact on your projection for the remainder of the season.

If we're discussing whether a team that's at 101 is due for regression, or trying to put an exact figure on the best possible projection for a team that's at 104, then it's important to factor in what we know about a team's shooting and goaltending talent. But by far the most common use of PDO is as a first screen to pick out the teams running so hot (or cold) that it would be crazy to expect them to continue at that level.

Even in the long run, when the randomness in shooting percentage has largely evened out, we still see shot differential being three times as important a factor as shot quality in today's NHL.

Basing a projection on a team's shooting and save percentages over the short run -- with all the attendant issues of randomness -- is a very shaky prospect.