Forecasting a goalie's future has proven to be quite difficult, and Brian Macdonald has published some really nice work on goalie analysis that I'd like to highlight.
From Cristobal Huet to Rick DiPietro to Ilya Bryzgalov, recent history is littered with laughably bad goalie contracts. My favorite example is the Tim Thomas double-bluff, in which his contract was briefly placed in the hysterically bad category, a year before he set the all-time record for highest save percentage.
Why are goalies so tough to predict? It's not because they're all crazy; it's a pretty simple issue of sample size.
A save percentage of .920 is a top-ten performance, while .910 would be nothing more than a good backup. But the difference between the two is just one save every 100 shots. That's an awfully narrow margin, and it takes an awfully large sample size to reliably identify differences that small. Much larger than you might expect -- several thousand shots, and even more if you want finer distinctions than top-ten versus good backup.
An 18th century minister's thoughts on goalies
So let's say we have a goalie who's posted a strong save percentage in his rookie year and we want to figure out how likely he is to keep it up. The best way to handle situations like this is usually with Bayesian analysis, a method that accounts for two things:
- How rare would it be to find a goalie who's really as good as the player in question seems to be?
- How rare would it be to see a performance like this just from a random hot streak?
Brian's done that in his series and has put together some beautiful illustrations of how it works. He's looking specifically at even strength save percentage, so the numbers are a bit different from the more common overall save percentage, but the principles are exactly the same. He's done a really nice job of explaining it, so I encourage you to read his series.
I'll just skip to the fantastic illustration of how our estimate of a goalie's true skill evolves with sample size. In this case, we're using Roberto Luongo as an example.
In this image, the red curve is Brian's estimate of the league-wide distribution of talent and the red dot is for the league-wide average. The blue dot is for Luongo's performance over the period in question. The black curve is for the estimated range of possibilities for his true talent, and the black dot is for the most likely guess of his true talent.
You can see that as he faces more shots, we get more confidence that what we've seen from him tells us how good he really is -- the black dot (our best estimate) moves closer to the blue dot (his observed performance), and the curve representing the range of possibilities gets narrower and narrower.
It's really nice work.
Neat. How do I do that?
Here's a do-it-yourself table. Brian kindly shared with me his estimate of the underlying talent distribution, which I used to calculate the following:
|Even strength shots faced|
The rows represent the even strength save percentage posted by the goalie you're interested in; the columns represent the number of shots he's faced.
For example, if your goalie has posted a .930 even-strength save percentage over 1,000 shots, you'd look at the .930 SV% row and the 1,000 shot column. There, you'd see "924±9", which means that we'd guess his true long-run talent is .924, and that there's a 95% chance he'll be within .009 of that (i.e. between .915 and .933). So you can use this table to estimate how confident your team should be in their young goalie, or which free agent they should pursue.
The fine print
There is one caveat about the assumed underlying distribution of talent. Brian is using a curve that is roughly bell-shaped, centered at about .922. This is fairly reasonable if we're talking about players who will get prolonged opportunities as starters or career backups.
The table above probably breaks down for replacement level guys -- this calculation will markedly overestimate the chances that the Alex Stalock's and Jeff Frazee's of the world are .922 goalies. Since we don't often get called on to forecast their careers, I'm not too worried about that; If you're bothering to wonder about a goalie's future, it's probably someone for whom the table above will apply reasonably well.
But Stalock and Frazee are examples of a broader issue that is worth thinking about. The reason they don't work out right is that this analysis doesn't use anything except the goalie's NHL history to date, and in the absence of data to the contrary, it assumes that the goalie is league average. But for Stalock and Frazee, things like their AHL performance and depth chart position tell us they're probably much worse than that, so in reality our prior assumption on them is very different from what this calculation assumes.
In a similar vein, pre-NHL performance can make us more optimistic about a goalie than we might otherwise be, as with (for example) Viktor Fasth's .929 save percentage on almost 3,000 shots in Sweden, or Jonathan Bernier's .927 on over 3,000 AHL shots. In general, these corrections will be smaller -- it's easy to see how a goalie's AHL performance would make you quite sure that he's not going to make it in the NHL, but it's harder for his AHL performance to make you sure that he will.
Still, it's worth bearing in mind that while NHL performance is the most useful evidence of how good a goalie is, it's not the only factor.
I think it's important to acknowledge these minor caveats so that those who pursue this line of analysis can be as thorough as possible. Nevertheless, I don't want to distract from the major point here: Brian has written a very nice series laying out a strong approach to evaluating goalies, and has done it in an easy-to-read style with some fantastic graphics.