How do we know if NHL analysis is any good?

Suppose you decide to renovate your bathroom. You're putting in a whole new shower stall, and you're going to need to find a plumber. You call around, meet with a few people, get quotes, and then what?

You'll know who returns phone calls promptly, whose schedule works well for you, who's easy to talk to. But it's awfully hard to tell at that point who's actually good at the job. You can read reviews online, but you don't necessarily know whether any of the customers know any more than you do about how to judge plumbers.

In an area where you're not an expert, it can be awfully hard to tell who is.

More Outnumbered

• Projecting scoring rates • The be all, end all stat • Does last year matter? • Track record vs. Hot streak • Ryan Suter's incredible workload

Searching for flaws

I had a conversation about this challenge at the Sloan Sports Analytics Conference this year. Someone from a team said that when analysts approach them, it's easy enough for them to make a rough cut to remove the obviously unqualified, but hard for them to differentiate beyond that.

He asked me what they should look for. My answer might be of use to fans as well as teams.

What I think really separates the best analysis is a creative search for ways the analysis could be wrong. Hockey doesn't often provide us with clean experiments, so there are almost always confounding factors to consider or ways that cause and effect can be difficult to tease apart.

Problems big and small

In some cases, those factors are so strong that the initial result becomes obviously nonsensical and the conclusion that something is missing is unavoidable.

If you try to assess the impact of quality of competition on Corsi by correlating the two, you find that facing stronger competition correlates with higher Corsi. Of course, that's not because it's easier to outshoot better competition; it's because better players tend to (on average) have a good Corsi and face good opponents.

Or if you try to assess the impact of fatigue on scoring rates, you find that when forwards' ice time increases, they actually score more points per minute. That's not because they get stronger with more minutes; it's because good results lead to them getting more ice time.

When an issue like this has such a big impact that the result is obviously wrong, anyone would catch it. And at the other extreme, if the issue has a negligible effect, it doesn't matter whether we try to control for it. But there's a vast middle ground where issues can have a significant-but-not-obviously-destructive impact, and that's dangerous.

What you want to see from an analyst is that they work through as many of these issues as they can. They control for the things they can, discuss the possible impact of the ones they can't, and are open to suggestions about things they might have missed.

Extraordinary results require extraordinary proof

With that being said, not every piece needs (or benefits from) excruciating scrutiny. There are some holes in conventional wisdom, but in general, conventional wisdom isn't all that bad. If a piece of analysis seems to confirm traditional thought, the odds are that they're probably both right and I probably wouldn't spend a ton of effort trying to find the holes.

But if a result is very surprising, we ought to be suspicious of it and expect very strong proof. We have to be open to change and willing to accept that what we've previously believed might be wrong, but it's fair to hold out for some pretty compelling evidence.

One part of "compelling evidence" is mathematical; we should expect a high degree of statistical significance. But arithmetic alone won't be able to identify potential confounding factors; quality analysis requires good judgment and creative thinking to go with the math.

At this stage in hockey's development, the models are still pretty simple and you don't need a deep background in statistics theory to make contributions, but you do need a good understanding of how hockey works and how its particular complexities can impact the analysis.