A review of analytics-based personnel assessments

I've written a few times about the importance of putting statistics to the test, of assessing whether they correlate with future results.

To do that, you might imagine making note of what the statistic says now and waiting a year or two to see how the future pans out. But we generally want results faster than that, so we take a shortcut. We look back to the past and ask questions like "ok, given the data we had in 2011, what would we have predicted for 2012?"

This is a very good thing to do, and I don't mean to discourage it. But it's not really the same as having a prediction on the record for testing.

There are a few prominent, oft-cited instances where stat-based predictions panned out -- the decline of the Stars in 2011 or the Wild in 2012 come to mind, and this year's Leafs are the latest closely-watched test case. But that's just a couple of examples; it'd be fair for the skeptic to ask how many bad predictions are being overlooked.

Like any decent stat guy, I don't want to rely on a couple of anecdotes. I wanted to take a more complete look at whether using our non-traditional stats really helps us make practical assessments.

Do the stats really give me an edge on the casual fan? Can they put me anywhere near an actual NHL team's evaluation skills?

Examining the record

I wanted to make my study comprehensive, but trying to find every hockey opinion I've ever had is obviously hopeless. I needed an organizing principle.

I'm a Flyers fan and a writer at Broad Street Hockey, so Flyers transactions will be a common source of commentary for me. The website flyershistory.com lists every transaction in Flyers history; I went back to when I joined Twitter in June of 2011 and for each transaction, searched for what opinions I put on the record, either on Twitter or on Broad Street Hockey.

What follows is a list of every occasion where I could find documented evidence that I disagreed with the team's move. The team, with a cadre of scouts and experts who know the game much better than I ever will, ought to crush me. If I can hold my own with a pro sports franchise -- a reasonably successful one at that -- then that says a lot about the value of the stats I rely on.

So how'd I do?

I have one clear loss: thus far, Mason has been more than worth the contract I didn't like.

There are two that are debatable. James van Riemsdyk has lived up to the contract that I didn't like, but given how disappointing his '11-12 season was, I still believe the team would have done even better if they'd taken a harder line in negotiations and waited until after that year to sign him if necessary. And we'll have to wait and see how Streit ages.

There are a couple that I think are very likely to be losses. Scott Hartnell's aging curve remains to be seen, but since the team hasn't been thrilled with his subsequent performance at ages 30-31, I can't imagine they're celebrating having him signed through age 36. The organization probably wouldn't agree with my readiness to write off Rosehill, but right now with him buried in the press box, that second year on the contract looks quite pointless.

The clutch save

There are times when a team needs the goalie to bail them out. Can we tell which goalies are most likely to rise to the occasion?

And then there are several that I think are clear wins. Sestito and Leighton played a combined 100 minutes in a Flyers uniform. Van Riemsdyk has put up exactly the numbers I expected of him, and Schenn still hasn't proven to be more than a third-pair defenseman.

And then there's Bryzgalov.

Even in the most charitable view of the record, I don't think you could conclude that the team has outperformed me. That's actually a pretty amazing testament to the power of the stats when you think about how much more the team sees of these players, how much more they know about their roles and their skills and about hockey in general.

I don't think I'm unique in this regard, either. I've used my own record because I know the most about where to find my historical comments, but I think we'd find a similar result if we analyzed Tyler Dellow's comments about the Oilers or the comments about the Maple Leafs at Pension Plan Puppets. Copper and Blue has had good success in this kind of test before, as has Jonathan Willis.

The point here isn't to congratulate myself; honestly, I don't think I've done anything special. Stat guys all over the internet were critical of the Schenn trade and the Leighton and Bryzgalov contracts the instant they occurred. But that's the point -- if all it takes to hold your own with a team full of hockey professionals is a basic understanding of hockey analytics, then the analytics must be awfully powerful.

My record isn't perfect, and neither are the analytics. Nothing ever will be. But the historical record of stat-based assessments is quite strong, and there's a good reason an increasing number of teams are starting to investigate their use.

More from SB Nation NHL:

• Winter Classic : Leafs win in shootout | Old lady dancing

• The Rink of Dreams: One Mich. family created an outdoor hockey paradise

• #Buttgoal | Did the officials get the call right?

• USA men’s Olympic roster revealed