Bayesian approach to analyzing goalies, Part 2: An example

In Part 1 of this series, we gave an introduction to this method of analyzing goalies and the motivation behind it.  We discussed why many analysts use even strength save percentage (ESSV%) instead of traditional SV%, and talked about two problems that ESSV% has which we'd like to deal with.  This series focuses on Problem 1: ESSV% tends to be very inconsistent especially for goalies who have faced a relatively small number of shots.  Our approach is a Bayesian analysis, where we can use "prior information".   

Let's begin with a simple example.  Let's consider the first 10 shots that Roberto Luongo faced during the 2008-09 season.  None of these shots were goals.  If this is the only information we have about Luongo, what should our estimate of his true ESSV% be?  In other words, what should we expect his ESSV% to be going forward?  We knew a lot about Luongo at the beginning of the 2008-09 season, but for the sake of explaining what's going on, let's pretend we didn't.

One approach to answering this question is to use his observed ESSV%. In this case, his ESSV% is 1.000 since he had 10 saves on these 10 shots.  This estimate is pretty high... 1.000 is way higher what any typical NHL goalie would ever have.  This is a pretty extreme example, since we are using only 10 shots, and it's sort of obvious that observed ESSV% should not be our estimate in this case.  But it will help us explain what is going on, and this kind of thing can still happen when using 100, 500, or 1000 shots, though the observed ESSV% won't typically be as extreme. 

Instead of using observed ESSV%, we could use a Bayesian estimate.  Suppose we know that the league's ESSV% are typically distributed like this:

 

The horizontal axis is ESSV% and the vertical axis indicates how likely each ESSV% is.  We expect an NHL goalie will hardly ever have an ESSV% less than .900 or greater than .940, and most of the time, ESSV% is between .910 and .930.   If we get a goalie that has an ESSV% much higher or lower than what is expected, then our method will tend to pull a goalie's ESSV% towards the mean.

(Note: There are different ways to get this curve, but let's pretend we got it from observed ESSV% for NHL goalies during the seasons in question.   This isn't exactly what we did, but we don't want to take the time to explain what we did just yet.) 

If we use this prior information in a particular way, we can get a Bayesian estimate of Luongo's true ESSV% after 10 shots that is much more conservative than his observed ESSV% of 1.000:

The red is our prior expectation for a goalie's ESSV% from the previous figure, and the black is our updated curve for Luongo after 10 shots.  The dots at the bottom are the mean for the prior (red) and the mean for Luongo (black). 

Note that this distribution curve for Luongo is very close to the prior distribution.  We only have 10 shots worth of information, and although Luongo had some pretty extreme results, there is not  overwhelming evidence that Luongo is much different than an average goalie. 

Also note that Luongo's curve is slightly shifted to the right, and that his dot is slightly to the right of the red dot.  This is expected since he saved 10 out of his first 10 shots.  Our estimate should be slightly higher than league average, since in this limited time period he had good results.  And that's what we see here.. slightly higher, but not much higher.

Luongo's curve is still pretty wide too.  A wider curve means we still aren't very sure about Luongo's ability.  In other words, we haven't narrowed down our beliefs much yet.  And we probably shouldn't after only 10 shots. 

What happens when we get more information?  Let's see what Luongo's curve looks like after 20 shots, 30 shots, etc., all the way up to 5230 shots:

(Issues uploading the 5200 shot version,
but here's the 4800 shot version for now... basically the same.)

We'll save the discussion of this animation for Part 3. 

Links to other parts:
Part 1 - Introduction
Part 2 - An example using only 10 shots
Part 3 - Updating estimates with more information
Part 4 - Regression to the mean, Luongo vs Schneider, Thomas vs Rask, Hiller vs Fasth