Recipe for Regression: a case study in bullpens

A good bullpen is a wonderful thing-look at where it got the Rays and the Phillies this year. The bullpen plays an undeniably major role in many teams' success and is often the piece that vaults them to the next level. It's also true, however, that compared to the starting rotation and the line-up, the bullpen provides far less value. Tom Tango estimates that only about 10% of the value provided to a team comes from the bullpen, right about the value Win Shares gives it. The relatively low value of the bullpen makes it difficult to build a team around long-term, but it is still possible to ride a hot bullpen to short-term success. The problem with that is that the high returns teams get out of these bullpens are usually not sustainable over multiple years, and the value of even a good bullpen is bound to come back down to earth.

Over the past 5 paired seasons, teams that ranked in the top 5 in baseball in WPA (win probability added-essentially how many wins they contributed over the year) by their bullpens one year averaged 7.43 wins from the bullpen in the year they were in the top 5 and dropped an average of 6.03 wins the following year. So teams with great bullpens one year carried over less than a fifth of the win value over average the next. Of the 25 teams to be in the top 5 in bullpen WPA from 2003-07, only 5 (20%) were able to stay in the top 5 the following year (for comparison, 17% would stay in the top 5 just by random chance), and 21 (84%) regressed the following year. So a great bullpen, while often a key ingredient to a successful team, generally does not hold extreme value from one year to the next.

This is of particular interest to this year's big surprise team, the aforementioned Rays. There was a lot of talk last year about the Rays' "different hero every night" on offense and stellar starting pitching, but the #1 reason they were where they were was their bullpen. The rotation, while it was good, wasn't even the best in the division-Toronto's staff earned that distinction-and was pretty comparable to Boston's staff. The offense was pretty much average and certainly worse than the Sox' line-up. Where they really distinguished themselves and made up that ground was in the bullpen.

Fangraphs lists the Rays' team bullpen WPA as 9.30, the best in baseball. That's more than double the wins contributed by their offense and starting pitching combined. With an average bullpen, they were basically the Blue Jays (about an 86 win team). Luckily for the Rays, they didn't have an average bullpen. Wins are wins no matter where you get them, and 9 or 10 wins from the bullpen are just as good to the Rays as getting those wins from anywhere else.

To the 2008 Rays, that is. What we want to know is what that does for the 2009 Rays. Can they repeat their 08 performance? Chances are, as shown above, no. Where does all that production go, though? It can't just disappear into thin air. So let's look at where the production came from and try to figure out what can happen to turn a great bullpen into a fairly pedestrian one.

The Rays' bullpen did abnormally well in high leverage situations when it counted more last year. Their WPA/LI, which measures what the pen did with every at bat equally weighted, was significantly lower than what you'd expect from a bullpen that won that many games, meaning the bullpen was not pitching well enough to win nearly that many games in general. By multiplying the bullpen's WPA/LI by its LI*, we can get how many wins they should have contributed based on how well they actually pitched, regardless of situation. Doing this leaves the expected win value of the Rays' bullpen 4 wins shy of their actual win value. Again, great for the 2008 Rays, but not such a good sign for the 2009 Rays.

Chances are, if you believe in clutch performance, you're shaking your head by now. The Rays won with a great clutch bullpen performance in 2008, and maybe they just have great clutch pitchers. You know, guys like Jason Hammel, J.P. Howell, Scott Dohmann, and Gary Glover (their top 4 relievers in outperforming their season numbers in high-leverage situations). Not exactly the best group of pitchers, but part of the Rays' success was that they got so many contributions from the weaker back end of their pen, so maybe these guys' best skill is their clutch ability, and maybe they really can repeat it. Let's check out their career WPA-(WPA/LI)xLI* before this year, and then in 2008-

WPA over expected

	career	2008
Hammel	-.81	.93
Howell	-1.02	1.45
Dohmann	-1.29	.31
Glover	-1.50	.28
combined	-4.62	2.97

It doesn't look like any of these pitchers has any repeatable clutch skill. In fact, every one of them had done worse in higher leverage situations over his career before this year, when they combined to contribute 3 wins over the expected value based on how well they pitched. Glover, who somehow managed to contribute .21 wins above average in 34 innings for the Rays, was cut mid-season and went to Detroit, where he managed to post a WPA of -1.07 in just 20 innings. That, in case you're not familiar with the relative scale, is horrible. In part, it was avoiding those kind of performances from their worst pitchers that made the Rays so successful.

Judging from where their clutch performances came from, it doesn't seem likely that they can come close to repeating them. So that's bad news to the tune of 4 wins for the 09 Rays, even if they pitch just as well. Even that could be difficult for them, though. They not only had pitchers contributing significantly beyond their performances, they had pitchers performing well beyond their career norms. The most exaggerated example of this is Grant Balfour, who went from being a walking punch-line (7.3 BB/9 in 24.2 IP in 2007 for a guy with that name?) to one of the best relievers in baseball in 2008.

Balfour
	IP	K/9	BB/9	K/BB	ERA	WPA	WPA/LI	exWins**
pre-2008	92.2	10.10	5.63	1.79	5.44	-0.73	-0.76	-0.87
2008	58.1	12.65	3.7	3.42	1.54	2.69	1.79	2.24
Bill James '09	74.0	12.04	4.50	2.68	3.01	x	x	1.28
Marcel '09	57.0	9.16	3.95	2.32	3.55	x	x	0.57

Marcel is the more pessimistic projection here, but both projections have Balfour regressing quite a bit next year (though not nearly to his pre-2008 self). His regression, based on his '09 projections for ERA and IP, is likely to knock 1-1.5 wins off his 2008 value. He also picked up an extra half a win by pitching disproportionately well in higher leverage situations, as discussed above, so it would be well within reason for Balfour to be worth a full 2 wins less next season. As a player with little Major League history, injuries that sidetracked his career, and one big break-out season under his belt, on top of just being a reliever in general, he is about as difficult to project a player as you'll find, though, so he could also be better than that. Still, there is very little chance of him posting another 1.54 ERA.

If we apply the same exWins formula to the 2009 projections of Tampa's top 3 relievers from 2008, we can see how many wins difference to expect from all 3 of them. Here are the differences in their 2008 seasons and '09 projections, along with the difference in WPA in 2008 when distributed evenly across all leverage situtations (dist.).

exWin differences b/w 2008 and '09 projections

	James	Marcel	dist.
Balfour	-0.96	-1.67	-0.43
Howell	-2.54	-2.22	-1.45
Wheeler	-0.30	-0.61	0.24
combined	-3.80	-4.50	-1.64

Between the three of them, Tampa can likely expect a drop-off in the range of 5.5-6 wins. The specific distribution among the three could change if Balfour or Howell get more closing opportunities and Wheeler gets moved to a set-up role, but that's basically just shifting their leverage indexes around without changing the overall picture much.

Throughout their bullpen, they had guys outperforming their histories, and they didn't really have anyone flop on them. The chances of their whole bullpen repeating that are pretty slim. Guys are going to fall on the wrong side of their projections next year, and they aren't going to get such great over-performances. That doesn't mean they can't compete next year, but if they are going to contend with the Sox and the rest of the AL, they'd better start finding production in other places, and fast.

*Multiplying WPA/LI by LI would seem like it would just give you WPA, but it doesn't because WPA/LI is it's own separate stat calculated separately from the season WPA total and average LI. WPA/LI takes the WPA of every individual play and divides it by the LI for that play, and then adds those together. LI on its own is just the average LI of all plays. WPA/LI is a better measure of the intrinsic value of a player's performance, as it considers all situations equally, whereas WPA gives you a better idea of the actual effect a player had on his team's games. WPA/LI x LI is a good balance between the two.

**exWins is the number of wins above average expected from a pitcher using a pythagorean approach with his ERA as the runs allowed and the league average ERA as the runs scored. The formula is explained in one of Tango's posts on the linked article.