Baseball batting averages throw some curves at statisticians

“I had many years that I was not so successful as a ballplayer, as it is a game of skill.”
-- Casey Stengel (from testimony before United States Senate Anti-Trust and Monopoly Hearing, 1958)

Last week the University of Minnesota School of Statistics sponsored a talk titled “In-season prediction of batting averages: A field test of empirical Bayes and Bayes methodologies,” presented by Lawrence D. Brown from the Statistics Department of Wharton School at the University of Pennsylvania. My colleague Pat Whitcomb attended and told me about a few findings by Brown that baseball fanatics like me would find a bit surprising:
“The simplest prediction method that uses the first half [of a season’s] batting average …performs worse than simply ignoring individual performances and using the overall mean of batting averages as the predictor for all players.”*

Evidently these professional players perform at such a consistent level that the ones hitting at a higher than average rate up until the mid-season break tend to regress back to the mean the rest of the way, and vice-versa.

Of course, by looking at many years of past performance, one would gain some predictive powers. For example, in 1978, more than ten years into his Hall of Fame (HOF) career, Rod Carew batted .333 for the Minnesota Twins. He made it to the Major Leagues only a few years ahead of fellow Twin Rick Dempsey, who hit at an average of .259 in 1978. Carew finished up his 19-year playing career with a lifetime batting average (BA) of .328, whereas Dempsey hung on for an astounding 24 years with a BA of only .233! It would not require a sabermetrician to predict over any reasonable time frame a higher BA for a HOF ballplayer like Carew versus a dog (but lovable, durable and reliable defensively at catcher) such as Dempsey.

Brown also verifies this ‘no brainer’ for baseball fans: “The naıve prediction that uses the first month’s average to predict later performance is especially poor.” Dempsey demonstrated the converse of this caveat by batting .385 (5 for 13) for his Baltimore Oriole team in the 1983 World Series to earn the Most Valuable Player (MVP) award!

Statistical anomalies like this naturally occur due to the nature of such binomial events, where only two outcomes are possible: When a batter comes to the plate, he either gets a hit, or he does not (foregoing any credit for a walk or sacrifice). It is very tricky to characterize binomial events when very few occur, such as in any given Series of 4 to 7 games. However, as a rule-of-thumb the statistical umpires say that if np>10 (for example over 50 at-bats for a fellow hitting at an rate of 0.200), the normal approximation can be used for binomial distributions and the variance becomes approximately p(1-p)/n.** From this equation one can see that the bigger the n, that is – at-bats, the less the fraction (batting average) varies.

PS. I leave you with this paradoxical question: Is it possible for one player to hit for a higher batting average than another player during a given year, and to do so again the next year, but to have a lower BA when the two years are combined?

*Annals of Applied Statistics, Volume 2, Number 1 (2008), 113-152

**This Wikipedia entry on the binomial distribution says that “this approximation is a huge time-saver (exact calculations with large n are very onerous); historically, it was the first use of the normal distribution, introduced in Abraham de Moivre's book The Doctrine of Chances in 1733.”