PECOTA, an acronym for Player Empirical Comparison and Optimization Test Algorithm, is a sabermetric system for forecasting Major League Baseball player performance. The word is a backronym based on the name of journeyman major league player Bill Pecota, who, with a lifetime batting average of .249, is perhaps representative of the typical PECOTA entry. PECOTA was developed by Nate Silver in 2002–2003 and introduced to the public in the book Baseball Prospectus 2003.Nate Silver, "Introducing PECOTA," in Gary Huckabay, Chris Kahrl, Dave Pease et al., Eds., Baseball Prospectus 2003 (Dulles, VA: Brassey's Publishers, 2003): 507–514. Baseball Prospectus (BP) has owned PECOTA since 2003; Silver managed PECOTA from 2003 to 2009. Beginning in Spring 2009, BP assumed responsibility for producing the annual forecasts, making 2010 the first baseball season for which Silver played no role in producing PECOTA projections.Nate Silver and Kevin Goldstein, "State of the Prospectus: Spring 2009," BaseballProspectus.com, March 24, 2009 .
One of several widely publicized statistical systems of forecasts of player performance, PECOTA player forecasts are marketed by BP as a fantasy baseball product. Since 2003, annual PECOTA forecasts have been published both in the Baseball Prospectus annual books and, in more detailed form, on the BaseballProspectus.com subscription-based website.Illustrative PECOTA estimates and "cards" are available for inspection by nonsubscribers here: http://www.baseballprospectus.com/pecota/. PECOTA also inspired some analogous projection systems for other professional sports: KUBIAK for the National Football League, SCHOENEKevin Pelton, "Introducing SCHOENE: Our NBA Projection System," BasketballProspectus.com (October 20, 2008) and CARMELO for the National Basketball Association, and VUKOTAThomas Awad, "Introducing VUKOTA," PuckProspectus.com (July 20, 2009). for the National Hockey League.
PECOTA forecasts a player's performance in all of the major categories used in typical fantasy baseball games; it also forecasts production in advanced sabermetric categories developed by Baseball Prospectus (e.g., VORP and EqA). In addition, PECOTA forecasts several summary diagnostics such as breakout rates, improve rates, and attrition rates, as well as the market values of the players. The logic and methodology underlying PECOTA have been described in several publications, but the detailed formulas are proprietary and have not been shared with the baseball research community.
Methodology Silver described the inspiration for his approach as follows: The basic idea behind PECOTA is really a fusion of two different things – [Bill] James's work on similarity scores and Gary Huckabay's work on Vlad, [Baseball Prospectus's] previous projection system, which tried to assign players to a number of different career paths.Gary Huckabay, "6–4–3: Reasonable Person Standard," BaseballProspectus.com, August 2, 2002. I think Gary used something like thirteen or fifteen separate career paths, and all that PECOTA is really doing is carrying that to the logical extreme, where there is essentially a separate career path for every player in major league history. The comparability scores are the mechanism by which it picks and chooses from among those career paths.Rich Lederer, "An Unfiltered Interview with Nate Silver," Baseball Analysts, February 12, 2007.
Comparable players PECOTA relies on fitting a given player's past performance statistics to the performance of "comparable" Major League ballplayers by means of similarity scores. As is described in the Baseball Prospectus website's glossary:
PECOTA compares each player against a database of roughly 20,000 major league batter seasons since World War II. In addition, it also draws upon a database of roughly 15,000 translated minor league seasons (1997–2006) for players that spent most of their previous season in the minor leagues. ... PECOTA considers four broad categories of attributes in determining a player's comparability:
1. Production metrics – such as batting average, isolated power, and unintentional walk rate for hitters, or strikeout rate and groundball rate for pitchers.
2. Usage metrics, including career length and plate appearances or innings pitched.
3. Phenotypic attributes, including handedness, height, weight, career length (for major leaguers), and minor league level (for prospects).
4. Fielding Position (for hitters) or starting/relief role (for pitchers). ... In most cases, the database is large enough to provide a meaningfully large set of appropriate comparables. When it isn't, the program is designed to 'cheat' by expanding its tolerance for dissimilar players until a reasonable sample size is reached.
PECOTA uses nearest neighbor analysis to match the individual player with a set of other players who are most similar to him. Although drawing on the underlying concept of Bill James' similarity scores, PECOTA calculates these scores in a distinct way that leads to a very different set of "comparables" than James' method.This difference is explained and illustrated in Nate Silver, "Introducing PECOTA," Baseball Prospectus 2003, cited above. Furthermore, Silver describes the following distinct feature: The PECOTA similarity scores are based primarily on looking at a three-year window of a pitcher’s performance. Thus, we might look at what a pitcher did from ages 35–37, and compare that against the most similar age 35–37 performances, after adjusting for parks, league effects, and a whole host of other things. This is different from the similarity scores you might see at baseball-reference.com or in other places, which attempt to evaluate the totality of a player’s career up to a given age.http://www.baseballprospectus.com/unfiltered/?p=136. Also see Baseball Prospectus' glossary entry for "Comparable Players". Once a set of "comparables" is determined for each player, his future performance forecast is based on the historical performance of his "comparables". For example, a 26-year-old's forecast performance in the coming season will be based on how the most comparable Major League 26-year-olds performed in their subsequent season.
Separate sets of predictions are developed for hitters and pitchers.
Peripheral statistics PECOTA also relies a lot on the use of peripheral statistics to forecast a given player's future performance. For example, drawing on the insights coming out of the use of defense-independent pitching statistics, PECOTA forecasts a pitcher's future performance in a given area by using information about his past performance in other areas.See PERA for an example of the use of peripheral statistics to estimate a performance. As baseball analyst and journalist Alan Schwarz writes, "Silver ... designed a sophisticated variance algorithm that has examined every big-league pitcher's statistics since 1946 to determine which numbers best forecast effectiveness, specifically earned run average. His findings are counterintuitive to most fans. 'When you try to predict future E.R.A.'s with past E.R.A.'s, you're making a mistake,' Silver said. Silver found that the most predictive statistics, by a considerable margin, are a pitcher's strikeout rate and walk rate. Home runs allowed, lefty-righty breakdowns and other data tell less about a pitcher's future".Alan Schwarz, "Numbers Suggest Mets Are Gambling on Zambrano," New York Times, August 22, 2004.
Probability distributions Instead of focusing on making point estimates of a player's future performance (such as batting average, home runs, and strike-outs), PECOTA relies on the historical performance of the given player's "comparables" to produce a probability distribution of the given player's predicted performance during the next five years. Alan Schwarz has emphasized this feature of PECOTA: "What separates Pecota from the gaggle of projection systems that outsiders have developed over many decades is how it recognizes, even flaunts, the uncertainty of predicting a player's skills. Rather than generate one line of expected statistics, Pecota presents seven – some optimistic, some pessimistic – each with its own confidence level. The system greatly resembles the forecasting of hurricane paths: players can go in many directions, so preparing for just one is foolish".Alan Schwarz, "Predicting Futures in Baseball, and the Downside of Damon," New York Times, November 13, 2005. Silver has written, This procedure requires us to become comfortable with probabilistic thinking. While a majority of players of a certain type may progress a certain way – say, peak early – there will always be exceptions. Moreover, the comparable players may not always perform in accordance with their true level of ability. They will sometimes appear to exceed it in any given season, and other times fall short, because of the sample size problems that we described earlier.
PECOTA accounts for these sorts of factors by creating not a single forecast point, as other systems do, but rather a range of possible outcomes that the player could expect to achieve at different levels of probability. Instead of telling you that it's going to rain, we tell you that there's an 80% chance of rain, because 80% of the time that these atmospheric conditions have emerged on Tuesday, it has rained on Wednesday.
Surely, this approach is more complicated than the standard method of applying an age adjustment based on the 'average' course of development of all players throughout history. However, it is also leaps and bounds more representative of reality, and more accurate to boot.Nate Silver, "Baseball Prospectus Basics: The Science of Forecasting," BaseballProspectus.com, March 11, 2004.
Team effort Although Silver was the creator of PECOTA, producing PECOTA forecasts was a team effort: "I might be 'the PECOTA guy,' but it very much is a team effort," Silver has said of the BP staff. "We all do it. It's my baby, but it takes a village to run a PECOTA".William Hageman, "Baseball By the Numbers," Chicago Tribune, January 4, 2006. For example, PECOTA draws on Clay Davenport's translations (the so-called Davenport Translations or DT's) of minor league and international baseball statistics to estimate the major league equivalent performance of each player.See Clay Davenport, "DT's vs. MLEs — A Validation Study," BaseballProspectus.com, January 30, 1998; Clay Davenport, "Winter and Fall League Translations: Just How Good Are These Leagues, Anyway?," BaseballProspectus.com, January 27, 2004; and Clay Davenport, "Over There! A Second Review of Translating Japanese Statistics, and Translating the Mexican League," Baseball Prospectus 2004 (New York: Workman, 2004): 585–590. In this way, PECOTA is able to make projections for more than 1,600 players each year, including many players with little or no prior major league experience.
The 2009 preseason forecasts were the last ones for which Silver took primary responsibility. In March 2009, Silver announced that PECOTA's extremely complex and laborious set of database manipulations and calculations would be moving to a different platform. Although Baseball Prospectus had been the owner of PECOTA since Silver sold it to them in 2003 – and Silver stewarded and took responsibility for the forecasts – henceforth PECOTA forecasts would be generated by the Baseball Prospectus team, initially with Clay Davenport in charge of the effort,See, for example, Clay Davenport, "Depth Charts," BaseballProspectus.com, May 13, 2009. and later, through the 2013 season, with Colin Wyers heading up both production and improvements in PECOTA.
Alternative forecasting systems