Our Approach to Rating Players 

Our philosophy

The most important point we can make about our approach to rating players is that we rate players based on performance, not reputation. This philosophy influences the meaning of our ratings and how we come up with them in the first place.

Our outfielder throwing rating, for example, measures the fielder's ability to prevent runners from taking extra bases and to throw them out when they try to advance. A fielder earns a good rating by positioning himself well, getting to the ball quickly, making a quick release, throwing with power, throwing with accuracy, and throwing to the right base. Someone with a powerful arm might still get a subpar rating if he doesn't get into throwing position quickly, throws wildly, or often throws to the wrong base.

Similarly, our running rating is not just a measure of raw speed, but also the ability to read the ball off the bat, get a good jump, and make good decisions about when to go for the extra base. Speed does matter, and you'll find plenty of speedsters with top baserunning ratings. But there are other players who run the bases well because their instincts are so good, and some swift runners who don't make the most of their chances.

And our rating for defensive range measures the ability to make plays. An infielder earns a good rating through positioning, quickness, soft hands, and effective throwing (quick release, arm strength, accuracy). It's not always the flashiest player who makes the most of the balls that are hit his way.

In other words, the things you can see when you watch them play, and the things the scouts often talk about, don't always translate into results. Sometimes the things you cannot see -- such as the mental aspects of the game -- are as important or more important in how the player actually performs in game action. Our results-oriented analysis often uncovers players with good instincts who accomplish a lot more than the scouts might have predicted based on their raw tools.

The role of analysis

Because our ratings measure overall ability to succeed in a certain phase of the game, we look for ways to evaluate performance by analyzing play-by-play data. This approach is not a radical one. Baseball people have been doing this for over a century to measure batting and pitching performances. We don't, after all, give the batting title to the guy with the prettiest swing, we give it to the player who hit for the highest average. We don't give the Cy Young to the pitcher with the best mechanics, we give it to the one who was most effective. Using statistics to evaluate performance is part of the tradition of the game.

But this tradition usually extends only to hitting and pitching. You never hear a television or radio analyst talk about meaningful measures of baserunning, throwing or defense. Instead, they talk about their impressions of the player -- how fast he looks, his quickness, strength and athleticism. If we applied the same standard to hitters, we'd never talk about on-base percentage or slugging average. If we did the same with pitchers, we'd never talk about walk-strikeout ratios and hits per nine innings.

Our approach is to apply the time-honored tradition of using well-crafted statistics to evaluate baseball performance. The difference is that we don't stop at hitting and pitching. We design ways to measure results in all phases of the game.

We sometimes find players whose performance is better or worse than you would guess by watching them a few times a year. And our ratings are occasionally at odds with the opinions being expressed by some of baseball's most famous writers and TV personalities. But we sincerely believe that doing original research into player skills is an important part of producing an accurate baseball simulation.

Suppose a player has a reputation for great defense but our analysis doesn't show a superior performance. If we give in to public opinion and rate him higher than his performance justifies, we'd have these options:

• reduce the range rating of one of his teammates so the team's defense isn't overrated
• reduce the effectiveness of the team's pitchers to compensate for the extra plays this player will now make in the simulation
• disregard these side effects and allow the player, the team, and its pitchers to produce better results than they should

We don't think it's fair to downgrade teammates so we can give a popular player a better rating than he deserves. And we don't think you'd want us to disregard the side effects and publish a season database with players and teams who will overperform. So we do our best to rate players based on performance, even if that means we might take a little heat for a few of our ratings.

Bunting Ratings

Assigning bunt ratings would be easy if all we had to do was look at the leaders in sacrifice bunts each year. But it's not that simple, because a player's batting stats don't tell us how many times he failed to get a bunt down when called upon. And we have separate ratings for sacrificing and bunting for a hit, and if you're limited to the official batting stats, there's no way to tell how many bunt singles a player legged out this year.

So we wrote a program to compile bunting information from the play-by-play and pitch-by-pitch data we work with each year. This program looks at three situations -- bunting for a hit, sacrifices, and squeeze bunts -- and counts the number of opportunities to bunt, the number of bunt attempts and the results of those attempts. The results include separate categories for bunt hits, reaching on an error, being put out at first with other runners advancing, popping out, grounding into a fielders choice in which a lead runner is retired, grounding or popping into a double play, and fouling off one or more bunt attempts before swinging away.

The number of bunts fouled off is quite important. Typically, in ordinary sacrifice situations, batters are successful in advancing one or more runners more than 85% of the time. Provided they got the ball in play, that is. But roughtly 35% of the time, they bunt foul once or twice before swinging away. Counting this type of failure drops the success rate to 57%. In other words, there are many more bunting "failures" due to fouled bunts than to poorly-placed bunts, and we would be missing a lot of important information if we tried to measure bunting ability without considering fouls.

So our bunt ratings take all of these factors into consideration. And it sometimes means that a player with loads of successful sacrifices is given a mediocre bunt rating because he also failed on a lot of attempts.

Running and throwing

There are no official statistics for baserunning, though you may occasionally see some data on how often runners go from first to third on a single. Our studies examine how often runners took extra bases on hits and fly balls, taking into account where the ball was hit (it's much easier to go to third on a ball hit to right), the number of outs (getting the jump with two outs makes a big difference), and whether the playing surface is grass or turf (ground balls reach the outfielder more quickly on old-style artificial turf, so it's harder to take the extra base on many singles.) We also look at how often each runner was thrown out on the bases.

Before wrapping up our running ratings, we also review a report showing the number of times players were used as or replaced by a pinch runner. If a manager frequently uses player A to run for player B, it's a good indication that the manager feels A is a much better runner. It doesn't mean that A is a great runner or that B is a lousy runner. It could be an average runner replacing a poor one, or it could be an excellent runner replacing an average one. That's why we look at who's replacing whom, not just the raw count of pinch-running appearances or replacements.

Outfield assist totals give us some insight into outfielder throwing, but they can also be misleading. Some great throwing outfielders have their assist totals cut down by their opponent's unwillingness to run on them in the first place. And not all assists are created equal. Some assists result when a throw nails a lead runner, while others occur when the lead runners score but the batter is out at second advancing on the throw. So we've written a program that allows us to see how many quality assists each outfielder earned during the season.

Our throwing ratings focus on the ability to prevent the lead runners from advancing (and to throw out those who try) using a combination of skills: getting to the ball quickly, getting the throw off quickly, and throwing with strength and accuracy. Some players with strong arms are not very accurate. Some players with weak arms compensate by getting into throwing position more quickly. As with all of our ratings, we measure results, not just how often a player looks good making a throw.

Stolen base defense

When rating pitchers and catchers for the ability to prevent stolen bases, it's important to go beyond looking at simple statistics such as the percentage of runners thrown out by a catcher. Every play starts with the ball in the pitcher's hand, and some studies suggest that pitchers may be twice as important as catchers in preventing steals.

It's entirely possible for a catcher to look good or bad, not because of his own ability to throw, but because of the pitchers he is working with. Likewise, a catcher with a cannon for an arm can make his pitchers look good.

So our base stealing studies measure performance by pitcher-catcher tandems. By comparing a pitcher's results with different catchers and vice versa, we learn more than we would if we merely looked at totals for the pitchers and catchers alone.

We also log all stolen base attempts, so if there's a question about the rating of a particular pitcher and catcher, we can see who was stealing bases against them. If, by chance, the league's best runners were on base a lot in your games, it's hard for you to look good. On the other hand, if you were giving up stolen bases to runners who don't run all that often, it can mean that you're viewed as easy pickings.

Measuring Defensive Range

Defensive playmaking ability -- turning batted balls into outs -- is one of the hardest baseball skills to evaluate.

Traditional fielding statistics -- putouts, assists, and errors -- tell us how many plays a fielder was involved in, but they don't tell us how many times he had an opportunity to make a play. A sure-handed shortstop may have very few errors but let his team down by failing to reach ground balls that other shortstops turn into outs. You can't figure that out just by looking at traditional fielding statistics.

Some publications rank players based on range factors (putouts plus assists per game) or fielding runs (a metric developed by Pete Palmer for the Total Baseball encyclopedia). But these statistics are based on the traditional statistics and therefore suffer from the same limitations. Specifically, rankings based on range factors assume that each of those players had the same number of opportunities to make plays. But they don't for several reasons:

• some pitching staffs strike out more hitters and therefore allow fewer batted balls to enter the field

• the mix of ground balls and fly balls can vary quite substantially from one pitching staff to another

• the mix of left- and right-handed pitchers varies from team to team, meaning that those teams face different percentages of left-handed hitters, which in turn affects how many ground balls are hit to each side of the infield

• some pitchers are better than others, and weaker pitchers present fielders with a higher percentage of difficult chances

• some ballparks require that fielders position themselves differently, affecting how many batted balls can be reached with normal effort

For all of these reasons, it's much better to count the number of opportunities presented to each fielder and the number of times each fielder was able to turn those opportunities into outs. To that end, we have developed a series of analytical tools that use play-by-play data to assess fielding ability. Our methods, which are described in detail in an article ("Evaluating Defense") that you can find on our web site, take all of the above-mentioned factors into account.

We believe this analytical approach is much better than relying on the subjective opinions of baseball writers and broadcasters. And it is much better than relying on the Gold Glove voters to identify the best fielders. Much too often, these observers focus almost all of their attention on one aspect of defensive performance -- errors -- and neglect the ability to reach batted balls in the first place.

Wrapping up

We put a lot of effort into our player ratings each year. It would, of course, be much easier if we just carried forward a player's ratings from year to year or if we based our ratings on what we hear in the media. But we don't believe we'd be doing our job if we did that.

Everybody knows that batting and pitching performances go up and down over the course of a player's career. We believe the same is true of other skills such as defense and baserunning. Using sophisticated analytical techniques and the best information available helps us spot changes in performance when they happen, not later when the media picks up on them.

But judgment and common sense are essential, too. Many players do not get enough playing time in a single season to enable us to reach conclusions based on numbers alone. Sometimes the way a player is used can produce a biased view of his abilities. So we always look at the player's results in the context of his team, career (including minor league records for younger players) and an overall understanding of how baseball works.