2004 Predictions -- Keeping Score
By Tom Tippett
October 14, 2004
When we release our annual Projection Disk in the spring, we give our customers a chance to get a head start on the baseball season. With projected statistics and ratings for over 1600 established big leaguers and top minor-league prospects, plus league schedules, park factors, team rosters, projected pitching rotations, bullpen assignments, lineups and depth charts, the Projection Disk gives them everything they need to play out the new season using the Diamond Mind Baseball simulation game.
It also gives us a chance to get a head start on the season. Ever since we created the first Projection Disk in 1998, we've been publishing our projected standings along with comments on the outlook for all 30 teams. Those projected standings are based on the average of a number of full-season simulations using the Projection Disk.
Of course, nobody really knows what's going to happen when the real season starts, but we're always curious to see how our projected results compare to the real thing. And we're equally interested in seeing how our projections stack up against the predictions made by other leading baseball experts and publications. This article takes a look at those preseason predictions and identifies the folks who were closest to hitting the mark in 2004. And because anyone can get lucky and pick the winners in one season, we also look at how everyone has done over a period of years.
Comparing predictions
In addition to projecting the order of finish, our simulations provide us with projected win-loss records, projected runs for and against, and the probability that each team will make the postseason by winning its division or grabbing the wild card.
Unfortunately, most of the predictions that are published in major newspapers, magazines and web sites don't include projected win-loss records. Instead, they give the projected order of finish without indicating which races are expected to be hotly contested and which will be runaways. Some don't even bother to predict the order of finish, but settle instead for the division winners and wild card teams.
As a result, we do our best to assign a meaningful score to each prediction based solely on order of finish within each division. We borrowed the scoring system from our friend Pete Palmer, co-author of Total Baseball and The Hidden Game of Baseball, who has been projecting team standings for more than 35 years.
Pete's scoring system subtracts each team's actual placement from its projected placement, squares this difference, and adds them up for all the teams. For example, if you predict a team will finish fourth and they finish second, that's a difference of two places. Square the result, and you get four points. Do this for every team and you get a total score. The lower the score, the more accurate your predictions.
We don't try to break ties. If, for example, two teams tie for first, we say that each team finished in 1.5th place for the purposes of figuring out how many places a prediction was off. Suppose a team was projected to finish third and they tied for first instead. That's a difference of 1.5 places. The square of 1.5 is 2.25, so that would be the point total for this team. That's why you'll see some fractional scores in the tables below.
Keeping things in perspective
That first year, we created a little database with our projected standings and those of fourteen national publications, and we were pleased to see that we ended the year with the best accuracy score among those fifteen forecasts. When we wrote up the results and posted them to our web site, however, we were very careful not to make any grand claims, saying:
"I'm not sure what to make of all this. It's just one year, and it's entirely possible that we were just lucky. Time will tell whether our approach to projecting seasons is consistently better than average."
Over time, we expanded our database to include the predictions of prominent baseball writers from major newspapers and other publications. This is easier said than done because some publications and web sites change their approach from year to year. For example, we used to track the predictions of several ESPN.com writers and editors, but they limited their picks to division winners in 2003. So the number of entries in our database can rise and fall depending on what the various publications do and whether we were able to find those predictions in our spring survey.
In the sections below, we'll show you how various prognosticators ranked in 2004 and over a period of years, with the period varying in length depending on when we added that person or publication to our database. We don't make any claims of completeness here -- there are lots of other predictions that are not in our database -- but we think you'll find that our sample is an interesting one.
For several reasons, we want to emphasize that it's important that nobody take these rankings too seriously.
First, this isn't the only scoring system one could use to rank these projections, of course. A fellow named Gerry Hamilton runs a predictions contest every year (see http://www.tidepool.com/~ggh1/index.html) and assigns a score based on how many games each team finished out of their predicted place in the standings. (We came 22nd out of 195 predictions in their 2004 contest after finishing 4th in 2003.)
Second, because of publishing deadlines, the predictions in some spring baseball magazines are made long before spring training started, others are prepared in early-to-mid March, and some are compiled just before opening day. Obviously, the longer you wait, the more information you have on player movement and injuries.
Third, many newspaper editors ask staff writers to make predictions so their readers have something to chew on for a couple of days. Some writers hate doing them but comply because their editors insist. Some do it even though their main beat is a different sport. Others may make off-the-wall picks just for grins or feel compelled to favor the hometown teams.
Rankings for 2004
It's interesting to see how everyone did this year, but it's even more interesting to look back to see how different people perceived the baseball world before the season started. We'll start by showing you the prediction rankings for the current season, then we'll follow that up with a review of each division race and how those races affected these rankings.
Forecaster Score New York Times 30 Las Vegas over-under line 32.5 Tony DeMarco, MSNBC.com 40 Diamond Mind simulations 42 Bob Hohler, Boston Globe 42 Joe Sheehan, Baseball Prospectus 42 Michael Wolverton, Baseball Prospectus 42 David Lipman, ESPN.com 44 Michael Holley, Boston Globe 46 Gary Huckabay, Baseball Prospectus 46 Team payroll (per USA Today) 46 Poll of SABR members 48 Athlon 48 Eric Mack, CBS SportsLine 48 2003 final standings 48 MLB Yearbook 50 Baseball Prospectus 52 Nate Silver, Baseball Prospectus 52 Lindy's 52 Dan Shaughnessy, Boston Globe 52 ESPN.com power rankings 56 Phil Rogers, ESPN.com 56 Steve Mann 56 The Sporting News (Ken Rosenthal) 58 Rany Jazayerli, Baseball Prospectus 58 Charley McCarthey, CBS SportsLine 58 Baseball America 60 Sports Illustrated 60 Spring Training Yearbook 60 Tristan Cockroft, CBS SportsLine 60 USA Today 61.5 Street & Smith 62 Chris Kahrl, Baseball Prospectus 62 Miami Herald 64 Derek Zumsteg, Baseball Prospectus 64 USA Today Sports Weekly 66 Jonah Keri, Baseball Prospectus 66 Pete Palmer 68 Dallas Morning News 68 Seattle Times 68 CBS SportsLine 72 Gordon Edes, Boston Globe 72 Scott Miller, CBS SportsLine 72 ESPN the magazine (Peter Gammons) 74 Los Angeles Times 74 Bob Ryan, Boston Globe 76 Adam Reich, CBS SportsLine 80 Spring training results 134
The "Diamond Mind simulations" entry is the one representing the average result of simulating the season 100 times. These simulations were done about three weeks before the season started.
There are a few other entries in this list that don't represent the views of a writer or a publication. If you predicted that the 2004 standings would be the same as in 2003, your score would have been 48. If you put together a set of standings based on the Las Vegas over-under line, you'd have racked up an impressively low total of 32.5 points. If you thought the teams would finish in order from highest to lowest payroll, your score would have been 46.
And if you predicted that the regular season standings would match the 2004 spring training standings, your score would have been 134. In other words, the spring training results were almost useless as a predictor of the real season, and that's been true for at least the past four years.
Reviewing the divisions
Much more interesting than the overall scores, in our opinion, are the details. Which teams were consistently under- or over-estimated? Which divisions contained the biggest surprises? Did anyone predict that certain teams would have a sudden change of fortune?
Leaving out the entries that don't represent writers or publications, here are some observations about how the others saw things last spring:
AL East. Everyone had either New York or Boston winning the division, with the Yankees being picked first four more times than the Red Sox. Other than Gary Huckabay, who picked Toronto second and Boston third, everyone had this as a two-team race. A good number of people picked Baltimore third ahead of Toronto, but four people picked the Orioles to finish last, too, so there was no clear consensus on the Orioles.
AL Central. The Kansas City Royals were the downfall for many this year. The young Royals led the division for much of the 2003 season before fading down the stretch, then added some veteran players during the winter. As a result, they were a trendy pick to win the division or finish second behind Minnesota. A good part of the reason our score is among the leaders in 2004 is that we identified the Royals as one of the teams most likely to disappoint. That was based largely on our simulation results, but also based on the fact that the 2003 Royals didn't have the statistical foundation to justify their high placement. Surprisingly, seven predictions had Detroit finishing fourth, in every case because they thought the Indians would be even worse.
AL West. A year ago, our score was significantly improved because we chose to rank the Mariners ahead of the Angels when those two teams finished in a virtual tie for second in our simulations. This year, those teams were again neck and neck, with the Mariners averaging one more win but the Angels having a slightly better run margin. In a decision we'd love to have back, we gave the nod to Seattle. More than twice as many people chose Anaheim to win the division over Oakland, with three choosing the Mariners for first place. Everyone picked the Rangers to finish last, meaning that nobody in our survey got this division (or any other division) correct from top to bottom.
NL East. Before the season, the Phillies appeared to be loaded with talent, the Marlins were shedding payroll after winning the World Series, and the Braves seemed quite vulnerable. All three teams were selected by at least one person to win the division, with Philly being the choice about 80% of the time. Most predictions had a clear separation between the top three and the bottom two, but Montreal (five times) and New York (three times) snuck into third place on a few lists.
NL Central. Only two entries (Diamond Mind and Steve Mann) had the Cardinals finishing first in this division. The others seemed caught up in the hype surrounding the Cubs young pitching (Prior, Wood, Zambrano) and the Astros older pitching (Clemens, Pettitte). Just about every prediction had the Cubs and Astros duking it out for first with the Cardinals third. The picks for first place were almost evenly split between Chicago and Houston, with the Cubs having a very slight edge. There was some variation in the order of the bottom three teams, but nobody picked any of them to finish in the top half of the division.
NL West. Picking the Dodgers to finish at or near the top was a key to the better-scoring predictions this year, as was picking against the Diamondbacks. We were among those who thought Arizona would finish ahead of Los Angeles, but we were not alone. Approximately 2/3 of the predictions had Arizona beating the Dodgers, with thirteen people picking the D'backs to win the division outright. (In an example of the importance of timing, Arizona finished one game ahead of the Dodgers in our simulations, but the teams would have been reversed had we run them again after Milton Bradley was traded to LA.) It's clear that many people thought this division was wide open, as four of the five teams (everyone but the Rockies) were picked to finish first at least once.
Summing up. For the first time ever, not a single division was nailed by even a single predictor. Certain teams surprised a lot of people by overachieving (Texas, Los Angeles) or falling short (Arizona, Seattle, Kansas City, Toronto). As a result, the prediction scores were much higher this year than in 2003. A year ago, things went more in accordance with expectations.
Seven-year rankings
Here are the rankings for those who were included in our sample every year. There's a new entry this year. We went back and ranked all of the teams based on their payroll as reported in USA Today in April, and we computed a standings score based on the "prediction" that teams would finish in order from highest to lowest payroll. As you can see, that doesn't seem to be a very good predictor.
Forecaster 2004 2003 2002 2001 2000 1999 1998 Total Diamond Mind 42.0 28.0 40.0 54.5 68.0 42.0 44.5 319.0 Las Vegas over-under 32.5 30.0 46.0 65.5 51.5 48.0 52.0 325.5 Sports Illustrated 60.0 30.0 48.0 56.5 40.0 56.0 54.0 344.5 Steve Mann 56.0 48.0 60.0 38.5 58.0 54.0 44.0 358.5 Sports Weekly 66.0 38.0 42.0 46.5 58.0 51.5 60.0 362.0 Athlon 48.0 36.0 38.0 67.5 42.0 72.0 72.0 375.5 Sporting News 58.0 44.0 54.0 52.5 38.0 78.0 54.0 378.5 Pete Palmer 68.0 56.0 50.0 70.5 54.0 40.0 58.0 396.5 Street & Smith 62.0 36.0 70.0 68.5 58.0 68.0 64.0 426.5 Previous season 48.0 42.0 48.0 64.5 56.0 70.0 100.0 428.5 Payroll ranking 46.0 64.0 102.0 60.0 88.0 72.0 44.0 476.0
Six-year rankings
In 1999, we added some writers from the Boston Globe.
Forecaster 2004 2003 2002 2001 2000 1999 Total Gordon Edes, Boston Globe 52.0 32.0 54.0 56.5 26.0 28.0 248.5 Las Vegas over-under line 32.5 30.0 46.0 65.5 51.5 48.0 273.5 Diamond Mind simulations 42.0 28.0 40.0 54.5 68.0 42.0 274.5 Sports Illustrated 60.0 30.0 48.0 56.5 40.0 56.0 290.5 USA Today Sports Weekly 66.0 38.0 42.0 46.5 58.0 51.5 302.0 Athlon 48.0 36.0 38.0 67.5 42.0 72.0 303.5 Baseball America 60.0 28.0 48.0 54.5 54.0 70.0 314.5 Steve Mann 56.0 48.0 60.0 38.5 58.0 54.0 314.5 Sporting News 58.0 44.0 54.0 52.5 38.0 78.0 324.5 Previous season standings 48.0 42.0 48.0 64.5 56.0 70.0 328.5 Dan Shaughnessy, Globe 52.0 56.0 70.0 44.5 54.0 58.0 334.5 Pete Palmer 68.0 56.0 50.0 70.5 54.0 40.0 338.5 Bob Ryan, Boston Globe 76.0 40.0 58.0 84.5 58.0 40.0 356.5 Street & Smith 62.0 36.0 70.0 68.5 58.0 68.0 362.5 Payroll ranking 46.0 64.0 102.0 60.0 88.0 72.0 432.0
Five-year rankings
The Diamond Mind simulations missed the mark by quite a bit in 2000. We added a new concept to our projection system that year, but we were unhappy with the results, and we took that out of the model before generating our projections in 2001. The results have been much better since. As you can see, the Las Vegas over-under line has been getting much better in recent years.
Forecaster 2004 2003 2002 2001 2000 Total Las Vegas over-under line 32.5 30.0 46.0 65.5 51.5 225.5 Athlon 48.0 36.0 38.0 67.5 42.0 231.5 Diamond Mind simulations 42.0 28.0 40.0 54.5 68.0 232.5 Sports Illustrated 60.0 30.0 48.0 56.5 40.0 234.5 Gordon Edes, Boston Globe 72.0 32.0 54.0 56.5 26.0 240.5 Baseball America 60.0 28.0 48.0 54.5 54.0 244.5 Sporting News 58.0 44.0 54.0 52.5 38.0 246.5 Previous season standings 48.0 42.0 48.0 64.5 56.0 248.5 USA Today Sports Weekly 66.0 38.0 42.0 46.5 58.0 250.5 Steve Mann 56.0 48.0 60.0 38.5 58.0 260.5 Dan Shaughnessy, Globe 52.0 56.0 70.0 44.5 54.0 276.5 Street & Smith 62.0 36.0 70.0 68.5 58.0 294.5 Pete Palmer 68.0 56.0 50.0 70.5 54.0 298.5 Bob Ryan, Boston Globe 76.0 40.0 58.0 84.5 58.0 316.5 Payroll ranking 46.0 64.0 102.0 60.0 88.0 360.0
Four-year rankings
Lindy's was a strong addition to our survey in 2001. We also added the San Francisco Chronicle that year, but they've been dropped from this list because we couldn't find their 2004 predictions. That paper ranked second from 2001 to 2003.
Forecaster 2004 2003 2002 2001 Total Diamond Mind simulations 42.0 28.0 40.0 54.5 164.5 Lindy's 52.0 40.0 42.0 36.5 170.5 Las Vegas over-under line 32.5 30.0 46.0 65.5 174.0 Tony DeMarco, MSNBC.com 40.0 34.0 34.0 67.5 175.5 Athlon 48.0 36.0 38.0 67.5 189.5 Baseball America 60.0 28.0 48.0 54.5 190.5 USA Today Sports Weekly 66.0 38.0 42.0 46.5 192.5 Sports Illustrated 60.0 30.0 48.0 56.5 194.5 Steve Mann 56.0 48.0 60.0 38.5 202.5 Previous season standings 48.0 42.0 48.0 64.5 202.5 Sporting News 58.0 44.0 54.0 52.5 208.5 Los Angeles Times 74.0 18.0 44.0 73.5 209.5 Gordon Edes, Boston Globe 72.0 32.0 54.0 56.5 214.5 Dan Shaughnessy, Globe 52.0 56.0 70.0 44.5 222.5 Street & Smith 62.0 36.0 70.0 68.5 236.5 Pete Palmer 68.0 56.0 50.0 70.5 244.5 Bob Ryan, Boston Globe 76.0 40.0 58.0 84.5 258.5 Payroll ranking 46.0 64.0 102.0 60.0 272.0 Spring training results 134.0 70.0 86.0 113.5 403.5
Three-year rankings
Here's how things looked from 2002 to 2004. The LA Times was unable to follow up the excellent 2003 predictions that put them in top spot in last year's two-season rankings.
Forecaster 2004 2003 2002 Total Tony DeMarco, MSNBC.com 40.0 34.0 34.0 108.0 Las Vegas over-under line 32.5 30.0 46.0 108.5 Diamond Mind simulations 42.0 28.0 40.0 110.0 Bob Hohler, Boston Globe 42.0 32.0 38.0 112.0 Athlon 48.0 36.0 38.0 122.0 Lindy's 52.0 40.0 42.0 134.0 Los Angeles Times 74.0 18.0 44.0 136.0 Baseball America 60.0 28.0 48.0 136.0 Sports Illustrated 60.0 30.0 48.0 138.0 Previous season standings 48.0 42.0 48.0 138.0 USA Today Sports Weekly 66.0 38.0 42.0 146.0 USA Today 61.5 32.0 58.0 151.5 Sporting News 58.0 44.0 54.0 156.0 Gordon Edes, Boston Globe 72.0 32.0 54.0 158.0 Steve Mann 56.0 48.0 60.0 164.0 Street & Smith 62.0 36.0 70.0 168.0 Bob Ryan, Boston Globe 76.0 40.0 58.0 174.0 Pete Palmer 68.0 56.0 50.0 174.0 Dan Shaughnessy, Globe 52.0 56.0 70.0 178.0 Payroll ranking 46.0 64.0 102.0 212.0 Spring training results 134.0 70.0 86.0 290.0
Two-year rankings
Finally, here's how things have looked over the past two years.
Forecaster 2004 2003 Total Las Vegas over-under line 32.5 30.0 62.5 Diamond Mind simulations 42.0 28.0 70.0 Tony DeMarco, MSNBC.com 40.0 34.0 74.0 Bob Hohler, Boston Globe 42.0 32.0 74.0 Athlon 48.0 36.0 84.0 Baseball America 60.0 28.0 88.0 Sports Illustrated 60.0 30.0 90.0 Previous season standings 48.0 42.0 90.0 MLB Yearbook 50.0 40.0 90.0 Lindy's 52.0 40.0 92.0 Los Angeles Times 74.0 18.0 92.0
USA Today 61.5 32.0 93.5 Street & Smith 62.0 36.0 98.0 Sporting News 58.0 44.0 102.0 USA Today Sports Weekly 66.0 38.0 104.0 Gordon Edes, Boston Globe 72.0 32.0 104.0 Steve Mann 56.0 48.0 104.0 Dan Shaughnessy, Globe 52.0 56.0 108.0 Spring Training Yearbook 60.0 48.0 108.0 ESPN the magazine 74.0 36.0 110.0 Payroll ranking 46.0 64.0 110.0 Pete Palmer 68.0 56.0 114.0 Bob Ryan, Boston Globe 76.0 40.0 116.0 Spring training results 134.0 70.0 204.0
Summing up
Overall, we've been pretty happy with our results, and if there's one thing that stands out, it's our ability to identify over-rated teams.
In 2004, we saw the Royals as a 2003 overachiever that was unlikely to repeat, we projected the Blue Jays to finish below .500, and we didn't buy all of the hype surrounding the Cubs and Astros.
A year earlier, our simulations correctly indicated that the Mets were likely to finish at the bottom of their division again, the Angels were very unlikely to repeat their 2002 success, and the Dodgers wouldn't score enough runs to make a serious run at the NL West title.
Even so, we're always surprised by something that happens each year. We didn't anticipate the emergence of the Rangers and Dodgers in 2004 or the surprising finishes of the Marlins and Royals the year before. As a result, we have a bunch of test cases to study as we consider possible improvements to our projection system.
More than anything, this process -- projecting the season in March, watching the real thing for six months, and taking a look back after the season -- is highly educational for us. So we'll be back with our projected 2005 team standings in March.
- Tags: Projected Standings
