Why I'm Not a Political Forecaster

This weblog passed its one-year anniversary last week, and by way of celebration I thought I would post one of the blog entries I wrote last year but decided, for one reason or another, not to publish. The essay, which I completed in early October, was entitled "Electoral Predictions," and used a simple but persuasive (or so I thought) statistical model to determine the likely outcome of the 2006 U.S. Congressional elections. How accurate was my prediction? Take a look:

It's never wise to make predictions in American politics, and historians are traditionally the worst political prognosticators. Nonetheless, with 5 1/2 weeks to go before the fall Congressional elections, let me briefly observe that history does not appear to be on the Democrats' side.

In the 1930s, Congressional elections sometimes led to quite dramatic swings in the partisan composition of the House of Representatives. There were several elections in which one party picked up 80 or more seats, followed by elections in which the other party recaptured 60 or more seats. These oscillations, however, have become much smaller in recent decades, as redistricting has made more House districts "safe" for either Democrats or Republicans, and as the realignments caused by the New Deal and the Civil Rights movement have reached completion. There have been only a few dramatic midterm elections in the last forty years, and only one in the last twenty: the 1994 "revolution," in which a wave of Democratic retirements in Republican-trending districts helped give the Republican Party control of the House.

The chart above demonstrates just how anomalous the 1994 election was. The X-axis identifies each of the last ten House elections, and the Y axis indicates the net change in House seats from one party to the other. Since 1986, this figure has usually been less than 10 seats, sometimes less than 5. The average change (factoring in 1994) has been 10-11 seats. If this twenty-year trend continues in 2006, the minority party -- the Democrats -- will probably pick up some seats in the House of Representatives. The historical odds, however, indicate that they'll only win 10-11 seats, leaving the Republicans with a 219-216 (or so) majority. It's a thin margin, but enough to keep Dennis Hastert ensconced in the Speaker's office.

The Senate race is more interesting because it's more personal -- and thus harder to predict. I will guess that Democratic challengers will win Republican seats in Pennsylvania, Montana, and Ohio, but Republican Thomas Kean is polling ahead of his Democratic challenger in New Jersey, and name recognition may be enough to get him the seat. The McCaskill-Talent race in Missouri is rated a tossup by Slate Magazine. George Allen is unlikely to lose in Virginia, "macaca" reference or no, because his opponent is weak and inexperienced. Lincoln Chafee has run far enough away from the president to keep his seat in Rhode Island. Tennessee will not vote for a black Democrat, no matter how conservative he is or how much name recognition he enjoys.

So the Democrats will pick up 2-3 seats in the Senate, giving the Republicans a 52-46 majority -- still larger than their majority in the 109th Congress. The other two seats will almost certainly be held by Bernie Sanders, a Vermont Socialist who will caucus with the Democrats, and Joe Lieberman, the newly-independent Connecticut Senator who will probably caucus with the Democrats.

Well, I was right about a few of the Senate races. Otherwise, my predictions were wrong. The Democrats picked up 31 seats in the House of Representatives, and Kean, Talent, and Chafee all lost their races; moreover, George Allen's "macaca" gaffe ("macacalypse," if you prefer) helped the "weak and inexperienced" Jim Webb defeat him, thus giving the Democrats control of the U.S. Senate. Perhaps historians really do make bad "political prognosticators."

There's another way to look at my lousy prediction, though. A Republican economist friend of mine, whom I now owe a pizza (we made a bet about the outcome of the House race), says that I should have used the 1966 House race as my model for the 2006 race. In that year, voters punished an increasingly unpopular president who had mired the nation in an foreign war by "shellacking" the president's party in the midterm elections. (Republicans picked up 47 House seats in 1966.) What's interesting is that my economist friend used a historical model to predict the outcome of the election, and was proven right, whereas I, a historian, used a statistical model to predict the election and was dead wrong. Sometimes it pays to stick with what you know.