Sonia's Batting Average

The confirmation silliness has reached new heights—critics now are springing the “fact” that Judge Sonia Sotomayor, nominated by President Obama to succeed the retiring Justice Souter, has been reversed 60 percent of the time by the Supreme Court. There is no such fact, of course, and no basis for any speculation.

Here’s why.

Over her career as an appellate judge, Sotomayor has authored 380 majority opinions. The Supreme Court has reviewed five, and reversed three—thus the “calculation” of 60 percent.

But this argument ignores the fundamentals of statistics. In the first place, 375 of her opinions were not even deemed worthy of review: That is, either the loser chose not to appeal, or the Supreme Court decided not to hear the case. Thus, by the method of her critics, we should conclude that Sotomayor has been reversed in only three of 375 cases—that is, eight-tenths of 1 percent of the time.

By the method of Sotomayor’s critics, Samuel Alito’s time on the Court of Appeals yields a reversal rate of 100 percent.

Overall, the Supreme Court in recent years has reversed roughly 75 percent of the cases it hears. Thus, even if one accepts the 60 percent figure for Sotomayor, that would put her below the median reversal rate for all federal appellate judges. But, of course, one would expect the reversal rate to be high for most, if not all, judges. After all, if the court thinks the lower court judge is right, it rarely will agree to hear the appeal.

Moreover, the sample suffers from what statisticians call selection bias. Such a bias occurs when the sample you are studying does not reflect the larger population about which you are drawing conclusions. Suppose you wanted to determine what percentage of Americans have had flu shots, and you did your study by dispatching pollsters to hospital emergency rooms across the country and asking each patient if he or she has had a shot. This would not give you any information about how many Americans have had flu shots, because most people do not use emergency rooms very often, and the people who use them are more likely to be very sick or uninsured or both. At best, you would know what percentage of the very sick or uninsured have had flu shots.

The same is true when one tries to figure out whether a judge does or does not follow precedent by studying reversal rates. The Supreme Court has absolute discretion over its docket and takes only the hard cases—those likely to provoke the greatest disagreements. Judge Richard Posner of the United States Court of Appeals for the Seventh Circuit once put the point this way: “[N]ot only does the Supreme Court review only a minute percentage of court of appeals decisions, but the cases it does review are not a representative sample of judicial activity in the different fields of federal law.”

A judge might follow the law perfectly in hundreds of decisions that the Supreme Court never hears, but, because panels that hear appellate arguments are drawn at random, she also might be dealt a handful of particularly difficult or unusual cases in which precedent is not a reliable guide. These are the cases the Supreme Court is most likely to review. And we have no way of drawing conclusions about the sample without knowing the complexity of the cases.

To take one example, Judge Sotomayor’s decision in Riverkeeper v. Environmental Protection Agency, concerning cost-benefit analysis under the Clean Water Act, was indeed reversed by the Supreme Court (under a different case name), but the complexity of the issue is perhaps illustrated by the fact that four of the justices thought she was right.

The biggest problem with the claim that Sotomayor has a high reversal rate is what statisticians call the law of large numbers. This law states that the sample mean converges to the distribution mean as the size of the sample increases. Stripped of the jargon, when you have only a tiny sample, you cannot assume with any confidence that your sample represents the whole.

If you have 400 socks in a drawer, and you want to know how many are black and how many are blue, you might draw a few out to look. Suppose of your first five that three are black and two are blue. Would this really imply that 60 percent of the socks are black? Of course not. You would want to pick a few more before you reached your decision. The more you drew out, the more confident you would be of your prediction. That is how the law of large numbers works.

Five opinions out of almost 400 is simply too small a number for us to draw reliable conclusions. Indeed, if we could render a verdict based on so small a sample, we would have to conclude that Justice Samuel Alito holds no respect for precedent. After all, when Alito served on the Court of Appeals, the only two majority opinions he wrote that the Supreme Court reviewed were reversed. By the method of Sotomayor’s critics, this yields a reversal rate of 100 percent. This would have been an absurd argument to make against Alito. It is equally absurd when applied to Sotomayor.

The American people do not need another summer full of silly applause lines about a Supreme Court nominee. What we need, and deserve, is a thoughtful and dignified debate over issues of high principle. Let’s hope this time we can have one.

Stephen L. Carter is the William Nelson Cromwell Professor of Law at Yale. His seven nonfiction books include God’s Name in Vain: The Wrongs and Rights of Religion in Politics and Civility: Manners, Morals, and the Etiquette of Democracy. His first novel, The Emperor of Ocean Park (2002), spent 11 weeks on the New York Times bestseller list. His fourth novel, Jericho’s Fall, will be published in July.