“There’s about to be a backlash against the Ivy League lock on the court,” explained David Brooks this week. In fact, it has already begun. From left to right, just about the only thing that liberals and conservatives have agreed upon since Elena Kagan’s nomination is that there are too many pointy-heads on the Supreme Court. “I think it would be good to have a nominee that stood up against powerful interests like the elite law schools, which are a powerful interest in the U.S. and have done a lot of damage,” explained William Kristol recently. Several days later, in what may be a sign that the apocalypse is nigh, the Huffington Post’s Sam Stein quoted Kristol with approval.

I’d like to propose a backlash against the backlash. (And yes, shoot me, I attended an Ivy League college myself.) When critics bemoan the fact that if Kagan gets confirmed, every Supreme Court justice will be tainted by the Ivy League, what they generally mean is that the Supremes won’t have anything in common with average Americans. But in one sense, they’re not supposed to. The Supreme Court is not supposed to comprise a cross-section of the American public, or of the American legal profession. It is supposed to constitute an intellectual elite. Brains aren’t the only thing you want in a Supreme Court justice. But they’re up there. There’s nothing wrong with wanting the judge reading the brief—like the radiologist reading the X-ray—to have graduated at the top of her class, not in the middle.

In fact, very few of the current Supremes were raised with a silver spoon.

Fine, you say, we want intellectual elites on the court. But the Ivy League doesn’t only—or even mostly—constitute an intellectual elite. It also constitutes a cultural and economic elite, filled with mediocrities who gained admission because of their privileged upbringing and family connections. Absolutely right. Historically, a lot of the students at places like Harvard and Yale have fallen into exactly that category. And some still do. We just got finished having one of them as president.

• Read Our Full Coverage of Elena Kagan• Benjy Sarlin: Andrew Sullivan’s Kagan Crusade • Samuel P. Jacobs: What Really Happened at Harvard But that’s a problem with the Ivy League. It’s not a problem with the Ivy Leaguers on the Supreme Court. In fact, very few of the current Supremes were raised with a silver spoon. Kagan went to public school in Manhattan; her mother was an elementary school teacher and her father was a lawyer who represented tenants. Antonin Scalia went to a Jesuit high school; his mother was an elementary school teacher in Queens and his father was a Sicilian immigrant who eventually became a professor of Romance languages at Brooklyn College. Samuel Alito’s mother was also a teacher and his father was an Italian immigrant who began his career as a public school teacher before becoming a state bureaucrat; Alito attended public school in New Jersey. Clarence Thomas grew up in rural Georgia. His father was a farm worker and his mother was a maid; he attended public and Catholic schools. John Roberts’ father was a plant manager at Bethlehem Steel; he attended Catholic schools. Sonia Sotomayor’s father was tool and die worker with a third-grade education and her mother was a nurse; she went to Catholic schools in the Bronx. Ruth Bader Ginsburg went to public school in Brooklyn. Stephen Breyer’s father was a lawyer for the San Francisco Board of Education; he went to Lowell, a famously demanding Bay Area public high school.

Of the current members of the Supreme Court, only two had unusually privileged backgrounds. One is Anthony Kennedy, whose father was a prominent Sacramento lawyer who palled around with Earl Warren. The other is John Paul Stevens, whose paternal family were insurance and real-estate tycoons. Yet oddly, it is Stevens’ retirement—and his likely replacement by Kagan—that has sparked the current media wailing about elitism on the court. Stevens, who as a young man was given a caged dove by family friend Charles Lindbergh, presumably represents the lost era of Supreme Court populism because he attended Northwestern Law.

You don’t have to be Nathan Glazer to detect some patterns in the lives of the current Supremes. First, their parents valued education. Kagan, Scalia, and Alito are the children of teachers. Sotomayor’s mother raised her children in a Bronx housing project but bought them copies of the Encyclopedia Britannica. Ruth Bader Ginsburg’s mother took her kids on frequent trips to the library. Second, the Supremes attended excellent public and parochial schools. Kagan went to Hunter College high school, where tuition is free, and admission requires besting the smartest kids in New York in a sixth-grade exam. Breyer went to a West Coast equivalent, Lowell. Ginsburg went to James Madison, a public school in Brooklyn that also graduated Nobel Prize-winning economists Robert Solow and Gary Becker. Scalia, Thomas, Roberts, and Sotomayor went to highly competitive, academically rigorous Catholic schools. For the most part, the current Supremes grew up in circumstances that most Americans can relate to. They and their parents simply worked harder to transcend them.

Had they come of age a few decades earlier, when kids from Andover and Groton ruled Princeton and Harvard Law—when those institutions weren’t as friendly to Jews, blacks, Puerto Ricans, and the kids of Italian immigrants—today’s Supreme Court justices would likely have attended college and law school at places like City College, Rutgers, Berkeley, and Fordham instead. And as a result, they might never have made it to the Supreme Court. It is undoubtedly true that the connections that people like Kagan, Sotomayor and Roberts made in the Ivy League greased the wheels for their future success. But they gained those connections through talent, ambition and hard work, not because their parents had money or status. The media has been asking the wrong question. What matters isn’t how many current Supreme Court justices attended Ivy League schools. What matters is how many parents of current Supreme Court justices attended Ivy League schools. The answer, as far as I can tell, is none.

Under cover of anti-elitism, people like Sarah Palin love to conflate unearned privilege with hard-won educational achievement. The former is pernicious; the latter is glorious. And if the latter is the story of Elena Kagan, and our current Supreme Court, it should be a matter of national pride.

Peter Beinart, senior political writer for The Daily Beast, is associate professor of journalism and political science at City University of New York and a senior fellow at the New America Foundation. His new book, The Icarus Syndrome: A History of American Hubris, will be published by HarperCollins in June. Follow him on Twitter and Facebook.