Florida Democrat Alan Grayson is known for throwing bombs at his Republican colleagues, but he may have gone too far with his latest.

In a fundraising email to supporters, Grayson’s campaign compared the Tea Party movement to the Ku Klux Klan—while depicting a burning cross. Republicans are enraged (“There’s no excuse for the hateful words and imagery used by Congressman Grayson,” said Matt Gorman, a spokesman for the National Republican Congressional Committee) but Grayson is unapologetic, telling Fox News that “[T]here is overwhelming evidence that the Tea Party is the home of bigotry and discrimination in America today, just as the KKK was for an earlier generation.”

There’s no question that Grayson is in the wrong: The Tea Party is objectionable, but it’s a far cry from the white supremacist vigilantism of the Klan and its offshoots. At the same time, it would be needless political correctness to dismiss the Tea Party as completely unrelated to the Klan, or at least, the reactionary currents that gave it life.

For starters, it’s no accident that the Tea Party emerged during a period of mass immigration and rapid cultural change. Like the Know-Nothing Party of the 1850s—which directed its energies against Catholic immigrants from Ireland—or the “modern” Klan of the 1920s—which, in addition to blacks, targeted Italians, Jews, and Eastern Europeans—the Tea Party has its roots in demographic anxiety; the profound fear that the country is turning into something foreign and un-American. Earlier this month, pollster Stan Greenberg released results from several focus groups he held with Tea Party and other Republican voters, in which they expressed their fears and concerns. His conclusion?

“They have an acute sense that they are white in a country that is becoming increasingly “minority,” and their party is getting whooped by a Democratic Party that uses big government programs that benefit mostly minorities, create dependency and a new electoral majority.”

On the same token, a new study (PDF) from the liberal Center for American Progress on public attitudes about rising diversity finds that white conservatives are alone in their discomfort with a rapidly changing polity. When asked to give their views on a series of statements, 61 percent of white conservatives (and 56 percent of whites older than 65) agreed that “discrimination against whites will increase” as the United States became more diverse. Likewise, white conservatives were mostly likely to believe that diversity would bring an end to a “common American culture” and increase the likelihood of “racial strife.”

These twin fears—of being isolated and of being overrun—are common in American history, and have inspired a wide range of ideologies, tactics and reactionary crusades. To wit, the Tea Party’s radical, anti-majoritarian approach to congressional negotiating—which saw its pinnacle in the most recent government shutdown—has its roots in “nullification,” the antebellum idea that a political minority could cancel a law it opposed. Moreover, as Zack Beauchamp correctly argues in long-form for ThinkProgress, the Southernness of the Tea Party owes itself to the reactions of the past: “Today’s Republican radicalism, with all of its attendent terrifying brinksmanship, is the grandchild of the white South’s devastating defeats in the struggle over racial exclusion.”

Again, I don’t agree with Alan Grayson’s claim that the Tea Party is the Ku Klux Klan of the 21st century—it’s reductive, inaccurate, and offensive. But reactionary movements—like their ideological counterparts on the Left—aren’t isolated phenomena; they grow out of longstanding conflicts and anxieties in American life.

Or, to put this in concrete terms, it’s not an accident that the most extreme members of the right wing—like the challenger to Mississippi Senator Thad Cochran, or the former aide to Kentucky Senator Rand Paul—have ties to secessionist groups or neo-Confederates; their ideological grievances are a recurring force in our country’s history.