In his long interview with David Remnick in the latest issue of The New Yorker, President Obama gave a few thoughts on the dynamics behind his job approval rating. For anyone who studies public opinion, and the intersection of politics and race, they were banal:

“There’s no doubt that there’s some folks who just really dislike me because they don’t like the idea of a black President,” Obama said. “Now, the flip side of it is there are some black folks and maybe some white folks who really like me and give me the benefit of the doubt precisely because I’m a black President.”

Naturally, this led to an explosion of sputtering outrage from the right-wing, which was shocked that Obama would even mention race, much less in the context of his approval rating. Fox’s Neil Cavuto, oblivious to the race baiting of his network, gave a typical reaction, complete with a context-free nod to Martin Luther King Jr.: “It just demeans this president when he offers his race as an excuse for not doing better in the same position. Because I think Martin Luther King said, ‘I have a dream.’ Not once do I recall his ever saying, ‘I have an out.’”

There’s no firm evidence to gauge the exact affect of racial animus (or racial solidarity) on Obama’s job approval numbers, but a few things are true. The first is that, to a large degree, race is already baked in to the two-party system. Mid-century social and economic shifts—and the Civil Rights Movement in particular—sparked an exodus of whites to the Republican Party, and a corresponding shift of African Americans to the Democratic Party.

In the decades since the 1960s, there have been eight Democratic presidential candidates who received less white support than Obama did in the 2008 election, and four who received less than he did four years later. Overall, from 1972 to 2004—the pre-Obama era—white support for Democratic candidates ranged from a low of 32 percent (George McGovern) to a high of 43 percent (Bill Clinton) for an average of 39.4 percent, or just above Obama’s total for 2012.

This isn’t a coincidence. What political scientists call “racial resentment”—the intersection of anti-black sentiments and traditional American views on hard work and individualism—is one of the most reliable predicators of partisan affiliation. And according to a 2010 paper by political scientists Michael Tesler and David Sears, voters high on the racial resentment scale (reproduced here) became more partisan in their attachment to the Republican Party. It’s not a stretch to imagine these are some of the people Obama is referring to. And on the flip-side, as noted in the interview, Obama has benefited from blacks’ racial solidarity and their traditional support for Democrats.

Indeed, according to another paper from researchers at the University of Michigan, Stanford, and the University of Chicago, there’s been a marked increase in the number of voters with explicit anti-black attitudes in the last five years, which rose from 47.6 percent in 2008 to 50.9 percent in 2012. What’s more, anti-black attitudes are heavily distributed on the right side of the political divide, though they exist among Democrats and independents as well.

It’s not new for conservatives to cry that Obama is “playing the race card.” But in this instance, it’s silly. Not only is Obama not attributing his entire standing to race, but he’s not saying anything that hasn’t been confirmed by reams of research, to say nothing of common sense. For most of their time in the United States, blacks were a hated minority, and just fifty years ago, they lived under economic disenfranchisement and violent oppression. Given that background, and its deep roots in all of American life, it’s ludicrous to think it doesn’t play a part in today’s politics.

Conservatives can hold their breath and wish for colorblindness, but that doesn’t make it true, and pretending otherwise just makes it harder to explain the world we live in.