Barack Obama promised to transform health care and energy and America’s role in the world. But what made tens of millions of Americans loopy with joy is the implicit promise that the young biracial senator would, through the power of his biography and charisma, heal racial divides. Throughout the presidential campaign and the brief but blissful post-inaugural delirium, countless trend pieces vividly described how blacks and whites were—gasp!—chatting amiably. They were giving each other Obama-inspired fist-bumps. Though there was something silly and sad about this enthusiasm—shouldn’t we have been fist-bumping across the color barrier long before?—it was hard not to share in the excitement. Now, however, one senses that the excitement is fading.

Click Image Below to View Our Gallery

In a recent survey conducted by CNN and Essence, 55 percent of African Americans identified racial discrimination as a serious problem. What’s interesting is that when asked the same question during the 2008 campaign, only 38 percent said racial discrimination was a serious problem. One obvious interpretation is that Obama’s election has rightly raised expectations, which the real world—even an Obamaized world—couldn’t match.

Moreover, African Americans are being hit disproportionately hard by the downturn. By now you’ve no doubt heard about the yawning unemployment gap between men and women, but the gap between whites and blacks is far larger: 8.7 percent vs. 14.7 percent. And that number, of course, doesn’t include workers who’ve been forced to work part-time rather than full-time, or those who’ve simply grown too discouraged to keep actively looking for work. Were you to take a broader view of unemployment, which factored in the staggering number of Americans behind bars, for example, the gap would look bigger and more depressing. It’s worth noting that the black-white unemployment gap started to seriously widen in the early 1980s. A popular theory is that the expanding gap was driven in large part by the decline of blue-collar manufacturing jobs that offered a decent living to non-college-grads. If this recession kills what’s left of Detroit, the gap will likely get worse.

Some of the president’s most trenchant critics work for a small Jersey City-based website called Black Agenda Report, the brainchild of Glen Ford and Bruce Dixon, two passionately left-wing black journalists. But though the editors of Black Agenda Report undoubtedly prefer the president to, say, Eric Cantor, one suspects that they are among the 4 percent who disapprove of his job performance. At the 100 Days mark, Bruce Dixon issued a fascinating report card that tracked Obama’s progress across 18 issue areas he deemed most relevant to African Americans, ranging from health care reform to militarizing the border to policy toward black farmers to (in his words) investigating Bush-era crimes. Out of 100 possible points, Dixon gave the president a rather disappointing 25. Which is to say, Dixon gave Obama—by all accounts a brilliant student—a failing grade.

My guess is that David Axelrod isn’t losing any sleep over Black Agenda Report. Ford and Dixon are sharp writers, but they’re also far more ideological and far less sentimental than the average voter. The president has a deep connection with black voters, one that will incline many of them to interpret his actions in the best possible light. Indeed, among African Americans, his approval rating is 96 percent in the latest Gallup poll. Meanwhile, a cleavage is emerging in his popularity: Among non-Hispanic whites, his approval has fallen to 52 percent. It seems that, while white voters are more inclined to blame Obama for their economic woes, black voters are taking a different and perhaps more patient view of the president.

Rather than blame the president, who, lest we forget, has only been in office for a few months and who can only do so much to undo the damage of the last several decades, many African Americans are looking to deeper, structural causes for their discontent. Whites feel differently.

So what happens if the unemployment rate continues to climb and we see an anti-Obama backlash? Even during the campaign, Obama has played a delicate game of raising expectations—Yes We Can—and lowering them at the same time. And white anxiety about electing a black president may have been masked by the total collapse of the Republican brand. At some point, his political luck will run out. And it’s easy to imagine that African Americans will remain his staunch defenders. Rather than blame the president, who, lest we forget, has only been in office for a few months and who can only do so much to undo the damage of the last several decades, many African Americans are looking to deeper, structural causes for their discontent. Whites feel differently.

A key moment came this past weekend when Colin Powell offered sharp criticisms of Obama’s expansive agenda after endorsing him last fall. It’s very easy to imagine white voters in the suburbs of Dallas or Kansas City listening to Powell and nodding along. But as for black voters who are used to getting less than a fair deal, Powell’s criticisms may well have come across as churlish and short-sighted. What will our politics look like if this perception gap widens? Will tough criticism of President Obama be seen as racist or otherwise out-of-bounds? Will whites start feeling as though White House efforts to aid the disadvantaged are nothing but handouts to undeserving racial outsiders? Harvard economists Edward Glaeser and Alberto Alesina have found that racially diverse societies are less likely to have generous welfare states than homogeneous societies. It thus makes a lot of sense that the White House is spending so much time and energy courting middle-of-the-road white voters, to the frustration of the Democratic left. The danger is that political polarization will be compounded by racial polarization. And a cultural climate that seemed so promising just a few months ago could take a very depressing turn.

When it comes to the state of race in America, Barack Obama might be too big to fail.

Reihan Salam is a fellow at the New America Foundation and the co-author of Grand New Party.