During the Republican primaries, Texas Republican Rep. Ron Paul came under fire for his association with a series of noxious newsletters, many of which featured racist overtones. Paul's defense? "Libertarians are incapable of being a racist."
(I suggest you fast forward to around the :45 mark if you're short on time)
Taking Paul's utopian beliefes to their natural ends, if everyone was a "real" libertarian, the problem of racism would not exist. (This logic lead Paul's son, Republican Sen. Rand Paul, to oppose, on ideological grounds, government legislation limiting discrimation by private businesses.)
But in The Economist, Will Wilkinson, writing on a separate discussion, details why this is not how the world actually works.
When I was a Rand-toting libertarian lad, I believed, as I believe now, that racism of any stripe is a disgusting form of collectivism. Where my opinion has changed is that I used to think that if negative rights to non-interference were strictly observed, liberty was guaranteed, but I don't now. Here's how I had thought about the matter. One racist acting in a private capacity on his or her racist beliefs can't violate anyone's legitimate, negative rights. (No one is entitled to another's good opinion!)
Eventually I realised that actions that are individually non-coercive can add up to stable patterns of behaviour that are systematically or structurally coercive, depriving some individuals of their rightful liberty. In fact, rights-violating structures or patterns of behaviour are excellent examples of Hayekian spontaneous orders—of phenomena that are the product of human action, but not of human design.
This shift has led me to see racism and sexism themselves as threats to liberty. Racism and sexism have come to matter more to me in that I have come to see them in terms of the political value that matters most to me: liberty. And so I have become much more sympathetic to policies that would limit individual liberty in order to suppress patterns or norms of behaviour that might pose an even greater threat to freedom. So I've become fairly friendly toward federal anti-discrimination law, affirmative action, Title 9, the works.