A few years ago I was at one of my first glitzy media parties when a prominent media figure began excitedly chatting with me. I was flattered, until I realized that he had confused me with a famous black actress.

An actress I don’t resemble all that much. When he realized his error, he was incredibly embarrassed and apologetic. I watched him turn several shades of red. I accepted his apology and the incident became a running joke with friends since there are worse things than being confused with someone who is clearly far more beautiful than I will ever be. But I’ve been reflecting on that incident recently for a couple of reasons.

For starters, it dawned on me that it occurred in the days just before social media began turning private missteps into public scandals. There were people who heard what he said. My guess is had it happened in 2015 he would have been publicly humiliated. Second, I realize that he has demonstrated a measure of generosity to me professionally since that incident, that to be frank, he may not have had he not felt some guilt—guilt and probably gratitude that I accepted the moment for what it was: an embarrassing error in judgment from a well-meaning person and not some intentional offense by a racist. The fact that I made an effort to tell the difference in that moment, and gave him the benefit of the doubt, meant that he ultimately decided to make an effort to help me. Now thanks in part to him, I have had opportunities to write about all sorts of issues and people, including those of importance to the African-American community.

Which is why I find the fallout from the protests at Yale University, Dartmouth, and other places so troubling, and damaging to the African-American community. Increasingly the message from certain activists seems to be, if you are imperfect in how you talk about race, you are not allowed to be part of the conversation. This message will not only have a chilling effect on college campuses, but I believe is already having a devastating impact on the efforts of minorities to advance in fields in which we are underrepresented—and in which our presence is needed now more than ever.

Here’s what I mean.

I recently had a conversation with a high-ranking ethnic minority in the television world. (I will not identify the person’s gender or race because this conversation was not for attribution.) I was asking this person for career advice because I had recently worked as a staff writer on a television series for the first time. Though I had a wonderful experience, I had been surprised at how lacking in diversity much of Hollywood really is (for all of its liberalism) particularly when it comes to the writing staffs of shows. (Part of why Shonda Rhimes, the celebrated writer for Grey’s Anatomy and Scandal, has become so iconic is not just that she is immensely talented but because in terms of the television world, she is the equivalent of electing the first black president. There are so few black women writers working on writing staffs today that to see the level of success she has achieved is awe-inspiring.)

This person I was speaking with said that when it comes to discussing race in the writers room, even when you need to, such as when you perceive a particular piece of dialogue as offensive, it’s important to follow your conscience but to raise the issue very carefully because no one wants to work with “that guy.” The “that guy” he was referring to is the minority writer who seems perpetually aggrieved, perpetually offended, and perpetually angry. (And may end up perpetually unemployed.)

To be clear, the message was not “Don’t speak up.” It was choose your battles—wisely—and be careful whom you write off as an enemy because you never know who you may need as an ally one day.

Though it was a tough piece of advice to hear (after all, being a political columnist, sharing my heartfelt opinion is part of my DNA), it also made me reflect on something I have discussed privately but never publicly, which is this: I know there are rooms of opportunity and power—including some writers rooms—that people like me are increasingly excluded from simply because the people in them are petrified of saying the wrong thing in front of me or another person of color or another woman and ending up vilified in social media or on a blog or in a protest. So instead they surround themselves with fellow white guys. I know this in part because in the last two weeks I have had at least two white people I know say that family members or friends have made comments to that effect, i.e. “I really like my black co-worker but I’m nervous about dinner because what if things turn to politics and I say the wrong thing.”

In an age in which unarmed black men are getting shot by police this may sound like small potatoes. But consider this: A groundbreaking study published in the American Psychologist last year found that blatant racism is no longer the biggest obstacle facing minorities seeking workplace advancement today. Instead it has been replaced by what you might call accidental discrimination. According to the study, “in present-day America, discrimination results more from helping ingroup members than from harming outgroup members.”

The study gives eye-opening examples that can best be summarized as this: People hire other people like them, but more specifically people hire and promote other people in their social networks. That means while getting good grades is important, today it may not be as important as going to the right school, the right church, joining the right social clubs, or playing the right leisure sports and meeting the right people and leveraging those relationships into professional opportunity.

This theory was further reinforced by research published by Professor Lauren Rivera of the Kellogg School of Management, who found that in elite, competitive professions, people hire people who they believe fit in their social networks. Her research found this was not limited by race, but by what one of her subjects calls the “airport test.” The investment banker said, “Would I want to be stuck in an airport in Minneapolis in a snowstorm with them?”

In other words, people share power with people they are comfortable with. Which brings me back to the increasingly toxic environment on our nation’s campuses.

There is no question that there are still racists, and specific acts of racism, that warrant outrage and protests. I think the students at the University of Missouri are to be commended for challenging a culture in which racial slurs had become viewed as permissible. But if the message becomes that we are so sensitive that we are unable to have rational conversations with those who may be well-intentioned but wrong (see Yale’s Halloween kerfuffle or the Dartmouth library siege), the conclusion will be that disenfranchised people of color are no longer worth including in any conversation that matters.

Like it or not, white males, a number of whom may have a learning curve on issues like race or gender, run the world, or at least most of it. If we want to see power distributed more evenly, we have to be able to get into the halls of power. That will be tough to do if more and more white Americans in power become fearful of having conversations with us in the first place.