Will Wednesday’s White House “Summit on Countering Violent Extremism” focus just on Muslims? Or will the summit look at combatting extremist violence of all types including those related to white supremacists, “Patriot groups,” and even violence that is Christian inspired? If we truly want to save American lives, it needs to be the latter.

I will be attending the summit. In advance of this event, I spoke to several Muslim American leaders about their views of and concerns with it. The responses ranged from some having strong fears that it could further the false but often heard narrative that terrorism is committed only by Muslims. Others, however, unreservedly welcomed this summit as an opportunity to partner with the government to counter radicals.

There is agreement among all the Muslims leaders I reached out to that countering the radicalization of Muslims in the United States and abroad, together with the threats posed by ISIS and al Qaeda, must be addressed. Not only are innocent people being slaughtered by these groups in the name of our faith, but a poll released just last week alarmingly found that 27 percent of Americans believe that ISIS accurately represents Islam.

But then comes the divide. On the one side we have people like Farhana Khera, attorney and executive director of Muslim Advocates. Khera explained, “We are deeply troubled by the message that the administration is sending by primarily focusing on American Muslims at this week’s summit, when the facts show that perpetrators who are Muslim comprise a very tiny fraction of extremist violence in the United States.”

Khera noted that while the White House has stated that the summit will focus on countering all forms of violent extremism, its press release announcing this conference cited only three recent incidents, and all involved exclusively Muslim terrorists: Paris, Sydney, and Ottawa.

Khera is absolutely correct that Muslims account for a small percentage of the extremist violence in America. The Southern Poverty Law Center released a report last week in anticipation of this summit that looked at violent extremism between 2009 and 2015.

The SPLC found that a majority of terror attacks committed on U.S. soil in that time frame were perpetrated by the “the ideology of the antigovernment ‘Patriot’ movement, including ‘sovereign citizens’ ” as well as white supremacist groups. For example, there was the 2012 attack in which two white supremacists drove through a black neighborhood in Tulsa, Oklahoma, and randomly shot five black people, killing three. (I’m sure some haven’t heard of this but if the gunmen were Muslim, we would’ve all known about this in great detail.)

There was also the 2011 arrest of four members of a north Georgia militia group that plotted to blow up federal buildings and release deadly ricin. And the neo-Nazi who killed seven people in a Sikh temple in Wisconsin in 2012. This attack resulted in more Americans being killed in one day than the total number of Americans killed by Muslim-related terrorism on U.S. soil from 2010 through 2014, which was four people.

But that doesn’t mean the radicalization of Muslims is not an issue that needs to be addressed. As Azhar Azeez, the president of the Islamic Society of North America, explained, “We understand violent Muslim extremism is a concern,” but he added, “the summit should not focus on it to the exclusion of other threats.”

However, Dr. Hedieh Mirahmadi, who is attending the summit and is seen by some as politically more conservative, didn’t express concerns about a potential backlash that could be caused by the summit. Rather, she was more excited at the prospect that it could provide Muslims with an opportunity to be seen not as “aggressors or victims” but as “partners with the U.S. government.” She added, the “American people need to see that we are committed to keeping Americans safe.”

Salam Al-Marayati, the president of Muslim Public Affairs Council, viewed the summit in a similar vein as Mirahmadi. But he did express concerns that our government needs to ensure that it doesn’t give “legitimacy” to the claims of ISIS and al Qaeda that they are in fact Islamic.

Al-Marayati’s vitally important point is often lost on people on the right in American politics who continually demand that we use terms like “radical Islam” to define ISIS and al Qaeda. They simply don’t grasp that this is what these terrorists are hoping we do because it will help them frame their actions as being defenders of Islam, not as the barbaric, unIslamic killers they are.

Several Muslim leaders raised an issue that may surprise some: the Muslim community also needs to be protected from extremism. Despite what we see in our media, the reality is that the overwhelming victims of Muslim terrorism are fellow Muslims, be it in Pakistan or at the hands of ISIS in Syria and Iraq.

And as Rahat Husain, the advocacy director of the Universal Muslim Association of America, noted, we have recently seen violence directed against American Muslims at alarming levels. He cited three incidents from the last week alone, with the murder of the three Muslim American students in North Carolina, the suspected arson of an Islamic school building in Houston and the assault of a Muslim man in Michigan by a person yelling religious and racial slurs. Husain hopes that the summit will be “designed to protect every American, including the Muslim community,” from extremism.

The White House has not yet released the agenda for the summit. Obviously countering radicalizations of Muslims will, and must, be discussed. My hope though is that if the summit’s goal is to truly keep Americans safe, it will also explore extremism from any source—even if that means making some uncomfortable by examining our fellow Americans who are not Muslim.