Online, Gavin McInnes is the gung-ho, ex-punk leader of a hate group who rails against liberal decadence. At home in his wealthy New York suburb, he’s upset that liberals won’t accept him.

“I blame [George] Soros for all of this,” McInnes said during a September broadcast of Infowars, on which he bemoaned what he claimed was his supposedly degraded social status. “Soros is terrorizing me and trying to shut down my fraternal club, the Proud Boys. And he’s paying people to mess with my life, and spread these lies about me, spread fake news.”

McInnes, 48, built his profile as a New York media provocateur. As co-founder of the media company Vice, he was an architect of the ironic hipster culture of the early 2000s, a scene that sometimes gave him cover for racist remarks that he defended as jokes before leaving Vice in 2008. Years later, in 2016, he began a new chapter in life: founding the Proud Boys—a violent, ultranationalist men’s club—and selling his $2.5 million Brooklyn penthouse to buy a home in Larchmont, a tony Westchester County town 20 miles north of the city.

After months of high-profile street fights involving the Proud Boys, neighbors in McInnes’ upscale town are starting to take notice of who exactly moved next door.

A local man at a country club mocked McInnes for being involved with Infowars and tried to sabotage his application to join, he claimed in the September broadcast. One of his daughter’s tutors allegedly stopped working for the family, citing McInnes’ politics. In a June broadcast, he claimed to have gotten into an altercation with a neighbor and kicked the man’s dog.

“I see it in my neighborhood,” McInnes said. “I see whispers all the time. I’m walking down the street and a housewife goes,” he made an outraged face “like she’s finally seen Hannibal Lecter walk by. And what is the horror? The horror is that I like Trump, and that if I like Trump I might be conservative.”

Larchmont residents’ real problem, locals say, is with McInnes’ role in the Proud Boys, which has been designated as a hate group by the Southern Poverty Law Center. And the Proud Boys’ involvement in a Manhattan brawl brought McInnes to the neighborhood’s attention.

On Oct. 12, McInnes spoke at Manhattan’s Metropolitan Republican Club. While he received a police escort after leaving the club, a crowd of Proud Boys lingered behind, and they brawled with anti-fascist protesters. Police arrested five Proud Boys and are seeking the arrest of four others, as well as three anti-fascists involved in the brawl.

Michael Sabino, who lives near Larchmont, said he’d been unaware of McInnes’ residence until then.

“What made me aware was the brawl at that event in Manhattan,” he told The Daily Beast. “Among my friends and people I speak with online, everybody’s disgusted.”

On Oct. 20, a Larchmont local emailed neighbors.

“So the founder of the proud boys lives in [a Larchmont neighborhood],” read the email, which was reviewed by The Daily Beast. “I suggest we buy the ‘Hate Has No Home Here’ lawn signs and ask our friends and neighbors in [the neighborhood] to put one on their lawn. Imagine if everyone had one. It would send a clear message to him and his ilk. The signs are $20 each and bulk orders ship free. Spread the word.”

The email was quickly forwarded to other community members, a person familiar with the messages said. By the following day, the campaign had “gone way beyond me and farther than anything I can manage,” the original emailer wrote neighbors, reminding them that the campaign should preach “the opposite of hate and intimidation.”

Two days later, McInnes’ wife, Emily, sent emails to people involved with the lawn-sign campaign.

“Hello neighbor,” she began. “I want to make sure some misinformation about my family is not being spread. The media has recently accused my husband of being an alt-right hate group leader. This is simply not true.”

She described the Proud Boys as multiracial and blamed the media for implying otherwise.

“The club that is affiliated with my husband is multi-ethnic and it explicitly states it does not endorse these sentiments in any way,” she wrote. “The men affiliated with my husband who were in an altercation with the masked Antifa group that is getting press recently are either people of color or married with children to African American women. To be clear—they are the ones being painted in this slanderous and completely false light.

Unlike her husband, Emily said she is a Democrat.

“I myself am Native American with a full blood mother who grew up fully immersed in our tribe, speaking our traditional language,” she said. “I would never accept racism to surround me in my own life. It is true my husband is a Trump supporter and enjoys being a provocateur at times. I think this, paired with the climate right now and his passion for free speech has caused this horribly inaccurate narrative. I vote Democrat but I respect if you don’t. My kids have been raised this way as well—to respect all humans—and it pains me to think they may be suffering because of these very adult conflicts and narratives. I wanted to make this all clear and I am happy to discuss anything at any time if needed.”

The following week, liberal writer Amy Siskind wrote a Facebook post warning that McInnes lived in Larchmont. Siskind lives nearby and announced that she was planning an anti-hate vigil “in light of events last week.”

The evening after local news site LoHud ran a story on Siskind’s announcement, the McInnes family showed up on Siskind’s doorstep. Siskind, alarmed, called the police. McInnes later told LoHud that Siskind shouldn’t have been alarmed about the unannounced visit.

Of the people gathered outside, “three of them are little kids and one was holding his teddy bear,” he said. “The pipe dream was that she opened the door and my wife would say, ‘Look, let’s talk about this. I’m a Hillary supporter. Why are you putting my beautiful children in danger?’ This was not about intimidation and I was very open with everyone.”

Although the neighborhood skews liberal, Larchmont’s Sabino said he knows people on the political right who reject McInnes. “Even they see this guy and hear what he has to say, and they don’t like it,” he said. “They’re like ‘This guy’s a jerk.’ I’m not trying to say nobody likes him. He obviously has a pretty big following.”

But Emily McInnes’ email touched on a reality of the Proud Boys that has sometimes confused media who attempt to portray the group as strictly white-nationalist: Some members are men of color. But the Proud Boys’ ideology is more nebulous than other right-wing extremist groups. Some members and former members are undeniably affiliated with white nationalism: A 41-year-old Proud Boy arrested in the Manhattan brawl attended Unite the Right, the deadly white-supremacist rally in Charlottesville, Virginia, last year.

More than anything, the Proud Boys are anti-leftist. Members talk openly of beating leftist opponents (“commies” and “antifa”) in the streets. Proud Boys at a violent Portland, Oregon, rally wore T-shirts valorizing Augusto Pinochet, a Chilean dictator who murdered tortured and murdered leftist political foes. When a prominent Twitter user made videos mocking the Proud Boys, a member showed up at his home. Members have also variously expressed sexist, anti-Semitic, and anti-LGBT views. The ideological flexibility gives some space for members of color; the aforementioned Proud Boy who attended Unite the Right also runs with a primarily Latino skinhead crew. It also gives plausible deniability for figures like McInnes, who constantly toe the line of acceptable hatred.

“Racism isn’t a thing,” McInnes said in the same September Infowars appearance in which he described neighborhood spats. “So why are you so obsessed with it?”

McInnes uses these broadcasts to craft his public profile as an edgy, conservative grouch. He’s used punk-rock posturing to present himself as a counterculture voice in decidedly un-punk outlets like Fox News. When he talks about street violence, it’s nostalgic, hitting Fox News-friendly notes about the supposed decline of masculinity.

Members of McInnes’ old scenester circles have also called out the contradiction: Being a rich conservative isn’t very punk. When McInnes praised Sheer Terror, a hardcore band that sings about being broke and miserable, the band denounced him and the Proud Boys. “The whole ‘Conservatism Is the New Punk’ thought is a fucking travesty, and a sad joke,” Sheer Terror’s frontman told RVA Mag. “I don’t want to hear about how you’ve ‘grown up, and have kids and a mortgage.’ You don’t eat shit just because the toilet’s closer than the fridge.”

Even McInnes’ mockery of the suburbs—a common punk trope—takes on a right-wing bent. In the September Infowars broadcast, McInnes pushed an anti-Semitic conspiracy theory, claiming his neighborhood spats were the effects of a campaign by George Soros, a billionaire philanthropist and prolific Democratic donor who is Jewish.

“I’m laying on the couch here bitching about the sort of pariah status that has been imbued upon me,” he told viewers. “The reason it’s relevant to your life, is I believe it can be traced back to Soros.”

Right-wingers have blamed Soros for everything from immigration to Black Lives Matter to efforts to count every vote in the 2018 midterm elections. The blame game echoes previous efforts to portray wealthy Jews as mass manipulators, efforts that have led to pogroms and genocides against Jews. Most recently, Soros was sent a bomb allegedly by a rabid Trump supporter.

In McInnes’ case, he accused Soros of funding a network of “lesbian” lawyers who harass the far right. (He also falsely accused Soros’ staff of organizing Unite the Right “to discredit us all.”)

“I think I’m part of this sort of computer virus the Soros machine sends out in the system to mess with your life,” he said.

But there’s something more sinister than simple conservative shit-talk at play: With one foot in the world of right-wing street fights and the other in upstate country clubs, McInnes is marshaling an extremist group from a safe remove. One Gavin McInnes champions violence and reaps speaking fees, the other doesn’t care for critical emails from neighbors.

“Of course he’s got to live somewhere,” Sabino said. “But I consider this as factual: The Proud Boys are a gang. They’re listed as a hate group. When the leader of a hate group moves within driving range of you and people you care about, that’s concerning. That’s not petty, that’s real.”