If he were still hosting the Late Show and had Donald Trump on as his guest during this presidential campaign, David Letterman said he would have “gone right after” him. “I would have said something like, ‘Hey, nice to see you. Now, let me ask you: What gives you the right to make fun of a human who is less fortunate, physically, than you are?’” The comments came during an interview with The New York Times this week in which Letterman was asked to comment on the controversy that followed Jimmy Fallon’s lightweight interview with the GOP presidential nominee. On Trump’s decision to publicly mock a physically disabled Times reporter earlier in the campaign, Letterman added, “if you can do that in a national forum, that says to me that you are a damaged human being. If you can do that, and not apologize, you’re a person to be shunned.”

Letterman also responded to Hillary Clinton’s use of a 2012 clip of Trump on the Late Show in a campaign ad that debuted during the Olympics this summer. “I was flattered. I was pleased,” he said of the ad, in which he called out Trump for making his branded shirts and ties overseas. “I felt like I still have a small voice in this.” A year earlier, Letterman labeled Trump a “racist” on his show for questioning President Obama’s birth certificate and ability to get into Harvard.