Things are going well for painter Glenn Ligon. Barely 18 months ago, the Obamas hung one of his works in the White House. Right now the art world is toasting him too, with a retrospective at New York’s Whitney Museum.

But success doesn’t seem to have mellowed this artist.

Sitting for an interview in the middle of his excellent Whitney show, Ligon is balking at so many questions that you imagine asking, “Is the sky blue?” and him answering, “No, it’s orange. And what does color mean anyway?” Of course, you’re not asking about blue and orange, but about black and white, and those two tones as the color of skin. Ligon makes art that is praised for its treatment of blackness. But getting him to discuss race is like getting J. Edgar Hoover to talk crinolines.

How does Ligon feel about having his work selected for our first “black” White House? He answers with a neutral “Because [Barack Obama] is in an administration that I actually believe in, it sort of means more to me.” Fair enough. There’s no obligation to feel racial pride. “Obama is the first African-American president,” Ligon goes on, “and for some people that means a great deal, and for some people it means very little.”

Yet it seems bizarre to refuse to talk race when a lot of your art is so clearly about it. The picture in the White House, Black Like Me #2, is built around a phrase from the 1961 book of the same name, in which white journalist John Howard Griffin darkens his skin. Ligon takes the all-capped words “ALL TRACES OF THE GRIFFIN I HAD BEEN WERE WIPED FROM EXISTENCE,” and stencils them down the white surface of the work, in ever-thicker black pigment, until they almost disappear into darkness.

Ligon insists that the black and white of his trademark text paintings is simply about the colors of print, not skin: “If you read a newspaper, it’s in black and white, too …I think black and white is neutral … It’s information.”

As the artist talks, the 1988 picture hanging right behind him, titled I Am a Man, bears only those bold words, in black capitals on white. It is based on the placards black strikers carried in Memphis when Martin Luther King Jr. was killed. “If you don’t know that history, does it address race?” he asks, avoiding the fact that the painting prods us to recall its backstory.

• View Blake Gopnik’s Daily PicsYou can certainly see why an artist of his talents might not want to be pigeonholed as “just” a black artist. But while race is not the only thing Ligon’s art is about, it is clearly the scaffold on which it is built. We’ll all have to talk the subject through, even if he won’t.

Blake Gopnik writes about art and design for Newsweek and The Daily Beast. He previously spent a decade as chief art critic of the Washington Post and before that was an arts editor and critic in Canada. He has a doctorate in art history from Oxford University, and has written on aesthetic topics ranging from Facebook to gastronomy.