In 1645, when the London publisher Humphrey Moseley agreed to print the “Poems of Mr. John Milton,” he requested that Milton have his portrait engraved for the frontispiece. Until this point, author images were generally reserved for religious pamphlets and posthumous poetry collections. But Moseley believed that showing off his living authors in classical poses—often wearing Laurel wreaths—would elevate their work, their personas (and, probably, Moseley) to the high status as England’s dead literati.

Moseley sent Milton to William Marshall, a.k.a. engraver to the stars. (Marshall’s client list included John Donne and King Charles I.) Milton was happy to oblige—he did not shy away from celebrity. But the engraving was a disaster. Milton’s right eye looked swollen, his nose was overly long, and worse, the caption proclaimed him to be 21, when in fact, he was 37 (and most certainly looks it in the engraving). Despite Milton’s protestations, Moseley decided to run the engraving. But if the author couldn’t control his appearance, he could at least have the final word. He wrote a brief commentary of Marshall’s work in Greek and then had Marshall add these lines to the frontispiece:

“Looking at the form of the original, you could say, perhaps, that this likeness had been drawn by a rank beginner; but, my friends, since you do not recognize what is pictured here, have a chuckle at a caricature by a good-for-nothing artist.” (Trans. David Masson, The Life of John Milton, 1965.)

Clearly, Marshall did not read Greek.

These days, Milton’s engraving would never have made it past the publisher’s marketing team. Which isn’t to say that all writers are so vain as to actually care about submitting a photogenic author photo. When my first book was published, my editor advised me not to submit my driver’s license; it had happened before. Hannah Harlow, Manager of eMarketing at my current publisher, Houghton Mifflin Harcourt, says that author photos are now expected in the publishing world. Without a photo, “people would wonder what’s wrong with the author—super homely? Recluse?” She adds that a great photo can lead to coveted media bookings.

So most of us have to submit a photo, but at least we get some input about it. Authors don’t get final (or any) say over cover art. We may not even get final say over our title. But we have some liberty to dress, pose, and smile (or not) exactly as we please. It’s an exercise in vanity, often fraught, but it’s necessary protection for our vulnerable psyches. Like Milton’s snarky aside, the photo is our pre-emptive defense against the critics. We believe that if we can control how we look, then we can control what the world sees. But like the book itself, the photo is just another an object for scrutiny, judgment and, inevitably, misinterpretation.

Female authors, especially, have trouble escaping questions of appearance. Shortly after my first book went out on submission, my agent arranged a meeting with an interested publisher and dispensed two pieces of advice: “be self deprecating” and “look cute.” I was momentarily irked by this comment, but at 23, I was too wrapped up in the prospect of a book deal to really care. Looking back, this moment troubles me. My book was about the Israeli-Palestinian conflict. It would involve months of travel in violent locales. “Cute” should have been the last thing a publisher was looking for. Still, I don’t blame my agent. She was young too—only 28—and was simply doing what she thought necessary to secure a deal.

Other authors who heard this story were less forgiving. “No man is getting ‘remember to look cute’ on the way to a publisher’s meeting,” said Maggie Shipstead, author of the debut novel Seating Arrangements. “But if you do look cute, you’re more likely to get your photo printed places.” This struck her as unfair. “Men can be more aggressive about their artistic temperament and lifestyle,” she said, “but as a female novelist, you’re expected to be approachable, nice, friendly.”

Shipstead says that in person, she is all of these things. Still, she felt the need to fend off gender-based assumptions about her book. “Given that my book was about a wedding, it was going to be a struggle to distance it from the chick lit category,” she said. “So [the photo] was my little, tiny, only-for-me stance against that.” She also hoped that a serious photo would send an implicit message to her publisher that her novel deserved packaging that was more literary and less “beach read.” She lamented the fact that on Amazon, her book is categorized under “single women.” “Is it because I’m single?” she wondered. “Or because one character in the book is single?” This was exactly the kind of thing she hoped had the photo would guard against, though clearly, Amazon paid no attention.

Meanwhile, Shipstead’s hardcover photo did not turn out the way she’d hoped. She made the mistake of hiring a wedding photographer and regrets both the stylish braid in her hair and the new shirt she’d bought for the occasion—one she hasn’t worn since. Most of all, Shipstead regrets her expression. She was trying so hard not to smile that her face kept tensing up. “I look like a curious marmot,” she said. At her book launch, multiple people told her she needed a different picture.

In her paperback photo, she’s wearing her favorite shirt, very little makeup, and has her hair in a more natural style. She took the photo in the small Paris flat where she’d done a lot of her writing. Whereas her expression in the hardcover photo is assertive and challenging, her expression in the paperback photo is at once self assured and guarded. And yet readers continue to disapprove. “FIVE people in the last 24 hours have commented on my author photo,” she wrote in an email a few months ago. They seemed unduly concerned about whether Shipstead’s picture looked like her and even held the book up to her face for comparison. One woman said, “’You have GOT to get them to change this photo.’” Shipstead was at a loss for what to do and annoyed by the woman’s insensitivity. It’s “US Weekly” type criticism. But authors aren’t celebrities. Aren’t we far too irrelevant to invite that kind of negative attention?

So what’s it like to be a male author, with unlimited license to pose exactly as he pleases? In fact, men don’t necessarily have this freedom either. Lee Child is the author of nearly 20 books and his face appears on the back cover of most of them—i.e. his face is the back cover. He says that once he became established, he felt pressure from the publisher to reflect a certain aesthetic. “You’re Mr. Thriller Guy,” Child said. “Implicitly, you understand that you have to do mean and moody.”

Now that he’s an international sensation, Child says he exists in the “Tom Cruise embodies of your protagonist” phase of author photographs. “They’re selling you the author as much as they’re selling the book,” he explained. Which means lots of touch ups. “My eyes are bluer, my teeth are whiter and my skin is more tanned than in real life,” he said of his recent pictures. It’s only when fans ask for autographs that Child feels comfortable distributing pictures that reflect his actual personality. Then, he happily sends them an image with a smile.

Child admits that he doesn’t pay attention to the author photos of men. But he loves looking at the women. (Before you accuse Child of objectification, consider that Shipstead admitted to buying the Perfect Storm partly because Sebastian Junger was “really hot.”) Child acknowledges that female authors probably feel more pressure to look a certain way in their photos and are more vulnerable to criticism. But he says that a woman’s eyes have a way of pulling him in, so much so, that he feels a unique bond with the writer—and her book.

I asked Child whether he felt a bond with me, based on the picture for my debut novel, The Year of the Gadfly. He called my expression “unguarded.” “It’s like you know in advance that you’re going to entertain the reader and are smiling in triumph,” he said. Child thought my photo advertised a light-hearted book. But when I said that Gadfly included vicious bullying and teen suicide, he changed tack. “Then the photo doesn’t jive,” he said. “It needs to be less frank, more sardonic, more bleak.”

Another male author, roughly Child’s age, called the picture “coy.” I probably should have been offended. Instead, I was flattered. I liked to think that my photo conveyed more complicated undercurrents—and sent an implicit message to readers and reviewers: I—and therefore my work—is more complicated than you think, so judge me/it at your own risk. But Bill Wadman, a professional photographer who has shot the likes of Jumpa Lahiri and Malcolm Gladwell, had a different take entirely. He called my photo a “hero image,” which “could be ultra serious, ultra sexy, ultra “author-y,” but is always a front: an idealized version of who the writer is.” Maybe next time, I really will submit my driver’s license!

Clearly, everybody interprets these photos differently. (Shipstead thought my photo read “young,” “approachable,” and “mischievous.”) And there’s no quantitative way to assess how certain photographs influence readers—or to what extent they really matter. In this sense, the experience of debut author, Winston Churchill, is instructive. Churchill is thought to be the very first writer to have his photograph published on a dust jacket. The book is The Story of the Malakand Field Force: An Episode of Frontier War, released in 1898. The author was 24 years old.

Ted Morgan, who wrote a book on Churchill’s early life, described the discrepancy between the author photo and the book’s subject matter. “On the jacket was a pensive young man with thinning hair who, dressed in a morning coat with silk lapels, did not look in the least like the officer on active duty whose experiences the book recounted.”

What accounted for such an incongruous photograph? Maybe Churchill (or his publisher) wanted to position the author as an intellectual rather than as a military man. Or maybe Churchill had ideas about what a “published author” should look like. Or, maybe, he hated how the photo emphasized his expansive forehead. We just don’t know. But for Churchill, it didn’t matter. His debut sold well. The lucky bastard.