Invisible Manor

The House that Slavery Built

Mac Griswold tells Jane Ciabattari about an estate near the Hamptons that used to be one of the largest slave-owning plantations in the North.

In 1984, Mac Griswold and her friend, the poet Frederick Seidel, left the town harbor of Shelter Island—eight miles from the Hamptons—and canoed along Gardiner’s Creek. As they meandered along, she saw a yellow Georgian house, and then a stand of 12-foot boxwoods that were even older than the house.

“Boxwoods grow very, very slowly,” Griswold told me. “So how about the house, I said to myself? So how about the garden path those boxwoods guarded? I was determined to find out more.” Griswold, a garden historian and author of Washington’s Gardens at Mount Vernon, soon wangled an invitation from the owners, Andrew and Alice Fiske. They gave her a tour. The property was first settled by Andrew Fiske’s ancestor, the Dutch Quaker Nathaniel Sylvester, and his wife, Grizzell, in 1651. At one point, Fiske opened a door that led, he said, up the “slave staircase” to the attic where the slaves slept.

Sylvester Manor, Griswold learned, had been on one of the largest slave-owning plantations in the North. She was so intrigued that she embarked on a research project that ultimately became The Manor. The book follows 11 generations of the Sylvester family back to the 1650s, when the land was a provisioning location for a Dutch West Indian sugar company. The Manor is filled with astonishing discoveries drawn from original documents and precious archeological finds, including a slave burial ground, a Quaker cemetery, and a large pottery vessel with an Indian lip and an African- or European-style handle, an example of a startling mixing of cultures. The layers of history uncovered at Sylvester Manor show that slavery in the North was not just a passing phenomenon.

Griswold’s research took her abroad—to Europe, the West Indies, and Africa—“to see where the people came from who all ended up on Shelter Island, together with the Manhasset Indians already there,” she said. She spent years sorting, transcribing, and inventorying the papers. Her most crucial find was the 1680 will of Nathaniel Sylvester, which is a dream document for any cultural historian, not only because everything on the estate is documented, but also because it reveals how slaves were seen during that time. In Sylvester’s will, he gives the names of the enslaved “as people with family relationships,” Griswold said. By contrast, in the inventory of his estate, the slaves were listed as things.

Northern plantations differed from those in the South in treatment of the African-born slave population. “Slaves didn't live in quarters, as in the South, but in the houses of their captors, meaning that normal privacy and family life didn't exist,” Griswold said. “Also, as they weren't part of an immense agricultural system growing staple crops such as cotton, rice, and indigo, many were highly skilled and were hired out to other whites at slack times on their own ‘plantations,’ which we can really think of as large family farms. They worked alongside their owners and with indentured servants and wage laborers, but of course the pay-out for those other workers in eventual freedom or in wages didn't exist for slaves, or for their children, for many generations.”

The Manhassets, who were native to the region, were also enslaved, but “more informally,” Griswold said. Their wages were paid in alcohol (rum from Barbados) and goods such as kettles and blankets. Although a law was passed in 1676 in New York forbidding the enslavement of Indians, “Indian slaves” were often handed down as property in family wills. Others were indentured servants, like Isaac Pharaoh, a Montaukett Indian whose indenture papers Griswold found in the vault at the manor house. “Esther Pharoah, Isaac's mother, signs her son away,” Griswold tells me, “‘of his own free will’ at the age of 5 years.”

Although free to leave the manor at 21, Isaac spent his entire life with the Sylvester descendants. In winters he lived in the manor house attic, where he carved the outlines of dozens of fully rigged ships into the attic dormer walls. He was interred in the slave cemetery.

While Isaac was living there, the leading members of New England’s intelligentsia spent summers on Shelter Island as guests of the Sylvester descendants. Henry Wadsworth Longfellow, John Greenleaf Whittier, James Russell Lowell, Sarah Orne Jewett, and other writers saw the Sylvester Manor as a precious talisman of an American past, and, after the Civil War, as a sort of Tara of the North. Longfellow, author of the poem “The Song of Hiawatha,” about an Indian hero, likely met Isaac Pharaoh there.

Even after the Civil War, some blacks, who, like Pharoah, were free to leave the manor, remained tied to the estate. Griswold chronicles the fate of Julia Johnson, a free black woman who inherited land and a house from her enslaved parents, Comus and Dido. By the 1860s, Julia was convinced to sell her land back to a Sylvester descendant who wanted to develop several hundred acres as a summer resort. She sold her waterfront property for $757. Today, Griswold said, “a local real estate dealer says about $7.5 million would buy you a nice house and a view of the water.” Julia bounced around as a servant in various families, and died in Sag Harbor in 1907. “Evidently, so strong was the pull of the manor, where she had spent most of her life,” Griswold said, that she asked to be buried in the Sylvester Manor slave cemetery.

A boulder carved in 1884 marks the cemetery where Isaac Pharaoh, Julia Johnson, and some 200 others lie. The people laid to rest there were part of a society that “rejected them as full human beings,” Griswold writes. “But as they lie here, unmarked, they are also vividly present.” The Manor is a step toward restoring these once-forgotten souls to a place in our shared history.