Tangled Tree

Exclusive: Ben Affleck’s Ancestor Did Not Own Slaves, Tax Documents Show

Georgia tax and census records show that Affleck’s great-great-great-grandfather likely owned no slaves at all—though he was the executor of estates that did.

04.28.15 9:25 AM ET

None are safe from the Sony Hack, as Ben Affleck found out last week when leaked emails revealed his request to have a slave-owning ancestor edited out of the PBS program Finding Your Roots. Affleck came clean via Twitter, while the program’s producer and host, Henry Louis Gates, answered accusations of censorship by attributing the ommission to an entertainment-value editorial call: Other ancestors just had more interesting stories, Gates told Boston.com.

But new research conducted by The Daily Beast and Georgia genealogist Barbara Stock suggests that Benjamin L. Cole—Affleck’s great-great-great-grandfather on his mother’s side—was hardly the wealthy Southern plantation owner that Finding Your Roots claimed, and that the small-town Georgia sheriff may not have owned slaves at all.

Using Georgia tax digests from Chatham County and Savannah—which give an annual account of wealth, including slave ownership—in conjunction with the census, which took a count only once a decade, Stock found evidence that while Affleck’s ancestor was the executor of estates that included slaves, he owned none himself.

Stock found that the 1850 federal census slave schedule—used by PBS, Breitbart, and others to show Cole’s wealth and slave-holding status—is in conflict with the tax digests. Though the 1850 census record doesn’t indicate Cole was simply a trustee, the tax digests show he owned no slaves or land at that point.

Johni Cerny, the lead genealogist on Ben Affleck’s episode of Finding Your Roots, confirmed to The Daily Beast that she consulted the 1850 census, and said she drew the best possible conclusions based on that evidence.

According to the tax digests, Benjamin L. Cole did not own slaves, although in 1863 and 1864 his second wife, Georgia A. Cole (Affleck’s third-great-grandmother), did own one female slave over the age of 12—which of course means that Affleck is still very much a descendant of slave owners. (Breitbart claims to have found at least nine other ancestors of Affleck’s who owned slaves.)

“The 1860 federal census slave schedule shows Benjamin having 23 slaves as executor for the Ann S. Norton estate (Cole’s mother-in-law from his first marriage), eight slaves as trustee for S. L. Speissegger and her children, but none for himself,” Stock wrote in an email of her findings to The Daily Beast.

Cole’s mother-in-law left 11 slaves to him to hold in trust for her three grandsons until they reached age 21. Their names were Buffy, John, Betty, Robert, Lydia, Caroline, Alfred, Ned, Susan, Tommy, and Harry.

Georgia genealogist Barbara Stock says that while the 1860 census directed its census-takers to note the names of executors and trustees (as it did in Cole’s case), the 1850 census—the one that listed Cole as a slave-owner—“only had to list the name of the person managing the slaves.” In the instructions for the 1850 slave schedules, it was noted that “the person in whose family, or on whose plantation, the slave is found to be employed, is to be considered the owner—the principal object being to get the number of slaves, and not that of masters or owners.” This likely accounts for the discrepancy in Cole’s case between the 1850 and 1860 censuses, and the 1850 census and the yearly tax digests.

The legal difference between owner and executor is clear. An executor is someone who is appointed by a will to dispose of the deceased’s estate, as William Wiecek, professor of public law and legislation emeritus at Syracuse University College of Law, explained in an email to The Daily Beast. “You won’t own the property; you will distribute it to those who do.”

A trustee—as Cole was for Norton’s grandchildren—has “the control power, but not ownership,” Wiecek said. “Your legal duty would be to administer the property according to my directions, and distribute the income or profits to the beneficiary.”

All the same, Wiecek noted, even as an executor, “[Cole] would have been a facilitator of slave ownership, and certainly someone who supported the system, though not himself the legal owner.”

According to the law at the time, Cole wouldn’t have been permitted to benefit financially from these slaves. Had he used the slaves on his personal farm, he would have had to pay. Had he rented them out, he would have had to distribute the profits to the estate’s beneficiaries.

A trustee—as Cole was for Norton’s grandchildren—has “the control power, but not ownership,” Wiecek said. “Your legal duty would be to administer the property according to my directions, and distribute the income or profits to the beneficiary.”

“But you never know,” said Alfred Brophy, a law professor at the University of North Carolina–Chapel Hill. “He could have been managing the slaves, and saying, ‘In your free time, go pick some cotton on my plantation as well.’”

Get The Beast In Your Inbox!
By clicking "Subscribe," you agree to have read the Terms of Use and Privacy Policy
Thank You!
You are now subscribed to the Daily Digest and Cheat Sheet. We will not share your email with anyone for any reason

“Your finding just shows how deeply the institution of slavery penetrated American finance,” Brophy said, “how deeply the United States law is implicated in the sins of slavery.”

Though the technicality of holding slaves and owning them may be a distinction without a difference for many, it does change the narrative that Finding Your Roots and Gates presented to Affleck in the edited-out segment.

In the original transcript, published last week by Gawker, Gates shows Affleck the slave schedule of the 1850 Census and says, “Your third great-grandfather owned 25 slaves. He was a slave owner.”

And then a voice-over: “These holdings put Benjamin Cole among the southern elite. Only about 10 percent of all slave-holders owned 20 slaves or more.”

“God...It gives me kind of a sagging feeling to see, uh, a biological relationship to that,” Affleck said upon learning of the link.

Genealogist Barbara Stock’s extensive research counters this argument.

“If the slaves had been his, I would agree [that Cole was among the elite],” she told The Daily Beast. “However, they were owned by estates and trusts—he did not own them himself. His farm was in the mid-size category for Georgia farms of that period. Of course, he also drew a wage as the sheriff.”

According to county records and the agricultural census schedule, Cole spent part of his 30s as a farmer, purchasing a 157-acre tract about eight miles outside the city of Savannah in 1846 for $800 and expanding his lot by 109 acres two years later for $500.  

On his farm, valued at $4,000 in 1860, lived two horses, one mule, four milking cows, 15 other cattle, and 20 pigs. Cole harvested 150 bushels of Indian corn, 18 bushels of peas and beans, 40 bushels of Irish potatoes, 50 bushels of sweet potatoes, one ton of hay, and made 50 pounds of butter.

Stock noted that a farm of this size would likely require workers outside the immediate family, and it’s not clear what kind of labor powered the Cole farm. Slave labor certainly shouldn’t be ruled out. He may have used the Norton estate slaves or could have hired outside slave laborers.

Either way, his 265-acre farm with 100 improved, farmable acres was a mid-sized farm for that time period. The 6 percent of white Georgians who made up the state’s planter class—or the Southern elite, as Gates put it—were generally defined as those owning 20 or more slaves and whose plantations ranged from 200 to 500 improved acres with a larger total acreage.

As the largest percentage of farms in Georgia fell in the 100- to 500-total acre range, Cole would have been considered an average Georgia farmer during that time.

Neither Affleck nor Gates returned requests for comment on The Daily Beast’s findings.

Did Gates know about Cole’s commonness and did it lead the Harvard professor to agree to Affleck’s request that Cole be excluded from the final product? In leaked emails between Gates and Sony CEO Michael Lynton, Gates writes, “One of our guests has asked us to edit out something about one of his ancestors—the fact that he owned slaves … Now, four or five of our guests this season descend from slave owners, including Ken Burns. We’ve never had anyone ever try to censor or edit what we found. He’s a megastar. What do we do?”

Gates went on to note, “And he wasn’t even a bad guy. We don’t demonize him at all.”

Responding to the allegations that he whitewashed Affleck’s history, Gates said he omitted Cole not after Affleck asked him to, as the leaked Sony emails suggest, but because Cole’s story wasn’t compelling enough. “In the case of Mr. Affleck—we focused on what we felt were the most interesting aspects of his ancestry—including a Revolutionary War ancestor, a 3rd great-grandfather who was an occult enthusiast, and his mother who marched for Civil Rights during the Freedom Summer of 1964,” Gates said in a statement.

Based on the public reaction to the omission, there does indeed seem to be interest in the life of Affleck’s third-great-grandfather—a farmer and longtime bailiff and county sheriff during the Civil War.

Benjamin L. Cole acted as Chatham County Sheriff briefly when he was about 42 years old, then again for nearly a decade from approximately age 45 to 54, serving during the run-up and through the fallout of the Civil War. At around age 50, Cole was too old to be drafted in the Civil War; the Confederate states required healthy white men between the ages of 18 and 35 to serve. But he did serve in the local guard, according to militia census data, and his obituary notes he was one of the original organizers of, and at one time a captain on, the Phoenix battalion, a volunteer infantry formed in 1861.

Cole likely awaited Sherman and his 60,000 plundering soldiers during the general’s 1864 March to the Sea. Though Cole was a rifleman, as a farm owner, he was perhaps relieved when his city gave up to Sherman’s “total war” campaign without a fight.  

Thirteen years prior, Cole had been a committee member of the Constitutional Union Association, a political party that fought against northerners and southerners alike who aimed to dissolve the union, “which we believe to be the noblest fabric of government which the wisdom of man has ever reared or the bounty of Heaven ever blessed,” according to a local paper’s write-up of one of their meetings. During an 1851 party meeting, Cole proposed a motion that urged Georgians to accept the Congressional compromises in 1850 that sought to resolve the national dispute over slavery. It read in part, “When the flag of the stars & stripes shall no longer wave over the Union of these States, the cause of liberty will have received its death warrant.”

After the Civil War, Cole still acted as deputy sheriff. In 1871, shortly before his death, the local paper wrote of an incident where Cole assisted in the apprehension of a drayman, or wagon driver. After the wanted, a black man named Jim Josephs, picked up the arresting officer, walked around the lot with him, and threw him into a crowd of other draymen, Cole appeared and an arrest was made.

Joseph would be one of Cole’s final arrests. Around age 57, according to his obituary, Cole succumbed to consumption of the bowels, a painful disease that had him confined to his home during the final months of his life.

His obituary read in part: “The deceased was widely known and respected as an honest, upright man, both in his public and private life. There are none of his many acquaintances but who will accord to him the reputation of always being willing and ready to do an act of kindness to all, it mattered not if it be the distressed criminal whom he might be serving in his official capacity of the stranger of but a few hours acquaintance.”

Affleck, released a statement last week on Facebook responding to the controversy surrounding his request to remove Cole from his televised family tree.

“I regret my initial thoughts that the issue of slavery not be included in the story. We deserve neither credit nor blame for our ancestors and the degree of interest in this story suggests that we are, as a nation, still grappling with the terrible legacy of slavery. It is an examination well worth continuing. I am glad that my story, however indirectly, will contribute to that discussion. While I don’t like that the guy is an ancestor, I am happy that aspect of our country’s history is being talked about.”