On Madagascar’s pirate island, the bones of ships fill the shallow shores, and the bones of the seafaring bandits fill the earth. The legacy of the pirating glory days is not only gold bullions or rubies or exotic textiles, but the graves of the pirates themselves.

Île Sainte-Marie, a long, thin island off the eastern African coast, was once the notorious home of an estimated 1,000 pirates. Today, it contains what may be the world’s only pirate graveyard.

The island, located four miles from Madagascar, was the ideal dock for sea-weary bandits who wanted to build a home base and still dabble in the shipping trade in the 1600 and 1700s.

In the 17th century, French sailors landed on the island and attempted to settle, but were wiped out by fever. The residents who took their place were a motley bunch: American, French, Portuguese and English pirates who’d spent years terrorizing the high seas.

The nearby waterway had a constant traffic of ships filled with spices, linens and other treasures being plied from the East Indies. It made easy pickings for the inhabitants of the pirate island, and the area became known as a hub for the illicit trade.

Today, these riches are rumored to be buried at the bottom of the waters surrounding Île Sainte-Marie. And the swashbuckling thieves who took them six feet under in what may be the world’s only pirate graveyard.

There are reportedly only 30 graves left, from what used to be hundreds. They’re shaded by wide, fanning palm trees, with skulls visibly engraved on a few of the tombstones.

A black tomb in the middle is rumored to contain the island’s most famous inhabitant. According to local lore, Captain William Kidd is buried in the tomb, sitting in an upright position as punishment for his sins.

In fact, Kidd was hung and buried in England, but his legendary loot may be just beneath the waters of Île Sainte-Marie.

In Kidd’s short career as a professional pirate—he started out in the pirate prevention business—he amassed untold riches. These lost treasures have inspired centuries of treasure hunters.

His most famous boat, the Adventure Galley, was left docked near the island at the time of his death, and it stayed there, rotting into the ocean. Though it was said to be emptied of valuables, it has drawn many an explorer to Île Sainte-Marie.

Marine archaeologist Barry Clifford has unearthed another half a dozen shipwrecks off the island’s coast in the 15 years he’s spent exploring the area for sunken treasure.

In early May, the island made breathless news when Clifford emerged from the water with a large silver ingot and claimed to find Kidd’s long-lost shipwreck in the shallow waters, though this has not been confirmed.

John de Bry, an archaeologist working with Clifford on shipwrecks in the area, has reported finding multiple identifications for the island during his research in the Paris archives.

This includes a 1733 map, which called the land mass the “Island of Pirates” and identified the location three pirate ship wrecks, he told the Cape Cod Times.

According to another record he found, ship allegiances were clearly defined on the island, and the living situation of the resident pirates was described as wooden hut accommodations with pirate flags flying above.

Some believe that the mythical anarchist pirate-founded utopia of Libertalia was actually Île Sainte-Marie, and others believe it was the model for Robert Louis Stevenson’s Treasure Island and its waters are rumored to hold not just the Adventure Galley of Captain Kidd, but other priceless wrecks.

This includes the Fiery Dragon of Captain WIlliam “Billy One-Hand” Condon, filled with rare porcelain, and perhaps some belongings of Robert Culliford, who once sailed with Kidd and at one point made off with $200,000 from a ship captured off the island’s coast.

But the pirates’ reign last didn’t last long. The French continued attempts to seize the island, and finally succeeded in the late 1700s. It wasn’t returned to Madagascar until 1960.

By all accounts, it remains a utopia, even if it’s not dripping with riches and waving pirate flags. Today it’s one of Madagascar’s most popular tourist destinations, a place where visitors can bike ride around the bays and watch the annual humpback whale migration—all while keeping an eye out for a frayed black flag.