LEWES, Delaware—The whispers about Dr. Earl Bradley started years ago in this sleepy fishing village of 3,000. But most parents didn’t believe the rumors until Bradley was arrested and subsequently charged this week in the largest child-abuse case in U.S. history.

The shock and grief and guilt is everywhere in this town. As her daughter throws bread crumbs to the geese in a marsh pond, a mother stays within arm’s reach. A father at the Lewes Bakery on Second Avenue holds his daughter’s hand tightly while he tries to balance his cup of coffee. Every parent had either taken their child to Bradley or knows someone who did.

“We’re dealing, I think, with a systemic problem.”

Bradley’s office, on a busy highway surrounded by strip malls and convenience stores, looked from the outside like one of the many little amusement parks that dot the landscape of these small beach communities. There was a Ferris wheel, a merry-go-round, and a couple of yellow VW bugs with “BayBees Pediatrics” painted on the side. But beneath the façade, in the basement of the little white house and in the adjacent out-buildings, the kindly bearded doctor was, authorities now say, systematically raping and abusing the children that parents trusted him to care for.

Monday, Delaware Attorney General Beau Biden charged Bradley with 571 separate counts of abusing 103 children and said he expects the number of victims to grow dramatically as the investigation continues. Biden also is investigating how the system failed these children, how complaints about Bradley were ignored, and why colleagues with suspicions never reported him.

Bradley’s lawyer declined our request to make him available for comment.

The Bradley case may be the worst in history, but it is not an aberration. Our investigation for the CBS Evening News With Katie Couric has revealed that dozens of pediatricians, in towns and cities across the country, have been arrested for sexually molesting child patients. In the last decade alone, at least 20 pediatricians were charged—often years after initial complaints were made. And those are the criminal cases. Most complaints, child advocates say, never get past state medical boards.

Just last year, for example, Alabama pediatrician Michael Roy Sharpe was charged with raping a patient, a teenage girl. He had previously been fired from two Tennessee hospitals for sexual misconduct, but he was never disciplined by medical boards and no charges were ever filed. Instead, he left the state with a clean record and set up a pediatrics practice in Alabama. Since his arrest, several other children have come forward to say they, too, were sexually abused.

Sharpe is in jail without bond facing state charges of rape and federal charges for possession of child pornography, and he surrendered his license to practice medicine in Alabama.

Child advocates like Dr. Eli Newberger, a professor at Harvard Medical School, compare pediatric pedophilia to the priest scandal that rocked the Catholic Church. People trust their pediatricians instinctively, he says, and simply cannot process the idea that they could abuse children.

But the evidence has become too great to ignore.

"We're dealing, I think, with a systemic problem, in which there is a reluctance to act on the part of colleagues for the various business and collegial reasons, and an organized coverup," Newberger told us.

Colleagues of Dr. Robert Marion in South Carolina, for example, allegedly heard complaints about him abusing children, but they simply asked him to leave the practice. He moved into another office in the same building and kept many of his same patients. Patients had no clue about his predatory behavior until he was charged with abusing four children.

"If the perpetrator is one of their colleagues and the reporting would ruin that man's life and career, they would much sooner not report, even if it endangered children," Newberger said.

Biden says he is investigating whether doctors and nurses in Delaware violated their duty to report Bradley’s behavior—which allowed him to continue abusing children for years.

"There is a code of silence," said attorney Craig Karsnitz, who is representing parents of victims who are suing Bradley, his colleagues, and the local Beebe Medical Center, where he practiced. "And I think that code was upheld to the nth degree in this case."

We talked to one mother who complained to police in 2005 about Bradley. She had just strapped her 3-year-old daughter into her car seat after an appointment when she got a chilling question.

“She just came out and said, 'Why did Dr. Bradley kiss my tongue?'" the mother said. “We called police that day.”

But prosecutors, after police interviewed Bradley, concluded there wasn’t enough evidence to charge him or go in with a search warrant, the mother said.

Then she and her husband were advised to stay quiet. If word got out about their complaint, they were warned, Bradley could sue them for slander.

“We were also told that when Dr. Bradley was interviewed about our incident, all he cared about was, ‘Who was saying I did this? I never did this. Who was saying these things about me?’” the mother recalled. “We feared retaliation.”

Others complained as well. Bradley’s own sister, who worked as his office manager, alerted the medical society of Delaware in 2005 that parents had complained the doctor inappropriately touched their daughters. According to an affidavit filed by a Delaware State Police detective, a fellow pediatrician, Dr. Lowell Scott, routinely referred to Bradley as a “pedophile.”

Still, nothing was done, and no action was taken, until state police began investigating last year. And throughout that investigation, Bradley continued to see patients. We talked to one woman who said her 5-year-old daughter was assaulted the day before he was arrested in mid-December.

“He molested her in front of me. He said he was checking her for a bladder infection, and he actually inserted fingers inside of her,” the mother said. “And it was very quick and I didn’t question it, because I trusted him.”

In the Bradley case, the 103 children identified as victims were videotaped by an elaborate system Bradley set up in his office, police said.

His videotapes didn’t name the victims, so police have taken still photos from the video to show parents, so they can help identify the children. We talked to a half-dozen parents who said their children were abused, including the father of a 3½-year-old girl, who explained the process.

“They brought by still photography, not showing the actual act, but because they need us to recognize her and make sure it was her. You could pretty much see her position and that she had a lollipop in her hand,” the father told us. “The police officer described that he disrobed both of them and that he was rubbing his penis on her vagina.”

It haunts these parents that they left their children alone with Bradley, who set up practice in Lewes in 1994 and who had thousands of patients. He was their trusted pediatrician. He’d been chief of pediatrics in the local medical center. Some of the mothers would also have their other small children in tow for the visit.

“I had to take my other daughter to get a urine sample in the bathroom. I was out of the room for three minutes or less,” said one mother, whose 3½-year-old was on the videotapes. “You don’t think anything of that at the time, but now looking back, the detectives have told us it took less than 45 seconds.”

And so when Bradley offered to take a child to his “toy room” for a treat after the appointment, they didn’t question it.

“He took her to the toy room downstairs every single appointment,” one mother of a 3-year-old said. “He’d have her appointment, and everything would be done, and he’d say you want a prize? And take her right downstairs, two to three minutes, and come pack up and she’d have… a toy.”

One lawyer for the parents, Bruce Hudson, told us, “The best estimate I’ve heard from the authorities is over a thousand,” referring to the number of children who may have been abused by Bradley during his career. For parents, that’s terrifying. They have no idea when the phone call will come from the police, telling them their child is a victim or has been seen on videotape.

One father, a local carpenter, gave police photos of his 5-year-old daughter and 8-year-old son.

“Sometimes my daughter did go down in the basement with him—that’s all I think about, still,” he said. "I look at her when she's laying there sleeping, and just think, the poor girl. I just pray she's not one of them because she's so sweet. She's my little baby."

Jan Crawford is the chief legal correspondent for CBS News. Her 2007 book, Supreme Conflict, was a New York Times bestseller. Before joining CBS, Crawford was a legal correspondent for ABC and spent 19 years as a reporter at the Chicago Tribune. She is a graduate of the University of Chicago Law School and the University of Alabama. She lives in Washington D.C. with her four children.

An award-winning investigative reporter, Howard L. Rosenberg is a Washington-based producer for CBS News 60 Minutes and chief foreign affairs correspondent Lara Logan. Prior to returning to CBS in 2009, Rosenberg spent 11 years as a producer for ABC’s Nightline , PrimeTime with Diane Sawyer and Charles Gibson , and 20/20 . A native of Nebraska, he is a U.S. Navy veteran.