New York’s 15th Congressional District, which includes parts of Central and East Harlem, Washington Heights, and the Upper West Side, has sent only two representatives to the House in the last 65 years: Civil-rights pioneer Adam Clayton Powell Jr. and the man who defeated him, Charlie Rangel.

Now Powell’s son, New York State Assemblyman Adam Clayton Powell IV, is looking to unseat Rangel, whose descent into scandal over the last two years has left his seat potentially vulnerable. Rangel is a Manhattan institution and still considered a strong favorite by political observers, having easily defeated the younger Powell in 1994. But the ethics trial set to heat up in the weeks leading up to the September 14 primary offers an opening for an upset. The only polling in the race, in July, showed Rangel generating a weak 39 percent support from voters in the four-man field, but with a significant lead over Clayton Powell, who came in second at 21 percent.

Another scenario: Rangel could stay in the race but resign after winning the general election.

Powell has no qualms about hitting Rangel hard over the ethics investigation, which is looking into whether Rangel skirted taxes while living in four rent-controlled apartments, failed to disclose income from a Dominican vacation home, and raised funds for City College of New York through his congressional office. (Rangel has denied wrongdoing.)

“He lost his chair of the Ways and Means Committee, he’s 80 years old, he’s been there 40 years, what does he need two more for?” Powell said in an interview. “People are literally ready to turn a page. Perhaps the ethics charges may push it, but people were ready before.”

It’s an obvious irony (and one Powell complains of having to constantly acknowledge) that his father faced serious ethics charges in the House himself and was even expelled—then reelected—before being defeated by Rangel. Nonetheless, the elder Powell is still an important political pioneer and there is a bronze statue of him today outside the Adam Clayton Powell Jr. State Office Building on Adam Clayton Powell Jr. Boulevard. (The elder Powel died in 1972.)

• Tom Watson: Where the Billions Will Go

Powell doesn’t think those scrapes will be a factor in the race. “When you’re a public figure, people take shots at you and that’s the reality,” he says. “But people know my record and they’ll see I’ve been an honest, decent public servant for 20 years.”

David Bositis, who researches black electoral politics at the Washington, D.C.-based Joint Center for Political and Economic Studies, predicted that Powell’s father, while offering a boost to his name recognition, would not likely give the son a huge advantage over Rangel.

“My guess would be a substantial majority of the voters in Harlem don't even know who Adam Clayton Powell Jr. is,” Bositis said. He noted that the African-American electorate skews younger, meaning fewer voters have a personal connection to the former congressman. On the other hand, the district is now 46 percent Hispanic, and Powell’s campaign is hoping to capitalize on his heritage—his mother is from Puerto Rico, where he was raised—to make electoral gains.

Powell’s biggest fear, based on pure speculation, is that Rangel will win the primary, then resign and handpick his successor—who would be appointed by party leaders and cruise to an easy victory in the heavily Democratic district, where winning the primary is tantamount to victory.

Another scenario: Rangel could stay in the race but resign after winning the general election, leading to a special election that would undoubtedly include a more competitive field than the one Powell faces now. The challenger and his staff are already pre-emptively attacking Rangel over such possibilities—even before the incumbent has indicated his plans.

“I find it ironic that Rangel, who fought and served with distinction in Korea, now wants to do the same kind of thing they do in North Korea,” one campaign aide told The Daily Beast, “which is make sure the outcome of the election is known before they even have elected the successor.”

Plenty of longtime incumbents have survived serious ethics charges to be reelected at least once in recent years, including former Rep. William Jefferson (D-LA) and Rep. Don Young (R-AK). But voters’ patience is not infinite, and this year’s anti-incumbent sentiment threatens even the most secure veterans. Just this week, seven-term Rep. Carolyn Cheeks Kilpatrick (D-MI) was defeated in her party’s primary largely due to the travails of her son, former Detroit Mayor Kwame Kilpatrick, who was convicted on felony perjury charges.

With attention to Rangel’s scandals ramping up just as he prepares for the voters’ judgment, he knows he can’t take his challenger for granted—especially with the ghosts of Harlem past knocking at his door.

Benjamin Sarlin is Washington correspondent for The Daily Beast. He previously covered New York City politics for The New York Sun and has worked for talkingpointsmemo.com.