The numbers are stark: of President Obama’s nine new Cabinet appointments, three are women and one is Hispanic.

This has prompted African-Americans, who voted for Obama in record numbers, to question whether they are getting their fair share of representation.

Ohio Democrat Marcia Fudge, who chairs the Congressional Black Caucus, sent Obama a letter last week saying that his appointments “have hardly been reflective of this country’s diversity.” She noted numerous phone calls from constituents to the offices of the CBC’s 42 members “questioning why none of the new appointees will be able to speak to the unique needs of African Americans.”

It’s totally understandable that the CBC wants to see more African-Americans with a seat at the table, and it’s the group’s job to keep the heat on the president. But not all CBC members share the criticism voiced by their chair.

Rep. Chaka Fattah (D-PA) points out the obvious—that the president himself is black and that Eric Holder as attorney general is still in the Cabinet and responsive to the black community. “Much more important than the personalities are the policy priorities,” Fattah told The Daily Beast, adding that he is confident that when the Cabinet selection process is complete, it will reflect the country.

A few months ago, it was women complaining that the president wasn’t keeping his promise of diversity, and women weren’t being named to top positions commensurate with their clout at the ballot box. That has since worked out to everybody’s satisfaction, and after Fudge’s letter was made public, White House officials, led by Valerie Jarrett, assured her that nominees who would please her are being vetted. When press secretary Jay Carney was asked at his Monday briefing about the lack of black appointees, he replied cryptically, saying the process is not yet complete and that “posts that will be empty have not all been filled.”

At least four Cabinet-level posts are awaiting nominees: Commerce, EPA, U.S. trade representative, and the Small Business Administration. “If it turns out he has made an insufficient number of black appointments, he will deserve to get some hassle,” says David Bositis, a senior research associate at the Joint Center for Political and Economic Studies, which focuses on issues of concern to the black community. “At this time, I give him the benefit of the doubt.”

At Monday’s announcement at the White House of Thomas Perez to head the Labor Department, a number of African-American leaders were among the invited guests, and Bositis cites their presence as evidence that Obama is not aloof to the community. Ben Jealous, who heads the NAACP, is a regular visitor at the White House, along with MSNBC’s Al Sharpton, who is there so often that Bositis quips they must have a room for him.

If Democrats held a majority in the House, then the CBC’s relationship with the White House would be different. They’d be chairing committees, and they’d have more access because they would have power. “They’re in a really unpleasant situation in terms of clout—they have virtually none,” says Bositis. “[Obama] meets with black leaders all the time, not occasionally, but they’re not necessarily members of the black caucus.”

It’s probably not a coincidence that Fudge’s letter arrived at the White House soon after Obama met with the Hispanic caucus to discuss immigration reform.

“A lot of this could be bruised egos,” says Kimberly Adams, an associate professor of political science at East Stroudsburg University. She points to a website noting it is 675 days and counting since Obama met with the black caucus. “They’re saying, ‘We’re still relevant,’” says Adams, who is writing a book on the utility and evolving role of the black caucus.

In her research, she says, she finds that the CBC’s priorities of focusing on unemployment, education, poverty, and health care have not changed since the group was founded with just nine members in 1969. “Whether he meets with them or not, he’s well aware of their concerns,” says Adams, adding that black unemployment, 13.8 percent compared to 7.7 percent overall, is “worthy of conversation.” That is a conversation that more black Cabinet members might bring out of the shadows and into the forefront of the country’s priorities.