“The financial system failed the American people…,” Goldman Sachs CEO Lloyd Blankfein told Charlie Rose on Friday. "How could you not blame the people [on Wall Street] that had awfully nice lives while things were going well? How could you not blame them when things turned badly?”

How right he was. Perhaps more surprising is that the man who said that with such eloquence and awareness is the one currently in the national hotseat. First, the SEC’s April 16 fraud charges for allegedly not disclosing the provenance of the doomed-to-fail CDOs they were peddling, then, hours before the Charlie Rose interview aired, news that the U.S. Attorney’s office in New York has begun a criminal investigation.

Even among Goldman’s bitterest competitors, a groundswell of concern has emerged on Wall Street about what is happening to the firm, and specifically how Blankfein has emerged as a Gekko-like figure.

“If he had a Bentley, I’d throw up. That’s not the kind of guy he is.”

“Lloyd Blankfein, public enemy No. 1?” the thinking goes. “Give everyone a break.”

While Blankfein, 55, has been one of the most handsomely paid executives on Wall Street –compensation of $54 million in 2006 and $70 million the following year–he’s hardly proved one of its flashier players. Compared with the hedge-fund titans he serviced, his haul was relatively small. And in terms of excesses, his behavior had nothing on former colleague John Thain (who reportedly spent $1.2 million having his Merrill Lynch office renovated) or private-equity honcho Steve Schwarzman (whose 60th birthday party is now considered a benchmark for the excesses of the last decade). “For Wall Street, he’s a pretty cerebral guy,” says one financial player who’s done business with Blankfein and asked to remain anonymous because he will likely have to do business with him again, “and I say that as someone who hates Goldman Sachs.”

• Tunku Varadarajan: Lloyd Blankfein’s PR Coup• Big Fat Story: Goldman’s Troubles DeepenSure, there’s a big apartment on Central Park West and a house in Sagaponack in the Hamptons, but nothing about his wealth or his behavior could truly be described as overly conspicuous. “The house is totally tasteful,” says a person who’s been inside of it. “It’s not showy.” Says another person who knows him well, “If he had a Bentley, I’d throw up. That’s not the kind of guy he is.”

Throughout his career, Blankfein has given primarily (and generously) to Democratic politicians like Hillary Clinton and Chris Dodd, putting him somewhat more in line with the sorts of people who now support financial reform than the financial wizards who have historically resisted it. He’s been on the boards of prominent New York organizations like the New York Historical Society and the Robin Hood Foundation. His wife Laura, a former lawyer to whom he’s been married since 1983, is extremely active in charity work, but not in the conspicuous socialite-y way that’s typical of many Park Avenue wives.

Says Blankfein’s good friend Liz Robbins, a Washington lobbyist who got to know Laura through their children at Fieldston and has since become a good friend of her husband’s as well, “She volunteers every Saturday for the Association to Benefit Children. She and Lloyd don’t just say, ‘Here’s some money to try and fix something.’ They get in there.”

Shortish and baldish, Blankfein is known for being witty and self-deprecating, which has served both to disarm people who might find him intimidating and to obscure the fact that behind the smile are shark’s teeth. “He is very quick,” says the Wall Street player. “He has that outer borough, sassy, funny Jewish presentation, like if Woody Allen hadn’t gone into the arts and played the clarinet but instead went into finance. You could imagine him doing that because of his looks and because he likes being funny and because he is funny.” At the same time, says a former partner: “You don’t get to where he’s gotten by being a wuss.”

In some way, his humor was a defense mechanism. Born in the South Bronx and raised in a housing project in Brooklyn, his father was a postal worker and his mother was a receptionist. He attended public school, earned pocket money selling concessions at Yankee Stadium (he still goes to games), graduated early, and headed to Harvard at 16, putting himself through with student loans and a scholarship. (Blankfein’s now the co-chair of Harvard’s financial task force and is said by Robbins to go back regularly to actually meet with and mentor students.)

“It wasn’t like he didn’t eat, but it was a pretty impoverished background,” says another former partner who’s known him for the better part of two decades. As Blankfein once told his former colleague Robert Steel, who recently served as undersecretary of the Treasury, about his tough upbringing: “You were either a great athlete or funny and entertaining, and I decided to go with funny and entertaining.”

After receiving his bachelor's degree, Blankfein returned to Harvard for law school, then practiced as a tax lawyer. He also had a Las Vegas habit. “Gambling is about probabilities and so is trading,” says one of the former partners. “There’s a certain constant with risk management but with Lloyd I don’t think there’s a need to take personal risks the way there was with Clinton [and sex]. I don’t think Lloyd is like that.”

Around 1981, Blankfein applied unsuccessfully for jobs at different banks. Eventually, he got hired as a gold trader at J Aron, a little known company recently acquired by Goldman Sachs. Among those who knew it, J Aron maintained a certain renegade reputation, and Blankfein, who smoked heavily and wore a beard, was in some way more of a fit there than at the parent company.

But as profits at Goldman began shifting from investment banking to trading, Blankfein’s rise continued apace. “In the end, Goldman’s culture is about making money for the firm. That’s what comes first,” says the Wall Street player. “It’s a puritan work ethic on steroids and to some extent that description fits Lloyd.”

In 2006, when Goldman’s CEO Hank Paulson was hired to go take over the treasury under George W. Bush, Blankfein became the company’s 11th leader in 140 years. (He’s since quit smoking and no longer has a beard.)

To those who do not like Blankfein—and there are many—his rise is seen as proof that in the transition from being a trusted investment bank with a long-term outlook to a trading outfit looking for short-term profits and rosy quarterly earnings, Goldman lost its way. (Of course, those profits and earnings reports have been pretty damn good, lately.)

In the course of reporting this story, nearly half a dozen former partners told The Daily Beast that Blankfein is as good as it gets where it comes to trading (“the smartest guy in the room,” says one) but diminishes, sidelines and pushes out people who come from investment banking and are not “yes” men. “He’s a kiss up kind of guy,” as another partner puts it .

Blankfein’s closest ally within the company, Gary D. Cohn, the firm’s chief operating officer, worked with him for many years on the trading side. And as one of these sources points out, Cohn has a house in Bridgehampton just minutes away from where his boss and his boss’ wife spend their spare time. Other top allies in the company include CFO David Viniar, who comes from credit risk management, Vice Chairmen J. Michael Evans and Michael Sherwood, who both came from Goldman’s securities divisions.

All of this has contributed to a sense within the financial services industries that Goldman has become so insular that it mimics a crazy John Grisham novel come to life. “It’s like that book, The Firm,” says the Wall Street player. “They can’t shake hands with you, they can’t let you in. They have to cut their hands and smoosh their blood together to gain entrance to the club.” (Indeed, besides Blankfein, not a single person currently working at Goldman agreed to speak for this story.)

Still, even among his financial world detractors, not a single one that I spoke to believed Goldman was guilty of anything illegal. Rather, says the Wall Street player who’s done business with the firm, the problem is that they screwed too many people for far too long–and that with the government now looking for a scapegoat, the firm sits in a precarious position.

“Why do people hate Goldman?” the source says. “They hate them because they’ve operated with enormous success and arrogance. If there’s a nickel on the ground that your grandmother dropped, they’ll dive down and take it before she can pick it up. There is just this relentless drive for success.”

Some believe that argument extends to Blankfein as well.

“Lloyd is a very smart guy,” says yet another former partner. “He’s a commercial animal, but his battle has been to balance those instincts with the long-term best interests of the firm and the client. And the question is, did he allow his commerciality to overtake everything else?”

Correction: Blankfein's apartment was first reported to be on Park Ave.

Jacob Bernstein is a senior reporter at The Daily Beast. Previously, he was a features writer at WWD and W Magazine. He has also written for New York magazine, Paper, and The Huffington Post.