Many of Summers' former colleagues and proteges, meanwhile, say he's misunderstood. Now they're trying to marshal support to counter a robust political opposition that appears to have caught both Summers and Obama by surprise.
The charges against the one-time Treasury Secretary and director of the National Economic Council are familiar. Critics say he's been on the wrong side of major financial debates and has an abrasive if not antiquated attitude toward women.
But these complaints, according to Summers enthusiasts, are bluntly political and often ignore other important, more positive aspects of his record.
"He gave a speech that he apologized for but a lot of people took a lot of offense to," Facebook COO Sheryl Sandberg told The Huffington Post in an interview, referencing Summers' infamous remark that women were less capable of tackling advanced science and engineering work. "It wasn't his policy. It was a speech he gave and actually the rest of that speech was fine. A lot of that speech said things that ... all of us really agree with, that it is really hard for women in terms of work-life balance to catch up."
Sandberg, who served as Summers' chief of staff when he was Treasury Secretary under Bill Clinton, noted that he gave a speech as chief economist for the World Bank in which he argued girls' education was an economic imperative.
"That speech put girls' education on the table as an economic issue, and economic issues are always taken more seriously than development issues by countries," she said. "Before that, girls' education was something education ministers worried about, which was good. But when it became something as the linchpin of economic growth, that obviously put many more forces behind it."
Sandberg has made this argument before, including in a Huffington Post blog post in 2008. Which begs the question: Why has his perception not improved more over time? After serving in the early years of Obama's administration, Summers remains the most divisive economic figure in Democratic Party politics.
Not even his harshest critics deny his brilliance. He has been integral to some of the thorniest dilemmas of the past few decades, and he has generally been seen as the smartest guy in the room when those dilemmas were addressed. But Summers' judgment and the way he treats colleagues has routinely come under fire.
Summers clashed with fellow members of the Obama team. Noam Scheiber, in his book The Escape Artists, reported that he sidelined Christina Romer, then the chair of the Council of Economic Advisers, when she tried to persuade the president to pursue a larger stimulus package. Meanwhile, Sheila Bair, the former Federal Deposit Insurance Corporation chair who had warned of trouble in the subprime lending market, was left out of most major financial policy decisions.
"The attitude was that they already had all the answers and were talking to us only because the president wanted them to," Bair wrote in her recent book regarding Obama's top economic advisers. "Larry and [Treasury Secretary Tim Geithner] didn't seem to care about the political beating the president took on the hundreds of billions of dollars thrown at the big-banks bailouts and AIG bonuses, but ... I don't think helping homeowners was ever a priority for them."
That it was two prominent women with whom Summers clashed has not gone unnoticed.
"Summers' record of sexism speaks for itself -- and it's disturbing," said Shaunna Thomas, co-founder of the feminist group UltraViolet. "Character and the capacity to see women as equals should factor heavily into Obama's choice for head of the Federal Reserve -- and ignoring those things would speak volumes to women across America."
But other women who have worked with Summers argue that his bluntness shouldn't be confused with insensitivity or sexism.
Diana Farrell, former deputy director at the National Economic Council, told The Huffington Post that she worked "closely and very collaboratively" with Summers as he helped pull the economy out of crisis. He "has a formidable command of the full range of economic, regulatory and monetary issues and applied these -- loyally and brilliantly -- to the service of our country," Farrell said.
Kathleen McCartney, now the president of Smith College, served as dean of the Harvard Graduate School of Education when Summers was president of the university. McCartney said that she and Summers had "a great working relationship," and they "worked together to use the Harvard platform to make a difference in children's lives."
Certainly, Summers has jousted with male colleagues as well as female ones. Former Office of Management and Budget Director Peter Orszag can attest to that. And for a number of Summers' detractors, it's the positions he's taken in those battles -- not the person he's battling -- that's cause for concern.
Clinton and others in office in the late 1990s have said Summers erred in pushing a de-regulatory platform at the time, during a period when Glass-Steagall financial regulations were repealed and the derivatives market was unshackled.
"Anyone who is being considered on Fed chairmanship should have to talk about Glass-Steagall, about the repeal and about their position on a modern-day version of it," former Sen. Byron Dorgan (D-N.D.) told HuffPost. Dorgan, now a lobbyist with Arent Fox, was one of only a handful of senators to vote against repealing the regulations.
"My hope is that [Obama will] look outside the very small talent pool that they usually look to," he said. "You know what I'm talking about? A talent pool that comes from Wall Street. There are a lot of really talented well qualified people outside that talent pool."
Critics also point out that Summers was part of an Obama economic team that let the banking industry off easy after it cratered the economy.
"Summers protected the big banks from being taken over when they were insolvent, and he has been a consistent foe of any serious effort to rework the financial industry," said economist Dean Baker, co-director of the Center on Economic and Policy Research. For good measure, Baker noted that Summers earned millions from financial sector interests, including bailed-out Citigroup, around his stints in the public sector.
But here too, Summers defenders argue that these critics are omitting key facts. Summers was more resistant to immediate deficit reduction in 2009 and 2010 than other Obama advisers, they say, and he made the argument at the time that additional stimulus was needed. As for his divisiveness, Brad DeLong, a University of California, Berkeley economist and Summers supporter, had three explanations.
"A byproduct of his personality, a habit -- learned in graduate school, and before -- of overstating points in order to provoke in the belief that he is going to be judged on the smartest thing he says, and most important, an American left that does not understand who Obama is," DeLong said.
"The Democratic Party -- the left especially -- chooses somebody without a national political reputation, projects their hopes and their fantasies onto them as a blank canvas, enthusiastically supports their run for president, is then bitterly disappointed, and blames evil advisors who force Obama to listen to Jamie Dimon rather than Elizabeth Warren," he explained.
As DeLong sees it, Summers became a convenient bogeyman for Democrats disaffected with the president's performance. Obama himself told House Democrats on Wednesday that progressives were treating his former adviser as a "whipping boy." He urged them to not believe everything they read, singling out articles from this website in particular.
But part of the problem facing Summers is not just that he's taking one for Obama. It's that his main areas of perceived weakness happen to be the areas of perceived strength for the other top contender for the Fed post.
Fed Vice Chair Janet Yellen picked up on signals of impending financial troubles and highlighted them in speeches for years preceding the 2008 crash. She warned about low-quality loans and an unstable banking system, and supported transparency at the Fed before it was politically popular to do so. Her defenders also argue that she's attuned to blue-collar economic issues -- like wage growth and housing -- and not just the balance sheets of the major financial institutions. She'd also be the first woman to head the Fed.
And so, while many expect both she and Summers to have similar approaches to monetary policy -- both appear to believe the Fed should still use the tools at its disposal to help the economy -- there are contrasts to be drawn. Even Summers' defenders admit that they'd be pleased to see the Fed's glass ceiling broken.
"Obviously, I've known Larry for over 20 years as a teacher, colleague and friend and I think he would be a remarkable choice for the job," said Sandberg, who famously wrote a book about female leaders at the highest levels of government and business. "I don't know Janet as well but she would be a good selection, too. We're lucky as a nation that we have such a great choices."