America’s Disappearing Millionaires

It hasn't been a good recession for the rich. The late boom was extraordinarily top-heavy, with the overwhelming majority of economic gains seemingly defying gravity and flowing to the top rung of the economic ladder. Now those with the most assets and income have the most to lose. Add together the declining markets, an imploding finance sector, a real estate rot that has eaten its way up from the ground floor to the penthouse, and the predations of Bernie Madoff and Sir Allen Stanford, millionaires who ripped off other millionaires, and, as my Newsweek colleague Robert Samuelson notes, these are tough times for the wealthy.

As if market forces and malevolent actors weren't enough, the rich are now finding themselves targeted by politicians. Strapped for cash, states, cities, and the federal government are seeking to soak the rich—or at least to make them pay taxes at the same marginal rates as they did in the Reagan years, which many on the right regard as an act equivalent to executing landed gentry. Some politicians have even suggested that we fund health care by slapping a surtax on people with annual incomes of more than $1 million.

This tactic isn't likely to work, in large part because people who make a lot of money are quite effective at swaying public policy. What's more, the wealthy have many defenders who argue that taxing the golden geese will cause them to fly away. In May, the Wall Street Journal op-ed page argued that millionaires fled Maryland after the state legislature boosted the top marginal state income tax rate to 6.25 percent on the top 0.3 percent of filers. "In 2008 roughly 3,000 million-dollar income tax returns were filed by the end of April," the Journal notes. "This year there were 2,000, which the state comptroller's office concedes is a 'substantial decline.' "The Journal uses this small sample to warn the federal government and states with progressive tax structures and lots of rich people—New York, New Jersey, California—to heed the lesson. Tax the wealthy too much, and they'll leave.

Such logic makes sense to the Journal's op-ed page staffers, who inhabit an alternative universe in which people wake up in the morning and decide whether to go to work, innovate, or buy a bagel based on marginal tax rates. But if people were motivated to choose residences based solely on high state income taxes, then California and New York wouldn't have any wealthy entrepreneurs, venture capitalists, or investment bankers—and the several states that have no state income tax, which include South Dakota, Alaska, and Wyoming, would be really crowded with rich people. Maybe Maryland's rich folks just had a crappy year in 2008. Robert Frank, who chronicles the ultra-rich economy for the reality-based portion of the Journal, notes, citing data from the Institute on Taxation and Economic Policy, that there's "evidence that [the state's] millionaires didn't disappear because they moved, they disappeared because they are no longer millionaires."

Now, there are millionaires—the kind who make more than $1 million per year in taxable income, of which there are very few—and there are millionaires, the kind who have assets totaling $1 million, of which there are more. A lifetime of careful saving and investing can vault seemingly middle-class Americans into that rarefied realm, as The Millionaire Next Door argues. Many people whose annual income places them in middling tax brackets have more than $1 million in assets. But their ranks are thinning, too, even though they haven't been singled out for punishment.

Consulting firm CapGemini conducts an annual census of high-net-worth individuals, defined as people with at least $1 million in investable assets, excluding primary residences. "We've been doing this report for 13 years and haven't seen this kind of loss of wealth since we started," said Ileana van der Linde, principal at CapGemini's wealth-management practice. North America saw an 18.5 percent decline in its high-net-worth population, from 3.02 million in 2007 to 2.46 million in 2008. Even though they were more likely to be diversified in bonds and cash—instead of simply plowing money into stocks—the HNWIs' collective net worth fell from $10.85 trillion in 2007 to $8.44 trillion in 2008, down 22.2 percent. The ultra-HNWIs—those with at least $30 million in assets—suffered even more. That segment of the population fell 25 percent.

CapGemini's survey contains some interesting geographic wrinkles. High-tax areas like New York and California—places where politicians have been talking about potentially raising taxes on the rich to deal with budget crises—held up better than the national average. The New York region, which has the most HNWIs, at 561,000, lost only 13.6 percent of its rich population last year. Los Angeles and San Francisco lost 17.8 percent and 15.3 percent, respectively. As for Maryland, the Baltimore metro area had a 19 percent drop in high-net-worth population.

Comparative tax havens like Florida, Nevada, and Arizona didn't see an influx of millionaires in 2008. Far from it. In 2008, Las Vegas lost 38 percent of its HNWIs, and Phoenix lost 34 percent. Florida, which has no state income tax and hasn't been talking about one, was a killing field for the rich. The three major metro areas that lost more than 40 percent of millionaires in 2008 were all in no-income-tax Florida—Orlando (42 percent), Miami (42 percent), and Tampa (51 percent). The decline has nothing to do with taxes and everything to do with bursting asset bubbles. Florida, Vegas, and Phoenix are places whose economies were dominated by ultra-bubbly real estate markets and where lots of businesses owned by wealthy people are dependent on construction, tourism, and leisure—sectors that got seriously whacked in 2008.

Of course, there's evidence that some millionaires have moved out of high-tax states. Bernie Madoff, for example, recently left New York to take up residence in North Carolina.