Their "key findings": (1) Northeastern states, generally liberal... oh, and also Utah, which ISN'T... have better upward mobility; and (2) ex-Confederate states either have worse upward mobility or more downward mobility.
But their determination of this seemed, well, a bit glib. I wanted to get into the guts of this study and see what it really means.
Details:
(1) The study focused on baby-boomers and slightly before- people born from 1943 to 1958.
(2) The study covered the ten year period from 1998 to 2007. (This means the numbers include the peak of the dot-com boom, the dot-com/9-11 bust, but doesn't include the economic meltdown of 2008.)
(3) The study used US Census Department and other anonymous sources and statistical analysis; it did NOT follow specific individuals over a ten-year period.
(4) The absolute percentage of income growth DOES include inflation; the 2007 income is compared to what the similar 1998 income would be in 2007 dollars. (This is a good thing, since according to data I looked up at US Inflation Calculator, cumulative annual inflation means prices went up by about 29% from 1998 to 2007- and NO state in the Pew study showed an absolute income growth of 29%, or even close.)
(5) "Relative income mobility" is measured twice.
(5a) Positive mobility is measured by tracking people in the bottom half of all earners and counting those who moved up by ten percentile points. (For example, someone who makes more than 35% of Americans moving up to making more than 47% of Americans counts; someone only making more than 8% of all Americans moving up to 15% does NOT. Nor does someone moving from 51% to 81%.
(5b) Negative mobility is measured by tracking people in the top half of all earners and counting those who moved down by ten percentile points. That means someone dropping from lower-middle-class (say, the 40% mark) into poverty (16%) doesn't count, but someone dropping from 90% to 75% does.
In both cases, the ten percentile movement minimum is meant to eliminate statistical noise- small movements that don't show actual long-term trends.
(6) Only people who had positive income for all ten years were counted. Take a loss one year for tax purposes, or hit a rough patch in life's highway, and Pew didn't count you.
(7) Some states had sample sizes so small as to be statistically questionable, and nine small enough that they had to be lumped together into supergroups to return any meaningful conclusions at all. Thus, Maine and Vermont are considered interchangable; Alaska is considered the same as Idaho, Montana and Wyoming; and Iowa, apparently, is an extension of North and South Dakota (or vice versa).
These last three items give me SEVERE misgivings about this study. It seems to me that a lot is being ignored for the sake of being able to give simple, happy, pat answers at the end of the study. If any statistician or student of demographics reads this, I'd appreciate their views on how Pew handled this. Myself, I'm not all that impressed.
But still I am curious as to the details. So- I looked at their actual data sheet rather than the interactive map linked to above to see what their actual numbers were.
The national averages to compare each individual state to:
ABSOLUTE MOBILITY: 17% (the mean American earner saw earnings grow by 17% after inflation)
RELATIVE UPWARD MOBILITY: 34% (of the bottom half of earners, 34% rose ten percentile points or more in the absolute ranking of American earners)
RELATIVE DOWNWARD MOBILITY: 28% (of the top half of earners, 28% dropped ten percentile points or more in the absolute ranking of American earners)
And already there are all sorts of unanswered questions here. Why don't #2 and #3 equal out? How many of the top half got richer than their peers, and how many of the bottom half sank deeper? There really isn't much meat in this study, is there?
But to persevere. Texas was ranked as one of the "loser" states because, according to their definitions and statistical analysis, it had worse-than-national-average results in two of the three stats. Oh so?
Here's a direct comparison:
ABSOLUTE MOBILITY: Texas 15%, national avg. 17%, marked FAIL (interesting: Arizona also shows 15%, but is NOT marked fail. Pew's excuse is that they didn't mark as above or below average states with small sample sizes unless the difference was BIG, due to margin of error calculations.)
R. UPWARD MOBILITY: Texas 31%, national avg. 34%, marked FAIL
R. DOWNWARD MOBILITY: Texas 30%, national avg. 28%, marked AVERAGE (apparently 2% is within margin of error on this, but not on Absolute Mobility)
And that's just my home state. The more I look at the results from the other states, the more they seem all over the map, thanks in no small part to Pew's margin-of-error issues. Well, to hell with their margin of error issues- let's ignore that and go by the actual numbers they used.
If we go by a straight 3% deviancy threshhold- that is, any measure 3% off from the national average counts as "above" or "below" average, then here are the states that come out very good (all three above average), good (2/3 above), bad (2/3 below) or very bad (all below). Boldface indicates states Pew didn't list in their key findings. Italics indicate a state Pew listed which, by a 3% threshhold, this method eliminates.
VERY GOOD: Maryland, New Jersey, New York
GOOD: Oregon, Michigan, Delaware, Pennsylvania, Connecticut, Massachusetts, Rhode Island, Kansas, Utah
BAD: Missouri, Nebraska, Arkansas, Florida, Kentucky, Mississippi, North Carolina, New Mexico, Texas (just out)
VERY BAD: District of Columbia, Alabama (promoted from Bad), Louisiana, South Carolina, West Virginia, Oklahoma
So, yes, there's still a big clump of bad in the Confederacy and a big clump of good in the Northeast and Mid-Atlantic. But what do Utah and Kansas have in common with the Northeast? What does Oregon have in common with either group? And what does New Mexico, or the national capitol, share with the old South that makes them crappy?
As an experiment, let's try adding the categories of Good-ish and Bad-ish to see how things develop. Good-ish is 1 above-average stat and no below-average stats; Bad-ish is the opposite.
GOOD-ISH: California, Michigan
BAD-ISH: Nevada, Maine/Vermont, Alaska/Idaho/Montana/Wyoming, Florida, Georgia, Tennessee, Texas
No, the closest that comes to resolving anything is that it sucks to be in the old South. Not all red states suck, though- the Dakotas are OK (unless Pew's stats have Iowa bailing them out in that stupid supergroup), as is Arizona. New England apparently isn't all roses either (is Maine dragging Vermont down in THAT supergroup, or do they both kind of suck equally?).
For your own graphic aid, here's a quick and dirty flood-fill map I made of the states: dark green = very good, blue-green is good-ish, yellow is bad-ish, dark red is very bad.
So, what does all of this tell me? It tells me that this study is useless if you intend to use it for any advocacy position. The methodolgy and data analysis can be questioned half a dozen ways by a layman- and, I'm sure, much more thoroughly by experts. It just so happened that, if you looked at the stats in one particular way, it showed a couple of patterns. Look at the stats more deeply, though, and you see two things: (1) the patterns go away, except for a bias against the old South; and (2) the stats look VERY shaky for the purpose of comparing states, when there are at least fifteen states whose stats are too weak to compare reliably on a one-on-one basis to the rest.
If you were planning on using this study to say things like, "Blue states GOOD! Red states BA-A-A-AD!" please don't. There's probably good evidence for the assertion... but this isn't good evidence for much of ANYTHING. Don't discredit your cause with junk numbers; that's what conservatives do.
1 comment:
As for grouping states: if their demographics are similar, that's actually better (statistically speaking) than reporting results for a state with a sample size too small to be valid, especially if they used a chi-square test as one of their analysis methods. Of course, getting a larger sample size would be even better, but in a state like Montana or Alaska, the population may be too low for that to be workable.
My immediate questions (being too lazy at the moment to read the initial report) would be:
a) Did they control for race as a compounding variable? If not, then it seems likely that states with relatively high African-American and Native American populations would score poorly, as social mobility in those groups is known to be lower than for Asian- and Euro-Americans. (Last I checked, the data for Latino/as was in flux.) That might explain Dixie's relatively poor showing right there.
b) Possibly hard to disentangle from the previous question: Did they control for proportion of the population who had done prison time? Almost the only way to achieve upward social mobility as an ex-felon is to start your own business, as you're practically unhireable - and that requires both great skill and great luck.
Post a Comment