A spider on your face won't stop me from criticizing this article

So there's this story about how your fear response may explain your politics:

The results seem to suggest that our ideas about the world are shaped by deep, involuntary reactions to the things we see. As evidence, the study found that greater sensitivity to the images was linked to more fervent support for a conservative agendaincluding opposition to immigration, gun control, gay marriage, abortion rights and pacifism, and support for military spending, warrantless searches, the Iraq War, school prayer and the truth of the Bible. In other words, on the level of physiological reactions in the conservative mind, illegal immigrants may = spiders = gay marriages = maggot-filled wounds = abortion rights = bloodied faces.

That's the Newsweek version; other media outlets are running with similar accounts. This has all the hallmarks of junk science -- the verrry, verrry conveeeenient direction of the association (conservatives are "more fearful"), the synergistic connection to an imminent public event, the vivid imagery ("spider on face", "maggots in open wound").

Maybe it's my hyper-reactive fear response talking, but I'm primed to be skeptical about this sort of thing. I'm especially bothered by the idea that it's somehow wrong to suggest that genetic variance underlies a behavioral trait like IQ, but perfectly fine to suggest that it underlies the voting tendencies of conservatives in Nebraska.

As you might guess, I think we should be skeptical of everything, skepticism being a scientist's day job. It's not that much work, or really very difficult at all, to work through the methods and statistics here, even if you're not a psychologist.

In a nutshell, here's what we have:

  • The authors called up 1310 people in Lincoln, Nebraska. 143 of these people had strong political attitudes and were willing to come in for further testing. Ultimately 46 of them completed the testing.
  • They constructed a dependent variable, which they call "Support for Protective Policies", by adding up responses to 18 questions about different social attitudes. These included:
    Military spending Warrantless Searches Death Penalty
    Patriot Act Obedience Patriotism
    Iraq War School Prayer Biblical Truth
    Pacifism Illegal Immigration Gun Control
    Foreign Aid Compromise Pre-marital Sex
    Gay Marriage Abortion Rights Pornography
  • They did physiological tests, including measuring skin conductance in response to ugly pictures (spider on face, maggots) and blinking in response to loud noises.
  • They ran three multiple regression models, trying to predict the dependent variable (Support for Protective Policies) as a function of several independent variables, including their physiological tests as well as sex, income, age and education.

The authors found a statistically significant association between their dependent variable ("Support for Protective Policies") and the physiological responses. Hence, they find statistical support for a connection between them. That says nothing at all about genetics -- we are left only with a plausible suggestion that the physiological responses have a genetic basis, and previous work that suggests that political attitudes have nonzero heritability.

Now, in general, I think that many behaviors have some genetic variation underlying them. The earlier twin studies by Alford's group demonstrate that "political behavior" has a nonzero genetic component. It is plausible (but not necessary) that this genetic component might have other observable physiological effects. It's even plausible (but not necessary) that the fear or stress responses might be connected to the larger-scale behavioral patterns. And I agree pretty much with Alford's attitude, as expressed in that Newsweek interview:

I don't think that biology is destiny, but for the general public, I want people to believe that it's something. Right now it's seen as nothing. It's given zero weight.

So in short, I'm willing to believe the premises of the study. And yet, when I read the widespread uncritical acceptance of this story, I have to say, "Hold on a second, here!"

Some commenters have tried to atomize the responses examined in the study. For instance, why are "gun control" and "pornography control" on different sides of the "socially protective" index? And what does "Biblical literalism" have to do with it?

But those kinds of questions are really going in the wrong direction. You can't break apart this result; there are simply too few observations. What you have to do is examine whether the assumptions justify the conclusions.

So, here is a list of possible criticisms of assumptions or methods that I would have noted if I were reviewing this paper.

  1. Sampling: By going from a sample of over 1310 down to 46 individuals, the authors were in a position to cull the sample in many ways that might influence their outcome. They report that each step of reduction of the sample happened by clear objective (or incidental) criteria. I have no reason to doubt them. But a skeptic should consider that factors like age, education, sex, income, marital status, number of children, and frequency of religious observance might well influence a subject's willingness to participate in experiments, and subtle efforts to correct for such a bias may themselves induce biases. The last three factors were not controlled in the study; a clever person might well think of others.
  2. Presentation of results: The methods employed are multiple regressions, using the dependent as a continuous variable. Yet the graphs in the paper show the dependent as a dichotomous variable. The graphs are designed to show the result to be as visually large as possible, but in so doing, they don't report the actual results, which are visually much weaker (while still statistically significant).
  3. The missing range: In a small sample like this (20 or so in either category) it is normal to show the range of responses in the data. It would also be normal to show plots of the dependent variable against other independent variables (such as income or education), at least in the supplementary information. In a sample like this, a small number of individuals with extreme responses will affect the mean as well as the standard error (both shown), in ways that can easily be detected on a plot of the range or a plot of the data themselves.
  4. Priming: So a bunch of conservatives march down to the university to be interviewed by academics about their political attitudes. I wonder if that might have some different effect on their stress levels than the political liberals who took the same trip and answered the same questions? The non-threatening baseline pictures are intended to correct for such biases, but do they in this case?
  5. The not-so-missing moderates: The distribution of "Support for Protective Policies" in the sample is highly bimodal, as the authors intended. But it is not symmetrical. There actually are "moderates" in the sample, at least by their criterion of intermediate score on their index. Most of these moderates would seem to have been lumped in with the "conservatives." So we're not looking at a comparison of "conservative" and "liberal," we're looking to some extent at "liberal" versus "everyone else." This makes little difference to the analysis, but a great big difference to how we should interpret it!
  6. Dependent variable: "Support for Protective Policies" is clearly a made-up category. Different sets of policies corresponded to "liberal" and "conservative" viewpoints 50 and 100 years ago, and do so in other countries. Why did the authors not select a simpler dependent variable? This has the look of a fishing expedition.
  7. Religious attitudes: The sample is divided most evenly by these social attitudes: "Biblical Truth," "School Prayer," "Pacifism," "Iraq War," "Gun Control," "Pre-marital Sex," "Gay Marriage," "Abortion Rights," and "Pornography." Ummm...if you subtract out the ones involving guns, isn't there an obvious underlying variable that explains most of these? Do you think the authors should have controlled it? Do you think a reviewer of the study ought to have pointed this out? I mean we're talking about 46 people here, and a shared uncontrolled variable among just five or six of them would entirely determine a significant outcome. Just asking...

What do these criticisms mean?

None of them exclude the hypothesis that genetic variation explains the behavioral associations. Again, the twin studies are enough to show that genes are relevant (but not sufficient) for predicting political attitudes. And to be perfectly clear, the study does not claim to explain the mechanism underlying the "relationship" between political attitudes and these physiological responses. It just claims that the relationship exists and supports that with its sample of individuals. I think the argument is insufficient (why didn't they include religious affiliation, again?) but we should keep in mind (as usual) that the paper, however flawed, is not making an extravagant claim.

But the criticisms raise questions about the specific attitudes that are supposed to be explained, and how those attitudes are "bundled" into "conservative" and "liberal" camps in the U.S. For example, it seems plausible to me that social constraints entirely determine the package of political attitudes that comprise "conservative" and "liberal" (and we may add "libertarian", "blue-dog" and many others), while people affiliate with those positions for individual reasons (such as religious affiliation, personal experience, and income level) that are themselves influenced by genetic variation.

And the criticisms raise reasonable questions about the power of the methods to detect an association. "Statistically significant" is not a magical charm to drive away questions about sampling and methods. I may be unusually sensitive, having just read that biography of Vavilov, but calling people down to a psychological laboratory to discuss their political beliefs looks, well, a lot like an interrogation. No matter how friendly it is, people of different political affiliations might well react differently.

More of a skeptic might well go farther. How do we know, for example, that the responses to the pictures and sounds are constant across different parts of the U.S.? There are clear urban-rural differences in political behavior. Are we to imagine that the FDR Democrats who used to live in the country all moved to the big city and left their spider and maggot-fearing relations behind in the sticks?

If I were publishing a paper that claimed a genetic basis for political behavior, scheduled to come out in a major journal around six weeks before a national election, I would make darned sure I got these details right!

Of course, if I were in the press, writing a story about this research, I would make darned sure I asked these kinds of questions, too...

(Elsewhere: Razib suggests that a more complicated model of causation should be assumed, and I agree, assuming there's any validity to the results.)

References:

Oxley DR, Smith KB, Alford JR, Hibbing MV, Miller JL, Scalora M, Hatemi PK, Hibbing JR. 2008. Political attitudes vary with physiological traits. Science 321:1667-1670. doi:10.1126/science.1157627