Language Log

Snap judgments of competence and the rhetoric of statistics

This morning, Andrew Sullivan linked to a news report about a study said to show that "[a] split-second glance at two candidates' faces is often enough to determine which one will win an election".

Sullivan's comment: "Maybe everything else is just make-work".

Actually, no.

The paper is Charles C. Ballew II and Alexander Todorov, "Predicting political elections from rapid and unreflective face judgments", PNAS published online 10/24/2007.

In their experiments, snap judgments of competence from facial appearance accounted for between 2% and 14% of the variance of vote share, wherefore "everything else" accounted for between 86% and 98%. These different outcomes represent the results of various different experimental techniques -- different lengths of picture presentation, presence or absence of response deadlines, etc. -- applied to different collections of gubernatorial and senatorial races.

To get a graphical sense of what this amount of prediction is like, here's a scatter plot, charting the proportion of subjects who judged a candidate as more competent-looking than his or her opponent, against that candidate's share of the two-party vote, for their largest experiment (89 gubernatorial races) and the best-performing experimental condition (250 msec exposure to the pictures, r²=0.053, i.e. 5.3% of variance accounted for):

SI Fig. 5. Scatter plot of the two-party vote share for the candidates and their perceived competence (Experiment 1). Each point represents a gubernatorial race. The line represents the best fitting linear curve.

Now, these were two-party races, so you should expect to predict the outcome from the flip of a fair coin 50% of the time, leaving 50% to be predicted from facts about the candidates and the voters. In their experiment 1, predicting 89 gubernatorial races, subjects' snap judgments of competence in their best-performing condition were able to predict the winner 68.5% of the time (the other conditions were right 59.6% and 62.9%). That decreased the chance error rate by (68.5-50)/50 = 37%, which is certainly a considerable help in prediction, almost half as good as the (huge) effect of incumbency for House races. The results in their experiment 3 (35 2006 governor's races) were similar: 68.6% prediction of the winner from subjects' aggregate snap judgments of competence.

So there's definitely something going on there. But it's a small effect on vote share (accounting for less than 10% of the variance), whose leverage as a predictor of election winners is perhaps made larger by the fact that most races are fairly close, so that swinging a small number of votes can sometimes determine the outcome. This is a long way from justifying the conclusion that people's votes are in general determined (or even very reliably predicted) by their first impression of a candidate's appearance, or the conclusion that everything in politics except what a candidate looks like is "just make-work".

I like Andrew Sullivan's writing a lot -- and his reaction in this case was surely a wry joke -- but it would be a better world if influential people would read and understand the primary sources in cases like this one, rather than relying on the (reliably unreliable) press releases and mass media to tell them what such "studies" and "research" mean. I guess it would be almost as good if the journalists who act as front-line interpreters did a better job.

Of course, this would require everyone involved to have basic statistical literacy, which in this case means understanding what "percent of variance accounted for" means. And people would have to get used to evaluating (reports of) scientific results with the same skeptical care as policy recommendations.

This story is on the wires now, and will be hitting the papers over the next few days, so you can judge for yourself how helpful and accurate the media's interpretations are, and what the uptake from political commentators is like. My own guess is that we'll see plenty more evidence of the kind of pop platonism that characterizes most people's way of thinking about quantitative properties of groups.

[You might wonder how incumbency and snap judgments of appearance interact. Funny you should ask: Ballew and Todorov devote a section of their paper to showing that in these experiments, the effect of snap competence judgments is independent of incumbency. But I was very interested in this note from their "supplementary material":

Incumbency Status and Competence Judgments. Although we showed that the effect of competence judgments was independent of incumbency status for Senate races in our prior work (3), this was not the case for the House races. For these races, competence judgments predicted the winner only in races in which the incumbents won. There are a number of differences between House and Senate races and it is not clear how to interpret the latter finding. There is less media exposure to House candidates than to Senate candidates, and it is likely that many voters are unfamiliar with the faces of their House candidates, a possibility that suggests different accounts of voting decisions in House and Senate races. It was also impossible to obtain pictures of both candidates for all House races and this may have introduced unknown biases in the sample of these races.

Or maybe better campaign organizations hire better photographers and hairdressers, and pick better head shots from the available possibilities?]

[Update -- Andrew Gelman at Statistical Modeling, Causal Inference, and Social Science blogged about an earlier Todorov paper back in April ("Baby-faced politicians lose", 4/27/2007). He counsels care in interpreting such results:

You have to be careful in interpreting the results, however. Todorov et al. seem to be saying that individual voters' visual "inferences of competence" are affecting votes. Another story, perhaps more plausible, is that the more competent-looking people are the ones who rose to political success.

My main point here is that no matter which way(s) the causal arrows point, the overall effect is a fairly small one, which is a very long way from licensing the belief that that vote share is entirely determined by appearance.]

Posted by Mark Liberman at October 26, 2007 02:03 PM