Language Log

Secrets of the BBC sexes

I should have known that there would be a BBC science angle to this words-per-day business. David Beaver writes:

At http://www.bbc.co.uk/science/humanbody/sex there's a six part test to see whether you're more female or male (I'm... male). One of the later tests concerns verbal ability, and along with your result on this part of the test is the following text:

Did you know that, on average, women use 15,000 words a day while men use 7,000?

Reader, did you know that, on average, there are 23 different versions of this phony comparison published every day in the world's media? Well, that's not true either -- but I think this is the 11th different pair of values that I've come across recently, ranging from a high of 50,000 vs. 25,000 to a low of 5,000 vs. 2,500. (And in case you're coming late to this discussion, all of these numbers seem to have been made up or copied from someone who made them up -- the many studies that actually count words and correlate the counts with sex find no group difference, or a relatively small difference. When a difference is found, it's usually in the direction of more words from men.)

David reports that the BBC sex-test message continues:

Women took about twice as long as men to end their online instant messenging conversations in a 2003 study of US university students. The study, which was published in the Journal of Language and Social Psychology, also found that women were much more likely to use emoticons (representations of emotions using punctuation marks).

The most popular emoticon was the smiley face :- )

(He observes that "it really said 'messenging', a nice combination of 'messenger' and 'messaging'". This is a fairly common blend -- 432,000 ghits -- but it seems to be a sporadic error rather than an up-and-coming variant.)

The "2003 study of US university students" seems to be Naomi Baron, "See you online", Journal of Language and Social Psychology 23(4) 397-423, 2004. This paper describes its raw materials as follows:

An IM corpus was collected in April 2003 from 22 college-aged students who were attending school or had graduated the semester before the study was undertaken. [...]

The corpus consisted of 23 distinct IM conversations. A total of 9 conversations took place between females (FF) and 9 between males (MM). An additional 5 conversations involved female/male dyads (FM). In a number of the FF and FM conversations, a single student experimenter had conversations with several people on his or her Buddy List. Because several student experimenters withdrew from the project and could not be replaced, most of the MM conversations were between the same two interlocutors.

Taken collectively, the 23 IM conversations contained a total of 2,185 conversational turns, made up of 11,718 words.

The first ironic statistic of the day: the paper reporting on this corpus is 27 printed pages, comprising roughly 11,000 words, so that the paper is about the same size as the corpus it's based on.

The comparison of conversation length and ending length involved only the 9 FF and 9 MM conversations, with the numbers provided in the table below. The second ironic statistic of the day: since "most of the MM conversations were between the same two interlocutors", the BBC science journalists (or their advisors, if any) are making general statements about "women" and "men" based mostly on a sample of two male college students. OK, I admit, it's a step up from the cows in terms of sampling methodology.

Anyhow, here's the data:

Baron's footnote indicates that the difference in conversation length were "not significant":

Large variances associated with the means on turns per conversations (female to female [FF] M = 121.89, SD = 82.68; male to male [MM] M = 85.22, SD = 88.96) and on minutes per conversation (FF M = 31.33, SD = 25.57; MM M = 18.89, SD = 17.27) rendered no statistically significant differences between gender pairs.

But I have to say that this is beside the point. Even if the observed differences had been statistically "significant", they could not have been scientifically meaningful. The male part of the sample was mostly derived from conversations between just two males (and there were not a whole lot of females)! If you could prove beyond a shadow of a statistical doubt that those particular two male college students had shorter IM conversations than the dozen or so (geographically and socially uniform) female college students in the sample, then what?

The business about turns and time "to close" is about the semi-ritualized exchanges at the very end of conversations. Baron's example:

Gale: hey I gotta run
Sally: Okay.
Sally: I’ll ttyl?
Gale: gotta do errands.
Gale: yep!!
Sally: Okay.
Sally: :)
Gale: talk to you soon
Sally: Alrighty.

I guess it's plausible that American women generally spend more turns and time in closing conversations than American men do, whether in face-to-face interactions, in telephone conversations, or in IM exchanges. However, I don't know whether this is true or not, in fact, and Baron's article doesn't really help us get closer to an answer. It's true that in her corpus, there was a "statistically significant" difference in closings:

For instant message (IM) closings, however, FF/MM differences were significant both for number of turns per closing (FFM= 9.78,SD = 5.12;MMM= 4.29,SD = 1.13), t (14) = 2.77, p = .015; as well as seconds taken to close (FF M = 41.00, SD = 20.12; MM M = 16.29, SD = 15.48), t (14) = 2.68, p = .018.

And from this we can conclude that Baron's two college males used more abrupt IM closings -- at least in messaging each other -- than her dozen college females did. But generalizing from a sample of two guys to American males college students in general is... well, let's say we shouldn't do it. Using this method, we could easily prove a large number of pretty surprising things about any group you care to name.

OK, enough. I'll just point out that the BBC science writers use this (in my opinion meaningless) result in a highly misleading way. Consider again what they wrote:

Did you know that, on average, women use 15,000 words a day while men use 7,000? Women took about twice as long as men to end their online instant messenging conversations in a 2003 study of US university students. The study, which was published in the Journal of Language and Social Psychology, also found that women were much more likely to use emoticons (representations of emotions using punctuation marks).

I believe that most readers will interpret that passage to mean that women's IM conversations were found to be twice as long as men's. But the fact of the cited study, you'll recall, was that there was no significant difference in conversation length. The only difference was in the length of closings -- and the data, again, came mostly from two men.

Science journalism is pretty bad overall, but I wonder, is there is any corner of it that is worse than reporting on sex differences? Perhaps reporting on animal communication is also in the running for the booby prize.

[Update -- one good result of the sad Foley scandal (aside from a marginal reduction in the proportion of hypocrites on Capitol Hill) will apparently be a doubling (or more) of Baron's sample size...]

Posted by Mark Liberman at September 29, 2006 07:57 AM