Language Log

A tale of two media

You've probably read about or heard about Glenn Wilson's study allegedly showing that email lowers IQ more than marijuana does, and you probably even remember the amount of alleged damage (10 points) and the alleged explanation (the cognitive wear and tear of jumping around among many topics developing in parallel). Now you have a chance to read Steven Johnson's argument in the NYT Sunday Magazine that "Watching TV Makes You Smarter", and evaluate his proposed explanation:

Think of the cognitive benefits conventionally ascribed to reading: attention, patience, retention, the parsing of narrative threads. Over the last half-century, programming on TV has increased the demands it places on precisely these mental faculties. This growing complexity involves three primary elements: multiple threading, flashing arrows and social networks.

In other words, for Johnson, jumping around among many topics developing in parallel is a "cognitive benefit".

It's certainly possible that dealing with the multiple email threads linking your own social network makes you stupider, while dealing with the multiple TV-show threads linking Tony Soprano's social network makes you smarter. On the other hand, a more parsimonious explanation is available: both Glenn Wilson and Steven Johnson are blowing smoke.

Let's look at the evidence.

You probably don't remember anything about Glenn Wilson's evidence for the effects of email on IQ, except that "a study showed" it, because he hasn't presented any. Not even a sketch of how the experiment was done has appeared in any of the stories that I've read, and searching several databases of scientific information leads me to conclude that no details have so far been published anywhere at all, not even in the most obscure psychometric journal. It's possible that no details will ever be published, because this was apparently part of a study privately commissioned by HP, and its author has a history of hopping around from topic to topic himself: political psychodynamics, sex differences, love addiction, and the psychology of bubble baths, among other things.

As I suggested in the blog entry linked above, there could be lots of confounding factors in an experiment on this topic. In fact, it's not easy to see how to design an experiment on the cognitive effects of email that doesn't have serious confounds. But instead of a series of carefully-documented and well-controlled experiments, we've got a single small experiment, documented only by a press release that doesn't even sketch the experimental design, and carried out by a psychiatrist whose previous work strikes me as higher in topicality than in scientific rigor.

What about Steven Johnson's evidence that TV makes you smarter? Well, Johnson is a writer of popular science books -- his last book was a tour of neuroscience called Mind Wide Open, and the NYT piece is adapted from his forthcoming book ''Everything Bad Is Good for You: How Today's Popular Culture Is Actually Making Us Smarter.'' So he's wearing his bias on his sleeve, so to speak -- we can assume that he's looking for a good story that will sell books, not seeking the truth in a careful and dispassionate way.

Still, in contrast to Wilson's press release on email and IQ, which was basically just a guy in a metaphorical white coat pushing the media's buttons, Johnson actually presents evidence and makes an argument. His evidence and arguments are all about developments in modern culture, specifically the design of TV shows, which have been getting more complicated in specific ways that he describes. His argument about the psychological effects of these cultural changes is a pretty weak one -- basically, he just asserts that more complicated experiences must make you smarter than simpler ones do. You could make the same argument about email.

Nevertheless, Johnson does actually present some supporting evidence. The part I liked best was the way he represents the plot of TV shows, as a sort of checkerboard graph in which "the vertical axis represents the number of individual threads, and the horizontal axis is time." Here's his graph of an episode of The Sopranos:

I'm not sure whether this sort of plot graph ("plot plot"?) is Johnson's invention -- he doesn't credit it to anyone else -- but I haven't seen it used before. I'd think that graphs like this would be a natural starting point for critical study of story-telling techniques, and it's easy to think of all sorts of interesting measures to derive from them. They should apply well to TV dramas like The Sopranos, where I imagine that the writers have such displays in mind as they plan episodes and seasons, and maybe even use something like them explicitly. The notion of "thread" may be harder to define for the plots of some other genres, where more of the structure is in the evolution of individual narrative strands than in the way the strands are woven together.

Anyhow, Johnson's plot plots impressed me, but his use of them didn't. He supports his generalizations with examples, without demonstrating (other than by assertion) that the examples are typical; some of the crucial cases are what we might call "generic examples", unsupported claims about typical examples of a type; and it turns out that some of the crucial aspects of his examples are not actually exemplified in the specific cases that he presents. This is normal and reasonable for journalism, but Johnson is presenting an original argument, not reporting on someone else's scholarship.

He asserts that the complexity of TV dramas has developed in four stages, of which The Sopranos is the culmination. He describes the first two stages this way:

Draw an outline of the narrative threads in almost every ''Dragnet'' episode, and it will be a single line: from the initial crime scene, through the investigation, to the eventual cracking of the case. A typical ''Starsky and Hutch'' episode offers only the slightest variation on this linear formula: the introduction of a comic subplot that usually appears only at the tail ends of the episode ...

and he depicts the "typical" episode of Starsky and Hutch like this:

The next stage is represented by the plot of a specific Hill Street Blues episode:

I'm sure that Johnson is describing a real trend, but it bothers me that the first two stages in his claimed evolution are not supported by any specific facts at all, and the change from Hill Street to The Sopranos turns out not really to be about the number of parallel plot threads at all, but about some more subtle developments:

The total number of active threads [in "The Sopranos"] equals the multiple plots of ''Hill Street,'' but here each thread is more substantial. The show doesn't offer a clear distinction between dominant and minor plots; each story line carries its weight in the mix. The episode also displays a chordal mode of storytelling entirely absent from ''Hill Street'': a single scene in ''The Sopranos'' will often connect to three different threads at the same time, layering one plot atop another. And every single thread in this ''Sopranos'' episode builds on events from previous episodes and continues on through the rest of the season and beyond.

Again, I'm sure that there's some truth here, but I can remember plenty of examples of "chordal storytelling" and cross-episode continuity in Hill Street Blues. And I'm sure that Dragnet and Starsky and Hutch had a very different plot layout from current shows, but it'd be nice to see at least one specific example rather than an assertion about what is typical. For all four stages, it'd be even better to see an argument based on analysis of a reasonable sample of shows. Overall, this is the kind of argument from assertion that often establishes as conventional wisdom a proposition that turns out to be completely false when someone finally gets around to checking it.

I was going to start the conclusion by writing "If Johnson were a scientist...", but that's misleading. This is not about science vs. the humanities, or even about good science vs. bad science. It's about rational investigation.

Everyone these days seems to believe that modern life is complex, fragmented and disjointed, in comparison to the life of the past. And most people have some kind of opinion about the effects on our attention span, our intelligence, our culture, our politics, our morals. Given how important this is, you'd think that people would look at it in a serious way, rather than just marshalling stereotypes and counter-stereotypes in rhetorical parade.

It's unfair for me to complain so much about Steven Johnson's article. He makes a clear and interesting argument about the evolution of modern mass culture, leading to a contrarian conclusion, and he supports it with a large number of interesting examples. I wish that more contemporary literary critics did this sort of thing as well as he does.

But why didn't Johnson consider doing this kind of investigation with some scholarly (or scientific) care? Alternatively, why hasn't someone else done this, so that Johnson could base his popular book and articles on a solid foundation of fact rather than a flimsy scaffolding of anecdote and rhetoric? Oh, I know, it's because the fragmentary and disjointed nature of modern life has left us without the attention span required by scholarship and science. Or wait, I guess it's actually because the experience of modern complexity has made us smart enough to transcend the plodding path of scholarship, leaping to valid conclusions in a cyberintuitive blink. One or the other, anyhow: whatever.

[Update 9/25/2005: for the truth about the experimental design of the "email lowers IQ" studies, and an apology for blaming the media's excesses on Glen Wilson, see this post.]

Posted by Mark Liberman at April 30, 2005 09:39 AM