According to Jim Holt's discussion in the New Yorker of Gerald Cohen's paper "Deeper into Bullshit", Cohen independently discovered what I have always privately called Labov's Test:
...how could one prove ... that a given statement is hopelessly unclear, and hence bullshit? One proposed test is to add a “not” to the statement and see if that makes any difference to its plausibility. If it doesn't, that statement is bullshit.
I described and exemplified Labov's Test in one of my first Language Log posts, though I attributed it to an anonymous "colleague" rather than to its author, Bill Labov. Since I was relying on my memory of a dinner-table anecdote from many years ago, I wasn't sure that I had the story exactly right. And Bill is a person who is very serious about quantitative validation of his empirical claims, so I thought he might be uneasy about being cited as author of an informal experiment without a large enough N or a double-blind design. And I was writing about whether or not there is any signal in Jacques Derrida's noise, so I felt constrained by contrast to be careful and exact.
But finding the truth depends as much on open presentation and discussion as on private research and reasoning; and this is a blog, not a refereed journal; and Labov's Test is a worthwhile rule of thumb, whose (co-)invention Bill deserves credit for. So I'm outing him as the author. I suppose we could call it the Labov-Cohen test. Some might object that Cohen actually wrote about it, while Labov's contribution was only through the oral tradition, but basic methodological innovations of this kind are often introduced by what scholars call "personal communication".
Anyhow, I have another purpose in bringing this up, which is to criticize Holt for carelessness in interpreting this useful test. (I'm not sure whether to bring Cohen into this or not, since I haven't yet read his paper “Deeper into Bullshit,” in Sarah Buss, Ed., Contours of Agency: Essays on Themes from Harry Frankfurt.)
A key thing about the test, missing from Holt's discussion, was front-and-center in Bill Labov's original presentation, as I recalled and described it:
This ... reminds me of a parlor game that a colleague of mine claims to have played, back in the day when it was easier to find academics who took Derrida seriously.
My colleague would open one of Derrida's works to a random page, pick a random sentence, write it down, and then (above or below it) write a variant in which positive and negative were interchanged, or a word or phrase was replaced with one of opposite meaning. He would then challenge the assembled Derrida partisans to guess which was the original and which was the variant. The point was that Derrida's admirers are generally unable to distinguish his pronouncements from their opposites at better than chance level, suggesting that the content is a sophisticated form of white noise. On this view, as Wolfgang Pauli once said of someone else, Derrida is "not even wrong.".
The point is that this is a test of communication from author to audience, not a test of the author's meaningfulness in itself. And it is framed as a behavioral test, not as a test of the author's intentions with respect to the relation between text and truth, or any other aspect of the author's state of mind. Labov's test could fail as easily because the audience is ignorant as because the text is nonsense.
For example, a 1999 JHEP paper by Seiberg and Whitten contains one of these two sentences, differing only in the introduction of "not" in the second one:
(1) ... at nonzero B (unless B is anti-self-dual) a configuration of a threebrane and a separated −1-brane is BPS, so an instanton on the threebrane cannot shrink to a point and escape.
(2) ... at nonzero B (unless B is anti-self-dual) a configuration of a threebrane and a separated −1-brane is not BPS, so an instanton on the threebrane cannot shrink to a point and escape.
Both sentences seem equally plausible to me. However, I don't take this as evidence that string theory is bullshit, but rather as evidence that I don't understand its mathematics. If I claimed to understand the mathematics of string theory as discussed in this paper, but was unable to pass Labov's test with respect to a set of pairs of sentences like this one, you'd be justified in concluding that I was bullshitting about understanding the paper, but not that Seiberg and Whitten were bullshitting in writing it. (By the way, the second sentence is the original one; and BPS is short for Bogomol'nyi, Prasad and Sommerfeld, and is discussed at greater length here, if that helps...)
(My interpretation of) Labov's claim about Derrida and similar writers is that all of his readers will fail the test (statistically speaking) all the time. If this were true, then we could conclude that everyone who claims to have understood Derrida (for example) is a bullshitter, or at least is in some sense deluded. This universal obscurity would certainly raise the suspicion that there was no suitable object of understanding available, for instance because the work is simply (or rather, complexly) nonsense.
I suspect that (my memory of) Bill's experimental hypothesis is often true, in the sense that in a controlled experiment, the partisans of such "theory" would fail to distinguish its statements from their negation at greater than chance level, in a large proportion of replications across statements and subjects. Of course, the ostentatious obscurity of such work suggests that its practitioners might be pleased rather than distressed by this result. And we'd need another test to distinguish this case from the similar results obtained for any sufficiently difficult mathematics. However, the point of my original post was that the test would not always be negative. Sometimes, Derrida was just wrong.
Posted by Mark Liberman at August 17, 2005 09:58 AM