Language Log

Baboons and Daubert

A recent New York Times article, How Baboons Think (Yes, Think), describes recent, interesting research by Dorothy Cheney and Robert Seyfarth on the Moremi baboons in Botswana. I won't comment on the language and cognition issues discussed there but I admit that I was very interested in the problem these wildlife biologists have with claims made by others-- that current ape studies can help us understand the evolution of communication:

Many linguists face similar problems. We gather data and experiment with it in our laboratories but many times we can't be sure that our results reflect what happens under more natural, non-laboratory conditions. For example, a great deal of the research on deceptive language is based on experiments in which people (usually undergrads) alternatively are asked to make truthful and untruthful statements for a stimulus that is then given to subjects to assess any deception. Not surprisingly, the subjects aren't very good at this. Another way to do this might be to use naturally occurring language data that might be taken from police interrogations or, in some cases courtroom testimony, that already  has been proved to be untrue, and then to produce experimental stimuli from this.

Linguists who study naturally occurring language are somewhat handicapped when their findings are compared with the much neater results produced by controlled experiments. One of the problems facing such linguists who serve as expert witnesses is the standard of admissibility that the US Supreme Court established in 1993 in Daubert v. Merrell Dow Pharmaceuticals, Inc., now commonly referred to as Daubert. Before this decision, the courts had relied on the 1923 decision of Frye v. United States, in which experts were selected using what is called "the general acceptance test," which specified that a proposed expert witness should be qualified in that field, that the testimony should be relevant to an issue in dispute in the case, and that the scientific theory or technique must be shown to be generally accepted by the relevant scientific community.

In most US jurisdictions today Daubert has replaced the Frye general acceptance test with a reliability assessment, an independent judicial evaluation of the reliability of the proposed testimony. The Court suggested four factors to assist judges:

The Court further noted that these criteria are not fixed or invariable but are meant to be helpful to judges, rather than definitive.

For a few years there was considerable confusion about the scope of Daubert. Did it apply only to scientific testing and analysis or did it apply to all forms of expert witness testimony, including groups such as engineers, automobile mechanics, and therapists? In 1999 the US Supreme Court ruled in Kumho Tire v. Carmichael that the Daubert reliability assessment applies to all forms of expert testimony, even to palmists and astrologists.

Generally speaking, linguistic analysis can satisfy the Daubert factors. The scientific nature and credentials of our field help meet factor 1. Publications and peer reviews help with factor 2. Factor 4 is not a problem because linguists get research funding from the National Science Foundation and other government agencies supporting our work. But factor 3 with its potential error rate can pose a serious problem for linguists who study language as it occurs in its natural contexts. Language evidence in the form of tape-recorded conversations is virtually impossible to replicate or convert into controlled experiments. Other social sciences have a similar problem, especially when they are not doing experimental studies.

I've worked on many such cases but the one that was most frustrating was a case in which there were two tape-recorded conversations in evidence. The prosecution claimed that one of them was a genuine conversation but the other one was a faked, staged conversation between the same two men. My task was to try to determine whether or not the allegedly faked conversation was actually staged.

Even linguists who specialize in conversation data haven't given this topic much thought. I couldn't find research studies on this topic so I couldn't tell the court that my findings had been subjected to previous peer reviews and publications (factor 2). Nor could I say that there was any known error rate for this kind of analysis (factor 3). I had compared the known, real conversation with the allegedly staged one and concluded that in terms of the phonology, morphology, syntax, discourse structure, speech errors, pauses, pause fillers, vocabulary, and other factors, there was no substantive difference between them. I couldn't testify that the speakers had or did not have the intention of producing a staged conversation, because that would go beyond the scope my field (or any other field, as far as I know). But I was prepared to testify that there was no linguistic evidence that would mark the two conversations as different. This didn't satisfy the judge, who from all I could tell may have been predisposed not to allow it anyway. She stuck rigidly to the Daubert factors and didn't allow me to testify.

When I read the complaint of Cheney and Seyfarth about the lack of research on chimpanzees in their wild, natural context, I couldn't help thinking that it parallels the Daubert factors that face some linguistic expert witnesses. Sometimes experimental research, for all its great values, can't tell us all that we really need to know.