It's reassuring to see that David Beaver, while not occupied with his duties as a mascot, has been applying the power of logic to foil would-be terrorists. I myself, in my secret life as a young Israeli mathematician, have applied the power of statistical pattern recognition develop something even more important, namely the "First ever PC voice analysis software that can detect Love!"
Here for the first time, I will reveal my method, a fundamental innovation that can be applied to nearly any problem. I've already revealed that this technique can determine with nearly 100% accuracy whether or not a passenger is planning to blow up an airplane, or whether or not someone who calls you on the phone is secretly in love with you. Believe it or not, my method can also be applied to determine whether a car's engine is about to break down, or whether a lottery ticket is going to win the grand prize or not.
While I can't claim the perfect results achievable using David's logical methods, I'm proud to be able to submit irrefutable proof that in detecting whether or not a Lotto 6/49 ticket will win the jackpot, my method will be correct exactly 99.999992849% of the time. For detecting airplane hijackers, the results are not quite so exact, but a maximum likelihoood extrapolation from the past three years of experience suggests a success rate of approximately 99.99999894%. This is much better than the mere 98% success rate claimed for the Nemesysco product in the news article that David cited. The details of my method are given, in full, below.
Seriously, Amir Liberman of Nemesysco is not a secret identity of mine, nor even any relation to me, as far as I know. And I have no idea whatsoever how he and his colleagues achieve whatever results their products are able to achieve, whether in detecting love or terrorism. That's the whole problem with these systems, as far as I'm concerned -- because there is no implementable explanation of what they're doing, it's impossible to evaluate the underlying science. Nemesysco's explanation of its "Layered Voice Analysis" technology on its website is hopelessly vague -- that's fair enough on a website for the general public, but there are no pointers to papers or technical explanations. Their statement that LVA uses "wide range spectrum analysis and micro-changes in the speech waveform itself (not micro tremors!)" is about as helpful as saying that a cancer treatment uses "chemical compounds and natural plant extracts (not laetrile!)". They do say that "[t]he LVA uses a patented and unique technology to detect 'Brain activity finger prints' using the voice as a 'medium' to the brain and analyzes the complete emotional structure of your subject". If the technology is patented, there must be some degree of disclosure, though past experience leaves me without a lot of hope of finding implementable details.
Although I don't know anything beyond guesswork about what the Nemesysco system is doing, I was quite serious about the performance of my detection algorithm. Its secret is a simple one, which I'm happy to share with you: just figure out the commonest outcome, and guess that way every time. Most lottery tickets lose -- this website explains that in the Lotto 6/49 game in particular, the odds of picking the right sequence of six numbers is 1/13,983,816 = 0.00000007151. Subtracting that quantity from 1 gives you the probability of a correct diagnosis if you guess "this ticket will not win the jackpot" on every occasion.
A similar line of reasoning applies to airplane hijacking by terrorists -- roughly 19 out of 1.8 billion U.S. domestic airline passengers over the past three years have been terrorists planning to hijack the plane they were boarding, so if you guess "this passenger is not a terrorist" every time, you'd have been right with probability 1 - (19/1,800,000,000). The future success rate for this algorithm is probably not enormously different -- there will continue to be about 600 million passengers boarding planes in the U.S. every year, and only a handful of hijackers.
You can now guess how to apply this method to the love analysis test. Here the technique does need to be personalized, and the success rate will vary with the individual, though results approaching 100% correct are likely in many cases...
Needless to say, I'm not recommending that anyone use this method for any purpose at all. The point is that you have to be careful about interpreting percentages that are cited as indications of performance levels. There's a branch of statistics called signal detection theory devoted to analyzing decision in the presence of uncertainty, and a basic understanding of its concepts -- like ROC curves and d' -- should be part of everyone's basic mathematical education.
I do need to disagree respectfully with one aspect of David's post. He writes:
The insurance companies have nothing to lose, in the sense that they start off with no sensible way of telling whether claims are fraudulent, or which parts of claims are fraudulent. But they do know that if the claimants think that the insurance companies have a way of telling truth from fiction, then fewer fraudulent claims will made or sustained. ... The insurance company has little interest, then, in whether Nemesysco's software really works. For what Nemesysco is really selling is is a great patter.
What we have here, I believe, is a technologically updated new release of that psychologically sophisticated version of the Pinocchio effect that evil parents have been using on their innocent children for generations, an effect whereby one lie spawns another, and all in the cause of establishing a norm of honesty:
Mom: Have you washed your hands.
Kid: Yes, Mom.
Mom: I can see your nose growing...
Based on interactions over the years with people in different areas of fraud prevention and investigation, including in the insurance industry, I believe that there is some truth in this, but it's by no means the whole story. Investigations are expensive, and it's important to be able to use them efficiently. You need to decide, somehow, which cases to examine at what level of scrutiny. Insurance companies do have methods for flagging suspicious claims, and I'm fairly confident that these methods have some genuine diagnostic value. If voice analysis were actually able to detect attempted fraud, even to a modest degree that would be useless as courtroom evidence, it might still be quite valuable as part of the process for deciding which cases to investigate further. This value doesn't depend on any deterrent effect -- it's gravy if things work out that way -- but it does require that the algorithms have some real ability to discriminate valid from fraudulent claims.
How the currently available commercial voice-analysis systems rate on this question is not known to me. But as I wrote before, I think it's appalling that this industry (i.e. proprietary methods for speech-based lie detection) continues, decade after decade, to market products in a way that would probably result in criminal charges for a pharmaceutical company. After all, it's potentially just as damaging to (fail to) identify people as law enforcement suspects or insurance fraudsters as it is to (fail to) diagnose them with diseases. And it would not be hard to evaluate the science underlying the technology -- whatever it is -- and to apply standard testing methods -- double blind, independent evaluation -- to determine how well the technology works in particular applications.
Posted by Mark Liberman at August 29, 2004 10:22 AM