Statisticians call that the base rate—how common something is within a broader class. Daniel Kahneman has a much more evocative visual term for it. He calls it the “outside view”—in contrast to the “inside view,” which is the specifics of the particular case.
False Positives
Rare signals drown in false positives.
← Back to trails
The conclusion is startling. I think that most forty-year-old women who have a positive mammogram would be astounded to learn that they still have less than a 1 percent chance of having breast cancer. Figure 3.3 might make the reason easier to understand: the tiny number of true positives (i.e., women with breast cancer) is overwhelmed by the number of false positives.
False positives are innocent emails that get mistakenly identified as spams. For most users, missing legitimate email is an order of magnitude worse than receiving spam, so a filter that yields false positives is like an acne cure that carries a risk of death to the patient.
The moral is that testing may cause more trouble than the product itself. Much product is condemned falsely in industry only because of measurement processes that give answers that do not agree with other answers.
But as bad as the false positives experience for innocent users and resulting negative publicity for the company might have been, it was an acceptable cost. The fact that spiraling fraud losses contributed to many of our competitors like eMoneyMail, PayMe, and PayPlace ceasing operation made this an easy choice—had PayPal not found a way to get fraud under control, it would have destroyed the company.
A possible exception would be software hooked up to sufficiently powerful actuators, such as software in early warning systems if connected directly to nuclear warheads or to human officers authorized to launch a nuclear strike. Malfunctions in such software can result in high-risk situations. This has happened at least twice within living memory. On November 9, 1979, a computer problem led NORAD (North American Aerospace Defense Command) to make a false report of an incoming full-scale Soviet attack on the United States. The USA made emergency retaliation preparations before data from early-warning radar systems showed that no attack had been launched (McLean and Stewart 1979). On September 26, 1983, the malfunctioning Soviet Oko nuclear early-warning system reported an incoming US missile strike. The report was correctly identified as a false alarm by the duty officer at the command center, Stanislav Petrov: a decision that has been credited with preventing thermonuclear war (Lebedev 2004).
As a rational trader (all traders boast so) I believe, as I discussed before, that there is a difference between noise and signal, and that noise needs to be ignored while a signal needs to be taken seriously. I use elementary (but robust) methods that allow me to calculate the expected noise and signal composition of any fluctuation in my trading performance. For example, after registering a profit of $100,000 on a given strategy, I may assign a 2% probability to the hypothesis of the strategy being profitable and 98% probability to the hypothesis that the performance may be the result of mere noise. A gain of $1 million on the other hand, certifies that the strategy is a profitable one, with a 99% probability. A rational person would act accordingly in the selection of strategies, and set his emotions in accordance with his results. Yet I have experienced leaps of joy over results that I knew were mere noise, and bouts of unhappiness over results that did not carry the slightest degree of statistical significance.
Doing it wrong is worse than doing nothing at all. When you know you’re clueless, you tend to be careful. But collecting a fistful of false positives is like convincing a drunk he’s sober: not an improvement.