Picture a letter from a screening program. Your test came back positive, and the test is described as 99% accurate. Most of us would read that as a 99% chance of having cancer. It is often far from the truth.
In July 1994, the statisticians Douglas Altman and Martin Bland explained why in a one-page note in the BMJ [1]. “The whole point of a diagnostic test is to use it to make a diagnosis,” they wrote. What matters is how often the test gets it right for the person holding the result.
In my view, a positive screening result means you should take the next step, but it does not mean you have the disease. A test labeled 99% accurate tells you less than it seems, because what the result means also depends on how common the disease is among the people being tested.
What can accuracy mean?
The word accurate can mean at least three different numbers. Altman and Bland defined the first two in an earlier note from June 1994 [2]. Sensitivity is the share of people with the disease whom the test correctly flags. Specificity is the share of people without the disease whom the test correctly clears.
Their example was a liver scan checked against a firm diagnosis in 344 patients. Of the 258 patients with liver disease, the scan flagged 231, a sensitivity of 90%. Of the 86 patients without it, the scan cleared 54, a specificity of 63%.
A third meaning is overall accuracy: the share of all results that are right. Think of a disease that affects 1 in 1,000 people. A fake test that tells everyone they are healthy would be right 99.9% of the time, and it would never find a single case.
None of these numbers answers the question a patient actually has. As Altman and Bland put it, “In clinical practice, however, the test result is all that is known”. You know your result, and you want to know what it means.
The two numbers a patient needs
The numbers that answer a patient’s question have their own names [1]. Positive predictive value is the share of people with a positive result who really have the disease. Negative predictive value is the share of people with a negative result who really do not.
For the liver scan, the positive predictive value was 88%: 231 of the 263 patients with an abnormal scan really had liver disease. Of the 81 patients with a normal scan, 54 really were free of it. By my arithmetic, that is a negative predictive value of about 67%, or 2 in 3.
These two numbers are not fixed features of a test. They depend on how common the disease is in the group being tested, which doctors call prevalence. In the liver study, 3 out of 4 patients had disease. Altman and Bland worked out that if only 1 in 4 did, the same scan’s positive predictive value would fall to 45%. Its negative predictive value would rise to 95%.
A made-up example: screening 100,000 people
The numbers below are invented to show the arithmetic, not taken from any real test. Suppose we screen 100,000 people for a cancer that 1 in 1,000 of them have. That means 100 people have the cancer and 99,900 do not.
Now give the test 99% sensitivity and 99% specificity. Of the 100 people with cancer, the test flags 99 and misses 1. The problem comes from the healthy group. The test is wrong about 1% of them, and 1% of 99,900 is 999 people.
So the test gives 99 true positives plus 999 false positives. That is 1,098 positive results in all, and only 99 are real. The positive predictive value is about 9%, roughly 1 in 11. About 10 of every 11 people who get a positive letter are healthy.
The negative results are much more reliable. Of the 98,902 negative results, only 1 is wrong. The overall accuracy is exactly 99%, just as the label promised. The label was true, but it did not tell people what a positive result means.
Imagine a hall holding everyone in this example whose test came back positive, about 1,100 people. Fewer than 100 of them have cancer. The other thousand or so are healthy people waiting, worried, for their next appointment.
Nothing is wrong with the test. But a 1% error rate among a very large number of healthy people still adds up to many false alarms.
Why does a rare disease mean more false alarms?
Now run the same made-up test on a group where 1 in 100 people have the cancer. Out of 100,000 people, 1,000 are sick and 99,000 are not. The test flags 990 of the sick people. It also flags 1% of the healthy ones, which is another 990 people. Now half of the positive results are real.
The test did not change. Only the group being tested changed. Altman and Bland wrote that when a disease is very rare, even a very good test cannot make a positive result close to certain [1]. They concluded that in screening the general population, “it is inevitable that many people with positive test results will be false positives.”
They also noted the opposite effect for negative results. The rarer a condition is, the more you can trust a negative result, and the less you can trust a positive one.
In fairness, false alarms can be a cost worth paying. A screening test may be set up to accept more false alarms so that it misses fewer cancers. A missed cancer can cost a life, while a false alarm often costs a follow-up test and some worry. I think that trade is reasonable if people are told about it honestly. I work on AI for breast cancer imaging, and the same trade-off appears there.
What should you ask after a positive screening result?
A positive screening result means you need a closer look. It does not mean you have cancer. A few questions can help:
- Ask how many people with a positive result on this test turn out to have cancer.
- Ask how common this cancer is in people of your age and risk.
- Ask what the next test is, and how much more certain it will make the answer.
The first question matters most, because it asks for the positive predictive value in plain words. If most positive results are false alarms, that is useful news. It means the chance that you have cancer is still small.
A test’s accuracy describes the test. What you need to know is what your own result means, and that depends on how rare the disease is. In my view, a positive screening result means you need more tests. It does not give you the final answer.
When you see a claim like this, start with how many people have the disease. Then count the false alarms among everyone else. After a few tries, a headline about a 99% accurate test will no longer sound like a promise. It will make you ask how often a positive result is real.
References
[1] D. G. Altman and J. M. Bland, “Statistics Notes: Diagnostic tests 2: predictive values,” BMJ, vol. 309, no. 6947, p. 102, Jul. 1994, doi: 10.1136/bmj.309.6947.102.
[2] D. G. Altman and J. M. Bland, “Statistics Notes: Diagnostic tests 1: sensitivity and specificity,” BMJ, vol. 308, no. 6943, p. 1552, Jun. 1994, doi: 10.1136/bmj.308.6943.1552.
Related Articles
