Understanding Bayesian Reasoning: Unraveling Diagnostic Test Reliability and Error Analysis

Bayesian reasoning offers a perspective on how we update beliefs in light of new evidence i.e., instead of concluding about the situation, new evidence is used to update our prior beliefs and obtain a posterior belief. In our preceding discussion, we offered a derivation of the Bayes’ formula as well as an intuitive exploration of its concise expression. In this post, we will be starting with the takeaway practice from that post, and delve deeper into how Bayes’s formula takes into account both the accuracy of the test and the overall prevalence of the disease. Let's investigate this very common example:

A patient goes to see a doctor. The doctor performs a test with 99 percent reliability. This means 99% of sick people get a positive result, and 99% of healthy people get a negative result. In this country, only 1% of the population is actually sick. Given this information, if a person receives a positive test result, what is the probability that this person is truly sick?

If I didn’t know Bayes theorem I would say %99. After all, the test is %99 reliable and it says that the patient is positive. So why are we questioning that? Let's investigate this case in a population of 10.000 people.

Table 1: The distribution of values in the question to help illustrate how the population is divided based on the given probabilities and test results.

Let me explain this table step by step. From the information, “In this country, only 1% of the population is actually sick”, it can be inferred that 100 out of every 10,000 people in the population have this specific disease, which gives the very bottom row in Table 1. Considering the information that “99% of sick people get a positive result”, one can conclude that approximately 1 percent of sick people (or 1 out of 100) test negative, which then gives the second column in Table 1. Finally, the statement “99% of healthy people get a negative result”, implies that about 1 percent of the healthy population (or 99 out of 9900 people) test positive, which finally gives the third column in Table 1.

At this point, we don’t actually need Bayes’s rule, as the probability we’re seeking can be directly calculated from the data in Table 1. Remember, our goal is to determine: if a patient tests positive, what is the probability that this patient is actually sick? Meaning, that out of the 198 people who test positive, what is the probability that they are sick? With a population of 198 people, this includes 99 sick individuals and 99 healthy individuals, if we randomly select one of these 198 individuals, the probability that they are sick is 99/198, which equals 0,5.

Now let's put this problem in Bayes’s perspective:

P(H) represents the prior probability of having the disease, calculated before any evidence is obtained, that is, before conducting any tests. Given that “In this country, only 1% of the population is actually sick”, one can understand that P(H) = 0.01

P(E|H) is the probability of a positive test result among diseased individuals, which is given as 99% in the problem, so P(E|H) = 0.99

P(E|¬H) is the probability of a positive test result among healthy individuals. So P(E|¬H) = 99/9900 = 0.01

P(¬H) is the probability of not having the disease. calculated as 1 — P(H), which is 1–0,01 = 0,99. Table 1 also shows that 9,900 out of 10,000 people do not have the disease, confirming P(¬H) = 9,900/10,000 = 0.99.

Table 2: The distribution of values in the question to help illustrate how the population is divided from Bayes’ perspective

P(E) is the overall probability of obtaining a positive test result, regardless of whether the individual is sick or healthy. This is calculated by considering both the probability of sick individuals testing positive and the probability of healthy individuals testing positive. So it can be calculated as:

P(E) can also be interpreted as the unconditional probability of a positive test. Table 1 (and 2) also confirms that;

P(E) = Total Positive Tests / Population = 198 / 10.000 = 0.0198

All in all, we have:

This problem is a classic example of how our intuitions can sometimes mislead us. Our initial intuition might lead us to believe that if a test has a 99% reliability, then if we get a positive test result, there’s a 99% chance we are actually sick.

Bayesian reasoning allows us to consider not only the reliability of the test but also the prevalence or incidence of the disease in the population. It helps us see that the probability of being sick after a positive test result is not solely determined by the test’s reliability but is also heavily influenced by the prevalence of the disease in the population. In situations with a low disease prevalence, even a highly reliable test can result in a relatively low probability of actually being sick after a positive result.

While we use the term “reliability” so often, and I really find it ambiguous, I want to open a parenthesis here to investigate what should we understand from this statement.

In the question, one can see that a medical test is being conducted. The patient who is subject to a test should be diseased or non-diseased, and the test result should be positive or negative. Here are all the scenarios for those cases;

Table 3: Illustration of outcomes of the test.

In the table, you can see the false results are highlighted in red font. Please pause and reflect for a moment; are these two types of false results the same? Can you see any differences in terms of cost, etc.?

The cost of these tasks should vary depending on the context. Let’s exaggerate, assume you are developing an app that classifies skin cancer.

Table 4: Illustration of outcomes in the test in the context of skin cancer to emphasize the cost of misclassification.

Now can you see a difference in these wrong classifications? There is indeed a significant difference: on one hand, a false negative leads to a missed diagnosis of skin cancer, potentially preventing the patient from seeking medical care on time and starting appropriate treatment. On the other hand, the false result causes much less damage; it causes some anxiety. After all, the patient visits the clinic and eventually learns he does not have skin cancer.

So, these two types of errors are different, and depending on the context, their cost should be totally different. Let's emphasize the two differences in the following paradigm:

Table 5: Illustration of the concepts of True Positive, True Negative, False Positive, and False Negative.

Diseased & Test +: A diseased person is truly classified as positive by the test. Therefore, it is called a true positive.

Diseased & Test -: A diseased person is falsely classified as negative by the test. Therefore, it is called a false negative. This is also called a Type II Error. In our skin cancer example, the cost of false negatives is devastating. The wrong diagnosis caused the death of the patient. Think of it for a second. Does it look like a miss?

Table 6: Illustration of the concepts of Type I and Type II error.

Non-diseased & Test +: A non-diseased person is falsely classified as positive by the test. Therefore, it is called a false positive. This is also called a Type I error. In our skin cancer example, the cost of false positives is not devastating. The wrong diagnosis caused anxiety, but eventually, the patient understood that he was actually not sick. Think of it for a second. Does it look like a false alarm?

Non-Diseased & Test -: Non-Diseased and truly classified as negative by the test. Therefore, it is called a true negative.

Many of the readers who are acquainted with statistics and machine learning grasped that we are dealing with binary classification, and going through the metrics of sensitivity, specificity, and accuracy. However, our primary interest, for now, is not to investigate the metrics to measure the performance of a diagnostic test, but only to relate these concepts with the Bayesian perspective.

In the context of binary classification, a Type 1 error occurs when a test incorrectly indicates the presence of a condition, or in other words, creates false alarms. A Type 2 error, on the other hand, occurs when a test fails to detect a condition in a person who is actually sick, or in other words, it misses. In our example, Bayesian reasoning helped us recognize that even a test with high reliability can yield false positives, particularly when the prevalence of the disease in the population is low. In some other scenarios, say high disease prevalence, it may be more useful to help recognize false negatives.

I believe now we have a better understanding of the term reliability of a test. In the next practice, we will be more specific, and get rid of the ambiguous term “reliable”. Here’s a practice to deepen our insights from this investigation:

In a population, a specific illness is found in 0.1% of the individuals. For a diagnostic test for this illness, the rate of false negatives is 10%, and the rate of false positives is 1%. Determine the probability that an individual who receives a positive result on this test actually has the disease.

1. Solve the question without using Bayes’ formula (Hint: create a table of population consisting of 10.000).
2. Solve the question using Bayes’ formula.