Bayesian Decision Making-1

Statistical Decision Making Part-I

What is Decision Making?

In this chapter we look at making decisions and classifying things. That’s really what pattern recognition is about. A classifier is built using a set of labelled examples to train it; then we check how well it does on a set of examples.

When we have an idea of what the probability distribution looks like for each group we call this type of decision making parametric. The details of the distribution, like the average and how spread out it’s are not known and have to be figured out from the data.

The Big Picture Workflow

Example 3.1 — Cold & Fever

When classes C₁, …, C_k are mutually exclusive and exhaustive, the evidence P(x) expands via the total probability theorem:

Bayes’ Theorem for k Classes

The density p(x|C_i) tells us how concentrated the probability mass is near x for class C_i. High density means x is very typical for that class.

Gaussian Classes — Discriminant Function

For classes that follow a distribution we can equate the weighted densities and then take the logarithm to get a quadratic equation in x.

When we have classes A and B with

• mean of class A as μ_A

• mean of class B as μ_B

• standard deviation of class A as σ_A

• standard deviation of class B, as σ_B

it works like this:

Substituting into Bayes’ Theorem:

Continuous Features — Diagonal Covariance

For the features that’re continuous, being independent of each other within a certain class let us say class C_i means that all the numbers that are not, on the main diagonal of the covariance matrix Σ_i are zero.

For two bivariate normal classes, the general boundary has the form:

When Covariance Matrices are Equal

Pooled Covariance Matrix

When covariances are assumed equal, estimate Σ from all classes jointly:

The decision rule “Choose A” applies when:

Model-Based Error Rate

The probability of error for class C_i is the probability that the feature falls outside its decision region R_i:

Confidence Intervals via Simple Counting

With n test samples and k errors, the estimated error rate is P̂(ε) = k/n. The 95% confidence interval is approximately:

§ 3.9 — Characteristic Curves (ROC)

In classification we have to deal with two kinds of mistakes. The first one is called a positive. This is when we say something is true when it is actually not true. The other kind of mistake is called a negative. This is when we say something is not true when it actually is true. When we try to find the point to make our decision we have to choose between these two kinds of mistakes. We can make one kind of mistake often but that means we will make the other kind of mistake more often. Binary classification is about finding a good balance, between false positives and false negatives.

In binary classification we face two error types: false positive (classifying 
T̄ as T) and false negative (classifying T as T̄). Moving the decision boundary 
trades one type of error for the other.

§ 3.9 — Confusion Matrices

A confusion matrix summarises all classification outcomes at once. For 3 classes A, B, C:

Reference

1. Pattern Recognition Book

NEXT - https://medium.com/p/4508c092bf5b?postPublishedType=initial