What is Parameter Estimation Problem?

How We Use Math to Find the Truth in Data?

Have you ever tried to guess something about a large group of people based on sample sample? Perhaps guessing the average height in your city or average salary. This simple act of using limited information to draw large-scale conclusion is the essence of statistics, and it culminates in one powerful concept: Parameter Estimation

This post gently introduces this powerful concept with the help of simple examples.

🚶‍♂️ The Intuitive Example: Regional Heights

Lets start with a relatable, simple example. Imagine we have two distinct populations: Region A and Region B. Due to cultural or geographical factors, the people in these regions have measurable different average heights.

We are given a sample — a random group of people whose height we have measured, but we don’t know if they come from A or B.

Figure 1. Two populations of different heights with an unknown sample.

Our initial task is simple one: classification. Based on average height of our sample, does this group seem more likely to belong to Population A or Population B.

Our common sense tells us that the sample looks more likely from Population A than B because A has relatively more taller people than B on average.

🧠 From Guesswork to the Mathematical Problem

In this simple scenario (A vs. B), we are trying to classify the data. But in reality, the problem is much deeper. Nature has its own processes which determine populations heights along with other properties.

Those processes are what we are interested in, but not directly known to us. One approach to understand those processes is by gaining insights into the observed data – simply put: number of times each observed height data occurred in the sample, also referred to as probability.

This knowledge into probability of each possible height data is represented in the form of data distribution. Data distribution is basically a mapping between height data and its probability of observing them in a sample.

Thus, we can represent each of our population through their corresponding data distribution –a set of probability measures one for each height data.

Assuming there are N distinct height measures then each population will require N-1 probability measures/parameters (the total probability sum to 1, therefore, if we know N-1 probability values, we can compute Nth probability).

To simplify it further, we can use a function for mapping height data to its corresponding probability measure. In that case, the number of parameters for each population reduces from N-1 to the number of parameters the employed function requires.

For example, Gaussian distribution function (the famous bell curve) is widely used for population’s characteristics, and that requires two parameters, mean and variance. So using this function, our initial problem becomes the one presented in the below figure.

Figure 2. Two populations represented through model’s parameters.

The Shift: Why Estimation is Needed

In our simple A vs. B example, we are saying “sample has properties like A.” But in the real world, neither population is predefined.

We usually don’t have menu of choices, we have only a single body of data- a sample of height data. Our task is to use this sample and ask: “What specific values for mean and variance are most likely responsible for generating this exact set of data?”.

Figure 3. Real-world problem of parameter estimation

This is the core of Parameter Estimation.

Hopefully, this overview has helped solidify your understanding of why parameter estimation is the foundation of statistical inference. In our next post, we move from the what to the how, exploring Maximum Likelihood Estimation (MLE) for parameter estimation.

Happy Statistical Learning !