Information Processing and Communication: A Beginner’s Guide to Information Processing

In a world flooded with information from voice assistants recognising our speech to algorithms sorting our photos, how do machines make sense of the data they receive? Whether it’s the rhythm of spoken words, the shape of a handwritten letter, or the colour of a traffic light, every piece of data must first be measured, processed, and interpreted. This process, known as information processing, lies at the heart of modern artificial intelligence and machine learning systems. Let’s begin with the core base of Information Processing.

Information

Information measures how much we learn from an event and how surprised we are by its outcome. If something expected happens, we gain little to no new information. However, when something unlikely occurs, we receive a great deal of information.

Information can appear in many formats: text, sound, image, signal or video. A song, a news article, or a photo on your phone all carry data, just in a different structure. But the essence is the same: information reduces uncertainty.

Imagine you’re trying to guess the weather tomorrow. If someone tells you, “There will be weather,” that doesn’t help. But if they say, “It will snow in August,” that’s unexpected and contains a lot of information, especially in a place where it rarely snows. The more surprising the outcome, the more information it carries.

In 1948, Claude Shannon formalised this idea through the concept of information entropy, giving us tools to quantify uncertainty in data. His theory underpins everything from text compression and file sharing to how modern devices interpret sounds, images, and messages.

Communication as Information Transfer

In any communication system, information follows a series of steps: it begins as an idea in the sender’s mind, gets encoded into words or signals, is transmitted through a chosen channel, and finally gets decoded by the receiver. This process ensures that the message is understood and can be responded to, completing the communication loop.

Encoding

Encoding refers to the process of creating a message using symbols that others can understand. These symbols can be words, images, gestures, or sounds. The person sending the message (the sender) chooses these based on what they think the other person (the receiver) will understand.

The encoding of a message depends on what the sender wants to convey and how they want the receiver to feel or react.

Encoding is, in other words, the process of converting data into a format that computers can understand, typically binary code. The simplest way to do this is to assign each character, symbol, or instruction a numeric value (usually in decimal), and then convert that value into binary, octal, or hexadecimal — the number systems used by computers to process and store information.

Transmission

Transmission is the step where the encoded message is sent from the sender to the receiver through a specific channel. This channel can be spoken words, printed text, phone lines, radio waves, or digital networks.

The quality and clarity of the transmission depend on the medium and can be affected by noise, distance, or interference.

Decoding

Decoding is the process the receiver uses to interpret the message. The receiver takes the symbols (words, gestures, sounds) and tries to understand the meaning behind them, based on their own experiences, context, and language.

In brief, decoding is the reverse of encoding. It means converting binary or encoded data back into a readable or usable format. For example, when you receive a file or a video, your computer decodes the compressed or encoded format (like .zip, .mp4) so you can read or watch it.

Information Processing

As we see in the diagram, the process starts with raw information like text, sound so on. First, it gets encoded, then it’s transmitted (maybe through the internet, a phone, or even a letter), and finally, it gets decoded on the other side so the receiver can understand it.

For example, in the heart disease detection system shown in the diagram, the process begins with raw clinical data such as patient symptoms or medical test results. This data is first preprocessed, and the encoded noise is removed, and only the most important features are selected. Then, the processed data is passed through a machine learning pipeline (similar to transmission), where feature extraction and oversampling techniques help prepare it for analysis. Finally, the data is decoded by a classification model, which decides whether heart disease is present or not, just like a receiver interpreting the original message.

Entropy and Measuring Information

Before any information can be processed, transmitted, or interpreted, as seen in systems like heart disease detection, we must first understand what information is and how much of it is present in a message. This is where entropy comes in. In information theory, entropy measures the amount of uncertainty or unpredictability in a probability distribution. It tells us how much “surprise” or “information” is associated with the possible outcomes of a random variable.

High and Low Entropy

When the probabilities of all outcomes are spread out evenly, like in a uniform distribution, the entropy is high, because it’s hard to predict which outcome will occur. In contrast, if one outcome is much more likely than the others, the entropy is low, meaning the system is more predictable.

Entropy is measured using the Shannon entropy formula, which quantifies the average amount of information (or uncertainty) in a probability distribution. If we have a random variable X that can take different values, and we know the probability of each value (given by pX​(x)), then the self-information of seeing the outcome x is defined as:

Self Information

As we mentioned before, entropy is the average amount of information (expected information content) in a probability distribution. Therefore, if we multiply each possible outcome by its information content and sum them, we obtain the entropy.

Shannon Entropy

Types of Entropy

As we mentioned before, entropy measures the average amount of information in a probability distribution. However, depending on the context and the type of data we’re dealing with, entropy can take different forms. For example, when analysing the combined uncertainty of two random variables, we use joint entropy. When we want to measure the uncertainty of one variable given that we know another, we use conditional entropy.

Types of Entropy

Joint Entropy H(X,Y)

The joint entropy measures the total uncertainty or randomness associated with both variables taken together. Just like we define entropy for a single random variable, we can extend this concept to a pair of random variables (X, Y). The joint entropy is defined as:

If X and Y are discrete random variables, the joint entropy becomes:

If X and Y are continuous, we define the differential joint entropy as:

Let’s consider two extreme scenarios:

If X = Y (i.e., perfectly correlated), then knowing one gives complete information about the other. In this case entropy of X is the same as the entropy of Y:

If X and Y are independent, there is no shared information, and the joint entropy equals the sum of individual entropies(uncertainty):

Independent events

Another important fact to know is that for any two random variables X and Y, the joint entropy H(X, Y) always satisfies the following inequality, as it captures the combined uncertainty of X and Y, ranging from full overlap (perfect dependence) to no overlap (complete independence):

So, Joint Entropy quantifies the total information needed to describe both X and Y

Conditional Entropy (H(X|Y)

In information theory, conditional entropy measures how much uncertainty remains about a variable Y when we already know the value of another variable X. It tells us how much more information we still need to fully describe Y, given that X is known.

Conditional Entropy

Mutual Information

Mutual information intuitively measures how much information is shared between two random variables, X and Y. It tells us how much knowing one variable reduces the uncertainty about the other. We calculate mutual information as below:

Mutual Information

If you take a closer look at the Venn diagram aga, you’ll notice that the mutual information of the shared part between X and Y can be expressed in several different but equivalent ways.

Each of these forms offers a slightly different perspective on how much information is shared between the two variables and how uncertainty is reduced when one variable is known.

These different expressions highlight that mutual information is not just about overlap; it also tells us how much uncertainty is removed, or how much redundancy exists between two sources of information.

Another important fact about mutual information is that if X and Y are independent, then the mutual information between them is zero. In this case, knowing the value of X gives you no insight into Y, and vice versa.

Mutual Information for Independent Events

Proving that mutual information is zero for independent events(x,y)

If two variables (x,y) are independent, the joint probability would be calculated as follows:

Let's plug our joint probability into our mutual information equation:

As we see above, if there is no connection between events, our independent mutual information will be zero.

For dependent events, the joint probability p(x, y) does not equal the product of the marginals p(x) times p(y). This means there is shared information between X and Y, and mutual information becomes greater than zero. In such cases, knowing one variable reduces the uncertainty of the other, which is exactly what mutual information captures.

Mutual information’s relationship with joint entropy and conditional entropy

Prove mutual information for dependent events.

First, as a reminder, we write the joint probability for dependent variables as follows.

Joint Probability for Dependent Variables

Second, substitute this into our mutual information formula.

And then expand our logarithmic term of the equation

Remember that we can write the total entropy of X and the conditional entropy as follows:

If we substitute entropy into our equation, we can end up with the following equation.

Mutual Information

The idea of measuring uncertainty is not just theoretical; it’s central to many real-world applications, especially in machine learning.

In machine learning, entropy is a fundamental concept used to measure uncertainty, make informed decisions, and guide model learning. It plays a key role in various areas, such as Decision tree algorithms or classification models.

In decision tree learning, entropy is used to evaluate how mixed a dataset is at each node, allowing the algorithm to choose the attribute that best splits the data and reduces entropy (uncertainty).

Desicion Tree

In classification tasks such as logistic regression and neural networks, entropy appears in the form of cross-entropy loss, which measures the difference between the predicted probabilities and the actual labels. By minimising this loss, the model becomes better at making accurate predictions.

Overall, entropy helps guide models toward more certainty, better decisions, and improved performance.

References:

• https://web.njit.edu/~lipuma/352comproc/comproc.htm
• https://www.sciencedirect.com/topics/engineering/shannon-entropy#:~:text=Shannon%20entropy%20is%20defined%20as,of%20obtaining%20a%20value%20xi.
• https://www.geeksforgeeks.org/computer-networks/encoding-and-decoding-in-communication-process/
• https://noone.io/blog/blockchain-security-random-numbers-entropy-cryptography/
• https://www.mdpi.com/1099-4300/21/7/691
• https://medium.com/data-science/decision-tree-ba64f977f7c3
• https://www.youtube.com/watch?v=i78buAHKXpI
• https://bernstein-network.de/wp-content/uploads/2021/02/18_Lecture-18-Application-to-neurons.pdf
• https://pub.towardsai.net/quantifying-gain-fb5a9c18cdac
• https://sccskp.lovestoblog.com/ics-1/data-communication/?i=1