Perceptron

Understanding the Perceptron: The Simplest Neural Network

If you want to understand modern Artificial Intelligence, you need to start with one simple idea: the perceptron.

The perceptron is the most basic form of a neural network, and it plays an important role as the foundation of deep learning. Even though it is simple, it introduces key ideas such as learning from data, decision boundaries, and model training.

What is a Perceptron?

A perceptron is a binary classification model. It takes several inputs and produces a single output: either 0 or 1.

In simple terms:

A perceptron is a model that makes a decision based on input values.

It works by:

• Receiving input data
• Assigning importance (weights)
• Combining them
• Producing a decision

This simple idea later evolves into complex neural networks.

Basic Components of a Perceptron

A perceptron consists of a few key parts:

1. Inputs (Features)

These are the data we use for prediction. Example:

• Study hours
• Attendance

2. Weights

Each input has a weight that represents its importance.

Higher weight = more influence

3. Bias

Bias is a constant value added to the model.

It helps shift the decision boundary and makes the model more flexible.

4. Weighted Sum

All inputs are combined mathematically: z = w1x1 + w2x2 + b

5. Activation Function

The perceptron uses a step function:

If (z ≥ 0) → output = 1

If (z < 0) → output = 0

This turns the result into a decision.

How Does a Perceptron Work?

The process is very simple:

1. Multiply inputs by weights
2. Add bias
3. Apply activation function
4. Produce output

This is called forward propagation

How Does It Learn? (Training Process)

A perceptron learns from mistakes.

Steps:

1. Make a prediction
2. Compare with the actual result
3. Calculate error
4. Update weights

The update rule is:

w = w + h (t — y) x

b = b + h (t — y)

Where:

t = true value

y = prediction

h = learning rate

b = bias

The model improves gradually over time.

Simple Example

Imagine predicting whether a student passes:

Inputs: study hours, attendance

Output:

• 1 = pass
• 0 = fail

The perceptron combines inputs and decides based on a threshold.

If the score is high enough → pass Otherwise → fail

Simple Perceptron Calculation Example

then

then

Decision Boundary (Key Idea)

A perceptron separates data using a straight line.

This line is called the decision boundary.

This is important:

• Works well if data can be separated linearly
• Fails if data is complex

Limitations of Perceptron

The perceptron has an important limitation:

It can only solve linearly separable problems

For example:

It cannot solve XOR problem

This limitation led to the development of more advanced models.

Perceptron training examples (step by step)

then

• Perceptron learns by trial and error
• It improves step-by-step
• Learning = updating weights using mistakes

Here is the simple code to show how perceptron learning

import numpy as np

# =========================
# 1. DATA
# =========================
# x1 = study hours, x2 = attendance
X = np.array([
    [4, 5],
    [2, 3]
])

y = np.array([1, 0])  # target

# =========================
# 2. INITIALIZATION
# =========================
w = np.array([0.0, 0.0])  # weights
b = 0.0                   # bias
alpha = 1.0               # learning rate

# =========================
# 3. ACTIVATION FUNCTION
# =========================
def step(z):
    return 1 if z >= 0 else 0

# =========================
# 4. TRAINING LOOP
# =========================
epochs = 5

for epoch in range(epochs):
    print(f"\nEpoch {epoch+1}")

    for i in range(len(X)):
        x = X[i]
        target = y[i]

        # forward
        z = np.dot(w, x) + b
        pred = step(z)

        # error
        error = target - pred

        print(f"\nData: {x}, Target: {target}")
        print(f"Prediction: {pred}, Error: {error}")

        # update
        w = w + alpha * error * x
        b = b + alpha * error

        print(f"Updated weights: {w}")
        print(f"Updated bias: {b}")

the result of this coding is,

Epoch 1

Data: [4 5], Target: 1
Prediction: 1, Error: 0
Updated weights: [0. 0.]
Updated bias: 0.0

Data: [2 3], Target: 0
Prediction: 1, Error: -1
Updated weights: [-2. -3.]
Updated bias: -1.0

Epoch 2

Data: [4 5], Target: 1
Prediction: 0, Error: 1
Updated weights: [2. 2.]
Updated bias: 0.0

Data: [2 3], Target: 0
Prediction: 1, Error: -1
Updated weights: [ 0. -1.]
Updated bias: -1.0

Epoch 3

Data: [4 5], Target: 1
Prediction: 0, Error: 1
Updated weights: [4. 4.]
Updated bias: 0.0

Data: [2 3], Target: 0
Prediction: 1, Error: -1
Updated weights: [2. 1.]
Updated bias: -1.0

Epoch 4

Data: [4 5], Target: 1
Prediction: 1, Error: 0
Updated weights: [2. 1.]
Updated bias: -1.0

Data: [2 3], Target: 0
Prediction: 1, Error: -1
Updated weights: [ 0. -2.]
Updated bias: -2.0

Epoch 5

Data: [4 5], Target: 1
Prediction: 0, Error: 1
Updated weights: [4. 3.]
Updated bias: -1.0

Data: [2 3], Target: 0
Prediction: 1, Error: -1
Updated weights: [2. 0.]
Updated bias: -2.0

The output shows:

• Prediction for each data point
• Error (true/false)
• Change in weights for each step

From Perceptron to Deep Learning

To overcome its limitations, researchers introduced:

• Multi-Layer Perceptron (MLP)
• Hidden layers
• Non-linear activation functions

These improvements allow models to:

• Learn complex patterns
• Create non-linear decision boundaries
• Power modern AI systems

Why is Perceptron Important?

Even though it is simple, the perceptron is:

• The first step toward neural networks
• A foundation of deep learning
• A great tool to understand how machines learn

Conclusion

The perceptron is more than just a simple model. It represents the beginning of machine learning as we know it today.

From a simple weighted sum to deep neural networks, everything starts here.

But there is a problem of Perceptron

The limitation of the perceptron was first solved by Multi-Layer Perceptron (MLP)

Step-by-Step Explanation

1. The Problem with Perceptron

A perceptron can only:

• Create a linear decision boundary (a straight line)

Because of this:

• It cannot solve problems like XOR
• It cannot handle complex patterns

2. The First Idea to Solve It

Researchers thought:

“If one perceptron is not enough… what if we use more?”

3. Multi-Layer Perceptron (MLP)

Structure:

Input → Hidden Layer → Output

Key idea:

• The hidden layer transforms the data
• So the data becomes easier to separate

4. Why MLP Works

• Each neuron creates a line
• Many neurons together create a complex shape

So:

• Perceptron → one straight line
• MLP → combination of lines → looks like a curve

5. New Problem Appears

Even though MLP was invented…

It was very hard to train

Why?

• No efficient method to update weights in multiple layers

6. The Next Big Solution

Backpropagation

What Backpropagation Does

• Sends the error from output back to earlier layers
• Updates all weights step by step

This made neural networks trainable and practical

Historical Flow (Very Important)

1. Perceptron
2. Fails on XOR (-)
3. Multi-Layer Perceptron
4. Hard to train (-)
5. Backpropagation
6. Modern Neural Networks

“The perceptron failed because it was too simple. The solution was to stack more layers.”

Key Insight

• Hidden layers → make models non-linear
• Backpropagation → makes models learn

import numpy as np
from sklearn.neural_network import MLPClassifier

# =========================
# 1. DATA (XOR)
# =========================
X = np.array([
    [0, 0],
    [0, 1],
    [1, 0],
    [1, 1]
])

y = np.array([0, 1, 1, 0])

# =========================
# 2. MODEL MLP
# =========================
model = MLPClassifier(
    hidden_layer_sizes=(2,),  # 1 hidden layer, 2 neurons
    activation='tanh',        # non-linear activation
    solver='lbfgs',           # optimizer
    max_iter=10000
)

# =========================
# 3. TRAINING
# =========================
model.fit(X, y)

# =========================
# 4. PREDICTION
# =========================
predictions = model.predict(X)

print("Predictions:", predictions)
print("Actual     :", y)

# =========================
# 5. CHECK ACCURACY
# =========================
accuracy = model.score(X, y)
print("Accuracy:", accuracy)

give result

Predictions: [0 0 1 1]
Actual     : [0 1 1 0]
Accuracy: 0.5

important insight

“MLP is not just a deeper perceptron. It also needs a different activation function to learn.”

import numpy as np
from sklearn.neural_network import MLPClassifier

# =========================
# 1. DATA XOR
# =========================
X = np.array([
    [0, 0],
    [0, 1],
    [1, 0],
    [1, 1]
])

y = np.array([0, 1, 1, 0])

# =========================
# 2. MODEL
# =========================
model = MLPClassifier(
    hidden_layer_sizes=(2,),
    activation='tanh',
    solver='sgd',
    learning_rate_init=0.1,
    max_iter=1,        # 1 epoch saja
    warm_start=True    # lanjut dari bobot sebelumnya
)

# =========================
# 3. TRAINING PER EPOCH
# =========================
epochs = 20

for epoch in range(epochs):
    model.fit(X, y)

    pred = model.predict(X)
    acc = model.score(X, y)

    print(f"Epoch {epoch+1}")
    print("Prediction:", pred)
    print("Accuracy :", acc)
    print("-" * 30)

give the result

Epoch 1
Prediction: [0 0 0 0]
Accuracy : 0.5
------------------------------
Epoch 2
Prediction: [0 0 0 0]
Accuracy : 0.5
------------------------------
Epoch 3
Prediction: [0 0 0 0]
Accuracy : 0.5
------------------------------
Epoch 4
Prediction: [0 0 0 0]
Accuracy : 0.5
------------------------------
Epoch 5
Prediction: [0 0 0 0]
Accuracy : 0.5
------------------------------
Epoch 6
Prediction: [0 0 0 0]
Accuracy : 0.5
------------------------------
Epoch 7
Prediction: [0 0 0 0]
Accuracy : 0.5
------------------------------
Epoch 8
Prediction: [0 0 0 0]
Accuracy : 0.5
------------------------------
Epoch 9
Prediction: [0 0 0 0]
Accuracy : 0.5
------------------------------
Epoch 10
Prediction: [0 0 0 0]
Accuracy : 0.5
------------------------------
Epoch 11
Prediction: [0 0 1 0]
Accuracy : 0.75
------------------------------
Epoch 12
Prediction: [0 0 1 1]
Accuracy : 0.5
------------------------------
Epoch 13
Prediction: [0 0 1 1]
Accuracy : 0.5
------------------------------
Epoch 14
Prediction: [0 0 1 1]
Accuracy : 0.5
------------------------------
Epoch 15
Prediction: [0 0 1 1]
Accuracy : 0.5
------------------------------
Epoch 16
Prediction: [0 0 1 1]
Accuracy : 0.5
------------------------------
Epoch 17
Prediction: [0 0 1 1]
Accuracy : 0.5
------------------------------
Epoch 18
Prediction: [0 0 1 1]
Accuracy : 0.5
------------------------------
Epoch 19
Prediction: [0 0 1 1]
Accuracy : 0.5
------------------------------
Epoch 20
Prediction: [0 1 1 1]
Accuracy : 0.75
------------------------------

here it is the visual of MLP (Multi Layer Perceptron)

I think this one is enough for today

what is the next problem?