My Study Notes on Deep Learning with Python Book

François Chollet's “Deep Learning with Python” book summary.

My notes on this book are quite superficial, serving as reminders of the content. I haven’t touched on many things, from code examples to detailed architectural explanations. If the content you see here interests you, you can buy the book and read the relevant section.

Summary

The book serves as a practical guide to understanding and building deep learning models using Python and the Keras library. It bridges the gap between the theoretical mathematical foundations of neural networks and their real-world applications in computer vision, natural language processing, and generative AI.

1. Foundations of Deep Learning & Mathematics

• Deep Learning Defined: A subset of machine learning that uses multi-layered artificial neural networks to extract complex patterns from data automatically.
• Tensors: The foundational data structures in deep learning. They are multidimensional arrays containing numerical data (e.g., 2D for tabular data, 3D for time series, 4D for images, 5D for video).
• The Learning Process: Neural networks learn through tensor operations (dot products, element-wise additions). The network makes a prediction, a loss function measures the error, and an optimisation algorithm (Stochastic Gradient Descent/RMSProp) updates the network’s weights in the opposite direction of the gradient to reduce the loss.
• Backpropagation: The algorithm that uses the calculus chain rule to compute these gradients across all layers.

2. Building Neural Networks & Machine Learning Workflow

• Architecture: Models are built using layers (e.g., Dense for tabular data) connected sequentially or via complex graphs. Activation functions such as ReLU are crucial because they introduce nonlinearity, enabling the network to learn complex patterns.
• Data Preparation: Neural networks cannot process raw data. Data must be vectorised, standardised (normalized to a 0–1 scale), and formatted correctly.
• Validation: For small datasets, K-fold cross-validation is recommended over simple train/test splits to ensure statistical reliability.
• Combating Overfitting: The central challenge in machine learning is balancing optimisation (learning the training data) and generalisation (performing well on unseen data). Overfitting can be mitigated by:
• Reducing network capacity (fewer layers/units).
• Applying Dropout (randomly ignoring neurons during training).
• Using Early Stopping (halting training when validation metrics stop improving).

3. Computer Vision (CNNs)

• Convolutional Neural Networks (CNNs): The standard for image processing. They use convolution layers to extract spatial hierarchies of features (from simple edges to complex shapes) and pooling layers (like MaxPooling2D) to downsample data and save computational power.
• Data Augmentation: A technique to prevent overfitting by artificially altering training images (rotating, zooming, flipping) to create more data.
• Transfer Learning: Utilising pre-trained models (like VGG16). You can use feature extraction (using the pre-trained base to process new images) or fine-tuning (unfreezing top layers to train them on your specific dataset).

4. Text and Sequences (RNNs and 1D CNNs)

• Tokenisation: Text must be broken down into words, characters, or subwords (the modern standard for LLMs) before being fed into a network.
• Recurrent Neural Networks (RNNs): Unlike feedforward networks, RNNs have “memory” and process sequences step-by-step.
• LSTMs and GRUs: Advanced RNN variants designed to solve the vanishing gradient problem, allowing the network to remember older, historical data in a sequence.
• Advanced Techniques: Using recurrent dropout, stacking RNN layers, and using Bidirectional RNNs (which read sequences both forward and backward) can drastically improve performance. 1D CNNs are also introduced as a faster, cheaper alternative for text processing.

5. Advanced Best Practices

• The Functional API: While the Sequential model is easy, Keras’ Functional API allows for complex architectures, like models with multiple inputs or outputs.
• Callbacks & TensorBoard: Tools like ModelCheckpoint and EarlyStopping automate saving and stopping models during training. TensorBoard provides browser-based visualisation of metrics and architectures.
• Hyperparameter Tuning & Ensembling: Relying on automated searches to find the best model parameters (layer count, learning rate) and combining multiple models (ensembling) yields the most robust predictions.

6. Generative AI & Conclusion

• The book touches upon the early foundations of GenAI, noting architectures like VAEs (Variational Autoencoders), which are great for continuous, structured latent spaces (like smoothly altering a face), and GANs (Generative Adversarial Networks), which use competing generator and discriminator networks to create highly realistic images.
• Conclusion: While deep learning is incredibly powerful at mapping inputs to outputs, it lacks human-like abstract generalisation and long-term planning. However, it remains a transformative tool that is becoming highly democratised, allowing developers of all levels to build intelligent applications.

The summary has ended, let’s begin.

1. Fundamentals of Deep Learning

An algorithm is a sequence of steps followed to solve a problem or perform a specific task, and the programs that execute these algorithms are called software. Artificial intelligence (AI) is an approach that enables software to perform tasks that typically require human-like intelligence.

Machine learning, an important subfield of artificial intelligence, allows systems to improve their performance by learning from data without being explicitly programmed. In this learning process, systems examine input–output pairs and learn to transform an input into the correct output.

Deep learning, on the other hand, is a more advanced machine learning method that uses artificial neural networks, inspired by the human brain and its multi-layered structure, to extract complex patterns from data.

Today, deep learning achieves remarkable success not only in classification and prediction problems but also in the field of Generative AI. Thanks to architectures such as seq2seq and transformers, it is now possible to perform tasks such as text-to-text translation, text-to-image generation, text-to-video generation, and text-to-music generation. These models can create realistic images, compose songs, generate videos, and write text that closely resembles natural human language. Deep learning models, fueled by large datasets and powerful neural networks, are producing results that increasingly approach human creativity even in fields that require creative output.

Deep learning became popular in the 2010s, driven by advances in graphics processing units (GPUs) and breakthroughs such as ImageNet. It offers high performance and automates the feature-extraction step, making problem-solving easier. Today, it is important to know gradient boosting for relatively shallow problems and deep learning for cognitive problems. For this purpose, we will use XGBoost through Keras.

2. Mathematical Foundations of Deep Learning

To understand deep learning, it is necessary to be familiar with simple mathematical concepts such as tensors, derivatives, and gradient descent. Understanding these basic concepts is important for understanding practical examples.

We will use the MNIST dataset, which contains handwritten digits from 0 to 9, and detect them from visual input.

The building blocks of deep networks are layers, which are data-processing modules that can be thought of as filters. At each layer, the network tries to extract more meaningful relationships and features.

You can better understand the working mechanism with the visualisation in the video below:

[embed]

from keras.datasets import mnist
(train_images, train_labels), (test_images, test_labels) = mnist.load_data()

train_images.shape # (60000,28,28)
len(train_images) # 60000
train_labels # array([5,0,4,...,5,6,8], dtype=unit8)
test_images.shape #(10000,28,28)
len(test_labels) #10000
test_labels # array([7,2,1,..,4,5,6],dtype=uint8)

from keras import models
from keras import layers

network = models.Sequential()
network.add(layers.Dense(512, activation='relu', input_shape=(28*28,)))
network.add(layers.Dense(10, activation='softmax'))

# compilation
network.compile(optimizer='rmsprop', loss='categorical_crossentropy', metrics=['accuracy'])

Our model contains two consecutive Dense layers. The last layer returns a vector of 10 probability scores whose sum equals 1, representing the probabilities for the 10 possible classes. The element with the highest probability corresponds to the predicted class.

The loss function allows the network to evaluate its performance on the training dataset and determine the correct learning direction.

Optimisation is the process of updating the network parameters based on the loss computed from the input data.

The metric we track during training and testing is accuracy, which is the ratio of correctly classified images to the total number of images.

Before starting training, we must scale all inputs to the range [0,1]. Our images are stored as a (60000, 28, 28) array with elements of type uint8 in the range [0,255]. We convert them to float32 and scale them to values between 0 and 1.

from keras.utils import to_categorical

train_images = train_images.reshape((60000, 28*28))
train_images = train_images.astype('float32') / 255
test_images = test_images.reshape((1000, 28*28))
test_images = test_images.astype('float32') / 255

train_labels = to_categorical(train_labels)
test_labels = to_categorical(test_labels)

network.fit(train_images, train_labels, epochs=5, batch_size=128)

test_loss, test_acc = network.evaluate(test_images, test_labels)
print('test_acc', test_acc) # 0.9875

If the accuracy on the training data is higher than that observed during validation, it indicates overfitting. This means the model overfits to the training data and cannot generalise effectively to real-world examples.

We have now seen the practical steps for a simple deep learning approach. Next, we will learn how things work behind the scenes.

Tensors

We represent data as tensors, which are multidimensional NumPy arrays. They mainly act as containers and usually store numerical data.

• Matrices are 2-dimensional tensors.
• The dimensions of a tensor are called axes.

Examples:

import numpy as np
x = np.array(12)
x.ndim # 0

y = np.array([12,3,6,7])
y.ndim # 1

z = np.array([[1,2,3],[4,5,6],[7,8,9]])
z.ndim # 2

A tensor has three key properties:

1. Number of axes (ndim), also called the tensor’s rank.
2. Shape: how many elements exist along each axis.
3. Data type (dtype).

Possible data types include uint8, float32, float64, and rarely char. Tensors cannot store strings because tensors occupy contiguous memory, while strings have variable length. Example:

import matplotlib.pyplot as plt
digit = train_images[3]
plt.imshow(digit, cmap=plt.cm.binary)
plt.show()

With train_images[3]we selected a specific image located along the first axis of the tensor. Selecting specific elements within a tensor is called tensor slicing.

If you use :, you select the entire axis. You can also use negative values, which means selecting from the end.

In real-world examples:

• Data with samples and features are 2D (vectors),
• If there is a time dimension, they are 3D,
• Images containing height, width, channels, and samples are 4D,
• Videos containing frames, height, width, and channels are 5D tensors.

For example:

• If we store how many times the 20,000 most common words appear in text documents, each document is encoded as a vector. For 500 documents, this becomes (500, 20000).
• If we store tweets of up to 280 characters using 128 possible characters and have 1 million tweets, the tensor becomes (1000000, 280, 128).
• For stock market data, if we store the minimum, maximum, and current price every minute for 250 days, the tensor would be (250, 390, 3).
• For images, the tensor could be (128, 256, 256, 3) — 128 samples of 256×256 images with 3 colour channels.
• A 60-second YouTube video at 144×256 resolution sampled at 4 frames per second results in 240 frames. For a batch of 4 videos, the tensor would be (4, 240, 144, 256, 3). If stored as float32, this would take about 405MB, though in reality, videos are much smaller because they are compressed (e.g., MPEG).

We can think of the layer keras.layers.Dense(512, activation='relu') as a function that transforms a 2D input into a 2D output for the next layer. With W as a 2D tensor and b as a vector:

output = relu(dot(W, input) + b)

Here, three tensor operations occur:

1. The dot product between the input data and W,
2. Adding the vector b to the result,
3. Applying the ReLU function (relu(x) = max(x, 0)).

Both the ReLU and addition operations are element-wise, meaning they are applied independently to each element of the tensor. This makes them well-suited for parallel computation.

To better understand what happens behind the scenes, we could implement this naively in Python. However, keep in mind that the libraries used in practice are written in C and are highly optimised, making them more accurate and efficient. Writing it from scratch is mainly for learning purposes.

def naive_relu(x):
  assert len(x.shape) == 2
  x = x.copy() # to not override the input
  for i in range(x.shape([0])):
    for j in range(x.shape[1]):
      x[i,j] = max(x[i,j],0)
  return x

def naive_add(x,y):
  assert len(x.shape) == 2
  assert x.shape == y.shape
  x = x.copy()
  for i in range(x.shape[0]):
    for j in range(x.shape[1]):
      x[i,j] += y[i,j]
  return x

Alternatively, you can do it using numpy:

import numpy as np

z = x + y # element based addition
z = np.maximum(z, 0.) # element base relu

Adding two tensors with different shapes is called broadcasting. Generally, the smaller tensor is spread across the shape of the larger tensor. Axes are added to the smaller tensor to make it equal to the ndim of the larger tensor, and the smaller tensor is repeated on the new axes to make it the same shape as the larger tensor. For example, let the x tensor be (32,10), and the y vector be (10,). First, we add one dimension to y (1,10), then we add y 32 times, thus obtaining the shape (32,10).

def naive_add_matrix_and_vector(x,y):
  assert len(x.shape) == 2
  assert len(y.shape) == 1
  assert x.shape[1] == y.shape[0]

  x = x.copy()
  for i in range(x.shape[0]):
    for j in range(y.shape[1]):
      x[i, j] += y[j]

  return x

Tensor inner product combines two inputs, unlike the dot product, which is an element-based operation.

import numpy as np

z = np.dot(x, y)

def naive_vector_dot(x,y):
  assert len(x.shape) == 1
  assert len(y.shape) == 1
  assert x.shape[0] = y.shape[0]

  z = 0
  for i in range(x.shape[0]):
    z += x[i] * y[i]
  return z

If two vectors have the same number of elements, an inner product can be performed, and the result is a scalar. Similarly, the inner product of the x matrix and the y vector can be performed, resulting in a vector.

import numpy as np

def naive_matrix_vector_dot(x,y):
  assert len(x.shape)
  assert len(y.shape)
  assert.xshape[1] == y.shape[0] # first dimension of x matrix and y vectors zeroed dimension must be the same

  z = np.zeroes(x.shape[0]) # creates a vector full of 0's has the same shape as y

  for i in range(x.shape[0]):
    for j in range(x.shape[1]):
      z[i] += x[i,j] * y[j]

  return z

# 0R

def naive_matrix_vector_dot(x,y):
  z = np.zeroes(x.shape[0])
  for i in range(x.shape[0]):
    z[i] = naive_vector_dot(x[i,:], y)
  return z

def naive_matrix_dot(x,y):
  assert len(x.shape)
  assert len(y.shape)

  assert x.shape[1] == y.shape[0]

  z = np.zeroes((x.shape[0], y .shape[1]))

  for i in range(x.shape[0]):
    for j range(y.shape[1]):
      row_x = x[i,:]
      column_y = y[:, j]
      z[i,j] = naive_vector_dot(row_x, column_y)
   return z

(a, b, c, d) . (d, ) -> (a, b, c)

(a, b, c, d) . (d, e) -> (a, b, c, e)

Transposing a matrix means swapping the rows and columns. The sum of two vectors also has the following geometric meaning:

In the example output = relu(dot(W, input) + b), W and b are randomly assigned initial values. With each training step, these values receive feedback called backpropagation based on where they should be, and they adjust themselves accordingly.

Determining what the weights of this network should be involves computing the gradient of the network’s loss with respect to each of its weights, which is possible because all functions are differentiable. Moving in the opposite direction of the resulting gradient reduces the loss value.

A derivative indicates how much the output of a function changes with respect to a change in its input.

Gradients are a derivative of tensor operations, meaning they are a generalisation from functions to multidimensional inputs.

It is theoretically possible to find the minimum of a differentiable function analytically; the minimum occurs at the point where its derivative is zero. To do this with tensors, you would need to compute the gradient and solve the equation gradient(f)(W) = 0. While this might be solvable forN = 3, in neural networks, the number of parameters can range from several thousand to a trillion, making this approach impossible.

Instead, we use the four-step mini-batch stochastic gradient descent (SGD) algorithm. We look at the current loss value and slightly adjust the parameters in random batches of data. If you update the weights in the opposite direction of the gradient, the loss decreases a little with each step:

1. Form a batch of x inputs and their corresponding y targets from the training data.
2. Perform a feedforward pass to obtain predictions y_pred from the inputs x.
3. Calculate the loss for the batch by measuring the difference between the predicted y_pred and the target y.
4. Compute the gradient of the loss with respect to the network parameters using backpropagation.
5. Update the parameters in the direction opposite to the gradient to slightly reduce the batch loss:

W -= step * gradient

If the step parameter is too small, it requires many iterations to descend, and local minimum compression occurs; if it is too large, random values ​​can be encountered on the curve. SGD has variants like AdaGrad and RMSProp that use optimisation methods. These incorporate the concept of momentum. Momentum addresses two problems: convergence speed and local minimum. If the ball has enough momentum, it can escape the local pit and fall to the deepest point. In practice, the w parameter is updated not only based on the gradient value of the current step but also by looking at the previous update value.

past_velocity = 0
momentum = 0.1

while loss > 0.01:
  w, loss, gadient = get_current_parameters()
  velocity = past_velocity * momentum - learning_rate * gradient
  w = w + momentum * velocity - learning_rate * gradient
  past_velocity = velocity
  update_parameter(w)

In the sections above, we assumed that if a function is differentiable, we can calculate its derivative. However, in practice, neural networks contain chained tensor operations, each with a known simple derivative. From calculus, the derivative of chained functions can be calculated using the chain rule f(g(x) = f’g(x)) * g’(x). The application of the chain rule to calculate the gradient values ​​of neural networks is called backpropagation. Backpropagation starts from the end, takes the missing value, and applies the chain rule downwards to each layer, calculating how much that layer contributes to the missing value. Libraries like TensorFlow can perform symbolic derivative calculations, meaning they use a gradient function that maps the network parameters to their gradient values. Thanks to symbolic derivatives, we don’t have to perform backpropagation manually. Therefore, we should focus on the necessary information about backpropagation and understand how gradient-based optimisation works.

3. Introduction to Neural Networks

In this section, we will try to solve binary classification, multiple classification, and linear regression problems using neural networks. In neural networks, some layers are stateless, meaning they do not carry state information, but most layers have weights consisting of one or more tensors learned as a result of stochastic gradient descent, and these weights hold the information the network has learned. Different tensor formats and data types are suitable for different layers; for example, a simple 2D tensor is generally processed in densely connected layers, serial data is stored in 3D tensors and processed in recurrent layers such as LSTM, and image data is stored in 4D tensors and generally processed in 2D convolutional layers.

Deep learning models are composed of layers arranged in a directed acyclic graph (DAG). The most common examples are linear layers, which map a single input to a single output. However, as you progress, you will see larger networks with different topologies. The most common ones include branched networks, multi-input networks, and Inception blocks.

Even after deciding on a network’s topology, you still need to choose a loss function and an optimisation algorithm. The loss function is the value that will be minimised during training and measures success for the task at hand. The optimisation algorithm determines how the network is updated based on the loss function, typically applying a form of stochastic gradient descent (SGD).

A neural network with multiple outputs can have multiple loss functions, one for each output. However, gradient descent operates on a scalar loss value, so if multiple loss functions exist, they must be combined into a single scalar value, usually by averaging. Choosing the correct loss function for the problem is critical.

Keras is a universal syntax for multiple deep learning engines. You can use Keras code with TensorFlow, PyTorch, and JAX.

In Keras, you can create layers using either the Sequential class or the Functional API. Below is a comparison:

from keras import models, layers
model = models.Sequential()
model.add(layers.Dense(32,activation='reul', input_shape=(784,)))
model.add(layers.Dense(10, activation='softmax')(x)

# or

input_tensor = layers.Input(shape=(784,))
x=layers.Dense(32, activation='relu')(input_tensor)
output_tensor = layers.Dense(10, activation='softmax')(x)

model = Model(inputs=input_tensor, outputs=output_tensor)

In the Functional API, layers are applied to data tensors as if they were functions.

In deep learning, NVIDIA GPUs on Ubuntu are commonly used, which is the most efficient approach. You can also use free Google TPUs via Colab.

You cannot feed lists of numbers directly into neural networks; these lists must be converted into vectors.

Cross-entropy is the best solution for models in terms of probabilities. It comes from information theory and measures the distance between probability distributions; in our case, it measures the distance between our predictions and the true distribution.

A dense layer without an activation function like ReLU only performs two linear operations: a dot product and a summation:

output = dot(W, input) + b

Thus, the network can only learn linear transformations of the inputs. The layer’s hypothesis space consists of all possible linear transformations in 16 dimensions (or the input/output dimensionality).

To obtain a richer hypothesis space and take advantage of deep representations, you need an activation function that breaks linearity.

from keras import optimizers
x_val = x_train[:10000]
partial_x_train = x_tran[1000:]

y_val=y_train[:1000]
partial_y_train = y_train[10000:]
model.complie(optimizer=optimiers.RMSprop(lr=0.001),loss='binary_crossentropy',metrics=['accuracy'])

history = model.fit(partial_x_train, partial_y_train, epochs=20, batch_size=512, validation_data=(x_val,y_val))
history_dict = history.history
histor_dict.keys()
# [u'acc',u'loss, u'vall_acc', u'val_loss']

The model.fit() method returns a history object, which is a dictionary containing information about what happened during training. This dictionary contains four entries recorded during training and validation.

import matplotlib.pyplot as plt

history_dict = history.history
loss_values = history_dict['loss']
valloss_values = history_dict['val_loss']

epochs = range(1, len(loss_values) + 1)

plt.plot(epochs, loss_values, 'bo', label='Train Loss') # bo stands for blue dot
plot.plot(epochs, val_loss_values , 'b' , label= 'Validation Loss') # b is for straight blue line
plt.title('Train and accuracy loss')
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.legend()
plt.show()

plt.clf() # Cleans the pilot

acc = history_dict['acc']
val_acc = history_dict['val_acc']
plt.plot(epochs, acc, 'bo', label='train accuracy')
plt.plot(epochs, val_acc, 'b', label='validation accuracy')

plt.xlabel('Epochs')
plt.ylabel('Accuracy')

plt.legend()
plt.show()

As can be seen, while the training loss decreases with each epoch, the training accuracy increases. When using gradient descent, we want the target we are minimising to decrease at each epoch.

A model that performs very well during training may not perform equally well on examples it has never seen; this is called overfitting. In other words, the model is learning but cannot generalise, effectively memorising the training data. In such cases, to prevent overfitting, you should stop training at an earlier epoch, for example, the third epoch. Some techniques can mitigate the effects of overfitting, such as training for only a few epochs (e.g., four).

After training the network, you want to use it practically. Using the predict method, you can generate probabilities for the outcomes.

Experiment with changing the network architecture:

• Try 1 or 3 hidden layers instead of 2 and observe how validation and test accuracy change.
• Try using fewer or more hidden units per layer.
• Try using MSE as the loss function instead of binary cross-entropy.
• Try using tanh instead of ReLU as the activation function and observe the results.

In summary:

• Before feeding your data into neural networks, you must vectorise it.
• Using dense layers with ReLU activations sequentially can solve many problems.
• For binary classification, your network should end with a single dense layer with a sigmoid activation so that the output is either 0 or 1.
• For a scalar sigmoid output, use binary cross-entropy as the loss function.
• In most cases, the RMSprop optimiser is sufficient.

But what if there are more than two classes? In the next section, we will build a network to classify Reuters news articles into 46 classes, which is a multi-class classification problem.

from keras.datasets import reuters
(train_data, train_labels), (test_data, test_labels) = returers.load_data(num_words = 10000)

len(train_data) # 8982
len(test_data) # 2246

Each word is assigned a number; this is called one-hot encoding.

We build a Sequential model with two hidden layers of 64 units with ReLU activations, and a final layer of 46 units with softmax. We train it using the RMSprop optimiser, categorical cross-entropy loss, and accuracy metrics for 20 epochs with a batch size of 512. When we plot the network’s results, we see that it overfits after the 9th epoch, so we retrain it for only 9 epochs. This approach achieves about 80% accuracy. If we had used a completely random model, accuracy would be around 50%, so this result is quite good.

Since our output has 46 dimensions, we emphasised that the hidden layers should contain at least 46 units. As an experiment, we reduce a hidden layer to 4 units, and the model reaches a maximum of 71% accuracy, meaning an 8% drop. This happens because the hidden layer is too small to capture and separate the information properly. By increasing hidden units or changing layer structures, we can improve results and analyse the effects.

So far, we’ve focused on categorical problems. Now let’s focus on predicting numerical data, i.e., regression problems. Note: Logistic regression is not regression but a classification problem.

We will predict real estate prices from the 1970s, with 506 examples. Our dataset is small. Each feature measures different scales — for example, some are 0–1, some 1–12, and some 0–100. Feeding such heterogeneous data directly into a neural network can cause problems and make learning difficult. Typically, features should be normalised. For each column, we subtract the column mean and divide by the standard deviation. This centres the data around 0 with a unit standard deviation. Even when normalising the test set, we use the mean and standard deviation calculated from the training set; test data biases are never introduced into training.

As the dataset size decreases, overfitting increases, and reducing the network size can help. If a sigmoid output layer is used, the final predictions are constrained between 0 and 1. For training the network, we use MSE (mean squared error) as the loss function. MAE (mean absolute error) measures the absolute difference between predictions and true values — for example, an MAE of 0.5 would correspond to $500.

If the dataset is very small, splitting it for validation can leave very few examples (e.g., 100). The sampling can heavily influence results. To prevent this, k-fold cross-validation is used. The data is split into k parts, and k models are trained, each on k–1 parts with the remaining part used for evaluation. This process is repeated multiple times to account for randomness, and the average result is reliable.

In this system, we can experiment with changes, such as increasing the number of epochs to 500.

4. Fundamentals of Machine Learning

Machine learning is basically divided into four fields: supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning.

Supervised learning is the most popular; it involves learning the mapping from inputs in the dataset to known outputs. Some examples of these tasks include recognising an image caption from a given sequence, parsing a syntax tree from a sentence, identifying and drawing bounding boxes around objects in an image, or creating masks. The goal of unsupervised learning is to visualise, compress, or denoise data. Unsupervised learning is the core of data analytics and can also be performed to understand the data before supervised learning. Dimensionality reduction and clustering are among the most popular approaches.

Semi-supervised learning is a special subset of supervised learning. In semi-supervised learning, there are still labels, but they are discovered from the inputs without human supervision. Autoencoders, for example, are a popular instance of this, where targets are learned from the input.

In reinforcement learning, agents receive information about their environment and select actions to maximise their reward. Today, it can be preferred for optimising an agent in environments with specific rules, or as a post-training phase to align large language models with human preferences.

Scalar regression is a task where the target is a continuous scalar value. Vector regression, on the other hand, is a task where the target is a set of continuous values. If you are performing regression for multiple values, you are doing vector regression — for example, the coordinates of a bounding box enclosing an object in an image. A mini-batch or batch is a small set of data processed by the model simultaneously, typically ranging from 8 to 128. Generally, powers of 2 are preferred, aiming to facilitate memory allocation on the GPU. During training, the stochastic gradient descent calculated on the mini-batch is applied to all the weights of the model.

Although splitting data into training, validation, and test datasets seems obvious, when you have little data, there are more advanced and easier ways to do this. These are hold-out validation, k-fold validation, and shuffled repeated k-fold validation.

In hold-out validation, a portion of the data is set aside for testing, the model is trained on the rest, and then evaluated on the test set. To prevent information leakage, the model is not updated based on the test set; therefore, a validation set is also required. However, if your data is scarce, you will end up with validation and test sets that are too small to statistically represent the dataset. You realise this when you get different performance results with different shuffles. K-fold and repeated k-fold are two methods that can remedy this.

In k-fold validation, the data is divided into k equal partitions. For every i-th partition, the network is trained on the remaining k-1 partitions and evaluated on the i-th partition. Finally, the average of the k scores is taken. As a result, you train and evaluate (number of loops/epochs × k) models, which can be computationally expensive.

Shuffle your data randomly before splitting it. However, if you are looking at the past to predict the future, you should not shuffle it; temporal leakage will occur. If you have duplicate data, it means your test set might leak into your training set, so you need to make sure this doesn’t happen.

Scaling all your data to the 0–1 range, or setting its mean to 0 and standard deviation to 1 internally, is very commonly used and beneficial, regardless of the time it requires, and it can be easily done with NumPy.

In neural networks, models eventually learn that a value of zero means empty (missing). Keep in mind that if there are missing values in your test set but none in your training set, this will cause problems. This means you might need to artificially create missing data in the training set.

Deep learning has freed us from feature engineering because neural networks can automatically learn useful representations from raw data.

Deep learning is an effort to maintain the balance between optimisation and overfitting, and this is called regularisation. Data can be augmented, or training can be cut short (early stopping). The goal is to learn well enough to generalise without reaching the point of memorisation.

Dropout was developed by Hinton and his students at the University of Toronto, and it is suitable for regularising neural networks. During training, dropout randomly sets a portion of the information learned by that layer to zero. Some layer units become zero. Dropout is not used during test time. Instead, because there are more active units than during training time, the layer’s outputs are scaled down by the dropout rate. Hinton noted that he was inspired by bank tellers for this. Tellers are constantly rotated; the reason for this is to reduce internal cooperation to defraud the bank. From this, he realised that randomly removing some neurons could disrupt these established “conspiracies” and prevent overfitting. The main idea is that by adding noise to the layer’s output, you destroy irrelevant patterns that the network might otherwise memorise by chance without that noise.

To summarise, to prevent overfitting in neural networks, data is augmented, the network’s capacity is reduced, weights are regularised, and dropout is applied.

If there is no correlation between your inputs and outputs, neural networks cannot find one. For example, you might try to predict future stock market data from past data, but stock market values are disconnected from the past; they constantly adapt to a changing world, though they might have a slight correlation with the very recent past. Another issue is non-stationary problems. For example, you cannot train your clothing recommendation system in the summer and ask for recommendations in the winter. This is a non-stationary random process, meaning it is time-dependent. The solution for this is to retrain the model with the recent past, spread it over a broader time frame, and also provide time as an input.

Choosing a metric of success: to control something, you need to be able to observe it. If all classes are in equal numbers in a balanced classification problem, performance is measured using ROC AUC (Area Under the Receiver Operating Characteristic curve), but in imbalanced classification problems, precision and recall are used. In ranking or multi-label classification, mean average precision is used. It is common to use your own custom metric to measure your success. However, if you want to develop a feel for where different metrics are used, you can look at Kaggle competition platforms.

For validation, hold-out validation can be used when you have plenty of data; k-fold cross-validation can be used when you have too little data for hold-out to be reliable; or repeated k-fold validation can be used to evaluate the model with high precision when data is scarce. One of these will do the job for you.

Data is converted into tensors and scaled to the -1 to 1 or 0 to 1 range. If features are in different ranges, they must be homogenised (standardised), and feature engineering should be performed for small datasets.

Below you can see the last-layer activations and loss functions for commonly used problem types:

• Binary Classification: sigmoid, binary crossentropy
• Single-label Multi-class Classification: softmax, categorical cross-entropy
• Multi-label Multi-class Classification: sigmoid, binary crossentropy
• Regression to arbitrary values: — (None), MSE
• Regression in the 0–1 range: sigmoid, MSE or binary crossentropy

Regularisation and hyperparameter tuning take up the most time: the model is modified, trained, evaluated on the validation dataset, and modified again, repeating until the best possible model is obtained. Dropout is added, different architectures are tried (layers are added or removed), L1 or L2 regularisation is added, different parameters are tested until the best configuration is found, and optionally, new features are added via feature engineering while unnecessary ones are removed.

Even just analysing and using your validation set can lead to overfitting without you realising it, forcing you to switch to k-fold validation.

5. Deep Learning for Computer Vision

We will use convolutional neural networks to recognise CNNs, diversify data to prevent overfitting, extract features using pre-trained convolutional networks, fine-tune what they have learned, visualise it, and understand how they make decisions.

from keras import layers
from keras import models

model = models.Sequential()
model.add(layers.Conv2D(32, (3,3), activation='relu', input_shape=(28,28,1))
model.add(layers.MaxPooling2D((2,2))
model.add(layers.Conv2D(64, (3,3), activation='relu'))
model.add(layers.MaxPooling2D((2,2))
model.add(layers.Conv2D(64, (3,3), activation='relu')

model.compile(optimizer='rmsprop', loss='categorical_crossentropy',metrics=['accuracy'])
model.fit(train_images, train_labels, epochs=5, batch_size=64)

In the MNIST project to recognise handwritten digits, convolutional neural networks achieved better results than dense models.

The learned patterns are transformation invariant, meaning orientation does not matter. A pattern recognised in the bottom-left corner can be recognised anywhere in the image.

Patterns can learn spatial hierarchies: the first layer learns fine details, and the next layer learns broader patterns built on them, allowing the network to efficiently learn increasingly complex concepts.

Convolutions operate on 3D tensors, which are called feature maps.

When the input size differs from the output size, padding is applied. In pooling, the dimensions are reduced to lower computational cost, and a summary approach is used — either taking the average or the maximum. Using the maximum generally performs better because it tends to encode the spatial presence of features across different layers of the feature map, carrying more information than averaging.

Deep learning works well when you have a lot of data, but the amount of data required depends on the size of your network. A small, well-regularised network can perform well with only a few hundred examples. You can also fine-tune a pre-trained model.

In all convolutional neural network (CNN) architectures, the number of channels in the feature maps increases as the network goes deeper, while the spatial dimensions of the feature maps decrease.

Below, you can see a CNN model for classifying cats and dogs:

from keras import layers
from keras import models

model = models.Sequential()
model.add(layers.Conv2D(32, (3,3), activation='relu', input_shape=(150,150,3))
model.add(layers.MaxPooling2D((2,2))
model.add(layers.Conv2D(64, (3,3), activation='relu'))
model.add(layers.MaxPooling2D((2,2))
model.add(layers.Conv2D(128, (3,3), activation='relu')
model.add(layers.MaxPooling2D((2,2))
model.add(layers.Conv2D(128, (3,3), activation='relu')
model.add(layers.MaxPooling2D((2,2))
model.add(layers.Flatten())
model.add(layers.Dense(512, activation='relu'))
model.add(layers.Dense(1, activation='sigmoid'))

model.compile(optimizer='rmsprop', loss='categorical_crossentropy',metrics=['accuracy'])
model.fit(train_images, train_labels, epochs=5, batch_size=64)

history = model.fit_generator(train_generator, steps_per_epoch=100, epochs=30, validation_data=validation_generator, validation_steps=50)
model.save('cats_and_dogs_small_1.h5')

Before feeding data into the network, we need to preprocess it and convert the tensors to floating-point numbers. Currently, the data is stored on disk as JPEG images. Before feeding them into the network, the image files must be read, the JPEG content decoded, converted into RGB pixel values, and transformed into floating-point tensors. The pixel values should then be scaled from 0–255 to 0–1. Libraries like Keras provide built-in tools to handle these steps automatically.

To increase our data, we can use data augmentation, which includes rotation, width and height shifts, shearing, zooming, horizontal and vertical flips, and filling in any resulting extra pixels. In Keras, these operations are implemented via the **ImageDataGenerator class**.

If your dataset is small, another approach is to use pre-trained models, which can be applied in two ways: feature extraction and fine-tuning.

• Feature extraction uses representations learned by the network to extract useful features from new examples. These features can then be fed into a newly trained classifier.
• In feature extraction, the representations learned by the network are used to extract interesting features for new examples, and a classifier trained from scratch can be used. Convolutional features are more general and reusable, while fully connected layers are less useful if the object’s position in the image is important. Also, deeper layers in the network learn more abstract features.

To train a classifier on top of a randomly initialised classifier, the convolutional base of VGG16 must be frozen. Similarly, to fine-tune upper layers, the classifier must be frozen; otherwise, the large error signal during training could destroy the randomly initialised layers. Therefore, add your own layers on top of the pre-trained network. Freeze the base model, unfreeze some layers of the base, and train the newly added layers together with the unfrozen layers of the pre-trained model.

For example, in VGG16, the first four convolutional blocks (Conv2D + Conv2D + MaxPooling2D) are frozen, and fine-tuning is applied to the last convolutional block and the fully connected classifier (Flatten + Dense + Dense).

Increasing the number of model parameters increases the risk of overfitting. Convolutional neural networks are not black boxes — the learned features can be visualised.

Evrişimli ağın öğrendikleri

6. Deep Learning for Text and Sequences

In this section, we will examine deep learning models that process text, time series, and sequence data. For sequence data, RNNs and 1D convolutional neural networks (CNNs) are commonly used. Applications include:

• Comparing time series or sequences, such as examining the similarity of two documents or stock price movements.
• Sequence-to-sequence learning, such as English–Turkish translation.
• Sentiment analysis, for example, analysing the sentiment of tweets or movie reviews.
• Time series forecasting, like predicting future weather based on past data.

Tokenisation is the process of splitting text into units suitable for the model. In modern NLP and machine learning, three main tokenisation approaches are popular:

1. Word-level tokenisation — splits text into words based on spaces and punctuation, but requires a large vocabulary.
2. Subword-level tokenisation (e.g., Byte-Pair Encoding — BPE, WordPiece, SentencePiece) — learns frequently occurring subword units and can efficiently represent new or unknown words.
3. Character-level tokenisation — treats each character as a separate token, allowing flexibility for rare words and spelling variations.

Modern language models (GPT-like) mostly use subword methods because they keep the vocabulary size manageable and capture linguistic patterns more effectively, allowing them to represent both common and rare word forms efficiently.

So far, the fully connected and convolutional networks we have seen do not have memory. They process each input independently, without retaining information about previous inputs. To process sequence- or time-dependent data with such networks, you would have to feed the entire sequence at once, treating it as a single large input. For example, in the IMDB dataset, the entire movie review would be converted into one large vector and processed in a single pass. These networks are called feedforward networks.

However, in real-world reading, for example, your eyes move across words sequentially, and each word builds on the previous ones. Your biological intelligence maintains an internal model from previous inputs while incrementally updating it with new information.

Recurrent neural networks (RNNs) implement a simple version of this principle. They process each element of a sequence one step at a time, carrying forward the information they have learned as a state. In essence, an RNN is a network with an internal loop.

The RNN state is reset whenever two different sequences are processed, so we are still feeding the network one sequence at a time as a single data instance. The difference is that we now process the sequence element by element, rather than all at once.

The RNN takes vector arrays as input, which are tensors in the form of 2D time and input_features. It iterates through the loop by a time step, and the current state t and the current input input_features are used to obtain the output t. The state of the next step becomes the output of this step. The first step starts without a state because the previous step does not exist; the initial state is a vector of zeros. Only the last output is needed because the entire history of the loop is stored.

model = Sequential()
model.add(Embedding(10000, 32))
model.add(SimpleRNN(32, return_sequences=True))
model.add(SimpleRNN(32, return_sequences=True))
model.add(SimpleRNN(32, return_sequences=True))
model.add(SimpleRNN(32))
model.summary()

RNNs are not very suitable for use with very long strings like text; iterative layers provide better performance. Keras includes SimpleRNN, LSTM, and GRU layers. SimpleRNN can theoretically hold information about all inputs from previous steps at time t, but in practice, learning such long-term requirements is impossible. The gradient disappearance problem, seen in very deep propagation networks, is where the network becomes untrainable as more layers are added.

LSTM and GRU are designed to solve these problems. The Long Short-Term Memory algorithm was developed by Hochreiter and Schmidhuber in 1997 and is the culmination of work on the gradient vanishing problem. LSTM is the RNN version and adds a way to carry information across the time step. Imagine a conveyor belt running in parallel with the array you are trying to process; information in the array can be transferred to the conveyor belt at any point, thus being delivered to the next time step, and then transferred back to its original position when you need it again. This is the fundamental thing that LSTM does: it prevents the old signal from being affected by the gradient vanishing problem while storing the information for later use.

LSTM architecture pseudocode:

output_t = activation(dot(state_t, Uo) + dot(input_t, Wo) + dot(C_t, Vo) + bo)
i_t = activation(dot(state_t, Ui) + dot(input_t, Wi) + bi)
f_t = activation(dot(state_t, Uf) + dot(input_t, Wf) + bf)
k_t = activation(dot(state_t, Uk) + dot(input_t, Wk) + bk)

c_t+1 = i_t * k_t + c_t * f_t

You don’t need to understand the full architecture of an LSTM cell, but you should know what LSTMs are used for: injecting past information into future predictions to mitigate the vanishing gradient problem.

We will discuss three advanced techniques to improve the performance and generalisation of recurrent neural networks (RNNs). Using these techniques, we will solve a challenging problem: predicting the temperature 24 hours ahead based on readings from sensors on a building’s roof (temperature, air pressure, humidity). This problem exposes many of the difficulties encountered in time series analysis.

1. Recurrent dropout: a special dropout technique for recurrent layers to combat overfitting.
2. Stacked recurrent layers: increase the network’s representational power but also increase computational cost.
3. Bidirectional recurrent layers: feed the existing sequence into the recurrent network in multiple directions to improve performance and help deal with forgetting problems.

Gated Recurrent Unit (GRU) layers use the same principles as LSTMs but operate with lower computational cost. However, they do not have the same representational power as LSTMs.

Since a network regularised with dropout takes longer to converge, it is often trained for twice as many epochs.

from keras.models import Sequential
from keras import layers
from keras.optimizers import RMSprop

model = Sequential()
model.add(layers.GRU(32,dropout=0.2, recurrent_dropout=0.2, input_shape=(None, float_data.shape[-1])))

model.add(layers.Dense(1))
model.compile(optimizers=RMSprop(), loss='mae')

history = model.fit_generator(train_gen, steps_per_epoch=500, epochs=40, validation_data=val_gen, validation_steps=val_steps)

Even after training for 30 epochs, our network does not exhibit overfitting. However, we still haven’t reached our best possible score.

In a machine learning workflow, the network’s capacity is increased until overfitting becomes the main limiting factor. The capacity of a network can be increased by adding more units or layers. If adding a new layer does not lead to improvement, there is no need to continue adding layers.

Bidirectional RNNs are a type of RNN that can provide better performance on certain tasks. They are often called the “Swiss Army knife” of natural language processing (NLP).

Standard RNNs are sequence- and time-dependent, processing inputs in chronological order. Bidirectional RNNs look at the sequence from both directions, capturing patterns that may be missed when reading only forward. This provides a richer representation.

For example, in Keras:

model.add(layers.Embedding(max_features,32))
model.add(layers.Bidirectional(layers.LSTM(32))
model.add(layers.Dense(1, activation='sigmoid'))

model.compile(optimizers='rmsprop', loss='binary_crossentropy', metrics=['acc'])
history = model.fit(x_train, y_train, epochs=10, batch_size=128, validation_split=0.2)

We are getting slightly better performance than with LSTM alone, achieving 89% accuracy. However, it leads to faster overfitting, which is expected because it uses twice the number of parameters.

To improve performance on temperature forecasting problems, you can also try the following:

• Change the number of units in the stacked recurrent layers.
• Adjust the learning rate of the RMSprop optimiser.
• Use LSTM layers instead of GRU.
• Add fully connected Dense layers on top of the recurrent layers; even stacked Dense layers can be used.
• Evaluate the best model on the test set according to the validation MAE score. Otherwise, you may have a model that overfits the validation set.

Deep learning is more of a science than engineering, and there is no fixed guide. Each problem is unique, and you need to experiment with all strategies based on observation.

Note: Methods used for market prediction are unlikely to succeed, because the market is constantly changing and never exactly follows the past. It is more reasonable to extract real-time insights from indicators.

Just as 2D CNNs perform well on visual patterns, 1D CNNs perform well on temporal patterns. In some problems, 1D CNNs can be a fast alternative to RNNs, especially in natural language processing.

A 1D CNN is the equivalent of a 2D convolutional network in computer vision. It is built by stacking Conv1D and MaxPooling1D layers, followed by global pooling and flattening layers at the end.

Running RNNs on very long sequences is computationally expensive. Therefore, 1D convolutional networks can be used as a preprocessing step: they shorten sequences and produce useful representations for the RNN to process, all with lower computational cost.

7. Best Practices in Advanced Deep Learning

We will examine powerful tools that can be used to achieve the best results on the most challenging problems. Using the Keras Functional API, we can build graph-based models, share layers across different inputs, and use Keras models like Python functions.

During training, we can monitor models using Keras callbacks and the TensorBoard browser-based visualisation tool. We will also discuss batch normalisation, residual connections, hyperparameter optimisation, and ensemble methods.

Some networks may require multiple independent inputs, some may have multiple outputs, and some may require different branching between layers.

For example, consider a typical question-answering model:

• The question and the short text needed to answer it are embedded separately.
• Each embedding passes through an LSTM, then they are combined and passed through a Dense layer to produce the answer.

You can also build models with multiple output heads. For example, a network predicting age and gender from social media posts would need different loss functions for each output head.

When starting training, you cannot know the optimal number of epochs in a single run — you must experiment and optimise repeatedly, which can be costly. One approach is early stopping, which halts training when the validation loss stops improving. Keras provides callbacks to implement this.

• You can use callbacks like ModelCheckpoint to save weights at different times.
• EarlyStopping implements the approach we just mentioned.
• You can dynamically adjust parameters such as learning rate and optimiser settings during training.
• Training and validation metrics are logged, allowing you to visualise the representations the model learns.

Keras provides predefined callbacks for all these purposes.

Example of using a callback:

import keras
callbacks_list = [keras.callbacks.EarlyStopping(monitor='acc', patience=1,), 
keras.callbacks.ModelCheckpoint(filepath='my_model.h5',
monitor='val_loss',
save_best_only=True,
]
model.compile(optimize='rmsprop', loss='binary_corssentropy',metrics=['acc'])

model.fit(x,y,epochs=10,batch_size=32,callbacks=callbacks_list, validation_data=(x_val, y_val))

You can also write your own callbacks.

TensorBoard is a browser-based experiment visualisation tool. You can visualise metrics, model architecture, gradient and activation histograms, and explore embeddings in 3D.

SeparableConv2D is an alternative to the Conv2D layer that makes the model lighter and faster, while sometimes yielding slightly higher accuracy.

You should train your models repeatedly, but you don’t need to manually tune all hyperparameters all day — you can let the machine do it.

When developing a deep learning architecture, you need to decide on things like:

• Number of layers
• Filters or units per layer
• Type of activation function
• Dropout rate

These are called hyperparameters and are set experimentally.

An automatic hyperparameter search works as follows:

1. A hyperparameter set is selected automatically.
2. The corresponding model is built and trained on the training data.
3. Its performance is measured on the validation set.
4. A new hyperparameter set is chosen, and the process repeats.
5. The final model is evaluated on the test set.

Ensemble methods combine predictions from different models to produce the best result. In ensemble classifiers, assigning high weight to good classifiers and low weight to weaker ones helps maintain generalisation while preserving accuracy.

We do not recommend training the same model multiple times with different random initialisations just to create an ensemble.

8. Generative Deep Learning

In this section, we will look at the potential of deep learning to enhance artistic production from different angles. We will produce sequences using DeepDream, Variational Autoencoders (VAE), and generate images with Generative Adversarial Networks (GANs).

Note: I won’t go into much detail about approaches like VAEs and GANs because they are somewhat older methods; for more details, check the book. They work by learning the latent space that preserves the statistical information of an image dataset. Points are sampled from this latent space and decoded to generate unseen images. VAEs and GANs are used for this purpose.

Greedy sampling makes the most likely prediction, which is why it does not produce natural text.

VAEs learn structured and continuous latent representations. Because of this, they work well in many tasks such as changing poses in the latent space or adding a smile to a face. They also perform well in animations, producing images that gradually change and continuously vary as you move from one end of the latent space to the other.

GANs, on the other hand, reduce continuity and structure slightly in each output.

GANs have a discriminative network that distinguishes between real and fake images, and a generator network. The generator is optimised to produce images that the discriminator cannot distinguish from real ones, meaning they work adversarially. This optimisation differs slightly from standard SGD: the surface changes with each descent, making it a difficult optimisation problem and requiring careful tuning of both the model architecture and its parameters.

9. Conclusion

If we compare human abilities with deep neural networks, we see that there are many fundamental differences. Humans have the ability to generalise and adapt to situations they have never encountered before. They can make long-term plans and combine learned concepts with experiences they have never had — this is a sign of a much more abstract and powerful form of generalisation.

Artificial intelligence and deep learning are a journey, and this book is only the first step you take in that journey. In the future, deep learning will be used by all developers, and everyone will build intelligent applications. Therefore, we must continue to create tools that make this usage increasingly easier.