What I Learnt After a Week: Deep Learning -Set up ML application (Learning in Public #3)

“Beauty can’t amuse you, but brainwork: reading, writing, thinking — can.” by Helen Gurley Brown

This week, I spent nearly five full days into Andrew Ng’s Deep Learning Specialization. These notes cover only about half of what I absorbed and the other half is still processing in my mind. I’ll share more soon.

What makes me truly happy is realizing how much my basic math background is helping now: understanding formulas, vectorization, and derivatives feels rewarding after years of not touching them. Of course, there are those moments when I stare at an equation and feel completely numb, but taking detailed notes and drawing quick visualizations always pulls me through.

Iam sharing multiple Deep Learning notes as I learn Andrew Ng’s course. Early-access notebook here → Deep Learning Journey

I even managed to slip away for a short Christmas break with family 🎄🎄🎄. Beneath twinkling lights and over shared meals ❤️ stomach full, heart fuller ❤️ my thoughts drifted quietly to this journey. I wondered where AI, machine learning, and deep learning might lead me in the year ahead. In those moments, a gentle truth settled in: I hold the rare gift of time and the ability to grasp these concepts 🌷. With it comes a quiet responsibility to cherish and nurture them, day after day, with appreciation and respect.

If you’re just starting out like me, you feel curious about AI, or you just dipped your toes into machine learning or deep learning, feeling that mix of excitement and overwhelm , this is for you. You’re not alone in this. Let’s explore what clicked for me this week.

Working with Datasets:

When your dataset is small (fewer than 10,000 examples), the classic split of 70/30 or 60/20/20 for train/dev/test feels right. The key rule I now appreciate: all three sets must come from the same distribution. Otherwise, your test results won’t tell you anything useful about real-world performance.

In today’s big-data world, though, scarcity isn’t usually the problem. With hundreds of thousands or millions of examples, it’s common to put 98% or more into training and leave just tiny slices for validation and testing. More data in training almost always beats perfectly balanced splits.

Reading Your Model’s Behaviour

To figure out whether a trained model’s is performing well, we observe proxy indicators (e.g., training error and validation ( or test) error. The gap between them tell us whether the model has a high variance issue (overfitting) or a high bias issue (underfitting). Below is what I have summarised:

Great reference: Cornel University Lecture on Bias/Variance and Model Selection https://www.cs.cornell.edu/courses/cs4780/2015fa/web/lecturenotes/lecturenote13.html

Wait! What’s the Difference Between a “Model” and a “Neural Network”?

I kept mixing these up, so I did another round of reading and here’s the clarity that finally clicked for me:

1. Machine learning: refers to enabling computers to learn patterns from data and make predictions or decisions without being explicitly programmed for every specific task. Unlike traditional programming (input + rules → output), ML uses data (inputs, outputs) to discover the rules. It can then perform tasks on new data. The main component that learns and discovers the rules (data patterns) is the “Model.” Traditional programming: Inputs + (programmed) Rules → Outputs ML: Data + Labels → Trained Model → Predictions on new Inputs
2. Deep learning: A subset of ML that uses neural networks with many layers to capture complex patterns from large amounts of data.
3. Model: is what you get after an ML training process

OK, so now the job of a software engineer and a machine learning engineer are clarified for me.

1. In traditional software development, a programmer writes rules (code) that produce the correct output for a given input.
2. In machine learning, an ML engineer trains a model using data, a model architecture, and parameters, so the model can make predictions on new data in the future (inference).

ML training process: Data + Algorithm → Trained Model → Predictions

To dive a bit deeper:

1. In traditional Machine Learning: A model is a learned function or algorithm with parameters (e.g., predictor) Linear Regression: with weights and bias Decision Tree: threshold in nodes
2. In Deep Learning: A deep neural network (multi-layered structure of interconnected nodes) is a type of architecture that can be used to train a model. And a trained neural network is a model that consists of: Architecture: number of nodes, layers *Hyperparameters: Specific choices (number of layers, nodes, learning rate…) Parameters: *Learned values — weights and bias During training, data flows through the network, and backpropagation updates the weights and bias to minimize error.

Regularization: Teaching the Model to Generalise Instead of Memorise

First, I think it’s worth understanding overfitting: it’s the case where a trained model memorises the training examples instead of discovering real patterns. That’s why it can produce great results on the training dataset but has high error rates on the test dataset.

With regularization techniques, we have different ways to punish this memorisation and force the model to “learn”, aka create general rules so that it can perform better on new datasets.

Different techniques:

1. L2 Regularization (also called Weight Decay)

• The idea: Penalize large weights → force the model to be trained using all features instead of relying on a few heavily-weighted features.
• Formula: Cost = Original_Cost + λ × Σ(w²)
• The larger the weight, the more the cost increases by λ × Σ(w²)

2. Dropout

• The idea: Randomly “turn off” some nodes in each layer in each iteration, so the neurons network can’t rely on any specific nodes. This forces the model to learn robust patterns that work with different subsets of neurons.
• Downside: The calculated cost becomes noisy (each iteration uses different subnet), making it harder to monitor training progress. It is important to remember the following with Drop-out When Training: Dropout ON — randomly drop neurons. When Testing: Dropout OFF — use all neurons.
• The scaling issue of dropout: When we use for example 50% dropout rate during training, 50% neurons are randomly turned off. The network learns to produce correct outputs using only half its neurons. However, at test time when ALL neurons are active, the output naturally becomes twice as large — breaking the predictions! In other words: We train the network with half the neurons dropped, expecting it to produce the correct output with this reduced capacity. But when testing with all neurons active, the combined output is scaled up (approximately doubled), causing incorrect predictions.
• Scaling solutions:
• Inverted dropout (train time): output = output / keep_prob → no scaling needed at test.
• Regular dropout (test time): output = output * (1-keep_prob) → scale down at test.

3. Early Stopping We stop training the model just before the validation error starts increasing.

4. Data Augmentation

• The idea: Force the model to learn true features, as flipped, rotated, or zoomed features are still the same features but in different positions, pixels, or resolutions. In classic cat classifications, we force the model to recognise that a cat is still a cat even if it’s picture is upside down, or cropped.
• Techniques include below for images: Horizontal flip Random zoom-in Random small distortions

Setting Up Optimization Properly — Part 1

1. Normalising inputs: to my understanding, is similar to creating a level playing field for your optimizer (model) to perform better.

So, what are the problems with unnormalised datasets? Below are few examples

1. Gradients vary based on feature scales: When features have different scales, their gradients will differ. For example:

• Feature 1: Age (0–100) → ∂L/∂w_age = ∂L/∂z × x_age (small x) → small gradient
• Feature 2: Income (0–100,000) → ∂L/∂w_income = ∂L/∂z × x_income (large x) → large gradient

The weight for age (w_age) barely moves while the weight for income (w_income) takes big steps. The optimizer takes huge steps in one direction and tiny steps in others, zigzagging slowly toward the minimum instead of following a smooth path.

2. Vanishing/Exploding Gradients: When input values vary widely like example below

X = [0.01, 1000, 0.5, 50000] # tiny, HUGE, normal, GIGANTIC

After weights (assume W=1 for simplicity): *

z = W x X = [0.01, 1000, 0.5, 50000]*

Sigmoid output *

σ(z) = [≈0, ≈1, 0.62, ≈1]*

Sigmoid GRADIENTS at these outputs: *

σ’(z) = σ(z) × (1 — σ(z)) = [0x1 , 1x0, 0.62 x 0.38, 1x0 ] = [≈0, ≈0, 0.24, ≈0] *# Neurons at 0 and 1 are saturated # Sigmoid’(0)≈0

*Gradient = ∂L/∂W ≈ 0 *→ Learning stops!

3. No single learning rate fits all features

• Feature 1: Age (0–100) → small gradient → needs large learning rate (e.g., 1.0)
• Feature 2: Income (0–100,000) → large gradient → needs small learning rate (e.g., 0.0001)

Common normalisation methods:

Method 1: Standardisation

X_normalized = (X - mean_trainset) / std_trainset

Method 2: Min-Max Scaling

X_normalized = (X - min_trainset) / (max - min) of train set

NOTE: You must normalize new data based on training statistics. Otherwise, the model sees a different distribution → wrong predictions. Process: Split → Calculate stats → Normalize → Train

# Original data (not normalised yet, not trained yet)
X_all = [data...]

# SPLIT FIRST!
X_train, X_test = train_test_split(X_all)
# Calculate from X_train (not X_all!)
mean_train = X_train.mean()
std_train = X_train.std()
# Normalize train set
X_train_norm = (X_train - mean_train) / std_train
# Normalize test set ( SAME mean/std from train!)
X_test_norm = (X_test - mean_train) / std_train
                          ↑            ↑
                    train stats!  train stats!

“In today’s knowledge-based economy, what you earn depends on what you learn.” by William J. Clinton

That’s half of everything for this week 😄. If you’re feeling a little overwhelmed right now, welcome to the club 💐. I definitely am. These concepts are dense, and I already know I’ll revisit these notes many times before they truly feel like mine.

But that’s also the fun part: every reread uncovers something new. And remember, just like me, you have time on your side if you show up consistently.

Happy learning, friends. Keep showing up: your future self will thank you.

This is part 3 of my “Learning in Public” series, documenting my restart of Andrew Ng’s Deep Learning Specialization.

My early-access notebook compiles multiple plain-language notes to help you understand and remember Deep Learning concepts faster. Get it here → Deep Learning Journey