From Tokens to Transformers: Addendum A

From DNNs to RNNs to LSTMs: A Three-Part Journey

Prefer the nuts-and-bolts? So, there…

Photo by Marcos Paulo Prado on Unsplash

Not a member? Read the full story here.

You’ll find the mathematical expressions in the main flow. Prefer to build intuition first? Skip to the Intuition sections, where the ideas are unpacked in plain language and short analogies before we circle back to the formalism. Or, you may choose to ignore the math altogether.

Recurrent Neural Network

Schematic diagram of LSTM

Intuition

Think of a student reading a long passage, one sentence at a time. The student keeps a notebook and also gives a quick summary after each sentence. The notebook is the LSTM’s long-term memory (cell state ct).The summary is the short-term, outward message (hidden state ht​).

Three knobs control behavior each step: forget, input, and output gates. Each knob is a soft dial from 0 to 1 (not just on/off).

At each sentence (time step):

Forget gate — the eraser. The student first skims the notebook and decides what’s now irrelevant. The forget knob turns the eraser up or down. High forget → old, useless notes are crossed out; low forget → keep most of the past.

Input gate — what to write. From the new sentence, the student proposes candidate notes (potential new facts). The input knob decides how much of that to actually write. If the sentence is crucial, the knob opens; if it’s fluff, it stays mostly closed.

Update the notebook — additive memory. The new notebook equals kept old notes plus newly written notes. Because it’s an add (keep + write), important information can pass forward for a long time without being repeatedly reprocessed — this is why LSTMs handle long context better than vanilla RNNs.

Output gate — what to say out loud. The student decides how much of the notebook to reveal in the quick whisper. The output knob controls that: wide open when a summary is needed, nearly closed when it’s better to stay quiet.

Why this beats a plain RNN. A vanilla RNN rewrites its memory every step; early facts fade as new ones overwrite them. The LSTM’s notebook + eraser + write dials let it carry key facts forward for many steps, purge what’s stale, and expose only what’s needed — so vital information doesn’t vanish.

How learning fits. After the student answers, the teacher compares it with the key and gives feedback. Training uses that feedback to tune the three knobs for future steps:

If the student kept junk, the forget dial gets nudged higher next time those patterns appear.

If they missed a crucial clue, the input dial opens more when similar cues show up. If they overshared noisy details, the output dial closes a bit in those contexts.

Mental picture.

Notebook c_t (c subscript t)​: durable, running notes.

Summary h_t​ (h subscript t)​: what the class hears.

Forget: erase stale notes. Input: write new notes. Output: reveal the right amount. Together, they let the student remember what matters, for as long as it’s useful.

About the author: Dr. Debdarsan Niyogi is an AI researcher and writer exploring the crossroads of modern AI, cognitive science, and philosophy. His work aims to make complex ideas accessible and thought-provoking. Follow him for more articles that examine profound questions starting from ground zero.

Have questions? Write in the comment section.