A Complete Mental Model for Generative AI — Part 1

Generative AI is evolving rapidly. Even experienced learners often struggle to organize the terminology around it. You encounter terms like Transformers, GPT, Diffusion Models, GANs, VAEs, Autoregressive Models, Flow Matching, NeRFs, Latent Space across articles, research papers, YouTube videos and Reddit discussions. At first, you might assume they all belong to the same category of ideas.

But they don’t.

Some of these terms describe neural network architectures, while others describe generation mechanisms, training objectives, conditioning strategies or even ways of representing data such as 3D scenes. The problem is that most explanations focus on individual concepts in isolation rather than showing how they actually connect. That is why most beginners and even some intermediate learners eventually ask the same question:

“What category do these things actually belong to?”

This article aims to answer that question by building a complete mental model of generative AI from the ground up. Instead of treating concepts like Transformers, Diffusion, GANs and NeRFs as isolated ideas, we will organize them into a structured hierarchy and then follow their historical evolution from early probabilistic models to modern multimodal foundation models.

By the end, you will not only understand what these terms mean individually, but also how they fit together into the broader landscape of modern generative AI.

The Mental Model: Five Components of Generative AI

Here is the core idea. To understand generative AI clearly, we need to stop thinking in terms of isolated models like “Transformers”, “GANs” or “Diffusion” and start thinking in terms of a set of components that every system is built from.

Almost every modern generative AI system you are familiar with like ChatGPT, DALL-E, Stable Diffusion, Sora can be fully understood through these five fundamental components.

1. Representation — How the AI “sees” data; as text tokens, pixels, or 3D space.
2. Architecture — The structure of the model; like CNNs, Transformers, or other network designs.
3. Generation Paradigm — How it creates new things (like predicting the next word or clearing up noise).
4. Training Objective — What the model is trying to optimize while learning.
5. Conditioning — How we guide the model; like text prompts, images, or other inputs.

There is an important point to highlight at the very beginning. These are largely separate design choices you can make when building a generative AI system rather than a fixed stack that data passes through. They aren’t perfectly independent. Some choices make certain others more natural. But treating them as distinct dimensions is the key to keeping the landscape organized in your head.

This first part of the blog series covers the first two components, Representation and Architecture. The rest will be covered in upcoming parts.

1. Representation: The Hidden Language of Generative AI

Before generating anything, a model needs a way to represent data. And before we talk about representation, we have to clearly understand the difference between discriminative and generative models.

Many traditional machine learning systems are discriminative. Their goal is to predict a label or category for a given input. For example, a cat-vs-dog classifier takes a pixel grid and outputs a probability. It only needs to learn some patterns that can differentiate a dog from a cat, but it doesn’t need to learn the full distribution of cat images.

But generative AI is different. To generate a cat image, the model must have some internal representation of “cat”, more specifically the textures, shapes, colors and poses that are inherent to a cat. This requires modeling the distributions of data, not just the boundary between classes.

Representations come in two broad families. The first is direct (or raw) representations that store data more or less as it naturally exists. The second is learned representations, where a model discovers a more compact and meaningful way to encode the data. Almost every modern system relies on the second family, but it is better to start with the first to get a complete picture. Different modalities (data formats) require different representation methods.

Direct Representations: Raw Data Formats

• Images → Pixel Grids: The most direct representation is raw pixel grids; high dimensional numerical arrays. A 512x512 RGB image has 786,432 numbers. Working directly in pixel space is computationally expensive, which is why most modern models avoid it.
• Audio → Waveforms or Spectrograms: A waveform is a long sequence of amplitude values over time. A spectrogram converts audio into a 2D image of frequency vs time, which is easier for models to process. WaveNet (2016) works directly on raw waveforms, but modern audio generation models operate on spectrograms or compressed audio tokens.
• Text → Tokens: Tokens are small units that may correspond to whole words, subwords, punctuation marks or character fragments. For example, in OpenAI’s o200k_base tokenizer (used in GPT-5), the phrase “Generative AI” is broken into [“Gener”, “ative”, “ AI”]. Each token is mapped to an integer ID and that ID is used to look up a vector in a learned vector table. The model then processes these learned vectors called embeddings which actually belong to the second family.

Learned Representations: Compressed and Meaningful

Raw representations are simple but not efficient. So, modern generative systems rarely work on raw data directly. Instead, they learn a compact encoding that keeps what matters and discards the rest.

• The latent Space — Imagine compressing every possible image of a human face into a map where similar faces are placed close together. Moving in one direction on the map makes faces older while another direction changes hair color. This continuous navigable space of compressed representations is called a latent space. One common way to construct a latent space is using an encoder-decoder architecture. In practice, an encoder compresses a high dimensional input (X) into a lower dimensional compressed representation (Z), while a decoder reconstructs the original data from the latent representation. Many modern generative systems, especially image and video generation models operate in latent spaces rather than directly in raw pixel space. This is why Stable Diffusion which performs diffusion in the latent space instead of raw pixels, is called a “Latent Diffusion Model”. Working in latent space makes generation significantly faster while also allowing the model to focus on higher-level semantic concepts rather than individual pixels.

• Embeddings — A particularly important form of latent representation is the embedding. These are vectors that capture semantic meaning by placing related concepts close together in the learned space. This is the same idea we met briefly in the text tokens section above. Modern systems such as CLIP, GPT and multimodal foundation models rely heavily on embeddings to connect text, images, audio and other modalities because embeddings allow models to reason about similarity and meaning rather than raw symbols.

Discrete Image Tokens — Quantized Latent Space

Once continuous latent spaces became successful, researchers explored whether images could also be represented using discrete symbolic tokens similar to language.

VQ-VAE (Vector Quantized Variational Autoencoders, 2017) answers this. It trains an encoder that maps image regions to the nearest entry in a dictionary of “visual words”, a fixed-size table, learned during training, called the codebook. The image then becomes a short grid of these integer IDs, and the decoder can reconstruct a high-quality image from that token grid.

3D Scenes: Explicit and Implicit Representations

So far, we have discussed representations for text, images, and audio. But once generative AI moved toward full 3D scene understanding and generation, entirely new representations became necessary. This is one of the fastest moving areas in this field today.

• Voxels are 3D equivalent of pixels, a grid of cubes where each cell stores density or color value. While this concept is very easy to understand, it scales very poorly due to cubic complexity O(N³). A 512x512x512 voxel grid has 134 million cells making high resolution 3D generation prohibitively expensive.
• Point Clouds represent 3D scene as a collection of (x, y, z) coordinate points each carrying an optional color or surface information. While they are lightweight and flexible, they lack surface continuity. They describe where the objects are, but not what the surface looks like.
• Neural Radiance Fields (NeRFs) introduced in 2020, took a fundamentally different approach. Instead of storing 3D structure explicitly, NeRF trains a neural network to represent the scene implicitly. You feed the network a specific 3D coordinate and a viewing direction, then it outputs the color and density of that point. The 3D scene is stored inside weights of the neural network rather than on a physical grid or list of points. While this achieves exceptional visual quality, training is slow and rendering is computationally expensive because calculating a pixel requires hundreds of network queries at different depths along its line of sight.
• 3D Gaussian Splatting introduced in 2023 addressed NeRF’s speed problem with an explicit representation. Instead of a neural network, the scene is stored as millions of small 3D Gaussians each with a position, size, orientation, color and opacity. Rendering works by projecting these onto the camera plane and blending them. Because no neural network inference is needed at rendering time, Gaussian Splatting renders scenes in real time while achieving visual quality comparable to NeRF. This has quickly become one of the most important real-time 3D scene representations.

2. Architecture: What Is the Model’s Internal Structure?

Architecture answers the question “What kind of computational machine is doing the processing?”

If representation defines how information is stored, architecture defines how information flows and is transformed.

An architecture is the structural design of the neural network, the building block that defines how information flows through the system, patterns are extracted and relationships are learned.

Different architectures exist because different modalities (data types) require different processing methods. For example:

• Images contain spatial relationships
• Language contains sequential dependencies.
• Videos contain both spatial and temporal structure.

To keep these architectures straight, it helps to split them into two families. The first is the set of core building blocks; primitives, where each one defines a fundamental mechanism for how information is mixed. The second is composite architectures; larger structures built by arranging those primitives for a specific modality or role.

Core building blocks: The Primitives

Each architecture in this group defines a distinct mechanism for processing information and a fundamental way of mixing it across a sequence, a spatial grid, or a graph. These are the true peers; the later architectures are built out of them.

2.1 MLP (Multi-Layer Perceptron) Fully Connected Networks

The Multi-Layer Perceptron (MLP) is the most fundamental neural network architecture. It stacks layers of neurons where each neuron in one layer is connected to every neuron in the next layer, followed by a non-linear activation function.

Input Layer → [Linear + Activation] → [Linear + Activation] → … → Output Layer

MLPs are powerful function approximators and form the foundation of modern deep learning. However, due to issues like parameter inefficiency, complete lack of spatial awareness, inability of sequential understating, modern generative models rarely use pure MLP as their primary backbone. Instead, MLP style “feed-forward” blocks appear inside more complex architectures.

2.2 CNN (Convolutional Neural Network)

CNNs were designed to handle spatial structure more efficiently than fully connected networks. Instead of connecting every input to every neuron, CNN applies small learnable filters called “Convolution Kernels” that slides across the input and detects local patterns.

Three key properties make CNNs more powerful for visual data.

• Locality — each filter looks at small neighborhood at a time, not whole image allowing the network can identify local patterns like edges and textures.
• Translation equivariance— Since the same filter is reused across the entire image a feature can still be recognized even if it appears in a different position.
• Hierarchical feature extraction — Early layers learn simple features like edges while deeper layers learn more complex structures like shapes and objects.

Because of these properties, CNNs became the dominant architecture for image understanding, appearing as the backbone of many early generative models.

2.3 RNN (Recurrent Neural Network)

While CNNs are excelled at handling spatial data like images, they struggle with sequential data where order matters. In a sentence, the meaning of a word often depends on the words that came before it.

For example, consider these two sentences:

1. “I need to deposit money in the bank”
2. “The bank of the river is muddy”

A model can’t understand the word “bank” in isolation, it must be look at surrounding word sequence.

Recurrent Neural Networks (RNNs) were designed to model this sequential structure. They introduce a “hidden state” an internal memory loop that passes information from one step to the next. As the model reads a sentence word by word, it updates the memory, allowing the model retain information from earlier parts of the sequence.

Input₁ → Hidden State₁ Input₂ + Hidden State₁ → Hidden State₂ Input₃ + Hidden State₂ → Hidden State₃ …

because of this recurrent structure, RNNs became one of the first successful architectures for language modeling, machine translation, speech recognition and text generation.

However, as the sequences gets longer, a major limitation rises. Information from very earlier time steps (the beginning of a long paragraph), gradually became difficult to preserve making it challenging for RNNs to learn long-range dependencies. Also, since the processing is sequential, parallel training is not possible.

2.4 LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit)

The main weakness of standard RNNs was the “vanishing gradient problem” which caused them to forget information from the very beginning of long sequences.

Long Short-Term Memory (LSTM) networks, introduced in 1997, addressed this issue by introducing memory cells. The information flow in an LSTM is strictly controlled by three key mechanisms called gates. These gates decides whether to keep or throw away information.

1. Input Gate — decides what new information should be stored.
2. Forget Gate — decides what old information should be discarded.
3. Output Gate — decides what information should be exposed to the next step.

Gated recurrent Units (GRUs) introduced later provide a simpler alternative with fewer parameters while retaining the advantages of LSTM.

For nearly a decade, LSTMs and GRUs became the dominant architecture for language modeling, machine translation, speech recognition and sequence generation.

Still one limitation remained; their training was inherently sequential which limited the scalability of models.

2.5 Transformers: The Attention Revolution

In 2017, Google Researchers published a paper titled “Attention is All You Need” introducing the Transformer architecture. This completely revolutionized the entire generative AI field.

This architecture finally solved the parallelization problem of RNNs, LSTMs and GRUs through a mechanism called Self-Attention. Instead of processing sequence word-by-word from left to right, transformers allow every element in a sequence to directly interact with every other element simultaneously.

For example, when processing the sentence “The animal didn’t cross the street because it was too tired” the Self-Attention mechanism allows the word “it” can directly connect to “animal” even though there are several words between them. If the sentence ends with “because it is too wide”, the model will automatically shift its attention to connect “it” with “street”. This ability to model long-range relationships makes Transformers excellent for language understanding.

Also, because the architecture completely eliminates step-by-step sequential loops, the entire training process can be run in parallel across multiple GPU cores, unlocking massive scalability.

Today, the Transformer serves as the foundational backbone architecture for almost all the state-of-the-art foundation models including large language models (LLMs) like GPT.

2.6 SSM (State Space Models)

For several years, Transformers dominated sequence modeling tasks such as language understanding and text generation. However, as models began to process longer contexts a new challenge emerged. The computational cost and memory requirements of self-attention grow rapidly with sequence length (O(N²)) making very long context increasingly expensive.

Researchers introduced State Space models (SSMs) as an alternative approach for handling long sequences more efficiently. The core idea is surprisingly similar to the original motivation behind RNNs. Instead of allowing every token to interact with every other token through self-attention, an SSMs maintain a compact internal state that continuously summarizes information from previous inputs.

Unlike traditional RNNs, modern State Space Models are designed so that this state update can be computed efficiently on modern hardware. This allows SSMs to combine efficient parallel training with computationally efficient inference.

One of the most influential examples is Mamba, introduced in 2023. Mamba demonstrated how State Space Models could be implemented efficiently on modern hardware while achieving performance competitive with Transformers on many sequence modeling tasks. Because its computational cost scales linearly with sequence length (O(N)), Mamba attracted significant attention as a potential solution for long-context processing.

2.7 GNN (Graph Neural Network)

Not all data is naturally organized as a grid or a sequence. Some data is best represented as a graph such as molecules (atoms joined by bonds), social networks, or road maps.

Graph Neural Networks (GNNs) are architectures designed specifically for this type of data. Their core mechanism is message passing, where each node repeatedly gathers information from its neighboring nodes and updates its own representation. Through multiple rounds of message passing, information can propagate across the graph and capture complex relationships between connected entities.

Like CNNs, RNNs and Transformers, GNNs define a specific way for information to flow through a neural network. The difference is that information is exchanged along graph connections rather than across a spatial grid or a sequence.

GNNs are less common in mainstream generative AI systems, but they play central role in scientific applications such as molecular generation, protein design and material discovery.

Composite Architectures: Built From the Primitives

Everything above is a primitive. Each defines a distinct mechanism for mixing information. The architectures in this group are different in kind. They are not new primitives, but rather strategic methods of arranging those foundational building blocks into larger structures.

2.7 The Encoder-Decoder & Decoder-Only Layouts: Macro-Shapes of Language

When we talk about Transformers as a primitive, we are talking about the self-attention mechanism. But how you arrange those attention blocks dictates what the model can actually do. In language processing, there are two dominant layouts built entirely from the Transformer primitive:

• Encoder-Decoder —The *encoder processes the full input sequence and converts it into a contextual representation that captures its meaning. The decoder then takes this representation and generates a new sequence conditioned on it. The core idea is the model first understands the input as a whole, then generates an output conditioned on that understanding. Input → ENCODER → latent summary → DECODER → Output *This design is fundamentally suited for sequence-to-sequence tasks, where the input and output are different modalities or languages. Examples: Google’s T5, OpenAI’s Whisper (for audio-to-text).
• Decoder-Only Layout —The decoder-only architecture uses a single stack of Transformer blocks with causal masking. Causal masking ensures that each token can only attend to previous tokens in the sequence, preventing access to future information. This enforces the model learns to predict the next token step by step. Token 1 → Token 2 → Token 3 → Token 4 (each token attends only to past tokens) Unlike encoder–decoder models, there is no separate encoding stage. The same network both conditions on context and generates output by continuing the sequence. Examples: The entire GPT family, Llama, Claude, and almost every modern conversational LLM.

2.8 U-Net: A Multi-Scale Encoder–Decoder Architecture

A U-Net, introduced in 2015 for biomedical image segmentation is not a new building block but a particular shape and a way of wiring existing blocks together. At its core, it follows a simple idea; compress information, then reconstruct it, but do both in a way that preserves detail.

The architecture has two symmetric paths;

• The encoder (down path) — gradually reduces spatial resolution while increasing feature depth. This is where the model learns what is in the image capturing global context, structure, and semantics.
• The decoder (up path) — then progressively restores resolution, reconstructing spatial detail to produce a full-size output. This is where the model learns where things are.

What makes U-Net powerful is the skip connections between these two paths. Instead of forcing the decoder to rely only on compressed representations, U-Net directly passes high-resolution features from the encoder to the corresponding decoder layers.

A U-Net can be built from CNN blocks (the classic version) or even from modern Transformer blocks. In generative AI, it became the standard denoising backbone of diffusion models such as Stable Diffusion, which is why you will see it mentioned constantly alongside diffusion (in part 2).

2.9 ViT (Vision Transformer)

The Vision Transformer, introduced in 2020, answers a simple question: can the Transformer, designed for text, also process images? The key idea is to turn an image into a sequence of tokens.

A ViT splits an image into a grid of fixed-size patches (for example, 16×16 pixels), flattens each patch, and linearly projects it into a vector called a patch embedding. Position information is added so the model knows where each patch came from. From that point on, the image is simply a sequence of patch tokens fed into an ordinary Transformer encoder.

This is the important takeaway: a ViT introduces no new mechanism. It is the same self-attention from section 2.5, pointed at image patches instead of word tokens. Given enough training data, ViTs match or surpass CNNs on many vision tasks, which is how Transformers spread from language into vision.

In Part 2 we will explore how these architectures actually generate new content through different generation paradigms, conditioning mechanisms, and training objectives. There we’ll cover the terms still left open; GANs, Diffusion, Autoregressive Models, and Flow Matching.