LLM Series 08: LLM Architectures & Their Use Cases

In recent years, Large Language Models (LLMs) have transformed the way we interact with technology — powering everything from intelligent chatbots to document summarization tools, virtual assistants, content creation platforms, and even code generation systems. These models are trained on vast amounts of text data and are capable of understanding, generating, and transforming human language with remarkable fluency.

But not all LLMs are created the same. Under the hood, different LLMs are built using different architectures, and this structural variation has a significant impact on their capabilities.

a. Some models are designed to generate text one word at a time like a storyteller.

b. Others are built to deeply understand context from all directions like a detective piecing together clues.

c. Some specialize in translating text from one language to another, while others are experts at classifying entire documents in a single glance.

In this blog, we’ll dive into four key types of LLM architectures:

• Causal (Autoregressive) Models
• Masked Language Models (Bidirectional)
• Sequence-to-Sequence (Encoder-Decoder) Models

1. Causal Language Models (Autoregressive Models)

Causal Language Modelling, also known as autoregressive modelling, is a foundational architecture for many generative large language models. The term “causal” means that the model generates each token based only on the tokens that came before it — never on future tokens. This approach mimics how we typically construct sentences when speaking or writing: we generate one word at a time, based on what we’ve already said.

Formally, a causal language model estimates the probability of a sequence of tokens by factoring it as:

This means that the model learns to predict the next word xt​ given all the previous words in the sequence x1, x2,…, xt−1​.

where,

• t is timesteps
• n is total context length

How It Works

At the core of a causal LM is the Transformer decoder architecture. During training and inference, it uses causal (or masked) self-attention, where each token can only attend to previous tokens, not future ones.

This masking is what makes the attention causal:

• During training: The model is shown sequences where it learns to predict each token by seeing only the tokens that came before it.
• During generation: The model produces one token at a time, feeding its previous outputs back into itself to generate the next token.

For example:

• Input: “The quick brown”
• Model generates: “fox”
• Then it updates the input to: “The quick brown fox”
• And generates the next word, and so on.

Use Cases:

• Text generation (chatbots, creative writing)
• Code completion and generation
• Auto-suggest features
• Step-by-step reasoning and agent tasks
• Open-ended conversations

When to Use:

• When future context is unknown or not needed
• When generating coherent text incrementally
• For tasks requiring creative, diverse outputs
• For conversational AI applications

Avoid causal models if your task needs full context understanding from both directions (like sentence classification or masked token recovery or advanced text summarization)

Code Implementation

from transformers import GPT2LMHeadModel, GPT2Tokenizer
import torch

# Load pre-trained GPT-2 model and tokenizer
model_name = "gpt2"
tokenizer = GPT2Tokenizer.from_pretrained(model_name)
model = GPT2LMHeadModel.from_pretrained(model_name)

# Set the model to evaluation mode
model.eval()

# Provide a prompt for the model to continue
prompt = "Once upon a time in a quiet village,"
input_ids = tokenizer.encode(prompt, return_tensors="pt")

# Generate continuation
output_ids = model.generate(input_ids, max_length=50, num_return_sequences=1, do_sample=True)

# Decode the output
generated_text = tokenizer.decode(output_ids[0], skip_special_tokens=True)

print("Generated Text:\n")
print(generated_text)

Generated Text:

Once upon a time in a quiet village, there lived a young girl named Ella who loved to read stories about dragons, pirates, and magical lands. One day, while exploring the attic of her...

2. Masked Language Models (Bidirectional)

Masked Language Modeling is a bidirectional pretraining approach used by models like BERT (Bidirectional Encoder Representations from Transformers). Unlike causal models, which only look at past tokens, MLMs use the entire context (both left and right) to predict missing tokens in a sentence.

This enables a much deeper understanding of the input text — making MLMs ideal for classification, question answering, and token-level prediction tasks like Named Entity Recognition (NER).

How It Works

During training:

• Some tokens in the input are randomly replaced with a special [MASK] token.
• The model learns to predict the masked tokens using the surrounding context (both left and right).

For example:

Input: The cat sat on the [MASK].

Target: mat

• We input a sentence with the [MASK] token.
• MLM uses the surrounding words (both left and right) to predict what the masked word should be.
• This demonstrates bidirectional understanding, which makes BERT powerful for comprehension tasks.

The model is trained to fill in that missing word based on context. Unlike causal models, MLMs don’t generate text — they understand and interpret it.

Popular Models Using MLM are

• BERT (Base, Large)
• RoBERTa
• DistilBERT
• DeBERTa
• ALBERT

These models are encoders only (no decoder) and are typically fine-tuned for downstream NLP tasks like classification, NER, or QA.

When to Use Masked LMs?

Choose MLMs when:

• You need deep contextual understanding of the full input.
• Your task is classification, entity recognition, or question answering.
• You want high-quality embeddings for downstream tasks.

Avoid using MLMs for open-ended text generation — that’s the domain of causal models like GPT.

Code Implementation

from transformers import BertTokenizer, BertForMaskedLM
import torch

# Load pre-trained BERT and tokenizer
model_name = "bert-base-uncased"
tokenizer = BertTokenizer.from_pretrained(model_name)
model = BertForMaskedLM.from_pretrained(model_name)
model.eval()

# Input sentence with a masked token
text = "The capital of France is [MASK]."
inputs = tokenizer(text, return_tensors="pt")

# Predict the masked token
with torch.no_grad():
    outputs = model(**inputs)
    predictions = outputs.logits

# Get the index of [MASK] token
mask_token_index = (inputs.input_ids == tokenizer.mask_token_id)[0].nonzero(as_tuple=True)[0]

# Get top predicted token
predicted_token_id = predictions[0, mask_token_index].argmax(axis=-1)
predicted_token = tokenizer.decode(predicted_token_id)

print(f"Original: {text}")
print(f"Predicted: The capital of France is {predicted_token}.")

Original: The capital of France is [MASK].
Predicted: The capital of France is paris.

3. Sequence-to-Sequence (Encoder-Decoder) Models

Sequence-to-Sequence (Seq2Seq) models are a type of architecture designed to transform one sequence into another. This is incredibly useful for tasks where the input and output are both sequences but may differ in length, structure, or content.

• The encoder processes the entire input sequence and creates a contextual representation.
• The decoder takes that representation and generates a corresponding output sequence, one token at a time.

Originally used for machine translation, this architecture now powers a wide range of tasks, including summarization, question answering, and text rewriting.

How It Works

1. The encoder reads the entire input sequence and converts it into a set of hidden representations.
2. These hidden representations are passed to the decoder.
3. The decoder generates the output sequence one token at a time, attending to relevant parts of the encoder’s output at each step.
4. Example: Input: "Translate English to French: Hello, how are you?" Output: "Bonjour, comment ça va ?"

This approach allows for context-aware generation based on the entire input sequence.

How it is different from Causal LM?

Difference between Sequence-to-Sequence (Seq2Seq) and Causal Language Models (Causal LMs) is essential because while they both deal with sequences, their architecture, training objectives, and use cases are quite different.

Key Differences Explained

• Seq2Seq has two parts: the encoder encodes the entire input, and the decoder generates the output using information from the encoder. Causal LMs have one decoder-only block that generates the next token based solely on past tokens.
• Seq2Seq decoders use cross-attention to look at the entire input sequence while generating the output. Causal LMs are unidirectional, meaning they generate one token at a time with no access to future tokens.
• Use Seq2Seq for transformation tasks where the output differs in form/content from the input (e.g., translation, summarization).Use Causal LMs for generation tasks where you’re extending the input (e.g., stories, chats, code).

Popular Models Using Encoder-Decoder Architecture

• T5 (Text-To-Text Transfer Transformer)
• BART
• mBART (Multilingual BART)
• MarianMT (machine translation)
• FLAN-T5 (instruction tuning)

Use Cases

Encoder-decoder models are ideal for:

• You need to transform text from one form to another.
• Your task involves translation, summarization, or rewriting.
• You want to encode full input context and generate a structured output.

Avoid them if your task is simply next-token prediction or classification — use causal or masked models instead for those.

Code Implementation

from transformers import T5Tokenizer, T5ForConditionalGeneration

# Load model and tokenizer
model_name = "t5-small"
tokenizer = T5Tokenizer.from_pretrained(model_name)
model = T5ForConditionalGeneration.from_pretrained(model_name)

# Input text (summarization task)
input_text = "summarize: The Eiffel Tower is a wrought-iron lattice tower on the Champ de Mars in Paris, France. " \
             "It is named after the engineer Gustave Eiffel, whose company designed and built the tower."

# Tokenize and encode
input_ids = tokenizer.encode(input_text, return_tensors="pt")

# Generate summary
output_ids = model.generate(input_ids, max_length=40, num_beams=4, early_stopping=True)

# Decode and print summary
summary = tokenizer.decode(output_ids[0], skip_special_tokens=True)
print("Summary:", summary)

Summary: The Eiffel Tower is a wrought-iron tower in Paris named after Gustave Eiffel.

Choosing the Right Architecture

Conclusion

The architecture of an LLM fundamentally shapes its capabilities and optimal use cases. While causal models excel at generation tasks, masked language models provide deeper bidirectional understanding, and sequence-to-sequence models offer transformation capabilities. Matching the right architecture to your specific task requirements is essential for maximizing performance and efficiency in real-world applications.