BERT Complete Architecture Guide: From Embeddings to Fine Tuning for Text Classification

BERT = Bidirectional Encoder Representations from Transformers

BERT is a language model that reads a full sentence from both left and right side, so it understands the meaning of a word using the full context.

Why BERT exists

Before BERT (2018), NLP models read text in one direction only — left to right (GPT) or right to left. They never saw the full context of a word simultaneously.

The problem:

"I went to the bank to deposit money"
"I sat by the river bank"

A left-to-right model reading “bank” hasn’t seen “deposit” or “river” yet — it guesses context too early.

Bert Model

BERT solved this by reading the entire sentence at once in both directions simultaneously — hence Bidirectional.

This is the main power of BERT.

It does not only look at words. It looks at words + context + position + relationship with other words.

When to use BERT

Use BERT when your data is text.

Product review sentiment Seller complaint classification Delivery issue detection Customer support ticket routing Fake review detection Product quality complaint detection Supplier feedback classification Return reason classification

The full architecture — every component

12 Transformer encoder layers
12 attention heads
768 hidden dimensions
110 million parameters
Maximum input length = 512 tokens

How BERT works — every layer explained

Text
 ↓
Tokenizer
 ↓
Token Embedding + Segment Embedding + Position Embedding
 ↓
Transformer Encoder Layer 1
 ↓
Transformer Encoder Layer 2
 ↓
...
 ↓
Transformer Encoder Layer 12
 ↓
Classification Head
 ↓
Prediction

1. Input layer — three embeddings summed

Every token gets three embeddings added together:

Final input vector = Token embedding + Segment embedding + Position embedding

Token embedding — what is the word (vocabulary of 30,000 wordpieces) Segment embedding — is this sentence A or B (for sentence-pair tasks)

Position embedding — where is this word (position 0, 1, 2… up to 512)

Each embedding is 768 dimensions. Sum them → one 768-dim vector per token.

BERT does not directly understand raw words.
It converts every word/token into numbers.
For every token, BERT creates 3 vectors:

What is the word?
"product"
"good"
"bad"
"delivery"
"refund"

Each token is converted into a vector.

product → [0.12, -0.45, 0.88, ...]
good    → [0.67, 0.21, -0.34, ...]

In BERT-base, each vector has:

768 numbers

so
"product" = 768-dimensional vector

2. Self-attention — the core mechanism

Every token attends to every other token simultaneously. For each attention head:

Q = X · Wq    # Query — "what am I looking for?"
K = X · Wk    # Key   — "what do I contain?"
V = X · Wv    # Value — "what do I return?"

Attention(Q,K,V) = softmax(Q · Kᵀ / √dk) · V

BERT-base uses 12 heads in parallel, each with dk=64 dims. Outputs concatenated → linear projection back to 768.

The √dk scaling prevents dot products from growing too large and saturating softmax.

Every word looks at every other word and decides which words are important for understanding its meaning.

Example

The delivery was not fast.

not

fast = positive
not fast = negative

Self-attention helps BERT understand this.

good ↔ not = strong relationship
good ↔ product = medium relationship
good ↔ the = weak relationship

Sentence A: Customer said product is damaged.
Sentence B: Seller says product was shipped safely.

BERT needs to know which token belongs to sentence A and which token belongs to sentence B.

For simple sentiment classification, usually we only have one sentence, so segment embedding is less important.

3. Feed-forward network — per token, independently

FFN(x) = GELU(x · W1 + b1) · W2 + b2

W1: 768 → 3072   (expand 4×)
W2: 3072 → 768   (compress back)

GELU activation (smoother than ReLU) — empirically better for language.

Where is the word in the sentence?

Because BERT sees all words together, it needs position information.

Dog bites man.
Man bites dog.

Same words, different meaning.

Position matters.

So BERT adds position numbers:

position 0
position 1
position 2
...

4. Pre-training tasks — why BERT is powerful

Masked Language Model (MLM):

• Randomly mask 15% of tokens with [MASK]
• Model must predict the original token
• Forces bidirectional context learning

Next Sentence Prediction (NSP):

• Feed two sentences, predict if B follows A
• Trains [CLS] token to capture sentence-level meaning

Final input vector

final vector = token meaning + sentence info + position info

"delivery" vector 
= token embedding 
+ segment embedding 
+ position embedding

Lazada working code product review classifier

This applies BERT to classify Lazada supplier/product feedback from O2O reviews or B2B complaint text:

transformers — Hugging Face library. Contains pre-trained BERT weights, tokenizer, and Trainer API. torch PyTorch. The tensor computation engine BERT runs on.

# Cell 1
%pip install transformers torch

import torch                                        # tensor math engine
import pandas as pd                                 # dataframe handling
import numpy as np                                  # array operations

from transformers import (
    AutoTokenizer,                                  # converts text → token IDs
    AutoModelForSequenceClassification,             # BERT + classification head
    TrainingArguments,                              # hyperparameter config object
    Trainer                                         # training loop engine
)
from torch.utils.data import Dataset                # base class for your dataset
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report

#AutoTokenizer — auto-detects the right tokenizer for any model name. For bert-base-multilingual-cased it loads the WordPiece tokenizer with 119,547 vocabulary tokens covering 104 languages including Thai.
#AutoModelForSequenceClassification — loads BERT's 12 transformer layers + adds a classification head on top:

# Cell 3 — sample lazada TH review data
# Replace with your actual review/complaint table from Delta
data = {
    'text': [
        "สินค้าหมดอายุก่อนกำหนด ไม่ควรวางขาย",          # expired product
        "ราคาถูก คุณภาพดี คุ้มค่ามาก",                    # good value
        "จัดส่งช้ามาก รอนานเกิน 2 สัปดาห์",               # slow delivery
        "พนักงานบริการดีมาก ช่วยเหลือดี",                  # good service
        "สินค้าแตกหักมาในกล่อง บรรจุภัณฑ์แย่มาก",         # damaged goods
        "ของสดคุณภาพดี สดมาก ราคาเหมาะสม",                # fresh produce
        "ไม่ตรงกับรูปที่โฆษณา หลอกลวงลูกค้า",              # misrepresentation
        "โปรโมชั่นดีมาก ประหยัดได้เยอะ",                   # good promo
    ],
    'label': [0, 1, 0, 1, 0, 1, 0, 1]  # 0=negative, 1=positive
}
df = pd.DataFrame(data)
print(f"Samples: {len(df)} | Positive: {df['label'].sum()} | Negative: {(df['label']==0).sum()}")

# Cell 4 — load multilingual BERT (handles Thai + English mixed text)
MODEL_NAME = "bert-base-multilingual-cased"   # supports Thai
# Alternative: "airesearch/wangchanberta-base-att-spm-uncased"  ← Thai-specific, better

tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)

# Test tokenization on Makro text
sample = "สินค้าหมดอายุ lazada branch Bangkok"
tokens = tokenizer(sample, return_tensors="pt")
print(f"Input IDs : {tokens['input_ids']}")
print(f"Token count: {tokens['input_ids'].shape[1]}")

# Cell 5 — dataset class
class LazadaReviewDataset(Dataset):
    def __init__(self, texts, labels, tokenizer, max_len=128):
        self.texts     = texts
        self.labels    = labels
        self.tokenizer = tokenizer
        self.max_len   = max_len

    def __len__(self):
        return len(self.texts)

    def __getitem__(self, idx):
        enc = self.tokenizer(
            self.texts[idx],
            max_length=self.max_len,
            padding='max_length',
            truncation=True,
            return_tensors='pt'
        )
        return {
            'input_ids':      enc['input_ids'].squeeze(),
            'attention_mask': enc['attention_mask'].squeeze(),
            'labels':         torch.tensor(self.labels[idx], dtype=torch.long)
        }

# Split
train_texts, val_texts, train_labels, val_labels = train_test_split(
    df['text'].tolist(), df['label'].tolist(),
    test_size=0.2, random_state=42, stratify=df['label']
)

train_dataset = LazadaReviewDataset(train_texts, train_labels, tokenizer)
val_dataset   = LazadaReviewDataset(val_texts,   val_labels,   tokenizer)

print(f"Train: {len(train_dataset)} | Val: {len(val_dataset)}")

# Cell 6 — load BERT with classification head
model = AutoModelForSequenceClassification.from_pretrained(
    MODEL_NAME,
    num_labels=2,           # negative / positive
    id2label={0:"negative", 1:"positive"},
    label2id={"negative":0, "positive":1}
)

# BERT has 110M params — freeze base, fine-tune only classifier
# Unfreeze last 2 layers + classifier for faster training
for name, param in model.named_parameters():
    if 'encoder.layer.10' in name or \
       'encoder.layer.11' in name or \
       'classifier' in name:
        param.requires_grad = True
    else:
        param.requires_grad = False

trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
total     = sum(p.numel() for p in model.parameters())
print(f"Trainable params: {trainable:,} / {total:,}  ({trainable/total:.1%})")

# Cell 7 — training
def compute_metrics(eval_pred):
    logits, labels = eval_pred
    preds = np.argmax(logits, axis=1)
    report = classification_report(labels, preds,
                                   target_names=['negative','positive'],
                                   output_dict=True)
    return {
        'accuracy':  report['accuracy'],
        'f1_neg':    report['negative']['f1-score'],
        'f1_pos':    report['positive']['f1-score'],
        'f1_macro':  report['macro avg']['f1-score'],
    }

training_args = TrainingArguments(
    output_dir            = '/tmp/makro_bert_reviews',
    num_train_epochs      = 5,
    per_device_train_batch_size = 8,
    per_device_eval_batch_size  = 8,
    learning_rate         = 2e-5,        # standard BERT fine-tune LR
    warmup_steps          = 10,
    weight_decay          = 0.01,
    evaluation_strategy   = 'epoch',
    save_strategy         = 'epoch',
    load_best_model_at_end= True,
    metric_for_best_model = 'f1_macro',
    logging_steps         = 5,
    report_to             = 'none'       # set to 'mlflow' to log to your experiment
)

trainer = Trainer(
    model           = model,
    args            = training_args,
    train_dataset   = train_dataset,
    eval_dataset    = val_dataset,
    compute_metrics = compute_metrics
)

trainer.train()

# Cell 8 — inference on new Makro text
def predict_sentiment(texts, model, tokenizer, threshold=0.5):
    model.eval()
    results = []

    for text in texts:
        enc = tokenizer(
            text,
            max_length=128,
            padding='max_length',
            truncation=True,
            return_tensors='pt'
        )
        with torch.no_grad():
            logits = model(**enc).logits
            probs  = torch.softmax(logits, dim=1).squeeze()

        results.append({
            'text':           text,
            'sentiment':      'positive' if probs[1] > threshold else 'negative',
            'confidence':     float(probs.max()),
            'prob_positive':  float(probs[1]),
            'prob_negative':  float(probs[0])
        })

    return pd.DataFrame(results)

# Test on new Makro product/supplier feedback
new_reviews = [
    "สินค้า Nestle คุณภาพดีมาก ส่งตรงเวลา",
    "supplier ส่งสินค้าไม่ครบ ขาดไป 3 รายการ",
    "ราคา Makro แพงกว่า Big C มาก",
    "Fresh food zone ดีขึ้นมากเลย สดสะอาด",
    "แอปใช้งานยาก สั่งของแล้วระบบค้าง"
]

results = predict_sentiment(new_reviews, model, tokenizer)
print(results[['text','sentiment','confidence']].to_string(index=False))

# Cell 9 — save model to MLflow + DBFS
import mlflow

mlflow.set_experiment("/Users/pvishnoi@lazada.com/ML_USE_CASE/bert_reviews")

with mlflow.start_run(run_name="bert_multilingual_makro_v1"):
    mlflow.log_params({
        'model_name':  MODEL_NAME,
        'epochs':      5,
        'lr':          2e-5,
        'max_len':     128,
        'num_labels':  2
    })

    # Log final metrics
    final_metrics = trainer.evaluate()
    mlflow.log_metrics({
        'val_f1_macro': final_metrics['eval_f1_macro'],
        'val_accuracy': final_metrics['eval_accuracy']
    })

    # Save model
    trainer.save_model('/dbfs/tmp/prem/bert_makro_reviews')
    mlflow.log_artifacts('/dbfs/tmp/prem/bert_makro_reviews', 'model')
    print(" Model saved to DBFS and MLflow")

Why BERT improves Makro sales — specifically

The key advantage over LightGBM for these tasks: BERT understands “สินค้าดี” and “good product” as the same sentiment your tabular model cannot do this.



