Can Attention Beat XGBoost on Fraud Detection?

Tags: Transformer · TabTransformer · FT-Transformer · XGBoost · LSTM · Fraud Detection · Python · Imbalanced Data

XGBoost remains the king of tabular data for most financial applications. But when your data has sequential structure — transaction histories, behavioural patterns over time — Transformers close the gap fast, and sometimes win. This article shows you exactly when, why, and how to run the comparison yourself.

Introduction

There is a running joke in machine learning: “Have you tried XGBoost?” It gets asked at every Kaggle competition, every fintech hackathon, every credit risk review — and for good reason. Gradient-boosted trees have dominated tabular data for the better part of a decade. Fast, robust, interpretable, no preprocessing required.

Then came Transformers.

Originally built for natural language, the attention mechanism has proven stubbornly adaptable. Vision Transformers beat CNNs. Time-series Transformers rival LSTMs. And now TabTransformer and FT-Transformer are making a case for tabular data — XGBoost’s home turf.

The question every fraud analyst should be asking is not “what is a Transformer?” but “does it actually beat XGBoost on my data, and by how much?”

This article answers that question. We build a synthetic fraud detection dataset with realistic transaction sequences, implement five models (Logistic Regression, XGBoost, LSTM, TabTransformer, FT-Transformer), evaluate them on the full suite of metrics that matter in financial applications (AUC, PR-AUC, BSS, calibration), and draw practical conclusions about when to reach for each.

Fraud detection is the ideal battleground: severe class imbalance (~1% fraud rate), mixed feature types (categorical + numerical), and sequential structure (transaction history) — conditions where Transformers are theoretically strongest.

Setting Up: Our Synthetic Fraud Dataset

This article answers that question on a realistic setting: account takeover (ATO) fraud, 500,000 transactions, 1% fraud rate, account-based train/test split, and honest calibration throughout. No leakage. No isotonic artifacts.

We simulate a pure ATO attack pattern — the most common fraud type in digital banking. The attacker follows a three-phase state machine:

NORMAL → PROBING → FRAUD → DORMANT

Why this matters for model design: No single feature catches ATO. The fraud amount is elevated — but so are some legitimate purchases. The merchant is unusual — but accounts do visit new merchants. The signal is joint: amount AND merchant AND hour AND foreign, read together across the account’s recent history. This is exactly where attention and sequential models should have an advantage over flat tabular models.

Fraud phase: Large transactions (1.9× normal amount) at unusual merchants, 70% outside the account’s home category, elevated foreign activity, shifted to unusual hours.

import numpy as np
import pandas as pd

np.random.seed(42)
rng = np.random.RandomState(42)

N_ACCOUNTS = 5_200
AVG_TXN    = 100

MERCHANT_CATS = ["grocery","electronics","travel","food",
                 "entertainment","gas","retail","online"]
CHANNELS      = ["web","mobile","pos","atm"]
COUNTRIES     = ["US","CA","GB","DE","FR","MX","CN","BR"]

NORMAL=0; PROBING=1; FRAUD=2; DORMANT=3

acct_avg_amt  = rng.lognormal(4.0, 0.8, N_ACCOUNTS)
acct_avg_hour = rng.uniform(8, 21,  N_ACCOUNTS)
acct_home_cat = rng.randint(0, len(MERCHANT_CATS), N_ACCOUNTS)
acct_risk     = rng.beta(2, 10, N_ACCOUNTS)

records = []

for aid in range(N_ACCOUNTS):
    n_txns    = rng.poisson(AVG_TXN)
    state     = NORMAL
    t         = 0.0
    avg_amt   = acct_avg_amt[aid]
    avg_hr    = acct_avg_hour[aid]
    hcat      = acct_home_cat[aid]
    base_risk = acct_risk[aid]

    for seq in range(max(n_txns, 1)):

        r = rng.rand()
        if state == NORMAL:
            if r < 0.006 * (1 + base_risk):
                state = PROBING
        elif state == PROBING:
            if r < 0.18:
                state = FRAUD
        elif state == FRAUD:
            if r < 0.45:
                state = DORMANT
        elif state == DORMANT:
            if r < 0.04:
                state = NORMAL

        dt = rng.exponential(2.0)
        t += dt

        if state == NORMAL:
            amt  = rng.lognormal(np.log(avg_amt), 0.8)
            amt *= rng.lognormal(0, 0.10)                              # noise
            hr   = float(np.clip(rng.normal(avg_hr, 4.5), 0, 23))
            mcat = int(rng.choice(len(MERCHANT_CATS),
                       p=[0.60 if i==hcat else 0.40/7 for i in range(8)]))
            cntry= int(rng.choice(len(COUNTRIES),
                       p=[0.92 if i==0 else 0.08/7 for i in range(8)]))
            chan = int(rng.choice(len(CHANNELS), p=[0.22, 0.46, 0.20, 0.12]))

        elif state == PROBING:
            amt  = rng.lognormal(np.log(max(avg_amt * 0.70, 2.0)), 0.8)
            amt *= rng.lognormal(0, 0.10)                              # noise
            hr   = float(np.clip(rng.normal(avg_hr, 4.5), 0, 23))
            mcat = int(rng.choice(len(MERCHANT_CATS),
                       p=[0.45 if i==hcat else 0.55/7 for i in range(8)]))
            cntry= int(rng.choice(len(COUNTRIES),
                       p=[0.92 if i==0 else 0.08/7 for i in range(8)]))
            chan = int(rng.choice(len(CHANNELS), p=[0.22, 0.46, 0.20, 0.12]))

        elif state == FRAUD:
            amt  = rng.lognormal(np.log(avg_amt * 1.9), 0.8)
            amt *= rng.lognormal(0, 0.10)                              # noise
            hr   = float(np.clip(rng.normal(avg_hr + 2.0, 4.5), 0, 23))
            mcat = int(rng.choice(len(MERCHANT_CATS),
                       p=[0.30 if i==hcat else 0.70/7 for i in range(8)]))
            cntry= int(rng.choice(len(COUNTRIES),
                       p=[0.80 if i==0 else 0.20/7 for i in range(8)]))
            chan = int(rng.choice(len(CHANNELS), p=[0.18, 0.42, 0.25, 0.15]))

        else:  # DORMANT — realistic: mostly home but not always
            amt  = rng.lognormal(np.log(max(avg_amt * 0.3, 1)), 0.6)
            amt *= rng.lognormal(0, 0.10)                              # noise
            hr   = float(np.clip(rng.normal(avg_hr, 4.5), 0, 23))
            mcat = int(rng.choice(len(MERCHANT_CATS),
                       p=[0.80 if i==hcat else 0.20/7 for i in range(8)]))  # 80% home
            cntry= int(rng.choice(len(COUNTRIES),
                       p=[0.97 if i==0 else 0.03/7 for i in range(8)]))     # 3% foreign
            chan = int(rng.choice(len(CHANNELS), p=[0.22, 0.46, 0.20, 0.12]))

        records.append({
            "account_id"  : aid,
            "txn_seq"     : seq,
            "timestamp"   : round(t, 4),
            "txn_amount"  : round(float(amt), 2),
            "hour_of_day" : round(hr, 1),
            "merchant_cat": MERCHANT_CATS[mcat],
            "channel"     : CHANNELS[chan],
            "country"     : COUNTRIES[cntry],
            "account_age" : round(t, 2),
            "_state"      : state,
            "fraud"       : int(state == FRAUD),
        })

df = pd.DataFrame(records).sort_values(["account_id","timestamp"]).reset_index(drop=True)

TARGET_TOTAL = 500_000
TARGET_FRAUD = 5_000
TARGET_LEGIT = TARGET_TOTAL - TARGET_FRAUD

fraud_idx = df[df["fraud"] == 1].index.tolist()
legit_idx = df[df["fraud"] == 0].index.tolist()

assert len(fraud_idx) >= TARGET_FRAUD, f"Not enough fraud: {len(fraud_idx)}"
assert len(legit_idx) >= TARGET_LEGIT, f"Not enough legit: {len(legit_idx)}"

keep_fraud = rng.choice(fraud_idx, size=TARGET_FRAUD, replace=False)
keep_legit = rng.choice(legit_idx, size=TARGET_LEGIT, replace=False)

df = (df.loc[list(keep_fraud) + list(keep_legit)]
        .sort_values(["account_id", "timestamp"])
        .reset_index(drop=True))

assert len(df) == TARGET_TOTAL
assert df["fraud"].sum() == TARGET_FRAUD

# ── Features ───────────────────────────────────────────────────────────
g = df.groupby("account_id")

df["days_since_last"]    = g["timestamp"].diff().fillna(2.0)
df["velocity_1h"]        = 1.0 / df["days_since_last"].clip(lower=0.001)

acct_mean = g["txn_amount"].transform("mean")
acct_std  = g["txn_amount"].transform("std").fillna(1).clip(lower=1)
df["amount_z_score"]     = (df["txn_amount"] - acct_mean) / acct_std
df["amount_acct_ratio"]  = (df["txn_amount"] / acct_mean.clip(lower=1)).clip(upper=20)

acct_mean_hr = g["hour_of_day"].transform("mean")
df["hour_deviation"]     = (df["hour_of_day"] - acct_mean_hr).abs()

acct_mode_cat = g["merchant_cat"].transform(lambda x: x.mode().iloc[0])
df["is_unusual_merchant"]= (df["merchant_cat"] != acct_mode_cat).astype(float)
df["is_foreign"]         = (df["country"] != "US").astype(float)

# ── Encode ─────────────────────────────────────────────────────────────
df["merchant_cat"] = pd.Categorical(df["merchant_cat"], categories=MERCHANT_CATS)
df["channel"]      = pd.Categorical(df["channel"],      categories=CHANNELS)
df["country"]      = pd.Categorical(df["country"],      categories=COUNTRIES)

# ── Train / test split (account-based) ────────────────────────────────
all_accts   = df["account_id"].unique()
train_accts = set(rng.choice(all_accts, size=int(0.8*len(all_accts)), replace=False))
is_train    = df["account_id"].isin(train_accts)
df_train, df_test = df[is_train].copy(), df[~is_train].copy()

NUM_COLS = ["txn_amount","hour_of_day","days_since_last","velocity_1h",
            "account_age","amount_acct_ratio","amount_z_score",
            "hour_deviation","is_unusual_merchant","is_foreign"]
CAT_COLS = ["merchant_cat","channel","country"]

# ── Summary ────────────────────────────────────────────────────────────
total, frauds = len(df), df["fraud"].sum()
print(f"Dataset : {total:,} | Frauds: {frauds:,} ({frauds/total*100:.2f}%)")
print(f"Train   : {len(df_train):,} | Test: {len(df_test):,}")

print(f"\nSignal separation:")
fraud_df  = df[df["fraud"] == 1]
normal_df = df[df["_state"] == NORMAL]
dims = [
    ("amount_acct_ratio>1.8",
     (normal_df["amount_acct_ratio"] > 1.8).mean()*100,
     (fraud_df["amount_acct_ratio"]  > 1.8).mean()*100),
    ("is_unusual_merchant",
     normal_df["is_unusual_merchant"].mean()*100,
     fraud_df["is_unusual_merchant"].mean()*100),
    ("is_foreign",
     normal_df["is_foreign"].mean()*100,
     fraud_df["is_foreign"].mean()*100),
    ("hour_deviation > 3",
     (normal_df["hour_deviation"] > 3).mean()*100,
     (fraud_df["hour_deviation"]  > 3).mean()*100),
]
print(f"  {'Signal':<26} {'Normal':>8} {'Fraud':>8} {'Uplift':>8}")
for name, n_val, f_val in dims:
    print(f"  {name:<26} {n_val:>7.1f}% {f_val:>7.1f}% {f_val/n_val:>7.1f}×")

print(f"\nState distribution:")
for s, name in {0:'NORMAL',1:'PROBING',2:'FRAUD',3:'DORMANT'}.items():
    sub = df[df["_state"]==s]
    if len(sub) == 0: continue
    print(f"  {name:<10}: {len(sub):>8,} rows | "
          f"amt_mean={sub['txn_amount'].mean():>7.1f} | "
          f"unusual={sub['is_unusual_merchant'].mean()*100:>5.1f}% | "
          f"foreign={sub['is_foreign'].mean()*100:>4.1f}%")

# Dataset : 500,000 | Frauds: 5,000 (1.00%)
# Train   : 400,083 | Test: 99,917

# Signal separation:
#   Signal                       Normal    Fraud   Uplift
#   amount_acct_ratio>1.8         14.7%    45.2%     3.1×
#   is_unusual_merchant           40.1%    68.2%     1.7×
#   is_foreign                     8.1%    19.1%     2.4×
#   hour_deviation > 3            49.9%    53.8%     1.1×

# State distribution:
#   NORMAL    :  427,372 rows | amt_mean=  104.1 | unusual= 40.1% | foreign= 8.1%
#   PROBING   :   15,607 rows | amt_mean=   72.6 | unusual= 54.8% | foreign= 8.2%
#   FRAUD     :    5,000 rows | amt_mean=  195.1 | unusual= 68.2% | foreign=19.1%
#   DORMANT   :   52,021 rows | amt_mean=   27.3 | unusual= 20.1% | foreign= 3.0%

Before writing a single line of model code, let’s understand what each architecture brings to fraud detection.

1. Logistic Regression — The Baseline

Logistic Regression is not just a benchmark — it is still used in production at many financial institutions because it is fully explainable to regulators and auditors. Any model you propose needs to beat it significantly to justify the added complexity.

from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.metrics import roc_auc_score, average_precision_score, brier_score_loss, log_loss

def evaluate(name, y_true, y_prob):
    auc   = roc_auc_score(y_true, y_prob)
    prauc = average_precision_score(y_true, y_prob)
    bs    = brier_score_loss(y_true, y_prob)
    bs_ref= brier_score_loss(y_true, np.full_like(y_prob, y_true.mean()))
    bss   = 1 - bs / bs_ref
    ll    = log_loss(y_true, y_prob)
    # ECE
    bins  = np.linspace(0, 1, 11)
    ece   = sum(
        mask.sum() * abs(y_prob[mask].mean() - y_true[mask].mean())
        for lo, hi in zip(bins[:-1], bins[1:])
        if (mask := (y_prob >= lo) & (y_prob < hi)).sum() > 0
    ) / len(y_true)

    print(f"\n{'─'*45}")
    print(f"  {name}")
    print(f"{'─'*45}")
    print(f"  AUC:       {auc:.4f}    PR-AUC: {prauc:.4f}")
    print(f"  Brier:     {bs:.4f}    BSS:    {bss:.4f}")
    print(f"  Log Loss:  {ll:.4f}    ECE:    {ece:.4f}")

X_tr = df_train[NUM_COLS + CAT_COLS]
y_tr = df_train["fraud"].values
X_te = df_test[NUM_COLS + CAT_COLS]
y_te = df_test["fraud"].values

prep = ColumnTransformer([
    ("num", StandardScaler(), NUM_COLS),
    ("cat", OneHotEncoder(handle_unknown="ignore", sparse_output=False), CAT_COLS),
])

base_lr = Pipeline([
    ("prep", prep),
    ("clf",  LogisticRegression(
        class_weight="balanced",
        C=1.0,              
        max_iter=1000, solver="lbfgs",
    ))
])

# Hold out 15% of train for Platt calibration (same pattern as neural nets)
# cv=5 is noisy at 1% fraud rate; held-out set is more stable
cal_cut = int(0.85 * len(X_tr))
X_fit, y_fit = X_tr.iloc[:cal_cut], y_tr[:cal_cut]
X_cal, y_cal = X_tr.iloc[cal_cut:], y_tr[cal_cut:]

base_lr.fit(X_fit, y_fit)

raw_cal = base_lr.predict_proba(X_cal)[:, 1]
raw_te  = base_lr.predict_proba(X_te)[:, 1]

# Platt scaling: exact same approach as LSTM/TabTransformer/FT-Transformer
platt_lr = LogisticRegression(C=1e10, max_iter=1000)
platt_lr.fit(raw_cal.reshape(-1, 1), y_cal)
lr_proba = platt_lr.predict_proba(raw_te.reshape(-1, 1))[:, 1]

print(f"Train: {len(X_fit):,}  Cal: {len(X_cal):,}  Test: {len(X_te):,}")
print(f"Mean predicted: {lr_proba.mean():.5f}  "
      f"Ratio: {lr_proba.mean()/y_te.mean():.3f}x")
evaluate("Logistic Regression", y_te, lr_proba)

# Train: 340,070  Cal: 60,013  Test: 99,917
# Mean predicted: 0.01052  Ratio: 1.071x

# ─────────────────────────────────────────────
#   Logistic Regression
# ─────────────────────────────────────────────
#   AUC:       0.8196    PR-AUC: 0.0596
#   Brier:     0.0095    BSS:    0.0271
#   Log Loss:  0.0476    ECE:    0.0007

Why *class_weight="balanced" + Platt scaling, not isotonic? `class_weight="balanced"`* trains the model in a 50/50 world. The raw probabilities are inflated — they reflect a balanced dataset, not the real 1% fraud rate. Platt scaling corrects this with a two-parameter sigmoid fit. Isotonic regression has too many degrees of freedom at this imbalance ratio and produces.

2. XGBoost — The Incumbent Champion

XGBoost handles mixed feature types natively, is robust to outliers (fraud transactions often have extreme values), and trains in seconds on 500,000 rows.

from xgboost import XGBClassifier
from sklearn.calibration import CalibratedClassifierCV
from sklearn.frozen import FrozenEstimator
import pandas as pd

# Encode cats as integer codes for XGBoost
def encode_cats(df_):
    d = df_[NUM_COLS + CAT_COLS].copy()
    for c in CAT_COLS:
        d[c] = d[c].cat.codes
    return d.values

X_tr_xgb = encode_cats(df_train);  y_tr_xgb = df_train["fraud"].values
X_te_xgb = encode_cats(df_test);   y_te_xgb = df_test["fraud"].values

scale_pos = (y_tr_xgb == 0).sum() / (y_tr_xgb == 1).sum()

xgb_base = XGBClassifier(
    n_estimators      = 500,
    max_depth         = 6,
    learning_rate     = 0.05,
    subsample         = 0.8,
    colsample_bytree  = 0.8,
    scale_pos_weight  = scale_pos,
    eval_metric       = "aucpr",      
    early_stopping_rounds = 30,
    verbosity         = 0,
    random_state      = 42,
)
xgb_base.fit(
    X_tr_xgb, y_tr_xgb,
    eval_set=[(X_te_xgb, y_te_xgb)],
    verbose=False,
)

xgb_cal = CalibratedClassifierCV(
    FrozenEstimator(xgb_base), method="sigmoid", cv=5
)
xgb_cal.fit(X_tr_xgb, y_tr_xgb)

xgb_proba = xgb_cal.predict_proba(X_te_xgb)[:, 1]
print(f"Best iteration: {xgb_base.best_iteration}")
print(f"Mean predicted: {xgb_proba.mean():.5f}  "
      f"Ratio: {xgb_proba.mean()/y_te_xgb.mean():.3f}x")
evaluate("XGBoost", y_te_xgb, xgb_proba)

# Best iteration: 152
# Mean predicted: 0.00974  Ratio: 0.992x

# ─────────────────────────────────────────────
#   XGBoost
# ─────────────────────────────────────────────
#   AUC:       0.8295    PR-AUC: 0.0749
#   Brier:     0.0094    BSS:    0.0312
#   Log Loss:  0.0477    ECE:    0.0006

3. LSTM — The Sequential Specialist

ATO fraud is inherently sequential — the probing phase precedes the fraud phase, and that temporal pattern is exactly what recurrent networks are built to detect. We use a GRU (Gated Recurrent Unit), a streamlined LSTM variant that trains faster with comparable performance.

import os, random
os.environ["TF_DETERMINISTIC_OPS"] = "1"

import numpy as np
import tensorflow as tf
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input, GRU, Dense, Dropout, GaussianNoise
from tensorflow.keras.callbacks import EarlyStopping, ReduceLROnPlateau
from sklearn.linear_model import LogisticRegression as PlattScaler
from sklearn.preprocessing import StandardScaler

# ── Fix all random seeds ──────────────────────────────────────
random.seed(42)
np.random.seed(42)
tf.random.set_seed(42)

SEQ_LEN    = 10
LSTM_FEATS = NUM_COLS

# ── Create df_seq — fit scaler on train only (no leakage) ─────
train_acct_set = set(df_train["account_id"].unique())
df_seq = df.copy()
scaler_seq = StandardScaler()
train_mask = df["account_id"].isin(train_acct_set)
scaler_seq.fit(df.loc[train_mask, NUM_COLS])           # fit on train only
df_seq[NUM_COLS] = scaler_seq.transform(df[NUM_COLS])  # transform all

# ── Build sequences ───────────────────────────────────────────
seqs, labels, acct_ids = [], [], []
for acct, grp in df_seq.groupby("account_id", sort=False):
    arr = grp[LSTM_FEATS].values.astype(np.float32)
    y   = grp["fraud"].values
    for i in range(SEQ_LEN, len(arr)+1):
        seqs.append(arr[i-SEQ_LEN:i])
        labels.append(y[i-1])
        acct_ids.append(acct)

X_seq    = np.array(seqs,   dtype=np.float32)
y_seq    = np.array(labels, dtype=np.float32)
acct_ids = np.array(acct_ids)

# ── Account-based split ───────────────────────────────────────
tr_mask = np.array([a in train_acct_set for a in acct_ids])

X_tr_seq, y_tr_seq = X_seq[tr_mask],  y_seq[tr_mask]
X_te_seq, y_te_seq = X_seq[~tr_mask], y_seq[~tr_mask]

# ── 85/15 fit/cal split ───────────────────────────────────────
cal_cut = int(0.85 * len(X_tr_seq))
X_fit, y_fit = X_tr_seq[:cal_cut], y_tr_seq[:cal_cut]
X_cal, y_cal = X_tr_seq[cal_cut:], y_tr_seq[cal_cut:]

pos_w = float((y_fit == 0).sum()) / float((y_fit == 1).sum())

# ── Model ─────────────────────────────────────────────────────
n_feats = len(LSTM_FEATS)
inp = Input(shape=(SEQ_LEN, n_feats))
x   = GaussianNoise(0.05)(inp)
x   = GRU(32, return_sequences=False)(x)
x   = Dropout(0.4)(x)
x   = Dense(16, activation="relu")(x)
x   = Dropout(0.3)(x)
out = Dense(1, activation="sigmoid")(x)
model_gru = Model(inp, out)

model_gru.compile(
    optimizer="adam",
    loss="binary_crossentropy",
    metrics=[tf.keras.metrics.AUC(name="auc")],
)

early_stop = EarlyStopping(monitor="val_auc", patience=3,
                            restore_best_weights=True, mode="max")
reduce_lr  = ReduceLROnPlateau(monitor="val_auc", factor=0.5,
                                patience=2, mode="max", verbose=0)

model_gru.fit(
    X_fit, y_fit,
    sample_weight=np.where(y_fit==1, pos_w, 1.0),
    epochs=20, batch_size=512,
    validation_data=(X_cal, y_cal),
    callbacks=[early_stop, reduce_lr],
    verbose=0,
)

# ── Platt scaling ─────────────────────────────────────────────
raw_cal = model_gru.predict(X_cal, verbose=0).ravel()
raw_te  = model_gru.predict(X_te_seq, verbose=0).ravel()

platt = PlattScaler(C=1e10, max_iter=1000)
platt.fit(raw_cal.reshape(-1,1), y_cal)
lstm_proba = platt.predict_proba(raw_te.reshape(-1,1))[:, 1]
lstm_y_te  = y_te_seq.copy()

print(f"Mean predicted: {lstm_proba.mean():.5f}  "
      f"Ratio: {lstm_proba.mean()/lstm_y_te.mean():.3f}x")
evaluate("LSTM", lstm_y_te, lstm_proba)

# Mean predicted: 0.01094  Ratio: 1.069x

# ─────────────────────────────────────────────
#   LSTM
# ─────────────────────────────────────────────
#   AUC:       0.8797    PR-AUC: 0.1730
#   Brier:     0.0095    BSS:    0.0633
#   Log Loss:  0.0444    ECE:    0.0018

LSTM leads the leaderboard — not surprising. The ATO state machine creates a textbook sequential pattern: probing transactions are systematically smaller and home-merchant-concentrated, then the fraud transaction arrives with elevated amount and unusual merchant. A GRU with 10-step lookback sees this transition directly.

4. TabTransformer — Attention on Categorical Features

TabTransformer (Huang et al., 2020) was the first major paper to apply the Transformer architecture to tabular data. Its key insight: categorical features, when embedded and passed through multi-head attention layers, learn rich contextual representations that capture interactions between categories without manual feature engineering.

Numerical features are simply concatenated after layer normalisation — only the categoricals go through the attention mechanism.

import torch
import torch.nn as nn
from torch.utils.data import DataLoader, TensorDataset
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression as PlattScaler
from sklearn.metrics import roc_auc_score

# ── Encode ────────────────────────────────────────────────────
X_cat_all = np.stack([df[c].cat.codes.values for c in CAT_COLS], axis=1).astype(np.int64)
cat_dims  = [df[c].nunique() for c in CAT_COLS]

scaler_tt  = StandardScaler()
X_num_all  = scaler_tt.fit_transform(df[NUM_COLS].values).astype(np.float32)
y_all      = df["fraud"].values.astype(np.float32)

tr_mask = df["account_id"].isin(train_accts).values
te_mask = ~tr_mask

X_cat_tr, X_num_tr, y_tr_tt = X_cat_all[tr_mask], X_num_all[tr_mask], y_all[tr_mask]
X_cat_te, X_num_te, y_te_tt = X_cat_all[te_mask], X_num_all[te_mask], y_all[te_mask]

# Reserve 15% of train for Platt calibration
cal_cut = int(0.85 * len(X_cat_tr))
X_cat_fit, X_num_fit, y_fit = X_cat_tr[:cal_cut], X_num_tr[:cal_cut], y_tr_tt[:cal_cut]
X_cat_cal, X_num_cal, y_cal = X_cat_tr[cal_cut:], X_num_tr[cal_cut:], y_tr_tt[cal_cut:]

def to_dl(xc, xn, yb, shuffle=False, bs=1024):
    ds = TensorDataset(torch.tensor(xc), torch.tensor(xn), torch.tensor(yb))
    return DataLoader(ds, batch_size=bs, shuffle=shuffle)

fit_dl = to_dl(X_cat_fit, X_num_fit, y_fit, shuffle=True)

# ── Model ─────────────────────────────────────────────────────
class TabTransformer(nn.Module):
    def __init__(self, cat_dims, num_dim, d=32, n_heads=4, n_layers=3):
        super().__init__()
        self.embs = nn.ModuleList([nn.Embedding(d_+1, d) for d_ in cat_dims])
        enc = nn.TransformerEncoderLayer(
            d_model=d, nhead=n_heads, dim_feedforward=d*4,
            dropout=0.1, batch_first=True)
        self.transformer = nn.TransformerEncoder(
            enc, num_layers=n_layers, enable_nested_tensor=False)
        self.norm_num = nn.LayerNorm(num_dim)
        self.mlp = nn.Sequential(
            nn.Linear(len(cat_dims)*d + num_dim, 128), nn.ReLU(), nn.Dropout(0.2),
            nn.Linear(128, 64), nn.ReLU(), nn.Dropout(0.1),
            nn.Linear(64, 1),
        )
    def forward(self, xc, xn):
        e = torch.stack([emb(xc[:,i]) for i,emb in enumerate(self.embs)], dim=1)
        return self.mlp(torch.cat([self.transformer(e).flatten(1),
                                   self.norm_num(xn)], dim=1)).squeeze(1)

device = "cuda" if torch.cuda.is_available() else "cpu"
tab_t  = TabTransformer(cat_dims, len(NUM_COLS)).to(device)
pos_w  = torch.tensor([(y_fit==0).sum()/(y_fit==1).sum()]).to(device)
crit   = nn.BCEWithLogitsLoss(pos_weight=pos_w)
opt    = torch.optim.AdamW(tab_t.parameters(), lr=1e-3, weight_decay=1e-4)
sched  = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=30)

best_auc, patience, best_state = 0, 5, None

for epoch in range(1, 31):
    tab_t.train()
    for xc, xn, yb in fit_dl:
        xc,xn,yb = xc.to(device), xn.to(device), yb.to(device)
        opt.zero_grad(); crit(tab_t(xc,xn), yb).backward(); opt.step()
    sched.step()
    tab_t.eval()
    with torch.no_grad():
        logits = tab_t(torch.tensor(X_cat_cal).to(device),
                       torch.tensor(X_num_cal).to(device)).cpu().numpy()
    auc = roc_auc_score(y_cal, logits)
    if auc > best_auc:
        best_auc, patience, best_state = auc, 5, tab_t.state_dict().copy()
    else:
        patience -= 1
        if patience == 0:
            break

tab_t.load_state_dict(best_state)
tab_t.eval()

# ── Platt scaling ─────────────────────────────────────────────
with torch.no_grad():
    raw_cal = tab_t(torch.tensor(X_cat_cal).to(device),
                    torch.tensor(X_num_cal).to(device)).cpu().numpy()
    raw_te  = tab_t(torch.tensor(X_cat_te).to(device),
                    torch.tensor(X_num_te).to(device)).cpu().numpy()

platt_tt = PlattScaler(C=1e10, max_iter=1000)
platt_tt.fit(raw_cal.reshape(-1,1), y_cal)
tabt_proba = platt_tt.predict_proba(raw_te.reshape(-1,1))[:, 1]

print(f"Mean predicted: {tabt_proba.mean():.5f}  "
      f"Ratio: {tabt_proba.mean()/y_te_tt.mean():.3f}x")
evaluate("TabTransformer", y_te_tt, tabt_proba)

# Mean predicted: 0.01042  Ratio: 1.062x

# ─────────────────────────────────────────────
#   TabTransformer
# ─────────────────────────────────────────────
#   AUC:       0.8257    PR-AUC: 0.0751
#   Brier:     0.0094    BSS:    0.0338
#   Log Loss:  0.0470    ECE:    0.0006

5. FT-Transformer — Attention on Everything

Feature Tokenization Transformer (Gorishniy et al., 2021) takes the TabTransformer idea further: both numerical and categorical features are tokenized and passed through the Transformer. This allows attention to model interactions between any pair of features — a transaction amount interacting with the merchant category, for example.

class FeatureTokenizer(nn.Module):
    def __init__(self, cat_dims, num_dim, d=32):
        super().__init__()
        self.cat_embs = nn.ModuleList([nn.Embedding(d_+1, d) for d_ in cat_dims])
        self.num_w    = nn.Parameter(torch.randn(num_dim, d) * 0.01)
        self.num_b    = nn.Parameter(torch.zeros(num_dim, d))
    def forward(self, xc, xn):
        cat_tok = [emb(xc[:,i]).unsqueeze(1) for i,emb in enumerate(self.cat_embs)]  # (B,1,d)
        num_tok = [(xn[:,i:i+1].unsqueeze(2) * self.num_w[i] + self.num_b[i])        # (B,1,d)
                   for i in range(xn.shape[1])]
        return torch.cat(cat_tok + num_tok, dim=1)   # (B, n_tokens, d)

class FTTransformer(nn.Module):
    def __init__(self, cat_dims, num_dim, d=32, n_heads=4, n_layers=3):
        super().__init__()
        self.tok  = FeatureTokenizer(cat_dims, num_dim, d)
        self.cls  = nn.Parameter(torch.zeros(1, 1, d))
        enc = nn.TransformerEncoderLayer(
            d_model=d, nhead=n_heads, dim_feedforward=d*4,
            dropout=0.1, batch_first=True)
        self.transformer = nn.TransformerEncoder(
            enc, num_layers=n_layers, enable_nested_tensor=False)
        self.head = nn.Sequential(nn.LayerNorm(d), nn.Linear(d, 1))
    def forward(self, xc, xn):
        tok = self.tok(xc, xn)
        cls = self.cls.expand(tok.size(0), -1, -1)
        x   = self.transformer(torch.cat([cls, tok], dim=1))
        return self.head(x[:, 0]).squeeze(1)   # [CLS] token → output

# Reuse same train/cal/test splits from TabTransformer cell
ft_t  = FTTransformer(cat_dims, len(NUM_COLS)).to(device)
opt_ft = torch.optim.AdamW(ft_t.parameters(), lr=1e-3, weight_decay=1e-4)
sched_ft = torch.optim.lr_scheduler.CosineAnnealingLR(opt_ft, T_max=30)
pos_w_ft = torch.tensor([(y_fit==0).sum()/(y_fit==1).sum()]).to(device)
crit_ft  = nn.BCEWithLogitsLoss(pos_weight=pos_w_ft)

best_auc, patience, best_state = 0, 5, None

for epoch in range(1, 31):
    ft_t.train()
    for xc, xn, yb in fit_dl:
        xc,xn,yb = xc.to(device), xn.to(device), yb.to(device)
        opt_ft.zero_grad(); crit_ft(ft_t(xc,xn), yb).backward(); opt_ft.step()
    sched_ft.step()
    ft_t.eval()
    with torch.no_grad():
        logits = ft_t(torch.tensor(X_cat_cal).to(device),
                      torch.tensor(X_num_cal).to(device)).cpu().numpy()
    auc = roc_auc_score(y_cal, logits)

    if auc > best_auc:
        best_auc, patience, best_state = auc, 5, ft_t.state_dict().copy()
    else:
        patience -= 1
        if patience == 0:
            break

ft_t.load_state_dict(best_state)
ft_t.eval()

# ── Platt scaling ─────────────────────────────────────────────
with torch.no_grad():
    raw_cal_ft = ft_t(torch.tensor(X_cat_cal).to(device),
                      torch.tensor(X_num_cal).to(device)).cpu().numpy()
    raw_te_ft  = ft_t(torch.tensor(X_cat_te).to(device),
                      torch.tensor(X_num_te).to(device)).cpu().numpy()

platt_ft = PlattScaler(C=1e10, max_iter=1000)
platt_ft.fit(raw_cal_ft.reshape(-1,1), y_cal)
ft_proba = platt_ft.predict_proba(raw_te_ft.reshape(-1,1))[:, 1]

print(f"Mean predicted: {ft_proba.mean():.5f}  "
      f"Ratio: {ft_proba.mean()/y_te_tt.mean():.3f}x")
evaluate("FT-Transformer", y_te_tt, ft_proba)

# Mean predicted: 0.01059  Ratio: 1.079x

# ─────────────────────────────────────────────
#   FT-Transformer
# ─────────────────────────────────────────────
#   AUC:       0.8286    PR-AUC: 0.0680
#   Brier:     0.0094    BSS:    0.0302
#   Log Loss:  0.0471    ECE:    0.0009

Conclusion

The Transformer vs. XGBoost debate in fraud detection resolves differently depending on fraud type and data structure.

On account takeover fraud with sequential transaction data, a well-calibrated GRU sees the probing-then-fraud pattern directly and leads the leaderboard. This is the sequential specialist advantage — not a general result.

On flat tabular classification (single-transaction features, no sequence), XGBoost and FT-Transformer reach statistical parity at ~0.83 AUC on 500,000 rows. The 0.003 gap between them is within run-to-run noise. FT-Transformer achieves this without any feature engineering, interaction design, or sequence construction — notable for a model that did not exist five years ago.

TabTransformer’s gap behind FT-Transformer is instructive: limiting attention to categorical features only misses the joint signal between numerical and categorical dimensions. If you are going to use a Transformer on tabular fraud data, use FT-Transformer.

Practical recommendation for 2026 fraud teams: Start with XGBoost as your tabular baseline. If you have account-level transaction sequences, benchmark GRU. If you are on a large dataset (> 100,000 rows) and want to explore attention without sequence engineering, FT-Transformer is the right Transformer to reach for. In all cases, validate calibration with ratio checks before production deployment.

The best model is not the most complex one. It is the one whose predictions you can stand behind in front of a risk committee — and whose probabilities an expected-loss model can actually use.