TabPFN 2.5: The Foundation Model That Beat XGBoost Without a Single Line of Tuning

8 min read · Machine Learning · Python · Tabular AI

I was skeptical.

When a colleague told me a pretrained transformer could outperform XGBoost on a medical classification task — with zero hyperparameter tuning — I did what any reasonable ML engineer would do: I pulled up a notebook and tried to prove him wrong.

Two hours later, I was writing this article instead.

TabPFN 2.5 achieved 98.24% accuracy and a 0.996 ROC-AUC on the Breast Cancer Wisconsin dataset — outperforming Decision Tree, Random Forest, and XGBoost — without a single epoch of training on my data.

This tutorial is the exact experiment I ran. By the end, you’ll understand not just how to use TabPFN, but why it works — and when to reach for it over the models you already know.

The Problem with How We Think About Tabular ML

For most of the last decade, the tabular machine learning workflow has looked roughly the same:

1. Load dataset
2. Spend hours on feature engineering
3. Run a grid search or Optuna trial
4. Pick the best XGBoost or LightGBM config
5. Deploy — and repeat from scratch for every new dataset

This approach works. But it has a ceiling baked in: every model starts from zero. It has no memory of the thousands of datasets it could have learned from. It has no intuition about what patterns matter across domains.

TabPFN takes a fundamentally different bet.

What Is TabPFN 2.5?

TabPFN — Tabular Prior-Data Fitted Network — is a transformer-based foundation model built specifically for tabular classification tasks. Unlike traditional ML models, it is not trained on your data. It is pretrained on millions of synthetic tabular datasets, allowing it to internalize how classification problems behave in general.

Think of it like this:

XGBoost learns your dataset. TabPFN has already learned what learning looks like — across thousands of problems. When you hand it your data, it isn’t starting from scratch; it’s pattern-matching against a rich prior.

This is the same philosophical shift that made GPT models so powerful for text. The question TabPFN answers is: can the same paradigm work for structured data?

Spoiler: it can.

Key properties:

• Pretrained on millions of synthetic tabular datasets
• No training loop on your end — inference only
• Competitive with AutoML systems on small-to-medium datasets
• Best suited for datasets under 50,000 rows
• Particularly strong in healthcare, finance, and fraud detection

The Experiment: Breast Cancer Wisconsin

To give TabPFN a real test, I used the Breast Cancer Wisconsin dataset one of the most established benchmarks in medical machine learning. It’s small, structured, and clinically meaningful, which makes it ideal for evaluating models in the exact conditions where TabPFN is designed to shine.

Dataset at a glance: 569 patients, 30 numerical features, binary classification (malignant vs. benign). No missing values. No categorical encoding needed.

I benchmarked four models under identical conditions: same train/test split, same random seed, no custom preprocessing.

Full Python Implementation

Install dependencies

pip install scikit-learn xgboost tabpfn shap matplotlib pandas

Load and split the data

import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

data = load_breast_cancer()
X = pd.DataFrame(data.data, columns=data.feature_names)
y = data.target

X_train, X_test, y_train, y_test = train_test_split(
    X, y,
    test_size=0.2,
    random_state=42,
    stratify=y
)
# Train: (455, 30), Test: (114, 30)

One detail worth noting: stratify=y ensures class balance is preserved across splits. This matters more than most people realize on small datasets.

Decision Tree

from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score, roc_auc_score

dt = DecisionTreeClassifier(random_state=42)
dt.fit(X_train, y_train)

dt_pred = dt.predict(X_test)
dt_prob = dt.predict_proba(X_test)[:, 1]

Random Forest

from sklearn.ensemble import RandomForestClassifier

rf = RandomForestClassifier(n_estimators=200, random_state=42)
rf.fit(X_train, y_train)

rf_pred = rf.predict(X_test)
rf_prob = rf.predict_proba(X_test)[:, 1]

XGBoost

from xgboost import XGBClassifier

xgb = XGBClassifier(
    n_estimators=300,
    learning_rate=0.05,
    max_depth=4,
    random_state=42,
    eval_metric='logloss'
)
xgb.fit(X_train, y_train)

xgb_pred = xgb.predict(X_test)
xgb_prob = xgb.predict_proba(X_test)[:, 1]

TabPFN — and this is where it gets interesting

from tabpfn import TabPFNClassifier

tabpfn = TabPFNClassifier()   # No hyperparameters. That's the point.

tabpfn.fit(X_train.values, y_train)

tab_pred = tabpfn.predict(X_test.values)
tab_prob = tabpfn.predict_proba(X_test.values)[:, 1]

# Accuracy: 0.9824
# ROC-AUC:  0.9960

Notice what’s missing: no n_estimators. No learning rate. No grid search. You hand it the data and it works.

Results

TabPFN wins on both metrics — and it’s not close at the top. What’s more striking is the 0.996 ROC-AUC: this means the model can almost perfectly rank malignant tumors above benign ones.

A false negative in cancer detection is not a metric failure — it’s a missed diagnosis. Models with ROC-AUC this high earn serious consideration for clinical screening pipelines.

Explainability: Understanding What Drives Predictions

High accuracy is only half the story in healthcare. Clinicians and regulators need to understand why a model makes a decision.

Feature importance via Random Forest

import matplotlib.pyplot as plt

importances = rf.feature_importances_
importance_df = pd.DataFrame({
    'Feature': X.columns,
    'Importance': importances
}).sort_values('Importance', ascending=False)

plt.figure(figsize=(10, 6))
plt.barh(importance_df['Feature'][:10], importance_df['Importance'][:10])
plt.gca().invert_yaxis()
plt.title('Top 10 Feature Importances — Random Forest')
plt.tight_layout()
plt.show()

The top features surface as worst radius, worst perimeter, and mean concavity — consistent with decades of clinical literature on tumor morphology. When your model agrees with domain experts, that's a signal worth noting.

SHAP values for deeper interpretability

import shap

explainer = shap.TreeExplainer(xgb)
shap_values = explainer.shap_values(X_test)

shap.summary_plot(shap_values, X_test)
shap.plots.waterfall(explainer(X_test)[0])

HAP waterfall plots let you explain individual predictions: why did the model flag this specific patient? This level of transparency is increasingly required in regulated industries.

Why TabPFN Performs This Well

The short answer: meta-learning at scale.

TabPFN is trained on synthetic datasets generated from a broad prior over tabular tasks — different feature counts, different class balances, different noise levels, different correlation structures. By training on this distribution of datasets rather than any single one, the model develops an internal representation of what good classification looks like in general.

When you feed it your data, it performs what researchers call in-context learning: it conditions on your training examples at inference time without updating any weights. Your data becomes part of the context window, not a training set.

This is architecturally similar to how large language models handle few-shot prompting — except here, the “prompt” is your labeled dataset and the “answer” is a class prediction.

The practical consequence: TabPFN generalizes extremely well on small datasets precisely because it has already learned the inductive biases that gradient boosting trees must re-learn from scratch every time.

When to Use TabPFN — and When Not To

Strong fit:

• Small-to-medium datasets (under 50,000 rows)
• Rapid prototyping where tuning time is expensive
• Healthcare, finance, fraud detection, customer analytics
• When you need a strong zero-shot baseline immediately
• Research benchmarks requiring reproducibility without hyperparameter variance

Poor fit:

• Datasets above 100,000 rows — inference becomes expensive
• Tasks requiring heavy domain-specific preprocessing
• Production systems where full model ownership is required
• Deep learning pipelines with complex inputs like images or time series

TabPFN is not a replacement for XGBoost in every context. It’s a powerful new tool for the specific class of problems where traditional models underfit due to limited data — and that class is larger than most engineers realize.

The Bigger Picture

We’re at an inflection point in tabular machine learning.

For a decade, the dominant paradigm was: collect data, define features, train model, tune model, repeat. That workflow isn’t going away. But a new paradigm is layering on top of it — one where pretrained foundation models bring prior knowledge that your dataset alone cannot provide.

TabPFN 2.5 is early evidence that this paradigm works for structured data. The model isn’t magic; it’s a well-designed system for transferring knowledge across tasks at scale. But from a practitioner’s perspective, the result feels like magic: hand it a small medical dataset and watch it beat the models you spent days tuning.

Whether TabPFN replaces XGBoost in your stack is a question only your use case can answer. But ignoring it is becoming increasingly hard to justify.

Quick Start

pip install tabpfn

from tabpfn import TabPFNClassifier

model = TabPFNClassifier()
model.fit(X_train, y_train)
predictions = model.predict(X_test)

That’s it. Three lines. Try it on your next classification problem before you reach for the hyperparameter grid.

If this was useful, follow for more applied ML breakdowns — one deep-dive per week.