This AI Model Beats XGBoost in 3 Seconds (No Training Required)

How a transformer-based foundation model is challenging the decade-long dominance of XGBoost and Random Forests

Image by Author

The Problem with Traditional Machine Learning

If you’re reading this, you’ve likely been through the process of loading tabular data into a program, splitting the data into training and test sets, creating a model using XGBoost and tuning the model’s hyperparameters for hours (possibly by ensembling multiple models), and eventually getting results that were satisfactory.

Now imagine that rather than taking an eternity to train and fine-tune your XGBoost model, there is a single model that can create an equivalent or better set of results in less than three seconds, absolutely no training and hyperparameter tuning required.

Introducing TabPFN (Tabular Prior-data Fitted Network) — a new type of foundation model that uses transformers to change how we build machine learning models for tabular datasets.

What Makes TabPFN Different?

Traditional ML: Train for Every Dataset

When you use XGBoost, Random Forest, or CatBoost, here’s what happens:

1. Load your dataset then separate it into train and test sets
2. Train the model: this will involve gradient descent (a type of iterative optimization) as well as build trees based on historical data (minutes to hours)
3. Find the best hyperparameters through a search (Grid Search or Bayesian Optimization) (this will take hours)
4. Repeat for every new dataset

TabPFN: Zero-Shot Learning

The process of using TabPFN for Zero-Shot Learning is completely different than using the above. It involves:

1. Loading your dataset and separating it into train and test sets
2. Simply using the TabPFN library by calling the appropriate .fit() and .predict() methods and it will do everything in seconds
3. There is no need for hyperparameter tuning
4. You use the same pre-trained model across all datasets

The magic? TabPFN does not learn from your data like the above mentioned methods do. Instead, TabPFN is an example of “in-context learning” — and is based on the same concept as GPT models.

How Does TabPFN Actually Work?

The Core Idea: Learning to Learn

TabPFN is based on a brilliant insight: instead of using your dataset to create a supervised learning algorithm, what if we took all of the datasets and built a model that will use that information to create its own supervised learning algorithm?

Here’s the process:

1. Pre-training on Synthetic Data

Researchers created millions of different synthetic tabular datasets using:

• Random causal graphs to model relationships between variables based on real-world examples
• Many different types of data distributions (Gaussian distribution, heavy-tailed and skewed)
• Different feature types (numerical, categorical)
• Many different target variable relationships (e.g., linear/non-linear; interaction effects)

The transformer was trained to predict labels on these synthetic datasets, essentially learning the meta-pattern of supervised learning.

2. In-Context Learning at Inference

The fit() function takes your real data and:

• treats it like Context (like a prompt to GPT)
• it utilises the knowledge it has learned to find patterns in your training data when you call predict() on new samples.
• There are no gradient updates. There is no backpropagation. It’s simply pattern matching.

This method is vastly different than how traditional machine learning works

Traditional ML: Model parameters ← Learned from your data
TabPFN: Model parameters ← Fixed (pre-trained)
        Pattern recognition ← Done via attention mechanism

3. The Transformer Architecture

Under the hood, TabPFN employs a Transformer Encoder structure similar to what is used in BERT:

• Self-attention layers are used to model relationships between all samples and features on a tokenized input.
• Positional encodings help to preserve the sequential order of the various features.
• Approximate Bayesian inference in the forward pass
• Uncertainty quantification comes naturally from the probabilistic approach

Performance: The Numbers Don’t Lie

Benchmark Results from the Nature Paper

The researchers evaluated TabPFNv2 on 57 real-world datasets from AutoML Benchmark and OpenML-CTR23, with datasets having:

• 29 classification datasets
• 28 regression datasets
• Datasets with up to 10,000 samples, 500 features, 10 classes

Key Results:

Classification Performance

Over 29 classification datasets TabPFN performed better than XGBoost, CatBoost, LightGBM, Random Forest, and neural networks on average in terms of ROC AUC. TabPFN matched an ensemble of tuned baselines after a single forward pass which took 2.8 seconds; while each baseline was fine-tuned for 4 hours.

Regression Performance

Over 28 regression datasets TabPFN outperformed all tree-based methods with respect to average R² scores. TabPFN also produced improved uncertainty estimates (calibrated probabilistic predictions).

TabPFN-2.5: The Latest Evolution

On industry benchmarks (up to 50,000 samples, 2,000 features):

• 100% win rate against the default XGBoost on datasets ≤10,000 samples
• 87% win rate against the default XGBoost on datasets up to 100K samples (classification)
• Matches AutoGluon 1.4 (tuned exsemble in four hours) in a single forward pass
• Inference time: less than 3 seconds

When to Use TabPFN vs Traditional Models

TabPFN is an Excellent Choice If:

• You’re working with datasets with less than 50,000 rows.
• You need to develop a prototype quickly.
• You do not have time to tune hyperparameters.
• You want to quantify the uncertainty of the outputs.
• You want to do minimal pre-processing of your data.
• You will have no more than 2000 features.
• You will have no more than 10 classes.

If You Use XGBoost/CatBoost, You Should:

• You’re working with datasets greater than 100,000 rows.
• You expect to have more than 10 classes.
• You plan to deploy an application in production and expect little latency to the user.
• You use feature engineering as your competitive advantage.
• You have custom features with domain knowledge coded into them.

The Sweet Spot

According to research, TabPFN performs exceptionally well on:

• Datasets containing irregularly distributed features (the TabPFN algorithm exceeds expectations when it comes to skewed and heavy-tailed datasets)
• Minimal training data (where traditional models typically fail)
• Various feature types without significant preprocessing

Hands-On: Using TabPFN

Installation

pip install tabpfn

Classification Example

from sklearn.datasets import load_breast_cancer
from sklearn.metrics import accuracy_score, roc_auc_score
from sklearn.model_selection import train_test_split
from tabpfn import TabPFNClassifier

# Load data
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.5, random_state=42
)

# That's it - no hyperparameters needed!
clf = TabPFNClassifier()
clf.fit(X_train, y_train)

# Predictions with uncertainty
probabilities = clf.predict_proba(X_test)
predictions = clf.predict(X_test)

print(f"ROC AUC: {roc_auc_score(y_test, probabilities[:, 1]):.4f}")
print(f"Accuracy: {accuracy_score(y_test, predictions):.4f}")

Regression Example

from sklearn.datasets import fetch_openml
from sklearn.metrics import mean_squared_error, r2_score
from sklearn.model_selection import train_test_split
from tabpfn import TabPFNRegressor

# Load Boston Housing data
df = fetch_openml(data_id=531, as_frame=True)
X = df.data
y = df.target.astype(float)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.5, random_state=42
)
# One line to rule them all
regressor = TabPFNRegressor()
regressor.fit(X_train, y_train)
predictions = regressor.predict(X_test)
print(f"MSE: {mean_squared_error(y_test, predictions):.4f}")
print(f"R² Score: {r2_score(y_test, predictions):.4f}")

Advanced Features

# Use older TabPFN v2 weights (if needed)
from tabpfn.constants import ModelVersion
clf_v2 = TabPFNClassifier.create_default_for_version(ModelVersion.V2)

# Enable GPU acceleration
clf_gpu = TabPFNClassifier(device='cuda')
# Handle larger datasets (experimental)
clf_large = TabPFNClassifier(ignore_pretraining_limits=True)
# Get uncertainty estimates
probabilities = clf.predict_proba(X_test)
# Higher entropy = more uncertain predictions

Real-World Performance Insights

What Researchers Found

Multiple independent benchmarks reveal consistent patterns:

Small datasets (less than 1,000 samples)

• TabPFNv2 consistently outperformed tuned XGBoost (specific margins vary by dataset)
• It took approximately 2 to 3 seconds to train TabPFN as compared to multiple minutes to train XGBoost.

Medium datasets (1,000 to 10,000 samples)

• The performance of TabPFNv2 was equal to or slightly better than the tuned baselines.
• Like the small datasets, TabPFN required no hyperparameter tuning.

Large datasets (10,000 to 50,000 samples)

• The performance of TabPFN-2.5 is still competitive with tuned XGBoost, and the performance difference lessened as the dataset increased in size.

Very large datasets (more than 50,000 samples)

• XGBoost and CatBoost regained dominance, and TabPFN used significantly more memory than before.

The Latency Trade-off

• Although TabPFN is trained very quickly, it may take much longer to perform inference; specifically, tree models (XGBoost/LightGBM) take less than 0.4 seconds to make 100K predictions.
• TabPFN inference is slower than tree models for large-scale predictions (specific times vary by dataset size and hardware).
• TabICL (another foundational model) generally have slower inference times

For production systems that require real-time predictions, tree-based models will have a performance advantage over TabPFN. However, TabPFN also has a VD (very-dense) version, called TabPFN-as-MLP, which takes the TabPFN model and reformats it as a small neural network or tree ensemble. While still providing most of the accuracy, this format significantly reduces latency.

The Technical Innovation: Why It Works

1. Synthetic Data Generation

The key innovation is the quality and diversity of synthetic training data:

For each synthetic dataset:
1. Sample a causal graph structure
2. Initialize random parameters
3. Generate features following causal relationships
4. Add realistic noise patterns
5. Create diverse target relationships

This creates datasets that mirror real-world complexity without overfitting to specific domains.

2. Bayesian Meta-Learning

TabPFN approximates Bayesian inference:

Traditional ML: p(y|x) = f(x; θ*) where θ* = argmax p(D|θ)
TabPFN: p(y|x, D_train) ≈ ∫ p(y|x, θ) p(θ|D_train) dθ

In plain English: TabPFN does not define only one optimal model, but uses the likelihood that other models explain your training data well to construct a distribution of all potential models.

3. Attention Mechanism

The transformer’s self-attention learns:

• Feature interactions: Determine how features relate to one another.
• Sample relationships: Determine how similar training samples relate.
• Context-aware predictions: Differentiate between random chance and true benefits by adjusting to the patterns in your dataset.

By using the built-in attention mechanism of the transformer, TabPFN is able to generalize very well because it is identifying patterns that exist in the dataset from learning patterns from other datasets, rather than trying to memorize data.

Limitations and Considerations

Existing Limitations

1. Dataset size: Maximum support of 50K rows (TabPFN-2.5)
2. Number of classes: Maximum of 10 classes are best, although this support can be extended
3. Hardware Requirements: Server-class/GPU (larger datasets)
4. Latency of Prediction: Slower than tree-based methods for real-time predictions
5. Interpretability: More difficult to understand than a tree-based method

Not a Silver Bullet

TabPFN is not the only method:

• For Kaggle Competitions — months of tuning time → Feature engineering + XGBoost still wins
• For very large datasets — i.e., more efficient methods of predictive modeling exist (e.g., XGBoost/CatBoost, all other methods are very scalable)
• For Time Series Forecasting — more effective methods exist (i.e., specialized method)
• For Highly Unbalanced Data — may require additional techniques

The TabPFN Ecosystem

There are many other components that complement our existing core model:

Extensions & Tools

• TabPFN Client: API in the cloud allowing for inference without requiring a graphics processing unit (GPU).
• TabPFN Extensions: Methods for SHAP usage/extension, for feature selection (FS), and for the detection of outliers (OD).
• AutoTabPFN: Automated hyperparameter optimization and method to build ensemble models.
• TabPFN UX: No-code graphical user interface for business users.

Enterprises’ Capabilities

This is how TabPFN will deliver value from the perspective of production environment configuration:

• Fast Inference Mode: By using a distilled multi-layered perceptron (MLP) or decision tree ensemble model (1000× faster).
• Large Data Mode: Capable of handling tabular datasets with as many as 10 million rows.
• Commercial License: For use in production environments.

The Future of Tabular ML

TabPFN represents a revolutionary way of training machine learning models:

From: Dedicated machine learning model training, specifically for each dataset

To: General purpose models that learn without labeled examples

This is similar to the revolution that we have seen in natural language processing through the advent of GPT models and in computer vision with CLIP and SAM. There are significant implications associated with this new paradigm for table-based ML:

1. Fast experimentation: a data scientist can test concepts in seconds versus the traditional hours
2. Lower barrier to entry: significantly less ML expertise is now required to produce strong baseline predictions
3. Improved uncertainty quantification: built-in capability to provide probabilistic predictions
4. Ability to leverage learning from millions of datasets and apply to your data

Conclusion: A Tool, Not a Replacement

TabPFN is not intended to replace XGBoost or Random Forests as an alternative tool set, rather it complements other tools that you already have in your toolbox:

• When rapid prototyping: TabPFN cannot be topped.
• When working with small sets of data: TabPFN usually provides better results than all other choices.
• When producing at scale: Traditional models continue to be better than TabPFN.

The field of machine learning is changing rapidly; there are now many foundation models (such as TabPFN) that have opened the doors to new, high-performance methods of building advanced models that have never been accessible before!

Resources

• Paper: TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second
• Paper: TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Models
• GitHub: https://github.com/PriorLabs/TabPFN
• Documentation: https://priorlabs.ai/docs
• Nature Publication: Accurate predictions on small data with a tabular foundation model