Tabular Foundation Models, Explained

Part I: How TabPFN, Prior-Data Fitted Networks, and In-Context Learning Are Changing the Way We Build Models for Structured Data.

Every other data modality got its foundation model years ago. Text got GPT. Images got CLIP and diffusion backbones. Even time series got pretrained transformers. Tabular data — however, that actually runs most of the world’s decisions — kept losing to a 20-year-old algorithm: gradient-boosted trees.

If deep learning eats every other domain, why did it take until 2025 for a neural network to convincingly beat XGBoost on small, everyday tables? And what changed?

“The short answer is that we stopped trying to make a network learn a table and instead made a network learn how to learn from any table”. That shift — from training a fresh model per dataset to having one pretrained model absorb new data at inference time — is what we now call a tabular foundation model (TFM). This post sets up the rest of the series: what a TFM actually is, why the field was stuck for so long, the core mechanism (in-context Bayesian inference) that broke the stalemate, and a hands-on first run with the model that started it, TabPFN.

What is a tabular foundation model?

A tabular foundation model is a single pretrained network — almost always a transformer — that can make predictions on a brand-new table of rows and columns without being retrained on that table. You hand it your training rows as context and your test rows as a query, it does one forward pass, and outputs predictions. No gradient descent on your data, no hyperparameter search, and no specific feature engineering pipeline.

This is easy to confuse with two things. It isn’t AutoML, which still fits a fresh model (or many) per dataset, just faster and with less human tuning. And it isn’t a tabular embedding model in the sense of, say, a learned representation for one fixed schema — a TFM is explicitly built to generalize across datasets with different numbers of columns, different column meanings, and different target types.

Think of it like a calculator versus a custom circuit. A calculator is built once and handles any arithmetic problem you give it. Soldering a one-off circuit for every new equation is what tabular ML has done since the 1990s — fit trees or a network from scratch, per dataset. A TFM does the soldering once, during pretraining, and reuses that same circuit afterward.

So the real question driving this field is: what does a model need to be trained on, and trained to do, before it can generalize to tables it’s never seen — different columns, different scales, different relationships between features and targets? That’s what the rest of this post covers.

Why Tabular Foundation Models Matter

For nearly two decades, the tabular machine learning playbook remained largely unchanged: preprocess the data, engineer features, train XGBoost or LightGBM, tune hyperparameters, and deploy. A widely cited 2022 benchmark study reinforced this status quo, showing that gradient-boosted trees consistently outperformed most neural-network-based approaches across a broad range of tabular datasets. Unlike images and text, tabular data presents unique challenges — irrelevant features, heterogeneous data types, and complex decision boundaries — that tree-based methods handle particularly well.

However, this success comes at a cost. Every new tabular ML problem — whether in credit risk, customer churn, fraud detection, or healthcare — typically requires extensive preprocessing, feature engineering, hyperparameter tuning, and retraining. While foundation models have transformed domains like NLP by enabling a “pretrain once, use many times” paradigm, tabular machine learning has remained largely dependent on training a new model for every dataset.

Several attempts were made to bridge this gap, including transformer-based architectures such as TabTransformer. Yet none consistently challenged the dominance of gradient-boosted trees.

The breakthrough came from rethinking the problem itself. Instead of training a model to solve a specific dataset, researchers trained a model to learn the process of supervised learning across millions of synthetic datasets. This idea led to TabPFN (Tabular Prior-data Fitted Network).

For the first time, tabular machine learning had a credible foundation-model paradigm. While the original TabPFN was limited to small datasets, TabPFN v2 (2025) greatly expanded its applicability, making tabular foundation models a practical alternative to gradient-boosted trees.This marks a major shift in tabular machine learning and the start of a new era for structured data modeling.

The Core Intuition: In-Context Learning for Tables

To understand Tabular Foundation Models (TFMs), it’s helpful to step away from architectures and think about the underlying idea.

Imagine a weather expert who has spent years studying how temperature, humidity, wind, and pressure interact across thousands of different climates. Now, suppose you give them a handful of observations from a city they’ve never seen before. Even without conducting a detailed study of that city, they can make a reasonable forecast by comparing those observations to patterns they’ve encountered before.

That’s essentially what a TFM does for tabular data.

During pretraining, the model is exposed to millions of synthetic datasets generated from diverse statistical relationships. Instead of learning one specific task, it learns the broader patterns and structures that commonly appear across tabular datasets. Later, when presented with a new dataset, the model uses those learned patterns to make predictions directly from the examples provided in the input.

This process is known as in-context learning (ICL). Rather than fitting a new model, updating parameters, or running an optimization procedure, the training data itself is provided as context alongside the query examples. The model then infers predictions in a single forward pass, much like how a large language model performs a new task from examples in a prompt.

What makes this particularly interesting is the theory behind it. By training on a vast collection of synthetic datasets sampled from known data-generating processes, models such as TabPFN effectively learn an approximation to Bayesian inference. In other words, the model learns to answer a question of the form:

“Given everything I’ve learned about how tabular datasets generally behave, and given these specific examples, what is the most likely prediction for this new row?”

This is fundamentally different from traditional machine learning approaches. A decision tree learns dataset-specific splits, and a neural network learns dataset-specific weights. A TFM, on the other hand, learns a reusable inference procedure that can be applied across datasets without retraining.

This intuition also helps explain both the strengths and limitations of TFMs. They tend to perform best when a new dataset resembles the types of relationships encountered during pretraining and typically degrade gracefully as the data distribution shifts. However, these models primarily reason about statistical patterns rather than the semantic meaning of columns. Unless explicitly designed to incorporate metadata such as feature names, they operate on the numerical structure of the data rather than understanding what each feature represents.

As we’ll see later in this series, addressing this limitation has become a major focus of recent TFM research and is one of the key motivations behind newer generations of tabular foundation models.

The Mathematical Foundation: Prior-Data Fitted Networks (PFNs)

At the heart of every Tabular Foundation Model lies a simple but powerful idea: Prior-Data Fitted Networks (PFNs).

While the equation may look intimidating, the intuition is straightforward.

Instead of training on rows from a single dataset, PFNs train on entire datasets sampled from a distribution of possible data-generating processes. For each synthetic dataset, the model is shown a set of training examples (Dtrain) and is asked to predict the label of a held-out query example (xquery). The objective is to maximize the probability assigned to the correct answer.

The key difference from conventional machine learning is that the model is not learning one dataset-specific function. Every training step exposes it to a completely new synthetic table with different features, relationships, and noise patterns. To succeed, the network must learn a general strategy for using observed examples to make predictions, rather than memorizing any particular dataset.

This is what allows a single pretrained model to generalize to unseen datasets without retraining. During inference, the model simply receives the training examples as context and applies the inference procedure it learned during pretraining.

From a Bayesian perspective, PFNs can be viewed as learning an approximation to posterior inference. The synthetic datasets used during training define a prior over possible tabular problems, and the model learns how to update its beliefs when presented with observed data. As more context examples become available, its predictions become increasingly tailored to the dataset at hand. When only a small amount of data is available, the model naturally falls back on the broader statistical patterns learned during pretraining.

This characteristic is particularly important because many real-world tabular problems operate in the small-data regime. Traditional models often struggle when training data is limited, whereas PFNs are explicitly trained to leverage prior knowledge accumulated across millions of datasets.

In essence, a PFN learns not just a predictive model, but a reusable inference algorithm. The knowledge resides in the pretrained weights, while the dataset itself becomes part of the input context. The “model fitting” happens through the examples provided at inference time rather than through an optimization process on the new dataset.

Architecture at a Glance

Before diving into the attention mechanisms and architectural innovations — which we’ll cover in Part 2 — it’s helpful to understand the high-level workflow of a Tabular Foundation Model.

Unlike traditional machine learning models that must be retrained for every new dataset, TabPFN stores its learned knowledge entirely within its pretrained weights. At inference time, both the training examples and the rows to be predicted are provided together as input to the model.

The process is surprisingly simple. Training rows and test rows are tokenized and passed through a transformer architecture. Through attention mechanisms, each test instance can identify and focus on the most relevant training examples, effectively learning which past observations are most useful for making its prediction. The model then produces predictions for all test rows in a single forward pass.

A key advantage of this design is that learning happens through context rather than optimization. Instead of fitting parameters on a new dataset, the model leverages the examples provided in the input to adapt its predictions.

To improve robustness, TabPFN typically performs inference across multiple random feature permutations and transformations, combining the resulting predictions into an ensemble. This helps ensure that performance is not dependent on arbitrary choices such as column ordering, which generally carries no meaningful information in tabular datasets.

While newer models such as TabICL, TabDPT, and TabPFN v2 introduce important innovations in attention mechanisms, scaling strategies, and synthetic data generation, they all share the same core principle: using in-context learning to perform tabular prediction without task-specific retraining.

That shared foundation is what makes them part of the broader family of Tabular Foundation Models — and the focus of this introductory post.

Where Are Tabular Foundation Models Being Used?

The strongest early adoption of Tabular Foundation Models (TFMs) has come from domains where labeled data is scarce, expensive to collect, and difficult to reproduce.

Drug Discovery and Chemistry

Drug discovery is a natural fit for TFMs. Early-stage molecular screening datasets are often small, noisy, and subject to significant distribution shifts between research and real-world deployment. Recent studies have shown that TabPFN can match the performance of well-tuned gradient-boosted trees for classification tasks while often delivering superior results on regression problems, particularly in low-data and out-of-distribution settings.

Biomedical Research

Biomedical applications face similar challenges: limited datasets, high data collection costs, and the need for rapid model development. Researchers have applied TabPFN to tasks such as disease risk prediction, biomarker discovery, and compound efficacy estimation, where the ability to achieve strong performance with minimal tuning is often as important as achieving marginal accuracy improvements.

AutoML and Enterprise ML Platforms

Perhaps the strongest signal of industry adoption is the growing integration of TFMs into AutoML ecosystems. Rather than replacing existing approaches outright, models such as TabPFN are increasingly being incorporated into ensemble pipelines alongside gradient-boosted trees and other traditional methods.

This highlights an important reality: for many organizations, TFMs are not necessarily a replacement for established tabular models but an additional tool that can strengthen existing machine learning workflows.

Common Pitfalls and Failure Modes

Despite the excitement surrounding TFMs, they are not a universal solution. Understanding their limitations is critical for successful adoption.

Best Practices for Using Tabular Foundation Models

If you’re experimenting with TFMs today, a few practical guidelines can help you get the most out of them:

Key Takeaways

Tabular Foundation Models represent a fundamental shift in how we think about tabular machine learning.

Rather than learning to fit a specific dataset, models such as TabPFN learn the process of fitting itself. By training on millions of synthetic datasets, they acquire a reusable inference procedure that can be applied to entirely new problems through in-context learning. This reframing — from learning a task to learning how to learn tasks — is what enabled TFMs to emerge as a credible alternative to traditional tabular modeling approaches.

The arrival of TabPFN v2 marked an important milestone, demonstrating that this paradigm could move beyond a research curiosity and become a practical tool for real-world machine learning workflows. As a result, the key questions facing the field have shifted from “Can this work?” to “How far can it scale?”

In this first part of the series, we explored the core ideas behind Tabular Foundation Models: the motivation, the intuition behind in-context learning, the foundations of Prior-Data Fitted Networks, and the architecture that powers TabPFN.

Try TabPFN Yourself

If you’d like to get a hands-on feel for TabPFN, I’ve put together a simple Google Colab notebook that walks through the end-to-end implementation — from loading a dataset and training a model to generating predictions and evaluating performance.

📓 Google Colab Notebook: Hands-on TabPFN Tutorial

One of the best ways to appreciate the promise of Tabular Foundation Models is to see how quickly you can get a competitive model running with minimal setup and no hyperparameter tuning.

In Part 2, we’ll dive deeper into the models that pushed the field forward — exploring architectures such as TabICL, TabDPT, and TabPFN v2, and examining how innovations in attention mechanisms, pretraining strategies, and scaling techniques are enabling TFMs to tackle increasingly large and complex tabular datasets.