How TabPFN “Sees” Letters: Classifying 26 Characters with the Many Class Extension

Photo by Kevin Murray on Unsplash

I recently discovered something fascinating. TabPFN can literally “see” patterns in tabular data using its *Many Class Classifier* extension.

Of course, it’s not “seeing” in the visual sense like a CNN would. But when you look at how certain datasets encode shapes and textures as numbers, TabPFN starts to feel surprisingly perceptive.

Other tabular models can also do this, but with lower accuracy, which makes them a bit myopic in that sense. Here, TabPFN outperforms every baseline, including strong models like Random Forests.

Let’s go through this idea step-by-step and code it together.

Before explaining how is it doing that, first I need to explain a dataset, namely the UCI Letter Recognition dataset.

UCI Letter Recognition dataset

The UCI Letter Recognition dataset is a classic benchmark in machine learning.

Its goal: classify printed capital letters (A–Z) based on their geometric properties.

Here’s what makes it interesting:

• It has 20,000 samples of distorted black-and-white letters (A–Z) generated using 20 different fonts.
• Instead of images, each letter is represented by 16 numeric features. For example, pixel distribution, edge count, and shape statistics.

That means we’re dealing with a 26-class classification problem, where each letter’s shape is encoded as numbers.

In other words, it’s a tabular representation of vision data and that’s where TabPFN gets fun.

Let’s code and see it together.

Installing and Preparing the Data

You can install the dataset and the TabPFN extension easily:

pip install ucimlrepo
pip install "tabpfn-extensions[many_class]"

Then, load the dataset:

from ucimlrepo import fetch_ucirepo 

letter_recognition = fetch_ucirepo(id=59) 

X = letter_recognition.data.features 
y = letter_recognition.data.targets 

Using the Many Class Extension

This is a new extension that came with TabPFNv2.5. It is an output-coding wrapper that enables TabPFN to handle dozens of classes.

Here’s a minimal setup:

from tabpfn_extensions import ManyClassClassifier
from tabpfn import TabPFNClassifier
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)

estimator = TabPFNClassifier(device="cuda")

classifier = ManyClassClassifier(estimator=estimator, alphabet_size=3)
classifier.fit(X_train, y_train)
predictions = classifier.predict(X_test)

Checking the Accuracy and Visualization

from sklearn.metrics import accuracy_score
accuracy = accuracy_score(y_test, predictions)

print(accuracy)
# 0.9862121212121212

That’s 98.6% accuracy, impressive for a model that has never been explicitly trained on image related data!

You can also plot the confusion matrix to see which letters are hardest to distinguish:

import matplotlib.pyplot as plt
import seaborn as sns
import string

cm = confusion_matrix(y_test, predictions)

plt.figure(figsize=(12, 10))
labels = list(string.ascii_uppercase[:26])
sns.heatmap(cm, annot=False, cmap="Blues", fmt="d",
            xticklabels=labels, yticklabels=labels)
plt.title("Confusion Matrix - Letter Recognition (ManyClassClassifier + TabPFN)")
plt.xlabel("Predicted")
plt.ylabel("True")
plt.show()

Even though some letters are more distinguishable than others (like A, D), you can see that all of the letters are distinguishable.

Also, when we compare with other models:

TabPFN outperforms every baseline and even strong models like Random Forests.

And remember: it did this without any hyperparameter tuning, feature scaling, or gradient-based optimization.

That’s what makes it fascinating: a pre-trained probabilistic model, operating in a purely tabular world, still manages to “see” letter shapes through numerical patterns.

When you think about it, that’s a glimpse of what tabular foundation models might bring to perception-heavy problems in the future. It may bridge symbolic and statistical representations.

If you’d like to learn more or get involved, you can join the official Prior Labs Community on Discord here: https://discord.gg/rN9GEBEZ

And if you enjoy what I write, feel free to leave a comment or a few claps. Getting feedback on Medium can be surprisingly hard, and it really helps.