Relational Graph Transformers: The Backbone of Relational Foundation Models

The Relational Foundation Model (RFM) redefines predictive analytics by offering a unified architecture capable of being fine-tuned on any relational dataset, such as a data warehouse. By interpreting database inputs, the RFM generates high-accuracy insights, including churn prediction, fraud detection, and personalized recommendations, in sub-second latency. This approach eliminates the traditional, months-long lifecycle of manually engineering and training bespoke models for every individual task.

This article provides a comprehensive technical details of Relational Graph Transformers (RelGT)[1], exploring their role as the architectural backbone for Relational Foundation Models.

Relational Foundational Model (Kumo.ai) [2]

Relational Graph Transformer[1]

In standard Transformers, text is represented by two primary components: token embeddings and positional encodings. Similarly, RelGT utilizes a node feature and multiple graph structure information to capture complex relational structures.

In RelGT, the input relational entity graph is discretized into tokens by sampling a fixed neighborhood for each training seed node. These neighbors are then processed through a multi-element tokenization strategy that encodes both content and topology. This tokenized sequence is passed into a Transformer network that synthesizes local and global representations, providing the necessary context for downstream predictive layers.

RelGT Architecture[1]

Let’s take a closer look at the individual building blocks of the Relational Graph Transformer. We will explore how each component, from the initial embedding layers to the attention mechanism, contributes to the model’s overall performance.

Relation Entity Graph (REG)

The structure of relational databases inherently forms a graph representation, called as relational entity graphs. They are constructed via primary-foreign key relationships within database tables. Due to the massive scale of these graphs, RelGT requires sampling a localized subgraph around each seed node. These seed nodes represent the specific entities related to the predictive task such as “users” for a churn model or “items” for a sales forecast. Consequently, the model generates a contextual subgraph for each seed node, providing the necessary structural and feature-based information for training.

Relation entity graph example

Tokenization

RelGT utilizes a multi-element token representation that operates without additional computational overhead relative to the number of nodes in the input Relational Entity Graph (REG). Rather than attempting to compress all structural information into a single positional encoding, the architecture decomposes the token into distinct, decoupled elements. This design allows the model to explicitly capture the unique characteristics of relational data: Node Features represent the intrinsic attributes of an entity, Node Types encode the heterogeneity inherent in multi-table database schemas, Hop Distance preserves relative proximity and structural distance within a local context, Time Encodings integrate temporal dynamics and sequence history, GNN-based Positional Encodings capture and preserve the local topology of the graph.

The RelGT tokenization process[1]

• Node Features: Represent the columnar attributes of an entity in the REG (corresponding to a specific database row), projected into a d-dimensional embedding space. To handle diverse data types, including numerical, categorical, multi-categorical, text, and image data, each modality is processed via a specialized encoder. These modality-specific representations are subsequently aggregated into a unified d-dimensional vector that serves as the node’s initial latent representation.

• Node Type Encoder: The Node Type Encoding steps converts each table-specific entity type into a d-dimensional representation. This mechanism explicitly incorporates schema heterogeneity, allowing the model to distinguish between different entity classes, such as “Users”, “Products”, or “Transactions” within the high-dimensional latent space.

• Hop Encoder: The Relative Hop Distance captures the structural proximity between the seed node and its neighbors. This distance is projected into a d-dimensional embedding, providing the Transformer with a topological prior that allows it to weight information based on its relational distance from the target entity.

• Time Encoder: The Time Encoder applies a linear transformation to the temporal delta between a neighbor node and the seed node. This allows the model to capture the ‘freshness’ of a relationship, effectively incorporating temporal dynamics into the node’s representation.

• Subgraph PE Encoder: To capture local graph structures that other token elements might overlook, a lightweight GNN encoder is applied to the sampled subgraph. This component preserves critical topological relationships, such as complex cycles, quasi-cliques, and hierarchical parent-child dependencies (e.g., the link between a “Product” and its associated “Transactions”). By pre-processing these structures, the GNN provides the Transformer with a rich structural context that raw node features alone cannot provide.

In the final stage, the effective token representation is synthesized by integrating all encoded elements.

This multi-element approach provides a comprehensive embedding that explicitly captures node features, entity types, structural positioning, temporal dynamics, and local topology. Indeed, this decoupled design achieves high expressivity without the prohibitive computational costs typically associated with large-scale graph processing.

Transformer Network

The Relational Graph Transformer (RelGT) processes the tokenized relational entity graph by integrating both local and global attention mechanisms.

The Transformer Network[1]

• Local Attention: This mechanism enables each seed node to attend to its K local tokens selected during the tokenization phase. It is designed to capture fine-grained, high-fidelity relationships defined by the underlying database schema.

• Global Attention: To provide context beyond the immediate neighborhood, seed nodes attend to a set of B global tokens. These tokens represent centroids that summarize the entire graph. They are dynamically updated during training using an Exponential Moving Average (EMA) K-Means algorithm, which is applied to the seed node features of each mini-batch.

• Integration and Output: The final node representation is synthesized by fusing the local and global embeddings. This composite representation is then passed through a Feed-Forward Network (FFN). Each stage of the Transformer follows a standard architecture, incorporating layer normalization and residual connections to ensure stable training and efficient gradient flow.

Prediction Heads

For downstream prediction, the synthesized representation of the seed node is passed through a specialized prediction head tailored to the target task. The entire model is trained end-to-end, utilizing task-specific loss functions such as cross-entropy for classification or mean squared error for regression. By integrating multi-element token representations within a hybrid local-global Transformer architecture, RelGT provides a robust solution to the core challenges of relational data: schema heterogeneity, complex temporal dynamics, and the deep structural dependencies inherent in large-scale relational entity graphs.

RelGT as backbone of RFM

RelGT introduces a powerful backbone for Relational Deep Learning, offering new possibilities for scaling architectures toward Foundation Models tailored for relational data. Its hybrid attention mechanism and multi-element tokenization provide the necessary efficiency to move from task-specific models to general-purpose relational intelligence such as KumoRFM[2].

KumoRFM represents a paradigm shift, a pre-trained foundation model capable of zero-shot prediction on heterogeneous relational schemas. By applying in-context learning to the multi-table relational graph setting, it bypasses the traditional requirements for custom model training. It leverages a table-agnostic encoder and a Relational Graph Transformer to synthesize information from diverse multimodal sources across tables. Consequently, KumoRFM provides low-latency inference for critical enterprise use cases, such as recommendations and fraud forecasting, effectively automating the model development lifecycle for the modern data stack.

KumoRFM architecture[2]

To facilitate in-context learning, KumoRFM dynamically generates context and prediction subgraphs directly from a relational database. This process relies on a dual-sampling strategy: a backward-looking sampler ensures temporal consistency by extracting historical context, while a forward-looking sampler identifies ground-truth labels for training. These subgraphs are then processed by a Relational Graph Transformer, which extracts generalized structural representations that allow the model to reason across diverse schemas.

Alternatively to KumoRFM, there are other RFMs such as Relational Transformer[3] and ConTextTab[4].

The Relational Transformer (RT) is a foundation architecture designed for zero-shot generalization across heterogeneous databases. Unlike traditional models, it can be deployed on unseen datasets and tasks without requiring dataset-specific fine-tuning or in-context retrieval. RT operates by tokenizing database cells alongside their respective table and column metadata. It is pre-trained using a masked token prediction objective and leverages a specialized Relational Attention mechanism to capture dependencies across columns, rows, and primary-foreign key relationships.

**ConTextTab** is a table-native In-Context Learning (ICL) framework that integrates semantic understanding with structural alignment. By utilizing specialized embeddings tailored to diverse data modalities and leveraging pre-training on large-scale, real-world tabular datasets.

References

[1] V. Dwivedi1, S. Jaladi, Y. Shen, F. Lopez, C. Kanatsoulis, R. Puri, M. Fey, J. Leskovec. Relational Graph Transformer. arXiv: 2505.10960. 2025

[2] M. Fey, V. Kocijan, F. Lopez, J. Lenssen, J. Leskovec. KumoRFM: A Foundation Model for In-Context Learning on Relational Data. https://kumo.ai/research/kumo_relational_foundation_model.pdf

[3] V. Hudovernik , M. Znidar, C. Kanatsoulis. Relational Transformer: Toward Zero-shot Foundational Models for Relational Data. arXiv: 2510.06377v2. 2025

[4] M. Spinaci, M. Polewczyk, M. Schambach, S. Thelin. ConTextTab: A Semantics-Aware Tabular In-Context Learner. arXiv: 2506.10707. 2025