Real-Time Semantic Search Inside Palantir Foundry: Building a Voyager-Powered Compute Module

Introduction

The Problem: Interactive Search Over Millions of Embeddings

While building a personal music analytics platform, I needed a way to interactively search through a massive dataset of 1024-dimensional song lyric vectors. I wanted to be able to query a theme like “existential dread” and get the top five semantically matching lyrics back in under two seconds. Serving this kind of real-time vector search over millions of embeddings is a completely different problem than processing batch data, and standard pipelines just weren’t designed for that kind of interactivity.

To build this, I started with a 25-million-track database sourced from LRCLIB. Processing that data at scale - cleaning, embedding, and classifying the lyrics was a perfect fit for a standard Palantir Foundry pipeline. But once the data was prepped, the final hurdle was actually surfacing it. I needed a way to interactively search those embeddings by free text, artist, or theme, without exporting everything out of the platform.

Why Not Just Self-Host?

The conventional approach would be to deploy a standalone service like a FastAPI container on GCP Cloud Run, an AWS Lambda behind API Gateway, or a Kubernetes pod running a vector database. This works, but introduces some challenges:

• Data duplication: You need to export embeddings from Foundry, transfer them to your cloud provider, and keep them in sync as the pipeline evolves.
• Operational overhead: You’re now maintaining infrastructure outside Foundry (CI/CD, monitoring, scaling, certificates, network policies) all for a single search endpoint.

Foundry’s **Compute Modules offer an alternative: deploy your custom container in any language, any library directly inside the Foundry platform. Compute Modules follow a “bring your own container” philosophy under a zero-trust security model: you package everything you need inside the container, and the platform provides scaling, governance, and strict network isolation. Your code runs alongside the data**, inherits the platform’s auth, and scales automatically with demand. One relevant precedent: Palantir has run Cohere’s embedding models inside Compute Modules by packaging model weights directly in the container. My situation was different - I use a proprietary, API-only model, which forced a more creative architecture, as we’ll see below.

Why Voyager?

I chose Spotify’s **Voyager** library (an open-source, in-memory approximate nearest neighbor search library) for several reasons:

• Zero infrastructure: Unlike vector databases, Voyager is a library, not a service. No database to manage, no cluster to scale. Just load the index into memory and query it.
• Fast: Sub-millisecond query times for 50K vectors. Scales well into the millions.
• Portable: The index is a single .voy file. Build it in a Spark transform, serve it in a Compute Module, or download it to your laptop.

That said, this architecture is flexible. You could swap Voyager for Annoy (also by Spotify), or any other ANN library, the Compute Module pattern remains the same. The core idea is: build the index in a pipeline, serve it in a container.

Architecture and Decisions

System Overview

The final system consists of three components working together:

1. Ingestion Data Pipeline (Python Transforms) — Downloads raw lyrics, cleans, deduplicates, embeds, classifies themes, and builds the Voyager index.
2. Embedding Proxy (Python Functions) — A lightweight function that wraps the Language Model Service via the OpenAI-compatible proxy, exposing text embedding as a callable Query API with native dimension control.
3. Search Module (Python Compute Module with Voyager) — Loads the pre-built index, embeds user queries via the proxy, and returns nearest-neighbor results.

Batch Data Pipeline Architecture

Serving Layer Architecture

How the Index Gets Built

The ingestion pipeline processes raw lyrics through 8 transform steps:

Note: This is the current working process, though I am actively running into edge cases that will likely alter the cleaning steps in future iterations

1. Download a 25GB compressed SQLite dump from LRCLIB (24.8M tracks)
2. Parse, deduplicate, and clean artist names (11.7M tracks)
3. Detect language using fastText, filter to English (7.4M tracks)
4. Chunk lyrics by verse (54.3M chunks)
5. Stratified sample to 50K chunks (preserving top artists)
6. Embed each chunk using text-embedding-3-large at 1024 dimensions.
7. Classify each chunk into one of 30 lyrical themes via cosine similarity
8. Build a Voyager ANN index from all 50K embeddings and save as a .voy file

The output is two datasets: a voyager_chunk_index (containing the .voy binary) and lyric_chunks_themed (containing the metadata for each chunk, useful for reverse lookups for the response). The Compute Module downloads both at startup.

How a Query Works

When a user searches for “existential dread”, the following happens:

1. The Compute Module receives the query via its function endpoint.
2. It calls the Embedding Proxy, a Python function via the Ontology Query API
3. The proxy calls the Language Model Service via the OpenAI-compatible client, requesting 1024-dimensional vectors directly.
4. Voyager performs an approximate nearest neighbor search (cosine distance) and returns the top-k chunk indices.
5. The module looks up metadata (artist, track, lyrics, theme) and returns enriched results with similarity scores.

End-to-end latency is typically 2–3 seconds, dominated by the embedding API call (~1.5s). The Voyager search itself takes <1ms.

Query Flow Sequence

Problems Along the Way

Getting this architecture working required navigating several platform constraints that aren’t immediately obvious from the documentation. Each of these took some debugging time and represents a genuine lesson for anyone building Compute Modules in Foundry.

1: The Container Can’t Talk to Anything (By Default)

Symptom: Every HTTP request from the container to anything external to it (Foundry APIs, Language Model Service) failed.

Root cause: Compute Modules operate under a zero-trust network model. By default, containers have no outbound network access whatsoever, not even to other Foundry services running on the same infrastructure.

Fix: Created a REST API Source in Data Connection for the Foundry stack hostname and attached it to the Compute Module configuration. Sources are the only mechanism for granting network egress to a container. I am unsure if this is the recommended approach, but it solved my issue of not being able to access anything in Foundry.

2: The Language Model Service is Unreachable

This was the most architecturally significant challenge.

Symptom: The search functions needed to embed user queries at runtime. I tried calling the Language Model Service HTTP API directly but the problem was the same as in the first problem.

Root cause: The LMS embedding API is an internal service that is not accessible via personal access tokens or any token available in a Compute Module. It is only accessible through:

• The palantir_models SDK (available only in Python Transforms environments)
• The foundry_sdk.v2.language_models package with OpenAI client (available in Python Functions environments)

I also confirmed that the internal palantir libraries/sdks aren’t installable as conda packages in Compute Module environments.

Fix: Introduced the Embedding Proxy pattern — a separate Python Functions repository that imports the embedding model and exposes it via the Functions API. The Compute Module calls this function via the standard Ontology Query API endpoint:

POST /api/v2/ontologies/{ontologyRid}/queries/embedTextV2/execute
{
  "parameters": { "text": "existential dread" }
}

Because the Python Function runs in Foundry’s native Functions runtime, it has direct access to the Language Model Service via the OpenAI-compatible proxy, no special tokens needed.

Quick gotcha: Python functions with api_name require at least one ontology entity (e.g., an object type) to be imported into the repository, even if the function doesn't use ontology objects. Without this, the publish step fails. Additionally, functions without api_name are not callable via REST — only functions with an api name exposed through the API gateway.

Lessons Learned & Conclusion

1. Design around the platform, not against it. Compute Modules are powerful but opinionated. The zero-trust network model and execution mode separation are non-negotiable constraints, map your architecture to them from day one.
2. Proxy internal services through Functions with api_name. If a Foundry service isn’t accessible from your container, wrap it in a Python or TypeScript Function with an api_name. This pattern works for any internal API, not just embeddings
3. Always configure Sources for egress. Compute Modules are black boxes — network egress requires an explicitly configured Source in Data Connection. Even “internal” Foundry APIs are external from the container’s perspective. No Source, no network.
4. Don’t assume Foundry-internal packages are available. palantir-models, language-model-service-api, and similar packages only exist in transforms environments. Compute Module conda environments are generic, plan your dependencies accordingly.

When I started this project, my initial goal was simply to build a fast semantic search for song lyrics. What it evolved into was a blueprint for escaping the limits of the standard, lightweight Functions API. It proves you can run heavy custom code — complete with external packages, bespoke data formats, and in-memory storage — inside a container that is hosted and queryable directly in Foundry. Maybe the use cases for this are niche, but I found one I was interested in and was able to successfully make it work.