Beyond Parquet: Why the Data World Needs Lance — and How to Start Using It Today

The file format wars are heating up. For over a decade, Parquet has been the undisputed king of columnar storage. Now, a new challenger is reshaping what “fast” really means — especially for AI-era workloads.

A Brief History: The Famous Formats of Data Engineering

Before we talk about what’s changing, let’s appreciate what we had — and still have.

The data engineering ecosystem has cycled through several generations of file formats, each solving the pain points of its era:

CSV / TSV — The universal lingua franca. Simple, human-readable, terrible for analytics. No types, no compression, no schema. But everything can read it.

Avro — Row-oriented, schema-embedded, excellent for Kafka and streaming ingestion. Built for writes, not reads.

ORC (Optimized Row Columnar) — Hive’s answer to columnar storage. Fast for Hive workloads, strong compression. Less portable outside the Hadoop world.

Apache Parquet — The one that won. Columnar, open, highly compressed, compatible with every engine from Spark to Trino to DuckDB. The default substrate for Apache Iceberg, Delta Lake, and Apache Hudi. If your data lake has a heartbeat, it’s beating in Parquet.

But something has shifted. Quietly, persistently, inevitably.

Why the Paradigm Must Change Now

Parquet was designed in the Hadoop era, when workloads were dominated by sequential batch scans, hardware was largely CPU-bound, and storage was spinning disks. It solved those problems beautifully.

Today’s reality looks very different across three dimensions:

Workloads have mutated. AI and ML pipelines don’t just scan tables — they do random point lookups across millions of records, retrieve embeddings, serve RAG pipelines, and train models on multimodal data that mixes text, images, audio, and high-dimensional vectors. These patterns break Parquet’s core assumptions.

Hardware has evolved. Modern NVMe SSDs deliver 850,000+ IOPS, enabling random access patterns that were once impractical. Cloud object stores like S3 are high-bandwidth but high-latency — a profile that punishes “chatty” formats requiring dozens of sequential metadata round-trips just to answer one query.

Data shapes have grown complex. Vector embeddings (768 to 4096 dimensions), wide feature tables with thousands of columns, and blobs like images and video frames are now first-class citizens — not edge cases.

Parquet was never designed to handle these. And that friction is now measurable.

Where Parquet Falls Short

To be fair to Parquet: for classic OLAP batch analytics — large scans, a few columns, aggregations over terabytes — it remains extremely hard to beat. The ecosystem gravity alone (Iceberg, Delta Lake, Spark, Trino, Flink) makes it the safe default.

But its structural decisions create real bottlenecks in modern workflows:

The “chatty” S3 problem. Reading a Parquet file on object storage is a cascade of sequential requests: fetch the footer, fetch row group metadata, fetch page indexes, then fetch actual data pages. For a selective query on a large file, the number of sequential round-trips can number in the dozens — each one paying S3’s latency tax. This serial dependency chain kills performance.

Point lookups are expensive by design. Parquet’s encodings are not designed to be “sliceable.” To access a single row, you typically load an entire page (~1MB). In ML inference, vector search, or feature serving, where you need random access to thousands of specific samples, this becomes catastrophic overhead.

Row group size is a tuning nightmare. Row groups in Parquet are fixed horizontal slices (typically ~128MB). Small data types prefer larger row groups; large data types prefer smaller ones. But a single row group size must be chosen for the entire file — and getting it wrong degrades either scan performance or random access by orders of magnitude.

No native vector or multimodal support. Parquet has no concept of embeddings, blob data, or vector indexes. Storing a 1536-dimensional float array in Parquet means treating it as a repeated field — a format that can’t be efficiently indexed, searched, or retrieved.

Schema evolution is painful. Adding a new derived column (say, an embedding generated after the fact) to a Parquet-backed table typically requires rewriting the entire dataset. At petabyte scale, that’s often not feasible.

Enter Lance: Storage Engineered for the AI Era

Lance is an open-source columnar format built from first principles for AI and ML workloads. Created by contributors who include co-authors of the Pandas library, Lance published its research paper at VLDB 2025 — a signal of serious academic and engineering credibility.

The philosophy behind Lance is simple: make data storage behave like memory, not a warehouse. Everything in its architecture is optimized for low-latency access, reproducibility, and AI-native data patterns.

Core Architecture: Fragments, Not Row Groups

At the heart of Lance is a fundamentally different layout. Instead of Parquet’s fixed row groups, Lance stores data in fragments — small, self-contained columnar chunks (~64MB by default), each with its own statistics and optional indexes.

A Lance dataset on disk looks like this:

my_dataset.lance/
├── data/
│   ├── fragment_0.lance    ← columnar data chunk
│   ├── fragment_1.lance
│   └── fragment_2.lance
├── _latest.manifest        ← current version pointer
└── _versions/             ← version history (append-only)
    ├── 1.manifest
    └── 2.manifest

Crucially, Lance treats its dataset as always having a single row group conceptually, while achieving parallel scans through fragment-level parallelism — without the rigid sizing tradeoffs Parquet imposes.

The Shining Features of Lance

1. Lightning-Fast Random Access — 100x faster than Parquet

Lance achieves random access at the row level without loading entire pages. Its adaptive structural encodings — mini-block for small data types and full-zip for large multimodal data — allow individual rows to be accessed with minimal read amplification (reading only the bytes you actually need). The VLDB 2025 paper demonstrates this delivers up to 100x faster random access than Parquet or Iceberg without sacrificing sequential scan performance.

2. Native Multimodal Data Support

Lance stores images, videos, audio, text, and embeddings in a single unified format using efficient blob encoding and lazy loading. A computer vision dataset with raw frames, captions, and CLIP embeddings can live in one Lance table — no gluing together separate systems. Lance v2.2 demonstrates up to 68x faster blob reads than Parquet while cutting storage by 50%+.

3. Built-in Vector + Hybrid Search

This is Lance’s most disruptive feature for AI pipelines. Lance natively indexes vectors (HNSW-based ANN search), supports BM25 full-text search, and combines both with SQL predicates in a single hybrid query — all without an external service. The dream of asking “find the 10 most semantically similar documents, where category = ‘finance’ and created_after = ‘2024–01–01’” against a flat file is now reality.

4. Zero-Copy Versioning and Time Travel

Every write to a Lance dataset creates a new version via an append-only manifest log — similar to how Git tracks commits. This gives you ACID transactions, time travel, and reproducible experiments without a Hive Metastore, a catalog service, or a table format wrapper. Lance v2.2 introduced Git-style branching and shallow cloning, making dataset experimentation feel like code branching.

5. Zero-Copy Column Evolution

Adding a derived column (embeddings, features, predictions) to a Lance dataset is a metadata-only operation. Only new data is written. The expensive existing data — raw images, video frames, audio blobs — remains untouched. In Parquet-backed systems, this typically means a full table rewrite.

6. Arrow-Native and Engine-Agnostic

Lance is built on Apache Arrow at its core. Zero-copy data sharing with any Arrow-compatible engine is a first-class feature. The ecosystem integrations are real and growing: Apache Spark, Ray, PyTorch, DuckDB, Trino, Apache Flink, Polars, Pandas, and open catalogs including Apache Polaris, Unity Catalog, and Apache Gravitino.

Lance vs. Parquet: A Direct Comparison

Dimension Apache Parquet Lance Primary design goal Batch analytics (OLAP) AI/ML workloads + analytics Random access Slow — full page loads (~1MB) Fast — row-level precision, 100x improvement Vector/embedding support None native First-class, ANN-indexed Multimodal blobs Workaround via repeated fields Native blob encoding, lazy loading Schema evolution Full rewrite for new columns Zero-copy column addition Versioning Relies on Iceberg/Delta Lake Built-in, zero-cost, Git-style S3 request pattern Chatty (dozens of sequential GETs) Optimized range requests Hybrid search Not possible without external systems Native (vector + FTS + SQL) Ecosystem maturity Decade+ of support everywhere Growing fast; DuckDB, Trino, Spark supported BI tool support Universal Limited (growing) Best motto Iceberg for BI Lance for AI

• Design Goal: Parquet → Batch OLAP; Lance → AI/ML + analytics.
• Random Access: Parquet slow (page loads); Lance row-level fast.
• Vectors/Blobs: Parquet workaround; Lance native + indexed.
• Schema Evolution: Parquet often full rewrite; Lance zero-copy.
• Versioning: Parquet relies on Iceberg/Delta; Lance built-in.
• S3 Pattern: Parquet chatty; Lance optimized range requests.
• Hybrid Search: Parquet needs externals; Lance native.
• Ecosystem maturity: Parquet Decade+ of support everywhere; Lance Growing fast; DuckDB, Trino, Spark supported
• BI tool support: Parquet Universal; Lance Limited (growing)

The last row deserves emphasis. Chang She, CEO of LanceDB, puts it plainly: “Lance for AI and Iceberg for BI.” Lance is not trying to replace Parquet for traditional data warehousing — it’s carving out the AI-native territory that Parquet was never built for.

Using Lance with DuckDB

This is where things get immediately practical. Lance became a core extension in DuckDB — meaning you install it once and query .lance datasets with plain SQL. No JVM, no Spark cluster required.

Setup

-- Install once
INSTALL lance;
LOAD lance;

Read a Lance Dataset Like a Table

-- Local file
SELECT * FROM 'path/to/my_dataset.lance' LIMIT 10;

-- S3 (with credential setup)
CREATE SECRET (
  TYPE lance,
  PROVIDER credential_chain,
  SCOPE 's3://my-bucket/'
);

SELECT * FROM 's3://my-bucket/embeddings.lance'
WHERE created_at > '2025-01-01'
LIMIT 100;

Write DuckDB Query Results to Lance

COPY (
  SELECT user_id, event_type, embedding_vector
  FROM raw_events
  WHERE partition_date = '2025-04-01'
) TO 'output.lance' (FORMAT lance, MODE 'overwrite');

-- Append new data
COPY (
  SELECT * FROM new_batch
) TO 'output.lance' (FORMAT lance, MODE 'append');

Vector Search via SQL Table Function

SELECT
  user_id,
  content,
  _distance
FROM lance_vector_search(
  'embeddings.lance',
  'embedding',                          -- vector column name
  [0.12, -0.85, 0.34, ...]::FLOAT[],   -- query vector
  k = 20,                               -- top-k results
  prefilter = true
)
ORDER BY _distance ASC;

Attach a Lance Namespace for Multi-Table Workflows

ATTACH './lance_catalog' AS lns (TYPE LANCE);

-- Create and populate tables
CREATE TABLE lns.main.features AS
SELECT * FROM source_data;

-- Join Lance tables with regular DuckDB tables
SELECT f.user_id, f.embedding, m.label
FROM lns.main.features f
JOIN local_metadata m ON f.user_id = m.user_id;

The DuckDB-Lance integration means you can build complete feature pipelines, RAG preprocessing steps, and embedding workflows entirely in SQL — with Lance as the durable storage layer and DuckDB as the compute layer.

Python: Convert Parquet to Lance in 2 Lines

import lance
import pyarrow.dataset as pa_dataset

# Read existing Parquet
parquet = pa_dataset.dataset("/path/to/data.parquet", format="parquet")

# Write as Lance — that's it
lance.write_dataset(parquet, "/path/to/data.lance")

Using Lance with Trino

Trino support for Lance is active and production-ready through the Lance catalog integration. The recommended path connects via Unity Catalog, Apache Polaris, or Apache Gravitino as the catalog layer, with Lance as the underlying file format.

# trino/catalog/lance.properties
connector.name=iceberg
hive.metastore=glue  # or polaris/gravitino REST catalog
# Lance datasets registered as Iceberg-compatible tables

For teams already running Trino on an Iceberg stack, Lance tables can be registered in the catalog alongside Parquet-backed Iceberg tables. The query engine sees them through the same catalog interface — the difference is that Lance-backed tables will exhibit dramatically better performance for AI workload patterns: selective reads, vector retrieval, and multimodal data access.

The LanceDB team’s stated catalog roadmap supports Apache Polaris, Unity Catalog, and Apache Gravitino — covering the major Trino catalog backends teams already use. For pure Trino-on-Iceberg shops, the practical path is: use Iceberg for BI tables, use Lance-native tables for AI/ML tables, governed through a shared catalog.

The Honest Shortcomings

Lance is genuinely exciting engineering. It is also a young project, and intellectual honesty requires acknowledging real limitations:

Ecosystem immaturity relative to Parquet. Parquet has a decade of battle-hardened support across every BI tool, visualization platform, and query engine on the planet. Lance’s integration list is growing fast — but if you need Tableau, Power BI, or legacy reporting tools to read your files directly, Parquet is still the only answer.

Compaction overhead. As you append data to Lance, fragments accumulate. Without regular compaction runs, query latency increases as the engine must open and scan many small files. Compaction merges fragments — but temporarily increases disk usage, as old versions are retained until explicitly cleaned. This requires operational discipline that Parquet (being append-only-by-design) doesn’t demand.

Sparse data compression is weaker. For sparse matrix data (e.g., single-cell genomics COO format), Lance v2.x achieves significantly less compression than Parquet with Zstd/Snappy. If your data is sparse and compression ratio matters more than random access speed, Lance may not be the right fit yet.

Breaking changes at the SDK level. Lance SDK 1.0.0 was released in December 2025, signaling production maturity. However, the team is explicit: because Lance depends on DataFusion at a low level, major SDK version bumps (with API-level breaking changes) can come reasonably often. Data on disk remains stable, but code relying on Lance’s Python or Java SDK may require updates.

Traditional OLAP queries are not Lance’s sweet spot. Full-column scans across billions of rows — the workload Parquet was born for — are where Parquet still wins or matches Lance. Lance’s value accretes as access patterns become more search-like, random, or multimodal.

BI tool and governance tooling gaps. Data lineage, column-level access control, and observability tooling in the Parquet/Iceberg ecosystem is deep and mature. Lance’s governance story is still being written, primarily through integration with existing catalog layers.

When to Choose Lance (and When Not To)

Reach for Lance when:

• You’re building RAG pipelines, semantic search, or vector-backed applications
• Your training datasets mix structured features with embeddings, images, or audio
• You need fast point lookups on millions of records (inference time feature serving)
• You want built-in dataset versioning for reproducible ML experiments
• You’re adding derived columns (embeddings, predictions) to existing datasets without rewrites

Stick with Parquet (and Iceberg) when:

• Your primary consumers are BI tools, SQL analytics, and reporting dashboards
• Your data warehouse is already Iceberg/Delta-native and working well
• You need maximum compression for sequential batch scan workloads
• Your organization requires established governance, lineage, and audit tooling
• You operate in a pure OLAP environment with no AI/ML component

The honest answer for most modern data platforms: you’ll run both. Iceberg on Parquet for your analytical layer, Lance for your AI layer. The same open catalogs — Polaris, Unity Catalog, Gravitino — can govern both.

Closing Thoughts

The Parquet era isn’t ending — it’s bifurcating. For the analytical patterns it was designed for, Parquet remains exceptional. But data engineering is no longer just analytics. It’s embeddings and RAG and model training and feature stores and multimodal pipelines.

Lance represents a genuinely different bet on what a data format should do: bring vector search, random access, versioning, and multimodal storage into a single open file format — and make it work with the engines (DuckDB, Trino, Spark) your team already operates.

For data engineers navigating the AI era, Lance is no longer a curiosity. It’s becoming infrastructure.

Siddique Ahmad is a Senior Data Engineer and Applied AI Consultant with 15+ years of experience building data platforms on open-source stacks including Apache Iceberg, Trino, ClickHouse, and DuckDB. He writes about practical data engineering, LLM integration, and the evolving lakehouse landscape.

Found this useful? Follow for more posts on the intersection of data engineering and AI infrastructure.