Variant Shred Coverage Report: See What Iceberg Actually Shredded Fully VS partially Shredded

TL;DR: Iceberg v3 VARIANT columns look like one field in DESCRIBE TABLE, but Parquet may store dozens of shredded sub-columns underneath. This free tool reads your data files and tells you what's FULLY shredded, PARTIALLY shredded, or NOT shredded — and which variant_get paths your query engine can actually optimize.

[embed]

The problem

You enabled variant shredding:

CREATE TABLE events (
  id BIGINT,
  v VARIANT
)
USING iceberg
TBLPROPERTIES (
  'format-version' = '3',
  'write.parquet.shred-variants' = 'true'
);

You write JSON. You query with:

SELECT variant_get(v, '$.type', 'string')
FROM events
WHERE variant_get(v, '$.actor.login', 'string') = 'octocat';

But did shredding actually help?

• DESCRIBE TABLE still shows one VARIANT column
• Some JSON paths get Parquet columns (typed_value)
• Some stay in binary (value)
• You can’t tell from SQL alone

Without knowing the physical layout, you might filter on paths that still force a full variant read.

The tool: Variant Shred Coverage Report

Script: tools/variant_shred_audit.py

What it does:

Based on the Parquet Variant Shredding specification:

• FULL → field.value is NULL, data is in typed_value → column projection + stats skipping work
• PARTIAL → column exists, but some rows still store binary in field.value
• NOT SHREDDED → no typed_value column; entire JSON lives in variant binary

Install and run

Requirements: Python 3 + PyArrow (no Spark needed for --parquet-dir mode)

pip install pyarrow

Mode 1: Parquet directory (fastest, no Spark)

python3 tools/variant_shred_audit.py \
  --parquet-dir /path/to/warehouse/demo/my_table/data \
  --variant-col v

Or use the wrapper:

./tools/variant_shred_coverage.sh \
  --parquet-dir /path/to/table/data \
  --variant-col v

Mode 2: Iceberg table via spark-submit (recommended)

Configure warehouse + packages via env vars — nothing hardcoded:

export ICEBERG_WAREHOUSE=file:///path/to/warehouse
export ICEBERG_PACKAGES=org.apache.iceberg:iceberg-spark-runtime-4.1_2.13:1.11.0
export ICEBERG_CATALOG=dev

./tools/variant_shred_coverage.sh \
  --table dev.demo.github_with_shredding \
  --variant-col v

Or raw spark-submit:

spark-submit \
  --packages org.apache.iceberg:iceberg-spark-runtime-4.1_2.13:1.11.0 \
  --conf spark.sql.catalog.dev=org.apache.iceberg.spark.SparkCatalog \
  --conf spark.sql.catalog.dev.type=hadoop \
  --conf spark.sql.catalog.dev.warehouse=file:///path/to/warehouse \
  --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
  tools/variant_shred_audit.py \
  --use-existing-spark \
  --table dev.demo.github_with_shredding \
  --variant-col v

Mode 3: Compare two tables (parquet dirs)

python3 tools/variant_shred_audit.py \
  --no-shred-dir /path/to/github_no_shredding/data \
  --with-shred-dir /path/to/github_with_shredding/data \
  --variant-col v

Optional Spark config (standalone Python + — table)

python3 tools/variant_shred_audit.py \
  --table dev.demo.my_table \
  --warehouse file:///path/to/warehouse \
  --packages org.apache.iceberg:iceberg-spark-runtime-4.1_2.13:1.11.0 \
  --catalog dev \
  --variant-col v

Example 1: Shredding OFF

python3 tools/variant_shred_audit.py \
  --parquet-dir /path/to/github_no_shredding/data \
  --variant-col v

Output:

==============================================================================================================
VARIANT SHRED COVERAGE REPORT
==============================================================================================================
  Location:      .../github_no_shredding/data
  Variant col:   v
  Parquet scan:  COMPLETE — 1 file(s), 180,387 rows
  Method:        Parquet metadata + column null statistics (not row sampling)

==============================================================================================================
RESULT: NOT SHREDDED
==============================================================================================================
  No typed_value columns found for variant column 'v'.
  Parquet layout: id, v.metadata, v.value  (3 columns)
  All 180,387 rows store the full JSON payload in v.value binary.

  Query impact:
    variant_get() works logically, but every access reads the full variant binary
    No column projection or Parquet stats skipping on JSON sub-fields
    Query benefit: NONE for all paths

  To enable shredding on new writes:
    ALTER TABLE <table> SET TBLPROPERTIES (
      'format-version' = '3',
      'write.parquet.shred-variants' = 'true'
    )

How this helps you: One glance — don’t expect query speedups from variant_get filters. Every path reads the full blob.

Example 2: Shredding ON

Same data with shredding enabled:

python3 tools/variant_shred_audit.py \
  --parquet-dir /path/to/github_with_shredding/data \
  --variant-col v

==============================================================================================================
VARIANT SHRED COVERAGE REPORT
==============================================================================================================
  Location:      /Users/sshah/IdeaProjects/study-learn/warehouse/demo/github_with_shredding/data
  Variant col:   v
  Parquet scan:  COMPLETE — 1 file(s), 180,387 rows
  Method:        Parquet metadata + column null statistics (not row sampling)
  B2 add-on:     off (unshredded path list requires --scan-rows)

  Parquet columns:  80
  Shredded paths:   35
  Root FULL rows:   180,387 (100.0%)  ← v.value is NULL
  Root PARTIAL rows: 0 (0.0%)  ← v.value has binary

SHREDDED PATHS (extracted to typed_value columns)
----------------------------------------------------------------------------------------------------
JSON PATH                                        FILES   COVERAGE       STATUS
----------------------------------------------------------------------------------------------------
actor.avatar_url                                     1       100%     SHREDDED
actor.display_login                                  1       100%     SHREDDED
actor.gravatar_id                                    1       100%     SHREDDED
actor.id                                             1       100%     SHREDDED
actor.login                                          1       100%     SHREDDED
actor.url                                            1       100%     SHREDDED
created_at                                           1       100%     SHREDDED
id                                                   1       100%     SHREDDED
org.avatar_url                                       1       100%     SHREDDED
org.gravatar_id                                      1       100%     SHREDDED
org.id                                               1       100%     SHREDDED
org.login                                            1       100%     SHREDDED
org.url                                              1       100%     SHREDDED
payload.action                                       1       100%     SHREDDED
payload.before                                       1       100%     SHREDDED
payload.commits[][].author.email                     1       100%     SHREDDED
payload.commits[][].author.name                      1       100%     SHREDDED
payload.commits[][].distinct                         1       100%     SHREDDED
payload.commits[][].message                          1       100%     SHREDDED
payload.commits[][].sha                              1       100%     SHREDDED
payload.commits[][].url                              1       100%     SHREDDED
payload.distinct_size                                1       100%     SHREDDED
payload.head                                         1       100%     SHREDDED
payload.master_branch                                1       100%     SHREDDED
payload.push_id                                      1       100%     SHREDDED
payload.pusher_type                                  1       100%     SHREDDED
payload.ref                                          1       100%     SHREDDED
payload.ref_type                                     1       100%     SHREDDED
payload.repository_id                                1       100%     SHREDDED
payload.size                                         1       100%     SHREDDED
public                                               1       100%     SHREDDED
repo.id                                              1       100%     SHREDDED
repo.name                                            1       100%     SHREDDED
repo.url                                             1       100%     SHREDDED
type                                                 1       100%     SHREDDED

NESTED PARTIAL SHRED (object still has leftover binary in .value)
----------------------------------------------------------------------------------------------------
JSON PATH / OBJECT                              PARTIAL ROWS  PARTIAL %
----------------------------------------------------------------------------------------------------
payload                                              180,387     100.0%
payload.distinct_size                                118,886      65.9%
payload.size                                         118,838      65.9%
actor.id                                                 464       0.3%
org.id                                                    74       0.0%
repo.id                                                   12       0.0%
payload.repository_id                                      2       0.0%

PER FILE
----------------------------------------------------------------------------------------------------
FILE                                               ROWS   COLS  ROOT FULL%   #PATHS           MODE
----------------------------------------------------------------------------------------------------
...37-4115-a60e-5925b82d1323-0-00001.parquet    180,387     80      100.0%       35    FULL (root)
==============================================================================================================
A — SPEC SUMMARY (Parquet VariantShredding.md)
==============================================================================================================
  FULLY SHREDDED field  → field.value is NULL, field.typed_value holds the value
                         → enables column projection + stats-based data skipping
  PARTIALLY SHREDDED    → field.value has binary on some rows (type mismatch / fallback)
                         OR parent object.value has binary (un-inferred subfields)
  NOT SHREDDED          → no typed_value column; value lives only in variant binary

  Files inspected:           1
  Rows inspected (Parquet):  180,387
  Shredded paths in schema:  35
    FULL (spec):             29  (83% of shredded)
    PARTIAL (spec):          6  (17% of shredded)
  Root variant FULL:         100.0% rows (v.value NULL — object fields in typed_value)

==============================================================================================================
B — SHREDDED PATHS: FULL vs PARTIAL (all files, spec-aligned)
==============================================================================================================
JSON PATH                                      FILES  ROW FULL%       SPEC    QUERY
--------------------------------------------------------------------------------------------------------------
actor.avatar_url                                 1/1     100.0%       FULL     HIGH
actor.display_login                              1/1     100.0%       FULL     HIGH
actor.gravatar_id                                1/1     100.0%       FULL     HIGH
actor.id                                         1/1      99.7%    PARTIAL   MEDIUM
actor.login                                      1/1     100.0%       FULL     HIGH
actor.url                                        1/1     100.0%       FULL     HIGH
created_at                                       1/1     100.0%       FULL     HIGH
id                                               1/1     100.0%       FULL     HIGH
org.avatar_url                                   1/1     100.0%       FULL     HIGH
org.gravatar_id                                  1/1     100.0%       FULL     HIGH
org.id                                           1/1     100.0%    PARTIAL   MEDIUM
org.login                                        1/1     100.0%       FULL     HIGH
org.url                                          1/1     100.0%       FULL     HIGH
payload.action                                   1/1     100.0%       FULL     HIGH
payload.before                                   1/1     100.0%       FULL     HIGH
payload.commits[][].author.email                 1/1       0.0%       FULL     HIGH
payload.commits[][].author.name                  1/1       0.0%       FULL     HIGH
payload.commits[][].distinct                     1/1       0.0%       FULL     HIGH
payload.commits[][].message                      1/1       0.0%       FULL     HIGH
payload.commits[][].sha                          1/1       0.0%       FULL     HIGH
payload.commits[][].url                          1/1       0.0%       FULL     HIGH
payload.distinct_size                            1/1      34.1%    PARTIAL      LOW
payload.head                                     1/1     100.0%       FULL     HIGH
payload.master_branch                            1/1     100.0%       FULL     HIGH
payload.push_id                                  1/1     100.0%       FULL     HIGH
payload.pusher_type                              1/1     100.0%       FULL     HIGH
payload.ref                                      1/1     100.0%       FULL     HIGH
payload.ref_type                                 1/1     100.0%       FULL     HIGH
payload.repository_id                            1/1     100.0%    PARTIAL   MEDIUM
payload.size                                     1/1      34.1%    PARTIAL      LOW
public                                           1/1     100.0%       FULL     HIGH
repo.id                                          1/1     100.0%    PARTIAL   MEDIUM
repo.name                                        1/1     100.0%       FULL     HIGH
repo.url                                         1/1     100.0%       FULL     HIGH
type                                             1/1     100.0%       FULL     HIGH

  (B2 skipped — pass --table + --scan-rows N or --json-file for unshredded path frequency)

==============================================================================================================
C — QUERY BENEFIT SUMMARY (which filters/projections win)
==============================================================================================================
  HIGH benefit (29 paths) — use these in WHERE / SELECT (typed_value + stats skipping):
    • variant_get(v, '$.actor.avatar_url', ...)  [100% row FULL, 100% files]
    • variant_get(v, '$.actor.display_login', ...)  [100% row FULL, 100% files]
    • variant_get(v, '$.actor.gravatar_id', ...)  [100% row FULL, 100% files]
    • variant_get(v, '$.actor.login', ...)  [100% row FULL, 100% files]
    • variant_get(v, '$.actor.url', ...)  [100% row FULL, 100% files]
    • variant_get(v, '$.created_at', ...)  [100% row FULL, 100% files]
    • variant_get(v, '$.id', ...)  [100% row FULL, 100% files]
    • variant_get(v, '$.org.avatar_url', ...)  [100% row FULL, 100% files]
    • variant_get(v, '$.org.gravatar_id', ...)  [100% row FULL, 100% files]
    • variant_get(v, '$.org.login', ...)  [100% row FULL, 100% files]
    • variant_get(v, '$.org.url', ...)  [100% row FULL, 100% files]
    • variant_get(v, '$.payload.action', ...)  [100% row FULL, 100% files]
    • variant_get(v, '$.payload.before', ...)  [100% row FULL, 100% files]
    • variant_get(v, '$.payload.commits[][].author.email', ...)  [0% row FULL, 100% files]
    • variant_get(v, '$.payload.commits[][].author.name', ...)  [0% row FULL, 100% files]
    ... +14 more

  MEDIUM benefit (4 paths) — column exists, partial fallback or not all files:
    • $.actor.id  (FULL 100%, files 100%)
    • $.org.id  (FULL 100%, files 100%)
    • $.payload.repository_id  (FULL 100%, files 100%)
    • $.repo.id  (FULL 100%, files 100%)

  LOW benefit (2 paths) — shredded column but heavy binary fallback:
    • $.payload.distinct_size  (65.9% rows still in field.value binary)
    • $.payload.size  (65.9% rows still in field.value binary)

  Rule of thumb:
    HIGH   → filter/project on this path; Iceberg/Parquet can skip row groups
    NONE   → variant_get still works logically, but reads full variant binary
    Prefer HIGH paths for dashboard filters; avoid NONE paths in hot queries
sshah@D345YG773G iceberg % 

Stop guessing. Read the Parquet file once. Know what’s shredded.

Code

https://github.com/soumilshah1995/variant_shred_audit