Creating Icerberg Table in MinIO with Spark

Prerequisites

• Colima: Version 0.7.0 or later for Docker runtime.
• Docker CLI and Docker Compose: For running Docker commands.

Project Structure

project/
├── notebooks/
│   └── iceberg_example.ipynb
├── warehouse/
└── docker-compose.yml

Set Up the Docker Compose Environment

• Install Colima: brew install colima
• Install Docker CLI: brew install docker
• Start Colima with sufficient resources for Spark and MinIO: colima start --cpu 6 --memory 12 --disk 20

Create a docker-compose.yml file with the following content to set up Spark, Iceberg REST catalog, and MinIO:

services:
  spark-iceberg:
    image: tabulario/spark-iceberg
    container_name: spark-iceberg
    networks:
      iceberg_net:
    depends_on:
      - rest
      - minio
    volumes:
      - ./warehouse:/home/iceberg/warehouse
      - ./notebooks:/home/iceberg/notebooks/notebooks
    environment:
      - AWS_ACCESS_KEY_ID=admin
      - AWS_SECRET_ACCESS_KEY=password
      - AWS_REGION=us-east-1
    ports:
      - 8888:8888
      - 8080:8080
      - 10000:10000
      - 10001:10001
  rest:
    image: tabulario/iceberg-rest
    container_name: iceberg-rest
    networks:
      iceberg_net:
    ports:
      - 8181:8181
    environment:
      - AWS_ACCESS_KEY_ID=admin
      - AWS_SECRET_ACCESS_KEY=password
      - AWS_REGION=us-east-1
      - CATALOG_WAREHOUSE=s3://warehouse/
      - CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO
      - CATALOG_S3_ENDPOINT=http://minio:9000
  minio:
    image: minio/minio
    container_name: minio
    environment:
      - MINIO_ROOT_USER=admin
      - MINIO_ROOT_PASSWORD=password
      - MINIO_DOMAIN=minio
    networks:
      iceberg_net:
        aliases:
          - warehouse.minio
    ports:
      - 9001:9001
      - 9000:9000
    command: ["server", "/data", "--console-address", ":9001"]
  mc:
    depends_on:
      - minio
    image: minio/mc
    container_name: mc
    networks:
      iceberg_net:
    environment:
      - AWS_ACCESS_KEY_ID=admin
      - AWS_SECRET_ACCESS_KEY=password
      - AWS_REGION=us-east-1
    entrypoint: >
      /bin/sh -c "
      until (/usr/bin/mc alias set minio http://minio:9000 admin password) do echo '...waiting...'; sleep 1; done;
      /usr/bin/mc mb minio/warehouse --ignore-existing;
      tail -f /dev/null"
networks:
  iceberg_net:

This configuration:

• Uses the tabulario/spark-iceberg image for Spark with Iceberg support.
• Sets up a MinIO server for object storage.
• Configures an Iceberg REST catalog to manage table metadata.
• Maps ports for Jupyter Notebook (8888), Spark UI (8080), and MinIO (9000, 9001).
• Creates a warehouse bucket in MinIO for storing Iceberg data and metadata.

Start the Docker Environment

docker compose up -d

Wait for the services to start. You can verify:

• MinIO is accessible at http://localhost:9001 (login with admin/password).
• Jupyter Notebook is accessible at http://localhost:8888.

Create a New Iceberg Table in the Notebook

In the Jupyter Notebook, add the following code to create a new Iceberg table using the NYC Taxi dataset (or any dataset of your choice). This example creates a table, loads sample data, and stores it in the MinIO warehouse bucket.

# Import required libraries
from pyspark.sql import SparkSession

# Configure Spark session
spark = SparkSession.builder \
    .appName("IcebergTableCreation") \
    .config("spark.sql.catalog.demo", "org.apache.iceberg.spark.SparkCatalog") \
    .config("spark.sql.catalog.demo.type", "rest") \
    .config("spark.sql.catalog.demo.uri", "http://iceberg-rest:8181") \
    .config("spark.sql.catalog.demo.io-impl", "org.apache.iceberg.aws.s3.S3FileIO") \
    .config("spark.sql.catalog.demo.warehouse", "s3://warehouse/") \
    .config("spark.sql.catalog.demo.s3.endpoint", "http://minio:9000") \
    .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") \
    .config("spark.hadoop.fs.s3a.access.key", "admin") \
    .config("spark.hadoop.fs.s3a.secret.key", "password") \
    .config("spark.hadoop.fs.s3a.endpoint", "http://minio:9000") \
    .getOrCreate()
# Create a database
spark.sql("CREATE DATABASE IF NOT EXISTS demo.nyc")
# Create an Iceberg table
spark.sql("""
CREATE TABLE demo.nyc.taxis (
    vendor_id BIGINT,
    trip_id BIGINT,
    trip_distance FLOAT,
    fare_amount DOUBLE,
    store_and_fwd_flag STRING
)
USING iceberg
PARTITIONED BY (vendor_id)
LOCATION 's3://warehouse/nyc/taxis'
""")
# Insert sample data
spark.sql("""
INSERT INTO demo.nyc.taxis
VALUES 
    (1, 1000371, 1.8, 15.32, 'N'),
    (2, 1000372, 2.5, 22.15, 'N'),
    (2, 1000373, 0.9, 9.01, 'N'),
    (1, 1000374, 8.4, 42.13, 'Y')
""")
# Verify the data
spark.sql("SELECT * FROM demo.nyc.taxis").show()
# Stop the Spark session
spark.stop()

If you want to use a real dataset like the NYC Taxi dataset (~112M rows, 10GB), download it from MinIO’s public dataset or another source and upload it to the warehouse bucket using the MinIO console or mc commands. Then, modify the notebook to read the data (e.g., spark.read.parquet("s3://warehouse/your_dataset.parquet").write.saveAsTable("demo.nyc.taxis")).

Verify the Table in MinIO

• Open the MinIO console at http://localhost:9001 and log in with admin/password.
• Navigate to the warehouse bucket. You should see a folder structure like nyc/taxis/ containing:
• data/: Parquet files with the actual table data.
• metadata/: Iceberg metadata files (e.g., .metadata.json, snap-.avro).

Clean Up

To stop and remove the Docker containers: docker compose down