Migrating data from HBase to Databricks using HBase Snapshots

Photo by Fr. Daniel Ciucci on Unsplash

Introduction

Migrating large HBase tables to Databricks can be challenging, especially when dealing with tables in the tens of terabytes. Common approaches like the Spark HBase Connector often run into issues with Kerberos authentication, excessive load on the HBase cluster, inconsistent data, and timeouts.

This guide walks through a battle-tested approach using HBase Snapshots combined with Apache Spark’s TableSnapshotInputFormat to perform a reliable, scalable, and minimally disruptive migration. The technique was developed and validated during a real-world migration of a 70 TB HBase table from a Cloudera-managed cluster to Databricks on Azure.

Why Snapshots?

Using HBase Snapshots for migration offers several advantages over direct HBase reads:

• Consistent point-in-time view of the table data
• Zero load on the HBase RegionServers during the read — the snapshot is read directly from the underlying HFiles
• Scalable parallelism — Spark can read the snapshot with as many tasks as there are HBase regions
• No Kerberos dependency — once exported to cloud storage, no HBase cluster connectivity is needed from Databricks

Any other method of importing large amounts of data from HBase is likely to lead to issues such as unnecessary load on the HBase cluster, inconsistent data, and timeouts.

Migration Process Overview

The end-to-end migration follows these 5 broad steps:

• Create the snapshot in HBase
• Export the snapshot from HBase to cloud storage (e.g., ADLS, S3)
• Delete the snapshot in HBase (to free up storage)
• Ingest the exported snapshot into Databricks using Spark
• Delete the exported snapshot from cloud storage (cleanup)

Step 1: Create and Export the HBase Snapshot

Creating the Snapshot

On your HBase cluster (e.g., Cloudera), create a snapshot of the target table:

hbase shell  
 > snapshot 'your_table_name', 'your_snapshot_name'

Storage considerations:

• Creating the snapshot itself does not consume additional storage — it is a metadata-only operation.
• If the HBase table is updated and flushed after the snapshot is created, the older HFiles will be preserved until the snapshot is deleted. This is the only source of extra storage consumption.
• Unless the table is experiencing heavy updates to existing data, the additional storage should not be a major concern. A modest number of older HFiles are typically retained until the next compaction and cleanup process.
• Tip: The best time to create the snapshot is when the table is not actively being updated, to minimize extra HFile retention.

Exporting the Snapshot to Cloud Storage

Use the HBase ExportSnapshot tool to copy the snapshot to your cloud storage (e.g., Azure Data Lake Storage):

hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot \  
   -snapshot your_snapshot_name \  
   -copy-to abfss://container@storageaccount.dfs.core.windows.net/hbase-snapshots/ \  
   -mappers 16

Data transfer considerations for large tables:

• For very large tables, the export can take significant time depending on network bandwidth.
• Microsoft provides guidance on data transfer options for large datasets: Azure Storage Solutions for Large Datasets.
• To avoid impacting production workloads, you can configure the distcp/export job to run in a separate YARN queue with a limited set of resources.
• If a single node can fully utilize the available bandwidth, running distcp locally outside of YARN is also an option.

Delete the Snapshot in HBase

Once the export is confirmed complete:

hbase shell  
 > delete_snapshot 'your_snapshot_name'

Step 2: Setting Up the Databricks Cluster

Required JAR Files

The following JAR files are needed to read HBase snapshots in Databricks. Note that the versions must match (or be compatible with) your source HBase version:

For HBase 2.6.x (upstream Apache HBase):

• hbase-shaded-mapreduce-2.6.3.jar
• opentelemetry-semconv-1.29.0-alpha.jar
• opentelemetry-context-1.49.0.jar
• opentelemetry-api-1.49.0.jar

For Cloudera HBase (e.g., CDP 7.1.7):

• hbase-shaded-mapreduce-2.2.3.7.1.7-2032-1.jar (Cloudera-specific version)
• opentelemetry-api-0.12.0.jar
• opentelemetry-context-0.12.0.jar Important: Check Maven for compatible OpenTelemetry versions that match your HBase distribution. The Cloudera-specific HBase shaded MapReduce JAR may require different OpenTelemetry versions than the upstream Apache HBase JAR.

JAR Installation: Use an Init Script (Not Cluster Libraries)

Do not install the JAR files as cluster libraries. Hadoop and HBase use reflection to load metrics classes, which causes a class loader conflict when JARs are provided via Databricks cluster libraries.

Instead, use a **cluster init script** to copy the JARs directly to /databricks/jars/:

#!/bin/bash  
 # init_script.sh - upload this to DBFS or a Unity Catalog Volume
cp /dbfs/path/to/your/jars/hbase-shaded-mapreduce-*.jar /databricks/jars/  
cp /dbfs/path/to/your/jars/opentelemetry-*.jar /databricks/jars/

Upload this script and configure it as an init script on your cluster.

Cluster Configuration

• Databricks Runtime: DBR 16.4 LTS (Scala 2.12) or later
• Access Mode: Single User or Dedicated (not Shared)
• Cluster type: Multi-node
• Worker sizing: Prefer more smaller workers over fewer large workers for better parallelism. For example, many medium-sized instances will outperform a few E64_v3 instances because the work is partitioned by HBase region.

Required Spark Configuration

Set the following Spark property in your cluster configuration:

spark.hadoopRDD.ignoreEmptySplits false

This ensures that incorrect split sizes(0) reported by the HFile format reader does not cause the regions to be skipped. This is required for correct snapshot reads.

A Note on reading HBase snapshots in Databricks

HBase snapshot reads may encounter issues when reading directly from ADLS/S3 or UC Volumes using UC storage credentials. A known workaround is to copy the snapshot files from Unity Catalog Volumes to DBFS and read the snapshot through the DBFS path. Alternatively you can also override the abfs/s3 file implementation to use static credentials.

Step 3: Read the Snapshot in Databricks

The code below is written in Scala and runs in a Databricks notebook. It uses TableSnapshotInputFormat with newAPIHadoopRDD to read the exported HBase snapshot.

Step 3a: Define the Snapshot Reader Function

This function configures HBase and Spark to read a snapshot from cloud storage:

import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.hbase.HBaseConfiguration
import org.apache.hadoop.hbase.client.{Result, Scan}
import org.apache.hadoop.hbase.io.ImmutableBytesWritable
import org.apache.hadoop.mapreduce.Job
import org.apache.hadoop.hbase.mapreduce.{IdentityTableMapper, TableSnapshotInputFormat, TableMapReduceUtil}
import org.apache.hadoop.fs.Path
import org.apache.spark.rdd.RDD

def hbaseTableSnapshotRDD(
    snapshotName: String,
    snapshotRoot: String,
    restoreRoot: String
): RDD[(ImmutableBytesWritable, Result)] = {

    val conf: Configuration = HBaseConfiguration.create(sc.hadoopConfiguration)

    conf.set("hbase.rootdir", snapshotRoot)
    conf.set("fs.defaultFS", snapshotRoot)

    // If using S3/ABFS override the filesystem implementation
    // conf.set("fs.s3a.impl", "shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem")
    // conf.set("fs.s3a.aws.credentials.provider",
    //     "shaded.databricks.org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider")
    // conf.set("fs.s3a.access.key", "<YOUR_ACCESS_KEY>")
    // conf.set("fs.s3a.secret.key", "<YOUR_SECRET_KEY>")
    // conf.set("fs.s3a.session.token", "<YOUR_SESSION_TOKEN>")

    val job = Job.getInstance(conf)

    val scan = new Scan
    scan.setScanMetricsEnabled(false)
    scan.setCaching(10000)
    scan.setCacheBlocks(false)

    TableMapReduceUtil.initTableSnapshotMapperJob(
        snapshotName,
        scan,
        classOf[IdentityTableMapper],
        null,
        null,
        job,
        false,
        new Path(restoreRoot))

    sc.newAPIHadoopRDD(
        job.getConfiguration,
        classOf[TableSnapshotInputFormat],
        classOf[ImmutableBytesWritable],
        classOf[Result])
}

Step 3b: Read the Snapshot

Point the function to the snapshot location and a temporary restore directory:

val snapshot_name = "your_snapshot_name"  
// Use dbfs or cloud storage locations  
//val snapshot_root = "s3a://your-bucket/hbase_snapshots/your_snapshot_name"  
//val restore_root  = "s3a://your-bucket/hbase_restored/your_snapshot_name"  
val snapshot_root = "dbfs:/hbase_snapshots/your_snapshot_name"  
val restore_root  = "dbfs:/hbase_restored/your_snapshot_name"
val hbaseRDD = hbaseTableSnapshotRDD(snapshot_name, snapshot_root, restore_root)

Note: If using cloud file systems, make sure appropriate overrides for FS implementation and static credentials are configured

Step 3c: Define the Schema and Convert to DataFrame

Map the HBase Result objects to Spark Rows using your table's column families and qualifiers:

import org.apache.spark.sql.types.{StructType, StructField, StringType}  
import org.apache.spark.sql.{DataFrame, Row}  
import org.apache.hadoop.hbase.util.Bytes
// Define the schema matching your HBase table columns  
 val schema = new StructType()  
   .add(StructField("row_key", StringType, false))  
   .add(StructField("first_name", StringType, true))  
   .add(StructField("last_name", StringType, true))  
   .add(StructField("company", StringType, true))  
   .add(StructField("city", StringType, true))  
   .add(StructField("country", StringType, true))  
   .add(StructField("email", StringType, true))
// Extract columns from HBase Result objects  
// Replace "cf" with your column family name and column names with your qualifiers  
val rowRDD = hbaseRDD.map { case (_, result: Result) =>  
   Row(  
     Bytes.toString(result.getRow),  
     Option(result.getValue("cf".getBytes, "first_name".getBytes)).map(Bytes.toString).orNull,  
     Option(result.getValue("cf".getBytes, "last_name".getBytes)).map(Bytes.toString).orNull,  
     Option(result.getValue("cf".getBytes, "company".getBytes)).map(Bytes.toString).orNull,  
     Option(result.getValue("cf".getBytes, "city".getBytes)).map(Bytes.toString).orNull,  
     Option(result.getValue("cf".getBytes, "country".getBytes)).map(Bytes.toString).orNull,  
     Option(result.getValue("cf".getBytes, "email".getBytes)).map(Bytes.toString).orNull  
   )  
 }
val df = spark.createDataFrame(rowRDD, schema)

Note: You will need to customize the schema, column family name ("cf"), and column qualifiers to match your specific HBase table structure.

Step 3d: Write to Delta Table

df.write.mode("overwrite").saveAsTable("your_catalog.your_schema.your_table")

Step 4: Performance Tuning

Understanding Parallelism

The default input splitter for TableSnapshotInputFormat partitions the work by HBase region. This means:

• The number of Spark tasks equals the number of regions in the HBase table
• All tasks can run concurrently if you have enough cluster resources

Diagnosing Slow Reads

If snapshot reads are taking longer than expected, investigate:

1. How many regions are in the table? Fewer regions means less parallelism.
2. What is the average size per region? Large regions take longer per task.
3. Are regions uniformly distributed, or are there skews? Skewed regions cause stragglers.
4. Are all tasks running concurrently? Check the Spark UI to see if tasks are waiting for executor allocation.

Sizing Guidelines

• Use more, smaller workers rather than fewer large ones — this gives Spark more executors to read regions in parallel.
• Autoscaling may not react quickly enough — consider starting with a fixed-size cluster based on your region count.

Custom Input Splitting

If your table has few, very large regions, the default per-region splitting may bottleneck parallelism. In this case, you can implement a custom input splitter to subdivide large regions into smaller splits. This requires additional development effort but can dramatically improve throughput for tables with uneven region distributions.

Step 5: Cleanup

After the data has been successfully ingested into Databricks and validated:

1. Verify the Delta table data quality and completeness
2. Delete the restored snapshot temporary directory
3. Delete the exported snapshot from cloud storage

Troubleshooting

ClassLoader / Reflection Errors

Symptom: Errors related to metrics class loading or reflection failures when reading the snapshot.

Cause: Hadoop/HBase use reflection to load metrics classes. When JARs are provided as cluster libraries, the Databricks class loader handles them differently, causing conflicts.

Fix: Use an init script to copy JARs to /databricks/jars/ instead of attaching them as cluster libraries.

JAR Version Mismatches

Symptom: NoSuchMethodError, ClassNotFoundException, or dependency errors.

Cause: The HBase shaded MapReduce JAR version doesn’t match the source HBase cluster version, or the OpenTelemetry JARs are incompatible.

Fix: Use the exact HBase version from your source cluster. Check Maven for compatible transitive dependency versions (especially OpenTelemetry).

HFile reader cannot read files on cloud storage

Symptom: Errors when reading snapshot files from ADLS/ABFSS/S3/Volume paths.

Cause: UC overrides cloud file system classes

Fix: Copy snapshot files from Volumes to DBFS and read from the DBFS path instead or override file system classes and use static credentials

Empty Splits Error

Symptom: Missing data or errors during snapshot scan.

Cause: Incorrect split size reported by the HFile format reader is causing the regions to be skipped.

Fix: Set spark.hadoopRDD.ignoreEmptySplits to false in the Spark configuration.

Summary

[embed]

This approach has been validated on tables up to 70 TB in production environments, migrating from Cloudera-managed HBase to Databricks on Azure. The key advantages are zero impact on the source HBase cluster during the read phase, consistent point-in-time data, and the ability to scale out reads across the full set of HBase regions.