Hello Ozone from Spark using s3a:// protocol

Quick dive into a basic POC with Dharmesh Jain and Anurag Dwivedi to test that s3a:// based connectivity can work between Apache Spark and Apache Ozone Storage systems. Note we are not enabling security here. The ozone container in this examples is running remotely from the machine where Spark is accessing the ozone file system.

Starting the ozone docker

docker run -d -p 9878:9878 -p 9876:9876 apache/ozone:1.3.0

Creating a bucket using aws CLI

aws s3api --endpoint http://localhost:9878 
create-bucket --bucket buckettest

# Executing above command is confirmed as below:
{
 "Location": "http://localhost:9878/buckettest"
}

Copying a file from local file system to bucket

aws s3 --endpoint http://localhost:9878 cp \
--storage-class REDUCED_REDUNDANCY  \
./employees.csv  s3://buckettest/employees.csv

# Executing above command is confirmed as below:
upload: ./employees.csv to s3://buckettest/employees.csv

where employees.csv is as below:

employee_name,employee_id
Ram,111
Shyam,222
Sita,333
Radha,444

Listing the file in the bucket

aws s3 --endpoint http://localhost:9878 ls s3://buckettest/employees.csv

#results in :
2024-06-22 06:56:59         63 employees.csv

Accessing ozone bucket from Spark

./pyspark \
--conf spark.hadoop.fs.s3a.endpoint=http://9.46.195.2:9878 \
--conf spark.hadoop.fs.s3a.connection.ssl.enabled=false \
--conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
--conf spark.hadoop.fs.s3a.path.style.access=true \
--conf spark.hadoop.fs.s3a.access.key=dummy \
--conf spark.hadoop.fs.s3a.secret.key=dummy \
--conf spark.hadoop.fs.s3a.bucket.probe=0 \
--conf spark.hadoop.fs.s3a.change.detection.version.required=false \
--conf spark.hadoop.fs.s3a.change.detection.mode=none

Read the file uploaded earlier

 spark.read.option("header",True).csv("s3a://buckettest/employees.csv").show()
+-------------+-----------+                                                     
|employee_name|employee_id|
+-------------+-----------+
|          Ram|        111|
|        Shyam|        222|
|         Sita|        333|
|        Radha|        444|
+-------------+-----------+

Note — in this case the values used for access key and secret key is actually dummy. You can use any value here and it would still work. This is because security has not been enabled on the ozone

“If security is not enabled, you can use any AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY”

You can use the same mechanism to connect to ozone from Analytics Engine Spark from IBM Cloud Pak for Data, IBM Cloud Serverless Spark and WatsonX.Data Spark Engine

References:

[embed]Documentation for Apache Ozone Apache Ozone Documentationozone.apache.org