Becoming Interview-Ready for Data Engineering Roles

Data Engineering Interview Prep Challenge: Day 20: Apache Spark: RDD and DataFrame.

Continuing my interview prep challenge, I am exploring Spark’s RDD and DataFrame.

Welcome to Day 20 of my Data Engineering Interview Prep Challenge! Yesterday, I completed Apache Spark fundamentals, gaining an understanding of the need for Spark and learning about the Spark ecosystem and its architecture. Today, I am going to dive further into Spark’s data abstraction, starting from lower-level fundamentals of RDDs and then moving to the higher-level API control offered by DataFrames.

Before we dive further, I just wanted to give a brief introductory understanding of SparkContext and SparkSession. They are the entry points to Spark, which needs a connection before we can start working with RDDs or DataFrames.

SparkContext and SparkSession

Think of these as your remote control for the entire Spark cluster.

SparkContext:

This is the main entry point for Spark functionality when working with RDDs. It’s like the ignition key for your Spark application.

What it does is:

• Connects the application to the Spark cluster.
• Coordinates all the Executors and manages resources
• Creates RDDs from a data source
• Broadcast variables and manage shared data

Creating a SparkContext in PySpark:

from pyspark import SparkContext

sc = SparkContext("local", "My App")
# Now you can create RDDs
rdd = sc.textFile("data.txt")

Note: You can have only one active SparkContext per JVM. If you try to create another, Spark will complain.

SparkSession:

It was introduced later into Spark (specifically in Spark 2.0) as a one-stop shop for all Spark functionality. It’s the newer and better way to work with Spark.

What it does is Everything SparkContext does, plus:

• Creates DataFrames and DataSets
• Executes SQL Queries
• Manages configuration
• Provides access to SparkContext (if you need RDDs)

Creating a SparkSession:

from pyspark.sql import SparkSession

spark = SparkSession.builder \ 
    .appName("My App") \
    .config("spark.some.config", "value") \
    .getOrCreate()

# Create DataFrames
df = spark.read.csv("data.csv")

# Execute SQL
spark.sql("SELECT * FROM table")

# Still need RDDs? Access SparkContext through SparkSession
sc = spark.sparkContext
rdd = sc.textFile("data.txt")

Always start with SparkSession in modern Spark applications. It’s the unified entry point that gives you access to everything.

What is an RDD (Resilient Distributed Dataset)?

Think of an RDD as a huge pile of puzzle pieces spread across multiple tables in different rooms. You can’t see all the pieces at once because they are distributed, but you have instructions (from your mom!) to work with all of them.

The Three Key Words Explained:

• Resilient: If someone accidentally knocks over one table and the pieces scatter, you can rebuild them because you remember the instructions (lineage)
• Distributed: The pieces are spread across multiple tables (machines), so many people can work simultaneously.
• Dataset: It’s just a collection of data, whether it could be numbers, texts, objects, or anything.

Key Characteristics:

1. Immutable (Read-Only): Once you create an RDD, you can’t change it. If you want to modify data, you create a NEW RDD. This makes things safer in distributed systems. Multiple processes can read the same data without worrying about someone else changing it.
2. Lazy Evaluation: RDDs are like a lazy college student (not me!) who takes notes on what to do but doesn’t actually start working until the deadline (when you call an action) (yes, it’s me!)
3. Type-Safe (In Scala/Java): RDDs know what type of data they contain. An RDD[String] contains strings, and an RDD[Int] contains integers. The compiler catches mistakes before runtime.
4. Low-Level Control: RDDs give you fine-grained control. You can tell Spark exactly how to partition data, cache specific operations, and control every transformation. It’s like driving a manual transmission car.

Common RDD Operations:

Transformation (Lazy):

• map() - Transform each element (like applying a function to every item)
• filter() - Keep only elements that match a condition
• flatMap() - Like map, but can produce multiple outputs per input
• reduceByKey() - Combine values with the same key
• join() - Combine two RDDs based on keys

Actions (Execute Everything):

• count() - Count elements
• collect() - Bring all data to the driver (careful with big data!)
• take(n) - Get the first n elements
• saveAsTextFile() - Write to storage

Example:

"""Word Count with RDD"""
# Read file (creates RDD)
lines = sc.textFile("book.txt")

# Split into words (transformation - lazy)
words = lines.flatMap(lambda line: line.split(" "))

# Create pairs (word, 1) (transformation - lazy)
pairs = words.map(lambda word: (word, 1))

# Sum up counts for each word (transformation - lazy)
wordCounts = pairs.reduceByKey(lambda a, b: a + b)

# Save results (action - NOW everything executes!)
wordCounts.saveAsTextFile("output")

What is a DataFrame?

If RDD is a box of random puzzle pieces, a DataFrame is a neatly organised Puzzle instruction manual with labelled sections. Everything has structure, names, and types. A DataFrame is basically a table. It is the same as Pandas DataFrame, just when used in the context of Spark, the characteristics differ slightly.

Key Characteristics:

1. Structured Data with Schema: Every DataFrame has a schema. Schema is a definition of column names and data types.
2. High-Level API: You work with column names and SQL-like operations instead of writing low-level functions.
3. Language Agnostic: DataFrames work the same way in Python, Scala, Java, and R. The same operations produce the same results because they all compile down to the same optimised execution plan.
4. Lazy Evaluation (Like RDDs): DataFrames also wait until you call an action before executing. Planning first, working later.

Common DataFrame Operations in action:

# Create a DataFrame
df = spark.read.csv("users.csv", header=True, inferSchema=True)

# See the structure
df.printSchema()

# Show first few rows
df.show()

# Select specific columns
df.select("name", "age").show()

# Filter rows
df.filter(df.age > 30).show()

# Group and aggregate
df.groupBy("city").count().show()

# SQL queries (yes, actual SQL!)
df.createOrReplaceTempView("users")
spark.sql("SELECT city, AVG(age) FROM users GROUP BY city").show()

Example:

"""Word Count with DataFrame"""
# Read file as DataFrame
df = spark.read.text("book.txt")

# Split into words and count
from pyspark.sql.functions import explode, split

wordCounts = df.select(
    explode(split(df.value, " ")).alias("word")
).groupBy("word").count()

wordCounts.show()

When to use RDD vs DataFrame?

Use RDD When:

1. You need fine-grained control: You know exactly how you want to partition, cache, or process data
2. Working with unstructured data: Raw text files, binary data, complex nested objects
3. Custom operations: You need to do something very specific that DataFrame APIs don’t support well
4. Legacy code: You’re maintaining older Spark applications written with RDDs

Example Scenario: You’re processing raw sensor data with irregular formats, custom binary protocols, or need precise control over data partitioning for a specialised algorithm.

Use DataFrames When:

1. Working with structured data: CSVs, JSON, Parquet files, database tables
2. Standard analytics: Filtering, grouping, aggregating, joining
3. SQL familiarity: Your team knows SQL better than functional programming
4. Performance matters: You want Spark to automatically optimise your queries
5. Cross-language consistency: Team uses multiple languages (Python, Scala, Java)

Example Scenario: Analysing sales data, processing logs with consistent structure, building ETL pipelines, and creating reports.

Conclusion

RDDs give developers fine‑grained control over distributed data and transformations, making them ideal for unstructured inputs, custom algorithms, or legacy codebases.

DataFrames, on the other hand, provide a higher‑level, schema‑aware interface that simplifies analytics, integrates seamlessly with SQL, and leverages Spark’s built‑in optimisations for performance.

Tomorrow, on Day 21, I’ll continue building on this foundation by exploring Transformations vs Actions in Spark, and understanding how Spark’s lazy evaluation model drives performance behind the scenes.

Today’s Resource

• Apache Spark Documentation
• Spark: The Definitive Guide
• Complete Spark (One of the best playlists on Apache Spark). Note: The videos are recorded in Hindi, you can translate to English with Auto dub.