Cracking the Ultimate PySpark Interview Question: How to Flatten Dynamic JSON Without Hardcoding Schema

How to flatten a completely dynamic, semi-structured JSON column in PySpark without hardcoding a single schema or melting your cluster

Imagine you’re a data engineer at a fast-growing startup. Your pipeline ingests millions of records from an external API. Every record has a column called attributes — a plain string. Sounds simple, right?

Then you open it.

User A → {'age': 25, 'city': 'NY'}
User B → {'signup_date': '2026-01-01', 'tier': 'gold'}
User C → {'device': 'iPhone', 'country': 'IN', 'referral': 'google'}

No two users are the same. The keys are completely dynamic. And your manager just said: “Can you flatten that into proper columns by tomorrow morning?”

This is the dynamic PySpark JSON flattening problem — and if you’ve ever tried to solve it by hardcoding a schema, you already know why that approach falls apart at scale.

Let’s fix it the right way.

🎯 The Problem: Why “Just Parse It” Doesn’t Cut It

Most tutorials tell you to define a StructType schema and use from_json(). That works great — until the schema changes. And with dynamic APIs, it always changes.

Here’s what goes wrong when teams try to handle this naively:

• Hardcoded schemas break the moment a new key appears in production
• **explode() on maps** causes row multiplication nightmares
• **collect() tricks** pull everything to the driver — a one-way ticket to OOM errors
• Re-running jobs manually every time the API evolves = technical debt spiral

What you actually need is a pipeline that:

1. Discovers all unique keys dynamically
2. Flattens them into individual columns without schema assumptions
3. Scales to millions of rows without killing your cluster

💡 The Concept: How Dynamic Flattening Works in PySpark

Before jumping to code, let’s build the mental model.

Think of your attributes column like a bag of random items left by each user. Some bags have apples, some have oranges, some have both, some have neither.

Your job? Create one column per fruit type across all users, and fill in null where a user didn't bring that fruit.

In PySpark terms, this is a pivot-from-map operation. Here’s the playbook:

Step 1: Parse the string into a MapType Use from_json() with a MapType(StringType(), StringType()) schema — this is the trick that handles any key-value structure without knowing keys upfront.

Step 2: Discover all unique keys Sample or scan the dataset to collect every key name that appears across all rows.

Step 3: Pivot each key into its own column Use map_keys() / map_values() or simply access parsed_col["key_name"] dynamically.

Step 4: Build the final DataFrame programmatically Loop over discovered keys and build select() expressions — no hardcoding, pure metaprogramming.

🧪 The Code: Step-by-Step PySpark Solution

Let’s walk through a complete, production-ready example.

Setup: Sample Data

from pyspark.sql import SparkSession
from pyspark.sql.functions import from_json, col, map_keys, explode_outer
from pyspark.sql.types import MapType, StringType

spark = SparkSession.builder.appName("DynamicFlatten").getOrCreate()

# Simulated external API payload
data = [
    (1, "{'age': '25', 'city': 'NY'}"),
    (2, "{'signup_date': '2026-01-01', 'tier': 'gold'}"),
    (3, "{'device': 'iPhone', 'country': 'IN', 'referral': 'google'}"),
    (4, "{'age': '30', 'tier': 'silver', 'city': 'LA'}"),
]

df = spark.createDataFrame(data, ["user_id", "attributes"])
df.show(truncate=False)

What this does: Creates a raw DataFrame mimicking what you’d receive from an API. The attributes column is a plain string — Python dict notation, not even proper JSON yet.

Step 1: Parse the String into a Map

# Replace single quotes with double quotes for valid JSON
from pyspark.sql.functions import regexp_replace

df_parsed = df.withColumn(
    "attrs_map",
    from_json(
        regexp_replace(col("attributes"), "'", '"'),  # fix single → double quotes
        MapType(StringType(), StringType())            # dynamic schema: any key, string value
    )
)

Why MapType(StringType(), StringType())? Because you're telling Spark: "Trust me, it's a key-value map — I don't know the keys yet." This is the entire trick. Spark will parse any valid JSON object into this map without complaining about unknown keys.

Why regexp_replace? Real-world API data often uses Python-style single quotes. JSON requires double quotes. This one-liner fixes it before parsing.

Step 2: Discover All Keys Dynamically

# Collect all unique keys across the entire dataset
all_keys = (
    df_parsed
    .select(explode_outer(map_keys(col("attrs_map"))).alias("key"))
    .distinct()
    .rdd
    .flatMap(lambda x: x)
    .collect()
)

print("Discovered keys:", all_keys)
# Output: ['age', 'city', 'signup_date', 'tier', 'device', 'country', 'referral']

What’s happening here?

• map_keys() extracts only the keys from each map
• explode_outer() expands the array of keys into individual rows (one row per key per user)
• .distinct() deduplicates across all rows
• .collect() brings only the key names (tiny data!) to the driver — not the actual values

This is the efficient version. You’re not pulling millions of records to the driver. You’re pulling a handful of strings.

Step 3: Flatten Into Independent Columns

# Dynamically build select expressions for each key
flatten_exprs = [col("user_id")] + [
    col("attrs_map")[key].alias(key)
    for key in all_keys
]

df_flat = df_parsed.select(*flatten_exprs)
df_flat.show()

Expected Output:

+-------+----+----+-----------+------+------+-------+--------+
|user_id| age|city|signup_date|  tier|device|country|referral|
+-------+----+----+-----------+------+------+-------+--------+
|      1|  25|  NY|       null|  null|  null|   null|    null|
|      2|null|null| 2026-01-01|  gold|  null|   null|    null|
|      3|null|null|       null|  null|iPhone|     IN|  google|
|      4|  30|  LA|       null|silver|  null|   null|    null|
+-------+----+----+-----------+------+------+-------+--------+

Every key becomes its own column. Missing values are null. Zero hardcoding. Zero schema drift issues.

Bonus: Handle Type Casting

Since we parsed everything as StringType, you may want to cast known numeric-looking columns:

from pyspark.sql.functions import when

# Optional: infer and cast numeric columns
for key in all_keys:
    df_flat = df_flat.withColumn(
        key,
        when(col(key).rlike(r"^\d+$"), col(key).cast("integer"))
        .otherwise(col(key))
    )

This conditionally casts columns that look like integers, leaving strings alone.

⚠️ Common Mistakes to Avoid

❌ Mistake 1: Using collect() on the Full DataFrame

# WRONG — never do this
all_data = df.collect()
keys = set()
for row in all_data:
    keys.update(json.loads(row['attributes']).keys())

This pulls all rows to the driver. On a 100M row dataset, your driver dies instantly.

Fix: Only collect the distinct keys after aggregation (as shown above).

❌ Mistake 2: Hardcoding the Schema

# WRONG — brittle and breaks with any API change
schema = StructType([
    StructField("age", IntegerType()),
    StructField("city", StringType()),
])

The moment the API adds a new field, your pipeline silently drops it.

Fix: Use MapType(StringType(), StringType()) and discover keys dynamically.

❌ Mistake 3: Using explode() on the Map Directly

# WRONG — multiplies your rows
df.select(explode(col("attrs_map")))

This turns 4 rows into 15+ rows, one per key-value pair. Now your joins and aggregations are all wrong.

Fix: Use map["key"] access syntax to pivot, not explode.

❌ Mistake 4: Scanning 100% of Data for Key Discovery

If your dataset is truly massive (billions of rows), scanning all of it for keys is slow.

Fix: Sample a representative fraction:

df_sample = df_parsed.sample(fraction=0.01, seed=42)
all_keys = (
    df_sample
    .select(explode_outer(map_keys(col("attrs_map"))).alias("key"))
    .distinct()
    .rdd.flatMap(lambda x: x)
    .collect()
)

A 1% sample is often enough to discover all key variations in a well-distributed dataset.

🚀 Pro Tips and Best Practices

Tip 1: Cache the Parsed DataFrame If you’re running key discovery and then flattening in the same job, cache df_parsed to avoid parsing the JSON string twice.

df_parsed.cache()

Tip 2: Use a Schema Registry for Production In production pipelines, don’t discover keys at runtime every single run. Instead, write discovered keys to a schema registry (Confluent, Glue, or even a Delta table), and update it incrementally as new keys appear.

Tip 3: Sanitize Column Names API keys sometimes contain spaces, dots, or special characters that break Spark column names. Sanitize them:

import re
clean_key = re.sub(r"[^a-zA-Z0-9_]", "_", key)

Tip 4: Write to Delta Lake with Schema Evolution When writing flattened data to a Delta table, enable schema evolution so new columns are added automatically:

df_flat.write.format("delta").option("mergeSchema", "true").mode("append").save(path)

Tip 5: Parallelize Key Discovery Across Partitions Instead of collect(), use reduceByKey or Spark's native agg(collect_set()) to keep key discovery fully distributed.

🔄 Real-World Use Cases

E-commerce Platforms: Product attributes vary wildly by category. A shirt has size and color; a laptop has RAM and GPU. This pattern handles both in one pipeline.

Marketing Analytics: User event metadata from tools like Segment or Mixpanel is semi-structured. Flattening it dynamically enables ad-hoc analysis without schema changes.

Financial Data Pipelines: Transaction metadata from payment processors often contains optional, provider-specific fields. Dynamic flattening absorbs new providers without code changes.

Data Lakehouses at Scale: Companies like Airbnb, Uber, and Netflix use variants of this pattern in their Spark-based ETL pipelines to handle schema-flexible ingestion into Delta Lake or Iceberg tables.

Interview Scenarios: This exact pattern appears in senior data engineering interviews at FAANG and unicorn startups. Being able to reason about MapType, key discovery efficiency, and avoiding collect() pitfalls is a strong differentiator.

📌 Quick Recap

• The attributes column holds semi-structured JSON with dynamic, unpredictable keys
• Parse it using from_json() with MapType(StringType(), StringType()) — no schema needed
• Discover all unique keys efficiently using explode_outer(map_keys(...)).distinct().collect() — only tiny key strings reach the driver
• Flatten by building select() expressions dynamically using a list comprehension
• Never collect() the full dataset, never hardcode a schema, never use explode() on the map directly
• In production: use sampling for key discovery, cache parsed DataFrames, sanitize column names, and write with schema evolution enabled

🚀 Level Up Your Career — Don’t Wait, Start NOW!

If you’re serious about growing in tech and staying ahead of the curve, this is your moment. No shortcuts — just real skills that actually make a difference.

🌐 Let’s Connect & Grow Together

Follow me for practical insights, real-world learning, and career tips:

🐦 Twitter: https://x.com/SriwWorld 📺 YouTube: https://www.youtube.com/@sriwworldofcoding?sub_confirmation=1 ✍️ Medium: https://medium.com/@sriwworldofcoding 🧵 Threads: https://www.threads.com/@sriwworldofcoding 📸 Instagram: https://www.instagram.com/sriwworldofcoding/ 📘 Facebook: https://www.facebook.com/profile.php?id=61576419014220 🌌 Bluesky: https://bsky.app/profile/sriwworldofcoding.bsky.social

🎯 Want Real Skills? Start With These Hands-On Courses

⚙️ Apache Airflow Bootcamp (Workflow Automation)

👉 https://www.udemy.com/course/apache-airflow-bootcamp-hands-on-workflow-automation/ 💡 Go from beginner to advanced — master DAGs, scheduling, operators, sensors, and build real production workflows.

🔥 PySpark for Data Engineers (Architecture + Interviews)

👉 https://www.udemy.com/course/pyspark-for-data-engineers-architecture-interviews/ 💡 Deep dive into Spark architecture, optimization, and performance tuning — plus crack interviews with confidence.

☁️ Crack Azure Data Engineer Interviews: The Ultimate Q&A Guide

👉 https://www.udemy.com/course/crack-azure-data-engineer-interviews-the-ultimate-qa-guide/ 💡 Get interview-ready with real-world questions on ADF, Synapse, Databricks, Event Hubs, Data Lake, Azure Functions & more.

💥 The difference between where you are and where you want to be? ACTION. Start learning today — your future self will thank you.

Tags: PySpark, Data Engineering, Apache Spark, Python, Big Data, ETL, Schema Evolution, Dynamic Schema, Interview Prep, Data Lakehouse