How to Calculate Spark Submit Parameters Based on File Size and Partitioning

When running a Spark job, choosing the right spark-submit parameters can significantly impact performance and resource utilization. In this article, I’ll show you how to estimate these parameters based on a real-world scenario: processing a 10 GB file with a partition size of 128 MB.

Why It Matters

Spark is a distributed engine. When you process large files, it splits the data into partitions. Each partition becomes a task. Understanding this helps us determine how many executors, cores, and memory Spark will need.

Step-by-Step Calculation

1. Calculate Number of Partitions

Partition size = 128 MB File size = 10 GB = 10,240 MB

Number of partitions = 10,240 / 128 = 80

So Spark will create 80 tasks.

2. Set Executor Cores

Let’s choose:

--executor-cores 4

This means each executor can process 4 tasks in parallel.

3. Determine Number of Executors

Total tasks = 80 Tasks per executor = 4

--num-executors = 80 / 4 = 20

This ensures that all tasks can be processed in parallel, without underutilizing or overloading resources.

4. Estimate Executor Memory

Each task processes 128 MB, but Spark needs 2–3x overhead for:

• Shuffling
• Serialization
• Caching

So memory per task:

128 MB × 3 = 384 MB

Memory per executor (4 tasks):

384 MB × 4 = 1,536 MB (~1.5 GB)

Add JVM overhead (~0.5 GB), and we round it up:

--executor-memory = 2G

Final spark-submit Command

spark-submit \
  --executor-memory 2G \
  --executor-cores 4 \
  --num-executors 20 \
  --driver-memory 2G \
  --conf spark.sql.shuffle.partitions=80 \
  your_spark_script.py

Total memory required to process 10GB of data

We are here! lets finalize on total memory to be required to process 10GB of data. Each executor has 2GB of memory

We have total of 20 executors

Therefore, 20*2 = 40 GB of total minimum memory required to process 10GB of data completely in parallel.

Key Takeaways

• Use file size and partition size to calculate number of tasks.
• Base executor count on tasks ÷ cores per executor.
• Estimate memory per executor as: partition_size × overhead × executor_cores + memoryOverhead
• Tune spark.sql.shuffle.partitions to match your data size and executor count.

Bonus Tip: If you’re running on a dynamic cluster like YARN or EMR, you can use dynamic allocation:

--conf spark.dynamicAllocation.enabled=true
--conf spark.dynamicAllocation.minExecutors=4
--conf spark.dynamicAllocation.maxExecutors=30

Hope this helps you better understand Spark resource planning!

#ApacheSpark #BigData #PerformanceTuning #DataEngineering #SparkSubmit #SparkTips

Follow my youtube channel for more such intresting topics and updates : https://www.youtube.com/@sriwworldofcoding

I have launched new udemy course named : Apache Airflow Bootcamp: Hands-On Workflow Automation . This course will take you from beginner to expert, covering everything from the basics to advanced concepts. Whether you’re a data engineer, a developer, or just looking to upskill, this course is for you!

Detailed Content : https://youtu.be/J23wLi8tTyQ

Follow me on : Twitter : https://x.com/SriwWorld Instagram : https://www.instagram.com/sriwworldofcoding/ Youtube : https://www.youtube.com/@sriwworldofcoding?sub_confirmation=1 Medium : https://medium.com/@sriwworldofcoding Threads : https://www.threads.com/@sriwworldofcoding Facebook : https://www.facebook.com/profile.php?id=61576419014220

🔥 If you want to stay ahead in your career, start learning NOW!

I highly recommend these 2 practical, hands-on courses 👇 📌 Apache Airflow Bootcamp (Workflow Automation) 👉 https://www.udemy.com/course/apache-airflow-bootcamp-hands-on-workflow-automation/ 💡 Learn everything from basics to advanced: DAGs, scheduling, operators, sensors & real workflows

📌 PySpark for Data Engineers (Architecture + Interviews) 👉 https://www.udemy.com/course/pyspark-for-data-engineers-architecture-interviews/ 💡 Master Spark architecture, optimization, performance tuning & crack interviews like a pro

A message from our Founder

Hey, Sunil here. I wanted to take a moment to thank you for reading until the end and for being a part of this community. Did you know that our team run these publications as a volunteer effort to over 3.5m monthly readers? We don’t receive any funding, we do this to support the community.

If you want to show some love, please take a moment to follow me on LinkedIn, TikTok, Instagram. You can also subscribe to our weekly newsletter. And before you go, don’t forget to clap and follow the writer️!