Spark Dynamic Resource Allocation

Spark Scheduling: Allocation Strategies

Scenarios:

• Multiple Spark applications running on a shared cluster.

Challenges:

• Efficient resource utilization: Static allocation might lead to underutilized resources while an application holds them for its entire duration.

Spark Allocation Strategies:

• Static Allocation (default): (ex: — number of executors: 10, executor-memory: 8GB etc.)
• Application requests all resources at startup from the cluster resource manager.
• Holds onto all resources throughout its execution, regardless of actual usage.
• Less efficient for applications with varying resource needs across stages.
• Dynamic Allocation:
• Application requests resources dynamically as needed.
• Releases unused executors back to the cluster manager.
• Acquires new executors when more resources are required.
• Requires enabling configurations (spark.dynamicAllocation.enabled and spark.dynamicAllocation.testingMode).

Dynamic Allocation Configuration:

• spark.dynamicAllocation.enabled: Set to true to enable dynamic allocation.
• spark.dynamicAllocation.idleTimeout: Time (default: 60 seconds) an executor can be idle before being released.
• spark.dynamicAllocation.backlogTimeout: Time (default: 1 second) pending tasks wait for an executor before requesting more resources.

Benefits of Dynamic Allocation (for shared clusters):

• Improved resource utilization by adapting to application needs.
• Allows multiple applications to share resources more efficiently.

Key Points:

• Dynamic allocation is a Spark-level configuration, not a cluster resource manager decision.
• Consider enabling dynamic allocation for shared clusters to optimize resource usage.

Serial vs. Parallel Execution within application

• Default (Serial): Spark jobs/tasks run sequentially, one after another.
• Parallel: You can submit jobs from multiple threads, allowing them to run concurrently.

Benefits of Parallel Execution:

• Improved performance by utilizing cluster resources more efficiently.

Challenges of Parallel Execution:

• Resource competition: Parallel jobs require resources to run tasks concurrently.
• Scheduling: Spark needs to manage resource allocation between parallel jobs.

Spark Schedulers:

• FIFO (default): Jobs are processed in the order they are submitted. Earlier jobs get priority on all available resources, potentially delaying later jobs.
• FAIR: Jobs are assigned tasks in a round-robin fashion, ensuring all parallel jobs get a fair share of resources and avoid waiting for each other.

Configuration:

• Enable FAIR scheduler using spark.scheduler.mode=FAIR.

if __name__ == "__main__":
    spark = SparkSession \
        .builder \
        .appName("Demo") \
        .master("local[3]") \
        .config("spark.sql.autoBroadcastJoinThreshold", "508")   
 \
        .config("spark.scheduler.mode", "FAIR") \
        .getOrCreate()

    file_prefix = "data/d"
    jobs = []
    outputs = []

    for i in range(0, 2):
        file1 = file_prefix + str(i + 1)
        file2 = file_prefix + str(i + 2)
        thread = threading.Thread(target=do_job, args=(file1, file2))
        jobs.append(thread)   

    for j in jobs:
        j.start()

    for j in jobs:
        j.join()   

    print(outputs)

Key Points:

• Consider parallel execution and FAIR scheduler for multithreaded Spark applications to optimize resource usage and job completion times.
• Spark UI provides insights into the active scheduler and resource allocation details.