Article 2: How SparkContext Boots — A Step-by-Step Source Code Walk

Series: “Deep Dive into Apache Spark Source Code” · Article 2 of 20 Source: Apache Spark 4.0.0 · github.com/apache/spark · Scala 2.13

What Happens When You Write new SparkContext(conf)?

Every Spark application starts with one line:

val sc = new SparkContext(conf)

Or in PySpark:

spark = SparkSession.builder.appName("MyApp").getOrCreate()

That single call triggers a precise sequence of ~20 initialization steps. By the time the constructor returns, Spark has wired up a scheduler, registered with the cluster, started a web UI, and is ready to run jobs.

In this article we trace the exact boot sequence inside SparkContext, referencing the real source code.

File: [core/src/main/scala/org/apache/spark/SparkContext.scala](https://github.com/apache/spark/blob/v4.0.0/core/src/main/scala/org/apache/spark/SparkContext.scala)

The Boot Sequence at a Glance

new SparkContext(conf)
  │
  ├─ 1. Validate SparkConf
  ├─ 2. Create LiveListenerBus
  ├─ 3. Create SparkEnv  ◄─── biggest step: serializer, RPC, memory, blocks
  ├─ 4. Create Spark UI (port 4040)
  ├─ 5. Register HeartbeatReceiver
  ├─ 6. Init ShuffleManager + MemoryManager
  ├─ 7. Create TaskScheduler + SchedulerBackend  ◄─── picks local/YARN/K8s
  ├─ 8. Create DAGScheduler
  ├─ 9. Start TaskScheduler
  ├─ 10. Get Application ID
  ├─ 11. Initialize BlockManager
  ├─ 12. Start MetricsSystem
  ├─ 13. Setup EventLogger
  ├─ 14. Create ContextCleaner
  └─ 15. Start ListenerBus

Let’s walk through each step.

Step 1 — Validate SparkConf (line 410)

// SparkContext.scala:410
_conf = config.clone()
_conf.validateSettings()
_conf.set("spark.app.id", "local-" + Utils.random.nextInt())

SparkConf is cloned first — Spark never modifies the user's original config object. validateSettings() checks for required keys: spark.master and spark.app.name. If either is missing, it throws an exception immediately.

Key configs to know:

Step 2 — Create LiveListenerBus (line 485)

// SparkContext.scala:485
_listenerBus = new LiveListenerBus(_conf)

The LiveListenerBus is Spark's internal event bus — an async message queue that broadcasts events to all registered listeners. Every significant event in a Spark job (job start, stage complete, task end) is posted here.

Who listens?

• SparkUI — updates the web dashboard
• EventLogger — writes events to disk for the History Server
• AppStatusStore — maintains job/stage status for sc.statusTracker

This must be created before SparkEnv because some environment components register listeners.

Step 3 — Create SparkEnv (line 495)

// SparkContext.scala:495
_env = createSparkEnv(_conf, isLocal, listenerBus)
SparkEnv.set(_env)

This is the biggest single step in the boot sequence. SparkEnv is the container for all low-level runtime services the driver needs.

File: [core/src/main/scala/org/apache/spark/SparkEnv.scala](https://github.com/apache/spark/blob/v4.0.0/core/src/main/scala/org/apache/spark/SparkEnv.scala)

Inside SparkEnv.createDriverEnv(), the following are built in order:

Step 4 — Create Spark UI (line 513)

// SparkContext.scala:513
_ui = if (conf.get(UI_ENABLED)) {
  Some(SparkUI.create(Some(this), _statusStore, _conf, _env.securityManager,
                      appName, "", startTime))
} else None
_ui.foreach(_.bind())

If spark.ui.enabled = true (the default), Spark starts a Jetty web server on port 4040. This is the dashboard you see at http://localhost:4040 — it shows live job progress, stage DAGs, executor status, and SQL query plans.

Step 5 — Register HeartbeatReceiver (lines 584–585)

// SparkContext.scala:584
_heartbeatReceiver = env.rpcEnv.setupEndpoint(
  HeartbeatReceiver.ENDPOINT_NAME,
  new HeartbeatReceiver(this))

Executors send periodic heartbeats to the driver to report they are alive and to share task metrics. The HeartbeatReceiver is an RPC endpoint on the driver side that receives these pings.

If an executor misses enough heartbeats, it is marked as lost and its tasks are rescheduled.

Step 6 — Init ShuffleManager + MemoryManager (lines 589–590)

These two were created lazily inside SparkEnv but are now explicitly started:

• ShuffleManager — manages how shuffle data is written and read. Default: SortShuffleManager
• MemoryManager — manages memory allocation between execution (tasks) and storage (cache). Default: UnifiedMemoryManager

UnifiedMemoryManager allows execution and storage to share a single memory pool, which is more efficient than the old static split.

Step 7 — Create TaskScheduler + SchedulerBackend (lines 593–595)

This is where Spark decides how to run tasks based on the spark.master URL.

The createTaskScheduler method (line 3267) matches the master URL with regex patterns:

master match {
  case "local" =>
    // Single-threaded local mode
    new LocalSchedulerBackend(conf, scheduler, 1)

  case LOCAL_N_REGEX(threads) =>
    // local[4] or local[*]
    new LocalSchedulerBackend(conf, scheduler, threads)

  case SPARK_REGEX(sparkUrl) =>
    // spark://host:port — Standalone cluster
    new StandaloneSchedulerBackend(scheduler, sc, masterUrls)

  case masterUrl =>
    // YARN, Kubernetes, etc. — loaded via ServiceLoader
    val cm = getClusterManager(masterUrl)
    cm.createSchedulerBackend(sc, masterUrl, scheduler)
}

This is the pluggable cluster manager design — adding support for a new cluster is just implementing ExternalClusterManager and registering it as a Java service.

Important: TaskScheduler is created here but not yet started. It starts at step 9.

Step 8 — Create DAGScheduler (line 596)

// SparkContext.scala:596
_dagScheduler = new DAGScheduler(this)

DAGScheduler is created after TaskScheduler because it needs a reference to it. Inside the DAGScheduler constructor (line 287):

// DAGScheduler.scala:287
taskScheduler.setDAGScheduler(this)

This call creates the two-way link between them:

• DAGScheduler holds a reference to TaskScheduler (to submit task sets)
• TaskScheduler holds a reference to DAGScheduler (to report stage failures back up)

DAGScheduler also initializes its core data structures:

val stageIdToStage   = new HashMap[Int, Stage]    // stage lookup
val activeJobs       = new HashSet[ActiveJob]      // running jobs
val waitingStages    = new HashSet[Stage]          // waiting on deps
val runningStages    = new HashSet[Stage]          // currently running
val failedStages     = new HashSet[Stage]          // needs retry

Step 9 — Start TaskScheduler (line 622)

// SparkContext.scala:622
_taskScheduler.start()

Now the scheduler actually connects to the cluster. For YARN, this is where the ResourceManager is contacted and the ApplicationMaster is launched. For Kubernetes, this is where the driver registers with the API server. For local mode, thread pools are created.

Step 10 — Get Application ID (lines 624–625)

// SparkContext.scala:624
_applicationId = _taskScheduler.applicationId()
_applicationAttemptId = _taskScheduler.applicationAttemptId()
_conf.set("spark.app.id", _applicationId)

The application ID is assigned by the cluster manager (e.g., application_1234567890_0001 in YARN). This ID is used to:

• Name the event log file
• Identify shuffle data in the BlockManager
• Label metrics in the monitoring system

Step 11 — Initialize BlockManager (line 643)

// SparkContext.scala:643
_env.blockManager.initialize(_applicationId)

BlockManager was created in step 3 but waited for the application ID to fully initialize. Now it registers with BlockManagerMaster on the driver and is ready to store broadcast variables and cached RDD partitions.

Step 12 — Start MetricsSystem (line 648)

// SparkContext.scala:648
_env.metricsSystem.start()
_env.metricsSystem.getServletHandlers.foreach(handler => ui.foreach(_.attachHandler(handler)))

The metrics system (backed by Dropwizard Metrics) starts publishing metrics. Metrics are attached to the Spark UI if enabled, and can also be sent to Graphite, Prometheus, or other sinks via configuration.

Step 13 — Setup EventLogger (lines 650–660)

// SparkContext.scala:650
if (_conf.get(EVENT_LOG_ENABLED)) {
  _eventLogger = Some(new EventLoggingListener(_applicationId, ...))
  _listenerBus.addToEventLogQueue(_eventLogger.get)
}

If spark.eventLog.enabled = true, Spark writes all ListenerBus events to a file. The History Server later reads these files to reconstruct completed application UIs.

Step 14 — Create ContextCleaner (lines 662–668)

// SparkContext.scala:662
_cleaner = if (_conf.get(CLEANER_REFERENCE_TRACKING)) {
  Some(new ContextCleaner(this, _shuffleDriverComponents))
} else None
_cleaner.foreach(_.start())

ContextCleaner runs a background thread that tracks RDDs, broadcasts, and shuffles via weak references. When your code no longer references an RDD, the cleaner automatically releases its shuffle files, broadcasts, and cached blocks — preventing memory leaks in long-running applications.

Step 15 — Start ListenerBus (line 687)

// SparkContext.scala:687
setupAndStartListenerBus()
postEnvironmentUpdate()
postApplicationStart()

The LiveListenerBus is started last. It begins dispatching all queued events to listeners. postApplicationStart() fires the first event, notifying all listeners (UI, logger, etc.) that the application has started.

SparkContext is now fully initialized and ready to accept jobs.

The Complete Boot Sequence Timeline

new SparkContext(conf)
│
├── [1]  Validate SparkConf                    line 410
├── [2]  Create LiveListenerBus                line 485
├── [3]  Create SparkEnv                       line 495
│         ├── RpcEnv (network)
│         ├── Serializer
│         ├── BroadcastManager
│         ├── MapOutputTracker
│         └── BlockManager
├── [4]  Create Spark UI (port 4040)           line 513
├── [5]  Register HeartbeatReceiver            line 584
├── [6]  Init ShuffleManager + MemoryManager   line 589
├── [7]  Create TaskScheduler + Backend        line 593
│         └── picks: local / standalone / YARN / K8s
├── [8]  Create DAGScheduler                   line 596
│         └── taskScheduler.setDAGScheduler()
├── [9]  Start TaskScheduler                   line 622
│         └── connects to cluster
├── [10] Get Application ID                    line 624
├── [11] Initialize BlockManager               line 643
├── [12] Start MetricsSystem                   line 648
├── [13] Setup EventLogger                     line 650
├── [14] Create ContextCleaner                 line 662
└── [15] Start ListenerBus                     line 687
         └── postApplicationStart() → app is live

Key Takeaways

• new SparkContext(conf) executes ~15 ordered initialization steps — order matters because later components depend on earlier ones
• SparkEnv (step 3) is the most complex step, creating all low-level runtime services
• TaskScheduler is selected dynamically based on spark.master — this is how Spark supports multiple cluster managers without code changes
• DAGScheduler and TaskScheduler are linked bidirectionally at creation time
• The ListenerBus is started last, ensuring all components are ready before events flow

Where to Go from Here

In Article 3, we’ll look at RDDs — the original abstraction in Spark. We’ll trace how sc.textFile() creates an RDD, what an RDD really is in the source code, and how transformations like map() and filter() chain together without executing anything.

Source Code References

All links point to Apache Spark 4.0.0 on GitHub.

• SparkContext — SparkContext.scala
• SparkEnv — SparkEnv.scala
• SparkConf — SparkConf.scala
• DAGScheduler — DAGScheduler.scala
• LiveListenerBus — LiveListenerBus.scala
• ContextCleaner — ContextCleaner.scala

All source code referenced in this article is from the Apache Spark open-source project, licensed under Apache License 2.0.