Databricks parallel notebook runs, clusters’ uses and cost implications

I have multiple child notebooks to run from a Parent notebook. As the notebooks do not have any dependancies on each other, so we can plan to run them in parallel. Below is the common Google provided code for achieving this task. But lets deep dive with this approach:

from concurrent.futures import ThreadPoolExecutor
# Define the child notebooks and their parameters
notebooks = [
 {"path": "/Path/To/Notebook1", "timeout": 3600, "params": {"arg1": "val1"}},
 {"path": "/Path/To/Notebook2", "timeout": 3600, "params": {"arg1": "val2"}}
]
# Wrapper function for dbutils call
def run_notebook(nb_info):
 return dbutils.notebook.run(nb_info["path"], nb_info["timeout"], nb_info["params"])
# Execute in parallel
with ThreadPoolExecutor(max_workers=len(notebooks)) as executor:
 results = list(executor.map(run_notebook, notebooks))
print(results)

Now there is a job cluster JC_01: Standard_E16_v3 (128 GB RAM, 16 cores), Autoscale: 1–4 workers, Spark Version: 17.3.x-scala2.13 which you have to use to run the above Parent notebook so at least 2 child notebooks can run in parallel. Will the JC suffice the need?

Yes, this cluster should be fine for running 2 parallel notebooks. Here’s why:

• Driver capacity: The driver has 128 GB RAM and 16 cores — dbutils.notebook.run executes on the driver via threads, so 2 concurrent threads are lightweight.
• Worker autoscale (1–4): With 2 notebooks running concurrently, the cluster can scale up to 4 workers to handle the combined Spark workloads. Each worker also has 128 GB / 16 cores, giving you up to 512 GB / 64 cores at peak.

The one thing to watch: if the child notebooks are memory-heavy, having 2 running simultaneously doubles the concurrent Spark resource demand. The autoscaler should handle this, but if you notice OOM errors, you could bump max_workers from 4 to 6.

Do you think that notebook timeout=3600 is justified for any scenarios?

For this estimation you have to check the notebook completion time. For example if each child notebook finishes in ~60 sec, then 10x headroom give ~600 sec for reasonable timeout.

The current 3600s isn’t harmful, but it means a stuck notebook would hang for a full hour before failing. A 600-second timeout per child notebook is a good balance — generous enough for occasional slowdowns but catches real issues much faster.

If the run parent notebook in a task of the job JOB_0001 with and without parallel child notebook then which is more cost effective?

If child notebooks are running sequentially then Total wall-clock time = T1 + T2 + … adding up. If child notebooks are called using parallelNotebooks() with max_workers=2, running 2 concurrently at any time. Total wall-clock time ≈ ~60% of sequential. The parallel version is more cost-effective. Since both use dbutils.notebook.run() on the same shared job cluster— you simply reduce the total cluster uptime, which directly lowers DBU and VM billing.

Key insight from Databricks best practices: If your workload scales linearly, doubling workers halves the time — so total cost stays roughly the same. But if it doesn’t scale linearly (which is common for ETL notebooks using dbutils.notebook.run()), more workers ≠ proportionally faster, and you end up paying more.

Closure Question: Then how do you decide which cluster size to use for a notebook run? Can we use a higher end cluster size with less worker nodes or lower end cluster size with more worker nodes?

Practical Steps which everyone should follow while choosing a cluster:

1. Start small — run your notebook on a single-node or 2-worker cluster and check the Spark UI (Stages tab) for spill metrics, task duration variance, and shuffle read/write.
2. Look for bottlenecks — if you see disk spill → add memory; if tasks are slow but CPU is idle → add cores; if shuffle is huge → consider larger fewer nodes.
3. Scale up and compare — double the workers and check if runtime halves (linear scaling). If it doesn’t, adding more workers won’t help cost-wise.
4. Use autoscaling with a reasonable min/max range so you only pay for peak resources when needed.

Databricks recommendation is fewer larger nodes over many smaller nodes. Hers is why:

Databricks states: “2 workers with 16 cores and 128 GB each has the same total compute as 8 workers with 4 cores and 32 GB each” — but the fewer-node config performs better because:

1. Shuffle stays local — joins and aggregations move less data across the network when more data fits on one node.
2. Less coordination overhead — fewer executors = less Spark scheduler work.