Databricks SQL Warehouse - Best Practices

When it comes to SQL Warehouses, everyone needs

Low latency — Low cost — Autoscaling.

There is no magic formula to determine this right away, so how to size the SQL Warehouse?

Databricks SQL Warehouse

Start with Medium size cluster. Depending on the performance, it can be adjusted to a larger or smaller size.

Default scaling Min 1 — Max 5. Due to cost factors or VPC limitations, some organizations won’t allow more than 3, then set it to max allowed limit.

As a rule of thumb,

To bring down latency, increase the Endpoint Size (Medium to XL or 2XL)

To Process a massive volume of data, increase the Endpoint Size.

For higher concurrency, increase the MIN/MAX values scaling values.

Larger Endpoint = Lower Latency

Higher scaling MAX Value = Higher Concurrency

So if you need Lower Latency + High Concurrency? You know what to do, right?

Thank you for reading and supporting my articles.

I wish you all a Happy and Prosperous 2023.

[embed]Join Medium with my referral link - Ganesh Chandrasekaran Read every story from Ganesh Chandrasekaran (and thousands of other writers on Medium). Your membership fee directly…ganeshchandrasekaran.com

[embed]Get an email whenever Ganesh Chandrasekaran publishes. Edit descriptionganeshchandrasekaran.com