Time Zones in Databricks: How to Work with Date and Time Correctly (Full Practical Guide)

An analyst recently started working at Databricks. He wrote his first script and wanted to add a report execution date— REPORT_DATE. He wrote a simple code and displayed the result on the screen:

from datetime import datetime
import pytz

current_time = datetime.now()
print(current_time)

He gets the result: 2026–01–18 05:30:00, even though he knows for sure that he ran the report at 9:30 AM Toronto time.

Having seen an example somewhere on the Internet, he adds the following line to his Notebook:

spark.conf.set("spark.sql.session.timeZone", "America/New_York")

Now the time is correct, but wait, for some reason, the numbers in the daily reports are now inconsistent! It turns out that an incorrectly set time zone can affect not only variables but also offset the time in tables if the TIMESTAMP format is set. Let's take a closer look.

Databricks works with multiple time zone levels, and confusion between them leads to time shifts in your data, sometimes by hours or even days. Time is a critical data attribute that impacts the accuracy of reports, business logic, and auditing.

Databricks has three levels of time management

1. Databricks Cluster (system settings)
2. Spark Session (time zone offset in tables in TIMESTAMP fields)
3. Python or SQL runtime (variables in the current cell in your Notebook)

Each level can have its own time zone, and they are not synchronized automatically. Below are examples of setting the time zone and its impact. We'll skip the JVM setting.

How to work with time zones correctly?

Scenario 1: I want to record the current execution time of a report

PROBLEM: datetime.now() returns time in UTC (cluster timezone):

from datetime import datetime
import pytz

report_date = datetime.now()
print(report_date)

SOLUTION 1: Use the local timezone for display:

from datetime import datetime
import pytz

report_date = datetime.now(pytz.timezone('America/Toronto'))
print(report_date)

It can also be used in an SQL query:

df = spark.sql(f"""
    SELECT
        '{report_date}' as REPORT_DATE,
        *
    FROM SCHEMA.CATALOG.TABLE
""")

Important: pytz.timezone() only affects Python variables, not Spark tables.

Scenario 2: I'm working with data in Spark SQL

For example, if we're working with SQL, we can use Python + SQL, as seen above. However, we can also set the time zone for SQL without Python using from_utc_timestamp :

SELECT
  from_utc_timestamp(current_timestamp(), 'America/Toronto') AS CURRENT_TIME_TORONTO,
  *
FROM SCHEMA.CATALOG.TABLE

CAUTION: Changing the session timezone affects all TIMESTAMP columns. This does not apply to DATE or TIMESTAMP_NTZ formats.

spark.conf.set("spark.sql.session.timeZone", "America/New_York")

Now ALL timestamp values ​​in SQL variables and tables will be interpreted as America/New_York instead of UTC.

If your data was stored in UTC, it will be shifted by 5 hours. Sales for January 20th will become sales for January 19th–20th.

Best Practices

• Store all timestamps in the database in UTC
• Do not change spark.sql.session.timeZone unless necessary.
• For display and logging, use local TZ
• When working on a schedule, consider the time zone.
• Check your current settings

Checking the cluster timezone:

import subprocess

result = subprocess.run(['date', '+%Z'], capture_output=True, text=True)
print(f"Cluster timezone: {result.stdout.strip()}")

Check Spark session timezone:

print(f"Spark session timezone: {spark.conf.get('spark.sql.session.timeZone')}")

Checking Python timezone:

from datetime import datetime

print(f"Python datetime.now(): {datetime.now()}")
print(f"Python datetime.utcnow(): {datetime.utcnow()}")

What problems and nuances might arise in real cases

Case 1: Migrating from Oracle without a timezone

Problem: In Oracle, the database stores dates in DATE columns (without a timezone). The Oracle server is in the America/Toronto timezone (UTC-5):

SELECT order_date FROM orders WHERE order_id =  123 ; 

-- Result: 2026-01-18 23:30:00 (stored as is, without timezone)

When loaded into Databricks, this field is loaded as is, without any shifting. However, Databricks stores this date in the UTC zone.

Consequences: An order placed at 23:30 Toronto time (actually 2026–01–19 04:30 UTC ) will be recorded as 2026–01–18 23:30 UTC – a difference of 5 hours!

The analyst has completed the training and thinks the data has been loaded correctly. However, they start calculating the report in UTC, changing the Notebook time zone to America/Toronto, and the date ends up shifting by another 5 hours. Therefore, it's important to know how data is loaded into Databricks.

Case 2: I changed the session timezone, and now everything is broken

Day 1: You've written your query and downloaded the data in UTC:

CREATE TABLE sales AS
  SELECT
    '2026-01-18 10:00:00' AS sale_time,
    100 AS amount

Day 10: Someone set a time zone for the session on the Notebook:

spark.conf.set("spark.sql.session.timeZone", "America/Toronto")

Day 11: Request for sales after 10:00:

  SELECT * FROM sales 
  WHERE sale_time >= '2026-01-18 10:00:00'

The result is empty! Because the filter is looking for 10:00 Toronto = 15:00 UTC.

Step-by-step guide: How to set the timezone correctly

Below, I've outlined the steps for changing the time zone from UTC to Canada. Be careful when changing the time zone during a session. Potential issues were described in the case studies above.

Remember: Time isn't just a number; it's a contract between systems. Violating this contract leads to errors that are difficult to detect and easily propagate across the entire analytics platform. Be mindful of time zones!

Subscribe to my blog on Medium to stay up-to-date with new insights!