🚀File Arrival Triggers in Databricks Workflows

The File Arrival Trigger in Databricks Workflows is now generally available (GA). If you haven’t tried it yet, it’s quick and easy to set up and integrate into your existing pipelines.

What Is a File Arrival Trigger?

A File Arrival Trigger monitors a specified storage location and automatically starts a job when new files appear. It’s simple to configure: just copy the path to an external location you want to monitor, paste it into the trigger configuration, and you’re good to go.

It supports external locations like Amazon S3, Azure Data Lake Storage (ADLS), and Google Cloud Storage. Whether your Databricks workspace is hosted on Azure, AWS, or GCP, this feature works seamlessly. The trigger polls for new files every minute at no additional cost — aside from the standard cost of listing files in the storage system.

You can configure it to monitor either a root folder or a specific subfolder, using either a volume path or an abfss:// path.

/Volumes/mycatalog/myschema/myvolume/
or
abfss://bronze@xxxx.dfs.core.windows.net/landing/

How to Configure File Arrival Triggers

1. Copy the path from the external location or volume you wish to monitor.

2. Open the pipeline you want to attach the trigger to.

3. In the Schedules & Triggers section, paste the path you copied into the location field.

4. Upload a file to the specified location. Within a minute, the job should trigger automatically.

Why Use File Arrival Triggers?

File Arrival Triggers are particularly useful when:

• File delivery timing is unpredictable: For example, an external data ingestion tool uploads files to your storage account on an ad hoc basis.
• You want to eliminate idle compute: Instead of keeping a cluster running and checking for files, this trigger can kick off a job only when files arrive, saving on compute costs.
• You’re orchestrating ML/AI workflows that depend on the presence of new data and need to process it immediately.

If you found this article insightful, please click the ‘clap’ button and follow me on Medium and LinkedIn. For any questions or advice, feel free to reach out to me on LinkedIn.