Using Teams + Power Automate for Databricks Job Execution

I recently received a request from an individual who wanted to provide ID(s) and receive some data quality metrics back. No dashboards, no querying — just simply send a Teams message and receive information back. I designed a very rudimentary solution that is working well for us and did not see similar ones posted elsewhere, so figured I would share.

Prior to beginning, you will need to set up or designate a channel in Teams for this messaging to take place.

The solution is based on an extremely simple flow in Power Automate:

From a step by step perspective

• When a new channel message is added
• Select the Team and Channel where messaging will take place. You can check for new items at whatever cadence you wish.
• Get message details
• Add in the Message id from the precious step as a your Message parameter
• Parse JSON
• Parse the Body from the previous step. It may be helpful to have a sample JSON output from the previous step to generate a schema with.
• HTTP
• Here I make a POST to the Databricks Jobs API to trigger a new job run. They have great documentation that explains the call in detail here: Trigger a new job run | Jobs API | REST API reference | Databricks on AWS
• As a job parameter, I include the Body PlainTextContent from the previous activity: this will be the ID(s) to run the checks for.

From a Power Automate side, that is it! You can now monitor a Teams channel for activity and send a request to Databricks Jobs API with information posted in that channel.

Just as a disclaimer — the information sent as a job parameter in this methodology comes across as a string: the onus is on you to handle any necessary parsing, formatting, or sanity checks needed. It could easily break stuff if you do not have sufficient exception handling in place.

The job I am triggering simply passes a parameter into a notebook and executes it. The notebook has a script which queries some tables in our catalog for these IDs and returns data quality metrics formatted in a string. I then send a message back to the Teams channel with the results.

To send your message string back to Teams, you will first need to generate an Incoming Webhook. You can them send whatever message you want to your channel, like the below example.

import urllib3
import json

class ConnectorCard:
    def __init__(self, hookurl, http_timeout=60):
        self.http = urllib3.PoolManager()
        self.payload = {}
        self.hookurl = hookurl
        self.http_timeout = http_timeout

    def text(self, mtext):
        self.payload['text'] = mtext
        return self

    def send(self):
        headers = {'Content-Type':'application/json'}
        r = self.http.request(
                'POST',
                f'{self.hookurl}',
                body=json.dumps(self.payload).encode('utf-8'),
                headers=headers, timeout=self.http_timeout)
        if r.status == 200:
            return True
        else:
            raise Exception(f'Request failed with status {r.status}')

webhook = 'https://your_teams_webhook'

myTeamsMessage = ConnectorCard(webhook)

myTeamsMessage.text('Hello World!')

myTeamsMessage.send()

That is it — you now have a simple way to monitor a Teams channel for incoming activity, trigger a Databricks job based on the content of that activity, and return the result back via Teams.