Drasi: Microsoft’s New Open-Source Project for Real-Time Change Detection

Drasi is a data processing platform designed to simplify the detection of changes in data systems (eg., Azure Cosmos DB, PostgreSQL, Event Hubs, Microsoft Dataverse etc.,) and trigger automatic responses to those changes. It provides a comprehensive solution for tracking logs and change feeds across various systems, evaluating them for relevance, and reacting in real time.

Unlike traditional methods that require copying data to a central location or repeated queries, Drasi uses continuous queries that monitor data in real time. When a change occurs that matches predefined criteria, Drasi triggers context-aware actions, making the process both efficient and scalable.

For example, Drasi can be applied in real-world scenarios such as:

• Automatically adjusting building sensors to maintain a comfortable environment for occupants.
• Enhancing risk management by tracking incidents and alerting key personnel when employees or assets are at risk.
• Optimizing delivery systems by triggering actions when customers arrive at a pickup zone.
• Improving security by detecting and responding to potential threats in Kubernetes clusters.

The key advantage of Drasi is its ability to track changes without data duplication or performance bottlenecks, making it ideal for real-time systems where immediate action is required.

Figure 1. Components of Drasi (Source: Microsoft)

In traditional systems, achieving what Drasi offers would require combining various tools and techniques. For example, to detect changes in databases, you’d use a tool like *Debezium for Change Data Capture (CDC) or implement custom polling mechanisms. Event processing would need stream-processing tools such as Apache Kafka to handle real-time data, and for automated reactions, custom services or functions* would be needed. This fragmented approach increases complexity, introduces maintenance challenges, and often leads to delays due to polling or batch processing, unlike Drasi’s streamlined, real-time solution.

Understanding Drasi

Drasi is composed of three fundamental components that work together to provide real-time change detection and automatic reactions within systems: Sources, Continuous Queries, and Reactions.

Figure.2 End to End Drasi scenario (Source: Microsoft)

Sources

In Drasi, Sources are responsible for connecting Drasi to the external systems (databases, message systems, etc.) and monitoring them for changes. These sources perform three key functions:

Processing Change Logs: They track the change log or feed from the system and push the data to subscribed **Continuous Queries.**

Translating Data: Non-graph systems like PostgreSQL or Kubernetes are transformed into a property graph model that Drasi can understand, making it easier to work with both relational and graph data sources.

Initializing Queries: Sources provide initial state data to Continuous Queries when they start, ensuring queries have the full picture of the data as changes begin flowing in.

Example syntax to define a source in YAML might look like:

apiVersion: v1
kind: Source
name: my-db
spec:
  kind: PostgreSQL
  properties:
    user: my-user
    password:
      kind: Secret
      name: pg-creds
      key: password

Continuous Queries

Continuous Queries, unlike traditional instantaneous queries that only capture data at a specific point in time, are designed to continuously monitor the database. These queries remain active and constantly update as changes occur in the data. When a change matches the criteria specified in a Continuous Query, Drasi not only updates the result set but also identifies exactly which data elements have been added, updated, or deleted. This eliminates the need to periodically rerun queries or compare snapshots of data to detect changes. Continuous Queries are written using ***Cypher Query Language, allowing developers to write complex graph-based queries to monitor and track changes across multiple sources. The unique aspect of Continuous Queries lies in their ability to provide an ongoing, real-time*** view of data as it changes, allowing developers to track specific conditions and execute context-aware actions. For example, in a logistics scenario, a Continuous Query can track when a product’s inventory level drops below a threshold and immediately trigger a replenishment process.

Here is a simple example of the definition for the Incident Alerting Continuous Query. Imagine an Incident Alerting Service which notifies managers if any of the employees in their team are at risk due to dangerous incidents happening in their location (e.g. fires, storms, protests, etc).

apiVersion: v1
kind: ContinuousQuery
name: manager-incident-alert
spec:
  sources:    
    subscriptions:
      - id: human-resources
  query: > 
    MATCH
      (e:Employee)-[:ASSIGNED_TO]->(t:Team),
      (m:Employee)-[:MANAGES]->(t:Team),
      (e:Employee)-[:LOCATED_IN]->(:Building)-[:LOCATED_IN]->(r:Region),
      (i:Incident {type:'environmental'})-[:OCCURS_IN]->(r:Region) 
    WHERE
      elementId(e) <> elementId(m) AND i.severity IN [‘critical’, ‘extreme’] AND i.endTimeMs IS NULL
    RETURN 
      m.name AS ManagerName, m.email AS ManagerEmail, 
      e.name AS EmployeeName, e.email AS EmployeeEmail,
      r.name AS RegionName, 
      elementId(i) AS IncidentId, i.severity AS IncidentSeverity, i.description AS IncidentDescription

Reactions

Reactions are triggered when Continuous Queries detect relevant data changes. Drasi provides standard reactions that forward these changes to external systems or execute commands directly within the system. For example:

• Changes can be forwarded to Azure Event Grid or SignalR, enabling real-time updates in applications or dashboards.
• Reactions can trigger stored procedures or Gremlin commands to update databases based on the detected changes, without requiring an intermediary layer or service.

This capability allows Drasi to integrate seamlessly with existing infrastructures and ensures immediate responses to changes, making it highly effective for real-time business processes.

Practical Example

In smart building management, Drasi can monitor room conditions stored in a Cosmos DB database, automatically updating dashboards when comfort levels change. A source in Drasi reads the change logs, continuous queries calculate real-time updates, and reactions push the data to a dashboard, allowing facilities managers to make timely adjustments.

Figure 3. Smart Building Management example

Architectural Design with Drasi

When designing solutions with Drasi, you can adopt multiple approaches depending on the level of complexity required. At the most basic level, Observing Changes allows Drasi to detect when data elements are created, modified, or deleted in source systems, similar to traditional notification systems. This method can track and respond to changes without manual intervention, making it ideal for simple setups.

For more sophisticated scenarios, Observing Conditions lets you detect when changes meet specific criteria or conditions. For instance, Drasi can trigger actions when a sensor value exceeds a threshold or when a data relationship meets predefined conditions. This eliminates the need for custom services to check for these events, significantly simplifying solution architecture.

The most advanced approach involves Observing Collections, where Drasi tracks dynamic collections of data and triggers actions based on changes within those collections. For example, in an e-commerce system, Drasi can manage collections of orders, updating relevant teams when orders move through different stages. This approach allows Drasi to manage complex workflows by continuously updating and processing real-time data changes without manual coding.

Drasi’s architecture provides flexibility, scalability, and automation, making it a powerful tool for event-driven solutions. By allowing architects to model data and processes efficiently, Drasi reduces the need for complex infrastructure and code, offering real-time insights and reactions.

When not to use Drasi

While Drasi is a powerful tool, there are situations where it might not be the best fit. If your system already has a mature change notification system optimized for its specific data model, using Drasi may add unnecessary complexity. For high-volume streaming data that requires advanced stream analytics, dedicated tools like Apache Kafka or Apache Flink might be more efficient. Additionally, if your system has highly complex data models that are abstracted away by an existing eventing mechanism, it may be simpler to stick with the built-in solution.

Conclusion

Drasi simplifies the complexities of managing data changes in modern systems by integrating multiple data sources, continuously monitoring for changes, and triggering automated, real-time reactions. It eliminates the need for building cumbersome custom solutions, enabling businesses to operate more efficiently. Drasi’s submission to the *CNCF *as a Sandbox project signals its potential as a key player in the cloud-native ecosystem, furthering the development of open-source, cloud-neutral technologies and empowering developers with flexible, scalable tools for modern applications.

References

1. Drasi (github.com)
2. Drasi.io