Mastering Graph Analytics in Azure Data Explorer

created by DALL·E with my prompt

Let’s imagine you’re staring at millions of rows of interconnected data — employees reporting to managers, devices communicating with other devices, or customers interacting with products. Traditional SQL joins start to get overcomplicated, inefficient and difficult to follow. This is exactly where Azure Data Explorer (ADX) graph functionality comes handy, and today we’re diving deep into how it transforms complex relationship analysis into an elegant solution.

Quick Azure Data Explorer and Kusto Query Language refresher

Azure Data Explorer (ADX) is Microsoft’s lightning-fast analytics service that excels at ingesting and querying massive amounts of time-series and log data. Kusto Query Language (KQL) is a powerful query language that feels natural for data exploration. If you’re new to KQL, I’d recommend checking out my previous article first (link), as we’ll be building on those foundational concepts throughout this piece.

Graph Analytics — core concepts

At its core, graph analytics represents data as nodes (entities) and edges (relationships) rather than rows and columns. Instead of wrestling with complex multi-table joins to understand how Alice reports to Bob who manages the DevOps team, graphs let you traverse these relationships naturally. Think social networks, organizational hierarchies, IoT device topologies, or supply chain dependencies — anywhere relationships matter as much as the data itself. Graphs mirror how we naturally think about connected information, making complex relationship queries very intuitive.

Two Flavors of Graph Functionality in ADX

ADX offers two distinct approaches to graph analytics, each serving different scenarios and scale requirements. Understanding when to use each approach can save you both time and resources.

1. Transient Graphs

Photo by Emmanuel Acua on Unsplash

Transient graphs are created dynamically using the make-graph operator. These graphs exist in memory during query execution and are automatically discarded when the query completes. Think of them as the perfect tool for exploratory analysis and one-off investigations.

Key characteristics:

• Zero setup overhead: No preprocessing, no schema definitions — just query and go
• Always current: Built from live data every time, so you’re always analyzing the latest state
• Memory-bound: Performance depends on your cluster’s available memory
• Cost-effective: No additional storage costs beyond your regular data

When to use

The sweet spot for transient graphs is when you need immediate insights into live data without the commitment of building persistent infrastructure and when your graph size isunder 10 million nodes and edges.

Example use cases:

• Ad-hoc analysis of organizational structures during a reorg
• Real-time IoT device relationship monitoring
• Exploring customer journey patterns for a specific campaign
• Prototyping graph patterns before committing to persistent model

2. Persistent Graphs

Photo by cant on Unsplash

Persistent graphs enable you to store, manage, and query graph data structures at scale. Persistent graphs are durable database objects that persist beyond individual query executions, providing enterprise-grade graph analytics capabilities.

What makes persistent graphs powerful:

• Schema-driven approach: Define node and edge types with properties for consistency
• Snapshot versioning: Capture graph states at different points in time
• Collaborative access: Multiple teams can share the same graph structure
• Performance optimization: No reconstruction overhead for repeated queries
• Enterprise scale: Handle graphs exceeding single-node memory limitations

When to use

The key insight here is that persistent graphs aren’t just “bigger transient graphs” — they’re fundamentally different tools optimized for different use cases. They are needed when you need repeated analysis, collaboration, and historical context especially on bigger data size.

Example use cases:

• Analyzing supply chain networks with millions of suppliers and products
• Digital twin implementations tracking complex IoT ecosystems over time
• Social network analysis requiring historical comparison
• Security analytics where you need to track user behavior patterns across time

Note: throughout this article I will be providing code examples available in Microsoft documentation, unless specified otherwise.

Building Transient Graphs

1. Transient Graphs (make-graph)

It starts with the make-graph operator. Here's a practical example using an organizational structure:

// Sample employee and reporting data
let employees = datatable(name: string, department: string, level: int)
[
    "Alice", "Engineering", 5,
    "Bob", "Engineering", 3,
    "Carol", "Marketing", 4,
    "Dave", "Engineering", 6
];
let reportsTo = datatable(employee: string, manager: string)
[
    "Bob", "Alice",
    "Alice", "Dave",
    "Carol", "Dave"
];
// Create and query the graph
reportsTo
| make-graph employee --> manager with employees on name
| graph-match (employee)-[reports*1..3]-(topManager)
where employee.name == "Bob"
project employee = employee.name, 
          managerChain = reports.manager,
          hops = array_length(reports)

This query creates a graph where employees are nodes connected by “reports to” relationships, then finds Bob’s reporting chain up to 3 levels. No complex joins, just natural graph traversal.

Pro tip for performance optimization: Always filter your data before creating the graph — select only relevant nodes, edges, and properties before graph creation.

Below code uses [hasManager2..5] *pattern. It matches paths that traverse the “hasManager” relationship between 2 and 5 times. In other words, it’s looking for management chains that are 2 to 5 levels deep.

let allEmployees = datatable(organization: string, name:string, age:long)
[
  "R&D", "Alice", 32,
  "R&D","Bob", 31,
  "R&D","Eve", 27,
  "R&D","Mallory", 29,
  "Marketing", "Alex", 35
];
let allReports = datatable(employee:string, manager:string, modificationDate: datetime)
[
  "Bob", "Alice", datetime(2022-05-23),
  "Bob", "Eve", datetime(2023-01-01),
  "Eve", "Mallory", datetime(2022-05-23),
  "Alice", "Dave", datetime(2022-05-23)
];
let filteredEmployees =
    allEmployees
    | where organization == "R&D"
    | project-away age, organization;
let filteredReports =
    allReports
    | summarize arg_max(modificationDate, *) by employee
    | project-away modificationDate;
filteredReports
| make-graph employee --> manager with filteredEmployees on name
| graph-match (employee)-[hasManager*2..5]-(manager)
  where employee.name == "Bob"
  project employee = employee.name, topManager = manager.name

Here's an example finding potential security risks in an IT infrastructure:

// Find devices that can reach critical assets through multiple hops
let devices = datatable(deviceId: string, deviceType: string, riskLevel: string)
[
    "srv-01", "server", "low",
    "router-02", "network", "medium", 
    "db-03", "database", "critical"
];
let connections = datatable(source: string, destination: string, connectionType: string)
[
    "srv-01", "router-02", "network",
    "router-02", "db-03", "network"
];
connections
| make-graph source --> destination with devices on deviceId
| graph-match (device)-[path*1..5]->(criticalAsset)
where device.riskLevel != "critical" and criticalAsset.riskLevel == "critical"
| project riskyDevice = device.deviceId,
          criticalAsset = criticalAsset.deviceId,
          pathLength = array_length(path),
          connectionPath = path.connectionType

This query identifies all paths from non-critical devices to critical assets, helping you understand potential attack vectors in your infrastructure.

2. Persistent Graphs (model & data snapshot)

Persistent graphs require definitely more upfront work. The process involves creating a graph model, then materializing it as snapshots.

Step 1: Define the Graph Model

It contains information about nodes, edges, parameters and connections between them. It needs to be defined with cautious because it defines how graph will be created.

.create-or-alter graph_model OrganizationalNetwork ```
{
    "Schema": {
        "Nodes": {
            "Employee": {
                "Name": "string",
                "Department": "string", 
                "Level": "int",
                "JoinDate": "datetime"
            },
            "Team": {
                "Name": "string",
                "Budget": "long"
            }
        },
        "Edges": {
            "ReportsTo": {
                "Since": "datetime",
                "Relationship": "string"
            },
            "MemberOf": {
                "Role": "string"
            }
        }
    },
    "Definition": {
        "Steps": [
            {
                "Kind": "AddNodes",
                "Query": "Employees | where IsActive == true | project Name, Department, Level, JoinDate",
                "NodeIdColumn": "Name",
                "Labels": ["Employee"]
            },
            {
                "Kind": "AddNodes", 
                "Query": "Teams | project TeamName as Name, Budget",
                "NodeIdColumn": "Name",
                "Labels": ["Team"]
            },
            {
                "Kind": "AddEdges",
                "Query": "ReportingStructure | project Employee, Manager, PromotionDate as Since | extend Relationship = 'DirectReport'",
                "SourceColumn": "Employee",
                "TargetColumn": "Manager", 
                "Labels": ["ReportsTo"]
            },
            {
                "Kind": "AddEdges",
                "Query": "TeamMembership | project Employee, TeamName, Role",
                "SourceColumn": "Employee",
                "TargetColumn": "TeamName",
                "Labels": ["MemberOf"]
            }
        ]
    }
}

Step 2: Create a Snapshot

This applies created graph model to the current data giving a snapshot, Thanks to this approach multiple historical graphs can be maintained which allows to history trends/scenario analysis.

.create graph snapshot OrgChart_2024Q1 from OrganizationalNetwork

Step 3: Query the Persistent Graph

graph("OrganizationalNetwork")
| graph-match (employee:Employee)-[reports:ReportsTo*1..5]->(executive:Employee)
where employee.Department == "Engineering" 
| graph-match (employee)-[memberOf:MemberOf]->(team:Team)
where team.Budget > 1000000
| project employeeName = employee.Name,
          executiveName = executive.Name,
          teamName = team.Name,
          reportingLevels = array_length(reports)

This query finds all Engineering employees who report up to executives and are also members of teams with budgets over $1M, showing the reporting hierarchy depth for each.

Historical analysis in Persistent Graph

Historical analysis is possible using multiple snapshots and lifecycle management:

.create graph snapshot SecurityAnalytics_Current from SecurityAnalyticsGraph

// Compare behavior patterns between time periods
let previous_patterns = graph("SecurityAnalyticsGraph", "SecurityAnalytics_LastMonth")
    | graph-match (person)-[browses]->(domain)
    | summarize domains = make_set(domain.DomainName) by person = person.Name;
let current_patterns = graph("SecurityAnalyticsGraph", "SecurityAnalytics_Current") 
    | graph-match (person)-[browses]->(domain)
    | summarize domains = make_set(domain.DomainName) by person = person.Name;
// Find employees with significantly changed browsing patterns
previous_patterns
| join kind=inner current_patterns on person
| extend new_domains = set_difference(domains1, domains)
| extend dropped_domains = set_difference(domains, domains1) 
| where array_length(new_domains) > 10 or array_length(dropped_domains) > 10
| project person, new_domains, dropped_domains

Key lifecycle management commands:

There are few important commands that help to keep track of models and snapshot created:

• .show graph_models - List all available graph models
• .show graph_snapshots - View all snapshots and their metadata
• .drop graph_snapshot <name> - Clean up old snapshots to manage storage costs

Remember that snapshots are point-in-time views, so you’ll need to refresh them periodically. In production, I recommend setting up automated processes to create daily or weekly snapshots depending on how frequently your underlying data changes.

ADX graphs vs traditional graph solutions (Neo4j)

While ADX’s graph capabilities are impressive for many scenarios, it is important to emphasize that this is not dedicated graph solution as such, rather an analytical addition to an existing data platform.

Below I will focus on main differences and use cases taking Neo4j as an example of robust, dedicated graph solution.

Key differences

The most important is that ADX graphs are tables pretending to be graphs. They take existing tabular data and apply graph queries on top. The data stays in columns and rows underneath. Wheras Neo4j stores actual graph structures. Nodes and relationships are real objects in storage, not derived from table joins. What this means in practice? Complex queries get messy in ADX. Multiple graph-match statements become hard to read.

Another differenceis that ADX doesn’t have more complex (yet usefull and popular) graph algorithms. Want PageRank or community detection? You’ll need to build it yourself in KQL. At the same time Neo4j comes with many algorithms built-in. Graph algorithms for recommendations, fraud detection, influence analysis — these require dedicated graph database capabilities.

ADX uses snapshots based on given time data content, Neo4j updates in real-time. There are pros and cons of these approaches, for ADX I find it really useful to be able to track historical changes and patterns. Modeling that behaviours in Neo4j would be more complex for sure.

When to Use What?

**Use ADX graphs when:

• **You’re already doing analytics in ADX and want to add graph analysis
• You need to combine graph patterns with time-series data
• Cost matters — you’re already paying for ADX

**Use dedicated graph solution when:

• **Graph traversal is your main use case, not just part of analytics
• You want built-in graph algorithms (PageRank, community detection, etc.)
• Your queries involve complex multi-step patterns

Or use both!

Wrapping Up — choosing the graph strategy

• Start with transient graphs when you need immediate insights, are working with smaller datasets, or are still exploring what questions to ask. They’re perfect for that investigative mindset where you’re not sure what you’re looking for yet.
• Graduate to persistent graphs when your analysis becomes routine, your datasets grow beyond 10 million nodes and edges, or multiple people need to collaborate on the same graph structure. The upfront investment in modeling pays off quickly with improved performance and consistency, especially for production workflows requiring reliable graph access.

I recommend to try it out in testing environment provided by Microsoft with sample data already loaded: Azure Data Explorer (KQL Playground)

Whether you’re untangling complex organizational structures, optimizing supply chains, or monitoring IoT ecosystems, graph analytics in ADX gives you the tools to see patterns that traditional row-and-column thinking might miss.

I hope you found this article helpful. Have you implemented graph analytics in your organization? I’d love to hear about your use cases and challenges in the comments below. You can also reach out to me directly via linkedin.