From Logs to Analytics: Designing ADX for Long-Term Observability

The customer operated multiple Kubernetes clusters generating high-volume application logs. They already had a mature log shipping layer using Vector, which gave them strong control over parsing, enrichment, filtering, and routing.

Their requirements were clear:

• Sustain very high ingestion throughput
• Support deep investigative analytics
• Remain cost-efficient at scale
• Allow future growth without re-architecture

Initially, observability looked simple.

A managed logging platform. Clean dashboards. Alerts configured. Everything under control.

Then scale happened.

Log volume grew from a few gigabytes per day to hundreds. Retention requirements expanded beyond 30 days. The questions changed.

Instead of “Is the service down?” engineers began asking:

• Why did latency increase across three clusters last Tuesday?
• What is the error distribution by deployment version?
• Can we reconstruct a transaction across services six months back?
• How does memory pressure correlate with rollout windows?

At that point, traditional logging platforms started to feel restrictive. Sometimes financially. Sometimes analytically.

That is where Azure Data Explorer entered the conversation.

Architecture Is the Real Observability Decision

Real systems are messy.

Schemas drift. Applications evolve. Logs break. Teams grow. Security wants its own pipeline. Compliance asks for reprocessing six months later.

We evaluated several ingestion patterns.

Option 1: Direct HTTP Ingestion

Architecture:

Application → Vector → ADX HTTP ingestion endpoint → Table

Vector uses its HTTP sink to push logs directly into ADX using the ingestion REST API. Example from local file:

# Data Directory
data_dir = "./vector-data"

# Global Configuration
[api]
enabled = true
address = "127.0.0.1:8686"

# Source - Read all events from both apps
[sources.all_apps]
type = "file"
include = ["logs/events.log"]
read_from = "end"

# Transform - Parse JSON and add metadata
[transforms.parse_events]
type = "remap"
inputs = ["all_apps"]
source = '''
  # Parse the JSON log entry
  . = parse_json!(string!(.message))

  # Add host information if not present
  if !exists(.host) {
    .host = get_hostname!()
  }

  # Add source identifier
  if !exists(.source) {
    .source = "vector-demo"
  }

  # Ensure app_type exists (critical for routing)
  if !exists(.app_type) {
    .app_type = "unknown"
  }
'''

# Route - Split events by app_type
[transforms.route_by_app]
type = "route"
inputs = ["parse_events"]

route.app_a = '.app_type == "app_a"'
route.app_b = '.app_type == "app_b"'

# Sink - App A (Logs) to ADX via HTTP
[sinks.adx_app_a]
type = "http"
inputs = ["route_by_app.app_a"]
uri = "${ADX_APP_A_ENDPOINT}"
encoding.codec = "json"

# Authentication
auth.strategy = "bearer"
auth.token = "${ADX_BEARER_TOKEN}"

# Headers
[sinks.adx_app_a.request.headers]
Content-Type = "application/json"
Accept = "application/json"

# Batch settings (ADX can handle larger batches)
[sinks.adx_app_a.batch]
timeout_secs = 5
max_bytes = 1048576  # 1MB

# Sink - App B (Metrics) to ADX via HTTP
[sinks.adx_app_b]
type = "http"
inputs = ["route_by_app.app_b"]
uri = "${ADX_APP_B_ENDPOINT}"
encoding.codec = "json"

# Authentication
auth.strategy = "bearer"
auth.token = "${ADX_BEARER_TOKEN}"

# Headers
[sinks.adx_app_b.request.headers]
Content-Type = "application/json"
Accept = "application/json"

# Batch settings
[sinks.adx_app_b.batch]
timeout_secs = 5
max_bytes = 1048576  # 1MB

# Console sink for debugging
[sinks.console]
type = "console"
inputs = ["parse_events"]
encoding.codec = "json"

Pros

• Minimal architecture
• Lower infrastructure cost
• Fewer components to manage
• Fast implementation
• Straightforward ownership

Cons

• No external durable buffer
• If ADX throttles, retries happen at the shipper level
• No independent replay capability
• Harder to introduce additional consumers
• Tighter coupling between producers and ADX schema
• Routing complexity if many apps dynamically map to tables

ADX does provide internal ingestion queuing, but there is no user-controlled streaming layer in this model.

At small scale, this works well.

At high scale with long retention and evolving schemas, it becomes fragile.

Option 2: Single Event Hub with Multiple Consumer Groups

Architecture:

Applications → Vector → Event Hub → Multiple Consumers

ADX reads with one consumer group. Security tooling or ML pipelines use others.

Example from local file:

# Vector Configuration - Multi-App Routing
# Routes multiple apps to same Event Hub with app_type field and different consumer groups
# ADX update policies handle the routing to different tables

data_dir = "./vector-data"

[api]
enabled = true
address = "127.0.0.1:8686"

# Source - Read all events (both apps write to same file)
[sources.all_apps]
type = "file"
include = ["logs/events.log"]
read_from = "end"

# Transform - Parse and ensure app_type field exists
[transforms.parse_events]
type = "remap"
inputs = ["all_apps"]
source = '''
. = parse_json!(string!(.message))

# Add host information if not present
if !exists(.host) {
  .host = get_hostname!()
}

# Add source identifier
.source = "vector-demo"

# Ensure app_type exists (critical for ADX routing)
if !exists(.app_type) {
  .app_type = "unknown"
}
'''

# Route - Split events by app_type
[transforms.route_by_app]
type = "route"
inputs = ["parse_events"]
route.app_a = '.app_type == "app_a"'
route.app_b = '.app_type == "app_b"'

# Transform - Add Table field for ADX routing (App A)
[transforms.add_table_app_a]
type = "remap"
inputs = ["route_by_app.app_a"]
source = '''
.Table = "AppA_Data"
'''

# Transform - Add Table field for ADX routing (App B)
[transforms.add_table_app_b]
type = "remap"
inputs = ["route_by_app.app_b"]
source = '''
.Table = "AppB_Data"
'''

# Sink - App A to Event Hub/Kafka
[sinks.kafka_app_a]
type = "kafka"
inputs = ["add_table_app_a"]
bootstrap_servers = "${EVENTHUB_NAMESPACE}.servicebus.windows.net:9093"
topic = "${EVENTHUB_NAME}"
compression = "none"
group_id = "app_a"

[sinks.kafka_app_a.encoding]
codec = "json"

[sinks.kafka_app_a.sasl]
enabled = true
mechanism = "PLAIN"
username = "$$ConnectionString"
password = "${EVENTHUB_CONNECTION_STRING}"

[sinks.kafka_app_a.tls]
enabled = true

[sinks.kafka_app_a.librdkafka_options]
"api.version.request" = "false"
"broker.version.fallback" = "1.0.0"
"request.required.acks" = "1"

[sinks.kafka_app_a.batch]
timeout_secs = 1
max_bytes = 1048576

# Sink - App B to Event Hub/Kafka
[sinks.kafka_app_b]
type = "kafka"
inputs = ["add_table_app_b"]
bootstrap_servers = "${EVENTHUB_NAMESPACE}.servicebus.windows.net:9093"
topic = "${EVENTHUB_NAME}"
compression = "none"
group_id = "app_b"

[sinks.kafka_app_b.encoding]
codec = "json"

[sinks.kafka_app_b.sasl]
enabled = true
mechanism = "PLAIN"
username = "$$ConnectionString"
password = "${EVENTHUB_CONNECTION_STRING}"

[sinks.kafka_app_b.tls]
enabled = true

[sinks.kafka_app_b.librdkafka_options]
"api.version.request" = "false"
"broker.version.fallback" = "1.0.0"
"request.required.acks" = "1"

[sinks.kafka_app_b.batch]
timeout_secs = 1
max_bytes = 1048576

# Console sinks for debugging
[sinks.console_app_a]
type = "console"
inputs = ["add_table_app_a"]

[sinks.console_app_a.encoding]
codec = "json"

[sinks.console_app_b]
type = "console"
inputs = ["add_table_app_b"]

[sinks.console_app_b.encoding]
codec = "json"

Pros

• Independent readers
• Replay flexibility per consumer
• Clean extensibility model
• Supports multi-pipeline architectures

Cons

• Still shared ingestion bottleneck
• Partition scaling must satisfy all workloads
• Throughput tuning impacts everyone
• Monitoring and capacity planning become more complex

This improves extensibility but does not address producer isolation.

Option 3: Event Hub per Application or Domain

Architecture:

Applications → Vector → Dedicated Event Hub → ADX Table

Example from local file:

# Vector Configuration - Multi-App Routing
# Routes multiple apps to different EventHubs
# ADX update policies handle the routing to different tables

data_dir = "./vector-data"

[api]
enabled = true
address = "127.0.0.1:8686"

# -------------------
# Sources
# -------------------

[sources.all_apps]
type = "file"
include = ["logs/events.log"]
read_from = "end"

# -------------------
# Transforms
# -------------------

[transforms.parse_events]
type = "remap"
inputs = ["all_apps"]
source = '''
. = parse_json!(string!(.message))

# Add host information if not present
if !exists(.host) {
  .host = get_hostname!()
}

# Add source identifier
.source = "vector-demo"

# Ensure app_type exists (critical for ADX routing)
if !exists(.app_type) {
  .app_type = "unknown"
}
'''

[transforms.route_by_app]
type = "route"
inputs = ["parse_events"]
route.app_a = '.app_type == "app_a"'
route.app_b = '.app_type == "app_b"'

[transforms.add_table_app_a]
type = "remap"
inputs = ["route_by_app.app_a"]
source = '''
.Table = "AppA_Data"
'''

[transforms.add_table_app_b]
type = "remap"
inputs = ["route_by_app.app_b"]
source = '''
.Table = "AppB_Data"
'''

# -------------------
# Sinks - App A
# -------------------

[sinks.kafka_app_a]
type = "kafka"
inputs = ["add_table_app_a"]
bootstrap_servers = "${EVENTHUB_NAMESPACE_A}.servicebus.windows.net:9093"
topic = "${EVENTHUB_NAME_A}"
compression = "none"

[sinks.kafka_app_a.encoding]
codec = "json"

[sinks.kafka_app_a.sasl]
enabled = true
mechanism = "PLAIN"
username = "$$ConnectionString"
password = "${EVENTHUB_CONNECTION_STRING_A}"

[sinks.kafka_app_a.tls]
enabled = true

[sinks.kafka_app_a.librdkafka_options]
"api.version.request" = "false"
"broker.version.fallback" = "1.0.0"
"request.required.acks" = "1"

[sinks.kafka_app_a.batch]
timeout_secs = 1
max_bytes = 1048576

# -------------------
# Sinks - App B
# -------------------

[sinks.kafka_app_b]
type = "kafka"
inputs = ["add_table_app_b"]
bootstrap_servers = "${EVENTHUB_NAMESPACE_B}.servicebus.windows.net:9093"
topic = "${EVENTHUB_NAME_B}"
compression = "none"

[sinks.kafka_app_b.encoding]
codec = "json"

[sinks.kafka_app_b.sasl]
enabled = true
mechanism = "PLAIN"
username = "$$ConnectionString"
password = "${EVENTHUB_CONNECTION_STRING_B}"

[sinks.kafka_app_b.tls]
enabled = true

[sinks.kafka_app_b.librdkafka_options]
"api.version.request" = "false"
"broker.version.fallback" = "1.0.0"
"request.required.acks" = "1"

[sinks.kafka_app_b.batch]
timeout_secs = 1
max_bytes = 1048576

# -------------------
# Console Sinks
# -------------------

[sinks.console_app_a]
type = "console"
inputs = ["add_table_app_a"]

[sinks.console_app_a.encoding]
codec = "json"

[sinks.console_app_b]
type = "console"
inputs = ["add_table_app_b"]

[sinks.console_app_b.encoding]
codec = "json"

This is the architecture we chose.

Pros

• Fault isolation between applications
• Independent partition scaling
• Clear ownership boundaries
• Reduced noisy neighbor risk
• Cleaner monitoring per workload
• Clear cost attribution

Cons

• Higher infrastructure cost
• More Event Hubs to manage
• Increased operational complexity
• Requires governance to prevent sprawl

In high-scale systems, isolation is often more valuable than consolidation.

This model introduced a durable streaming backbone while preserving clean separation between producers and consumers.

Option 4: Raw Staging Tables with Update Policies

We also evaluated introducing a raw staging pattern inside ADX.

Architecture:

Applications → Vector → Event Hub → ADX Raw Table → Update Policies → Curated Tables

Example from local file:

# Vector Configuration
# Applications → Vector → Event Hub
# --------------------------------------------

data_dir = "./vector-data"

[api]
enabled = true
address = "127.0.0.1:8686"

# Source - Application logs
[sources.app_logs]
type = "file"
include = ["logs/events.log"]
read_from = "end"

# Transform - Minimal normalization (raw preservation)
[transforms.normalize]
type = "remap"
inputs = ["app_logs"]
source = """
# Parse JSON payload
. = parse_json!(string!(.message))

# Preserve full raw payload for ADX Raw Table
.raw_payload = .

# Add ingestion metadata
.ingest_time = now()
.source = "vector"
"""

# Sink - Event Hub (Kafka endpoint)
[sinks.eventhub_raw]
type = "kafka"
inputs = ["normalize"]
bootstrap_servers = "${EVENTHUB_NAMESPACE}.servicebus.windows.net:9093"
topic = "${EVENTHUB_NAME}"
compression = "none"

[sinks.eventhub_raw.encoding]
codec = "json"

[sinks.eventhub_raw.sasl]
enabled = true
mechanism = "PLAIN"
username = "$$ConnectionString"
password = "${EVENTHUB_CONNECTION_STRING}"

[sinks.eventhub_raw.tls]
enabled = true

[sinks.eventhub_raw.librdkafka_options]
"api.version.request" = "false"
"broker.version.fallback" = "1.0.0"
"request.required.acks" = "1"

[sinks.eventhub_raw.batch]
timeout_secs = 1
max_bytes = 1048576

In this pattern, the raw table stores the full dynamic payload. Update policies project structured columns into curated tables used for analytics.

Pros

• Raw payload preservation
• Safe schema evolution
• Reprocessing capability
• Clear separation between ingestion and analytics
• Reduced ingestion-time transformation risk

Cons

• Higher storage footprint
• More complex data model
• Additional operational overhead
• Requires strong lifecycle governance

This pattern is powerful, especially in environments with frequent schema drift or strong data engineering requirements.

However it becomes complex when each application has different log schema.

The Final Architecture

Applications → Vector → Event Hub (per application or domain) → Corresponding ADX Table

This provided:

• Durable buffering
• Replay capability
• Independent scaling characteristics
• Clear isolation boundaries
• Long-term analytical power in ADX

Yes, this architecture costs more than direct HTTP ingestion.

But architecture should be evaluated in terms of risk and evolution.

The streaming backbone allowed:

• Absorbing traffic spikes
• Adding new consumers without redesign
• Scaling workloads independently
• Maintaining clean failure boundaries
• Preserving long-term retention with analytical depth

The HTTP sink model was cheaper and simpler. It also shifted risk upstream and limited extensibility.

For small systems, that trade-off may be acceptable.

For high-scale observability platforms with long retention and investigative workloads, it becomes a liability.

Final Reflection

Observability is not about dashboards. It is not even about ADX.

It is about designing an ingestion path that can evolve without collapsing under growth.

In this case, introducing Event Hub per application was not about adding complexity.

It was about reducing systemic risk.

In distributed systems, reducing risk is often the most cost-effective decision you can make.