Designing a Mini Telemetry Collector: How Senior Engineers Approach LLD Differently

Recently, I decided to design a Mini Telemetry Collector as a learning project for distributed systems and observability. The goal wasn’t to build another toy application. The goal was to practice the same design thinking used when building systems like Prometheus, Grafana Alloy, or OpenTelemetry Collector.

Instead of jumping into code, I approached it like a Low-Level Design (LLD) interview.

Step 1: Understand Requirements

The requirements sounded deceptively simple:

• Applications expose metrics over HTTP
• Collector periodically scrapes metrics
• Support multiple targets
• Store metrics in memory
• Allow querying collected metrics

Example:

GET /metrics

Outputs:
http_requests_total 150
memory_usage_bytes 4096
cpu_usage_percent 52

At first glance, it feels like a straightforward coding exercise, it isn’t.

The interesting part is figuring out how responsibilities should be distributed across components.

Step 2: Identify Entities

A common mistake is creating classes directly from nouns in the problem statement.

The problem says “Telemetry Collector”, so many engineers immediately create:

Collector

without asking:

What responsibility does it actually own?

Instead, I started by identifying data objects and behavioral components separately.

Data Objects

Target Metric MetricQuery TargetStatus

Components

Scraper Parser Storage Orchestrator

Notice what’s missing:

Collector

We’ll come back to that.

Step 3: Assign Responsibilities

Every component should have a single clear responsibility.

Scraper

Responsible for fetching metrics from targets.

Parser

Responsible for converting raw text into structured metrics.

Input:

cpu_usage_percent 52

Output:

Metric{ Name: “cpu_usage_percent”, Value: 52 }

Storage

Responsible for:

• Storing metrics
• Retrieving metrics
• Managing concurrency

Orchestrator

Responsible for wiring components together.

Scraper
   |
Parser
   |
Storage

Nothing more.

Step 4: Delete Unnecessary Components

One of the most valuable design skills is knowing what NOT to build.

Initially, I had:

Collector

Then I asked What responsibility does Collector own?

After assigning responsibilities to Scraper, Parser, and Storage, the answer became “Nothing”. So it was removed.

This sounds simple, but many systems become difficult to maintain because they accumulate classes with no clear ownership. If a component has no meaningful responsibility, delete it.

Step 5: Design for Queries, Not Storage

At first, it was tempting to expose methods like:

GetMetricByName()
GetMetricByNameAndServer()
GetMetricByNameAndServerAndTimeRange()

This works for today’s requirements. It breaks tomorrow. Instead, I introduced:

MetricQuery{
  MetricName
  ServerName
  StartTime
  EndTime
}

Storage exposes:

Query(MetricQuery)

Now new requirements don’t require interface changes.

This is a simple example of designing for extensibility without over-engineering.

Step 6: Not Every Component Needs an Interface

This is where many LLD discussions go wrong.

I’ve seen designs like:

IScraper
IParser
IStorage
IQueryService
IManager
IProvider

with exactly one implementation each.Creating abstractions before they’re needed adds complexity without delivering value.

For this project Scraperand Parserneeds no Interface because there is only one implementation. Currently, we will need interface only for Storage

Because future implementations are likely:

MemoryStorage
RedisStorage
TSDBStorage
PostgresStorage

A useful rule:

Introduce abstractions where variability is expected. Not everywhere.

Step 7: Concurrency Is a Storage Problem

The collector supports multiple targets. The obvious solution is one goroutine per target

For 100 targets, 15-second scrape interval, this is perfectly reasonable.

However, multiple goroutines introduce another problem Scraper A, Scraper B, Scraper C may all write simultaneously. At the same time Query API may be reading.

Many engineers try to solve synchronization in the scraper, I prefer a different rule:

The component that owns the data owns synchronization.

Storage owns the data. Therefore Storage owns the mutex.

Storage
  RWMutex

This keeps concurrency concerns localized.

Step 8: Model Storage Around Access Patterns

Another common mistake is optimizing writes before understanding reads.

The first design considered:

map[metricName][]Metric

Then I looked at the query pattern:

server=app1 metric=cpu_usage_percent

This led to:

map[serverName]map[metricName][]Metric

Now both dimensions are indexed. A useful principle “Organize data based on how it is queried”, not how it is stored.

Step 9: Handle Failures Like a Production System

What happens if a target is down? Many toy projects simply log an error. A more realistic design:

Retry N times

If retries fail:

TargetStatus{ LastError LastSuccessTime ConsecutiveFailures }

This enables future features such as:

GET /targets/status

without touching metric storage. Separating operational state from business data keeps the model clean.

Final Thoughts

When I first started learning system design, I thought great engineers were distinguished by the patterns they knew. Over time, I realized the opposite is often true.

Great engineers spend less time adding abstractions and more time questioning whether those abstractions should exist at all.

In this design alone, we removed:

• Collector
• QueryService
• Several unnecessary interfaces

before writing a single line of code. That’s not simplification after the fact.

That’s design. If you enjoyed this article, you may also like my earlier post:

Good Engineers Code. Great Engineers Design.

https://medium.com/@sumitbhanushali16/good-engineers-code-great-engineers-design-4fedc3dc9b30

In the next article, I’ll start implementing this telemetry collector in Go and explore how design decisions influence code structure.