Observability in Backend Systems: Making Production Failures Understandable

How logs, metrics, traces, and correlation IDs help teams understand what happens in production

Observability in backend systems — logs, metrics, traces, and making production failures understandable

In Part 1, I wrote about boundaries. In Part 2, I wrote about aggregates and invariants. In Part 3, I wrote about domain events. In Part 4, I wrote about commands and queries. In Part 5, I wrote about modular monoliths and microservices. In Part 6, I wrote about synchronous vs asynchronous communication. In Part 7, I wrote about HTTP, messaging, and gRPC. In Part 8, I wrote about resiliency in backend systems.

Now we move to a topic that is often added too late:

observability.

Many teams build systems, deploy them, and only then ask:

• Why did this request fail?
• Why is this API slow?
• Did the message get processed?
• Where did the payment workflow stop?
• Which dependency is causing the issue?
• Why did the user see an error?

If the system cannot answer those questions, production support becomes guesswork.

And guesswork is not a strategy.

Observability is what allows teams to understand what is happening inside a system without manually debugging production.

Observability is more than logging

A common mistake is thinking observability means adding logs everywhere.

Logs are important, but they are only one part of the picture.

A backend system usually needs three main signals:

• logs
• metrics
• traces

Together, they help answer different questions.

Logs tell us what happened. Metrics tell us how often and how badly it is happening. Traces tell us where time was spent across the flow.

A production-ready system needs all three.

Why observability matters in backend architecture

Architecture is not only about designing clean modules and correct flows.

It is also about making the system understandable when it is running.

A system may look clean in diagrams, but still be painful in production if no one can answer basic operational questions.

For example:

• A booking request failed, but no one knows why
• Payment was confirmed, but the consultation status did not update
• Notification was sent twice
• A video session could not start
• A background job failed silently
• A downstream service became slow and caused timeouts

These are not rare situations.

They are normal production realities.

Observability helps us investigate these problems quickly and confidently.

The real goal of observability

The goal is not to collect as much data as possible.

That creates noise.

The real goal is to make important system behavior visible.

A useful observability setup should help answer:

• What happened?
• When did it happen?
• Who or what triggered it?
• Which system component handled it?
• How long did it take?
• Did it succeed or fail?
• Where did the failure happen?
• Was this isolated or widespread?

That is the difference between raw logs and useful operational insight.

Logs: recording meaningful events

Logs are useful when they capture meaningful events in the system.

But many systems either log too little or too much.

Too little logging

You only see:

Error occurred.

This is almost useless.

It does not tell you:

• Which request failed
• Which entity was involved
• Which user was affected
• Which dependency failed
• Which business operation was running

Too much logging

You log every small internal step.

Now production logs become noisy, expensive, and difficult to search.

Good logging is intentional.

It focuses on important technical and business events.

What should be logged?

In backend systems, useful logs usually include:

• important state transitions
• failed business operations
• external dependency calls
• retry attempts
• timeout failures
• message processing failures
• security-sensitive actions
• unexpected exceptions

For example, in a telemedicine system, useful business logs might include:

• consultation requested
• doctor accepted consultation
• payment confirmed
• video session started
• consultation completed
• prescription generated

These logs help the team understand the business workflow, not just the technical execution.

Structured logs are better than plain text logs

A weak log message looks like this:

Consultation completed successfully

It is readable, but not very useful for searching and filtering.

A better structured log includes important properties:

logger.LogInformation(
    "Consultation {ConsultationId} completed by Doctor {DoctorId}. CorrelationId: {CorrelationId}",
    consultationId,
    doctorId,
    correlationId);

This allows logs to be searched by:

• ConsultationId
• DoctorId
• CorrelationId
• event type
• timestamp
• severity

Structured logs make production investigation much easier.

Correlation IDs: connecting the whole flow

One of the most important observability concepts is the correlation ID.

A correlation ID is a value that connects all logs, traces, and operations related to the same request or workflow.

Without it, logs from one user action may be scattered across:

• API layer
• application handler
• domain event handler
• database operation
• external provider call
• background job
• notification service

The team then has to manually guess which logs belong together.

A correlation ID solves this by giving the entire flow one shared identifier.

CorrelationId: 8f7c9b72-8e4a-4f8d-b65e-2a5f7c1d9132

Every log related to that request or workflow should include it.

Why correlation IDs matter

Imagine this flow:

1. patient books a consultation
2. booking succeeds
3. payment request is created
4. payment callback is received
5. notification is triggered
6. video session is prepared

If something fails at step 5, the team should be able to trace the full path from the original booking request to the notification failure.

That is almost impossible without correlation.

Correlation IDs turn disconnected logs into a readable story.

Metrics: seeing the health of the system

Logs are useful for investigating individual events.

Metrics are useful for understanding system behavior at scale.

A metric answers questions like:

• How many requests are failing?
• What is the average response time?
• How many payments are pending?
• How many messages are stuck?
• How many retries are happening?
• Is the system getting slower over time?
• Did the failure rate increase after deployment?

Metrics are essential because humans cannot read logs all day and detect patterns manually.

The system should expose health indicators clearly.

Useful backend metrics

For a backend system, useful metrics may include:

API metrics

• request count
• request duration
• error rate
• timeout count
• status code distribution

Database metrics

• query duration
• connection pool usage
• failed database operations
• transaction duration

Messaging metrics

• queue length
• message processing duration
• retry count
• dead-letter count
• duplicate message count

Business metrics

• consultations booked
• payments confirmed
• consultations completed
• failed payment confirmations
• prescription generation failures

Technical metrics show system health.

Business metrics show workflow health.

You usually need both.

Business metrics are often more useful than teams expect

A system may be technically healthy but business-broken.

For example:

• APIs are returning 200 OK
• CPU usage is normal
• The database is responsive

But:

• Payment confirmations are not updating consultations
• Notifications are not being delivered
• Prescriptions are not generated
• Doctors are not seeing new requests

From a pure infrastructure perspective, the system may look fine.

From a business perspective, it is failing.

That is why business-level metrics matter.

Traces: understanding the journey of a request

A trace shows the path of a request or workflow across components.

It helps answer:

• Where did the request go?
• Which component took the most time?
• Which dependency failed?
• How long did each step take?
• Where did the timeout happen?

This is especially important in systems with:

• multiple modules
• external APIs
• message processing
• background jobs
• distributed services

Traces are extremely useful when one user action crosses many boundaries.

Example: tracing a consultation booking

A consultation booking flow may include:

1. API endpoint receives request
2. Command handler validates input
3. Scheduling module checks availability
4. The consultation aggregate is created
5. Database transaction commits
6. Domain event is stored in outbox
7. The background worker publishes event
8. The notification handler sends message

A trace helps show how long each step took and where the flow failed.

Without tracing, you may only see the final error.

With tracing, you can see the journey.

Observability in synchronous workflows

Synchronous workflows are usually easier to follow because the caller is waiting for a response.

But they still need strong observability.

For synchronous operations, observe:

• request duration
• validation failures
• dependency latency
• timeout events
• retry attempts
• final success or failure

For example:

• booking request duration
• payment provider response time
• video provider session creation latency
• database transaction duration

If an API becomes slow, traces and metrics should help identify whether the problem is:

• application logic
• database query
• external API
• network delay
• lock contention
• retry behavior

Without observability, every slowdown becomes a mystery.

Observability in asynchronous workflows

Asynchronous workflows need even more careful observability.

Why?

Because the user request may already be completed while background work continues later.

For example:

• Appointment booked successfully
• Notification is queued
• Reminder job is created
• The reporting update happens later

If the notification fails, the user may not immediately know.

The system must still make that failure visible.

For asynchronous processing, observe:

• message received
• message processed
• message failed
• retry attempted
• message was moved to the dead-letter
• duplicate message ignored
• processing duration

This is critical for event-driven systems.

Dead-letter queues need monitoring

A dead-letter queue is useful only if someone monitors it.

Otherwise, it becomes a graveyard for failed messages.

If messages are moved to dead-letter, the system should expose:

• How many messages failed
• What type of message failed
• When failures started
• Which handler failed
• Why processing failed

A growing dead-letter queue is usually a signal that something needs attention.

It should not be invisible.

Observability and resiliency are connected

Part 8 focused on resiliency.

Observability and resiliency are deeply connected.

A resilient system needs to know:

• How often retries happen
• Which dependencies timeout
• When circuit breakers open
• How many requests fail fast
• How many duplicate messages are ignored
• Whether fallback behavior is being used too often

Without observability, resiliency patterns may hide problems.

For example, retries may make the system appear successful while quietly increasing latency and load.

Fallbacks may keep the UI running while an important dependency is failing repeatedly.

Circuit breakers may protect the system, but they also signal that something is unhealthy.

So every resiliency pattern should be observable.

Alerts: when the system should ask for attention

Observability is not useful if no one notices important failures.

That is where alerts come in.

But alerts should be designed carefully.

Too many alerts create noise. Too few alerts allow problems to go unnoticed.

Good alerts are usually tied to user impact or business impact.

For example:

• The payment failure rate is above normal
• Consultation booking error rate increased
• Video session creation failures are rising
• Dead-letter messages are increasing
• API latency is above the acceptable threshold
• Prescription generation failed repeatedly

Alerts should not wake people up for every small technical detail.

They should focus on symptoms that matter.

Dashboards: making system health visible

Dashboards are useful when they show the right information.

A backend dashboard might include:

• API error rate
• API latency
• request volume
• database latency
• queue length
• failed message count
• payment confirmation failures
• notification delivery failures
• consultation completion count

For a telemedicine platform, a useful dashboard might show:

• consultations requested today
• consultations accepted
• payments pending
• active consultations
• completed consultations
• failed video session creations
• prescription generation failures

This gives both technical and business visibility.

Common mistake: logging only technical events

Many systems log technical events but ignore business events.

For example:

Request completed
Database query executed
Message consumed

These are useful, but incomplete.

In business systems, we also need logs that reflect the domain:

Consultation accepted
Payment confirmed
Prescription generated
Appointment cancelled

Business event logs make it easier to understand what actually happened from the user’s perspective.

A production issue is rarely just technical.

It usually affects a business workflow.

Common mistake: no consistent identifiers

Another common mistake is logging useful messages without consistent identifiers.

For example:

Payment confirmed

But which payment? Which consultation? Which patient? Which request? Which provider reference?

Better:

logger.LogInformation(
    "Payment confirmed for Consultation {ConsultationId}. PaymentReference: {PaymentReference}. CorrelationId: {CorrelationId}",
    consultationId,
    paymentReference,
    correlationId);

Identifiers make logs searchable and useful.

Without identifiers, logs become stories with missing names.

Common mistake: logging sensitive data

Observability must not become a security risk.

Backend logs should not casually include:

• passwords
• tokens
• full payment details
• sensitive medical notes
• personal health information
• unnecessary personal data

This is especially important in domains like healthcare, finance, and legal systems.

Log what is needed for diagnosis.

Do not log everything just because it is available.

Good observability respects privacy and security boundaries.

Observability should be designed, not sprinkled

A common anti-pattern is adding logs randomly after bugs appear.

That leads to inconsistent, noisy, and incomplete observability.

A better approach is to design observability around important workflows.

For each critical workflow, ask:

• What is the start of the flow?
• What are the key state transitions?
• What external dependencies are involved?
• What failures should be visible?
• What metrics indicate health?
• What identifiers connect the flow?
• What alerts are needed?

This turns observability into an architecture concern, not an afterthought.

Example: observability for payment confirmation

For payment confirmation, we might observe:

Logs

• Payment callback received
• Payment reference already processed
• Payment confirmed
• Consultation status updated
• Failure reason if confirmation fails

Metrics

• payment confirmation count
• failed confirmation count
• duplicate callback count
• confirmation processing duration

Traces

• callback endpoint
• payment validation
• database update
• consultation update
• event/outbox write

Alerts

• payment confirmation failures exceed threshold
• duplicate callbacks spike unexpectedly
• payment provider latency increases

This gives the team real operational visibility into an important business flow.

Example: observability for video sessions

For video session creation, we might observe:

Logs

• Video session requested
• Provider call started
• Provider call succeeded
• Provider call failed
• Session token generated

Metrics

• session creation latency
• provider failure rate
• successful session count
• failed session count

Traces

• API request
• authorization check
• consultation state check
• provider call
• response returned to the client

Alerts

• provider failure rate increases
• session creation latency crosses threshold
• video join failures spike

This is practical observability.

It connects technical events to user experience.

Observability in a modular monolith

Observability is not only for microservices.

A modular monolith also needs it.

Even inside one deployable application, workflows can cross multiple modules:

• Scheduling
• Consultation
• Payments
• Notifications
• Prescription

A request may still involve multiple handlers, domain events, database operations, and background workers.

So the modular monolith should still use:

• structured logs
• correlation IDs
• metrics
• traces
• workflow-level visibility

This makes future extraction easier, too.

If one module later becomes a separate service, the observability model is already in place.

A simple observability checklist

For each important backend workflow, ask:

• Do we log the start and end of the workflow?
• Do we log important state transitions?
• Do we include correlation IDs?
• Do we include business identifiers?
• Do we avoid sensitive data?
• Do we measure latency?
• Do we measure failure rate?
• Do we trace external calls?
• Do we monitor background processing?
• Do we alert on business-impacting failures?

If the answer is no, production support will likely be harder than it needs to be.

Final thought

Observability is not just a production add-on.

It is part of the backend architecture.

A system that cannot explain itself in production is incomplete, no matter how clean the code looks.

Logs help us understand what happened. Metrics help us understand how the system is behaving. Traces help us understand where time and failures occur. Correlation IDs connect the story across the workflow.

Together, they turn production issues from mysteries into investigations.

That is the real value of observability.

It helps teams move from guessing to understanding.

And in production, that difference matters.

Coming next

In Part 10, I will cover:

**Transactions and eventual consistency — when backend systems need immediate consistency, and when they can safely allow delayed consistency**