Practical observability checklist for APIs, workers & jobs. Part 2

Logs, traces, and alerts that actually help when things go wrong.

In the first part, we focused on metrics: the signals that help answer whether an API, worker, or scheduled job is actually doing its job. But metrics only tell part of the story — they can tell you that something is wrong, not necessarily what happened or where.

This time, we’ll look at the other pieces of the observability puzzle: logs, traces, alerting, and a practical rollout path for building a setup that actually helps during real incidents.

Logs, metrics, and traces: what “good enough” looks like

Once the workload-specific signals are clear, the next question is where logs, metrics, and tracing fit. Metrics tell you something is wrong. Logs explain what happened. Traces connect behavior across components. They’re complementary — each answers a different question, and none of them replaces the others. If you’re not sure which one you need:

Metrics: answer operational questions

Good metrics are not “everything we could measure” — they’re signals that answer operational questions. Before adding a metric, it’s worth asking: what question does this answer, who will look at it, during what kind of incident, and what action could follow? If the answer is unclear, the metric will become noise.

Common metric pitfalls:

• high-cardinality labels such as raw user IDs, account IDs, full URLs, request IDs,
• too many dashboard panels with no investigation flow,
• histograms with unusable buckets,
• counters without clear outcome labels,
• gauges that are updated only on success and silently become stale.

Logs: reconstruct what happened

If logs are meant to be searched, correlated, filtered, or used during incidents, they should carry explicit fields instead of hiding meaning in string position. Unstructured logs may still be acceptable for low-volume, human-facing, local, or legacy logs, but for production application logs, structured logging should be the default.

A good log line is a structured event:

{
  "event": "event_processing_failed",
  "service": "analytics-api",
  "environment": "production",
  "handler": "ScoringUpdatedHandler",
  "event_type": "scoring_updated",
  "error_class": "DatabaseTimeoutError",
  "outcome": "failed",
  "duration_ms": 843,
  "trace_id": "..."
}

The format is JSON, but that’s not the point. The point is that the meaning is explicit.

Useful fields:

• timestamp,
• level,
• service name,
• environment,
• version or deploy SHA,
• request ID / correlation ID / trace ID,
• operation name,
• workload type,
• entity ID when safe and bounded,
• error class,
• outcome,
• duration when relevant.

Log the events that help reconstruct the story:

• request received,
• dependency call failed,
• event handling started/finished/failed,
• job started/finished/failed,
• retry scheduled,
• cache invalidated,
• unexpected state detected.

Common logging pitfalls:

• unstructured strings that are hard to query,
• missing correlation IDs,
• logging sensitive data,
• logging too much at error level,
• logs without stable event names,
• logs that describe symptoms but omit context.

Be careful with log levels. They help, but don’t decide which events must always be visible. Don’t hide events essential for understanding production behavior only behind DEBUG — log them at INFO with structured context. Reserve ERROR for failures that actually represent failed work or real user impact.

Tracing: connect the path

Tracing doesn’t have to be all-or-nothing. For most teams “minimal viable tracing” is enough to start — just add spans at the boundaries that matter:

• API request enters the service,
• database query happens,
• cache call happens,
• external HTTP call happens,
• event is published or consumed
• job step starts and finishes.

Two spans: one slow DB query and one slow downstream call account for ∽80% of the latency.

Minimal viable tracing is also about being intentional with sampling. You may not need to keep every successful trace, especially in a high-traffic system. But don’t let sampling hide the ones you need during debugging. A practical rule is to sample successful executions, but always keep 100% of traces for errors — 5xx responses, unhandled exceptions, failed worker handlers, failed jobs. If something failed, the trace is probably worth keeping.

Tracing becomes especially valuable when:

• one user request crosses several services,
• API behavior depends on downstream systems,
• workers and jobs interact with databases, caches, or external APIs,
• latency can come from multiple places,
• logs alone do not show the full path.

Common tracing pitfalls:

• tracing everything without a purpose,
• missing service/deploy context,
• not propagating trace IDs across components,
• collecting spans but not using them in incident workflows,
• assuming tracing replaces metrics or logs.

It’s not the first thing I’d add. But once basic metrics and structured logs are in place, minimal tracing removes a lot of guesswork.

Alerting that actually helps

Having the right signals is only half the problem. The other half is alerting on them well — and this is where teams often suffer twice: first from too little observability, then from too many bad alerts.

A good alert tells you that something actionable is wrong. “A datapoint looked weird for 30 seconds” or “a graph moved” don’t do that.

Useful API alerts

Good API alert candidates:

• service not ready for sustained period,
• sustained 5xx ratio above threshold,
• sustained p95/p99 latency above threshold,
• critical endpoint failure rate,
• traffic drops to zero unexpectedly,
• dependency failures affecting request handling.

Better alert: “p99 latency for critical API endpoint is above threshold for 10 minutes, and request volume is non-trivial.”

Worse alert: “one slow request happened.”

Useful worker alerts

Good worker alert candidates:

• backlog is growing and processing rate is low,
• backlog exists but no successful read/process happened recently (to avoid alerting just because there is no work to do),
• processing failure ratio above threshold,
• repeated failures for one event type or handler,
• dead-letter queue grows,
• oldest message age exceeds threshold.

Better alert: “queue depth > threshold and last successful processing timestamp is older than N minutes.”

Worse alert: “worker pod exists / does not exist” as the only signal.

Useful scheduled job alerts

Good scheduled job alert candidates:

• last success too old,
• latest run failed,
• duration exceeds expected window,
• records processed is unexpectedly zero,
• output freshness violates product expectations.

Better alert: last success timestamp is older than expected interval plus grace period — for example, 26 hours for a daily job.

Before moving on, three more things about alerting that are easy to get wrong:

Include enough context for the alert to be actionable: what failed, since when, current value and threshold, affected service, environment, dashboard and log links, runbook link, recent deploy. It will help to reduce the time between “something is wrong” and “we know where to look first.”

Start with a small number of high-confidence alerts and expand carefully. Alert fatigue is a reliability risk — once people learn that alerts can be ignored, that habit eventually applies to the important ones too.

Not every alert deserves the same channel. Some alerts should wake someone up. These should be urgent, actionable, and tied to real impact. Other signals may still be useful, but they do not require immediate action. They can become warnings, tickets, dashboard annotations, or low-priority notifications. Treating every interesting event as urgent is one of the fastest ways to destroy trust in monitoring. When in doubt, do not start with the notification channel. Start with the expected human response:

Where to start

Trying to build a perfect observability setup from day one is a good way to get stuck. Good news, you don’t have to — it can be rolled out gradually:

Phase 1: Basic service visibility

First, get the basics: can you tell whether the service is up, ready, and failing loudly? Start with:

• health endpoint,
• readiness endpoint,
• structured logging baseline,
• service name / environment / version in logs,
• request ID or correlation ID,
• basic request metrics for APIs,
• basic error logging.

Phase 2: Workload-specific metrics

Next, add signals based on workload type — can you tell whether it’s actually doing useful work?

For APIs:

• request rate,
• error rate,
• latency histogram,
• endpoint labels,
• dependency/cache visibility where useful.

For workers:

• read rate,
• processed outcomes,
• failure counters,
• backlog,
• last successful progress timestamp,
• processing duration.

For scheduled jobs:

• last run timestamp,
• last success timestamp,
• last run status,
• duration,
• records processed.

Phase 3: High-confidence alerts

Now it’s time to add a small number of alerts for real operational risk, so you find out about important failures before users or other teams do:

• API not ready,
• sustained 5xx increase,
• sustained p99 latency increase,
• worker backlog + no progress,
• scheduled job last success too old,
• latest scheduled job failed.

Phase 4: Dashboards and runbooks

Clean up dashboards around investigation paths, not around random metric collections. A useful dashboard has an overview (is the service healthy?), workload-specific behavior, dependencies, errors and log links, deploy markers or version panel.

Every common alert should have a runbook — so every alert has a starting point. A basic one should answer: what this alert means, common causes, where to look first, what dashboards/logs/traces to open, what actions are safe, when to escalate.

Phase 5: Minimal viable tracing and deeper instrumentation

Once metrics, logs, and basic alerting are in place, add tracing where it removes real guesswork — so you can connect behavior across components and boundaries.

Good first targets:

• API request path through DB/cache/external dependencies,
• worker event processing path,
• scheduled job steps,
• expensive DB queries,
• calls to critical third-party services.

Observability will not stop systems from failing. But it changes what failure feels like.

With the right signals, a production incident stops being a panic and becomes an investigation. You know whether the API is slow or broken, whether the worker is doing useful work or just sitting there, whether the scheduled job ran successfully and recently enough.

This is the article I wish I’d had earlier. Not to avoid incidents entirely — that’s not realistic. But to set things up right from the start, sleep through more nights, and spend less time guessing in the dark when something does go wrong.

P.S. “Every move you make, every step you take, I’ll be watching you.” At this point I’m convinced Sting accidentally wrote a song about observability. We’re all a little bit stalkers of our systems. The good kind 😉