← Back to list

My Observability trace went dark at thread spawn. Here’s what fixed It.

When you spawn a new thread in Spring Boot, your Dynatrace trace doesn’t follow. Here’s why — and the one TaskDecorator configuration…

Sriram Mahalingam in Stackademic · 2026-06-11 12:00 · 0 claps · 6.6 min read paywalled
#spring-boot #dynatrace #observability #opentelemetry #platform-engineering
Open on Medium ↗
Wiki topics: SOC · Sociology & Politics

SPRING BOOT & OBSERVABILITY

My Observability trace went dark at thread spawn. Here’s what fixed It.

When you spawn a new thread in Spring Boot,

your Dynatrace trace doesn’t follow. Here’s why — and the one TaskDecorator configuration that fixes it permanently

⭐️ (Not a Medium member yet? Read the rest of this story paywall-free here.)

Photo by Nicolas Gonzalez on Unsplash

Photo by Nicolas Gonzalez on Unsplash

GCP Pub/Sub, a 30-second backend process, a new thread, and a Dynatrace trace that just… stopped. The fix was one Spring feature most developers walk past every day.

The trace looked fine.

  • GCP Pub/Sub received the message, the backend picked it up, the initial processing started — everything green in Dynatrace.
  • Then at exactly the point where I spawned a new thread to handle the long-running work, the trace just stopped.
  • Not an error. Not a timeout. Just gone.
  • The thread was running, the work was completing, the response was 200. But from Dynatrace’s perspective, the flow had vanished into a void.

I stared at that gap for longer than I’d like to admit.

This is the story of what caused it and the Spring feature that fixed it in a way that felt almost embarrassingly elegant once I understood what it was doing.

The Setup — Why a New Thread in the First Place

The flow was straightforward on paper. A GCP Pub/Sub subscription pushes messages to a Spring Boot 3.5 backend. The backend processes each message and acknowledges it. Standard pattern, works fine for quick operations.

The problem was that our processing occasionally took more than 30 seconds. GCP Pub/Sub has an acknowledgement deadline — if your backend doesn’t acknowledge within the configured window, Pub/Sub assumes the message was not processed and redelivers it. With processing times occasionally stretching past 30 seconds, we were getting redeliveries — the same message processed twice, downstream effects that nobody wanted.

The obvious fix was to decouple the acknowledgement from the processing. Acknowledge immediately, process asynchronously. Return 200 to Pub/Sub right away, handle the actual work in a separate thread.

// What we did first — the naive approach
@Service
public class MessageProcessor {

    @PubSubMessageHandler
    public void handleMessage(PubsubMessage message, AckReplyConsumer consumer) {

        // Acknowledge immediately — don't let Pub/Sub redeliver
        consumer.ack();

        // Spawn a new thread for the long-running work
        new Thread(() -> {
            processLongRunningWork(message);
        }).start();
    }

    private void processLongRunningWork(PubsubMessage message) {
        // 30+ seconds of processing
        // Database operations, external API calls, transformations
    }
}

This worked. Messages were processed. Redeliveries stopped. The 200 responses came back within milliseconds.

And then I opened Dynatrace.

The Trace That Stopped at Thread Spawn

The distributed trace told a clear story up to a point:

Pub/Sub message received
        ↓
handleMessage() entered — trace ID: abc-123
        ↓
consumer.ack() called
        ↓
new Thread().start()
        ↓
[TRACE ENDS]

After “new Thread().start()”, nothing. The trace for the actual processing — the database calls, the external API interactions, the transformations that took 30 seconds — was completely absent from Dynatrace. Not erroring, not slow, not visible at all.

The work was happening. I could see it in the database. I could see the downstream effects. But the observability layer had no record of any of it.

The reason is straightforward once you understand how distributed tracing works.

  • Dynatrace — like any OpenTelemetry-compatible APM tool — propagates trace context through a thread-local variable.
  • When your code runs on a thread, it inherits the trace context of that thread.
  • When you spawn a plain new thread with new Thread(() -> {…}).start(), that new thread has no parent thread context.
  • It starts fresh, with no trace ID, no span context, no connection to the originating request.

The child thread was invisible to Dynatrace because it was born without an identity.

What We Tried First — And Why It Wasn’t Enough

The first instinct was to use Spring’s Async annotation with a configured ThreadPoolTaskExecutor. This is the standard Spring pattern for async execution and it felt like the right direction.

// First attempt — @Async with executor
@Configuration
@EnableAsync
public class AsyncConfig {

    @Bean(name = "pubSubExecutor")
    public Executor pubSubExecutor() {
        ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
        executor.setCorePoolSize(10);
        executor.setMaxPoolSize(20);
        executor.setQueueCapacity(100);
        executor.setThreadNamePrefix("pubsub-async-");
        executor.initialize();
        return executor;
    }
}

@Service
public class MessageProcessor {

    @Async("pubSubExecutor")
    public CompletableFuture<Void> processAsync(PubsubMessage message) {
        processLongRunningWork(message);
        return CompletableFuture.completedFuture(null);
    }
}
  • The thread pool was better than raw thread spawning — managed, bounded, named. But the Dynatrace problem persisted.
  • The trace still went dark at the async boundary because Async alone does not propagate the tracing context from the calling thread into the executing thread.

The missing piece was context propagation. Spring’s executor needed to know about the observability context — the trace ID, the span information — and carry it across the thread boundary automatically.

The Fix — TaskDecorator and Context Propagation

Spring’s ThreadPoolTaskExecutor has a feature called TaskDecorator — a hook that wraps each task before it executes on a thread pool thread. The decorator runs in the context of the submitting thread (where the trace context exists) and can capture that context, then restore it on the executing thread.

@Configuration
@EnableAsync
public class AsyncConfig {

    @Bean(name = "pubSubExecutor")
    public Executor pubSubExecutor() {
        ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
        executor.setCorePoolSize(10);
        executor.setMaxPoolSize(20);
        executor.setQueueCapacity(100);
        executor.setThreadNamePrefix("pubsub-async-");

        // This is the key — wrap each task with context propagation
        executor.setTaskDecorator(new ContextPropagatingTaskDecorator());

        executor.initialize();
        return executor;
    }
}

The ContextPropagatingTaskDecorator is where the actual work happens:

public class ContextPropagatingTaskDecorator implements TaskDecorator {

    @Override
    public Runnable decorate(Runnable runnable) {
        // Capture the current context on the SUBMITTING thread
        // This is where the Dynatrace trace ID lives
        Context currentContext = Context.current();

        // Return a new Runnable that restores the context
        // on the EXECUTING thread before running the actual work
        return () -> {
            try (Scope scope = currentContext.makeCurrent()) {
                // The executing thread now has the same trace context
                // as the submitting thread
                // Dynatrace can see the full flow
                runnable.run();
            }
        };
    }
}
  • What Context.current() captures is the full OpenTelemetry context — trace ID, span ID, baggage, all of it.
  • When makeCurrent() is called on the executing thread, it restores that context into the thread-local storage that Dynatrace reads from.
  • The child thread is now connected to the parent trace.

The complete updated configuration:

@Service
public class MessageProcessor {

    private final AsyncMessageService asyncMessageService;

    @PubSubMessageHandler
    public void handleMessage(PubsubMessage message, AckReplyConsumer consumer) {
        // Acknowledge immediately — Pub/Sub won't redeliver
        consumer.ack();

        // Submit to context-aware executor
        // The trace context follows the work into the thread pool
        asyncMessageService.processAsync(message);
    }
}

@Service
public class AsyncMessageService {

    @Async("pubSubExecutor")
    public CompletableFuture<Void> processAsync(PubsubMessage message) {
        // This thread now carries the full trace context
        // Every operation inside here is visible in Dynatrace
        processLongRunningWork(message);
        return CompletableFuture.completedFuture(null);
    }

    private void processLongRunningWork(PubsubMessage message) {
        // 30+ seconds of processing
        // All of this is now traced end-to-end in Dynatrace
    }
}

What Dynatrace Showed After the Fix

After deploying with the TaskDecorator in place, the trace told the complete story:

Pub/Sub message received
        ↓
handleMessage() entered — trace ID: abc-123
        ↓
consumer.ack() called
        ↓
Thread pool task submitted — span continues
        ↓
processLongRunningWork() entered — same trace ID: abc-123
        ↓
Database operation — traced ✅
        ↓
External API call — traced ✅
        ↓
Transformation complete — traced ✅
        ↓
CompletableFuture completed — trace closed ✅
  • The entire 30-second flow was now visible as a single continuous trace.
  • The thread boundary that had previously been an observability black hole was transparent.

When something went wrong in the long-running processing — and things do go wrong

  • we could see exactly where in the trace it happened,
  • how long each operation took, and what the state was at every point.

The gap in Dynatrace that had been making production debugging essentially impossible was gone.

Why This Matters Beyond Dynatrace

  • The trace propagation problem is not specific to Dynatrace. Any OpenTelemetry-compatible APM tool — Datadog, New Relic, Jaeger, Zipkin — relies on the same thread-local context propagation mechanism.
  • Spawning a plain thread or using a non-decorated executor breaks that propagation in every one of them.
  • The same issue affects MDC (Mapped Diagnostic Context) in your logging — if you use MDC to carry a correlation ID through your logs, that correlation ID will also be lost at a plain thread boundary.
  • The TaskDecorator pattern fixes MDC propagation too, with a small addition:
public class ContextPropagatingTaskDecorator implements TaskDecorator {

    @Override
    public Runnable decorate(Runnable runnable) {
        // Capture both OpenTelemetry context AND MDC
        Context currentContext = Context.current();
        Map<String, String> mdcContext = MDC.getCopyOfContextMap();

        return () -> {
            // Restore MDC on the executing thread
            if (mdcContext != null) {
                MDC.setContextMap(mdcContext);
            }

            try (Scope scope = currentContext.makeCurrent()) {
                runnable.run();
            } finally {
                // Clean up MDC after task completes
                MDC.clear();
            }
        };
    }
}

With this version, both your Dynatrace traces and your correlation IDs in logs propagate correctly across thread boundaries. A single TaskDecorator implementation makes your entire async execution context-aware.

The Broader Lesson

There is a category of bugs in distributed systems that are not bugs in the traditional sense — the code runs correctly, the results are right, the response codes are appropriate. But the observability layer goes blind at a specific point and you lose the ability to understand what actually happened.

Thread context loss is one of those. It does not break your application. It breaks your ability to see your application.

In a production system where something eventually goes wrong — and something always eventually goes wrong — that visibility is not a nice-to-have. It is the difference between a 10-minute diagnosis and a 3-hour war room.

The TaskDecorator pattern is one of those Spring features that sits quietly in the documentation, easy to miss, solving a problem you do not know you have until the first time Dynatrace shows you a trace that just stops at a thread boundary and you spend an afternoon wondering where your flow went.

Now you know where it went. And you know how to make it stay visible.

If you are running Spring Boot on GCP with Pub/Sub and Dynatrace — or any combination of async processing and APM tooling — the TaskDecorator configuration above is worth adding before you need it rather than after.


메타데이터
post_id
bd4e889bdfd6
slug
my-observability-trace-went-dark-at-thread-spawn-heres-what-fixed-it-bd4e889bdfd6
url
https://blog.stackademic.com/my-observability-trace-went-dark-at-thread-spawn-heres-what-fixed-it-bd4e889bdfd6
canonical_url
https://blog.stackademic.com/my-observability-trace-went-dark-at-thread-spawn-heres-what-fixed-it-bd4e889bdfd6
author_url
https://medium.com/@sriram.chennai64
status
ok
fetched_at
2026-07-10 21:34:04