From signals to answers: connecting monitoring and ownership so engineers can resolve incidents fast.

Observability at Pennylane

We don’t wait for smoke to find a fire; we make observability part of how we build, onboard, and scale engineering knowledge.

At Pennylane, observability is not just a set of tools we use when production is on fire. It is part of how we build software, how we debug, how we onboard engineers, and how we scale engineering knowledge across the company.

If you join Pennylane, you will quickly notice that we invest a lot in giving engineers context when something goes wrong. The goal is simple: make issues easier to detect, easier to understand, and easier to investigate.

Our observability stack is mostly built around a few tools:

• Sentry for runtime error tracking
• Datadog for everything else: logs, traces, monitors, dashboards, and metrics
• Change Data Capture (CDC) (from the Postgres replication log) streams changes into a dedicated datastore that we can query directly from Metabase. This datastore keeps full history, so we can inspect how a given record evolved over time. We won’t dig into CDC in this article, as it would deserve its own dedicated one.

What matters most to us is not just the tooling itself, but how we connect all of these pieces together, and how observability is tied to ownership inside the engineering organization.

Observability is part of ownership

One thing that matters a lot at Pennylane is ownership.

Our product and technical scopes are owned by squads, and observability is part of that ownership. A squad is not only responsible for shipping features in its area. It is also responsible for understanding how that scope behaves in production, defining the relevant monitors, and making sure failures can be detected early.

In other words, teams are expected to build and maintain the monitoring for the parts of the system they own.

This is an important aspect of our engineering culture. Observability is not something fully centralized and delegated to a platform team. The teams closest to a product or technical scope are expected to know what “healthy” looks like, what can fail, what needs to be measured, and what should alert.

At the same time, we also maintain platform-wide monitoring. We have an on-call rotation with a broader view of the platform. Its role is not to replace squad ownership, but to provide an additional safety net and to watch the health of shared infrastructure and cross-cutting systems.

This ownership model translates directly into how our monitoring is organized. We rely on a lot of Datadog monitors to track the health of the platform, but not all monitors serve the same purpose.

Squad monitoring: the scoped view

This view is owned by the squads closest to the code.

For example, for Rails controllers, we can monitor things such as the number of distinct impacted accounts on a given controller. This is useful because it helps a squad understand not only that a controller is failing, but also how broad the impact is. A team can then detect that something is wrong in the area they own and react accordingly.

We apply similar ideas to background jobs. We want teams to know not only that something is failing, but also who is impacted and how large the blast radius is.

This type of monitoring is strongly aligned with ownership: if a squad owns a controller, a workflow, or a job family, it should also own the signals that tell whether this part of the system is healthy.

Platform monitoring: the global view

Platform monitoring watches the bigger picture and shared systems. It is especially useful as a second line of defense: if something was not captured well enough by squad monitoring, the global view can still detect that something abnormal is happening.

This broader view is also more focused on shared infrastructure and platform-wide health signals, for example:

• Redis CPU usage
• Redis memory levels
• Postgres replication lag
• broader platform error signals

One of the most useful feedback loops we get from platform monitoring is detecting where squad monitoring was missing or set at the wrong threshold. When a global alert fires before any squad-level signal does, we treat it as a prompt to push a more precise monitor back down to the squad that owns that code path. Over time, this is what keeps squad-level coverage accurate.

This combination helps us balance local ownership with global safety.

Maximizing production context

Strong monitoring tells us that something is wrong. To go further and understand why, we lean heavily on the context we keep around every request and job.

One of the main things we rely on at Pennylane is logs.

In Datadog, we retain 100% of our logs, which means we can reconstruct in detail what happened during a Rails controller execution or inside a Sidekiq job. In the event of a failure, this is essential. Instead of only having a high-level error message, we can often follow the full execution path and recover enough context to understand the issue.

We do not only rely on the default logs produced by the platform and libraries. We also enrich logs directly from our Ruby on Rails codebase.

For example, we often add custom tags such as:

• an internal tenant reference (not a customer-identifying value)
• domain-specific context
• business-related metadata useful for debugging
• a request id, propagated as parent_request_id into Sidekiq jobs and into the SQL comment prefix, so a faulty request can be traced from the database insights back to its origin

A concrete example: we can tag logs with an internal tenant reference and then enrich it with additional business context, such as whether the underlying account is large or small. We do this using Datadog Reference Tables, which act like lookup tables on the Datadog side and let us join extra metadata to our telemetry. Reference tables are kept up to date automatically through S3 syncs, so the business metadata we join against (e.g. the large-SME segmentation) stays fresh without manual maintenance.

This is very useful because raw technical events are often not enough. Being able to say “this issue only affects large accounts” or “this is happening only on a certain category of customers” changes the speed and quality of the investigation a lot.

Traces are sampled, but errors are kept

Logs give us breadth across a request. Traces give us its structure across services.

We also use Datadog APM tracing, but with a relatively low sampling rate. We do not keep every trace, because that would be too costly and often unnecessary.

What we do keep systematically are error traces and long-running traces.

This gives us a good tradeoff:

• low enough cost on the happy path
• high context when something actually breaks

And again, we do not stop at the default instrumentation. We enrich spans with extra contextual information when it is useful.

For example, when we hit a SQL timeout, we attach the parameterized query to the relevant span — literal values are stripped, so nothing customer-identifying ever lands in our Datadog telemetry. From there, an internal, access-controlled tool lets engineers replay the query plan against a production-grade replica, with realistic parameters provided rather than through the span itself. The two paths are intentionally separated: telemetry stays safe to read broadly, while the replay capability lives behind stricter controls. In practice, an engineer can go from “we saw a timeout” to “we understand which query plan is problematic” much faster.

This kind of enrichment is one of the themes of our observability approach: the default signals are helpful, but the highest leverage comes from adding the context that is specific to your business and your architecture.

Dashboards are part of the daily workflow

When the same context shows up over and over, it usually deserves a dashboard.

We use lots of dashboards as a way to monitor systems and understand trends over time.

Depending on the use case, we build dashboards on top of either a metric or a log. There is always a tradeoff.

Metrics are great because:

• retention is longer
• granularity is often better for historical analysis
• they are well suited for dashboards and trend monitoring

Logs are great because:

• cardinality is much higher
• they preserve more detailed event-level context
• they allow much richer filtering when you need to debug something specific
• you can work with them right away — metrics often need to be defined and instrumented up-front, so building a dashboard from a metric usually takes more time and effort than building one from logs

So in practice, the question is not “metrics or logs?” but “what kind of signal do we need for this problem?”

Sometimes a metric is clearly the right abstraction. Sometimes only logs will give enough detail. The craft of observability is knowing which one to use.

Sentry is our runtime error entry point

Dashboards help us watch trends. For individual exceptions in production, we start somewhere else.

For runtime exceptions, Sentry plays a central role.

We try to look at all new exceptions. Every new Sentry issue is pushed to Slack, so engineers are notified quickly. This creates a tight feedback loop: when something new appears in production, the team can react early.

Notifications are routed deliberately:

• each squad has its own dedicated Sentry / alerting Slack channel, so alerts land in front of the team that owns the affected scope
• a global error channel acts as a cross-cutting safety net for issues that don’t have a clear squad owner yet, or that need broader visibility
• detection is only half the loop — we also track resolution through reporting and SLOs, so we know how a squad is trending over time, not just whether the latest alert was acknowledged

What is useful is that Sentry is not isolated from the rest of the stack.

Correlate Sentry with Datadog

From a Sentry issue, we can correlate to the related Datadog logs, and from those logs we can usually reach the associated trace. This means that a single runtime exception can give access to a surprisingly rich investigation chain:

• the exception itself
• the deployed commit
• the Sentry context
• the related logs
• the trace data
• any custom metadata we added ourselves

This saves a lot of time in practice. Instead of jumping manually between disconnected tools, engineers can navigate through connected signals.

Context is the king

That investigation chain only works because every layer carries enough context. A recurring theme in our setup is that out-of-the-box observability is good, but not enough.

In Sentry, we add manual context on top of what Sentry collects automatically.

In Datadog logs, we rely both on standard fields and on custom enrichment from our application.

In traces, we combine standard instrumentation with extra business or technical context when it improves debuggability.

This might sound like a detail, but it changes the experience a lot. A plain stack trace is useful. A stack trace tied to the exact tenant, the exact request context, the exact commit, the exact SQL query, and the exact trace is much more useful.

We invest in AI-assisted investigation

By enriching our telemetry with consistent metadata, we have been able to layer AI directly on top of our stack to assist with the first steps of any triage.

All engineers have access to Claude, and we built an internal skill connected to:

• Sentry MCP
• Datadog MCP
• our internal command-line tooling

These are used as pre-investigation tools.

When a new Sentry issue appears, an engineer does not always know where to start. In many cases, just launching an AI-assisted investigation is cheap and useful. The goal is not to fully automate debugging, but to gather the available context quickly in one place and help engineers get to a first understanding faster.

This is particularly useful for:

• triaging new issues
• reducing the cost of the first investigation step
• helping newer engineers navigate unfamiliar code and tooling
• making investigation workflows more consistent

For experienced engineers, it saves time. For newer engineers, it can also be a way to learn how incidents are explored at Pennylane.

Observability is part of onboarding

That last point on newer engineers is not an accident. Observability is something we deliberately introduce from day one.

We have a synchronous one-hour training for every new engineer joining the tech team.

This training is part of our onboarding weeks and introduces:

• Datadog
• Sentry
• how do we use them at Pennylane
• what we have built on top of them
• how engineers are expected to investigate production issues in practice

This is important for us because observability is not an advanced topic reserved for a few senior engineers. It is part of the normal developer experience.

When someone joins the company, we want them to quickly understand not just the codebase, but also how to operate it.

Observability is shared knowledge

Onboarding only goes so far. To keep operational knowledge alive after the first weeks, we also need a way for it to keep circulating.

One thing we care about is that observability should not stay in the hands of a few experts.

We have an Observability Fellowship, which is a group of people available to:

• share their knowledge
• answer questions
• help others improve their Datadog and Sentry usage
• spread good practices across the engineering organization

This matters because observability tooling can become very powerful but also quite unevenly used if knowledge does not circulate. The fellowship is one of the ways we try to make sure the organization keeps leveling up collectively.

What we are really trying to optimize

If we had to summarize our philosophy, it would probably be this:

We want engineers to have the shortest possible path from “something is wrong” to “I understand what is happening.”

That requires:

• good signals
• enough retention
• strong correlation between tools
• custom context added from the application
• clear ownership of monitoring
• a global safety net at the platform level
• operational habits around looking at issues
• shared knowledge across the team

Observability at Pennylane is therefore not only a technical stack. It is a combination of tooling, conventions, ownership, training, and internal culture.

And for us, that is probably the key point: observability is not just there to watch systems. It is there to help engineers make sense of reality quickly, and to make sure every team can own its production scope with confidence.