Measuring What Actually Matters: SLI-Based Monitoring for Payment Services

There’s a moment every on-call engineer dreads. It’s 2 AM, an alert fires, and you’re staring at a dashboard full of green — CPU fine, memory fine, pods running — yet payments are silently failing. Users are getting errors. The business is bleeding. And your monitoring told you everything was okay.

That gap between “infra is healthy” and “users are happy” is exactly the problem that Service Level Indicators (SLIs) are designed to close.

This article walks through how to design and implement SLI-based monitoring specifically for a payments service — choosing the right metric, pulling it from the right place, and wiring it into Prometheus in a way that actually pages you when something real goes wrong.

SLI, SLO, SLA — Quick Recap

These three terms get thrown around a lot, often interchangeably, which causes confusion.

SLI (Service Level Indicator) is the actual measured metric. It’s a ratio or a number that says something about how well your system is serving users. Key word: measured, not estimated, not assumed.

SLO (Service Level Objective) is the target you set for that indicator. “95% of payment transactions should succeed” is an SLO. It’s an internal commitment, not a legal document.

SLA (Service Level Agreement) is the contractual version — the thing you sign with your customers that usually comes with financial penalties if you miss it.

The relationship flows in one direction: your SLI tells you where you are, your SLO tells you where you should be, and your SLA tells you what happens if your SLO fails too often.

Most teams obsess over SLAs and forget to build proper SLIs first. Get the SLI right, and the rest becomes much easier to reason about.

Picking the Right SLI for Payments

Not every metric makes a good SLI. A good SLI has three properties:

1. It directly reflects user experience
2. It’s explainable to a non-technical stakeholder in one sentence
3. It’s actually measurable without heroic effort

For a payment service, there’s one question that cuts through everything:

From the customer’s perspective, when is a payment successful?

The answer is simple: when the payment goes through. When the money moves. When the bank says yes.

That maps cleanly to a ratio:

Payment SLI = Successful Payments / Total Payments

If you have 1,000 payment attempts and 980 succeed, your SLI is 98%. Simple to compute, simple to explain, and directly tied to what your users care about.

Contrast this with metrics like “API response time under 200ms” or “pod restart count.” Those might be useful signals, but they don’t tell you whether payments are actually completing. A fast failure is still a failure.

Why the Database Is the Right Source of Truth

Now comes the implementation question: where do you actually measure this?

You could instrument your application code. Add counters to your payment processing logic. Log outcomes. Build a pipeline to aggregate those logs. That works, but it has a few problems:

• It couples your monitoring to your app restarts. If the pod crashes mid-transaction, you might lose the outcome.
• It gets complicated with legacy flows or synchronous processing paths where the app doesn’t see the final state.
• You need code changes every time the payment flow evolves.

The database doesn’t have these problems. By the time a payment reaches its terminal state — whether that’s success, failure, or rejection — that state is written to the database. It’s durable. It doesn’t disappear when a pod restarts. It covers every path, including the legacy ones.

For a payments service, the DB is the ground truth. So that’s where we measure.

Exposing DB State as Prometheus Metrics

Most observability stacks are built around Prometheus. The challenge is that Prometheus scrapes HTTP endpoints — your database doesn’t expose one. This is where a DB exporter comes in.

Tools like the oracle-db-exporter or similar project-specific exporters let you write SQL queries and expose the results as Prometheus metrics. No application code changes, no new dependencies in your service — just a sidecar or standalone process that talks to the DB and speaks Prometheus.

The setup is straightforward:

• Write a SQL query that counts payment records grouped by their final status
• The exporter runs that query on a configurable interval
• Results get exposed as a gauge metric with labels for payment type and status

For example, a metric called payment_query_record_count with labels like status=SUCCESS and status=FAILURE gives Prometheus everything it needs.

Complete Installation Reference— https://oracle.github.io/oracle-db-appdev-monitoring/docs/intro/

Designing the Query

The tricky part is translating database states into clean success/failure semantics. Payment systems are messy — there are intermediate states, partial acknowledgments, and edge cases around bank reference numbers.

A reasonable mapping looks something like this:

PROCESSED is clean — the payment cleared. ACCEPTED needs a secondary check: if the bank returned a reference number, the payment made it through; if not, something went wrong downstream. REJECTED is always a failure.

This logic lives entirely in SQL. The query groups records by their resolved final status, hands the counts to the exporter, and Prometheus does the rest. Business logic stays explicit and reviewable, not buried in application code.

Prometheus Recording Rules and Alerting

Once the raw counts are flowing into Prometheus, you need a recording rule to compute the ratio:

groups:
  - name: payments.rules
    interval: 30s
    rules:
      - record: payments:success:30m
        expr: |
          sum(increase(payment_query_record_count{status="SUCCESS"}[30m]))

      - record: payments:total:30m
        expr: |
          sum(increase(payment_query_record_count[30m]))

      - record: payments:success_ratio
        expr: |
          payments:success:30m / payments:total:30m

And the alert:

- alert: PaymentSuccessRateLow
  expr: payments:success_ratio < 0.99
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "Payment success rate dropped below 99%"
    description: "Current rate: {{ $value | humanizePercentage }}"

A 30-minute window balances sensitivity and noise. Too short and you’ll get alert storms from brief blips; too long and you’re slow to catch a real degradation. Tune based on your traffic volume — high-throughput systems can afford shorter windows.

The for: 5m in the alert adds a brief confirmation period so a single slow scrape doesn't wake someone up at 3 AM.

What You Get Out of This

The real payoff isn’t just the alert. It’s the cultural shift that comes with having a clear SLI.

When you can say “our payment success rate is 99.2% over the last 30 days” in a team meeting, conversations change. Engineering decisions get grounded in something measurable. Incident postmortems become more focused — instead of debating whether something was “a real outage,” you look at the SLI graph and the answer is right there.

And when users report that payments aren’t working, you stop saying “nothing looks wrong on our end” and start looking at the right metric from the start.

Takeaways

• SLIs should reflect user experience, not infrastructure health
• For payments, the success ratio is the clearest possible SLI
• The database is a more reliable source of truth than application-level instrumentation
• A DB exporter lets you expose SQL query results as Prometheus metrics with zero application changes
• Status-to-outcome mapping belongs in the query, not in the alert rule
• Recording rules make ratio computation clean and composable

If you’ve been relying on RED metrics or infra dashboards to tell you whether payments are working — they’re useful, but they’re not enough. Build the SLI first.