Surviving a Telemetry Crisis: How We Tamed a Data Tsunami with Grafana’s LGTM Stack

Background & Motivation

At **Super.Money**, our system’s reliability and performance are the bedrock of our business. As a platform that handles millions of daily UPI transactions, our engineering teams need to see what’s happening in real-time. Even a short outage or slow performance can lead to lost user trust and a direct financial hit.

To get this level of visibility, we had to move past our old, disconnected monitoring tools. We were managing separate systems for logs, metrics, and traces. It was inefficient and made it hard to quickly find the root cause of problems. Our big move was to switch to the open-source **LGTM stack** — with Loki for scalable log aggregation, Grafana for unified visualization and alerting, and Mimir for metrics storage. This new, self-hosted setup gave us the control and flexibility we needed to operate at a massive scale without breaking the bank.

But we quickly learned that even the best tools are only useful if they’re set up correctly. A recent incident showed us that in the fast-paced world of fintech, an unoptimized system can turn from a great asset into a costly problem.

The Problem: A Tsunami of Telemetry

The challenge started innocently enough. After we increased our log ingestion limits, our observability platform was hit by a huge and unexpected surge of data. The problem came from a few of our core services, and the data volumes they were generating were far beyond anything we had prepared for.

Before this, their log volumes were easy to manage. But within a few days, the log volume from one key service, responsible for our bill payments gateway, went from a few gigabytes to tens of terabytes a day, representing a growth of over 1,500%. Another service saw its log volume jump by over 10,000% in just a few days.

This flood of data had an immediate impact. Our Grafana dashboards became sluggish and our queries started timing out, making it impossible to get a clear picture of what was happening. The massive increase in data also drove up our cloud-related costs dramatically, with our logging expenses alone rising by over 200%. Most importantly, our engineers were struggling to find anything useful. Debugging became a monumental task, with critical logs getting buried in a mountain of noise.

The Investigation: Tracing the Root Cause

Our engineering team immediately launched an investigation to figure out what was happening. This was a methodical process, starting from the symptoms and working our way to the source.

1. Identifying the Data Culprit

Our first step was to find out what was causing all this data. We quickly found a few unusually large log entries. We discovered that our applications were logging the full content of entire API responses, some of which were hundreds of kilobytes each. This was a bad practice. These large pieces of data didn’t help with debugging and were taking up a huge amount of our resources. The logs were primarily from API calls that returned large data payloads, like responses for a user’s transaction history or a list of available payment plans.

2. The Fluent Bit Anomaly Explained

Once we found the data, we looked at our log forwarder, **Fluent Bit**. We realized the problem wasn’t just the large logs — it was how our system was trying to handle them. Fluent Bit processes logs in a small internal memory space. When a single log line was too big for that space, the system’s default setting would kick in. Instead of simply dropping the line, it would keep trying to send it. This created a continuous cycle of retries, causing the same data to be sent again and again, leading to the exponential growth we saw. This behavior, while intended for robustness, became a major vulnerability at our scale.

The Corrective Actions: A Two-Phase Solution

Our solution was a direct attack on the root cause. We knew the problem wasn’t the logs themselves, but our system’s reaction to them.

Phase 1: Initial Buffer Optimization (The Incomplete Fix)

Our first attempt at a solution was to optimize Fluent Bit’s internal buffer sizes. We adjusted parameters such as Buffer_Chunk_size and Buffer_Max_size. The hypothesis was that by giving the agent larger buffers, we could accommodate the big log lines and prevent the retry loop. While this did help with memory management, it was not the complete solution. It merely raised the bar for what would be considered a "long line." An even larger log entry would still trigger the same destructive behavior. This step was a necessary part of our debugging process, but it showed us that the root cause was not a lack of buffer size, but the agent's default behavior when faced with an overflow.

Phase 2: The Critical Skip_Long_Lines Fix

The breakthrough came with the discovery and implementation of a single, powerful configuration parameter: **Skip_Long_Lines. By default, this parameter is set to false, allowing the retry behavior we had observed. We changed the setting to `true`. This single, seemingly small change instructed Fluent Bit to gracefully and silently discard any log line that exceeded its internal buffer capacity. It was a direct instruction to the agent to stop** trying to do the impossible. This simple, elegant solution directly addressed the core issue by preventing the wasteful retry loop and the exponential data multiplication it caused.

The Impact: A Return to Excellence

The results were immediate and transformative. Our log volumes from the problematic services plummeted. We saw a reduction of more than 99% in log volume from one service and over 99.5% from another, leading to a log rate that was over 600x lower than during the peak of the incident.

This chart illustrates the sharp spike in log volume immediately following our initial ingestion rate increase, followed by an immediate and dramatic drop to a sustainable baseline on the day we deployed our solution.

This chart visually compares the days leading up to and after the fix, clearly showing the chaotic, high-volume periods giving way to a new, flattened, and sustainable log rate, proving the effectiveness of our solution.

This dramatic drop had a huge impact on our company:

• Significant Cost Savings: The data reduction translated directly into substantial savings on our data and storage costs, bringing our budget back in line. We saw our overall logging expenses drop by over 50%.
• Restored Efficiency: Our LGTM platform is now fast and reliable again. Dashboards load instantly, and queries that used to fail now work in milliseconds. This has directly led to a major improvement in how fast we can fix issues.
• Trust in Our Tools: Our team’s confidence in our observability tools is back. By getting rid of the noise, we can now easily find the data we need to make quick decisions.

You can see the results for yourself in these graphs. One shows a sharp spike in log volume right after we increased our limits, followed by a dramatic drop to a stable baseline on the day we deployed our fix. Another chart visually compares the old, chaotic high-volume days with the new, flat, and sustainable log rate, proving our solution worked.

By turning a crisis into a major learning moment, we not only fixed a critical problem but also matured our engineering philosophy. We proved that by using the right open-source tools and having a smart engineering approach, we can solve big scalability challenges and build a more robust, cost-effective system.

Lessons Learned & Our Future Strategy

This incident was a powerful reminder that observability isn’t just about the tools you use; it’s about the discipline and practices you follow. We have learned a few key lessons that will guide us moving forward:

• Logging is a Conscious Choice: We now treat logging as a serious part of our development process. We’re asking ourselves if every log is really needed. We’ve learned that full API responses and large data payloads usually don’t belong in production logs.
• Embrace Proactive Monitoring: We’ve set up automated alerts for our log ingestion rates. This helps us find unusual spikes early, so we can prevent a similar situation from ever happening again.
• Integrate Cost Awareness: Our newfound understanding of data volume has made us more aware of the financial implications of our logging practices. We now regularly monitor our costs alongside our system metrics to ensure our observability solution remains a good investment.
• Use Tools to Their Full Potential: We are exploring more advanced filtering using tools like Fluent Bit’s grep and lua plugins. This will give us more control to drop specific noisy logs or add valuable context to the data before we store it.

Our journey from a data firehose to a clear, actionable dashboard is a testament to the power of a unified, open-source stack and an engineering team committed to getting better. The LGTM stack, now properly tuned, will continue to be a cornerstone of our operational excellence as we scale to new heights.