Slashing Cloud Costs by Over 80%: How Tata Digital Optimized Storage and Compute at Scale

As Tata Digital scaled its petabyte-level data platform across brands and categories, storage and compute costs began to spike dramatically. The primary reason was that during the initial phase of platform growth, our focus was on speed of execution and rapid business onboarding. To meet aggressive GTM timelines and support anticipated user volumes, infrastructure decisions are leaned toward scale and agility rather than optimization.

While engineering delivered swiftly to support business timelines, deep cost-efficiency planning was deferred to the post-launch stabilization phase. This led to high-spec clusters and real-time pipelines being set up even where they weren’t essential. Brand-specific customizations also led to fragmented pipelines and non-standardized configurations. These were reasonable trade-offs at the time.

As the platform matured, we made a deliberate pivot to cost governance, architectural streamlining, and long-term sustainability. After investigating root causes and aligning both platform usage and business needs, our data engineering team achieved an 80% cost reduction without sacrificing performance or stakeholder experience.

In this blog, we share how we diagnosed, optimized, and architected a sustainable cost-efficient ecosystem across Azure ADLS, Databricks, and VM infrastructure.

Problem Statement

As the data platform expanded, the cost of maintaining it grew disproportionately high. To understand and control this surge, we first needed to identify where and why the costs were accumulating and whether these were genuine system costs or if we were bleeding money due to misconfigurations or inefficient infrastructure choices.

We conducted a detailed cost diagnostics and monitoring exercise across all services and identified three major components contributing most significantly to the monthly cloud spend:

· Databricks processing costs

· ADLS storage costs

· Virtual machine (VM) costs

Combined, these three components contributed to a substantial portion of the monthly cloud spend.

This forced us to rethink our architecture, redesign capacity models, and implement systemic changes to how we planned, deployed, and governed workloads.

Issue Identification and Diagnosis

Our diagnostic phase revealed deeper systemic issues. High Databricks cost wasn’t an isolated expense, it also drove up VM and storage costs. More Databricks activity meant more VMs were spun up and more frequent read/write operations on storage (ADLS).

We discovered:

· Default Settings Misuse: Features like Photon Engine were often unknowingly enabled because they were set as default in Databricks clusters. Photon clusters are approximately 2.5 times more expensive than standard all-purpose clusters. While Photon is designed for high-performance, short-duration processing, it is generally not required for typical daily load jobs, making it default usage and unnecessary cost driver in most cases.

· High-frequency Streaming: Several pipelines were initially configured for real-time streaming based on business expectations for near-instant data availability. However, during our cost diagnostic phase, we revisited these assumptions with stakeholders and evaluated whether such real-time capabilities were genuinely required. In many cases, we discovered that daily or batch processing would sufficiently meet business needs. This insight revealed that high-frequency streaming was a key driver of unnecessary ADLS I/O and compute costs.

· Oversized Clusters: In the pursuit of identifying high-cost workloads, we revisited Databricks cluster configurations and discovered underutilized compute capacity. Clusters were often provisioned generously to safeguard performance under GTM pressure, which led to underutilization in some cases

· Job Fragmentation: Due to the pressure of meeting aggressive GTM timelines, multiple jobs were developed independently for specific use cases. In many cases, these jobs could have been stitched together into unified pipelines or reused to serve multiple needs. This fragmentation led to under-utilized clusters and scheduling inefficiencies that compounded over time.

The Fix — Optimisation Strategy

It became clear that heavy or unoptimised data processing was the primary cost driver and needed to be addressed first. Optimising Databricks usage through better governance, capacity alignment, and platform-aware engineering was the most impactful starting point to manage our overall cloud footprint.

Databricks Processing Optimization

To bring Databricks costs under control, we took a structured and surgical approach. We started by aligning job refresh frequencies with actual business requirements, ensuring that data was processed only when needed. Capacity planning was reevaluated job by job to ensure resource provisioning was right-sized from the beginning, not scaled by default.

We introduced strict cluster governance using Databricks cluster policies and role-based access controls automating guardrails to prevent misuse of high-end configurations. Our mindset shifted toward always choosing the least expensive path first, scaling up only if justified by real-time business needs. These principles formed the foundation of our Databricks optimization strategy.

· Optimized Scheduling: All jobs were re-evaluated and aligned with actual downstream data consumption patterns to avoid unnecessary compute usage.

· Cluster Efficiency: We conducted audits to address underutilized clusters and merged fragmented jobs to reduce the number of compute instances.

· Configuration Cleanup: Removed default-enabled premium features like Photon Engine where not essential and enforced automation-led governance using Databricks cluster policies.

VM Optimization

As we streamlined Databricks usage and eliminated inefficiencies in job scheduling and processing, a significant byproduct was the reduction in VM utilization. With fewer and better-optimized workloads running, the compute footprint across virtual machines naturally declined.

One of the most impactful infrastructure changes we implemented was the Intel to AMD transition. By shifting from high-cost Intel VMs to cost-effective AMD machines, we were able to maintain performance while significantly reducing compute expenses.

Fun fact: During our diagnostics, we observed that for every ₹1 spent on VMs, we were spending nearly ₹2.5 on Databricks processing (DBUs), a revealing ratio that reinforced the need to address processing inefficiencies first.

Storage Optimization

Storage costs were significantly impacted by unnecessary data replication and lack of lifecycle management. One of the primary contributors was the widespread use of Geo-Redundant Storage (GRS). While GRS offers higher fault tolerance, it wasn’t necessary for every layer of our data warehouse. We kept only the final gold layer GRS enabled, while lower layers such as Raw and Silver could be safely maintained with LRS to reduce unnecessary duplication across geographies.

Additionally, we identified that systems like Databricks and EventHub, which temporarily use Azure Data Lake Gen2 for intermediate storage, also had GRS enabled by default. This configuration didn’t add business value and contributed directly to inflated storage bills. We reclassified these storage accounts and transitioned them to Locally Redundant Storage (LRS), appropriate for their short-term usage nature.

We also addressed inefficiencies tied to Delta Lake versioning. Delta tables retain multiple versions of data, including deleted or overwritten files, unless explicitly cleaned. This behavior, when unmanaged, led to substantial storage bloat. A disciplined vacuuming schedule was implemented across critical Delta tables to free up space and reduce accumulation.

· GRS to LRS Conversion: Identified and transitioned non-critical and temporary storage accounts from GRS to LRS.

· Lifecycle Management: Applied TTL policies and migrated infrequently accessed data to Cool or Archive tiers.

· Delta Table Maintenance: Automated vacuum jobs to clean up obsolete Delta Lake file versions, optimizing data retention.

Impact Summary

Our cost optimization efforts led to substantial savings across all major components of the data platform, even as data volume and processing demand increased. Here’s a concise summary of the reduction impact and key levers:

Learnings

Optimization wasn’t a one-time fix but a deliberate, systematic journey across the stack.

“There are no silver bullets, only lead bullets.” — The Hard Thing About Hard Things.

This quote became our guiding principle, we tackled each inefficiency job by job, cluster by cluster.

We learned to challenge default configurations and vendor recommendations not just for functionality but for cost benefit analysis.

Teams should be empowered to move fast, but this must be balanced with clear governance through automated policies to avoid unintentional overspending.

Features like geo-replication, auto-scaling, and Delta versioning are powerful but their costs can compound quickly if left to default configurations. Our approach shifted to evaluating each feature for its cost-benefit impact and relevance to business goals. We aligned data refresh frequency to actual SLAs, enabled TTL and lifecycle rules, vacuumed Delta tables regularly, and used premium features like Photon and Serverless only when they clearly justified the cost.

Final Thoughts

This transformation underscores the maturity of the platform moving from speed-first execution to sustainable engineering excellence.

At Tata Digital, we view cost optimization as more than a financial metric, it’s a key pillar of responsible platform engineering. The changes we implemented not only reduced spend but helped us build a more stable, scalable, and efficient data foundation.