Data Within Reach: What I Stopped Tolerating in Our Fintech Data Stack

Field notes from a Senior Technical Program Manager who got tired of watching good engineers fight bad infrastructure.

It’s 4 pm on a Thursday. The board review is in two days. My lead data scientist just walked into my office and said our fraud detection model isn’t running — not because it’s broken, but because it can’t access the production data it needs. The model is fine. The pipeline is fine. The data exists. It just lives somewhere the model can’t reach, locked behind a permissions gap no one budgeted time to close.

That moment crystallized something I had been circling around for months. In fintech, data infrastructure failures rarely announce themselves as infrastructure failures. They show up as model delays, compliance gaps, engineer frustration, and product decisions made on stale information. The data gravity problem — where compute has to travel to data rather than data moving to where it’s needed — is invisible until it isn’t.

What follows are my field notes. Not a product pitch. Not a vendor comparison. Notes from someone who has sat in that chair and had to explain to a board why something that should have been simple wasn’t.

The Production Gap, and Why It’s Your Problem

The problem isn’t that fintech companies don’t have data. They have too much of it. The problem is that the data lives in the wrong place at the wrong time, under the wrong governance model.

In traditional fintech architectures, production data sits in secured, regulated environments. Your models and analytics workloads live somewhere else — a separate cloud account, a different region, an analytics cluster with its own networking and Identity and Access Management (IAM) configuration. Moving data between these zones requires engineering work: pipelines, copies, transformations, latency. Each hop introduces delay and governance risk.

What I stopped tolerating: the assumption that this friction is a fixed cost. It isn’t. It’s an architectural choice, and it has a better alternative.

What Happened When the Data Stopped Moving

The board review scenario I opened with wasn’t a one-time event. Over the prior twelve months, we had four similar incidents where the gap between where our data lived and where our models needed it created operational failure. None of them showed up in sprint planning. All of them showed up in post-mortems.

The pattern was consistent: a model or analytics job needed data that was technically available but practically inaccessible. Engineering had to intervene. Time passed. The business window closed. Someone senior asked why.

I started tracking our data movement latency as a separate KPI. Not pipeline latency — data movement latency. The time between when production data was generated and when it was usable in a model training or inference context. The number was always worse than anyone expected, and always invisible until I made it visible.

The implication for you: if you aren’t measuring data movement latency as a distinct metric, you are flying blind on one of your most significant operational risks.

What ‘Governance Built In’ Actually Looks Like

Every compliance-adjacent fintech conversation eventually lands on governance. Most of the time, the word is used to describe a layer added on top of existing infrastructure — a policy engine, an audit log, a permissions model retrofitted onto data pipelines that were never designed for it.

What I’ve learned is that this retrofit approach is the source of most of our governance incidents. When Role-Based Access Control (RBAC) is an afterthought, it gets inconsistently applied. When data lineage tracking is bolted on, it has gaps. When audit logs are generated by a separate tool, they don’t match the operational reality of what actually ran.

Governance built in means the compute environment enforces access controls as a native capability, not a plugin. It means lineage is generated automatically from the execution engine, not reconstructed from logs after the fact. It means your audit trail is a byproduct of normal operation, not an additional workload.

The implication: when evaluating infrastructure for regulated workloads, the question isn’t ‘does it support governance?’ It’s ‘where does governance live in the architecture?’ If the answer is ‘in a separate tool,’ that’s a risk you’re accepting.

The Architecture Decision That Determines Your Cost Structure

There’s a specific architectural choice that happens early in fintech platform design that determines more of your future cost structure than almost any other decision: where does compute live relative to data?

Option A: compute lives in a separate environment (a different cluster, a different cloud service, a managed notebook environment). Data has to move to compute. This creates pipelines, copies, and latency. It also creates attack surface — every data movement is a potential leak vector.

Option B: compute runs inside the data environment. Models, containers, and inference jobs execute where the data already lives, under the same governance controls, without data ever leaving the perimeter.

Snowflake’s Snowpark Container Services (SPCS) is an implementation of Option B. You deploy containerized workloads — Python environments, model servers, custom APIs — directly inside Snowflake, with access to the data as a first-class capability rather than a network hop. The Open Container Initiative (OCI)-compliant containers run in Snowflake’s managed infrastructure. The data governance that already controls your Snowflake environment extends automatically to the compute layer.

The implication for architecture decisions: the cost of Option A is not just the pipeline infrastructure. It’s the latency, the governance overhead, the security review of every data movement, and the engineering time spent on plumbing instead of product. That cost compounds.

The Compute Pool as a Fiscal Control Mechanism

One of the things that consistently surprises TPMs who are new to Snowflake’s compute pool model is how much fiscal control it gives you compared to traditional virtual machine (VM)-based architectures.

In a VM-based world, you provision capacity and pay for it whether it runs or not. Auto-scaling helps, but scaling events have latency and the billing model tends toward always-on. In Snowflake’s model, compute pools are defined resources that can be suspended when not in use. You pay for what runs, not what exists.

From a program management perspective, this changes how you budget for ML infrastructure. Instead of reserving headroom and negotiating cloud commitments based on peak load estimates, you define the pool configuration that matches your workload profile and pay for actual execution. That’s a different kind of conversation with your finance team.

The implication: if you’re presenting ML infrastructure costs to your CFO or VP of Engineering, the compute pool model gives you unit economics that are actually defensible. You’re not showing utilization rates — you’re showing execution costs per workload.

What I’d Actually Do Next Quarter

If I were starting fresh with a fintech data platform initiative next quarter, here’s what I’d actually prioritize, in order.

First: map your data gravity problem. Before any tool selection, I’d spend two weeks documenting every place where data has to move to reach a compute workload. Latency per hop. Engineering hours per month maintaining those pipelines. Incidents per quarter caused by data movement failures. That map will tell you where your actual risk is.

Second: identify your highest-friction model deployment path. Where does it take longest to get a model from development to production inference on real data? That’s your biggest operational risk and your clearest case for compute-near-data architecture.

Third: run a proof of concept on your worst-case workload, not your easiest. The worst-case workload — the one with the most complex data requirements, the tightest governance constraints, the most problematic latency — is the one that will tell you whether the infrastructure actually works for your organization. Easy workloads succeed everywhere.

Fourth: price the Large Language Model (LLM) use case into your compute pool planning from day one. The organizations that will struggle with AI infrastructure costs in 18 months are the ones that didn’t factor inference workloads into their platform architecture today. LLM inference is not a bolt-on — it’s a first-class workload that requires compute pool sizing decisions made at platform design time.

Fifth: define your governance success metrics before you ship anything. What does ‘governance built in’ mean for your regulatory obligations? What audit artifacts do you need? What RBAC policies are non-negotiable? If you can’t answer those questions before the first model goes to production, you’re going to answer them in an incident debrief.

The fraud model we couldn’t run before that board review? We eventually got it working. It took two engineers three days to close the permissions gap manually. Three days of calendar time, six days of engineering time, and a board meeting where I had to explain why a capability we’d budgeted for wasn’t operational.

If your data infrastructure is working the way it should, you’ll notice it the same way I eventually did — in what disappears from your calendar. No emergency data access tickets. No pre-board scrambles. No post-mortem line items for pipeline failures. Just models running on the data they were designed to use, when they were designed to run.

That’s the infrastructure problem worth solving. Not because it’s technically elegant, but because it stops showing up in your board reviews.