Breaking the Black Box: How AI Embeddings Cut Cloud Costs by 50% While Boosting Matching by 65%

By Paul Beata, Patrick Cosmo

Every month, streaming platforms like Spotify and social platforms like YouTube send the music publishing industry a stack of CSV files. Inside each one: metadata for millions of songs they’ve streamed but don’t know who to pay the publishing royalties to. They call it the “Black Box”. The same files land in every publisher’s inbox. Whoever identifies their own catalog in those rows fastest and most accurately gets paid; the rest is left on the table. At Warner Chappell Music, the publishing division of WMG, Global Match solves this.

Quick context on the rights side: a recording is a specific recorded version of a song; the underlying composition (the lyrics, melody, and structure) is a separate piece of intellectual property. Record labels administer rights to recordings; music publishers like Warner Chappell administer rights to the compositions behind them.

Global Match is a tool that identifies recordings that belong to our ‘rights’ catalog within these CSV files. For a long time, Global Match did this with brute-force SQL string-matching against our catalog. As the volume of global music and publishing data skyrocketed, our cloud bills spiked right alongside it.

So, we decided to re-architect the entire system. By shifting from heavy database processing to in-memory machine learning, we did what sounds like a contradiction: we slashed our cloud operational costs by over 50% while simultaneously boosting our matching output by 65%.

When you are dealing with high-volume data matching, scaling up usually means spending up. Here is exactly how we pulled off the opposite.

The Wall: Why SQL Stored Procedures Couldn’t Scale

To understand how we fixed it, you have to look at where we started. Our original matching infrastructure relied heavily on Snowflake stored procedures and sequential string-matching.

On paper, it made sense initially. Our music catalog data lived in the database, so we processed it in the database. But strict string-matching is notoriously rigid. If an artist’s name had a typo, a missing middle initial, or a non-Western character, the database missed it. To catch those edge cases, we had to write increasingly complex, computationally heavy SQL queries.

As a result, our database compute costs were rising, the system could only process files with up to 30K rows, and a single batch could take 12+ hours just to churn through. The user experience was equally cumbersome: upload a file to a region-specific cloud folder and then wait until tomorrow to see the results. We were essentially throwing expensive cloud compute at a problem that required a smarter algorithmic approach. And while we were waiting on those 12-hour jobs, the same monthly files were getting processed by every other publisher.

The Paradigm Shift: From SQL to AI Embeddings, From Always-On to On-Demand

We realized we needed a smarter matching approach, and a move to on-demand compute. This led us to Global Match 2.0, where we abandoned the heavy database-layer processing entirely in favor of text embeddings and vector spaces.

To maximize this efficiency, we rebuilt our underlying cloud architecture using a highly parallel, event-driven model. Instead of keeping massive servers provisioned and waiting, we built a system that scales compute dynamically up and down. When a massive batch file drops, a fleet of lightweight workers spin up (AWS Lambda functions), process the data in parallel at a rate of 20,000 rows per minute, and immediately shut down.

The parallel processing takes place in 2 phases:

1. The Preprocessor: A “preprocessor” AWS lambda function is triggered by the upload of a new file to S3 (uploaded by the user via our UI). This lambda effectively shards each file for parallel processing by enqueueing multiple separate work requests into an AWS SQS worker queue. The lambda doesn’t literally split the incoming file into separate files; instead it points each work request to a range of bytes (rows) in the original file that denote the boundaries of the shard assigned to the work request.

2. The Worker Fleet: The presence of a new work request in SQS then automatically triggers an elastic pool of worker lambda functions, which read their shard’s data into memory using the S3 Get Request API and RFC 9110 range headers to read only a subset of the original request’s data, and feed it to our AI matching service, a long-running ECS container that holds the embedding model and our vectorized catalog in memory.

How parallelized on-demand processing works in Global Match

The Matching Engine

Instead of asking a database to compare characters, we built our matching around AI-generated vector embeddings.

Our catalog’s text data like recording titles, creator names, and artist metadata lives in Snowflake. Before encoding, we drop historical and non-claimable works, narrowing the ~10M-record catalog down to the ~2M currently claimable compositions, keeping things tight enough to stay in memory. We then encode each of those 2M records into a vector embedding using a BERT-based language model. In this learned vector space, semantic similarity becomes geometric proximity — typos, transliterations, and ‘feat.’-vs-’featuring’ variations all sit close to the canonical version, where strict string-matching would have missed them entirely.

To query this massive space efficiently without running out of memory, we leveraged scikit-learn’s standard K-Nearest Neighbors search to find the best potential matches. The full set of catalog embeddings, together with the spatial index sklearn builds over them, gets serialized into a single pickle file weighing in at a few hundred megabytes. Our AI matching service, a long-running ECS container fronting the worker Lambdas, loads the pickle into memory at startup and keeps it resident. When a worker hands it a row of CSV text, the model encodes the input, runs a nearest-neighbors lookup against the in-memory index, and returns the top candidates.

After the AI identifies the best “mathematical” matches, an additional business layer with claiming rules and rights information makes sure that what we claim is actually claimable, not just mathematically close.

Suddenly, we weren’t running brutally long, nested string-comparison queries anymore: we were doing lightning-fast AI vector mathematics in memory.

The Results: Doing More with Less

The impact was immediate and dramatic. By moving the heavy lifting from disk-heavy database compute to an on-demand, optimized AI-based matching system, we completely decoupled our data volume from our infrastructure costs.

• Cost Reduction: Our cloud operational costs dropped by over 50%
• Speed Boost: Through parallelization our processing speed increased by over 100x
• Match Rate Coverage: Total matching output increased by 65%

Even better, this fast AI-driven backend completely transformed our user experience. Because the system could serve up matches instantly, we built a clean web UI to replace our old SharePoint workflows. This created a perfect “human-in-the-loop” flywheel: our ops teams could review matches 25% to 50% faster, and their manual corrections can be fed right back into the model as fresh training data, making the system even smarter over time.

The Lesson Learned

Don’t mistake a bigger hammer for a better tool. When performance lags or data scales, the easiest temptation in the cloud era is to provision larger instances, buy higher database tiers, and throw money at the problem. But horizontal scaling of an inefficient architecture just acts as a multiplier for your cloud bill.

Take a hard look at the core math of your problem. Sometimes, the path to a 50% cost reduction isn’t found in optimizing your infrastructure configurations, but in completely rethinking the data structures and algorithms powering your code.