Scaling to 300K Files: How We Built Bulletproof Rate Limit Handling for Large Shopify Stores

When Small Store Assumptions Meet Enterprise Scale

How we evolved our media scanner to handle Shopify’s largest stores — and what we learned about building resilient API integrations.

The Challenge of Scale

At Media Cleanup, we help Shopify merchants identify unused media files. Our scanner works brilliantly for the typical store: a few hundred products, maybe a thousand files.

But then we onboarded a merchant with 300,000 files.

This wasn’t a bug — it was a scale challenge we hadn’t encountered before. And it taught us valuable lessons about building for Shopify’s full ecosystem, from small boutiques to enterprise operations.

Understanding Shopify’s API Limits

Shopify uses a “leaky bucket” rate limiter for their GraphQL API:

• Standard stores: 1,000 points, restored at 50/second
• Shopify Plus: 2,000 points, restored at 100/second

For a store with 1,000 files, you need maybe 4 API pages. Quick and easy.

For a store with 300,000 files, you need 1,200+ API pages. That’s a fundamentally different engineering challenge.

Small store:   4 pages × 50 points = 200 points ✓ Easy
Large store:   1,200 pages × 50 points = 60,000 points 
               → Requires careful pacing over 20+ minutes

Our Original Design

Our initial implementation was optimized for speed — perfect for the 99% of stores under 10,000 files:

while (hasNextPage) {
  const response = await admin.graphql(QUERY, { variables });
  const data = await response.json();

  if (data.errors) {
    console.error("GraphQL errors:", data.errors);
    break; // Move on, don't block the user
  }

  processData(data);
  cursor = data.pageInfo.endCursor;
}

This worked great — until Shopify started returning THROTTLED responses for our largest merchants.

The Enterprise Solution

When we identified that our largest stores were hitting rate limits, we implemented what we call “resilient pagination” — a pattern that gracefully handles Shopify’s throttling without compromising accuracy:

async function fetchWithResilience(queryFn, pageLabel) {
  for (let attempt = 1; attempt <= 10; attempt++) {
    const response = await queryFn();
    const data = await response.json();

    if (isThrottled(data)) {
      const backoff = Math.min(5000 * attempt, 60000);
      console.log(`[${pageLabel}] Rate limited, waiting ${backoff/1000}s...`);
      await sleep(backoff);
      continue;
    }

    return data;
  }
}

Why This Approach?

1. Respects Shopify’s infrastructure — We don’t hammer their API; we wait politely
2. Guarantees complete data — Every file, every product, every reference
3. Transparent logging — Merchants and our team can see exactly what’s happening
4. Progressive backoff — 5s → 10s → 15s → up to 60s maximum

The Results

After implementing resilient pagination, our largest merchant’s scan completed successfully:

Total files scanned — 299,858

Products analyzed — 10,107

Files correctly categorized — 100%

Scan duration~55 minutes

For context, a 55-minute scan for 300K files means we’re processing ~91 files per second while respecting API limits. That’s efficient.

What We Built Along the Way

This experience led us to build several features that benefit all our merchants:

1. Shop-Specific Maintenance Mode

npm run maintenance -- block store.myshopify.com

We can now pause access for individual stores while investigating issues — without affecting anyone else.

2. Comprehensive Health Checks

DATABASE HEALTH CHECK
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✓ Session valid
✓ Scan complete  
✓ 299,858 files indexed
✓ Data integrity verified

3. Real-Time Progress Logging

[FILES] Page 202: Rate limited, waiting 5s...
[FILES] Page 202: Success (retry 1)
[PRODUCTS] Processing page 100 (3,000 products scanned)

Lessons for Fellow Shopify App Developers

1. Design for the Long Tail

Your app might work for 99% of stores. But that 1% of large stores will push your architecture to its limits. Plan for them.

2. Shopify’s Rate Limits Are a Feature, Not a Bug

They protect the platform (and your app) from overload. Work with them using backoff strategies, not against them.

3. Visibility Is Everything

When a scan takes 55 minutes, merchants need to know it’s working. Progress logs, status updates, and transparency build trust.

4. Build Your Safety Rails Early

Maintenance modes, health checks, and audit logs seem like overkill — until you need them. Then they’re invaluable.

The Technical Pattern

For anyone building Shopify apps that handle large data sets, here’s our battle-tested approach:

const MAX_RETRIES = 10;
const BASE_DELAY = 5000; // 5 seconds
const MAX_DELAY = 60000; // 60 seconds

async function paginateWithResilience<T>(
  fetcher: (cursor: string | null) => Promise<Response>,
  extractor: (data: any) => { items: T[]; nextCursor: string | null }
): Promise<T[]> {
  const results: T[] = [];
  let cursor: string | null = null;

  do {
    let data: any;

    for (let attempt = 1; attempt <= MAX_RETRIES; attempt++) {
      const response = await fetcher(cursor);
      data = await response.json();

      if (data.errors?.some(e => e.extensions?.code === "THROTTLED")) {
        const delay = Math.min(BASE_DELAY * attempt, MAX_DELAY);
        console.log(`Throttled, attempt ${attempt}/${MAX_RETRIES}, waiting ${delay}ms`);
        await new Promise(r => setTimeout(r, delay));
        continue;
      }
      break;
    }

    const { items, nextCursor } = extractor(data);
    results.push(...items);
    cursor = nextCursor;

  } while (cursor);

  return results;
}

Conclusion

Supporting a 300,000-file store wasn’t in our original roadmap. But it pushed us to build a more resilient, more transparent, and ultimately better product.

Every Shopify app eventually faces this inflection point: the moment when a large merchant reveals the limits of your current architecture. How you respond defines your product.

We chose to build for scale. And now every merchant — from the 50-file boutique to the 300K-file enterprise — gets the same bulletproof scanning experience.

Building for Shopify? Start with the small stores, but architect for the big ones.

About Media Cleanup

We help Shopify merchants find and remove unused media files, saving storage costs and improving site performance. Our scanner handles stores of any size — from startups to enterprise.

Try Media Cleanup → *Intall Link*

Learn More -> *Media Cleanup*