What 10,000 API Calls Taught Me About Writing Code

I needed to build an automated inventory of everything in my ArcGIS portal. Feature layers, web maps, dashboards-all of it. The goal was a dashboard that would show us what we had, how much storage we were using, and which items were chonky space munchers and which ones are authoritative.

My script worked perfectly on a small test set. Few hundred items. A minute or two. No problem.

Then I ran it on my actual portal with thousands of items.

And waited.

And waited.

For ~2,500 items, the script took over an hour. For what should have been a simple data-gathering exercise, this was unusable.

So I dug in. What I found was a hidden performance trap-and a lesson I still use today when working with any API.

The Problem: I must have called a thousand times…

The ArcGIS Python API is beautifully designed. You search for items, you get objects that feel like native Python. Access item.title, get the title. Access item.size, get the size. It just works.

But there’s a catch most tutorials don’t mention: not all data is loaded upfront.

When you search, the API returns just enough metadata to display a list-usually ID, title, and owner. Need more details? The first time you access certain attributes, the SDK quietly makes a separate network call to fetch them.

In an interactive session, you’d never notice. A few extra round-trips. Who cares?

In a batch script processing thousands of items? That “few extra” multiplied fast.

My initial script looked something like this:

The Initial Script example with Additional API calls.

Nothing complicated. Search for everything, grab the fields I needed. Why was this so slow?

I added timing logs to my loop and watched what happened. For 2,500 items, my script was making over 12,500 network calls. Each attribute-size, created, modified, content_status-was triggering a separate call. Multiply by 2,500 items, and suddenly I understood why it was taking an hour.

The worst part? I didn’t even need those extra calls. The data was already there.

The Hidden Culprit: Lazy Loading

The ArcGIS Python API uses a pattern called lazy loading. When you search for items, you get lightweight objects with just enough data to display a list-usually just the ID, title, and owner.

When you access other attributes-like size, created, or content_status-the API quietly makes a separate network call to fetch that data. Each attribute. Each item.

For 2,500 items accessing 5 attributes each, that’s 12,500 hidden API calls.

No wonder it was slow.

The “Aha!” Moment

While digging through the ArcGIS REST API documentation, I found something unexpected: the search endpoint actually returns all the metadata I needed in a single response. Fields like size, created, modified, and even content_status are all there from the start.

The Python API just doesn’t expose them immediately unless you ask.

So I tried a simple change: I started accessing the attributes directly, trusting they were already loaded.

What the optimized one looks like without additional calls.

Same code. Same fields. No additional API calls.

Result: Processing time dropped from over an hour to under a minute. (May be like 6–7)

What I Learned

1. The Data is Often Already There

Before optimizing, check what’s already in your search results. Many APIs return more than they let on. Read the docs. Inspect the responses. You might be surprised what you find.

2. Measure Before You Optimize

I assumed the bottleneck was network speed. Only after adding timing logs did I discover the real problem was hidden calls. Profiling saves you from optimizing the wrong thing.

3. Simple Beats Complex

I spent days researching async patterns and direct REST calls. The fix was a 30-second code change. Complexity is tempting. Simplicity is better.

4. Know what your code is actually doing.

That “simple” attribute access? It might be hiding a network call. Log your operations. Understand the cost of each line.

5. Work With the Tool, Not Against It

The Python API isn’t slow, my misuse of it was. Once I understood its behavior, it performed excellently.

The Final Result

My inventory script now processes all our items in under 10 minutes; down from what would have been over 2 hours. The code is clean, easy to maintain, and runs on the standard ArcGIS Python API.

No async magic. No direct REST calls. Just a better understanding of how the tools I already use actually work.

For Anyone Building Automation

If you’re writing scripts against any API, here’s some thoughts:

• Add timing logs early. Know where your time is actually going.
• Read the API documentation. Understanding the underlying response structure saves hours of debugging.
• Check what’s already there. Many APIs return rich metadata by default.
• Don’t assume you need extra calls. Profile first. Often, the data you need is already in your hands.
• Keep it simple. Before reaching for complex optimization patterns, make sure you’re not fighting the tool you’re using.

Thoughts

Sometimes the best optimization isn’t writing faster code-it’s understanding the code you already have.

Have you ever uncovered a hidden performance trap in an API you were using? I’d love to hear your stories.