Avoiding an AI Dystopia

It feels like I can’t peruse LinkedIn without seeing posts about massive spending of data centers and the insane CAPEX investments by big tech companies. Interspersed with these posts are friends being laid off by the same big tech companies or voluntarily decamping to greener pastures. The latest proliferation of posts have been tech companies nuking token leader boards, executives bemoaning AI spend without productivity gains and the start of token caps.

In my own AI usage there are not only token timeouts during the day, but also weekly quotas and increasingly “no servers available” errors. The AI tokens we are increasingly addicted to are heavily subsidized. Will the true token cost not emerge until we are all hooked or will innovation replace subsidization keeping costs manageable? Samuel Colvin, of Pydantic fame, lays out the trajectory really well in this interview.

The Information Retrieval Option

So what is the alternative? Our co-founder, Pramukta Rao, has long been a fan of information retrieval strategies (a.k.a. RAG) and how they harken back to the more classic computer science separation of memory and computation. Instead of trying to force a multi-billion-parameter foundation model to memorize and reason over the entire sum of human knowledge, why not use modern information retrieval strategies to decouple the two?

With agentic workflows fluidly mixing data and algorithms, pairing smaller models with an external index or vector database allows users to retrieve precisely what they need for a given prompt. This architecture mitigates the context bloat that triggers frequent timeouts and degrades reasoning performance in foundation models. A smaller model can tap into the exact same knowledge reservoir, reducing token expenditure before the compute layer engages.

This approach is a pragmatic return to classical computer science architectures. Trying to bake all data directly into a model’s parameters is the computational equivalent of buying an entire library just to read a single book. Information retrieval strategies mirror traditional memory hierarchies, where the language model functions like a CPU processing logic. The immediate model context operates as an expensive L1 cache, and the external database serves as the cheap+big hard drive. This approach treats context as a managed resource, rather than an endless destination for capital expenditure.

How Does this Work for Geospatial Data

Arguably geospatial data is one of the best candidates for an information retrieval approach. Our data primarily consists of facts, which AI models are prone to hallucinate left to their own devices. RAG replaces messy text matching with precise, mathematical coordinates. This allows AI models to anchor unstructured text directly to exact physical locations. Geography also provides a built-in hierarchy through boundaries and proximity rules. This natural structure optimizes how data is chunked and easily scales the context window for large language models.

Probably the most popular geospatial RAG today are POI grounding services used to prevent local search question in chatbots from hallucinating. There are several providers, and by way of example here is grounding data from work we’ve been doing with Overtures’s POI data.

Using this style of grounding data we can then help AI models provide better answers for local search related questions from users. The grounding allows the model to premise its answer with factual locations, while enhancing the standard local search query with useful or entertaining context.

Since Arc’teryx is directly in front of us lets ask the model/grounding how to get to Montbell.

If you give the examples a close read you’ll notice a few features not common in AI chatbot responses. First, the model understand what direction you are facing, and if destination or landmark is on your right or left. Also the chatbot knows what places look like, so it can describe the exterior to give you a visual reference.

Local Models for Local Queries

Grounding services are reasonably clever and useful, but they don’t directly solve the problem of needing data centers to deliver our answers. In this example Claude is still hosted at an xAI Colossus data center burning compute/energy/water. To truly shift the compute dynamics we need to run the model locally on the device that is navigating you.

To this end Google recently launched Gemma4 — their open models for running locally on edge devices. We were instantly big fans, but there were challenges. Specifically to have the model support our grounding service for sustained conversations we needed to adjust a few things.

• Tool-call parameter degradation over long conversations — the Gemma4 model can stop emitting valid tool-call parameter values as turn count grows. We’ve traced this to a precision issues in the LiteRT math implementation and built a workaround.
• Unreliable tool-call structure — the model would intermittently fail to produce a valid tool-call structure at all. We’ve addressed this though a grammar-constrained tool-call.
• RAG latency from split contexts — using separate LiteRT contexts for text generation and embedding could add avoidable latency to our local RAG setup. The new the runtime consolidates this for lower latency.
• Poor diagnostic visibility for hardware-specific builds — we needed more insight to correctly produce/deploy model variants tuned for the different LiteRT accelerator delegates.
• Context-window management — we needed a place to manage conversation history as it approaches the max context for the mode.
• VLM statefulness during tool calls — while our initial use case is geospatial reasoning for local search conversations, we see a lot of opportunity of collecting data with the same architecture. To that end to needed to upgrade the handling of the model’s conversational state when tool calls issue against the VLM.

I’m sure Google will address all these small optimizations in time, but in the spirit of open source we want to contribute our fixes back to the community sooner. The capabilities of the Gemma4 model for geospatial workflows are really impressive, and I’d encourage everyone to check it out. We’ll be adding our grounding service to compliment it soon.

While this work is a drop in the bucket to solving our current compute trajectory I believe that many hands make light work. If enough of us see the problem as important someone will discover the DWDM of the AI bubble.