From Prompting to Context: Techniques for Efficient AI Applications

AI applications are rapidly moving from experimentation to production

AI applications are rapidly moving from experimentation to production, where efficiency, accuracy, and cost-effectiveness matter as much as innovation. Designing systems that balance these trade-offs requires more than just good prompting — it calls for thoughtful orchestration of models, caching strategies, and context generation. In this post, we’ll explore some of the proven techniques we’ve adopted while building production-grade AI agents, highlighting how small design choices can significantly impact accuracy, latency, and cost.

Double Checking Accuracy with Independent Verification

Not relying on the first generated response of theLLM

One effective technique we’ve adopted to improve the accuracy of our agents is output verification using an independent LLM. Instead of relying solely on the first generated response, we add an extra layer of validation.

Take our example from one of the AI powered applications AdChoreo. Role for one of the agents in the application was to come up with an SQL query based on the user request. While the agent is often able to generate the required query, there are times when the result is either syntactically incorrect or does not fully capture the user’s intent. Rather than risking execution of a faulty query, we route the output through a second, independent model that acts as a verifier.

This verifier checks the query for accuracy, alignment with the original user request, and potential execution issues. If errors are found, it can correct them before the query is executed. This extra safeguard not only minimizes mistakes but also improves trust in the system’s outputs leading to more reliable performance overall.

Picking the right model also plays an important role in independent verification. You may pick the same model as the agent’s but it may have a self-bias, where a model favors its own reponses over the response generated by other models. You can pick a smaller model because it’s able to judge the outputs of a stronger model. To make the evaluation consistent adjust sampling variables for a consistent response.

This comes with its own trade-offs, including higher costs and added latency, but the gains in accuracy and reliability often outweigh these concerns.

Leveraging In-Session Context Caching

Caching has long been integral to software applications to reduce latency and cost. Many ideas from software caching can be used for AI applications. Here we’re going to talk about Exact caching for context construction.

For example, consider an SQL agent that takes a natural language request, converts it into a SQL query, and then runs that query against a database. If a user asks “What were the total sales last month?”, the agent might generate the SQL:

SELECT SUM(amount) 
FROM sales 
WHERE DATE_TRUNC('month', sale_date) = DATE '2025-08-01';

If this query is executed frequently perhaps by multiple users or dashboards — the result can be cached along with the query string itself. The next time the same request comes in, instead of hitting the database again, the system can quickly return the cached result. This saves both time (lower latency for the user) and resources (fewer database reads).

However, caching introduces a tradeoff. If the underlying data changes often, a cached result may become stale. This was the case in our implementation where we were caching Ad Spend data. In such cases, it’s best practice to associate a time-to-live (TTL) with the cached entry, depending on how frequently the business needs real-time accuracy. This way, the system gets the benefit of faster responses while ensuring that data remains reasonably up-to-date.

In essence, exact caching allows AI agents to feel more responsive and cost-efficient, without sacrificing correctness when coupled with smart expiry strategies.

Smarter Context Generation for Complex Workflows

…where human-guided orchestration comes in…

Context generation is one of the most important aspects of building Agents and RAG systems. When there are multiple data sources involved, it becomes a non-trivial problem. Leaving everything to the agent is rarely advisable — especially for critical tasks — because LLMs are not always reliable in following multi-step instructions exactly as defined. This is where human-guided orchestration comes in.

Take our use case with the Facebook Marketing API as an example. We needed ad-related metrics using the Insights endpoint, but to know the current status of an ad, a separate API call was required. Relying on the agent to figure this out dynamically would be inefficient and error-prone. Instead, we programmatically decided which API endpoints should be used for which information. The agent only guided us on what information was required, but the logic of how to fetch it was managed in our system. This reduced latency and saved costs by avoiding unnecessary LLM calls.

By following this approach we gain tighter control, faster execution, and reduced costs. In short, context generation works best when agents provide direction, but the system enforces structure. This hybrid balance ensures both flexibility and reliability in production-grade applications.

Reducing Costs with Sharper Prompting Techniques

Another best practice to consider when designing system prompts for agents is to be explicit about optimization strategies the agent should follow. While LLMs are good at reasoning about what information to fetch, they don’t always choose the most efficient path unless guided.

For example, when I wanted the agent to fetch metrics using the Facebook Marketing API, the default behavior was to pull all raw data for a particular field. This resulted in large, verbose payloads, much of which was irrelevant to the user’s query. The unnecessary data increased both the cost (more tokens for the LLM to process) and the response time.

Knowing that the API provided filtering capabilities, I updated the system prompt to explicitly instruct the agent to apply a filtering parameter when fetching data. Instead of retrieving every possible value for that field, the agent would now request only the subset of data matching the user’s need. This not only cut down the volume of data retrieved but also reduced the amount of information the LLM had to parse downstream.

The result was twofold: lower latency and reduced costs, without sacrificing accuracy. By embedding these types of domain-specific optimization hints directly into the system prompt, we can ensure that agents make better choices consistently, leading to a smoother and more efficient user experience.

Conclusion

As AI systems transition from prototypes to production-ready applications, the small design decisions matter the most. Independent verification ensures accuracy, caching improves responsiveness, smarter context orchestration reduces errors, and sharper prompts cut costs. Together, these strategies allow teams to strike the right balance between efficiency, reliability, and scalability. By applying these principles, developers can unlock the true potential of AI agents making them not only powerful but also trustworthy in real-world use cases.