Optimization strategies for agentic systems

The main challenges of Agentic AI systems are largely the same across use cases: fast responses with minimal latency, lower token usage to control costs, strong security and guardrails, and effective handling of long-term memory and context history.

Most frameworks focus on optimizing for speed in the early stages. Standards and well-designed architectures, however, optimize for longevity

In this article, I want to show how a few practical approaches can help us reduce costs, make AI systems faster, and keep them more secure:

• Semantic and prompt caching
• Context compression
• Choosing the right data format
• Selecting the right model
• Using the right agentic design patterns
• LLM & AI gateways and proper security guardrails

Semantic Caching

In real AI applications, users ask the same or very similar questions far more often than we expect especially in chatbots, enterprise assistants, and support systems. In many production workloads, 20–40% of incoming queries are semantically similar. That means a large share of LLM calls are spent generating answers that are almost identical.

Semantic caching solves this by reusing answers based on meaning, not exact wording. When applied correctly, it can:

• reduce token usage and costs by 30–70%
• improve response times by 10–15×
• significantly reduce unnecessary LLM calls

How semantic caching works

Users express the same intent in many different ways: Forgot-password example (same intent):

• “I forgot my password.”
• “How can I reset my password?”
• “What should I do if I lost my password?”

Different sentences, same meaning. Semantic caching allows the system to reuse the same answer instead of calling the model every time.

Some implementation tips for semantic caching:

• Tune similarity thresholds to balance accuracy and cache hit rate
• Invalidate cache on data changes (event-driven or TTL)
• Handle embedding model upgrades with versioning or re-embedding
• Prevent cache poisoning using validation and context metadata

Right Model Selection (latency vs intelligence)

For agentic AI systems, latency is often the biggest pain point. At the same time, accuracy and logical reasoning still matter — especially for complex tasks.

The problem is simple: you usually can’t optimize for speed and deep reasoning at the same time. That’s why model selection should always be use-case driven, and in many cases, hybrid.

https://artificialanalysis.ai/

• Standard models are optimized for speed and scale. They work best for simple Q&A, chat, content generation, and high-throughput use cases.
• Reasoning models trade speed for accuracy and deep logic. They are slower but much stronger at math, complex reasoning, and advanced coding.

Key strategies include:

• Dynamic model routing to send simple tasks to fast models and complex tasks to reasoning models
• Semantic or intent-based routing at the ai gateway layer
• Using smaller models first for classification or drafting, then escalating when needed
• Combining LLMs with rules or classic ML to reduce cost and improve predictability
• Fallback and load balancing to handle provider latency or outages

The goal is not to use the smartest model everywhere, but to use the right model at the right time balancing latency, cost, and accuracy.

Data Formats

The text compares common data formats based on their performance with specific models (GPT-5 Nano, Gemini 2.5 Flash Lite, and Llama 3.2 3B).

• YAML: The top performer for output quality on GPT-5 Nano and Gemini 2.5 Flash Lite.
• Markdown: The most token-efficient format. It uses ~10% fewer tokens than YAML and 34–38% fewer than JSON.
• JSON: Performed poorly on GPT-5 Nano and Gemini 2.5 Flash Lite. It is token-heavy due to structural symbols (curly braces, quotes) and whitespace.
• XML: The worst performer for GPT-5 Nano and Gemini 2.5 Flash Lite.
• Llama 3.2 3B Instruct: This model was format-agnostic, showing similar performance regardless of the data structure used.

If you are using JSON Use as possible less nested objects cause all curly braces and symbols use tokens: Many of the tokens are white space characters, control characters, and various combinations of brackets and other characters.

TOON (Token-Oriented Object Notation) is introduced as a new standard designed specifically to fix the inefficiencies of JSON.1

• Token Efficiency: Reduces token count by 30–60% compared to JSON.2
• Key Features: Balances human readability with token optimization; includes structured validation (explicit array lengths and field definitions).3

Which Format to Use?

For Maximum Efficiency: → CSV / For Structure + Efficiency: → TOON / For Complex Nesting: → JSON

Prompt Caching

https://platform.openai.com/docs/guides/prompt-caching

Unlike Semantic Caching (which retrieves previous answers to skip the LLM entirely), Prompt Caching optimizes the processing of the LLM call itself. It is designed for workflows where the System Prompt includes lengthy, static elements such as detailed instructions, tool definitions, or few-shot examples.

Key Benefits: Prompt Caching is enabled automatically for gpt-4o and newer models, requiring no code changes and incurring no extra fees. Cached input tokens are roughly 10× cheaper than regular input tokens on both the OpenAI and Anthropic APIs.

Structuring Prompts: To maximize cache hits, prompts must be structured to ensure exact prefix matches.

• Beginning (Static): Place all static content here. This includes system instructions, examples, images, and tool definitions. These must remain identical across requests.
• End (Variable): Place user-specific or dynamic content at the very end of the prompt.

Note: Cache hits rely on the initial portion of the prompt matching exactly. If variable content is placed at the start, the cache will be invalidated.

Caching is enabled automatically for prompts that are 1024 tokens or longer. When you make an API request, the following steps occur:

1. Cache Routing:

• Requests are routed to a machine based on a hash of the initial prefix of the prompt. The hash typically uses the first 256 tokens, though the exact length varies depending on the model.
• If you provide the [prompt_cache_key](https://platform.openai.com/docs/api-reference/responses/create#responses-create-prompt_cache_key) parameter, it is combined with the prefix hash, allowing you to influence routing and improve cache hit rates. This is especially beneficial when many requests share long, common prefixes.
• If requests for the same prefix and prompt_cache_key combination exceed a certain rate (approximately 15 requests per minute), some may overflow and get routed to additional machines, reducing cache effectiveness.

1. Cache Lookup: The system checks if the initial portion (prefix) of your prompt exists in the cache on the selected machine.
2. Cache Hit: If a matching prefix is found, the system uses the cached result. This significantly decreases latency and reduces costs.
3. Cache Miss: If no matching prefix is found, the system processes your full prompt, caching the prefix afterward on that machine for future requests.

Context Compression (shortening memory without losing meaning)

Long-running agent sessions can easily generate millions of tokens of conversation history, far exceeding what any model can realistically keep in working memory, which makes context compression unavoidable in real systems.

For example :

• DeepSeek-OCR: Takes snapshots of pages (like being that smart friend who remembers “the key point was in the blue box on page 47”) but traditional AI memorizes every word of every book (like being that friend who recites entire movie scripts)

Contextual compression is proposed as a superior method to address these limitations. It offers four primary benefits:

• Overcoming Token Limits: It packs more information into restricted token windows, allowing models to process larger inputs.
• Efficiency: It enables the LLM to handle long documents or histories by focusing only on relevant information without being overwhelmed.
• Redundancy Reduction: It strips away unnecessary repetition, ensuring only critical data is retained for higher accuracy.
• Resource Optimization: It lowers memory usage and speeds up inference times, which is crucial for resource-constrained environments.

Agentic AI Design Patterns

To get a deeper understanding of how enterprise-scale agents are architected in the real world, you can start by reading this BCG report on building effective enterprise agents: https://www.bcg.com/assets/2025/building-effective-enterprise-agents.pdf

• ReAct (Reasoning + Acting): The agent loops between thinking and using tools (search, APIs, DB) step-by-step, adjusting based on what it observes, which makes it great for unpredictable tasks but often slower and more expensive.
• Reflection: The agent generates a first draft, critiques its own output, and rewrites to improve quality, which reduces mistakes and hallucinations but increases cost and latency because it adds extra LLM passes.
• Planning: Before acting, the agent breaks a big goal into smaller ordered steps and then follows them, improving structure and predictability for complex tasks but sometimes wasting work if earlier steps make later ones unnecessary.
• Multi-Agent Systems: The system splits work across specialist agents (either a manager-led hierarchy or a swarm where agents coordinate directly), which can boost capability and parallelism but becomes harder to control and debug as complexity grows.

Memory & Context

Yet memory in AI Agents is deeply intertwined with context. Without context, even the most sophisticated conversation becomes meaningless.

Short-term memory is the immediate context window the model sees while generating responses. It contains : The last few user messages , The agent’s own reasoning or scratchpad, Recent steps of a task

Long-term memory gives agents continuity. Using vector stores, databases, or memory APIs, agents can store : User preferences , Project metadata , Past decisions , Summaries of previous session , Documentation or code relevant to future tasks

RAG is how agents bridge memory and reasoning. Here’s how it works :

1. The agent takes your query.
2. It uses embeddings to search long-term memory or external docs.
3. It retrieves the most relevant chunks.
4. It injects those into the model’s context window.
5. The model generates responses using both short-term and retrieved long-term memory.

This gives agents : Up to date knowledge , Accuracy across large datasets , Recall of past conversations , The ability to stay consistent across days, weeks, or projects

LLM Gateway & AI GATEWAY

At the core of a strong LLM gateway is its ability to hide the complexity of individual model APIs behind a single, unified interface, allowing developers to work with multiple models without changing application code. Platforms like OpenRouter simplify this by exposing one API endpoint for many LLMs, making it easy to switch models based on cost, performance, or availability. Similarly, Kong AI Gateway builds on mature API management capabilities to provide dynamic routing, load balancing, and centralized control. Together, these gateways reduce integration effort while enabling smarter, production-ready model selection

Kong AI Gateway offers extensive capabilities beyond the core features detailed in this article, including:

• Semantic caching
• Prompt compression
• AI model failover and retry mechanisms
• Automated RAG injection
• Content safety guardrails
• PII sanitization, prompt engineering templates
• Request/response transformations
• MCP traffic gateway support
• Streaming capabilities, secrets management
• Advanced analytics
• Audit logging
• Custom plugin extensibility to address comprehensive enterprise AI governance and operational requirements

Security & Guardrails Checklist (for agentic AI systems)

Identity, Access, and Permissions

• Use strong AuthN/AuthZ for every agent, tool, and MCP/server call (OAuth2, mTLS, short-lived tokens).
• Least-privilege by default: agents should start with “read-only” and earn write permissions per workflow.
• Fine-grained tool permissions (RBAC/ABAC): allow “read spreadsheet” but deny “delete rows,” allow “transfer preview” but deny “execute.”
• Step-up verification for risky actions (re-auth, OTP, manager approval, or human-in-the-loop).
• Per-tenant isolation: separate credentials, policies, and storage boundaries for each tenant.

Data Protection and Privacy

• PII masking / redaction before LLM calls (<EMAIL>, <PHONE>, ***1234) and apply it consistently across tools.
• Field-level allowlists: only send explicitly permitted fields; drop everything else by default.
• Data classification tags (public / internal / confidential / regulated) and route accordingly.
• Data minimization: send only what’s needed for the current step, not entire records or schemas.
• Tenant-specific encryption keys (KMS + rotation) for stored prompts, embeddings, and agent memory.
• Retention policies: automatically expire agent memory, cache entries, and logs based on risk level.

Prompt Injection & Input Hardening

• Treat user text as untrusted and keep it clearly separated from system instructions.
• Prompt injection sanitizer on every input (strip tool instructions, hidden directives, “system override” patterns).
• Jailbreak phrase detector (e.g., “ignore previous instructions…”, “act as system…”) + escalation to safe mode.
• Content boundary formatting (explicit delimiters + quoted user content) to reduce instruction confusion.
• RAG hygiene: sanitize retrieved documents too (docs can contain injections).

Output Guardrails (Leakage Prevention)

• Post-generation scanning for PII/secrets before returning results to the user.
• Secret detection (API keys, tokens, credentials, private URLs) + automatic redaction/blocking.
• Policy checks for disallowed content and disallowed actions (financial transfers, account changes, etc.).
• Citations/grounding requirement for factual answers in regulated workflows (reduce hallucinations).
• Safe fallback responses: when confidence is low or policy triggers, respond with “I can’t complete that” and ask for confirmation.

Tool Safety and Execution Controls

• Human approval for irreversible actions (payments, deletion, customer data export, production changes).
• Two-phase execution: “plan → show proposed action → approve → execute.”
• Tool argument validation with strict schemas (reject unknown fields, unexpected parameters).
• Rate limiting per agent and per tool to prevent abuse and runaway loops.
• Sandboxed code execution with timeouts, memory caps, network restrictions, and filesystem isolation.
• Egress controls: block agents from sending data to unauthorized external domains/APIs.
• Read vs write separation: separate tools/endpoints for read-only vs write operations.

Gateway / Central Policy Layer (the “air traffic control” hub)

• Single proxy layer to inject API keys and enforce policies (never put secrets in prompts).
• Centralized policy engine (OPA-style rules or similar) applied consistently across all tools/providers.
• Model routing by data sensitivity: regulated inputs go only to approved models/endpoints.
• No-train / no-log mode for regulated workloads; enforce at gateway level, not just in app code.
• Vendor fallback with policy parity: switching providers should not bypass guardrails.

Logging, Monitoring, and Incident Readiness

• Audit logs for everything: prompts, tool calls, approvals, outputs, and policy decisions (with redaction).
• Strip or tokenize logs: never store raw prompts containing personal data; store hashes or structured traces.
• Anomaly detection: alert on spikes in tool usage, token usage, unusual exports, or repeated jailbreak attempts.
• Traceability: attach correlation IDs to every agent step so incidents can be reconstructed.
• Kill switch: ability to instantly disable a tool, model, or entire agent workflow.

Memory, Caching, and Vector DB Security

• Prevent cache poisoning: validate cached answers, store provenance, and bind cache entries to tenant/session/domain.
• Cache invalidation rules: TTL + event-driven invalidation when policies/knowledge changes.
• Access control on embeddings: vectors can leak meaning; protect them like sensitive data.
• Encrypted vector stores and strict tenant isolation (no cross-tenant nearest-neighbor leakage).

Testing and Continuous Hardening

• Monthly red-teaming with adversarial prompts (prompt injection, data exfiltration, tool misuse).
• Regression test suite for security prompts (known jailbreak attempts should stay blocked).
• Canary / shadow mode for new models and new tools with stricter monitoring before full rollout.
• Policy-as-code: versioned guardrails, peer-reviewed like normal code changes.