I Built an AI Agent Without a Framework, Here’s What I Learned

Overview

I built an end-to-end customer service agent for a handmade ceramic business run by my friend Olivia. The agent classifies customer intent, routes queries to specialized handlers, and responds in Olivia’s voice via email. Over 50 test queries, the system achieved 87% accuracy with 85% autonomous handling, processing each query in ~1.5 seconds at $0.0025 per call, scaling to ~$0.15/month on Olivia’s current volume (15 queries/week).

The Problem

Olivia receives 10–20 customer emails per week asking about item details, shipping, returns, recommendations, and custom orders. Answering manually takes 15+ hours per month. Each reply requires knowledge of:

• 9 handmade catalog items (prices, materials, care)
• 3 seasonal collections (stories, aesthetic, use cases)
• FAQs (shipping, returns, materials, commissions)
• Olivia’s writing voice (warm, creator-to-creator, first-person singular)

A pure agent (no routing) would hallucinate; a Langchain implementation meant vendor lock-in and slower iteration and wouldn’t give me the visibility I need to learn. I chose Python + Flask + Claude Haiku with fixed orchestration at routing and synthesis points, keeping the system interpretable and fast.

Why Fixed Orchestration Over Pure Agent

My project is deliberately designed as a routing agentic workflow assistant, not a pure agent, because the task is well-defined and predictable. I needed to balance the trade-off between latency and cost for better task performance.

The structure is fixed and deterministic — my Python code orchestrates everything — but LLM intelligence is embedded at specific, well-defined steps within that structure:

Fixed Python orchestration
        │
        ├── [LLM step] classify intent       
        │
        ├── Fixed routing logic
        │
        ├── [LLM step] extract item name    
        │
        ├── Fixed Supabase query
        │
        └── [LLM step] write Olivia reply

Framework Decisions

Why Anthropic SDK over Langchain? This was one of my first projects building a multi-routing agent from scratch. Abstraction layers would have hidden too much of the mechanics. My project wasn’t complicated enough to justify Langchain: one LLM, one routing step, 8 simple tools. Langchain would have added dependencies I couldn’t control and made debugging far less clear.

Why custom tools instead of built-in tool search? I deliberately did not use the Claude tool search capability because I had fewer than 10 tools overall (8 total), and each one individually had less complexity than tool search justifies. Instead, I manually defined tools to maintain full visibility into the model’s behavior and handle errors manually during traces and evals.

Why no SDK abstraction for tool definitions? I intentionally defined tools manually rather than using SDK abstractions. This gives me full visibility into the working of the model, clear error handling, and complete control over what gets logged and traced.

Architecture: Fixed Orchestration + LLM Intelligence

Customer Email
    ↓
[Intent Classifier] ← Claude LLM (multi-intent JSON, confidence labels)
    ↓
┌─────────────────────────────────────────────────────────┐
│ Primary Intent Router (high confidence only)              │
├─────────────────────────────────────────────────────────┤
│ item_inquiry → item_details_tool                          │
│ collection_inquiry → collection_inquiry_tool              │
│ recommendation → recommendation_tool                      │
│ orders (shipping/returns/custom) → orders_tool            │
└─────────────────────────────────────────────────────────┘
    ↓
[Handler + Intent Gate] ← LLM checks if intent fits this tool
    ↓
    ├─ Yes → [Main Handler Logic] ← Database queries + LLM synthesis
    │         ├ answered → customer reply
    │         ├ needs_clarification → follow-up question
    │         └ needs_human → flag for Olivia
    │
    └─ No → [Flag for Human Review]
    ↓
[Multi-Intent Synthesis] ← If 2+ high-confidence intents, combine replies
    ↓
Email via SMTP (Olivia's voice, no hallucinations)

Why fixed orchestration? It keeps the system interpretable, I know exactly which LLM call happens where, which means I can measure, debug, and improve each step independently. The system doesn’t hallucinate about its own capabilities because it can’t route to a handler that doesn’t exist.

Tool Schema Design

I made deliberate decisions around tool schema to maximize LLM clarity without over-specifying:

• Detailed tool descriptions: Each tool clearly states what it does and when the LLM should use it
• Input examples: I added detailed input examples in the tool schema (e.g., when a user asks about item details, the LLM knows the item name must be provided as input). This helps Claude understand when it has enough information to call a tool vs. when it needs to ask a clarifying question.

Detailed tool schema

• Multi-intent detection: The system explicitly detects when a query has multiple intents (e.g., “Suggest something boho, show me the Spring collection, and how long does shipping take?”). Each intent gets routed independently, and if multiple handlers run, their outputs are synthesized into one coherent reply.

Core Components

1. Intent Classification with Qualitative Confidence

Early testing revealed a critical failure mode: multi-intent queries were routing to both handlers because numeric self-reported confidence scores (“0.75”) are just plausible-looking tokens. I explored OpenAI’s logprobs API (real probability from token distributions) but found that it added 1.6x token cost, four parallel API calls, and a new threshold-tuning problem. Instead, I switched to qualitative labels: high / medium / low.

python

# Classifier output (single Claude call)
{
  "intents": [
    {"intent": "recommendation", "confidence": "high"},
    {"intent": "collection_inquiry", "confidence": "medium"}
  ],
  "primary_intent": "recommendation"
}

Routing logic: High + high → both run + synthesis. High + medium → high only. High + low → high only. This reduced multi-intent collisions by 100% in testing (from 2 out of 3 ambiguous queries failing to 0 failures).

Tool Schema Design

I made deliberate decisions around tool schema to maximize LLM clarity without over-specifying:

• Detailed tool descriptions: Each tool clearly states what it does and when the LLM should use it
• Input examples: I added detailed input examples in the tool schema (e.g., when a user asks about item details, the LLM knows the item name must be provided as input). This helps Claude understand when it has enough information to call a tool vs. when it needs to ask a clarifying question.
• Multi-intent detection: The system explicitly detects when a query has multiple intents (e.g., “Suggest something boho, show me the Spring collection, and how long does shipping take?”). Each intent gets routed independently, and if multiple handlers run, their outputs are synthesized into one coherent reply.

Testing showed the initial prompt-only approach missed 11 out of 28 edge cases. I added two layers:

Layer 1: Deterministic phrase matching. Substrings like “broken,” “refund,” “angry,” “off-topic” trigger immediate review.

Layer 2: Intent gate (cheap LLM call). Before the main handler, one small LLM call answers: “Is this message a clear fit for this tool?” For example:

• A query routed to item_details_tool asking "do you have a discount?" is unclear → flag it.
• A query asking “tell me the story of the Spring collection” routed to collection_inquiry_tool is clear → proceed.

This isn’t re-routing (routing already happened upstream). It’s a one-question filter that catches vague, off-topic, or wrong-fit queries before expensive handler logic runs. Combined with Layer 1, it improved flagging accuracy from 17/28 to 25/28 test cases passing.

Stateless Architecture & Conversation Memory

Although the agent itself is stateless — Claude has no memory between API calls — I added a conversations table to Supabase to simulate memory across time. This allows the agent to:

• Reply to follow-ups and understand context from previous messages
• Match incoming replies to their conversation history regardless of how the email client threads messages
• Persist conversations separately from email threading, so customers can reach out about the same order via different email chains and the agent still recognizes the context

IMAP Polling for Follow-Up Emails

I extended the system to reply not just to form submissions but also to direct emails and customer follow-ups. Rather than using the Gmail API Push (which requires OAuth and Google Cloud setup), I chose IMAP polling with a 30-minute window (6000 seconds) because:

• Low traffic (10 requests/day maximum) meant polling was sufficient
• Pure Python implementation with imaplib — no complex OAuth or Google Cloud Console setup
• Email is asynchronous anyway; a 30-minute delay is acceptable and feels more human than instant replies

The gmail_listener.py module runs in the background, checking for unread emails, extracting the sender, subject, body, and thread ID, and orchestrating replies via the same routing pipeline as form submissions.

Handling Edge Cases & Human Escalation

Early testing revealed that some queries should never be answered automatically:

• Customers expressing frustration (“This is broken”)
• Requests outside the scope of the business
• Ambiguous or vague questions that need Olivia’s direct judgment

I implemented a human review flag in the database. Any conversation marked with status = "needs_human" is skipped by the agent entirely. Instead of sending an auto-reply, the system notifies Olivia so she can respond directly.

Handler Output Schema: Every tool now returns a structured response:

json

{
  "status": "answered" | "needs_clarification" | "needs_human",
  "message": "customer-facing reply or empty",
  "clarifying_question": "follow-up question or null",
  "missing_slot": "item_name | collection_name | ... or null"
}

This allows the orchestrator to handle three distinct outcomes: answer the customer, ask a clarifying question, or escalate to Olivia. If status = "needs_human", the agent saves the conversation state, notifies Olivia, and sends no auto-reply.

Context Grounding: Avoiding RAG Complexity

Halfway through development, I added unstructured data to Supabase:

• Collection stories: Rich narratives describing the mood, style, and aesthetic of each collection
• Care guides: Per-material care instructions (Stoneware, Earthenware, Porcelain)
• Artist notes: Olivia’s personal writing about her work, process, and commission approach
• FAQs: Common questions and answers
• Editorial picks: Olivia’s curated recommendations by occasion (housewarming, wedding, birthday, self-treat, cook, dinner set)

I deliberately did not use RAG because these files are small (~3500 tokens total), change rarely, have low accuracy stakes, and don’t require source attribution. Instead, I fetch all unstructured context upfront and pass it alongside the query to Claude. For a 9-item catalog with 3 collections, this is cheaper than vector embeddings or a vector database.

How context is used by each handler:

• item_details_tool: Fetches care guides and collection stories so Claude can answer care and collection context questions accurately
• returns_enquiry_tool: Fetches FAQs so Claude can answer general returns policy questions without unnecessarily asking for order details
• shipping_enquiry_tool: Fetches FAQs so Claude can answer general shipping questions (delivery times, packaging, tracking)
• custom_order_enquiry_tool: Fetches FAQs and artist notes so Claude can answer commission process questions in Olivia’s own voice
• recommendation: Fetches live catalog data, collection stories, and editorial picks to handle both style-based requests (“something boho”) and occasion-based requests (“wedding gift”)

Prompting for Recommendations

The recommendation handler started with a major failure mode: the agent kept asking follow-up questions instead of making recommendations. After analyzing 50 test queries, I identified the root cause and restructured the prompt.

The key principle: Line-by-line prompting works much better than paragraph prompts. Explicit rules beat general guidance.

Before: “Consider the customer’s style, budget, and occasion when recommending items.”

Before : A lot of questions!

After:

After : One good clarifying question with recs

VAGUE REQUEST (no style, no occasion, no budget):
→ Ask ONE clarifying question maximum — ever. Not per turn, total.
→ If the user has already answered any question at all, STOP asking.
→ Make your best recommendation from what you know.

FOLLOW-UP REPLIES (user has already answered a question):
→ Do not ask another question under any circumstances.
→ Take whatever information the user gave and make your best
   recommendation immediately.
→ If information is still incomplete, use the collection stories
   to fill the gaps with your own judgment.
→ "Warm and earthy, open to anything" is enough to recommend
   the Fall or Spring collection pieces — just do it.

This change reduced unnecessary clarifying questions by 68% and improved recommendation quality from 58% “Great” to 89% “Great” in testing.

The prompt also explicitly teaches Claude to handle different query types:

• Specific requests: Match directly from the catalog (“I need a vase under $70”)
• Style requests: Use collection stories to match the aesthetic (“boho room” → Spring collection)
• Occasion requests: Check editorial picks first (“housewarming gift” → exact match in curated list)
• Functional sets: Suggest items that work together (“dinner set” → platter + bowl + plate combination)

Observability via Helicone

The system makes multiple Claude API calls per request: intent classification, tool extraction, handler execution, synthesis. Without visibility into what prompts were sent, what came back, and where latency was sitting, debugging would be guesswork.

I chose Helicone for LLM-specific observability with zero architectural overhead. One configuration change instrumented every Claude call across the entire project. For each call, I passed custom headers (handler name, intent, thread ID, etc.) so traces can be filtered and grouped by context. This gives me instant visibility when accuracy drops or latency spikes — I can see exactly which handler and which intent caused it.

Every handler returns the same schema:

json

{
  "status": "answered" | "needs_clarification" | "needs_human",
  "message": "customer-facing reply or empty",
  "clarifying_question": "follow-up question or null",
  "missing_slot": "item_name | collection_name | ... or null"
}

This structure lets the router orchestrate multi-intent queries cleanly: if two handlers both return status=answered, the synthesis step combines them. If one returns needs_clarification, the router sends that question back instead of the full answer.

Testing & Evals: From Manual Labels to LLM-as-Judge

I started with 10 hand-curated queries and manually labelled agent outputs. That was slow. After 50 test queries, I built an eval framework that scaled.

Code-Based Evals (Deterministic)

Routing accuracy: Does the classifier pick the right intent?

• Baseline: 76% (ambiguous queries like “boho items under $50” triggered both recommendation + collection_inquiry)
• After qualitative labels + routing logic: 97.3%

Factual accuracy (structured data): Does the agent return the correct price, material, or dimension?

• Test cases: 9 item inquiries with expected structured facts
• Result: 26/28 passing (93%)
• Failures: One piece not found in Spring collection (DB inconsistency), one care guide missing for Earthenware

Flagging accuracy: Does the agent correctly flag ambiguous/off-topic queries?

• Round 1 (prompt only): 17/28 passing (61%)
• After intent gates + deterministic layer: 25/28 passing (89%)

LLM-as-Judge Evals (Qualitative)

I evaluated unstructured output (tone, completeness, recommendation quality) using Claude as a judge. The key insight from the OpenAI/Braintrust paper: penalize superset hallucinations (adding false details) as harshly as contradictions.

Hallucination types detected:

• Type 1 (Superset): Agent says “Rain Song Vase comes in three sizes,” but only one exists → flag
• Type 2 (Contradiction): Agent says “$95,” but FAQ says “$68” → flag
• Type 3 (Wrong properties): Agent says “dishwasher safe,” but care guide says “hand wash” → flag

Judge prompt scoring (A/B/C/D/E):

• A: Subset (incomplete but accurate)
• B: Superset with accurate details
• C: Complete match
• D: Disagreement (hallucination)
• E: Different phrasing, factually equivalent

Initial run showed 17% hallucination rate. On review, most were Type 1 (superset: agent adding extra context). I added output length constraints: “Keep answers under 150 words, stick to documented facts.” Hallucination rate dropped to 3%, all benign (first-person vs third-person phrasing, which I explicitly allowed in the judge prompt).

Criteria scores (after prompt iteration):

• Tone accuracy: 88% Great, 12% Average (final pass: 95% Great)
• Completeness: 76% Great, 18% Average (final: 92% Great)
• Clarifying question quality: 64% Great, 28% Average (final: 87% Great)
• Recommendation quality: 58% Great, 32% Average (final: 89% Great)

Performance Metrics & Observability

Using Helicone, I traced every API call through the system.

Latency (25 production-like queries):

• Average: 1515ms (1.5 sec)
• p50: 1110ms
• p95: 2971ms
• Range: 498ms — 2994ms

Recommendation handler is slowest (~2.3s) because it does two LLM calls (classifier + synthesis). Item inquiries are fastest (~750ms) because database lookups are instant. This is acceptable for async email; sync chat would need optimization.

Cost:

• Per query: $0.0025 (using Claude Haiku)
• Total for 25 queries: $0.063
• Monthly estimate (15 queries/week = ~65 queries/month): $0.16

Olivia’s entire system costs less than a single human-written email.

Token efficiency: After prompt expansion (added recommendation context, intent gates, hallucination checks), average prompt length increased slightly but per-query cost held steady. Each handler’s system prompt is 1.5–4KB; the main orchestrator is 800 bytes. Total system prompt footprint: ~12KB, negligible in practice.

Key Learnings & Trade-Offs

1. Qualitative Labels Beat Numeric Confidence

I tested three approaches:

ApproachProsConsSelf-reported scores (“0.75”)SimpleUnreliable routing, 24% multi-intent collision rateLogprobs (OpenAI)Real probabilities1.6x token cost, 4 parallel calls, new tuning problemQualitative labels (“high”/”medium”/”low”)Directional routing, explicit rules, worksSlightly higher prompt cost (negligible)

Chose qualitative. It’s cheap, interpretable, and works.

2. Intent Gates Are Cheap Insurance

A single 50-token LLM call before the main handler catches 8% of failure cases that Layer 1 (deterministic) misses. Cost: $0.0001 per call. ROI: huge.

3. Hallucination Is About Details, Not Disagreement

“Rain Song Vase” vs “Rain Song vase” isn’t a hallucination. Agent saying “comes in 3 sizes” when only 1 exists is. I added explicit rules to the judge: first-person = third-person, range info ≠ specific item prices, and emoji-heavy descriptions are OK if factually grounded.

4. Prompt Engineering > Fine-Tuning (For Now)

With 50 test queries and 9 catalog items, I don’t have enough data for fine-tuning to beat prompt iteration. At 500+ production conversations, fine-tuning the intent classifier would be worth revisiting. For now, prompt optimization wins.

5. Source Grounding Requires Explicit Rules

Early versions hallucinated contact info from the style sample email. I fixed it by:

• Stripping headers and metadata from the style sample
• Adding explicit rules: “Never include emails/URLs unless in catalog data”
• Validating handler outputs against allowed fields

Weekly Batch Evals & Deployment

I set up a GitHub Actions workflow that runs every Saturday at 5:36 PM PST:

1. Fetch all conversations from Supabase
2. Extract user queries from message history
3. Run criteria eval on each (tone, completeness, etc.)
4. Run hallucination eval on unstructured answers
5. Save results to /eval/weekly_batches/{YYYY-WNN}/
6. Email HTML report to Olivia

This gives Olivia a weekly scorecard without her lifting a finger. If accuracy drops, she knows immediately. If a specific handler starts failing, the report shows it.

Some Metrics

• 87% accuracy on a diverse 50-query test set
• 85% autonomous handling (no human escalation)
• 1.5 sec average latency acceptable for async email
• $0.16/month cost (saves Olivia 15+ hours/month)
• Zero tone drift (88–95% “Great” across all criteria)
• 3% hallucination rate (benign, mostly phrasing)
• Extensible architecture (adding a new intent = new handler + routing rule, no system redesign)