LLMCYCLE

Building a Production-Ready Multi-Provider LLM App with llmcycle

From battling 2 AM rate limits to achieving zero‑downtime LLM streaming

The 2 AM Wake‑Up Call

Picture this: your AI product is gaining traction, users are flooding in, and then it happens. A key provider hits its rate limit — at 2 AM. Your application crashes. By the time you wake up, you’ve lost hours of uptime and frustrated countless users.

This nightmare scenario inspired Bishwajit Garai to build llmcycle — a production‑grade universal LLM routing framework that handles the chaos of multi‑provider AI infrastructure so you don’t have to.

🔗 Resources — PyPI: https://pypi.org/project/llmcycle/ — GitHub: https://github.com/Bishwajitgarai/llmcycle

Why Most LLM Routers Fall Short

• ❌ Single key per provider — When one key hits 429, the entire provider goes dark.
• ❌ No mid‑stream recovery — A connection drop at token 1800 out of 2000 means starting from scratch.
• ❌ Manual provider configuration — Every SDK has different auth headers, error formats, and quirks.
• ❌ Zero observability — No dashboard, no analytics, no way to see which keys are healthy.

LLMCycle was built to fix all of this — and it’s completely open source (MIT licensed).

What Makes LLMCycle Different

• 🔑 Unlimited multi‑key rotation — Automatic round‑robin across keys. Rate limit? Rotate instantly.
• 🔄 Per‑key cooldown + auto‑recovery — 429 triggers cooldown for that key only, not the whole provider.
• 🌊 Seamless mid‑stream failover — If a stream dies at token 1800, llmcycle resumes from another provider. The user never notices.
• 🚀 Provider auto‑discovery — Drop keys into .env, llmcycle finds them. No boilerplate.
• 📊 Real‑time FastAPI dashboard — Run llmcycle ui to see key health, latency, token usage.

How LLMCycle Compares

| Feature | LLMCycle | LiteLLM | LangChain | OpenAI SDK | Portkey | | — — — — — — — — — — — — — — — — — | — — — — — | — — — — -| — — — — — -| — — — — — — | — — — — -| | Multi‑key per provider | Unlimited| ❌ | ❌ | ❌ | ✅ Paid | | Auto key round‑robin | ✅ | ❌ | ❌ | ❌ | ✅ Paid | | 429 per‑key cooldown + recovery | ✅ | Basic | ❌ | ❌ | ✅ Paid | | 401 → auto disable key | ✅ | ❌ | ❌ | ❌ | ❌ | | Mid‑stream failover | ✅ | ❌ | ❌ | ❌ | ❌ | | Provider auto‑discovery from .env| ✅ | ❌ | ❌ | ❌ | ❌ | | Web dashboard | ✅ | ❌ | ❌ | ❌ | ✅ Paid | | Fully open source & free | ✅ MIT | ✅ MIT | ✅ MIT | ✅ MIT | ❌ Freemium |

Building a Real Project: Multi‑Provider Research Assistant

Let’s build an async research assistant that queries multiple LLMs in parallel, gathers diverse perspectives, and synthesizes the final answer.

Prerequisites

• Python 3.9+
• API keys from Groq, Together AI, OpenRouter (free tiers work)

Installation

pip install llmcycle python-dotenv

Configuration

Create a .env file:

GROQ_API_KEYS=gsk_your_groq_key_here TOGETHER_API_KEYS=tg_your_together_key_here OPENROUTER_API_KEYS=sk-or_your_openrouter_key_here

Complete Application Code

Create a file main.py with the code below. (The code is complete and tested.)

import asyncio import time from llmcycle import LLMCycle from llmcycle.core.router import RoutingStrategy

async def get_llm_response(client, model, prompt, source): print(f”Requesting from [{source}]: {model}…”) start = time.perf_counter() try: resp = await client.complete( model=model, prompt=prompt, cache_ttl=300, strategy=RoutingStrategy.ROUND_ROBIN ) elapsed = (time.perf_counter() — start) * 1000 print(f”Success from {model} ({elapsed:.1f}ms)”) return { “source”: source, “model”: model, “content”: resp.content, “latency_ms”: elapsed } except Exception as e: print(f”Failed from {model}: {e}”) return None

async def generate_final_answer(client, question, partial_responses): print(“\nGenerating final answer…”) synthesis_prompt = f””” You are a research assistant synthesizing information.

Original Question: {question}

Research Notes from Different Models: {chr(10).join([f”- [{r[‘source’]} via {r[‘model’]}]: {r[‘content’]}” for r in partial_responses if r])}

Task: Based on the research notes above, provide a final, well-structured answer. If there are disagreements, note them and provide the most likely correct information. “”” final = await client.complete( model=”openrouter/meta-llama/llama-3.1–70b-instruct”, prompt=synthesis_prompt, cache_ttl=0 ) print(“Final answer ready”) return final.content

async def main(): print(“LLM Research Assistant”) question = input(“Enter your research topic: “)

client = LLMCycle( groups={ “research_tier”: [ “groq/llama-3.1–70b-versatile”, “together_ai/meta-llama/Llama-3.1–70B-Instruct-Turbo”, “openrouter/meta-llama/llama-3.1–8b-instruct” ] }, fallbacks={“openrouter”: [“groq”, “together_ai”]}, cache=True, guardrail=True )

models_to_query = [ (“openrouter/gryphe/mythomax-l2–13b”, “OpenRouter (MythoMax)”), (“together_ai/meta-llama/Llama-3.1–8B-Instruct-Turbo”, “Together AI (Llama 3.1 8B)”), (“groq/llama-3.1–70b-versatile”, “Groq (Llama 3.1 70B)”) ]

research_prompt = f””” Please provide a detailed, factual answer to: “{question}” Focus on key concepts, recent developments, and practical implications. “””

print(“\nGathering initial research in parallel…”) tasks = [get_llm_response(client, model, research_prompt, name) for model, name in models_to_query] results = await asyncio.gather(*tasks) successful = [r for r in results if r is not None]

if not successful: print(“All providers failed. Check your API keys.”) return

final_answer = await generate_final_answer(client, question, successful)

print(“\n” + “=”40) print(“RESEARCH SUMMARY”) print(“=”40) for r in successful: print(f”\nSource: {r[‘source’]}”) print(f”Model: {r[‘model’]}”) print(f”Latency: {r[‘latency_ms’]:.1f}ms”) print(f”Preview: {r[‘content’][:300]}…”)

print(“\n” + “=”40) print(“FINAL SYNTHESIZED ANSWER”) print(“=”40) print(final_answer)

print(“\nProvider Health:”) for provider in client.get_providers(): stats = client.key_manager.key_count(provider) print(f” {provider}: {stats[‘active’]}/{stats[‘total’]} keys active”)

await client.close()

if name == “main”: asyncio.run(main())

Running the Project

1. Save the code as main.py.
2. Run python main.py
3. Enter a research question, e.g., “What are the key benefits of a multi-provider LLM router?”
4. Watch as the assistant queries three providers in parallel and synthesizes the final answer.

What This Demonstrates

• Parallel execution — queries three models simultaneously.
• Automatic failover — if one provider fails, others still work.
• Caching — repeated questions answered from cache (cache_ttl=300 seconds).
• Guardrails — PII and API keys are masked automatically.
• Key health dashboard — shows active vs total keys per provider.

Next Steps

• Add a web interface (Gradio / FastAPI).
• Use the built‑in dashboard: llmcycle ui
• Extend with Redis for distributed caching.

Conclusion

LLMCycle is not just another LLM wrapper. It’s a production‑ready router that brings enterprise‑grade reliability to your AI applications — for free. If you’re tired of 2 AM alerts and brittle provider integrations, give llmcycle a try.

Star the repo, share the post, and build something resilient.