Build Your Own Perplexity in 100 Lines of Python

Perplexity crossed 100 million users. People treat it like magic.

It isn’t. At its core, it runs a pipeline that any engineer can replicate in an afternoon. I know because I built one and in this post I’ll walk you through exactly how it works, including the async trick that makes it fast, and the reranking step that makes it accurate.

Here’s what we’re building:

User Query 
↓ 
Generate 5 sub-questions [Gemini] 
↓ 
Search each in parallel [DuckDuckGo] 
↓ 
Scrape top URLs in parallel [Trafilatura] 
↓ 
Chunk → Deduplicate → Rerank [Cohere] 
↓ 
Generate answer with citations [Gemini]

Let’s go step by step.

When a user types “give me in full detail how ML and AI is different”, searching that exact phrase gives you 3 mediocre results.

Instead, we generate 5 related sub-questions first:

• What are the key differences between AI and ML?
• Is machine learning a subset of artificial intelligence?
• How does ML differ from traditional AI rule-based systems?
• What are real-world examples where AI and ML overlap?
• How do objectives differ between AI systems and ML models?

Now you’re covering the topic from 5 angles. Each question fetches 3 URLs. That’s 15 data sources for one query which is exactly the multi-query retrieval strategy that makes Perplexity answers feel comprehensive.

The gotcha: Don’t just generate random questions. They need to be semantically diverse covering definition, relationship, methodology, examples, and edge cases. Prompt your LLM explicitly for this. Vague question generation = redundant search results = wasted tokens.

Here’s where most tutorials fall apart.

DDGS (DuckDuckGo Search) and Trafilatura (web scraper) are both blocking libraries. They don't support async/await natively.

If you run them directly in an async function, you block the entire event loop. All 5 searches run one after another. With 15 network calls total that’s painfully slow.

The fix is run_in_executor. It pushes a blocking function into a thread pool and gives you back an awaitable so your event loop stays free:

executor = ThreadPoolExecutor(max_workers=10)

async def search_question_async(question: str):
    loop = asyncio.get_event_loop()

    # Blocking library → thread pool → awaitable
    results = await loop.run_in_executor(executor, _ddgs_search, question)

    # Scrape all URLs concurrently
    scrape_tasks = [
        loop.run_in_executor(executor, _trafilatura_fetch, r["href"])
        for r in results if r.get("href")
    ]
    scraped_texts = await asyncio.gather(*scrape_tasks)

This pattern works for any blocking library old Redis clients, legacy HTTP wrappers, database drivers that predate asyncio. Whenever you’re stuck with a sync library in an async codebase, run_in_executor is your escape hatch.

The gotcha: Don’t set max_workers too high. With 50 threads hammering DuckDuckGo, you'll get rate-limited fast. 10 is a safe ceiling for search + scrape combined.

After scraping, we ran our pipeline on a real query: “give me in full detail how ML and AI is different from each other”

The raw numbers:

Total chunks after splitting:  212
Unique chunks after dedup:     199
Top chunks after reranking:     15
Total tokens in context:      2014

Three separate operations here — each one matters.

Chunking splits scraped pages into 400-token pieces with 50-token overlap. Why overlap? Because answers often span chunk boundaries. Without overlap, you lose context at every split point. The chunk size of 400 is intentional small enough that each chunk is focused on one idea, large enough that it has enough context to be useful.

Deduplication removes chunks that are either from the same URL+position, or have identical content hashes. We went from 212 → 199 that’s 13 duplicate chunks removed. In production, the same content often appears across mirror sites and aggregators. Without dedup, your LLM sees the same sentence three times and treats it as stronger evidence than it actually is.

Reranking is where the real quality gate is. We have 199 unique chunks — but we can only send so many tokens to the LLM. Cohere’s reranker scores every chunk against the original query and returns the top 15:

def rerank_chunks(query: str, chunks: list[dict], top_n: int = 10):
    response = co.rerank(
        model="rerank-v3.5",
        query=query,
        documents=[c["text"] for c in chunks],
        top_n=min(top_n, len(chunks)),
    )
    return [chunks[hit.index] for hit in response.results]

Without reranking, you’re feeding the LLM whatever chunks happen to contain matching keywords. With reranking, you’re feeding it the 15 most semantically relevant chunks. The difference in answer quality is significant — especially for nuanced queries where keyword overlap doesn’t equal relevance.

The gotcha: Reranking costs money and adds latency. For a prototype, it’s fine. In production, cache reranked results for repeated queries. The same question asked 1000 times shouldn’t hit Cohere 1000 times.

With 2014 tokens of clean, reranked context, we call Gemini twice:

Two-pass generation matters here. A single call asking for “answer + citations” often produces hallucinated citations — the model invents sources that sound plausible. Separating the steps forces grounding in the first pass, then formatting in the second.

The output for our ML vs AI query came back structured across five sections: Definition and Scope, Relationship, Objectives, Methodology, and Real-World Application Overlap — each sentence cited to its source document.

That’s the same structure Perplexity uses. Not magic. Just a well-designed prompt.

To be honest about what this prototype isn’t:

The core loop -> search, retrieve, rank, generate -> is identical. The difference is engineering polish, not architecture.

The complete implementation is on GitHub: https://github.com/vishwajeetvishwakarma/AgentShortProjects/tree/main/perplexity_workflow

Stack used: Gemini Flash (question gen + answer gen), DuckDuckGo Search via ddgs, Trafilatura (scraping), LangChain text splitter (chunking), Cohere rerank-v3.5 (reranking), tiktoken (token counting).

Next post: I built a hallucination detection layer on top of this pipeline because reranking gets you good context, but it doesn’t guarantee a grounded answer. Here’s how to catch it before your user does. Subscribe so you don’t miss it.

Originally published at https://vishwajeetv2003.substack.com.