PixelRAG Proves That the Biggest Bottleneck in Web RAG Was the Text Parser All Along

Every retrieval-augmented generation pipeline in use today begins with the same quiet act of destruction. A document arrives, whether a web page, a PDF, or an internal knowledge base article, and the first thing the system does is strip away most of what makes that document meaningful. Tables are flattened into comma-separated rows. Charts become nothing. Page layout, which carries real semantic signal about how information is organized and which pieces are related to which, disappears entirely. Infographics, visual hierarchies, and multi-column structures become a jumbled linear sequence of tokens with no spatial context surviving the conversion.

This text extraction step is so deeply embedded in how retrieval systems are built that most practitioners treat it as infrastructure rather than as a design decision. But it is a decision, and according to new research from a team spanning UC Berkeley, Princeton University, EPFL, and Databricks, it may be the single most consequential source of wrong answers in web-based retrieval pipelines today. Their system, called PixelRAG, takes the opposite approach entirely. Rather than extracting text from documents and then working with the extracted text, it renders documents as screenshots and retrieves over the images directly. A vision-language model reads the answer straight off the visual tiles. HTML parsing is not simplified or improved in PixelRAG. It is eliminated.

The information loss problem in text-based web retrieval

Understanding why PixelRAG matters requires sitting with the magnitude of what standard HTML-to-text conversion actually discards. The problem is not that parsers are poorly implemented. It is that the conversion is fundamentally lossy in ways that cannot be fully recovered downstream.

HTML parsing may be one of the biggest self-inflicted bottlenecks in web RAG. Consider what happens when a retrieval system encounters a Wikipedia article containing a comparison table. The table has a clear two-dimensional structure, with column headers conveying one dimension of meaning, row labels conveying another, and the intersecting cells carrying the actual data. A human reading this table can instantly see which cell answers which question. An HTML-to-text parser converts this structure into a linear sequence of tokens, typically stripping the table formatting and leaving behind something that looks like a list of values with unclear relationships. The retrieval system that indexes this output is not indexing the table. It is indexing a degraded representation of the table that has lost most of the structural information that made the table useful.

Rather than parsing HTML or PDFs into raw strings, PixelRAG renders source material into screenshot tiles. Those tiles are then embedded into a visual index that preserves the original layout, tabular structures, and design signals that text parsers strip away. This is not a marginal improvement in information retention. It is a categorical difference in what the retrieval system is working with.

The problem compounds when considering that different HTML parsers make different choices about what to discard. Published research has found that swapping one parser for another on the same document collection can move accuracy by roughly ten percentage points on question answering benchmarks, with no other change to the pipeline. This means that a substantial fraction of the performance difference between retrieval systems that appear to differ on algorithm or model is actually noise introduced by the parser, and the research community building on these benchmarks has been measuring that noise without fully attributing it to its source.

What PixelRAG does instead

Instead of converting web pages into text, PixelRAG treats web pages like humans do: it takes screenshots, indexes the visual pages, retrieves image tiles, and feeds them directly into Vision-Language Models.

The pipeline has four stages that can be run independently or together. The first stage is rendering, handled by a tool called pixelshot that uses Playwright’s Chrome DevTools Protocol to capture a live rendered view of any URL or local file. For PDFs, the same tool renders each page as an image rather than extracting text. The output is a set of screenshot tiles that represent the document as it actually appears when a browser renders it, complete with all layout, visual hierarchy, and graphical content intact.

The second stage is embedding. A Qwen3-VL-Embedding model, LoRA-fine-tuned specifically on screenshot data from web pages, converts each image tile into a dense vector that captures the visual and semantic content of the screenshot. This fine-tuning step is what separates PixelRAG from simply running a generic image embedding model over page screenshots. The retrieval model was trained on a contrastive dataset synthesized from web page screenshots, using a pipeline that samples knowledge-intensive web pages, generates search queries with a language model, filters low quality examples, and mines hard negatives to create training pairs that teach the embedding model to distinguish between pages that answer a query and pages that do not.

The third stage is index construction, which builds a FAISS vector index over the embedded screenshot tiles. FAISS, the widely used approximate nearest neighbor search library from Meta AI Research, handles the scalability requirements: the Wikipedia index covering over eight million articles and thirty million screenshots uses a normalized index with efficient compressed representations that make both construction and querying tractable at this scale.

The fourth stage is serving, which provides a REST API that accepts text or image queries and returns the most relevant screenshot tiles ranked by similarity to the query. At query time, a vision-language model reads the retrieved tiles and generates the answer from what it sees in the images rather than from extracted text.

Installing and running PixelRAG

The entire pipeline is available as a Python package with optional extras that install only the stages a given deployment needs:

pip install pixelrag

The pixelshot command, which handles the rendering stage, is included in the base install. Rendering a web page to screenshot tiles requires one command:

pixelshot https://en.wikipedia.org/wiki/Python --output ./tiles

The pre-built Wikipedia index is available through the project’s hosted endpoint, which requires no API key and accepts both text and image queries:

curl -X POST https://api.pixelrag.ai/search \
  -H "Content-Type: application/json" \
  -d '{"queries": [{"text": "What is the capital of France?"}], "n_docs": 5}'

For teams that want to run the search API against their own downloaded copy of the index rather than the hosted endpoint, the full embed and serve pipeline can be installed and run locally:

pip install 'pixelrag[serve]'
huggingface-cli download StarTrail-org/pixelrag-faiss-indexes \
  --repo-type dataset --include "search_index_normed_v2/*" --local-dir ./index
pixelrag serve --index-dir ./index/search_index_normed_v2 --port 30001

Building a visual index over a custom document collection involves creating a configuration file and running the index build command:

pip install 'pixelrag[index]'
cat > pixelrag.yaml << 'EOF'
source:
  type: local
  path: ./my_docs
embed:
  model: Qwen/Qwen3-VL-Embedding-2B
  device: cuda
  gpu_ids: [0]
output: ./my_index
EOF
pixelrag index build
pixelrag serve --index-dir ./my_index --port 30001

Each stage of the pipeline can also be run independently without the orchestrator, which is useful for environments where only part of the pipeline needs to be updated:

pip install 'pixelrag[embed]'
pixelrag chunk --tiles-dir ./tiles
pixelrag embed --shard-dir ./tiles --output-dir ./embeddings --gpu-ids 0,1
pixelrag build-index --embeddings-dir ./embeddings --output-dir ./index

Rendering can also be called programmatically from within a Python application, which enables integration into agent frameworks and custom data processing pipelines:

from pixelrag_render import render_url
tiles = render_url("https://en.wikipedia.org/wiki/Python", "./tiles")

The Wikipedia index and what it demonstrates

The most ambitious demonstration of PixelRAG’s scalability is the pre-built index of all of Wikipedia. We built the first visual index covering all of Wikipedia: 30M+ webpage screenshots. At 8.28 million articles rendered to screenshot tiles and indexed as visual embeddings, this is the largest visual retrieval index publicly demonstrated, and it is served through the project’s hosted API endpoint with no authentication required.

Across six different question-answering benchmarks, PixelRAG consistently outperformed text-based RAG systems, with accuracy gains as high as 18.1%. The fact that these gains appear on text-only question answering benchmarks, where the questions have text answers embedded in the documents, is the counterintuitive result that makes the research most compelling. Visual retrieval is outperforming text retrieval not only on questions that require reading a chart or interpreting a table, but on questions where the answer is a straightforward text string embedded somewhere in the document. The reason is that the visual index preserves structural context around that text string: where it sits on the page, how it is formatted, what surrounds it, and what its relationship is to nearby headings and labels. All of that context helps the retrieval model identify the right document, and it helps the reader model find the right span within the retrieved tile.

The agent cost reduction finding adds another dimension to the performance story. PixelRAG also cuts AI agent token costs by up to 10x compared to legacy pipelines. Screenshots provide a far more compact representation of a web page than the full extracted text, because a screenshot captures the rendered output at a fixed resolution rather than including all the HTML structure and boilerplate that surrounds the actual content. When an agent retrieves a screenshot tile containing the relevant information, it processes fewer tokens to read the answer than it would when processing a chunk of extracted text containing the same information plus surrounding context. At the scale of agentic workloads, this token efficiency difference accumulates into a significant cost reduction.

The scaling property that makes this fundamentally different

One of the most practically significant properties of PixelRAG’s architecture is the decoupling between the retrieval index and the reader model. In a text-based RAG system, improving the reader model while keeping the index fixed provides limited returns, because the quality of the index representation is a ceiling on what any reader model can extract from it. If the index has discarded the table structure, no reader model can reconstruct it from the text chunks.

Text RAG stays flat. PixelRAG rides the VLM scaling curve. A PixelRAG index stores the raw pixel content of the documents. As vision-language models improve, which they have been doing rapidly, the same index becomes more capable of answering questions, because the reader is operating on richer input. A stronger reader model lifts accuracy with no re-indexing, since the index contains the same visual information it always did. The investment in building and maintaining the index retains and increases its value over time rather than becoming obsolete when the reader model is upgraded.

This scaling property also has implications for the economics of maintaining a retrieval system. Re-indexing a large document collection is expensive and time consuming. A system where upgrading the reader model requires re-indexing has a significantly higher maintenance cost than one where the reader can be swapped independently. PixelRAG’s architecture makes the costly operation, index construction and embedding, a one time investment rather than a recurring cost tied to model upgrades.

Giving AI agents visual perception of the web

Beyond the retrieval use case, the project ships a Claude Code plugin called pixelbrowse that gives coding agents direct visual access to rendered web pages rather than requiring them to parse raw HTML. PixelRAG renders documents as screenshots and retrieves over the images directly. Visual structure that HTML parsing throws away stays intact, so the reader model can actually answer questions about it.

Installing the plugin requires two steps after the base package is installed:

pip install pixelrag
claude plugin marketplace add StarTrail-org/PixelRAG
claude plugin install pixelbrowse@pixelrag-plugins

After installation, an agent session can screenshot and read any URL by stating a natural language goal:

claude -p "screenshot https://news.ycombinator.com and summarize the top stories"
claude -p "screenshot https://arxiv.org/abs/2404.12387 and explain the key findings"

Inside an interactive Claude Code session, the slash command invokes the same capability:

/screenshot https://example.com

The plugin works by calling pixelshot locally through Playwright and the Chrome DevTools Protocol, producing a rendered screenshot that the agent reads as an image rather than processing raw HTML. There is no MCP server, no backend service, and no additional infrastructure required beyond the pixelshot command that ships with the base package. For an agent working with a live web page, an arXiv paper, a documentation site, or any other rendered document, this approach gives it the same visual understanding a human would have when opening the same URL in a browser.

This is meaningful in agentic contexts for the same reason it is meaningful in retrieval contexts: the information a human sees when looking at a rendered page is not fully recoverable from the HTML source. A documentation site that uses visual layout to distinguish primary API methods from deprecated ones, a news site that uses visual hierarchy to distinguish headlines from subheadings, or a research paper that presents findings in a table are all examples where a human reading the rendered page understands something that an agent parsing the raw HTML might miss.

The training pipeline and what was released

The retrieval model that powers PixelRAG is a LoRA adaptation of Qwen3-VL-Embedding-2B, trained on a dataset of web page screenshots with synthetic contrastive examples. The training pipeline is published as a separate project with its own dependency environment, using pinned versions of PyTorch, Transformers, and CUDA libraries to ensure reproducibility. The trained adapters are published at Chrisyichuan/wiki-screenshot-embedding-lora on Hugging Face, and the full training dataset is published at Chrisyichuan/screenshot-training-natural-filtered-v2, which means other research groups can use the same data to fine-tune different embedding models, including larger Qwen variants or other visual embedding architectures, without needing to reproduce the data generation pipeline from scratch.

This level of openness in releasing both the trained model and the training data is notable in the current research environment, where many high performing models are released with closed training pipelines. It allows the community to study what properties of the training data drove the performance gains, to experiment with architectural variations, and to adapt the approach to new domains beyond web pages and Wikipedia.

Where visual RAG fits in the broader retrieval landscape

PixelRAG’s own authors point to hybrid deployment as the most practical near-term path, layering visual retrieval on top of existing text systems rather than replacing them. For teams already running RAG pipelines, the path toward capturing the benefits of visual retrieval does not necessarily require a full infrastructure replacement. A hybrid approach that uses visual retrieval for documents with rich visual structure, such as tables, charts, and multi-column layouts, while retaining text retrieval for documents where plain text extraction is lossless, captures most of the accuracy gain while minimizing the operational disruption.

VB Pulse Q1 2026 data from qualified enterprise respondents found intent to adopt hybrid retrieval tripling from 10.3% in January to 33.3% in March, the fastest-growing strategic position in the dataset. This acceleration suggests that the research community’s findings about text parser limitations are being absorbed into production deployment decisions. The practical bottleneck for most teams is not awareness that visual information is being lost in text extraction. It is having a practical, scalable, open source pipeline for doing something about it, which is precisely what PixelRAG provides.

The project complements related work in the visual document understanding space. Research on ColPali demonstrated that document page images could be embedded and retrieved directly at the page level, while work on VisRAG extended this approach to question answering. PixelRAG operates at the tile level rather than the page level, which provides finer grained retrieval that reduces the amount of irrelevant visual content passed to the reader model, and it focuses specifically on the web-scale retrieval use case that makes the Wikipedia index demonstration meaningful.

Conclusion

PixelRAG makes a compelling empirical case that the first step of nearly every existing web retrieval pipeline is also its biggest source of error. By rendering documents as screenshot tiles rather than extracting text, embedding those tiles with a vision-language model trained on screenshot data, and letting the reader model work from what the page actually looks like rather than a degraded text approximation, the system achieves accuracy gains that beat text-based retrieval even on benchmarks where all answers are text strings.

The thirty million screenshot Wikipedia index demonstrates that the approach is not just theoretically appealing but practically deployable at the scale of the entire world’s largest reference resource. The open source release under the Apache 2.0 license, the published training adapters, the public training dataset, and the Claude Code plugin for giving agents direct visual web access together make this one of the most complete and practically usable open releases in the visual retrieval space.

For anyone maintaining or building a retrieval pipeline that works with web pages, PDFs, or any document with meaningful visual structure, the question this research raises is difficult to ignore: how much accuracy is being left on the table by the parser that runs before anything else?

The repository is available at: https://github.com/StarTrail-org/PixelRAG