The Needle in a Gigapixel Haystack: Rethinking Pathological Report Generation with BiGen

In the world of computational pathology, Whole Slide Images (WSIs) present a unique paradox: they are visually massive, often reaching gigapixel resolutions, yet the information critical for a definitive diagnosis is incredibly sparse. While deep learning has made strides in image analysis, vision-only models often struggle to bridge the gap between raw pixels and the high-level semantic concepts — such as specific cancer subtypes, grading, or molecular markers — that pathologists use every day.

A recent piece of work addresses this by grounding its architecture in the actual diagnostic workflow. Pathologists don’t scan every micron of a slide with equal intensity; they rapidly home in on “regions of interest” and subconsciously draw upon a mental library of similar historical cases to frame their findings.

This intuition is the engine behind BiGen, a dual-branch parallel learning framework. While the end goal remains the standard translation of a WSI into a textual sequence, BiGen fundamentally changes how the intermediate representations are learned.

Distilling the Visual Chaos

The first challenge is noise. Instead of forcing a model to perform self-attention across thousands of image patches — which is computationally expensive and noisy — BiGen introduces a learnable visual token. This token acts as a query in a cross-attention mechanism, aggregating features from all patches into a highly compressed global visual representation.

The byproduct of this step is a set of attention scores. These scores are used to filter the “Top-K” most critical patches, ensuring that the model’s “eyes” stay fixed on diagnostically relevant tissue while ignoring the surrounding “white space” or stroma.

Retrieval-Augmented Clinical Memory

Once the key patches are identified, the model shifts from pure vision to semantic alignment. These high-attention patches are fed into PLIP (a vision-language foundation model specifically pretrained on medical data) to obtain embeddings already aligned with the text space.

These patches are then spatially aggregated into “tissue regions” — essentially partitioning the sorted patch sequence into fixed groups and averaging them. But here is where the model gains its “experience”: each region is used to query a massive knowledge base of historical pathological reports. By retrieving the Top-V most relevant descriptions from past cases, the model builds a “knowledge feature” for every region.

The Power of Shared Weights

To ensure the visual and textual information speak the same language, the authors introduce a learnable text token that performs a similar compression on the retrieved historical descriptions. Interestingly, the cross-attention layers in both the visual and textual branches share weights, enforcing a strict alignment between the two modalities.

In the final decoding stage, the heavy lifting is already done. The generator doesn’t struggle with thousands of raw patch vectors; instead, it receives just two distilled vectors: the visual token and the text token. These serve as the Key and Value for the decoder’s cross-attention layers to produce a coherent, expert-level report.

From Pixels to Prose

The entire system is trained via supervised learning, using a dataset of WSIs paired with original reports written by professional pathologists. By mimicking both the “selective looking” and “historical referencing” of human experts, BiGen doesn’t just improve standard text metrics — it demonstrates a tangible gain in performance for downstream clinical tasks. It suggests that the future of medical AI isn’t just about seeing more data, but about seeing it through the lens of established clinical wisdom.

| Find papers faster on arXivSub with AI summary (CVPR/ICCV/ICML/ICLR/NeurIPS/AAAI/MICCAI)