Bridging Vision and Language: My GSoC 2025 Journey Building the Multimodal Embedding Explorer (MMEE)

How I spent my summer visualizing intelligence, detecting noisy captions, and learning to see AI differently.

The Beginning — Seeing the Invisible

When I first encountered CLIP — OpenAI’s model that can understand both images and text — I was fascinated by one question:

“If models can understand both words and pixels, can we understand what they see?”

CLIP encodes each image and caption into a 512-dimensional vector, representing their meaning in mathematical form. These vectors are powerful — they capture semantics, relationships, and context. But to humans, they’re invisible: just sequences of numbers.

During my GSoC 2025 journey with OpenVINO, I wanted to make that invisible world visible.

That curiosity became my proposal: MMEE — the Multimodal Embedding Explorer, a tool to visualize, analyze, and debug multimodal embeddings interactively. My goal was to help researchers and engineers see how well image and text embeddings align — and where they break.

From Idea to Proposal

When I first wrote my proposal, the idea seemed deceptively simple: “Visualize embeddings in 2D.” But as I dug deeper, I realized this project touched one of AI’s hardest open questions — how do we represent meaning jointly across modalities?

Figure 1: Contrastive pre-training setup from CLIP. Each image–text pair is encoded independently by an image encoder and a text encoder, and their embeddings are aligned through a contrastive loss computed over all pairwise similarities in the batch. Source: Radford et al., “Learning Transferable Visual Models From Natural Language Supervision,” OpenAI, 2021.

Understanding the Joint Embedding Space

At the heart of multimodal AI lies the idea of a joint embedding space — a shared mathematical world where both images and text can coexist.

Models like CLIP (OpenAI, 2021), ALIGN (Google, 2021), and LiT (2022) learn this space using contrastive learning: pulling matching image-text pairs closer and pushing unrelated ones apart. These models are remarkably effective, and research around how to build such spaces has been extensive.

But as I began working on MMEE, I realized something:

While embedding alignment is well-studied, understanding and visualizing how these embeddings behave — especially when they go wrong remains an open problem.

When we talk about “joint embedding,” we often assume both modalities align perfectly. Yet, in practice, alignment drifts. Text embeddings often scatter more widely than image embeddings. Semantically similar captions may end up far apart because of subtle wording differences. And when you introduce noisy or mislabeled captions, the geometry of the space becomes even more chaotic.

This is where visualization becomes critical — not as a pretty chart, but as a diagnostic tool. The paper “Visualizing CLIP Models from Natural Language Supervision” (2022) touches on this — showing how even state-of-the-art models exhibit local distortions, cluster overlap, and modality imbalance. It inspired me to treat embedding visualization not as an afterthought, but as the main lens of understanding.

FIgure 2: Embeddings representing multi modal and combining them into one joint vector space to show similar relationships

So I built MMEE to make that invisible geometry visible to show, in real time, how images and captions align, cluster, and deviate.

Bringing these embeddings into a truly shared space wasn’t easy. At first, image and text clusters floated apart — like two constellations slightly misaligned. To fix this, I used Orthogonal Procrustes alignment, which computes a rotation matrix that best aligns image vectors with their caption counterparts without distorting their structure. After applying it, something remarkable happened — clusters began to make sense. The dog images sat near their captions again, the car clusters tightened, and the overall geometry felt more semantic.

Figure 3: We start with two sets of embeddings X and Y that approximately preserve distances but are unaligned (left). We find X¯ by apply the orthogonal Procrustes transformation to X (center). X¯ retains the exact geometry of X but the embeddings are now aligned with Y (right)

Even then, the alignment wasn’t perfect — and that’s the most important lesson. The joint space is not a solved problem; it’s a reflection of meaning itself. It changes with data, model biases, and language variation.

I wanted MMEE to be both powerful and simple. It’s built with:

• Streamlit — for a clean, interactive UI.
• Plotly — for dynamic 2D projections and hover-based exploration.

app/
├── config/          # Configuration files (paths, constants, model settings)
├── data_loaders/    # Functions to load datasets, captions, and embeddings
├── embedding/       # Computes embeddings & projections (CLIP, PCA, UMAP, etc.)
├── detection/       # Outlier detection logic (IsolationForest, LOF, DBSCAN)
├── ui/              # Streamlit UI components (tabs, layout, inputs)
├── utils/           # Shared helper functions (file I/O, caching, metrics)
├── visualization/   # Plotly visualization utilities (scatter plots, clusters)
└── main.py          # Entry point (orchestrates UI)

The user uploads a dataset containing:

images/
captions.json
labels.json

The MMEE tool visualizes joint image–text embeddings generated via CLIP. After loading the dataset, it computes projections using PCA, t-SNE, or UMAP and enables users to analyze alignment, clusters, and outliers. The sidebar provides controls for choosing projection methods, tuning parameters, and running global or class-wise outlier detection.

Technical Deep Dive; Detecting Truth in Noisy Captions

The core feature of MMEE is its ability to identify noisy captions — mislabeled, misleading, or semantically inconsistent image-text pairs.

Step 1: Injecting Noise

To evaluate the robustness of MMEE’s outlier detection pipeline, I introduced a controlled noise mechanism that simulates real-world captioning errors. The goal was to mimic how dataset noise arises when image–caption pairs are mismatched or mislabeled across classes.

The process is implemented in a custom utility script, **add_bad_captions.py**, which deliberately corrupts a subset of captions while maintaining a clear ground truth reference for evaluation.

How noise is injected:

1. The script loads the original dataset’s image–caption mapping (for example, captions.json or dataset_annotations.json).
2. A subset of images (typically 10–20%) is randomly selected for corruption.
3. For each selected image, its original caption is replaced by a caption belonging to a different class — specifically, a caption taken from another image that has a distinct semantic label.
4. For instance, an image labeled “dog running in the park” might be reassigned a caption like “a red sports car on the highway.” This ensures that the injected noise is plausible text but semantically inconsistent with the visual content.
5. The modified mapping is saved as **captions_with_noise.json**, representing the corrupted dataset.
6. In parallel, a ground truth file, **bad_caption_gt.json**, is generated. It stores the indices or filenames of all images whose captions were deliberately replaced, acting as the “answer key” for evaluation.
7. This controlled corruption allows MMEE to test whether its outlier detection modules (IsolationForest, Local Outlier Factor, DBSCAN, etc.) can successfully identify mismatched image–text pairs based purely on embedding geometry.

By aligning image and text embeddings in the same 2D/3D projection space, noisy samples typically appear as isolated or displaced clusters, helping visualize and validate how semantic inconsistency manifests numerically.

Step 2: Detection and Evaluation

Once the noisy captions were injected, the next stage involved detecting them using MMEE’s outlier detection suite and quantitatively evaluating how well the model could separate genuine image–caption pairs from corrupted ones.

Outlier Detection

After loading both clean and noisy embeddings, MMEE applies multiple detection algorithms — each operating on the joint image–text embedding space. The underlying assumption is that semantically mismatched pairs (e.g., a “car” image described as a “dog”) will occupy regions of lower density or lie far from their class clusters.

The following methods were tested:

• Isolation Forest: Identifies anomalies based on how easily a point can be isolated in a random partitioning tree structure.
• Local Outlier Factor (LOF): Detects points with substantially lower local density compared to their neighbors, ideal for dense semantic clusters.
• DBSCAN: Groups points into dense clusters and flags those that do not belong to any cluster as outliers.
• k-NN Distance Scoring: Computes distances to nearest neighbors within the same class; unusually distant points are considered semantically inconsistent.

Each method outputs a binary label (0 = normal, 1 = outlier), which is compared against the ground truth noise annotations from **bad_caption_gt.json**.

Evaluation Metrics

To assess detection quality, MMEE computes standard classification metrics:

• Precision: The proportion of detected outliers that were truly noisy.
• Recall: The proportion of all noisy samples that were correctly detected.
• F1-score: Harmonic mean of Precision and Recall, providing a single measure of accuracy–coverage balance.
• Confusion Matrix: Summarizes true positives (TP), false positives (FP), false negatives (FN), and true negatives (TN) for a visual performance snapshot.

For example, a representative run might yield:

Precision = 0.992 • Recall = 0.938 • F1 = 0.965 (TP = 394, FP = 3, FN = 26, TN = 1047)

These scores show that the model accurately differentiates between correctly labeled and semantically noisy samples while maintaining an extremely low false-positive rate.

The Global vs Class-wise Detection Problem

At first, I tried global outlier detection — treating the entire embedding space as one big dataset and running algorithms like Isolation Forest and LOF.

It didn’t work.

Different classes have different densities — “flowers” form tight clusters, while “birds” are more dispersed. The global detector couldn’t define what “normal” meant when everything looked different.

As a result:

• Dense classes dominated decisions.
• Sparse classes generated false positives.
• Precision dropped to ~0.42, recall to ~0.38.

It was frustrating. The visualizations looked messy, and the metrics were disappointing. I remember sitting there late at night, plotting point clouds that didn’t make sense — red (noisy) and blue (clean) points blending together with no clear separation.

Then my mentor Rajesh Suggested me: What if “normal” isn’t global? What if it’s local to each class?

The Magic of Class-wise Detection

I restructured the detection pipeline. Instead of one global detector, MMEE now:

1. Groups samples by class.
2. Fits an individual detector (Isolation Forest, LOF, kNN, or DBSCAN) for each cluster.
3. Flags points that deviate from the local density.

Suddenly, it worked.

The plots became clearer. Each cluster had a natural boundary. Precision jumped to ~0.83, recall to ~0.91. A “weird dog” was now compared only to other dogs — not to cars or flowers.

Figure 4: How a local (classwise) outlier method can help in improving the accuracy of the outlier detections.

The Fusion Scorer- Combining Multiple Signals

Detection alone had limits. Some captions were syntactically correct but semantically off — like “a small animal on the grass” for an image of a lion.

So I built a Fusion Scorer, a logistic regression model combining:

• CLIP cosine similarity,
• Residual distance between image and text vectors, caption length, and outlier scores from detectors.

This produced an authenticity score — a single measure of how “real” a caption felt. It improved the model’s ability to catch subtle misalignments without overreacting to small variations.

Mentorship and Growth

Working with Laurens, Rajesh and Samet at OpenVINO was the best part of GSoC. They didn’t just guide me technically — they helped me think like a researcher. Laurens and Rajesh taught me to focus on interpretability and clarity; Samet pushed me to structure my code modularly and think long-term.

Our discussions weren’t about fixing code; they were about understanding behavior — why embeddings behave the way they do, and how to make that behavior visible to others.

Results — Making Semantics Visible

Once MMEE stabilized, I was happy to see:

Figure 1: PCA projections showing cluster separation

Figure 2: Outlier overlay — noisy captions marked in red

Figure 3: Confusion matrix comparing predictions vs. ground truth

The visuals showed exactly what the model struggled with — captions drifting away, clusters overlapping, semantic noise becoming visible.

That’s when I realized: MMEE isn’t just a visualization tool. It’s an X-ray for multimodal models.

🌱 Lessons Beyond Code

This summer taught me that research is not about building perfect systems — it’s about understanding imperfection deeply. I learned patience through debugging, humility through failed experiments, and confidence through iteration.

I also learned that open-source isn’t about solo work — it’s about community. From my mentors’ guidance to the feedback loops on GitHub, every suggestion shaped MMEE into something more meaningful.

Closing Thoughts

GSoC taught me that clarity is a form of innovation. If you can help others see what’s happening inside AI models, you’ve already made a difference.

To anyone aspiring to apply for GSoC: Pick a problem that makes you curious — one that keeps you up at night not because it’s hard, but because it’s meaningful. Build something that helps others understand.

That’s what MMEE did for me. It helped me see — not just data, but how machines interpret the world around us.

Special Thanks to My Mentors: [Laurens Hogeweg, Rajesh Gangireddy & Samet Akcay ] — OpenVINO Toolkit.

Aarya Pandey Google Summer of Code 2025 | OpenVINO Toolkit. Project: Multimodal Embedding Explorer (MMEE)

LINK TO THE REPO: https://github.com/geeky33/mmee2025