I Spent a Semester Reproducing a Research Paper. Here’s What Surprised Me.

When my professor said “reproduce these results,” I thought: how hard can it be? The paper is published. The method is described. The metrics are there in Table 1.

Three months later I had a working pipeline, results that mostly matched, and a very different understanding of what “reproducibility” means in deep learning research.

The paper was SAM-REF — a method for interactive image segmentation that builds on Meta’s Segment Anything Model (SAM). The goal: given an image and some user clicks, produce a clean segmentation mask. SAM-REF improves on plain SAM by adding a refiner on top of the base model. My job was to reimplement it from scratch in PyTorch and verify the results.

The paper told me what. It didn’t tell me how.

The method has two stages. Stage A fine-tunes SAM’s decoder. Stage B trains a refiner on top. Both are described clearly enough in the paper.

What wasn’t described clearly: the exact training setup, the click simulation strategy, how embeddings were cached (or whether they were), which subset of COCO was used for training, and a handful of implementation details that turn out to matter a lot.

None of this is unusual. Research papers are not codebases. They’re descriptions of experiments, written after the fact, compressed to fit a page limit. The gap between “we describe our method” and “you can run our method” is almost always larger than it looks.

I spent the first two weeks figuring out what I didn’t know yet.

Embedding caching was not optional

SAM’s image encoder is a heavy Vision Transformer. Running it for every training example, every epoch, would have taken forever. The solution is to precompute and cache the image embeddings once — then at training time, you only run the decoder.

The paper mentions this in passing. I had to figure out the indexing scheme, the storage format, and how to make sure the cached embeddings and the training examples stayed aligned as I shuffled the dataset. It took longer than it should have. Getting it wrong meant training runs that silently produced garbage because the image and the mask were out of sync.

No crash. No error message. Just quietly wrong outputs for a day and a half until I caught it.

Click simulation is harder than it sounds

Interactive segmentation is trained by simulating user clicks. You generate a mask prediction, find where it’s wrong, place a click there, and iterate. Simple in theory.

In practice: where exactly do you place the click? The centroid of the error region? A random point inside it? The point of maximum distance from the mask boundary? Each choice produces slightly different training dynamics. The paper’s description was vague enough that I had to make a judgment call, run experiments, and compare.

I ended up logging everything — which click strategy, how many iterations, what the loss looked like — because the only way to understand what was happening was to have a record of it.

My numbers were close. Not identical.

After getting Stage A and Stage B working, I evaluated on GrabCut and Berkeley — the same benchmarks the paper uses, with mIoU@K and NoC@90/95 as metrics.

The numbers were close to the paper. Not identical.

Is that a successful reproduction? Honestly, I’m not sure. Some of the gap is almost certainly from implementation differences in the training details I had to infer. Some might be from hardware differences. Some might be from click simulation choices.

The paper authors would probably say the results reproduce within a reasonable margin. A stricter reading says: I produced a pipeline that works and behaves similarly, but I can’t fully explain the delta.

That’s uncomfortable to sit with. It also feels like the honest answer. I suspect most ML papers, if you tried to reproduce them carefully, would give you a similar one.

What I’d do differently

Log implementation decisions, not just training runs. I had detailed logs of loss curves and metric values. I had loose notes on why I chose a particular click strategy or how I structured the dataset indexing. By the time I was writing up, I was reconstructing choices from git commits and memory.

Also: the bugs that cost me the most time were in the data pipeline, not in the model. The embedding caching, the dataset indexing, the alignment between images and masks — that’s where things went wrong quietly. Model architecture bugs are usually obvious. Data pipeline bugs often aren’t.

The thing I keep thinking about

Reproducing a paper made me read papers differently. Now when I read a methods section, I’m looking for what’s missing — not what’s wrong, but what the authors knew and didn’t think to write down because it was obvious to them.

The gap between a published result and a working implementation is normal. It doesn’t mean the research is bad. It means papers are summaries, not manuals.

If you’re about to reproduce a paper: budget more time than you think, start with the data pipeline, and log everything. The interesting part isn’t whether you match the numbers. It’s what breaks on the way there.

MSc AI at FAU, researching how vision models break under real-world image conditions.