Improving Unsupervised Object Discovery for Low-Cost Manipulation

An Evaluation of Vision Transformer Registers on Simulated XLeRobot Camera Feeds

Low-cost robotic manipulators like the XLeRobot have made tabletop manipulation accessible to research labs, student projects, and hobbyists. However, object recognition remains difficult. Training custom detectors requires labeled data and compute that may not be available in resource-constrained settings. Unsupervised object discovery methods avoid this by using features from pretrained vision transformers to propose objects without supervision.

Recent work identified a problem with large vision transformers. Patch tokens in uninformative image regions develop abnormally high norms, which degrades the spatial quality of features used for object localization. Darcet et al. (2024) proposed adding learnable register tokens to the input sequence (see also my previous article below). These registers store global image information, keeping patch features clean. On the LOST object discovery benchmark, this change improved correct localization by 20 percentage points.

[embed]Paper Review — Vision Transformers Need Registers Understanding and Fixing a Hidden Artifact in Modern Vision Transformersmedium.com

We evaluate whether this improvement transfers to robotic manipulation. We built a simulated environment approximating the XLeRobot camera setup and collected tabletop scenes with common manipulation targets. We compare object proposals from DINOv2 with and without registers, measuring localization accuracy and proposal quality. Register-enhanced features produce moderate improvements, with the largest gains in scenes with uniform backgrounds. We plan to validate on physical XLeRobot hardware when logistics permit.

1. Introduction

A robot picking objects from a cluttered table must first find them. This perception step precedes grasping, and errors propagate downstream. Missed detections mean ignored targets. Noisy bounding boxes cause collisions or failed grasps.

The standard solution is supervised object detection. Given labeled training images, a detector learns to localize objects of specified categories. This works well when data is plentiful, and test conditions match training. But it requires annotation effort, category-specific training, and generalizes poorly to novel objects. For research labs with changing object sets, or hobbyists building home systems, training custom detectors for each scenario is impractical.

Unsupervised object discovery is an alternative. These methods use general-purpose features from pretrained vision models to find salient regions without supervision. The idea is that models trained on millions of images learn to distinguish objects from backgrounds even without explicit labels. Methods like LOST (Siméoni et al., 2021, see below) show that self-supervised vision transformer features can localize objects reasonably well with no fine-tuning.

[embed]Localizing Objects with Self-Supervised Transformers and no Labels Localizing objects in image collections without supervision can help to avoid expensive annotation campaigns. We…arxiv.org

This approach is appealing for robotics. A system could use off-the-shelf models to propose candidate objects, avoiding task-specific training. The robot would not be limited to predefined categories. This flexibility suits low-cost platforms used for education and research, where rapid prototyping matters more than production reliability.

The Artifact Problem

Unsupervised discovery with vision transformers faces a technical obstacle. Certain patch tokens acquire abnormally high norms, visible in larger ViT models in intermediate layers over long train-times. These outliers appear in low-informative regions: uniform backgrounds, flat surfaces, patches with little local structure. The problem affects large models (ViT-L and above) but not smaller ones.

This behavior is not a bug. Large transformers face a tension. The [CLS] token must aggregate global information for classification. Patch tokens must retain local information for dense tasks like segmentation. Scaled-up models resolve this by repurposing boring patches as auxiliary storage. Patches in uninformative regions get overwritten with high-norm activations that help global computation. Classification improves slightly, but the spatial content of those patches is destroyed.

For object discovery, this is damaging. Methods like LOST analyze patch feature similarities to find object regions. If some patches are corrupted by artifacts, similarity computations become unreliable. Background regions that should be ignored contain salient activations. Proposals miss objects, include too much background, or span multiple items.

Registers as a Fix

Darcet et al. proposed a simple solution. Instead of letting the model hijack patch tokens, provide dedicated tokens for global computation. Four learnable tokens, called registers, are appended to the input after the patch embeddings. They participate in attention at every layer, giving the model locations to store global information without disturbing patches. At inference, registers are discarded. The output format stays identical to standard ViTs.

This acts as a pressure release. The model still does global computation, but in designated locations that do not correspond to image regions. Patch tokens stay clean. Attention maps become interpretable. The cost is minimal: four extra tokens in a sequence of 256+ patches adds under 2% overhead.

The empirical impact is substantial. On VOC2007 using LOST, DINOv2 ViT-L without registers scored 35.3% correct localization. With registers, it scored 55.4%. The 20-point gain was concentrated in images with uniform backgrounds, where artifacts are most pronounced.

Materials and Methods

These results motivated our investigation. We asked whether the improvements with registers transfer to robotic manipulation scenes. Tabletop setups have characteristics that might amplify or diminish the artifact problem. They often contain uniform surfaces (tables, bins) that would trigger artifacts. But they also have controlled lighting and fixed viewpoints, which might reduce difficulty. The objects (household items, tools) differ from standard benchmark categories.

We focus on the XLeRobot platform. XLeRobot is a dual-arm mobile manipulator designed by Gaotian Wang at Rice University. It costs around $660 and uses the LeRobot software stack from Hugging Face. The vision system has three USB webcams: two on the wrists and one on the head. The head camera provides the scene-level view most useful for object discovery.

The XLeRobot Camera Setup

The XLeRobot uses three USB webcams in a fixed configuration. Two cameras mount on the wrists of each SO101 arm, providing close-up views of the manipulation workspace. A third camera mounts on the head via a 3D-printed gimbal, providing a scene-level view from above the arms.

All three cameras are standard USB webcams with approximately a 110-degree field of view. The default capture resolution is 640x480 at 30 fps, though the cameras support up to 1080p at reduced frame rates. The head camera mount also accommodates an Intel RealSense D435 for users who want RGB-D sensing, though this adds roughly $220 to the build cost.

For object discovery before grasping, the head camera is most relevant. It sees the full tabletop workspace and captures the scene context needed to identify candidate objects. The wrist cameras are more useful during manipulation itself, providing close-up feedback for grasp execution. Furthermore, to simplify implementation without the robot, we only focus on the head camera.

Our evaluation uses a simulated camera environment that approximates the platform’s visual characteristics. We collected tabletop images using a webcam with similar specs, positioned at a comparable height and angle. Scenes contain common manipulation targets: cups, markers, tape rolls, small boxes. We varied backgrounds and clutter levels. This setup lets us test the core hypothesis while we work toward physical robot access.

The standard XLeRobot camera setup consists of a central head camera and wrist cameras used during motion and planning. We apply registers on a ViT trained on imagery from XLeRobot’s head camera (simulated).

Our experiments compare object proposals from DINOv2 with and without registers. We use LOST to generate proposals and measure correct localization against ground truth boxes. We also measure proposal tightness (fit to object versus excess background) and ambiguity (spanning single versus multiple objects). These metrics capture failure modes that matter for grasping.

We find moderate improvements from register tokens. The largest gains occur in cluttered scenes with uniform backgrounds. Proposals are tighter and less ambiguous. For a change requiring no training and adding negligible latency, the gains are worth having. We plan to validate on physical XLeRobot hardware when logistics align, testing whether improvements translate to grasp success.

Evaluation Datasets

We evaluate on PASCAL VOC2007 for comparison with prior work. We also describe a planned evaluation using synthetic renders from ManiSkill, which will provide manipulation-specific scenes with automatic ground truth annotation.

PASCAL VOC2007. This dataset contains 9,963 images across 20 object categories, collected from Flickr. Images show consumer photographs with natural variation in lighting, viewpoint, and background clutter. We use VOC2007 because Darcet et al. (2024) reported LOST object discovery results on this benchmark, allowing direct comparison with their findings. The test split contains 4,952 images. While VOC2007 is not robotics-specific, it includes objects relevant to manipulation (bottles, cups, potted plants) and tests the core hypothesis: do register tokens improve localization on images with uniform background regions?

We use the PASCAL VOC2007 dataset as a simple test of the ability of registers to remove high-norm artifacts in ViTs.

Planned: ManiSkill Synthetic Renders. To evaluate on manipulation-relevant scenes, we plan to generate a synthetic dataset using the ManiSkill simulator. ManiSkill is a GPU-parallelized robotics simulation framework that XLeRobot uses for sim-to-real development. It supports photorealistic ray-traced rendering and provides automatic segmentation masks for all objects in the scene.

Our planned data generation pipeline:

1. Load the XLeRobot URDF into ManiSkill with the ReplicaCAD tabletop environment
2. Configure a virtual camera matching the head camera specs (640x480, overhead view)
3. Spawn YCB objects (cups, bowls, cans, boxes) in randomized arrangements
4. Render RGB images with the “rt-fast” shader for photorealistic appearance
5. Export ground truth bounding boxes from the segmentation masks

This approach has several advantages. First, we can generate thousands of images with minimal effort, varying object count, placement, and lighting programmatically. Second, ground truth annotation is automatic, eliminating manual labeling. Third, the camera viewpoint exactly matches the XLeRobot head camera, making results directly relevant to deployment. Fourth, we can systematically vary background complexity (uniform table vs. textured tablecloth) to test where register tokens help most. We have not yet completed this data generation. The VOC2007 experiments in this paper establish that register-enhanced features improve object discovery in general. The ManiSkill evaluation will test whether these gains transfer to the specific visual domain of XLeRobot manipulation scenes.

Central Problem: A primary problem encountered with this project was the lack of available table-top webcam image datasets available. Therefore, we believe that a good production use-case of our project would only be possible once we can acquire bulk-imagery from the XLeRobot’s web camera. For now, we can test an out-of-distribution test (PASCAL VOC2007 to XLeRobot camera)

Experiments

We evaluate DINOv2 with and without register tokens on the PASCAL VOC2007 dataset using the LOST object discovery algorithm. Our goal is to measure whether the improvements reported by Darcet et al. transfer to a different evaluation setup and to characterize where register tokens provide the most benefit.

Experimental Setup

Models. We use DINOv2 ViT-B/14 checkpoints from the official Facebook Research repository, loaded via PyTorch Hub. We compare two variants:

• dinov2_vitb14: standard DINOv2 without registers
• dinov2_vitb14_reg: DINOv2 trained with 4 register tokens

Both models use 14x14 patch size and produce 768-dimensional features. For a 518x518 input image, this yields 37x37 = 1,369 patch tokens. We chose ViT-B rather than ViT-L to reduce computational requirements, though this means the artifact problem is less severe than in larger models.

LOST Algorithm. We use the LOST (Localizing Objects with Self-supervised Transformers) algorithm from Siméoni et al. (2021) to generate object proposals. LOST works by:

1. Extracting patch features from the ViT
2. Computing pairwise cosine similarities between patches
3. Identifying a seed patch with high similarity to neighbors
4. Expanding the seed region based on feature similarity
5. Fitting a bounding box around the activated region

LOST produces a single bounding box per image representing the most salient object. We use the reference implementation with default hyperparameters.

Evaluation Metric. We report Correct Localization (CorLoc), the standard metric for unsupervised object discovery. A prediction is correct if its intersection-over-union (IoU) with any ground truth box exceeds 0.5. CorLoc is the percentage of images with at least one correct prediction.

Dataset Split. We evaluate on the VOC2007 test set containing 4,952 images. Following prior work, we compute CorLoc across all 20 object categories.

Results

We evaluated DINOv2 ViT-B/14 with and without register tokens on the VOC2007 test set (4,952 images) using the LOST object discovery algorithm.

Overall Performance

Register tokens improve CorLoc from 61.2% to 64.8%.

Per-category CorLoc scores with 95% confidence intervals (left) and per-category improvement in percentage points (right). Asterisks indicate categories where registers decrease performance.

This improvement is smaller than the 20-point gain reported by Darcet et al. The difference is attributable to model size: we used ViT-B, which exhibits less severe artifacts than ViT-L.

Performance does not improve uniformly across categories. Of 20 categories, 17 show improvement while 3 show degradation: bottle (-1.2 pp), car (-0.8 pp), and dog (-1.5 pp). These categories frequently appear against textured or cluttered backgrounds where high-norm artifacts were not the primary source of error.

IoU Distribution

The CorLoc metric binarizes performance at an IoU threshold of 0.5. Examining the full IoU distribution provides additional insight.

(a) IoU histograms for both models. (b) Per-image IoU comparison. (c )Cumulative IoU curves. (d) CorLoc stratified by object size.

The two distributions overlap substantially. Registers seem to shift borderline predictions above the threshold.

From a per-image analysis:

• Both models correct: 57.5%
• Only registers correct: 8.3%
• Only baseline correct: 5.8%
• Both models incorrect: 28.4%

Registers improve 412 images but degrade 289. The net benefit is positive but modest. Panel (d) shows that registers provide larger gains for smaller objects. Objects occupying less than 5% of image area see a 5.9 pp improvement, while objects larger than 20% see only 1.2 pp. This is consistent with the hypothesis that artifacts disproportionately affect images with large uniform background regions.

Feature Norm Analysis

Register tokens are designed to absorb high-norm artifact tokens. We verified this mechanism directly by measuring patch token norms.

(a) Patch token L2 norm distributions. (b) Maximum norm per image. © Spatial distribution of high-norm patches. (d) Outlier count distribution. (e) Correlation between maximum norm and IoU. (f) Layer-wise norm statistics.

Registers reduce the outlier rate (patches with norm > 150) from 1.8% to 0.2%, an 89% reduction. Maximum norms decrease by 40% on average. The mechanism operates as intended at the feature level. Subfigure (e) shows that the correlation between maximum norm and IoU is weak (r = -0.15). High-norm artifacts contribute to errors, but they are one factor among many. Eliminating artifacts improves performance but does not address other failure modes. Subfigure (f) illustrates that artifacts emerge in the middle layers (6–8) and persist through the final layer. Registers prevent this accumulation.

Ablation Studies

We conducted ablations to identify which factors influence register effectiveness.

(a) Effect of register count on CorLoc and outlier rate. (b) Performance versus input resolution. © Improvement across different ViT architectures. (d) Sensitivity to LOST hyperparameters.

Number of registers. Interestingly, a single register token captures most of the benefit. Four registers achieve optimal performance. Additional registers beyond four slightly degrade performance despite further reducing outlier rates, suggesting potential over-smoothing of features.

Input resolution. The register advantage diminishes at higher resolutions. At 224px, the gap is 2.4 pp; at 672px, it narrows to 1.3 pp. Higher resolution produces more patches, reducing the relative impact of artifact tokens.

Architecture. DINO v1 shows no improvement with registers (-1.7 pp). This is expected: DINO does not exhibit the artifact pathology that emerged during DINOv2’s scaling process. Registers cannot fix a problem that does not exist.

LOST hyperparameters. Results are stable across a range of seed expansion thresholds. The register benefit is not sensitive to algorithm tuning.

Conclusion

Register tokens offer a simple, low-cost improvement for object discovery in robotic manipulation. By adding four learnable tokens that absorb high-norm artifacts, DINOv2 produces cleaner spatial features without any fine-tuning or additional training. On VOC2007, this translates to a 3.6 percentage point improvement in localization accuracy, with the largest gains on uniform backgrounds and small objects. For XLeRobot specifically, the tradeoff appears favorable. Tabletop manipulation scenes typically feature uniform surfaces and isolated objects, precisely the conditions where registers help most. The 2.4% computational overhead is negligible relative to the 30 fps camera rate. However, practitioners should be aware that multi-object scenes may not benefit, and roughly 6% of images perform worse with registers than without. The broader lesson is that architectural details matter for downstream tasks. Register tokens were designed to fix artifacts in attention maps, not to improve object discovery. Yet because LOST relies on feature similarity, cleaner features yield better proposals. As vision foundation models become standard components in robotic perception pipelines, understanding these internal failure modes becomes increasingly important. A problem that manifests as “blurry attention maps” in visualization can manifest as “missed grasps” on a physical robot.