SAM 2: Segment Anything… Images and Video!

From SAM 2 paper

The ability to precisely identify and segment objects within images and videos is a cornerstone of computer vision: with the advent of foundation models like the Segment Anything Model (SAM), this capability has become remarkably accessible for static images, however, extending this promptable segmentation prowess to the dynamic world of videos presents a new set of challenges. In fact, objects in motion, occlusions, and the sheer volume of data in video streams require an architecture capable of not only understanding individual frames but also maintaining a temporal context of the scene… But now there’s SAM 2!

More details here:

1. From Images to Videos: The Challenge of Temporal Context
2. SAM 2 Architecture: Memory Makes the Difference
3. The SA-V Dataset: A Foundation for “Segmenting Anything” in Videos
4. Evaluating SAM 2: A Versatile Performer
5. Ablations: Insights into Design Choices
6. Conclusion

From Images to Videos: The Challenge of Temporal Context

While SAM revolutionized image segmentation by allowing users to identify objects with simple prompts like clicks or boxes, video segmentation necessitates a deeper understanding of how objects persist and evolve over time. For example, imagine tracking a soccer ball as it’s passed between players, momentarily disappearing behind their legs, and then reappearing: a robust video segmentation model must be able to maintain a memory of the ball’s appearance and trajectory, predicting its location even when temporarily obscured.

Nonetheless, existing video segmentation datasets primarily focus on specific object categories (people, cars, animals) and complete objects rather than parts. This limits their usefulness in training a truly versatile model capable of “segmenting anything” in a video, including parts of objects like a person’s hand or a bike’s tire.

SAM 2 Architecture: Memory Makes the Difference

SAM 2 addresses these challenges by introducing a novel streaming architecture that incorporates memory, enabling the model to track objects across video frames.

SAM 2 architecture

Let’s break down its key components:

Image Encoder: A streaming approach is adopted for real-time processing of arbitrarily long videos. An efficient MAE pre-trained Hiera image encoder (a hierarchical Vision Transformer) is used to extract per-frame features. This encoder is run only once per interaction, providing a compact representation of each frame.

Memory Attention: The heart of SAM 2’s video capability lies in its memory attention module. This module takes the per-frame features from the image encoder and conditions them on the information stored in the memory bank.

Memory Bank: This bank holds two types of memories:

1. Recent Frames: Stores features from the last N frames, allowing the model to represent short-term object motion and appearance changes. This memory is updated as the video progresses, capturing the most recent information about the target object.
2. Prompted Frames: Stores memories of frames where the user provided prompts (clicks, boxes, masks). This long-term memory helps the model recall the object’s characteristics, even if it’s been occluded for an extended period.

Object Pointers: Lightweight vector representations that capture high-level semantic information about the object being segmented. These pointers, derived from the mask decoder’s output, provide an additional layer of contextual information to the memory attention module.

Prompt Encoder and Mask Decoder: Similar to SAM, this component takes user prompts (points, boxes, or masks) and combines them with the conditioned frame embedding to predict a segmentation mask for the current frame. It can predict multiple masks for ambiguous prompts and even predict if the object is present in the frame.

Memory Encoder: Downsamples the predicted mask and combines it with the original frame embedding, generating a compact memory representation to be stored in the memory bank for future use.

The model is trained jointly on image and video data, simulating interactive prompting. During training, the model receives a sequence of frames, some of which are randomly selected for prompts. It then predicts the ground-truth masklet (a sequence of masks), allowing it to learn both spatial segmentation and temporal tracking.

The SA-V Dataset: A Foundation for “Segmenting Anything” in Videos

To train a model capable of segmenting anything in videos, a massive and diverse dataset is crucial. SAM 2 utilizes the SA-V (Segment Anything Video) dataset, the largest video segmentation dataset to date, collected through an innovative data engine.

The SA-V dataset was built using a data engine that leverages SAM and SAM 2 in the loop with human annotators. This approach significantly accelerates the annotation process while maintaining high quality:

• Phase 1: SAM per Frame: Annotators used SAM to segment objects in each frame individually, resulting in high-quality spatial annotations but requiring considerable time.
• Phase 2: SAM + SAM 2 Mask: Annotators used SAM to segment the object in the first frame and then utilized SAM 2 (accepting only mask prompts) to propagate the mask to other frames. This significantly reduced annotation time.
• Phase 3: Full SAM 2: Annotators used the full capabilities of SAM 2 with point and mask prompts, further reducing annotation time by leveraging the model’s memory and allowing for easy refinement with occasional clicks.

To enhance diversity, the data engine also includes automatic masklet generation. SAM 2 is prompted with a grid of points in the first frame, and the resulting masklets are verified by human annotators. Unsatisfactory masklets are either rejected or sent back for refinement, further improving the model’s capabilities and identifying failure cases.

The SA-V Dataset Details:

• Scale: 50.9K videos with 642.6K masklets (35.5M masks), 53 times larger than any existing video segmentation dataset.
• Diversity: Covers diverse scenes (indoor, outdoor), objects (whole objects and parts), and motion patterns, including occlusions and reappearances.
• Geographic and Demographic Representation: Videos were collected by crowdworkers spanning 47 countries, ensuring a wide range of real-world scenarios and minimizing bias.

Comparison of the SA-V dataset with existing video object segmentation datasets

In comparison with SA-V Manual, that contains only manually annotated labels, the SA-V Manual+Auto version reaches 642.6k masklets and 35.5M masks, having manually annotated labels and also automatically generated masklets.

Also, another dataset with available licensed video data has been used internally during the training.

Evaluating SAM 2: A Versatile Performer

SAM 2’s effectiveness is evaluated across a range of zero-shot video and image segmentation tasks, demonstrating its ability to generalize to diverse scenarios.

Promptable Video Segmentation:

SAM 2 is tested in a simulated interactive setting where users provide prompts (clicks) to segment objects in a video. It consistently outperforms baselines constructed by combining SAM with state-of-the-art video object segmentation models (XMem++ and Cutie) on 9 densely annotated datasets. SAM 2 achieves higher accuracy with significantly fewer user interactions (over 3x fewer), showcasing its superior interactive segmentation capabilities.

Semi-supervised Video Object Segmentation (VOS):

In the traditional VOS setting (where mask prompts are provided only on the first frame and scribbles on multiple frames to segment objects throughout a video), the model receives a mask prompt only in the first frame. SAM 2 outperforms specialized VOS models like XMem++ and Cutie on 17 datasets, demonstrating its strong performance even on tasks for which it’s not specifically designed.

Image Segmentation:

Even on static images, SAM 2 surpasses the original SAM in accuracy on a benchmark of 37 datasets while achieving a 6x speedup. This improvement is attributed to a smaller but more efficient image encoder and the diversity of the training data.

Fairness Evaluation:

SAM 2 is evaluated on the Ego-Exo4D dataset, which includes demographic information about the people in the videos. While there are minor performance discrepancies for single-click prompts, these are likely due to prompt ambiguity. At 3 clicks and with ground-truth mask prompts, the model demonstrates minimal performance differences across gender and age groups.

Ablations: Insights into Design Choices

Extensive ablations provide valuable insights into the design decisions behind SAM 2.

Data Ablations:

• Data Mix: Combining SA-V, internally available licensed video data, and SA-1B (image data) yields the best performance across both video and image segmentation tasks, highlighting the importance of diverse training data.
• Data Quantity: A clear power-law relationship is observed between the amount of training data and segmentation accuracy, indicating that further scaling the dataset could lead to even better performance.
• Data Quality: Filtering SA-V data based on the number of edits made by annotators during collection results in better performance than random sampling, suggesting that focusing on challenging cases is beneficial.

Model Ablations:

• Capacity: Increasing model capacity by using a larger image encoder or more layers in the memory attention module generally improves performance.
• Relative Positional Encoding: Removing relative positional biases from the image encoder and using 2D RoPE in the memory attention module enhances speed without sacrificing accuracy.
• Memory Architecture: Directly storing memory features in the memory bank proves to be sufficient and more efficient than using recurrent memory with GRUs. Additionally, incorporating object pointers into the memory bank boosts performance on challenging datasets.

Conclusion

Concluding, we can recap the main contributions of SAM 2 as:

• Extends promptable segmentation to video: Allows users to interact with videos using familiar prompts (clicks, boxes, masks).
• Introduces a streaming architecture with memory: Enables real-time processing of arbitrarily long videos and maintains temporal context of objects.
• Provides the SA-V dataset: Offers the largest and most diverse video segmentation dataset to date, accelerating research in this field.

Also, the model of SAM 2 is under Apache 2.0 License, available on GitHub, so it has the potential to democratize access to advanced video understanding capabilities. Isn’t it really nice?

PS: This article coming from my website, feel free to subscribe, follow me, or whatever you want!

This story is published on Generative AI. Connect with us on LinkedIn and follow Zeniteq to stay in the loop with the latest AI stories.

Subscribe to our newsletter and YouTube channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!