FoundationPoseROS2: Revolutionizing Multi-Object Pose Estimation with ROS2

Demo for Multi-Object Tracking of Novel Objects using ROS and RealSense

Robotics Application

Introduction

In the world of robotics and automation, accurate object pose estimation plays a crucial role in enabling intelligent manipulation, navigation, and interaction. FoundationPoseROS2 offers a groundbreaking solution by integrating 6D object pose estimation and tracking capabilities within the ROS2 framework, powered by the latest advancements in segmentation and GPU technologies.

This system leverages RealSense2 cameras alongside the Segment Anything Model 2 (SAM2) for a seamless end-to-end pipeline, capable of real-time multi-object tracking and pose estimation. By optimizing GPU requirements to just 8GB, FoundationPoseROS2 democratizes access to high-performance robotics solutions.

What Sets FoundationPoseROS2 Apart?

1. ROS2 Real-Time Framework: Designed for real-time performance with minimal hardware requirements, offering an accessible alternative to isaac_ros_foundationpose which demands over 64GB of GPU memory.
2. SAM2-Based Segmentation: Utilizes advanced segmentation for object identification and tracking.
3. Multi-Object Pose Estimation: Tracks multiple objects simultaneously in real-world environments.
4. Interactive GUI: Simplifies model-to-mask assignment for a streamlined multi-pose estimation workflow.

Key Features

• Interactive Object Selection GUI: Select, reorder, and visualize object models through an intuitive interface.
• End-to-End Pose Estimation: Integrates segmentation, tracking, and pose calculation for novel objects.
• Lightweight GPU Requirements: Designed to run on a minimum 8GB NVIDIA GPU, with CUDA 12.x support.
• Real-Time Visualization: Displays pose information with bounding boxes and 3D axes for a comprehensive view.

Getting Started

Prerequisites

• Ubuntu: Compatible with ROS2 distributions like Foxy or Humble.
• ROS2: Install and configure for your environment.
• Intel RealSense Camera: Required for real-time image capture.
• CUDA 12.x: Ensure compatibility with NVIDIA GPUs.

Installation Steps

Set Up ROS2 and Dependencies:

sudo apt install ros-<ROS_DISTRO>-desktop ros-<ROS_DISTRO>-realsense2-* ros-<ROS_DISTRO>-librealsense2*

Install CUDA and Miniconda:

# Install CUDA
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.0-1_all.deb
sudo dpkg -i cuda-keyring_1.0-1_all.deb
sudo apt-get update && sudo apt-get install cuda

# Install Miniconda
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh -b -u -p ~/miniconda3
source ~/miniconda3/bin/activate

Clone and Build the Repository:

git clone https://github.com/ammar-n-abbas/FoundationPoseROS2.git
conda create -n foundationpose_ros python=3.10 -y
conda activate foundationpose_ros
cd FoundationPoseROS2 && bash build_all_conda.sh

Running the System

Live Camera Feed

Launch RealSense Camera Node:

source /opt/ros/<ROS_DISTRO>/setup.bash
ros2 launch realsense2_camera rs_launch.py enable_rgbd:=true align_depth.enable:=trueRun Pose Estimation:

Run Pose Estimation:

conda activate foundationpose_ros
python ./FoundationPoseROS2/foundationpose_ros_multi.py

[embed]Tutorial

Rosbag2 Playback

Download and Play Rosbag:

[embed]cube_demo_data_rosbag2 - Google Drive This browser version is no longer supported. Please upgrade to a supported browser.drive.google.com

source /opt/ros/<ROS_DISTRO>/setup.bash && ros2 bag play cube_demo_data_rosbag2/cube_demo_data_rosbag2.db3

Run the Pose Estimation Node:

python ./FoundationPoseROS2/foundationpose_ros_multi.py

[embed]Rosbag demo tutorial

Running on Novel Objects

FoundationPoseROS2 is designed to handle novel objects by utilizing custom object models in .obj or .stl formats. Follow these steps to integrate and track new objects:

Add Object Mesh: Place the mesh file of the object in the following directory:

./FoundationPoseROS2/demo_data/object_name/<OBJECT_MESH>.obj

Run the Pose Estimation Script: Activate the conda environment, configure ROS2, and execute the pose estimation script:

conda activate foundationpose_ros
source /opt/ros/<ROS_DISTRO>/setup.bash
export PATH=/usr/local/<YOUR_cuda-12.X_VERSION>/bin${PATH:+:${PATH}}~
python ./FoundationPoseROS2/foundationpose_ros_multi.py

[!NOTE] Verify and update the camera topics in the Python script to match your setup for seamless operation.

By integrating new object meshes, FoundationPoseROS2 allows you to expand its applicability to any custom object in real-world environments.

Acknowledgements

This project is funded by the EU Commission Recovery and Resilience Facility and supported under the Science Foundation Ireland Future Digital Challenge (Grant №22/NCF/FD/10929).

Conclusion

FoundationPoseROS2 represents a leap forward in robotics, enabling robust and scalable pose estimation for novel objects in dynamic environments. Its innovative use of ROS2 and SAM2 frameworks ensures compatibility and performance, empowering researchers and engineers to achieve new levels of efficiency and precision in robotic systems.

Try it out today and transform the way you approach multi-object tracking and pose estimation!

For more information, visit:

[embed]GitHub - ammar-n-abbas/FoundationPoseROS2: FoundationPoseROS2 is a ROS2-integrated system for 6D… FoundationPoseROS2 is a ROS2-integrated system for 6D object pose estimation and tracking, based on the FoundationPose…github.com