Project IDRIS: Indoor Delivery Robot Intelligence System

ROS2, Industrial Robotics, Turtlebot 4, PincherX, Mediapipe, Human-Robot Collaboration

Manual object transport and sorting are inefficient and repetitive in indoor environments such as warehouses, labs, and facilities. An autonomous system that combines mobile navigation and vision-based manipulation for intelligent object delivery and sorting.

Core Concept: TurtleBot4 → Autonomous Navigation → Object Delivery → PincherX 100 → MediaPipe Gesture Control → Color-Based Sorting/ Recognition techniques.

Integration Layer:

ROS2 State Machine coordinating all subsystems

System Architecture

System Architecture — IDRIS

[embed]Demonstration (Robotics & Computer Vision)

Phase1

We work on mapping the workspace using SLAM Toolbox and configuring Nav2 for autonomous navigation. Define waypoints for the delivery station. Implement a robot status publishing system. Define a complete 2D map of the operating environment.

Key deliverables: Complete 2D map of operating environment, Autonomous navigation to PincherX station, Robot status topic (ARRIVED state broadcast)

Turtlebot 4- At Home location (0,0)

Phase2

During our next phase, we will configure the Interobotix SDK for ROS2 in Docker, test joint control, and integrate it with MoveIt to implement basic pick-and-place operations. Later, we will set up a camera feed for MediaPipe. This phase focuses primarily on the robotic arm, the PincherX 100. We will use the Interbotix SDK via a Docker container to run ROS2 Humble. Next, we will test joint control and movement integration for inverse kinematics (both rotational and linear movement) using Python scripts for pick-and-place operations. The final objective is to set up the camera for MediaPipe and integrate the camera and robotic arm together.

Key Deliverables: Workspace is divided into cells. Pre-computed Inverse Kinematic(IK) solutions are generated for each cell. The hand position maps to the nearest grid cell, enabling fast, stable, and intuitive control. Functional pick-and-place operations with hardcoded poses and 2–3 defined sorting bin locations. Working camera feed for vision processing (MediaPipe).

Tech Stack: Interbotix SDK (control), MoveIt2, OpenCV, ROS2 Humble (as a replacement for ROS2 Jazzy), Docker.

We divide the input camera feed into different cells (grids) and assign different pre-computed inverse kinematic values for various purposes. Fast, stable, and intuitive.

Key deliverables

• Hand Tracking with MediaPipe: Enables real-time tracking of the operator’s hand for intuitive robot control.
• Grid-to-Workspace Mapping: The workspace is divided into grid cells with pre-computed IK solutions. The detected hand position maps to the nearest cell for fast and stable arm movement.
• Gesture-Based Gripper Control:

Hand Position → Arm Position, Pinch Gesture → Gripper Close, Open Hand → Gripper Open

PincherX 100 (Robotic Arm)

Smooth Cell Transitions: Ensures fluid arm movement when switching between adjacent cells.

Functional Pick-and-Place: Performs pick-and-place operations using hardcoded poses.

Defined Sorting Bins: Supports 2–3 predefined sorting bin locations.

Working Camera Feed: Real-time camera feed integrated with MediaPipe for vision processing.

Phase4 — Integrated Delivery and Sorting Operations

In this phase, we achieve full system integration between the TurtleBot4 mobile base and the PincherX 100 robotic arm, enabling end-to-end autonomous delivery, vision-guided manipulation, and gesture-assisted sorting. The workflow combines navigation, perception, and manual intervention for flexible operation in semi-autonomous environments.

Workflow States

NAVIGATING — The TurtleBot4 autonomously navigates to the designated delivery station using ROS 2 navigation and mapping.

ARRIVED — Upon arrival, the system activates MediaPipe-based gesture control, allowing the operator to take over local manipulation.

MANUAL_CONTROL — The operator guides the robotic arm using hand tracking and gestures for precise object picking.

SORTING — The object is placed into one of the predefined sorting bins, guided by the operator’s gestures and the grid-to-workspace mapping.

DONE — The system resets or loops automatically to handle the next object, maintaining continuous workflow operation.

Key deliverables

Complete Robot Integration: Seamless coordination between TurtleBot4 (navigation) and PincherX100 (manipulation).

Cross-Robot Communication: Robust ROS 2 communication architecture linking navigation, manipulation, and vision modules.

Error Handling: Built-in mechanisms for fault detection, recovery, and manual override during navigation or manipulation.

System Architecture Documentation: Comprehensive documentation of ROS 2 nodes, topics, and inter-robot communication flow.

Implementation Details: Clear explanations of integration steps, Docker configurations, and ROS 2 launch workflows.

Performance Metrics: Evaluation of navigation accuracy, manipulation success rate, and overall system latency.

Demo Video Recording: Recorded demonstration showcasing navigation, gesture control, and sorting workflow with polished final demo: Refined and edited version of the demo for presentation or deployment showcase.

Complete Documentation (IDRIS): End-to-end project documentation covering design, implementation, testing, and evaluation.

Reproducible Codebase: Fully containerized, version-controlled code ensuring easy replication and deployment.

Mediapipe (CV2 Libraries)

Parallel mediapipe functions with color-based sorting functions. Waypoint Once we have the map, we can set the origin as home (0,0) and target locations as point 1 (10,0) and so on for moving.

Levels of Implementation

Level 1

• TurtleBot4 autonomous navigation
• PincherX 100 basic pick-and-place
• MediaPipe hand tracking working

Level 2

• Camera-arm integration complete
• Full system integration with robot waypoints
• Functional sorting pipeline

Level 3

• VR control via Quest 2
• Multiple human avoidance
• Completely autonomous (no human)

This slide outlines some hand-picked technical challenges and their solutions for our IDRIS robotics concept:

Challenge 1: Real-time hand tracking latency: Solved using a discrete grid system with pre-computed poses to reduce processing delays. Involved in Phase 1 and Phase 2.

Challenge 2: TurtleBot4-PincherX100 coordination: Addressed through ROS2 status topics and manual coordination between the mobile base and robotic arm.

Challenge 3: Consistent pick-and-place accuracy — Achieved by using a fixed delivery pose combined with manual gesture control for precise positioning. Unknown limit of pinctures, gipper limits, and weight of predefined objects

Challenge 4: Unknown object handling — Implemented color-based sorting with visual feedback to identify and categorize objects without prior knowledge. Colors of objects.

Challenge 5: Workspace collision avoidance — Pre-validates all grid cells using MoveIt collision checking and includes an emergency stop for unexpected contact. We use Moveit2

Challenge 6: Object slippage in gripper — Uses force feedback monitoring and adaptive grip strength based on object weight to maintain secure grasping. Controlling the Torque in the servos using the functions available in the InterbotixSDK and playing around with it

These challenges cover the key integration issues between gesture recognition, navigation, manipulation, and safety in our Industrial orbot, performing an autonomous delivery and sorting system.

Team IDRIS — Isaac Premkumar, Abdul Rahman, Rongxuan Zhang, Xian Li (Affiliated with Northeastern University)

Full Demo (IDRIS)

[embed]