From CheatCode to Cross-Modal Attention: Building MgACT for Contact-Rich Manipulation

The Challenge: When Vision Isn’t Enough

In robotic manipulation, particularly in tasks like precise cable insertion (the focus of the AI of Industry Challenge), standard vision-based policies often hit a wall. As the robot’s end-effector approaches the target port, the most critical part of the operation — the actual insertion — is heavily occluded by the robot’s own arm and the cable itself.

When you can’t see, you have to feel. This realization led us to move beyond standard Vision-Language-Action (VLA) models and build MgACT, a novel cross-modal architecture designed to seamlessly fuse visual information with high-fidelity haptic feedback.

Gathering the Data: 500 Episodes with CheatCode

The foundation of any good imitation learning policy is high-quality demonstration data. To collect this, we developed a custom teleoperation policy based on the “CheatCode” methodology.

Over an intensive collection period on AWS EC2 instance, we gathered 500 expert episodes. The resulting dataset is a goldmine: each episode contains perfectly synchronized 16.7 Hz streams of 3-camera RGB vision, 21-D proprioceptive joint states, and crucial 6-DOF force/torque (wrench) readings. We also capture a 19-dimensional variable-impedance action space, directly predicting compliance to ensure gentle, safe insertions.

The MgACT Architecture

With our data in hand, we designed Modality Gated Action Chunking Transformer (MgACT). Building upon the robust Action Chunking with Transformers (ACT) framework, we introduced several key innovations to handle the intricacies of contact-rich tasks:

1. Cross-Modal Token Fusion:

We process our three camera streams through a shared ResNet-18 backbone and our 8-step haptic wrench window through a 1D-CNN. Instead of simple concatenation, we employ bidirectional cross-attention. Visual tokens attend to haptic tokens, and haptic tokens attend to visual tokens, creating deeply fused representations before they ever hit the ACT transformer encoder.

2. Contact-Conditioned Modality Dropout:

To force the network to rely on haptics when occlusion occurs, we implemented a clever regularizer. During training, whenever the recorded force exceeds a contact threshold (e.g., > 5 Newtons), we aggressively mask out the visual tokens. This directly simulates occlusion and prevents causal confusion where the model becomes overly reliant on vision.

3. Auxiliary Understanding Heads:

We added auxiliary losses to ensure the model actually understands the physics of the task. A reconstruction head forces the network to predict the haptic wrench window, while a phase-classification head predicts whether the robot is in free-space, approaching, or making contact.

Early Results and Looking Ahead

Training an architecture of this complexity requires significant compute, and we are currently running our jobs on Colab L4 GPUs.

The early training logs are incredibly promising. Within just the first epoch, the model demonstrated rapid convergence:

Epoch 8/30 train_loss=1.1697 val_loss=0.0217 val_L_action=0.0156

Achieving an L1 action validation loss of 0.0156 this early indicates that the cross-modal fusion is effectively learning the expert distribution without collapsing.

We are currently completing the full 30-epoch training run and expect to extract the best weights soon and evaluate them in the simulation.