Before VLAs Had a Name: My 2020 Forecast on General-Purpose Robotics

I ran across this talk I gave at MSR’s 2020 Disruptive Technology Review, a highly curated annual event for the senior leadership of Microsoft. The event is meant to highlight non-obvious technological advances and opportunities for the next decade. Below is the complete script of one of the final iterations on my talk. Five years later, my predictions and timeline seem to hold well!

This talk is about an approaching disruption that will radically change the landscape of robotic manipulation.

The market for human-safe robotic manipulators is growing exponentially at more than 30% compound annual growth rate. Capital investment in robotic manipulation startups already exceeds 700 million this year. Human-safe robotic arms are becoming commodities, the result of hardware convergence and increased competition. Robotic hands are getting better, approaching human-level dexterity. Robotic manipulation is moving to new application domains, like agriculture, healthcare, supply chain and many more.

There has been remarkable progress on the software side too. Analytical control methods and probabilistic path planners made it possible to use robotic arms in open environments. Data-driven methods have advanced the state of the art in grasping and in-hand manipulation.

These examples, by themselves, might seem rather unconvincing. Most of these demo videos are sped up, and the experiments are narrow, carefully selected and done under lab conditions. Seemingly trivial tasks require several PhD years and enormous compute. But looking at them in isolation risks missing the forest for the trees.

All manipulation tasks share the same fundamental structure. Picking tomatoes is not that different from packing cosmetics or arranging hospital supplies, or even mixing food or pouring water. The physical world evolves according to a set of rules that children are able to capture intuitively from a very early age.

You can probably guess what I’m about to say.

The big, looming disruption in physical manipulation is a pretrained, deep model that can enable all of these tasks and more, a physical manipulation version of GPT3, which I call the Artificial Motor Cortex, or AMC. There is a real possibility that this disruption could happen over the next 5 years.

To be clear, we are not talking about a human brain equivalent. Dexterous manipulation is well within the capability of an octopus with about 500 million neurons and 1 trillion synapses. GPT3 proved that, with enough data and capacity, a single pretrained network can achieve excellent performance on many different tasks. Recent work from Google on Visual Transformers shows that these properties carry over to the visual domain too. Given the complexity of learning from pixels, we can expect AMC to be one or two orders of magnitude bigger than GPT3. The state of the art in manipulation is nowhere close to that.

Whatever the architecture, we do know what we want AMC to do. Like GPT3, it should be fundamentally a generative model, but one over the visual, tactile and motor domains. It should model the world through the prism of embodied manipulation, and capture essential properties like object permanence, affordances and intuitive physics. It should be able to imagine causal explanations for a given world state and produce plans for moving the world to a desired state. Getting there will require a massive engineering effort and will likely take months of cloud-scale resources to train. However, once trained, fine-tuning it for a specific manipulation task should take significantly less effort.

Whoever succeeds at this will have an incredible asset, one that can be provided as a service and used over and over again. Today, even training a task-specific model is incredibly difficult and expensive, if at all possible. To compensate, each commercial manipulation application is addressed as a one-off, with painstaking hardware design and many limitations. By replacing training with cheap fine-tuning, AMC will enable a much wider variety of motor skills on general purpose hardware. This will open the door to countless new physical manipulation applications, from personal to industrial and from assistive to autonomous.

The market forces driving automation are only going to get stronger. The labor force will continue to shrink due to aging and immigration policies, and the demand for automation will only get bigger. The hardware is already capable of human-level dexterity. The missing piece is AMC. We, along with Amazon, Nvidia and Google, are well positioned to make this happen. The question is, who’s going to get there first?