What Does It Actually Take for a Vision System to Understand How an Object Can Be Used?

Let me share a CVPR 2026 paper that digs into a deceptively simple question: when a vision system “gets” how an object can be used, what ability is actually doing the work? The “how it can be used” here is affordance — a knife can be gripped, a cup can be grasped, a button can be pressed.

The authors break this down into two complementary abilities. One is geometric perception, responsible for identifying the structural parts of an object that support interaction — for instance, which segment of a knife is the handle. The other is interactive perception, which models how an action actually operates on those parts. Their claim is that real affordance understanding hinges precisely on these two abilities.

Instead of training a new model, they take off-the-shelf vision foundation models and probe them, asking which model each ability actually lives in.

The finding: self-supervised ViTs like DINO already encode part-level geometric structure — they know roughly which chunks an object decomposes into. And generative models like Flux, when conditioned on a given verb, hide a rich, verb-bound spatial attention inside their cross-attention maps. Feed in “grasp the cup,” and the attention naturally collapses onto the handle. That’s essentially an implicit interaction prior.

The key move is that the authors show these two dimensions aren’t just correlated — they’re building blocks you can assemble affordance out of.

They run PCA on the DINO features, pull out the principal directions of shape variation, and get a handful of interpretable vectors, each corresponding to a class of part. These are the part-level geometric prototypes. Combine all the vectors and you get a projection matrix; once it’s fixed, any other image can just be projected through that same matrix.

That heatmap then gets aligned and fused with Flux’s interaction attention— the whole thing training-free and zero-shot, no training, no annotation.

And just from stitching these together, on the UMD and AGD20K benchmarks, the results on affordance segmentation and affordance heatmaps end up matching or even beating weakly-supervised methods. So affordance turns out to be a combination of two abilities that already exist inside foundation models — geometry and interaction — not something you necessarily have to train from scratch.

| Find papers faster on arXivSub with AI summary (CVPR、ICCV、ECCV、ICML、ICLR、NeurIPS、AAAI、MICCAI、CoRL、RSS、ACL、EMNLP、ICRA、IROS、IJCAI)