What TouchDesigner Can and Cannot Do for XR and AR Development

TouchDesigner is widely used in interactive installations and real-time visual work, but many creators still wonder how it fits into AR or XR workflows.

This article summarizes what TouchDesigner can and cannot do for AR, how it communicates with mobile devices, and how to combine it with Unity or 8thWall for practical XR projects.

What is TouchDesigner?

It’s my first learning step of TouchDesigner!

TouchDesigner is a node-based visual programming tool used for interactive art, installations, real-time visuals, projection mapping, and sensor-based experiences.

It runs locally on a computer and generates visuals or data in real time.

What TouchDesigner can or cannot do?

TouchDesigner can…

• Generate real-time visuals and animations
• Process camera input (motion, face detection using AI TOPs)
• Receive sensor data (Arduino, MIDI, OSC devices)
• Send data to external apps via OSC, WebSocket, TCP/UDP
• Output live video streams (NDI, Spout/Syphon, RTMP)

But,

• It does not provide AR functions (no plane tracking, image tracking, SLAM, etc.)
• It does not run on the web and cannot directly power a website
• It cannot export 3D models for use in other engines
• It requires a local machine; it is not cloud-friendly

Inputs Output

Inputs to TouchDesigner are…

• Webcam video
• Sensors (Arduino, IMU, distance sensors, etc.)
• Microphones / audio signals
• OSC / WebSocket / MIDI / serial input
• Images, videos, shaders

And outputs from TouchDesigner are…

• Real-time video (as a texture stream or projection)
• Data values (OSC, WebSocket, UDP/TCP, MIDI)
• Processed camera tracking values (face points, motion amount)
• Audio-reactive values

Use Cases for XR (AR/VR)

1. AR powered by real-time data from TouchDesigner

TouchDesigner → sends OSC/WebSocket data AR app (Unity / 8thWall) → changes effects based on that data

Examples:

• Audio-reactive AR effects (particles, colors, lights)
• Sensor-driven AR (distance sensor, light sensor, gesture devices)
• “World state” controlled by environment (TD reads sensors → AR changes scene)

2. Real-time video from TouchDesigner used as textures in AR

TD generates a moving pattern or effect → send via NDI/RTMP → AR engine uses it as a dynamic texture in the 3D scene.

3. Installation-style interactive XR

TouchDesigner acts as the interaction hub:

• It reads sensors, microphones, cameras
• It processes and converts those signals
• It sends unified data to the AR application

Useful for exhibitions or public installations.

What TouchDesigner is NOT good for

• Web-based face tracking on websites
• Mobile-only AR experiences
• Cloud-hosted interactivity (These are better achieved with 8thWall, ARKit/ARCore, or Web APIs.)

How to Develop? (Practical Notes)

TouchDesigner is designed to run on a local PC. It is not meant for cloud deployment or running directly on mobile devices.

If you want to connect a smartphone or tablet, both devices simply need to be on the same Wi-Fi network, and you can communicate via OSC, WebSocket, or UDP/TCP.

TouchDesigner cannot handle AR tracking (plane detection, image tracking, SLAM). For AR functionality, you still need AR Foundation, ARKit/ARCore, or 8thWall.

TouchDesigner’s main role in an AR workflow is: generate visuals and send real-time data to the AR app. For building the AR app itself, Unity + AR Foundation is likely the more reliable option, especially for performance. (8thWall works too, but depending on the complexity, it may use more memory and be less stable for large, dynamic scenes.)

[Check out TouchDesigner “AR filter tutorials” here](https: //www.youtube.com/watch?v=NJbSm9UbCqk&t=13s)

However, these still run entirely on a local PC webcam — they do not provide mobile AR features.

I first heard about TouchDesigner from a friend, and it got me thinking about using it in my art projects. I was especially curious about using sensors in AR experiences, where you can interactively communicate with a fantasy world.

I want to learn more about TouchDesigner and bring new ideas to life!