Part 6: Inside m{ai}geXR-3d-mcp — Architecture Deep Dive

How WebSocket broadcasting, per-framework adapters, state management, and creative AI prompts work together to enable real-time multi-framework 3D scene control

📖 The m{ai}geXR Journey 0–1:

• Part 1: The Future of AI-Powered 3D Development on iOS
• Part 2: Multi-Provider AI meets Multi-Framework XR on iOS
• Part 3: Local Development Meets Universal Platform Vision
• Part 4: From Multi-Platform Dreams to Production Reality
• Part 5: Welcome to m{ai}geXR — The Evolution of AI-Powered XR Development

The Challenge: Four Frameworks, One Truth

Building m{ai}geXR-3d-mcp meant solving a deceptively hard problem: How do you let AI control four different 3D engines simultaneously while maintaining visual consistency and creative intelligence?

The constraints were brutal:

• Each framework has completely different APIs (Three.js != A-Frame != Babylon.js != React Three Fiber)
• Coordinate systems vary ({x,y,z} objects vs “x y z” strings vs [x,y,z] arrays)
• Material models are incompatible (Three.js MeshStandardMaterial vs Babylon StandardMaterial)
• The AI must generate framework-specific code but maintain a single source of truth
• Scene updates must broadcast to all connected clients in real-time
• VR chat must work seamlessly across desktop and headset modes

This article is the deep dive — the architecture, design decisions, and technical details behind how we made it work.

System Architecture Overview

┌─────────────────────────────────────────────────────────────────┐
│                  MCP Host (Copilot / Claude / Cursor)           │
│                                                                  │
│  User: "Create a glowing portal with neon lights"               │
└──────────────────────────┬──────────────────────────────────────┘
                           │ stdio / JSON-RPC
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│                    MCP Server (Node.js)                          │
│  Port: stdio (MCP) + WebSocket Server (:8083)                   │
│                                                                  │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │  handlers/                                                 │  │
│  │   └─ toolHandler.ts    → 33 MCP tool definitions          │  │
│  │   └─ promptHandler.ts  → Creative AI prompts               │  │
│  └───────────────────────────────────────────────────────────┘  │
│                                                                  │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │  state/                                                    │  │
│  │   └─ SceneStateManager.ts → Canonical scene state          │  │
│  │   └─ UndoStack.ts         → 20-deep snapshot history       │  │
│  └───────────────────────────────────────────────────────────┘  │
│                                                                  │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │  chat/                                                     │  │
│  │   └─ ChatRelay.ts         → 9 AI providers (OpenAI, etc.)  │  │
│  │   └─ MessageQueue.ts      → In-world message buffer        │  │
│  └───────────────────────────────────────────────────────────┘  │
│                                                                  │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │  ws/                                                       │  │
│  │   └─ WebSocketServer.ts   → Broadcast coordinator          │  │
│  │   └─ adapters/                                             │  │
│  │       ├─ ThreeJSAdapter.ts    → {x,y,z} objects            │  │
│  │       ├─ AFrameAdapter.ts     → "x y z" strings            │  │
│  │       ├─ BabylonAdapter.ts    → degrees → radians          │  │
│  │       └─ R3FAdapter.ts        → [x,y,z] tuples             │  │
│  └───────────────────────────────────────────────────────────┘  │
│                                                                  │
│  Broadcast Flow:                                                │
│    MCP Tool Call → State Update → Adapter Transform →          │
│    WebSocket Broadcast → All Connected Clients                 │
└──────────────────┬─────────────┬─────────────┬─────────────────┘
                   │             │             │
        WebSocket  │  WebSocket  │  WebSocket  │  WebSocket
                   ▼             ▼             ▼             ▼
         ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
         │  Three.js    │ │  A-Frame     │ │  Babylon.js  │ │  R3F/React   │
         │  :5173       │ │  :5174       │ │  :5175       │ │  :5176       │
         │              │ │              │ │              │ │              │
         │ ┌──────────┐ │ │ ┌──────────┐ │ │ ┌──────────┐ │ │ ┌──────────┐ │
         │ │dispatcher│ │ │ │dispatcher│ │ │ │dispatcher│ │ │ │dispatcher│ │
         │ │  ↓       │ │ │ │  ↓       │ │ │ │  ↓       │ │ │ │  ↓       │ │
         │ │Scene Mgr │ │ │ │Scene Mgr │ │ │ │Scene Mgr │ │ │ │Zustand   │ │
         │ │  ↓       │ │ │ │  ↓       │ │ │ │  ↓       │ │ │ │Store     │ │
         │ │Renderer  │ │ │ │Renderer  │ │ │ │Renderer  │ │ │ │  ↓       │ │
         │ │          │ │ │ │          │ │ │ │          │ │ │ │Renderer  │ │
         │ ├──────────┤ │ │ ├──────────┤ │ │ ├──────────┤ │ │ ├──────────┤ │
         │ │VR Chat   │ │ │ │VR Chat   │ │ │ │VR Chat   │ │ │ │VR Chat   │ │
         │ │Panel     │ │ │ │Panel     │ │ │ │Panel     │ │ │ │Panel     │ │
         │ └──────────┘ │ │ └──────────┘ │ │ └──────────┘ │ │ └──────────┘ │
         └──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘

Layer 1: MCP Protocol & Tool Definitions

At the foundation, m{ai}geXR-3d-mcp is a Model Context Protocol (MCP) server. This means it speaks the standardized JSON-RPC protocol that AI assistants (GitHub Copilot, Claude Desktop, Cursor) understand.

Tool Handler Architecture

The handlers/toolHandler.ts file defines 33 MCP tools that the AI can call. Each tool specifies:

• name — e.g., createObject, setCamera, animateObject
• description — What the tool does (visible to the AI)
• inputSchema — JSON Schema defining required/optional parameters
• handler function — TypeScript code that executes when the AI calls the tool

Example: createObject Tool

{
  name: 'createObject',
  description: 'Create a 3D object (box, sphere, cylinder, etc.) in the scene',
  inputSchema: {
    type: 'object',
    properties: {
      id: { type: 'string', description: 'Unique identifier' },
      type: {
        type: 'string',
        enum: ['box', 'sphere', 'cylinder', 'cone', 'torus', ...18 total],
        description: 'Geometry type'
      },
      position: {
        type: 'object',
        properties: { x: {type: 'number'}, y: {type: 'number'}, z: {type: 'number'} }
      },
      material: {
        type: 'object',
        properties: {
          color: { type: 'string' },
          metalness: { type: 'number' },
          roughness: { type: 'number' },
          emissive: { type: 'string' },
          emissiveIntensity: { type: 'number' },
          opacity: { type: 'number' }
        }
      }
      // ...scale, rotation, visibility, shadows, etc.
    },
    required: ['id', 'type', 'position']
  }
}

Why This Matters

The AI doesn’t need to know how the objects are created — it just needs to know what parameters are valid. The JSON Schema acts as a contract between the AI and the server.

When the AI calls createObject, the server:

Step 1: Validates Input

Checks that the parameters match the schema (type-safe validation).

Step 2: Updates Canonical State

Adds the object to SceneStateManager (the single source of truth).

Step 3: Saves Snapshot

Pushes current state to the undo stack for rollback capability.

Step 4: Broadcasts Command

Sends the command through adapters to all connected WebSocket clients.

Layer 2: Scene State Management

The state/SceneStateManager.ts file maintains the canonical representation of the 3D scene. This is the single source of truth that all four framework clients sync against.

Data Structure

interface SceneState {
  objects:     Record;      // All 3D objects by ID
  lights:      Record;       // All lights by ID
  particles:   Record;      // Particle systems by ID
  behaviors:   Record;      // Continuous effects by ID
  camera:      SceneCamera;                      // Camera position/target/FOV
  environment: EnvironmentDef;                   // Background/fog/postFX
  animations?: Record;     // Persistent animations
}

interface SceneObject {
  id: string;
  type: 'box' | 'sphere' | 'cylinder' | ...;    // 18 geometry types
  position: { x: number; y: number; z: number };
  scale?: { x: number; y: number; z: number };
  rotation?: { x: number; y: number; z: number };  // Degrees (server-side)
  material?: {
    color?: string;               // Hex color
    metalness?: number;           // 0-1
    roughness?: number;           // 0-1
    emissive?: string;            // Hex color
    emissiveIntensity?: number;   // Brightness multiplier
    opacity?: number;             // 0-1
    wireframe?: boolean;
  };
  visible?: boolean;
  castShadow?: boolean;
  receiveShadow?: boolean;
  parentId?: string;              // For object grouping/hierarchies
}

State Update Flow

When the AI modifies the scene:

1. Tool handler receives parameters from MCP JSON-RPC call
2. Scene State Manager validates and merges changes into canonical state
3. Undo Stack captures snapshot before the change (20-deep history)
4. WebSocket server broadcasts the update to all connected clients
5. Each client adapter transforms the command for its framework
6. Client dispatchers route commands to scene managers
7. Frameworks render the changes in their own rendering loops

Key Insight: The server never touches Three.js, Babylon.js, or any rendering code. It only manages abstract scene state. The clients are responsible for translating that state into framework-specific rendering calls.

Layer 3: WebSocket Broadcasting & Adapters

The ws/WebSocketServer.ts file manages real-time bidirectional communication between the MCP server and all connected clients.

Connection Flow

// Client connects with framework identifier
ws://localhost:8083?framework=threejs
ws://localhost:8083?framework=aframe
ws://localhost:8083?framework=babylonjs
ws://localhost:8083?framework=r3f

// Server registers client and selects appropriate adapter
const adapter = adapters[framework];  // ThreeJSAdapter, AFrameAdapter, etc.
clients.set(clientId, { ws, framework, adapter });
// When broadcasting commands:
for (const client of clients.values()) {
  const transformed = client.adapter.transform(command);
  client.ws.send(JSON.stringify(transformed));
}

Per-Framework Adapters

Each adapter implements a transform(command) method that converts the canonical command format into framework-specific representation.

Three.js Adapter (ws/adapters/ThreeJSAdapter.ts)

class ThreeJSAdapter {
  transform(cmd: Command): Command {
    // Three.js expects {x, y, z} objects
    // Rotation is in degrees (Three.js uses radians but client converts)
    // Material properties map directly to MeshStandardMaterial
    return cmd;  // Minimal transformation needed
  }
}

A-Frame Adapter (ws/adapters/AFrameAdapter.ts)

class AFrameAdapter {
  transform(cmd: Command): Command {
    // A-Frame expects "x y z" string format
    if (cmd.position) {
      cmd.position = `${cmd.position.x} ${cmd.position.y} ${cmd.position.z}`;
    }
    if (cmd.rotation) {
      cmd.rotation = `${cmd.rotation.x} ${cmd.rotation.y} ${cmd.rotation.z}`;
    }
    if (cmd.scale) {
      cmd.scale = `${cmd.scale.x} ${cmd.scale.y} ${cmd.scale.z}`;
    }
    // Material properties remain as objects (A-Frame components accept both)
    return cmd;
  }
}

Babylon.js Adapter (ws/adapters/BabylonAdapter.ts)

class BabylonAdapter {
  transform(cmd: Command): Command {
    // Babylon.js needs degrees converted to radians
    // (Client-side BabylonSceneManager handles the conversion)
    // Material naming: Three.js "roughness" = Babylon "roughness"
    // But MeshStandardMaterial != StandardMaterial (different defaults)
    return cmd;  // Client handles rotation conversion
  }
}

React Three Fiber Adapter (ws/adapters/R3FAdapter.ts)

class R3FAdapter {
  transform(cmd: Command): Command {
    // R3F expects {x, y, z} objects (converts to tuples internally)
    // State is managed via Zustand store (reactive updates)
    return cmd;  // R3F client normalizes to React patterns
  }
}

Broadcast Strategy

The server uses a fan-out broadcast pattern:

1. MCP tool handler updates SceneStateManager
2. WebSocketServer.broadcast(command) is called
3. Server iterates through all connected clients
4. For each client, applies the appropriate adapter transform
5. Sends transformed command via WebSocket

Performance Note: Broadcasts are asynchronous and non-blocking. The MCP tool handler returns immediately after updating state — WebSocket sends happen in parallel without blocking the MCP response.

Layer 4: Client-Side Dispatch & Rendering

Each of the four clients follows the same pattern:

1. WebSocket listener receives command from server
2. Dispatcher routes command to appropriate handler (createObject → scene.createObject)
3. Scene manager translates command to framework-specific API calls
4. Renderer updates the 3D scene (requestAnimationFrame loop)

Three.js Client Flow

// packages/client-threejs/src/main.ts
ws.onmessage = (event) => {
  const cmd = JSON.parse(event.data);
  dispatch(sceneManager, cmd);  // Route to dispatcher
};

// packages/client-threejs/src/commands/dispatch.ts
function dispatch(scene: ThreeSceneManager, cmd: Command) {
  switch (cmd.action) {
    case 'createObject':
      scene.createObject(cmd as SceneObject);
      break;
    case 'setCamera':
      scene.setCamera(cmd as CameraCommand);
      break;
    // ...33 total command types
  }
}
// packages/client-threejs/src/scene/SceneManager.ts
class ThreeSceneManager {
  createObject(def: SceneObject) {
    let geometry;
    switch (def.type) {
      case 'box':
        geometry = new THREE.BoxGeometry(
          def.scale?.x ?? 1,
          def.scale?.y ?? 1,
          def.scale?.z ?? 1
        );
        break;
      case 'sphere':
        geometry = new THREE.SphereGeometry(def.radius ?? 0.5, 32, 32);
        break;
      // ...18 geometry types
    }
    const material = new THREE.MeshStandardMaterial({
      color: new THREE.Color(def.material?.color ?? '#ffffff'),
      metalness: def.material?.metalness ?? 0.3,
      roughness: def.material?.roughness ?? 0.7,
      emissive: new THREE.Color(def.material?.emissive ?? '#000000'),
      emissiveIntensity: def.material?.emissiveIntensity ?? 0,
      opacity: def.material?.opacity ?? 1,
      transparent: (def.material?.opacity ?? 1) < 1,
      wireframe: def.material?.wireframe ?? false,
    });
    const mesh = new THREE.Mesh(geometry, material);
    mesh.position.set(def.position.x, def.position.y, def.position.z);
    mesh.rotation.set(
      THREE.MathUtils.degToRad(def.rotation?.x ?? 0),
      THREE.MathUtils.degToRad(def.rotation?.y ?? 0),
      THREE.MathUtils.degToRad(def.rotation?.z ?? 0)
    );
    mesh.castShadow = def.castShadow ?? true;
    mesh.receiveShadow = def.receiveShadow ?? true;
    this.scene.add(mesh);
    this.meshes.set(def.id, mesh);  // Store reference for updates
  }
}

Babylon.js Client Flow (Different API, Same Pattern)

// packages/client-babylonjs/src/scene.ts
class BabylonSceneManager {
  createObject(def: SceneObject) {
    let mesh;
    switch (def.type) {
      case 'box':
        mesh = BABYLON.MeshBuilder.CreateBox(def.id, {
          width: def.scale?.x ?? 1,
          height: def.scale?.y ?? 1,
          depth: def.scale?.z ?? 1,
        }, this.scene);
        break;
      case 'sphere':
        mesh = BABYLON.MeshBuilder.CreateSphere(def.id, {
          diameter: (def.radius ?? 0.5) * 2,
          segments: 32,
        }, this.scene);
        break;
      // ...18 geometry types
    }

const material = new BABYLON.StandardMaterial(def.id + '_mat', this.scene);
    material.diffuseColor = BABYLON.Color3.FromHexString(def.material?.color ?? '#ffffff');
    material.specularColor = BABYLON.Color3.Black();  // Babylon-specific
    material.emissiveColor = BABYLON.Color3.FromHexString(def.material?.emissive ?? '#000000');
    material.alpha = def.material?.opacity ?? 1;
    material.wireframe = def.material?.wireframe ?? false;
    // Babylon uses radians for rotation (server sends degrees)
    mesh.position = new BABYLON.Vector3(def.position.x, def.position.y, def.position.z);
    mesh.rotation = new BABYLON.Vector3(
      toRadians(def.rotation?.x ?? 0),
      toRadians(def.rotation?.y ?? 0),
      toRadians(def.rotation?.z ?? 0)
    );
    mesh.material = material;
    mesh.receiveShadows = def.receiveShadow ?? true;
    this.meshes.set(def.id, mesh);
  }
}

React Three Fiber Client Flow (Reactive State)

// packages/client-r3f/src/store/sceneStore.ts (Zustand store)
interface SceneStore {
  objects: Record;
  lights: Record;
  camera: SceneCamera;
  environment: EnvironmentDef;
  createObject: (def: SceneObject) => void;
  updateObject: (updates: Partial & {id: string}) => void;
  deleteObject: (id: string) => void;
  // ...
}
const useSceneStore = create((set) => ({
  objects: {},
  lights: {},
  camera: { position: {x:0, y:5, z:10}, target: {x:0, y:0, z:0} },
  environment: {},
  createObject: (def) => set((state) => ({
    objects: { ...state.objects, [def.id]: def }
  })),
  updateObject: (updates) => set((state) => ({
    objects: {
      ...state.objects,
      [updates.id]: { ...state.objects[updates.id], ...updates }
    }
  })),
  deleteObject: (id) => set((state) => {
    const { [id]: removed, ...remaining } = state.objects;
    return { objects: remaining };
  }),
}));
// packages/client-r3f/src/components/SceneRenderer.tsx
function SceneRenderer() {
  const objects = useSceneStore((state) => state.objects);
  const lights = useSceneStore((state) => state.lights);
  return (

  );
}
// State changes trigger React re-renders, R3F updates Three.js scene graph

Layer 5: Creative AI System Prompts

This is where architecture meets artistry. The handlers/promptHandler.ts file defines two critical MCP prompts:

1. Base System Prompt (3d-world-assistant)

This is the creative director prompt that teaches the AI how to think like a visual artist:

Creative Philosophy Teaching

You are a creative 3D scene designer and spatial artist.
Think like a cinematographer, lighting designer, and architect combined.

**Lighting creates emotion:**
- Warm lights (#ff9966) = cozy, inviting, sunset
- Cool lights (#66ccff) = clinical, futuristic, moonlight
- Colored point lights = drama, neon, cyberpunk
- Low ambient + directional = high contrast, cinematic
**Composition creates interest:**
- Use the rule of thirds - don't center everything
- Create depth with foreground, midground, background layers
- Vary object sizes - large anchors, small details
- Lead the eye with lighting and object placement
**Color theory matters:**
- Complementary colors create vibrance (blue/orange, purple/yellow)
- Analogous colors feel harmonious (blues/purples, reds/oranges)
- Desaturated + one saturated accent = sophisticated focus
**Atmosphere is everything:**
- Fog adds depth and mystery (near:5, far:30 for dramatic)
- Dark backgrounds (#0a0a0f) make lights pop
- Emissive materials create mood without lights
- Subtle animations add life (slow rotation, gentle bobbing)

2. Per-Framework Prompts (framework-guide)

When a client connects, it tells the server which framework it’s using. The server injects a framework-specific prompt that teaches the AI the correct syntax and creative opportunities for that engine:

Framework Prompt Focus Three.js PBR materials for realistic metals/glass, combine torus + emissive for sci-fi portals, layer transparent planes for glass/water effects A-Frame Room-scale VR experiences, immersive galleries, spatial UI, objects sized 1–3 units for human scale Babylon.js Photorealistic architectural visualizations, advanced shadow systems, particle-heavy effects (explosions, magic, weather) React Three Fiber Data-driven art, parametric design, live 3D dashboards, component-based reusable scene pieces

Chat Relay Enhanced Prompts

When using Direct Chat Mode (with API key in .env), the chat/ChatRelay.ts system injects an even more detailed creative library:

• Postprocessing recipes: Bloom (strength, radius, threshold), vignette (darkness, offset), chromatic aberration — with pro tips like “dark backgrounds make bloom POP dramatically”
• Material recipes by style: Water (metalness 0.1, roughness 0.1, opacity 0.7), enchanted gems, hologram, carbon fiber
• Lighting recipes: Natural (sun + sky + ambient), indoor (warm point lights), dramatic (side directional + low ambient + rim lights), moody horror, sci-fi neon
• Particle recipes: Stars/snow (twinkle + additive), fireflies (glow + emissive), sparkles (high emissive Intensity)
• Behavior guidance: When to use continuous behaviors (spinning planets) vs timed animations (A→B transitions)

Layer 6: In-World Chat Architecture

The in-world chat system is what makes m{ai}geXR-3d-mcp truly revolutionary. Here’s how it works:

Message Queue (chat/MessageQueue.ts)

class MessageQueue {
  private messages: Array<{role: 'user' | 'ai', text: string, timestamp: number}> = [];
// User presses ~ in 3D viewport, types message, presses Enter
  addUserMessage(text: string) {
    this.messages.push({ role: 'user', text, timestamp: Date.now() });
  }
  // MCP host AI or ChatRelay polls for new messages
  getPending(): Array<{role: 'user', text: string}> {
    const pending = this.messages.filter(m => m.role === 'user');
    this.messages = this.messages.filter(m => m.role !== 'user');  // Clear after read
    return pending;
  }
  // AI responds via sendChatMessage tool
  addAIMessage(text: string) {
    this.messages.push({ role: 'ai', text, timestamp: Date.now() });
    // Broadcast to all clients for display in chat overlay
  }
}

Client-Side Chat Overlay

Each client has a floating chat panel that appears when you press ~ (backtick):

// packages/client-threejs/src/overlay/ChatOverlay.ts
class ChatOverlay {
  constructor() {
    this.panel = document.createElement('div');
    this.panel.id = 'chat-panel';
    this.panel.style.cssText = `
      position: fixed;
      bottom: 20px;
      left: 20px;
      width: 400px;
      max-height: 300px;
      background: rgba(10, 10, 30, 0.95);
      border: 2px solid rgba(100, 100, 220, 0.5);
      border-radius: 8px;
      padding: 15px;
      display: none;  /* Hidden by default */
      z-index: 1000;
    `;

this.input = document.createElement('input');
    this.input.placeholder = 'Type your message...';
    this.input.addEventListener('keydown', (e) => {
      if (e.key === 'Enter') {
        this.sendMessage(this.input.value);
        this.input.value = '';
      }
    });
    document.addEventListener('keydown', (e) => {
      if (e.key === '`' || e.key === '~') {
        e.preventDefault();
        this.toggle();
      }
    });
  }
  sendMessage(text: string) {
    // Send to server via WebSocket
    ws.send(JSON.stringify({
      type: 'chat',
      role: 'user',
      text
    }));
  }
  displayAIMessage(text: string) {
    const msg = document.createElement('div');
    msg.textContent = `🤖 ${text}`;
    this.messagesContainer.appendChild(msg);
  }
}

VR Chat Panel (WebXR Integration)

In VR mode, the chat panel becomes a 3D floating canvas texture:

// packages/client-babylonjs/src/scene.ts
async initXR() {
  const xr = await this.scene.createDefaultXRExperienceAsync({...});
// Create 3D plane for chat
  this.vrChatMesh = BABYLON.MeshBuilder.CreatePlane('__vr-chat', {
    width: 1.4,
    height: 0.9
  }, this.scene);
  this.vrChatMesh.position = new BABYLON.Vector3(0, 1.5, -2);  // Float in front of user
  // Render chat messages to dynamic texture
  this.vrChatTexture = new BABYLON.DynamicTexture('__vr-chat-tex', {
    width: 700,
    height: 450
  }, this.scene);
  const mat = new BABYLON.StandardMaterial('__vr-chat-mat', this.scene);
  mat.diffuseTexture = this.vrChatTexture;
  mat.emissiveTexture = this.vrChatTexture;  // Glow in dark VR
  mat.disableLighting = true;
  this.vrChatMesh.material = mat;
  // Redraw canvas when messages update
  redrawVRChat() {
    const ctx = this.vrChatTexture.getContext();
    ctx.fillStyle = 'rgba(10, 10, 30, 0.95)';
    ctx.fillRect(0, 0, 700, 450);
    // Draw messages
    let y = 56;
    for (const msg of this.vrChatMessages) {
      ctx.fillStyle = msg.role === 'user' ? '#c7d2fe' : '#86efac';
      ctx.fillText(`${msg.role === 'user' ? '🗣' : '🤖'} ${msg.text}`, 16, y);
      y += 24;
    }
    this.vrChatTexture.update();  // Upload to GPU
  }
}

Complete Request Flow: User → AI → 4 Clients

Let’s trace a complete request through the system:

1. User Types In-World Message

User presses ~ in Three.js client, types: "Create a glowing purple portal with neon lights", presses Enter.

// Client sends to server
ws.send({ type: 'chat', role: 'user', text: '...' });
// Server adds to MessageQueue
messageQueue.addUserMessage('Create a glowing purple portal...');

2. MCP Host AI Polls for Messages

GitHub Copilot (or Claude Desktop) calls the getPendingUserMessages tool.

// MCP JSON-RPC request
{
  jsonrpc: '2.0',
  method: 'tools/call',
  params: {
    name: 'getPendingUserMessages'
  }
}

// Server responds
{
  result: [
    { role: 'user', text: 'Create a glowing purple portal...' }
  ]
}

3. AI Generates Scene Commands

The AI (enhanced by creative prompts) generates a thoughtful scene:

// AI calls multiple MCP tools
[
  {
    name: 'setEnvironment',
    params: {
      background: '#0a0a1e',
      fog: { color: '#6600ff', near: 8, far: 25 },
      bloom: { strength: 0.8, threshold: 0.5, radius: 0.4 }
    }
  },
  {
    name: 'createObject',
    params: {
      id: 'portal',
      type: 'torus',
      position: {x:0, y:1.5, z:0},
      scale: {x:2, y:2, z:0.3},
      rotation: {x:90, y:0, z:0},
      material: {
        color: '#1a1a1a',
        emissive: '#ff00ff',
        emissiveIntensity: 2.0,
        metalness: 0.9
      }
    }
  },
  {
    name: 'createLight',
    params: {
      id: 'neon1',
      lightType: 'point',
      color: '#ff00ff',
      intensity: 3,
      position: {x:0, y:1.5, z:0}
    }
  },
  {
    name: 'addBehavior',
    params: {
      id: 'spin_portal',
      objectId: 'portal',
      type: 'spin',
      params: {speedY: 1}
    }
  },
  {
    name: 'sendChatMessage',
    params: {
      text: 'I\'ve created a mystical portal with glowing purple energy and neon lighting. The portal spins continuously for a hypnotic effect.'
    }
  }
]

4. Server Updates State & Broadcasts

For each tool call:

// setEnvironment
SceneStateManager.updateEnvironment({ background, fog, bloom });
UndoStack.push(snapshot);
WebSocketServer.broadcast({ action: 'setEnvironment', background, fog, bloom });
// createObject
SceneStateManager.addObject({ id: 'portal', type: 'torus', ... });
UndoStack.push(snapshot);
WebSocketServer.broadcast({ action: 'createObject', id: 'portal', ... });
// createLight
SceneStateManager.addLight({ id: 'neon1', lightType: 'point', ... });
WebSocketServer.broadcast({ action: 'createLight', id: 'neon1', ... });
// addBehavior
SceneStateManager.addBehavior({ id: 'spin_portal', objectId: 'portal', ... });
WebSocketServer.broadcast({ action: 'addBehavior', id: 'spin_portal', ... });
// sendChatMessage
MessageQueue.addAIMessage('I\'ve created a mystical portal...');
WebSocketServer.broadcast({ type: 'chat', role: 'ai', text: '...' });

5. All Four Clients Receive Broadcasts

Each client gets the same commands, but transforms them for its framework:

Three.js Client:

dispatch(sceneManager, { action: 'createObject', id: 'portal', type: 'torus', ... });
→ sceneManager.createObject({ id: 'portal', ... });
→ new THREE.TorusGeometry(...);
→ new THREE.MeshStandardMaterial({ emissive: '#ff00ff', emissiveIntensity: 2.0, ... });
→ scene.add(mesh);

A-Frame Client:

dispatch(sceneManager, { action: 'createObject', id: 'portal', type: 'torus', ... });
→ sceneManager.createObject({ id: 'portal', ... });
→ const entity = document.createElement('a-torus');
→ entity.setAttribute('position', '0 1.5 0');
→ entity.setAttribute('rotation', '90 0 0');
→ entity.setAttribute('material', 'color: #1a1a1a; emissive: #ff00ff; ...');
→ scene.appendChild(entity);

Babylon.js Client:

dispatch(sceneManager, { action: 'createObject', id: 'portal', type: 'torus', ... });
→ sceneManager.createObject({ id: 'portal', ... });
→ BABYLON.MeshBuilder.CreateTorus('portal', {...}, scene);
→ material.emissiveColor = BABYLON.Color3.FromHexString('#ff00ff');
→ mesh.rotation = new BABYLON.Vector3(toRadians(90), 0, 0);

React Three Fiber Client:

dispatch(store, { action: 'createObject', id: 'portal', type: 'torus', ... });
→ store.createObject({ id: 'portal', ... });
→ Zustand state updated: objects['portal'] = { type: 'torus', ... };
→ React re-renders:

6. Chat Message Displayed

All clients receive the sendChatMessage broadcast and display the AI's response in their chat overlays (or VR panels if in WebXR mode).

Result: Four different 3D engines render the exact same scene from a single set of AI-generated commands.

Key Architecture Decisions

1. Why WebSocket Instead of HTTP?

Bidirectional real-time communication. HTTP is request-response — the client has to poll for updates. WebSocket keeps a persistent connection, allowing the server to push updates instantly to all clients. Essential for in-world chat and multi-client synchronization.

2. Why Canonical State on the Server?

Single source of truth. If each client maintained its own state, they’d drift out of sync. The server’s SceneStateManager is the authoritative representation. Clients are views of that state.

3. Why Per-Framework Adapters?

Separation of concerns. The server doesn’t need to know about Three.js vs Babylon.js APIs. Adapters encapsulate framework-specific transformations. Adding a fifth framework (PlayCanvas, Wonderland Engine) means writing one new adapter — no changes to the server core.

4. Why Creative AI Prompts?

Quality over quantity. Without artistic guidance, the AI generates technically correct but visually bland scenes. The creative prompts teach why lighting, composition, and color matter — transforming the AI from a code generator into a design partner.

5. Why 20-Deep Undo Stack?

Experimentation safety. Users should feel free to try things without fear of losing work. The undo stack captures snapshots before every state change, allowing rollback through 20 iterations.

Performance Considerations

WebSocket Message Size

Commands are JSON-serialized. Typical sizes:

• createObject: ~300-500 bytes
• setCamera: ~150 bytes
• setEnvironment: ~400-600 bytes
• Chat message: ~100–200 bytes

Optimization: We don’t send the entire scene state on every update — only the delta (the specific command). Clients apply incremental updates.

Rendering Performance

Each client runs its own rendering loop (requestAnimationFrame at 60fps). Scene updates trigger re-renders, but:

• Three.js / Babylon.js / A-Frame: Direct DOM/scene graph manipulation — fast
• React Three Fiber: Zustand state changes trigger React reconciliation — slightly slower but still performant (React optimizes via diffing)

Behavior System Performance

Behaviors run every frame. To avoid lag:

• Math operations are simple (sin/cos for bobbing, incremental rotation)
• No heap allocations per frame (reuse Vector3 instances)
• Clients can disable behaviors if frame rate drops

Undo Stack Memory

Each snapshot stores the entire SceneState. For a typical scene:

• 10 objects: ~3KB
• 5 lights: ~500 bytes
• Camera + environment: ~500 bytes
• Total per snapshot: ~4KB
• 20-deep stack: ~80KB

Negligible memory footprint. We could increase the stack depth to 50+ without issues.

Testing & Type Safety

Comprehensive Test Suite

We’ve added 30 passing tests across three test files:

Test Coverage

• dispatch.test.ts (13 tests): Command routing verification — ensures createObject, updateObject, setCamera, etc. route to correct scene manager methods
• scene.test.ts (11 tests): Utility functions — hex to RGB color conversion, degrees to radians, easing curves (lerp, cubic ease-in-out), vector math (distance, normalization)
• integration.test.ts (6 tests): Full command pipeline — object lifecycle (create → update → animate → delete), scene composition workflows, particle systems, behaviors

100% TypeScript Type Safety

All clients now have zero TypeScript errors:

• Fixed R3F dispatch.ts:125: Added Vec3 type import and type guard for animation handling
• Fixed Babylon.js Vite config: Resolved shader loading 504 errors by configuring optimizeDeps for @babylonjs/core
• Strict type checking: All command parameters validated against interfaces

What We’ve Built: The Numbers

System Statistics (April 2026)

• 4 production-ready clients — Three.js, A-Frame, Babylon.js, React Three Fiber
• 9 AI providers — OpenAI, Anthropic, Google Gemini, Mistral, Groq, xAI/Grok, Cohere, Together.ai, Ollama
• 33 MCP tools — Complete scene control (objects, lights, camera, animation, behaviors, particles, environment, scripting, I/O, chat)
• 18 geometry types — box, sphere, cylinder, torusKnot, platonic solids, tube, ring, line, and more
• 5 behavior types — spin, bob, orbit, lookAt, pulse (continuous per-frame effects)
• 4 WebSocket adapters — Per-framework command transformation
• 4 WebXR implementations — VR support with floating 3D chat panels
• 30 passing tests — Dispatch, utilities, integration coverage
• 100% TypeScript type safety — Zero type errors across all clients
• Creative AI system — Lighting, composition, color theory, atmosphere, material/postprocessing recipes
• 20-turn conversation history — Scene-aware AI with context memory
• 20-deep undo stack — Snapshot-based rollback capability
• Real-time broadcasting — Sub-100ms latency from AI to all clients
• In-world chat — Desktop and VR, relay or direct modes

Try It Yourself

Experience the Architecture in Action

One command. Four frameworks. Nine AI providers. Zero friction.

npx maige-3d-mcp

**Clone the Repository • [Read the Docs](https://github.com/m-ai-geXR/maigeXR/tree/main/mcp-webgpu) • [Join Discussions](https://github.com/m-ai-geXR/maigeXR/discussions)**

Quick Start (5 Minutes)

# Clone repository
git clone https://github.com/m-ai-geXR/maigeXR.git
cd maigeXR/mcp-webgpu

# Install dependencies
pnpm install
# Configure (add at least one API key)
cp .env.example .env
# Build and run
pnpm build:server
pnpm dev

The server auto-opens the Three.js client at http://localhost:5173. Press ~ to open the chat overlay and start talking to the AI from inside the scene.

Final Thoughts

Part 5 was about intelligence — making the AI smarter with RAG and multimodal capabilities.

Part 6 is about architecture — how we built a system that:

• Maintains a single source of truth while supporting four different rendering engines
• Broadcasts scene changes in real-time to all connected clients
• Transforms commands per-framework without server-side rendering knowledge
• Teaches AI to be a creative design partner with lighting, composition, and color theory
• Enables in-world chat that works seamlessly in desktop and VR modes
• Provides comprehensive testing and 100% TypeScript type safety

The result is a system that feels magical to use but is solid under the hood.

“Architecture is not just about making things work — it’s about making them work beautifully, reliably, and extensibly.”

We’ve built the foundation. Now it’s time to build on it.

Join us. 🚀✨