AI for Frontend Developers — Day 64

AI for Frontend Developers — Week 10 Day 1

Teaching My AI to Understand Images (Vision AI)

Until now, my AI system could understand:

• Text ✅
• Voice input ✅
• Voice output ✅

But users still had one major limitation:

they could only describe things using words

Today I added one of the biggest upgrades in the entire project:

Vision AI

Now users can:

• upload screenshots
• upload diagrams
• upload UI designs
• upload photos
• ask questions about images directly

This transformed the application from a traditional chatbot into a:

multimodal AI system

🚀 Today’s Goal

Build a complete:

Image → AI Understanding → Response

pipeline.

Final flow:

User uploads image
↓
Frontend converts image to Base64
↓
Backend sends multimodal request
↓
Vision model analyzes image
↓
AI responds with understanding

Existing Upload Architecture Saved HUGE Time

The best part?

I already had:

• file upload system
• preview system
• attachment UI
• upload validation
• file persistence

from previous document upload implementation.

So instead of building:

• separate image infrastructure
• separate image upload flow

I simply extended the existing architecture into:

multimodal upload infrastructure

This was a huge realization about good system design.

🚀 Extending Allowed File Types

I already supported:

• PDFs
• text files

Now I extended support for images.

const allowedTypes = [
  "application/pdf",
  "text/plain",
  "image/png",
  "image/jpeg",
  "image/jpg",
  "image/webp",
];

Detecting Image Uploads

Inside the upload handler, I added image detection logic.

if (file.type.startsWith("image/")) {
  setSelectedImage(file);
}

This became the key state for Vision AI processing.

The Biggest Bug I Faced

Initially, image uploads worked perfectly:

• preview worked
• attachment worked
• upload worked

But the AI kept replying:

"I can't analyze images directly..."

Even though I was sending images.

After debugging carefully, I discovered the REAL issue:

selectedImage was never being set

So:

• Base64 conversion never happened
• image was never sent to backend
• OpenAI received only plain text

This was one of those bugs where:

• UI looked completely correct
• backend looked mostly correct
• but one missing frontend state broke the entire feature

Converting Images to Base64

Vision models commonly accept:

• image URLs
• Base64 image data

I used Base64 conversion with FileReader.

const convertToBase64 = (file) => {
  return new Promise((resolve, reject) => {
    const reader = new FileReader();
    reader.readAsDataURL(file);
    reader.onload = () => {
      resolve(reader.result);
    };
    reader.onerror = (error) => {
      reject(error);
    };
  });
};

This converts images into:

data:image/png;base64,...

which OpenAI vision models can process directly.

Sending Image to Backend

Before the API request, I generated Base64 image data.

let imageBase64 = null;

if (selectedImage) {
  imageBase64 =
    await convertToBase64(
      selectedImage
    );
}

Then I included it in the request body.

body: JSON.stringify({
  messages: updatedMessages,
  mode,
  chatId,
  image: imageBase64,
}),

The Most Important Backend Change

Traditional OpenAI requests use:

content: "plain text"

But Vision AI requires:

content: [
  {
    type: "text",
    text: "..."
  },
  {
    type: "image_url",
    image_url: {
      url: image
    }
  }
]

This was the core multimodal architecture change.

Building Multimodal User Content

I created dynamic user content based on whether image exists.

const userContent = image
  ? [
      {
        type: "text",
        text: userQuestion,
      },
      {
        type: "image_url",
        image_url: {
          url: image,
        },
      },
    ]
  : userQuestion;

Now:

• text-only chats still work normally
• image chats become multimodal automatically

Creating OpenAI Messages Properly

Another important realization:

I could NOT directly use:

...recentMessages

because the latest message needed:

• multimodal structure
• image content array

So I created a reusable message structure.

const openAIMessages = [
  {
    role: "system",
    content: finalSystemPrompt,
  },
  ...recentMessages.slice(0, -1),
  {
    role: "user",
    content: userContent,
  },
];

This became the centralized message architecture for:

• normal chats
• streaming chats
• tool flows
• vision flows

Another Critical Bug

I fixed the first OpenAI API call…

but Vision AI STILL failed.

Why?

Because my streaming API calls were still using:

...recentMessages

instead of:

• multimodal message arrays

This meant:

• initial call saw image
• final response generation lost image context

The fix was:

• using openAIMessages
• consistently across ALL OpenAI calls

This was a very important debugging lesson.

🚀 Final Result

My AI can now:

• Understand screenshots ✅
• Analyze UI designs ✅
• Explain diagrams ✅
• Process uploaded images ✅
• Combine text + image prompts ✅
• Handle multimodal conversations ✅

Examples:

"What is wrong in this UI?"
"Explain this architecture diagram"
"What error is shown in this screenshot?"
"What is this image about?"

This was one of the biggest capability jumps in the project so far.

🧠 Key Learnings

• Vision AI requires multimodal message structures instead of plain text content.
• OpenAI vision models accept Base64 image data directly.
• Multimodal systems require careful frontend + backend coordination.
• Small missing frontend state can completely break multimodal workflows.
• Existing upload infrastructure can evolve into powerful multimodal architecture.
• All OpenAI calls must consistently use multimodal message formatting.
• Vision AI is much more than OCR — it enables visual reasoning and contextual understanding.

🔗 Live Demo & Code

👉 Live App: https://ai-chat-app-learning.netlify.app

👉 GitHub Repo: https://github.com/RohitKuwar/ai-chat-app/tree/feature/vision-ai-image-understanding

🚀 What’s Next

Today my AI evolved from a text-and-voice assistant into a true multimodal AI capable of understanding visual information.

Next, I’ll improve the image experience further with better multimodal memory, visual context handling, and advanced image interactions 🚀

I’ll be posting here daily as I learn. Let’s grow together and stay ahead in the AI era 🚀

Happy learning ✌️