Sharing My Real-World Experience Testing an AI-Based Chatbot Scheduler.

Artificial intelligence is rapidly becoming a core component of customer service, with many companies replacing traditional rule-based chatbots with AI-powered conversational agents. One of the most interesting and challenging projects in my QA career involved testing such an AI-driven scheduling assistant — a system designed to help customers book repair appointments in real time.

In this article, I’ll share my learnings, testing techniques, and the practical challenges of validating an AI-based conversational agent.

Understanding the AI Scheduling Agent

The assistant I tested was far from a simple scripted chatbot. It could:

• Interact with users through natural, conversational messages
• Fetch real-time data from backend systems
• Offer available repair time slots
• Help users create, update, or cancel appointments
• Handle multiple user journeys dynamically
• Communicate via WebSockets for instant message exchange

My primary responsibility was to ensure the end-to-end experience was correct, stable, and user-friendly — regardless of how the AI behaved internally.

🔹 1. UI & Conversation Flow Testing

My first focus area was validating the front-end chat experience. Even though the AI decided the responses, I ensured the UI handled:

• Correct message order
• Accurate questions and prompts
• Helpful error or fallback messages
• Clear flow for collecting user details
• Smooth navigation through the full conversation journey

I also used Selenium for functional automation testing, which allowed me to automate repetitive conversation scenarios, validate UI behavior, and quickly check multiple user journeys for correctness and stability.

Key scenarios I tested:

• First-time user repair booking
• Returning user with an existing appointment
• Browser refresh mid-conversation
• Interrupted user inputs
• Incorrect or missing information
• Switching between steps

The goal was to ensure a predictable and smooth experience, even when the AI took unexpected paths.

🔹 2. Real-Time WebSocket Testing

This was one of the most technically challenging and educational parts of the project. Unlike normal HTTP requests, the agent communicated through WebSockets — enabling two-way, real-time communication. I tested:

• Connection reliability
• Message timing and ordering
• Handling of partial messages
• Behavior under slow or unstable network conditions
• Reconnect and retry logic
• Real-time backend updates

verified WebSocket behavior manually using browser DevTools (Network → WS). This allowed me to inspect real-time incoming and outgoing messages, check message order, monitor Base64 audio chunks, and ensure connection stability under both normal and slow network conditions.

This approach helped uncover issues like delayed messages, dropped connections, and occasional UI freezes during long conversations.

🔹 3. Backend API Integration Testing

Behind every AI message, the agent interacted with backend microservices to fetch:

• Product details
• Warranty status
• Customer information
• Available appointment slots
• Confirmation numbers

My API testing covered:

✔ Response accuracy ✔ Edge cases and error codes ✔ Latency handling ✔ Data consistency between UI & backend ✔ Behavior when an API failed

This ensured data reliability throughout the scheduling journey.

🔹 4. Appointment Scheduling Flow Testing

The agent’s primary function was scheduling repairs, so I tested all variations:

• New appointment creation
• Selecting product and issue category
• Availability and time-slot logic
• Rescheduling and cancellations
• Confirmation message validation
• Notification triggers (email/SMS)

This ensured the entire customer journey worked correctly, even during complex scenarios.

🔹 5. Performance Testing with JMeter + InfluxDB + Grafana

To validate system performance, I created load scenarios using:

• JMeter — simulate concurrent users
• InfluxDB — store performance metrics
• Grafana — visualize performance dashboards

I tested:

• Peak traffic impact
• WebSocket message throughput
• API response time patterns
• Behavior under stress
• Long-duration conversation stability

This exposed bottlenecks in message handling and backend performance, enabling the team to optimize system stability.

🔹 Testing Audio-to-Base64 Messages Under Load

The system also supported audio messages, where the browser converted voice input into Base64-encoded chunks streamed over WebSockets.

For this, I:

• Used prerecorded audio samples
• Converted them to Base64
• Sent them in WebSocket messages via JMeter
• Simulated multiple users sending audio simultaneously
• Monitored delays and chunk-processing bottlenecks

I also verified real-time delivery manually using browser DevTools (Network → WS) to ensure audio chunks were received in correct order and without loss.

This approach ensured stability not only for text chat but also for real-time audio interactions.

🔹 What Was Not Part of My Role — But I’m Curious About

Although I worked extensively on UI, functional, API, and performance testing, I wasn’t directly involved in:

• NLP model training
• Intent creation or tuning
• AI model development

However, this experience sparked my interest in AI testing. I’m now exploring:

• How NLP intents are designed
• How AI responses are evaluated
• How AI accuracy and safety are tested

AI testing is a fascinating area I’m eager to grow into.

🔹 Key Learnings from Testing an AI Agent

Some key insights from this experience:

⭐ Real-time systems require deep testing WebSockets behave differently from normal APIs, making synchronization and timing critical.

⭐ AI responses add unpredictability Even if the AI logic changes, the UI must remain stable and reliable.

⭐ Backend performance affects conversation flow Slow APIs or backend errors directly impact user experience.

⭐ AI testing combines multiple testing types Functional + API + UI + performance + real-time behavior — all must be tested together.

🔹 Final Thoughts

Testing an AI-driven scheduling agent gave me a deeper understanding of:

• Real-time communication patterns
• Modern conversational systems
• Complex backend integrations
• Performance engineering
• End-to-end customer experience design

It was one of the most valuable testing experiences of my career, and I hope this article helps other QA professionals understand what AI-agent testing looks like in real projects.