When Automation Meets Unpredictability: My Experience Building a Human-in-the-Loop Question-Answer System

Introduction

In one of my recent automation projects, I had to work with a website that asked different questions based on the scenario. At first, the automation script worked fine, but soon we realized a major challenge: the questions were not always the same. Sometimes new questions appeared, and sometimes the same question was shown in a completely different format. Because of this, our automation kept breaking.

Since I had only seven months of experience in automation, I wasn’t sure how to handle this problem. After discussing with my senior, we decided to design a system where automation and human input could work together. This is where the concept of Human-in-the-Loop (HIL) became important. The idea was simple: whenever a new question appears, a human gives the answer once, and the system learns it for the future.

This blog explains the exact problem we faced, the issues involved, and how we built a solution using embeddings, semantic search, and PostgreSQL to make the automation smarter over time.

Problem

The main challenge in our project was that the website displayed dynamic questions. These questions changed frequently based on different scenarios. Sometimes completely new questions appeared, and other times the same question was shown in a different wording. Because of this, our automation script couldn’t recognize the question correctly.

Even the client couldn’t remember all the questions they had answered earlier, so they couldn’t always guide us on what the correct response should be. As a result:

• The automation kept failing whenever a new or differently phrased question appeared.
• Our script didn’t know how to decide the right answer without human help.
• We had no simple way to store and reuse answers for future runs.

This made the overall automation unstable, manual, and time-consuming.

Key Issues

While working on this problem, we faced several challenges:

1. Same question in different formats The website often asked the same question but with different wording. Our automation treated each version as a completely new question.
2. Unable to identify new vs. old questions Without understanding the meaning of the text, automation couldn’t tell whether a question was truly new or just a slight variation.
3. Capturing the correct answer was difficult When a human selected the answer manually, we needed a reliable way to detect which option they clicked.
4. HTML element behavior was inconsistent The selected option showed a “checked=active” state, but it appeared only for a short moment. We had to check the HTML element every 2 seconds to detect this change.
5. No existing storage for learned answers Before implementing our solution, there was no database system to store questions, answers, and their meanings for future automation.

These issues made it clear that a simple rule-based automation was not enough. We needed a smarter, learning-based approach.

Solution

To make the automation smarter and more reliable, we decided to combine semantic search, embeddings, human feedback, and a PostgreSQL database. The goal was to let the system learn from every new question a human answers.

This approach helped us transform a failing rule-based automation into a self-learning system that gets smarter with every human intervention.

Flow of the System

1. Question appears on the website The automation script reads the question shown on the page.
2. Embed the question using bge-large-en-v1.5 Instead of comparing raw text, we convert the question into an embedding vector using the bge-large-en-v1.5 model. This helps the system understand the meaning of the question.
3. Semantic search in PostgreSQL The embedding is used to search for similar questions stored in the database. If a semantically similar question already exists, its answer is immediately retrieved.
4. If match found → Answer automatically If the similarity score is high, the automation uses the stored answer and selects the corresponding option on the website.
5. If no match found → Human answers When the system cannot find a similar question, it waits for a human to manually click the correct option on the website.
6. Detect the human-selected answer The automation checks the HTML element every 2 seconds. When an option is selected, the page shows a state like checked=active. This is used to capture the correct answer.
7. Store in PostgreSQL Once the answer is captured:

• The question is embedded again using bge-large-en-v1.5
• The embedding, question text, and answer are stored in PostgreSQL

System becomes smarter

The next time a similar question appears, the automation can answer it without any human help, thanks to semantic similarity.

Conclusion

This project showed that traditional rule-based automation is not enough when website content changes frequently. Dynamic questions, different word formats, and unpredictable user flows can easily break an automation script. By adding Human-in-the-Loop, we allowed the system to learn from human input whenever a new question appeared. Using bge-large-en-v1.5 embeddings, semantic search, and PostgreSQL, we converted every new interaction into reusable knowledge.

Over time, the automation became more stable and accurate because fewer human interventions were needed. Each answered question made the database smarter, and similar questions could be handled automatically in the future. This experience helped me understand that combining AI, automation, and human feedback can solve real-world problems more effectively than automation alone.

Thank you for reading this blog! 🙏 If you found this topic helpful, feel free to explore my other articles and follow me on Medium for more content. 📚✨ I am still learning and improving in the automation and AI space, and I would love to share that journey with you. 🤝🤖 If you have any suggestions or new topics you want me to write about, let’s learn and grow together. 🌱🚀