Did the Dog Eat AI’s Logic?

Why AI data analysis tools should show their work

Photo by Lesli Whitecotton on Unsplash.

AI tools are getting surprisingly good at helping with data analysis. You can now upload a dataset into a chatbot and ask a question — e.g. “Which products have the highest ratings?” — and the LLM will generate code and results for you. When I first saw tools like this, it felt almost like magic. But it also introduces a new challenge: how do we know the AI actually did the analysis correctly?

A recent research paper by Kazemitabaar et al. (2024) explores this problem and comes up with a new way to design AI-assisted data analysis tools that gives users more control and transparency. It takes its starting point in most current tools that are based on a conversational interface. Using these tools, a user submits a prompt describing a data analysis task, and the AI generates the code and results for the entire task in one step.

Conversational workflow.

While this workflow looks simple, it creates two challenges.

First of all users often struggle with verification. The AI generates a large block of code and output, and the user has to carefully read through everything to verify whether the results are correct. If the AI made incorrect assumptions about the dataset, those mistakes can be difficult to detect.

Secondly, users struggle with steering the AI. So if something is wrong with the output the AI produces, the only options is to write another prompt and hope the AI produces a better result the next time, or to tediously manually fix the issue yourself. This process can quickly become frustrating.

To better understand these issues, the researchers conducted a study with 15 participants performing common data analysis tasks using a conversational AI tool. They found that a large portion of user interactions actually involved trying to just steer the AI’s process or verify its results. However, the conversational interface offered very little support for doing either effectively.

To address these challenges, the researchers proposed a design concept called *interactive task decomposition* where, instead of generating the entire analysis in one step, the system breaks the task into smaller parts and makes the AI’s reasoning visible to the user. This approach allows users to intervene at any time during the analysis process rather than only after the AI has already generated the final result. The researchers implemented this idea in two prototype systems.

The Stepwise system breaks a task into a series of smaller subgoals. For each step, the AI generates assumptions about the data and the actions required to proceed.

Stepwise workflow.

Users can edit these assumptions before the system generates code. The code is then executed and the results are shown, including intermediate variables that users can inspect. Once a step is completed, the system generates the next subgoal. By focusing on smaller steps, users can verify the analysis gradually and correct problems early in the process.

The Phasewise system takes a different approach. Instead of many small steps, it organizes the analysis into three main phases:

1. Input and output assumptions;
2. An execution plan describing the analysis process; and
3. Generated code and results.

Phasewise workflow.

Each phase is editable, allowing users to modify the AI’s assumptions, adjust the analysis plan, and inspect the generated code. This structure gives users a clearer overview of the entire analysis.

Both system also adopted a sidebar for exploring the assumptions and AI produced code, and for verifying the correctness of the output. Here the user is able to ask questions about specific parts of code, assumpions or the plan and the it is also possible to generate code directly here. This was done on a different thread, so this way the main thread would not get overcluttered as users asks clarification questions and other smaller code generation requests.

Sidebar.

Sidebar for query.

To evaluate the systems, the researchers conducted a study with 18 participants who completed data analysis tasks using three tools: the traditional conversational interface, the Stepwise system, and the Phasewise system.

Participants reported that the Stepwise and Phasewise systems made it easier to verify results and intervene when something went wrong. They also felt more in control of the AI’s analysis process.

The most interesting is that participants did not necessarily complete tasks faster. Instead, the main improvement was in how transparent and controllable the AI system felt. Many participants reported that they “felt happier using their system” compared to the baseline conversational system and “felt more in control”.

This work highlights an important lesson for human-centered AI: powerful AI systems are not enough on their own. Users also need tools that help them understand and guide the AI’s reasoning.

By exposing the AI’s assumptions and allowing users to intervene during the process, systems like Stepwise and Phasewise show how future AI-assisted data analysis tools could become more transparent, trustworthy, and collaborative by allowing users to steer the AI throughout the entire process.

Further Reading

• Kazemitabaar, M., Williams, J., Drosos, I., Grossman, T., Henley, A. Z., Negreanu, C., & Sarkar, A. (2024). Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decomposition. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST).