How reliable are AI-generated analysis results?
Let’s Talk about the Validity of AI Results
Today, I want to share a small experiment I conducted to compare the results of analyses conducted on different days using QInsights.
For this experiment, I analyzed interviews on the topic "The Meaning of Friendship", featuring four respondents: Deborah, Shazia, Alexander, and Trevor. I conducted the analysis on three separate occasions—January 17, February 7, and February 8, 2025. The specific dates aren’t important; what matters is that each analysis was performed independently at different times.
On each of these days, I asked the following questions: 1. How did the respondents describe the way they first met their friend? 2. What were their first impressions of the person who later became their friend? 3. What shared interests or experiences initially helped to build their friendship? 4. What activities did they participate in that helped strengthen their bond? 5. Was there a moment or event early in the relationship that deepened their emotional connection? 6. Were there any barriers that initially hindered the development of your friendship? Conversely, what factors seemed to facilitate the friendship? 7. Based on this initial analysis, write a detailed and cohesive essay about Deborah’s experiences with her friend. The essay should flow as a narrative and be suitable for inclusion in a formal report. Begin with how Deborah initially met her friend, discuss her first impressions, their shared interests, the activities that bonded them, and the emotional moments that deepened their friendship. Also, address the barriers and facilitators in their friendship. Use direct quotes from Deborah to illustrate key points. Do not use bullet points or lists—write this as a seamless text. The latter question was asked for each respondent.
For this experiment, I focused solely on the responses to Question 7, which essentially serves as a summary of Questions 1 to 6, to identify similarities in the answers. I began by reading through them myself. My initial impression was that I was encountering the same overarching narratives, though certain summaries included details that were absent in others.
Next, I used an AI assistant to compare the three versions from each respondent at the content level. This analysis yielded the following results (abreviated version, for full results and data set, click here: Results and data set)
drive.google.com/drive/folders/1ceyQ_B6mDcuXE2RTFn…
Alexander • All three documents communicate the same core story. • The documents tell the same main story with slight variations in emphasis.
Are they saying the same thing? Yes and no.
Yes: The fundamental story remains identical in all three. No: Each document shapes the narrative differently: 1. Feb 7 version = Full, detailed, and reflective. 2. Feb 8 version = Condensed, casual, humorous. 3. Jan 17 version = Personal, emotional, deep. Deborah • All three documents convey the same essential themes. • The core story remains identical across all three documents.
Are they saying the same thing? Yes and no.
Yes: The events and main themes are identical in all three documents. No: Each version shapes the story differently: • Feb 7 version = Most complete, emotional, and hopeful. • Feb 8 version = Concise, practical, and direct. • Jan 17 version = Personal, reflective, and analytical. Shazia • All three documents communicate the same key themes. • The documents share the same narrative structure, with variations in emphasis.
Are they saying the same thing? Yes and no.
Yes: The events and themes are consistent in all three versions. No: Each document approaches the friendship differently: • Feb 7 version = Comprehensive, reflective, and balanced. • Feb 8 version = Concise, practical, humorous. • Jan 17 version = Personal, deep, and philosophical.
Trevor • All three documents consistently highlight the following key themes. • The core story remains the same across all three versions.
Are they saying the same thing? Yes and no.
Yes: The core events and themes remain identical. No: Each document presents a different tone and focus: • Feb 7 version = Well-structured, complete, and balanced. • Feb 8 version = Casual, slightly humorous, and condensed. • Jan 17 version = Deeply personal, emotional, and introspective.
Looking at the ratings across all four cases, it becomes clear that the AI assistant consistently maintained a similar tone of voice within each respective analysis. On February 7, the responses were longer and more detailed; on February 8, they were shorter and more pragmatic; and on January 17, they had a more profound depth.
However, the core themes and overarching narratives remained consistent across all three versions. In addition we know that the results are derived directly from the data rather than being AI-generated fabrications, since QInsights employs a Retrieval-Augmented Generation (RAG) system. Plus, the results can be verified.
Verifying AI answers in the original source data in QInsights
How Can the Results Be Interpreted? One could argue that the results are unreliable because each analysis includes different details. However, this argument only holds if the analysis is conducted in a passive, one-directional manner—simply asking a fixed set of questions without engaging with the responses, verifying the sources, or asking follow-up questions.
If analysis is approached this way, the responsibility shifts entirely to AI, turning it into an automated reporting tool rather than an analytical partner. But qualitative analysis, especially when using AI, should be an interactive, iterative process where the researcher critically engages with the insights, refines inquiries, and ensures alignment with the underlying data.
If you use the tool as an assistant rather than a passive generator, the workflow shifts: 1. Read through each response carefully – rather than accepting it at face value, you engage with it critically. 2. Verify the references – if confirmation is needed, you check the supporting sources and review the original files. 3. Ask follow-up questions – if more detail is required, you refine your queries to explore nuances that may not have surfaced initially. 4. Continue iteratively – moving to the next question, informed by the insights gathered so far. In this interactive process, the final result isn’t just a static summary of the initial questions, but a richer synthesis that includes everything explored in between. Any missing nuances or details from the first round of summaries would naturally emerge through this dynamic back-and-forth engagement with the data.
Recommendation: Write the Summaries Yourself While the AI assistant provides structured summaries, its writing style remains distinctly AI-generated. To ensure clarity, coherence, and a personal touch, it’s recommended that you write the summaries yourself.
The starting point for your summary should be: 1. The AI’s responses to each individual question. 2. The follow-up questions and their answers. This initial summary is not yet the full analysis, but rather the first step. Since interviews cover a broad range of topics, the next stage involves moving to a different set of questions in a new chat and writing another summary.
At this stage, you may start noticing patterns and connections within the data. Instead of simply progressing to the next topic, you might decide to shift your focus toward these emerging relationships—deepening your analysis. With each round of exploration, the process gradually moves beyond simple summaries to a more in-depth analysis and interpretation.
Is This Process Time-Consuming? Surprisingly, no—especially when compared to traditional coding-based qualitative analysis. • A focused conversation on one topic might take around 30 minutes, plus additional time for writing. • In a single day, depending on the volume of data, you can easily explore two to four research questions. This workflow streamlines qualitative analysis while maintaining rigor, flexibility, and depth, making AI-powered research both efficient and insightful.
I call this process "Conversational Analysis with AI." And yes, I know—I need to write it up as a paper you can cite. Stay tuned, it's coming this spring!
As always, have a great start to your week. With best regards,