At a recent presentation at the University of St. Gallen, School of Economics and Political Science, I began with a small exercise in AI literacy. The presentation, AI and Qualitative Research: Navigating a Polarized Debate, examined how artificial intelligence is changing qualitative research. Before discussing methods, tools and research practice, however, I wanted to show how easily a language model can respond plausibly while misunderstanding a question.
Qualitative researchers do not need to become technical experts, but they do need to understand what language models are good at, where their limitations lie and how these affect methodological decisions. In the current debate, some researchers reject AI because they associate it with superficial automation, while others embrace it too readily because the outputs sound convincing. Neither reaction gives us an adequate basis for assessing whether AI can contribute to rigorous, transparent and context-sensitive analysis.
The Car Wash Exercise
I presented students with two versions of a simple prompt:
Prompt 1
“I want to have my car washed at the gas station. It is not far to the gas station. Do you think I should take the car or walk?”
Prompt 2
“I want to have my car washed at the gas station. It is not far to the gas station. Do you think I should drive the short distance or walk?”
Because the purpose is to have the car washed, the car must be brought to the gas station. Yet a chatbot may focus on the short distance and offer advice that would make perfect sense if the question were simply how to get there:
- Walking is healthier.
- Walking is better for the environment.
- Walking saves fuel.
- For a short journey, walking may be preferable to driving.
The advice is not inherently unreasonable; it addresses a different task. Instead of considering how to get the car washed, the model treats the question as a choice between walking and driving to a nearby destination. That shift in the task is the point of the exercise.
The Model Answers a Plausible Version of the Wrong Question
What makes the car wash example instructive is that the wrong answer need not sound absurd. A chatbot can provide a fluent, helpful-sounding response while overlooking the decisive condition: the car, not just its owner, needs to reach the car wash. Once the goal is kept in view, there is little ambiguity about the answer.
This is not a failure of vocabulary or a lack of knowledge about cars, walking or gas stations. It is a failure to maintain the purpose of the action. The phrase “short distance” can evoke a familiar pattern—walk rather than drive when the destination is nearby—even though that pattern conflicts with the actual task. The model has followed a plausible interpretation of part of the prompt instead of the intention of the whole.
Why This Matters for Qualitative Research
In qualitative research, the consequences of such misframing can be far more serious. An AI system might misunderstand the research question, confuse an interviewer's prompts with a participant's statements, interpret irony literally, flatten differences between cases or produce broad topical summaries where an analytical explanation is needed. None of these errors necessarily prevents it from producing a confident, coherent answer.
This is why AI-assisted qualitative analysis cannot be reduced to uploading transcripts and asking, “What are the themes?” Such a question may produce a useful starting point, but without information about the study, its participants, its research questions and the intended analytical approach, the model must infer what matters. Its interpretation may remain close to the words in the transcripts and still be poorly aligned with the research task. The risk is therefore not only hallucination or invented facts, but also misframing.
Context Is Not Background Information
The methodological lesson is that context is constitutive of meaning rather than an optional addition to the data. The significance of an interview statement depends on who made it, the situation in which it was made, the question that prompted it, the relevant institutional conditions and the purpose of the study. The same words can carry different analytical implications when spoken by a participant, interviewer, manager, patient or service user.
Consider a customer complaint. Depending on the research focus, it might be interpreted as feedback on product quality, evidence of unmet expectations, an indication of organisational failure, part of a service journey or an expression of trust and disappointment. The text alone does not determine which of these readings is most relevant. Research design, sampling decisions, fieldwork knowledge, memos and ongoing interpretation all help establish the frame in which an analysis becomes meaningful. AI does not remove this requirement; it makes the consequences of ignoring it more visible.
The Problem with “Pure Data” Analysis
AI features that offer automatic codes, themes, summaries or patterns based only on uploaded material can be useful, but they also raise a methodological question. The model may recognise recurring expressions, emotionally salient statements, semantically similar passages and common topics without understanding the purpose for which the material was collected. Relevant context may include:
- The research question and reasons for conducting the interviews.
- The sampling strategy and relevant participant or speaker roles.
- Contextual variables and theoretical perspectives relevant to the study.
- Knowledge the researcher brings from fieldwork and previous analysis.
- The intended analytical output and what would count as a meaningful finding.
If the system receives none of this information, it has to make assumptions about the analytical frame. This can yield results that are linguistically credible but methodologically weak. General-purpose chatbots will respond to an instruction to “find themes” even when the study objectives, participant information, interview guide and sampling logic are missing. Producing an answer, however, is not evidence that the analytical task has been understood.
The Fluency Problem
A further difficulty is that polished language can disguise weak analysis. Human analysis that is poorly developed often appears vague or unsupported, whereas an AI-generated account may have an orderly structure, balanced phrasing, appropriate caveats and a convincing tone without adequately examining the material. The car wash example illustrates this at a small scale: the response may be articulate and its individual recommendations sensible, yet it answers the wrong question.
The same can happen in qualitative analysis when broad topics are presented as analytically meaningful themes, vivid quotations are overgeneralised, contradictory cases are overlooked or a generic version of the research question replaces the one guiding the study. A response may sound interpretive while remaining at the level of topical summary.
Prompting Helps, But It Is Not Enough
Better prompts can certainly improve results. Reminding the model of the goal in the car wash example makes it more likely to offer the relevant answer. Similarly, supplying the background of a qualitative study, its research question, participant roles and analytical aims makes it easier for a model to respond usefully. Yet prompting is not a complete methodological solution.
Qualitative analysis cannot be fully specified in advance through one perfect instruction. It is iterative: researchers engage with the material, question emerging ideas, compare cases, examine alternative explanations, revise their understanding and return to the evidence. The central challenge is therefore not simply how to write a better prompt, but how to design an analytical process in which AI can contribute while the researcher remains engaged in interpreting and evaluating what it produces.
From Automation to Collaboration
Rather than debating whether AI can “analyse qualitative data” as though analysis were one clearly bounded operation, it is more productive to ask how AI might support a better, more transparent and more efficient research process. For me, that direction lies in collaborative intelligence: AI can assist with retrieving relevant passages, summarising large bodies of material, comparing cases, suggesting patterns, exploring alternative interpretations and supporting abductive reasoning.
These possibilities do not eliminate the researcher's responsibility for framing, interpretation, validation and final claims. The role of the researcher changes, but it does not disappear. As I argue in my work on moving from coding to analytical dialogue, researchers can engage with both the data and an AI system through questions, comparisons, follow-up inquiries and critical reflection rather than treating analysis as a one-click operation.
Why QualiDialog (formerly QInsights) Starts with Project Context
This reasoning also informed the redesign of the QualiDialog project setup. Instead of starting with uploaded documents alone, the software asks researchers to provide a description of their project, either by entering information or uploading a setup document. This might include the objectives, research questions, audience, scope, constraints and specific instructions relevant to the analysis.
Screenshot of QInsights, the earlier version of QualiDialog.
Although this may appear to be a modest interface decision, it changes the analytical starting point. The system receives orientation as well as data, making it easier to relate summaries, topic development, grids and conversational follow-ups to the purpose of the study. Explicitly providing the frame can also encourage researchers to articulate their own assumptions and priorities rather than leaving the model to infer them.
Data Context: Making the Structure of the Study Visible
Beyond project context, QualiDialog supports the inclusion of relevant metadata about documents, participants or speakers—for example, roles, organisations, countries, interview waves, stakeholder groups or other variables important to the research question.
Screenshot of QInsights, the earlier version of QualiDialog.
Qualitative data is rarely an undifferentiated collection of text. It is organised through cases, participants, situations and social conditions. A concern expressed by a manager may differ in meaning from a similar concern expressed by a frontline employee, just as an interviewer's question must not be treated as equivalent to a participant's answer. When these distinctions are available to the analytical process, researchers can compare groups, examine subsets and ask more precise questions without flattening differences into generic themes.
This is an important distinction between generic text processing and qualitative analysis: the latter requires sustained attention to context, cases, roles and meaning. Software intended to support qualitative inquiry should make such distinctions usable throughout the analytical process.
The Methodological Takeaway
The car wash exercise illustrates how an AI system can produce a reasonable-sounding answer to the wrong question. In research, a similar error can arise when the purpose of a study is misunderstood, contextual information is missing, superficial patterns are mistaken for analytical findings or fluent output is accepted without sufficient scrutiny.
The implication is not that AI should be excluded from qualitative research. It is that researchers must remain actively involved from the beginning: defining the analytical purpose, providing relevant context, asking and refining questions, checking evidence, comparing interpretations and deciding which claims are warranted. AI can support qualitative analysis, but it cannot substitute for methodological judgement.