What changes, what stays when using AI for qualitative data analysis

Tools are changing. The underlying methodology however stays.

I'm currently writing a number of articles and book chapters and ChatGPT (as my currently preferred chatbot for this activity) has been substantially involved in this process. We basically "write" together. As I engage in dialog when analysing data, I also engage in dialog when writing. It is a very similar iterative process.

I select and read the literature. I bring my methodological knowledge to it. I tell ChatGPT what I think is important, where I disagree with authors, what I think they have misunderstood, and where I want the argument to go.

ChatGPT helps me connect those ideas, structure them and, quite often, puts them into much better academic prose than I would.

That is considerably more than grammar correction. Therefore I wanted to know how different the outcome is to what I wrote without AI. To test this, I gave it a chapter I wrote for my 2019 book: Analysing qualitative data with ATLAS.ti.

What happened next was not what I expected.

ChatGPT noticed a line of argument running from that 2019 chapter straight into what I am writing about AI today. Back then, I was writing about coding with ATLAS.ti. One sentence reads:

“The software itself does not dictate how to use a code. It only provides this entity as an item in the toolbox.”

Later in the chapter: “it is the user who must add methodological meaning to the entity called ‘code’”

Years later, I am arguing something remarkably similar about AI.

The researcher should guide the analytical process and bring GenAI into the loop for particular purposes. The technology can contribute, sometimes substantially, but it should not determine the direction of the inquiry.

There was another continuity I had not consciously noticed.

In 2019 I wrote that codes are not “hard printed in the data” and that there are no fixed rules for constructing categories. What we notice and how we conceptualise it depends on our research questions, disciplinary background, literature and experience.

Today, one of my concerns about automated AI analysis is precisely the assumption that we can ask a model to “find the themes” or “identify the categories,” as though these analytical objects were sitting in the data waiting to be extracted.

I then even went further back to the first edition of the book written in 2010. The strongest continuity is not actually about coding. It is about what I have always considered to be the locus of analysis: the researcher’s intellectual engagement with the material. I wrote: “A lot of the analysis happens as you write, not by clicking on some buttons and outputting some results.” And immediately afterwards: “You need to look at what the software retrieves, read through it and write it up in your own words in order to gain an understanding of what is happening in the data.”

This is close to my current position on AI. The technology can process and retrieve; that output is not yet the analysis. Analysis happens when the researcher engages with it, interprets it, questions it and develops an understanding.

Other continuities are:

Retrieval is a means of thinking, not an answer-producing mechanism.

I explained to my readers that the query tool finds material relevant to a question, but that the researcher then has to inspect what has been retrieved. “the really important result is the content of the quotations. The quotation frequency is merely an initial point of orientation.”

If we replace “query tool” with RAG/LLM retrieval, then this is very close to my current Conversational Analysis with AI logic. The machine helps bring relevant material into view. The researcher determines what it means.

Questions drive analysis.

I described and explained the following sequence how to proceed with analysis after coding:

Research question → formulate query → retrieve relevant material → read it → interpret it → write → formulate further questions.

I explained that researchers “develop questions and hypotheses along the way” and that questions can be “adjust[ed] and modif[ied]” as the analysis progresses.

That is close to the architecture of my present conversational approach. The technological interface has changed radically: instead of translating an analytical question into Boolean/proximity queries over coded segments, we can now formulate the analytical question in natural language and converse with the material. But the epistemic logic is surprisingly similar.

Dialogic conception of analysis. I wrote: When you “engage with your data, and enter into an internal dialogue, you advance your analytical thoughts and ways of thinking.”

That dialogue is largely internal: the researcher writes memos, examines retrievals, reflects and develops further questions. With GenAI, part of that internal dialogue can become an external dialogue with a language model.

And this is especially what I enjoy when working with an AI Assistant. In QInsights, for all chats we have started to integrate the possibility to use the microphone to literally "start talking with your data". It currently works if you use the Chrome Browser.

Thus, in my early writing, the idea of the dialogic analytical process was already there; GenAI provides a new interlocutor within it.

The researchers create meaning.

In 2012, I wrote: “To the software, a code is just an object that can be attached to various other objects and whose content can be searched and retrieved. Everything else is up to you.”

That has also not changed.

Traceability. I wrote about "research-question memos" where you record the question, the query used, the researcher's answer and interpretation, and supporting quotations. I connected this to “trustworthiness, credibility, transparency and dependability.”

This is very close to my current insistence that AI-supported analysis should preserve the path from analytical claim back to source material rather than simply generate a plausible synthesis.

However, there is one major discontinuity!

In my books about ATLAS.ti, I still treated coding as the necessary representational infrastructure for analysis: “Coding is a necessary step for describing what’s in the data.”

“the first step is always to develop some codes and to build up a coding system. Without that, you cannot query the data.”

This wasn't necessarily my methodological preference. It was partly an affordance constraint. The software literally required the researcher to convert the material into a coded representational structure before sophisticated retrieval became possible. The query tool could only query what had previously been encoded.

LLMs change that.

We can now ask questions of the source material without first transforming the entire corpus into a code system. So something that in 2012 looked methodologically necessary no longer is. If we compare 'Now' and 'Then', the analytic process enfolds as follows:

Then: Data → coding → structured retrieval → researcher reads → researcher writes/thinks → further questions → conceptual understanding.

Now: Data → researcher asks/explores → AI processes/retrieves/synthesises → researcher evaluates/interprets → further questions → recursive dialogue → conceptual understanding.

The endpoint hasn't moved very much. The middle has.

What I have consistently resisted is equating — what the software produces — with analysis. When using CAQDAS tools, clicking buttons and producing query results wasn't analysis. Today, prompting an LLM and receiving themes, summaries or interpretations isn't analysis either.

I have consistently regarded qualitative analysis as an iterative process of questioning, reading, writing, comparison and reflection carried out by the researcher. The tools available since the 1990s required coding to make that process computationally tractable. LLMs no longer do.

And I am very excited about this possibility.

Some things to ponder about this week. All the best and until next Monday,