How well can AI handle iterative qualitative coding?

AI Advancements - Moving fast

I am currently preparing a book chapter titled "Qualitative Content Analysis in the Context of Generative Artificial Intelligence" for a handbook on qualitative content analysis, edited by Margrit Schreier and Nicole Weydmann.

As I outlined the chapter, I realized it would be worthwhile to revisit some of my experiments using a chatbot for coding—this time testing them with OpenAI’s latest reasoning models.

The results were impressive. What did I test?

Just a few months ago, it wasn’t possible to guide a large language model (LLM) through an iterative process when "coding" data. But what does that mean?

When a human researcher codes qualitative data, the process is dynamic—we continuously compare and contrast. As we read a new data segment, we check whether an existing code fits, whether a code could work with slight refinement or abstraction, or whether we need to create a new one. When coding a second interview, we keep the first in mind, aiming to apply the same codes whenever possible to maintain consistency. This process repeats as we move through the data.

I started a chat with o3-mini as my AI assistant and asked how it would approach coding four interview transcripts. It responded:

"When coding multiple interviews, I typically use an iterative and flexible process that blends both deductive and inductive approaches. Here’s how I would approach coding four interviews:..."

The description that followed was spot on. Just a few months ago, similar responses tended to mix elements of thematic analysis and grounded theory. Curious to see it in action, I uploaded my four test interviews and asked the AI to actually perform the coding. Below is a condensed version of o3-mini describing its process:

Step 1: Read Through Each Interview I started by reading each transcript (Alexander, Deborah, Shazia, and Trevor) to grasp the overall narratives around friendship. This helped me understand the context in which each interviewee discussed topics like how friendships were formed, maintained, and changed over time.

Step 2: Develop Initial Codes (Using Interview 1 as a Starting Point)

Step 3: Apply and Compare Codes Across Interviews

Step 4: Developing a Codebook After coding each interview individually, I consolidated the codes into a draft codebook. While some codes (like Friendship Formation and Shared Activities) are consistent across interviews, others are unique to the content of a specific interview (e.g., Transition Experiences in Alexander’s case or Conflict and Dissolution in Trevor’s narrative).

Step 5: Consistent Application with Room for New Insights • Consistent Application: I used the same codes across interviews when the content was similar. For instance, every time a respondent mentioned the initial meeting or bonding moments, I applied the Friendship Formation code. • Flexibility for New Themes: When new aspects emerged—like Deborah’s reflections on memory and loss or Trevor’s mention of friendship dissolution—I introduced additional codes to capture these nuances.

Step 6: Constant Comparison I regularly compared sections of one interview with others. This allowed me to decide when to: • Apply an Existing Code: If a similar concept was discussed (e.g., shared activities), the same code was used. • Develop a New Code: If a new theme arose that wasn’t captured by the existing codes (e.g., the impact of personal growth or illness on friendships), I added a new code. • Flexibility was preserved to account for each interview’s unique contributions, ensuring that subtle nuances and contextual details were not lost. If you want to read the entire chat, here is the link

chatgpt.com/share/67cd9a23-187c-800b-b3ba-066… .

When I asked whether this code list was complete, it responded: "The initial code list was intended as a starting point based on early readings. In qualitative coding, it's normal—and expected—that the codebook evolves as you work more deeply with the data." (Chat 1: March 9, 2025) 😜

So, I asked it to go through two more iterations. By the end, the codebook contained 12 categories or themes. My manually created coding frame, in comparison, had 62 codes organized into the 12 categories. While the wording of the categories differed, the overall topics were largely the same. I did not continue to probe for subcategories. However, I am sure it would also worked with me to explore each category or theme in more depth.

After the first iteration, I tested whether the AI assistant could retrieve all data segments associated with a specific category. In the past, it typically provided only one or two example quotes rather than a complete set of relevant text segments. To check this, I asked: "Please give me a complete list of text segments for the code: Shared Experiences."

The AI assistant returned three to five quotes per respondent. When I repeated this request after the third iteration, the quotes remained unchanged, demonstrating consistency.

However, the AI chatbot is not yet a true code-and-retrieve tool where you can click on a code to instantly view all associated segments. Instead, you have to run a query, prompting the AI assistant to analyze the data again and retrieve relevant quotes. This also means you cannot easily reassign a specific quote to a different code. Additionally, the codebook remains at the category level and lacks detailed subcodes.

When I tested the iterative coding process with 20 interviews, it still worked. The o3-mini model has a larger context window, allowing up to 200k tokens of data. However, it only returned eight categories (themes). I believe I mentioned this in an earlier newsletter: when querying a large dataset, AI tends to generate broad or generalized answers—something we see reflected in this result.

Still, I was impressed. Would I return to a traditional coding approach? No.

When you think about it, I wasn’t coding—I was engaging in a dialogue with the AI. Rather than creating subcategories, I would now take the suggested categories and explore them in more depth. I’d pick one and ask further questions—not to develop subcategories but to gain a deeper understanding: uncovering nuances, exploring variations in how different respondent groups experience the topic, and exploring relationships to other topics in the data.

Speaking of the rapid pace of AI development—if you have a subscription to The New York Times, I highly recommend listening to this podcast by Ezra Klein: The Government Knows AGI is Coming

podcasts.apple.com/us/podcast/the-government-knows-a…

At the end of the episode, Ben Buchanan ( former White House Special Advisor on AI under the Biden Administration), suggests a few books to read, including The Structure of Scientific Revolutions by Thomas Kuhn.

Some of you may recall my webinar, "Redefining Qualitative Research: Is AI Leading Us to a New Paradigm?

youtube.com/watch In that session, I provided a brief walkthrough of the different phases of a paradigm shift as described by Kuhn. If you don't have time to read the book, click the link and watch the webinar.

Ben Buchanan recommends this book because he sees it as a fitting summary of their discussion—highlighting how shifts in technology and scientific understanding bring profound societal implications. Based on what he knows as former advisor on AI, his prediction is that transformative AI (another word for AGI - Artificial General Intelligence) will be here in two to three years.