I tried hard to get ChatGPT to code open-ended responses
Dr Susanne Friese · · Experiments & evidence
Belated Monday in Your Inbox
I really tried hard to get ChatGPT to help me code open-ended responses
You might rememember that I recently worte about a dialog
qeludra.com/blog/what-is-really-happening-whe… I had with an LLM about why it used NLP procedures to suggest codes, even though the prompt did not ask for it.
I have now found out why it happened — but also more than that.
We had a proposal for a chapter for the book Qualitative Data Analysis with Artificial Intelligence: Theory, Methods and Practice, which I am editing together with David L. Morgan. The proposal was about using ChatGPT to analyze open-ended survey responses. Even though my perspective is that we no longer need to code data, there are still plenty of researchers who cannot let go of the ways they are used to.
Because of the nature of open-ended responses — they come neatly structured in the form of an Excel table, with each answer listed in a separate cell and thus, the coding unit is defined — a chatbot might be able to assist with coding.
So I set out to develop a prompt — first to generate a coding frame and then to apply that coding frame to the data. Developing the coding frame worked after a few iterations, by guiding the chatbot to better understand what a well-developed coding frame should look like (see prompt at the end of the newsletter). I coded parts of the data myself, so I could evaluate the qualitity of the categories and subcodes that ChatGPT was suggesting.
The next task was to apply the coding frame to two questions and 150 imaginary respondents. I had created the data together with ChatGPT-4o.
The 4o model did not perform very well with coding the data. So I tried Gemini 2.5, one of the top models, and then also 3o. Since 3o is a reasoning model, I gained some insights into why it was struggling to apply the coding frame.
And this is what I want to share with you. It is actually quite funny:
o3: “The dataset is large, and it seems like managing 150 lines of data with manual coding is becoming tricky. I keep running into issues with exceeding character limits, and closing quotes are causing interruptions. An alternative idea is to simplify things by manually defining the code outside Python, or parsing data directly from the prompt text. Still, manually entering 150 rows is a lot. Maybe I should create a simplified table for now, even if the full dataset is challenging to handle all at once.
It seems we've missed lines 98–150 because the dataset tail got cut off. Restarting the compile might be too time-consuming and prone to errors. [One idea is…. ] Another option could be to generate a zip link to the full CSV, but that still requires the data to be written […]. The task feels a bit daunting but doable; basically, I need to generate rows and create the table manually. I’ll have to write out all 150 rows, which will be long, but it's part of the plan—time-consuming, though. It's a big task, but I’m ready.”
It looks like I lost the dataset_tail variable due to an earlier incomplete attempt. I'll need to recreate it by copying the lines 31-49 again. […]
It looks like we missed rows 50-97 in the dataset while putting things together earlier. Here's what we know: dataset_head covered lines 1-30, dataset_tail handled 31-49, and dataset_tail2 jumped to lines 98-150. So, we still need to grab lines 50-97. I can create a new section, dataset_tail_mid, to fill in that gap. Let’s copy those rows 50-97 and include them properly. Should have everything sorted out soon! Oh, and it looks like we never fully included lines 50-60 in the dataset_tail2…. […]
“Hmm, there’s an issue with row 3 where "Contributing to Scientific Progress" seems to be incorrectly flagged under "Not Satisfied Codes." It looks like the pattern "research" is triggering a match, but that's not expected since the row talks about "documentation" and not "research." I should refine the patterns to ensure they select the correct text. I’ll go ahead and take a closer look at row 3 to fix this. Something seems off with how patterns are being matched.
Looking at the dissatisfaction text, it mentions "pressure to publish results quickly," which includes the word "research." This matches a pattern, so Contributing to Scientific Progress was coded — but it wasn’t expected in this case as it really should be "Rushed by Deadlines." The heuristics aren't precise. The phrase "quality of the research" suggests dissatisfaction with the process, but Contributing to Science is about pride, not process.
Given the complexity, maybe manual coding is still the best approach." (end of quote - 03 reasoning).
In the end, I did get a table with about 75% of the responses coded — but not all of it was correct. So, in addition to filling in the missing responses, a researcher would also need to check the rest. This is still faster, I suppose, than coding from scratch — but putting the prompt together took so many iterations that, in that time, I could have coded all 300 responses five times over.
I did compare the results of Gemini’s coding with o3’s coding. They did not code in the same way, and I would expect yet another result with a third or fourth run. If you want good-quality results, there is no other option than for the researcher to check everything.
Coding the data is, of course, not yet the final result. Having the LLM count the frequency for each code was also not successful — for that, I probably would have needed to engage Python again.
And then, what's next? Generating themes on the basis of the coded data and making sense of it all?
If the prompt and the process would have been straight forward, the results and frequency counts more reliable, maybe they could have been some merit in it.
After 2 days, by then I knew the data quite well, I thought, let's see what QInsights comes up with. The same main topics, I had developed with ChatGPT for the coding frame?
Yes, also with QInsights, I get slightly different results if I run a theme analysis two or three times. But they complement each other, and it takes just a minute. QInsights matched all the main topics that were in the coding frame and even found two more that were actually relevant.
Since a few weeks we have an option for semi-structured data like open-ends to export all data segments that belong to a theme. This is just a click on your part, and not an LLM desperately looking for rows it had missed and finding it a daunting task.
When using QInsights, the next step is to work with those themes and ask questions. This is also where LLMs shine - and where you get the support you are looking for.
I did not start the challenge of developing a good prompt for a general-purpose chatbot to code open-ends with the aim of proving myself right. I genuinely thought it would be possible for semi-structured data. It just turned out contrary to what I had expected.
Even though I am the co-founder of QInsights, and much of my vision went into its development, I am probably also the one most critical about it — always noticing the little flaws and thinking about what could be done better.
After this experience with general-purpose chatbots that make use of the same LLMs as we do, I was really happy with what we have achieved with QInsights.
Sometimes it’s okay, I guess, to be a little bit proud of what one has accomplished 😉. There is probably also something you can be proud of?
I wish you a good rest of the week, Susanne
Prompt for developing a coding frame: You are assisting in qualitative analysis of open-ended survey responses. [Describe your project]
The dataset is structured with the following columns: • desribe the columns of your data set
Task: Read through all No. of responses [comment: this is important, as otherwise, the LLM will only take a sample of your data, and the list of suggested codes will not be exhaustive]
Suggest a comprehensive list of interpretive codes that capture the meanings expressed in the data.
(Detailed Description: For each response, generate one or more interpretive codes that reflect the underlying idea or sentiment, rather than repeating words from the text. Codes should be short (2–5 words), descriptive, and interpretive. Avoid simply extracting frequent words. Instead, focus on meaning. Where possible, suggest codes that could apply to multiple responses, rather than inventing a unique code for every single response. Organize the codes into categories and, if needed, subcategories.
Ensure that: Homogeneity within codes: Each code refers to one clearly defined idea; avoid mixing phenomena (e.g., split “Conflict” & “Poor Communication” into two separate codes). Heterogeneity between codes: Each code must have a definition that makes it conceptually unique and clearly distinguishable from all other codes. This applies both to codes within the same category and to subcodes across different categories. The key requirement is that each code label and definition refers to one specific, non-overlapping meaning.
Multiple coding is allowed: A single response segment may be assigned to more than one code if it legitimately addresses different ideas.
Avoid mixing positive and negative content in one code: • If a concept can be expressed positively (e.g., “adequate resources”) and negatively (e.g., “lack of resources”), create two separate codes with distinct labels and definitions. • Category definitions should remain neutral, but subcodes must clearly indicate whether they capture a supportive/adequate experience or a hindering/inadequate experience. • Check every proposed code label to confirm it refers to one direction of meaning only. If it covers both, split it into two codes.
Exhaustiveness: Together, the codes should capture the full range of meanings in the data, so no relevant content is left uncoded [comment: there is no guarantee for this. As descrobed above, QInsights discovered two more topics that were not covered by the suggested coding frame].
For each code, at both the category and subcode level, provide a concise definition (one to two sentences) specifying what types of responses the code covers and how it should be applied
Do not apply the codes yet — only propose the coding frame with definitions for review.
Required table format: Category + Definition Subcode Subcode Definition [Category name: Category definition] [Subcode label] [Subcode definition]
NOTE: If you want to try it out, you need to copy and paste your data also into the prompt field. Do not upload your data as separte file.