Can generative AI analyse vast amounts of qualitative data?

Can we really analyse "vast" amounts of data with generative AI?

You’ve probably heard claims that generative AI can now analyze vast amounts of data effortlessly. Unfortunately, that’s not entirely true—and that’s what I want to talk about this Monday morning.

The Context Window: A Fundamental Limitation Theoretically, generative AI could analyze large datasets. However, there’s a crucial limitation that often goes unnoticed: the context window. Think of it as the AI’s short-term memory—only a fixed amount of data can be processed at any given time.

This limit exists for a reason. Every time you double the context window size, the computational cost quadruples. If we allowed unlimited data processing, the sheer computing power required would be unsustainable—even the largest data centers wouldn’t be able to keep up.

How Much Data Can AI Process at Once? The actual size of the context window varies depending on the AI model and whether you’re using a pro version: • ChatGPT-4 (Pro version): 128k tokens (~250-350 pages or 10-15 one-hour interviews) • Claude Pro (Anthropic’s model): 200k tokens (~400-500 pages) This is already a significant amount of data, but I often hear researchers hoping to analyze entire projects with 200-300 interviews in one go. While strategies exist to handle large datasets, it doesn’t mean you can simply upload everything and have an AI assistant search across all documents simultaneously.

qeludra.com

Analyzing Large-Scale Data: What Actually Works? If you need to analyze thousands of documents or large-scale datasets, generative AI is not the solution—at least, not on its own. Instead, this is where traditional AI methods shine. These methods have existed for over a decade and are designed for large-scale text analysis, including: • Natural Language Processing (NLP): Identifying named entities, parsing sentence structure, and tagging parts of speech. • TF-IDF (Term Frequency-Inverse Document Frequency): Ranking key terms and clustering documents based on keyword importance. • Topic Modeling: Identifying hidden themes in massive text collections. • Clustering (K-means, Hierarchical): Grouping similar documents for structured analysis. These techniques are built for scale and ensure consistent, systematic processing of vast amounts of text.

Generative AI for Qualitative Analysis: What’s Possible? When people want to use generative AI for qualitative analysis, they’re not referring to statistical NLP techniques. They want AI to support interpretive, thematic analysis. This is possible—but only within the constraints of the context window.

So what happens when you exceed the context window? 1. Data beyond the limit is ignored. Worst case scenario: you don’t even realize that part of your dataset wasn’t analyzed. 2. AI starts hallucinating. If it can’t access all the data, it may fabricate details rather than acknowledge the missing information. 3. The AI assistant warns you. Ideally, the system notifies you that the data exceeds its limit and asks you to refine your request.

Another option is a Retrieval-Augmented Generation (RAG) system, which for instance is used by QInsights

qinsights.ai/ . If you select a large number of documents that exceed the context window, QInsights pre-processes the data based on your question, retrieves only the relevant segments, and processes those selectively.

For example, you might select 30 interviews for analysis. But rather than sending all 30 interviews in full, QInsights first extracts relevant excerpts in order not to exceed the AI’s context window. This allows you to analyze larger projects. However, also here, there’s a trade-off you should know about: • When querying a large dataset, the AI generates high-level summaries rather than deep, granular insights • The AI prioritizes frequently mentioned themes, meaning subtle patterns or unique perspectives may be overshadowed • Responses can become overly generalized, making it harder to identify distinct patterns

If you want more depth in your analysis, a better approach is to narrow your focus:

Select a targeted subset of interviews based on criteria like demographic group or thematic relevance. • The AI can then explore nuances and provide richer insights instead of just summarizing. • You’ll see patterns more clearly as the AI isn’t overwhelmed by too much data at once.

The Bottom Line: Right Tool, Right Job Generative AI excels at qualitative, nuanced analysis of smaller datasets—a few hundred pages, perfect for interview studies or focus groups. But if you need systematic analysis at scale, you’ll still need traditional NLP and machine learning techniques.

It’s all about choosing the right tool for the job. Don’t let the hype mislead you—understanding the context window is key to getting meaningful, actionable insights from AI-powered analysis.