​WORKFLOWS: DIGITAL METHODS FOR REPRODUCIBLE RESEARCH PRACTICES IN THE ARTS AND HUMANITIES - DARIAH ANNUAL EVENT 2024, LISBOA, PORTUGAL

ABSTRACT: 

Uncovering Inconsistencies in AI Text Analysis: A Review of ChatGPT 4.0’s Stop Word Frequency Counting​

​

This study explores structural irregularities in textual data analysis using ChatGPT 4.0, one of several Large Language Models available today. The research focuses on applying ChatGPT's functions to a Portuguese newspaper, specifically analysing the frequency of stop words across ten iterations, with a consistent prompt used throughout. The main aim is to use textual data analysis methods to identify the key themes and ideas presented in the newspaper. Findings reveal significant asymmetries and inconsistencies in ChatGPT's performance, particularly pronounced in later iterations. Notably, even stop words that initially appeared to have relatively balanced frequencies displayed significant inconsistencies upon closer examination. Furthermore, a correlation was observed between instances of null frequency and increased irregularities in word frequency analysis. These results underscore the challenges in maintaining consistency across multiple iterations and raise critical questions about the reliability and reproducibility of AI-generated textual analysis, especially when dealing with nuanced linguistic features. While ChatGPT 4.0 shows promise for textual analysis, its current limitations demand a cautious approach in academic research. This study emphasizes the necessity for rigorous validation protocols and human oversight when integrating AI tools into linguistic and social science research methodologies. Future investigations should prioritize understanding the root causes of these inconsistencies and developing robust strategies to enhance the reliability of AI-driven textual analysis in academic contexts, ultimately bridging the gap between AI capabilities and the stringent requirements of linguistic and social science research.