Towards Effective Argumentation: Design and Implementation of a Generative AI-Based Evaluation and Feedback System

This study aimed at examining the performance of generative artificial intelligence to extract argumentation elements from text. Thus, the researchers developed a web-based framework to provide automated assessment and feedback relying on a large language model, ChatGPT. The results produced by ChatGPT were compared to human experts across scientific and non-scientific contexts. The findings revealed marked discrepancies in the performance of AI for extracting argument components, with a significant variance between issues of a scientific nature and those that are not. Higher accuracy was noted in identifying claims, data, and qualifiers, as opposed to rebuttals, backing, and warrants. The study illuminated AI's promise for educational applications but also its shortcomings, such as the increased frequency of erroneous element identification when accuracy was low. This highlights the essential need for more in-depth comparative research on models and the further development of AI to enhance its role in supporting argumentation training.