Search

Background: Recently, ChatGPT and similar generative AI models have attracted hundreds of millions of users and become part of the public discourse. Many believe that such models will disrupt society and will result in a significant change in the education system and information generation in the future. So far, this belief is based on either colloquial evidence or benchmarks from the owners of the models -- both lack scientific rigour. Objective: Through a large-scale study comparing human-written versus ChatGPT-generated argumentative student essays, we systematically assess the quality of the AI-generated content. Methods: A large corpus of essays was rated using standard criteria by a large number of human experts (teachers). We augment the analysis with a consideration of the linguistic characteristics of the generated essays. Results: Our results demonstrate that ChatGPT generates essays that are rated higher for quality than human-written essays. The writing style of the AI models exhibits linguistic characteristics that are different from those of the human-written essays, e.g., it is characterized by fewer discourse and epistemic markers, but more nominalizations and greater lexical diversity. Conclusions: Our results clearly demonstrate that models like ChatGPT outperform humans in generating argumentative essays. Since the technology is readily available for anyone to use, educators must act immediately. We must re-invent homework and develop teaching concepts that utilize these AI models in the same way as math utilized the calculator: teach the general concepts first and then use AI tools to free up time for other learning objectives., Comment: Submitted

Introducing computational thinking in primary school curricula implies that teachers have to prepare appropriate lesson material. Typically this includes creating programming tasks, which may overwhelm primary school teachers with lacking programming subject knowledge. Inadequate resulting example code may negatively affect learning, and students might adopt bad programming habits or misconceptions. To avoid this problem, automated program analysis tools have the potential to help scaffolding task creation processes. For example, static program analysis tools can automatically detect both good and bad code patterns, and provide hints on improving the code. To explore how teachers generally proceed when creating programming tasks, whether tool support can help, and how it is perceived by teachers, we performed a pre-study with 26 and a main study with 59 teachers in training and the LitterBox static analysis tool for Scratch. We find that teachers in training (1) often start with brainstorming thematic ideas rather than setting learning objectives, (2) write code before the task text, (3) give more hints in their task texts and create fewer bugs when supported by LitterBox, and (4) mention both positive aspects of the tool and suggestions for improvement. These findings provide an improved understanding of how to inform teacher training with respect to support needed by teachers when creating programming tasks., Comment: To be published in the proceedings of the 28th Annual ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE)