Search

Block-based programming environments like Scratch are widely used in introductory programming courses. They facilitate learning pivotal programming concepts by eliminating syntactical errors, but logical errors that break the desired program behaviour are nevertheless possible. Finding such errors requires testing, i.e., running the program and checking its behaviour. In many programming environments, this step can be automated by providing executable tests as code; in Scratch, testing can only be done manually by invoking events through user input and observing the rendered stage. While this is arguably sufficient for learners, the lack of automated testing may be inhibitive for teachers wishing to provide feedback on their students' solutions. In order to address this issue, we introduce a new category of blocks in Scratch that enables the creation of automated tests. With these blocks, students and teachers alike can create tests and receive feedback directly within the Scratch environment using familiar block-based programming logic. To facilitate the creation and to enable batch processing sets of student solutions, we extend the Scratch user interface with an accompanying test interface. We evaluated this testing framework with 28 teachers who created tests for a popular Scratch game and subsequently used these tests to assess and provide feedback on student implementations. An overall accuracy of 0.93 of teachers' tests compared to manually evaluating the functionality of 21 student solutions demonstrates that teachers are able to create and effectively use tests. A subsequent survey confirms that teachers consider the block-based test approach useful., Comment: To appear at the 24th Koli Calling International Conference on Computing Education Research

Introducing computational thinking in primary school curricula implies that teachers have to prepare appropriate lesson material. Typically this includes creating programming tasks, which may overwhelm primary school teachers with lacking programming subject knowledge. Inadequate resulting example code may negatively affect learning, and students might adopt bad programming habits or misconceptions. To avoid this problem, automated program analysis tools have the potential to help scaffolding task creation processes. For example, static program analysis tools can automatically detect both good and bad code patterns, and provide hints on improving the code. To explore how teachers generally proceed when creating programming tasks, whether tool support can help, and how it is perceived by teachers, we performed a pre-study with 26 and a main study with 59 teachers in training and the LitterBox static analysis tool for Scratch. We find that teachers in training (1) often start with brainstorming thematic ideas rather than setting learning objectives, (2) write code before the task text, (3) give more hints in their task texts and create fewer bugs when supported by LitterBox, and (4) mention both positive aspects of the tool and suggestions for improvement. These findings provide an improved understanding of how to inform teacher training with respect to support needed by teachers when creating programming tasks., Comment: To be published in the proceedings of the 28th Annual ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE)

Background: Recently, ChatGPT and similar generative AI models have attracted hundreds of millions of users and become part of the public discourse. Many believe that such models will disrupt society and will result in a significant change in the education system and information generation in the future. So far, this belief is based on either colloquial evidence or benchmarks from the owners of the models -- both lack scientific rigour. Objective: Through a large-scale study comparing human-written versus ChatGPT-generated argumentative student essays, we systematically assess the quality of the AI-generated content. Methods: A large corpus of essays was rated using standard criteria by a large number of human experts (teachers). We augment the analysis with a consideration of the linguistic characteristics of the generated essays. Results: Our results demonstrate that ChatGPT generates essays that are rated higher for quality than human-written essays. The writing style of the AI models exhibits linguistic characteristics that are different from those of the human-written essays, e.g., it is characterized by fewer discourse and epistemic markers, but more nominalizations and greater lexical diversity. Conclusions: Our results clearly demonstrate that models like ChatGPT outperform humans in generating argumentative essays. Since the technology is readily available for anyone to use, educators must act immediately. We must re-invent homework and develop teaching concepts that utilize these AI models in the same way as math utilized the calculator: teach the general concepts first and then use AI tools to free up time for other learning objectives., Comment: Submitted

Computational thinking is increasingly introduced at primary school level, usually with some form of programming activity. In particular, educational robots provide an opportunity for engaging students with programming through hands-on experiences. However, primary school teachers might not be adequately prepared for teaching computer science related topics, and giving feedback to students can often be challenging: Besides the content of the feedback (e.g., what problems have to be handled), the way the feedback is given is also important, as it can lead to negative emotional effects. To support teachers with the way of giving feedback on common problems when teaching programming with robotics, we conducted a study consisting of seven workshops with three third and four fourth grade primary school classes. Within seven different activities, the 116 primary school children first programmed the Ozobot Evo robot in the pen-and-paper mode and then on a digital device. Throughout these activities we collected data on the problems the students encountered, the feedback given, and the fun they experienced. Our analysis reveals eight categories of problems, which we summarise in this paper together with corresponding possible feedback. We observed that problems that are urgent or can harm the students' self-efficacy have a negative impact on how enjoyable an activity is perceived. While direct instruction significantly decreased the experienced fun, hints had a positive effect. Generally, we found programming the Ozobot Evo to be encouraging for both girls and boys. To support teachers, we discuss ideas for giving encouraging feedback on common problems of Ozobot Evo programming activities and how our findings transfer to other robots., Comment: To be published in the Proceedings of the 17th Workshop in Primary and Secondary Computing Education (WIPSCE'22)

Programmable robots are engaging and fun to play with, interact with the real world, and are therefore well suited to introduce young learners to programming. Introductory robot programming languages often extend existing block-based languages such as Scratch. While teaching programming with such languages is well established, the interaction with the real world in robot programs leads to specific challenges, for which learners and educators may require assistance and feedback. A practical approach to provide this feedback is by identifying and pointing out patterns in the code that are indicative of good or bad solutions. While such patterns have been defined for regular block-based programs, robot-specific programming aspects have not been considered so far. The aim of this paper is therefore to identify patterns specific to robot programming for the Scratch-based mBlock programming language, which is used for the popular mBot and Codey Rocky robots. We identify: (1) 26 bug patterns, which indicate erroneous code; (2) three code smells, which indicate code that may work but is written in a confusing or difficult to understand way; and (3) 18 code perfumes, which indicate aspects of code that are likely good. We extend the LitterBox analysis framework to automatically identify these patterns in mBlock programs. Evaluated on a dataset of 3,540 mBlock programs, we find a total of 6,129 instances of bug patterns, 592 code smells and 14,495 code perfumes. This demonstrates the potential of our approach to provide feedback and assistance to learners and educators alike for their mBlock robot programs., Comment: To be published in the Proceedings of the 17th Workshop in Primary and Secondary Computing Education (WIPSCE'22)

Bugs in Scratch programs can spoil the fun and inhibit learning success. Many common bugs are the result of recurring patterns of bad code. In this paper we present a collection of common code patterns that typically hint at bugs in Scratch programs, and the LitterBox tool which can automatically detect them. We empirically evaluate how frequently these patterns occur, and how severe their consequences usually are. While fixing bugs inevitably is part of learning, the possibility to identify the bugs automatically provides the potential to support learners, Comment: Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education (pp. 89-95)

Bugs in learners' programs are often the result of fundamental misconceptions. Teachers frequently face the challenge of first having to understand such bugs, and then suggest ways to fix them. In order to enable teachers to do so effectively and efficiently, it is desirable to support them in recognising and fixing bugs. Misconceptions often lead to recurring patterns of similar bugs, enabling automated tools to provide this support in terms of hints on occurrences of common bug patterns. In this paper, we investigate to what extent the hints improve the effectiveness and efficiency of teachers in debugging learners' programs using a cohort of 163 primary school teachers in training, tasked to correct buggy Scratch programs, with and without hints on bug patterns. Our experiment suggests that automatically generated hints can reduce the effort of finding and fixing bugs from 8.66 to 5.24 minutes, while increasing the effectiveness by 34% more correct solutions. While this improvement is convincing, arguably teachers in training might first need to learn debugging "the hard way" to not miss the opportunity to learn by relying on tools. We therefore investigate whether the use of hints during training affects their ability to recognise and fix bugs without hints. Our experiment provides no significant evidence that either learning to debug with hints or learning to debug "the hard way" leads to better learning effects. Overall, this suggests that bug patterns might be a useful concept to include in the curriculum for teachers in training, while tool-support to recognise these patterns is desirable for teachers in practice., Comment: To be published in the proceedings of the 16th Workshop in Primary and Secondary Computing Education (WIPSCE)

Block-based programming languages like Scratch enable children to be creative while learning to program. Even though the block-based approach simplifies the creation of programs, learning to program can nevertheless be challenging. Automated tools such as linters therefore support learners by providing feedback about potential bugs or code smells in their programs. Even when this feedback is elaborate and constructive, it still represents purely negative criticism and by construction ignores what learners have done correctly in their programs. In this paper we introduce an orthogonal approach to linting: We complement the criticism produced by a linter with positive feedback. We introduce the concept of code perfumes as the counterpart to code smells, indicating the correct application of programming practices considered to be good. By analysing not only what learners did wrong but also what they did right we hope to encourage learners, to provide teachers and students a better understanding of learners' progress, and to support the adoption of automated feedback tools. Using a catalogue of 25 code perfumes for Scratch, we empirically demonstrate that these represent frequent practices in Scratch, and we find that better programs indeed contain more code perfumes., Comment: To be published in the proceedings of the 16th Workshop in Primary and Secondary Computing Education (WIPSCE)

Learning basic programming with Scratch can be hard for novices and tutors alike: Students may not know how to advance when solving a task, teachers may face classrooms with many raised hands at a time, and the problem is exacerbated when novices are on their own in online or virtual lessons. It is therefore desirable to generate next-step hints automatically to provide individual feedback for students who are stuck, but current approaches rely on the availability of multiple hand-crafted or hand-selected sample solutions from which to draw valid hints, and have not been adapted for Scratch. Automated testing provides an opportunity to automatically select suitable candidate solutions for hint generation, even from a pool of student solutions using different solution approaches and varying in quality. In this paper we present Catnip, the first next-step hint generation approach for Scratch, which extends existing data-driven hint generation approaches with automated testing. Evaluation of Catnip on a dataset of student Scratch programs demonstrates that the generated hints point towards functional improvements, and the use of automated tests allows the hints to be better individualized for the chosen solution path., Comment: To be published in the Proceedings of the 26th annual conference on Innovation and Technology in Computer Science Education (ITiCSE)

Creating programs with block-based programming languages like Scratch is easy and fun. Block-based programs can nevertheless contain bugs, in particular when learners have misconceptions about programming. Even when they do not, Scratch code is often of low quality and contains code smells, further inhibiting understanding, reuse, and fun. To address this problem, in this paper we introduce LitterBox, a linter for Scratch programs. Given a program or its public project ID, LitterBox checks the program against patterns of known bugs and code smells. For each issue identified, LitterBox provides not only the location in the code, but also a helpful explanation of the underlying reason and possible misconceptions. Learners can access LitterBox through an easy to use web interface with visual information about the errors in the block-code, while for researchers LitterBox provides a general, open source, and extensible framework for static analysis of Scratch programs., Comment: To be published in the Proceedings of the International Conference on Software Engineering (ICSE'21), Joint Track on Software Engineering Education and Training