Search

AI-powered education technologies can support students and teachers in computer science education. However, with the recent developments in generative AI, and especially the increasingly emerging popularity of ChatGPT, the effectiveness of using large language models for solving programming tasks has been underexplored. The present study examines ChatGPT's ability to generate code solutions at different difficulty levels for introductory programming courses. We conducted an experiment where ChatGPT was tested on 127 randomly selected programming problems provided by Kattis, an automatic software grading tool for computer science programs, often used in higher education. The results showed that ChatGPT independently could solve 19 out of 127 programming tasks generated and assessed by Kattis. Further, ChatGPT was found to be able to generate accurate code solutions for simple problems but encountered difficulties with more complex programming tasks. The results contribute to the ongoing debate on the utility of AI-powered tools in programming education., QC 20240321 Part of ISBN 9798400716188

AI-powered education technologies can support students and teachers in computer science education. However, with the recent developments in generative AI, and especially the increasingly emerging popularity of ChatGPT, the effectiveness of using large language models for solving programming tasks has been underexplored. The present study examines ChatGPT’s ability to generate code solutions at different difficulty levels for introductory programming courses. We conducted an experiment where ChatGPT was tested on 127 randomly selected programming problems provided by Kattis, an automatic software grading tool for computer science programs, often used in higher education. The results showed that ChatGPT independently could solve 19 out of 127 programming tasks generated and assessed by Kattis. Further, ChatGPT was found to be able to generate accurate code solutions for simple problems but encountered difficulties with more complex programming tasks. The results contribute to the ongoing debate on the utility of AI-powered tools in programming education.

Automated programming assignment grading tools have become integral to CS courses at introductory as well as advanced levels. However a lot of these tools have their own custom approaches to setting up assignments and describing how solutions should be tested, requiring instructors to make a significant learning investment to begin using a new tool. In addition, differences between tools mean that initial investment must be repeated when switching tools or adding a new one. Worse still, tool-specific strategies further reduce the ability of educators to share and reuse their assignments. As a solution to this problem, we describe our experiences working with PEML, the Programming Exercise Markup Language, which provides an easy to use, instructor friendly approach for writing programming assignments. Unlike tool-oriented data interchange formats, PEML is designed to provide a human friendly authoring format that has been developed to be intuitive, expressive and not be a technological or notational barrier to instructors. We describe the design of PEML and also discuss its implementation as a programming library, a web application, and a microservice that provides full parsing and rendering capabilities for easy integration into any tools or scripting libraries. We also describe the integration of PEML into two automated testing and grading tools used at Virginia Tech by the CS department: Code Workout and Web-CAT. We then describe our experiences using PEML to describe a full range of programming assignments, laboratory exercises, and small coding questions of varying complexity in demonstrating the practicality of the notation. We evaluate the feasibility of PEML using this encoding exercise as well as the effect of its integration into the aforementioned automated grading tools. We finally present a framework for integrating PEML into existing grading tools and then draw our conclusions as well as list down avenues PEML can be expanded into in the future.

The version of record of this article, first published in "Education and information technologies", is available online at Publisher’s website: 10.1007/s10639-021-10876-7, The version of record is available online at: Education and information technologies http://dx.doi.org/10.1007/s10639-021-10876-7, Research on impact in student achievement of online homework systems compared to traditional methods is ambivalent. Methodological issues in the study design, besides of technological diversity, can account for this uncertainty. Hypothesis This study aims to estimate the effect size of homework practice with exercises automatically provided by the ‘e-status’ platform, in students from five Engineering programs. Instead of comparing students using the platform with others not using it, we distributed the subject topics into two blocks, and created nine probability problems for each block. After that, the students were randomly assigned to one block and could solve the related exercises through e-status. Teachers and evaluators were masked to the assignation. Five weeks after the assignment, all students answered a written test with questions regarding all topics. The study outcome was the difference between both blocks’ scores obtained from the test. The two groups comprised 163 and 166 students. Of these, 103 and 107 respectively attended the test, while the remainder were imputed with 0. Those assigned to the first block obtained an average outcome of −1.85, while the average in the second block was −3.29 (95% confidence interval of difference, −2.46 to −0.43). During the period in which they had access to the platform before the test, the average total time spent solving problems was less than three hours. Our findings provide evidence that a small amount of active online work can positively impact on student performance., Peer Reviewed, Postprint (author's final draft)

In this thesis, we have defined a symbolic execution technique to automatically generate test suites for programs written in functional programming languages that can find the behavioural differences between a reference implementation and a set of potentially different implementations. Our symbolic execution technique uses a constraint solver in order to find a model that satisfies all constraints that together represent an execution path through the program. Furthermore, our technique utilises manually defined budget constraints to guide the symbolic execution to more interesting areas of the programs. These budget constraints define an initial budget that dictates when the symbolic execution terminates, and a set of costs associated to certain operations such as calling (specific) functions or performing (specific) pattern matches. This allows the symbolic execution to deplete budgets faster when it explores functions or branches of a program that are deemed "unlikely to be interesting". This results in less system resources being used on exploring these uninteresting execution paths, and instead allows for exploration of deeper paths in more interesting areas of the programs. Our budget constraint optimisation strategy works alongside other well-known optimisation strategies for symbolic execution such as early branch pruning and reusing intermediate results, more commonly known in the field of symbolic execution as compositioning. In order to perform our symbolic execution on functional programs, we have defined a variation of Control Flow Graphs (CFG) for functional programming languages which allow modelling the execution flow between the different expressions that form the body of functions in functional programming. Additionally, our technique uses a constraint solver in order to find what paths through the program are feasible and to generate inputs that will lead to the execution of said paths. To allow us to specify these constraints on a higher, Computer Science

Automated grading of free-text exam responses is a very challenging task due to the complex nature of the problem, such as lack of training data and biased ground-truth of the graders. In this paper, we focus on the automated grading of free-text responses. We formulate the problem as a binary classification problem of two class labels: low- and high-grade. We present a benchmark on four machine learning methods using three experiment protocols on two real-world datasets, one from Cyber-crime exams in Arabic and one from Data Mining exams in English that is presented first time in this work. By providing various metrics for binary classification and answer ranking, we illustrate the benefits and drawbacks of the benchmarked methods. Our results suggest that standard models with individual word representations can in some cases achieve competitive predictive performance against deep neural language models using context-based representations on both binary classification and answer ranking for free-text response grading tasks. Lastly, we discuss the pedagogical implications of our findings by identifying potential pitfalls and challenges when building predictive models for such tasks.

In the education context, open-ended works generally entail a series of benefits as the possibility of develop original ideas and a more productive learning process to the student rather than closed-answer activities. Nevertheless, such works suppose a significant correction workload to the teacher in contrast to the latter ones that can be self-corrected. Furthermore, such workload turns to be intractable with large groups of students. In order to maintain the advantages of open-ended works with a reasonable amount of correction effort, this article proposes a novel methodology: students perform the corrections using a rubric (closed Likert scale) as a guideline in a peer-review fashion; then, their markings are automatically analyzed with statistical tools to detect possible biased scorings; finally, in the event the statistical analysis detects a biased case, the teacher is required to intervene to manually correct the assignment. This methodology has been tested on two different assignments with two heterogeneous groups of people to assess the robustness and reliability of the proposal. As a result, we obtain values over 95% in the confidence of the intra-class correlation test (ICC) between the grades computed by our proposal and those directly resulting from the manual correction of the teacher. These figures confirm that the evaluation obtained with the proposed methodology is statistically similar to that of the manual correction of the teacher with a remarkable decrease in terms of effort.

In the education context, open-ended works generally entail a series of benefits as the possibility of develop original ideas and a more productive learning process to the student rather than closed-answer activities. Nevertheless, such works suppose a significant correction workload to the teacher in contrast to the latter ones that can be self-corrected. Furthermore, such workload turns to be intractable with large groups of students. In order to maintain the advantages of open-ended works with a reasonable amount of correction effort, this article proposes a novel methodology: students perform the corrections using a rubric (closed Likert scale) as a guideline in a peer-review fashion; then, their markings are automatically analyzed with statistical tools to detect possible biased scorings; finally, in the event the statistical analysis detects a biased case, the teacher is required to intervene to manually correct the assignment. This methodology has been tested on two different assignments with two heterogeneous groups of people to assess the robustness and reliability of the proposal. As a result, we obtain values over 95% in the confidence of the intra-class correlation test (ICC) between the grades computed by our proposal and those directly resulting from the manual correction of the teacher. These figures confirm that the evaluation obtained with the proposed methodology is statistically similar to that of the manual correction of the teacher with a remarkable decrease in terms of effort.

In 2014, Massive Open Online Courses (MOOCs) are expected towitness a phenomenal growth in student registration compared to the previous years. As MOOCs continue to grow in number, therehas been an increasing focus on assessment and evaluation. Because of the huge enrollments in a MOOC, it is impossible for the instructor to grade homework and evaluate each student. The enormous data generated by learners in a MOOC can be used for developing and refining automated assessment techniques. As a result, “Smart Systems” are being designed to track and predict learner behavior while completing MOOC assessments. These automated assessments for MOOCs can automatically score and provide feedback to students multiple choice questions, mathematical problems and essays. Automated assessments help teachers with grading and also support students in the learning processes. These assessments are prompt, consistent, and support objectivity in assessment and evaluation (Ala-Mutka, 2005). This paper reviews the emerging trends in MOOC assessments and their application in supporting student learning and achievement. The paper concludes by describing how assessment techniques in MOOCs can help to maximize learning outcomes.

Educators in many disciplines are too often forced to rely on intuition about how students learn and the effectiveness of teaching to guide changes and improvements to their curricula. In computer science, systems that perform automated collection and assessment of programming assignments are seeing increased adoption, and these systems generate a great deal of meaningful intermediate data and statistics during the grading process. Continuous collection of these data and long-term retention of collected data present educators with a new resource to assess both learning (how well students understand a topic or how they behave on assignments) and teaching (how effective a response, intervention, or assessment instrument was in evaluating knowledge or changing behavior), by basing their decisions on evidence rather than intuition. It is only possible to achieve these goals, however, if such data are easily accessible. I present an infrastructure that has been added to one such automated grading system, Web-CAT, in order to facilitate routine data collection and access while requiring very little added effort by instructors. Using this infrastructure, I present three case studies that serve as representative examples of educational questions that can be explored thoroughly using pre-existing data from required student work. The first case study examines student time management habits and finds that students perform better when they start earlier but that offering extra credit for finishing earlier did not encourage them to do so. The second case study evaluates a tool used to improve student understanding of manual memory management and finds that students made fewer errors when using the tool. The third case study evaluates the reference tests used to grade student code on a selected assignment and confirms that the tests are a suitable instrument for assessing student ability. In each case study, I use a data-driven, evidence-based approach spanning multiple semesters

Automated grading systems have been researched at various universities for several years. Numerous systems have been developed that automate the process of grading by compiling, executing and testing the students submitted source code. However, such systems are mostly written as UNIX scripts and are restricted to performing one kind of activity. The instructors or teaching assistants have to resort to other methods in order to provide their feedback to the students. The core of this thesis is to research a TA feedback mechanism which will streamline the grading process for the professors and teaching assistants. A web-based grading tool has been developed that allows course staff to enter comments for students' programs directly through a web browser. The interface provides for full direct-manipulation editing of comments, which are then immediately viewable by students when they look up assignment results. Such an interface also has potential to support peer grading of assignments. Teaching assistants of introductory programming level courses were interviewed to learn about the different grading methods they use and were asked their opinion of our new grading interface. TAs were also asked to grade assignments using the traditional paper method as well as the computer using our new grading tool for comparison. Finally, an anonymous survey was sent out to various computer science faculties in different universities to gather information about the expectations they have with respect to TA grading activities for programming assignments and the learning outcomes that these professors desire for their students.

Practise is one of the most important steps in learning the art of computer programming. Unfortunately, human grading of programming assignments is a tedious and error-prone task, a problem compounded by the large enrolments of many programming courses. As a result, students in such courses tend to be given fewer programming assignments than should be ideally given. One solution to this problem is to automate the grading process such that students can electronically submit their programming assignments and receive instant feedback. This paper studies the implementation of one such automated grading system, called the Online Judge, in the School of Computing of the National University of Singapore for a compulsory first-year course that teaches basic programming techniques with over 700 students, describing the student reactions and behavior as well as the difficulties encountered. The Online Judge was also successfully employed for an advanced undergraduate course and an introductory high school course. (C) 2003 Elsevier Ltd. All rights reserved.

Practise is one of the most important steps in learning the art of computer programming. Unfortunately, human grading of programming assignments is a tedious and error-prone task, a problem compounded by the large enrolments of many programming courses. As a result, students in such courses tend to be given fewer programming assignments than should be ideally given. One solution to this problem is to automate the grading process such that students can electronically submit their programming assignments and receive instant feedback. This paper studies the implementation of one such automated grading system, called the Online Judge, in the School of Computing of the National University of Singapore for a compulsory first-year course that teaches basic programming techniques with over 700 students, describing the student reactions and behavior as well as the difficulties encountered. The Online Judge was also successfully employed for an advanced undergraduate course and an introductory high school course. (C) 2003 Elsevier Ltd. All rights reserved.

Practise is one of the most important steps in learning the art of computer programming. Unfortunately, human grading of programming assignments is a tedious and error-prone task, a problem compounded by the large enrolments of many programming courses. As a result, students in such courses tend to be given fewer programming assignments than should be ideally given. One solution to this problem is to automate the grading process such that students can electronically submit their programming assignments and receive instant feedback. This paper studies the implementation of one such automated grading system, called the Online Judge, in the School of Computing of the National University of Singapore for a compulsory first-year course that teaches basic programming techniques with over 700 students, describing the student reactions and behavior as well as the difficulties encountered. The Online Judge was also successfully employed for an advanced undergraduate course and an introductory high school course. (C) 2003 Elsevier Ltd. All rights reserved.

This paper describes how conventional image processing techniques can be applied to the grading of Southern Rock Lobsters (SRL) to produce a high quality data layer which could be an input into product traceability. The research is part of a broader investigation into designing a low-cost biometric identification solution for use along the entire lobster supply chain. In approaching the image processing for lobster grading a key consideration is to develop a system capable of using low cost consumer grade cameras readily available in mobile phones. The results confirm that by combining a number of common techniques in computer vision it is possible to capture and process a set of valuable attributes from sampled lobster image including color, length, weight, legs and sex. By combining this image profile with other pre-existing data on catch location and landing port each lobster can be verifiably tracked along the supply chain journey to markets in China. The image processing research results achieved in the laboratory show high accuracy in measuring lobster carapace length that is vital for weight conversion calculations. The results also demonstrate the capability to obtain reliable values for average color, tail shape and number of legs on a lobster used in grading classifications. The findings are a major first step in the development of individual lobster biometric identification and will directly contribute to automating lobster grading in this valuable Australian fishery.