Search

Automated programming assignment grading tools have become integral to CS courses at introductory as well as advanced levels. However a lot of these tools have their own custom approaches to setting up assignments and describing how solutions should be tested, requiring instructors to make a significant learning investment to begin using a new tool. In addition, differences between tools mean that initial investment must be repeated when switching tools or adding a new one. Worse still, tool-specific strategies further reduce the ability of educators to share and reuse their assignments. As a solution to this problem, we describe our experiences working with PEML, the Programming Exercise Markup Language, which provides an easy to use, instructor friendly approach for writing programming assignments. Unlike tool-oriented data interchange formats, PEML is designed to provide a human friendly authoring format that has been developed to be intuitive, expressive and not be a technological or notational barrier to instructors. We describe the design of PEML and also discuss its implementation as a programming library, a web application, and a microservice that provides full parsing and rendering capabilities for easy integration into any tools or scripting libraries. We also describe the integration of PEML into two automated testing and grading tools used at Virginia Tech by the CS department: Code Workout and Web-CAT. We then describe our experiences using PEML to describe a full range of programming assignments, laboratory exercises, and small coding questions of varying complexity in demonstrating the practicality of the notation. We evaluate the feasibility of PEML using this encoding exercise as well as the effect of its integration into the aforementioned automated grading tools. We finally present a framework for integrating PEML into existing grading tools and then draw our conclusions as well as list down avenues PEML can be expanded into in the future. Master of Science Automated grading tools have become ubiquitous to CS courses focused on programming concepts at both the undergraduate as well as graduate level. These tools allow instructors to provide near instant feedback to students as well as spend more time focusing on the curriculum rather than grading. However, these tools use a variety programming assignment representation formats and without a standardized representation, instructors and educators may struggle to share and reuse assignments across different tools and platforms. To address this need, we have developed the Programming Exercise Markup Language (PEML), a standardized format for representing programming exercises, designed to be human-friendly as well as easy to learn and use. PEML includes information about the problem statement, input and output formats, constraints, and sample test cases, and can be used for a wide range of exercise types and programming languages. As part of this master's thesis project, we encoded 50 assignments of varying size and difficulty into PEML as well as integrated support for PEML into Web-CAT and Code Workout, two commonly used automated grading tools used at Virginia Tech. Building upon our experience performing this task, we also designed a framework that can be utilized when integrating PEML into other automated grading tools. By providing a standardized way of representing programming assignments, PEML can help to streamline programming education and make it easier for instructors and educators to create and share assignments across different tools and platforms.

This is a pre-registration form for an upcoming systematic review, including search terms and, inclusion and exclusion criteria. The form is based on the "Inclusive Systematic Review Registration Form" (https://osf.io/by27q/)

Background: Reliable, time- and cost-effective, and clinician-friendly diagnostic tools are cornerstones in facial palsy (FP) patient management. Different automated FP grading systems have been developed but revealed persisting downsides such as insufficient accuracy and cost-intensive hardware. We aimed to overcome these barriers and programmed an automated grading system for FP patients utilizing the House and Brackmann scale (HBS). Methods: Image datasets of 86 patients seen at the Department of Plastic, Hand, and Reconstructive Surgery at the University Hospital Regensburg, Germany, between June 2017 and May 2021, were used to train the neural network and evaluate its accuracy. Nine facial poses per patient were analyzed by the algorithm. Results: The algorithm showed an accuracy of 100%. Oversampling did not result in altered outcomes, while the direct form displayed superior accuracy levels when compared to the modular classification form (n = 86; 100% vs. 99%). The Early Fusion technique was linked to improved accuracy outcomes in comparison to the Late Fusion and sequential method (n = 86; 100% vs. 96% vs. 97%). Conclusions: Our automated FP grading system combines high-level accuracy with cost- and time-effectiveness. Our algorithm may accelerate the grading process in FP patients and facilitate the FP surgeon’s workflow.

Massive open online courses (MOOCs) provide hundreds of students with teaching materials, assessment tools, and collaborative instruments. The assessment activity, in particular, is demanding in terms of both time and effort; thus, the use of artificial intelligence can be useful to address and reduce the time and effort required. This paper reports on a system and related experiments finalised to improve both the performance and quality of formative and summative assessments in specific data science courses. The system is developed to automatically grade assignments composed of R commands commented with short sentences written in natural language. In our opinion, the use of the system can (i) shorten the correction times and reduce the possibility of errors and (ii) support the students while solving the exercises assigned during the course through automated feedback. To investigate these aims, an ad-hoc experiment was conducted in three courses containing the specific topic of statistical analysis of health data. Our evaluation demonstrated that automated grading has an acceptable correlation with human grading. Furthermore, the students who used the tool did not report usability issues, and those that used it for more than half of the exercises obtained (on average) higher grades in the exam. Finally, the use of the system reduced the correction time and assisted the professor in identifying correction errors.

Computer science class enrollments have rapidly risen in the past decade. With current class sizes, standard approaches to grading and providing personalized feedback are no longer possible and new techniques become both feasible and necessary. In this paper, we present the third version of Automata Tutor, a tool for helping teachers and students in large courses on automata and formal languages. The second version of Automata Tutor supported automatic grading and feedback for finite-automata constructions and has already been used by thousands of users in dozens of countries. This new version of Automata Tutor supports automated grading and feedback generation for a greatly extended variety of new problems, including problems that ask students to create regular expressions, context-free grammars, pushdown automata and Turing machines corresponding to a given description, and problems about converting between equivalent models - e.g., from regular expressions to nondeterministic finite automata. Moreover, for several problems, this new version also enables teachers and students to automatically generate new problem instances. We also present the results of a survey run on a class of 950 students, which shows very positive results about the usability and usefulness of the tool.

The version of record of this article, first published in "Education and information technologies", is available online at Publisher’s website: 10.1007/s10639-021-10876-7 The version of record is available online at: Education and information technologies http://dx.doi.org/10.1007/s10639-021-10876-7 Research on impact in student achievement of online homework systems compared to traditional methods is ambivalent. Methodological issues in the study design, besides of technological diversity, can account for this uncertainty. Hypothesis This study aims to estimate the effect size of homework practice with exercises automatically provided by the ‘e-status’ platform, in students from five Engineering programs. Instead of comparing students using the platform with others not using it, we distributed the subject topics into two blocks, and created nine probability problems for each block. After that, the students were randomly assigned to one block and could solve the related exercises through e-status. Teachers and evaluators were masked to the assignation. Five weeks after the assignment, all students answered a written test with questions regarding all topics. The study outcome was the difference between both blocks’ scores obtained from the test. The two groups comprised 163 and 166 students. Of these, 103 and 107 respectively attended the test, while the remainder were imputed with 0. Those assigned to the first block obtained an average outcome of −1.85, while the average in the second block was −3.29 (95% confidence interval of difference, −2.46 to −0.43). During the period in which they had access to the platform before the test, the average total time spent solving problems was less than three hours. Our findings provide evidence that a small amount of active online work can positively impact on student performance.

Grading student activities in online courses is a time-expensive task, especially with a high number of students in the course. To avoid a bottleneck in the continuous evaluation process, quizzes with multiple choice questions are frequently used. However, a quiz fails on the provision of formative feedback to the student. This work presents PLeNTaS, a system for the automatic grading of short answers from open domains, that reduces the time required for the grading task and offers formative feedback to the students. It is based on the analysis of the text from the point of view of three different levels: orthography, syntax, and semantics. The validation of the system will consider the correlation of the assigned grade with the human grade, the utility of the automatically generated feedback and the pedagogical impact caused by the system usage in the course.

Programming Style is an important part of program quality and it should be taken into account while assessing programs designedby competitors in informatics. In InternationalOlympiad in Informatics grading is automated and based on testing results only, while programming style is not taken into account. However there exists practice in universities in programming courses where programming style of submitted programs is evaluated automatically. The paper reviews existing experience and discusses possibilities for automated grading of programming style in informatics olympiads.

Programming is a very complex and challenging subject to teach and learn. A strategy guaranteed to deliver proven results has been intensive and continual training. However, this strategy holds an extra workload for the teachers with huge numbers of programming assignments to evaluate in a fair and timely manner. Furthermore, under the current COVID-19 distance teaching circumstances, regular assessment is a fundamental feedback mechanism. It ensures that students engage in learning as well as determines the extent to which they reached the expected learning goals, in this new learning reality. In sum, automating the assessment process will be particularly appreciated by the instructors and highly beneficial to the students. The purpose of this paper is to investigate the feasibility of automatic assessment in the context of computer programming courses. Thus, a prototype based on merging static and dynamic analysis was developed. Empirical evaluation of the proposed grading tool within an introductory C-language course has been presented and compared to manually assigned marks. The outcomes of the comparative analysis have shown the reliability of the proposed automatic assessment prototype.

The paper deals with the automated grading of assignments made up of R commands, their output and comments written in natural language. Compared to other tools presented in the literature, our tool supports both students and teachers, uses static source code analysis for the code-snippets and a supervised classifier based on sentence embeddings for the open-ended answers, and provides a feedback to students and includes the instructor review. After more than one year of use, improvements in terms of the feedback provided to the students are discussed in the paper, that in turn should offer manifold benefits to both students and teachers. Finally, the paper proposes a study finalised to refine and test the effectiveness of the proposal.

Software testing has become a standard for most software projects. However, there is a lack of testing in many curricula, and if present, courses lack instant feedback using automated systems. In this work, we show our realization of an exercise to teach software testing using an automated grading system. We illustrate our didactic goals, describe the task design and technical implementation. Evaluation shows that students experience only a slight increase in difficulty compared to other tasks and perceive the task description as sufficient.

Mobile applications are very challenging to test as they usually have a complex graphical user interface and advanced functionality that involves interacting with remote services. Due to these features, student assessment in courses about mobile application development usually relies on assignments or projects that are manually checked by teaching assistants for grading. This approach clearly does not scale to large classrooms, especially for online courses. This article presents a novel system for automated assessment of Android exercises with cloud-native technologies. Different from the state of the art, the proposed solution leverages a mobile app testing framework that is largely used in the industry instead of custom libraries. Furthermore, the devised system employs software containers and scales with the availability of resources in a data center, which is essential for massive open online courses. The system design and implementation is detailed, together with the results from a deployment within a master-level course with 120 students. The received feedback demonstrates that the proposed solution was effective, as it provided insightful feedback and supported independent learning of mobile application development.

Background MRI scanning has revolutionized the clinical diagnosis of lumbar spinal stenosis (LSS). However, there is currently no consensus as to how best to classify MRI findings which has hampered the development of robust longitudinal epidemiological studies of the condition. We developed and tested an automated system for grading lumbar spine MRI scans for central LSS for use in epidemiological research. Methods Using MRI scans from the large population-based cohort study (the Wakayama Spine Study), all graded by a spinal surgeon, we trained an automated system to grade central LSS in four gradings of the bone and soft tissue margins: none, mild, moderate, severe. Subsequently, we tested the automated grading against the independent readings of our observer in a test set to investigate reliability and agreement. Results Complete axial views were available for 4855 lumbar intervertebral levels from 971 participants. The machine used 4365 axial views to learn (training set) and graded the remaining 490 axial views (testing set). The agreement rate for gradings was 65.7% (322/490) and the reliability (Lin’s correlation coefficient) was 0.73. In 2.2% of scans (11/490) there was a difference in classification of 2 and in only 0.2% (1/490) was there a difference of 3. When classified into 2 groups as ‘severe’ vs ‘no/mild/moderate’. The agreement rate was 94.1% (461/490) with a kappa of 0.75. Conclusions This study showed that an automated system can “learn” to grade central LSS with excellent performance against the reference standard. Thus SpineNet offers potential to grade LSS in large-scale epidemiological studies involving a high volume of MRI spine data with a high level of consistency and objectivity.

Educational systems providing Automated Grading can be very useful for both learner and teacher. After their design and implementation, systems providing automated grading have to be tested and validated, which let the need for real world experiments arise. In this paper we describe an approach to “simulated experiments” that we hope can ease the task of the researchers developing the above mentioned systems. The concept of our proposal is in defining a simulated class by means of statistical distributions of the features that model the student and her/his work in the system. Since such features can be very different from a system to another, we try and keep the description of our framework as general as possible, and define a simulated class by the distribution of the grades that the learners should get (and the system should infer, if correctly working). We show the use of our approach with two systems, that do grading and model the students quite differently. We conclude that our simulation framework can be beneficial in validating an automated grading system, and in allowing to reflect on possible updates of the system, to make its grading more correct.

Assessing the academic capabilities of students should play a key role in both stimulating their learning process (formative assessment) and in the accurate evaluation of their knowledge and capabilities in relation to a topic (summative assessment). Therefore, according to the principle of constructive alignment, any form of assessment needs to be carefully designed to match the learning outcomes of a course and needs to be delivered in an appropriate format (paper-based vs. computer-based) and graded in a suitable manner. However, this is a challenging task, due to the substantial amount of time teachers need to spend on grading open questions. From our experience, this results in using less appropriate assessment methods (e.g.: Multiple Choice questions) or in less time spent by teachers on innovating their courses (e.g.: implementation of formative assessment). Inspired by recent developments in academia and practice, we propose to investigate the application of machine learning technology for supporting grading of open questions, with a focus on summative assessment and exploring possibilities for formative assessment. Our expected results include the design of a method for supporting grading of open questions with machine learning, an investigation into the most suitable machine learning algorithms for small samples of tests.

Assessing the academic capabilities of students should play a key role in both stimulating their learning process (formative assessment) and in the accurate evaluation of their knowledge and capabilities in relation to a topic (summative assessment). Therefore, according to the principle of constructive alignment, any form of assessment needs to be carefully designed to match the learning outcomes of a course and needs to be delivered in an appropriate format (paper-based vs. computer-based) and graded in a suitable manner. However, this is a challenging task, due to the substantial amount of time teachers need to spend on grading open questions. From our experience, this results in using less appropriate assessment methods (e.g.: Multiple Choice questions) or in less time spent by teachers on innovating their courses (e.g.: implementation of formative assessment). Inspired by recent developments in academia and practice, we propose to investigate the application of machine learning technology for supporting grading of open questions, with a focus on summative assessment and exploring possibilities for formative assessment. Our expected results include the design of a method for supporting grading of open questions with machine learning, an investigation into the most suitable machine learning algorithms for small samples of tests.

Automatic short-answer grading (ASAG) is a system that aims to help speed up the assessment process without an instructor’s intervention. Previous research had successfully built an ASAG system whose performance had a correlation of 0.66 and mean absolute error (MAE) starting from 0.94 with a conventionally graded set. However, this study had a weakness in the need for more than one reference answer for each question. It used a string-based equation method and keyword matching process to measure the sentences’ similarity in order to produce an assessment rubric. Thus, our study aimed to build a more concise short-answer automatic scoring system using a single reference answer. The mechanism used a semantic similarity measurement approach through word embedding techniques and syntactic analysis to assess the learner’s accuracy. Based on the experiment results, the semantic similarity approach showed a correlation value of 0.70 and an MAE of 0.70 when compared with the grading reference.

Automated grading tools either execute the submitted code against sample data or statically analyse it. The paper presents a tool for the automated grading of R code snippets, submitted by students learning Health Informatics in the degree course of Medicine and Surgery of the University of L’Aquila (Italy). The tool performs a static analysis of the R commands, with the respective output, as well as of the sentences written in natural language. The paper details the problem in general and through examples. Then, it describes the proposed solution and reports on the comparison between the automated grading and the human one. Finally, the paper ends by describing the foreseen use of the tool and the needed improvements.

Programming and software engineering courses in computer science curricula typically focus on both providing theoretical knowledge of programming languages and best-practices, and developing practical development skills. In a massive course - several hundred students - the teachers are not able to adequately attend to the practical part, therefore process automation and incentives to students must be used to drive the students in the right direction. Our goals was to design an automated programming assignment infrastructure capable of supporting massive courses. The infrastructure should encourage students to apply the key software engineering (SE) practices - automated testing, con guration management, and Integrated Development Environment (IDE) - and acquire the basic skills for using the corresponding tools. We selected a few widely adopted development tools used to support the key software engineering practices and mapped them to the basic activities in our exam assignment management process. This experience report describes the results from the past academic year. The infrastructure we built has been used for a full academic year and supported four exam sessions for a total of over a thousand students. The satisfaction level reported by the students is generally high., Accepted for SEEM 2018 - Software Engineering Education for Millennials, colocated with ICSE 2018

In the education context, open-ended works generally entail a series of benefits as the possibility of develop original ideas and a more productive learning process to the student rather than closed-answer activities. Nevertheless, such works suppose a significant correction workload to the teacher in contrast to the latter ones that can be self-corrected. Furthermore, such workload turns to be intractable with large groups of students. In order to maintain the advantages of open-ended works with a reasonable amount of correction effort, this article proposes a novel methodology: students perform the corrections using a rubric (closed Likert scale) as a guideline in a peer-review fashion; then, their markings are automatically analyzed with statistical tools to detect possible biased scorings; finally, in the event the statistical analysis detects a biased case, the teacher is required to intervene to manually correct the assignment. This methodology has been tested on two different assignments with two heterogeneous groups of people to assess the robustness and reliability of the proposal. As a result, we obtain values over 95% in the confidence of the intra-class correlation test (ICC) between the grades computed by our proposal and those directly resulting from the manual correction of the teacher. These figures confirm that the evaluation obtained with the proposed methodology is statistically similar to that of the manual correction of the teacher with a remarkable decrease in terms of effort. This work has been supported by the Vicerrectorado de Calidad e Innovación Educativa-Instituto de Ciencias de la Educación of the Universidad de Alicante (2016-17 edition) through the Programa de Redes-I3CE de investigación en docencia universitaria (ref. 3690).

Purpose Uveitis is associated with accumulation of exudate in the vitreous, which reduces fundus visibility. The condition is assessed in patients by subjectively matching fundus photographs to a six-level (NIH) or nine-level (Miami) haze scale. This study aimed to develop an objective method of assessing vitreous haze. Methods An image-processing algorithm was designed that quantifies vitreous haze via high-pass filtering, entropy analysis, and power spectrum integration. The algorithm was refined using nine published photographs that represent incremental levels of fundus blur and applied without further refinement to 120 random fundus photographs from a uveitis image library. Computed scores were compared against the grades of two trained readers of vitreous haze and against acutance, a generic measure of image clarity, using Cohen's κ and Gwet's AC statistics. Results Exact agreement between algorithm scores and reader grades was substantial for both NIH and Miami scales (κ = 0.61 and 0.67, AC = 0.82 and 0.92). Within-one (κ = 0.78 and 0.82) and within-two (κ = 0.80 and 0.84) levels of agreement were almost perfect. The correspondence was comparable to that between readers. Whereas, exact (κ = 0.45 and 0.44, AC = 0.73 and 0.75), within-one (κ = 0.69 and 0.68), and within-two (κ = 0.73 and 0.72) levels of agreement for the two scales were moderate to substantial for acutance calculations. Conclusions The computer algorithm produces a quantitative measure of vitreous haze that correlates strongly with the perception of expert graders. Translational relevance The work offers a rapid, unbiased, standardized means of assessing vitreous haze for clinical and telemedical monitoring of uveitis patients.

Context: The number of students enrolled in universities at standard and on-line programming courses is rapidly increasing. This calls for automated evaluation of students assignments. Objective: We aim to develop methods and tools for objective and reljable automated grading that can also provide substantial and comprehensible feedback. Our approach targets introductory programming courses, which have a number of specific features and goals. The benefits are twofold: reducing the workload for teachers, and providing helpful feedback to students in the process of learning. Method: For sophisticated automated evaluation of students' programs, our grading framework combines results of three approaches (i) testing, (ii) software verification, and (iii) control flow graph similarity measurement. We present our tools for software verification and control flow graph similarity measurement, which are publicly available and open source. The tools are based on an intermediate code representation, so they could be applied to a number of programming languages. Results: Empirical evaluation of the proposed grading framework is performed on a corpus of programs written by university students in programming language C within an introductory programming course. Results of the evaluation show that the synergy of proposed approaches improves the quality and precision of automated grading and that automatically generated grades are highly correlated with instructor-assigned grades. Also, the results show that our approach can be trained to adapt to teacher's grading style. Conclusions: In this paper we integrate several techniques for evaluation of student's assignments. The obtained results suggest that the presented tools can find real-world applications in automated grading. (C) 2012 Elsevier B.V. All rights reserved.

Students in higher education require satisfactory computer skills to be successful. While today’s students may have greater exposure to technology, research shows that their actual computer knowledge and skills are superficial and narrow. As a result, the freshman computer literacy course remains an important curricular component. This study investigates the implementation of an innovative Web-based technology for delivering software proficiency training for Microsoft Office. Building upon decades of end-user computing satisfaction and technology acceptance research, the purpose of the study is to describe the instructor and student experiences that result from the implementation and use of MyITLab educational software. The nature of the study is descriptive, rather than evaluative, with the following goals: (a) to describe instructors’ experiences with the software, (b) to identify patterns of technology usage and utility, and (c) to elucidate levels of computing satisfaction and technology acceptance among users. The study applies a mixed-method, single-unit, embedded case study design to focus the inquiry on an introductory computer applications course, offered in the Fall 2011 semester at a college in western Canada. The embedded units consist of five instructors, with 322 students enrolled across 10 sections. Data were analyzed from course documents, classroom observations, instructor interviews, and a student survey that produced 149 satisfactory responses. The survey was constructed by adapting instruments based on the Wixom and Todd (2005) integrated research model and the Unified Theory of Acceptance and Use of Technology (UTAUT) model. Results of the study are summarized into five assertions: 1) MyITLab effectively eliminates or, at least, reduces instructor grading workloads for assignments, 2) MyITLab provides students with frequent corrective feedback on assignments, 3) the step-by-step presentation of instructions in MyITLab may not solely meet the needs of solution-based learning outcomes, 4) instructors should be trained on MyITLab to maximize the software’s utility, and 5) the MyITLab solution bank of acceptable responses should be expanded to reduce potential grading inaccuracies. An enhanced Wixom and Todd (2005) model is also presented for future research of educational software. Lastly, the reader is encouraged to reconsider the information presented and generalize it for their own purposes.

Pastaruoju metu mokslinėje visuomenėje vis daugiau atgarsio sulaukia įvairios informatikos (algoritmavimo) olimpiados. Ši sritis tampa mokslinių tyrimų objektu. Informatikos olimpiadų dalyviai turi sukurti algoritmus ir juos užrašyti nepriekaištingai veikiančiomis programomis viena iš leidžiamų programavimo kalbų. Darbai yra vertinami automatiškai testuojant juodosios dėžės principu, visiškai nebandant analizuoti paties algoritmo. Toks vertinimas sulaukia nemažai kritikos, tačiau kol kas gerų alternatyvų nėra. Straipsnyje pateikiama automatinio vertinimo informatikos olimpiadose raida nuo pirmųjų olimpiadų iki šių dienų, apžvelgiami darbai, kuriuose nagrinėjama ši problematika, aptariamos olimpiadose ir mokslinėje literatūroje siūlytos patobulintos vertinimo schemos, numatomos ateities gairės.Development and perspectives of automated grading in informatics olympiadsJūratė Skūpienė SummaryInternational Olympiads in Informatics (IOI) lately are gaining more attention in the scientific community. Contestants in IOI have to design algorithms and implement them as programs in one of allowed programming languages. Currently all the submissions are graded automatically using black-box method and grading is based on execution of compiled programs with different tests (input data). The algorithm itself is not analysed or revealed in any other possible way. Current gradingsystem receives a lot of criticism due its unfairness (mistyping the name of a variable might lead to zero points), however, no better grading models have been proposed so far. The paper gives an overview of the development of grading in IOI starting from the very first IOI’s where the verbal description of an algorithm had to be presented and evaluated. Many grading problems emerged in 1990’s due to input/output format requirements which where inevitable in order to be able to perform automated testing of submissions. Input/output details required a lot of concentration from participants for mistyping a file name or redundant/missing end of line symbol might have resulted in zero points. Sometimes that even shifted focus from algorithm to formatting details. These problems were solved with the appearance of the first Informatics Contest Management System in 2001 which allowed the contestants to submit and run their program with sample tests thus checking their programs for compatibility with format requirements and correcting if there was such a need during contest time. After the contest management systems found their place in IOI’s the attention was shifted to the relationship between the grades given for a submission and the algorithm implemented. The paper presents overview of published papers and discussions in IOI community on these topics. The papers ends with perspectives and directions for future improvement of grading in IOI’s.

Undergraduates are taught the database language SQL using internet based tools. The managed learning environment includes assessment.

Programming tasks are an important part of teaching computer programming as they foster students to develop essential programming skills and techniques through practice. The design of educational problems plays a crucial role in the extent to which the experiential knowledge is imparted to the learner both in terms of quality and quantity. Badly designed tasks have been known to put-off students from practicing programming. Hence, there is a need for carefully designed problems. Cellular Automata programming lends itself as a very suitable candidate among problems designed for programming practice. In this paper, we describe how various types of problems can be designed using concepts from Cellular Automata and discuss the features which make them good practice problems with regard to instructional pedagogy. We also present a case study on a Cellular Automata programming exercise used in a MOOC on Test Driven Development using JUnit, and discuss the automated evaluation of code submissions and the feedback about the reception of this exercise by participants in this course. Finally, we suggest two ideas to facilitate an easier approach of creating such programming exercises.

In 2014, Massive Open Online Courses (MOOCs) are expected towitness a phenomenalgrowth in student registration compared to the previous years. As MOOCs continue to grow in number, therehas been an increasing focus on assessment and evaluation. Becauseof the huge enrollments in a MOOC, it is impossible for the instructor to grade homeworkand evaluate each student. The enormous data generated by learners in a MOOC can beused for developing and refining automated assessment techniques. As a result, SmartSystems are being designed to track and predict learner behavior while completingMOOC assessments. These automated assessments for MOOCs can automatically scoreand provide feedback to students multiple choice questions, mathematical problems andessays. Automated assessments help teachers with grading and also support students in thelearning processes. These assessments are prompt, consistent, and support objectivity inassessment and evaluation (Ala-Mutka, 2005). This paper reviews the emerging trends inMOOC assessments and their application in supporting student learning and achievement.The paper concludes by describing how assessment techniques in MOOCs can help tomaximize learning outcomes., In 2014, Massive Open Online Courses (MOOCs) are expected to witness a phenomenal growth in student registration compared to the previous years (Lee, Stewart, Claugar-Pop, 2014). As MOOCs continue to grow in number, there has been an increasing focus on assessment and evaluation. Because of the huge enrollments in a MOOC, it is impossible for the instructor to grade homework and evaluate each student. The enormous data generated by learners in a MOOC can be used for developing and refining automated assessment techniques. As a result, “Smart Systems” are being designed to track and predict learner behavior while completing MOOC assessments. These automated assessments for MOOCs can automatically score and provide feedback to students multiple choice questions, mathematical problems and essays. Automated assessments help teachers with grading and also support students in the learning processes. Theseassessments are prompt, consistent, and support objectivity in assessment and evaluation (Ala-Mutka, 2005). This paper reviews the emerging trends in MOOC assessments and their application in supporting student learning and achievement. The paper concludes by describing how assessment techniques in MOOCs can help to maximize learning outcomes.

Educators in many disciplines are too often forced to rely on intuition about how students learn and the effectiveness of teaching to guide changes and improvements to their curricula. In computer science, systems that perform automated collection and assessment of programming assignments are seeing increased adoption, and these systems generate a great deal of meaningful intermediate data and statistics during the grading process. Continuous collection of these data and long-term retention of collected data present educators with a new resource to assess both learning (how well students understand a topic or how they behave on assignments) and teaching (how effective a response, intervention, or assessment instrument was in evaluating knowledge or changing behavior), by basing their decisions on evidence rather than intuition. It is only possible to achieve these goals, however, if such data are easily accessible. I present an infrastructure that has been added to one such automated grading system, Web-CAT, in order to facilitate routine data collection and access while requiring very little added effort by instructors. Using this infrastructure, I present three case studies that serve as representative examples of educational questions that can be explored thoroughly using pre-existing data from required student work. The first case study examines student time management habits and finds that students perform better when they start earlier but that offering extra credit for finishing earlier did not encourage them to do so. The second case study evaluates a tool used to improve student understanding of manual memory management and finds that students made fewer errors when using the tool. The third case study evaluates the reference tests used to grade student code on a selected assignment and confirms that the tests are a suitable instrument for assessing student ability. In each case study, I use a data-driven, evidence-based approach spanning multiple semesters and students, allowing me to answer each question in greater detail than was possible using previous methods and giving me significantly increased confidence in my conclusions. Ph. D.

Automated grading systems have been researched at various universities for several years. Numerous systems have been developed that automate the process of grading by compiling, executing and testing the students submitted source code. However, such systems are mostly written as UNIX scripts and are restricted to performing one kind of activity. The instructors or teaching assistants have to resort to other methods in order to provide their feedback to the students. The core of this thesis is to research a TA feedback mechanism which will streamline the grading process for the professors and teaching assistants. A web-based grading tool has been developed that allows course staff to enter comments for students' programs directly through a web browser. The interface provides for full direct-manipulation editing of comments, which are then immediately viewable by students when they look up assignment results. Such an interface also has potential to support peer grading of assignments. Teaching assistants of introductory programming level courses were interviewed to learn about the different grading methods they use and were asked their opinion of our new grading interface. TAs were also asked to grade assignments using the traditional paper method as well as the computer using our new grading tool for comparison. Finally, an anonymous survey was sent out to various computer science faculties in different universities to gather information about the expectations they have with respect to TA grading activities for programming assignments and the learning outcomes that these professors desire for their students. Master of Science

Purpose: To increase the efficiency of retinal image grading, algorithms for automated grading have been developed, such as the IDx-DR 2.0 device. We aimed to determine the ability of this device, incorporated in clinical work flow, to detect retinopathy in persons with type 2 diabetes. Methods: Retinal images of persons treated by the Hoorn Diabetes Care System (DCS) were graded by the IDx-DR device and independently by three retinal specialists using the International Clinical Diabetic Retinopathy severity scale (ICDR) and EURODIAB criteria. Agreement between specialists was calculated. Results of the IDx-DR device and experts were compared using sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV), distinguishing between referable diabetic retinopathy (RDR) and vision-threatening retinopathy (VTDR). Area under the receiver operating characteristic curve (AUC) was calculated. Results: Of the included 1415 persons, 898 (63.5%) had images of sufficient quality according to the experts and the IDx-DR device. Referable diabetic retinopathy (RDR) was diagnosed in 22 persons (2.4%) using EURODIAB and 73 persons (8.1%) using ICDR classification. Specific intergrader agreement ranged from 40% to 61%. Sensitivity, specificity, PPV and NPV of IDx-DR to detect RDR were 91% (95% CI: 0.69-0.98), 84% (95% CI: 0.81-0.86), 12% (95% CI: 0.08-0.18) and 100% (95% CI: 0.99-1.00; EURODIAB) and 68% (95% CI: 0.56-0.79), 86% (95% CI: 0.84-0.88), 30% (95% CI: 0.24-0.38) and 97% (95% CI: 0.95-0.98; ICDR). The AUC was 0.94 (95% CI: 0.88-1.00; EURODIAB) and 0.87 (95% CI: 0.83-0.92; ICDR). For detection of VTDR, sensitivity was lower and specificity was higher compared to RDR. AUC's were comparable. Conclusion: Automated grading using the IDx-DR device for RDR detection is a valid method and can be used in primary care, decreasing the demand on ophthalmologists.

Manual grading of structured query language (SQL) statements after an exam can be tedious and time consuming for the teaching assistant. Additionally, it can also be subjective to her current state of mind and, thus, prone to errors. In this paper we propose an automated method for grading individual SQL statements. The method uses several common and simple string similarity metrics for comparing the student devised statements against the reference statements. These are then used, along with the manually assigned grades, for building the predictive logistic regression model. The proposed method was evaluated on a dataset consisting of 314 pairs of student-reference statements, along with the discretized average grade assigned by three independent evaluators. The model achieved the expected classification accuracy of 78% on a binary class, thus exhibiting its potential for real-life application. The model can be used as is with the suggested calculated features and reported learnt parameters, or adapted to other examiners' evaluation criteria, presuming their willingness to build manually graded datasets of their own.

We present an automated approach to verify the correctness of programming assignments, such as the ones that arise in a functional programming course. Our approach takes as input student submissions and reference solutions, and uses equivalence checking to automatically prove or disprove correctness of each submission. To be effective in the context of a real-world programming course, an automated grading system must be both robust, to support programs written in a variety of style, and scalable, to treat hundreds of submissions at once. We achieve robustness by handling recursion using functional induction and by handling auxiliary functions using function call matching. We achieve scalability using a clustering algorithm that leverages the transitivity of equivalence to discover intermediate reference solutions among student submissions. We implement our approach on top of the Stainless verification system, to support equivalence checking of Scala programs. We evaluate our system and its components on over 4000 programs drawn from a functional programming course and from the program equivalence checking literature; this is the largest such evaluation to date. We show that our system is capable of proving program correctness by generating inductive equivalence proofs, and providing counterexamples for incorrect programs, with a high success rate.

We introduce a method for automated grading of handwritten essays written by foreign language learners of French. The handwriting recognition system allows digitising the essays for further processing and functions at a low character error rate. The transcriptions are then vectorised using embeddings from state-of-the-art pre-trained natural language processing models. On top of the extracted word-level features, a deep recurrent network was trained for grade predictions for essays, using the nine different grading criteria as target variables. Scores on these criteria were previously obtained from human expert raters for more than 6’000 student essays. We present preliminary findings on prediction accuracy and discuss possible future developments and applications of the system.