Search

Peer-review systems such as SWoRD lack intelligence for detecting and responding to problems with students’ reviewing performance. While prior work has demonstrated the feasibility of automatically identifying desirable feedback features in free-text reviews of student papers, similar methods have not yet been developed for feedback regarding argument diagrams. One desirable feedback feature is problem localization, which has been shown to positively correlate with feedback implementation in both student papers and argument diagrams. In this paper we demonstrate that features previously developed for identifying localization in paper reviews do not work well when applied to peer reviews of argument diagrams. We develop a novel algorithm tailored for reviews of argument diagrams, and demonstrate significant performance improvements in identifying problem localization in an experimental evaluation.

Peer review and citation analysis are the two most common approaches for quality evaluations of scientific publications, although they are subject to criticism for various reasons. This paper outlines the problems of citation analysis and peer review and introduces Collaborative Document Evaluation as a supplement or possibly even a substitute. Collaborative Document Evaluation aims to enable the readers of publications to act as peer reviewers and share their evaluations in the form of ratings, annotations, links and classifications via the internet. In addition, Collaborative Document Evaluation might well enhance the search for publications. In this paper the implications of Collaborative Document Evaluation for the scientific community are discussed and questions are asked as to how to create incentives for scientists to participate.

In this paper we propose an Open Access model for Legal Information Institutes (LIIs) publications in three steps: Accredited Public Archival (APA), Comment-Open Publication (COP) and peer reviewed Publication (PRP). This raises some ethical and legal issues on privacy and intellectual property which cannot be ignored. We would like to foster dialogue and discussion as the unique means to create an interactive framework among research communities, LIIs and users.

Peer review lies at the core of current scientific research. It is composed of a set of social norms, practices and processes that connect the abstract scientific method with the society of people that apply the method. As a social construct, peer review should be understood by building theory-informed models and comparing them with data collection. Both these activities are evolving in the era of automated computation and communication: new modeling tools and large bodies of data become available to the interested scholar. In this paper, starting from abstract principles, we develop and present a model of the peer review process. We also propose a working implementation of a subset of the general model, developed with Jason, a framework that implements the Belief-Desire-Intention (BDI) model for multi agent systems. After running a set of simulations, varying the initial distribution of reviewer skill, we compare the aggregates that our simplified model produces with recent findings, showing how for some parameter choice the model can generate data in qualitative agreement with measures.

It is common that students peer-review other students' writing, and these reviews are useful information to instructors, both on the particulars of the essay being reviewed, the feedback provided and the overall progress of the class. This paper describes a novel approach to summarising feedback in academic essay writing. We present a summarisation method for identifying and extracting representative opinion sentences from each feedback. Sentiment score-based techniques are employed and SentiWordNet is used as a linguistic lexical resource for sentiment summarisation. We evaluate our approach with the reviews written by a group of 50 engineering students.

In this paper, we use supervised machine learning to automatically identify the problem localization of peer-review feedback Using five features extracted via Natural Language Processing techniques, the learned model significantly outperforms a standard baseline Our work suggests that it is feasible for future tutoring systems to generate assessments regarding the use of localization in student peer reviews.

This paper addresses how the peer review system (FD Commons on WEB) was used as an analytical tool to identify key principles and criteria for assessing teaching and learning in higher education. The focus of this study was to understand various reviewers' viewpoints of educational events during “lesson study” by examining type of knowledge shared by reviewers. We present the overview of our Peer Review Process Project and report on five pilot studies conducted during spring term 2009. The successive trials revealed the following three points: (1) the lecture summary content differed between reviewers depending on specialty and teaching experience, (2) reviewers wrote more comments concerning lecture quality than basic teaching skills, (3) as keywords for self-reflection, lecturers used content information rather than pedagogical points.

There are many ways to model software development processes. This paper reports a feature analysis of four process modeling techniques using criteria specified by a software development organization. The evaluation used a single process, peer review, modeled using all four techniques. Performing the modeling activity highlighted the usefulness of the modeling activity and the usefulness of metamodels in structuring processes.

In the context of Intelligent Tutoring Systems, there is a potential for adapting either content or its sequence to student as to enhance the learning experience. Recent theories propose the use of team-working environments to improve even further this experience. In this paper an effective matching algorithm is presented in the context of peer reviewing applied to an educational setting. The problem is formulated as an optimization problem to search a solution that satisfies a set of given criteria modeled as "profiles". These profiles represent regions of the solution space to be either favored or avoided when searching for a solution. The proposed technique was deployed in a first semester computer engineering course and proved to be both effective and well received by the students.

In this survey talk we discuss several problems related to peer to peer networks. A host of issues arises in the context of peer to peer networks, including efficiency issues, censorship issues, anonymity issues, etc. While many of these problems have been studied in the past, the file swapping application has taken over the Internet, given these problems renewed impetus. I will discuss papers co-authored with J. Saia, E. Cohen, H. Kaplan, R. Berman, A. Ta-Sham, and others.

This paper focuses on peer review as a quality management technique used in open-source software (OSS) development and the similarities and differences with those of traditional development. The organizational commitment of OSS developers to quality is also explored. A comprehensive web-based questionnaire was completed by OSS and traditional developers. It was found that peer review is generally considered very useful for detecting both defects and flaws in code, as well as being important in contributing to the quality of the software. It is suggested that OSS developers commit to quality through internalization - adopting the ideas as their own, as the majority of the developers indicated that they would perform peer reviews without management direction. Encouragement to perform peer reviews and an organizational culture of peer review make it more likely for the developers to perform peer review under their own initiative, but neither are essential.

Despite the fact the hypermedia has long been discussed as a suitable format for academic research publications, the academic community has made little progress toward realizing hypermedia publishing. In this paper, we provide an initial framework for considering the problems facing any system designed for publishing academic hypertexts, and propose some first steps our community can take toward realizing this goal.