Search

A new advancement in test automation is the use of natural language processing (NLP) to generate test cases (or test scripts) from natural language text. NLP is innovative in this context and promises of reducing test cases creation time and simplifying understanding for "non‐developer" software testers as well. Recently, many vendors have launched on the market many proposals of NLP‐based tools and testing frameworks but their superiority has never been empirically validated. This paper investigates the adoption of NLP‐based test automation in the web context with a series of case studies conducted to compare the costs of the NLP testing approach—measured in terms of test cases development and test cases evolution—with respect to more consolidated approaches, that is, programmable (or script‐based) testing and capture&replay testing. The results of our study show that NLP‐based test automation appears to be competitive for small‐ to medium‐sized test suites such as those considered in our empirical study. It minimizes the total cumulative cost (development and evolution) and does not require software testers with programming skills. [ABSTRACT FROM AUTHOR]

The amount of grey literature and 'softer' intelligence from social media or websites is vast. Given the long lead‐times of producing high‐quality peer‐reviewed health information, this is causing a demand for new ways to provide prompt input for secondary research. To our knowledge, this is the first review of automated data extraction methods or tools for health‐related grey literature and soft data, with a focus on (semi)automating horizon scans, health technology assessments (HTA), evidence maps, or other literature reviews. We searched six databases to cover both health‐ and computer‐science literature. After deduplication, 10% of the search results were screened by two reviewers, the remainder was single‐screened up to an estimated 95% sensitivity; screening was stopped early after screening an additional 1000 results with no new includes. All full texts were retrieved, screened, and extracted by a single reviewer and 10% were checked in duplicate. We included 84 papers covering automation for health‐related social media, internet fora, news, patents, government agencies and charities, or trial registers. From each paper, we extracted data about important functionalities for users of the tool or method; information about the level of support and reliability; and about practical challenges and research gaps. Poor availability of code, data, and usable tools leads to low transparency regarding performance and duplication of work. Financial implications, scalability, integration into downstream workflows, and meaningful evaluations should be carefully planned before starting to develop a tool, given the vast amounts of data and opportunities those tools offer to expedite research. [ABSTRACT FROM AUTHOR]

Background: Preoperative risky alcohol use is one of the most common surgical risk factors. Accurate and early identification of risky alcohol use could enhance surgical safety. Artificial Intelligence‐based approaches, such as natural language processing (NLP), provide an innovative method to identify alcohol‐related risks from patients' electronic health records (EHR) before surgery. Methods: Clinical notes (n = 53,629) from pre‐operative patients in a tertiary care facility were analyzed for evidence of risky alcohol use and alcohol use disorder. One hundred of these records were reviewed by experts and labeled for comparison. A rule‐based NLP model was built, and we assessed the clinical notes for the entire population. Additionally, we assessed each record for the presence or absence of alcohol‐related International Classification of Diseases (ICD) diagnosis codes as an additional comparator. Results: NLP correctly identified 87% of the human‐labeled patients classified with risky alcohol use. In contrast, diagnosis codes alone correctly identified only 29% of these patients. In terms of specificity, NLP correctly identified 84% of the non‐risky cohort, while diagnosis codes correctly identified 90% of this cohort. In the analysis of the full dataset, the NLP‐based approach identified three times more patients with risky alcohol use than ICD codes. Conclusions: NLP, an artificial intelligence‐based approach, efficiently and accurately identifies alcohol‐related risk in patients' EHRs. This approach could supplement other alcohol screening tools to identify patients in need of intervention, treatment, and/or postoperative withdrawal prophylaxis. Alcohol‐related ICD diagnosis had limited utility relative to NLP, which extracts richer information within clinical notes to classify patients. [ABSTRACT FROM AUTHOR]

There is a growing interest in the appropriate evaluation of penicillin adverse drug reaction (ADR) labels. We have developed machine learning models for classifying penicillin ADR labels using free‐text reaction descriptions, and here report external and practical validation. The models performed comparably with expert criteria for the categorisation of allergy or intolerance and identification of high‐risk allergies. These models have practical applications in detecting individuals suitable for penicillin ADR evaluation. Implementation studies are required. [ABSTRACT FROM AUTHOR]

Background: Teachers often rely on the use of open‐ended questions to assess students' conceptual understanding of assigned content. Particularly in the context of mathematics; teachers use these types of questions to gain insight into the processes and strategies adopted by students in solving mathematical problems beyond what is possible through more close‐ended problem types. While these types of problems are valuable to teachers, the variation in student responses to these questions makes it difficult, and time‐consuming, to evaluate and provide directed feedback. It is a well‐studied concept that feedback, both in terms of a numeric score but more importantly in the form of teacher‐authored comments, can help guide students as to how to improve, leading to increased learning. It is for this reason that teachers need better support not only for assessing students' work but also in providing meaningful and directed feedback to students. Objectives: In this paper, we seek to develop, evaluate, and examine machine learning models that support automated open response assessment and feedback. Methods: We build upon the prior research in the automatic assessment of student responses to open‐ended problems and introduce a novel approach that leverages student log data combined with machine learning and natural language processing methods. Utilizing sentence‐level semantic representations of student responses to open‐ended questions, we propose a collaborative filtering‐based approach to both predict student scores as well as recommend appropriate feedback messages for teachers to send to their students. Results and Conclusion: We find that our method outperforms previously published benchmarks across three different metrics for the task of predicting student performance. Through an error analysis, we identify several areas where future works may be able to improve upon our approach. Lay Description: What is already known about this topic: Open‐ended questions are used by teachers in the domain of mathematics to assess their students understanding but automated support for these types of questions are limited in online learning platforms. Recent advancements in areas of machine learning and natural language processing have led to promising results in a range of domains and applications. What this paper adds: Emulating how teachers identify similar student answers can be used to build tools that support teachers in assessing and providing feedback to student open‐ended work. Implications for practice: Developing better automated supports can increase the amount of direct feedback students receive to guide their learning. [ABSTRACT FROM AUTHOR]

Background: Formative assessments are needed to enable monitoring how student knowledge develops throughout a unit. Constructed response items which require learners to formulate their own free‐text responses are well suited for testing their active knowledge. However, assessing such constructed responses in an automated fashion is a complex task and requires the application of natural language processing methodology. In this article, we implement and evaluate multiple machine learning models for coding energy knowledge in free‐text responses of German K‐12 students to items in formative science assessments which were conducted during synchronous online learning sessions. Dataset: The dataset we collected for this purpose consists of German constructed responses from 38 different items dealing with aspects of energy such as manifestation and transformation. The units and items were implemented with the help of project‐based pedagogy and evidence‐centered design, and the responses were coded for seven core ideas concerning the manifestation and transformation of energy. The data was collected from students in seventh, eighth and ninth grade. Methodology: We train various transformer‐ and feature‐based models and compare their ability to recognize the respective ideas in students' writing. Moreover, as domain knowledge and its development can be formally modeled through knowledge networks, we evaluate how well the detection of the ideas within responses translated into accurate co‐occurrence‐based knowledge networks. Finally, in terms of the descriptive accuracy of our models, we inspect what features played a role for which prediction outcome and if the models pick up on undesired shortcuts. In addition to this, we analyze how much the models match human coders in what evidence within responses they consider important for their coding decisions. Results: A model based on a modified GBERT‐large can achieve the overall most promising results, although descriptive accuracy varies much more than predictive accuracy for the different ideas assessed. For reasons of comparability, we also evaluate the same machine learning architecture using the SciEntsBank 3‐Way benchmark with an English RoBERTa‐large model, where it achieves state‐of‐the‐art results in two out of three evaluation categories. Lay Description: What is already known about this topic?: Formative assessments are needed to test and monitor the development of learners' knowledge throughout a unit to provide them with appropriate automated feedback.Constructed response items which require learners to formulate their own free‐text responses are well suited for testing their active knowledge.Assessing constructed responses in an automated fashion is a widely researched topic, but the problem is far from solved and most of the work focuses on predicting holistic scores or grades.To allow for a more fine‐grained and analytic assessment of learners' knowledge, systems which go beyond predicting simple grades are required.To guarantee that models are stable and make their predictions for the correct reasons, methods for explaining the models are required. What this papers adds?: A core topic in physics education is the concept of energy.We implement and evaluate multiple systems based on natural language processing technology for assessing learners' conceptual knowledge about energy physics using transformer language models as well as feature‐based approaches.The systems assess students' knowledge about various forms of energy, indicators for the same and the transformation of energy from form into another.As our systems are based on machine learning methodology, we introduce a novel German short answer dataset for training them to detect the respective knowledge elements within students' free‐text responses.We evaluate the performance of these systems using this dataset as well as the well‐established SciEntsBand‐3‐Way dataset and manage to achieve, to our best knowledge, new state‐of‐the‐art results for the latter.Moreover, we apply methodology for explaining model predictions to assess whether predictions are carried out for the correct reasons. Implications for practice and/or policy: It is indeed possible to assess constructed responses for the demonstrated knowledge about energy physics in an analytic fashion using natural language processing.Transformer language models can outperform more specialized feature‐based approaches for this task in terms of predictive and descriptive accuracy.Co‐occurrences of different concepts within the same responses can lead models to learn undesired shortcuts which make them unstable. [ABSTRACT FROM AUTHOR]

Background: Assignments that involve writing based on several texts are challenging to many learners. Formative feedback supporting learners in these tasks should be informed by the characteristics of evolving written product and by the characteristics of learning processes learners enacted while developing the product. However, formative feedback in writing tasks based on multiple texts has almost exclusively focused on essay product and rarely included SRL processes. Objectives: We explored the viability of using product and process features to develop machine learning classifiers that identify low‐ and high‐performing essays in a multi‐text writing task. Methods: We examined learning processes and essay submissions of 163 graduate students working on an authentic multi‐text writing assignment. We utilised learners' trace data to obtain process features and state‐of‐the‐art natural language processing methods to obtain product features for our classifiers. Results and Conclusions: Of four popular classifiers examined in this study, Random Forest achieved the best performance (accuracy = 0.80 and recall = 0.77). The analysis of important features identified in the Random Forest classification model revealed one product (coverage of reading topics) and three process (elaboration/organisation, re‐reading and planning) features as important predictors of writing quality. Major Takeaways: The classifier can be used as a part of a future automated writing evaluation system that will support at scale formative assessment in writing tasks based on multiple texts in different courses. Based on important predictors of essay performance, a guidance can be tailored to learners at the outset of a multi‐text writing task to help them do well in the task. Lay Description: What is already known about this topic?: Both product and process features should be used to inform formative feedback on writing.Providing product‐ and process‐oriented feedback to learners is challenging.Automatic writing evaluation systems have mainly relied upon product features.Automated analysis of learners' trace data and their essay drafts is a promising venue. What this paper adds?: An accurate machine learning classifier that identifies low‐ and high‐scoring essays.The classifier utilized both product and process features.We obtained process features from learners' trace data in digital learning environment.We computed product features using state‐of‐the‐art text analytical methods. Implications for practice and/or policy: The classifier can be used as a part of a future automated writing evaluation system.We revealed learning processes and essay characteristics that influence performance.Based on important predictors of performance, formative feedback can be given to learners. [ABSTRACT FROM AUTHOR]

Background: In the context of large‐scale educational assessments, the effort required to code open‐ended text responses is considerably more expensive and time‐consuming than the evaluation of multiple‐choice responses because it requires trained personnel and long manual coding sessions. Aim: Our semi‐supervised coding method eco (exploring coding assistant) dynamically supports human raters by automatically coding a subset of the responses. Method: We map normalized response texts into a semantic space and cluster response vectors based on their semantic similarity. Assuming that similar codes represent semantically similar responses, we propagate codes to responses in optimally homogeneous clusters. Cluster homogeneity is assessed by strategically querying informative responses and presenting them to a human rater. Following each manual coding, the method estimates the code distribution respecting a certainty interval and assumes a homogeneous distribution if certainty exceeds a predefined threshold. If a cluster is determined to certainly comprise homogeneous responses, all remaining responses are coded accordingly automatically. We evaluated the method in a simulation using different data sets. Results: With an average miscoding of about 3%, the method reduced the manual coding effort by an average of about 52%. Conclusion: Combining the advantages of automatic and manual coding produces considerable coding accuracy and reduces the required manual effort. Lay Description: We present a new (semi‐)automatic method for coding text responses.The method supports the human coders in their work and acts as an assistant. The exploring coding assistant eco groups all responses in a first unsupervised step and explore the clusters by continuously selecting responses for manual coding to gather supervised information.Based on the coding information, eco automatically codes similar responses and reduces the manual coding effort. [ABSTRACT FROM AUTHOR]

Main path analysis is a popular method for extracting the scientific backbone from the citation network of a research domain. Existing approaches ignored the semantic relationships between the citing and cited publications, resulting in several adverse issues, in terms of coherence of main paths and coverage of significant studies. This paper advocated the semantic main path network analysis approach to alleviate these issues based on citation function analysis. A wide variety of SciBERT‐based deep learning models were designed for identifying citation functions. Semantic citation networks were built by either including important citations, for example, extension, motivation, usage and similarity, or excluding incidental citations like background and future work. Semantic main path network was built by merging the top‐K main paths extracted from various time slices of semantic citation network. In addition, a three‐way framework was proposed for the quantitative evaluation of main path analysis results. Both qualitative and quantitative analysis on three research areas of computational linguistics demonstrated that, compared to semantics‐agnostic counterparts, different types of semantic main path networks provide complementary views of scientific knowledge flows. Combining them together, we obtained a more precise and comprehensive picture of domain evolution and uncover more coherent development pathways between scientific ideas. [ABSTRACT FROM AUTHOR]

The article provides information on a transformer-based clustering model named ClusterBot developed to generate a curriculum map. Topics discussed include the potential of generative artificial intelligence (Gen-AI) to revolutionise medical education, how the accuracy of ClusterBot in identifying relevant text was evaluated, and lessons learned from the human-AI hybrid approach.

Combining computational technologies and humanities is an ongoing effort aimed at making resources such as texts, images, audio, video, and other artifacts digitally available, searchable, and analyzable. In recent years, deep neural networks (DNN) dominate the field of automatic text analysis and natural language processing (NLP), in some cases presenting a super‐human performance. DNNs are the state‐of‐the‐art machine learning algorithms solving many NLP tasks that are relevant for Digital Humanities (DH) research, such as spell checking, language detection, entity extraction, author detection, question answering, and other tasks. These supervised algorithms learn patterns from a large number of "right" and "wrong" examples and apply them to new examples. However, using DNNs for analyzing the text resources in DH research presents two main challenges: (un)availability of training data and a need for domain adaptation. This paper explores these challenges by analyzing multiple use‐cases of DH studies in recent literature and their possible solutions and lays out a practical decision model for DH experts for when and how to choose the appropriate deep learning approaches for their research. Moreover, in this paper, we aim to raise awareness of the benefits of utilizing deep learning models in the DH community. [ABSTRACT FROM AUTHOR]

Commonly used machine learning applications seem to relate to big data. This article provides a gentle review of machine learning and shows why machine learning can be applied to small data too. An example of applying machine learning to screen irregularity reports is presented. In the example, the support vector machine and multinomial naïve Bayes methods were used and compared. The performance of machine learning was compared to human experts in terms of flagging records to be excluded from equating. The application of machine learning seemed to be successful, although the data only consisted of a couple of thousand records. Recommendations in using machine learning are provided. [ABSTRACT FROM AUTHOR]

Automated information extraction might be able to assist with the collection of stroke key performance indicators (KPI). The feasibility of using natural language processing for classification‐based KPI and datetime field extraction was assessed. Using free‐text discharge summaries, random forest models achieved high levels of performance in classification tasks (area under the receiver operator curve 0.95–1.00). The datetime field extraction method was successful in 29 of 43 (67.4%) cases. Further studies are indicated. [ABSTRACT FROM AUTHOR]

Objective: Online forums allow people to semi‐anonymously discuss their struggles, often leading to greater honesty. This characteristic makes forums valuable for identifying users in need of immediate help from mental health professionals. Because it would be impractical to manually review every post on a forum to identify users in need of urgent help, there may be value to developing algorithms for automatically detecting posts reflecting a heightened risk of imminent plans to engage in disordered behaviors. Method: Five natural language processing techniques (tools to perform computational text analysis) were used on a data set of 4,812 posts obtained from six eating disorder‐related subreddits. Two licensed clinical psychologists labeled 53 of these posts, deciding whether or not the content of the post indicated that its author needed immediate professional help. The remaining 4,759 posts were unlabeled. Results: Each of the five techniques ranked the 50 posts most likely to be intervention‐worthy (the "top‐50"). The two most accurate detection techniques had an error rate of 4% for their respective top‐50. Discussion: This article demonstrates the feasibility of automatically detecting—with only a few dozen labeled examples—the posts of individuals in need of immediate mental health support for an eating disorder. [ABSTRACT FROM AUTHOR]

The authors address the problem of unsupervised ensemble ranking. Traditional approaches either combine multiple ranking criteria into a unified representation to obtain an overall ranking score or to utilize certain rank fusion or aggregation techniques to combine the ranking results. Beyond the aforementioned “combine-then-rank” and “rank-then-combine” approaches, the authors propose a novel “rank-learn-combine” ranking framework, called Interactive Ranking (iRANK), which allows two base rankers to “teach” each other before combination during the ranking process by providing their own ranking results as feedback to the others to boost the ranking performance. This mutual ranking refinement process continues until the two base rankers cannot learn from each other any more. The overall performance is improved by the enhancement of the base rankers through the mutual learning mechanism. The authors further design two ranking refinement strategies to efficiently and effectively use the feedback based on reasonable assumptions and rational analysis. Although iRANK is applicable to many applications, as a case study, they apply this framework to the sentence ranking problem in query-focused summarization and evaluate its effectiveness on the DUC 2005 and 2006 data sets. The results are encouraging with consistent and promising improvements. [ABSTRACT FROM AUTHOR]

Authorship attribution supported by statistical or computational methods has a long history starting from the 19th century and is marked by the seminal study of Mosteller and Wallace (1964) on the authorship of the disputed “Federalist Papers.” During the last decade, this scientific field has been developed substantially, taking advantage of research advances in areas such as machine learning, information retrieval, and natural language processing. The plethora of available electronic texts (e.g., e-mail messages, online forum messages, blogs, source code, etc.) indicates a wide variety of applications of this technology, provided it is able to handle short and noisy text from multiple candidate authors. In this article, a survey of recent advances of the automated approaches to attributing authorship is presented, examining their characteristics for both text representation and text classification. The focus of this survey is on computational requirements and settings rather than on linguistic or literary issues. We also discuss evaluation methodologies and criteria for authorship attribution studies and list open questions that will attract future work in this area. [ABSTRACT FROM AUTHOR]