Back to Search
Start Over
Towards automatic annotation of collaborative problem‐solving skills in technology‐enhanced environments.
- Source :
-
Journal of Computer Assisted Learning . Oct2022, Vol. 38 Issue 5, p1434-1447. 14p. - Publication Year :
- 2022
-
Abstract
- Background: Collaborative problem solving (CPS) is important for success in the 21st century, especially for teamwork and communication in technology‐enhanced environments. Measurement of CPS skills has emerged as an essential aspect in educational assessment. Modern research in CPS relies on theory‐driven measurements that are usually carried out as manual annotations over recorded logs of collaborative activities. However, manual annotation has limited scalability and is not conductive towards CPS assessments at scale. Objective: We explore possibilities for automated annotation of actions in collaborative‐teams, especially chat messages. We evaluate two approaches that employ machine learning for automated classification of CPS events. Method: Data were collected from engineering, physics and electronics students' participation in a simulation‐based task on electronics concepts, in which participants communicated via text‐chat messages. All task activities were logged and time stamped. Data have been manually classified for the CPS skills, using an ontology that includes both social and cognitive dimensions. In this article, we describe computational linguistic methods for automatically classifying the CPS skills from logged data, with a view towards automating CPS assessments. Results: We applied two machine learning methods to our data. A Naïve Bayes classifier has been previously used in CPS research, but it is only moderately successful on our data. We also present a k‐nearest‐neighbours (kNN) classifier that uses distributional semantic models for measuring text similarity. This classifier shows strong agreement between automated and human annotations. The study also demonstrates that automatic spelling correction and slang normalization of chat texts are useful for accurate automated annotation. Implications: Our results suggest that a kNN classifier can be very effective for accurate annotation of CPS events. It achieves reasonably strong results even when trained on only half of the available data. This shows a promise towards reduction of manual data annotation for CPS measurement. Lay Description: What is already known about this topic: Collaborative problem solving (CPS) is important for success in the 21st century, especially when teamwork and communication occur in technology‐enhanced environments.Measurement of CPS skills requires classification and annotation of team interactions, in particular their communications, such as chat messages. What this paper adds: We explore possibilities for automated classification of CPS activities using machine learning approaches. Implications for practice: One of the approaches we explored is particularly effective and can help reduce the amount of manual annotations needed for CPS measurements. [ABSTRACT FROM AUTHOR]
Details
- Language :
- English
- ISSN :
- 02664909
- Volume :
- 38
- Issue :
- 5
- Database :
- Academic Search Index
- Journal :
- Journal of Computer Assisted Learning
- Publication Type :
- Academic Journal
- Accession number :
- 158963683
- Full Text :
- https://doi.org/10.1111/jcal.12689