Exploring latent states of problem‐solving competence using hidden Markov model on process data.

The response process of problem‐solving items contains rich information about respondents' behaviours and cognitive process in the digital tasks, while the information extraction is a big challenge. The aim of the study is to use a data‐driven approach to explore the latent states and state transitions underlying problem‐solving process to reflect test‐takers' behavioural patterns, and to investigate how these states and state transitions could be associated with test‐takers' performance. We employed the Hidden Markov Modelling approach to identify test takers' hidden states during the problem‐solving process and compared the frequency of states and/or state transitions between different performance groups. We conducted comparable studies in two problem‐solving items with a focus on the US sample that was collected in PIAAC 2012, and examined the correlation between those frequencies from two items. Latent states and transitions between them underlying the problem‐solving process were identified and found significantly different by performance groups. The groups with correct responses in both items were found more engaged in tasks and more often to use efficient tools to solve problems, while the group with incorrect responses was found more likely to use shorter action sequences and exhibit hesitative behaviours. Consistent behavioural patterns were identified across items. This study demonstrates the value of data‐driven based HMM approach to better understand respondents' behavioural patterns and cognitive transmissions underneath the observable action sequences in complex problem‐solving tasks. Lay Description: What is already known about this topic?: Process data of interactive problem‐solving tasks contains rich information about test‐takers.To extract underlying information from process data is a big challenge. ​ What this paper adds?: The hidden Markov model helps identify latent states underlying problem‐solving process.Different performance groups displayed different behaviour characteristics.A consistent behavioural pattern was observed across problem‐solving items. ​ Implications for practitioners: The use of HMM holds promise in better visualize and understand behaviour patterns in response process.HMM is potential in understanding cognitive transmissions underneath the observable action sequences. ​ [ABSTRACT FROM AUTHOR]