Search

This study explores the challenge of sentence-level AI-generated text detection within human-AI collaborative hybrid texts. Existing studies of AI-generated text detection for hybrid texts often rely on synthetic datasets. These typically involve hybrid texts with a limited number of boundaries. We contend that studies of detecting AI-generated content within hybrid texts should cover different types of hybrid texts generated in realistic settings to better inform real-world applications. Therefore, our study utilizes the CoAuthor dataset, which includes diverse, realistic hybrid texts generated through the collaboration between human writers and an intelligent writing system in multi-turn interactions. We adopt a two-step, segmentation-based pipeline: (i) detect segments within a given hybrid text where each segment contains sentences of consistent authorship, and (ii) classify the authorship of each identified segment. Our empirical findings highlight (1) detecting AI-generated sentences in hybrid texts is overall a challenging task because (1.1) human writers' selecting and even editing AI-generated sentences based on personal preferences adds difficulty in identifying the authorship of segments; (1.2) the frequent change of authorship between neighboring sentences within the hybrid text creates difficulties for segment detectors in identifying authorship-consistent segments; (1.3) the short length of text segments within hybrid texts provides limited stylistic cues for reliable authorship determination; (2) before embarking on the detection process, it is beneficial to assess the average length of segments within the hybrid text. This assessment aids in deciding whether (2.1) to employ a text segmentation-based strategy for hybrid texts with longer segments, or (2.2) to adopt a direct sentence-by-sentence classification strategy for those with shorter segments., Comment: Camera-Ready version of our IJCAI 2024 accepted paper (Special Track: AI and Social Good)

Multimodal Learning Analytics (MMLA) integrates novel sensing technologies and artificial intelligence algorithms, providing opportunities to enhance student reflection during complex, collaborative learning experiences. Although recent advancements in MMLA have shown its capability to generate insights into diverse learning behaviours across various learning settings, little research has been conducted to evaluate these systems in authentic learning contexts, particularly regarding students' perceived fairness, accountability, transparency, and ethics (FATE). Understanding these perceptions is essential to using MMLA effectively without introducing ethical complications or negatively affecting how students learn. This study aimed to address this gap by assessing the FATE of MMLA in an authentic, collaborative learning context. We conducted semi-structured interviews with 14 undergraduate students who used MMLA visualisations for post-activity reflection. The findings highlighted the significance of accurate and comprehensive data representation to ensure visualisation fairness, the need for different levels of data access to foster accountability, the imperative of measuring and cultivating transparency with students, and the necessity of transforming informed consent from dichotomous to continuous and measurable scales. While students value the benefits of MMLA, they also emphasise the importance of ethical considerations, highlighting a pressing need for the LA and MMLA community to investigate and address FATE issues actively., Comment: 16 pages, 1 figure

Automatic Essay Scoring (AES) is a well-established educational pursuit that employs machine learning to evaluate student-authored essays. While much effort has been made in this area, current research primarily focuses on either (i) boosting the predictive accuracy of an AES model for a specific prompt (i.e., developing prompt-specific models), which often heavily relies on the use of the labeled data from the same target prompt; or (ii) assessing the applicability of AES models developed on non-target prompts to the intended target prompt (i.e., developing the AES models in a cross-prompt setting). Given the inherent bias in machine learning and its potential impact on marginalized groups, it is imperative to investigate whether such bias exists in current AES methods and, if identified, how it intervenes with an AES model's accuracy and generalizability. Thus, our study aimed to uncover the intricate relationship between an AES model's accuracy, fairness, and generalizability, contributing practical insights for developing effective AES models in real-world education. To this end, we meticulously selected nine prominent AES methods and evaluated their performance using seven metrics on an open-sourced dataset, which contains over 25,000 essays and various demographic information about students such as gender, English language learner status, and economic status. Through extensive evaluations, we demonstrated that: (1) prompt-specific models tend to outperform their cross-prompt counterparts in terms of predictive accuracy; (2) prompt-specific models frequently exhibit a greater bias towards students of different economic statuses compared to cross-prompt models; (3) in the pursuit of generalizability, traditional machine learning models coupled with carefully engineered features hold greater potential for achieving both high accuracy and fairness than complex neural network models.

Integrating generative AI (GAI) into higher education is crucial for preparing a future generation of GAI-literate students. Yet a thorough understanding of the global institutional adoption policy remains absent, with most of the prior studies focused on the Global North and the promises and challenges of GAI, lacking a theoretical lens. This study utilizes the Diffusion of Innovations Theory to examine GAI adoption strategies in higher education across 40 universities from six global regions. It explores the characteristics of GAI innovation, including compatibility, trialability, and observability, and analyses the communication channels and roles and responsibilities outlined in university policies and guidelines. The findings reveal a proactive approach by universities towards GAI integration, emphasizing academic integrity, teaching and learning enhancement, and equity. Despite a cautious yet optimistic stance, a comprehensive policy framework is needed to evaluate the impacts of GAI integration and establish effective communication strategies that foster broader stakeholder engagement. The study highlights the importance of clear roles and responsibilities among faculty, students, and administrators for successful GAI integration, supporting a collaborative model for navigating the complexities of GAI in education. This study contributes insights for policymakers in crafting detailed strategies for its integration.

Peer feedback has been widely used in computer-supported collaborative learning (CSCL) setting to improve students’ engagement with massive courses. Although the peer feedback process increases students’ self-regulatory practice, metacognition, and academic achievement, instructors need to go through large amounts of feedback text data which is much more time-consuming. To address this challenge, the present study proposes an automated content analysis approach to identify relevant categories in peer feedback based on traditional and sequence-based classifiers using TF-IDF and content-independent features. We use a data set from an extensive course (N = 231 students) in the setting of engineering higher education. In particular, a total of 2,444 peer feedback messages were analyzed. The CRF classification model based on the TF-IDF features achieved the best performance. The results illustrate that the ability to scale up the automatic analysis of peer feedback provides new opportunities for student-improved learning and improved teacher support in higher education at scale., Part of ISBN 9783031426810QC 20231009

A complex challenge for the society is to offer equal learning opportunities at various life stages and to support students, teachers, and institutions in their various tasks and roles related to learning and teaching. Learning analytics (LA) provides an opportunity to address these societal challenges. As the LA field matures, tool development is aimed at aiding informed human decision-making and combating inequalities. For example, detecting students at risk of dropping out or supporting self-regulated learning. The inception of LA was catalysed by an increasing amount of available data and what could be done with these data to improve learner support and teaching. Simultaneously, an increase in the computational power, machine learning methods, and tools at hand offer renewing affordances to analyse and visualise data both retrospectively and for predictive purposes. Employing LA as a solution also brings potential problems, such as unequal treatment, privacy concerns, and unethical practices. Through selected example cases, this chapter presents and addresses these potentials and risks.

Educational technology innovations leveraging large language models (LLMs) have shown the potential to automate the laborious process of generating and analysing textual content. While various innovations have been developed to automate a range of educational tasks (e.g., question generation, feedback provision, and essay grading), there are concerns regarding the practicality and ethicality of these innovations. Such concerns may hinder future research and the adoption of LLMs-based innovations in authentic educational contexts. To address this, we conducted a systematic scoping review of 118 peer-reviewed papers published since 2017 to pinpoint the current state of research on using LLMs to automate and support educational tasks. The findings revealed 53 use cases for LLMs in automating education tasks, categorised into nine main categories: profiling/labelling, detection, grading, teaching support, prediction, knowledge representation, feedback, content generation, and recommendation. Additionally, we also identified several practical and ethical challenges, including low technological readiness, lack of replicability and transparency, and insufficient privacy and beneficence considerations. The findings were summarised into three recommendations for future studies, including updating existing innovations with state-of-the-art models (e.g., GPT-3/4), embracing the initiative of open-sourcing models/systems, and adopting a human-centred approach throughout the developmental process. As the intersection of AI and education is continuously evolving, the findings of this study can serve as an essential reference point for researchers, allowing them to leverage the strengths, learn from the limitations, and uncover potential research opportunities enabled by ChatGPT and other generative AI models.

Multimodal Learning Analytics (MMLA) innovations make use of rapidly evolving sensing and artificial intelligence algorithms to collect rich data about learning activities that unfold in physical learning spaces. The analysis of these data is opening exciting new avenues for both studying and supporting learning. Yet, practical and logistical challenges commonly appear while deploying MMLA innovations "in-the-wild". These can span from technical issues related to enhancing the learning space with sensing capabilities, to the increased complexity of teachers' tasks and informed consent. These practicalities have been rarely discussed. This paper addresses this gap by presenting a set of lessons learnt from a 2-year human-centred MMLA in-the-wild study conducted with 399 students and 17 educators. The lessons learnt were synthesised into topics related to i) technological/physical aspects of the deployment; ii) multimodal data and interfaces; iii) the design process; iv) participation, ethics and privacy; and v) the sustainability of the deployment., Comment: Submitted to TOCHI

Peer feedback has been widely used in computer-supported collaborative learning (CSCL) setting to improve students’ engagement with massive courses. Although the peer feedback process increases students’ self-regulatory practice, metacognition, and academic achievement, instructors need to go through large amounts of feedback text data which is much more time-consuming. To address this challenge, the present study proposes an automated content analysis approach to identify relevant categories in peer feedback based on traditional and sequence-based classifiers using TF-IDF and content-independent features. We use a data set from an extensive course (N = 231 students) in the setting of engineering higher education. In particular, a total of 2,444 peer feedback messages were analyzed. The CRF classification model based on the TF-IDF features achieved the best performance. The results illustrate that the ability to scale up the automatic analysis of peer feedback provides new opportunities for student-improved learning and improved teacher support in higher education at scale.

The rapid expansion of Learning Analytics (LA) and Artificial Intelligence in Education (AIED) offers new scalable, data-intensive systems but also raises concerns about data privacy and agency. Excluding stakeholders -- like students and teachers -- from the design process can potentially lead to mistrust and inadequately aligned tools. Despite a shift towards human-centred design in recent LA and AIED research, there remain gaps in our understanding of the importance of human control, safety, reliability, and trustworthiness in the design and implementation of these systems. We conducted a systematic literature review to explore these concerns and gaps. We analysed 108 papers to provide insights about i) the current state of human-centred LA/AIED research; ii) the extent to which educational stakeholders have contributed to the design process of human-centred LA/AIED systems; iii) the current balance between human control and computer automation of such systems; and iv) the extent to which safety, reliability and trustworthiness have been considered in the literature. Results indicate some consideration of human control in LA/AIED system design, but limited end-user involvement in actual design. Based on these findings, we recommend: 1) carefully balancing stakeholders' involvement in designing and deploying LA/AIED systems throughout all design phases, 2) actively involving target end-users, especially students, to delineate the balance between human control and automation, and 3) exploring safety, reliability, and trustworthiness as principles in future human-centred LA/AIED systems., Comment: 40 pages, 6 figures, 1 table in Appendix

Individual behavioral engagement is an important indicator of active learning in collaborative settings, encompassing multidimensional behaviors mediated through various interaction modes. Little existing work has explored the use of multimodal process data to understand individual behavioral engagement in face-to-face collaborative learning settings. In this study we bridge this gap, for the first time, introducing a heterogeneous tripartite network approach to analyze the interconnections among multimodal process data in collaborative learning. Students' behavioral engagement strategies are analyzed based on their interaction patterns with various spatial locations and verbal communication types using a heterogeneous tripartite network. The multimodal collaborative learning process data were collected from 15 teams of four students. We conducted stochastic blockmodeling on a projection of the heterogeneous tripartite network to cluster students into groups that shared similar spatial and oral engagement patterns. We found two distinct clusters of students, whose characteristic behavioural engagement strategies were identified by extracting interaction patterns that were statistically significant relative to a multinomial null model. The two identified clusters also exhibited a statistically significant difference regarding students' perceived collaboration satisfaction and teacher-assessed team performance level. This study advances collaboration analytics methodology and provides new insights into personalized support in collaborative learning.

Generative artificial intelligence (GenAI), exemplified by ChatGPT, Midjourney, and other state-of-the-art large language models and diffusion models, holds significant potential for transforming education and enhancing human productivity. While the prevalence of GenAI in education has motivated numerous research initiatives, integrating these technologies within the learning analytics (LA) cycle and their implications for practical interventions remain underexplored. This paper delves into the prospective opportunities and challenges GenAI poses for advancing LA. We present a concise overview of the current GenAI landscape and contextualise its potential roles within Clow's generic framework of the LA cycle. We posit that GenAI can play pivotal roles in analysing unstructured data, generating synthetic learner data, enriching multimodal learner interactions, advancing interactive and explanatory analytics, and facilitating personalisation and adaptive interventions. As the lines blur between learners and GenAI tools, a renewed understanding of learners is needed. Future research can delve deep into frameworks and methodologies that advocate for human-AI collaboration. The LA community can play a pivotal role in capturing data about human and AI contributions and exploring how they can collaborate most effectively. As LA advances, it is essential to consider the pedagogical implications and broader socioeconomic impact of GenAI for ensuring an inclusive future.

Generative artificial intelligence (GenAI) offers promising potential for advancing human-AI collaboration in qualitative research. However, existing works focused on conventional machine-learning and pattern-based AI systems, and little is known about how researchers interact with GenAI in qualitative research. This work delves into researchers' perceptions of their collaboration with GenAI, specifically ChatGPT. Through a user study involving ten qualitative researchers, we found ChatGPT to be a valuable collaborator for thematic analysis, enhancing coding efficiency, aiding initial data exploration, offering granular quantitative insights, and assisting comprehension for non-native speakers and non-experts. Yet, concerns about its trustworthiness and accuracy, reliability and consistency, limited contextual understanding, and broader acceptance within the research community persist. We contribute five actionable design recommendations to foster effective human-AI collaboration. These include incorporating transparent explanatory mechanisms, enhancing interface and integration capabilities, prioritising contextual understanding and customisation, embedding human-AI feedback loops and iterative functionality, and strengthening trust through validation mechanisms.

The recent large language models (LLMs), e.g., ChatGPT, have been able to generate human-like and fluent responses when provided with specific instructions. While admitting the convenience brought by technological advancement, educators also have concerns that students might leverage LLMs to complete their writing assignments and pass them off as their original work. Although many AI content detection studies have been conducted as a result of such concerns, most of these prior studies modeled AI content detection as a classification problem, assuming that a text is either entirely human-written or entirely AI-generated. In this study, we investigated AI content detection in a rarely explored yet realistic setting where the text to be detected is collaboratively written by human and generative LLMs (i.e., hybrid text). We first formalized the detection task as identifying the transition points between human-written content and AI-generated content from a given hybrid text (boundary detection). Then we proposed a two-step approach where we (1) separated AI-generated content from human-written content during the encoder training process; and (2) calculated the distances between every two adjacent prototypes and assumed that the boundaries exist between the two adjacent prototypes that have the furthest distance from each other. Through extensive experiments, we observed the following main findings: (1) the proposed approach consistently outperformed the baseline methods across different experiment settings; (2) the encoder training process can significantly boost the performance of the proposed approach; (3) when detecting boundaries for single-boundary hybrid essays, the proposed approach could be enhanced by adopting a relatively large prototype size, leading to a 22% improvement in the In-Domain evaluation and an 18% improvement in the Out-of-Domain evaluation., Comment: Accepted as an AAAI 2024 (Vancouver, Canada) full paper

Reference to teachers as agents of change has become commonplace in the education literature, including change toward more inclusive practice in response to the changing demographic of schooling. Yet, little is known about how teacher agency relates to (1) their understanding of, and commitment to any given change agenda and (2) the institutional and social structures through which they are able to access knowledge and resources within and beyond their schools. This study combined social and epistemic network analysis to examine teachers’ understanding of change and their sense of agency as they use their social networks to mobilise support for furthering change that matters to them. Our study is the first to apply this learning analytic approach in a real setting context. We used theories of teacher agency and inclusive pedagogy to interpret teachers’ social interactions in light of the extent to which they seek to make a difference toward greater inclusion. We collected data with an online log completed by teachers and other staff in two schools in Sweden over 6 months. The findings suggest that teachers understanding of change is embedded in their day-to-day activities such as student support, lesson planning, improvement of programs, and working conditions. Teachers tend to exercise agency toward inclusion when they seek to support student learning and well-being. When teachers act as agents of change, their social networks are bigger, more diverse and more collaborative than in situations in which they act as role implementers. We discuss substantive and methodological implications of these findings.

Predictors of student academic success do not always replicate well across different learning designs, subject areas, or educational institutions. This suggests that characteristics of a particular discipline and learning design have to be carefully considered when creating predictive models in order to scale up learning analytics. This study aimed to examine if and to what extent frequently used predictors of study success are portable across a homogenous set of courses. The research was conducted in an integrated blended problem-based curriculum with trace data (n = 2,385 students) from 50 different course offerings across four academic years. We applied the statistical method of single paper meta-analysis to combine correlations of several indicators with students' success. Total activity and the forum indicators exhibited the highest prediction intervals, where the former represented proxies of the overall engagement with online tasks, and the latter with online collaborative learning activities. Indicators of lecture reading (frequency of lecture view) showed statistically insignificant prediction intervals and, therefore, are less likely to be portable across course offerings. The findings show moderate amounts of variability both within iterations of the same course and across courses. The results suggest that the use of the meta-analytic statistical method for the examination of study success indicators across courses with similar learning design and subject area can offer valuable quantitative means for the identification of predictors that reasonably well replicate and consequently can be reliably portable in the future.

Massive Open Online Courses (MOOCs) have already shown a great potential to be used as an alternative model for teacher professional development (TPD). Not only do MOOCs offer relevant content and activities, but they also provide opportunities for strengthening the skills for self regulated learning of teachers as life-long learners. In this study, we focused on a unique subgroup of MOOC learners - retakers who take the same TPD MOOC multiple times. In the empirical study reported in this paper, we examined what learning strategies they choose when taking a TPD MOOC and the extent to which they are able to adapt their learning strategies to improve their performance in the subsequent attempts. By using learning analytic methods, we detected five learning strategies and eight major forms of strategy change adapted by MOOC retakers. We found that two strategy changes (from Content-Oriented Strategy to Intensive-Thorough Strategy or Balanced Strategy) with associated with significantly higher performance in the subsequent attempts compared to no change in strategy. Our findings have implications for MOOC instructors and providers about the ways they can support MOOC retakers. Our findings also suggest that MOOCs are not only valuable TPD resources for teachers, but they also have unique opportunities for strengthening the self-regulation skills of teacher learners as the "by-products of learning".

CRUE-CSIC (Acuerdos Transformativos 2021), Collaboration is considered as one of the main drivers of learning and it has been broadly studied across numerous contexts, including Massive Open Online Courses (MOOCs). The research on MOOCs has risen exponentially during the last years and there have been a number of works focused on studying collaboration. However, these previous studies have been restricted to the analysis of collaboration based on the forum and social interactions, without taking into account other possibilities such as the synchronicity in the interactions with the platform. Therefore, in this work we performed a case study with the goal of implementing a data-driven approach to detect and characterize collaboration in MOOCs. We applied an algorithm to detect synchronicity links based on their submission times to quizzes as an indicator of collaboration, and applied it to data from two large Coursera MOOCs. We found three different profiles of user accounts, that were grouped in couples and larger communities exhibiting different types of associations between user accounts. The characterization of these user accounts suggested that some of them might represent genuine online learning collaborative associations, but that in other cases dishonest behaviors such as free-riding or multiple account cheating might be present. These findings call for additional research on the study of the kind of collaborations that can emerge in online settings., Ministerio de Economía y Competitividad (MINECO), Depto. de Ingeniería de Software e Inteligencia Artificial (ISIA), Fac. de Informática, TRUE, pub

Reference to teachers as agents of change has become commonplace in the education literature, including change toward more inclusive practice in response to the changing demographic of schooling. Yet, little is known about how teacher agency relates to (1) their understanding of, and commitment to any given change agenda and (2) the institutional and social structures through which they are able to access knowledge and resources within and beyond their schools. This study combined social and epistemic network analysis to examine teachers’ understanding of change and their sense of agency as they use their social networks to mobilise support for furthering change that matters to them. Our study is the first to apply this learning analytic approach in a real setting context. We used theories of teacher agency and inclusive pedagogy to interpret teachers’ social interactions in light of the extent to which they seek to make a difference toward greater inclusion. We collected data with an online log completed by teachers and other staff in two schools in Sweden over 6 months. The findings suggest that teachers understanding of change is embedded in their day-to-day activities such as student support, lesson planning, improvement of programs, and working conditions. Teachers tend to exercise agency toward inclusion when they seek to support student learning and well-being. When teachers act as agents of change, their social networks are bigger, more diverse and more collaborative than in situations in which they act as role implementers. We discuss substantive and methodological implications of these findings.

Predictive modelling of academic success and retention has been a key research theme in Learning Analytics. While the initial work on predictive modelling was focused on the development of general predictive models, portable across different learning settings, later studies demonstrated the drawbacks of not considering the specificities of course design and disciplinary context. This study builds on the methods and findings of related earlier studies to further explore factors predictive of learners' academic success in blended learning. In doing so, it differentiates itself by (i) relying on a larger and homogeneous course sample (15 courses, 50 course offerings in total), and (ii) considering both internal and external conditions as factors affecting the learning process. We apply mixed effect linear regression models, to examine: i) to what extent indicators of students' online learning behaviour can explain the variability in the final grades, and ii) to what extent that variability is attributable to the course and students' internal conditions, not captured by the logged data. Having examined different types of behaviour indicators (e.g., indicators of the overall activity level, those indicative of regularity of study, etc), we found little difference, if any, in their predictive power. Our results further indicate that a low proportion of variance is explained by the behaviour-based indicators, while a significant portion of variability stems from the learners' internal conditions. Hence, when variability in external conditions is largely controlled for (the same institution, discipline, and nominal pedagogical model), students' internal state is the key predictor of their course performance., QC 20210802

In this article, we empirically validate Tinto's Student Integration model, in particular, the predictions the model makes regarding both students' academic outcomes and their dropout decisions. In doing so, we analyze three decades' worth of student enrollments at an Australian university and present a novel methodological approach using graph embedding techniques to capture both structural and neighborhood-based features of the co-enrollment network. In keeping with Tinto's model, we find that not only do these embedded representations of students' social network predict their final grade point average (GPA), but also are able to successfully classify students who dropout. Our results show that these embedded representations of a student's social network can achieve F1-scores of up to 0.79 when classifying dropout and explain up to 10% of the variance in student's final GPA. When controlling for a small set of covariates and variables common to the literature, this performance increases to 0.83 and 24%, respectively. Furthermore, the performance of these methods is robust to both changes in their parameterization and to corruption of the underlying social networks. Importantly, this implies that hyperparameters may be selected to reduce the computational demands of this method without loss of predictive power. The novelty of this method, and its ability to identify student dropout, merits further investigation to preemptively identify at-risk students.

For service implementations to be widely adopted, it is necessary for the expectations of the key stakeholders to be considered. Failure to do so may lead to services reflecting ideological gaps, which will inadvertently create dissatisfaction among its users. Learning analytics research has begun to recognise the importance of understanding the student perspective towards the services that could be potentially offered; however, student engagement remains low. Furthermore, there has been no attempt to explore whether students can be segmented into different groups based on their expectations towards learning analytics services. In doing so, it allows for a greater understanding of what is and is not expected from learning analytics services within a sample of students. The current exploratory work addresses this limitation by using the three-step approach to latent class analysis to understand whether student expectations of learning analytics services can clearly be segmented, using self-report data obtained from a sample of students at an Open University in the Netherlands. The findings show that student expectations regarding ethical and privacy elements of a learning analytics service are consistent across all groups; however, those expectations of service features are quite variable. These results are discussed in relation to previous work on student stakeholder perspectives, policy development, and the European General Data Protection Regulation (GDPR).

This paper aims to explore time management strategies followed by students in a flipped classroom through the analysis of trace data. Specifically, an exploratory study was conducted on the dataset collected in three consecutive offerings of an undergraduate computer engineering course (N = 1,134). Trace data about activities were initially coded for the timeliness of activity completion. Such data were then analysed using agglomerative hierarchical clustering based on Ward's algorithm, first order Markov chains, and inferential statistics to (a) detect time management tactics and strategies from students' learning activities and (b) analyse the effects of personalized analytics-based feedback on time management. The results indicate that meaningful and theoretically relevant time management patterns can be detected from trace data as manifestations of students' tactics and strategies. The study also showed that time management tactics had significant associations with academic performance and were associated with different interventions in personalized analytics-based feedback.

Learning design in a massive open online course (MOOC) intends to promote creativity, autonomy, and social networked learning, amongst other things. Students in a MOOC are required to self-regulate their learning to properly self-monitor their learning process and effectiveness of the adopted learning strategies. This paper presents the results of a study among 279 students enrolled in a MOOC that was enriched with a set of scaffolding interventions for social mirroring. The mirroring interventions supported social awareness and social embeddedness of learners. Associations between the use of the interventions and microlevel self-regulated learning processes were measured and analyzed. The extent to which those associations are affected by learner demographics and motivational characteristics was also investigated. Findings show that interventions that provide students, throughout the course, with learning updates and progress of peers are associated with the students' engagement with learning tasks and applying changes in strategies for completing those tasks. Social awareness scaffold influenced more students low in need for cognition, with a higher education degree, high in performance-approach orientation and low in grit, to engage with their learning tasks, while its effect on the change in learning strategies was higher with those early and towards the end of their careers and high in performance-approach strategy. The social comparison scaffold affected more students low in mastery goal orientation and high in grit to work on their learning tasks.

Studying online requires well-developed self-regulated learning skills to properly manage one's learning strategies. Learning analytics research has proposed novel methods for extracting theoretically meaningful learning strategies from trace data originating from formal learning settings (online, blended, or flipped classroom). Thus identified strategies proved to be associated with academic achievement. However, automated extraction of theoretically meaningful learning strategies from trace data in the context of massive open online courses (MOOCs) is still under-explored. Moreover, there is a lacuna in research on the relations between automatically detected strategies and the established psychological constructs. The paper reports on a study that (a) applied a state-of-the-art analytic method that combines process and sequence mining techniques to detect learning strategies from the trace data collected in a MOOC (N=1,397), and (b) explored associations of the detected strategies with academic performance and personality traits (Big Five). Four learning strategies detected with the adopted analytics method were shown to be theoretically interpretable as the wellknown approaches to learning. The results also revealed that the four detected learning strategies were predicted by conscientiousness, emotional instability, and agreeableness and were associated with academic performance. Implications for theoretical validity and practical application of analytics-detected learning strategies are also provided.

Generalizability of the value of methods based on learning analytics remains one of the big challenges in the field of learning analytics. One approach to testing generalizability of a method is to apply it consistently in different learning contexts. This study extends a previously published work by examining the generalizability of a learning analytics method proposed for detecting learning tactics and strategies from trace data. The method was applied to the datasets collected in three different course designs and delivery modalities, including flipped classroom, blended learning, and massive open online course. The proposed method combines process mining and sequence analysis. The detected learning strategies are explored in terms of their association with academic performance. The results indicate the applicability of the proposed method across different learning contexts. Moreover, the findings contribute to the understanding of the learning tactics and strategies identified in the trace data: learning tactics proved to be responsive to the course design, whereas learning strategies were found to be more sensitive to the delivery modalities than to the course design. These findings, well aligned with self-regulated learning theory, highlight the association of learning contexts with the choice of learning tactics and strategies.

This paper reports on the findings of a study that proposed a novel learning analytics methodology that combines three complimentary techniques - agglomerative hierarchical clustering, epistemic network analysis, and process mining. The methodology allows for identification and interpretation of self-regulated learning in terms of the use of learning strategies. The main advantage of the new technique over the existing ones is that it combines the time management and learning tactic dimensions of learning strategies, which are typically studied in isolation. The new technique allows for novel insights into learning strategies by studying the frequency of, strength of connections between, and ordering and time of execution of time management and learning tactics. The technique was validated in a study that was conducted on the trace data of first-year undergraduate students who were enrolled into two consecutive offerings (N-2017 = 250 and N-2018 = 232) of a course at an Australian university. The application of the proposed technique identified four strategy groups derived from three distinct time management tactics and five learning tactics. The tactics and strategies identified with the technique were correlated with academic performance and were interpreted according to the established theories and practices of self-regulated learning.

Learning in computer-mediated setting represents a complex, multidimensional process. This complexity calls for a comprehensive analytical approach that would allow for understanding of various dimensions of learner generated discourse and the structure of the underlying social interactions. Current research, however, primarily focuses on manual or, more recently, supervised methods for discourse analysis. Moreover, discourse and social structures are typically analyzed separately without the use of computational methods that can offer a holistic perspective. This paper proposes an approach that addresses these two challenges, first, by using an unsupervised machine learning approach to extract speech acts as representations of knowledge construction processes and finds transition probabilities between speech acts across different messages, and second, by integrating the use of discovered speech acts to explain the formation of social ties and predicting course outcomes. We extracted six categories of speech acts from messages exchanged in discussion forums of two MOOCs and each category corresponded to knowledge construction processes from well-established theoretical models. We further showed how measures derived from discourse analysis explained the ways how social ties were created that framed emerging social networks. Multiple regression models showed that the combined use of measures derived from discourse analysis and social ties predicted learning outcomes.

Samoregulisano učenje predstavlja formu učenja koja stavlja akcenat na samog učenika i pruža mu mogućnost da upravlja i kontinuirano reguliše svoj proces učenja. Poznato je da društveni kontekst i interakcije koje se ostvaruju tokom učenja imaju uticaj na akcije učenika i da se samoregulacija može posmatrati kao regulacija individue u okviru socijalnog okruženja. Međutim, postojeći sistemi za učenje na Vebu pružaju učenicima ograničene mogućnosti za samoregulaciju tokom učenja u neformalnim i informalnim kontekstima. Predmet ove doktorske disertacije je dizajniranje i projektovanje softverskog sistema koji je namenjen samoregulisanom učenju na Vebu. Cilj je da ovaj softverski sistem poseduje funkcionalnosti koje podržavaju procese samoregulisanog učenja, da omogućavaju uređivanje kohorti, kurseva, materijala i aktivnosti učenja, kao i da omogućavaju kolektivnu regulaciju između učenika. Prototip softverskog sistema, koji je razvijen na osnovu predloženog dizajna i arhitekture, evaluiran je u okviru MOOC kursa u trajanju od 9 nedelja. Dobijeni rezultati su pokazali da postoje izražene asocijacije između funkcionalnosti zasnovanim na socijalnim interakcijama i procesa samoregulisanog učenja koji je zasnovan na kolektivnoj regulaciji. Konkretno, pokazano je da postoje asocijacije između funkcionalnosti koje omogućavaju uvid u aktivnosti i progres drugih učenika sa procesima samoregulisanog učenja koji se odnose na izvršenje zadataka učenika. Takođe, pronađene su razlike u ovim asocijacijama kod učenika sa različitim demografskim i motivacionim karakteristikama., Self-regulated learning is a form of learning that puts emphasis on a learner and gives them the ability to manage and continually regulate their learning process. It is known that the social context and interactions that occur during learning can impact students' actions and that self-regulation can be viewed as a regulation of an individual within their social context. However, existing web learning systems provide students with limited functionalities to self-regulate their learning in informal and informal contexts. The topic of this doctoral dissertation is the design and architecture of a software system that enables self-regulated learning on the Web. The goal of this system is to enable a learner with functionalities that support self-regulated learning processes, to enable management of cohorts, courses, materials and learning activities, as well as to allow collective regulation among learners. A software prototype, developed based on the proposed design and architecture, was evaluated within the MOOC course for 9 weeks. The obtained results showed that there are significant associations between functionalities based on social interactions and the self-regulated learning process based on collective regulation. In particular, it has been shown that there are associations between functionalities that allow insight into the learning activities and progress of other students with self-regulated learning processes related to engagement in learning. Also, variations in these associations were found among students with different demographic and motivational characteristics.

Research in learning analytics proposed different computational techniques to detect learning tactics and strategies adopted by learners in digital environments through the analysis of students' trace data. While many promising insights have been produced, there has been much less understanding about how and to what extent different data analytic approaches influence results. This paper presents a comparison of three analytic approaches including process, sequence, and network approaches for detection of learning tactics and strategies. The analysis was performed on a dataset collected in a massive open online course on software programming. All three approaches produced four tactics and three strategy groups. The tactics detected by using the sequence analysis approach differed from those identified by the other two methods. The process and network analytic approaches had more than 66% of similarity in the detected tactics. Learning strategies detected by the three approaches proved to be highly similar.

The use of learning trace data together with various analytical methods has proven successful in detecting patterns in learning behaviour, identifying student profiles, and clustering learning resources. However, interpretation of the findings is often difficult and uncertain due to a lack of contextual data (e.g., data on student motivation, emotion or curriculum design). In this study we explored the integration of student selfreports about cognitive load and self-efficacy into the learning process and collection of relevant students' perceptions as learning traces. Our objective was to examine the association of traced measures of relevant learning constructs (cognitive load and self-efficacy) with i) indicators of the students' learning behaviour derived from trace data, and ii) the students' academic performance. The results indicated the presence of association between some indicators of students' engagement with learning activities and traced measures of cognitive load and self-efficacy. Correlational analysis demonstrated significant positive correlation between the students' course performance and traced measures of cognitive load and self-efficacy.

Flipped classroom (FC) is an active learning design requiring students to complete assigned pre-class learning activities in preparation for face-to-face sessions. Students' timely, regular, and productive engagement with the pre-class activities is considered critical for the success of the overall FC design, as these activities serve to prepare students for effective participation in face-to-face sessions. However, there is limited empirical evidence on the strength of association between students' regularity of engagement with the pre-class activities and their learning performance in a FC course. Hence, the current study uses learning trace data from three consecutive offerings of a FC course to examine students' regularity of pre-class learning activities and its association with the students' course performance. In particular, the study derives several indicators of regularity from the trace data, including indicators related to time management and those reflecting regularity in the pattern of engagement with pre-class learning activities. The association with course performance is examined by building predictive regression models with the defined indicators as features. To examine the relevance of incorporating the specificities of the instructional design in predictive models, we designed and compared two kinds of indicators: generic (i.e. course-design-agnostic) and course-design-specific indicators. The study identified several indicators of regularity of pre-class activities as significant predictors of course performance. It also demonstrated that predictive models with only generic indicators were able to explain only a small portion of the overall variability in the students' course performance, and were significantly outperformed by models that incorporated coursespecific indicators. Finally, the study findings point to the importance of assisting students in regulating their use of learning resources during class preparation activities in a FC.

There is little debate regarding the importance of student feedback for improving the learning process. However, there remain significant workload barriers for instructors that impede their capacity to provide timely and meaningful feedback. The increasing role technology is playing in the education space may provide novel solutions to this impediment. As students interact with the various learning technologies in their course of study, they create digital traces that can be captured and analysed. These digital traces form the new kind of data that are frequently used in learning analytics to develop actionable recommendations that can support student learning. This paper explores the use of such analytics to address the challenges impeding the capacity of instructors to provide personalised feedback at scale. The case study reported in the paper showed how the approach was associated with a positive impact on student perception of feedback quality and on academic achievement. The study was conducted with first year undergraduate engineering students enrolled in a computer systems course with a blended learning design across three consecutive years (N-2013 = 290, N-2014 = 316 and N-2015 = 415).

The success of the flipped classroom (FC) is effectively reliant on the level of student engagement with the preparatory activities prior to attending face-to-face teaching sessions. Information about the nature and level of student engagement with these activities can help instructors make informed decisions regarding how to best support student learning. Despite the comprehensive data collection enabled by the increasing presence of education technologies, few studies have used these data to investigate the range of strategies students employ in FC models. This study addresses this deficit by proposing an analytical approach that allows for exploring, first, the strategies students use to interact with online preparation activities; and, second, evolution of those strategies over the duration of a course delivered with a FC pedagogy. The proposed approach identified eight learning strategies and six trajectories of strategy change over the duration of the course, with statistically significant effects of strategy change trajectories on academic performance.

Research into self-regulated learning has traditionally relied upon self-reported data. While there is a rich body of literature that has extracted invaluable information from such sources, it suffers from a number of shortcomings. For instance, it has been shown that surveys often provide insight into students' perceptions about learning rather than how students actually employ study tactics and learning strategies. Accordingly, recent research has sought to assess students' learning strategies and, by extension, their self-regulated learning via trace data collected from digital learning environments. A number of studies have amply demonstrated the ability of educational data mining and learning analytics methods to identify patterns indicative of learning strategies within trace log data. However, many of these methods are limited in their ability to describe and interpret differences between extracted latent representations at varying levels of granularity (for instance, in terms of the underlying data of student actions and behavior). To address this limitation, the present study proposes a new methodology whereby interpretable representations of student's self-regulating behavior are derived at two theoretically inspired levels: that of learning strategies, and the study tactics that compose them.

Learning analytics has the potential to detect and explain characteristics of learning strategies through analysis of trace data and communicate the findings via feedback. However, the role of learning analytics-based feedback in selection and regulation of learning strategies is still insufficiently explored and understood. This research aims to examine the sequential and temporal characteristics of learning strategies and investigate their association with feedback. Three years of trace data were collected from online pre-class activities of a flipped classroom, where different types of feedback were employed in each year. Clustering, sequence mining, and process mining were used to detect and interpret learning tactics and strategies. Inferential statistics were used to examine the association of feedback with the learning performance and the detected learning strategies. The results suggest a positive association between the personalised feedback and the effective strategies.

This paper reports the findings of a study that proposed a novel learning analytic methodology that combines process mining with cluster analysis to study time management in the context of blended and online learning. The study was conducted with first-year students (N = 241) who were enrolled in blended learning of a health science course. The study identified four distinct time management tactics and three strategies. The tactics and strategies were interpreted according to the established theoretical framework of self-regulated learning in terms of student decisions about what to study, how long to study, and how to study. The study also identified significant differences in academic performance among students who followed different time management strategies.

The capacity to foster interpersonal interactions in massive open online courses (MOOCs) has frequently been contested, particularly when learner interactions are limited to MOOC forums. The establishment of social presence-a perceived sense of somebody being present and "real"-is among the strategies to tackle the challenges of online learning and could be applied in MOOCs. Thus far, social presence in MOOCs has been under-researched. Studies that previously examined social presence in MOOCs did not account for the peculiar nature of open online learning. In contrast to the existing work, this study seeks to understand how learners perceive social presence, and the different nuances of social presence in diverse MOOC populations. In particular, we compare perceptions of social presence across the groups of learners with different patterns of forum participation in three edX MOOCs. The findings reveal substantial differences in how learners with varying forum activity perceive social presence. Perceptions of social presence also differed in courses with the varying volume of forum interaction and duration. Finally, learners with sustained forum activity generally reported higher social presence scores that included low affectivity and strong group cohesion perceptions. With this in mind, this study is significant because of the insights into brings to the current body of knowledge around social presence in MOOCs. The study's findings also raise questions about the effectiveness of transferring existing socio-constructivist constructs into the MOOC contexts., System Engineering

The rapid growth of blended and online learning models in higher education has resulted in a parallel increase in the use of audio-visual resources among students and teachers. Despite the heavy adoption of video resources, there have been few studies investigating their effect on learning processes and even less so in the context of academic development. This paper uses learning analytic techniques to examine how academic teaching staff engage with a set of prescribed videos and video annotations in a professional development course. The data was collected from two offerings of the course at a large research-intensive university in Australia. The data was used to identify patterns of activity and transition states as users engaged with the course videos and video annotations. Latent class analysis and hidden Markov models were used to characterise the evolution of engagement throughout the course. The results provide a detailed description of the evolution of learner engagement that can be readily translated into action aimed at increasing the quality of the learning experience.

Learning analytics research has often been touted as a means to address concerns regarding student retention outcomes. However, few research studies to date, have examined the impact of the implemented intervention strategies designed to address such retention challenges. Moreover, the methodological rigor of some of the existing studies has been challenged. This study evaluates the impact of a pilot retention program. The study contrasts the findings obtained by the use of different methods for analysis of the effect of the intervention. The pilot study was undertaken between 2012 and 2014 resulting in a combined enrolment of 11,160 students. A model to predict attrition was developed, drawing on data from student information system, learning management system interactions, and assessment. The predictive model identified some 1868 students as academically at-risk. Early interventions were implemented involving learning and remediation support. Common statistical methods demonstrated a positive association between the intervention and student retention. However, the effect size was low. The use of more advanced statistical methods, specifically mixed-effect methods explained higher variability in the data (over 99%), yet found the intervention had no effect on the retention outcomes. The study demonstrates that more data about individual differences is required to not only explain retention but to also develop more effective intervention approaches.

The pervasive collection of data has opened the possibility for educational institutions to use analytics methods to improve the quality of the student experience. However, the adoption of these methods faces multiple challenges particularly at the course level where instructors and students would derive the most benefit from the use of analytics and predictive models. The challenge lies in the knowledge gap between how the data is captured, processed and used to derive models of student behavior, and the subsequent interpretation and the decision to deploy pedagogical actions and interventions by instructors. Simply put, the provision of learning analytics alone has not necessarily led to changing teaching practices. In order to support pedagogical change and aid interpretation, this paper proposes a model that can enable instructors to readily identify subpopulations of students to provide specific support actions. The approach was applied to a first year course with a large number of students. The resulting model classifies students according to their predicted exam scores, based on indicators directly derived from the learning design.

Websites offering daily deal offers have received widespread attention from the end-users. The objective of such Websites is to provide time limited discounts on goods and services in the hope of enticing more customers to purchase such goods or services. The success of daily deal Websites has given rise to meta-level daily deal aggregator services that collect daily deal information from across the Web. Due to some of the unique characteristics of daily deal Websites such as high update frequency, time sensitivity, and lack of coherent information representation, many deal aggregators rely on human intervention to identify and extract deal information. In this paper, we propose an approach where daily deal information is identified, classified and properly segmented and localized. Our approach is based on a semi-supervised method that uses sentence-level features of daily deal information on a given Web page. Our work offers (i) a set of computationally inexpensive discriminative features that are able to effectively distinguishWeb pages that contain daily deal information; (ii) the construction and systematic evaluation of machine learning techniques based on these features to automatically classify daily deal Web pages; and (iii) the development of an accurate segmentation algorithm that is able to localize and extract individual deals from within a complex Web page. We have extensively evaluated our approach from different perspectives, the results of which show notable performance.

Considering the ever-increasing speed at which new textual content is generated, an efficient and effective use of large text corpora requires automated natural language processing and text analysis tools. A subset of such tools, namely automated semantic annotation tools, are capable of interlinking syntactical forms of text with their underlying semantic concepts. The optimal performance of automated semantic annotation tools often depends on tuning the values of the tools' adjustable parameters to the specificities of the annotation task, and particularly to the characteristics of the text to be annotated. Such characteristics include the text domain, terseness or verbosity level, text length, structure and style. Since the default configuration of annotation tools is not suitable for the large variety of input texts that different combinations of these attributes can produce, users often need to adjust the annotators' tunable parameters in order to get the best results. However, the configuration of semantic annotators is presently a tedious and time consuming task as it is primarily based on a manual trial-and-error process. In this paper, we propose a Parameter Tuning Architecture (PTA) for automating the task of configuring parameter values of semantic annotation tools. We describe the core fitness functions of PTA that operate on the quality of the annotations produced, and offer a solution, based on a genetic algorithm, for searching the space of possible parameter values. Our experiments demonstrate that PTA enables effective configuration of parameter values of many semantic annotation tools.

The concept of social goals refers to organizational goals that are defined in an open and transparent manner; they serve as social objects that incite both formal and informal collaboration around shared interests/objectives. Our objective is to facilitate the comprehension of social goals and examine the role of social goals as scaffolds of social learning in an organization. To this end, we followed an approach based on the visualization of social goals and explored how different presentations of goals, specifically, faceted goal browsing, graph-based visualization, and timeline-based visualization, contribute to the realization of the stated objective. To assess this approach, we conducted a between subjects study where each participant performed a set of goal comprehension tasks with one of the examined presentations of goals. The study demonstrated that our visualizations of goals increase the accuracy of the overall comprehension of an organization's goals; this positive effect is also present when the comprehension of relationships-either explicit or implicit ties-among social goals is needed. The results also confirmed that our graph-based visualization of social goals could serve as a facilitator of social learning in an organization.

Ова дисертација је фокусирана на примену семантичких технологија за решавање проблема оптималне конфигурације сервисно-оријентисаних архитектура (енгл. Service Oriented Architecture – SOA) на основу нефункционалних захтева корисника. Решење је базирано на проширењу АHP алгоритма за рад са различитим врстама захтева и развоју хеуристичког приступа заснованог на генетичким алгоритмима за решавање проблема оптималне конфигурације. Постојећа решења у овој области су показала изузетно мали ниво персонализације, тј корисницима није дозвољено дефинисање разних софистициранијих врста захтева који осликавају њихове жеље, очекивања и строге захтеве за које захтевају потпуно испуњење. Такође, постојећа решења су била перманентно фокусирана на испуњење захтева функционалности, након чега се врши одабир конфигурације сходно захтевима о смањењу вредности карактеристика које имају тенденцију раста (нпр., цена и време извршавања), односно повећању вредности карактеристика које имају тенденцију опадања (нпр., поузданост и доступност). Међутим, када се посматрају целе фамилије SOA, од посебног значаја постаје проблем конструкције конфигурације при истовременом задовољењу функционалних и нефункционалних захтева. Предложено интегрално решење под називом OptConfSOAFобезбеђује представљање различитих врста захтева (безусловни, условни, захтеви о лексикографском поретку) о нефункционалним карактеристикама и оптималну конфигурацију фамилија SOA на основу дефинисаних захтева. Приступ који се предлаже обезбеђује истовремено задовољење захтева који се тичу функционалности система као и нефункционалних захтева који могу бити различитог нивоа приоритета, односити се на поједине делове или сервисно- оријентисану архитектуру у целости. Предложено решење је опште и није ограничено само на веб сервисе, иако се појам семантичких технологија обично везује за дати домен примене. Решење се може применити у било ком домену у којем се SOA парадигма може применити посматрањем сервиса као било које компоне, This dissertation is focused on the application of semantic technologies for solving the problem of optimal configuration of service-oriented architectures (SOA) based on stakeholders’ non-functional requirements. The proposed solution is developed as an extension of the AHP algorithm to allow for processing of different kinds of requirements. To address the problem of optimal configuration of SOA, a heuristic approach based on genetic algorithms has also been proposed and validated. Existing approaches in this field have shown low level of personalization, i.e. stakeholders are neither enabled to define sophisticated requirements that reflect their own expectations and attitudes, nor they are able to indicate hard requirements that have to be fully satisfied. Furthermore, existing approaches were primarily addressing the problem of fulfilling functional requirements, while the selection of an appropriate configuration is driven by the goal of decreasing the values of monotonically decreasing features (e.g., price and execution time) and simultaneous increasing the values of monotonically increasing features (e.g., availability and reliability). By considering the whole SOA families, the problem of configuration based on both functional and nonfunctional requirements gets special importance for research and further applications. The proposed solution, titled OptConfSOAF provides a framework for specification and processing of different kinds of requirements (unconditional, conditional, and requirements about lexicographical order) over non-functional features, and further optimal configuration of SOA families. The proposed approach provides simultaneous fulfillment of functional requirements (i.e., requirements related to the system’s functionalities) and non-functional requirements, where the latter could be defined with different level of importance, for specific parts of a SOA-based system or the system in its entirety. The proposed solution is general and is not

Present research and development offer various learning analytics tools providing insights into different aspects of learning processes. Adoption of a specific tool for practice is based on how its learning analytics are perceived by educators to support their pedagogical and organizational goals. In this paper, we propose and empirically validate a Learning Analytics Acceptance Model (LAAM) of factors influencing the beliefs of educators concerning the adoption a learning analytics tool. In particular, our model explains how the usage beliefs (i.e., ease-of-use and usefulness perceptions) about the learning analytics of a tool are associated with the intention to adopt the tool. In our study, we considered several factors that could potentially affect the adoption beliefs: i) pedagogical knowledge and information design skills of educators; ii) educators' perceived utility of a learning analytics tool; and iii) educators' perceived ease-of-use of a learning analytics tool. By following the principles of Technology Acceptance Model, the study is done with a sample of educators who experimented with a LOCO-Analyst tool. Our study also determined specific analytics types that are primary antecedence of perceived usefulness (concept comprehension and social interaction) and ease-of-use (interactive visualization).