Search

Student understanding of climate change is an active and growing area of research, but little research has documented undergraduate students' knowledge about the biotic impacts of climate change. Here, we address this literature gap by presenting the Inventory of Biotic Climate Literacy (IBCL), a concept inventory developed to assess undergraduate biology student knowledge of how climate change impacts living things. We developed the IBCL through literature review, student and expert interviews, student field tests, and expert review. We implemented two large nationwide field tests and conducted multiple psychometric analyses on these datasets. These analyses resulted in a final tool of 30 items measuring 16 constructs related to the biotic impacts of climate change. We discovered that the final IBCL does not represent a single, simple construct but rather the complicated and interactive concepts that comprise this topic. We suggest that sum scores are still a valuable measure, as certain groups (upperclassmen and politically liberal individuals) scored significantly higher. We also found value in analyzing individual student performance on the IBCL by developing student profiles. The IBCL represents an important tool in assessing student understanding of the complex and growing problem of climate change and its impact on the living world. [ABSTRACT FROM AUTHOR]

Background: Concept inventories (CIs) have become widely used tools for assessing students' learning and assisting with educational decisions. Over the past three decades, CI developers have utilized various design approaches and methodologies. As a result, it can be challenging for those developing new CIs to identify the most effective and appropriate methods and approaches. This scoping review aimed to identify and map key design stages, summarize methodologies, identify design gaps and provide guidance for future efforts in the development and validation of CI tools. Methods: A preliminary literature review combined theoretical thematic analysis (deductive, researcher-driven) focusing on specific data aspects, and inductive thematic analysis (data-driven), using emerging themes independent of specific research questions or theoretical interests. Expert discussions complemented the analysis process. Results: The scoping review analyzed 106 CI articles and identified five key development stages: define the construct, determine and validate content domain; identify misconceptions; item formation and response processes design; test item selection and validation; and test application and refinement. A descriptive design model was developed using a mixed-method approach, incorporating expert input, literature review, student-oriented analysis, and statistical tests. Various psychometric assessments were employed to validate the test and its items. Substantial gaps were noted in defining and determining the validity and reliability of CI tools, and in the evidence required to establish these attributes. Conclusion: The growing interest in utilizing CIs for educational purposes has highlighted the importance of identifying and refining the most effective design stages and methodologies. CI developers need comprehensive guidance to establish and evaluate the validity and reliability of their instruments. Future research should focus on establishing a unified typology of CI instrument validity and reliability requirements, as well as the types of evidence needed to meet these standards. This effort could optimize the effectiveness of CI tools, foster a cohesive evaluation approach, and bridge existing gaps. [ABSTRACT FROM AUTHOR]

Most biology undergraduates learn about mutations in multiple classrooms throughout their college career. Understanding personalised genome test results, genome editing controversies, and the appearance of new variants of viruses or antibiotic resistant bacteria all require foundational knowledge about mutations. However, the abstract nature of molecular processes surrounding mutations makes them one of the more difficult topics for students to understand and apply. Instructors need valid assessment tools to document student understanding and tailor their instructional methods to address student knowledge gaps. We describe here the development and validation of the Mutations Criterion Referenced Assessment (MuCRA). This formative assessment was developed through an iterative process involving expert feedback and student responses to both open-ended and multiple-choice questions. The final MuCRA is composed of 10 multiple-choice questions aligned with three learning objectives. The item difficulty for each question was between 0.32–0.65, while the discrimination index ranged from 0.31–0.75 and the reliability (KR20) for the MuCRA was 0.69. The congruence analyses demonstrated distractors are capturing student misconceptions in 9/10 questions. These data indicate that the MuCRA can be used to reliably assess student learning and common misconceptions about mutations. [ABSTRACT FROM AUTHOR]

BackgroundConcept inventories (CIs) have become widely used tools for assessing students’ learning and assisting with educational decisions. Over the past three decades, CI developers have utilized various design approaches and methodologies. As a result, it can be challenging for those developing new CIs to identify the most effective and appropriate methods and approaches. This scoping review aimed to identify and map key design stages, summarize methodologies, identify design gaps and provide guidance for future efforts in the development and validation of CI tools.MethodsA preliminary literature review combined theoretical thematic analysis (deductive, researcher-driven) focusing on specific data aspects, and inductive thematic analysis (data-driven), using emerging themes independent of specific research questions or theoretical interests. Expert discussions complemented the analysis process.ResultsThe scoping review analyzed 106 CI articles and identified five key development stages: define the construct, determine and validate content domain; identify misconceptions; item formation and response processes design; test item selection and validation; and test application and refinement. A descriptive design model was developed using a mixed-method approach, incorporating expert input, literature review, student-oriented analysis, and statistical tests. Various psychometric assessments were employed to validate the test and its items. Substantial gaps were noted in defining and determining the validity and reliability of CI tools, and in the evidence required to establish these attributes.ConclusionThe growing interest in utilizing CIs for educational purposes has highlighted the importance of identifying and refining the most effective design stages and methodologies. CI developers need comprehensive guidance to establish and evaluate the validity and reliability of their instruments. Future research should focus on establishing a unified typology of CI instrument validity and reliability requirements, as well as the types of evidence needed to meet these standards. This effort could optimize the effectiveness of CI tools, foster a cohesive evaluation approach, and bridge existing gaps.

The study aimed to analyze the content validation of the STEM-integrated on thermal and transport concept inventory instrument used to measure the problem-solving abilities of high school students. The instrument questions developed amounted to nine description questions. This type of study is development research. The steps in this research are define, design, development, and dissemination. However, in this study, it was limited to the design stage (expert validation). Furthermore, the instrument was assessed qualitatively and quantitatively by a validator consisting of three experts and five users. Assessment aspects include material, language, and construction. The qualitative data collected was analyzed qualitatively, and the quantitative data were analyzed using the Content Validity Ratio (CVR). The results showed that nine items had a CVR value exceeding 0.75, which meant that all items had valid criteria for content validity. The CVR value obtained is between 0.88 and 0.96, which means that the instrument has high criterion validity. [ABSTRACT FROM AUTHOR]

Solid-state physics has important applications in the development of modern technologies. Although solid-state concepts, such as electric and thermal conductivity, are included in the curricula at all educational levels, even university students have many incorrect ideas about them. The incorrect ideas identified in previous studies are mainly related to macroscopic aspects of solid-state physics. With the aim of gaining a more comprehensive insight into students' understanding of solid-state physics, we have developed a multiple-choice concept inventory on microscopic models of electric and thermal conductivity of solids (METCS). The inventory has been field-tested with a sample consisting of 233 first-year medical faculty and engineering students from the University of Rijeka (Croatia). METCS proved to have good psychometric features and it helped to uncover many incorrect ideas that have not been reported in the earlier physics education literature. The findings from this study could be a good starting point for the development of evidence-based, university-level tutorials on microscopic models of electric and thermal conductivity. [ABSTRACT FROM AUTHOR]

Abstract Background Random genetic drift is a difficult concept for biology undergraduates to understand. Active learning activities in a collaborative setting have the potential to improve student learning outcomes compared to traditional lectures alone and have been shown to help foster success for underrepresented students. However, few activities in this content area have been evaluated for effectiveness in improving student outcomes using peer-reviewed instruments backed by evidence of their validity and reliability. Our aim in this study was to use the Genetic Drift Instrument (GeDI) to evaluate and compare student learning gains in an upper division genetics course in which two different genetic drift activities, a faculty-developed collaborative exercise and a commercially published lab tutorial, were administered in an active-learning classroom with students working in small groups. Methods The GeDI was administered in both pre- and post-testing in two semesters (n = 95 and 98 students), with the semesters differing in which activity was assigned. Instrument dimensionality, person and item fit, and reliability were evaluated using Rasch analysis. Hierarchical Linear Models (HLMs) with two-way interactions were fitted to assess whether being in a certain Intervention Type, Race/ethnicity, Gender, or First Generation Status affected learning gains. HLMs with three-way interactions were used to assess whether the activities benefited students of all backgrounds equivalently. Results We found that the GeDI demonstrated unidimensionality, with high item reliability and relatively low person reliability, consistent with previous studies. Both the faculty-developed activity and the commercially available lab tutorial were associated with significant learning gains on genetic drift concepts. Students in the SimBio group had higher learning gains but the difference in effect size was small. No significant differences in learning gains were found between students from different demographic groups, and both activities appeared to benefit students of different backgrounds equivalently. Conclusions The GeDI instrument could be improved by adding items that more consistently differentiate students of different ability levels, especially at high ability levels. The greater impact on learning gains in the SimBio group while statistically significant does not translate into actual meaningful differences in student understanding. While students of different background variables in the sample have equivalent learning gains and are benefitted equivalently by the different interventions, our interventions did not ameliorate inequities in genetic drift understanding as measured by the GeDI that were uncovered in pre-testing.

Background: Random genetic drift is a difficult concept for biology undergraduates to understand. Active learning activities in a collaborative setting have the potential to improve student learning outcomes compared to traditional lectures alone and have been shown to help foster success for underrepresented students. However, few activities in this content area have been evaluated for effectiveness in improving student outcomes using peer-reviewed instruments backed by evidence of their validity and reliability. Our aim in this study was to use the Genetic Drift Instrument (GeDI) to evaluate and compare student learning gains in an upper division genetics course in which two different genetic drift activities, a faculty-developed collaborative exercise and a commercially published lab tutorial, were administered in an active-learning classroom with students working in small groups. Methods: The GeDI was administered in both pre- and post-testing in two semesters (n = 95 and 98 students), with the semesters differing in which activity was assigned. Instrument dimensionality, person and item fit, and reliability were evaluated using Rasch analysis. Hierarchical Linear Models (HLMs) with two-way interactions were fitted to assess whether being in a certain Intervention Type, Race/ethnicity, Gender, or First Generation Status affected learning gains. HLMs with three-way interactions were used to assess whether the activities benefited students of all backgrounds equivalently. Results: We found that the GeDI demonstrated unidimensionality, with high item reliability and relatively low person reliability, consistent with previous studies. Both the faculty-developed activity and the commercially available lab tutorial were associated with significant learning gains on genetic drift concepts. Students in the SimBio group had higher learning gains but the difference in effect size was small. No significant differences in learning gains were found between students from different demographic groups, and both activities appeared to benefit students of different backgrounds equivalently. Conclusions: The GeDI instrument could be improved by adding items that more consistently differentiate students of different ability levels, especially at high ability levels. The greater impact on learning gains in the SimBio group while statistically significant does not translate into actual meaningful differences in student understanding. While students of different background variables in the sample have equivalent learning gains and are benefitted equivalently by the different interventions, our interventions did not ameliorate inequities in genetic drift understanding as measured by the GeDI that were uncovered in pre-testing. [ABSTRACT FROM AUTHOR]

Concept inventories are multiple choice exams designed with the intention to test core concepts on specific subjects and evaluate common misconceptions. These tests serve as a useful tool in the classroom to assess value added by the instructor's educational methods and to better understand how students learn. They can provide educators with a method to evaluate their current teaching strategies and to make modifications that enhance student learning and ultimately elevate the quality of medical physics education. The use of concept inventories in introductory college physics courses revealed important gaps in conceptual understanding of physics by undergraduate students and motivated a shift of physics teaching towards more effective methods, such as active learning techniques. The goal of this review is to introduce medical physicists to concept inventories as educational evaluation tools and discuss potential applications to medical physics education by development through multi‐institutional collaboration. [ABSTRACT FROM AUTHOR]

Since the introduction of the Force Concept Inventory (FCI) in 1992, the CI tests have been widely used for measuring conceptual knowledge and for studying teaching issues in almost all disciplines and levels of study. A standard concept inventory analysis includes the design of a qualitative test, adequate realization of testing, calibration procedure, and comprehensive analysis of its findings. Usually, the CI test calibration is carried out through the Rasch sociometric technique, which is also used for calculating crucial indicators of knowledge such as item difficulties, students’ abilities, and many more. Whereas the quality of the tests’ design can be guaranteed by using certified and professional CI tests, the statistical adequacy of the testing merits critical attention before going on to the final step of the analysis. Also, the analysis of CI outcomes can be advanced by contemplating auxiliary tools and complementary techniques. In this framework, we propose to enforce the test index validity requirement for qualifying the CI outcomes as local or global. Specifically, the conclusions of CI analysis are acceptable for the whole population from which the sample has been extracted if the test's indexes comply with the validity requirements provided by the index theory. In the case when test indexes are out of validity range and re-conducting them is impractical for some objective circumstances or research specifics, we suggest injecting some new records into the existing one or mixing the data gathered from different samples until the new indexes are in the desired range. Using this methodology, we have reviewed our previous FCI tests, which were initially intended to demonstrate the impairment of learning in the physics discipline triggered by online learning during the pandemic closure. Through this renormalization procedure, we obtained a credible assessment of the understanding of mechanics and electromagnetism in high school students who followed online lectures during the pandemic closure. Also, by using indexes’ validity as an auxiliary tool, we identified that for measuring the knowledge of electromagnetism in students enrolled in branches where physics is a basic discipline, a shortened version of the BEMA test was a better instrument than the corresponding shortened EMCI test. Next, we used the optimal histogram idea borrowed from distribution fitting procedures to identify the natural levels of students’ abilities for solving a certain CI test. Another intriguing proposal presented in this work consists of combining an ad-hoc Likert scale assignment for usual errors in physics exams with the FCI designation of the basic commonsense confusion in mechanics for identifying their pairing features in common exams. We believe that the methods proposed herein can improve CI analysis in more general senses. Doi: 10.28991/HEF-2023-04-01-08 Full Text: PDF

Concept inventories (Cis) have become popular assessment tools in science, technology, engineering, and mathematics education. Some researchers use CI scores when looking at differences in conceptual understanding or learning gains across demographic groups, but very few Cis have been evaluated for measurement bias or other aspects that threaten the fair assessment of learners. The most common psychometric evaluation models are shaped primarily by the majority demographic group, so these models can hide biases in the assessment against minority groups. The purpose of this study was to evaluate the extent to which the validity, reliability, and fairness evidence supports the use of the total score on a 12-item Abbreviated Dynamics Concept Inventory (aDCI) as a measure of a student's overall conceptual understanding of dynamics. Because of the strong relationship between the aDCI and the Force Concept Inventory, which has previously been shown to include item-level gender biases, we examined threats to fair measurement across gender scores of the aDCI. We employed an argument-based validation approach which tested: (1) the fit of a single-factor latent structure for the aDCI scores via a confirmatory factor analysis (CFA), (2) the difficulty and discrimination of each item using item response theory, (3) the correlation between the aDCI scores and similar measures of conceptual understanding, and (4) the differential item functioning of the aDCI items across gender groups via a multiple-group CFA. We found that one item had face-level construct validity concerns and two others were slightly biased against women. Possible sources of gender bias included the question's content and context. Our results suggest that the interpretation of a student's total aDCI score should consider the differential item functioning of two items across gender and the construct-alignment concerns of a third item. This work highlights the importance and challenge of designing inclusive assessments and validating them with fair psychometric models. [ABSTRACT FROM AUTHOR]

The engineering schools usually target problem-solving skills in students instead of conceptual development, which is an essential skill for transformation from novice to professional engineer as per the program objective. Improving a student's conceptual knowledge can help students understand a problem better and develop a better solution. Conceptual understanding also assists students in identifying gaps in their problem-solving techniques. This paper attempts to administer a Signal and System Concept Inventory (SSCI) to test the conceptual knowledge of core concepts of signals and systems course and then identify the correlation of post-test scores with the student's performance in the end-term exam. The result shows that the students who scored above 80% in concept inventory also performed outstanding in the end-term examination. The result also indicates that most of the students able solve questions on background mathematics and polezero plots but struggled with convolution and Fourier analysis. [ABSTRACT FROM AUTHOR]

Solid-state physics has important applications in the development of modern technologies. Although solid-state concepts, such as electric and thermal conductivity, are included in the curricula at all educational levels, even university students have many incorrect ideas about them. The incorrect ideas identified in previous studies are mainly related to macroscopic aspects of solid-state physics. With the aim of gaining a more comprehensive insight into students’ understanding of solid-state physics, we have developed a multiple-choice concept inventory on microscopic models of electric and thermal conductivity of solids (METCS). The inventory has been field-tested with a sample consisting of 233 first-year medical faculty and engineering students from the University of Rijeka (Croatia). METCS proved to have good psychometric features and it helped to uncover many incorrect ideas that have not been reported in the earlier physics education literature. The findings from this study could be a good starting point for the development of evidence-based, university-level tutorials on microscopic models of electric and thermal conductivity.

Quantum technologies have outgrown mere fundamental research in laboratories over recent years, and will facilitate more and more potentially disruptive applications in a wide range of fields in the future. In foresight, qualification opportunities need to be implemented in order to train qualified specialists, referred to as the future quantum workforce, in various fields. Universities world-wide have launched qualification programmes for engineers focusing on quantum optics and photonics. In many of these programmes, students attend courses on quantum physics contextualized via quantum optics experiments with heralded photons, because: (1) their experimental and physical foundations may be directly leveraged to teaching a number of quantum technology applications, and (2) physics education research has provided empirical evidence, according to which such quantum optics-based approaches are conducive to learning about quantum concepts. While many teachers are confident about the effectiveness of their concepts, there is little empirical evidence due to the lack of content-area-specific research tools. We present a 16-item concept inventory to assess students’ conceptual understanding of quantum optics concepts in the context of experiments with heralded photons adopted from a test instrument published in the literature. We have administered this Quantum Optics Concept Inventory as a post-test to N=216 students after instruction on quantum optics as part of an undergraduate engineering course. We evaluated the instruments’ psychometric quality, both in terms of classical test theory, and using a Rasch scaling approach. The Quantum Optics Concept Inventory enables a reliable measure (α=0.74), and the data gathered show a good fit to the Rasch model. The students’ scores suggest that fundamental quantum effects pose striking learning hurdles to the engineering students. In contrast, most of the students are able to cope with the experimental and technical foundations of quantum optics experiments with heralded photons and their underlying principles, such as the coincidence technique used for the preparation of single-photon states. These findings are in accordance with prior research, and hence, the Quantum Optics Concept Inventory may serve as a fruitful starting point for future empirical research with regard to the education of the future quantum workforce.

This study aimed at exploring the understanding of concept inventories for classroom assessment by physics tutors and pre-service teachers in selected diploma teachers' training colleges in Tanzania. The study employed a qualitative research approach and a case study design. Eight physics tutors and 20 second-year pre-service physics teachers from two selected diploma colleges were involved in the study. Data were collected using tutors' interviews and pre-service teachers' focus group discussions. Analysis of data was carried out by using the deductive thematic analysis technique.The findings revealed that these physics tutors and pre-service teachers understand concept inventory as a question-and-answer assessment strategy and as a criterion-referenced test. It was also revealed that both groups were not familiar with the format and examples of concept inventory tests found in physics. Moreover, the findings indicated that participants were conducting diagnostic assessments by using other forms of assessment tools apart from concept inventories. The assessment practices were dominated by traditional assessment methods such as written tests and examinations. The study concluded that assessments through concept inventories remain a big challenge to the participants. The study recommends providing in-service training for physics tutors on the use of concept inventory for effective assessments of pre-service teachers and training the latter on how to use concept inventories. [ABSTRACT FROM AUTHOR]

This paper discusses the development and validation of a concept inventory for interpreting kinematics graphs in the Tanzanian context. The study involved 61 participants comprising physics pre-service teachers, secondary school teachers, diploma college tutors, and a university lecturer from Tanzania. We developed 25 multiple-choice questions for interpreting kinematics graphs. The different steps in the development process used are selecting the topic, setting objectives, constructing questions, validating questions, and reliability testing. We carried out descriptive and inferential statistical analysis by using Statistical Package for Social Science (SPSS) version 22 followed by item analysis for pre-and post-piloting. Findings revealed normal distribution scores with a mean and standard deviation of 39.28±10.893 for pre-piloting and 40.16±8.08 for post-piloting. It also revealed no significant difference between pre-and post-piloting results with a p-value of 0.414. In addition, correlation coefficients for test re-test reliability were .783 and .878 for single and average measures respectively. Moreover, item analysis in terms of difficulty index, discrimination index, and distractor efficiency agreed with the published standards. Based on these findings, the study recommends the use of developed and validated kinematics graphs concept inventory by physics educators in both research and classroom instructions in the Tanzanian context. [ABSTRACT FROM AUTHOR]

This article reports on the development and piloting of a German version of a concept test to assess students’ conceptual knowledge of density. The concept test was administered in paper-pencil format to 222 German secondary school students as a post-test after instruction in all relevant concepts of density. We provide a psychometric characterization of the instrument based on the student responses. Therefore, we use methods from classical test theory and item response theory. Our analysis shows that our concept test has good psychometric properties. Furthermore, it is shown to enable a reliable (α= 0.77) survey of students’ conceptual knowledge of density. Finally, we use the student responses to identify secondary school students’ difficulties with the density concept. We replicate student difficulties on density from the literature, but our study also reveals some new difficulties.

In this study, we administered and evaluated circular motion concept question items with a view to developing an inventory suitable for the Ugandan context. Before administering the circular concept items, six physics experts and ten undergraduate physics students carried out the face and content validation. One hundred eighteen undergraduate students responded to the 42 circular motion concept items. The data were analysed using the classical test theory (CTT) and item response curve (IRC) analyses. We calculated the difficulty level and index of discrimination and gauged the distraction efficiency of items. The IRCs revealed insights that were not evident from those provided by the CTT. Based on the IRCs, the circular concept items are classified into three categories: efficient, moderately efficient, and inefficient. This helped us better evaluate the quality of the items and their appropriateness for the population under consideration. We ended up with 22 circular motion concept question items which we call the circular motion concept inventory (CMCI). This inventory is particularly relevant to Ugandan context and may be useful to other countries in the East African region which share similar syllabi. [ABSTRACT FROM AUTHOR]

The purpose of this article is to validate the relevance of a concept inventory on fractions by measuring the presence and evolution of misconceptions among prospective primary and pre-school teachers, including the overcoming of their misconceptions during and at the end of the instructional intervention. Seven text statements were defined and composed a "Likert scale" concept inventory. This was administered to students at three different stages of the learning process to measure the understanding gain: before the start of the course, after the instructional intervention consisting of a lecture and a seminar on fractions, and finally at the end of the semester. The results from the initial testing confirmed the misconceptions about fractions of the students. Based on the experiment, it is thus obvious the need to allocate a longer time for understanding and fixing the concept of fraction for prospective primary and preschool teachers. [ABSTRACT FROM AUTHOR]

Standard assessments can give instructors reliable objective measures of student skills at the beginning and end of a term. We describe seven standard assessments that we have developed for commonly taught economics courses, and introduce a new website, econ-assessments.org, that allows instructors worldwide to set up any of our assessments for their students. Instructors get a link to their chosen assessment that can be shared with students, and students take the timed assessment when they follow the link. At the end of the test-taking period, instructors receive a report that summarizes performance in aggregate and by learning goal. [ABSTRACT FROM AUTHOR]

Cosmology concepts encompass complex spatial and temporal relations that are counterintuitive. Cosmology findings, because of their intrinsic interest, are often reported in the public domain with enthusiasm, and students come to cosmology with a range of conceptions some aligned and some at variance with the current science. This makes cosmology concepts challenging to teach, and also challenging to evaluate students' conceptual understanding. This study builds on previous research of the authors investigating the methodological challenges for characterising students' cosmology conceptions and the reasoning underlying these. Insights from student responses in two iterations of an open-ended instrument were used to develop a concept inventory that combined cosmological conceptions with reasoning levels based on the SOLO taxonomy. This paper reports on the development and validation of the Cosmology Concept Inventory (CosmoCI) for high school. CosmoCI is a 28-item multiple-choice instrument that was implemented with grade 10 and 11 school students (n = 234) in Australia and Sweden. Using Rasch analysis in the form of a partial credit model (PCM), the paper describes a validated progression in student reasoning in cosmology across four conceptual dimensions, supporting the utility of CosmoCI as an assessment tool which can also instigate rich discussions in the science classroom. [ABSTRACT FROM AUTHOR]

Quantum technologies have outgrown mere fundamental research in laboratories over recent years, and will facilitate more and more potentially disruptive applications in a wide range of fields in the future. In foresight, qualification opportunities need to be implemented in order to train qualified specialists, referred to as the future quantum workforce, in various fields. Universities world-wide have launched qualification programmes for engineers focusing on quantum optics and photonics. In many of these programmes, students attend courses on quantum physics contextualized via quantum optics experiments with heralded photons, because: (1) their experimental and physical foundations may be directly leveraged to teaching a number of quantum technology applications, and (2) physics education research has provided empirical evidence, according to which such quantum optics-based approaches are conducive to learning about quantum concepts. While many teachers are confident about the effectiveness of their concepts, there is little empirical evidence due to the lack of content-area-specific research tools. We present a 16-item concept inventory to assess students' conceptual understanding of quantum optics concepts in the context of experiments with heralded photons adopted from a test instrument published in the literature. We have administered this Quantum Optics Concept Inventory as a post-test to N = 216 students after instruction on quantum optics as part of an undergraduate engineering course. We evaluated the instruments' psychometric quality, both in terms of classical test theory, and using a Rasch scaling approach. The Quantum Optics Concept Inventory enables a reliable measure ( α = 0.74 ), and the data gathered show a good fit to the Rasch model. The students' scores suggest that fundamental quantum effects pose striking learning hurdles to the engineering students. In contrast, most of the students are able to cope with the experimental and technical foundations of quantum optics experiments with heralded photons and their underlying principles, such as the coincidence technique used for the preparation of single-photon states. These findings are in accordance with prior research, and hence, the Quantum Optics Concept Inventory may serve as a fruitful starting point for future empirical research with regard to the education of the future quantum workforce. [ABSTRACT FROM AUTHOR]

IntroductionAs artificial intelligence (AI) technology becomes more widespread in the classroom environment, educators have relied on data-driven machine learning (ML) techniques and statistical frameworks to derive insights into student performance patterns. Bayesian methodologies have emerged as a more intuitive approach to frequentist methods of inference since they link prior assumptions and data together to provide a quantitative distribution of final model parameter estimates. Despite their alignment with four recent ML assessment criteria developed in the educational literature, Bayesian methodologies have received considerably less attention by academic stakeholders prompting the need to empirically discern how these techniques can be used to provide actionable insights into student performance.MethodsTo identify the factors most indicative of student retention and attrition, we apply a Bayesian framework to comparatively examine the differential impact that the amalgamation of traditional and AI-driven predictors has on student performance in an undergraduate in-person science, technology, engineering, and mathematics (STEM) course.ResultsInteraction with the course learning management system (LMS) and performance on diagnostic concept inventory (CI) assessments provided the greatest insights into final course performance. Establishing informative prior values using historical classroom data did not always appreciably enhance model fit.DiscussionWe discuss how Bayesian methodologies are a more pragmatic and interpretable way of assessing student performance and are a promising tool for use in science education research and assessment.

In this study, we administered and evaluated circular motion concept question items with a view to developing an inventory suitable for the Ugandan context. Before administering the circular concept items, six physics experts and ten undergraduate physics students carried out the face and content validation. One hundred eighteen undergraduate students responded to the 42 circular motion concept items. The data were analysed using the classical test theory (CTT) and item response curve (IRC) analyses. We calculated the difficulty level and index of discrimination and gauged the distraction efficiency of items. The IRCs revealed insights that were not evident from those provided by the CTT. Based on the IRCs, the circular concept items are classified into three categories: efficient, moderately efficient, and inefficient. This helped us better evaluate the quality of the items and their appropriateness for the population under consideration. We ended up with 22 circular motion concept question items which we call the circular motion concept inventory (CMCI). This inventory is particularly relevant to Ugandan context and may be useful to other countries in the East African region which share similar syllabi. [ABSTRACT FROM AUTHOR]

Many disciplines have used concept inventories (CI) to better understand the alternative ideas held by students. Administering a CI at both the beginning and the end of the semester can yield much information. For example, a comparison of both pre- and post-test responses can help determine if specific gains in student learning were met. In this paper, we describe the development of a concept inventory over four academic years for the one-semester general, organic, and biochemistry (GOB) course required for nursing students. Questions were designed around course topics and to identify common misconceptions, which were gathered from the literature, student interviews, and content experts. The instrument was refined over several iterations to ensure that all questions were appropriate and understandable while providing useful information. The final version demonstrated appropriate gains in the comparison of pre- and post-test results for an active learning classroom and results are discussed. In addition, item quality was ascertained via the discrimination and difficulty values, and Cronbach's alpha coefficient was computed for the entire test as a measure of its reliability. [ABSTRACT FROM AUTHOR]

This article reports on the development and piloting of a German version of a concept test to assess students’ conceptual knowledge of density. The concept test was administered in paper-pencil format to 222 German secondary school students as a post-test after instruction in all relevant concepts of density. We provide a psychometric characterization of the instrument based on the student responses. Therefore, we use methods from classical test theory and item response theory. Our analysis shows that our concept test has good psychometric properties. Furthermore, it is shown to enable a reliable (α= 0.77) survey of students’ conceptual knowledge of density. Finally, we use the student responses to identify secondary school students’ difficulties with the density concept. We replicate student difficulties on density from the literature, but our study also reveals some new difficulties. [ABSTRACT FROM AUTHOR]

Pharmacology education currently lacks a research‐based consensus on which core concepts all graduates should know and understand, as well as a valid and reliable means to assess core conceptual learning. The Core Concepts in Pharmacology Expert Group (CC‐PEG) from Australia and New Zealand recently identified a set of core concepts of pharmacology education as a first step toward developing a concept inventory—a valid and reliable tool to assess learner attainment of concepts. In the current study, CC‐PEG used established methodologies to define each concept and then unpack its key components. Expert working groups of three to seven educators were formed to unpack concepts within specific conceptual groupings: what the body does to the drug (pharmacokinetics); what the drug does to the body (pharmacodynamics); and system integration and modification of drug–response. First, a one‐sentence definition was developed for each core concept. Next, sub‐concepts were established for each core concept. These twenty core concepts, along with their respective definitions and sub‐concepts, can provide pharmacology educators with a resource to guide the development of new curricula and the evaluation of existing curricula. The unpacking and articulation of these core concepts will also inform the development of a pharmacology concept inventory. We anticipate that these resources will advance further collaboration across the international pharmacology education community to improve curricula, teaching, assessment, and learning.The second stage of development of core concepts of pharmacology education; defining each concept, and unpacking the concept by articulation of sub‐concepts. [ABSTRACT FROM AUTHOR]

Biochemistry curricula present a particular challenge to undergraduate students with abstract concepts which can lead to misconceptions that impede learning. In particular, these students have difficulty understanding enzyme structure and function concepts. Targeted learning activities and three‐dimensional (3D) physical models are proposed to help students challenge these misconceptions and increase conceptual understanding. Here we assessed such pedagogical tools using the Enzyme‐Substrate Interactions Concept Inventory (ESICI) to measure (mis)conceptual changes from Pre‐ to Post‐ time points in a single semester undergraduate biochemistry course. A Control group of students engaged with the active learning activities without the 3D physical models and students in the Intervention group utilized these activities with the 3D physical models. At the Post‐ time point both groups had higher, yet similar ESICI scores of the same magnitude as the highest scoring group from the national sample. Concomitantly, many misconception markers decreased compared to the national sample, although some of these differed between the Control and Intervention groups. Based on this assessment, both pedagogical approaches successfully increased conceptual understanding and targeted many of the misconceptions measured by the ESICI, however, several misconceptions persisted. Surprisingly, the students who used the 3D physical models did not demonstrate a further decrease in the misconception markers. Additionally, psychometric evaluation of the ESICI with our sample recommends the revision of several questions to improve the validity of this assessment. We also offer suggestions to improve instruction and pedagogical tools with further avenues for research on learning. [ABSTRACT FROM AUTHOR]

Undergraduate STEM students majoring in various science sub-disciplines (e.g. chemistry, physics, engineering) must develop strong understandings of core foundational thermodynamics concepts. The ability for course instructors and researchers to effectively refine instruction and develop interventions to support students' learning hinges on their ability to accurately gauge students' knowledge through the use of established measures. The Thermodynamics Conceptual Reasoning Inventory (TCRI) is designed to gauge undergraduate students' understanding of introductory thermodynamics concepts. The present study extends the findings of a previous publication by positioning the TCRI within the broader international literature of thermodynamics concept inventories and generating an argument for the reliability and validity of TCRI scores in a broader context. Participants (n = 278) took the revised 36-item TCRI (available in the supplementary online materials). Findings revealed that TCRI scores are useful in the broader context (e.g. no evidence of floor or ceiling effects, evidence of high reliability, no differences for students across majors, and TCRI scores were moderately correlated with both course exam scores and GPA). No further revisions are recommended based on analysis of item properties. The cumulative body of evidence related to the TCRI suggests that scores are useful indicators of undergraduate students' conceptual understanding of introductory thermodynamics concepts. [ABSTRACT FROM AUTHOR]

Abstract An investigation of the gender gap in engineering students’ understanding of Newtonian physics has been done. The results indicate a noticeable gender gap, in addition to an effect due to differences in preparation, i.e. physics courses in secondary school. The gender gap after university instruction decreases in the case of a higher degree of preparation, while the gender gap is unchanged or widens for students with a lower degree of preparation after university instruction.

Abstract Pharmacology education currently lacks an agreed knowledge curriculum. Evidence from physics and biology education indicates that core concepts are useful and effective structures around which such a curriculum can be designed to facilitate student learning. Building on previous work, we developed a novel, criterion‐based method to identify the core concepts of pharmacology education. Five novel criteria were developed, based on a literature search, to separate core concepts in pharmacology from topics and facts. Core concepts were agreed to be big ideas, enduring, difficult, applicable across contexts, and useful to solve problems. An exploratory survey of 33 pharmacology educators from Australia and New Zealand produced 109 terms, which were reduced to a working list of 26 concepts during an online workshop. Next, an expert group of 12 educators refined the working list to 19 concepts, by applying the five criteria and consolidating synonyms, and added three additional concepts that emerged during discussions. A confirmatory survey of a larger group resulted in 17 core concepts of pharmacology education. This list may be useful for educators to evaluate existing curricula, design new curricula, and to inform the development of a concept inventory to test attainment of the core concepts in pharmacology.

Pharmacology education currently lacks an agreed knowledge curriculum. Evidence from physics and biology education indicates that core concepts are useful and effective structures around which such a curriculum can be designed to facilitate student learning. Building on previous work, we developed a novel, criterion‐based method to identify the core concepts of pharmacology education. Five novel criteria were developed, based on a literature search, to separate core concepts in pharmacology from topics and facts. Core concepts were agreed to be big ideas, enduring, difficult, applicable across contexts, and useful to solve problems. An exploratory survey of 33 pharmacology educators from Australia and New Zealand produced 109 terms, which were reduced to a working list of 26 concepts during an online workshop. Next, an expert group of 12 educators refined the working list to 19 concepts, by applying the five criteria and consolidating synonyms, and added three additional concepts that emerged during discussions. A confirmatory survey of a larger group resulted in 17 core concepts of pharmacology education. This list may be useful for educators to evaluate existing curricula, design new curricula, and to inform the development of a concept inventory to test attainment of the core concepts in pharmacology.Over 50 pharmacology educators from Australia and New Zealand worked together to identify the core concepts of pharmacology education. The figure shows the 19 concepts identified and the process taken to produce this list [ABSTRACT FROM AUTHOR]

Background: Accreditation organizations broadened program assessment criteria in ways that present challenges for the evaluation of learning outcomes. This is especially the case in courses where there are not narrowly defined questions or definitive solutions, such as engineering ethics. While protocols for learning assessment exist, there is limited research exploring sociotechnical learning outcomes in a manner that combines theoretical, empirical, and procedural aspects of assessment. Purpose/Hypothesis: This paper shares 3 years of research into the effectiveness of concept maps as an assessment tool for engineering students in courses that prioritize professional skills development and sociotechnical thinking. We show that concept maps can offer evidence of knowledge formation and learning outcomes associated with courses that introduce complex problems with multiple possible interpretations or viable solutions. Design/Method: A concept mapping activity was completed by 614 undergraduate engineering students at the start and end of three different courses to evaluate sociotechnical thinking. Student‐level longitudinal changes were evaluated using paired t tests and simple linear regression. The concept mapping activity was improved iteratively in response to pilot tests and focus groups. Results: Undergraduate engineers demonstrated greater complexity in the representation of sociotechnical relationships as observed in the structure and content of the concept maps. The methodological results offer lessons about the administration and analysis of concept maps often excluded from conversations on course and curriculum level assessment. Conclusions: Concept maps offer insights into student learning outcomes, and they can be deployed and analyzed with minimal resources. However, assessments must be carefully designed to account for administrative and analytical challenges. [ABSTRACT FROM AUTHOR]

In this paper, we propose an experiential learning module for the course Signals and Systems (SNS). This course is designed in the second year of the Electronics and Communication Engineering stream and challenging course as it demands a higher level of cognition skills. Experiential learning is a series of activities that allow students to be actively involved in ensuring learning. The experiential learning module enhances the problemsolving skills towards reaching higher levels of cognition, moving from the lower levels of remembering, understanding. Typically, the conventional modes are teacher-centric, visual learners and sequential learners find challenges in stimulating their cognition skills. Towards this, we design an experiential learning framework with two techniques: Concept inventory and SNS with Python to facilitate sequential learners and visual learners respectively. The challenges associated with the deployment of our framework are time and resource management. The proposed framework facilitates achieving a higher level of cognitive skills; also the introduction of the simulating tool ensures the conceptual learning of the course beyond classroom learning. [ABSTRACT FROM AUTHOR]

In 2011, we published a description of 15 core concepts of physiology, and in 2017 we described how core concepts could be used to teach physiology. On the basis of publications and conference presentations, it is clear that the core concepts, conceptual frameworks, and the homeostasis concept inventory have been used by faculty in many ways to improve and assess student learning and align instruction and programs. A growing number of colleagues focus their teaching on physiology core concepts, and some core concepts have been used as explicit themes or organizing principles in physiology or anatomy and physiology textbooks. The core concepts published in 2011 were derived from inputs from a diverse group of physiology instructors and articulated what this group of instructors expressed a decade ago. On the basis of current feedback from the physiology teaching community as a consequence of the use of core concepts in teaching and learning, we have revisited these concepts and made revisions to address issues that have emerged. In this article, we offer revised definitions and explanations of the core concepts, propose an additional core concept (“physical properties of matter” which combines two previous concepts), and describe three broad categories for the revised core concepts. Finally, we catalog published resources for each of the core concepts that provide instructors tools to focus facilitation of student learning on goals (learning outcomes), activities and assessments to enable students to develop and apply their understanding of the core concepts of physiology. [ABSTRACT FROM AUTHOR]

Recently, the University of Waikato introduced a first-year physics course requiring no specific physics or mathematics background, to support degrees in other subjects and allowing maximum diversity in student intake. This course uses the Investigative Science Learning Environment (ISLE) approach in which students develop their own experiments to tackle scientific questions. ISLE has proved successful with mathematically able classes; we have adapted it for a less capable cohort in a regional New Zealand context. We have assessed our approach for 2018 using: pre- and post-course concept tests; performance in exams; a student focus group; analysis of laboratory notebooks and marksheets; interviews with teaching staff; and student comments from the university's course evaluations. Concept tests showed a mean normalised gain of 0.61. Students did well in the exam. Many students were initially unfamiliar with practical scientific investigation and record keeping but laboratory notebooks showed desired learning outcomes were achieved by the end. Students and staff engaged well with this course. Students' final grades were not significantly different between genders and between under-represented minority and non-minority students. We cannot yet say whether this approach provides a sufficient platform for progress in engineering but students have learned fundamental practical physical concepts. [ABSTRACT FROM AUTHOR]

We present ideas about concepts underlying climate change, held by students in years 9 and 10. Misconceptions about climate change are common among students, and may be due to misconceptions about underlying concepts. To investigate this, we developed the Climate Change Concept Inventory (CCCI), and trialed it with 229 students; corroborating findings through focus group interviews. Our interview method and data analysis methods are described. Findings included overestimation of human contributions to atmospheric carbon inputs, ultra violet radiation in sunlight, and greenhouse gases in the atmosphere. Students were unaware that CO2 dissolves in water, and of the role of oceans in the carbon cycle. Greenhouse gases other than CO2 were rarely known. Earth's energy balance and black body radiation were not well understood. There were misconceptions about interactions between electromagnetic radiation and atmospheric gases; and limited understanding of carbon chemistry. The CCCI is available from the corresponding author. [ABSTRACT FROM AUTHOR]

The discovery and characterisation of planets orbiting distant stars has shed light on the origin of our own Solar System. It is important that college-level introductory astronomy students have a general understanding of the planet formation process before they are able to draw parallels between extrasolar systems and our own Solar System. In this work, we introduce the Planet Formation Concept Inventory (PFCI), an educational research tool used to assess student learning on the topic of planet formation. The PFCI Version 3 was administered to N = 561 students pre-instruction and N = 374 students post-instruction. Here, we present a Classical Test Theory (CTT) analysis of the PFCI Version 3. Ultimately, we conclude that the PFCI is a reliable and valid instrument that can differentiate experts from novices, and can be used to assess college-level introductory astronomy students' learning on the topic of planet formation. Initial findings on class normalised gain scores indicate that the PFCI may be capable of assessing the effectiveness of different instructional models. In the future, we recommend a national study of the PFCI to discern its ability to provide insight regarding the ascribed characteristics of learners and the effectiveness of different instructional strategies being used to teach this topic. [ABSTRACT FROM AUTHOR]

Concept inventories have become increasingly common in STEM disciplines as a means of assessing student conceptual understanding on a given topic, and overall they have led to significant reform in the teaching and learning of content in their respective disciplines. In mathematics, the use of the Calculus Concept Inventory seems, anecdotally and based on a review of the literature, to be growing. Yet peer-reviewed literature on its development and psychometric properties is lacking. Using data from approximately 1800 students across four institutions, we analysed its content validity, internal structure validity, and reliability. We conclude that the data is consistent with a unidimensional model and that the instrument lacks sufficiently strong validity and reliability for its intended use. We therefore recommend the development of a new instrument to better measure conceptual understanding of calculus concepts based on a stronger theoretical framework. [ABSTRACT FROM AUTHOR]

This paper examines engineering students' misconceptions of thermodynamics. A summary of the pedagogical theories of misconceptions is presented and the cognitive roots of misconceptions are briefly reviewed. Concept tests are discussed as one of the most effective tools for diagnosing misconceptions, followed by a summary of the available thermodynamics concept tests. Finally, an overview is presented of the most frequent thermodynamics misconceptions among engineering students. [ABSTRACT FROM AUTHOR]

The automated design of a set of practice problems that co-adapts to a population of learners is a challenging problem. Fortunately, coevolutionary computation offers a rich framework to study interactions between two co-adapting populations of teachers and learners. This framework is also relevant in scenarios in which a population of students solve practice exercises that are synthesized by an evolutionary algorithm. In this study, we propose to leverage coevolutionary optimization to evolve a population of Parsons puzzles (a relatively recent new type of practice exercise for novice computer programmers). To this end, we start by experimenting with successive simulations that progressively introduce the characteristics that we anticipate finding in our target application. Using these simulations, we refine a set of guidelines that capture insights on how to successfully coevolve Parsons puzzles. These guidelines are then used to implement the proposed "EvoParsons" software, with which we conduct preliminary evaluations on real human students enrolled in an introductory Java programming course at the University of South Florida. We also propose several quantitative metrics to assess the quality of puzzles produced by EvoParsons. Both simulations and experiments establish the feasibility of evolving pedagogically relevant practice problems that cover most of the dimensions underlying the interactions between problems and students. In addition, a generation-by-generation detailed analysis of the evolving population of Parsons puzzles confirms the occurrence of incremental improvements that can be explained in pedagogical terms. [ABSTRACT FROM AUTHOR]

We describe the development of a new, freely available, online, programmatic-level assessment tool, Measuring Achievement and Progress in Science in Physiology, or Phys-MAPS (http://cperl.lassp.cornell.edu/bio-maps). Aligned with the conceptual frameworks of Core Principles of Physiology, and Vision and Change Core Concepts, Phys-MAPS can be used to evaluate student learning of core physiology concepts at multiple time points in an undergraduate physiology program, providing a valuable longitudinal tool to gain insight into student thinking and aid in the data-driven reform of physiology curricula. Phys-MAPS questions have a modified multiple true/false design and were developed using an iterative process, including student interviews and physiology expert review to verify scientific accuracy, appropriateness for physiology majors, and clarity. The final version of Phys-MAPS was tested with 2,600 students across 13 universities, has evidence of reliability, and has no significant statement biases. Over 90% of the physiology experts surveyed agreed that each Phys-MAPS statement was scientifically accurate and relevant to a physiology major. When testing each statement for bias, differential item functioning analysis demonstrated only a small effect size (<0.008) of any tested demographic variable. Regarding student performance, Phys-MAPS can also distinguish between lower and upper division students, both across different institutions (average overall scores increase with each level of class standing; two-way ANOVA, P < 0.001) and within each of three sample institutions (each ANOVA, P ≤ 0.001). Furthermore, at the level of individual concepts, only evolution and homeostasis do not demonstrate the typical increase across class standing, suggesting these concepts likely present consistent conceptual challenges for physiology students. [ABSTRACT FROM AUTHOR]

Concept inventories (Cls) have become popular assessment tools in science, technology, engineering, and mathematics education. Some researchers use Cl scores when looking at differences in conceptual understanding or learning gains across demographic groups, but very few Cls have been evaluated for m easurement bias or other aspects that threaten the fair assessment of learners. The most common psychometric evaluation models are shaped primarily by the majority demographic group, so these models can hide biases in the assessment against minority groups. The purpose of this study was to evaluate the extent to which the validity, reliability, and fairness evidence supports the use of the total score on a 12-item Abbreviated Dynamics Concept Inventory (aDCl) as a measure of a student's overall conceptual understanding of dynamics. Because of the strong relationship between the aDCl and the Force Concept Inventory, which has previously been shown to include item-level gender biases, we examined threats to fair measurement across gender scores of the aD C l. Wc employed an argument-based validation approach which tested: (1) the fit of a single-factor latent structure for the aDCl scores via a confirmatory factor analysis (CFA), (2) the difficulty and discrimination of each item using item response theory, (3) the correlation between the aD Cl scores and similar measures of conceptual understanding, and (4) the differential item functioning of the aDCl items across gender groups via a multiple-group CFA. We found that one item had face-level construct validity concerns and two others were slightly biased against women. Possible sources of gender bias included the question's content and context. Our results suggest that the interpretation of a student's total aDCl score should consider the differential item functioning of two items across gender and the construct-alignment concerns of a third item. This work highlights the importance and challenge of designing inclusive assessments and validating them with fair psychometric models. [ABSTRACT FROM AUTHOR]

Measuring understanding is notoriously difficult. Indeed, in formulating learning outcomes the word 'understanding' is usually avoided, but in the sciences, developing understanding is one of the main aims of instruction. Scientific knowledge is factual, having been tested against empirical observation and experimentation, but knowledge of facts alone is not enough. There are also models and theories containing complex ideas and interrelationships that must be understood, and considerable attention has been devoted across a range of scientific disciplines to measuring understanding. This case study will focus on one of the main tools employed: the concept inventory and in particular the Force Concept Inventory (FCI). The success of concept inventories in physics has spawned concept inventories in chemistry, biology, astronomy, materials science and maths, to name a few. We focus here on the FCI and ask how useful concept inventories are for evaluating learning gains. Finally, we report on recent work by the authors to extend conceptual testing beyond the multiple-choice format. [ABSTRACT FROM AUTHOR]

Background and Purpose: Genomic nursing education requires a genomic literacy assessment supported by evidence of reliability and validity. This study applied psychometric analyses to provide support for the Genomic Nursing Concept Inventory (GNCI). Methods: Over nine semesters, baccalaureate nursing students (N = 1,065) completed the GNCI on the first and last days of genomics instruction. Psychometric analyses assessed scale and item performance pre- and post-instruction. Results: Exploratory factor analysis supported scale unidimensionality and identified items with low pattern coefficients. Analyses supported test–retest and internal consistency reliability and criterion validity. Scale difficulty decreased by 28% from pre- to post-instruction. Underperforming items were identified for further inventory refinement. Conclusions: Findings support use of the GNCI to measure learning needs pre-instruction and learning gains post-instruction. Data also inform planned inventory revision. [ABSTRACT FROM AUTHOR]