Search

In this essay, we explore the consequences of the University of California's policies to address racial disparities and its support for social justice activism as influences on its commitment to academic freedom and other intellectual values. This is a story of the interaction between two essential public university missions -- one civic, the other intellectual -- and the slow effacement of one by the other. The University's expressed commitments to academic freedom and the culture of rationalism have not been abandoned, but they are too often considered secondary or when confronted by new administrative initiatives and social movement activism related to diversity, equity, and inclusion (DEI). The experimental use of mandatory DEI statements on a number of the ten UC campuses, within willing academic departments, as initial screening mechanisms in faculty hiring is the most dramatic of the new administrative policies that have been put into place to advance faculty diversity. This policy can be considered the most problematic of a series of efforts that the UC campuses and the UC Office of the President have taken for more than a decade to prioritize representation in academic appointments. Our intent is to encourage a discussion of these policies within UC in light of the University's fundamental commitments to open intellectual inquiry, the discovery and dissemination of a wide range of new knowledge, and a culture of rationalism.

The guided pathways model, comprising 14 different practices, is a framework for comprehensive, whole-college reform undertaken by community colleges to help all students choose, enter, progress through, and complete a program of study that enables them to secure sustaining-wage employment or transfer with junior standing in a major. Since its introduction in 2015, it has been adopted by hundreds of community colleges across the United States. This paper asks whether guided pathways practices implemented at 62 community and technical colleges in three states--Tennessee, Ohio, and Washington--are associated with improvements in student outcomes during the first year of college. Specifically, using institutional survey and rich administrative data, we construct measures of adoption of guided pathways reforms to examine the association between guided pathways practices and fall-to-fall persistence, college credits earned, college math credits earned, and STEM credits earned. Our study reveals substantial variation in the adoption of guided pathways reforms across the states and across community colleges within the states over time. While we cannot establish a causal relationship between guided pathways adoption and student outcomes, we find significant positive associations between the statewide adoption of guided pathways reforms and early student outcomes in Tennessee. The observed improvements in that state are likely the result of concurrent reforms--guided pathways and others--implemented simultaneously, rather than of guided pathways reforms alone. We do not find evidence of improved student outcomes in either Ohio or Washington following the launch of statewide guided pathways initiatives. Our findings suggest that complementarities among adopted practices within and across areas of practice--rather than the adoption of individual practices or the intensity of adoption--seem to drive larger improvements in early academic success across the three states. Our study is the first of its kind to explore the potential of guided pathways reforms in contributing to improved early academic success, representing a significant descriptive contribution given that whole-college reforms in higher education are understudied.

It is now well established that the COVID-19 pandemic had a devastating and unequal impact on student achievement. Test score declines were disproportionately large for historically marginalized students, exacerbating preexisting achievement gaps and threatening educational and economic inequality. In this paper, we use longitudinal student-level NWEA MAP Growth test data to estimate differences in test score declines for students at different points on the prepandemic test distribution. We also test the extent to which students' schools and districts accounted for these differences in declines. We find significant differences in learning loss by baseline achievement, with lower-achieving student's scores dropping 0.100 SD more in math and 0.113 SD more in reading than higher-achieving students' scores. We additionally show that the school a student attended accounts for about three-quarters of this widening gap in math achievement and about one-third in reading. The findings suggest school and district-level policies may have mattered more for learning loss than individual students' experiences within schools and districts. Such nuanced information regarding the variation in the pandemic's impacts on students is critical for policymakers and practitioners designing targeted academic interventions and for tracking disparities in academic recovery. [Additional funding for this report was provided by Kenneth C. Griffin.]

In 1993, the Commonwealth of Massachusetts dramatically overhauled its K-12 education system and created a new school finance formula, building an educational accountability structure to ensure every child has access to a high-quality education. The Massachusetts Education Reform Act (MERA) established academic standards in core subjects, mandated assessments to measure student outcomes on those standards, and established a system for holding schools accountable when students failed to meet basic expectations. This system has helped Massachusetts' public schools become the highest performing in the country. Student outcomes in all tested subjects and across demographic groups have improved steadily over time, but disparities in achievement and attainment exist between the Commonwealth's most privileged students and their less privileged counterparts, many of whom are black or Hispanic. Without the MERA and its requirement to assess every student and publish aggregate academic outcomes, policymakers may not understand the extent of disparity or how to address it as student outcomes data are integral to understanding where Massachusetts' public schools have been, where they are going, and how they can get there. This paper illustrates the importance of the Massachusetts Education Reform Act and how it has positively impacted students over time. It explains why the current accountability system evolved as it did and why preserving the most important aspects of that system is critical if the state is going to fulfill its constitutional obligation to educate all children to a high common standard.

Each year, more than a million high school students nationally take college dual enrollment courses, which have been shown to increase college access and success among participants. Yet racial/ethnic and other equity gaps in dual enrollment participation are widespread. To broaden the benefits of dual enrollment, the state of Ohio passed legislation in 2017 establishing the Innovative Programs (IP) policy, allowing waivers to test-based eligibility requirements--a frequently identified barrier to equitable access--for specific high school-college partnerships providing expanded outreach and support for students underrepresented in the state's dual enrollment program. This paper describes a multiple methods study of IP we conducted to examine how these partnerships were implemented to address the needs of underrepresented students and to evaluate whether the partnerships were successful in broadening access to and success in dual enrollment, as measured by course participation, pass rates, and college matriculation after high school. We find that the IP increased participation in dual enrollment among Black and Hispanic students. And while the implementation of the policy broadened access without changing course outcomes, the impacts on college enrollment after high school were mixed. Our results underscore the importance of pairing increased access to dual enrollment with adequate financial, advising, and academic resources to promote student success in and beyond dual enrollment courses.

In this position paper, the authors lay out the imperative for equitable artificial intelligence (AI), highlighting the essential role of access-oriented institutions and calling on technology companies (both large and small), foundations, and local, state, and federal regulators to consult with the newly convened Complete College America Council on Equitable AI in Higher Education. Their belief is that equitable AI spans far beyond the risk of mis-trained data. How schools adopt or reject these tools, the priorities of AI vendors, access to resources that enable the use of these tools, and the systemic integration of historically underrepresented and underserved voices will shape whether technology amplifies privilege or fosters inclusivity. A three-fold framework is presented for understanding Equity in AI, considering not just the quality and unbiased nature of the data used to train generative AI machines but also who has access to conversations around policy and product, as well as which institutions have access to the resources and safety nets that enable innovation and experimentation in the field of AI. A disruptive new advisory council is proposed, the Complete College America Council on Equitable AI in Higher Education, composed of representatives from historically excluded institutions and, by extension, students. The authors urge policymakers, technologists, and funders to proactively consult the Council and disrupt systemic inequities by integrating AI into higher education rather than continue to perpetuate them. [This paper was created in partnership with T3 Advisory.]

Graduates of special education teacher education programs can teach in a range of special education settings, raising the potential that their training can occur in very different settings than where they find their first jobs. We follow 263 completers of Moderate Disabilities programs in Massachusetts from their field placements to their early-career teaching positions and study the characteristics of their field placements and the degree to which these are aligned with their early-career teaching positions. We also assess the degree to which alignment is associated with early-career teacher turnover. We found that many of these teachers student-taught in an inclusive setting but were hired into a self-contained special education setting and vice versa, and teachers who experienced this misalignment were more likely to leave the workforce early in their careers. Teachers who student taught with a supervising practitioner without a special education license were also more likely to leave early. Findings suggest that teachers training to educate students with learning disabilities should student teach in a setting that is aligned with where they are likely to be hired, and with a supervising practitioner who is trained in special education.

Correctional education is a prevalent form of rehabilitation programming for prisoners in the United States. There is limited evidence, however, about the labor market returns to credentials received while incarcerated. Using incarceration, educational, and labor market data in Washington State, we study the labor market returns to GEDs and short-term vocational certificates earned in prison. We identify the returns to credentials by a difference-indifferences design that compares changes in earnings and employment for incarcerated persons who earn a credential to those who enroll in a program but fail to complete a GED or certificate. We estimate that GEDs increase post-incarceration earnings by about $450 per quarter and that vocational certificates increase earnings by about $250 per quarter. Degree completers have higher hourly wages, are more likely to be employed, and work more hours following release. For vocational programs, earnings increases are driven by certificates in construction and manufacturing. [The research presented presented in this report uses confidential data from the Education Research and Data Center (ERDC) located within the Washington Office of Financial Management (OFM).]

Civic learning is an increasingly salient topic in research, policy, and practice. However, the recent empirical evidence on access to civic learning opportunities is limited. We build on prior research using survey items from the 2018 National Assessment of Educational Progress civics assessment and provide descriptive evidence on disparities in access to three categories of civics content and three evidence-based instructional approaches. We highlight inequalities in opportunities by student characteristics, school characteristics, and state characteristics among a national sample of more than 10,000 8th-grade students enrolled in a course with at least some civics focus (controlling for variation in the extent of civics focus). Our findings conflict with most of the prior evidence regarding disparities in access by race, ethnicity, or socioeconomic background, favoring Black students, Hispanic students, and students of relatively lower socioeconomic backgrounds. This suggests a shift in recent years, potentially due to an increased focus on equity. English learners and students with disabilities also reported greater access than their counterparts. Other findings include inequalities across school types, school location (city students reporting greater opportunity than both rural and suburban students), census region, and state testing policy. Additional findings are presented, and implications and opportunities for future research are discussed.

Emotional, behavioral, and mental health challenges make it difficult for many children and adolescents to engage and succeed at school. Research indicates that at least 20% of all children and adolescents have been diagnosed with one more mental health disorders. Behavioral problems, anxiety, and depression are the most diagnosed mental health issues, and they often co-occur. Moreover, these conditions are being diagnosed at increasingly younger ages. In the past several years there has been a rise in the number of adolescents and young adults with serious mental health issues such as major depression and suicidal ideation, and the COVID-19 pandemic has exacerbated mental health problems for children and adolescents. Schools offer promise for providing intensive clinical support to the most at-risk students, and schools are necessary environment to explore the implementation of multi-modal youth mental health services. This paper provides an analysis of an intensive, in-school mental health services model developed and implemented by Effective School Solutions (ESS), a New Jersey based provider of high acuity school based mental health services for K-12 students. We analyze this multi-modal model for its effectiveness in improving educational outcomes for over 3,000 students identified as requiring intensive clinical mental health support across the 2021-22 school year. This analysis reveals that those students receiving High- versus Low-fidelity programming (i.e., multiple sessions per week for at least half of the school year versus for less than half of the school year) had better educational outcomes. Students receiving High-fidelity programming had greater improvements in grade point average (GPA) and greater reductions in absences across the school year. A higher number of in-school clinical sessions per week significantly predicted a greater increase in GPA and a greater reduction in total disciplinary incidents (including out of school suspensions) across the school year. This report provides initial promising evidence that in-school intensive mental health clinical services yield positive effects on students' educational outcomes. Though future research is needed to validate and extend these findings, schools may consider implementing such services onsite to meet students where they are and to optimize students' mental, behavioral, and educational well-being. [This white paper report was published by the Yale Child Study Center."]

Interest in the field of career and technical education (CTE) has experienced a resurgence over the last decade, as the global economy has grown increasingly competitive while students have continued to leave school underprepared for well-paying twenty-first century jobs. Together and separately, the education and workforce sectors have sought to address these challenges and better prepare students for viable economic futures. The results have been many new, innovative programs at both the secondary and postsecondary education levels that seek to give students technical training for specific careers, general training to prepare them for the workplace, and work-based learning opportunities where they can develop connections to employers and the workforce. While there are still many under-researched areas in CTE, this paper attempts to capture the evidence that has emerged--identifying areas where there is more evidence as well as areas where gaps in evidence still exist. The studies that have been conducted on CTE have demonstrated that it shows promise, but it is imperative to continue building evidence, particularly where there is policy interest and momentum but little data. Doing so will help demonstrate how those programs and models serve students and ensure that the continued scaling up of CTE is supported by a rigorous evidence base. This paper begins with an overview of the issues in the education system and the labor market that have led to the current revival of CTE. It argues that the skills today's employers need are not the ones schools are providing. The paper continues with a description of how various policies have fostered the growth of CTE. In the next section, it provides details on the types of programs and institutions that offer CTE, and the evidence base to support each of them. The paper provides evidence on the effectiveness of CTE at different educational levels, and for specific subgroups, including students with disabilities, and by gender. Further, the paper provides an overview of the available evidence to support different kinds of programs offered at both secondary and postsecondary education levels, touching on the amount of evidence available in each area and the level of rigor used in the studies that generated that evidence. The paper concludes by suggesting that while CTE instruction at the secondary and postsecondary levels could bolster students' economic mobility by helping them gain postsecondary credentials and obtain higher-paying jobs, there are challenges involved in turning that promise into reality. Investments in evidence-based practices can give CTE programs a better chance at success.

The purpose of this technical assistance paper is to assist education leaders and administrators in the consistent implementation of the Florida Career and Professional Education (CAPE) Act in Section 1003.491, Florida Statutes (F.S.). This technical assistance paper addresses questions on recent legislation, funding, and data reporting. [For the previous report, see ED616038.]

The United States has been cultivating STEM talent for decades with great success, but that robust talent pipeline is threatened by a growing STEM teacher shortage. Left unaddressed, that shortage could curtail employment in STEM professions and the many related careers that increasingly require some degree of scientific and technological literacy. While the STEM teacher shortage is a national problem, the effects are likely to be felt more acutely in a state such as Massachusetts with a high concentration of STEM jobs. This paper provides an overview of the causes of the STEM teacher shortage and offers three recommendations to address it.

A large share of Wisconsin's 400+ school districts are rural, and nearly half of all students enrolled in public schools statewide attend rural schools. Persistent challenges faced by rural districts in attracting and retaining educators, however, are contributing to an increasingly dire situation for the state's rural communities. This report summarizes findings from a yearlong research project investigating strategies being used by rural districts to address educator shortages, including a diverse set of "grow your own" (GYO) initiatives which involve school districts working with educator preparation programs (EPPs) and other local partners to identify, recruit, and prepare local candidates to become educators. The mixed-methods research design draws upon two statewide surveys (of rural superintendents and local site coordinators of the Educators Rising GYO program), along with site visits to five rural districts, to accomplish two main goals: (1) to identify strategies being used by rural districts to attract, recruit, and hire teachers; and (2) to describe the landscape of GYO initiatives being utilized by rural Wisconsin districts and local partners to address educator shortages. Broadly speaking, the work suggests that rural school districts are not waiting on state or federal policymakers to solve longstanding staffing challenges for them, but instead are actively engaged in implementing a mix of national GYO programs such as Educators Rising and "homegrown" GYO initiatives as one solution to chronic educator shortages. Not surprisingly, a distinct set of both successes and challenges have emerged as rural districts attempt to implement, launch, sustain, and scale up GYO initiatives that meet their needs. [Paper's original version August 2023; revised version September 2023. This work was made possible by a research grant from UW-Madison's Tommy G. Thompson Center for Public Leadership.]

The United States retains many aspects of a healthy open society, but there are indicators of trouble and deep divisions around the meaning and importance of democratic values. This debate has significant repercussions for universities and their academic communities. In the most-simple terms, there is a red and blue state divide over the role and importance of public institutions, including universities -- red representing largely rural states in which most voters vote Republican and blue being majority Democratic voters, often with one of the two parties having majorities in their respective state legislatures. Then there are so-called purple states in which both parties are vying for dominance, but they are fewer in number. This brief discusses this contemporary dynamic and its implication for higher education and science policy.

This brief discusses cases of neo-nationalist violations of academic freedom in Hungary and Poland. The most prominent case of neo-nationalist violation of academic freedom in Hungary is the fate of the Central European University (CEU). The circumstances of CEU's forced move out of Hungary came before the European Court of Justice regarding it a possible violation of EU law. The Court cited the General Agreement on Trade in Services (GATS) under one of the three pillars of the World Trade Organisation (WTO) 1994 agreement, free trade, and the determination that CEU was a form of international educational services that should not be denied to the people of Hungary. Poland has a similar hostile environment to academics and academic freedom, although with a glimmer of hope following recent elections. The brief also discusses how such open breaches of academic freedom as in Hungary or Poland, in which politicians directly try to exert influence on research institutions and professors, are fortunately rather rare in Germany. However, a confluence of factors perhaps obscures the differences between "academic freedom" and the "freedom of opinion." In Germany, academic freedom includes the search for topics, rigorous methodical investigation, and professional norms to express findings and competent opinions, whereas the free speech is outside of these professional norms. The brief concludes with a discussion of the role of universities and the future of democracy in the context of ensuring a space for free and open debate.

Starting this year, tens of thousands of Russian freshmen found themselves attending a new mandatory course -- "Foundations of Russian Statehood." Swiftly designed under the auspices of Putin's administration, this ideologically charged course aims to position Russia as a unique civilization-state, bolstering Putin's political narrative and providing justification for the full-scale invasion of Ukraine. Consider, for example, this excerpt from the course's instructional video: "The 'Russian world' extends beyond current Russian borders, transcending ethnicities, territories, religions, political systems, and ideological preferences." As this curriculum becomes standard in Russian universities, it contributes to the emerging trend of weaponizing Russian universities and turning them into instruments in Russia's war of attrition with Ukraine and its broader stand-off with the West. This report discusses this weaponization process and the impact it is having on Russian universities, faculty, students, and the academic communities they belong to. It is regrettably a story of back to the future, reminiscent of the Soviet era of repression and attempts at control and manipulation of academics.

Since 2020, the federal government has distributed almost $2 billion to American school districts to mitigate the effects of COVID-19 pandemic school closures. Released in three separate tranches through the Elementary and Secondary School Emergency Relief (ESSER) fund, the Trump and Biden administrations outlined parameters for spending this unprecedented amount of money and ensured that most of it went directly to local school districts, with a comparative sliver going to state education agencies. This report gives an overview of the most common ESSER expenditures to date and assesses the factors that might influence whether district ESSER investments will make a difference for students. It also provides recommendations for how stakeholders should evaluate the ESSER experiment once it officially ends.

China's higher-education system has been shaped in recent years by a trio of factors: the COVID-19 pandemic, the ambitions of Chinese leader Xi Jinping to make his country into an innovation superpower that is loyal to the Communist Party, and western alarm about those ambitions. But a fourth development, the slowing of China's formerly super-charged economy, could play a more prominent role going forward. This article examines these four factors.

Since establishing its first campus in 1868, the University of California (UC), California's land-grant university, developed into the nation's first multi-campus system in the United States, and is today widely recognized as the world's premier network of public research universities. This short essay provides an historical brief on the role that shared governance, and specifically the role of the Academic Senate, played in creating an academic culture of excellence and high achievement in pursuing its tripartite mission of teaching and learning, research and knowledge production, and public service. A key component in understanding the critical role of the Senate in UC's evolution from a single campus in Berkeley to now a ten-campus system is the university's unusual designation as a public trust in the state constitution that, beginning in 1879, protected the university at critical times from external political pressures and allowed the university to develop an internal academic culture guided by the Academic Senate. By the 1920s, the emergence of California's unique and innovative public system of higher education, with UC as the sole public provider of doctoral degrees and state funded research, also helps explain the ability of the UC system to maintain its mission and formulate what is termed a "One University" model. The Academic Senate has created coherency and shared values within UC, and a culture and expectation for faculty performance that is unique among universities around the world. This essay also offers a brief reflection on the Academic Senate's past influence, its current status, and prospective role. The overall intent is to provide context for the current academic community and higher education scholars regarding the past and future role of faculty in university governance and management, and what distinguishes UC in the pantheon of major research universities.

Following up on an individual-level randomized controlled trial of a Dana Center Math Pathways (DCMP) model, this study assessed longer-term impacts on students' math completion, academic progress, and academic attainment. The version of the DCMP that was assessed in this study diversified the developmental and college-level math course content that students take, separating it into distinct pathways that better aligned with their career interests. It also streamlined developmental math sequences into a one-semester developmental course for all students, regardless of placement level, and implemented evidence-based curricula and pedagogy to engage students in active problem solving that was pertinent to real-life situations. The study, which followed 1,411 students from four Texas community colleges and ten campuses, found that, in the five years after random assignment, program group students were consistently more likely to successfully complete their first college-level math courses than control group students. The study did not find impacts after five years on the number of overall college credits that students accrued or on the likelihood that students attained a credential or transferred to a four-year college.

Multiple measures assessment (MMA) has gained considerable momentum over the past decade as an alternative to traditional test-based procedures for placing incoming students into developmental or college-level coursework in math and English at broad-access colleges. Compared to standardized tests, which measure student performance at a single point in time, MMA (which often emphasizes high school GPA as a measure) provides a more holistic picture of students' academic preparation. Despite positive impacts on student outcomes that have been found by recent research on MMA, questions remain about whether the positive effects of MMA are sustained over time. This study--a follow-up to prior research using the same sample of students--employs a randomized controlled trial to investigate whether algorithmic MMA placement used at seven State University of New York (SUNY) community colleges led to better student outcomes, for up to four and a half years after randomization, than a system based on test scores alone. Nearly 13,000 incoming students who arrived at the seven colleges in fall 2016, spring 2017, and fall 2017 took placement tests and were randomly assigned to be placed using either the status quo method (business-as-usual group) or the alternative, algorithmic MMA method (program group). Using this sample, we estimate the overall treatment effects on placement into, enrollment in, and completion of college-level math and English as well as effects on other outcomes. We conduct similar analyses on race/ethnicity, Pell recipient status, and gender subgroups. We also descriptively examine the proportion of program group students who were bumped up (i.e., their placement changed from a developmental course placement to a college-level course placement) and bumped down (i.e., their placement changed from a college-level course placement to a developmental course placement) by the MMA algorithm, and we perform a cost-effectiveness analysis. We find that the MMA method used at the colleges improved access to and success in college-level courses and that lower cut scores in English rather than math are associated with larger and longer lasting impacts on completion of college-level coursework. While MMA improved outcomes among student subgroups, it had little to no impact on gaps in outcomes between subgroups. We also find that bumped-up students had substantially better outcomes in both math and English, while bumped-down students had substantially worse outcomes. Our results suggest that increased access to college-level courses is the driving factor in the positive outcomes experienced by program group students and that placement into standalone developmental courses can have detrimental effects on student outcomes. In the discussion of the study's results, we make recommendations for adopting MMA at colleges. Implemented together with other initiatives to support students, MMA can be a first step on the path to success for incoming students. [For the accompanying research brief, see ED632528.]

The growth of homeschooling from a somewhat fringe movement during the 1970s and 1980s to a more widespread and socially accepted approach in recent decades has provided a strong foundation of flexible learning models. When Florida's school choice expansion, House Bill 1, was introduced in January 2023, one of its goals was to allow more homeschoolers to participate in the education savings account (ESA) program. But many homeschoolers and homeschool advocacy groups balked at the proposal. Having worked hard to achieve independence, they were loath to be linked to government funding. Even if the program were optional, they feared that associated regulations would eventually extend to traditional homeschoolers. In the end, the bill passed with new language that satisfied traditional homeschoolers by creating a new option for parents to educate their children at home. As states continue to enact and expand education savings accounts, Colleen Hroncich argues that it is crucial that policymakers craft bills in a way that maximizes freedom and minimizes roadblocks.

This study investigates racial disparities in school closures both within California and nationally. Findings highlight an alarming pattern: Schools enrolling higher proportions of Black students are at significantly increased risk of closure relative to those enrolling fewer Black students, a pattern that is more pronounced in California than elsewhere in the United States. This study also finds that conventional explanations for school closures--such as declining enrollments, poverty rates, and achievement differences--cannot fully account for why schools enrolling larger shares of Black students have greater odds of closure. These findings underscore that school closures in California and elsewhere reflect racial inequalities that require adequate policymaking to ensure equitable and fair school-closure proceedings

We investigate the impact of attending a free school in England -- that is, a new start-up school that enjoys considerable autonomy while remaining in the state sector. We analyse the effects of two secondary free schools with different teaching philosophies: one follows a 'no excuse' paradigm, while the other one adopts a 'classical liberal', knowledge-rich approach. We establish causal effects exploiting admission lotteries and a distance-based regression discontinuity design. Both schools have a strong positive impact on student test scores on average. However, we also find heterogeneous effects: the 'no excuse' school mostly benefits boys, while the 'classical liberal' school mainly benefits White British and non-poor students. Both schools similarly reduce student absences and school mobility. Peer quality, teacher characteristics, and inspectorate ratings cannot fully explain the schools' effectiveness. Instead, a quantitative text analysis of the schools' 'vision and ethos' statements shows that the 'no excuse' and 'classical liberal' philosophies adopted by the two free schools clearly set them apart from the counterfactual schools where rejected applicants enrol, and likely explain their heterogeneous effects.

With the rapid growth of scholarly archives, researchers subscribe to "paper alert" systems that periodically provide them with recommendations of recently published papers that are similar to previously collected papers. However, researchers sometimes struggle to make sense of nuanced connections between recommended papers and their own research context, as existing systems only present paper titles and abstracts. To help researchers spot these connections, we present PaperWeaver, an enriched paper alerts system that provides contextualized text descriptions of recommended papers based on user-collected papers. PaperWeaver employs a computational method based on Large Language Models (LLMs) to infer users' research interests from their collected papers, extract context-specific aspects of papers, and compare recommended and collected papers on these aspects. Our user study (N=15) showed that participants using PaperWeaver were able to better understand the relevance of recommended papers and triage them more confidently when compared to a baseline that presented the related work sections from recommended papers., Comment: Accepted to CHI 2024

This occasional paper is the ninth in a series on the National Quality Framework (NQF). The Education and Care Services National Law and National Regulations govern the minimum standards and requirements that all providers of NQF regulated services must meet, including health and safety requirements. Quality Area 2 of the National Quality Standard (NQS) upholds children's right to be protected and kept safe. The approved provider, nominated supervisors, coordinators and educators have responsibility for supporting the health, safety and wellbeing of all children. In exercising their responsibilities, they must take reasonable care to protect children from foreseeable risk of harm, injury and infection. This paper uses data from the National Quality Agenda Information Technology System (NQA ITS) to provide analysis of trends in serious incidents resulting in injury, trauma or illness in Australian children's education and care services between 1 April 2020 and 31 March 2022.

The iterative character of work in machine learning (ML) and artificial intelligence (AI) and reliance on comparisons against benchmark datasets emphasize the importance of reproducibility in that literature. Yet, resource constraints and inadequate documentation can make running replications particularly challenging. Our work explores the potential of using downstream citation contexts as a signal of reproducibility. We introduce a sentiment analysis framework applied to citation contexts from papers involved in Machine Learning Reproducibility Challenges in order to interpret the positive or negative outcomes of reproduction attempts. Our contributions include training classifiers for reproducibility-related contexts and sentiment analysis, and exploring correlations between citation context sentiment and reproducibility scores. Study data, software, and an artifact appendix are publicly available at https://github.com/lamps-lab/ccair-ai-reproducibility ., Comment: 9 pages, 4 figures

Reading scientific research papers is a skill that many students do not learn before entering PhD programs, but it is critical to their success. This paper describes our diagramming technique for teaching this skill, which helps them identify the structure and the scientific argument of the paper. This has made our students more effective readers., Comment: 6 pages, 6 figures, working paper

Current research on defending against adversarial examples focuses primarily on achieving robustness against a single attack type such as $\ell_2$ or $\ell_{\infty}$-bounded attacks. However, the space of possible perturbations is much larger and currently cannot be modeled by a single attack type. The discrepancy between the focus of current defenses and the space of attacks of interest calls to question the practicality of existing defenses and the reliability of their evaluation. In this position paper, we argue that the research community should look beyond single attack robustness, and we draw attention to three potential directions involving robustness against multiple attacks: simultaneous multiattack robustness, unforeseen attack robustness, and a newly defined problem setting which we call continual adaptive robustness. We provide a unified framework which rigorously defines these problem settings, synthesize existing research in these fields, and outline open directions. We hope that our position paper inspires more research in simultaneous multiattack, unforeseen attack, and continual adaptive robustness.

As AI promises to accelerate scientific discovery, it remains unclear whether fully AI-driven research is possible and whether it can adhere to key scientific values, such as transparency, traceability and verifiability. Mimicking human scientific practices, we built data-to-paper, an automation platform that guides interacting LLM agents through a complete stepwise research process, while programmatically back-tracing information flow and allowing human oversight and interactions. In autopilot mode, provided with annotated data alone, data-to-paper raised hypotheses, designed research plans, wrote and debugged analysis codes, generated and interpreted results, and created complete and information-traceable research papers. Even though research novelty was relatively limited, the process demonstrated autonomous generation of de novo quantitative insights from data. For simple research goals, a fully-autonomous cycle can create manuscripts which recapitulate peer-reviewed publications without major errors in about 80-90%, yet as goal complexity increases, human co-piloting becomes critical for assuring accuracy. Beyond the process itself, created manuscripts too are inherently verifiable, as information-tracing allows to programmatically chain results, methods and data. Our work thereby demonstrates a potential for AI-driven acceleration of scientific discovery while enhancing, rather than jeopardizing, traceability, transparency and verifiability.

Citation text plays a pivotal role in elucidating the connection between scientific documents, demanding an in-depth comprehension of the cited paper. Constructing citations is often time-consuming, requiring researchers to delve into extensive literature and grapple with articulating relevant content. To address this challenge, the field of citation text generation (CTG) has emerged. However, while earlier methods have primarily centered on creating single-sentence citations, practical scenarios frequently necessitate citing multiple papers within a single paragraph. To bridge this gap, we propose a method that leverages Large Language Models (LLMs) to generate multi-citation sentences. Our approach involves a single source paper and a collection of target papers, culminating in a coherent paragraph containing multi-sentence citation text. Furthermore, we introduce a curated dataset named MCG-S2ORC, composed of English-language academic research papers in Computer Science, showcasing multiple citation instances. In our experiments, we evaluate three LLMs LLaMA, Alpaca, and Vicuna to ascertain the most effective model for this endeavor. Additionally, we exhibit enhanced performance by integrating knowledge graphs from target papers into the prompts for generating citation text. This research underscores the potential of harnessing LLMs for citation generation, opening a compelling avenue for exploring the intricate connections between scientific documents., Comment: 14 pages, 7 figures, 11th International Conference, BDA 2023, Delhi, India

For undergraduate students of computing, learning to solve complex practical problems in a team is an essential skill for their future careers. This skill is needed in various fields, such as in cybersecurity and IT governance. Tabletop exercises are an innovative teaching method used in practice for training teams in incident response and evaluation of contingency plans. However, tabletop exercises are not yet widely established in university education. This paper presents data and teaching experience from a cybersecurity course that introduces tabletop exercises in classrooms using a novel technology: INJECT Exercise Platform (IXP), a web-based learning environment for delivering and evaluating the exercises. This technology substantially improves the prior practice, since tabletop exercises worldwide have usually been conducted using pen and paper. Unlike in traditional tabletop exercises, which are difficult to evaluate manually, IXP provides insights into students' behavior and learning based on automated analysis of interaction data. We demonstrate IXP's capabilities and evolution by comparing exercise sessions hosted throughout three years at different stages of the platform's readiness. The analysis of student data is supplemented by the discussion of the lessons learned from employing IXP in computing education contexts. The data analytics enabled a detailed comparison of the teams' performance and behavior. Instructors who consider innovating their classes with tabletop exercises may use IXP and benefit from the insights in this paper., Comment: Published in ACM ITiCSE 2024 conference proceedings, see https://doi.org/10.1145/3649217.3653639

In the paper [3] the value of the invariant denoted by d and often called "rank" was misstated for a narrow class of examples of ECS manifolds: they were identified as having d = 1 instead of the correct value d = 2. The error was repeated, by citing [3], in [1], [2] and [4] -- [7]. We briefly describe the class in question, explain why it has d = 2, and list the required corrections of the affected papers., Comment: 4 pages, to be included in a future paper as an appendix

This paper introduces a new method of paper circuit fabrication that overcomes design barriers and increases flexibility in circuit design. Conventional circuit boards rely on thin traces, which limits the complexity and accuracy when applied to paper circuits. To address this issue, we propose a method that uses large conductive zones in paper circuits and performs subtractive processing during their fabrication. This approach eliminates design barriers and allows for more flexibility in circuit design. We introduce PaperCAD, a software tool that simplifies the design process by converting traditional circuit design to paper circuit design. We demonstrate our technique by creating two paper circuit boards. Our approach has the potential to promote the development of new applications for paper circuits., Comment: ACM CHI 2023 Workshop 02: Beyond Prototyping Boards: Future Paradigms for Electronics Toolkits

This paper explores the implementation of embedded magnets to enhance paper-based interactions. The integration of magnets in paper-based interactions simplifies the fabrication process, making it more accessible for building soft robotics systems. We discuss various interaction patterns achievable through this approach and highlight their potential applications., Comment: ACM DIS 2023 Workshop 02: Soft Robotics and Programmable Materials for Human-Computer Interaction

Behavioural types provide a promising way to achieve lightweight, language-integrated verification for communication-centric software. However, a large barrier to the adoption of behavioural types is that the current state of the art expects software to be written using the same tools and typing discipline throughout a system, and has little support for components over which a developer has no control. This position paper describes the outcomes of a working group discussion at Dagstuhl Seminar 24051 (Next-Generation Protocols for Heterogeneous Systems). We propose a methodology for integrating multiple behaviourally-typed components, written in different languages. Our proposed approach involves an extensible protocol description language, a session IR that can describe data transformations and boundary monitoring and which can be compiled into program-specific session proxies, and finally a session middleware to aid session establishment. We hope that this position paper will stimulate discussion on one of the most pressing challenges facing the widespread adoption of behavioural typing., Comment: In Proceedings PLACES 2024, arXiv:2404.03712

Context: Annually, ICSE acknowledges a range of papers, a subset of which are paired with research artifacts such as source code, datasets, and supplementary materials, adhering to the Open Science Policy. However, no prior systematic inquiry dives into gauging the influence of ICSE papers using artifact attributes. Objective: We explore the mutual impact between artifacts and their associated papers presented at ICSE over ten years. Method: We collect data on usage attributes from papers and their artifacts, conduct a statistical assessment to identify differences, and analyze the top five papers in each attribute category. Results: There is a significant difference between paper citations and the usage of associated artifacts. While statistical analyses show no notable difference between paper citations and GitHub stars, variations exist in views and/or downloads of papers and artifacts. Conclusion: We provide a thorough overview of ICSE's accepted papers from the last decade, emphasizing the intricate relationship between research papers and their artifacts. To enhance the assessment of artifact influence in software research, we recommend considering key attributes that may be present in one platform but not in another., Comment: This paper has been accepted for publication and presentation at The 22nd IEEE/ACIS International Conference on Software Engineering, Management and Applications (SERA 2024) to be held in Honolulu, USA on May 30-June 1, 2024

This paper presents and evaluates a set of methods to classify individual Scopus publications using their references back to the second generation, where each publication can be assigned fractionally into up to five ASJC (All Science Journal Classifications) categories, excluding the Multidisciplinary area and the miscellaneous categories. Based on proposals by Glanzel et al. (1999a, 1999b, 2021), some additional parameters are established that allow different results to be obtained depending on how category membership is weighted or how the acceptance thresholds for multiple assignments are established. Various classifications are obtained, and then compared with each other, with the original ASJC Scopus journal classification, and with the AAC (Authors Assignation Collection) classification of a previous study (Alvarez-Llorente et al., 2023) in which the papers corresponding authors assign them the most appropriate categories. Classifications in which a high threshold is set for allowing assignments to multiple categories, combined with the use of first- and second-generation references and averaging over the number of references, provide the most promising results, improving over other reference-based reclassification proposals in terms of granularity, and over the Scopus classification itself in such aspects as the homogeneity of the publications assigned to a category. They also show greater coincidence with the AAC classification.

This white paper presents the current status of the R&D for Analysis Facilities (AFs) and attempts to summarize the views on the future direction of these facilities. These views have been collected through the High Energy Physics (HEP) Software Foundation's (HSF) Analysis Facilities forum, established in March 2022, the Analysis Ecosystems II workshop, that took place in May 2022, and the WLCG/HSF pre-CHEP workshop, that took place in May 2023. The paper attempts to cover all the aspects of an analysis facility.

This paper explored the temporal characteristics of clinical citations of biomedical papers, including how long it takes to receive its first clinical citation (the initial stage) and how long it takes to receive two or more clinical citations after its first clinical citation (the build-up stage). Over 23 million biomedical papers in PubMed between 1940 and 2013 and their clinical citations are used as the research data. We divide these biomedical papers into three groups and four categories from clinical citation level and translational science perspectives. We compare the temporal characteristics of biomedical papers of different groups or categories. From the perspective of clinical citation level, the results show that highly clinically cited papers had obvious advantages of receiving clinical citations over medium and lowly clinically cited papers in both the initial and build-up stages. Meanwhile, as the number of clinical citations increased in the build-up stage, the difference in the length of time to receive the corresponding number of clinical citations among the three groups of biomedical papers significantly increased. From the perspective of translational science, the results reveal that biomedical papers closer to clinical science more easily receive clinical citations than papers closer to basic science in both the initial and build-up stages. Moreover, we found that highly clinically cited papers had the desperate advantage of receiving clinical citations over even the clinical guidelines or clinical trials. The robustness analysis of the two aspects demonstrates the reliability of our results.

Scientific publishing lays the foundation of science by disseminating research findings, fostering collaboration, encouraging reproducibility, and ensuring that scientific knowledge is accessible, verifiable, and built upon over time. Recently, there has been immense speculation about how many people are using large language models (LLMs) like ChatGPT in their academic writing, and to what extent this tool might have an effect on global scientific practices. However, we lack a precise measure of the proportion of academic writing substantially modified or produced by LLMs. To address this gap, we conduct the first systematic, large-scale analysis across 950,965 papers published between January 2020 and February 2024 on the arXiv, bioRxiv, and Nature portfolio journals, using a population-level statistical framework to measure the prevalence of LLM-modified content over time. Our statistical estimation operates on the corpus level and is more robust than inference on individual instances. Our findings reveal a steady increase in LLM usage, with the largest and fastest growth observed in Computer Science papers (up to 17.5%). In comparison, Mathematics papers and the Nature portfolio showed the least LLM modification (up to 6.3%). Moreover, at an aggregate level, our analysis reveals that higher levels of LLM-modification are associated with papers whose first authors post preprints more frequently, papers in more crowded research areas, and papers of shorter lengths. Our findings suggests that LLMs are being broadly used in scientific writings.

In 2024, we will hold a research paper competition (the third Human Understanding AI Paper Challenge) for the research and development of artificial intelligence technologies to understand human daily life. This document introduces the datasets that will be provided to participants in the competition, and summarizes the issues to consider in data processing and learning model development., Comment: 7 pages, 3 figures

Examining limitations is a crucial step in the scholarly research reviewing process, revealing aspects where a study might lack decisiveness or require enhancement. This aids readers in considering broader implications for further research. In this article, we present a novel and challenging task of Suggestive Limitation Generation (SLG) for research papers. We compile a dataset called LimGen, encompassing 4068 research papers and their associated limitations from the ACL anthology. We investigate several approaches to harness large language models (LLMs) for producing suggestive limitations, by thoroughly examining the related challenges, practical insights, and potential opportunities. Our LimGen dataset and code can be accessed at https://github.com/armbf/LimGen., Comment: 16 pages, 3 figures

This non-archival index is not complete, as some accepted papers chose to opt-out of inclusion. The list of all accepted papers is available on the workshop website.