Search

Despite an abundance of proposed systems, the verification of units-of-measure within programs remains rare in scientific computing. We attempt to address this issue by providing a lightweight static verification system for units-of-measure in Fortran programs which supports incremental annotation of large projects. We take the opposite approach to the most mainstream existing deployment of units-of-measure typing (in F#) and generate a global, rather than local, constraints system for a program. We show that such a system can infer (and check) polymorphic units specifications for under-determined parts of the program. Not only does this ability allow checking of partially annotated programs but it also allows the global constraint problem to be partitioned. This partitioning means we can scale to large programs by solving constraints for each program module independently and storing inferred units at module boundaries (separate verification). We provide an implementation of our approach as an extension to an open-source Fortran analysis tool., Comment: Please note that this article was written in 2018 and has been relocated to arXiv simply for archival purposes

Neural code synthesis has reached a point where snippet generation is accurate enough to be considered for integration into human software development workflows. Commercial products aim to increase programmers' productivity, without being able to measure it directly. In this case study, we asked users of GitHub Copilot about its impact on their productivity, and sought to find a reflection of their perception in directly measurable user data. We find that the rate with which shown suggestions are accepted, rather than more specific metrics regarding the persistence of completions in the code over time, drives developers' perception of productivity., Comment: To appear in: Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming (MAPS '22), June 13, 2022

This brief applies value-added models to student surveys in the CORE Districts to explore whether social-emotional learning (SEL) surveys can be used to measure effective classroom-level supports for SEL. The authors find that classrooms differ in their effect on students' growth in self-reported SEL--even after accounting for school-level effects. Results suggest that classroom-level effects within schools may be larger than school-level effects. However, the low explanatory power of the SEL models means it is unclear that these are causal effects that have appropriately controlled for student-level characteristics. Finally, there are generally low correlations between classroom-level growth in SEL and classroom-level growth in English language arts (ELA) or math, suggesting the SEL measures may capture growth not measured by academic test scores. Although results are preliminary, they indicate there might be measurable student growth in SEL impacted by the environment of classrooms within schools. [For the full working paper, see "Can We Measure Classroom Supports for Social-Emotional Learning? Applying Value-Added Models to Student Surveys in the CORE Districts. Working Paper" (ED600451).]

This executive summary describes the larger study, "Teacher Preparation Experiences and Early Teaching Effectiveness. NCEE 2019-4007" (ED598664). This study provides information about new teachers' preparation experiences and explores whether particular types of experiences are related to teachers' effectiveness in improving their students' test scores. Understanding whether certain ways of preparing teachers are more effective than others is important, but very little is currently known about what teacher preparation emphasizes at a fine-grained level and how those experiences are related to teaching effectiveness. Research identifying which preparation experiences are related to improved student performance can generate hypotheses for improving teacher preparation. Through an online survey administered in spring 2015, this study collected information from new teachers about their preparation experiences. It also examined the relationship between the frequency of these experiences and teachers' effectiveness in improving student test scores. The summary describes how the study was conducted, key findings, and concluding thoughts.

This report examines the frequency of particular teacher preparation experiences and explores their relationships to beginning teachers' effectiveness in improving student test scores once they get to the classroom. The report found both differences in how teachers prepare for their certification in the field and that certain experiences are related to improving test scores in the classroom. The findings provide a detailed look into current teacher preparation practices and identify potential avenues for improving them. [For the appendices, see ED598665. For the executive summary, see ED598666. For the study highlights, see ED598667.]

The 13 appendices in this publication accompany the full report, "Teacher Preparation Experiences and Early Teaching Effectiveness. NCEE 2019-4007" (ED598664). They include: (1) Teacher Survey Response Rates by District/State; (2) Comparison of the Subsample Used for Relational Analyses and the Full Sample; (3) Development of the Teacher Survey; (4) Teacher Survey Questions on Preparation Experiences; (5) Measures of Preparation Experiences; (6) Teachers' Assessment of the Usefulness of Preparation Experiences; (7) Factor Analysis Methodology; (8) Assessing Measurement Error in Teacher Survey Responses; (9) Value-Added Modeling Methodology; (10) Teachers' Preparation Experiences by Preparation Provider and Degree Program; (11) Robustness of Results to Alternative Statistical Models; (12) Subgroup Analyses of Relational Findings; and (13) Relationships between Preparation Experiences and Effectiveness by Year of Teaching.

School value-added models are increasingly used to measure schools' contributions to student success. At the same time, policymakers and researchers agree that schools should support students' socialemotional learning (SEL) as well as academic development. Yet, the evidence regarding whether schools can influence SEL and whether statistical growth models can appropriately measure this influence is limited. Recent work shows meaningful differences across schools in changes in SEL scores by grade (Loeb, Christian, Hough, Meyer, Rice, & West, 2019), but whether these differences represent the effects of schools is still unclear. The current paper builds upon this earlier work by examining the stability of the estimated school-by-grade effects on SEL across two years, using a large-scale SEL survey administered in California's CORE districts. We find that correlations among school effects in the same grades across different years are positive, but they are lower than those for math and English Language Arts (ELA). Schools in the top or the bottom of the school effect distribution are more persistent in their impacts across years than those in the middle of the distribution. Overall, the results provide evidence that these school effects measure real contributions to SEL. However, the low stability of effects from one year to the next draw into question whether including these school value-added measures of self-reported SEL in school performance frameworks and systems would be beneficial.

This extended abstract reports on previous work of the CamFort project in which we developed an external units-of-measure type system for Fortran code, targeted at scientists. Our approach can guide the programmer in adding specifications (type annotations) to existing code, with the aim of easing adoption on legacy code. Pertinent to the topics of the HATRA workshop, we discuss the human-aspects of the tool here. CamFort is open-source and freely available online., Comment: Presented at HATRA 2020 (Workshop on Human Aspects of Types and Reasoning Assistants) colocated with SPLASH 2020

The forgetting curve has been extensively explored by psychologists, educationalists and cognitive scientists alike. In the context of Intelligent Tutoring Systems, modelling the forgetting curve for each user and knowledge component (e.g. vocabulary word) should enable us to develop optimal revision strategies that counteract memory decay and ensure long-term retention. In this study we explore a variety of forgetting curve models incorporating psychological and linguistic features, and we use these models to predict the probability of word recall by learners of English as a second language. We evaluate the impact of the models and their features using data from an online vocabulary teaching platform and find that word complexity is a highly informative feature which may be successfully learned by a neural network model., Comment: Artificial Intelligence for Education 2020 (AIED)

Mounting evidence demonstrates that social-emotional skills are important for students' academic and life success, yet we have limited evidence on how these skills develop over time and how this development varies across student subgroups. In this study, we use the first large-scale panel survey of social-emotional learning (SEL) to describe how four SEL constructs--growth-mindset, self-efficacy, self-management, and social awareness--develop from Grade 4 to Grade 12, and how these trends vary by gender, socioeconomic status, and race/ethnicity. Our results are based on self-report student surveys administered in the 2014-15 and 2015-16 school years to roughly 400,000 students within the CORE districts, a network of urban school districts in California. We simulate cohort trends for students continuously enrolled in six districts within CORE by adding the mean change at each grade level observed among students surveyed in both years to the mean scores of Grade 8 students in 2015-16. We find that, with the exception of growth mindset, social-emotional skills do not increase monotonically as students move through school. Self-efficacy, social awareness, and to a lesser degree self-management decrease after Grade 6. Trends in SEL also vary by subgroup. Female students report higher self-management and social awareness than boys, but their self-efficacy drops sharply relative to boys in middle and high school. Economically disadvantaged students report lower levels of each social-emotional construct, but gaps in self-management, growth mindset, and self-efficacy narrow in high school. White students report higher levels of social-emotional skills than African American and Latinx students; Asian students report levels of self-management similar to White students but exhibit declining self-efficacy over time. While our findings cannot be generalized beyond the California districts we study, the scope and scale of our data far exceeds anything in the extant literature. [For a related report, see ED591084.]

Measures of school-level growth in student outcomes are common tools used to assess the impacts of schools. The vast majority of these measures are based on standardized tests, even though emerging evidence demonstrates the importance of social-emotional skills (SEL). This paper uses the first large-scale panel surveys of students on SEL to produce and evaluate school-level value-added measures by grade for growth mindset, self-efficacy, self-management, and social awareness. We find substantive differences across schools in SEL growth, of magnitudes similar to those for academic achievement. This result suggests that schools might contribute to students' SEL. However, we also find that the models are not as well specified for SEL as they are for achievement gains, raising the possibility that the estimated school effects include school-level measurement error and potential omitted variables bias. In addition, the across-school variance in the average level of the SEL measures is proportionally much smaller than for academic measures, which would not be expected if substantial impacts of schools on SEL outcomes persisted over time. These findings recommend caution in interpreting measures as the causal impacts of schools on SEL, though they also do not rule out important school effects.

With an increased appreciation of students' social-emotional skills among researchers and policy makers, many states and school districts are moving toward a systematic process to measure Social-Emotional Learning (SEL). In this study, we examine the measurement properties of California's CORE Districts' SEL survey administered to over 400,000 students in grades 3 to 12 during the 2015-16 school year. We conduct analyses through both classical test theory and item response theory frameworks, applying three different polytomous IRT models on both the full student sample and on separate samples from each grade. From these analyses, we summarize the psychometric properties of items at each grade level, compare items' functionality across grades, compare student outcomes from IRT models and the classical approach, make suggestions on approaches to modeling and scaling the SEL survey data, and identify items, by grade, that do not contribute positively to measurement of each outcome. Finally, we discuss policy implications in using SEL measures among educators, administrators, policy makers, and other stakeholders.

Distributed analytics engines such as Spark are a common choice for processing extremely large datasets. However, finding good configurations for these systems remains challenging, with each workload potentially requiring a different setup to run optimally. Using suboptimal configurations incurs significant extra runtime costs. %Furthermore, Spark and similar platforms are gaining traction within data-scientists communities where awareness of such issues is relatively low. We propose Tuneful, an approach that efficiently tunes the configuration of in-memory cluster computing systems. Tuneful combines incremental Sensitivity Analysis and Bayesian optimization to identify near-optimal configurations from a high-dimensional search space, using a small number of executions. This setup allows the tuning to be done online, without any previous training. Our experimental results show that Tuneful reduces the search time for finding close-to-optimal configurations by 62\% (at the median) when compared to existing state-of-the-art techniques. This means that the amortization of the tuning cost happens significantly faster, enabling practical tuning for new classes of workloads.

ReLU is widely seen as the default choice for activation functions in neural networks. However, there are cases where more complicated functions are required. In particular, recurrent neural networks (such as LSTMs) make extensive use of both hyperbolic tangent and sigmoid functions. These functions are expensive to compute. We used function approximation techniques to develop replacements for these functions and evaluated them empirically on three popular network configurations. We find safe approximations that yield a 10% to 37% improvement in training times on the CPU. These approximations were suitable for all cases we considered and we believe are appropriate replacements for all networks using these activation functions. We also develop ranged approximations which only apply in some cases due to restrictions on their input domain. Our ranged approximations yield a performance improvement of 20% to 53% in network training time. Our functions also match or considerably out perform the ad-hoc approximations used in Theano and the implementation of Word2Vec., Comment: 10 Pages, 5 Figures, 1 Table

Professional software developers spend a significant amount of time fixing builds, but this has received little attention as a problem in automatic program repair. We present a new deep learning architecture, called Graph2Diff, for automatically localizing and fixing build errors. We represent source code, build configuration files, and compiler diagnostic messages as a graph, and then use a Graph Neural Network model to predict a diff. A diff specifies how to modify the code's abstract syntax tree, represented in the neural network as a sequence of tokens and of pointers to code locations. Our network is an instance of a more general abstraction that we call Graph2Tocopo, which is potentially useful in any development tool for predicting source code changes. We evaluate the model on a dataset of over 500k real build errors and their resolutions from professional developers. Compared to the approach of DeepDelta (Mesbah et al., 2019), our approach tackles the harder task of predicting a more precise diff but still achieves over double the accuracy., Comment: Submitted for review on Aug 23, 2019

The purpose of this report is to present the results of a two-year randomized control trial evaluation of the SPARK literacy program. SPARK is an early grade literacy program developed by Boys & Girls Clubs of Greater Milwaukee. In 2010, SPARK was awarded an Investing in Innovation (i3) Department of Education grant to further develop the program and test its impact in seven Milwaukee Public Schools (MPS). The evaluation used a randomized control trial selection to test the impact of SPARK across three domains: reading achievement, literacy, and school attendance. Informed consent was obtained from 576 parents for their students to participate in the study. A random sample of kindergarten, 1st, and 2nd grade students in seven MPS schools was selected in October and November of 2013 to participate. 286 students were randomly selected as SPARK participants and 290 students were randomly selected as control students. Stratification was done by school and grade level within school. The specific number of students selected to receive SPARK within each strata was determined both by the number of consented students and the capacity to serve students within each site. Students with a reading-related IEP or who were English Language Learners were not eligible to participate in the evaluation but were eligible to receive tutoring. All other students were eligible to participate. The results suggest that SPARK had statistically significant, positive impacts on reading achievement, literacy, and regular school day attendance. Tables are appended. [SREE documents are structured abstracts of SREE conference symposium, panel, and paper or poster submissions.]

Student growth measures are increasingly common in K-12 school and teacher accountability policy in the United States. At this time, it is still rare for educator preparation programs (EPPs) to be evaluated by similar measures for either low stakes program improvement or higher stakes accreditation processes. Growth in student achievement reflects the contributions of multiple people and multiple entities, including EPPs, via their roles in selecting educators into the profession and providing candidates with education and training. This paper presents a partial list of technical considerations that researchers, policy makers, and, ultimately, EPPs must address when trying to measure or manage performance. We identify two categories of considerations: those that affect the calculation of teacher value added, and those that affect the aggregation of teacher estimates to the EPP level. Consistent with the current literature, we distinguish the effects of preparation programs on teacher performance from the effects of selection into programs and into districts/schools. One of the most perplexing conclusions of several recent papers is the lack of correlation between teachers' training institution and on-the-job performance. We demonstrate how this result can be generated through a teacher-school matching process under certain labor market conditions. Support for this research was provided by the Council for the Accreditation of Educator Preparation.