Search

The objective of this study was to collect online information about COVID-19 mask mandates in United States (U.S.) school districts using artificial intelligence to verify and contribute to official statistics collected by the U.S. Department of Education (ED) and linking to data reported by U.S. states. To collect the data, the authors developed a customized web scraping tool using the Python package BeautifulSoup to automate Google searches of mask mandates in school districts in the U.S. These automated searches retrieved a pre-determined number of top search results in a tabular format. Next, the authors developed Natural Language Processing (NLP) code that read the search results and classified which school districts had implemented a mask mandate. This classification was achieved by developing and training a supervised machine learning algorithm using the search results data that were manually labelled by the authors. As a pilot study, the authors were able to successfully develop the automated Google search tool to query and retrieve school district level mask mandate information for the state of Ohio. The algorithm trained using this data classified which school districts had implemented a mask mandate with an accuracy of 87 percent. The data predicted by the algorithm was used to verify the same data collected by ED through monthly surveys of public schools, and will inform masking policy data for states that had low response rates to the ED survey. These data will also be linked to data on the number of COVID-19 cases in schools reported by state governments to assess the success of masking policies in reducing COVID-19 case counts. The results of the study demonstrate that large scale data collection and validation activities can be conducted with high accuracy at a low cost and can be repeated more frequently than surveys can without incurring any additional burden on potential respondents.

While rigorous evaluation of donor funded education interventions are quite common in developing countries, the data collected and analyzed to assess the interventions are not always published. Even when published, the data often lack codebooks and technical documentation, guidance on how to correctly use the data, and most frustratingly only contain meta-information pertaining to the design of the intervention and evaluation in free text in PDF reports. Such practices reduce the overall quality of the data and meta-data making it very difficult to be used for secondary analysis to answer policy relevant questions. To address such challenges, this paper describes the integrated data system developed to securely receive, review, harmonize, ingest, curate, and publish standardized public use Early Grade Reading Assessment data and meta-data to track the progress of the 2011-2015 USAID Education Strategy. The paper discusses the underlying methodological framework for systematically reviewing the data for quality and completeness as well as the approach to profile the design of the intervention and evaluation to assess its rigor using the system. Lastly, the paper illustrates how the meta-data created through the systematic review process combined with the data collected and processed through the system can be used for developing dynamic data products that reduce the technical barriers for such data to be used by a wide range of stakeholders to inform critical policy decisions.

While the nFORM administrative data system is used to collect operations and performance data from HHS funded Healthy Marriage and Responsible Fatherhood (HMRF) grantees to evaluate program performance using top line measures like enrollment and attendance of workshops, the rich data collected through the system provides an unparalleled window into the workings of the HMRF programs and the population they serve. This paper describes how raw data exported from the nFORM system have been used to develop econometric models to understand the relationship of demographic and socio-economic characteristics on program enrollment and attendance, the effectiveness of incentives and behavioral nudges on program participation, as well as changes in behaviors and attitudes due to the intervention. Moreover, the paper discusses how patterns revealed through mining the raw nFORM data combined with other administrative data has provided insights into deficiencies in outreach and recruitment efforts, and highlights how the relative effectiveness of steps taken to remedy the deficiencies can also be tracked using the data. Lastly, the paper presents recommendations and best practices in using nFORM and other similar administrative data to inform continuous quality improvement of HMRF programs without stretching the budget.