Robust Data Submission Pipeline For AgMIP Global Economics Modelers

Agricultural Model Intercomparison and Improvement Project (AgMIP) is an international effort to improve agricultural models and scientific and technological capabilities for assessing the impacts of climate variability and other driving factors on agriculture, food security, and poverty. The Global Economics (GlobalEcon) team in AgMIP conducts extensive model intercomparison to assess the agricultural effects of climate change, bioenergy policies, and socioeconomic factors. Currently, the preparation and sharing of GlobalEcon model data are done manually, which is time-consuming and error-prone. This project aims to address this problem by designing a robust data submission pipeline capable of harmonizing data from diverse modeling groups within the team. The result is a web application implemented with Jupyter Notebook and hosted on MyGeoHub that can accept, validate, and clean up model data from the GlobalEcon modelers, diagnose the data for errors, and visualize potential outliers in the data. The created tool performs data sanitization and diagnosis based on data format protocols agreed by the modeling team. It also lets individual users request a protocol update should the need arise. In addition, for protocol violations that are easy to fix, the application allows users to correct them on the go. Upon successful submission, the tool stores the uploaded data in a shared repository on MyGeoHub, which an existing data exploration tool can query and access. The creation of this feature-rich data submission tool helps the GlobalEcon team build an accessible and interoperable data repository and streamline their ensemble modeling activities.