Navigating the development challenges in creating complex data systems

Data science systems (DSSs) are a fundamental tool in many areas of research and are now developed by people with a myriad of backgrounds. This is coupled with a crisis in reproducibility of such DSSs despite the wide availability of powerful tools for data science and machine learning over the last decade. We believe that perverse incentives and a lack of widespread software engineering skills are among the many causes of this crisis and analyze why software engineering and building large complex systems is, in general, hard. Based on these insights, we identify how software engineering addresses those difficulties and how one might apply and generalize software engineering methods to make DSSs more fit for purpose. We advocate two key development philosophies: one should incrementally grow – not plan then build – DSSs, and one should use two types of feedback loops during development: one which tests the code’s correctness and another that evaluates the code’s efficacy.