Data matters: Towards a data-centric theory of generalisation

The ability of a learning machine to perform outside the training data is referred to as its generalisation performance. Despite being researched for many years, generalisation is one of the key unresolved puzzles in machine learning. In this thesis we start building the understanding needed to construct a new framework for reasoning about generalisation. We start with a theoretical perspective but conclude that the field needs to build stronger intuitions before being able to formalise generalisation in a meaningful way. Our theoretical exploration, however, highlights that the data plays a much more central role than previously acknowledged. To better understand how the data can be incorporated in generalisation studies, we start exploring the practice of modifying images. The modifications we consider are mixed data augmentation, patch-shuffling, and patch-based occlusion. We find that there are a number of incorrect implicit assumptions in the literature regarding the side effects of data modification. These assumptions deem some distortion-based approaches to evaluating model attributes to be incorrect. In the case of modifying data to assess robustness to occlusion, we propose a solution that addresses the side effects. The existence of these incorrect assumptions attests to the fact that the field has a poor understanding of data modification. Despite the field’s limited understanding, data distortion has most recently been used to empirically predict generalisation performance. We focus on this practice and claim that data modification has been carelessly used in this case as well. We argue that it is the limited evaluation settings that caused the modification-based predictors to appear successful despite relying on poorly founded intuitions. We end by proposing the backbone for an extensive evaluation of empirical predictors of generalisation. We believe that such a practical approach to generalisation, when thoroughly designed, has the potential to provid