Search

Unlike primates, training artificial neural networks on changing data distributions leads to a rapid decrease in performance on old tasks. This phenomenon is commonly referred to as catastrophic forgetting. In this paper, we investigate the representational changes that underlie this performance decrease and identify three distinct processes that together account for the phenomenon. The largest component is a misalignment between hidden representations and readout layers. Misalignment occurs due to learning on additional tasks and causes internal representations to shift. Representational geometry is partially conserved under this misalignment and only a small part of the information is irrecoverably lost. All types of representational changes scale with the dimensionality of hidden representations. These insights have implications for deep learning applications that need to be continuously updated, but may also aid aligning ANN models to the rather robust biological vision., Comment: 3 pages, 1 figure; published at the 2023 Conference on Cognitive Computational Neuroscience

Continual learning algorithms strive to acquire new knowledge while preserving prior information. Often, these algorithms emphasise stability and restrict network updates upon learning new tasks. In many cases, such restrictions come at a cost to the model's plasticity, i.e. the model's ability to adapt to the requirements of a new task. But is all change detrimental? Here, we approach this question by proposing that activation spaces in neural networks can be decomposed into two subspaces: a readout range in which change affects prior tasks and a null space in which change does not alter prior performance. Based on experiments with this novel technique, we show that, indeed, not all activation change is associated with forgetting. Instead, only change in the subspace visible to the readout of a task can lead to decreased stability, while restricting change outside of this subspace is associated only with a loss of plasticity. Analysing various commonly used algorithms, we show that regularisation-based techniques do not fully disentangle the two spaces and, as a result, restrict plasticity more than need be. We expand our results by investigating a linear model in which we can manipulate learning in the two subspaces directly and thus causally link activation changes to stability and plasticity. For hierarchical, nonlinear cases, we present an approximation that enables us to estimate functionally relevant subspaces at every layer of a deep nonlinear network, corroborating our previous insights. Together, this work provides novel means to derive insights into the mechanisms behind stability and plasticity in continual learning and may serve as a diagnostic tool to guide developments of future continual learning algorithms that stabilise inference while allowing maximal space for learning., Comment: 17 pages, 7 figures, Revision now accepted at COLLAS 2024

Imaging airway tissue, optical coherence tomography (OCT) provides cross-sectional images of tissue structures, shows cilia movement and mucus secretion, but does not provide sufficient contrast to differentiate individual cells. By using fast sequences of microscopic resolution OCT (mOCT) images, OCT can picture small signal fluctuations (dynamic) OCT and overcomes the lack in contrast caused by speckle noise. In this way, OCT visualizes airway morphology on a cellular level and allows to track the dynamic behavior of immune cells, as well as mucus transport and secretion. Here, we demonstrate that mOCT, by using temporal tissue fluctuation as contrast (dynamic OCT), provides the possibility to study physiological and pathological tissue processes in vivo., Comment: 12 pages, 4 figures

Language interfaces with many other cognitive domains. This paper explores how interactions at these interfaces can be studied with deep learning methods, focusing on the relation between language emergence and visual perception. To model the emergence of language, a sender and a receiver agent are trained on a reference game. The agents are implemented as deep neural networks, with dedicated vision and language modules. Motivated by the mutual influence between language and perception in cognition, we apply systematic manipulations to the agents' (i) visual representations, to analyze the effects on emergent communication, and (ii) communication protocols, to analyze the effects on visual representations. Our analyses show that perceptual biases shape semantic categorization and communicative content. Conversely, if the communication protocol partitions object space along certain attributes, agents learn to represent visual information about these attributes more accurately, and the representations of communication partners align. Finally, an evolutionary analysis suggests that visual representations may be shaped in part to facilitate the communication of environmentally relevant distinctions. Aside from accounting for co-adaptation effects between language and perception, our results point out ways to modulate and improve visual representation learning and emergent communication in artificial agents.

Most artificial neural networks used for object detection and recognition are trained in a fully supervised setup. This is not only very resource consuming as it requires large data sets of labeled examples but also very different from how humans learn. We introduce a setup in which an artificial agent first learns in a simulated world through self-supervised exploration. Following this, the representations learned through interaction with the world can be used to associate semantic concepts such as different types of doors. To do this, we use a method we call fast concept mapping which uses correlated firing patterns of neurons to define and detect semantic concepts. This association works instantaneous with very few labeled examples, similar to what we observe in humans in a phenomenon called fast mapping. Strikingly, this method already identifies objects with as little as one labeled example which highlights the quality of the encoding learned self-supervised through embodiment using curiosity-driven exploration. It therefor presents a feasible strategy for learning concepts without much supervision and shows that through pure interaction with the world meaningful representations of an environment can be learned., Comment: 10 pages. Find associated code and data here: https://github.com/vkakerbeck/FastConceptMapping

With the further development of highly automated vehicles, drivers will engage in non-related tasks while being driven. Still, drivers have to take over control when requested by the car. Here the question arises, how potentially distracted drivers get back into the control-loop quickly and safely when the car requests a takeover. To investigate effective human-machine interactions in mobile, versatile, and cost-efficient setup is needed. We developed a virtual reality toolkit for the Unity 3D game engine containing all necessary code and assets to enable fast adaptations to various human-machine interaction experiments, including close monitoring of the subject. The presented project contains all needed functionalities for realistic traffic behavior, cars, and pedestrians, as well as a large, open-source, scriptable, and modular VR environment. It covers roughly 25 square km, a package of 125 animated pedestrians and numerous vehicles, including motorbikes, trucks, and cars. It also contains all needed nature assets to make it both highly dynamic and realistic. The presented repository contains a C++ library made for LoopAR that enables force feedback for gaming steering wheels as a fully supported component. It also includes All necessary scripts for eye-tracking in the used devices. All main functions are integrated into the graphical user interface of the Unity Editor or are available as prefab variants to ease the use of the embedded functionalities. The primary purpose of this project is to serve as open access, cost-efficient toolkit that enables interested researchers to conduct realistic virtual reality research studies without costly and immobile simulators.

While optical coherence tomography (OCT) provides a resolution down to 1 micrometer it has difficulties to visualize cellular structures due to a lack of scattering contrast. By evaluating signal fluctuations, a significant contrast enhancement was demonstrated using time-domain full-field OCT (FF-OCT), which makes cellular and subcellular structures visible. The putative cause of the dynamic OCT signal is ATP-dependent motion of cellular structures in a sub-micrometer range, which provides histology-like contrast. Here we demonstrate dynamic contrast with a scanning frequency-domain OCT (FD-OCT). Given the inherent sectional imaging geometry, scanning FD-OCT provides depth-resolved images across tissue layers, a perspective known from histopathology, much faster and more efficiently than FF-OCT. Both, shorter acquisition times and tomographic depth-sectioning reduce the sensitivity of dynamic contrast for bulk tissue motion artifacts and simplify their correction in post-processing. The implementation of dynamic contrast makes microscopic FD-OCT a promising tool for histological analysis of unstained tissues., Comment: 7 pages, 3 figures, 1 Video available on request

We present a new two-stage pipeline for predicting frames of traffic scenes where relevant objects can still reliably be detected. Using a recent video prediction network, we first generate a sequence of future frames based on past frames. A second network then enhances these frames in order to make them appear more realistic. This ensures the quality of the predicted frames to be sufficient to enable accurate detection of objects, which is especially important for autonomously driving cars. To verify this two-stage approach, we conducted experiments on the Cityscapes dataset. For enhancing, we trained two image-to-image translation methods based on generative adversarial networks, one for blind motion deblurring and one for image super-resolution. All resulting predictions were quantitatively evaluated using both traditional metrics and a state-of-the-art object detection network showing that the enhanced frames appear qualitatively improved. While the traditional image comparison metrics, i.e., MSE, PSNR, and SSIM, failed to confirm this visual impression, the object detection evaluation resembles it well. The best performing prediction-enhancement pipeline is able to increase the average precision values for detecting cars by about 9% for each prediction step, compared to the non-enhanced predictions., Comment: Accepted for presentation at the IEEE Intelligent Transportation Systems Conference -- ITSC 2019

The effects of particle roughness and short-ranged non-central forces on colloidal gels are studied using computer simulations in which particles experience a sinusoidal variation in energy as they rotate. The number of minima $n$ and energy scale $K$ are the key parameters; for large $K$ and $n$, particle rotation is strongly hindered, but for small $K$ and $n$ particle rotation is nearly free. A series of systems are simulated and characterized using fractal dimensions, structure factors, coordination number distributions, bond-angle distributions and linear rheology. When particles rotate easily, clusters restructure to favor dense packings. This leads to longer gelation times and gels with strand-like morphology. The elastic moduli of such gels scale as $G' \propto \omega^{0.5}$ at high shear frequencies $\omega$. In contrast, hindered particle rotation inhibits restructuring and leads to rapid gelation and diffuse morphology. Such gels are stiffer, with $G'\propto\omega^{0.35}$. The viscous moduli $G''$ in the low-barrier and high-barrier regimes scale according to exponents $0.53$ and $0.5$, respectively. The crossover frequency between elastic and viscous behaviors generally increases with the barrier to rotation. These findings agree qualitatively with some recent experiments on heterogeneously-surface particles and with studies of DLCA-type gels and gels of smooth spheres., Comment: 32 pages, 8 figures

Data augmentation is a popular technique largely used to enhance the training of convolutional neural networks. Although many of its benefits are well known by deep learning researchers and practitioners, its implicit regularization effects, as compared to popular explicit regularization techniques, such as weight decay and dropout, remain largely unstudied. As a matter of fact, convolutional neural networks for image object classification are typically trained with both data augmentation and explicit regularization, assuming the benefits of all techniques are complementary. In this paper, we systematically analyze these techniques through ablation studies of different network architectures trained with different amounts of training data. Our results unveil a largely ignored advantage of data augmentation: networks trained with just data augmentation more easily adapt to different architectures and amount of training data, as opposed to weight decay and dropout, which require specific fine-tuning of their hyperparameters., Comment: Preprint of the manuscript accepted for presentation at the International Conference on Artificial Neural Networks (ICANN) 2018. Best Paper Award

Deep convolutional neural networks trained for image object categorization have shown remarkable similarities with representations found across the primate ventral visual stream. Yet, artificial and biological networks still exhibit important differences. Here we investigate one such property: increasing invariance to identity-preserving image transformations found along the ventral stream. Despite theoretical evidence that invariance should emerge naturally from the optimization process, we present empirical evidence that the activations of convolutional neural networks trained for object categorization are not robust to identity-preserving image transformations commonly used in data augmentation. As a solution, we propose data augmentation invariance, an unsupervised learning objective which improves the robustness of the learned representations by promoting the similarity between the activations of augmented image samples. Our results show that this approach is a simple, yet effective and efficient (10 % increase in training time) way of increasing the invariance of the models while obtaining similar categorization performance., Comment: 6 pages, 2 figures, work in progress

Contrary to most machine learning models, modern deep artificial neural networks typically include multiple components that contribute to regularization. Despite the fact that some (explicit) regularization techniques, such as weight decay and dropout, require costly fine-tuning of sensitive hyperparameters, the interplay between them and other elements that provide implicit regularization is not well understood yet. Shedding light upon these interactions is key to efficiently using computational resources and may contribute to solving the puzzle of generalization in deep learning. Here, we first provide formal definitions of explicit and implicit regularization that help understand essential differences between techniques. Second, we contrast data augmentation with weight decay and dropout. Our results show that visual object categorization models trained with data augmentation alone achieve the same performance or higher than models trained also with weight decay and dropout, as is common practice. We conclude that the contribution on generalization of weight decay and dropout is not only superfluous when sufficient implicit regularization is provided, but also such techniques can dramatically deteriorate the performance if the hyperparameters are not carefully tuned for the architecture and data set. In contrast, data augmentation systematically provides large generalization gains and does not require hyperparameter re-tuning. In view of our results, we suggest to optimize neural networks without weight decay and dropout to save computational resources, hence carbon emissions, and focus more on data augmentation and other inductive biases to improve performance and robustness., Comment: Major changes: 1. updated figures; 2. statistical significance analysis of results through bootstrap; 3. information about carbon emissions produced with the experiments; 4. extended discussion about the need for weight decay and dropout; 5. literature review updated; 6. slight update of the definitions of explicit and implicit regularization, with several examples for illustration

The impressive success of modern deep neural networks on computer vision tasks has been achieved through models of very large capacity compared to the number of available training examples. This overparameterization is often said to be controlled with the help of different regularization techniques, mainly weight decay and dropout. However, since these techniques reduce the effective capacity of the model, typically even deeper and wider architectures are required to compensate for the reduced capacity. Therefore, there seems to be a waste of capacity in this practice. In this paper we build upon recent research that suggests that explicit regularization may not be as important as widely believed and carry out an ablation study that concludes that weight decay and dropout may not be necessary for object recognition if enough data augmentation is introduced., Comment: Minor changes: more explicit legend of the figure, clearer second paragraph of the introduction, details of weight decay and dropout in the appendix

Ethical thought experiments such as the trolley dilemma have been investigated extensively in the past, showing that humans act in a utilitarian way, trying to cause as little overall damage as possible. These trolley dilemmas have gained renewed attention over the past years; especially due to the necessity of implementing moral decisions in autonomous driving vehicles. We conducted a set of experiments in which participants experienced modified trolley dilemmas as the driver in a virtual reality environment. Participants had to make decisionsbetween two discrete options: driving on one of two lanes where different obstacles came into view. Obstacles included a variety of human-like avatars of different ages and group sizes. Furthermore, we tested the influence of a sidewalk as a potential safe harbor and a condition implicating a self-sacrifice. Results showed that subjects, in general, decided in a utilitarian manner, sparing the highest number of avatars possible with a limited influence of the other variables. Our findings support that human behavior is in line with the utilitarian approach to moral decision making. This may serve as a guideline for the implementation of moral decisions in ADVs., Comment: 24 pages, 7 figures