Search

Due to their architecture and how they are trained, artificial neural networks are typically not robust toward pruning, replacing, or shuffling layers at test time. However, such properties would be desirable for different applications, such as distributed neural network architectures where the order of execution cannot be guaranteed or parts of the network can fail during inference. In this work, we address these issues through a number of proposed training approaches for vision transformers whose most important component is randomizing the execution order of attention modules at training time. We show that with our proposed approaches, vision transformers are indeed capable to adapt to arbitrary layer execution orders at test time assuming one tolerates a reduction (about 20\%) in accuracy at the same model size. We also find that our trained models can be randomly merged with each other resulting in functional ("Frankenstein") models without loss of performance compared to the source models. Finally, we layer-prune our models at test time and find that their performance declines gracefully.

Image-generation models are changing how we express ourselves in visual art. However, what people think of AI-generated art is still largely unexplored, especially compared to traditional art. In this paper, we present the design of an interactive research product, GenFrame - an image-generating picture frame that appears as a traditional painting but offers the viewer the agency to modify the depicted painting. In the current paper, we report on a study where we deployed the GenFrame in a traditional art museum and interviewed visitors about their views on AI art. When provoked by AI-generated art, people need more of the artist's backstory and emotional journey to make the artwork commensurate with traditional art. However, generative AI-enabled interactive experiences open new ways of engaging with art when a turn of a dial can modify art styles or motifs on a painting. A demo can be seen here: https://youtu.be/1rhW4fazaBY., Comment: Design Research Society conference 2024 (DRS2024), Boston 24-28 June 2024

This Research through Design paper explores how object detection may be applied to a large digital art museum collection to facilitate new ways of encountering and experiencing art. We present the design and evaluation of an interactive application called SMKExplore, which allows users to explore a museum's digital collection of paintings by browsing through objects detected in the images, as a novel form of open-ended exploration. We provide three contributions. First, we show how an object detection pipeline can be integrated into a design process for visual exploration. Second, we present the design and development of an app that enables exploration of an art museum's collection. Third, we offer reflections on future possibilities for museums and HCI researchers to incorporate object detection techniques into the digitalization of museums.

While diversity has become a debated issue in design, very little research exists on positive use-cases for diversity beyond scholarly criticism. The current work addresses this gap through the case of a diversity-aware chatbot, exploring what benefits a diversity-aware chatbot could bring to people and how do people interpret diversity when being presented with it. In this paper, we motivate a Q&A chatbot as a technology probe and deploy it in two student communities within a study. During the study, we collected contextual data on people's expectations and perceptions when presented with diversity during the study. Our key findings show that people seek out others with shared niche interests, or their search is driven by exploration and inspiration when presented with diversity. Although interacting with chatbots is limited, participants found the engagement novel and interesting to motivate future research., Comment: Design Research Society conference 2022, Bilbao, 25 June - 3 July, 2022

Models leveraging both visual and textual data such as Contrastive Language-Image Pre-training (CLIP), are the backbone of many recent advances in artificial intelligence. In this work, we show that despite their versatility, such models are vulnerable to what we refer to as fooling master images. Fooling master images are capable of maximizing the confidence score of a CLIP model for a significant number of widely varying prompts, while being either unrecognizable or unrelated to the attacked prompts for humans. The existence of such images is problematic as it could be used by bad actors to maliciously interfere with CLIP-trained image retrieval models in production with comparably small effort as a single image can attack many different prompts. We demonstrate how fooling master images for CLIP (CLIPMasterPrints) can be mined using stochastic gradient descent, projected gradient descent, or blackbox optimization. Contrary to many common adversarial attacks, the blackbox optimization approach allows us to mine CLIPMasterPrints even when the weights of the model are not accessible. We investigate the properties of the mined images, and find that images trained on a small number of image captions generalize to a much larger number of semantically related captions. We evaluate possible mitigation strategies, where we increase the robustness of the model and introduce an approach to automatically detect CLIPMasterPrints to sanitize the input of vulnerable models. Finally, we find that vulnerability to CLIPMasterPrints is related to a modality gap in contrastive pre-trained multi-modal networks. Code available at https://github.com/matfrei/CLIPMasterPrints., Comment: This work was supported by a research grant (40575) from VILLUM FONDEN

Smartphones enable understanding human behavior with activity recognition to support people's daily lives. Prior studies focused on using inertial sensors to detect simple activities (sitting, walking, running, etc.) and were mostly conducted in homogeneous populations within a country. However, people are more sedentary in the post-pandemic world with the prevalence of remote/hybrid work/study settings, making detecting simple activities less meaningful for context-aware applications. Hence, the understanding of (i) how multimodal smartphone sensors and machine learning models could be used to detect complex daily activities that can better inform about people's daily lives and (ii) how models generalize to unseen countries, is limited. We analyzed in-the-wild smartphone data and over 216K self-reports from 637 college students in five countries (Italy, Mongolia, UK, Denmark, Paraguay). Then, we defined a 12-class complex daily activity recognition task and evaluated the performance with different approaches. We found that even though the generic multi-country approach provided an AUROC of 0.70, the country-specific approach performed better with AUROC scores in [0.79-0.89]. We believe that research along the lines of diversity awareness is fundamental for advancing human behavior understanding through smartphones and machine learning, for more real-world utility across countries., Comment: ACM CHI 2023

Mood inference with mobile sensing data has been studied in ubicomp literature over the last decade. This inference enables context-aware and personalized user experiences in general mobile apps and valuable feedback and interventions in mobile health apps. However, even though model generalization issues have been highlighted in many studies, the focus has always been on improving the accuracies of models using different sensing modalities and machine learning techniques, with datasets collected in homogeneous populations. In contrast, less attention has been given to studying the performance of mood inference models to assess whether models generalize to new countries. In this study, we collected a mobile sensing dataset with 329K self-reports from 678 participants in eight countries (China, Denmark, India, Italy, Mexico, Mongolia, Paraguay, UK) to assess the effect of geographical diversity on mood inference models. We define and evaluate country-specific (trained and tested within a country), continent-specific (trained and tested within a continent), country-agnostic (tested on a country not seen on training data), and multi-country (trained and tested with multiple countries) approaches trained on sensor data for two mood inference tasks with population-level (non-personalized) and hybrid (partially personalized) models. We show that partially personalized country-specific models perform the best yielding area under the receiver operating characteristic curve (AUROC) scores of the range 0.78-0.98 for two-class (negative vs. positive valence) and 0.76-0.94 for three-class (negative vs. neutral vs. positive valence) inference. Overall, we uncover generalization issues of mood inference models to new countries and how the geographical similarity of countries might impact mood inference., Comment: ACM IMWUT 2022, To be presented at ACM Ubicomp 2023

In crystalline solids the interactions of charge and spin can result in a variety of emergent quantum ground states, especially in partially filled, topological flat bands such as Landau levels or in 'magic-angle' bilayer graphene. Much less explored is rhombohedral graphite (RG), perhaps the simplest and structurally most perfect condensed matter system to host a flat band protected by symmetry. By scanning tunneling microscopy we map the flat band charge density of 8, 10 and 17 layers and identify a domain structure emerging from a competition between a sublattice antiferromagnetic insulator and a gapless correlated paramagnet. Our density-matrix renormalization group calculations explain the observed features and demonstrate that the correlations are fundamentally different from graphene based magnetism identified until now, forming the ground state of a quantum magnet. Our work establishes RG as a new platform to study many-body interactions beyond the mean-field approach, where quantum fluctuations and entanglement dominate., Comment: supplementary information included

Detecting conductance quantization in graphene nanostructures turned out more challenging than expected. The observation of well-defined conductance plateaus through graphene nanoconstrictions so far has only been accessible in the highest quality suspended or h-BN encapsulated devices. However, reaching low conductance quanta in zero magnetic field, is a delicate task even with such ultra-high mobility devices. Here, we demonstrate a simple AFM-based nanopatterning technique for defining graphene constrictions with high precision (down to 10 nm width) and reduced edge-roughness (+/- 1 nm). The patterning process is based on the in-plane mechanical cleavage of graphene by the AFM tip, along its high symmetry crystallographic directions. As-defined, narrow graphene constrictions with improved edge quality enable an unprecedentedly robust QPC operation, allowing the observation of conductance quantization even on standard $SiO_2/Si$ substrates, down to low conductance quanta. Conductance plateaus, were observed at $ne^2/h$, evenly spaced by $2e^2/h$ (corresponding to n = 3, 5, 7, 9, 11) in the absence of an external magnetic field, while spaced by $e^2/h$ (n = 1, 2, 3, 4, 5, 6) in 8T magnetic field., Comment: Main paper and supplementary informations

Non-reciprocal transport, where the left to right flowing current differs from the right to left flowing one, is an unexpected phenomenon in bulk crystals. BiTeBr is a non-centrosymmetric material, with a giant Rashba spin-orbit coupling which presents this unusual effect when placed in an in-plane magnetic field. It has been shown that this effect depends strongly on the carrier density, however, in-situ tuning has not yet been demonstrated. We developed a method where thin BiTeBr flakes are gate tuned via ionic-liquid gating through a thin protective hBN layer. Tuning the carrier density allows a more than \SI{400}{\percent} variation of the non-reciprocal response. Our study serves as a milestone on how a few-atomic-layer-thin van der Waals protection layer allows ionic gating of chemically sensitive, exotic nanocrystals.

Graphene on noble-metal nanostructures constitutes an attractive nanocomposite with possible applications in sensors or energy conversion. In this work we study the properties of hybrid graphene/gold nanoparticle structures by Raman spectroscopy and Scanning Probe Methods. The nanoparticles (NPs) were prepared by local annealing of gold thin films using focused laser beam. The method resulted in a patterned surface, with NPs formed at arbitrarily chosen microscale areas. Graphene grown by chemical vapour deposition was transferred onto the prepared, closely spaced gold NPs. While we found that successive higher intensity (6 mW) laser irradiation increased gradually the doping and the defect concentration in SiO2 supported graphene, the same irradiation procedure did not induce such irreversible effects in the graphene supported by gold NPs. Moreover, the laser irradiation induced dynamic hydrostatic strain in the graphene on Au NPs, which turned out to be completely reversible. These results can have implications in the development of graphene/plasmonic nanoparticle based high temperature sensors operating in dynamic regimes., Comment: 18 pages, 5 figures

The pseudo-magnetic field generated by mechanical strain in graphene can have dramatic consequences on the behavior of electrons and holes. Here we show that pseudo-magnetic field fluctuations present in crumpled graphene can induce significant intravalley scattering of charge carriers. We detect this by measuring the confocal Raman spectra of crumpled areas, where we observe an increase of the D'/D peak intensity ratio by up to a factor of 300. We reproduce our observations by numerical calculation of the double resonant Raman spectra and interpret the results as experimental evidence of the phase shift suffered by Dirac charge carriers in the presence of a pseudo-magnetic field. This lifts the restriction on complete intravalley backscattering of Dirac fermions.

Spin orbit interaction can be strongly boosted when a heavy element is embedded into an inversion asymmetric crystal field. A simple structure to realize this concept in a 2D crystal contains three atomic layers, a middle one built up from heavy elements generating strong atomic spin-orbit interaction and two neighboring atomic layers with different electron negativity. BiTeI is a promising candidate for such a 2D crystal, since it contains heavy Bi layer between Te and I layers. Recently the bulk form of BiTeI attracted considerable attention due to its giant Rashba interaction, however, 2D form of this crystal was not yet created. In this work we report the first exfoliation of single layer BiTeI using a recently developed exfoliation technique on stripped gold. Our combined scanning probe studies and first principles calculations show that SL BiTeI flakes with sizes of 100 $\mu$m were achieved which are stable at ambient conditions. The giant Rashba splitting and spin-momentum locking of this new member of 2D crystals open the way towards novel spintronic applications and synthetic topological heterostructures., Comment: 20 pages, 5 figures

Graphene nanoplatelets (GNPs) have emerged as one of the most promising filler materials for improving the tribological performance of ceramic composites due to their outstanding solid lubricant properties as well as mechanical and thermal stability. Yet, the addition of GNPs has so far provided only a very limited improvement in the tribological properties of ceramics, particularly concerning the reduction of their friction coefficient. This is most likely due to the challenges of achieving a lubricating and protecting tribo-film through a high GNP coverage of the exposed surfaces. Here we show that this can be achieved by efficiently increasing the exfoliation degree of GNPs down to the few-layer (FL) range. By employing FL-GNPs as filler material, the wear resistance of Si3N4 composites can be increased by about twenty times, the friction coefficient reduced to nearly its half, while the other mechanical properties are also preserved or improved. Using confocal Raman microscopy, we were able to demonstrate the formation of a continuous FL- GNP tribo-film, already at 5wt% FL-GNP content.