Search

Teams can outperform individuals; could adding AI teammates further bolster performance of teams solving problems collaboratively? Collaborative problem solving (CPS) research commonly studies teams with two agents (human-human or human-AI), but team research literature finds that, for complex tasks, larger teams are more effective. Progress in studying collaboration with more than two agents, through textual records of team interactions, is hindered by a major data challenge: available CPS corpora are predominantly dyadic, and adapting pre-existing CPS tasks to more agents is non-trivial. We address this data challenge by developing a CPS task generator, CPS-TaskForge, that can produce environments for studying CPS under a wide array of conditions, and releasing a CPS task design checklist grounded in the theoretical PISA 2015 CPS framework to help facilitate the development of CPS corpora with more agents. CPS-TaskForge takes the form of a resource management (tower defense) game, and different CPS tasks can be studied by manipulating game design parameters. We conduct a case study with groups of 3-4 humans to validate production of diverse natural language CPS communication in a game instance produced by CPS-TaskForge. We discuss opportunities for advancing research in CPS (both with human-only and human-AI teams) using different task configurations. We will release data and code.

Distributed execution of deep learning training involves a dynamic interplay between hardware accelerator architecture and device placement strategy. This is the first work to explore the co-optimization of determining the optimal architecture and device placement strategy through novel algorithms, improving the balance of computational resources, memory usage, and data distribution. Our architecture search leverages tensor and vector units, determining their quantity and dimensionality, and on-chip and off-chip memory configurations. It also determines the microbatch size and decides whether to recompute or stash activations, balancing the memory footprint of training and storage size. For each explored architecture configuration, we use an Integer Linear Program (ILP) to find the optimal schedule for executing operators on the accelerator. The ILP results then integrate with a dynamic programming solution to identify the most effective device placement strategy, combining data, pipeline, and tensor model parallelism across multiple accelerators. Our approach achieves higher throughput on large language models compared to the state-of-the-art TPUv4 and the Spotlight accelerator search framework. The entire source code of PHAZE is available at https://github.com/msr-fiddle/phaze., Comment: Accepted at the 41st International Conference on Machine Learning (ICML), 2024

Confining light illumination in the three dimensions of space is a challenge for various applications. Among these, optogenetic methods developed for live experiments in cell biology would benefit from such a localized illumination as it would improve the spatial resolution of diffusive photosensitive proteins leading to spatially constrained biological responses in specific subcellular organelles. Here, we describe a method to create and move a focused evanescent spot across the field of view of a high numerical aperture microscope objective, using a digital micro-mirror device (DMD). We show that, after correcting the optical aberrations, light is confined within a spot of sub-micron lateral size and $\sim$100~nm axial depth, resulting in a volume of illumination drastically smaller than the one generated by a standard propagative focus. This evanescent focus is sufficient to induce a more intense and localized recruitment compared to a propagative focus on the optogenetic system CRY2-CIBN, improving the resolution of its pattern of activation., Comment: 12 pages, 6 figures, submitted to 'Optics Express'

Federated Learning (FL) allows machine learning models to train locally on individual mobile devices, synchronizing model updates via a shared server. This approach safeguards user privacy; however, it also generates a heterogeneous training environment due to the varying performance capabilities across devices. As a result, straggler devices with lower performance often dictate the overall training time in FL. In this work, we aim to alleviate this performance bottleneck due to stragglers by dynamically balancing the training load across the system. We introduce Invariant Dropout, a method that extracts a sub-model based on the weight update threshold, thereby minimizing potential impacts on accuracy. Building on this dropout technique, we develop an adaptive training framework, Federated Learning using Invariant Dropout (FLuID). FLuID offers a lightweight sub-model extraction to regulate computational intensity, thereby reducing the load on straggler devices without affecting model quality. Our method leverages neuron updates from non-straggler devices to construct a tailored sub-model for each straggler based on client performance profiling. Furthermore, FLuID can dynamically adapt to changes in stragglers as runtime conditions shift. We evaluate FLuID using five real-world mobile clients. The evaluations show that Invariant Dropout maintains baseline model efficiency while alleviating the performance bottleneck of stragglers through a dynamic, runtime approach., Comment: Accepted at the 37th Conference on Neural Information Processing Systems (NeurIPS), 2023

The existence of a shift-shift memory effect, whereby any translation of the input field induces translations in the output field in four symmetrical directions, has been observed in square waveguides by correlation measurements. Here we demonstrate that this memory effect is also observed in real space and can be put to use for imaging purposes. First, a focus is created at the output of a square-core multimode fiber, by wavefront shaping based on feedback from a guide-star. Then, thanks to the memory effect, four symmetrical spots can be scanned at the fiber output by shifting the wavefront at the fiber input. We demonstrate that this property can be exploited to perform fluorescence imaging through the multimode fiber, without requiring the measurement of a transmission matrix.

Speckle-correlation imaging techniques are widely used for non-invasive imaging through complex scattering media. While light propagation through multimode fibers and scattering media share many analogies, reconstructing images through multimode fibers from speckle correlations remains an unsolved challenge. Here, we exploit a kaleidoscopic memory effect emerging in square-core multimode fibers and demonstrate fluorescence imaging with no prior knowledge on the fiber. Experimentally, our approach simply requires to translate random speckle patterns at the input of a square-core fiber and to measure the resulting fluorescence intensity with a bucket detector. The image of the fluorescent object is then reconstructed from the autocorrelation of the measured signal by solving an inverse problem. This strategy does not require the knowledge of the fragile deterministic relation between input and output fields, which makes it promising for the development of flexible minimally-invasive endoscopes.

The widespread adoption of handheld devices have fueled rapid growth in new applications. Several of these new applications employ machine learning models to train on user data that is typically private and sensitive. Federated Learning enables machine learning models to train locally on each handheld device while only synchronizing their neuron updates with a server. While this enables user privacy, technology scaling and software advancements have resulted in handheld devices with varying performance capabilities. This results in the training time of federated learning tasks to be dictated by a few low-performance straggler devices, essentially becoming a bottleneck to the entire training process. In this work, we aim to mitigate the performance bottleneck of federated learning by dynamically forming sub-models for stragglers based on their performance and accuracy feedback. To this end, we offer the Invariant Dropout, a dynamic technique that forms a sub-model based on the neuron update threshold. Invariant Dropout uses neuron updates from the non-straggler clients to develop a tailored sub-models for each straggler during each training iteration. All corresponding weights which have a magnitude less than the threshold are dropped for the iteration. We evaluate Invariant Dropout using five real-world mobile clients. Our evaluations show that Invariant Dropout obtains a maximum accuracy gain of 1.4% points over state-of-the-art Ordered Dropout while mitigating performance bottlenecks of stragglers., Comment: Undergraduate thesis completed under advisor Dr. Prashant Nair and committee member Dr. Divya Mahajan

We establish a method, called pbFFS for photobleaching Fluctuation Fluorescence Spectroscopy, which aims at characterizing molecules or particles labelled with a unknown distribution or fluorophores. Using photobleaching as a control parameter, pbFFS provides information on the distribution of labels and a more reliable estimation of the absolute density or concentration of these molecules. We present a complete theoretical derivation of the pbFFS approach and experimentally apply it to measure the surface density of a monolayer of fluorescently tagged streptavidin molecules that can be used as a base platform for biomimetic systems. The surface density measured by pbFFS is consistent with the results of more standard surface techniques, such as ellipsometry. Compared to those techniques, pbFFS has two main advantages: it enables in situ characterization (no dedicated substrates are required) and is applicable to low masses of adsorbed molecules, as demonstrated here by quantifying the density of biotin-Atto molecules that bind to the streptavidin layer. Besides molecules immobilized on surfaces, we also applied pbFFS to molecules diffusing in solution, to confirm the distribution of labels found on surfaces. Hence, pbFFS provides a set of tools to investigate molecules attached to a variable number of fluorophores, with the aim toquantify either the number of molecules or the distribution of fluorophores, the latter case being especially relevant for oligomerization studies., Comment: 31 pages, 7 figures and supporting information (4 pages, including 2 figures)

We present a minimally-invasive endoscope based on a multimode fiber that combines photoacoustic and fluorescence sensing. From the measurement of a transmission matrix during a prior calibration step, a focused spot is produced and raster-scanned over a sample at the distal tip of the fiber by use of a fast spatial light modulator. An ultra-sensitive fiber-optic ultrasound sensor for photoacoustic detection placed next to the fiber is combined with a photodetector to obtain both fluorescence and photoacoustic images with a distal imaging tip no larger than 250um. The high signal-to-noise ratio provided by wavefront shaping based focusing and the ultra-sensitive ultrasound sensor enables imaging with a single laser shot per pixel, demonstrating fast two-dimensional hybrid imaging of red blood cells and fluorescent beads.

We describe the effect of optical aberrations on fluorescence fluctuations microscopy (FFM), when focusing through a single living cell. FFM measurements are performed in an aqueous fluorescent solution and prove to be a highly sensitive tool to assess the optical aberrations introduced by the cell. We demonstrate an adaptive optics (AO) system to remove the aberration-related bias in the FFM measurements. Our data show that AO is not only useful when imaging deep in tissues but also when performing FFM measurements through a single cellular layer. This work paves the way for the application of FFM to complex three-dimensional multicellular samples.

We report, for the first time, a multi-confocal Fluorescence Correlation Spectroscopy (mFCS) technique which allows parallel measurements at different locations, by combining a Spatial Light Modulator (SLM), with an Electron Multiplying-CCD camera (EM-CCD). The SLM is used to produce a series of laser spots, while the pixels of the EM-CCD play the roles of virtual pinholes. The phase map addressed to the SLM is calculated by using the spherical wave approximation and makes it possible to produce several diffraction limited laser spots, either aligned or spread over the field of view. To attain fast enough imaging rates, the camera has been used in different acquisition modes, the fastest of which leads to a time resolution of 100 $\mu$s. We qualified the experimental set-up by using solutions of sulforhodamine G in glycerol and demonstrated that the observation volumes are similar to that of a standard confocal set-up. To demonstrate that our mFCS method is suitable for intracellular studies, experiments have been conducted on two stable cell lines: mouse embryonic fibroblasts expressing eGFP-actin and H1299 cells expressing the heat shock factor fusion protein HSF1-eGFP. In the first case we could recover, by analyzing the auto-correlation curves, the diffusion constant of G-actin within the cytoplasm, although we were also sensitive to the complex network of interactions with F-actin. Concerning HSF1, we could clearly observe the modifications of the number of molecules and of the HSF1 dynamics during heat shock.

Fluorescence Correlation Spectroscopy (FCS) yields measurement parameters (number of molecules, diffusion time) that characterize the concentration and kinetics of fluorescent molecules within a supposedly known observation volume. Absolute derivation of concentrations and diffusion constants therefore requires preliminary calibrations of the confocal Point Spread Function with phantom solutions under perfectly controlled environmental conditions. In this paper, we quantify the influence of optical aberrations on single photon FCS and demonstrate a simple Adaptive Optics system for aberration correction. Optical aberrations are gradually introduced by focussing the excitation laser beam at increasing depths in fluorescent solutions with various refractive indices, which leads to drastic depth-dependent bias in the estimated FCS parameters. Aberration correction with a Deformable Mirror stabilizes these parameters within a range of several tens of \mum into the solution. We also demonstrate, both theoretically and experimentally, that the molecular brightness scales as the Strehl ratio squared.

Determining the number of fluorescent entities that are coupled to a given molecule (DNA, protein, etc.) is a key point of numerous biological studies, especially those based on a single molecule approach. Reliable methods are important, in this context, not only to characterize the labeling process, but also to quantify interactions, for instance within molecular complexes. We combined Fluorescence Correlation Spectroscopy (FCS) and photobleaching experiments to measure the effective number of molecules and the molecular brightness as a function of the total fluorescence count rate on solutions of cDNA (containing a few percent of C bases labeled with Alexa Fluor 647). Here, photobleaching is used as a control parameter to vary the experimental outputs (brightness and number of molecules). Assuming a Poissonian distribution of the number of fluorescent labels per cDNA, the FCS-photobleaching data could be easily fit to yield the mean number of fluorescent labels per cDNA strand (@ 2). This number could not be determined solely on the basis of the cDNA brightness, because of both the statistical distribution of the number of fluorescent labels and their unknown brightness when incorporated in cDNA. The statistical distribution of the number of fluorophores labeling cDNA was confirmed by analyzing the photon count distribution (with the cumulant method), which showed clearly that the brightness of cDNA strands varies from one molecule to the other., Comment: 38 pages (avec les figures)