Search

We present a method for controlling a simulated humanoid to grasp an object and move it to follow an object trajectory. Due to the challenges in controlling a humanoid with dexterous hands, prior methods often use a disembodied hand and only consider vertical lifts or short trajectories. This limited scope hampers their applicability for object manipulation required for animation and simulation. To close this gap, we learn a controller that can pick up a large number (>1200) of objects and carry them to follow randomly generated trajectories. Our key insight is to leverage a humanoid motion representation that provides human-like motor skills and significantly speeds up training. Using only simplistic reward, state, and object representations, our method shows favorable scalability on diverse object and trajectories. For training, we do not need dataset of paired full-body motion and object trajectories. At test time, we only require the object mesh and desired trajectories for grasping and transporting. To demonstrate the capabilities of our method, we show state-of-the-art success rates in following object trajectories and generalizing to unseen objects. Code and models will be released., Comment: Project page: https://www.zhengyiluo.com/Omnigrasp/

The Magnetic Resonance Fingerprinting (MRF) approach aims to estimate multiple MR or physiological parameters simultaneously with a single fast acquisition sequence. Most of the MRF studies proposed so far have used simple MR sequence types to measure relaxation times (T1, T2). In that case, deep learning algorithms have been successfully used to speed up the reconstruction process. In theory, the MRF concept could be used with a variety of other MR sequence types and should be able to provide more information about the tissue microstructures. Yet, increasing the complexity of the numerical models often leads to prohibited simulation times, and estimating multiple parameters from one sequence implies new dictionary dimensions whose sizes become too large for standard computers and DL architectures.In this paper, we propose to analyze the MRF signal coming from a complex balance Steady-state free precession (bSSFP) type sequence to simultaneously estimate relaxometry maps (T1, T2), Field maps (B1, B0) as well as microvascular properties such as the local Cerebral Blood Volume (CBV) or the averaged vessel Radius (R).To bypass the curse of dimensionality, we propose an efficient way to simulate the MR signal coming from numerical voxels containing realistic microvascular networks as well as a Bidirectional Long Short-Term Memory network used for the matching process.On top of standard MRF maps, our results on 3 human volunteers suggest that our approach can quickly produce high-quality quantitative maps of microvascular parameters that are otherwise obtained using longer dedicated sequences and intravenous injection of a contrast agent. This approach could be used for the management of multiple pathologies and could be tuned to provide other types of microstructural information.

Reinforcement Learning has achieved significant success in generating complex behavior but often requires extensive reward function engineering. Adversarial variants of Imitation Learning and Inverse Reinforcement Learning offer an alternative by learning policies from expert demonstrations via a discriminator. Employing discriminators increases their data- and computational efficiency over the standard approaches; however, results in sensitivity to imperfections in expert data. We propose RILe, a teacher-student system that achieves both robustness to imperfect data and efficiency. In RILe, the student learns an action policy while the teacher dynamically adjusts a reward function based on the student's performance and its alignment with expert demonstrations. By tailoring the reward function to both performance of the student and expert similarity, our system reduces dependence on the discriminator and, hence, increases robustness against data imperfections. Experiments show that RILe outperforms existing methods by 2x in settings with limited or noisy expert data.

I present all the details in calculating the posterior distribution of the conjugate Normal-Gamma prior in Bayesian Linear Models (BLM), including correlated observations, prediction, model selection and comments on efficient numeric implementations. A Python implementation is also presented. These have been presented and available in many books and texts but, I believe, a general compact and simple presentation is always welcome and not always simple to find. Since correlated observations are also included, these results may also be useful for time series analysis and spacial statistics. Other particular cases presented include regression, Gaussian processes and Bayesian Dynamic Models., Comment: 13 pages, 4 figures, Python implementation

Monocular egocentric 3D human motion capture is a challenging and actively researched problem. Existing methods use synchronously operating visual sensors (e.g. RGB cameras) and often fail under low lighting and fast motions, which can be restricting in many applications involving head-mounted devices. In response to the existing limitations, this paper 1) introduces a new problem, i.e., 3D human motion capture from an egocentric monocular event camera with a fisheye lens, and 2) proposes the first approach to it called EventEgo3D (EE3D). Event streams have high temporal resolution and provide reliable cues for 3D human motion capture under high-speed human motions and rapidly changing illumination. The proposed EE3D framework is specifically tailored for learning with event streams in the LNES representation, enabling high 3D reconstruction accuracy. We also design a prototype of a mobile head-mounted device with an event camera and record a real dataset with event observations and the ground-truth 3D human poses (in addition to the synthetic dataset). Our EE3D demonstrates robustness and superior 3D accuracy compared to existing solutions across various challenging experiments while supporting real-time 3D pose update rates of 140Hz., Comment: 14 pages, 11 figures and 6 tables; project page: https://4dqv.mpi-inf.mpg.de/EventEgo3D/; Computer Vision and Pattern Recognition (CVPR) 2024

Expulsion and suspension (or exclusionary discipline) in early care and education (ECE) are widely recognized as harmful to children. Numerous states have developed expulsion and suspension prevention policies through legislation or administrative rules. However, there is limited research on the implementation of state policies designed to reduce exclusionary discipline and on ECE providers' use of program supports that could help reduce exclusionary discipline practices. In this report, we use survey data to investigate Arkansas ECE program leaders' and teachers' understanding of the state's expulsion prevention policy, their use of suspension and expulsion and their participation in professional development (PD) and other supports to meet the needs of children with challenging behavior. We also explore how teachers' attributions of the reasons for challenging behavior predict their use of the available PD and supports. Key findings include: (1) Publicly funded programs in Arkansas (which are subject to state policy) engaged in exclusionary discipline at lower rates than non-publicly funded programs, despite reporting a higher percentage of children with challenging behavior; (2) Many Arkansas ECE program leaders were aware of the state's expulsion prevention policy, but fewer demonstrated a strong understanding of the requirements; (3) More than three quarters of Arkansas ECE teachers reported receiving some form of PD or training on topics such as encouraging positive teacher-child interactions, promoting children's positive behavior and helping children develop skills like managing their emotions; and (4) The use of exclusionary discipline practices was related to teachers' views of families as a reason for children's challenging behavior. At the end of the report, we offer recommendations based on these findings.

A recently launched professional development initiative in Arkansas, called Array, is designed to proactively identify early care and education (ECE) programs that could benefit from social-emotional-focused (SE-focused) professional development and support, and to offer assistance tailored to a program's particular needs. Because Array is designed to initiate contact with ECE programs, it has the potential to intervene early and prevent serious escalations of teachers' frustration in response to children's challenging behavior. This report examines results from case studies in three ECE programs that participated in Array between June and December 2022. The results focus on unique features of Array, how it operates, reported benefits to programs, and recommendations for continued improvement.

Human hands possess the dexterity to interact with diverse objects such as grasping specific parts of the objects and/or approaching them from desired directions. More importantly, humans can grasp objects of any shape without object-specific skills. Recent works synthesize grasping motions following single objectives such as a desired approach heading direction or a grasping area. Moreover, they usually rely on expensive 3D hand-object data during training and inference, which limits their capability to synthesize grasping motions for unseen objects at scale. In this paper, we unify the generation of hand-object grasping motions across multiple motion objectives, diverse object shapes and dexterous hand morphologies in a policy learning framework GraspXL. The objectives are composed of the graspable area, heading direction during approach, wrist rotation, and hand position. Without requiring any 3D hand-object interaction data, our policy trained with 58 objects can robustly synthesize diverse grasping motions for more than 500k unseen objects with a success rate of 82.2%. At the same time, the policy adheres to objectives, which enables the generation of diverse grasps per object. Moreover, we show that our framework can be deployed to different dexterous hands and work with reconstructed or generated objects. We quantitatively and qualitatively evaluate our method to show the efficacy of our approach. Our model, code, and the large-scale generated motions are available at https://eth-ait.github.io/graspxl/., Comment: Camera ready for ECCV2024. Project Page: https://eth-ait.github.io/graspxl/

Generating natural hand-object interactions in 3D is challenging as the resulting hand and object motions are expected to be physically plausible and semantically meaningful. Furthermore, generalization to unseen objects is hindered by the limited scale of available hand-object interaction datasets. We propose DiffH2O, a novel method to synthesize realistic, one or two-handed object interactions from provided text prompts and geometry of the object. The method introduces three techniques that enable effective learning from limited data. First, we decompose the task into a grasping stage and a text-based interaction stage and use separate diffusion models for each. In the grasping stage, the model only generates hand motions, whereas in the interaction phase both hand and object poses are synthesized. Second, we propose a compact representation that tightly couples hand and object poses. Third, we propose two different guidance schemes to allow more control of the generated motions: grasp guidance and detailed textual guidance. Grasp guidance takes a single target grasping pose and guides the diffusion model to reach this grasp at the end of the grasping stage, which provides control over the grasping pose. Given a grasping motion from this stage, multiple different actions can be prompted in the interaction phase. For textual guidance, we contribute comprehensive text descriptions to the GRAB dataset and show that they enable our method to have more fine-grained control over hand-object interactions. Our quantitative and qualitative evaluation demonstrates that the proposed method outperforms baseline methods and leads to natural hand-object motions. Moreover, we demonstrate the practicality of our framework by utilizing a hand pose estimate from an off-the-shelf pose estimator for guidance, and then sampling multiple different actions in the interaction stage., Comment: Project Page: https://diffh2o.github.io/

This paper proposes a new approach for analyzing seismic accelerograms using the evolutionary Power Spectral Density function (ePSDF). The accelerogram of an earthquake can be accurately modeled and simulated from its spectrogram, based on the oscillatory stochastic processes theory. To adequately characterize a spectrogram that is consistent with the response spectra, a parametric model with 16 parameters is proposed. This model describes the square of the amplitude spectrum, an envelope of the square of the accelerogram, and a copula that constructs a time-frequency model from the time and frequency marginals. The use of copulas to model a bivariate probability distribution is a common practice in statistics, particularly when the marginal distributions are known. The periodogram can be viewed as an unnormalized probability density function, where the total energy serves as the normalization constant, since the total energy of a seismic motion is always finite. Additionally, a reduced model consisting of only 10 parameters is presented, which may be especially valuable when only shear wave effects are relevant., Comment: 20 pages, 13 figures. Keywords:Spectral Density, Oscillatory process, Copula, Stochastic simulation, Response Spectrum

A central challenge in quantum networking is transferring quantum states between different physical modalities, such as between flying photonic qubits and stationary quantum memories. One implementation entails using spin-photon interfaces that combine solid-state spin qubits, such as color centers in diamond, with photonic nanostructures. However, while high-fidelity spin-photon interactions have been demonstrated on isolated devices, building practical quantum repeaters requires scaling to large numbers of interfaces yet to be realized. Here, we demonstrate integration of nanophotonic cavities containing tin-vacancy (SnV) centers in a photonic integrated circuit (PIC). Out of a six-channel quantum micro-chiplet (QMC), we find four coupled SnV-cavity devices with an average Purcell factor of ~7. Based on system analyses and numerical simulations, we find with near-term improvements this multiplexed architecture can enable high-fidelity quantum state transfer, paving the way towards building large-scale quantum repeaters., Comment: to be published in Optica Quantum

Programmable spatiotemporal control of light is crucial for advancements in optical communications, imaging, and quantum technologies. Commercial spatial light modulators (SLMs) typically have megapixel-scale apertures but are limited to ~kHz operational speeds. Developing a device that controls a similar number of spatial modes at high speeds could potentially transform fields such as imaging through scattering media, quantum computing with cold atoms and ions, and high-speed machine vision, but to date remains an open challenge. In this work we introduce and demonstrate a free-form, resonant electro-optic (EO) modulator with megapixel apertures using CMOS integration. The optical layer features a Lithium Niobate (LN) thin-film integrated with a photonic crystal (PhC), yielding a guided mode resonance (GMR) with a Q-factor>1000, a field overlap coefficient ~90% and a 1.6 GHz 3-dB modulation bandwidth (detector limited). To realize a free-form and scalable SLM, we fabricate the PhC via interference lithography and develop a procedure to bond the device to a megapixel CMOS backplane. We identify limitations in existing EO materials and CMOS backplanes that must be overcome to simultaneously achieve megapixel-scale, GHz-rate operation. The `LN on Silicon' (LNoS) architecture we present is a blueprint towards realizing such devices.

The Open Databases Integration for Materials Design (OPTIMADE) application programming interface (API) empowers users with holistic access to a growing federation of databases, enhancing the accessibility and discoverability of materials and chemical data. Since the first release of the OPTIMADE specification (v1.0), the API has undergone significant development, leading to the upcoming v1.2 release, and has underpinned multiple scientific studies. In this work, we highlight the latest features of the API format, accompanying software tools, and provide an update on the implementation of OPTIMADE in contributing materials databases. We end by providing several use cases that demonstrate the utility of the OPTIMADE API in materials research that continue to drive its ongoing development.

Indium incorporation into wurtzite (0001)-oriented In$_{x}$Al$_{y}$Ga$_{1-x-y}$N layers grown by plasma-assisted molecular beam epitaxy was studied as a function of the growth temperature (565-635 $^{\circ}$C) and the AlN mole fraction (0.01 < y < 0.27). The layer stoichiometry was determined by Rutherford backscattering spectrometry (RBS). RBS shows that Indium incorporation decreased continuously with increasing growth temperature due to thermally enhanced dissociation of In-N bonds and for increasing AlN mole fractions. High resolution x-ray diffraction and transmission electron microscopy (TEM) measurements did not show evidence of phase separation. The mosaicity of the quaternary layers was found to be mainly determined by the growth temperature and independent on alloy composition within the range studied. However, depending on the AlN mole fraction, nanometer-sized composition fluctuations were detected by TEM. Photoluminescence spectra showed a single broad emission at room temperature, with energy and bandwidth S- and W-shaped temperature dependences typical of exciton localization by alloy inhomogeneities. Cathodo-luminescence measurements demonstrated that the alloy inhomogeneities, responsible of exciton localization, occur on a lateral length scale below 150 nm, which is corroborated by TEM.

Cluster repair methods aim to determine errors in clusters and modify them so that each cluster consists of records representing the same entity. Current cluster repair methodologies primarily assume duplicate-free data sources, where each record from one source corresponds to a unique record from another. However, real-world data often deviates from this assumption due to quality issues. Recent approaches apply clustering methods in combination with link categorization methods so they can be applied to data sources with duplicates. Nevertheless, the results do not show a clear picture since the quality highly varies depending on the configuration and dataset. In this study, we introduce a novel approach for cluster repair that utilizes graph metrics derived from the underlying similarity graphs. These metrics are pivotal in constructing a classification model to distinguish between correct and incorrect edges. To address the challenge of limited training data, we integrate an active learning mechanism tailored to cluster-specific attributes. The evaluation shows that the method outperforms existing cluster repair methods without distinguishing between duplicate-free or dirty data sources. Notably, our modified active learning strategy exhibits enhanced performance when dealing with datasets containing duplicates, showcasing its effectiveness in such scenarios.

Existing research has studied the underrepresentation of Black, Indigenous and People of Color (BIPOC) students enrolling in and graduating from science, technology, engineering, and mathematics (STEM) fields in college (e.g., Okahana et al., 2018; Rincón & Lane, 2017). However, there is a dearth of research that examines the precollegiate factors that impact whether a student majors in a STEM field (Moakler & Kim, 2014). This study uses binary logistic regression and moderated binary logistic regression to examine the influences that gender, mathematics (math) identity, science identity, career expectations at age 30, and high school STEM credit completion have on BIPOC students' postsecondary major (STEM or non-STEM). Based on the logistic regression results, our study indicates that gender, science identity, career expectations at age 30, and high school STEM credit completion significantly predict the odds of postsecondary enrollment in a STEM major. In addition, our ad hoc analysis confirms that gender moderates the relationship between science identity and the likelihood of a STEM collegiate major. These results can aid researchers and practitioners in investigating opportunities to improve STEM participation for BIPOC students.

Thin-Film Lithium Niobate (TFLN) is an emerging integrated photonic platform showing great promise due to its large second-order nonlinearity at microwave and optical frequencies, cryogenic compatibility, large piezoelectric response, and low optical loss at visible and near-infrared wavelengths. These properties enabled Mach-Zehnder interferometer-based devices to demonstrate amplitude- and in-phase/quadrature (IQ) modulation at voltage levels compatible with complementary metal-oxide-semiconductor (CMOS) electronics. Maintaining low-voltage operation requires centimeter-scale device lengths, making it challenging to realize the large-scale circuits required by ever-increasing bandwidth demands in data communications. Reduced device sizes reaching the 10 um scale are possible with photonic crystal (PhC) cavities. So far, their operation has been limited to modulation of amplitudes and required circulators or lacked cascadability. Here, we demonstrate a compact IQ modulator using two PhC cavities operating as phase shifters in a Fabry-Perot-enhanced Michelson interferometer configuration. It supports cascadable amplitude and phase modulation at GHz bandwidths with CMOS-compatible voltages. While the bandwidth limitation of resonant devices is often considered detrimental, their compactness enables dense co-integration with CMOS electronics where clock-rate-level operation (few GHz) removes power-hungry electrical time-multiplexing. Recent demonstrations of chip-scale transceivers with dense-wavelength division multiplied transceivers could be monolithically implemented and driven toward ultimate information densities using TFLN electro-optic frequency combs and our PhC IQ modulators., Comment: 27 pages, 27 figures

3D hand tracking from a monocular video is a very challenging problem due to hand interactions, occlusions, left-right hand ambiguity, and fast motion. Most existing methods rely on RGB inputs, which have severe limitations under low-light conditions and suffer from motion blur. In contrast, event cameras capture local brightness changes instead of full image frames and do not suffer from the described effects. Unfortunately, existing image-based techniques cannot be directly applied to events due to significant differences in the data modalities. In response to these challenges, this paper introduces the first framework for 3D tracking of two fast-moving and interacting hands from a single monocular event camera. Our approach tackles the left-right hand ambiguity with a novel semi-supervised feature-wise attention mechanism and integrates an intersection loss to fix hand collisions. To facilitate advances in this research domain, we release a new synthetic large-scale dataset of two interacting hands, Ev2Hands-S, and a new real benchmark with real event streams and ground-truth 3D annotations, Ev2Hands-R. Our approach outperforms existing methods in terms of the 3D reconstruction accuracy and generalises to real data under severe light conditions., Comment: 17 pages, 12 figures, 7 tables; project page: https://4dqv.mpi-inf.mpg.de/Ev2Hands/

The paper addresses Bayesian inferences in inverse problems with uncertainty quantification involving a computationally expensive forward map associated with solving a partial differential equations. To mitigate the computational cost, the paper proposes a new surrogate model informed by the physics of the problem, specifically when the forward map involves solving a linear elliptic partial differential equation. The study establishes the consistency of the posterior distribution for this surrogate model and demonstrates its effectiveness through numerical examples with synthetic data. The results indicate a substantial improvement in computational speed, reducing the processing time from several months with the exact forward map to a few minutes, while maintaining negligible loss of accuracy in the posterior distribution.

Vision-based human-to-robot handover is an important and challenging task in human-robot interaction. Recent work has attempted to train robot policies by interacting with dynamic virtual humans in simulated environments, where the policies can later be transferred to the real world. However, a major bottleneck is the reliance on human motion capture data, which is expensive to acquire and difficult to scale to arbitrary objects and human grasping motions. In this paper, we introduce a framework that can generate plausible human grasping motions suitable for training the robot. To achieve this, we propose a hand-object synthesis method that is designed to generate handover-friendly motions similar to humans. This allows us to generate synthetic training and testing data with 100x more objects than previous work. In our experiments, we show that our method trained purely with synthetic data is competitive with state-of-the-art methods that rely on real human motion data both in simulation and on a real system. In addition, we can perform evaluations on a larger scale compared to prior work. With our newly introduced test set, we show that our model can better scale to a large variety of unseen objects and human motions compared to the baselines. Project page: https://eth-ait.github.io/synthetic-handovers/

Climate change is increasing the intensity and frequency of many extreme weather events, including heatwaves, which results in increased thermal discomfort and mortality rates. While global mitigation action is undoubtedly necessary, so is climate adaptation, e.g., through climate-sensitive urban planning. Among the most promising strategies is harnessing the benefits of urban trees in shading and cooling pedestrian-level environments. Our work investigates the challenge of optimal placement of such trees. Physical simulations can estimate the radiative and thermal impact of trees on human thermal comfort but induce high computational costs. This rules out optimization of tree placements over large areas and considering effects over longer time scales. Hence, we employ neural networks to simulate the point-wise mean radiant temperatures--a driving factor of outdoor human thermal comfort--across various time scales, spanning from daily variations to extended time scales of heatwave events and even decades. To optimize tree placements, we harness the innate local effect of trees within the iterated local search framework with tailored adaptations. We show the efficacy of our approach across a wide spectrum of study areas and time scales. We believe that our approach is a step towards empowering decision-makers, urban designers and planners to proactively and effectively assess the potential of urban trees to mitigate heat stress.