Search

Image-based Joint-Embedding Predictive Architecture (IJEPA) offers an attractive alternative to Masked Autoencoder (MAE) for representation learning using the Masked Image Modeling framework. IJEPA drives representations to capture useful semantic information by predicting in latent rather than input space. However, IJEPA relies on carefully designed context and target windows to avoid representational collapse. The encoder modules in IJEPA cannot adaptively modulate the type of predicted and/or target features based on the feasibility of the masked prediction task as they are not given sufficient information of both context and targets. Based on the intuition that in natural images, information has a strong spatial bias with spatially local regions being highly predictive of one another compared to distant ones. We condition the target encoder and context encoder modules in IJEPA with positions of context and target windows respectively. Our "conditional" encoders show performance gains on several image classification benchmark datasets, improved robustness to context window size and sample-efficiency during pretraining., Comment: NeurIPS 2024 Workshop on Self-Supervised Learning - Theory and Practice. Comments welcome!

Future communication systems are anticipated to facilitate applications requiring high data transmission rates while maintaining energy efficiency. Hexagonal quadrature amplitude modulation (HQAM) offers this owing to its compact symbol arrangement within the two-dimensional (2D) plane. Building on the limitations of the current approaches, this letter presents a straightforward and precise approximation for calculating the symbol error probability (SEP) of HQAM in additive white Gaussian noise (AWGN) channel. The analytical and simulation results align across several signal-to-noise ratios (SNR). In addition, the proposed approximation is effective for accurately estimating the SEP of HQAM in scenarios with slow-fading, including those subject to Rayleigh fading., Comment: 5 pages, 7 figures

This study explores using embedding rank as an unsupervised evaluation metric for general-purpose speech encoders trained via self-supervised learning (SSL). Traditionally, assessing the performance of these encoders is resource-intensive and requires labeled data from the downstream tasks. Inspired by the vision domain, where embedding rank has shown promise for evaluating image encoders without tuning on labeled downstream data, this work examines its applicability in the speech domain, considering the temporal nature of the signals. The findings indicate rank correlates with downstream performance within encoder layers across various downstream tasks and for in- and out-of-domain scenarios. However, rank does not reliably predict the best-performing layer for specific downstream tasks, as lower-ranked layers can outperform higher-ranked ones. Despite this limitation, the results suggest that embedding rank can be a valuable tool for monitoring training progress in SSL speech models, offering a less resource-demanding alternative to traditional evaluation methods., Comment: Submitted to ICASSP 2025

Despite researchers having extensively studied various ways to track body pose on-the-go, most prior work does not take into account wheelchair users, leading to poor tracking performance. Wheelchair users could greatly benefit from this pose information to prevent injuries, monitor their health, identify environmental accessibility barriers, and interact with gaming and VR experiences. In this work, we present WheelPoser, a real-time pose estimation system specifically designed for wheelchair users. Our system uses only four strategically placed IMUs on the user's body and wheelchair, making it far more practical than prior systems using cameras and dense IMU arrays. WheelPoser is able to track a wheelchair user's pose with a mean joint angle error of 14.30 degrees and a mean joint position error of 6.74 cm, more than three times better than similar systems using sparse IMUs. To train our system, we collect a novel WheelPoser-IMU dataset, consisting of 167 minutes of paired IMU sensor and motion capture data of people in wheelchairs, including wheelchair-specific motions such as propulsion and pressure relief. Finally, we explore the potential application space enabled by our system and discuss future opportunities. Open-source code, models, and dataset can be found here: https://github.com/axle-lab/WheelPoser., Comment: Accepted by ASSETS 2024

Spatial disorientation is a leading cause of fatal aircraft accidents. This paper explores the potential of AI agents to aid pilots in maintaining balance and preventing unrecoverable losses of control by offering cues and corrective measures that ameliorate spatial disorientation. A multi-axis rotation system (MARS) was used to gather data from human subjects self-balancing in a spaceflight analog condition. We trained models over this data to create "digital twins" that exemplified performance characteristics of humans with different proficiency levels. We then trained various reinforcement learning and deep learning models to offer corrective cues if loss of control is predicted. Digital twins and assistant models then co-performed a virtual inverted pendulum (VIP) programmed with identical physics. From these simulations, we picked the 5 best-performing assistants based on task metrics such as crash frequency and mean distance from the direction of balance. These were used in a co-performance study with 20 new human subjects performing a version of the VIP task with degraded spatial information. We show that certain AI assistants were able to improve human performance and that reinforcement-learning based assistants were objectively more effective but rated as less trusted and preferable by humans., Comment: 10 pages, To be published in the International Conference on Human-Agent Interaction (HAI '24) proceedings

A catalytic Turing machine is a variant of a Turing machine in which there exists an auxiliary tape in addition to the input tape and the work tape. This auxiliary tape is initially filled with arbitrary content. The machine can read and write on the auxiliary tape, but it is constrained to restore its initial content when it halts. Studying such a model and finding its powers and limitations has practical applications. In this paper, we study catalytic Turing machines with O(log n)-sized work tape and polynomial-sized auxiliary tape that are allowed to lose at most constant many bits of the auxiliary tape when they halt. We show that such catalytic Turing machines can only decide the same set of languages as standard catalytic Turing machines with the same size work and auxiliary tape.

Recent video masked autoencoder (MAE) works have designed improved masking algorithms focused on saliency. These works leverage visual cues such as motion to mask the most salient regions. However, the robustness of such visual cues depends on how often input videos match underlying assumptions. On the other hand, natural language description is an information dense representation of video that implicitly captures saliency without requiring modality-specific assumptions, and has not been explored yet for video MAE. To this end, we introduce a novel text-guided masking algorithm (TGM) that masks the video regions with highest correspondence to paired captions. Without leveraging any explicit visual cues for saliency, our TGM is competitive with state-of-the-art masking algorithms such as motion-guided masking. To further benefit from the semantics of natural language for masked reconstruction, we next introduce a unified framework for joint MAE and masked video-text contrastive learning. We show that across existing masking algorithms, unifying MAE and masked video-text contrastive learning improves downstream performance compared to pure MAE on a variety of video recognition tasks, especially for linear probe. Within this unified framework, our TGM achieves the best relative performance on five action recognition and one egocentric datasets, highlighting the complementary nature of natural language for masked video modeling., Comment: Accepted to ECCV 2024

Two competing paradigms exist for self-supervised learning of data representations. Joint Embedding Predictive Architecture (JEPA) is a class of architectures in which semantically similar inputs are encoded into representations that are predictive of each other. A recent successful approach that falls under the JEPA framework is self-distillation, where an online encoder is trained to predict the output of the target encoder, sometimes using a lightweight predictor network. This is contrasted with the Masked AutoEncoder (MAE) paradigm, where an encoder and decoder are trained to reconstruct missing parts of the input in the data space rather, than its latent representation. A common motivation for using the JEPA approach over MAE is that the JEPA objective prioritizes abstract features over fine-grained pixel information (which can be unpredictable and uninformative). In this work, we seek to understand the mechanism behind this empirical observation by analyzing the training dynamics of deep linear models. We uncover a surprising mechanism: in a simplified linear setting where both approaches learn similar representations, JEPAs are biased to learn high-influence features, i.e., features characterized by having high regression coefficients. Our results point to a distinct implicit bias of predicting in latent space that may shed light on its success in practice., Comment: Technical report

Artificial neural networks play a crucial role in machine learning and there is a need to improve their performance. This paper presents FOXANN, a novel classification model that combines the recently developed Fox optimizer with ANN to solve ML problems. Fox optimizer replaces the backpropagation algorithm in ANN; optimizes synaptic weights; and achieves high classification accuracy with a minimum loss, improved model generalization, and interpretability. The performance of FOXANN is evaluated on three standard datasets: Iris Flower, Breast Cancer Wisconsin, and Wine. The results presented in this paper are derived from 100 epochs using 10-fold cross-validation, ensuring that all dataset samples are involved in both the training and validation stages. Moreover, the results show that FOXANN outperforms traditional ANN and logistic regression methods as well as other models proposed in the literature such as ABC-ANN, ABC-MNN, CROANN, and PSO-DNN, achieving a higher accuracy of 0.9969 and a lower validation loss of 0.0028. These results demonstrate that FOXANN is more effective than traditional methods and other proposed models across standard datasets. Thus, FOXANN effectively addresses the challenges in ML algorithms and improves classification performance., Comment: 12 pages

Finding a simple path of even length between two designated vertices in a directed graph is a fundamental NP-complete problem known as the EvenPath problem. Nedev proved in 1999, that for directed planar graphs, the problem can be solved in polynomial time. More than two decades since then, we make the first progress in extending the tractable classes of graphs for this problem. We give a polynomial time algorithm to solve the EvenPath problem for classes of H-minor-free directed graphs,1 where H is a single-crossing graph. We make two new technical contributions along the way, that might be of independent interest. The first, and perhaps our main, contribution is the construction of small, planar, parity-mimicking networks. These are graphs that mimic parities of all possible paths between a designated set of terminals of the original graph. Finding vertex disjoint paths between given source-destination pairs of vertices is another fundamental problem, known to be NP-complete in directed graphs, though known to be tractable in planar directed graphs. We encounter a natural variant of this problem, that of finding disjoint paths between given pairs of vertices, but with constraints on parity of the total length of paths. The other significant contribution of our paper is to give a polynomial time algorithm for the 3-disjoint paths with total parity problem, in directed planar graphs with some restrictions (and also in directed graphs of bounded treewidth).

Goal-oriented communication (GoC) is a form of semantic communication where the effectiveness of information transmission is measured by its impact on achieving the desired goal. In the context of the Internet of Things (IoT), GoC can make IoT sensors to selectively transmit data pertinent to the intended goals of the receiver. Therefore, GoC holds significant value for IoT networks as it facilitates timely decision-making at the receiver, reduces network congestion, and enhances spectral efficiency. In this paper, we consider a scenario where an edge node polls sensors monitoring the state of a non-linear dynamic system (NLDS) to respond to the queries of several clients. Our work delves into the foregoing GoC problem, which we term goal-oriented scheduling (GoS). Our proposed GoS utilizes deep reinforcement learning (DRL) with meticulously devised action space, state space, and reward function. The devised action space and reward function play a pivotal role in reducing the number of sensor transmissions. Meanwhile, the devised state space empowers our DRL scheduler to poll the sensor whose observation is expected to minimize the mean square error (MSE) of the query responses. Our numerical analysis demonstrates that the proposed GoS can either effectively minimize the query response MSE further or obtain a resembling MSE compared to benchmark scheduling methods, depending on the type of query. Furthermore, the proposed GoS proves to be energy-efficient for the sensors and of lower complexity compared to benchmark scheduling methods.

This paper summarizes our acoustic modeling efforts in the Johns Hopkins University speech recognition system for the CHiME-5 challenge to recognize highly-overlapped dinner party speech recorded by multiple microphone arrays. We explore data augmentation approaches, neural network architectures, front-end speech dereverberation, beamforming and robust i-vector extraction with comparisons of our in-house implementations and publicly available tools. We finally achieved a word error rate of 69.4% on the development set, which is a 11.7% absolute improvement over the previous baseline of 81.1%, and release this improved baseline with refined techniques/tools as an advanced CHiME-5 recipe., Comment: Published in: ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

We explore a generalization of nonrelativistic fermionic statistics that interpolates between bosons and fermions, in which up to $K$ particles may occupy a single-particle state. We show that it can be mapped exactly to $K$ flavors of fermions with imaginary polarization. In particular, for $K\!=\!2$, we use such a mapping to derive the virial coefficients and relate them to those of conventional spin-1/2 fermions in an exact fashion. We also use the mapping to derive next-to-leading-order perturbative results for the pressure equation of state. Our results indicate that the $K\!=\!2$ particles are more strongly coupled than conventional spin-$1/2$ fermions, as measured by the interaction effects on the virial expansion and on the pressure equation of state. In the regime set by the unitary limit, the proposed $K\!=\!2$ deformation represents a universal many-body system whose properties remain largely unknown. In particular, the system can be expected to become superfluid at a critical temperature $T_c$ higher than that of the unitary limit. We suggest it may be possible to realize this system experimentally by engineering a polarized coupling to an electrostatic potential. Finally, we show that the $K\!=\!2$ system does not display a sign problem for determinantal Monte Carlo calculations, which indicates that $T_c$ can at least in principle be calculated with conventional methods., Comment: 7 pages, 3 figures

Whilst holding great promise for low noise, ease of operation and networking, useful photonic quantum computing has been precluded by the need for beyond-state-of-the-art components, manufactured by the millions. Here we introduce a manufacturable platform for quantum computing with photons. We benchmark a set of monolithically-integrated silicon photonics-based modules to generate, manipulate, network, and detect photonic qubits, demonstrating dual-rail photonic qubits with $99.98\% \pm 0.01\%$ state preparation and measurement fidelity, Hong-Ou-Mandel quantum interference between independent photon sources with $99.50\%\pm0.25\%$ visibility, two-qubit fusion with $99.22\%\pm0.12\%$ fidelity, and a chip-to-chip qubit interconnect with $99.72\%\pm0.04\%$ fidelity, not accounting for loss. In addition, we preview a selection of next generation technologies, demonstrating low-loss silicon nitride waveguides and components, fabrication-tolerant photon sources, high-efficiency photon-number-resolving detectors, low-loss chip-to-fiber coupling, and barium titanate electro-optic phase shifters., Comment: 8 pages, 5 figures

Majorana bound states emerge in topological superconductors as zero-energy edge states exhibiting spatial nonlocality. Despite the enormous advances, the detection of Majorana bound states is still challenging mainly because topologically trivial Andreev bound states produce similar signatures. In this work we consider a topological superconductor with Majorana bound states coupled to quantum dots and investigate the dynamics of their quantum correlations with the aim to explore their entanglement properties. In particular, we characterize entanglement by using concurrence and discord, which are also complemented by the entanglement dynamics and return probability. We find that Majorana bound states at truly zero energy can transform an initially entangled system into its classical state, while they can create maximally entangled states at a finite energy overlap. Interestingly, we show that the system can generate a maximally entangled state between MBSs and a quantum dot by simply controlling the Majorana nonlocality. We demonstrate that these results hold in the scenarios when the initial state is either maximally entangled or separable, albeit in the latter maximally entangled states are achieved in the long time dynamics. Furthermore, we contrast our findings with those produced by a regular fermion and obtain very distinct entanglement signatures. Our work offers an alternative approach to characterize Majorana bound states, which can be also useful towards their utilization for quantum information tasks., Comment: 14 pages, 5 figures

Connectionist temporal classification (CTC) models are known to have peaky output distributions. Such behavior is not a problem for automatic speech recognition (ASR), but it can cause inaccurate forced alignments (FA), especially at finer granularity, e.g., phoneme level. This paper aims at alleviating the peaky behavior for CTC and improve its suitability for forced alignment generation, by leveraging label priors, so that the scores of alignment paths containing fewer blanks are boosted and maximized during training. As a result, our CTC model produces less peaky posteriors and is able to more accurately predict the offset of the tokens besides their onset. It outperforms the standard CTC model and a heuristics-based approach for obtaining CTC's token offset timestamps by 12-40% in phoneme and word boundary errors (PBE and WBE) measured on the Buckeye and TIMIT data. Compared with the most widely used FA toolkit Montreal Forced Aligner (MFA), our method performs similarly on PBE/WBE on Buckeye, yet falls behind MFA on TIMIT. Nevertheless, our method has a much simpler training pipeline and better runtime efficiency. Our training recipe and pretrained model are released in TorchAudio., Comment: Accepted by ICASSP 2024. Github repo: https://github.com/huangruizhe/audio/tree/aligner_label_priors

Exploring new strategies for controlling the magnetic domain propagation is the key to realize ultrafast, high-density domain wall-based memory and logic devices for next generation computing. These strategies include strain modulation in multiferroic devices, geometric confinement and area-selective pinning of domain wall. 2D Van der Waals materials introduce localized modifications to the interfacial magnetic order, enabling control over the propagation of magnetic domains. Here, using Lorentz-Transmission Electron Microscopy (L-TEM) along with the Modified Transport of Intensity equations (MTIE), we demonstrate controlled domain expansion with in-situ magnetic field in a ferromagnet (Permalloy, NiFe) interfacing with a 2D Van der Waals material Graphene (Gr). The Gr/NiFe interface exhibits distinctive domain expansion rate with magnetic field selectively near the interface which is further analyzed using micromagnetic simulations. Our findings are crucial for comprehending direct visualization of interface controlled magnetic domain expansion, offering insights for developing future domain wall-based technology., Comment: Merged Manuscript and supplementary file. Submitted to Communications Physics (under review)

Hand gesture recognition is becoming a more prevalent mode of human-computer interaction, especially as cameras proliferate across everyday devices. Despite continued progress in this field, gesture customization is often underexplored. Customization is crucial since it enables users to define and demonstrate gestures that are more natural, memorable, and accessible. However, customization requires efficient usage of user-provided data. We introduce a method that enables users to easily design bespoke gestures with a monocular camera from one demonstration. We employ transformers and meta-learning techniques to address few-shot learning challenges. Unlike prior work, our method supports any combination of one-handed, two-handed, static, and dynamic gestures, including different viewpoints, and the ability to handle irrelevant hand movements. We implement three real-world applications using our customization method, conduct a user study, and achieve up to 94% average recognition accuracy from one demonstration. Our work provides a viable path for vision-based gesture customization, laying the foundation for future advancements in this domain., Comment: 2024 (UIST' 24). USA, 14 pages

With a motive of ubiquitous connectivity over the globe with enhanced spectral efficiency, intelligent reflecting surfaces (IRS) integrated satellite-terrestrial communications is a topic of research interest in an infrastructure-deficient remote terrains. In line with this vision, this paper entails the performance analysis of satellite-terrestrial networks leveraging both aerial and terrestrial IRS nodes, with the support of high altitude platforms over diverse fading channels including shadowed Rician, Rician, and Nakagami-$m$ fading channels. The merits of IRS in enhancing spectral efficiency is analyzed through closed-form expressions of outage probability and ergodic rate. Further, the average symbol error rate analysis for the higher-order quadrature amplitude modulation (QAM) schemes such as hexagonal QAM, rectangular QAM, cross QAM, and square QAM is performed. Practical constraints like antenna gains, path loss, and link fading are considered to characterize the satellite terrestrial links. Finally, a comparison between the high-altitude platforms based IRS node and terrestrial IRS nodes is performed and various insights are drawn under various fading scenarios and path loss conditions. This paper contribute towards understanding and potential implementation of IRS-integrated satellite-terrestrial networks for efficient and reliable communication.

Joint embedding (JE) architectures have emerged as a promising avenue for acquiring transferable data representations. A key obstacle to using JE methods, however, is the inherent challenge of evaluating learned representations without access to a downstream task, and an annotated dataset. Without efficient and reliable evaluation, it is difficult to iterate on architectural and training choices for JE methods. In this paper, we introduce LiDAR (Linear Discriminant Analysis Rank), a metric designed to measure the quality of representations within JE architectures. Our metric addresses several shortcomings of recent approaches based on feature covariance rank by discriminating between informative and uninformative features. In essence, LiDAR quantifies the rank of the Linear Discriminant Analysis (LDA) matrix associated with the surrogate SSL task -- a measure that intuitively captures the information content as it pertains to solving the SSL task. We empirically demonstrate that LiDAR significantly surpasses naive rank based approaches in its predictive power of optimal hyperparameters. Our proposed criterion presents a more robust and intuitive means of assessing the quality of representations within JE architectures, which we hope facilitates broader adoption of these powerful techniques in various domains., Comment: Technical report