Search

We present VLMnav, an embodied framework to transform a Vision-Language Model (VLM) into an end-to-end navigation policy. In contrast to prior work, we do not rely on a separation between perception, planning, and control; instead, we use a VLM to directly select actions in one step. Surprisingly, we find that a VLM can be used as an end-to-end policy zero-shot, i.e., without any fine-tuning or exposure to navigation data. This makes our approach open-ended and generalizable to any downstream navigation task. We run an extensive study to evaluate the performance of our approach in comparison to baseline prompting methods. In addition, we perform a design analysis to understand the most impactful design decisions. Visual examples and code for our project can be found at https://jirl-upenn.github.io/VLMnav/

We present an approach to learn general robot manipulation priors from 3D hand-object interaction trajectories. We build a framework to use in-the-wild videos to generate sensorimotor robot trajectories. We do so by lifting both the human hand and the manipulated object in a shared 3D space and retargeting human motions to robot actions. Generative modeling on this data gives us a task-agnostic base policy. This policy captures a general yet flexible manipulation prior. We empirically demonstrate that finetuning this policy, with both reinforcement learning (RL) and behavior cloning (BC), enables sample-efficient adaptation to downstream tasks and simultaneously improves robustness and generalizability compared to prior approaches. Qualitative experiments are available at: \url{https://hgaurav2k.github.io/hop/}.

Automatically detecting and recovering from failures is an important but challenging problem for autonomous robots. Most of the recent work on learning to plan from demonstrations lacks the ability to detect and recover from errors in the absence of an explicit state representation and/or a (sub-) goal check function. We propose an approach (blending learning with symbolic search) for automated error discovery and recovery, without needing annotated data of failures. Central to our approach is a neuro-symbolic state representation, in the form of dense scene graph, structured based on the objects present within the environment. This enables efficient learning of the transition function and a discriminator that not only identifies failures but also localizes them facilitating fast re-planning via computation of heuristic distance function. We also present an anytime version of our algorithm, where instead of recovering to the last correct state, we search for a sub-goal in the original plan minimizing the total distance to the goal given a re-planning budget. Experiments on a physics simulator with a variety of simulated failures show the effectiveness of our approach compared to existing baselines, both in terms of efficiency as well as accuracy of our recovery mechanism., Comment: This work has been submitted to the IEEE for possible publication

Our goal is to enable embodied agents to learn inductively generalizable spatial concepts, e.g., learning staircase as an inductive composition of towers of increasing height. Given a human demonstration, we seek a learning architecture that infers a succinct ${program}$ representation that explains the observed instance. Additionally, the approach should generalize inductively to novel structures of different sizes or complex structures expressed as a hierarchical composition of previously learned concepts. Existing approaches that use code generation capabilities of pre-trained large (visual) language models, as well as purely neural models, show poor generalization to a-priori unseen complex concepts. Our key insight is to factor inductive concept learning as (i) ${\it Sketch:}$ detecting and inferring a coarse signature of a new concept (ii) ${\it Plan:}$ performing MCTS search over grounded action sequences (iii) ${\it Generalize:}$ abstracting out grounded plans as inductive programs. Our pipeline facilitates generalization and modular reuse, enabling continual concept learning. Our approach combines the benefits of the code generation ability of large language models (LLM) along with grounded neural representations, resulting in neuro-symbolic programs that show stronger inductive generalization on the task of constructing complex structures in relation to LLM-only and neural-only approaches. Furthermore, we demonstrate reasoning and planning capabilities with learned concepts for embodied instruction following.

One of the central challenge for extracting governing principles of dynamical system via Dynamic Mode Decomposition (DMD) is about the limit data availability or formally called as Limited Data Acquisition in the present paper. In the interest of discovering the governing principles for a dynamical system with limited data acquisition, we provide a variant of Kernelized Extended DMD (KeDMD) based on the Koopman operator which employ the notion of Gaussian random matrix to recover the dominant Koopman modes for the standard fluid flow across cylinder experiment. It turns out that the traditional kernel function, Gaussian Radial Basis Function Kernel, unfortunately, is not able to generate the desired Koopman modes in the scenario of executing KeDMD with limited data acquisition. However, the Laplacian Kernel Function successfully generates the desired Koopman modes when limited data is provided in terms of data-set snapshot for the aforementioned experiment and this manuscripts serves the purpose of reporting these exciting experimental insights. This paper also explores the functionality of the Koopman operator when it interacts with the reproducing kernel Hilbert space (RKHS) that arises from the normalized probability Lebesgue measure $d\mu_{\sigma,1,\mathbb{C}^n}(z)=(2\pi\sigma^2)^{-n}\exp\left(-\frac{\|z\|_2}{\sigma}\right)dV(z)$ when it is embedded in $L^2-$sense for the holomorphic functions over $\mathbb{C}^n$, in the aim of determining the Koopman modes for fluid flow across cylinder experiment. We explore the operator-theoretic characterizations of the Koopman operator on the RKHS generated by the normalized Laplacian measure $d\mu_{\sigma,1,\mathbb{C}^n}(z)$ in the $L^2-$sense. In doing so, we provide the compactification & closable characterization of Koopman operator over the RKHS generated by the normalized Laplacian measure in the $L^2-$sense., Comment: 52 Pages, 16 Figures, JULIA Coding Result for Dynamic Mode Decomposition, Part of this work selected for 42nd Annual Dynamic Days 2024 Conference (January 8 to 10) at University of California, Davis

Gaussian Radial Basis Function (RBF) Kernels are the most-often-employed kernels in artificial intelligence and machine learning routines for providing optimally-best results in contrast to their respective counter-parts. However, a little is known about the application of the Generalized Gaussian Radial Basis Function on various machine learning algorithms namely, kernel regression, support vector machine (SVM) and pattern-recognition via neural networks. The results that are yielded by Generalized Gaussian RBF in the kernel sense outperforms in stark contrast to Gaussian RBF Kernel, Sigmoid Function and ReLU Function. This manuscript demonstrates the application of the Generalized Gaussian RBF in the kernel sense on the aforementioned machine learning routines along with the comparisons against the aforementioned functions as well., Comment: 20 pages, MATLAB CODE, 11 figures, Results presented in AMS Spring Eastern Sectional Meeting on April 2023

Adversarial purification using generative models demonstrates strong adversarial defense performance. These methods are classifier and attack-agnostic, making them versatile but often computationally intensive. Recent strides in diffusion and score networks have improved image generation and, by extension, adversarial purification. Another highly efficient class of adversarial defense methods known as adversarial training requires specific knowledge of attack vectors, forcing them to be trained extensively on adversarial examples. To overcome these limitations, we introduce a new framework, namely Language Guided Adversarial Purification (LGAP), utilizing pre-trained diffusion models and caption generators to defend against adversarial attacks. Given an input image, our method first generates a caption, which is then used to guide the adversarial purification process through a diffusion network. Our approach has been evaluated against strong adversarial attacks, proving its effectiveness in enhancing adversarial robustness. Our results indicate that LGAP outperforms most existing adversarial defense techniques without requiring specialized network training. This underscores the generalizability of models trained on large datasets, highlighting a promising direction for further research.

The performance of large language models (LLMs) on existing reasoning benchmarks has significantly improved over the past years. In response, we present JEEBench, a considerably more challenging benchmark dataset for evaluating the problem solving abilities of LLMs. We curate 515 challenging pre-engineering mathematics, physics and chemistry problems from the highly competitive IIT JEE-Advanced exam. Long-horizon reasoning on top of deep in-domain knowledge is essential for solving problems in this benchmark. Our evaluation on various open-source and proprietary models reveals that the highest performance, even after using techniques like self-consistency, self-refinement and chain-of-thought prompting, is less than 40%. The typical failure modes of GPT-4, the best model, are errors in algebraic manipulation, difficulty in grounding abstract concepts into mathematical equations accurately and failure in retrieving relevant domain-specific concepts. We also observe that by mere prompting, GPT-4 is unable to assess risk introduced by negative marking for incorrect answers. For this, we develop a post-hoc confidence-thresholding method over self-consistency, which enables effective response selection. We hope that our challenging benchmark will guide future re-search in problem-solving using LLMs., Comment: EMNLP 2023

Given a natural language instruction and an input scene, our goal is to train a model to output a manipulation program that can be executed by the robot. Prior approaches for this task possess one of the following limitations: (i) rely on hand-coded symbols for concepts limiting generalization beyond those seen during training [1] (ii) infer action sequences from instructions but require dense sub-goal supervision [2] or (iii) lack semantics required for deeper object-centric reasoning inherent in interpreting complex instructions [3]. In contrast, our approach can handle linguistic as well as perceptual variations, end-to-end trainable and requires no intermediate supervision. The proposed model uses symbolic reasoning constructs that operate on a latent neural object-centric representation, allowing for deeper reasoning over the input scene. Central to our approach is a modular structure consisting of a hierarchical instruction parser and an action simulator to learn disentangled action representations. Our experiments on a simulated environment with a 7-DOF manipulator, consisting of instructions with varying number of steps and scenes with different number of objects, demonstrate that our model is robust to such variations and significantly outperforms baselines, particularly in the generalization settings. The code, dataset and experiment videos are available at https://nsrmp.github.io, Comment: International Conference on Robotics and Automation (ICRA), 2023

We propose a systematic method to construct crystal-based molecular structures often needed as input for computational chemistry studies. These structures include crystal ``slabs" with periodic boundary conditions (PBCs) and non-periodic solids such as Wulff structures. We also introduce a method to build crystal slabs with orthogonal PBC vectors. These methods are integrated into our code, Los Alamos Crystal Cut (LCC), which is open source and thus fully available to the community. Examples showing the use of these methods are given throughout the manuscript.

We study the numerical range of the Weighted Composition Operator over the Mittag-Leffler space of entire functions., Comment: Has critical issues with the basic definition and machinery of theme!

We present important characterizations of the Weighted Composition Operator over the Mittag Leffler space of entire functions. These characterizations include the Hilbert-Schmidt and Unitary char-acterizations of the Weighted Composition Operator over the Mittag-Leffler space. We also see the result for the spectrum of Weighted com-position operator over the Mittag-Leffler space, Comment: Results are very specific to the problem and unable to encompass the essential outline of the problem!

We present the Littlewood-Paley Identity for the Mittag-Leffler space ML2(C; {\alpha}) of entire functions. We also briefly demonstrate the connection between the Littlewood-Paley Identity and the compactness of the weighted composition operator on ML2(C; {\alpha})., Comment: The principle investigation of the paper was already performed with better mathematics then in the present paper!

We present the upper bound of the essential norm of the composition operator over the Polylogarithmic Hardy space PL2(D;s).The results involve the Nevanlinna counting function for PL2(D;s). We first prove the Littlewood-Paley Identity for PL2(D;s) which leads to the Nevanlinna counting function for PL2(D;s). With all these results, not only we get the upper bound of the essential norm of the composition operator over PL2(D;s) but also we get an upper bound in terms of the angular derivative and essential norm of composition operator over the Hardy space H2., Comment: Certain sections of the paper doesn't follow right explanation. Further investigation is needed

The Hilbert spaces are common. But the direct connection between them is rare. The aim of this paper is to establish a direct relation among the three Hilbert spaces Hardy, Bergman and Dirichlet without defining any of the Hilbert space in weighted sense. In order to accomplish this goal, this paper develops the Littlewood-Paley type Identities for Bergman and Dirichlet space. After defining these identities, the vision of connecting all the three Hilbert spaces via a direct connection is achieved., Comment: Definition fails crucially at the basic level. Also results are drawn at early-stage of the investigation