Generalized Learning from Demonstrations for Embodied AI

Bridging the gap between human capabilities and AI, this dissertation explores Learning from Demonstrations (LfD) for embodied AI. While traditional imitation learning methods struggle with generalization to new environments and complex tasks, this work introduces novel approaches that enable AI agents to learn generalized and multi-task policies from diverse and even imperfect human demonstrations.We first delve into dexterous manipulation, drawing inspiration from the remarkable versatility of human hands. We introduce a novel platform and pipeline for learning from raw human videos, enabling dexterous manipulation for high-dimensional action spaces. Additionally, we propose a generalized policy learning approach based on human hand affordances and a behavior cloning regularization technique, empowering embodied agents to manipulate novel objects.We further explore multi-task real robot learning, integrating spatial and semantic information for enhanced decision-making. We propose a method to distill semantic knowledge from a vision-language foundation model (VLM) using a 3D volumetric representation inspired by human spatial understanding. Additionally, we improve the efficiency and generalizability of multi-task learning by decoupling knowledge distillation from action learning and incorporating diffusion training for more precise sequential decision-making.Finally, we tackle the real-world challenge of imperfect demonstrations, a common issue in practical scenarios. We investigate and address the trajectory stitching problem in Decision Transformers, proposing a solution that learns a superior multi-task policy by adaptively adjusting the model's context length with suboptimal data. This work underscores the development of robust AI systems capable of effectively leveraging imperfect human demonstrations.