Search

Achieving human-like planning and control with multimodal observations in an open world is a key milestone for more functional generalist agents. Existing approaches can handle certain long-horizon tasks in an open world. However, they still struggle when the number of open-world tasks could potentially be infinite and lack the capability to progressively enhance task completion as game time progresses. We introduce JARVIS-1, an open-world agent that can perceive multimodal input (visual observations and human instructions), generate sophisticated plans, and perform embodied control, all within the popular yet challenging open-world Minecraft universe. Specifically, we develop JARVIS-1 on top of pre-trained multimodal language models, which map visual observations and textual instructions to plans. The plans will be ultimately dispatched to the goal-conditioned controllers. We outfit JARVIS-1 with a multimodal memory, which facilitates planning using both pre-trained knowledge and its actual game survival experiences. JARVIS-1 is the existing most general agent in Minecraft, capable of completing over 200 different tasks using control and observation space similar to humans. These tasks range from short-horizon tasks, e.g., "chopping trees" to long-horizon tasks, e.g., "obtaining a diamond pickaxe". JARVIS-1 performs exceptionally well in short-horizon tasks, achieving nearly perfect performance. In the classic long-term task of $\texttt{ObtainDiamondPickaxe}$, JARVIS-1 surpasses the reliability of current state-of-the-art agents by 5 times and can successfully complete longer-horizon and more challenging tasks. The project page is available at https://craftjarvis-jarvis1.github.io., Comment: arXiv admin note: text overlap with arXiv:2206.11795 by other authors

We study the problem of building a controller that can follow open-ended instructions in open-world environments. We propose to follow reference videos as instructions, which offer expressive goal specifications while eliminating the need for expensive text-gameplay annotations. A new learning framework is derived to allow learning such instruction-following controllers from gameplay videos while producing a video instruction encoder that induces a structured goal space. We implement our agent GROOT in a simple yet effective encoder-decoder architecture based on causal transformers. We evaluate GROOT against open-world counterparts and human players on a proposed Minecraft SkillForge benchmark. The Elo ratings clearly show that GROOT is closing the human-machine gap as well as exhibiting a 70% winning rate over the best generalist agent baseline. Qualitative analysis of the induced goal space further demonstrates some interesting emergent properties, including the goal composition and complex gameplay behavior synthesis. Code and video can be found on the website https://craftjarvis-groot.github.io.

We study different restricted variations of the obnoxious facility location problem on a plane. The first is the constrained obnoxious facility location on a line segment (COFL-Line) problem. We provide an efficient algorithm for this problem that executes in $O(n ^ 2 \log k + n \log k \log (n^2 + k))$ time. Our result improves on the best known result of $O((nk)^2 \log(nk) + (n + k) \log (nk))$ time obtained by Singireddy and Basappa\cite{singireddy2022dispersing}. We also study the same problem where the facilities must be placed on a given circle (the constrained obnoxious facility location on a circle (COFL-Circ) problem). We provide an efficient algorithm for this problem that executes in $O(n ^ 2 \log k + n \log k \log (n^2 + k))$ time. Our result improves on the best known result of $O((nk)^2 \log(nk) + (n + k) \log (nk))$ time obtained by Singireddy and Basappa\cite{singireddy2022dispersing}. The third problem we study is the min-sum obnoxious facility location (MOFL) problem.We provide an efficient algorithm that executes in $O(nk\cdot \alpha(nk) \log^3 {nk})$ time, where $\alpha(.)$ is the inverse Ackermann function. The best known previous result is an $O(n^3k)$ time obtained by Singireddy and Basappa\cite{singireddy2022dispersing}.

Using the galaxy clusters from The Three Hundred Project, we define a new parameter: $\lambda_{DS}$ to describe the dynamical state of clusters, which assumes a double-Gaussian distribution in logarithm scale for our mass-complete cluster sample at $z=0$ from the dark-matter-only (DMO) run. Therefore, the threshold for distinguishing relaxed and unrelaxed clusters is naturally determined by the crossing point of the double-Gaussian fitting which has a value of $\lambda_{DS} = 3.424$. By applying $\lambda_{DS}$ with the same parameters from the DMO run to the hydro-dynamically simulated clusters (Gadget-X run and GIZMO-SIMBA run), we investigate the effect of baryons on the cluster dynamical state. We find a weak baryon-model dependence for the $\lambda_{DS}$ parameter. Finally, we study the evolution of $\lambda_{DS}$ along with clusters mass accretion history. We notice an upper limit of halo mass change $\frac{\Delta M_{200}}{M_{200}} \sim 0.12$ that don't alter the cluster dynamical state, i.e. from relaxed to unrelaxed. We define relaxation period (from the most relaxed state to disturb and relaxed again) which reflects how long the dynamical state of a cluster restores its relaxation state, and propose a correlation between this relaxation period and the strength of halo mass change $\frac{\Delta M_{200}}{M_{200}}$. With the proposed fitting to such correlation, we verify the relaxation period can be estimated from $\frac{\Delta M_{200}}{M_{200}}$ (including multi mass change peaks) with considerably small error., Comment: 13 pages, 12 figures, comments are welcome