Search

Reinforcement learning algorithms typically rely on the assumption that the environment dynamics and value function can be expressed in terms of a Markovian state representation. However, when state information is only partially observable, how can an agent learn such a state representation, and how can it detect when it has found one? We introduce a metric that can accomplish both objectives, without requiring access to--or knowledge of--an underlying, unobservable state space. Our metric, the $\lambda$-discrepancy, is the difference between two distinct temporal difference (TD) value estimates, each computed using TD($\lambda$) with a different value of $\lambda$. Since TD($\lambda$=0) makes an implicit Markov assumption and TD($\lambda$=1) does not, a discrepancy between these estimates is a potential indicator of a non-Markovian state representation. Indeed, we prove that the $\lambda$-discrepancy is exactly zero for all Markov decision processes and almost always non-zero for a broad class of partially observable environments. We also demonstrate empirically that, once detected, minimizing the $\lambda$-discrepancy can help with learning a memory function to mitigate the corresponding partial observability. We then train a reinforcement learning agent that simultaneously constructs two recurrent value networks with different $\lambda$ parameters and minimizes the difference between them as an auxiliary loss. The approach scales to challenging partially observable domains, where the resulting agent frequently performs significantly better (and never performs worse) than a baseline recurrent agent with only a single value network., Comment: GitHub URL: https://github.com/brownirl/lambda_discrepancy; Videos: https://lambda-discrepancy.github.io/

Additive manufacturing is an expanding multidisciplinary field encompassing applications including medical devices, aerospace components, microfabrication strategies, and artificial organs. Among additive manufacturing approaches, light-based printing technologies, including two-photon polymerization, projection micro stereolithography, and volumetric printing, have garnered significant attention due to their speed, resolution and/or potential applications for biofabrication. In this study, we introduce dynamic interface printing (DIP), a new 3D printing approach that leverages an acoustically modulated, constrained air-liquid boundary to rapidly generate cm-scale three-dimensional structures within tens of seconds. Distinct from volumetric approaches, this process eliminates the need for intricate feedback systems, specialized chemistry, or complex optics while maintaining rapid printing speeds. We demonstrate the versatility of this technique across a broad array of materials and intricate geometries, including those that would be impossible to print via conventional layer-by-layer methods. In doing so, we demonstrate the rapid fabrication of complex structures in-situ, overprinting, structural parallelisation, and biofabrication utility. Moreover, we showcase that the formation of surface waves at this boundary enables enhanced mass transport, material flexibility, and permits three-dimensional particle patterning. We therefore anticipate that this approach will be invaluable for applications where high resolution, scalable throughput, and biocompatible printing is required., Comment: 63 Pages, 4 Figures, 18 Supplementary Figures

Neural MMO 2.0 is a massively multi-agent environment for reinforcement learning research. The key feature of this new version is a flexible task system that allows users to define a broad range of objectives and reward signals. We challenge researchers to train agents capable of generalizing to tasks, maps, and opponents never seen during training. Neural MMO features procedurally generated maps with 128 agents in the standard setting and support for up to. Version 2.0 is a complete rewrite of its predecessor with three-fold improved performance and compatibility with CleanRL. We release the platform as free and open-source software with comprehensive documentation available at neuralmmo.github.io and an active community Discord. To spark initial research on this new platform, we are concurrently running a competition at NeurIPS 2023.

This report builds on work of the Delphi Project on the Changing Faculty and Student Success, initiated in 2012, that has documented changes in the academic profession and its implications for higher education. While the Delphi Project has aimed to address specific issues--for example, by conducting and disseminating research on how faculty working conditions shape their ability to perform as teachers--this report takes a look at how broader trends and issues affecting faculty are reshaping the profession. In this report, the authors explore the issues and trends that have affected faculty in the United States over the past year. Their goal in developing this annual report is to provide a snapshot from varied sources about the state of the profession. Topics covered include the future job market, faculty accountability policies, shared governance, tenure, unionization, non-tenure track faculty, salaries, and benefits. The report also addresses issues specifically relevant for women and faculty of color in particular, along with a focus on trends in teaching and learning in general. A section of the report is also dedicated to challenging common myths about faculty. [This report is a resource created by The Delphi Project on the Changing Faculty and Student Success, an initiative of the Pullias Center for Higher Education at the University of Southern California.]