Search

Inverse Reinforcement Learning (IRL) and Reinforcement Learning from Human Feedback (RLHF) are pivotal methodologies in reward learning, which involve inferring and shaping the underlying reward function of sequential decision-making problems based on observed human demonstrations and feedback. Most prior work in reward learning has relied on prior knowledge or assumptions about decision or preference models, potentially leading to robustness issues. In response, this paper introduces a novel linear programming (LP) framework tailored for offline reward learning. Utilizing pre-collected trajectories without online exploration, this framework estimates a feasible reward set from the primal-dual optimality conditions of a suitably designed LP, and offers an optimality guarantee with provable sample efficiency. Our LP framework also enables aligning the reward functions with human feedback, such as pairwise trajectory comparison data, while maintaining computational tractability and sample efficiency. We demonstrate that our framework potentially achieves better performance compared to the conventional maximum likelihood estimation (MLE) approach through analytical examples and numerical experiments., Comment: ICML 2024

We consider the Calogero--Moser derivative nonlinear Schr\"odinger equation (CM-DNLS), which is an $L^{2}$-critical nonlinear Schr\"odinger equation with explicit solitons, self-duality, and pseudo-conformal symmetry. More importantly, this equation is known to be completely integrable in the Hardy space $L_{+}^{2}$ and the solutions in this class are referred to as \emph{chiral} solutions. A rigorous PDE analysis of this equation with complete integrability was recently initiated by G\'erard and Lenzmann. Our main result constructs smooth, chiral, and finite energy finite-time blow-up solutions with mass arbitrarily close to that of soliton, answering the global regularity question for chiral solutions raised by G\'erard and Lenzmann. The blow-up rate obtained for these solutions is different from the pseudo-conformal rate. Our proof also gives a construction of a codimension one set of smooth finite energy initial data (but without addressing chirality) leading to the same blow-up dynamics. Our blow-up construction in the Hardy space might also be contrasted with the global well-posedness of the derivative nonlinear Schr\"odinger equation (DNLS), which is another integrable $L^{2}$-critical Schr\"odinger equation. The overall scheme of our proof is the forward construction of blow-up dynamics with modulation analysis and is not reliant on complete integrability. We begin with developing a linear theory for the near soliton dynamics. We discover a nontrivial conjugation identity, which unveils a surprising connection from the linearized (CM-DNLS) to the 1D free Schr\"odinger equation, which is a crucial ingredient for overcoming the difficulties from the non-local nonlinearity. Another principal challenge in this work, the slow decay of soliton, is overcome by introducing a trick of decomposing solutions depending on topologies, which we believe is of independent interest., Comment: 99 pages

We consider the global dynamics of finite energy solutions to energy-critical equivariant harmonic map heat flow (HMHF) and radial nonlinear heat equation (NLH). It is known that any finite energy equivariant solutions to (HMHF) decompose into finitely many harmonic maps (bubbles) separated by scales and a body map, as approaching to the maximal time of existence. Our main result for (HMHF) gives a complete classification of their dynamics for equivariance indices $D\geq3$; (i) they exist globally in time, (ii) the number of bubbles and signs are determined by the energy class of the initial data, and (iii) the scales of bubbles are asymptotically given by a universal sequence of rates up to scaling symmetry. In parallel, we also obtain a complete classification of $\dot{H}^{1}$-bounded radial solutions to (NLH) in dimensions $N\geq7$, building upon soliton resolution for such solutions. To our knowledge, this provides the first rigorous classification of bubble tree dynamics within symmetry. We introduce a new approach based on the energy method that does not rely on maximum principle. The key ingredient of the proof is a monotonicity estimate near any bubble tree configurations, which in turn requires a delicate construction of modified multi-bubble profiles also., Comment: 44 pages

Distributed Deep Learning (DDL), as a paradigm, dictates the use of GPU-based clusters as the optimal infrastructure for training large-scale Deep Neural Networks (DNNs). However, the high cost of such resources makes them inaccessible to many users. Public cloud services, particularly Spot Virtual Machines (VMs), offer a cost-effective alternative, but their unpredictable availability poses a significant challenge to the crucial checkpointing process in DDL. To address this, we introduce DeepVM, a novel solution that recommends cost-effective cluster configurations by intelligently balancing the use of Spot and On-Demand VMs. DeepVM leverages a four-stage process that analyzes instance performance using the FLOPP (FLoating-point Operations Per Price) metric, performs architecture-level analysis with linear programming, and identifies the optimal configuration for the user-specific needs. Extensive simulations and real-world deployments in the AWS environment demonstrate that DeepVM consistently outperforms other policies, reducing training costs and overall makespan. By enabling cost-effective checkpointing with Spot VMs, DeepVM opens up DDL to a wider range of users and facilitates a more efficient training of complex DNNs., Comment: 14 pages, 8 figures

An exact r-coloring of a set $S$ is a surjective function $c:S \rightarrow \{1, 2, \ldots,r\}$. A rainbow solution to an equation over $S$ is a solution such that all components are a different color. We prove that every 3-coloring of $\mathbb{N}$ with an upper density greater than $(4^s-1)/(3 \cdot 4^s)$ contains a rainbow solution to $x-y=z^k$. The rainbow number for an equation in the set $S$ is the smallest integer $r$ such that every exact $r$-coloring has a rainbow solution. We compute the rainbow numbers of $\mathbb{Z}_p$ for the equation $x-y=z^k$, where $p$ is prime and $k\geq 2$.

We consider the long time dynamics for the self-dual Chern-Simons-Schr\"odinger equation (CSS) within equivariant symmetry. (CSS) is a self-dual $L^{2}$-critical equation having pseudoconformal invariance and solitons. In this paper, we show that any $m$-equivariant, $m\geq1$, $H^{3}$ finite-time blow-up solution to (CSS) is a pseudoconformal blow-up solution. More precisely, such a solution decomposes into the sum of one modulated soliton that contracts at the pseudoconformal rate $\lambda(t)\sim T-t$, and a radiation. Applying the pseudoconformal transform in reverse, we also obtain a refined soliton resolution theorem for $m$-equivariant, $m\geq1$, $H^{3,3}$ solutions: any such solutions blow up in the pseudoconformal regime, scatter (to $0$), or scatter to a modulated soliton with some fixed scale and phase. To our knowledge, this is the first result on the full classification of the dynamics of arbitrary smooth and spatially decaying solutions, including the dynamics of scale and phase, in the class of nonlinear Schr\"odinger equations which are not known to be completely integrable. Our analysis not only builds upon the previous works, especially the soliton resolution theorem by the author, Kwon, and Oh, but also refines all steps of the arguments typically employed in the forward construction of blow-up dynamics. The key feature of the proof is that we can identify the singular and regular parts of any $H^{3}$ finite-time blow-up solutions, such that the evolution of the singular part is governed by a universal modulation dynamics while the regular part is kept $H^{3}$-bounded even up to the blow-up time. As a byproduct, we also observe that the asymptotic profile has a universal singular structure., Comment: 81 pages, to appear in J. Eur. Math. Soc

We consider the energy-critical (corotational) 1-equivariant wave maps into the two-sphere. By the seminal work [53] of Rapha\"el and Rodnianski, there is an open set of initial data whose forward-in-time development blows up in finite time with the blow-up rate $\lambda(t)=(T-t)e^{-\sqrt{|\log(T-t)|}+O(1)}$. In this paper, we show that this $e^{O(1)}$-factor in fact converges to the universal constant $2e^{-1}$, and hence these solutions contract at the universal rate $\lambda(t)=2e^{-1}(T-t)e^{-\sqrt{|\log(T-t)|}}(1+o_{t\to T}(1))$. Our proof is inspired by recent works on type-II blow-up dynamics for parabolic equations. The key improvement is in the construction of an explicit invariant subspace decomposition for the linearized operator perturbed by the scaling generator in the dispersive case, from which we obtain a more precise ODE system determining $\lambda(t)$., Comment: 66 pages

We consider the self-dual Chern-Simons-Schr\"odinger equation (CSS) under equivariant symmetry, which is a $L^{2}$-critical equation. It is known that (CSS) admits solitons and finite-time blow-up solutions. In this paper, we show soliton resolution for any solutions with equivariant data in the weighted Sobolev space $H^{1,1}$: every maximal solution decomposes into at most one modulated soliton and a radiation. A striking fact is that the nonscattering part must be a single modulated soliton. To our knowledge, this is the first result on soliton resolution in a class of nonlinear Schr\"odinger equations which are not known to be completely integrable. The key ingredient is the defocusing nature of the equation in the exterior of a soliton profile. This is a consequence of two distinctive features of (CSS): self-duality and non-local nonlinearity., Comment: 26 pages, to appear in Amer. J. Math

To address the issue of inaccurate distributions in practical stochastic systems, a minimax linear-quadratic control method is proposed using the Wasserstein metric. Our method aims to construct a control policy that is robust against errors in an empirical distribution of underlying uncertainty, by adopting an adversary that selects the worst-case distribution. The opponent receives a Wasserstein penalty proportional to the amount of deviation from the empirical distribution. A closed-form expression of the finite-horizon optimal policy pair is derived using a Riccati equation. The result is then extended to the infinite-horizon average cost setting by identifying conditions under which the Riccati recursion converges to the unique positive semi-definite solution to an algebraic Riccati equation. Our method is shown to possess several salient features including closed-loop stability, and an out-of-sample performance guarantee. We also discuss how to optimize the penalty parameter for enhancing the distributional robustness of our control policy. Last but not least, a theoretical connection to the classical $H_\infty$-method is identified from the perspective of distributional robustness., Comment: arXiv admin note: text overlap with arXiv:2003.13258

We propose a novel approach to optimize fleet management by combining multi-agent reinforcement learning with graph neural network. To provide ride-hailing service, one needs to optimize dynamic resources and demands over spatial domain. While the spatial structure was previously approximated with a regular grid, our approach represents the road network with a graph, which better reflects the underlying geometric structure. Dynamic resource allocation is formulated as multi-agent reinforcement learning, whose action-value function (Q function) is approximated with graph neural networks. We use stochastic policy update rule over the graph with deep Q-networks (DQN), and achieve superior results over the greedy policy update. We design a realistic simulator that emulates the empirical taxi call data, and confirm the effectiveness of the proposed model under various conditions., Comment: 7 pages, 4 figures

We consider the finite-time blow-up dynamics of solutions to the self-dual Chern-Simons-Schr\"odinger (CSS) equation (also referred to as the Jackiw-Pi model) near the radial soliton $Q$ with the least $L^{2}$-norm (ground state). While a formal application of pseudoconformal symmetry to $Q$ gives rise to an $L^{2}$-continuous curve of initial data sets whose solutions blow up in finite time, they all have infinite energy due to the slow spatial decay of $Q$. In this paper, we exhibit initial data sets that are smooth finite energy radial perturbations of $Q$, whose solutions blow up in finite time. Interestingly, their blow-up rate differs from the pseudoconformal rate by a power of logarithm. Applying pseudoconformal symmetry in reverse, this also yields a first example of an infinite-time blow-up solution, whose blow-up profile contracts at a logarithmic rate. Our analysis builds upon the ideas of previous works of the first two authors on (CSS) [21,22], as well as the celebrated works on energy-critical geometric equations by Merle, Rapha\"el, and Rodnianski [33,38]. A notable feature of this paper is a systematic use of nonlinear covariant conjugations by the covariant Cauchy-Riemann operators in all parts of the argument. This not only overcomes the nonlocality of the problem, which is the principal challenge for (CSS), but also simplifies the structure of nonlinearity arising in the proof., Comment: 81 pages, to appear in Ann. Sci. \'Ec. Norm. Sup\'er

We consider the self-dual Chern-Simons-Schr\"odinger equation (CSS) under equivariance symmetry. Among others, (CSS) has a static solution $Q$ and pseudoconformal symmetry. We study the conditional stability of pseudoconformal blow-up solutions $u$ such that \[ u(t,r)-\frac{e^{i\gamma_{\ast}}}{T-t}Q\Big(\frac{r}{T-t}\Big)\to u^{\ast}\quad\text{as }t\to T^{-}. \] When the equivariance index $m\geq1$, we construct a codimension one blow-up manifold, i.e. a codimension one set of initial data yielding pseudoconformal blow-up solutions. Moreover, when $m\geq3$, we establish the Lipschitz regularity of the constructed blow-up manifold (the conditional stability). This is a forward construction of blow-up solutions, as opposed to authors' previous work [25] (arXiv:1909.01055), which is a backward construction of blow-up solutions with prescribed asymptotic profiles. In view of the instability result of [25], the codimension one condition is expected to be optimal. We perform the modulation analysis with a robust energy method developed by Merle, Rapha\"el, Rodnianski, and others. One of our crucial inputs is a remarkable conjugation identity, which enables the method of supersymmetric conjugates as like Schr\"odinger maps and wave maps. It suggests how we define adapted derivatives. More interestingly, it shows a deep connection with the Schr\"odinger maps at the linearized level and allows us to find a repulsivity structure. The nonlocal nonlinearities become obstacles in many places. For instance, we need to capture non-perturbative contributions from the nonlocal nonlinearities and absorb them into phase corrections in a spirit of [25]. More importantly, we need to take a nonlinear pathway to construct modified profiles. This is suggested from [25] and becomes available thanks to the self-duality. From this, we also recognize the stable modes and unstable modes., Comment: 96 pages

New low regularity well-posedness results for the generalized Benjamin-Ono equations with quartic or higher nonlinearity and periodic boundary conditions are shown. We use the short-time Fourier transform restriction method and modified energies to overcome the derivative loss. Previously, Molinet--Ribaud established local well-posedness in $H^{1}(\mathbb{T},\mathbb{R})$ via gauge transforms. We show local existence and a priori estimates in $H^{s}(\mathbb{T},\mathbb{R})$, $s>1/2$, and local well-posedness in $H^{s}(\mathbb{T},\mathbb{R})$, $s\geq3/4$ without using gauge transforms. In case of quartic nonlinearity we prove global existence of solutions conditional upon small initial data., Comment: 46 pages, accepted to JHDE

In this paper, we propose a minimax linear-quadratic control method to address the issue of inaccurate distribution information in practical stochastic systems. To construct a control policy that is robust against errors in an empirical distribution of uncertainty, our method is to adopt an adversary, which selects the worst-case distribution. To systematically adjust the conservativeness of our method, the opponent receives a penalty proportional to the amount, measured with the Wasserstein metric, of deviation from the empirical distribution. In the finite-horizon case, using a Riccati equation, we derive a closed-form expression of the unique optimal policy and the opponent's policy that generates the worst-case distribution. This result is then extended to the infinite-horizon setting by identifying conditions under which the Riccati recursion converges to the unique positive semi-definite solution to an associated algebraic Riccati equation (ARE). The resulting optimal policy is shown to stabilize the expected value of the system state under the worst-case distribution. We also discuss that our method can be interpreted as a distributional generalization of the $H_\infty$-method.

This paper proposes a framework which is able to generate a sequence of three-dimensional human dance poses for a given music. The proposed framework consists of three components: a music feature encoder, a pose generator, and a music genre classifier. We focus on integrating these components for generating a realistic 3D human dancing move from music, which can be applied to artificial agents and humanoid robots. The trained dance pose generator, which is a generative autoregressive model, is able to synthesize a dance sequence longer than 5,000 pose frames. Experimental results of generated dance sequences from various songs show how the proposed method generates human-like dancing move to a given music. In addition, a generated 3D dance sequence is applied to a humanoid robot, showing that the proposed framework can make a robot to dance just by listening to music., Comment: 8 pages, 10 figures

We consider the self-dual Chern-Simons-Schr\"odinger equation (CSS). CSS is $L^{2}$-critical, admits solitons, and has the pseudoconformal symmetry. In this work, we consider pseudoconformal blow-up solutions under $m$-equivariance, $m\geq1$. Our result is threefold. Firstly, we construct a pseudoconformal blow-up solution $u$ with given asymptotic profile $z^{\ast}$: \[ \Big[u(t,r)-\frac{1}{|t|}Q\Big(\frac{r}{|t|}\Big)e^{-i\frac{r^{2}}{4|t|}}\Big]e^{im\theta}\to z^{\ast}\qquad\text{in }H^{1} \] as $t\to0^{-}$, where $Q(r)e^{im\theta}$ is a static solution. Secondly, we show that such blow-up solutions are unique in a suitable class. Lastly, we exhibit an instability mechanism of $u$. We construct a continuous family of solutions $u^{(\eta)}$, $0\leq\eta\ll1$, such that $u^{(0)}=u$ and for $\eta>0$, $u^{(\eta)}$ is a global scattering solution exhibiting a rotational instability as $\eta\to0^{+}$: $u^{(\eta)}$ takes an abrupt spatial rotation by the angle \[ \Big(\frac{m+1}{m}\Big)\pi \] on the time interval $|t|\lesssim\eta$. We are inspired by works in the $L^{2}$-critical NLS. In the seminal work of Bourgain and Wang (1997), they constructed such pseudoconformal blow-up solutions. Merle, Rapha\"el, and Szeftel (2013) showed an instability of Bourgain-Wang solutions. Although CSS shares many features with NLS, there are essential differences and obstacles over NLS. To name a few, the soliton profile to CSS shows a slow decay $r^{-(m+2)}$, CSS has nonlocal nonlinearities responsible for strong long-range interactions, and the instability mechanism of CSS is completely different from that of NLS. Here, the phase rotation is the main source of the instability. On the other hand, the self-dual structure of CSS is our sponsor to overcome these obstacles. We exploited the self-duality in many places such as the linearization, spectral properties, and construction of modified profiles., Comment: 117 pages