Search

Self-play methods have demonstrated remarkable success in enhancing model capabilities across various domains. In the context of Reinforcement Learning from Human Feedback (RLHF), self-play not only boosts Large Language Model (LLM) performance but also overcomes the limitations of traditional Bradley-Terry (BT) model assumptions by finding the Nash equilibrium (NE) of a preference-based, two-player constant-sum game. However, existing methods either guarantee only average-iterate convergence, incurring high storage and inference costs, or converge to the NE of a regularized game, failing to accurately reflect true human preferences. In this paper, we introduce Magnetic Preference Optimization (MPO), a novel approach capable of achieving last-iterate convergence to the NE of the original game, effectively overcoming the limitations of existing methods. Building upon Magnetic Mirror Descent (MMD), MPO attains a linear convergence rate, making it particularly suitable for fine-tuning LLMs. To ensure our algorithm is both theoretically sound and practically viable, we present a simple yet effective implementation that adapts the theoretical insights to the RLHF setting. Empirical results demonstrate that MPO can significantly enhance the performance of LLMs, highlighting the potential of self-play methods in alignment., Comment: Under review

The ex ante equilibrium for two-team zero-sum games, where agents within each team collaborate to compete against the opposing team, is known to be the best a team can do for coordination. Many existing works on ex ante equilibrium solutions are aiming to extend the scope of ex ante equilibrium solving to large-scale team games based on Policy Space Response Oracle (PSRO). However, the joint team policy space constructed by the most prominent method, Team PSRO, cannot cover the entire team policy space in heterogeneous team games where teammates play distinct roles. Such insufficient policy expressiveness causes Team PSRO to be trapped into a sub-optimal ex ante equilibrium with significantly higher exploitability and never converges to the global ex ante equilibrium. To find the global ex ante equilibrium without introducing additional computational complexity, we first parameterize heterogeneous policies for teammates, and we prove that optimizing the heterogeneous teammates' policies sequentially can guarantee a monotonic improvement in team rewards. We further propose Heterogeneous-PSRO (H-PSRO), a novel framework for heterogeneous team games, which integrates the sequential correlation mechanism into the PSRO framework and serves as the first PSRO framework for heterogeneous team games. We prove that H-PSRO achieves lower exploitability than Team PSRO in heterogeneous team games. Empirically, H-PSRO achieves convergence in matrix heterogeneous games that are unsolvable by non-heterogeneous baselines. Further experiments reveal that H-PSRO outperforms non-heterogeneous baselines in both heterogeneous team games and homogeneous settings.

With recent advancements in large language models (LLMs), alignment has emerged as an effective technique for keeping LLMs consensus with human intent. Current methods primarily involve direct training through Supervised Fine-tuning (SFT) or Reinforcement Learning from Human Feedback (RLHF), both of which require substantial computational resources and extensive ground truth data. This paper explores an efficient method for aligning black-box large models using smaller models, introducing a model-agnostic and lightweight Bayesian Persuasion Alignment framework. We formalize this problem as an optimization of the signaling strategy from the small model's perspective. In the persuasion process, the small model (Advisor) observes the information item (i.e., state) and persuades large models (Receiver) to elicit improved responses. The Receiver then generates a response based on the input, the signal from the Advisor, and its updated belief about the information item. Through training using our framework, we demonstrate that the Advisor can significantly enhance the performance of various Receivers across a range of tasks. We theoretically analyze our persuasion framework and provide an upper bound on the Advisor's regret, confirming its effectiveness in learning the optimal signaling strategy. Our Empirical results demonstrates that GPT-2 can significantly improve the performance of various models, achieving an average enhancement of 16.1% in mathematical reasoning ability and 13.7% in code generation. We hope our work can provide an initial step toward rethinking the alignment framework from the Bayesian Persuasion perspective.

Two-team zero-sum games are one of the most important paradigms in game theory. In this paper, we focus on finding an unexploitable equilibrium in large team games. An unexploitable equilibrium is a worst-case policy, where members in the opponent team cannot increase their team reward by taking any policy, e.g., cooperatively changing to other joint policies. As an optimal unexploitable equilibrium in two-team zero-sum games, correlated-team maxmin equilibrium remains unexploitable even in the worst case where players in the opponent team can achieve arbitrary cooperation through a joint team policy. However, finding such an equilibrium in large games is challenging due to the impracticality of evaluating the exponentially large number of joint policies. To solve this problem, we first introduce a general solution concept called restricted correlated-team maxmin equilibrium, which solves the problem of being impossible to evaluate all joint policy by a sample factor while avoiding an exploitation problem under the incomplete joint policy evaluation. We then develop an efficient sequential correlation mechanism, and based on which we propose an algorithm for approximating the unexploitable equilibrium in large games. We show that our approach achieves lower exploitability than the state-of-the-art baseline when encountering opponent teams with different exploitation ability in large team games including Google Research Football.

The burgeoning integration of artificial intelligence (AI) into human society brings forth significant implications for societal governance and safety. While considerable strides have been made in addressing AI alignment challenges, existing methodologies primarily focus on technical facets, often neglecting the intricate sociotechnical nature of AI systems, which can lead to a misalignment between the development and deployment contexts. To this end, we posit a new problem worth exploring: Incentive Compatibility Sociotechnical Alignment Problem (ICSAP). We hope this can call for more researchers to explore how to leverage the principles of Incentive Compatibility (IC) from game theory to bridge the gap between technical and societal components to maintain AI consensus with human societies in different contexts. We further discuss three classical game problems for achieving IC: mechanism design, contract theory, and Bayesian persuasion, in addressing the perspectives, potentials, and challenges of solving ICSAP, and provide preliminary implementation conceptions., Comment: 13 pages, 2 figures

High-quality rare-earth iron garnet (ReIG) Tb$_3$Fe$_5$O$_{12}$ (TbIG) thin films are epitaxially grown on a series of (111)-oriented garnet substrates with various lattice constants. The coherent growth induces a substrate-dependent in-plane tensile or compressive strain in the TbIG film. Measurements of the anomalous Hall-like effect (AHLE) in TbIG/Pt heterostructures show that the compensation temperature of TbIG films monotonically changes with the film strain. The strain results in a variation of the distances between magnetic atoms in the TbIG crystal and therefore the corresponding exchange interactions. The latter is explicitly calculated as a function of the lattice strain based on density functional theory, reproducing the observed experimental results. This work provides a versatile way to optimize ReIG-based spin-orbit torque devices., Comment: 28 pages, 5 figures

We investigate the shadows of Konoplya-Zhidenko naked singularity. In the spacetime of Konoplya-Zhidenko naked singularity, not only can unstable retrograde light ring (LR) exist, but also unstable prograde LR, leading to the formation of a complete photon sphere (PS). Due to the absence of an event horizon, a dark disc-shaped shadow does not appear; instead, a ring-shaped shadow is observed. The ring-shaped shadow appears as an infinite number of relativistic Einstein rings in the image of the naked singularity. For some parameter values, only the unstable retrograde LR exists, resulting in an incomplete unstable PS and consequently giving rise to the arc-shaped shadow for Konoplya-Zhidenko naked singularity. The shadow of Konoplya-Zhidenko naked singularity gradually shifts to the right as the rotation parameter $a$ increases, and gradually becomes smaller as the deformation parameter $|\eta|$ increases. Moreover, the stable LRs and stable photon spherical orbits can also exist in Konoplya-Zhidenko naked singularity spacetime, but they have no effect on the image of the naked singularity. This study demonstrates that rotating naked singularity can exhibit not only an arc-shaped shadow but also a ring-shaped shadow., Comment: 14 pages, 11 figures. It is to be published in Chinese Physics C

We studied the relativistic images caused by strong gravitational lensing in Kerr black hole images, which carry some essential signatures about the black hole space-time. We defined a new celestial coordinates whose origin is the center of black hole shadow to locate the relativistic images. Under the influences of the dragging effect caused by rotating black hole and the inclination angle of observer, the relative positions between the primary and secondary images are different with the different Kerr spin parameter $a$ and the observer's inclination angle $i$, so it can be used to determine the value of $a$ and $i$. We provided the specific approach to measure the value of $a$ and $i$ by the relativistic images. The time delays between the primary and secondary images are different with the different $a$ and $i$. The time delays in conjunction with the relative positions between the primary and secondary images could allow us to measure the value of $a$ and $i$ more precisely. These relativistic images are as unique as fingerprints for black hole space-time, by which one can further determine other parameters of all kinds of compact objects and verify various theories of gravity. Our results provide a new method to implement parameter estimation in the study of black hole physics and astrophysics., Comment: 16 pages, 12 figures

We give a brief review on the formation and the calculation of black hole shadow. Firstly, we introduce the conception of black hole shadow and the current works on a variety of black hole shadows. Secondly, we present main methods of calculating photon sphere radius and shadow radius, and then explain how the photon sphere affect the boundary of black hole shadow. We review the analytical calculation for black hole shadows which have analytic expressions for shadow boundary due to the integrable photon motion system. And we introduce the fundamental photon orbits which can explain the patterns of black hole shadow shape. Finally, we review the numerical calculation of black hole shadows with the backward ray-tracing method, and introduce some chaotic black hole shadows with self-similar fractal structures. Since the gravitational waves from the merger of binary black holes have been detected, we introduce a couple of shadows of binary black holes, which all have the eyebrowlike shadows around the main shadows with the fractal structures. We discuss the invariant phase space structures of photon motion system in black hole space-time, and explain the formation of black hole shadow is dominated by the invariant manifolds of certain Lyapunov orbits near the fixed points., Comment: 16 Pages, 18 figures, An invited short review and accepted for publication in Commun. Theor. Phys. Some minor error in the original version are corrected

We have studied the surface geometry and shadows of Schwarzschild black hole with a halo containing the quadrupolar and octopolar terms. We found the quadrupole term makes the Schwarzschild black hole prolate for the quadrupole strength $\mathcal{Q}<0$ and oblate for $\mathcal{Q}>0$, and the octopole term makes shadow stretch upward for the octopolar strength $\mathcal{O}<0$ and stretch downward for $\mathcal{O}>0$. The shadow of Schwarzschild black hole with halo stretches and squeezes along the horizontal direction for $\mathcal{Q}<0$ and $\mathcal{Q}>0$ respectively. Meanwhile, black hole shadow shifts upward for $\mathcal{O}<0$ and shifts downward for $\mathcal{O}>0$. We exhibited the light rays that form the shadow boundary to explain the emergence of the extraordinary patterns of black hole shadow with the quadrupole and octopole terms. From the observable width $W$, height $H$, oblateness $K$ and distortion parameter $\delta_{c}$ of black hole shadow, one can determine the quadrupole and octopolar strengths of Schwarzschild black hole with halo. Our results show that the quadrupolar and octopolar terms yield a series of interesting patterns for the shadow of a Schwarzschild black hole with halo., Comment: 20 pages,12 figures

We have studied the spacetime of a Kerr black hole immersed in Melvin magnetic field, and found not only unstable light rings could exist, but also stable light rings could exist. Both the prograde and retrograde unstable light rings radiuses increase with the magnetic field parameter $B$, but it is the opposite for stable light rings. The existence of unstable, stable light rings depend on both the rotation parameter $a$ and the magnetic field parameter $B$. For a certain $a$, there are both the prograde and retroprade unstable (stable) light rings when $B$ is less than a critical value $B_{c}$ of retrograde light ring. In this case, the shadows of Melvin-Kerr black hole have two gray regions on both sides of the middle main shadow, which correspond to the prograde and retrograde stable photon orbits. The photons in stable orbits are always moving around Melvin-Kerr black hole, they can't enter the black hole or escape to infinity. As $B$ continues to increase, there is only the prograde unstable (stable) light ring. In this case, the gray region only emerges in the life of the main shadow, which corresponds to the prograde stable photon orbits. The absence of the retrograde unstable (stable) light rings makes the Melvin-Kerr black hole shadow an half-panoramic (equatorial) shadow. When $B$ is bigger than $B_{C}$ of prograde light ring, neither prograde nor retroprade unstable (stable) light rings exist. In this case, the shadow of Melvin-Kerr black hole has no gray region for stable photon orbits, and becomes a panoramic (equatorial) shadow. In addition, there also exist some self-similar fractal structures in the shadow of Melvin-Kerr black hole arising from the chaotic motion of photon., Comment: 16 pages, 9 figures

We have studied the shadow of a disformal Kerr black hole with an extra deformation parameter, which belongs to non-stealth rotating solutions in quadratic Degenerate Higher Order Scalar Tensor (DHOST) theory. Our result show that the size of the shadow increases with the deformation parameter for the black hole with arbitrary spin parameter. However, the effect of the deformation parameter on the shadow shape depends heavily on the spin parameter of black hole and the sign of the deformation parameter. The change of the shadow shape becomes more distinct for the black hole with the more quickly rotation and the more negative deformation parameter. Especially, for the near-extreme black hole with negative deformation parameter, there exist a "pedicel"-like structure appeared in the shadow, which increases with the absolute value of deformation parameter. The eyebrow-like shadow and the self-similar fractal structures also appear in the shadow for the disformal Kerr black hole in DHOST theory. These features in the black hole shadow originating from the scalar field could help us to understand the non-stealth disformal Kerr black hole and quadratic DHOST theory., Comment: 13 pages, 5 figures

We present firstly the equation of motion for the photon coupled to a special bumblebee vector field in a Kerr black hole spacetime and find that the propagation of light depends on its polarization due to the birefringence phenomenon. The dependence of black hole shadow on the light's polarization is dominated by the rotation of black hole. In the non-rotating case, we find that the black hole shadow is independent of the polarization of light. However, the status is changed in the rotating case, in which the black hole shadow depends on the light's polarization and the coupling between bumblebee vector field and electromagnetic field. These features of black hole shadow casted by polarized lights could help us to understand the bumblebee vector field with Lorentz symmetry breaking and its interaction with electromagnetic field., Comment: 16 pages, 8 figures. Some references are added and accepted for pubication in JHEP

We have studied the shadows of a Schwarzschild black hole under a special polar gravitational perturbation, which is a particular solution of Einstein equations expanded up to first order. It is shown that the black hole shadow changes periodically with time and the change of shadow depends on the Legendre polynomial order parameter l and the frequency {\sigma} of gravitational wave. For the odd order of Legendre polynomial, the center of shadow oscillates along the direction which is vertical to equatorial plane. For even l, the center of shadow does not move, but the shadow alternately stretches and squeezes with time along the vertical direction. Moreover, the presence of the gravitational wave leads to the self-similar fractal structures appearing in the boundary of the black hole shadow. We also find that this special gravitational wave has a greater influence on the vertical direction of black hole shadow., Comment: 20 pages, 12 figures

We have studied the shadows of a Schwarzschild black hole surrounded by a Bach-Weyl ring through the backward ray-tracing method. The presence of Bach-Weyl ring leads to that the photon dynamical system is non-integrable and then chaos would appear in the photon motion, which affects sharply the black hole shadow. The size and shape the black hole shadow depend on the black hole parameter, the Bach-Weyl ring mass and the Weyl radius between black hole and ring. Some self-similar fractal structures also appear in the black hole shadow, which originates from the chaotic lensing. We also study the change of the image of Bach-Weyl ring with the ring mass and the Weyl radius. Finally, we analyze the invariant manifolds of Lyapunov orbits near the fixed points and discuss further the formation of the shadow of a Schwarzschild black hole with Bach-Weyl ring., Comment: 16 pages,8 figures, the version published in EPJC