Evolution of magnetic phase in two dimensional van der Waals Mn$_{1-x}$Ni$_x$PS$_3$ single crystals

Metal thio(seleno)phosphates MPX$_3$ have attracted considerable attentions with wide spanned band gaps and rich magnetic properties. In this series, two neighboring members MnPS$_3$ and NiPS$_3$ differ in magnetic atoms, magnetic easy axes, spin anisotropy, as well as nearest-neighbor magnetic interactions. The competition between these components may cause intriguing physical phenomena. In this article, the evolution of magnetism of Mn$_{1-x}$Ni$_x$PS$_3$ series is reported. Despite the incompatible antiferromagnetic orders of two end members, the antiferromagnetism persists as the ground state in the whole substitution region. The magnetic ordering temperature $T_{\rm N}$ show nonmonotonic V-shape behavior, and the reentrant spin glass phase at x=0.5 is observed. In addition, abnormal bifurcation of $T_{\rm N}$ occurs at x=0.75, which may be due to the temperature-dependent spin reorientation or phase separation. The evolution of magnetism is further confirmed semi-quantitatively by our density functional theory calculations. Our study indicates that exotic magnetism can be intrigued when multi-degrees of freedom are involved in these low-dimensional systems, which call for more in-depth microscopic studies in future.