Interplay of the Jahn-Teller effect and spin-orbit coupling: The case of trigonal vibrations

We study an interplay between the orbital degeneracy and the spin-orbit coupling (SOC) giving rise to spin-orbital entangled states in concentrated systems (cooperative Jahn-Teller [JT] effect). As a specific example, we analyze the interaction of electrons occupying triply degenerate single-ion t2g levels with trigonal vibrations (the t - T problem). A more general problem of the electron-lattice interaction involving both tetragonal and trigonal vibrations is also considered. It is shown that the result of such interaction crucially depends on the occupation of t2g levels leading to either the suppression or the enhancement of the JT effect by the SOC. © 2022 American Physical Society.