Exotic phases in frustrated quantum systems

In this work, we study the extended Shastry-Sutherland model which is a Quantum spin system with geometrical frustration. It is originated from the compound SrCu2(BO3)2 which exhibits nontrivial magnetization plateaus in the external magnetic field. In the first part, we use a generalized spin wave theory (spin wave theory in plaquette representation) to investigate the intermediate phase (extremely frustrated range) of the Shastry-Sutherland model and the associated quasiparticle dispersions. We confirm the existence of this plaquette singlet state by explictly calculating its energy with quantum corrections from second order perturbation theory. We also propose a more general plaquette valence-bond-solid (PVBS) phase when the anisotropy is turned on. The quasi- particle dispersion changes qualitatively when it passes a critical line within the PVBS phase. The gap splits from k = (0,0) to four degenerate points which may imply a crossover to the resonanting valence bond state. In the second part, we study the magnetization plateaus and supersolid phases in the extended Shastry-Sutherland model which is expected to be the effective model of the rare earth tetraborides family. Analytical (spiral plaquette representation) and numerical (Stochastic Series Expansion QMC) methods are applied to the Ising and Ising-like XXZ models, respectively. Results from the two methods are qualitatively consistent. Expected plateaus 1/2 and 1/3 have been observed in the phase diagram and a few new plateaus including 5/9 and 2/9 have been observed in a narrow regime of the phase diagram. Besides, we also study the Spin-1 Heisenberg model on a bipartite lattice to test the consistency of the QMC and spin wave method. The excellent quantitative and qualitative agreement guarantees the efficiency of the lowest order spin wave theory. DOCTOR OF PHILOSOPHY (SPMS)