Rare-earth compounds have the special property that they usually form ionic bonds with large localized magnetic moments. The interaction of these moments with each other, as well as nuclear spins and other magnetic moments in the host material, leads to a multitude of interesting physical phenomena, which have been studied over the last century -- this includes not only the field of magnetism, but also e.g. lasers, superconductivity and quantum information processing, giving rise to fast technological advances. In this thesis, we study quantum coherence of dilute rare-earth ions. The main objective is to gain a good theoretical understanding of coherence-limiting mechanisms in solid-state materials, which ultimately allows us to construct protected quantum objects. These can be used e.g. as highly coherent qubits to store and process quantum information, or as quantum sensors to probe their environment. While the main results of this thesis are of theoretical nature, most of them were obtained in the context of understanding and analyzing real-world experiments in the host material LiYF4. In the first part of this thesis, we prepare the ground for the study of coherence-limiting effects. The main results concern the quantitative analysis of line-broadening mechanisms in lightly doped Ho(3+):LiYF4 as probed by Fourier-transform infrared spectroscopy. More importantly, we also present a quantitative theoretical model of thermally activated quantum tunneling in this material -- as measured by Faraday rotation during a hysteresis loop -- which allows us to indirectly measure the lifetime of hyperfine states. In the second and central part of the thesis, we turn to the quantum coherence of different kinds of rare-earth excitations. We first study spin-echo experiments performed in Tb(3+):LiYF4, which measure the coherence of crystal-field excitations. A new kind of quantum mechanical fluctuation and dephasing mechanism based on the virtual, cyclic exchange of neutral excitations is identified, which we call ring exchange. The theoretical understanding of the coherence of crystal-field excitations showed that pairs of ions host particularly coherent excitations. Their dephasing is sensitive to the dynamics of fast flipping spins in their environment and thus may serve as quantum sensors. Another central insight of this thesis is that -- given a targeted coherence time -- the maximal density of coherent degrees of freedom is usually obtained in more densely doped systems, where coherent pairs are surrounded by many less coherent single ions, rather than in more dilute samples, where the entirety of ions achieves the targeted coherence time. Then, we turn to another magnetic coherence phenomenon, revisiting mysteriously long-lived coherent low-frequency excitations in Ho(3+):LiYF4. We eliminate electronic spins as candidates to explain the experiments and present several arguments pointing to the nuclear origin of the phenomenon. Motivated by this, we analyze the coherence of low-energy excitations of Ho(3+) nuclear spins, finding interesting electro-nuclear degrees of freedom that might serve as future quantum memories. In a final chapter, we combine the various ingredients of rare-earth physics to propose a platform for universal quantum computing with single rare-earth ions. This scheme has several advantages over similar ones based on phosphorous doped silicon, in particular faster gate times and reduced charge noise. We also compare our proposal to previous rare-earth schemes with two-qubit gates based on the dipole blockade, showing that our gate time is improved by more than an order of magnitude. While the ideas developed in this thesis are inspired by experiments in specific materials, the derived results apply to rather generic Hamiltonians covering a multitude of rare-earth materials and other quantum magnets. This prepares the floor for future experimental research, in particular in the implementation of quantum computing with rare-earth ions, as well as the study of localization of dipolar interacting moments in quantum magnets.