We study two classes of object to gain a better understanding of the evolution of Active Galactic Nuclei (AGN): Infrared-Faint Radio Sources (IFRSs) and Gigahertz Peaked Spectrum (GPS) / Compact Steep Spectrum (CSS) sources. IFRSs are a recently discovered rare class of object, which were found to be strong in the radio but undetectable in extremely sensitive infrared observations from the Spitzer Space Telescope, even in stacked images with σ < 1μJy. IFRSs were found to exhibit a relatively high sky density, and were thought to represent AGN at z > 3. Therefore, IFRSs may significantly increase the number of known high-redshift galaxies. However, their non-detections in the optical and infrared prevented confirmation of their nature. Previous studies of IFRSs focused on very sensitive observations of a few small regions of the sky, and the largest sample consisted of 55 IFRSs. However, we follow the strategy of combining radio data with IR and optical data for a large region of the sky. Using these data, we discover a population of >1300 brighter IFRSs which are, for the first time, reliably detected in the infrared and optical. We present the first spectroscopic redshifts of IFRSs and show that the brightest IFRSs are at z > 2. Furthermore, we rule out that IFRSs are Star Forming Galaxies, hotspots, lobes or misidentifications. We find the first X-ray counterparts of IFRSs, and increase the number of known polarised IFRSs five-fold. We present an analysis of their radio spectra and show that IFRSs consist of GPS, CSS and ultra-steep-spectrum sources. We follow up >50 of these using VLBI observations, and confirm the AGN status of IFRSs. We conclude that IFRSs represent a new population of high-redshift radio galaxies, which, for the faintest IFRSs, may have redshifts as high as z = 7 and consist of a few hundred thousand objects across the μJy sky. GPS and CSS sources are compact radio sources with a convex radio spectrum. They are widely thought to represent young and evolving radio galaxies that have recently launched their jets. However, good evidence exists in individual cases that GPS and CSS sources are one of the following: 1) frustrated by interactions with dense gas and dust in their environment; 2) prematurely dying radio sources; 3) recurrent radio galaxies. Their convex spectrum is generally thought to be caused by Synchrotron Self Absorption (SSA), an internal process in which the same population of electrons is responsible for the synchrotron emission and self-absorption. However, recent studies have shown that the convex spectrum may be caused by Free-Free Absorption (FFA), an external process in which an inhomogeneous screen absorbs the synchrotron emission. The majority of GPS and CSS samples consist of Jy-level and therefore, high-luminosity sources. VLBI images show that GPS and CSS sources typically have double-lobed, edge-brightened morphologies on mas scales, appearing as scaled down versions of Fanaroff-Riley Class II (FR II) galaxies. Recently, two low-luminosity GPS sources were found to have jet-brightened morphologies, which appeared as scaled down versions of Fanaroff-Riley Class I (FR I) galaxies. From this, it was proposed that there exists a morphology-luminosity break analogous to the FR I/II break and that low-luminosity GPS and CSS sources are the compact counterparts of FR I galaxies. However, this hypothesis remains unconfirmed, since very few samples of low-luminosity GPS and CSS sources exist. We have observed the faintest population of GPS and CSS sources to date, consisting ⇠150 sources, many of which are low-luminosity. We use high-resolution radio observations to determine their linear size, resolve their jets and observe their small-scale morphology. We combine these data with a large number of radio observations at other frequencies to model their radio spectra using SSA and FFA models. In particular, we use very low frequency observations that have only recently become available to constrain their spectral peaks. We follow up eight of the most compact sources with VLBI and detect six of them. We find that our GPS and CSS sources are well modelled by an inhomogeneous FFA model (hereafter ‘FFA’). Furthermore, we find a number of very compact GPS and CSS sources that are inconsistent with SSA theory. We show that a single inhomogeneous SSA model fits poorly to the majority of radio spectra, predicting far too steep a slope below the peak. We resolve all of the sources with VLBI and derive their kinematic ages based on the jet sizes. Even when assuming more complex SSA models, we derive magnetic field strengths several orders of magnitude too high for one source. A few sources are well modelled by an FFA model in which the inhomogeneous absorption is dominated by clouds of high density, consistent with the frustration hypothesis. However, the majority of sources are well modelled by FFA models with low-density clouds. These models suggest that an inhomogeneous and clumpy medium surrounds the sources, implying they may undergo recurrent activity. Furthermore, the spectral model of one CSS source suggests it is a prematurely dying radio galaxy whose jets have been switched off for ∽600 years. However, we find no evidence of restarted radio galaxies within our high resolution observations. The kinematic and spectral ages we derive are consistent with the hypothesis that GPS and CSS sources are young and evolving. We find tentative evidence that at mJy-levels, the fraction of CSS sources is smaller than the fraction estimated for Jy-level sources. This may be accounted for as a selection effect or if a smaller fraction of mJy-level GPS sources evolve into CSS sources as compared to the Jy-level GPS sources. We find a few GPS sources with low luminosities, which we will follow up with VLBI to test whether they are the compact counterparts of FR I galaxies. We conclude that, despite being historically favoured, single inhomogeneous SSA is not the dominant form of absorption amongst a large fraction of GPS and CSS sources. We find that FFA provides a good model for the majority of the spectra with observable turnovers, suggesting an inhomogeneous and clumpy ambient medium. Furthermore, we conclude that the majority of our GPS and CSS sources are young and evolving and may undergo recurrent activity over small time scales. We conclude that a very small fraction of GPS and CSS sources consists of frustrated, dying or restarted radio galaxies.