Search

We introduce Falcon2-11B, a foundation model trained on over five trillion tokens, and its multimodal counterpart, Falcon2-11B-vlm, which is a vision-to-text model. We report our findings during the training of the Falcon2-11B which follows a multi-stage approach where the early stages are distinguished by their context length and a final stage where we use a curated, high-quality dataset. Additionally, we report the effect of doubling the batch size mid-training and how training loss spikes are affected by the learning rate. The downstream performance of the foundation model is evaluated on established benchmarks, including multilingual and code datasets. The foundation model shows strong generalization across all the tasks which makes it suitable for downstream finetuning use cases. For the vision language model, we report the performance on several benchmarks and show that our model achieves a higher average score compared to open-source models of similar size. The model weights and code of both Falcon2-11B and Falcon2-11B-vlm are made available under a permissive license.

Large language models are commonly trained on a mixture of filtered web data and curated high-quality corpora, such as social media conversations, books, or technical papers. This curation process is believed to be necessary to produce performant models with broad zero-shot generalization abilities. However, as larger models requiring pretraining on trillions of tokens are considered, it is unclear how scalable is curation and whether we will run out of unique high-quality data soon. At variance with previous beliefs, we show that properly filtered and deduplicated web data alone can lead to powerful models; even significantly outperforming models from the state-of-the-art trained on The Pile. Despite extensive filtering, the high-quality data we extract from the web is still plentiful, and we are able to obtain five trillion tokens from CommonCrawl. We publicly release an extract of 600 billion tokens from our RefinedWeb dataset, and 1.3/7.5B parameters language models trained on it.

In a previous paper, we reviewed theoretically some of the available processing schemes for X-ray wavefront sensing based on random modulation. We here show experimental applications of the technique for characterising both refractive and reflective optical components. These fast and accurate X-ray at-wavelength metrology methods can assist the manufacture of X-ray optics that transport X-ray beams with minimum amount of wavefront distortion. We also recall how such methods can facilitate online optimization of active optics., Comment: 13 pages, 13 figures. arXiv admin note: substantial text overlap with arXiv:1902.09418

X-ray near-field speckle-based phase-sensing approaches provide efficient means to characterise optical elements. Here, we present a theoretical review of several of these speckle methods in the frame of optical characterisation and provide a generalization of the concept. As we also demonstrate experimentally in another paper, the methods theoretically developed here can be applied with different beams and optics and within a variety of situations where at-wavelength metrology is desired. By understanding the differences between the various processing methods, it is possible to find and implement the best suited approach for each metrology scenario., Comment: 10 pages, 3 figures, 1 table

It has been known for a long time that stellar metallicity plays a significant role in the determination of the ages of the different Galactic stellar populations, when main sequence evolutionary tracks are employed. Here we analyze the role that metallicity plays on the white dwarf luminosity function of the Galactic disk, which is often used to determine its age. We employ a Monte Carlo population synthesis code that accounts for the properties of the population of Galactic disk white dwarfs. Our code incorporates the most up-to-date evolutionary cooling sequences for white dwarfs with hydrogen-rich and hydrogen-deficient atmospheres for both carbon-oxygen and oxygen-neon cores. We use two different models to assess the evolution of the metallicity, one in which the adopted metallicity is constant with time, but with a moderate dispersion, and a second one in which the metallicity increases with time. We found that our theoretical results are in a very satisfactory agreement with the observational luminosity functions obtained from the Sloan Digital Sky Survey (SDSS) and from the SuperCOSMOS Sky Survey (SSS), independently of the adopted age-metallicity law. In particular, we found that the age-metallicity law has no noticeable impact in shaping the bright branch of the white dwarf luminosity function, and that the position of its cut-off is almost insensitive to the adopoted age-metallicity relationship. Because the shape of the bright branch of the white dwarf luminosity function is insensitive to the age-metallicity law, it can be safely employed to test the theoretical evolutionary sequences, while due to the limited sensitivity of the position of the drop-off to the distribution of metallicities, its location provides a robust indicator of the age of the Galactic disk., Comment: 7 pages, 5 figures, accepted for publication in A&A