Search

LLMs have transformed NLP and shown promise in various fields, yet their potential in finance is underexplored due to a lack of thorough evaluations and the complexity of financial tasks. This along with the rapid development of LLMs, highlights the urgent need for a systematic financial evaluation benchmark for LLMs. In this paper, we introduce FinBen, the first comprehensive open-sourced evaluation benchmark, specifically designed to thoroughly assess the capabilities of LLMs in the financial domain. FinBen encompasses 35 datasets across 23 financial tasks, organized into three spectrums of difficulty inspired by the Cattell-Horn-Carroll theory, to evaluate LLMs' cognitive abilities in inductive reasoning, associative memory, quantitative reasoning, crystallized intelligence, and more. Our evaluation of 15 representative LLMs, including GPT-4, ChatGPT, and the latest Gemini, reveals insights into their strengths and limitations within the financial domain. The findings indicate that GPT-4 leads in quantification, extraction, numerical reasoning, and stock trading, while Gemini shines in generation and forecasting; however, both struggle with complex extraction and forecasting, showing a clear need for targeted enhancements. Instruction tuning boosts simple task performance but falls short in improving complex reasoning and forecasting abilities. FinBen seeks to continuously evaluate LLMs in finance, fostering AI development with regular updates of tasks and models., Comment: 19 pages, 10 figures

Background: Traditional Chinese medicine has used Peucedanum praeruptorum Dunn (Apiaceae) for a long time. Various coumarins, including the significant constituents praeruptorin (A-E), are the active constituents in the dried roots of P. praeruptorum. Previous transcriptomic and metabolomic studies have attempted to elucidate the distribution and biosynthetic network of these medicinal-valuable compounds. However, the lack of a high-quality reference genome impedes an in-depth understanding of genetic traits and thus the development of better breeding strategies. Results: A telomere-to-telomere (T2T) genome was assembled for P. praeruptorum by combining PacBio HiFi, ONT ultra-long, and Hi-C data. The final genome assembly was approximately 1.798 Gb, assigned to 11 chromosomes with genome completeness >98%. Comparative genomic analysis suggested that P. praeruptorum experienced 2 whole-genome duplication events. By the transcriptomic and metabolomic analysis of the coumarin metabolic pathway, we presented coumarins' spatial and temporal distribution and the expression patterns of critical genes for its biosynthesis. Notably, the COSY and cytochrome P450 genes showed tandem duplications on several chromosomes, which may be responsible for the high accumulation of coumarins. Conclusions: A T2T genome for P. praeruptorum was obtained, providing molecular insights into the chromosomal distribution of the coumarin biosynthetic genes. This high-quality genome is an essential resource for designing engineering strategies for improving the production of these valuable compounds.