Back to Search Start Over

LP Data Pipeline: Lightweight, Purpose-driven Data Pipeline for Large Language Models

Authors :
Kim, Yungi
Ha, Hyunsoo
Yang, Seonghoon
Lee, Sukyung
Kim, Jihoo
Park, Chanjun
Publication Year :
2024

Abstract

Creating high-quality, large-scale datasets for large language models (LLMs) often relies on resource-intensive, GPU-accelerated models for quality filtering, making the process time-consuming and costly. This dependence on GPUs limits accessibility for organizations lacking significant computational infrastructure. To address this issue, we introduce the Lightweight, Purpose-driven (LP) Data Pipeline, a framework that operates entirely on CPUs to streamline the processes of dataset extraction, filtering, and curation. Based on our four core principles, the LP Data Pipeline significantly reduces preparation time and cost while maintaining high data quality. Importantly, our pipeline enables the creation of purpose-driven datasets tailored to specific domains and languages, enhancing the applicability of LLMs in specialized contexts. We anticipate that our pipeline will lower the barriers to LLM development, enabling a wide range of organizations to access LLMs more easily.

Details

Database :
arXiv
Publication Type :
Report
Accession number :
edsarx.2411.11289
Document Type :
Working Paper