Back to Search
Start Over
DP-MemArc: Differential Privacy Transfer Learning for Memory Efficient Language Models
- Publication Year :
- 2024
-
Abstract
- Large language models have repeatedly shown outstanding performance across diverse applications. However, deploying these models can inadvertently risk user privacy. The significant memory demands during training pose a major challenge in terms of resource consumption. This substantial size places a heavy load on memory resources, raising considerable practical concerns. In this paper, we introduce DP-MemArc, a novel training framework aimed at reducing the memory costs of large language models while emphasizing the protection of user data privacy. DP-MemArc incorporates side network or reversible network designs to support a variety of differential privacy memory-efficient fine-tuning schemes. Our approach not only achieves in memory optimization but also ensures robust privacy protection, keeping user data secure and confidential. Extensive experiments have demonstrated that DP-MemArc effectively provides differential privacy-efficient fine-tuning across different task scenarios.<br />Comment: 9 pages second version
Details
- Database :
- arXiv
- Publication Type :
- Report
- Accession number :
- edsarx.2406.11087
- Document Type :
- Working Paper