Back to Search Start Over

Intriguing Properties of Compression on Multilingual Models

Authors :
Ogueji, Kelechi
Ahia, Orevaoghene
Onilude, Gbemileke
Gehrmann, Sebastian
Hooker, Sara
Kreutzer, Julia
Publication Year :
2022

Abstract

Multilingual models are often particularly dependent on scaling to generalize to a growing number of languages. Compression techniques are widely relied upon to reconcile the growth in model size with real world resource constraints, but compression can have a disparate effect on model performance for low-resource languages. It is thus crucial to understand the trade-offs between scale, multilingualism, and compression. In this work, we propose an experimental framework to characterize the impact of sparsifying multilingual pre-trained language models during fine-tuning. Applying this framework to mBERT named entity recognition models across 40 languages, we find that compression confers several intriguing and previously unknown generalization properties. In contrast to prior findings, we find that compression may improve model robustness over dense models. We additionally observe that under certain sparsification regimes compression may aid, rather than disproportionately impact the performance of low-resource languages.<br />Comment: Accepted to EMNLP 2022

Details

Database :
arXiv
Publication Type :
Report
Accession number :
edsarx.2211.02738
Document Type :
Working Paper
Full Text :
https://doi.org/10.48550/arXiv.2211.02738