---
title: "DistilBERT, a distilled version of BERT: smaller, faster, cheaper"
description: "The authors from Hugging Face propose DistilBERT: a compact general-purpose model obtained… 22 references, cited by 2,404 papers here."
authors: ["Victor Sanh", "Lysandre Debut", "Julien Chaumond", "Thomas Wolf"]
year: 2019
venue: "arXiv:1910.01108"
source: "https://arxiv.org/abs/1910.01108"
doi: "10.48550/arxiv.1910.01108"
arxiv: "1910.01108"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sanh-2019-distilbert-distilled-version-bert-smaller/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sanh-2019-distilbert-distilled-version-bert-smaller/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/) / DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

# DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

**Authors:** [Victor Sanh](https://md.research.elamur.ai/free/en/authors/victor-sanh/), [Lysandre Debut](https://md.research.elamur.ai/free/en/authors/lysandre-debut/), [Julien Chaumond](https://md.research.elamur.ai/free/en/authors/julien-chaumond/) and [Thomas Wolf](https://md.research.elamur.ai/free/en/authors/thomas-wolf/) · **Year:** 2019 · **Venue:** arXiv:1910.01108 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1910.01108) · [PDF](https://arxiv.org/pdf/1910.01108) · [DOI 10.48550/arxiv.1910.01108](https://doi.org/10.48550/arxiv.1910.01108) · [arXiv 1910.01108](https://arxiv.org/abs/1910.01108) · [OpenAlex](https://openalex.org/W2978017171)

**Citations:** 4,347 citations (OpenAlex, 2026-10-14) · [2404](https://md.research.elamur.ai/free/en/papers/sanh-2019-distilbert-distilled-version-bert-smaller/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary

Large pre-trained language models such as BERT deliver high quality. However, they require a lot of memory and computation, which makes them hard to run on smartphones or on a limited budget. The authors from Hugging Face propose DistilBERT: a compact general-purpose model obtained through knowledge distillation already during pre-training rather than for a specific task. The student's architecture is the same as BERT-base, but with half as many layers, and the token-type embeddings and the pooler are removed. The initial weights are taken from every second layer of the teacher. Training combines three loss functions: a distillation loss on the teacher's softened distributions, masked language modeling, and a cosine loss that aligns the directions of the hidden states. The corpus is the same as BERT's: English Wikipedia and the Toronto Book Corpus. As a result, DistilBERT is 40% smaller and 60% faster than BERT and retains about 97% of its performance on GLUE. The weights and code are released openly in the Transformers library. The model has therefore become a practical foundation for cheaper inference and for NLP on mobile devices.

## Key points

- **Size and speed:** 66M parameters versus 110M in BERT-base and 180M in ELMo. A full pass over the STS-B dev set on CPU with batch size 1 takes 410 s versus 668 s for BERT.
- **Quality:** GLUE macro-score of 77.0 versus 79.5 for BERT-base and 68.7 for the ELMo baseline. On IMDb, accuracy is 92.82 versus 93.46. On SQuAD 1.1, EM/F1 is 77.7/85.8 versus 81.2/88.5, and with a second distillation step during fine-tuning it is 79.1/86.9 (according to Table 2; the main text gives different numbers for this variant).
- **Training:** about 90 hours on 8 V100 GPUs with 16 GB each. Batches of up to 4K examples via gradient accumulation, dynamic masking, no next sentence prediction objective (following the RoBERTa recipe).
- **Ablation (change in GLUE macro-score):** without the distillation loss L_ce −2.96, without the cosine loss L_cos −1.46, without L_mlm only −0.31. Random initialization instead of the teacher's weights gives −3.69.
- **On-device:** in a question answering app on the iPhone 7 Plus, DistilBERT runs 71% faster than BERT (excluding tokenization), and the model weighs 207 MB. The code is published in the swift-coreml-transformers repository.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Robert Bosch](https://md.research.elamur.ai/free/en/companies/robert-bosch-germany/) — Company, Stuttgart, Germany · 12 works · 9,057 citations
- [Freie Universität Berlin](https://md.research.elamur.ai/free/en/universities/freie-universitat-berlin/) — University, Berlin, Germany · 13 works · 7,043 citations
- [Fraunhofer Institute for Open Communication Systems](https://md.research.elamur.ai/free/en/labs/fraunhofer-institute-for-open-communication-systems/) — Research institute, Berlin, Germany · 2 works · 6,805 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (1): [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 297 works

## Cite

Cite as: Victor Sanh, Lysandre Debut, Julien Chaumond and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv:1910.01108.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{sanh2019distilbert,
  title = {{DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter}},
  author = {Victor Sanh, Lysandre Debut, Julien Chaumond and Thomas Wolf},
  year = {2019},
  doi = {10.48550/arxiv.1910.01108},
  eprint = {1910.01108},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1910.01108}
}
```

- [← previous BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://md.research.elamur.ai/free/en/papers/lee-2019-biobert-pre-trained-biomedical-language/)
- [next → A Survey on Bias and Fairness in Machine Learning](https://md.research.elamur.ai/free/en/papers/mehrabi-2019-survey-bias-fairness-machine-learning/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/)

## References

22 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In NAACL-HLT, 2018.
2. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+Language+models+are+unsupervised+multitask+learners%2E+2019%2E)
3. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar S. Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke S. Zettlemoyer, and Veselin Stoyanov. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). ArXiv, abs/1907.11692, 2019.
4. Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. [Green ai](https://md.research.elamur.ai/free/en/papers/schwartz-2019-green-ai/). ArXiv, abs/1907.10597, 2019.
5. Emma Strubell, Ananya Ganesh, and Andrew McCallum. [Energy and policy considerations for deep learning in nlp](https://md.research.elamur.ai/free/en/papers/strubell-2019-energy-policy-considerations-deep-learning/). In ACL, 2019.
6. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In NIPS, 2017.
7. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. [Transformers: State-of-the-art natural language processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/), 2019.
8. Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In KDD, 2006. [find](https://scholar.google.com/scholar?q=Cristian+Bucila%2C+Rich+Caruana%2C+and+Alexandru+Niculescu%2DMizil%2E+Model+compression%2E+In+KDD%2C+2006%2E)
9. Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. [Distilling the knowledge in a neural network](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/). ArXiv, abs/1503.02531, 2015.
10. Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. [Aligning books and movies: Towards story-like visual explanations by watching movies and reading books](https://md.research.elamur.ai/free/en/papers/zhu-2015-aligning-books-movies-towards-story/). 2015 IEEE International Conference on Computer Vision (ICCV), pages 19–27, 2015.
11. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. [Glue: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In ICLR, 2018.
12. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. [Deep contextualized word representations](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/). In NAACL, 2018.
13. Alex Wang, Ian F. Tenney, Yada Pruksachatkun, Katherin Yu, Jan Hula, Patrick Xia, Raghu Pappagari, Shuning Jin, R. Thomas McCoy, Roma Patel, Yinghui Huang, Jason Phang, Edouard Grave, Najoung Kim, Phu Mon Htut, Thibault F’evry, Berlin Chen, Nikita Nangia, Haokun Liu, Anhad Mohananey, Shikha Bordia,… [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Ian+F%2E+Tenney%2C+Yada+Pruksachatkun%2C+Katherin+Yu%2C+Jan+Hula%2C+Patrick+Xia%2C+Raghu+Pappagari%2C+Shuning+Jin%2C+R%2E+Thomas+McCoy%2C+Roma+Patel%2C+Yinghui+Huang%2C+Jason+Phang%2C+Edouard+Grave%2C+Najoung+Kim%2C+Phu)
14. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. [Learning word vectors for sentiment analysis](https://md.research.elamur.ai/free/en/papers/maas-2011-learning-word-vectors-sentiment-analysis/). In ACL, 2011.
15. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100, 000+ questions for machine comprehension of text. In EMNLP, 2016. — [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
16. Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. Distilling task-specific knowledge from bert into simple neural networks. ArXiv, abs/1903.12136, 2019. [find](https://scholar.google.com/scholar?q=Raphael+Tang%2C+Yao+Lu%2C+Linqing+Liu%2C+Lili+Mou%2C+Olga+Vechtomova%2C+and+Jimmy+Lin%2E+Distilling+task%2Dspecific+knowledge+from+bert+into+simple+neural+networks%2E+ArXiv%2C+abs%2F1903%2E12136%2C+2019%2E)
17. Debajyoti Chatterjee. Making neural machine reading comprehension faster. ArXiv, abs/1904.00796, 2019. — [arXiv:1904.00796](https://arxiv.org/abs/1904.00796) [find](https://scholar.google.com/scholar?q=Debajyoti+Chatterjee%2E+Making+neural+machine+reading+comprehension+faster%2E+ArXiv%2C+abs%2F1904%2E00796%2C+2019%2E+%E2%80%94+%5BarXiv%3A1904%2E00796%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1904%2E00796%29)
18. Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better: The impact of student initialization on knowledge distillation. ArXiv, abs/1908.[08962, 2019](https://md.research.elamur.ai/free/en/papers/turc-2019-well-read-students-learn-better/).
19. Ze Yang, Linjun Shou, Ming Gong, Wutao Lin, and Daxin Jiang. Model compression with multi-task knowledge distillation for web-scale question answering system. ArXiv, abs/1904.09636, 2019. — [arXiv:1904.09636](https://arxiv.org/abs/1904.09636) [find](https://scholar.google.com/scholar?q=Ze+Yang%2C+Linjun+Shou%2C+Ming+Gong%2C+Wutao+Lin%2C+and+Daxin+Jiang%2E+Model+compression+with+multi%2Dtask+knowledge+distillation+for+web%2Dscale+question+answering+system%2E+ArXiv%2C+abs%2F1904%2E09636%2C+2019%2E+%E2%80%94+%5BarXiv%3A190)
20. Henry Tsai, Jason Riesa, Melvin Johnson, Naveen Arivazhagan, Xin Li, and Amelia Archer. Small and practical bert models for sequence labeling. In EMNLP-IJCNLP, 2019. [find](https://scholar.google.com/scholar?q=Henry+Tsai%2C+Jason+Riesa%2C+Melvin+Johnson%2C+Naveen+Arivazhagan%2C+Xin+Li%2C+and+Amelia+Archer%2E+Small+and+practical+bert+models+for+sequence+labeling%2E+In+EMNLP%2DIJCNLP%2C+2019%2E)
21. Paul Michel, Omer Levy, and Graham Neubig. [Are sixteen heads really better than one?](https://md.research.elamur.ai/free/en/papers/michel-2019-sixteen-heads-really-better-than/) In NeurIPS, 2019.
22. Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In ICML, 2015. [find](https://scholar.google.com/scholar?q=Suyog+Gupta%2C+Ankur+Agrawal%2C+Kailash+Gopalakrishnan%2C+and+Pritish+Narayanan%2E+Deep+learning+with+limited+numerical+precision%2E+In+ICML%2C+2015%2E)

*Updated 2 November 2026*