---
title: "Efficient Intent Detection with Dual — Casanueva 2020"
description: "Task-oriented dialogue systems must classify each user utterance into an intent, yet commercial projects often start with only a handful of labelled…"
authors: ["Iñigo Casanueva", "Tadas Temčinas", "Daniela Gerz", "Matthew Henderson", "Ivan Vulić"]
year: 2020
venue: "Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI"
source: "https://aclanthology.org/2020.nlp4convai-1.5/"
doi: "10.18653/v1/2020.nlp4convai-1.5"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/casanueva-2020-efficient-intent-detection-dual-sentence/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/casanueva-2020-efficient-intent-detection-dual-sentence/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/2/) / Efficient Intent Detection with Dual Sentence Encoders

# Efficient Intent Detection with Dual Sentence Encoders

**Authors:** [Iñigo Casanueva](https://md.research.elamur.ai/free/en/authors/inigo-casanueva/), Tadas Temčinas, [Daniela Gerz](https://md.research.elamur.ai/free/en/authors/daniela-gerz/), [Matthew Henderson](https://md.research.elamur.ai/free/en/authors/matthew-henderson/), [Ivan Vulić](https://md.research.elamur.ai/free/en/authors/ivan-vulic/) · **Year:** 2020 · **Venue:** Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.nlp4convai-1.5/) · [PDF](https://aclanthology.org/2020.nlp4convai-1.5.pdf) · [DOI 10.18653/v1/2020.nlp4convai-1.5](https://doi.org/10.18653/v1/2020.nlp4convai-1.5) · [OpenAlex](https://openalex.org/W3012396850)

**Citations:** 336 citations (openalex-snapshot · 2026-10-14) · [286](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Task-oriented dialogue systems must classify each user utterance into an intent, yet commercial projects often start with only a handful of labelled examples per class. The authors argue that fully fine-tuning LM-pretrained encoders such as BERT is expensive and prone to overfitting in such conditions. Instead, they keep two dual sentence encoders frozen. Both were pretrained on conversational response selection: USE and ConveRT. Only a small MLP classifier is trained on their sentence vectors, used separately or concatenated. They evaluate on HWU 64, CLINC 150 and a newly released banking dataset, BANKING 77, against BERT used as a frozen feature extractor and against fully fine-tuned BERT-Large. The dual-encoder classifiers perform best overall, with the largest margins at 10 or 30 examples per intent. They train in minutes on a laptop CPU and barely react to hyperparameter changes, so scarce labels can be used for training instead of validation. The result makes intent detection cheap to build, and BANKING 77 adds a fine-grained single-domain benchmark.

## Key points
- **BANKING 77**: 13,083 customer-service queries labelled with 77 fine-grained intents in a single banking domain. Utterances average about 12 words, compared with 7 in HWU 64 (25,716 examples, 64 intents, 21 domains) and 8 in CLINC 150 (23,700 examples, 150 intents, 10 domains).
- **Classifier setup**: a single 512-unit ReLU hidden layer with a softmax output, placed on fixed USE, ConveRT or USE+ConveRT encodings. It is trained with SGD (learning rate 0.7, linear decay), heavy dropout of 0.75 and 500 iterations.
- **Few-shot evaluation**: training sets are subsampled to 10 or 30 examples per intent, while test sets stay unchanged. The training subsets are released for reproducibility.
- **Model compactness**: ConveRT is about 59MB, with 6 Transformer layers and 512-dimensional embeddings, compared with 24 layers and 1024 dimensions in BERT-Large. Frozen BERT features clearly underperform BERT in fine-tuning mode.
- **Released resources**: code is at github.com/PolyAI-LDN/polyai-models and the dataset at github.com/PolyAI-LDN/task-specific-datasets. Planned extensions include multilingual transfer and out-of-scope detection.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

286 papers of our library cite it (of the 140884 whose bibliographies we read); 10 of them have a page here, 276 not yet.

- [ConveRT: Efficient and Accurate Conversational Representations from Transformers](https://md.research.elamur.ai/free/en/papers/henderson-2020-convert-efficient-accurate-conversational-representations/)
- [NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models](https://md.research.elamur.ai/free/en/papers/lee-2024-nv-embed-improved-techniques-training/)
- [Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Words](https://md.research.elamur.ai/free/en/papers/lex-2021-hofmann-superbizarre-derivational/)
- [Long-context LLMs Struggle with Long In-context Learning](https://md.research.elamur.ai/free/en/papers/li-2024-long-context-llms-struggle-long/)
- [On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey](https://md.research.elamur.ai/free/en/papers/long-2024-llms-driven-synthetic-data-generation/)
- [MTEB: Massive Text Embedding Benchmark](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-mteb-massive-text-embedding-benchmark/)
- [DIDECO: An Annotated Dataset for Intent Detection in Digital Communications](https://md.research.elamur.ai/free/en/papers/popovic-2026-dideco-annotated-dataset-intent-detection/)
- [DarijaBanking: A New Resource for Overcoming Language Barriers in Banking Intent Detection for Moroccan Arabic Speakers](https://md.research.elamur.ai/free/en/papers/skiredj-2024-darijabanking-new-resource-overcoming-language/)
- [Probing Pretrained Language Models for Lexical Semantics](https://md.research.elamur.ai/free/en/papers/vulic-2020-probing-pretrained-language-models-lexical/)
- [Can Your Model Tell a Negation from an Implicature? Unravelling Challenges With Intent Encoders](https://md.research.elamur.ai/free/en/papers/zhang-2024-can-your-model-tell-negation/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{casanueva2020efficient,
  title = {{Efficient Intent Detection with Dual Sentence Encoders}},
  author = {Casanueva, Iñigo and Temčinas, Tadas and Gerz, Daniela and Henderson, Matthew and Vulić, Ivan},
  booktitle = {Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI},
  year = {2020},
  doi = {10.18653/v1/2020.nlp4convai-1.5},
  url = {https://doi.org/10.18653/v1/2020.nlp4convai-1.5}
}
```

- [← previous Efficient Dialogue State Tracking by Selectively Overwriting Memory](https://md.research.elamur.ai/free/en/papers/kim-2020-efficient-dialogue-state-tracking-selectively/)
- [next → Efficient Second-Order TreeCRF for Neural Dependency Parsing](https://md.research.elamur.ai/free/en/papers/acl-2020-zhang-efficient-second-order-treecrf-neural/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/2/)

## References

30 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rami Al-Rfou, Marc Pickett, Javier Snaider, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2016. Conversational contextual cues: The case of personalization and history for response ranking. CoRR, abs/1606.00372. — [arXiv:1606.00372](https://arxiv.org/abs/1606.00372) [find](https://scholar.google.com/scholar?q=Rami+Al%2DRfou%2C+Marc+Pickett%2C+Javier+Snaider%2C+Yun%2DHsuan+Sung%2C+Brian+Strope%2C+and+Ray+Kurzweil%2E+2016%2E+Conversational+contextual+cues%3A+The+case+of+personalization+and+history+for+response+ranking%2E+CoRR%2C+ab)
2. Matthew Henderson, Iñigo Casanueva, Nikola Mrkšić, Pei-Hao Su, Ivan Vulić, et al. 2019b. ConveRT: Efficient and accurate conversational representations from transformers. arXiv preprint arXiv:1911.03688. [find](https://scholar.google.com/scholar?q=Matthew+Henderson%2C+In%CC%83igo+Casanueva%2C+Nikola+Mrks%CC%8Cic%CC%81%2C+Pei%2DHao+Su%2C+Ivan+Vulic%CC%81%2C+et+al%2E+2019b%2E+ConveRT%3A+Efficient+and+accurate+conversational+representations+from+transformers%2E+arXiv+preprint+arXiv%3A1911) [our card](https://md.research.elamur.ai/free/en/papers/henderson-2020-convert-efficient-accurate-conversational-representations/)
3. Yuki Arase and Jun’ichi Tsujii. 2019. Transfer fine-tuning: A BERT case study. In Proceedings of EMNLP-IJCNLP, pages 5392–5403. [find](https://scholar.google.com/scholar?q=Yuki+Arase+and+Jun%E2%80%99ichi+Tsujii%2E+2019%2E+Transfer+fine%2Dtuning%3A+A+BERT+case+study%2E+In+Proceedings+of+EMNLP%2DIJCNLP%2C+pages+5392%E2%80%935403%2E)
4. Daniel Braun, Adrian Hernandez Mendez, Florian Matthes, and Manfred Langen. 2017. Evaluating natural language understanding services for conversational question answering systems. In Proceedings of SIGDIAL, pages 174–185. [find](https://scholar.google.com/scholar?q=Daniel+Braun%2C+Adrian+Hernandez+Mendez%2C+Florian+Matthes%2C+and+Manfred+Langen%2E+2017%2E+Evaluating+natural+language+understanding+services+for+conversational+question+answering+systems%2E+In+Proceedings+of+SI)
5. Matthew Henderson, Ivan Vulić, Daniela Gerz, Iñigo Casanueva, Paweł Budzianowski, Sam Coope, Georgios Spithourakis, Tsung-Hsien Wen, Nikola Mrkšić, and Pei-Hao Su. 2019c. Training neural response selection for task-oriented dialogue systems. In Proceedings of ACL, pages 5392–5404. [find](https://scholar.google.com/scholar?q=Matthew+Henderson%2C+Ivan+Vulic%CC%81%2C+Daniela+Gerz%2C+In%CC%83igo+Casanueva%2C+Pawe%C5%82+Budzianowski%2C+Sam+Coope%2C+Georgios+Spithourakis%2C+Tsung%2DHsien+Wen%2C+Nikola+Mrks%CC%8Cic%CC%81%2C+and+Pei%2DHao+Su%2E+2019c%2E+Training+neural+response+)
6. Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018. MultiWOZ - A large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In Proceedings of EMNLP, pages 5016–5026. [find](https://scholar.google.com/scholar?q=Pawe%C5%82+Budzianowski%2C+Tsung%2DHsien+Wen%2C+Bo%2DHsiang+Tseng%2C+In%CC%83igo+Casanueva%2C+Stefan+Ultes%2C+Osman+Ramadan%2C+and+Milica+Gas%CC%8Cic%CC%81%2E+2018%2E+MultiWOZ+%2D+A+large%2Dscale+multi%2Ddomain+wizard%2Dof%2Doz+dataset+for+task%2Dorien)
7. Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2020. Poly-encoders: Transformer architectures and pre-training strategies for fast and accurate multi-sentence scoring. In Proceedings of ICLR, volume abs/1905.01969. — [arXiv:1905.01969](https://arxiv.org/abs/1905.01969) [find](https://scholar.google.com/scholar?q=Samuel+Humeau%2C+Kurt+Shuster%2C+Marie%2DAnne+Lachaux%2C+and+Jason+Weston%2E+2020%2E+Poly%2Dencoders%3A+Transformer+architectures+and+pre%2Dtraining+strategies+for+fast+and+accurate+multi%2Dsentence+scoring%2E+In+Proceedin)
8. Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018. Universal sentence encoder. CoRR, abs/1803.11175. [find](https://scholar.google.com/scholar?q=Daniel+Cer%2C+Yinfei+Yang%2C+Sheng%2Dyi+Kong%2C+Nan+Hua%2C+Nicole+Limtiaco%2C+Rhomni+St%2E+John%2C+Noah+Constant%2C+Mario+Guajardo%2DCespedes%2C+Steve+Yuan%2C+Chris+Tar%2C+Yun%2DHsuan+Sung%2C+Brian+Strope%2C+and+Ray+Kurzweil%2E+2018%2E+) [our card](https://md.research.elamur.ai/free/en/papers/cer-2018-universal-sentence-encoder/)
9. Talaat Khalil, Kornel Kiełczewski, Georgios Christos Chouliaras, Amina Keldibek, and Maarten Versteegh. 2019. Cross-lingual intent classification in a low resource industrial setting. In Proceedings of EMNLP-IJCNLP, pages 6418–6423. [find](https://scholar.google.com/scholar?q=Talaat+Khalil%2C+Kornel+Kie%C5%82czewski%2C+Georgios+Christos+Chouliaras%2C+Amina+Keldibek%2C+and+Maarten+Versteegh%2E+2019%2E+Cross%2Dlingual+intent+classification+in+a+low+resource+industrial+setting%2E+In+Proceedings+o)
10. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. CoRR, abs/1910.01108. [find](https://scholar.google.com/scholar?q=Victor+Sanh%2C+Lysandre+Debut%2C+Julien+Chaumond%2C+and+Thomas+Wolf%2E+2019%2E+DistilBERT%2C+a+distilled+version+of+BERT%3A+Smaller%2C+faster%2C+cheaper+and+lighter%2E+CoRR%2C+abs%2F1910%2E01108%2E)
11. Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019. An evaluation dataset for intent classification and out-of-scope prediction. In Proceedings of EMNLP-IJCNLP, p… [find](https://scholar.google.com/scholar?q=Stefan+Larson%2C+Anish+Mahendran%2C+Joseph+J%2E+Peper%2C+Christopher+Clarke%2C+Andrew+Lee%2C+Parker+Hill%2C+Jonathan+K%2E+Kummerfeld%2C+Kevin+Leach%2C+Michael+A%2E+Laurenzano%2C+Lingjia+Tang%2C+and+Jason+Mars%2E+2019%2E+An+evaluat)
12. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958. [find](https://scholar.google.com/scholar?q=Nitish+Srivastava%2C+Geoffrey+E%2E+Hinton%2C+Alex+Krizhevsky%2C+Ilya+Sutskever%2C+and+Ruslan+Salakhutdinov%2E+2014%2E+Dropout%3A+A+simple+way+to+prevent+neural+networks+from+overfitting%2E+Journal+of+Machine+Learning+R)
13. Xingkun Liu, Arash Eshghi, Pawel Swietojanski, and Verena Rieser. 2019a. Benchmarking natural language understanding services for building conversational agents. In Proceedings of IWSDS. [find](https://scholar.google.com/scholar?q=Xingkun+Liu%2C+Arash+Eshghi%2C+Pawel+Swietojanski%2C+and+Verena+Rieser%2E+2019a%2E+Benchmarking+natural+language+understanding+services+for+building+conversational+agents%2E+In+Proceedings+of+IWSDS%2E)
14. Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of ACL, pages 3645–3650. [find](https://scholar.google.com/scholar?q=Emma+Strubell%2C+Ananya+Ganesh%2C+and+Andrew+McCallum%2E+2019%2E+Energy+and+policy+considerations+for+deep+learning+in+NLP%2E+In+Proceedings+of+ACL%2C+pages+3645%E2%80%933650%2E) [our card](https://md.research.elamur.ai/free/en/papers/strubell-2019-energy-policy-considerations-deep-learning/)
15. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. RoBERTa: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019b%2E+RoBERTa%3A+A+robustly+optimized+BERT+pretraining+approach%2E+)
16. Shyam Upadhyay, Manaal Faruqui, Gökhan Tür, Dilek Hakkani-Tür, and Larry P. Heck. 2018. (almost) zero-shot cross-lingual spoken language understanding. In Proceedings of ICASSP, pages 6034–6038. [find](https://scholar.google.com/scholar?q=Shyam+Upadhyay%2C+Manaal+Faruqui%2C+Go%CC%88khan+Tu%CC%88r%2C+Dilek+Hakkani%2DTu%CC%88r%2C+and+Larry+P%2E+Heck%2E+2018%2E+%28almost%29+zero%2Dshot+cross%2Dlingual+spoken+language+understanding%2E+In+Proceedings+of+ICASSP%2C+pages+6034%E2%80%936038%2E)
17. Zihan Liu, Jamin Shin, Yan Xu, Genta Indra Winata, Peng Xu, Andrea Madotto, and Pascale Fung. 2019c. Zero-shot cross-lingual dialogue systems with transferable latent variables. In Proceedings of EMNLP-IJCNLP, pages 1297–1303. [find](https://scholar.google.com/scholar?q=Zihan+Liu%2C+Jamin+Shin%2C+Yan+Xu%2C+Genta+Indra+Winata%2C+Peng+Xu%2C+Andrea+Madotto%2C+and+Pascale+Fung%2E+2019c%2E+Zero%2Dshot+cross%2Dlingual+dialogue+systems+with+transferable+latent+variables%2E+In+Proceedings+of+EMNL)
18. Tsung-Hsien Wen, David Vandyke, Nikola Mrkšić, Milica Gašić, Lina M. Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017. A network-based end-to-end trainable task-oriented dialogue system. In Proceedings of EACL, pages 438–449. [find](https://scholar.google.com/scholar?q=Tsung%2DHsien+Wen%2C+David+Vandyke%2C+Nikola+Mrks%CC%8Cic%CC%81%2C+Milica+Gas%CC%8Cic%CC%81%2C+Lina+M%2E+Rojas%2DBarahona%2C+Pei%2DHao+Su%2C+Stefan+Ultes%2C+and+Steve+Young%2E+2017%2E+A+network%2Dbased+end%2Dto%2Dend+trainable+task%2Doriented+dialogue+sy)
19. Andrew L. Maas, Awni Y. Hannun, and Andrew Y. Ng. 2013. Rectifier nonlinearities improve neural network acoustic models. In Proceedings of ICML. [find](https://scholar.google.com/scholar?q=Andrew+L%2E+Maas%2C+Awni+Y%2E+Hannun%2C+and+Andrew+Y%2E+Ng%2E+2013%2E+Rectifier+nonlinearities+improve+neural+network+acoustic+models%2E+In+Proceedings+of+ICML%2E)
20. Jason Williams. 2012. A critical analysis of two statistical spoken dialog systems in public use. In Proceedings of SLT. [find](https://scholar.google.com/scholar?q=Jason+Williams%2E+2012%2E+A+critical+analysis+of+two+statistical+spoken+dialog+systems+in+public+use%2E+In+Proceedings+of+SLT%2E)
21. Shikib Mehri, Evgeniia Razumovskaia, Tiancheng Zhao, and Maxine Eskenazi. 2019. Pretraining methods for dialog context representation learning. In Proceedings of ACL, pages 3836–3845. [find](https://scholar.google.com/scholar?q=Shikib+Mehri%2C+Evgeniia+Razumovskaia%2C+Tiancheng+Zhao%2C+and+Maxine+Eskenazi%2E+2019%2E+Pretraining+methods+for+dialog+context+representation+learning%2E+In+Proceedings+of+ACL%2C+pages+3836%E2%80%933845%2E)
22. Anbang Xu, Zhe Liu, Yufan Guo, Vibha Sinha, and Rama Akkiraju. 2017. A new chatbot for customer service on social media. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pages 3506–3510. [find](https://scholar.google.com/scholar?q=Anbang+Xu%2C+Zhe+Liu%2C+Yufan+Guo%2C+Vibha+Sinha%2C+and+Rama+Akkiraju%2E+2017%2E+A+new+chatbot+for+customer+service+on+social+media%2E+In+Proceedings+of+the+2017+CHI+Conference+on+Human+Factors+in+Computing+Systems)
23. Matthew Olson, Abraham J. Wyner, and Richard Berk. 2018. Modern neural networks generalize on small data sets. In Proceedings of NeurIPS, pages 3623– 3632. [find](https://scholar.google.com/scholar?q=Matthew+Olson%2C+Abraham+J%2E+Wyner%2C+and+Richard+Berk%2E+2018%2E+Modern+neural+networks+generalize+on+small+data+sets%2E+In+Proceedings+of+NeurIPS%2C+pages+3623%E2%80%93+3632%2E)
24. Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernández Ábrego, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2019. Multilingual universal sentence encoder for semantic retrieval. CoRR, abs/1907.04307. [find](https://scholar.google.com/scholar?q=Yinfei+Yang%2C+Daniel+Cer%2C+Amin+Ahmad%2C+Mandy+Guo%2C+Jax+Law%2C+Noah+Constant%2C+Gustavo+Herna%CC%81ndez+A%CC%81brego%2C+Steve+Yuan%2C+Chris+Tar%2C+Yun%2DHsuan+Sung%2C+Brian+Strope%2C+and+Ray+Kurzweil%2E+2019%2E+Multilingual+universal+) [our card](https://md.research.elamur.ai/free/en/papers/yang-2020-multilingual-universal-sentence-encoder-semantic/)
25. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of NAACL-HLT, pages 2227–2237. [find](https://scholar.google.com/scholar?q=Matthew+Peters%2C+Mark+Neumann%2C+Mohit+Iyyer%2C+Matt+Gardner%2C+Christopher+Clark%2C+Kenton+Lee%2C+and+Luke+Zettlemoyer%2E+2018%2E+Deep+contextualized+word+representations%2E+In+Proceedings+of+NAACL%2DHLT%2C+pages+2227%E2%80%9322)
26. Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. 2019. Q8BERT: Quantized 8bit BERT. CoRR, abs/1910.06188. — [arXiv:1910.06188](https://arxiv.org/abs/1910.06188) [find](https://scholar.google.com/scholar?q=Ofir+Zafrir%2C+Guy+Boudoukh%2C+Peter+Izsak%2C+and+Moshe+Wasserblat%2E+2019%2E+Q8BERT%3A+Quantized+8bit+BERT%2E+CoRR%2C+abs%2F1910%2E06188%2E+%E2%80%94+%5BarXiv%3A1910%2E06188%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1910%2E06188%29)
27. Matthew E. Peters, Sebastian Ruder, and Noah A. Smith. 2019. To tune or not to tune? Adapting pre-trained representations to diverse tasks. In Proceedings of the 4th Workshop on Representation Learning for NLP, pages 7–14. [find](https://scholar.google.com/scholar?q=Matthew+E%2E+Peters%2C+Sebastian+Ruder%2C+and+Noah+A%2E+Smith%2E+2019%2E+To+tune+or+not+to+tune%3F+Adapting+pre%2Dtrained+representations+to+diverse+tasks%2E+In+Proceedings+of+the+4th+Workshop+on+Representation+Learnin)
28. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8). [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+Blog%2C+1%288%29%2E)
29. Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2019. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. arXiv preprint arXiv:1909.05855. [find](https://scholar.google.com/scholar?q=Abhinav+Rastogi%2C+Xiaoxue+Zang%2C+Srinivas+Sunkara%2C+Raghav+Gupta%2C+and+Pranav+Khaitan%2E+2019%2E+Towards+scalable+multi%2Ddomain+conversational+agents%3A+The+schema%2Dguided+dialogue+dataset%2E+arXiv+preprint+arXiv%3A1)
30. Sebastian Ruder, Matthew E. Peters, Swabha Swayamdipta, and Thomas Wolf. 2019. Transfer learning in natural language processing. In Proceedings of NAACL-HLT: Tutorials, pages 15–18. [find](https://scholar.google.com/scholar?q=Sebastian+Ruder%2C+Matthew+E%2E+Peters%2C+Swabha+Swayamdipta%2C+and+Thomas+Wolf%2E+2019%2E+Transfer+learning+in+natural+language+processing%2E+In+Proceedings+of+NAACL%2DHLT%3A+Tutorials%2C+pages+15%E2%80%9318%2E) [our card](https://md.research.elamur.ai/free/en/papers/ruder-2019-transfer-learning-natural-language-processing/)