---
title: "Investigating Meta-Learning Algorithms for Low-Resource Natural Language Understanding Tasks (2019)"
description: "The authors test whether optimization-based meta-learning gives a better starting point for low-resource English… 35 references, cited by 51 papers here."
authors: ["Zi-Yi Dou", "Keyi Yu", "Antonios Anastasopoulos"]
year: 2019
venue: "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)"
source: "https://aclanthology.org/D19-1112/"
doi: "10.18653/v1/d19-1112"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/dou-2019-investigating-meta-learning-algorithms-low/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/dou-2019-investigating-meta-learning-algorithms-low/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) / Investigating Meta-Learning Algorithms for Low-Resource Natural Language Understanding Tasks

# Investigating Meta-Learning Algorithms for Low-Resource Natural Language Understanding Tasks

**Authors:** [Zi-Yi Dou](https://md.research.elamur.ai/free/en/authors/zi-yi-dou/), Keyi Yu and [Antonios Anastasopoulos](https://md.research.elamur.ai/free/en/authors/antonios-anastasopoulos/) · **Year:** 2019 · **Venue:** [Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2019](https://md.research.elamur.ai/free/en/venues/emnlp/2019/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D19-1112/) · [PDF](https://aclanthology.org/D19-1112.pdf) · [DOI 10.18653/v1/d19-1112](https://doi.org/10.18653/v1/d19-1112) · [OpenAlex](https://openalex.org/W2970678056)

**Citations:** 99 citations (OpenAlex, 2026-10-14) · [51](https://md.research.elamur.ai/free/en/papers/dou-2019-investigating-meta-learning-algorithms-low/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
General-purpose text representations such as BERT and multi-task models like MT-DNN often underperform when a target task has little labeled data. Multi-task training also tends to favor the tasks with the most data. The authors test whether optimization-based meta-learning gives a better starting point for low-resource English NLU tasks. They compare MAML, first-order MAML and Reptile. The pipeline has three stages: BERT BASE pre-training, a meta-learning phase on four data-rich GLUE tasks (SST-2, QQP, MNLI, QNLI), and fine-tuning on four small GLUE tasks (CoLA, MRPC, STS-B, RTE). WNLI is excluded, and BERT and MT-DNN serve as baselines. On the GLUE test sets, the meta-learned models generally beat both baselines, and Reptile is the strongest overall. Ablations cover how tasks are sampled and the settings of the inner update loop. A transfer experiment on the unseen SciTail dataset shows the largest benefit when very little training data is available. This suggests meta-learning is a practical way to initialize models for data-scarce NLU settings.

## Key points
- **GLUE test results for Reptile vs. MT-DNN:** CoLA 53.2 vs 51.7, MRPC 90.2/86.7 vs 89.9/86.3, STS-B 88.7/88.1 vs 87.6/86.8, RTE 77.0 vs 75.4. BERT reaches only 66.4 on RTE. Vanilla MAML scores best on CoLA (53.4).
- **Task sampling:** sampling auxiliary tasks in proportion to dataset size (PPS) beat uniform and mixed sampling. On RTE dev, Reptile scored 83.0 with PPS and 75.7 with uniform sampling.
- **Meta-learning setup:** 5 inner update steps, inner learning rate α = 1e-3 and 8 tasks per step. Training used Adam with batch size 32 and learning rate 5e-5 for 5 epochs.
- **Inner-loop ablations:** both fewer and more inner steps hurt, since k = 1 reduces Reptile to plain joint training. A larger α also degraded results.
- **Transfer to SciTail:** with 0.1%, 1%, 10% and 100% of the training data, Reptile consistently outperformed MT-DNN. Its gains over BERT were largest in the smallest-data settings.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 365 works · 158,048 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🌱 Low-resource languages](https://md.research.elamur.ai/free/en/topics/low-resource-languages/) · 166 works

## Cite

Cite as: Zi-Yi Dou, Keyi Yu and Antonios Anastasopoulos. 2019. Investigating Meta-Learning Algorithms for Low-Resource Natural Language Understanding Tasks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{dou2019investigating,
  title = {{Investigating Meta-Learning Algorithms for Low-Resource Natural Language Understanding Tasks}},
  author = {Zi-Yi Dou, Keyi Yu and Antonios Anastasopoulos},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1112},
  url = {https://doi.org/10.18653/v1/d19-1112}
}
```

- [← previous Answering Complex Open-domain Questions Through Iterative Query Generation](https://md.research.elamur.ai/free/en/papers/qi-2019-answering-complex-open-domain-questions/)
- [next → The Effect of Translationese in Machine Translation Test Sets](https://md.research.elamur.ai/free/en/papers/zhang-2019-effect-translationese-machine-translation-test/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/)

## References

35 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Daniel Cera, Mona Diabb, Eneko Agirrec, Inigo Lopez-Gazpioc, Lucia Speciad, and Basque Country Donostia. 2017. Semeval-2017 task 1: Semantic textual similarity multilingual and cross-lingual focused evaluation. In 11th International Workshop on Semantic Evaluations. [our card](https://md.research.elamur.ai/free/en/papers/cer-2017-semeval-2017-task-1-semantic/)
2. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. JMLR. [our card](https://md.research.elamur.ai/free/en/papers/collobert-2011-natural-language-processing-almost-scratch/)
3. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, pages 177–190. Springer. [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Oren+Glickman%2C+and+Bernardo+Magnini%2E+2005%2E+The+pascal+recognising+textual+entailment+challenge%2E+In+Machine+Learning+Challenges+Workshop%2C+pages+177%E2%80%93190%2E+Springer%2E)
4. Andrew M Dai and Quoc V Le. 2015. Semi-supervised sequence learning. In NeurIPS. [find](https://scholar.google.com/scholar?q=Andrew+M+Dai+and+Quoc+V+Le%2E+2015%2E+Semi%2Dsupervised+sequence+learning%2E+In+NeurIPS%2E)
5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL. [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)
6. William B Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005). [our card](https://md.research.elamur.ai/free/en/papers/dolan-2005-automatically-constructing-corpus-sentential-paraphrases/)
7. Yang Fan, Fei Tian, Tao Qin, Xiang-Yang Li, and Tie-Yan Liu. 2018. Learning to teach. In ICLR. [find](https://scholar.google.com/scholar?q=Yang+Fan%2C+Fei+Tian%2C+Tao+Qin%2C+Xiang%2DYang+Li%2C+and+Tie%2DYan+Liu%2E+2018%2E+Learning+to+teach%2E+In+ICLR%2E)
8. Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML. [find](https://scholar.google.com/scholar?q=Chelsea+Finn%2C+Pieter+Abbeel%2C+and+Sergey+Levine%2E+2017%2E+Model%2Dagnostic+meta%2Dlearning+for+fast+adaptation+of+deep+networks%2E+In+ICML%2E)
9. Jiatao Gu, Yong Wang, Yun Chen, Victor OK Li, and Kyunghyun Cho. 2018. Meta-learning for low-resource neural machine translation. In EMNLP. [our card](https://md.research.elamur.ai/free/en/papers/gu-2018-meta-learning-low-resource-neural/)
10. Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. 2017. Mask r-cnn. In CVPR. [find](https://scholar.google.com/scholar?q=Kaiming+He%2C+Georgia+Gkioxari%2C+Piotr+Dolla%CC%81r%2C+and+Ross+Girshick%2E+2017%2E+Mask+r%2Dcnn%2E+In+CVPR%2E)
11. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In ACL. [find](https://scholar.google.com/scholar?q=Jeremy+Howard+and+Sebastian+Ruder%2E+2018%2E+Universal+language+model+fine%2Dtuning+for+text+classification%2E+In+ACL%2E)
12. Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. Scitail: A textual entailment dataset from science question answering. In AAAI. [find](https://scholar.google.com/scholar?q=Tushar+Khot%2C+Ashish+Sabharwal%2C+and+Peter+Clark%2E+2018%2E+Scitail%3A+A+textual+entailment+dataset+from+science+question+answering%2E+In+AAAI%2E)
13. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR. [find](https://scholar.google.com/scholar?q=Diederik+P+Kingma+and+Jimmy+Ba%2E+2015%2E+Adam%3A+A+method+for+stochastic+optimization%2E+In+ICLR%2E)
14. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In NeurIPS. [find](https://scholar.google.com/scholar?q=Ryan+Kiros%2C+Yukun+Zhu%2C+Ruslan+R+Salakhutdinov%2C+Richard+Zemel%2C+Raquel+Urtasun%2C+Antonio+Torralba%2C+and+Sanja+Fidler%2E+2015%2E+Skip%2Dthought+vectors%2E+In+NeurIPS%2E)
15. Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. 2015. Human-level concept learning through probabilistic program induction. Science. [find](https://scholar.google.com/scholar?q=Brenden+M+Lake%2C+Ruslan+Salakhutdinov%2C+and+Joshua+B+Tenenbaum%2E+2015%2E+Human%2Dlevel+concept+learning+through+probabilistic+program+induction%2E+Science%2E)
16. Quoc Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In ICML. [our card](https://md.research.elamur.ai/free/en/papers/le-2014-distributed-representations-sentences-documents/)
17. Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning. [find](https://scholar.google.com/scholar?q=Hector+Levesque%2C+Ernest+Davis%2C+and+Leora+Morgenstern%2E+2012%2E+The+winograd+schema+challenge%2E+In+Thirteenth+International+Conference+on+the+Principles+of+Knowledge+Representation+and+Reasoning%2E)
18. Xiaodong Liu, Jianfeng Gao, Xiaodong He, Li Deng, Kevin Duh, and Ye-Yi Wang. 2015. Representation learning using multi-task deep neural networks for semantic classification and information retrieval. In NAACL. [our card](https://md.research.elamur.ai/free/en/papers/liu-2015-representation-learning-multi-task-deep/)
19. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019. Multi-task deep neural networks for natural language understanding. arXiv. [our card](https://md.research.elamur.ai/free/en/papers/liu-2019-multi-task-deep-neural-networks/)
20. Minh-Thang Luong, Quoc V Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2015. Multi-task sequence to sequence learning. arXiv. [our card](https://md.research.elamur.ai/free/en/papers/luong-2015-multi-task-sequence-sequence-learning/)
21. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In NeurIPS. [our card](https://md.research.elamur.ai/free/en/papers/mccann-2017-learned-translation-contextualized-word-vectors/)
22. Kun Qian and Zhou Yu. 2019. Domain adaptive dialog generation via meta learning. In ACL. [find](https://scholar.google.com/scholar?q=Kun+Qian+and+Zhou+Yu%2E+2019%2E+Domain+adaptive+dialog+generation+via+meta+learning%2E+In+ACL%2E)
23. Alec Radford, Karthik Narasimhan, Time Salimans, and Ilya Sutskever. 2018. Improving language understanding with unsupervised learning. Technical report, OpenAI. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Time+Salimans%2C+and+Ilya+Sutskever%2E+2018%2E+Improving+language+understanding+with+unsupervised+learning%2E+Technical+report%2C+OpenAI%2E)
24. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392. [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
25. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1… [find](https://scholar.google.com/scholar?q=Richard+Socher%2C+Alex+Perelygin%2C+Jean+Wu%2C+Jason+Chuang%2C+Christopher+D+Manning%2C+Andrew+Ng%2C+and+Christopher+Potts%2E+2013%2E+Recursive+deep+models+for+semantic+compositionality+over+a+sentiment+treebank%2E+In+)
26. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS. [our card](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/)
27. Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. Glue: A multi-task benchmark and analysis platform for natural language understanding. In ICLR. [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)
28. Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2018. Neural network acceptability judgments. arXiv preprint arXiv:1805.12471. [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Amanpreet+Singh%2C+and+Samuel+R+Bowman%2E+2018%2E+Neural+network+acceptability+judgments%2E+arXiv+preprint+arXiv%3A1805%2E12471%2E)
29. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL. [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+Bowman%2E+2018%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+NAACL%2E)
30. Fisher Yu, Dequan Wang, Evan Shelhamer, and Trevor Darrell. 2018. Deep layer aggregation. In CVPR. [find](https://scholar.google.com/scholar?q=Fisher+Yu%2C+Dequan+Wang%2C+Evan+Shelhamer%2C+and+Trevor+Darrell%2E+2018%2E+Deep+layer+aggregation%2E+In+CVPR%2E)
31. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In NeurIPS. [our card](https://md.research.elamur.ai/free/en/papers/mikolov-2013-distributed-representations-words-phrases-their/)
32. Alex Nichol, Joshua Achiam, and John Schulman. 2018. On first-order meta-learning algorithms. arXiv. [find](https://scholar.google.com/scholar?q=Alex+Nichol%2C+Joshua+Achiam%2C+and+John+Schulman%2E+2018%2E+On+first%2Dorder+meta%2Dlearning+algorithms%2E+arXiv%2E)
33. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In EMNLP. [our card](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/)
34. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In NAACL. [our card](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/)
35. Daniel Povey, Gaofeng Cheng, Yiming Wang, Ke Li, Hainan Xu, Mahsa Yarmohamadi, and Sanjeev Khudanpur. 2018. Semi-orthogonal low-rank matrix factorization for deep neural networks. In Inter-Speech. [find](https://scholar.google.com/scholar?q=Daniel+Povey%2C+Gaofeng+Cheng%2C+Yiming+Wang%2C+Ke+Li%2C+Hainan+Xu%2C+Mahsa+Yarmohamadi%2C+and+Sanjeev+Khudanpur%2E+2018%2E+Semi%2Dorthogonal+low%2Drank+matrix+factorization+for+deep+neural+networks%2E+In+Inter%2DSpeech%2E)

*Updated 24 October 2026*