--- title: "MPNet: Masked and Permuted Pre-training for — Song 2020" description: "BERT and XLNet each use a pre-training objective with its own weakness." authors: ["Kaitao Song", "Xu Tan", "Tao Qin", "Jianfeng Lu", "Tie-Yan Liu"] year: 2020 venue: "arXiv:2004.09297" source: "https://arxiv.org/abs/2004.09297" doi: "10.48550/arxiv.2004.09297" arxiv: "2004.09297" licence: "arXiv — non-exclusive licence to distribute" licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/song-2020-mpnet-masked-permuted-pre-training/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/song-2020-mpnet-masked-permuted-pre-training/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/3/) / MPNet: Masked and Permuted Pre-training for Language Understanding # MPNet: Masked and Permuted Pre-training for Language Understanding **Authors:** [Kaitao Song](https://md.research.elamur.ai/free/en/authors/kaitao-song/), [Xu Tan](https://md.research.elamur.ai/free/en/authors/xu-tan/), [Tao Qin](https://md.research.elamur.ai/free/en/authors/tao-qin/), [Jianfeng Lu](https://md.research.elamur.ai/free/en/authors/jianfeng-lu/), [Tie-Yan Liu](https://md.research.elamur.ai/free/en/authors/tie-yan-liu/) · **Year:** 2020 · **Venue:** arXiv:2004.09297 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2004.09297) · [PDF](https://arxiv.org/pdf/2004.09297) · [DOI 10.48550/arxiv.2004.09297](https://doi.org/10.48550/arxiv.2004.09297) · [arXiv 2004.09297](https://arxiv.org/abs/2004.09297) · [OpenAlex](https://openalex.org/W3016473712) **Citations:** 514 citations (openalex-snapshot · 2026-10-14) · [430](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary BERT and XLNet each use a pre-training objective with its own weakness. BERT's masked language modeling (MLM) predicts each masked token independently of the others. XLNet's permuted language modeling (PLM) captures dependencies between predicted tokens, but it hides the positions of tokens that come later. As a result, the model sees an incomplete sentence during pre-training, which does not match what it gets at fine-tuning. The authors show that both objectives can be described in one framework: a permuted sequence is split into a part used as context and a part to be predicted. Building on this, MPNet predicts tokens one after another in permuted order, as PLM does. It also adds mask symbols that carry the positions of all predicted tokens (called "position compensation"), so the model always knows the full sentence layout. This is implemented with two-stream self-attention. MPNet was pre-trained on over 160GB of text and fine-tuned on GLUE, SQuAD, RACE and IMDB. With the same base model size, it beats BERT, XLNet and RoBERTa. The work matters because it combines the strengths of both objectives in one model, and the model is publicly released. ## Key points - On GLUE dev sets (BERT-BASE setting), MPNet scores 4.8, 3.4 and 1.5 points higher than BERT, XLNet and RoBERTa respectively. - On RACE, MPNet pre-trained on only Wikipedia + BooksCorpus (16GB) reaches 70.4 accuracy, compared with 65.0 for BERT and 66.8 for XLNet. With the full 160GB corpus it reaches 76.1. On IMDB, its error rate drops to 4.4 (BERT: 5.4, XLNet: 4.9). - Ablation (16GB, 1M steps): removing position compensation turns MPNet into PLM and costs 0.6–2.3 points. Reducing it to plain MLM costs 0.5–3.7 points, measured on SQuAD v1.1/v2.0, MNLI and SST-2. - MPNet also clearly beats BERT, XLNet and RoBERTa on SQuAD v1.1 and v2.0 (single model, no data augmentation). - Published at NeurIPS 2020. The code and pre-trained models are available at github.com/microsoft/MPNet. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/) ## Cited by Cited by 430 papers of our library cite it (of the 140673 whose bibliographies we read); 16 of them have a page here, 414 not yet. - [Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering](https://md.research.elamur.ai/free/en/papers/baek-2023-knowledge-augmented-language-model-prompting/) - [Character Coreference Resolution in Movie Screenplays](https://md.research.elamur.ai/free/en/papers/baruah-2023-character-coreference-resolution-movie-screenplays/) - [Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation](https://md.research.elamur.ai/free/en/papers/gao-2023-text-sql-empowered-large-language/) - [BERTopic: Neural topic modeling with a class-based TF-IDF procedure](https://md.research.elamur.ai/free/en/papers/grootendorst-2022-bertopic-neural-topic-modeling-class/) - [TATA: Stance Detection via Topic-Agnostic and Topic-Aware Embeddings](https://md.research.elamur.ai/free/en/papers/hanley-2023-tata-stance-detection-topic-agnostic/) - [Intertextual Parallel Detection in Biblical Hebrew: A Transformer-Based Benchmark](https://md.research.elamur.ai/free/en/papers/hbo-2025-smiley-intertextual-parallel-detection-biblical-hebrew/) - [Bridging Cultures in the Kitchen: A Framework and Benchmark for Cross-Cultural Recipe Retrieval](https://md.research.elamur.ai/free/en/papers/hu-2024-bridging-cultures-kitchen-framework-benchmark/) - [LooGLE: Can Long-Context Language Models Understand Long Contexts?](https://md.research.elamur.ai/free/en/papers/li-2023-loogle-long-context-language-models/) - [The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning](https://md.research.elamur.ai/free/en/papers/lin-2023-unlocking-spell-base-llms-rethinking/) - [Can we obtain significant success in RST discourse parsing by using Large Language Models?](https://md.research.elamur.ai/free/en/papers/maekawa-2024-can-we-obtain-significant-success/) - [MTEB: Massive Text Embedding Benchmark](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-mteb-massive-text-embedding-benchmark/) - [Assessing and Improving Punctuation Robustness in English-Marathi Machine Translation](https://md.research.elamur.ai/free/en/papers/shejole-2026-assessing-improving-punctuation-robustness-english/) - [A Massive Scale Semantic Similarity Dataset of Historical English](https://md.research.elamur.ai/free/en/papers/silcock-2023-massive-scale-semantic-similarity-dataset/) - [An Extreme Multi-label Text Classification (XMTC) Library Dataset: What If We Took “Use of Practical AI in Digital Libraries” Seriously?](https://md.research.elamur.ai/free/en/papers/souza-2026-extreme-multi-label-text-classification/) - [Pregnant Questions: The Importance of Pragmatic Awareness in Maternal Health Question Answering](https://md.research.elamur.ai/free/en/papers/srikanth-2024-pregnant-questions-importance-pragmatic-awareness/) - [Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity](https://md.research.elamur.ai/free/en/papers/wang-2023-survey-factuality-large-language-models/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{song2020mpnet, title = {{MPNet: Masked and Permuted Pre-training for Language Understanding}}, author = {Song, Kaitao and Tan, Xu and Qin, Tao and Lu, Jianfeng and Liu, Tie-Yan}, year = {2020}, doi = {10.48550/arxiv.2004.09297}, eprint = {2004.09297}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2004.09297} } ``` - [← previous Morphology-rich Alphasyllabary Embeddings](https://md.research.elamur.ai/free/en/papers/lingbulk-2020-mersha-morphology-rich-alphasyllabary-embeddings/) - [next → Multi-Domain Dialogue Acts and Response Co-Generation](https://md.research.elamur.ai/free/en/papers/wang-2020-multi-domain-dialogue-acts-response/) - [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/3/) ## References 32 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. URL [https://s3-us-west-2.amazonaws](https://s3-us-west-2.amazonaws). com/openai-assets/researchcovers/languageunsupervised/language understanding paper. pdf, 2018. [↗](https://s3-us-west-2.amazonaws) 2. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, pages 4171–4186, 2019. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+NAACL%2C+pages+4171%E2%80%934186%2C+2019%2E) 3. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, 1(8), 2019. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+Blog%2C+1%288%29%2C+2019%2E) 4. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. MASS: Masked sequence to sequence pre-training for language generation. In ICML, pages 5926–5936, 2019. [find](https://scholar.google.com/scholar?q=Kaitao+Song%2C+Xu+Tan%2C+Tao+Qin%2C+Jianfeng+Lu%2C+and+Tie%2DYan+Liu%2E+MASS%3A+Masked+sequence+to+sequence+pre%2Dtraining+for+language+generation%2E+In+ICML%2C+pages+5926%E2%80%935936%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/song-2019-mass-masked-sequence-sequence-pre/) 5. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. In NIPS, pages 5754–5764, 2019. [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Zihang+Dai%2C+Yiming+Yang%2C+Jaime+Carbonell%2C+Russ+R+Salakhutdinov%2C+and+Quoc+V+Le%2E+Xlnet%3A+Generalized+autoregressive+pretraining+for+language+understanding%2E+In+NIPS%2C+pages+5754%E2%80%935764%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/) 6. Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. Unified language model pre-training for natural language understanding and generation. In NIPS, pages 13042–13054, 2019. [find](https://scholar.google.com/scholar?q=Li+Dong%2C+Nan+Yang%2C+Wenhui+Wang%2C+Furu+Wei%2C+Xiaodong+Liu%2C+Yu+Wang%2C+Jianfeng+Gao%2C+Ming+Zhou%2C+and+Hsiao%2DWuen+Hon%2E+Unified+language+model+pre%2Dtraining+for+natural+language+understanding+and+generation%2E+In+) [our card](https://md.research.elamur.ai/free/en/papers/dong-2019-unified-language-model-pre-training/) 7. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+arXiv+p) 8. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019. [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+Wei+Li%2C+and+Peter+J+Liu%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtext+trans) 9. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. Spanbert: Improving pre-training by representing and predicting spans. arXiv preprint arXiv:1907.10529, 2019. [find](https://scholar.google.com/scholar?q=Mandar+Joshi%2C+Danqi+Chen%2C+Yinhan+Liu%2C+Daniel+S+Weld%2C+Luke+Zettlemoyer%2C+and+Omer+Levy%2E+Spanbert%3A+Improving+pre%2Dtraining+by+representing+and+predicting+spans%2E+arXiv+preprint+arXiv%3A1907%2E10529%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/joshi-2019-spanbert-improving-pre-training-representing/) 10. Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555, 2020. [find](https://scholar.google.com/scholar?q=Kevin+Clark%2C+Minh%2DThang+Luong%2C+Quoc+V+Le%2C+and+Christopher+D+Manning%2E+Electra%3A+Pre%2Dtraining+text+encoders+as+discriminators+rather+than+generators%2E+arXiv+preprint+arXiv%3A2003%2E10555%2C+2020%2E) 11. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pages 5998–6008, 2017. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81+ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+Attention+is+all+you+need%2E+In+NIPS%2C+pages+5998%E2%80%936008%2C+2017%2E) 12. Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang, Shijin Wang, and Guoping Hu. Pre-training with whole word masking for chinese BERT. CoRR, abs/1906.08101, 2019. [find](https://scholar.google.com/scholar?q=Yiming+Cui%2C+Wanxiang+Che%2C+Ting+Liu%2C+Bing+Qin%2C+Ziqing+Yang%2C+Shijin+Wang%2C+and+Guoping+Hu%2E+Pre%2Dtraining+with+whole+word+masking+for+chinese+BERT%2E+CoRR%2C+abs%2F1906%2E08101%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/cui-2019-pre-training-whole-word-masking/) 13. Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In NAACL, pages 464–468, June 2018. [find](https://scholar.google.com/scholar?q=Peter+Shaw%2C+Jakob+Uszkoreit%2C+and+Ashish+Vaswani%2E+Self%2Dattention+with+relative+position+representations%2E+In+NAACL%2C+pages+464%E2%80%93468%2C+June+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/shaw-2018-self-attention-relative-position-representations/) 14. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints, 2019. [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+Wei+Li%2C+and+Peter+J%2E+Liu%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtext+tran) 15. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In ICCV, pages 19–27, December 2015. [find](https://scholar.google.com/scholar?q=Yukun+Zhu%2C+Ryan+Kiros%2C+Rich+Zemel%2C+Ruslan+Salakhutdinov%2C+Raquel+Urtasun%2C+Antonio+Torralba%2C+and+Sanja+Fidler%2E+Aligning+books+and+movies%3A+Towards+story%2Dlike+visual+explanations+by+watching+movies+and+re) [our card](https://md.research.elamur.ai/free/en/papers/zhu-2015-aligning-books-movies-towards-story/) 16. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+Language+models+are+unsupervised+multitask+learners%2E+2019%2E) 17. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692, 2019. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+Roberta%3A+A+robustly+optimized+BERT+pretraining+approach%2E+CoRR%2C+a) 18. Trieu H. Trinh and Quoc V. Le. A simple method for commonsense reasoning. CoRR, abs/1806.02847, 2018. [find](https://scholar.google.com/scholar?q=Trieu+H%2E+Trinh+and+Quoc+V%2E+Le%2E+A+simple+method+for+commonsense+reasoning%2E+CoRR%2C+abs%2F1806%2E02847%2C+2018%2E) 19. Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014. [find](https://scholar.google.com/scholar?q=Diederik+P%2E+Kingma+and+Jimmy+Ba%2E+Adam%3A+A+method+for+stochastic+optimization%2C+2014%2E) 20. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR, 2019. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+R%2E+Bowman%2E+GLUE%3A+A+multi%2Dtask+benchmark+and+analysis+platform+for+natural+language+understanding%2E+In+ICLR%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/) 21. Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. CoRR, abs/1805.12471, 2018. [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Amanpreet+Singh%2C+and+Samuel+R%2E+Bowman%2E+Neural+network+acceptability+judgments%2E+CoRR%2C+abs%2F1805%2E12471%2C+2018%2E) 22. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, pages 1631–1642, October 2013. [find](https://scholar.google.com/scholar?q=Richard+Socher%2C+Alex+Perelygin%2C+Jean+Wu%2C+Jason+Chuang%2C+Christopher+D%2E+Manning%2C+Andrew+Ng%2C+and+Christopher+Potts%2E+Recursive+deep+models+for+semantic+compositionality+over+a+sentiment+treebank%2E+In+EMNLP) 23. William B. Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005. [find](https://scholar.google.com/scholar?q=William+B%2E+Dolan+and+Chris+Brockett%2E+Automatically+constructing+a+corpus+of+sentential+paraphrases%2E+In+Proceedings+of+the+Third+International+Workshop+on+Paraphrasing+%28IWP2005%29%2C+2005%2E) [our card](https://md.research.elamur.ai/free/en/papers/dolan-2005-automatically-constructing-corpus-sentential-paraphrases/) 24. Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14, August 2017. [find](https://scholar.google.com/scholar?q=Daniel+Cer%2C+Mona+Diab%2C+Eneko+Agirre%2C+I%C3%B1igo+Lopez%2DGazpio%2C+and+Lucia+Specia%2E+SemEval%2D2017+task+1%3A+Semantic+textual+similarity+multilingual+and+crosslingual+focused+evaluation%2E+In+Proceedings+of+the+11th) 25. Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL, pages 1112–1122, June 2018. [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+Bowman%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+NAACL%2C+pages+1112%E2%80%931122%2C+June+2018%2E) 26. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP, pages 2383–2392, 2016. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Jian+Zhang%2C+Konstantin+Lopyrev%2C+and+Percy+Liang%2E+SQuAD%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+In+EMNLP%2C+pages+2383%E2%80%932392%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/) 27. Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment, pages 177–190, 2006. [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Oren+Glickman%2C+and+Bernardo+Magnini%2E+The+pascal+recognising+textual+entailment+challenge%2E+In+Machine+Learning+Challenges%2E+Evaluating+Predictive+Uncertainty%2C+Visual+Object+Classification%2C+an) 28. Hector J. Levesque, Ernest Davis, and Leora Morgenstern. The Winograd Schema Challenge. In Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning, pages 552–561. Rome, Italy, 2012. [find](https://scholar.google.com/scholar?q=Hector+J%2E+Levesque%2C+Ernest+Davis%2C+and+Leora+Morgenstern%2E+The+Winograd+Schema+Challenge%2E+In+Proceedings+of+the+Thirteenth+International+Conference+on+Principles+of+Knowledge+Representation+and+Reasonin) 29. Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable questions for SQuAD. In ACL, pages 784–789, 2018. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Robin+Jia%2C+and+Percy+Liang%2E+Know+what+you+don%E2%80%99t+know%3A+Unanswerable+questions+for+SQuAD%2E+In+ACL%2C+pages+784%E2%80%93789%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/) 30. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. RACE: Large-scale ReAding comprehension dataset from examinations. In EMNLP, pages 785–794, 2017. [find](https://scholar.google.com/scholar?q=Guokun+Lai%2C+Qizhe+Xie%2C+Hanxiao+Liu%2C+Yiming+Yang%2C+and+Eduard+Hovy%2E+RACE%3A+Large%2Dscale+ReAding+comprehension+dataset+from+examinations%2E+In+EMNLP%2C+pages+785%E2%80%93794%2C+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/) 31. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142–150, June 2011. [find](https://scholar.google.com/scholar?q=Andrew+L%2E+Maas%2C+Raymond+E%2E+Daly%2C+Peter+T%2E+Pham%2C+Dan+Huang%2C+Andrew+Y%2E+Ng%2C+and+Christopher+Potts%2E+Learning+word+vectors+for+sentiment+analysis%2E+In+Proceedings+of+the+49th+Annual+Meeting+of+the+Associati) [our card](https://md.research.elamur.ai/free/en/papers/maas-2011-learning-word-vectors-sentiment-analysis/) 32. Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. How to fine-tune BERT for text classification? CoRR, abs/1905.05583, 2019. [find](https://scholar.google.com/scholar?q=Chi+Sun%2C+Xipeng+Qiu%2C+Yige+Xu%2C+and+Xuanjing+Huang%2E+How+to+fine%2Dtune+BERT+for+text+classification%3F+CoRR%2C+abs%2F1905%2E05583%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/sun-2019-fine-tune-bert-text-classification/)