---
title: "Out of Order — paper by Pham et al. (2021)"
description: "The paper asks whether BERT-style classifiers that beat humans on GLUE actually rely on word order. 2.0 F1. 40 references, cited by 66 papers here."
authors: ["Thang Pham", "Trung Bui", "Long Mai", "Anh Nguyen"]
year: 2021
venue: "Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021"
source: "https://aclanthology.org/2021.findings-acl.98/"
doi: "10.18653/v1/2021.findings-acl.98"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/pham-2021-out-order-how-important-is/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/pham-2021-out-order-how-important-is/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/) / Out of Order: How important is the sequential order of words in a sentence in Natural Language Understanding tasks?

# Out of Order: How important is the sequential order of words in a sentence in Natural Language Understanding tasks?

**Authors:** Thang Pham, [Trung Bui](https://md.research.elamur.ai/free/en/authors/trung-bui/), Long Mai and [Anh Nguyen](https://md.research.elamur.ai/free/en/authors/anh-nguyen/) · **Year:** 2021 · **Venue:** [Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2021](https://md.research.elamur.ai/free/en/venues/findings/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.findings-acl.98/) · [PDF](https://aclanthology.org/2021.findings-acl.98.pdf) · [DOI 10.18653/v1/2021.findings-acl.98](https://doi.org/10.18653/v1/2021.findings-acl.98) · [OpenAlex](https://openalex.org/W3117312003)

**Citations:** 79 citations (OpenAlex, 2026-10-14) · [66](https://md.research.elamur.ai/free/en/papers/pham-2021-out-order-how-important-is/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper asks whether BERT-style classifiers that beat humans on GLUE actually rely on word order. The authors fine-tuned base versions of BERT, RoBERTa and ALBERT on seven GLUE tasks: CoLA, SST-2, QQP, MRPC, STS-B, RTE and QNLI. They then re-tested each model on correctly classified, class-balanced dev examples in which a single sentence had its 1-, 2- or 3-grams randomly permuted. A Word-Order Sensitivity (WOS) score measures how far accuracy falls toward the 50% chance level. Apart from CoLA, the models mostly kept their original predictions on scrambled input, so they behave much like bag-of-words systems. Analysis with LIME and attention shows what they use instead. Sentiment models lean on the polarity of key words, and pair-input models have self-attention heads that link matching tokens across the two inputs regardless of position. When RoBERTa was first fine-tuned on synthetic tasks that force attention to word order, results improved on most GLUE tasks and on SQuAD 2.0. The study suggests GLUE does not really require syntax or deep reasoning.

## Key points
- Between 75% and 90% of correct predictions stayed the same after input words were shuffled. About 65% of gold labels across five GLUE tasks could still be recovered from shuffled sentences.
- In SST-2, the sentiment of one salient word predicts the label of the whole sentence in at least 60% of cases.
- WOS is defined as s = (100 − p)/(100 − b), with b = 50. CoLA is the clearly order-sensitive task, followed by RTE among the rest.
- Extra fine-tuning on synthetic tasks raised WOS by 148–194%. Gains included:
  - RTE's average climbed to 73.21% (up from 72.2%), with the best model reaching 74.73%.
  - SQuAD 2.0 F1 rose from 80.62 to 81.08.
  - SST-2 was the only task that did not improve.
- RoBERTa remained fairly insensitive to shuffling even on the harder ANLI benchmark, with a WOS of 0.63.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Adobe Systems](https://md.research.elamur.ai/free/en/companies/adobe-systems/) — Company, San Jose, United States · 14 works · 3,493 citations
- [The University of Queensland](https://md.research.elamur.ai/free/en/universities/the-university-of-queensland/) — University, Brisbane, Australia · 2 works · 4,399 citations
- [Auburn University](https://md.research.elamur.ai/free/en/universities/auburn-university/) — University, Auburn, United States · 2 works · 151 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

## Cite

Cite as: Thang Pham, Trung Bui, Long Mai and Anh Nguyen. 2021. Out of Order: How important is the sequential order of words in a sentence in Natural Language Understanding tasks?. Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{pham2021out,
  title = {{Out of Order: How important is the sequential order of words in a sentence in Natural Language Understanding tasks?}},
  author = {Thang Pham, Trung Bui, Long Mai and Anh Nguyen},
  booktitle = {Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021},
  year = {2021},
  doi = {10.18653/v1/2021.findings-acl.98},
  url = {https://doi.org/10.18653/v1/2021.findings-acl.98}
}
```

- [← previous Including Signed Languages in Natural Language Processing](https://md.research.elamur.ai/free/en/papers/yin-2021-including-signed-languages-natural-language/)
- [next → PLATO-2: Towards Building an Open-Domain Chatbot via Curriculum Learning](https://md.research.elamur.ai/free/en/papers/bao-2021-plato-2-building-open-domain/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/)

## References

40 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural Language Processing with Python: Analyzing Text with the Natural Language Toolkit. O’Reilly, Beijing. [find](https://scholar.google.com/scholar?q=Steven+Bird%2C+Ewan+Klein%2C+and+Edward+Loper%2E+2009%2E+Natural+Language+Processing+with+Python%3A+Analyzing+Text+with+the+Natural+Language+Toolkit%2E+O%E2%80%99Reilly%2C+Beijing%2E)
2. Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14, Vancouver, Canada.… [our card](https://md.research.elamur.ai/free/en/papers/cer-2017-semeval-2017-task-1-semantic/)
3. Peijie Chen, Chirag Agarwal, and Anh Nguyen. 2020. The shape and simplicity biases of adversarially robust imagenet-trained cnns. arXiv e-prints, pages arXiv–2006. [find](https://scholar.google.com/scholar?q=Peijie+Chen%2C+Chirag+Agarwal%2C+and+Anh+Nguyen%2E+2020%2E+The+shape+and+simplicity+biases+of+adversarially+robust+imagenet%2Dtrained+cnns%2E+arXiv+e%2Dprints%2C+pages+arXiv%E2%80%932006%2E)
4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)
5. William B Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005). [our card](https://md.research.elamur.ai/free/en/papers/dolan-2005-automatically-constructing-corpus-sentential-paraphrases/)
6. Jeffrey L Elman. 1990. Finding structure in time. Cognitive science, 14(2):179–211. [find](https://scholar.google.com/scholar?q=Jeffrey+L+Elman%2E+1990%2E+Finding+structure+in+time%2E+Cognitive+science%2C+14%282%29%3A179%E2%80%93211%2E)
7. Kawin Ethayarajh. 2019. How contextual are contextualized word representations? comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process… [find](https://scholar.google.com/scholar?q=Kawin+Ethayarajh%2E+2019%2E+How+contextual+are+contextualized+word+representations%3F+comparing+the+geometry+of+BERT%2C+ELMo%2C+and+GPT%2D2+embeddings%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+i)
8. Allyson Ettinger. 2020. What bert is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34–48. [our card](https://md.research.elamur.ai/free/en/papers/ettinger-2020-what-bert-is-not-lessons/)
9. Lorenzo Gatti, Marco Guerini, and Marco Turchi. 2015. Sentiwords: Deriving a high precision and high coverage lexicon for sentiment analysis. IEEE Transactions on Affective Computing, 7(4):409–421. [find](https://scholar.google.com/scholar?q=Lorenzo+Gatti%2C+Marco+Guerini%2C+and+Marco+Turchi%2E+2015%2E+Sentiwords%3A+Deriving+a+high+precision+and+high+coverage+lexicon+for+sentiment+analysis%2E+IEEE+Transactions+on+Affective+Computing%2C+7%284%29%3A409%E2%80%93421%2E)
10. Edward Gibson, Leon Bergen, and Steven T Piantadosi. 2013. Rational integration of noisy evidence and prior semantic expectations in sentence interpretation. Proceedings of the National Academy of Sciences, 110(20):8051–8056. [find](https://scholar.google.com/scholar?q=Edward+Gibson%2C+Leon+Bergen%2C+and+Steven+T+Piantadosi%2E+2013%2E+Rational+integration+of+noisy+evidence+and+prior+semantic+expectations+in+sentence+interpretation%2E+Proceedings+of+the+National+Academy+of+Sci)
11. GLUE leaderboard. Glue benchmark. [https://gluebenchmark.com/leaderboard](https://gluebenchmark.com/leaderboard) . (Accessed on 01/29/2021). [↗](https://gluebenchmark.com/leaderboard)
12. Eva Hasler, Felix Stahlberg, Marcus Tomalin, Adrià de Gispert, and Bill Byrne. 2017. A comparison of neural models for word ordering. In Proceedings of the 10th International Conference on Natural Language Generation, pages 208–212, Santiago de Compostela, Spain. Association for Computational Lingu… [find](https://scholar.google.com/scholar?q=Eva+Hasler%2C+Felix+Stahlberg%2C+Marcus+Tomalin%2C+Adria%CC%80+de+Gispert%2C+and+Bill+Byrne%2E+2017%2E+A+comparison+of+neural+models+for+word+ordering%2E+In+Proceedings+of+the+10th+International+Conference+on+Natural+La)
13. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+1997%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%3A1735%E2%80%931780%2E)
14. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’04, page 168–177, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Minqing+Hu+and+Bing+Liu%2E+2004%2E+Mining+and+summarizing+customer+reviews%2E+In+Proceedings+of+the+Tenth+ACM+SIGKDD+International+Conference+on+Knowledge+Discovery+and+Data+Mining%2C+KDD+%E2%80%9904%2C+page+168%E2%80%93177%2C+N)
15. Huggingface. 2020. Pretrained models — transformers 3.3.0 documentation. huggingface.co/ transformers/pretrained_models.html . (Accessed on 09/30/2020). [find](https://scholar.google.com/scholar?q=Huggingface%2E+2020%2E+Pretrained+models+%E2%80%94+transformers+3%2E3%2E0+documentation%2E+huggingface%2Eco%2F+transformers%2Fpretrained%5Fmodels%2Ehtml+%2E+%28Accessed+on+09%2F30%2F2020%29%2E)
16. Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky. 2018. Sharp nearby, fuzzy far away: How neural language models use context. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 284–294, Melbourne, Australia. Association for C… [find](https://scholar.google.com/scholar?q=Urvashi+Khandelwal%2C+He+He%2C+Peng+Qi%2C+and+Dan+Jurafsky%2E+2018%2E+Sharp+nearby%2C+fuzzy+far+away%3A+How+neural+language+models+use+context%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Compu)
17. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. Published as a conference paper at the 3rd International Conference for Learning Representations, San Diego. [find](https://scholar.google.com/scholar?q=Diederik+P+Kingma+and+Jimmy+Ba%2E+2015%2E+Adam%3A+A+method+for+stochastic+optimization%2E+Published+as+a+conference+paper+at+the+3rd+International+Conference+for+Learning+Representations%2C+San+Diego%2E)
18. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Zhenzhong+Lan%2C+Mingda+Chen%2C+Sebastian+Goodman%2C+Kevin+Gimpel%2C+Piyush+Sharma%2C+and+Radu+Soricut%2E+2020%2E+Albert%3A+A+lite+bert+for+self%2Dsupervised+learning+of+language+representations%2E+In+International+Confe)
19. Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning. Citeseer. [find](https://scholar.google.com/scholar?q=Hector+Levesque%2C+Ernest+Davis%2C+and+Leora+Morgenstern%2E+2012%2E+The+winograd+schema+challenge%2E+In+Thirteenth+International+Conference+on+the+Principles+of+Knowledge+Representation+and+Reasoning%2E+Citeseer%2E)
20. Yongjie Lin, Yi Chern Tan, and Robert Frank. 2019. Open sesame: Getting inside bert’s linguistic knowledge. Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. [our card](https://md.research.elamur.ai/free/en/papers/lin-2019-open-sesame-getting-inside-bert/)
21. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+a)
22. Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černockỳ, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Eleventh annual conference of the international speech communication association. [find](https://scholar.google.com/scholar?q=Toma%CC%81s%CC%8C+Mikolov%2C+Martin+Karafia%CC%81t%2C+Luka%CC%81s%CC%8C+Burget%2C+Jan+C%CC%8Cernocky%CC%80%2C+and+Sanjeev+Khudanpur%2E+2010%2E+Recurrent+neural+network+based+language+model%2E+In+Eleventh+annual+conference+of+the+international+speech)
23. Jeff Mitchell and Jeffrey Bowers. 2020. Priorless recurrent networks learn curiously. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5147–5158. [find](https://scholar.google.com/scholar?q=Jeff+Mitchell+and+Jeffrey+Bowers%2E+2020%2E+Priorless+recurrent+networks+learn+curiously%2E+In+Proceedings+of+the+28th+International+Conference+on+Computational+Linguistics%2C+pages+5147%E2%80%935158%2E)
24. Francis Mollica, Matthew Siegelman, Evgeniia Diachek, Steven T Piantadosi, Zachary Mineroff, Richard Futrell, Hope Kean, Peng Qian, and Evelina Fedorenko. 2020. Composition is the core driver of the language-selective network. Neurobiology of Language, 1(1):104–134. [find](https://scholar.google.com/scholar?q=Francis+Mollica%2C+Matthew+Siegelman%2C+Evgeniia+Diachek%2C+Steven+T+Piantadosi%2C+Zachary+Mineroff%2C+Richard+Futrell%2C+Hope+Kean%2C+Peng+Qian%2C+and+Evelina+Fedorenko%2E+2020%2E+Composition+is+the+core+driver+of+the+l)
25. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4885–4901, Online. Association for Computa… [find](https://scholar.google.com/scholar?q=Yixin+Nie%2C+Adina+Williams%2C+Emily+Dinan%2C+Mohit+Bansal%2C+Jason+Weston%2C+and+Douwe+Kiela%2E+2020%2E+Adversarial+NLI%3A+A+new+benchmark+for+natural+language+understanding%2E+In+Proceedings+of+the+58th+Annual+Meetin)
26. NLTK. nltk.metrics.distance — nltk 3.5 documentation. [https://www.nltk.org/_modules/](https://www.nltk.org/_modules/) nltk/metrics/distance.html . (Accessed on 12/25/2020). [↗](https://www.nltk.org/_modules/)
27. Chinnadhurai Sankar, Sandeep Subramanian, Chris Pal, Sarath Chandar, and Yoshua Bengio. 2019. Do neural dialog systems use the conversation history effectively? an empirical study. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 32–37, Florence, Ital… [find](https://scholar.google.com/scholar?q=Chinnadhurai+Sankar%2C+Sandeep+Subramanian%2C+Chris+Pal%2C+Sarath+Chandar%2C+and+Yoshua+Bengio%2E+2019%2E+Do+neural+dialog+systems+use+the+conversation+history+effectively%3F+an+empirical+study%2E+In+Proceedings+of+t)
28. Allen Schmaltz, Alexander M. Rush, and Stuart” Shieber. 2016. Word ordering without syntax. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2319–2324, Austin, Texas. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Allen+Schmaltz%2C+Alexander+M%2E+Rush%2C+and+Stuart%E2%80%9D+Shieber%2E+2016%2E+Word+ordering+without+syntax%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+2319%E2%80%932324%2C+)
29. Chenglei Si, Shuohang Wang, Min-Yen Kan, and Jing Jiang. 2019. What does bert learn from multiple-choice reading comprehension datasets? arXiv preprint arXiv:1910.12391. — [arXiv:1910.12391](https://arxiv.org/abs/1910.12391) [find](https://scholar.google.com/scholar?q=Chenglei+Si%2C+Shuohang+Wang%2C+Min%2DYen+Kan%2C+and+Jing+Jiang%2E+2019%2E+What+does+bert+learn+from+multiple%2Dchoice+reading+comprehension+datasets%3F+arXiv+preprint+arXiv%3A1910%2E12391%2E+%E2%80%94+%5BarXiv%3A1910%2E12391%5D%28https%3A%2F%2Fa)
30. Koustuv Sinha, Prasanna Parthasarathi, Joelle Pineau, and Adina Williams. 2020. Unnatural language inference. arXiv preprint arXiv:2101.00010. [our card](https://md.research.elamur.ai/free/en/papers/sinha-2021-unnatural-language-inference/)
31. Swapnil Parekh, Yaman Kumar Singla, Changyou Chen, Junyi Jessy Li, and Rajiv Ratn Shah. 2020. My teacher thinks the world is flat! interpreting automatic essay scoring mechanism. arXiv preprint arXiv:2012.13872. — [arXiv:2012.13872](https://arxiv.org/abs/2012.13872) [find](https://scholar.google.com/scholar?q=Swapnil+Parekh%2C+Yaman+Kumar+Singla%2C+Changyou+Chen%2C+Junyi+Jessy+Li%2C+and+Rajiv+Ratn+Shah%2E+2020%2E+My+teacher+thinks+the+world+is+flat%21+interpreting+automatic+essay+scoring+mechanism%2E+arXiv+preprint+arXiv%3A)
32. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631… [find](https://scholar.google.com/scholar?q=Richard+Socher%2C+Alex+Perelygin%2C+Jean+Wu%2C+Jason+Chuang%2C+Christopher+D+Manning%2C+Andrew+Y+Ng%2C+and+Christopher+Potts%2E+2013%2E+Recursive+deep+models+for+semantic+compositionality+over+a+sentiment+treebank%2E+I)
33. Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2249–2255, Austin, Texas. Association for Computational Linguistic… [our card](https://md.research.elamur.ai/free/en/papers/parikh-2016-decomposable-attention-model-natural-language/)
34. Saku Sugawara, Pontus Stenetorp, Kentaro Inui, and Akiko Aizawa. 2020. Assessing the benchmarking capacity of machine reading comprehension datasets. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8918–8927. [find](https://scholar.google.com/scholar?q=Saku+Sugawara%2C+Pontus+Stenetorp%2C+Kentaro+Inui%2C+and+Akiko+Aizawa%2E+2020%2E+Assessing+the+benchmarking+capacity+of+machine+reading+comprehension+datasets%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificia)
35. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+R%2E+Bowman%2E+2019%2E)
36. Quora. 2017. (1) first quora dataset release: GLUE: A multi-task benchmark and analysis plat-Question pairs - data @ quora - quora. form for natural language understanding. In Inter-https://www.quora.com/q/quoradata/ national Conference on Learning Representations. First-Quora-Dataset-Release-Questi… [find](https://scholar.google.com/scholar?q=Quora%2E+2017%2E+%281%29+first+quora+dataset+release%3A+GLUE%3A+A+multi%2Dtask+benchmark+and+analysis+plat%2DQuestion+pairs+%2D+data+%40+quora+%2D+quora%2E+form+for+natural+language+understanding%2E+In+Inter%2Dhttps%3A%2F%2Fwww%2Equora%2E)
37. Wei Wang, Bin Bi, Ming Yan, Chen Wu, Jiangnan Xia, Zuyi Bao, Liwei Peng, and Luo Si. 2020. [find](https://scholar.google.com/scholar?q=Wei+Wang%2C+Bin+Bi%2C+Ming+Yan%2C+Chen+Wu%2C+Jiangnan+Xia%2C+Zuyi+Bao%2C+Liwei+Peng%2C+and+Luo+Si%2E+2020%2E)
38. Structbert: Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Incorporating language structures into pre-training Know what you don’t know: Unanswerable ques-for deep language understanding. In International tions for SQuAD. In Proceedings of the 56th An-Conference on Learning Representations. nua… [find](https://scholar.google.com/scholar?q=Structbert%3A+Pranav+Rajpurkar%2C+Robin+Jia%2C+and+Percy+Liang%2E+2018%2E+Incorporating+language+structures+into+pre%2Dtraining+Know+what+you+don%E2%80%99t+know%3A+Unanswerable+ques%2Dfor+deep+language+understanding%2E+In+Inte)
39. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for Fabio Massimo Zanzotto, Andrea Santilli, Leonardo machine comprehension of text. In Proceedings of Ranaldi, Dario Onorati, Pierfrancesco Tommasino, the 2016 Conference on Empirical Methods in Natu-… [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Jian+Zhang%2C+Konstantin+Lopyrev%2C+and+Percy+Liang%2E+2016%2E+SQuAD%3A+100%2C000%2B+questions+for+Fabio+Massimo+Zanzotto%2C+Andrea+Santilli%2C+Leonardo+machine+comprehension+of+text%2E+In+Proceedings+o)
40. Yue Zhang and Stephen Clark. 2015. Discriminative syntax-based word ordering for text generation. Computational linguistics, 41(3):503–538. [find](https://scholar.google.com/scholar?q=Yue+Zhang+and+Stephen+Clark%2E+2015%2E+Discriminative+syntax%2Dbased+word+ordering+for+text+generation%2E+Computational+linguistics%2C+41%283%29%3A503%E2%80%93538%2E)

*Updated 24 October 2026*