---
title: "Who cites RoBERTa: A Robustly — 9,837 citing papers"
description: "9,837 papers of the Elamur Research library cite «RoBERTa: A Robustly Optimized BERT Pretraining Approach», found in their…"
canonical: "https://research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/cited-by/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/cited-by/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [RoBERTa: A Robustly Optimized BERT Pretraining…](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/) / cited by

# Who cites «RoBERTa: A Robustly Optimized BERT Pretraining Approach»

9,837 papers of our library cite it (of the 149,439 whose bibliographies we read) · 1293 with a page here, the most cited first · page 1 of 13 · 1–100 · [← back to «RoBERTa: A Robustly Optimized BERT Pretraining Approach»](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/)

**1** [2](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/cited-by/page/2/) … [10](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/cited-by/page/10/) … [13](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/cited-by/page/13/) [next →](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/cited-by/page/2/)

1. **[Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/) [Conference on Empirical Methods in Natural Language Processing (EMNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/)**Nils Reimers, Iryna Gurevych*2019 · 17,096 citations*
2. **[Transformers: State-of-the-Art Natural Language Processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/) [Conference on Empirical Methods in Natural Language Processing (EMNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/)**Thomas Wolf, Lysandre Debut, Victor Sanh, et al.*2020 · 15,755 citations*
3. [**Language Models are Few-Shot Learners arXiv:2005.14165**Tom B. Brown, Benjamin Mann, Nick Ryder, et al.*2020 · 14,178 citations*](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
4. **[BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/) [Annual Meeting of the Association for Computational Linguistics (ACL)](https://md.research.elamur.ai/free/en/venues/acl/)**Mike Lewis, Yinhan Liu, Naman Goyal, et al.*2020 · 12,856 citations*
5. [**Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer arXiv:1910.10683**Colin Raffel, Noam Shazeer, Adam Roberts, et al.*2019 · 11,325 citations*](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/)
6. [**Llama 2: Open Foundation and Fine-Tuned Chat Models arXiv:2307.09288**Hugo Touvron, Louis Martin, Kevin Stone, et al.*2023 · 9,669 citations*](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
7. [**LoRA: Low-Rank Adaptation of Large Language Models arXiv:2106.09685**Edward J. Hu, Yelong Shen, Phillip Wallis, et al.*2021 · 8,661 citations*](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/)
8. [**The Llama 3 Herd of Models arXiv:2407.21783**Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al.*2024 · 7,682 citations*](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/)
9. [**ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education**Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, et al.*2023 · 7,440 citations*](https://md.research.elamur.ai/free/en/papers/kasneci-2023-chatgpt-good-opportunities-challenges-large/)
10. **[Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/papers/liu-2021-pre-train-prompt-predict-systematic/) [ACM Computing Surveys](https://md.research.elamur.ai/free/en/venues/acm-computing-surveys/)**Pengfei Liu, Weizhe Yuan, Jinlan Fu, et al.*2021 · 5,423 citations*
11. **[SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://md.research.elamur.ai/free/en/papers/gao-2021-simcse-simple-contrastive-learning-sentence/) [Conference on Empirical Methods in Natural Language Processing (EMNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/)**Tianyu Gao, Xingcheng Yao, Danqi Chen*2021 · 3,942 citations*
12. [**Longformer: The Long-Document Transformer arXiv:2004.05150**Iz Beltagy, Matthew E. Peters, Arman Cohan*2020 · 3,880 citations*](https://md.research.elamur.ai/free/en/papers/beltagy-2020-longformer-long-document-transformer/)
13. [**XLNet: Generalized Autoregressive Pretraining for Language Understanding arXiv:1906.08237**Zhilin Yang, Zihang Dai, Yiming Yang, et al.*2019 · 3,857 citations*](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/)
14. [**Measuring Massive Multitask Language Understanding arXiv:2009.03300**Dan Hendrycks, Collin Burns, Steven Basart, et al.*2020 · 3,620 citations*](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/)
15. **[HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://md.research.elamur.ai/free/en/papers/hsu-2021-hubert-self-supervised-speech-representation/) [IEEE/ACM Transactions on Audio Speech and Language Processing](https://md.research.elamur.ai/free/en/venues/ieee-acm-transactions-on-audio-speech-and-language-processing/)**Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, et al.*2021 · 3,458 citations*
16. [**Scaling Laws for Neural Language Models arXiv:2001.08361**Jared Kaplan, Sam McCandlish, Tom Henighan, et al.*2020 · 3,434 citations*](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/)
17. **[A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://md.research.elamur.ai/free/en/papers/huang-2023-survey-hallucination-large-language-models/) [ACM Transactions on Information Systems](https://md.research.elamur.ai/free/en/venues/acm-transactions-on-information-systems/)**Lei Huang, Weijiang Yu, Weitao Ma, et al.*2023 · 3,231 citations*
18. **[mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer](https://md.research.elamur.ai/free/en/papers/xue-2021-mt5-massively-multilingual-pre-trained/) [Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)](https://md.research.elamur.ai/free/en/venues/naacl/)**Linting Xue, Noah Constant, Adam Roberts, et al.*2021 · 3,226 citations*
19. [**On the Opportunities and Risks of Foundation Models arXiv:2108.07258**Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, et al.*2021 · 3,160 citations*](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/)
20. **[CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://md.research.elamur.ai/free/en/papers/feng-2020-codebert-pre-trained-model-programming/) [Findings of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/findings/)**Zhangyin Feng, Daya Guo, Duyu Tang, et al.*2020 · 2,967 citations*
21. **[QLoRA: Efficient Finetuning of Quantized LLMs](https://md.research.elamur.ai/free/en/papers/dettmers-2023-qlora-efficient-finetuning-quantized-llms/) [Conference on Neural Information Processing Systems (NeurIPS)](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/)**Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, et al.*2023 · 2,929 citations*
22. **[A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/) [Frontiers of Computer Science](https://md.research.elamur.ai/free/en/venues/frontiers-of-computer-science/)**Wayne Xin Zhao, Kun Zhou, Junyi Li, et al.*2023 · 2,875 citations*
23. [**ALBERT: A Lite BERT for Self-supervised Learning of Language Representations arXiv:1909.11942**Zhenzhong Lan, Mingda Chen, Sebastian Goodman, et al.*2019 · 2,806 citations*](https://md.research.elamur.ai/free/en/papers/lan-2019-albert-lite-bert-self-supervised/)
24. [**Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing arXiv:2007.15779**Yu Gu; Robert Tinn; Hao Cheng; Michael Lucas; Naoto Usuyama; Xiaodong Liu; Tristan Naumann; Jianfeng Gao;, Hoifung Poon*2020 · 2,674 citations*](https://md.research.elamur.ai/free/en/papers/gu-2020-domain-specific-language-model-pretraining/)
25. [**Evaluating Large Language Models Trained on Code arXiv:2107.03374**Mark Chen, Jerry Tworek, Heewoo Jun, et al.*2021 · 2,659 citations*](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
26. **[SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://md.research.elamur.ai/free/en/papers/joshi-2019-spanbert-improving-pre-training-representing/) [Transactions of the Association for Computational Linguistics (TACL)](https://md.research.elamur.ai/free/en/venues/tacl/)**Mandar Joshi, Danqi Chen, Yinhan Liu, et al.*2019 · 2,616 citations*
27. **[UNITER: UNiversal Image-TExt Representation Learning](https://md.research.elamur.ai/free/en/papers/chen-2019-uniter-universal-image-text-representation/) [Lecture notes in computer science](https://md.research.elamur.ai/free/en/venues/lecture-notes-in-computer-science/)**Yen-Chun Chen, Linjie Li, Licheng Yu, et al.*2019 · 2,418 citations*
28. **[TinyBERT: Distilling BERT for Natural Language Understanding](https://md.research.elamur.ai/free/en/papers/jiao-2020-tinybert-distilling-bert-natural-language/) [Findings of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/findings/)**Xiaoqi Jiao, Yichun Yin, Lifeng Shang, et al.*2020 · 2,378 citations*
29. **[BLEURT: Learning Robust Metrics for Text Generation](https://md.research.elamur.ai/free/en/papers/sellam-2020-bleurt-learning-robust-metrics-text/) [Annual Meeting of the Association for Computational Linguistics (ACL)](https://md.research.elamur.ai/free/en/venues/acl/)**Thibault Sellam, Dipanjan Das, Ankur Parikh*2020 · 2,032 citations*
30. **[Making Pre-trained Language Models Better Few-shot Learners](https://md.research.elamur.ai/free/en/papers/gao-2021-making-pre-trained-language-models/) [Annual Meeting of the Association for Computational Linguistics (ACL)](https://md.research.elamur.ai/free/en/venues/acl/)**Tianyu Gao, Adam Fisch, Danqi Chen*2021 · 1,987 citations*
31. **[On Faithfulness and Factuality in Abstractive Summarization](https://md.research.elamur.ai/free/en/papers/maynez-2020-faithfulness-factuality-abstractive-summarization/) [Annual Meeting of the Association for Computational Linguistics (ACL)](https://md.research.elamur.ai/free/en/venues/acl/)**Joshua Maynez, Shashi Narayan, Bernd Bohnet, et al.*2020 · 1,857 citations*
32. [**DeBERTa: Decoding-enhanced BERT with Disentangled Attention arXiv:2006.03654**Pengcheng He, Xiaodong Liu, Jianfeng Gao, et al.*2020 · 1,828 citations*](https://md.research.elamur.ai/free/en/papers/he-2020-deberta-decoding-enhanced-bert-disentangled/)
33. **[Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference](https://md.research.elamur.ai/free/en/papers/schick-2021-exploiting-cloze-questions-few-shot/) [Conference of the European Chapter of the Association for Computational Linguistics (EACL)](https://md.research.elamur.ai/free/en/venues/eacl/)**Timo Schick, Hinrich Schütze*2021 · 1,817 citations*
34. [**Pre-trained Models for Natural Language Processing: A Survey arXiv:2003.08271 · SCIENCE CHINA Technological Sciences,2020, 63, 1872-1897**Xipeng Qiu, Tianxiang Sun, Yige Xu, et al.*2020 · 1,759 citations*](https://md.research.elamur.ai/free/en/papers/qiu-2020-pre-trained-models-natural-language/)
35. [**Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models arXiv:2206.04615 · Transactions on Machine Learning Research, May/2022, https://openreview.net/forum?id=uyTL5Bvosj**Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, et al.*2022 · 1,756 citations*](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/)
36. [**OPT: Open Pre-trained Transformer Language Models arXiv:2205.01068**Susan Zhang, Stephen Roller, Naman Goyal, et al.*2022 · 1,755 citations*](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/)
37. [**BERTopic: Neural topic modeling with a class-based TF-IDF procedure arXiv:2203.05794**Maarten Grootendorst*2022 · 1,737 citations*](https://md.research.elamur.ai/free/en/papers/grootendorst-2022-bertopic-neural-topic-modeling-class/)
38. [**Qwen Technical Report arXiv:2309.16609**Jinze Bai, Shuai Bai, Yunfei Chu, et al.*2023 · 1,669 citations*](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/)
39. **[Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://md.research.elamur.ai/free/en/papers/jia-2021-scaling-up-visual-vision-language/) [International Conference on Machine Learning (ICML)](https://md.research.elamur.ai/free/en/venues/international-conference-on-machine-learning/)**Chao Jia, Yinfei Yang, Ye Xia, et al.*2021 · 1,615 citations*
40. [**REALM: Retrieval-Augmented Language Model Pre-Training arXiv:2002.08909**Kelvin Guu, Kenton Lee, Zora Tung, et al.*2020 · 1,580 citations*](https://md.research.elamur.ai/free/en/papers/guu-2020-realm-retrieval-augmented-language-model/)
41. **[Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/min-2021-recent-advances-natural-language-processing/) [ACM Computing Surveys](https://md.research.elamur.ai/free/en/venues/acm-computing-surveys/)**Bonan Min, Hayley Ross, Elior Sulem, et al.*2021 · 1,556 citations*
42. **[Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://md.research.elamur.ai/free/en/papers/pan-2023-unifying-large-language-models-knowledge/) [IEEE Transactions on Knowledge and Data Engineering](https://md.research.elamur.ai/free/en/venues/ieee-transactions-on-knowledge-and-data-engineering/)**Shirui Pan, Linhao Luo, Yufei Wang, et al.*2023 · 1,551 citations*
43. **[LEGAL-BERT: The Muppets straight out of Law School](https://md.research.elamur.ai/free/en/papers/chalkidis-2020-legal-bert-muppets-straight-out/) [Findings of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/findings/)**Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, et al.*2020 · 1,493 citations*
44. **[Pre-Training with Whole Word Masking for Chinese BERT](https://md.research.elamur.ai/free/en/papers/cui-2019-pre-training-whole-word-masking/) [IEEE/ACM Transactions on Audio Speech and Language Processing](https://md.research.elamur.ai/free/en/venues/ieee-acm-transactions-on-audio-speech-and-language-processing/)**Yiming Cui, Wanxiang Che, Ting Liu, et al.*2019 · 1,490 citations*
45. **[GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://md.research.elamur.ai/free/en/papers/du-2022-glm-general-language-model-pretraining/) [Annual Meeting of the Association for Computational Linguistics (ACL)](https://md.research.elamur.ai/free/en/venues/acl/)**Zhengxiao Du, Yujie Qian, Xiao Liu, et al.*2022 · 1,455 citations*
46. **[Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://md.research.elamur.ai/free/en/papers/min-2022-rethinking-role-demonstrations-what-makes/) [Conference on Empirical Methods in Natural Language Processing (EMNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/)**Sewon Min, Xinxi Lyu, Ari Holtzman, et al.*2022 · 1,444 citations*
47. [**Code Llama: Open Foundation Models for Code arXiv:2308.12950**Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, et al.*2024 · 1,430 citations*](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/)
48. **[wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://md.research.elamur.ai/free/en/papers/baevski-2020-wav2vec-2-0-framework-self/) [Conference on Neural Information Processing Systems (NeurIPS)](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/)**Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, et al.*2020 · 1,421 citations*
49. [**LaMDA: Language Models for Dialog Applications arXiv:2201.08239**Romal Thoppilan, Daniel De Freitas, Jamie Hall, et al.*2022 · 1,344 citations*](https://md.research.elamur.ai/free/en/papers/thoppilan-2022-lamda-language-models-dialog-applications/)
50. [**DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing arXiv:2111.09543**Pengcheng He, Jianfeng Gao, Weizhu Chen*2021 · 1,320 citations*](https://md.research.elamur.ai/free/en/papers/he-2021-debertav3-improving-deberta-using-electra/)
51. **[RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://md.research.elamur.ai/free/en/papers/gehman-2020-realtoxicityprompts-evaluating-neural-toxic-degeneration/) [Findings of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/findings/)**Samuel Gehman, Suchin Gururangan, Maarten Sap, et al.*2020 · 1,306 citations*
52. [**AraBERT: Transformer-based Model for Arabic Language Understanding arXiv:2003.00104**Wissam Antoun, Fady Baly, Hazem Hajj*2020 · 1,304 citations*](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/)
53. **[Adversarial NLI: A New Benchmark for Natural Language Understanding](https://md.research.elamur.ai/free/en/papers/nie-2020-adversarial-nli-new-benchmark-natural/) [Annual Meeting of the Association for Computational Linguistics (ACL)](https://md.research.elamur.ai/free/en/venues/acl/)**Yixin Nie, Adina Williams, Emily Dinan, et al.*2020 · 1,296 citations*
54. **[A Comprehensive Overview of Large Language Models](https://md.research.elamur.ai/free/en/papers/naveed-2023-comprehensive-overview-large-language-models/) [ACM Transactions on Intelligent Systems and Technology](https://md.research.elamur.ai/free/en/venues/acm-transactions-on-intelligent-systems-and-technology/)**Humza Naveed, Asad Ullah Khan, Shi Qiu, et al.*2023 · 1,287 citations*
55. **[CamemBERT: a Tasty French Language Model](https://md.research.elamur.ai/free/en/papers/martin-2019-camembert-tasty-french-language-model/) [Annual Meeting of the Association for Computational Linguistics (ACL)](https://md.research.elamur.ai/free/en/venues/acl/)**Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, et al.*2019 · 1,269 citations*
56. **[PIQA: Reasoning about Physical Commonsense in Natural Language](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/) [Proceedings of the AAAI Conference on Artificial Intelligence](https://md.research.elamur.ai/free/en/venues/proceedings-of-the-aaai-conference-on-artificial-intelligence/)**Yonatan Bisk, Rowan Zellers, Ronan Le Bras, et al.*2019 · 1,264 citations*
57. [**What Makes Good In-Context Examples for GPT-3? Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures**Jiachang Liu, Dinghan Shen, Yizhe Zhang, et al.*2022 · 1,256 citations*](https://md.research.elamur.ai/free/en/papers/liu-2022-what-makes-good-context-examples/)
58. **[Recipes for Building an Open-Domain Chatbot](https://md.research.elamur.ai/free/en/papers/roller-2021-recipes-building-open-domain-chatbot/) [Conference of the European Chapter of the Association for Computational Linguistics (EACL)](https://md.research.elamur.ai/free/en/venues/eacl/)**Stephen Roller, Emily Dinan, Naman Goyal, et al.*2021 · 1,214 citations*
59. **[Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://md.research.elamur.ai/free/en/papers/lu-2022-fantastically-ordered-prompts-where-find/) [Annual Meeting of the Association for Computational Linguistics (ACL)](https://md.research.elamur.ai/free/en/venues/acl/)**Yao Lu, Max Bartolo, Alastair Moore, et al.*2022 · 1,209 citations*
60. **[BERTweet: A pre-trained language model for English Tweets](https://md.research.elamur.ai/free/en/papers/nguyen-2020-bertweet-pre-trained-language-model/) [Conference on Empirical Methods in Natural Language Processing (EMNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/)**Dat Quoc Nguyen, Thanh Vu, Anh Tuan Nguyen*2020 · 1,144 citations*
61. **[StereoSet: Measuring stereotypical bias in pretrained language models](https://md.research.elamur.ai/free/en/papers/nadeem-2021-stereoset-measuring-stereotypical-bias-pretrained/) [Annual Meeting of the Association for Computational Linguistics (ACL)](https://md.research.elamur.ai/free/en/venues/acl/)**Moin Nadeem, Anna Bethke, Siva Reddy*2021 · 1,141 citations*
62. **[It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/schick-2021-it-s-not-just-size/) [Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)](https://md.research.elamur.ai/free/en/venues/naacl/)**Timo Schick, Hinrich Schütze*2021 · 1,132 citations*
63. **[Bias and Fairness in Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/gallegos-2023-bias-fairness-large-language-models/) [Computational Linguistics](https://md.research.elamur.ai/free/en/venues/cl/)**Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, et al.*2023 · 1,082 citations*
64. [**VL-BERT: Pre-training of Generic Visual-Linguistic Representations arXiv:1908.08530**Weijie Su, Xizhou Zhu, Yue Cao, et al.*2019 · 1,066 citations*](https://md.research.elamur.ai/free/en/papers/su-2019-vl-bert-pre-training-generic/)
65. **[GPT Understands, Too](https://md.research.elamur.ai/free/en/papers/liu-2021-gpt-understands-too/) [AI Open](https://md.research.elamur.ai/free/en/venues/ai-open/)**Xiao Liu, Yanan Zheng, Zhengxiao Du, et al.*2021 · 1,054 citations*
66. **[Big Bird: Transformers for Longer Sequences](https://md.research.elamur.ai/free/en/papers/zaheer-2020-big-bird-transformers-longer-sequences/) [Conference on Neural Information Processing Systems (NeurIPS)](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/)**Manzil Zaheer, Guru Guruganesh, Avinava Dubey, et al.*2020 · 1,023 citations*
67. **[K-BERT: Enabling Language Representation with Knowledge Graph](https://md.research.elamur.ai/free/en/papers/liu-2019-k-bert-enabling-language-representation/) [Proceedings of the AAAI Conference on Artificial Intelligence](https://md.research.elamur.ai/free/en/venues/proceedings-of-the-aaai-conference-on-artificial-intelligence/)**Weijie Liu, Peng Zhou, Zhe Zhao, et al.*2019 · 1,019 citations*
68. [**GPQA: A Graduate-Level Google-Proof Q&A Benchmark arXiv:2311.12022**David Rein, Betty Li Hou, Asa Cooper Stickland, et al.*2023 · 1,018 citations*](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/)
69. **[P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks](https://md.research.elamur.ai/free/en/papers/liu-2022-p-tuning-prompt-tuning-can/) [Annual Meeting of the Association for Computational Linguistics (ACL)](https://md.research.elamur.ai/free/en/venues/acl/)**Xiao Liu, Kaixuan Ji, Yicheng Fu, et al.*2022 · 1,017 citations*
70. **[MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices](https://md.research.elamur.ai/free/en/papers/sun-2020-mobilebert-compact-task-agnostic-bert/) [Annual Meeting of the Association for Computational Linguistics (ACL)](https://md.research.elamur.ai/free/en/venues/acl/)**Zhiqing Sun, Hongkun Yu, Xiaodan Song, et al.*2020 · 1,016 citations*
71. **[Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models](https://md.research.elamur.ai/free/en/papers/zhang-2023-siren-s-song-ai-ocean/) [Computational Linguistics](https://md.research.elamur.ai/free/en/venues/cl/)**Yue Zhang, Yafu Li, Leyang Cui, et al.*2023 · 977 citations*
72. [**AdapterFusion: Non-Destructive Task Composition for Transfer Learning arXiv:2005.00247 · Proceedings of EACL 2021**Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, et al.*2020 · 958 citations*](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-adapterfusion-non-destructive-task-composition/)
73. [**MPNet: Masked and Permuted Pre-training for Language Understanding arXiv:2004.09297**Kaitao Song, Xu Tan, Tao Qin, et al.*2020 · 957 citations*](https://md.research.elamur.ai/free/en/papers/song-2020-mpnet-masked-permuted-pre-training/)
74. [**TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification arXiv:2010.12421**Francesco Barbieri, Jose Camacho-Collados, Leonardo Neves, et al.*2020 · 956 citations*](https://md.research.elamur.ai/free/en/papers/barbieri-2020-tweeteval-unified-benchmark-comparative-evaluation/)
75. [**Extracting Training Data from Large Language Models arXiv:2012.07805**Nicholas Carlini, Florian Tramer, Eric Wallace, et al.*2020 · 955 citations*](https://md.research.elamur.ai/free/en/papers/carlini-2020-extracting-training-data-large-language/)
76. [**Patient Knowledge Distillation for BERT Model Compression arXiv:1908.09355**Siqi Sun, Yu Cheng, Zhe Gan, et al.*2019 · 947 citations*](https://md.research.elamur.ai/free/en/papers/sun-2019-patient-knowledge-distillation-bert-model/)
77. **[C-Pack: Packed Resources For General Chinese Embeddings](https://md.research.elamur.ai/free/en/papers/xiao-2023-c-pack-packed-resources/) [International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR)](https://md.research.elamur.ai/free/en/venues/international-acm-sigir-conference-on-research-and-development-in-information/)**Shitao Xiao, Zheng Liu, Peitian Zhang, et al.*2023 · 922 citations*
78. **[CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models](https://md.research.elamur.ai/free/en/papers/nangia-2020-crows-pairs-challenge-dataset-measuring/) [Conference on Empirical Methods in Natural Language Processing (EMNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/)**Nikita Nangia, Clara Vania, Rasika Bhalerao, et al.*2020 · 897 citations*
79. [**LayoutLM: Pre-training of Text and Layout for Document Image Understanding arXiv:1912.13318**Yiheng Xu, Minghao Li, Lei Cui, et al.*2019 · 896 citations*](https://md.research.elamur.ai/free/en/papers/xu-2019-layoutlm-pre-training-text-layout/)
80. [**Are NLP Models really able to Solve Simple Math Word Problems?**Arkil Patel, Satwik Bhattamishra, Navin Goyal*2021 · 890 citations*](https://md.research.elamur.ai/free/en/papers/patel-2021-are-nlp-models-really-able/)
81. **[UNIFIEDQA: Crossing Format Boundaries with a Single QA System](https://md.research.elamur.ai/free/en/papers/khashabi-2020-unifiedqa-crossing-format-boundaries-single/) [Findings of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/findings/)**Daniel Khashabi, Sewon Min, Tushar Khot, et al.*2020 · 857 citations*
82. **[TaPas: Weakly Supervised Table Parsing via Pre-training](https://md.research.elamur.ai/free/en/papers/herzig-2020-tapas-weakly-supervised-table-parsing/) [Annual Meeting of the Association for Computational Linguistics (ACL)](https://md.research.elamur.ai/free/en/venues/acl/)**Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, et al.*2020 · 847 citations*
83. **[On the Sentence Embeddings from Pre-trained Language Models](https://md.research.elamur.ai/free/en/papers/li-2020-sentence-embeddings-pre-trained-language/) [Conference on Empirical Methods in Natural Language Processing (EMNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/)**Bohan Li, Hao Zhou, Junxian He, et al.*2020 · 833 citations*
84. **[Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://md.research.elamur.ai/free/en/papers/qin-2023-chatgpt-general-purpose-natural-language/) [Conference on Empirical Methods in Natural Language Processing (EMNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/)**Chengwei Qin, Aston Zhang, Zhuosheng Zhang, et al.*2023 · 828 citations*
85. **[Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond](https://md.research.elamur.ai/free/en/papers/yang-2023-harnessing-power-llms-practice-survey/) [ACM Transactions on Knowledge Discovery from Data](https://md.research.elamur.ai/free/en/venues/acm-transactions-on-knowledge-discovery-from-data/)**Jingfeng Yang, Hongye Jin, Ruixiang Tang, et al.*2023 · 814 citations*
86. **[In-Context Retrieval-Augmented Language Models](https://md.research.elamur.ai/free/en/papers/ram-2023-context-retrieval-augmented-language-models/) [Transactions of the Association for Computational Linguistics (TACL)](https://md.research.elamur.ai/free/en/venues/tacl/)**Ori Ram, Yoav Levine, Itay Dalmedigos, et al.*2023 · 793 citations*
87. **[Can Large Language Models Transform Computational Social Science?](https://md.research.elamur.ai/free/en/papers/ziems-2023-large-language-models-transform-computational/) [Computational Linguistics](https://md.research.elamur.ai/free/en/venues/cl/)**Caleb Ziems, William Held, Omar Shaikh, et al.*2023 · 790 citations*
88. **[Explainability for Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/zhao-2023-explainability-large-language-models-survey/) [ACM Transactions on Intelligent Systems and Technology](https://md.research.elamur.ai/free/en/venues/acm-transactions-on-intelligent-systems-and-technology/)**Haiyan Zhao, Hanjie Chen, Fan Yang, et al.*2023 · 790 citations*
89. [**Revisiting Pre-Trained Models for Chinese Natural Language Processing arXiv:2004.13922**Yiming Cui, Wanxiang Che, Ting Liu, et al.*2020 · 790 citations*](https://md.research.elamur.ai/free/en/papers/cui-2020-revisiting-pre-trained-models-chinese/)
90. **[Towards Reasoning in Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/huang-2023-reasoning-large-language-models-survey/) [Findings of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/findings/)**Jie Huang, Kevin Chen-Chuan Chang*2023 · 780 citations*
91. **[LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking](https://md.research.elamur.ai/free/en/papers/huang-2022-layoutlmv3-pre-training-document-ai/) [ACM International Conference on Multimedia (ACM MM)](https://md.research.elamur.ai/free/en/venues/acm-international-conference-on-multimedia-acm-mm/)**Yupan Huang, Tengchao Lv, Lei Cui, et al.*2022 · 778 citations*
92. **[AdapterHub: A Framework for Adapting Transformers](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-adapterhub-framework-adapting-transformers/) [Conference on Empirical Methods in Natural Language Processing (EMNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/)**Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, et al.*2020 · 754 citations*
93. **[Unified Pre-training for Program Understanding and Generation](https://md.research.elamur.ai/free/en/papers/ahmad-2021-unified-pre-training-program-understanding/) [Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)](https://md.research.elamur.ai/free/en/venues/naacl/)**Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, et al.*2021 · 750 citations*
94. **[Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://md.research.elamur.ai/free/en/papers/ho-2020-constructing-multi-hop-qa-dataset/) [International Conference on Computational Linguistics (COLING)](https://md.research.elamur.ai/free/en/venues/coling/)**Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, et al.*2020 · 741 citations*
95. **[IndicNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages](https://md.research.elamur.ai/free/en/papers/kakwani-2020-indicnlpsuite-monolingual-corpora-evaluation-benchmarks/) [Findings of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/findings/)**Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, et al.*2020 · 732 citations*
96. **[Fact or Fiction: Verifying Scientific Claims](https://md.research.elamur.ai/free/en/papers/wadden-2020-fact-or-fiction-verifying-scientific/) [Conference on Empirical Methods in Natural Language Processing (EMNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/)**David Wadden, Shanchuan Lin, Kyle Lo, et al.*2020 · 709 citations*
97. **[SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://md.research.elamur.ai/free/en/papers/cohan-2020-specter-document-level-representation-learning/) [Annual Meeting of the Association for Computational Linguistics (ACL)](https://md.research.elamur.ai/free/en/venues/acl/)**Arman Cohan, Sergey Feldman, Iz Beltagy, et al.*2020 · 697 citations*
98. **[CORD-19: The COVID-19 Open Research Dataset](https://md.research.elamur.ai/free/en/papers/wang-2020-cord-19-covid-19-open/) [Workshop on NLP for COVID-19 at ACL](https://md.research.elamur.ai/free/en/venues/nlpcovid19/)**Lucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar, et al.*2020 · 695 citations*
99. **[TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data](https://md.research.elamur.ai/free/en/papers/yin-2020-tabert-pretraining-joint-understanding-textual/) [Annual Meeting of the Association for Computational Linguistics (ACL)](https://md.research.elamur.ai/free/en/venues/acl/)**Pengcheng Yin, Graham Neubig, Wen-tau Yih, et al.*2020 · 692 citations*
100. [**Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval arXiv:2007.00808**Lee Xiong, Chenyan Xiong, Ye Li, et al.*2020 · 688 citations*](https://md.research.elamur.ai/free/en/papers/xiong-2020-approximate-nearest-neighbor-negative-contrastive/)

**1** [2](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/cited-by/page/2/) … [10](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/cited-by/page/10/) … [13](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/cited-by/page/13/) [next →](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/cited-by/page/2/)

- [← back to](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/) — RoBERTa: A Robustly Optimized BERT Pretraining Approach