--- title: "GPT Understands, Too — Liu 2021" description: "Hand-written natural-language prompts help pretrained language models on NLU tasks, but the authors show these prompts are fragile." authors: ["Xiao Liu", "Yanan Zheng", "Zhengxiao Du", "Ming Ding", "Yujie Qian", "Zhilin Yang", "Jie Tang"] year: 2021 venue: "arXiv:2103.10385" source: "https://arxiv.org/abs/2103.10385" doi: "10.48550/arxiv.2103.10385" arxiv: "2103.10385" licence: "CC BY 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/liu-2021-gpt-understands-too/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liu-2021-gpt-understands-too/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/2/) / GPT Understands, Too # GPT Understands, Too **Authors:** [Xiao Liu](https://md.research.elamur.ai/free/en/authors/xiao-liu/), Yanan Zheng, [Zhengxiao Du](https://md.research.elamur.ai/free/en/authors/zhengxiao-du/), [Ming Ding](https://md.research.elamur.ai/free/en/authors/ming-ding/), [Yujie Qian](https://md.research.elamur.ai/free/en/authors/yujie-qian/), [Zhilin Yang](https://md.research.elamur.ai/free/en/authors/zhilin-yang/), [Jie Tang](https://md.research.elamur.ai/free/en/authors/jie-tang/) · **Year:** 2021 · **Venue:** arXiv:2103.10385 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2103.10385) · [PDF](https://arxiv.org/pdf/2103.10385) · [DOI 10.48550/arxiv.2103.10385](https://doi.org/10.48550/arxiv.2103.10385) · [arXiv 2103.10385](https://arxiv.org/abs/2103.10385) · [OpenAlex](https://openalex.org/W3139080614) **Citations:** 596 citations (openalex-snapshot · 2026-10-14) · [443](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Hand-written natural-language prompts help pretrained language models on NLU tasks, but the authors show these prompts are fragile. Swapping a single word in a template can move accuracy by roughly 20 points. Automatic prompt search finds stronger templates, yet it still operates over discrete tokens, so the fragility persists and gradient information is only partly used. The paper proposes P-Tuning, which places trainable continuous vectors ("pseudo" prompt tokens) next to the ordinary discrete prompt words and optimizes them by backpropagation on the task objective. A small prompt encoder, either an LSTM or an MLP, produces these vectors so they are not learned in isolation. Experiments cover LAMA knowledge probing with frozen models and SuperGLUE with fine-tuned models, in both few-shot and fully-supervised settings. P-Tuning improves scores and greatly reduces the variation between manual templates. It beats hand-crafted and searched prompts on LAMA and outperforms PET on SuperGLUE. In practice, adapting a model becomes far less sensitive to exact prompt wording, whether the model is frozen or tuned. ## Key points - On LAMA-TREx P17 with BERT-base-cased, four manual templates ranged from 19.8 to 51.1 precision@1. With P-Tuning, all four reached about 57.8–58.1. - On LAMA with frozen models, P-Tuning exceeded manual prompts by over 20 points and searched prompts by about 9 points. The standard deviation across templates dropped from 10.1 to 0.46. - On SuperGLUE with fine-tuned models, P-Tuning outperformed PET using its best discrete prompts, in both few-shot and fully-supervised settings. - In few-shot ALBERT experiments, LSTM and MLP prompt encoders were reliable. A plain embedding (identity) encoder was unstable, for example CB-F1 of 67.27 versus 84.24 with MLP. On CB, four continuous tokens worked best (avg 86.76), while eight hurt performance (78.40). - Adding continuous prompts on top of LM-BFF's automatically searched templates gave small gains on MNLI (58.70 vs 57.53) and MRPC (69.49 vs 68.26). Results on SST-2 were roughly unchanged. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/) ## Cited by Cited by 443 papers of our library cite it (of the 140673 whose bibliographies we read); 35 of them have a page here, 408 not yet. - [On the Opportunities and Risks of Foundation Models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/) - [Exploring the Potential of ChatGPT on Sentence Level Relations: A Focus on Temporal, Causal, and Discourse Relations](https://md.research.elamur.ai/free/en/papers/chan-2024-exploring-potential-chatgpt-sentence-level/) - [RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/deng-2022-rlprompt-optimizing-discrete-text-prompts/) - [Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution](https://md.research.elamur.ai/free/en/papers/fernando-2023-promptbreeder-self-referential-self-improvement/) - [Bias and Fairness in Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/gallegos-2023-bias-fairness-large-language-models/) - [PPT: Pre-trained Prompt Tuning for Few-shot Learning](https://md.research.elamur.ai/free/en/papers/gu-2022-ppt-pre-trained-prompt-tuning/) - [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://md.research.elamur.ai/free/en/papers/han-2024-parameter-efficient-fine-tuning-large/) - [Towards a Unified View of Parameter-Efficient Transfer Learning](https://md.research.elamur.ai/free/en/papers/he-2021-towards-unified-view-parameter-efficient/) - [Few-shot Learning with Multilingual Language Models](https://md.research.elamur.ai/free/en/papers/lin-2021-few-shot-learning-multilingual-language/) - [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://md.research.elamur.ai/free/en/papers/liu-2021-p-tuning-v2-prompt-tuning/) - [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/papers/liu-2021-pre-train-prompt-predict-systematic/) - [P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks](https://md.research.elamur.ai/free/en/papers/liu-2022-p-tuning-prompt-tuning-can/) - [Generating Training Data with Language Models: Towards Zero-Shot Language Understanding](https://md.research.elamur.ai/free/en/papers/meng-2022-generating-training-data-language-models/) - [Noisy Channel Language Model Prompting for Few-Shot Text Classification](https://md.research.elamur.ai/free/en/papers/min-2021-noisy-channel-language-model-prompting/) - [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/min-2021-recent-advances-natural-language-processing/) - [Reframing Instructional Prompts to GPTk's Language](https://md.research.elamur.ai/free/en/papers/mishra-2022-reframing-instructional-prompts-gptk-s/) - [A Comprehensive Overview of Large Language Models](https://md.research.elamur.ai/free/en/papers/naveed-2023-comprehensive-overview-large-language-models/) - [True Few-Shot Learning with Language Models](https://md.research.elamur.ai/free/en/papers/perez-2021-true-few-shot-learning-language/) - [Learning How to Ask: Querying LMs with Mixtures of Soft Prompts](https://md.research.elamur.ai/free/en/papers/qin-2021-learning-ask-querying-lms-mixtures/) - [Pre-trained Models for Natural Language Processing: A Survey](https://md.research.elamur.ai/free/en/papers/qiu-2020-pre-trained-models-natural-language/) - [mGPT: Few-Shot Learners Go Multilingual](https://md.research.elamur.ai/free/en/papers/shliazhko-2022-mgpt-few-shot-learners-go/) - [SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer](https://md.research.elamur.ai/free/en/papers/vu-2022-spot-better-frozen-model-adaptation/) - [CogVLM: Visual Expert for Pretrained Language Models](https://md.research.elamur.ai/free/en/papers/wang-2023-cogvlm-visual-expert-pretrained/) - [Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data](https://md.research.elamur.ai/free/en/papers/xu-2023-baize-open-source-chat-model/) - [Large Language Models as Optimizers](https://md.research.elamur.ai/free/en/papers/yang-2023-large-language-models-optimizers/) - [视觉语言预训练综述](https://md.research.elamur.ai/free/en/papers/yin-2022-vision-language-pre/) - [GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation](https://md.research.elamur.ai/free/en/papers/yoo-2021-gpt3mix-leveraging-large-scale-language/) - [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://md.research.elamur.ai/free/en/papers/yu-2023-language-models-super-mario-absorbing/) - [GLM-130B: An Open Bilingual Pre-trained Model](https://md.research.elamur.ai/free/en/papers/zeng-2022-glm-130b-open-bilingual-pre/) - [Aspect Sentiment Quad Prediction as Paraphrase Generation](https://md.research.elamur.ai/free/en/papers/zhang-2021-aspect-sentiment-quad-prediction-as/) - [LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention](https://md.research.elamur.ai/free/en/papers/zhang-2023-llama-adapter-efficient-fine-tuning/) - [Infusing Hierarchical Guidance into Prompt Tuning: A Parameter-Efficient Framework for Multi-level Implicit Discourse Relation Recognition](https://md.research.elamur.ai/free/en/papers/zhao-2023-infusing-hierarchical-guidance-into-prompt/) - [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/) - [Factual Probing Is MASK: Learning vs. Learning to Recall](https://md.research.elamur.ai/free/en/papers/zhong-2021-factual-probing-is-mask-learning/) - [Large Language Models Are Human-Level Prompt Engineers](https://md.research.elamur.ai/free/en/papers/zhou-2022-large-language-models-human-level/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{liu2021gpt, title = {{GPT Understands, Too}}, author = {Liu, Xiao and Zheng, Yanan and Du, Zhengxiao and Ding, Ming and Qian, Yujie and Yang, Zhilin and Tang, Jie}, year = {2021}, doi = {10.48550/arxiv.2103.10385}, eprint = {2103.10385}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2103.10385} } ``` - [← previous GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://md.research.elamur.ai/free/en/papers/du-2021-glam-efficient-scaling-language-models/) - [next → GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation](https://md.research.elamur.ai/free/en/papers/yoo-2021-gpt3mix-leveraging-large-scale-language/) - [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/2/) ## References 41 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165. [find](https://scholar.google.com/scholar?q=Tom+B+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+et+al%2E+2020%2E+Language+models+are+few%2Dshot+lear) 2. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019a. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics… [find](https://scholar.google.com/scholar?q=Christopher+Clark%2C+Kenton+Lee%2C+Ming%2DWei+Chang%2C+Tom+Kwiatkowski%2C+Michael+Collins%2C+and+Kristina+Toutanova%2E+2019a%2E+Boolq%3A+Exploring+the+surprising+difficulty+of+natural+yes%2Fno+questions%2E+In+Proceedings+o) [our card](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/) 3. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. 2019b. What does bert look at? an analysis of bert’s attention. arXiv preprint arXiv:1906.04341. [find](https://scholar.google.com/scholar?q=Kevin+Clark%2C+Urvashi+Khandelwal%2C+Omer+Levy%2C+and+Christopher+D+Manning%2E+2019b%2E+What+does+bert+look+at%3F+an+analysis+of+bert%E2%80%99s+attention%2E+arXiv+preprint+arXiv%3A1906%2E04341%2E) 4. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, pages 177–190. Springer. [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Oren+Glickman%2C+and+Bernardo+Magnini%2E+2005%2E+The+pascal+recognising+textual+entailment+challenge%2E+In+Machine+Learning+Challenges+Workshop%2C+pages+177%E2%80%93190%2E+Springer%2E) 5. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860. [find](https://scholar.google.com/scholar?q=Zihang+Dai%2C+Zhilin+Yang%2C+Yiming+Yang%2C+Jaime+Carbonell%2C+Quoc+V+Le%2C+and+Ruslan+Salakhutdinov%2E+2019%2E+Transformer%2Dxl%3A+Attentive+language+models+beyond+a+fixed%2Dlength+context%2E+arXiv+preprint+arXiv%3A1901%2E028) 6. Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. 2020b. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423–438. [find](https://scholar.google.com/scholar?q=Zhengbao+Jiang%2C+Frank+F+Xu%2C+Jun+Araki%2C+and+Graham+Neubig%2E+2020b%2E+How+can+we+know+what+language+models+know%3F+Transactions+of+the+Association+for+Computational+Linguistics%2C+8%3A423%E2%80%93438%2E) [our card](https://md.research.elamur.ai/free/en/papers/jiang-2020-how-can-we-know-what/) 7. Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics:… [find](https://scholar.google.com/scholar?q=Daniel+Khashabi%2C+Snigdha+Chaturvedi%2C+Michael+Roth%2C+Shyam+Upadhyay%2C+and+Dan+Roth%2E+2018%2E+Looking+beyond+the+surface%3A+A+challenge+set+for+reading+comprehension+over+multiple+sentences%2E+In+Proceedings+of+) [our card](https://md.research.elamur.ai/free/en/papers/khashabi-2018-looking-beyond-surface-challenge-set/) 8. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. ArXiv, abs/2104.08691. [find](https://scholar.google.com/scholar?q=Brian+Lester%2C+Rami+Al%2DRfou%2C+and+Noah+Constant%2E+2021%2E+The+power+of+scale+for+parameter%2Defficient+prompt+tuning%2E+ArXiv%2C+abs%2F2104%2E08691%2E) 9. Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning. Citeseer. [find](https://scholar.google.com/scholar?q=Hector+Levesque%2C+Ernest+Davis%2C+and+Leora+Morgenstern%2E+2012%2E+The+winograd+schema+challenge%2E+In+Thirteenth+International+Conference+on+the+Principles+of+Knowledge+Representation+and+Reasoning%2E+Citeseer%2E) 10. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190. [find](https://scholar.google.com/scholar?q=Xiang+Lisa+Li+and+Percy+Liang%2E+2021%2E+Prefix%2Dtuning%3A+Optimizing+continuous+prompts+for+generation%2E+arXiv+preprint+arXiv%3A2101%2E00190%2E) 11. Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021. What makes good in-context examples for gpt-3? arXiv preprint arXiv:2101.06804. [find](https://scholar.google.com/scholar?q=Jiachang+Liu%2C+Dinghan+Shen%2C+Yizhe+Zhang%2C+Bill+Dolan%2C+Lawrence+Carin%2C+and+Weizhu+Chen%2E+2021%2E+What+makes+good+in%2Dcontext+examples+for+gpt%2D3%3F+arXiv+preprint+arXiv%3A2101%2E06804%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2022-what-makes-good-context-examples/) 12. Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. 2019. The commitmentbank: Investigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, volume 23, pages 107–124. [find](https://scholar.google.com/scholar?q=Marie%2DCatherine+De+Marneffe%2C+Mandy+Simons%2C+and+Judith+Tonhauser%2E+2019%2E+The+commitmentbank%3A+Investigating+projection+in+naturally+occurring+discourse%2E+In+proceedings+of+Sinn+und+Bedeutung%2C+volume+23%2C+p) [our card](https://md.research.elamur.ai/free/en/papers/enprag-2019-marneffe-commitmentbank-investigating-projection-naturally/) 13. Xiao Liu, Fanjin Zhang, Zhenyu Hou, Zhaoyu Wang, Li Mian, Jing Zhang, and Jie Tang. 2020. Self-supervised learning: Generative or contrastive. arXiv preprint arXiv:2006.08218, 1(2). — [arXiv:2006.08218](https://arxiv.org/abs/2006.08218) [find](https://scholar.google.com/scholar?q=Xiao+Liu%2C+Fanjin+Zhang%2C+Zhenyu+Hou%2C+Zhaoyu+Wang%2C+Li+Mian%2C+Jing+Zhang%2C+and+Jie+Tang%2E+2020%2E+Self%2Dsupervised+learning%3A+Generative+or+contrastive%2E+arXiv+preprint+arXiv%3A2006%2E08218%2C+1%282%29%2E+%E2%80%94+%5BarXiv%3A2006%2E0821) 14. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2018%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+arXiv+preprint+arXiv%3A1810%2E04805%2E) 15. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+a) 16. Tianyu Gao, Adam Fisch, and Danqi Chen. 2020. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723. [find](https://scholar.google.com/scholar?q=Tianyu+Gao%2C+Adam+Fisch%2C+and+Danqi+Chen%2E+2020%2E+Making+pre%2Dtrained+language+models+better+few%2Dshot+learners%2E+arXiv+preprint+arXiv%3A2012%2E15723%2E) [our card](https://md.research.elamur.ai/free/en/papers/gao-2021-making-pre-trained-language-models/) 17. Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Liang Zhang, Wentao Han, Minlie Huang, et al. 2021. Pre-trained models: Past, present and future. AI Open. [find](https://scholar.google.com/scholar?q=Xu+Han%2C+Zhengyan+Zhang%2C+Ning+Ding%2C+Yuxian+Gu%2C+Xiao+Liu%2C+Yuqi+Huo%2C+Jiezhong+Qiu%2C+Liang+Zhang%2C+Wentao+Han%2C+Minlie+Huang%2C+et+al%2E+2021%2E+Pre%2Dtrained+models%3A+Past%2C+present+and+future%2E+AI+Open%2E) 18. John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=John+Hewitt+and+Christopher+D%2E+Manning%2E+2019%2E+A+structural+probe+for+finding+syntax+in+word+representations%2E+In+North+American+Chapter+of+the+Association+for+Computational+Linguistics%3A+Human+Language+) [our card](https://md.research.elamur.ai/free/en/papers/hewitt-2019-structural-probe-finding-syntax-word/) 19. Zhengbao Jiang, Antonios Anastasopoulos, Jun Araki, Haibo Ding, and Graham Neubig. 2020a. X-factr: Multilingual factual knowledge retrieval from pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5943–5959. [find](https://scholar.google.com/scholar?q=Zhengbao+Jiang%2C+Antonios+Anastasopoulos%2C+Jun+Araki%2C+Haibo+Ding%2C+and+Graham+Neubig%2E+2020a%2E+X%2Dfactr%3A+Multilingual+factual+knowledge+retrieval+from+pre%2Dtrained+language+models%2E+In+Proceedings+of+the+2020) 20. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2021. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. CoRR, abs/2104.08786. [find](https://scholar.google.com/scholar?q=Yao+Lu%2C+Max+Bartolo%2C+Alastair+Moore%2C+Sebastian+Riedel%2C+and+Pontus+Stenetorp%2E+2021%2E+Fantastically+ordered+prompts+and+where+to+find+them%3A+Overcoming+few%2Dshot+prompt+order+sensitivity%2E+CoRR%2C+abs%2F2104%2E08) [our card](https://md.research.elamur.ai/free/en/papers/lu-2022-fantastically-ordered-prompts-where-find/) 21. Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim Rocktäschel, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. 2020. How context affects language models’ factual predictions. arXiv preprint arXiv:2005.04611. [find](https://scholar.google.com/scholar?q=Fabio+Petroni%2C+Patrick+Lewis%2C+Aleksandra+Piktus%2C+Tim+Rockt%C3%A4schel%2C+Yuxiang+Wu%2C+Alexander+H+Miller%2C+and+Sebastian+Riedel%2E+2020%2E+How+context+affects+language+models%E2%80%99+factual+predictions%2E+arXiv+preprint+a) [our card](https://md.research.elamur.ai/free/en/papers/petroni-2020-context-affects-language-models-factual/) 22. Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. 2019. Language models as knowledge bases? arXiv preprint arXiv:1909.01066. [find](https://scholar.google.com/scholar?q=Fabio+Petroni%2C+Tim+Rockt%C3%A4schel%2C+Patrick+Lewis%2C+Anton+Bakhtin%2C+Yuxiang+Wu%2C+Alexander+H+Miller%2C+and+Sebastian+Riedel%2E+2019%2E+Language+models+as+knowledge+bases%3F+arXiv+preprint+arXiv%3A1909%2E01066%2E) [our card](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/) 23. Mohammad Taher Pilehvar and José Camacho-Collados. 2018. Wic: 10, 000 example pairs for evaluating context-sensitive representations. CoRR, abs/1808.09121. [find](https://scholar.google.com/scholar?q=Mohammad+Taher+Pilehvar+and+Jos%C3%A9+Camacho%2DCollados%2E+2018%2E+Wic%3A+10%2C+000+example+pairs+for+evaluating+context%2Dsensitive+representations%2E+CoRR%2C+abs%2F1808%2E09121%2E) [our card](https://md.research.elamur.ai/free/en/papers/pilehvar-2019-wic-word-context-dataset-evaluating/) 24. Guanghui Qin and J. Eisner. 2021. Learning how to ask: Querying lms with mixtures of soft prompts. ArXiv, abs/2104.06599. [find](https://scholar.google.com/scholar?q=Guanghui+Qin+and+J%2E+Eisner%2E+2021%2E+Learning+how+to+ask%3A+Querying+lms+with+mixtures+of+soft+prompts%2E+ArXiv%2C+abs%2F2104%2E06599%2E) [our card](https://md.research.elamur.ai/free/en/papers/qin-2021-learning-ask-querying-lms-mixtures/) 25. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E) 26. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+Wei+Li%2C+and+Peter+J+Liu%2E+2019%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtext) 27. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning, pages 90–95. [find](https://scholar.google.com/scholar?q=Melissa+Roemmele%2C+Cosmin+Adrian+Bejan%2C+and+Andrew+S+Gordon%2E+2011%2E+Choice+of+plausible+alternatives%3A+An+evaluation+of+commonsense+causal+reasoning%2E+In+AAAI+Spring+Symposium%3A+Logical+Formalizations+of+C) 28. Timo Schick and Hinrich Schütze. 2020. It’s not just size that matters: Small language models are also few-shot learners. Computing Research Repository, arXiv:2009.07118. [find](https://scholar.google.com/scholar?q=Timo+Schick+and+Hinrich+Sch%C3%BCtze%2E+2020%2E+It%E2%80%99s+not+just+size+that+matters%3A+Small+language+models+are+also+few%2Dshot+learners%2E+Computing+Research+Repository%2C+arXiv%3A2009%2E07118%2E) [our card](https://md.research.elamur.ai/free/en/papers/schick-2021-it-s-not-just-size/) 29. Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980. [find](https://scholar.google.com/scholar?q=Taylor+Shin%2C+Yasaman+Razeghi%2C+Robert+L+Logan+IV%2C+Eric+Wallace%2C+and+Sameer+Singh%2E+2020%2E+Autoprompt%3A+Eliciting+knowledge+from+language+models+with+automatically+generated+prompts%2E+arXiv+preprint+arXiv%3A2) [our card](https://md.research.elamur.ai/free/en/papers/shin-2020-autoprompt-eliciting-knowledge-language-models/) 30. Jesse Vig. 2019. A multiscale visualization of attention in the transformer model. arXiv preprint arXiv:1906.05714. [find](https://scholar.google.com/scholar?q=Jesse+Vig%2E+2019%2E+A+multiscale+visualization+of+attention+in+the+transformer+model%2E+arXiv+preprint+arXiv%3A1906%2E05714%2E) [our card](https://md.research.elamur.ai/free/en/papers/vig-2019-multiscale-visualization-attention-transformer-model/) 31. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019a. Superglue: A stickier benchmark for general-purpose language understanding systems. arXiv preprint arXiv:1905.00537. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Yada+Pruksachatkun%2C+Nikita+Nangia%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+R+Bowman%2E+2019a%2E+Superglue%3A+A+stickier+benchmark+for+general%2Dpurpose+language+understan) [our card](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/) 32. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems. In NeurIPS 2019, pages 3261–3275. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Yada+Pruksachatkun%2C+Nikita+Nangia%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+R%2E+Bowman%2E+2019b%2E+SuperGLUE%3A+A+Stickier+Benchmark+for+General%2DPurpose+Language+Understa) [our card](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/) 33. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. ArXiv, abs/1804.07461. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+R%2E+Bowman%2E+2018%2E+Glue%3A+A+multi%2Dtask+benchmark+and+analysis+platform+for+natural+language+understanding%2E+ArXiv%2C+abs%2F1804%2E07) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/) 34. Chenguang Wang, Xiao Liu, and Dawn Song. 2020. Language models are open knowledge graphs. arXiv preprint arXiv:2010.11967. — [arXiv:2010.11967](https://arxiv.org/abs/2010.11967) [find](https://scholar.google.com/scholar?q=Chenguang+Wang%2C+Xiao+Liu%2C+and+Dawn+Song%2E+2020%2E+Language+models+are+open+knowledge+graphs%2E+arXiv+preprint+arXiv%3A2010%2E11967%2E+%E2%80%94+%5BarXiv%3A2010%2E11967%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2010%2E11967%29) 35. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237. [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Zihang+Dai%2C+Yiming+Yang%2C+Jaime+Carbonell%2C+Ruslan+Salakhutdinov%2C+and+Quoc+V+Le%2E+2019%2E+Xlnet%3A+Generalized+autoregressive+pretraining+for+language+understanding%2E+arXiv+preprint+arXiv%3A1906%2E08) [our card](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/) 36. Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. Record: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint arXiv:1810.12885. [find](https://scholar.google.com/scholar?q=Sheng+Zhang%2C+Xiaodong+Liu%2C+Jingjing+Liu%2C+Jianfeng+Gao%2C+Kevin+Duh%2C+and+Benjamin+Van+Durme%2E+2018%2E+Record%3A+Bridging+the+gap+between+human+and+machine+commonsense+reading+comprehension%2E+arXiv+preprint+arX) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2018-record-bridging-gap-between-human/) 37. Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q. Weinberger, and Yoav Artzi. 2020. Revisiting few-sample BERT fine-tuning. CoRR, abs/2006.05987. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Felix+Wu%2C+Arzoo+Katiyar%2C+Kilian+Q%2E+Weinberger%2C+and+Yoav+Artzi%2E+2020%2E+Revisiting+few%2Dsample+BERT+fine%2Dtuning%2E+CoRR%2C+abs%2F2006%2E05987%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2020-revisiting-few-sample-bert-fine/) 38. Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021a. Calibrate before use: Improving few-shot performance of language models. CoRR, abs/2102.09690. [find](https://scholar.google.com/scholar?q=Tony+Z%2E+Zhao%2C+Eric+Wallace%2C+Shi+Feng%2C+Dan+Klein%2C+and+Sameer+Singh%2E+2021a%2E+Calibrate+before+use%3A+Improving+few%2Dshot+performance+of+language+models%2E+CoRR%2C+abs%2F2102%2E09690%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhao-2021-calibrate-before-use-improving-few/) 39. Tony Z Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021b. Calibrate before use: Improving few-shot performance of language models. arXiv preprint arXiv:2102.09690. [find](https://scholar.google.com/scholar?q=Tony+Z+Zhao%2C+Eric+Wallace%2C+Shi+Feng%2C+Dan+Klein%2C+and+Sameer+Singh%2E+2021b%2E+Calibrate+before+use%3A+Improving+few%2Dshot+performance+of+language+models%2E+arXiv+preprint+arXiv%3A2102%2E09690%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhao-2021-calibrate-before-use-improving-few/) 40. Yanan Zheng, Jing Zhou, Yujie Qian, Ming Ding, Jian Li, Ruslan Salakhutdinov, Jie Tang, Sebastian Ruder, and Zhilin Yang. 2021. Fewnlu: Benchmarking state-of-the-art methods for few-shot natural language understanding. [find](https://scholar.google.com/scholar?q=Yanan+Zheng%2C+Jing+Zhou%2C+Yujie+Qian%2C+Ming+Ding%2C+Jian+Li%2C+Ruslan+Salakhutdinov%2C+Jie+Tang%2C+Sebastian+Ruder%2C+and+Zhilin+Yang%2E+2021%2E+Fewnlu%3A+Benchmarking+state%2Dof%2Dthe%2Dart+methods+for+few%2Dshot+natural+langu) 41. Zexuan Zhong, Dan Friedman, and Danqi Chen. 2021. Factual probing is [mask]: Learning vs. learning to recall. ArXiv, abs/2104.05240. [find](https://scholar.google.com/scholar?q=Zexuan+Zhong%2C+Dan+Friedman%2C+and+Danqi+Chen%2E+2021%2E+Factual+probing+is+%5Bmask%5D%3A+Learning+vs%2E+learning+to+recall%2E+ArXiv%2C+abs%2F2104%2E05240%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhong-2021-factual-probing-is-mask-learning/)