---
title: "Language Model Behavior: A Comprehensive Survey — paper by Chang and Bergen (2024)"
description: "These studies treat models as black boxes and probe them with controlled inputs, often borrowing methods from… 154 references, cited by 45 papers here."
authors: ["Tyler A. Chang", "Benjamin K. Bergen"]
year: 2024
venue: "Computational Linguistics"
source: "https://aclanthology.org/2024.cl-1.9/"
doi: "10.1162/coli_a_00492"
licence: "CC BY-NC-ND 4.0 (Computational Linguistics, MIT Press)"
licence_url: "https://creativecommons.org/licenses/by-nc-nd/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chang-2024-language-model-behavior-comprehensive-survey/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chang-2024-language-model-behavior-comprehensive-survey/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/2/) / Language Model Behavior: A Comprehensive Survey

# Language Model Behavior: A Comprehensive Survey

**Authors:** [Tyler A. Chang](https://md.research.elamur.ai/free/en/authors/tyler-a-chang/) and [Benjamin K. Bergen](https://md.research.elamur.ai/free/en/authors/benjamin-bergen/) · **Year:** 2024 · **Venue:** [Computational Linguistics](https://md.research.elamur.ai/free/en/venues/cl/) ([Computational Linguistics 2024](https://md.research.elamur.ai/free/en/venues/cl/2024/)) · **Publisher:** [The MIT Press](https://md.research.elamur.ai/free/en/publishers/the-mit-press/) · **Licence:** [CC BY-NC-ND 4.0 (Computational Linguistics, MIT Press)](https://creativecommons.org/licenses/by-nc-nd/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.cl-1.9/) · [PDF](https://aclanthology.org/2024.cl-1.9.pdf) · [DOI 10.1162/coli\_a\_00492](https://doi.org/10.1162/coli_a_00492) · [OpenAlex](https://openalex.org/W4388691863)

**Citations:** 103 citations (OpenAlex, 2026-10-14) · [45](https://md.research.elamur.ai/free/en/papers/chang-2024-language-model-behavior-comprehensive-survey/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Transformer language models are now deployed widely, often with little or no task-specific tuning, but their outputs remain hard to predict, and the research on their behavior is scattered across hundreds of papers. Chang and Bergen consolidate this evidence by reviewing over 250 behavioral studies of English masked and autoregressive models that were not fine-tuned. These studies treat models as black boxes and probe them with controlled inputs, often borrowing methods from psycholinguistics. The review covers social bias, memorization, reasoning, and knowledge of the world, along with the linguistic areas of syntax, semantics, and pragmatics. Its central conclusion is that models have real but fragile abilities in each of these areas. Performance depends heavily on exact wording and surface cues. Larger models produce much better text, yet they still make factual and commonsense mistakes, reproduce memorized passages, and show biases. The authors suggest that many of these failures can be understood as patterns in text that the model generalizes too broadly or too narrowly. The survey serves as a reference for practitioners and for researchers in neighboring fields. Note that multilingual and non-English models, including Ukrainian, fall outside its scope.

## Key points
- Papers were collected through Semantic Scholar. A seed set of 271 papers led to more than 15K citing works, scraped on February 4, 2023. Screening by title reduced these to about 1,500, and screening by abstract to about 400 highly relevant papers.
- The scope excludes fine-tuned models, because fine-tuning can latch onto spurious correlations and erode knowledge from pre-training. It also excludes multilingual, vision-language, code, speech, knowledge-augmented, sparse, and compressed models.
- The models covered range from roughly 100M to 500B parameters, with vocabularies of about 30K–250K tokens. Examples include GPT-3/4, PaLM, OPT, and BERT. The survey also notes when instruction-tuned or RLHF-tuned GPT-3 variants such as text-davinci-003 are involved.
- The authors pose an open question: do gains from scale come from learning sturdier generalizations, or from memorizing more specific cases? They lean toward the second explanation.
- Behavioral and mechanistic analyses are related through Marr's levels of analysis. Code and key points are available at github.com/tylerachang/llm-behavior-survey.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of California San Diego](https://md.research.elamur.ai/free/en/universities/university-of-california-san-diego/) — University, San Diego, United States · 34 works · 51,167 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works

## Cite

Cite as: Tyler A. Chang and Benjamin K. Bergen. 2024. Language Model Behavior: A Comprehensive Survey. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{chang2024language,
  title = {{Language Model Behavior: A Comprehensive Survey}},
  author = {Tyler A. Chang and Benjamin K. Bergen},
  journal = {Computational Linguistics},
  year = {2024},
  doi = {10.1162/coli\_a\_00492},
  url = {https://doi.org/10.1162/coli_a_00492}
}
```

- [← previous Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Models](https://md.research.elamur.ai/free/en/papers/yu-2024-chain-note-enhancing-robustness-retrieval/)
- [next → BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages](https://md.research.elamur.ai/free/en/papers/myung-2024-blend-benchmark-llms-everyday-knowledge/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/2/)

## References

154 references, 54 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abdou, Mostafa, Vinit Ravishankar, Artur Kulmizev, and Anders Søgaard. 2022. Word order does matter and shuffled language models know it. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6907–6919. https://doi.org/10.18653/v1 /… [find](https://scholar.google.com/scholar?q=Abdou%2C+Mostafa%2C+Vinit+Ravishankar%2C+Artur+Kulmizev%2C+and+Anders+S%C3%B8gaard%2E+2022%2E+Word+order+does+matter+and+shuffled+language+models+know+it%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+f)
2. Aher, Gati, Rosa Arriaga, and Adam Kalai. 2022. Using large language models to simulate multiple humans. ArXiv, arXiv:2208.10264. [find](https://scholar.google.com/scholar?q=Aher%2C+Gati%2C+Rosa+Arriaga%2C+and+Adam+Kalai%2E+2022%2E+Using+large+language+models+to+simulate+multiple+humans%2E+ArXiv%2C+arXiv%3A2208%2E10264%2E)
3. Aina, Laura and Tal Linzen. 2021. The language model understood the prompt was ambiguous: Probing syntactic uncertainty through generation. In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 42–57. https://doi.org/10.18653/v1 /2021.blackbox… [find](https://scholar.google.com/scholar?q=Aina%2C+Laura+and+Tal+Linzen%2E+2021%2E+The+language+model+understood+the+prompt+was+ambiguous%3A+Probing+syntactic+uncertainty+through+generation%2E+In+Proceedings+of+the+Fourth+BlackboxNLP+Workshop+on+Analyzi)
4. Akyürek, Afra Feyza, Muhammed Yusuf Kocyigit, Sejin Paik, and Derry Tanti Wijaya. 2022. Challenges in measuring bias via open-ended language generation. In Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pages 76–76. https://doi.org/10.18653/v1/2022.gebnlp-1… [↗](https://doi.org/10.18653/v1/2022.gebnlp-1…)
5. Aribandi, Vamsi, Yi Tay, and Donald Metzler. 2021. How reliable are model diagnostics? In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1778–1785. https://doi.org/10.18653/v1/2021.findings -acl.155 Armeni, Kristijan, Christopher Honey, and Tal Linzen. 2022. Chara… [↗](https://doi.org/10.18653/v1/2021.findings)
6. Bang, Yejin, Nayeon Lee, Etsuko Ishii, Andrea Madotto, and Pascale Fung. 2021. Assessing political prudence of open-domain chatbots. In Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 548–555. https://doi.org/10.18653/v1/2021.sigdial-1.57 Bartl,… [↗](https://doi.org/10.18653/v1/2021.sigdial-1.57)
7. Bechtel, William and Oron Shagrir. 2015. The non-redundant contributions of Marr’s three levels of analysis for explaining information-processing mechanisms. [find](https://scholar.google.com/scholar?q=Bechtel%2C+William+and+Oron+Shagrir%2E+2015%2E+The+non%2Dredundant+contributions+of+Marr%E2%80%99s+three+levels+of+analysis+for+explaining+information%2Dprocessing+mechanisms%2E)
8. Topics in Cognitive Science, 7(2):312–322. https://doi.org/10.1111/tops.12141, PubMed: 25900887 Belinkov, Yonatan. 2022. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207–219. https://doi.org/10.1162/coli_a_00422 [↗](https://doi.org/10.1111/tops.12141) [our card](https://md.research.elamur.ai/free/en/papers/belinkov-2022-probing-classifiers-promises-shortcomings-advances/)
9. Beloucif, Meriem and Chris Biemann. 2021. Probing pre-trained language models for semantic attributes and their values. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2554–2559. https://doi.org/10.18653/v1/2021 .findings-emnlp.218 Bender, Emily M., Timnit Gebru, Ange… [↗](https://doi.org/10.18653/v1/2021)
10. Binz, Marcel and Eric Schulz. 2023. Using cognitive psychology to understand GPT-3. Proceedings of the National Academy of Sciences of the United States of America, 120(6):e2218523120. https://doi.org/10.1073/pnas.2218523120, PubMed: 36730192 Blodgett, Su Lin, Solon Barocas, Hal Daumé III, and Han… [↗](https://doi.org/10.1073/pnas.2218523120)
11. Borgeaud, Sebastian, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin… [find](https://scholar.google.com/scholar?q=Borgeaud%2C+Sebastian%2C+Arthur+Mensch%2C+Jordan+Hoffmann%2C+Trevor+Cai%2C+Eliza+Rutherford%2C+Katie+Millican%2C+George+Bm+Van+Den+Driessche%2C+Jean%2DBaptiste+Lespiau%2C+Bogdan+Damoc%2C+Aidan+Clark%2C+Diego+De+Las+Casas%2C+Au)
12. Bowman, Samuel. 2022. The dangers of underclaiming: Reasons for caution when reporting how NLP systems fail. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7484–7499. https://doi.org/10.18653/v1/2022.acl-long.516 Brandl, Ste… [↗](https://doi.org/10.18653/v1/2022.acl-long.516)
13. Broyde, Joshua and Claire Palmer. 2021. Build a medical sentence matching application using BERT and Amazon SageMaker. [find](https://scholar.google.com/scholar?q=Broyde%2C+Joshua+and+Claire+Palmer%2E+2021%2E+Build+a+medical+sentence+matching+application+using+BERT+and+Amazon+SageMaker%2E)
14. AWS Machine Learning Blog. Cai, Xingyu, Jiaji Huang, Yu-Lan Bian, and Kenneth Ward Church. 2021. Isotropy in the contextual embedding space: Clusters and manifolds. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=AWS+Machine+Learning+Blog%2E+Cai%2C+Xingyu%2C+Jiaji+Huang%2C+Yu%2DLan+Bian%2C+and+Kenneth+Ward+Church%2E+2021%2E+Isotropy+in+the+contextual+embedding+space%3A+Clusters+and+manifolds%2E+In+International+Conference+on+Lear)
15. Cao, Boxi, Hongyu Lin, Xianpei Han, Fangchao Liu, and Le Sun. 2022. Can prompt probe pretrained language models? Understanding the invisible risks from a causal view. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5796–5808.… [find](https://scholar.google.com/scholar?q=Cao%2C+Boxi%2C+Hongyu+Lin%2C+Xianpei+Han%2C+Fangchao+Liu%2C+and+Le+Sun%2E+2022%2E+Can+prompt+probe+pretrained+language+models%3F+Understanding+the+invisible+risks+from+a+causal+view%2E+In+Proceedings+of+the+60th+Annual)
16. Carlini, Nicholas, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel. 2021. Extracting training data from large language models. In USENIX Security Symposium, pages 2633–2650. [our card](https://md.research.elamur.ai/free/en/papers/carlini-2020-extracting-training-data-large-language/)
17. Syntax: A Generative Introduction. Blackwell. Caron, Graham and Shashank Srivastava. 2022. Identifying and manipulating the personality traits of language models. ArXiv, arXiv:2212.10276. https://doi.org/10.18653/v1/2023.findings-emnlp.156 Chang, Tyler, Zhuowen Tu, and Benjamin Bergen. 2022. The ge… [↗](https://doi.org/10.18653/v1/2023.findings-emnlp.156)
18. Chaves, Rui P. and Stephanie N. Richter. 2021. Look at that! BERT can be easily distracted from paying attention to morphosyntax. In Proceedings of the Society for Computation in Linguistics 2021, pages 28–38. [find](https://scholar.google.com/scholar?q=Chaves%2C+Rui+P%2E+and+Stephanie+N%2E+Richter%2E+2021%2E+Look+at+that%21+BERT+can+be+easily+distracted+from+paying+attention+to+morphosyntax%2E+In+Proceedings+of+the+Society+for+Computation+in+Linguistics+2021%2C+pag)
19. Chen, Kaiping, Anqi Shao, Jirayu Burapacheep, and Yixuan Li. 2022. A critical appraisal of equity in conversational AI: Evidence from auditing GPT-3’s dialogues with different publics on climate change and Black Lives Matter. ArXiv, arXiv:2209.13627. — [arXiv:2209.13627](https://arxiv.org/abs/2209.13627) [find](https://scholar.google.com/scholar?q=Chen%2C+Kaiping%2C+Anqi+Shao%2C+Jirayu+Burapacheep%2C+and+Yixuan+Li%2E+2022%2E+A+critical+appraisal+of+equity+in+conversational+AI%3A+Evidence+from+auditing+GPT%2D3%E2%80%99s+dialogues+with+different+publics+on+climate+chang)
20. Chen, Mark, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alet… [our card](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
21. Chiang, Cheng Han, Sung-Feng Huang, and Hung-yi Lee. 2020. Pretrained language model embryology: The birth of ALBERT. [find](https://scholar.google.com/scholar?q=Chiang%2C+Cheng+Han%2C+Sung%2DFeng+Huang%2C+and+Hung%2Dyi+Lee%2E+2020%2E+Pretrained+language+model+embryology%3A+The+birth+of+ALBERT%2E)
22. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6813–6828. https://doi.org/10.18653/v1/2020.emnlp-main.553 Chiang, Ting Rui and Yun-Nung Chen. 2021. Relating neural text degeneration to exposure bias. In Proceedings of the Fourth BlackboxNLP… [↗](https://doi.org/10.18653/v1/2020.emnlp-main.553) [our card](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
23. Chuang, Chengyu and Yi Yang. 2022. Buy Tesla, sell Ford: Assessing implicit stock market preference in pre-trained language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 100–105. [https://doi.org](https://doi.org) /10.18653/v1/2022.ac… [↗](https://doi.org)
24. Cong, Yan. 2022. Psycholinguistic diagnosis of language models’ commonsense reasoning. In Proceedings of the First Workshop on Commonsense Representation and Reasoning (CSRR 2022), pages 17–22. https://doi.org/10.18653/v1/2022.csrr-1.3 Czarnowska, Paula, Yogarshi Vyas, and Kashif Shah. 2021. Quanti… [↗](https://doi.org/10.18653/v1/2022.csrr-1.3)
25. Czinczoll, Tamara, Helen Yannakoudakis, Pushkar Mishra, and Ekaterina Shutova. 2022. Scientific and creative analogies in pretrained language models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2094–2100. https://doi.org/10.18653/v1/2022 .findings-emnlp.153 Dabrow… [↗](https://doi.org/10.18653/v1/2022) [our card](https://md.research.elamur.ai/free/en/papers/dai-2023-why-can-gpt-learn-context/)
26. Dar, Guy, Mor Geva, Ankit Gupta, and Jonathan Berant. 2022. Analyzing Transformers in embedding space. ArXiv, arXiv:2209.02535. https://doi.org/10.18653/v1/2023.acl-long.893 Dasgupta, Ishita, Andrew Lampinen, Stephanie Chan, Antonia Creswell, Dharshan Kumaran, James McClelland, and Felix Hill. 2022… [↗](https://doi.org/10.18653/v1/2023.acl-long.893) [our card](https://md.research.elamur.ai/free/en/papers/dar-2023-analyzing-transformers-embedding-space/)
27. Davis, Forrest and Marten van Schijndel. 2020. Discourse structure interacts with reference but not syntax in neural language models. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 396–407. https://doi.org/10.18653/v1/2020.conll-1.32 Davison, Joe, Joshua Fel… [↗](https://doi.org/10.18653/v1/2020.conll-1.32)
28. De Bruyn, Maxime, Ehsan Lotfi, Jeska Buhmann, and Walter Daelemans. 2022. Is it smaller than a tennis ball? Language models play the game of twenty questions. In Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 80–90. https://doi.org/10.18653… [find](https://scholar.google.com/scholar?q=De+Bruyn%2C+Maxime%2C+Ehsan+Lotfi%2C+Jeska+Buhmann%2C+and+Walter+Daelemans%2E+2022%2E+Is+it+smaller+than+a+tennis+ball%3F+Language+models+play+the+game+of+twenty+questions%2E+In+Proceedings+of+the+Fifth+BlackboxNLP+W)
29. Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies… [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)
30. Dhamala, Jwala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. 2021. BOLD: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the ACM Conference on Fairness, Accountability, and Transparency, pages 862–872. ht… [find](https://scholar.google.com/scholar?q=Dhamala%2C+Jwala%2C+Tony+Sun%2C+Varun+Kumar%2C+Satyapriya+Krishna%2C+Yada+Pruksachatkun%2C+Kai%2DWei+Chang%2C+and+Rahul+Gupta%2E+2021%2E+BOLD%3A+Dataset+and+metrics+for+measuring+biases+in+open%2Dended+language+generation%2E+I)
31. Du, Yifan, Zikang Liu, Junyi Li, and Wayne Xin Zhao. 2022b. A survey of vision-language pre-trained models. In Proceedings of the International Joint Conference on Artificial Intelligence, pages 5436–5443. https://doi.org/10.24963/ijcai.2022/762 Dufter, Philipp, Martin Schmitt, and Hinrich Schütze… [↗](https://doi.org/10.24963/ijcai.2022/762)
32. Dugan, Liam, Daphne Ippolito, Arun Kirubarajan, Sherry Shi, and Chris Callison-Burch. 2023. Real or fake text?: Investigating human ability to detect boundaries between human-written and machine-generated text. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 12763–12771. http… [find](https://scholar.google.com/scholar?q=Dugan%2C+Liam%2C+Daphne+Ippolito%2C+Arun+Kirubarajan%2C+Sherry+Shi%2C+and+Chris+Callison%2DBurch%2E+2023%2E+Real+or+fake+text%3F%3A+Investigating+human+ability+to+detect+boundaries+between+human%2Dwritten+and+machine%2Dgener)
33. Fedus, William, Barret Zoph, and Noam Shazeer. 2022. Switch Transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23:1–39. [find](https://scholar.google.com/scholar?q=Fedus%2C+William%2C+Barret+Zoph%2C+and+Noam+Shazeer%2E+2022%2E+Switch+Transformers%3A+Scaling+to+trillion+parameter+models+with+simple+and+efficient+sparsity%2E+Journal+of+Machine+Learning+Research%2C+23%3A1%E2%80%9339%2E)
34. Felkner, Virginia K., Ho-Chun Herbert Chang, Eugene Jang, and Jonathan May. 2022. Towards WinoQueer: Developing a benchmark for anti-queer bias in large language models. In Queer in AI Workshop. https://doi.org/10.18653/v1/2023.acl-long.507 Finlayson, Matthew, Aaron Mueller, Sebastian Gehrmann, Stu… [↗](https://doi.org/10.18653/v1/2023.acl-long.507)
35. Freitag, Markus and Yaser Al-Onaizan. 2017. Beam search strategies for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 56–60. https://doi.org/10 .18653/v1/W17-3207 Fricke, Suzanne. 2018. Semantic Scholar. Journal of the Medical Library Associatio… [our card](https://md.research.elamur.ai/free/en/papers/freitag-2017-beam-search-strategies-neural-machine/)
36. Ganguli, Deep, Danny Hernandez, Liane Lovitt, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova Dassarma, Dawn Drain, Nelson Elhage, Sheer El Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan, Scott Johnston, Andy Jones, Nicholas Joseph, Jackson Kernian, Shauna Kravec, Ben Mann, Neel Nan… [find](https://scholar.google.com/scholar?q=Ganguli%2C+Deep%2C+Danny+Hernandez%2C+Liane+Lovitt%2C+Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+Tom+Conerly%2C+Nova+Dassarma%2C+Dawn+Drain%2C+Nelson+Elhage%2C+Sheer+El+Showk%2C+Stanislav+Fort%2C+Zac+Hatfield%2DDodds%2C+Tom+Henig)
37. Gauthier, Jon, Jennifer Hu, Ethan Wilcox, Peng Qian, and Roger Levy. 2020. SyntaxGym: An online platform for targeted evaluation of language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 70–76. https://doi.org/10.1865… [our card](https://md.research.elamur.ai/free/en/papers/gauthier-2020-syntaxgym-online-platform-targeted-evaluation/)
38. Geeraerts, Dirk. 2017. Lexical semantics. Oxford Research Encyclopedia of Linguistics. https://doi.org/10.1093/acrefore/9780199384655.013.29 Gehman, Samuel, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxicityPrompts: Evaluating neural toxic degeneration in language mod… [↗](https://doi.org/10.1093/acrefore/9780199384655.013.29) [our card](https://md.research.elamur.ai/free/en/papers/goldberg-2019-assessing-bert-s-syntactic-abilities/)
39. Grice, H. P. 1975. Logic and conversation. Syntax and Semantics: Vol. 3: Speech Acts, pages 41–58. https://doi.org/10.1163/9789004368811 003 Griciūtė, Bernadeta, Marc Tanti, and Lucia Donatelli. 2022. On the cusp of comprehensibility: Can language models distinguish between metaphors and nonsense… [↗](https://doi.org/10.1163/9789004368811)
40. Hagendorff, Thilo, Sarah Fabi, and Michal Kosinski. 2022. Machine intuition: Uncovering human-like intuitive decision-making in GPT-3.5. ArXiv, arXiv:2212.05206. https://doi.org/10.1038/s43588-023-00527-x Hahn, Michael. 2020. Theoretical limitations of self-attention in neural sequence models. Tran… [↗](https://doi.org/10.1038/s43588-023-00527-x)
41. Han, Simeng, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Luke Benson, Lucy Sun, Ekaterina Zubova, Yujie Qiao, Matthew Burtell, David Peng, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Shafiq Joty, Alexander R. Fabbri, Wo… [find](https://scholar.google.com/scholar?q=Han%2C+Simeng%2C+Hailey+Schoelkopf%2C+Yilun+Zhao%2C+Zhenting+Qi%2C+Martin+Riddell%2C+Luke+Benson%2C+Lucy+Sun%2C+Ekaterina+Zubova%2C+Yujie+Qiao%2C+Matthew+Burtell%2C+David+Peng%2C+Jonathan+Fan%2C+Yixin+Liu%2C+Brian+Wong%2C+Malcolm+)
42. Hendrycks, Dan, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021b. Measuring mathematical problem solving with the MATH dataset. In Advances in Neural Information Processing Systems Datasets and Benchmarks Track. [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/)
43. Hernandez, Danny, Tom B. Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, Scott Johnston, Benjamin Mann, Christopher Olah, Catherine Olsson, Dario Amodei, Nicholas Joseph, Jared Kaplan, and Sam McCandlish. 2022. Scaling law… [our card](https://md.research.elamur.ai/free/en/papers/hernandez-2022-scaling-laws-interpretability-learning-repeated/)
44. Hershcovich, Daniel, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders Søgaard. 2022. Challenges and strategies in cross-cultural… [our card](https://md.research.elamur.ai/free/en/papers/hershcovich-2022-challenges-strategies-cross-cultural-nlp/)
45. Holtzman, Ari, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Holtzman%2C+Ari%2C+Jan+Buys%2C+Li+Du%2C+Maxwell+Forbes%2C+and+Yejin+Choi%2E+2020%2E+The+curious+case+of+neural+text+degeneration%2E+In+International+Conference+on+Learning+Representations%2E)
46. Hossain, Md Mosharaf, Dhivya Chinnappa, and Eduardo Blanco. 2022. An analysis of negation in natural language understanding corpora. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 716–723. [https://doi.org](https://doi.org) /10.18653/v1/2022.a… [↗](https://doi.org)
47. Hu, Jennifer, Sammy Floyd, Olessia Jouravlev, Evelina Fedorenko, and Edward Gibson. 2022. A fine-grained comparison of pragmatic language understanding in humans and language models. ArXiv, arXiv:2212.06801. https://doi.org/10.18653/v1/2023.acl-long.230 Hu, Jennifer, Jon Gauthier, Peng Qian, Ethan… [↗](https://doi.org/10.18653/v1/2023.acl-long.230) [our card](https://md.research.elamur.ai/free/en/papers/hu-2023-fine-grained-comparison-pragmatic-language/)
48. Huynh, Hien, Tomas O. Lentz, and Emiel van Miltenburg. 2022. Implicit causality in GPT-2: A case study. ArXiv, arXiv:2212.04348. [find](https://scholar.google.com/scholar?q=Huynh%2C+Hien%2C+Tomas+O%2E+Lentz%2C+and+Emiel+van+Miltenburg%2E+2022%2E+Implicit+causality+in+GPT%2D2%3A+A+case+study%2E+ArXiv%2C+arXiv%3A2212%2E04348%2E)
49. Ippolito, Daphne, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. 2020. Automatic detection of generated text is easiest when humans are fooled. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1808–1822. [https://doi.org](https://doi.org) /10.18653/v1/2020.acl… [↗](https://doi.org) [our card](https://md.research.elamur.ai/free/en/papers/ippolito-2019-automatic-detection-generated-text-easiest/)
50. Jain, Sarthak and Byron C. Wallace. 2019. Attention is not Explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3543–3556. [our card](https://md.research.elamur.ai/free/en/papers/jain-2019-attention-is-not-explanation/)
51. Jakesch, Maurice, Jeffrey T. Hancock, and Mor Naaman. 2023. Human heuristics for AI-generated language are flawed. Proceedings of the National Academy of Sciences, 120(11):e2208839120. https://doi.org/10.1073/pnas.2208839120 PubMed: 36881628 Jang, Joel, Seonghyeon Ye, and Minjoon Seo. 2022. Can lar… [↗](https://doi.org/10.1073/pnas.2208839120)
52. Jawahar, Ganesh, Muhammad Abdul-Mageed, and Laks Lakshmanan, V. S. 2020. Automatic detection of machine generated text: A critical survey. In Proceedings of the 28th International Conference on Computational Linguistics, pages 2296–2309. https://doi.org/10.18653/v1/2020.coling-main.208 Jiang, Cheng… [↗](https://doi.org/10.18653/v1/2020.coling-main.208) [our card](https://md.research.elamur.ai/free/en/papers/jawahar-2020-automatic-detection-machine-generated-text/)
53. Jiang, Guangyuan, Manjie Xu, Song-Chun Zhu, Wenjuan Han, Chi Zhang, and Yixin Zhu. 2022. MPI: Evaluating and inducing personality in pre-trained language models. ArXiv, arXiv:2206.07550. [find](https://scholar.google.com/scholar?q=Jiang%2C+Guangyuan%2C+Manjie+Xu%2C+Song%2DChun+Zhu%2C+Wenjuan+Han%2C+Chi+Zhang%2C+and+Yixin+Zhu%2E+2022%2E+MPI%3A+Evaluating+and+inducing+personality+in+pre%2Dtrained+language+models%2E+ArXiv%2C+arXiv%3A2206%2E07550%2E)
54. Jiang, Tianyu and Ellen Riloff. 2021. Learning prototypical functions for physical artifacts. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6941–6951… [find](https://scholar.google.com/scholar?q=Jiang%2C+Tianyu+and+Ellen+Riloff%2E+2021%2E+Learning+prototypical+functions+for+physical+artifacts%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+the+11th+In)
55. Jin, Zijia, Xingyu Zhang, Mo Yu, and Lifu Huang. 2022b. Probing script knowledge from pre-trained models. In Proceedings of the Workshop on Unimodal and Multimodal Induction of Linguistic Structures (UM-IoS), pages 87–93. https://doi.org/10.18653/v1/2022.umios-1.10 Johnson, Rebecca L, Giada Pistill… [↗](https://doi.org/10.18653/v1/2022.umios-1.10)
56. Johnson, Steven. 2022. A.I. is mastering language. Should we trust what it says? The New York Times. [find](https://scholar.google.com/scholar?q=Johnson%2C+Steven%2E+2022%2E+A%2EI%2E+is+mastering+language%2E+Should+we+trust+what+it+says%3F+The+New+York+Times%2E)
57. Jones, Cameron and Benjamin Bergen. 2021. The role of physical inference in pronoun resolution. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 43, pages 2876–2882. [find](https://scholar.google.com/scholar?q=Jones%2C+Cameron+and+Benjamin+Bergen%2E+2021%2E+The+role+of+physical+inference+in+pronoun+resolution%2E+In+Proceedings+of+the+Annual+Meeting+of+the+Cognitive+Science+Society%2C+volume+43%2C+pages+2876%E2%80%932882%2E)
58. Jones, Cameron R., Tyler A. Chang, Seana Coulson, James Michaelov, Sean Trott, and Benjamin Bergen. 2022. Distributional semantics still can’t account for affordances. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 44, pages 482–489. [find](https://scholar.google.com/scholar?q=Jones%2C+Cameron+R%2E%2C+Tyler+A%2E+Chang%2C+Seana+Coulson%2C+James+Michaelov%2C+Sean+Trott%2C+and+Benjamin+Bergen%2E+2022%2E+Distributional+semantics+still+can%E2%80%99t+account+for+affordances%2E+In+Proceedings+of+the+Annual+Mee)
59. Joshi, Pratik, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293. [https://doi.org](https://doi.org) /10.18653/v… [↗](https://doi.org) [our card](https://md.research.elamur.ai/free/en/papers/joshi-2020-state-fate-linguistic-diversity-inclusion/)
60. Kadavath, Saurav, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zachary Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hern… [find](https://scholar.google.com/scholar?q=Kadavath%2C+Saurav%2C+Tom+Conerly%2C+Amanda+Askell%2C+Tom+Henighan%2C+Dawn+Drain%2C+Ethan+Perez%2C+Nicholas+Schiefer%2C+Zachary+Dodds%2C+Nova+DasSarma%2C+Eli+Tran%2DJohnson%2C+Scott+Johnston%2C+Sheer+El%2DShowk%2C+Andy+Jones%2C+Nels)
61. Kalo, Jan Christoph and Leandra Fichtel. 2022. KAMEL: Knowledge analysis with multitoken entities in language models. In 4th Conference on Automated Knowledge Base Construction. [find](https://scholar.google.com/scholar?q=Kalo%2C+Jan+Christoph+and+Leandra+Fichtel%2E+2022%2E+KAMEL%3A+Knowledge+analysis+with+multitoken+entities+in+language+models%2E+In+4th+Conference+on+Automated+Knowledge+Base+Construction%2E)
62. Kandpal, Nikhil, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2022. Large language models struggle to learn long-tail knowledge. ArXiv, arXiv:2211.08411. [our card](https://md.research.elamur.ai/free/en/papers/kandpal-2022-large-language-models-struggle-learn/)
63. Kandpal, Nikhil, Eric Wallace, and Colin Raffel. 2022. Deduplicating training data mitigates privacy risks in language models. In International Conference on Machine Learning, pages 10697–10707. [our card](https://md.research.elamur.ai/free/en/papers/kandpal-2022-deduplicating-training-data-mitigates-privacy/)
64. Kaplan, Jared, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei. 2020. Scaling laws for neural language models. ArXiv, arXiv:2001.08361. [our card](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/)
65. Karpas, Ehud, Omri Abend, Yonatan Belinkov, Barak Lenz, Opher Lieber, Nir Ratner, Yoav Shoham, Hofit Bata, Yoav Levine, Kevin Leyton-Brown, Dor Muhlgay, Noam Rozen, Erez Schwartz, Gal Shachaf, Shai Shalev-Shwartz, Amnon Shashua, and Moshe Tenenholtz. 2022. MRKL Systems: A modular, neuro-symbolic arc… [our card](https://md.research.elamur.ai/free/en/papers/kassner-2020-are-pretrained-language-models-symbolic/)
66. Kavumba, Pride, Ryo Takahashi, and Yusuke Oda. 2022. Are prompt-based models clueless? In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2333–2352. https://doi.org/10.18653/v1/2022.acl-long.166 Kementchedjhieva, Yova, Mark Ande… [↗](https://doi.org/10.18653/v1/2022.acl-long.166)
67. Kharitonov, Eugene, Marco Baroni, and Dieuwke Hupkes. 2021. How BPE affects memorization in Transformers. ArXiv, arXiv:2110.02782. — [arXiv:2110.02782](https://arxiv.org/abs/2110.02782) [find](https://scholar.google.com/scholar?q=Kharitonov%2C+Eugene%2C+Marco+Baroni%2C+and+Dieuwke+Hupkes%2E+2021%2E+How+BPE+affects+memorization+in+Transformers%2E+ArXiv%2C+arXiv%3A2110%2E02782%2E+%E2%80%94+%5BarXiv%3A2110%2E02782%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2110%2E02782%29)
68. Kim, Sanghee J., Lang Yu, and Allyson Ettinger. 2022. “no, they did not”: Dialogue response dynamics in pre-trained language models. In Proceedings of the 29th International Conference on Computational Linguistics, pages 863–874. [find](https://scholar.google.com/scholar?q=Kim%2C+Sanghee+J%2E%2C+Lang+Yu%2C+and+Allyson+Ettinger%2E+2022%2E+%E2%80%9Cno%2C+they+did+not%E2%80%9D%3A+Dialogue+response+dynamics+in+pre%2Dtrained+language+models%2E+In+Proceedings+of+the+29th+International+Conference+on+Computationa)
69. Kirchenbauer, John, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. 2023. A watermark for large language models. ArXiv, arXiv:2301.10226. [our card](https://md.research.elamur.ai/free/en/papers/kirchenbauer-2023-watermark-large-language-models/)
70. Kirk, Hannah Rose, Yennie Jun, Filippo Volpin, Haider Iqbal, Elias Benussi, Frederic Dreyer, Aleksandar Shtedritski, and Yuki Asano. 2021. Bias out-of-the-box: An empirical analysis of intersectional occupational biases in popular generative language models. In Advances in Neural Information Process… [find](https://scholar.google.com/scholar?q=Kirk%2C+Hannah+Rose%2C+Yennie+Jun%2C+Filippo+Volpin%2C+Haider+Iqbal%2C+Elias+Benussi%2C+Frederic+Dreyer%2C+Aleksandar+Shtedritski%2C+and+Yuki+Asano%2E+2021%2E+Bias+out%2Dof%2Dthe%2Dbox%3A+An+empirical+analysis+of+intersectional+)
71. Ko, Wei Jen and Junyi Jessy Li. 2020. Assessing discourse relations in language generation from GPT-2. In Proceedings of the 13th International Conference on Natural Language Generation, pages 52–59. https://doi.org/10.18653/v1/2020.inlg-1.8 Kojima, Takeshi, Shixiang Shane Gu, Machel Reid, Yutaka M… [↗](https://doi.org/10.18653/v1/2020.inlg-1.8) [our card](https://md.research.elamur.ai/free/en/papers/ko-2020-assessing-discourse-relations-language-generation/)
72. Kovaleva, Olga, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019. Revealing the dark secrets of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4365–43… [our card](https://md.research.elamur.ai/free/en/papers/kovaleva-2019-revealing-dark-secrets-bert/)
73. The Keyword: Google Blog. Kudo, Taku. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66–75. https://doi.org/10.18653/… [our card](https://md.research.elamur.ai/free/en/papers/kurita-2019-measuring-bias-contextualized-word-representations/)
74. Kwon, Sunjae, Cheongwoong Kang, Jiyeon Han, and Jaesik Choi. 2019. Why do masked neural language models still need common sense knowledge? ArXiv, arXiv:1911.03024. — [arXiv:1911.03024](https://arxiv.org/abs/1911.03024) [find](https://scholar.google.com/scholar?q=Kwon%2C+Sunjae%2C+Cheongwoong+Kang%2C+Jiyeon+Han%2C+and+Jaesik+Choi%2E+2019%2E+Why+do+masked+neural+language+models+still+need+common+sense+knowledge%3F+ArXiv%2C+arXiv%3A1911%2E03024%2E+%E2%80%94+%5BarXiv%3A1911%2E03024%5D%28https%3A%2F%2Farxiv%2Eo)
75. Lakhotia, Kushal, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, and Emmanuel Dupoux. 2021. On generative spoken language modeling from raw audio. Transactions of the Association for Computational Linguistics,… [our card](https://md.research.elamur.ai/free/en/papers/lakhotia-2021-generative-spoken-language-modeling-raw/)
76. Lal, Yash Kumar, Niket Tandon, Tanvi Aggarwal, Horace Liu, Nathanael Chambers, Raymond Mooney, and Niranjan Balasubramanian. 2022. Using commonsense knowledge to answer why-questions. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1204–1219. https://… [find](https://scholar.google.com/scholar?q=Lal%2C+Yash+Kumar%2C+Niket+Tandon%2C+Tanvi+Aggarwal%2C+Horace+Liu%2C+Nathanael+Chambers%2C+Raymond+Mooney%2C+and+Niranjan+Balasubramanian%2E+2022%2E+Using+commonsense+knowledge+to+answer+why%2Dquestions%2E+In+Proceedings+o)
77. Lampinen, Andrew, Ishita Dasgupta, Stephanie Chan, Kory Mathewson, Mh Tessler, Antonia Creswell, James McClelland, Jane Wang, and Felix Hill. 2022. Can language models learn from explanations in context? In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 537–563. https:/… [our card](https://md.research.elamur.ai/free/en/papers/lampinen-2022-can-language-models-learn-explanations/)
78. Lee, Angie. 2023. What are large language models used for? NVIDIA Blog. [find](https://scholar.google.com/scholar?q=Lee%2C+Angie%2E+2023%2E+What+are+large+language+models+used+for%3F+NVIDIA+Blog%2E)
79. Lee, Jooyoung, Thai Le, Jinghui Chen, and Dongwon Lee. 2023. Do language models plagiarize? In The ACM Web Conference, pages 3637–3647. https://doi.org/10.1145/3543507.3583199 Lee, Katherine, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 20… [↗](https://doi.org/10.1145/3543507.3583199)
80. Lees, Alyssa, Vinh Q. Tran, Yi Tay, Jeffrey Sorensen, Jai Gupta, Donald Metzler, and Lucy Vasserman. 2022. A new generation of perspective API: Efficient multilingual character-level Transformers. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3197–320… [find](https://scholar.google.com/scholar?q=Lees%2C+Alyssa%2C+Vinh+Q%2E+Tran%2C+Yi+Tay%2C+Jeffrey+Sorensen%2C+Jai+Gupta%2C+Donald+Metzler%2C+and+Lucy+Vasserman%2E+2022%2E+A+new+generation+of+perspective+API%3A+Efficient+multilingual+character%2Dlevel+Transformers%2E+In+)
81. Lehman, Eric, Sarthak Jain, Karl Pichotta, Yoav Goldberg, and Byron Wallace. 2021. Does BERT pretrained on clinical notes reveal sensitive data? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 94… [find](https://scholar.google.com/scholar?q=Lehman%2C+Eric%2C+Sarthak+Jain%2C+Karl+Pichotta%2C+Yoav+Goldberg%2C+and+Byron+Wallace%2E+2021%2E+Does+BERT+pretrained+on+clinical+notes+reveal+sensitive+data%3F+In+Proceedings+of+the+2021+Conference+of+the+North+Amer)
82. Li, Junyi, Tianyi Tang, Wayne Xin Zhao, and Ji-Rong Wen. 2021. Pretrained language models for text generation: A survey. In International Joint Conference on Artificial Intelligence, pages 4492–4499. https://doi.org/10.24963/ijcai.2021/612 Li, Xiang Lorraine, Adhiguna Kuncoro, Jordan Hoffmann, Cypr… [↗](https://doi.org/10.24963/ijcai.2021/612)
83. Lieber, Opher, Or Sharir, Barak Lenz, and Yoav Shoham. 2021. Jurassic-1: Technical details and evaluation. White Paper. AI21 Labs. [find](https://scholar.google.com/scholar?q=Lieber%2C+Opher%2C+Or+Sharir%2C+Barak+Lenz%2C+and+Yoav+Shoham%2E+2021%2E+Jurassic%2D1%3A+Technical+details+and+evaluation%2E+White+Paper%2E+AI21+Labs%2E)
84. Lin, Bill Yuchen, Seyeon Lee, Rahul Khanna, and Xiang Ren. 2020. Birds have four legs?! NumerSense: Probing numerical commonsense knowledge of pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6862–6868. [https://doi.…](https://doi.…) [↗](https://doi.…) [our card](https://md.research.elamur.ai/free/en/papers/lin-2020-birds-have-four-legs-numersense/)
85. Liu, Ruibo, Chenyan Jia, Jason Wei, Guangxuan Xu, and Soroush Vosoughi. 2022c. Quantifying and alleviating political bias in language models. Artificial Intelligence, 304:103654. https://doi.org/10.1016/j.artint.2021.103654 Liu, Yinhan, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, O… [↗](https://doi.org/10.1016/j.artint.2021.103654)
86. Liu, Zeyu, Yizhong Wang, Jungo Kasai, Hannaneh Hajishirzi, and Noah A. Smith. 2021. Probing across time: What does RoBERTa know and when? In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 820–842. https://doi.org/10.18653/v1/2021.findings-emnlp.71 Magee, Liam, Lida Gha… [↗](https://doi.org/10.18653/v1/2021.findings-emnlp.71) [our card](https://md.research.elamur.ai/free/en/papers/liu-2021-probing-across-time-what-does/)
87. Mahowald, Kyle. 2023. A discerning several thousand judgments: GPT-3 rates the article + adjective + numeral + noun construction. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 265–273. [https://doi.org](https://doi.org) /10.18653/v1/2023.eacl-main… [↗](https://doi.org)
88. Malkin, Nikolay, Sameera Lanka, Pranav Goel, and Nebojsa Jojic. 2021. Studying word order through iterative shuffling. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10351–10366. https://doi.org/10.18653/v1/2021.emnlp-main.809 Mallen, Alex, Akari As… [↗](https://doi.org/10.18653/v1/2021.emnlp-main.809)
89. Mattern, Justus, Zhijing Jin, Mrinmaya Sachan, Rada Mihalcea, and B. Schölkopf. 2022. Understanding stereotypes in language models: Towards robust measurement and zero-shot debiasing. ArXiv, arXiv:2212.10678. — [arXiv:2212.10678](https://arxiv.org/abs/2212.10678) [find](https://scholar.google.com/scholar?q=Mattern%2C+Justus%2C+Zhijing+Jin%2C+Mrinmaya+Sachan%2C+Rada+Mihalcea%2C+and+B%2E+Scho%CC%88lkopf%2E+2022%2E+Understanding+stereotypes+in+language+models%3A+Towards+robust+measurement+and+zero%2Dshot+debiasing%2E+ArXiv%2C+arXiv%3A22)
90. McCoy, R. Thomas, Paul Smolensky, Tal Linzen, Jianfeng Gao, and Asli Celikyilmaz. 2021. How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN. ArXiv, arXiv:2111.09509. [our card](https://md.research.elamur.ai/free/en/papers/mccoy-2023-how-much-do-language-models/)
91. McCoy, Thomas, Robert Frank, and Tal Linzen. 2018. Revisiting the poverty of the stimulus: Hierarchical generalization without a hierarchical bias in recurrent neural networks. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 40, pages 2096–2101. [find](https://scholar.google.com/scholar?q=McCoy%2C+Thomas%2C+Robert+Frank%2C+and+Tal+Linzen%2E+2018%2E+Revisiting+the+poverty+of+the+stimulus%3A+Hierarchical+generalization+without+a+hierarchical+bias+in+recurrent+neural+networks%2E+In+Proceedings+of+the+A)
92. McCoy, Tom, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448. https://doi.org/10.18653/v1/P19 -1334 Meade, Nicho… [↗](https://doi.org/10.18653/v1/P19) [our card](https://md.research.elamur.ai/free/en/papers/meade-2022-empirical-survey-effectiveness-debiasing-techniques/)
93. Miaschi, Alessio, Dominique Brunato, Felice Dell’Orletta, and Giulia Venturi. 2020. Linguistic profiling of a neural language model. In Proceedings of the 28th International Conference on Computational Linguistics, pages 745–756. https://doi.org/10.18653/v1/2020.coling-main.65 Michaelov, James and… [↗](https://doi.org/10.18653/v1/2020.coling-main.65)
94. Misra, Kanishka, Allyson Ettinger, and Julia Rayz. 2020. Exploring BERT’s sensitivity to lexical cues using tests from semantic priming. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4625–4635. https://doi.org/10.18653/v1/2020.findings-emnlp.415 Misra, Kanishka, Al… [↗](https://doi.org/10.18653/v1/2020.findings-emnlp.415)
95. Misra, Kanishka, Julia Taylor Rayz, and Allyson Ettinger. 2023. COMPS: Conceptual minimal pair sentences for testing property knowledge and inheritance in pre-trained language models. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, page… [find](https://scholar.google.com/scholar?q=Misra%2C+Kanishka%2C+Julia+Taylor+Rayz%2C+and+Allyson+Ettinger%2E+2023%2E+COMPS%3A+Conceptual+minimal+pair+sentences+for+testing+property+knowledge+and+inheritance+in+pre%2Dtrained+language+models%2E+In+Proceedings+o)
96. Mosbach, Marius, Anna Khokhlova, Michael A. Hedderich, and Dietrich Klakow. 2020. On the interplay between fine-tuning and sentence-level probing for linguistic knowledge in pre-trained Transformers. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2502–2516. https://d… [find](https://scholar.google.com/scholar?q=Mosbach%2C+Marius%2C+Anna+Khokhlova%2C+Michael+A%2E+Hedderich%2C+and+Dietrich+Klakow%2E+2020%2E+On+the+interplay+between+fine%2Dtuning+and+sentence%2Dlevel+probing+for+linguistic+knowledge+in+pre%2Dtrained+Transformers%2E+)
97. The Keyword: Google Blog. Newman, Benjamin, Kai-Siang Ang, Julia Gong, and John Hewitt. 2021. Refining targeted syntactic evaluation of language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… [find](https://scholar.google.com/scholar?q=The+Keyword%3A+Google+Blog%2E+Newman%2C+Benjamin%2C+Kai%2DSiang+Ang%2C+Julia+Gong%2C+and+John+Hewitt%2E+2021%2E+Refining+targeted+syntactic+evaluation+of+language+models%2E+In+Proceedings+of+the+2021+Conference+of+the+No)
98. Nozza, Debora, Federico Bianchi, and Dirk Hovy. 2021. HONEST: Measuring hurtful sentence completion in language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2398–2406. https://doi.org/… [our card](https://md.research.elamur.ai/free/en/papers/nozza-2021-honest-measuring-hurtful-sentence-completion/)
99. OpenAI. 2022. ChatGPT: Optimizing language models for dialogue. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2022%2E+ChatGPT%3A+Optimizing+language+models+for+dialogue%2E)
100. OpenAI Blog. OpenAI. 2023a. GPT-4 technical report. [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
101. OpenAI. OpenAI. 2023b. Model index for researchers. [find](https://scholar.google.com/scholar?q=OpenAI%2E+OpenAI%2E+2023b%2E+Model+index+for+researchers%2E)
102. OpenAI. Ousidhoum, Nedjma, Xinran Zhao, Tianqing Fang, Yangqiu Song, and Dit-Yan Yeung. 2021. Probing toxic content in large pre-trained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural… [find](https://scholar.google.com/scholar?q=OpenAI%2E+Ousidhoum%2C+Nedjma%2C+Xinran+Zhao%2C+Tianqing+Fang%2C+Yangqiu+Song%2C+and+Dit%2DYan+Yeung%2E+2021%2E+Probing+toxic+content+in+large+pre%2Dtrained+language+models%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+t)
103. Patel, Roma and Ellie Pavlick. 2021. “was it “stated” or was it “claimed”?: How linguistic bias affects generative language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10080–10095. https://doi.org/10.18653/v1/2021.emnlp-main.790 Pedinotti… [↗](https://doi.org/10.18653/v1/2021.emnlp-main.790) [our card](https://md.research.elamur.ai/free/en/papers/perez-2022-discovering-language-model-behaviors-model/)
104. Petty, Jackson and Robert Frank. 2021. Transformers generalize linearly. ArXiv, arXiv:2109.12036. — [arXiv:2109.12036](https://arxiv.org/abs/2109.12036) [find](https://scholar.google.com/scholar?q=Petty%2C+Jackson+and+Robert+Frank%2E+2021%2E+Transformers+generalize+linearly%2E+ArXiv%2C+arXiv%3A2109%2E12036%2E+%E2%80%94+%5BarXiv%3A2109%2E12036%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2109%2E12036%29)
105. Piantadosi, Steven and Evelina Fedorenko. 2017. Infinitely productive language can arise from chance under communicative pressure. Journal of Language Evolution, 2(2):141–147. https://doi.org/10.1093/jole/lzw013 Podkorytov, Maksim, Daniel Bis, and Xiuwen Liu. 2021. How can the [MASK] know? The sour… [↗](https://doi.org/10.1093/jole/lzw013)
106. Porada, Ian, Alessandro Sordoni, and Jackie Cheung. 2022. Does pre-training induce systematic inference? How masked language models acquire commonsense knowledge. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech… [find](https://scholar.google.com/scholar?q=Porada%2C+Ian%2C+Alessandro+Sordoni%2C+and+Jackie+Cheung%2E+2022%2E+Does+pre%2Dtraining+induce+systematic+inference%3F+How+masked+language+models+acquire+commonsense+knowledge%2E+In+Proceedings+of+the+2022+Conference)
107. Press, Ofir, Noah Smith, and Mike Lewis. 2022. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Press%2C+Ofir%2C+Noah+Smith%2C+and+Mike+Lewis%2E+2022%2E+Train+short%2C+test+long%3A+Attention+with+linear+biases+enables+input+length+extrapolation%2E+In+International+Conference+on+Learning+Representations%2E)
108. Press, Ofir, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. 2022. Measuring and narrowing the compositionality gap in language models. ArXiv, arXiv:2210.03350. https://doi.org/10.18653/v1/2023.findings-emnlp.378 Qin, Lianhui, Aditya Gupta, Shyam Upadhyay, Luheng He, Yejin Cho… [↗](https://doi.org/10.18653/v1/2023.findings-emnlp.378)
109. Radford, Alec, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. [find](https://scholar.google.com/scholar?q=Radford%2C+Alec%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E)
110. OpenAI. Radford, Alec, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI. Rae, Jack W., Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susa… [find](https://scholar.google.com/scholar?q=OpenAI%2E+Radford%2C+Alec%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI%2E+Rae%2C+Jack+W%2E%2C+Sebastian+Borgeaud%2C+Trevor+C)
111. Raffel, Colin, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text Transformer. Journal of Machine Learning Research, 21(140):5485–5551. [our card](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/)
112. Raj, Harsha, Domenic Rosati, and Subhabrata Majumdar. 2022. Measuring reliability of large language models through semantic consistency. In NeurIPS ML Safety Workshop. [find](https://scholar.google.com/scholar?q=Raj%2C+Harsha%2C+Domenic+Rosati%2C+and+Subhabrata+Majumdar%2E+2022%2E+Measuring+reliability+of+large+language+models+through+semantic+consistency%2E+In+NeurIPS+ML+Safety+Workshop%2E)
113. Ravichander, Abhilasha, Eduard Hovy, Kaheer Suleman, Adam Trischler, and Jackie Chi Kit Cheung. 2020. On the systematicity of probing contextualized word representations: The case of hypernymy in BERT. In Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics, pages 88–102. [find](https://scholar.google.com/scholar?q=Ravichander%2C+Abhilasha%2C+Eduard+Hovy%2C+Kaheer+Suleman%2C+Adam+Trischler%2C+and+Jackie+Chi+Kit+Cheung%2E+2020%2E+On+the+systematicity+of+probing+contextualized+word+representations%3A+The+case+of+hypernymy+in+BERT)
114. Razeghi, Yasaman, Robert Logan IV, Matt Gardner, and Sameer Singh. 2022. Impact of pretraining term frequencies on few-shot numerical reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 840–854. https://doi.org/10.18653/v1/2022.findings-emnlp.59 Reif, Emily, D… [↗](https://doi.org/10.18653/v1/2022.findings-emnlp.59) [our card](https://md.research.elamur.ai/free/en/papers/razeghi-2022-impact-pretraining-term-frequencies-few/)
115. Rogers, Anna, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842–866. https://doi.org/10.1162/tacl_a_00349 [↗](https://doi.org/10.1162/tacl_a_00349) [our card](https://md.research.elamur.ai/free/en/papers/rogers-2020-primer-bertology-what-we-know/)
116. Romero, Julien and Simon Razniewski. 2022. Do children texts hold the key to commonsense knowledge? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10954–10959. https://doi.org/10.18653/v1/2022.emnlp-main.752 Ruis, Laura, Akbir Khan, Stella Rose Bide… [↗](https://doi.org/10.18653/v1/2022.emnlp-main.752)
117. Ryu, Soo Hyun and Richard Lewis. 2021. Accounting for agreement phenomena in sentence comprehension with transformer language models: Effects of similarity-based interference on surprisal and attention. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 61–71.… [find](https://scholar.google.com/scholar?q=Ryu%2C+Soo+Hyun+and+Richard+Lewis%2E+2021%2E+Accounting+for+agreement+phenomena+in+sentence+comprehension+with+transformer+language+models%3A+Effects+of+similarity%2Dbased+interference+on+surprisal+and+attentio)
118. Sancheti, Abhilasha and Rachel Rudinger. 2022. What do large language models learn about scripts? In Proceedings of the 11th Joint Conference on Lexical and Computational Semantics, pages 1–11. https://doi.org/10.18653/v1/2022.starsem-1.1 Sanh, Victor, Lysandre Debut, Julien Chaumond, and Thomas Wo… [↗](https://doi.org/10.18653/v1/2022.starsem-1.1)
119. Sap, Maarten, Ronan Le Bras, Daniel Fried, and Yejin Choi. 2022. Neural theory-of-mind? On the limits of social intelligence in large LMs. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3762–3780. https://doi.org/10.18653/v1/2022.emnlp-main.248 Sapa… [↗](https://doi.org/10.18653/v1/2022.emnlp-main.248)
120. Schuster, Sebastian and Tal Linzen. 2022. When a sentence does not introduce a discourse entity, transformer-based models still sometimes refer to it. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pa… [find](https://scholar.google.com/scholar?q=Schuster%2C+Sebastian+and+Tal+Linzen%2E+2022%2E+When+a+sentence+does+not+introduce+a+discourse+entity%2C+transformer%2Dbased+models+still+sometimes+refer+to+it%2E+In+Proceedings+of+the+2022+Conference+of+the+Nort)
121. Shaikh, Omar, Hongxin Zhang, William B. Held, Michael Bernstein, and Diyi Yang. 2022. On second thought, let’s not think step by step! Bias and toxicity in zero-shot reasoning. ArXiv, arXiv:2212.08061. https://doi.org/10.18653/v1/2023 .acl-long.244 Shardlow, Matthew and Piotr Przybyla. 2022. Deanthr… [↗](https://doi.org/10.18653/v1/2023) [our card](https://md.research.elamur.ai/free/en/papers/shaikh-2023-second-thought-let-s-not/)
122. Shaw, Peter, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 464–468. ht… [our card](https://md.research.elamur.ai/free/en/papers/shi-2023-large-language-models-easily-distracted/)
123. Shi, Haohan and Phillip Wolff. 2021. What Transformers might know about the physical world: T5 and the origins of knowledge. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 43, pages 2218–2224. [find](https://scholar.google.com/scholar?q=Shi%2C+Haohan+and+Phillip+Wolff%2E+2021%2E+What+Transformers+might+know+about+the+physical+world%3A+T5+and+the+origins+of+knowledge%2E+In+Proceedings+of+the+Annual+Meeting+of+the+Cognitive+Science+Society%2C+volu)
124. Shwartz, Vered and Yejin Choi. 2020. Do neural language models overcome reporting bias? In Proceedings of the 28th International Conference on Computational Linguistics, pages 6863–6870. https://doi.org/10.18653/v1/2020.coling-main.605 Shwartz, Vered, Rachel Rudinger, and Oyvind Tafjord. 2020. “you… [↗](https://doi.org/10.18653/v1/2020.coling-main.605)
125. Spitale, Giovanni, Nikola Biller-Andorno, and Federico Germani. 2023. AI model GPT-3 (dis)informs us better than humans. ArXiv, arXiv:2301.11924. https://doi.org/10.1126/sciadv.adh1850, PubMed: 37379395 Srivastava, Aarohi, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch,… [↗](https://doi.org/10.1126/sciadv.adh1850) [our card](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/)
126. Stammbach, Dominik, Maria Antoniak, and Elliott Ash. 2022. Heroes, villains, and victims, and GPT-3: Automated extraction of character roles without training data. In Proceedings of the 4th Workshop of Narrative Understanding (WNU2022), pages 47–56. https://doi.org/10.18653/v1/2022.wnu-1.6 [↗](https://doi.org/10.18653/v1/2022.wnu-1.6)
127. Stevenson, Claire, Iris Smal, Matthijs Baas, Raoul Grasman, and Han van der Maas. 2022. Putting GPT-3’s creativity to the (alternative uses) test. In International Conference on Computational Creativity, pages 164–168. Stolfo, Alessandro, Zhijing Jin, Kumar Shridhar, Bernhard Schölkopf, and Mrinmay… [our card](https://md.research.elamur.ai/free/en/papers/stolfo-2023-causal-framework-quantify-robustness-mathematical/)
128. Su, Jianlin, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2021. RoFormer: Enhanced Transformer with rotary position embedding. ArXiv, arXiv:2104.09864. [our card](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/)
129. Summers-Stay, Douglas, Claire Bonial, and Clare Voss. 2021. What can a generative language model answer about a passage? In Proceedings of the 3rd Workshop on Machine Reading for Question Answering, pages 73–81. https://doi.org/10.18653/v1/2021.mrqa-1.7 Suzgun, Mirac, Nathan Scales, Nathanael Schar… [↗](https://doi.org/10.18653/v1/2021.mrqa-1.7) [our card](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/)
130. Tal, Yarden, Inbal Magar, and Roy Schwartz. 2022. Fewer errors, but more stereotypes? The effect of model size on gender bias. In Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pages 112–120. https://doi.org/10.18653/v1/2022.gebnlp-1.13 Tamborrino, Alexandre… [↗](https://doi.org/10.18653/v1/2022.gebnlp-1.13)
131. Tay, Yi, Mostafa Dehghani, Samira Abnar, Hyung Won Chung, W. Fedus, Jinfeng Rao, Sharan Narang, Vinh Tran, Dani Yogatama, and Donald Metzler. 2022a. Scaling laws vs model architectures: How does inductive bias influence scaling? ArXiv, arXiv:2207.10551. https://doi.org/10.18653/v1 /2023.findings-emn… [our card](https://md.research.elamur.ai/free/en/papers/tay-2023-scaling-laws-vs-model-architectures/)
132. Tejada, Giorgia Nidia Carranza, Johannes Scholtes, and Gerasimos Spanakis. 2021. A study of BERT’s processing of negations to determine sentiment. Benelux Conference on Artificial Intelligence and the Belgian Dutch Conference on Machine Learning, pages 47–59. [find](https://scholar.google.com/scholar?q=Tejada%2C+Giorgia+Nidia+Carranza%2C+Johannes+Scholtes%2C+and+Gerasimos+Spanakis%2E+2021%2E+A+study+of+BERT%E2%80%99s+processing+of+negations+to+determine+sentiment%2E+Benelux+Conference+on+Artificial+Intelligence+and+the)
133. Tenney, Ian, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593–4601. https://doi.org/10.18653/v1/P19-1452 [↗](https://doi.org/10.18653/v1/P19-1452) [our card](https://md.research.elamur.ai/free/en/papers/tenney-2019-bert-rediscovers-classical-nlp-pipeline/)
134. Thewsey, Alex. 2021. Bring structure to diverse documents with Amazon Textract and transformer-based models on Amazon SageMaker. [find](https://scholar.google.com/scholar?q=Thewsey%2C+Alex%2E+2021%2E+Bring+structure+to+diverse+documents+with+Amazon+Textract+and+transformer%2Dbased+models+on+Amazon+SageMaker%2E)
135. AWS Machine Learning Blog. Tirumala, Kushal, Aram H. Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. 2022. Memorization without overfitting: Analyzing the training dynamics of large language models. In Advances in Neural Information Processing Systems, volume 35, pages 38274–38290. [our card](https://md.research.elamur.ai/free/en/papers/tirumala-2022-memorization-overfitting-analyzing-training-dynamics/)
136. Touileb, Samia. 2022. Exploring the effects of negation and grammatical tense on bias probes. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short… [find](https://scholar.google.com/scholar?q=Touileb%2C+Samia%2E+2022%2E+Exploring+the+effects+of+negation+and+grammatical+tense+on+bias+probes%2E+In+Proceedings+of+the+2nd+Conference+of+the+Asia%2DPacific+Chapter+of+the+Association+for+Computational+Ling)
137. Traylor, Aaron, Roman Feiman, and Ellie Pavlick. 2021. AND does not mean OR: Using formal languages to study language models’ representations. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Pr… [find](https://scholar.google.com/scholar?q=Traylor%2C+Aaron%2C+Roman+Feiman%2C+and+Ellie+Pavlick%2E+2021%2E+AND+does+not+mean+OR%3A+Using+formal+languages+to+study+language+models%E2%80%99+representations%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Associat)
138. Blog Post. Trott, Sean, Cameron J. Jones, Tyler A. Chang, James Michaelov, and Benjamin Bergen. 2023. Do large language models know what humans know? Cognitive Science, 47(7):e13309. https://doi.org/10.1111/cogs.13309, PubMed: 37401923 Truong, Thinh Hung, Yulia Otmakhova, Timothy Baldwin, Trevor Co… [↗](https://doi.org/10.1111/cogs.13309) [our card](https://md.research.elamur.ai/free/en/papers/truong-2022-not-another-negation-benchmark-nan/)
139. Tuckute, Greta, Aalok Sathe, Mingye Wang, Harley Yoder, Cory Shain, and Evelina Fedorenko. 2022. SentSpace: Large-scale benchmarking and evaluation of text using cognitively motivated lexical, syntactic, and semantic features. In Proceedings of the 2022 Conference of the North American Chapter of th… [find](https://scholar.google.com/scholar?q=Tuckute%2C+Greta%2C+Aalok+Sathe%2C+Mingye+Wang%2C+Harley+Yoder%2C+Cory+Shain%2C+and+Evelina+Fedorenko%2E+2022%2E+SentSpace%3A+Large%2Dscale+benchmarking+and+evaluation+of+text+using+cognitively+motivated+lexical%2C+syntact)
140. Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, pages 5998–6008. [our card](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/)
141. Vig, Jesse and Yonatan Belinkov. 2019. Analyzing the structure of attention in a transformer language model. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 63–76. https://doi.org/10.18653/v1/W19 -4808 Vig, Jesse, Sebastian Gehrmann, Yon… [↗](https://doi.org/10.18653/v1/W19) [our card](https://md.research.elamur.ai/free/en/papers/vig-2019-analyzing-structure-attention-transformer-language/)
142. Wahle, Jan Philip, Terry Ruas, Frederic Kirstein, and Bela Gipp. 2022. How large language models are transforming machine-paraphrase plagiarism. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 952–963. https://doi.org/10.18653/v1/2022.emnlp-main.62 W… [↗](https://doi.org/10.18653/v1/2022.emnlp-main.62)
143. Wang, Boshi, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. 2022a. Towards understanding chain-of-thought prompting: An empirical study of what matters. ArXiv, arXiv:2212.10001. https://doi.org/10.18653/v1/2023.acl-long.153 Wang, Benyou, Lifeng Shang, Christina Lioma,… [↗](https://doi.org/10.18653/v1/2023.acl-long.153) [our card](https://md.research.elamur.ai/free/en/papers/wang-2023-understanding-chain-thought-prompting-empirical/)
144. Wang, Cunxiang, Boyuan Zheng, Yuchen Niu, and Yue Zhang. 2021b. Exploring generalization ability of pretrained language models on arithmetic and logical reasoning. In Natural Language Processing and Chinese Computing, pages 758–769. https://doi.org/10.1007/978-3-030 -88480-2 61 Wang, Tianlu, Rohit S… [↗](https://doi.org/10.1007/978-3-030)
145. Wei, Jason, Dan Garrette, Tal Linzen, and Ellie Pavlick. 2021. Frequency effects on syntactic rule learning in transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 932–948. https://doi.org/10.18653/v1/2021.emnlp-main.72 Wei, Jason, Najoung K… [↗](https://doi.org/10.18653/v1/2021.emnlp-main.72) [our card](https://md.research.elamur.ai/free/en/papers/wei-2021-frequency-effects-syntactic-rule-learning/)
146. Wei, Jason, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022c. Chain of thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824–24837. [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)
147. Weidinger, Laura, John F. J. Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, Zachary Kenton, Sande Minnich Brown, William T. Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hendric… [our card](https://md.research.elamur.ai/free/en/papers/weidinger-2021-ethical-social-risks-harm-language/)
148. Weidinger, Laura, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, Courtney Biles, Sasha Brown, Zac Kenton, Will Hawkins, Tom Stepleton, Abeba Birhane, Lisa Anne Hendricks, Laura Rimell, William Isaac, Julia Haas, Se… [find](https://scholar.google.com/scholar?q=Weidinger%2C+Laura%2C+Jonathan+Uesato%2C+Maribeth+Rauh%2C+Conor+Griffin%2C+Po%2DSen+Huang%2C+John+Mellor%2C+Amelia+Glaese%2C+Myra+Cheng%2C+Borja+Balle%2C+Atoosa+Kasirzadeh%2C+Courtney+Biles%2C+Sasha+Brown%2C+Zac+Kenton%2C+Will+Haw)
149. Linguistic Inquiry, pages 1–88. https://doi.org/10.1162/ling a 00491 Williams, Adina, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Co… [↗](https://doi.org/10.1162/ling)
150. Zhang, Lining, Mengchen Wang, Liben Chen, and Wenxin Zhang. 2022a. Probing GPT-3’s linguistic knowledge on semantic tasks. In Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 297–304. https://doi.org/10.18653/v1/2022.blackboxnlp-1.24 Zhang,… [↗](https://doi.org/10.18653/v1/2022.blackboxnlp-1.24)
151. Zhang, Sheng, Xin Zhang, Weiming Zhang, and Anders Søgaard. 2021a. Sociolectal analysis of pretrained language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4581–4588. https://doi.org/10.18653/v1/2021.emnlp-main.375 Zhang, Tianyi, Varsha Ki… [↗](https://doi.org/10.18653/v1/2021.emnlp-main.375)
152. Zhang, Yian, Alex Warstadt, Xiaocheng Li, and Samuel R. Bowman. 2021b. When do you need billions of words of pretraining data? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume… [find](https://scholar.google.com/scholar?q=Zhang%2C+Yian%2C+Alex+Warstadt%2C+Xiaocheng+Li%2C+and+Samuel+R%2E+Bowman%2E+2021b%2E+When+do+you+need+billions+of+words+of+pretraining+data%3F+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computat)
153. Zhang, Zhengyan, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: Enhanced language representation with informative entities. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1441–1451. https://doi.org/10.18653/v1/P19 -1139 Zhao,… [↗](https://doi.org/10.18653/v1/P19)
154. Zwaan, Rolf A. 2016. Situation models, mental simulations, and abstract concepts in discourse comprehension. Psychonomic Bulletin & Review, 23:1028–1034. https://doi.org/10.3758/s13423-015-0864-x, PubMed: 26088667 — [doi:10.3758/s13423-015-0864-x](https://doi.org/10.3758/s13423-015-0864-x) [↗](https://doi.org/10.3758/s13423-015-0864-x)

*Updated 24 October 2026*