---
title: "Generative Models as a Complex Systems Science: How can we"
description: "The authors present no new data or model. 142 references, cited by 17 papers here. Among them: 4."
authors: ["Ari Holtzman", "Peter West", "Luke Zettlemoyer"]
year: 2023
venue: "arXiv:2308.00189"
source: "https://arxiv.org/abs/2308.00189"
doi: "10.48550/arxiv.2308.00189"
arxiv: "2308.00189"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/holtzman-2023-generative-models-complex-systems-science/"
---
Elamur: [Mova](https://md.mova.elamur.ai/free/en/) · [Dictionary](https://md.dictionary.elamur.ai/free/en/) · **Research** · [Groupants](https://md.groupants.elamur.ai/free/en/)

[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/holtzman-2023-generative-models-complex-systems-science/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/13/) / Generative Models as a Complex Systems Science: How can we make sense of large language model behavior?

# Generative Models as a Complex Systems Science: How can we make sense of large language model behavior?

**Authors:** [Ari Holtzman](https://md.research.elamur.ai/free/en/authors/ari-holtzman/), [Peter West](https://md.research.elamur.ai/free/en/authors/peter-west/) and [Luke Zettlemoyer](https://md.research.elamur.ai/free/en/authors/luke-zettlemoyer/) · **Year:** 2023 · **Venue:** arXiv:2308.00189 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2308.00189) · [PDF](https://arxiv.org/pdf/2308.00189) · [DOI 10.48550/arxiv.2308.00189](https://doi.org/10.48550/arxiv.2308.00189) · [arXiv 2308.00189](https://arxiv.org/abs/2308.00189) · [OpenAlex](https://openalex.org/W4385507660)

**Citations:** 4 citations (OpenAlex, 2026-10-14) · [17](https://md.research.elamur.ai/free/en/papers/holtzman-2023-generative-models-complex-systems-science/cited-by/) in our library, of the 150,927 papers whose bibliographies we read

## Summary
This position paper argues that research on large language models lacks a systematic vocabulary for what these models actually do. Benchmarks keep multiplying, but they do not explain which underlying behaviors let models succeed across tasks. The authors present no new data or model. Instead, they build their case around a thought experiment: a hypothetical "Newformer" that beats Transformers yet is architecturally alien and too costly for anyone to retrain. Such a model would defeat most current interpretability tools. They propose a top-down taxonomy of output-level behaviors, formalized as metamodels that predict limited aspects of a model's output. This taxonomy would then direct bottom-up mechanistic work. They frame generative models as complex systems whose emergent behaviors are usually discovered rather than engineered. They also note that, unlike natural systems, these models can be simulated and support repeatable experiments, but only if they are open-source. The main message is to establish *what* models do before asking how and why. This would make analysis more robust to future architecture changes.

## Key points
- **Newformer thought experiment:** an attention-free, partly non-differentiable model that is unreplicable at scale shows how fragile Transformer-specific interpretation methods are. The authors compare this to earlier gating-based RNN/LSTM analyses (e.g., ELMo era), which did not carry over to Transformers such as BERT.
- **Behavior guiding mechanisms:** Al-Rfou et al. (2019) observed copying behavior, which prompted researchers first to identify copying heads and later to find the mechanism tied to in-context learning known as induction heads, as reported by Elhage et al. (2021) and Olsson et al. (2022).
- **Bottom-up analysis can mislead:** embedding anisotropy was widely treated as a flaw. Later work suggests it may not limit expressivity and may even help LMs.
- **Working definition of behavior:** building on Fong and Vedaldi (2017), a behavior is a partial explanation, meaning a metamodel that predicts certain aspects of a black-box model's output.
- **Access and openness:** training a GPT-3-level model costs about half a million dollars in compute, so most labs cannot do it. Proprietary APIs such as ChatGPT and GPT-4 drift over time and are silently patched, which introduces observer effects. Open-source models lag roughly 1–2 years behind and are essential for replicable science.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 422 works · 213,644 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 4,145 works

## Cite

Cite as: Ari Holtzman, Peter West and Luke Zettlemoyer. 2023. Generative Models as a Complex Systems Science: How can we make sense of large language model behavior?. arXiv:2308.00189.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{holtzman2023generative,
  title = {{Generative Models as a Complex Systems Science: How can we make sense of large language model behavior?}},
  author = {Ari Holtzman, Peter West and Luke Zettlemoyer},
  year = {2023},
  doi = {10.48550/arxiv.2308.00189},
  eprint = {2308.00189},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2308.00189}
}
```

- [← previous Efficiently Adapting Pretrained Language Models To New Languages](https://md.research.elamur.ai/free/en/papers/csaki-2023-efficiently-adapting-pretrained-language-models/)
- [next → Large-scale Language Model Rescoring on Long-form Data](https://md.research.elamur.ai/free/en/papers/chen-2023-large-scale-language-model-rescoring/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/13/)

## References

142 references, 57 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Linden Li Abhinav Venigalla. 2022. Mosaic LLMs (part 2): GPT-3 quality for \\<$500k. [https://www.mosaicml.com/blog/gpt-3-quality-for-500k](https://www.mosaicml.com/blog/gpt-3-quality-for-500k). Accessed: 2023-7-7. [↗](https://www.mosaicml.com/blog/gpt-3-quality-for-500k)
2. Armen Aghajanyan, Dmytro Okhonko, Mike Lewis, Mandar Joshi, Hu Xu, Gargi Ghosh, and Luke Zettlemoyer. 2022. HTLM: Hyper-text pre-training and prompting of language models. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Armen+Aghajanyan%2C+Dmytro+Okhonko%2C+Mike+Lewis%2C+Mandar+Joshi%2C+Hu+Xu%2C+Gargi+Ghosh%2C+and+Luke+Zettlemoyer%2E+2022%2E+HTLM%3A+Hyper%2Dtext+pre%2Dtraining+and+prompting+of+language+models%2E+In+International+Conference+)
3. Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. 2019. [Character-level language modeling with deeper self-attention](https://md.research.elamur.ai/free/en/papers/al-rfou-2018-character-level-language-modeling-deeper/). In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 3159–3166.
4. Meysam Alizadeh, Maël Kubli, Zeynab Samei, Shirin Dehghani, Juan Diego Bermeo, Maria Ko-robeynikova, and Fabrizio Gilardi. 2023. Open-Source large language models outperform crowd workers and approach ChatGPT in Text-Annotation tasks. [find](https://scholar.google.com/scholar?q=Meysam+Alizadeh%2C+Ma%C3%ABl+Kubli%2C+Zeynab+Samei%2C+Shirin+Dehghani%2C+Juan+Diego+Bermeo%2C+Maria+Ko%2Drobeynikova%2C+and+Fabrizio+Gilardi%2E+2023%2E+Open%2DSource+large+language+models+outperform+crowd+workers+and+approach)
5. Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. Falcon-40B: an open large language model with state-of-t… [find](https://scholar.google.com/scholar?q=Ebtesam+Almazrouei%2C+Hamza+Alobeidli%2C+Abdulaziz+Alshamsi%2C+Alessandro+Cappelli%2C+Ruxandra+Cojocaru%2C+Merouane+Debbah%2C+Etienne+Goffinet%2C+Daniel+Heslow%2C+Julien+Launay%2C+Quentin+Malartic%2C+Badreddine+Noune%2C+Ba)
6. Jacob Andreas. 2022. [Language models as agent models](https://md.research.elamur.ai/free/en/papers/andreas-2022-language-models-as-agent-models/). In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5769–5779.
7. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/).
8. C M Barry. 2014. Who sharpened occam’s razor? [https://www.irishphilosophy.com/2014/05/](https://www.irishphilosophy.com/2014/05/) 27/who-sharpened-occams-razor/. Accessed: 2023-7-7. [↗](https://www.irishphilosophy.com/2014/05/)
9. Russell R Barton and Martin Meckesheimer. 2006. Metamodel-based simulation optimization. Handbooks in operations research and management science, 13:535–574. [find](https://scholar.google.com/scholar?q=Russell+R+Barton+and+Martin+Meckesheimer%2E+2006%2E+Metamodel%2Dbased+simulation+optimization%2E+Handbooks+in+operations+research+and+management+science%2C+13%3A535%E2%80%93574%2E)
10. Jasmijn Bastings, Yonatan Belinkov, Yanai Elazar, Dieuwke Hupkes, Naomi Saphra, and Sarah Wiegreffe. 2022. Proceedings of the fifth blackboxnlp workshop on analyzing and interpreting neural networks for nlp. In Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networ… [find](https://scholar.google.com/scholar?q=Jasmijn+Bastings%2C+Yonatan+Belinkov%2C+Yanai+Elazar%2C+Dieuwke+Hupkes%2C+Naomi+Saphra%2C+and+Sarah+Wiegreffe%2E+2022%2E+Proceedings+of+the+fifth+blackboxnlp+workshop+on+analyzing+and+interpreting+neural+networks+f)
11. Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. [Longformer: The long-document transformer](https://md.research.elamur.ai/free/en/papers/beltagy-2020-longformer-long-document-transformer/). arXiv preprint arXiv:2004.05150.
12. BigScience. 2022. Bigscience model training launched. BigScience Blog. [find](https://scholar.google.com/scholar?q=BigScience%2E+2022%2E+Bigscience+model+training+launched%2E+BigScience+Blog%2E)
13. Daniel Biś, Maksim Podkorytov, and Xiuwen Liu. 2021. Too much in common: Shifting of embeddings in transformer language models and its implications. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pag… [find](https://scholar.google.com/scholar?q=Daniel+Bis%CC%81%2C+Maksim+Podkorytov%2C+and+Xiuwen+Liu%2E+2021%2E+Too+much+in+common%3A+Shifting+of+embeddings+in+transformer+language+models+and+its+implications%2E+In+Proceedings+of+the+2021+Conference+of+the+North)
14. Terra Blevins and Luke Zettlemoyer. 2022. Language contamination explains the cross-lingual capabilities of english pretrained models. arXiv preprint arXiv:2204.08110. — [arXiv:2204.08110](https://arxiv.org/abs/2204.08110) [find](https://scholar.google.com/scholar?q=Terra+Blevins+and+Luke+Zettlemoyer%2E+2022%2E+Language+contamination+explains+the+cross%2Dlingual+capabilities+of+english+pretrained+models%2E+arXiv+preprint+arXiv%3A2204%2E08110%2E+%E2%80%94+%5BarXiv%3A2204%2E08110%5D%28https%3A%2F%2Farx)
15. Ludwig Boltzmann. 1868. Studien über das gle-ichgewicht der lebendigen kraft zwischen bewegten materiellen punkten [studies on the balance of living force between moving material points]. Wiener Berichte, 58:517–560. [find](https://scholar.google.com/scholar?q=Ludwig+Boltzmann%2E+1868%2E+Studien+%C3%BCber+das+gle%2Dichgewicht+der+lebendigen+kraft+zwischen+bewegten+materiellen+punkten+%5Bstudies+on+the+balance+of+living+force+between+moving+material+points%5D%2E+Wiener+Beric)
16. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
17. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. [Sparks of artificial general intelligence: Early experiments with gpt-4](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/). arXiv preprint arXiv:2303.12712.
18. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2023. Quantifying memorization across neural language models. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Nicholas+Carlini%2C+Daphne+Ippolito%2C+Matthew+Jagielski%2C+Katherine+Lee%2C+Florian+Tramer%2C+and+Chiyuan+Zhang%2E+2023%2E+Quantifying+memorization+across+neural+language+models%2E+In+The+Twelfth+International+Confe)
19. Mike Caro. 2003. Caro’s book of poker tells. Cardoza Publishing. [find](https://scholar.google.com/scholar?q=Mike+Caro%2E+2003%2E+Caro%E2%80%99s+book+of+poker+tells%2E+Cardoza+Publishing%2E)
20. Tyler A Chang and Benjamin K Bergen. 2023. [Language model behavior: A comprehensive survey](https://md.research.elamur.ai/free/en/papers/chang-2024-language-model-behavior-comprehensive-survey/). arXiv preprint arXiv:2303.11504.
21. 百度百科. 2022. 普通高等学校招生全国统一考试. [find](https://scholar.google.com/scholar?q=%E7%99%BE%E5%BA%A6%E7%99%BE%E7%A7%91%2E+2022%2E+%E6%99%AE%E9%80%9A%E9%AB%98%E7%AD%89%E5%AD%A6%E6%A0%A1%E6%8B%9B%E7%94%9F%E5%85%A8%E5%9B%BD%E7%BB%9F%E4%B8%80%E8%80%83%E8%AF%95%2E)
22. Chacha Chen, Shi Feng, Amit Sharma, and Chenhao Tan. 2023. Machine explanations and human understanding. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, pages 1–1. [find](https://scholar.google.com/scholar?q=Chacha+Chen%2C+Shi+Feng%2C+Amit+Sharma%2C+and+Chenhao+Tan%2E+2023%2E+Machine+explanations+and+human+understanding%2E+In+Proceedings+of+the+2023+ACM+Conference+on+Fairness%2C+Accountability%2C+and+Transparency%2C+pages+)
23. Yonatan Belinkov and Yonatan Bisk. 2017. Synthetic and natural noise both break neural machine translation. International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Yonatan+Belinkov+and+Yonatan+Bisk%2E+2017%2E+Synthetic+and+natural+noise+both+break+neural+machine+translation%2E+International+Conference+on+Learning+Representations%2E)
24. Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. [Generating long sequences with sparse transformers](https://md.research.elamur.ai/free/en/papers/child-2019-generating-long-sequences-sparse-transformers/). arXiv preprint arXiv:1904.10509.
25. Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. [On the properties of neural machine translation: Encoder–decoder approaches](https://md.research.elamur.ai/free/en/papers/cho-2014-properties-neural-machine-translation-encoder/). In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pages 103–111.
26. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416. [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Le+Hou%2C+Shayne+Longpre%2C+Barret+Zoph%2C+Yi+Tay%2C+William+Fedus%2C+Eric+Li%2C+Xuezhi+Wang%2C+Mostafa+Dehghani%2C+Siddhartha+Brahma%2C+et+al%2E+2022%2E+Scaling+instruction%2Dfinetuned+language+models%2E+arXi)
27. Thomas M Cover. 1999. Elements of information theory. John Wiley & Sons. [find](https://scholar.google.com/scholar?q=Thomas+M+Cover%2E+1999%2E+Elements+of+information+theory%2E+John+Wiley+%26+Sons%2E)
28. Aleksandra Ćwiek, Susanne Fuchs, Christoph Draxler, Eva Liina Asu, Dan Dediu, Katri Hiovain, Shigeto Kawahara, Sofia Koutalidis, Manfred Krifka, Pär-tel Lippus, Gary Lupyan, Grace E Oh, Jing Paul, Caterina Petrone, Rachid Ridouane, Sabine Reiter, Nathalie Schümchen, Ádám Szalontai, Özlem Ünal-Logac… [find](https://scholar.google.com/scholar?q=Aleksandra+C%CC%81wiek%2C+Susanne+Fuchs%2C+Christoph+Draxler%2C+Eva+Liina+Asu%2C+Dan+Dediu%2C+Katri+Hiovain%2C+Shigeto+Kawahara%2C+Sofia+Koutalidis%2C+Manfred+Krifka%2C+P%C3%A4r%2Dtel+Lippus%2C+Gary+Lupyan%2C+Grace+E+Oh%2C+Jing+Paul%2C+Ca)
29. Giannis Daras and Alexandros G Dimakis. 2022. Discovering the hidden vocabulary of dalle-2. arXiv preprint arXiv:2206.00169. — [arXiv:2206.00169](https://arxiv.org/abs/2206.00169) [find](https://scholar.google.com/scholar?q=Giannis+Daras+and+Alexandros+G+Dimakis%2E+2022%2E+Discovering+the+hidden+vocabulary+of+dalle%2D2%2E+arXiv+preprint+arXiv%3A2206%2E00169%2E+%E2%80%94+%5BarXiv%3A2206%2E00169%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2206%2E00169%29)
30. Yuntian Deng. 2023. OpenAI watch. [https://openaiwatch.com/](https://openaiwatch.com/). Accessed: 2023-7-6, source: https://twitter.com/yuntiandeng/status/ 1641108596510343168?s=20. [↗](https://openaiwatch.com/)
31. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. [Qlora: Efficient finetuning of quantized llms](https://md.research.elamur.ai/free/en/papers/dettmers-2023-qlora-efficient-finetuning-quantized-llms/). arXiv preprint arXiv:2305.14314.
32. Tyna Eloundou, Sam Manning, Pamela Mishkin, and Daniel Rock. 2023. Gpts are gpts: An early look at the labor market impact potential of large language models. arXiv preprint arXiv:2303.10130. — [arXiv:2303.10130](https://arxiv.org/abs/2303.10130) [find](https://scholar.google.com/scholar?q=Tyna+Eloundou%2C+Sam+Manning%2C+Pamela+Mishkin%2C+and+Daniel+Rock%2E+2023%2E+Gpts+are+gpts%3A+An+early+look+at+the+labor+market+impact+potential+of+large+language+models%2E+arXiv+preprint+arXiv%3A2303%2E10130%2E+%E2%80%94+%5BarXiv)
33. Kawin Ethayarajh. 2019. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process… [find](https://scholar.google.com/scholar?q=Kawin+Ethayarajh%2E+2019%2E+How+contextual+are+contextualized+word+representations%3F+comparing+the+geometry+of+bert%2C+elmo%2C+and+gpt%2D2+embeddings%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+i)
34. Ruth C Fong and Andrea Vedaldi. 2017. Interpretable explanations of black boxes by meaningful perturbation. In Proceedings of the IEEE international conference on computer vision, pages 3429–3437. [find](https://scholar.google.com/scholar?q=Ruth+C+Fong+and+Andrea+Vedaldi%2E+2017%2E+Interpretable+explanations+of+black+boxes+by+meaningful+perturbation%2E+In+Proceedings+of+the+IEEE+international+conference+on+computer+vision%2C+pages+3429%E2%80%933437%2E)
35. National Association for College Admission Counseling. 2008. Report of the commission on the use of standardized tests in undergraduate admission. ERIC Clearinghouse. [find](https://scholar.google.com/scholar?q=National+Association+for+College+Admission+Counseling%2E+2008%2E+Report+of+the+commission+on+the+use+of+standardized+tests+in+undergraduate+admission%2E+ERIC+Clearinghouse%2E)
36. Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tieyan Liu. 2019. [Representation degeneration problem in training natural language generation models](https://md.research.elamur.ai/free/en/papers/gao-2019-representation-degeneration-problem-training-natural/). In International Conference on Learning Representations.
37. Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, et al. 2021. [The gem benchmark: Natural language generation, its evaluation and metrics](https://md.research.elamur.ai/free/en/papers/gehrmann-2021-gem-benchmark-natural-language-generation/). In Proceedings…
38. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
39. A Shaji George and AS Hovan George. 2023. A review of chatgpt ai’s impact on several business sectors. Partners Universal International Innovation Journal, 1(1):9–23. [find](https://scholar.google.com/scholar?q=A+Shaji+George+and+AS+Hovan+George%2E+2023%2E+A+review+of+chatgpt+ai%E2%80%99s+impact+on+several+business+sectors%2E+Partners+Universal+International+Innovation+Journal%2C+1%281%29%3A9%E2%80%9323%2E)
40. Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan†, Nicholas Joseph†, Ben Mann†, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown,… [find](https://scholar.google.com/scholar?q=Nelson+Elhage%2C+Neel+Nanda%2C+Catherine+Olsson%2C+Tom+Henighan%E2%80%A0%2C+Nicholas+Joseph%E2%80%A0%2C+Ben+Mann%E2%80%A0%2C+Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+Tom+Conerly%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Deep+Ganguli%2C+Zac+Hatfield%2DDodds%2C)
41. Andreas Grivas, Nikolay Bogoychev, and Adam Lopez. 2022. Low-rank softmax can have unargmaxable classes in theory but rarely in practice. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6738–6758, Dublin, Ireland. Association… [find](https://scholar.google.com/scholar?q=Andreas+Grivas%2C+Nikolay+Bogoychev%2C+and+Adam+Lopez%2E+2022%2E+Low%2Drank+softmax+can+have+unargmaxable+classes+in+theory+but+rarely+in+practice%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+f)
42. Nathan Godey, Éric de la Clergerie, and Benoît Sagot. 2023. Is anisotropy inherent to transformers? arXiv preprint arXiv:2306.07656. — [arXiv:2306.07656](https://arxiv.org/abs/2306.07656) [find](https://scholar.google.com/scholar?q=Nathan+Godey%2C+%C3%89ric+de+la+Clergerie%2C+and+Beno%C3%AEt+Sagot%2E+2023%2E+Is+anisotropy+inherent+to+transformers%3F+arXiv+preprint+arXiv%3A2306%2E07656%2E+%E2%80%94+%5BarXiv%3A2306%2E07656%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2306%2E07656%29)
43. Peter D Grünwald. 2007. The minimum description length principle. MIT press. [find](https://scholar.google.com/scholar?q=Peter+D+Gr%C3%BCnwald%2E+2007%2E+The+minimum+description+length+principle%2E+MIT+press%2E)
44. Eray Eliaçık. Playing with fire: The leaked plugin DAN Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio unchains ChatGPT from its moral and ethical restric-César Teodoro Mendes, Allie Del Giorno, Sivakanth tions. [https://dataconomy.com/2023/03/31/](https://dataconomy.com/2023/03/31/) Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo chatgp… [↗](https://dataconomy.com/2023/03/31/)
45. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A Smith. 2018. [Annotation artifacts in natural language inference data](https://md.research.elamur.ai/free/en/papers/gururangan-2018-annotation-artifacts-natural-language-inference/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technol…
46. Nikhil Kandpal, Eric Wallace, and Colin Raffel. 2022. [Deduplicating training data mitigates privacy risks in language models](https://md.research.elamur.ai/free/en/papers/kandpal-2022-deduplicating-training-data-mitigates-privacy/). In International Conference on Machine Learning, pages 10697–10707. PMLR.
47. Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. 2019. [Learning latent dynamics for planning from pixels](https://md.research.elamur.ai/free/en/papers/hafner-2018-learning-latent-dynamics-planning-pixels/). In International conference on machine learning, pages 2555–2565. PMLR.
48. Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019. [Ctrl: A conditional transformer language model for controllable generation](https://md.research.elamur.ai/free/en/papers/keskar-2019-ctrl-conditional-transformer-language-model/). arXiv preprint arXiv:1909.05858.
49. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In International Conference on Learning Representations.
50. Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415. [find](https://scholar.google.com/scholar?q=Dan+Hendrycks+and+Kevin+Gimpel%2E+2016%2E+Gaussian+error+linear+units+%28gelus%29%2E+arXiv+preprint+arXiv%3A1606%2E08415%2E)
51. Benjamin Hilton. 2022. No, DALL-E doesn’t have a secret language.(or at least, we haven’t found one yet)this viral DALL-E thread has some pretty astounding claims. but maybe the reason they’re so astounding is that, for the most part, they’re not true. thread (1/15) [https://t.co/8F2WDp7lTK](https://t.co/8F2WDp7lTK). https://… [↗](https://t.co/8F2WDp7lTK)
52. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735– 1780. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+J%C3%BCrgen+Schmidhuber%2E+1997%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%3A1735%E2%80%93+1780%2E)
53. John H Holland. 2014. Complexity: A very short introduction. OUP Oxford. [find](https://scholar.google.com/scholar?q=John+H+Holland%2E+2014%2E+Complexity%3A+A+very+short+introduction%2E+OUP+Oxford%2E)
54. Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. 2020. [The curious case of neural text degeneration](https://md.research.elamur.ai/free/en/papers/holtzman-2019-curious-case-neural-text-degeneration/). International Conference on Learning Representations.
55. Gabriel Ilharco, Rowan Zellers, Ali Farhadi, and Hannaneh Hajishirzi. 2021. [Probing contextual language models for common ground with visual representations](https://md.research.elamur.ai/free/en/papers/ilharco-2021-probing-contextual-language-models-common/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog…
56. Alon Jacovi and Yoav Goldberg. 2020. [Towards faithfully interpretable nlp systems: How should we define and evaluate faithfulness?](https://md.research.elamur.ai/free/en/papers/jacovi-2020-faithfully-interpretable-nlp-systems-how/) In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4198–4205.
57. Neel Jain, Khalid Saifullah, Yuxin Wen, John Kirchenbauer, Manli Shu, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein. 2023. Bring your own data! self-supervised evaluation for large language models. arXiv preprint arXiv:2306.13651. — [arXiv:2306.13651](https://arxiv.org/abs/2306.13651) [find](https://scholar.google.com/scholar?q=Neel+Jain%2C+Khalid+Saifullah%2C+Yuxin+Wen%2C+John+Kirchenbauer%2C+Manli+Shu%2C+Aniruddha+Saha%2C+Micah+Goldblum%2C+Jonas+Geiping%2C+and+Tom+Goldstein%2E+2023%2E+Bring+your+own+data%21+self%2Dsupervised+evaluation+for+large+)
58. Andrej Karpathy, Justin Johnson, and Li Fei-Fei. 2015. [Visualizing and understanding recurrent networks](https://md.research.elamur.ai/free/en/papers/karpathy-2015-visualizing-understanding-recurrent-networks/).
59. Urvashi Khandelwal, Kevin Clark, Dan Jurafsky, and Lukasz Kaiser. 2019. Sample Efficient Text Summarization Using a Single Pre-Trained Transformer. arXiv preprint arXiv:1905.08836. [find](https://scholar.google.com/scholar?q=Urvashi+Khandelwal%2C+Kevin+Clark%2C+Dan+Jurafsky%2C+and+Lukasz+Kaiser%2E+2019%2E+Sample+Efficient+Text+Summarization+Using+a+Single+Pre%2DTrained+Transformer%2E+arXiv+preprint+arXiv%3A1905%2E08836%2E)
60. Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky. 2018. [Sharp nearby, fuzzy far away: How neural language models use context](https://md.research.elamur.ai/free/en/papers/khandelwal-2018-sharp-nearby-fuzzy-far-away/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 284–294.
61. Lee Kiho. ChatGPT_DAN: ChatGPT DAN, jailbreaks prompt. [find](https://scholar.google.com/scholar?q=Lee+Kiho%2E+ChatGPT%5FDAN%3A+ChatGPT+DAN%2C+jailbreaks+prompt%2E)
62. Yair Lakretz, German Kruszewski, Theo Desbordes, Dieuwke Hupkes, Stanislas Dehaene, and Marco Baroni. 2019. [The emergence of number and syntax units in LSTM language models](https://md.research.elamur.ai/free/en/papers/lakretz-2019-emergence-number-syntax-units-lstm/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human La…
63. Angeliki Lazaridou and Marco Baroni. 2020. Emergent multi-agent communication in the deep learning era. arXiv preprint arXiv:2006.02419. — [arXiv:2006.02419](https://arxiv.org/abs/2006.02419) [find](https://scholar.google.com/scholar?q=Angeliki+Lazaridou+and+Marco+Baroni%2E+2020%2E+Emergent+multi%2Dagent+communication+in+the+deep+learning+era%2E+arXiv+preprint+arXiv%3A2006%2E02419%2E+%E2%80%94+%5BarXiv%3A2006%2E02419%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2006%2E02419%29)
64. Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, and Wei-Ning Hsu. 2023. Voicebox: Text-Guided multilingual universal speech generation at scale. [find](https://scholar.google.com/scholar?q=Matthew+Le%2C+Apoorv+Vyas%2C+Bowen+Shi%2C+Brian+Karrer%2C+Leda+Sari%2C+Rashel+Moritz%2C+Mary+Williamson%2C+Vimal+Manohar%2C+Yossi+Adi%2C+Jay+Mahadeokar%2C+and+Wei%2DNing+Hsu%2E+2023%2E+Voicebox%3A+Text%2DGuided+multilingual+univer)
65. Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2022. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),… [find](https://scholar.google.com/scholar?q=Katherine+Lee%2C+Daphne+Ippolito%2C+Andrew+Nystrom%2C+Chiyuan+Zhang%2C+Douglas+Eck%2C+Chris+Callison%2DBurch%2C+and+Nicholas+Carlini%2E+2022%2E+Deduplicating+training+data+makes+language+models+better%2E+In+Proceedings+o)
66. Belinda Z Li, Jane Yu, Madian Khabsa, Luke Zettlemoyer, Alon Halevy, and Jacob Andreas. 2021. Quantifying adaptability in pre-trained language models with 500 tasks. arXiv preprint arXiv:2112.03204. [find](https://scholar.google.com/scholar?q=Belinda+Z+Li%2C+Jane+Yu%2C+Madian+Khabsa%2C+Luke+Zettlemoyer%2C+Alon+Halevy%2C+and+Jacob+Andreas%2E+2021%2E+Quantifying+adaptability+in+pre%2Dtrained+language+models+with+500+tasks%2E+arXiv+preprint+arXiv%3A2112%2E03204%2E)
67. Thomas Liao, Rohan Taori, Inioluwa Deborah Raji, and Ludwig Schmidt. 2021. Are we learning yet? a meta review of evaluation failures across machine learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). [find](https://scholar.google.com/scholar?q=Thomas+Liao%2C+Rohan+Taori%2C+Inioluwa+Deborah+Raji%2C+and+Ludwig+Schmidt%2E+2021%2E+Are+we+learning+yet%3F+a+meta+review+of+evaluation+failures+across+machine+learning%2E+In+Thirty%2Dfifth+Conference+on+Neural+Infor)
68. Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, et al. 2021. [Few-shot learning with multilingual language models](https://md.research.elamur.ai/free/en/papers/lin-2021-few-shot-learning-multilingual-language/). arXiv preprint arXiv:2112.10668.
69. Tal Linzen, Grzegorz Chrupała, Yonatan Belinkov, and Dieuwke Hupkes, editors. 2019. Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. Association for Computational Linguistics, Florence, Italy. [find](https://scholar.google.com/scholar?q=Tal+Linzen%2C+Grzegorz+Chrupa%C5%82a%2C+Yonatan+Belinkov%2C+and+Dieuwke+Hupkes%2C+editors%2E+2019%2E+Proceedings+of+the+2019+ACL+Workshop+BlackboxNLP%3A+Analyzing+and+Interpreting+Neural+Networks+for+NLP%2E+Association+fo)
70. Zachary C Lipton. 2018. The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue, 16(3):31–57. [find](https://scholar.google.com/scholar?q=Zachary+C+Lipton%2E+2018%2E+The+mythos+of+model+interpretability%3A+In+machine+learning%2C+the+concept+of+interpretability+is+both+important+and+slippery%2E+Queue%2C+16%283%29%3A31%E2%80%9357%2E)
71. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023a. [Lost in the middle: How language models use long contexts](https://md.research.elamur.ai/free/en/papers/liu-2023-lost-middle-language-models-use/).
72. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. [Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing](https://md.research.elamur.ai/free/en/papers/liu-2021-pre-train-prompt-predict-systematic/). arXiv preprint arXiv:2107.13586.
73. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023b. [Gpteval: Nlg evaluation using gpt-4 with better human alignment](https://md.research.elamur.ai/free/en/papers/liu-2023-g-eval-nlg-evaluation-gpt/). arXiv preprint arXiv:2303.16634.
74. Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. 2023. Videofusion: Decomposed diffusion models for high-quality video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). [find](https://scholar.google.com/scholar?q=Zhengxiong+Luo%2C+Dayou+Chen%2C+Yingya+Zhang%2C+Yan+Huang%2C+Liang+Wang%2C+Yujun+Shen%2C+Deli+Zhao%2C+Jingren+Zhou%2C+and+Tieniu+Tan%2E+2023%2E+Videofusion%3A+Decomposed+diffusion+models+for+high%2Dquality+video+generation%2E+)
75. Christopher D Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy. 2020. Emergent linguistic structure in artificial neural networks trained by self-supervision. Proceedings of the National Academy of Sciences, 117(48):30046–30054. [find](https://scholar.google.com/scholar?q=Christopher+D+Manning%2C+Kevin+Clark%2C+John+Hewitt%2C+Urvashi+Khandelwal%2C+and+Omer+Levy%2E+2020%2E+Emergent+linguistic+structure+in+artificial+neural+networks+trained+by+self%2Dsupervision%2E+Proceedings+of+the+Na)
76. R Thomas McCoy, Paul Smolensky, Tal Linzen, Jianfeng Gao, and Asli Celikyilmaz. 2023. [How much do language models copy from their training data? evaluating linguistic novelty in text generation using raven](https://md.research.elamur.ai/free/en/papers/mccoy-2023-how-much-do-language-models/). Transactions of the Association for Computational Linguistics, 11:652–670.
77. Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. [Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference](https://md.research.elamur.ai/free/en/papers/mccoy-2019-right-wrong-reasons-diagnosing-syntactic/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448, Florence, Italy. Association for Computational Ling…
78. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. [Distributed representations of words and phrases and their compositionality](https://md.research.elamur.ai/free/en/papers/mikolov-2013-distributed-representations-words-phrases-their/). Advances in neural information processing systems, 26.
79. Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022. Cross-task generalization via natural language crowdsourcing instructions. In 60th Annual Meeting of the Association for Computational Linguistics, ACL 2022, pages 3470–3487. Association for Computational Linguistics (ACL)… [find](https://scholar.google.com/scholar?q=Swaroop+Mishra%2C+Daniel+Khashabi%2C+Chitta+Baral%2C+and+Hannaneh+Hajishirzi%2E+2022%2E+Cross%2Dtask+generalization+via+natural+language+crowdsourcing+instructions%2E+In+60th+Annual+Meeting+of+the+Association+for+C)
80. Saurabh Mittal, Saikou Diallo, and Andreas Tolk. 2018. Emergent behavior in complex systems engineering: A modeling and simulation approach. John Wiley & Sons. [find](https://scholar.google.com/scholar?q=Saurabh+Mittal%2C+Saikou+Diallo%2C+and+Andreas+Tolk%2E+2018%2E+Emergent+behavior+in+complex+systems+engineering%3A+A+modeling+and+simulation+approach%2E+John+Wiley+%26+Sons%2E)
81. Masahiro Mori. 2012. The uncanny valley: The original essay by masahiro mori. [https://spectrum.ieee.org/the-uncanny-valley](https://spectrum.ieee.org/the-uncanny-valley). Accessed: 2023-7-6. [↗](https://spectrum.ieee.org/the-uncanny-valley)
82. Miguel Morin and Matthew Willetts. 2020. Non-determinism in tensorflow resnets. arXiv preprint arXiv:2001.11396. [find](https://scholar.google.com/scholar?q=Miguel+Morin+and+Matthew+Willetts%2E+2020%2E+Non%2Ddeterminism+in+tensorflow+resnets%2E+arXiv+preprint+arXiv%3A2001%2E11396%2E)
83. Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023. [Orca: Progressive learning from complex explanation traces of GPT-4](https://md.research.elamur.ai/free/en/papers/mukherjee-2023-orca-progressive-learning-complex-explanation/).
84. Mathias Müller, Annette Rios Gonzales, and Rico Sennrich. 2020. [Domain robustness in neural machine translation](https://md.research.elamur.ai/free/en/papers/muller-2020-domain-robustness-neural-machine-translation/). In Proceedings of the 14th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track), pages 151–164.
85. MEJ Newman. 2011. Resource letter cs–1: Complex systems. American Journal of Physics, 79(8):800– 810. [find](https://scholar.google.com/scholar?q=MEJ+Newman%2E+2011%2E+Resource+letter+cs%E2%80%931%3A+Complex+systems%2E+American+Journal+of+Physics%2C+79%288%29%3A800%E2%80%93+810%2E)
86. Chris Olah. 2015. Understanding LSTM networks. [https://colah.github.io/posts/](https://colah.github.io/posts/) Accessed: 2015-08-Understanding-LSTMs/. 2023-7-7. [↗](https://colah.github.io/posts/)
87. Chris Olah. 2022. Mechanistic interpretability, variables, and the importance of interpretable bases. [https://transformer-circuits.pub/2022/mech-interp-essay/index.html](https://transformer-circuits.pub/2022/mech-interp-essay/index.html). Accessed: 2023-7-8. [↗](https://transformer-circuits.pub/2022/mech-interp-essay/index.html)
88. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amode… — [In-context Learning and Induction Heads](https://md.research.elamur.ai/free/en/papers/olsson-2022-context-learning-induction-heads/)
89. OpenAI. 2023. GPT-4 API general availability and deprecation of older models in the completions API. [https://openai.com/blog/](https://openai.com/blog/) [↗](https://openai.com/blog/)
90. Accessed: the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5493–5505. [find](https://scholar.google.com/scholar?q=Accessed%3A+the+Association+for+Computational+Linguistics+and+the+11th+International+Joint+Conference+on+Natural+Language+Processing+%28Volume+1%3A+Long+Papers%29%2C+pages+5493%E2%80%935505%2E)
91. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35:27730–27744.
92. Wolfgang Prinz. 2006. Messung kontra augenschein. Psychologische Rundschau, 57(2):106–111. gpt-4-api-general-availability. 2023-7-6. [find](https://scholar.google.com/scholar?q=Wolfgang+Prinz%2E+2006%2E+Messung+kontra+augenschein%2E+Psychologische+Rundschau%2C+57%282%29%3A106%E2%80%93111%2E+gpt%2D4%2Dapi%2Dgeneral%2Davailability%2E+2023%2D7%2D6%2E)
93. OpenAI. 2023. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/).
94. Letitia Parcalabescu, Albert Gatt, Anette Frank, and Iacer Calixto. 2021. Seeing past words: Testing the cross-modal capabilities of pretrained v&l models on counting tasks. In Proceedings of the 1st Workshop on Multimodal Semantic Representations (MMSR), pages 32–44. [find](https://scholar.google.com/scholar?q=Letitia+Parcalabescu%2C+Albert+Gatt%2C+Anette+Frank%2C+and+Iacer+Calixto%2E+2021%2E+Seeing+past+words%3A+Testing+the+cross%2Dmodal+capabilities+of+pretrained+v%26l+models+on+counting+tasks%2E+In+Proceedings+of+the+1st+)
95. Roma Patel and Ellie Pavlick. 2021. “was it “stated” or was it “claimed”?: How linguistic bias affects generative language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10080–10095, Online and Punta Cana, Dominican Republic. Association for… [find](https://scholar.google.com/scholar?q=Roma+Patel+and+Ellie+Pavlick%2E+2021%2E+%E2%80%9Cwas+it+%E2%80%9Cstated%E2%80%9D+or+was+it+%E2%80%9Cclaimed%E2%80%9D%3F%3A+How+linguistic+bias+affects+generative+language+models%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural)
96. Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. 2020. [Bpe-dropout: Simple and effective subword regularization](https://md.research.elamur.ai/free/en/papers/provilkov-2020-bpe-dropout-simple-effective-subword/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1882–1892.
97. Alec Radford, Rafal Jozefowicz, and Ilya Sutskever. 2017. [Learning to generate reviews and discovering sentiment](https://md.research.elamur.ai/free/en/papers/radford-2017-learning-generate-reviews-discovering-sentiment/). arXiv preprint arXiv:1704.01444.
98. Alec Radford, Jeffrey Wu, Dario Amodei, Daniela Amodei, Jack Clark, Miles Brundage, and Ilya Sutskever. 2019a. Better language models and their implications. OpenAI Blog, 1:2. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Dario+Amodei%2C+Daniela+Amodei%2C+Jack+Clark%2C+Miles+Brundage%2C+and+Ilya+Sutskever%2E+2019a%2E+Better+language+models+and+their+implications%2E+OpenAI+Blog%2C+1%3A2%2E)
99. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019b. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+2019b%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E)
100. Alex Perry. 2023. OpenAI updates GPT-4 with Partha Pratim Ray. 2023. Chatgpt: A comprehensive new features. [https://mashable.com/article/](https://mashable.com/article/) openai-chatgpt-gpt-4-function-calling-update. review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet of Accessed… [↗](https://mashable.com/article/)
101. Things and Cyber-Physical Systems. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, Gardner, Christopher Clark, Kenton Lee, and Luke and Sameer Singh. 2020. Beyond accuracy: Behav-Zettlemoyer. 2018. Deep contextualized word repre-ioral testing o… [find](https://scholar.google.com/scholar?q=Things+and+Cyber%2DPhysical+Systems%2E+Matthew+E%2E+Peters%2C+Mark+Neumann%2C+Mohit+Iyyer%2C+Matt+Marco+Tulio+Ribeiro%2C+Tongshuang+Wu%2C+Carlos+Guestrin%2C+Gardner%2C+Christopher+Clark%2C+Kenton+Lee%2C+and+Luke+and+Sameer+S)
102. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. [Language models as knowledge bases?](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/) In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural…
103. Sundar Pichai. 2023. An important next step on our AI journey. [https://blog.google/technology/ai/](https://blog.google/technology/ai/) bard-google-ai-search-updates/. Accessed: 2023-7-6. [↗](https://blog.google/technology/ai/)
104. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. [Hypothesis only baselines in natural language inference](https://md.research.elamur.ai/free/en/papers/poliak-2018-hypothesis-only-baselines-natural-language/). In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 180– 191.
105. Ofir Press, Noah A Smith, and Mike Lewis. 2021. Shortformer: Better language modeling using shorter inputs. In Proceedings of the 59th Annual Meeting of Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. [Object hallucination in image captioning](https://md.research.elamur.ai/free/en/papers/rohrbach-2018-object-hallucination-image-captioning/). In Proceedings o…
106. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2021. [High-resolution image synthesis with latent diffusion models](https://md.research.elamur.ai/free/en/papers/rombach-2021-high-resolution-image-synthesis-latent/).
107. Robert Rosenthal. 1976. Experimenter effects in behavioral research. [find](https://scholar.google.com/scholar?q=Robert+Rosenthal%2E+1976%2E+Experimenter+effects+in+behavioral+research%2E)
108. William Rudman and Carsten Eickhoff. 2023. Stable anisotropic regularization. arXiv preprint arXiv:2305.19358. — [arXiv:2305.19358](https://arxiv.org/abs/2305.19358) [find](https://scholar.google.com/scholar?q=William+Rudman+and+Carsten+Eickhoff%2E+2023%2E+Stable+anisotropic+regularization%2E+arXiv+preprint+arXiv%3A2305%2E19358%2E+%E2%80%94+%5BarXiv%3A2305%2E19358%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2305%2E19358%29)
109. John Schulman, Barret Zoph, Christina Kim, Jacob Hilton, Jacob Menick, Jiayi Weng, Juan Felipe Ceron Uribe, Liam Fedus, Luke Metz, Michael Pokorny, Rapha Gontijo Lopes, Shengjia Zhao, Arun Vijayvergiya, Eric Sigler, Adam Perelman, Chelsea Voss, Mike Heaton, Joel Parish, Dave Cummings, Rajeev Nayak,… [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Barret+Zoph%2C+Christina+Kim%2C+Jacob+Hilton%2C+Jacob+Menick%2C+Jiayi+Weng%2C+Juan+Felipe+Ceron+Uribe%2C+Liam+Fedus%2C+Luke+Metz%2C+Michael+Pokorny%2C+Rapha+Gontijo+Lopes%2C+Shengjia+Zhao%2C+Arun+Vijayvergiy)
110. Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073– 1083. [find](https://scholar.google.com/scholar?q=Abigail+See%2C+Peter+J+Liu%2C+and+Christopher+D+Manning%2E+2017%2E+Get+to+the+point%3A+Summarization+with+pointer%2Dgenerator+networks%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computation)
111. Abigail See, Aneesh Pappu, Rohun Saxena, Akhila Yerukola, and Christopher D Manning. 2019. [Do massively pretrained language models make better storytellers?](https://md.research.elamur.ai/free/en/papers/see-2019-massively-pretrained-language-models-make/) In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 843–861.
112. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. [Neural machine translation of rare words with subword units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725.
113. Xing Shi, Yijun Xiao, and Kevin Knight. 2020. Why neural machine translation prefers empty outputs. arXiv preprint arXiv:2012.13454. — [arXiv:2012.13454](https://arxiv.org/abs/2012.13454) [find](https://scholar.google.com/scholar?q=Xing+Shi%2C+Yijun+Xiao%2C+and+Kevin+Knight%2E+2020%2E+Why+neural+machine+translation+prefers+empty+outputs%2E+arXiv+preprint+arXiv%3A2012%2E13454%2E+%E2%80%94+%5BarXiv%3A2012%2E13454%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2012%2E13454%29)
114. Matt G Southern. 2023. [no title]. [https://www.searchenginejournal.com/](https://www.searchenginejournal.com/) Accessed: openai-chatgpt-update/476116/. 2023-7-6. company Stability AI. 2023. StableLM: StableLM: Stability AI language models. [↗](https://www.searchenginejournal.com/)
115. Felix Stahlberg and Bill Byrne. 2019. [On nmt search errors and model errors: Cat got your tongue?](https://md.research.elamur.ai/free/en/papers/stahlberg-2019-nmt-search-errors-model-errors/) In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3356–3362.
116. Felix Stahlberg, Ilia Kulikov, and Shankar Kumar. 2022. Uncertainty determines the adequacy of the mode and the tractability of decoding in sequence-to-sequence models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8634–8645… [find](https://scholar.google.com/scholar?q=Felix+Stahlberg%2C+Ilia+Kulikov%2C+and+Shankar+Kumar%2E+2022%2E+Uncertainty+determines+the+adequacy+of+the+mode+and+the+tractability+of+decoding+in+sequence%2Dto%2Dsequence+models%2E+In+Proceedings+of+the+60th+Annu)
117. Shane Steinert-Threlkeld, Xuhui Zhou, Zeyu Liu, and C M Downey. 2022. Emergent communication fine-tuning (EC-FT) for pretrained language models. [find](https://scholar.google.com/scholar?q=Shane+Steinert%2DThrelkeld%2C+Xuhui+Zhou%2C+Zeyu+Liu%2C+and+C+M+Downey%2E+2022%2E+Emergent+communication+fine%2Dtuning+%28EC%2DFT%29+for+pretrained+language+models%2E)
118. ICLR 2022 EmeCom Workshop. [find](https://scholar.google.com/scholar?q=ICLR+2022+EmeCom+Workshop%2E)
119. Simeng Sun, Kalpesh Krishna, Andrew Mattarella-Micke, and Mohit Iyyer. 2021. [Do long-range language models actually use long-range context?](https://md.research.elamur.ai/free/en/papers/sun-2021-do-long-range-language-models/) In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 807– 822.
120. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca). [↗](https://github.com/tatsu-lab/stanford_alpaca)
121. Ryan Teehan, Miruna Clinciu, Oleg Serikov, Eliza Szczechla, Natasha Seelam, Shachar Mirkin, and Aaron Gokaslan. 2022. Emergent structures and training dynamics in large language models. In Proceedings of BigScience Episode\\\# 5–Workshop on Challenges & Perspectives in Creating Large Language Models,… [find](https://scholar.google.com/scholar?q=Ryan+Teehan%2C+Miruna+Clinciu%2C+Oleg+Serikov%2C+Eliza+Szczechla%2C+Natasha+Seelam%2C+Shachar+Mirkin%2C+and+Aaron+Gokaslan%2E+2022%2E+Emergent+structures+and+training+dynamics+in+large+language+models%2E+In+Proceedings)
122. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. [Bert rediscovers the classical nlp pipeline](https://md.research.elamur.ai/free/en/papers/tenney-2019-bert-rediscovers-classical-nlp-pipeline/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593–4601.
123. Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. 2022. [Memorization without overfitting: Analyzing the training dynamics of large language models](https://md.research.elamur.ai/free/en/papers/tirumala-2022-memorization-overfitting-analyzing-training-dynamics/). Advances in Neural Information Processing Systems, 35:38274–38290.
124. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. [LLaMA: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/).
125. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30.
126. The Vicuna Team. Vicuna: An Open-Source chatbot impressing GPT-4 with 90%* ChatGPT quality. [https://lmsys.org/blog/2023-03-30-vicuna/](https://lmsys.org/blog/2023-03-30-vicuna/). Accessed: 2023-7-6. [↗](https://lmsys.org/blog/2023-03-30-vicuna/)
127. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. [Glue: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355.
128. Chaojun Wang and Rico Sennrich. 2020. [On exposure bias, hallucination and domain shift in neural machine translation](https://md.research.elamur.ai/free/en/papers/wang-2020-exposure-bias-hallucination-domain-shift/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3544–3552.
129. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. [Defending against neural fake news](https://md.research.elamur.ai/free/en/papers/zellers-2019-defending-against-neural-fake-news/). Advances in neural information processing systems, 32.
130. Lingxiao Wang, Jing Huang, Kevin Huang, Ziniu Hu, Guangtao Wang, and Quanquan Gu. 2020. Improving neural language generation with spectrum control. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Lingxiao+Wang%2C+Jing+Huang%2C+Kevin+Huang%2C+Ziniu+Hu%2C+Guangtao+Wang%2C+and+Quanquan+Gu%2E+2020%2E+Improving+neural+language+generation+with+spectrum+control%2E+In+International+Conference+on+Learning+Representati)
131. Chaoning Zhang, Chenshuang Zhang, Chenghao Li, Yu Qiao, Sheng Zheng, Sumit Kumar Dam, Mengchun Zhang, Jung Uk Kim, Seong Tae Kim, Jinwoo Choi, et al. 2023. One small step for generative ai, one giant leap for agi: A complete survey on chatgpt in aigc era. arXiv preprint arXiv:2304.06488. — [arXiv:2304.06488](https://arxiv.org/abs/2304.06488) [find](https://scholar.google.com/scholar?q=Chaoning+Zhang%2C+Chenshuang+Zhang%2C+Chenghao+Li%2C+Yu+Qiao%2C+Sheng+Zheng%2C+Sumit+Kumar+Dam%2C+Mengchun+Zhang%2C+Jung+Uk+Kim%2C+Seong+Tae+Kim%2C+Jinwoo+Choi%2C+et+al%2E+2023%2E+One+small+step+for+generative+ai%2C+one+giant+)
132. Alex Warstadt, Leshem Choshen, Aaron Mueller, Adina Williams, Ethan Wilcox, and Chengxu Zhuang. 2023. Call for papers–the babylm challenge: Sample-efficient pretraining on a developmentally plausible corpus. arXiv preprint arXiv:2301.11796. [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Leshem+Choshen%2C+Aaron+Mueller%2C+Adina+Williams%2C+Ethan+Wilcox%2C+and+Chengxu+Zhuang%2E+2023%2E+Call+for+papers%E2%80%93the+babylm+challenge%3A+Sample%2Defficient+pretraining+on+a+developmentally+plausible+)
133. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. [Emergent abilities of large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/). arXiv preprint arX…
134. Gail Weiss, Yoav Goldberg, and Eran Yahav. 2021. Thinking like transformers. In International Conference on Machine Learning, pages 11080–11090. PMLR. [find](https://scholar.google.com/scholar?q=Gail+Weiss%2C+Yoav+Goldberg%2C+and+Eran+Yahav%2E+2021%2E+Thinking+like+transformers%2E+In+International+Conference+on+Machine+Learning%2C+pages+11080%E2%80%9311090%2E+PMLR%2E)
135. Andrew Wilson. How to jailbreak ChatGPT to unlock its full potential. [https://approachableai.com/](https://approachableai.com/) how-to-jailbreak-chatgpt/. Accessed: 2023-7-6. [↗](https://approachableai.com/)
136. D H Wolpert and W G Macready. 1997. No free lunch theorems for optimization. IEEE Trans. Evol. Comput., 1(1):67–82. [find](https://scholar.google.com/scholar?q=D+H+Wolpert+and+W+G+Macready%2E+1997%2E+No+free+lunch+theorems+for+optimization%2E+IEEE+Trans%2E+Evol%2E+Comput%2E%2C+1%281%29%3A67%E2%80%9382%2E)
137. Zhuohan Xie, Trevor Cohn, and Jey Han Lau. 2023. Can very large pretrained language models learn storytelling with a few examples? [find](https://scholar.google.com/scholar?q=Zhuohan+Xie%2C+Trevor+Cohn%2C+and+Jey+Han+Lau%2E+2023%2E+Can+very+large+pretrained+language+models+learn+storytelling+with+a+few+examples%3F)
138. Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Bing Yin, and Xia Hu. 2023. [Harnessing the power of llms in practice: A survey on chatgpt and beyond](https://md.research.elamur.ai/free/en/papers/yang-2023-harnessing-power-llms-practice-survey/). arXiv preprint arXiv:2304.13712.
139. Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. 2021. [CrossFit: A few-shot learning challenge for cross-task generalization in NLP](https://md.research.elamur.ai/free/en/papers/ye-2021-crossfit-few-shot-learning-challenge/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7163–7189, Online and Punta Cana, Dominican Republic. Association for Compu…
140. Lili Yu, Dániel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. 2023. Megabyte: Predicting million-byte sequences with multiscale transformers. arXiv preprint arXiv:2305.07185. [find](https://scholar.google.com/scholar?q=Lili+Yu%2C+D%C3%A1niel+Simig%2C+Colin+Flaherty%2C+Armen+Aghajanyan%2C+Luke+Zettlemoyer%2C+and+Mike+Lewis%2E+2023%2E+Megabyte%3A+Predicting+million%2Dbyte+sequences+with+multiscale+transformers%2E+arXiv+preprint+arXiv%3A2305%2E071)
141. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. [Judging LLM-as-a-judge with MT-Bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/).
142. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2023. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206. [find](https://scholar.google.com/scholar?q=Chunting+Zhou%2C+Pengfei+Liu%2C+Puxin+Xu%2C+Srini+Iyer%2C+Jiao+Sun%2C+Yuning+Mao%2C+Xuezhe+Ma%2C+Avia+Efrat%2C+Ping+Yu%2C+Lili+Yu%2C+et+al%2E+2023%2E+Lima%3A+Less+is+more+for+alignment%2E+arXiv+preprint+arXiv%3A2305%2E11206%2E)

*Updated 3 November 2026*