---
title: "GPTEval: A Survey on Assessments of ChatGPT and GPT-4"
description: "Many separate studies have tested ChatGPT and GPT-4 on individual tasks or… 77.4% accuracy. 130 references, cited by 38 papers here."
authors: ["Rui Mao", "Guanyi Chen", "Xulang Zhang", "Frank Guerin", "Erik Cambria"]
year: 2024
venue: "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)"
source: "https://aclanthology.org/2024.lrec-main.693/"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/mao-2024-gpteval-survey-assessments-chatgpt-gpt/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mao-2024-gpteval-survey-assessments-chatgpt-gpt/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / GPTEval: A Survey on Assessments of ChatGPT and GPT-4

# GPTEval: A Survey on Assessments of ChatGPT and GPT-4

**Authors:** [Rui Mao](https://md.research.elamur.ai/free/en/authors/rui-mao/), [Guanyi Chen](https://md.research.elamur.ai/free/en/authors/guanyi-chen/), [Xulang Zhang](https://md.research.elamur.ai/free/en/authors/xulang-zhang/), [Frank Guerin](https://md.research.elamur.ai/free/en/authors/frank-guerin/) and [Erik Cambria](https://md.research.elamur.ai/free/en/authors/erik-cambria/) · **Year:** 2024 · **Venue:** [Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)](https://md.research.elamur.ai/free/en/venues/lrec/) ([LREC 2024](https://md.research.elamur.ai/free/en/venues/lrec/2024/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.lrec-main.693/) · [PDF](https://aclanthology.org/2024.lrec-main.693.pdf) · [OpenAlex](https://openalex.org/W7131795684)

**Citations:** 37 citations (OpenAlex, 2026-10-14) · [38](https://md.research.elamur.ai/free/en/papers/mao-2024-gpteval-survey-assessments-chatgpt-gpt/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary

Many separate studies have tested ChatGPT and GPT-4 on individual tasks or disciplines, but no overview had combined their findings. This survey gathers quantitative evaluations of both models and groups them by area:

- classic NLP tasks: dialogue, generation, translation, summarization, affective computing, information extraction, and use as an annotator or evaluator
- multilingual performance and reasoning
- knowledge in fields such as mathematics, physics, chemistry, law, economics, finance and medicine
- ethical concerns

The authors set the reported results against expert and fine-tuned baselines and review how the evaluations were done. The models understand and produce language very well and can explain their outputs. However, they often fall behind specialized fine-tuned systems on domain-heavy tasks, make mistakes in multi-step reasoning, and perform clearly worse on low-resource languages.

The authors also argue that current evaluations are fragile. Scores change with the prompt, baselines are often tested on their own in-domain data, and leakage of benchmarks into GPT training data cannot be excluded. For multilingual and low-resource work, the survey shows where GPT models are still unreliable.

## Key points

- Lai et al. tested ChatGPT across 36 languages. Task-specific fine-tuned models beat it on most tasks except PoS tagging, and English prompts improved its results even on non-English tasks.
- On the SuperCLUE Chinese benchmark, GPT-4 scored 76.67 and ChatGPT 66.18, against a human score of 96.50. Both models were weak on classical Chinese, poetry and couplets.
- In affective computing, a RoBERTa-embedding model outperformed ChatGPT and GPT-4 on 22 of 34 tasks.
- GPT-4 reached 77.4% accuracy on personally identifiable information detection, compared with 40.8% for the previous SOTA model.
- Of 1008 jokes generated by ChatGPT, about 90% repeated one of the same 25 jokes. In contrast, it ranked in the top 9% of students in microeconomics and the top 1% in macroeconomics.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Rui Mao, Guanyi Chen, Xulang Zhang, Frank Guerin and Erik Cambria. 2024. GPTEval: A Survey on Assessments of ChatGPT and GPT-4. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{mao2024gpteval,
  title = {{GPTEval: A Survey on Assessments of ChatGPT and GPT-4}},
  author = {Rui Mao, Guanyi Chen, Xulang Zhang, Frank Guerin and Erik Cambria},
  booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
  year = {2024},
  doi = {10.63317/3dhqzf3tvt2z},
  url = {https://doi.org/10.63317/3dhqzf3tvt2z}
}
```

- [← previous Enhancing Noise Robustness of Retrieval-Augmented Language Models with Adaptive Adversarial Training](https://md.research.elamur.ai/free/en/papers/fang-2024-enhancing-noise-robustness-retrieval-augmented/)
- [next → PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilities](https://md.research.elamur.ai/free/en/papers/sravanthi-2024-pub-pragmatics-understanding-benchmark-assessing/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

130 references, 43 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rachith Aiyappa, Jisun An, Haewoon Kwak, and Yong-Yeol Ahn. 2023. [Can we trust the evaluation on ChatGPT?](https://md.research.elamur.ai/free/en/papers/aiyappa-2023-can-we-trust-evaluation-chatgpt/) arXiv preprint arXiv:2303.12767.
2. Mostafa M Amin, Rui Mao, Erik Cambria, and Björn W Schuller. 2023. A wide evaluation of ChatGPT on affective computing tasks. arXiv preprint arXiv:2308.13911. — [arXiv:2308.13911](https://arxiv.org/abs/2308.13911) [find](https://scholar.google.com/scholar?q=Mostafa+M+Amin%2C+Rui+Mao%2C+Erik+Cambria%2C+and+Bj%C3%B6rn+W+Schuller%2E+2023%2E+A+wide+evaluation+of+ChatGPT+on+affective+computing+tasks%2E+arXiv+preprint+arXiv%3A2308%2E13911%2E+%E2%80%94+%5BarXiv%3A2308%2E13911%5D%28https%3A%2F%2Farxiv%2Eorg%2Fab)
3. Fares Antaki, Samir Touma, Daniel Milad, Jonathan El-Khoury, and Renaud Duval. 2023. Evaluating the performance of ChatGPT in ophthalmology: An analysis of its successes and shortcomings. Ophthalmology Science, page 100324. [find](https://scholar.google.com/scholar?q=Fares+Antaki%2C+Samir+Touma%2C+Daniel+Milad%2C+Jonathan+El%2DKhoury%2C+and+Renaud+Duval%2E+2023%2E+Evaluating+the+performance+of+ChatGPT+in+ophthalmology%3A+An+analysis+of+its+successes+and+shortcomings%2E+Ophthalmolog)
4. Sai Anirudh Athaluri, Sandeep Varma Manthena, VSR Krishna Manoj Kesapragada, Vineel Yarlagadda, Tirth Dave, and Rama Tulasi Siri Duddumpudi. 2023. Exploring the boundaries of reality: Investigating the phenomenon of artificial intelligence hallucination in scientific writing through ChatGPT referenc… [find](https://scholar.google.com/scholar?q=Sai+Anirudh+Athaluri%2C+Sandeep+Varma+Manthena%2C+VSR+Krishna+Manoj+Kesapragada%2C+Vineel+Yarlagadda%2C+Tirth+Dave%2C+and+Rama+Tulasi+Siri+Duddumpudi%2E+2023%2E+Exploring+the+boundaries+of+reality%3A+Investigating+th)
5. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. [A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity](https://md.research.elamur.ai/free/en/papers/bang-2023-multitask-multilingual-multimodal-evaluation-chatgpt/). arXiv preprint arXiv:2302.04023.
6. Anja Belz, Craig Thomson, and Ehud Reiter. 2023. Missing information, unresponsive authors, experimental flaws: The impossibility of assessing the reproducibility of previous human evaluations in NLP. In The Fourth Workshop on Insights from Negative Results in NLP, pages 1–10. [find](https://scholar.google.com/scholar?q=Anja+Belz%2C+Craig+Thomson%2C+and+Ehud+Reiter%2E+2023%2E+Missing+information%2C+unresponsive+authors%2C+experimental+flaws%3A+The+impossibility+of+assessing+the+reproducibility+of+previous+human+evaluations+in+NLP%2E)
7. Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2013. [Representation learning: A review and new perspectives](https://md.research.elamur.ai/free/en/papers/bengio-2012-representation-learning-review-new-perspectives/). IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798–1828.
8. Manik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, Pengfei Liu, and Graham Neubig. 2020. [find](https://scholar.google.com/scholar?q=Manik+Bhandari%2C+Pranav+Narayan+Gour%2C+Atabak+Ashfaq%2C+Pengfei+Liu%2C+and+Graham+Neubig%2E+2020%2E)
9. [Re-evaluating evaluation in text summarization](https://md.research.elamur.ai/free/en/papers/bhandari-2020-re-evaluating-evaluation-text-summarization/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9347–9359.
10. Sebastian Bordt and Ulrike von Luxburg. 2023. ChatGPT participates in a computer science exam. arXiv preprint arXiv:2303.09461. — [arXiv:2303.09461](https://arxiv.org/abs/2303.09461) [find](https://scholar.google.com/scholar?q=Sebastian+Bordt+and+Ulrike+von+Luxburg%2E+2023%2E+ChatGPT+participates+in+a+computer+science+exam%2E+arXiv+preprint+arXiv%3A2303%2E09461%2E+%E2%80%94+%5BarXiv%3A2303%2E09461%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2303%2E09461%29)
11. Ali Borji. 2023. [A categorical archive of ChatGPT failures](https://md.research.elamur.ai/free/en/papers/borji-2023-categorical-archive-chatgpt-failures/). arXiv preprint arXiv:2302.03494.
12. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. [Sparks of artificial general intelligence: Early experiments with GPT-4](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/). arXiv prepr…
13. Alex Cabrera and Graham Neubig. 2023. Zeno chatbot report. Accessed: 18/05/2023. [find](https://scholar.google.com/scholar?q=Alex+Cabrera+and+Graham+Neubig%2E+2023%2E+Zeno+chatbot+report%2E+Accessed%3A+18%2F05%2F2023%2E)
14. Erik Cambria, Qian Liu, Sergio Decherchi, Frank Xing, and Kenneth Kwok. 2022. SenticNet 7: A commonsense-based neurosymbolic AI framework for explainable sentiment analysis. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 3829–3839. [find](https://scholar.google.com/scholar?q=Erik+Cambria%2C+Qian+Liu%2C+Sergio+Decherchi%2C+Frank+Xing%2C+and+Kenneth+Kwok%2E+2022%2E+SenticNet+7%3A+A+commonsense%2Dbased+neurosymbolic+AI+framework+for+explainable+sentiment+analysis%2E+In+Proceedings+of+the+Thir)
15. Erik Cambria, Rui Mao, Melvin Chen, Zhaoxia Wang, and Seng-Beng Ho. 2023. Seven pillars for the future of Artificial Intelligence. IEEE Intelligent Systems, 38(6):62–69. [find](https://scholar.google.com/scholar?q=Erik+Cambria%2C+Rui+Mao%2C+Melvin+Chen%2C+Zhaoxia+Wang%2C+and+Seng%2DBeng+Ho%2E+2023%2E+Seven+pillars+for+the+future+of+Artificial+Intelligence%2E+IEEE+Intelligent+Systems%2C+38%286%29%3A62%E2%80%9369%2E)
16. Erik Cambria and Bebo White. 2014. Jumping NLP curves: A review of natural language processing research. IEEE Computational Intelligence Magazine, 9(2):48–57. [find](https://scholar.google.com/scholar?q=Erik+Cambria+and+Bebo+White%2E+2014%2E+Jumping+NLP+curves%3A+A+review+of+natural+language+processing+research%2E+IEEE+Computational+Intelligence+Magazine%2C+9%282%29%3A48%E2%80%9357%2E)
17. Inês Carvalho and Stanislav Ivanov. 2023. Chat-GPT for tourism: Applications, benefits and risks. Tourism Review. [find](https://scholar.google.com/scholar?q=In%C3%AAs+Carvalho+and+Stanislav+Ivanov%2E+2023%2E+Chat%2DGPT+for+tourism%3A+Applications%2C+benefits+and+risks%2E+Tourism+Review%2E)
18. Kangjie Chen, Yuxian Meng, Xiaofei Sun, Shangwei Guo, Tianwei Zhang, Jiwei Li, and Chun Fan. 2021a. BadPre: Task-agnostic backdoor attacks to pre-trained NLP foundation models. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Kangjie+Chen%2C+Yuxian+Meng%2C+Xiaofei+Sun%2C+Shangwei+Guo%2C+Tianwei+Zhang%2C+Jiwei+Li%2C+and+Chun+Fan%2E+2021a%2E+BadPre%3A+Task%2Dagnostic+backdoor+attacks+to+pre%2Dtrained+NLP+foundation+models%2E+In+International+Confer)
19. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021b. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374.
20. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. [find](https://scholar.google.com/scholar?q=Wei%2DLin+Chiang%2C+Zhuohan+Li%2C+Zi+Lin%2C+Ying+Sheng%2C+Zhanghao+Wu%2C+Hao+Zhang%2C+Lianmin+Zheng%2C+Siyuan+Zhuang%2C+Yonghao+Zhuang%2C+Joseph+E%2E+Gonzalez%2C+Ion+Stoica%2C+and+Eric+P%2E+Xing%2E+2023%2E)
21. Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality. Accessed: 16/05/2023. [find](https://scholar.google.com/scholar?q=Vicuna%3A+An+open%2Dsource+chatbot+impressing+GPT%2D4+with+90%25%2A+ChatGPT+quality%2E+Accessed%3A+16%2F05%2F2023%2E)
22. Jonathan H Choi, Kristin E Hickman, Amy Monahan, and Daniel Schwarcz. 2023. ChatGPT goes to law school. Journal of Legal Education (Forthcoming). [find](https://scholar.google.com/scholar?q=Jonathan+H+Choi%2C+Kristin+E+Hickman%2C+Amy+Monahan%2C+and+Daniel+Schwarcz%2E+2023%2E+ChatGPT+goes+to+law+school%2E+Journal+of+Legal+Education+%28Forthcoming%29%2E)
23. Haoran Chu and Sixiao Liu. 2023. Can AI tell good stories? Narrative transportation and persuasion with ChatGPT. PsyArXiv. [find](https://scholar.google.com/scholar?q=Haoran+Chu+and+Sixiao+Liu%2E+2023%2E+Can+AI+tell+good+stories%3F+Narrative+transportation+and+persuasion+with+ChatGPT%2E+PsyArXiv%2E)
24. Ted M Clark. 2023. Investigating the use of an artificial intelligence chatbot with general chemistry exam questions. Journal of Chemical Education. [find](https://scholar.google.com/scholar?q=Ted+M+Clark%2E+2023%2E+Investigating+the+use+of+an+artificial+intelligence+chatbot+with+general+chemistry+exam+questions%2E+Journal+of+Chemical+Education%2E)
25. Katherine M. Collins, Albert Qiaochu Jiang, Simon Frieder, Li Siang Wong, Miri Zilka, Umang Bhatt, Thomas Lukasiewicz, Yuhuai Wu, Joshua B. Tenenbaum, William Hart, Timothy Gowers, Wen-Ding Li, Adrian Weller, and Mateja Jamnik. 2023. Evaluating language models for mathematics through interactions. a… [find](https://scholar.google.com/scholar?q=Katherine+M%2E+Collins%2C+Albert+Qiaochu+Jiang%2C+Simon+Frieder%2C+Li+Siang+Wong%2C+Miri+Zilka%2C+Umang+Bhatt%2C+Thomas+Lukasiewicz%2C+Yuhuai+Wu%2C+Joshua+B%2E+Tenenbaum%2C+William+Hart%2C+Timothy+Gowers%2C+Wen%2DDing+Li%2C+Adrian)
26. Merten Nikolay Dahlkemper, Simon Zacharias Lahme, and Pascal Klein. 2023. How do physics students evaluate artificial intelligence responses on comprehension questions? A study on the perceived scientific accuracy and linguistic quality. arXiv preprint arXiv:2304.05906. — [arXiv:2304.05906](https://arxiv.org/abs/2304.05906) [find](https://scholar.google.com/scholar?q=Merten+Nikolay+Dahlkemper%2C+Simon+Zacharias+Lahme%2C+and+Pascal+Klein%2E+2023%2E+How+do+physics+students+evaluate+artificial+intelligence+responses+on+comprehension+questions%3F+A+study+on+the+perceived+scient)
27. Ernest Davis. 2023. Benchmarks for automated commonsense reasoning: A survey. arXiv preprint arXiv:2302.04752. [find](https://scholar.google.com/scholar?q=Ernest+Davis%2E+2023%2E+Benchmarks+for+automated+commonsense+reasoning%3A+A+survey%2E+arXiv+preprint+arXiv%3A2302%2E04752%2E)
28. Adrian de Wynter, Xun Wang, Alex Sokolov, Qilong Gu, and Si-Qing Chen. 2023. An evaluation on large language model outputs: Discourse and memorization. arXiv preprint arXiv:2304.08637. [find](https://scholar.google.com/scholar?q=Adrian+de+Wynter%2C+Xun+Wang%2C+Alex+Sokolov%2C+Qilong+Gu%2C+and+Si%2DQing+Chen%2E+2023%2E+An+evaluation+on+large+language+model+outputs%3A+Discourse+and+memorization%2E+arXiv+preprint+arXiv%3A2304%2E08637%2E)
29. Erik Derner and Kristina Batistič. 2023. Beyond the safeguards: Exploring the security risks of ChatGPT. arXiv preprint arXiv:2305.08005. — [arXiv:2305.08005](https://arxiv.org/abs/2305.08005) [find](https://scholar.google.com/scholar?q=Erik+Derner+and+Kristina+Batisti%C4%8D%2E+2023%2E+Beyond+the+safeguards%3A+Exploring+the+security+risks+of+ChatGPT%2E+arXiv+preprint+arXiv%3A2305%2E08005%2E+%E2%80%94+%5BarXiv%3A2305%2E08005%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2305%2E08005%29)
30. Aniket Deroy, Kripabandhu Ghosh, and Saptarshi Ghosh. 2023. How ready are pre-trained abstractive models and LLMs for legal case judgement summarization? arXiv preprint arXiv:2306.01248. [find](https://scholar.google.com/scholar?q=Aniket+Deroy%2C+Kripabandhu+Ghosh%2C+and+Saptarshi+Ghosh%2E+2023%2E+How+ready+are+pre%2Dtrained+abstractive+models+and+LLMs+for+legal+case+judgement+summarization%3F+arXiv+preprint+arXiv%3A2306%2E01248%2E)
31. Dragomir Radev. 2021. SummEval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409. [find](https://scholar.google.com/scholar?q=Dragomir+Radev%2E+2021%2E+SummEval%3A+Re%2Devaluating+summarization+evaluation%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+9%3A391%E2%80%93409%2E)
32. Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, Alexis Chevalier, and Julius Berner. 2023. [Mathematical capabilities of ChatGPT](https://md.research.elamur.ai/free/en/papers/frieder-2023-mathematical-capabilities-chatgpt/). arXiv preprint arXiv:2301.13867.
33. Jinglong Gao, Xiao Ding, Bing Qin, and Ting Liu. 2023. Is ChatGPT a good causal reasoner? A comprehensive evaluation. arXiv preprint arXiv:2305.07375. [find](https://scholar.google.com/scholar?q=Jinglong+Gao%2C+Xiao+Ding%2C+Bing+Qin%2C+and+Ting+Liu%2E+2023%2E+Is+ChatGPT+a+good+causal+reasoner%3F+A+comprehensive+evaluation%2E+arXiv+preprint+arXiv%3A2305%2E07375%2E)
34. Mengshi Ge, Rui Mao, and Erik Cambria. 2023. A survey on computational metaphor processing techniques: From identification, interpretation, generation to application. Artificial Intelligence Review, 56:1829–1895. [find](https://scholar.google.com/scholar?q=Mengshi+Ge%2C+Rui+Mao%2C+and+Erik+Cambria%2E+2023%2E+A+survey+on+computational+metaphor+processing+techniques%3A+From+identification%2C+interpretation%2C+generation+to+application%2E+Artificial+Intelligence+Review%2C+5)
35. Wayne Geerling, G Dirk Mateer, Jadrian Wooten, and Nikhil Damodaran. 2023. ChatGPT has aced the test of understanding in college economics: Now what? The American Economist, page 05694345231169654. [find](https://scholar.google.com/scholar?q=Wayne+Geerling%2C+G+Dirk+Mateer%2C+Jadrian+Wooten%2C+and+Nikhil+Damodaran%2E+2023%2E+ChatGPT+has+aced+the+test+of+understanding+in+college+economics%3A+Now+what%3F+The+American+Economist%2C+page+05694345231169654%2E)
36. Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. [ChatGPT outperforms crowdworkers for text-annotation tasks](https://md.research.elamur.ai/free/en/papers/gilardi-2023-chatgpt-outperforms-crowd-workers-text/). arXiv preprint arXiv:2303.15056.
37. Aidan Gilson, Conrad W Safranek, Thomas Huang, Vimig Socrates, Ling Chi, Richard Andrew Taylor, David Chartash, et al. 2023. How does ChatGPT perform on the United States Medical Licensing Examination? The implications of large language models for medical education and knowledge assessment. JMIR Med… [find](https://scholar.google.com/scholar?q=Aidan+Gilson%2C+Conrad+W+Safranek%2C+Thomas+Huang%2C+Vimig+Socrates%2C+Ling+Chi%2C+Richard+Andrew+Taylor%2C+David+Chartash%2C+et+al%2E+2023%2E+How+does+ChatGPT+perform+on+the+United+States+Medical+Licensing+Examination)
38. Max Grusky, Mor Naaman, and Yoav Artzi. 2018. [NEWSROOM: A dataset of 1.3 million summaries with diverse extractive strategies](https://md.research.elamur.ai/free/en/papers/grusky-2018-newsroom-dataset-1-3-million/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 708–719.
39. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional Transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
40. Jian Guan, Zhexin Zhang, Zhuoer Feng, Zitao Liu, Wenbiao Ding, Xiaoxi Mao, Changjie Fan, and Minlie Huang. 2021. OpenMEVA: A benchmark for evaluating open-ended story generation metrics. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internati… [find](https://scholar.google.com/scholar?q=Jian+Guan%2C+Zhexin+Zhang%2C+Zhuoer+Feng%2C+Zitao+Liu%2C+Wenbiao+Ding%2C+Xiaoxi+Mao%2C+Changjie+Fan%2C+and+Minlie+Huang%2E+2021%2E+OpenMEVA%3A+A+benchmark+for+evaluating+open%2Dended+story+generation+metrics%2E+In+Proceeding)
41. Alexander R Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023. [The false promise of imitating proprietary LLMs](https://md.research.elamur.ai/free/en/papers/gudibande-2023-false-promise-imitating-proprietary-llms/). arXiv preprint arXiv:2305.15717.
42. RJan Hatzius, Joseph Briggs, Devesh Kodnani, and Giovanni Pierdomenico. 2023. The potentially large effects of artificial intelligence on economic growth. Technical report, Goldman Sachs. [find](https://scholar.google.com/scholar?q=RJan+Hatzius%2C+Joseph+Briggs%2C+Devesh+Kodnani%2C+and+Giovanni+Pierdomenico%2E+2023%2E+The+potentially+large+effects+of+artificial+intelligence+on+economic+growth%2E+Technical+report%2C+Goldman+Sachs%2E)
43. Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How good are GPT models at machine translation? A comprehensive evaluation. arXiv preprint arXiv:2302.09210. [find](https://scholar.google.com/scholar?q=Amr+Hendy%2C+Mohamed+Abdelrehim%2C+Amr+Sharaf%2C+Vikas+Raunak%2C+Mohamed+Gabr%2C+Hitokazu+Matsushita%2C+Young+Jin+Kim%2C+Mohamed+Afify%2C+and+Hany+Hassan+Awadalla%2E+2023%2E+How+good+are+GPT+models+at+machine+translation)
44. Takanobu Hirosawa, Yukinori Harada, Masashi Yokose, Tetsu Sakamoto, Ren Kawamura, and Taro Shimizu. 2023. Diagnostic accuracy of differential-diagnosis lists generated by generative pretrained Transformer 3 chatbot for clinical vignettes with common chief complaints: A pilot study. International Jou… [find](https://scholar.google.com/scholar?q=Takanobu+Hirosawa%2C+Yukinori+Harada%2C+Masashi+Yokose%2C+Tetsu+Sakamoto%2C+Ren+Kawamura%2C+and+Taro+Shimizu%2E+2023%2E+Diagnostic+accuracy+of+differential%2Ddiagnosis+lists+generated+by+generative+pretrained+Transfo)
45. Mark K Ho, Michael L Littman, Fiery Cushman, and Joseph L Austerweil. 2015. Teaching with rewards and punishments: Reinforcement or communication? In CogSci. [find](https://scholar.google.com/scholar?q=Mark+K+Ho%2C+Michael+L+Littman%2C+Fiery+Cushman%2C+and+Joseph+L+Austerweil%2E+2015%2E+Teaching+with+rewards+and+punishments%3A+Reinforcement+or+communication%3F+In+CogSci%2E)
46. Bart Holterman and Kees van Deemter. 2023. Does ChatGPT have theory of mind? arXiv preprint arXiv:2305.14020. — [arXiv:2305.14020](https://arxiv.org/abs/2305.14020) [find](https://scholar.google.com/scholar?q=Bart+Holterman+and+Kees+van+Deemter%2E+2023%2E+Does+ChatGPT+have+theory+of+mind%3F+arXiv+preprint+arXiv%3A2305%2E14020%2E+%E2%80%94+%5BarXiv%3A2305%2E14020%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2305%2E14020%29)
47. Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, et al. 2023. [C-EVAL: A multi-level multi-discipline Chinese evaluation suite for foundation models](https://md.research.elamur.ai/free/en/papers/huang-2023-c-eval-multi-level-multi/). arXiv preprint arXiv:2305.08322.
48. Sophie Jentzsch and Kristian Kersting. 2023. Chat-GPT is fun, but it is not funny! humor is still challenging large language models. arXiv preprint arXiv:2306.04563. [find](https://scholar.google.com/scholar?q=Sophie+Jentzsch+and+Kristian+Kersting%2E+2023%2E+Chat%2DGPT+is+fun%2C+but+it+is+not+funny%21+humor+is+still+challenging+large+language+models%2E+arXiv+preprint+arXiv%3A2306%2E04563%2E)
49. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. [Survey of hallucination in natural language generation](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/). ACM Computing Surveys, 55(12):1–38.
50. Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Xing Wang, and Zhaopeng Tu. 2023. [Is ChatGPT a good translator? Yes with GPT-4 as the engine](https://md.research.elamur.ai/free/en/papers/jiao-2023-chatgpt-good-translator-yes-gpt/). arXiv preprint arXiv:2301.08745.
51. Daniel Kahneman. 2000. Thinking Fast and Slow. Farrar. [find](https://scholar.google.com/scholar?q=Daniel+Kahneman%2E+2000%2E+Thinking+Fast+and+Slow%2E+Farrar%2E)
52. Marzena Karpinska and Mohit Iyyer. 2023. [Large language models effectively leverage document-level context for literary translation, but critical errors persist](https://md.research.elamur.ai/free/en/papers/karpinska-2023-large-language-models-effectively-leverage/). arXiv preprint arXiv:2304.03245.
53. Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. 2023. Chat-GPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differenc… [find](https://scholar.google.com/scholar?q=Enkelejda+Kasneci%2C+Kathrin+Se%C3%9Fler%2C+Stefan+K%C3%BCchemann%2C+Maria+Bannert%2C+Daryna+Dementieva%2C+Frank+Fischer%2C+Urs+Gasser%2C+Georg+Groh%2C+Stephan+G%C3%BCnnemann%2C+Eyke+H%C3%BCllermeier%2C+et+al%2E+2023%2E+Chat%2DGPT+for+good%3F+On+op)
54. Emre Kıcıman, Robert Ness, Amit Sharma, and Chenhao Tan. 2023. Causal reasoning and large language models: Opening a new frontier for causality. arXiv preprint arXiv:2305.00050. [find](https://scholar.google.com/scholar?q=Emre+K%C4%B1c%C4%B1man%2C+Robert+Ness%2C+Amit+Sharma%2C+and+Chenhao+Tan%2E+2023%2E+Causal+reasoning+and+large+language+models%3A+Opening+a+new+frontier+for+causality%2E+arXiv+preprint+arXiv%3A2305%2E00050%2E)
55. Tom Kocmi and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality. arXiv preprint arXiv:2302.14520. [find](https://scholar.google.com/scholar?q=Tom+Kocmi+and+Christian+Federmann%2E+2023%2E+Large+language+models+are+state%2Dof%2Dthe%2Dart+evaluators+of+translation+quality%2E+arXiv+preprint+arXiv%3A2302%2E14520%2E)
56. Jan Kocoń, Igor Cichecki, Oliwier Kaszyca, Mateusz Kochanek, Dominika Szydło, Joanna Baran, Julita Bielaniewicz, Marcin Gruza, Arkadiusz Janz, Kamil Kanclerz, et al. 2023. [ChatGPT: Jack of all trades, master of none](https://md.research.elamur.ai/free/en/papers/kocon-2023-chatgpt-jack-all-trades-master/). arXiv preprint arXiv:2302.10724.
57. Gerd Kortemeyer. 2023. Could an artificial-intelligence agent pass an introductory physics course? Physical Review Physics Education Research, 19(1):010132. [find](https://scholar.google.com/scholar?q=Gerd+Kortemeyer%2E+2023%2E+Could+an+artificial%2Dintelligence+agent+pass+an+introductory+physics+course%3F+Physical+Review+Physics+Education+Research%2C+19%281%29%3A010132%2E)
58. Michal Kosinski. 2023. [Theory of mind may have spontaneously emerged in large language models](https://md.research.elamur.ai/free/en/papers/kosinski-2023-evaluating-large-language-models-theory/). arXiv preprint arXiv:2302.02083.
59. Tiffany H Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, et al. 2023. [Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models](https://md.research.elamur.ai/free/en/papers/kung-2023-performance-chatgpt-usmle-potential-ai/). PLoS Digital H…
60. Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Huu Nguyen. 2023. [ChatGPT beyond English: Towards a comprehensive evaluation of large language models in multilingual learning](https://md.research.elamur.ai/free/en/papers/lai-2023-chatgpt-beyond-english-comprehensive-evaluation/). arXiv preprint arXiv:2304.05613.
61. Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, and Jimmy Xiangji Huang. 2023. [A systematic study and comprehensive evaluation of ChatGPT on benchmark datasets](https://md.research.elamur.ai/free/en/papers/laskar-2023-systematic-study-comprehensive-evaluation-chatgpt/). arXiv preprint arXiv:2305.18486.
62. Hyunsu Lee. 2023. The rise of ChatGPT: Exploring its potential in medical education. Anatomical Sciences Education. [find](https://scholar.google.com/scholar?q=Hyunsu+Lee%2E+2023%2E+The+rise+of+ChatGPT%3A+Exploring+its+potential+in+medical+education%2E+Anatomical+Sciences+Education%2E)
63. Jan Leike, Miljan Martic, and Shane Legg. 2017. Learning through human feedback. Technical report, Google DeepMind. [find](https://scholar.google.com/scholar?q=Jan+Leike%2C+Miljan+Martic%2C+and+Shane+Legg%2E+2017%2E+Learning+through+human+feedback%2E+Technical+report%2C+Google+DeepMind%2E)
64. Aitor Lewkowycz, Anders Johan Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Venkatesh Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. 2022. [Solving quantitative reasoning problems with language models](https://md.research.elamur.ai/free/en/papers/lewkowycz-2022-solving-quantitative-reasoning-problems-language/). In Advances in Neural Information Processing Systems.
65. Jinyang Li, Binyuan Hui, Ge Qu, Binhua Li, Jiaxi Yang, Bowen Li, Bailin Wang, Bowen Qin, Rongyu Cao, Ruiying Geng, et al. 2023. [Can LLM already serve as a database interface? A big bench for large-scale database grounded text-to-SQLs](https://md.research.elamur.ai/free/en/papers/li-2023-llm-already-serve-database-interface/). arXiv preprint arXiv:2305.03111.
66. Chin-Yew Lin. 2004. [ROUGE: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text Summarization Branches Out, pages 74–81.
67. Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. 2023a. [We’re afraid language models aren’t modeling ambiguity](https://md.research.elamur.ai/free/en/papers/liu-2023-we-re-afraid-language-models/). arXiv preprint arXiv:2304.14399.
68. Gabrielle Kaili-May Liu. 2023. Perspectives on the social impacts of reinforcement learning with human feedback. arXiv preprint arXiv:2303.02891. — [arXiv:2303.02891](https://arxiv.org/abs/2303.02891) [find](https://scholar.google.com/scholar?q=Gabrielle+Kaili%2DMay+Liu%2E+2023%2E+Perspectives+on+the+social+impacts+of+reinforcement+learning+with+human+feedback%2E+arXiv+preprint+arXiv%3A2303%2E02891%2E+%E2%80%94+%5BarXiv%3A2303%2E02891%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2303%2E02891%29)
69. Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. 2023b. [Evaluating the logical reasoning ability of ChatGPT and GPT-4](https://md.research.elamur.ai/free/en/papers/liu-2023-evaluating-logical-reasoning-ability-chatgpt/). arXiv preprint arXiv:2304.03439.
70. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023c. [G-EVAL: NLG evaluation using GPT-4 with better human alignment](https://md.research.elamur.ai/free/en/papers/liu-2023-g-eval-nlg-evaluation-gpt/). arXiv preprint arXiv:2303.16634.
71. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. [RoBERTa: A robustly optimized BERT pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). arXiv preprint arXiv:1907.11692.
72. François Mairesse, Milica Gasic, Filip Jurcicek, Simon Keizer, Blaise Thomson, Kai Yu, and Steve Young. 2010. Phrase-based statistical language generation using graphical models and active learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 155… [find](https://scholar.google.com/scholar?q=Fran%C3%A7ois+Mairesse%2C+Milica+Gasic%2C+Filip+Jurcicek%2C+Simon+Keizer%2C+Blaise+Thomson%2C+Kai+Yu%2C+and+Steve+Young%2E+2010%2E+Phrase%2Dbased+statistical+language+generation+using+graphical+models+and+active+learning%2E+I)
73. Rui Mao, Guanyi Chen, Xiao Li, Mengshi Ge, and Erik Cambria. 2024a. A comparative analysis of metaphorical cognition in ChatGPT and human minds. Technical report, Nanyang Technological University. [find](https://scholar.google.com/scholar?q=Rui+Mao%2C+Guanyi+Chen%2C+Xiao+Li%2C+Mengshi+Ge%2C+and+Erik+Cambria%2E+2024a%2E+A+comparative+analysis+of+metaphorical+cognition+in+ChatGPT+and+human+minds%2E+Technical+report%2C+Nanyang+Technological+University%2E)
74. Rui Mao, Kai He, Xulang Zhang, Guanyi Chen, Jinjie Ni, Zonglin Yang, and Erik Cambria. 2024b. A survey on semantic processing techniques. Information Fusion, 101:101988. [find](https://scholar.google.com/scholar?q=Rui+Mao%2C+Kai+He%2C+Xulang+Zhang%2C+Guanyi+Chen%2C+Jinjie+Ni%2C+Zonglin+Yang%2C+and+Erik+Cambria%2E+2024b%2E+A+survey+on+semantic+processing+techniques%2E+Information+Fusion%2C+101%3A101988%2E)
75. Rui Mao, Xiao Li, Kai He, Mengshi Ge, and Erik Cambria. 2023a. MetaPro Online: A computational metaphor processing online system. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), volume 3, pages 127–135. [find](https://scholar.google.com/scholar?q=Rui+Mao%2C+Xiao+Li%2C+Kai+He%2C+Mengshi+Ge%2C+and+Erik+Cambria%2E+2023a%2E+MetaPro+Online%3A+A+computational+metaphor+processing+online+system%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+Compu)
76. Rui Mao, Qian Liu, Kai He, Wei Li, and Erik Cambria. 2023b. The biases of pre-trained language models: An empirical study on prompt-based sentiment analysis and emotion detection. IEEE Transactions on Affective Computing, 14(3):1743–1753. [find](https://scholar.google.com/scholar?q=Rui+Mao%2C+Qian+Liu%2C+Kai+He%2C+Wei+Li%2C+and+Erik+Cambria%2E+2023b%2E+The+biases+of+pre%2Dtrained+language+models%3A+An+empirical+study+on+prompt%2Dbased+sentiment+analysis+and+emotion+detection%2E+IEEE+Transactions+on)
77. Gary Marcus and Ernest Davis. 2023. How not to test GPT. Accessed: 29/05/2023. [find](https://scholar.google.com/scholar?q=Gary+Marcus+and+Ernest+Davis%2E+2023%2E+How+not+to+test+GPT%2E+Accessed%3A+29%2F05%2F2023%2E)
78. Eric Martínez. 2023. Re-evaluating GPT-4’s bar exam performance. Available at SSRN 4441311. [find](https://scholar.google.com/scholar?q=Eric+Mart%C3%ADnez%2E+2023%2E+Re%2Devaluating+GPT%2D4%E2%80%99s+bar+exam+performance%2E+Available+at+SSRN+4441311%2E)
79. Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, Anton Belyi, Haotian Zhang, Karanjeet Singh, Doug Kang, Hongyu Hè, Max Schwarzer, Tom Gunter, Xiang Kong, Aonan Zhang, Jianyu Wang, Chong Wang, Nan Du, Tao… [find](https://scholar.google.com/scholar?q=Brandon+McKinzie%2C+Zhe+Gan%2C+Jean%2DPhilippe+Fauconnier%2C+Sam+Dodge%2C+Bowen+Zhang%2C+Philipp+Dufter%2C+Dhruti+Shah%2C+Xianzhi+Du%2C+Futang+Peng%2C+Floris+Weers%2C+Anton+Belyi%2C+Haotian+Zhang%2C+Karanjeet+Singh%2C+Doug+Kang%2C)
80. Lars Mehnen, Stefanie Gruarin, Mina Vasileva, and Bernhard Knapp. 2023. ChatGPT as a medical doctor? A diagnostic accuracy study on common and rare diseases. medRxiv, pages 2023–04. [find](https://scholar.google.com/scholar?q=Lars+Mehnen%2C+Stefanie+Gruarin%2C+Mina+Vasileva%2C+and+Bernhard+Knapp%2E+2023%2E+ChatGPT+as+a+medical+doctor%3F+A+diagnostic+accuracy+study+on+common+and+rare+diseases%2E+medRxiv%2C+pages+2023%E2%80%9304%2E)
81. Shikib Mehri and Maxine Eskenazi. 2020. [USR: An unsupervised and reference free evaluation metric for dialog generation](https://md.research.elamur.ai/free/en/papers/mehri-2020-usr-unsupervised-reference-free-evaluation/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 681–707.
82. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. [Rethinking the role of demonstrations: What makes in-context learning work?](https://md.research.elamur.ai/free/en/papers/min-2022-rethinking-role-demonstrations-what-makes/) In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048–11064.
83. Marvin L Minsky. 1991. Logical versus analogical or symbolic versus connectionist or neat versus scruffy. AI magazine, 12(2):34–34. [find](https://scholar.google.com/scholar?q=Marvin+L+Minsky%2E+1991%2E+Logical+versus+analogical+or+symbolic+versus+connectionist+or+neat+versus+scruffy%2E+AI+magazine%2C+12%282%29%3A34%E2%80%9334%2E)
84. Shima Rahimi Moghaddam and Christopher J. Honey. 2023. Boosting theory-of-mind performance in large language models via prompting. arXiv preprint arXiv:2304.11490. — [arXiv:2304.11490](https://arxiv.org/abs/2304.11490) [find](https://scholar.google.com/scholar?q=Shima+Rahimi+Moghaddam+and+Christopher+J%2E+Honey%2E+2023%2E+Boosting+theory%2Dof%2Dmind+performance+in+large+language+models+via+prompting%2E+arXiv+preprint+arXiv%3A2304%2E11490%2E+%E2%80%94+%5BarXiv%3A2304%2E11490%5D%28https%3A%2F%2Farxiv%2Eo)
85. NLP Team MosaicML. 2023. Introducing MPT-7B: A new standard for open-source, commercially usable LLMs. Accessed: 18/05/2023. [find](https://scholar.google.com/scholar?q=NLP+Team+MosaicML%2E+2023%2E+Introducing+MPT%2D7B%3A+A+new+standard+for+open%2Dsource%2C+commercially+usable+LLMs%2E+Accessed%3A+18%2F05%2F2023%2E)
86. Fabio Motoki, Valdemar Pinho Neto, and Victor Rodrigues. 2023. More human than human: Measuring ChatGPT political bias. Available at SSRN 4372349. [find](https://scholar.google.com/scholar?q=Fabio+Motoki%2C+Valdemar+Pinho+Neto%2C+and+Victor+Rodrigues%2E+2023%2E+More+human+than+human%3A+Measuring+ChatGPT+political+bias%2E+Available+at+SSRN+4372349%2E)
87. NASEM, National Academies of Sciences, Engineering, and Medicine and others. 2018. How people learn II: Learners, contexts, and cultures. National Academies Press. [find](https://scholar.google.com/scholar?q=NASEM%2C+National+Academies+of+Sciences%2C+Engineering%2C+and+Medicine+and+others%2E+2018%2E+How+people+learn+II%3A+Learners%2C+contexts%2C+and+cultures%2E+National+Academies+Press%2E)
88. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Technical report, OpenAI. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+Technical+report%2C+OpenAI%2E)
89. Arya S Rao, Michael Pang, John Kim, Meghana Kamineni, Winston Lie, Anoop K Prasad, Adam Landman, Keith Dryer, and Marc D Succi. 2023. Assessing the utility of ChatGPT throughout the entire clinical workflow. medRxiv. [find](https://scholar.google.com/scholar?q=Arya+S+Rao%2C+Michael+Pang%2C+John+Kim%2C+Meghana+Kamineni%2C+Winston+Lie%2C+Anoop+K+Prasad%2C+Adam+Landman%2C+Keith+Dryer%2C+and+Marc+D+Succi%2E+2023%2E+Assessing+the+utility+of+ChatGPT+throughout+the+entire+clinical+wo)
90. David Rapaport, Merton Gill, and Roy Schafer. 1946. Diagnostic psychological testing: The theory, statistical evaluation, and diagnostic application of a battery of tests: Volume II. The Year Book Publishers. [find](https://scholar.google.com/scholar?q=David+Rapaport%2C+Merton+Gill%2C+and+Roy+Schafer%2E+1946%2E+Diagnostic+psychological+testing%3A+The+theory%2C+statistical+evaluation%2C+and+diagnostic+application+of+a+battery+of+tests%3A+Volume+II%2E+The+Year+Book+Pub)
91. Xuan Ren and Lingqiao Liu. 2023. You can generate it again: Data-to-text generation with verification and correction prompting. arXiv preprint arXiv:2306.15933. — [arXiv:2306.15933](https://arxiv.org/abs/2306.15933) [find](https://scholar.google.com/scholar?q=Xuan+Ren+and+Lingqiao+Liu%2E+2023%2E+You+can+generate+it+again%3A+Data%2Dto%2Dtext+generation+with+verification+and+correction+prompting%2E+arXiv+preprint+arXiv%3A2306%2E15933%2E+%E2%80%94+%5BarXiv%3A2306%2E15933%5D%28https%3A%2F%2Farxiv%2Eorg%2F)
92. OpenAI. 2023a. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
93. Pablo Rivas and Liang Zhao. 2023. Marketing with ChatGPT: Navigating the ethical terrain of GPT-based chatbot technology. AI, 4(2):375–384. [find](https://scholar.google.com/scholar?q=Pablo+Rivas+and+Liang+Zhao%2E+2023%2E+Marketing+with+ChatGPT%3A+Navigating+the+ethical+terrain+of+GPT%2Dbased+chatbot+technology%2E+AI%2C+4%282%29%3A375%E2%80%93384%2E)
94. OpenAI. 2023b. Introducing ChatGPT. Accessed: 11/05/2023. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2023b%2E+Introducing+ChatGPT%2E+Accessed%3A+11%2F05%2F2023%2E)
95. David Rozado. 2023. The political biases of Chat-GPT. Social Sciences, 12(3):148. [find](https://scholar.google.com/scholar?q=David+Rozado%2E+2023%2E+The+political+biases+of+Chat%2DGPT%2E+Social+Sciences%2C+12%283%29%3A148%2E)
96. Regina Pally. 2007. The predicting brain: Unconscious repetition, conscious reflection and therapeutic change. The International Journal of Psychoanalysis, 88(4):861–881. [find](https://scholar.google.com/scholar?q=Regina+Pally%2E+2007%2E+The+predicting+brain%3A+Unconscious+repetition%2C+conscious+reflection+and+therapeutic+change%2E+The+International+Journal+of+Psychoanalysis%2C+88%284%29%3A861%E2%80%93881%2E)
97. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. 2015. [ImageNet large scale visual recognition challenge](https://md.research.elamur.ai/free/en/papers/russakovsky-2014-imagenet-large-scale-visual-recognition/). International Journal of Computer Vision, 115:211–252.
98. Brad Payne. 2020. Privacy protection with AI: Survey of data-anonymization techniques. Accessed: 03/06/2023. [find](https://scholar.google.com/scholar?q=Brad+Payne%2E+2020%2E+Privacy+protection+with+AI%3A+Survey+of+data%2Danonymization+techniques%2E+Accessed%3A+03%2F06%2F2023%2E)
99. Xutan Peng, Yipeng Zhang, Jingfeng Yang, and Mark Stevenson. 2023. On the security vulnerabilities of text-to-SQL models. arXiv preprint arXiv:2211.15363. — [arXiv:2211.15363](https://arxiv.org/abs/2211.15363) [find](https://scholar.google.com/scholar?q=Xutan+Peng%2C+Yipeng+Zhang%2C+Jingfeng+Yang%2C+and+Mark+Stevenson%2E+2023%2E+On+the+security+vulnerabilities+of+text%2Dto%2DSQL+models%2E+arXiv+preprint+arXiv%3A2211%2E15363%2E+%E2%80%94+%5BarXiv%3A2211%2E15363%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F22)
100. Maja Popović. 2015. chrF: character n-gram Fscore for automatic MT evaluation. In Proceedings of the 10th Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Maja+Popovi%C4%87%2E+2015%2E+chrF%3A+character+n%2Dgram+Fscore+for+automatic+MT+evaluation%2E+In+Proceedings+of+the+10th+Workshop+on+Statistical+Machine+Translation%2E)
101. Anantha Krishnan Pradeep. 2010. The buying brain: Secrets for selling to the subconscious mind. John Wiley & Sons. [find](https://scholar.google.com/scholar?q=Anantha+Krishnan+Pradeep%2E+2010%2E+The+buying+brain%3A+Secrets+for+selling+to+the+subconscious+mind%2E+John+Wiley+%26+Sons%2E)
102. Xiao Pu, Mingqi Gao, and Xiaojun Wan. 2023. Summarization is (almost) dead. arXiv preprint arXiv:2309.09558. — [arXiv:2309.09558](https://arxiv.org/abs/2309.09558) [find](https://scholar.google.com/scholar?q=Xiao+Pu%2C+Mingqi+Gao%2C+and+Xiaojun+Wan%2E+2023%2E+Summarization+is+%28almost%29+dead%2E+arXiv+preprint+arXiv%3A2309%2E09558%2E+%E2%80%94+%5BarXiv%3A2309%2E09558%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2309%2E09558%29)
103. Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. [Is ChatGPT a general-purpose natural language processing task solver?](https://md.research.elamur.ai/free/en/papers/qin-2023-chatgpt-general-purpose-natural-language/) arXiv preprint arXiv:2302.06476.
104. Jérôme Rutinowski, Sven Franke, Jan Endendyk, Ina Dormuth, and Markus Pauly. 2023. The self-perception and political biases of ChatGPT. arXiv preprint arXiv:2304.07333. — [arXiv:2304.07333](https://arxiv.org/abs/2304.07333) [find](https://scholar.google.com/scholar?q=J%C3%A9r%C3%B4me+Rutinowski%2C+Sven+Franke%2C+Jan+Endendyk%2C+Ina+Dormuth%2C+and+Markus+Pauly%2E+2023%2E+The+self%2Dperception+and+political+biases+of+ChatGPT%2E+arXiv+preprint+arXiv%3A2304%2E07333%2E+%E2%80%94+%5BarXiv%3A2304%2E07333%5D%28https%3A%2F%2Far)
105. Malik Sallam. 2023. ChatGPT utility in healthcare education, research, and practice: Systematic review on the promising perspectives and valid concerns. In Healthcare, volume 11, page 887. [find](https://scholar.google.com/scholar?q=Malik+Sallam%2E+2023%2E+ChatGPT+utility+in+healthcare+education%2C+research%2C+and+practice%3A+Systematic+review+on+the+promising+perspectives+and+valid+concerns%2E+In+Healthcare%2C+volume+11%2C+page+887%2E)
106. Erik Tjong Kim Sang and Fien De Meulder. 2003. [Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition](https://md.research.elamur.ai/free/en/papers/sang-2003-introduction-conll-2003-shared-task/). In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142–147.
107. Glorin Sebastian. 2023. Do ChatGPT and other AI chatbots pose a cybersecurity risk?: An exploratory study. International Journal of Security and Privacy in Pervasive Computing (IJSPPC), 15(1):1–11. [find](https://scholar.google.com/scholar?q=Glorin+Sebastian%2E+2023%2E+Do+ChatGPT+and+other+AI+chatbots+pose+a+cybersecurity+risk%3F%3A+An+exploratory+study%2E+International+Journal+of+Security+and+Privacy+in+Pervasive+Computing+%28IJSPPC%29%2C+15%281%29%3A1%E2%80%9311%2E)
108. Mohamed L Seghier. 2023. ChatGPT: Not all languages are equal. Nature, 615(7951):216–216. [find](https://scholar.google.com/scholar?q=Mohamed+L+Seghier%2E+2023%2E+ChatGPT%3A+Not+all+languages+are+equal%2E+Nature%2C+615%287951%29%3A216%E2%80%93216%2E)
109. Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. 2023. [The curse of recursion: Training on generated data makes models forget](https://md.research.elamur.ai/free/en/papers/shumailov-2023-curse-recursion-training-generated-data/). arXiv preprint arXiv:2305.17493.
110. Mayank Soni and Vincent Wade. 2023. Comparing abstractive summaries generated by ChatGPT to real summaries through blinded reviewers and text classification algorithms. arXiv preprint arXiv:2303.17650. — [arXiv:2303.17650](https://arxiv.org/abs/2303.17650) [find](https://scholar.google.com/scholar?q=Mayank+Soni+and+Vincent+Wade%2E+2023%2E+Comparing+abstractive+summaries+generated+by+ChatGPT+to+real+summaries+through+blinded+reviewers+and+text+classification+algorithms%2E+arXiv+preprint+arXiv%3A2303%2E17650)
111. Weiwei Sun, Lingyong Yan, Xinyu Ma, Pengjie Ren, Dawei Yin, and Zhaochun Ren. 2023. [Is Chat-GPT good at search?](https://md.research.elamur.ai/free/en/papers/sun-2023-chatgpt-good-search-investigating-large/) Investigating large language models as re-ranking agent. arXiv preprint arXiv:2304.09542.
112. Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2021. [Multilingual translation from denoising pre-training](https://md.research.elamur.ai/free/en/papers/tang-2021-multilingual-translation-denoising-pre-training/). In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3450–3466.
113. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Alpaca: A strong, replicable instruction-following model. Technical report, Stanford Center for Research on Foundation Models. [find](https://scholar.google.com/scholar?q=Rohan+Taori%2C+Ishaan+Gulrajani%2C+Tianyi+Zhang%2C+Yann+Dubois%2C+Xuechen+Li%2C+Carlos+Guestrin%2C+Percy+Liang%2C+and+Tatsunori+B+Hashimoto%2E+2023%2E+Alpaca%3A+A+strong%2C+replicable+instruction%2Dfollowing+model%2E+Technical)
114. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. [LLaMa: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971.
115. Shangqing Tu, Chunyang Li, Jifan Yu, Xiaozhi Wang, Lei Hou, and Juanzi Li. 2023. ChatLog: Recording and analyzing ChatGPT across time. arXiv preprint arXiv:2304.14106. — [arXiv:2304.14106](https://arxiv.org/abs/2304.14106) [find](https://scholar.google.com/scholar?q=Shangqing+Tu%2C+Chunyang+Li%2C+Jifan+Yu%2C+Xiaozhi+Wang%2C+Lei+Hou%2C+and+Juanzi+Li%2E+2023%2E+ChatLog%3A+Recording+and+analyzing+ChatGPT+across+time%2E+arXiv+preprint+arXiv%3A2304%2E14106%2E+%E2%80%94+%5BarXiv%3A2304%2E14106%5D%28https%3A%2F%2Farx)
116. Ruben S van Bergen and Nikolaus Kriegeskorte. 2020. Going in circles is the way forward: The role of recurrence in visual inference. Current Opinion in Neurobiology, 65:176–193. [find](https://scholar.google.com/scholar?q=Ruben+S+van+Bergen+and+Nikolaus+Kriegeskorte%2E+2020%2E+Going+in+circles+is+the+way+forward%3A+The+role+of+recurrence+in+visual+inference%2E+Current+Opinion+in+Neurobiology%2C+65%3A176%E2%80%93193%2E)
117. Boshi Wang, Xiang Yue, and Huan Sun. 2023a. Can ChatGPT defend the truth? Automatic dialectical evaluation elicits LLMs’ deficiencies in reasoning. arXiv preprint arXiv:2305.13160. evaluator? A preliminary study. arXiv preprint arXiv:2303.04048. — [arXiv:2305.13160](https://arxiv.org/abs/2305.13160) [find](https://scholar.google.com/scholar?q=Boshi+Wang%2C+Xiang+Yue%2C+and+Huan+Sun%2E+2023a%2E+Can+ChatGPT+defend+the+truth%3F+Automatic+dialectical+evaluation+elicits+LLMs%E2%80%99+deficiencies+in+reasoning%2E+arXiv+preprint+arXiv%3A2305%2E13160%2E+evaluator%3F+A+prelim)
118. Jindong Wang, Xixu Hu, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang, Haojun Huang, Wei Ye, Xiubo Geng, Binxin Jiao, Yue Zhang, and Xing Xie. 2023d. On the robustness of ChatGPT: An adversarial and out-of-distribution perspective. arXiv preprint arXiv:2302.12095. [find](https://scholar.google.com/scholar?q=Jindong+Wang%2C+Xixu+Hu%2C+Wenxin+Hou%2C+Hao+Chen%2C+Runkai+Zheng%2C+Yidong+Wang%2C+Linyi+Yang%2C+Haojun+Huang%2C+Wei+Ye%2C+Xiubo+Geng%2C+Binxin+Jiao%2C+Yue+Zhang%2C+and+Xing+Xie%2E+2023d%2E+On+the+robustness+of+ChatGPT%3A+An+adve)
119. Xiao Wang, Guangyao Chen, Guangwu Qian, Pengcheng Gao, Xiao-Yong Wei, Yaowei Wang, Yonghong Tian, and Wen Gao. 2023e. Large-scale multi-modal pre-trained models: A comprehensive survey. Machine Intelligence Research, 20(4):447–482. [find](https://scholar.google.com/scholar?q=Xiao+Wang%2C+Guangyao+Chen%2C+Guangwu+Qian%2C+Pengcheng+Gao%2C+Xiao%2DYong+Wei%2C+Yaowei+Wang%2C+Yonghong+Tian%2C+and+Wen+Gao%2E+2023e%2E+Large%2Dscale+multi%2Dmodal+pre%2Dtrained+models%3A+A+comprehensive+survey%2E+Machine+Intell)
120. Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang, Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Meishan Zhang, et al. 2023. [Zero-shot information extraction via chatting with Chat-GPT](https://md.research.elamur.ai/free/en/papers/wei-2023-chatie-zero-shot-information-extraction/). arXiv preprint arXiv:2302.10205.
121. Qianqian Xie, Weiguang Han, Yanzhao Lai, Min Peng, and Jimin Huang. 2023. The Wall Street neophyte: A zero-shot analysis of ChatGPT over multimodal stock movement prediction challenges. arXiv preprint arXiv:2304.05351. — [arXiv:2304.05351](https://arxiv.org/abs/2304.05351) [find](https://scholar.google.com/scholar?q=Qianqian+Xie%2C+Weiguang+Han%2C+Yanzhao+Lai%2C+Min+Peng%2C+and+Jimin+Huang%2E+2023%2E+The+Wall+Street+neophyte%3A+A+zero%2Dshot+analysis+of+ChatGPT+over+multimodal+stock+movement+prediction+challenges%2E+arXiv+preprint)
122. Liang Xu and others from SuperCLUE team. 2023. SuperCLUE: A benchmark for foundation models in Chinese. Accessed: 18/05/2023. [find](https://scholar.google.com/scholar?q=Liang+Xu+and+others+from+SuperCLUE+team%2E+2023%2E+SuperCLUE%3A+A+benchmark+for+foundation+models+in+Chinese%2E+Accessed%3A+18%2F05%2F2023%2E)
123. Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. 2023. [The dawn of lmms: Preliminary explorations with gpt-4v(ision)](https://md.research.elamur.ai/free/en/papers/yang-2023-dawn-lmms-preliminary-explorations-gpt/).
124. Wentao Ye, Mingfeng Ou, Tianyi Li, Yipeng chen, Xuetao Ma, Yifan Yanggong, Sai Wu, Jie Fu, Gang Chen, Haobo Wang, and Junbo Zhao. 2023. Assessing hidden risks of LLMs: An empirical study on robustness, consistency, and credibility. arXiv preprint arXiv:2305.10235. — [arXiv:2305.10235](https://arxiv.org/abs/2305.10235) [find](https://scholar.google.com/scholar?q=Wentao+Ye%2C+Mingfeng+Ou%2C+Tianyi+Li%2C+Yipeng+chen%2C+Xuetao+Ma%2C+Yifan+Yanggong%2C+Sai+Wu%2C+Jie+Fu%2C+Gang+Chen%2C+Haobo+Wang%2C+and+Junbo+Zhao%2E+2023%2E+Assessing+hidden+risks+of+LLMs%3A+An+empirical+study+on+robustness)
125. Zheng-Xin Yong, Cristina Menghini, and Stephen H Bach. 2023a. [Low-resource languages jailbreak GPT-4](https://md.research.elamur.ai/free/en/papers/yong-2023-low-resource-languages-jailbreak-gpt/). arXiv preprint arXiv:2310.02446.
126. Jiaan Wang, Yunlong Liang, Fandong Meng, Zhixu Li, Jianfeng Qu, and Jie Zhou. 2023b. Zero-shot cross-lingual summarization via large language models. arXiv preprint arXiv:2302.14229. [find](https://scholar.google.com/scholar?q=Jiaan+Wang%2C+Yunlong+Liang%2C+Fandong+Meng%2C+Zhixu+Li%2C+Jianfeng+Qu%2C+and+Jie+Zhou%2E+2023b%2E+Zero%2Dshot+cross%2Dlingual+summarization+via+large+language+models%2E+arXiv+preprint+arXiv%3A2302%2E14229%2E)
127. Zheng-Xin Yong, Ruochen Zhang, Jessica Zosa Forde, Skyler Wang, Samuel Cahyawijaya, Holy Lovenia, Genta Indra Winata, Lintang Sutawika, Jan Christian Blaise Cruz, Long Phan, Yin Lin Tan, and Alham Fikri Aji. 2023b. [Prompting multilingual large language models to generate code-mixed texts: The case o…](https://md.research.elamur.ai/free/en/papers/yong-2023-prompting-multilingual-large-language-models/)
128. Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023c. Is ChatGPT a good NLG Shuzhou Yuan and Michael F"arber. 2023. Evaluating generative models for graph-to-text generation. arXiv preprint arXiv:2307.14712. [find](https://scholar.google.com/scholar?q=Jiaan+Wang%2C+Yunlong+Liang%2C+Fandong+Meng%2C+Haoxiang+Shi%2C+Zhixu+Li%2C+Jinan+Xu%2C+Jianfeng+Qu%2C+and+Jie+Zhou%2E+2023c%2E+Is+ChatGPT+a+good+NLG+Shuzhou+Yuan+and+Michael+F%22arber%2E+2023%2E+Evaluating+generative+models+)
129. Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. [BARTScore: Evaluating generated text as text generation](https://md.research.elamur.ai/free/en/papers/yuan-2021-bartscore-evaluating-generated-text-text/). Advances in Neural Information Processing Systems, 34:27263–27277.
130. Haopeng Zhang, Xiao Liu, and Jiawei Zhang. 2023a. Extractive summarization via ChatGPT for faithful summary generation. arXiv preprint arXiv:2304.04193. [find](https://scholar.google.com/scholar?q=Haopeng+Zhang%2C+Xiao+Liu%2C+and+Jiawei+Zhang%2E+2023a%2E+Extractive+summarization+via+ChatGPT+for+faithful+summary+generation%2E+arXiv+preprint+arXiv%3A2304%2E04193%2E)

*Updated 31 October 2026*