---
title: "Can we trust the evaluation on ChatGPT? — Aiyappa 2023"
description: "Those releases are the early version presumably used by Zhang et al. (V1), the Jan 30 version and… 32 references, cited by 35 papers here."
authors: ["Rachith Aiyappa", "Jisun An", "Haewoon Kwak", "Yong-yeol Ahn"]
year: 2023
venue: "Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023)"
source: "https://aclanthology.org/2023.trustnlp-1.5/"
doi: "10.18653/v1/2023.trustnlp-1.5"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/aiyappa-2023-can-we-trust-evaluation-chatgpt/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/aiyappa-2023-can-we-trust-evaluation-chatgpt/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / Can we trust the evaluation on ChatGPT?

# Can we trust the evaluation on ChatGPT?

**Authors:** Rachith Aiyappa, [Jisun An](https://md.research.elamur.ai/free/en/authors/jisun-an/), [Haewoon Kwak](https://md.research.elamur.ai/free/en/authors/haewoon-kwak/) and Yong-yeol Ahn · **Year:** 2023 · **Venue:** [Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023)](https://md.research.elamur.ai/free/en/venues/trustnlp/) ([Workshop on Trustworthy NLP 2023](https://md.research.elamur.ai/free/en/venues/trustnlp/2023/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.trustnlp-1.5/) · [PDF](https://aclanthology.org/2023.trustnlp-1.5.pdf) · [DOI 10.18653/v1/2023.trustnlp-1.5](https://doi.org/10.18653/v1/2023.trustnlp-1.5) · [OpenAlex](https://openalex.org/W4385734111)

**Citations:** 68 citations (OpenAlex, 2026-10-14) · [35](https://md.research.elamur.ai/free/en/papers/aiyappa-2023-can-we-trust-evaluation-chatgpt/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper examines whether published evaluations of ChatGPT can be trusted. The model is closed, its training data are undisclosed, and it is repeatedly updated through RLHF using user queries. Benchmark test sets that researchers submit could therefore leak into later versions and inflate scores.

As a case study, the authors take zero-shot stance detection on tweets. They replicate the prompt of Zhang et al. (2022) on the SemEval 2016 Task 6 and P-Stance test sets and compare several ChatGPT releases. Those releases are the early version presumably used by Zhang et al. (V1), the Jan 30 version and the Feb 13 ChatGPT plus version. Stance labels are extracted by hand from the model's often wordy answers.

Overall, scores rose after the earlier evaluation study appeared, which fits with possible contamination. However, ordinary model improvement explains the trend equally well. The authors stress that, without access to the model, leakage can be neither confirmed nor excluded. This matters for anyone using closed LLMs as research instruments or as benchmark baselines.

## Key points
- **Data:** 860 SemEval 2016 Task 6 test tweets for the targets Hillary Clinton, Feminist Movement and Legalization of Abortion, plus 2,157 P-Stance test tweets about Trump, Biden and Bernie Sanders.
- **P-Stance results (Micro-F1/Macro-F1, V1 → Feb 13 ChatGPT plus):**
  - Trump: 82.8/83.2 → 83.76/83.09
  - Biden: 82.3/82.0 → 83.07/82.69
  - Bernie: 79.4/79.4 → 79.7/79.6
- **SemEval drop and possible causes:** On SemEval, Feb 13 performed below Jan 30. The authors offer several possible explanations:
  - shared chat context in the manual Jan 30 sessions;
  - catastrophic forgetting;
  - a more "diplomatic", restricted model.
- **Practical constraints:** No access to retired versions, a cap of roughly 40 queries per hour and no public API at the time. These also meant uncertainty across repeated queries could not be estimated.
- **Recommendations:** Model creators should document possible contamination, allow leakage audits and block contamination coming from user inputs. The BIG-bench "canary" string is cited as one existing safeguard.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Indiana University Bloomington](https://md.research.elamur.ai/free/en/universities/indiana-university-bloomington/) — University, Bloomington, United States · 37 works · 3,685 citations
- [Indiana University](https://md.research.elamur.ai/free/en/universities/indiana-university/) — University, Bloomington, United States · 41 works · 3,663 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 543 works

🧭 Topics (1): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works

## Cite

Cite as: Rachith Aiyappa, Jisun An, Haewoon Kwak and Yong-yeol Ahn. 2023. Can we trust the evaluation on ChatGPT?. Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{aiyappa2023can,
  title = {{Can we trust the evaluation on ChatGPT?}},
  author = {Rachith Aiyappa, Jisun An, Haewoon Kwak and Yong-yeol Ahn},
  booktitle = {Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023)},
  year = {2023},
  doi = {10.18653/v1/2023.trustnlp-1.5},
  url = {https://doi.org/10.18653/v1/2023.trustnlp-1.5}
}
```

- [← previous AnglE-optimized Text Embeddings](https://md.research.elamur.ai/free/en/papers/li-2023-angle-optimized-text-embeddings/)
- [next → Rethinking Benchmark and Contamination for Language Models with Rephrased Samples](https://md.research.elamur.ai/free/en/papers/yang-2023-rethinking-benchmark-contamination-language-models/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/)

## References

32 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. 2023. Artificial hallucinations in chatgpt: Implications in scientific writing. Cureus, 15(2). [find](https://scholar.google.com/scholar?q=2023%2E+Artificial+hallucinations+in+chatgpt%3A+Implications+in+scientific+writing%2E+Cureus%2C+15%282%29%2E)
2. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610–623. [find](https://scholar.google.com/scholar?q=Emily+M+Bender%2C+Timnit+Gebru%2C+Angelina+McMillan%2DMajor%2C+and+Shmargaret+Shmitchell%2E+2021%2E+On+the+dangers+of+stochastic+parrots%3A+Can+language+models+be+too+big%3F+In+Proceedings+of+the+2021+ACM+conference+)
3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda sometimes factors like network errors which made Chat-GPT unresponsive forced us to open a new chat session in the same sitting. But for a major chunk, a… [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+sometimes+factors+like+network+errors+which+made+Chat%2D)
4. Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
5. Dan Milmo. 2023b. Reinventing search with a new ai-powered microsoft bing and edge, your copilot for the web. The Official Microsoft Blog. [https://archive.is/mCmFq](https://archive.is/mCmFq). [↗](https://archive.is/mCmFq)
6. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B Brown, Dawn Song, Ulfar Erlingsson, et al. 2021. [Extracting training data from large language models](https://md.research.elamur.ai/free/en/papers/carlini-2020-extracting-training-data-large-language/). In USENIX Security Symposium, volume 6.
7. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. [Rethinking the role of demonstrations: What makes in-context learning work?](https://md.research.elamur.ai/free/en/papers/min-2022-rethinking-role-demonstrations-what-makes/) Preprint arXiv:2202.12837.
8. Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. Preprint arXiv:2104.08758. [find](https://scholar.google.com/scholar?q=Jesse+Dodge%2C+Maarten+Sap%2C+Ana+Marasovic%CC%81%2C+William+Agnew%2C+Gabriel+Ilharco%2C+Dirk+Groeneveld%2C+Margaret+Mitchell%2C+and+Matt+Gardner%2E+2021%2E+Documenting+large+webtext+corpora%3A+A+case+study+on+the+colossal+cl)
9. Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016. [Semeval-2016 task 6: Detecting stance in tweets](https://md.research.elamur.ai/free/en/papers/mohammad-2016-semeval-2016-task-6-detecting/). In Proceedings of the 10th international workshop on semantic evaluation (SemEval-2016), pages 31–41.
10. Michael Dowling and Brian Lucey. 2023. Chatgpt for (finance) research: The bananarama conjecture. Finance Research Letters, 53. [find](https://scholar.google.com/scholar?q=Michael+Dowling+and+Brian+Lucey%2E+2023%2E+Chatgpt+for+%28finance%29+research%3A+The+bananarama+conjecture%2E+Finance+Research+Letters%2C+53%2E)
11. Saif M Mohammad, Parinaz Sobhani, and Svetlana Kiritchenko. 2017. Stance and sentiment in tweets. ACM Transactions on Internet Technology (TOIT), 17(3):1–23. [find](https://scholar.google.com/scholar?q=Saif+M+Mohammad%2C+Parinaz+Sobhani%2C+and+Svetlana+Kiritchenko%2E+2017%2E+Stance+and+sentiment+in+tweets%2E+ACM+Transactions+on+Internet+Technology+%28TOIT%29%2C+17%283%29%3A1%E2%80%9323%2E)
12. Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, Alexis Chevalier, and Julius Berner. 2023. [Mathematical capabilities of chatgpt](https://md.research.elamur.ai/free/en/papers/frieder-2023-mathematical-capabilities-chatgpt/). Preprint arXiv:2301.13867.
13. OpenAI. 2022. Chatgpt: Optimizing language models for dialogue. [https://archive.is/4snnY](https://archive.is/4snnY). [↗](https://archive.is/4snnY)
14. OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv:2303.08774.
15. Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. 2022. What can transformers learn in-context? a case study of simple function classes. Preprint arXiv:2208.01066. [find](https://scholar.google.com/scholar?q=Shivam+Garg%2C+Dimitris+Tsipras%2C+Percy+Liang%2C+and+Gregory+Valiant%2E+2022%2E+What+can+transformers+learn+in%2Dcontext%3F+a+case+study+of+simple+function+classes%2E+Preprint+arXiv%3A2208%2E01066%2E)
16. Preprint Sajan B Patel and Kyle Lam. 2023. Chatgpt: the future of discharge summaries? The Lancet Digital Health, 5(3):e107–e108. [find](https://scholar.google.com/scholar?q=Preprint+Sajan+B+Patel+and+Kyle+Lam%2E+2023%2E+Chatgpt%3A+the+future+of+discharge+summaries%3F+The+Lancet+Digital+Health%2C+5%283%29%3Ae107%E2%80%93e108%2E)
17. Jan Kocoń, Igor Cichecki, Oliwier Kaszyca, Mateusz Kochanek, Dominika Szydło, Joanna Baran, Julita Bielaniewicz, Marcin Gruza, Arkadiusz Janz, Kamil Kanclerz, et al. 2023. [Chatgpt: Jack of all trades, master of none](https://md.research.elamur.ai/free/en/papers/kocon-2023-chatgpt-jack-all-trades-master/). Preprint arXiv:2302.10724.
18. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E)
19. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. Preprint arXiv:1707.06347. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+2017%2E+Proximal+policy+optimization+algorithms%2E+Preprint+arXiv%3A1707%2E06347%2E)
20. Dilek Küçük and Fazli Can. 2020. Stance detection: A survey. ACM Computing Surveys (CSUR), 53(1):1– 37. [find](https://scholar.google.com/scholar?q=Dilek+K%C3%BC%C3%A7%C3%BCk+and+Fazli+Can%2E+2020%2E+Stance+detection%3A+A+survey%2E+ACM+Computing+Surveys+%28CSUR%29%2C+53%281%29%3A1%E2%80%93+37%2E)
21. Tiffany H Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, et al. 2023. [Performance of chatgpt on usmle: Potential for ai-assisted medical education using large language models](https://md.research.elamur.ai/free/en/papers/kung-2023-performance-chatgpt-usmle-potential-ai/). PLOS Digital H…
22. Dominik Sobania, Martin Briesch, Carol Hanna, and Justyna Petke. 2023. An analysis of the automatic bug fixing performance of chatgpt. Preprint arXiv:2301.08653. — [arXiv:2301.08653](https://arxiv.org/abs/2301.08653) [find](https://scholar.google.com/scholar?q=Dominik+Sobania%2C+Martin+Briesch%2C+Carol+Hanna%2C+and+Justyna+Petke%2E+2023%2E+An+analysis+of+the+automatic+bug+fixing+performance+of+chatgpt%2E+Preprint+arXiv%3A2301%2E08653%2E+%E2%80%94+%5BarXiv%3A2301%2E08653%5D%28https%3A%2F%2Farxiv%2Eorg)
23. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022. [Beyond the imitation game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/). Preprint arXiv:2206.04615.
24. Yingjie Li, Tiberiu Sosea, Aditya Sawant, Ajith Jayaraman Nair, Diana Inkpen, and Cornelia Caragea. 2021. [P-stance: A large dataset for stance detection in political domain](https://md.research.elamur.ai/free/en/papers/li-2021-p-stance-large-dataset-stance/). In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2355–2365.
25. Chris Stokel-Walker and Richard Van Noorden. 2023. What chatgpt and generative ai mean for science. Nature, 614(7947):214–216. [find](https://scholar.google.com/scholar?q=Chris+Stokel%2DWalker+and+Richard+Van+Noorden%2E+2023%2E+What+chatgpt+and+generative+ai+mean+for+science%2E+Nature%2C+614%287947%29%3A214%E2%80%93216%2E)
26. Michael McCloskey and Neal J Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109–165. Elsevier. [find](https://scholar.google.com/scholar?q=Michael+McCloskey+and+Neal+J+Cohen%2E+1989%2E+Catastrophic+interference+in+connectionist+networks%3A+The+sequential+learning+problem%2E+In+Psychology+of+learning+and+motivation%2C+volume+24%2C+pages+109%E2%80%93165%2E+Else)
27. Viriya Taecharungroj. 2023. What can chatgpt do? analyzing early reactions to the innovative ai chatbot on twitter. Big Data and Cognitive Computing, 7(1):35. [find](https://scholar.google.com/scholar?q=Viriya+Taecharungroj%2E+2023%2E+What+can+chatgpt+do%3F+analyzing+early+reactions+to+the+innovative+ai+chatbot+on+twitter%2E+Big+Data+and+Cognitive+Computing%2C+7%281%29%3A35%2E)
28. Christian Terwiesch. 2023. Would chatgpt3 get a wharton mba. a prediction based on its performance in the operations management course. [https://archive.is/HsSUf](https://archive.is/HsSUf). [↗](https://archive.is/HsSUf)
29. Dan Milmo. 2023a. Chatgpt reaches 100 million users two months after launch. The Guardian. [https://archive.is/u3HLx](https://archive.is/u3HLx). [↗](https://archive.is/u3HLx)
30. Eva AM van Dis, Johan Bollen, Willem Zuidema, Robert van Rooij, and Claudi L Bockting. 2023. Chatgpt: five priorities for research. Nature, 614(7947):224–226. [find](https://scholar.google.com/scholar?q=Eva+AM+van+Dis%2C+Johan+Bollen%2C+Willem+Zuidema%2C+Robert+van+Rooij%2C+and+Claudi+L+Bockting%2E+2023%2E+Chatgpt%3A+five+priorities+for+research%2E+Nature%2C+614%287947%29%3A224%E2%80%93226%2E)
31. Patrick Y Wu, Joshua A Tucker, Jonathan Nagler, and Solomon Messing. 2023. Large language models can be used to estimate the ideologies of politicians in a zero-shot learning setting. Preprint arXiv:2303.12057. — [arXiv:2303.12057](https://arxiv.org/abs/2303.12057) [find](https://scholar.google.com/scholar?q=Patrick+Y+Wu%2C+Joshua+A+Tucker%2C+Jonathan+Nagler%2C+and+Solomon+Messing%2E+2023%2E+Large+language+models+can+be+used+to+estimate+the+ideologies+of+politicians+in+a+zero%2Dshot+learning+setting%2E+Preprint+arXiv%3A2)
32. Bowen Zhang, Daijun Ding, and Liwen Jing. 2022. How would stance detection techniques evolve after the launch of chatgpt? Preprint arXiv:2212.14548. [find](https://scholar.google.com/scholar?q=Bowen+Zhang%2C+Daijun+Ding%2C+and+Liwen+Jing%2E+2022%2E+How+would+stance+detection+techniques+evolve+after+the+launch+of+chatgpt%3F+Preprint+arXiv%3A2212%2E14548%2E)

*Updated 31 October 2026*