---
title: "A Systematic Survey and Critical Review on Evaluating Large"
description: "The authors first lay out a typical evaluation workflow in three stages: the evaluation setup (choosing… 211 references, cited by 41 papers here."
authors: ["Md Tahmid Rahman Laskar", "Sawsan Alqahtani", "M Saiful Bari", "Mizanur Rahman", "Mohammad Abdullah Matin Khan", "Haidar Khan", "Israt Jahan", "Amran Bhuiyan", "Chee Wei Tan", "Md Rizwan Parvez", "Enamul Hoque", "Shafiq Joty", "Jimmy Xiangji Huang"]
year: 2024
venue: "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2024.emnlp-main.764/"
doi: "10.18653/v1/2024.emnlp-main.764"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/laskar-2024-systematic-survey-critical-review-evaluating/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/laskar-2024-systematic-survey-critical-review-evaluating/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations

# A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations

**Authors:** [Md Tahmid Rahman Laskar](https://md.research.elamur.ai/free/en/authors/md-tahmid-rahman-laskar/), [Sawsan Alqahtani](https://md.research.elamur.ai/free/en/authors/sawsan-alqahtani/), [M Saiful Bari](https://md.research.elamur.ai/free/en/authors/m-saiful-bari/), [Mizanur Rahman](https://md.research.elamur.ai/free/en/authors/mizanur-rahman/), [Mohammad Abdullah Matin Khan](https://md.research.elamur.ai/free/en/authors/mohammad-abdullah-matin-khan/), Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, [Md Rizwan Parvez](https://md.research.elamur.ai/free/en/authors/md-rizwan-parvez/), [Enamul Hoque](https://md.research.elamur.ai/free/en/authors/enamul-hoque/), [Shafiq Joty](https://md.research.elamur.ai/free/en/authors/shafiq-joty/) and [Jimmy Xiangji Huang](https://md.research.elamur.ai/free/en/authors/jimmy-xiangji-huang/) · **Year:** 2024 · **Venue:** [Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2024](https://md.research.elamur.ai/free/en/venues/emnlp/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.emnlp-main.764/) · [PDF](https://aclanthology.org/2024.emnlp-main.764.pdf) · [DOI 10.18653/v1/2024.emnlp-main.764](https://doi.org/10.18653/v1/2024.emnlp-main.764) · [OpenAlex](https://openalex.org/W4404782882)

**Citations:** 64 citations (OpenAlex, 2026-10-14) · [41](https://md.research.elamur.ai/free/en/papers/laskar-2024-systematic-survey-critical-review-evaluating/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
LLM evaluation results often conflict across studies because each study sets up its experiments differently. This EMNLP 2024 paper by Laskar et al. examines where those inconsistencies come from. The authors first lay out a typical evaluation workflow in three stages: the evaluation setup (choosing benchmarks and models), response generation (prompt design and decoding parameters), and evaluation methodology (parsing scripts and automatic, human, or LLM-based judging). They then review the literature at each stage along three dimensions: reproducibility, reliability, and robustness. Recurring problems include unreleased data subsets, unstated model versions, undisclosed prompts and decoding settings, and missing parsing code. Closed-source models that change silently, and closed models used as judges, make these problems worse. The paper ends with stage-by-stage recommendations and supporting code. It offers a practical checklist for anyone benchmarking LLMs. The authors note that their analysis centres on English, so multilingual and low-resource evaluation remains an open area.

## Key points
- The paper cites an analysis by Balloccu et al. (2024) of 212 papers:
  - prompts were available in 90.6% of them;
  - code was available in 53.3%;
  - both prompts and code were available in 50.0%;
  - the model version was stated in only 29.3%;
  - just 20.7% made fair model comparisons.
- Conflicting results are common. For example, adding output-length limits to prompts lowered scores on SAMSum in one study (Qin et al., 2023) but raised them in another (Laskar et al., 2023a).
- Gains from few-shot prompting reported by model developers often fail to appear in independent evaluations. This raises questions about undisclosed prompt engineering or selective choice of examples.
- The recommendations include:
  - publishing parsing scripts and model outputs;
  - testing parsers on varied response types;
  - pairing ROUGE with BERTScore;
  - using several LLMs as juries rather than a single judge;
  - tracking latency and GPU usage with pyNVML.
- The data and code are available at github.com/ntunlp/Critical-Review-of-LLM-Eval. Training-phase issues and multimodal benchmarks are outside the paper's scope.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Nanyang Technological University](https://md.research.elamur.ai/free/en/universities/nanyang-technological-university/) — University, Singapore, Singapore · 156 works · 25,644 citations
- [York University](https://md.research.elamur.ai/free/en/universities/york-university/) — University, Toronto, Canada · 13 works · 3,075 citations
- [York University](https://md.research.elamur.ai/free/en/universities/york-university-us/) — University, York, United States · 4 works · 381 citations
- [Royal Bank of Canada](https://md.research.elamur.ai/free/en/labs/royal-bank-of-canada/) — Research institute, Toronto, Canada · 3 works · 561 citations
- [Princess Nourah bint Abdulrahman University](https://md.research.elamur.ai/free/en/universities/princess-nourah-bint-abdulrahman-university/) — University, Riyadh, Saudi Arabia · 4 works · 112 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works

## Cite

Cite as: Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty and Jimmy Xiangji Huang. 2024. A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{laskar2024systematic,
  title = {{A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations}},
  author = {Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty and Jimmy Xiangji Huang},
  booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.764},
  url = {https://doi.org/10.18653/v1/2024.emnlp-main.764}
}
```

- [← previous R-Judge: Benchmarking Safety Risk Awareness for LLM Agents](https://md.research.elamur.ai/free/en/papers/yuan-2024-r-judge-benchmarking-safety-risk/)
- [next → Arcee's MergeKit: A Toolkit for Merging Large Language Models](https://md.research.elamur.ai/free/en/papers/goddard-2024-arcee-s-mergekit-toolkit-merging/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

211 references, 113 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ahmed Abdelali, Hamdy Mubarak, Shammur Chowdhury, Maram Hasanain, Basel Mousi, Sabri Boughorbel, Samir Abdaljalil, Yassine El Kheir, Daniel Izham, Fahim Dalvi, et al. 2024. [Larabench: Benchmarking arabic ai with large language models](https://md.research.elamur.ai/free/en/papers/abdelali-2024-larabench-benchmarking-arabic-ai-large/). In Proceedings of the 18th Conference of the European Chapter of…
2. Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. 2024. [Phi-3 technical report: A highly capable language model locally on your phone](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-3-technical-report-highly/). arXiv preprint arXiv:2404.14219.
3. Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Maxamed Axmed, et al. 2023. [Mega: Multilingual evaluation of generative ai](https://md.research.elamur.ai/free/en/papers/ahuja-2023-mega-multilingual-evaluation-generative-ai/). arXiv preprint arXiv:2303.12528.
4. Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. 2023. [The falcon series of open language models](https://md.research.elamur.ai/free/en/papers/almazrouei-2023-falcon-series-open-language-models/). arXiv preprint arXiv:2311.16867.
5. Iñigo Alonso, Maite Oronoz, and Rodrigo Agerri. 2024. Medexpqa: Multilingual benchmarking of large language models for medical question answering. arXiv preprint arXiv:2404.05590. — [arXiv:2404.05590](https://arxiv.org/abs/2404.05590) [find](https://scholar.google.com/scholar?q=I%C3%B1igo+Alonso%2C+Maite+Oronoz%2C+and+Rodrigo+Agerri%2E+2024%2E+Medexpqa%3A+Multilingual+benchmarking+of+large+language+models+for+medical+question+answering%2E+arXiv+preprint+arXiv%3A2404%2E05590%2E+%E2%80%94+%5BarXiv%3A2404%2E05590%5D)
6. Norah Alzahrani, Hisham Abdullah Alyahya, Yazeed Alnumay, Sultan Alrashed, Shaykhah Alsubaie, Yusef Almushaykeh, Faisal Mirza, Nouf Alotaibi, Nora Altwairesh, Areeb Alowisheq, M Saiful Bari, and Haidar Khan. 2024. When benchmarks are targets: Revealing the sensitivity of large language model leaderb… [find](https://scholar.google.com/scholar?q=Norah+Alzahrani%2C+Hisham+Abdullah+Alyahya%2C+Yazeed+Alnumay%2C+Sultan+Alrashed%2C+Shaykhah+Alsubaie%2C+Yusef+Almushaykeh%2C+Faisal+Mirza%2C+Nouf+Alotaibi%2C+Nora+Altwairesh%2C+Areeb+Alowisheq%2C+M+Saiful+Bari%2C+and+Haida)
7. Shengnan An, Bo Zhou, Zeqi Lin, Qiang Fu, Bei Chen, Nanning Zheng, Weizhu Chen, and Jian-Guang Lou. 2023. Skill-based few-shot selection for in-context learning. arXiv preprint arXiv:2305.14210. [find](https://scholar.google.com/scholar?q=Shengnan+An%2C+Bo+Zhou%2C+Zeqi+Lin%2C+Qiang+Fu%2C+Bei+Chen%2C+Nanning+Zheng%2C+Weizhu+Chen%2C+and+Jian%2DGuang+Lou%2E+2023%2E+Skill%2Dbased+few%2Dshot+selection+for+in%2Dcontext+learning%2E+arXiv+preprint+arXiv%3A2305%2E14210%2E)
8. Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. [Palm 2 technical report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/). arXiv preprint arXiv:2305.10403.
9. Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku. [find](https://scholar.google.com/scholar?q=Anthropic%2E+2024%2E+The+claude+3+model+family%3A+Opus%2C+sonnet%2C+haiku%2E)
10. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). arXiv preprint arXiv:2108.07732.
11. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. [Qwen-vl: A frontier large vision-language model with versatile abilities](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-vl-versatile-vision-language/). arXiv preprint arXiv:2308.12966.
12. Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, et al. 2024. Benchmarking foundation models with language-model-as-an-examiner. Advances in Neural Information Processing Systems, 36. [find](https://scholar.google.com/scholar?q=Yushi+Bai%2C+Jiahao+Ying%2C+Yixin+Cao%2C+Xin+Lv%2C+Yuze+He%2C+Xiaozhi+Wang%2C+Jifan+Yu%2C+Kaisheng+Zeng%2C+Yijia+Xiao%2C+Haozhe+Lyu%2C+et+al%2E+2024%2E+Benchmarking+foundation+models+with+language%2Dmodel%2Das%2Dan%2Dexaminer%2E+Advan)
13. Simone Balloccu, Patrícia Schmidtová, Mateusz Lango, and Ondřej Dušek. 2024. [Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source llms](https://md.research.elamur.ai/free/en/papers/balloccu-2024-leak-cheat-repeat-data-contamination/). arXiv preprint arXiv:2402.03927.
14. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. [A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity](https://md.research.elamur.ai/free/en/papers/bang-2023-multitask-multilingual-multimodal-evaluation-chatgpt/). arXiv preprint arXiv:2302.04023.
15. Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, Anthony DiPofi, Julen Etxaniz, Benjamin Fattori, Jessica Zosa Forde, Charles Foster, Mimansa Jaiswal, Wilson Y. Lee, Haonan Li, Charles… [find](https://scholar.google.com/scholar?q=Stella+Biderman%2C+Hailey+Schoelkopf%2C+Lintang+Sutawika%2C+Leo+Gao%2C+Jonathan+Tow%2C+Baber+Abbasi%2C+Alham+Fikri+Aji%2C+Pawan+Sasanka+Ammanamanchi%2C+Sidney+Black%2C+Jordan+Clive%2C+Anthony+DiPofi%2C+Julen+Etxaniz%2C+Benja)
16. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. [Piqa: Reasoning about physical commonsense in natural language](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/). In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439.
17. Meriem Boubdir, Edward Kim, Beyza Ermis, Sara Hooker, and Marzieh Fadaee. 2023. Elo uncovered: Robustness and best practices in language model evaluation. arXiv preprint arXiv:2311.17295. [find](https://scholar.google.com/scholar?q=Meriem+Boubdir%2C+Edward+Kim%2C+Beyza+Ermis%2C+Sara+Hooker%2C+and+Marzieh+Fadaee%2E+2023%2E+Elo+uncovered%3A+Robustness+and+best+practices+in+language+model+evaluation%2E+arXiv+preprint+arXiv%3A2311%2E17295%2E)
18. Sabri Boughorbel, MD Parvez, and Majd Hawasly. 2024. Improving language models trained with translated data via continual pre-training and dictionary learning analysis. arXiv preprint arXiv:2405.14277. — [arXiv:2405.14277](https://arxiv.org/abs/2405.14277) [find](https://scholar.google.com/scholar?q=Sabri+Boughorbel%2C+MD+Parvez%2C+and+Majd+Hawasly%2E+2024%2E+Improving+language+models+trained+with+translated+data+via+continual+pre%2Dtraining+and+dictionary+learning+analysis%2E+arXiv+preprint+arXiv%3A2405%2E14277)
19. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). arXiv preprint arXiv:2005.14165.
20. Jannis Bulian, Christian Buck, Wojciech Gajewski, Benjamin Boerschinger, and Tal Schuster. 2022. [Tomayto, tomahto. beyond token-level answer equivalence for question answering evaluation](https://md.research.elamur.ai/free/en/papers/bulian-2022-tomayto-tomahto-beyond-token-level/). arXiv preprint arXiv:2202.07654.
21. Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. 2024. A survey on mixture of experts. arXiv preprint arXiv:2407.06204. [find](https://scholar.google.com/scholar?q=Weilin+Cai%2C+Juyong+Jiang%2C+Fan+Wang%2C+Jing+Tang%2C+Sunghun+Kim%2C+and+Jiayi+Huang%2E+2024%2E+A+survey+on+mixture+of+experts%2E+arXiv+preprint+arXiv%3A2407%2E06204%2E)
22. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2024. [A survey on evaluation of large language models](https://md.research.elamur.ai/free/en/papers/chang-2023-survey-evaluation-large-language-models/). ACM Trans. Intell. Syst. Technol., 15(3).
23. Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J Pappas, Florian Tramer, et al. 2024. [Jailbreakbench: An open robustness benchmark for jailbreaking large language models](https://md.research.elamur.ai/free/en/papers/chao-2024-jailbreakbench-open-robustness-benchmark-jailbreaking/). arXiv preprint arXiv:2404.…
24. Anthony Chen, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2020. Mocha: A dataset for training and evaluating generative reading comprehension metrics. arXiv preprint arXiv:2010.03636. [find](https://scholar.google.com/scholar?q=Anthony+Chen%2C+Gabriel+Stanovsky%2C+Sameer+Singh%2C+and+Matt+Gardner%2E+2020%2E+Mocha%3A+A+dataset+for+training+and+evaluating+generative+reading+comprehension+metrics%2E+arXiv+preprint+arXiv%3A2010%2E03636%2E)
25. Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. 2024a. [Benchmarking large language models in retrieval-augmented generation](https://md.research.elamur.ai/free/en/papers/chen-2023-benchmarking-large-language-models-retrieval/). In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17754– 17762.
26. Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. 2024b. [Are we on the right way for evaluating large vision-language models?](https://md.research.elamur.ai/free/en/papers/chen-2024-right-way-evaluating-large-vision/) arXiv preprint arXiv:2403.20330.
27. Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. 2023a. [Sharegpt4v: Improving large multi-modal models with better captions](https://md.research.elamur.ai/free/en/papers/chen-2023-sharegpt4v-improving-large-multi-modal/). arXiv preprint arXiv:2311.12793.
28. Lingjiao Chen, Matei Zaharia, and James Zou. 2023b. How is ChatGPT’s behavior changing over time? arXiv preprint arXiv:2307.09009. [find](https://scholar.google.com/scholar?q=Lingjiao+Chen%2C+Matei+Zaharia%2C+and+James+Zou%2E+2023b%2E+How+is+ChatGPT%E2%80%99s+behavior+changing+over+time%3F+arXiv+preprint+arXiv%3A2307%2E09009%2E)
29. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374.
30. Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. 2023c. [Longlora: Efficient fine-tuning of long-context large language models](https://md.research.elamur.ai/free/en/papers/chen-2023-longlora-efficient-fine-tuning-long/). arXiv preprint arXiv:2309.12307.
31. Steffi Chern, Ethan Chern, Graham Neubig, and Pengfei Liu. 2024. Can large language models be trusted for evaluation? scalable meta-evaluation of llms as evaluators via agent debate. arXiv preprint arXiv:2401.16788. [find](https://scholar.google.com/scholar?q=Steffi+Chern%2C+Ethan+Chern%2C+Graham+Neubig%2C+and+Pengfei+Liu%2E+2024%2E+Can+large+language+models+be+trusted+for+evaluation%3F+scalable+meta%2Devaluation+of+llms+as+evaluators+via+agent+debate%2E+arXiv+preprint+ar)
32. Cheng-Han Chiang and Hung-yi Lee. 2023. Can large language models be an alternative to human evaluations? arXiv preprint arXiv:2305.01937. [find](https://scholar.google.com/scholar?q=Cheng%2DHan+Chiang+and+Hung%2Dyi+Lee%2E+2023%2E+Can+large+language+models+be+an+alternative+to+human+evaluations%3F+arXiv+preprint+arXiv%3A2305%2E01937%2E)
33. Sanghyun Choo and Wonjoon Kim. 2023. A study on the evaluation of tokenizer performance in natural language processing. Applied Artificial Intelligence, 37(1):2175112. [find](https://scholar.google.com/scholar?q=Sanghyun+Choo+and+Wonjoon+Kim%2E+2023%2E+A+study+on+the+evaluation+of+tokenizer+performance+in+natural+language+processing%2E+Applied+Artificial+Intelligence%2C+37%281%29%3A2175112%2E)
34. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416. [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Le+Hou%2C+Shayne+Longpre%2C+Barret+Zoph%2C+Yi+Tay%2C+William+Fedus%2C+Eric+Li%2C+Xuezhi+Wang%2C+Mostafa+Dehghani%2C+Siddhartha+Brahma%2C+et+al%2E+2022%2E+Scaling+instruction%2Dfinetuned+language+models%2E+arXi)
35. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. [Boolq: Exploring the surprising difficulty of natural yes/no questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/). arXiv preprint arXiv:1905.10044.
36. Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. [TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages](https://md.research.elamur.ai/free/en/papers/clark-2020-tydi-qa-benchmark-information-seeking/). arXiv preprint arXiv:2003.05002.
37. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). arXiv preprint arXiv:1803.05457.
38. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. 2024. [Instructblip: Towards general-purpose vision-language models with instruction tuning](https://md.research.elamur.ai/free/en/papers/dai-2023-instructblip-towards-general-purpose-vision/). Advances in Neural Information Processing Systems, 36.
39. Fahim Dalvi, Maram Hasanain, Sabri Boughorbel, Basel Mousi, Samir Abdaljalil, Nizi Nazar, Ahmed Abdelali, Shammur Absar Chowdhury, Hamdy Mubarak, Ahmed Ali, et al. 2023. Llmebench: A flexible framework for accelerating llms benchmarking. arXiv preprint arXiv:2308.04945. [find](https://scholar.google.com/scholar?q=Fahim+Dalvi%2C+Maram+Hasanain%2C+Sabri+Boughorbel%2C+Basel+Mousi%2C+Samir+Abdaljalil%2C+Nizi+Nazar%2C+Ahmed+Abdelali%2C+Shammur+Absar+Chowdhury%2C+Hamdy+Mubarak%2C+Ahmed+Ali%2C+et+al%2E+2023%2E+Llmebench%3A+A+flexible+framewor)
40. Bosheng Ding, Chengwei Qin, Linlin Liu, Lidong Bing, Shafiq Joty, and Boyang Li. 2022. [Is gpt-3 a good data annotator?](https://md.research.elamur.ai/free/en/papers/ding-2022-gpt-3-good-data-annotator/) arXiv preprint arXiv:2212.10450.
41. Markus Freitag and Yaser Al-Onaizan. 2017. [Beam search strategies for neural machine translation](https://md.research.elamur.ai/free/en/papers/freitag-2017-beam-search-strategies-neural-machine/). arXiv preprint arXiv:1702.01806.
42. Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. 2023a. [Mme: A comprehensive evaluation benchmark for multimodal large language models](https://md.research.elamur.ai/free/en/papers/fu-2023-mme-comprehensive-evaluation-benchmark-multimodal/). arXiv preprint arXiv:2306.13394.
43. Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. 2024a. Blink: Multimodal large language models can see but not perceive. arXiv preprint arXiv:2404.12390. [find](https://scholar.google.com/scholar?q=Xingyu+Fu%2C+Yushi+Hu%2C+Bangzheng+Li%2C+Yu+Feng%2C+Haoyu+Wang%2C+Xudong+Lin%2C+Dan+Roth%2C+Noah+A+Smith%2C+Wei%2DChiu+Ma%2C+and+Ranjay+Krishna%2E+2024a%2E+Blink%3A+Multimodal+large+language+models+can+see+but+not+perceive%2E+ar)
44. Xue-Yong Fu, Md Tahmid Rahman Laskar, Cheng Chen, and Shashi Bhushan Tn. 2023b. Are large language models reliable judges? a study on the factuality evaluation capabilities of LLMs. In Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM), pages 310–316, Sin… [find](https://scholar.google.com/scholar?q=Xue%2DYong+Fu%2C+Md+Tahmid+Rahman+Laskar%2C+Cheng+Chen%2C+and+Shashi+Bhushan+Tn%2E+2023b%2E+Are+large+language+models+reliable+judges%3F+a+study+on+the+factuality+evaluation+capabilities+of+LLMs%2E+In+Proceedings+of+)
45. Xue-Yong Fu, Md Tahmid Rahman Laskar, Elena Khasanova, Cheng Chen, and Shashi Bhushan TN. 2024b. Tiny titans: Can smaller large language models punch above their weight in the real world for meeting summarization? arXiv preprint arXiv:2402.00841. [find](https://scholar.google.com/scholar?q=Xue%2DYong+Fu%2C+Md+Tahmid+Rahman+Laskar%2C+Elena+Khasanova%2C+Cheng+Chen%2C+and+Shashi+Bhushan+TN%2E+2024b%2E+Tiny+titans%3A+Can+smaller+large+language+models+punch+above+their+weight+in+the+real+world+for+meeting+s)
46. Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang, and Xiaojun Wan. 2023a. Human-like summarization evaluation with ChatGPT. arXiv preprint arXiv:2304.02554. [find](https://scholar.google.com/scholar?q=Mingqi+Gao%2C+Jie+Ruan%2C+Renliang+Sun%2C+Xunjian+Yin%2C+Shiping+Yang%2C+and+Xiaojun+Wan%2E+2023a%2E+Human%2Dlike+summarization+evaluation+with+ChatGPT%2E+arXiv+preprint+arXiv%3A2304%2E02554%2E)
47. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023b. [Retrieval-augmented generation for large language models: A survey](https://md.research.elamur.ai/free/en/papers/gao-2023-retrieval-augmented-generation-large-language/). arXiv preprint arXiv:2312.10997.
48. Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, et al. 2024. Are we done with mmlu? arXiv preprint arXiv:2406.04127. [find](https://scholar.google.com/scholar?q=Aryo+Pradipta+Gema%2C+Joshua+Ong+Jun+Leang%2C+Giwon+Hong%2C+Alessio+Devoto%2C+Alberto+Carlo+Maria+Mancino%2C+Rohit+Saxena%2C+Xuanli+He%2C+Yu+Zhao%2C+Xiaotang+Du%2C+Mohammad+Reza+Ghasemi+Madani%2C+et+al%2E+2024%2E+Are+we+done)
49. Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. [Samsum corpus: A human-annotated dialogue dataset for abstractive summarization](https://md.research.elamur.ai/free/en/papers/gliwa-2019-samsum-corpus-human-annotated-dialogue/). In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70–79.
50. Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Synnaeve. 2024. Better & faster large language models via multi-token prediction. arXiv preprint arXiv:2404.19737. [find](https://scholar.google.com/scholar?q=Fabian+Gloeckle%2C+Badr+Youbi+Idrissi%2C+Baptiste+Rozi%C3%A8re%2C+David+Lopez%2DPaz%2C+and+Gabriel+Synnaeve%2E+2024%2E+Better+%26+faster+large+language+models+via+multi%2Dtoken+prediction%2E+arXiv+preprint+arXiv%3A2404%2E19737%2E)
51. Shahriar Golchin and Mihai Surdeanu. 2023. [Time travel in llms: Tracing data contamination in large language models](https://md.research.elamur.ai/free/en/papers/golchin-2023-time-travel-llms-tracing-data/). arXiv preprint arXiv:2308.08493.
52. Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. 2023. [Hallusionbench: An advanced diagnostic suite for entangled language hallucination & visual illusion in large vision-language models](https://md.research.elamur.ai/free/en/papers/guan-2023-hallusionbench-advanced-diagnostic-suite-entangled/). arXiv preprint arXiv:2310.1…
53. Yue Guo, Zian Xu, and Yi Yang. 2023a. Is Chat-GPT a financial expert? evaluating language models on financial natural language processing. arXiv preprint arXiv:2310.12664. — [arXiv:2310.12664](https://arxiv.org/abs/2310.12664) [find](https://scholar.google.com/scholar?q=Yue+Guo%2C+Zian+Xu%2C+and+Yi+Yang%2E+2023a%2E+Is+Chat%2DGPT+a+financial+expert%3F+evaluating+language+models+on+financial+natural+language+processing%2E+arXiv+preprint+arXiv%3A2310%2E12664%2E+%E2%80%94+%5BarXiv%3A2310%2E12664%5D%28https%3A%2F)
54. Zishan Guo, Renren Jin, Chuang Liu, Yufei Huang, Dan Shi, Linhao Yu, Yan Liu, Jiaxuan Li, Bojian Xiong, Deyi Xiong, et al. 2023b. [Evaluating large language models: A comprehensive survey](https://md.research.elamur.ai/free/en/papers/guo-2023-evaluating-large-language-models-comprehensive/). arXiv:2310.19736. arXiv preprint Rishav Hada, Varun Gumma, Adrian de Wynter, Harshita Diddee, Mohamed Ahmed, Mon…
55. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. [Measuring coding challenge competence with APPS](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-coding-challenge-competence-apps/). In Proceedings of the Neural Information Processing Systems Track on Datasets and Benc…
56. Mohammed Saidul Islam, Raian Rahman, Ahmed Masry, Md Tahmid Rahman Laskar, Mir Tafseer Nayeem, and Enamul Hoque. 2024b. Are large vision language models up to the challenge of chart comprehension and reasoning? an extensive investigation into the capabilities and limitations of lvlms. arXiv preprint… [find](https://scholar.google.com/scholar?q=Mohammed+Saidul+Islam%2C+Raian+Rahman%2C+Ahmed+Masry%2C+Md+Tahmid+Rahman+Laskar%2C+Mir+Tafseer+Nayeem%2C+and+Enamul+Hoque%2E+2024b%2E+Are+large+vision+language+models+up+to+the+challenge+of+chart+comprehension+and+)
57. Israt Jahan, Md Tahmid Rahman Laskar, Chun Peng, and Jimmy Huang. 2023. Evaluation of ChatGPT on biomedical tasks: A zero-shot comparison with fine-tuned generative transformers. In The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 326–336, Toronto, Canada. A… [find](https://scholar.google.com/scholar?q=Israt+Jahan%2C+Md+Tahmid+Rahman+Laskar%2C+Chun+Peng%2C+and+Jimmy+Huang%2E+2023%2E+Evaluation+of+ChatGPT+on+biomedical+tasks%3A+A+zero%2Dshot+comparison+with+fine%2Dtuned+generative+transformers%2E+In+The+22nd+Workshop+)
58. Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2020a. [Aligning ai with shared human values](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-aligning-ai-shared-human-values/). arXiv preprint arXiv:2008.02275.
59. Israt Jahan, Md Tahmid Rahman Laskar, Chun Peng, and Jimmy Xiangji Huang. 2024. A comprehensive evaluation of large language models on benchmark biomedical text processing tasks. Computers in Biology and Medicine, page 108189. [find](https://scholar.google.com/scholar?q=Israt+Jahan%2C+Md+Tahmid+Rahman+Laskar%2C+Chun+Peng%2C+and+Jimmy+Xiangji+Huang%2E+2024%2E+A+comprehensive+evaluation+of+large+language+models+on+benchmark+biomedical+text+processing+tasks%2E+Computers+in+Biology+)
60. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020b. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). arXiv preprint arXiv:2009.03300.
61. Fred Jelinek, Robert L Mercer, Lalit R Bahl, and James K Baker. 1977. Perplexity—a measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America, 62(S1):S63–S63. [find](https://scholar.google.com/scholar?q=Fred+Jelinek%2C+Robert+L+Mercer%2C+Lalit+R+Bahl%2C+and+James+K+Baker%2E+1977%2E+Perplexity%E2%80%94a+measure+of+the+difficulty+of+speech+recognition+tasks%2E+The+Journal+of+the+Acoustical+Society+of+America%2C+62%28S1%29%3AS63%E2%80%93S)
62. Yutong Hu, Quzhe Huang, Mingxu Tao, Chen Zhang, and Yansong Feng. 2024. Can perplexity reflect large language model’s ability in long text understanding? arXiv preprint arXiv:2405.06105. — [arXiv:2405.06105](https://arxiv.org/abs/2405.06105) [find](https://scholar.google.com/scholar?q=Yutong+Hu%2C+Quzhe+Huang%2C+Mingxu+Tao%2C+Chen+Zhang%2C+and+Yansong+Feng%2E+2024%2E+Can+perplexity+reflect+large+language+model%E2%80%99s+ability+in+long+text+understanding%3F+arXiv+preprint+arXiv%3A2405%2E06105%2E+%E2%80%94+%5BarXiv%3A2405)
63. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+et+al%2E+2023%2E+Mist)
64. Hui Huang, Yingqi Qu, Jing Liu, Muyun Yang, and Tiejun Zhao. 2024a. An empirical study of llm-as-a-judge for llm evaluation: Fine-tuned judge models are task-specific classifiers. arXiv preprint arXiv:2403.02839. — [arXiv:2403.02839](https://arxiv.org/abs/2403.02839) [find](https://scholar.google.com/scholar?q=Hui+Huang%2C+Yingqi+Qu%2C+Jing+Liu%2C+Muyun+Yang%2C+and+Tiejun+Zhao%2E+2024a%2E+An+empirical+study+of+llm%2Das%2Da%2Djudge+for+llm+evaluation%3A+Fine%2Dtuned+judge+models+are+task%2Dspecific+classifiers%2E+arXiv+preprint+arXiv)
65. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. [Swe-bench: Can language models resolve real-world github issues?](https://md.research.elamur.ai/free/en/papers/jimenez-2024-swe-bench-can-language-models/) arXiv preprint arXiv:2310.06770.
66. Wei Huang, Xudong Ma, Haotong Qin, Xingyu Zheng, Chengtao Lv, Hong Chen, Jie Luo, Xiaojuan Qi, Xianglong Liu, and Michele Magno. 2024b. How good are low-bit quantized llama3 models? an empirical study. arXiv preprint arXiv:2404.14047. — [arXiv:2404.14047](https://arxiv.org/abs/2404.14047) [find](https://scholar.google.com/scholar?q=Wei+Huang%2C+Xudong+Ma%2C+Haotong+Qin%2C+Xingyu+Zheng%2C+Chengtao+Lv%2C+Hong+Chen%2C+Jie+Luo%2C+Xiaojuan+Qi%2C+Xianglong+Liu%2C+and+Michele+Magno%2E+2024b%2E+How+good+are+low%2Dbit+quantized+llama3+models%3F+an+empirical+study)
67. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. [Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611.
68. Md Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez. 2024a. Mapcoder: Multi-agent code generation for competitive problem solving. arXiv preprint arXiv:2405.11403. [find](https://scholar.google.com/scholar?q=Md+Ashraful+Islam%2C+Mohammed+Eunus+Ali%2C+and+Md+Rizwan+Parvez%2E+2024a%2E+Mapcoder%3A+Multi%2Dagent+code+generation+for+competitive+problem+solving%2E+arXiv+preprint+arXiv%3A2405%2E11403%2E)
69. Mohsinul Kabir, Mohammed Saidul Islam, Md Tahmid Rahman Laskar, Mir Tafseer Nayeem, M Saiful Bari, and Enamul Hoque. 2023. [find](https://scholar.google.com/scholar?q=Mohsinul+Kabir%2C+Mohammed+Saidul+Islam%2C+Md+Tahmid+Rahman+Laskar%2C+Mir+Tafseer+Nayeem%2C+M+Saiful+Bari%2C+and+Enamul+Hoque%2E+2023%2E)
70. Benllmeval: A comprehensive evaluation into the potentials and pitfalls of large language models on bengali NLP. arXiv preprint arXiv:2309.13173. — [arXiv:2309.13173](https://arxiv.org/abs/2309.13173) [find](https://scholar.google.com/scholar?q=Benllmeval%3A+A+comprehensive+evaluation+into+the+potentials+and+pitfalls+of+large+language+models+on+bengali+NLP%2E+arXiv+preprint+arXiv%3A2309%2E13173%2E+%E2%80%94+%5BarXiv%3A2309%2E13173%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2309%2E13173%29)
71. Tom Kocmi and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality. arXiv preprint arXiv:2302.14520. [find](https://scholar.google.com/scholar?q=Tom+Kocmi+and+Christian+Federmann%2E+2023%2E+Large+language+models+are+state%2Dof%2Dthe%2Dart+evaluators+of+translation+quality%2E+arXiv+preprint+arXiv%3A2302%2E14520%2E)
72. Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. 2016. [A diagram is worth a dozen images](https://md.research.elamur.ai/free/en/papers/kembhavi-2016-diagram-worth-dozen-images/). In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pages 235–251. Springer.
73. Jan Kocoń, Igor Cichecki, Oliwier Kaszyca, Mateusz Kochanek, Dominika Szydło, Joanna Baran, Julita Bielaniewicz, Marcin Gruza, Arkadiusz Janz, Kamil Kanclerz, et al. 2023. [Chatgpt: Jack of all trades, master of none](https://md.research.elamur.ai/free/en/papers/kocon-2023-chatgpt-jack-all-trades-master/). Information Fusion, 99:101861.
74. Zachary Kenton, Noah Y Siegel, János Kramár, Jonah Brown-Cohen, Samuel Albanie, Jannis Bulian, Rishabh Agarwal, David Lindner, Yunhao Tang, Noah D Goodman, et al. 2024. On scalable oversight with weak llms judging strong llms. arXiv preprint arXiv:2407.04622. — [arXiv:2407.04622](https://arxiv.org/abs/2407.04622) [find](https://scholar.google.com/scholar?q=Zachary+Kenton%2C+Noah+Y+Siegel%2C+J%C3%A1nos+Kram%C3%A1r%2C+Jonah+Brown%2DCohen%2C+Samuel+Albanie%2C+Jannis+Bulian%2C+Rishabh+Agarwal%2C+David+Lindner%2C+Yunhao+Tang%2C+Noah+D+Goodman%2C+et+al%2E+2024%2E+On+scalable+oversight+with+weak)
75. Mohammad Abdullah Matin Khan, M Saiful Bari, Xuan Long Do, Weishi Wang, Md Rizwan Parvez, and Shafiq Joty. 2023. [xcodeeval: A large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval](https://md.research.elamur.ai/free/en/papers/khan-2023-xcodeeval-large-scale-multilingual-multitask/). arXiv preprint arXiv:2303.03004.
76. Md Tawkat Islam Khondaker, Abdul Waheed, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. [Gptaraeval: a comprehensive evaluation of ChatGPT on arabic NLP](https://md.research.elamur.ai/free/en/papers/khondaker-2023-gptaraeval-comprehensive-evaluation-chatgpt-arabic/). arXiv preprint arXiv:2305.14976.
77. Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024a. [Prometheus 2: An open source language model specialized in evaluating other language models](https://md.research.elamur.ai/free/en/papers/kim-2024-prometheus-2-open-source-language/). arXiv preprint arXiv:2405.01535.
78. Thomas Kosch and Sebastian Feger. 2024. Risk or chance? large language models and reproducibility in human-computer interaction research. arXiv preprint arXiv:2404.15782. — [arXiv:2404.15782](https://arxiv.org/abs/2404.15782) [find](https://scholar.google.com/scholar?q=Thomas+Kosch+and+Sebastian+Feger%2E+2024%2E+Risk+or+chance%3F+large+language+models+and+reproducibility+in+human%2Dcomputer+interaction+research%2E+arXiv+preprint+arXiv%3A2404%2E15782%2E+%E2%80%94+%5BarXiv%3A2404%2E15782%5D%28https%3A%2F%2F)
79. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. [Natural questions: a benchmark for question answering research](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/). Transactions of the Association for Computational Linguistic…
80. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. [Race: Large-scale reading comprehension dataset from examinations](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/). arXiv preprint arXiv:1704.04683.
81. Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Huu Nguyen. 2023. [ChatGPT beyond english: Towards a comprehensive evaluation of large language models in multilingual learning](https://md.research.elamur.ai/free/en/papers/lai-2023-chatgpt-beyond-english-comprehensive-evaluation/). arXiv preprint arXiv:2304.05613.
82. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. 2024. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787. [find](https://scholar.google.com/scholar?q=Nathan+Lambert%2C+Valentina+Pyatkin%2C+Jacob+Morrison%2C+LJ+Miranda%2C+Bill+Yuchen+Lin%2C+Khyathi+Chandu%2C+Nouha+Dziri%2C+Sachin+Kumar%2C+Tom+Zick%2C+Yejin+Choi%2C+et+al%2E+2024%2E+Rewardbench%3A+Evaluating+reward+models+for+)
83. Tae Soo Kim, Yoonjoo Lee, Jamin Shin, Young-Ho Kim, and Juho Kim. 2024b. Evallm: Interactive evaluation of large language model prompts on user-defined criteria. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pages 1–21. [find](https://scholar.google.com/scholar?q=Tae+Soo+Kim%2C+Yoonjoo+Lee%2C+Jamin+Shin%2C+Young%2DHo+Kim%2C+and+Juho+Kim%2E+2024b%2E+Evallm%3A+Interactive+evaluation+of+large+language+model+prompts+on+user%2Ddefined+criteria%2E+In+Proceedings+of+the+CHI+Conference+o)
84. Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, et al. 2023. [Measuring faithfulness in chain-of-thought reasoning](https://md.research.elamur.ai/free/en/papers/lanham-2023-measuring-faithfulness-chain-thought-reasoning/). arXiv preprint arXiv:2307.13702.
85. Masamune Kobayashi, Masato Mita, and Mamoru Komachi. 2024. Large language models are state-of-the-art evaluator for grammatical error correction. arXiv preprint arXiv:2403.17540. [find](https://scholar.google.com/scholar?q=Masamune+Kobayashi%2C+Masato+Mita%2C+and+Mamoru+Komachi%2E+2024%2E+Large+language+models+are+state%2Dof%2Dthe%2Dart+evaluator+for+grammatical+error+correction%2E+arXiv+preprint+arXiv%3A2403%2E17540%2E)
86. Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, and Jimmy Huang. 2023a. [A systematic study and comprehensive evaluation of ChatGPT on benchmark datasets](https://md.research.elamur.ai/free/en/papers/laskar-2023-systematic-study-comprehensive-evaluation-chatgpt/). In Findings of the Association for Computational Linguistics: ACL 2023, pages 431–469, Toronto, Cana…
87. Md Tahmid Rahman Laskar, Cheng Chen, Xue-yong Fu, and Shashi Bhushan Tn. 2022a. Improving named entity recognition in telephone conversations via effective active learning with human in the loop. arXiv preprint arXiv:2211.01354. [find](https://scholar.google.com/scholar?q=Md+Tahmid+Rahman+Laskar%2C+Cheng+Chen%2C+Xue%2Dyong+Fu%2C+and+Shashi+Bhushan+Tn%2E+2022a%2E+Improving+named+entity+recognition+in+telephone+conversations+via+effective+active+learning+with+human+in+the+loop%2E+arXi)
88. Md Tahmid Rahman Laskar, Xue-Yong Fu, Cheng Chen, and Shashi Bhushan Tn. 2023b. Building real-world meeting summarization systems using large language models: A practical perspective. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 343… [find](https://scholar.google.com/scholar?q=Md+Tahmid+Rahman+Laskar%2C+Xue%2DYong+Fu%2C+Cheng+Chen%2C+and+Shashi+Bhushan+Tn%2E+2023b%2E+Building+real%2Dworld+meeting+summarization+systems+using+large+language+models%3A+A+practical+perspective%2E+In+Proceedings+o)
89. Md Tahmid Rahman Laskar, Enamul Hoque, and Jimmy Xiangji Huang. 2022b. [Domain adaptation with pre-trained transformers for query-focused abstractive text summarization](https://md.research.elamur.ai/free/en/papers/laskar-2022-domain-adaptation-pre-trained-transformers/). Computational Linguistics, 48(2):279–320.
90. Md Tahmid Rahman Laskar, Xiangji Huang, and Enamul Hoque. 2020. Contextualized embeddings based transformer encoder for sentence similarity modeling in answer selection task. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 5505–5514. [find](https://scholar.google.com/scholar?q=Md+Tahmid+Rahman+Laskar%2C+Xiangji+Huang%2C+and+Enamul+Hoque%2E+2020%2E+Contextualized+embeddings+based+transformer+encoder+for+sentence+similarity+modeling+in+answer+selection+task%2E+In+Proceedings+of+the+Twe)
91. Md Tahmid Rahman Laskar, Mizanur Rahman, Israt Jahan, Enamul Hoque, and Jimmy Huang. 2023c. Can large language models fix data annotation errors? an empirical study using debatepedia for query-focused text summarization. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages… [find](https://scholar.google.com/scholar?q=Md+Tahmid+Rahman+Laskar%2C+Mizanur+Rahman%2C+Israt+Jahan%2C+Enamul+Hoque%2C+and+Jimmy+Huang%2E+2023c%2E+Can+large+language+models+fix+data+annotation+errors%3F+an+empirical+study+using+debatepedia+for+query%2Dfocused)
92. Md Tahmid Rahman Laskar, Mizanur Rahman, Israt Jahan, Enamul Hoque, and Jimmy Huang. 2023d. CQSumDP: a ChatGPT-annotated resource for query-focused abstractive summarization based on debatepedia. arXiv preprint arXiv:2305.06147. — [arXiv:2305.06147](https://arxiv.org/abs/2305.06147) [find](https://scholar.google.com/scholar?q=Md+Tahmid+Rahman+Laskar%2C+Mizanur+Rahman%2C+Israt+Jahan%2C+Enamul+Hoque%2C+and+Jimmy+Huang%2E+2023d%2E+CQSumDP%3A+a+ChatGPT%2Dannotated+resource+for+query%2Dfocused+abstractive+summarization+based+on+debatepedia%2E+arXi)
93. Chris van der Lee, Albert Gatt, Emiel van Miltenburg, and Emiel Krahmer. 2021. Human evaluation of automatically generated text: Current trends and best practice guidelines. Computer Speech & Language, 67:101151. [find](https://scholar.google.com/scholar?q=Chris+van+der+Lee%2C+Albert+Gatt%2C+Emiel+van+Miltenburg%2C+and+Emiel+Krahmer%2E+2021%2E+Human+evaluation+of+automatically+generated+text%3A+Current+trends+and+best+practice+guidelines%2E+Computer+Speech+%26+Language)
94. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. [Retrieval-augmented generation for knowledge-intensive NLP tasks](https://md.research.elamur.ai/free/en/papers/lewis-2020-retrieval-augmented-generation-knowledge-intensive/). Advances in Neural Information Processing Systems, 33:9459–9474.
95. Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. 2023a. Seed-bench-2: Benchmarking multimodal large language models. arXiv preprint arXiv:2311.17092. — [arXiv:2311.17092](https://arxiv.org/abs/2311.17092) [find](https://scholar.google.com/scholar?q=Bohao+Li%2C+Yuying+Ge%2C+Yixiao+Ge%2C+Guangzhi+Wang%2C+Rui+Wang%2C+Ruimao+Zhang%2C+and+Ying+Shan%2E+2023a%2E+Seed%2Dbench%2D2%3A+Benchmarking+multimodal+large+language+models%2E+arXiv+preprint+arXiv%3A2311%2E17092%2E+%E2%80%94+%5BarXiv%3A2311)
96. Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023b. [Seed-bench: Benchmarking multimodal llms with generative comprehension](https://md.research.elamur.ai/free/en/papers/li-2023-seed-bench-benchmarking-multimodal-llms/). arXiv preprint arXiv:2307.16125.
97. Changmao Li and Jeffrey Flanigan. 2023. Task contamination: Language models may not be few-shot anymore. arXiv preprint arXiv:2312.16337. [find](https://scholar.google.com/scholar?q=Changmao+Li+and+Jeffrey+Flanigan%2E+2023%2E+Task+contamination%3A+Language+models+may+not+be+few%2Dshot+anymore%2E+arXiv+preprint+arXiv%3A2312%2E16337%2E)
98. Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023c. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161. [find](https://scholar.google.com/scholar?q=Raymond+Li%2C+Loubna+Ben+Allal%2C+Yangtian+Zi%2C+Niklas+Muennighoff%2C+Denis+Kocetkov%2C+Chenghao+Mou%2C+Marc+Marone%2C+Christopher+Akiki%2C+Jia+Li%2C+Jenny+Chim%2C+et+al%2E+2023c%2E+Starcoder%3A+may+the+source+be+with+you%21+ar)
99. Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023d. [Evaluating object hallucination in large vision-language models](https://md.research.elamur.ai/free/en/papers/li-2023-evaluating-object-hallucination-large-vision/). arXiv preprint arXiv:2305.10355.
100. Yinheng Li, Shaofei Wang, Han Ding, and Hang Chen. 2023e. Large language models in finance: A survey. In Proceedings of the Fourth ACM International Conference on AI in Finance, pages 374–382. [find](https://scholar.google.com/scholar?q=Yinheng+Li%2C+Shaofei+Wang%2C+Han+Ding%2C+and+Hang+Chen%2E+2023e%2E+Large+language+models+in+finance%3A+A+survey%2E+In+Proceedings+of+the+Fourth+ACM+International+Conference+on+AI+in+Finance%2C+pages+374%E2%80%93382%2E)
101. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James M… — [Competition-Level Code Generation with AlphaCode](https://md.research.elamur.ai/free/en/papers/li-2022-competition-level-code-generation-alphacode/)
102. Zongxia Li, Ishani Mondal, Yijun Liang, Huy Nghiem, and Jordan Lee Boyd-Graber. 2024. Pedants (precise evaluations of diverse answer nominee text for skinflints): Efficient evaluation analysis and benchmarking for open-domain question answering. [find](https://scholar.google.com/scholar?q=Zongxia+Li%2C+Ishani+Mondal%2C+Yijun+Liang%2C+Huy+Nghiem%2C+and+Jordan+Lee+Boyd%2DGraber%2E+2024%2E+Pedants+%28precise+evaluations+of+diverse+answer+nominee+text+for+skinflints%29%3A+Efficient+evaluation+analysis+and+ben)
103. Xiao Liu, Xuanyu Lei, Shengyuan Wang, Yue Huang, Zhuoer Feng, Bosi Wen, Jiale Cheng, Pei Ke, Yifan Xu, Weng Lam Tam, et al. 2023c. [Alignbench: Benchmarking chinese alignment of large language models](https://md.research.elamur.ai/free/en/papers/liu-2023-alignbench-benchmarking-chinese-alignment/). arXiv preprint arXiv:2311.18743.
104. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110. [find](https://scholar.google.com/scholar?q=Percy+Liang%2C+Rishi+Bommasani%2C+Tony+Lee%2C+Dimitris+Tsipras%2C+Dilara+Soylu%2C+Michihiro+Yasunaga%2C+Yian+Zhang%2C+Deepak+Narayanan%2C+Yuhuai+Wu%2C+Ananya+Kumar%2C+et+al%2E+2022%2E+Holistic+evaluation+of+language+models%2E+)
105. Yi Liu, Lianzhe Huang, Shicheng Li, Sishuo Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. 2023d. Recall: A benchmark for llms robustness against external counterfactual knowledge. arXiv preprint arXiv:2311.08147. — [arXiv:2311.08147](https://arxiv.org/abs/2311.08147) [find](https://scholar.google.com/scholar?q=Yi+Liu%2C+Lianzhe+Huang%2C+Shicheng+Li%2C+Sishuo+Chen%2C+Hao+Zhou%2C+Fandong+Meng%2C+Jie+Zhou%2C+and+Xu+Sun%2E+2023d%2E+Recall%3A+A+benchmark+for+llms+robustness+against+external+counterfactual+knowledge%2E+arXiv+preprint+)
106. Chin-Yew Lin. 2004. [ROUGE: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text summarization branches out, pages 74–81.
107. Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. [Truthfulqa: Measuring how models mimic human falsehoods](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/). arXiv preprint arXiv:2109.07958.
108. Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. 2023a. [Mitigating hallucination in large multi-modal models via robust instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-mitigating-hallucination-large-multi-modal/). In The Twelfth International Conference on Learning Representations.
109. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023b. [Improved baselines with visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-improved-baselines-visual-instruction-tuning/). arXiv preprint arXiv:2310.03744.
110. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024a. [Visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/). Advances in neural information processing systems, 36.
111. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2024b. [Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation](https://md.research.elamur.ai/free/en/papers/liu-2023-your-code-generated-chatgpt-really/). Advances in Neural Information Processing Systems, 36.
112. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. 2023e. [Mmbench: Is your multi-modal model an all-around player?](https://md.research.elamur.ai/free/en/papers/liu-2023-mmbench-multi-modal-model-all/) arXiv preprint ar…
113. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2023. [Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts](https://md.research.elamur.ai/free/en/papers/lu-2023-mathvista-evaluating-mathematical-reasoning-foundation/). arXiv preprint arXiv:2310.02255.
114. Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507–2521. [find](https://scholar.google.com/scholar?q=Pan+Lu%2C+Swaroop+Mishra%2C+Tanglin+Xia%2C+Liang+Qiu%2C+Kai%2DWei+Chang%2C+Song%2DChun+Zhu%2C+Oyvind+Tafjord%2C+Peter+Clark%2C+and+Ashwin+Kalyan%2E+2022%2E+Learn+to+explain%3A+Multimodal+reasoning+via+thought+chains+for+scienc)
115. Yujie Lu, Xianjun Yang, Xiujun Li, Xin Eric Wang, and William Yang Wang. 2024. Llm-score: Unveiling the power of large language models in text-to-image synthesis evaluation. Advances in Neural Information Processing Systems, 36. [find](https://scholar.google.com/scholar?q=Yujie+Lu%2C+Xianjun+Yang%2C+Xiujun+Li%2C+Xin+Eric+Wang%2C+and+William+Yang+Wang%2E+2024%2E+Llm%2Dscore%3A+Unveiling+the+power+of+large+language+models+in+text%2Dto%2Dimage+synthesis+evaluation%2E+Advances+in+Neural+Informa)
116. Gen Luo, Yiyi Zhou, Tianhe Ren, Shengxin Chen, Xiaoshuai Sun, and Rongrong Ji. 2024. Cheap and quick: Efficient vision-language instruction tuning for large language models. Advances in Neural Information Processing Systems, 36. [find](https://scholar.google.com/scholar?q=Gen+Luo%2C+Yiyi+Zhou%2C+Tianhe+Ren%2C+Shengxin+Chen%2C+Xiaoshuai+Sun%2C+and+Rongrong+Ji%2E+2024%2E+Cheap+and+quick%3A+Efficient+vision%2Dlanguage+instruction+tuning+for+large+language+models%2E+Advances+in+Neural+Informa)
117. Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023. [ChatGPT as a factual inconsistency evaluator for text summarization](https://md.research.elamur.ai/free/en/papers/luo-2023-chatgpt-factual-inconsistency-evaluator-text/). arXiv preprint arXiv:2303.15621.
118. Yuanjie Lyu, Zhiyu Li, Simin Niu, Feiyu Xiong, Bo Tang, Wenjin Wang, Hao Wu, Huanyong Liu, Tong Xu, and Enhong Chen. 2024. Crud-rag: A comprehensive chinese benchmark for retrieval-augmented generation of large language models. arXiv preprint arXiv:2401.17043. — [arXiv:2401.17043](https://arxiv.org/abs/2401.17043) [find](https://scholar.google.com/scholar?q=Yuanjie+Lyu%2C+Zhiyu+Li%2C+Simin+Niu%2C+Feiyu+Xiong%2C+Bo+Tang%2C+Wenjin+Wang%2C+Hao+Wu%2C+Huanyong+Liu%2C+Tong+Xu%2C+and+Enhong+Chen%2E+2024%2E+Crud%2Drag%3A+A+comprehensive+chinese+benchmark+for+retrieval%2Daugmented+generatio)
119. Oscar Mañas, Benno Krojer, and Aishwarya Agrawal. 2024. Improving automatic vqa evaluation using large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4171–4179. [find](https://scholar.google.com/scholar?q=Oscar+Ma%C3%B1as%2C+Benno+Krojer%2C+and+Aishwarya+Agrawal%2E+2024%2E+Improving+automatic+vqa+evaluation+using+large+language+models%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+volume+38%2C+pag)
120. Rui Mao, Guanyi Chen, Xulang Zhang, Frank Guerin, and Erik Cambria. 2023. [Gpteval: A survey on assessments of chatgpt and gpt-4](https://md.research.elamur.ai/free/en/papers/mao-2024-gpteval-survey-assessments-chatgpt-gpt/). arXiv preprint arXiv:2308.12488.
121. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789. [find](https://scholar.google.com/scholar?q=Todor+Mihaylov%2C+Peter+Clark%2C+Tushar+Khot%2C+and+Ashish+Sabharwal%2E+2018%2E+Can+a+suit+of+armor+conduct+electricity%3F+a+new+dataset+for+open+book+question+answering%2E+arXiv+preprint+arXiv%3A1809%2E02789%2E)
122. Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. 2024. [Large language models: A survey](https://md.research.elamur.ai/free/en/papers/minaee-2024-large-language-models-survey/).
123. Abhika Mishra, Akari Asai, Vidhisha Balachandran, Yizhong Wang, Graham Neubig, Yulia Tsvetkov, and Hannaneh Hajishirzi. 2024. [Fine-grained hallucination detection and editing for language models](https://md.research.elamur.ai/free/en/papers/mishra-2024-fine-grained-hallucination-detection-editing/). arXiv preprint arXiv:2401.06855.
124. Jinjie Ni, Fuzhao Xue, Xiang Yue, Yuntian Deng, Mahir Shah, Kabir Jain, Graham Neubig, and Yang You. 2024. [Mixeval: Deriving wisdom of the crowd from llm benchmark mixtures](https://md.research.elamur.ai/free/en/papers/ni-2024-mixeval-deriving-wisdom-crowd-llm/).
125. OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/).
126. Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. [Chartqa: A benchmark for question answering about charts with visual and logical reasoning](https://md.research.elamur.ai/free/en/papers/masry-2022-chartqa-benchmark-question-answering-about/). arXiv preprint arXiv:2203.10244.
127. Yonatan Oren, Nicole Meister, Niladri Chatterji, Faisal Ladhak, and Tatsunori B Hashimoto. 2023. Proving test set contamination in black box language models. arXiv preprint arXiv:2310.17623. [find](https://scholar.google.com/scholar?q=Yonatan+Oren%2C+Nicole+Meister%2C+Niladri+Chatterji%2C+Faisal+Ladhak%2C+and+Tatsunori+B+Hashimoto%2E+2023%2E+Proving+test+set+contamination+in+black+box+language+models%2E+arXiv+preprint+arXiv%3A2310%2E17623%2E)
128. Ahmed Masry, Mehrad Shahmohammadi, Md Rizwan Parvez, Enamul Hoque, and Shafiq Joty. 2024. Chartinstruct: Instruction tuning for chart comprehension and reasoning. arXiv preprint arXiv:2403.09028. [find](https://scholar.google.com/scholar?q=Ahmed+Masry%2C+Mehrad+Shahmohammadi%2C+Md+Rizwan+Parvez%2C+Enamul+Hoque%2C+and+Shafiq+Joty%2E+2024%2E+Chartinstruct%3A+Instruction+tuning+for+chart+comprehension+and+reasoning%2E+arXiv+preprint+arXiv%3A2403%2E09028%2E)
129. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35:27730–27744.
130. Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. 2021. [Docvqa: A dataset for vqa on document images](https://md.research.elamur.ai/free/en/papers/mathew-2020-docvqa-dataset-vqa-document-images/). In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209.
131. Alexandre Matton, Tom Sherborne, Dennis Aumiller, Elena Tommasone, Milad Alizadeh, Jingyi He, Raymond Ma, Maxime Voisin, Ellen Gilsenan-McMahon, and Matthias Gallé. 2024. On leakage of code generation evaluation datasets. [find](https://scholar.google.com/scholar?q=Alexandre+Matton%2C+Tom+Sherborne%2C+Dennis+Aumiller%2C+Elena+Tommasone%2C+Milad+Alizadeh%2C+Jingyi+He%2C+Raymond+Ma%2C+Maxime+Voisin%2C+Ellen+Gilsenan%2DMcMahon%2C+and+Matthias+Gall%C3%A9%2E+2024%2E+On+leakage+of+code+generation)
132. Timothy R McIntosh, Teo Susnjak, Tong Liu, Paul Watters, and Malka N Halgamuge. 2024. Inadequacies of large language model benchmarks in the era of generative artificial intelligence. arXiv preprint arXiv:2402.09880. — [arXiv:2402.09880](https://arxiv.org/abs/2402.09880) [find](https://scholar.google.com/scholar?q=Timothy+R+McIntosh%2C+Teo+Susnjak%2C+Tong+Liu%2C+Paul+Watters%2C+and+Malka+N+Halgamuge%2E+2024%2E+Inadequacies+of+large+language+model+benchmarks+in+the+era+of+generative+artificial+intelligence%2E+arXiv+preprint+a)
133. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
134. Md Rizwan Parvez. 2024. Evidence to generate (e2g): A single-agent two-step prompting for context grounded and retrieval augmented reasoning. arXiv preprint arXiv:2401.05787. [find](https://scholar.google.com/scholar?q=Md+Rizwan+Parvez%2E+2024%2E+Evidence+to+generate+%28e2g%29%3A+A+single%2Dagent+two%2Dstep+prompting+for+context+grounded+and+retrieval+augmented+reasoning%2E+arXiv+preprint+arXiv%3A2401%2E05787%2E)
135. Md Rizwan Parvez, Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. [Retrieval augmented code generation and summarization](https://md.research.elamur.ai/free/en/papers/parvez-2021-retrieval-augmented-code-generation-summarization/). In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2719–2734, Punta Cana, Dominican Republic. Association for Computational Li…
136. Md Rizwan Parvez, Tolga Bolukbasi, Kai-Wei Chang, and Venkatesh Saligrama. 2019. Robust text classifier on test-time budgets. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJC… [find](https://scholar.google.com/scholar?q=Md+Rizwan+Parvez%2C+Tolga+Bolukbasi%2C+Kai%2DWei+Chang%2C+and+Venkatesh+Saligrama%2E+2019%2E+Robust+text+classifier+on+test%2Dtime+budgets%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Lang)
137. Md Rizwan Parvez, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2018. Building language models for text with named entities. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2373–2383, Melbourne, Australia. Association… [find](https://scholar.google.com/scholar?q=Md+Rizwan+Parvez%2C+Saikat+Chakraborty%2C+Baishakhi+Ray%2C+and+Kai%2DWei+Chang%2E+2018%2E+Building+language+models+for+text+with+named+entities%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Co)
138. Md Rizwan Parvez and Kai-Wei Chang. 2021. Evaluating the values of sources in transfer learning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5084–5116, Online. Association for Computational L… [find](https://scholar.google.com/scholar?q=Md+Rizwan+Parvez+and+Kai%2DWei+Chang%2E+2021%2E+Evaluating+the+values+of+sources+in+transfer+learning%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+of+the+Association+for+Computationa)
139. Md Rizwan Parvez, Jianfeng Chi, Wasi Uddin Ahmad, Yuan Tian, and Kai-Wei Chang. 2023. Retrieval enhanced data augmentation for question answering on privacy policies. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 201–210, Dubrov… [find](https://scholar.google.com/scholar?q=Md+Rizwan+Parvez%2C+Jianfeng+Chi%2C+Wasi+Uddin+Ahmad%2C+Yuan+Tian%2C+and+Kai%2DWei+Chang%2E+2023%2E+Retrieval+enhanced+data+augmentation+for+question+answering+on+privacy+policies%2E+In+Proceedings+of+the+17th+Confer)
140. Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. 2022. [Red teaming language models with language models](https://md.research.elamur.ai/free/en/papers/perez-2022-red-teaming-language-models-language/). arXiv preprint arXiv:2202.03286.
141. Sarah Masud Preum, Md Rizwan Parvez, Kai-Wei Chang, and John Stankovic. 2018. A corpus of drug usage guidelines annotated with type of advice. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). [find](https://scholar.google.com/scholar?q=Sarah+Masud+Preum%2C+Md+Rizwan+Parvez%2C+Kai%2DWei+Chang%2C+and+John+Stankovic%2E+2018%2E+A+corpus+of+drug+usage+guidelines+annotated+with+type+of+advice%2E+In+Proceedings+of+the+Eleventh+International+Conference+o)
142. Xiao Pu, Mingqi Gao, and Xiaojun Wan. 2023. Summarization is (almost) dead. arXiv preprint arXiv:2309.09558. — [arXiv:2309.09558](https://arxiv.org/abs/2309.09558) [find](https://scholar.google.com/scholar?q=Xiao+Pu%2C+Mingqi+Gao%2C+and+Xiaojun+Wan%2E+2023%2E+Summarization+is+%28almost%29+dead%2E+arXiv+preprint+arXiv%3A2309%2E09558%2E+%E2%80%94+%5BarXiv%3A2309%2E09558%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2309%2E09558%29)
143. Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. [Is ChatGPT a general-purpose natural language processing task solver?](https://md.research.elamur.ai/free/en/papers/qin-2023-chatgpt-general-purpose-natural-language/) arXiv preprint arXiv:2302.06476.
144. Haoyi Qiu, Wenbo Hu, Zi-Yi Dou, and Nanyun Peng. 2024. Valor-eval: Holistic coverage and faithfulness evaluation of large vision-language models. arXiv preprint arXiv:2404.13874. [find](https://scholar.google.com/scholar?q=Haoyi+Qiu%2C+Wenbo+Hu%2C+Zi%2DYi+Dou%2C+and+Nanyun+Peng%2E+2024%2E+Valor%2Deval%3A+Holistic+coverage+and+faithfulness+evaluation+of+large+vision%2Dlanguage+models%2E+arXiv+preprint+arXiv%3A2404%2E13874%2E)
145. Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. Improving language understanding by generative pre-training. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+Ilya+Sutskever%2C+et+al%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E)
146. Raian Rahman, Rizvi Hasan, Abdullah Al Farhad, Md. Tahmid Rahman Laskar, Md. Hamjajul Ashmafee, and Abu Raihan Mostofa Kamal. 2023. [Chartsumm: A comprehensive benchmark for automatic chart summarization of long and short summaries](https://md.research.elamur.ai/free/en/papers/rahman-2023-chartsumm-comprehensive-benchmark-automatic-chart/). Proceedings of the Canadian Conference on Artificial Intelligence.
147. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. [Know what you don’t know: Unanswerable questions for squad](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/). arXiv preprint arXiv:1806.03822.
148. Mathieu Ravaut, Bosheng Ding, Fangkai Jiao, Hailin Chen, Xingxuan Li, Ruochen Zhao, Chengwei Qin, Caiming Xiong, and Shafiq Joty. 2024. How much are llms contaminated? a comprehensive survey and the llmsanitize library. arXiv preprint arXiv:2404.00699. [find](https://scholar.google.com/scholar?q=Mathieu+Ravaut%2C+Bosheng+Ding%2C+Fangkai+Jiao%2C+Hailin+Chen%2C+Xingxuan+Li%2C+Ruochen+Zhao%2C+Chengwei+Qin%2C+Caiming+Xiong%2C+and+Shafiq+Joty%2E+2024%2E+How+much+are+llms+contaminated%3F+a+comprehensive+survey+and+the+l)
149. Vipula Rawte, Amit Sheth, and Amitava Das. 2023. [A survey of hallucination in large foundation models](https://md.research.elamur.ai/free/en/papers/rawte-2023-survey-hallucination-large-foundation-models/). arXiv preprint arXiv:2309.05922.
150. Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023. [Code llama: Open foundation models for code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/). arXiv preprint arXiv:2308.12950.
151. Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. [How good is your tokenizer? on the monolingual performance of multilingual language models](https://md.research.elamur.ai/free/en/papers/rust-2021-how-good-is-your-tokenizer/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Join…
152. Mobashir Sadat, Zhengyu Zhou, Lukas Lange, Jun Araki, Arsalan Gundroo, Bingqing Wang, Rakesh Menon, Md Parvez, and Zhe Feng. 2023. DelucionQA: Detecting hallucinations in domain-specific question answering. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 822–835, Sing… [find](https://scholar.google.com/scholar?q=Mobashir+Sadat%2C+Zhengyu+Zhou%2C+Lukas+Lange%2C+Jun+Araki%2C+Arsalan+Gundroo%2C+Bingqing+Wang%2C+Rakesh+Menon%2C+Md+Parvez%2C+and+Zhe+Feng%2E+2023%2E+DelucionQA%3A+Detecting+hallucinations+in+domain%2Dspecific+question+answ)
153. Sander Schulhoff, Jeremy Pinto, Anaum Khan, Louis-François Bouchard, Chenglei Si, Svetlina Anati, Valen Tagliabue, Anson Kost, Christopher Carnahan, and Jordan Boyd-Graber. 2023. Ignore this title and hackaprompt: Exposing systemic vulnerabilities of llms through a global prompt hacking competition.… [find](https://scholar.google.com/scholar?q=Sander+Schulhoff%2C+Jeremy+Pinto%2C+Anaum+Khan%2C+Louis%2DFran%C3%A7ois+Bouchard%2C+Chenglei+Si%2C+Svetlina+Anati%2C+Valen+Tagliabue%2C+Anson+Kost%2C+Christopher+Carnahan%2C+and+Jordan+Boyd%2DGraber%2E+2023%2E+Ignore+this+title+and)
154. Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2023. Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. arXiv preprint arXiv:2310.11324. [find](https://scholar.google.com/scholar?q=Melanie+Sclar%2C+Yejin+Choi%2C+Yulia+Tsvetkov%2C+and+Alane+Suhr%2E+2023%2E+Quantifying+language+models%E2%80%99+sensitivity+to+spurious+features+in+prompt+design+or%3A+How+i+learned+to+start+worrying+about+prompt+formatt)
155. Oscar Sainz, Jon Ander Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023. [NLP evaluation in trouble: On the need to measure llm data contamination for each benchmark](https://md.research.elamur.ai/free/en/papers/sainz-2023-nlp-evaluation-trouble-need-measure/). arXiv preprint arXiv:2310.18018.
156. Shreya Shankar, JD Zamfirescu-Pereira, Björn Hartmann, Aditya G Parameswaran, and Ian Arawjo. 2024. Who validates the validators? aligning llm-assisted evaluation of llm outputs with human preferences. arXiv preprint arXiv:2404.12272. — [arXiv:2404.12272](https://arxiv.org/abs/2404.12272) [find](https://scholar.google.com/scholar?q=Shreya+Shankar%2C+JD+Zamfirescu%2DPereira%2C+Bj%C3%B6rn+Hartmann%2C+Aditya+G+Parameswaran%2C+and+Ian+Arawjo%2E+2024%2E+Who+validates+the+validators%3F+aligning+llm%2Dassisted+evaluation+of+llm+outputs+with+human+preferences)
157. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99– 106. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+2021%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+Communications+of+the+ACM%2C+64%289%29%3A99%E2%80%93+106%2E)
158. Chenhui Shen, Liying Cheng, Xuan-Phi Nguyen, Yang You, and Lidong Bing. 2023. [Large language models are not yet human-level evaluators for abstractive summarization](https://md.research.elamur.ai/free/en/papers/shen-2023-large-language-models-are-not/). In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 4215–4233.
159. SambaNova. 2024. Samba-coe v0.3: The power of routing ml models at scale. [find](https://scholar.google.com/scholar?q=SambaNova%2E+2024%2E+Samba%2Dcoe+v0%2E3%3A+The+power+of+routing+ml+models+at+scale%2E)
160. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019. [Socialiqa: Commonsense reasoning about social interactions](https://md.research.elamur.ai/free/en/papers/sap-2019-socialiqa-commonsense-reasoning-about-social/). arXiv preprint arXiv:1904.09728.
161. Sander Schulhoff, Michael Ilie, Nishant Balepur, Konstantine Kahadze, Amanda Liu, Chenglei Si, Yinheng Li, Aayush Gupta, Hyo-Jung Han, Sevien Schulhoff, Pranav Sandeep Dulepet, Saurav Vidyadhara, Dayeon Ki, Sweta Agrawal, Chau Pham, Gerson C. Kroiz, Feileen Li, Hudson Tao, Ashay Srivastava, Hevander… — [A Thorough Examination of Decoding Methods in the Era of LLMs](https://md.research.elamur.ai/free/en/papers/shi-2024-thorough-examination-decoding-methods-era/)
162. Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2023. Detecting pretraining data from large language models. arXiv preprint arXiv:2310.16789. [find](https://scholar.google.com/scholar?q=Weijia+Shi%2C+Anirudh+Ajith%2C+Mengzhou+Xia%2C+Yangsibo+Huang%2C+Daogao+Liu%2C+Terra+Blevins%2C+Danqi+Chen%2C+and+Luke+Zettlemoyer%2E+2023%2E+Detecting+pretraining+data+from+large+language+models%2E+arXiv+preprint+arXiv%3A)
163. Rickard Stureborg, Dimitris Alikaniotis, and Yoshi Suhara. 2024. [Large language models are inconsistent and biased evaluators](https://md.research.elamur.ai/free/en/papers/stureborg-2024-large-language-models-inconsistent-biased/). arXiv preprint arXiv:2405.01724.
164. Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, et al. 2024. [Trustllm: Trustworthiness in large language models](https://md.research.elamur.ai/free/en/papers/huang-2024-trustllm-trustworthiness-large-language-models/). arXiv preprint arXiv:2401.05561.
165. Alon Talmor and Jonathan Berant. 2018. [The web as a knowledge-base for answering complex questions](https://md.research.elamur.ai/free/en/papers/talmor-2018-web-as-knowledge-base-answering/). arXiv preprint arXiv:1803.06643.
166. Yixuan Tang and Yi Yang. 2024. [Multihop-rag: Benchmarking retrieval-augmented generation for multi-hop queries](https://md.research.elamur.ai/free/en/papers/tang-2024-multihop-rag-benchmarking-retrieval-augmented/). arXiv preprint arXiv:2401.15391.
167. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805.
168. Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295. [find](https://scholar.google.com/scholar?q=Gemma+Team%2C+Thomas+Mesnard%2C+Cassidy+Hardin%2C+Robert+Dadashi%2C+Surya+Bhupatiraju%2C+Shreya+Pathak%2C+Laurent+Sifre%2C+Morgane+Rivi%C3%A8re%2C+Mihir+Sanjay+Kale%2C+Juliette+Love%2C+et+al%2E+2024%2E+Gemma%3A+Open+models+based+on)
169. Simone Tedeschi, Felix Friedrich, Patrick Schramowski, Kristian Kersting, Roberto Navigli, Huu Nguyen, and Bo Li. 2024. Alert: A comprehensive benchmark for assessing large language models’ safety through red teaming. arXiv preprint arXiv:2404.08676. — [arXiv:2404.08676](https://arxiv.org/abs/2404.08676) [find](https://scholar.google.com/scholar?q=Simone+Tedeschi%2C+Felix+Friedrich%2C+Patrick+Schramowski%2C+Kristian+Kersting%2C+Roberto+Navigli%2C+Huu+Nguyen%2C+and+Bo+Li%2E+2024%2E+Alert%3A+A+comprehensive+benchmark+for+assessing+large+language+models%E2%80%99+safety+thr)
170. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971.
171. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288.
172. Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. 2024. [Replacing judges with juries: Evaluating llm generations with a panel of diverse models](https://md.research.elamur.ai/free/en/papers/verga-2024-replacing-judges-juries-evaluating-llm/). arXiv preprint arXiv:2404.18796.
173. Jindong Wang, Xixu Hu, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang, Haojun Huang, Wei Ye, Xiubo Geng, et al. 2023a. On the robustness of ChatGPT: An adversarial and out-of-distribution perspective. arXiv preprint arXiv:2302.12095. [find](https://scholar.google.com/scholar?q=Jindong+Wang%2C+Xixu+Hu%2C+Wenxin+Hou%2C+Hao+Chen%2C+Runkai+Zheng%2C+Yidong+Wang%2C+Linyi+Yang%2C+Haojun+Huang%2C+Wei+Ye%2C+Xiubo+Geng%2C+et+al%2E+2023a%2E+On+the+robustness+of+ChatGPT%3A+An+adversarial+and+out%2Dof%2Ddistribution)
174. Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023b. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926. [find](https://scholar.google.com/scholar?q=Peiyi+Wang%2C+Lei+Li%2C+Liang+Chen%2C+Zefan+Cai%2C+Dawei+Zhu%2C+Binghuai+Lin%2C+Yunbo+Cao%2C+Qi+Liu%2C+Tianyu+Liu%2C+and+Zhifang+Sui%2E+2023b%2E+Large+language+models+are+not+fair+evaluators%2E+arXiv+preprint+arXiv%3A2305%2E1792)
175. Tong Wang, Ninad Kulkarni, and Yanjun Qi. 2024a. Less is more for improving automatic evaluation of factual consistency. arXiv preprint arXiv:2404.06579. [find](https://scholar.google.com/scholar?q=Tong+Wang%2C+Ninad+Kulkarni%2C+and+Yanjun+Qi%2E+2024a%2E+Less+is+more+for+improving+automatic+evaluation+of+factual+consistency%2E+arXiv+preprint+arXiv%3A2404%2E06579%2E)
176. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024b. [Mmlu-pro: A more robust and challenging multi-task language understanding benchmar…](https://md.research.elamur.ai/free/en/papers/wang-2024-mmlu-pro-more-robust-challenging/)
177. Zhiruo Wang, Jun Araki, Zhengbao Jiang, Md Rizwan Parvez, and Graham Neubig. 2023c. [Learning to filter context for retrieval-augmented generation](https://md.research.elamur.ai/free/en/papers/wang-2023-learning-filter-context-retrieval-augmented/). arXiv preprint arXiv:2311.08377.
178. Jason Wei, Yi Tay, and Quoc V Le. 2022a. Inverse scaling can become u-shaped. arXiv preprint arXiv:2211.02011. [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Yi+Tay%2C+and+Quoc+V+Le%2E+2022a%2E+Inverse+scaling+can+become+u%2Dshaped%2E+arXiv+preprint+arXiv%3A2211%2E02011%2E)
179. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022b. [Chain of thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). arXiv preprint arXiv:2201.11903.
180. Sheng-Lun Wei, Cheng-Kuang Wu, Hen-Hsen Huang, and Hsin-Hsi Chen. 2024. Unveiling selection biases: Exploring order and token sensitivity in large language models. arXiv preprint arXiv:2406.03009. [find](https://scholar.google.com/scholar?q=Sheng%2DLun+Wei%2C+Cheng%2DKuang+Wu%2C+Hen%2DHsen+Huang%2C+and+Hsin%2DHsi+Chen%2E+2024%2E+Unveiling+selection+biases%3A+Exploring+order+and+token+sensitivity+in+large+language+models%2E+arXiv+preprint+arXiv%3A2406%2E03009%2E)
181. Man-Fai Wong, Shangxin Guo, Ching-Nam Hang, Siu-Wai Ho, and Chee-Wei Tan. 2023. [find](https://scholar.google.com/scholar?q=Man%2DFai+Wong%2C+Shangxin+Guo%2C+Ching%2DNam+Hang%2C+Siu%2DWai+Ho%2C+and+Chee%2DWei+Tan%2E+2023%2E)
182. Natural Xing Xie, and Yue Zhang. 2022. Glue-x: Evaluating natural language understanding models from an out-of-distribution generalization perspective. arXiv preprint arXiv:2211.08073. language generation and understanding of big code for ai-assisted programming: A review. Entropy, 25(6):888. [find](https://scholar.google.com/scholar?q=Natural+Xing+Xie%2C+and+Yue+Zhang%2E+2022%2E+Glue%2Dx%3A+Evaluating+natural+language+understanding+models+from+an+out%2Dof%2Ddistribution+generalization+perspective%2E+arXiv+preprint+arXiv%3A2211%2E08073%2E+language+genera)
183. Honghan Wu, Minhong Wang, Jinge Wu, Farah Francis, Yun-Hsuan Chang, Alex Shavick, Hang Dong, Michael TC Poon, Natalie Fitzpatrick, Adam P Levine, et al. 2022. A survey on clinical natural language processing in the united kingdom from 2007 to 2022. NPJ digital medicine, 5(1):186. [find](https://scholar.google.com/scholar?q=Honghan+Wu%2C+Minhong+Wang%2C+Jinge+Wu%2C+Farah+Francis%2C+Yun%2DHsuan+Chang%2C+Alex+Shavick%2C+Hang+Dong%2C+Michael+TC+Poon%2C+Natalie+Fitzpatrick%2C+Adam+P+Levine%2C+et+al%2E+2022%2E+A+survey+on+clinical+natural+language+pro)
184. Minghao Wu and Alham Fikri Aji. 2023. [Style over substance: Evaluation biases for large language models](https://md.research.elamur.ai/free/en/papers/wu-2025-style-over-substance-evaluation-biases/). arXiv preprint arXiv:2307.03025.
185. Congying Xia, Chen Xing, Jiangshu Du, Xinyi Yang, Yihao Feng, Ran Xu, Wenpeng Yin, and Caiming Xiong. 2024. Fofo: A benchmark to evaluate llms’ format-following capability. arXiv preprint arXiv:2402.18667. [find](https://scholar.google.com/scholar?q=Congying+Xia%2C+Chen+Xing%2C+Jiangshu+Du%2C+Xinyi+Yang%2C+Yihao+Feng%2C+Ran+Xu%2C+Wenpeng+Yin%2C+and+Caiming+Xiong%2E+2024%2E+Fofo%3A+A+benchmark+to+evaluate+llms%E2%80%99+format%2Dfollowing+capability%2E+arXiv+preprint+arXiv%3A2402%2E1)
186. Guangzhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang. 2024. [Benchmarking retrieval-augmented generation for medicine](https://md.research.elamur.ai/free/en/papers/xiong-2024-benchmarking-retrieval-augmented-generation-medicine/). arXiv preprint arXiv:2402.13178.
187. Ruijie Xu, Zengzhi Wang, Run-Ze Fan, and Pengfei Liu. 2024. Benchmarking benchmark leakage in large language models. arXiv preprint arXiv:2404.18824. [find](https://scholar.google.com/scholar?q=Ruijie+Xu%2C+Zengzhi+Wang%2C+Run%2DZe+Fan%2C+and+Pengfei+Liu%2E+2024%2E+Benchmarking+benchmark+leakage+in+large+language+models%2E+arXiv+preprint+arXiv%3A2404%2E18824%2E)
188. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai D… [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chang+Zhou%2C+Chengpeng+Li%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Guanting+Dong%2C+Haoran+Wei%2C+Huan+Lin%2C+Jialong+Tang%2C+Jialin+Wang%2C+Jian+Yang%2C+Jian)
189. Linyi Yang, Shuibai Zhang, Libo Qin, Yafu Li, Yidong Wang, Hanmeng Liu, Jindong Wang, Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, Zekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, Yang Shen, et al. 2023a. A comprehensive capability analysis of gpt-3 and gpt-3.5 series models. arXiv preprint… [find](https://scholar.google.com/scholar?q=Linyi+Yang%2C+Shuibai+Zhang%2C+Libo+Qin%2C+Yafu+Li%2C+Yidong+Wang%2C+Hanmeng+Liu%2C+Jindong+Wang%2C+Junjie+Ye%2C+Xuanting+Chen%2C+Nuo+Xu%2C+Can+Zu%2C+Zekai+Shao%2C+Shichun+Liu%2C+Yuhan+Cui%2C+Zeyang+Zhou%2C+Chao+Gong%2C+Yang+Shen%2C+e)
190. Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. 2023b. [mplug-owl: Modularization empowers large language models with multimodality](https://md.research.elamur.ai/free/en/papers/ye-2023-mplug-owl-modularization-empowers-large/). arXiv preprint arXiv:2304.14178.
191. Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2023. [Mm-vet: Evaluating large multimodal models for integrated capabilities](https://md.research.elamur.ai/free/en/papers/yu-2023-mm-vet-evaluating-large-multimodal/). arXiv preprint arXiv:2308.02490.
192. Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. [Bartscore: Evaluating generated text as text generation](https://md.research.elamur.ai/free/en/papers/yuan-2021-bartscore-evaluating-generated-text-text/). Advances in Neural Information Processing Systems, 34:27263–27277.
193. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. 2023. [Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi](https://md.research.elamur.ai/free/en/papers/yue-2023-mmmu-massive-multi-discipline-multimodal/). arXiv preprint arXiv:2311.16502.
194. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+2019%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F+arXiv+preprint+arXiv%3A1905%2E07830%2E)
195. Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting Hu. 2023. [Alignscore: Evaluating factual consistency with a unified alignment function](https://md.research.elamur.ai/free/en/papers/zha-2023-alignscore-evaluating-factual-consistency-unified/). arXiv preprint arXiv:2305.16739.
196. Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Dylan Slack, Qin Lyu, et al. 2024a. A careful examination of large language model performance on grade school arithmetic. arXiv preprint arXiv:2405.00332. [find](https://scholar.google.com/scholar?q=Hugh+Zhang%2C+Jeff+Da%2C+Dean+Lee%2C+Vaughn+Robinson%2C+Catherine+Wu%2C+Will+Song%2C+Tiffany+Zhao%2C+Pranav+Raja%2C+Dylan+Slack%2C+Qin+Lyu%2C+et+al%2E+2024a%2E+A+careful+examination+of+large+language+model+performance+on+gra)
197. Pan Zhang, Xiaoyi Dong Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Shuangrui Ding, Songyang Zhang, Haodong Duan, Hang Yan, et al. 2023. [Internlm-xcomposer: A vision-language large model for advanced text-image comprehension and composition](https://md.research.elamur.ai/free/en/papers/zhang-2023-internlm-xcomposer-vision-language/). arXiv preprint arXiv:2309.15112.
198. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q+Weinberger%2C+and+Yoav+Artzi%2E+2019%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E+In+International+Conference+on+Learning+Representations%2E)
199. Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B Hashimoto. 2024b. [Benchmarking large language models for news summarization](https://md.research.elamur.ai/free/en/papers/zhang-2024-benchmarking-large-language-models-news/). Transactions of the Association for Computational Linguistics, 12:39–57.
200. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023a. [A survey of large language models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/). arXiv preprint arXiv:2303.18223.
201. Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. 2023b. [Slic-hf: Sequence likelihood calibration with human feedback](https://md.research.elamur.ai/free/en/papers/zhao-2023-slic-hf-sequence-likelihood-calibration/). arXiv preprint arXiv:2305.10425.
202. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). Advances in Neural Information Processing Systems, 36.
203. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023a. [Instruction-following evaluation for large language models](https://md.research.elamur.ai/free/en/papers/zhou-2023-instruction-following-evaluation-large-language/). arXiv preprint arXiv:2311.07911.
204. Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. 2023b. Don’t make your llm an evaluation benchmark cheater. arXiv preprint arXiv:2311.01964. [find](https://scholar.google.com/scholar?q=Kun+Zhou%2C+Yutao+Zhu%2C+Zhipeng+Chen%2C+Wentong+Chen%2C+Wayne+Xin+Zhao%2C+Xu+Chen%2C+Yankai+Lin%2C+Ji%2DRong+Wen%2C+and+Jiawei+Han%2E+2023b%2E+Don%E2%80%99t+make+your+llm+an+evaluation+benchmark+cheater%2E+arXiv+preprint+arXiv%3A2311)
205. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023a. [Minigpt-4: Enhancing vision-language understanding with advanced large language models](https://md.research.elamur.ai/free/en/papers/zhu-2023-minigpt-4-enhancing-vision-language/). arXiv preprint arXiv:2304.10592.
206. Kaijie Zhu, Jindong Wang, Qinlin Zhao, Ruochen Xu, and Xing Xie. 2024. Dyval 2: [Dynamic evaluation of large language models by meta probing agents](https://md.research.elamur.ai/free/en/papers/zhu-2024-dynamic-evaluation-large-language-models/).
207. Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Neil Zhenqiang Gong, Yue Zhang, et al. 2023b. [Promptbench: Towards evaluating the robustness of large language models on adversarial prompts](https://md.research.elamur.ai/free/en/papers/zhu-2023-promptrobust-towards-evaluating-robustness-large/). arXiv preprint arXiv:2306.04528.
208. Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Zhicheng Dou, and Ji-Rong Wen. 2023c. [Large language models for information retrieval: A survey](https://md.research.elamur.ai/free/en/papers/zhu-2023-large-language-models-information-retrieval/). arXiv preprint arXiv:2308.07107.
209. Ziyu Zhuang, Qiguang Chen, Longxuan Ma, Mingda Li, Yi Han, Yushan Qian, Haopeng Bai, Zixian Feng, Weinan Zhang, and Ting Liu. 2023. [Through the lens of core competency: Survey on evaluation of large language models](https://md.research.elamur.ai/free/en/papers/zhuang-2023-through-lens-core-competency-survey/). arXiv preprint arXiv:2308.07902.
210. Terry Yue Zhuo, Yujin Huang, Chunyang Chen, and Zhenchang Xing. 2023. [Red teaming ChatGPT via jailbreaking: Bias, robustness, reliability and toxicity](https://md.research.elamur.ai/free/en/papers/zhuo-2023-red-teaming-chatgpt-jailbreaking-bias/). arXiv preprint arXiv:2301.12867.
211. Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2024. [Can large language models transform computational social science?](https://md.research.elamur.ai/free/en/papers/ziems-2023-large-language-models-transform-computational/) Computational Linguistics, pages 1–55.

*Updated 31 October 2026*