---
title: "Benchmarking Large Language Models in — Chen 2023"
description: "Retrieval-augmented generation (RAG) is widely promoted as a way to reduce LLM hallucination and outdated knowledge."
authors: ["Jiawei Chen", "Hongyu Lin", "Xianpei Han", "Le Sun"]
year: 2023
venue: "arXiv:2309.01431"
source: "https://arxiv.org/abs/2309.01431"
doi: "10.48550/arxiv.2309.01431"
arxiv: "2309.01431"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chen-2023-benchmarking-large-language-models-retrieval/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chen-2023-benchmarking-large-language-models-retrieval/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/) / Benchmarking Large Language Models in Retrieval-Augmented Generation

# Benchmarking Large Language Models in Retrieval-Augmented Generation

**Authors:** Jiawei Chen, [Hongyu Lin](https://md.research.elamur.ai/free/en/authors/hongyu-lin/), [Xianpei Han](https://md.research.elamur.ai/free/en/authors/xianpei-han/), [Le Sun](https://md.research.elamur.ai/free/en/authors/le-sun/) · **Year:** 2023 · **Venue:** arXiv:2309.01431 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2309.01431) · [PDF](https://arxiv.org/pdf/2309.01431) · [DOI 10.48550/arxiv.2309.01431](https://doi.org/10.48550/arxiv.2309.01431) · [arXiv 2309.01431](https://arxiv.org/abs/2309.01431) · [OpenAlex](https://openalex.org/W4393147129)

**Citations:** 433 citations (openalex-snapshot · 2026-10-14) · [191](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Retrieval-augmented generation (RAG) is widely promoted as a way to reduce LLM hallucination and outdated knowledge. However, there has been little systematic evidence about where it fails for different models. The authors build RGB (Retrieval-Augmented Generation Benchmark), an English/Chinese corpus divided into four testbeds. Each testbed probes one skill:
- coping with related but unhelpful documents;
- declining to answer when no document holds the answer;
- combining facts spread across several documents;
- detecting false statements in retrieved text.

To keep the models' memorized knowledge from skewing results, questions are built from recent news. ChatGPT drafts question–answer pairs, a search API gathers web pages, a dense retriever reranks passages, and humans filter the output. Six LLMs are tested, including ChatGPT, ChatGLM2-6B and Vicuna-7b. The models tolerate some noise but often mix up similar facts. They seldom refuse when evidence is missing, handle multi-document questions poorly, and usually believe retrieved misinformation over their own knowledge, even when warned. RGB offers a diagnostic tool and shows that RAG pipelines need careful design.

## Key points
- **Four target abilities:** withstanding noisy documents, refusing to answer when evidence is absent, merging information from multiple sources, and resisting factually wrong retrieved content. Each has its own testbed in RGB. Code and data are at github.com/chen700564/RGB.
- **Data pipeline:** news articles → event/question/answer extraction with gpt-3.5-turbo → Google Search API → dense-retrieval reranking of chunks (top-30) → human adjustment and filtering.
- **Models evaluated:** ChatGPT, ChatGLM-6B, ChatGLM2-6B, Vicuna-7b, Qwen-7B-Chat and BELLE-7B.
- **Integration errors (ChatGLM2-6B):** merging sub-answers (28% of errors), ignoring one sub-question (28%) and misaligning documents with sub-questions (6%). The authors suggest chain-of-thought decomposition but note it slows inference.
- **Counterfactual robustness:** only models with over 70% closed-book accuracy were tested. Their accuracy of 77–91% without documents fell to 9–17% when given counterfactual documents. Error correction rates ranged from 25% to about 57%.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

191 papers of our library cite it (of the 140673 whose bibliographies we read); 12 of them have a page here, 179 not yet.

- [Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models](https://md.research.elamur.ai/free/en/papers/dahl-2024-large-legal-fictions-profiling-legal/)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/gao-2023-retrieval-augmented-generation-large-language/)
- [Survey of Hallucination in Natural Language Generation](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/)
- [A Survey on Large Language Models for Code Generation](https://md.research.elamur.ai/free/en/papers/jiang-2024-survey-large-language-models-code/)
- [Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools](https://md.research.elamur.ai/free/en/papers/magesh-2024-hallucination-free-assessing-reliability-leading/)
- [Uncovering Ideological Bias in RAG with Lexical Multidimensional Analysis: A Case Study on COVID-19](https://md.research.elamur.ai/free/en/papers/salari-2026-uncovering-ideological-bias-rag-lexical/)
- [MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents](https://md.research.elamur.ai/free/en/papers/tang-2024-minicheck-efficient-fact-checking-llms/)
- [MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries](https://md.research.elamur.ai/free/en/papers/tang-2024-multihop-rag-benchmarking-retrieval-augmented/)
- [Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity](https://md.research.elamur.ai/free/en/papers/wang-2023-survey-factuality-large-language-models/)
- [华谱通: 基于知识推理的家谱问答大语言模型](https://md.research.elamur.ai/free/en/papers/wu-2025-huaputong-large-language/)
- [Hallucination is Inevitable: An Innate Limitation of Large Language Models](https://md.research.elamur.ai/free/en/papers/xu-2024-hallucination-inevitable-innate-limitation-large/)
- [Knowledge Conflicts for LLMs: A Survey](https://md.research.elamur.ai/free/en/papers/xu-2024-knowledge-conflicts-llms-survey/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{chen2023benchmarking,
  title = {{Benchmarking Large Language Models in Retrieval-Augmented Generation}},
  author = {Chen, Jiawei and Lin, Hongyu and Han, Xianpei and Sun, Le},
  year = {2023},
  doi = {10.48550/arxiv.2309.01431},
  eprint = {2309.01431},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2309.01431}
}
```

- [← previous BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset](https://md.research.elamur.ai/free/en/papers/ji-2023-beavertails-towards-improved-safety-alignment/)
- [next → Benchmarking Machine Translation with Cultural Awareness](https://md.research.elamur.ai/free/en/papers/yao-2023-benchmarking-machine-translation-cultural-awareness/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/)

## References

47 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Adlakha, V.; BehnamGhader, P.; Lu, X. H.; Meade, N.; and Reddy, S. 2023. Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering. arXiv:2307.16877. [find](https://scholar.google.com/scholar?q=Adlakha%2C+V%2E%3B+BehnamGhader%2C+P%2E%3B+Lu%2C+X%2E+H%2E%3B+Meade%2C+N%2E%3B+and+Reddy%2C+S%2E+2023%2E+Evaluating+Correctness+and+Faithfulness+of+Instruction%2DFollowing+Models+for+Question+Answering%2E+arXiv%3A2307%2E16877%2E)
2. Bang, Y.; Cahyawijaya, S.; Lee, N.; Dai, W.; Su, D.; Wilie, B.; Lovenia, H.; Ji, Z.; Yu, T.; Chung, W.; Do, Q. V.; Xu, Y.; and Fung, P. 2023. A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity. arXiv:2302.04023. [find](https://scholar.google.com/scholar?q=Bang%2C+Y%2E%3B+Cahyawijaya%2C+S%2E%3B+Lee%2C+N%2E%3B+Dai%2C+W%2E%3B+Su%2C+D%2E%3B+Wilie%2C+B%2E%3B+Lovenia%2C+H%2E%3B+Ji%2C+Z%2E%3B+Yu%2C+T%2E%3B+Chung%2C+W%2E%3B+Do%2C+Q%2E+V%2E%3B+Xu%2C+Y%2E%3B+and+Fung%2C+P%2E+2023%2E+A+Multitask%2C+Multilingual%2C+Multimodal+Evaluation+of+ChatGP) [our card](https://md.research.elamur.ai/free/en/papers/bang-2023-multitask-multilingual-multimodal-evaluation-chatgpt/)
3. Bian, N.; Liu, P.; Han, X.; Lin, H.; Lu, Y.; He, B.; and Sun, L. 2023. A Drop of Ink Makes a Million Think: The Spread of False Information in Large Language Models. arXiv:2305.04812. — [arXiv:2305.04812](https://arxiv.org/abs/2305.04812) [find](https://scholar.google.com/scholar?q=Bian%2C+N%2E%3B+Liu%2C+P%2E%3B+Han%2C+X%2E%3B+Lin%2C+H%2E%3B+Lu%2C+Y%2E%3B+He%2C+B%2E%3B+and+Sun%2C+L%2E+2023%2E+A+Drop+of+Ink+Makes+a+Million+Think%3A+The+Spread+of+False+Information+in+Large+Language+Models%2E+arXiv%3A2305%2E04812%2E+%E2%80%94+%5BarXiv%3A2305%2E04)
4. Borgeaud, S.; Mensch, A.; Hoffmann, J.; Cai, T.; Rutherford, E.; Millican, K.; van den Driessche, G.; Lespiau, J.-B.; Damoc, B.; Clark, A.; de Las Casas, D.; Guy, A.; Menick, J.; Ring, R.; Hennigan, T.; Huang, S.; Maggiore, L.; Jones, C.; Cassirer, A.; Brock, A.; Paganini, M.; Irving, G.; Vinyals, O… [find](https://scholar.google.com/scholar?q=Borgeaud%2C+S%2E%3B+Mensch%2C+A%2E%3B+Hoffmann%2C+J%2E%3B+Cai%2C+T%2E%3B+Rutherford%2C+E%2E%3B+Millican%2C+K%2E%3B+van+den+Driessche%2C+G%2E%3B+Lespiau%2C+J%2E%2DB%2E%3B+Damoc%2C+B%2E%3B+Clark%2C+A%2E%3B+de+Las+Casas%2C+D%2E%3B+Guy%2C+A%2E%3B+Menick%2C+J%2E%3B+Ring%2C+R%2E%3B+Hennigan%2C+T)
5. Cai, D.; Wang, Y.; Bi, W.; Tu, Z.; Liu, X.; Lam, W.; and Shi, S. 2019a. Skeleton-to-Response: Dialogue Generation Guided by Retrieval Memory. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (L… [find](https://scholar.google.com/scholar?q=Cai%2C+D%2E%3B+Wang%2C+Y%2E%3B+Bi%2C+W%2E%3B+Tu%2C+Z%2E%3B+Liu%2C+X%2E%3B+Lam%2C+W%2E%3B+and+Shi%2C+S%2E+2019a%2E+Skeleton%2Dto%2DResponse%3A+Dialogue+Generation+Guided+by+Retrieval+Memory%2E+In+Proceedings+of+the+2019+Conference+of+the+North+America)
6. Cai, D.; Wang, Y.; Bi, W.; Tu, Z.; Liu, X.; and Shi, S. 2019b. Retrieval-guided Dialogue Response Generation via a Matching-to-Generation Framework. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Languag… [find](https://scholar.google.com/scholar?q=Cai%2C+D%2E%3B+Wang%2C+Y%2E%3B+Bi%2C+W%2E%3B+Tu%2C+Z%2E%3B+Liu%2C+X%2E%3B+and+Shi%2C+S%2E+2019b%2E+Retrieval%2Dguided+Dialogue+Response+Generation+via+a+Matching%2Dto%2DGeneration+Framework%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+)
7. Cao, M.; Dong, Y.; Wu, J.; and Cheung, J. C. K. 2020. Factual Error Correction for Abstractive Summarization Models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 6251– 6258. Online: Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Cao%2C+M%2E%3B+Dong%2C+Y%2E%3B+Wu%2C+J%2E%3B+and+Cheung%2C+J%2E+C%2E+K%2E+2020%2E+Factual+Error+Correction+for+Abstractive+Summarization+Models%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+Proc) [our card](https://md.research.elamur.ai/free/en/papers/cao-2020-factual-error-correction-abstractive-summarization/)
8. Chang, Y.; Wang, X.; Wang, J.; Wu, Y.; Yang, L.; Zhu, K.; Chen, H.; Yi, X.; Wang, C.; Wang, Y.; Ye, W.; Zhang, Y.; Chang, Y.; Yu, P. S.; Yang, Q.; and Xie, X. 2023. A Survey on Evaluation of Large Language Models. arXiv:2307.03109. [find](https://scholar.google.com/scholar?q=Chang%2C+Y%2E%3B+Wang%2C+X%2E%3B+Wang%2C+J%2E%3B+Wu%2C+Y%2E%3B+Yang%2C+L%2E%3B+Zhu%2C+K%2E%3B+Chen%2C+H%2E%3B+Yi%2C+X%2E%3B+Wang%2C+C%2E%3B+Wang%2C+Y%2E%3B+Ye%2C+W%2E%3B+Zhang%2C+Y%2E%3B+Chang%2C+Y%2E%3B+Yu%2C+P%2E+S%2E%3B+Yang%2C+Q%2E%3B+and+Xie%2C+X%2E+2023%2E+A+Survey+on+Evaluation+of+Large+Lan) [our card](https://md.research.elamur.ai/free/en/papers/chang-2023-survey-evaluation-large-language-models/)
9. Chiang, W.-L.; Li, Z.; Lin, Z.; Sheng, Y.; Wu, Z.; Zhang, H.; Zheng, L.; Zhuang, S.; Zhuang, Y.; Gonzalez, J. E.; Stoica, I.; and Xing, E. P. 2023. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality. [find](https://scholar.google.com/scholar?q=Chiang%2C+W%2E%2DL%2E%3B+Li%2C+Z%2E%3B+Lin%2C+Z%2E%3B+Sheng%2C+Y%2E%3B+Wu%2C+Z%2E%3B+Zhang%2C+H%2E%3B+Zheng%2C+L%2E%3B+Zhuang%2C+S%2E%3B+Zhuang%2C+Y%2E%3B+Gonzalez%2C+J%2E+E%2E%3B+Stoica%2C+I%2E%3B+and+Xing%2C+E%2E+P%2E+2023%2E+Vicuna%3A+An+Open%2DSource+Chatbot+Impressing+GPT%2D4+with)
10. Cui, J.; Li, Z.; Yan, Y.; Chen, B.; and Yuan, L. 2023. Chat-Law: Open-Source Legal Large Language Model with Integrated External Knowledge Bases. arXiv:2306.16092. [find](https://scholar.google.com/scholar?q=Cui%2C+J%2E%3B+Li%2C+Z%2E%3B+Yan%2C+Y%2E%3B+Chen%2C+B%2E%3B+and+Yuan%2C+L%2E+2023%2E+Chat%2DLaw%3A+Open%2DSource+Legal+Large+Language+Model+with+Integrated+External+Knowledge+Bases%2E+arXiv%3A2306%2E16092%2E) [our card](https://md.research.elamur.ai/free/en/papers/cui-2023-chatlaw-multi-agent-legal-assistant/)
11. Drozdov, A.; Schärli, N.; Akyürek, E.; Scales, N.; Song, X.; Chen, X.; Bousquet, O.; and Zhou, D. 2023. Compositional Semantic Parsing with Large Language Models. In The Eleventh International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Drozdov%2C+A%2E%3B+Scha%CC%88rli%2C+N%2E%3B+Akyu%CC%88rek%2C+E%2E%3B+Scales%2C+N%2E%3B+Song%2C+X%2E%3B+Chen%2C+X%2E%3B+Bousquet%2C+O%2E%3B+and+Zhou%2C+D%2E+2023%2E+Compositional+Semantic+Parsing+with+Large+Language+Models%2E+In+The+Eleventh+International+Confe)
12. Edward Beeching, N. H. S. H. N. L. N. R. O. S. L. T. T. W., Clémentine Fourrier. 2023. Open LLM Leaderboard. [https://huggingface.co/spaces/HuggingFaceH4/open](https://huggingface.co/spaces/HuggingFaceH4/open) llm leaderboard. [↗](https://huggingface.co/spaces/HuggingFaceH4/open)
13. Guo, B.; Zhang, X.; Wang, Z.; Jiang, M.; Nie, J.; Ding, Y.; Yue, J.; and Wu, Y. 2023. How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection. arXiv:2301.07597. [find](https://scholar.google.com/scholar?q=Guo%2C+B%2E%3B+Zhang%2C+X%2E%3B+Wang%2C+Z%2E%3B+Jiang%2C+M%2E%3B+Nie%2C+J%2E%3B+Ding%2C+Y%2E%3B+Yue%2C+J%2E%3B+and+Wu%2C+Y%2E+2023%2E+How+Close+is+ChatGPT+to+Human+Experts%3F+Comparison+Corpus%2C+Evaluation%2C+and+Detection%2E+arXiv%3A2301%2E07597%2E) [our card](https://md.research.elamur.ai/free/en/papers/guo-2023-close-chatgpt-human-experts-comparison/)
14. Guu, K.; Lee, K.; Tung, Z.; Pasupat, P.; and Chang, M.-W. 2020. REALM: Retrieval-Augmented Language Model Pre-Training. In Proceedings of the 37th International Conference on Machine Learning, ICML’20. JMLR.org. [find](https://scholar.google.com/scholar?q=Guu%2C+K%2E%3B+Lee%2C+K%2E%3B+Tung%2C+Z%2E%3B+Pasupat%2C+P%2E%3B+and+Chang%2C+M%2E%2DW%2E+2020%2E+REALM%3A+Retrieval%2DAugmented+Language+Model+Pre%2DTraining%2E+In+Proceedings+of+the+37th+International+Conference+on+Machine+Learning%2C+ICML%E2%80%9920)
15. He, H.; Zhang, H.; and Roth, D. 2022. Rethinking with Retrieval: Faithful Large Language Model Inference. arXiv:2301.00303. [find](https://scholar.google.com/scholar?q=He%2C+H%2E%3B+Zhang%2C+H%2E%3B+and+Roth%2C+D%2E+2022%2E+Rethinking+with+Retrieval%3A+Faithful+Large+Language+Model+Inference%2E+arXiv%3A2301%2E00303%2E)
16. Hendrycks, D.; Burns, C.; Basart, S.; Zou, A.; Mazeika, M.; Song, D.; and Steinhardt, J. 2021. Measuring Massive Multitask Language Understanding. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Hendrycks%2C+D%2E%3B+Burns%2C+C%2E%3B+Basart%2C+S%2E%3B+Zou%2C+A%2E%3B+Mazeika%2C+M%2E%3B+Song%2C+D%2E%3B+and+Steinhardt%2C+J%2E+2021%2E+Measuring+Massive+Multitask+Language+Understanding%2E+In+International+Conference+on+Learning+Representatio) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/)
17. Huang, Y.; Bai, Y.; Zhu, Z.; Zhang, J.; Zhang, J.; Su, T.; Liu, J.; Lv, C.; Zhang, Y.; Lei, J.; Fu, Y.; Sun, M.; and He, J. 2023. C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models. arXiv preprint arXiv:2305.08322. [find](https://scholar.google.com/scholar?q=Huang%2C+Y%2E%3B+Bai%2C+Y%2E%3B+Zhu%2C+Z%2E%3B+Zhang%2C+J%2E%3B+Zhang%2C+J%2E%3B+Su%2C+T%2E%3B+Liu%2C+J%2E%3B+Lv%2C+C%2E%3B+Zhang%2C+Y%2E%3B+Lei%2C+J%2E%3B+Fu%2C+Y%2E%3B+Sun%2C+M%2E%3B+and+He%2C+J%2E+2023%2E+C%2DEval%3A+A+Multi%2DLevel+Multi%2DDiscipline+Chinese+Evaluation+Suite+for+Fo) [our card](https://md.research.elamur.ai/free/en/papers/huang-2023-c-eval-multi-level-multi/)
18. Izacard, G.; and Grave, E. 2021. Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, 874–880. Online: Association for Computational Linguis… [find](https://scholar.google.com/scholar?q=Izacard%2C+G%2E%3B+and+Grave%2C+E%2E+2021%2E+Leveraging+Passage+Retrieval+with+Generative+Models+for+Open+Domain+Question+Answering%2E+In+Proceedings+of+the+16th+Conference+of+the+European+Chapter+of+the+Associatio) [our card](https://md.research.elamur.ai/free/en/papers/izacard-2021-leveraging-passage-retrieval-generative-models/)
19. Izacard, G.; Lewis, P.; Lomeli, M.; Hosseini, L.; Petroni, F.; Schick, T.; Dwivedi-Yu, J.; Joulin, A.; Riedel, S.; and Grave, E. 2022. Atlas: Few-shot Learning with Retrieval Augmented Language Models. arXiv:2208.03299. [find](https://scholar.google.com/scholar?q=Izacard%2C+G%2E%3B+Lewis%2C+P%2E%3B+Lomeli%2C+M%2E%3B+Hosseini%2C+L%2E%3B+Petroni%2C+F%2E%3B+Schick%2C+T%2E%3B+Dwivedi%2DYu%2C+J%2E%3B+Joulin%2C+A%2E%3B+Riedel%2C+S%2E%3B+and+Grave%2C+E%2E+2022%2E+Atlas%3A+Few%2Dshot+Learning+with+Retrieval+Augmented+Language+Models)
20. Ji, Z.; Lee, N.; Frieske, R.; Yu, T.; Su, D.; Xu, Y.; Ishii, E.; Bang, Y. J.; Madotto, A.; and Fung, P. 2023. Survey of Hallucination in Natural Language Generation. ACM Comput. [find](https://scholar.google.com/scholar?q=Ji%2C+Z%2E%3B+Lee%2C+N%2E%3B+Frieske%2C+R%2E%3B+Yu%2C+T%2E%3B+Su%2C+D%2E%3B+Xu%2C+Y%2E%3B+Ishii%2C+E%2E%3B+Bang%2C+Y%2E+J%2E%3B+Madotto%2C+A%2E%3B+and+Fung%2C+P%2E+2023%2E+Survey+of+Hallucination+in+Natural+Language+Generation%2E+ACM+Comput%2E) [our card](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/)
21. Surv., 55(12). Lewis, P.; Perez, E.; Piktus, A.; Petroni, F.; Karpukhin, V.; Goyal, N.; Küttler, H.; Lewis, M.; Yih, W.-t.; Rocktäschel, T.; Riedel, S.; and Kiela, D. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Proceedings of the 34th International Conference on Neur… [find](https://scholar.google.com/scholar?q=Surv%2E%2C+55%2812%29%2E+Lewis%2C+P%2E%3B+Perez%2C+E%2E%3B+Piktus%2C+A%2E%3B+Petroni%2C+F%2E%3B+Karpukhin%2C+V%2E%3B+Goyal%2C+N%2E%3B+Ku%CC%88ttler%2C+H%2E%3B+Lewis%2C+M%2E%3B+Yih%2C+W%2E%2Dt%2E%3B+Rockta%CC%88schel%2C+T%2E%3B+Riedel%2C+S%2E%3B+and+Kiela%2C+D%2E+2020%2E+Retrieval%2DAugmented+Gener)
22. Li, D.; Rawat, A. S.; Zaheer, M.; Wang, X.; Lukasik, M.; Veit, A.; Yu, F.; and Kumar, S. 2023a. Large Language Models with Controllable Working Memory. In Findings of the Association for Computational Linguistics: ACL 2023, 1774–1793. Toronto, Canada: Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Li%2C+D%2E%3B+Rawat%2C+A%2E+S%2E%3B+Zaheer%2C+M%2E%3B+Wang%2C+X%2E%3B+Lukasik%2C+M%2E%3B+Veit%2C+A%2E%3B+Yu%2C+F%2E%3B+and+Kumar%2C+S%2E+2023a%2E+Large+Language+Models+with+Controllable+Working+Memory%2E+In+Findings+of+the+Association+for+Computational) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-large-language-models-controllable-working/)
23. Li, X.; Zhang, T.; Dubois, Y.; Taori, R.; Gulrajani, I.; Guestrin, C.; Liang, P.; and Hashimoto, T. B. 2023b. AlpacaEval: An Automatic Evaluator of Instruction-following Models. [https://github.com/tatsu-lab/alpaca](https://github.com/tatsu-lab/alpaca) eval. [↗](https://github.com/tatsu-lab/alpaca)
24. Li, X.; Zhu, X.; Ma, Z.; Liu, X.; and Shah, S. 2023c. Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? An Examination on Several Typical Tasks. arXiv:2305.05862. — [arXiv:2305.05862](https://arxiv.org/abs/2305.05862) [find](https://scholar.google.com/scholar?q=Li%2C+X%2E%3B+Zhu%2C+X%2E%3B+Ma%2C+Z%2E%3B+Liu%2C+X%2E%3B+and+Shah%2C+S%2E+2023c%2E+Are+ChatGPT+and+GPT%2D4+General%2DPurpose+Solvers+for+Financial+Text+Analytics%3F+An+Examination+on+Several+Typical+Tasks%2E+arXiv%3A2305%2E05862%2E+%E2%80%94+%5BarXiv%3A23)
25. Liu, N. F.; Zhang, T.; and Liang, P. 2023. Evaluating Verifiability in Generative Search Engines. arXiv:2304.09848. [find](https://scholar.google.com/scholar?q=Liu%2C+N%2E+F%2E%3B+Zhang%2C+T%2E%3B+and+Liang%2C+P%2E+2023%2E+Evaluating+Verifiability+in+Generative+Search+Engines%2E+arXiv%3A2304%2E09848%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-evaluating-verifiability-generative-search-engines/)
26. Maynez, J.; Narayan, S.; Bohnet, B.; and McDonald, R. 2020. On Faithfulness and Factuality in Abstractive Summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 1906–1919. [find](https://scholar.google.com/scholar?q=Maynez%2C+J%2E%3B+Narayan%2C+S%2E%3B+Bohnet%2C+B%2E%3B+and+McDonald%2C+R%2E+2020%2E+On+Faithfulness+and+Factuality+in+Abstractive+Summarization%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+)
27. OpenAI. 2022. Chatgpt: Optimizing language models for dialogue. [https://openai.com/blog/chatgpt](https://openai.com/blog/chatgpt). [↗](https://openai.com/blog/chatgpt)
28. Peng, B.; Galley, M.; He, P.; Cheng, H.; Xie, Y.; Hu, Y.; Huang, Q.; Liden, L.; Yu, Z.; Chen, W.; and Gao, J. 2023. [find](https://scholar.google.com/scholar?q=Peng%2C+B%2E%3B+Galley%2C+M%2E%3B+He%2C+P%2E%3B+Cheng%2C+H%2E%3B+Xie%2C+Y%2E%3B+Hu%2C+Y%2E%3B+Huang%2C+Q%2E%3B+Liden%2C+L%2E%3B+Yu%2C+Z%2E%3B+Chen%2C+W%2E%3B+and+Gao%2C+J%2E+2023%2E)
29. Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback. arXiv:2302.12813. Qin, Y.; Liang, S.; Ye, Y.; Zhu, K.; Yan, L.; Lu, Y.; Lin, Y.; Cong, X.; Tang, X.; Qian, B.; Zhao, S.; Tian, R.; Xie, R.; Zhou, J.; Gerstein, M.; Li, D.; Liu, Z.; and Sun… [find](https://scholar.google.com/scholar?q=Check+Your+Facts+and+Try+Again%3A+Improving+Large+Language+Models+with+External+Knowledge+and+Automated+Feedback%2E+arXiv%3A2302%2E12813%2E+Qin%2C+Y%2E%3B+Liang%2C+S%2E%3B+Ye%2C+Y%2E%3B+Zhu%2C+K%2E%3B+Yan%2C+L%2E%3B+Lu%2C+Y%2E%3B+Lin%2C+Y%2E%3B+Cong%2C+X) [our card](https://md.research.elamur.ai/free/en/papers/peng-2023-check-facts-try-again-improving/)
30. ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs. arXiv:2307.16789. [find](https://scholar.google.com/scholar?q=ToolLLM%3A+Facilitating+Large+Language+Models+to+Master+16000%2B+Real%2Dworld+APIs%2E+arXiv%3A2307%2E16789%2E) [our card](https://md.research.elamur.ai/free/en/papers/qin-2023-toolllm-facilitating-large-language-models/)
31. QwenLM. 2023. Qwen-7B. [https://github.com/QwenLM/](https://github.com/QwenLM/) [↗](https://github.com/QwenLM/)
32. Raunak, V.; Menezes, A.; and Junczys-Dowmunt, M. 2021. [find](https://scholar.google.com/scholar?q=Raunak%2C+V%2E%3B+Menezes%2C+A%2E%3B+and+Junczys%2DDowmunt%2C+M%2E+2021%2E)
33. The Curious Case of Hallucinations in Neural Machine Translation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1172– 1183. Online: Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=The+Curious+Case+of+Hallucinations+in+Neural+Machine+Translation%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics%3A+Human+Language+T) [our card](https://md.research.elamur.ai/free/en/papers/raunak-2021-curious-case-hallucinations-neural-machine/)
34. Ren, R.; Wang, Y.; Qu, Y.; Zhao, W. X.; Liu, J.; Tian, H.; Wu, H.; Wen, J.-R.; and Wang, H. 2023. Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation. arXiv:2307.11019. [find](https://scholar.google.com/scholar?q=Ren%2C+R%2E%3B+Wang%2C+Y%2E%3B+Qu%2C+Y%2E%3B+Zhao%2C+W%2E+X%2E%3B+Liu%2C+J%2E%3B+Tian%2C+H%2E%3B+Wu%2C+H%2E%3B+Wen%2C+J%2E%2DR%2E%3B+and+Wang%2C+H%2E+2023%2E+Investigating+the+Factual+Knowledge+Boundary+of+Large+Language+Models+with+Retrieval+Augmentation%2E+arX)
35. Shen, X.; Chen, Z.; Backes, M.; and Zhang, Y. 2023. In ChatGPT We Trust? Measuring and Characterizing the Reliability of ChatGPT. arXiv:2304.08979. — [arXiv:2304.08979](https://arxiv.org/abs/2304.08979) [find](https://scholar.google.com/scholar?q=Shen%2C+X%2E%3B+Chen%2C+Z%2E%3B+Backes%2C+M%2E%3B+and+Zhang%2C+Y%2E+2023%2E+In+ChatGPT+We+Trust%3F+Measuring+and+Characterizing+the+Reliability+of+ChatGPT%2E+arXiv%3A2304%2E08979%2E+%E2%80%94+%5BarXiv%3A2304%2E08979%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2304%2E0897)
36. Shi, W.; Min, S.; Yasunaga, M.; Seo, M.; James, R.; Lewis, M.; Zettlemoyer, L.; and tau Yih, W. 2023. RE-PLUG: Retrieval-Augmented Black-Box Language Models. arXiv:2301.12652. [find](https://scholar.google.com/scholar?q=Shi%2C+W%2E%3B+Min%2C+S%2E%3B+Yasunaga%2C+M%2E%3B+Seo%2C+M%2E%3B+James%2C+R%2E%3B+Lewis%2C+M%2E%3B+Zettlemoyer%2C+L%2E%3B+and+tau+Yih%2C+W%2E+2023%2E+RE%2DPLUG%3A+Retrieval%2DAugmented+Black%2DBox+Language+Models%2E+arXiv%3A2301%2E12652%2E) [our card](https://md.research.elamur.ai/free/en/papers/shi-2023-replug-retrieval-augmented-black-box/)
37. THUDM. 2023a. ChatGLM-6B. [https://github.com/](https://github.com/) [↗](https://github.com/)
38. THUDM. 2023b. ChatGLM2-6B. [https://github.com/](https://github.com/) [↗](https://github.com/)
39. Trivedi, H.; Balasubramanian, N.; Khot, T.; and Sabharwal, A. 2023. Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 10014–10037. To… [find](https://scholar.google.com/scholar?q=Trivedi%2C+H%2E%3B+Balasubramanian%2C+N%2E%3B+Khot%2C+T%2E%3B+and+Sabharwal%2C+A%2E+2023%2E+Interleaving+Retrieval+with+Chain%2Dof%2DThought+Reasoning+for+Knowledge%2DIntensive+Multi%2DStep+Questions%2E+In+Proceedings+of+the+61st+Annu) [our card](https://md.research.elamur.ai/free/en/papers/trivedi-2022-interleaving-retrieval-chain-thought-reasoning/)
40. Wang, A.; Pruksachatkun, Y.; Nangia, N.; Singh, A.; Michael, J.; Hill, F.; Levy, O.; and Bowman, S. R. 2019a. SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems. [find](https://scholar.google.com/scholar?q=Wang%2C+A%2E%3B+Pruksachatkun%2C+Y%2E%3B+Nangia%2C+N%2E%3B+Singh%2C+A%2E%3B+Michael%2C+J%2E%3B+Hill%2C+F%2E%3B+Levy%2C+O%2E%3B+and+Bowman%2C+S%2E+R%2E+2019a%2E+SuperGLUE%3A+A+Stickier+Benchmark+for+General%2DPurpose+Language+Understanding+Systems%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/)
41. Red Hook, NY, USA: Curran Associates Inc. Wang, A.; Singh, A.; Michael, J.; Hill, F.; Levy, O.; and Bowman, S. R. 2019b. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Red+Hook%2C+NY%2C+USA%3A+Curran+Associates+Inc%2E+Wang%2C+A%2E%3B+Singh%2C+A%2E%3B+Michael%2C+J%2E%3B+Hill%2C+F%2E%3B+Levy%2C+O%2E%3B+and+Bowman%2C+S%2E+R%2E+2019b%2E+GLUE%3A+A+Multi%2DTask+Benchmark+and+Analysis+Platform+for+Natural+Language+Underst) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)
42. Xu, G.; Liu, J.; Yan, M.; Xu, H.; Si, J.; Zhou, Z.; Yi, P.; Gao, X.; Sang, J.; Zhang, R.; Zhang, J.; Peng, C.; Huang, F.; and Zhou, J. 2023a. CValues: Measuring the Values of Chinese Large Language Models from Safety to Responsibility. arXiv:2307.09705. [find](https://scholar.google.com/scholar?q=Xu%2C+G%2E%3B+Liu%2C+J%2E%3B+Yan%2C+M%2E%3B+Xu%2C+H%2E%3B+Si%2C+J%2E%3B+Zhou%2C+Z%2E%3B+Yi%2C+P%2E%3B+Gao%2C+X%2E%3B+Sang%2C+J%2E%3B+Zhang%2C+R%2E%3B+Zhang%2C+J%2E%3B+Peng%2C+C%2E%3B+Huang%2C+F%2E%3B+and+Zhou%2C+J%2E+2023a%2E+CValues%3A+Measuring+the+Values+of+Chinese+Large+Language+Mo)
43. Xu, S.; Pang, L.; Shen, H.; Cheng, X.; and Chua, T.-S. 2023b. Search-in-the-Chain: Towards Accurate, Credible and Traceable Large Language Models for Knowledge-intensive Tasks. arXiv:2304.14732. — [arXiv:2304.14732](https://arxiv.org/abs/2304.14732) [find](https://scholar.google.com/scholar?q=Xu%2C+S%2E%3B+Pang%2C+L%2E%3B+Shen%2C+H%2E%3B+Cheng%2C+X%2E%3B+and+Chua%2C+T%2E%2DS%2E+2023b%2E+Search%2Din%2Dthe%2DChain%3A+Towards+Accurate%2C+Credible+and+Traceable+Large+Language+Models+for+Knowledge%2Dintensive+Tasks%2E+arXiv%3A2304%2E14732%2E+%E2%80%94+%5Bar)
44. Yunjie Ji, Y. G. Y. P. Q. N. B. M. X. L., Yong Deng. 2023. BELLE: Bloom-Enhanced Large Language model Engine. [https://github.com/LianjiaTech/BELLE](https://github.com/LianjiaTech/BELLE). [↗](https://github.com/LianjiaTech/BELLE)
45. Zhang, W.; Aljunied, S. M.; Gao, C.; Chia, Y. K.; and Bing, L. 2023. M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Models. [find](https://scholar.google.com/scholar?q=Zhang%2C+W%2E%3B+Aljunied%2C+S%2E+M%2E%3B+Gao%2C+C%2E%3B+Chia%2C+Y%2E+K%2E%3B+and+Bing%2C+L%2E+2023%2E+M3Exam%3A+A+Multilingual%2C+Multimodal%2C+Multilevel+Benchmark+for+Examining+Large+Language+Models%2E)
46. Zhong, W.; Cui, R.; Guo, Y.; Liang, Y.; Lu, S.; Wang, Y.; Saied, A.; Chen, W.; and Duan, N. 2023. AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models. arXiv:2304.06364. Zhou, D.; Schärli, N.; Hou, L.; Wei, J.; Scales, N.; Wang, X.; Schuurmans, D.; Cui, C.; Bousquet, O.; Le, Q. V.; a… [find](https://scholar.google.com/scholar?q=Zhong%2C+W%2E%3B+Cui%2C+R%2E%3B+Guo%2C+Y%2E%3B+Liang%2C+Y%2E%3B+Lu%2C+S%2E%3B+Wang%2C+Y%2E%3B+Saied%2C+A%2E%3B+Chen%2C+W%2E%3B+and+Duan%2C+N%2E+2023%2E+AGIEval%3A+A+Human%2DCentric+Benchmark+for+Evaluating+Foundation+Models%2E+arXiv%3A2304%2E06364%2E+Zhou%2C+D%2E%3B+Scha%CC%88) [our card](https://md.research.elamur.ai/free/en/papers/zhong-2024-agieval-human-centric-benchmark-evaluating/)
47. Zhou, S.; Alon, U.; Xu, F. F.; Jiang, Z.; and Neubig, G. 2023b. DocPrompting: Generating Code by Retrieving the Docs. In The Eleventh International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Zhou%2C+S%2E%3B+Alon%2C+U%2E%3B+Xu%2C+F%2E+F%2E%3B+Jiang%2C+Z%2E%3B+and+Neubig%2C+G%2E+2023b%2E+DocPrompting%3A+Generating+Code+by+Retrieving+the+Docs%2E+In+The+Eleventh+International+Conference+on+Learning+Representations%2E)