---
title: "Context-faithful Prompting for Large Language — Zhou 2023"
description: "Large language models often favor facts memorized during pretraining over the passage they are given."
authors: ["Wenxuan Zhou", "Sheng Zhang", "Hoifung Poon", "Muhao Chen"]
year: 2023
venue: "Findings of the Association for Computational Linguistics: EMNLP 2023"
source: "https://aclanthology.org/2023.findings-emnlp.968/"
doi: "10.18653/v1/2023.findings-emnlp.968"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhou-2023-context-faithful-prompting-large-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhou-2023-context-faithful-prompting-large-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/2/) / Context-faithful Prompting for Large Language Models

# Context-faithful Prompting for Large Language Models

**Authors:** Wenxuan Zhou, [Sheng Zhang](https://md.research.elamur.ai/free/en/authors/sheng-zhang/), [Hoifung Poon](https://md.research.elamur.ai/free/en/authors/hoifung-poon/), [Muhao Chen](https://md.research.elamur.ai/free/en/authors/muhao-chen/) · **Year:** 2023 · **Venue:** Findings of the Association for Computational Linguistics: EMNLP 2023 · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.findings-emnlp.968/) · [PDF](https://aclanthology.org/2023.findings-emnlp.968.pdf) · [DOI 10.18653/v1/2023.findings-emnlp.968](https://doi.org/10.18653/v1/2023.findings-emnlp.968) · [OpenAlex](https://openalex.org/W4389523675)

**Citations:** 46 citations (openalex-snapshot · 2026-10-14) · [99](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Large language models often favor facts memorized during pretraining over the passage they are given. This makes them unreliable for context-dependent tasks such as reading comprehension and information extraction. The authors study this "contextual faithfulness" in two settings. In entity-based knowledge conflict, the context contradicts what the model memorized. In prediction with abstention, the context simply lacks the answer. They test training-free prompting strategies: instructions, phrases limiting the question to the given text, and two new ideas. Opinion-based prompts recast the passage as something a narrator said and ask for that narrator's view. Counterfactual demonstrations are few-shot examples built on altered, false facts. Experiments use text-davinci-003 and other LLMs on Re-TACRED (relation extraction), Natural Questions and RealTime QA (MRC). Opinion-based framing gives the biggest single gain, and combining it with counterfactual examples works best. The gains do not hurt accuracy when context and memory agree. This matters because it offers a cheap, black-box way to make retrieval-augmented LLMs respect up-to-date or document-specific information.

## Key points
- With the proposed prompts, text-davinci-003's memorization ratio on Natural Questions drops from 35.2% to 3.0%. This ratio is the share of answers taken from memory rather than from the context.
- Counterfactual demonstrations help in knowledge-conflict cases, while ordinary factual demonstrations bring little or even negative effect.
- On the original, non-conflicting Natural Questions contexts, opinion-based prompts match or slightly beat the base prompts, so faithfulness gains do not hurt normal performance.
- Compared with smaller LLMs, larger ones are quicker to revise memorized answers, whether or not the new prompts are used. All experiments are English-only.
- The methods need no fine-tuning or white-box access. Code and data are available at github.com/wzhouad/context-faithful-llm.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

99 papers of our library cite it (of the 140884 whose bibliographies we read); 8 of them have a page here, 91 not yet.

- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://md.research.elamur.ai/free/en/papers/huang-2023-survey-hallucination-large-language-models/)
- [Trusting Your Evidence: Hallucinate Less with Context-aware Decoding](https://md.research.elamur.ai/free/en/papers/shi-2024-trusting-your-evidence-hallucinate-less/)
- [MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents](https://md.research.elamur.ai/free/en/papers/tang-2024-minicheck-efficient-fact-checking-llms/)
- [Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity](https://md.research.elamur.ai/free/en/papers/wang-2023-survey-factuality-large-language-models/)
- [Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts](https://md.research.elamur.ai/free/en/papers/xie-2023-adaptive-chameleon-stubborn-sloth-revealing/)
- [Knowledge Conflicts for LLMs: A Survey](https://md.research.elamur.ai/free/en/papers/xu-2024-knowledge-conflicts-llms-survey/)
- [Making Retrieval-Augmented Language Models Robust to Irrelevant Context](https://md.research.elamur.ai/free/en/papers/yoran-2023-making-retrieval-augmented-language-models/)
- [Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models](https://md.research.elamur.ai/free/en/papers/zhang-2023-siren-s-song-ai-ocean/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhou2023context,
  title = {{Context-faithful Prompting for Large Language Models}},
  author = {Zhou, Wenxuan and Zhang, Sheng and Poon, Hoifung and Chen, Muhao},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.968},
  url = {https://doi.org/10.18653/v1/2023.findings-emnlp.968}
}
```

- [← previous Content-Localization based System for Analyzing Sentiment and Hate Behaviors in Low-Resource Dialectal Arabic: English to Levantine and Gulf](https://md.research.elamur.ai/free/en/papers/alzamzami-2023-content-localization-based-system-analyzing/)
- [next → Coreference Resolution through a seq2seq Transition-Based System](https://md.research.elamur.ai/free/en/papers/bohnet-2023-coreference-resolution-through-seq2seq-transition/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/2/)

## References

56 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Johannes Bjerva, Nikita Bhutani, Behzad Golshan, Wang-Chiew Tan, and Isabelle Augenstein. 2020. SubjQA: A Dataset for Subjectivity and Review Comprehension. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5480–5494, Online. Association for Com… [find](https://scholar.google.com/scholar?q=Johannes+Bjerva%2C+Nikita+Bhutani%2C+Behzad+Golshan%2C+Wang%2DChiew+Tan%2C+and+Isabelle+Augenstein%2E+2020%2E+SubjQA%3A+A+Dataset+for+Subjectivity+and+Review+Comprehension%2E+In+Proceedings+of+the+2020+Conference+on+Em)
2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+et+al%2E+2020%2E+Language+models+are+few%2Dshot+lear)
3. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. QuAC: Question answering in context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2174–2184, Brussels, Belgium. Association for Computa… [find](https://scholar.google.com/scholar?q=Eunsol+Choi%2C+He+He%2C+Mohit+Iyyer%2C+Mark+Yatskar%2C+Wen%2Dtau+Yih%2C+Yejin+Choi%2C+Percy+Liang%2C+and+Luke+Zettlemoyer%2E+2018%2E+QuAC%3A+Question+answering+in+context%2E+In+Proceedings+of+the+2018+Conference+on+Empirical) [our card](https://md.research.elamur.ai/free/en/papers/choi-2018-quac-question-answering-context/)
4. C Chow. 1970. On optimum recognition error and reject tradeoff. IEEE Transactions on information theory, 16(1):41–46. [find](https://scholar.google.com/scholar?q=C+Chow%2E+1970%2E+On+optimum+recognition+error+and+reject+tradeoff%2E+IEEE+Transactions+on+information+theory%2C+16%281%29%3A41%E2%80%9346%2E)
5. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311. [find](https://scholar.google.com/scholar?q=Aakanksha+Chowdhery%2C+Sharan+Narang%2C+Jacob+Devlin%2C+Maarten+Bosma%2C+Gaurav+Mishra%2C+Adam+Roberts%2C+Paul+Barham%2C+Hyung+Won+Chung%2C+Charles+Sutton%2C+Sebastian+Gehrmann%2C+et+al%2E+2022%2E+Palm%3A+Scaling+language+mode)
6. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416. [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Le+Hou%2C+Shayne+Longpre%2C+Barret+Zoph%2C+Yi+Tay%2C+William+Fedus%2C+Eric+Li%2C+Xuezhi+Wang%2C+Mostafa+Dehghani%2C+Siddhartha+Brahma%2C+et+al%2E+2022%2E+Scaling+instruction%2Dfinetuned+language+models%2E+arXi)
7. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:… [find](https://scholar.google.com/scholar?q=Christopher+Clark%2C+Kenton+Lee%2C+Ming%2DWei+Chang%2C+Tom+Kwiatkowski%2C+Michael+Collins%2C+and+Kristina+Toutanova%2E+2019%2E+BoolQ%3A+Exploring+the+surprising+difficulty+of+natural+yes%2Fno+questions%2E+In+Proceedings+of) [our card](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/)
8. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457. [find](https://scholar.google.com/scholar?q=Peter+Clark%2C+Isaac+Cowhey%2C+Oren+Etzioni%2C+Tushar+Khot%2C+Ashish+Sabharwal%2C+Carissa+Schoenick%2C+and+Oyvind+Tafjord%2E+2018%2E+Think+you+have+solved+question+answering%3F+try+arc%2C+the+ai2+reasoning+challenge%2E+arX)
9. Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. 2016. Learning with rejection. In Algorithmic Learning Theory: 27th International Conference, ALT 2016, Bari, Italy, October 19-21, 2016, Proceedings 27, pages 67–82. Springer. [find](https://scholar.google.com/scholar?q=Corinna+Cortes%2C+Giulia+DeSalvo%2C+and+Mehryar+Mohri%2E+2016%2E+Learning+with+rejection%2E+In+Algorithmic+Learning+Theory%3A+27th+International+Conference%2C+ALT+2016%2C+Bari%2C+Italy%2C+October+19%2D21%2C+2016%2C+Proceedings)
10. Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021. Editing factual knowledge in language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6491– 6506, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nicola+De+Cao%2C+Wilker+Aziz%2C+and+Ivan+Titov%2E+2021%2E+Editing+factual+knowledge+in+language+models%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+6491%E2%80%93+6)
11. Giorgio Fumera and Fabio Roli. 2002. Support vector machines with embedded reject option. In Pattern Recognition with Support Vector Machines: First International Workshop, SVM 2002 Niagara Falls, Canada, August 10, 2002 Proceedings, pages 68–82. Springer. [find](https://scholar.google.com/scholar?q=Giorgio+Fumera+and+Fabio+Roli%2E+2002%2E+Support+vector+machines+with+embedded+reject+option%2E+In+Pattern+Recognition+with+Support+Vector+Machines%3A+First+International+Workshop%2C+SVM+2002+Niagara+Falls%2C+Can)
12. Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050–1059. PMLR. [find](https://scholar.google.com/scholar?q=Yarin+Gal+and+Zoubin+Ghahramani%2E+2016%2E+Dropout+as+a+bayesian+approximation%3A+Representing+model+uncertainty+in+deep+learning%2E+In+international+conference+on+machine+learning%2C+pages+1050%E2%80%931059%2E+PMLR%2E)
13. Luyu Gao and Jamie Callan. 2022. Unsupervised corpus aware language model pre-training for dense passage retrieval. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2843–2853, Dublin, Ireland. Association for Computational Ling… [find](https://scholar.google.com/scholar?q=Luyu+Gao+and+Jamie+Callan%2E+2022%2E+Unsupervised+corpus+aware+language+model+pre%2Dtraining+for+dense+passage+retrieval%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computational+Lingu) [our card](https://md.research.elamur.ai/free/en/papers/gao-2022-unsupervised-corpus-aware-language-model/)
14. Mansi Gupta, Nitish Kulkarni, Raghuveer Chanda, Anirudha Rayasam, and Zachary C Lipton. 2019. Amazonqa: A review-based question answering task. In International Joint Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Mansi+Gupta%2C+Nitish+Kulkarni%2C+Raghuveer+Chanda%2C+Anirudha+Rayasam%2C+and+Zachary+C+Lipton%2E+2019%2E+Amazonqa%3A+A+review%2Dbased+question+answering+task%2E+In+International+Joint+Conference+on+Artificial+Intellig)
15. Dan Hendrycks and Kevin Gimpel. 2017. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Dan+Hendrycks+and+Kevin+Gimpel%2E+2017%2E+A+baseline+for+detecting+misclassified+and+out%2Dof%2Ddistribution+examples+in+neural+networks%2E+In+International+Conference+on+Learning+Representations%2E)
16. Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. 2020. Pretrained transformers improve out-of-distribution robustness. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2744–2751, Online. Association for Comput… [find](https://scholar.google.com/scholar?q=Dan+Hendrycks%2C+Xiaoyuan+Liu%2C+Eric+Wallace%2C+Adam+Dziedzic%2C+Rishabh+Krishnan%2C+and+Dawn+Song%2E+2020%2E+Pretrained+transformers+improve+out%2Dof%2Ddistribution+robustness%2E+In+Proceedings+of+the+58th+Annual+Meeti) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-pretrained-transformers-improve-out-distribution/)
17. Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Few-shot learning with retrieval augmented language models. arXiv preprint arXiv:2208.03299. [find](https://scholar.google.com/scholar?q=Gautier+Izacard%2C+Patrick+Lewis%2C+Maria+Lomeli%2C+Lucas+Hosseini%2C+Fabio+Petroni%2C+Timo+Schick%2C+Jane+Dwivedi%2DYu%2C+Armand+Joulin%2C+Sebastian+Riedel%2C+and+Edouard+Grave%2E+2022%2E+Few%2Dshot+learning+with+retrieval+au)
18. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, Vancouver… [find](https://scholar.google.com/scholar?q=Mandar+Joshi%2C+Eunsol+Choi%2C+Daniel+Weld%2C+and+Luke+Zettlemoyer%2E+2017%2E+TriviaQA%3A+A+large+scale+distantly+supervised+challenge+dataset+for+reading+comprehension%2E+In+Proceedings+of+the+55th+Annual+Meeting+) [our card](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/)
19. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Onli… [find](https://scholar.google.com/scholar?q=Vladimir+Karpukhin%2C+Barlas+Oguz%2C+Sewon+Min%2C+Patrick+Lewis%2C+Ledell+Wu%2C+Sergey+Edunov%2C+Danqi+Chen%2C+and+Wen%2Dtau+Yih%2E+2020%2E+Dense+passage+retrieval+for+open%2Ddomain+question+answering%2E+In+Proceedings+of+th) [our card](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/)
20. Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A Smith, Yejin Choi, and Kentaro Inui. 2022. Realtime qa: What’s the answer right now? arXiv preprint arXiv:2207.13332. [find](https://scholar.google.com/scholar?q=Jungo+Kasai%2C+Keisuke+Sakaguchi%2C+Yoichi+Takahashi%2C+Ronan+Le+Bras%2C+Akari+Asai%2C+Xinyan+Yu%2C+Dragomir+Radev%2C+Noah+A+Smith%2C+Yejin+Choi%2C+and+Kentaro+Inui%2E+2022%2E+Realtime+qa%3A+What%E2%80%99s+the+answer+right+now%3F+arXi) [our card](https://md.research.elamur.ai/free/en/papers/kasai-2022-realtime-qa-s-answer-right/)
21. Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. 2022. Demonstrate-search-predict: Composing retrieval and language models for knowledge-intensive nlp. arXiv preprint arXiv:2212.14024. [find](https://scholar.google.com/scholar?q=Omar+Khattab%2C+Keshav+Santhanam%2C+Xiang+Lisa+Li%2C+David+Hall%2C+Percy+Liang%2C+Christopher+Potts%2C+and+Matei+Zaharia%2E+2022%2E+Demonstrate%2Dsearch%2Dpredict%3A+Composing+retrieval+and+language+models+for+knowledge%2Din) [our card](https://md.research.elamur.ai/free/en/papers/khattab-2022-demonstrate-search-predict-composing-retrieval/)
22. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural… [find](https://scholar.google.com/scholar?q=Tom+Kwiatkowski%2C+Jennimaria+Palomaki%2C+Olivia+Redfield%2C+Michael+Collins%2C+Ankur+Parikh%2C+Chris+Alberti%2C+Danielle+Epstein%2C+Illia+Polosukhin%2C+Jacob+Devlin%2C+Kenton+Lee%2C+Kristina+Toutanova%2C+Llion+Jones%2C+Matt) [our card](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)
23. Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30. [find](https://scholar.google.com/scholar?q=Balaji+Lakshminarayanan%2C+Alexander+Pritzel%2C+and+Charles+Blundell%2E+2017%2E+Simple+and+scalable+predictive+uncertainty+estimation+using+deep+ensembles%2E+Advances+in+neural+information+processing+systems%2C+3)
24. Angeliki Lazaridou, Elena Gribovskaya, Wojciech Stokowiec, and Nikolai Grigorev. 2022. Internet-augmented language models through few-shot prompting for open-domain question answering. arXiv preprint arXiv:2203.05115. [find](https://scholar.google.com/scholar?q=Angeliki+Lazaridou%2C+Elena+Gribovskaya%2C+Wojciech+Stokowiec%2C+and+Nikolai+Grigorev%2E+2022%2E+Internet%2Daugmented+language+models+through+few%2Dshot+prompting+for+open%2Ddomain+question+answering%2E+arXiv+preprint+) [our card](https://md.research.elamur.ai/free/en/papers/lazaridou-2022-internet-augmented-language-models-through/)
25. Angeliki Lazaridou, Adhi Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d’Autume, Tomas Kocisky, Sebastian Ruder, et al. 2021. Mind the gap: Assessing temporal generalization in neural language models. Advances in Neural Information Processing Sy… [find](https://scholar.google.com/scholar?q=Angeliki+Lazaridou%2C+Adhi+Kuncoro%2C+Elena+Gribovskaya%2C+Devang+Agrawal%2C+Adam+Liska%2C+Tayfun+Terzi%2C+Mai+Gimenez%2C+Cyprien+de+Masson+d%E2%80%99Autume%2C+Tomas+Kocisky%2C+Sebastian+Ruder%2C+et+al%2E+2021%2E+Mind+the+gap%3A+Asses)
26. Daliang Li, Ankit Singh Rawat, Manzil Zaheer, Xin Wang, Michal Lukasik, Andreas Veit, Felix Yu, and Sanjiv Kumar. 2022. Large language models with controllable working memory. arXiv preprint arXiv:2211.05110. [find](https://scholar.google.com/scholar?q=Daliang+Li%2C+Ankit+Singh+Rawat%2C+Manzil+Zaheer%2C+Xin+Wang%2C+Michal+Lukasik%2C+Andreas+Veit%2C+Felix+Yu%2C+and+Sanjiv+Kumar%2E+2022%2E+Large+language+models+with+controllable+working+memory%2E+arXiv+preprint+arXiv%3A221) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-large-language-models-controllable-working/)
27. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics… [find](https://scholar.google.com/scholar?q=Stephanie+Lin%2C+Jacob+Hilton%2C+and+Owain+Evans%2E+2022%2E+TruthfulQA%3A+Measuring+how+models+mimic+human+falsehoods%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+) [our card](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/)
28. Adam Liska, Tomas Kocisky, Elena Gribovskaya, Tayfun Terzi, Eren Sezener, Devang Agrawal, D’Autume Cyprien De Masson, Tim Scholtes, Manzil Zaheer, Susannah Young, et al. 2022. Streamingqa: A benchmark for adaptation to new knowledge over time in question answering models. In International Conference… [find](https://scholar.google.com/scholar?q=Adam+Liska%2C+Tomas+Kocisky%2C+Elena+Gribovskaya%2C+Tayfun+Terzi%2C+Eren+Sezener%2C+Devang+Agrawal%2C+D%E2%80%99Autume+Cyprien+De+Masson%2C+Tim+Scholtes%2C+Manzil+Zaheer%2C+Susannah+Young%2C+et+al%2E+2022%2E+Streamingqa%3A+A+benchmark)
29. Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 100–114… [find](https://scholar.google.com/scholar?q=Jiachang+Liu%2C+Dinghan+Shen%2C+Yizhe+Zhang%2C+Bill+Dolan%2C+Lawrence+Carin%2C+and+Weizhu+Chen%2E+2022%2E+What+makes+good+in%2Dcontext+examples+for+GPT%2D3%3F+In+Proceedings+of+Deep+Learning+Inside+Out+%28DeeLIO+2022%29%3A+The) [our card](https://md.research.elamur.ai/free/en/papers/liu-2022-what-makes-good-context-examples/)
30. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+a)
31. Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021. Entity-based knowledge conflicts in question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7052–7063, Online and Punta Cana, Dominican Re… [find](https://scholar.google.com/scholar?q=Shayne+Longpre%2C+Kartik+Perisetla%2C+Anthony+Chen%2C+Nikhil+Ramesh%2C+Chris+DuBois%2C+and+Sameer+Singh%2E+2021%2E+Entity%2Dbased+knowledge+conflicts+in+question+answering%2E+In+Proceedings+of+the+2021+Conference+on+Em) [our card](https://md.research.elamur.ai/free/en/papers/longpre-2021-entity-based-knowledge-conflicts-question/)
32. Keming Lu, I-Hung Hsu, Wenxuan Zhou, Mingyu Derek Ma, and Muhao Chen. 2022. Summarization as indirect supervision for relation extraction. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6575–6594, Abu Dhabi, United Arab Emirates. Association for Computational Linguis… [find](https://scholar.google.com/scholar?q=Keming+Lu%2C+I%2DHung+Hsu%2C+Wenxuan+Zhou%2C+Mingyu+Derek+Ma%2C+and+Muhao+Chen%2E+2022%2E+Summarization+as+indirect+supervision+for+relation+extraction%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A)
33. Kevin Meng, David Bau, Alex J Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Kevin+Meng%2C+David+Bau%2C+Alex+J+Andonian%2C+and+Yonatan+Belinkov%2E+2022%2E+Locating+and+editing+factual+associations+in+gpt%2E+In+Advances+in+Neural+Information+Processing+Systems%2E)
34. Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. 2023. Mass-editing memory in a transformer. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Kevin+Meng%2C+Arnab+Sen+Sharma%2C+Alex+Andonian%2C+Yonatan+Belinkov%2C+and+David+Bau%2E+2023%2E+Mass%2Dediting+memory+in+a+transformer%2E+In+International+Conference+on+Learning+Representations%2E)
35. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, Brussels, Belgium. Association fo… [find](https://scholar.google.com/scholar?q=Todor+Mihaylov%2C+Peter+Clark%2C+Tushar+Khot%2C+and+Ashish+Sabharwal%2E+2018%2E+Can+a+suit+of+armor+conduct+electricity%3F+a+new+dataset+for+open+book+question+answering%2E+In+Proceedings+of+the+2018+Conference+on+)
36. Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. 2022. Fast model editing at scale. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Eric+Mitchell%2C+Charles+Lin%2C+Antoine+Bosselut%2C+Chelsea+Finn%2C+and+Christopher+D+Manning%2E+2022%2E+Fast+model+editing+at+scale%2E+In+International+Conference+on+Learning+Representations%2E)
37. Ella Neeman, Roee Aharoni, Or Honovich, Leshem Choshen, Idan Szpektor, and Omri Abend. 2022. Disentqa: Disentangling parametric and contextual knowledge with counterfactual question answering. arXiv preprint arXiv:2211.05655. [find](https://scholar.google.com/scholar?q=Ella+Neeman%2C+Roee+Aharoni%2C+Or+Honovich%2C+Leshem+Choshen%2C+Idan+Szpektor%2C+and+Omri+Abend%2E+2022%2E+Disentqa%3A+Disentangling+parametric+and+contextual+knowledge+with+counterfactual+question+answering%2E+arXiv+p)
38. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789, Melbourne, Australia. Association for Computational Ling… [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Robin+Jia%2C+and+Percy+Liang%2E+2018%2E+Know+what+you+don%E2%80%99t+know%3A+Unanswerable+questions+for+SQuAD%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguisti) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/)
39. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Jian+Zhang%2C+Konstantin+Lopyrev%2C+and+Percy+Liang%2E+2016%2E+SQuAD%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natu) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
40. Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249–266. [find](https://scholar.google.com/scholar?q=Siva+Reddy%2C+Danqi+Chen%2C+and+Christopher+D%2E+Manning%2E+2019%2E+CoQA%3A+A+conversational+question+answering+challenge%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+7%3A249%E2%80%93266%2E) [our card](https://md.research.elamur.ai/free/en/papers/reddy-2019-coqa-conversational-question-answering-challenge/)
41. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992,… [find](https://scholar.google.com/scholar?q=Nils+Reimers+and+Iryna+Gurevych%2E+2019%2E+Sentence%2DBERT%3A+Sentence+embeddings+using+Siamese+BERT%2Dnetworks%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+the)
42. Erik F Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. arXiv preprint cs/0306050. [find](https://scholar.google.com/scholar?q=Erik+F+Sang+and+Fien+De+Meulder%2E+2003%2E+Introduction+to+the+conll%2D2003+shared+task%3A+Language%2Dindependent+named+entity+recognition%2E+arXiv+preprint+cs%2F0306050%2E)
43. Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. 2022. Col-BERTv2: Effective and efficient retrieval via lightweight late interaction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan… [find](https://scholar.google.com/scholar?q=Keshav+Santhanam%2C+Omar+Khattab%2C+Jon+Saad%2DFalcon%2C+Christopher+Potts%2C+and+Matei+Zaharia%2E+2022%2E+Col%2DBERTv2%3A+Effective+and+efficient+retrieval+via+lightweight+late+interaction%2E+In+Proceedings+of+the+2022+)
44. Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Boyd-Graber, and Lijuan Wang. 2023. Prompting gpt-3 to be reliable. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Chenglei+Si%2C+Zhe+Gan%2C+Zhengyuan+Yang%2C+Shuohang+Wang%2C+Jianfeng+Wang%2C+Jordan+Boyd%2DGraber%2C+and+Lijuan+Wang%2E+2023%2E+Prompting+gpt%2D3+to+be+reliable%2E+In+International+Conference+on+Learning+Representations%2E) [our card](https://md.research.elamur.ai/free/en/papers/si-2022-prompting-gpt-3-reliable/)
45. George Stoica, Emmanouil Antonios Platanios, and Barnabás Póczos. 2021. Re-tacred: Addressing shortcomings of the tacred dataset. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 13843–13850. [find](https://scholar.google.com/scholar?q=George+Stoica%2C+Emmanouil+Antonios+Platanios%2C+and+Barnab%C3%A1s+P%C3%B3czos%2E+2021%2E+Re%2Dtacred%3A+Addressing+shortcomings+of+the+tacred+dataset%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+volu)
46. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+2023%2E+Llama+2%3A+Open+foundation+and+fi)
47. Haoyu Wang, Hongming Zhang, Yuqian Deng, Jacob Gardner, Dan Roth, and Muhao Chen. 2023. Extracting or guessing? improving faithfulness of event temporal relation extraction. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 541–553,… [find](https://scholar.google.com/scholar?q=Haoyu+Wang%2C+Hongming+Zhang%2C+Yuqian+Deng%2C+Jacob+Gardner%2C+Dan+Roth%2C+and+Muhao+Chen%2E+2023%2E+Extracting+or+guessing%3F+improving+faithfulness+of+event+temporal+relation+extraction%2E+In+Proceedings+of+the+17th)
48. Yiwei Wang, Muhao Chen, Wenxuan Zhou, Yujun Cai, Yuxuan Liang, Dayiheng Liu, Baosong Yang, Juncheng Liu, and Bryan Hooi. 2022. Should we rely on entity mentions for relation extraction? debiasing relation extraction with counterfactual analysis. In Proceedings of the 2022 Conference of the North Ame… [find](https://scholar.google.com/scholar?q=Yiwei+Wang%2C+Muhao+Chen%2C+Wenxuan+Zhou%2C+Yujun+Cai%2C+Yuxuan+Liang%2C+Dayiheng+Liu%2C+Baosong+Yang%2C+Juncheng+Liu%2C+and+Bryan+Hooi%2E+2022%2E+Should+we+rely+on+entity+mentions+for+relation+extraction%3F+debiasing+rela)
49. Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2022. Finetuned language models are zero-shot learners. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Maarten+Bosma%2C+Vincent+Zhao%2C+Kelvin+Guu%2C+Adams+Wei+Yu%2C+Brian+Lester%2C+Nan+Du%2C+Andrew+M+Dai%2C+and+Quoc+V+Le%2E+2022%2E+Finetuned+language+models+are+zero%2Dshot+learners%2E+In+International+Conference)
50. Ji Xin, Raphael Tang, Yaoliang Yu, and Jimmy Lin. 2021. The art of abstention: Selective prediction and error regularization for natural language processing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natu… [find](https://scholar.google.com/scholar?q=Ji+Xin%2C+Raphael+Tang%2C+Yaoliang+Yu%2C+and+Jimmy+Lin%2E+2021%2E+The+art+of+abstention%3A+Selective+prediction+and+error+regularization+for+natural+language+processing%2E+In+Proceedings+of+the+59th+Annual+Meeting+)
51. Mark Yatskar. 2019. A qualitative comparison of CoQA, SQuAD 2.0 and QuAC. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2318–2323, Minneapolis, Minnesota. Asso… [find](https://scholar.google.com/scholar?q=Mark+Yatskar%2E+2019%2E+A+qualitative+comparison+of+CoQA%2C+SQuAD+2%2E0+and+QuAC%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics%3A+Human+La)
52. Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Position-aware attention and supervised data improve slot filling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 35–45, Copenhagen, Denmark. Association for Compu… [find](https://scholar.google.com/scholar?q=Yuhao+Zhang%2C+Victor+Zhong%2C+Danqi+Chen%2C+Gabor+Angeli%2C+and+Christopher+D%2E+Manning%2E+2017%2E+Position%2Daware+attention+and+supervised+data+improve+slot+filling%2E+In+Proceedings+of+the+2017+Conference+on+Empir)
53. Wenxuan Zhou and Muhao Chen. 2022. An improved baseline for sentence-level relation extraction. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Shor… [find](https://scholar.google.com/scholar?q=Wenxuan+Zhou+and+Muhao+Chen%2E+2022%2E+An+improved+baseline+for+sentence%2Dlevel+relation+extraction%2E+In+Proceedings+of+the+2nd+Conference+of+the+Asia%2DPacific+Chapter+of+the+Association+for+Computational+Li)
54. Wenxuan Zhou, Fangyu Liu, and Muhao Chen. 2021. Contrastive out-of-distribution detection for pre-trained transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1100–1111, Online and Punta Cana, Dominican Republic. Association for Computational… [find](https://scholar.google.com/scholar?q=Wenxuan+Zhou%2C+Fangyu+Liu%2C+and+Muhao+Chen%2E+2021%2E+Contrastive+out%2Dof%2Ddistribution+detection+for+pre%2Dtrained+transformers%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural+Language+P)
55. RealTime QA: Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022. Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910. [find](https://scholar.google.com/scholar?q=RealTime+QA%3A+Yongchao+Zhou%2C+Andrei+Ioan+Muresanu%2C+Ziwen+Han%2C+Keiran+Paster%2C+Silviu+Pitis%2C+Harris+Chan%2C+and+Jimmy+Ba%2E+2022%2E+Large+language+models+are+human%2Dlevel+prompt+engineers%2E+arXiv+preprint+arXiv%3A) [our card](https://md.research.elamur.ai/free/en/papers/zhou-2022-large-language-models-human-level/)
56. Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. 2020. Modifying memories in transformer models. arXiv preprint arXiv:2012.00363. [find](https://scholar.google.com/scholar?q=Chen+Zhu%2C+Ankit+Singh+Rawat%2C+Manzil+Zaheer%2C+Srinadh+Bhojanapalli%2C+Daliang+Li%2C+Felix+Yu%2C+and+Sanjiv+Kumar%2E+2020%2E+Modifying+memories+in+transformer+models%2E+arXiv+preprint+arXiv%3A2012%2E00363%2E)