---
title: "Rich Knowledge Sources Bring Complex Knowledge Conflicts:"
description: "Open-retrieval QA systems can draw on two kinds of knowledge: facts memorized in a language… 46 references, cited by 73 papers here."
authors: ["Hung-Ting Chen", "Michael Zhang", "Eunsol Choi"]
year: 2022
venue: "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2022.emnlp-main.146/"
doi: "10.18653/v1/2022.emnlp-main.146"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chen-2022-rich-knowledge-sources-bring-complex/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chen-2022-rich-knowledge-sources-bring-complex/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/3/) / Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidence

# Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidence

**Authors:** Hung-Ting Chen, [Michael Zhang](https://md.research.elamur.ai/free/en/authors/michael-zhang/) and [Eunsol Choi](https://md.research.elamur.ai/free/en/authors/eunsol-choi/) · **Year:** 2022 · **Venue:** [Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2022](https://md.research.elamur.ai/free/en/venues/emnlp/2022/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.emnlp-main.146/) · [PDF](https://aclanthology.org/2022.emnlp-main.146.pdf) · [DOI 10.18653/v1/2022.emnlp-main.146](https://doi.org/10.18653/v1/2022.emnlp-main.146) · [OpenAlex](https://openalex.org/W4385574183)

**Citations:** 30 citations (OpenAlex, 2026-10-14) · [73](https://md.research.elamur.ai/free/en/papers/chen-2022-rich-knowledge-sources-bring-complex/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Open-retrieval QA systems can draw on two kinds of knowledge: facts memorized in a language model's weights and up to a hundred retrieved passages. Earlier work tended to treat these sources as mutually consistent, and it studied conflicts only with single passages. This paper asks which source models trust when the sources disagree. The authors use Fusion-in-Decoder (T5-large reader) and RAG, both with a DPR retriever over Wikipedia, on NQ-Open and TriviaQA. They simulate several conflict scenarios: substituting answer entities in some passages, applying semantic edits such as negation, and building real ambiguity sets from AmbigQA and SituatedQA.

With a high-recall retriever, the models mostly copy spans from a few top passages rather than generating answers from memory. When passages disagree, parametric knowledge acts mainly as a tie-breaker. Model confidence barely drops when evidence conflicts. A separately trained XGBoost calibrator, fed with augmented conflict data, learns to abstain to some degree, but this skill transfers poorly across conflict types. Trustworthy QA over messy, outdated or contradictory corpora therefore remains an open problem.

## Key points
- FiD produces answers absent from the retrieved passages for only 3.4% (NQ) and 6.2% (TriviaQA) of dev examples; RAG does so for 20.2% on NQ.
- The knowledge corpus is the English Wikipedia dump of Dec. 20, 2018, split into 100-word passages. The SituatedQA setting additionally retrieves from a 2021 snapshot.
- Models largely ignore perturbations such as negation and still return the answer entity, much like extractive readers.
- Calibrators trained on realistic conflict sets (AmbigQA, SituatedQA) generalize better than those trained on simulated partial substitution, even though the realistic sets are much smaller. Training on all conflict types jointly gives only modest gains.
- FiD's accuracy gap between training data (80%) and test data (52%) may explain why raw generation probability beat the base calibrator out of domain.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [The University of Texas at Austin](https://md.research.elamur.ai/free/en/universities/the-university-of-texas-at-austin/) — University, Austin, United States · 72 works · 55,722 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 523 works

## Cite

Cite as: Hung-Ting Chen, Michael Zhang and Eunsol Choi. 2022. Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidence. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{chen2022rich,
  title = {{Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidence}},
  author = {Hung-Ting Chen, Michael Zhang and Eunsol Choi},
  booktitle = {Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.146},
  url = {https://doi.org/10.18653/v1/2022.emnlp-main.146}
}
```

- [← previous NTREX-128 – News Test References for MT Evaluation of 128 Languages](https://md.research.elamur.ai/free/en/papers/federmann-2022-ntrex-128-news-test-references/)
- [next → Learning to Reason Deductively: Math Word Problem Solving as Complex Relation Extraction](https://md.research.elamur.ai/free/en/papers/jie-2022-learning-reason-deductively-math-word/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/3/)

## References

46 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Max Bartolo, A Roberts, Johannes Welbl, Sebastian Riedel, and Pontus Stenetorp. 2020. Beat the ai: Investigating adversarial human annotation for reading comprehension. Transactions of the Association for Computational Linguistics, 8:662–678. [find](https://scholar.google.com/scholar?q=Max+Bartolo%2C+A+Roberts%2C+Johannes+Welbl%2C+Sebastian+Riedel%2C+and+Pontus+Stenetorp%2E+2020%2E+Beat+the+ai%3A+Investigating+adversarial+human+annotation+for+reading+comprehension%2E+Transactions+of+the+Association)
2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
3. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. [Reading Wikipedia to answer open-domain questions](https://md.research.elamur.ai/free/en/papers/chen-2017-reading-wikipedia-answer-open-domain/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870–1879, Vancouver, Canada. Association for Computational Lin…
4. Sihao Chen, Daniel Khashabi, Wenpeng Yin, Chris Callison-Burch, and Dan Roth. 2019. [Seeing things from a different angle:discovering diverse perspectives about claims](https://md.research.elamur.ai/free/en/papers/chen-2019-seeing-things-different-angle-discovering/). ArXiv, abs/1906.03538.
5. Tianqi Chen and Carlos Guestrin. 2016. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, pages 785–794, New York, NY, USA. ACM. [find](https://scholar.google.com/scholar?q=Tianqi+Chen+and+Carlos+Guestrin%2E+2016%2E+XGBoost%3A+A+scalable+tree+boosting+system%2E+In+Proceedings+of+the+22nd+ACM+SIGKDD+International+Conference+on+Knowledge+Discovery+and+Data+Mining%2C+KDD+%E2%80%9916%2C+pages+7)
6. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). arXiv preprint arXiv:2204.02311.
7. Chris Donahue, Mina Lee, and Percy Liang. 2020. [Enabling language models to fill in the blanks](https://md.research.elamur.ai/free/en/papers/donahue-2020-enabling-language-models-fill-blanks/). arXiv preprint arXiv:2005.05339.
8. Y. Du, Antoine Bosselut, and Christopher D. Manning. 2022. Synthetic disinformation attacks on automated fact verification systems. ArXiv, abs/2202.09381. — [arXiv:2202.09381](https://arxiv.org/abs/2202.09381) [find](https://scholar.google.com/scholar?q=Y%2E+Du%2C+Antoine+Bosselut%2C+and+Christopher+D%2E+Manning%2E+2022%2E+Synthetic+disinformation+attacks+on+automated+fact+verification+systems%2E+ArXiv%2C+abs%2F2202%2E09381%2E+%E2%80%94+%5BarXiv%3A2202%2E09381%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F22)
9. Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. [ELI5: Long form question answering](https://md.research.elamur.ai/free/en/papers/fan-2019-eli5-long-form-question-answering/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3558–3567, Florence, Italy. Association for Computational Linguistics.
10. Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. 2019. [find](https://scholar.google.com/scholar?q=Adam+Fisch%2C+Alon+Talmor%2C+Robin+Jia%2C+Minjoon+Seo%2C+Eunsol+Choi%2C+and+Danqi+Chen%2E+2019%2E)
11. [MRQA 2019 shared task: Evaluating generalization in reading comprehension](https://md.research.elamur.ai/free/en/papers/fisch-2019-mrqa-2019-shared-task-evaluating/). In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 1–13, Hong Kong, China. Association for Computational Linguistics.
12. [Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, N…](https://md.research.elamur.ai/free/en/papers/gardner-2020-evaluating-models-local-decision-boundaries/)
13. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. REALM: Retrieval-augmented language model pre-training. Proceedings of the International Conference on Machine Learning (ICML). [find](https://scholar.google.com/scholar?q=Kelvin+Guu%2C+Kenton+Lee%2C+Zora+Tung%2C+Panupong+Pasupat%2C+and+Ming%2DWei+Chang%2E+2020%2E+REALM%3A+Retrieval%2Daugmented+language+model+pre%2Dtraining%2E+Proceedings+of+the+International+Conference+on+Machine+Learning+%28)
14. Gautier Izacard and Edouard Grave. 2020. Distilling knowledge from reader to retriever for question answering. arXiv preprint arXiv:2012.04584. — [arXiv:2012.04584](https://arxiv.org/abs/2012.04584) [find](https://scholar.google.com/scholar?q=Gautier+Izacard+and+Edouard+Grave%2E+2020%2E+Distilling+knowledge+from+reader+to+retriever+for+question+answering%2E+arXiv+preprint+arXiv%3A2012%2E04584%2E+%E2%80%94+%5BarXiv%3A2012%2E04584%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2012%2E04584%29)
15. Gautier Izacard and Edouard Grave. 2021. [Leveraging passage retrieval with generative models for open domain question answering](https://md.research.elamur.ai/free/en/papers/izacard-2021-leveraging-passage-retrieval-generative-models/). In EACL.
16. Robin Jia and Percy Liang. 2017. [Adversarial examples for evaluating reading comprehension systems](https://md.research.elamur.ai/free/en/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/). Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), abs/1707.07328.
17. Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. [Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/). In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
18. Amita Kamath, Robin Jia, and Percy Liang. 2020. Selective question answering under domain shift. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=Amita+Kamath%2C+Robin+Jia%2C+and+Percy+Liang%2E+2020%2E+Selective+question+answering+under+domain+shift%2E+In+Proceedings+of+the+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28ACL%29%2E)
19. Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. [Dense passage retrieval for open-domain question answering](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/). Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
20. Divyansh Kaushik, Eduard Hovy, and Zachary C Lipton. 2020. Learning the difference that makes a difference with counterfactually augmented data. International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Divyansh+Kaushik%2C+Eduard+Hovy%2C+and+Zachary+C+Lipton%2E+2020%2E+Learning+the+difference+that+makes+a+difference+with+counterfactually+augmented+data%2E+International+Conference+on+Learning+Representations+%28I)
21. Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021. [Hurdles to progress in long-form question answering](https://md.research.elamur.ai/free/en/papers/krishna-2021-hurdles-progress-long-form-question/). ArXiv, abs/2103.06332.
22. T. Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, C. Alberti, D. Epstein, Illia Polosukhin, J. Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. [Natural questions:…](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)
23. Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. [Latent retrieval for weakly supervised open domain question answering](https://md.research.elamur.ai/free/en/papers/lee-2019-latent-retrieval-weakly-supervised-open/). In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 6086–6096, Florence, Italy. Association for Computational Linguistics.
24. Patrick Lewis, Ethan Perez, Aleksandara Piktus, F. Petroni, V. Karpukhin, Naman Goyal, Heinrich Kuttler, M. Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Proceedings of Advances in Neural Information Pr… [find](https://scholar.google.com/scholar?q=Patrick+Lewis%2C+Ethan+Perez%2C+Aleksandara+Piktus%2C+F%2E+Petroni%2C+V%2E+Karpukhin%2C+Naman+Goyal%2C+Heinrich+Kuttler%2C+M%2E+Lewis%2C+Wen+tau+Yih%2C+Tim+Rockt%C3%A4schel%2C+Sebastian+Riedel%2C+and+Douwe+Kiela%2E+2020%2E+Retrieval%2Daugm)
25. Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel. 2021. Question and answer test-train overlap in open-domain question answering datasets. In EACL. [find](https://scholar.google.com/scholar?q=Patrick+Lewis%2C+Pontus+Stenetorp%2C+and+Sebastian+Riedel%2E+2021%2E+Question+and+answer+test%2Dtrain+overlap+in+open%2Ddomain+question+answering+datasets%2E+In+EACL%2E)
26. Shayne Longpre, Kartik Kumar Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021. [Entity-based knowledge conflicts in question answering](https://md.research.elamur.ai/free/en/papers/longpre-2021-entity-based-knowledge-conflicts-question/). ArXiv, abs/2109.05052.
27. Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2018%2E+Decoupled+weight+decay+regularization%2E+In+International+Conference+on+Learning+Representations%2E)
28. Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. Ambigqa: Answering ambiguous open-domain questions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Sewon+Min%2C+Julian+Michael%2C+Hannaneh+Hajishirzi%2C+and+Luke+Zettlemoyer%2E+2020%2E+Ambigqa%3A+Answering+ambiguous+open%2Ddomain+questions%2E+In+Proceedings+of+the+Conference+on+Empirical+Methods+in+Natural+Languag)
29. Liangming Pan, Wenhu Chen, Min-Yen Kan, and William Yang Wang. 2021. Contraqa: Question answering under contradicting contexts. arXiv preprint arXiv:2110.07803. — [arXiv:2110.07803](https://arxiv.org/abs/2110.07803) [find](https://scholar.google.com/scholar?q=Liangming+Pan%2C+Wenhu+Chen%2C+Min%2DYen+Kan%2C+and+William+Yang+Wang%2E+2021%2E+Contraqa%3A+Question+answering+under+contradicting+contexts%2E+arXiv+preprint+arXiv%3A2110%2E07803%2E+%E2%80%94+%5BarXiv%3A2110%2E07803%5D%28https%3A%2F%2Farxiv%2Eorg%2F)
30. Bhargavi Paranjape, Matthew Lamm, and Ian Tenney. 2021. Retrieval-guided counterfactual generation for qa. ArXiv, abs/2110.07596. [find](https://scholar.google.com/scholar?q=Bhargavi+Paranjape%2C+Matthew+Lamm%2C+and+Ian+Tenney%2E+2021%2E+Retrieval%2Dguided+counterfactual+generation+for+qa%2E+ArXiv%2C+abs%2F2110%2E07596%2E)
31. Fabio Petroni, Tim Rocktäschel, A. H. Miller, P. Lewis, A. Bakhtin, Y. Wu, and S. Riedel. 2019. [Language models as knowledge bases?](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/) In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
32. Peng Qi, Timothy Dozat, Yuhao Zhang, and Christopher D. Manning. 2018. Universal dependency parsing from scratch. In Proceedings of the [CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies](https://md.research.elamur.ai/free/en/papers/zeman-2018-conll-2018-shared-task-multilingual/), pages 160–170, Brussels, Belgium. Association for Computational Linguistics.
33. Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446. [find](https://scholar.google.com/scholar?q=Jack+W+Rae%2C+Sebastian+Borgeaud%2C+Trevor+Cai%2C+Katie+Millican%2C+Jordan+Hoffmann%2C+Francis+Song%2C+John+Aslanides%2C+Sarah+Henderson%2C+Roman+Ring%2C+Susannah+Young%2C+et+al%2E+2021%2E+Scaling+language+models%3A+Methods%2C+a)
34. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of Machine Learning Research, 21(140):1–67.
35. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. ["why should i trust you?": Explaining the predictions of any classifier](https://md.research.elamur.ai/free/en/papers/ribeiro-2016-why-should-i-trust-explaining/). Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.
36. Marco Túlio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of nlp models with checklist. [find](https://scholar.google.com/scholar?q=Marco+T%C3%BAlio+Ribeiro%2C+Tongshuang+Wu%2C+Carlos+Guestrin%2C+and+Sameer+Singh%2E+2020%2E+Beyond+accuracy%3A+Behavioral+testing+of+nlp+models+with+checklist%2E)
37. Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Adam+Roberts%2C+Colin+Raffel%2C+and+Noam+Shazeer%2E+2020%2E+How+much+knowledge+can+you+pack+into+the+parameters+of+a+language+model%3F+In+Proceedings+of+the+Conference+on+Empirical+Methods+in+Natural+Language+P)
38. Devendra Singh Sachan, Siva Reddy, William Hamilton, Chris Dyer, and Dani Yogatama. 2021. End-to-end training of multi-document reader and retriever for [open-domain question answering](https://md.research.elamur.ai/free/en/papers/chen-2020-open-domain-question-answering/). NeurIPS, abs/2106.05346.
39. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. [Bidirectional attention flow for machine comprehension](https://md.research.elamur.ai/free/en/papers/seo-2016-bidirectional-attention-flow-machine-comprehension/). International Conference on Learning Representations (ICLR), abs/1611.01603.
40. Darsh J. Shah, Tal Schuster, and Regina Barzilay. 2020. Automatic fact-guided sentence modification. In AAAI. [find](https://scholar.google.com/scholar?q=Darsh+J%2E+Shah%2C+Tal+Schuster%2C+and+Regina+Barzilay%2E+2020%2E+Automatic+fact%2Dguided+sentence+modification%2E+In+AAAI%2E)
41. Haitian Sun, William W. Cohen, and Ruslan Salakhutdinov. 2022. Reasoning over logically interacted conditions for question answering. ArXiv, abs/2205.12898. — [arXiv:2205.12898](https://arxiv.org/abs/2205.12898) [find](https://scholar.google.com/scholar?q=Haitian+Sun%2C+William+W%2E+Cohen%2C+and+Ruslan+Salakhutdinov%2E+2022%2E+Reasoning+over+logically+interacted+conditions+for+question+answering%2E+ArXiv%2C+abs%2F2205%2E12898%2E+%E2%80%94+%5BarXiv%3A2205%2E12898%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F)
42. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. [Lamda: Language models for dialog applications](https://md.research.elamur.ai/free/en/papers/thoppilan-2022-lamda-language-models-dialog-applications/). arXiv preprint arXiv:2201.08239.
43. Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel Weld. 2021. [Polyjuice: Generating counterfactuals for explaining, evaluating, and improving models](https://md.research.elamur.ai/free/en/papers/wu-2021-polyjuice-generating-counterfactuals-explaining-evaluating/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on N…
44. Sohee Yang and Minjoon Seo. 2020. Is retriever merely an approximator of reader? ArXiv, abs/2010.10999. — [arXiv:2010.10999](https://arxiv.org/abs/2010.10999) [find](https://scholar.google.com/scholar?q=Sohee+Yang+and+Minjoon+Seo%2E+2020%2E+Is+retriever+merely+an+approximator+of+reader%3F+ArXiv%2C+abs%2F2010%2E10999%2E+%E2%80%94+%5BarXiv%3A2010%2E10999%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2010%2E10999%29)
45. Michael J.Q. Zhang and Eunsol Choi. 2021. Situatedqa: Incorporating extra-linguistic contexts into qa. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Michael+J%2EQ%2E+Zhang+and+Eunsol+Choi%2E+2021%2E+Situatedqa%3A+Incorporating+extra%2Dlinguistic+contexts+into+qa%2E+In+Proceedings+of+the+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2E)
46. Shujian Zhang, Chengyue Gong, and Eunsol Choi. 2021. Knowing more about questions can help: Improving calibration in question answering. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1958–1970, Online. Association for Computational Linguistics. \\\#Passages neg.… [find](https://scholar.google.com/scholar?q=Shujian+Zhang%2C+Chengyue+Gong%2C+and+Eunsol+Choi%2E+2021%2E+Knowing+more+about+questions+can+help%3A+Improving+calibration+in+question+answering%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+A)

*Updated 24 October 2026*