---
title: "Out-of-Distribution Detection and Selective Generation for"
description: "They then propose cheap OOD scores computed from the model's existing Transformer… 68 references, cited by 68 papers here."
authors: ["Jie Ren", "Jiaming Luo", "Yao Zhao", "Kundan Krishna", "Mohammad Saleh", "Balaji Lakshminarayanan", "Peter J. Liu"]
year: 2022
venue: "arXiv:2209.15558"
source: "https://arxiv.org/abs/2209.15558"
doi: "10.48550/arxiv.2209.15558"
arxiv: "2209.15558"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ren-2022-out-distribution-detection-selective-generation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ren-2022-out-distribution-detection-selective-generation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/5/) / Out-of-Distribution Detection and Selective Generation for Conditional Language Models

# Out-of-Distribution Detection and Selective Generation for Conditional Language Models

**Authors:** [Jie Ren](https://md.research.elamur.ai/free/en/authors/jie-ren/), [Jiaming Luo](https://md.research.elamur.ai/free/en/authors/jiaming-luo/), [Yao Zhao](https://md.research.elamur.ai/free/en/authors/yao-zhao/), [Kundan Krishna](https://md.research.elamur.ai/free/en/authors/kundan-krishna/), [Mohammad Saleh](https://md.research.elamur.ai/free/en/authors/mohammad-saleh/), Balaji Lakshminarayanan and [Peter J. Liu](https://md.research.elamur.ai/free/en/authors/peter-j-liu/) · **Year:** 2022 · **Venue:** arXiv:2209.15558 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2209.15558) · [PDF](https://arxiv.org/pdf/2209.15558) · [DOI 10.48550/arxiv.2209.15558](https://doi.org/10.48550/arxiv.2209.15558) · [arXiv 2209.15558](https://arxiv.org/abs/2209.15558) · [OpenAlex](https://openalex.org/W4300980582)

**Citations:** 12 citations (OpenAlex, 2026-10-14) · [68](https://md.research.elamur.ai/free/en/papers/ren-2022-out-distribution-detection-selective-generation/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Conditional language models (CLMs) used for summarization and translation can fail badly when user inputs differ from the training data, and their own likelihood is a poor warning signal. The authors show that perplexity distributions for in-domain and out-of-distribution (OOD) data overlap heavily. They then propose cheap OOD scores computed from the model's existing Transformer encoder-decoder representations. Averaged final-layer encoder states (input) and decoder states (output) are modeled with Gaussians, and the Mahalanobis distance to the training domain is contrasted with the distance to a broad background domain. This yields a Relative Mahalanobis Distance (RMD). A logistic-regression variant on the same embeddings is also tested. No separate detector network is needed. The scores clearly beat perplexity-based baselines. The paper also introduces selective generation, in which the model abstains on outputs likely to be poor. Combining perplexity with the OOD score predicts quality better under distribution shift than perplexity alone. This work matters for deploying generative models safely on real, shifted inputs.

## Key points
- **Summarization setup.** The summarization model is trained on xsum (BBC news).
  - cnn_dailymail and newsroom serve as near-OOD sets.
  - forumsum, samsum and reddit_tifu serve as far-OOD sets.
  - Perplexity even rates cnn_dailymail as more "in-domain" than xsum itself.
- **Translation setup.** The translation model is trained on WMT15 and tested on other WMT splits, OPUS100 and MTNT. Here, perplexity overlap between in-domain and OOD data is even worse.
- **Background data for RMD.** The background Gaussians are fitted on C4 for summarization and on ParaCrawl English-French for translation. The output background is built from the model's own decoded outputs, so no labeled pairs are required.
- **Observed failure mode.** On CNN articles, the BBC-trained summarizer sometimes produces the summary "All images are copyrighted".
- **Evaluation framework.** The authors provide a framework for OOD detection and selective generation that includes human quality ratings of summaries. These ratings are used to assess a combined score of perplexity and output RMD.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 492 works · 440,463 citations
- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 598 works · 191,226 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [📝 Summarization](https://md.research.elamur.ai/free/en/topics/summarization/) · 243 works

## Cite

Cite as: Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mohammad Saleh, Balaji Lakshminarayanan and Peter J. Liu. 2022. Out-of-Distribution Detection and Selective Generation for Conditional Language Models. arXiv:2209.15558.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{ren2022out,
  title = {{Out-of-Distribution Detection and Selective Generation for Conditional Language Models}},
  author = {Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mohammad Saleh, Balaji Lakshminarayanan and Peter J. Liu},
  year = {2022},
  doi = {10.48550/arxiv.2209.15558},
  eprint = {2209.15558},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2209.15558}
}
```

- [← previous AraBART: a Pretrained Arabic Sequence-to-Sequence Model for Abstractive Summarization](https://md.research.elamur.ai/free/en/papers/eddine-2022-arabart-pretrained-arabic-sequence-sequence/)
- [next → Thor: Wielding Hammers to Integrate Language Models and Automated Theorem Provers](https://md.research.elamur.ai/free/en/papers/jiang-2022-thor-wielding-hammers-integrate-language/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/5/)

## References

68 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Udit Arora, William Huang, and He He. Types of out-of-distribution texts and how to detect them. arXiv preprint arXiv:2109.06827, 2021. [find](https://scholar.google.com/scholar?q=Udit+Arora%2C+William+Huang%2C+and+He+He%2E+Types+of+out%2Dof%2Ddistribution+texts+and+how+to+detect+them%2E+arXiv+preprint+arXiv%3A2109%2E06827%2C+2021%2E)
2. Mikko Aulamo and Jörg Tiedemann. The OPUS resource repository: An open package for creating parallel corpora and machine translation services. In Proceedings of the 22nd Nordic Conference on Computational Linguistics, pp. 389–394, Turku, Finland, September–October 2019. Linköping University Electr… [find](https://scholar.google.com/scholar?q=Mikko+Aulamo+and+Jo%CC%88rg+Tiedemann%2E+The+OPUS+resource+repository%3A+An+open+package+for+creating+parallel+corpora+and+machine+translation+services%2E+In+Proceedings+of+the+22nd+Nordic+Conference+on+Computat)
3. Marta Bañón, Pinzhen Chen, Barry Haddow, Kenneth Heafield, Hieu Hoang, Miquel Esplà-Gomis, Mikel L. Forcada, Amir Kamran, Faheem Kirefu, Philipp Koehn, Sergio Ortiz Rojas, Leopoldo Pla Sempere, Gema Ramı́rez-Sánchez, Elsa Sarrı́as, Marek Strelec, Brian Thompson, William Waites, Dion Wiggins, and… — [ParaCrawl: Web-Scale Acquisition of Parallel Corpora](https://md.research.elamur.ai/free/en/papers/banon-2020-paracrawl-web-scale-acquisition-parallel/)
4. Peter L Bartlett and Marten H Wegkamp. Classification with a reject option using a hinge loss. Journal of Machine Learning Research, 9(8), 2008. [find](https://scholar.google.com/scholar?q=Peter+L+Bartlett+and+Marten+H+Wegkamp%2E+Classification+with+a+reject+option+using+a+hinge+loss%2E+Journal+of+Machine+Learning+Research%2C+9%288%29%2C+2008%2E)
5. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Barry Haddow, Matthias Huck, Chris Hokamp, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Carolina Scarton, Lucia Specia, and Marco Turchi. [Findings of the 2015 workshop on statistical machine translation](https://md.research.elamur.ai/free/en/papers/bojar-2015-findings-2015-workshop-statistical-machine/). In Proceeding…
6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretchen)
7. Curran Associates, Inc., 2020. URL [https://proceedings.neurips.cc/paper/](https://proceedings.neurips.cc/paper/) 2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf . [↗](https://proceedings.neurips.cc/paper/)
8. Saikiran Bulusu, Bhavya Kailkhura, Bo Li, Pramod K Varshney, and Dawn Song. Anomalous example detection in deep learning: A survey. IEEE Access, 8:132330–132347, 2020. [find](https://scholar.google.com/scholar?q=Saikiran+Bulusu%2C+Bhavya+Kailkhura%2C+Bo+Li%2C+Pramod+K+Varshney%2C+and+Dawn+Song%2E+Anomalous+example+detection+in+deep+learning%3A+A+survey%2E+IEEE+Access%2C+8%3A132330%E2%80%93132347%2C+2020%2E)
9. Chi-Keung Chow. An optimum character recognition system using decision functions. IRE Transactions on Electronic Computers, (4):247–254, 1957. [find](https://scholar.google.com/scholar?q=Chi%2DKeung+Chow%2E+An+optimum+character+recognition+system+using+decision+functions%2E+IRE+Transactions+on+Electronic+Computers%2C+%284%29%3A247%E2%80%93254%2C+1957%2E)
10. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… — [PaLM: Scaling Language Modeling with Pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
11. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. [Unsupervised cross-lingual representation learning at scale](https://md.research.elamur.ai/free/en/papers/conneau-2020-unsupervised-cross-lingual-representation-learning/). In Proceedings of the 58th Annual Meeting of the Association for Com…
12. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ondřej Bojar. [Results of the WMT21 metrics shared task: Evaluating metrics with expert-based human evaluations on TED and news domain](https://md.research.elamur.ai/free/en/papers/freitag-2021-results-wmt21-metrics-shared-task/). In Proceedings of the Sixth Conference on Machine Translation,…
13. Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. Advances in neural information processing systems, 30, 2017. [find](https://scholar.google.com/scholar?q=Yonatan+Geifman+and+Ran+El%2DYaniv%2E+Selective+classification+for+deep+neural+networks%2E+Advances+in+neural+information+processing+systems%2C+30%2C+2017%2E)
14. Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. [SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization](https://md.research.elamur.ai/free/en/papers/gliwa-2019-samsum-corpus-human-annotated-dialogue/). In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pp. 70–79, Hong Kong, China, November 2019. Association for Computational Linguistic…
15. Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. [Explaining and harnessing adversarial examples](https://md.research.elamur.ai/free/en/papers/goodfellow-2014-explaining-harnessing-adversarial-examples/). arXiv preprint arXiv:1412.6572, 2014.
16. Ben Goodrich, Vinay Rao, Peter J. Liu, and Mohammad Saleh. Assessing the factual accuracy of generated text. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’19, pp. 166–175, New York, NY, USA, 2019. Association for Computing Machinery. ISBN… — [doi:10.1145/3292500.3330955](https://doi.org/10.1145/3292500.3330955) [find](https://scholar.google.com/scholar?q=Ben+Goodrich%2C+Vinay+Rao%2C+Peter+J%2E+Liu%2C+and+Mohammad+Saleh%2E+Assessing+the+factual+accuracy+of+generated+text%2E+In+Proceedings+of+the+25th+ACM+SIGKDD+International+Conference+on+Knowledge+Discovery+and+D)
17. Max Grusky, Mor Naaman, and Yoav Artzi. [Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies](https://md.research.elamur.ai/free/en/papers/grusky-2018-newsroom-dataset-1-3-million/). Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), 2018. doi:…
18. Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02136, 2016. [find](https://scholar.google.com/scholar?q=Dan+Hendrycks+and+Kevin+Gimpel%2E+A+baseline+for+detecting+misclassified+and+out%2Dof%2Ddistribution+examples+in+neural+networks%2E+arXiv+preprint+arXiv%3A1610%2E02136%2C+2016%2E)
19. Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint arXiv:1812.04606, 2018. [find](https://scholar.google.com/scholar?q=Dan+Hendrycks%2C+Mantas+Mazeika%2C+and+Thomas+Dietterich%2E+Deep+anomaly+detection+with+outlier+exposure%2E+arXiv+preprint+arXiv%3A1812%2E04606%2C+2018%2E)
20. Dan Hendrycks, Steven Basart, Mantas Mazeika, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. Scaling out-of-distribution detection for real-world settings. arXiv preprint arXiv:1911.11132, 2019. [find](https://scholar.google.com/scholar?q=Dan+Hendrycks%2C+Steven+Basart%2C+Mantas+Mazeika%2C+Mohammadreza+Mostajabi%2C+Jacob+Steinhardt%2C+and+Dawn+Song%2E+Scaling+out%2Dof%2Ddistribution+detection+for+real%2Dworld+settings%2E+arXiv+preprint+arXiv%3A1911%2E11132%2C+2)
21. Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. [Pretrained transformers improve out-of-distribution robustness](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-pretrained-transformers-improve-out-distribution/). arXiv preprint arXiv:2004.06100, 2020.
22. Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. [Teaching machines to read and comprehend](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/). In NIPS, pp. 1693– 1701, 2015. URL [http://papers.nips.cc/paper/5945-teaching-machines-to-read-and-comprehend](http://papers.nips.cc/paper/5945-teaching-machines-to-read-and-comprehend) . [↗](http://papers.nips.cc/paper/5945-teaching-machines-to-read-and-comprehend)
23. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. [The curious case of neural text degeneration](https://md.research.elamur.ai/free/en/papers/holtzman-2019-curious-case-neural-text-degeneration/). In International Conference on Learning Representations, 2020. URL [https://openreview.net/forum?id=rygGQyrFvH](https://openreview.net/forum?id=rygGQyrFvH) . [↗](https://openreview.net/forum?id=rygGQyrFvH)
24. Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. [TRUE: Re-evaluating factual consistency evaluation](https://md.research.elamur.ai/free/en/papers/honovich-2022-true-re-evaluating-factual-consistency/). In Proceedings of the 2022 Conference of the North American Chapter of the Association f…
25. Misha Khalman, Yao Zhao, and Mohammad Saleh. Forumsum: A multi-speaker conversation summarization dataset. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 4592–4599, 2021. [find](https://scholar.google.com/scholar?q=Misha+Khalman%2C+Yao+Zhao%2C+and+Mohammad+Saleh%2E+Forumsum%3A+A+multi%2Dspeaker+conversation+summarization+dataset%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2021%2C+pp%2E+4592%E2%80%934599%2C+2021)
26. Byeongchang Kim, Hyunwoo Kim, and Gunhee Kim. [Abstractive summarization of reddit posts with multi-level memory networks](https://md.research.elamur.ai/free/en/papers/kim-2019-abstractive-summarization-reddit-posts-multi/), 2018.
27. Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017. [find](https://scholar.google.com/scholar?q=Balaji+Lakshminarayanan%2C+Alexander+Pritzel%2C+and+Charles+Blundell%2E+Simple+and+scalable+predictive+uncertainty+estimation+using+deep+ensembles%2E+Advances+in+neural+information+processing+systems%2C+30%2C+201)
28. Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. NeurIPS, 2018. [find](https://scholar.google.com/scholar?q=Kimin+Lee%2C+Kibok+Lee%2C+Honglak+Lee%2C+and+Jinwoo+Shin%2E+A+simple+unified+framework+for+detecting+out%2Dof%2Ddistribution+samples+and+adversarial+attacks%2E+NeurIPS%2C+2018%2E)
29. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. [BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). In Proceedings of the 58th Annual Meeting of the Assoc…
30. Shiyu Liang, Yixuan Li, and R Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. arXiv preprint arXiv:1706.02690, 2017. — [arXiv:1706.02690](https://arxiv.org/abs/1706.02690) [find](https://scholar.google.com/scholar?q=Shiyu+Liang%2C+Yixuan+Li%2C+and+R+Srikant%2E+Enhancing+the+reliability+of+out%2Dof%2Ddistribution+image+detection+in+neural+networks%2E+arXiv+preprint+arXiv%3A1706%2E02690%2C+2017%2E+%E2%80%94+%5BarXiv%3A1706%2E02690%5D%28https%3A%2F%2Farxiv%2Eor)
31. Zi Lin, Jeremiah Zhe Liu, and Jingbo Shang. Towards collaborative neural-symbolic graph semantic parsing via uncertainty. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 4160–4173, 2022. [find](https://scholar.google.com/scholar?q=Zi+Lin%2C+Jeremiah+Zhe+Liu%2C+and+Jingbo+Shang%2E+Towards+collaborative+neural%2Dsymbolic+graph+semantic+parsing+via+uncertainty%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL+2022%2C+pp%2E+41)
32. Jeremiah Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax Weiss, and Balaji Lakshminarayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. Advances in Neural Information Processing Systems, 33:7498–7512, 2020a. [find](https://scholar.google.com/scholar?q=Jeremiah+Liu%2C+Zi+Lin%2C+Shreyas+Padhy%2C+Dustin+Tran%2C+Tania+Bedrax+Weiss%2C+and+Balaji+Lakshminarayanan%2E+Simple+and+principled+uncertainty+estimation+with+deterministic+deep+learning+via+distance+awareness%2E)
33. Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. [Generating wikipedia by summarizing long sequences](https://md.research.elamur.ai/free/en/papers/liu-2018-generating-wikipedia-summarizing-long-sequences/). In International Conference on Learning Representations, 2018. URL [https://openreview.net/forum?id=Hyg0vbWC-](https://openreview.net/forum?id=Hyg0vbWC-) . [↗](https://openreview.net/forum?id=Hyg0vbWC-)
34. Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. Advances in Neural Information Processing Systems, 33:21464–21475, 2020b. [find](https://scholar.google.com/scholar?q=Weitang+Liu%2C+Xiaoyun+Wang%2C+John+Owens%2C+and+Yixuan+Li%2E+Energy%2Dbased+out%2Dof%2Ddistribution+detection%2E+Advances+in+Neural+Information+Processing+Systems%2C+33%3A21464%E2%80%9321475%2C+2020b%2E)
35. Denis Lukovnikov, Sina Daubener, and Asja Fischer. Detecting compositionally out-of-distribution examples in semantic parsing. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 591–598, 2021. [find](https://scholar.google.com/scholar?q=Denis+Lukovnikov%2C+Sina+Daubener%2C+and+Asja+Fischer%2E+Detecting+compositionally+out%2Dof%2Ddistribution+examples+in+semantic+parsing%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2021%2C)
36. Andrey Malinin and Mark Gales. Uncertainty estimation in autoregressive structured prediction. arXiv preprint arXiv:2002.07650, 2020. [find](https://scholar.google.com/scholar?q=Andrey+Malinin+and+Mark+Gales%2E+Uncertainty+estimation+in+autoregressive+structured+prediction%2E+arXiv+preprint+arXiv%3A2002%2E07650%2C+2020%2E)
37. Andrey Malinin, Neil Band, German Chesnokov, Yarin Gal, Mark JF Gales, Alexey Noskov, Andrey Ploskonosov, Liudmila Prokhorenkova, Ivan Provilkov, Vatsal Raina, et al. Shifts: A dataset of real distributional shift across multiple large-scale tasks. arXiv preprint arXiv:2107.07455, 2021. — [arXiv:2107.07455](https://arxiv.org/abs/2107.07455) [find](https://scholar.google.com/scholar?q=Andrey+Malinin%2C+Neil+Band%2C+German+Chesnokov%2C+Yarin+Gal%2C+Mark+JF+Gales%2C+Alexey+Noskov%2C+Andrey+Ploskonosov%2C+Liudmila+Prokhorenkova%2C+Ivan+Provilkov%2C+Vatsal+Raina%2C+et+al%2E+Shifts%3A+A+dataset+of+real+distrib)
38. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. [On faithfulness and factuality in abstractive summarization](https://md.research.elamur.ai/free/en/papers/maynez-2020-faithfulness-factuality-abstractive-summarization/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1906–1919, Online, July 2020. Association for Computational Linguistics. doi: 1…
39. Paul Michel and Graham Neubig. [MTNT: A testbed for machine translation of noisy text](https://md.research.elamur.ai/free/en/papers/michel-2018-mtnt-testbed-machine-translation-noisy/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 543–553, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1050…
40. Jishnu Mukhoti, Andreas Kirsch, Joost van Amersfoort, Philip HS Torr, and Yarin Gal. Deep deterministic uncertainty: A simple baseline. arXiv e-prints, pp. arXiv–2102, 2021. [find](https://scholar.google.com/scholar?q=Jishnu+Mukhoti%2C+Andreas+Kirsch%2C+Joost+van+Amersfoort%2C+Philip+HS+Torr%2C+and+Yarin+Gal%2E+Deep+deterministic+uncertainty%3A+A+simple+baseline%2E+arXiv+e%2Dprints%2C+pp%2E+arXiv%E2%80%932102%2C+2021%2E)
41. Kevin P. Murphy. Probabilistic Machine Learning: Advanced Topics. MIT Press, 2023. URL probml.ai . [find](https://scholar.google.com/scholar?q=Kevin+P%2E+Murphy%2E+Probabilistic+Machine+Learning%3A+Advanced+Topics%2E+MIT+Press%2C+2023%2E+URL+probml%2Eai+%2E)
42. Vaishnavh Nagarajan, Anders Andreassen, and Behnam Neyshabur. Understanding the failure modes of out-of-distribution generalization. arXiv preprint arXiv:2010.15775, 2020. — [arXiv:2010.15775](https://arxiv.org/abs/2010.15775) [find](https://scholar.google.com/scholar?q=Vaishnavh+Nagarajan%2C+Anders+Andreassen%2C+and+Behnam+Neyshabur%2E+Understanding+the+failure+modes+of+out%2Dof%2Ddistribution+generalization%2E+arXiv+preprint+arXiv%3A2010%2E15775%2C+2020%2E+%E2%80%94+%5BarXiv%3A2010%2E15775%5D%28https%3A%2F)
43. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 [Conference on Empirical Methods in Natural Language Processing, pp](https://md.research.elamur.ai/free/en/papers/narayan-2018-don-t-give-me-details/). 1797–1807, Brussels, Belgium, October-No…
44. A Nguyen, J Yosinski, and J Clune. Deep neural networks are easily fooled: high confidence predictions for unrecognizable images. arxiv. arXiv preprint arXiv:1412.1897, 2014. [find](https://scholar.google.com/scholar?q=A+Nguyen%2C+J+Yosinski%2C+and+J+Clune%2E+Deep+neural+networks+are+easily+fooled%3A+high+confidence+predictions+for+unrecognizable+images%2E+arxiv%2E+arXiv+preprint+arXiv%3A1412%2E1897%2C+2014%2E)
45. Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems, 32, 2019. [find](https://scholar.google.com/scholar?q=Yaniv+Ovadia%2C+Emily+Fertig%2C+Jie+Ren%2C+Zachary+Nado%2C+David+Sculley%2C+Sebastian+Nowozin%2C+Joshua+Dillon%2C+Balaji+Lakshminarayanan%2C+and+Jasper+Snoek%2E+Can+you+trust+your+model%E2%80%99s+uncertainty%3F+evaluating+predic)
46. Amy Pu, Hyung Won Chung, Ankur Parikh, Sebastian Gehrmann, and Thibault Sellam. [Learning compact metrics for MT](https://md.research.elamur.ai/free/en/papers/pu-2021-learning-compact-metrics-mt/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 751–762, Online and Punta Cana, Dominican Republic, November 2021. Association for Computati…
47. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+Language+models+are+unsupervised+multitask+learners%2E+2019%2E)
48. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of Machine Learning Research, 21(140):1–67, 2020. URL [http://jmlr.org/papers/v21/2…](http://jmlr.org/papers/v21/2…) [↗](http://jmlr.org/papers/v21/2…)
49. Mrinal Rawat, Ramya Hebbalaguppe, and Lovekesh Vig. Pnpood: Out-of-distribution detection for text classification via plug andplay data augmentation. arXiv preprint arXiv:2111.00506, 2021. — [arXiv:2111.00506](https://arxiv.org/abs/2111.00506) [find](https://scholar.google.com/scholar?q=Mrinal+Rawat%2C+Ramya+Hebbalaguppe%2C+and+Lovekesh+Vig%2E+Pnpood%3A+Out%2Dof%2Ddistribution+detection+for+text+classification+via+plug+andplay+data+augmentation%2E+arXiv+preprint+arXiv%3A2111%2E00506%2C+2021%2E+%E2%80%94+%5BarXiv%3A21)
50. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. [Comet: A neural framework for mt evaluation](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/). arXiv preprint arXiv:2009.09025, 2020.
51. Jie Ren, Peter J Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark A DePristo, Joshua V Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distribution detection. NeurIPS, 2019. [find](https://scholar.google.com/scholar?q=Jie+Ren%2C+Peter+J+Liu%2C+Emily+Fertig%2C+Jasper+Snoek%2C+Ryan+Poplin%2C+Mark+A+DePristo%2C+Joshua+V+Dillon%2C+and+Balaji+Lakshminarayanan%2E+Likelihood+ratios+for+out%2Dof%2Ddistribution+detection%2E+NeurIPS%2C+2019%2E)
52. Jie Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padhy, and Balaji Lakshminarayanan. A simple fix to mahalanobis distance for improving near-ood detection. arXiv preprint arXiv:2106.09022, 2021. — [arXiv:2106.09022](https://arxiv.org/abs/2106.09022) [find](https://scholar.google.com/scholar?q=Jie+Ren%2C+Stanislav+Fort%2C+Jeremiah+Liu%2C+Abhijit+Guha+Roy%2C+Shreyas+Padhy%2C+and+Balaji+Lakshminarayanan%2E+A+simple+fix+to+mahalanobis+distance+for+improving+near%2Dood+detection%2E+arXiv+preprint+arXiv%3A2106%2E09)
53. Lukas Ruff, Jacob R Kauffmann, Robert A Vandermeulen, Grégoire Montavon, Wojciech Samek, Marius Kloft, Thomas G Dietterich, and Klaus-Robert Müller. A unifying review of deep and shallow anomaly detection. Proceedings of the IEEE, 109(5):756–795, 2021. [find](https://scholar.google.com/scholar?q=Lukas+Ruff%2C+Jacob+R+Kauffmann%2C+Robert+A+Vandermeulen%2C+Gre%CC%81goire+Montavon%2C+Wojciech+Samek%2C+Marius+Kloft%2C+Thomas+G+Dietterich%2C+and+Klaus%2DRobert+Mu%CC%88ller%2E+A+unifying+review+of+deep+and+shallow+anomaly+det)
54. Mohammadreza Salehi, Hossein Mirzaei, Dan Hendrycks, Yixuan Li, Mohammad Hossein Rohban, and Mohammad Sabokrou. A unified survey on anomaly, novelty, open-set, and out-of-distribution detection: Solutions and future challenges. arXiv preprint arXiv:2110.14051, 2021. — [arXiv:2110.14051](https://arxiv.org/abs/2110.14051) [find](https://scholar.google.com/scholar?q=Mohammadreza+Salehi%2C+Hossein+Mirzaei%2C+Dan+Hendrycks%2C+Yixuan+Li%2C+Mohammad+Hossein+Rohban%2C+and+Mohammad+Sabokrou%2E+A+unified+survey+on+anomaly%2C+novelty%2C+open%2Dset%2C+and+out%2Dof%2Ddistribution+detection%3A+Solut)
55. Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1073–1083, Vancouver, Canada, July 2017. Association for Com… [find](https://scholar.google.com/scholar?q=Abigail+See%2C+Peter+J%2E+Liu%2C+and+Christopher+D%2E+Manning%2E+Get+to+the+point%3A+Summarization+with+pointer%2Dgenerator+networks%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computational+L)
56. Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pp. 4596–4604. PMLR, 2018. [find](https://scholar.google.com/scholar?q=Noam+Shazeer+and+Mitchell+Stern%2E+Adafactor%3A+Adaptive+learning+rates+with+sublinear+memory+cost%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+4596%E2%80%934604%2E+PMLR%2C+2018%2E)
57. Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. arXiv preprint arXiv:2204.06507, 2022. — [arXiv:2204.06507](https://arxiv.org/abs/2204.06507) [find](https://scholar.google.com/scholar?q=Yiyou+Sun%2C+Yifei+Ming%2C+Xiaojin+Zhu%2C+and+Yixuan+Li%2E+Out%2Dof%2Ddistribution+detection+with+deep+nearest+neighbors%2E+arXiv+preprint+arXiv%3A2204%2E06507%2C+2022%2E+%E2%80%94+%5BarXiv%3A2204%2E06507%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2204%2E065)
58. Brian Thompson and Matt Post. [Automatic machine translation evaluation in many languages via zero-shot paraphrasing](https://md.research.elamur.ai/free/en/papers/thompson-2020-automatic-machine-translation-evaluation-many/). arXiv preprint arXiv:2004.14564, 2020.
59. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. [Lamda: Language models for dialog applications](https://md.research.elamur.ai/free/en/papers/thoppilan-2022-lamda-language-models-dialog-applications/). arXiv preprint arXiv:2201.08239, 2022.
60. Jörg Tiedemann. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pp. 2214–2218, Istanbul, Turkey, May 2012. European Language Resources Association (ELRA). URL [http://www.lrec-conf.org/proceedings/lre…](http://www.lrec-conf.org/proceedings/lre…) [↗](http://www.lrec-conf.org/proceedings/lre…)
61. Dustin Tran, Jeremiah Liu, Michael W Dusenberry, Du Phan, Mark Collier, Jie Ren, Kehang Han, Zi Wang, Zelda Mariet, Huiyi Hu, et al. Plex: Towards reliability using pretrained large model extensions. arXiv preprint arXiv:2207.07411, 2022. — [arXiv:2207.07411](https://arxiv.org/abs/2207.07411) [find](https://scholar.google.com/scholar?q=Dustin+Tran%2C+Jeremiah+Liu%2C+Michael+W+Dusenberry%2C+Du+Phan%2C+Mark+Collier%2C+Jie+Ren%2C+Kehang+Han%2C+Zi+Wang%2C+Zelda+Mariet%2C+Huiyi+Hu%2C+et+al%2E+Plex%3A+Towards+reliability+using+pretrained+large+model+extensions%2E+)
62. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30, 2017.
63. Jim Winkens, Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, Joseph R Ledsam, Patricia MacWilliams, Pushmeet Kohli, Alan Karthikesalingam, Simon Kohl, et al. Contrastive training for improved out-of-distribution detection. arXiv preprint arXiv:2007.05566, 2020. — [arXiv:2007.05566](https://arxiv.org/abs/2007.05566) [find](https://scholar.google.com/scholar?q=Jim+Winkens%2C+Rudy+Bunel%2C+Abhijit+Guha+Roy%2C+Robert+Stanforth%2C+Vivek+Natarajan%2C+Joseph+R+Ledsam%2C+Patricia+MacWilliams%2C+Pushmeet+Kohli%2C+Alan+Karthikesalingam%2C+Simon+Kohl%2C+et+al%2E+Contrastive+training+for+)
64. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… [find](https://scholar.google.com/scholar?q=Yonghui+Wu%2C+Mike+Schuster%2C+Zhifeng+Chen%2C+Quoc+V%2E+Le%2C+Mohammad+Norouzi%2C+Wolfgang+Macherey%2C+Maxim+Krikun%2C+Yuan+Cao%2C+Qin+Gao%2C+Klaus+Macherey%2C+Jeff+Klingner%2C+Apurva+Shah%2C+Melvin+Johnson%2C+Xiaobing+Liu%2C+Luk)
65. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016b. [find](https://scholar.google.com/scholar?q=Yonghui+Wu%2C+Mike+Schuster%2C+Zhifeng+Chen%2C+Quoc+V+Le%2C+Mohammad+Norouzi%2C+Wolfgang+Macherey%2C+Maxim+Krikun%2C+Yuan+Cao%2C+Qin+Gao%2C+Klaus+Macherey%2C+et+al%2E+Google%E2%80%99s+neural+machine+translation+system%3A+Bridging+th)
66. Tim Z Xiao, Aidan N Gomez, and Yarin Gal. Wat zei je? detecting out-of-distribution translations with variational transformers. arXiv preprint arXiv:2006.08344, 2020. — [arXiv:2006.08344](https://arxiv.org/abs/2006.08344) [find](https://scholar.google.com/scholar?q=Tim+Z+Xiao%2C+Aidan+N+Gomez%2C+and+Yarin+Gal%2E+Wat+zei+je%3F+detecting+out%2Dof%2Ddistribution+translations+with+variational+transformers%2E+arXiv+preprint+arXiv%3A2006%2E08344%2C+2020%2E+%E2%80%94+%5BarXiv%3A2006%2E08344%5D%28https%3A%2F%2Farxi)
67. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. [Pegasus: Pre-training with extracted gap-sentences for abstractive summarization](https://md.research.elamur.ai/free/en/papers/zhang-2019-pegasus-pre-training-extracted-gap/). In Proceedings of the 37th International Conference on Machine Learning, ICML’20. JMLR.org, 2020.
68. Wenxuan Zhou, Fangyu Liu, and Muhao Chen. Contrastive out-of-distribution detection for pre-trained transformers. arXiv preprint arXiv:2104.08812, 2021. — [arXiv:2104.08812](https://arxiv.org/abs/2104.08812) [find](https://scholar.google.com/scholar?q=Wenxuan+Zhou%2C+Fangyu+Liu%2C+and+Muhao+Chen%2E+Contrastive+out%2Dof%2Ddistribution+detection+for+pre%2Dtrained+transformers%2E+arXiv+preprint+arXiv%3A2104%2E08812%2C+2021%2E+%E2%80%94+%5BarXiv%3A2104%2E08812%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2104)

*Updated 2 November 2026*