---
title: "Optimizing the Factual Correctness of a Summary — Zhang 2020"
description: "Abstractive summarizers trained to match human references can score well on overlap metrics like ROUGE and still state wrong facts."
authors: ["Yuhao Zhang", "Derek Merck", "Emily Tsai", "Christopher D. Manning", "Curtis Langlotz"]
year: 2020
venue: "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/2020.acl-main.458/"
doi: "10.18653/v1/2020.acl-main.458"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2020-optimizing-factual-correctness-summary-study/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2020-optimizing-factual-correctness-summary-study/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/) / Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports

# Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports

**Authors:** [Yuhao Zhang](https://md.research.elamur.ai/free/en/authors/yuhao-zhang/), Derek Merck, Emily Tsai, [Christopher D. Manning](https://md.research.elamur.ai/free/en/authors/christopher-d-manning/), Curtis Langlotz · **Year:** 2020 · **Venue:** Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.acl-main.458/) · [PDF](https://aclanthology.org/2020.acl-main.458.pdf) · [DOI 10.18653/v1/2020.acl-main.458](https://doi.org/10.18653/v1/2020.acl-main.458) · [OpenAlex](https://openalex.org/W3034863243)

**Citations:** 160 citations (openalex-snapshot · 2026-10-14) · [80](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Abstractive summarizers trained to match human references can score well on overlap metrics like ROUGE and still state wrong facts. For example, a summary may reach a high ROUGE-L while reversing a finding such as the presence of pneumothorax. Zhang et al. address this for radiology, where the findings section of a report is condensed into an impression and errors can cause medical harm.

They build an automatic fact-checking setup. An information extraction module pulls clinical observations from both the generated and the reference summary, and comparing the two yields a factual accuracy score. This score serves as a reward in reinforcement learning, alongside a ROUGE-based reward and a language model objective, to train a background-augmented pointer-generator.

On two hospital datasets (Stanford and RIH), automatic metrics and blinded radiologist judgments both show clear gains in factual correctness and overall quality over the baseline, approaching human-written summaries. Optimizing ROUGE alone via RL already improves correctness compared with maximum likelihood training. The authors describe this as the first work to directly optimize a summarizer for factual correctness.

## Key points
- Backbone model: the background-augmented pointer-generator (Bi-LSTM encoder, LSTM decoder, attention, copy mechanism, separately encoded background section). The best variant, RL R+C, combines ROUGE and factual correctness rewards.
- Motivation: earlier work found that about 30% of neural radiology summaries contained factual errors.
- Human evaluation: two board-certified radiologists ranked 50 Stanford test examples on fluency, factual correctness/completeness, and overall quality. RL R+C beat the baseline on correctness and overall quality but was slightly less preferred on fluency, where 60% of comparisons were ties.
- Style analysis: a Mixture of Softmaxes LM was trained on about 222,000 human summaries. RL R+C's perplexity was much closer to human references, while the baseline overused generic text: the sentence "no significant interval change" appeared in 34% of baseline outputs on the Stanford dev set, versus 24% for RL R+C and 17% for humans.
- Limitations: the method depends on a rule-based IE module that suits only a limited fact space, so open-domain generalization remains unsolved. It also targets mainly binary errors, such as whether a disease is present.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

80 papers of our library cite it (of the 140884 whose bibliographies we read); 8 of them have a page here, 72 not yet.

- [Factual Error Correction for Abstractive Summarization Models](https://md.research.elamur.ai/free/en/papers/cao-2020-factual-error-correction-abstractive-summarization/)
- [CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization](https://md.research.elamur.ai/free/en/papers/cao-2021-cliff-contrastive-learning-improving-faithfulness/)
- [Survey of Hallucination in Natural Language Generation](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/)
- [Hurdles to Progress in Long-form Question Answering](https://md.research.elamur.ai/free/en/papers/krishna-2021-hurdles-progress-long-form-question/)
- [A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation](https://md.research.elamur.ai/free/en/papers/liu-2022-token-level-reference-free-hallucination/)
- [Evaluating the Retrieval Component in a Retrieval-Augmented Summarization System for Patient Records in French](https://md.research.elamur.ai/free/en/papers/naguib-2026-evaluating-retrieval-component-retrieval-augmented/)
- [Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors](https://md.research.elamur.ai/free/en/papers/tang-2023-understanding-factual-errors-summarization-errors/)
- [Fine-tuning Language Models for Factuality](https://md.research.elamur.ai/free/en/papers/tian-2023-fine-tuning-language-models-factuality/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhang2020optimizing,
  title = {{Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports}},
  author = {Zhang, Yuhao and Merck, Derek and Tsai, Emily and Manning, Christopher D. and Langlotz, Curtis},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.458},
  url = {https://doi.org/10.18653/v1/2020.acl-main.458}
}
```

- [← previous On the syntax of object pronouns in Old English and Early Middle English](https://md.research.elamur.ai/free/en/papers/bastiani-2020-syntax-object-pronouns-old-english/)
- [next → Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://md.research.elamur.ai/free/en/papers/li-2020-oscar-object-semantics-aligned-pre/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/)

## References

43 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. [find](https://scholar.google.com/scholar?q=Dzmitry+Bahdanau%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2015%2E+Neural+machine+translation+by+jointly+learning+to+align+and+translate%2E) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
2. The 2015 International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=The+2015+International+Conference+on+Learning+Representations%2E)
3. Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2017. Faithful to the original: Fact aware neural abstractive summarization. The Thirty-First AAAI Conference on Artificial Intelligence (AAAI-2017). [find](https://scholar.google.com/scholar?q=Ziqiang+Cao%2C+Furu+Wei%2C+Wenjie+Li%2C+and+Sujian+Li%2E+2017%2E+Faithful+to+the+original%3A+Fact+aware+neural+abstractive+summarization%2E+The+Thirty%2DFirst+AAAI+Conference+on+Artificial+Intelligence+%28AAAI%2D2017%29%2E)
4. Arun Chaganty, Stephen Mussmann, and Percy Liang. 2018. The price of debiasing automatic metrics in natural language evalaution. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL 2018). [find](https://scholar.google.com/scholar?q=Arun+Chaganty%2C+Stephen+Mussmann%2C+and+Percy+Liang%2E+2018%2E+The+price+of+debiasing+automatic+metrics+in+natural+language+evalaution%2E+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computati)
5. Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL 2018). [find](https://scholar.google.com/scholar?q=Yen%2DChun+Chen+and+Mohit+Bansal%2E+2018%2E+Fast+abstractive+summarization+with+reinforce%2Dselected+sentence+rewriting%2E+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguistics) [our card](https://md.research.elamur.ai/free/en/papers/chen-2018-fast-abstractive-summarization-reinforce-selected/)
6. Jianpeng Cheng and Mirella Lapata. 2016. Neural summarization by extracting sentences and words. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016). [find](https://scholar.google.com/scholar?q=Jianpeng+Cheng+and+Mirella+Lapata%2E+2016%2E+Neural+summarization+by+extracting+sentences+and+words%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28ACL+2016%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/cheng-2016-neural-summarization-extracting-sentences-words/)
7. Wojciech Kryściński, Bryan McCann, Caiming Xiong, and Richard Socher. 2019b. Evaluating the factual consistency of abstractive text summarization. arXiv preprint arXiv:1910.12840. [find](https://scholar.google.com/scholar?q=Wojciech+Krys%CC%81cin%CC%81ski%2C+Bryan+McCann%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+2019b%2E+Evaluating+the+factual+consistency+of+abstractive+text+summarization%2E+arXiv+preprint+arXiv%3A1910%2E12840%2E)
8. Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming-Wei Chang, Dipanjan Das, and William Cohen. 2019. Handling divergent reference texts when evaluating table-to-text generation. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL 2019). [find](https://scholar.google.com/scholar?q=Bhuwan+Dhingra%2C+Manaal+Faruqui%2C+Ankur+Parikh%2C+Ming%2DWei+Chang%2C+Dipanjan+Das%2C+and+William+Cohen%2E+2019%2E+Handling+divergent+reference+texts+when+evaluating+table%2Dto%2Dtext+generation%2E+Proceedings+of+the+57t) [our card](https://md.research.elamur.ai/free/en/papers/dhingra-2019-handling-divergent-reference-texts-when/)
9. Christy Y. Li, Xiaodan Liang, Zhiting Hu, and Eric P. Xing. 2019. Hybrid retrieval-generation reinforced agent for medical image report generation. Advances in neural information processing systems. [find](https://scholar.google.com/scholar?q=Christy+Y%2E+Li%2C+Xiaodan+Liang%2C+Zhiting+Hu%2C+and+Eric+P%2E+Xing%2E+2019%2E+Hybrid+retrieval%2Dgeneration+reinforced+agent+for+medical+image+report+generation%2E+Advances+in+neural+information+processing+systems%2E)
10. Yue Dong, Yikang Shen, Eric Crawford, Herke van Hoof, and Jackie Chi Kit Cheung. 2018. Bandit-Sum: Extractive summarization as a contextual bandit. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP 2018). [find](https://scholar.google.com/scholar?q=Yue+Dong%2C+Yikang+Shen%2C+Eric+Crawford%2C+Herke+van+Hoof%2C+and+Jackie+Chi+Kit+Cheung%2E+2018%2E+Bandit%2DSum%3A+Extractive+summarization+as+a+contextual+bandit%2E+Proceedings+of+the+2018+Conference+on+Empirical+Meth)
11. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. Text Summarization Branches Out: ACL Workshop. [find](https://scholar.google.com/scholar?q=Chin%2DYew+Lin%2E+2004%2E+ROUGE%3A+A+package+for+automatic+evaluation+of+summaries%2E+Text+Summarization+Branches+Out%3A+ACL+Workshop%2E)
12. Guanxiong Liu, Tzu-Ming Harry Hsu, Matthew Mc-Dermott, Willie Boag, Wei-Hung Weng, Peter Szolovits, and Marzyeh Ghassemi. 2019. Clinically accurate chest X-ray report generation. arXiv preprint arXiv:1904.02633. — [arXiv:1904.02633](https://arxiv.org/abs/1904.02633) [find](https://scholar.google.com/scholar?q=Guanxiong+Liu%2C+Tzu%2DMing+Harry+Hsu%2C+Matthew+Mc%2DDermott%2C+Willie+Boag%2C+Wei%2DHung+Weng%2C+Peter+Szolovits%2C+and+Marzyeh+Ghassemi%2E+2019%2E+Clinically+accurate+chest+X%2Dray+report+generation%2E+arXiv+preprint+arXiv%3A)
13. Günes Erkan and Dragomir R Radev. 2004. LexRank: Graph-based lexical centrality as salience in text summarization. Journal of Artificial Intelligence Research, 22:457–479. [find](https://scholar.google.com/scholar?q=Gu%CC%88nes+Erkan+and+Dragomir+R+Radev%2E+2004%2E+LexRank%3A+Graph%2Dbased+lexical+centrality+as+salience+in+text+summarization%2E+Journal+of+Artificial+Intelligence+Research%2C+22%3A457%E2%80%93479%2E)
14. Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating wikipedia by summarizing long sequences. [find](https://scholar.google.com/scholar?q=Peter+J%2E+Liu%2C+Mohammad+Saleh%2C+Etienne+Pot%2C+Ben+Goodrich%2C+Ryan+Sepassi%2C+Lukasz+Kaiser%2C+and+Noam+Shazeer%2E+2018%2E+Generating+wikipedia+by+summarizing+long+sequences%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2018-generating-wikipedia-summarizing-long-sequences/)
15. The 2018 International Conference for Learning Representations. [find](https://scholar.google.com/scholar?q=The+2018+International+Conference+for+Learning+Representations%2E)
16. Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Lingui… [find](https://scholar.google.com/scholar?q=Tobias+Falke%2C+Leonardo+F%2E+R%2E+Ribeiro%2C+Prasetya+Ajie+Utama%2C+Ido+Dagan%2C+and+Iryna+Gurevych%2E+2019%2E+Ranking+generated+summaries+by+correctness%3A+An+interesting+but+challenging+application+for+natural+langu) [our card](https://md.research.elamur.ai/free/en/papers/falke-2019-ranking-generated-summaries-correctness-interesting/)
17. Sean MacAvaney, Sajad Sotudeh, Arman Cohan, Nazli Goharian, Ish Talati, and Ross W. Filice. 2019. Ontology-aware clinical abstractive summarization. Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 19). [find](https://scholar.google.com/scholar?q=Sean+MacAvaney%2C+Sajad+Sotudeh%2C+Arman+Cohan%2C+Nazli+Goharian%2C+Ish+Talati%2C+and+Ross+W%2E+Filice%2E+2019%2E+Ontology%2Daware+clinical+abstractive+summarization%2E+Proceedings+of+the+42nd+International+ACM+SIGIR+Con)
18. Ben Goodrich, Vinay Rao, Peter J. Liu, and Mohammad Saleh. 2019. Assessing the factual accuracy of generated text. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 19). [find](https://scholar.google.com/scholar?q=Ben+Goodrich%2C+Vinay+Rao%2C+Peter+J%2E+Liu%2C+and+Mohammad+Saleh%2E+2019%2E+Assessing+the+factual+accuracy+of+generated+text%2E+Proceedings+of+the+25th+ACM+SIGKDD+International+Conference+on+Knowledge+Discovery+an)
19. Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. 2019. CheXpert: A large chest radiograph dataset with uncertainty labels and expert comparison. The Thirty-Third AAAI Conference on Artifici… [find](https://scholar.google.com/scholar?q=Jeremy+Irvin%2C+Pranav+Rajpurkar%2C+Michael+Ko%2C+Yifan+Yu%2C+Silviana+Ciurea%2DIlcus%2C+Chris+Chute%2C+Henrik+Marklund%2C+Behzad+Haghgoo%2C+Robyn+Ball%2C+Katie+Shpanskaya%2C+et+al%2E+2019%2E+CheXpert%3A+A+large+chest+radiograph)
20. Charles E Kahn Jr, Curtis P Langlotz, Elizabeth S Burnside, John A Carrino, David S Channin, David M Hovsepian, and Daniel L Rubin. 2009. Toward best practices in radiology reporting. Radiology, 252(3):852–856. [find](https://scholar.google.com/scholar?q=Charles+E+Kahn+Jr%2C+Curtis+P+Langlotz%2C+Elizabeth+S+Burnside%2C+John+A+Carrino%2C+David+S+Channin%2C+David+M+Hovsepian%2C+and+Daniel+L+Rubin%2E+2009%2E+Toward+best+practices+in+radiology+reporting%2E+Radiology%2C+252%283)
21. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In The 2015 International Conference for Learning Representations. [find](https://scholar.google.com/scholar?q=Diederik+P+Kingma+and+Jimmy+Ba%2E+2015%2E+Adam%3A+A+method+for+stochastic+optimization%2E+In+The+2015+International+Conference+for+Learning+Representations%2E)
22. Wojciech Kryściński, Nitish Shirish Keskar, Bryan Mc-Cann, Caiming Xiong, and Richard Socher. 2019a. Neural text summarization: A critical evaluation. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP 2019). [find](https://scholar.google.com/scholar?q=Wojciech+Krys%CC%81cin%CC%81ski%2C+Nitish+Shirish+Keskar%2C+Bryan+Mc%2DCann%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+2019a%2E+Neural+text+summarization%3A+A+critical+evaluation%2E+Proceedings+of+the+2019+Conference+on+Empirical) [our card](https://md.research.elamur.ai/free/en/papers/kryscinski-2019-neural-text-summarization-critical-evaluation/)
23. Christopher D Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Introduction to information retrieval. Cambridge University Press. [find](https://scholar.google.com/scholar?q=Christopher+D+Manning%2C+Prabhakar+Raghavan%2C+and+Hinrich+Schu%CC%88tze%2E+2008%2E+Introduction+to+information+retrieval%2E+Cambridge+University+Press%2E)
24. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Mc-Closky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations. [find](https://scholar.google.com/scholar?q=Christopher+D%2E+Manning%2C+Mihai+Surdeanu%2C+John+Bauer%2C+Jenny+Finkel%2C+Steven+J%2E+Bethard%2C+and+David+Mc%2DClosky%2E+2014%2E+The+Stanford+CoreNLP+natural+language+processing+toolkit%2E+In+Association+for+Computation) [our card](https://md.research.elamur.ai/free/en/papers/manning-2014-stanford-corenlp-natural-language-processing/)
25. Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2016a. SummaRuNNer: A recurrent neural network based sequence model for extractive summarization of documents. The Thirty-First AAAI Conference on Artificial Intelligence (AAAI-2017). [find](https://scholar.google.com/scholar?q=Ramesh+Nallapati%2C+Feifei+Zhai%2C+and+Bowen+Zhou%2E+2016a%2E+SummaRuNNer%3A+A+recurrent+neural+network+based+sequence+model+for+extractive+summarization+of+documents%2E+The+Thirty%2DFirst+AAAI+Conference+on+Artifi) [our card](https://md.research.elamur.ai/free/en/papers/nallapati-2016-summarunner-recurrent-neural-network-based/)
26. Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. 2016b. Abstractive text summarization using sequence-to-sequence RNNs and beyond. Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning. [find](https://scholar.google.com/scholar?q=Ramesh+Nallapati%2C+Bowen+Zhou%2C+Cicero+dos+Santos%2C+Caglar+Gulcehre%2C+and+Bing+Xiang%2E+2016b%2E+Abstractive+text+summarization+using+sequence%2Dto%2Dsequence+RNNs+and+beyond%2E+Proceedings+of+The+20th+SIGNLL+Confe)
27. Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Romain+Paulus%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+2018%2E+A+deep+reinforced+model+for+abstractive+summarization%2E+In+International+Conference+on+Learning+Representations%2E) [our card](https://md.research.elamur.ai/free/en/papers/paulus-2017-deep-reinforced-model-abstractive-summarization/)
28. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Jeffrey+Pennington%2C+Richard+Socher%2C+and+Christopher+D+Manning%2E+2014%2E+GloVe%3A+Global+vectors+for+word+representation%2E+In+Proceedings+of+the+2014+Conference+on+Empirical+Methods+in+Natural+Language+Proce)
29. Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. Sequence level training with recurrent neural networks. [find](https://scholar.google.com/scholar?q=Marc%E2%80%99Aurelio+Ranzato%2C+Sumit+Chopra%2C+Michael+Auli%2C+and+Wojciech+Zaremba%2E+2016%2E+Sequence+level+training+with+recurrent+neural+networks%2E) [our card](https://md.research.elamur.ai/free/en/papers/ranzato-2015-sequence-level-training-recurrent-neural/)
30. The 2016 International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=The+2016+International+Conference+on+Learning+Representations%2E)
31. Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. 2017. Self-critical sequence training for image captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7008–7024. [find](https://scholar.google.com/scholar?q=Steven+J+Rennie%2C+Etienne+Marcheret%2C+Youssef+Mroueh%2C+Jerret+Ross%2C+and+Vaibhava+Goel%2E+2017%2E+Self%2Dcritical+sequence+training+for+image+captioning%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision) [our card](https://md.research.elamur.ai/free/en/papers/rennie-2016-self-critical-sequence-training-image/)
32. Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. In The 2017 Annual Meeting of the Association of Computational Linguistics (ACL 2017). [find](https://scholar.google.com/scholar?q=Abigail+See%2C+Peter+J+Liu%2C+and+Christopher+D+Manning%2E+2017%2E+Get+to+the+point%3A+Summarization+with+pointer%2Dgenerator+networks%2E+In+The+2017+Annual+Meeting+of+the+Association+of+Computational+Linguistics+%28)
33. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958. [find](https://scholar.google.com/scholar?q=Nitish+Srivastava%2C+Geoffrey+Hinton%2C+Alex+Krizhevsky%2C+Ilya+Sutskever%2C+and+Ruslan+Salakhutdinov%2E+2014%2E+Dropout%3A+a+simple+way+to+prevent+neural+networks+from+overfitting%2E+Journal+of+Machine+Learning+Rese)
34. Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In Advances in Neural Information Processing Systems, pages 2692–2700. [find](https://scholar.google.com/scholar?q=Oriol+Vinyals%2C+Meire+Fortunato%2C+and+Navdeep+Jaitly%2E+2015%2E+Pointer+networks%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+2692%E2%80%932700%2E)
35. Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3-4):229–256. [find](https://scholar.google.com/scholar?q=Ronald+J+Williams%2E+1992%2E+Simple+statistical+gradient%2Dfollowing+algorithms+for+connectionist+reinforcement+learning%2E+Machine+Learning%2C+8%283%2D4%29%3A229%E2%80%93256%2E)
36. Ronald J Williams and David Zipser. 1989. A learning algorithm for continually running fully recurrent neural networks. Neural Computation, 1(2):270– 280. [find](https://scholar.google.com/scholar?q=Ronald+J+Williams+and+David+Zipser%2E+1989%2E+A+learning+algorithm+for+continually+running+fully+recurrent+neural+networks%2E+Neural+Computation%2C+1%282%29%3A270%E2%80%93+280%2E)
37. Sam Wiseman, Stuart Shieber, and Alexander Rush. 2017. Challenges in data-to-document generation. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP 2017). [find](https://scholar.google.com/scholar?q=Sam+Wiseman%2C+Stuart+Shieber%2C+and+Alexander+Rush%2E+2017%2E+Challenges+in+data%2Dto%2Ddocument+generation%2E+Proceedings+of+the+2017+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP+2017%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/wiseman-2017-challenges-data-document-generation/)
38. Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen. 2018. Breaking the softmax bottleneck: A high-rank RNN language model. [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Zihang+Dai%2C+Ruslan+Salakhutdinov%2C+and+William+W%2E+Cohen%2E+2018%2E+Breaking+the+softmax+bottleneck%3A+A+high%2Drank+RNN+language+model%2E)
39. The 2018 International Conference for Learning Representations. [find](https://scholar.google.com/scholar?q=The+2018+International+Conference+for+Learning+Representations%2E)
40. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In The 2020 International Conference for Learning Representations. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q+Weinberger%2C+and+Yoav+Artzi%2E+2020%2E+BERTScore%3A+Evaluating+text+generation+with+BERT%2E+In+The+2020+International+Conference+for+Learning+Representations%2E)
41. Yuhao Zhang, Daisy Yi Ding, Tianpei Qian, Christopher D. Manning, and Curtis P. Langlotz. 2018. Learning to summarize radiology findings. In EMNLP 2018 Workshop on Health Text Mining and Information Analysis. [find](https://scholar.google.com/scholar?q=Yuhao+Zhang%2C+Daisy+Yi+Ding%2C+Tianpei+Qian%2C+Christopher+D%2E+Manning%2C+and+Curtis+P%2E+Langlotz%2E+2018%2E+Learning+to+summarize+radiology+findings%2E+In+EMNLP+2018+Workshop+on+Health+Text+Mining+and+Information+A)
42. Split Train Dev Test Time Coverage Stanford RIH 2009/01 – 2014/04 2014/05 – 2014/08 2014/09 – 2014/12 [find](https://scholar.google.com/scholar?q=Split+Train+Dev+Test+Time+Coverage+Stanford+RIH+2009%2F01+%E2%80%93+2014%2F04+2014%2F05+%E2%80%93+2014%2F08+2014%2F09+%E2%80%93+2014%2F12)
43. Variable 2017/11 – 2018/06 2018/07 – 2018/09 2018/10 – 2018/12 *Table 6: Time coverage of different splits in the Stanford and RIH datasets.* [find](https://scholar.google.com/scholar?q=Variable+2017%2F11+%E2%80%93+2018%2F06+2018%2F07+%E2%80%93+2018%2F09+2018%2F10+%E2%80%93+2018%2F12+%2ATable+6%3A+Time+coverage+of+different+splits+in+the+Stanford+and+RIH+datasets%2E%2A)