---
title: "Answers Unite! Unsupervised Metrics for — Scialom 2019"
description: "Reinforcement learning lets abstractive summarizers optimise non-differentiable rewards, but the usual reward, ROUGE, measures n-gram overlap with a…"
authors: ["Thomas Scialom", "Sylvain Lamprier", "Benjamin Piwowarski", "Jacopo Staiano"]
year: 2019
venue: "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)"
source: "https://aclanthology.org/D19-1320/"
doi: "10.18653/v1/d19-1320"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/scialom-2019-answers-unite-unsupervised-metrics-reinforced/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/scialom-2019-answers-unite-unsupervised-metrics-reinforced/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/) / Answers Unite! Unsupervised Metrics for Reinforced Summarization Models

# Answers Unite! Unsupervised Metrics for Reinforced Summarization Models

**Authors:** [Thomas Scialom](https://md.research.elamur.ai/free/en/authors/thomas-scialom/), [Sylvain Lamprier](https://md.research.elamur.ai/free/en/authors/sylvain-lamprier/), [Benjamin Piwowarski](https://md.research.elamur.ai/free/en/authors/benjamin-piwowarski/), [Jacopo Staiano](https://md.research.elamur.ai/free/en/authors/jacopo-staiano/) · **Year:** 2019 · **Venue:** Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D19-1320/) · [PDF](https://aclanthology.org/D19-1320.pdf) · [DOI 10.18653/v1/d19-1320](https://doi.org/10.18653/v1/d19-1320) · [OpenAlex](https://openalex.org/W2970892365)

**Citations:** 120 citations (openalex-snapshot · 2026-10-14) · [90](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Reinforcement learning lets abstractive summarizers optimise non-differentiable rewards, but the usual reward, ROUGE, measures n-gram overlap with a reference. It therefore favours lexical similarity and captures fluency poorly. The authors build on APES, which masks named entities in a reference summary to create cloze-style questions and then checks whether a QA model can answer them from the generated summary. They score summaries by QA F1 and by QA confidence (the probability given to the correct answer). They also add unsupervised variants that generate the questions from the source article, so no reference summary is needed.

Against human ratings of CNN/Daily Mail outputs, the unsupervised QA metrics track readability and relevance better than ROUGE, TextRank, Novelty or BERT LM perplexity. When used as RL rewards on top of the Gehrmann et al. (2018) model, they improve human-judged relevance without the readability collapse caused by a pure ROUGE reward. Because the metrics need no references, unlabeled in-domain and out-of-domain text can also be used for training.

## Key points
- The metrics are QA f score and QA conf, each in a supervised (sup) and an unsupervised (unsup) version, plus a learned combination ("QA learned") fitted by Ridge regression on 50 human-rated samples from Paulus et al. (2017).
- **Correlation study:** the authors used 100 CNN/Daily Mail samples from 3 systems, rated 1–10 for Readability and Relevance on Amazon Mechanical Turk.
  - QA conf (unsup) reached Spearman ρ of 0.33 for readability and 0.31 for relevance.
  - ROUGE variants stayed at roughly 0.12–0.18.
- **Human evaluation against the baseline (readability 7.07, relevance 5.82):**
  - A ROUGE-only reward dropped readability to 2.14.
  - QA learned trained with unlabeled CNN-DM TEST documents scored 6.89 for readability and 6.80 for relevance.
- **Effect of unlabeled data:** benefits grew as the unlabeled text got closer to the test documents. TL;DR (out-of-domain) gave a slight gain, CNN-DM VAL a larger one, and CNN-DM TEST the largest.
- **Code:** the metrics are to be released as the Python package summa-qa (github.com/recitalAI/summa-qa).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

90 papers of our library cite it (of the 140884 whose bibliographies we read); 10 of them have a page here, 80 not yet.

- [CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization](https://md.research.elamur.ai/free/en/papers/cao-2021-cliff-contrastive-learning-improving-faithfulness/)
- [FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization](https://md.research.elamur.ai/free/en/papers/durmus-2020-feqa-question-answering-evaluation-framework/)
- [QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization](https://md.research.elamur.ai/free/en/papers/fabbri-2022-qafacteval-improved-qa-based-factual/)
- [Q²: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering](https://md.research.elamur.ai/free/en/papers/honovich-2021-q-2-evaluating-factual-consistency/)
- [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://md.research.elamur.ai/free/en/papers/keskar-2019-ctrl-conditional-transformer-language-model/)
- [Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation](https://md.research.elamur.ai/free/en/papers/liu-2022-revisiting-gold-standard-grounding-summarization/)
- [A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation](https://md.research.elamur.ai/free/en/papers/liu-2022-token-level-reference-free-hallucination/)
- [MLSUM: The Multilingual Summarization Corpus](https://md.research.elamur.ai/free/en/papers/scialom-2020-mlsum-multilingual-summarization-corpus/)
- [QuestEval: Summarization Asks for Fact-based Evaluation](https://md.research.elamur.ai/free/en/papers/scialom-2021-questeval-summarization-asks-fact-based/)
- [Recursively Summarizing Books with Human Feedback](https://md.research.elamur.ai/free/en/papers/wu-2021-recursively-summarizing-books-human-feedback/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{scialom2019answers,
  title = {{Answers Unite! Unsupervised Metrics for Reinforced Summarization Models}},
  author = {Scialom, Thomas and Lamprier, Sylvain and Piwowarski, Benjamin and Staiano, Jacopo},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1320},
  url = {https://doi.org/10.18653/v1/d19-1320}
}
```

- [← previous Answer Extraction for Why Arabic Questions Answering Systems: EWAQ](https://md.research.elamur.ai/free/en/papers/al-2019-khawaldeh-answer-extraction-why-arabic-questions/)
- [next → APE at Scale and Its Implications on MT Evaluation Biases](https://md.research.elamur.ai/free/en/papers/freitag-2019-ape-scale-its-implications-mt/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/)

## References

32 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Kristjan Arumae and Fei Liu. 2019. Guiding extractive summarization with question-answering rewards. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2566–2577, M… [find](https://scholar.google.com/scholar?q=Kristjan+Arumae+and+Fei+Liu%2E+2019%2E+Guiding+extractive+summarization+with+question%2Danswering+rewards%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chapter+of+the+Association+for+Computat)
2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. [find](https://scholar.google.com/scholar?q=Dzmitry+Bahdanau%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2014%2E+Neural+machine+translation+by+jointly+learning+to+align+and+translate%2E+arXiv+preprint+arXiv%3A1409%2E0473%2E) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
3. Ping Chen, Fei Wu, Tong Wang, and Wei Ding. 2018. A semantic qa-based approach for text summarization evaluation. In Thirty-Second AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Ping+Chen%2C+Fei+Wu%2C+Tong+Wang%2C+and+Wei+Ding%2E+2018%2E+A+semantic+qa%2Dbased+approach+for+text+summarization+evaluation%2E+In+Thirty%2DSecond+AAAI+Conference+on+Artificial+Intelligence%2E)
4. Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 675–686, Melbourne, Australia. Association for Computational Ling… [find](https://scholar.google.com/scholar?q=Yen%2DChun+Chen+and+Mohit+Bansal%2E+2018%2E+Fast+abstractive+summarization+with+reinforce%2Dselected+sentence+rewriting%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguist) [our card](https://md.research.elamur.ai/free/en/papers/chen-2018-fast-abstractive-summarization-reinforce-selected/)
5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+the+2019+Conference+of+the+N)
6. Matan Eyal, Tal Baumel, and Michael Elhadad. 2019. Question answering as an automatic evaluation metric for news article summarization. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for C… [find](https://scholar.google.com/scholar?q=Matan+Eyal%2C+Tal+Baumel%2C+and+Michael+Elhadad%2E+2019%2E+Question+answering+as+an+automatic+evaluation+metric+for+news+article+summarization%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chap)
7. Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4098–4109. [find](https://scholar.google.com/scholar?q=Sebastian+Gehrmann%2C+Yuntian+Deng%2C+and+Alexander+Rush%2E+2018%2E+Bottom%2Dup+abstractive+summarization%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+4098%E2%80%934) [our card](https://md.research.elamur.ai/free/en/papers/gehrmann-2018-bottom-up-abstractive-summarization/)
8. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in neural information processing systems, pages 1693– 1701. [find](https://scholar.google.com/scholar?q=Karl+Moritz+Hermann%2C+Tomas+Kocisky%2C+Edward+Grefenstette%2C+Lasse+Espeholt%2C+Will+Kay%2C+Mustafa+Suleyman%2C+and+Phil+Blunsom%2E+2015%2E+Teaching+machines+to+read+and+comprehend%2E+In+Advances+in+neural+information) [our card](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/)
9. Wojciech Kryściński, Romain Paulus, Caiming Xiong, and Richard Socher. 2018. Improving abstraction in text summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1808–1817, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Wojciech+Krys%CC%81cin%CC%81ski%2C+Romain+Paulus%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+2018%2E+Improving+abstraction+in+text+summarization%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Langua)
10. Piji Li, Lidong Bing, and Wai Lam. 2018. Actor-critic based training framework for abstractive summarization. arXiv preprint arXiv:1803.11070. — [arXiv:1803.11070](https://arxiv.org/abs/1803.11070) [find](https://scholar.google.com/scholar?q=Piji+Li%2C+Lidong+Bing%2C+and+Wai+Lam%2E+2018%2E+Actor%2Dcritic+based+training+framework+for+abstractive+summarization%2E+arXiv+preprint+arXiv%3A1803%2E11070%2E+%E2%80%94+%5BarXiv%3A1803%2E11070%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1803%2E11070%29)
11. C-Y Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Proc. of Workshop on Text Summarization Branches Out, Post Conference Workshop of ACL 2004. [find](https://scholar.google.com/scholar?q=C%2DY+Lin%2E+2004%2E+Rouge%3A+A+package+for+automatic+evaluation+of+summaries%2E+In+Proc%2E+of+Workshop+on+Text+Summarization+Branches+Out%2C+Post+Conference+Workshop+of+ACL+2004%2E)
12. Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural L… [find](https://scholar.google.com/scholar?q=Chia%2DWei+Liu%2C+Ryan+Lowe%2C+Iulian+Serban%2C+Mike+Noseworthy%2C+Laurent+Charlin%2C+and+Joelle+Pineau%2E+2016%2E+How+not+to+evaluate+your+dialogue+system%3A+An+empirical+study+of+unsupervised+evaluation+metrics+for+d)
13. Rada Mihalcea and Paul Tarau. 2004. Textrank: Bringing order into text. In Proceedings of the 2004 conference on empirical methods in natural language processing. [find](https://scholar.google.com/scholar?q=Rada+Mihalcea+and+Paul+Tarau%2E+2004%2E+Textrank%3A+Bringing+order+into+text%2E+In+Proceedings+of+the+2004+conference+on+empirical+methods+in+natural+language+processing%2E)
14. Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 280–290. [find](https://scholar.google.com/scholar?q=Ramesh+Nallapati%2C+Bowen+Zhou%2C+Cicero+dos+Santos%2C+Caglar+Gulcehre%2C+and+Bing+Xiang%2E+2016%2E+Abstractive+text+summarization+using+sequence%2Dto%2Dsequence+rnns+and+beyond%2E+In+Proceedings+of+The+20th+SIGNLL+Con)
15. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Ranking sentences for extractive summarization with reinforcement learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Paper… [find](https://scholar.google.com/scholar?q=Shashi+Narayan%2C+Shay+B%2E+Cohen%2C+and+Mirella+Lapata%2E+2018%2E+Ranking+sentences+for+extractive+summarization+with+reinforcement+learning%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter) [our card](https://md.research.elamur.ai/free/en/papers/narayan-2018-ranking-sentences-extractive-summarization-reinforcement/)
16. Ani Nenkova. 2011. Automatic Summarization. Foundations and Trends in Information Retrieval. Now. [find](https://scholar.google.com/scholar?q=Ani+Nenkova%2E+2011%2E+Automatic+Summarization%2E+Foundations+and+Trends+in+Information+Retrieval%2E+Now%2E)
17. Jun-Ping Ng and Viktoria Abrecht. 2015. Better summarization evaluation with word embeddings for rouge. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1925–1930. [find](https://scholar.google.com/scholar?q=Jun%2DPing+Ng+and+Viktoria+Abrecht%2E+2015%2E+Better+summarization+evaluation+with+word+embeddings+for+rouge%2E+In+Proceedings+of+the+2015+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages)
18. Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. 1999. The pagerank citation ranking: Bringing order to the web. Technical report, Stanford InfoLab. [find](https://scholar.google.com/scholar?q=Lawrence+Page%2C+Sergey+Brin%2C+Rajeev+Motwani%2C+and+Terry+Winograd%2E+1999%2E+The+pagerank+citation+ranking%3A+Bringing+order+to+the+web%2E+Technical+report%2C+Stanford+InfoLab%2E) [our card](https://md.research.elamur.ai/free/en/papers/page-1999-pagerank-citation-ranking-bringing/)
19. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+Bleu%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+Proceedings+of+the+40th+annual+meeting+on+association+for+computat) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
20. Ramakanth Pasunuru and Mohit Bansal. 2018. Multi-reward reinforced summarization with saliency and entailment. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 646– 653, N… [find](https://scholar.google.com/scholar?q=Ramakanth+Pasunuru+and+Mohit+Bansal%2E+2018%2E+Multi%2Dreward+reinforced+summarization+with+saliency+and+entailment%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of+the+Association+fo) [our card](https://md.research.elamur.ai/free/en/papers/pasunuru-2018-multi-reward-reinforced-summarization-saliency/)
21. Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304. [find](https://scholar.google.com/scholar?q=Romain+Paulus%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+2017%2E+A+deep+reinforced+model+for+abstractive+summarization%2E+arXiv+preprint+arXiv%3A1705%2E04304%2E) [our card](https://md.research.elamur.ai/free/en/papers/paulus-2017-deep-reinforced-model-abstractive-summarization/)
22. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Jian+Zhang%2C+Konstantin+Lopyrev%2C+and+Percy+Liang%2E+2016%2E+SQuAD%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natu) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
23. Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2015. Sequence level training with recurrent neural networks. arXiv preprint arXiv:1511.06732. [find](https://scholar.google.com/scholar?q=Marc%E2%80%99Aurelio+Ranzato%2C+Sumit+Chopra%2C+Michael+Auli%2C+and+Wojciech+Zaremba%2E+2015%2E+Sequence+level+training+with+recurrent+neural+networks%2E+arXiv+preprint+arXiv%3A1511%2E06732%2E) [our card](https://md.research.elamur.ai/free/en/papers/ranzato-2015-sequence-level-training-recurrent-neural/)
24. Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. 2017. Self-critical sequence training for image captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7008–7024. [find](https://scholar.google.com/scholar?q=Steven+J+Rennie%2C+Etienne+Marcheret%2C+Youssef+Mroueh%2C+Jerret+Ross%2C+and+Vaibhava+Goel%2E+2017%2E+Self%2Dcritical+sequence+training+for+image+captioning%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision) [our card](https://md.research.elamur.ai/free/en/papers/rennie-2016-self-critical-sequence-training-image/)
25. Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073– 1083. [find](https://scholar.google.com/scholar?q=Abigail+See%2C+Peter+J+Liu%2C+and+Christopher+D+Manning%2E+2017%2E+Get+to+the+point%3A+Summarization+with+pointer%2Dgenerator+networks%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computation)
26. Elaheh ShafieiBavani, Mohammad Ebrahimi, Raymond Wong, and Fang Chen. 2018. Summarization evaluation in the absence of human model summaries using the compositionality of word embeddings. In Proceedings of the 27th International Conference on Computational Linguistics, pages 905– 914, Santa Fe, New… [find](https://scholar.google.com/scholar?q=Elaheh+ShafieiBavani%2C+Mohammad+Ebrahimi%2C+Raymond+Wong%2C+and+Fang+Chen%2E+2018%2E+Summarization+evaluation+in+the+absence+of+human+model+summaries+using+the+compositionality+of+word+embeddings%2E+In+Proceedin)
27. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+V+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+Advances+in+neural+information+processing+systems%2C+pages+3104%E2%80%933112%2E)
28. Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In Advances in Neural Information Processing Systems, pages 2692–2700. [find](https://scholar.google.com/scholar?q=Oriol+Vinyals%2C+Meire+Fortunato%2C+and+Navdeep+Jaitly%2E+2015%2E+Pointer+networks%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+2692%E2%80%932700%2E)
29. Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. 2017. Tl;dr: Mining reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization, pages 59– 63. [find](https://scholar.google.com/scholar?q=Michael+Vo%CC%88lske%2C+Martin+Potthast%2C+Shahbaz+Syed%2C+and+Benno+Stein%2E+2017%2E+Tl%3Bdr%3A+Mining+reddit+to+learn+automatic+summarization%2E+In+Proceedings+of+the+Workshop+on+New+Frontiers+in+Summarization%2C+pages+59) [our card](https://md.research.elamur.ai/free/en/papers/volske-2017-tl-dr-mining-reddit-learn/)
30. Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229–256. [find](https://scholar.google.com/scholar?q=Ronald+J+Williams%2E+1992%2E+Simple+statistical+gradient%2Dfollowing+algorithms+for+connectionist+reinforcement+learning%2E+Machine+learning%2C+8%283%2D4%29%3A229%E2%80%93256%2E)
31. Ronald J Williams and David Zipser. 1989. A learning algorithm for continually running fully recurrent neural networks. Neural computation, 1(2):270– 280. [find](https://scholar.google.com/scholar?q=Ronald+J+Williams+and+David+Zipser%2E+1989%2E+A+learning+algorithm+for+continually+running+fully+recurrent+neural+networks%2E+Neural+computation%2C+1%282%29%3A270%E2%80%93+280%2E)
32. Harris Wu, Dragomir R Radev, and Weiguo Fan. 2002. Towards answer-focused summarization. In Proceedings of the 1st International Conference on Information Technology and Applications. [find](https://scholar.google.com/scholar?q=Harris+Wu%2C+Dragomir+R+Radev%2C+and+Weiguo+Fan%2E+2002%2E+Towards+answer%2Dfocused+summarization%2E+In+Proceedings+of+the+1st+International+Conference+on+Information+Technology+and+Applications%2E)