---
title: "Document Ranking with a Pretrained — Nogueira 2020"
description: "The paper reframes neural document reranking as a generation task, departing from the usual approach of encoder-only classifiers such as BERT."
authors: ["Rodrigo Nogueira", "Zhiying Jiang", "Ronak Pradeep", "Jimmy Lin"]
year: 2020
venue: "Findings of the Association for Computational Linguistics: EMNLP 2020"
source: "https://aclanthology.org/2020.findings-emnlp.63/"
doi: "10.18653/v1/2020.findings-emnlp.63"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/nogueira-2020-document-ranking-pretrained-sequence-sequence/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/nogueira-2020-document-ranking-pretrained-sequence-sequence/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/2/) / Document Ranking with a Pretrained Sequence-to-Sequence Model

# Document Ranking with a Pretrained Sequence-to-Sequence Model

**Authors:** [Rodrigo Nogueira](https://md.research.elamur.ai/free/en/authors/rodrigo-nogueira/), Zhiying Jiang, Ronak Pradeep, [Jimmy Lin](https://md.research.elamur.ai/free/en/authors/jimmy-lin/) · **Year:** 2020 · **Venue:** Findings of the Association for Computational Linguistics: EMNLP 2020 · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.findings-emnlp.63/) · [PDF](https://aclanthology.org/2020.findings-emnlp.63.pdf) · [DOI 10.18653/v1/2020.findings-emnlp.63](https://doi.org/10.18653/v1/2020.findings-emnlp.63) · [OpenAlex](https://openalex.org/W3100107515)

**Citations:** 448 citations (openalex-snapshot · 2026-10-14) · [90](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
The paper reframes neural document reranking as a generation task, departing from the usual approach of encoder-only classifiers such as BERT. The authors fine-tune T5 (base, large, 3B) on MS MARCO passage. Each query–document pair goes into a simple text template, and the model learns to output the word "true" or "false". At inference time, a softmax over just those two token logits produces a relevance probability. BM25 candidates are then reordered by that score. On MS MARCO, this approach beats strong encoder-only rerankers. Applied zero-shot to Robust04, Core17 and Core18, it surpasses earlier state-of-the-art systems that needed in-domain cross-validation. Its biggest advantage over BERT appears when little training data is available. Probing experiments swap in different target words and show that this choice changes effectiveness, even among semantically close words. This suggests the model draws on knowledge acquired during pretraining. The work matters because it establishes encoder–decoder models as practical, data-efficient rankers, with code and models released at pygaggle.ai.

## Key points
- Input template: `Query: [Q] Document: [D] Relevant:`. Target tokens are single SentencePiece words, so no subword logit aggregation is needed.
- Training uses MS MARCO passage (8.8M passages, about 530K relevant training pairs) with a constant learning rate of 10⁻³, 100K iterations, and class-balanced batches of 128. On one TPU v3-8, T5-base, large and 3B took roughly 12, 48 and 160 hours. T5-11B was not tested because of its cost.
- Zero-shot transfer collections:
  - Robust04: 249 topics, about 528K documents.
  - Core17: 50 topics, New York Times Annotated Corpus.
  - Core18: 50 topics, TREC Washington Post Corpus.
  - Topic descriptions served as T5 queries, and long documents were split into passages.
- With all training data, the choice of target tokens hardly matters. With 1K or 10K examples, most substitutions reduce effectiveness, and the "Subwords" variant falls below BM25 at 1K.
- The method should carry over to other seq2seq models, such as BART, MASS, UniLM and Pegasus, which are left for future work.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

90 papers of our library cite it (of the 140884 whose bibliographies we read); 4 of them have a page here, 86 not yet.

- [Promptagator: Few-shot Dense Retrieval From 8 Examples](https://md.research.elamur.ai/free/en/papers/dai-2022-promptagator-few-shot-dense-retrieval/)
- [Re2G: Retrieve, Rerank, Generate](https://md.research.elamur.ai/free/en/papers/glass-2022-re2g-retrieve-rerank-generate/)
- [MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages](https://md.research.elamur.ai/free/en/papers/zhang-2023-miracl-multilingual-retrieval-dataset-covering/)
- [Large Language Models for Information Retrieval: A Survey](https://md.research.elamur.ai/free/en/papers/zhu-2023-large-language-models-information-retrieval/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{nogueira2020document,
  title = {{Document Ranking with a Pretrained Sequence-to-Sequence Model}},
  author = {Nogueira, Rodrigo and Jiang, Zhiying and Pradeep, Ronak and Lin, Jimmy},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2020},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.63},
  url = {https://doi.org/10.18653/v1/2020.findings-emnlp.63}
}
```

- [← previous Distinguish Confusing Law Articles for Legal Judgment Prediction](https://md.research.elamur.ai/free/en/papers/xu-2020-distinguish-confusing-law-articles-legal/)
- [next → Document-level Neural MT: A Systematic Comparison](https://md.research.elamur.ai/free/en/papers/lopes-2020-document-level-neural-mt-systematic/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/2/)

## References

37 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. James Allan, Donna Harman, Evangelos Kanoulas, Dan Li, Christophe Van Gysel, and Ellen Voorhees. 2017. [find](https://scholar.google.com/scholar?q=James+Allan%2C+Donna+Harman%2C+Evangelos+Kanoulas%2C+Dan+Li%2C+Christophe+Van+Gysel%2C+and+Ellen+Voorhees%2E+2017%2E)
2. TREC 2017 common core track overview. In Proceedings of the Twenty-Sixth Text REtrieval Conference (TREC 2017). [find](https://scholar.google.com/scholar?q=TREC+2017+common+core+track+overview%2E+In+Proceedings+of+the+Twenty%2DSixth+Text+REtrieval+Conference+%28TREC+2017%29%2E)
3. James Allan, Donna Harman, Evangelos Kanoulas, and Ellen Voorhees. 2018. [find](https://scholar.google.com/scholar?q=James+Allan%2C+Donna+Harman%2C+Evangelos+Kanoulas%2C+and+Ellen+Voorhees%2E+2018%2E)
4. TREC 2018 common core track overview. In Proceedings of the Twenty-Seventh Text REtrieval Conference (TREC 2018). [find](https://scholar.google.com/scholar?q=TREC+2018+common+core+track+overview%2E+In+Proceedings+of+the+Twenty%2DSeventh+Text+REtrieval+Conference+%28TREC+2018%29%2E)
5. Nima Asadi and Jimmy Lin. 2013. Effectiveness/efficiency tradeoffs for candidate generation in multi-stage retrieval architectures. In Proceedings of the 36th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2013), pages 997–1000. [find](https://scholar.google.com/scholar?q=Nima+Asadi+and+Jimmy+Lin%2E+2013%2E+Effectiveness%2Fefficiency+tradeoffs+for+candidate+generation+in+multi%2Dstage+retrieval+architectures%2E+In+Proceedings+of+the+36th+Annual+International+ACM+SIGIR+Conference)
6. Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2016. MS MARCO: A human generated MAchine Reading COmprehension dataset. arXiv:1611.09268… [find](https://scholar.google.com/scholar?q=Payal+Bajaj%2C+Daniel+Campos%2C+Nick+Craswell%2C+Li+Deng%2C+Jianfeng+Gao%2C+Xiaodong+Liu%2C+Rangan+Majumder%2C+Andrew+McNamara%2C+Bhaskar+Mitra%2C+Tri+Nguyen%2C+Mir+Rosenberg%2C+Xia+Song%2C+Alina+Stoica%2C+Saurabh+Tiwary%2C+and+) [our card](https://md.research.elamur.ai/free/en/papers/bajaj-2016-ms-marco-human-generated/)
7. Zhuyun Dai and Jamie Callan. 2019. Deeper text understanding for IR with contextual neural language modeling. In Proceedings of the 42nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2019), pages 985–988, Paris, France. [find](https://scholar.google.com/scholar?q=Zhuyun+Dai+and+Jamie+Callan%2E+2019%2E+Deeper+text+understanding+for+IR+with+contextual+neural+language+modeling%2E+In+Proceedings+of+the+42nd+Annual+International+ACM+SIGIR+Conference+on+Research+and+Devel)
8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+the+2019+Conference+of+the+N)
9. Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. 2019. Show your work: Improved reporting of experimental results. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Pro… [find](https://scholar.google.com/scholar?q=Jesse+Dodge%2C+Suchin+Gururangan%2C+Dallas+Card%2C+Roy+Schwartz%2C+and+Noah+A%2E+Smith%2E+2019%2E+Show+your+work%3A+Improved+reporting+of+experimental+results%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Metho)
10. Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In Advances in Neural Information Processing Systems, pages 13063–13075. for ad-hoc retrieval. In… [find](https://scholar.google.com/scholar?q=Li+Dong%2C+Nan+Yang%2C+Wenhui+Wang%2C+Furu+Wei%2C+Xiaodong+Liu%2C+Yu+Wang%2C+Jianfeng+Gao%2C+Ming+Zhou%2C+and+Hsiao%2DWuen+Hon%2E+2019%2E+Unified+language+model+pre%2Dtraining+for+natural+language+understanding+and+generatio) [our card](https://md.research.elamur.ai/free/en/papers/dong-2019-unified-language-model-pre-training/)
11. Kai Hui, Andrew Yates, Klaus Berberich, and Gerard de Melo. 2018. Co-PACRR: A context-aware neural IR model for ad-hoc retrieval. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining (WSDM 2018), pages 279– 287. [find](https://scholar.google.com/scholar?q=Kai+Hui%2C+Andrew+Yates%2C+Klaus+Berberich%2C+and+Gerard+de+Melo%2E+2018%2E+Co%2DPACRR%3A+A+context%2Daware+neural+IR+model+for+ad%2Dhoc+retrieval%2E+In+Proceedings+of+the+Eleventh+ACM+International+Conference+on+Web+Sea)
12. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv:2001.08361. [find](https://scholar.google.com/scholar?q=Jared+Kaplan%2C+Sam+McCandlish%2C+Tom+Henighan%2C+Tom+B%2E+Brown%2C+Benjamin+Chess%2C+Rewon+Child%2C+Scott+Gray%2C+Alec+Radford%2C+Jeffrey+Wu%2C+and+Dario+Amodei%2E+2020%2E+Scaling+laws+for+neural+language+models%2E+arXiv%3A2001)
13. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71. [find](https://scholar.google.com/scholar?q=Taku+Kudo+and+John+Richardson%2E+2018%2E+SentencePiece%3A+A+simple+and+language+independent+subword+tokenizer+and+detokenizer+for+neural+text+processing%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+M) [our card](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/)
14. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the… [find](https://scholar.google.com/scholar?q=Mike+Lewis%2C+Yinhan+Liu%2C+Naman+Goyal%2C+Marjan+Ghazvininejad%2C+Abdelrahman+Mohamed%2C+Omer+Levy%2C+Veselin+Stoyanov%2C+and+Luke+Zettlemoyer%2E+2020%2E+BART%3A+Denoising+sequence%2Dto%2Dsequence+pre%2Dtraining+for+natural+l) [our card](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/)
15. Canjia Li, Andrew Yates, Sean MacAvaney, Ben He, and Yingfei Sun. 2020. PARADE: Passage representation aggregation for document reranking. arXiv:2008.09093. — [arXiv:2008.09093](https://arxiv.org/abs/2008.09093) [find](https://scholar.google.com/scholar?q=Canjia+Li%2C+Andrew+Yates%2C+Sean+MacAvaney%2C+Ben+He%2C+and+Yingfei+Sun%2E+2020%2E+PARADE%3A+Passage+representation+aggregation+for+document+reranking%2E+arXiv%3A2008%2E09093%2E+%E2%80%94+%5BarXiv%3A2008%2E09093%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F)
16. Hang Li. 2011. Learning to Rank for Information Retrieval and Natural Language Processing. Morgan & Claypool Publishers. [find](https://scholar.google.com/scholar?q=Hang+Li%2E+2011%2E+Learning+to+Rank+for+Information+Retrieval+and+Natural+Language+Processing%2E+Morgan+%26+Claypool+Publishers%2E)
17. Tie-Yan Liu. 2009. Learning to rank for information retrieval. Foundations and Trends in Information Retrieval, 3(3):225–331. [find](https://scholar.google.com/scholar?q=Tie%2DYan+Liu%2E+2009%2E+Learning+to+rank+for+information+retrieval%2E+Foundations+and+Trends+in+Information+Retrieval%2C+3%283%29%3A225%E2%80%93331%2E)
18. Sean MacAvaney, Andrew Yates, Arman Cohan, and Nazli Goharian. 2019. CEDR: Contextualized embeddings for document ranking. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1101–1104. [find](https://scholar.google.com/scholar?q=Sean+MacAvaney%2C+Andrew+Yates%2C+Arman+Cohan%2C+and+Nazli+Goharian%2E+2019%2E+CEDR%3A+Contextualized+embeddings+for+document+ranking%2E+In+Proceedings+of+the+42nd+International+ACM+SIGIR+Conference+on+Research+and)
19. Bhaskar Mitra and Nick Craswell. 2019. An introduction to neural information retrieval. Foundations and Trends in Information Retrieval, 13(1):1–126. [find](https://scholar.google.com/scholar?q=Bhaskar+Mitra+and+Nick+Craswell%2E+2019%2E+An+introduction+to+neural+information+retrieval%2E+Foundations+and+Trends+in+Information+Retrieval%2C+13%281%29%3A1%E2%80%93126%2E)
20. Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. arXiv:1805.04833. [find](https://scholar.google.com/scholar?q=Angela+Fan%2C+Mike+Lewis%2C+and+Yann+Dauphin%2E+2018%2E+Hierarchical+neural+story+generation%2E+arXiv%3A1805%2E04833%2E)
21. Bhaskar Mitra, Fernando Diaz, and Nick Craswell. 2017. Learning to match using local and distributed representations of text for web search. In Proceedings of the 26th International Conference on World Wide Web (WWW 2017), pages 1291–1299. [find](https://scholar.google.com/scholar?q=Bhaskar+Mitra%2C+Fernando+Diaz%2C+and+Nick+Craswell%2E+2017%2E+Learning+to+match+using+local+and+distributed+representations+of+text+for+web+search%2E+In+Proceedings+of+the+26th+International+Conference+on+Worl)
22. Jiafeng Guo, Yixing Fan, Qingyao Ai, and W. Bruce Croft. 2016. A deep relevance matching model Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage re-ranking with BERT. arXiv:1901.04085. [find](https://scholar.google.com/scholar?q=Jiafeng+Guo%2C+Yixing+Fan%2C+Qingyao+Ai%2C+and+W%2E+Bruce+Croft%2E+2016%2E+A+deep+relevance+matching+model+Rodrigo+Nogueira+and+Kyunghyun+Cho%2E+2019%2E+Passage+re%2Dranking+with+BERT%2E+arXiv%3A1901%2E04085%2E) [our card](https://md.research.elamur.ai/free/en/papers/nogueira-2019-passage-re-ranking-bert/)
23. Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. 2019a. Multi-stage document ranking with BERT. arXiv:1910.14424. — [arXiv:1910.14424](https://arxiv.org/abs/1910.14424) [find](https://scholar.google.com/scholar?q=Rodrigo+Nogueira%2C+Wei+Yang%2C+Kyunghyun+Cho%2C+and+Jimmy+Lin%2E+2019a%2E+Multi%2Dstage+document+ranking+with+BERT%2E+arXiv%3A1910%2E14424%2E+%E2%80%94+%5BarXiv%3A1910%2E14424%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1910%2E14424%29)
24. Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. 2019b. Document expansion by query prediction. arXiv:1904.08375. [find](https://scholar.google.com/scholar?q=Rodrigo+Nogueira%2C+Wei+Yang%2C+Jimmy+Lin%2C+and+Kyunghyun+Cho%2E+2019b%2E+Document+expansion+by+query+prediction%2E+arXiv%3A1904%2E08375%2E) [our card](https://md.research.elamur.ai/free/en/papers/nogueira-2019-document-expansion-query-prediction/)
25. Kezban Dilek Onal, Ye Zhang, Ismail Sengor Altingovde, Md Mustafizur Rahman, Pinar Karagoz, Alex Braylan, Brandon Dang, Heng-Lu Chang, Henna Kim, Quinten McNamara, Aaron Angert, Edward Banner, Vivek Khetan, Tyler McDonnell, An Thanh Nguyen, Dan Xu, Byron C. Wallace, Maarten de Rijke, and Matthew Lea… [find](https://scholar.google.com/scholar?q=Kezban+Dilek+Onal%2C+Ye+Zhang%2C+Ismail+Sengor+Altingovde%2C+Md+Mustafizur+Rahman%2C+Pinar+Karagoz%2C+Alex+Braylan%2C+Brandon+Dang%2C+Heng%2DLu+Chang%2C+Henna+Kim%2C+Quinten+McNamara%2C+Aaron+Angert%2C+Edward+Banner%2C+Vivek+K)
26. Raul Puri and Bryan Catanzaro. 2019. Zero-shot text classification with generative language models. arXiv:1912.10165. — [arXiv:1912.10165](https://arxiv.org/abs/1912.10165) [find](https://scholar.google.com/scholar?q=Raul+Puri+and+Bryan+Catanzaro%2E+2019%2E+Zero%2Dshot+text+classification+with+generative+language+models%2E+arXiv%3A1912%2E10165%2E+%E2%80%94+%5BarXiv%3A1912%2E10165%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1912%2E10165%29)
27. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1–67. [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+Wei+Li%2C+and+Peter+J+Liu%2E+2020%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtext)
28. Stephen E. Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, and Mike Gatford. 1994. Okapi at TREC-3. In Proceedings of the 3rd Text REtrieval Conference (TREC-3), pages 109– 126. [find](https://scholar.google.com/scholar?q=Stephen+E%2E+Robertson%2C+Steve+Walker%2C+Susan+Jones%2C+Micheline+Hancock%2DBeaulieu%2C+and+Mike+Gatford%2E+1994%2E+Okapi+at+TREC%2D3%2E+In+Proceedings+of+the+3rd+Text+REtrieval+Conference+%28TREC%2D3%29%2C+pages+109%E2%80%93+126%2E) [our card](https://md.research.elamur.ai/free/en/papers/robertson-1994-okapi-trec-3/)
29. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019. MASS: Masked sequence to sequence pre-training for language generation. In Proceedings of the 36th International Conference on Machine Learning, pages 5926–5936. [find](https://scholar.google.com/scholar?q=Kaitao+Song%2C+Xu+Tan%2C+Tao+Qin%2C+Jianfeng+Lu%2C+and+Tie%2DYan+Liu%2E+2019%2E+MASS%3A+Masked+sequence+to+sequence+pre%2Dtraining+for+language+generation%2E+In+Proceedings+of+the+36th+International+Conference+on+Machine) [our card](https://md.research.elamur.ai/free/en/papers/song-2019-mass-masked-sequence-sequence-pre/)
30. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N%2E+Gomez%2C+%C5%81ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+Advances+in+Neural+Information+Process)
31. Ellen M. Voorhees. 2004. Overview of the TREC 2004 Robust Track. In Proceedings of the Thirteenth Text REtrieval Conference (TREC 2004), pages 52–69. [find](https://scholar.google.com/scholar?q=Ellen+M%2E+Voorhees%2E+2004%2E+Overview+of+the+TREC+2004+Robust+Track%2E+In+Proceedings+of+the+Thirteenth+Text+REtrieval+Conference+%28TREC+2004%29%2C+pages+52%E2%80%9369%2E)
32. Chenyan Xiong, Zhuyun Dai, Jamie Callan, Zhiyuan Liu, and Russell Power. 2017. End-to-end neural ad-hoc ranking with kernel pooling. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2017), pages 55–64. [find](https://scholar.google.com/scholar?q=Chenyan+Xiong%2C+Zhuyun+Dai%2C+Jamie+Callan%2C+Zhiyuan+Liu%2C+and+Russell+Power%2E+2017%2E+End%2Dto%2Dend+neural+ad%2Dhoc+ranking+with+kernel+pooling%2E+In+Proceedings+of+the+40th+International+ACM+SIGIR+Conference+on+Re)
33. Peilin Yang, Hui Fang, and Jimmy Lin. 2017. Anserini: Enabling the use of Lucene for information retrieval research. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1253–1256. [find](https://scholar.google.com/scholar?q=Peilin+Yang%2C+Hui+Fang%2C+and+Jimmy+Lin%2E+2017%2E+Anserini%3A+Enabling+the+use+of+Lucene+for+information+retrieval+research%2E+In+Proceedings+of+the+40th+International+ACM+SIGIR+Conference+on+Research+and+Devel)
34. Wei Yang, Kuang Lu, Peilin Yang, and Jimmy Lin. 2019. Critically examining the “neural hype” weak baselines and the additivity of effectiveness gains from neural ranking models. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages… [find](https://scholar.google.com/scholar?q=Wei+Yang%2C+Kuang+Lu%2C+Peilin+Yang%2C+and+Jimmy+Lin%2E+2019%2E+Critically+examining+the+%E2%80%9Cneural+hype%E2%80%9D+weak+baselines+and+the+additivity+of+effectiveness+gains+from+neural+ranking+models%2E+In+Proceedings+of+the+)
35. Emine Yilmaz, Nick Craswell, Bhaskar Mitra, and Daniel Campos. 2020. On the reliability of test collections for evaluating systems of different types. In Proceedings of the 43rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2020), pages 2101– 2… [find](https://scholar.google.com/scholar?q=Emine+Yilmaz%2C+Nick+Craswell%2C+Bhaskar+Mitra%2C+and+Daniel+Campos%2E+2020%2E+On+the+reliability+of+test+collections+for+evaluating+systems+of+different+types%2E+In+Proceedings+of+the+43rd+Annual+International+A)
36. Zeynep Akkalyoncu Yilmaz, Wei Yang, Haotian Zhang, and Jimmy Lin. 2019. Cross-domain modeling of sentence-level evidence for document retrieval. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Pr… [find](https://scholar.google.com/scholar?q=Zeynep+Akkalyoncu+Yilmaz%2C+Wei+Yang%2C+Haotian+Zhang%2C+and+Jimmy+Lin%2E+2019%2E+Cross%2Ddomain+modeling+of+sentence%2Dlevel+evidence+for+document+retrieval%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Meth)
37. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2020. PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization. In Proceedings of the 37th International Conference on Machine Learning (ICML 2020), pages 2021– 2032. [find](https://scholar.google.com/scholar?q=Jingqing+Zhang%2C+Yao+Zhao%2C+Mohammad+Saleh%2C+and+Peter+J%2E+Liu%2E+2020%2E+PEGASUS%3A+Pre%2Dtraining+with+extracted+gap%2Dsentences+for+abstractive+summarization%2E+In+Proceedings+of+the+37th+International+Conference+) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2019-pegasus-pre-training-extracted-gap/)