---
title: "Do Long-Range Language Models Actually Use Long-Range"
description: "Long-range Transformer language models can now read thousands of tokens of preceding text, and… 41 references, cited by 58 papers here."
authors: ["Simeng Sun", "Kalpesh Krishna", "Andrew Mattarella-Micke", "Mohit Iyyer"]
year: 2021
venue: "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2021.emnlp-main.62/"
doi: "10.18653/v1/2021.emnlp-main.62"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sun-2021-do-long-range-language-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sun-2021-do-long-range-language-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) / Do Long-Range Language Models Actually Use Long-Range Context?

# Do Long-Range Language Models Actually Use Long-Range Context?

**Authors:** [Simeng Sun](https://md.research.elamur.ai/free/en/authors/simeng-sun/), [Kalpesh Krishna](https://md.research.elamur.ai/free/en/authors/kalpesh-krishna/), [Andrew Mattarella-Micke](https://md.research.elamur.ai/free/en/authors/andrew-mattarella-micke/) and [Mohit Iyyer](https://md.research.elamur.ai/free/en/authors/mohit-iyyer/) · **Year:** 2021 · **Venue:** [Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2021](https://md.research.elamur.ai/free/en/venues/emnlp/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.emnlp-main.62/) · [PDF](https://aclanthology.org/2021.emnlp-main.62.pdf) · [DOI 10.18653/v1/2021.emnlp-main.62](https://doi.org/10.18653/v1/2021.emnlp-main.62) · [OpenAlex](https://openalex.org/W3200409031)

**Citations:** 42 citations (OpenAlex, 2026-10-14) · [58](https://md.research.elamur.ai/free/en/papers/sun-2021-do-long-range-language-models/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Long-range Transformer language models can now read thousands of tokens of preceding text, and they reach lower perplexity on benchmarks such as PG-19. This paper asks whether that extra distant context is actually useful to them. The authors used public checkpoints of the Routing Transformer (RT, up to 8192 tokens) and a Local Transformer (LT) built from the same weights. They evaluated both on a sampled part of the PG-19 validation set. They varied the prefix length and perturbed the distant context by shuffling words or swapping in text from another document. They also measured effects on specific token types and on sequence-level tasks.

Overall perplexity stops improving once the prefix passes about 2K tokens. Only a narrow set of tokens benefits from more distant text, such as pieces of multi-subword words or rare tokens copied from far back, and fiction gains the most. Corrupting the distant context barely changes perplexity, and sequence-level prediction gains nothing. The study shows that longer input windows do not automatically bring discourse understanding. The authors argue that similar analyses should accompany future long-context models.

## Key points
- **Models:** RT has 490M parameters, 22 layers with 8 heads each, and a vocabulary of about 98K subwords; only its top two layers use clustering attention. It scores 33.2 test perplexity on PG-19. LT replaces the clustering heads with 256-token local attention, giving an effective receptive field of about 5K tokens.
- **Data:** The validation split of PG-19 has 50 books and about 3M tokens. The authors sampled 220K target tokens, scoring 10 targets per context. They dropped an annotated edition of *The Canterbury Tales*, which shrank the test/validation perplexity gap from about 10 to about 3.
- **Perplexity:** Aggregate perplexity levels off after a 2K-token prefix. Tokens inside subword clusters keep improving with RT up to about 5K tokens. Shuffling or replacing the distant context has little effect on perplexity.
- **Suffix identification:** The task has 7K examples, each asking which of six 128-token continuations follows a prefix, in a SWAG-style setup. Accuracy does not rise beyond 2K tokens of prefix, and RT and LT perform almost equally poorly.
- **Sequence copying:** Both models exploit a duplicated target sequence placed within the previous 512 tokens. They lose this advantage when the duplicate sits more than 2K tokens back.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Massachusetts Amherst](https://md.research.elamur.ai/free/en/universities/university-of-massachusetts-amherst/) — University, Amherst Center, United States · 67 works · 64,591 citations
- [Intuit](https://md.research.elamur.ai/free/en/companies/intuit/) — Company, Mountain View, United States · 4 works · 887 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,344 works

## Cite

Cite as: Simeng Sun, Kalpesh Krishna, Andrew Mattarella-Micke and Mohit Iyyer. 2021. Do Long-Range Language Models Actually Use Long-Range Context?. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{sun2021do,
  title = {{Do Long-Range Language Models Actually Use Long-Range Context?}},
  author = {Simeng Sun, Kalpesh Krishna, Andrew Mattarella-Micke and Mohit Iyyer},
  booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.62},
  url = {https://doi.org/10.18653/v1/2021.emnlp-main.62}
}
```

- [← previous Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Words](https://md.research.elamur.ai/free/en/papers/hofmann-2021-superbizarre-derivational/)
- [next → Data-to-text Generation with Macro Planning](https://md.research.elamur.ai/free/en/papers/puduppully-2021-data-text-generation-macro-planning/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/)

## References

41 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. [Longformer: The long-document transformer](https://md.research.elamur.ai/free/en/papers/beltagy-2020-longformer-long-document-transformer/).
2. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). arXiv preprint arXiv:2005.14165.
3. Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long sequences with sparse transformers. [find](https://scholar.google.com/scholar?q=Rewon+Child%2C+Scott+Gray%2C+Alec+Radford%2C+and+Ilya+Sutskever%2E+2019%2E+Generating+long+sequences+with+sparse+transformers%2E)
4. Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J Colwell, and Adrian Weller. 2021. Rethinking attention with performers. In International Conf… [find](https://scholar.google.com/scholar?q=Krzysztof+Marcin+Choromanski%2C+Valerii+Likhosherstov%2C+David+Dohan%2C+Xingyou+Song%2C+Andreea+Gane%2C+Tamas+Sarlos%2C+Peter+Hawkins%2C+Jared+Quincy+Davis%2C+Afroz+Mohiuddin%2C+Lukasz+Kaiser%2C+David+Benjamin+Belanger%2C+)
5. Gonçalo M. Correia, Vlad Niculae, and André F. T. Martins. 2019. [Adaptively sparse transformers](https://md.research.elamur.ai/free/en/papers/correia-2019-adaptively-sparse-transformers/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2174– 2184, Hong…
6. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978–2988, Florence, Italy. Assoc… [find](https://scholar.google.com/scholar?q=Zihang+Dai%2C+Zhilin+Yang%2C+Yiming+Yang%2C+Jaime+Carbonell%2C+Quoc+Le%2C+and+Ruslan+Salakhutdinov%2E+2019%2E+Transformer%2DXL%3A+Attentive+language+models+beyond+a+fixed%2Dlength+context%2E+In+Proceedings+of+the+57th+Annu)
7. Energy costs We conduct all our analysis experiments on RTX8000 GPUs. Although our work does not include training large language models, the energy costs of evaluating large pre-trained LMs, such as the Routing Transformer, should not be Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutano… [find](https://scholar.google.com/scholar?q=Energy+costs+We+conduct+all+our+analysis+experiments+on+RTX8000+GPUs%2E+Although+our+work+does+not+include+training+large+language+models%2C+the+energy+costs+of+evaluating+large+pre%2Dtrained+LMs%2C+such+as+t)
8. Barbara J. Grosz, Aravind K. Joshi, and Scott Weinstein. 1995. [Centering: A framework for modeling the local coherence of discourse](https://md.research.elamur.ai/free/en/papers/grosz-1995-centering-framework-modeling-local-coherence/). Computational Linguistics, 21(2):203–225.
9. Jerry R Hobbs. 1979. Coherence and coreference. Cognitive science, 3(1):67–90. [find](https://scholar.google.com/scholar?q=Jerry+R+Hobbs%2E+1979%2E+Coherence+and+coreference%2E+Cognitive+science%2C+3%281%29%3A67%E2%80%9390%2E)
10. Sebastian Hofstätter, Hamed Zamani, Bhaskar Mitra, Nick Craswell, and Allan Hanbury. 2020. Local self-attention over long text for efficient document retrieval. [find](https://scholar.google.com/scholar?q=Sebastian+Hofst%C3%A4tter%2C+Hamed+Zamani%2C+Bhaskar+Mitra%2C+Nick+Craswell%2C+and+Allan+Hanbury%2E+2020%2E+Local+self%2Dattention+over+long+text+for+efficient+document+retrieval%2E)
11. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Ari+Holtzman%2C+Jan+Buys%2C+Li+Du%2C+Maxwell+Forbes%2C+and+Yejin+Choi%2E+2020%2E+The+curious+case+of+neural+text+degeneration%2E+In+International+Conference+on+Learning+Representations%2E)
12. Yangfeng Ji, Trevor Cohn, Lingpeng Kong, Chris Dyer, and Jacob Eisenstein. 2015. Document context language models. arXiv preprint arXiv:1511.03962. — [arXiv:1511.03962](https://arxiv.org/abs/1511.03962) [find](https://scholar.google.com/scholar?q=Yangfeng+Ji%2C+Trevor+Cohn%2C+Lingpeng+Kong%2C+Chris+Dyer%2C+and+Jacob+Eisenstein%2E+2015%2E+Document+context+language+models%2E+arXiv+preprint+arXiv%3A1511%2E03962%2E+%E2%80%94+%5BarXiv%3A1511%2E03962%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1511%2E0396)
13. Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Francois Fleuret. 2020. Transformers are rnns: Fast autoregressive transformers with linear attention. In Proceedings of the 37th International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Angelos+Katharopoulos%2C+Apoorv+Vyas%2C+Nikolaos+Pappas%2C+and+Francois+Fleuret%2E+2020%2E+Transformers+are+rnns%3A+Fast+autoregressive+transformers+with+linear+attention%2E+In+Proceedings+of+the+37th+International)
14. Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky. 2018. Sharp nearby, fuzzy far away: How neural language models use context. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 284–294, Melbourne, Australia. Association for C… [find](https://scholar.google.com/scholar?q=Urvashi+Khandelwal%2C+He+He%2C+Peng+Qi%2C+and+Dan+Jurafsky%2E+2018%2E+Sharp+nearby%2C+fuzzy+far+away%3A+How+neural+language+models+use+context%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Compu)
15. Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. [Reformer: The efficient transformer](https://md.research.elamur.ai/free/en/papers/kitaev-2020-reformer-efficient-transformer/). In International Conference on Learning Representations.
16. Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021. [Hurdles to progress in long-form question answering](https://md.research.elamur.ai/free/en/papers/krishna-2021-hurdles-progress-long-form-question/). In North American Association for Computational Linguistics.
17. William Labov and Joshua Waletzky. 1997. Narrative analysis: Oral versions of personal experience. [find](https://scholar.google.com/scholar?q=William+Labov+and+Joshua+Waletzky%2E+1997%2E+Narrative+analysis%3A+Oral+versions+of+personal+experience%2E)
18. Yi-An Lai, Garima Lalwani, and Yi Zhang. 2020. Context analysis for pre-trained masked language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3789–3804, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yi%2DAn+Lai%2C+Garima+Lalwani%2C+and+Yi+Zhang%2E+2020%2E+Context+analysis+for+pre%2Dtrained+masked+language+models%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2020%2C+pages+3789%E2%80%933804%2C+Onlin)
19. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. [Effective approaches to attention-based neural machine translation](https://md.research.elamur.ai/free/en/papers/luong-2015-effective-approaches-attention-based-neural/). In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412–1421, Lisbon, Portugal. Association for Computational Linguistics.
20. Raymond J Mooney and Gerald DeJong. 1985. Learning schemata for natural language processing. In IJ-CAI, pages 681–687. [find](https://scholar.google.com/scholar?q=Raymond+J+Mooney+and+Gerald+DeJong%2E+1985%2E+Learning+schemata+for+natural+language+processing%2E+In+IJ%2DCAI%2C+pages+681%E2%80%93687%2E)
21. Ofir Press, Noah A. Smith, and Mike Lewis. 2020. Shortformer: Better language modeling using shorter inputs. [find](https://scholar.google.com/scholar?q=Ofir+Press%2C+Noah+A%2E+Smith%2C+and+Mike+Lewis%2E+2020%2E+Shortformer%3A+Better+language+modeling+using+shorter+inputs%2E)
22. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E)
23. Jack Rae and Ali Razavi. 2020. Do transformers need deep long-range memory? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7524–7529, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jack+Rae+and+Ali+Razavi%2E+2020%2E+Do+transformers+need+deep+long%2Drange+memory%3F+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+7524%E2%80%937529%2C+Online%2E+Associa)
24. Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. 2020. [Compressive transformers for long-range sequence modelling](https://md.research.elamur.ai/free/en/papers/rae-2019-compressive-transformers-long-range-sequence/). In International Conference on Learning Representations.
25. Roni Rosenfeld. 1996. A maximum entropy approach to adaptive statistical language modeling. [find](https://scholar.google.com/scholar?q=Roni+Rosenfeld%2E+1996%2E+A+maximum+entropy+approach+to+adaptive+statistical+language+modeling%2E)
26. Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. 2021. [Efficient content-based sparse attention with routing transformers](https://md.research.elamur.ai/free/en/papers/roy-2020-efficient-content-based-sparse-attention/). Transactions of the Association for Computational Linguistics, 9:53–68.
27. Roger C Schank and Robert P Abelson. 1977. Scripts, plans, goals, and understanding: an inquiry into human knowledge structures. [find](https://scholar.google.com/scholar?q=Roger+C+Schank+and+Robert+P+Abelson%2E+1977%2E+Scripts%2C+plans%2C+goals%2C+and+understanding%3A+an+inquiry+into+human+knowledge+structures%2E)
28. Vatsal Sharan, Sham Kakade, Percy Liang, and Gregory Valiant. 2018. Prediction with a short memory. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing, STOC 2018, page 1074–1087, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Vatsal+Sharan%2C+Sham+Kakade%2C+Percy+Liang%2C+and+Gregory+Valiant%2E+2018%2E+Prediction+with+a+short+memory%2E+In+Proceedings+of+the+50th+Annual+ACM+SIGACT+Symposium+on+Theory+of+Computing%2C+STOC+2018%2C+page+1074%E2%80%93)
29. Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. 2019. [Adaptive attention span in transformers](https://md.research.elamur.ai/free/en/papers/sukhbaatar-2019-adaptive-attention-span-transformers/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 331–335, Florence, Italy. Association for Computational Linguistics.
30. Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. 2021. Synthesizer: Rethinking self-attention for transformer models. [find](https://scholar.google.com/scholar?q=Yi+Tay%2C+Dara+Bahri%2C+Donald+Metzler%2C+Da%2DCheng+Juan%2C+Zhe+Zhao%2C+and+Che+Zheng%2E+2021%2E+Synthesizer%3A+Rethinking+self%2Dattention+for+transformer+models%2E)
31. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
32. Chaojun Wang and Rico Sennrich. 2020. [On exposure bias, hallucination and domain shift in neural machine translation](https://md.research.elamur.ai/free/en/papers/wang-2020-exposure-bias-hallucination-domain-shift/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3544–3552, Online. Association for Computational Linguistics.
33. A Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. 2020. Linformer: Self-attention with linear complexity. [find](https://scholar.google.com/scholar?q=A+Sinong+Wang%2C+Belinda+Z%2E+Li%2C+Madian+Khabsa%2C+Han+Fang%2C+and+Hao+Ma%2E+2020%2E+Linformer%3A+Self%2Dattention+with+linear+complexity%2E)
34. Tian Wang and Kyunghyun Cho. 2016. Larger-context language modelling with recurrent neural network. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1319–1329, Berlin, Germany. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tian+Wang+and+Kyunghyun+Cho%2E+2016%2E+Larger%2Dcontext+language+modelling+with+recurrent+neural+network%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1)
35. Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2020. [Neural text generation with unlikelihood training](https://md.research.elamur.ai/free/en/papers/welleck-2019-neural-text-generation-unlikelihood-training/). In International Conference on Learning Representations.
36. Jason Weston, Sumit Chopra, and Antoine Bordes. 2015. Memory networks. [find](https://scholar.google.com/scholar?q=Jason+Weston%2C+Sumit+Chopra%2C+and+Antoine+Bordes%2E+2015%2E+Memory+networks%2E)
37. Zhanghao Wu, Zhijian Liu, Ji Lin, Yujun Lin, and Song Han. 2020. Lite transformer with long-short range attention. [find](https://scholar.google.com/scholar?q=Zhanghao+Wu%2C+Zhijian+Liu%2C+Ji+Lin%2C+Yujun+Lin%2C+and+Song+Han%2E+2020%2E+Lite+transformer+with+long%2Dshort+range+attention%2E)
38. Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. [Big bird: Transformers for longer sequences](https://md.research.elamur.ai/free/en/papers/zaheer-2020-big-bird-transformers-longer-sequences/). Advances in Neural Information Processing Systems, 33.
39. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 93– 104, Brussels, Belgium. Association for Computational Ling… [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Yonatan+Bisk%2C+Roy+Schwartz%2C+and+Yejin+Choi%2E+2018%2E+SWAG%3A+A+large%2Dscale+adversarial+dataset+for+grounded+commonsense+inference%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+)
40. Jiacheng Zhang, Huanbo Luan, Maosong Sun, Feifei Zhai, Jingfang Xu, Min Zhang, and Yang Liu. 2018. [Improving the transformer translation model with document-level context](https://md.research.elamur.ai/free/en/papers/zhang-2018-improving-transformer-translation-model-document/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 533–542, Brussels, Belgium. Ass…
41. Pei Zhang, Boxing Chen, Niyu Ge, and Kai Fan. 2020. Long-short term masking transformer: A simple but effective baseline for document-level neural machine translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1081–1087, Online. Associat… [find](https://scholar.google.com/scholar?q=Pei+Zhang%2C+Boxing+Chen%2C+Niyu+Ge%2C+and+Kai+Fan%2E+2020%2E+Long%2Dshort+term+masking+transformer%3A+A+simple+but+effective+baseline+for+document%2Dlevel+neural+machine+translation%2E+In+Proceedings+of+the+2020+Confe)

*Updated 24 October 2026*