---
title: "What Context Features Can Transformer Language Models Use?"
description: "Transformer language models achieve lower perplexity when they see hundreds of preceding tokens… 37 references, cited by 42 papers here."
authors: ["Joe O'Connor", "Jacob Andreas"]
year: 2021
venue: "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)"
source: "https://aclanthology.org/2021.acl-long.70/"
doi: "10.18653/v1/2021.acl-long.70"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/connor-2021-what-context-features-can-transformer/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/connor-2021-what-context-features-can-transformer/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) / What Context Features Can Transformer Language Models Use?

# What Context Features Can Transformer Language Models Use?

**Authors:** Joe O'Connor and [Jacob Andreas](https://md.research.elamur.ai/free/en/authors/jacob-andreas/) · **Year:** 2021 · **Venue:** [Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2021](https://md.research.elamur.ai/free/en/venues/acl/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.acl-long.70/) · [PDF](https://aclanthology.org/2021.acl-long.70.pdf) · [DOI 10.18653/v1/2021.acl-long.70](https://doi.org/10.18653/v1/2021.acl-long.70) · [OpenAlex](https://openalex.org/W3173681001)

**Citations:** 35 citations (OpenAlex, 2026-10-14) · [42](https://md.research.elamur.ai/free/en/papers/connor-2021-what-context-features-can-transformer/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Transformer language models achieve lower perplexity when they see hundreds of preceding tokens, but it is unclear which properties of that long context they actually exploit. O'Connor and Andreas tackle this question using V-information, a measure introduced by Xu et al. (2020) that quantifies how much information a computationally bounded predictor can extract from an input. Khandelwal et al. (2018) perturbed context only at test time. Here, separate models are trained and evaluated on deliberately damaged contexts, such as shuffled word order or deleted word classes. The damage is applied only to the distant part of the context, and nearby tokens stay intact. An "ablated information" ratio then shows what share of the benefit from the extra context is lost. The models use the GPT-2 architecture and are trained from scratch on WikiText-103. Even harsh ablations, like shuffling words within sentences or keeping only nouns, cost under 15% of the usable information. Long-range context therefore helps mainly through content words and local co-occurrence, not detailed syntax. This points toward better context representations rather than simply longer inputs.

## Key points
- Setup: GPT-2 architecture (Hugging Face implementation, default hyperparameters) trained from scratch on WikiText-103, which has about 103.2M training words and 217,646 evaluation words. Results are averaged over two random initializations, and the work took roughly 100 training runs.
- Extending context from 256 to 768 tokens lowers perplexity by about 4%. Ablations are applied to the far context while the nearest tokens are left unablated, with a separator token between the two parts.
- Shuffling words within sentences or deleting everything except nouns removes less than 15% of the usable information. Removing function words and shuffling within local windows (e.g., trigrams) has very little effect.
- Retaining only document identity or only named entities causes large accuracy drops, so topic information alone does not explain the benefit of long context. Deleting function words even improves held-out accuracy, which suggests the effects are partly model-specific.
- Code is released at github.com/lingo-mit/context-ablations.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Massachusetts Institute of Technology](https://md.research.elamur.ai/free/en/universities/massachusetts-institute-of-technology/) — University, Cambridge, United States · 211 works · 69,000 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Joe O'Connor and Jacob Andreas. 2021. What Context Features Can Transformer Language Models Use?. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{oconnor2021what,
  title = {{What Context Features Can Transformer Language Models Use?}},
  author = {Joe O'Connor and Jacob Andreas},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.70},
  url = {https://doi.org/10.18653/v1/2021.acl-long.70}
}
```

- [← previous Explanation-Based Human Debugging of NLP Models: A Survey](https://md.research.elamur.ai/free/en/papers/lertvittayakumjorn-2021-explanation-based-human-debugging-nlp/)
- [next → Arabic Speech Recognition by End-to-End, Modular Systems and Human](https://md.research.elamur.ai/free/en/papers/hussein-2021-arabic-speech-recognition-end-end/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/)

## References

37 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Regina Barzilay and Mirella Lapata. 2008. [Modeling local coherence: An entity-based approach](https://md.research.elamur.ai/free/en/papers/barzilay-2008-modeling-local-coherence-entity-based/). Computational Linguistics, 34(1):1–34.
2. Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. [Longformer: The long-document transformer](https://md.research.elamur.ai/free/en/papers/beltagy-2020-longformer-long-document-transformer/). arXiv:2004.05150.
3. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. [A neural probabilistic language model](https://md.research.elamur.ai/free/en/papers/bengio-2003-neural-probabilistic-language-model/). The Journal of Machine Learning Research, 3:1137–1155.
4. Peter F Brown, Stephen A Della Pietra, Vincent J Della Pietra, Jennifer C Lai, and Robert L Mercer. 1992. [An estimate of an upper bound for the entropy of english](https://md.research.elamur.ai/free/en/papers/brown-1992-estimate-upper-bound-entropy-english/). Computational Linguistics, 18(1):31–40.
5. Ralf D Brown. 2011. The CMU-EBMT machine translation system. Machine translation, 25(2):179. [find](https://scholar.google.com/scholar?q=Ralf+D+Brown%2E+2011%2E+The+CMU%2DEBMT+machine+translation+system%2E+Machine+translation%2C+25%282%29%3A179%2E)
6. Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Charlin. 2020. Language GANs falling short. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Massimo+Caccia%2C+Lucas+Caccia%2C+William+Fedus%2C+Hugo+Larochelle%2C+Joelle+Pineau%2C+and+Laurent+Charlin%2E+2020%2E+Language+GANs+falling+short%2E+In+International+Conference+on+Learning+Representations%2E)
7. Ozan Caglayan, Pranava Madhyastha, and Lucia Specia. 2020. Curious case of language generation evaluation metrics: A cautionary tale. In Proceedings of the 28th International Conference on Computational Linguistics, pages 2322–2328, Barcelona, Spain (Online). International Committee on Computational… [find](https://scholar.google.com/scholar?q=Ozan+Caglayan%2C+Pranava+Madhyastha%2C+and+Lucia+Specia%2E+2020%2E+Curious+case+of+language+generation+evaluation+metrics%3A+A+cautionary+tale%2E+In+Proceedings+of+the+28th+International+Conference+on+Computation)
8. Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. [Generating long sequences with sparse transformers](https://md.research.elamur.ai/free/en/papers/child-2019-generating-long-sequences-sparse-transformers/). URL [https://openai.com/blog/sparse-transformers](https://openai.com/blog/sparse-transformers). [↗](https://openai.com/blog/sparse-transformers)
9. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276–286, Florence, Italy. Association for Computationa… [find](https://scholar.google.com/scholar?q=Kevin+Clark%2C+Urvashi+Khandelwal%2C+Omer+Levy%2C+and+Christopher+D%2E+Manning%2E+2019%2E+What+does+BERT+look+at%3F+an+analysis+of+BERT%E2%80%99s+attention%2E+In+Proceedings+of+the+2019+ACL+Workshop+BlackboxNLP%3A+Analyzing+an)
10. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978–2988, Florence, Italy. Assoc… [find](https://scholar.google.com/scholar?q=Zihang+Dai%2C+Zhilin+Yang%2C+Yiming+Yang%2C+Jaime+Carbonell%2C+Quoc+Le%2C+and+Ruslan+Salakhutdinov%2E+2019%2E+Transformer%2DXL%3A+Attentive+language+models+beyond+a+fixed%2Dlength+context%2E+In+Proceedings+of+the+57th+Annu)
11. Jeffrey L Elman. 1990. Finding structure in time. Cognitive science, 14(2):179–211. [find](https://scholar.google.com/scholar?q=Jeffrey+L+Elman%2E+1990%2E+Finding+structure+in+time%2E+Cognitive+science%2C+14%282%29%3A179%E2%80%93211%2E)
12. Joshua T Goodman. 2001. A bit of progress in language modeling. Computer Speech & Language, 15(4):403–434. [find](https://scholar.google.com/scholar?q=Joshua+T+Goodman%2E+2001%2E+A+bit+of+progress+in+language+modeling%2E+Computer+Speech+%26+Language%2C+15%284%29%3A403%E2%80%93434%2E)
13. Michael Hahn. 2020. Theoretical limitations of self-attention in neural sequence models. Transactions of the Association for Computational Linguistics, 8:156–171. [find](https://scholar.google.com/scholar?q=Michael+Hahn%2E+2020%2E+Theoretical+limitations+of+self%2Dattention+in+neural+sequence+models%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+8%3A156%E2%80%93171%2E)
14. Tatsunori Hashimoto, Hugh Zhang, and Percy Liang. 2019. Unifying human and statistical evaluation for natural language generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Sho… [find](https://scholar.google.com/scholar?q=Tatsunori+Hashimoto%2C+Hugh+Zhang%2C+and+Percy+Liang%2E+2019%2E+Unifying+human+and+statistical+evaluation+for+natural+language+generation%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chapter+o)
15. Mikael Henaff, Jason Weston, Arthur Szlam, Antoine Bordes, and Yann LeCun. 2016. Tracking the world state with recurrent entity networks. In ICLR. [find](https://scholar.google.com/scholar?q=Mikael+Henaff%2C+Jason+Weston%2C+Arthur+Szlam%2C+Antoine+Bordes%2C+and+Yann+LeCun%2E+2016%2E+Tracking+the+world+state+with+recurrent+entity+networks%2E+In+ICLR%2E)
16. Matthew Honnibal, Ines Montani, Sofie Van Lan-deghem, and Adriane Boyd. 2020. spaCy: Industrial-strength Natural Language Processing in Python. [find](https://scholar.google.com/scholar?q=Matthew+Honnibal%2C+Ines+Montani%2C+Sofie+Van+Lan%2Ddeghem%2C+and+Adriane+Boyd%2E+2020%2E+spaCy%3A+Industrial%2Dstrength+Natural+Language+Processing+in+Python%2E)
17. Sarthak Jain and Byron C. Wallace. 2019. [Attention is not explanation](https://md.research.elamur.ai/free/en/papers/jain-2019-attention-is-not-explanation/). In NAACL-HLT.
18. Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky. 2018. Sharp nearby, fuzzy far away: How neural language models use context. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 284–294, Melbourne, Australia. Association for C… [find](https://scholar.google.com/scholar?q=Urvashi+Khandelwal%2C+He+He%2C+Peng+Qi%2C+and+Dan+Jurafsky%2E+2018%2E+Sharp+nearby%2C+fuzzy+far+away%3A+How+neural+language+models+use+context%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Compu)
19. Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. [Reformer: The efficient transformer](https://md.research.elamur.ai/free/en/papers/kitaev-2020-reformer-efficient-transformer/). In International Conference on Learning Representations.
20. Reinhard Kneser and Hermann Ney. 1995. Improved backing-off for m-gram language modeling. In 1995 International Conference on Acoustics, Speech, and Signal Processing, volume 1, pages 181–184. IEEE. [find](https://scholar.google.com/scholar?q=Reinhard+Kneser+and+Hermann+Ney%2E+1995%2E+Improved+backing%2Doff+for+m%2Dgram+language+modeling%2E+In+1995+International+Conference+on+Acoustics%2C+Speech%2C+and+Signal+Processing%2C+volume+1%2C+pages+181%E2%80%93184%2E+IEEE%2E)
21. Jiwei Li, Will Monroe, and Dan Jurafsky. 2016. Understanding neural networks through representation erasure. arXiv preprint arXiv:1612.08220. [find](https://scholar.google.com/scholar?q=Jiwei+Li%2C+Will+Monroe%2C+and+Dan+Jurafsky%2E+2016%2E+Understanding+neural+networks+through+representation+erasure%2E+arXiv+preprint+arXiv%3A1612%2E08220%2E)
22. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. [Pointer sentinel mixture models](https://md.research.elamur.ai/free/en/papers/merity-2016-pointer-sentinel-mixture-models/). CoRR, abs/1609.07843.
23. Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černockỳ, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Eleventh annual conference of the International Speech Communication Association. [find](https://scholar.google.com/scholar?q=Toma%CC%81s%CC%8C+Mikolov%2C+Martin+Karafia%CC%81t%2C+Luka%CC%81s%CC%8C+Burget%2C+Jan+C%CC%8Cernocky%CC%80%2C+and+Sanjeev+Khudanpur%2E+2010%2E+Recurrent+neural+network+based+language+model%2E+In+Eleventh+annual+conference+of+the+International+Speech)
24. F. Mollica, Matthew Siegelman, Evgeniia Diachek, S. Piantadosi, Zachary Mineroff, Richard Futrell, Hope H. Kean, Peng Qian, and E. Fedorenko. 2020. Composition is the core driver of the language-selective network. Neurobiology of Language, 1:104–134. [find](https://scholar.google.com/scholar?q=F%2E+Mollica%2C+Matthew+Siegelman%2C+Evgeniia+Diachek%2C+S%2E+Piantadosi%2C+Zachary+Mineroff%2C+Richard+Futrell%2C+Hope+H%2E+Kean%2C+Peng+Qian%2C+and+E%2E+Fedorenko%2E+2020%2E+Composition+is+the+core+driver+of+the+language%2Dselec)
25. Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. Rissanen data analysis: Examining dataset characteristics via description length. arXiv preprint arXiv:2103.03872. [find](https://scholar.google.com/scholar?q=Ethan+Perez%2C+Douwe+Kiela%2C+and+Kyunghyun+Cho%2E+2021%2E+Rissanen+data+analysis%3A+Examining+dataset+characteristics+via+description+length%2E+arXiv+preprint+arXiv%3A2103%2E03872%2E)
26. Thang M Pham, Trung Bui, Long Mai, and Anh Nguyen. 2020. [Out of order: How important is the sequential order of words in a sentence in natural language understanding tasks?](https://md.research.elamur.ai/free/en/papers/pham-2021-out-order-how-important-is/) arXiv preprint arXiv:2012.15180.
27. Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. 2020. [Information-theoretic probing for linguistic structure](https://md.research.elamur.ai/free/en/papers/pimentel-2020-information-theoretic-probing-linguistic-structure/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4609–4622, Online. Association for Comp…
28. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E)
29. Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, and Timothy P. Lillicrap. 2019. [Compressive transformers for long-range sequence modelling](https://md.research.elamur.ai/free/en/papers/rae-2019-compressive-transformers-long-range-sequence/).
30. Chinnadhurai Sankar, Sandeep Subramanian, Christopher Pal, Sarath Chandar, and Yoshua Bengio. 2019. Do neural dialog systems use the conversation history effectively? an empirical study. arXiv preprint arXiv:1906.01603. [find](https://scholar.google.com/scholar?q=Chinnadhurai+Sankar%2C+Sandeep+Subramanian%2C+Christopher+Pal%2C+Sarath+Chandar%2C+and+Yoshua+Bengio%2E+2019%2E+Do+neural+dialog+systems+use+the+conversation+history+effectively%3F+an+empirical+study%2E+arXiv+preprin)
31. Claude E Shannon. 1948. A mathematical theory of communication. The Bell system technical journal, 27(3):379–423. [find](https://scholar.google.com/scholar?q=Claude+E+Shannon%2E+1948%2E+A+mathematical+theory+of+communication%2E+The+Bell+system+technical+journal%2C+27%283%29%3A379%E2%80%93423%2E)
32. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Proces…
33. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. [Super-GLUE: A stickier benchmark for general-purpose language understanding systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/). arXiv preprint arXiv:1905.00537.
34. Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and Hao Ma. 2020. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768. [find](https://scholar.google.com/scholar?q=Sinong+Wang%2C+Belinda+Li%2C+Madian+Khabsa%2C+Han+Fang%2C+and+Hao+Ma%2E+2020%2E+Linformer%3A+Self%2Dattention+with+linear+complexity%2E+arXiv+preprint+arXiv%3A2006%2E04768%2E)
35. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtow-icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin L… — [Transformers: State-of-the-Art Natural Language Processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
36. Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. 2020. A theory of usable information under computational constraints. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Yilun+Xu%2C+Shengjia+Zhao%2C+Jiaming+Song%2C+Russell+Stewart%2C+and+Stefano+Ermon%2E+2020%2E+A+theory+of+usable+information+under+computational+constraints%2E+In+International+Conference+on+Learning+Representations)
37. Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. 2016. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530. [find](https://scholar.google.com/scholar?q=Chiyuan+Zhang%2C+Samy+Bengio%2C+Moritz+Hardt%2C+Benjamin+Recht%2C+and+Oriol+Vinyals%2E+2016%2E+Understanding+deep+learning+requires+rethinking+generalization%2E+arXiv+preprint+arXiv%3A1611%2E03530%2E)

*Updated 29 October 2026*