---
title: "Modeling Localness for Self-Attention Networks — Yang 2018"
description: "It offers a cheap, task-independent way to inject locality into attention, and it combines well with… 27 references, cited by 59 papers here."
authors: ["Baosong Yang", "Zhaopeng Tu", "Derek F. Wong", "Fandong Meng", "Lidia S. Chao", "Tong Zhang"]
year: 2018
venue: "Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/D18-1475/"
doi: "10.18653/v1/d18-1475"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yang-2018-modeling-localness-self-attention-networks/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yang-2018-modeling-localness-self-attention-networks/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/3/) / Modeling Localness for Self-Attention Networks

# Modeling Localness for Self-Attention Networks

**Authors:** [Baosong Yang](https://md.research.elamur.ai/free/en/authors/baosong-yang/), [Zhaopeng Tu](https://md.research.elamur.ai/free/en/authors/zhaopeng-tu/), [Derek F. Wong](https://md.research.elamur.ai/free/en/authors/derek-f-wong/), [Fandong Meng](https://md.research.elamur.ai/free/en/authors/fandong-meng/), [Lidia S. Chao](https://md.research.elamur.ai/free/en/authors/lidia-s-chao/) and [Tong Zhang](https://md.research.elamur.ai/free/en/authors/tong-zhang/) · **Year:** 2018 · **Venue:** [Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2018](https://md.research.elamur.ai/free/en/venues/emnlp/2018/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D18-1475/) · [PDF](https://aclanthology.org/D18-1475.pdf) · [DOI 10.18653/v1/d18-1475](https://doi.org/10.18653/v1/d18-1475) · [OpenAlex](https://openalex.org/W2964302946)

**Citations:** 196 citations (OpenAlex, 2026-10-14) · [59](https://md.research.elamur.ai/free/en/papers/yang-2018-modeling-localness-self-attention-networks/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Self-attention in the Transformer links every token directly to all other tokens. This handles long-distance dependencies well, but it spreads attention weight thinly, so relations between neighbouring words can get lost. The authors add a learnable Gaussian bias to the attention logits before the softmax. For each query, the model predicts a centre position and a window size, which sets the standard deviation. Weights near the centre are boosted and distant ones are damped.

The window can take three forms:
- a fixed constant,
- one value per layer, derived from the mean of the keys,
- a separate value for every query.

Each attention head receives its own bias. The bias is applied only to the lower encoder layers, where short-range patterns dominate, so the upper layers stay free to model global context.

On WMT14 English⇒German and WMT17 Chinese⇒English, the method consistently improves over a strong Transformer baseline. It offers a cheap, task-independent way to inject locality into attention, and it combines well with relative position encoding.

## Key points
- The Gaussian bias is added to the dot-product energy. Its centre and window are scaled by a sigmoid to the sentence length. The fixed-window variant uses D = 10, following Luong et al. (2015).
- The query-specific window gave the best results. It shares the projection W_p with the centre-position predictor, which keeps it lighter than the layer-specific variant.
- The baseline Transformer has 6 encoder and 6 decoder layers. Restricting localness to the lower layers preserves long-range modelling. Localness helps more with multi-head attention than with a single head.
- The method is complementary to the relative position encoding of Shaw et al. (2018), and combining the two adds further gains.
- N-gram analysis shows slight losses on unigrams but gains on longer n-grams, with the query-specific variant strongest on 4–8-grams. Visualisation shows that learned windows widen in higher layers.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tencent](https://md.research.elamur.ai/free/en/companies/tencent/) — Company, Shenzhen, China · 66 works · 17,945 citations
- [University of Macau](https://md.research.elamur.ai/free/en/universities/university-of-macau/) — University, Macao, China · 10 works · 1,905 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

## Cite

Cite as: Baosong Yang, Zhaopeng Tu, Derek F. Wong, Fandong Meng, Lidia S. Chao and Tong Zhang. 2018. Modeling Localness for Self-Attention Networks. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yang2018modeling,
  title = {{Modeling Localness for Self-Attention Networks}},
  author = {Baosong Yang, Zhaopeng Tu, Derek F. Wong, Fandong Meng, Lidia S. Chao and Tong Zhang},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1475},
  url = {https://doi.org/10.18653/v1/d18-1475}
}
```

- [← previous Hurtlex: A Multilingual Lexicon of Words to Hurt](https://md.research.elamur.ai/free/en/papers/bassignana-2018-hurtlex-multilingual-lexicon-words-hurt/)
- [next → Retrieve and Refine: Improved Sequence Generation Models For Dialogue](https://md.research.elamur.ai/free/en/papers/weston-2018-retrieve-refine-improved-sequence-generation/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/3/)

## References

27 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Antonios Anastasopoulos and David Chiang. 2018. [Tied Multitask Learning for Neural Speech Translation](https://md.research.elamur.ai/free/en/papers/anastasopoulos-2018-tied-multitask-learning-neural-speech/). In NAACL.
2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. [Neural Machine Translation by Jointly Learning to Align and Translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). In ICLR.
3. Michał Daniluk, Tim Rocktäschel, Johannes Welbl, and Sebastian Riedel. 2017. Frustratingly short attention spans in neural language modeling. In ICLR. [find](https://scholar.google.com/scholar?q=Micha%C5%82+Daniluk%2C+Tim+Rockta%CC%88schel%2C+Johannes+Welbl%2C+and+Sebastian+Riedel%2E+2017%2E+Frustratingly+short+attention+spans+in+neural+language+modeling%2E+In+ICLR%2E)
4. Ziyi Dou, Zhaopeng Tu, Xing Wang, Shuming Shi, and Tong Zhang. 2018. Exploiting Deep Representations for Neural Machine Translation. In EMNLP. [find](https://scholar.google.com/scholar?q=Ziyi+Dou%2C+Zhaopeng+Tu%2C+Xing+Wang%2C+Shuming+Shi%2C+and+Tong+Zhang%2E+2018%2E+Exploiting+Deep+Representations+for+Neural+Machine+Translation%2E+In+EMNLP%2E)
5. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. [Convolutional Sequence to Sequence Learning](https://md.research.elamur.ai/free/en/papers/gehring-2017-convolutional-sequence-sequence-learning/). In ICML.
6. Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, et al. 2018. [Achieving Human Parity on Automatic Chinese to English News Translation](https://md.research.elamur.ai/free/en/papers/hassan-2018-achieving-human-parity-automatic-chinese/). arXiv:1803.05567.
7. Po Sen Huang, Chong Wang, Sitao Huang, Dengyong Zhou, and Li Deng. 2018. Towards Neural Phrase-based Machine Translation. In ICLR. [find](https://scholar.google.com/scholar?q=Po+Sen+Huang%2C+Chong+Wang%2C+Sitao+Huang%2C+Dengyong+Zhou%2C+and+Li+Deng%2E+2018%2E+Towards+Neural+Phrase%2Dbased+Machine+Translation%2E+In+ICLR%2E)
8. Jinbae Im and Sungzoon Cho. 2017. Distance-based Self-Attention Network for Natural Language Inference. arXiv:1712.02047. — [arXiv:1712.02047](https://arxiv.org/abs/1712.02047) [find](https://scholar.google.com/scholar?q=Jinbae+Im+and+Sungzoon+Cho%2E+2017%2E+Distance%2Dbased+Self%2DAttention+Network+for+Natural+Language+Inference%2E+arXiv%3A1712%2E02047%2E+%E2%80%94+%5BarXiv%3A1712%2E02047%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1712%2E02047%29)
9. Philipp Koehn. 2004. [Statistical Significance Tests for Machine Translation Evaluation](https://md.research.elamur.ai/free/en/papers/koehn-2004-statistical-significance-tests-machine-translation/). In EMNLP.
10. Jian Li, Zhaopeng Tu, Baosong Yang, Michael R. Lyu, and Tong Zhang. 2018. Multi-Head Attention with Disagreement Regularization. In EMNLP. [find](https://scholar.google.com/scholar?q=Jian+Li%2C+Zhaopeng+Tu%2C+Baosong+Yang%2C+Michael+R%2E+Lyu%2C+and+Tong+Zhang%2E+2018%2E+Multi%2DHead+Attention+with+Disagreement+Regularization%2E+In+EMNLP%2E)
11. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. [Effective Approaches to Attention-based Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/luong-2015-effective-approaches-attention-based-neural/). In EMNLP.
12. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [BLEU: A Method for Automatic Evaluation of Machine Translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In ACL.
13. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. [Deep Contextualized Word Representations](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/). In NAACL.
14. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. [Neural Machine Translation of Rare Words with Subword Units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/). In ACL.
15. Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. [Self-Attention with Relative Position Representations](https://md.research.elamur.ai/free/en/papers/shaw-2018-self-attention-relative-position-representations/). In NAACL.
16. Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Shirui Pan, and Chengqi Zhang. 2018a. DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding. In AAAI. [find](https://scholar.google.com/scholar?q=Tao+Shen%2C+Tianyi+Zhou%2C+Guodong+Long%2C+Jing+Jiang%2C+Shirui+Pan%2C+and+Chengqi+Zhang%2E+2018a%2E+DiSAN%3A+Directional+Self%2DAttention+Network+for+RNN%2FCNN%2DFree+Language+Understanding%2E+In+AAAI%2E)
17. Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, and Chengqi Zhang. 2018b. Bi-Directional Block Self-Attention for Fast and Memory-Efficient Sequence Modeling. In ICLR. [find](https://scholar.google.com/scholar?q=Tao+Shen%2C+Tianyi+Zhou%2C+Guodong+Long%2C+Jing+Jiang%2C+and+Chengqi+Zhang%2E+2018b%2E+Bi%2DDirectional+Block+Self%2DAttention+for+Fast+and+Memory%2DEfficient+Sequence+Modeling%2E+In+ICLR%2E)
18. Xing Shi, Inkit Padhi, and Kevin Knight. 2016. [Does String-based Neural MT Learn Source Syntax?](https://md.research.elamur.ai/free/en/papers/shi-2016-does-string-based-neural-mt/) In EMNLP.
19. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s Neural Machine Translation System: Bridging the Gap Between Human and Machine Translation. arXiv:1609.08144. [find](https://scholar.google.com/scholar?q=Yonghui+Wu%2C+Mike+Schuster%2C+Zhifeng+Chen%2C+Quoc+V+Le%2C+Mohammad+Norouzi%2C+Wolfgang+Macherey%2C+Maxim+Krikun%2C+Yuan+Cao%2C+Qin+Gao%2C+Klaus+Macherey%2C+et+al%2E+2016%2E+Google%E2%80%99s+Neural+Machine+Translation+System%3A+Bridg)
20. Chang Xu, Cecile Paris, Surya Nepal, and Ross Sparks. 2018. Cross-Target Stance Classification with Self-Attention Networks. In ACL. [find](https://scholar.google.com/scholar?q=Chang+Xu%2C+Cecile+Paris%2C+Surya+Nepal%2C+and+Ross+Sparks%2E+2018%2E+Cross%2DTarget+Stance+Classification+with+Self%2DAttention+Networks%2E+In+ACL%2E)
21. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C Courville, Ruslan Salakhutdinov, Richard S Zemel, and Yoshua Bengio. 2015. [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://md.research.elamur.ai/free/en/papers/xu-2015-show-attend-tell-neural-image/). In ICML.
22. Matthias Sperber, Jan Niehues, Graham Neubig, Sebastian Stüker, and Alex Waibel. 2018. Self-Attentional Acoustic Models. In Interspeech. [find](https://scholar.google.com/scholar?q=Matthias+Sperber%2C+Jan+Niehues%2C+Graham+Neubig%2C+Sebastian+Stu%CC%88ker%2C+and+Alex+Waibel%2E+2018%2E+Self%2DAttentional+Acoustic+Models%2E+In+Interspeech%2E)
23. Baosong Yang, Derek F. Wong, Tong Xiao, Lidia S. Chao, and Jingbo Zhu. 2017. Towards Bidirectional Hierarchical Representations for Attention-based Neural Machine Translation. In EMNLP. [find](https://scholar.google.com/scholar?q=Baosong+Yang%2C+Derek+F%2E+Wong%2C+Tong+Xiao%2C+Lidia+S%2E+Chao%2C+and+Jingbo+Zhu%2E+2017%2E+Towards+Bidirectional+Hierarchical+Representations+for+Attention%2Dbased+Neural+Machine+Translation%2E+In+EMNLP%2E)
24. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is All You Need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In NIPS.
25. Biao Zhang, Deyi Xiong, and Jinsong Su. 2018. Accelerating Neural Transformer via an Average Attention Network. In ACL. [find](https://scholar.google.com/scholar?q=Biao+Zhang%2C+Deyi+Xiong%2C+and+Jinsong+Su%2E+2018%2E+Accelerating+Neural+Transformer+via+an+Average+Attention+Network%2E+In+ACL%2E)
26. Xing Wang, Zhaopeng Tu, Deyi Xiong, and Min Zhang. 2017. Translating Phrases in Neural Machine Translation. In EMNLP. [find](https://scholar.google.com/scholar?q=Xing+Wang%2C+Zhaopeng+Tu%2C+Deyi+Xiong%2C+and+Min+Zhang%2E+2017%2E+Translating+Phrases+in+Neural+Machine+Translation%2E+In+EMNLP%2E)
27. Jiacheng Zhang, Yanzhuo Ding, Shiqi Shen, Yong Cheng, Maosong Sun, Huanbo Luan, and Yang Liu. 2017. THUMT: An Open Source Toolkit for Neural Machine Translation. arXiv:1706.06415. — [arXiv:1706.06415](https://arxiv.org/abs/1706.06415) [find](https://scholar.google.com/scholar?q=Jiacheng+Zhang%2C+Yanzhuo+Ding%2C+Shiqi+Shen%2C+Yong+Cheng%2C+Maosong+Sun%2C+Huanbo+Luan%2C+and+Yang+Liu%2E+2017%2E+THUMT%3A+An+Open+Source+Toolkit+for+Neural+Machine+Translation%2E+arXiv%3A1706%2E06415%2E+%E2%80%94+%5BarXiv%3A1706%2E06415%5D)

*Updated 24 October 2026*