--- title: "Probing the Attention Representation of — Yao 2026" description: "Earlier surprisal studies found that language models are sensitive to filler-gap dependencies (FGDs)." authors: ["Ruoqing Yao", "Pranav Anand"] year: 2026 venue: "Proceedings of the Society for Computation in Linguistics 2026" source: "https://aclanthology.org/2026.scil-main.23/" doi: "10.18653/v1/2026.scil-main.23" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/yao-2026-probing-attention-representation-filler-gap/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yao-2026-probing-attention-representation-filler-gap/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/8/) / Probing the Attention Representation of Filler-Gap Dependency in Transformers # Probing the Attention Representation of Filler-Gap Dependency in Transformers **Authors:** Ruoqing Yao, [Pranav Anand](https://md.research.elamur.ai/free/en/authors/pranav-anand/) · **Year:** 2026 · **Venue:** Proceedings of the Society for Computation in Linguistics 2026 · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.scil-main.23/) · [PDF](https://aclanthology.org/2026.scil-main.23.pdf) · [DOI 10.18653/v1/2026.scil-main.23](https://doi.org/10.18653/v1/2026.scil-main.23) · [OpenAlex](https://openalex.org/W7166876875) **Citations:** 0 citations (openalex-snapshot · 2026-10-14) ## Summary Earlier surprisal studies found that language models are sensitive to filler-gap dependencies (FGDs). These effects weaken as more clauses separate filler and gap, and they also weaken, puzzlingly, when the complementizer "that" is overt. This paper looks inside GPT-2 (12 layers × 12 heads) to see what attention does in these cases. The measure is the attention weight from the verb at the gap site back to the wh-filler. The authors test it on controlled item sets that vary: - clausal boundaries (0C/1C/2C), - linear distance, - island type, - relative clauses, - whether complementizers are overt. Two heads, h(5,2) and h(8,9), selectively mark grammatical gaps. Their attention also correlates with filled-gap surprisal. This attention survives long linear distance but drops with clausal embedding and islands. Head h(5,2) moves its attention toward the nearest overt C. This offers a possible explanation for why overt Cs dampen surprisal effects. Single-head ablation with TransformerLens supports a role for h(5,2) but not for h(8,9). The strongest ablation effect comes from h(1,10), a head that probing did not flag. The authors conclude that FGD processing is spread across partly specialized heads, and that attention weights tell only part of the story. ## Key points - **Probing results:** h(5,2) has the highest verb-to-filler attention of all 144 heads in wh- and relative-clause conditions. Both h(5,2) and h(8,9) correlate significantly with filled-gap surprisal (β = 6.33 and 5.61, p \< .001). - **Islands:** the materials are 50 island item sets from Kobzeva et al. (2025). h(5,2) is graded, dropping more for CNPC (strong) islands than for whether-islands (weak). h(8,9) treats both island types alike. - **Overt complementizers:** h(5,2) follows an "attend closest C" pattern, and its filler attention correlates with surprisal (β = 12.59). h(8,9) attends less to the filler when Cs are overt but still focuses mainly on it. - **Ablation:** one head at a time was zeroed out using TransformerLens. h(1,10) has the strongest effect and h(5,2) the second strongest in [+wh]. Ablating h(8,9) has no predicted effect. Effects are much weaker in [+th] items, which lack a filler. - **Limitations and future work:** the authors stress that neither probing nor ablation shows true causality. They propose activation patching as the next step. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{yao2026probing, title = {{Probing the Attention Representation of Filler-Gap Dependency in Transformers}}, author = {Yao, Ruoqing and Anand, Pranav}, booktitle = {Proceedings of the Society for Computation in Linguistics 2026}, year = {2026}, doi = {10.18653/v1/2026.scil-main.23}, url = {https://doi.org/10.18653/v1/2026.scil-main.23} } ``` - [← previous Probing Discrete Speech Tokens of Spoken Language Models](https://md.research.elamur.ai/free/en/papers/naber-2026-probing-discrete-speech-tokens-spoken/) - [next → Probing the Dynamics of Syntactic Ability Acquisition Throughout LLM Pretraining](https://md.research.elamur.ai/free/en/papers/matsuda-2026-probing-dynamics-syntactic-ability-acquisition/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/8/) ## References 6 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. S. Boguraev, C. Potts, and K. Mahowald. 2025. Causal interventions reveal shared structure across english filler–gap constructions. In Proceedings of the 2025 EMNLP, pages 25032–25053. [find](https://scholar.google.com/scholar?q=S%2E+Boguraev%2C+C%2E+Potts%2C+and+K%2E+Mahowald%2E+2025%2E+Causal+interventions+reveal+shared+structure+across+english+filler%E2%80%93gap+constructions%2E+In+Proceedings+of+the+2025+EMNLP%2C+pages+25032%E2%80%9325053%2E) [our card](https://md.research.elamur.ai/free/en/papers/boguraev-2025-causal-interventions-reveal-shared-structure/) 2. C.-Y. Chang, X. Huang, H. Nasir, S. Storks, O. Akingbade, and H. Dai. 2025. Mind the gap. In EMNLP 2025, pages 15060–15076. [find](https://scholar.google.com/scholar?q=C%2E%2DY%2E+Chang%2C+X%2E+Huang%2C+H%2E+Nasir%2C+S%2E+Storks%2C+O%2E+Akingbade%2C+and+H%2E+Dai%2E+2025%2E+Mind+the+gap%2E+In+EMNLP+2025%2C+pages+15060%E2%80%9315076%2E) 3. A. Kobzeva, S. Arehalli, T. Linzen, and D. Kush. 2025. Learning filler-gap dependencies with neural language models. JML, 144:104663. [find](https://scholar.google.com/scholar?q=A%2E+Kobzeva%2C+S%2E+Arehalli%2C+T%2E+Linzen%2C+and+D%2E+Kush%2E+2025%2E+Learning+filler%2Dgap+dependencies+with+neural+language+models%2E+JML%2C+144%3A104663%2E) 4. N. Nanda and J. Bloom. 2022. Transformerlens. [https://github.com/TransformerLensOrg/](https://github.com/TransformerLensOrg/) TransformerLens. [↗](https://github.com/TransformerLensOrg/) 5. A. Ritchart, G. Goodall, and M. Garellek. 2016. Prosody and the that-trace effect. In WCCFL 33, pages 320–328. Cascadilla Proceedings Project. [find](https://scholar.google.com/scholar?q=A%2E+Ritchart%2C+G%2E+Goodall%2C+and+M%2E+Garellek%2E+2016%2E+Prosody+and+the+that%2Dtrace+effect%2E+In+WCCFL+33%2C+pages+320%E2%80%93328%2E+Cascadilla+Proceedings+Project%2E) 6. E. Wilcox, R. Futrell, and R. Levy. 2024. Using computational models to test syntactic learnability. LI, 55(4):805–848. [find](https://scholar.google.com/scholar?q=E%2E+Wilcox%2C+R%2E+Futrell%2C+and+R%2E+Levy%2E+2024%2E+Using+computational+models+to+test+syntactic+learnability%2E+LI%2C+55%284%29%3A805%E2%80%93848%2E)