--- title: "Infusing Finetuning with Semantic Dependencies — Wu 2021" description: "Building on this, they propose SIFT (semantics-infused finetuning). 81.7 LAS. 5 references, cited by 23 papers here." authors: ["Zhaofeng Wu", "Hao Peng", "Noah A. Smith"] year: 2021 venue: "Transactions of the Association for Computational Linguistics" source: "https://aclanthology.org/2021.tacl-1.14/" doi: "10.1162/tacl_a_00363" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/wu-2021-infusing-finetuning-semantic-dependencies/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wu-2021-infusing-finetuning-semantic-dependencies/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/5/) / Infusing Finetuning with Semantic Dependencies # Infusing Finetuning with Semantic Dependencies **Authors:** [Zhaofeng Wu](https://md.research.elamur.ai/free/en/authors/zhaofeng-wu/), [Hao Peng](https://md.research.elamur.ai/free/en/authors/hao-peng/) and [Noah A. Smith](https://md.research.elamur.ai/free/en/authors/noah-a-smith/) · **Year:** 2021 · **Venue:** [Transactions of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/tacl/) ([Transactions of the Association for Computational Linguistics (TACL) 2021](https://md.research.elamur.ai/free/en/venues/tacl/2021/)) · **Publisher:** [The MIT Press](https://md.research.elamur.ai/free/en/publishers/the-mit-press/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2021.tacl-1.14/) · [PDF](https://aclanthology.org/2021.tacl-1.14.pdf) · [DOI 10.1162/tacl\_a\_00363](https://doi.org/10.1162/tacl_a_00363) · [OpenAlex](https://openalex.org/W3111186274) **Citations:** 12 citations (OpenAlex, 2026-10-14) · [23](https://md.research.elamur.ai/free/en/papers/wu-2021-infusing-finetuning-semantic-dependencies/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Pretrained language models clearly encode a lot of syntax. Whether they also capture meaning, specifically predicate-argument structure, is less clear. The authors test this with a new full-graph probe for semantic dependency parsing on the DM formalism (CoNLL 2015 data). They compare it with a syntactic probe on the Penn Treebank converted to Stanford Dependencies. For RoBERTa and BERT, a linear probe over frozen representations falls much further below a finetuned ceiling parser on DM than on syntax. This suggests semantic structure stays largely implicit in these models. Building on this, they propose SIFT (semantics-infused finetuning). Sentences are first parsed into DM graphs. A relational graph convolutional network (RGCN) then encodes these graphs alongside RoBERTa during task finetuning. A cheaper SIFT-Light variant needs no parser at inference time. On GLUE, SIFT consistently beats both base and large RoBERTa baselines. The gains are biggest when finetuning data is scarce. The work shows that general-purpose linguistic supervision can still add value on top of pretraining. ## Key points - **Probing data and gaps:** PTB SD has 39,832 training and 2,416 test sentences; DM has 33,964 and 1,410. - With RoBERTa-base, the probe drops from 95.2 to 81.7 LAS on syntax (−14.2% relative). - On DM it drops from 94.2 to 70.7 labeled F1 (−24.9% relative). - With RoBERTa-large the gaps are −18.5% for syntax and −28.3% for DM. - **DM formalism:** DM is derived from MRS/HPSG. It represents a sentence as a labeled directed graph with 59 relation labels. Unlike most syntactic trees, it can skip semantically empty words. - **GLUE averages (base size):** SIFT 87.0, SIFT-Light 86.7, RoBERTa 86.4. SIFT also outperforms other ways of injecting semantics: - GCN 86.8 - GAT 86.5 - Hidden 86.4 - Scaffold 85.5 Modeling relation labels helps, though GCN beats SIFT on CoLA. - **Low-data MNLI:** with only 392 training examples (0.1%), SIFT improves on RoBERTa by 2.5 points in-domain (58.7 → 61.2) and 3.2 points out-of-domain (60.5 → 63.7). With full data the gain is just 0.2–0.4. - **Code:** available at github.com/ZhaofengWu/SIFT. Diagnostic analyses use HANS and the GLUE diagnostic set. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 353 works · 200,043 citations - [Allen Institute for Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/allen-institute-for-artificial-intelligence/) — Research institute, Seattle, United States · 275 works · 97,634 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works ## Cite Cite as: Zhaofeng Wu, Hao Peng and Noah A. Smith. 2021. Infusing Finetuning with Semantic Dependencies. Transactions of the Association for Computational Linguistics. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @article{wu2021infusing, title = {{Infusing Finetuning with Semantic Dependencies}}, author = {Zhaofeng Wu, Hao Peng and Noah A. Smith}, journal = {Transactions of the Association for Computational Linguistics}, year = {2021}, doi = {10.1162/tacl\_a\_00363}, url = {https://doi.org/10.1162/tacl_a_00363} } ``` - [← previous ArCOV19-Rumors: Arabic COVID-19 Twitter Dataset for Misinformation Detection](https://md.research.elamur.ai/free/en/papers/haouari-2021-arcov19-rumors-arabic-covid-19/) - [next → Wino-X: Multilingual Winograd Schemas for Commonsense Reasoning and Coreference Resolution](https://md.research.elamur.ai/free/en/papers/emelin-2021-wino-x-multilingual-winograd-schemas/) - [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/5/) ## References 5 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Omri Abend and Ari Rappoport. 2013. [Universal conceptual cognitive annotation (UCCA)](https://md.research.elamur.ai/free/en/papers/abend-2013-universal-conceptual-cognitive-annotation-ucca/). In Proceedings of ACL. 2. Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. [Fine-grained analysis of sentence embeddings using auxiliary prediction tasks](https://md.research.elamur.ai/free/en/papers/adi-2016-fine-grained-analysis-sentence-embeddings/). In Proceedings of ICLR. 3. Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998. [The Berkeley FrameNet project](https://md.research.elamur.ai/free/en/papers/baker-1998-berkeley-framenet/). In Proceedings of ACL. DOI: https://doi.org/10.3115/980845.980860 [↗](https://doi.org/10.3115/980845.980860) 4. Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, and Danilo Giampiccolo. 2006. The second PASCAL recognising textual entailment challenge. Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment. [find](https://scholar.google.com/scholar?q=Roy+Bar%2DHaim%2C+Ido+Dagan%2C+Bill+Dolan%2C+Lisa+Ferro%2C+and+Danilo+Giampiccolo%2E+2006%2E+The+second+PASCAL+recognising+textual+entailment+challenge%2E+Proceedings+of+the+Second+PASCAL+Challenges+Workshop+on+Recog) 5. Jasmijn Bastings, Ivan Titov, Wilker Aziz, Diego Marcheggiani, and Khalil Sima’an. 2017. [Graph convolutional encoders for syntax-aware neural machine translation](https://md.research.elamur.ai/free/en/papers/bastings-2017-graph-convolutional-encoders-syntax-aware/). In Proceedings of EMNLP. DOI: https://doi.org/10 .18653/v1/D17-1209 \#\# 8 Conclusion *Updated 30 October 2026*