---
title: "Diachronic Usage Relatedness (DURel) — Schlechtweg 2018"
description: "Computational models of lexical semantic change are hard to evaluate because no language had a reliable benchmark beyond a few hand-picked examples."
authors: ["Dominik Schlechtweg", "Sabine Schulte im Walde", "Stefanie Eckmann"]
year: 2018
venue: "Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)"
source: "https://aclanthology.org/N18-2027/"
doi: "10.18653/v1/n18-2027"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/schlechtweg-2018-diachronic-usage-relatedness-durel-framework/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/schlechtweg-2018-diachronic-usage-relatedness-durel-framework/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/2/) / Diachronic Usage Relatedness (DURel): A Framework for the Annotation of Lexical Semantic Change

# Diachronic Usage Relatedness (DURel): A Framework for the Annotation of Lexical Semantic Change

**Authors:** [Dominik Schlechtweg](https://md.research.elamur.ai/free/en/authors/dominik-schlechtweg/), [Sabine Schulte im Walde](https://md.research.elamur.ai/free/en/authors/sabine-schulte-im-walde/), Stefanie Eckmann · **Year:** 2018 · **Venue:** Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/N18-2027/) · [PDF](https://aclanthology.org/N18-2027.pdf) · [DOI 10.18653/v1/n18-2027](https://doi.org/10.18653/v1/n18-2027) · [OpenAlex](https://openalex.org/W2796841076)

**Citations:** 76 citations (openalex-snapshot · 2026-10-14) · [82](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Computational models of lexical semantic change are hard to evaluate because no language had a reliable benchmark beyond a few hand-picked examples. The authors propose DURel, an annotation framework that adapts graded synchronic polysemy annotation to diachronic data. Annotators never define or label word senses. They rate how related two uses of the same word are on a 4-point scale adapted from Brown (2008). Use pairs are drawn from an earlier period, a later period, and a mixed COMPARE group that pairs one use from each. Change is read from shifts in mean relatedness:
* A drop from the earlier to the later period signals a new meaning.
* A rise signals a lost meaning.
* Low COMPARE relatedness signals strong change of either kind.

Applied to German texts from DTA, the framework produced a test set of 1,320 rated use pairs for 22 target words. Inter-annotator agreement was good, and the measures broadly matched the changes reported in the historical literature. The framework is language-independent, so it offers a reusable recipe for building semantic change benchmarks for other languages.

## Key points
- **Data and design.** Five native German speakers, all linguistics students, rated 1,320 use pairs for 22 target words. Each word had 20 pairs in each of three groups: EARLIER (1750–1800), LATER (1850–1900) and COMPARE. Uses came from DTA (Deutsches Textarchiv).
- **Scale.** The scale runs from 4 (Identical) down to 1 (Unrelated), with 0 for "Cannot decide". Each use was shown with its preceding and following sentence.
- **Agreement.** Mean pairwise Spearman's ρ was 0.66, with 0-judgments omitted and pairwise values ranging from 0.57 to 0.68. This is slightly above the 0.55–0.62 reported by Erk et al. (2013) for synchronic data. The two annotators with a historical-linguistics background agreed most closely.
- **Target words.** Targets were chosen from changes documented by Paul (2002) and Osman (1971), checked against the corpus, giving 19 innovative and 9 reductive cases. Anstalt, Anstellung and Vorwort undergo both kinds of change and appear more than once.
- **Measures.**
  - Mean ΔLATER was 0.39 for reductive cases and −0.18 for innovative ones.
  - ΔLATER failed for Presse, whose new meaning became dominant, but COMPARE caught that change.
  - COMPARE wrongly flagged the stably polysemous Feder as changed.
  - Guidelines and data are released at www.ims.uni-stuttgart.de/data/durel/.

## Interesting ideas
- **Rate relatedness instead of assigning senses.** Graded relatedness judgments between two uses sidestep the need for a sense inventory, which is often missing for historical stages or low-resource languages. This would suit building semantic resources for Ukrainian, where fine-grained historical dictionaries may be incomplete.
- **Add a cross-period comparison group.** Pairing one early and one late use measures change directly. It does not rely on the old meaning still being present, which helps when a meaning is fully replaced. The same trick applies to comparing domains or genres, not only time periods.
- **Normalize by baseline polysemy.** The proposed ΔCOMPARE subtracts within-early relatedness from cross-period relatedness, so a word that is simply polysemous is not mistaken for one that changed. This idea transfers to any drift-detection metric that confuses inherent variability with genuine shift.
- **Choose targets with a known spread of effects.** Selecting words whose documented change is visible in the corpus, both weakly and strongly, keeps a test set from collapsing into many near-zero effects. This makes rankings more informative when comparing models.
- **Hide the study's purpose and shuffle everything.** Annotators were not told the study concerned semantic change. Uses within pairs and pairs across groups were randomized, which reduces expectation bias. This is a cheap safeguard for any annotation project measuring differences between conditions.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Word sense disambiguation](https://md.research.elamur.ai/free/en/topics/word-sense-disambiguation/)

## Cited by

Cited by

82 papers of our library cite it (of the 140884 whose bibliographies we read); 5 of them have a page here, 77 not yet.

- [Analysing Lexical Semantic Change with Contextualised Word Representations](https://md.research.elamur.ai/free/en/papers/giulianelli-2020-analysing-lexical-semantic-change-contextualised/)
- [SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection](https://md.research.elamur.ai/free/en/papers/schlechtweg-2020-semeval-2020-task-1-unsupervised/)
- [The LSCD Benchmark: a Testbed for Diachronic Word Meaning Tasks](https://md.research.elamur.ai/free/en/papers/schlechtweg-2026-lscd-benchmark-testbed-diachronic-word/)
- [Transparent Semantic Change Detection with Dependency-Based Profiles](https://md.research.elamur.ai/free/en/papers/tat-2026-transparent-semantic-change-detection-dependency/)
- [AmDi - Ambiguous Words Diachronic Dataset](https://md.research.elamur.ai/free/en/papers/thielen-2026-amdi-ambiguous-words-diachronic-dataset/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{schlechtweg2018diachronic,
  title = {{Diachronic Usage Relatedness (DURel): A Framework for the Annotation of Lexical Semantic Change}},
  author = {Schlechtweg, Dominik and Walde, Sabine Schulte im and Eckmann, Stefanie},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)},
  year = {2018},
  doi = {10.18653/v1/n18-2027},
  url = {https://doi.org/10.18653/v1/n18-2027}
}
```

- [← previous Developing the Bangla RST Discourse Treebank](https://md.research.elamur.ai/free/en/papers/das-2018-developing-bangla-rst-discourse-treebank/)
- [next → Diachronic word embeddings and semantic shifts: a survey](https://md.research.elamur.ai/free/en/papers/kutuzov-2018-diachronic-word-embeddings-semantic-shifts/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/2/)

## References

20 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. D. Bamman and G. Crane. 2011. Measuring historical word sense variation. In Proceedings of the 11th Annual International ACM/IEEE Joint Conference on Digital Libraries. ACM, New York, NY, USA, pages 1–10. [find](https://scholar.google.com/scholar?q=D%2E+Bamman+and+G%2E+Crane%2E+2011%2E+Measuring+historical+word+sense+variation%2E+In+Proceedings+of+the+11th+Annual+International+ACM%2FIEEE+Joint+Conference+on+Digital+Libraries%2E+ACM%2C+New+York%2C+NY%2C+USA%2C+pages+1)
2. J. H. Lau, P. Cook, D. McCarthy, D. Newman, and T. Baldwin. 2012. Word sense induction for novel sense detection. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics. Stroudsburg, PA, USA, pages 591–601. [find](https://scholar.google.com/scholar?q=J%2E+H%2E+Lau%2C+P%2E+Cook%2C+D%2E+McCarthy%2C+D%2E+Newman%2C+and+T%2E+Baldwin%2E+2012%2E+Word+sense+induction+for+novel+sense+detection%2E+In+Proceedings+of+the+13th+Conference+of+the+European+Chapter+of+the+Association+for+C)
3. A. Blank. 1997. Prinzipien des lexikalischen Bedeutungswandels am Beispiel der romanischen Sprachen. Niemeyer, Tübingen. [find](https://scholar.google.com/scholar?q=A%2E+Blank%2E+1997%2E+Prinzipien+des+lexikalischen+Bedeutungswandels+am+Beispiel+der+romanischen+Sprachen%2E+Niemeyer%2C+Tu%CC%88bingen%2E)
4. S. W. Brown. 2008. Choosing sense distinctions for WSD: Psycholinguistic evidence. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies: Short Papers. Stroudsburg, PA, USA, pages 249–252. [find](https://scholar.google.com/scholar?q=S%2E+W%2E+Brown%2E+2008%2E+Choosing+sense+distinctions+for+WSD%3A+Psycholinguistic+evidence%2E+In+Proceedings+of+the+46th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+on+Human+Language+Technolo)
5. N. Osman. 1971. Kleines Lexikon untergegangener Wörter: Wortuntergang seit dem Ende des 18. Jahrhunderts. Beck, München. [find](https://scholar.google.com/scholar?q=N%2E+Osman%2E+1971%2E+Kleines+Lexikon+untergegangener+Wo%CC%88rter%3A+Wortuntergang+seit+dem+Ende+des+18%2E+Jahrhunderts%2E+Beck%2C+Mu%CC%88nchen%2E)
6. H. Paul. 2002. Deutsches Wörterbuch: Bedeu-tungsgeschichte und Aufbau unseres Wortschatzes. Niemeyer, Tübingen, 10. edition. [find](https://scholar.google.com/scholar?q=H%2E+Paul%2E+2002%2E+Deutsches+Wo%CC%88rterbuch%3A+Bedeu%2Dtungsgeschichte+und+Aufbau+unseres+Wortschatzes%2E+Niemeyer%2C+Tu%CC%88bingen%2C+10%2E+edition%2E)
7. P. Cook, J. H. Lau, D. McCarthy, and T. Baldwin. 2014. Novel word-sense identification. In 25th International Conference on Computational Linguistics, Proceedings of the Conference: Technical Papers, August 23-29, 2014, Dublin, Ireland. pages 1624–1635. [find](https://scholar.google.com/scholar?q=P%2E+Cook%2C+J%2E+H%2E+Lau%2C+D%2E+McCarthy%2C+and+T%2E+Baldwin%2E+2014%2E+Novel+word%2Dsense+identification%2E+In+25th+International+Conference+on+Computational+Linguistics%2C+Proceedings+of+the+Conference%3A+Technical+Papers%2C+)
8. E. Rosch and C.B. Mervis. 1975. Family resemblances: Studies in the internal structure of categories. Cognitive Psychology 7:573–605. [find](https://scholar.google.com/scholar?q=E%2E+Rosch+and+C%2EB%2E+Mervis%2E+1975%2E+Family+resemblances%3A+Studies+in+the+internal+structure+of+categories%2E+Cognitive+Psychology+7%3A573%E2%80%93605%2E)
9. Deutsches Textarchiv. 2017. Grundlage für ein Referenzkorpus der neuhochdeutschen Sprache. Herausgegeben von der Berlin-Brandenburgischen Akademie der Wissenschaften. [http://www.deutschestextarchiv.de/](http://www.deutschestextarchiv.de/). [↗](http://www.deutschestextarchiv.de/)
10. D. Schlechtweg, S. Eckmann, E. Santus, S. Schulte im Walde, and D. Hole. 2017. German in flux: Detecting metaphoric change via word entropy. In Proceedings of the 21st Conference on Computational Natural Language Learning. Vancouver, Canada, pages 354–367. [find](https://scholar.google.com/scholar?q=D%2E+Schlechtweg%2C+S%2E+Eckmann%2C+E%2E+Santus%2C+S%2E+Schulte+im+Walde%2C+and+D%2E+Hole%2E+2017%2E+German+in+flux%3A+Detecting+metaphoric+change+via+word+entropy%2E+In+Proceedings+of+the+21st+Conference+on+Computational+Natu)
11. Haim Dubossarsky, Daphna Weinshall, and Eitan Grossman. 2017. Outta control: Laws of semantic change and inherent biases in word representation models. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Copenhagen, Denmark, pages 1147–1156. [find](https://scholar.google.com/scholar?q=Haim+Dubossarsky%2C+Daphna+Weinshall%2C+and+Eitan+Grossman%2E+2017%2E+Outta+control%3A+Laws+of+semantic+change+and+inherent+biases+in+word+representation+models%2E+In+Proceedings+of+the+2017+Conference+on+Empiric)
12. A. Soares da Silva. 1992. Homonı́mia e polissemia: Análise sémica e teoria do campoléxico. In Actas do XIX Congreso Internacional de Lingüı́stica e Filoloxı́a Románicas. Fundación Pedro Barrié de la Maza, La Coruña, volume 2 of Lexicoloxı́a e Met-alexicografı́a, pages 257–287. [find](https://scholar.google.com/scholar?q=A%2E+Soares+da+Silva%2E+1992%2E+Homon%C4%B1%CC%81mia+e+polissemia%3A+Ana%CC%81lise+se%CC%81mica+e+teoria+do+campole%CC%81xico%2E+In+Actas+do+XIX+Congreso+Internacional+de+Lingu%CC%88%C4%B1%CC%81stica+e+Filolox%C4%B1%CC%81a+Roma%CC%81nicas%2E+Fundacio%CC%81n+Pedro+Barrie%CC%81+)
13. K. Erk, D. McCarthy, and N. Gaylord. 2009. Investigations on word senses and word usages. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 1. Stroudsburg, PA, USA, pages 10–18. [find](https://scholar.google.com/scholar?q=K%2E+Erk%2C+D%2E+McCarthy%2C+and+N%2E+Gaylord%2E+2009%2E+Investigations+on+word+senses+and+word+usages%2E+In+Proceedings+of+the+Joint+Conference+of+the+47th+Annual+Meeting+of+the+ACL+and+the+4th+International+Joint+C)
14. David Tuggy. 1993. Ambiguity, polysemy, and vagueness. Cognitive Linguistics 4(3):273–290. [find](https://scholar.google.com/scholar?q=David+Tuggy%2E+1993%2E+Ambiguity%2C+polysemy%2C+and+vagueness%2E+Cognitive+Linguistics+4%283%29%3A273%E2%80%93290%2E)
15. J. Zlatev. 2003. Polysemy or generality? Mu, Mouton de Gruyter, volume 23 of Cognitive Approaches to Lexical Semantics, pages 447–494. [find](https://scholar.google.com/scholar?q=J%2E+Zlatev%2E+2003%2E+Polysemy+or+generality%3F+Mu%2C+Mouton+de+Gruyter%2C+volume+23+of+Cognitive+Approaches+to+Lexical+Semantics%2C+pages+447%E2%80%93494%2E)
16. K. Erk, D. McCarthy, and N. Gaylord. 2013. Measuring word meaning in context. Computational Linguistics 39(3):511–554. [find](https://scholar.google.com/scholar?q=K%2E+Erk%2C+D%2E+McCarthy%2C+and+N%2E+Gaylord%2E+2013%2E+Measuring+word+meaning+in+context%2E+Computational+Linguistics+39%283%29%3A511%E2%80%93554%2E)
17. L. Frermann and M. Lapata. 2016. A Bayesian model of diachronic meaning change. Transactions of the Association for Computational Linguistics 4:31–45. [find](https://scholar.google.com/scholar?q=L%2E+Frermann+and+M%2E+Lapata%2E+2016%2E+A+Bayesian+model+of+diachronic+meaning+change%2E+Transactions+of+the+Association+for+Computational+Linguistics+4%3A31%E2%80%9345%2E)
18. K. Gulordava and M. Baroni. 2011. A distributional similarity approach to the detection of semantic change in the Google Books Ngram corpus. In Proceedings of the Workshop on Geometrical Models of Natural Language Semantics. Stroudsburg, PA, USA, pages 67–71. [find](https://scholar.google.com/scholar?q=K%2E+Gulordava+and+M%2E+Baroni%2E+2011%2E+A+distributional+similarity+approach+to+the+detection+of+semantic+change+in+the+Google+Books+Ngram+corpus%2E+In+Proceedings+of+the+Workshop+on+Geometrical+Models+of+Nat) [our card](https://md.research.elamur.ai/free/en/papers/gulordava-2011-distributional-similarity-approach-detection-semantic/)
19. W. L. Hamilton, J. Leskovec, and D. Jurafsky. 2016. Cultural shift or linguistic drift? Comparing two computational measures of semantic change. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Austin, Texas, pages 2116–2121. [find](https://scholar.google.com/scholar?q=W%2E+L%2E+Hamilton%2C+J%2E+Leskovec%2C+and+D%2E+Jurafsky%2E+2016%2E+Cultural+shift+or+linguistic+drift%3F+Comparing+two+computational+measures+of+semantic+change%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Meth) [our card](https://md.research.elamur.ai/free/en/papers/hamilton-2016-cultural-shift-or-linguistic-drift/)
20. N. Ide and J. Pustejovsky, editors. 2017. Handbook of Linguistic Annotation. Springer, Dordrecht. [find](https://scholar.google.com/scholar?q=N%2E+Ide+and+J%2E+Pustejovsky%2C+editors%2E+2017%2E+Handbook+of+Linguistic+Annotation%2E+Springer%2C+Dordrecht%2E)