---
title: "“Shakespeare in the Vectorian Age” – An evaluation of different"
description: "The paper looks at how to find Shakespeare quotations, both exact and altered, in contemporary fiction using computers. 34 references."
authors: ["Bernhard Liebl", "Manuel Burghardt"]
year: 2020
venue: "Proceedings of the 4th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature"
source: "https://aclanthology.org/2020.latechclfl-1.7/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/liebl-2020-shakespeare-vectorian-age-evaluation-different/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liebl-2020-shakespeare-vectorian-age-evaluation-different/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/10/) / “Shakespeare in the Vectorian Age” – An evaluation of different word embeddings and NLP parameters for the detection of Shakespeare quotes

# “Shakespeare in the Vectorian Age” – An evaluation of different word embeddings and NLP parameters for the detection of Shakespeare quotes

**Authors:** Bernhard Liebl and [Manuel Burghardt](https://md.research.elamur.ai/free/en/authors/manuel-burghardt/) · **Year:** 2020 · **Venue:** [Proceedings of the 4th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature](https://md.research.elamur.ai/free/en/venues/latechclfl/) ([LaTeCH-CLfL 2020](https://md.research.elamur.ai/free/en/venues/latechclfl/2020/)) · **Publisher:** [International Committee on Computational Linguistics (ICCL)](https://md.research.elamur.ai/free/en/publishers/international-committee-on-computational-linguistics-iccl/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.latechclfl-1.7/) · [PDF](https://aclanthology.org/2020.latechclfl-1.7.pdf) · [OpenAlex](https://openalex.org/W3120910537)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
The paper looks at how to find Shakespeare quotations, both exact and altered, in contemporary fiction using computers. The authors built the Vectorian, a search engine that takes a Shakespeare phrase as a query and ranks corpus sentences by how well they align with it. Corpus and query are tagged with spaCy. Token similarities come from word embeddings (fasttext, wnet2vec) and are adjusted by ten explicit parameters covering POS influence, embedding similarity and alignment scoring. A Waterman-Smith-Beyer alignment then scores each sentence pair. The parameters were tuned against a ground truth of known references, and an ablation study checked what each one contributes. The best setups reach about 80% nDCG@50. Optimizing the arithmetic mean gives a configuration that makes heavy use of embeddings and POS cues. Optimizing the harmonic mean switches most of the pipeline off and behaves almost like a verbatim word matcher, which helps on hard queries. The study shows which features actually matter, and that query difficulty should be modelled explicitly.

## Key points
- Preprocessing used spaCy 2.3.2 with en_core_web_lg-2.3.1. The authors note this model ignores Early Modern English, and their embeddings do not handle historical meaning shift (HistWords is left as future work).
- The parameters fall into three groups:
  - POS: XDT, SPW, PMP.
  - Embedding similarity: EMI, ESM (cosine / nicdm / apsynp), IFS, SIF, SIT.
  - Alignment: MLP, SBO.
- wnet2vec was built from WordNet for 58,492 words drawn from 66 novels, but it covers only 27,718 of them.
- After some extra hits were judged correct, nDCG was 81.2% for the arithmetic-mean variant (A) and 78.9% for the harmonic-mean variant (H).
- Results for variant A:
  - The best EMI value was 0.55, so mixing the two embeddings helped.
  - Without mixing, wnet2vec beat fasttext.
  - Cosine slightly beat nicdm, and both clearly beat apsynp.
  - SPW worked best around 0.4–0.5.
  - A full POS-mismatch penalty was optimal.

## Interesting ideas
- Tuning one system for the arithmetic mean and again for the harmonic mean of per-query scores produces two different strategies: one boosts typical queries, the other rescues the worst ones.
- Starting from the best configuration, they varied one parameter at a time and showed that several were effectively no-ops. This makes a tuned pipeline interpretable rather than opaque.
- A POS mismatch penalty can override embedding similarity entirely, so two tokens with different UPOS count as unrelated however close their vectors are.
- Two very different embeddings, one distributional and one derived from the WordNet graph, were merged by taking the maximum of their interpolated similarities. Each one covers the other's gaps.
- Gapped sequence alignment over token similarities catches altered quotes, with substituted words and insertions, while keeping word order meaningful.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Report both means in evaluations, so that a small set of total failures is not hidden behind a good average.
- **I2** (frames): Run the same one-knob-at-a-time ablation on the weights used to score frame and role matches, and drop the weights that change nothing.
- **I3** (answer): When aligning question words to passage tokens, gate lexical similarity by UPOS (and lemma) to avoid spurious matches.
- **I4** (world): Combine WordNet-based relatedness with corpus-based similarity through a max-style blend when linking text words to ontology concepts.
- **I5** (back): Check text → graph → text compressor output against the source with such an alignment, instead of a bag-of-words overlap.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [📚 Literature and Literary Theory](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/literature-and-literary-theory/) · 519 works

🧭 Topics (1): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works

## Cite

Cite as: Bernhard Liebl and Manuel Burghardt. 2020. “Shakespeare in the Vectorian Age” – An evaluation of different word embeddings and NLP parameters for the detection of Shakespeare quotes. Proceedings of the 4th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{liebl2020shakespeare,
  title = {{“Shakespeare in the Vectorian Age” – An evaluation of different word embeddings and NLP parameters for the detection of Shakespeare quotes}},
  author = {Bernhard Liebl and Manuel Burghardt},
  booktitle = {Proceedings of the 4th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature},
  year = {2020},
  url = {https://aclanthology.org/2020.latechclfl-1.7/}
}
```

- [← previous Vom x-qatal zur Informationsstruktur – und retour. Bemerkungen zur Analyse althebräischer Sätze in der Poesie (From x-qatal to information structure – and back again)](https://md.research.elamur.ai/free/en/papers/schiller-2020-x-qatal-information-structure-back/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/10/)

## References

34 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016. [SemEval-2016 Task 1: Semantic Textual Similarity, Monolingual and Cross-Lingual Evaluation](https://md.research.elamur.ai/free/en/papers/agirre-2016-semeval-2016-task-1-semantic/). In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 497–511, San Diego, California. Association for Computational Linguistics.
2. Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. [Optuna: A Next-generation Hyperparameter Optimization Framework](https://md.research.elamur.ai/free/en/papers/akiba-2019-optuna-next-generation-hyperparameter-optimization/). In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2623–2631, Anchorage AK USA, July. ACM.
3. David Bamman and Gregory Crane. 2008. The logic and discovery of textual allusion. In In Proceedings of the 2008 LREC Workshop on Language Technology for Cultural Heritage Data. [find](https://scholar.google.com/scholar?q=David+Bamman+and+Gregory+Crane%2E+2008%2E+The+logic+and+discovery+of+textual+allusion%2E+In+In+Proceedings+of+the+2008+LREC+Workshop+on+Language+Technology+for+Cultural+Heritage+Data%2E)
4. Daniel Bär, Torsten Zesch, and Iryna Gurevych. 2012. Text reuse detection using a composition of text similarity measures. In Proceedings of COLING 2012, pages 167–184, Mumbai, India, December. [find](https://scholar.google.com/scholar?q=Daniel+Ba%CC%88r%2C+Torsten+Zesch%2C+and+Iryna+Gurevych%2E+2012%2E+Text+reuse+detection+using+a+composition+of+text+similarity+measures%2E+In+Proceedings+of+COLING+2012%2C+pages+167%E2%80%93184%2C+Mumbai%2C+India%2C+December%2E)
5. The COLING 2012 Organizing Committee. [find](https://scholar.google.com/scholar?q=The+COLING+2012+Organizing+Committee%2E)
6. Vuk Batanović and Dragan Bojić. 2015. Using Part-of-Speech Tags as Deep Syntax Indicators in Determining Short Text Semantic Similarity. Computer Science and Information Systems, 12(1):1–31, January. [↗](https://doi.org/10.2298/csis131127082b)
7. Maximilian Bryan, Manuel Burghardt, and Johannes Molz. 2020. A computational expedition into the undiscovered country - evaluating neural networks for the identification of hamlet text reuse. Proceedings of the 1st Workshop on Computational Humanities Research (CHR). [find](https://scholar.google.com/scholar?q=Maximilian+Bryan%2C+Manuel+Burghardt%2C+and+Johannes+Molz%2E+2020%2E+A+computational+expedition+into+the+undiscovered+country+%2D+evaluating+neural+networks+for+the+identification+of+hamlet+text+reuse%2E+Proceedi)
8. Marco Büchler, Annette Geßner, Monica Berti, and Thomas Eckart. 2013. Measuring the influence of a work by text re-use. Bulletin of the Institute of Classical Studies. Supplement, pages 63–79. [find](https://scholar.google.com/scholar?q=Marco+Bu%CC%88chler%2C+Annette+Ge%C3%9Fner%2C+Monica+Berti%2C+and+Thomas+Eckart%2E+2013%2E+Measuring+the+influence+of+a+work+by+text+re%2Duse%2E+Bulletin+of+the+Institute+of+Classical+Studies%2E+Supplement%2C+pages+63%E2%80%9379%2E)
9. Manuel Burghardt, Selina Meyer, Stephanie Schmidtbauer, and Johannes Molz. 2019. “The Bard meets the Doctor” – Computergestützte Identifikation intertextueller Shakespearebezüge in der Science Fiction-Serie Dr. Who. Book of Abstracts, DHd. [find](https://scholar.google.com/scholar?q=Manuel+Burghardt%2C+Selina+Meyer%2C+Stephanie+Schmidtbauer%2C+and+Johannes+Molz%2E+2019%2E+%E2%80%9CThe+Bard+meets+the+Doctor%E2%80%9D+%E2%80%93+Computergestu%CC%88tzte+Identifikation+intertextueller+Shakespearebezu%CC%88ge+in+der+Science+Ficti)
10. Neil Coffee, Jean-Pierre Koenig, Shakthi Poornima, Christopher Forstall, Roelant Ossewaarde, and Sarah Jacobson. 2012a. The Tesserae Project: intertextual analysis of Latin poetry. Literary and Linguistic Computing, 28(2):221–228, 07. [↗](https://doi.org/10.1093/llc/fqs033)
11. Neil Coffee, Jean-Pierre Koenig, Shakthi Poornima, Roelant Ossewaarde, Christopher Forstall, and Sarah Jacobson. 2012b. Intertextuality in the digital age. Transactions of the American Philological Association (1974-), pages 383–422. [↗](https://doi.org/10.1353/apa.2012.0010)
12. Christopher Forstall, Neil Coffee, Thomas Buck, Katherine Roache, and Sarah Jacobson. 2015. Modeling the scholars: Detecting intertextuality through enhanced word-level n-gram matching. Digital Scholarship in the Humanities, 30(4):503–515. [↗](https://doi.org/10.1093/llc/fqu014)
13. Greta Franzini, Emily Franzini, Gabriela Rotari, Franziska Pannach, Mahdi Solhdoust, and Marco Büchler. 2017. The digital breadcrumb trail of brothers grimm. Poster at the DATECH conference, Göttingen. [find](https://scholar.google.com/scholar?q=Greta+Franzini%2C+Emily+Franzini%2C+Gabriela+Rotari%2C+Franziska+Pannach%2C+Mahdi+Solhdoust%2C+and+Marco+Bu%CC%88chler%2E+2017%2E+The+digital+breadcrumb+trail+of+brothers+grimm%2E+Poster+at+the+DATECH+conference%2C+Go%CC%88tting)
14. Jean-Gabriel Ganascia, Peirre Glaudes, and Andrea Del Lungo. 2014. Automatic detection of reuses and citations in literary texts. Literary and Linguistic Computing, 29(3):412–421, 06. [↗](https://doi.org/10.1093/llc/fqu020)
15. Marjorie Garber. 2005. Shakespeare After All. Anchor Books. [find](https://scholar.google.com/scholar?q=Marjorie+Garber%2E+2005%2E+Shakespeare+After+All%2E+Anchor+Books%2E)
16. Gérard Genette. 1993. Palimpseste. Die Literatur auf zweiter Stufe. Suhrkamp. [find](https://scholar.google.com/scholar?q=Ge%CC%81rard+Genette%2E+1993%2E+Palimpseste%2E+Die+Literatur+auf+zweiter+Stufe%2E+Suhrkamp%2E)
17. William L. Hamilton, Jure Leskovec, and Dan Jurafsky. 2016. [Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change](https://md.research.elamur.ai/free/en/papers/hamilton-2016-diachronic-word-embeddings-reveal-statistical/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1489–1501, Berlin, Germany. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/p16-1141)
18. Regula Hohl-Trillini. 2019. ’Look thee, I speak play scraps’: Digitally Mapping Intertextuality in Early Modern Drama. Oxford University, Bodleian and Folger Libraries, July. [find](https://scholar.google.com/scholar?q=Regula+Hohl%2DTrillini%2E+2019%2E+%E2%80%99Look+thee%2C+I+speak+play+scraps%E2%80%99%3A+Digitally+Mapping+Intertextuality+in+Early+Modern+Drama%2E+Oxford+University%2C+Bodleian+and+Folger+Libraries%2C+July%2E)
19. Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems, 20(4):422–446, October. [↗](https://doi.org/10.1145/582415.582418)
20. Herve Jegou, Cordelia Schmid, Hedi Harzallah, and Jakob Verbeek. 2010. Accurate Image Search Using the Contextual Dissimilarity Measure. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTEL-LIGENCE, 32(1):10. [find](https://scholar.google.com/scholar?q=Herve+Jegou%2C+Cordelia+Schmid%2C+Hedi+Harzallah%2C+and+Jakob+Verbeek%2E+2010%2E+Accurate+Image+Search+Using+the+Contextual+Dissimilarity+Measure%2E+IEEE+TRANSACTIONS+ON+PATTERN+ANALYSIS+AND+MACHINE+INTEL%2DLIGENCE)
21. Cyril Labbé and Dominique Labbé. 2005. A Tool for Literary Studies: Intertextual Distance and Tree Classification. Literary and Linguistic Computing, 21(3):311–326, 10. [↗](https://doi.org/10.1093/llc/fqi063)
22. Jurij Leskovec, Anand Rajaraman, and Jeffrey D. Ullman. 2020. Mining of Massive Datasets. Cambridge University Press, New York, NY, third edition edition. [↗](https://doi.org/10.1017/9781108684163)
23. Enrique Manjavacas, Brian Long, and Mike Kestemont. 2019. On the Feasibility of Automated Detection of Allusive Text Reuse. arXiv:1905.02973 [cs], May. [↗](https://arxiv.org/abs/1905.02973)
24. Christopher Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Introduction to Information Retrieval. Cambridge University Press, USA. [↗](https://doi.org/10.1017/cbo9780511809071)
25. Julie Maxwell and Kate Rumbold. 2018. Shakespeare and Quotation. Cambridge University Press. [find](https://scholar.google.com/scholar?q=Julie+Maxwell+and+Kate+Rumbold%2E+2018%2E+Shakespeare+and+Quotation%2E+Cambridge+University+Press%2E)
26. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. [Efficient estimation of word representations in vector space](https://md.research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/). arXiv preprint arXiv:1301.3781. [↗](https://arxiv.org/abs/1301.3781)
27. Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. [Advances in Pre-Training Distributed Word Representations](https://md.research.elamur.ai/free/en/papers/mikolov-2017-advances-pre-training-distributed-word/). In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018). [↗](https://doi.org/10.63317/4b3prw5a5tze)
28. Christoph Molnar. 2020. Interpretable Machine Learning. A Guide for Making Black Box Models Explainable. [https://christophm.github.io/interpretable-ml-book/](https://christophm.github.io/interpretable-ml-book/). [↗](https://christophm.github.io/interpretable-ml-book/)
29. Johannes Molz. 2019. A close and distant reading of Shakespearean intertextuality. Ludwig-Maximilians-Universität München, Juli. [↗](https://doi.org/10.5282/edoc.25432)
30. Chakaveh Saedi, António Branco, João António Rodrigues, and João Silva. 2018. WordNet Embeddings. In Proceedings of The Third Workshop on Representation Learning for NLP, pages 122–131, Melbourne, Australia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chakaveh+Saedi%2C+Anto%CC%81nio+Branco%2C+Joa%CC%83o+Anto%CC%81nio+Rodrigues%2C+and+Joa%CC%83o+Silva%2E+2018%2E+WordNet+Embeddings%2E+In+Proceedings+of+The+Third+Workshop+on+Representation+Learning+for+NLP%2C+pages+122%E2%80%93131%2C+Melbourne%2C)
31. Enrico Santus, Hongmin Wang, Emmanuele Chersoni, and Yue Zhang. 2018. A Rank-Based Similarity Metric for Word Embeddings. arXiv:1805.01923 [cs], May. [↗](https://arxiv.org/abs/1805.01923)
32. Walter Scheirer, Christopher Forstall, and Neil Coffee. 2014. The sense of a connection: Automatic tracing of intertextuality by meaning. Digital Scholarship in the Humanities, 31(1):204–217, 10. [↗](https://doi.org/10.1093/llc/fqu058)
33. Ann Taylor, Mitchell Marcus, and Beatrice Santorini. 2003. The Penn Treebank: An Overview. In Nancy Ide, Jean Véronis, and Anne Abeillé, editors, Treebanks, volume 20, pages 5–22. Springer Netherlands, Dordrecht. [↗](https://doi.org/10.1007/978-94-010-0201-1_1)
34. M.S Waterman, T.F Smith, and W.A Beyer. 1976. Some biological sequence metrics. Advances in Mathematics, 20(3):367–387, June. [↗](https://doi.org/10.1016/0001-8708(76)90202-4)

*Updated 1 November 2026*