--- title: "Comparative Analysis of Document-Level Embedding Methods" description: "The work offers a small, transparent template for comparing representations on stylistically difficult text. 21 references." authors: ["Klara Kramer"] year: 2024 venue: "arXiv:2412.17552" source: "https://arxiv.org/abs/2412.17552" doi: "10.48550/arxiv.2412.17552" arxiv: "2412.17552" licence: "CC BY-NC-SA 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by-nc-sa/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/kramer-2024-comparative-analysis-document-level-embedding/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kramer-2024-comparative-analysis-document-level-embedding/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/13/) / Comparative Analysis of Document-Level Embedding Methods for Similarity Scoring on Shakespeare Sonnets and Taylor Swift Lyrics # Comparative Analysis of Document-Level Embedding Methods for Similarity Scoring on Shakespeare Sonnets and Taylor Swift Lyrics **Authors:** Klara Kramer · **Year:** 2024 · **Venue:** arXiv:2412.17552 · **Licence:** [CC BY-NC-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-nc-sa/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2412.17552) · [PDF](https://arxiv.org/pdf/2412.17552) · [DOI 10.48550/arxiv.2412.17552](https://doi.org/10.48550/arxiv.2412.17552) · [arXiv 2412.17552](https://arxiv.org/abs/2412.17552) · [OpenAlex](https://openalex.org/W4405768198) **Citations:** 1 citations (OpenAlex, 2026-10-14) ## Summary The paper asks how the choice of document representation changes cosine similarity scores when texts come from two very different genres. Krämer takes Shakespeare's 154 sonnets and an equally sized random sample of Taylor Swift songs. Each text is lowercased and tokenised, with stopwords and punctuation removed. Every document is then embedded in three ways: TF-IDF weights, the mean of its Word2Vec word vectors, and BERT embeddings. Pairwise cosine matrices are computed for each corpus, for a merged corpus, and for a cross-only "Distinct" matrix that holds only sonnet–song pairs. Four hypotheses about mean scores are tested with one-way ANOVA, Tukey HSD and Mann-Whitney tests. TF-IDF produces low scores overall, with its highest values on the Swift lyrics. Averaged Word2Vec stays close to the ceiling everywhere, including across genres. BERT does not rate the sonnets as more alike than the other methods do, which the author attributes to a lack of domain-specific fine-tuning. The work offers a small, transparent template for comparing representations on stylistically difficult text. Its numbers also show that a mean similarity score on its own can be hard to interpret. ## Key points - Data come from GitHub repositories by Marquez (sonnets) and Finch (lyrics). From 232 Swift songs, 154 were drawn at random to match the 154 sonnets. - Exploratory statistics: the Shakespeare set has 17,507 words, 3,027 types and lexical diversity 0.1729. The Swift set has 56,932 words, 3,734 types and diversity 0.0656. - Mean similarities by method: - Averaged Word2Vec: 0.9539 on Shakespeare, 0.9993 on Swift, 0.9998 on Combined, 0.9997 on Distinct. - TF-IDF: 0.1157, 0.1486, 0.1075 and 0.0734 on the same four sets. - BERT: 0.9207, 0.9088, 0.7983 and 0.6820. - Hypothesis outcomes: - TF-IDF scoring Swift highest was supported (ANOVA p \< 0.001, confirmed by Tukey HSD). - Word2Vec scoring Swift above Shakespeare was not supported (Mann-Whitney p ≈ 1). - BERT scoring the sonnets highest was rejected. - Words missing from a model's vocabulary are treated as zero vectors. Cosine similarity is computed with sklearn's pairwise function, and it is the only metric used; the author lists this as a limitation. ## Interesting ideas - Building a separate cross-only matrix (rows from one corpus, columns from the other) keeps within-corpus pairs from inflating a cross-domain average. - The paper's own table shows averaged Word2Vec at about 0.95–0.9998 on every set, so such a score barely separates related pairs from unrelated ones; a measure needs a spread check before its numbers mean anything. - Each prediction is phrased as a null hypothesis and checked with a test suited to the comparison (ANOVA plus Tukey for three groups, Mann-Whitney for two), instead of eyeballing averages. - The extreme pairs are read by hand: the most similar sonnets share a theme of unreturned love, while the least similar songs differ in tone (struggle versus joy). This explains what each representation is actually responding to. - A quick type/token lexical diversity profile per corpus (0.17 versus 0.07 here) makes genre differences visible before any modelling. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (learn): when scoring a new parser or frame extractor, report agreement on cross-genre pairs as its own block, so in-domain matches cannot hide domain transfer failures. - **I2** (learn): any similarity or confidence signal used to accept silver data should be tested against deliberately unrelated pairs as a negative control, so a saturated signal is rejected before it is trusted. - **I3** (learn): comparisons between parser versions or LLM teacher tiers could follow the same pattern, attaching a significance test to each claimed improvement. - **I4** (frames): inspecting the most and least similar frame graphs could show whether tone and stance (speech acts, irony) are captured or lost, guiding where prag needs extra features. - **I5** (tokens): computing this profile on each new text source would flag archaic or highly repetitive input where the lemmatiser and tagger are likely to struggle. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works ## Cite Cite as: Klara Kramer. 2024. Comparative Analysis of Document-Level Embedding Methods for Similarity Scoring on Shakespeare Sonnets and Taylor Swift Lyrics. arXiv:2412.17552. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{kramer2024comparative, title = {{Comparative Analysis of Document-Level Embedding Methods for Similarity Scoring on Shakespeare Sonnets and Taylor Swift Lyrics}}, author = {Klara Kramer}, year = {2024}, doi = {10.48550/arxiv.2412.17552}, eprint = {2412.17552}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2412.17552} } ``` - [← previous Communicating with Speakers and Listeners of Different Pragmatic Levels](https://md.research.elamur.ai/free/en/papers/naszadi-2024-communicating-speakers-listeners-different-pragmatic/) - [next → Computational Methods for the Analysis of Complementizer Variability in Language and Literature: The Case of Hebrew "she-" and "ki"](https://md.research.elamur.ai/free/en/papers/shmidman-2024-computational-methods-analysis-complementizer-variability/) - [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/13/) ## References 21 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Refat Aljumily. Hierarchical and non-hierarchical linear and non-linear clustering methods to “shakespeare authorship question”. Social Sciences, 4(3):758–799, 2015. [find](https://scholar.google.com/scholar?q=Refat+Aljumily%2E+Hierarchical+and+non%2Dhierarchical+linear+and+non%2Dlinear+clustering+methods+to+%E2%80%9Cshakespeare+authorship+question%E2%80%9D%2E+Social+Sciences%2C+4%283%29%3A758%E2%80%93799%2C+2015%2E) 2. Ahmed Shamsul Arefin, Renato Vimieiro, Carlos Riveros, Hugh Craig, and Pablo Moscato. An information theoretic clustering approach for unveiling authorship affinities in shakespearean era plays and poems. PloS one, 9(10):e111445, 2014. [↗](https://doi.org/10.1371/journal.pone.0111445) 3. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. [Enriching word vectors with subword information](https://md.research.elamur.ai/free/en/papers/bojanowski-2017-enriching-word-vectors-subword-information/). Transactions of the association for computational linguistics, 5:135– 146, 2017. [↗](https://doi.org/10.1162/tacl_a_00051) 4. Derek Finch. Corpus-of-taylor-swift. Corpus-of-Taylor-Swift/blob/main/, 2024. [find](https://scholar.google.com/scholar?q=Derek+Finch%2E+Corpus%2Dof%2Dtaylor%2Dswift%2E+Corpus%2Dof%2DTaylor%2DSwift%2Fblob%2Fmain%2F%2C+2024%2E) 5. Faruq Ahmad Kendong, Afifah Sakinah Daud, and Aeisha Joharry. A corpus-driven analysis of taylor swift’s song lyrics. International Journal of Modern Languages and Applied Linguistics, 7(2):59–82, 2023. [find](https://scholar.google.com/scholar?q=Faruq+Ahmad+Kendong%2C+Afifah+Sakinah+Daud%2C+and+Aeisha+Joharry%2E+A+corpus%2Ddriven+analysis+of+taylor+swift%E2%80%99s+song+lyrics%2E+International+Journal+of+Modern+Languages+and+Applied+Linguistics%2C+7%282%29%3A59%E2%80%9382%2C+202) 6. Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of naacL-HLT, volume 1, page 2. Minneapolis, Minnesota, 2019. 7. Peter Knees and Markus Schedl. A survey of music similarity and recommendation from music context data. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 10(1):1–21, 2013. [↗](https://doi.org/10.1145/2542205.2542206) 8. Moshe Koppel and Shachar Seidman. Detecting pseudepigraphic texts using novel similarity measures. Digital Scholarship in the Humanities, 33(1):72–81, 2018. [↗](https://doi.org/10.1093/llc/fqx011) 9. Quoc Le and Tomas Mikolov. [Distributed representations of sentences and documents](https://md.research.elamur.ai/free/en/papers/le-2014-distributed-representations-sentences-documents/). In International conference on machine learning, pages 1188–1196. PMLR, 2014. [↗](https://doi.org/10.48550/arxiv.1405.4053) 10. Aaron Marquez. Generate-shakespeare-sonnets. Generate-Shakespeare-Sonnets/blob/master/, 2017. [find](https://scholar.google.com/scholar?q=Aaron+Marquez%2E+Generate%2Dshakespeare%2Dsonnets%2E+Generate%2DShakespeare%2DSonnets%2Fblob%2Fmaster%2F%2C+2017%2E) 11. James H Martin and Daniel Jurafsky. Speech and Language Processing. Pearson Prentice Hall, 2024. [find](https://scholar.google.com/scholar?q=James+H+Martin+and+Daniel+Jurafsky%2E+Speech+and+Language+Processing%2E+Pearson+Prentice+Hall%2C+2024%2E) 12. N Reimers. [Sentence-bert: Sentence embeddings using siamese bert-networks](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/). arXiv preprint arXiv:1908.10084, 2019. [↗](https://arxiv.org/abs/1908.10084) 13. Gerard Salton and Chris Buckley. Approaches to text retrieval for structured documents. Technical report, Cornell University, 1990. [https://github.com/sagesolar/https://github.com/enerrio/REFERENCES](https://github.com/sagesolar/https://github.com/enerrio/REFERENCES) [↗](https://github.com/sagesolar/https://github.com/enerrio/REFERENCES) 14. Omid Shahmirzadi, Adam Lugowski, and Kenneth Younge. Text similarity in vector space models: a comparative study. In 2019 18th IEEE international conference on machine learning and applications (ICMLA), pages 659–666. IEEE, 2019. [find](https://scholar.google.com/scholar?q=Omid+Shahmirzadi%2C+Adam+Lugowski%2C+and+Kenneth+Younge%2E+Text+similarity+in+vector+space+models%3A+a+comparative+study%2E+In+2019+18th+IEEE+international+conference+on+machine+learning+and+applications+%28ICMLA) 15. William Shakespeare. Complete Works of W. Shakespeare. WP Nimmo, 1864. [find](https://scholar.google.com/scholar?q=William+Shakespeare%2E+Complete+Works+of+W%2E+Shakespeare%2E+WP+Nimmo%2C+1864%2E) 16. Karen Sparck Jones. A statistical interpretation of term specificity and its application in retrieval. Journal of documentation, 28(1):11–21, 1972. [↗](https://doi.org/10.1108/eb026526) 17. Taylor Swift. 1989 (Taylor’s Version). Republic Records, New York City, 2023. [find](https://scholar.google.com/scholar?q=Taylor+Swift%2E+1989+%28Taylor%E2%80%99s+Version%29%2E+Republic+Records%2C+New+York+City%2C+2023%2E) 18. Greg Corrado Tomas Mikolov, Kai Chen and Jeffrey Dean. [Efficient estimation of word representations in vector space](https://md.research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/). ICLR Workshop, 2013. [↗](https://doi.org/10.48550/arxiv.1301.3781) 19. Jiapeng Wang and Yihong Dong. Measurement of text similarity: a survey. Information, 11(9):421, 2020. [↗](https://doi.org/10.3390/info11090421) 20. Zhibo Wang, Long Ma, and Yanqing Zhang. A novel method for document summarization using word2vec. In 2016 IEEE 15th International Conference on Cognitive Informatics & Cognitive Computing (ICCI* CC), pages 523–529. IEEE, 2016. [find](https://scholar.google.com/scholar?q=Zhibo+Wang%2C+Long+Ma%2C+and+Yanqing+Zhang%2E+A+novel+method+for+document+summarization+using+word2vec%2E+In+2016+IEEE+15th+International+Conference+on+Cognitive+Informatics+%26+Cognitive+Computing+%28ICCI%2A+CC%29%2C+) 21. Kenneth A Younge and Jeffrey M Kuhn. Patent-to-patent similarity: A vector space model. Available at SSRN 2709238, 2016. [↗](https://doi.org/10.2139/ssrn.2709238) *Updated 1 November 2026*