--- title: "Why is unsupervised alignment of English embeddings from" description: "The authors suspect that differing inductive biases create a loss surface full of local optima, which traps… 6 references, cited by 4 papers here." authors: ["Mareike Hartmann", "Yova Kementchedjhieva", "Anders Søgaard"] year: 2018 venue: "Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing" source: "https://aclanthology.org/D18-1056/" doi: "10.18653/v1/d18-1056" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/hartmann-2018-why-is-unsupervised-alignment-english/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/hartmann-2018-why-is-unsupervised-alignment-english/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/6/) / Why is unsupervised alignment of English embeddings from different algorithms so hard? # Why is unsupervised alignment of English embeddings from different algorithms so hard? **Authors:** [Mareike Hartmann](https://md.research.elamur.ai/free/en/authors/mareike-hartmann/), [Yova Kementchedjhieva](https://md.research.elamur.ai/free/en/authors/yova-kementchedjhieva/) and [Anders Søgaard](https://md.research.elamur.ai/free/en/authors/anders-sogaard/) · **Year:** 2018 · **Venue:** [Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2018](https://md.research.elamur.ai/free/en/venues/emnlp/2018/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/D18-1056/) · [PDF](https://aclanthology.org/D18-1056.pdf) · [DOI 10.18653/v1/d18-1056](https://doi.org/10.18653/v1/d18-1056) · [OpenAlex](https://openalex.org/W2891231879) **Citations:** 10 citations (OpenAlex, 2026-10-14) · 4 in our library, of the 149,439 papers whose bibliographies we read ## Summary The authors ask when unsupervised bilingual dictionary induction can work at all. To remove differences between languages, they align English with English. They train 300-dimensional embeddings with five algorithms (Hyperwords-SVD, Hyperwords-SGNS, CBOW, GloVe, FastText) on two disjoint 10% samples of a March 2018 English Wikipedia dump. They then align the spaces with MUSE, a GAN with a linear generator followed by Procrustes refinement, and compare against supervised Procrustes alignment that uses 5000 seed words. The result is stark. Pairs built with the same algorithm align almost perfectly. Pairs built with different algorithms reach zero precision, even when trained on the same text. Supervised Procrustes still maps those failing pairs almost perfectly, so a linear map does exist. Normalization, centering and hyper-parameter changes do not help. The authors suspect that differing inductive biases create a loss surface full of local optima, which traps the discriminator. They present this as an open challenge for the stability of adversarial alignment and unsupervised MT. ## Key points - **Data:** Each Wikipedia sample has over 400 million tokens. The vocabularies have 463,576 and 528,556 words, with 351,858 shared. - **Evaluation and main results:** Alignment is scored by Precision@1 on the 1500 test words from Bojanowski et al. (2017). - Same-algorithm pairs score 0.992–0.997. - Every cross-algorithm pair scores 0.000 in the unsupervised setting. - Supervised Procrustes reaches 0.967–0.999 on those same cross-algorithm pairs. - Precision@10 shows the same pattern. - **MUSE settings:** The defaults were used. The discriminator has 2 fully connected layers of 2048 units and is trained for 5 epochs of 1,000,000 iterations, with 5 discriminator steps per iteration and batch size 32. The generator is initialized as the identity matrix. - **Embedding geometry:** Average inner product with the mean vector differs strongly across algorithms: | Algorithm | Avg. inner product with mean | |---|---| | Hyperwords-SVD | 0.0032 | | Hyperwords-SGNS | 0.0002 | | GloVe | 0.0002 | | FastText | 0.2988 | | CBOW | 4.2985 | However, normalizing and/or centering the embeddings still leaves cross-algorithm P@1 at 0. - **Extra analyses:** - A learning curve for FastText (evaluated on 878 shared words) shows how much data a same-algorithm alignment needs. - Discriminator loss curves show two distinct behaviours. Same-algorithm runs display a genuine min-max contest. Cross-algorithm runs see the discriminator loss collapse early and stay stuck. ## Interesting ideas - Pairing the unsupervised method with a supervised upper bound on the same pair separates "no solution exists" from "the optimizer cannot find it," which is a cleaner diagnosis than reporting a failure alone. - Aligning a language with itself removes linguistic variation and gives a control condition where success should be trivial, so any failure exposes the method's own weakness. - Ruling out cheap explanations one by one (hyper-parameters, normalization, centering, initialization) before proposing a deep cause keeps the final claim honest. - Watching training dynamics, here whether two adversaries keep pushing each other or one collapses early, is a fast health signal for a run that might otherwise look merely "bad." - A single geometric statistic, the average inner product with the mean vector, quickly reveals how differently two representation spaces are shaped. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (learn): When a self-induced repair rule or silver-data learner fails on a phenomenon, first check whether a supervised fit on gold examples succeeds, so search failures are not mistaken for missing signal. - **I2** (learn): Run Mova's agreement and evaluation pipelines on deliberately self-identical inputs, such as one treebank split two ways, as a negative or positive control before trusting cross-source comparisons. - **I3** (learn): In error analysis of the witness parsers, write down and test the trivial explanations for a disagreement before attributing it to a real linguistic ambiguity. - **I4** (tree): Log loss trajectories of gparse training runs and flag runs whose curves flatten abnormally early, as a cheap sign of a stuck optimizer rather than a data problem. - **I5** (world): When comparing compiled knowledge vectors or ontology embeddings from different sources (WordNet, Wikidata), compute such summary statistics first to anticipate whether they can be mapped onto each other. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [University of Copenhagen](https://md.research.elamur.ai/free/en/universities/university-of-copenhagen/) — University, Copenhagen, Denmark · 116 works · 15,292 citations 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works 🧭 Topics (1): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,379 works 💡 Concepts (1): [💡 Latent semantic indexing](https://md.research.elamur.ai/free/en/concepts/latent-semantic-indexing/) · 29 works ## Cite Cite as: Mareike Hartmann, Yova Kementchedjhieva and Anders Søgaard. 2018. Why is unsupervised alignment of English embeddings from different algorithms so hard?. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{hartmann2018why, title = {{Why is unsupervised alignment of English embeddings from different algorithms so hard?}}, author = {Mareike Hartmann, Yova Kementchedjhieva and Anders Søgaard}, booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing}, year = {2018}, doi = {10.18653/v1/d18-1056}, url = {https://doi.org/10.18653/v1/d18-1056} } ``` - [← previous Universal Dependencies and Quantitative Typological Trends. A Case Study on Word Order](https://md.research.elamur.ai/free/en/papers/alzetta-2018-universal-dependencies-quantitative-typological-trends/) - [next → YNU-HPCC at SemEval-2018 Task 3: Ensemble Neural Network Models for Irony Detection on Twitter](https://md.research.elamur.ai/free/en/papers/peng-2018-ynu-hpcc-semeval-2018-task/) - [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/6/) ## References 6 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Martin Arjovsky and Leon Bottou. 2017. Towards principled methods for training generative adversarial networks. In ICLR. [↗](https://doi.org/10.48550/arxiv.1701.04862) 2. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018. Generalizing and improving bilingual word embedding mappings with a multi-step framework of linear transformations. In Proceedings of AAAI. [find](https://scholar.google.com/scholar?q=Mikel+Artetxe%2C+Gorka+Labaka%2C+and+Eneko+Agirre%2E+2018%2E+Generalizing+and+improving+bilingual+word+embedding+mappings+with+a+multi%2Dstep+framework+of+linear+transformations%2E+In+Proceedings+of+AAAI%2E) 3. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. [Enriching word vectors with subword information](https://md.research.elamur.ai/free/en/papers/bojanowski-2017-enriching-word-vectors-subword-information/). Transactions of the Association for Computational Linguistics 5:135–146. [↗](https://doi.org/10.1162/tacl_a_00051) 4. Alexis Conneau, Guillaume Lample, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018. Word Translation Without Parallel Data. In Proceedings of ICLR. [↗](https://doi.org/10.48550/arxiv.1710.04087) 5. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David WardeFarley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial networks. In Proceedings of NIPS. [↗](https://doi.org/10.48550/arxiv.1406.2661) 6. Omer Levy, Yoav Goldberg, and Ido Dagan. 2015. [Improving distributional similarity with lessons learned from word embeddings](https://md.research.elamur.ai/free/en/papers/levy-2015-improving-distributional-similarity-lessons-learned/). TACL 3:211–225. [↗](https://doi.org/10.1162/tacl_a_00134) *Updated 1 November 2026*