---
title: "What Kind of Language Is Hard to Language-Model? (2019)"
description: "The authors warn against generalizing from small language samples and offer a reusable tool for… 54 references, cited by 64 papers here."
authors: ["Sabrina J. Mielke", "Ryan Cotterell", "Kyle Gorman", "Brian Roark", "Jason Eisner"]
year: 2019
venue: "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/P19-1491/"
doi: "10.18653/v1/p19-1491"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/mielke-2019-what-kind-language-is-hard/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mielke-2019-what-kind-language-is-hard/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) / What Kind of Language Is Hard to Language-Model?

# What Kind of Language Is Hard to Language-Model?

**Authors:** [Sabrina J. Mielke](https://md.research.elamur.ai/free/en/authors/sabrina-j-mielke/), [Ryan Cotterell](https://md.research.elamur.ai/free/en/authors/ryan-cotterell/), [Kyle Gorman](https://md.research.elamur.ai/free/en/authors/kyle-gorman/), [Brian Roark](https://md.research.elamur.ai/free/en/authors/brian-roark/) and [Jason Eisner](https://md.research.elamur.ai/free/en/authors/jason-eisner/) · **Year:** 2019 · **Venue:** [Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2019](https://md.research.elamur.ai/free/en/venues/acl/2019/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/P19-1491/) · [PDF](https://aclanthology.org/P19-1491.pdf) · [DOI 10.18653/v1/p19-1491](https://doi.org/10.18653/v1/p19-1491) · [OpenAlex](https://openalex.org/W2951412882)

**Citations:** 67 citations (OpenAlex, 2026-10-14) · [64](https://md.research.elamur.ai/free/en/papers/mielke-2019-what-kind-language-is-hard/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper tests whether current language-modeling methods work equally well across languages. It revisits the authors' earlier claim (Cotterell et al., 2018) that rich inflectional morphology makes languages harder to model. That study used the fully parallel Europarl corpus, which covers few languages, most of them related. To go beyond it, the authors fit a paired-sample multiplicative mixed-effects model. The model jointly estimates how much information each aligned sentence ("intent") carries and a difficulty coefficient for each language, and it tolerates missing translations. They train open-vocabulary Char-RNNLM and BPE-RNNLM models on Bible translations from 13 language families (69 languages per the abstract, 62 per the body) and on the 21 Europarl languages. The difficulty estimates are stable across corpora and architectures. However, morphological complexity and WALS typological features do not predict them. Only vocabulary size and, more weakly, sentence length in characters correlate with difficulty. Translationese also proves no easier to model once training data are balanced. The authors warn against generalizing from small language samples and offer a reusable tool for cross-lingual evaluation.

## Key points
- Models: Char-RNNLM, an LSTM using the Merity et al. (2018) implementation with char-PTB settings, and BPE-RNNLM, which uses 0.4|V| BPE merges. Data were split 2/3 train, 1/6 dev and 1/6 test.
- The earlier morphology finding, measured with MCC (morphological counting complexity), could not be reproduced on the larger language sample. No WALS feature showed a tenable correlation with difficulty.
- Word inventory size was the strongest predictor of difficulty. Raw character sequence length was a weaker one. Unigram word entropy added nothing beyond vocabulary size.
- Translationese test: Europarl was split into native and translated sub-languages (42 in total). Native text first looked slightly harder, by 0.027 ± 0.023 across 8 languages. With balanced training on 10 fully parallel Europarl languages, the gap shrank to −0.0044 ± 0.022.
- Main recommendation: typology is a small-data problem. Future cross-linguistic studies should use far more languages, which this method makes feasible.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 353 works · 364,806 citations
- [Johns Hopkins University](https://md.research.elamur.ai/free/en/universities/johns-hopkins-university/) — University, Baltimore, United States · 187 works · 92,194 citations
- [City University of New York](https://md.research.elamur.ai/free/en/universities/city-university-of-new-york/) — University, New York, United States · 13 works · 2,531 citations
- [The Graduate Center, CUNY](https://md.research.elamur.ai/free/en/universities/the-graduate-center-cuny/) — University, New York, United States · 6 works · 182 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works · [🧩 Morphology](https://md.research.elamur.ai/free/en/topics/morphology/) · 277 works

## Cite

Cite as: Sabrina J. Mielke, Ryan Cotterell, Kyle Gorman, Brian Roark and Jason Eisner. 2019. What Kind of Language Is Hard to Language-Model?. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{mielke2019what,
  title = {{What Kind of Language Is Hard to Language-Model?}},
  author = {Sabrina J. Mielke, Ryan Cotterell, Kyle Gorman, Brian Roark and Jason Eisner},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1491},
  url = {https://doi.org/10.18653/v1/p19-1491}
}
```

- [← previous Findings of the WMT 2019 Shared Task on Parallel Corpus Filtering for Low-Resource Conditions](https://md.research.elamur.ai/free/en/papers/koehn-2019-findings-wmt-2019-shared-task/)
- [next → ParaCrawl: Web-scale parallel corpora for the languages of the EU](https://md.research.elamur.ai/free/en/papers/espla-2019-paracrawl-web-scale-parallel-corpora/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/)

## References

54 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+1997%2E+Long+short%2Dterm+memory%2E+Neural+Computation%2C+9%288%29%3A1735%E2%80%931780%2E)
2. Željko Agić, Anders Johannsen, Barbara Plank, Héctor Martı́nez Alonso, Natalie Schluter, and Anders Søgaard. 2016. [Multilingual projection for parsing truly low-resource languages](https://md.research.elamur.ai/free/en/papers/agic-2016-multilingual-projection-parsing-truly-low/). Transactions of the Association for Computational Linguistics, 4:301–312.
3. Christo Kirov, Ryan Cotterell, John Sylak-Glassman, Géraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Arya McCarthy, Sabrina J. Mielke, Sandra Kübler, David Yarowsky, Jason Eisner, and Mans Hulden. 2018. Unimorph 2.0: Universal morphology. In Proceedings of the Ninth Internationa… [find](https://scholar.google.com/scholar?q=Christo+Kirov%2C+Ryan+Cotterell%2C+John+Sylak%2DGlassman%2C+Ge%CC%81raldine+Walther%2C+Ekaterina+Vylomova%2C+Patrick+Xia%2C+Manaal+Faruqui%2C+Arya+McCarthy%2C+Sabrina+J%2E+Mielke%2C+Sandra+Ku%CC%88bler%2C+David+Yarowsky%2C+Jason+Eisner%2C)
4. Mona Baker. 1993. Corpus linguistics and translation studies: Implications and applications. Text and Technology: In Honour of John Sinclair, pages 233– 250. [find](https://scholar.google.com/scholar?q=Mona+Baker%2E+1993%2E+Corpus+linguistics+and+translation+studies%3A+Implications+and+applications%2E+Text+and+Technology%3A+In+Honour+of+John+Sinclair%2C+pages+233%E2%80%93+250%2E)
5. Emily M. Bender. 2009. Linguistically naı̈ve != language independent: Why NLP needs linguistic typology. In EACL 2009 Workshop on the Interaction between Linguistics and Computational Linguistics, pages 26–32. [find](https://scholar.google.com/scholar?q=Emily+M%2E+Bender%2E+2009%2E+Linguistically+na%C4%B1%CC%88ve+%21%3D+language+independent%3A+Why+NLP+needs+linguistic+typology%2E+In+EACL+2009+Workshop+on+the+Interaction+between+Linguistics+and+Computational+Linguistics%2C+pag)
6. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In MT Summit, pages 79–86. [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2E+2005%2E+Europarl%3A+A+parallel+corpus+for+statistical+machine+translation%2E+In+MT+Summit%2C+pages+79%E2%80%9386%2E)
7. Charles J. Kowalski. 1972. On the effects of non-normality on the distribution of the sample product-moment correlation coefficient. Journal of the Royal Statistical Society. Series C (Applied Statistics), 21(1):1–12. [find](https://scholar.google.com/scholar?q=Charles+J%2E+Kowalski%2E+1972%2E+On+the+effects+of+non%2Dnormality+on+the+distribution+of+the+sample+product%2Dmoment+correlation+coefficient%2E+Journal+of+the+Royal+Statistical+Society%2E+Series+C+%28Applied+Statist)
8. Yoav Benjamini and Yosef Hochberg. 1995. Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society. Series B (Methodological), 57(1):289–300. [find](https://scholar.google.com/scholar?q=Yoav+Benjamini+and+Yosef+Hochberg%2E+1995%2E+Controlling+the+false+discovery+rate%3A+A+practical+and+powerful+approach+to+multiple+testing%2E+Journal+of+the+Royal+Statistical+Society%2E+Series+B+%28Methodological)
9. Taku Kudo. 2018. [Subword regularization: Improving neural network translation models with multiple subword candidates](https://md.research.elamur.ai/free/en/papers/kudo-2018-subword-regularization-improving-neural-network/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66–75, Melbourne, Australia.
10. Bob Carpenter, Andrew Gelman, Matthew Hoffman, Daniel Lee, Ben Goodrich, Michael Betancourt, Marcus Brubaker, Jiqiang Guo, Peter Li, and Allen Riddell. 2017. Stan: A probabilistic programming language. Journal of Statistical Software, Articles, 76(1):1–32. [find](https://scholar.google.com/scholar?q=Bob+Carpenter%2C+Andrew+Gelman%2C+Matthew+Hoffman%2C+Daniel+Lee%2C+Ben+Goodrich%2C+Michael+Betancourt%2C+Marcus+Brubaker%2C+Jiqiang+Guo%2C+Peter+Li%2C+and+Allen+Riddell%2E+2017%2E+Stan%3A+A+probabilistic+programming+language)
11. Gennadi Lembersky, Noam Ordan, and Shuly Wintner. 2012. Adapting translation models to translationese improves SMT. In Proceedings of EACL, pages 255–265. [find](https://scholar.google.com/scholar?q=Gennadi+Lembersky%2C+Noam+Ordan%2C+and+Shuly+Wintner%2E+2012%2E+Adapting+translation+models+to+translationese+improves+SMT%2E+In+Proceedings+of+EACL%2C+pages+255%E2%80%93265%2E)
12. Ryan Cotterell, Sabrina J. Mielke, Jason Eisner, and Brian Roark. 2018. [Are all languages equally hard to language-model?](https://md.research.elamur.ai/free/en/papers/cotterell-2018-are-all-languages-equally-hard/) In Proceedings of NAACL, pages 536–541.
13. Haitao Liu. 2008. Dependency distance as a metric of language comprehension difficulty. Journal of Cognitive Science, 9(2):159–191. [find](https://scholar.google.com/scholar?q=Haitao+Liu%2E+2008%2E+Dependency+distance+as+a+metric+of+language+comprehension+difficulty%2E+Journal+of+Cognitive+Science%2C+9%282%29%3A159%E2%80%93191%2E)
14. Mathieu Dehouck and Pascal Denis. 2018. A framework for understanding the role of morphology in universal dependency parsing. In Proceedings of EMNLP, pages 2864–2870. [find](https://scholar.google.com/scholar?q=Mathieu+Dehouck+and+Pascal+Denis%2E+2018%2E+A+framework+for+understanding+the+role+of+morphology+in+universal+dependency+parsing%2E+In+Proceedings+of+EMNLP%2C+pages+2864%E2%80%932870%2E)
15. Haitao Liu, Chunshan Xu, and Junying Liang. 2017. Dependency distance: A new perspective on syntactic patterns in natural languages. Physics of Life Reviews, 21:171–193. [find](https://scholar.google.com/scholar?q=Haitao+Liu%2C+Chunshan+Xu%2C+and+Junying+Liang%2E+2017%2E+Dependency+distance%3A+A+new+perspective+on+syntactic+patterns+in+natural+languages%2E+Physics+of+Life+Reviews%2C+21%3A171%E2%80%93193%2E)
16. Chris Drummond. 2009. Replicability is not reproducibility: Nor is it good science. In Proceedings of the Evaluation Methods for Machine Learning Workshop at the 26th ICML. [find](https://scholar.google.com/scholar?q=Chris+Drummond%2E+2009%2E+Replicability+is+not+reproducibility%3A+Nor+is+it+good+science%2E+In+Proceedings+of+the+Evaluation+Methods+for+Machine+Learning+Workshop+at+the+26th+ICML%2E)
17. Thomas Mayer and Michael Cysouw. 2014. [Creating a massively parallel Bible corpus](https://md.research.elamur.ai/free/en/papers/mayer-2014-creating-massively-parallel-bible-corpus/). In Proceedings of LREC, pages 3158–3163.
18. Matthew S. Dryer and Martin Haspelmath, editors. 2013. WALS Online. Max Planck Institute for Evolutionary Anthropology, Leipzig. [find](https://scholar.google.com/scholar?q=Matthew+S%2E+Dryer+and+Martin+Haspelmath%2C+editors%2E+2013%2E+WALS+Online%2E+Max+Planck+Institute+for+Evolutionary+Anthropology%2C+Leipzig%2E)
19. Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018. An analysis of neural language modeling at multiple scales. arXiv preprint arXiv:1803.08240. [find](https://scholar.google.com/scholar?q=Stephen+Merity%2C+Nitish+Shirish+Keskar%2C+and+Richard+Socher%2E+2018%2E+An+analysis+of+neural+language+modeling+at+multiple+scales%2E+arXiv+preprint+arXiv%3A1803%2E08240%2E)
20. Lawrence Fenton. 1960. The sum of log-normal probability distributions in scatter transmission systems. IRE Transactions on Communications Systems, 8(1):57–67. [find](https://scholar.google.com/scholar?q=Lawrence+Fenton%2E+1960%2E+The+sum+of+log%2Dnormal+probability+distributions+in+scatter+transmission+systems%2E+IRE+Transactions+on+Communications+Systems%2C+8%281%29%3A57%E2%80%9367%2E)
21. Bart van Merriënboer, Amartya Sanyal, Hugo Larochelle, and Yoshua Bengio. 2017. Multiscale sequence modeling with a learned dictionary. arXiv preprint arXiv:1707.00762. — [arXiv:1707.00762](https://arxiv.org/abs/1707.00762) [find](https://scholar.google.com/scholar?q=Bart+van+Merrie%CC%88nboer%2C+Amartya+Sanyal%2C+Hugo+Larochelle%2C+and+Yoshua+Bengio%2E+2017%2E+Multiscale+sequence+modeling+with+a+learned+dictionary%2E+arXiv+preprint+arXiv%3A1707%2E00762%2E+%E2%80%94+%5BarXiv%3A1707%2E00762%5D%28https%3A%2F%2Fa)
22. Richard Futrell, Kyle Mahowald, and Edward Gibson. 2015. Large-scale evidence of dependency length minimization in 37 languages. Proceedings of the National Academy of Sciences, 112(33):10336– 10341. [find](https://scholar.google.com/scholar?q=Richard+Futrell%2C+Kyle+Mahowald%2C+and+Edward+Gibson%2E+2015%2E+Large%2Dscale+evidence+of+dependency+length+minimization+in+37+languages%2E+Proceedings+of+the+National+Academy+of+Sciences%2C+112%2833%29%3A10336%E2%80%93+10341%2E)
23. Sabrina J. Mielke and Jason Eisner. 2018. Spell once, summon anywhere: A two-level open-vocabulary language model. arXiv preprint arXiv:1804.08205. — [arXiv:1804.08205](https://arxiv.org/abs/1804.08205) [find](https://scholar.google.com/scholar?q=Sabrina+J%2E+Mielke+and+Jason+Eisner%2E+2018%2E+Spell+once%2C+summon+anywhere%3A+A+two%2Dlevel+open%2Dvocabulary+language+model%2E+arXiv+preprint+arXiv%3A1804%2E08205%2E+%E2%80%94+%5BarXiv%3A1804%2E08205%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1804%2E0820)
24. NIST Multimodal Information Group. 2010a. [find](https://scholar.google.com/scholar?q=NIST+Multimodal+Information+Group%2E+2010a%2E)
25. NIST 2002 Open Machine Translation (OpenMT) evaluation LDC2010T10. [find](https://scholar.google.com/scholar?q=NIST+2002+Open+Machine+Translation+%28OpenMT%29+evaluation+LDC2010T10%2E)
26. Johannes Graën, Dolores Batinic, and Martin Volk. 2014. Cleaning the Europarl corpus for linguistic applications. In Konvens, pages 222–227. [find](https://scholar.google.com/scholar?q=Johannes+Grae%CC%88n%2C+Dolores+Batinic%2C+and+Martin+Volk%2E+2014%2E+Cleaning+the+Europarl+corpus+for+linguistic+applications%2E+In+Konvens%2C+pages+222%E2%80%93227%2E)
27. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. [Neural machine translation of rare words with subword units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/). In Proceedings of ACL, pages 1715– 1725.
28. NIST Multimodal Information Group. 2010b. [find](https://scholar.google.com/scholar?q=NIST+Multimodal+Information+Group%2E+2010b%2E)
29. NIST 2003 Open Machine Translation (OpenMT) evaluation LDC2010T11. [find](https://scholar.google.com/scholar?q=NIST+2003+Open+Machine+Translation+%28OpenMT%29+evaluation+LDC2010T11%2E)
30. NIST Multimodal Information Group. 2010c. [find](https://scholar.google.com/scholar?q=NIST+Multimodal+Information+Group%2E+2010c%2E)
31. NIST 2004 Open Machine Translation (OpenMT) evaluation LDC2010T12. [find](https://scholar.google.com/scholar?q=NIST+2004+Open+Machine+Translation+%28OpenMT%29+evaluation+LDC2010T12%2E)
32. Claude E. Shannon. 1951. Prediction and entropy of printed English. Bell Labs Technical Journal, 30(1):50–64. [find](https://scholar.google.com/scholar?q=Claude+E%2E+Shannon%2E+1951%2E+Prediction+and+entropy+of+printed+English%2E+Bell+Labs+Technical+Journal%2C+30%281%29%3A50%E2%80%9364%2E)
33. NIST Multimodal Information Group. 2010d. [find](https://scholar.google.com/scholar?q=NIST+Multimodal+Information+Group%2E+2010d%2E)
34. NIST 2005 Open Machine Translation (OpenMT) evaluation LDC2010T14. [find](https://scholar.google.com/scholar?q=NIST+2005+Open+Machine+Translation+%28OpenMT%29+evaluation+LDC2010T14%2E)
35. Milan Straka, Jan Hajič, and Jana Straková. 2016. UD-Pipe: Trainable pipeline for processing CoNLL-U files performing tokenization, morphological analysis, POS tagging and parsing. In Proceedings of LREC, pages 4290–4297. [find](https://scholar.google.com/scholar?q=Milan+Straka%2C+Jan+Hajic%CC%8C%2C+and+Jana+Strakova%CC%81%2E+2016%2E+UD%2DPipe%3A+Trainable+pipeline+for+processing+CoNLL%2DU+files+performing+tokenization%2C+morphological+analysis%2C+POS+tagging+and+parsing%2E+In+Proceedings+of)
36. NIST Multimodal Information Group. 2010e. [find](https://scholar.google.com/scholar?q=NIST+Multimodal+Information+Group%2E+2010e%2E)
37. NIST 2006 Open Machine Translation (OpenMT) evaluation LDC2010T17. [find](https://scholar.google.com/scholar?q=NIST+2006+Open+Machine+Translation+%28OpenMT%29+evaluation+LDC2010T17%2E)
38. Milan Straka and Jana Straková. 2017. [Tokenizing, POS tagging, lemmatizing and parsing UD 2.0 with UDPipe](https://md.research.elamur.ai/free/en/papers/straka-2017-tokenizing-pos-tagging-lemmatizing-parsing/). In CoNLL 2017 Shared Task: Multilingual parsing from raw text to Universal Dependencies, pages 88–99.
39. NIST Multimodal Information Group. 2010f. [find](https://scholar.google.com/scholar?q=NIST+Multimodal+Information+Group%2E+2010f%2E)
40. NIST 2008 Open Machine Translation (OpenMT) evaluation LDC2010T21. [find](https://scholar.google.com/scholar?q=NIST+2008+Open+Machine+Translation+%28OpenMT%29+evaluation+LDC2010T21%2E)
41. NIST Multimodal Information Group. 2010g. [find](https://scholar.google.com/scholar?q=NIST+Multimodal+Information+Group%2E+2010g%2E)
42. NIST 2009 Open Machine Translation (OpenMT) evaluation LDC2010T23. [find](https://scholar.google.com/scholar?q=NIST+2009+Open+Machine+Translation+%28OpenMT%29+evaluation+LDC2010T23%2E)
43. Ilya Sutskever, James Martens, and Geoffrey Hinton. 2011. Generating text with recurrent neural networks. In Proceedings of ICML, pages 1017–1024. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+James+Martens%2C+and+Geoffrey+Hinton%2E+2011%2E+Generating+text+with+recurrent+neural+networks%2E+In+Proceedings+of+ICML%2C+pages+1017%E2%80%931024%2E)
44. NIST Multimodal Information Group. 2013a. NIST 2008-2012 Open Machine Translation (OpenMT) progress test sets LDC2013T07. [find](https://scholar.google.com/scholar?q=NIST+Multimodal+Information+Group%2E+2013a%2E+NIST+2008%2D2012+Open+Machine+Translation+%28OpenMT%29+progress+test+sets+LDC2013T07%2E)
45. David Yarowsky, Grace Ngai, and Richard Wicentowski. 2001. [Inducing multilingual text analysis tools via robust projection across aligned corpora](https://md.research.elamur.ai/free/en/papers/yarowsky-2001-inducing-multilingual-text-analysis-tools/). In Proceedings of the First International Conference on Human Language Technology Research, pages 1–8.
46. NIST Multimodal Information Group. 2013b. [find](https://scholar.google.com/scholar?q=NIST+Multimodal+Information+Group%2E+2013b%2E)
47. NIST 2012 Open Machine Translation (OpenMT) evaluation LDC2013T03. [find](https://scholar.google.com/scholar?q=NIST+2012+Open+Machine+Translation+%28OpenMT%29+evaluation+LDC2013T03%2E)
48. Lewis M. Paul, Gary F. Simons, Charles D. Fennig, et al. 2009. Ethnologue: Languages of the world, 19 edition. SIL International, Dallas. [find](https://scholar.google.com/scholar?q=Lewis+M%2E+Paul%2C+Gary+F%2E+Simons%2C+Charles+D%2E+Fennig%2C+et+al%2E+2009%2E+Ethnologue%3A+Languages+of+the+world%2C+19+edition%2E+SIL+International%2C+Dallas%2E)
49. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. [Deep contextualized word representations](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/). In Proceedings of NAACL, pages 2227– 2237.
50. Ella Rabinovich and Shuly Wintner. 2015. Unsupervised identification of translationese. Transactions of the Association for Computational Linguistics, 3:419–432. [find](https://scholar.google.com/scholar?q=Ella+Rabinovich+and+Shuly+Wintner%2E+2015%2E+Unsupervised+identification+of+translationese%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+3%3A419%E2%80%93432%2E)
51. Ella Rabinovich, Shuly Wintner, and Ofek Luis Lewinsohn. 2016. A parallel corpus of translationese. In International Conference on Intelligent Text Processing and Computational Linguistics, pages 140–155. Springer. [find](https://scholar.google.com/scholar?q=Ella+Rabinovich%2C+Shuly+Wintner%2C+and+Ofek+Luis+Lewinsohn%2E+2016%2E+A+parallel+corpus+of+translationese%2E+In+International+Conference+on+Intelligent+Text+Processing+and+Computational+Linguistics%2C+pages+140%E2%80%93)
52. Philip Resnik, Mari Broman Olsen, and Mona Diab. 1999. The bible as a parallel corpus: Annotating the ‘book of 2000 tongues’. Computers and the Humanities, 33(1):129–153. [find](https://scholar.google.com/scholar?q=Philip+Resnik%2C+Mari+Broman+Olsen%2C+and+Mona+Diab%2E+1999%2E+The+bible+as+a+parallel+corpus%3A+Annotating+the+%E2%80%98book+of+2000+tongues%E2%80%99%2E+Computers+and+the+Humanities%2C+33%281%29%3A129%E2%80%93153%2E)
53. Benoı̂t Sagot. 2013. Comparing complexity measures. In Computational Approaches to Morphological Complexity. [find](https://scholar.google.com/scholar?q=Beno%C4%B1%CC%82t+Sagot%2E+2013%2E+Comparing+complexity+measures%2E+In+Computational+Approaches+to+Morphological+Complexity%2E)
54. S. C. Schwartz and Y. S. Yeh. 1982. On the distribution function and moments of power sums with log-normal components. The Bell System Technical Journal, 61(7):1441–1462. [find](https://scholar.google.com/scholar?q=S%2E+C%2E+Schwartz+and+Y%2E+S%2E+Yeh%2E+1982%2E+On+the+distribution+function+and+moments+of+power+sums+with+log%2Dnormal+components%2E+The+Bell+System+Technical+Journal%2C+61%287%29%3A1441%E2%80%931462%2E)

*Updated 24 October 2026*