--- title: "Large Language Models in Machine Translation — Brants 2007" description: "The paper asks how far n-gram language models can be scaled for statistical machine translation and whether translation quality keeps improving as they…" authors: ["Thorsten Brants", "Ashok C. Popat", "Peng Xu", "Franz J. Och", "Jeffrey Dean"] year: 2007 venue: "Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)" source: "https://aclanthology.org/D07-1090/" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/brants-2007-large-language-models-machine-translation/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/brants-2007-large-language-models-machine-translation/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2007](https://md.research.elamur.ai/free/en/papers/2007/) / Large Language Models in Machine Translation # Large Language Models in Machine Translation **Authors:** [Thorsten Brants](https://md.research.elamur.ai/free/en/authors/thorsten-brants/), Ashok C. Popat, [Peng Xu](https://md.research.elamur.ai/free/en/authors/peng-xu/), Franz J. Och, [Jeffrey Dean](https://md.research.elamur.ai/free/en/authors/jeffrey-dean/) · **Year:** 2007 · **Venue:** Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/D07-1090/) · [PDF](https://aclanthology.org/D07-1090.pdf) · [OpenAlex](https://openalex.org/W2109664771) **Citations:** 531 citations (openalex-snapshot · 2026-10-14) · [170](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary The paper asks how far n-gram language models can be scaled for statistical machine translation and whether translation quality keeps improving as they grow. The Google authors built a distributed system on MapReduce that handles vocabulary building, n-gram counting and model generation. Sharding by the first two words of each n-gram keeps the workload balanced. The resulting models store smoothed probabilities rather than raw counts and are queried through batched client–server requests. This allows them to be used directly inside a single-pass decoder instead of only for re-ranking candidate translations afterwards. The authors also propose Stupid Backoff, an unnormalized scoring scheme that uses plain relative frequencies with a fixed backoff factor and is cheap to compute at massive scale. Using up to 2 trillion training tokens, they trained Arabic–English systems and charted BLEU learning curves. Quality rose roughly linearly with each doubling of data, even at the largest sizes, and Stupid Backoff caught up with Kneser-Ney smoothing once data became large. The work showed that sheer data volume, paired with simple estimators, can beat more sophisticated smoothing. ## Key points - Training data reached up to 2 trillion tokens, yielding 5-gram models with up to 300 billion n-grams. That is about two orders of magnitude beyond earlier distributed LMs, which used 3–4 billion words. - Stupid Backoff uses α = 0.4 without discounting or normalization. Its scores are not true probabilities, which is acceptable because the log-linear decoder relies only on relative feature values. - The experiments used four LM data sources (target, ldcnews, webnews, web) with an Arabic–English system. The full version of this system scored 0.4535 BLEU at NIST 2006. - On target data, gains per doubling were about 0.62 BP for Kneser-Ney and 0.66 BP for Stupid Backoff. The gap between the two methods disappeared on larger sets. The web-scale Stupid Backoff model beat the best Kneser-Ney model by 0.82 BP. - Test-set 5-gram coverage grew from 0.06 with 13 million tokens to 0.56 with 2 trillion tokens. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/) Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) ## Cited by Cited by 170 papers of our library cite it (of the 140673 whose bibliographies we read); 12 of them have a page here, 158 not yet. - [N-gram Counts and Language Models from the Common Crawl](https://md.research.elamur.ai/free/en/papers/buck-2014-n-gram-counts-language-models/) - [One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling](https://md.research.elamur.ai/free/en/papers/chelba-2013-one-billion-word-benchmark-measuring/) - [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-2024-team-gemini-1-5-unlocking-multimodal/) - [Scalable Modified Kneser-Ney Language Model Estimation](https://md.research.elamur.ai/free/en/papers/heafield-2013-scalable-modified-kneser-ney-language/) - [Improving Text Simplification Language Modeling Using Unsimplified Text Data](https://md.research.elamur.ai/free/en/papers/kauchak-2013-improving-text-simplification-language-modeling/) - [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://md.research.elamur.ai/free/en/papers/keskar-2019-ctrl-conditional-transformer-language-model/) - [Probabilistic Modelling of Morphologically Rich Languages](https://md.research.elamur.ai/free/en/papers/langstream-2015-botha-probabilistic-modelling-morphologically-rich-languages/) - [Intelligent Selection of Language Model Training Data](https://md.research.elamur.ai/free/en/papers/moore-2010-intelligent-selection-language-model-training/) - [ChatDev: Communicative Agents for Software Development](https://md.research.elamur.ai/free/en/papers/qian-2023-chatdev-communicative-agents-software/) - [BloombergGPT: A Large Language Model for Finance](https://md.research.elamur.ai/free/en/papers/wu-2023-bloomberggpt-large-language-model-finance/) - [Unsupervised Text Style Transfer using Language Models as Discriminators](https://md.research.elamur.ai/free/en/papers/yang-2018-unsupervised-text-style-transfer-using/) - [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{brants2007large, title = {{Large Language Models in Machine Translation}}, author = {Brants, Thorsten and Popat, Ashok C. and Xu, Peng and Och, Franz J. and Dean, Jeffrey}, booktitle = {Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)}, year = {2007}, url = {https://aclanthology.org/D07-1090/} } ``` - [← previous Joint Determination of Anaphoricity and Coreference Resolution using Integer Programming](https://md.research.elamur.ai/free/en/papers/denis-2007-joint-determination-anaphoricity-coreference-resolution/) - [next → Learning Synchronous Grammars for Semantic Parsing with Lambda Calculus](https://md.research.elamur.ai/free/en/papers/wong-2007-learning-synchronous-grammars-semantic-parsing/) - [↑ back to papers of 2007](https://md.research.elamur.ai/free/en/papers/2007/) ## References 14 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Peter F. Brown, Stephen Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, 19(2):263–311. [find](https://scholar.google.com/scholar?q=Peter+F%2E+Brown%2C+Stephen+Della+Pietra%2C+Vincent+J%2E+Della+Pietra%2C+and+Robert+L%2E+Mercer%2E+1993%2E+The+mathematics+of+statistical+machine+translation%3A+Parameter+estimation%2E+Computational+Linguistics%2C+19%282%29%3A26) 2. Stanley F. Chen and Joshua Goodman. 1998. An empirical study of smoothing techniques for language modeling. Technical Report TR-10-98, Harvard, Cambridge, MA, USA. [find](https://scholar.google.com/scholar?q=Stanley+F%2E+Chen+and+Joshua+Goodman%2E+1998%2E+An+empirical+study+of+smoothing+techniques+for+language+modeling%2E+Technical+Report+TR%2D10%2D98%2C+Harvard%2C+Cambridge%2C+MA%2C+USA%2E) [our card](https://md.research.elamur.ai/free/en/papers/chen-1996-empirical-study-smoothing-techniques-language/) 3. Jeffrey Dean and Sanjay Ghemawat. 2004. Mapreduce: Simplified data processing on large clusters. In Sixth Symposium on Operating System Design and Implementation (OSDI-04), San Francisco, CA, USA. [find](https://scholar.google.com/scholar?q=Jeffrey+Dean+and+Sanjay+Ghemawat%2E+2004%2E+Mapreduce%3A+Simplified+data+processing+on+large+clusters%2E+In+Sixth+Symposium+on+Operating+System+Design+and+Implementation+%28OSDI%2D04%29%2C+San+Francisco%2C+CA%2C+USA%2E) 4. Ahmad Emami, Kishore Papineni, and Jeffrey Sorensen. 2007. Large-scale distributed language modeling. In Proceedings of ICASSP-2007, Honolulu, HI, USA. [find](https://scholar.google.com/scholar?q=Ahmad+Emami%2C+Kishore+Papineni%2C+and+Jeffrey+Sorensen%2E+2007%2E+Large%2Dscale+distributed+language+modeling%2E+In+Proceedings+of+ICASSP%2D2007%2C+Honolulu%2C+HI%2C+USA%2E) 5. Joshua Goodman. 2001. A bit of progress in language modeling. Technical Report MSR-TR-2001-72, Microsoft Research, Redmond, WA, USA. [find](https://scholar.google.com/scholar?q=Joshua+Goodman%2E+2001%2E+A+bit+of+progress+in+language+modeling%2E+Technical+Report+MSR%2DTR%2D2001%2D72%2C+Microsoft+Research%2C+Redmond%2C+WA%2C+USA%2E) 6. Frederick Jelinek and Robert L. Mercer. 1980. Interpolated estimation of Markov source parameters from sparse data. In Pattern Recognition in Practice, pages 381–397. North Holland. [find](https://scholar.google.com/scholar?q=Frederick+Jelinek+and+Robert+L%2E+Mercer%2E+1980%2E+Interpolated+estimation+of+Markov+source+parameters+from+sparse+data%2E+In+Pattern+Recognition+in+Practice%2C+pages+381%E2%80%93397%2E+North+Holland%2E) 7. Slava Katz. 1987. Estimation of probabilities from sparse data for the language model component of a speech recognizer. IEEE Transactions on Acoustics, Speech, and Signal Processing, 35(3). [find](https://scholar.google.com/scholar?q=Slava+Katz%2E+1987%2E+Estimation+of+probabilities+from+sparse+data+for+the+language+model+component+of+a+speech+recognizer%2E+IEEE+Transactions+on+Acoustics%2C+Speech%2C+and+Signal+Processing%2C+35%283%29%2E) 8. Reinhard Kneser and Hermann Ney. 1995. Improved backing-off for m-gram language modeling. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, pages 181– 184. [find](https://scholar.google.com/scholar?q=Reinhard+Kneser+and+Hermann+Ney%2E+1995%2E+Improved+backing%2Doff+for+m%2Dgram+language+modeling%2E+In+Proceedings+of+the+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processing%2C+pages+181%E2%80%93+184) 9. Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of EMNLP-04, Barcelona, Spain. [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2E+2004%2E+Statistical+significance+tests+for+machine+translation+evaluation%2E+In+Proceedings+of+EMNLP%2D04%2C+Barcelona%2C+Spain%2E) 10. Hermann Ney and Stefan Ortmanns. 1999. Dynamic programming search for continuous speech recognition. IEEE Signal Processing Magazine, 16(5):64–83. [find](https://scholar.google.com/scholar?q=Hermann+Ney+and+Stefan+Ortmanns%2E+1999%2E+Dynamic+programming+search+for+continuous+speech+recognition%2E+IEEE+Signal+Processing+Magazine%2C+16%285%29%3A64%E2%80%9383%2E) 11. Eric W. Noreen. 1989. Computer-Intensive Methods for Testing Hypotheses. John Wiley & Sons. [find](https://scholar.google.com/scholar?q=Eric+W%2E+Noreen%2E+1989%2E+Computer%2DIntensive+Methods+for+Testing+Hypotheses%2E+John+Wiley+%26+Sons%2E) 12. Franz Josef Och and Hermann Ney. 2004. The alignment template approach to statistical machine translation. Computational Linguistics, 30(4):417–449. [find](https://scholar.google.com/scholar?q=Franz+Josef+Och+and+Hermann+Ney%2E+2004%2E+The+alignment+template+approach+to+statistical+machine+translation%2E+Computational+Linguistics%2C+30%284%29%3A417%E2%80%93449%2E) 13. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of ACL-02, pages 311–318, Philadelphia, PA, USA. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+BLEU%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+Proceedings+of+ACL%2D02%2C+pages+311%E2%80%93318%2C+Philadelphia%2C+PA%2C+USA%2E) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/) 14. Ying Zhang, Almut Silja Hildebrand, and Stephan Vogel. 2006. Distributed language modeling for n-best list re-ranking. In Proceedings of EMNLP-2006, pages 216–223, Sydney, Australia. [find](https://scholar.google.com/scholar?q=Ying+Zhang%2C+Almut+Silja+Hildebrand%2C+and+Stephan+Vogel%2E+2006%2E+Distributed+language+modeling+for+n%2Dbest+list+re%2Dranking%2E+In+Proceedings+of+EMNLP%2D2006%2C+pages+216%E2%80%93223%2C+Sydney%2C+Australia%2E)