---
title: "A Phrase-Based Statistical Model for SMS Text Normalization"
description: "SMS messages are full of ad-hoc short-forms, phonetic spellings, dropped words and… 0.8070 BLEU. 20 references, cited by 78 papers here."
authors: ["AiTi Aw", "Min Zhang", "Juan Xiao", "Jian Su"]
year: 2006
venue: "Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions"
source: "https://aclanthology.org/P06-2005/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/aw-2006-phrase-based-statistical-model-sms/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/aw-2006-phrase-based-statistical-model-sms/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2006](https://md.research.elamur.ai/free/en/papers/2006/) / A Phrase-Based Statistical Model for SMS Text Normalization

# A Phrase-Based Statistical Model for SMS Text Normalization

**Authors:** [AiTi Aw](https://md.research.elamur.ai/free/en/authors/aiti-aw/), [Min Zhang](https://md.research.elamur.ai/free/en/authors/min-zhang/), Juan Xiao and [Jian Su](https://md.research.elamur.ai/free/en/authors/jian-su/) · **Year:** 2006 · **Venue:** [Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2006](https://md.research.elamur.ai/free/en/venues/acl/2006/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/P06-2005/) · [PDF](https://aclanthology.org/P06-2005.pdf) · [OpenAlex](https://openalex.org/W2163942301)

**Citations:** 278 citations (OpenAlex, 2026-10-14) · [78](https://md.research.elamur.ai/free/en/papers/aw-2006-phrase-based-statistical-model-sms/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
SMS messages are full of ad-hoc short-forms, phonetic spellings, dropped words and missing punctuation, so off-the-shelf machine translation handles them poorly. Rather than retuning each MT system for this style, the authors clean up English SMS text into standard English first and leave the MT engine unchanged. They frame this normalization as monolingual translation from "SMS language" to English. Their system pairs a bigram language model with a phrase-based lexical mapping model, which allows multi-word to multi-word mappings and a null word for deletions. They built a parallel corpus of 5000 raw messages with manual normalizations. In 5-fold cross validation, the system reaches 0.8070 BLEU, compared with a baseline of about 0.70. Using it as a preprocessing step for English-to-Chinese translation of 402 messages nearly doubles average BLEU, from 0.1926 to 0.3770. This shows that a separate normalization module can adapt general MT systems to noisy text without modifying them.

## Key points
- **Data:** The source collection has 55,000 English SMS messages from a chat room and from university students' correspondence. The 5000-message parallel corpus (raw text plus manual normalization) was used for training and testing.
- **Corpus analysis:** In a sample of 700 messages, 621 (88.71%) needed changes, totaling 2300 edits. Substitutions made up 86.43% of the edits, insertions 8.09% and deletions 5.48%.
- **Normalization results:** The phrase-based model averaged 0.8070 BLEU (3-gram) across 5 folds, with individual folds ranging from 0.7908 to 0.8236. The paper reports the baseline as 0.6958 in some places and 0.6985 in others. The learning curve levels off at around 3000 training messages.
- **MT impact (402 messages, two references, 3-gram BLEU):**

  | Input | Systran | I2R | Average |
  |---|---|---|---|
  | Raw messages | 0.1219 | 0.2633 | 0.1926 |
  | Dictionary lookup | 0.1690 | 0.3485 | 0.2588 |
  | Proposed normalization | 0.3116 | 0.4423 | 0.3770 |

- **Limitations and future work:** The system does not yet restore missing punctuation. The authors plan to use pronunciation information to handle OOV words caused by phonetic spelling.

## Interesting ideas
- **Normalize before translating, and treat MT as a black box.** Separating noisy-text handling from the downstream system means one module can serve MT, text-to-speech, search and other applications. This is useful whenever you cannot or do not want to retrain the main model, for example when processing informal Ukrainian or surzhyk social media text.
- **Cast normalization as monolingual phrase-based translation.** Contiguous multi-word mappings plus a null token cover all three edit types: substitution, deletion and insertion. Merged forms like "lemme" or "duno" show why word-by-word dictionary replacement is not enough.
- **Profile the edit types before designing the model.** Sampling a few hundred messages and counting substitutions, insertions and deletions showed which phenomena the model must cover. This kind of cheap analysis can guide the design of any normalization or correction task.
- **Evaluate both intrinsically and extrinsically.** Besides BLEU against manual normalizations, the authors measure the gain in downstream MT across two different engines. They also include a dictionary-lookup baseline, which shows how much context-aware modeling adds over a simple lingo list.
- **Use a learning curve to check data sufficiency.** Plotting score against training size, which plateaued at about 3000 pairs, gives a practical estimate of how much parallel data to annotate for a similar task.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Institute for Infocomm Research](https://md.research.elamur.ai/free/en/labs/institute-for-infocomm-research/) — Research institute, Singapore, Singapore · 33 works · 14,373 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

## Cite

Cite as: AiTi Aw, Min Zhang, Juan Xiao and Jian Su. 2006. A Phrase-Based Statistical Model for SMS Text Normalization. Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{aw2006phrase,
  title = {{A Phrase-Based Statistical Model for SMS Text Normalization}},
  author = {AiTi Aw, Min Zhang, Juan Xiao and Jian Su},
  booktitle = {Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions},
  year = {2006},
  doi = {10.3115/1273073.1273078},
  url = {https://doi.org/10.3115/1273073.1273078}
}
```

- [← previous Arabic Preprocessing Schemes for Statistical Machine Translation](https://md.research.elamur.ai/free/en/papers/habash-2006-arabic-preprocessing-schemes-statistical-machine/)
- [next → Reranking and Self-Training for Parser Adaptation](https://md.research.elamur.ai/free/en/papers/mcclosky-2006-reranking-self-training-parser-adaptation/)
- [↑ back to papers of 2006](https://md.research.elamur.ai/free/en/papers/2006/)

## References

20 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. A.T. Aw, M. Zhang, Z.Z. Fan, P.K. Yeo and J. Su. 2005. Input Normalization for an English-to-Chinese SMS Translation System. MT Summit-2005 [find](https://scholar.google.com/scholar?q=A%2ET%2E+Aw%2C+M%2E+Zhang%2C+Z%2EZ%2E+Fan%2C+P%2EK%2E+Yeo+and+J%2E+Su%2E+2005%2E+Input+Normalization+for+an+English%2Dto%2DChinese+SMS+Translation+System%2E+MT+Summit%2D2005)
2. S. Bangalore, V. Murdock and G. Riccardi. 2002. Bootstrapping Bilingual Data using Consensus Translation for a Multilingual Instant Messaging System. COLING-2002 [find](https://scholar.google.com/scholar?q=S%2E+Bangalore%2C+V%2E+Murdock+and+G%2E+Riccardi%2E+2002%2E+Bootstrapping+Bilingual+Data+using+Consensus+Translation+for+a+Multilingual+Instant+Messaging+System%2E+COLING%2D2002)
3. R. Barzilay and K. R. McKeown. 2001. [Extracting paraphrases from a parallel corpus](https://md.research.elamur.ai/free/en/papers/barzilay-2001-extracting-paraphrases-parallel-corpus/). ACL-2001
4. E. Brill and R. C. Moore. 2000. [An Improved Error Model for Noisy Channel Spelling Correction](https://md.research.elamur.ai/free/en/papers/brill-2000-improved-error-model-noisy-channel/). ACL-2000
5. P. F. Brown, S. D. Pietra, V. D. Pietra and R. Mercer. 1993. The Mathematics of Statistical Machine Translation: Parameter Estimation. Computational Linguistics: 19(2) [find](https://scholar.google.com/scholar?q=P%2E+F%2E+Brown%2C+S%2E+D%2E+Pietra%2C+V%2E+D%2E+Pietra+and+R%2E+Mercer%2E+1993%2E+The+Mathematics+of+Statistical+Machine+Translation%3A+Parameter+Estimation%2E+Computational+Linguistics%3A+19%282%29)
6. A. Clark. 2003. Pre-processing very noisy text. In Proceedings of Workshop on Shallow Processing of Large Corpora, Lancaster, 2003 [find](https://scholar.google.com/scholar?q=A%2E+Clark%2E+2003%2E+Pre%2Dprocessing+very+noisy+text%2E+In+Proceedings+of+Workshop+on+Shallow+Processing+of+Large+Corpora%2C+Lancaster%2C+2003)
7. F. J. Damerau. 1964. A technique for computer detection and correction of spelling errors. Communications ACM 7, 171-176 [find](https://scholar.google.com/scholar?q=F%2E+J%2E+Damerau%2E+1964%2E+A+technique+for+computer+detection+and+correction+of+spelling+errors%2E+Communications+ACM+7%2C+171%2D176)
8. A.P. Dempster, N.M. Laird and D.B. Rubin. 1977. Maximum likelihood from incomplete data via the EM algorithm, Journal of the Royal Statistical Society, Series B, Vol. 39, 1-38 [find](https://scholar.google.com/scholar?q=A%2EP%2E+Dempster%2C+N%2EM%2E+Laird+and+D%2EB%2E+Rubin%2E+1977%2E+Maximum+likelihood+from+incomplete+data+via+the+EM+algorithm%2C+Journal+of+the+Royal+Statistical+Society%2C+Series+B%2C+Vol%2E+39%2C+1%2D38)
9. A. Golding and D. Roth. 1999. A Winnow-Based Approach to Spelling Correction. Machine Learning 34: 107-130 [find](https://scholar.google.com/scholar?q=A%2E+Golding+and+D%2E+Roth%2E+1999%2E+A+Winnow%2DBased+Approach+to+Spelling+Correction%2E+Machine+Learning+34%3A+107%2D130)
10. F. Jelinek. 1991. Self-organized language modeling for speech recognition. In A. Waibel and K.F. Lee, editors, Readings in Speech Recognition, pages 450-506. [find](https://scholar.google.com/scholar?q=F%2E+Jelinek%2E+1991%2E+Self%2Dorganized+language+modeling+for+speech+recognition%2E+In+A%2E+Waibel+and+K%2EF%2E+Lee%2C+editors%2C+Readings+in+Speech+Recognition%2C+pages+450%2D506%2E)
11. M. D. Kernighan, K Church and W. Gale. 1990. [A spelling correction program based on a noisy channel model](https://md.research.elamur.ai/free/en/papers/kernighan-1990-spelling-correction-program-based-noisy/). COLING-1990
12. K. Kukich. 1992. Techniques for automatically correcting words in text. ACM Computing Surveys, 24(4):377-439 [find](https://scholar.google.com/scholar?q=K%2E+Kukich%2E+1992%2E+Techniques+for+automatically+correcting+words+in+text%2E+ACM+Computing+Surveys%2C+24%284%29%3A377%2D439)
13. K. A. Papineni, S. Roukos, T. Ward and W. J. Zhu. 2002. [BLEU : a Method for Automatic Evaluation of Machine Translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). ACL-2002
14. P. Koehn, F.J. Och and D. Marcu. 2003. Statistical Phrase-Based Translation. HLT-NAACL-2003 [find](https://scholar.google.com/scholar?q=P%2E+Koehn%2C+F%2EJ%2E+Och+and+D%2E+Marcu%2E+2003%2E+Statistical+Phrase%2DBased+Translation%2E+HLT%2DNAACL%2D2003)
15. C. Shannon. 1948. A mathematical theory of communication. Bell System Technical Journal 27(3): 379-423 [find](https://scholar.google.com/scholar?q=C%2E+Shannon%2E+1948%2E+A+mathematical+theory+of+communication%2E+Bell+System+Technical+Journal+27%283%29%3A+379%2D423)
16. M. Shimohata and E. Sumita 2002. Automatic Paraphrasing Based on Parallel Corpus for Normalization. LREC-2002 [find](https://scholar.google.com/scholar?q=M%2E+Shimohata+and+E%2E+Sumita+2002%2E+Automatic+Paraphrasing+Based+on+Parallel+Corpus+for+Normalization%2E+LREC%2D2002)
17. R. Sproat, A. Black, S. Chen, S. Kumar, M. Ostendorf and C. Richards. 2001. Normalization of Non-Standard Words. Computer Speech and Language, 15(3):287-333 [find](https://scholar.google.com/scholar?q=R%2E+Sproat%2C+A%2E+Black%2C+S%2E+Chen%2C+S%2E+Kumar%2C+M%2E+Ostendorf+and+C%2E+Richards%2E+2001%2E+Normalization+of+Non%2DStandard+Words%2E+Computer+Speech+and+Language%2C+15%283%29%3A287%2D333)
18. A. Stolcke. 2002. SRILM – An extensible language modeling toolkit. ICSLP-2002 [find](https://scholar.google.com/scholar?q=A%2E+Stolcke%2E+2002%2E+SRILM+%E2%80%93+An+extensible+language+modeling+toolkit%2E+ICSLP%2D2002)
19. K. Toutanova and R. C. Moore. 2002. [Pronunciation Modeling for Improved Spelling Correction](https://md.research.elamur.ai/free/en/papers/toutanova-2002-pronunciation-modeling-improved-spelling-correction/). ACL-2002
20. R. Zens and H. Ney. 2004. Improvements in Phrase-Based Statistical MT. HLT-NAALL-2004 [find](https://scholar.google.com/scholar?q=R%2E+Zens+and+H%2E+Ney%2E+2004%2E+Improvements+in+Phrase%2DBased+Statistical+MT%2E+HLT%2DNAALL%2D2004)

*Updated 24 October 2026*