--- title: "IHS_RD: Lexical Normalization for English Tweets (2015)" description: "The paper addresses lexical normalization of English tweets: rewriting slang, typos… 0.8083 F1. 9 references, cited by 9 papers here." authors: ["Dmitry Supranovich", "Viachaslau Patsepnia"] year: 2015 venue: "Proceedings of the Workshop on Noisy User-generated Text" source: "https://aclanthology.org/W15-4311/" doi: "10.18653/v1/w15-4311" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/supranovich-2015-ihs-rd-lexical-normalization-english/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/supranovich-2015-ihs-rd-lexical-normalization-english/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2015](https://md.research.elamur.ai/free/en/papers/2015/page/3/) / IHS\_RD: Lexical Normalization for English Tweets # IHS\_RD: Lexical Normalization for English Tweets **Authors:** Dmitry Supranovich and Viachaslau Patsepnia · **Year:** 2015 · **Venue:** [Proceedings of the Workshop on Noisy User-generated Text](https://md.research.elamur.ai/free/en/venues/wnut/) ([Workshop on Noisy User-generated Text 2015](https://md.research.elamur.ai/free/en/venues/wnut/2015/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/W15-4311/) · [PDF](https://aclanthology.org/W15-4311.pdf) · [DOI 10.18653/v1/w15-4311](https://doi.org/10.18653/v1/w15-4311) · [OpenAlex](https://openalex.org/W2250409328) **Citations:** 19 citations (OpenAlex, 2026-10-14) · [9](https://md.research.elamur.ai/free/en/papers/supranovich-2015-ihs-rd-lexical-normalization-english/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary The paper addresses lexical normalization of English tweets: rewriting slang, typos, abbreviations and stretched spellings into canonical English for the ACL 2015 W-NUT shared task. Normalizations can be one-to-one, many-to-one or one-to-many, and proper nouns and hashtags/mentions must be left alone. The training data has 2950 annotated tweets. The authors split the task into two parts: - **Detection:** a CRF labels each token as a likely normalization target or not. - **Replacement:** a lexicon supplies the rewrite. The lexicon combines the training pairs, earlier normalization datasets (Han; Liu) and a public list of social-media abbreviations. All output is then converted to American spelling with VarCon. An in-house SVM "did-you-mean" spelling corrector is applied afterwards to catch words the lexicon lacks. On the 1967-tweet test set the submitted system scored F1 0.8272. It ranked first of 5 unconstrained entries and second of all 11. Dropping the corrector actually raised F1 slightly, because it overcorrected non-English and morphologically varied words. The study shows that a simple, transparent pipeline built around a lexicon can be highly competitive. ## Key points - **Candidate CRF:** - Labels are CAND / NOT_CAND. - Features are the token itself, its left and right neighbours, and compliance with the guidelines (alphanumeric only). - Membership in the normalization lexicon and in the canonical lexicon are also features. - So are word length, vowel count and their interaction, plus whether the token is within edit distance ≤2 of a canonical word. - **Effect of features (CRF stage alone):** - The baseline features alone give CRF precision 0.991 but recall only 0.57 (F1 0.7237). - Adding the dictionary features lifts CRF F1 to 0.8635. - With all features enabled, CRF F1 is 0.8632, and the final system F1 is 0.8272. - **With vs. without the corrector:** - Submitted system with DYM: P 0.8469, R 0.8083, F1 0.8272. - Without DYM: P 0.8765, R 0.7949, F1 0.8337. - **The DYM corrector:** - It is an SVM over candidate corrections. - Its features include an error-model score from an autocorrection index of 12.4M documents and a 2-gram language model trained on 177K patents. - It was not adapted to social media text. - **Approaches that failed:** - Language detection did not help, because tweets are short and often mix languages. - A non-social-media language model could not pick the right variant among ambiguous entries (e.g. ur → you're / your / you). - As a fallback, the most frequent variant is listed first. ## Interesting ideas - Separate "does this token need changing?" (a sequence classifier) from "what does it become?" (a lexicon lookup); each half then stays simple, inspectable and fixable on its own. - A cheap length-versus-vowel-count feature catches vowel-stripped abbreviations like bcz or pls without any dictionary. - Score each component on its own and also end-to-end; here a slightly better candidate detector did not translate into a much better final result, and an extra corrector lowered overall F1. - The corrector treats "make no correction" as an explicit candidate with its own confidence, rather than as the absence of an output. - A corrector trained on patents and scientific text overcorrected foreign and inflected words in tweets, which shows the danger of applying a module to text from a different domain. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (tokens): a pre-tagger normalization pass in crate en could flag noisy tokens with a classifier and rewrite them only from an explicit lexicon, so every change is traceable to an entry. - **I2** (tokens): the tokenizer/tagger could use this signal to route odd tokens to an abbreviation-expansion step instead of guessing a POS for an unknown word. - **I3** (learn): evaluation should report per-stage metrics next to the final answer score, so that a module that looks good locally but harms the pipeline is visible. - **I4** (learn): self-induced repair rules could always compete against a scored "leave unchanged" option, which serves as a built-in guard against overcorrection. - **I5** (learn): when silver data or repair modules are reused on a new genre, run them first against held-out text from that genre as a negative control before trusting their changes. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* 📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🎯 Decision Sciences](https://md.research.elamur.ai/free/en/shelves/decision-sciences/) · 249 works · [🎯 Management Science and Operations Research](https://md.research.elamur.ai/free/en/shelves/decision-sciences/management-science-and-operations-research/) · 134 works · [🖱️ Human-Computer Interaction](https://md.research.elamur.ai/free/en/shelves/computer-science/human-computer-interaction/) · 160 works 🧭 Topics (1): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,379 works ## Cite Cite as: Dmitry Supranovich and Viachaslau Patsepnia. 2015. IHS\_RD: Lexical Normalization for English Tweets. Proceedings of the Workshop on Noisy User-generated Text. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{supranovich2015ihs, title = {{IHS\_RD: Lexical Normalization for English Tweets}}, author = {Dmitry Supranovich and Viachaslau Patsepnia}, booktitle = {Proceedings of the Workshop on Noisy User-generated Text}, year = {2015}, doi = {10.18653/v1/w15-4311}, url = {https://doi.org/10.18653/v1/w15-4311} } ``` - [← previous World Englishes and English Language Teaching: A pragmatic and humanistic approach](https://md.research.elamur.ai/free/en/papers/bhowmik-2015-world-englishes-english-language-teaching/) - [next → ValenTo: Sentiment Analysis of Figurative Language Tweets with Irony and Sarcasm](https://md.research.elamur.ai/free/en/papers/farias-2015-valento-sentiment-analysis-figurative-language/) - [↑ back to papers of 2015](https://md.research.elamur.ai/free/en/papers/2015/page/3/) ## References 9 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Kevin Atkinson. VarCon. Vers. 2015.02.15. Web. 01 Apr. 2015. [http://wordlist.aspell.net/varcon/](http://wordlist.aspell.net/varcon/) [↗](http://wordlist.aspell.net/varcon/) 2. Timothy Baldwin, Marie-Catherine de Marneffe, Bo Han, Young-Bum Kim, Alan Ritter, and Wei Xu. 2015. [Shared tasks of the 2015 workshop on noisy user-generated text: Twitter lexical normalization and named entity recognition](https://md.research.elamur.ai/free/en/papers/baldwin-2015-shared-tasks-2015-workshop-noisy/). In Proceedings of the Workshop on Noisy User-generated Text (WNUT 2015), Beijing, China. 3. Vangie Beal. Text messaging and online chat abbreviations. Web. 01 Apr. 2015. [http://www.webopedia.com/quick_ref/textmessage](http://www.webopedia.com/quick_ref/textmessage) abbreviations.asp Grzegorz Chrupała. 2014. Normalizing tweets with edit scripts and recurrent neural embeddings. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL 2014), pages 680–686, Baltimore, USA. [↗](http://www.webopedia.com/quick_ref/textmessage) 4. Stephan Gouws, Dirk Hovy, and Donald Metzler. 2011. Unsupervised mining of lexical variants from noisy text. In Proceedings of the First workshop on Unsupervised Learning in NLP, pages 82–90, Edinburgh, UK. [find](https://scholar.google.com/scholar?q=Stephan+Gouws%2C+Dirk+Hovy%2C+and+Donald+Metzler%2E+2011%2E+Unsupervised+mining+of+lexical+variants+from+noisy+text%2E+In+Proceedings+of+the+First+workshop+on+Unsupervised+Learning+in+NLP%2C+pages+82%E2%80%9390%2C+Edinburg) 5. Bo Han and Timothy Baldwin. 2011. Lexical normalization of short text messages: Makn sens a #twitter. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (ACL 2011), pages 368-378, Portland, USA. [find](https://scholar.google.com/scholar?q=Bo+Han+and+Timothy+Baldwin%2E+2011%2E+Lexical+normalization+of+short+text+messages%3A+Makn+sens+a+%23twitter%2E+In+Proceedings+of+the+49th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28ACL+20) 6. John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional Random Fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning, ICML'01, pages 282–289. [find](https://scholar.google.com/scholar?q=John+D%2E+Lafferty%2C+Andrew+McCallum%2C+and+Fernando+C%2E+N%2E+Pereira%2E+2001%2E+Conditional+Random+Fields%3A+Probabilistic+models+for+segmenting+and+labeling+sequence+data%2E+In+Proceedings+of+the+Eighteenth+Interna) 7. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA Fei Liu, Fuliang Weng, Bingqing Wang, and Yang Liu. 2011. Insertion, deletion, or substitution? Normalizing text messages without pre-categorization nor supervision. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (ACL 2011), pages 71-76, Portland, USA. [find](https://scholar.google.com/scholar?q=Morgan+Kaufmann+Publishers+Inc%2E%2C+San+Francisco%2C+CA%2C+USA+Fei+Liu%2C+Fuliang+Weng%2C+Bingqing+Wang%2C+and+Yang+Liu%2E+2011%2E+Insertion%2C+deletion%2C+or+substitution%3F+Normalizing+text+messages+without+pre%2Dcategoriza) 8. Fei Liu, Fuliang Weng, and Xiao Jiang. 2012. A broad-coverage normalization system for social media language. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL 2012), pages 1035-1044, Jeju Island, Korea. [find](https://scholar.google.com/scholar?q=Fei+Liu%2C+Fuliang+Weng%2C+and+Xiao+Jiang%2E+2012%2E+A+broad%2Dcoverage+normalization+system+for+social+media+language%2E+In+Proceedings+of+the+50th+Annual+Meeting+of+the+Association+for+Computational+Linguistics) 9. Yi Yang and Jacob Eisenstein. 2013. A log-linear model for unsupervised text normalization. In Proceedings of Conference on Empirical Methods in Natura Language Processing (EMNLP), pages 61-72, Seattle, USA. [find](https://scholar.google.com/scholar?q=Yi+Yang+and+Jacob+Eisenstein%2E+2013%2E+A+log%2Dlinear+model+for+unsupervised+text+normalization%2E+In+Proceedings+of+Conference+on+Empirical+Methods+in+Natura+Language+Processing+%28EMNLP%29%2C+pages+61%2D72%2C+Seattl) *Updated 30 October 2026*