--- title: "Two Database Resources for Processing Social Media English Text" description: "The authors present two hand-built resources. 7 references. Among them: 3." authors: ["Eleanor Clark", "Kenji Araki"] year: 2012 venue: "Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)" source: "https://aclanthology.org/L12-1124/" licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)" licence_url: "https://creativecommons.org/licenses/by-nc/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/clark-2012-two-database-resources-processing-social/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/clark-2012-two-database-resources-processing-social/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2012](https://md.research.elamur.ai/free/en/papers/2012/page/3/) / Two Database Resources for Processing Social Media English Text # Two Database Resources for Processing Social Media English Text **Authors:** Eleanor Clark and [Kenji Araki](https://md.research.elamur.ai/free/en/authors/kenji-araki/) · **Year:** 2012 · **Venue:** [Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)](https://md.research.elamur.ai/free/en/venues/lrec/) ([LREC 2012](https://md.research.elamur.ai/free/en/venues/lrec/2012/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/) **Links:** [Publisher's page](https://aclanthology.org/L12-1124/) · [PDF](https://aclanthology.org/L12-1124.pdf) · [OpenAlex](https://openalex.org/W2250939229) **Citations:** 3 citations (OpenAlex, 2026-10-14) ## Summary Casual English on Twitter and YouTube is full of shortened forms, acronyms, dropped apostrophes, slang and emoticons. This noise hurts machine translation and makes posts hard for non-native readers to follow. The authors present two hand-built resources. The first is the CECS lexicon, with 1,255 entries. A rule-based normalizer uses it, looking up tokens and short phrases in a trie and swapping in standard English. Each entry carries a category label and an explanatory note. On 100 tweets translated into Japanese with Google Translate and Systran, normalizing first sharply reduced the number of words the MT systems left untranslated. Ten learners of English also reported clearly better comprehension. The second resource is the CEGS phoneme database, which runs in the opposite direction. It maps each CMU Pronouncing Dictionary phoneme to slang-like spellings, so a generator can turn ordinary text into plausible "tweet-speak". This generator has not yet been evaluated. Together, the two resources show that a small, curated, explainable lexicon can make a practical difference with noisy social-media input. ## Key points - **CECS database:** 1,255 manually verified entries (single words and mostly 2–3-word phrases) drawn from Twitter and YouTube comments. Meanings were checked against Wiktionary and Urban Dictionary. Each entry has four fields: error word, regular word, category and notes. - **Eight categories of casual English:** shortform abbreviation, acronym, typo/misspelling, punctuation omission, non-dictionary slang (including slang senses of standard words), wordplay, censor avoidance, and emoticons. - **MT evaluation:** 100 tweets from Choudhury et al.'s corpus were translated English→Japanese with Google Translate and Systran, before and after CECS. Errors were counted by hand as non-translated words (NTW) and wrongly translated words (WTW). - NTW fell from about 3.3 to roughly 0.8–0.9 per sentence (average sentence length 15.35 words). - WTW also dropped, from about 1.2 to about 0.7. - **Reader study:** 10 non-native learners aged 23–64 rated 20 tweets on a 1–5 understanding scale. The mean rose from 2.89 (raw) to 3.89 (normalized), and the weakest learners gained the most. - **CEGS:** - Maps the 39 CMU phonemes to alternative spellings, chosen at random among candidates. - Pre-filters with a reversed slice of the CECS database. - Selects 22% of eligible tokens for conversion. This rate comes from manual analysis of 320 tweets, which found 21.67% casual tokens per sentence. ## Interesting ideas - Keying a trie on the same tokenizer's output allows multi-word lexicon entries, so slang phrases are matched as units rather than word by word. - Ambiguous shortforms like "ur" are resolved by storing small context bigrams ("ur so" → you're, "ur friend" → your) instead of a statistical model, which keeps every decision traceable. - Every lexicon entry stores a category and a human-readable note on its origin, so the system can explain each replacement to the user, not just make it. - A cheap, task-grounded metric was used: counting words the downstream system simply failed to process (NTW), alongside outright wrong outputs. - The reverse generator corrupts clean text at the phoneme level, at a realistic rate measured from real tweets (about 22% of tokens). ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (tokens): A normalization pass before the tagger in crate en could use one shared tokenizer for both lexicon and input, so phrase entries never drift out of alignment with real token boundaries. - **I2** (tokens): Context-keyed rewrite entries could resolve "ur", "its", "im" and similar forms before lemmatization, and each rewrite would carry a visible rule ID in the explanation trace. - **I3** (world): Category and provenance notes on normalized tokens could flow into the text's world model, recording that a speaker used a censored vulgarity or an emoticon as a stance marker that later stages can cite. - **I4** (learn): An "unhandled token" counter per sentence (OOV for the tagger, unattached nodes in the tree) before and after a preprocessing step would give a clear, explainable signal of whether a repair rule helps. - **I5** (learn): Phoneme-based noising of clean treebank sentences at a calibrated rate could create noisy-input test sets, where the gold tree is known, to check the parsers' robustness to social-media spelling. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Hokkaido University](https://md.research.elamur.ai/free/en/universities/hokkaido-university/) — University, Sapporo, Japan · 4 works · 17 citations 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🖱️ Human-Computer Interaction](https://md.research.elamur.ai/free/en/shelves/computer-science/human-computer-interaction/) · 141 works 🧭 Topics (1): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,263 works ## Cite Cite as: Eleanor Clark and Kenji Araki. 2012. Two Database Resources for Processing Social Media English Text. Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{clark2012two, title = {{Two Database Resources for Processing Social Media English Text}}, author = {Eleanor Clark and Kenji Araki}, booktitle = {Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)}, year = {2012}, doi = {10.63317/2hig8zibqoyk}, url = {https://doi.org/10.63317/2hig8zibqoyk} } ``` - [← previous The Theophoric Element Ba'Al in Ancient Phoenician Inscriptions](https://md.research.elamur.ai/free/en/papers/aurelian-2012-theophoric-element-ba-al-ancient/) - [next → A Tool for Extracting Conversational Implicatures](https://md.research.elamur.ai/free/en/papers/tatu-2012-tool-extracting-conversational-implicatures/) - [↑ back to papers of 2012](https://md.research.elamur.ai/free/en/papers/2012/page/3/) ## References 7 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Bird, S; Klein, E. and Loper, E. (2009), Natural Language Processing with Python. O'Reilly Media. [find](https://scholar.google.com/scholar?q=Bird%2C+S%3B+Klein%2C+E%2E+and+Loper%2C+E%2E+%282009%29%2C+Natural+Language+Processing+with+Python%2E+O%27Reilly+Media%2E) 2. Choudhury, M. D.; Lin, Y.R.; Sundaram, H.; Candan, K, S.; Xie, L. and A. Kelliher (2010). How does the sampling strategy impact the discovery of information diffusion in social media? In Proceedings of the 4th International Conference on Weblogs and Social Media, Washington DC, USA, May 2010. [find](https://scholar.google.com/scholar?q=Choudhury%2C+M%2E+D%2E%3B+Lin%2C+Y%2ER%2E%3B+Sundaram%2C+H%2E%3B+Candan%2C+K%2C+S%2E%3B+Xie%2C+L%2E+and+A%2E+Kelliher+%282010%29%2E+How+does+the+sampling+strategy+impact+the+discovery+of+information+diffusion+in+social+media%3F+In+Proceedings+o) 3. Clark, E and Araki, K. (2011) Text Normalization in Social Media: Progress, Problems and Applications for a Pre-Processing System of Casual English. In Proceedings of the Twelfth Conference of the Pacific Association of Computational Linguistics. [find](https://scholar.google.com/scholar?q=Clark%2C+E+and+Araki%2C+K%2E+%282011%29+Text+Normalization+in+Social+Media%3A+Progress%2C+Problems+and+Applications+for+a+Pre%2DProcessing+System+of+Casual+English%2E+In+Proceedings+of+the+Twelfth+Conference+of+the+Pac) 4. Kuala Lumpur, Malaysia, July 2011. [find](https://scholar.google.com/scholar?q=Kuala+Lumpur%2C+Malaysia%2C+July+2011%2E) 5. Henriquez, C. A. and Hernandez, A. (2009) A ngram-based statistical machine translation approach for text normalization on chat-speak style communications. In Proceedings of CAW2.0, Madrid, Spain, August 2009, pp.1–5. [find](https://scholar.google.com/scholar?q=Henriquez%2C+C%2E+A%2E+and+Hernandez%2C+A%2E+%282009%29+A+ngram%2Dbased+statistical+machine+translation+approach+for+text+normalization+on+chat%2Dspeak+style+communications%2E+In+Proceedings+of+CAW2%2E0%2C+Madrid%2C+Spain%2C+Aug) 6. Ritter, A; Cherry, C. and Dolan, B. [Unsupervised modeling of Twitter Conversations](https://md.research.elamur.ai/free/en/papers/ritter-2010-unsupervised-modeling-twitter-conversations/). In Proceedings of HLT-NAACL 2010, Los Angeles, California, June 2010, pp. 172–180. 7. Wong, W; Liu, W; and Bennamoun, M. (2007) Enhanced integrated scoring for cleaning dirty texts. In Proceedings of IJCAI 2007 Workshop on Analytics for Noisy Unstructured Text Data, Hyderabad, India, January 2007, pp. 55–6. [↗](https://doi.org/10.48550/arxiv.0810.0332) *Updated 1 November 2026*