--- title: "A Hidden Markov Model Based System for Entity Extraction from" description: "The paper describes a named entity recognizer for English tweets, built for ESM-IL, a FIRE 2015 shared… 5 references, cited by 1 paper here." authors: ["Kamal Sarkar"] year: 2015 venue: "arXiv:1512.03950" source: "https://arxiv.org/abs/1512.03950" doi: "10.48550/arxiv.1512.03950" arxiv: "1512.03950" licence: "arXiv — non-exclusive license to distribute" licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/sarkar-2015-hidden-markov-model-based-system/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sarkar-2015-hidden-markov-model-based-system/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2015](https://md.research.elamur.ai/free/en/papers/2015/page/3/) / A Hidden Markov Model Based System for Entity Extraction from Social Media English Text at FIRE 2015 # A Hidden Markov Model Based System for Entity Extraction from Social Media English Text at FIRE 2015 **Authors:** Kamal Sarkar · **Year:** 2015 · **Venue:** arXiv:1512.03950 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/) **Links:** [Publisher's page](https://arxiv.org/abs/1512.03950) · [PDF](https://arxiv.org/pdf/1512.03950) · [DOI 10.48550/arxiv.1512.03950](https://doi.org/10.48550/arxiv.1512.03950) · [arXiv 1512.03950](https://arxiv.org/abs/1512.03950) · [OpenAlex](https://openalex.org/W2263524470) **Citations:** 5 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 141,840 papers whose bibliographies we read ## Summary The paper describes a named entity recognizer for English tweets, built for ESM-IL, a FIRE 2015 shared task on identifying entities in social-media posts written in Indian languages. The task covers a fine-grained inventory of entity types, including person, location, organization, artifact, facilities, money, date, period, and living thing. The author converts the organizers' offset-based annotation files into IOB format and trains a trigram Hidden Markov Model. The main twist is the observation unit. Each token is not observed as a bare word but as a composite symbol with three parts: - the word itself; - either its POS tag from MontyTagger or, if the word appears in a gazetteer, a gazetteer tag; - a rule-derived "meta tag" for surface shape, such as capitalization, hashtag, @-mention, colon, or hyphen. Unseen symbols are scored through suffix statistics of rare training symbols. Decoding uses Viterbi. A small post-processing rule then removes malformed entity spans. The system ranked first among English submissions with P/R/F of 61.96/39.46/48.21. This shows that a cheap generative model enriched with lexical resources and shape features can compete on noisy social-media text. ## Key points - The model is a trigram HMM. Transition probabilities are smoothed by deleted interpolation over trigram, bigram, and unigram tag counts, and the best sequence is found with Viterbi. - Each observation symbol is a triplet ``, where the X-tag is a MontyTagger POS tag or a gazetteer tag. Gazetteer lists include first names (Bperson), later name parts (Iperson), facilities, months, days, periods, counts, and money. - Meta tags come from ordered hand rules. Examples: - ICAP for an initial capital; - ABBR for abbreviations; - CHAS/HASH for hashtags; - ATSY for @-mentions; - CCOL/COLN for a trailing colon; - CHYP/HYPH for hyphens; - ALDT for all-digit tokens. - Unknown triplets are handled by appending the X-tag and meta-tag to the word to form a pseudo word. Its probability is then estimated by suffix analysis over rare pseudo words (frequency ≤ 2) in training. - The English data lists the figures 11,003 and 9641 in the training/test columns, with 21 NE types. The official result was F = 48.21, ahead of the next-best English run at 42.64. ## Interesting ideas - Packing extra evidence (a POS or lexicon tag plus a shape class) into one composite observed symbol lets a plain HMM use rich features without changing its equations or decoder. - When a token hits a gazetteer, the lexicon tag replaces the POS tag. This gives the resource priority over a statistical guess, and the switch stays visible in the output. - Appending the feature tags to the word before running suffix analysis makes the unknown-word model sensitive to both morphology and context class at once. - A deterministic post-check removes any I-XXX run that starts directly after O. The author treats such runs as model errors, not as broken entities to be repaired. - Shape features specific to social media (hashtags, @-mentions, colons after names) are encoded as a small ordered rule cascade, which is trivial to audit. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (tokens): The `en` tagger could use composite keys like word+shape+lexicon class as a cheap, fully inspectable backoff signal for noisy or out-of-domain input. - **I2** (world): Entity typing in the text world could record "typed by lexicon hit" versus "typed by model" as separate provenance, which keeps each entity's class explainable. - **I3** (tokens): Unknown-word handling in the lemmatizer/tagger could condition suffix statistics on shape and lexicon class, not on the raw string alone. - **I4** (learn): This works as a self-induced repair rule. Its effect can be checked against a negative control: compare the rule's deletions with gold spans to confirm it removes more false entities than true ones. - **I5** (tokens): The tokenizer could emit an explicit shape tag for each token, so later stages can tell a Twitter handle from an ordinary proper noun. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Jadavpur University](https://md.research.elamur.ai/free/en/universities/jadavpur-university/) — University, Kolkata, India · 4 works · 310 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works 🧭 Topics (1): [English](https://md.research.elamur.ai/free/en/topics/english/) · 1,373 works ## Cite Cite as: Kamal Sarkar. 2015. A Hidden Markov Model Based System for Entity Extraction from Social Media English Text at FIRE 2015. arXiv:1512.03950. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{sarkar2015hidden, title = {{A Hidden Markov Model Based System for Entity Extraction from Social Media English Text at FIRE 2015}}, author = {Kamal Sarkar}, year = {2015}, doi = {10.48550/arxiv.1512.03950}, eprint = {1512.03950}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.1512.03950} } ``` - [← previous Prepositional Phrase Attachment Problem Revisited: How VERBNET Can Help](https://md.research.elamur.ai/free/en/papers/bailey-2015-prepositional-phrase-attachment-problem-revisited/) - [next → A Frobenius Model of Information Structure in Categorical Compositional Distributional Semantics](https://md.research.elamur.ai/free/en/papers/kartsaklis-2015-frobenius-model-information-structure-categorical/) - [↑ back to papers of 2015](https://md.research.elamur.ai/free/en/papers/2015/page/3/) ## References 5 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Grishman, R. 1995. The New York University System MUC-6 or Where’s the syntax? In Proceedings of the Sixth Message Understanding Conference. [find](https://scholar.google.com/scholar?q=Grishman%2C+R%2E+1995%2E+The+New+York+University+System+MUC%2D6+or+Where%E2%80%99s+the+syntax%3F+In+Proceedings+of+the+Sixth+Message+Understanding+Conference%2E) 2. McDonald, D. D. 1996. Internal and external evidence in the identification and semantic categorization of proper names. In B. Boguraev and J. Pustejovsky, editors, Corpus Processing for Lexical Acquisition, 21–39. [find](https://scholar.google.com/scholar?q=McDonald%2C+D%2E+D%2E+1996%2E+Internal+and+external+evidence+in+the+identification+and+semantic+categorization+of+proper+names%2E+In+B%2E+Boguraev+and+J%2E+Pustejovsky%2C+editors%2C+Corpus+Processing+for+Lexical+Acquis) 3. Wakao, T., Gaizauskas, R. and Wilks, Y. 1996. Evaluationof an algorithm for the recognition and classification of proper names. In Proceedings of COLING-96. [find](https://scholar.google.com/scholar?q=Wakao%2C+T%2E%2C+Gaizauskas%2C+R%2E+and+Wilks%2C+Y%2E+1996%2E+Evaluationof+an+algorithm+for+the+recognition+and+classification+of+proper+names%2E+In+Proceedings+of+COLING%2D96%2E) 4. Bikel, D. M., Miller, S., Schwartz, R. and Weischedel, R. 1997. [Nymble: A High Performance Learning Name-finder](https://md.research.elamur.ai/free/en/papers/bikel-1997-nymble-high-performance-learning-name/). In Proceedings of the Fifth Conference on Applied Natural Language Processing, 194–201. [↗](https://doi.org/10.48550/arxiv.cmp-lg/9803003) 5. Borthwick, A. 1999. A Maximum Entropy Approach to Named Entity Recognition. Ph.D. thesis, Computer Science Department, New York University. English Malayalam [find](https://scholar.google.com/scholar?q=Borthwick%2C+A%2E+1999%2E+A+Maximum+Entropy+Approach+to+Named+Entity+Recognition%2E+Ph%2ED%2E+thesis%2C+Computer+Science+Department%2C+New+York+University%2E+English+Malayalam) *Updated 24 October 2026*