--- title: "NLTK: The Natural Language Toolkit — Bird 2006" description: "The paper describes a reworked version of NLTK, an open-source (GPL) collection of Python modules, corpus samples, tutorials and exercises for teaching…" authors: ["Steven Bird"] year: 2006 venue: "Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions" source: "https://aclanthology.org/P06-4018/" doi: "10.3115/1225403.1225421" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/bird-2006-nltk-natural-language-toolkit/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/bird-2006-nltk-natural-language-toolkit/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2006](https://md.research.elamur.ai/free/en/papers/2006/) / NLTK: The Natural Language Toolkit # NLTK: The Natural Language Toolkit **Authors:** [Steven Bird](https://md.research.elamur.ai/free/en/authors/steven-bird/) · **Year:** 2006 · **Venue:** Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/P06-4018/) · [PDF](https://aclanthology.org/P06-4018.pdf) · [DOI 10.3115/1225403.1225421](https://doi.org/10.3115/1225403.1225421) · [OpenAlex](https://openalex.org/W4251372957) **Citations:** 833 citations (openalex-snapshot · 2026-10-14) · [539](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary The paper describes a reworked version of NLTK, an open-source (GPL) collection of Python modules, corpus samples, tutorials and exercises for teaching and prototyping in computational linguistics. Earlier releases had grown complicated. NLTK 1.4 in particular tried to let independent tasks, such as tagging and stemming, share one representation through a blackboard architecture for tokens. That design forced programmers into clumsy code. The new release drops much of this complexity. It uses simpler linguistic data structures and newer Python language features. Through short interactive sessions, the author shows tokenization, Porter stemming, corpus readers, a range of taggers chained by backoff, regex-based chunking, and context-free and probabilistic parsing. The paper also explains how the toolkit helps a single-semester NLP course cover both theory and practice. It does this through graphical step-by-step algorithm demonstrations and assignments that grow in difficulty. The work matters because it lowers the entry barrier to hands-on NLP for students and researchers. ## Key points - The toolkit ships samples and readers for the Penn Treebank, the CMU Pronunciation Dictionary, the Brown Corpus, the NIST IEER Corpus, the CoNLL-2000 Chunking Corpus and the PP Attachment Corpus, and it also supports the SIL Shoebox format. - Taggers range from a default tagger (alone it gets only about 10–20% of tokens right) to regexp, unigram, bigram, Brill (contributed by Christopher Maloof) and HMM (contributed by Trevor Cohn). Accuracy is measured with `tag.accuracy()` against gold-standard text. - The parsers include recursive-descent, shift-reduce, chart parsing with a bottom-up strategy, a Viterbi-style PCFG parser and bottom-up probabilistic chart parsers. A demo sentence yields two parses because of PP-attachment ambiguity. - Chunking supports bracket and IOB notations, regular-expression chunk parsers, cascades of chunkers, and evaluation and error-analysis tools. - The rewrite was motivated by NLTK 1.4's blackboard token architecture, which made code awkward for programmers. ## Interesting ideas - **Backoff chains of taggers:** a bigram tagger falls back to a unigram tagger, which in turn falls back to regex rules. Every token then gets a tag while the statistical models are still used where they apply. The same cascade suits low-resource languages, where trained models leave many unknown words. - **Simplicity over a unified architecture:** the authors admit that a general shared-data design made the toolkit harder to use, and they rolled it back. This is a useful reminder when designing data pipelines. Plain, task-specific structures can serve users better than one elegant universal format. - **Self-running demo modes:** most components can perform a meaningful task without user input, and their state can be viewed step by step in a GUI. Shipping this kind of runnable illustration with any library or dataset helps onboarding and teaching. - **Graduated assignments built on existing modules:** students first use a module, then modify it (for example, turning a recursive-descent parser into a left-corner parser), then build and integrate new components. This staged path also works for training new contributors on a shared codebase. - **Corpus readers that know the file layout:** these readers load annotated data straight into native data structures. This removes repetitive parsing code and is worth copying for any family of in-house corpora. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) Topics: [Part-of-speech tagging](https://md.research.elamur.ai/free/en/topics/part-of-speech-tagging/) ## Cited by Cited by 539 papers of our library cite it (of the 140884 whose bibliographies we read); 36 of them have a page here, 503 not yet. - [SarcasmDet at SemEval-2022 Task 6: Detecting Sarcasm using Pre-trained Transformers in English and Arabic Languages](https://md.research.elamur.ai/free/en/papers/abdullah-2022-sarcasmdet-semeval-2022-task-6/) - [A unified approach to sentence segmentation of punctuated text in many languages](https://md.research.elamur.ai/free/en/papers/acl-2021-wicks-unified-approach-sentence-segmentation-punctuated/) - [Improving a Strong Neural Parser with Conjunction-Specific Features](https://md.research.elamur.ai/free/en/papers/aclbulk-2017-ficler-improving-strong-neural-parser-conjunction/) - [Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks](https://md.research.elamur.ai/free/en/papers/adi-2016-fine-grained-analysis-sentence-embeddings/) - [CCOHA: Clean Corpus of Historical American English](https://md.research.elamur.ai/free/en/papers/alatrash-2020-ccoha-clean-corpus-historical-american/) - [MacBERTh: Development and Evaluation of a Historically Pre-trained Language Model for English (1450-1950)](https://md.research.elamur.ai/free/en/papers/arevalo-2021-macberth-development-evaluation-historically-pre/) - [Dialogue Act Modeling in a Complex Task-Oriented Domain](https://md.research.elamur.ai/free/en/papers/boyer-2010-dialogue-act-modeling-complex-task/) - [Modeling Empathy and Distress in Reaction to News Stories](https://md.research.elamur.ai/free/en/papers/buechel-2018-modeling-empathy-distress-reaction-news/) - [Logical Natural Language Generation from Open-Domain Tables](https://md.research.elamur.ai/free/en/papers/chen-2020-logical-natural-language-generation-open/) - [Pragmatic inference of scalar implicature by LLMs](https://md.research.elamur.ai/free/en/papers/cho-2024-pragmatic-inference-scalar-implicature-llms/) - [NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation](https://md.research.elamur.ai/free/en/papers/dhole-2023-nl-augmenter-framework-task-sensitive/) - [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://md.research.elamur.ai/free/en/papers/du-2021-glam-efficient-scaling-language-models/) - [MultiUN: A Multilingual Corpus from United Nation Documents](https://md.research.elamur.ai/free/en/papers/eisele-2010-multiun-multilingual-corpus-united-nation/) - [UMBC_EBIQUITY-CORE: Semantic Textual Similarity Systems](https://md.research.elamur.ai/free/en/papers/han-2013-umbc-ebiquity-core-semantic-textual/) - [Jointly Predicting Predicates and Arguments in Neural Semantic Role Labeling](https://md.research.elamur.ai/free/en/papers/he-2018-jointly-predicting-predicates-arguments-neural/) - [SimLex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation](https://md.research.elamur.ai/free/en/papers/hill-2015-simlex-999-evaluating-semantic-models/) - [How Do Cultural Differences Impact the Quality of Sarcasm Annotation?: A Case Study of Indian Annotators and American Text](https://md.research.elamur.ai/free/en/papers/joshi-2016-how-do-cultural-differences-impact/) - [Sarcasm Target Identification: Dataset and An Introductory Approach](https://md.research.elamur.ai/free/en/papers/joshi-2018-sarcasm-target-identification-dataset-introductory/) - [Datasets: A Community Library for Natural Language Processing](https://md.research.elamur.ai/free/en/papers/lhoest-2021-datasets-community-library-natural-language/) - [A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation](https://md.research.elamur.ai/free/en/papers/liu-2022-token-level-reference-free-hallucination/) - [Hindsight Quality Prediction Experiments in Multi-Candidate Human-Post-Edited Machine Translation](https://md.research.elamur.ai/free/en/papers/marmonier-2026-hindsight-quality-prediction-experiments-multi/) - [MUSS: Multilingual Unsupervised Sentence Simplification by Mining Paraphrases](https://md.research.elamur.ai/free/en/papers/martin-2022-muss-multilingual-unsupervised-sentence-simplification/) - [DTED: Evaluation of Machine Translation Structure Using Dependency Parsing and Tree Edit Distance](https://md.research.elamur.ai/free/en/papers/mccaffery-2016-dted-evaluation-machine-translation-structure/) - [Document-Level Neural Machine Translation with Hierarchical Attention Networks](https://md.research.elamur.ai/free/en/papers/miculicich-2018-document-level-neural-machine-translation/) - [Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation](https://md.research.elamur.ai/free/en/papers/moryossef-2019-step-step-separating-planning-realization/) - [Interlinear Glosses as a Multilingual Pivot for Machine Translation: An Updated Study on Turkish with Restricted Resources](https://md.research.elamur.ai/free/en/papers/ozer-2026-interlinear-glosses-as-multilingual-pivot/) - [Improving Implicit Discourse Relation Recognition Through Feature Set Optimization](https://md.research.elamur.ai/free/en/papers/park-2012-improving-implicit-discourse-relation-recognition/) - [Automatic Prompt Optimization with “Gradient Descent” and Beam Search](https://md.research.elamur.ai/free/en/papers/pryzant-2023-automatic-prompt-optimization-gradient-descent/) - [Improving the Inference of Implicit Discourse Relations via Classifying Explicit Discourse Connectives](https://md.research.elamur.ai/free/en/papers/rutherford-2015-improving-inference-implicit-discourse-relations/) - [PySBD: Pragmatic Sentence Boundary Disambiguation](https://md.research.elamur.ai/free/en/papers/sadvilkar-2020-pysbd-pragmatic-sentence-boundary-disambiguation/) - [WANLP 2021 Shared-Task: Towards Irony and Sentiment Detection in Arabic Tweets using Multi-headed-LSTM-CNN-GRU and MaRBERT](https://md.research.elamur.ai/free/en/papers/salam-2021-wanlp-2021-shared-task-irony/) - [DREAM: A Challenge Data Set and Models for Dialogue-Based Reading Comprehension](https://md.research.elamur.ai/free/en/papers/sun-2019-dream-challenge-data-set-models/) - [MAVEN: A Massive General Domain Event Detection Dataset](https://md.research.elamur.ai/free/en/papers/wang-2020-maven-massive-general-domain-event/) - [Mind the GAP: A Balanced Corpus of Gendered Ambiguous Pronouns](https://md.research.elamur.ai/free/en/papers/webster-2018-mind-gap-balanced-corpus-gendered/) - [Challenges in Data-to-Document Generation](https://md.research.elamur.ai/free/en/papers/wiseman-2017-challenges-data-document-generation/) - [SciDTB: Discourse Dependency TreeBank for Scientific Abstracts](https://md.research.elamur.ai/free/en/papers/yang-2018-scidtb-discourse-dependency-treebank-scientific/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{bird2006nltk, title = {{NLTK: The Natural Language Toolkit}}, author = {Bird, Steven}, booktitle = {Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions}, year = {2006}, doi = {10.3115/1225403.1225421}, url = {https://doi.org/10.3115/1225403.1225421} } ``` - [← previous N-gram-based Machine Translation](https://md.research.elamur.ai/free/en/papers/marino-2006-n-gram-based-machine-translation/) - [next → Noun Phrase Chunking in Hebrew: Influence of Lexical and Morphological Features](https://md.research.elamur.ai/free/en/papers/aclbulk-2006-goldberg-noun-phrase-chunking-hebrew-influence/) - [↑ back to papers of 2006](https://md.research.elamur.ai/free/en/papers/2006/) ## References 3 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Marti Hearst. 2005. Teaching applied natural language processing: Triumphs and tribulations. In Proc 2nd ACL Workshop on Effective Tools and Methodologies for Teaching NLP and CL, pages 1–8, ACL Elizabeth Liddy and Nancy McCracken. 2005. Hands-on NLP for an interdisciplinary audience. In Proc 2nd AC… [find](https://scholar.google.com/scholar?q=Marti+Hearst%2E+2005%2E+Teaching+applied+natural+language+processing%3A+Triumphs+and+tribulations%2E+In+Proc+2nd+ACL+Workshop+on+Effective+Tools+and+Methodologies+for+Teaching+NLP+and+CL%2C+pages+1%E2%80%938%2C+ACL+Eliza) 2. David Beasley. 2006. Python Essential Reference, 3rd Edition. Sams. [find](https://scholar.google.com/scholar?q=David+Beasley%2E+2006%2E+Python+Essential+Reference%2C+3rd+Edition%2E+Sams%2E) 3. Rune Sætre, Amund Tveit, Tonje S. Steigedal, and Astrid Lægreid. 2005. Semantic annotation of biomedical literature using Google. In Data Mining and Bioinformatics Workshop, volume 3482 of Lecture Notes in Computer Science. Springer. [find](https://scholar.google.com/scholar?q=Rune+S%C3%A6tre%2C+Amund+Tveit%2C+Tonje+S%2E+Steigedal%2C+and+Astrid+L%C3%A6greid%2E+2005%2E+Semantic+annotation+of+biomedical+literature+using+Google%2E+In+Data+Mining+and+Bioinformatics+Workshop%2C+volume+3482+of+Lecture+Not)