---
title: "‘Indicatements' that character language models learn English"
description: "The authors train a 'wordless' character LSTM LM (spaces are ordinary symbols) on 7M characters of… 13 references, cited by 5 papers here."
authors: ["Yova Kementchedjhieva", "Adam Lopez"]
year: 2018
venue: "Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP"
source: "https://aclanthology.org/W18-5417/"
doi: "10.18653/v1/w18-5417"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kementchedjhieva-2018-indicatements-that-character-language-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kementchedjhieva-2018-indicatements-that-character-language-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/6/) / ‘Indicatements' that character language models learn English morpho-syntactic units and regularities

# ‘Indicatements' that character language models learn English morpho-syntactic units and regularities

**Authors:** [Yova Kementchedjhieva](https://md.research.elamur.ai/free/en/authors/yova-kementchedjhieva/) and [Adam Lopez](https://md.research.elamur.ai/free/en/authors/adam-lopez/) · **Year:** 2018 · **Venue:** [Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP](https://md.research.elamur.ai/free/en/venues/emnlp-workshop-blackboxnlp-analyzing-and-interpreting-neural-networks-for-nlp/) ([EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP 2018](https://md.research.elamur.ai/free/en/venues/emnlp-workshop-blackboxnlp-analyzing-and-interpreting-neural-networks-for-nlp/2018/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/W18-5417/) · [PDF](https://aclanthology.org/W18-5417.pdf) · [DOI 10.18653/v1/w18-5417](https://doi.org/10.18653/v1/w18-5417) · [OpenAlex](https://openalex.org/W2891343966)

**Citations:** 12 citations (OpenAlex, 2026-10-14) · [5](https://md.research.elamur.ai/free/en/papers/kementchedjhieva-2018-indicatements-that-character-language-models/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper asks whether a character-level language model that reads raw, unsegmented English actually picks up morphology, or only seems to because it shares parameters across similar strings. The authors train a 'wordless' character LSTM LM (spaces are ordinary symbols) on 7M characters of English Wikipedia and examine it in several ways. They inspect sampled nonce words, look for interpretable hidden units, attach a frozen-encoder segmentation probe (C2M) and a POS probe (C2T), and measure how likely derivational suffixes are after bases of different word classes.

The LM produces plausible over-regularized forms such as *breaked* and *indicatement*. It also develops a single "word unit" that fires at word ends and at word-like bases inside longer words. Morpheme boundaries appear to be learned mainly by extending knowledge of word boundaries. The LM respects the selectional restrictions of most suffixes tested. This matters because it gives direct evidence for, and a limit on, claims that character models make morphological analysis unnecessary. The mechanism should work well for English but less well for languages whose morphemes rarely coincide with words.

## Key points
- **Model and training:** a one-layer LSTM with 256 hidden units, learning rate 0.003, minibatch 50 and dropout 0.2, trained on the first 7M characters of English Wikipedia (90/10 train/dev split, not lowercased).
- **Sampled text:** roughly 1 in 20 sampled tokens is a nonce word, checked against the training data and the pyenchant dictionaries.
- **Hidden units:** about 40 units looked interpretable. Examples include a punctuation unit, a Latinate *-ion* suffix unit and a word-boundary unit. The word unit's activation correlates only 0.08 with the predicted probability of a following space, so it is not simply a space predictor.
- **Segmentation probe:** C2M was trained on gold segmentations from MorphoChallenge 2010 and Hutmegs 1.0 (2275 word forms), using the LM's frozen encoder. It reached an overall F1 of 53.3. Scores per boundary type ranged widely, from 27.6 to 90.9.
- **Suffix selection:** 11 of 19 derivational suffixes were more probable after bases of the correct class, among them *-ment*, *-able*, *-ness* and *-ity*. The *-ion* errors trace to frequency: the suffix appears in 18,945 words, while its bases appear alone only about 9,755 times, so the LM treats bases like *populat* as nominal.

## Interesting ideas
- **Frozen-encoder probes:** attach small task heads (segmentation, tagging) to a frozen pretrained encoder, so the head's score measures what the encoder already knows rather than what fine-tuning adds.
- **Over-regularization as a diagnostic:** nonce forms like *breaked* or *indicatement* show that a model applies a regular rule which a lexical exception should have blocked. Collecting such errors shows which rules were learned and which exceptions are missing.
- **Ruling out a trivial explanation:** the authors checked that the word unit is not just a "space comes next" detector by correlating its activation with the model's space probability.
- **Selectional restrictions via conditional probability:** they tested whether a suffix is more probable after bases of its required word class, which turns a morpho-syntactic constraint into a measurable score.
- **Frequency explains category drift:** a base seen far more often with a suffix than without it gets absorbed into the suffix's word class.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Use frozen-encoder probes on gparse's transformer layers to check which layers carry morphology versus syntax before deciding where to tap features.
- **I2** (tokens): Feed generated or misspelled forms to the lemmatizer and flag cases where a regular analysis overrides a known irregular, as tests for the exception tables.
- **I3** (learn): When a feature or repair rule seems to capture a linguistic notion, add a negative control that correlates it with the obvious surface cue it might be proxying.
- **I4** (world): Apply the same contrast to compiled knowledge, such as which derivational suffixes attach to which verb classes, to validate WordNet/VerbNet-derived derivation links.
- **I5** (tokens): When the tagger mislabels a bound base, check base-alone versus base+suffix counts in the corpus before blaming the model.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Edinburgh](https://md.research.elamur.ai/free/en/universities/university-of-edinburgh/) — University, Edinburgh, United Kingdom · 381 works · 80,747 citations
- [University of Copenhagen](https://md.research.elamur.ai/free/en/universities/university-of-copenhagen/) — University, Copenhagen, Denmark · 116 works · 15,292 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,379 works · [🧩 Morphology](https://md.research.elamur.ai/free/en/topics/morphology/) · 420 works

## Cite

Cite as: Yova Kementchedjhieva and Adam Lopez. 2018. ‘Indicatements' that character language models learn English morpho-syntactic units and regularities. Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{kementchedjhieva2018indicatements,
  title = {{‘Indicatements' that character language models learn English morpho-syntactic units and regularities}},
  author = {Yova Kementchedjhieva and Adam Lopez},
  booktitle = {Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP},
  year = {2018},
  doi = {10.18653/v1/w18-5417},
  url = {https://doi.org/10.18653/v1/w18-5417}
}
```

- [← previous Multi-Relational Question Answering from Narratives: Machine Reading and Reasoning in Simulated Worlds](https://md.research.elamur.ai/free/en/papers/labutov-2018-multi-relational-question-answering-narratives/)
- [next → Sarcasm Target Identification: Dataset and An Introductory Approach](https://md.research.elamur.ai/free/en/papers/joshi-2018-sarcasm-target-identification-dataset-introductory/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/6/)

## References

13 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals. 2016. Listen, attend and spell: A neural network for large vocabulary conversational speech recognition. In Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on, pages 4960–4964. IEEE. [find](https://scholar.google.com/scholar?q=William+Chan%2C+Navdeep+Jaitly%2C+Quoc+Le%2C+and+Oriol+Vinyals%2E+2016%2E+Listen%2C+attend+and+spell%3A+A+neural+network+for+large+vocabulary+conversational+speech+recognition%2E+In+Acoustics%2C+Speech+and+Signal+Proce)
2. Linlin Wang, Zhu Cao, Yu Xia, and Gerard de Melo. 2016. Morphological segmentation with window lstm neural networks. In AAAI, pages 2842–2848. [find](https://scholar.google.com/scholar?q=Linlin+Wang%2C+Zhu+Cao%2C+Yu+Xia%2C+and+Gerard+de+Melo%2E+2016%2E+Morphological+segmentation+with+window+lstm+neural+networks%2E+In+AAAI%2C+pages+2842%E2%80%932848%2E)
3. Ron J Weiss, Jan Chorowski, Navdeep Jaitly, Yonghui Wu, and Zhifeng Chen. 2017. Sequence-to-sequence models can directly translate foreign speech. arXiv preprint arXiv:1703.08581. [↗](https://arxiv.org/abs/1703.08581)
4. Junyoung Chung, Kyunghyun Cho, and Yoshua Bengio. 2016. [A character-level decoder without explicit segmentation for neural machine translation](https://md.research.elamur.ai/free/en/papers/chung-2016-character-level-decoder-without-explicit/). arXiv preprint arXiv:1603.06147. [↗](https://arxiv.org/abs/1603.06147)
5. Nigel Fabb. 1988. English suffixation is constrained only by selectional restrictions. Natural Language and Linguistic Theory, 6(4):527–539. [↗](https://doi.org/10.1007/bf00134491)
6. Akos Kadar, Grzegorz Chrupala, and Afra Alishahi. 2016. [Representation of linguistic form and function in recurrent neural networks](https://md.research.elamur.ai/free/en/papers/kadar-2017-representation-linguistic-form-function-recurrent/). [↗](https://doi.org/10.1162/coli_a_00300)
7. Ákos Kádár, Marc-Alexandre Côté, Grzegorz Chrupała, and Afra Alishahi. 2018. Revisiting the hierarchical multiscale lstm. arXiv preprint arXiv:1807.03595. [↗](https://arxiv.org/abs/1807.03595)
8. Andrej Karpathy. 2015. The unreasonable effectiveness of recurrent neural networks. Andrej Karpathy blog. [find](https://scholar.google.com/scholar?q=Andrej+Karpathy%2E+2015%2E+The+unreasonable+effectiveness+of+recurrent+neural+networks%2E+Andrej+Karpathy+blog%2E)
9. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. 2016. [Character-aware neural language models](https://md.research.elamur.ai/free/en/papers/kim-2015-character-aware-neural-language-models/). In Proceedings of AAAI, pages 2741–2749.
10. Jason Lee, Kyunghyun Cho, and Thomas Hofmann. 2016. [Fully Character-Level Neural Machine Translation without Explicit Segmentation](https://md.research.elamur.ai/free/en/papers/lee-2017-fully-character-level-neural-machine/). Acl-2016, pages 1693–1703. [↗](https://doi.org/10.1162/tacl_a_00067)
11. Peng Qian, Xipeng Qiu, and Xuanjing Huang. 2016. Analyzing linguistic knowledge in sequential model of sentence. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 826–835. [find](https://scholar.google.com/scholar?q=Peng+Qian%2C+Xipeng+Qiu%2C+and+Xuanjing+Huang%2E+2016%2E+Analyzing+linguistic+knowledge+in+sequential+model+of+sentence%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natural+Language+Processi)
12. Teemu Ruokolainen, Oskar Kohonen, Sami Virpioja, and Mikko Kurimo. 2013. Supervised morphological segmentation in a low-resource learning setting using conditional random fields. In CoNLL, pages 29–37. [find](https://scholar.google.com/scholar?q=Teemu+Ruokolainen%2C+Oskar+Kohonen%2C+Sami+Virpioja%2C+and+Mikko+Kurimo%2E+2013%2E+Supervised+morphological+segmentation+in+a+low%2Dresource+learning+setting+using+conditional+random+fields%2E+In+CoNLL%2C+pages+29%E2%80%9337)
13. Cicero Nogueira dos Santos and Bianca Zadrozny. 2014. Learning Character-level Representations for Part-of-Speech Tagging. Proceedings of the 31st International Conference on Machine Learning, ICML-14(2011):1818–1826. [find](https://scholar.google.com/scholar?q=Cicero+Nogueira+dos+Santos+and+Bianca+Zadrozny%2E+2014%2E+Learning+Character%2Dlevel+Representations+for+Part%2Dof%2DSpeech+Tagging%2E+Proceedings+of+the+31st+International+Conference+on+Machine+Learning%2C+ICML%2D14)

*Updated 1 November 2026*