---
title: "Effective Use of Word Order for Text — Johnson 2015"
description: "The paper asks how text classifiers can benefit from word order when bag-of-words models discard it and adding n-grams often fails to help, especially…"
authors: ["Rie Johnson", "Tong Zhang"]
year: 2015
venue: "Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies"
source: "https://aclanthology.org/N15-1011/"
doi: "10.3115/v1/n15-1011"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/johnson-2015-effective-use-word-order-text/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/johnson-2015-effective-use-word-order-text/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2015](https://md.research.elamur.ai/free/en/papers/2015/) / Effective Use of Word Order for Text Categorization with Convolutional Neural Networks

# Effective Use of Word Order for Text Categorization with Convolutional Neural Networks

**Authors:** Rie Johnson, [Tong Zhang](https://md.research.elamur.ai/free/en/authors/tong-zhang/) · **Year:** 2015 · **Venue:** Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/N15-1011/) · [PDF](https://aclanthology.org/N15-1011.pdf) · [DOI 10.3115/v1/n15-1011](https://doi.org/10.3115/v1/n15-1011) · [OpenAlex](https://openalex.org/W2963921497)

**Citations:** 864 citations (openalex-snapshot · 2026-10-14) · [92](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
The paper asks how text classifiers can benefit from word order when bag-of-words models discard it and adding n-grams often fails to help, especially for topic categorization. Johnson and Zhang feed high-dimensional one-hot word vectors directly into a convolutional neural network instead of first looking up low-dimensional word vectors. The convolution layer therefore learns embeddings of short text regions directly. Efficient sparse computation on GPU makes this practical. They propose two variants. seq-CNN concatenates the one-hot vectors inside each region. bow-CNN collapses each region into a bag of words, which cuts the parameter count. A "parallel" extension combines several convolution layers so that different region embeddings complement each other. In experiments on sentiment data (e.g., Elec) and topic data (e.g., RCV1), seq-CNN performs better on sentiment and bow-CNN on topics. The better variant generally beats bag-of-n-gram baselines and more complex earlier text CNNs. The authors present this as the first successful use of word order to improve topic classification. Their code is publicly released.

## Key points
- Input is a document treated as a |D|×1 "image" with |V| channels, one one-hot vector per word. No pre-trained or separately learned word vectors are used, and the setting is purely supervised.
- seq-CNN region vectors have p|V| dimensions, while bow-CNN regions have only |V| dimensions. With vocabularies of around 100K, bow-CNN keeps large regions affordable.
- The authors argue that a word-vector lookup layer is just a convolution with region size one. They question this choice when bi-grams discriminate better than unigrams.
- On Elec, an SVM with uni-, bi- and tri-gram features still relies mostly on unigrams. Its top 100 features contain only 28 bi-grams and 4 tri-grams. NB-weighting favors longer n-grams (60 tri-grams in its top 100) but does not always give the best accuracy.
- A seq-CNN with region size 3 and 1000 neurons learns components that respond to test-set phrases never seen in training, such as "am entirely satisfied" and "best X ever".

## Interesting ideas
- **Embed regions, not words.** Applying convolution straight to sparse one-hot input lets the model learn phrase-level features in one step. This could simplify pipelines for low-resource languages such as Ukrainian, where good pre-trained word vectors may be scarce.
- **Bag-of-words inside small windows.** Ignoring order only within a short region sits between full sequence models and plain bag-of-words. It is a cheap compromise for tasks like topic labeling, where exact local order matters less than which words appear together.
- **Exploit sparsity rather than shrinking the vocabulary.** Computing convolutions on sparse data efficiently avoids cutting the vocabulary or region size to save compute. This design lesson carries over to any model with huge categorical inputs.
- **Combining several region embeddings in parallel.** Layers with different region sizes or representations can run side by side and feed a single classifier. This is a simple ensemble-like trick for capturing patterns at multiple scales.
- **Interpretability check through unseen n-grams.** Listing test phrases that strongly activate predictive components but never occurred in training shows the model generalizes beyond memorized n-grams. This evaluation idea is useful whenever one needs to argue that a model captures compositional patterns.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

92 papers of our library cite it (of the 140884 whose bibliographies we read); 6 of them have a page here, 86 not yet.

- [Intra-Sentential Subject Zero Anaphora Resolution using Multi-Column Convolutional Neural Network](https://md.research.elamur.ai/free/en/papers/iida-2016-intra-sentential-subject-zero-anaphora/)
- [Adversarial Training Methods for Semi-Supervised Text Classification](https://md.research.elamur.ai/free/en/papers/miyato-2016-adversarial-training-methods-semi-supervised/)
- [Document Modeling with Gated Recurrent Neural Network for Sentiment Classification](https://md.research.elamur.ai/free/en/papers/tang-2015-document-modeling-gated-recurrent-neural/)
- [Hierarchical Attention Networks for Document Classification](https://md.research.elamur.ai/free/en/papers/yang-2016-hierarchical-attention-networks-document-classification/)
- [Character-level Convolutional Networks for Text Classification](https://md.research.elamur.ai/free/en/papers/zhang-2015-character-level-convolutional-networks-text/)
- [A Sensitivity Analysis of (and Practitioners' Guide to) Convolutional Neural Networks for Sentence Classification](https://md.research.elamur.ai/free/en/papers/zhang-2017-sensitivity-analysis-practitioners-guide-convolutional/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{johnson2015effective,
  title = {{Effective Use of Word Order for Text Categorization with Convolutional Neural Networks}},
  author = {Johnson, Rie and Zhang, Tong},
  booktitle = {Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2015},
  doi = {10.3115/v1/n15-1011},
  url = {https://doi.org/10.3115/v1/n15-1011}
}
```

- [← previous Does Universal Dependencies need a parsing representation? An investigation of English](https://md.research.elamur.ai/free/en/papers/silveira-2015-does-universal-dependencies-need-parsing/)
- [next → End-to-end learning of semantic role labeling using recurrent neural networks](https://md.research.elamur.ai/free/en/papers/zhou-2015-end-end-learning-semantic-role/)
- [↑ back to papers of 2015](https://md.research.elamur.ai/free/en/papers/2015/)

## References

28 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Christopher Bishop. 1995. Neural networks for pattern recognition. Oxford University Press. [find](https://scholar.google.com/scholar?q=Christopher+Bishop%2E+1995%2E+Neural+networks+for+pattern+recognition%2E+Oxford+University+Press%2E)
2. John Blitzer, Mark Dredze, and Fernando Pereira. 2007. Biographies, bollywood, boom-boxes, and blenders: Domain adaptation for sentiment classification. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=John+Blitzer%2C+Mark+Dredze%2C+and+Fernando+Pereira%2E+2007%2E+Biographies%2C+bollywood%2C+boom%2Dboxes%2C+and+blenders%3A+Domain+adaptation+for+sentiment+classification%2E+In+Proceedings+of+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/blitzer-2007-biographies-bollywood-boom-boxes-blenders/)
3. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12:2493– 2537. [find](https://scholar.google.com/scholar?q=Ronan+Collobert%2C+Jason+Weston%2C+Le%CC%81on+Bottou%2C+Michael+Karlen%2C+Koray+Kavukcuoglu%2C+and+Pavel+Kuksa%2E+2011%2E+Natural+language+processing+%28almost%29+from+scratch%2E+Journal+of+Machine+Learning+Research%2C+12%3A2493%E2%80%93) [our card](https://md.research.elamur.ai/free/en/papers/collobert-2011-natural-language-processing-almost-scratch/)
4. Jianfeng Gao, Patric Pantel, Michael Gamon, Xiaodong He, and Li dent. 2014. Modeling interestingness with deep neural networks. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Jianfeng+Gao%2C+Patric+Pantel%2C+Michael+Gamon%2C+Xiaodong+He%2C+and+Li+dent%2E+2014%2E+Modeling+interestingness+with+deep+neural+networks%2E+In+Proceedings+of+EMNLP%2E)
5. Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 2011. Domain adaptation for large-scale sentiment classification: A deep learning approach. In Proceedings of ICML. [find](https://scholar.google.com/scholar?q=Xavier+Glorot%2C+Antoine+Bordes%2C+and+Yoshua+Bengio%2E+2011%2E+Domain+adaptation+for+large%2Dscale+sentiment+classification%3A+A+deep+learning+approach%2E+In+Proceedings+of+ICML%2E)
6. Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R. Salakhutdinov. 2012. Improving neural networks by preventing co-adaptation of feature detectors. arXiv:1207.0580. [find](https://scholar.google.com/scholar?q=Geoffrey+E%2E+Hinton%2C+Nitish+Srivastava%2C+Alex+Krizhevsky%2C+Ilya+Sutskever%2C+and+Ruslan+R%2E+Salakhutdinov%2E+2012%2E+Improving+neural+networks+by+preventing+co%2Dadaptation+of+feature+detectors%2E+arXiv%3A1207%2E0580%2E)
7. Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014. A convolutional neural network for modeling sentences. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Nal+Kalchbrenner%2C+Edward+Grefenstette%2C+and+Phil+Blunsom%2E+2014%2E+A+convolutional+neural+network+for+modeling+sentences%2E+In+Proceedings+of+ACL%2E)
8. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of EMNLP, pages 1746–1751. [find](https://scholar.google.com/scholar?q=Yoon+Kim%2E+2014%2E+Convolutional+neural+networks+for+sentence+classification%2E+In+Proceedings+of+EMNLP%2C+pages+1746%E2%80%931751%2E) [our card](https://md.research.elamur.ai/free/en/papers/kim-2014-convolutional-neural-networks-sentence-classification/)
9. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet classification with deep convolutional neural networks. In Proceedings of NIPS. [find](https://scholar.google.com/scholar?q=Alex+Krizhevsky%2C+Ilya+Sutskever%2C+and+Geoffrey+E%2E+Hinton%2E+2012%2E+ImageNet+classification+with+deep+convolutional+neural+networks%2E+In+Proceedings+of+NIPS%2E)
10. Quoc Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In Proceedings of ICML. [find](https://scholar.google.com/scholar?q=Quoc+Le+and+Tomas+Mikolov%2E+2014%2E+Distributed+representations+of+sentences+and+documents%2E+In+Proceedings+of+ICML%2E) [our card](https://md.research.elamur.ai/free/en/papers/le-2014-distributed-representations-sentences-documents/)
11. Yann LeCun, León Bottou, Yoshua Bengio, and Patrick Haffner. 1986. Gradient-based learning applied to document recognition. In Proceedings of the IEEE, pages 2278–2324. [find](https://scholar.google.com/scholar?q=Yann+LeCun%2C+Leo%CC%81n+Bottou%2C+Yoshua+Bengio%2C+and+Patrick+Haffner%2E+1986%2E+Gradient%2Dbased+learning+applied+to+document+recognition%2E+In+Proceedings+of+the+IEEE%2C+pages+2278%E2%80%932324%2E)
12. David D. Lewis, Yiming Yang, Tony G. Rose, and Fan Li. 2004. RCV1: A new benchmark collection for text categorization research. Journal of Marchine Learning Research, 5:361–397. [find](https://scholar.google.com/scholar?q=David+D%2E+Lewis%2C+Yiming+Yang%2C+Tony+G%2E+Rose%2C+and+Fan+Li%2E+2004%2E+RCV1%3A+A+new+benchmark+collection+for+text+categorization+research%2E+Journal+of+Marchine+Learning+Research%2C+5%3A361%E2%80%93397%2E) [our card](https://md.research.elamur.ai/free/en/papers/lewis-2004-rcv1-benchmark-collection-text-categorization/)
13. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Andrew+L%2E+Maas%2C+Raymond+E%2E+Daly%2C+Peter+T%2E+Pham%2C+Dan+Huang%2C+Andrew+Y%2E+Ng%2C+and+Christopher+Potts%2E+2011%2E+Learning+word+vectors+for+sentiment+analysis%2E+In+Proceedings+of+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/maas-2011-learning-word-vectors-sentiment-analysis/)
14. Julian McAuley and Jure Leskovec. 2013. Hidden factors and hidden topics: Understanding rating dimensions with review text. In RecSys. [find](https://scholar.google.com/scholar?q=Julian+McAuley+and+Jure+Leskovec%2E+2013%2E+Hidden+factors+and+hidden+topics%3A+Understanding+rating+dimensions+with+review+text%2E+In+RecSys%2E)
15. Grégoire Mesnil, Tomas Mikolov, Marc’Aurelio Ranzato, and Yoshua Bengio. 2014. Ensemble of generative and discriminative techniques for sentiment analysis of movie reviews. arXiv:1412.5335v5 (4 Feb 2015 version). — [arXiv:1412.5335](https://arxiv.org/abs/1412.5335) [find](https://scholar.google.com/scholar?q=Gre%CC%81goire+Mesnil%2C+Tomas+Mikolov%2C+Marc%E2%80%99Aurelio+Ranzato%2C+and+Yoshua+Bengio%2E+2014%2E+Ensemble+of+generative+and+discriminative+techniques+for+sentiment+analysis+of+movie+reviews%2E+arXiv%3A1412%2E5335v5+%284+Feb+2)
16. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of NIPS. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Ilya+Sutskever%2C+Kai+Chen%2C+Greg+Corrado%2C+and+Jeffrey+Dean%2E+2013%2E+Distributed+representations+of+words+and+phrases+and+their+compositionality%2E+In+Proceedings+of+NIPS%2E)
17. Bo Pang and Lillian Lee. 2008. Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval, 2(1–2):1–135. [find](https://scholar.google.com/scholar?q=Bo+Pang+and+Lillian+Lee%2E+2008%2E+Opinion+mining+and+sentiment+analysis%2E+Foundations+and+Trends+in+Information+Retrieval%2C+2%281%E2%80%932%29%3A1%E2%80%93135%2E)
18. Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs up? sentiment classification using machine learning techniques. In Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 79–86. [find](https://scholar.google.com/scholar?q=Bo+Pang%2C+Lillian+Lee%2C+and+Shivakumar+Vaithyanathan%2E+2002%2E+Thumbs+up%3F+sentiment+classification+using+machine+learning+techniques%2E+In+Proceedings+of+Conference+on+Empirical+Methods+in+Natural+Language+P) [our card](https://md.research.elamur.ai/free/en/papers/pang-2002-thumbs-up-sentiment-classification-machine/)
19. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2014. ImageNet Large Scale Visual Recognition Challenge. arXiv:1409.0575. [find](https://scholar.google.com/scholar?q=Olga+Russakovsky%2C+Jia+Deng%2C+Hao+Su%2C+Jonathan+Krause%2C+Sanjeev+Satheesh%2C+Sean+Ma%2C+Zhiheng+Huang%2C+Andrej+Karpathy%2C+Aditya+Khosla%2C+Michael+Bernstein%2C+Alexander+C%2E+Berg%2C+and+Li+Fei%2DFei%2E+2014%2E+ImageNet+Larg) [our card](https://md.research.elamur.ai/free/en/papers/russakovsky-2014-imagenet-large-scale-visual-recognition/)
20. Mehran Sahami, Susan Dumais, David Heckerman, and Eric Horvitz. 1998. A bayesian approach to filtering junk e-mail. In Proceedings of AAAI’98 Workshop on Learning for Text Categorization. [find](https://scholar.google.com/scholar?q=Mehran+Sahami%2C+Susan+Dumais%2C+David+Heckerman%2C+and+Eric+Horvitz%2E+1998%2E+A+bayesian+approach+to+filtering+junk+e%2Dmail%2E+In+Proceedings+of+AAAI%E2%80%9998+Workshop+on+Learning+for+Text+Categorization%2E)
21. Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng, and Grégoire Mensnil. 2014. A latent semantic model with convolutional-pooling structure for information retrieval. In Proceedings of CIKM. [find](https://scholar.google.com/scholar?q=Yelong+Shen%2C+Xiaodong+He%2C+Jianfeng+Gao%2C+Li+Deng%2C+and+Gre%CC%81goire+Mensnil%2E+2014%2E+A+latent+semantic+model+with+convolutional%2Dpooling+structure+for+information+retrieval%2E+In+Proceedings+of+CIKM%2E)
22. Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2014. Going deeper with convolutions. arXiv:1409.4842. — [arXiv:1409.4842](https://arxiv.org/abs/1409.4842) [find](https://scholar.google.com/scholar?q=Christian+Szegedy%2C+Wei+Liu%2C+Yangqing+Jia%2C+Pierre+Sermanet%2C+Scott+Reed%2C+Dragomir+Anguelov%2C+Dumitru+Erhan%2C+Vincent+Vanhoucke%2C+and+Andrew+Rabinovich%2E+2014%2E+Going+deeper+with+convolutions%2E+arXiv%3A1409%2E4842)
23. Chade-Meng Tan, Yuan-Fang Wang, and Chan-Do Lee. 2002. The use of bigrams to enhance text categorization. Information Processing and Management, 38:529–546. [find](https://scholar.google.com/scholar?q=Chade%2DMeng+Tan%2C+Yuan%2DFang+Wang%2C+and+Chan%2DDo+Lee%2E+2002%2E+The+use+of+bigrams+to+enhance+text+categorization%2E+Information+Processing+and+Management%2C+38%3A529%E2%80%93546%2E)
24. Duyu Tang, Furu Wei, Nan Yang, Ming Zhou, Ting Liu, and Bing Qin. 2014. Learning sentiment-specific word embedding for twitter sentiment classification. In Proceedings of ACL, pages 1555–1565. [find](https://scholar.google.com/scholar?q=Duyu+Tang%2C+Furu+Wei%2C+Nan+Yang%2C+Ming+Zhou%2C+Ting+Liu%2C+and+Bing+Qin%2E+2014%2E+Learning+sentiment%2Dspecific+word+embedding+for+twitter+sentiment+classification%2E+In+Proceedings+of+ACL%2C+pages+1555%E2%80%931565%2E) [our card](https://md.research.elamur.ai/free/en/papers/tang-2014-learning-sentiment-specific-word-embedding/)
25. Sida Wang and Christopher D. Manning. 2012. Baselines and bigrams: Simple, good sentiment and topic classification. In Proceedings of ACL (short paper). [find](https://scholar.google.com/scholar?q=Sida+Wang+and+Christopher+D%2E+Manning%2E+2012%2E+Baselines+and+bigrams%3A+Simple%2C+good+sentiment+and+topic+classification%2E+In+Proceedings+of+ACL+%28short+paper%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2012-baselines-bigrams-simple-good-sentiment/)
26. Jason Weston, Sumit Chopra, and Keith Adams. 2014. \#tagspace: Semantic embeddings from hashtags. In Proceedings of EMNLP, pages 1822–1827. [find](https://scholar.google.com/scholar?q=Jason+Weston%2C+Sumit+Chopra%2C+and+Keith+Adams%2E+2014%2E+%5C%23tagspace%3A+Semantic+embeddings+from+hashtags%2E+In+Proceedings+of+EMNLP%2C+pages+1822%E2%80%931827%2E)
27. Puyang Xu and Ruhi Sarikaya. 2013. Convolutional neural network based triangular crf for joint intent detection and slot filling. In ASRU. [find](https://scholar.google.com/scholar?q=Puyang+Xu+and+Ruhi+Sarikaya%2E+2013%2E+Convolutional+neural+network+based+triangular+crf+for+joint+intent+detection+and+slot+filling%2E+In+ASRU%2E)
28. Liheng Xu, Kang Liu, Siwei Lai, and Jun Zhao. 2014. Product feature mining: Semantic clues versus syntactic constituents. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Liheng+Xu%2C+Kang+Liu%2C+Siwei+Lai%2C+and+Jun+Zhao%2E+2014%2E+Product+feature+mining%3A+Semantic+clues+versus+syntactic+constituents%2E+In+Proceedings+of+ACL%2E)