---
title: "A Neural Knowledge Language Model — paper by Ahn et al. (2016)"
description: "The authors propose the Neural Knowledge Language Model (NKLM), an LSTM-based RNN model… 35 references, cited by 45 papers here."
authors: ["Sungjin Ahn", "Heeyoul Choi", "Tanel Pärnamaa", "Yoshua Bengio"]
year: 2016
venue: "arXiv:1608.00318"
source: "https://arxiv.org/abs/1608.00318"
doi: "10.48550/arxiv.1608.00318"
arxiv: "1608.00318"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ahn-2016-neural-knowledge-language-model/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ahn-2016-neural-knowledge-language-model/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/) / A Neural Knowledge Language Model

# A Neural Knowledge Language Model

**Authors:** [Sungjin Ahn](https://md.research.elamur.ai/free/en/authors/sungjin-ahn/), Heeyoul Choi, Tanel Pärnamaa and [Yoshua Bengio](https://md.research.elamur.ai/free/en/authors/yoshua-bengio/) · **Year:** 2016 · **Venue:** arXiv:1608.00318 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1608.00318) · [PDF](https://arxiv.org/pdf/1608.00318) · [DOI 10.48550/arxiv.1608.00318](https://doi.org/10.48550/arxiv.1608.00318) · [arXiv 1608.00318](https://arxiv.org/abs/1608.00318) · [OpenAlex](https://openalex.org/W2476140796)

**Citations:** 101 citations (OpenAlex, 2026-10-14) · [45](https://md.research.elamur.ai/free/en/papers/ahn-2016-neural-knowledge-language-model/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Conventional neural language models are poor at conveying factual knowledge. Names of people, places, and dates occur rarely in a corpus. As a result, the model either lacks such words in its vocabulary or substitutes the \<unk> token for them. The authors propose the Neural Knowledge Language Model (NKLM), an LSTM-based RNN model combined with a knowledge graph. At each step, the model first determines whether the next word corresponds to some fact about the topic. It then either takes the word from the regular vocabulary or copies it from the name of that fact's object. This allows the model to produce words it did not see during training. To train it, the authors built the WikiFact dataset, in which Wikipedia descriptions are automatically aligned with facts from Freebase. They also proposed the Unknown-Penalized Perplexity metric, which prevents a model from getting a good score merely because it predicts \<unk> accurately. NKLM outperforms the baseline RNNLM on all three perplexity variants and generates far fewer unknown words. The work is an early example of combining symbolic knowledge with a neural language model through a copy mechanism.

## Key points
- **WikiFact**: Wikipedia texts aligned with Freebase facts by simple string matching. Words with no fact behind them (e.g., function words) get a separate class, Not-a-Fact (NaF).
- **Fact representation**: each fact is a (subject, relation, object) triple with 100-dimensional TransE embeddings. These vectors are kept fixed during training, so facts can be updated without retraining the model. When the place of birth Oklahoma was replaced with Chicago, the model immediately wrote the new city and left the rest of the text unchanged.
- **Setup**: both models are two-layer LSTMs with dropout and 1000 hidden units. The baseline RNNLM has a 10K-word vocabulary. NKLM has slightly fewer parameters than RNNLM.
- **New metrics UPP and UPP-fact**: the probability of \<unk> is divided by the number of out-of-vocabulary words. In UPP these are all such words in the corpus; in UPP-fact, only the knowledge words about the current topic.
- **The main source of the gain is word copying**: without it, NKLM is only slightly better than RNNLM. Limitation: the model does not use facts it has not recognized. For example, it writes "he" about a woman even though a gender fact is available.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Sungjin Ahn, Heeyoul Choi, Tanel Pärnamaa and Yoshua Bengio. 2016. A Neural Knowledge Language Model. arXiv:1608.00318.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{ahn2016neural,
  title = {{A Neural Knowledge Language Model}},
  author = {Sungjin Ahn, Heeyoul Choi, Tanel Pärnamaa and Yoshua Bengio},
  year = {2016},
  doi = {10.48550/arxiv.1608.00318},
  eprint = {1608.00318},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1608.00318}
}
```

- [← previous Two negations for the price of one](https://md.research.elamur.ai/free/en/papers/thornton-2016-two-negations-price-one/)
- [next → Simpler Context-Dependent Logical Forms via Model Projections](https://md.research.elamur.ai/free/en/papers/long-2016-simpler-context-dependent-logical-forms/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/)

## References

35 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bahdanau, Dzmitry, Cho, Kyunghyun, and Bengio, Yoshua. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). arXiv preprint arXiv:1409.0473, 2014.
2. Bastien, Frédéric, Lamblin, Pascal, Pascanu, Razvan, Bergstra, James, Goodfellow, Ian, Bergeron, Arnaud, Bouchard, Nicolas, Warde-Farley, David, and Bengio, Yoshua. [Theano: new features and speed improvements](https://md.research.elamur.ai/free/en/papers/bastien-2012-theano-new-features-speed-improvements/). arXiv preprint arXiv:1211.5590, 2012.
3. Bengio, Yoshua, Ducharme, Réjean, Vincent, Pascal, and Jauvin, Christian. [A neural probabilistic language model](https://md.research.elamur.ai/free/en/papers/bengio-2003-neural-probabilistic-language-model/). In Journal of Machine Learning Research, 2003.
4. Blei, David M, Ng, Andrew Y, and Jordan, Michael I. [Latent dirichlet allocation](https://md.research.elamur.ai/free/en/papers/blei-2003-latent-dirichlet-allocation/). the Journal of machine Learning research, 3:993–1022, 2003.
5. Bordes, Antoine, Usunier, Nicolas, Garcia-Duran, Alberto, Weston, Jason, and Yakhnenko, Oksana. [Translating embeddings for modeling multi-relational data](https://md.research.elamur.ai/free/en/papers/bordes-2013-translating-embeddings-modeling-multi-relational/). In Advances in Neural Information Processing Systems, pp. 2787–2795, 2013.
6. Bordes, Antoine, Usunier, Nicolas, Chopra, Sumit, and Weston, Jason. Large-scale simple question answering with memory networks. arXiv preprint arXiv:1506.02075, 2015. [find](https://scholar.google.com/scholar?q=Bordes%2C+Antoine%2C+Usunier%2C+Nicolas%2C+Chopra%2C+Sumit%2C+and+Weston%2C+Jason%2E+Large%2Dscale+simple+question+answering+with+memory+networks%2E+arXiv+preprint+arXiv%3A1506%2E02075%2C+2015%2E)
7. Celikyilmaz, Asli, Hakkani-Tur, Dilek, Pasupat, Panupong, and Sarikaya, Ruhi. Enriching word embeddings using knowledge graph for semantic tagging in conversational dialog systems. In 2015 AAAI Spring Symposium Series, 2015. [find](https://scholar.google.com/scholar?q=Celikyilmaz%2C+Asli%2C+Hakkani%2DTur%2C+Dilek%2C+Pasupat%2C+Panupong%2C+and+Sarikaya%2C+Ruhi%2E+Enriching+word+embeddings+using+knowledge+graph+for+semantic+tagging+in+conversational+dialog+systems%2E+In+2015+AAAI+Spring)
8. Chelba, Ciprian, Mikolov, Tomas, Schuster, Mike, Ge, Qi, Brants, Thorsten, Koehn, Phillipp, and Robinson, Tony. [One billion word benchmark for measuring progress in statistical language modeling](https://md.research.elamur.ai/free/en/papers/chelba-2013-one-billion-word-benchmark-measuring/). arXiv preprint arXiv:1312.3005, 2013.
9. Gildea, Daniel and Hofmann, Thomas. Topic-based language models using em. EuroSpeech 1999, 1999. [find](https://scholar.google.com/scholar?q=Gildea%2C+Daniel+and+Hofmann%2C+Thomas%2E+Topic%2Dbased+language+models+using+em%2E+EuroSpeech+1999%2C+1999%2E)
10. Graves, Alex, Wayne, Greg, and Danihelka, Ivo. Neural turing machines. arXiv preprint arXiv:1410.5401, 2014. [find](https://scholar.google.com/scholar?q=Graves%2C+Alex%2C+Wayne%2C+Greg%2C+and+Danihelka%2C+Ivo%2E+Neural+turing+machines%2E+arXiv+preprint+arXiv%3A1410%2E5401%2C+2014%2E)
11. Gu, Jiatao, Lu, Zhengdong, Li, Hang, and Li, Victor O. K. [Incorporating copying mechanism in sequence-to-sequence learning](https://md.research.elamur.ai/free/en/papers/gu-2016-incorporating-copying-mechanism-sequence-sequence/). CoRR, abs/1603.06393, 2016.
12. Gulcehre, Caglar, Ahn, Sungjin, Nallapati, Ramesh, Zhou, Bowen, and Bengio, Yoshua. [Pointing the unknown words](https://md.research.elamur.ai/free/en/papers/gulcehre-2016-pointing-unknown-words/). ACL 2016, 2016.
13. Hochreiter, Sepp and Schmidhuber, Jürgen. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. [find](https://scholar.google.com/scholar?q=Hochreiter%2C+Sepp+and+Schmidhuber%2C+Ju%CC%88rgen%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%3A1735%E2%80%931780%2C+1997%2E)
14. Iyyer, Mohit, Boyd-Graber, Jordan L, Claudino, Leonardo Max Batista, Socher, Richard, and Daumé III, Hal. A neural network for factoid question answering over paragraphs. In EMNLP 2014, pp. 633–644, 2014. [find](https://scholar.google.com/scholar?q=Iyyer%2C+Mohit%2C+Boyd%2DGraber%2C+Jordan+L%2C+Claudino%2C+Leonardo+Max+Batista%2C+Socher%2C+Richard%2C+and+Daume%CC%81+III%2C+Hal%2E+A+neural+network+for+factoid+question+answering+over+paragraphs%2E+In+EMNLP+2014%2C+pp%2E+633%E2%80%93644%2C+)
15. Jean, Sebastien, Cho, Kyunghyun, Memisevic, Roland, and Bengio, Yoshua. [On using very large target vocabulary for neural machine translation](https://md.research.elamur.ai/free/en/papers/jean-2015-very-large-target-vocabulary-neural/). ACL 2015, 2015.
16. Jozefowicz, Rafal, Vinyals, Oriol, Schuster, Mike, Shazeer, Noam, and Wu, Yonghui. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016. [find](https://scholar.google.com/scholar?q=Jozefowicz%2C+Rafal%2C+Vinyals%2C+Oriol%2C+Schuster%2C+Mike%2C+Shazeer%2C+Noam%2C+and+Wu%2C+Yonghui%2E+Exploring+the+limits+of+language+modeling%2E+arXiv+preprint+arXiv%3A1602%2E02410%2C+2016%2E)
17. Kim, Yoon. [Convolutional neural networks for sentence classification](https://md.research.elamur.ai/free/en/papers/kim-2014-convolutional-neural-networks-sentence-classification/). EMNLP 2014, 2014.
18. Lebret, Rémi, Grangier, David, and Auli, Michael. [Neural text generation from structured data with application to the biography domain](https://md.research.elamur.ai/free/en/papers/lebret-2016-neural-text-generation-structured-data/). arXiv preprint arXiv:1603.07771, 2016.
19. Long, Teng, Lowe, Ryan, Cheung, Jackie Chi Kit, and Precup, Doina. Leveraging lexical resources for learning entity embeddings in multi-relational data. 2016. [find](https://scholar.google.com/scholar?q=Long%2C+Teng%2C+Lowe%2C+Ryan%2C+Cheung%2C+Jackie+Chi+Kit%2C+and+Precup%2C+Doina%2E+Leveraging+lexical+resources+for+learning+entity+embeddings+in+multi%2Drelational+data%2E+2016%2E)
20. Marcus, Mitchell P, Marcinkiewicz, Mary Ann, and Santorini, Beatrice. [Building a large annotated corpus of english: The penn treebank](https://md.research.elamur.ai/free/en/papers/marcus-1993-building-large-annotated-corpus-english/). Computational linguistics, 19(2):313–330, 1993.
21. Merity, Stephen, Xiong, Caiming, Bradbury, James, and Socher, Richard. [Pointer sentinel mixture models](https://md.research.elamur.ai/free/en/papers/merity-2016-pointer-sentinel-mixture-models/). arXiv preprint arXiv:1609.07843, 2016.
22. Mikolov, Tomas and Zweig, Geoffrey. Context dependent recurrent neural network language model. In Spoken Language Technology Workshop (SLT), 2012 IEEE, pp. 234–239. [find](https://scholar.google.com/scholar?q=Mikolov%2C+Tomas+and+Zweig%2C+Geoffrey%2E+Context+dependent+recurrent+neural+network+language+model%2E+In+Spoken+Language+Technology+Workshop+%28SLT%29%2C+2012+IEEE%2C+pp%2E+234%E2%80%93239%2E)
23. Mikolov, Tomas, Karafiát, Martin, Burget, Lukas, Cernockỳ, Jan, and Khudanpur, Sanjeev. Recurrent neural network based language model. In INTERSPEECH 2010, volume 2, pp. 3, 2010. [find](https://scholar.google.com/scholar?q=Mikolov%2C+Tomas%2C+Karafia%CC%81t%2C+Martin%2C+Burget%2C+Lukas%2C+Cernocky%CC%80%2C+Jan%2C+and+Khudanpur%2C+Sanjeev%2E+Recurrent+neural+network+based+language+model%2E+In+INTERSPEECH+2010%2C+volume+2%2C+pp%2E+3%2C+2010%2E)
24. Miller, George A. [Wordnet: a lexical database for english](https://md.research.elamur.ai/free/en/papers/miller-1994-wordnet/). Communications of the ACM, 38(11):39–41, 1995.
25. Mnih, Andriy and Hinton, Geoffrey E. A scalable hierarchical distributed language model. In Advances in neural information processing systems, pp. 1081–1088, 2009. [find](https://scholar.google.com/scholar?q=Mnih%2C+Andriy+and+Hinton%2C+Geoffrey+E%2E+A+scalable+hierarchical+distributed+language+model%2E+In+Advances+in+neural+information+processing+systems%2C+pp%2E+1081%E2%80%931088%2C+2009%2E)
26. Mnih, Andriy and Teh, Yee Whye. [A fast and simple algorithm for training neural probabilistic language models](https://md.research.elamur.ai/free/en/papers/mnih-2012-fast-simple-algorithm-training-neural/). ICML 2012, 2012.
27. Morin, Frederic and Bengio, Yoshua. Hierarchical probabilistic neural network language model. AISTATS 2005, pp. 246, 2005. [find](https://scholar.google.com/scholar?q=Morin%2C+Frederic+and+Bengio%2C+Yoshua%2E+Hierarchical+probabilistic+neural+network+language+model%2E+AISTATS+2005%2C+pp%2E+246%2C+2005%2E)
28. Nickel, Maximilian, Murphy, Kevin, Tresp, Volker, and Gabrilovich, Evgeniy. A review of relational machine learning for knowledge graphs: From multi-relational link prediction to automated knowledge graph construction. arXiv preprint arXiv:1503.00759, 2015. [find](https://scholar.google.com/scholar?q=Nickel%2C+Maximilian%2C+Murphy%2C+Kevin%2C+Tresp%2C+Volker%2C+and+Gabrilovich%2C+Evgeniy%2E+A+review+of+relational+machine+learning+for+knowledge+graphs%3A+From+multi%2Drelational+link+prediction+to+automated+knowledge+g)
29. Reed, Scott and de Freitas, Nando. Neural programmer-interpreters. ICLR 2016, 2016. [find](https://scholar.google.com/scholar?q=Reed%2C+Scott+and+de+Freitas%2C+Nando%2E+Neural+programmer%2Dinterpreters%2E+ICLR+2016%2C+2016%2E)
30. Serban, Iulian V, Sordoni, Alessandro, Bengio, Yoshua, Courville, Aaron, and Pineau, Joelle. Building end-to-end dialogue systems using generative hierarchical neural networks. 30th AAAI Conference on Artificial Intelligence, 2015. [find](https://scholar.google.com/scholar?q=Serban%2C+Iulian+V%2C+Sordoni%2C+Alessandro%2C+Bengio%2C+Yoshua%2C+Courville%2C+Aaron%2C+and+Pineau%2C+Joelle%2E+Building+end%2Dto%2Dend+dialogue+systems+using+generative+hierarchical+neural+networks%2E+30th+AAAI+Conference+on)
31. Vinyals, Oriol and Le, Quoc. [A neural conversational model](https://md.research.elamur.ai/free/en/papers/vinyals-2015-neural-conversational-model/). arXiv preprint arXiv:1506.05869, 2015.
32. Vinyals, Oriol, Fortunato, Meire, and Jaitly, Navdeep. Pointer networks. NIPS 2015, 2015. [find](https://scholar.google.com/scholar?q=Vinyals%2C+Oriol%2C+Fortunato%2C+Meire%2C+and+Jaitly%2C+Navdeep%2E+Pointer+networks%2E+NIPS+2015%2C+2015%2E)
33. Weston, Jason, Chopra, Sumit, and Bordes, Antoine. Memory networks. ICLR 2015, 2015. [find](https://scholar.google.com/scholar?q=Weston%2C+Jason%2C+Chopra%2C+Sumit%2C+and+Bordes%2C+Antoine%2E+Memory+networks%2E+ICLR+2015%2C+2015%2E)
34. Weston, Jason, Bordes, Antoine, Chopra, Sumit, and Mikolov, Tomas. [Towards ai-complete question answering: A set of prerequisite toy tasks](https://md.research.elamur.ai/free/en/papers/weston-2015-towards-ai-complete-question-answering/). ICLR 2016, 2016.
35. Zaremba, Wojciech, Sutskever, Ilya, and Vinyals, Oriol. [Recurrent neural network regularization](https://md.research.elamur.ai/free/en/papers/zaremba-2014-recurrent-neural-network-regularization/). arXiv preprint arXiv:1409.2329, 2014.

*Updated 2 November 2026*