---
title: "Keep CALM and Explore — paper by Yao et al. (2020)"
description: "The authors propose CALM (Contextual Action Language Model), which is trained on human play… 28 references, cited by 54 papers here."
authors: ["Shunyu Yao", "Rohan Rao", "Matthew Hausknecht", "Karthik Narasimhan"]
year: 2020
venue: "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)"
source: "https://aclanthology.org/2020.emnlp-main.704/"
doi: "10.18653/v1/2020.emnlp-main.704"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yao-2020-keep-calm-explore-language-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yao-2020-keep-calm-explore-language-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/5/) / Keep CALM and Explore: Language Models for Action Generation in Text-based Games

# Keep CALM and Explore: Language Models for Action Generation in Text-based Games

**Authors:** [Shunyu Yao](https://md.research.elamur.ai/free/en/authors/shunyu-yao/), Rohan Rao, [Matthew Hausknecht](https://md.research.elamur.ai/free/en/authors/matthew-hausknecht/) and [Karthik Narasimhan](https://md.research.elamur.ai/free/en/authors/karthik-narasimhan/) · **Year:** 2020 · **Venue:** [Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2020](https://md.research.elamur.ai/free/en/venues/emnlp/2020/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.emnlp-main.704/) · [PDF](https://aclanthology.org/2020.emnlp-main.704.pdf) · [DOI 10.18653/v1/2020.emnlp-main.704](https://doi.org/10.18653/v1/2020.emnlp-main.704) · [OpenAlex](https://openalex.org/W3099204253)

**Citations:** 73 citations (OpenAlex, 2026-10-14) · [54](https://md.research.elamur.ai/free/en/papers/yao-2020-keep-calm-explore-language-models/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Text-based games are hard for reinforcement learning agents because the space of possible text commands is combinatorially huge, and only a few commands in each state are valid or useful. Earlier agents often depended on Jericho's handicap that lists admissible actions. The authors propose CALM (Contextual Action Language Model), which is trained on human play transcripts to suggest a small set of plausible actions for the current situation. The model sees the previous observation, the previous action and the current observation. A DRRN agent then scores these candidates with Q-values learned from game rewards. They compare n-gram and GPT-2 versions. None of the evaluation games appear in CALM's training data. On 28 Jericho games, CALM (GPT-2) with DRRN raises the average normalized score by 69% relative to the previous best handicap-free method. On several games it matches or beats agents that use the admissible-action handicap. The work shows that general linguistic and gameplay knowledge from humans can transfer to unseen games and make exploration tractable.

## Key points
- New dataset: 426 human gameplay transcripts covering 590 games, collected from ClubFloyd. The dataset and code are released at github.com/princeton-nlp/calm-textgame.
- The action space is very large. ZORK1 alone has a 697-word vocabulary, which gives roughly 200 billion possible 4-word commands.
- On 8 games CALM outperforms handicap-assisted agents. For example, on INHUMANE it scores 25.7, while KG-A2C scores 3.
- Ablations show that both GPT-2 pre-training and ClubFloyd fine-tuning matter. A candidate set size of k = 10 performs clearly worse than k = 20–40. Replacing DRRN with a random agent gives only 1.8%.
- During exploration the agent reached higher scores than it finally kept. If RL fully exploited the best trajectories it saw, the average normalized score would reach 14.7%, which exceeds all compared models, including those with handicaps.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/) — Company, Redmond, United States · 240 works · 123,537 citations
- [Princeton University](https://md.research.elamur.ai/free/en/universities/princeton-university/) — University, Princeton, United States · 61 works · 36,471 citations

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,344 works

## Cite

Cite as: Shunyu Yao, Rohan Rao, Matthew Hausknecht and Karthik Narasimhan. 2020. Keep CALM and Explore: Language Models for Action Generation in Text-based Games. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yao2020keep,
  title = {{Keep CALM and Explore: Language Models for Action Generation in Text-based Games}},
  author = {Shunyu Yao, Rohan Rao, Matthew Hausknecht and Karthik Narasimhan},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.704},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.704}
}
```

- [← previous ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora](https://md.research.elamur.ai/free/en/papers/ouyang-2020-ernie-m-enhanced-multilingual/)
- [next → Open-Domain Question Answering](https://md.research.elamur.ai/free/en/papers/chen-2020-open-domain-question-answering/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/5/)

## References

28 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Prithviraj Ammanabrolu, William Broniec, Alex Mueller, Jeremy Paul, and Mark O Riedl. 2019. Toward automated quest generation in text-adventure games. arXiv preprint arXiv:1909.06283. [find](https://scholar.google.com/scholar?q=Prithviraj+Ammanabrolu%2C+William+Broniec%2C+Alex+Mueller%2C+Jeremy+Paul%2C+and+Mark+O+Riedl%2E+2019%2E+Toward+automated+quest+generation+in+text%2Dadventure+games%2E+arXiv+preprint+arXiv%3A1909%2E06283%2E)
2. Prithviraj Ammanabrolu and Matthew Hausknecht. 2020. Graph constrained reinforcement learning for natural language action spaces. arXiv preprint arXiv:2001.08837. — [arXiv:2001.08837](https://arxiv.org/abs/2001.08837) [find](https://scholar.google.com/scholar?q=Prithviraj+Ammanabrolu+and+Matthew+Hausknecht%2E+2020%2E+Graph+constrained+reinforcement+learning+for+natural+language+action+spaces%2E+arXiv+preprint+arXiv%3A2001%2E08837%2E+%E2%80%94+%5BarXiv%3A2001%2E08837%5D%28https%3A%2F%2Farxiv%2Eor)
3. Prithviraj Ammanabrolu and Mark Riedl. 2019. Playing text-adventure games with graph-based deep reinforcement learning. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers),… [find](https://scholar.google.com/scholar?q=Prithviraj+Ammanabrolu+and+Mark+Riedl%2E+2019%2E+Playing+text%2Dadventure+games+with+graph%2Dbased+deep+reinforcement+learning%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chapter+of+the+Assoc)
4. Hongshen Chen, Xiaorui Liu, Dawei Yin, and Jiliang Tang. 2017. A survey on dialogue systems: Recent advances and new frontiers. Acm Sigkdd Explorations Newsletter, 19(2):25–35. [find](https://scholar.google.com/scholar?q=Hongshen+Chen%2C+Xiaorui+Liu%2C+Dawei+Yin%2C+and+Jiliang+Tang%2E+2017%2E+A+survey+on+dialogue+systems%3A+Recent+advances+and+new+frontiers%2E+Acm+Sigkdd+Explorations+Newsletter%2C+19%282%29%3A25%E2%80%9335%2E)
5. Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. [On the properties of neural machine translation: Encoder–decoder approaches](https://md.research.elamur.ai/free/en/papers/cho-2014-properties-neural-machine-translation-encoder/). In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pages 103–111, Doha, Qatar. Associati…
6. Marc-Alexandre Côté, Ákos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, et al. 2018. [Textworld: A learning environment for text-based games](https://md.research.elamur.ai/free/en/papers/cote-2018-textworld-learning-environment-text-based/). In Workshop on Computer Games, pages 41–75. Springer.
7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805.
8. Nancy Fulda, Daniel Ricks, Ben Murdoch, and David Wingate. 2017. What can you do with a rock? affordance extraction via word embeddings. CoRR, abs/1703.03429. [find](https://scholar.google.com/scholar?q=Nancy+Fulda%2C+Daniel+Ricks%2C+Ben+Murdoch%2C+and+David+Wingate%2E+2017%2E+What+can+you+do+with+a+rock%3F+affordance+extraction+via+word+embeddings%2E+CoRR%2C+abs%2F1703%2E03429%2E)
9. Matthew Hausknecht, Prithviraj Ammanabrolu, Marc-Alexandre Côté, and Xingdi Yuan. 2019a. Interactive fiction games: A colossal adventure. CoRR, abs/1909.05398. — [arXiv:1909.05398](https://arxiv.org/abs/1909.05398) [find](https://scholar.google.com/scholar?q=Matthew+Hausknecht%2C+Prithviraj+Ammanabrolu%2C+Marc%2DAlexandre+Co%CC%82te%CC%81%2C+and+Xingdi+Yuan%2E+2019a%2E+Interactive+fiction+games%3A+A+colossal+adventure%2E+CoRR%2C+abs%2F1909%2E05398%2E+%E2%80%94+%5BarXiv%3A1909%2E05398%5D%28https%3A%2F%2Farxiv%2Eorg)
10. Matthew Hausknecht, Ricky Loynd, Greg Yang, Adith Swaminathan, and Jason D Williams. 2019b. Nail: A general interactive fiction agent. arXiv preprint arXiv:1902.04259. — [arXiv:1902.04259](https://arxiv.org/abs/1902.04259) [find](https://scholar.google.com/scholar?q=Matthew+Hausknecht%2C+Ricky+Loynd%2C+Greg+Yang%2C+Adith+Swaminathan%2C+and+Jason+D+Williams%2E+2019b%2E+Nail%3A+A+general+interactive+fiction+agent%2E+arXiv+preprint+arXiv%3A1902%2E04259%2E+%E2%80%94+%5BarXiv%3A1902%2E04259%5D%28https%3A%2F%2Farx)
11. Ji He, Jianshu Chen, Xiaodong He, Jianfeng Gao, Lihong Li, Li Deng, and Mari Ostendorf. 2015. Deep reinforcement learning with a natural language action space. arXiv preprint arXiv:1511.04636. [find](https://scholar.google.com/scholar?q=Ji+He%2C+Jianshu+Chen%2C+Xiaodong+He%2C+Jianfeng+Gao%2C+Lihong+Li%2C+Li+Deng%2C+and+Mari+Ostendorf%2E+2015%2E+Deep+reinforcement+learning+with+a+natural+language+action+space%2E+arXiv+preprint+arXiv%3A1511%2E04636%2E)
12. Vishal Jain, William Fedus, Hugo Larochelle, Doina Precup, and Marc G. Bellemare. 2019. Algorithmic improvements for deep reinforcement learning applied to interactive fiction. CoRR, abs/1903.03094. [find](https://scholar.google.com/scholar?q=Vishal+Jain%2C+William+Fedus%2C+Hugo+Larochelle%2C+Doina+Precup%2C+and+Marc+G%2E+Bellemare%2E+2019%2E+Algorithmic+improvements+for+deep+reinforcement+learning+applied+to+interactive+fiction%2E+CoRR%2C+abs%2F1903%2E03094%2E)
13. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427–431. Association for Computation… [find](https://scholar.google.com/scholar?q=Armand+Joulin%2C+Edouard+Grave%2C+Piotr+Bojanowski%2C+and+Tomas+Mikolov%2E+2017%2E+Bag+of+tricks+for+efficient+text+classification%2E+In+Proceedings+of+the+15th+Conference+of+the+European+Chapter+of+the+Associati)
14. Daniel Jurafsky and James H. Martin. 2009. Speech and Language Processing (2nd Edition). Prentice-Hall, Inc., USA. [find](https://scholar.google.com/scholar?q=Daniel+Jurafsky+and+James+H%2E+Martin%2E+2009%2E+Speech+and+Language+Processing+%282nd+Edition%29%2E+Prentice%2DHall%2C+Inc%2E%2C+USA%2E)
15. Bartosz Kostka, Jaroslaw Kwiecien, Jakub Kowalski, and Pawel Rychlikowski. 2017. Text-based adventures of the golovin AI agent. CoRR, abs/1705.05637. — [arXiv:1705.05637](https://arxiv.org/abs/1705.05637) [find](https://scholar.google.com/scholar?q=Bartosz+Kostka%2C+Jaroslaw+Kwiecien%2C+Jakub+Kowalski%2C+and+Pawel+Rychlikowski%2E+2017%2E+Text%2Dbased+adventures+of+the+golovin+AI+agent%2E+CoRR%2C+abs%2F1705%2E05637%2E+%E2%80%94+%5BarXiv%3A1705%2E05637%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1705%2E05)
16. Angeliki Lazaridou, Anna Potapenko, and Olivier Tieleman. 2020. Multi-agent communication meets natural language: Synergies between functional and structural language learning. arXiv preprint arXiv:2005.07064. [find](https://scholar.google.com/scholar?q=Angeliki+Lazaridou%2C+Anna+Potapenko%2C+and+Olivier+Tieleman%2E+2020%2E+Multi%2Dagent+communication+meets+natural+language%3A+Synergies+between+functional+and+structural+language+learning%2E+arXiv+preprint+arXiv%3A20)
17. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. [Distributed representations of words and phrases and their compositionality](https://md.research.elamur.ai/free/en/papers/mikolov-2013-distributed-representations-words-phrases-their/). In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 26, pa…
18. Karthik Narasimhan, Tejas D. Kulkarni, and Regina Barzilay. 2015. Language understanding for text-based games using deep reinforcement learning. In EMNLP, pages 1–11. [find](https://scholar.google.com/scholar?q=Karthik+Narasimhan%2C+Tejas+D%2E+Kulkarni%2C+and+Regina+Barzilay%2E+2015%2E+Language+understanding+for+text%2Dbased+games+using+deep+reinforcement+learning%2E+In+EMNLP%2C+pages+1%E2%80%9311%2E)
19. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+Blog%2C+1%288%29%3A9%2E)
20. Yiping Song, Rui Yan, Xiang Li, Dongyan Zhao, and Ming Zhang. 2016. Two are better than one: An ensemble of retrieval-and generation-based dialog systems. arXiv preprint arXiv:1610.07149. — [arXiv:1610.07149](https://arxiv.org/abs/1610.07149) [find](https://scholar.google.com/scholar?q=Yiping+Song%2C+Rui+Yan%2C+Xiang+Li%2C+Dongyan+Zhao%2C+and+Ming+Zhang%2E+2016%2E+Two+are+better+than+one%3A+An+ensemble+of+retrieval%2Dand+generation%2Dbased+dialog+systems%2E+arXiv+preprint+arXiv%3A1610%2E07149%2E+%E2%80%94+%5BarXiv%3A161)
21. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+V+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+Advances+in+neural+information+processing+systems%2C+pages+3104%E2%80%933112%2E)
22. Ruo Yu Tao, Marc-Alexandre Côté, Xingdi Yuan, and Layla El Asri. 2018. Towards solving text-based games by producing adaptive action spaces. arXiv preprint arXiv:1812.00855. — [arXiv:1812.00855](https://arxiv.org/abs/1812.00855) [find](https://scholar.google.com/scholar?q=Ruo+Yu+Tao%2C+Marc%2DAlexandre+Co%CC%82te%CC%81%2C+Xingdi+Yuan%2C+and+Layla+El+Asri%2E+2018%2E+Towards+solving+text%2Dbased+games+by+producing+adaptive+action+spaces%2E+arXiv+preprint+arXiv%3A1812%2E00855%2E+%E2%80%94+%5BarXiv%3A1812%2E00855%5D%28htt)
23. Jack Urbanek, Angela Fan, Siddharth Karamcheti, Saachi Jain, Samuel Humeau, Emily Dinan, Tim Rocktäschel, Douwe Kiela, Arthur Szlam, and Jason Weston. 2019. Learning to speak and act in a fantasy text adventure game. CoRR, abs/1903.03094. [find](https://scholar.google.com/scholar?q=Jack+Urbanek%2C+Angela+Fan%2C+Siddharth+Karamcheti%2C+Saachi+Jain%2C+Samuel+Humeau%2C+Emily+Dinan%2C+Tim+Rockta%CC%88schel%2C+Douwe+Kiela%2C+Arthur+Szlam%2C+and+Jason+Weston%2E+2019%2E+Learning+to+speak+and+act+in+a+fantasy+tex)
24. Nick Walton. 2019. Ai dungeon 2: Creating infinitely generated text adventures with deep learning language models. [find](https://scholar.google.com/scholar?q=Nick+Walton%2E+2019%2E+Ai+dungeon+2%3A+Creating+infinitely+generated+text+adventures+with+deep+learning+language+models%2E)
25. Jason D. Williams, Kavosh Asadi, and Geoffrey Zweig. 2017. Hybrid code networks: practical and efficient end-to-end dialog control with supervised and reinforcement learning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 665… [find](https://scholar.google.com/scholar?q=Jason+D%2E+Williams%2C+Kavosh+Asadi%2C+and+Geoffrey+Zweig%2E+2017%2E+Hybrid+code+networks%3A+practical+and+efficient+end%2Dto%2Dend+dialog+control+with+supervised+and+reinforcement+learning%2E+In+Proceedings+of+the+55t)
26. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s [transformers: State-of-the-art natural language processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/). ArXiv, abs/1910.03771.
27. Tom Zahavy, Matan Haroush, Nadav Merlis, Daniel J Mankowitz, and Shie Mannor. 2018. Learn what not to learn: Action elimination with deep reinforcement learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing S… [find](https://scholar.google.com/scholar?q=Tom+Zahavy%2C+Matan+Haroush%2C+Nadav+Merlis%2C+Daniel+J+Mankowitz%2C+and+Shie+Mannor%2E+2018%2E+Learn+what+not+to+learn%3A+Action+elimination+with+deep+reinforcement+learning%2E+In+S%2E+Bengio%2C+H%2E+Wallach%2C+H%2E+Larochell)
28. Tiancheng Zhao and Maxine Eskenazi. 2016. Towards end-to-end learning for dialog state tracking and management using deep reinforcement learning. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 1–10, Los Angeles. Association for Computational… [find](https://scholar.google.com/scholar?q=Tiancheng+Zhao+and+Maxine+Eskenazi%2E+2016%2E+Towards+end%2Dto%2Dend+learning+for+dialog+state+tracking+and+management+using+deep+reinforcement+learning%2E+In+Proceedings+of+the+17th+Annual+Meeting+of+the+Speci)

*Updated 24 October 2026*