---
title: "Decoding, Fast and Slow — paper by Zarrieß et al. (2021)"
description: "It builds on the character-level Rational Speech Acts (RSA) captioner of Cohn-Gordon… 54.1 accuracy. 28 references, cited by 3 papers here."
authors: ["Sina Zarrieß", "Hendrik Buschmeier", "Ting Han", "Simeon Schüz"]
year: 2021
venue: "Proceedings of the 14th International Conference on Natural Language Generation"
source: "https://aclanthology.org/2021.inlg-1.41/"
doi: "10.18653/v1/2021.inlg-1.41"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zarriess-2021-decoding-fast-slow-case-study/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zarriess-2021-decoding-fast-slow-case-study/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/7/) / Decoding, Fast and Slow: A Case Study on Balancing Trade-Offs in Incremental, Character-level Pragmatic Reasoning

# Decoding, Fast and Slow: A Case Study on Balancing Trade-Offs in Incremental, Character-level Pragmatic Reasoning

**Authors:** [Sina Zarrieß](https://md.research.elamur.ai/free/en/authors/sina-zarriess/), [Hendrik Buschmeier](https://md.research.elamur.ai/free/en/authors/hendrik-buschmeier/), Ting Han and [Simeon Schüz](https://md.research.elamur.ai/free/en/authors/simeon-schuz/) · **Year:** 2021 · **Venue:** [Proceedings of the 14th International Conference on Natural Language Generation](https://md.research.elamur.ai/free/en/venues/inlg/) ([INLG 2021](https://md.research.elamur.ai/free/en/venues/inlg/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.inlg-1.41/) · [PDF](https://aclanthology.org/2021.inlg-1.41.pdf) · [DOI 10.18653/v1/2021.inlg-1.41](https://doi.org/10.18653/v1/2021.inlg-1.41) · [OpenAlex](https://openalex.org/W3202614407)

**Citations:** 3 citations (OpenAlex, 2026-10-14) · 3 in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper looks at the cost of pragmatic decoding in neural image captioning. It builds on the character-level Rational Speech Acts (RSA) captioner of Cohn-Gordon et al. (2018). That speaker runs recursive speaker–listener reasoning at every character so its caption singles out a target image among distractors, which is informative but expensive. The authors propose a cheap alternative, a mixed "fast and slow" speaker S x. It runs the rational S 1 step only on the first character of each word and lets the plain language model (S 0) fill in the rest.

They train a character-level version of Lu et al. (2017)'s adaptive attention model on MSCOCO. They then compare S 0, S 1 and S x under greedy and beam search, scoring:
- informativeness, via the accuracy of a separately trained listener;
- quality, via CIDEr and out-of-vocabulary counts;
- diversity, via several type-token ratios.

Fully rational decoding raises informativeness but lowers quality, invents non-words and repeats words more. S x keeps most of the informativeness gain while losing much less quality. This shows that applying reasoning only at selected steps can balance the competing goals.

## Key points
- **Data:**
  - MSCOCO, with 82,783 training and 40,504 validation images.
  - The training set was randomly halved so the speaker and the listener learn from disjoint data.
  - 1,000 validation images were used for evaluation.
- **Models:**
  - The captioner is Lu et al. (2017)'s adaptive attention architecture with ResNet152 image features, retrained to emit characters.
  - RSA follows Cohn-Gordon et al. (2018), with rationality parameter α ∈ {1, 3, 5}.
- **Evaluation contexts:** each target gets two distractors, chosen as the images whose reference captions have the highest Jaccard similarity to the target's captions.
- **Main result (beam search, α=5):**
  - Versus S 1, S x gains more than 12 CIDEr points (0.606 vs 0.481).
  - S x produces 8 OOV types against 72.
  - Its listener accuracy is only about 3 points lower (66.9 vs 70.4).
  - The literal S 0 scores 54.1 accuracy and 0.778 CIDEr.
- **Diversity:** pragmatic decoding makes individual captions more repetitive (lower TTR cap, TTR cap c and TTR cap 2), yet it raises global vocabulary size and TTR g. Beam search interacts badly with a highly rational S 1 and produces a sharp rise in OOV words.

## Interesting ideas
- Expensive reasoning does not need to run at every step; triggering it only at real choice points (word starts) captures most of the benefit, while cheap defaults handle predictable continuations.
- The informativeness judge (listener) is trained on a different half of the data than the speaker, so the evaluator cannot share the generator's learned biases.
- Hard test contexts are built automatically by picking distractors whose descriptions overlap most (Jaccard) with the target's, so success requires truly discriminative output.
- Diversity is split into a global type-token ratio and per-output ratios (all words, content words only, bigrams), which separates "rich vocabulary" from "repeats itself".
- A single rationality knob (α) is swept and the results are reported as trade-off curves across informativeness, quality and invented words, rather than as one best number.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (back): The generator could call its costly pragmatic or referring-expression check only when picking a new content word or opening a noun phrase, and fill function words and inflection from the cheap model.
- **I2** (learn): Mova's evaluators and silver-data witnesses should be trained on splits disjoint from the model they judge, which makes agreement a more honest signal.
- **I3** (answer): QA on fables and tales could use near-duplicate entities or passages as distractors, testing whether answers really pick out the right referent.
- **I4** (back): The text → graph → text compressor could report both measures, so repetitive regenerated text is caught even when overall vocabulary looks healthy.
- **I5** (frames): When tuning how strongly implicature or speech-act reasoning overrides the literal reading, Mova could sweep its strength parameter and log both the gains and the new errors.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Bielefeld University](https://md.research.elamur.ai/free/en/universities/bielefeld-university/) — University, Bielefeld, Germany · 19 works · 968 citations
- [Artificial Intelligence Research Center](https://md.research.elamur.ai/free/en/labs/artificial-intelligence-research-center/) — Research institute, Tokyo, Japan · 8 works · 1,556 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 510 works

## Cite

Cite as: Sina Zarrieß, Hendrik Buschmeier, Ting Han and Simeon Schüz. 2021. Decoding, Fast and Slow: A Case Study on Balancing Trade-Offs in Incremental, Character-level Pragmatic Reasoning. Proceedings of the 14th International Conference on Natural Language Generation.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zarriess2021decoding,
  title = {{Decoding, Fast and Slow: A Case Study on Balancing Trade-Offs in Incremental, Character-level Pragmatic Reasoning}},
  author = {Sina Zarrieß, Hendrik Buschmeier, Ting Han and Simeon Schüz},
  booktitle = {Proceedings of the 14th International Conference on Natural Language Generation},
  year = {2021},
  doi = {10.18653/v1/2021.inlg-1.41},
  url = {https://doi.org/10.18653/v1/2021.inlg-1.41}
}
```

- [← previous Combining Context-Free and Contextualized Representations for Arabic Sarcasm Detection and Sentiment Identification](https://md.research.elamur.ai/free/en/papers/hengle-2021-combining-context-free-contextualized-representations/)
- [next → Dependency Parsing Evaluation for Low-resource Spontaneous Speech](https://md.research.elamur.ai/free/en/papers/liu-2021-dependency-parsing-evaluation-low-resource/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/7/)

## References

28 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jacob Andreas and Dan Klein. 2016. [Reasoning about pragmatics with neural listeners and speakers](https://md.research.elamur.ai/free/en/papers/andreas-2016-reasoning-about-pragmatics-neural-listeners/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1173–1182, Austin, Texas. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/d16-1125)
2. Herbert H. Clark. 1996. Using Language. Cambridge University Press, Cambridge, UK. [find](https://scholar.google.com/scholar?q=Herbert+H%2E+Clark%2E+1996%2E+Using+Language%2E+Cambridge+University+Press%2C+Cambridge%2C+UK%2E)
3. Reuben Cohn-Gordon, Noah Goodman, and Christopher Potts. 2019. [An incremental iterated response model of pragmatics](https://md.research.elamur.ai/free/en/papers/gordon-2019-incremental-iterated-response-model-pragmatics/). In Proceedings of the Society for Computation in Linguistics (SCiL) 2019, pages 81–90. [↗](https://doi.org/10.48550/arxiv.1810.00367)
4. Judith Degen and Michael K Tanenhaus. 2016. Availability of alternatives and the processing of scalar implicatures: A visual world eye-tracking study. Cognitive science, 40(1):172–201. [↗](https://doi.org/10.1111/cogs.12227)
5. Judith Degen and Michael K Tanenhaus. 2019. Constraint-based pragmatic processing. Handbook of Experimental Semantics and Pragmatics. [↗](https://doi.org/10.1093/oxfordhb/9780198791768.013.8)
6. Michael C Frank and Noah D Goodman. 2012. [Predicting pragmatic reasoning in language games](https://md.research.elamur.ai/free/en/papers/frank-2012-predicting-pragmatic-reasoning-language-games/). Science, 336(6084):998–998. [↗](https://doi.org/10.1126/science.1218633)
7. Daniel Fried, Jacob Andreas, and Dan Klein. 2018. [Unified pragmatic models for generating and following instructions](https://md.research.elamur.ai/free/en/papers/fried-2018-unified-pragmatic-models-generating-following/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1951–1963, New Orleans, Louisiana. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1711.04987)
8. Alexia Galati and Susan E. Brennan. 2010. Attenuating information in spoken communication: For the speaker, or for the addressee? Journal of Memory and Language, 62:35–51. [↗](https://doi.org/10.1016/j.jml.2009.09.002)
9. H. P. Grice. 1975. Logic and conversation. In Peter Cole and Jerry L. Morgan, editors, Syntax and Semantics: Vol. 3: Speech Acts, pages 41–58. Academic Press, New York. [↗](https://doi.org/10.1163/9789004368811_003)
10. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations. [↗](https://doi.org/10.48550/arxiv.1904.09751)
11. Daphne Ippolito, Reno Kriz, João Sedoc, Maria Kustikova, and Chris Callison-Burch. 2019. Comparison of diverse decoding methods from conditional language models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3752–3762, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Daphne+Ippolito%2C+Reno+Kriz%2C+Jo%C3%A3o+Sedoc%2C+Maria+Kustikova%2C+and+Chris+Callison%2DBurch%2E+2019%2E+Comparison+of+diverse+decoding+methods+from+conditional+language+models%2E+In+Proceedings+of+the+57th+Annual+Meet)
12. Daniel Kahneman. 2011. Macmillan. Thinking, Fast and Slow. [find](https://scholar.google.com/scholar?q=Daniel+Kahneman%2E+2011%2E+Macmillan%2E+Thinking%2C+Fast+and+Slow%2E)
13. Boaz Keysar, Dale J. Barr, and William S. Horton. 1998. The egocentric basis of language use: Insights from a processing approach. Current Directions in Psychological Science, 7:46–49. [↗](https://doi.org/10.1111/1467-8721.ep13175613)
14. Jiwei Li, Will Monroe, and Dan Jurafsky. 2016. A simple, fast diverse decoding algorithm for neural generation. arXiv preprint arXiv:1611.08562. [↗](https://arxiv.org/abs/1611.08562)
15. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. [Microsoft coco: Common objects in context](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/). In European conference on computer vision, pages 740–755. Springer. [↗](https://doi.org/10.4230/lipics.cosit.2024.1)
16. Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher. 2017. Knowing when to look: Adaptive attention via a visual sentinel for image captioning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 375–383. [find](https://scholar.google.com/scholar?q=Jiasen+Lu%2C+Caiming+Xiong%2C+Devi+Parikh%2C+and+Richard+Socher%2E+2017%2E+Knowing+when+to+look%3A+Adaptive+attention+via+a+visual+sentinel+for+image+captioning%2E+In+Proceedings+of+the+IEEE+conference+on+computer+)
17. Emiel van Miltenburg, Desmond Elliott, and Piek Vossen. 2018. Measuring the diversity of automatic image descriptions. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1730–1741, Santa Fe, New Mexico, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Emiel+van+Miltenburg%2C+Desmond+Elliott%2C+and+Piek+Vossen%2E+2018%2E+Measuring+the+diversity+of+automatic+image+descriptions%2E+In+Proceedings+of+the+27th+International+Conference+on+Computational+Linguistics%2C)
18. Ehud Reiter. 1991. A new model of lexical choice for nouns. Computational Intelligence, 7(4):240–251. [find](https://scholar.google.com/scholar?q=Ehud+Reiter%2E+1991%2E+A+new+model+of+lexical+choice+for+nouns%2E+Computational+Intelligence%2C+7%284%29%3A240%E2%80%93251%2E)
19. Walter Schneider and Richard M Shiffrin. 1977. Controlled and automatic human information processing: I. detection, search, and attention. Psychological review, 84(1):1. [↗](https://doi.org/10.1037//0033-295x.84.1.1)
20. Simeon Schüz, Ting Han, and Sina Zarrieß. 2021. Diversity as a by-product: Goal-oriented language generation leads to linguistic variation. In Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 411– 422, Singapore and Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Simeon+Sch%C3%BCz%2C+Ting+Han%2C+and+Sina+Zarrie%C3%9F%2E+2021%2E+Diversity+as+a+by%2Dproduct%3A+Goal%2Doriented+language+generation+leads+to+linguistic+variation%2E+In+Proceedings+of+the+22nd+Annual+Meeting+of+the+Special+Int)
21. John Searle. 1969. Speech acts: An essay in the philosophy of language, volume 626. Cambridge university press. [find](https://scholar.google.com/scholar?q=John+Searle%2E+1969%2E+Speech+acts%3A+An+essay+in+the+philosophy+of+language%2C+volume+626%2E+Cambridge+university+press%2E)
22. Sheng Shen, Daniel Fried, Jacob Andreas, and Dan Klein. 2019. [Pragmatically informative text generation](https://md.research.elamur.ai/free/en/papers/shen-2019-pragmatically-informative-text-generation/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4060–4067, Minneapolis, Minnesota. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n19-1410)
23. Yik-Cheung Tam. 2020. Cluster-based beam search for pointer-generator chatbot grounded by knowledge. Computer Speech & Language, 64:101094. [↗](https://doi.org/10.1016/j.csl.2020.101094)
24. Ramakrishna Vedantam, Samy Bengio, Kevin Murphy, Devi Parikh, and Gal Chechik. 2017. Context-aware captions from context-agnostic supervision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 251–260. [find](https://scholar.google.com/scholar?q=Ramakrishna+Vedantam%2C+Samy+Bengio%2C+Kevin+Murphy%2C+Devi+Parikh%2C+and+Gal+Chechik%2E+2017%2E+Context%2Daware+captions+from+context%2Dagnostic+supervision%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+)
25. Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. [Cider: Consensus-based image description evaluation](https://md.research.elamur.ai/free/en/papers/vedantam-2014-cider-consensus-based-image-description/). In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575. [↗](https://doi.org/10.48550/arxiv.1411.5726)
26. Ashwin Vijayakumar, Michael Cogswell, Ramprasaath Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2018. Diverse beam search for improved description of complex scenes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32. [find](https://scholar.google.com/scholar?q=Ashwin+Vijayakumar%2C+Michael+Cogswell%2C+Ramprasaath+Selvaraju%2C+Qing+Sun%2C+Stefan+Lee%2C+David+Crandall%2C+and+Dhruv+Batra%2E+2018%2E+Diverse+beam+search+for+improved+description+of+complex+scenes%2E+In+Proceedings)
27. Julia White, Jesse Mu, and Noah Goodman. 2020. Learning to refer informatively by amortizing pragmatic reasoning. In Proceedings of the 42nd Annual Meeting of the Cognitive Science Society. [↗](https://doi.org/10.48550/arxiv.2006.00418)
28. Gilbert Youmans. 1990. Measuring lexical style and competence: The type-token vocabulary curve. Style, 24:584–599. [find](https://scholar.google.com/scholar?q=Gilbert+Youmans%2E+1990%2E+Measuring+lexical+style+and+competence%3A+The+type%2Dtoken+vocabulary+curve%2E+Style%2C+24%3A584%E2%80%93599%2E)

*Updated 24 October 2026*