--- title: "Referring Expressions with Rational Speech Act — Le 2022" description: "The paper generates referring expressions (\"the dog with the frisbee\") for objects in photographs with the Rational Speech Act (RSA) model instead of a…" authors: ["Hieu Le", "Taufiq Daryanto", "Fabian Zhafransyah", "Derry Wijaya", "Elizabeth Coppock", "Sang Chin"] year: 2022 venue: "arXiv:2205.07795" source: "https://arxiv.org/abs/2205.07795" arxiv: "2205.07795" licence: "CC BY 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/le-2022-referring-expressions-rational-speech-act/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/le-2022-referring-expressions-rational-speech-act/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/3/) / Referring Expressions with Rational Speech Act Framework: A Probabilistic Approach # Referring Expressions with Rational Speech Act Framework: A Probabilistic Approach **Authors:** Hieu Le, Taufiq Daryanto, Fabian Zhafransyah, Derry Wijaya, [Elizabeth Coppock](https://md.research.elamur.ai/free/en/authors/elizabeth-coppock/), Sang Chin · **Year:** 2022 · **Venue:** arXiv:2205.07795 (preprint) **Link:** [arxiv.org/abs/2205.07795](https://arxiv.org/abs/2205.07795) · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) (arXiv) [**1** citations](https://openalex.org/works?filter=cites:W4307656474) OpenAlex · 2026-10-14 · [**1** citations in our library](https://md.research.elamur.ai/free/en/papers/le-2022-referring-expressions-rational-speech-act/cited-by/) of the 140673 papers whose bibliographies we read · [arXiv:2205.07795](https://openalex.org/S4306400194) · [licence](http://creativecommons.org/licenses/by/4.0/) - [Publisher's page](https://arxiv.org/abs/2205.07795) - [PDF](https://arxiv.org/pdf/2205.07795) - [arXiv 2205.07795](https://arxiv.org/abs/2205.07795) - [OpenAlex](https://openalex.org/W4307656474) **Authors (6)** 1. [Hieu Le](https://openalex.org/A5101977591) [ORCID](https://orcid.org/0000-0001-7855-2778) · h 17 · 1415 citations · 51 works 2. [Taufiq Daryanto](https://openalex.org/A5051502019) h 1 · 1 citations · 3 works 3. [Fabian Zhafransyah](https://openalex.org/A5024229022) h 1 · 11 citations · 4 works 4. [Derry Wijaya](https://openalex.org/A5027174211) [ORCID](https://orcid.org/0000-0002-0848-4703) · h 19 · 1800 citations · 119 works 5. [Elizabeth Coppock](https://openalex.org/A5074610868) [ORCID](https://orcid.org/0000-0001-9987-344X) · h 17 · 1120 citations · 69 works 6. [Sang Chin](https://openalex.org/A5103044811) [ORCID](https://orcid.org/0000-0002-1913-4223) · h 15 · 981 citations · 81 works Citations and author metrics: OpenAlex, 2026-10-14; authors with the same name are not always told apart. ## What it is about The paper generates referring expressions ("the dog with the frisbee") for objects in photographs with the Rational Speech Act (RSA) model instead of a neural speaker. A scene graph (object types, attributes, relations) is first extracted with pre-trained Detectron2 and Graph R-CNN, and ordinal relations (left, right, second from left) are added by sorting objects by x-coordinate. The speaker adds one descriptor at a time until the listener's entropy over objects falls below a threshold or four descriptors are used. On RefCOCO and RefCOCO+ the expressions are checked by human choice among boxes. The method does not beat an end-to-end neural model trained on full data, and the authors trace many failures (boxes with no match, or not highlighted) to the vision step rather than to the pragmatic one. The tables are hard to align in the text read here, so the card quotes only one clear figure: an adjusted accuracy of 52.54% on RefCOCO+. ## Ideas - I1. `prag/src/rsa.rs`: a deterministic version of the speaker. At each step choose the descriptor that most reduces the set of candidate referents, with ties broken by the descriptor string in `BTreeMap` order, and stop when one candidate remains or after a fixed length. Compare length and success with the sampling version on the school lessons. - I2. `world` thresholds: the paper fixes one threshold for attributes and one for relations by hand, and says it would learn them from data. Fit each threshold on a training split and report it on a held-out split (`cv`), never on the test set. - I3. Ordinal descriptors as a rule in the story world: sort the same-kind objects by position in the text (first, second, last) and add "second from left"-style descriptors. Test on the school scene questions, with a negative control where the order is shuffled. - I4. Keep the trace: store the descriptor sequence and the entropy after each step for each referring expression. This makes each choice explainable (the "why" the project needs), and the trace is the input to an error table like the paper's. ## For Mova The speaker and listener belong with `prag/src/rsa.rs` and `prag/src/infer.rs`. The scene and facts are in `world/src/knowledge.rs` and `world/src/events.rs`. Referring expressions and the choice of descriptor relate to [languages/en/35-reference-and-coreference.md](https://mova.elamur.ai/free/en/docs/languages/en/35-reference-and-coreference/) and to the quantity implicature in [languages/en/33-implicature-and-presupposition.md](https://mova.elamur.ai/free/en/docs/languages/en/33-implicature-and-presupposition/). Measures: expression length and success on the school scene questions, with the random-order control, and the four error categories on a held-out scene set. Topics: [Speech act theory](https://md.research.elamur.ai/free/en/topics/speech-act-theory/) ## Cite The paper itself, not this card. BibTeX from the library's record of the work. [arXiv 2205.07795](https://arxiv.org/abs/2205.07795) ```bibtex @misc{le2022referring, title = {{Referring Expressions with Rational Speech Act Framework: A Probabilistic Approach}}, author = {Le, Hieu and Daryanto, Taufiq and Zhafransyah, Fabian and Wijaya, Derry and Coppock, Elizabeth and Chin, Sang}, year = {2022}, eprint = {2205.07795}, archivePrefix = {arXiv}, url = {https://arxiv.org/abs/2205.07795} } ``` - [← previous Reducing Activation Recomputation in Large Transformer Models](https://md.research.elamur.ai/free/en/papers/korthikanti-2022-reducing-activation-recomputation-large-transformer/) - [next → Reflect, Not Reflex: Inference-Based Common Ground Improves Dialogue Response Quality](https://md.research.elamur.ai/free/en/papers/zhou-2022-reflect-not-reflex-inference-based/) - [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/3/) ## References 18 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. W. Monroe and C. Potts, “Learning in the Rational Speech Acts model,” CoRR, vol. abs/1510.06807, 2015. [↗](https://arxiv.org/abs/1510.06807) 2. S. Kazemzadeh, V. Ordonez, M. Matten, and T. Berg, “ReferItGame: Referring to objects in photographs of natural scenes,” in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), (Doha, Qatar), pp. 787–798, ACL, Oct. 2014. [↗](https://doi.org/10.3115/v1/d14-1086) 3. M. C. Frank and N. D. Goodman, “Predicting pragmatic reasoning in language games,” Science, vol. 336, no. 6084, p. 998, 2012. [↗](https://doi.org/10.1126/science.1218633) [our card](https://md.research.elamur.ai/free/en/papers/frank-2012-predicting-pragmatic-reasoning-language-games/) 4. J. Degen, R. X. D. Hawkins, C. Graf, E. Kreiss, and N. D. Goodman, “When redundancy is rational: A Bayesian approach to ’overinformative’ referring expressions,” CoRR, vol. abs/1903.08237, 2019. [↗](https://arxiv.org/abs/1903.08237) 5. J. Andreas and D. Klein, “Reasoning about pragmatics with neural listeners and speakers,” in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, (Austin, Texas), pp. 1173–1182, ACL, Nov. 2016. [↗](https://doi.org/10.18653/v1/d16-1125) [our card](https://md.research.elamur.ai/free/en/papers/andreas-2016-reasoning-about-pragmatics-neural-listeners/) 6. R. Cohn-Gordon, N. D. Goodman, and C. Potts, “Pragmatically informative image captioning with character-level reference,” CoRR, vol. abs/1804.05417, 2018. [↗](https://arxiv.org/abs/1804.05417) 7. J. Yang, J. Lu, S. Lee, D. Batra, and D. Parikh, “Graph R-CNN for scene graph generation,” CoRR, vol. abs/1808.00191, 2018. [↗](https://arxiv.org/abs/1808.00191) 8. Y. Wu, A. Kirillov, F. Massa, W.-Y. Lo, and R. Girshick, “Detectron2.” [https://github.com/facebookresearch/detectron2](https://github.com/facebookresearch/detectron2), 2019. [↗](https://github.com/facebookresearch/detectron2) 9. L. Yu, P. Poirson, S. Yang, A. C. Berg, and T. L. Berg, “Modeling context in referring expressions,” CoRR, vol. abs/1608.00272, 2016. [↗](https://arxiv.org/abs/1608.00272) [our card](https://md.research.elamur.ai/free/en/papers/yu-2016-modeling-context-referring-expressions/) 10. G. Scontras, M. H. Tessler, and M. Franke, “Probabilistic language understanding: An introduction to the rational speech act framework,” 2018. [find](https://scholar.google.com/scholar?q=G%2E+Scontras%2C+M%2E+H%2E+Tessler%2C+and+M%2E+Franke%2C+%E2%80%9CProbabilistic+language+understanding%3A+An+introduction+to+the+rational+speech+act+framework%2C%E2%80%9D+2018%2E) 11. S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards real-time object detection with region proposal networks,” in Advances in Neural Information Processing Systems (NIPS), 2015. [↗](https://doi.org/10.1109/tpami.2016.2577031) [our card](https://md.research.elamur.ai/free/en/papers/ren-2015-faster-r-cnn-towards-real/) 12. W. Abdulla, “Mask R-CNN for object detection and instance segmentation on Keras and TensorFlow.” [https://github.com/matterport/Mask_RCNN](https://github.com/matterport/Mask_RCNN), 2017. [↗](https://github.com/matterport/Mask_RCNN) 13. R. Luo and G. Shakhnarovich, “Comprehension-guided referring expressions,” CoRR, vol. abs/1701.03439, 2017. [↗](https://arxiv.org/abs/1701.03439) 14. D. Lassiter and N. Goodman, “Adjectival vagueness in a Bayesian model of interpretation,” Synthese, vol. 10, pp. 3801–3836, 2017. [find](https://scholar.google.com/scholar?q=D%2E+Lassiter+and+N%2E+Goodman%2C+%E2%80%9CAdjectival+vagueness+in+a+Bayesian+model+of+interpretation%2C%E2%80%9D+Synthese%2C+vol%2E+10%2C+pp%2E+3801%E2%80%933836%2C+2017%2E) 15. D. Edgington, “The philosophical problem of vagueness,” Legal Theory, vol. 7, pp. 371–378, 2001. [↗](https://doi.org/10.1017/s1352325201704028) 16. L. Yu, H. Tan, M. Bansal, and T. L. Berg, “A joint speaker-listener-reinforcer model for referring expressions,” 2017. [find](https://scholar.google.com/scholar?q=L%2E+Yu%2C+H%2E+Tan%2C+M%2E+Bansal%2C+and+T%2E+L%2E+Berg%2C+%E2%80%9CA+joint+speaker%2Dlistener%2Dreinforcer+model+for+referring+expressions%2C%E2%80%9D+2017%2E) 17. J. Kim, H. Ko, and J. Wu, “CoNAN: A complementary neighboring-based attention network for referring expression generation,” in Proceedings of the 28th International Conference on Computational Linguistics, (Barcelona, Spain (Online)), pp. 1952– 1962, International Committee on Computational Linguistics, Dec. 2020. [find](https://scholar.google.com/scholar?q=J%2E+Kim%2C+H%2E+Ko%2C+and+J%2E+Wu%2C+%E2%80%9CCoNAN%3A+A+complementary+neighboring%2Dbased+attention+network+for+referring+expression+generation%2C%E2%80%9D+in+Proceedings+of+the+28th+International+Conference+on+Computational+Linguis) 18. P. Joshi, S. Santy, A. Budhiraja, K. Bali, and M. Choudhury, “The state and fate of linguistic diversity and inclusion in the NLP world,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, (Online), pp. 6282–6293, Association for Computational Linguistics, July 2020. [find](https://scholar.google.com/scholar?q=P%2E+Joshi%2C+S%2E+Santy%2C+A%2E+Budhiraja%2C+K%2E+Bali%2C+and+M%2E+Choudhury%2C+%E2%80%9CThe+state+and+fate+of+linguistic+diversity+and+inclusion+in+the+NLP+world%2C%E2%80%9D+in+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+f) [our card](https://md.research.elamur.ai/free/en/papers/joshi-2020-state-fate-linguistic-diversity-inclusion/)