---
title: "GRAVL-BERT: Graphical Visual-Linguistic — Guo 2022"
description: "The paper addresses multimodal coreference resolution (MCR): working out which objects in a scene image a user means during a multi-turn, task-oriented…"
authors: ["Danfeng Guo", "Arpit Gupta", "Sanchit Agarwal", "Jiun-Yu Kao", "Shuyang Gao", "Arijit Biswas", "Chien-Wei Lin", "Tagyoung Chung", "Mohit Bansal"]
year: 2022
venue: "Proceedings of the 29th International Conference on Computational Linguistics"
source: "https://aclanthology.org/2022.coling-1.22/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/guo-2022-gravl-bert-graphical-visual-linguistic/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/guo-2022-gravl-bert-graphical-visual-linguistic/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2022](https://md.research.elamur.ai/free/papers/2022/page/2/) / GRAVL-BERT: Graphical Visual-Linguistic Representations for Multimodal Coreference Resolution

# GRAVL-BERT: Graphical Visual-Linguistic Representations for Multimodal Coreference Resolution

**Authors:** Danfeng Guo, Arpit Gupta, [Sanchit Agarwal](https://md.research.elamur.ai/free/authors/sanchit-agarwal/), Jiun-Yu Kao, [Shuyang Gao](https://md.research.elamur.ai/free/authors/shuyang-gao/), Arijit Biswas, Chien-Wei Lin, [Tagyoung Chung](https://md.research.elamur.ai/free/authors/tagyoung-chung/), [Mohit Bansal](https://md.research.elamur.ai/free/authors/mohit-bansal/) · **Year:** 2022 · **Venue:** Proceedings of the 29th International Conference on Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.coling-1.22/) · [PDF](https://aclanthology.org/2022.coling-1.22.pdf) · [OpenAlex](https://openalex.org/W6891920313)

**Citations:** 0 citations (openalex-snapshot · 2026-10-14) · [6](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
The paper addresses multimodal coreference resolution (MCR): working out which objects in a scene image a user means during a multi-turn, task-oriented dialogue. This is harder than visual grounding. References may rely on spatial relations between objects, on earlier dialogue turns, or on non-visual catalog attributes such as brand or price. Scenes can also contain over a hundred similar items. The authors present GRAVL-BERT, which extends VL-BERT so that each candidate object is scored on its own. The model input combines:
- Fast-RCNN features for the scene, the object and sampled neighboring regions
- a GCN layer that encodes relative positions among objects
- flattened metadata text and generated captions
- a token marking how many turns ago the agent last offered the object

The model is further pre-trained with MLM on dialogue data. On SIMMC 2.0, a shopping-domain dataset, it ranked first in the DSTC-10 MCR track. The work shows that combining dialogue context, scene graphs and external knowledge in a single BERT-style model substantially improves reference resolution for multimodal assistants.

## Key points
- GRAVL-BERT ranked 1st in the DSTC-10 SIMMC 2.0 MCR challenge, with object-level F1 of 0.783 on test and about 0.76 on devtest. The code is released at github.com/alexa/gravl-bert.
- It beats the GPT-2-based SIMMC 2.0 baseline by 33.9 points absolute and concurrent systems by about 2.5 points.
- Further pre-training on SIMMC 2.0 dialogues gave a large gain over the original VL-BERT. VD-BERT pre-trained on VisDial showed similar benefits.
- In ablations, the GCN and the reference-distance token gave the main gains. Surrounding features added 0.41% and captions added 0.80% in F1.
- Compared with pre-trained VL-BERT, object-level error rates fell by 58–73% across all reference types: spatial, non-target objects, dialogue history, metadata and purely visual. Remaining failures involve global scene reasoning, such as counting cabinets.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

Topics: [Coreference resolution](https://md.research.elamur.ai/free/topics/coreference-resolution/)

## Cited by

Cited by

6 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 5 not yet.

- [Multimodal Cross-Document Event Coreference Resolution Using Linear Semantic Transfer and Mixed-Modality Ensembles](https://md.research.elamur.ai/free/papers/nath-2024-multimodal-cross-document-event-coreference/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{guo2022gravl,
  title = {{GRAVL-BERT: Graphical Visual-Linguistic Representations for Multimodal Coreference Resolution}},
  author = {Guo, Danfeng and Gupta, Arpit and Agarwal, Sanchit and Kao, Jiun-Yu and Gao, Shuyang and Biswas, Arijit and Lin, Chien-Wei and Chung, Tagyoung and Bansal, Mohit},
  booktitle = {Proceedings of the 29th International Conference on Computational Linguistics},
  year = {2022},
  doi = {10.48448/phm0-3037},
  url = {https://doi.org/10.48448/phm0-3037}
}
```

- [← previous Graph Pre-training for AMR Parsing and Generation](https://md.research.elamur.ai/free/papers/bai-2022-graph-pre-training-amr-parsing/)
- [next → GuoFeng: A Benchmark for Zero Pronoun Recovery and Translation](https://md.research.elamur.ai/free/papers/xu-2022-guofeng-benchmark-zero-pronoun-recovery/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/papers/2022/page/2/)

## References

28 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Marc Brockschmidt. 2020. GNN-FiLM: Graph neural networks with feature-wise linear modulation. In ICML, volume 119 of PMLR, pages 1144–1152. PMLR. [find](https://scholar.google.com/scholar?q=Marc+Brockschmidt%2E+2020%2E+GNN%2DFiLM%3A+Graph+neural+networks+with+feature%2Dwise+linear+modulation%2E+In+ICML%2C+volume+119+of+PMLR%2C+pages+1144%E2%80%931152%2E+PMLR%2E)
2. Pingping Huang, Jianhui Huang, Yuqing Guo, Min Qiao, and Yong Zhu. 2019b. Multi-grained attention with object-level grounding for visual question answering. In ACL, pages 3595–3600, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Pingping+Huang%2C+Jianhui+Huang%2C+Yuqing+Guo%2C+Min+Qiao%2C+and+Yong+Zhu%2E+2019b%2E+Multi%2Dgrained+attention+with+object%2Dlevel+grounding+for+visual+question+answering%2E+In+ACL%2C+pages+3595%E2%80%933600%2C+Florence%2C+Italy%2E+A)
3. Yichen Huang, Yuchen Wang, and Yik-Cheung Tam. 2021. Uniter-based situated coreference resolution with rich multimodal input. arXiv preprint arXiv:2112.03521. — [arXiv:2112.03521](https://arxiv.org/abs/2112.03521) [find](https://scholar.google.com/scholar?q=Yichen+Huang%2C+Yuchen+Wang%2C+and+Yik%2DCheung+Tam%2E+2021%2E+Uniter%2Dbased+situated+coreference+resolution+with+rich+multimodal+input%2E+arXiv+preprint+arXiv%3A2112%2E03521%2E+%E2%80%94+%5BarXiv%3A2112%2E03521%5D%28https%3A%2F%2Farxiv%2Eorg%2Fab)
4. Kanishk Jain and Vineet Gandhi. 2021. Comprehensive multi-modal interactions for referring image segmentation. arXiv preprint arXiv:abs/2104.10412. [find](https://scholar.google.com/scholar?q=Kanishk+Jain+and+Vineet+Gandhi%2E+2021%2E+Comprehensive+multi%2Dmodal+interactions+for+referring+image+segmentation%2E+arXiv+preprint+arXiv%3Aabs%2F2104%2E10412%2E)
5. Satwik Kottur, Seungwhan Moon, Alborz Geramifard, and Babak Damavandi. 2021. Simmc 2.0: A task-oriented dialog dataset for immersive multimodal conversations. arXiv preprint arXiv:2104.08667. [find](https://scholar.google.com/scholar?q=Satwik+Kottur%2C+Seungwhan+Moon%2C+Alborz+Geramifard%2C+and+Babak+Damavandi%2E+2021%2E+Simmc+2%2E0%3A+A+task%2Doriented+dialog+dataset+for+immersive+multimodal+conversations%2E+arXiv+preprint+arXiv%3A2104%2E08667%2E)
6. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2017. Imagenet classification with deep convolutional neural networks. ACM, 60(6):84–90. [find](https://scholar.google.com/scholar?q=Alex+Krizhevsky%2C+Ilya+Sutskever%2C+and+Geoffrey+E%2E+Hinton%2E+2017%2E+Imagenet+classification+with+deep+convolutional+neural+networks%2E+ACM%2C+60%286%29%3A84%E2%80%9390%2E)
7. Hung Le, Doyen Sahoo, Nancy F. Chen, and S. Hoi. 2019. Multimodal transformer networks for end-to-end video-grounded dialogue systems. In ACL. [find](https://scholar.google.com/scholar?q=Hung+Le%2C+Doyen+Sahoo%2C+Nancy+F%2E+Chen%2C+and+S%2E+Hoi%2E+2019%2E+Multimodal+transformer+networks+for+end%2Dto%2Dend+video%2Dgrounded+dialogue+systems%2E+In+ACL%2E)
8. Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Uniter: Universal image-text representation learning. In ECCV. [find](https://scholar.google.com/scholar?q=Yen%2DChun+Chen%2C+Linjie+Li%2C+Licheng+Yu%2C+Ahmed+El+Kholy%2C+Faisal+Ahmed%2C+Zhe+Gan%2C+Yu+Cheng%2C+and+Jingjing+Liu%2E+2020%2E+Uniter%3A+Universal+image%2Dtext+representation+learning%2E+In+ECCV%2E) [our card](https://md.research.elamur.ai/free/papers/chen-2019-uniter-universal-image-text-representation/)
9. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In acl, pages 7871–7880. [find](https://scholar.google.com/scholar?q=Mike+Lewis%2C+Yinhan+Liu%2C+Naman+Goyal%2C+Marjan+Ghazvininejad%2C+Abdelrahman+Mohamed%2C+Omer+Levy%2C+Veselin+Stoyanov%2C+and+Luke+Zettlemoyer%2E+2020%2E+Bart%3A+Denoising+sequence%2Dto%2Dsequence+pre%2Dtraining+for+natural+l) [our card](https://md.research.elamur.ai/free/papers/lewis-2020-bart-denoising-sequence-sequence-pre/)
10. Vinay Damodaran, Sharanya Chakravarthy, Akshay Kumar, et al. 2021. Understanding the role of scene graphs in visual question answering. CoRR, abs/2101.05479. — [arXiv:2101.05479](https://arxiv.org/abs/2101.05479) [find](https://scholar.google.com/scholar?q=Vinay+Damodaran%2C+Sharanya+Chakravarthy%2C+Akshay+Kumar%2C+et+al%2E+2021%2E+Understanding+the+role+of+scene+graphs+in+visual+question+answering%2E+CoRR%2C+abs%2F2101%2E05479%2E+%E2%80%94+%5BarXiv%3A2101%2E05479%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs)
11. Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, Stefan Lee, José M. F. Moura, Devi Parikh, and Dhruv Batra. 2019. Visual dialog. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41:1242–1256. [find](https://scholar.google.com/scholar?q=Abhishek+Das%2C+Satwik+Kottur%2C+Khushi+Gupta%2C+Avi+Singh%2C+Deshraj+Yadav%2C+Stefan+Lee%2C+Jos%C3%A9+M%2E+F%2E+Moura%2C+Devi+Parikh%2C+and+Dhruv+Batra%2E+2019%2E+Visual+dialog%2E+IEEE+Transactions+on+Pattern+Analysis+and+Machine+)
12. Chaorui Deng, Qi Wu, Qingyao Wu, et al. 2018. Visual grounding via accumulated attention. In CVPR. [find](https://scholar.google.com/scholar?q=Chaorui+Deng%2C+Qi+Wu%2C+Qingyao+Wu%2C+et+al%2E+2018%2E+Visual+grounding+via+accumulated+attention%2E+In+CVPR%2E)
13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In ACL, pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+ACL%2C+pages+4171%E2%80%934186%2C+Minneapolis%2C+Minnesot)
14. Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang. 2021. Vision-language transformer and query generation for referring segmentation. In ICCV. [find](https://scholar.google.com/scholar?q=Henghui+Ding%2C+Chang+Liu%2C+Suchen+Wang%2C+and+Xudong+Jiang%2E+2021%2E+Vision%2Dlanguage+transformer+and+query+generation+for+referring+segmentation%2E+In+ICCV%2E)
15. Gen Li, Nan Duan, Yuejian Fang, et al. 2020a. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. AAAI, 34(07):11336–11344. [find](https://scholar.google.com/scholar?q=Gen+Li%2C+Nan+Duan%2C+Yuejian+Fang%2C+et+al%2E+2020a%2E+Unicoder%2Dvl%3A+A+universal+encoder+for+vision+and+language+by+cross%2Dmodal+pre%2Dtraining%2E+AAAI%2C+34%2807%29%3A11336%E2%80%9311344%2E)
16. Xiujun Li, Xi Yin, Chunyuan Li, et al. 2020b. Oscar: Object-semantics aligned pre-training for vision-language tasks. In ECCV. [find](https://scholar.google.com/scholar?q=Xiujun+Li%2C+Xi+Yin%2C+Chunyuan+Li%2C+et+al%2E+2020b%2E+Oscar%3A+Object%2Dsemantics+aligned+pre%2Dtraining+for+vision%2Dlanguage+tasks%2E+In+ECCV%2E) [our card](https://md.research.elamur.ai/free/papers/li-2020-oscar-object-semantics-aligned-pre/)
17. J. Lu, V. Goswami, M. Rohrbach, et al. 2020. 12-in-1: Multi-task vision and language representation learning. In CVPR, pages 10434–10443, Los Alamitos, CA, USA. IEEE Computer Society. [find](https://scholar.google.com/scholar?q=J%2E+Lu%2C+V%2E+Goswami%2C+M%2E+Rohrbach%2C+et+al%2E+2020%2E+12%2Din%2D1%3A+Multi%2Dtask+vision+and+language+representation+learning%2E+In+CVPR%2C+pages+10434%E2%80%9310443%2C+Los+Alamitos%2C+CA%2C+USA%2E+IEEE+Computer+Society%2E)
18. Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguis-tic representations for vision-and-language tasks. In NIPS. [find](https://scholar.google.com/scholar?q=Jiasen+Lu%2C+Dhruv+Batra%2C+Devi+Parikh%2C+and+Stefan+Lee%2E+2019%2E+Vilbert%3A+Pretraining+task%2Dagnostic+visiolinguis%2Dtic+representations+for+vision%2Dand%2Dlanguage+tasks%2E+In+NIPS%2E)
19. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation, 9(8):1735–1780. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+J%C3%BCrgen+Schmidhuber%2E+1997%2E+Long+Short%2DTerm+Memory%2E+Neural+Computation%2C+9%288%29%3A1735%E2%80%931780%2E)
20. Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556–2565,… [find](https://scholar.google.com/scholar?q=Piyush+Sharma%2C+Nan+Ding%2C+Sebastian+Goodman%2C+and+Radu+Soricut%2E+2018%2E+Conceptual+captions%3A+A+cleaned%2C+hypernymed%2C+image+alt%2Dtext+dataset+for+automatic+image+captioning%2E+In+Proceedings+of+the+56th+Annual) [our card](https://md.research.elamur.ai/free/papers/sharma-2018-conceptual-captions-cleaned-hypernymed-image/)
21. L Huang, W Wang, Y Xia, and J Chen. 2019a. Adaptively aligned image captioning via adaptive attention time. In NIPS, pages 8942–8951. [find](https://scholar.google.com/scholar?q=L+Huang%2C+W+Wang%2C+Y+Xia%2C+and+J+Chen%2E+2019a%2E+Adaptively+aligned+image+captioning+via+adaptive+attention+time%2E+In+NIPS%2C+pages+8942%E2%80%938951%2E)
22. Weijie Su, Xizhou Zhu, Yue Cao, et al. 2020. Vl-bert: Pre-training of generic visual-linguistic representations. In ICLR. [find](https://scholar.google.com/scholar?q=Weijie+Su%2C+Xizhou+Zhu%2C+Yue+Cao%2C+et+al%2E+2020%2E+Vl%2Dbert%3A+Pre%2Dtraining+of+generic+visual%2Dlinguistic+representations%2E+In+ICLR%2E) [our card](https://md.research.elamur.ai/free/papers/su-2019-vl-bert-pre-training-generic/)
23. Ross Girshick. 2015. Fast r-cnn. In International Conference on Computer Vision (ICCV). [find](https://scholar.google.com/scholar?q=Ross+Girshick%2E+2015%2E+Fast+r%2Dcnn%2E+In+International+Conference+on+Computer+Vision+%28ICCV%29%2E)
24. Hao Tan and Mohit Bansal. 2019. LXMERT: learning cross-modality encoder representations from transformers. In EMNLP. [find](https://scholar.google.com/scholar?q=Hao+Tan+and+Mohit+Bansal%2E+2019%2E+LXMERT%3A+learning+cross%2Dmodality+encoder+representations+from+transformers%2E+In+EMNLP%2E) [our card](https://md.research.elamur.ai/free/papers/tan-2019-lxmert-learning-cross-modality-encoder/)
25. Yue Wang, Shafiq Joty, Michael Lyu, et al. 2020. VD-BERT: A Unified Vision and Dialog Transformer with BERT. In EMNLP, pages 3325–3338, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yue+Wang%2C+Shafiq+Joty%2C+Michael+Lyu%2C+et+al%2E+2020%2E+VD%2DBERT%3A+A+Unified+Vision+and+Dialog+Transformer+with+BERT%2E+In+EMNLP%2C+pages+3325%E2%80%933338%2C+Online%2E+Association+for+Computational+Linguistics%2E)
26. Jianwei Yang, Jiasen Lu, Stefan Lee, et al. 2018. Graph r-cnn for scene graph generation. In ECCV, pages 690–706, Cham. Springer International Publishing. [find](https://scholar.google.com/scholar?q=Jianwei+Yang%2C+Jiasen+Lu%2C+Stefan+Lee%2C+et+al%2E+2018%2E+Graph+r%2Dcnn+for+scene+graph+generation%2E+In+ECCV%2C+pages+690%E2%80%93706%2C+Cham%2E+Springer+International+Publishing%2E)
27. Xu Yang, Kaihua Tang, Hanwang Zhang, and Jianfei Cai. 2019. Auto-encoding scene graphs for image captioning. In CVPR. [find](https://scholar.google.com/scholar?q=Xu+Yang%2C+Kaihua+Tang%2C+Hanwang+Zhang%2C+and+Jianfei+Cai%2E+2019%2E+Auto%2Dencoding+scene+graphs+for+image+captioning%2E+In+CVPR%2E)
28. Ting Yao, Yingwei Pan, Yehao Li, and Tao Mei. 2018. Exploring visual relationship for image captioning. In ECCV. [find](https://scholar.google.com/scholar?q=Ting+Yao%2C+Yingwei+Pan%2C+Yehao+Li%2C+and+Tao+Mei%2E+2018%2E+Exploring+visual+relationship+for+image+captioning%2E+In+ECCV%2E)