---
title: "PAIR: A Pilot Dataset for Dual — Watson 2026"
description: "In many collaborative settings, such as human–robot interaction or remote teamwork, partners see different parts of the same scene."
authors: ["Lewis N. Watson", "Carl Strathearn", "Kenny Mitchell", "Yanchao Yu"]
year: 2026
venue: "Proceedings of the Fifteenth Language Resources and Evaluation Conference"
source: "https://aclanthology.org/2026.lrec-1.216/"
doi: "10.63317/5gun3w98ovtb"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/watson-2026-pair-pilot-dataset-dual-perspective/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/watson-2026-pair-pilot-dataset-dual-perspective/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/7/) / PAIR: A Pilot Dataset for Dual Perspective-based Video-Grounded Dialogue and Reconciliation

# PAIR: A Pilot Dataset for Dual Perspective-based Video-Grounded Dialogue and Reconciliation

**Authors:** [Lewis N. Watson](https://md.research.elamur.ai/free/en/authors/lewis-n-watson/), [Carl Strathearn](https://md.research.elamur.ai/free/en/authors/carl-strathearn/), [Kenny Mitchell](https://md.research.elamur.ai/free/en/authors/kenny-mitchell/), [Yanchao Yu](https://md.research.elamur.ai/free/en/authors/yanchao-yu/) · **Year:** 2026 · **Venue:** Proceedings of the Fifteenth Language Resources and Evaluation Conference · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.lrec-1.216/) · [PDF](https://aclanthology.org/2026.lrec-1.216.pdf) · [DOI 10.63317/5gun3w98ovtb](https://doi.org/10.63317/5gun3w98ovtb) · [OpenAlex](https://openalex.org/W7160092827)

**Citations:** 0 citations (openalex-snapshot · 2026-10-14)

## Summary
In many collaborative settings, such as human–robot interaction or remote teamwork, partners see different parts of the same scene. Most dialogue corpora, however, either give both speakers identical visual input or limit them to question–answer exchanges. PAIR is a small pilot corpus built to study how people reach a common account when their perceptual access differs.

Thirty participants from a university were paired. In each session, one partner watched the egocentric recording of a skilled activity from Ego-Exo4D and the other watched the exocentric one. They then talked face to face, without a script, until they agreed on the events. The audio was transcribed with WhisperX and corrected by hand, especially speaker diarization. Utterances were split at 500 ms pauses and labelled with 42 dialogue act categories.

The authors analysed turn distribution, participation balance and dialogue act composition. Reconciliation took different shapes: some pairs built the narrative jointly, others ran lopsided, clarification-driven exchanges, and some mixed both. The authors present PAIR as a controlled benchmark for studying perspective integration, not as training data.

## Key points
- 15 dialogues: 5 Ego-Exo4D clips (about 5 minutes each; e.g., cooking, sports, cycling), each discussed by 3 independent pairs. Each discussion lasted roughly 10 minutes.
- Pipeline: WhisperX transcription, manual verification of speakers and tokens, 500 ms segmentation threshold, and annotation with 42 dialogue act categories.
- Two interaction styles were identified, "Iterative Freeflow" and "Q&A", plus mixed cases. They were characterised through participation symmetry, floor exchange and dialogue act sequencing.
- The release contains only anonymised transcripts, annotations and metadata. Ego-Exo4D videos must be obtained separately, and non-verbal cues such as gaze and gesture were not preserved.
- The corpus is available at github.com/lewiswatson55/PAIR_Corpus. Suggested uses include dialogue act classification, video-grounded dialogue modelling and multi-agent reasoning research.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

Topics: [Speech act theory](https://md.research.elamur.ai/free/en/topics/speech-act-theory/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{watson2026pair,
  title = {{PAIR: A Pilot Dataset for Dual Perspective-based Video-Grounded Dialogue and Reconciliation}},
  author = {Watson, Lewis N. and Strathearn, Carl and Mitchell, Kenny and Yu, Yanchao},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/5gun3w98ovtb},
  url = {https://doi.org/10.63317/5gun3w98ovtb}
}
```

- [← previous Paintings, Mosaics, Icons. A Syriac Hymn on the Cathedral of Edessa](https://md.research.elamur.ai/free/en/papers/polanski-2026-paintings-mosaics-icons-syriac-hymn/)
- [next → ParaCLEAN: Improving Translation Quality through Systematic Parallel Data Cleaning](https://md.research.elamur.ai/free/en/papers/mash-2026-paraclean-improving-translation-quality-through/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/7/)

## References

19 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Lavisha Aggarwal, Vikas Bahirwani, Lin Li, and Andrea Colaco. 2025. Generating dialogues from egocentric instructional videos for task assistance: Dataset, method and benchmark. arXiv preprint. [find](https://scholar.google.com/scholar?q=Lavisha+Aggarwal%2C+Vikas+Bahirwani%2C+Lin+Li%2C+and+Andrea+Colaco%2E+2025%2E+Generating+dialogues+from+egocentric+instructional+videos+for+task+assistance%3A+Dataset%2C+method+and+benchmark%2E+arXiv+preprint%2E)
2. James F Allen, Lenhart K Schubert, George Ferguson, Peter Heeman, Chung Hee Hwang, Tsuneaki Kato, Marc Light, Nathaniel Martin, Bradford Miller, Massimo Poesio, et al. 1995. The trains project: A case study in building a conversational planning agent. Journal of Experimental & Theoretical Artificial… [find](https://scholar.google.com/scholar?q=James+F+Allen%2C+Lenhart+K+Schubert%2C+George+Ferguson%2C+Peter+Heeman%2C+Chung+Hee+Hwang%2C+Tsuneaki+Kato%2C+Marc+Light%2C+Nathaniel+Martin%2C+Bradford+Miller%2C+Massimo+Poesio%2C+et+al%2E+1995%2E+The+trains+project%3A+A+case)
3. Anne H Anderson, Miles Bader, Ellen Gurman Bard, Elizabeth Boyle, Gwyneth Doherty, Simon Garrod, Stephen Isard, Jacqueline Kowtko, Jan McAllister, Jim Miller, et al. 1991. The hcrc map task corpus. Language and Speech, 34(4):351– 366. [find](https://scholar.google.com/scholar?q=Anne+H+Anderson%2C+Miles+Bader%2C+Ellen+Gurman+Bard%2C+Elizabeth+Boyle%2C+Gwyneth+Doherty%2C+Simon+Garrod%2C+Stephen+Isard%2C+Jacqueline+Kowtko%2C+Jan+McAllister%2C+Jim+Miller%2C+et+al%2E+1991%2E+The+hcrc+map+task+corpus%2E+La)
4. Max Bain, Jaesung Huh, Tengda Han, and Andrew Zisserman. 2023. Whisperx: Time-accurate speech transcription of long-form audio. [find](https://scholar.google.com/scholar?q=Max+Bain%2C+Jaesung+Huh%2C+Tengda+Han%2C+and+Andrew+Zisserman%2E+2023%2E+Whisperx%3A+Time%2Daccurate+speech+transcription+of+long%2Dform+audio%2E)
5. Jean Carletta, Simone Ashby, Sebastien Bourban, Mike Flynn, Mael Guillemot, Thomas Hain, Jaroslav Kadlec, Vasilis Karaiskos, Wessel Kraaij, Melissa Kronenthal, et al. 2005. The ami meeting corpus: A pre-announcement. In International Workshop on Machine Learning for Multimodal Interaction, pages 28–… [find](https://scholar.google.com/scholar?q=Jean+Carletta%2C+Simone+Ashby%2C+Sebastien+Bourban%2C+Mike+Flynn%2C+Mael+Guillemot%2C+Thomas+Hain%2C+Jaroslav+Kadlec%2C+Vasilis+Karaiskos%2C+Wessel+Kraaij%2C+Melissa+Kronenthal%2C+et+al%2E+2005%2E+The+ami+meeting+corpus%3A+A+p)
6. Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. 2017. Visual dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 326– 335. [find](https://scholar.google.com/scholar?q=Abhishek+Das%2C+Satwik+Kottur%2C+Khushi+Gupta%2C+Avi+Singh%2C+Deshraj+Yadav%2C+Jos%C3%A9+MF+Moura%2C+Devi+Parikh%2C+and+Dhruv+Batra%2E+2017%2E+Visual+dialog%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Patt)
7. Harm De Vries, Florian Strub, Sarath Chandar, Olivier Pietquin, Hugo Larochelle, and Aaron Courville. 2017. Guesswhat?! visual object discovery through multi-modal dialogue. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5503–5512. [find](https://scholar.google.com/scholar?q=Harm+De+Vries%2C+Florian+Strub%2C+Sarath+Chandar%2C+Olivier+Pietquin%2C+Hugo+Larochelle%2C+and+Aaron+Courville%2E+2017%2E+Guesswhat%3F%21+visual+object+discovery+through+multi%2Dmodal+dialogue%2E+In+Proceedings+of+the+IEEE)
8. Andrew Gargett, Konstantina Garoufi, Alexander Koller, and Kristina Striegnitz. 2010. The GIVE-2 corpus of giving instructions in virtual environments. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10), Valletta, Malta. European Language Resources… [find](https://scholar.google.com/scholar?q=Andrew+Gargett%2C+Konstantina+Garoufi%2C+Alexander+Koller%2C+and+Kristina+Striegnitz%2E+2010%2E+The+GIVE%2D2+corpus+of+giving+instructions+in+virtual+environments%2E+In+Proceedings+of+the+Seventh+International+Conf)
9. John J. Godfrey, Edward C. Holliman, and Jane Mc-Daniel. 1992. Switchboard: Telephone speech corpus for research and development. In IEEE International Conference on Acoustics, Speech, and Signal Processing. IEEE. [find](https://scholar.google.com/scholar?q=John+J%2E+Godfrey%2C+Edward+C%2E+Holliman%2C+and+Jane+Mc%2DDaniel%2E+1992%2E+Switchboard%3A+Telephone+speech+corpus+for+research+and+development%2E+In+IEEE+International+Conference+on+Acoustics%2C+Speech%2C+and+Signal+Proc)
10. Kristen Grauman, Andrew Westbury, et al. 2024. Ego-exo4d: Understanding skilled human activity from first- and third-person perspectives. [find](https://scholar.google.com/scholar?q=Kristen+Grauman%2C+Andrew+Westbury%2C+et+al%2E+2024%2E+Ego%2Dexo4d%3A+Understanding+skilled+human+activity+from+first%2D+and+third%2Dperson+perspectives%2E)
11. Markus Guhe and Alex Lascarides. 2014. Game strategies for the settlers of catan. In 2014 IEEE Conference on Computational Intelligence and Games, pages 1–8. IEEE. [find](https://scholar.google.com/scholar?q=Markus+Guhe+and+Alex+Lascarides%2E+2014%2E+Game+strategies+for+the+settlers+of+catan%2E+In+2014+IEEE+Conference+on+Computational+Intelligence+and+Games%2C+pages+1%E2%80%938%2E+IEEE%2E)
12. Janosch Haber, Tim Baumgärtner, Ece Takmaz, Lieke Gelderloos, Elia Bruni, and Raquel Fernández. 2019. The photobook dataset: Building common ground through visually-grounded dialogue. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1895–1910. [find](https://scholar.google.com/scholar?q=Janosch+Haber%2C+Tim+Baumg%C3%A4rtner%2C+Ece+Takmaz%2C+Lieke+Gelderloos%2C+Elia+Bruni%2C+and+Raquel+Fern%C3%A1ndez%2E+2019%2E+The+photobook+dataset%3A+Building+common+ground+through+visually%2Dgrounded+dialogue%2E+In+Proceedings+o) [our card](https://md.research.elamur.ai/free/en/papers/haber-2019-photobook-dataset-building-common-ground/)
13. Wiradee Imrattanatrai, Masaki Asada, Kimihiro Hasegawa, Zhi-Qi Cheng, Ken Fukuda, and Teruko Mitamura. 2025. A video-grounded dialogue dataset and metric for event-driven activities. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 24203– 24211. [find](https://scholar.google.com/scholar?q=Wiradee+Imrattanatrai%2C+Masaki+Asada%2C+Kimihiro+Hasegawa%2C+Zhi%2DQi+Cheng%2C+Ken+Fukuda%2C+and+Teruko+Mitamura%2E+2025%2E+A+video%2Dgrounded+dialogue+dataset+and+metric+for+event%2Ddriven+activities%2E+In+Proceedings+of)
14. Satwik Kottur, José MF Moura, Devi Parikh, Dhruv Batra, and Marcus Rohrbach. 2019. Clevr-dialog: A diagnostic dataset for multi-round reasoning in visual dialog. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techn… [find](https://scholar.google.com/scholar?q=Satwik+Kottur%2C+Jos%C3%A9+MF+Moura%2C+Devi+Parikh%2C+Dhruv+Batra%2C+and+Marcus+Rohrbach%2E+2019%2E+Clevr%2Ddialog%3A+A+diagnostic+dataset+for+multi%2Dround+reasoning+in+visual+dialog%2E+In+Proceedings+of+the+2019+Conference+)
15. Mike Lewis, Denis Yarats, Yann N Dauphin, Devi Parikh, and Dhruv Batra. 2017. Deal or no deal? end-to-end learning for negotiation dialogues. arXiv preprint arXiv:1706.05125. — [arXiv:1706.05125](https://arxiv.org/abs/1706.05125) [find](https://scholar.google.com/scholar?q=Mike+Lewis%2C+Denis+Yarats%2C+Yann+N+Dauphin%2C+Devi+Parikh%2C+and+Dhruv+Batra%2E+2017%2E+Deal+or+no+deal%3F+end%2Dto%2Dend+learning+for+negotiation+dialogues%2E+arXiv+preprint+arXiv%3A1706%2E05125%2E+%E2%80%94+%5BarXiv%3A1706%2E05125%5D%28http)
16. Hongpeng Lin, Ludan Ruan, Wenke Xia, Peiyu Liu, Jingyuan Wen, Yixin Xu, Di Hu, Ruihua Song, Wayne Xin Zhao, Qin Jin, et al. 2023. Tiktalk: A video-based dialogue dataset for multi-modal chitchat in real world. In Proceedings of the 31st ACM International Conference on Multimedia, pages 1303–1313. [find](https://scholar.google.com/scholar?q=Hongpeng+Lin%2C+Ludan+Ruan%2C+Wenke+Xia%2C+Peiyu+Liu%2C+Jingyuan+Wen%2C+Yixin+Xu%2C+Di+Hu%2C+Ruihua+Song%2C+Wayne+Xin+Zhao%2C+Qin+Jin%2C+et+al%2E+2023%2E+Tiktalk%3A+A+video%2Dbased+dialogue+dataset+for+multi%2Dmodal+chitchat+in+re)
17. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022. Robust speech recognition via large-scale weak supervision. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jong+Wook+Kim%2C+Tao+Xu%2C+Greg+Brockman%2C+Christine+McLeavey%2C+and+Ilya+Sutskever%2E+2022%2E+Robust+speech+recognition+via+large%2Dscale+weak+supervision%2E) [our card](https://md.research.elamur.ai/free/en/papers/radford-2022-robust-speech-recognition-large-scale/)
18. Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca A. Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational Linguistics, 26(3):339–373. [find](https://scholar.google.com/scholar?q=Andreas+Stolcke%2C+Klaus+Ries%2C+Noah+Coccaro%2C+Elizabeth+Shriberg%2C+Rebecca+A%2E+Bates%2C+Daniel+Jurafsky%2C+Paul+Taylor%2C+Rachel+Martin%2C+Carol+Van+Ess%2DDykema%2C+and+Marie+Meteer%2E+2000%2E+Dialogue+act+modeling+for+au) [our card](https://md.research.elamur.ai/free/en/papers/prag-2000-stolcke-dialogue-act-modeling-automatic-tagging/)
19. Carl Strathearn, Yanchao Yu, and Dimitra Gkatzia. 2023. Taskmaster: A novel cross-platform task-based spoken dialogue system for human-robot interaction. In Proceedings of the Workshop on Human-Robot Conversational Interaction (HRCI) at the ACM/IEEE International Conference on Human-Robot Interactio… [find](https://scholar.google.com/scholar?q=Carl+Strathearn%2C+Yanchao+Yu%2C+and+Dimitra+Gkatzia%2E+2023%2E+Taskmaster%3A+A+novel+cross%2Dplatform+task%2Dbased+spoken+dialogue+system+for+human%2Drobot+interaction%2E+In+Proceedings+of+the+Workshop+on+Human%2DRobot+)