---
title: "LEDA — paper by Karan et al. (2023) — summary and ideas"
description: "The authors build LEDA from public IETF mailing-list threads about four Internet-standard drafts, which an IETF expert… 24 references."
authors: ["Vanja Mladen Karan", "Prashant Khare", "Ravi Shekhar", "Stephen McQuistin", "Ignacio Castro", "Gareth Tyson", "Colin Perkins", "Patrick Healey", "Matthew Purver"]
year: 2023
venue: "Findings of the Association for Computational Linguistics: ACL 2023"
source: "https://aclanthology.org/2023.findings-acl.378/"
doi: "10.18653/v1/2023.findings-acl.378"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/karan-2023-leda-large-organization-email-based/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/karan-2023-leda-large-organization-email-based/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/14/) / LEDA: a Large-Organization Email-Based Decision-Dialogue-Act Analysis Dataset

# LEDA: a Large-Organization Email-Based Decision-Dialogue-Act Analysis Dataset

**Authors:** [Vanja Mladen Karan](https://md.research.elamur.ai/free/en/authors/vanja-mladen-karan/), Prashant Khare, [Ravi Shekhar](https://md.research.elamur.ai/free/en/authors/ravi-shekhar/), Stephen McQuistin, [Ignacio Castro](https://md.research.elamur.ai/free/en/authors/ignacio-castro/), Gareth Tyson, Colin Perkins, [Patrick Healey](https://md.research.elamur.ai/free/en/authors/patrick-g-t-healey/) and [Matthew Purver](https://md.research.elamur.ai/free/en/authors/matthew-purver/) · **Year:** 2023 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2023](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2023](https://md.research.elamur.ai/free/en/venues/findings/2023/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.findings-acl.378/) · [PDF](https://aclanthology.org/2023.findings-acl.378.pdf) · [DOI 10.18653/v1/2023.findings-acl.378](https://doi.org/10.18653/v1/2023.findings-acl.378) · [OpenAlex](https://openalex.org/W4385571057)

**Citations:** 1 citations (OpenAlex, 2026-10-14)

## Summary
Large, globally distributed organizations make important decisions through asynchronous online discussion, yet existing decision-oriented dialogue corpora come from small face-to-face meetings (AMI, ICSI) or from one closed company (Enron). The authors build LEDA from public IETF mailing-list threads about four Internet-standard drafts, which an IETF expert selected to cover different technical areas. Emails are split into authored, quoted and signature parts. The authored parts receive multi-label dialogue-act (DA) tags from a simplified taxonomy derived from ISO 24617-2. Their analysis tracks how DA use shifts over a draft's life-cycle and how draft authors, influential participants and WG chairs differ in communication style. For example, chairs' questions are answered most often, and roughly two thirds of all questions go unanswered. They also train a baseline tagger that encodes each segment with BERT and runs a BiLSTM over the segment sequence. The resource matters for studying transparency and accountability in standards bodies. It also adds an online, many-participant setting to dialogue-act research.

## Key points
- LEDA has 8230 segments from 2196 messages by 363 authors, covering complete threads for four IETF drafts drawn from an archive of more than 2M emails.
- The taxonomy has two levels: 3 coarse labels (InformationProviding, InformationSeeking, Social) split into 11 fine ones. It was reached after four calibration rounds that pruned and merged ISO 24617-2 labels.
- Seven linguistics students annotated the data, with at least two per thread. Agreement was measured as averaged pairwise F1 on 15 threads that every annotator labeled. All individual labels are released.
- Share of questions that receive an answer, by group: chairs 51%, all participants 37%, influential participants 34%, draft authors 22%.
- The baseline tagger (BERT [CLS] → BiLSTM → linear layer, binary cross-entropy) reaches macro F1 of about 0.59 with bert-base and about 0.52 with bert-base-ietf. Additional language-model training on IETF mail therefore did not help.

## Interesting ideas
- Instead of adopting a standard DA scheme as is, the authors annotated in rounds and after each one dropped labels that were irrelevant to email and merged labels annotators kept confusing, so the final set is one humans can apply reliably.
- Multi-label agreement was measured by treating one annotator as gold and the other as a classifier, then averaging F1 over all pairs, because Kappa and Alpha cannot handle multi-label data.
- The authors kept every annotator's labels rather than forcing a single adjudicated answer, treating some disagreement as inherent to subjective categories.
- Each email was segmented into the writer's own text, quoted text and signature, with links recorded from each quote to its source. This removes noise and preserves who actually said what.
- Reply trees were turned into root-to-leaf paths for sequence modelling. Segments that appear on several paths are resolved by majority vote across those paths.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (frames): The speech-act inventory in prag could be pruned and merged the same way, using confusion observed on real texts, so each act the system outputs is one it can actually justify.
- **I2** (learn): Agreement among witness parsers or LLM teacher tiers on multi-valued outputs (several roles or acts per span) could be scored with pairwise F1 to decide what counts as silver.
- **I3** (learn): Silver data could keep the full set of witness votes, so that ambiguous items act as soft targets or are kept out of strict evaluation sets.
- **I4** (frames): Detecting quoted spans and linking them to their source would let prag attribute speech acts and claims to the original speaker, not the person forwarding them.
- **I5** (world): Branching dialogues or threads could be linearized per branch to order events, with conflicting state updates on shared nodes settled by voting across branches.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Hong Kong University of Science and Technology](https://md.research.elamur.ai/free/en/universities/hong-kong-university-of-science-and-technology/) — University, Hong Kong, China · 142 works · 25,136 citations
- [University of Essex](https://md.research.elamur.ai/free/en/universities/university-of-essex/) — University, Colchester, United Kingdom · 34 works · 3,537 citations
- [Queen Mary University of London](https://md.research.elamur.ai/free/en/universities/queen-mary-university-of-london/) — University, London, United Kingdom · 54 works · 3,542 citations
- [Jožef Stefan Institute](https://md.research.elamur.ai/free/en/labs/jozef-stefan-institute/) — Research institute, Ljubljana, Slovenia · 17 works · 2,165 citations
- [Jožef Stefan International Postgraduate School](https://md.research.elamur.ai/free/en/universities/jozef-stefan-international-postgraduate-school/) — University, Ljubljana, Slovenia · 11 works · 1,776 citations
- [University of Glasgow](https://md.research.elamur.ai/free/en/universities/university-of-glasgow/) — University, Glasgow, United Kingdom · 11 works · 635 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [🗨️ Dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/) · 624 works · [💬 Speech act theory](https://md.research.elamur.ai/free/en/topics/speech-act-theory/) · 229 works

## Cite

Cite as: Vanja Mladen Karan, Prashant Khare, Ravi Shekhar, Stephen McQuistin, Ignacio Castro, Gareth Tyson, Colin Perkins, Patrick Healey and Matthew Purver. 2023. LEDA: a Large-Organization Email-Based Decision-Dialogue-Act Analysis Dataset. Findings of the Association for Computational Linguistics: ACL 2023.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{karan2023leda,
  title = {{LEDA: a Large-Organization Email-Based Decision-Dialogue-Act Analysis Dataset}},
  author = {Vanja Mladen Karan, Prashant Khare, Ravi Shekhar, Stephen McQuistin, Ignacio Castro, Gareth Tyson, Colin Perkins, Patrick Healey and Matthew Purver},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.378},
  url = {https://doi.org/10.18653/v1/2023.findings-acl.378}
}
```

- [← previous Language Models are Bounded Pragmatic Speakers: Understanding RLHF from a Bayesian Cognitive Modeling Perspective](https://md.research.elamur.ai/free/en/papers/nguyen-2023-language-models-bounded-pragmatic-speakers/)
- [next → Modeling Construction Grammar's Way into NLP: Insights from negative results in automatically identifying schematic clausal constructions in Brazilian Portuguese](https://md.research.elamur.ai/free/en/papers/lorenzi-2023-modeling-construction-grammar-s-way/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/14/)

## References

24 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. John Aberdeen and Lisa Ferro. 2003. Dialogue patterns and misunderstandings. Technical report, MITRE Corp. McLean VA. [find](https://scholar.google.com/scholar?q=John+Aberdeen+and+Lisa+Ferro%2E+2003%2E+Dialogue+patterns+and+misunderstandings%2E+Technical+report%2C+MITRE+Corp%2E+McLean+VA%2E)
2. Valerio Basile, Michael Fell, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, Massimo Poesio, and Alexandra Uma. 2021. [We need to consider disagreement in evaluation](https://md.research.elamur.ai/free/en/papers/basile-2021-we-need-consider-disagreement-evaluation/). In Proceedings of the For details see [https://www.ietf.org/about/](https://www.ietf.org/about/) note-well/ and the IETF privacy policy available at https://www.ietf.org/privacy-statement/.1st Workshop on Benchmarking: Past, Present and Future, pages 15–21, Online. Association for Computational Linguistics. [↗](https://www.ietf.org/about/)
3. Trung Bui and Stanley Peters. 2010. Decision detection using hierarchical graphical models. In Proceedings of the ACL 2010 Conference Short Papers, pages 307–312. [find](https://scholar.google.com/scholar?q=Trung+Bui+and+Stanley+Peters%2E+2010%2E+Decision+detection+using+hierarchical+graphical+models%2E+In+Proceedings+of+the+ACL+2010+Conference+Short+Papers%2C+pages+307%E2%80%93312%2E)
4. Harry Bunt, Jan Alexandersson, Jae-Woong Choe, Alex C Fang, Koiti Hasida, Volha Petukhova, Andrei Popescu-Belis, and David Traum. 2012. [Iso 24617-2: A semantically-based standard for dialogue annotation](https://md.research.elamur.ai/free/en/papers/bunt-2012-iso-24617-2-semantically-based/). Technical report, University of Southern California Los Angeles.
5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n19-1423)
6. Raquel Fernández, Matthew Frampton, Patrick Ehlen, Matthew Purver, and Stanley Peters. 2008. Modelling and detecting decisions in multi-party dialogue. In Proceedings of the 9th SIGdial Workshop on Discourse and Dialogue, pages 156–163. [find](https://scholar.google.com/scholar?q=Raquel+Fern%C3%A1ndez%2C+Matthew+Frampton%2C+Patrick+Ehlen%2C+Matthew+Purver%2C+and+Stanley+Peters%2E+2008%2E+Modelling+and+detecting+decisions+in+multi%2Dparty+dialogue%2E+In+Proceedings+of+the+9th+SIGdial+Workshop+on+Di)
7. H Flanagan. 2019. RFC 8700: Fifty Years of RFCs. [find](https://scholar.google.com/scholar?q=H+Flanagan%2E+2019%2E+RFC+8700%3A+Fifty+Years+of+RFCs%2E)
8. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735– 1780. [↗](https://doi.org/10.1162/neco.1997.9.8.1735)
9. Pei-Yun Hsueh and Johanna D Moore. 2007. What decisions have you made?: Automatic decision detection in meeting conversations. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference, pages 25–32. [find](https://scholar.google.com/scholar?q=Pei%2DYun+Hsueh+and+Johanna+D+Moore%2E+2007%2E+What+decisions+have+you+made%3F%3A+Automatic+decision+detection+in+meeting+conversations%2E+In+Human+Language+Technologies+2007%3A+The+Conference+of+the+North+American)
10. Adam Janin, Don Baron, Jane Edwards, Dan Ellis, David Gelbart, Nelson Morgan, Barbara Peskin, Thilo Pfau, Elizabeth Shriberg, Andreas Stolcke, et al. 2003. The icsi meeting corpus. In 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings.(ICASSP’03)., volume 1, pages I–I. IEEE. [↗](https://doi.org/10.7916/d83b68g8)
11. Prashant Khare, Mladen Karan, Stephen McQuistin, Colin Perkins, Gareth Tyson, Matthew Purver, Patrick Healey, and Ignacio Castro. 2022. The web we weave: Untangling the social graph of the IETF. In Proceedings of the 16th International AAAI Conference on Web and Social Media (ICWSM), pages 500–511, Palo Alto, CA, USA. AAAI Press. [find](https://scholar.google.com/scholar?q=Prashant+Khare%2C+Mladen+Karan%2C+Stephen+McQuistin%2C+Colin+Perkins%2C+Gareth+Tyson%2C+Matthew+Purver%2C+Patrick+Healey%2C+and+Ignacio+Castro%2E+2022%2E+The+web+we+weave%3A+Untangling+the+social+graph+of+the+IETF%2E+In+Pr)
12. Bryan Klimt and Yiming Yang. 2004. The enron corpus: A new dataset for email classification research. In European conference on machine learning, pages 217–226. Springer. [find](https://scholar.google.com/scholar?q=Bryan+Klimt+and+Yiming+Yang%2E+2004%2E+The+enron+corpus%3A+A+new+dataset+for+email+classification+research%2E+In+European+conference+on+machine+learning%2C+pages+217%E2%80%93226%2E+Springer%2E)
13. Elisa Leonardelli, Stefano Menini, Alessio Palmero Aprosio, Marco Guerini, and Sara Tonelli. 2021. [Agreeing to disagree: Annotating offensive language datasets with annotators’ disagreement](https://md.research.elamur.ai/free/en/papers/leonardelli-2021-agreeing-disagree-annotating-offensive-language/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10528–10539.
14. Ruizhe Li, Chenghua Lin, Matthew Collinson, Xiao Li, and Guanyi Chen. 2019. A dual-attention hierarchical recurrent neural network for dialogue act classification. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 383–392, Hong Kong, China. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1810.09154)
15. Iain McCowan, Jean Carletta, Wessel Kraaij, Simone Ashby, S Bourban, M Flynn, M Guillemot, Thomas Hain, J Kadlec, Vasilis Karaiskos, et al. 2005. The AMI meeting corpus. In Proceedings of the 5th International Conference on Methods and Techniques in Behavioral Research, volume 88, page 100. Citeseer. [find](https://scholar.google.com/scholar?q=Iain+McCowan%2C+Jean+Carletta%2C+Wessel+Kraaij%2C+Simone+Ashby%2C+S+Bourban%2C+M+Flynn%2C+M+Guillemot%2C+Thomas+Hain%2C+J+Kadlec%2C+Vasilis+Karaiskos%2C+et+al%2E+2005%2E+The+AMI+meeting+corpus%2E+In+Proceedings+of+the+5th+Inte)
16. Stephen McQuistin, Mladen Karan, Prashant Khare, Colin Perkins, Gareth Tyson, Matthew Purver, Patrick Healey, Waleed Iqbal, Junaid Qadir, and Ignacio Castro. 2021. Characterising the IETF through the lens of RFC deployment. In Proceedings of the 21st ACM Internet Measurement Conference, pages 137–149. [find](https://scholar.google.com/scholar?q=Stephen+McQuistin%2C+Mladen+Karan%2C+Prashant+Khare%2C+Colin+Perkins%2C+Gareth+Tyson%2C+Matthew+Purver%2C+Patrick+Healey%2C+Waleed+Iqbal%2C+Junaid+Qadir%2C+and+Ignacio+Castro%2E+2021%2E+Characterising+the+IETF+through+the+)
17. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te-jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. [PyTorch: An imperative style, high-performance deep learning library](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/). In H. Wallach, H. Larochelle, A. Beygelzimer, F. d’ Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc. [↗](https://doi.org/10.48550/arxiv.1912.01703)
18. Ellie Pavlick and Tom Kwiatkowski. 2019. [Inherent disagreements in human textual inferences](https://md.research.elamur.ai/free/en/papers/pavlick-2019-inherent-disagreements-human-textual-inferences/). Transactions of the Association for Computational Linguistics, 7:677–694. [↗](https://doi.org/10.1162/tacl_a_00293)
19. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830. [↗](https://doi.org/10.48550/arxiv.1201.0490)
20. Elizabeth Shriberg, Raj Dhillon, Sonali Bhagat, Jeremy Ang, and Hannah Carvey. 2004. [The icsi meeting recorder dialog act (mrda) corpus](https://md.research.elamur.ai/free/en/papers/shriberg-2004-icsi-meeting-recorder-dialog-act/). Technical report, International Computer Science Institute, Berkeley CA. [↗](https://doi.org/10.21236/ada460980)
21. Jennifer Stromer-Galley and Peter Muhlberger. 2009. Agreement and disagreement in group deliberation: Effects on deliberation satisfaction, future engagement, and decision legitimacy. Political communication, 26(2):173–192. [↗](https://doi.org/10.1080/10584600902850775)
22. Alexandra N Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2021. Learning from disagreement: A survey. Journal of Artificial Intelligence Research, 72:1385–1470. [↗](https://doi.org/10.1613/jair.1.12752)
23. Arkaitz Zubiaga, Elena Kochkina, Maria Liakata, Rob Procter, and Michal Lukasik. 2016. Stance classification in rumours as a sequential task exploiting the tree structure of social media conversations. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2438–2448, Osaka, Japan. [↗](https://doi.org/10.48550/arxiv.1609.09028)
24. The COL-ING 2016 Organizing Committee. [find](https://scholar.google.com/scholar?q=The+COL%2DING+2016+Organizing+Committee%2E)

*Updated 1 November 2026*