---
title: "Multi-Source Emotion Annotation in Children's Language: When"
description: "The authors conclude that high inter-LLM agreement cannot replace human validation for affect labelling. 26 references."
authors: ["Farida Said", "Jeanne Villaneau"]
year: 2026
venue: "Proceedings of Computational Affective Science (CAS) @ LREC 2026"
source: "https://aclanthology.org/2026.cas-1.11/"
doi: "10.63317/39tk69v6ca4p"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/said-2026-multi-source-emotion-annotation-children/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/said-2026-multi-source-emotion-annotation-children/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/42/) / Multi-Source Emotion Annotation in Children's Language: When LLM Consensus Diverges from Human Judgment

# Multi-Source Emotion Annotation in Children's Language: When LLM Consensus Diverges from Human Judgment

**Authors:** Farida Said and [Jeanne Villaneau](https://md.research.elamur.ai/free/en/authors/jeanne-villaneau/) · **Year:** 2026 · **Venue:** [Proceedings of Computational Affective Science (CAS) @ LREC 2026](https://md.research.elamur.ai/free/en/venues/cas/) ([Computational Affective Science (CAS) @ LREC 2026](https://md.research.elamur.ai/free/en/venues/cas/2026/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.cas-1.11/) · [PDF](https://aclanthology.org/2026.cas-1.11.pdf) · [DOI 10.63317/39tk69v6ca4p](https://doi.org/10.63317/39tk69v6ca4p) · [OpenAlex](https://openalex.org/W7166560225)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
The paper asks whether agreement among several LLM annotators reliably signals label quality for emotion annotation. The test case is children's speech, where feelings are usually implied rather than named. The authors use 2,106 clause-level segments from interviews with 184 French-speaking children (ages 6–11), who described the concepts FATHER and MOTHER. Claude Sonnet 4 and GPT-5.2 each labelled valence and intensity under three zero-shot prompt postures, giving six pseudo-annotators. Their outputs were combined with Majority Vote, Dawid–Skene and MACE. FEEL-based lexicon baselines with Stanza lemmatization were also evaluated. Three experts annotated a stratified 500-segment gold standard, which serves as the external reference.

The internal reliability estimates from Dawid–Skene turned out to be badly miscalibrated. Both models tend to over-assign neutral labels, and this shared error looks like consensus to the aggregator. The gold standard reveals the problem and even reverses which model looks better. The authors conclude that high inter-LLM agreement cannot replace human validation for affect labelling. They release the gold subset and the probabilistic corpus.

## Key points
- Dawid–Skene credits GPT-5.2 with 90.7% valence accuracy, but the gold standard gives 71.0%, an overestimate of 19.7 points. Claude Sonnet 4 is underestimated by 6.8 points, which flips the model ranking.
- The three aggregation methods produce nearly the same labels: 98.6% three-way agreement (493/500). Their gold accuracies are also nearly identical (MV 69.2%, DS 69.0%, MACE 69.2%; κ ≈ 0.475). This points to annotator bias, not the aggregation rule, as the source of error.
- A conservative probe used a 61-entry simplified FEEL lexicon with Stanza lemmatization. It finds no overt affect words in about 76% of segments.
- The gold labels are 68.4% positive and 26.0% neutral. The LLM-predicted distribution was 53.7% positive and 40.8% neutral, consistent with systematic over-neutralization.
- Expert inter-annotator agreement was κ = 0.682–0.731. The final gold matches a Dawid–Skene aggregation of the human labels alone (κ = 0.983–0.991). The source data is the COS-MOS corpus (CC-BY-4.0).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Institut de Recherche en Informatique et Systèmes Aléatoires](https://md.research.elamur.ai/free/en/labs/institut-de-recherche-en-informatique-et-systemes-aleatoires/) — Research institute, Rennes, France · 7 works · 1,944 citations
- [Université de Bretagne Sud](https://md.research.elamur.ai/free/en/universities/universite-de-bretagne-sud/) — University, Lorient, France · 2 works · 35 citations
- [Laboratoire de Mathématiques de Bretagne Atlantique](https://md.research.elamur.ai/free/en/labs/laboratoire-de-mathematiques-de-bretagne-atlantique/) — Research institute, Brest, France · 2 works

📚 Shelves (8): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,954 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 1,649 works · [💭 Experimental and Cognitive Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/experimental-and-cognitive-psychology/) · 930 works · [🤝 General Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/general-social-sciences/) · 254 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [😊 Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) · 1,035 works

## Cite

Cite as: Farida Said and Jeanne Villaneau. 2026. Multi-Source Emotion Annotation in Children's Language: When LLM Consensus Diverges from Human Judgment. Proceedings of Computational Affective Science (CAS) @ LREC 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{said2026multi,
  title = {{Multi-Source Emotion Annotation in Children's Language: When LLM Consensus Diverges from Human Judgment}},
  author = {Farida Said and Jeanne Villaneau},
  booktitle = {Proceedings of Computational Affective Science (CAS) @ LREC 2026},
  year = {2026},
  doi = {10.63317/39tk69v6ca4p},
  url = {https://doi.org/10.63317/39tk69v6ca4p}
}
```

- [← previous Multi-lingual Functional Evaluation for Large Language Models](https://md.research.elamur.ai/free/en/papers/ojewale-2026-multi-lingual-functional-evaluation-large/)
- [next → Multi-Task Instruction Tuning via Data Scheduling for Low-Resource Arabic SpeechLLMs](https://md.research.elamur.ai/free/en/papers/bhatti-2026-multi-task-instruction-tuning-data/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/42/)

## References

26 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ron Artstein and Massimo Poesio. Inter-coder agreement for computational linguistics. Computational Linguistics, 34:555–596, 2008. [find](https://scholar.google.com/scholar?q=Ron+Artstein+and+Massimo+Poesio%2E+Inter%2Dcoder+agreement+for+computational+linguistics%2E+Computational+Linguistics%2C+34%3A555%E2%80%93596%2C+2008%2E)
2. Jacob Cohen. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46, 1960. [find](https://scholar.google.com/scholar?q=Jacob+Cohen%2E+A+coefficient+of+agreement+for+nominal+scales%2E+Educational+and+Psychological+Measurement%2C+20%281%29%3A37%E2%80%9346%2C+1960%2E)
3. A. Philip Dawid and Allan M. Skene. Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society: Series C, 28:20–28, 1979. [find](https://scholar.google.com/scholar?q=A%2E+Philip+Dawid+and+Allan+M%2E+Skene%2E+Maximum+likelihood+estimation+of+observer+error%2Drates+using+the+EM+algorithm%2E+Journal+of+the+Royal+Statistical+Society%3A+Series+C%2C+28%3A20%E2%80%9328%2C+1979%2E)
4. Joseph L. Fleiss. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76:378–382, 1971. [find](https://scholar.google.com/scholar?q=Joseph+L%2E+Fleiss%2E+Measuring+nominal+scale+agreement+among+many+raters%2E+Psychological+Bulletin%2C+76%3A378%E2%80%93382%2C+1971%2E)
5. Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. [ChatGPT outperforms crowd-workers for text-annotation tasks](https://md.research.elamur.ai/free/en/papers/gilardi-2023-chatgpt-outperforms-crowd-workers-text/). arXiv:2303.15056, 2023.
6. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning (ICML 2017), pages 1321–1330. [find](https://scholar.google.com/scholar?q=Chuan+Guo%2C+Geoff+Pleiss%2C+Yu+Sun%2C+and+Kilian+Q%2E+Weinberger%2E+On+calibration+of+modern+neural+networks%2E+In+Proceedings+of+the+34th+International+Conference+on+Machine+Learning+%28ICML+2017%29%2C+pages+1321%E2%80%93133)
7. Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. [Learning whom to trust with MACE](https://md.research.elamur.ai/free/en/papers/hovy-2013-learning-whom-trust-mace/). In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1120– 1130, Atlanta, Georgia, 2013. Associat…
8. Saurav Kadavath et al. Language models (mostly) know what they don’t know. arXiv preprint arXiv:2207.05221, 2022. [find](https://scholar.google.com/scholar?q=Saurav+Kadavath+et+al%2E+Language+models+%28mostly%29+know+what+they+don%E2%80%99t+know%2E+arXiv+preprint+arXiv%3A2207%2E05221%2C+2022%2E)
9. Klaus Krippendorff. Computing Krippendorff’s alpha-reliability. Departmental Papers (ASC), Annenberg School for Communication, University of Pennsylvania, 2011. [find](https://scholar.google.com/scholar?q=Klaus+Krippendorff%2E+Computing+Krippendorff%E2%80%99s+alpha%2Dreliability%2E+Departmental+Papers+%28ASC%29%2C+Annenberg+School+for+Communication%2C+University+of+Pennsylvania%2C+2011%2E)
10. Chao Liu and Charis Chen. Text mining and sentiment analysis: A new lens to explore the emotion dynamics of mother–child interactions. Social Development, 33, 2024. https://doi.org/10.1 111/sode.12733. [find](https://scholar.google.com/scholar?q=Chao+Liu+and+Charis+Chen%2E+Text+mining+and+sentiment+analysis%3A+A+new+lens+to+explore+the+emotion+dynamics+of+mother%E2%80%93child+interactions%2E+Social+Development%2C+33%2C+2024%2E+https%3A%2F%2Fdoi%2Eorg%2F10%2E1+111%2Fsode%2E12733)
11. Minxue Niu, Mimansa Jaiswal, and Emily Mower Provost. From text to emotion: Unveiling the emotion annotation capabilities of LLMs. In Proceedings of Interspeech 2024, pages 2650–2654, 2024. https://doi.org/10.2 1437/Interspeech.2024-2282. [find](https://scholar.google.com/scholar?q=Minxue+Niu%2C+Mimansa+Jaiswal%2C+and+Emily+Mower+Provost%2E+From+text+to+emotion%3A+Unveiling+the+emotion+annotation+capabilities+of+LLMs%2E+In+Proceedings+of+Interspeech+2024%2C+pages+2650%E2%80%932654%2C+2024%2E+https%3A%2F%2Fdo)
12. Minxue Niu, Yara El-Tawil, Amrit Romana, and Emily Mower Provost. Rethinking emotion annotations in the era of large language models. IEEE Transactions on Affective Computing, 16 (04):2668–2679, 2025. [find](https://scholar.google.com/scholar?q=Minxue+Niu%2C+Yara+El%2DTawil%2C+Amrit+Romana%2C+and+Emily+Mower+Provost%2E+Rethinking+emotion+annotations+in+the+era+of+large+language+models%2E+IEEE+Transactions+on+Affective+Computing%2C+16+%2804%29%3A2668%E2%80%932679%2C+2025%2E)
13. Marissa Ogren and Catherine M. Sandhofer. Emotion words in early childhood: A language transcript analysis. Cognitive Development, 60: 101122, 2021. https://doi.org/10.1016/ j.cogdev.2021.101122. [find](https://scholar.google.com/scholar?q=Marissa+Ogren+and+Catherine+M%2E+Sandhofer%2E+Emotion+words+in+early+childhood%3A+A+language+transcript+analysis%2E+Cognitive+Development%2C+60%3A+101122%2C+2021%2E+https%3A%2F%2Fdoi%2Eorg%2F10%2E1016%2F+j%2Ecogdev%2E2021%2E101122%2E)
14. Nicholas Pangakis, Samuel Wolken, and Neil Fasching. [Automated annotation with generative AI requires validation](https://md.research.elamur.ai/free/en/papers/pangakis-2023-automated-annotation-generative-ai-requires/). arXiv preprint arXiv:2306.00176, 2023. https://doi.org/ 10.48550/arXiv.2306.00176.
15. Silviu Paun, Bob Carpenter, Jon Chamberlain, Dirk Hovy, Udo Kruschwitz, and Massimo Poesio. [Comparing bayesian models of annotation](https://md.research.elamur.ai/free/en/papers/paun-2018-comparing-bayesian-models-annotation/). Transactions of the Association for Computational Linguistics, 6:571–585, 2018.
16. Barbara Plank. The “problem” of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 [Conference on Empirical Methods in Natural Language Processing, pages 10671–10682, 2022](https://md.research.elamur.ai/free/en/papers/plank-2022-problem-human-label-variation-ground/). https://doi.org/10.18653/v1/2022.emnlp-m ain.731. [↗](https://doi.org/10.18653/v1/2022.emnlp-m)
17. Peng Qi et al. Stanza: A python NLP toolkit for many human languages. In Proceedings of ACL 2020: System Demonstrations, pages 101–108, 2020. [find](https://scholar.google.com/scholar?q=Peng+Qi+et+al%2E+Stanza%3A+A+python+NLP+toolkit+for+many+human+languages%2E+In+Proceedings+of+ACL+2020%3A+System+Demonstrations%2C+pages+101%E2%80%93108%2C+2020%2E)
18. Kenji Sagae, Eric Davis, Alon Lavie, Brian MacWhinney, and Shuly Wintner. Morphosyntactic annotation of CHILDES transcripts. Journal of Child Language, 37(3):705–729, 2010. [find](https://scholar.google.com/scholar?q=Kenji+Sagae%2C+Eric+Davis%2C+Alon+Lavie%2C+Brian+MacWhinney%2C+and+Shuly+Wintner%2E+Morphosyntactic+annotation+of+CHILDES+transcripts%2E+Journal+of+Child+Language%2C+37%283%29%3A705%E2%80%93729%2C+2010%2E)
19. Woosuk Seo, Chanmo Yang, and Young-Ho Kim. Chacha: Leveraging large language models to prompt children to share their emotions about personal events. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pages 1–20. ACM, 2024. CHI ’24. [find](https://scholar.google.com/scholar?q=Woosuk+Seo%2C+Chanmo+Yang%2C+and+Young%2DHo+Kim%2E+Chacha%3A+Leveraging+large+language+models+to+prompt+children+to+share+their+emotions+about+personal+events%2E+In+Proceedings+of+the+CHI+Conference+on+Human+Fact)
20. Carlo Strapparava and Rada Mihalcea. [Semeval-2007 task 14: Affective text](https://md.research.elamur.ai/free/en/papers/strapparava-2007-semeval-2007-task-14-affective/). In Proceedings of SemEval-2007, pages 70–74, 2007. spans. If uncertain, use confidence in [0,1] accordingly and keep evidence_span minimal but valid. Do not add any extra keys.
21. Petter Törnberg. ChatGPT-4 outperforms experts and crowd workers in annotating political Twitter messages. arXiv:2304.[06588, 2023](https://md.research.elamur.ai/free/en/papers/tornberg-2023-chatgpt-4-outperforms-experts-crowd/).
22. Posture variants (appended to the base prompt). P1 (conservative): “Annotate carefully and conservatively. Prefer fewer labels if ambiguous.” P2 (implicit): “Annotate based on the child’s expressed feeling/attitude in the text, even if implicit.” P3 (lexical): “Annotate with emphasis on explicit lex… — [doi:10.1613/jair.1](https://doi.org/10.1613/jair.1) [find](https://scholar.google.com/scholar?q=Posture+variants+%28appended+to+the+base+prompt%29%2E+P1+%28conservative%29%3A+%E2%80%9CAnnotate+carefully+and+conservatively%2E+Prefer+fewer+labels+if+ambiguous%2E%E2%80%9D+P2+%28implicit%29%3A+%E2%80%9CAnnotate+based+on+the+child%E2%80%99s+expressed+fe)
23. Language Resource References User message (per segment). { "segment_id": \\<int\>, "child_id": \\<int\|null\>, "concept": "father\|mother", "question": "\\<question_type\>", "segment_text": "\\<exact child , → utterance\>" Amine Abdaoui, Jérôme Azé, Sandra Bringay, and Pascal Poncelet. FEEL: a french expanded emo… [find](https://scholar.google.com/scholar?q=Language+Resource+References+User+message+%28per+segment%29%2E+%7B+%22segment%5Fid%22%3A+%5C%3Cint%5C%3E%2C+%22child%5Fid%22%3A+%5C%3Cint%5C%7Cnull%5C%3E%2C+%22concept%22%3A+%22father%5C%7Cmother%22%2C+%22question%22%3A+%22%5C%3Cquestion%5Ftype%5C%3E%22%2C+%22segment%5Ftext%22%3A+%22%5C%3Cexact+chil)
24. Abdelhadi Bellachhab, Olga Galatanu, Frédéric Pugniere-Saavedra, Valérie Rochaix, Farida Saïd, and Jeanne Villaneau. Projet COS-MOS : Transcriptions anonymisées d’interviews d’enfants sur la construction sémantique et les stéréotypes. HAL: hal-04893220, 2025. [find](https://scholar.google.com/scholar?q=Abdelhadi+Bellachhab%2C+Olga+Galatanu%2C+Fr%C3%A9d%C3%A9ric+Pugniere%2DSaavedra%2C+Val%C3%A9rie+Rochaix%2C+Farida+Sa%C3%AFd%2C+and+Jeanne+Villaneau%2E+Projet+COS%2DMOS+%3A+Transcriptions+anonymis%C3%A9es+d%E2%80%99interviews+d%E2%80%99enfants+sur+la+construct)
25. Humberto Pérez-Espinosa, Juan Martínez-Miranda, Ismael Espinosa-Curiel, Josefina Rodríguez-Jacobo, Luis Villaseñor-Pineda, and Himer Avila-George. IESC-Child: An interactive emotional children’s speech corpus. Computer Speech & Language, 59:55–74, 2020. https://doi.org/10.1016/j.csl.2019.06.006. — [doi:10.1016/j.csl.2019.06.006](https://doi.org/10.1016/j.csl.2019.06.006) [↗](https://doi.org/10.1016/j.csl.2019.06.006)
26. Arielle Syssau, Adil Yakhloufi, Edouard Giudicelli, Catherine Monnier, and Royce Anders. FAN-Cat: French affective norms for ten emotional categories. Behavior Research Methods, 53:447– 465, 2021. [find](https://scholar.google.com/scholar?q=Arielle+Syssau%2C+Adil+Yakhloufi%2C+Edouard+Giudicelli%2C+Catherine+Monnier%2C+and+Royce+Anders%2E+FAN%2DCat%3A+French+affective+norms+for+ten+emotional+categories%2E+Behavior+Research+Methods%2C+53%3A447%E2%80%93+465%2C+2021%2E)

*Updated 2 November 2026*