---
title: "LabelBuddy — paper by Prokopiou et al. (2026)"
description: "The authors present LabelBuddy, an open-source web tool for collaborative audio annotation with AI support. 23 references."
authors: ["Ioannis Prokopiou", "Ioannis Sina", "Agisilaos Kounelis", "Pantelis Vikatos", "Themos Stafylakis"]
year: 2026
venue: "Proceedings of the 4th Workshop on NLP for Music and Audio (NLP4MusA 2026)"
source: "https://aclanthology.org/2026.nlp4musa-1.2/"
doi: "10.18653/v1/2026.nlp4musa-1.2"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/prokopiou-2026-labelbuddy-open-source-music-audio/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/prokopiou-2026-labelbuddy-open-source-music-audio/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/34/) / LabelBuddy: An Open Source Music and Audio Language Annotation Tagging Tool Using AI Assistance

# LabelBuddy: An Open Source Music and Audio Language Annotation Tagging Tool Using AI Assistance

**Authors:** Ioannis Prokopiou, Ioannis Sina, Agisilaos Kounelis, Pantelis Vikatos and Themos Stafylakis · **Year:** 2026 · **Venue:** [Proceedings of the 4th Workshop on NLP for Music and Audio (NLP4MusA 2026)](https://md.research.elamur.ai/free/en/venues/nlp4musa/) ([Workshop on NLP for Music and Audio 2026](https://md.research.elamur.ai/free/en/venues/nlp4musa/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.nlp4musa-1.2/) · [PDF](https://aclanthology.org/2026.nlp4musa-1.2.pdf) · [DOI 10.18653/v1/2026.nlp4musa-1.2](https://doi.org/10.18653/v1/2026.nlp4musa-1.2) · [OpenAlex](https://openalex.org/W7140085853)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Building audio and music datasets that link sound with language is slow. Current practice also spreads segmentation, text handling and listening tests across separate tools, which makes human-in-the-loop (HITL) pipelines for Large Audio-Language Models (LALMs) hard to run. The authors present LabelBuddy, an open-source web tool for collaborative audio annotation with AI support. Its central design choice separates the interface (Django, wavesurfer.js waveform view) from model inference. Inference runs in Docker containers that are declared in YAML files and called through a Flask REST API. Models such as YOHO, musicnn, PANNs and Music Flamingo propose tags or captions, and annotators edit them, so the work shifts from writing labels to checking them. Approved labels can then be used to fine-tune the models. Role-based access for managers, annotators and reviewers supports a review and consensus stage. A case study shows how to build a music captioning dataset. The paper reports no measured results yet. Instead, it outlines a pilot evaluation and a roadmap toward RLHF-style preference collection. Its value is a pluggable, open workbench for language-grounded audio data.

## Key points
- Table 1 compares LabelBuddy with Audino, BAT, Aubio, Gecko, Prodigy and Label Studio (CE) on four features: audio-specific design, decoupled AI assistance, open-source licensing and collaborative consensus. LabelBuddy is the only tool marked as having all four.
- **Backend:** a relational database stores Projects, Users and Tasks.
  - Managers upload WAV/MP3 audio and assign it to annotators.
  - Annotations are saved as JSON with time boundaries and ontology tags.
  - Data can be exported as CSV, JSON or JSONL.
  - Inference containers can be deployed on remote cloud nodes such as AWS or Azure.
- **Case study:** a container wrapping Music Flamingo generates a candidate caption for each track.
  - The annotator corrects hallucinated details and adjusts timestamps.
  - Reviewers see where annotators disagree.
  - The final export consists of (audio_path, text_caption) pairs.
- **Planned pilot on DCASE 2024 data:**
  - time saved compared with labeling from scratch;
  - inter-annotator agreement measured with Fleiss' Kappa;
  - PSDS gains for SED baselines trained on data curated in LabelBuddy.
- **Roadmap:**
  - chain-of-thought assistance from models such as Qwen-Audio;
  - a pairwise preference interface aggregated with Bayesian Bradley–Terry (BBQ);
  - timestamp-required QA templates, motivated by the RUListening benchmark.
  - The code is on GitHub and originated in Google Summer of Code.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works · [🔊 Signal Processing](https://md.research.elamur.ai/free/en/shelves/computer-science/signal-processing/) · 452 works

🧭 Topics (1): [🎙️ Speech & spoken language](https://md.research.elamur.ai/free/en/topics/speech/) · 480 works

## Cite

Cite as: Ioannis Prokopiou, Ioannis Sina, Agisilaos Kounelis, Pantelis Vikatos and Themos Stafylakis. 2026. LabelBuddy: An Open Source Music and Audio Language Annotation Tagging Tool Using AI Assistance. Proceedings of the 4th Workshop on NLP for Music and Audio (NLP4MusA 2026).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{prokopiou2026labelbuddy,
  title = {{LabelBuddy: An Open Source Music and Audio Language Annotation Tagging Tool Using AI Assistance}},
  author = {Ioannis Prokopiou, Ioannis Sina, Agisilaos Kounelis, Pantelis Vikatos and Themos Stafylakis},
  booktitle = {Proceedings of the 4th Workshop on NLP for Music and Audio (NLP4MusA 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.nlp4musa-1.2},
  url = {https://doi.org/10.18653/v1/2026.nlp4musa-1.2}
}
```

- [← previous L3IA-Subtask 1 at AraSentEval Shared Task: Multi-Dialect Arabic Sentiment Classification via a Transformer-Based Approach](https://md.research.elamur.ai/free/en/papers/haouach-2026-l3ia-subtask-1-arasenteval-shared/)
- [next → LabelFusion: Fusing Large Language Models with Transformer Encoders for Robust Financial News Classification](https://md.research.elamur.ai/free/en/papers/schlee-2026-labelfusion-fusing-large-language-models/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/34/)

## References

23 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Juan Gutiérrez, Ángel Mora, Pablo Regodón, Silvia Rodriguez, and José Luis Blanco. 2025. Ai-boosted video annotation: Assessing the process enhancement. arXiv preprint arXiv:2510.21798. — [arXiv:2510.21798](https://arxiv.org/abs/2510.21798) [find](https://scholar.google.com/scholar?q=Juan+Guti%C3%A9rrez%2C+%C3%81ngel+Mora%2C+Pablo+Regod%C3%B3n%2C+Silvia+Rodriguez%2C+and+Jos%C3%A9+Luis+Blanco%2E+2025%2E+Ai%2Dboosted+video+annotation%3A+Assessing+the+process+enhancement%2E+arXiv+preprint+arXiv%3A2510%2E21798%2E+%E2%80%94+%5BarXiv%3A2510%2E)
2. Paul Brossier, Tintamar, Eduard Müller, Nils Philippsen, Tres Seaver, Hannes Fritz, cyclopsian, Sam Alexander, Jon Williams, James Cowgill, and Ancor Cruz. 2019. aubio/aubio: 0.4.9. [find](https://scholar.google.com/scholar?q=Paul+Brossier%2C+Tintamar%2C+Eduard+M%C3%BCller%2C+Nils+Philippsen%2C+Tres+Seaver%2C+Hannes+Fritz%2C+cyclopsian%2C+Sam+Alexander%2C+Jon+Williams%2C+James+Cowgill%2C+and+Ancor+Cruz%2E+2019%2E+aubio%2Faubio%3A+0%2E4%2E9%2E)
3. Faria Binte Kader and Santu Karmaker. 2025. A survey on evaluation metrics for music generation. arXiv preprint arXiv:2509.00051. — [arXiv:2509.00051](https://arxiv.org/abs/2509.00051) [find](https://scholar.google.com/scholar?q=Faria+Binte+Kader+and+Santu+Karmaker%2E+2025%2E+A+survey+on+evaluation+metrics+for+music+generation%2E+arXiv+preprint+arXiv%3A2509%2E00051%2E+%E2%80%94+%5BarXiv%3A2509%2E00051%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2509%2E00051%29)
4. Mark Cartwright, Ayanna Seals, Justin Salamon, Alex Williams, Stefanie Mikloska, Duncan MacConnell, Edith Law, Juan P Bello, and Oded Nov. 2017. Seeing sound: Investigating the effects of visualizations and complexity on crowdsourced audio annotations. Proceedings of the ACM on Human-Computer Intera… [find](https://scholar.google.com/scholar?q=Mark+Cartwright%2C+Ayanna+Seals%2C+Justin+Salamon%2C+Alex+Williams%2C+Stefanie+Mikloska%2C+Duncan+MacConnell%2C+Edith+Law%2C+Juan+P+Bello%2C+and+Oded+Nov%2E+2017%2E+Seeing+sound%3A+Investigating+the+effects+of+visualizatio)
5. Jin-Dong Kim and Yue Wang. 2012. Pubannotation-a persistent and sharable corpus and annotation repository. In BioNLP: Proceedings of the 2012 Workshop on Biomedical Natural Language Processing, pages 202–205. [find](https://scholar.google.com/scholar?q=Jin%2DDong+Kim+and+Yue+Wang%2E+2012%2E+Pubannotation%2Da+persistent+and+sharable+corpus+and+annotation+repository%2E+In+BioNLP%3A+Proceedings+of+the+2012+Workshop+on+Biomedical+Natural+Language+Processing%2C+pages+)
6. Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. 2023. Qwen-audio: Advancing universal Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D Plumbley. 2020. Panns: Large-scale pretrained audio neural networks for audio pattern… [find](https://scholar.google.com/scholar?q=Yunfei+Chu%2C+Jin+Xu%2C+Xiaohuan+Zhou%2C+Qian+Yang%2C+Shiliang+Zhang%2C+Zhijie+Yan%2C+Chang+Zhou%2C+and+Jingren+Zhou%2E+2023%2E+Qwen%2Daudio%3A+Advancing+universal+Qiuqiang+Kong%2C+Yin+Cao%2C+Turab+Iqbal%2C+Yuxuan+Wang%2C+Wenwu+Wa)
7. Pontus Stenetorp, Sampo Pyysalo, Goran Topić, Tomoko Ohta, Sophia Ananiadou, and Jun’ichi Tsujii. 2012. [Brat: a web-based tool for nlp-assisted text annotation](https://md.research.elamur.ai/free/en/papers/stenetorp-2012-brat-web-based-tool-nlp/). In Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics, page…
8. Golan Levy, Raquel Sitman, Ido Amir, Eduard Gol-shtein, Ran Mochary, Eilon Reshef, Roi Reichart, and Omri Allouche. 2019. Gecko-a tool for effective annotation of human conversations. In INTERSPEECH, pages 3677–3678. [find](https://scholar.google.com/scholar?q=Golan+Levy%2C+Raquel+Sitman%2C+Ido+Amir%2C+Eduard+Gol%2Dshtein%2C+Ran+Mochary%2C+Eilon+Reshef%2C+Roi+Reichart%2C+and+Omri+Allouche%2E+2019%2E+Gecko%2Da+tool+for+effective+annotation+of+human+conversations%2E+In+INTERSPEECH%2C+)
9. Maxim Tkachenko, Mikhail Malyuk, Andrey Holmanyuk, and Nikolai Liubimov. 2020-2022. Label Studio: Data labeling software. Open source software available from [https://github.com/heartexlabs/label-studio](https://github.com/heartexlabs/label-studio). [↗](https://github.com/heartexlabs/label-studio)
10. Cheng Liu, Hui Wang, Jinghua Zhao, Shiwan Zhao, Hui Bu, Xin Xu, Jiaming Zhou, Haoqin Sun, and Yong Qin. 2025. Musiceval: A generative music dataset with expert ratings for automatic text-to-music evaluation. arXiv preprint arXiv:2501.10811. — [arXiv:2501.10811](https://arxiv.org/abs/2501.10811) [find](https://scholar.google.com/scholar?q=Cheng+Liu%2C+Hui+Wang%2C+Jinghua+Zhao%2C+Shiwan+Zhao%2C+Hui+Bu%2C+Xin+Xu%2C+Jiaming+Zhou%2C+Haoqin+Sun%2C+and+Yong+Qin%2E+2025%2E+Musiceval%3A+A+generative+music+dataset+with+expert+ratings+for+automatic+text%2Dto%2Dmusic+eval)
11. Satvik Venkatesh, David Moffat, and Eduardo Reck Miranda. 2022. You only hear once: a yolo-like algorithm for audio segmentation and sound event detection. Applied Sciences, 12(7):3293. [find](https://scholar.google.com/scholar?q=Satvik+Venkatesh%2C+David+Moffat%2C+and+Eduardo+Reck+Miranda%2E+2022%2E+You+only+hear+once%3A+a+yolo%2Dlike+algorithm+for+audio+segmentation+and+sound+event+detection%2E+Applied+Sciences%2C+12%287%29%3A3293%2E)
12. Blai Meléndez Catalán, Emilio Molina, and Emilia Gómez Gutiérrez. 2017. Bat: An open-source, web-based audio events annotation tool. [find](https://scholar.google.com/scholar?q=Blai+Mel%C3%A9ndez+Catal%C3%A1n%2C+Emilio+Molina%2C+and+Emilia+G%C3%B3mez+Guti%C3%A9rrez%2E+2017%2E+Bat%3A+An+open%2Dsource%2C+web%2Dbased+audio+events+annotation+tool%2E)
13. Paul Voigtlaender, Lishu Luo, Chun Yuan, Yong Jiang, and Bastian Leibe. 2021. Reducing the annotation effort for video object segmentation datasets. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3060–3069. [find](https://scholar.google.com/scholar?q=Paul+Voigtlaender%2C+Lishu+Luo%2C+Chun+Yuan%2C+Yong+Jiang%2C+and+Bastian+Leibe%2E+2021%2E+Reducing+the+annotation+effort+for+video+object+segmentation+datasets%2E+In+Proceedings+of+the+IEEE%2FCVF+Winter+Conference+on)
14. Christian M Meyer, Darina Benikova, Margot Mieskes, and Iryna Gurevych. 2016. Mdswriter: annotation tool for creating high-quality multi-document summarization corpora. In Proceedings of ACL-2016 System Demonstrations, pages 97–102. [find](https://scholar.google.com/scholar?q=Christian+M+Meyer%2C+Darina+Benikova%2C+Margot+Mieskes%2C+and+Iryna+Gurevych%2E+2016%2E+Mdswriter%3A+annotation+tool+for+creating+high%2Dquality+multi%2Ddocument+summarization+corpora%2E+In+Proceedings+of+ACL%2D2016+Syst)
15. Max Wiechmann, Seid Muhie Yimam, and Chris Biemann. 2021. Activeanno: General-purpose document-level annotation tool with active learning integration. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: De… [find](https://scholar.google.com/scholar?q=Max+Wiechmann%2C+Seid+Muhie+Yimam%2C+and+Chris+Biemann%2E+2021%2E+Activeanno%3A+General%2Dpurpose+document%2Dlevel+annotation+tool+with+active+learning+integration%2E+In+Proceedings+of+the+2021+Conference+of+the+Nort)
16. Ines Montani and Matthew Honnibal. 2018. Prodigy: A new annotation tool for radically efficient machine teaching. Artificial Intelligence to appear. [find](https://scholar.google.com/scholar?q=Ines+Montani+and+Matthew+Honnibal%2E+2018%2E+Prodigy%3A+A+new+annotation+tool+for+radically+efficient+machine+teaching%2E+Artificial+Intelligence+to+appear%2E)
17. Minh-Quoc Nghiem and Sophia Ananiadou. 2018. Aplenty: annotation tool for creating high-quality datasets using active and proactive learning. In EMNLP (Demonstration), pages 108–113. [find](https://scholar.google.com/scholar?q=Minh%2DQuoc+Nghiem+and+Sophia+Ananiadou%2E+2018%2E+Aplenty%3A+annotation+tool+for+creating+high%2Dquality+datasets+using+active+and+proactive+learning%2E+In+EMNLP+%28Demonstration%29%2C+pages+108%E2%80%93113%2E)
18. Yongyi Zang, Sean O’Brien, Taylor Berg-Kirkpatrick, Julian McAuley, and Zachary Novack. 2025. Are you really listening? boosting perceptual awareness in music-qa benchmarks. arXiv preprint arXiv:2504.00369. — [arXiv:2504.00369](https://arxiv.org/abs/2504.00369) [find](https://scholar.google.com/scholar?q=Yongyi+Zang%2C+Sean+O%E2%80%99Brien%2C+Taylor+Berg%2DKirkpatrick%2C+Julian+McAuley%2C+and+Zachary+Novack%2E+2025%2E+Are+you+really+listening%3F+boosting+perceptual+awareness+in+music%2Dqa+benchmarks%2E+arXiv+preprint+arXiv%3A2504%2E)
19. Minh-Quoc Nghiem, Paul Baylis, and Sophia Ananiadou. 2021. Paladin: an annotation tool based on active and proactive learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 238–243. [find](https://scholar.google.com/scholar?q=Minh%2DQuoc+Nghiem%2C+Paul+Baylis%2C+and+Sophia+Ananiadou%2E+2021%2E+Paladin%3A+an+annotation+tool+based+on+active+and+proactive+learning%2E+In+Proceedings+of+the+16th+Conference+of+the+European+Chapter+of+the+Asso)
20. Sylvaine Picard, Camille Chapdelaine, Cyril Cappi, Laurent Gardes, Eric Jenn, Baptiste Lefevre, and Thomas Soumarmon. 2020. Ensuring dataset quality for machine learning certification. In 2020 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW), pages 275–282. IEEE. [find](https://scholar.google.com/scholar?q=Sylvaine+Picard%2C+Camille+Chapdelaine%2C+Cyril+Cappi%2C+Laurent+Gardes%2C+Eric+Jenn%2C+Baptiste+Lefevre%2C+and+Thomas+Soumarmon%2E+2020%2E+Ensuring+dataset+quality+for+machine+learning+certification%2E+In+2020+IEEE+In)
21. Jordi Pons and Xavier Serra. 2019. musicnn: Pre-trained convolutional neural networks for music audio tagging. arXiv preprint arXiv:1909.06654. — [arXiv:1909.06654](https://arxiv.org/abs/1909.06654) [find](https://scholar.google.com/scholar?q=Jordi+Pons+and+Xavier+Serra%2E+2019%2E+musicnn%3A+Pre%2Dtrained+convolutional+neural+networks+for+music+audio+tagging%2E+arXiv+preprint+arXiv%3A1909%2E06654%2E+%E2%80%94+%5BarXiv%3A1909%2E06654%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1909%2E06654%29)
22. Michael Schoeffler, Sarah Bartoschek, Fabian-Robert Stöter, Marlene Roess, Susanne Westphal, Bernd Edler, and Jürgen Herre. 2018. webmushra—a comprehensive framework for web-based listening tests. Journal of Open Research Software, 6(1). [find](https://scholar.google.com/scholar?q=Michael+Schoeffler%2C+Sarah+Bartoschek%2C+Fabian%2DRobert+St%C3%B6ter%2C+Marlene+Roess%2C+Susanne+Westphal%2C+Bernd+Edler%2C+and+J%C3%BCrgen+Herre%2E+2018%2E+webmushra%E2%80%94a+comprehensive+framework+for+web%2Dbased+listening+tests%2E+Jou)
23. Erik Smistad, Andreas Østvik, and Lasse Løvstakken. 2021. Annotation web-an open-source web-based annotation tool for ultrasound images. In 2021 IEEE International Ultrasonics Symposium (IUS), pages 1–4. IEEE. [find](https://scholar.google.com/scholar?q=Erik+Smistad%2C+Andreas+%C3%98stvik%2C+and+Lasse+L%C3%B8vstakken%2E+2021%2E+Annotation+web%2Dan+open%2Dsource+web%2Dbased+annotation+tool+for+ultrasound+images%2E+In+2021+IEEE+International+Ultrasonics+Symposium+%28IUS%29%2C+pages+1)

*Updated 1 November 2026*