--- title: "Hierarchical Fusion for Online Multimodal Dialog — Miah 2023" description: "This paper tackles online dialog act (DA) classification." authors: ["Md Messal Monem Miah", "Adarsh Pyarelal", "Ruihong Huang"] year: 2023 venue: "Findings of the Association for Computational Linguistics: EMNLP 2023" source: "https://aclanthology.org/2023.findings-emnlp.505/" doi: "10.18653/v1/2023.findings-emnlp.505" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/papers/miah-2023-hierarchical-fusion-online-multimodal-dialog/" --- [Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/miah-2023-hierarchical-fusion-online-multimodal-dialog/) [home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2023](https://md.research.elamur.ai/free/papers/2023/page/3/) / Hierarchical Fusion for Online Multimodal Dialog Act Classification # Hierarchical Fusion for Online Multimodal Dialog Act Classification **Authors:** Md Messal Monem Miah, Adarsh Pyarelal, [Ruihong Huang](https://md.research.elamur.ai/free/authors/ruihong-huang/) · **Year:** 2023 · **Venue:** Findings of the Association for Computational Linguistics: EMNLP 2023 · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2023.findings-emnlp.505/) · [PDF](https://aclanthology.org/2023.findings-emnlp.505.pdf) · [DOI 10.18653/v1/2023.findings-emnlp.505](https://doi.org/10.18653/v1/2023.findings-emnlp.505) · [OpenAlex](https://openalex.org/W4389520130) **Citations:** 2 citations (openalex-snapshot · 2026-10-14) · [5](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary This paper tackles online dialog act (DA) classification. Here no human (oracle) transcripts are available, and a system must work from raw speech plus whatever an ASR system produces. Earlier multimodal DA approaches used audio poorly and merged modalities only at a late stage. The authors transcribe speech with Whisper and take audio frame features from Whisper's last encoder layer. Text token features come from a frozen RoBERTa. Their Hierarchical Fusion (HF) model encodes each modality with a Bi-LSTM and self-attention pooling. It joins text and audio at the utterance level before attention-based dialog-context modeling. Separate unimodal context encoders add further information, and a late-fusion classifier merges all branches. On MRDA and EMOTyDA, the model beats prior state-of-the-art approaches by about 3 F1 points. It also beats a text-only baseline by roughly 4.5–5.5 F1 points. The work shows that fusing modalities early, with strong pretrained audio features, makes DA recognition more robust to noisy ASR. This matters for real-time conversational agents. ## Key points - **Inputs and features:** Whisper serves as both the ASR system and the audio feature extractor, and RoBERTa supplies text embeddings. Whisper features outperformed MFCC, eGeMAPS and Wav2Vec2 embeddings. - **Ablation results:** - Early fusion improved F1 by about 2 points over late fusion on both datasets. - Adding the unimodal encoders raised accuracy by about 2 points. - Cross-attention in utterance encoding gave no notable benefit. - **Class-wise gains:** HF improved 32 of 52 MRDA classes, including +24 F1 on Or Question (qr), and 10 of 12 EMOTyDA classes. - **MRDA with oracle transcripts:** HF reaches 91.8 accuracy, which is behind only Li et al. (2019) at 92.2 and Chapuis et al. (2020) at 92.4. The online HF accuracy is 82.6. - **EMOTyDA:** - With oracle input, HF (63.42 accuracy / 50.3 F1) exceeds the best Saha et al. (2020) model (51.00 / 44.52). - In the online setting, HF reaches 55.58 accuracy and 41.95 F1. - The code is publicly released. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/) Topics: [Speech act theory](https://md.research.elamur.ai/free/topics/speech-act-theory/) ## Cited by Cited by 5 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 4 not yet. - [Do LLMs Understand Dialogues? A Case Study on Dialogue Acts](https://md.research.elamur.ai/free/papers/qamar-2025-do-llms-understand-dialogues-case/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{miah2023hierarchical, title = {{Hierarchical Fusion for Online Multimodal Dialog Act Classification}}, author = {Miah, Md Messal Monem and Pyarelal, Adarsh and Huang, Ruihong}, booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023}, year = {2023}, doi = {10.18653/v1/2023.findings-emnlp.505}, url = {https://doi.org/10.18653/v1/2023.findings-emnlp.505} } ``` - [← previous Hexatagging: Projective Dependency Parsing as Tagging](https://md.research.elamur.ai/free/papers/acl-2023-amini-hexatagging-projective-dependency-parsing-as/) - [next → How Are Idioms Processed Inside Transformer Language Models?](https://md.research.elamur.ai/free/papers/tian-2023-how-are-idioms-processed-inside/) - [↑ back to papers of 2023](https://md.research.elamur.ai/free/papers/2023/page/3/) ## References 2 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Ali Ahmadvand, Jason Ingyu Choi, and Eugene Agichtein. 2019. Contextual dialogue act classification for open-domain conversational agents. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR’19, page 1273–1276, New York, NY, USA. Association for Computing Machinery. [↗](https://doi.org/10.1145/3331184.3331375) 2. Jeremy Ang, Yang Liu, and Elizabeth Shriberg. 2005. Automatic dialog act segmentation and classification in multiparty meetings. Proceedings. (ICASSP ’05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005., 1:I/1061–I/1064 Vol. 1. [find](https://scholar.google.com/scholar?q=Jeremy+Ang%2C+Yang+Liu%2C+and+Elizabeth+Shriberg%2E+2005%2E+Automatic+dialog+act+segmentation+and+classification+in+multiparty+meetings%2E+Proceedings%2E+%28ICASSP+%E2%80%9905%29%2E+IEEE+International+Conference+on+Acoustics%2C+)