---
title: "DualFact+ — paper by Oguz et al. (2026) — summary and ideas"
description: "The authors propose DualFact, which splits factuality into two layers. 0.429 Pearson. 24 references."
authors: ["Cennet Oguz", "Yasser Hamidullah", "Josef van Genabith", "Simon Ostermann"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.1912/"
doi: "10.18653/v1/2026.findings-acl.1912"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/oguz-2026-dualfact-multimodal-fact-verification-framework/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/oguz-2026-dualfact-multimodal-fact-verification-framework/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/30/) / DualFact+: A Multimodal Fact Verification Framework for Procedural Video Captioning

# DualFact+: A Multimodal Fact Verification Framework for Procedural Video Captioning

**Authors:** [Cennet Oguz](https://md.research.elamur.ai/free/en/authors/cennet-oguz/), [Yasser Hamidullah](https://md.research.elamur.ai/free/en/authors/yasser-hamidullah/), [Josef van Genabith](https://md.research.elamur.ai/free/en/authors/josef-van-genabith/) and [Simon Ostermann](https://md.research.elamur.ai/free/en/authors/simon-ostermann/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.1912/) · [PDF](https://aclanthology.org/2026.findings-acl.1912.pdf) · [DOI 10.18653/v1/2026.findings-acl.1912](https://doi.org/10.18653/v1/2026.findings-acl.1912) · [OpenAlex](https://openalex.org/W7166859371)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
The paper addresses a gap in how procedural (how-to) video captions are evaluated. Lexical and embedding-based metrics cannot tell whether a caption assigns objects, tools and locations to the correct roles. The authors propose DualFact, which splits factuality into two layers. Conceptual facts are abstract role labels such as Action, Ingredient/Object, Tool and Location. Contextual facts are grounded predicate–argument tuples, for example cut(onion).

Facts are checked with NLI models in two variants: DualFact-T uses textual evidence and DualFact-V uses video evidence. The framework also fills in arguments that captions leave unstated (VIA) and builds sets of contrasting facts. Errors are sorted into three types: hallucination, salience and omission.

Two benchmarks were built:
- **YouCook3-Fact**, derived from YouCook2 by splitting captions into clauses.
- **CraftBench-Fact**, a new set of furniture-crafting videos annotated from scratch.

Tests on current multimodal LLMs show that their captions often miss arguments or mix up roles. Judging captions without the video inflates hallucination counts. Fact-based scores agree with human judgments better than standard metrics do. This makes the framework a more interpretable way to check visual grounding.

## Key points
- **Dataset sizes.**
  - YouCook3 train: 300 videos, 2,570 clips, 3,759 VIA arguments, 7,725 conceptual and 6,826 contextual facts.
  - YouCook3 test: 200 videos, 1,800 clips.
  - CraftBench train: 120 videos, 1,735 clips.
  - CraftBench test: 100 videos, 1,468 clips.
  - Code and data are available at github.com/OguzCennet/DualFact.
- **Correlation with human judgments.** Caption-based conceptual fact scores correlate best (Spearman ρ = 0.429, Pearson r = 0.534). Baselines score much lower: CIDEr 0.140, BERTScore −0.050, EMScore (Text) 0.280.
  - Contextual fact scores correlate less well than conceptual ones in this table, which conflicts with the abstract's claim of a particular advantage for contextual facts.
- **Human study.** With captions only, annotators judged 45.66% of conceptual and 21.55% of contextual facts correct. With video, these rose to 84.30% and 60.34%.
- **Predicted captions.** Textual verification of predicted captions against gold facts reaches only 19.92% accuracy.
  - Caption-only analysis inflates hallucination rates (e.g., 83.21% for locations).
  - Video grounding reclassifies many of these errors as salience (e.g., 36.17% for objects).
- **Naming inconsistencies.**
  - The PDF title reads "…Procedural Video Understanding", while the library title says "Captioning".
  - The conclusion calls the framework "MultiFactScore" rather than DualFact.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [German Research Centre for Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/german-research-centre-for-artificial-intelligence/) — Research institute, Kaiserslautern, Germany · 79 works · 29,970 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (1): [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 811 works

## Cite

Cite as: Cennet Oguz, Yasser Hamidullah, Josef van Genabith and Simon Ostermann. 2026. DualFact+: A Multimodal Fact Verification Framework for Procedural Video Captioning. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{oguz2026dualfact,
  title = {{DualFact+: A Multimodal Fact Verification Framework for Procedural Video Captioning}},
  author = {Cennet Oguz, Yasser Hamidullah, Josef {van Genabith} and Simon Ostermann},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1912},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1912}
}
```

- [← previous DualAxis AI at SemEval-2026 Task 3: Dimensional Aspect-Based Sentiment Analysis](https://md.research.elamur.ai/free/en/papers/missaoui-2026-dualaxis-ai-semeval-2026-task/)
- [next → DuIVRS-2: An LLM-based Interactive Voice Response System for Large-scale POI Attribute Acquisition](https://md.research.elamur.ai/free/en/papers/zhang-2026-duivrs-2-llm-based-interactive/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/30/)

## References

24 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. 2023. [Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks](https://md.research.elamur.ai/free/en/papers/chen-2023-internvl-scaling-up-vision/). arXiv preprint arXiv:2312.142…
2. Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. 2018. Scaling egocentric vision: The epic-kitchens dataset. In Proceedings of the European conference on computer vision (ECCV),… [find](https://scholar.google.com/scholar?q=Dima+Damen%2C+Hazel+Doughty%2C+Giovanni+Maria+Farinella%2C+Sanja+Fidler%2C+Antonino+Furnari%2C+Evangelos+Kazakos%2C+Davide+Moltisanti%2C+Jonathan+Munro%2C+Toby+Perrett%2C+Will+Price%2C+et+al%2E+2018%2E+Scaling+egocentric+vis)
3. Michael Denkowski and Alon Lavie. 2014. [Meteor universal: Language specific translation evaluation for any target language](https://md.research.elamur.ai/free/en/papers/denkowski-2014-meteor-universal-language-specific-translation/). In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 376–380, Baltimore, Maryland, USA. Association for Computational Linguistics.
4. Nouha Dziri et al. 2023. [Faith and fate: Limits of transformers on compositionality](https://md.research.elamur.ai/free/en/papers/dziri-2023-faith-fate-limits-transformers-compositionality/). Transactions of the Association for Computational Linguistics.
5. Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. [Clipscore: A reference-free evaluation metric for image captioning](https://md.research.elamur.ai/free/en/papers/hessel-2021-clipscore-reference-free-evaluation-metric/). arXiv preprint arXiv:2104.08718.
6. Wei Jiang et al. 2023. Hallucination in video captioning. IEEE Transactions on Pattern Analysis and Machine Intelligence. [find](https://scholar.google.com/scholar?q=Wei+Jiang+et+al%2E+2023%2E+Hallucination+in+video+captioning%2E+IEEE+Transactions+on+Pattern+Analysis+and+Machine+Intelligence%2E)
7. Liqiang Jing, Viet Lai, Seunghyun Yoon, Trung Bui, and Xinya Du. 2025. Fifa: Unified faithfulness evaluation framework for text-to-video and video-to-text generation. arXiv preprint arXiv:2507.06523. [find](https://scholar.google.com/scholar?q=Liqiang+Jing%2C+Viet+Lai%2C+Seunghyun+Yoon%2C+Trung+Bui%2C+and+Xinya+Du%2E+2025%2E+Fifa%3A+Unified+faithfulness+evaluation+framework+for+text%2Dto%2Dvideo+and+video%2Dto%2Dtext+generation%2E+arXiv+preprint+arXiv%3A2507%2E06523%2E)
8. Liqiang Jing, Ruosen Li, Yunmo Chen, and Xinya Du. 2023. Faithscore: Fine-grained evaluations of hallucinations in large vision-language models. arXiv preprint arXiv:2311.01477. [find](https://scholar.google.com/scholar?q=Liqiang+Jing%2C+Ruosen+Li%2C+Yunmo+Chen%2C+and+Xinya+Du%2E+2023%2E+Faithscore%3A+Fine%2Dgrained+evaluations+of+hallucinations+in+large+vision%2Dlanguage+models%2E+arXiv+preprint+arXiv%3A2311%2E01477%2E)
9. Chloé Kiddon, Ganesa Thandavam Ponnuraj, Luke Zettlemoyer, and Yejin Choi. 2015. Mise en place: Unsupervised interpretation of instructional recipes. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 982–992. [find](https://scholar.google.com/scholar?q=Chlo%C3%A9+Kiddon%2C+Ganesa+Thandavam+Ponnuraj%2C+Luke+Zettlemoyer%2C+and+Yejin+Choi%2E+2015%2E+Mise+en+place%3A+Unsupervised+interpretation+of+instructional+recipes%2E+In+Proceedings+of+the+2015+Conference+on+Empirical)
10. Saehyung Lee, Seunghyun Yoon, Trung Bui, Jing Shi, and Sungroh Yoon. 2024. Toward robust hyper-detailed image captioning: A multiagent approach and dual evaluation metrics for factuality and coverage. arXiv preprint arXiv:2412.15484. — [arXiv:2412.15484](https://arxiv.org/abs/2412.15484) [find](https://scholar.google.com/scholar?q=Saehyung+Lee%2C+Seunghyun+Yoon%2C+Trung+Bui%2C+Jing+Shi%2C+and+Sungroh+Yoon%2E+2024%2E+Toward+robust+hyper%2Ddetailed+image+captioning%3A+A+multiagent+approach+and+dual+evaluation+metrics+for+factuality+and+coverage%2E)
11. Chin-Yew Lin. 2004. [Rouge: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text summarization branches out, pages 74–81.
12. Hui Liu and Xiaojun Wan. 2023. Models see hallucinations: Evaluating the factuality in video captioning. arXiv preprint arXiv:2303.02961. [find](https://scholar.google.com/scholar?q=Hui+Liu+and+Xiaojun+Wan%2E+2023%2E+Models+see+hallucinations%3A+Evaluating+the+factuality+in+video+captioning%2E+arXiv+preprint+arXiv%3A2303%2E02961%2E)
13. Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. 2019. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2630–2640. [find](https://scholar.google.com/scholar?q=Antoine+Miech%2C+Dimitri+Zhukov%2C+Jean%2DBaptiste+Alayrac%2C+Makarand+Tapaswi%2C+Ivan+Laptev%2C+and+Josef+Sivic%2E+2019%2E+Howto100m%3A+Learning+a+text%2Dvideo+embedding+by+watching+hundred+million+narrated+video+clips%2E)
14. Cennet Oguz, Pascal Denis, Emmanuel Vincent, Simon Ostermann, and Josef van Genabith. 2023. Find-2-find: Multitask learning for anaphora resolution and object localization. In 2023 Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Cennet+Oguz%2C+Pascal+Denis%2C+Emmanuel+Vincent%2C+Simon+Ostermann%2C+and+Josef+van+Genabith%2E+2023%2E+Find%2D2%2Dfind%3A+Multitask+learning+for+anaphora+resolution+and+object+localization%2E+In+2023+Conference+on+Empir)
15. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
16. Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. [Object hallucination in image captioning](https://md.research.elamur.ai/free/en/papers/rohrbach-2018-object-hallucination-image-captioning/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4035–4045. Association for Computational Linguistics.
17. Michael Roth and Anette Frank. 2015. Inducing implicit arguments from comparable texts. Computational Linguistics, 41(4):625–664. [find](https://scholar.google.com/scholar?q=Michael+Roth+and+Anette+Frank%2E+2015%2E+Inducing+implicit+arguments+from+comparable+texts%2E+Computational+Linguistics%2C+41%284%29%3A625%E2%80%93664%2E)
18. Sara Sarto et al. 2023. Positive-augmented contrastive learning for image and video captioning evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE. [find](https://scholar.google.com/scholar?q=Sara+Sarto+et+al%2E+2023%2E+Positive%2Daugmented+contrastive+learning+for+image+and+video+captioning+evaluation%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognition%2E+IEEE%2E)
19. Yaya Shi, Xu Yang, Haiyang Xu, Chunfeng Yuan, Bing Li, Weiming Hu, and Zheng-Jun Zha. 2022. Emscore: Evaluating video captioning via coarse-grained and fine-grained embedding matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17929–17938. [find](https://scholar.google.com/scholar?q=Yaya+Shi%2C+Xu+Yang%2C+Haiyang+Xu%2C+Chunfeng+Yuan%2C+Bing+Li%2C+Weiming+Hu%2C+and+Zheng%2DJun+Zha%2E+2022%2E+Emscore%3A+Evaluating+video+captioning+via+coarse%2Dgrained+and+fine%2Dgrained+embedding+matching%2E+In+Proceedings+)
20. Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. 2019. Coin: A large-scale dataset for comprehensive instructional video analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1207–1216. [find](https://scholar.google.com/scholar?q=Yansong+Tang%2C+Dajun+Ding%2C+Yongming+Rao%2C+Yu+Zheng%2C+Danyang+Zhang%2C+Lili+Zhao%2C+Jiwen+Lu%2C+and+Jie+Zhou%2E+2019%2E+Coin%3A+A+large%2Dscale+dataset+for+comprehensive+instructional+video+analysis%2E+In+Proceedings+of+)
21. Jialin Yuan et al. 2021. Video fact-checking and explanation generation. In Proceedings of the 29th ACM International Conference on Multimedia. ACM. [find](https://scholar.google.com/scholar?q=Jialin+Yuan+et+al%2E+2021%2E+Video+fact%2Dchecking+and+explanation+generation%2E+In+Proceedings+of+the+29th+ACM+International+Conference+on+Multimedia%2E+ACM%2E)
22. Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022. Towards a unified multidimensional evaluator for text generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Association for Computational… [find](https://scholar.google.com/scholar?q=Ming+Zhong%2C+Yang+Liu%2C+Da+Yin%2C+Yuning+Mao%2C+Yizhu+Jiao%2C+Pengfei+Liu%2C+Chenguang+Zhu%2C+Heng+Ji%2C+and+Jiawei+Han%2E+2022%2E+Towards+a+unified+multidimensional+evaluator+for+text+generation%2E+In+Proceedings+of+the)
23. Metric Positive recall Negative specificity Overall accuracy Count Value (%) 4 329 / 7 221 6 524 / 7 878 – 59.95 82.81 71.88 V . We compute grounding quality separately for positive and negative objects. Positive grounding recall. Recall pos = *Table 12: Dataset-level object grounding performance.*… [find](https://scholar.google.com/scholar?q=Metric+Positive+recall+Negative+specificity+Overall+accuracy+Count+Value+%28%25%29+4+329+%2F+7+221+6+524+%2F+7+878+%E2%80%93+59%2E95+82%2E81+71%2E88+V+%2E+We+compute+grounding+quality+separately+for+positive+and+negative+objec)
24. Luowei Zhou, Chenliang Xu, and Jason Corso. 2018. Towards automatic learning of procedures from web instructional videos. In Proceedings of the AAAI conference on artificial intelligence, volume 32. [find](https://scholar.google.com/scholar?q=Luowei+Zhou%2C+Chenliang+Xu%2C+and+Jason+Corso%2E+2018%2E+Towards+automatic+learning+of+procedures+from+web+instructional+videos%2E+In+Proceedings+of+the+AAAI+conference+on+artificial+intelligence%2C+volume+32%2E)

*Updated 1 November 2026*