---
title: "Grounding Action Descriptions in Videos — Regneri 2013"
description: "Text-only distributional models describe word meaning incompletely, and earlier multimodal work… 25 references, cited by 60 papers here."
authors: ["Michaela Regneri", "Marcus Rohrbach", "Dominikus Wetzel", "Stefan Thater", "Bernt Schiele", "Manfred Pinkal"]
year: 2013
venue: "Transactions of the Association for Computational Linguistics"
source: "https://aclanthology.org/Q13-1003/"
doi: "10.1162/tacl_a_00207"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/regneri-2013-grounding-action-descriptions-videos/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/regneri-2013-grounding-action-descriptions-videos/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2013](https://md.research.elamur.ai/free/en/papers/2013/) / Grounding Action Descriptions in Videos

# Grounding Action Descriptions in Videos

**Authors:** [Michaela Regneri](https://md.research.elamur.ai/free/en/authors/michaela-regneri/), [Marcus Rohrbach](https://md.research.elamur.ai/free/en/authors/marcus-rohrbach/), Dominikus Wetzel, Stefan Thater, Bernt Schiele and [Manfred Pinkal](https://md.research.elamur.ai/free/en/authors/manfred-pinkal/) · **Year:** 2013 · **Venue:** [Transactions of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/tacl/) ([Transactions of the Association for Computational Linguistics (TACL) 2013](https://md.research.elamur.ai/free/en/venues/tacl/2013/)) · **Publisher:** [The MIT Press](https://md.research.elamur.ai/free/en/publishers/the-mit-press/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/Q13-1003/) · [PDF](https://aclanthology.org/Q13-1003.pdf) · [DOI 10.1162/tacl\_a\_00207](https://doi.org/10.1162/tacl_a_00207) · [OpenAlex](https://openalex.org/W2111078031)

**Citations:** 467 citations (OpenAlex, 2026-10-14) · [60](https://md.research.elamur.ai/free/en/papers/regneri-2013-grounding-action-descriptions-videos/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Text-only distributional models describe word meaning incompletely, and earlier multimodal work grounded semantics only in still images. This paper grounds descriptions of actions in video instead. The authors build TACOS (Saarbrücken Corpus of Textually Annotated Cooking Scenes) on top of the MPII Composites cooking videos. Mechanical Turk workers wrote several descriptions for each video, and every sentence was linked to a video segment and to the existing low-level activity annotations. The authors also create ASim, a gold standard of paired action descriptions with their video segments, rated for similarity in the style of the Usage Similarity dataset of Erk et al. (2009). They then test existing textual and visual similarity measures against ASim, both alone and combined by averaging z-scores. Video-based measures correlate significantly with human judgments, and mixing the two modalities works best, coming close to inter-annotator agreement. Visual features help mostly when pairs share an object but describe different actions, which suggests they capture action similarity rather than object similarity. All data are publicly released.

## Key points
- MPII Composites provides 212 high-resolution videos (4.5 min on average) of 41 cooking tasks performed by 22 subjects. It includes 8818 segments annotated with 60 low-level activity labels (e.g. PEEL, STIR).
- For TACOS, 127 videos covering 26 ingredient-manipulation tasks each received 20 MTurk descriptions, giving 2540 assignments. Each description contained 5–15 sentences and was collected with an adapted Vatic tool.
- On the full ASim set, the ALL COMBINED model (textual and visual measures together) reached the best correlation with human ratings, 0.55.
- Visual models beat textual ones on the SAME OBJECT subset but added nothing on the SAME VERB subset. Raw visual features alone account for most of the gain, without needing the supervised classifier.
- The released resources are the videos, low-level annotations, aligned descriptions, the ASim dataset, and the visual features. Apart from the visual classifier, all models are unsupervised and could transfer to other domains with observable actions.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Saarland University](https://md.research.elamur.ai/free/en/universities/saarland-university/) — University, Saarbrücken, Germany · 102 works · 17,447 citations
- [Max Planck Institute for Informatics](https://md.research.elamur.ai/free/en/labs/max-planck-institute-for-informatics/) — Research institute, Saarbrücken, Germany · 11 works · 4,289 citations
- [Max Planck Society](https://md.research.elamur.ai/free/en/labs/max-planck-society/) — Research institute, Munich, Germany · 10 works · 42,142 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

## Cite

Cite as: Michaela Regneri, Marcus Rohrbach, Dominikus Wetzel, Stefan Thater, Bernt Schiele and Manfred Pinkal. 2013. Grounding Action Descriptions in Videos. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{regneri2013grounding,
  title = {{Grounding Action Descriptions in Videos}},
  author = {Michaela Regneri, Marcus Rohrbach, Dominikus Wetzel, Stefan Thater, Bernt Schiele and Manfred Pinkal},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2013},
  doi = {10.1162/tacl\_a\_00207},
  url = {https://doi.org/10.1162/tacl_a_00207}
}
```

- [← previous Deterministic Coreference Resolution Based on Entity-Centric, Precision-Ranked Rules](https://md.research.elamur.ai/free/en/papers/lee-2013-deterministic-coreference-resolution-based-entity/)
- [next → Learning Whom to Trust with MACE](https://md.research.elamur.ai/free/en/papers/hovy-2013-learning-whom-trust-mace/)
- [↑ back to papers of 2013](https://md.research.elamur.ai/free/en/papers/2013/)

## References

25 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Luis von Ahn and Laura Dabbish. 2004. Labeling images with a computer game. In Proceedings of SIGCHI 2004. [find](https://scholar.google.com/scholar?q=Luis+von+Ahn+and+Laura+Dabbish%2E+2004%2E+Labeling+images+with+a+computer+game%2E+In+Proceedings+of+SIGCHI+2004%2E)
2. Elia Bruni, Giang Binh Tran, and Marco Baroni. 2011. Distributional semantics from text and images. In Proceedings of GEMS 2011. [find](https://scholar.google.com/scholar?q=Elia+Bruni%2C+Giang+Binh+Tran%2C+and+Marco+Baroni%2E+2011%2E+Distributional+semantics+from+text+and+images%2E+In+Proceedings+of+GEMS+2011%2E)
3. David L. Chen and William B. Dolan. 2011. [Collecting highly parallel data for paraphrase evaluation](https://md.research.elamur.ai/free/en/papers/chen-2011-collecting-highly-parallel-data-paraphrase/). In Proceedings of ACL 2011.
4. Timothee Cour, Chris Jordan, Eleni Miltsakaki, and Ben Taskar. 2008. Movie/script: Alignment and parsing of video and text transcription. In Computer Vision – ECCV 2008, volume 5305 of Lecture Notes in Computer Science, pages 158–171. [find](https://scholar.google.com/scholar?q=Timothee+Cour%2C+Chris+Jordan%2C+Eleni+Miltsakaki%2C+and+Ben+Taskar%2E+2008%2E+Movie%2Fscript%3A+Alignment+and+parsing+of+video+and+text+transcription%2E+In+Computer+Vision+%E2%80%93+ECCV+2008%2C+volume+5305+of+Lecture+Notes+i)
5. Springer Berlin Heidelberg. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The PASCAL recognising textual entailment challenge. In Proceedings of MLCW 2005. Jesse Dodge, Amit Goyal, Xufeng Han, Alyssa Mensch, Margaret Mitchell, Karl Stratos, Kota Yamaguchi, Yejin Choi, Hal Daumé III, Alexand… [find](https://scholar.google.com/scholar?q=Springer+Berlin+Heidelberg%2E+Ido+Dagan%2C+Oren+Glickman%2C+and+Bernardo+Magnini%2E+2005%2E+The+PASCAL+recognising+textual+entailment+challenge%2E+In+Proceedings+of+MLCW+2005%2E+Jesse+Dodge%2C+Amit+Goyal%2C+Xufeng+Han%2C)
6. Katrin Erk and Diana McCarthy. 2009. Graded word sense assignment. In Proceedings of EMNLP 2009. [find](https://scholar.google.com/scholar?q=Katrin+Erk+and+Diana+McCarthy%2E+2009%2E+Graded+word+sense+assignment%2E+In+Proceedings+of+EMNLP+2009%2E)
7. Katrin Erk, Diana McCarthy, and Nicholas Gaylord. 2009. Investigations on word senses and word usages. In Proceedings of ACL/AFNLP 2009. [find](https://scholar.google.com/scholar?q=Katrin+Erk%2C+Diana+McCarthy%2C+and+Nicholas+Gaylord%2E+2009%2E+Investigations+on+word+senses+and+word+usages%2E+In+Proceedings+of+ACL%2FAFNLP+2009%2E)
8. Katrin Erk, Diana McCarthy, and Nick Gaylord. 2012. [Measuring word meaning in context](https://md.research.elamur.ai/free/en/papers/erk-2013-measuring-word-meaning-context/).
9. CL. Yansong Feng and Mirella Lapata. 2010. Visual information in semantic representation. In Proceedings of HLT-NAACL 2010. [find](https://scholar.google.com/scholar?q=CL%2E+Yansong+Feng+and+Mirella+Lapata%2E+2010%2E+Visual+information+in+semantic+representation%2E+In+Proceedings+of+HLT%2DNAACL+2010%2E)
10. A. M. Glenberg. 2002. Grounding language in action. [find](https://scholar.google.com/scholar?q=A%2E+M%2E+Glenberg%2E+2002%2E+Grounding+language+in+action%2E)
11. Psychonomic Bulletin & Review. Sonal Gupta and Raymond J. Mooney. 2010. Using closed captions as supervision for video activity recognition. In Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence (AAAI-2010), pages 1083–1088, Atlanta, GA, July. [find](https://scholar.google.com/scholar?q=Psychonomic+Bulletin+%26+Review%2E+Sonal+Gupta+and+Raymond+J%2E+Mooney%2E+2010%2E+Using+closed+captions+as+supervision+for+video+activity+recognition%2E+In+Proceedings+of+the+Twenty%2DFourth+AAAI+Conference+on+Arti)
12. Abhinav Gupta, Praveen Srinivasan, Jianbo Shi, and Larry S. Davis. 2009. Understanding videos, constructing plots learning a visually grounded storyline model from annotated videos. In Proceedings of CVPR 2009. [find](https://scholar.google.com/scholar?q=Abhinav+Gupta%2C+Praveen+Srinivasan%2C+Jianbo+Shi%2C+and+Larry+S%2E+Davis%2E+2009%2E+Understanding+videos%2C+constructing+plots+learning+a+visually+grounded+storyline+model+from+annotated+videos%2E+In+Proceedings+of+)
13. Steve R. Howell, Damian Jankowicz, and Suzanna Becker. 2005. A model of grounded language acquisition: Sensorimotor features improve lexical and grammatical learning. [find](https://scholar.google.com/scholar?q=Steve+R%2E+Howell%2C+Damian+Jankowicz%2C+and+Suzanna+Becker%2E+2005%2E+A+model+of+grounded+language+acquisition%3A+Sensorimotor+features+improve+lexical+and+grammatical+learning%2E)
14. JML. S. Mathe, A. Fazly, S. Dickinson, and S. Stevenson. 2008. Learning the abstract motion semantics of verbs from captioned videos. pages 1–8. [find](https://scholar.google.com/scholar?q=JML%2E+S%2E+Mathe%2C+A%2E+Fazly%2C+S%2E+Dickinson%2C+and+S%2E+Stevenson%2E+2008%2E+Learning+the+abstract+motion+semantics+of+verbs+from+captioned+videos%2E+pages+1%E2%80%938%2E)
15. Jeff Mitchell and Mirella Lapata. 2008. Vector-based models of semantic composition. In Proceedings of ACL 2008. [find](https://scholar.google.com/scholar?q=Jeff+Mitchell+and+Mirella+Lapata%2E+2008%2E+Vector%2Dbased+models+of+semantic+composition%2E+In+Proceedings+of+ACL+2008%2E)
16. Tanvi S. Motwani and Raymond J. Mooney. 2012. Improving video activity recognition using object recognition and text mining. In Proceedings of the 20th European Conference on Artificial Intelligence (ECAI-2012), pages 600–605, August. [find](https://scholar.google.com/scholar?q=Tanvi+S%2E+Motwani+and+Raymond+J%2E+Mooney%2E+2012%2E+Improving+video+activity+recognition+using+object+recognition+and+text+mining%2E+In+Proceedings+of+the+20th+European+Conference+on+Artificial+Intelligence+%28)
17. Jeff Orkin and Deb Roy. 2009. Automatic learning and generation of social behavior from collective human gameplay. In Proceedings of AAMAS 2009. [find](https://scholar.google.com/scholar?q=Jeff+Orkin+and+Deb+Roy%2E+2009%2E+Automatic+learning+and+generation+of+social+behavior+from+collective+human+gameplay%2E+In+Proceedings+of+AAMAS+2009%2E)
18. Hilke Reckman, Jeff Orkin, and Deb Roy. 2011. Extracting aspects of determiner meaning from dialogue in a virtual world environment. In Proceedings of CCS 2011, IWCS ’11. [find](https://scholar.google.com/scholar?q=Hilke+Reckman%2C+Jeff+Orkin%2C+and+Deb+Roy%2E+2011%2E+Extracting+aspects+of+determiner+meaning+from+dialogue+in+a+virtual+world+environment%2E+In+Proceedings+of+CCS+2011%2C+IWCS+%E2%80%9911%2E)
19. Marcus Rohrbach, Sikandar Amin, Mykhaylo Andriluka, and Bernt Schiele. 2012a. A database for fine grained activity detection of cooking activities. In Proceedings of CVPR 2012. [find](https://scholar.google.com/scholar?q=Marcus+Rohrbach%2C+Sikandar+Amin%2C+Mykhaylo+Andriluka%2C+and+Bernt+Schiele%2E+2012a%2E+A+database+for+fine+grained+activity+detection+of+cooking+activities%2E+In+Proceedings+of+CVPR+2012%2E)
20. Marcus Rohrbach, Michaela Regneri, Micha Andriluka, Sikandar Amin, Manfred Pinkal, and Bernt Schiele. 2012b. Script data for attribute-based recognition of composite activities. In Proceedings of ECCV 2012. [find](https://scholar.google.com/scholar?q=Marcus+Rohrbach%2C+Michaela+Regneri%2C+Micha+Andriluka%2C+Sikandar+Amin%2C+Manfred+Pinkal%2C+and+Bernt+Schiele%2E+2012b%2E+Script+data+for+attribute%2Dbased+recognition+of+composite+activities%2E+In+Proceedings+of+ECCV)
21. Carina Silberer and Mirella Lapata. 2012. Grounded models of semantic representation. In Proceedings of EMNLP-CoNLL 2012. [find](https://scholar.google.com/scholar?q=Carina+Silberer+and+Mirella+Lapata%2E+2012%2E+Grounded+models+of+semantic+representation%2E+In+Proceedings+of+EMNLP%2DCoNLL+2012%2E)
22. Mark Steyvers. 2010. Combining feature norms and text data with topic models. Acta Psychologica, 133(3):234 – 243. ¡ce:title¿Formal modeling of semantic concepts¡/ce:title¿. Stefan Thater, Hagen Fürstenau, and Manfred Pinkal. 2011. Word meaning in context: A simple and effective vector model. In Pr… [find](https://scholar.google.com/scholar?q=Mark+Steyvers%2E+2010%2E+Combining+feature+norms+and+text+data+with+topic+models%2E+Acta+Psychologica%2C+133%283%29%3A234+%E2%80%93+243%2E+%C2%A1ce%3Atitle%C2%BFFormal+modeling+of+semantic+concepts%C2%A1%2Fce%3Atitle%C2%BF%2E+Stefan+Thater%2C+Hagen+Fu%CC%88rs)
23. Peter D. Turney and Patrick Pantel. 2010. From frequency to meaning. vector space models for semantics. [find](https://scholar.google.com/scholar?q=Peter+D%2E+Turney+and+Patrick+Pantel%2E+2010%2E+From+frequency+to+meaning%2E+vector+space+models+for+semantics%2E)
24. JAIR. E. Tzoukermann, J. Neumann, J. Kosecka, C. Fermuller, I. Perera, F. Ferraro, B. Sapp, R. Chaudhry, and G. Singh. 2011. Language models for semantic extraction and filtering in video action recognition. In AAAI Workshop on Language-Action Tools for Cognitive Artificial Agents. [find](https://scholar.google.com/scholar?q=JAIR%2E+E%2E+Tzoukermann%2C+J%2E+Neumann%2C+J%2E+Kosecka%2C+C%2E+Fermuller%2C+I%2E+Perera%2C+F%2E+Ferraro%2C+B%2E+Sapp%2C+R%2E+Chaudhry%2C+and+G%2E+Singh%2E+2011%2E+Language+models+for+semantic+extraction+and+filtering+in+video+action+recog)
25. Carl Vondrick, Donald Patterson, and Deva Ramanan. 2012. Efficiently scaling up crowdsourced video annotation. IJCV. Heng Wang, Alexander Kläser, Cordelia Schmid, and Cheng-Lin Liu. 2011. Action Recognition by Dense Trajectories. In Proceedings of CVPR 2011. [find](https://scholar.google.com/scholar?q=Carl+Vondrick%2C+Donald+Patterson%2C+and+Deva+Ramanan%2E+2012%2E+Efficiently+scaling+up+crowdsourced+video+annotation%2E+IJCV%2E+Heng+Wang%2C+Alexander+Kla%CC%88ser%2C+Cordelia+Schmid%2C+and+Cheng%2DLin+Liu%2E+2011%2E+Action+Reco)

*Updated 24 October 2026*