---
title: "Multimodal Semantic Learning from Child-Directed Input — Lazaridou 2016"
description: "The paper models how children acquire word meanings. 25 references, cited by 9 papers here."
authors: ["Angeliki Lazaridou", "Grzegorz Chrupała", "Raquel Fernández", "Marco Baroni"]
year: 2016
venue: "Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies"
source: "https://aclanthology.org/N16-1043/"
doi: "10.18653/v1/n16-1043"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lazaridou-2016-multimodal-semantic-learning-child-directed/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lazaridou-2016-multimodal-semantic-learning-child-directed/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/4/) / Multimodal Semantic Learning from Child-Directed Input

# Multimodal Semantic Learning from Child-Directed Input

**Authors:** [Angeliki Lazaridou](https://md.research.elamur.ai/free/en/authors/angeliki-lazaridou/), [Grzegorz Chrupała](https://md.research.elamur.ai/free/en/authors/grzegorz-chrupala/), [Raquel Fernández](https://md.research.elamur.ai/free/en/authors/raquel-fernandez/) and [Marco Baroni](https://md.research.elamur.ai/free/en/authors/marco-baroni/) · **Year:** 2016 · **Venue:** Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/N16-1043/) · [PDF](https://aclanthology.org/N16-1043.pdf) · [DOI 10.18653/v1/n16-1043](https://doi.org/10.18653/v1/n16-1043)

**Citations:** 9 in our library, of the 140,884 papers whose bibliographies we read

## Summary

The paper models how children acquire word meanings. Previous computational models typically represented a scene as a set of artificial symbolic labels, whereas the authors instead work with real images of objects. They extend the Multimodal Skip-Gram (MSG) model. It learns vector representations of words by simultaneously predicting neighboring words and the visual vectors of objects in the scene, but it does not know in advance which object corresponds to which word. An attention mechanism is added to the model, assigning greater weight to word–object pairs that the model already considers related. A social cue is also added: whether the mother is holding the object in her hands. For the experiments, the Illustrated Frank et al. Corpus (IFC) was created. It is a fragment of CHILDES in which a mother addresses an infant, with the symbolic labels replaced by image features from ImageNet. The best variant, Attentive Social MSG, outperforms almost all previous approaches. Control experiments show that the gain comes specifically from the visual information and the natural order of utterances. The model often correctly links a word to an object after a single occurrence of the word and transfers its knowledge to new images. It therefore reproduces children's learning more plausibly.

## Key points

- **The IFC corpus.** It contains 624 utterances and 2,533 words. Each utterance is accompanied by up to 7 objects, 2 on average. Only 33% of utterances explicitly name a possible referent, and 8 of the 37 test words occur only once.
- **Visual vectors.** For each object, 100 images were taken from ImageNet. From each, a 4096-dimensional feature vector was extracted using Caffe and the CNN of Krizhevsky et al. These vectors were averaged and reduced to 200 dimensions.
- **Best-F.** MSG scores 0.64, AttentiveMSG 0.70, and AttentiveSocialMSG 0.73. For comparison: Bayesian CSL scores 0.54, BEAGLE 0.55, and PMI 0.53. The BEAGLE+PMI combination yields 0.83, but its parameters were tuned on the test data with access to the gold-standard annotation, so it cannot be fairly compared with MSG.
- **Control experiments.** If the visual vectors are shuffled among objects, the result drops to 0.65. If the sentence order is shuffled, it drops to 0.59.
- **Generalization.** The test covered 19 words, with 40 new ImageNet images for each. Precision@10 is 62% versus 25% for random selection. False matches are mostly visually similar to the correct object, for example a sheep for the word lamb. This resembles the overextension errors typical of children.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

Topics: [🧒 Psycholinguistics & language acquisition](https://md.research.elamur.ai/free/en/topics/psycholinguistics/) · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/)

## Cite

Cite as: Angeliki Lazaridou, Grzegorz Chrupała, Raquel Fernández and Marco Baroni. 2016. Multimodal Semantic Learning from Child-Directed Input. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{lazaridou2016multimodal,
  title = {{Multimodal Semantic Learning from Child-Directed Input}},
  author = {Angeliki Lazaridou, Grzegorz Chrupała, Raquel Fernández and Marco Baroni},
  booktitle = {Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2016},
  doi = {10.18653/v1/n16-1043},
  url = {https://doi.org/10.18653/v1/n16-1043}
}
```

- [← previous IMS HotCoref DE: A Data-driven Co-reference Resolver for German](https://md.research.elamur.ai/free/en/papers/roesiger-2016-ims-hotcoref-de-data-driven/)
- [next → To resume or not to resume: Some remarks on 'resumption' in left dislocation constructions in Polish, and its relevance for Biblical Hebrew](https://md.research.elamur.ai/free/en/papers/andrason-2016-resume-resume-some-remarks-resumption/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/4/)

## References

25 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of ICLR Conference Track, San Diego, CA. Published online: [http://www.iclr.cc/doku.php?id=](http://www.iclr.cc/doku.php?id=) iclr2015:main. [↗](http://www.iclr.cc/doku.php?id=) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
2. Marco Baroni, Alessandro Lenci, and Luca Onnis. 2007. ISA meets Lara: An incremental word space model for cognitively plausible simulations of semantic learning. In Proceedings of the ACL Workshop on Cognitive Aspects of Computational Language Acquisition, pages 49–56. [find](https://scholar.google.com/scholar?q=Marco+Baroni%2C+Alessandro+Lenci%2C+and+Luca+Onnis%2E+2007%2E+ISA+meets+Lara%3A+An+incremental+word+space+model+for+cognitively+plausible+simulations+of+semantic+learning%2E+In+Proceedings+of+the+ACL+Workshop+on+)
3. Elia Bruni, Nam Khanh Tran, and Marco Baroni. 2014. Multimodal distributional semantics. Journal of Artificial Intelligence Research, 49:1–47. [find](https://scholar.google.com/scholar?q=Elia+Bruni%2C+Nam+Khanh+Tran%2C+and+Marco+Baroni%2E+2014%2E+Multimodal+distributional+semantics%2E+Journal+of+Artificial+Intelligence+Research%2C+49%3A1%E2%80%9347%2E)
4. Jia Deng, Wei Dong, Richard Socher, Lia-Ji Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In Proceedings of CVPR, pages 248–255, Miami Beach, FL. [find](https://scholar.google.com/scholar?q=Jia+Deng%2C+Wei+Dong%2C+Richard+Socher%2C+Lia%2DJi+Li%2C+and+Li+Fei%2DFei%2E+2009%2E+Imagenet%3A+A+large%2Dscale+hierarchical+image+database%2E+In+Proceedings+of+CVPR%2C+pages+248%E2%80%93255%2C+Miami+Beach%2C+FL%2E)
5. Afsaneh Fazly, Afra Alishahi, and Suzanne Stevenson. 2010. A probabilistic computational model of cross-situational word learning. Cognitive Science, 34:1017–1063. [find](https://scholar.google.com/scholar?q=Afsaneh+Fazly%2C+Afra+Alishahi%2C+and+Suzanne+Stevenson%2E+2010%2E+A+probabilistic+computational+model+of+cross%2Dsituational+word+learning%2E+Cognitive+Science%2C+34%3A1017%E2%80%931063%2E)
6. Michael Frank, Noah Goodman, and Joshua Tenenbaum. 2007. A Bayesian framework for cross-situational word-learning. In Proceedings of NIPS, pages 457– 464, Vancouver, Canada. [find](https://scholar.google.com/scholar?q=Michael+Frank%2C+Noah+Goodman%2C+and+Joshua+Tenenbaum%2E+2007%2E+A+Bayesian+framework+for+cross%2Dsituational+word%2Dlearning%2E+In+Proceedings+of+NIPS%2C+pages+457%E2%80%93+464%2C+Vancouver%2C+Canada%2E)
7. Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093. Ákos Kádár, Afra Alishahi, and Grzegorz Chrupała. 2015. Learning wor… [our card](https://md.research.elamur.ai/free/en/papers/jia-2014-caffe-convolutional-architecture-fast-feature/)
8. Casey Kennington and David Schlangen. 2015. Simple learning and compositional application of perceptually grounded word meanings for incremental reference resolution. In Proceedings of the Conference for the Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=Casey+Kennington+and+David+Schlangen%2E+2015%2E+Simple+learning+and+compositional+application+of+perceptually+grounded+word+meanings+for+incremental+reference+resolution%2E+In+Proceedings+of+the+Conference+)
9. Brent Kievit-Kylar and Michael Jones. 2011. The Semantic Pictionary project. In Proceedings of CogSci, pages 2229–2234, Austin, TX. [find](https://scholar.google.com/scholar?q=Brent+Kievit%2DKylar+and+Michael+Jones%2E+2011%2E+The+Semantic+Pictionary+project%2E+In+Proceedings+of+CogSci%2C+pages+2229%E2%80%932234%2C+Austin%2C+TX%2E)
10. Brent Kievit-Kylar, George Kachergis, and Michael Jones. 2013. Naturalistic word-concept pair learning with semantic spaces. In Proceedings of CogSci, pages 2716–2721, Berlin, Germany. [find](https://scholar.google.com/scholar?q=Brent+Kievit%2DKylar%2C+George+Kachergis%2C+and+Michael+Jones%2E+2013%2E+Naturalistic+word%2Dconcept+pair+learning+with+semantic+spaces%2E+In+Proceedings+of+CogSci%2C+pages+2716%E2%80%932721%2C+Berlin%2C+Germany%2E)
11. Ryan Kiros, Ruslan Salakhutdinov, and Richard Zemel. 2014. Unifying visual-semantic embeddings with multimodal neural language models. In Proceedings of the NIPS Deep Learning and Representation Learning Workshop, Montreal, Canada. Published online: [http://www.dlworkshop.org/](http://www.dlworkshop.org/) accepted-papers. [↗](http://www.dlworkshop.org/) [our card](https://md.research.elamur.ai/free/en/papers/kiros-2014-unifying-visual-semantic-embeddings-multimodal/)
12. Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. 2012. ImageNet classification with deep convolutional neural networks. In Proceedings of NIPS, pages 1097– 1105, Lake Tahoe, Nevada. [find](https://scholar.google.com/scholar?q=Alex+Krizhevsky%2C+Ilya+Sutskever%2C+and+Geoffrey+Hinton%2E+2012%2E+ImageNet+classification+with+deep+convolutional+neural+networks%2E+In+Proceedings+of+NIPS%2C+pages+1097%E2%80%93+1105%2C+Lake+Tahoe%2C+Nevada%2E)
13. Angeliki Lazaridou, Nghia The Pham, and Marco Baroni. 2015. Combining language and vision with a multimodal skip-gram model. In Proceedings of NAACL, pages 153–163, Denver, CO. [our card](https://md.research.elamur.ai/free/en/papers/lazaridou-2015-combining-language-vision-multimodal-skip/)
14. Brian MacWhinney. 2000. The CHILDES Project: Tools for analyzing talk. Lawrence Erlbaum Associates, 3rd edition. [find](https://scholar.google.com/scholar?q=Brian+MacWhinney%2E+2000%2E+The+CHILDES+Project%3A+Tools+for+analyzing+talk%2E+Lawrence+Erlbaum+Associates%2C+3rd+edition%2E)
15. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. http://arxiv.org/abs/ 1301.3781/. Adria Recasens ∗ , Aditya Khosla ∗ , Carl Vondrick, and Antonio Torralba. 2015. Where are they looking? In Advances in Neural Information Pro… [↗](http://arxiv.org/abs/) [our card](https://md.research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/)
16. Leslie Rescorla. 1980. Overextension in early language development. Journal of Child Language, 7(2):321– 335. [find](https://scholar.google.com/scholar?q=Leslie+Rescorla%2E+1980%2E+Overextension+in+early+language+development%2E+Journal+of+Child+Language%2C+7%282%29%3A321%E2%80%93+335%2E)
17. Brandon C. Roy, Michael C. Frank, Philip DeCamp, Matthew Miller, and Deb Roy. 2015. Predicting the birth of a spoken word. Proceedings of the National Academy of Sciences, 112(41):12663–12668. [find](https://scholar.google.com/scholar?q=Brandon+C%2E+Roy%2C+Michael+C%2E+Frank%2C+Philip+DeCamp%2C+Matthew+Miller%2C+and+Deb+Roy%2E+2015%2E+Predicting+the+birth+of+a+spoken+word%2E+Proceedings+of+the+National+Academy+of+Sciences%2C+112%2841%29%3A12663%E2%80%9312668%2E)
18. Deb Roy. 2000. A computational model of word learning from multimodal sensory input. In Proceedings of the International Conference of Cognitive Modeling (ICCM2000), Groningen, Netherlands. [find](https://scholar.google.com/scholar?q=Deb+Roy%2E+2000%2E+A+computational+model+of+word+learning+from+multimodal+sensory+input%2E+In+Proceedings+of+the+International+Conference+of+Cognitive+Modeling+%28ICCM2000%29%2C+Groningen%2C+Netherlands%2E)
19. Deb Roy. 2009. New horizons in the study of child language acquisition. In Proceedings of Interspeech. [find](https://scholar.google.com/scholar?q=Deb+Roy%2E+2009%2E+New+horizons+in+the+study+of+child+language+acquisition%2E+In+Proceedings+of+Interspeech%2E)
20. Carina Silberer and Mirella Lapata. 2014. Learning grounded meaning representations with autoencoders. In Proceedings of ACL, pages 721–732, Baltimore, Maryland. [find](https://scholar.google.com/scholar?q=Carina+Silberer+and+Mirella+Lapata%2E+2014%2E+Learning+grounded+meaning+representations+with+autoencoders%2E+In+Proceedings+of+ACL%2C+pages+721%E2%80%93732%2C+Baltimore%2C+Maryland%2E)
21. Tanya Stivers and Jack Sidnell. 2005. Introduction: Multimodal interaction. Semiotica, pages 1–20. [find](https://scholar.google.com/scholar?q=Tanya+Stivers+and+Jack+Sidnell%2E+2005%2E+Introduction%3A+Multimodal+interaction%2E+Semiotica%2C+pages+1%E2%80%9320%2E)
22. John Trueswell, Tamara Medina, Alon Hafri, and Lila Gleitman. 2013. Propose but verify: Fast mapping meets cross-situational word learning. Cognitive Psychology, 66(1):126–156. [find](https://scholar.google.com/scholar?q=John+Trueswell%2C+Tamara+Medina%2C+Alon+Hafri%2C+and+Lila+Gleitman%2E+2013%2E+Propose+but+verify%3A+Fast+mapping+meets+cross%2Dsituational+word+learning%2E+Cognitive+Psychology%2C+66%281%29%3A126%E2%80%93156%2E)
23. Chen Yu and Dana H. Ballard. 2007. A unified model of early word learning: Integrating statistical and social cues. Neurocomputing, 70(13-15):2149–2165. [find](https://scholar.google.com/scholar?q=Chen+Yu+and+Dana+H%2E+Ballard%2E+2007%2E+A+unified+model+of+early+word+learning%3A+Integrating+statistical+and+social+cues%2E+Neurocomputing%2C+70%2813%2D15%29%3A2149%E2%80%932165%2E)
24. Chen Yu and Linda B. Smith. 2013. Joint attention without gaze following: human infants and their parents coordinate visual attention to objects through eye-hand coordination. PLoS ONE, 8(11). [find](https://scholar.google.com/scholar?q=Chen+Yu+and+Linda+B%2E+Smith%2E+2013%2E+Joint+attention+without+gaze+following%3A+human+infants+and+their+parents+coordinate+visual+attention+to+objects+through+eye%2Dhand+coordination%2E+PLoS+ONE%2C+8%2811%29%2E)
25. C. Yu. 2005. The emergence of links between lexical acquisition and object categorization: A computational study. Connection Science, 17(3):381–397. [find](https://scholar.google.com/scholar?q=C%2E+Yu%2E+2005%2E+The+emergence+of+links+between+lexical+acquisition+and+object+categorization%3A+A+computational+study%2E+Connection+Science%2C+17%283%29%3A381%E2%80%93397%2E)