---
title: "Jointly Learning to Parse and Perceive: Connecting Natural"
description: "This paper addresses grounded language acquisition: connecting natural-language statements to the… 28 references, cited by 49 papers here."
authors: ["Jayant Krishnamurthy", "Thomas Kollar"]
year: 2013
venue: "Transactions of the Association for Computational Linguistics"
source: "https://aclanthology.org/Q13-1016/"
doi: "10.1162/tacl_a_00220"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/krishnamurthy-2013-jointly-learning-parse-perceive-connecting/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/krishnamurthy-2013-jointly-learning-parse-perceive-connecting/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2013](https://md.research.elamur.ai/free/en/papers/2013/) / Jointly Learning to Parse and Perceive: Connecting Natural Language to the Physical World

# Jointly Learning to Parse and Perceive: Connecting Natural Language to the Physical World

**Authors:** Jayant Krishnamurthy and Thomas Kollar · **Year:** 2013 · **Venue:** [Transactions of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/tacl/) ([Transactions of the Association for Computational Linguistics (TACL) 2013](https://md.research.elamur.ai/free/en/venues/tacl/2013/)) · **Publisher:** [The MIT Press](https://md.research.elamur.ai/free/en/publishers/the-mit-press/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/Q13-1016/) · [PDF](https://aclanthology.org/Q13-1016.pdf) · [DOI 10.1162/tacl\_a\_00220](https://doi.org/10.1162/tacl_a_00220) · [OpenAlex](https://openalex.org/W2186283982)

**Citations:** 156 citations (OpenAlex, 2026-10-14) · [49](https://md.research.elamur.ai/free/en/papers/krishnamurthy-2013-jointly-learning-parse-perceive-connecting/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
This paper addresses grounded language acquisition: connecting natural-language statements to the objects they describe when the world is available only as raw perceptual features, not as a ready-made logical database. The authors propose Logical Semantics with Perception (LSP), which has three parts. A perception module builds a logical knowledge base using separate binary classifiers for one-argument categories (such as colour or object type) and two-argument relations (such as spatial "on"). A semantic parser maps the statement to a lambda-calculus logical form. A deterministic evaluator then runs that form against the knowledge base to return the denotation and grounding. The authors also design a weakly supervised training scheme. It learns only from statements paired with their denotations, with no parse annotations, word-to-object links or hand-built knowledge bases. LSP was evaluated on image scene understanding and on geographical question answering over map polygons. It clearly outperformed a comparable model that handles attributes only, and weak supervision came close to full supervision. These results show that relational language can be learned from perception with inexpensive annotation, which is useful for robotics and spatial question answering.

## Key points
- LSP produces both a **denotation** (the answer set for the whole phrase) and a **grounding** (the referents of each noun phrase). Training uses denotations only, while evaluation also checks groundings.
- Compared with Matuszek et al. (2012), LSP answers 34% more questions on geographical QA (GeoQA) and improves scene-understanding (Scene) accuracy by 16%. Unlike the G3 model, it allows one-to-many phrase–entity mappings.
- The weakly supervised variant (LSP-W) comes within 6% of the fully supervised one (LSP-F) and needs far less annotation. The category-only baseline LSP-CAT performs poorly because most examples involve relations.
- Exact-match semantic parser accuracy is 0.72 (LSP-W) and 0.73 (LSP-F) on Scene, against an upper bound of 0.79. On GeoQA it is 0.71 and 0.75, against 0.87. LSP-CAT reaches only 0.21 and 0.17.
- With a reweighted relation metric that emphasizes pairs relevant to real queries, the relation classifiers of LSP-W and LSP-F perform similarly (F1 0.98 vs 0.96 on Scene, 0.79 vs 0.76 on GeoQA). This suggests weak supervision tunes perception toward the relation instances that matter in practice.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 365 works · 158,048 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

## Cite

Cite as: Jayant Krishnamurthy and Thomas Kollar. 2013. Jointly Learning to Parse and Perceive: Connecting Natural Language to the Physical World. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{krishnamurthy2013jointly,
  title = {{Jointly Learning to Parse and Perceive: Connecting Natural Language to the Physical World}},
  author = {Jayant Krishnamurthy and Thomas Kollar},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2013},
  doi = {10.1162/tacl\_a\_00220},
  url = {https://doi.org/10.1162/tacl_a_00220}
}
```

- [← previous Squibs: What Is a Paraphrase?](https://md.research.elamur.ai/free/en/papers/bhagat-2013-squibs-what-is-paraphrase/)
- [next → Overcoming the Lack of Parallel Data in Sentence Compression](https://md.research.elamur.ai/free/en/papers/filippova-2013-overcoming-lack-parallel-data-sentence/)
- [↑ back to papers of 2013](https://md.research.elamur.ai/free/en/papers/2013/)

## References

28 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rehj Cantrell, Matthias Scheutz, Paul Schermerhorn, and Xuan Wu. 2010. Robust spoken instruction understanding for HRI. In Proceedings of the 5th ACM/IEEE International Conference on Human-Robot Interaction. [find](https://scholar.google.com/scholar?q=Rehj+Cantrell%2C+Matthias+Scheutz%2C+Paul+Schermerhorn%2C+and+Xuan+Wu%2E+2010%2E+Robust+spoken+instruction+understanding+for+HRI%2E+In+Proceedings+of+the+5th+ACM%2FIEEE+International+Conference+on+Human%2DRobot+Inter)
2. David L. Chen and Raymond J. Mooney. 2008. Learning to sportscast: a test of grounded language acquisition. In Proceedings of the 25th International Conference on Machine learning. [find](https://scholar.google.com/scholar?q=David+L%2E+Chen+and+Raymond+J%2E+Mooney%2E+2008%2E+Learning+to+sportscast%3A+a+test+of+grounded+language+acquisition%2E+In+Proceedings+of+the+25th+International+Conference+on+Machine+learning%2E)
3. David L. Chen and Raymond J. Mooney. 2011. Learning to interpret natural language navigation instructions from observations. In Proceedings of the 25th AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=David+L%2E+Chen+and+Raymond+J%2E+Mooney%2E+2011%2E+Learning+to+interpret+natural+language+navigation+instructions+from+observations%2E+In+Proceedings+of+the+25th+AAAI+Conference+on+Artificial+Intelligence%2E)
4. James Clarke, Dan Goldwasser, Ming-Wei Chang, and Dan Roth. 2010. Driving semantic parsing from the world’s response. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning. [find](https://scholar.google.com/scholar?q=James+Clarke%2C+Dan+Goldwasser%2C+Ming%2DWei+Chang%2C+and+Dan+Roth%2E+2010%2E+Driving+semantic+parsing+from+the+world%E2%80%99s+response%2E+In+Proceedings+of+the+Fourteenth+Conference+on+Computational+Natural+Language+Lear)
5. Navneet Dalal and Bill Triggs. 2005. Histograms of oriented gradients for human detection. In Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. [find](https://scholar.google.com/scholar?q=Navneet+Dalal+and+Bill+Triggs%2E+2005%2E+Histograms+of+oriented+gradients+for+human+detection%2E+In+Proceedings+of+the+2005+IEEE+Computer+Society+Conference+on+Computer+Vision+and+Pattern+Recognition%2E)
6. Juraj Dzifcak, Matthias Scheutz, Chitta Baral, and Paul Schermerhorn. 2009. What to do and how to do it: translating natural language directives into temporal and dynamic logic representation for goal management and action execution. In Proceedings of the 2009 IEEE International Conference on Roboti… [find](https://scholar.google.com/scholar?q=Juraj+Dzifcak%2C+Matthias+Scheutz%2C+Chitta+Baral%2C+and+Paul+Schermerhorn%2E+2009%2E+What+to+do+and+how+to+do+it%3A+translating+natural+language+directives+into+temporal+and+dynamic+logic+representation+for+goal)
7. Kai-yuh Hsiao, Nikolaos Mavridis, and Deb Roy. 2003. Coupling perception and simulation: Steps towards conversational robotics. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems. [find](https://scholar.google.com/scholar?q=Kai%2Dyuh+Hsiao%2C+Nikolaos+Mavridis%2C+and+Deb+Roy%2E+2003%2E+Coupling+perception+and+simulation%3A+Steps+towards+conversational+robotics%2E+In+Proceedings+of+the+IEEE%2FRSJ+International+Conference+on+Intelligent+R)
8. Rohit J. Kate and Raymond J. Mooney. 2006. [Using string-kernels for learning semantic parsers](https://md.research.elamur.ai/free/en/papers/kate-2006-string-kernels-learning-semantic-parsers/). In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the ACL.
9. Rohit J. Kate and Raymond J. Mooney. 2007. Learning language semantics from ambiguous supervision. In Proceedings of the 22nd Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Rohit+J%2E+Kate+and+Raymond+J%2E+Mooney%2E+2007%2E+Learning+language+semantics+from+ambiguous+supervision%2E+In+Proceedings+of+the+22nd+Conference+on+Artificial+Intelligence%2E)
10. Thomas Kollar, Stefanie Tellex, Deb Roy, and Nicholas Roy. 2010. Toward understanding natural language directions. In Proceedings of the 5th ACM/IEEE International Conference on Human-Robot Interaction. [find](https://scholar.google.com/scholar?q=Thomas+Kollar%2C+Stefanie+Tellex%2C+Deb+Roy%2C+and+Nicholas+Roy%2E+2010%2E+Toward+understanding+natural+language+directions%2E+In+Proceedings+of+the+5th+ACM%2FIEEE+International+Conference+on+Human%2DRobot+Interactio)
11. Jayant Krishnamurthy and Tom M. Mitchell. 2012. Weakly supervised training of semantic parsers. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. [find](https://scholar.google.com/scholar?q=Jayant+Krishnamurthy+and+Tom+M%2E+Mitchell%2E+2012%2E+Weakly+supervised+training+of+semantic+parsers%2E+In+Proceedings+of+the+2012+Joint+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+Comp)
12. Geert-Jan M. Kruijff, Hendrik Zender, Patric Jensfelt, and Henrik I. Christensen. 2007. Situated dialogue and spatial organization: What, where... and why. International Journal of Advanced Robotic Systems. [find](https://scholar.google.com/scholar?q=Geert%2DJan+M%2E+Kruijff%2C+Hendrik+Zender%2C+Patric+Jensfelt%2C+and+Henrik+I%2E+Christensen%2E+2007%2E+Situated+dialogue+and+spatial+organization%3A+What%2C+where%2E%2E%2E+and+why%2E+International+Journal+of+Advanced+Robotic+Sy)
13. Tom Kwiatkowski, Luke Zettlemoyer, Sharon Goldwater, and Mark Steedman. 2010. Inducing probabilistic CCG grammars from logical form with higher-order unification. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Tom+Kwiatkowski%2C+Luke+Zettlemoyer%2C+Sharon+Goldwater%2C+and+Mark+Steedman%2E+2010%2E+Inducing+probabilistic+CCG+grammars+from+logical+form+with+higher%2Dorder+unification%2E+In+Proceedings+of+the+2010+Conference)
14. Michael Levit and Deb Roy. 2007. Interpretation of spatial language in a map navigation task. IEEE Transactions on Systems, Man, and Cybernetics, Part B. Percy Liang, Michael I. Jordan, and Dan Klein. 2011. Learning dependency-based compositional semantics. In Proceedings of the Association for Comp… [find](https://scholar.google.com/scholar?q=Michael+Levit+and+Deb+Roy%2E+2007%2E+Interpretation+of+spatial+language+in+a+map+navigation+task%2E+IEEE+Transactions+on+Systems%2C+Man%2C+and+Cybernetics%2C+Part+B%2E+Percy+Liang%2C+Michael+I%2E+Jordan%2C+and+Dan+Klein%2E)
15. Matthew MacMahon, Brian Stankiewicz, and Benjamin Kuipers. 2006. Walk the talk: connecting language, knowledge, and action in route instructions. In Proceedings of the 21st National Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Matthew+MacMahon%2C+Brian+Stankiewicz%2C+and+Benjamin+Kuipers%2E+2006%2E+Walk+the+talk%3A+connecting+language%2C+knowledge%2C+and+action+in+route+instructions%2E+In+Proceedings+of+the+21st+National+Conference+on+Arti)
16. Cynthia Matuszek, Dieter Fox, and Karl Koscher. 2010. Following directions using statistical machine translation. In Proceedings of the 5th ACM/IEEE International Conference on Human-Robot Interaction. [find](https://scholar.google.com/scholar?q=Cynthia+Matuszek%2C+Dieter+Fox%2C+and+Karl+Koscher%2E+2010%2E+Following+directions+using+statistical+machine+translation%2E+In+Proceedings+of+the+5th+ACM%2FIEEE+International+Conference+on+Human%2DRobot+Interaction)
17. Cynthia Matuszek, Nicholas FitzGerald, Luke Zettlemoyer, Liefeng Bo, and Dieter Fox. 2012. A joint model of language and perception for grounded attribute learning. In Proceedings of the 29th International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Cynthia+Matuszek%2C+Nicholas+FitzGerald%2C+Luke+Zettlemoyer%2C+Liefeng+Bo%2C+and+Dieter+Fox%2E+2012%2E+A+joint+model+of+language+and+perception+for+grounded+attribute+learning%2E+In+Proceedings+of+the+29th+Internat)
18. Nathan D. Ratliff, J. Andrew Bagnell, and Martin A. Zinkevich. 2006. (online) subgradient methods for structured prediction. [find](https://scholar.google.com/scholar?q=Nathan+D%2E+Ratliff%2C+J%2E+Andrew+Bagnell%2C+and+Martin+A%2E+Zinkevich%2E+2006%2E+%28online%29+subgradient+methods+for+structured+prediction%2E)
19. Artificial Intelligence and Statistics. Deb Roy, Kai-Yuh Hsiao, and Nikolaos Mavridis. 2003. Conversational robots: building blocks for grounding word meaning. In Proceedings of the HLT-NAACL 2003 Workshop on Learning Word Meaning from Non-linguistic Data. [find](https://scholar.google.com/scholar?q=Artificial+Intelligence+and+Statistics%2E+Deb+Roy%2C+Kai%2DYuh+Hsiao%2C+and+Nikolaos+Mavridis%2E+2003%2E+Conversational+robots%3A+building+blocks+for+grounding+word+meaning%2E+In+Proceedings+of+the+HLT%2DNAACL+2003+Wor)
20. Nobuyuki Shimizu and Andrew Haas. 2009. Learning to follow navigational route instructions. In Proceedings of the 21st international joint conference on artifical intelligence. [find](https://scholar.google.com/scholar?q=Nobuyuki+Shimizu+and+Andrew+Haas%2E+2009%2E+Learning+to+follow+navigational+route+instructions%2E+In+Proceedings+of+the+21st+international+joint+conference+on+artifical+intelligence%2E)
21. Marjorie Skubic, Dennis Perzanowski, Sam Blisard, Alan Schultz, William Adams, Magda Bugajska, and Derek Brock. 2004. Spatial language for human-robot dialogs. IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews. [find](https://scholar.google.com/scholar?q=Marjorie+Skubic%2C+Dennis+Perzanowski%2C+Sam+Blisard%2C+Alan+Schultz%2C+William+Adams%2C+Magda+Bugajska%2C+and+Derek+Brock%2E+2004%2E+Spatial+language+for+human%2Drobot+dialogs%2E+IEEE+Transactions+on+Systems%2C+Man%2C+and+C)
22. Surface Structure and Interpretation. Ben Taskar, Carlos Guestrin, and Daphne Koller. 2004. Max-margin markov networks. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Surface+Structure+and+Interpretation%2E+Ben+Taskar%2C+Carlos+Guestrin%2C+and+Daphne+Koller%2E+2004%2E+Max%2Dmargin+markov+networks%2E+In+Advances+in+Neural+Information+Processing+Systems%2E)
23. Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew Walter, Ashis Banerjee, Seth Teller, and Nicholas Roy. 2011. Understanding natural language commands for robotic navigation and mobile manipulation. In AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Stefanie+Tellex%2C+Thomas+Kollar%2C+Steven+Dickerson%2C+Matthew+Walter%2C+Ashis+Banerjee%2C+Seth+Teller%2C+and+Nicholas+Roy%2E+2011%2E+Understanding+natural+language+commands+for+robotic+navigation+and+mobile+manipul)
24. Kristina Toutanova, Dan Klein, Christopher D. Manning, and Yoram Singer. 2003. [Feature-rich part-of-speech tagging with a cyclic dependency network](https://md.research.elamur.ai/free/en/papers/toutanova-2003-feature-rich-part-speech-tagging/). In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology.
25. Terry Winograd. 1970. Procedures as a representation for data in a computer program for understanding natural language. Ph.D. thesis, Massachusetts Institute of Technology. [find](https://scholar.google.com/scholar?q=Terry+Winograd%2E+1970%2E+Procedures+as+a+representation+for+data+in+a+computer+program+for+understanding+natural+language%2E+Ph%2ED%2E+thesis%2C+Massachusetts+Institute+of+Technology%2E)
26. Yuk Wah Wong and Raymond J. Mooney. 2006. [Learning for semantic parsing with statistical machine translation](https://md.research.elamur.ai/free/en/papers/wong-2006-learning-semantic-parsing-statistical-machine/). In Proceedings of the main conference on Human Language Technology Conference of NAACL.
27. John M. Zelle and Raymond J. Mooney. 1996. [Learning to parse database queries using inductive logic programming](https://md.research.elamur.ai/free/en/papers/zelle-1996-learning-parse-database-queries-using/). In Proceedings of the Thirteenth National Conference on Artificial Intelligence.
28. Luke S. Zettlemoyer and Michael Collins. 2005. [Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars](https://md.research.elamur.ai/free/en/papers/zettlemoyer-2012-learning-map-sentences-logical-form/). In Proceedings of the 21st Conference in Uncertainty in Artificial Intelligence.

*Updated 24 October 2026*