---
title: "Bimodal Modelling of Source Code and Natural Language (2015)"
description: "The authors build a probabilistic model that jointly describes short natural-language phrases and… 32 references, cited by 32 papers here."
authors: ["Miltos Allamanis", "Daniel Tarlow", "Andrew Gordon", "Yi Wei"]
year: 2015
venue: "ICML 2015, PMLR 37:2123–2132"
source: "https://proceedings.mlr.press/v37/"
licence: "All rights reserved — free to read at the source"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/allamanis-2015-bimodal-modelling-source-code-natural/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/allamanis-2015-bimodal-modelling-source-code-natural/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2015](https://md.research.elamur.ai/free/en/papers/2015/page/2/) / Bimodal Modelling of Source Code and Natural Language

# Bimodal Modelling of Source Code and Natural Language

**Authors:** Miltos Allamanis, [Daniel Tarlow](https://md.research.elamur.ai/free/en/authors/daniel-tarlow/), [Andrew Gordon](https://md.research.elamur.ai/free/en/authors/andrew-gordon/) and Yi Wei · **Year:** 2015 · **Venue:** ICML 2015, PMLR 37:2123–2132 · **Licence:** All rights reserved — free to read at the source

**Links:** [Publisher's page](https://proceedings.mlr.press/v37/) · [PDF](https://proceedings.mlr.press/v37/allamanis15.pdf) · [OpenAlex](https://openalex.org/W1860267373)

**Citations:** 147 citations (OpenAlex, 2026-10-14) · [32](https://md.research.elamur.ai/free/en/papers/allamanis-2015-bimodal-modelling-source-code-natural/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The authors build a probabilistic model that jointly describes short natural-language phrases and fragments of source code. The approach combines two lines of work: statistical models of code and bimodal models of image–text pairs. The model takes the form P(C | L). Code is represented as a parse tree that is built step by step. Each choice of child nodes depends on the text query and on the part of the tree already built. The text is encoded as the average of word vectors (bag of words). The tree context is given by the preceding tokens and the types of ancestor nodes. These two representations are combined multiplicatively. Training uses noise contrastive estimation with the AdaGrad optimizer, and a PCFG serves as the noise distribution. The data consist of "short description + code" pairs, in particular StackOverflow question titles together with code from the answers. The model was tested on two retrieval tasks: finding code for a query and selecting a description for code (captioning). For unseen code, the correct description often lands among the top positions of the ranking. Taking code structure into account consistently improves the results. This is one of the early attempts to transfer multimodal methods to the code–language pair, which opens the way to semantic code search and automatic documentation.

## Key points
- Probabilities are obtained by normalizing scores over the empirical set of child tuples for each node type. If a simple frequency-based estimate is used instead of the learned score, the model reduces to an ordinary PCFG.
- The partial-tree context consists of the 10 most recent tokens and the 10 node types on the path to the root. On synthetic data, the element-wise product of the text and tree vectors (multiplicative) proved better than their sum (additive).
- The model was evaluated on the perls-captions, perls-all and SO (StackOverflow) datasets. Selecting a description for code turned out to be easier than finding code from a description. Noisier text and code degrade quality only moderately.
- The difference between the maximum and the mean cross-entropy works as a confidence measure. If answers are given for only 16% of the fragments, the MRR in description retrieval rises from 0.517 to 0.698, i.e., by roughly 35%. The Spearman correlation with MRR is ρ = 0.315 for the multiplicative model versus 0.268 for nl-only.
- Models that take code structure into account consistently and noticeably outperform the nl-only variant, which ignores it. According to the authors, without structure the correlations within the code have to be explained through the text, which makes the task harder and increases the risk of overfitting.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Edinburgh](https://md.research.elamur.ai/free/en/universities/university-of-edinburgh/) — University, Edinburgh, United Kingdom · 396 works · 82,279 citations
- [Microsoft Research (United Kingdom)](https://md.research.elamur.ai/free/en/companies/microsoft-research-united-kingdom/) — Company, Cambridge, United Kingdom · 15 works · 3,537 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,888 works

🧭 Topics (1): [💻 Code & programming languages](https://md.research.elamur.ai/free/en/topics/code-and-programming/) · 194 works

## Cite

Cite as: Miltos Allamanis, Daniel Tarlow, Andrew Gordon and Yi Wei. 2015. Bimodal Modelling of Source Code and Natural Language. ICML 2015, PMLR 37:2123–2132.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{allamanis2015bimodal,
  title = {{Bimodal Modelling of Source Code and Natural Language}},
  author = {Miltos Allamanis, Daniel Tarlow, Andrew Gordon and Yi Wei},
  booktitle = {ICML 2015, PMLR 37:2123–2132},
  year = {2015},
  url = {https://proceedings.mlr.press/v37/}
}
```

- [← previous Joint Lemmatization and Morphological Tagging with Lemming](https://md.research.elamur.ai/free/en/papers/muller-2015-joint-lemmatization-morphological-tagging-lemming/)
- [next → Order-Embeddings of Images and Language](https://md.research.elamur.ai/free/en/papers/vendrov-2015-order-embeddings-images-language/)
- [↑ back to papers of 2015](https://md.research.elamur.ai/free/en/papers/2015/page/2/)

## References

32 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Allamanis, Miltiadis and Sutton, Charles. Mining source code repositories at massive scale using language modeling. In Working Conference on Mining Software Repositories (MSR), 2013. [find](https://scholar.google.com/scholar?q=Allamanis%2C+Miltiadis+and+Sutton%2C+Charles%2E+Mining+source+code+repositories+at+massive+scale+using+language+modeling%2E+In+Working+Conference+on+Mining+Software+Repositories+%28MSR%29%2C+2013%2E)
2. Allamanis, Miltiadis and Sutton, Charles. [Mining Idioms from Source Code](https://md.research.elamur.ai/free/en/papers/allamanis-2014-mining-idioms-source-code/). In Symposium on the Foundations of Software Engineering (FSE), 2014.
3. Allamanis, Miltiadis, Barr, Earl T, Bird, Christian, and Sutton, Charles. Learning natural coding conventions. In Symposium on the Foundations of Software Engineering (FSE), 2014. [find](https://scholar.google.com/scholar?q=Allamanis%2C+Miltiadis%2C+Barr%2C+Earl+T%2C+Bird%2C+Christian%2C+and+Sutton%2C+Charles%2E+Learning+natural+coding+conventions%2E+In+Symposium+on+the+Foundations+of+Software+Engineering+%28FSE%29%2C+2014%2E)
4. Bajracharya, Sushil, Ossher, Joel, and Lopes, Cristina. Sourcerer: An infrastructure for large-scale collection and analysis of open-source code. Science of Computer Programming, 79:241–259, 2014. [find](https://scholar.google.com/scholar?q=Bajracharya%2C+Sushil%2C+Ossher%2C+Joel%2C+and+Lopes%2C+Cristina%2E+Sourcerer%3A+An+infrastructure+for+large%2Dscale+collection+and+analysis+of+open%2Dsource+code%2E+Science+of+Computer+Programming%2C+79%3A241%E2%80%93259%2C+2014%2E)
5. Duchi, John, Hazan, Elad, and Singer, Yoram. Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research, 12:2121–2159, 2011. [find](https://scholar.google.com/scholar?q=Duchi%2C+John%2C+Hazan%2C+Elad%2C+and+Singer%2C+Yoram%2E+Adaptive+subgradient+methods+for+online+learning+and+stochastic+optimization%2E+The+Journal+of+Machine+Learning+Research%2C+12%3A2121%E2%80%932159%2C+2011%2E)
6. Fang, Hao, Gupta, Saurabh, Iandola, Forrest, Srivastava, Rupesh, Deng, Li, Dollár, Piotr, Gao, Jianfeng, He, Xiaodong, Mitchell, Margaret, Platt, John, et al. [From captions to visual concepts and back](https://md.research.elamur.ai/free/en/papers/fang-2014-captions-visual-concepts-back/). arXiv preprint arXiv:1411.4952, 2014.
7. Gulwani, Sumit and Marron, Mark. Nlyze: Interactive programming by natural language for spreadsheet data analysis and manipulation. SIGMOD, 2014. [find](https://scholar.google.com/scholar?q=Gulwani%2C+Sumit+and+Marron%2C+Mark%2E+Nlyze%3A+Interactive+programming+by+natural+language+for+spreadsheet+data+analysis+and+manipulation%2E+SIGMOD%2C+2014%2E)
8. Gutmann, Michael U and Hyvärinen, Aapo. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. The Journal of Machine Learning Research, 13(1):307–361, 2012. [find](https://scholar.google.com/scholar?q=Gutmann%2C+Michael+U+and+Hyva%CC%88rinen%2C+Aapo%2E+Noise%2Dcontrastive+estimation+of+unnormalized+statistical+models%2C+with+applications+to+natural+image+statistics%2E+The+Journal+of+Machine+Learning+Research%2C+13%281%29)
9. Hindle, Abram, Barr, Earl T, Su, Zhendong, Gabel, Mark, and Devanbu, Premkumar. On the naturalness of software. In International Conference on Software Engineering (ICSE), 2012. [find](https://scholar.google.com/scholar?q=Hindle%2C+Abram%2C+Barr%2C+Earl+T%2C+Su%2C+Zhendong%2C+Gabel%2C+Mark%2C+and+Devanbu%2C+Premkumar%2E+On+the+naturalness+of+software%2E+In+International+Conference+on+Software+Engineering+%28ICSE%29%2C+2012%2E)
10. Hochreiter, Sepp and Schmidhuber, Jürgen. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. [find](https://scholar.google.com/scholar?q=Hochreiter%2C+Sepp+and+Schmidhuber%2C+Ju%CC%88rgen%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%3A1735%E2%80%931780%2C+1997%2E)
11. Karaivanov, Svetoslav, Raychev, Veselin, and Vechev, Martin. Phrase-based statistical translation of programming languages. In Proceedings of the 2014 ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming & Software, pp. 173–184. [find](https://scholar.google.com/scholar?q=Karaivanov%2C+Svetoslav%2C+Raychev%2C+Veselin%2C+and+Vechev%2C+Martin%2E+Phrase%2Dbased+statistical+translation+of+programming+languages%2E+In+Proceedings+of+the+2014+ACM+International+Symposium+on+New+Ideas%2C+New+Par)
12. Karpathy, Andrej and Fei-Fei, Li. [Deep visual-semantic alignments for generating image descriptions](https://md.research.elamur.ai/free/en/papers/karpathy-2014-deep-visual-semantic-alignments-generating/). arXiv preprint arXiv:1412.2306, 2014.
13. Keivanloo, Iman, Rilling, Juergen, and Zou, Ying. Spotting working code examples. In Proceedings of the 36th International Conference on Software Engineering, pp. 664–675. ACM, 2014. [find](https://scholar.google.com/scholar?q=Keivanloo%2C+Iman%2C+Rilling%2C+Juergen%2C+and+Zou%2C+Ying%2E+Spotting+working+code+examples%2E+In+Proceedings+of+the+36th+International+Conference+on+Software+Engineering%2C+pp%2E+664%E2%80%93675%2E+ACM%2C+2014%2E)
14. Kiros, Ryan, Zemel, R, and Salakhutdinov, Ruslan. Multimodal neural language models. In Proc. NIPS Deep Learning Workshop, 2013. [find](https://scholar.google.com/scholar?q=Kiros%2C+Ryan%2C+Zemel%2C+R%2C+and+Salakhutdinov%2C+Ruslan%2E+Multimodal+neural+language+models%2E+In+Proc%2E+NIPS+Deep+Learning+Workshop%2C+2013%2E)
15. Kushman, Nate and Barzilay, Regina. Using semantic unification to generate regular expressions from natural language. North American Chapter of the Association for Computational Linguistics (NAACL), 2013. [find](https://scholar.google.com/scholar?q=Kushman%2C+Nate+and+Barzilay%2C+Regina%2E+Using+semantic+unification+to+generate+regular+expressions+from+natural+language%2E+North+American+Chapter+of+the+Association+for+Computational+Linguistics+%28NAACL%29%2C+2)
16. Liang, Percy, Jordan, Michael I, and Klein, Dan. Learning dependency-based compositional semantics. Computational Linguistics, 39(2):389–446, 2013. [find](https://scholar.google.com/scholar?q=Liang%2C+Percy%2C+Jordan%2C+Michael+I%2C+and+Klein%2C+Dan%2E+Learning+dependency%2Dbased+compositional+semantics%2E+Computational+Linguistics%2C+39%282%29%3A389%E2%80%93446%2C+2013%2E)
17. Maddison, Chris and Tarlow, Daniel. Structured generative models of natural source code. In Proceedings of The 31st International Conference on Machine Learning, pp. 649–657, 2014. [find](https://scholar.google.com/scholar?q=Maddison%2C+Chris+and+Tarlow%2C+Daniel%2E+Structured+generative+models+of+natural+source+code%2E+In+Proceedings+of+The+31st+International+Conference+on+Machine+Learning%2C+pp%2E+649%E2%80%93657%2C+2014%2E)
18. Memisevic, Roland and Hinton, Geoffrey. Unsupervised learning of image transformations. In Computer Vision and Pattern Recognition, 2007. CVPR’07. IEEE Conference on, pp. 1–8. [find](https://scholar.google.com/scholar?q=Memisevic%2C+Roland+and+Hinton%2C+Geoffrey%2E+Unsupervised+learning+of+image+transformations%2E+In+Computer+Vision+and+Pattern+Recognition%2C+2007%2E+CVPR%E2%80%9907%2E+IEEE+Conference+on%2C+pp%2E+1%E2%80%938%2E)
19. Mitchell, Jeff and Lapata, Mirella. Composition in distributional models of semantics. Cognitive Science, 34(8): 1388–1439, 2010. [find](https://scholar.google.com/scholar?q=Mitchell%2C+Jeff+and+Lapata%2C+Mirella%2E+Composition+in+distributional+models+of+semantics%2E+Cognitive+Science%2C+34%288%29%3A+1388%E2%80%931439%2C+2010%2E)
20. Mnih, Andriy and Kavukcuoglu, Koray. Learning word embeddings efficiently with noise-contrastive estimation. In Advances in Neural Information Processing Systems, pp. 2265–2273, 2013. [find](https://scholar.google.com/scholar?q=Mnih%2C+Andriy+and+Kavukcuoglu%2C+Koray%2E+Learning+word+embeddings+efficiently+with+noise%2Dcontrastive+estimation%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pp%2E+2265%E2%80%932273%2C+2013%2E)
21. Mnih, Andriy and Teh, Yee Whye. [A fast and simple algorithm for training neural probabilistic language models](https://md.research.elamur.ai/free/en/papers/mnih-2012-fast-simple-algorithm-training-neural/). In ICML, 2012.
22. Movshovitz-Attias, Dana, Cohen, WW, and W. Cohen, William. Natural Language Models for Predicting Programming Comments. In Proc of the ACL, 2013. [find](https://scholar.google.com/scholar?q=Movshovitz%2DAttias%2C+Dana%2C+Cohen%2C+WW%2C+and+W%2E+Cohen%2C+William%2E+Natural+Language+Models+for+Predicting+Programming+Comments%2E+In+Proc+of+the+ACL%2C+2013%2E)
23. MSDN. URL [https://MSDN.microsoft.com/en-us/library/bb397906.aspx](https://MSDN.microsoft.com/en-us/library/bb397906.aspx). .NET Compiler Platform. URL https://github. com/dotnet/roslyn. Nguyen, Tung Thanh, Nguyen, Anh Tuan, Nguyen, Hoan Anh, and Nguyen, Tien N. A statistical semantic language model for source code. In Joint Meeting of the European Softwar… [↗](https://MSDN.microsoft.com/en-us/library/bb397906.aspx)
24. Raychev, Veselin, Vechev, Martin, and Krause, Andreas. Predicting program properties from big code. In Proceedings of the 42nd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, pp. 111–124. ACM, 2015. [find](https://scholar.google.com/scholar?q=Raychev%2C+Veselin%2C+Vechev%2C+Martin%2C+and+Krause%2C+Andreas%2E+Predicting+program+properties+from+big+code%2E+In+Proceedings+of+the+42nd+Annual+ACM+SIGPLAN%2DSIGACT+Symposium+on+Principles+of+Programming+Language)
25. Socher, Richard, Karpathy, Andrej, Le, Quoc V, Manning, Christopher D, and Ng, Andrew Y. [Grounded compositional semantics for finding and describing images with sentences](https://md.research.elamur.ai/free/en/papers/socher-2014-grounded-compositional-semantics-finding-describing/). Transactions of the Association for Computational Linguistics, 2:207–218, 2014.
26. Srivastava, Nitish and Salakhutdinov, Ruslan R. Multimodal learning with deep boltzmann machines. In Advances in neural information processing systems, pp. 2222–2230, 2012. [find](https://scholar.google.com/scholar?q=Srivastava%2C+Nitish+and+Salakhutdinov%2C+Ruslan+R%2E+Multimodal+learning+with+deep+boltzmann+machines%2E+In+Advances+in+neural+information+processing+systems%2C+pp%2E+2222%E2%80%932230%2C+2012%2E)
27. Sutskever, Ilya, Vinyals, Oriol, and Le, Quoc VV. [Sequence to sequence learning with neural networks](https://md.research.elamur.ai/free/en/papers/sutskever-2014-sequence-sequence-learning-neural-networks/). In Advances in Neural Information Processing Systems, pp. 3104–3112, 2014.
28. Taylor, Graham W and Hinton, Geoffrey E. Factored conditional restricted boltzmann machines for modeling motion style. In Proceedings of the 26th annual international conference on machine learning, pp. 1025–1032. ACM, 2009. [find](https://scholar.google.com/scholar?q=Taylor%2C+Graham+W+and+Hinton%2C+Geoffrey+E%2E+Factored+conditional+restricted+boltzmann+machines+for+modeling+motion+style%2E+In+Proceedings+of+the+26th+annual+international+conference+on+machine+learning%2C+p)
29. Tu, Zhaopeng, Su, Zhendong, and Devanbu, Premkumar. On the localness of software. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 269–280. ACM, 2014. [find](https://scholar.google.com/scholar?q=Tu%2C+Zhaopeng%2C+Su%2C+Zhendong%2C+and+Devanbu%2C+Premkumar%2E+On+the+localness+of+software%2E+In+Proceedings+of+the+22nd+ACM+SIGSOFT+International+Symposium+on+Foundations+of+Software+Engineering%2C+pp%2E+269%E2%80%93280%2E+AC)
30. Vinyals, Oriol, Toshev, Alexander, Bengio, Samy, and Erhan, Dumitru. [Show and tell: A neural image caption generator](https://md.research.elamur.ai/free/en/papers/vinyals-2014-show-tell-neural-image-caption/). arXiv preprint arXiv:1411.4555, 2014.
31. Zelle, John M. and Mooney, Raymond J. [Learning to parse database queries using inductive logic programming](https://md.research.elamur.ai/free/en/papers/zelle-1996-learning-parse-database-queries-using/). In Proceedings of the National Conference on Artificial Intelligence (AAAI), 1996.
32. Zettlemoyer, Luke S and Collins, Michael. [Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars](https://md.research.elamur.ai/free/en/papers/zettlemoyer-2012-learning-map-sentences-logical-form/). In Proceedings of Uncertainty in Artificial Intelligence (UAI), 2005.

*Updated 3 November 2026*