---
title: "Learning Continuous Semantic Representations of Symbolic"
description: "The authors propose the EqNet architecture. 24 references, cited by 17 papers here. Among them: 40."
authors: ["Miltiadis Allamanis", "Pankajan Chanthirasegaran", "Pushmeet Kohli", "Charles Sutton"]
year: 2016
venue: "arXiv:1611.01423"
source: "https://arxiv.org/abs/1611.01423"
doi: "10.48550/arxiv.1611.01423"
arxiv: "1611.01423"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/allamanis-2016-learning-continuous-semantic-representations-symbolic/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/allamanis-2016-learning-continuous-semantic-representations-symbolic/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/4/) / Learning Continuous Semantic Representations of Symbolic Expressions

# Learning Continuous Semantic Representations of Symbolic Expressions

**Authors:** [Miltiadis Allamanis](https://md.research.elamur.ai/free/en/authors/miltiadis-allamanis/), Pankajan Chanthirasegaran, [Pushmeet Kohli](https://md.research.elamur.ai/free/en/authors/pushmeet-kohli/) and [Charles Sutton](https://md.research.elamur.ai/free/en/authors/charles-sutton/) · **Year:** 2016 · **Venue:** arXiv:1611.01423 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1611.01423) · [PDF](https://arxiv.org/pdf/1611.01423) · [DOI 10.48550/arxiv.1611.01423](https://doi.org/10.48550/arxiv.1611.01423) · [arXiv 1611.01423](https://arxiv.org/abs/1611.01423) · [OpenAlex](https://openalex.org/W2555308822)

**Citations:** 40 citations (OpenAlex, 2026-10-14) · [17](https://md.research.elamur.ai/free/en/papers/allamanis-2016-learning-continuous-semantic-representations-symbolic/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The authors set out to learn how to map algebraic and logical expressions to continuous vectors so that semantically equivalent formulas receive identical or very close representations, even when they are written very differently. They call these vectors SemVec. The main difficulty is that the representation is built from the syntax tree, while a small change in notation, such as an added negation, can completely change the meaning. Ordinary TreeNNs cannot handle this: they group expressions by similarity of notation rather than by meaning. The authors propose the EqNet architecture. It has a separate two-layer MLP with a residual connection for each node type, normalizes the vectors, and is trained on equivalence classes rather than on individual pairs. EqNet also uses subexpression autoencoding, which pulls the representations of a single class into tight clusters. For evaluation, the authors assembled datasets of Boolean and polynomial expressions. On these, EqNet substantially outperforms TreeNNs and RNNs. This is a step toward combining symbolic and neural reasoning, in particular toward heuristics for symbolic solvers.

## Key points
- The paper appeared at ICML 2017 (PMLR 70). The authors are from Microsoft Research, the University of Edinburgh, DeepMind and the Alan Turing Institute.
- Two classes of tasks were created for evaluation: BOOL (Boolean expressions) and POLY (polynomials). The code and data are available at groups.inf.ed.ac.uk/cup/semvec.
- The key components of EqNet are COMBINE (a two-layer MLP with a residual connection and L2 normalization) and SubexpAe (an autoencoder that reconstructs a node's children from the parent node and vice versa). Deeper networks did not improve the results.
- Compared with the work of Zaremba et al. (2014), which covered only homogeneous polynomials in a single variable, EqNet handles a broader class of expressions and does not require search.
- Limitations in the authors' words: fixed-size vectors have limited capacity because the space of values grows exponentially. More complex procedures will likely require variable-size representations.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Edinburgh](https://md.research.elamur.ai/free/en/universities/university-of-edinburgh/) — University, Edinburgh, United Kingdom · 396 works · 82,279 citations
- [Google DeepMind](https://md.research.elamur.ai/free/en/companies/google-deepmind/) — Company, London, United Kingdom · 82 works · 84,829 citations
- [The Alan Turing Institute](https://md.research.elamur.ai/free/en/labs/the-alan-turing-institute/) — Research institute, London, United Kingdom · 35 works · 5,111 citations
- [Microsoft Research (United Kingdom)](https://md.research.elamur.ai/free/en/companies/microsoft-research-united-kingdom/) — Company, Cambridge, United Kingdom · 15 works · 3,537 citations

📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🧮 Computational Theory and Mathematics](https://md.research.elamur.ai/free/en/shelves/computer-science/computational-theory-and-mathematics/) · 496 works

## Cite

Cite as: Miltiadis Allamanis, Pankajan Chanthirasegaran, Pushmeet Kohli and Charles Sutton. 2016. Learning Continuous Semantic Representations of Symbolic Expressions. arXiv:1611.01423.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{allamanis2016learning,
  title = {{Learning Continuous Semantic Representations of Symbolic Expressions}},
  author = {Miltiadis Allamanis, Pankajan Chanthirasegaran, Pushmeet Kohli and Charles Sutton},
  year = {2016},
  doi = {10.48550/arxiv.1611.01423},
  eprint = {1611.01423},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1611.01423}
}
```

- [← previous Joint Transition-based Dependency Parsing and Disfluency Detection for Automatic Speech Recognition Texts](https://md.research.elamur.ai/free/en/papers/yoshikawa-2016-joint-transition-based-dependency-parsing/)
- [next → Learning Connective-based Word Representations for Implicit Discourse Relation Identification](https://md.research.elamur.ai/free/en/papers/braud-2016-learning-connective-based-word-representations/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/4/)

## References

24 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alemi, Alex A, Chollet, Francois, Irving, Geoffrey, Szegedy, Christian, and Urban, Josef. DeepMath – Deep sequence models for premise selection. arXiv preprint arXiv:1606.[04442, 2016](https://md.research.elamur.ai/free/en/papers/alemi-2016-deepmath-deep-sequence-models-premise/).
2. Allamanis, Miltiadis, Peng, Hao, and Sutton, Charles. [A convolutional attention network for extreme summarization of source code](https://md.research.elamur.ai/free/en/papers/allamanis-2016-convolutional-attention-network-extreme-summarization/). In ICML, 2016.
3. Ba, Jimmy Lei, Kiros, Jamie Ryan, and Hinton, Geoffrey E. [Layer normalization](https://md.research.elamur.ai/free/en/papers/ba-2016-layer-normalization/). arXiv preprint arXiv:1607.06450, 2016.
4. Bahdanau, Dzmitry, Cho, Kyunghyun, and Bengio, Yoshua. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). In ICLR, 2015.
5. Chopra, Sumit, Hadsell, Raia, and LeCun, Yann. Learning a similarity metric discriminatively, with application to face verification. In CVPR, 2005. [find](https://scholar.google.com/scholar?q=Chopra%2C+Sumit%2C+Hadsell%2C+Raia%2C+and+LeCun%2C+Yann%2E+Learning+a+similarity+metric+discriminatively%2C+with+application+to+face+verification%2E+In+CVPR%2C+2005%2E)
6. Dyer, Chris, Ballesteros, Miguel, Ling, Wang, Matthews, Austin, and Smith, Noah A. [Transition-based dependency parsing with stack long short-term memory](https://md.research.elamur.ai/free/en/papers/dyer-2015-transition-based-dependency-parsing-stack/). In ACL, 2015.
7. Graves, Alex, Wayne, Greg, and Danihelka, Ivo. Neural Turing machines. arXiv preprint arXiv:1410.5401, 2014. [find](https://scholar.google.com/scholar?q=Graves%2C+Alex%2C+Wayne%2C+Greg%2C+and+Danihelka%2C+Ivo%2E+Neural+Turing+machines%2E+arXiv+preprint+arXiv%3A1410%2E5401%2C+2014%2E)
8. Grefenstette, Edward, Hermann, Karl Moritz, Suleyman, Mustafa, and Blunsom, Phil. Learning to transduce with unbounded memory. In NIPS, 2015. [find](https://scholar.google.com/scholar?q=Grefenstette%2C+Edward%2C+Hermann%2C+Karl+Moritz%2C+Suleyman%2C+Mustafa%2C+and+Blunsom%2C+Phil%2E+Learning+to+transduce+with+unbounded+memory%2E+In+NIPS%2C+2015%2E)
9. Gruau, Frédéric, Ratajszczak, Jean-Yves, and Wiber, Gilles. A neural compiler. Theoretical Computer Science, 1995. [find](https://scholar.google.com/scholar?q=Gruau%2C+Fre%CC%81de%CC%81ric%2C+Ratajszczak%2C+Jean%2DYves%2C+and+Wiber%2C+Gilles%2E+A+neural+compiler%2E+Theoretical+Computer+Science%2C+1995%2E)
10. Joulin, Armand and Mikolov, Tomas. Inferring algorithmic patterns with stack-augmented recurrent nets. In NIPS, 2015. [find](https://scholar.google.com/scholar?q=Joulin%2C+Armand+and+Mikolov%2C+Tomas%2E+Inferring+algorithmic+patterns+with+stack%2Daugmented+recurrent+nets%2E+In+NIPS%2C+2015%2E)
11. Kaiser, Łukasz and Sutskever, Ilya. Neural GPUs learn algorithms. In ICLR, 2016. [find](https://scholar.google.com/scholar?q=Kaiser%2C+%C5%81ukasz+and+Sutskever%2C+Ilya%2E+Neural+GPUs+learn+algorithms%2E+In+ICLR%2C+2016%2E)
12. Loos, Sarah, Irving, Geoffrey, Szegedy, Christian, and Kaliszyk, Cezary. [Deep network guided proof search](https://md.research.elamur.ai/free/en/papers/loos-2017-deep-network-guided-proof-search/). arXiv preprint arXiv:1701.06972, 2017.
13. Mou, Lili, Li, Ge, Zhang, Lu, Wang, Tao, and Jin, Zhi. Convolutional neural networks over tree structures for programming language processing. In AAAI, 2016. [find](https://scholar.google.com/scholar?q=Mou%2C+Lili%2C+Li%2C+Ge%2C+Zhang%2C+Lu%2C+Wang%2C+Tao%2C+and+Jin%2C+Zhi%2E+Convolutional+neural+networks+over+tree+structures+for+programming+language+processing%2E+In+AAAI%2C+2016%2E)
14. Neelakantan, Arvind, Le, Quoc V, and Sutskever, Ilya. Neural programmer: Inducing latent programs with gradient descent. In ICLR, 2015. [find](https://scholar.google.com/scholar?q=Neelakantan%2C+Arvind%2C+Le%2C+Quoc+V%2C+and+Sutskever%2C+Ilya%2E+Neural+programmer%3A+Inducing+latent+programs+with+gradient+descent%2E+In+ICLR%2C+2015%2E)
15. Neto, João Pedro, Siegelmann, Hava T, and Costa, J Félix. Symbolic processing in neural networks. Journal of the Brazilian Computer Society, 8(3):58–70, 2003. [find](https://scholar.google.com/scholar?q=Neto%2C+Joa%CC%83o+Pedro%2C+Siegelmann%2C+Hava+T%2C+and+Costa%2C+J+Fe%CC%81lix%2E+Symbolic+processing+in+neural+networks%2E+Journal+of+the+Brazilian+Computer+Society%2C+8%283%29%3A58%E2%80%9370%2C+2003%2E)
16. Piech, Chris, Huang, Jonathan, Nguyen, Andy, Phulsuk-sombati, Mike, Sahami, Mehran, and Guibas, Leonidas J. Learning program embeddings to propagate feedback on student code. In ICML, 2015. [find](https://scholar.google.com/scholar?q=Piech%2C+Chris%2C+Huang%2C+Jonathan%2C+Nguyen%2C+Andy%2C+Phulsuk%2Dsombati%2C+Mike%2C+Sahami%2C+Mehran%2C+and+Guibas%2C+Leonidas+J%2E+Learning+program+embeddings+to+propagate+feedback+on+student+code%2E+In+ICML%2C+2015%2E)
17. Reed, Scott and de Freitas, Nando. Neural programmer-interpreters. ICLR, 2016. [find](https://scholar.google.com/scholar?q=Reed%2C+Scott+and+de+Freitas%2C+Nando%2E+Neural+programmer%2Dinterpreters%2E+ICLR%2C+2016%2E)
18. Salimans, Tim and Kingma, Diederik P. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Advances in Neural Information Processing Systems, 2016. [find](https://scholar.google.com/scholar?q=Salimans%2C+Tim+and+Kingma%2C+Diederik+P%2E+Weight+normalization%3A+A+simple+reparameterization+to+accelerate+training+of+deep+neural+networks%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2016%2E)
19. Siegelmann, Hava T. Neural programming language. In Proceedings of the 12th National Conference on Artificial Intelligence, 1994. [find](https://scholar.google.com/scholar?q=Siegelmann%2C+Hava+T%2E+Neural+programming+language%2E+In+Proceedings+of+the+12th+National+Conference+on+Artificial+Intelligence%2C+1994%2E)
20. Snoek, Jasper, Larochelle, Hugo, and Adams, Ryan P. Practical Bayesian optimization of machine learning algorithms. In NIPS, 2012. [find](https://scholar.google.com/scholar?q=Snoek%2C+Jasper%2C+Larochelle%2C+Hugo%2C+and+Adams%2C+Ryan+P%2E+Practical+Bayesian+optimization+of+machine+learning+algorithms%2E+In+NIPS%2C+2012%2E)
21. Socher, Richard, Pennington, Jeffrey, Huang, Eric H, Ng, Andrew Y, and Manning, Christopher D. [Semi-supervised recursive autoencoders for predicting sentiment distributions](https://md.research.elamur.ai/free/en/papers/socher-2011-semi-supervised-recursive-autoencoders-predicting/). In EMNLP, 2011.
22. Socher, Richard, Huval, Brody, Manning, Christopher D, and Ng, Andrew Y. [Semantic compositionality through recursive matrix-vector spaces](https://md.research.elamur.ai/free/en/papers/socher-2012-semantic-compositionality-through-recursive-matrix/). In EMNLP, 2012.
23. Socher, Richard, Perelygin, Alex, Wu, Jean Y, Chuang, Jason, Manning, Christopher D, Ng, Andrew Y, and Potts, Christopher. [Recursive deep models for semantic compositionality over a sentiment treebank](https://md.research.elamur.ai/free/en/papers/socher-2013-recursive-deep-models-semantic-compositionality/). In EMNLP, 2013.
24. Zaremba, Wojciech, Kurach, Karol, and Fergus, Rob. Learning to discover efficient mathematical identities. In Advances in Neural Information Processing Systems, pp. 1278–1286, 2014. [find](https://scholar.google.com/scholar?q=Zaremba%2C+Wojciech%2C+Kurach%2C+Karol%2C+and+Fergus%2C+Rob%2E+Learning+to+discover+efficient+mathematical+identities%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pp%2E+1278%E2%80%931286%2C+2014%2E)

*Updated 3 November 2026*