---
title: "Neural Attribute Machines for Program Generation — Amodio 2017"
description: "The authors propose Neural Attribute Machines (NAMs). 32 references, cited by 6 papers here."
authors: ["Matthew Amodio", "Swarat Chaudhuri", "Thomas W. Reps"]
year: 2017
venue: "arXiv:1705.09231"
source: "https://arxiv.org/abs/1705.09231"
doi: "10.48550/arxiv.1705.09231"
arxiv: "1705.09231"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/amodio-2017-neural-attribute-machines-program-generation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/amodio-2017-neural-attribute-machines-program-generation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2017](https://md.research.elamur.ai/free/en/papers/2017/page/5/) / Neural Attribute Machines for Program Generation

# Neural Attribute Machines for Program Generation

**Authors:** Matthew Amodio, [Swarat Chaudhuri](https://md.research.elamur.ai/free/en/authors/swarat-chaudhuri/) and Thomas W. Reps · **Year:** 2017 · **Venue:** arXiv:1705.09231 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1705.09231) · [PDF](https://arxiv.org/pdf/1705.09231) · [DOI 10.48550/arxiv.1705.09231](https://doi.org/10.48550/arxiv.1705.09231) · [arXiv 1705.09231](https://arxiv.org/abs/1705.09231) · [OpenAlex](https://openalex.org/W2619311298)

**Citations:** 17 citations (OpenAlex, 2026-10-14) · [6](https://md.research.elamur.ai/free/en/papers/amodio-2017-neural-attribute-machines-program-generation/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The authors study how neural networks generate program source code. A recurrent network trained only on examples of correct programs learns the context-sensitive rules of a language poorly. For example, a variable must be declared before it is used, and its type must fit the place where it is used. The authors propose Neural Attribute Machines (NAMs). In this model, an RNN works together with a logical automaton based on an L-attributed grammar. The automaton tracks the structural context while the model builds an abstract syntax tree (AST) top-down and left to right. This context is encoded as a fixed-length vector and appended to the input sequence. The model is also trained with a three-valued loss function. It penalizes three cases differently: a correct prediction, an incorrect but admissible one, and an incorrect and inadmissible one. The experiments used a corpus of C programs that compile. NAMs violate constraints noticeably less often than a plain RNN trained on the same data. The paper concludes that formally known language rules are better built explicitly into training than left for the network to infer from examples on its own.

## Key points
- The AST is converted into a sequence of "nonterminal–production" pairs by depth-first traversal, and moves back up the tree are marked with pop markers. Variable names are replaced with Var1, Var2… in the order in which they appear.
- Two constraints were tested separately: declared-variable (a variable must be declared before use) and typesafe-variable (variable types must be consistent).
- NAM with the context vector generates 2.8 times more correct trees than the vanilla RNN. The full NAM (context plus three-valued loss) has a slightly higher NLL because of regularization, but generates best.
- Mean NLL per token on the test data. Declared-variable: vanilla RNN — 0.253, NAM w/context — 0.188, NAM w/both — 0.208. Typesafe-variable: 1.425 / 0.779 / 0.794, respectively. The baseline model SGWC generalizes poorly: 7.178 on the test set for the type constraint.
- The closest prior work is Maddison and Tarlow. Future plans: add more C language constraints so that generated programs compile more often, and apply NAMs to other structured sequences, for example proof trees in logic.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Wisconsin–Madison](https://md.research.elamur.ai/free/en/universities/university-of-wisconsin-madison/) — University, Madison, United States · 60 works · 49,213 citations
- [Rice University](https://md.research.elamur.ai/free/en/universities/rice-university/) — University, Houston, United States · 15 works · 2,036 citations
- [University of Wisconsin System](https://md.research.elamur.ai/free/en/universities/university-of-wisconsin-system/) — University, Madison, United States · 3 works · 202 citations

Also on the byline: [GrammaTech](https://ror.org/03mwfxd89).

📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🧮 Computational Theory and Mathematics](https://md.research.elamur.ai/free/en/shelves/computer-science/computational-theory-and-mathematics/) · 496 works

## Cite

Cite as: Matthew Amodio, Swarat Chaudhuri and Thomas W. Reps. 2017. Neural Attribute Machines for Program Generation. arXiv:1705.09231.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{amodio2017neural,
  title = {{Neural Attribute Machines for Program Generation}},
  author = {Matthew Amodio, Swarat Chaudhuri and Thomas W. Reps},
  year = {2017},
  doi = {10.48550/arxiv.1705.09231},
  eprint = {1705.09231},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1705.09231}
}
```

- [← previous Fermi at SemEval-2017 Task 7: Detection and Interpretation of Homographic puns in English Language](https://md.research.elamur.ai/free/en/papers/indurthi-2017-fermi-semeval-2017-task-7/)
- [next → Preserving Distributional Information in Dialogue Act Classification](https://md.research.elamur.ai/free/en/papers/tran-2017-preserving-distributional-information-dialogue-act/)
- [↑ back to papers of 2017](https://md.research.elamur.ai/free/en/papers/2017/page/5/)

## References

32 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. arXiv preprint arXiv:1601.06759, 2016. [find](https://scholar.google.com/scholar?q=Aaron+van+den+Oord%2C+Nal+Kalchbrenner%2C+and+Koray+Kavukcuoglu%2E+Pixel+recurrent+neural+networks%2E+arXiv+preprint+arXiv%3A1601%2E06759%2C+2016%2E)
2. David Ha and Douglas Eck. A neural representation of sketch drawings. arXiv preprint arXiv:1704.03477, 2017. — [arXiv:1704.03477](https://arxiv.org/abs/1704.03477) [find](https://scholar.google.com/scholar?q=David+Ha+and+Douglas+Eck%2E+A+neural+representation+of+sketch+drawings%2E+arXiv+preprint+arXiv%3A1704%2E03477%2C+2017%2E+%E2%80%94+%5BarXiv%3A1704%2E03477%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1704%2E03477%29)
3. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. [Show and tell: A neural image caption generator](https://md.research.elamur.ai/free/en/papers/vinyals-2014-show-tell-neural-image-caption/). In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3156–3164, 2015.
4. Christian Holler, Kim Herzig, and Andreas Zeller. Fuzzing with code fragments. In USENIX Security Symposium, pages 445–458, 2012. [find](https://scholar.google.com/scholar?q=Christian+Holler%2C+Kim+Herzig%2C+and+Andreas+Zeller%2E+Fuzzing+with+code+fragments%2E+In+USENIX+Security+Symposium%2C+pages+445%E2%80%93458%2C+2012%2E)
5. Abram Hindle, Earl T Barr, Zhendong Su, Mark Gabel, and Premkumar Devanbu. On the naturalness of software. In Software Engineering (ICSE), 2012 34th International Conference on, pages 837–847. IEEE, 2012. [find](https://scholar.google.com/scholar?q=Abram+Hindle%2C+Earl+T+Barr%2C+Zhendong+Su%2C+Mark+Gabel%2C+and+Premkumar+Devanbu%2E+On+the+naturalness+of+software%2E+In+Software+Engineering+%28ICSE%29%2C+2012+34th+International+Conference+on%2C+pages+837%E2%80%93847%2E+IEEE%2C+2)
6. Veselin Raychev, Martin Vechev, and Eran Yahav. Code completion with statistical language models. In PLDI, 2014. [find](https://scholar.google.com/scholar?q=Veselin+Raychev%2C+Martin+Vechev%2C+and+Eran+Yahav%2E+Code+completion+with+statistical+language+models%2E+In+PLDI%2C+2014%2E)
7. D.E. Knuth. Semantics of context-free languages. Mathematical Systems Theory, 2(2):127–145, June 1968. [find](https://scholar.google.com/scholar?q=D%2EE%2E+Knuth%2E+Semantics+of+context%2Dfree+languages%2E+Mathematical+Systems+Theory%2C+2%282%29%3A127%E2%80%93145%2C+June+1968%2E)
8. D.E. Knuth. Semantics of context-free languages: Correction. Mathematical Systems Theory, 5(1):95–96, March 1971. [find](https://scholar.google.com/scholar?q=D%2EE%2E+Knuth%2E+Semantics+of+context%2Dfree+languages%3A+Correction%2E+Mathematical+Systems+Theory%2C+5%281%29%3A95%E2%80%9396%2C+March+1971%2E)
9. P.M. Lewis, D.J. Rosenkrantz, and R.E. Stearns. Attributed translations. J. Comput. Syst. Sci., 9(3):279–307, December 1974. [find](https://scholar.google.com/scholar?q=P%2EM%2E+Lewis%2C+D%2EJ%2E+Rosenkrantz%2C+and+R%2EE%2E+Stearns%2E+Attributed+translations%2E+J%2E+Comput%2E+Syst%2E+Sci%2E%2C+9%283%29%3A279%E2%80%93307%2C+December+1974%2E)
10. Sophocles Efremidis, Christos H. Papadimitriou, and Martha Sideris. Complexity characterizations of attribute grammar languages. Inf. and Comp., 78(3):178–186, 1988. [find](https://scholar.google.com/scholar?q=Sophocles+Efremidis%2C+Christos+H%2E+Papadimitriou%2C+and+Martha+Sideris%2E+Complexity+characterizations+of+attribute+grammar+languages%2E+Inf%2E+and+Comp%2E%2C+78%283%29%3A178%E2%80%93186%2C+1988%2E)
11. T. Reps and T. Teitelbaum. The Synthesizer Generator: A System for Constructing Language-Based Editors. Springer-Verlag, NY, 1988. [find](https://scholar.google.com/scholar?q=T%2E+Reps+and+T%2E+Teitelbaum%2E+The+Synthesizer+Generator%3A+A+System+for+Constructing+Language%2DBased+Editors%2E+Springer%2DVerlag%2C+NY%2C+1988%2E)
12. Lili Mou, Hao Peng, Ge Li, Yan Xu, Lu Zhang, and Zhi Jin. Discriminative neural sentence modeling by tree-based convolution. arXiv preprint arXiv:1504.01106, 2015. [find](https://scholar.google.com/scholar?q=Lili+Mou%2C+Hao+Peng%2C+Ge+Li%2C+Yan+Xu%2C+Lu+Zhang%2C+and+Zhi+Jin%2E+Discriminative+neural+sentence+modeling+by+tree%2Dbased+convolution%2E+arXiv+preprint+arXiv%3A1504%2E01106%2C+2015%2E)
13. Bing Shuai, Zhen Zuo, Bing Wang, and Gang Wang. Dag-recurrent neural networks for scene labeling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3620–3629, 2016. [find](https://scholar.google.com/scholar?q=Bing+Shuai%2C+Zhen+Zuo%2C+Bing+Wang%2C+and+Gang+Wang%2E+Dag%2Drecurrent+neural+networks+for+scene+labeling%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+pages+3620%E2%80%933629%2C+201)
14. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+J%C3%BCrgen+Schmidhuber%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%3A1735%E2%80%931780%2C+1997%2E)
15. Horia Margarit and Raghav Subramaniam. A batch-normalized recurrent network for sentiment classification. Advances in Neural Information Processing Systems, 2016. [find](https://scholar.google.com/scholar?q=Horia+Margarit+and+Raghav+Subramaniam%2E+A+batch%2Dnormalized+recurrent+network+for+sentiment+classification%2E+Advances+in+Neural+Information+Processing+Systems%2C+2016%2E)
16. Marijn F. Stollenga, Wonmin Byeon, Marcus Liwicki, and Juergen Schmidhuber. Parallel multi-dimensional LSTM, with application to fast biomedical volumetric image segmentation. In Advances in Neural Information Processing Systems, pages 2998–3006, 2015. [find](https://scholar.google.com/scholar?q=Marijn+F%2E+Stollenga%2C+Wonmin+Byeon%2C+Marcus+Liwicki%2C+and+Juergen+Schmidhuber%2E+Parallel+multi%2Ddimensional+LSTM%2C+with+application+to+fast+biomedical+volumetric+image+segmentation%2E+In+Advances+in+Neural+In)
17. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. [Sequence to sequence learning with neural networks](https://md.research.elamur.ai/free/en/papers/sutskever-2014-sequence-sequence-learning-neural-networks/). In Advances in neural information processing systems, pages 3104–3112, 2014.
18. Diederik Kingma and Jimmy Ba. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). arXiv preprint arXiv:1412.6980, 2014.
19. Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. [Dropout: A simple way to prevent neural networks from overfitting](https://md.research.elamur.ai/free/en/papers/srivastava-2014-dropout-simple-way-prevent/). Journal of Machine Learning Research, 15(1):1929–1958, 2014.
20. George C. Necula, Scott McPeak, Shree P. Rahul, and Westley Weimer. CIL: Intermediate language and tools for analysis and transformation of C programs. In International Conference on Compiler Construction, pages 213–228. Springer, 2002. [find](https://scholar.google.com/scholar?q=George+C%2E+Necula%2C+Scott+McPeak%2C+Shree+P%2E+Rahul%2C+and+Westley+Weimer%2E+CIL%3A+Intermediate+language+and+tools+for+analysis+and+transformation+of+C+programs%2E+In+International+Conference+on+Compiler+Construc)
21. C.J. Maddison and D. Tarlow. Structured generative models of natural source code. In ICML, 2014. [find](https://scholar.google.com/scholar?q=C%2EJ%2E+Maddison+and+D%2E+Tarlow%2E+Structured+generative+models+of+natural+source+code%2E+In+ICML%2C+2014%2E)
22. Tung Thanh Nguyen, Anh Tuan Nguyen, Hoan Anh Nguyen, and Tien N. Nguyen. A statistical semantic language model for source code. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2013, pages 532–542, New York, NY, USA, 2013. ACM. [find](https://scholar.google.com/scholar?q=Tung+Thanh+Nguyen%2C+Anh+Tuan+Nguyen%2C+Hoan+Anh+Nguyen%2C+and+Tien+N%2E+Nguyen%2E+A+statistical+semantic+language+model+for+source+code%2E+In+Proceedings+of+the+2013+9th+Joint+Meeting+on+Foundations+of+Software+)
23. Anh Tuan Nguyen and Tien N. Nguyen. Graph-based statistical language model for code. In Proceedings of the 37th International Conference on Software Engineering - Volume 1, ICSE ’15, pages 858–868, Piscataway, NJ, USA, 2015. IEEE Press. [find](https://scholar.google.com/scholar?q=Anh+Tuan+Nguyen+and+Tien+N%2E+Nguyen%2E+Graph%2Dbased+statistical+language+model+for+code%2E+In+Proceedings+of+the+37th+International+Conference+on+Software+Engineering+%2D+Volume+1%2C+ICSE+%E2%80%9915%2C+pages+858%E2%80%93868%2C+Pi)
24. Pavol Bielik, Veselin Raychev, and Martin Vechev. PHOG: Probabilistic model for code. In ICML, pages 19–24, 2016. [find](https://scholar.google.com/scholar?q=Pavol+Bielik%2C+Veselin+Raychev%2C+and+Martin+Vechev%2E+PHOG%3A+Probabilistic+model+for+code%2E+In+ICML%2C+pages+19%E2%80%9324%2C+2016%2E)
25. Miltiadis Allamanis and Charles Sutton. [Mining idioms from source code](https://md.research.elamur.ai/free/en/papers/allamanis-2014-mining-idioms-source-code/). In Proceedings of the 22Nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2014, pages 472–483, New York, NY, USA, 2014. ACM.
26. Emilio Parisotto, Abdel-rahman Mohamed, Rishabh Singh, Lihong Li, Dengyong Zhou, and Pushmeet Kohli. [Neuro-symbolic program synthesis](https://md.research.elamur.ai/free/en/papers/parisotto-2016-neuro-symbolic-program-synthesis/). arXiv preprint arXiv:1611.01855, 2016.
27. Vijayaraghavan Murali, Swarat Chaudhuri, and Chris Jermaine. Bayesian sketch learning for program synthesis. arXiv preprint arXiv:1703.05698, 2017. [find](https://scholar.google.com/scholar?q=Vijayaraghavan+Murali%2C+Swarat+Chaudhuri%2C+and+Chris+Jermaine%2E+Bayesian+sketch+learning+for+program+synthesis%2E+arXiv+preprint+arXiv%3A1703%2E05698%2C+2017%2E)
28. Matej Balog, Alexander L. Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. Deepcoder: Learning to write programs. arXiv preprint arXiv:1611.01989, 2016. [find](https://scholar.google.com/scholar?q=Matej+Balog%2C+Alexander+L%2E+Gaunt%2C+Marc+Brockschmidt%2C+Sebastian+Nowozin%2C+and+Daniel+Tarlow%2E+Deepcoder%3A+Learning+to+write+programs%2E+arXiv+preprint+arXiv%3A1611%2E01989%2C+2016%2E)
29. J.-F. Raskin and F. Servais. Visibly pushdown transducers. In ICALP, 2008. [find](https://scholar.google.com/scholar?q=J%2E%2DF%2E+Raskin+and+F%2E+Servais%2E+Visibly+pushdown+transducers%2E+In+ICALP%2C+2008%2E)
30. Edward Grefenstette, Karl Moritz Hermann, Mustafa Suleyman, and Phil Blunsom. Learning to transduce with unbounded memory. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 1828–1836. Curran Associates, Inc., 20… [find](https://scholar.google.com/scholar?q=Edward+Grefenstette%2C+Karl+Moritz+Hermann%2C+Mustafa+Suleyman%2C+and+Phil+Blunsom%2E+Learning+to+transduce+with+unbounded+memory%2E+In+C%2E+Cortes%2C+N%2E+D%2E+Lawrence%2C+D%2E+D%2E+Lee%2C+M%2E+Sugiyama%2C+and+R%2E+Garnett%2C+editors)
31. Jen-Tzung Chien and Yuan-Chu Ku. Bayesian recurrent neural network for language modeling. IEEE transactions on neural networks and learning systems, 27(2):361–374, 2016. [find](https://scholar.google.com/scholar?q=Jen%2DTzung+Chien+and+Yuan%2DChu+Ku%2E+Bayesian+recurrent+neural+network+for+language+modeling%2E+IEEE+transactions+on+neural+networks+and+learning+systems%2C+27%282%29%3A361%E2%80%93374%2C+2016%2E)
32. Jing Bai and Yan Wu. SAE-RNN deep learning for RGB-D based object recognition. In International Conference on Intelligent Computing, pages 235–240. Springer, 2014. [find](https://scholar.google.com/scholar?q=Jing+Bai+and+Yan+Wu%2E+SAE%2DRNN+deep+learning+for+RGB%2DD+based+object+recognition%2E+In+International+Conference+on+Intelligent+Computing%2C+pages+235%E2%80%93240%2E+Springer%2C+2014%2E)

*Updated 3 November 2026*