---
title: "Theano: new features and speed improvements (2012)"
description: "This 2012 report describes how Theano, a Python-based compiler that turns symbolic linear-algebra… 11 references, cited by 149 papers here."
authors: ["Frédéric Bastien", "Pascal Lamblin", "Razvan Pascanu", "James Bergstra", "Ian Goodfellow", "Arnaud Bergeron", "Nicolas Bouchard", "David Warde-Farley", "Yoshua Bengio"]
year: 2012
venue: "arXiv:1211.5590"
source: "https://arxiv.org/abs/1211.5590"
doi: "10.48550/arxiv.1211.5590"
arxiv: "1211.5590"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/bastien-2012-theano-new-features-speed-improvements/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/bastien-2012-theano-new-features-speed-improvements/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2012](https://md.research.elamur.ai/free/en/papers/2012/) / Theano: new features and speed improvements

# Theano: new features and speed improvements

**Authors:** [Frédéric Bastien](https://md.research.elamur.ai/free/en/authors/frederic-bastien/), Pascal Lamblin, [Razvan Pascanu](https://md.research.elamur.ai/free/en/authors/razvan-pascanu/), James Bergstra, [Ian Goodfellow](https://md.research.elamur.ai/free/en/authors/ian-goodfellow/), Arnaud Bergeron, Nicolas Bouchard, [David Warde-Farley](https://md.research.elamur.ai/free/en/authors/david-warde-farley/) and [Yoshua Bengio](https://md.research.elamur.ai/free/en/authors/yoshua-bengio/) · **Year:** 2012 · **Venue:** arXiv:1211.5590 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1211.5590) · [PDF](https://arxiv.org/pdf/1211.5590) · [DOI 10.48550/arxiv.1211.5590](https://doi.org/10.48550/arxiv.1211.5590) · [arXiv 1211.5590](https://arxiv.org/abs/1211.5590) · [OpenAlex](https://openalex.org/W1606347560)

**Citations:** 962 citations (OpenAlex, 2026-10-14) · [149](https://md.research.elamur.ai/free/en/papers/bastien-2012-theano-new-features-speed-improvements/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
This 2012 report describes how Theano, a Python-based compiler that turns symbolic linear-algebra expressions into optimized CPU or GPU code, was extended and made faster. Its authors are the Université de Montréal team. On the feature side, the main additions are the Scan Op, which represents a whole loop as a single graph node that can be differentiated, and an R-operator that provides forward-mode differentiation for Hessian-Free optimization. On the speed side, the authors target time spent outside actual computation. They introduce a VM runtime that supports lazy evaluation, plus a C version (CVM), now the default, that avoids returning to the Python interpreter between operations. More ops were also given C implementations. They then benchmark Theano against Torch7 on feed-forward networks and against RNNLM on recurrent networks. Theano matches or beats Torch7 once mini-batches are used. RNNLM leads on small recurrent models, but Theano catches up as models grow. The work showed that a flexible symbolic framework can compete with hand-tuned libraries.

## Key points
- **Scan Op:**
  - Gradients use backpropagation through time.
  - Iteration counts can be symbolic, including "do while" loops.
  - Loops can run on the GPU.
  - Compatible loops are merged, and loop-invariant work is moved out of the body.
- **R-operator:** Combining `TT.Lop` with `TT.Rop` lets Theano compute a Gauss-Newton Hessian-vector product for an RNN in two passes instead of a naive three.
- **Overhead reductions:**
  - The `cvm_nogc` linker skips freeing intermediate results, which matters most on GPU.
  - Setting `trust_input` skips input type checks and conversions.
  - Calling `f.fn(n_calls=N)` runs several iterations without a Python loop.
- **Torch7 comparison:** The benchmark is 10-class classification of 784-dimensional inputs with three models: logistic regression, one hidden layer of 500 units, and three hidden layers of 1000 units. Torch7 wins only on logistic regression without mini-batches and on GPU with batch size 1. With batches of 10 or 60, Theano is faster or equal.
- **RNNLM comparison:** The benchmark uses batch size 1 and hidden layers of 50 to 1500 units. RNNLM is faster for small networks, while Theano closes the gap for larger ones.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Université de Montréal](https://md.research.elamur.ai/free/en/universities/universite-de-montreal/) — University, Montreal, Canada · 87 works · 91,597 citations

📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🔌 Hardware and Architecture](https://md.research.elamur.ai/free/en/shelves/computer-science/hardware-and-architecture/) · 167 works

## Cite

Cite as: Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, James Bergstra, Ian Goodfellow, Arnaud Bergeron, Nicolas Bouchard, David Warde-Farley and Yoshua Bengio. 2012. Theano: new features and speed improvements. arXiv:1211.5590.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{bastien2012theano,
  title = {{Theano: new features and speed improvements}},
  author = {Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, James Bergstra, Ian Goodfellow, Arnaud Bergeron, Nicolas Bouchard, David Warde-Farley and Yoshua Bengio},
  year = {2012},
  doi = {10.48550/arxiv.1211.5590},
  eprint = {1211.5590},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1211.5590}
}
```

- [← previous SemEval-2012 Task 6: A Pilot on Semantic Textual Similarity](https://md.research.elamur.ai/free/en/papers/agirre-2012-semeval-2012-task-6-pilot/)
- [next → Learning to Map Sentences to Logical Form: Structured Classification with Probabilistic Categorial Grammars](https://md.research.elamur.ai/free/en/papers/zettlemoyer-2012-learning-map-sentences-logical-form/)
- [↑ back to papers of 2012](https://md.research.elamur.ai/free/en/papers/2012/)

## References

11 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bergstra, J., Breuleux, O., Bastien, F., Lamblin, P., Pascanu, R., Desjardins, G., Turian, J., Warde-Farley, D., and Bengio, Y. (2010). Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy). Oral Presentation. [find](https://scholar.google.com/scholar?q=Bergstra%2C+J%2E%2C+Breuleux%2C+O%2E%2C+Bastien%2C+F%2E%2C+Lamblin%2C+P%2E%2C+Pascanu%2C+R%2E%2C+Desjardins%2C+G%2E%2C+Turian%2C+J%2E%2C+Warde%2DFarley%2C+D%2E%2C+and+Bengio%2C+Y%2E+%282010%29%2E+Theano%3A+a+CPU+and+GPU+math+expression+compiler%2E+In+Proceedings+o)
2. Bergstra, J., Bastien, F., Breuleux, O., Lamblin, P., Pascanu, R., Delalleau, O., Desjardins, G., Warde-Farley, D., Goodfellow, I., Bergeron, A., and Bengio, Y. (2011). Theano: Deep learning on gpus with python. In Big Learn workshop, NIPS’11. 500 hidden units 1500 hidden units 10000 8000 6000 4000… [find](https://scholar.google.com/scholar?q=Bergstra%2C+J%2E%2C+Bastien%2C+F%2E%2C+Breuleux%2C+O%2E%2C+Lamblin%2C+P%2E%2C+Pascanu%2C+R%2E%2C+Delalleau%2C+O%2E%2C+Desjardins%2C+G%2E%2C+Warde%2DFarley%2C+D%2E%2C+Goodfellow%2C+I%2E%2C+Bergeron%2C+A%2E%2C+and+Bengio%2C+Y%2E+%282011%29%2E+Theano%3A+Deep+learning+on+gpus+w)
3. Collobert, R., Kavukcuoglu, K., and Farabet, C. (2011). Torch7: A matlab-like environment for machine learning. In BigLearn, NIPS Workshop. [find](https://scholar.google.com/scholar?q=Collobert%2C+R%2E%2C+Kavukcuoglu%2C+K%2E%2C+and+Farabet%2C+C%2E+%282011%29%2E+Torch7%3A+A+matlab%2Dlike+environment+for+machine+learning%2E+In+BigLearn%2C+NIPS+Workshop%2E)
4. Hunter, J. D. (2007). Matplotlib: A 2d graphics environment. Computing in Science and Engineering, 9(3), 90–95. [find](https://scholar.google.com/scholar?q=Hunter%2C+J%2E+D%2E+%282007%29%2E+Matplotlib%3A+A+2d+graphics+environment%2E+Computing+in+Science+and+Engineering%2C+9%283%29%2C+90%E2%80%9395%2E)
5. Jones, E., Oliphant, T., Peterson, P., et al. (2001). SciPy: Open source scientific tools for Python. [find](https://scholar.google.com/scholar?q=Jones%2C+E%2E%2C+Oliphant%2C+T%2E%2C+Peterson%2C+P%2E%2C+et+al%2E+%282001%29%2E+SciPy%3A+Open+source+scientific+tools+for+Python%2E)
6. Martens, J. and Sutskever, I. (2011). Learning recurrent neural networks with hessian-free optimization. In L. Getoor and T. Scheffer, editors, Proceedings of the 28th International Conference on Machine Learning (ICML-11), ICML ’11, pages 1033–1040, New York, NY, USA. ACM. [find](https://scholar.google.com/scholar?q=Martens%2C+J%2E+and+Sutskever%2C+I%2E+%282011%29%2E+Learning+recurrent+neural+networks+with+hessian%2Dfree+optimization%2E+In+L%2E+Getoor+and+T%2E+Scheffer%2C+editors%2C+Proceedings+of+the+28th+International+Conference+on+Mach)
7. Mikolov, T., Deoras, A., Kombrink, S., Burget, L., and Cernocky, J. (2011). Empirical evaluation and combination of advanced language modeling techniques. In Proc. 12th annual conference of the international speech communication association (INTERSPEECH 2011). [find](https://scholar.google.com/scholar?q=Mikolov%2C+T%2E%2C+Deoras%2C+A%2E%2C+Kombrink%2C+S%2E%2C+Burget%2C+L%2E%2C+and+Cernocky%2C+J%2E+%282011%29%2E+Empirical+evaluation+and+combination+of+advanced+language+modeling+techniques%2E+In+Proc%2E+12th+annual+conference+of+the+intern)
8. Oliphant, T. E. (2007). Python for scientific computing. Computing in Science and Engineering, 9, 10–20. [find](https://scholar.google.com/scholar?q=Oliphant%2C+T%2E+E%2E+%282007%29%2E+Python+for+scientific+computing%2E+Computing+in+Science+and+Engineering%2C+9%2C+10%E2%80%9320%2E)
9. Pearlmutter, B. A. (1994). Fast exact multiplication by the hessian. Neural Computation, 6, 147–160. [find](https://scholar.google.com/scholar?q=Pearlmutter%2C+B%2E+A%2E+%281994%29%2E+Fast+exact+multiplication+by+the+hessian%2E+Neural+Computation%2C+6%2C+147%E2%80%93160%2E)
10. Pérez, F. and Granger, B. E. (2007). IPython: A system for interactive scientific computing. Computing in Science and Engineering, 9(3), 21–29. [find](https://scholar.google.com/scholar?q=Pe%CC%81rez%2C+F%2E+and+Granger%2C+B%2E+E%2E+%282007%29%2E+IPython%3A+A+system+for+interactive+scientific+computing%2E+Computing+in+Science+and+Engineering%2C+9%283%29%2C+21%E2%80%9329%2E)
11. Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1986). Learning internal representations by error propagation. volume 1, chapter 8, pages 318–362. MIT Press, Cambridge. [find](https://scholar.google.com/scholar?q=Rumelhart%2C+D%2E+E%2E%2C+Hinton%2C+G%2E+E%2E%2C+and+Williams%2C+R%2E+J%2E+%281986%29%2E+Learning+internal+representations+by+error+propagation%2E+volume+1%2C+chapter+8%2C+pages+318%E2%80%93362%2E+MIT+Press%2C+Cambridge%2E)

*Updated 2 November 2026*