---
title: "End-to-end symbolic regression with transformers (2022)"
description: "Symbolic regression (SR) looks for a readable mathematical formula that reproduces observed data. 29 references, cited by 18 papers here."
authors: ["Pierre-Alexandre Kamienny", "Stéphane d'Ascoli", "Guillaume Lample", "François Charton"]
year: 2022
venue: "arXiv:2204.10532"
source: "https://arxiv.org/abs/2204.10532"
doi: "10.48550/arxiv.2204.10532"
arxiv: "2204.10532"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "https://arxiv.org/licenses/nonexclusive-distrib/1.0/license.html"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kamienny-2022-end-end-symbolic-regression-transformers/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kamienny-2022-end-end-symbolic-regression-transformers/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/6/) / End-to-end symbolic regression with transformers

# End-to-end symbolic regression with transformers

**Authors:** [Pierre-Alexandre Kamienny](https://md.research.elamur.ai/free/en/authors/pierre-alexandre-kamienny/), [Stéphane d'Ascoli](https://md.research.elamur.ai/free/en/authors/stephane-d-ascoli/), [Guillaume Lample](https://md.research.elamur.ai/free/en/authors/guillaume-lample/) and [François Charton](https://md.research.elamur.ai/free/en/authors/francois-charton/) · **Year:** 2022 · **Venue:** arXiv:2204.10532 · **Licence:** [arXiv — non-exclusive licence to distribute](https://arxiv.org/licenses/nonexclusive-distrib/1.0/license.html)

**Links:** [Publisher's page](https://arxiv.org/abs/2204.10532) · [PDF](https://arxiv.org/pdf/2204.10532v1) · [DOI 10.48550/arxiv.2204.10532](https://doi.org/10.48550/arxiv.2204.10532) · [arXiv 2204.10532](https://arxiv.org/abs/2204.10532) · [OpenAlex](https://openalex.org/W4224611958)

**Citations:** 9 citations (OpenAlex, 2026-10-14) · [18](https://md.research.elamur.ai/free/en/papers/kamienny-2022-end-end-symbolic-regression-transformers/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Symbolic regression (SR) looks for a readable mathematical formula that reproduces observed data. Usual pipelines first guess an expression "skeleton" and then fit its constants with BFGS. Genetic programming (GP) repeats this loop many times, so it is accurate but slow. Earlier Transformer methods predicted only skeletons, lagged behind GP and handled at most three input variables. Kamienny et al. instead train a Transformer to output the whole formula, numerical constants included. Its vocabulary combines symbolic tokens for operators and variables with tokens that encode numbers. The model is pre-trained on large synthetic data: random expression trees evaluated on inputs sampled from mixtures of Gaussian and uniform clusters, which are then whitened. The predicted constants can optionally serve as a starting point for BFGS refinement. Ablations show that this end-to-end approach outperforms skeleton prediction. On SRBench, the model approaches leading GP systems such as Operon while running orders of magnitude faster and producing simpler formulas. This makes SR practical where results are needed quickly, for example in real-time settings.

## Key points
- The model works with up to 10 input features (D ≤ 10), compared with 3 in concurrent neural SR work. Numbers are tokenized in base-10 floating-point form with four significant digits.
- The evaluation used SRBench filtered to 190 datasets: 57 black-box problems, 119 Feynman problems and 14 ODE-Strogatz problems. Each dataset was split 75%/25% into train and test data.
- On Feynman problems, the model ranks fourth in accuracy (share of problems with R² > 0.99). Its formulas are less complex than those of the top two systems, Operon and SBP-GP.
- On real-world black-box tasks, it beats classic ML baselines such as XGBoost and Random Forests, with lower inference time and an interpretable output.
- Without refinement, the end-to-end model is sensitive to label noise, and BFGS refinement restores robustness. Extrapolation stays reasonable even at test-input scale σ = 32. The code is released as a Scikit-learn compatible regressor (github.com/facebookresearch/symbolicregression).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [⚛️ Physics and Astronomy](https://md.research.elamur.ai/free/en/shelves/physics-and-astronomy/) · 70 works · [⚛️ Statistical and Nonlinear Physics](https://md.research.elamur.ai/free/en/shelves/physics-and-astronomy/statistical-and-nonlinear-physics/) · 60 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,948 works

## Cite

Cite as: Pierre-Alexandre Kamienny, Stéphane d'Ascoli, Guillaume Lample and François Charton. 2022. End-to-end symbolic regression with transformers. arXiv:2204.10532.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{kamienny2022end,
  title = {{End-to-end symbolic regression with transformers}},
  author = {Pierre-Alexandre Kamienny, Stéphane d'Ascoli, Guillaume Lample and François Charton},
  year = {2022},
  doi = {10.48550/arxiv.2204.10532},
  eprint = {2204.10532},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2204.10532}
}
```

- [← previous DuReader_retrieval: A Large-scale Chinese Benchmark for Passage Retrieval from Web Search Engine](https://md.research.elamur.ai/free/en/papers/qiu-2022-dureader-retrieval-large-scale/)
- [next → Revisiting Pre-trained Language Models and their Evaluation for Arabic Natural Language Processing](https://md.research.elamur.ai/free/en/papers/ghaddar-2022-revisiting-pre-trained-language-models/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/6/)

## References

29 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Silviu-Marian Udrescu and Max Tegmark. [AI Feynman: a Physics-Inspired Method for Symbolic Regression](https://md.research.elamur.ai/free/en/papers/udrescu-2019-ai-feynman-physics-inspired/). 2020. arXiv: 1905.11481 [physics.comp-ph].
2. M. Cranmer et al. ‘[Discovering Symbolic Models from Deep Learning with Inductive Biases](https://md.research.elamur.ai/free/en/papers/cranmer-2020-discovering-symbolic-models-deep/)’. In: ArXiv abs/2006.11287 (2020).
3. Marta Garnelo, Kai Arulkumaran and Murray Shanahan. ‘Towards deep symbolic reinforcement learning’. In: arXiv preprint arXiv:1609.05518 (2016). — [arXiv:1609.05518](https://arxiv.org/abs/1609.05518) [find](https://scholar.google.com/scholar?q=Marta+Garnelo%2C+Kai+Arulkumaran+and+Murray+Shanahan%2E+%E2%80%98Towards+deep+symbolic+reinforcement+learning%E2%80%99%2E+In%3A+arXiv+preprint+arXiv%3A1609%2E05518+%282016%29%2E+%E2%80%94+%5BarXiv%3A1609%2E05518%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1609%2E05518%29)
4. Mikel Landajuela et al. ‘Discovering symbolic policies with deep reinforcement learning’. In: International Conference on Machine Learning. PMLR. 2021, pp. 5979–5989. [find](https://scholar.google.com/scholar?q=Mikel+Landajuela+et+al%2E+%E2%80%98Discovering+symbolic+policies+with+deep+reinforcement+learning%E2%80%99%2E+In%3A+International+Conference+on+Machine+Learning%2E+PMLR%2E+2021%2C+pp%2E+5979%E2%80%935989%2E)
5. Mojtaba Valipour et al. ‘SymbolicGPT: A Generative Transformer Model for Symbolic Regression’. In: arXiv preprint arXiv:2106.14131 (2021). Luca Biggio et al. Neural Symbolic Regression that Scales. 2021. arXiv: 2106.06427 [cs.LG]. — [arXiv:2106.14131](https://arxiv.org/abs/2106.14131) [find](https://scholar.google.com/scholar?q=Mojtaba+Valipour+et+al%2E+%E2%80%98SymbolicGPT%3A+A+Generative+Transformer+Model+for+Symbolic+Regression%E2%80%99%2E+In%3A+arXiv+preprint+arXiv%3A2106%2E14131+%282021%29%2E+Luca+Biggio+et+al%2E+Neural+Symbolic+Regression+that+Scales%2E+20)
6. William La Cava et al. ‘[Contemporary symbolic regression methods and their relative performance](https://md.research.elamur.ai/free/en/papers/cava-2021-contemporary-symbolic-regression-methods/)’. In: arXiv preprint arXiv:2107.14351 (2021).
7. Nikos Aréchiga et al. ‘Accelerating Understanding of Scientific Experiments with End to End Symbolic Regression’. In: ArXiv abs/2112.04023 (2021). — [arXiv:2112.04023](https://arxiv.org/abs/2112.04023) [find](https://scholar.google.com/scholar?q=Nikos+Ar%C3%A9chiga+et+al%2E+%E2%80%98Accelerating+Understanding+of+Scientific+Experiments+with+End+to+End+Symbolic+Regression%E2%80%99%2E+In%3A+ArXiv+abs%2F2112%2E04023+%282021%29%2E+%E2%80%94+%5BarXiv%3A2112%2E04023%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2112%2E04023)
8. Silviu-Marian Udrescu and Max Tegmark. ‘Symbolic Pregression: Discovering Physical Laws from Raw Distorted Video’. In: Physical review. E 103 4-1 (2021), p. 043307. [find](https://scholar.google.com/scholar?q=Silviu%2DMarian+Udrescu+and+Max+Tegmark%2E+%E2%80%98Symbolic+Pregression%3A+Discovering+Physical+Laws+from+Raw+Distorted+Video%E2%80%99%2E+In%3A+Physical+review%2E+E+103+4%2D1+%282021%29%2C+p%2E+043307%2E)
9. Anja Butter et al. ‘Back to the Formula – LHC Edition’. In: 2021. Michael Schmidt and Hod Lipson. ‘Age-fitness pareto optimization’. In: Genetic programming theory and practice VIII. Springer, 2011, pp. 129–146. [find](https://scholar.google.com/scholar?q=Anja+Butter+et+al%2E+%E2%80%98Back+to+the+Formula+%E2%80%93+LHC+Edition%E2%80%99%2E+In%3A+2021%2E+Michael+Schmidt+and+Hod+Lipson%2E+%E2%80%98Age%2Dfitness+pareto+optimization%E2%80%99%2E+In%3A+Genetic+programming+theory+and+practice+VIII%2E+Springer%2C+2011%2C+p)
10. Michael Schmidt and Hod Lipson. ‘Distilling free-form natural laws from experimental data’. In: science 324.5923 (2009), pp. 81–85. [find](https://scholar.google.com/scholar?q=Michael+Schmidt+and+Hod+Lipson%2E+%E2%80%98Distilling+free%2Dform+natural+laws+from+experimental+data%E2%80%99%2E+In%3A+science+324%2E5923+%282009%29%2C+pp%2E+81%E2%80%9385%2E)
11. William La Cava et al. ‘Learning concise representations for regression by evolving networks of trees’. In: arXiv preprint arXiv:1807.00981 (2018). Trent McConaghy. ‘FFX: Fast, scalable, deterministic symbolic regression technology’. In: Genetic Programming Theory and Practice IX. Springer, 2011, pp… — [arXiv:1807.00981](https://arxiv.org/abs/1807.00981) [find](https://scholar.google.com/scholar?q=William+La+Cava+et+al%2E+%E2%80%98Learning+concise+representations+for+regression+by+evolving+networks+of+trees%E2%80%99%2E+In%3A+arXiv+preprint+arXiv%3A1807%2E00981+%282018%29%2E+Trent+McConaghy%2E+%E2%80%98FFX%3A+Fast%2C+scalable%2C+deterministic)
12. Marco Virgolin et al. ‘Improving model-based genetic programming for symbolic regression of small expressions’. In: Evolutionary computation 29.2 (2021), pp. 211–237. [find](https://scholar.google.com/scholar?q=Marco+Virgolin+et+al%2E+%E2%80%98Improving+model%2Dbased+genetic+programming+for+symbolic+regression+of+small+expressions%E2%80%99%2E+In%3A+Evolutionary+computation+29%2E2+%282021%29%2C+pp%2E+211%E2%80%93237%2E)
13. Fabricio Olivetti de França and Guilherme Seidyo Imai Aldeia. ‘Interaction–Transformation Evolutionary Algorithm for Symbolic Regression’. In: Evolutionary computation 29.3 (2021), pp. 367– 390. [find](https://scholar.google.com/scholar?q=Fabricio+Olivetti+de+Fran%C3%A7a+and+Guilherme+Seidyo+Imai+Aldeia%2E+%E2%80%98Interaction%E2%80%93Transformation+Evolutionary+Algorithm+for+Symbolic+Regression%E2%80%99%2E+In%3A+Evolutionary+computation+29%2E3+%282021%29%2C+pp%2E+367%E2%80%93+390%2E)
14. Ignacio Arnaldo, Krzysztof Krawiec and Una-May O’Reilly. ‘Multiple regression genetic programming’. In: Proceedings of the 2014 Annual Conference on Genetic and Evolutionary Computation. 2014, pp. 879–886. [find](https://scholar.google.com/scholar?q=Ignacio+Arnaldo%2C+Krzysztof+Krawiec+and+Una%2DMay+O%E2%80%99Reilly%2E+%E2%80%98Multiple+regression+genetic+programming%E2%80%99%2E+In%3A+Proceedings+of+the+2014+Annual+Conference+on+Genetic+and+Evolutionary+Computation%2E+2014%2C+pp%2E+879)
15. Marco Virgolin, Tanja Alderliesten and Peter A. N. Bosman. ‘Linear Scaling with and within Semantic Backpropagation-Based Genetic Programming for Symbolic Regression’. In: Proceedings of the Genetic and Evolutionary Computation Conference. GECCO ’19. Prague, Czech Republic: Association for Computing… — [doi:10.1145/3321707.3321758](https://doi.org/10.1145/3321707.3321758) [find](https://scholar.google.com/scholar?q=Marco+Virgolin%2C+Tanja+Alderliesten+and+Peter+A%2E+N%2E+Bosman%2E+%E2%80%98Linear+Scaling+with+and+within+Semantic+Backpropagation%2DBased+Genetic+Programming+for+Symbolic+Regression%E2%80%99%2E+In%3A+Proceedings+of+the+Genetic+a)
16. Michael Kommenda et al. ‘Parameter identification for symbolic regression using nonlinear least squares’. In: Genetic Programming and Evolvable Machines 21.3 (2020), pp. 471–501. doi: 10.1007/s10710-019-09371-3. url: https://doi.org/10.1007/s10710-019-09371-3. — [doi:10.1007/s10710-019-09371-3](https://doi.org/10.1007/s10710-019-09371-3) [↗](https://doi.org/10.1007/s10710-019-09371-3)
17. Brenden K Petersen et al. ‘Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients’. In: arXiv preprint arXiv:1912.04871 (2019). Georg Martius and Christoph H Lampert. ‘Extrapolation and learning equations’. In: arXiv preprint arXiv:1610.02995 (2016)… — [arXiv:1912.04871](https://arxiv.org/abs/1912.04871) [find](https://scholar.google.com/scholar?q=Brenden+K+Petersen+et+al%2E+%E2%80%98Deep+symbolic+regression%3A+Recovering+mathematical+expressions+from+data+via+risk%2Dseeking+policy+gradients%E2%80%99%2E+In%3A+arXiv+preprint+arXiv%3A1912%2E04871+%282019%29%2E+Georg+Martius+and+Chr)
18. Subham Sahoo, Christoph Lampert and Georg Martius. ‘Learning equations for extrapolation and control’. In: International Conference on Machine Learning. PMLR. 2018, pp. 4442–4450. Ashish Vaswani et al. ‘[Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/)’. In: Advances in neural information processing systems. 2017, pp. 5998…
19. Guillaume Lample and François Charton. ‘[Deep learning for symbolic mathematics](https://md.research.elamur.ai/free/en/papers/lample-2019-deep-learning-symbolic-mathematics/)’. In: arXiv preprint arXiv:1912.01412 (2019).
20. François Charton, Amaury Hayat and Guillaume Lample. ‘[Learning advanced mathematical computations from examples](https://md.research.elamur.ai/free/en/papers/charton-2020-learning-advanced-mathematical-computations-examples/)’. In: arXiv preprint arXiv:2006.06462 (2020). François Charton. ‘Linear algebra with transformers’. In: arXiv preprint arXiv:2112.01898 (2021).
21. Christopher Hahn et al. ‘[Teaching temporal logics to neural networks](https://md.research.elamur.ai/free/en/papers/hahn-2020-teaching-temporal-logics-neural-networks/)’. In: arXiv preprint arXiv:2003.04218 (2020).
22. Stanislas Polu and Ilya Sutskever. ‘[Generative language modeling for automated theorem proving](https://md.research.elamur.ai/free/en/papers/polu-2020-generative-language-modeling-automated-theorem/)’. In: arXiv preprint arXiv:2009.03393 (2020). Stéphane d’Ascoli et al. ‘Deep Symbolic Regression for Recurrent Sequences’. In: arXiv preprint arXiv:2201.04600 (2022).
23. Roger Guimerà et al. ‘A Bayesian machine scientist to aid in the solution of challenging scientific problems’. In: Science advances 6.5 (2020), eaav6971. Patrick Kidger. SympyTorch. [https://github.com/patrick-kidger/sympytorch](https://github.com/patrick-kidger/sympytorch). 2021. [↗](https://github.com/patrick-kidger/sympytorch)
24. Richard Zou Horace He. functorch: JAX-like composable function transforms for PyTorch. [https://github.com/pytorch/functorch](https://github.com/pytorch/functorch). 2021. [↗](https://github.com/pytorch/functorch)
25. Sam Wiseman and Alexander M. Rush. Sequence-to-Sequence Learning as Beam-Search Optimization. 2016. doi: 10.48550/ARXIV.1606.02960. url: https://arxiv.org/abs/1606.02960. [↗](https://arxiv.org/abs/1606.02960)
26. Jerome H Friedman. ‘Greedy function approximation: a gradient boosting machine’. In: Annals of statistics (2001), pp. 1189–1232. [find](https://scholar.google.com/scholar?q=Jerome+H+Friedman%2E+%E2%80%98Greedy+function+approximation%3A+a+gradient+boosting+machine%E2%80%99%2E+In%3A+Annals+of+statistics+%282001%29%2C+pp%2E+1189%E2%80%931232%2E)
27. Steven H. Strogatz. Nonlinear Dynamics and Chaos: With Applications to Physics, Biology, Chemistry and Engineering. Westview Press, 2000. [find](https://scholar.google.com/scholar?q=Steven+H%2E+Strogatz%2E+Nonlinear+Dynamics+and+Chaos%3A+With+Applications+to+Physics%2C+Biology%2C+Chemistry+and+Engineering%2E+Westview+Press%2C+2000%2E)
28. Ying Jin et al. Bayesian Symbolic Regression. 2020. arXiv: 1910.08892 [stat.ME]. — [arXiv:1910.08892](https://arxiv.org/abs/1910.08892) [find](https://scholar.google.com/scholar?q=Ying+Jin+et+al%2E+Bayesian+Symbolic+Regression%2E+2020%2E+arXiv%3A+1910%2E08892+%5Bstat%2EME%5D%2E+%E2%80%94+%5BarXiv%3A1910%2E08892%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1910%2E08892%29)
29. T. Nathan Mundhenk et al. Symbolic Regression via Neural-Guided Genetic Programming Population Seeding. 2021. arXiv: 2111.00053 [cs.NE]. — [arXiv:2111.00053](https://arxiv.org/abs/2111.00053) [find](https://scholar.google.com/scholar?q=T%2E+Nathan+Mundhenk+et+al%2E+Symbolic+Regression+via+Neural%2DGuided+Genetic+Programming+Population+Seeding%2E+2021%2E+arXiv%3A+2111%2E00053+%5Bcs%2ENE%5D%2E+%E2%80%94+%5BarXiv%3A2111%2E00053%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2111%2E00053%29)

*Updated 3 November 2026*