---
title: "A Causal Framework to Quantify the Robustness of Mathematical"
description: "The authors replace these with a principled behavioral-testing framework based on causal inference. 55 references, cited by 49 papers here."
authors: ["Alessandro Stolfo", "Zhijing Jin", "Kumar Shridhar", "Bernhard Schölkopf", "Mrinmaya Sachan"]
year: 2023
venue: "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2023.acl-long.32/"
doi: "10.18653/v1/2023.acl-long.32"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/stolfo-2023-causal-framework-quantify-robustness-mathematical/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/stolfo-2023-causal-framework-quantify-robustness-mathematical/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/) / A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Models

# A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Models

**Authors:** [Alessandro Stolfo](https://md.research.elamur.ai/free/en/authors/alessandro-stolfo/), [Zhijing Jin](https://md.research.elamur.ai/free/en/authors/zhijing-jin/), [Kumar Shridhar](https://md.research.elamur.ai/free/en/authors/kumar-shridhar/), [Bernhard Schölkopf](https://md.research.elamur.ai/free/en/authors/bernhard-scholkopf/) and [Mrinmaya Sachan](https://md.research.elamur.ai/free/en/authors/mrinmaya-sachan/) · **Year:** 2023 · **Venue:** [Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2023](https://md.research.elamur.ai/free/en/venues/acl/2023/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.acl-long.32/) · [PDF](https://aclanthology.org/2023.acl-long.32.pdf) · [DOI 10.18653/v1/2023.acl-long.32](https://doi.org/10.18653/v1/2023.acl-long.32) · [OpenAlex](https://openalex.org/W4385565015)

**Citations:** 16 citations (OpenAlex, 2026-10-14) · [49](https://md.research.elamur.ai/free/en/papers/stolfo-2023-causal-framework-quantify-robustness-mathematical/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Large language models perform well on math benchmarks, but it is unclear whether they truly reason or exploit superficial cues in problem wording. Earlier robustness checks relied on hand-made problem variants. The authors replace these with a principled behavioral-testing framework based on causal inference. They model a math word problem as a causal graph with several factors: irrelevant surface text (S), the arithmetic operations (O), the operands (N) and the ground-truth answer (G). An idealized, human-inspired solver should let only G drive the model's answer R. By applying do-interventions to the numbers and to the templates, they estimate total and direct causal effects (TCE, DCE) as distances between the model's answer distributions before and after the change. Effects routed through G indicate sensitivity, and direct effects that bypass G indicate brittleness. Applied to thirteen GPT variants plus LLaMA and Stanford Alpaca, the analysis shows that robustness does not grow steadily with scale. The instruction-tuned GPT-3 Davinci models are a clear exception, although their robustness weakens on harder problems. The framework gives a reusable, formal method for auditing numerical reasoning.

## Key points
- The causal graph separates the desired path (operands and operations → ground truth G → prediction R) from spurious direct paths such as S → R and N → R. These are measured as TCE and DCE using distribution-distance metrics.
- Thirteen GPT models of varying size were evaluated, both instruction-tuned and not. Three LLaMA models and Stanford Alpaca were tested separately to isolate the effects of size and instruction tuning.
- Among non-instruction-tuned models, larger ones respond more to changes in the true answer but are not necessarily more robust. The GPT-3 Davinci models (175B) improve sharply on both sensitivity and robustness.
- On three-operand problems, the direct effect of N for Davinci-003 (δ_cp) jumps from 0.17 to 0.87. This means the model changes its answer 87% of the time even when the correct result stays the same.
- Alpaca-style instruction tuning raises robustness without notably boosting overall performance.
- The study has several limitations:
  - It is English-only.
  - Numbers are limited to integers from 0 to 300.
  - It assumes the test problems were unseen during training.
  - GPT-3 API restrictions limited the number of perturbations.
- Code and data are available at github.com/alestolfo/causal-math.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [ETH Zurich](https://md.research.elamur.ai/free/en/universities/eth-zurich/) — University, Zurich, Switzerland · 36 works · 4,534 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 510 works

## Cite

Cite as: Alessandro Stolfo, Zhijing Jin, Kumar Shridhar, Bernhard Schölkopf and Mrinmaya Sachan. 2023. A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Models. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{stolfo2023causal,
  title = {{A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Models}},
  author = {Alessandro Stolfo, Zhijing Jin, Kumar Shridhar, Bernhard Schölkopf and Mrinmaya Sachan},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.32},
  url = {https://doi.org/10.18653/v1/2023.acl-long.32}
}
```

- [← previous Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning](https://md.research.elamur.ai/free/en/papers/zadouri-2023-pushing-mixture-experts-limit-extremely/)
- [next → MMSD2.0: Towards a Reliable Multi-modal Sarcasm Detection System](https://md.research.elamur.ai/free/en/papers/qin-2023-mmsd2-0-reliable-multi-modal/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/)

## References

55 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Taylor Berg-Kirkpatrick and Daniel Spokoyny. 2020. An empirical investigation of contextualized number prediction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4754–4764, Online. Association for Computational Linguistics. 9 [find](https://scholar.google.com/scholar?q=Taylor+Berg%2DKirkpatrick+and+Daniel+Spokoyny%2E+2020%2E+An+empirical+investigation+of+contextualized+number+prediction%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+Proces)
2. Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. 2022. [GPT-NeoX-20B: An open-source autoregressive language model](https://md.research.elamur.ai/free/en/papers/black-2022-gpt-neox-20b-open-source/). arXiv preprint arXiv:2204.06745. 6
3. Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large scale autoregressive language modeling with mesh-tensorflow. If you use this software, please cite it using these metadata. 6 [find](https://scholar.google.com/scholar?q=Sid+Black%2C+Leo+Gao%2C+Phil+Wang%2C+Connor+Leahy%2C+and+Stella+Biderman%2E+2021%2E+GPT%2DNeo%3A+Large+scale+autoregressive+language+modeling+with+mesh%2Dtensorflow%2E+If+you+use+this+software%2C+please+cite+it+using+these)
4. Elizabeth M. Brannon. 2005. The independence of language and mathematical reasoning. Proceedings of the National Academy of Sciences, 102(9):3177– 3178. 3 [find](https://scholar.google.com/scholar?q=Elizabeth+M%2E+Brannon%2E+2005%2E+The+independence+of+language+and+mathematical+reasoning%2E+Proceedings+of+the+National+Academy+of+Sciences%2C+102%289%29%3A3177%E2%80%93+3178%2E+3)
5. [Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri…](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
6. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). arXiv preprint arXiv:2204.02311. 1, 9
7. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168. 1, 9
8. Amir Feder, Katherine A. Keith, Emaad Manzoor, Reid Pryzant, Dhanya Sridhar, Zach Wood-Doughty, Jacob Eisenstein, Justin Grimmer, Roi Reichart, Margaret E. Roberts, Brando n M. Stewart, Victor Veitch, and Diyi Yang. 2021a. Causal inference in natural language processing: Estimation, prediction, inte… [find](https://scholar.google.com/scholar?q=Amir+Feder%2C+Katherine+A%2E+Keith%2C+Emaad+Manzoor%2C+Reid+Pryzant%2C+Dhanya+Sridhar%2C+Zach+Wood%2DDoughty%2C+Jacob+Eisenstein%2C+Justin+Grimmer%2C+Roi+Reichart%2C+Margaret+E%2E+Roberts%2C+Brando+n+M%2E+Stewart%2C+Victor+Veitch%2C)
9. Amir Feder, Nadav Oved, Uri Shalit, and Roi Reichart. 2021b. [CausaLM: Causal model explanation through counterfactual language models](https://md.research.elamur.ai/free/en/papers/feder-2021-causalm-causal-model-explanation-through/). Computational Linguistics, 47(2):333–386. 9 states. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 288–301, Punta Cana, Dominican Re…
10. Matthew Finlayson, Aaron Mueller, Sebastian Gehrmann, Stuart Shieber, Tal Linzen, and Yonatan Belinkov. 2021. Causal analysis of syntactic agreement mechanisms in neural language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internati… [find](https://scholar.google.com/scholar?q=Matthew+Finlayson%2C+Aaron+Mueller%2C+Sebastian+Gehrmann%2C+Stuart+Shieber%2C+Tal+Linzen%2C+and+Yonatan+Belinkov%2E+2021%2E+Causal+analysis+of+syntactic+agreement+mechanisms+in+neural+language+models%2E+In+Proceeding)
11. Zhijing Jin, Julius von Kügelgen, Jingwei Ni, Tejas Vaidhya, Ayush Kaushal, Mrinmaya Sachan, and Bernhard Schoelkopf. 2021c. Causal direction of data collection matters: Implications of causal and anticausal learning for NLP. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lang… [find](https://scholar.google.com/scholar?q=Zhijing+Jin%2C+Julius+von+K%C3%BCgelgen%2C+Jingwei+Ni%2C+Tejas+Vaidhya%2C+Ayush+Kaushal%2C+Mrinmaya+Sachan%2C+and+Bernhard+Schoelkopf%2E+2021c%2E+Causal+direction+of+data+collection+matters%3A+Implications+of+causal+and+ant)
12. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. 6 [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+et+al%2E+2020%2E+The+pile%3A+An+800gb+dataset+of+diverse+text+for+lan)
13. Mor Geva, Ankit Gupta, and Jonathan Berant. 2020. [Injecting numerical reasoning skills into language models](https://md.research.elamur.ai/free/en/papers/geva-2020-injecting-numerical-reasoning-skills-into/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 946–958, Online. Association for Computational Linguistics. 9
14. Zhihua Jin, Xin Jiang, Xingbo Wang, Qun Liu, Yong Wang, Xiaozhe Ren, and Huamin Qu. 2021a. Numgpt: Improving numeracy ability of generative pre-trained models. arXiv preprint arXiv:2109.03137. 5 — [arXiv:2109.03137](https://arxiv.org/abs/2109.03137) [find](https://scholar.google.com/scholar?q=Zhihua+Jin%2C+Xin+Jiang%2C+Xingbo+Wang%2C+Qun+Liu%2C+Yong+Wang%2C+Xiaozhe+Ren%2C+and+Huamin+Qu%2E+2021a%2E+Numgpt%3A+Improving+numeracy+ability+of+generative+pre%2Dtrained+models%2E+arXiv+preprint+arXiv%3A2109%2E03137%2E+5+%E2%80%94+%5Bar)
15. Zhijing Jin, Yuen Chen, Felix Leeb, Luigi Gresele, Ojasv Kamal, Zhiheng Lyu, Kevin Blin, Fernando Gonzalez Adauto, Max Kleiman-Weiner, Mrinmaya Sachan, and Bernhard Schoelkopf. 2023a. Cladder: Assessing causal reasoning in language models. 9 [find](https://scholar.google.com/scholar?q=Zhijing+Jin%2C+Yuen+Chen%2C+Felix+Leeb%2C+Luigi+Gresele%2C+Ojasv+Kamal%2C+Zhiheng+Lyu%2C+Kevin+Blin%2C+Fernando+Gonzalez+Adauto%2C+Max+Kleiman%2DWeiner%2C+Mrinmaya+Sachan%2C+and+Bernhard+Schoelkopf%2E+2023a%2E+Cladder%3A+Assessi)
16. Zhijing Jin, Amir Feder, and Kun Zhang. 2022. CausalNLP tutorial: An introduction to causality for natural language processing. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts, pages 17– 22, Abu Dubai, UAE. Association for Computational L… [find](https://scholar.google.com/scholar?q=Zhijing+Jin%2C+Amir+Feder%2C+and+Kun+Zhang%2E+2022%2E+CausalNLP+tutorial%3A+An+introduction+to+causality+for+natural+language+processing%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+in+Natural+La)
17. Zhijing Jin, Jiarui Liu, Zhiheng Lyu, Spencer Poff, Mrinmaya Sachan, Rada Mihalcea, Mona Diab, and Bernhard Schoelkopf. 2023b. Can large language models infer causation from correlation? 9 [find](https://scholar.google.com/scholar?q=Zhijing+Jin%2C+Jiarui+Liu%2C+Zhiheng+Lyu%2C+Spencer+Poff%2C+Mrinmaya+Sachan%2C+Rada+Mihalcea%2C+Mona+Diab%2C+and+Bernhard+Schoelkopf%2E+2023b%2E+Can+large+language+models+infer+causation+from+correlation%3F+9)
18. Zhijing Jin, Zhiheng Lyu, Yiwen Ding, Mrinmaya Sachan, Kun Zhang, Rada Mihalcea, and Bernhard Schoelkopf. 2023c. AI Scholars: A dataset for NLP-involved causal inference. 9 [find](https://scholar.google.com/scholar?q=Zhijing+Jin%2C+Zhiheng+Lyu%2C+Yiwen+Ding%2C+Mrinmaya+Sachan%2C+Kun+Zhang%2C+Rada+Mihalcea%2C+and+Bernhard+Schoelkopf%2E+2023c%2E+AI+Scholars%3A+A+dataset+for+NLP%2Dinvolved+causal+inference%2E+9)
19. Zhijing Jin, Zeyu Peng, Tejas Vaidhya, Bernhard Schoelkopf, and Rada Mihalcea. 2021b. Mining the cause of political decision-making from social media: A case study of COVID-19 policies across the US Divyansh Kaushik, Eduard H. Hovy, and Zachary Chase Lipton. 2020. Learning the difference that makes… [find](https://scholar.google.com/scholar?q=Zhijing+Jin%2C+Zeyu+Peng%2C+Tejas+Vaidhya%2C+Bernhard+Schoelkopf%2C+and+Rada+Mihalcea%2E+2021b%2E+Mining+the+cause+of+political+decision%2Dmaking+from+social+media%3A+A+case+study+of+COVID%2D19+policies+across+the+US+D)
20. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). arXiv preprint arXiv:2205.11916. 9
21. Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. 2016. MAWPS: A math word problem repository. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1152–1157, San D… [find](https://scholar.google.com/scholar?q=Rik+Koncel%2DKedziorski%2C+Subhro+Roy%2C+Aida+Amini%2C+Nate+Kushman%2C+and+Hannaneh+Hajishirzi%2E+2016%2E+MAWPS%3A+A+math+word+problem+repository%2E+In+Proceedings+of+the+2016+Conference+of+the+North+American+Chapter+o)
22. Zhiheng Lyu, Zhijing Jin, Justus Mattern, Rada Mihalcea, Mrinmaya Sachan, and Bernhard Schölkopf. 2023. Psychologically-inspired causal prompts. CoRR, abs/2305.01764. 9 — [arXiv:2305.01764](https://arxiv.org/abs/2305.01764) [find](https://scholar.google.com/scholar?q=Zhiheng+Lyu%2C+Zhijing+Jin%2C+Justus+Mattern%2C+Rada+Mihalcea%2C+Mrinmaya+Sachan%2C+and+Bernhard+Sch%C3%B6lkopf%2E+2023%2E+Psychologically%2Dinspired+causal+prompts%2E+CoRR%2C+abs%2F2305%2E01764%2E+9+%E2%80%94+%5BarXiv%3A2305%2E01764%5D%28https%3A%2F%2Far)
23. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems, 35:17359–17372. 2, 9 [find](https://scholar.google.com/scholar?q=Kevin+Meng%2C+David+Bau%2C+Alex+Andonian%2C+and+Yonatan+Belinkov%2E+2022%2E+Locating+and+editing+factual+associations+in+GPT%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A17359%E2%80%9317372%2E+2%2C+9)
24. Shen-yun Miao, Chao-Chun Liang, and Keh-Yih Su. 2020. A diverse corpus for evaluating and developing English math word problem solvers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 975–984, Online. Association for Computational Linguistics. 5, 6 [find](https://scholar.google.com/scholar?q=Shen%2Dyun+Miao%2C+Chao%2DChun+Liang%2C+and+Keh%2DYih+Su%2E+2020%2E+A+diverse+corpus+for+evaluating+and+developing+English+math+word+problem+solvers%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for)
25. Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, et al. 2022a. [Lila: A unified benchmark for mathematical reasoning](https://md.research.elamur.ai/free/en/papers/mishra-2022-lila-unified-benchmark-mathematical-reasoning/). arXiv preprint arXiv:2210.17517. 3
26. Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Sachdeva, Peter Clark, Chitta Baral, and Ashwin Kalyan. 2022b. NumGLUE: A suite of fundamental yet challenging mathematical reasoning tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1:… [find](https://scholar.google.com/scholar?q=Swaroop+Mishra%2C+Arindam+Mitra%2C+Neeraj+Varshney%2C+Bhavdeep+Sachdeva%2C+Peter+Clark%2C+Chitta+Baral%2C+and+Ashwin+Kalyan%2E+2022b%2E+NumGLUE%3A+A+suite+of+fundamental+yet+challenging+mathematical+reasoning+tasks%2E+In)
27. Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 383–394, Online. Association for Computational Linguistics. 9 [find](https://scholar.google.com/scholar?q=Linguistics+and+the+11th+International+Joint+Conference+on+Natural+Language+Processing+%28Volume+2%3A+Short+Papers%29%2C+pages+383%E2%80%93394%2C+Online%2E+Association+for+Computational+Linguistics%2E+9)
28. Martin M Monti, Lawrence M Parsons, and Daniel N Osherson. 2012. Thought beyond language: Neural dissociation of algebra and natural language. Psychological science, 23(8):914–922. 3 [find](https://scholar.google.com/scholar?q=Martin+M+Monti%2C+Lawrence+M+Parsons%2C+and+Daniel+N+Osherson%2E+2012%2E+Thought+beyond+language%3A+Neural+dissociation+of+algebra+and+natural+language%2E+Psychological+science%2C+23%288%29%3A914%E2%80%93922%2E+3)
29. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. 6 [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+6)
30. Jingwei Ni, Zhijing Jin, Markus Freitag, Mrinmaya Sachan, and Bernhard Schölkopf. 2022. Original or translated? A causal analysis of the impact of translationese on machine translation performance. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computation… [find](https://scholar.google.com/scholar?q=Jingwei+Ni%2C+Zhijing+Jin%2C+Markus+Freitag%2C+Mrinmaya+Sachan%2C+and+Bernhard+Sch%C3%B6lkopf%2E+2022%2E+Original+or+translated%3F+A+causal+analysis+of+the+impact+of+translationese+on+machine+translation+performance%2E+In)
31. [Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Tr…](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
32. Kuntal Kumar Pal and Chitta Baral. 2021. Investigating numeracy learning ability of a text-to-text transfer model. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3095–3101, Punta Cana, Dominican Republic. Association for Computational Linguistics. 9 [find](https://scholar.google.com/scholar?q=Kuntal+Kumar+Pal+and+Chitta+Baral%2E+2021%2E+Investigating+numeracy+learning+ability+of+a+text%2Dto%2Dtext+transfer+model%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2021%2C+pages+3095%E2%80%93)
33. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. [Are NLP models really able to solve simple math word problems?](https://md.research.elamur.ai/free/en/papers/patel-2021-are-nlp-models-really-able/) In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080–2094, Online. Associat…
34. Judea Pearl. 1995. Causal diagrams for empirical research. Biometrika, 82(4):669–688. 2, 3, 4 [find](https://scholar.google.com/scholar?q=Judea+Pearl%2E+1995%2E+Causal+diagrams+for+empirical+research%2E+Biometrika%2C+82%284%29%3A669%E2%80%93688%2E+2%2C+3%2C+4)
35. Judea Pearl. 2001. Direct and indirect effects. In UAI ’01: Proceedings of the 17th Conference in Uncertainty in Artificial Intelligence, University of Washington, Seattle, Washington, USA, August 2-5, 2001, pages 411–420. Morgan Kaufmann. 2, 4 [find](https://scholar.google.com/scholar?q=Judea+Pearl%2E+2001%2E+Direct+and+indirect+effects%2E+In+UAI+%E2%80%9901%3A+Proceedings+of+the+17th+Conference+in+Uncertainty+in+Artificial+Intelligence%2C+University+of+Washington%2C+Seattle%2C+Washington%2C+USA%2C+August+2%2D5)
36. Judea Pearl. 2009. Causality. Cambridge University Press. 9 [find](https://scholar.google.com/scholar?q=Judea+Pearl%2E+2009%2E+Causality%2E+Cambridge+University+Press%2E+9)
37. Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. 2017. Elements of causal inference: Foundations and learning algorithms. The MIT Press. 9 [find](https://scholar.google.com/scholar?q=Jonas+Peters%2C+Dominik+Janzing%2C+and+Bernhard+Sch%C3%B6lkopf%2E+2017%2E+Elements+of+causal+inference%3A+Foundations+and+learning+algorithms%2E+The+MIT+Press%2E+9)
38. Piotr Pi˛ekos, Mateusz Malinowski, and Henryk Michalewski. 2021. Measuring and improving BERT’s mathematical abilities by predicting the order of reasoning. In Proceedings of the 59th Annual Meeting of the Association for Computational Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Sin… [find](https://scholar.google.com/scholar?q=Piotr+Pi%CB%9Bekos%2C+Mateusz+Malinowski%2C+and+Henryk+Michalewski%2E+2021%2E+Measuring+and+improving+BERT%E2%80%99s+mathematical+abilities+by+predicting+the+order+of+reasoning%2E+In+Proceedings+of+the+59th+Annual+Meeting+o)
39. Margaret E Roberts, Brandon M Stewart, and Richard A Nielsen. 2020. Adjusting for confounding with text matching. American Journal of Political Science, 64(4):887–903. 9 [find](https://scholar.google.com/scholar?q=Margaret+E+Roberts%2C+Brandon+M+Stewart%2C+and+Richard+A+Nielsen%2E+2020%2E+Adjusting+for+confounding+with+text+matching%2E+American+Journal+of+Political+Science%2C+64%284%29%3A887%E2%80%93903%2E+9)
40. Mrinmaya Sachan, Kumar Dubey, and Eric Xing. 2017. From textbooks to knowledge: A case study in harvesting axiomatic knowledge from textbooks to solve geometry problems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 773–784. 1 [find](https://scholar.google.com/scholar?q=Mrinmaya+Sachan%2C+Kumar+Dubey%2C+and+Eric+Xing%2E+2017%2E+From+textbooks+to+knowledge%3A+A+case+study+in+harvesting+axiomatic+knowledge+from+textbooks+to+solve+geometry+problems%2E+In+Proceedings+of+the+2017+Con)
41. Mrinmaya Sachan, Kumar Avinava Dubey, Tom M Mitchell, Dan Roth, and Eric P Xing. 2018. Learning pipelines with limited data and domain knowledge: A study in parsing physics problems. Advances in Neural Information Processing Systems, 31. 1 [find](https://scholar.google.com/scholar?q=Mrinmaya+Sachan%2C+Kumar+Avinava+Dubey%2C+Tom+M+Mitchell%2C+Dan+Roth%2C+and+Eric+P+Xing%2E+2018%2E+Learning+pipelines+with+limited+data+and+domain+knowledge%3A+A+study+in+parsing+physics+problems%2E+Advances+in+Neura)
42. Mrinmaya Sachan and Eric Xing. 2017. Learning to solve geometry problems from natural language demonstrations in textbooks. In Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (* SEM 2017), pages 251–261. 1 [find](https://scholar.google.com/scholar?q=Mrinmaya+Sachan+and+Eric+Xing%2E+2017%2E+Learning+to+solve+geometry+problems+from+natural+language+demonstrations+in+textbooks%2E+In+Proceedings+of+the+6th+Joint+Conference+on+Lexical+and+Computational+Sema)
43. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. In NeurIPS EMC 2 Workshop. 6 [find](https://scholar.google.com/scholar?q=Victor+Sanh%2C+Lysandre+Debut%2C+Julien+Chaumond%2C+and+Thomas+Wolf%2E+2019%2E+DistilBERT%2C+a+distilled+version+of+BERT%3A+Smaller%2C+faster%2C+cheaper+and+lighter%2E+In+NeurIPS+EMC+2+Workshop%2E+6)
44. Minjoon Seo, Hannaneh Hajishirzi, Ali Farhadi, Oren Etzioni, and Clint Malcolm. 2015. [Solving geometry problems: Combining text and diagram interpretation](https://md.research.elamur.ai/free/en/papers/seo-2015-solving-geometry-problems-combining-text/). In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1466–1476, Lisbon, Portugal. Association for Co…
45. Jianhao Shen, Yichun Yin, Lin Li, Lifeng Shang, Xin Jiang, Ming Zhang, and Qun Liu. 2021. Generate & rank: A multi-task framework for math word problems. arXiv preprint arXiv:2109.03034. 9 [find](https://scholar.google.com/scholar?q=Jianhao+Shen%2C+Yichun+Yin%2C+Lin+Li%2C+Lifeng+Shang%2C+Xin+Jiang%2C+Ming+Zhang%2C+and+Qun+Liu%2E+2021%2E+Generate+%26+rank%3A+A+multi%2Dtask+framework+for+math+word+problems%2E+arXiv+preprint+arXiv%3A2109%2E03034%2E+9)
46. Daniel Spokoyny, Ivan Lee, Zhao Jin, and Taylor Berg-Kirkpatrick. 2021. Masked measurement prediction: Learning to jointly predict quantities and units from textual context. arXiv preprint arXiv:2112.08616. 9 [find](https://scholar.google.com/scholar?q=Daniel+Spokoyny%2C+Ivan+Lee%2C+Zhao+Jin%2C+and+Taylor+Berg%2DKirkpatrick%2E+2021%2E+Masked+measurement+prediction%3A+Learning+to+jointly+predict+quantities+and+units+from+textual+context%2E+arXiv+preprint+arXiv%3A2112%2E)
47. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/](https://github.com/tatsu-lab/) stanford_alpaca. 2, 8 [↗](https://github.com/tatsu-lab/)
48. Avijit Thawani, Jay Pujara, Filip Ilievski, and Pedro Szekely. 2021. Representing numbers in NLP: a survey and a vision. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644–656, Online. Associati… [find](https://scholar.google.com/scholar?q=Avijit+Thawani%2C+Jay+Pujara%2C+Filip+Ilievski%2C+and+Pedro+Szekely%2E+2021%2E+Representing+numbers+in+NLP%3A+a+survey+and+a+vision%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+of+the+Asso)
49. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971. 2, 8
50. Victor Veitch, Dhanya Sridhar, and David M. Blei. 2020. Adapting text embeddings for causal inference. In Proceedings of the Thirty-Sixth Conference on Uncertainty in Artificial Intelligence, UAI 2020, virtual online, August 3-6, 2020, volume 124 of Proceedings of Machine Learning Research, pages 91… [find](https://scholar.google.com/scholar?q=Victor+Veitch%2C+Dhanya+Sridhar%2C+and+David+M%2E+Blei%2E+2020%2E+Adapting+text+embeddings+for+causal+inference%2E+In+Proceedings+of+the+Thirty%2DSixth+Conference+on+Uncertainty+in+Artificial+Intelligence%2C+UAI+2020)
51. Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020. Investigating gender bias in language models using causal mediation analysis. Advances in Neural Information Processing Systems, 33:12388– 12401. 2, 9 [find](https://scholar.google.com/scholar?q=Jesse+Vig%2C+Sebastian+Gehrmann%2C+Yonatan+Belinkov%2C+Sharon+Qian%2C+Daniel+Nevo%2C+Yaron+Singer%2C+and+Stuart+Shieber%2E+2020%2E+Investigating+gender+bias+in+language+models+using+causal+mediation+analysis%2E+Advance)
52. Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. 2019. [Do NLP models know numbers? probing numeracy in embeddings](https://md.research.elamur.ai/free/en/papers/wallace-2019-do-nlp-models-know-numbers/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing…
53. Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 billion parameter autoregressive language model. 6 [find](https://scholar.google.com/scholar?q=Ben+Wang+and+Aran+Komatsuzaki%2E+2021%2E+GPT%2DJ%2D6B%3A+A+6+billion+parameter+autoregressive+language+model%2E+6)
54. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022a. [Emergent abilities of large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/). arXiv preprint arXiv:2206.07682. 1, 8
55. Xikun Zhang, Deepak Ramachandran, Ian Tenney, Yanai Elazar, and Dan Roth. 2020. Do language embeddings capture scales? In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4889–4896, Online. Association for Computational Linguistics. 9 [find](https://scholar.google.com/scholar?q=Xikun+Zhang%2C+Deepak+Ramachandran%2C+Ian+Tenney%2C+Yanai+Elazar%2C+and+Dan+Roth%2E+2020%2E+Do+language+embeddings+capture+scales%3F+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2020%2C+pages+4)

*Updated 24 October 2026*