---
title: "Giving BERT a Calculator — paper by Andor et al. (2019)"
description: "Neural reading comprehension systems are good at extracting answer spans but… 82.2 F1 on CoQA. 13 references, cited by 54 papers here."
authors: ["Daniel Andor", "Luheng He", "Kenton Lee", "Emily Pitler"]
year: 2019
venue: "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)"
source: "https://aclanthology.org/D19-1609/"
doi: "10.18653/v1/d19-1609"
arxiv: "1909.00109"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/andor-2019-giving-bert-calculator-finding-operations/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/andor-2019-giving-bert-calculator-finding-operations/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/) / Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehension

# Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehension

**Authors:** [Daniel Andor](https://md.research.elamur.ai/free/en/authors/daniel-andor/), [Luheng He](https://md.research.elamur.ai/free/en/authors/luheng-he/), [Kenton Lee](https://md.research.elamur.ai/free/en/authors/kenton-lee/) and [Emily Pitler](https://md.research.elamur.ai/free/en/authors/emily-pitler/) · **Year:** 2019 · **Venue:** [Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2019](https://md.research.elamur.ai/free/en/venues/emnlp/2019/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D19-1609/) · [PDF](https://aclanthology.org/D19-1609.pdf) · [DOI 10.18653/v1/d19-1609](https://doi.org/10.18653/v1/d19-1609) · [arXiv 1909.00109](https://arxiv.org/abs/1909.00109) · [OpenAlex](https://openalex.org/W2970609357)

**Citations:** 72 citations (OpenAlex, 2026-10-14) · [54](https://md.research.elamur.ai/free/en/papers/andor-2019-giving-bert-calculator-finding-operations/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Neural reading comprehension systems are good at extracting answer spans but weak at arithmetic. On DROP, many answers must be computed instead of copied from the passage. The authors address this by letting a BERT-based reader choose a small executable program rather than generate the number itself. The program space includes:
- literals (yes/no/unknown, digits 0–9);
- text spans;
- arithmetic over numbers found in the passage (Sum, Diff, Diff100, and later Mul and Div);
- shallow two-step compositions (Sum3, Merge).

Each candidate is scored from BERT token representations, and an external calculator executes the best one. Training marginalizes over all oracle derivations that yield the gold answer, and pruning keeps the composed operations tractable. The approach gains over 33 absolute points over the previous best DROP system, NAQANet. It also learns new operations from few examples on the Illinois math word problems. The work matters because it moves computation out of the network and makes predictions partly interpretable. The operation inventory can also grow without redesigning the architecture.

## Key points
- **DROP results:** NAQANet reaches 47.77 test F1. The model pre-trained on CoQA and fine-tuned on DROP reaches 80.47 test F1, and a 6-model ensemble reaches 81.78. The oracle ceiling with all operations is 93.01 dev F1.
- **Ablations:** Diff100 adds about 9 points and Sum3 about 1.1. Merge helps multi-span answers, but they stay under-predicted because single spans outnumber them about 11:1.
- **CoQA and forgetting:** Fine-tuning on DROP alone drops CoQA dev F1 to 52.2. Joint training keeps both: 82.2 F1 on CoQA and 81.1 F1 on DROP.
- **Illinois math word problems:** Accuracy is 48.6 for the basic model, 74.0 after adding Mul/Div, and 83.2 when DROP data is also used. This exceeds prior systems at 73.3–80.1.
- **Training details:** 36% of answerable examples have several valid derivations, and the model learns to resolve this ambiguity during training. Pruning keeps the top 128 Span and Sum candidates before composing Merge and Sum3.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 492 works · 440,463 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (1): [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · 487 works

## Cite

Cite as: Daniel Andor, Luheng He, Kenton Lee and Emily Pitler. 2019. Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehension. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{andor2019giving,
  title = {{Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehension}},
  author = {Daniel Andor, Luheng He, Kenton Lee and Emily Pitler},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1609},
  eprint = {1909.00109},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d19-1609}
}
```

- [← previous Better Rewards Yield Better Summaries: Learning to Summarise Without References](https://md.research.elamur.ai/free/en/papers/bohm-2019-better-rewards-yield-better-summaries/)
- [next → Grammar Error Correction in Morphologically Rich Languages: The Case of Russian](https://md.research.elamur.ai/free/en/papers/rozovskaya-2019-grammar-error-correction-morphologically-rich/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/)

## References

13 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In NAACL.
2. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. [DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/). In NAACL.
3. Kenton Lee, Tom Kwiatkowski, Ankur P. Parikh, and Dipanjan Das. 2017. Learning recurrent span representations for extractive question answering. CoRR, abs/1611.01436. — [arXiv:1611.01436](https://arxiv.org/abs/1611.01436) [find](https://scholar.google.com/scholar?q=Kenton+Lee%2C+Tom+Kwiatkowski%2C+Ankur+P%2E+Parikh%2C+and+Dipanjan+Das%2E+2017%2E+Learning+recurrent+span+representations+for+extractive+question+answering%2E+CoRR%2C+abs%2F1611%2E01436%2E+%E2%80%94+%5BarXiv%3A1611%2E01436%5D%28https%3A%2F%2Farxi)
4. Chao-Chun Liang, Kuang-Yi Hsu, Chien-Tsung Huang, Chung-Min Li, Shen-Yu Miao, and Keh-Yih Su. 2016. A tag-based statistical english math word problem solver with understanding, reasoning and explanation. In IJCAI. [find](https://scholar.google.com/scholar?q=Chao%2DChun+Liang%2C+Kuang%2DYi+Hsu%2C+Chien%2DTsung+Huang%2C+Chung%2DMin+Li%2C+Shen%2DYu+Miao%2C+and+Keh%2DYih+Su%2E+2016%2E+A+tag%2Dbased+statistical+english+math+word+problem+solver+with+understanding%2C+reasoning+and+explanati)
5. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. [Know what you dont know: Unanswerable questions for SQuAD](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784– 789.
6. Abhilasha Ravichander, Aakanksha Naik, Carolyn Penstein Rosé, and Eduard H. Hovy. 2019. [Equate: A benchmark evaluation framework for quantitative reasoning in natural language inference](https://md.research.elamur.ai/free/en/papers/ravichander-2019-equate-benchmark-evaluation-framework-quantitative/). CoRR, abs/1901.03735.
7. Siva Reddy, Danqi Chen, and Christopher D. Manning. 2018. [CoQA: A conversational question answering challenge](https://md.research.elamur.ai/free/en/papers/reddy-2019-coqa-conversational-question-answering-challenge/). CoRR, abs/1808.07042.
8. Subhro Roy and Dan Roth. 2015. [Solving general arithmetic word problems](https://md.research.elamur.ai/free/en/papers/roy-2015-solving-general-arithmetic-word-problems/). In EMNLP.
9. Subhro Roy, Tim Vieira, and Dan Roth. 2015. Reasoning about quantities in natural language. Transactions of the Association for Computational Linguistics, 3:1–13. [find](https://scholar.google.com/scholar?q=Subhro+Roy%2C+Tim+Vieira%2C+and+Dan+Roth%2E+2015%2E+Reasoning+about+quantities+in+natural+language%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+3%3A1%E2%80%9313%2E)
10. David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. 2019. [Analysing mathematical reasoning abilities of neural models](https://md.research.elamur.ai/free/en/papers/saxton-2019-analysing-mathematical-reasoning-abilities-neural/). CoRR, abs/1904.01557.
11. Lei Wang, Dongxiang Zhang, Lianli Gao, Jingkuan Song, Long Guo, and Heng Tao Shen. 2018. Math-DQN: Solving arithmetic word problems via deep reinforcement learning. In AAAI. [find](https://scholar.google.com/scholar?q=Lei+Wang%2C+Dongxiang+Zhang%2C+Lianli+Gao%2C+Jingkuan+Song%2C+Long+Guo%2C+and+Heng+Tao+Shen%2E+2018%2E+Math%2DDQN%3A+Solving+arithmetic+word+problems+via+deep+reinforcement+learning%2E+In+AAAI%2E)
12. Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V Le. 2018. [QANet: Combining local convolution with global self-attention for reading comprehension](https://md.research.elamur.ai/free/en/papers/yu-2018-qanet-combining-local-convolution-global/). Proceedings of ICLR.
13. Dongxiang Zhang, Lei Wang, Nuo Xu, Bing Tian Dai, and Heng Tao Shen. 2018. The gap of semantic parsing: A survey on automatic math word problem solvers. IEEE transactions on pattern analysis and machine intelligence. [find](https://scholar.google.com/scholar?q=Dongxiang+Zhang%2C+Lei+Wang%2C+Nuo+Xu%2C+Bing+Tian+Dai%2C+and+Heng+Tao+Shen%2E+2018%2E+The+gap+of+semantic+parsing%3A+A+survey+on+automatic+math+word+problem+solvers%2E+IEEE+transactions+on+pattern+analysis+and+machi)

*Updated 2 November 2026*