---
title: "General-Purpose Question-Answering with Macaw — Tafjord 2021"
description: "The open release gives researchers a free baseline and a starting point for fine-tuning. 22 references, cited by 34 papers here."
authors: ["Oyvind Tafjord", "Peter Clark"]
year: 2021
venue: "arXiv:2109.02593"
source: "https://arxiv.org/abs/2109.02593"
doi: "10.48550/arxiv.2109.02593"
arxiv: "2109.02593"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/tafjord-2021-general-purpose-question-answering/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/tafjord-2021-general-purpose-question-answering/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/5/) / General-Purpose Question-Answering with Macaw

# General-Purpose Question-Answering with Macaw

**Authors:** [Oyvind Tafjord](https://md.research.elamur.ai/free/en/authors/oyvind-tafjord/) and [Peter Clark](https://md.research.elamur.ai/free/en/authors/peter-clark/) · **Year:** 2021 · **Venue:** arXiv:2109.02593 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2109.02593) · [PDF](https://arxiv.org/pdf/2109.02593) · [DOI 10.48550/arxiv.2109.02593](https://doi.org/10.48550/arxiv.2109.02593) · [arXiv 2109.02593](https://arxiv.org/abs/2109.02593) · [OpenAlex](https://openalex.org/W3196886373)

**Citations:** 31 citations (OpenAlex, 2026-10-14) · [34](https://md.research.elamur.ai/free/en/papers/tafjord-2021-general-purpose-question-answering/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Few strong, general-purpose question-answering systems were openly available in 2021. GPT-3 was closed, UnifiedQA mostly selected spans or multiple-choice options rather than writing free answers, and T5-CBQA lagged in the authors' tests. Tafjord and Clark address this gap with Macaw, a generative QA model built on UnifiedQA, which itself builds on T5. Macaw is trained "multi-angle": each example is split into slots (context, question, MC options, answer, explanation), and the model learns to predict any chosen subset of slots from others. This lets one model answer questions, write questions for a given answer, propose distractor options, or produce short explanations. On Challenge300, a hand-built set of 300 difficult probing questions, the 11B Macaw answers zero-shot more than 10 points (absolute) better than GPT-3, Jurassic-1, and T5-CBQA, despite being roughly sixteen times smaller than GPT-3. The authors also catalogue its failure modes, including false presuppositions, commonsense slips, and repetitive iterative outputs. The open release gives researchers a free baseline and a starting point for fine-tuning.

## Key points
- Inputs and outputs are encoded as plain text slot strings (e.g., `$answer$ ; $question$ = …`). Angles such as QMC→AE, QC→A, QAC→M and AC→QM are sampled during training, and slot order is shuffled, except that context is placed last because of T5's 512-token limit.
- Training data include multiple-choice datasets such as ARC and RACE. A Macaw-based model was reported as leading the ARC leaderboards.
- On Challenge300, Macaw (11B) beats GPT-3 (175B) by over 10% absolute. T5-CBQA (trained on Natural Questions) and AI21's Jurassic-1 also score lower.
- Known weaknesses include being fooled by false presuppositions (e.g., Zuckerberg "founding Google"), giving overly brief answers, and repeating itself in iterative or story-generation prompts. Like other pretrained models, it can produce biased or offensive answers.
- Code and model are available at https://github.com/allenai/macaw, together with all compared models' Challenge300 answers.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,621 works

🧭 Topics (1): [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · 491 works

## Cite

Cite as: Oyvind Tafjord and Peter Clark. 2021. General-Purpose Question-Answering with Macaw. arXiv:2109.02593.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{tafjord2021general,
  title = {{General-Purpose Question-Answering with Macaw}},
  author = {Oyvind Tafjord and Peter Clark},
  year = {2021},
  doi = {10.48550/arxiv.2109.02593},
  eprint = {2109.02593},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2109.02593}
}
```

- [← previous Evaluating Various Tokenizers for Arabic Text Classification](https://md.research.elamur.ai/free/en/papers/alyafeai-2021-evaluating-various-tokenizers-arabic-text/)
- [next → ParsiNLU: A Suite of Language Understanding Challenges for Persian](https://md.research.elamur.ai/free/en/papers/khashabi-2021-parsinlu-suite-language-understanding-challenges/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/5/)

## References

22 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. S. Bhakthavatsalam, D. Khashabi, T. Khot, B. D. Mishra, K. Richardson, A. Sabharwal, C. Schoenick, O. Tafjord, and P. Clark. Think you have solved direct-answer question answering? try arc-da, the direct-answer ai2 reasoning challenge. ArXiv, abs/2102.03315, 2021. — [arXiv:2102.03315](https://arxiv.org/abs/2102.03315) [find](https://scholar.google.com/scholar?q=S%2E+Bhakthavatsalam%2C+D%2E+Khashabi%2C+T%2E+Khot%2C+B%2E+D%2E+Mishra%2C+K%2E+Richardson%2C+A%2E+Sabharwal%2C+C%2E+Schoenick%2C+O%2E+Tafjord%2C+and+P%2E+Clark%2E+Think+you+have+solved+direct%2Danswer+question+answering%3F+try+arc%2Dda%2C+the+dir)
2. T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov. [Natural questions: a benchmark for question answering research](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/). Transactions of th…
3. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
4. G. Lai, Q. Xie, H. Liu, Y. Yang, and E. Hovy. [RACE: Large-scale reading comprehension dataset from examinations](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/). In EMNLP, 2017.
5. H. Levesque, E. Davis, and L. Morgenstern. The winograd schema challenge. In KR, 2011. [find](https://scholar.google.com/scholar?q=H%2E+Levesque%2C+E%2E+Davis%2C+and+L%2E+Morgenstern%2E+The+winograd+schema+challenge%2E+In+KR%2C+2011%2E)
6. P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. [Think you have solved question answering? Try ARC, the AI2 Reasoning Challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). ArXiv, abs/1803.05457, 2018.
7. T. Li, D. Khashabi, T. Khot, A. Sabharwal, and V. Srikumar. [Unqovering stereotyping biases via underspecified questions](https://md.research.elamur.ai/free/en/papers/li-2020-unqovering-stereotyping-biases-underspecified-questions/). In EMNLP, 2020. G. Marcus and E. Davis. Experiments testing gpt-3’s ability at commonsense reasoning: Results. Technical report, NYU, 2020a. (https://cs.nyu.edu/∼davise/papers/GPT…
8. P. Clark, O. Etzioni, T. Khot, A. Sabharwal, O. Tafjord, P. D. Turney, and D. Khashabi. Combining retrieval, statistics, and inference to answer elementary science questions. In AAAI, 2016. [find](https://scholar.google.com/scholar?q=P%2E+Clark%2C+O%2E+Etzioni%2C+T%2E+Khot%2C+A%2E+Sabharwal%2C+O%2E+Tafjord%2C+P%2E+D%2E+Turney%2C+and+D%2E+Khashabi%2E+Combining+retrieval%2C+statistics%2C+and+inference+to+answer+elementary+science+questions%2E+In+AAAI%2C+2016%2E)
9. G. Marcus and E. Davis. Gpt-3, bloviator: Openai’s language generator has no idea what it’s talking about. MIT Technology Review, Aug 2020b. [find](https://scholar.google.com/scholar?q=G%2E+Marcus+and+E%2E+Davis%2E+Gpt%2D3%2C+bloviator%3A+Openai%E2%80%99s+language+generator+has+no+idea+what+it%E2%80%99s+talking+about%2E+MIT+Technology+Review%2C+Aug+2020b%2E)
10. B. Dalvi, L. Huang, N. Tandon, W. tau Yih, and P. Clark. Tracking state changes in procedural text: a challenge dataset and models for process paragraph comprehension. In NAACL-HLT, 2018. [find](https://scholar.google.com/scholar?q=B%2E+Dalvi%2C+L%2E+Huang%2C+N%2E+Tandon%2C+W%2E+tau+Yih%2C+and+P%2E+Clark%2E+Tracking+state+changes+in+procedural+text%3A+a+challenge+dataset+and+models+for+process+paragraph+comprehension%2E+In+NAACL%2DHLT%2C+2018%2E)
11. A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever. Improving language understanding by generative pre-training. Technical report, OpenAI, 2018. [find](https://scholar.google.com/scholar?q=A%2E+Radford%2C+K%2E+Narasimhan%2C+T%2E+Salimans%2C+and+I%2E+Sutskever%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E+Technical+report%2C+OpenAI%2C+2018%2E)
12. E. Davis. How to write science questions that are easy for people and hard for computers. AI Mag., 37:13–22, 2016. [find](https://scholar.google.com/scholar?q=E%2E+Davis%2E+How+to+write+science+questions+that+are+easy+for+people+and+hard+for+computers%2E+AI+Mag%2E%2C+37%3A13%E2%80%9322%2C+2016%2E)
13. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). J. Mach. Learn. Res., 21:140:1–140:67, 2020.
14. P. Hase, S. Zhang, H. Xie, and M. Bansal. [Leakage-adjusted simulatability: Can models generate non-trivial explanations of their behavior in natural language?](https://md.research.elamur.ai/free/en/papers/hase-2020-leakage-adjusted-simulatability-can-models/) In EMNLP, 2020.
15. A. Roberts, C. Raffel, and N. M. Shazeer. How much knowledge can you pack into the parameters of a language model? In EMNLP, 2020. [find](https://scholar.google.com/scholar?q=A%2E+Roberts%2C+C%2E+Raffel%2C+and+N%2E+M%2E+Shazeer%2E+How+much+knowledge+can+you+pack+into+the+parameters+of+a+language+model%3F+In+EMNLP%2C+2020%2E)
16. M. Henaff, J. Weston, A. D. Szlam, A. Bordes, and Y. Le-Cun. [Tracking the world state with recurrent entity networks](https://md.research.elamur.ai/free/en/papers/henaff-2016-tracking-world-state-recurrent-entity/). In ICLR, 2016.
17. N. Tandon, B. Dalvi, J. Grus, W. tau Yih, A. Bosselut, and P. Clark. Reasoning about actions and state changes by injecting commonsense knowledge. In EMNLP, 2018. [find](https://scholar.google.com/scholar?q=N%2E+Tandon%2C+B%2E+Dalvi%2C+J%2E+Grus%2C+W%2E+tau+Yih%2C+A%2E+Bosselut%2C+and+P%2E+Clark%2E+Reasoning+about+actions+and+state+changes+by+injecting+commonsense+knowledge%2E+In+EMNLP%2C+2018%2E)
18. A. Holtzman, J. Buys, M. Forbes, and Y. Choi. [The curious case of neural text degeneration](https://md.research.elamur.ai/free/en/papers/holtzman-2019-curious-case-neural-text-degeneration/). ArXiv, abs/1904.09751, 2020.
19. J. Weston, A. Bordes, S. Chopra, and T. Mikolov. [Towards AI-Complete question answering: A set of prerequisite toy tasks](https://md.research.elamur.ai/free/en/papers/weston-2015-towards-ai-complete-question-answering/). In ICLR, 2016.
20. P. A. Jansen, E. Wainwright, S. Marmorstein, and C. T. Morrison. [Worldtree: A corpus of explanation graphs for elementary science questions supporting multi-hop inference](https://md.research.elamur.ai/free/en/papers/jansen-2018-worldtree-corpus-explanation-graphs-elementary/). In LREC, 2018. Also arXiv:1802.03052.
21. X. Zhou, M. Sap, S. Swayamdipta, N. A. Smith, and Y. Choi. [Challenges in automated debiasing for toxic language detection](https://md.research.elamur.ai/free/en/papers/zhou-2021-challenges-automated-debiasing-toxic-language/). In EACL, 2021.
22. A. Kalyan, A. Kumar, A. Chandrasekaran, A. Sabharwal, and P. Clark. How much coffee was consumed during [find](https://scholar.google.com/scholar?q=A%2E+Kalyan%2C+A%2E+Kumar%2C+A%2E+Chandrasekaran%2C+A%2E+Sabharwal%2C+and+P%2E+Clark%2E+How+much+coffee+was+consumed+during)

*Updated 3 November 2026*