---
title: "CODAH — paper by Chen et al. (2019) — summary and ideas"
description: "To address this, they build CODAH, a harder companion set in the same format: a… 65% accuracy. 9 references, cited by 48 papers here."
authors: ["Michael Chen", "Mike D'Arcy", "Alisa Liu", "Jared Fernandez", "Doug Downey"]
year: 2019
venue: "Proceedings of the 3rd Workshop on Evaluating Vector Space Representations for NLP"
source: "https://aclanthology.org/W19-2008/"
doi: "10.18653/v1/w19-2008"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chen-2019-codah-adversarially-authored-question-answering/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chen-2019-codah-adversarially-authored-question-answering/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/) / CODAH: An Adversarially-Authored Question Answering Dataset for Common Sense

# CODAH: An Adversarially-Authored Question Answering Dataset for Common Sense

**Authors:** Michael Chen, Mike D'Arcy, [Alisa Liu](https://md.research.elamur.ai/free/en/authors/alisa-liu/), [Jared Fernandez](https://md.research.elamur.ai/free/en/authors/jared-fernandez/) and [Doug Downey](https://md.research.elamur.ai/free/en/authors/doug-downey/) · **Year:** 2019 · **Venue:** [Proceedings of the 3rd Workshop on Evaluating Vector Space Representations for NLP](https://md.research.elamur.ai/free/en/venues/workshop-on-evaluating-vector-space-representations-for-nlp/) ([Workshop on Evaluating Vector Space Representations for NLP 2019](https://md.research.elamur.ai/free/en/venues/workshop-on-evaluating-vector-space-representations-for-nlp/2019/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/W19-2008/) · [PDF](https://aclanthology.org/W19-2008.pdf) · [DOI 10.18653/v1/w19-2008](https://doi.org/10.18653/v1/w19-2008) · [OpenAlex](https://openalex.org/W3088396740)

**Citations:** 42 citations (OpenAlex, 2026-10-14) · [48](https://md.research.elamur.ai/free/en/papers/chen-2019-codah-adversarially-authored-question-answering/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Pre-trained transformer models such as BERT reached or exceeded human scores on SWAG, a sentence-completion benchmark for commonsense inference. The authors argue that these models lack real human-like common sense and instead exploit regularities in that dataset. To address this, they build CODAH, a harder companion set in the same format: a prompt sentence, a subject, and four candidate endings, of which exactly one is sensible. Most questions were written by Northwestern computer science students who could see, in real time, how a BERT-base model fine-tuned on SWAG answered them. Students earned extra credit when a question fooled this model, and equal credit when it still fooled models fine-tuned on other submissions in cross-validation, which discouraged easily learnable artifacts. After manual cleaning, 2,801 questions remain. Even models fine-tuned on SWAG and then on CODAH reach only about 65% accuracy, while humans score 95.3%. The work offers a practical recipe for model-in-the-loop adversarial authoring and a tougher commonsense test. The authors note, however, that CODAH complements SWAG rather than being harder at equal size.

## Key points
- **Dataset construction:** 4,149 raw submissions were reduced to 2,801 questions from 116 named contributors, with 321 (11%) submitted anonymously. The data is released at github.com/Websail-NU/CODAH.
- **Main results:** In cross-validation, OpenAI GPT-1 scored 65.3% and BERT scored 64.5%, both after prior fine-tuning on SWAG. Human accuracy was 95.3%, compared with about 85% for humans on SWAG.
- **Question categories:** Questions were labeled as Idioms, Negation, Polysemy, Reference, and Quantitative Reasoning. Quantitative questions were especially hard for both BERT and GPT.
- **Answer-only baseline:** With the question hidden, BERT reached 28.4% and GPT-1 reached 53.9%. This indicates some signal lies in the answer options alone.
- **Size-matched comparison:** On a SWAG subset equal in size to CODAH, GPT scored 63.6% versus 62.4% on CODAH, and BERT scored 28.7% versus 49.6%. CODAH is therefore distinct from SWAG but not necessarily more difficult at equal size.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Northwestern University](https://md.research.elamur.ai/free/en/universities/northwestern-university/) — University, Evanston, United States · 12 works · 1,971 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (2): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 510 works · [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · 243 works

## Cite

Cite as: Michael Chen, Mike D'Arcy, Alisa Liu, Jared Fernandez and Doug Downey. 2019. CODAH: An Adversarially-Authored Question Answering Dataset for Common Sense. Proceedings of the 3rd Workshop on Evaluating Vector Space Representations for NLP.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{chen2019codah,
  title = {{CODAH: An Adversarially-Authored Question Answering Dataset for Common Sense}},
  author = {Michael Chen, Mike D'Arcy, Alisa Liu, Jared Fernandez and Doug Downey},
  booktitle = {Proceedings of the 3rd Workshop on Evaluating Vector Space Representations for NLP},
  year = {2019},
  doi = {10.18653/v1/w19-2008},
  url = {https://doi.org/10.18653/v1/w19-2008}
}
```

- [← previous The NiuTrans Machine Translation Systems for WMT19](https://md.research.elamur.ai/free/en/papers/li-2019-niutrans-machine-translation-systems-wmt19/)
- [next → Dialogue Act Classification with Context-Aware Self-Attention](https://md.research.elamur.ai/free/en/papers/raheja-2019-dialogue-act-classification-context-aware/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/)

## References

9 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. [Squad: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392.
2. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642. [find](https://scholar.google.com/scholar?q=Samuel+R+Bowman%2C+Gabor+Angeli%2C+Christopher+Potts%2C+and+Christopher+D+Manning%2E+2015%2E+A+large+annotated+corpus+for+learning+natural+language+inference%2E+In+Proceedings+of+the+2015+Conference+on+Empirical+)
3. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems, pages 5998–6008.
4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805.
5. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa… [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+Bowman%2E+2018%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chap)
6. Allyson Ettinger, Sudha Rao, Hal Daumé III, and Emily M Bender. 2017. Towards linguistically generalizable nlp systems: A workshop and shared task. In Proceedings of the First Workshop on Building Linguistically Generalizable NLP Systems, pages 1– 10. [find](https://scholar.google.com/scholar?q=Allyson+Ettinger%2C+Sudha+Rao%2C+Hal+Daume%CC%81+III%2C+and+Emily+M+Bender%2E+2017%2E+Towards+linguistically+generalizable+nlp+systems%3A+A+workshop+and+shared+task%2E+In+Proceedings+of+the+First+Workshop+on+Building+Li)
7. Reza Ghaeini, Sadid A Hasan, Vivek Datla, Joey Liu, Kathy Lee, Ashequl Qadir, Yuan Ling, Aaditya Prakash, Xiaoli Fern, and Oladimeji Farri. 2018. Dr-bilstm: Dependent reading bidirectional lstm for natural language inference. In Proceedings of the 2018 Conference of the North American Chapter of the… [find](https://scholar.google.com/scholar?q=Reza+Ghaeini%2C+Sadid+A+Hasan%2C+Vivek+Datla%2C+Joey+Liu%2C+Kathy+Lee%2C+Ashequl+Qadir%2C+Yuan+Ling%2C+Aaditya+Prakash%2C+Xiaoli+Fern%2C+and+Oladimeji+Farri%2E+2018%2E+Dr%2Dbilstm%3A+Dependent+reading+bidirectional+lstm+for+na)
8. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. Swag: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 93–104. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Yonatan+Bisk%2C+Roy+Schwartz%2C+and+Yejin+Choi%2E+2018%2E+Swag%3A+A+large%2Dscale+adversarial+dataset+for+grounded+commonsense+inference%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+)
9. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human [find](https://scholar.google.com/scholar?q=Suchin+Gururangan%2C+Swabha+Swayamdipta%2C+Omer+Levy%2C+Roy+Schwartz%2C+Samuel+Bowman%2C+and+Noah+A+Smith%2E+2018%2E+Annotation+artifacts+in+natural+language+inference+data%2E+In+Proceedings+of+the+2018+Conference+of)

*Updated 24 October 2026*