---
title: "A Surprisingly Robust Trick for the Winograd Schema Challenge"
description: "The paper addresses the Winograd Schema Challenge, a test in which a system must… 9.6 points. 19 references, cited by 41 papers here."
authors: ["Vid Kocijan", "Ana-Maria Cretu", "Oana-Maria Camburu", "Yordan Yordanov", "Thomas Lukasiewicz"]
year: 2019
venue: "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/P19-1478/"
doi: "10.18653/v1/p19-1478"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kocijan-2019-surprisingly-robust-trick-winograd-schema/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kocijan-2019-surprisingly-robust-trick-winograd-schema/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/) / A Surprisingly Robust Trick for the Winograd Schema Challenge

# A Surprisingly Robust Trick for the Winograd Schema Challenge

**Authors:** [Vid Kocijan](https://md.research.elamur.ai/free/en/authors/vid-kocijan/), [Ana-Maria Cretu](https://md.research.elamur.ai/free/en/authors/ana-maria-cretu/), [Oana-Maria Camburu](https://md.research.elamur.ai/free/en/authors/oana-maria-camburu/), [Yordan Yordanov](https://md.research.elamur.ai/free/en/authors/yordan-yordanov/) and [Thomas Lukasiewicz](https://md.research.elamur.ai/free/en/authors/thomas-lukasiewicz/) · **Year:** 2019 · **Venue:** [Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2019](https://md.research.elamur.ai/free/en/venues/acl/2019/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/P19-1478/) · [PDF](https://aclanthology.org/P19-1478.pdf) · [DOI 10.18653/v1/p19-1478](https://doi.org/10.18653/v1/p19-1478) · [OpenAlex](https://openalex.org/W2945290257)

**Citations:** 80 citations (OpenAlex, 2026-10-14) · [41](https://md.research.elamur.ai/free/en/papers/kocijan-2019-surprisingly-robust-trick-winograd-schema/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper addresses the Winograd Schema Challenge, a test in which a system must decide which of two candidate nouns an ambiguous pronoun refers to. The task is designed to require commonsense reasoning, and very little training data exists for it. The authors mask the pronoun and let a pre-trained language model (BERT-large, BERT-base or GPT) score each candidate by its masked-token probability. Fine-tuning combines a log-likelihood term with a margin-based ranking penalty. Two supervision sources are used. The first is the Definite Pronoun Resolution dataset (WSCR), an easier relative of WSC273. The second is MaskedWiki, a new set of 2.4M automatically built WSC-like items created from English Wikipedia by masking a repeated noun. Earlier work found little benefit in transferring from WSCR, yet here it improved every model tested. BERT trained on MaskedWiki and then WSCR reached 72.5% on WSC273 and 74.7% on WNLI. This exceeds the previous best results by 8.8 and 9.6 points and beats the WNLI majority baseline, which no earlier system had surpassed.

## Key points
- Best model (BERT_WIKI_WSCR): 72.5% on WSC273 and 74.7% on the WNLI test set. The prior state of the art was about 63.7% on WSC273, and the WNLI majority baseline is 65.1 (human performance 95.9).
- MaskedWiki was built from the enwiki-20181201 dump using the Stanford POS tagger. About 130M candidate items were randomly downsampled to 2.4M. A manual check of 200 items found 45% hard, 45.5% easy, 8.5% unsolvable and 1% noise.
- WSCR was used with 1322 training and 564 test examples, the test set serving for validation. Six items that overlap with WSC273 were removed beforehand.
- Training settings were α = 20, β = 0.2, a learning rate of 5·10⁻⁶ with Adam, and batch size 64. The models ran one epoch on MaskedWiki and up to 30 epochs on WSCR.
- MaskedWiki alone did not help BERT, but it did once WSCR fine-tuning followed. Keeping the paired "special word" examples in WSCR mattered: accuracy on WSC273 was 0.703 with pairs versus 0.663 without them.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Oxford](https://md.research.elamur.ai/free/en/universities/university-of-oxford/) — University, Oxford, United Kingdom · 104 works · 157,183 citations
- [Imperial College London](https://md.research.elamur.ai/free/en/universities/imperial-college-london/) — University, London, United Kingdom · 48 works · 12,181 citations
- [The Alan Turing Institute](https://md.research.elamur.ai/free/en/labs/the-alan-turing-institute/) — Research institute, London, United Kingdom · 29 works · 4,368 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

## Cite

Cite as: Vid Kocijan, Ana-Maria Cretu, Oana-Maria Camburu, Yordan Yordanov and Thomas Lukasiewicz. 2019. A Surprisingly Robust Trick for the Winograd Schema Challenge. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{kocijan2019surprisingly,
  title = {{A Surprisingly Robust Trick for the Winograd Schema Challenge}},
  author = {Vid Kocijan, Ana-Maria Cretu, Oana-Maria Camburu, Yordan Yordanov and Thomas Lukasiewicz},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1478},
  url = {https://doi.org/10.18653/v1/p19-1478}
}
```

- [← previous Automatic Argument Quality Assessment - New Datasets and Methods](https://md.research.elamur.ai/free/en/papers/toledo-2019-automatic-argument-quality-assessment-new/)
- [next → Collaborative Dialogue in Minecraft](https://md.research.elamur.ai/free/en/papers/chen-2019-collaborative-dialogue-minecraft/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/)

## References

19 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ernest Davis, Leora Morgenstern, and Charles L. Ortiz. 2017. The first Winograd Schema Challenge at IJCAI-16. AI Magazine, 38(3):97–98. [find](https://scholar.google.com/scholar?q=Ernest+Davis%2C+Leora+Morgenstern%2C+and+Charles+L%2E+Ortiz%2E+2017%2E+The+first+Winograd+Schema+Challenge+at+IJCAI%2D16%2E+AI+Magazine%2C+38%283%29%3A97%E2%80%9398%2E)
2. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). Computing Research Repository, arXiv:1810.04805.
3. Ali Emami, Noelia De La Cruz, Adam Trischler, Kaheer Suleman, and Jackie Chi Kit Cheung. 2018. A knowledge hunting framework for common sense reasoning. Computing Research Repository, arXiv:1810.01375. [find](https://scholar.google.com/scholar?q=Ali+Emami%2C+Noelia+De+La+Cruz%2C+Adam+Trischler%2C+Kaheer+Suleman%2C+and+Jackie+Chi+Kit+Cheung%2E+2018%2E+A+knowledge+hunting+framework+for+common+sense+reasoning%2E+Computing+Research+Repository%2C+arXiv%3A1810%2E01375)
4. Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zh… — [Achieving Human Parity on Automatic Chinese to English News Translation](https://md.research.elamur.ai/free/en/papers/hassan-2018-achieving-human-parity-automatic-chinese/)
5. Jeremy Howard and Sebastian Ruder. 2018. Fine-tuned language models for text classification. Computing Research Repository, arXiv:1801.06146. [find](https://scholar.google.com/scholar?q=Jeremy+Howard+and+Sebastian+Ruder%2E+2018%2E+Fine%2Dtuned+language+models+for+text+classification%2E+Computing+Research+Repository%2C+arXiv%3A1801%2E06146%2E)
6. Rie Johnson and Tong Zhang. 2017. Deep pyramid convolutional neural networks for text categorization. In Proceedings of ACL, pages 562–570. ACL. [find](https://scholar.google.com/scholar?q=Rie+Johnson+and+Tong+Zhang%2E+2017%2E+Deep+pyramid+convolutional+neural+networks+for+text+categorization%2E+In+Proceedings+of+ACL%2C+pages+562%E2%80%93570%2E+ACL%2E)
7. Hector J. Levesque, Ernest Davis, and Leora Morgenstern. 2011. The Winograd Schema Challenge. AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning, 46. [find](https://scholar.google.com/scholar?q=Hector+J%2E+Levesque%2C+Ernest+Davis%2C+and+Leora+Morgenstern%2E+2011%2E+The+Winograd+Schema+Challenge%2E+AAAI+Spring+Symposium%3A+Logical+Formalizations+of+Commonsense+Reasoning%2C+46%2E)
8. Hector J. Levesque, Ernest Davis, and Leora Morgenstern. 2012. The Winograd Schema Challenge. In Proceedings of KR. AAAI Press. [find](https://scholar.google.com/scholar?q=Hector+J%2E+Levesque%2C+Ernest+Davis%2C+and+Leora+Morgenstern%2E+2012%2E+The+Winograd+Schema+Challenge%2E+In+Proceedings+of+KR%2E+AAAI+Press%2E)
9. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019. [Multi-task deep neural networks for natural language understanding](https://md.research.elamur.ai/free/en/papers/liu-2019-multi-task-deep-neural-networks/). Computing Research Repository, arXiv:1901.11504.
10. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Mc-Closky. 2014. [The Stanford CoreNLP natural language processing toolkit](https://md.research.elamur.ai/free/en/papers/manning-2014-stanford-corenlp-natural-language-processing/). In Association for Computational Linguistics (ACL) System Demonstrations, pages 55–60.
11. Leora Morgenstern, Ernest Davis, and Charles L. Ortiz. 2016. Planning, executing and evaluating the Winograd Schema Challenge. AI Magazine. [find](https://scholar.google.com/scholar?q=Leora+Morgenstern%2C+Ernest+Davis%2C+and+Charles+L%2E+Ortiz%2E+2016%2E+Planning%2C+executing+and+evaluating+the+Winograd+Schema+Challenge%2E+AI+Magazine%2E)
12. Nikita Nangia and Samuel R. Bowman. 2018. A conservative human baseline estimate for GLUE: People still (mostly) beat machines. [find](https://scholar.google.com/scholar?q=Nikita+Nangia+and+Samuel+R%2E+Bowman%2E+2018%2E+A+conservative+human+baseline+estimate+for+GLUE%3A+People+still+%28mostly%29+beat+machines%2E)
13. Juri Opitz and Anette Frank. 2018. Addressing the Winograd Schema Challenge as a sequence ranking task. In Proceedings of the First International Workshop on Language Cognition and Computational Models, pages 41–52. ACL. [find](https://scholar.google.com/scholar?q=Juri+Opitz+and+Anette+Frank%2E+2018%2E+Addressing+the+Winograd+Schema+Challenge+as+a+sequence+ranking+task%2E+In+Proceedings+of+the+First+International+Workshop+on+Language+Cognition+and+Computational+Model)
14. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E)
15. Altaf Rahman and Vincent Ng. 2012. [Resolving complex cases of definite pronouns: The Winograd Schema Challenge](https://md.research.elamur.ai/free/en/papers/rahman-2012-resolving-complex-cases-definite-pronouns/). In Proceedings of EMNLP.
16. Paul Trichelair, Ali Emami, Jackie Chi Kit Cheung, Adam Trischler, Kaheer Suleman, and Fernando Diaz. 2018. On the evaluation of common-sense reasoning in natural language understanding. Computing Research Repository, arXiv:1811.01778. — [arXiv:1811.01778](https://arxiv.org/abs/1811.01778) [find](https://scholar.google.com/scholar?q=Paul+Trichelair%2C+Ali+Emami%2C+Jackie+Chi+Kit+Cheung%2C+Adam+Trischler%2C+Kaheer+Suleman%2C+and+Fernando+Diaz%2E+2018%2E+On+the+evaluation+of+common%2Dsense+reasoning+in+natural+language+understanding%2E+Computing+Res)
17. T. H. Trinh and Q. V. Le. 2018. A Simple Method for Commonsense Reasoning. Computing Research repository, arXiv:1806.02847. [find](https://scholar.google.com/scholar?q=T%2E+H%2E+Trinh+and+Q%2E+V%2E+Le%2E+2018%2E+A+Simple+Method+for+Commonsense+Reasoning%2E+Computing+Research+repository%2C+arXiv%3A1806%2E02847%2E)
18. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Computing Research Repository, arXiv:1706.03762.
19. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In Proceedings of ICLR.

*Updated 29 October 2026*