---
title: "DuReader_robust — paper by Tang et al. (2020)"
description: "Neural machine reading comprehension (MRC) models often fail in real-world use. 95% accuracy. 19 references, cited by 12 papers here."
authors: ["Hongxuan Tang", "Hongyu Li", "Jing Liu", "Yu Hong", "Hua Wu", "Haifeng Wang"]
year: 2020
venue: "arXiv:2004.11142"
source: "https://arxiv.org/abs/2004.11142"
doi: "10.48550/arxiv.2004.11142"
arxiv: "2004.11142"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/tang-2020-dureader-robust-chinese-dataset/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/tang-2020-dureader-robust-chinese-dataset/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/8/) / DuReader\_robust: A Chinese Dataset Towards Evaluating Robustness and Generalization of Machine Reading Comprehension in Real-World Applications

# DuReader\_robust: A Chinese Dataset Towards Evaluating Robustness and Generalization of Machine Reading Comprehension in Real-World Applications

**Authors:** Hongxuan Tang, [Hongyu Li](https://md.research.elamur.ai/free/en/authors/hongyu-li/), [Jing Liu](https://md.research.elamur.ai/free/en/authors/jing-liu/), [Yu Hong](https://md.research.elamur.ai/free/en/authors/yu-hong/), [Hua Wu](https://md.research.elamur.ai/free/en/authors/hua-ren-wu/) and [Haifeng Wang](https://md.research.elamur.ai/free/en/authors/haifeng-wang/) · **Year:** 2020 · **Venue:** arXiv:2004.11142 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2004.11142) · [PDF](https://arxiv.org/pdf/2004.11142) · [DOI 10.48550/arxiv.2004.11142](https://doi.org/10.48550/arxiv.2004.11142) · [arXiv 2004.11142](https://arxiv.org/abs/2004.11142) · [OpenAlex](https://openalex.org/W3184464794)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [12](https://md.research.elamur.ai/free/en/papers/tang-2020-dureader-robust-chinese-dataset/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Neural machine reading comprehension (MRC) models often fail in real-world use. They may answer paraphrased questions inconsistently (over-sensitivity), get distracted by passage spans that share many words with the question (over-stability), or struggle outside their training domain (generalization). Earlier benchmarks studied these weaknesses one at a time, usually with artificially edited questions or passages. The authors instead built DuReader_robust, a Chinese span-extraction dataset derived from DuReader, where questions come from real Baidu search users and contexts come from search results. It has standard in-domain splits plus a challenge test set covering all three weaknesses, assembled from natural text with crowdworker and linguistic-expert annotation and some model-based adversarial filtering. Baselines built on pre-trained LMs (BERT base, ERNIE 1.0 base, RoBERTa large) come close to human scores in-domain but drop sharply on the challenge set. The analysis suggests that more rephrasing, stronger question–trap overlap and missing domain knowledge each hurt performance. This makes the dataset a realistic testbed for robust MRC.

## Key points
- The dataset has about 21K instances: 14,520 train, 1,417 dev, 1,285 in-domain test and 3,556 challenge examples. Answer types are mostly dates (24.7%) and numbers (17.5%), and 10% of annotations were expert-checked to a 95% accuracy threshold.
- The challenge set has three subsets:
  - **Over-sensitivity (~1.2K):** paraphrases from a retrieval toolkit, half model-independent and half chosen by attacking ERNIE and RoBERTa models.
  - **Over-stability (~0.8K):** expert-written questions targeting same-type named-entity traps.
  - **Generalization:** out-of-domain data from education and finance.
- RoBERTa large reaches 71.20 EM / 84.16 F1 on the in-domain test set, against a human score of 72.00 / 86.43. On the challenge set it falls to 45.02 / 62.83.
- On the over-stability subset, EM ranges from only 16.78 to 28.18. Baselines change their answer for roughly 16–22% of paraphrased questions, and this rate rises as paraphrases diverge lexically from the original.
- In the education domain, RoBERTa large scores poorly on math (19.85 EM) and chemistry (37.46 EM), topics that are rare in training data. The dataset and code are released at github.com/baidu/DuReader.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Baidu](https://md.research.elamur.ai/free/en/companies/baidu/) — Company, Beijing, China · 74 works · 15,263 citations
- [Soochow University](https://md.research.elamur.ai/free/en/universities/soochow-university/) — University, Suzhou, China · 63 works · 4,507 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · 441 works

## Cite

Cite as: Hongxuan Tang, Hongyu Li, Jing Liu, Yu Hong, Hua Wu and Haifeng Wang. 2020. DuReader\_robust: A Chinese Dataset Towards Evaluating Robustness and Generalization of Machine Reading Comprehension in Real-World Applications. arXiv:2004.11142.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{tang2020dureader,
  title = {{DuReader\_robust: A Chinese Dataset Towards Evaluating Robustness and Generalization of Machine Reading Comprehension in Real-World Applications}},
  author = {Hongxuan Tang, Hongyu Li, Jing Liu, Yu Hong, Hua Wu and Haifeng Wang},
  year = {2020},
  doi = {10.48550/arxiv.2004.11142},
  eprint = {2004.11142},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2004.11142}
}
```

- [← previous Akkadian Treebank for early Neo-Assyrian Royal Inscriptions](https://md.research.elamur.ai/free/en/papers/luukko-2020-akkadian-treebank-early-neo-assyrian/)
- [next → I3rab: A New Arabic Dependency Treebank Based on Arabic Grammatical Theory](https://md.research.elamur.ai/free/en/papers/halabi-2020-i3rab-new-arabic-dependency-treebank/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/8/)

## References

19 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
2. Shi Feng, Eric Wallace, Alvin Grissom II, Mohit Iyyer, Pedro Rodriguez, and Jordan Boyd-Graber. 2018. Pathologies of neural models make interpretations difficult. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3719–3728. [find](https://scholar.google.com/scholar?q=Shi+Feng%2C+Eric+Wallace%2C+Alvin+Grissom+II%2C+Mohit+Iyyer%2C+Pedro+Rodriguez%2C+and+Jordan+Boyd%2DGraber%2E+2018%2E+Pathologies+of+neural+models+make+interpretations+difficult%2E+In+Proceedings+of+the+2018+Conference)
3. Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. 2019. [find](https://scholar.google.com/scholar?q=Adam+Fisch%2C+Alon+Talmor%2C+Robin+Jia%2C+Minjoon+Seo%2C+Eunsol+Choi%2C+and+Danqi+Chen%2E+2019%2E)
4. [MRQA 2019 shared task: Evaluating generalization in reading comprehension](https://md.research.elamur.ai/free/en/papers/fisch-2019-mrqa-2019-shared-task-evaluating/). In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 1–13, Hong Kong, China. Association for Computational Linguistics.
5. Wee Chung Gan and Hwee Tou Ng. 2019. Improving the robustness of question answering systems to question paraphrasing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6065–6075. [find](https://scholar.google.com/scholar?q=Wee+Chung+Gan+and+Hwee+Tou+Ng%2E+2019%2E+Improving+the+robustness+of+question+answering+systems+to+question+paraphrasing%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Lin)
6. Wei He, Kai Liu, Jing Liu, Yajuan Lyu, Shiqi Zhao, Xinyan Xiao, Yuan Liu, Yizhong Wang, Hua Wu, Qiaoqiao She, Xuan Liu, Tian Wu, and Haifeng Wang. 2018. [DuReader: a Chinese machine reading comprehension dataset from real-world applications](https://md.research.elamur.ai/free/en/papers/he-2017-dureader-chinese-machine-reading/). In Proceedings of the Workshop on Machine Reading for Quest…
7. Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. [Adversarial example generation with syntactically controlled paraphrase networks](https://md.research.elamur.ai/free/en/papers/iyyer-2018-adversarial-example-generation-syntactically-controlled/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vo…
8. Robin Jia and Percy Liang. 2017. [Adversarial examples for evaluating reading comprehension systems](https://md.research.elamur.ai/free/en/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021–2031, Copenhagen, Denmark. Association for Computational Linguistics.
9. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. [ALBERT: A lite BERT for self-supervised learning of language representations](https://md.research.elamur.ai/free/en/papers/lan-2019-albert-lite-bert-self-supervised/). In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRevi…
10. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). In arXiv preprint arXiv:1907.11692.
11. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. [Squad: 100, 000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 2383–2392. The A…
12. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018a. Anchors: High-precision model-agnostic explanations. In Thirty-Second AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Marco+Tulio+Ribeiro%2C+Sameer+Singh%2C+and+Carlos+Guestrin%2E+2018a%2E+Anchors%3A+High%2Dprecision+model%2Dagnostic+explanations%2E+In+Thirty%2DSecond+AAAI+Conference+on+Artificial+Intelligence%2E)
13. Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018b. Semantically equivalent adversarial rules for debugging NLP models. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pag… [find](https://scholar.google.com/scholar?q=Marco+Tu%CC%81lio+Ribeiro%2C+Sameer+Singh%2C+and+Carlos+Guestrin%2E+2018b%2E+Semantically+equivalent+adversarial+rules+for+debugging+NLP+models%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Com)
14. Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. [Bidirectional attention flow for machine comprehension](https://md.research.elamur.ai/free/en/papers/seo-2016-bidirectional-attention-flow-machine-comprehension/). In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
15. Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019. [Ernie: Enhanced representation through knowledge integration](https://md.research.elamur.ai/free/en/papers/sun-2019-ernie-enhanced-representation-through-knowledge/). arXiv preprint arXiv:1904.09223.
16. Alon Talmor and Jonathan Berant. 2019a. [Multiqa: An empirical investigation of generalization and transfer in reading comprehension](https://md.research.elamur.ai/free/en/papers/talmor-2019-multiqa-empirical-investigation-generalization-transfer/). In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, page…
17. Alon Talmor and Jonathan Berant. 2019b. [Multiqa: An empirical investigation of generalization and transfer in reading comprehension](https://md.research.elamur.ai/free/en/papers/talmor-2019-multiqa-empirical-investigation-generalization-transfer/). In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, page…
18. Shuohang Wang and Jing Jiang. 2017. [Machine comprehension using match-lstm and answer pointer](https://md.research.elamur.ai/free/en/papers/wang-2016-machine-comprehension-using-match-lstm/). In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
19. Johannes Welbl, Pasquale Minervini, Max Bartolo, Pontus Stenetorp, and Sebastian Riedel. 2020. Un-dersensitivity in neural reading comprehension. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, EMNLP 2020, Online Event, 16-20 November 2020, pages… [find](https://scholar.google.com/scholar?q=Johannes+Welbl%2C+Pasquale+Minervini%2C+Max+Bartolo%2C+Pontus+Stenetorp%2C+and+Sebastian+Riedel%2E+2020%2E+Un%2Ddersensitivity+in+neural+reading+comprehension%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Met)

*Updated 30 October 2026*