---
title: "LogiQA: A Challenge Dataset for Machine Reading — Liu 2020"
description: "Reading comprehension benchmarks have grown harder by adding multi-hop evidence and commonsense knowledge, but deductive logical reasoning has received…"
authors: ["Jian Liu", "Leyang Cui", "Hanmeng Liu", "Dandan Huang", "Yile Wang", "Yue Zhang"]
year: 2020
venue: "arXiv:2007.08124"
source: "https://arxiv.org/abs/2007.08124"
doi: "10.48550/arxiv.2007.08124"
arxiv: "2007.08124"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/liu-2020-logiqa-challenge-dataset-machine-reading/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liu-2020-logiqa-challenge-dataset-machine-reading/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/) / LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning

# LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning

**Authors:** [Jian Liu](https://md.research.elamur.ai/free/en/authors/jian-liu/), [Leyang Cui](https://md.research.elamur.ai/free/en/authors/leyang-cui/), [Hanmeng Liu](https://md.research.elamur.ai/free/en/authors/hanmeng-liu/), Dandan Huang, Yile Wang, [Yue Zhang](https://md.research.elamur.ai/free/en/authors/yue-zhang/) · **Year:** 2020 · **Venue:** arXiv:2007.08124 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2007.08124) · [PDF](https://arxiv.org/pdf/2007.08124) · [DOI 10.48550/arxiv.2007.08124](https://doi.org/10.48550/arxiv.2007.08124) · [arXiv 2007.08124](https://arxiv.org/abs/2007.08124) · [OpenAlex](https://openalex.org/W3034457116)

**Citations:** 131 citations (openalex-snapshot · 2026-10-14) · [263](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Reading comprehension benchmarks have grown harder by adding multi-hop evidence and commonsense knowledge, but deductive logical reasoning has received little attention in modern NLP. LogiQA fills this gap with 8,678 multiple-choice items, each pairing a passage and a question with four candidate answers. The items come from publicly available logic tests for civil servant examinations, written by domain experts rather than crowd workers. The authors manually excluded problems that rely on figures, charts or heavy mathematics, and kept a broad mix of reasoning types. Because answers usually do not appear verbatim in the passage and little outside knowledge is needed, success depends on inference rather than surface matching. Several neural readers were tested, from attention-based models to BERT and RoBERTa. The best system reached only 35.31% accuracy, against a human ceiling of 96.00%. This large gap makes LogiQA a demanding test for machine reading. It also offers a way to revisit classic logical AI questions with deep learning methods.

## Key points
- **Dataset:** 8,678 four-option questions from expert-authored civil servant logic exams, released at github.com/lgw863/LogiQA-dataset.
- **Reasoning types:** Items can carry more than one label. Shares are categorical 30.8%, sufficient conditional 27.6%, necessary conditional 24.7%, conjunctive 21.3% and disjunctive 18.5%.
- **Baselines:** The evaluated systems were BERT, RoBERTa, the Co-Matching Network, the Gated-Attention Reader, the Stanford Attentive Reader and a simple word-matching method. The best model scored 35.31%, close to the 25% chance level, while humans scored 96.00%.
- **RoBERTa by type:** Categorical reasoning reached 55.00%. The other four types scored only about 17–23%. Errors included ignoring negation and failing to enumerate constraint combinations.
- **Lexical overlap analysis:** The candidate with the highest word overlap is correct in only 28.37% of cases. Every model chose such candidates more often than that, showing a bias toward surface matching. RoBERTa showed the weakest bias.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

263 papers of our library cite it (of the 140673 whose bibliographies we read); 18 of them have a page here, 245 not yet.

- [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://md.research.elamur.ai/free/en/papers/black-2022-gpt-neox-20b-open-source/)
- [Training Verifiers to Solve Math Word Problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/)
- [LCHAIM - Investigating Long Context Reasoning in Hebrew](https://md.research.elamur.ai/free/en/papers/findings-2025-malul-lchaim-investigating-long-context-reasoning/)
- [Evaluating Large Language Models: A Comprehensive Survey](https://md.research.elamur.ai/free/en/papers/guo-2023-evaluating-large-language-models-comprehensive/)
- [FOLIO: Natural Language Reasoning with First-Order Logic](https://md.research.elamur.ai/free/en/papers/han-2022-folio-natural-language-reasoning-first/)
- [Measuring Coding Challenge Competence With APPS](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-coding-challenge-competence-apps/)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/)
- [Measuring Faithfulness in Chain-of-Thought Reasoning](https://md.research.elamur.ai/free/en/papers/lanham-2023-measuring-faithfulness-chain-thought-reasoning/)
- [Making Language Models Better Reasoners with Step-Aware Verifier](https://md.research.elamur.ai/free/en/papers/li-2023-making-language-models-better-reasoners/)
- [DataComp-LM: In search of the next generation of training sets for language models](https://md.research.elamur.ai/free/en/papers/li-2024-datacomp-lm-search-next-generation/)
- [s1: Simple test-time scaling](https://md.research.elamur.ai/free/en/papers/muennighoff-2025-s1-simple-test-time-scaling/)
- [A Comprehensive Overview of Large Language Models](https://md.research.elamur.ai/free/en/papers/naveed-2023-comprehensive-overview-large-language-models/)
- [AYN: A Tiny Yet Competitive Indian Legal Language Model Pretrained from Scratch](https://md.research.elamur.ai/free/en/papers/niyogi-2026-ayn-tiny-yet-competitive-indian/)
- [Leveraging Large Language Models for Multiple Choice Question Answering](https://md.research.elamur.ai/free/en/papers/robinson-2022-leveraging-large-language-models-multiple/)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/)
- [ThinkStruct: RST-Aware Attention for Logical Reasoning in Machine Reading Comprehension](https://md.research.elamur.ai/free/en/papers/tran-2026-thinkstruct-rst-aware-attention-logical/)
- [Large Language Models are Better Reasoners with Self-Verification](https://md.research.elamur.ai/free/en/papers/weng-2023-large-language-models-are-better/)
- [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://md.research.elamur.ai/free/en/papers/zhong-2024-agieval-human-centric-benchmark-evaluating/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{liu2020logiqa,
  title = {{LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning}},
  author = {Liu, Jian and Cui, Leyang and Liu, Hanmeng and Huang, Dandan and Wang, Yile and Zhang, Yue},
  year = {2020},
  doi = {10.48550/arxiv.2007.08124},
  eprint = {2007.08124},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2007.08124}
}
```

- [← previous Logical Natural Language Generation from Open-Domain Tables](https://md.research.elamur.ai/free/en/papers/chen-2020-logical-natural-language-generation-open/)
- [next → Long Range Arena: A Benchmark for Efficient Transformers](https://md.research.elamur.ai/free/en/papers/tay-2020-long-range-arena-benchmark-efficient/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/)

## References

32 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. P. Bramsen, M. Escobar-Molano, A. Patel, and R. Alonso. Extracting social power relationships from natural language. In NAACL, 2011. [find](https://scholar.google.com/scholar?q=P%2E+Bramsen%2C+M%2E+Escobar%2DMolano%2C+A%2E+Patel%2C+and+R%2E+Alonso%2E+Extracting+social+power+relationships+from+natural+language%2E+In+NAACL%2C+2011%2E)
2. Z. Cai, L. Tu, and K. Gimpel. Pay attention to the ending: Strong neural baselines for the roc story cloze task. In ACL, 2017. [find](https://scholar.google.com/scholar?q=Z%2E+Cai%2C+L%2E+Tu%2C+and+K%2E+Gimpel%2E+Pay+attention+to+the+ending%3A+Strong+neural+baselines+for+the+roc+story+cloze+task%2E+In+ACL%2C+2017%2E)
3. D. Chen, J. Bolton, and C. D. Manning. A thorough examination of the CNN/daily mail reading comprehension task. In ACL, 2016. [find](https://scholar.google.com/scholar?q=D%2E+Chen%2C+J%2E+Bolton%2C+and+C%2E+D%2E+Manning%2E+A+thorough+examination+of+the+CNN%2Fdaily+mail+reading+comprehension+task%2E+In+ACL%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/chen-2016-thorough-examination-cnn-daily-mail/)
4. A. Colmerauer and P. Roussel. The birth of prolog. In History of programming languages—II. ACM, 1996. [find](https://scholar.google.com/scholar?q=A%2E+Colmerauer+and+P%2E+Roussel%2E+The+birth+of+prolog%2E+In+History+of+programming+languages%E2%80%94II%2E+ACM%2C+1996%2E)
5. Y. Cui, W. Che, T. Liu, B. Qin, Z. Yang, S. Wang, and G. Hu. Pre-training with whole word masking for chinese bert. arXiv, 2019. [find](https://scholar.google.com/scholar?q=Y%2E+Cui%2C+W%2E+Che%2C+T%2E+Liu%2C+B%2E+Qin%2C+Z%2E+Yang%2C+S%2E+Wang%2C+and+G%2E+Hu%2E+Pre%2Dtraining+with+whole+word+masking+for+chinese+bert%2E+arXiv%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/cui-2019-pre-training-whole-word-masking/)
6. L. Cui, Y. Wu, S. Liu, Y. Zhang, and M. Zhou. Mutual: A dataset for multi-turn dialogue reasoning. In ACL, 2020. [find](https://scholar.google.com/scholar?q=L%2E+Cui%2C+Y%2E+Wu%2C+S%2E+Liu%2C+Y%2E+Zhang%2C+and+M%2E+Zhou%2E+Mutual%3A+A+dataset+for+multi%2Dturn+dialogue+reasoning%2E+In+ACL%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/cui-2020-mutual-dataset-multi-turn-dialogue/)
7. J. Devlin, M-W Chang, K. Lee, and K. Toutanova. BERT: Pre-training of bidirectional transformers for language understanding. In NAACL, 2019. [find](https://scholar.google.com/scholar?q=J%2E+Devlin%2C+M%2DW+Chang%2C+K%2E+Lee%2C+and+K%2E+Toutanova%2E+BERT%3A+Pre%2Dtraining+of+bidirectional+transformers+for+language+understanding%2E+In+NAACL%2C+2019%2E)
8. B. Dhingra, H. Liu, Z. Yang, W. Cohen, and R. Salakhutdinov. Gated-attention readers for text comprehension. ACL, 2017. [find](https://scholar.google.com/scholar?q=B%2E+Dhingra%2C+H%2E+Liu%2C+Z%2E+Yang%2C+W%2E+Cohen%2C+and+R%2E+Salakhutdinov%2E+Gated%2Dattention+readers+for+text+comprehension%2E+ACL%2C+2017%2E)
9. D. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In NAACL, 2019. [find](https://scholar.google.com/scholar?q=D%2E+Dua%2C+Y%2E+Wang%2C+P%2E+Dasigi%2C+G%2E+Stanovsky%2C+S%2E+Singh%2C+and+M%2E+Gardner%2E+DROP%3A+A+reading+comprehension+benchmark+requiring+discrete+reasoning+over+paragraphs%2E+In+NAACL%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/)
10. I. Habernal, H. Wachsmuth, I. Gurevych, and B. Stein. The argument reasoning comprehension task: Identification and reconstruction of implicit warrants. In NAACL, 2018. [find](https://scholar.google.com/scholar?q=I%2E+Habernal%2C+H%2E+Wachsmuth%2C+I%2E+Gurevych%2C+and+B%2E+Stein%2E+The+argument+reasoning+comprehension+task%3A+Identification+and+reconstruction+of+implicit+warrants%2E+In+NAACL%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/habernal-2018-argument-reasoning-comprehension-task-identification/)
11. K. Moritz Hermann, T. Kocisky, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, and P. Blunsom. Teaching machines to read and comprehend. In NIPS, 2015. [find](https://scholar.google.com/scholar?q=K%2E+Moritz+Hermann%2C+T%2E+Kocisky%2C+E%2E+Grefenstette%2C+L%2E+Espeholt%2C+W%2E+Kay%2C+M%2E+Suleyman%2C+and+P%2E+Blunsom%2E+Teaching+machines+to+read+and+comprehend%2E+In+NIPS%2C+2015%2E) [our card](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/)
12. L. Huang, R. Le Bras, C. Bhagavatula, and Y. Choi. Cosmos QA: Machine comprehension with contextual commonsense reasoning. In EMNLP, 2019. [find](https://scholar.google.com/scholar?q=L%2E+Huang%2C+R%2E+Le+Bras%2C+C%2E+Bhagavatula%2C+and+Y%2E+Choi%2E+Cosmos+QA%3A+Machine+comprehension+with+contextual+commonsense+reasoning%2E+In+EMNLP%2C+2019%2E)
13. Patrick J. Hurley. A concise introduction to logic. Nelson Education, 2014. [find](https://scholar.google.com/scholar?q=Patrick+J%2E+Hurley%2E+A+concise+introduction+to+logic%2E+Nelson+Education%2C+2014%2E)
14. M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In ACL, 2017. [find](https://scholar.google.com/scholar?q=M%2E+Joshi%2C+E%2E+Choi%2C+D%2E+Weld%2C+and+L%2E+Zettlemoyer%2E+TriviaQA%3A+A+large+scale+distantly+supervised+challenge+dataset+for+reading+comprehension%2E+In+ACL%2C+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/)
15. D. Khashabi, T. Khot, A. Sabharwal, P. Clark, O. Etzioni, and D. Roth. Question answering via integer programming over semi-structured knowledge. arXiv, 2016. [find](https://scholar.google.com/scholar?q=D%2E+Khashabi%2C+T%2E+Khot%2C+A%2E+Sabharwal%2C+P%2E+Clark%2C+O%2E+Etzioni%2C+and+D%2E+Roth%2E+Question+answering+via+integer+programming+over+semi%2Dstructured+knowledge%2E+arXiv%2C+2016%2E)
16. Tomáš Kočiský, J. Schwarz, P. Blunsom, C. Dyer, K. M. Hermann, Gábor Melis, and E. Grefenstette. The NarrativeQA reading comprehension challenge. TACL, 2018. [find](https://scholar.google.com/scholar?q=Toma%CC%81s%CC%8C+Koc%CC%8Cisky%CC%81%2C+J%2E+Schwarz%2C+P%2E+Blunsom%2C+C%2E+Dyer%2C+K%2E+M%2E+Hermann%2C+Ga%CC%81bor+Melis%2C+and+E%2E+Grefenstette%2E+The+NarrativeQA+reading+comprehension+challenge%2E+TACL%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/kocisky-2018-narrativeqa-reading-comprehension-challenge/)
17. G. Lai, Q. Xie, H. Liu, Y. Yang, and E. Hovy. RACE: Large-scale Reading Comprehension dataset from Examinations. In EMNLP, 2017. [find](https://scholar.google.com/scholar?q=G%2E+Lai%2C+Q%2E+Xie%2C+H%2E+Liu%2C+Y%2E+Yang%2C+and+E%2E+Hovy%2E+RACE%3A+Large%2Dscale+Reading+Comprehension+dataset+from+Examinations%2E+In+EMNLP%2C+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/)
18. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv, 2019. [find](https://scholar.google.com/scholar?q=Y%2E+Liu%2C+M%2E+Ott%2C+N%2E+Goyal%2C+J%2E+Du%2C+M%2E+Joshi%2C+D%2E+Chen%2C+O%2E+Levy%2C+M%2E+Lewis%2C+L%2E+Zettlemoyer%2C+and+V%2E+Stoyanov%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+arXiv%2C+2019%2E)
19. John McCarthy. Artificial intelligence, logic and formalizing common sense. In Philosophical logic and artificial intelligence. 1989. [find](https://scholar.google.com/scholar?q=John+McCarthy%2E+Artificial+intelligence%2C+logic+and+formalizing+common+sense%2E+In+Philosophical+logic+and+artificial+intelligence%2E+1989%2E)
20. Nils J. Nilsson. Logic and artificial intelligence. Artificial intelligence, 1991. [find](https://scholar.google.com/scholar?q=Nils+J%2E+Nilsson%2E+Logic+and+artificial+intelligence%2E+Artificial+intelligence%2C+1991%2E)
21. T. Niven and Hung-Yu Kao. Probing neural network comprehension of natural language arguments. In ACL, 2019. [find](https://scholar.google.com/scholar?q=T%2E+Niven+and+Hung%2DYu+Kao%2E+Probing+neural+network+comprehension+of+natural+language+arguments%2E+In+ACL%2C+2019%2E)
22. S. Ostermann, M. Roth, A. Modi, S. Thater, and M. Pinkal. Task 11: Machine comprehension using commonsense knowledge. In SemEval, 2018. [find](https://scholar.google.com/scholar?q=S%2E+Ostermann%2C+M%2E+Roth%2C+A%2E+Modi%2C+S%2E+Thater%2C+and+M%2E+Pinkal%2E+Task+11%3A+Machine+comprehension+using+commonsense+knowledge%2E+In+SemEval%2C+2018%2E)
23. P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP, 2016. [find](https://scholar.google.com/scholar?q=P%2E+Rajpurkar%2C+J%2E+Zhang%2C+K%2E+Lopyrev%2C+and+P%2E+Liang%2E+SQuAD%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+In+EMNLP%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
24. M. Richardson, C.J.C. Burges, and E. Renshaw. MCTest: A challenge dataset for the open-domain machine comprehension of text. In EMNLP, 2013. [find](https://scholar.google.com/scholar?q=M%2E+Richardson%2C+C%2EJ%2EC%2E+Burges%2C+and+E%2E+Renshaw%2E+MCTest%3A+A+challenge+dataset+for+the+open%2Ddomain+machine+comprehension+of+text%2E+In+EMNLP%2C+2013%2E) [our card](https://md.research.elamur.ai/free/en/papers/missing-2013-richardson-mctest-challenge-dataset-open-domain/)
25. A. Saha, R. Aralikatte, Mitesh M. Khapra, and K. Sankaranarayanan. DuoRC: Towards complex language understanding with paraphrased reading comprehension. In ACL, 2018. [find](https://scholar.google.com/scholar?q=A%2E+Saha%2C+R%2E+Aralikatte%2C+Mitesh+M%2E+Khapra%2C+and+K%2E+Sankaranarayanan%2E+DuoRC%3A+Towards+complex+language+understanding+with+paraphrased+reading+comprehension%2E+In+ACL%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/saha-2018-duorc-complex-language-understanding-paraphrased/)
26. M. Sap, H. Rashkin, D. Chen, R. Le Bras, and Y. Choi. Social iqa: Commonsense reasoning about social interactions. EMNLP, 2019. [find](https://scholar.google.com/scholar?q=M%2E+Sap%2C+H%2E+Rashkin%2C+D%2E+Chen%2C+R%2E+Le+Bras%2C+and+Y%2E+Choi%2E+Social+iqa%3A+Commonsense+reasoning+about+social+interactions%2E+EMNLP%2C+2019%2E)
27. K. Sinha, S. Sodhani, J. Dong, J. Pineau, and W. L. Hamilton. Clutrr: A diagnostic benchmark for inductive reasoning from text. In EMNLP, 2019. [find](https://scholar.google.com/scholar?q=K%2E+Sinha%2C+S%2E+Sodhani%2C+J%2E+Dong%2C+J%2E+Pineau%2C+and+W%2E+L%2E+Hamilton%2E+Clutrr%3A+A+diagnostic+benchmark+for+inductive+reasoning+from+text%2E+In+EMNLP%2C+2019%2E)
28. S. Wang, M. Yu, J. Jiang, and S. Chang. A co-matching model for multi-choice reading comprehension. In ACL, 2018. [find](https://scholar.google.com/scholar?q=S%2E+Wang%2C+M%2E+Yu%2C+J%2E+Jiang%2C+and+S%2E+Chang%2E+A+co%2Dmatching+model+for+multi%2Dchoice+reading+comprehension%2E+In+ACL%2C+2018%2E)
29. J. Welbl, P. Stenetorp, and S. Riedel. Constructing datasets for multi-hop reading comprehension across documents. TACL, 2018. [find](https://scholar.google.com/scholar?q=J%2E+Welbl%2C+P%2E+Stenetorp%2C+and+S%2E+Riedel%2E+Constructing+datasets+for+multi%2Dhop+reading+comprehension+across+documents%2E+TACL%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/welbl-2018-constructing-datasets-multi-hop-reading/)
30. T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, Rémi Louf, M. Funtowicz, and J. Brew. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, 2019. [find](https://scholar.google.com/scholar?q=T%2E+Wolf%2C+L%2E+Debut%2C+V%2E+Sanh%2C+J%2E+Chaumond%2C+C%2E+Delangue%2C+A%2E+Moi%2C+P%2E+Cistac%2C+T%2E+Rault%2C+Re%CC%81mi+Louf%2C+M%2E+Funtowicz%2C+and+J%2E+Brew%2E+Huggingface%E2%80%99s+transformers%3A+State%2Dof%2Dthe%2Dart+natural+language+processing%2E+ArXi) [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
31. Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. Cohen, R. Salakhutdinov, and C. D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In EMNLP, 2018. [find](https://scholar.google.com/scholar?q=Z%2E+Yang%2C+P%2E+Qi%2C+S%2E+Zhang%2C+Y%2E+Bengio%2C+W%2E+Cohen%2C+R%2E+Salakhutdinov%2C+and+C%2E+D%2E+Manning%2E+HotpotQA%3A+A+dataset+for+diverse%2C+explainable+multi%2Dhop+question+answering%2E+In+EMNLP%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/)
32. Wen-tau Yih, Ming-Wei Chang, C. Meek, and A. Pastusiak. Question answering using enhanced lexical semantic models. In ACL, 2013. [find](https://scholar.google.com/scholar?q=Wen%2Dtau+Yih%2C+Ming%2DWei+Chang%2C+C%2E+Meek%2C+and+A%2E+Pastusiak%2E+Question+answering+using+enhanced+lexical+semantic+models%2E+In+ACL%2C+2013%2E)