---
title: "End-to-End Bias Mitigation by Modelling Biases in Corpora — Mahabadi 2020"
description: "The authors offer two training-time remedies for cases where the bias is known in advance. 4.8 points on the SNLI. 39 references, cited by 98 papers here."
authors: ["Rabeeh Karimi Mahabadi", "Yonatan Belinkov", "James Henderson"]
year: 2020
venue: "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/2020.acl-main.769/"
doi: "10.18653/v1/2020.acl-main.769"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/mahabadi-2020-end-end-bias-mitigation-modelling/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mahabadi-2020-end-end-bias-mitigation-modelling/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/) / End-to-End Bias Mitigation by Modelling Biases in Corpora

# End-to-End Bias Mitigation by Modelling Biases in Corpora

**Authors:** [Rabeeh Karimi Mahabadi](https://md.research.elamur.ai/free/en/authors/rabeeh-karimi-mahabadi/), [Yonatan Belinkov](https://md.research.elamur.ai/free/en/authors/yonatan-belinkov/) and [James Henderson](https://md.research.elamur.ai/free/en/authors/james-henderson/) · **Year:** 2020 · **Venue:** Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.acl-main.769/) · [PDF](https://aclanthology.org/2020.acl-main.769.pdf) · [DOI 10.18653/v1/2020.acl-main.769](https://doi.org/10.18653/v1/2020.acl-main.769) · [OpenAlex](https://openalex.org/W3018505466)

**Citations:** 9 citations (OpenAlex, 2026-10-14) · [98](https://md.research.elamur.ai/free/en/papers/mahabadi-2020-end-end-bias-mitigation-modelling/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
Strong NLU models often succeed on benchmarks by exploiting spurious cues in the data. In NLI, for example, negation words in the hypothesis tend to signal contradiction, so such models break down on test sets built without those artifacts. The authors offer two training-time remedies for cases where the bias is known in advance. A separate bias-only model, such as a hypothesis-only classifier for NLI or a model using syntactic heuristics and similarity features for HANS, is trained jointly with the main model. Its predictions reshape the main model's loss so that examples the shortcut already solves count less, and learning shifts toward the hard cases.

The two variants are a Product of Experts (PoE) ensemble and a Debiased Focal Loss (DFL). Both train end-to-end, work with any architecture, and can handle several bias sources at once. Tested on SNLI, MNLI and FEVER with challenge sets such as HANS, they clearly improve out-of-domain accuracy and transfer to other entailment data. This makes models more reliable without the cost of collecting new datasets.

## Key points
- **How the methods work:**
  - PoE adds the log-probabilities of the bias-only model and the base model before computing the loss.
  - DFL uses the bias-only model's confidence in the correct label to directly scale down the base model's loss.
  - The bias-only loss is never back-propagated into parameters it shares with the base model, and the bias branch is discarded after training.
- **Headline gains:** the methods reached a new state of the art, improving on the baselines by 7.4 points on HANS, 4.8 points on the SNLI hard set and 9.8 points on the FEVER Symmetric test set. The authors also built hard MNLI subsets from the MNLI development sets.
- **Transfer results:** BERT models trained on SNLI were evaluated on 12 target datasets. Both debiased versions beat the cross-entropy baseline on every target except MNLI and MNLI-M, whose biases resemble SNLI's. The largest gain was on SciTail, with +3.8 for PoE and +1.7 for DFL.
- **Correlation analysis:** the losses of the debiased models correlate less with the bias-only model's loss than the baseline's losses do. PoE showed the lower correlation on MNLI, while DFL did so on HANS.
- **Trade-off and availability:** DFL's γ parameter balances in-domain against out-of-domain accuracy, which matters especially for weaker encoders such as InferSent. Code and data are available at github.com/rabeehk/robust-nli.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Institutions

- [Massachusetts Institute of Technology](https://md.research.elamur.ai/free/en/universities/massachusetts-institute-of-technology/) — University, Cambridge, United States · 125 works · 60,031 citations
- [Harvard University](https://md.research.elamur.ai/free/en/universities/harvard-university/) — University, Cambridge, United States · 64 works · 35,670 citations
- [École Polytechnique Fédérale de Lausanne](https://md.research.elamur.ai/free/en/universities/ecole-polytechnique-federale-de-lausanne/) — University, Lausanne, Switzerland · 19 works · 6,324 citations
- [Idiap Research Institute](https://md.research.elamur.ai/free/en/labs/idiap-research-institute/) — Research institute, Martigny-Combe, Switzerland · 17 works · 1,669 citations

## Cite

Cite as: Rabeeh Karimi Mahabadi, Yonatan Belinkov and James Henderson. 2020. End-to-End Bias Mitigation by Modelling Biases in Corpora. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{mahabadi2020end,
  title = {{End-to-End Bias Mitigation by Modelling Biases in Corpora}},
  author = {Rabeeh Karimi Mahabadi, Yonatan Belinkov and James Henderson},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.769},
  url = {https://doi.org/10.18653/v1/2020.acl-main.769}
}
```

- [← previous AMR-To-Text Generation with Graph Transformer](https://md.research.elamur.ai/free/en/papers/wang-2020-amr-text-generation-graph-transformer/)
- [next → LEDGAR: A Large-Scale Multi-label Corpus for Text Classification of Legal Provisions in Contracts](https://md.research.elamur.ai/free/en/papers/tuggener-2020-ledgar-large-scale-multi-label/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/)

## References

39 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yonatan Belinkov, Adam Poliak, Stuart Shieber, Benjamin Van Durme, and Alexander Rush. 2019a. Don’t take the premise for granted: Mitigating artifacts in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 877–891, Florence, I… [our card](https://md.research.elamur.ai/free/en/papers/belinkov-2019-don-t-take-premise-granted/)
2. Yonatan Belinkov, Adam Poliak, Stuart M Shieber, Benjamin Van Durme, and Alexander M Rush. 2019b. On adversarial removal of hypothesis-only bias in natural language inference. In Proceedings of the 8th Joint Conference on Lexical and Computational Semantics. [find](https://scholar.google.com/scholar?q=Yonatan+Belinkov%2C+Adam+Poliak%2C+Stuart+M+Shieber%2C+Benjamin+Van+Durme%2C+and+Alexander+M+Rush%2E+2019b%2E+On+adversarial+removal+of+hypothesis%2Donly+bias+in+natural+language+inference%2E+In+Proceedings+of+the+8t)
3. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Samuel+R+Bowman%2C+Gabor+Angeli%2C+Christopher+Potts%2C+and+Christopher+D+Manning%2E+2015%2E+A+large+annotated+corpus+for+learning+natural+language+inference%2E+In+Proceedings+of+the+2015+Conference+on+Empirical+)
4. Remi Cadene, Corentin Dancette, Hedi Ben-younes, Matthieu Cord, and Devi Parikh. 2019. Rubi: Reducing unimodal biases in visual question answering. In Advances in neural information processing systems. [find](https://scholar.google.com/scholar?q=Remi+Cadene%2C+Corentin+Dancette%2C+Hedi+Ben%2Dyounes%2C+Matthieu+Cord%2C+and+Devi+Parikh%2E+2019%2E+Rubi%3A+Reducing+unimodal+biases+in+visual+question+answering%2E+In+Advances+in+neural+information+processing+systems)
5. Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced lstm for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Qian+Chen%2C+Xiaodan+Zhu%2C+Zhen%2DHua+Ling%2C+Si+Wei%2C+Hui+Jiang%2C+and+Diana+Inkpen%2E+2017%2E+Enhanced+lstm+for+natural+language+inference%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computa)
6. Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. 2019. Don’t take the easy way out: Ensemble based methods for avoiding known dataset biases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Christopher+Clark%2C+Mark+Yatskar%2C+and+Luke+Zettlemoyer%2E+2019%2E+Don%E2%80%99t+take+the+easy+way+out%3A+Ensemble+based+methods+for+avoiding+known+dataset+biases%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+M)
7. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loı̈c Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. [our card](https://md.research.elamur.ai/free/en/papers/conneau-2017-supervised-learning-universal-sentence-representations/)
8. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment challenge. Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment. [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Oren+Glickman%2C+and+Bernardo+Magnini%2E+2006%2E+The+pascal+recognising+textual+entailment+challenge%2E+Machine+Learning+Challenges%2E+Evaluating+Predictive+Uncertainty%2C+Visual+Object+Classification%2C)
9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)
10. Yichen Gong, Heng Luo, and Jian Zhang. 2017. Natural language inference over interaction space. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Yichen+Gong%2C+Heng+Luo%2C+and+Jian+Zhang%2E+2017%2E+Natural+language+inference+over+interaction+space%2E+In+International+Conference+on+Learning+Representations%2E)
11. Gabriel Grand and Yonatan Belinkov. 2019. Adversarial regularization for visual question answering: Strengths, shortcomings, and side effects. In Proceedings of the Second Workshop on Shortcomings in Vision and Language, pages 1–13. [find](https://scholar.google.com/scholar?q=Gabriel+Grand+and+Yonatan+Belinkov%2E+2019%2E+Adversarial+regularization+for+visual+question+answering%3A+Strengths%2C+shortcomings%2C+and+side+effects%2E+In+Proceedings+of+the+Second+Workshop+on+Shortcomings+in+)
12. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technol… [find](https://scholar.google.com/scholar?q=Suchin+Gururangan%2C+Swabha+Swayamdipta%2C+Omer+Levy%2C+Roy+Schwartz%2C+Samuel+Bowman%2C+and+Noah+A+Smith%2E+2018%2E+Annotation+artifacts+in+natural+language+inference+data%2E+In+Proceedings+of+the+2018+Conference+of)
13. He He, Sheng Zha, and Haohan Wang. 2019. Unlearn dataset bias in natural language inference by fitting the residual. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pages 132–142, Hong Kong, China. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/he-2019-unlearn-dataset-bias-natural-language/)
14. Geoffrey E Hinton. 2002. Training products of experts by minimizing contrastive divergence. Neural computation. [find](https://scholar.google.com/scholar?q=Geoffrey+E+Hinton%2E+2002%2E+Training+products+of+experts+by+minimizing+contrastive+divergence%2E+Neural+computation%2E)
15. Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. [our card](https://md.research.elamur.ai/free/en/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/)
16. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Armand+Joulin%2C+Edouard+Grave%2C+Piotr+Bojanowski%2C+and+Tomas+Mikolov%2E+2017%2E+Bag+of+tricks+for+efficient+text+classification%2E+In+Proceedings+of+the+15th+Conference+of+the+European+Chapter+of+the+Associati)
17. Divyansh Kaushik and Zachary C Lipton. 2018. How much reading does reading comprehension require? a critical investigation of popular benchmarks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. [our card](https://md.research.elamur.ai/free/en/papers/kaushik-2018-how-much-reading-does-reading/)
18. Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. Scitail: A textual entailment dataset from science question answering. In Thirty-Second AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Tushar+Khot%2C+Ashish+Sabharwal%2C+and+Peter+Clark%2E+2018%2E+Scitail%3A+A+textual+entailment+dataset+from+science+question+answering%2E+In+Thirty%2DSecond+AAAI+Conference+on+Artificial+Intelligence%2E)
19. Alice Lai, Yonatan Bisk, and Julia Hockenmaier. 2017. Natural language inference from multiple premises. In Proceedings of the Eighth International Joint Conference on Natural Language Processing. [find](https://scholar.google.com/scholar?q=Alice+Lai%2C+Yonatan+Bisk%2C+and+Julia+Hockenmaier%2E+2017%2E+Natural+language+inference+from+multiple+premises%2E+In+Proceedings+of+the+Eighth+International+Joint+Conference+on+Natural+Language+Processing%2E)
20. Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision. [our card](https://md.research.elamur.ai/free/en/papers/lin-2017-focal-loss-dense-object-detection/)
21. Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014. A sick cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), Reykja… [our card](https://md.research.elamur.ai/free/en/papers/marelli-2014-sick-cure-evaluation-compositional-distributional/)
22. R Thomas McCoy, Junghyun Min, and Tal Linzen. 2019a. Berts of a feather do not generalize together: Large variability in generalization across models with similar test set performance. arXiv preprint arXiv:1911.02969. [find](https://scholar.google.com/scholar?q=R+Thomas+McCoy%2C+Junghyun+Min%2C+and+Tal+Linzen%2E+2019a%2E+Berts+of+a+feather+do+not+generalize+together%3A+Large+variability+in+generalization+across+models+with+similar+test+set+performance%2E+arXiv+preprint+)
23. Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019b. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448, Florence, Italy. Association for Computational Lin… [our card](https://md.research.elamur.ai/free/en/papers/mccoy-2019-right-wrong-reasons-diagnosing-syntactic/)
24. Ellie Pavlick, Travis Wolfe, Pushpendre Rastogi, Chris Callison-Burch, Mark Dredze, and Benjamin Van Durme. 2015. Framenet+: Fast paraphrastic tripling of framenet. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference o… [our card](https://md.research.elamur.ai/free/en/papers/pavlick-2015-framenet-fast-paraphrastic-tripling-framenet/)
25. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics. [find](https://scholar.google.com/scholar?q=Adam+Poliak%2C+Jason+Naradowsky%2C+Aparajita+Haldar%2C+Rachel+Rudinger%2C+and+Benjamin+Van+Durme%2E+2018%2E+Hypothesis+only+baselines+in+natural+language+inference%2E+In+Proceedings+of+the+Seventh+Joint+Conference+)
26. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E)
27. Altaf Rahman and Vincent Ng. 2012. Resolving complex cases of definite pronouns: the winograd schema challenge. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. [our card](https://md.research.elamur.ai/free/en/papers/rahman-2012-resolving-complex-cases-definite-pronouns/)
28. Sainandan Ramakrishnan, Aishwarya Agrawal, and Stefan Lee. 2018. Overcoming language priors in visual question answering with adversarial regularization. In Advances in Neural Information Processing Systems, pages 1541–1551. [find](https://scholar.google.com/scholar?q=Sainandan+Ramakrishnan%2C+Aishwarya+Agrawal%2C+and+Stefan+Lee%2E+2018%2E+Overcoming+language+priors+in+visual+question+answering+with+adversarial+regularization%2E+In+Advances+in+Neural+Information+Processing+S)
29. Drew Reisinger, Rachel Rudinger, Francis Ferraro, Craig Harman, Kyle Rawlins, and Benjamin Van Durme. 2015. Semantic proto-roles. Transactions of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Drew+Reisinger%2C+Rachel+Rudinger%2C+Francis+Ferraro%2C+Craig+Harman%2C+Kyle+Rawlins%2C+and+Benjamin+Van+Durme%2E+2015%2E+Semantic+proto%2Droles%2E+Transactions+of+the+Association+for+Computational+Linguistics%2E)
30. Tal Schuster, Darsh J Shah, Yun Jie Serene Yeo, Daniel Filizzola, Enrico Santus, and Regina Barzilay. 2019. Towards debiasing fact verification models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. [our card](https://md.research.elamur.ai/free/en/papers/schuster-2019-debiasing-fact-verification-models/)
31. Rishi Sharma, James Allen, Omid Bakhshandeh, and Nasrin Mostafazadeh. 2018. Tackling the story ending biases in the story cloze test. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rishi+Sharma%2C+James+Allen%2C+Omid+Bakhshandeh%2C+and+Nasrin+Mostafazadeh%2E+2018%2E+Tackling+the+story+ending+biases+in+the+story+cloze+test%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+C)
32. Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations. [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)
33. Lili Mou, Rui Men, Ge Li, Yan Xu, Lu Zhang, Rui Yan, and Zhi Jin. 2016. Natural language inference by tree-based convolution and heuristic matching. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/mou-2016-natural-language-inference-tree-based/)
34. Zhiguo Wang, Wael Hamza, and Radu Florian. 2017. Bilateral multi-perspective matching for natural language sentences. In Proceedings of the 26th International Joint Conference on Artificial Intelligence. [our card](https://md.research.elamur.ai/free/en/papers/wang-2017-bilateral-multi-perspective-matching-natural/)
35. Ellie Pavlick and Chris Callison-Burch. 2016. Most” babies” are” little” and most” problems” are” huge”: Compositional entailment in adjective-nouns. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ellie+Pavlick+and+Chris+Callison%2DBurch%2E+2016%2E+Most%E2%80%9D+babies%E2%80%9D+are%E2%80%9D+little%E2%80%9D+and+most%E2%80%9D+problems%E2%80%9D+are%E2%80%9D+huge%E2%80%9D%3A+Compositional+entailment+in+adjective%2Dnouns%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+A)
36. Aaron Steven White, Pushpendre Rastogi, Kevin Duh, and Benjamin Van Durme. 2017. Inference is everything: Recasting semantic resources into a unified evaluation framework. In Proceedings of the Eighth International Joint Conference on Natural Language Processing. [our card](https://md.research.elamur.ai/free/en/papers/white-2017-inference-is-everything-recasting-semantic/)
37. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+Bowman%2E+2018%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chap)
38. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
39. Sheng Zhang, Rachel Rudinger, Kevin Duh, and Benjamin Van Durme. 2017. Ordinal common-sense inference. Transactions of the Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/zhang-2017-ordinal-common-sense-inference/)