---
title: "Med-HALT: Medical Domain Hallucination Test for — Pal 2023"
description: "Large language models can produce fluent, confident answers that are wrong or unverifiable."
authors: ["Ankit Pal", "Logesh Kumar Umapathi", "Malaikannan Sankarasubbu"]
year: 2023
venue: "arXiv:2307.15343"
source: "https://arxiv.org/abs/2307.15343"
doi: "10.48550/arxiv.2307.15343"
arxiv: "2307.15343"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/pal-2023-med-halt-medical-domain-hallucination/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/pal-2023-med-halt-medical-domain-hallucination/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/) / Med-HALT: Medical Domain Hallucination Test for Large Language Models

# Med-HALT: Medical Domain Hallucination Test for Large Language Models

**Authors:** [Ankit Pal](https://md.research.elamur.ai/free/en/authors/ankit-pal/), [Logesh Kumar Umapathi](https://md.research.elamur.ai/free/en/authors/logesh-kumar-umapathi/), [Malaikannan Sankarasubbu](https://md.research.elamur.ai/free/en/authors/malaikannan-sankarasubbu/) · **Year:** 2023 · **Venue:** arXiv:2307.15343 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2307.15343) · [PDF](https://arxiv.org/pdf/2307.15343) · [DOI 10.48550/arxiv.2307.15343](https://doi.org/10.48550/arxiv.2307.15343) · [arXiv 2307.15343](https://arxiv.org/abs/2307.15343) · [OpenAlex](https://openalex.org/W4389519219)

**Citations:** 144 citations (openalex-snapshot · 2026-10-14) · [75](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Large language models can produce fluent, confident answers that are wrong or unverifiable. In healthcare this can directly harm patient decisions, yet hallucination had not been benchmarked specifically for medicine. The authors introduce Med-HALT, which they describe as the first benchmark of its kind for the medical domain. It draws on multiple-choice questions from medical licensing and entrance exams in India, Spain, the U.S., and Taiwan, plus a PubMed-based portion.

The benchmark has two test families:
- **Reasoning tests** check whether a model can reject a randomly proposed answer, recognise when "None of the above" is correct, and refuse nonsensical questions.
- **Memory tests** check whether a model can correctly link PubMed IDs, titles, abstracts, and URLs.

The authors evaluated Text-Davinci, GPT-3.5, Llama-2, MPT, and Falcon. All models showed substantial room for improvement, and results depended strongly on prompt wording and decoding settings. Med-HALT gives developers a reproducible way to measure this risk before deploying LLMs in clinical settings.

## Key points
- **Data size and sources:** Med-HALT has 18,866 samples per reasoning task and 4,916 PubMed samples per memory task. Sources include MedMCQA (AIIMS PG and NEET PG, 9,515 questions), Headqa (Spain, 4,068), MedQA USMILE (2,801), and MedQA Taiwan (TWMLE, 2,482).
- **Test types:** The reasoning tests are the False Confidence Test, None of the Above, and Fake Questions. The memory tests are PMID-to-Title, Title-to-PubMed-link, Abstract-to-PubMed-link, and PubMed-link-to-Title.
- **Reasoning results:** Llama-2 70B had the highest average accuracy (about 72.3%), well ahead of GPT-3.5 (about 44.5%).
- **Memory results:** Falcon 40B and its instruct variant scored best on average (about 30.4% accuracy). Producing a link from an abstract was close to 0% for nearly every model.
- **GPT-3.5 sensitivity experiments:**
  - Zero-shot accuracy was only 7.31%; adding examples helped, but gains flattened beyond 3 shots.
  - Temperature had a negligible effect.
  - Repeated runs ranged from 27.87% to 28.52%.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) · [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

## Cited by

Cited by

75 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 68 not yet.

- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://md.research.elamur.ai/free/en/papers/huang-2023-survey-hallucination-large-language-models/)
- [Survey of Hallucination in Natural Language Generation](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/)
- [The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Models](https://md.research.elamur.ai/free/en/papers/li-2024-dawn-after-dark-empirical-study/)
- [RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models](https://md.research.elamur.ai/free/en/papers/niu-2024-ragtruth-hallucination-corpus-developing-trustworthy/)
- [A Survey of Hallucination in Large Foundation Models](https://md.research.elamur.ai/free/en/papers/rawte-2023-survey-hallucination-large-foundation-models/)
- [MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning](https://md.research.elamur.ai/free/en/papers/tang-2024-medagents-large-language-models-as/)
- [Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models](https://md.research.elamur.ai/free/en/papers/zhang-2023-siren-s-song-ai-ocean/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{pal2023med,
  title = {{Med-HALT: Medical Domain Hallucination Test for Large Language Models}},
  author = {Pal, Ankit and Umapathi, Logesh Kumar and Sankarasubbu, Malaikannan},
  year = {2023},
  doi = {10.48550/arxiv.2307.15343},
  eprint = {2307.15343},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2307.15343}
}
```

- [← previous Measuring Fine-Grained Semantic Equivalence with Abstract Meaning Representation](https://md.research.elamur.ai/free/en/papers/wein-2023-measuring-fine-grained-semantic-equivalence/)
- [next → MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Data](https://md.research.elamur.ai/free/en/papers/han-2023-medalpaca-open-source-collection-medical/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/)

## References

25 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Monica Agrawal, Stefan Hegselmann, Hunter Lang, Yoon Kim, and David Sontag. 2022. Large language models are few-shot clinical information extractors. [find](https://scholar.google.com/scholar?q=Monica+Agrawal%2C+Stefan+Hegselmann%2C+Hunter+Lang%2C+Yoon+Kim%2C+and+David+Sontag%2E+2022%2E+Large+language+models+are+few%2Dshot+clinical+information+extractors%2E) [our card](https://md.research.elamur.ai/free/en/papers/agrawal-2022-large-language-models-are-few/)
2. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Sh… [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Andy+Jones%2C+Kamal+Ndousse%2C+Amanda+Askell%2C+Anna+Chen%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Stanislav+Fort%2C+Deep+Ganguli%2C+Tom+Henighan%2C+Nicholas+Joseph%2C+Saurav+Kadavath%2C+Jackson+Kernion%2C+Tom+Conerly%2C+)
3. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. A multitask, multilingual, Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind N… [find](https://scholar.google.com/scholar?q=Yejin+Bang%2C+Samuel+Cahyawijaya%2C+Nayeon+Lee%2C+Wenliang+Dai%2C+Dan+Su%2C+Bryan+Wilie%2C+Holy+Lovenia%2C+Ziwei+Ji%2C+Tiezheng+Yu%2C+Willy+Chung%2C+Quyet+V%2E+Do%2C+Yan+Xu%2C+and+Pascale+Fung%2E+2023%2E+A+multitask%2C+multilingual%2C)
4. Tianyu Han, Lisa C Adams, Jens-Michalis Papaioannou, Paul Grundmann, Tom Oberhauser, Alexander Löser, Daniel Truhn, and Keno K Bressem. 2023. Medalpaca–an open-source collection of medical conversational ai models and training data. arXiv preprint arXiv:2304.08247. [find](https://scholar.google.com/scholar?q=Tianyu+Han%2C+Lisa+C+Adams%2C+Jens%2DMichalis+Papaioannou%2C+Paul+Grundmann%2C+Tom+Oberhauser%2C+Alexander+Lo%CC%88ser%2C+Daniel+Truhn%2C+and+Keno+K+Bressem%2E+2023%2E+Medalpaca%E2%80%93an+open%2Dsource+collection+of+medical+conversati) [our card](https://md.research.elamur.ai/free/en/papers/han-2023-medalpaca-open-source-collection-medical/)
5. Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. CoRR, abs/1904.09751. [find](https://scholar.google.com/scholar?q=Ari+Holtzman%2C+Jan+Buys%2C+Maxwell+Forbes%2C+and+Yejin+Choi%2E+2019%2E+The+curious+case+of+neural+text+degeneration%2E+CoRR%2C+abs%2F1904%2E09751%2E)
6. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Wenliang Dai, Andrea Madotto, and Pascale Fung. 2022. Survey of hallucination in natural language generation. ACM Computing Surveys, 55:1 – 38. [find](https://scholar.google.com/scholar?q=Ziwei+Ji%2C+Nayeon+Lee%2C+Rita+Frieske%2C+Tiezheng+Yu%2C+Dan+Su%2C+Yan+Xu%2C+Etsuko+Ishii%2C+Yejin+Bang%2C+Wenliang+Dai%2C+Andrea+Madotto%2C+and+Pascale+Fung%2E+2022%2E+Survey+of+hallucination+in+natural+language+generation%2E) [our card](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/)
7. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2020. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. ArXiv, abs/2009.13081. [find](https://scholar.google.com/scholar?q=Di+Jin%2C+Eileen+Pan%2C+Nassim+Oufattole%2C+Wei%2DHung+Weng%2C+Hanyi+Fang%2C+and+Peter+Szolovits%2E+2020%2E+What+disease+does+this+patient+have%3F+a+large%2Dscale+open+domain+question+answering+dataset+from+medical+exams) [our card](https://md.research.elamur.ai/free/en/papers/jin-2020-disease-this-patient-have-large/)
8. Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jianyun Nie, and Ji rong Wen. 2023a. Halueval: A large-scale hallucination evaluation benchmark for large language models. ArXiv, abs/2305.11747. [find](https://scholar.google.com/scholar?q=Junyi+Li%2C+Xiaoxue+Cheng%2C+Wayne+Xin+Zhao%2C+Jianyun+Nie%2C+and+Ji+rong+Wen%2E+2023a%2E+Halueval%3A+A+large%2Dscale+hallucination+evaluation+benchmark+for+large+language+models%2E+ArXiv%2C+abs%2F2305%2E11747%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-halueval-large-scale-hallucination-evaluation/)
9. Yunxiang Li, Zihan Li, Kai Zhang, Ruilong Dan, Steve Jiang, and You Zhang. 2023b. Chatdoctor: A medical chat model fine-tuned on a large language model meta-ai (llama) using medical domain knowledge. Cureus, 15(6). [find](https://scholar.google.com/scholar?q=Yunxiang+Li%2C+Zihan+Li%2C+Kai+Zhang%2C+Ruilong+Dan%2C+Steve+Jiang%2C+and+You+Zhang%2E+2023b%2E+Chatdoctor%3A+A+medical+chat+model+fine%2Dtuned+on+a+large+language+model+meta%2Dai+%28llama%29+using+medical+domain+knowledge%2E+) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-chatdoctor-medical-chat-model-fine/)
10. Tianyu Liu, Yizhe Zhang, Chris Brockett, Yi Mao, Zhifang Sui, Weizhu Chen, and Bill Dolan. 2021. A token-level reference-free hallucination detection benchmark for free-form text generation. arXiv preprint arXiv:2104.08704. [find](https://scholar.google.com/scholar?q=Tianyu+Liu%2C+Yizhe+Zhang%2C+Chris+Brockett%2C+Yi+Mao%2C+Zhifang+Sui%2C+Weizhu+Chen%2C+and+Bill+Dolan%2E+2021%2E+A+token%2Dlevel+reference%2Dfree+hallucination+detection+benchmark+for+free%2Dform+text+generation%2E+arXiv+pre) [our card](https://md.research.elamur.ai/free/en/papers/liu-2022-token-level-reference-free-hallucination/)
11. MosaicML. 2023. Introducing mpt-30b: Raising the bar for open-source foundation models. Accessed: 2023-06-22. [find](https://scholar.google.com/scholar?q=MosaicML%2E+2023%2E+Introducing+mpt%2D30b%3A+Raising+the+bar+for+open%2Dsource+foundation+models%2E+Accessed%3A+2023%2D06%2D22%2E)
12. Ankit Pal. 2022. Promptify: Structured output from llms. [https://github.com/promptslab/](https://github.com/promptslab/) Promptify . Prompt-Engineering components for NLP tasks in Python. [↗](https://github.com/promptslab/)
13. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Proceedings of the Conference on Health, Inference, and Learning, volume 174 of Proceedings of Machine Learning Research, pages 24… [find](https://scholar.google.com/scholar?q=Ankit+Pal%2C+Logesh+Kumar+Umapathi%2C+and+Malaikannan+Sankarasubbu%2E+2022%2E+Medmcqa%3A+A+large%2Dscale+multi%2Dsubject+multi%2Dchoice+dataset+for+medical+domain+question+answering%2E+In+Proceedings+of+the+Conference+) [our card](https://md.research.elamur.ai/free/en/papers/pal-2022-medmcqa-large-scale-multi-subject/)
14. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Sou… [find](https://scholar.google.com/scholar?q=Adam+Paszke%2C+Sam+Gross%2C+Francisco+Massa%2C+Adam+Lerer%2C+James+Bradbury%2C+Gregory+Chanan%2C+Trevor+Killeen%2C+Zeming+Lin%2C+Natalia+Gimelshein%2C+Luca+Antiga%2C+Alban+Desmaison%2C+Andreas+Ko%CC%88pf%2C+Edward+Yang%2C+Zach+DeVi) [our card](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/)
15. Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023a. The refinedweb dataset for falcon llm: Outperforming curated corpora with web data, and web data only. [find](https://scholar.google.com/scholar?q=Guilherme+Penedo%2C+Quentin+Malartic%2C+Daniel+Hesslow%2C+Ruxandra+Cojocaru%2C+Alessandro+Cappelli%2C+Hamza+Alobeidli%2C+Baptiste+Pannier%2C+Ebtesam+Almazrouei%2C+and+Julien+Launay%2E+2023a%2E+The+refinedweb+dataset+for+)
16. Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023b. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:230… [find](https://scholar.google.com/scholar?q=Guilherme+Penedo%2C+Quentin+Malartic%2C+Daniel+Hesslow%2C+Ruxandra+Cojocaru%2C+Alessandro+Cappelli%2C+Hamza+Alobeidli%2C+Baptiste+Pannier%2C+Ebtesam+Almazrouei%2C+and+Julien+Launay%2E+2023b%2E+The+RefinedWeb+dataset+for+)
17. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E)
18. Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. CoRR, abs/2104.07567. [find](https://scholar.google.com/scholar?q=Kurt+Shuster%2C+Spencer+Poff%2C+Moya+Chen%2C+Douwe+Kiela%2C+and+Jason+Weston%2E+2021%2E+Retrieval+augmentation+reduces+hallucination+in+conversation%2E+CoRR%2C+abs%2F2104%2E07567%2E) [our card](https://md.research.elamur.ai/free/en/papers/shuster-2021-retrieval-augmentation-reduces-hallucination-conversation/)
19. K. Singhal, Shekoofeh Azizi, and Tao Tu. 2022. Large language models encode clinical knowledge. ArXiv, abs/2212.13138. [find](https://scholar.google.com/scholar?q=K%2E+Singhal%2C+Shekoofeh+Azizi%2C+and+Tao+Tu%2E+2022%2E+Large+language+models+encode+clinical+knowledge%2E+ArXiv%2C+abs%2F2212%2E13138%2E) [our card](https://md.research.elamur.ai/free/en/papers/singhal-2022-large-language-models-encode-clinical/)
20. K. Singhal, Tao Tu, and Juraj Gottweis. 2023. Towards expert-level medical question answering with large language models. ArXiv, abs/2305.09617. [find](https://scholar.google.com/scholar?q=K%2E+Singhal%2C+Tao+Tu%2C+and+Juraj+Gottweis%2E+2023%2E+Towards+expert%2Dlevel+medical+question+answering+with+large+language+models%2E+ArXiv%2C+abs%2F2305%2E09617%2E) [our card](https://md.research.elamur.ai/free/en/papers/singhal-2023-towards-expert-level-medical-question/)
21. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+Dan+Bikel%2C+Lukas+Blecher%2C+Cristian+Canton+Fe)
22. David Vilares and Carlos Gómez-Rodrı́guez. 2019. Head-qa: A healthcare dataset for complex reasoning. ArXiv, abs/1906.04701. [find](https://scholar.google.com/scholar?q=David+Vilares+and+Carlos+Go%CC%81mez%2DRodr%C4%B1%CC%81guez%2E+2019%2E+Head%2Dqa%3A+A+healthcare+dataset+for+complex+reasoning%2E+ArXiv%2C+abs%2F1906%2E04701%2E)
23. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language model with self generated instructions. [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A%2E+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+2022%2E+Self%2Dinstruct%3A+Aligning+language+model+with+self+generated+instructions%2E)
24. Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2021. Finetuned language models are zero-shot learners. CoRR, abs/2109.01652. [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Maarten+Bosma%2C+Vincent+Y%2E+Zhao%2C+Kelvin+Guu%2C+Adams+Wei+Yu%2C+Brian+Lester%2C+Nan+Du%2C+Andrew+M%2E+Dai%2C+and+Quoc+V%2E+Le%2E+2021%2E+Finetuned+language+models+are+zero%2Dshot+learners%2E+CoRR%2C+abs%2F2109%2E01652%2E)
25. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. [find](https://scholar.google.com/scholar?q=Thomas+Wolf%2C+Lysandre+Debut%2C+Victor+Sanh%2C+Julien+Chaumond%2C+Clement+Delangue%2C+Anthony+Moi%2C+Pier%2Dric+Cistac%2C+Tim+Rault%2C+Re%CC%81mi+Louf%2C+Morgan+Funtowicz%2C+and+Jamie+Brew%2E+2019%2E+Huggingface%E2%80%99s+transformers%3A+St) [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)