---
title: "CharacterBERT — paper by Boukkouri et al. (2020)"
description: "Their solution, CharacterBERT, replaces BERT's wordpiece embedding… 1.5 points on ChemProt. 35 references, cited by 73 papers here."
authors: ["Hicham El Boukkouri", "Olivier Ferret", "Thomas Lavergne", "Hiroshi Noji", "Pierre Zweigenbaum", "Jun'ichi Tsujii"]
year: 2020
venue: "Proceedings of the 28th International Conference on Computational Linguistics"
source: "https://aclanthology.org/2020.coling-main.609/"
doi: "10.18653/v1/2020.coling-main.609"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/boukkouri-2020-characterbert-reconciling-elmo-bert-word/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/boukkouri-2020-characterbert-reconciling-elmo-bert-word/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/3/) / CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters

# CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters

**Authors:** Hicham El Boukkouri, [Olivier Ferret](https://md.research.elamur.ai/free/en/authors/olivier-ferret/), [Thomas Lavergne](https://md.research.elamur.ai/free/en/authors/thomas-lavergne/), Hiroshi Noji, [Pierre Zweigenbaum](https://md.research.elamur.ai/free/en/authors/pierre-zweigenbaum/) and [Jun'ichi Tsujii](https://md.research.elamur.ai/free/en/authors/jun-ichi-tsujii/) · **Year:** 2020 · **Venue:** [Proceedings of the 28th International Conference on Computational Linguistics](https://md.research.elamur.ai/free/en/venues/coling/) ([COLING 2020](https://md.research.elamur.ai/free/en/venues/coling/2020/)) · **Publisher:** [International Committee on Computational Linguistics (ICCL)](https://md.research.elamur.ai/free/en/publishers/international-committee-on-computational-linguistics-iccl/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.coling-main.609/) · [PDF](https://aclanthology.org/2020.coling-main.609.pdf) · [DOI 10.18653/v1/2020.coling-main.609](https://doi.org/10.18653/v1/2020.coling-main.609) · [OpenAlex](https://openalex.org/W3115462295)

**Citations:** 131 citations (OpenAlex, 2026-10-14) · [73](https://md.research.elamur.ai/free/en/papers/boukkouri-2020-characterbert-reconciling-elmo-bert-word/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Since BERT, most Transformer encoders have kept its wordpiece tokenizer, although the architecture does not require it. When BERT is adapted to a specialized field through continued pre-training, its general-domain vocabulary stays fixed. Domain terms then get broken into many fragments that carry little meaning. The authors show this on medical text: about 13% of token occurrences stay whole under a medical vocabulary but are split by BERT's.

Their solution, CharacterBERT, replaces BERT's wordpiece embedding layer with the Character-CNN module from ELMo and leaves the Transformer layers unchanged. Each word gets one open-vocabulary representation computed from its characters, and the MLM objective predicts whole words. BERT and CharacterBERT are trained in parallel under identical settings: first on general text (English Wikipedia, OpenWebText), then on medical text (MIMIC-III, PMC OA abstracts).

On most clinical and biomedical tasks, the medical CharacterBERT outperforms its BERT twin, which ASO significance tests confirm. It also handles misspellings better. This makes it a simpler, word-level route to domain-specific models without a predefined subword vocabulary.

## Key points
- **Character-CNN design:**
  - Tokens are encoded as UTF-8 bytes (vocabulary of 263 symbols) and truncated to 50 characters.
  - Each character gets a 16-d embedding.
  - Seven 1-d CNNs are max-pooled, followed by two Highway layers and a projection to 768 dimensions.
  - The resulting model is smaller than BERT-base-uncased (≈104.6M vs ≈109.5M parameters).
- **Four models compared:** BERT general, CharacterBERT general, BERT medical and CharacterBERT medical. The medical versions are re-trained from the general ones, each on 16 Tesla V100 GPUs.
- **Medical task results:** CharacterBERT medical ensembles beat BERT medical by roughly 1.5 points on ChemProt, 2 on DDI and 0.5 on MEDNLI and i2b2. It is worse on the small ClinicalSTS set, where it also shows high variance.
- **Robustness to noise:** At 40% character-level noise on the MEDNLI test set, CharacterBERT leads BERT by ≈5 F1, compared with a gap under 1 F1 on clean data.
- **Trade-offs and availability:** Pre-training is slower, but inference is slightly faster. Code and pre-trained models are released at github.com/helboukkouri/character-bert. The study covers English only.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Centre National de la Recherche Scientifique](https://md.research.elamur.ai/free/en/labs/centre-national-de-la-recherche-scientifique/) — Research institute, Paris, France · 104 works · 60,877 citations
- [Laboratoire d'Informatique pour la Mécanique et les Sciences de l'Ingénieur](https://md.research.elamur.ai/free/en/labs/laboratoire-d-informatique-pour-la-mecanique-et-les-sciences-de-l-ingenieur/) — Research institute, Orsay, France · 21 works · 7,271 citations
- [Université Paris-Saclay](https://md.research.elamur.ai/free/en/universities/universite-paris-saclay/) — University, Gif-sur-Yvette, France · 34 works · 5,733 citations
- [National Institute of Advanced Industrial Science and Technology](https://md.research.elamur.ai/free/en/labs/national-institute-of-advanced-industrial-science-and-technology/) — Research institute, Tsukuba, Japan · 13 works · 2,747 citations
- [Artificial Intelligence Research Center](https://md.research.elamur.ai/free/en/labs/artificial-intelligence-research-center/) — Research institute, Tokyo, Japan · 8 works · 1,556 citations
- [Commissariat à l'Énergie Atomique et aux Énergies Alternatives](https://md.research.elamur.ai/free/en/government/commissariat-a-l-energie-atomique-et-aux-energies-alternatives/) — Government body, Paris, France · 5 works · 350 citations
- [Laboratoire d'Intégration des Systèmes et des Technologies](https://md.research.elamur.ai/free/en/government/laboratoire-d-integration-des-systemes-et-des-technologies/) — Government body, Palaiseau, France · 4 works · 297 citations

Also on the byline: [CEA Paris-Saclay - Etablissement de Saclay](https://ror.org/01fv25t22).

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

## Cite

Cite as: Hicham El Boukkouri, Olivier Ferret, Thomas Lavergne, Hiroshi Noji, Pierre Zweigenbaum and Jun'ichi Tsujii. 2020. CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters. Proceedings of the 28th International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{boukkouri2020characterbert,
  title = {{CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters}},
  author = {Hicham {El Boukkouri}, Olivier Ferret, Thomas Lavergne, Hiroshi Noji, Pierre Zweigenbaum and Jun'ichi Tsujii},
  booktitle = {Proceedings of the 28th International Conference on Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.coling-main.609},
  url = {https://doi.org/10.18653/v1/2020.coling-main.609}
}
```

- [← previous Training Question Answering Models From Synthetic Data](https://md.research.elamur.ai/free/en/papers/puri-2020-training-question-answering-models-synthetic/)
- [next → Contextualizing Hate Speech Classifiers with Post-hoc Explanation](https://md.research.elamur.ai/free/en/papers/kennedy-2020-contextualizing-hate-speech-classifiers-post/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/3/)

## References

35 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020. ELECTRA: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555. [find](https://scholar.google.com/scholar?q=Kevin+Clark%2C+Minh%2DThang+Luong%2C+Quoc+V+Le%2C+and+Christopher+D+Manning%2E+2020%2E+ELECTRA%3A+Pre%2Dtraining+text+encoders+as+discriminators+rather+than+generators%2E+arXiv+preprint+arXiv%3A2003%2E10555%2E)
2. Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang, Shijin Wang, and Guoping Hu. 2019. [Pre-training with whole word masking for chinese BERT](https://md.research.elamur.ai/free/en/papers/cui-2019-pre-training-whole-word-masking/). arXiv preprint arXiv:1906.08101.
3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
4. Rotem Dror, Segev Shlomov, and Roi Reichart. 2019. Deep dominance - how to properly compare deep neural models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2773–2785, Florence, Italy, July. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rotem+Dror%2C+Segev+Shlomov%2C+and+Roi+Reichart%2E+2019%2E+Deep+dominance+%2D+how+to+properly+compare+deep+neural+models%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguisti)
5. Hicham El Boukkouri, Olivier Ferret, Thomas Lavergne, and Pierre Zweigenbaum. 2019. Embedding strategies for specialized domains: Application to clinical entity recognition. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 2… [find](https://scholar.google.com/scholar?q=Hicham+El+Boukkouri%2C+Olivier+Ferret%2C+Thomas+Lavergne%2C+and+Pierre+Zweigenbaum%2E+2019%2E+Embedding+strategies+for+specialized+domains%3A+Application+to+clinical+entity+recognition%2E+In+Proceedings+of+the+57th)
6. Emad Elwany, Dave Moore, and Gaurav Oberoi. 2019. Bert goes to law school: Quantifying the competitive advantage of access to large legal corpora in contract understanding. arXiv preprint arXiv:1911.00473. [find](https://scholar.google.com/scholar?q=Emad+Elwany%2C+Dave+Moore%2C+and+Gaurav+Oberoi%2E+2019%2E+Bert+goes+to+law+school%3A+Quantifying+the+competitive+advantage+of+access+to+large+legal+corpora+in+contract+understanding%2E+arXiv+preprint+arXiv%3A1911%2E0)
7. Aaron Gokaslan and Vanya Cohen. 2019. OpenWebText corpus. [http://Skylion007.github.io/](http://Skylion007.github.io/) OpenWebTextCorpus. [↗](http://Skylion007.github.io/)
8. Marı́a Herrero-Zazo, Isabel Segura-Bedmar, Paloma Martı́nez, and Thierry Declerck. 2013. The DDI corpus: An annotated corpus with pharmacological substances and drug–drug interactions. Journal of biomedical informatics, 46(5):914–920. [find](https://scholar.google.com/scholar?q=Mar%C4%B1%CC%81a+Herrero%2DZazo%2C+Isabel+Segura%2DBedmar%2C+Paloma+Mart%C4%B1%CC%81nez%2C+and+Thierry+Declerck%2E+2013%2E+The+DDI+corpus%3A+An+annotated+corpus+with+pharmacological+substances+and+drug%E2%80%93drug+interactions%2E+Journal+of+biom)
9. Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016. [MIMIC-III, a freely accessible critical care database](https://md.research.elamur.ai/free/en/papers/johnson-2016-mimic-iii-freely-accessible-critical/). Scientific data, 3:160035.
10. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410. [https://abci.ai/](https://abci.ai/) [↗](https://abci.ai/)
11. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. 2016. [Character-aware neural language models](https://md.research.elamur.ai/free/en/papers/kim-2015-character-aware-neural-language-models/). In Thirtieth AAAI Conference on Artificial Intelligence.
12. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. [find](https://scholar.google.com/scholar?q=Diederik+P+Kingma+and+Jimmy+Ba%2E+2014%2E+Adam%3A+A+method+for+stochastic+optimization%2E+arXiv+preprint+arXiv%3A1412%2E6980%2E)
13. Martin Krallinger, Obdulia Rabal, Saber A Akhondi, et al. 2017. Overview of the BioCreative VI chemical-protein interaction track. In Proceedings of the sixth BioCreative challenge evaluation workshop, volume 1, pages 141–146. [find](https://scholar.google.com/scholar?q=Martin+Krallinger%2C+Obdulia+Rabal%2C+Saber+A+Akhondi%2C+et+al%2E+2017%2E+Overview+of+the+BioCreative+VI+chemical%2Dprotein+interaction+track%2E+In+Proceedings+of+the+sixth+BioCreative+challenge+evaluation+workshop)
14. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. AL-BERT: A lite BERT for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942. [find](https://scholar.google.com/scholar?q=Zhenzhong+Lan%2C+Mingda+Chen%2C+Sebastian+Goodman%2C+Kevin+Gimpel%2C+Piyush+Sharma%2C+and+Radu+Soricut%2E+2019%2E+AL%2DBERT%3A+A+lite+BERT+for+self%2Dsupervised+learning+of+language+representations%2E+arXiv+preprint+arXiv%3A)
15. Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. 1989. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541–551. [find](https://scholar.google.com/scholar?q=Yann+LeCun%2C+Bernhard+Boser%2C+John+S+Denker%2C+Donnie+Henderson%2C+Richard+E+Howard%2C+Wayne+Hubbard%2C+and+Lawrence+D+Jackel%2E+1989%2E+Backpropagation+applied+to+handwritten+zip+code+recognition%2E+Neural+computati)
16. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://md.research.elamur.ai/free/en/papers/lee-2019-biobert-pre-trained-biomedical-language/). Bioinformatics, 36(4):1234–1240.
17. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+RoBERTa%3A+A+robustly+optimized+BERT+pretraining+approach%2E+a)
18. Minh-Thang Luong and Christopher D. Manning. 2016. [Achieving open vocabulary neural machine translation with hybrid word-character models](https://md.research.elamur.ai/free/en/papers/luong-2016-achieving-open-vocabulary-neural-machine/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1054–1063, Berlin, Germany, August. Ass…
19. Andriy Mnih and Koray Kavukcuoglu. 2013. Learning word embeddings efficiently with noise-contrastive estimation. In Advances in neural information processing systems, pages 2265–2273. [find](https://scholar.google.com/scholar?q=Andriy+Mnih+and+Koray+Kavukcuoglu%2E+2013%2E+Learning+word+embeddings+efficiently+with+noise%2Dcontrastive+estimation%2E+In+Advances+in+neural+information+processing+systems%2C+pages+2265%E2%80%932273%2E)
20. Yifan Peng, Shankai Yan, and Zhiyong Lu. 2019. [Transfer learning in biomedical natural language processing: An evaluation of BERT and ELMo on ten benchmarking datasets](https://md.research.elamur.ai/free/en/papers/peng-2019-transfer-learning-biomedical-natural-language/). In Proceedings of the 2019 Workshop on Biomedical Natural Language Processing (BioNLP 2019), pages 58–65.
21. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. [Deep contextualized word representations](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog…
22. Danish Pruthi, Bhuwan Dhingra, and Zachary C. Lipton. 2019. [Combating adversarial misspellings with robust word recognition](https://md.research.elamur.ai/free/en/papers/pruthi-2019-combating-adversarial-misspellings-robust-word/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5582–5591, Florence, Italy, July. Association for Computational Linguistics.
23. Alexey Romanov and Chaitanya Shivade. 2018. [Lessons from natural language inference in the clinical domain](https://md.research.elamur.ai/free/en/papers/romanov-2018-lessons-natural-language-inference-clinical/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1586– 1596, Brussels, Belgium, October-November. Association for Computational Linguistics.
24. Yuqi Si, Jingqi Wang, Hua Xu, and Kirk Roberts. 2019. Enhancing clinical concept extraction with contextual embeddings. Journal of the American Medical Informatics Association, 26(11):1297–1304. [find](https://scholar.google.com/scholar?q=Yuqi+Si%2C+Jingqi+Wang%2C+Hua+Xu%2C+and+Kirk+Roberts%2E+2019%2E+Enhancing+clinical+concept+extraction+with+contextual+embeddings%2E+Journal+of+the+American+Medical+Informatics+Association%2C+26%2811%29%3A1297%E2%80%931304%2E)
25. Rupesh K Srivastava, Klaus Greff, and Jürgen Schmidhuber. 2015. Training very deep networks. In Advances in neural information processing systems, pages 2377–2385. [find](https://scholar.google.com/scholar?q=Rupesh+K+Srivastava%2C+Klaus+Greff%2C+and+Ju%CC%88rgen+Schmidhuber%2E+2015%2E+Training+very+deep+networks%2E+In+Advances+in+neural+information+processing+systems%2C+pages+2377%E2%80%932385%2E)
26. Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. 2019. [ERNIE 2.0: A continual pre-training framework for language understanding](https://md.research.elamur.ai/free/en/papers/sun-2019-ernie-2-0-continual/). arXiv preprint arXiv:1907.12412.
27. Lichao Sun, Kazuma Hashimoto, Wenpeng Yin, Akari Asai, Jia Li, Philip Yu, and Caiming Xiong. 2020. Adv-BERT: BERT is not robust on misspellings! generating nature adversarial samples on BERT. arXiv preprint arXiv:2003.04985. — [arXiv:2003.04985](https://arxiv.org/abs/2003.04985) [find](https://scholar.google.com/scholar?q=Lichao+Sun%2C+Kazuma+Hashimoto%2C+Wenpeng+Yin%2C+Akari+Asai%2C+Jia+Li%2C+Philip+Yu%2C+and+Caiming+Xiong%2E+2020%2E+Adv%2DBERT%3A+BERT+is+not+robust+on+misspellings%21+generating+nature+adversarial+samples+on+BERT%2E+arXiv+pr)
28. Özlem Uzuner, Brett R South, Shuying Shen, and Scott L DuVall. 2011. 2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text. Journal of the American Medical Informatics Association, 18(5):552– 556. [find](https://scholar.google.com/scholar?q=O%CC%88zlem+Uzuner%2C+Brett+R+South%2C+Shuying+Shen%2C+and+Scott+L+DuVall%2E+2011%2E+2010+i2b2%2FVA+challenge+on+concepts%2C+assertions%2C+and+relations+in+clinical+text%2E+Journal+of+the+American+Medical+Informatics+Associ)
29. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in neural information processing systems, pages 5998–6008.
30. Yanshan Wang, Naveed Afzal, Sunyang Fu, Liwei Wang, Feichen Shen, Majid Rastegar-Mojarad, and Hongfang Liu. 2018a. MedSTS: a resource for clinical semantic textual similarity. Language Resources and Evaluation, pages 1–16. [find](https://scholar.google.com/scholar?q=Yanshan+Wang%2C+Naveed+Afzal%2C+Sunyang+Fu%2C+Liwei+Wang%2C+Feichen+Shen%2C+Majid+Rastegar%2DMojarad%2C+and+Hongfang+Liu%2E+2018a%2E+MedSTS%3A+a+resource+for+clinical+semantic+textual+similarity%2E+Language+Resources+and+E)
31. Yanshan Wang, Naveed Afzal, Sijia Liu, Majid Rastegar-Mojarad, Liwei Wang, Feichen Shen, Sunyang Fu, and Hongfang Liu. 2018b. Overview of the BioCreative/OHNLP challenge 2018 task 2: clinical semantic textual similarity. Proceedings of the BioCreative/OHNLP Challenge, 2018. [find](https://scholar.google.com/scholar?q=Yanshan+Wang%2C+Naveed+Afzal%2C+Sijia+Liu%2C+Majid+Rastegar%2DMojarad%2C+Liwei+Wang%2C+Feichen+Shen%2C+Sunyang+Fu%2C+and+Hongfang+Liu%2E+2018b%2E+Overview+of+the+BioCreative%2FOHNLP+challenge+2018+task+2%3A+clinical+semantic)
32. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. [find](https://scholar.google.com/scholar?q=Yonghui+Wu%2C+Mike+Schuster%2C+Zhifeng+Chen%2C+Quoc+V+Le%2C+Mohammad+Norouzi%2C+Wolfgang+Macherey%2C+Maxim+Krikun%2C+Yuan+Cao%2C+Qin+Gao%2C+Klaus+Macherey%2C+et+al%2E+2016%2E+Google%E2%80%99s+neural+machine+translation+system%3A+Bridg)
33. Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. 2019. Large batch optimization for deep learning: Training BERT in 76 minutes. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Yang+You%2C+Jing+Li%2C+Sashank+Reddi%2C+Jonathan+Hseu%2C+Sanjiv+Kumar%2C+Srinadh+Bhojanapalli%2C+Xiaodan+Song%2C+James+Demmel%2C+Kurt+Keutzer%2C+and+Cho%2DJui+Hsieh%2E+2019%2E+Large+batch+optimization+for+deep+learning%3A+Trai)
34. Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: Enhanced language representation with informative entities. arXiv preprint arXiv:1905.07129. [find](https://scholar.google.com/scholar?q=Zhengyan+Zhang%2C+Xu+Han%2C+Zhiyuan+Liu%2C+Xin+Jiang%2C+Maosong+Sun%2C+and+Qun+Liu%2E+2019%2E+ERNIE%3A+Enhanced+language+representation+with+informative+entities%2E+arXiv+preprint+arXiv%3A1905%2E07129%2E)
35. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. [Aligning books and movies: Towards story-like visual explanations by watching movies and reading books](https://md.research.elamur.ai/free/en/papers/zhu-2015-aligning-books-movies-towards-story/). In Proceedings of the IEEE international conference on computer vision, pages 19–…

*Updated 24 October 2026*