---
title: "UER: An Open-Source Toolkit for Pre-training Models (2019)"
description: "The authors present UER (Universal Encoder Representations), a modular PyTorch toolkit that builds… 17 references, cited by 41 papers here."
authors: ["Zhe Zhao", "Hui Chen", "Jinbin Zhang", "Xin Zhao", "Tao Liu", "Wei Lu", "Xi Chen", "Haotang Deng", "Qi Ju", "Xiaoyong Du"]
year: 2019
venue: "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations"
source: "https://aclanthology.org/D19-3041/"
doi: "10.18653/v1/d19-3041"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhao-2019-uer-open-source-toolkit-pre/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhao-2019-uer-open-source-toolkit-pre/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/) / UER: An Open-Source Toolkit for Pre-training Models

# UER: An Open-Source Toolkit for Pre-training Models

**Authors:** [Zhe Zhao](https://md.research.elamur.ai/free/en/authors/zhe-zhao/), [Hui Chen](https://md.research.elamur.ai/free/en/authors/hui-chen/), Jinbin Zhang, [Xin Zhao](https://md.research.elamur.ai/free/en/authors/xin-zhao/), [Tao Liu](https://md.research.elamur.ai/free/en/authors/tao-liu/), [Wei Lu](https://md.research.elamur.ai/free/en/authors/wei-lu/), [Xi Chen](https://md.research.elamur.ai/free/en/authors/xi-chen/), [Haotang Deng](https://md.research.elamur.ai/free/en/authors/haotang-deng/), [Qi Ju](https://md.research.elamur.ai/free/en/authors/qi-ju/) and [Xiaoyong Du](https://md.research.elamur.ai/free/en/authors/xiaoyong-du/) · **Year:** 2019 · **Venue:** [Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2019](https://md.research.elamur.ai/free/en/venues/emnlp/2019/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D19-3041/) · [PDF](https://aclanthology.org/D19-3041.pdf) · [DOI 10.18653/v1/d19-3041](https://doi.org/10.18653/v1/d19-3041) · [OpenAlex](https://openalex.org/W2970390493)

**Citations:** 99 citations (OpenAlex, 2026-10-14) · [41](https://md.research.elamur.ai/free/en/papers/zhao-2019-uer-open-source-toolkit-pre/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary

Pre-training models such as ELMO, GPT and BERT differ in their training corpus, encoder architecture, pre-training target and fine-tuning strategy. No single configuration suits every downstream task or compute budget, yet most open-source code covers only one or a few models. The authors present UER (Universal Encoder Representations), a modular PyTorch toolkit that builds models from interchangeable parts in four components: subencoder, encoder, target, and downstream fine-tuning. With a few lines of code, users can rebuild established models like BERT or combine modules into designs nobody has tried yet. Several targets can be weighted together, and distributed training is supported. Using UER, the team released a Chinese model zoo covering varied corpora, encoders and targets. Their BERT reimplementation performs on par with the HuggingFace and ERNIE implementations. Choosing in-domain corpora, suitable targets or lighter encoders can beat or nearly equal the BERT baseline. The toolkit therefore makes it cheaper to reproduce published models and to explore new pre-training choices.

## Key points

- **Modules:**
  - Encoders: LSTM, GRU, CNN, GatedCNN, AttentionNN, Transformer, and their stacks.
  - Targets: LM, MLM (cloze), autoencoder, NSP, NMT, and CLS.
  - Chinese subword features: radicals and pinyin, with each character treated as a word.
- **GLUE dev set:** UER's BERT reimplementation matches HuggingFace overall. It is higher on RTE (66.8 vs 61.0) and QNLI (91.5 vs 89.0) and slightly lower on SST-2 (92.4 vs 93.0).
- **ERNIE benchmark:** UER is comparable to ERNIE and better on MSRA-NER (93.6 vs 92.6).
- **Amazon review corpus pre-training:** Starting from BERT-base Chinese, further pre-training with the CLS target raised ChnSentiCorp from 94.3 to 95.8 and Douban from 87.5 to 88.3.
- **Lighter encoder:** A pre-trained 2-layer LSTM (embedding size 512, hidden size 1024) reached 96.9 on Online shopping and 94.5 on ChnSentiCorp, slightly above BERT (96.3 and 94.3). It scored 86.5 on Douban versus BERT's 87.5. Code and model zoo: github.com/dbiir/UER-py.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) — University, Beijing, China · 315 works · 43,268 citations
- [Tencent](https://md.research.elamur.ai/free/en/companies/tencent/) — Company, Shenzhen, China · 163 works · 21,032 citations
- [Renmin University of China](https://md.research.elamur.ai/free/en/universities/renmin-university-of-china/) — University, Beijing, China · 139 works · 17,752 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Zhe Zhao, Hui Chen, Jinbin Zhang, Xin Zhao, Tao Liu, Wei Lu, Xi Chen, Haotang Deng, Qi Ju and Xiaoyong Du. 2019. UER: An Open-Source Toolkit for Pre-training Models. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhao2019uer,
  title = {{UER: An Open-Source Toolkit for Pre-training Models}},
  author = {Zhe Zhao, Hui Chen, Jinbin Zhang, Xin Zhao, Tao Liu, Wei Lu, Xi Chen, Haotang Deng, Qi Ju and Xiaoyong Du},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations},
  year = {2019},
  doi = {10.18653/v1/d19-3041},
  url = {https://doi.org/10.18653/v1/d19-3041}
}
```

- [← previous Investigating Entity Knowledge in BERT with Simple Neural End-To-End Entity Linking](https://md.research.elamur.ai/free/en/papers/broscheit-2019-investigating-entity-knowledge-bert-simple/)
- [next → OSIAN: Open Source International Arabic News Corpus - Preparation and Integration into the CLARIN-infrastructure](https://md.research.elamur.ai/free/en/papers/zeroual-2019-osian-open-source-international-arabic/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/)

## References

17 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. [Deep contextualized word representations](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/). arXiv preprint arXiv:1802.05365.
2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). arXiv preprint arXiv:1409.0473.
3. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E)
4. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. [Supervised learning of universal sentence representations from natural language inference data](https://md.research.elamur.ai/free/en/papers/conneau-2017-supervised-learning-universal-sentence-representations/). EMNLP.
5. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In NIPS.
6. Andrew M Dai and Quoc V Le. 2015. Semi-supervised sequence learning. In NIPS. [find](https://scholar.google.com/scholar?q=Andrew+M+Dai+and+Quoc+V+Le%2E+2015%2E+Semi%2Dsupervised+sequence+learning%2E+In+NIPS%2E)
7. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In ICLR.
8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805.
9. Jie Yang and Yue Zhang. 2018. [Ncrf++: An open-source neural sequence labeling toolkit](https://md.research.elamur.ai/free/en/papers/yang-2018-ncrf-open-source-neural-sequence/). arXiv preprint arXiv:1806.05626.
10. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8). [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+1997%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%2E)
11. Xiang Zhang and Yann LeCun. 2017. Which encoding is the best for text classification in chinese, english, japanese and korean? [find](https://scholar.google.com/scholar?q=Xiang+Zhang+and+Yann+LeCun%2E+2017%2E+Which+encoding+is+the+best+for+text+classification+in+chinese%2C+english%2C+japanese+and+korean%3F)
12. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146. [find](https://scholar.google.com/scholar?q=Jeremy+Howard+and+Sebastian+Ruder%2E+2018%2E+Universal+language+model+fine%2Dtuning+for+text+classification%2E+arXiv+preprint+arXiv%3A1801%2E06146%2E)
13. Zhe Zhao, Tao Liu, Shen Li, Bofang Li, and Xiaoyong Du. 2017. Ngram2vec: Learning improved word representations from ngram co-occurrence statistics. EMNLP. [find](https://scholar.google.com/scholar?q=Zhe+Zhao%2C+Tao+Liu%2C+Shen+Li%2C+Bofang+Li%2C+and+Xiaoyong+Du%2E+2017%2E+Ngram2vec%3A+Learning+improved+word+representations+from+ngram+co%2Doccurrence+statistics%2E+EMNLP%2E)
14. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016. Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759. [find](https://scholar.google.com/scholar?q=Armand+Joulin%2C+Edouard+Grave%2C+Piotr+Bojanowski%2C+and+Tomas+Mikolov%2E+2016%2E+Bag+of+tricks+for+efficient+text+classification%2E+arXiv+preprint+arXiv%3A1607%2E01759%2E)
15. Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, and Bo Xu. 2016. [Attention-based bidirectional long short-term memory networks for relation classification](https://md.research.elamur.ai/free/en/papers/zhou-2016-attention-based-bidirectional-long-short/). In ACL., volume 2.
16. Yoon Kim. 2014. [Convolutional neural networks for sentence classification](https://md.research.elamur.ai/free/en/papers/kim-2014-convolutional-neural-networks-sentence-classification/). EMNLP.
17. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In NIPS. [find](https://scholar.google.com/scholar?q=Ryan+Kiros%2C+Yukun+Zhu%2C+Ruslan+R+Salakhutdinov%2C+Richard+Zemel%2C+Raquel+Urtasun%2C+Antonio+Torralba%2C+and+Sanja+Fidler%2E+2015%2E+Skip%2Dthought+vectors%2E+In+NIPS%2E)

*Updated 1 November 2026*