---
title: "Unsupervised Domain Adaptation of Contextualized — Han 2019"
description: "Pretrained contextual encoders such as BERT are trained on modern, stylistically narrow text like Wikipedia, news and books."
authors: ["Xiaochuang Han", "Jacob Eisenstein"]
year: 2019
venue: "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)"
source: "https://aclanthology.org/D19-1433/"
doi: "10.18653/v1/d19-1433"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/han-2019-unsupervised-domain-adaptation-contextualized-embeddings/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/han-2019-unsupervised-domain-adaptation-contextualized-embeddings/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/) / Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling

# Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling

**Authors:** [Xiaochuang Han](https://md.research.elamur.ai/free/en/authors/xiaochuang-han/), [Jacob Eisenstein](https://md.research.elamur.ai/free/en/authors/jacob-eisenstein/) · **Year:** 2019 · **Venue:** Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D19-1433/) · [PDF](https://aclanthology.org/D19-1433.pdf) · [DOI 10.18653/v1/d19-1433](https://doi.org/10.18653/v1/d19-1433) · [OpenAlex](https://openalex.org/W2971277088)

**Citations:** 146 citations (openalex-snapshot · 2026-10-14) · [96](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Pretrained contextual encoders such as BERT are trained on modern, stylistically narrow text like Wikipedia, news and books. It is unclear how well they work on very different text when labeled data exists only in a standard source domain. Han and Eisenstein study this unsupervised domain adaptation problem for sequence labeling. Their main case is part-of-speech tagging of Early Modern English: they train on the Penn Treebank and test on PPCEME. A secondary case is named entity recognition on Twitter (WNUT 2016).

Their method, AdaptaBERT, first continues BERT's masked language modeling on unlabeled target-domain text, then fine-tunes on labeled source data. Plain BERT already matches the strongest earlier feature-based adaptation approach on historical text. The extra domain-tuning step adds clear gains, especially on out-of-vocabulary words, and also improves Twitter NER. Unlike supervised fine-tuning on target labels, it does not hurt source-domain accuracy. This makes it a cheap alternative to pretraining from scratch when target data or compute are scarce.

## Key points
- Data: PTB (20th-century English) is the labeled source and the Penn-Helsinki PPCEME is the historical target. Tags are mapped with the Moon and Baldridge (2007) scheme, and most evaluations use a coarse-grained PTB tagset.
- Domain-adaptive fine-tuning cuts out-of-vocabulary tagging errors on Early Modern English by more than 50%. Archaic pronouns are hard for the unadapted baseline: a BERT tagger misses 216 of 272 instances of *thee*.
- On WNUT NER, task-tuned BERT scores 57.7 F1, and AdaptaBERT variants reach up to 62.8 F1. The 2016 shared-task winner (Limsopatham and Collier) scored 65.9.
- AdaptaBERT keeps CoNLL F1 at about 97.6–97.8. BERT fine-tuned on in-domain WNUT labels drops to 80.9, a sign of catastrophic forgetting.
- Trained models are released in the AdaptaBERT GitHub repository (xhan77/AdaptaBERT).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Named entity recognition](https://md.research.elamur.ai/free/en/topics/named-entity-recognition/) · [Part-of-speech tagging](https://md.research.elamur.ai/free/en/topics/part-of-speech-tagging/)

## Cited by

Cited by

96 papers of our library cite it (of the 140884 whose bibliographies we read); 4 of them have a page here, 92 not yet.

- [MacBERTh: Development and Evaluation of a Historically Pre-trained Language Model for English (1450-1950)](https://md.research.elamur.ai/free/en/papers/arevalo-2021-macberth-development-evaluation-historically-pre/)
- [Cross-Domain Label-Adaptive Stance Detection](https://md.research.elamur.ai/free/en/papers/hardalov-2021-cross-domain-label-adaptive-stance/)
- [When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Models](https://md.research.elamur.ai/free/en/papers/muller-2021-when-being-unseen-mbert-is/)
- [Neural Unsupervised Domain Adaptation in NLP–-A Survey](https://md.research.elamur.ai/free/en/papers/ramponi-2020-neural-unsupervised-domain-adaptation-nlp/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{han2019unsupervised,
  title = {{Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling}},
  author = {Han, Xiaochuang and Eisenstein, Jacob},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1433},
  url = {https://doi.org/10.18653/v1/d19-1433}
}
```

- [← previous Unsupervised Data Augmentation for Consistency Training](https://md.research.elamur.ai/free/en/papers/xie-2019-unsupervised-data-augmentation-consistency-training/)
- [next → Usage patterns in the development of Hebrew grammatical subjects](https://md.research.elamur.ai/free/en/papers/lingbulk-2019-dattner-usage-patterns-development-hebrew-grammatical/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/)

## References

46 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Firoj Alam, Shafiq Joty, and Muhammad Imran. 2018. Domain adaptation with adversarial training and graph embeddings. In Proceedings of the Association for Computational Linguistics (ACL), pages 1077–1087. [find](https://scholar.google.com/scholar?q=Firoj+Alam%2C+Shafiq+Joty%2C+and+Muhammad+Imran%2E+2018%2E+Domain+adaptation+with+adversarial+training+and+graph+embeddings%2E+In+Proceedings+of+the+Association+for+Computational+Linguistics+%28ACL%29%2C+pages+1077%E2%80%931)
2. Martin Arjovsky, Soumith Chintala, and Léon Bottou. 2017. Wasserstein generative adversarial networks. In Proceedings of the International Conference on Machine Learning (ICML), pages 214–223. [find](https://scholar.google.com/scholar?q=Martin+Arjovsky%2C+Soumith+Chintala%2C+and+Le%CC%81on+Bottou%2E+2017%2E+Wasserstein+generative+adversarial+networks%2E+In+Proceedings+of+the+International+Conference+on+Machine+Learning+%28ICML%29%2C+pages+214%E2%80%93223%2E)
3. Timothy Baldwin, Paul Cook, Marco Lui, Andrew MacKinlay, and Li Wang. 2013. How noisy social media text, how diffrnt social media sources. In Proceedings of the 6th International Joint Conference on Natural Language Processing (IJCNLP 2013), pages 356–364. [find](https://scholar.google.com/scholar?q=Timothy+Baldwin%2C+Paul+Cook%2C+Marco+Lui%2C+Andrew+MacKinlay%2C+and+Li+Wang%2E+2013%2E+How+noisy+social+media+text%2C+how+diffrnt+social+media+sources%2E+In+Proceedings+of+the+6th+International+Joint+Conference+on+N) [our card](https://md.research.elamur.ai/free/en/papers/baldwin-2013-how-noisy-social-media-text/)
4. Alistair Baron and Paul Rayson. 2008. Vard2: A tool for dealing with spelling variation in historical corpora. In Postgraduate conference in corpus linguistics. [find](https://scholar.google.com/scholar?q=Alistair+Baron+and+Paul+Rayson%2E+2008%2E+Vard2%3A+A+tool+for+dealing+with+spelling+variation+in+historical+corpora%2E+In+Postgraduate+conference+in+corpus+linguistics%2E)
5. Iz Beltagy, Arman Cohan, and Kyle Lo. 2019. SciBERT: Pretrained contextualized embeddings for scientific text. arXiv preprint arXiv:1903.10676. [find](https://scholar.google.com/scholar?q=Iz+Beltagy%2C+Arman+Cohan%2C+and+Kyle+Lo%2E+2019%2E+SciBERT%3A+Pretrained+contextualized+embeddings+for+scientific+text%2E+arXiv+preprint+arXiv%3A1903%2E10676%2E) [our card](https://md.research.elamur.ai/free/en/papers/beltagy-2019-scibert-pretrained-language-model-scientific/)
6. Susan L Bryant, Andrea Forte, and Amy Bruckman. 2005. Becoming Wikipedian: transformation of participation in a collaborative online encyclopedia. In Proceedings of the 2005 international ACM SIGGROUP conference on Supporting group work, pages 1–10. ACM. [find](https://scholar.google.com/scholar?q=Susan+L+Bryant%2C+Andrea+Forte%2C+and+Amy+Bruckman%2E+2005%2E+Becoming+Wikipedian%3A+transformation+of+participation+in+a+collaborative+online+encyclopedia%2E+In+Proceedings+of+the+2005+international+ACM+SIGGROUP)
7. Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. 2013. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005. [find](https://scholar.google.com/scholar?q=Ciprian+Chelba%2C+Tomas+Mikolov%2C+Mike+Schuster%2C+Qi+Ge%2C+Thorsten+Brants%2C+Phillipp+Koehn%2C+and+Tony+Robinson%2E+2013%2E+One+billion+word+benchmark+for+measuring+progress+in+statistical+language+modeling%2E+arXiv) [our card](https://md.research.elamur.ai/free/en/papers/chelba-2013-one-billion-word-benchmark-measuring/)
8. Andrew M Dai and Quoc V Le. 2015. Semi-supervised sequence learning. In Neural Information Processing Systems (NIPS), pages 3079–3087. [find](https://scholar.google.com/scholar?q=Andrew+M+Dai+and+Quoc+V+Le%2E+2015%2E+Semi%2Dsupervised+sequence+learning%2E+In+Neural+Information+Processing+Systems+%28NIPS%29%2C+pages+3079%E2%80%933087%2E)
9. Xiang Dai, Sarvnaz Karimi, Ben Hachey, and Cecile Paris. 2019. Using similarity measures to select pre-training data for NER. In Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL). [find](https://scholar.google.com/scholar?q=Xiang+Dai%2C+Sarvnaz+Karimi%2C+Ben+Hachey%2C+and+Cecile+Paris%2E+2019%2E+Using+similarity+measures+to+select+pre%2Dtraining+data+for+NER%2E+In+Proceedings+of+the+North+American+Chapter+of+the+Association+for+Comput)
10. Hal Daumé III and Daniel Marcu. 2006. Domain adaptation for statistical classifiers. Journal of Artificial Intelligence Research, 26:101–126. [find](https://scholar.google.com/scholar?q=Hal+Daume%CC%81+III+and+Daniel+Marcu%2E+2006%2E+Domain+adaptation+for+statistical+classifiers%2E+Journal+of+Artificial+Intelligence+Research%2C+26%3A101%E2%80%93126%2E)
11. Stefania Degaetano-Ortlieb. 2018. Stylistic variation over 200 years of court proceedings according to gender and social class. In Proceedings of the Second Workshop on Stylistic Variation, pages 1–10. [find](https://scholar.google.com/scholar?q=Stefania+Degaetano%2DOrtlieb%2E+2018%2E+Stylistic+variation+over+200+years+of+court+proceedings+according+to+gender+and+social+class%2E+In+Proceedings+of+the+Second+Workshop+on+Stylistic+Variation%2C+pages+1%E2%80%9310)
12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL). [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+the+North+American+Chapter+o)
13. Jacob Eisenstein. 2013. What to do about bad language on the internet. In Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 359–369. [find](https://scholar.google.com/scholar?q=Jacob+Eisenstein%2E+2013%2E+What+to+do+about+bad+language+on+the+internet%2E+In+Proceedings+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics+%28NAACL%29%2C+pages+359%E2%80%93369%2E) [our card](https://md.research.elamur.ai/free/en/papers/eisenstein-2013-what-do-about-bad-language/)
14. Meng Fang and Trevor Cohn. 2017. Model transfer for tagging low-resource languages using a bilingual dictionary. In Proceedings of the Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=Meng+Fang+and+Trevor+Cohn%2E+2017%2E+Model+transfer+for+tagging+low%2Dresource+languages+using+a+bilingual+dictionary%2E+In+Proceedings+of+the+Association+for+Computational+Linguistics+%28ACL%29%2E)
15. Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. 2016. Domain-adversarial training of neural networks. The Journal of Machine Learning Research, 17(59):1–35. [find](https://scholar.google.com/scholar?q=Yaroslav+Ganin%2C+Evgeniya+Ustinova%2C+Hana+Ajakan%2C+Pascal+Germain%2C+Hugo+Larochelle%2C+Franc%CC%A7ois+Laviolette%2C+Mario+Marchand%2C+and+Victor+Lempitsky%2E+2016%2E+Domain%2Dadversarial+training+of+neural+networks%2E+The+J)
16. Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou. 2018. Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences, 115(16):E3635–E3644. [find](https://scholar.google.com/scholar?q=Nikhil+Garg%2C+Londa+Schiebinger%2C+Dan+Jurafsky%2C+and+James+Zou%2E+2018%2E+Word+embeddings+quantify+100+years+of+gender+and+ethnic+stereotypes%2E+Proceedings+of+the+National+Academy+of+Sciences%2C+115%2816%29%3AE3635%E2%80%93E)
17. Bo Han, Paul Cook, and Timothy Baldwin. 2012. Automatically constructing a normalisation dictionary for microblogs. In Proceedings of Empirical Methods for Natural Language Processing (EMNLP), pages 421–432. [find](https://scholar.google.com/scholar?q=Bo+Han%2C+Paul+Cook%2C+and+Timothy+Baldwin%2E+2012%2E+Automatically+constructing+a+normalisation+dictionary+for+microblogs%2E+In+Proceedings+of+Empirical+Methods+for+Natural+Language+Processing+%28EMNLP%29%2C+pages+4)
18. Martin Hilpert and Stefan Th Gries. 2016. Quantitative approaches to diachronic corpus linguistics. The Cambridge handbook of English historical linguistics, pages 36–53. [find](https://scholar.google.com/scholar?q=Martin+Hilpert+and+Stefan+Th+Gries%2E+2016%2E+Quantitative+approaches+to+diachronic+corpus+linguistics%2E+The+Cambridge+handbook+of+English+historical+linguistics%2C+pages+36%E2%80%9353%2E)
19. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the Association for Computational Linguistics (ACL), pages 328–339. [find](https://scholar.google.com/scholar?q=Jeremy+Howard+and+Sebastian+Ruder%2E+2018%2E+Universal+language+model+fine%2Dtuning+for+text+classification%2E+In+Proceedings+of+the+Association+for+Computational+Linguistics+%28ACL%29%2C+pages+328%E2%80%93339%2E)
20. Young-Bum Kim, Karl Stratos, and Dongchan Kim. 2017. Adversarial adaptation of synthetic or stale data. In Proceedings of the Association for Computational Linguistics (ACL), pages 1297–1307. [find](https://scholar.google.com/scholar?q=Young%2DBum+Kim%2C+Karl+Stratos%2C+and+Dongchan+Kim%2E+2017%2E+Adversarial+adaptation+of+synthetic+or+stale+data%2E+In+Proceedings+of+the+Association+for+Computational+Linguistics+%28ACL%29%2C+pages+1297%E2%80%931307%2E)
21. Anthony Kroch, Beatrice Santorini, and Ariel Diertani. 2004. Penn-Helsinki Parsed Corpus of Early Modern English. [http://www.ling.upenn.edu/hist-corpora/](http://www.ling.upenn.edu/hist-corpora/) PPCEME-RELEASE-2/index.html. [↗](http://www.ling.upenn.edu/hist-corpora/)
22. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. BioBERT: pre-trained biomedical language representation model for biomedical text mining. arXiv preprint arXiv:1901.08746. [find](https://scholar.google.com/scholar?q=Jinhyuk+Lee%2C+Wonjin+Yoon%2C+Sungdong+Kim%2C+Donghyeon+Kim%2C+Sunkyu+Kim%2C+Chan+Ho+So%2C+and+Jaewoo+Kang%2E+2019%2E+BioBERT%3A+pre%2Dtrained+biomedical+language+representation+model+for+biomedical+text+mining%2E+arXiv+pr) [our card](https://md.research.elamur.ai/free/en/papers/lee-2019-biobert-pre-trained-biomedical-language/)
23. Nut Limsopatham and Nigel Collier. 2016. Bidirectional LSTM for named entity recognition in twitter messages. In Proceedings of the 2nd Workshop on Noisy User-generated Text (WNUT), pages 145–152, Osaka, Japan. [find](https://scholar.google.com/scholar?q=Nut+Limsopatham+and+Nigel+Collier%2E+2016%2E+Bidirectional+LSTM+for+named+entity+recognition+in+twitter+messages%2E+In+Proceedings+of+the+2nd+Workshop+on+Noisy+User%2Dgenerated+Text+%28WNUT%29%2C+pages+145%E2%80%93152%2C+Osa)
24. The COLING 2016 Organizing Committee. [find](https://scholar.google.com/scholar?q=The+COLING+2016+Organizing+Committee%2E)
25. Bill Yuchen Lin and Wei Lu. 2018. Neural adaptation layers for cross-domain named entity recognition. In Proceedings of Empirical Methods for Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Bill+Yuchen+Lin+and+Wei+Lu%2E+2018%2E+Neural+adaptation+layers+for+cross%2Ddomain+named+entity+recognition%2E+In+Proceedings+of+Empirical+Methods+for+Natural+Language+Processing+%28EMNLP%29%2E)
26. Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313–330. [find](https://scholar.google.com/scholar?q=Mitchell+P%2E+Marcus%2C+Mary+Ann+Marcinkiewicz%2C+and+Beatrice+Santorini%2E+1993%2E+Building+a+large+annotated+corpus+of+English%3A+The+Penn+Treebank%2E+Computational+Linguistics%2C+19%282%29%3A313%E2%80%93330%2E) [our card](https://md.research.elamur.ai/free/en/papers/bank-1993-marcus-building-large-annotated-corpus-english/)
27. Michael McCloskey and Neal J Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109– 165. Elsevier. [find](https://scholar.google.com/scholar?q=Michael+McCloskey+and+Neal+J+Cohen%2E+1989%2E+Catastrophic+interference+in+connectionist+networks%3A+The+sequential+learning+problem%2E+In+Psychology+of+learning+and+motivation%2C+volume+24%2C+pages+109%E2%80%93+165%2E+Els)
28. Jean-Baptiste Michel, Yuan Kui Shen, Aviva Presser Aiden, Adrian Veres, Matthew K Gray, Joseph P Pickett, Dale Hoiberg, Dan Clancy, Peter Norvig, Jon Orwant, et al. 2011. Quantitative analysis of culture using millions of digitized books. Science, 331(6014):176–182. [find](https://scholar.google.com/scholar?q=Jean%2DBaptiste+Michel%2C+Yuan+Kui+Shen%2C+Aviva+Presser+Aiden%2C+Adrian+Veres%2C+Matthew+K+Gray%2C+Joseph+P+Pickett%2C+Dale+Hoiberg%2C+Dan+Clancy%2C+Peter+Norvig%2C+Jon+Orwant%2C+et+al%2E+2011%2E+Quantitative+analysis+of+cult)
29. Taesun Moon and Jason Baldridge. 2007. Part-of-speech tagging for middle English through alignment and projection of parallel diachronic texts. In Proceedings of Empirical Methods for Natural Language Processing (EMNLP), pages 390–399. [find](https://scholar.google.com/scholar?q=Taesun+Moon+and+Jason+Baldridge%2E+2007%2E+Part%2Dof%2Dspeech+tagging+for+middle+English+through+alignment+and+projection+of+parallel+diachronic+texts%2E+In+Proceedings+of+Empirical+Methods+for+Natural+Language) [our card](https://md.research.elamur.ai/free/en/papers/aclbulk-2007-moon-part-speech-tagging-middle-english/)
30. Lili Mou, Zhao Meng, Rui Yan, Ge Li, Yan Xu, Lu Zhang, and Zhi Jin. 2016. How transferable are neural networks in NLP applications? In Proceedings of Empirical Methods for Natural Language Processing (EMNLP), pages 479–489. [find](https://scholar.google.com/scholar?q=Lili+Mou%2C+Zhao+Meng%2C+Rui+Yan%2C+Ge+Li%2C+Yan+Xu%2C+Lu+Zhang%2C+and+Zhi+Jin%2E+2016%2E+How+transferable+are+neural+networks+in+NLP+applications%3F+In+Proceedings+of+Empirical+Methods+for+Natural+Language+Processing+)
31. Aditi Muralidharan and Marti A Hearst. 2013. Supporting exploratory text analysis in literature study. Literary and linguistic computing, 28(2):283–295. [find](https://scholar.google.com/scholar?q=Aditi+Muralidharan+and+Marti+A+Hearst%2E+2013%2E+Supporting+exploratory+text+analysis+in+literature+study%2E+Literary+and+linguistic+computing%2C+28%282%29%3A283%E2%80%93295%2E)
32. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL). [find](https://scholar.google.com/scholar?q=Matthew+E+Peters%2C+Mark+Neumann%2C+Mohit+Iyyer%2C+Matt+Gardner%2C+Christopher+Clark%2C+Kenton+Lee%2C+and+Luke+Zettlemoyer%2E+2018%2E+Deep+contextualized+word+representations%2E+In+Proceedings+of+the+North+American+Cha)
33. Matthew E. Peters, Sebastian Ruder, and Noah A. Smith. 2019. To tune or not to tune? adapting pre-trained representations to diverse tasks. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), pages 7–14, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Matthew+E%2E+Peters%2C+Sebastian+Ruder%2C+and+Noah+A%2E+Smith%2E+2019%2E+To+tune+or+not+to+tune%3F+adapting+pre%2Dtrained+representations+to+diverse+tasks%2E+In+Proceedings+of+the+4th+Workshop+on+Representation+Learnin)
34. Sanjay Purushotham, Wilka Carvalho, Tanachat Ni-lanon, and Yan Liu. 2017. Variational recurrent adversarial deep domain adaptation. In Proceedings of the International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Sanjay+Purushotham%2C+Wilka+Carvalho%2C+Tanachat+Ni%2Dlanon%2C+and+Yan+Liu%2E+2017%2E+Variational+recurrent+adversarial+deep+domain+adaptation%2E+In+Proceedings+of+the+International+Conference+on+Learning+Represent)
35. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. Technical report, OpenAI. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E+Technical+report%2C+OpenAI%2E)
36. Alexander Robertson and Sharon Goldwater. 2018. Evaluating historical text normalization systems: How well do they generalize? In Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 720– 725. [find](https://scholar.google.com/scholar?q=Alexander+Robertson+and+Sharon+Goldwater%2E+2018%2E+Evaluating+historical+text+normalization+systems%3A+How+well+do+they+generalize%3F+In+Proceedings+of+the+North+American+Chapter+of+the+Association+for+Compu) [our card](https://md.research.elamur.ai/free/en/papers/robertson-2018-evaluating-historical-text-normalization-systems/)
37. Benjamin Strauss, Bethany Toma, Alan Ritter, Marie-Catherine De Marneffe, and Wei Xu. 2016. Results of the wnut16 named entity recognition shared task. In Proceedings of the 2nd Workshop on Noisy User-generated Text (WNUT), pages 138–144. [find](https://scholar.google.com/scholar?q=Benjamin+Strauss%2C+Bethany+Toma%2C+Alan+Ritter%2C+Marie%2DCatherine+De+Marneffe%2C+and+Wei+Xu%2E+2016%2E+Results+of+the+wnut16+named+entity+recognition+shared+task%2E+In+Proceedings+of+the+2nd+Workshop+on+Noisy+User)
38. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Conference on Natural Language Learning (CoNLL), pages 142–147. [find](https://scholar.google.com/scholar?q=Erik+F%2E+Tjong+Kim+Sang+and+Fien+De+Meulder%2E+2003%2E+Introduction+to+the+CoNLL%2D2003+shared+task%3A+Language%2Dindependent+named+entity+recognition%2E+In+Proceedings+of+the+Conference+on+Natural+Language+Learni)
39. Romain Vuillemot, Tanya Clement, Catherine Plaisant, and Amit Kumar. 2009. What’s being said near “Martha”? Exploring name entities in literary text collections. In Symposium on Visual Analytics Science and Technology, pages 107–114. IEEE. [find](https://scholar.google.com/scholar?q=Romain+Vuillemot%2C+Tanya+Clement%2C+Catherine+Plaisant%2C+and+Amit+Kumar%2E+2009%2E+What%E2%80%99s+being+said+near+%E2%80%9CMartha%E2%80%9D%3F+Exploring+name+entities+in+literary+text+collections%2E+In+Symposium+on+Visual+Analytics+Scien)
40. Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V. Le. 2019. Unsupervised data augmentation. arXiv preprint arXiv:1904.12848. [find](https://scholar.google.com/scholar?q=Qizhe+Xie%2C+Zihang+Dai%2C+Eduard+Hovy%2C+Minh%2DThang+Luong%2C+and+Quoc+V%2E+Le%2E+2019%2E+Unsupervised+data+augmentation%2E+arXiv+preprint+arXiv%3A1904%2E12848%2E) [our card](https://md.research.elamur.ai/free/en/papers/xie-2019-unsupervised-data-augmentation-consistency-training/)
41. Yi Yang and Jacob Eisenstein. 2015. Unsupervised multi-domain adaptation with feature embeddings. In Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL). [find](https://scholar.google.com/scholar?q=Yi+Yang+and+Jacob+Eisenstein%2E+2015%2E+Unsupervised+multi%2Ddomain+adaptation+with+feature+embeddings%2E+In+Proceedings+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics+%28NAACL%29%2E)
42. Yi Yang and Jacob Eisenstein. 2016. Part-of-speech tagging for historical English. In Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL). [find](https://scholar.google.com/scholar?q=Yi+Yang+and+Jacob+Eisenstein%2E+2016%2E+Part%2Dof%2Dspeech+tagging+for+historical+English%2E+In+Proceedings+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics+%28NAACL%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/aclbulk-2016-yang-part-speech-tagging-historical-english/)
43. Zhilin Yang, Ruslan Salakhutdinov, and William W Cohen. 2016. Transfer learning for sequence tagging with hierarchical recurrent networks. In Proceedings of the International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Ruslan+Salakhutdinov%2C+and+William+W+Cohen%2E+2016%2E+Transfer+learning+for+sequence+tagging+with+hierarchical+recurrent+networks%2E+In+Proceedings+of+the+International+Conference+on+Learning+Re)
44. Dani Yogatama, Cyprien de Masson d’Autume, Jerome Connor, Tomas Kocisky, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer, et al. 2019. Learning and evaluating general linguistic intelligence. arXiv preprint arXiv:1901.11373. [find](https://scholar.google.com/scholar?q=Dani+Yogatama%2C+Cyprien+de+Masson+d%E2%80%99Autume%2C+Jerome+Connor%2C+Tomas+Kocisky%2C+Mike+Chrzanowski%2C+Lingpeng+Kong%2C+Angeliki+Lazaridou%2C+Wang+Ling%2C+Lei+Yu%2C+Chris+Dyer%2C+et+al%2E+2019%2E+Learning+and+evaluating+genera)
45. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the International Conference on Computer Vision (ICCV), pages 1… [find](https://scholar.google.com/scholar?q=Yukun+Zhu%2C+Ryan+Kiros%2C+Rich+Zemel%2C+Ruslan+Salakhutdinov%2C+Raquel+Urtasun%2C+Antonio+Torralba%2C+and+Sanja+Fidler%2E+2015%2E+Aligning+books+and+movies%3A+Towards+story%2Dlike+visual+explanations+by+watching+movies+) [our card](https://md.research.elamur.ai/free/en/papers/zhu-2015-aligning-books-movies-towards-story/)
46. Yftah Ziser and Roi Reichart. 2018. Pivot based language modeling for improved neural domain adaptation. In Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 1241–1251. [find](https://scholar.google.com/scholar?q=Yftah+Ziser+and+Roi+Reichart%2E+2018%2E+Pivot+based+language+modeling+for+improved+neural+domain+adaptation%2E+In+Proceedings+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics+%28)