---
title: "How Transferable are Neural Networks in NLP Applications? (2016)"
description: "The authors run a controlled study on six public datasets in two groups. 25 references, cited by 68 papers here."
authors: ["Lili Mou", "Zhao Meng", "Rui Yan", "Ge Li", "Yan Xu", "Lu Zhang", "Zhi Jin"]
year: 2016
venue: "Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/D16-1046/"
doi: "10.18653/v1/d16-1046"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/mou-2016-how-transferable-are-neural-networks/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mou-2016-how-transferable-are-neural-networks/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/2/) / How Transferable are Neural Networks in NLP Applications?

# How Transferable are Neural Networks in NLP Applications?

**Authors:** [Lili Mou](https://md.research.elamur.ai/free/en/authors/lili-mou/), Zhao Meng, [Rui Yan](https://md.research.elamur.ai/free/en/authors/rui-yan/), [Ge Li](https://md.research.elamur.ai/free/en/authors/ge-li/), [Yan Xu](https://md.research.elamur.ai/free/en/authors/yan-xu/), [Lu Zhang](https://md.research.elamur.ai/free/en/authors/lu-zhang/) and [Zhi Jin](https://md.research.elamur.ai/free/en/authors/zhi-jin/) · **Year:** 2016 · **Venue:** [Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2016](https://md.research.elamur.ai/free/en/venues/emnlp/2016/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D16-1046/) · [PDF](https://aclanthology.org/D16-1046.pdf) · [DOI 10.18653/v1/d16-1046](https://doi.org/10.18653/v1/d16-1046) · [OpenAlex](https://openalex.org/W2964352358)

**Citations:** 259 citations (OpenAlex, 2026-10-14) · [68](https://md.research.elamur.ai/free/en/papers/mou-2016-how-transferable-are-neural-networks/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Neural NLP models overfit easily when the dataset of interest is small, so reusing knowledge from a larger source dataset is attractive. Earlier neural NLP work applied transfer in an ad hoc way and reported contradictory outcomes: tiny gains in one study, large jumps in another. The authors run a controlled study on six public datasets in two groups. The first group is single-sentence classification with an LSTM-RNN, using IMDB as the source and MR and QC as targets. The second is sentence-pair classification with a Siamese CNN, using SNLI as the source and SICK and MSRP as targets. In each group, one target has the same task meaning as the source and the other does not, while the network architecture stays identical. The authors compare three strategies: initializing the target model with source parameters (INIT), training both tasks jointly (MULT), and combining the two. They also examine which layers carry over. Transfer success depends mainly on how semantically close the tasks are, which differs from image processing. This gives practitioners realistic expectations about when neural transfer in NLP pays off.

## Key points
- Dataset sizes (train/val/test):
  - IMDB: 550,000 / 50,000 / 2,000
  - MR: 8,500 / 1,100 / 1,100
  - QC: 4,800 / 600 / 500
  - SNLI: 550,152 / 10,000 / 10,000
  - SICK: 4,439 / 495 / 4,906
  - MSRP: 3,575 / 501 / 1,725
- Non-transfer baselines were averaged over 5 random initializations and are close to comparable published models. Examples include MR at 75.1 ± 0.6, SICK at 70.9 ± 1.3 and MSRP at 69.0 ± 0.5.
- Transfer helped the semantically equivalent pairs (IMDB→MR, SNLI→SICK) but not the semantically different targets (QC, MSRP). This held for both INIT and MULT.
- Layer-level findings:
  - The output layer is largely tied to its dataset and does not transfer.
  - Word embeddings (pretrained with word2vec, 100-dimensional) may still help semantically different tasks.
- MULT and INIT reached similar peak accuracy. MULT was slightly lower than INIT in Experiment I and slightly higher in Experiment II. MULT+INIT was stable across values of λ but did not beat either method alone.
- In SNLI→SICK, parameters taken from epoch 2 of source training gave the best SICK accuracy (78.98%). At that point SNLI accuracy was only 72.65%, versus 76.26% at epoch 23. Transfer accuracy later fell by about 2.7%, while IMDB→MR showed no such mismatch.

## Interesting ideas
- **Paired targets per source.** Each source gets one same-meaning target and one different-meaning target, all under a fixed architecture. This isolates task semantics as the variable and would suit testing transfer between, say, Ukrainian corpora or tasks.
- **Layer-by-layer transfer notation.** Each layer (embedding, hidden, output) is marked as not transferred, transferred and frozen, or transferred and fine-tuned. This systematically maps which components are reusable and is cheap to replicate for any layered model.
- **Transferring from intermediate source epochs.** Copying source parameters at several training checkpoints showed that a well-trained source model is not always the best donor. This could save compute when the source dataset is large.
- **Tuning the mixing weight λ.** In multi-task training, λ balances source and target losses. The authors first searched it coarsely, then finely near the small values that worked well. This is a simple, transferable recipe for weighting auxiliary data.
- **Fair baselines.** Baselines average several random seeds, while the source model is trained only once, since a specific trained model is what gets transferred. This keeps the comparison honest without inflating cost.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) — University, Beijing, China · 100 works · 35,618 citations
- [Institute of Software](https://md.research.elamur.ai/free/en/labs/institute-of-software/) — Research institute, Beijing, China · 10 works · 4,807 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

## Cite

Cite as: Lili Mou, Zhao Meng, Rui Yan, Ge Li, Yan Xu, Lu Zhang and Zhi Jin. 2016. How Transferable are Neural Networks in NLP Applications?. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{mou2016how,
  title = {{How Transferable are Neural Networks in NLP Applications?}},
  author = {Lili Mou, Zhao Meng, Rui Yan, Ge Li, Yan Xu, Lu Zhang and Zhi Jin},
  booktitle = {Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing},
  year = {2016},
  doi = {10.18653/v1/d16-1046},
  url = {https://doi.org/10.18653/v1/d16-1046}
}
```

- [← previous Problems With Evaluation of Word Embeddings Using Word Similarity Tasks](https://md.research.elamur.ai/free/en/papers/faruqui-2016-problems-evaluation-word-embeddings-word/)
- [next → Cultural Shift or Linguistic Drift? Comparing Two Computational Measures of Semantic Change](https://md.research.elamur.ai/free/en/papers/hamilton-2016-cultural-shift-or-linguistic-drift/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/2/)

## References

25 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. 2006. Greedy layer-wise training of deep networks. In Advances in Neural Information Processing Systems, pages 153–160. [find](https://scholar.google.com/scholar?q=Yoshua+Bengio%2C+Pascal+Lamblin%2C+Dan+Popovici%2C+and+Hugo+Larochelle%2E+2006%2E+Greedy+layer%2Dwise+training+of+deep+networks%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+153%E2%80%93160%2E)
2. John Blitzer, Ryan McDonald, and Fernando Pereira. 2006. [Domain adaptation with structural correspondence learning](https://md.research.elamur.ai/free/en/papers/blitzer-2006-domain-adaptation-structural-correspondence-learning/). In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 120–128.
3. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 632–642. [find](https://scholar.google.com/scholar?q=Samuel+R%2E+Bowman%2C+Gabor+Angeli%2C+Christopher+Potts%2C+and+Christopher+D%2E+Manning%2E+2015%2E+A+large+annotated+corpus+for+learning+natural+language+inference%2E+In+Proceedings+of+the+Conference+on+Empirical+Met)
4. Jane Bromley, James W Bentz, Léon Bottou, Isabelle Guyon, Yann LeCun, Cliff Moore, Eduard Säckinger, and Roopak Shah. 1993. Signature verification using a “Siamese” time delay neural network. International Journal of Pattern Recognition and Artificial Intelligence, 7(04):669–688. [find](https://scholar.google.com/scholar?q=Jane+Bromley%2C+James+W+Bentz%2C+Le%CC%81on+Bottou%2C+Isabelle+Guyon%2C+Yann+LeCun%2C+Cliff+Moore%2C+Eduard+Sa%CC%88ckinger%2C+and+Roopak+Shah%2E+1993%2E+Signature+verification+using+a+%E2%80%9CSiamese%E2%80%9D+time+delay+neural+network%2E+Intern)
5. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th International Conference on Machine Learning, pages 160–167. [find](https://scholar.google.com/scholar?q=Ronan+Collobert+and+Jason+Weston%2E+2008%2E+A+unified+architecture+for+natural+language+processing%3A+Deep+neural+networks+with+multitask+learning%2E+In+Proceedings+of+the+25th+International+Conference+on+Mac)
6. Hal Daumé III, Abhishek Kumar, and Avishek Saha. 2010. Frustratingly easy semi-supervised domain adaptation. In Proceedings of the Workshop on Domain Adaptation for Natural Language Processing, pages 53–59. [find](https://scholar.google.com/scholar?q=Hal+Daume%CC%81+III%2C+Abhishek+Kumar%2C+and+Avishek+Saha%2E+2010%2E+Frustratingly+easy+semi%2Dsupervised+domain+adaptation%2E+In+Proceedings+of+the+Workshop+on+Domain+Adaptation+for+Natural+Language+Processing%2C+pages)
7. Hal Daumé III. 2007. [Frustratingly easy domain adaptation](https://md.research.elamur.ai/free/en/papers/iii-2007-frustratingly-easy-domain-adaptation/). In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 256–263.
8. Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. 2014. DeCAF: A deep convolutional activation feature for generic visual recognition. In Proceedings of the 31st International Conference on Machine Learning, pages 647–655. [find](https://scholar.google.com/scholar?q=Jeff+Donahue%2C+Yangqing+Jia%2C+Oriol+Vinyals%2C+Judy+Hoffman%2C+Ning+Zhang%2C+Eric+Tzeng%2C+and+Trevor+Darrell%2E+2014%2E+DeCAF%3A+A+deep+convolutional+activation+feature+for+generic+visual+recognition%2E+In+Proceedings)
9. Li Dong, Furu Wei, Shujie Liu, Ming Zhou, and Ke Xu. 2015. A statistical parsing framework for sentiment classification. Computational Linguistics, 41(2):293–336. [find](https://scholar.google.com/scholar?q=Li+Dong%2C+Furu+Wei%2C+Shujie+Liu%2C+Ming+Zhou%2C+and+Ke+Xu%2E+2015%2E+A+statistical+parsing+framework+for+sentiment+classification%2E+Computational+Linguistics%2C+41%282%29%3A293%E2%80%93336%2E)
10. George Foster, Cyril Goutte, and Roland Kuhn. 2010. [Discriminative instance weighting for domain adaptation in statistical machine translation](https://md.research.elamur.ai/free/en/papers/foster-2010-discriminative-instance-weighting-domain-adaptation/). In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 451–459.
11. Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 2011. Domain adaptation for large-scale sentiment classification: A deep learning approach. In Proceedings of the 28th International Conference on Machine Learning, pages 513–520. [find](https://scholar.google.com/scholar?q=Xavier+Glorot%2C+Antoine+Bordes%2C+and+Yoshua+Bengio%2E+2011%2E+Domain+adaptation+for+large%2Dscale+sentiment+classification%3A+A+deep+learning+approach%2E+In+Proceedings+of+the+28th+International+Conference+on+Mac)
12. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+1997%2E+Long+short%2Dterm+memory%2E+Neural+Computation%2C+9%288%29%3A1735%E2%80%931780%2E)
13. Baotian Hu, Zhengdong Lu, Hang Li, and Qingcai Chen. 2014. Convolutional neural network architectures for matching natural language sentences. In Advances in Neural Information Processing Systems, pages 2042–2050. [find](https://scholar.google.com/scholar?q=Baotian+Hu%2C+Zhengdong+Lu%2C+Hang+Li%2C+and+Qingcai+Chen%2E+2014%2E+Convolutional+neural+network+architectures+for+matching+natural+language+sentences%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pag)
14. Jing Jiang and ChengXiang Zhai. 2007. [Instance weighting for domain adaptation in NLP](https://md.research.elamur.ai/free/en/papers/jiang-2007-instance-weighting-domain-adaptation-nlp/). In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 264– 271.
15. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, pages 1097–1105. [find](https://scholar.google.com/scholar?q=Alex+Krizhevsky%2C+Ilya+Sutskever%2C+and+Geoffrey+E+Hinton%2E+2012%2E+ImageNet+classification+with+deep+convolutional+neural+networks%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+1097%E2%80%931105%2E)
16. Yang Liu, Sujian Li, Xiaodong Zhang, and Zhifang Sui. 2016. [Implicit discourse relation classification via multi-task neural networks](https://md.research.elamur.ai/free/en/papers/liu-2016-implicit-discourse-relation-classification-multi/). In Proceedings of the 30th AAAI Conference on Artificial Intelligence, pages 2750–2756.
17. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. [Distributed representations of words and phrases and their compositionality](https://md.research.elamur.ai/free/en/papers/mikolov-2013-distributed-representations-words-phrases-their/). In Advances in Neural Information Processing Systems, pages 3111–3119.
18. Hao Peng, Lili Mou, Ge Li, Yunchuan Chen, Yangyang Lu, and Zhi Jin. 2015. A comparative study on regularization strategies for embedding-based neural networks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 2106–2111. [find](https://scholar.google.com/scholar?q=Hao+Peng%2C+Lili+Mou%2C+Ge+Li%2C+Yunchuan+Chen%2C+Yangyang+Lu%2C+and+Zhi+Jin%2E+2015%2E+A+comparative+study+on+regularization+strategies+for+embedding%2Dbased+neural+networks%2E+In+Proceedings+of+the+Conference+on+Empi)
19. Barbara Plank and Alessandro Moschitti. 2013. Embedding semantic similarity in tree kernels for domain adaptation of relation extraction. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 1498–1507. [find](https://scholar.google.com/scholar?q=Barbara+Plank+and+Alessandro+Moschitti%2E+2013%2E+Embedding+semantic+similarity+in+tree+kernels+for+domain+adaptation+of+relation+extraction%2E+In+Proceedings+of+the+51st+Annual+Meeting+of+the+Association+f)
20. Peter Prettenhofer and Benno Stein. 2010. [Cross-language text classification using structural correspondence learning](https://md.research.elamur.ai/free/en/papers/prettenhofer-2010-cross-language-text-classification-structural/). In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1118–1127.
21. Richard Socher, Jeffrey Pennington, Eric H Huang, Andrew Y Ng, and Christopher D Manning. 2011. [Semi-supervised recursive autoencoders for predicting sentiment distributions](https://md.research.elamur.ai/free/en/papers/socher-2011-semi-supervised-recursive-autoencoders-predicting/). In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 151–161.
22. Dong Wang and Thomas Fang Zheng. 2015. Transfer learning for speech and language processing. In Proceedings of the Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, pages 1225–1237. [find](https://scholar.google.com/scholar?q=Dong+Wang+and+Thomas+Fang+Zheng%2E+2015%2E+Transfer+learning+for+speech+and+language+processing%2E+In+Proceedings+of+the+Asia%2DPacific+Signal+and+Information+Processing+Association+Annual+Summit+and+Conferen)
23. Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014. How transferable are features in deep neural networks? In Advances in Neural Information Processing Systems, pages 3320–3328. [find](https://scholar.google.com/scholar?q=Jason+Yosinski%2C+Jeff+Clune%2C+Yoshua+Bengio%2C+and+Hod+Lipson%2E+2014%2E+How+transferable+are+features+in+deep+neural+networks%3F+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+3320%E2%80%933328%2E)
24. Matthew D Zeiler and Rob Fergus. 2014. [Visualizing and understanding convolutional networks](https://md.research.elamur.ai/free/en/papers/zeiler-2013-visualizing-understanding-convolutional-networks/). In Proceedings of 13th European Conference on Computer Vision, pages 818–833.
25. Han Zhao, Zhengdong Lu, and Pascal Poupart. 2015. Self-adaptive hierarchical sentence model. In International Joint Conference on Artificial Intelligence, pages 4069–4076. [find](https://scholar.google.com/scholar?q=Han+Zhao%2C+Zhengdong+Lu%2C+and+Pascal+Poupart%2E+2015%2E+Self%2Dadaptive+hierarchical+sentence+model%2E+In+International+Joint+Conference+on+Artificial+Intelligence%2C+pages+4069%E2%80%934076%2E)

*Updated 24 October 2026*