---
title: "Investigating Forgetting in Pre-Trained Representations Through"
description: "They introduce three measures: GD (generality destruction) compares accuracy reached… 2.13 points. 35 references, cited by 9 papers here."
authors: ["Yun Luo", "Zhen Yang", "Xuefeng Bai", "Fandong Meng", "Jie Zhou", "Yue Zhang"]
year: 2023
venue: "arXiv:2305.05968"
source: "https://arxiv.org/abs/2305.05968"
doi: "10.48550/arxiv.2305.05968"
arxiv: "2305.05968"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/luo-2023-investigating-forgetting-pre-trained-representations/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/luo-2023-investigating-forgetting-pre-trained-representations/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/14/) / Investigating Forgetting in Pre-Trained Representations Through Continual Learning

# Investigating Forgetting in Pre-Trained Representations Through Continual Learning

**Authors:** [Yun Luo](https://md.research.elamur.ai/free/en/authors/yun-luo/), [Zhen Yang](https://md.research.elamur.ai/free/en/authors/zhen-yang/), [Xuefeng Bai](https://md.research.elamur.ai/free/en/authors/xuefeng-bai/), [Fandong Meng](https://md.research.elamur.ai/free/en/authors/fandong-meng/), [Jie Zhou](https://md.research.elamur.ai/free/en/authors/jie-zhou/) and [Yue Zhang](https://md.research.elamur.ai/free/en/authors/yue-zhang/) · **Year:** 2023 · **Venue:** arXiv:2305.05968 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2305.05968) · [PDF](https://arxiv.org/pdf/2305.05968) · [DOI 10.48550/arxiv.2305.05968](https://doi.org/10.48550/arxiv.2305.05968) · [arXiv 2305.05968](https://arxiv.org/abs/2305.05968) · [OpenAlex](https://openalex.org/W4376167151)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · 9 in our library, of the 149,439 papers whose bibliographies we read

## Summary
When a pre-trained language model is fine-tuned on a sequence of tasks, its contextualized representations shift. Earlier work measured this "representation forgetting" only on tasks already learned. This paper asks whether the drift also erodes the model's general usefulness for future tasks. The authors run task-incremental continual learning, mainly on BERT, using classification tasks from GLUE and MBPA++ (including MNLI, CoLA, Yelp and Yahoo) in several task orders. They introduce three measures:

- **GD** (generality destruction) compares accuracy reached during continual training with accuracy from fine-tuning the original model on each task alone.
- **SynF** and **SemF** use MLP probes on the last encoder layer to track the loss of syntactic and semantic knowledge relative to the original model.

Across experiments, generality degrades and both kinds of linguistic knowledge erode. Two remedies emerge. Placing general linguistic tasks early in the sequence reduces the damage. Hybrid strategies that combine replay and regularization balance task accuracy and knowledge retention best. These findings inform how to add knowledge to pre-trained LMs incrementally.

## Key points
- Probing uses six SentEval tasks plus MRPC from GLUE, sampling 1,200 examples per class (about 1% of SentEval).
  - Syntactic probes: BShift, TreeDepth, SubjNum, ObjNum.
  - Semantic probes: Tense, MRPC, CoordInv.
- With experience replay, BERT's NLI accuracy fell only from 83.57% to 81.08%, yet the representations visibly drifted in t-SNE plots. Later sentiment accuracy was 2.13 points below fine-tuning the original BERT.
- Final average accuracy over six task orders differed sharply by strategy:
  - plain sequential fine-tuning (BERT-FT): roughly 43–58%;
  - replay (BERT-ER): about 65–69%;
  - DERPP: best accuracy and lowest GD.
- Each strategy had a distinct trade-off:
  - LwF (regularization) kept general knowledge well, with negative SynF/SemF values of -2.24% and -3.84%, but had higher GD than plain fine-tuning.
  - Replay showed higher GD, SynF and SemF than plain fine-tuning.
- GD correlates positively with SynF and SemF, and all three depend strongly on task order. They are lowest when the general linguistic task comes first.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tencent](https://md.research.elamur.ai/free/en/companies/tencent/) — Company, Shenzhen, China · 163 works · 21,032 citations
- [Westlake University](https://md.research.elamur.ai/free/en/universities/westlake-university/) — University, Hangzhou, China · 51 works · 8,572 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

## Cite

Cite as: Yun Luo, Zhen Yang, Xuefeng Bai, Fandong Meng, Jie Zhou and Yue Zhang. 2023. Investigating Forgetting in Pre-Trained Representations Through Continual Learning. arXiv:2305.05968.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{luo2023investigating,
  title = {{Investigating Forgetting in Pre-Trained Representations Through Continual Learning}},
  author = {Yun Luo, Zhen Yang, Xuefeng Bai, Fandong Meng, Jie Zhou and Yue Zhang},
  year = {2023},
  doi = {10.48550/arxiv.2305.05968},
  eprint = {2305.05968},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.05968}
}
```

- [← previous Expression Syntax Information Bottleneck for Math Word Problems](https://md.research.elamur.ai/free/en/papers/xiong-2023-expression-syntax-information-bottleneck-math/)
- [next → KSAA-RD Shared Task: Arabic Reverse Dictionary](https://md.research.elamur.ai/free/en/papers/matham-2023-ksaa-rd-shared-task-arabic/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/14/)

## References

35 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. 2018. [Memory aware synapses: Learning what (not) to forget](https://md.research.elamur.ai/free/en/papers/aljundi-2017-memory-aware-synapses-learning-forget/). In Proceedings of the European Conference on Computer Vision (ECCV), pages 139–154.
2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
3. Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. 2020. Dark experience for general continual learning: a strong, simple baseline. Advances in neural information processing systems, 33:15920–15930. [find](https://scholar.google.com/scholar?q=Pietro+Buzzega%2C+Matteo+Boschini%2C+Angelo+Porrello%2C+Davide+Abati%2C+and+Simone+Calderara%2E+2020%2E+Dark+experience+for+general+continual+learning%3A+a+strong%2C+simple+baseline%2E+Advances+in+neural+information+pr)
4. Alexis Conneau and Douwe Kiela. 2018. [Senteval: An evaluation toolkit for universal sentence representations](https://md.research.elamur.ai/free/en/papers/conneau-2018-senteval-evaluation-toolkit-universal-sentence/). arXiv preprint arXiv:1803.05449.
5. MohammadReza Davari, Nader Asadi, Sudhir Mudur, Rahaf Aljundi, and Eugene Belilovsky. 2022. Probing representation forgetting in supervised and unsupervised continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16712–16721. [find](https://scholar.google.com/scholar?q=MohammadReza+Davari%2C+Nader+Asadi%2C+Sudhir+Mudur%2C+Rahaf+Aljundi%2C+and+Eugene+Belilovsky%2E+2022%2E+Probing+representation+forgetting+in+supervised+and+unsupervised+continual+learning%2E+In+Proceedings+of+the+I)
6. Cyprien de Masson D’Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. 2019. Episodic memory in lifelong language learning. Advances in Neural Information Processing Systems, 32. [find](https://scholar.google.com/scholar?q=Cyprien+de+Masson+D%E2%80%99Autume%2C+Sebastian+Ruder%2C+Lingpeng+Kong%2C+and+Dani+Yogatama%2E+2019%2E+Episodic+memory+in+lifelong+language+learning%2E+Advances+in+Neural+Information+Processing+Systems%2C+32%2E)
7. Yufan Huang, Yanzhe Zhang, Jiaao Chen, Xuezhi Wang, and Diyi Yang. 2021. Continual learning for text classification with information disentanglement based regularization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langu… [find](https://scholar.google.com/scholar?q=Yufan+Huang%2C+Yanzhe+Zhang%2C+Jiaao+Chen%2C+Xuezhi+Wang%2C+and+Diyi+Yang%2E+2021%2E+Continual+learning+for+text+classification+with+information+disentanglement+based+regularization%2E+In+Proceedings+of+the+2021+Co)
8. Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. [What does BERT learn about the structure of language?](https://md.research.elamur.ai/free/en/papers/jawahar-2019-what-does-bert-learn-about/) In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3651–3657, Florence, Italy. Association for Computational Linguistics.
9. Shibo Jie, Zhi-Hong Deng, and Ziheng Li. 2022. Alleviating representational shift for continual fine-tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3810–3819. [find](https://scholar.google.com/scholar?q=Shibo+Jie%2C+Zhi%2DHong+Deng%2C+and+Ziheng+Li%2E+2022%2E+Alleviating+representational+shift+for+continual+fine%2Dtuning%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+pages)
10. Zixuan Ke and Bing Liu. 2022. Continual learning of natural language processing tasks: A survey. arXiv preprint arXiv:2211.12701. [find](https://scholar.google.com/scholar?q=Zixuan+Ke+and+Bing+Liu%2E+2022%2E+Continual+learning+of+natural+language+processing+tasks%3A+A+survey%2E+arXiv+preprint+arXiv%3A2211%2E12701%2E)
11. Diederik P Kingma and Jimmy Ba. 2014. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). arXiv preprint arXiv:1412.6980.
12. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521… [find](https://scholar.google.com/scholar?q=James+Kirkpatrick%2C+Razvan+Pascanu%2C+Neil+Rabinowitz%2C+Joel+Veness%2C+Guillaume+Desjardins%2C+Andrei+A+Rusu%2C+Kieran+Milan%2C+John+Quan%2C+Tiago+Ramalho%2C+Agnieszka+Grabska%2DBarwinska%2C+et+al%2E+2017%2E+Overcoming+catas)
13. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. [Albert: A lite bert for self-supervised learning of language representations](https://md.research.elamur.ai/free/en/papers/lan-2019-albert-lite-bert-self-supervised/). arXiv preprint arXiv:1909.11942.
14. Zhizhong Li and Derek Hoiem. 2017. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935–2947. [find](https://scholar.google.com/scholar?q=Zhizhong+Li+and+Derek+Hoiem%2E+2017%2E+Learning+without+forgetting%2E+IEEE+transactions+on+pattern+analysis+and+machine+intelligence%2C+40%2812%29%3A2935%E2%80%932947%2E)
15. Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. 2022. [Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning](https://md.research.elamur.ai/free/en/papers/liu-2022-few-shot-parameter-efficient-fine/). arXiv preprint arXiv:2205.05638.
16. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). arXiv preprint arXiv:1907.11692.
17. David Lopez-Paz and Marc’Aurelio Ranzato. 2017. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30. [find](https://scholar.google.com/scholar?q=David+Lopez%2DPaz+and+Marc%E2%80%99Aurelio+Ranzato%2E+2017%2E+Gradient+episodic+memory+for+continual+learning%2E+Advances+in+neural+information+processing+systems%2C+30%2E)
18. Amil Merchant, Elahe Rahimtoroghi, Ellie Pavlick, and Ian Tenney. 2020. [What happens to bert embeddings during fine-tuning?](https://md.research.elamur.ai/free/en/papers/merchant-2020-what-happens-bert-embeddings-during/) arXiv preprint arXiv:2004.14448.
19. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. [Rethinking the role of demonstrations: What makes in-context learning work?](https://md.research.elamur.ai/free/en/papers/min-2022-rethinking-role-demonstrations-what-makes/) arXiv preprint arXiv:2202.12837.
20. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E)
21. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. [SQuAD: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
22. Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. 2019. Learning to learn without forgetting by maximizing transfer and minimizing interference. In International Conference on Learning Representations. International Conference on Learning Representati… [find](https://scholar.google.com/scholar?q=Matthew+Riemer%2C+Ignacio+Cases%2C+Robert+Ajemian%2C+Miao+Liu%2C+Irina+Rish%2C+Yuhai+Tu%2C+and+Gerald+Tesauro%2E+2019%2E+Learning+to+learn+without+forgetting+by+maximizing+transfer+and+minimizing+interference%2E+In+Int)
23. Anthony Robins. 1995. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science, 7(2):123–146. [find](https://scholar.google.com/scholar?q=Anthony+Robins%2E+1995%2E+Catastrophic+forgetting%2C+rehearsal+and+pseudorehearsal%2E+Connection+Science%2C+7%282%29%3A123%E2%80%93146%2E)
24. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. [Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter](https://md.research.elamur.ai/free/en/papers/sanh-2019-distilbert-distilled-version-bert-smaller/). arXiv preprint arXiv:1910.01108.
25. Timo Schick and Hinrich Schütze. 2021. [It’s not just size that matters: Small language models are also few-shot learners](https://md.research.elamur.ai/free/en/papers/schick-2021-it-s-not-just-size/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2339–2352, Online. Associ…
26. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019a. [BERT rediscovers the classical NLP pipeline](https://md.research.elamur.ai/free/en/papers/tenney-2019-bert-rediscovers-classical-nlp-pipeline/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593– 4601, Florence, Italy. Association for Computational Linguistics.
27. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al. 2019b. [What do you learn from context? probing for sentence structure in contextualized word representations](https://md.research.elamur.ai/free/en/papers/tenney-2019-learn-context-probing-sentence-structure/). arXiv preprint arXiv:1905.06316.
28. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. [Glue: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355.
29. Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625–641. [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Amanpreet+Singh%2C+and+Samuel+R+Bowman%2E+2019%2E+Neural+network+acceptability+judgments%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+7%3A625%E2%80%93641%2E)
30. Adina Williams, Nikita Nangia, and Samuel R Bowman. 2017. [A broad-coverage challenge corpus for sentence understanding through inference](https://md.research.elamur.ai/free/en/papers/williams-2018-broad-coverage-challenge-corpus-sentence/). arXiv preprint arXiv:1704.05426.
31. Tongtong Wu, Massimo Caccia, Zhuang Li, Yuan-Fang Li, Guilin Qi, and Gholamreza Haffari. 2022. Pre-trained language model in continual learning: A comparative study. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net. [find](https://scholar.google.com/scholar?q=Tongtong+Wu%2C+Massimo+Caccia%2C+Zhuang+Li%2C+Yuan%2DFang+Li%2C+Guilin+Qi%2C+and+Gholamreza+Haffari%2E+2022%2E+Pre%2Dtrained+language+model+in+continual+learning%3A+A+comparative+study%2E+In+The+Tenth+International+Confere)
32. Lu Yu, Bartlomiej Twardowski, Xialei Liu, Luis Herranz, Kai Wang, Yongmei Cheng, Shangling Jui, and Joost van de Weijer. 2020. Semantic drift compensation for class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6982–6991. [find](https://scholar.google.com/scholar?q=Lu+Yu%2C+Bartlomiej+Twardowski%2C+Xialei+Liu%2C+Luis+Herranz%2C+Kai+Wang%2C+Yongmei+Cheng%2C+Shangling+Jui%2C+and+Joost+van+de+Weijer%2E+2020%2E+Semantic+drift+compensation+for+class%2Dincremental+learning%2E+In+Proceeding)
33. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. [Character-level convolutional networks for text classification](https://md.research.elamur.ai/free/en/papers/zhang-2015-character-level-convolutional-networks-text/). Advances in neural information processing systems, 28.
34. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. [Calibrate before use: Improving few-shot performance of language models](https://md.research.elamur.ai/free/en/papers/zhao-2021-calibrate-before-use-improving-few/). In International Conference on Machine Learning, pages 12697–12706. PMLR.
35. Zining Zhu, Soroosh Shahtalebi, and Frank Rudzicz. 2022. Predicting fine-tuning performance with probing. arXiv preprint arXiv:2210.07352. [find](https://scholar.google.com/scholar?q=Zining+Zhu%2C+Soroosh+Shahtalebi%2C+and+Frank+Rudzicz%2E+2022%2E+Predicting+fine%2Dtuning+performance+with+probing%2E+arXiv+preprint+arXiv%3A2210%2E07352%2E)

*Updated 2 November 2026*