--- title: "Temporal Effects on Pre-trained Models for Language Processing" description: "The paper offers practical guidance on when to retrain and how to design temporal evaluations. 1 reference, cited by 44 papers here." authors: ["Oshin Agarwal", "Ani Nenkova"] year: 2022 venue: "Transactions of the Association for Computational Linguistics" source: "https://aclanthology.org/2022.tacl-1.53/" doi: "10.1162/tacl_a_00497" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/agarwal-2022-temporal-effects-pre-trained-models/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/agarwal-2022-temporal-effects-pre-trained-models/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/) / Temporal Effects on Pre-trained Models for Language Processing Tasks # Temporal Effects on Pre-trained Models for Language Processing Tasks **Authors:** [Oshin Agarwal](https://md.research.elamur.ai/free/en/authors/oshin-agarwal/) and [Ani Nenkova](https://md.research.elamur.ai/free/en/authors/ani-nenkova/) · **Year:** 2022 · **Venue:** [Transactions of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/tacl/) ([Transactions of the Association for Computational Linguistics (TACL) 2022](https://md.research.elamur.ai/free/en/venues/tacl/2022/)) · **Publisher:** [The MIT Press](https://md.research.elamur.ai/free/en/publishers/the-mit-press/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2022.tacl-1.53/) · [PDF](https://aclanthology.org/2022.tacl-1.53.pdf) · [DOI 10.1162/tacl\_a\_00497](https://doi.org/10.1162/tacl_a_00497) · [OpenAlex](https://openalex.org/W3217756080) **Citations:** 40 citations (OpenAlex, 2026-10-14) · [44](https://md.research.elamur.ai/free/en/papers/agarwal-2022-temporal-effects-pre-trained-models/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Language and facts about the world shift over time, so a deployed NLP system may lose accuracy. Earlier studies usually tested on one final time slice, which cannot show whether performance declines. Agarwal and Nenkova separate two questions. Does a model trained once get worse on later data (temporal model deterioration)? Does retraining on fresher labeled data help (temporal domain adaptation)? They study four English tasks whose correct labels do not depend on time: named entity recognition, truecasing, sentiment classification and domain classification. Models are trained on each time split and tested on every later one, using summary scores and significance testing. Systems built on GloVe, GloVe+ELMo, BERT and RoBERTa, including several model sizes, often show no statistically significant decline. The same networks with randomly initialized word vectors degrade sharply. Retraining on recent data still helps, so the authors test two adaptation methods that need no new labels. An extra intermediate pre-training step proves unsuitable, while self-labeling works consistently. The paper offers practical guidance on when to retrain and how to design temporal evaluations. ## Key points - The evaluation trains on every temporal split and tests on all later ones. This avoids relying on a single test year, which can mislead: Twitter's late-2017 character-limit increase made 2018 tweets longer and richer in entities. - With pre-trained representations, deterioration depends on the task and is often not significant. Adaptation gains from newer gold data are typically significant. This holds whether or not the pre-training data overlaps in time with the task data. - Self-labeling improves adaptation in all settings. On NER-TTC it beats fine-tuning on new human annotations. For BERT, adding only 25% new self-labeled data to the old gold data already surpasses gold-standard fine-tuning. - Sentiment results report F1 for the negative class, because about 80% of the examples are positive. Macro-F1 is given in an appendix. - Hyperparameters are tuned on the oldest split and then reused for all later splits. Code and design details are published at github.com/oagarwal/temporal-effects. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [University of Pennsylvania](https://md.research.elamur.ai/free/en/universities/university-of-pennsylvania/) — University, Philadelphia, United States · 221 works · 63,234 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works ## Cite Cite as: Oshin Agarwal and Ani Nenkova. 2022. Temporal Effects on Pre-trained Models for Language Processing Tasks. Transactions of the Association for Computational Linguistics. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @article{agarwal2022temporal, title = {{Temporal Effects on Pre-trained Models for Language Processing Tasks}}, author = {Oshin Agarwal and Ani Nenkova}, journal = {Transactions of the Association for Computational Linguistics}, year = {2022}, doi = {10.1162/tacl\_a\_00497}, url = {https://doi.org/10.1162/tacl_a_00497} } ``` - [← previous Neuron-level Interpretation of Deep NLP Models: A Survey](https://md.research.elamur.ai/free/en/papers/sajjad-2022-neuron-level-interpretation-deep-nlp/) - [next → The Geometry of Multilingual Language Model Representations](https://md.research.elamur.ai/free/en/papers/chang-2022-geometry-multilingual-language-model-representations/) - [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/) ## References 1 references, 0 of them with a card of ours. ↗ leads to the work. 1. Oshin Agarwal, Yinfei Yang, Byron C. Wallace, and Ani Nenkova. 2021. Interpretability analysis for named entity recognition to understand system predictions and how they can improve. Computational Linguistics, 47(1): 117–140. https://doi.org/10.1162/coli a 00397 [↗](https://doi.org/10.1162/coli) *Updated 29 October 2026*