---
title: "TARE — paper by Wang and Zhao (2026) — summary and ideas"
description: "TARE builds on this observation. 41 references. Parameter-efficient fine-tuning (PEFT) involves a trade-off."
authors: ["Yulong Wang", "Siyu Zhao"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.841/"
doi: "10.18653/v1/2026.acl-long.841"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2026-tare-lightweight-token-aware-representation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2026-tare-lightweight-token-aware-representation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/51/) / TARE: Lightweight Token-Aware Representation Editing for Fine-tuning Transformer-like Models

# TARE: Lightweight Token-Aware Representation Editing for Fine-tuning Transformer-like Models

**Authors:** [Yulong Wang](https://md.research.elamur.ai/free/en/authors/yulong-wang/) and Siyu Zhao · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.841/) · [PDF](https://aclanthology.org/2026.acl-long.841.pdf) · [DOI 10.18653/v1/2026.acl-long.841](https://doi.org/10.18653/v1/2026.acl-long.841) · [OpenAlex](https://openalex.org/W7166838905)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Parameter-efficient fine-tuning (PEFT) involves a trade-off. Low-rank adapters such as LoRA add many trainable weights and need rank and scale settings. Cheap representation editors such as RED apply one transformation to every token and miss token-level context. The authors observe that the per-token optimal scaling and bias vectors in a layer group into a few recurring patterns. A small set of editors could therefore cover them.

TARE builds on this observation. After each FFN block of a frozen transformer, a small selector scores a pool of diagonal editors, each a per-dimension scale and bias. For every token, it keeps the top-k editors and blends their outputs with softmax weights. The authors stress that this is not a mixture-of-experts adapter, and the only settings are n and k.

TARE was tested on LLaMA-3-8B for knowledge and math reasoning and on RoBERTa-base/large for GLUE. It matches or beats LoRA, DoRA, MiLoRA, LoReFT and RED while training only 0.0392% of parameters with about 20 GiB of peak memory. This makes it attractive for low-budget adaptation.

## Key points
- **Averages:** 86.7% on eight knowledge tasks (BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC-e, ARC-c, OBQA), against LoReFT 86.6, LoRA 80.8 and RED 80.2. It reaches 76.7% on seven math tasks (incl. GSM8K, SVAMP, AQuA) and 88.3% on GLUE. Qwen-2.5-7B-Instruct was also evaluated on the math tasks.
- **Default configuration:** n=8 editors, Top-k=3, with an optional load-balancing loss. Raising k from 1 to 3 lifts the math average from 71.2 to 76.7, and larger k gives no further gain.
- **Placement ablation:** inserting at the FFN down projection is the best compromise. Placing editors at the up projection costs 0.1369% of parameters and about 29.6 GiB of memory.
- **Data efficiency:** with Math-10K, accuracy grows smoothly from 69.8 with 500 examples to 76.7 with the full 9,919 examples.
- **Limitations and code:** routing may collapse or use editors unevenly, and diagonal edits limit expressivity. Code is at github.com/PatriciaPulec/tare.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Beijing University of Posts and Telecommunications](https://md.research.elamur.ai/free/en/universities/beijing-university-of-posts-and-telecommunications/) — University, Beijing, China · 78 works · 33,365 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

## Cite

Cite as: Yulong Wang and Siyu Zhao. 2026. TARE: Lightweight Token-Aware Representation Editing for Fine-tuning Transformer-like Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wang2026tare,
  title = {{TARE: Lightweight Token-Aware Representation Editing for Fine-tuning Transformer-like Models}},
  author = {Yulong Wang and Siyu Zhao},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.841},
  url = {https://doi.org/10.18653/v1/2026.acl-long.841}
}
```

- [← previous TantaArabNLP at KSAA-2026 Task 2: Adapting CATT-Whisper for Arabic Speech Dictation with Automatic Diacritization](https://md.research.elamur.ai/free/en/papers/esmaeil-2026-tantaarabnlp-ksaa-2026-task-2/)
- [next → Targeted Exploration via Unified Entropy Control for Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/wang-2026-targeted-exploration-unified-entropy-control/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/51/)

## References

41 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jacob Austin, Augustus Odena, and Maxwell Nye. 2021. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). arXiv preprint arXiv:2108.07732.
2. Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. 2021. [Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models](https://md.research.elamur.ai/free/en/papers/ben-zaken-2021-bitfit-simple-parameter-efficient-fine/). arXiv preprint arXiv:2106.10199.
3. Yonatan Bisk, Rowan Zellers, and Ronan Le Bras. 2019. [Piqa: Reasoning about physical commonsense in natural language](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/). In AAAI Conference on Artificial Intelligence.
4. Mark Chen, Jerry Tworek, and Heewoo Jun. 2021. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). ArXiv, abs/2107.03374.
5. Christopher Clark, Kenton Lee, and Ming-Wei Chang. 2019. [Boolq: Exploring the surprising difficulty of natural yes/no questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/). ArXiv, abs/1905.10044.
6. Peter Clark, Isaac Cowhey, and Oren Etzioni. 2018. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). ArXiv, abs/1803.05457.
7. Karl Cobbe, Vineet Kosaraju, and Mohammad Bavarian. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168. Introduces the GSM8K dataset.
8. Abhimanyu Dubey, Abhinav Jauhri, and Abhinav Pandey. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). ArXiv, abs/2407.21783.
9. William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39. [find](https://scholar.google.com/scholar?q=William+Fedus%2C+Barret+Zoph%2C+and+Noam+Shazeer%2E+2022%2E+Switch+transformers%3A+Scaling+to+trillion+parameter+models+with+simple+and+efficient+sparsity%2E+Journal+of+Machine+Learning+Research%2C+23%28120%29%3A1%E2%80%9339%2E)
10. Ben Goodrich, Vinay Rao, and Peter J Liu. 2019. Assessing the factual accuracy of generated text. In proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 166–175. [find](https://scholar.google.com/scholar?q=Ben+Goodrich%2C+Vinay+Rao%2C+and+Peter+J+Liu%2E+2019%2E+Assessing+the+factual+accuracy+of+generated+text%2E+In+proceedings+of+the+25th+ACM+SIGKDD+international+conference+on+knowledge+discovery+%26+data+mining%2C+p)
11. Mohammad Javad Hosseini, Hannaneh Hajishirzi, and Oren Etzioni. 2014. [Learning to solve arithmetic word problems with verb categorization](https://md.research.elamur.ai/free/en/papers/hosseini-2014-learning-solve-arithmetic-word-problems/). In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 523– 533, Doha, Qatar. Introduces the AddSub dataset.
12. J. Edward Hu, Yelong Shen, and Phillip Wallis. 2021. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). ArXiv, abs/2106.09685.
13. Zhiqiang Hu, Yihuai Lan, and Lei Wang. 2023a. [Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models](https://md.research.elamur.ai/free/en/papers/hu-2023-llm-adapters-adapter-family-parameter/). ArXiv, abs/2304.01933.
14. Zhiqiang Hu, Lei Wang, and Yihuai Lan. 2023b. [Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models](https://md.research.elamur.ai/free/en/papers/hu-2023-llm-adapters-adapter-family-parameter/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5254–5276, Singapore. Introduces the Math10K dataset.
15. Rik Koncel-Kedziorski, Hannaneh Hajishirzi, and Ashish Sabharwal. 2015. Parsing algebraic word problems into equations. Transactions of the Association for Computational Linguistics, 3:585–597. Introduces the SingleEq dataset. [find](https://scholar.google.com/scholar?q=Rik+Koncel%2DKedziorski%2C+Hannaneh+Hajishirzi%2C+and+Ashish+Sabharwal%2E+2015%2E+Parsing+algebraic+word+problems+into+equations%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+3%3A585%E2%80%93597%2E+Introd)
16. Rik Koncel-Kedziorski, Subhro Roy, and Aida Amini. 2016. Mawps: A math word problem repository. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1152–1157, San Diego, California. [find](https://scholar.google.com/scholar?q=Rik+Koncel%2DKedziorski%2C+Subhro+Roy%2C+and+Aida+Amini%2E+2016%2E+Mawps%3A+A+math+word+problem+repository%2E+In+Proceedings+of+the+2016+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational)
17. Wang Ling, Dani Yogatama, and Chris Dyer. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 158–167, Vancouver, Canada. Intr… [find](https://scholar.google.com/scholar?q=Wang+Ling%2C+Dani+Yogatama%2C+and+Chris+Dyer%2E+2017%2E+Program+induction+by+rationale+generation%3A+Learning+to+solve+and+explain+algebraic+word+problems%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Assoc)
18. Haokun Liu, Derek Tam, and Mohammed Muqeeth. 2022. [Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning](https://md.research.elamur.ai/free/en/papers/liu-2022-few-shot-parameter-efficient-fine/). Advances in Neural Information Processing Systems, 35:1950– 1965.
19. Shih-Yang Liu, Chien-Yi Wang, and Hongxu Yin. 2024. Dora: Weight-decomposed low-rank adaptation. ArXiv, abs/2402.09353. [find](https://scholar.google.com/scholar?q=Shih%2DYang+Liu%2C+Chien%2DYi+Wang%2C+and+Hongxu+Yin%2E+2024%2E+Dora%3A+Weight%2Ddecomposed+low%2Drank+adaptation%2E+ArXiv%2C+abs%2F2402%2E09353%2E)
20. Yinhan Liu, Myle Ott, and Naman Goyal. 2019. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). arXiv preprint arXiv:1907.11692.
21. Fanxu Meng, Zhaohui Wang, and Muhan Zhang. 2024. [Pissa: Principal singular values and singular vectors adaptation of large language models](https://md.research.elamur.ai/free/en/papers/meng-2024-pissa-principal-singular-values-singular/). In The Thirty-eighth Annual Conference on Neural Information Processing Systems (NeurIPS).
22. Todor Mihaylov, Peter Clark, and Tushar Khot. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Todor+Mihaylov%2C+Peter+Clark%2C+and+Tushar+Khot%2E+2018%2E+Can+a+suit+of+armor+conduct+electricity%3F+a+new+dataset+for+open+book+question+answering%2E+In+Conference+on+Empirical+Methods+in+Natural+Language+Proc)
23. Jekaterina Novikova, Ondrej Dusek, and Verena Rieser. 2017. [The e2e dataset: New challenges for end-to-end generation](https://md.research.elamur.ai/free/en/papers/novikova-2017-e2e-dataset-new-challenges-end/). ArXiv, abs/1706.09254.
24. OpenAI, Josh Achiam, and Steven Adler. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). ArXiv, abs/2303.08774.
25. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. [Are nlp models really able to solve simple math word problems?](https://md.research.elamur.ai/free/en/papers/patel-2021-are-nlp-models-really-able/) In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080–2094. Introduces the S…
26. Subhro Roy and Dan Roth. 2015. [Solving general arithmetic word problems](https://md.research.elamur.ai/free/en/papers/roy-2015-solving-general-arithmetic-word-problems/). In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1743–1752, Lisbon, Portugal.
27. Tanik Saikh, Tirthankar Ghosal, and Amish Mittal. 2022. Scienceqa: a novel resource for question answering on scholarly articles. International Journal on Digital Libraries, 23:289 – 301. [find](https://scholar.google.com/scholar?q=Tanik+Saikh%2C+Tirthankar+Ghosal%2C+and+Amish+Mittal%2E+2022%2E+Scienceqa%3A+a+novel+resource+for+question+answering+on+scholarly+articles%2E+International+Journal+on+Digital+Libraries%2C+23%3A289+%E2%80%93+301%2E)
28. Keisuke Sakaguchi, Ronan Le Bras, and Chandra Bhagavatula. 2019. An adversarial winograd schema challenge at scale. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+and+Chandra+Bhagavatula%2E+2019%2E+An+adversarial+winograd+schema+challenge+at+scale%2E)
29. Maarten Sap, Hannah Rashkin, and Derek Chen. 2019. [Socialiqa: Commonsense reasoning about social interactions](https://md.research.elamur.ai/free/en/papers/sap-2019-socialiqa-commonsense-reasoning-about-social/). arXiv preprint arXiv:1904.09728.
30. Alex Wang, Amanpreet Singh, and Julian Michael. 2019. [Glue: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In Proceedings of the International Conference on Learning Representations (ICLR).
31. Hanqing Wang, Zeguan Xiao, and Yixia Li. 2024a. Milora: Harnessing minor singular components for parameter-efficient llm finetuning. In North American Chapter of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Hanqing+Wang%2C+Zeguan+Xiao%2C+and+Yixia+Li%2E+2024a%2E+Milora%3A+Harnessing+minor+singular+components+for+parameter%2Defficient+llm+finetuning%2E+In+North+American+Chapter+of+the+Association+for+Computational+Ling)
32. Shaowen Wang, Linxi Yu, and Jian Li. 2024b. Lora-ga: Low-rank adaptation with gradient approximation. ArXiv, abs/2407.05000. — [arXiv:2407.05000](https://arxiv.org/abs/2407.05000) [find](https://scholar.google.com/scholar?q=Shaowen+Wang%2C+Linxi+Yu%2C+and+Jian+Li%2E+2024b%2E+Lora%2Dga%3A+Low%2Drank+adaptation+with+gradient+approximation%2E+ArXiv%2C+abs%2F2407%2E05000%2E+%E2%80%94+%5BarXiv%3A2407%2E05000%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2407%2E05000%29)
33. Shaowen Wang, Linxi Yu, and Jian Li. 2024c. Lora-ga: Low-rank adaptation with gradient approximation. In NeurIPS. [find](https://scholar.google.com/scholar?q=Shaowen+Wang%2C+Linxi+Yu%2C+and+Jian+Li%2E+2024c%2E+Lora%2Dga%3A+Low%2Drank+adaptation+with+gradient+approximation%2E+In+NeurIPS%2E)
34. Jason Wei, Xuezhi Wang, and Dale Schuurmans. 2022. [Chain of thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). ArXiv, abs/2201.11903.
35. Muling Wu, Wenhao Liu, and Xiaohua Wang. 2024a. Advancing parameter efficiency in fine-tuning via representation editing. ArXiv, abs/2402.15179. [find](https://scholar.google.com/scholar?q=Muling+Wu%2C+Wenhao+Liu%2C+and+Xiaohua+Wang%2E+2024a%2E+Advancing+parameter+efficiency+in+fine%2Dtuning+via+representation+editing%2E+ArXiv%2C+abs%2F2402%2E15179%2E)
36. Zhengxuan Wu, Aryaman Arora, and Zheng Wang. 2024b. [Reft: Representation finetuning for language models](https://md.research.elamur.ai/free/en/papers/wu-2024-reft-representation-finetuning-language-models/). ArXiv, abs/2404.03592.
37. Can Xu, Qingfeng Sun, and Kai Zheng. 2024. [Wizardlm: Empowering large pre-trained language models to follow complex instructions](https://md.research.elamur.ai/free/en/papers/xu-2023-wizardlm-empowering-large-pre-trained/). In ICLR.
38. Rowan Zellers, Ari Holtzman, and Yonatan Bisk. 2019. Hellaswag: Can a machine really finish your sentence? In Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+and+Yonatan+Bisk%2E+2019%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F+In+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2E)
39. Fangzhao Zhang and Mert Pilanci. 2024. Spectral adapter: Fine-tuning in spectral space. ArXiv, abs/2405.13952. — [arXiv:2405.13952](https://arxiv.org/abs/2405.13952) [find](https://scholar.google.com/scholar?q=Fangzhao+Zhang+and+Mert+Pilanci%2E+2024%2E+Spectral+adapter%3A+Fine%2Dtuning+in+spectral+space%2E+ArXiv%2C+abs%2F2405%2E13952%2E+%E2%80%94+%5BarXiv%3A2405%2E13952%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2405%2E13952%29)
40. Yuanhe Zhang, Fanghui Liu, and Yudong Chen. 2025. Lora-one: One-step full gradient could suffice for fine-tuning large language models, provably and efficiently. arXiv, abs/2502.01235. — [arXiv:2502.01235](https://arxiv.org/abs/2502.01235) [find](https://scholar.google.com/scholar?q=Yuanhe+Zhang%2C+Fanghui+Liu%2C+and+Yudong+Chen%2E+2025%2E+Lora%2Done%3A+One%2Dstep+full+gradient+could+suffice+for+fine%2Dtuning+large+language+models%2C+provably+and+efficiently%2E+arXiv%2C+abs%2F2502%2E01235%2E+%E2%80%94+%5BarXiv%3A2502%2E0)
41. Lianmin Zheng, Wei-Lin Chiang, and Ying Sheng. 2023. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/).

*Updated 2 November 2026*