---
title: "Orthogonal Representation Editing — paper by Yu et al. (2026)"
description: "Editing many facts in an LLM at once tends to work less well as the batch grows. 24 references."
authors: ["Wenhao Yu", "Zhicong Lu", "Bo Lv", "Fangyin Ma", "Kaiwen Wei", "Shihao Yang", "Nayu Liu"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.1892/"
doi: "10.18653/v1/2026.findings-acl.1892"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yu-2026-orthogonal-representation-editing-decoupling-semantic/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yu-2026-orthogonal-representation-editing-decoupling-semantic/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/44/) / Orthogonal Representation Editing: Decoupling Semantic Entanglement in Batch Knowledge Editing of LLMs

# Orthogonal Representation Editing: Decoupling Semantic Entanglement in Batch Knowledge Editing of LLMs

**Authors:** [Wenhao Yu](https://md.research.elamur.ai/free/en/authors/wenhao-yu/), Zhicong Lu, [Bo Lv](https://md.research.elamur.ai/free/en/authors/bo-lv/), Fangyin Ma, [Kaiwen Wei](https://md.research.elamur.ai/free/en/authors/kaiwen-wei/), Shihao Yang and [Nayu Liu](https://md.research.elamur.ai/free/en/authors/nayu-liu/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.1892/) · [PDF](https://aclanthology.org/2026.findings-acl.1892.pdf) · [DOI 10.18653/v1/2026.findings-acl.1892](https://doi.org/10.18653/v1/2026.findings-acl.1892) · [OpenAlex](https://openalex.org/W7166840545)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Editing many facts in an LLM at once tends to work less well as the batch grows. The authors trace this to semantic representation entanglement: edits that share concepts or sentence templates, such as many "capital of X" facts, land in overlapping regions of hidden space and interfere with each other. The effect is stronger across languages. A controlled pilot study on LLaMA-3-8B shows MEMIT and AlphaEdit losing accuracy on such entangled batches.

Their method, Orthogonal Representation Editing (ORE), keeps the backbone frozen and builds on ReFT. It works in three steps:
- It collects hidden states from unrelated but structurally similar prompts drawn from ZsRE and CounterFact.
- It runs PCA on these states to estimate a general semantic subspace.
- It removes each edit vector's projection onto that subspace, so updates carry only fact-specific directions.

A gated non-linear representation head then learns where to intervene and how much to inject. The authors report that ORE beats existing editors in batch and cross-lingual settings. This matters for multilingual models, where shared semantics otherwise spread edits into non-target languages.

## Key points
- Pilot data: Gemini 3 generated 200 "capital" samples (100 French, 100 Chinese) with identical syntax. A control set of 100 random ZsRE items was translated into French and Chinese.
- MEMIT editing success on LLaMA-3-8B fell from 86% on random samples to 71% (French), 64% (Chinese) and 56% (mixed Fr–Zh). AlphaEdit showed a similar drop.
- The edit vector is the difference between hidden states for the target and the predicted declarative statement. It is then orthogonalized against the top-k PCA basis U_gs.
- The gated head combines an MLP bottleneck (DOWN_PROJ/UP_PROJ with SiLU), RMSNorm and a dynamic position gate, replacing ReFT's linear, manually positioned interventions.
- Limitations: experiments cover only 7B–8B models, standard editing settings and QA-style accuracy. Code is available at github.com/YVVH/ORE.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tencent](https://md.research.elamur.ai/free/en/companies/tencent/) — Company, Shenzhen, China · 163 works · 21,032 citations
- [University of Chinese Academy of Sciences](https://md.research.elamur.ai/free/en/universities/university-of-chinese-academy-of-sciences/) — University, Beijing, China · 188 works · 8,743 citations
- [Tianjin University](https://md.research.elamur.ai/free/en/universities/tianjin-university/) — University, Tianjin, China · 52 works · 1,128 citations
- [Chongqing University](https://md.research.elamur.ai/free/en/universities/chongqing-university/) — University, Chongqing, China · 14 works · 318 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Wenhao Yu, Zhicong Lu, Bo Lv, Fangyin Ma, Kaiwen Wei, Shihao Yang and Nayu Liu. 2026. Orthogonal Representation Editing: Decoupling Semantic Entanglement in Batch Knowledge Editing of LLMs. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yu2026orthogonal,
  title = {{Orthogonal Representation Editing: Decoupling Semantic Entanglement in Batch Knowledge Editing of LLMs}},
  author = {Wenhao Yu, Zhicong Lu, Bo Lv, Fangyin Ma, Kaiwen Wei, Shihao Yang and Nayu Liu},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1892},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1892}
}
```

- [← previous OrthoEdit: Principled and Stable Knowledge Editing via Orthogonal Subspace Projection](https://md.research.elamur.ai/free/en/papers/qiao-2026-orthoedit-principled-stable-knowledge-editing/)
- [next → Orthographic Constraint Satisfaction and Human Difficulty Alignment in Large Language Models](https://md.research.elamur.ai/free/en/papers/tuck-2026-orthographic-constraint-satisfaction-human-difficulty/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/44/)

## References

24 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong, and Zhang Xiong. 2023. [Transformer-patcher: One mistake worth one neuron](https://md.research.elamur.ai/free/en/papers/huang-2023-transformer-patcher-one-mistake-worth/). arXiv preprint arXiv:2301.09785.
2. Hervé Abdi and Lynne J Williams. 2010. Principal component analysis. Wiley interdisciplinary reviews: computational statistics, 2(4):433–459. [find](https://scholar.google.com/scholar?q=Herv%C3%A9+Abdi+and+Lynne+J+Williams%2E+2010%2E+Principal+component+analysis%2E+Wiley+interdisciplinary+reviews%3A+computational+statistics%2C+2%284%29%3A433%E2%80%93459%2E)
3. Himanshu Beniwal, D Kowsik, and Mayank Singh. 2024. Cross-lingual editing in multilingual language models. In Findings of the Association for Computational Linguistics: EACL 2024, pages 2078–2128. [find](https://scholar.google.com/scholar?q=Himanshu+Beniwal%2C+D+Kowsik%2C+and+Mayank+Singh%2E+2024%2E+Cross%2Dlingual+editing+in+multilingual+language+models%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EACL+2024%2C+pages+2078%E2%80%932128%2E)
4. Houcheng Jiang, Junfeng Fang, Ningyu Zhang, Guojun Ma, Mingyang Wan, Xiang Wang, Xiangnan He, and Tat-seng Chua. 2025a. Anyedit: Edit any knowledge encoded in language models. arXiv preprint arXiv:2502.05628. — [arXiv:2502.05628](https://arxiv.org/abs/2502.05628) [find](https://scholar.google.com/scholar?q=Houcheng+Jiang%2C+Junfeng+Fang%2C+Ningyu+Zhang%2C+Guojun+Ma%2C+Mingyang+Wan%2C+Xiang+Wang%2C+Xiangnan+He%2C+and+Tat%2Dseng+Chua%2E+2025a%2E+Anyedit%3A+Edit+any+knowledge+encoded+in+language+models%2E+arXiv+preprint+arXiv%3A250)
5. Baolong Bi, Shenghua Liu, Lingrui Mei, Yiwei Wang, Junfeng Fang, Pengliang Ji, and Xueqi Cheng. 2025. Decoding by contrasting knowledge: Enhancing large language model confidence on edited facts. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lo… [find](https://scholar.google.com/scholar?q=Baolong+Bi%2C+Shenghua+Liu%2C+Lingrui+Mei%2C+Yiwei+Wang%2C+Junfeng+Fang%2C+Pengliang+Ji%2C+and+Xueqi+Cheng%2E+2025%2E+Decoding+by+contrasting+knowledge%3A+Enhancing+large+language+model+confidence+on+edited+facts%2E+In+P)
6. Houcheng Jiang, Junfeng Fang, Tianyu Zhang, Baolong Bi, An Zhang, Ruipeng Wang, Tao Liang, and Xiang Wang. 2025b. [Neuron-level sequential editing for large language models](https://md.research.elamur.ai/free/en/papers/jiang-2025-neuron-level-sequential-editing-large/). In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16678…
7. Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. [Zero-shot relation extraction via reading comprehension](https://md.research.elamur.ai/free/en/papers/levy-2017-zero-shot-relation-extraction-reading/). arXiv preprint arXiv:1706.04115. null-space constraints. In Findings of the Association for Computational Linguistics: ACL 2025, pages 8796–8810, Vienna, Austria. Association for…
8. Qi Li and Xiaowen Chu. 2025. AdaEdit: Advancing continuous knowledge editing for large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4127–4149, Vienna, Austria. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Qi+Li+and+Xiaowen+Chu%2E+2025%2E+AdaEdit%3A+Advancing+continuous+knowledge+editing+for+large+language+models%2E+In+Proceedings+of+the+63rd+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volu)
9. Changyue Wang, Weihang Su, Qingyao Ai, Yujia Zhou, and Yiqun Liu. 2025. Decoupling reasoning and knowledge injection for in-context knowledge editing. arXiv preprint arXiv:2506.00536. [find](https://scholar.google.com/scholar?q=Changyue+Wang%2C+Weihang+Su%2C+Qingyao+Ai%2C+Yujia+Zhou%2C+and+Yiqun+Liu%2E+2025%2E+Decoupling+reasoning+and+knowledge+injection+for+in%2Dcontext+knowledge+editing%2E+arXiv+preprint+arXiv%3A2506%2E00536%2E)
10. Tianci Liu, Ruirui Li, Yunzhe Qi, Hui Liu, Xianfeng Tang, Tianqi Zheng, Qingyu Yin, Monica Xiao Cheng, Jun Huan, Haoyu Wang, and Jing Gao. 2025. Unlocking efficient, scalable, and continual knowledge editing with basis-level representation fine-tuning. In The Thirteenth International Conference on L… [find](https://scholar.google.com/scholar?q=Tianci+Liu%2C+Ruirui+Li%2C+Yunzhe+Qi%2C+Hui+Liu%2C+Xianfeng+Tang%2C+Tianqi+Zheng%2C+Qingyu+Yin%2C+Monica+Xiao+Cheng%2C+Jun+Huan%2C+Haoyu+Wang%2C+and+Jing+Gao%2E+2025%2E+Unlocking+efficient%2C+scalable%2C+and+continual+knowledge+)
11. Jun-Yu Ma, Hong Wang, Hao-Xiang Xu, Zhen-Hua Ling, and Jia-Chen Gu. 2025. Perturbation-restrained sequential model editing. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Jun%2DYu+Ma%2C+Hong+Wang%2C+Hao%2DXiang+Xu%2C+Zhen%2DHua+Ling%2C+and+Jia%2DChen+Gu%2E+2025%2E+Perturbation%2Drestrained+sequential+model+editing%2E+In+The+Thirteenth+International+Conference+on+Learning+Representations%2E)
12. Ben Mann, Nick Ryder, Melanie Subbiah, J Kaplan, P Dhariwal, A Neelakantan, P Shyam, G Sastry, A Askell, S Agarwal, and 1 others. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). arXiv preprint arXiv:2005.14165, 1(3):3.
13. Jiaan Wang, Yunlong Liang, Zengkui Sun, Yuxuan Cao, Jiarong Xu, and Fandong Meng. 2024a. [Cross-lingual knowledge editing in large language models](https://md.research.elamur.ai/free/en/papers/wang-2024-cross-lingual-knowledge-editing-large/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11676–11686.
14. Song Wang, Yaochen Zhu, Haochen Liu, Zaiyi Zheng, Chen Chen, and Jundong Li. 2024b. Knowledge editing for [large language models: A survey](https://md.research.elamur.ai/free/en/papers/minaee-2024-large-language-models-survey/). ACM Computing Surveys, 57(3):1–37.
15. Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D Manning, and Christopher Potts. 2024. ReFT: Representation fine-tuning for language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Zhengxuan+Wu%2C+Aryaman+Arora%2C+Zheng+Wang%2C+Atticus+Geiger%2C+Dan+Jurafsky%2C+Christopher+D+Manning%2C+and+Christopher+Potts%2E+2024%2E+ReFT%3A+Representation+fine%2Dtuning+for+language+models%2E+In+The+Thirty%2Deighth+An)
16. Kevin Meng, David Bau, Alex J Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Kevin+Meng%2C+David+Bau%2C+Alex+J+Andonian%2C+and+Yonatan+Belinkov%2E+2022%2E+Locating+and+editing+factual+associations+in+GPT%2E+In+Advances+in+Neural+Information+Processing+Systems%2E)
17. Haoyu Xu, Pengxiang Lan, Enneng Yang, Guibing Guo, Jianzhe Zhao, Linying Jiang, and Xingwei Wang. 2025. Knowledge decoupling via orthogonal projection for lifelong editing of large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1:… [find](https://scholar.google.com/scholar?q=Haoyu+Xu%2C+Pengxiang+Lan%2C+Enneng+Yang%2C+Guibing+Guo%2C+Jianzhe+Zhao%2C+Linying+Jiang%2C+and+Xingwei+Wang%2E+2025%2E+Knowledge+decoupling+via+orthogonal+projection+for+lifelong+editing+of+large+language+models%2E+In)
18. Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. 2023. Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Kevin+Meng%2C+Arnab+Sen+Sharma%2C+Alex+J+Andonian%2C+Yonatan+Belinkov%2C+and+David+Bau%2E+2023%2E+Mass%2Dediting+memory+in+a+transformer%2E+In+The+Eleventh+International+Conference+on+Learning+Representations%2E)
19. Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. 2023. Editing large language models: Problems, methods, and opportunities. arXiv preprint arXiv:2305.13172. [find](https://scholar.google.com/scholar?q=Yunzhi+Yao%2C+Peng+Wang%2C+Bozhong+Tian%2C+Siyuan+Cheng%2C+Zhoubo+Li%2C+Shumin+Deng%2C+Huajun+Chen%2C+and+Ningyu+Zhang%2E+2023%2E+Editing+large+language+models%3A+Problems%2C+methods%2C+and+opportunities%2E+arXiv+preprint+arXi)
20. Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn. 2022. Memory-based model editing at scale. In International Conference on Machine Learning, pages 15817–15831. PMLR. [find](https://scholar.google.com/scholar?q=Eric+Mitchell%2C+Charles+Lin%2C+Antoine+Bosselut%2C+Christopher+D+Manning%2C+and+Chelsea+Finn%2E+2022%2E+Memory%2Dbased+model+editing+at+scale%2E+In+International+Conference+on+Machine+Learning%2C+pages+15817%E2%80%9315831%2E+PM)
21. Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. 2020. Modifying memories in transformer models. arXiv preprint arXiv:2012.00363. [find](https://scholar.google.com/scholar?q=Chen+Zhu%2C+Ankit+Singh+Rawat%2C+Manzil+Zaheer%2C+Srinadh+Bhojanapalli%2C+Daliang+Li%2C+Felix+Yu%2C+and+Sanjiv+Kumar%2E+2020%2E+Modifying+memories+in+transformer+models%2E+arXiv+preprint+arXiv%3A2012%2E00363%2E)
22. David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. 2019. Experience replay for continual learning. Advances in neural information processing systems, 32. [find](https://scholar.google.com/scholar?q=David+Rolnick%2C+Arun+Ahuja%2C+Jonathan+Schwarz%2C+Timothy+Lillicrap%2C+and+Gregory+Wayne%2E+2019%2E+Experience+replay+for+continual+learning%2E+Advances+in+neural+information+processing+systems%2C+32%2E)
23. Marco Scialanga, Thibault Laugel, Vincent Grari, and Marcin Detyniecki. 2025. SAKE: Steering activations for knowledge editing. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15966–15978, Vienna, Austria. Association for Comp… [find](https://scholar.google.com/scholar?q=Marco+Scialanga%2C+Thibault+Laugel%2C+Vincent+Grari%2C+and+Marcin+Detyniecki%2E+2025%2E+SAKE%3A+Steering+activations+for+knowledge+editing%2E+In+Proceedings+of+the+63rd+Annual+Meeting+of+the+Association+for+Computa)
24. Wei Sun, Tingyu Qu, Mingxiao Li, Jesse Davis, and Marie-Francine Moens. 2025. Mitigating negative interference in multilingual knowledge editing through [find](https://scholar.google.com/scholar?q=Wei+Sun%2C+Tingyu+Qu%2C+Mingxiao+Li%2C+Jesse+Davis%2C+and+Marie%2DFrancine+Moens%2E+2025%2E+Mitigating+negative+interference+in+multilingual+knowledge+editing+through)

*Updated 2 November 2026*