---
title: "Let's Focus on Neuron — paper by Xu et al. (2025)"
description: "Xu et al. argue that training can be targeted much more finely, down to individual neurons, which they… 43 references, cited by 10 papers here."
authors: ["Haoyun Xu", "Runzhe Zhan", "Yingpeng Ma", "Derek F. Wong", "Lidia S. Chao"]
year: 2025
venue: "Proceedings of the 31st International Conference on Computational Linguistics"
source: "https://aclanthology.org/2025.coling-main.630/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xu-2025-let-s-focus-neuron-neuron/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xu-2025-let-s-focus-neuron-neuron/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/8/) / Let's Focus on Neuron: Neuron-Level Supervised Fine-tuning for Large Language Model

# Let's Focus on Neuron: Neuron-Level Supervised Fine-tuning for Large Language Model

**Authors:** Haoyun Xu, [Runzhe Zhan](https://md.research.elamur.ai/free/en/authors/runzhe-zhan/), Yingpeng Ma, [Derek F. Wong](https://md.research.elamur.ai/free/en/authors/derek-f-wong/) and [Lidia S. Chao](https://md.research.elamur.ai/free/en/authors/lidia-s-chao/) · **Year:** 2025 · **Venue:** [Proceedings of the 31st International Conference on Computational Linguistics](https://md.research.elamur.ai/free/en/venues/coling/) ([COLING 2025](https://md.research.elamur.ai/free/en/venues/coling/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.coling-main.630/) · [PDF](https://aclanthology.org/2025.coling-main.630.pdf) · [OpenAlex](https://openalex.org/W4392974112)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [10](https://md.research.elamur.ai/free/en/papers/xu-2025-let-s-focus-neuron-neuron/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Standard supervised fine-tuning updates every weight of a large language model. Popular PEFT methods such as adapters or LoRA reduce this cost, but they still decide what to train at the level of whole layers or modules. Xu et al. argue that training can be targeted much more finely, down to individual neurons, which they define as rows of a weight matrix.

Their method, NeFT, works in three steps:
- It runs a short full-parameter fine-tuning pass.
- It compares each neuron in the original and tuned models using cosine similarity, and marks the neurons that drifted most as task-sensitive.
- It fine-tunes the original model again while zeroing the gradients of all other neurons, so no extra modules are needed.

A probe-based pilot on XNLI with Llama-2-7b-chat came first, followed by experiments on translation and summarization. NeFT generally outperforms full fine-tuning and matches or beats LoRA at similar parameter budgets. The work matters because it reduces the number of trainable parameters. It also offers an interpretability view of which neurons a task actually depends on.

## Key points
- **Pilot study:** a Ridge probe trained on XNLI hidden states of Llama-2-7b-chat selected 100,000 neurons, about 6% of all parameters. Training only these beat full fine-tuning in accuracy:
  - English: 84.9 vs 82.8
  - French: 76.9 vs 70.7
  - German: 80.2 vs 67.0
- **Parameter-matched comparison with LoRA** on English-to-Chinese translation (20k examples): NeFT reached lower validation loss and converged faster in both pairings.
  - NeFT 0.4% (25.6M params) vs LoRA r=16 (40.0M)
  - NeFT 9% (614.4M) vs LoRA r=256 (639.6M)
- **Selection does not require a converged model:** neurons chosen after only 800 training steps still worked well. Neuron sets selected under different settings overlapped by 61–76%, and the overlap grew as the selected share rose from 3% to 12%.
- **Transfer:** neurons identified on one En-Zh dataset kept their performance when applied to a separate, non-overlapping En-Zh dataset.
- **Analysis tools:** the authors sort neurons into strongly affected, suppressed, and indirectly affected types, and introduce a "rank difference" measure of how neurons are used. Code and data are available at github.com/NLP2CT/NeFT.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Macau](https://md.research.elamur.ai/free/en/universities/university-of-macau/) — University, Macao, China · 33 works · 2,223 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Haoyun Xu, Runzhe Zhan, Yingpeng Ma, Derek F. Wong and Lidia S. Chao. 2025. Let's Focus on Neuron: Neuron-Level Supervised Fine-tuning for Large Language Model. Proceedings of the 31st International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{xu2025let,
  title = {{Let's Focus on Neuron: Neuron-Level Supervised Fine-tuning for Large Language Model}},
  author = {Haoyun Xu, Runzhe Zhan, Yingpeng Ma, Derek F. Wong and Lidia S. Chao},
  booktitle = {Proceedings of the 31st International Conference on Computational Linguistics},
  year = {2025},
  url = {https://aclanthology.org/2025.coling-main.630/}
}
```

- [← previous JustLogic: A Comprehensive Benchmark for Evaluating Deductive Reasoning in Large Language Models](https://md.research.elamur.ai/free/en/papers/chen-2025-justlogic-comprehensive-benchmark-evaluating-deductive/)
- [next → Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation](https://md.research.elamur.ai/free/en/papers/mahdavi-2025-leveraging-online-olympiad-level-math/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/8/)

## References

43 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. First of all, this work is limited by experimenting on task-specific benchmarks using Llama-2-7B model. Moreover, this work encounters certain constraints due to the capabilities of current distributed frameworks, particularly regarding gradient operations. Specifically, we are unable to apply gradi… [find](https://scholar.google.com/scholar?q=First+of+all%2C+this+work+is+limited+by+experimenting+on+task%2Dspecific+benchmarks+using+Llama%2D2%2D7B+model%2E+Moreover%2C+this+work+encounters+certain+constraints+due+to+the+capabilities+of+current+distribute)
2. Alan Ansell, Ivan Vulic, Hannah Sterz, Anna Korhonen, and Edoardo M. Ponti. 2024. Scaling sparse fine-tuning to large language models. CoRR, abs/2401.16405. — [arXiv:2401.16405](https://arxiv.org/abs/2401.16405) [find](https://scholar.google.com/scholar?q=Alan+Ansell%2C+Ivan+Vulic%2C+Hannah+Sterz%2C+Anna+Korhonen%2C+and+Edoardo+M%2E+Ponti%2E+2024%2E+Scaling+sparse+fine%2Dtuning+to+large+language+models%2E+CoRR%2C+abs%2F2401%2E16405%2E+%E2%80%94+%5BarXiv%3A2401%2E16405%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F)
3. Abhik Bhattacharjee, Tahmid Hasan, Wasi Uddin Ahmad, Yuan-Fang Li, Yong-Bin Kang, and Rifat Shahriyar. 2023. Crosssum: Beyond english-centric cross-lingual summarization for 1, 500+ language pairs. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1:… [find](https://scholar.google.com/scholar?q=Abhik+Bhattacharjee%2C+Tahmid+Hasan%2C+Wasi+Uddin+Ahmad%2C+Yuan%2DFang+Li%2C+Yong%2DBin+Kang%2C+and+Rifat+Shahriyar%2E+2023%2E+Crosssum%3A+Beyond+english%2Dcentric+cross%2Dlingual+summarization+for+1%2C+500%2B+language+pairs%2E+In)
4. Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. 2020. The lottery ticket hypothesis for pre-trained bert networks. Advances in neural information processing systems, 33:15834–15846. [find](https://scholar.google.com/scholar?q=Tianlong+Chen%2C+Jonathan+Frankle%2C+Shiyu+Chang%2C+Sijia+Liu%2C+Yang+Zhang%2C+Zhangyang+Wang%2C+and+Michael+Carbin%2E+2020%2E+The+lottery+ticket+hypothesis+for+pre%2Dtrained+bert+networks%2E+Advances+in+neural+informati)
5. Tri Dao, Beidi Chen, Nimit S Sohoni, Arjun Desai, Michael Poli, Jessica Grogan, Alexander Liu, Aniruddh Rao, Atri Rudra, and Christopher Ré. 2022. Monarch: Expressive structured matrices for efficient and accurate training. In International Conference on Machine Learning, pages 4690–4721. PMLR. [find](https://scholar.google.com/scholar?q=Tri+Dao%2C+Beidi+Chen%2C+Nimit+S+Sohoni%2C+Arjun+Desai%2C+Michael+Poli%2C+Jessica+Grogan%2C+Alexander+Liu%2C+Aniruddh+Rao%2C+Atri+Rudra%2C+and+Christopher+R%C3%A9%2E+2022%2E+Monarch%3A+Expressive+structured+matrices+for+efficient)
6. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez,… — [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/)
7. Nadir Durrani, Hassan Sajjad, Fahim Dalvi, and Yonatan Belinkov. 2020. Analyzing individual neurons in pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 4865–4880. Association for Co… [find](https://scholar.google.com/scholar?q=Nadir+Durrani%2C+Hassan+Sajjad%2C+Fahim+Dalvi%2C+and+Yonatan+Belinkov%2E+2020%2E+Analyzing+individual+neurons+in+pre%2Dtrained+language+models%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natura)
8. Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, and Xinchao Wang. 2023. Depgraph: Towards any structural pruning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 16091–16101. IEEE. [find](https://scholar.google.com/scholar?q=Gongfan+Fang%2C+Xinyin+Ma%2C+Mingli+Song%2C+Michael+Bi+Mi%2C+and+Xinchao+Wang%2E+2023%2E+Depgraph%3A+Towards+any+structural+pruning%2E+In+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+CVPR+2023%2C+Van)
9. Elias Frantar and Dan Alistarh. 2023. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 10323–10337. PMLR. [find](https://scholar.google.com/scholar?q=Elias+Frantar+and+Dan+Alistarh%2E+2023%2E+Sparsegpt%3A+Massive+language+models+can+be+accurately+pruned+in+one%2Dshot%2E+In+International+Conference+on+Machine+Learning%2C+ICML+2023%2C+23%2D29+July+2023%2C+Honolulu%2C+Ha)
10. Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022. The flores-101 evaluation benchmark for low-resource and multilingual machine translation. Trans. Assoc. Comput. Linguistics, 10:522–53… [find](https://scholar.google.com/scholar?q=Naman+Goyal%2C+Cynthia+Gao%2C+Vishrav+Chaudhary%2C+Peng%2DJen+Chen%2C+Guillaume+Wenzek%2C+Da+Ju%2C+Sanjana+Krishnan%2C+Marc%E2%80%99Aurelio+Ranzato%2C+Francisco+Guzm%C3%A1n%2C+and+Angela+Fan%2E+2022%2E+The+flores%2D101+evaluation+benchmark)
11. Demi Guo, Alexander M. Rush, and Yoon Kim. 2021a. [Parameter-efficient transfer learning with diff pruning](https://md.research.elamur.ai/free/en/papers/guo-2021-parameter-efficient-transfer-learning-diff/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1…
12. Demi Guo, Alexander M Rush, and Yoon Kim. 2021b. [Parameter-efficient transfer learning with diff pruning](https://md.research.elamur.ai/free/en/papers/guo-2021-parameter-efficient-transfer-learning-diff/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pag…
13. Wes Gurnee and Max Tegmark. 2023. Language models represent space and time. CoRR, abs/2310.02207. [find](https://scholar.google.com/scholar?q=Wes+Gurnee+and+Max+Tegmark%2E+2023%2E+Language+models+represent+space+and+time%2E+CoRR%2C+abs%2F2310%2E02207%2E)
14. Song Han, Jeff Pool, John Tran, and William J. Dally. 2015. Learning both weights and connections for efficient neural networks. CoRR, abs/1506.02626. [find](https://scholar.google.com/scholar?q=Song+Han%2C+Jeff+Pool%2C+John+Tran%2C+and+William+J%2E+Dally%2E+2015%2E+Learning+both+weights+and+connections+for+efficient+neural+networks%2E+CoRR%2C+abs%2F1506%2E02626%2E)
15. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
16. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Lon… [find](https://scholar.google.com/scholar?q=Neil+Houlsby%2C+Andrei+Giurgiu%2C+Stanislaw+Jastrzebski%2C+Bruna+Morrone%2C+Quentin+de+Laroussilhe%2C+Andrea+Gesmundo%2C+Mona+Attariyan%2C+and+Sylvain+Gelly%2E+2019%2E+Parameter%2Defficient+transfer+learning+for+NLP%2E+In+)
17. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
18. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+Las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
19. Eldar Kurtic, Denis Kuznedelev, Elias Frantar, Michael Goin, and Dan Alistarh. 2023. Sparse fine-tuning for inference acceleration of large language models. CoRR, abs/2310.06927. [find](https://scholar.google.com/scholar?q=Eldar+Kurtic%2C+Denis+Kuznedelev%2C+Elias+Frantar%2C+Michael+Goin%2C+and+Dan+Alistarh%2E+2023%2E+Sparse+fine%2Dtuning+for+inference+acceleration+of+large+language+models%2E+CoRR%2C+abs%2F2310%2E06927%2E)
20. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. [The power of scale for parameter-efficient prompt tuning](https://md.research.elamur.ai/free/en/papers/lester-2021-power-scale-parameter-efficient-prompt/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 3045– 3059.…
21. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Pa… [find](https://scholar.google.com/scholar?q=Xiang+Lisa+Li+and+Percy+Liang%2E+2021%2E+Prefix%2Dtuning%3A+Optimizing+continuous+prompts+for+generation%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+the+11t)
22. Chin-Yew Lin. 2004. [ROUGE: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
23. Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. 2022. [Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning](https://md.research.elamur.ai/free/en/papers/liu-2022-few-shot-parameter-efficient-fine/). In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processin…
24. Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Ré, and Beidi Chen. 2023. Deja vu: Contextual sparsity for efficient llms at inference time. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023,… [find](https://scholar.google.com/scholar?q=Zichang+Liu%2C+Jue+Wang%2C+Tri+Dao%2C+Tianyi+Zhou%2C+Binhang+Yuan%2C+Zhao+Song%2C+Anshumali+Shrivastava%2C+Ce+Zhang%2C+Yuandong+Tian%2C+Christopher+R%C3%A9%2C+and+Beidi+Chen%2E+2023%2E+Deja+vu%3A+Contextual+sparsity+for+efficient+l)
25. Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. [Llm-pruner: On the structural pruning of large language models](https://md.research.elamur.ai/free/en/papers/ma-2023-llm-pruner-structural-pruning-large/). CoRR, abs/2305.11627.
26. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA, pages 311–318. ACL.
27. Roma Patel and Ellie Pavlick. 2022. Mapping language models to grounded conceptual spaces. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net. [find](https://scholar.google.com/scholar?q=Roma+Patel+and+Ellie+Pavlick%2E+2022%2E+Mapping+language+models+to+grounded+conceptual+spaces%2E+In+The+Tenth+International+Conference+on+Learning+Representations%2C+ICLR+2022%2C+Virtual+Event%2C+April+25%2D29%2C+202)
28. Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2021. [Adapterfusion: Non-destructive task composition for transfer learning](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-adapterfusion-non-destructive-task-composition/). In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, On…
29. Sai Prasanna, Anna Rogers, and Anna Rumshisky. 2020. When bert plays the lottery, all tickets are winning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3208–3229. [find](https://scholar.google.com/scholar?q=Sai+Prasanna%2C+Anna+Rogers%2C+and+Anna+Rumshisky%2E+2020%2E+When+bert+plays+the+lottery%2C+all+tickets+are+winning%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EM)
30. Ricardo Rei, Craig Stewart, Ana C. Farinha, and Alon Lavie. 2020. [COMET: A neural framework for MT evaluation](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 2685–2702. Association for Computational Linguisti…
31. Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. 2021. [Adapterdrop: On the efficiency of adapters in transformers](https://md.research.elamur.ai/free/en/papers/ruckle-2021-adapterdrop-efficiency-adapters-transformers/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta C…
32. Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. 2023. Powerinfer: Fast large language model serving with a consumer-grade GPU. CoRR, abs/2312.12456. [find](https://scholar.google.com/scholar?q=Yixin+Song%2C+Zeyu+Mi%2C+Haotong+Xie%2C+and+Haibo+Chen%2E+2023%2E+Powerinfer%3A+Fast+large+language+model+serving+with+a+consumer%2Dgrade+GPU%2E+CoRR%2C+abs%2F2312%2E12456%2E)
33. Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. 2023. A simple and effective pruning approach for large language models. CoRR, abs/2306.11695. [find](https://scholar.google.com/scholar?q=Mingjie+Sun%2C+Zhuang+Liu%2C+Anna+Bair%2C+and+J%2E+Zico+Kolter%2E+2023%2E+A+simple+and+effective+pruning+approach+for+large+language+models%2E+CoRR%2C+abs%2F2306%2E11695%2E)
34. Yi-Lin Sung, Varun Nair, and Colin Raffel. 2021. Training neural networks with fixed sparse masks. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 24193–24205. [find](https://scholar.google.com/scholar?q=Yi%2DLin+Sung%2C+Varun+Nair%2C+and+Colin+Raffel%2E+2021%2E+Training+neural+networks+with+fixed+sparse+masks%2E+In+Advances+in+Neural+Information+Processing+Systems+34%3A+Annual+Conference+on+Neural+Information+Proc)
35. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Alpaca: A strong, replicable instruction-following model. Stanford Center for Research on Foundation Models. [https://crfm.stanford](https://crfm.stanford). edu/2023/03/13/alpaca. html, 3(6):… [↗](https://crfm.stanford)
36. Vithursan Thangarasa, Abhay Gupta, William Marshall, Tianda Li, Kevin Leong, Dennis DeCoste, Sean Lie, and Shreyas Saxena. 2023. Spdf: Sparse pre-training and dense fine-tuning for large language models. In Uncertainty in Artificial Intelligence, pages 2134– 2146. PMLR. [find](https://scholar.google.com/scholar?q=Vithursan+Thangarasa%2C+Abhay+Gupta%2C+William+Marshall%2C+Tianda+Li%2C+Kevin+Leong%2C+Dennis+DeCoste%2C+Sean+Lie%2C+and+Shreyas+Saxena%2E+2023%2E+Spdf%3A+Sparse+pre%2Dtraining+and+dense+fine%2Dtuning+for+large+language+mode)
37. Jörg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation, LREC 2012, Istanbul, Turkey, May 23-25, 2012, pages 2214–2218. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=J%C3%B6rg+Tiedemann%2E+2012%2E+Parallel+data%2C+tools+and+interfaces+in+OPUS%2E+In+Proceedings+of+the+Eighth+International+Conference+on+Language+Resources+and+Evaluation%2C+LREC+2012%2C+Istanbul%2C+Turkey%2C+May+23%2D25%2C+2)
38. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
39. Elena Voita, Javier Ferrando, and Christoforos Nalmpantis. 2023. Neurons in large language models: Dead, n-gram, positional. CoRR, abs/2309.04827. [find](https://scholar.google.com/scholar?q=Elena+Voita%2C+Javier+Ferrando%2C+and+Christoforos+Nalmpantis%2E+2023%2E+Neurons+in+large+language+models%3A+Dead%2C+n%2Dgram%2C+positional%2E+CoRR%2C+abs%2F2309%2E04827%2E)
40. Fei Yuan, Yinquan Lu, Wenhao Zhu, Lingpeng Kong, Lei Li, Yu Qiao, and Jingjing Xu. 2023a. Lego-mt: Learning detachable models for massively multilingual machine translation. In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pages 11518– 11533.… [find](https://scholar.google.com/scholar?q=Fei+Yuan%2C+Yinquan+Lu%2C+Wenhao+Zhu%2C+Lingpeng+Kong%2C+Lei+Li%2C+Yu+Qiao%2C+and+Jingjing+Xu%2E+2023a%2E+Lego%2Dmt%3A+Learning+detachable+models+for+massively+multilingual+machine+translation%2E+In+Findings+of+the+Associa)
41. Fei Yuan, Shuai Yuan, Zhiyong Wu, and Lei Li. 2023b. How multilingual is multilingual llm? CoRR, abs/2311.09071. — [arXiv:2311.09071](https://arxiv.org/abs/2311.09071) [find](https://scholar.google.com/scholar?q=Fei+Yuan%2C+Shuai+Yuan%2C+Zhiyong+Wu%2C+and+Lei+Li%2E+2023b%2E+How+multilingual+is+multilingual+llm%3F+CoRR%2C+abs%2F2311%2E09071%2E+%E2%80%94+%5BarXiv%3A2311%2E09071%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2311%2E09071%29)
42. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. [Instruction-following evaluation for large language models](https://md.research.elamur.ai/free/en/papers/zhou-2023-instruction-following-evaluation-large-language/). CoRR, abs/2311.07911.
43. Tao Zhuang, Zhixuan Zhang, Yuheng Huang, Xiaoyi Zeng, Kai Shuang, and Xiang Li. 2020. Neuron-level structured pruning using polarization regularizer. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12,… [find](https://scholar.google.com/scholar?q=Tao+Zhuang%2C+Zhixuan+Zhang%2C+Yuheng+Huang%2C+Xiaoyi+Zeng%2C+Kai+Shuang%2C+and+Xiang+Li%2E+2020%2E+Neuron%2Dlevel+structured+pruning+using+polarization+regularizer%2E+In+Advances+in+Neural+Information+Processing+Syste)

*Updated 30 October 2026*