---
title: "ArcLight — paper by Xu et al. (2026) — summary and ideas"
description: "The authors present ArcLight, a compact C++ inference engine of roughly ten source and header files, written from scratch for… 16 references."
authors: ["Yuzhuang Xu", "Xu Han", "Yuxuan Li", "Wanxiang Che"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)"
source: "https://aclanthology.org/2026.acl-demo.18/"
doi: "10.18653/v1/2026.acl-demo.18"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xu-2026-arclight-lightweight-llm-inference-architecture/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xu-2026-arclight-lightweight-llm-inference-architecture/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/23/) / ArcLight: A Lightweight LLM Inference Architecture for Many-Core CPUs

# ArcLight: A Lightweight LLM Inference Architecture for Many-Core CPUs

**Authors:** [Yuzhuang Xu](https://md.research.elamur.ai/free/en/authors/yuzhuang-xu/), [Xu Han](https://md.research.elamur.ai/free/en/authors/xu-han/), [Yuxuan Li](https://md.research.elamur.ai/free/en/authors/yuxuan-li/) and [Wanxiang Che](https://md.research.elamur.ai/free/en/authors/wanxiang-che/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-demo.18/) · [PDF](https://aclanthology.org/2026.acl-demo.18.pdf) · [DOI 10.18653/v1/2026.acl-demo.18](https://doi.org/10.18653/v1/2026.acl-demo.18) · [OpenAlex](https://openalex.org/W7166836230)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
CPU frameworks for LLM inference, such as llama.cpp, scale poorly on many-core server CPUs. These machines split cores and memory into several NUMA nodes, and reading memory attached to another node is much slower than reading local memory. Existing tools largely ignore this cost. The authors present ArcLight, a compact C++ inference engine of roughly ten source and header files, written from scratch for such hardware. It separates a decoding frontend (weight loading, model definition, the generation loop) from a backend. The backend provides an object-oriented tensor library, a static graph builder, a scheduler and kernels reused from llama.cpp. Its main ideas are:
- separate memory buffers allocated on each NUMA node;
- a thread pool that can be regrouped on the fly;
- tensor parallelism across NUMA nodes, using Scatter/Gather operators and subgraphs that synchronize only within their own thread group.

On a 192-core ARM server, ArcLight decodes up to 46% faster than llama.cpp. This shows that NUMA-aware design can unlock substantially more CPU inference performance. The minimal codebase is also meant to be easy to modify for research and teaching.

## Key points
- Test setup: 4 NUMA nodes, each with 48 HUAWEI Kunpeng-920 (ARMv8.2) cores and 6 DDR4 channels, running Ubuntu 22.04. The model was Qwen3-4B quantized to Q4_0, with NEON kernels, a 15-token prompt and 256 generated tokens.
- ArcLight reaches up to 46% higher decoding throughput than llama.cpp. Even on a single NUMA node it is slightly faster, because all of its memory is allocated locally.
- Letting thread groups run their subgraphs asynchronously, with global barriers only at the start and end, adds about 5 tokens/s.
- Memory handling uses one buffer per NUMA node and alternates two activation buffers by layer parity, which reduces runtime memory use.
- The code is open source at https://github.com/OpenBMB/ArcLight. So far it has been evaluated only on ARM, and x86 support is listed as future work.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 415 works · 94,028 citations
- [Harbin Institute of Technology](https://md.research.elamur.ai/free/en/universities/harbin-institute-of-technology/) — University, Harbin, China · 185 works · 29,140 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works · [🔌 Hardware and Architecture](https://md.research.elamur.ai/free/en/shelves/computer-science/hardware-and-architecture/) · 167 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Yuzhuang Xu, Xu Han, Yuxuan Li and Wanxiang Che. 2026. ArcLight: A Lightweight LLM Inference Architecture for Many-Core CPUs. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{xu2026arclight,
  title = {{ArcLight: A Lightweight LLM Inference Architecture for Many-Core CPUs}},
  author = {Yuzhuang Xu, Xu Han, Yuxuan Li and Wanxiang Che},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-demo.18},
  url = {https://doi.org/10.18653/v1/2026.acl-demo.18}
}
```

- [← previous ARCHITECT: Uncertainty-Aware Dynamic Tool Learning via Causal Intervention for Open-World Agents](https://md.research.elamur.ai/free/en/papers/wang-2026-architect-uncertainty-aware-dynamic-tool/)
- [next → Are All Prompt Components Value-Neutral? Understanding the Heterogeneous Adversarial Robustness of Dissected Prompt in LLMs](https://md.research.elamur.ai/free/en/papers/zheng-2026-are-all-prompt-components-value/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/23/)

## References

16 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Tri Dao. 2024. [FlashAttention-2: Faster attention with better parallelism and work partitioning](https://md.research.elamur.ai/free/en/papers/dao-2023-flashattention-2-faster-attention-better/). In Proceedings of the Twelfth International Conference on Learning Representations (ICLR).
2. Georgi Gerganov and contributors. 2023. llama.cpp: LLM inference in C/C++. [https://github.com/](https://github.com/) ggml-org/llama.cpp. Accessed: 2025-10-01. [↗](https://github.com/)
3. Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia Xu Chen, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, and Zhifeng Chen. 2019. GPipe: Efficient training of giant neural networks using pipeline parallelism. Advances in Neural Information Processing Systems (NeurIPS), pa… [find](https://scholar.google.com/scholar?q=Yanping+Huang%2C+Youlong+Cheng%2C+Ankur+Bapna%2C+Orhan+Firat%2C+Mia+Xu+Chen%2C+Dehao+Chen%2C+HyoukJoong+Lee%2C+Jiquan+Ngiam%2C+Quoc+V+Le%2C+Yonghui+Wu%2C+and+Zhifeng+Chen%2E+2019%2E+GPipe%3A+Efficient+training+of+giant+neural+)
4. Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. 2023. [Reducing activation recomputation in large transformer models](https://md.research.elamur.ai/free/en/papers/korthikanti-2022-reducing-activation-recomputation-large-transformer/). Proceedings of Machine Learning and Systems (ML-Sys), pages 341–353.
5. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. [Efficient memory management for large language model serving with PagedAttention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/). In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles (SOSP…
6. Microsoft Corporation. 2019. ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator. [https://github.com/microsoft/](https://github.com/microsoft/) onnxruntime. [↗](https://github.com/microsoft/)
7. NVIDIA Corporation. 2019. NVIDIA TensorRT: SDK for high-performance deep learning inference on NVIDIA GPUs. [https://github.com/NVIDIA/](https://github.com/NVIDIA/) TensorRT. [↗](https://github.com/NVIDIA/)
8. Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. 2024. [OmniQuant: Omnidirectionally calibrated quantization for large language models](https://md.research.elamur.ai/free/en/papers/shao-2023-omniquant-omnidirectionally-calibrated-quantization-large/). In Proceedings of the Twelfth International Conference on Learning Representations (ICLR).
9. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-LM: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053. [find](https://scholar.google.com/scholar?q=Mohammad+Shoeybi%2C+Mostofa+Patwary%2C+Raul+Puri%2C+Patrick+LeGresley%2C+Jared+Casper%2C+and+Bryan+Catanzaro%2E+2019%2E+Megatron%2DLM%3A+Training+multi%2Dbillion+parameter+language+models+using+model+parallelism%2E+arXiv+p)
10. Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. 2024. A simple and effective pruning approach for large language models. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Mingjie+Sun%2C+Zhuang+Liu%2C+Anna+Bair%2C+and+J+Zico+Kolter%2E+2024%2E+A+simple+and+effective+pruning+approach+for+large+language+models%2E+In+Proceedings+of+the+Twelfth+International+Conference+on+Learning+Repre)
11. Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. 2024. [QuIP\#: Even better LLM quantization with hadamard incoherence and lattice codebooks](https://md.research.elamur.ai/free/en/papers/tseng-2024-quip-even-better-llm-quantization/). In Proceedings of International Conference on Machine Learning (ICML), pages 48630–48656.
12. Yuzhuang Xu, Xu Han, Zonghan Yang, Shuo Wang, Qingfu Zhu, Zhiyuan Liu, Weidong Liu, and Wanxiang Che. 2024. Onebit: Towards extremely low-bit large language models. Advances in Neural Information Processing Systems (NeurIPS), pages 66357– 66382. [find](https://scholar.google.com/scholar?q=Yuzhuang+Xu%2C+Xu+Han%2C+Zonghan+Yang%2C+Shuo+Wang%2C+Qingfu+Zhu%2C+Zhiyuan+Liu%2C+Weidong+Liu%2C+and+Wanxiang+Che%2E+2024%2E+Onebit%3A+Towards+extremely+low%2Dbit+large+language+models%2E+Advances+in+Neural+Information+Proc)
13. Yuzhuang Xu, Xu Han, Yuanchi Zhang, Yixuan Wang, Yijun Liu, Shiyu Ji, Qingfu Zhu, and Wanxiang Che. 2026. CAMERA: Multi-matrix joint compression for MoE models via micro-expert redundancy analysis. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pages 27395–27404. [find](https://scholar.google.com/scholar?q=Yuzhuang+Xu%2C+Xu+Han%2C+Yuanchi+Zhang%2C+Yixuan+Wang%2C+Yijun+Liu%2C+Shiyu+Ji%2C+Qingfu+Zhu%2C+and+Wanxiang+Che%2E+2026%2E+CAMERA%3A+Multi%2Dmatrix+joint+compression+for+MoE+models+via+micro%2Dexpert+redundancy+analysis%2E+In)
14. Yuzhuang Xu, Shiyu Ji, Qingfu Zhu, and Wanxiang Che. 2025. CRVQ: Channel-relaxed vector quantization for extreme compression of LLMs. Transactions of the Association for Computational Linguistics (TACL), 13:1488–1506. [find](https://scholar.google.com/scholar?q=Yuzhuang+Xu%2C+Shiyu+Ji%2C+Qingfu+Zhu%2C+and+Wanxiang+Che%2E+2025%2E+CRVQ%3A+Channel%2Drelaxed+vector+quantization+for+extreme+compression+of+LLMs%2E+Transactions+of+the+Association+for+Computational+Linguistics+%28TAC)
15. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). arXiv preprint arXiv:2505.09388.
16. Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, and 1 others. 2024. [SGLang: Efficient execution of structured language model programs](https://md.research.elamur.ai/free/en/papers/zheng-2023-sglang-efficient-execution-structured-language/). Advances in Neural Information Processing Systems (NeurIPS), pa…

*Updated 2 November 2026*