---
title: "HuoziIME — paper by Shan et al. (2026) — summary and ideas"
description: "Chinese mobile keyboards still mostly convert pinyin to characters. 15 references. Among them: 0."
authors: ["Baocai Shan", "Yuzhuang Xu", "Wanxiang Che"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)"
source: "https://aclanthology.org/2026.acl-demo.32/"
doi: "10.18653/v1/2026.acl-demo.32"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shan-2026-huoziime-device-llm-enhanced-input/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shan-2026-huoziime-device-llm-enhanced-input/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/32/) / HuoziIME: An On-Device LLM-Enhanced Input Method for Deep Personalization

# HuoziIME: An On-Device LLM-Enhanced Input Method for Deep Personalization

**Authors:** [Baocai Shan](https://md.research.elamur.ai/free/en/authors/baocai-shan/), [Yuzhuang Xu](https://md.research.elamur.ai/free/en/authors/yuzhuang-xu/) and [Wanxiang Che](https://md.research.elamur.ai/free/en/authors/wanxiang-che/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-demo.32/) · [PDF](https://aclanthology.org/2026.acl-demo.32.pdf) · [DOI 10.18653/v1/2026.acl-demo.32](https://doi.org/10.18653/v1/2026.acl-demo.32) · [OpenAlex](https://openalex.org/W7166808692)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Chinese mobile keyboards still mostly convert pinyin to characters. Current LLM features in commercial IMEs are cloud-based extras that bring latency and privacy risks and offer only shallow personalization. HuoziIME addresses this by placing a small LLM at the centre of the typing loop and running it fully on the phone. The authors post-train Qwen3-0.6B on stylized conversational data covering three persona styles. The data come from Internet dialogues plus LLM-synthesized dialogues, refined through iterative filtering. Personalization relies on a three-tier memory: L1 holds cached KV states, L2 holds plaintext user facts with an HNSW index, and L3 logs interactions for later fine-tuning. A GRPO-trained trigger decides when retrieval is actually needed. On the systems side, the runtime adds a radix-tree KV cache for prefix reuse, KV-Splice injection of memory segments, and big.LITTLE thread scheduling. On a mobile CPU, the system decodes stably and keeps its memory footprint moderate. The work matters because it shows that private, memory-driven generative typing is feasible without cloud inference.

## Key points
- **Architecture and resources:**
  - Built on Qwen3-0.6B, extending the llama.cpp and YuyanIME codebases, with the result released under GPL-3.0.
  - Code is available at github.com/Shan-HIT/HuoziIME.
- **Memory and retrieval:**
  - Uses an L1/L2/L3 hierarchical memory.
  - A GRPO-based trigger chooses among four task classes: direct completion, memory retrieval, memory extraction, and refusal of invalid information.
- **Speed on device:**
  - Prefill throughput peaks above 260 tokens/s, and decoding holds at 24–25 tokens/s.
  - Cold-start Time to First Candidate grows from about 800 to 1700 ms as context reaches 512 tokens.
  - During continuous typing, KV reuse brings the perceived delay close to zero.
- **Memory footprint:**
  - The model is quantized from 1.34 GB to 485 MB (Q4_0).
  - The L1 KV cache is capped at 24 MB, and extra peak RAM settles around 1.12 GB.
- **Integration and evaluation:**
  - Context is shared across apps via the Model Context Protocol (MCP).
  - Suggestions appear as GhostText inline plus a secondary candidate panel.
  - User feedback is limited to four participants with NLP backgrounds.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Harbin Institute of Technology](https://md.research.elamur.ai/free/en/universities/harbin-institute-of-technology/) — University, Harbin, China · 158 works · 28,841 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Baocai Shan, Yuzhuang Xu and Wanxiang Che. 2026. HuoziIME: An On-Device LLM-Enhanced Input Method for Deep Personalization. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{shan2026huoziime,
  title = {{HuoziIME: An On-Device LLM-Enhanced Input Method for Deep Personalization}},
  author = {Baocai Shan, Yuzhuang Xu and Wanxiang Che},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-demo.32},
  url = {https://doi.org/10.18653/v1/2026.acl-demo.32}
}
```

- [← previous HumanLLM: Benchmarking and Improving LLM Anthropomorphism via Human Cognitive Patterns](https://md.research.elamur.ai/free/en/papers/wang-2026-humanllm-benchmarking-improving-llm-anthropomorphism/)
- [next → Hybrid CNN-Transformer Architecture for Arabic Speech Emotion Recognition](https://md.research.elamur.ai/free/en/papers/gheffari-2026-hybrid-cnn-transformer-architecture-arabic/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/32/)

## References

15 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anthropic. 2024. Model Context Protocol (MCP). Official Protocol Specification. [find](https://scholar.google.com/scholar?q=Anthropic%2E+2024%2E+Model+Context+Protocol+%28MCP%29%2E+Official+Protocol+Specification%2E)
2. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, and 29 others. 2023. [Qwen technical report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/). arXiv preprint, abs/2309.16609.
3. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. GPTQ: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323. [find](https://scholar.google.com/scholar?q=Elias+Frantar%2C+Saleh+Ashkboos%2C+Torsten+Hoefler%2C+and+Dan+Alistarh%2E+2022%2E+GPTQ%3A+Accurate+post%2Dtraining+quantization+for+generative+pre%2Dtrained+transformers%2E+arXiv+preprint+arXiv%3A2210%2E17323%2E)
4. Georgi Gerganov. 2023. llama.cpp: Port of Meta’s LLaMA model in C/C++. GitHub repository. [find](https://scholar.google.com/scholar?q=Georgi+Gerganov%2E+2023%2E+llama%2Ecpp%3A+Port+of+Meta%E2%80%99s+LLaMA+model+in+C%2FC%2B%2B%2E+GitHub+repository%2E)
5. Dan Gusfield. 1997. Algorithms on Strings, Trees, and Sequences: Computer Science and Computational Biology. Cambridge University Press, Cambridge, UK. [find](https://scholar.google.com/scholar?q=Dan+Gusfield%2E+1997%2E+Algorithms+on+Strings%2C+Trees%2C+and+Sequences%3A+Computer+Science+and+Computational+Biology%2E+Cambridge+University+Press%2C+Cambridge%2C+UK%2E)
6. Andrew Hard, Kanishka Rao, Rajiv Mathews, Françoise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage. 2018. Federated learning for mobile keyboard prediction. arXiv preprint, abs/1811.03604. [find](https://scholar.google.com/scholar?q=Andrew+Hard%2C+Kanishka+Rao%2C+Rajiv+Mathews%2C+Fran%C3%A7oise+Beaufays%2C+Sean+Augenstein%2C+Hubert+Eichner%2C+Chlo%C3%A9+Kiddon%2C+and+Daniel+Ramage%2E+2018%2E+Federated+learning+for+mobile+keyboard+prediction%2E+arXiv+preprint%2C)
7. Yafang Huang, Zuchao Li, Zhuosheng Zhang, and Hai Zhao. 2018. Moon IME: Neural-based Chinese Pinyin aided input method with customizable association. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 140–145. [find](https://scholar.google.com/scholar?q=Yafang+Huang%2C+Zuchao+Li%2C+Zhuosheng+Zhang%2C+and+Hai+Zhao%2E+2018%2E+Moon+IME%3A+Neural%2Dbased+Chinese+Pinyin+aided+input+method+with+customizable+association%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+A)
8. Yury A. Malkov and Dmitry A. Yashunin. 2018. [Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs](https://md.research.elamur.ai/free/en/papers/malkov-2016-efficient-robust-approximate-nearest-neighbor/). IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(4):824–836.
9. Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. 2024. MemGPT: Towards LLMs as operating systems. In Proceedings of the 12th International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Charles+Packer%2C+Sarah+Wooders%2C+Kevin+Lin%2C+Vivian+Fang%2C+Shishir+G%2E+Patil%2C+Ion+Stoica%2C+and+Joseph+E%2E+Gonzalez%2E+2024%2E+MemGPT%3A+Towards+LLMs+as+operating+systems%2E+In+Proceedings+of+the+12th+International+C)
10. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint, abs/1707.06347. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+2017%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint%2C+abs%2F1707%2E06347%2E)
11. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. [find](https://scholar.google.com/scholar?q=Zhihong+Shao%2C+Peiyi+Wang%2C+Qihao+Zhu%2C+Runxin+Xu%2C+Junxiao+Song%2C+Xiao+Bi%2C+Haowei+Zhang%2C+Mingchuan+Zhang%2C+Y%2E+K%2E+Li%2C+Y%2E+Wu%2C+and+Daya+Guo%2E+2024%2E)
12. [DeepSeekMath: Pushing the limits of mathematical reasoning in open language models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/). arXiv preprint, abs/2402.03300.
13. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2024. [RoFormer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/). Neurocomputing, 568:127063.
14. Yuzhuang Xu, Xu Han, Zonghan Yang, Shuo Wang, Qingfu Zhu, Zhiyuan Liu, Weidong Liu, and Wanxiang Che. 2024. Onebit: Towards extremely low-bit large language models. Advances in Neural Information Processing Systems (NeurIPS), pages 66357– 66382. [find](https://scholar.google.com/scholar?q=Yuzhuang+Xu%2C+Xu+Han%2C+Zonghan+Yang%2C+Shuo+Wang%2C+Qingfu+Zhu%2C+Zhiyuan+Liu%2C+Weidong+Liu%2C+and+Wanxiang+Che%2E+2024%2E+Onebit%3A+Towards+extremely+low%2Dbit+large+language+models%2E+Advances+in+Neural+Information+Proc)
15. Yuzhuang Xu, Xu Han, Yuanchi Zhang, Yixuan Wang, Yijun Liu, Shiyu Ji, Qingfu Zhu, and Wanxiang Che. 2026. CAMERA: Multi-matrix joint compression for MoE models via micro-expert redundancy analysis. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pages 27395–27404. [find](https://scholar.google.com/scholar?q=Yuzhuang+Xu%2C+Xu+Han%2C+Yuanchi+Zhang%2C+Yixuan+Wang%2C+Yijun+Liu%2C+Shiyu+Ji%2C+Qingfu+Zhu%2C+and+Wanxiang+Che%2E+2026%2E+CAMERA%3A+Multi%2Dmatrix+joint+compression+for+MoE+models+via+micro%2Dexpert+redundancy+analysis%2E+In)

*Updated 1 November 2026*