---
title: "MagicPIG: LSH Sampling for Efficient LLM Generation (2024)"
description: "They propose MagicPIG, which replaces selection with sampling. 79 references, cited by 64 papers here."
authors: ["Zhuoming Chen", "Ranajoy Sadhukhan", "Zihao Ye", "Yang Zhou", "Jianyu Zhang", "Niklas Nolte", "Yuandong Tian", "Matthijs Douze", "Leon Bottou", "Zhihao Jia", "Beidi Chen"]
year: 2024
venue: "arXiv:2410.16179"
source: "https://arxiv.org/abs/2410.16179"
doi: "10.48550/arxiv.2410.16179"
arxiv: "2410.16179"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chen-2024-magicpig-lsh-sampling-efficient-llm/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chen-2024-magicpig-lsh-sampling-efficient-llm/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / MagicPIG: LSH Sampling for Efficient LLM Generation

# MagicPIG: LSH Sampling for Efficient LLM Generation

**Authors:** Zhuoming Chen, Ranajoy Sadhukhan, Zihao Ye, Yang Zhou, Jianyu Zhang, Niklas Nolte, [Yuandong Tian](https://md.research.elamur.ai/free/en/authors/yuandong-tian/), [Matthijs Douze](https://md.research.elamur.ai/free/en/authors/matthijs-douze/), [Leon Bottou](https://md.research.elamur.ai/free/en/authors/leon-bottou/), Zhihao Jia and [Beidi Chen](https://md.research.elamur.ai/free/en/authors/beidi-chen/) · **Year:** 2024 · **Venue:** arXiv:2410.16179 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2410.16179) · [PDF](https://arxiv.org/pdf/2410.16179) · [DOI 10.48550/arxiv.2410.16179](https://doi.org/10.48550/arxiv.2410.16179) · [arXiv 2410.16179](https://arxiv.org/abs/2410.16179) · [OpenAlex](https://openalex.org/W4404312786)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [64](https://md.research.elamur.ai/free/en/papers/chen-2024-magicpig-lsh-sampling-efficient-llm/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Serving long-context LLMs is constrained by the KV cache. It consumes GPU memory and makes decoding memory-bound. Many speedups keep only the highest-scoring keys (TopK attention). The authors show this is a biased estimator that breaks down when attention is spread out, as in aggregation tasks like common- or frequent-word extraction. They propose MagicPIG, which replaces selection with sampling. SimHash-based locality sensitive hashing approximates sampling in proportion to attention scores, and the sampled terms are reweighted to estimate the attention output. The system splits work across devices: hash codes are computed on the GPU, while hash tables, the KV cache and the reduced attention computation live on the CPU. This frees GPU memory for larger batches and longer contexts. On RULER, MagicPIG stays close to full attention while touching only a few percent of the keys, and it beats Quest. Decoding throughput rises by 1.5–5× across A100, L20 and RTX 4090 setups. The work shows that estimation-based attention can outperform even exact TopK attention.

## Key points
- **Attention is less sparse than assumed.** In some layers, the top 20% of keys capture only about 70% of the attention mass. Sampling keys by their true attention weights ("oracle sampling") cuts estimation error by up to 4× compared with TopK.
- **RULER results with Llama-3.1-8B-Instruct (16K–96K context):**
  - Full attention averages 88.7.
  - MagicPIG scores 86.4–87.6 at a total cost of 0.02–0.05.
  - Quest scores 80.2–82.1 at a cost of about 0.10.
  - The same pattern holds for MegaBeam-Mistral-7B-512K and Llama3-8B-Prolong-512K.
- **Throughput gains:**
  - 1.5× on an A100 serving a 34B model at 16K context.
  - 5.0× on an L20 serving a 13B model at 16K context.
  - Up to 3.3× on a simulated RTX 4090 serving an 8B model at 96K context, with 54 ms single-request latency.
  - Offloading allows batches more than 12× larger than GPU full attention.
- **Implementation:** the GPU side is written in PyTorch and the CPU side in FBGEMM, using bfloat16. LSH is configured by (K, L): K hash bits per table and L tables.
- **Ablations:**
  - Without key centering, fewer than 0.1% of keys are retrieved. NIAH accuracy drops to near zero and FWE falls to 65%.
  - MagicPIG beats exact TopK on CWE by up to 3% and on FWE by up to 8%.

## Interesting ideas
- **Estimate instead of select.** Treat approximate attention as a sampling problem with bias correction rather than a search for the largest scores. Any pipeline that truncates a weighted sum to its top entries could gain from importance sampling with reweighting.
- **Center vectors before hashing.** Shifting keys before hashing fixes a geometric mismatch: queries and keys sit in nearly opposite narrow cones. This check is worth running in any cosine-based retrieval setup where the query and data distributions differ.
- **Handle the attention sink separately.** The sink key is nearly static and points away from the other keys, so it creates the impression of sparsity. Inspecting distributions with the sink excluded gives a more honest picture of how concentrated attention really is.
- **Split hardware by workload type.** Cheap, compute-heavy hashing runs on the GPU, while large tables and memory-bound lookups go to the CPU. This pattern can relieve VRAM limits on modest hardware such as a single RTX 4090.
- **Evaluate on aggregation tasks, not just retrieval.** CWE and FWE exposed failures that NIAH-style needle tests hid. Benchmarks for efficient-inference methods should include tasks that require using the whole context.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 365 works · 158,048 citations
- [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 263 works · 180,581 citations
- [New York University](https://md.research.elamur.ai/free/en/universities/new-york-university/) — University, New York, United States · 136 works · 86,390 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,344 works

## Cite

Cite as: Zhuoming Chen, Ranajoy Sadhukhan, Zihao Ye, Yang Zhou, Jianyu Zhang, Niklas Nolte, Yuandong Tian, Matthijs Douze, Leon Bottou, Zhihao Jia and Beidi Chen. 2024. MagicPIG: LSH Sampling for Efficient LLM Generation. arXiv:2410.16179.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{chen2024magicpig,
  title = {{MagicPIG: LSH Sampling for Efficient LLM Generation}},
  author = {Zhuoming Chen, Ranajoy Sadhukhan, Zihao Ye, Yang Zhou, Jianyu Zhang, Niklas Nolte, Yuandong Tian, Matthijs Douze, Leon Bottou, Zhihao Jia and Beidi Chen},
  year = {2024},
  doi = {10.48550/arxiv.2410.16179},
  eprint = {2410.16179},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2410.16179}
}
```

- [← previous Knowledge of Knowledge: Exploring Known-Unknowns Uncertainty with Large Language Models](https://md.research.elamur.ai/free/en/papers/amayuelas-2024-knowledge-knowledge-exploring-known-unknowns/)
- [next → Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescue](https://md.research.elamur.ai/free/en/papers/gu-2024-model-editing-harms-general-abilities/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

79 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774, 2023.
2. Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant Nair, Ilya Soloveychik, and Purushotham Kamath. Keyformer: Kv cache reduction through key tokens selection for efficient generative inference. Proceedings of Machine Learning and Systems, 6:114–127, 2024. [find](https://scholar.google.com/scholar?q=Muhammad+Adnan%2C+Akhil+Arunkumar%2C+Gaurav+Jain%2C+Prashant+Nair%2C+Ilya+Soloveychik%2C+and+Purushotham+Kamath%2E+Keyformer%3A+Kv+cache+reduction+through+key+tokens+selection+for+efficient+generative+inference%2E+Pr)
3. AI@Meta. Llama 3 model card. 2024. [https://github.com/meta-llama/llama3/blob/main/MODEL](https://github.com/meta-llama/llama3/blob/main/MODEL) CARD.md. [↗](https://github.com/meta-llama/llama3/blob/main/MODEL)
4. Josh Alman and Zhao Song. Fast attention requires bounded entries. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 63117–63135. Curran Associates, Inc., 2023. [https://proceedings.neurips.cc/paper](https://proceedings.neurips.cc/paper) fil… [↗](https://proceedings.neurips.cc/paper)
5. Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, et al. Deepspeed inference: Enabling efficient inference of transformer models at unprecedented scale. arXiv preprint arXiv:2207.00032, 2022. [find](https://scholar.google.com/scholar?q=Reza+Yazdani+Aminabadi%2C+Samyam+Rajbhandari%2C+Minjia+Zhang%2C+Ammar+Ahmad+Awan%2C+Cheng+Li%2C+Du+Li%2C+Elton+Zheng%2C+Jeff+Rasley%2C+Shaden+Smith%2C+Olatunji+Ruwase%2C+et+al%2E+Deepspeed+inference%3A+Enabling+efficient+inf)
6. Alexandr Andoni and Ilya Razenshteyn. Optimal data-dependent hashing for approximate near neighbors. In Proceedings of the forty-seventh annual ACM symposium on Theory of computing, pages 793–801, 2015. [find](https://scholar.google.com/scholar?q=Alexandr+Andoni+and+Ilya+Razenshteyn%2E+Optimal+data%2Ddependent+hashing+for+approximate+near+neighbors%2E+In+Proceedings+of+the+forty%2Dseventh+annual+ACM+symposium+on+Theory+of+computing%2C+pages+793%E2%80%93801%2C+201)
7. Alexandr Andoni, Piotr Indyk, Thijs Laarhoven, Ilya Razenshteyn, and Ludwig Schmidt. Practical and optimal LSH for angular distance. In Proceedings of the 28th International Conference on Neural Information Processing Systems-Volume 1, pages 1225–1233, 2015. [find](https://scholar.google.com/scholar?q=Alexandr+Andoni%2C+Piotr+Indyk%2C+Thijs+Laarhoven%2C+Ilya+Razenshteyn%2C+and+Ludwig+Schmidt%2E+Practical+and+optimal+LSH+for+angular+distance%2E+In+Proceedings+of+the+28th+International+Conference+on+Neural+Infor)
8. Arturs Backurs, Moses Charikar, Piotr Indyk, and Paris Siminelakis. Efficient density evaluation for smooth kernels. In 2018 IEEE 59th Annual Symposium on Foundations of Computer Science (FOCS), pages 615–626, 2018. doi: 10.1109/FOCS.2018.00065. — [doi:10.1109/focs.2018.00065](https://doi.org/10.1109/focs.2018.00065) [find](https://scholar.google.com/scholar?q=Arturs+Backurs%2C+Moses+Charikar%2C+Piotr+Indyk%2C+and+Paris+Siminelakis%2E+Efficient+density+evaluation+for+smooth+kernels%2E+In+2018+IEEE+59th+Annual+Symposium+on+Foundations+of+Computer+Science+%28FOCS%29%2C+pages)
9. Arturs Backurs, Piotr Indyk, and Tal Wagner. Space and time efficient kernel density estimation in high dimensions. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2… [find](https://scholar.google.com/scholar?q=Arturs+Backurs%2C+Piotr+Indyk%2C+and+Tal+Wagner%2E+Space+and+time+efficient+kernel+density+estimation+in+high+dimensions%2E+In+H%2E+Wallach%2C+H%2E+Larochelle%2C+A%2E+Beygelzimer%2C+F%2E+d%27Alche%CC%81%2DBuc%2C+E%2E+Fox%2C+and+R%2E+Garnet)
10. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. [Longbench: A bilingual, multitask benchmark for long context understanding](https://md.research.elamur.ai/free/en/papers/bai-2023-longbench-bilingual-multitask-benchmark/). arXiv preprint arXiv:2308.14508, 2023.
11. Jan van den Brand, Zhao Song, and Tianyi Zhou. Algorithm and hardness for dynamic attention maintenance in large language models. arXiv preprint arXiv:2304.02207, 2023. — [arXiv:2304.02207](https://arxiv.org/abs/2304.02207) [find](https://scholar.google.com/scholar?q=Jan+van+den+Brand%2C+Zhao+Song%2C+and+Tianyi+Zhou%2E+Algorithm+and+hardness+for+dynamic+attention+maintenance+in+large+language+models%2E+arXiv+preprint+arXiv%3A2304%2E02207%2C+2023%2E+%E2%80%94+%5BarXiv%3A2304%2E02207%5D%28https%3A%2F%2Far)
12. Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis, 2019. https://arxiv.org/abs/1809.11096. — [arXiv:1809.11096](https://arxiv.org/abs/1809.11096) [↗](https://arxiv.org/abs/1809.11096)
13. Moses S. Charikar. Similarity estimation techniques from rounding algorithms. In Proceedings of the Thiry-Fourth Annual ACM Symposium on Theory of Computing, STOC ’02, page 380–388, New York, NY, USA, 2002. Association for Computing Machinery. ISBN 1581134959. doi: 10.1145/509907.509965. https://doi… [find](https://scholar.google.com/scholar?q=Moses+S%2E+Charikar%2E+Similarity+estimation+techniques+from+rounding+algorithms%2E+In+Proceedings+of+the+Thiry%2DFourth+Annual+ACM+Symposium+on+Theory+of+Computing%2C+STOC+%E2%80%9902%2C+page+380%E2%80%93388%2C+New+York%2C+NY%2C+USA%2C)
14. Beidi Chen, Anshumali Shrivastava, and Rebecca C Steorts. Unique entity estimation with application to the syrian conflict. The Annals of Applied Statistics, 12(2):1039–1067, 2018. [find](https://scholar.google.com/scholar?q=Beidi+Chen%2C+Anshumali+Shrivastava%2C+and+Rebecca+C+Steorts%2E+Unique+entity+estimation+with+application+to+the+syrian+conflict%2E+The+Annals+of+Applied+Statistics%2C+12%282%29%3A1039%E2%80%931067%2C+2018%2E)
15. Beidi Chen, Yingchen Xu, and Anshumali Shrivastava. Fast and accurate stochastic gradient estimation. Advances in Neural Information Processing Systems, 32, 2019. [find](https://scholar.google.com/scholar?q=Beidi+Chen%2C+Yingchen+Xu%2C+and+Anshumali+Shrivastava%2E+Fast+and+accurate+stochastic+gradient+estimation%2E+Advances+in+Neural+Information+Processing+Systems%2C+32%2C+2019%2E)
16. Beidi Chen, Tharun Medini, James Farwell, sameh gobriel, Charlie Tai, and Anshumali Shrivastava. Slide : In defense of smart algorithms over hardware acceleration for large-scale deep learning systems. In I. Dhillon, D. Papailiopoulos, and V. Sze, editors, Proceedings of Machine Learning and Systems… [find](https://scholar.google.com/scholar?q=Beidi+Chen%2C+Tharun+Medini%2C+James+Farwell%2C+sameh+gobriel%2C+Charlie+Tai%2C+and+Anshumali+Shrivastava%2E+Slide+%3A+In+defense+of+smart+algorithms+over+hardware+acceleration+for+large%2Dscale+deep+learning+systems)
17. Beidi Chen, Tharun Medini, James Farwell, Charlie Tai, Anshumali Shrivastava, et al. SLIDE: In defense of smart algorithms over hardware acceleration for large-scale deep learning systems. Proceedings of Machine Learning and Systems, 2:291–306, 2020b. [find](https://scholar.google.com/scholar?q=Beidi+Chen%2C+Tharun+Medini%2C+James+Farwell%2C+Charlie+Tai%2C+Anshumali+Shrivastava%2C+et+al%2E+SLIDE%3A+In+defense+of+smart+algorithms+over+hardware+acceleration+for+large%2Dscale+deep+learning+systems%2E+Proceedings)
18. Beidi Chen, Tri Dao, Eric Winsor, Zhao Song, Atri Rudra, and Christopher Ré. Scatterbrain: Unifying sparse and low-rank attention. Advances in Neural Information Processing Systems, 34:17413–17426, 2021. [find](https://scholar.google.com/scholar?q=Beidi+Chen%2C+Tri+Dao%2C+Eric+Winsor%2C+Zhao+Song%2C+Atri+Rudra%2C+and+Christopher+Re%CC%81%2E+Scatterbrain%3A+Unifying+sparse+and+low%2Drank+attention%2E+Advances+in+Neural+Information+Processing+Systems%2C+34%3A17413%E2%80%9317426%2C+2)
19. Zhuoming Chen, Avner May, Ruslan Svirschevski, Yuhsun Huang, Max Ryabinin, Zhihao Jia, and Beidi Chen. Sequoia: Scalable, robust, and hardware-aware speculative decoding. arXiv preprint arXiv:2402.12374, 2024. [find](https://scholar.google.com/scholar?q=Zhuoming+Chen%2C+Avner+May%2C+Ruslan+Svirschevski%2C+Yuhsun+Huang%2C+Max+Ryabinin%2C+Zhihao+Jia%2C+and+Beidi+Chen%2E+Sequoia%3A+Scalable%2C+robust%2C+and+hardware%2Daware+speculative+decoding%2E+arXiv+preprint+arXiv%3A2402%2E123)
20. Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476, 2024. [find](https://scholar.google.com/scholar?q=Zesen+Cheng%2C+Sicong+Leng%2C+Hang+Zhang%2C+Yifei+Xin%2C+Xin+Li%2C+Guanzheng+Chen%2C+Yongxin+Zhu%2C+Wenqi+Zhang%2C+Ziyang+Luo%2C+Deli+Zhao%2C+et+al%2E+Videollama+2%3A+Advancing+spatial%2Dtemporal+modeling+and+audio+understandi)
21. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating llms by human preference. arXiv preprint arXiv:2403.04132, 2024. [find](https://scholar.google.com/scholar?q=Wei%2DLin+Chiang%2C+Lianmin+Zheng%2C+Ying+Sheng%2C+Anastasios+Nikolas+Angelopoulos%2C+Tianle+Li%2C+Dacheng+Li%2C+Hao+Zhang%2C+Banghua+Zhu%2C+Michael+Jordan%2C+Joseph+E+Gonzalez%2C+et+al%2E+Chatbot+arena%3A+An+open+platform+for)
22. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/), 2021.
23. Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. CoRR, abs/2307.08691, 2023. doi: 10.48550/ARXIV.2307.08691. https://doi.org/10.48550/arXiv.2307.08691. [↗](https://doi.org/10.48550/arXiv.2307.08691)
24. Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344–16359, 2022a. [find](https://scholar.google.com/scholar?q=Tri+Dao%2C+Dan+Fu%2C+Stefano+Ermon%2C+Atri+Rudra%2C+and+Christopher+Re%CC%81%2E+Flashattention%3A+Fast+and+memory%2Defficient+exact+attention+with+io%2Dawareness%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A1634)
25. Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Con… [find](https://scholar.google.com/scholar?q=Tri+Dao%2C+Daniel+Y%2E+Fu%2C+Stefano+Ermon%2C+Atri+Rudra%2C+and+Christopher+Re%CC%81%2E+Flashattention%3A+Fast+and+memory%2Defficient+exact+attention+with+io%2Dawareness%2E+In+Sanmi+Koyejo%2C+S%2E+Mohamed%2C+A%2E+Agarwal%2C+Danielle+Be)
26. Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. [A dataset of information-seeking questions and answers anchored in research papers](https://md.research.elamur.ai/free/en/papers/dasigi-2021-dataset-information-seeking-questions-answers/). arXiv preprint arXiv:2105.03011, 2021.
27. Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. [The faiss library](https://md.research.elamur.ai/free/en/papers/douze-2024-faiss-library/). arXiv preprint arXiv:2401.08281, 2024.
28. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. [find](https://scholar.google.com/scholar?q=Abhimanyu+Dubey%2C+Abhinav+Jauhri%2C+Abhinav+Pandey%2C+Abhishek+Kadian%2C+Ahmad+Al%2DDahle%2C+Aiesha+Letman%2C+Akhil+Mathur%2C+Alan+Schelten%2C+Amy+Yang%2C+Angela+Fan%2C+et+al%2E+The+llama+3+herd+of+models%2E+arXiv+preprint+ar)
29. Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, September… — [doi:10.5281/zenodo.5371628](https://doi.org/10.5281/zenodo.5371628) [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Jonathan+Tow%2C+Stella+Biderman%2C+Sid+Black%2C+Anthony+DiPofi%2C+Charles+Foster%2C+Laurence+Golding%2C+Jeffrey+Hsu%2C+Kyle+McDonell%2C+Niklas+Muennighoff%2C+Jason+Phang%2C+Laria+Reynolds%2C+Eric+Tang%2C+Anish+Thite)
30. Tianyu Gao, Alexander Wettig, Howard Yen, and Danqi Chen. [How to train long-context language models (effectively)](https://md.research.elamur.ai/free/en/papers/gao-2025-how-train-long-context-language/). arXiv preprint arXiv:2410.02660, 2024.
31. Jiaao He and Jidong Zhai. Fastdecode: High-throughput gpu-efficient llm serving using heterogeneous pipelines. arXiv preprint arXiv:2403.11421, 2024. [find](https://scholar.google.com/scholar?q=Jiaao+He+and+Jidong+Zhai%2E+Fastdecode%3A+High%2Dthroughput+gpu%2Defficient+llm+serving+using+heterogeneous+pipelines%2E+arXiv+preprint+arXiv%3A2403%2E11421%2C+2024%2E)
32. Pujiang He, Shan Zhou, Wenhuan Huang, Changqing Li, Duyi Wang, Bin Guo, Chen Meng, Sheng Gui, Weifei Yu, and Yi Xie. Inference performance optimization for large language models on cpus, 2024. https://arxiv.org/abs/2407.07304. [↗](https://arxiv.org/abs/2407.07304)
33. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). arXiv preprint arXiv:2009.03300, 2020.
34. Timothy Hesterberg. Advances in importance sampling. 01 2003. [find](https://scholar.google.com/scholar?q=Timothy+Hesterberg%2E+Advances+in+importance+sampling%2E+01+2003%2E)
35. Ke Hong, Guohao Dai, Jiaming Xu, Qiuli Mao, Xiuhong Li, Jun Liu, Kangdi Chen, Yuhan Dong, and Yu Wang. Flashdecoding++: Faster large language model inference on gpus, 2024. https://arxiv.org/abs/2311.01282. [↗](https://arxiv.org/abs/2311.01282)
36. Eduard Hovy, Laurie Gerber, Ulf Hermjakob, Chin-Yew Lin, and Deepak Ravichandran. [Toward semantics-based answer pinpointing](https://md.research.elamur.ai/free/en/papers/hovy-2001-semantics-based-answer-pinpointing/). In Proceedings of the First International Conference on Human Language Technology Research, 2001. [https://www.aclweb.org/anthology/H01-1069](https://www.aclweb.org/anthology/H01-1069). [↗](https://www.aclweb.org/anthology/H01-1069)
37. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.[06654, 2024](https://md.research.elamur.ai/free/en/papers/hsieh-2024-ruler-s-real-context-size/).
38. Omid Jafari, Preeti Maurya, Parth Nagarkar, Khandker Mushfiqul Islam, and Chidambaram Crushev. A survey on locality sensitive hashing algorithms and their applications. arXiv preprint arXiv:2102.08942, 2021. — [arXiv:2102.08942](https://arxiv.org/abs/2102.08942) [find](https://scholar.google.com/scholar?q=Omid+Jafari%2C+Preeti+Maurya%2C+Parth+Nagarkar%2C+Khandker+Mushfiqul+Islam%2C+and+Chidambaram+Crushev%2E+A+survey+on+locality+sensitive+hashing+algorithms+and+their+applications%2E+arXiv+preprint+arXiv%3A2102%2E08942)
39. Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. [Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/), 2017. https://arxiv.org/abs/1705.03551. [↗](https://arxiv.org/abs/1705.03551)
40. Daya Khudia, Jianyu Huang, Protonu Basu, Summer Deng, Haixin Liu, Jongsoo Park, and Mikhail Smelyanskiy. Fbgemm: Enabling high-performance low-precision deep learning inference. arXiv preprint arXiv:2101.05615, 2021. — [arXiv:2101.05615](https://arxiv.org/abs/2101.05615) [find](https://scholar.google.com/scholar?q=Daya+Khudia%2C+Jianyu+Huang%2C+Protonu+Basu%2C+Summer+Deng%2C+Haixin+Liu%2C+Jongsoo+Park%2C+and+Mikhail+Smelyanskiy%2E+Fbgemm%3A+Enabling+high%2Dperformance+low%2Dprecision+deep+learning+inference%2E+arXiv+preprint+arXiv%3A2)
41. Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. [Reformer: The efficient transformer](https://md.research.elamur.ai/free/en/papers/kitaev-2020-reformer-efficient-transformer/). arXiv preprint arXiv:2001.04451, 2020.
42. Teun Kloek and Herman K Van Dijk. Bayesian estimates of equation system parameters: an application of integration by monte carlo. Econometrica: Journal of the Econometric Society, pages 1–19, 1978. [find](https://scholar.google.com/scholar?q=Teun+Kloek+and+Herman+K+Van+Dijk%2E+Bayesian+estimates+of+equation+system+parameters%3A+an+application+of+integration+by+monte+carlo%2E+Econometrica%3A+Journal+of+the+Econometric+Society%2C+pages+1%E2%80%9319%2C+1978%2E)
43. Xin Li and Dan Roth. [Learning question classifiers](https://md.research.elamur.ai/free/en/papers/li-2002-learning-question-classifiers/). In COLING 2002: The 19th International Conference on Computational Linguistics, 2002. [https://www.aclweb.org/anthology/C02-1150](https://www.aclweb.org/anthology/C02-1150). [↗](https://www.aclweb.org/anthology/C02-1150)
44. Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. [Snapkv: Llm knows what you are looking for before generation](https://md.research.elamur.ai/free/en/papers/li-2024-snapkv-llm-knows-looking-before/). arXiv preprint arXiv:2404.14469, 2024.
45. Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. Qserve: W4a8kv4 quantization and system co-design for efficient llm serving. arXiv preprint arXiv:2405.04532, 2024. [find](https://scholar.google.com/scholar?q=Yujun+Lin%2C+Haotian+Tang%2C+Shang+Yang%2C+Zhekai+Zhang%2C+Guangxuan+Xiao%2C+Chuang+Gan%2C+and+Song+Han%2E+Qserve%3A+W4a8kv4+quantization+and+system+co%2Ddesign+for+efficient+llm+serving%2E+arXiv+preprint+arXiv%3A2405%2E0453)
46. Di Liu, Meng Chen, Baotong Lu, Huiqiang Jiang, Zhenhua Han, Qianxi Zhang, Qi Chen, Chengruidong Zhang, Bailu Ding, Kai Zhang, et al. Retrievalattention: Accelerating long-context llm inference via vector retrieval. arXiv preprint arXiv:2409.10516, 2024a. [find](https://scholar.google.com/scholar?q=Di+Liu%2C+Meng+Chen%2C+Baotong+Lu%2C+Huiqiang+Jiang%2C+Zhenhua+Han%2C+Qianxi+Zhang%2C+Qi+Chen%2C+Chengruidong+Zhang%2C+Bailu+Ding%2C+Kai+Zhang%2C+et+al%2E+Retrievalattention%3A+Accelerating+long%2Dcontext+llm+inference+via+vec)
47. Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmarking repository-level code auto-completion systems, 2023. https://arxiv.org/abs/2306.03091. [↗](https://arxiv.org/abs/2306.03091)
48. Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. [Kivi: A tuning-free asymmetric 2bit quantization for kv cache](https://md.research.elamur.ai/free/en/papers/liu-2024-kivi-tuning-free-asymmetric-2bit/). arXiv preprint arXiv:2402.02750, 2024b.
49. Sharon L Lohr. Sampling: design and analysis. Chapman and Hall/CRC, 2021. [find](https://scholar.google.com/scholar?q=Sharon+L+Lohr%2E+Sampling%3A+design+and+analysis%2E+Chapman+and+Hall%2FCRC%2C+2021%2E)
50. Paul Lukacs. Closed population capture-recapture models. Program MARK: a gentle introduction, 8, 2009. [find](https://scholar.google.com/scholar?q=Paul+Lukacs%2E+Closed+population+capture%2Drecapture+models%2E+Program+MARK%3A+a+gentle+introduction%2C+8%2C+2009%2E)
51. Qin Lv, William Josephson, Zhe Wang, Moses Charikar, and Kai Li. Intelligent probing for locality sensitive hashing: multi-probe lsh and beyond. Proc. VLDB Endow., 10(12):2021–2024, August 2017. ISSN 2150-8097. doi: 10.14778/ 3137765.3137836. https://doi.org/10.14778/3137765.3137836. — [doi:10.14778/3137765.3137836](https://doi.org/10.14778/3137765.3137836) [↗](https://doi.org/10.14778/3137765.3137836)
52. Yuzhen Mao, Martin Ester, and Ke Li. Iceformer: Accelerated inference with long-sequence transformers on cpus. arXiv preprint arXiv:2405.02842, 2024. — [arXiv:2405.02842](https://arxiv.org/abs/2405.02842) [find](https://scholar.google.com/scholar?q=Yuzhen+Mao%2C+Martin+Ester%2C+and+Ke+Li%2E+Iceformer%3A+Accelerated+inference+with+long%2Dsequence+transformers+on+cpus%2E+arXiv+preprint+arXiv%3A2405%2E02842%2C+2024%2E+%E2%80%94+%5BarXiv%3A2405%2E02842%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2405%2E02)
53. Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Rae Ying Yee Wong, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. Specinfer: Accelerating generative llm serving with speculative inference and token tree verification. arXiv preprint arXiv:2305.09781, 2023. [find](https://scholar.google.com/scholar?q=Xupeng+Miao%2C+Gabriele+Oliaro%2C+Zhihao+Zhang%2C+Xinhao+Cheng%2C+Zeyu+Wang%2C+Rae+Ying+Yee+Wong%2C+Zhuoming+Chen%2C+Daiyaan+Arfeen%2C+Reyna+Abhyankar%2C+and+Zhihao+Jia%2E+Specinfer%3A+Accelerating+generative+llm+serving+w)
54. Toan Nguyen Mau and Yasushi Inoguchi. Locality-sensitive hashing for information retrieval system on multiple gpgpu devices. Applied Sciences, 10(7), 2020. ISSN 2076-3417. doi: 10.3390/app10072539. [https://www.mdpi.com/](https://www.mdpi.com/) 2076-3417/10/7/2539. — [doi:10.3390/app10072539](https://doi.org/10.3390/app10072539) [↗](https://www.mdpi.com/)
55. Art B. Owen. Monte Carlo theory, methods and examples. [https://artowen.su.domains/mc/](https://artowen.su.domains/mc/), 2013. [↗](https://artowen.su.domains/mc/)
56. Zaifeng Pan, Feng Zhang, Hourun Li, Chenyang Zhang, Xiaoyong Du, and Dong Deng. G-slide: A gpu-based sub-linear deep learning engine via lsh sparsification. IEEE Transactions on Parallel and Distributed Systems, 33(11):3015–3027, 2022. doi: 10.1109/TPDS.2021.3132493. — [doi:10.1109/tpds.2021.3132493](https://doi.org/10.1109/tpds.2021.3132493) [find](https://scholar.google.com/scholar?q=Zaifeng+Pan%2C+Feng+Zhang%2C+Hourun+Li%2C+Chenyang+Zhang%2C+Xiaoyong+Du%2C+and+Dong+Deng%2E+G%2Dslide%3A+A+gpu%2Dbased+sub%2Dlinear+deep+learning+engine+via+lsh+sparsification%2E+IEEE+Transactions+on+Parallel+and+Distribut)
57. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. [Pytorch: An imperative style, high-performance deep learning library](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/). Advances in neural information processing systems, 32, 2019.
58. Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. arXiv preprint arXiv:2211.05102, 2022. [find](https://scholar.google.com/scholar?q=Reiner+Pope%2C+Sholto+Douglas%2C+Aakanksha+Chowdhery%2C+Jacob+Devlin%2C+James+Bradbury%2C+Anselm+Levskaya%2C+Jonathan+Heek%2C+Kefan+Xiao%2C+Shivani+Agrawal%2C+and+Jeff+Dean%2E+Efficiently+scaling+transformer+inference%2E+a)
59. Siva Reddy, Danqi Chen, and Christopher D Manning. [Coqa: A conversational question answering challenge](https://md.research.elamur.ai/free/en/papers/reddy-2019-coqa-conversational-question-answering-challenge/). Transactions of the Association for Computational Linguistics, 7:249–266, 2019.
60. Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défosse… — [Code Llama: Open Foundation Models for Code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/)
61. Caitlin Sadowski. Simhash : Hash-based similarity detection. 2007. [https://api.semanticscholar.org/CorpusID:](https://api.semanticscholar.org/CorpusID:) 199497165. [↗](https://api.semanticscholar.org/CorpusID:)
62. Anshumali Shrivastava and Ping Li. Asymmetric lsh (alsh) for sublinear time maximum inner product search (mips). In Advances in Neural Information Processing Systems (NeurIPS), pages 2321–2329, 2014. [find](https://scholar.google.com/scholar?q=Anshumali+Shrivastava+and+Ping+Li%2E+Asymmetric+lsh+%28alsh%29+for+sublinear+time+maximum+inner+product+search+%28mips%29%2E+In+Advances+in+Neural+Information+Processing+Systems+%28NeurIPS%29%2C+pages+2321%E2%80%932329%2C+2014%2E)
63. Prajwal Singhania, Siddharth Singh, Shwai He, Soheil Feizi, and Abhinav Bhatele. Loki: Low-rank keys for efficient sparse attention. arXiv preprint arXiv:2406.02542, 2024. — [arXiv:2406.02542](https://arxiv.org/abs/2406.02542) [find](https://scholar.google.com/scholar?q=Prajwal+Singhania%2C+Siddharth+Singh%2C+Shwai+He%2C+Soheil+Feizi%2C+and+Abhinav+Bhatele%2E+Loki%3A+Low%2Drank+keys+for+efficient+sparse+attention%2E+arXiv+preprint+arXiv%3A2406%2E02542%2C+2024%2E+%E2%80%94+%5BarXiv%3A2406%2E02542%5D%28https%3A%2F)
64. Malcolm Slaney, Yury Lifshits, and Junfeng He. Optimal parameters for locality-sensitive hashing. Proceedings of the IEEE, 100(9):2604–2623, 2012. doi: 10.1109/JPROC.2012.2193849. — [doi:10.1109/jproc.2012.2193849](https://doi.org/10.1109/jproc.2012.2193849) [find](https://scholar.google.com/scholar?q=Malcolm+Slaney%2C+Yury+Lifshits%2C+and+Junfeng+He%2E+Optimal+parameters+for+locality%2Dsensitive+hashing%2E+Proceedings+of+the+IEEE%2C+100%289%29%3A2604%E2%80%932623%2C+2012%2E+doi%3A+10%2E1109%2FJPROC%2E2012%2E2193849%2E+%E2%80%94+%5Bdoi%3A10%2E1109%2Fjproc)
65. Ryan Spring and Anshumali Shrivastava. A new unbiased and efficient class of lsh-based samplers and estimators for partition function computation in log-linear models. arXiv preprint arXiv:1703.05160, 2017. — [arXiv:1703.05160](https://arxiv.org/abs/1703.05160) [find](https://scholar.google.com/scholar?q=Ryan+Spring+and+Anshumali+Shrivastava%2E+A+new+unbiased+and+efficient+class+of+lsh%2Dbased+samplers+and+estimators+for+partition+function+computation+in+log%2Dlinear+models%2E+arXiv+preprint+arXiv%3A1703%2E05160%2C)
66. Hanshi Sun, Zhuoming Chen, Xinyu Yang, Yuandong Tian, and Beidi Chen. Triforce: Lossless acceleration of long sequence generation with hierarchical speculative decoding. arXiv preprint arXiv:2404.11912, 2024. [find](https://scholar.google.com/scholar?q=Hanshi+Sun%2C+Zhuoming+Chen%2C+Xinyu+Yang%2C+Yuandong+Tian%2C+and+Beidi+Chen%2E+Triforce%3A+Lossless+acceleration+of+long+sequence+generation+with+hierarchical+speculative+decoding%2E+arXiv+preprint+arXiv%3A2404%2E1191)
67. Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. [Quest: Query-aware sparsity for efficient long-context llm inference](https://md.research.elamur.ai/free/en/papers/tang-2024-quest-query-aware-sparsity-efficient/). arXiv preprint arXiv:2406.10774, 2024.
68. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805, 2023.
69. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+Dan+Bikel%2C+Lukas+Blecher%2C+Cristian+Canton+Fe)
70. Minzheng Wang, Longze Chen, Cheng Fu, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, et al. [Leave no document behind: Benchmarking long-context llms with extended multi-doc qa](https://md.research.elamur.ai/free/en/papers/wang-2024-leave-no-document-behind-benchmarking/). arXiv preprint arXiv:2406.17419, 2024.
71. Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. [Retrieval head mechanistically explains long-context factuality](https://md.research.elamur.ai/free/en/papers/wu-2024-retrieval-head-mechanistically-explains-long/). arXiv preprint arXiv:2404.15574, 2024.
72. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453, 2023. [find](https://scholar.google.com/scholar?q=Guangxuan+Xiao%2C+Yuandong+Tian%2C+Beidi+Chen%2C+Song+Han%2C+and+Mike+Lewis%2E+Efficient+streaming+language+models+with+attention+sinks%2E+arXiv+preprint+arXiv%3A2309%2E17453%2C+2023%2E)
73. Zihao Ye, Ruihang Lai, Bo-Ru Lu, Chien-Yu Lin, Size Zheng, Lequn Chen, Tianqi Chen, and Luis Ceze. Cascade inference: Memory bandwidth efficient shared prefix batch decoding, February 2024. [https://flashinfer.ai/2024/02/](https://flashinfer.ai/2024/02/) 02/cascade-inference.html. [↗](https://flashinfer.ai/2024/02/)
74. Amir Zandieh, Insu Han, Majid Daliri, and Amin Karbasi. Kdeformer: Accelerating transformers via kernel density estimation. In International Conference on Machine Learning, pages 40605–40623. PMLR, 2023. [find](https://scholar.google.com/scholar?q=Amir+Zandieh%2C+Insu+Han%2C+Majid+Daliri%2C+and+Amin+Karbasi%2E+Kdeformer%3A+Accelerating+transformers+via+kernel+density+estimation%2E+In+International+Conference+on+Machine+Learning%2C+pages+40605%E2%80%9340623%2E+PMLR%2C+20)
75. Hailin Zhang, Xiaodong Ji, Yilin Chen, Fangcheng Fu, Xupeng Miao, Xiaonan Nie, Weipeng Chen, and Bin Cui. Pqcache: Product quantization-based kvcache for long context llm inference. arXiv preprint arXiv:2407.12820, 2024. [find](https://scholar.google.com/scholar?q=Hailin+Zhang%2C+Xiaodong+Ji%2C+Yilin+Chen%2C+Fangcheng+Fu%2C+Xupeng+Miao%2C+Xiaonan+Nie%2C+Weipeng+Chen%2C+and+Bin+Cui%2E+Pqcache%3A+Product+quantization%2Dbased+kvcache+for+long+context+llm+inference%2E+arXiv+preprint+arX)
76. Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. [Draft & verify: Lossless large language model acceleration via self-speculative decoding](https://md.research.elamur.ai/free/en/papers/zhang-2024-draft-verify-lossless-large-language/). CoRR, abs/2309.08168, 2023a. doi: 10.48550/ARXIV.2309. 08168. https://doi.org/10.48550/arXiv.2309.08168. [↗](https://doi.org/10.48550/arXiv.2309.08168)
77. Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang ”Atlas” Wang, and Beidi Chen. H2o: Heavy-hitter oracle for efficient generative inference of large language models. In A. Oh, T. Naumann, A. Globerson,… [find](https://scholar.google.com/scholar?q=Zhenyu+Zhang%2C+Ying+Sheng%2C+Tianyi+Zhou%2C+Tianlong+Chen%2C+Lianmin+Zheng%2C+Ruisi+Cai%2C+Zhao+Song%2C+Yuandong+Tian%2C+Christopher+Re%CC%81%2C+Clark+Barrett%2C+Zhangyang+%E2%80%9DAtlas%E2%80%9D+Wang%2C+and+Beidi+Chen%2E+H2o%3A+Heavy%2Dhitter+orac)
78. Yang Zhou. Yangzhoumill/infini igsm 4k noise close. [https://huggingface.co/datasets/YangZhoumill/infini](https://huggingface.co/datasets/YangZhoumill/infini) igsm 4k noise close, 2024a. Accessed: 2024-10-20. [↗](https://huggingface.co/datasets/YangZhoumill/infini)
79. Yang Zhou. Yangzhoumill/infini igsm 8k noise close. [https://huggingface.co/datasets/YangZhoumill/infini](https://huggingface.co/datasets/YangZhoumill/infini) igsm 8k noise close, 2024b. Accessed: 2024-10-20. [↗](https://huggingface.co/datasets/YangZhoumill/infini)

*Updated 24 October 2026*