--- title: "GPU & parallel computing — papers" description: "GPU & parallel computing: 4 papers of the Elamur Research library, the most cited first. GPUs, parallel algorithms and the hardware that runs the models." canonical: "https://research.elamur.ai/free/en/topics/gpu-parallel-computing/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/topics/gpu-parallel-computing/) [home](https://md.research.elamur.ai/free/en/) / [topics](https://md.research.elamur.ai/free/en/topics/) / GPU & parallel computing # ⚡ GPU & parallel computing 4 papers · the most cited first · [all topics](https://md.research.elamur.ai/free/en/topics/) GPUs, parallel algorithms and the hardware that runs the models. A survey · about 4 minutes to read · every paper named here has its own page in this library **In this article** 1. [What it is](#a-what-it-is) 2. [History and milestones](#a-history-and-milestones) 3. [Landmark papers in this library](#a-landmark-papers-in-this-library) 4. [Common threads](#a-common-threads) 5. [People](#a-people) 6. [Institutions and places](#a-institutions-and-places) 7. [Open questions and neighbours](#a-open-questions-and-neighbours) Graphics processing units were designed to shade millions of pixels in parallel, and that same design, thousands of simple cores sharing fast but limited memory, turned out to suit the dense linear algebra at the heart of machine learning. Since NVIDIA's CUDA platform opened GPUs to general-purpose programming in 2007, much of applied machine learning has been shaped by what runs well on this hardware: which algorithms parallelise, how data moves between memory levels, and how far a model can be squeezed onto a single device. The papers under this topic are few but heavily cited. They show four views of the same problem, getting more useful work out of a GPU: a library for searching billions of vectors, an engine for running very large language models on one modest card, a benchmark asking whether language models can write fast GPU code themselves, and an attention mechanism designed from the start around the hardware. ## What it is The topic covers research where the GPU, or parallel execution more broadly, is part of the contribution rather than just the place where experiments ran. Typical concerns are memory hierarchy (registers, shared memory, GPU memory, CPU RAM, disk), arithmetic intensity, kernel fusion, quantisation for bandwidth, and scheduling of data movement. It overlaps with [Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/), which asks how to make models cheaper in general, and with [Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/), whose scale drives most of the recent hardware work. All the papers here stand on the [Computing & GPUs](https://md.research.elamur.ai/free/en/shelves/computing/) shelf; single papers also sit on [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/) and [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/). ## History and milestones The papers here span less than a decade, but they mark distinct stages. In 2017 the concern was classic retrieval at scale: nearest-neighbour search over image and text embeddings. By 2023 the bottleneck had become the language model itself, too large for a single accelerator. By 2025 two new directions appear: using language models to write GPU kernels, and redesigning attention so that its sparsity actually saves wall-clock time on modern hardware. Landmark papers on this topic: the 4 most cited, in time The 4 most cited papers on this topic span 2017–2025; the dot grows with the citations. **The numbers** | Paper | Citations | | --- | --- | | [2017 — Billion-scale similarity search with GPUs](https://md.research.elamur.ai/free/en/papers/johnson-2017-billion-scale-similarity-search-gpus/) | 4,687 | | [2023 — FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU](https://md.research.elamur.ai/free/en/papers/sheng-2023-flexgen-high-throughput-generative-inference/) | 399 | | [2025 — KernelBench: Can LLMs Write Efficient GPU Kernels?](https://md.research.elamur.ai/free/en/papers/ouyang-2025-kernelbench-llms-write-efficient-gpu/) | 170 | | [2025 — Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention](https://md.research.elamur.ai/free/en/papers/yuan-2025-native-sparse-attention-hardware/) | 135 | Papers on this topic by year 4 dated papers from 2017 to 2025; the busiest year is 2025 with 2, and 75% of them appeared in 2018 or later. **The numbers** | Year | Papers | | --- | --- | | 2025 | 2 | | 2023 | 1 | | 2017 | 1 | ## Landmark papers in this library **Vector search on GPUs.** [Billion-scale similarity search with GPUs](https://md.research.elamur.ai/free/en/papers/johnson-2017-billion-scale-similarity-search-gpus/) (2017), from Facebook AI Research, is by far the most cited work here. Earlier GPU search methods were limited by the step of picking the k smallest distances, which parallelises poorly. The authors wrote a k-selection routine that holds its working data in GPU registers and fuses with neighbouring kernels, and built exact and product-quantisation search on top of it. It beat previous GPU and CPU results and made nearest-neighbour graphs over a billion vectors feasible on a single machine with a few GPUs. Released as the Faiss library, it became a standard component of retrieval systems, including the dense retrieval used in [Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/). **Large models on one GPU.** [FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU](https://md.research.elamur.ai/free/en/papers/sheng-2023-flexgen-high-throughput-generative-inference/) (2023) targets batch workloads where latency is unimportant. It treats GPU memory, CPU memory and SSD as one hierarchy, uses linear programming to choose where weights, activations and the attention cache live and when they move, and compresses weights and cache to four bits without retraining. This let a 175-billion-parameter OPT model run for bulk processing on a single 16 GB card at far higher throughput than earlier offloading systems. **Can models write kernels?** [KernelBench: Can LLMs Write Efficient GPU Kernels?](https://md.research.elamur.ai/free/en/papers/ouyang-2025-kernelbench-llms-write-efficient-gpu/) (2025) poses PyTorch workloads, from single operators to full architectures, and asks a model to replace them with faster code. Its metric counts outputs that are both correct and faster than the reference by a chosen factor. Strong reasoning models rarely matched the baseline unaided, mostly failing on compilation and correctness, while feedback from execution and profiling raised scores considerably. The work links the topic to [Code & programming languages](https://md.research.elamur.ai/free/en/topics/code-and-programming/). **Attention built for the hardware.** [Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention](https://md.research.elamur.ai/free/en/papers/yuan-2025-native-sparse-attention-hardware/) (2025), from DeepSeek and Peking University, combines three attention branches (compressed block summaries, selected blocks and a sliding window) over contiguous blocks, with custom Triton kernels. Because the sparsity is present from pretraining and matches GPU memory access patterns, it speeds up both training and decoding on long sequences while matching full attention on the reported benchmarks. Citations of the papers on this topic, by when they were published Papers published in 2015–2019 gather 87% of the 5,391 citations of the papers on this topic. **The numbers** | Published | Papers and their citations | | --- | --- | | 2025–2029 | 2 papers, 305 citations | | 2020–2024 | 1 papers, 399 citations | | 2015–2019 | 1 papers, 4,687 citations | ## Common threads Three themes run through all four papers. First, memory movement rather than arithmetic is usually the limit, so the gains come from keeping data close to the compute units or moving it in large, regular blocks. Second, theoretical savings do not count until they show up in wall-clock time, a point the sparse attention paper makes explicitly about earlier methods. Third, software released with the paper (Faiss, FlexGen, KernelBench) is a large part of the impact. ## People Authors with the most papers on this topic Christopher Re leads with 2 papers; the 12 authors shown wrote 13 of the papers on this topic between them, counting each paper once per author. **The numbers** | Author | Papers here · their citations | | --- | --- | | [Christopher Re](https://md.research.elamur.ai/free/en/authors/christopher-re/) | 2 papers · 569 citations | | [Hervé Jégou](https://md.research.elamur.ai/free/en/authors/herve-jegou/) | 1 paper · 4,687 citations | | [Jeff Johnson](https://md.research.elamur.ai/free/en/authors/jeff-johnson/) | 1 paper · 4,687 citations | | [Matthijs Douze](https://md.research.elamur.ai/free/en/authors/matthijs-douze/) | 1 paper · 4,687 citations | | [Beidi Chen](https://md.research.elamur.ai/free/en/authors/beidi-chen/) | 1 paper · 399 citations | | [Clark Barrett](https://md.research.elamur.ai/free/en/authors/clark-barrett/) | 1 paper · 399 citations | | [Daniel Y. Fu](https://md.research.elamur.ai/free/en/authors/daniel-fu/) | 1 paper · 399 citations | | [Joseph E. Gonzalez](https://md.research.elamur.ai/free/en/authors/joseph-e-gonzalez/) | 1 paper · 399 citations | | [Lianmin Zheng](https://md.research.elamur.ai/free/en/authors/lianmin-zheng/) | 1 paper · 399 citations | | [Percy Liang](https://md.research.elamur.ai/free/en/authors/percy-liang/) | 1 paper · 399 citations | | [Ying Sheng](https://md.research.elamur.ai/free/en/authors/ying-sheng/) | 1 paper · 399 citations | | [Zhiqiang Xie](https://md.research.elamur.ai/free/en/authors/zhiqiang-xie/) | 1 paper · 399 citations | - [Jeff Johnson](https://md.research.elamur.ai/free/en/authors/jeff-johnson/), [Matthijs Douze](https://md.research.elamur.ai/free/en/authors/matthijs-douze/) and [Hervé Jégou](https://md.research.elamur.ai/free/en/authors/herve-jegou/) wrote the Faiss paper; Douze and Jégou are long known for product quantisation and large-scale similarity search. - [Christopher Re](https://md.research.elamur.ai/free/en/authors/christopher-re/) is the only author on two papers here, FlexGen and KernelBench, reflecting a sustained interest in systems for machine learning. - [Ying Sheng](https://md.research.elamur.ai/free/en/authors/ying-sheng/) led FlexGen, with co-authors including [Lianmin Zheng](https://md.research.elamur.ai/free/en/authors/lianmin-zheng/), [Zhuohan Li](https://md.research.elamur.ai/free/en/authors/zhuohan-li/), [Daniel Y. Fu](https://md.research.elamur.ai/free/en/authors/daniel-fu/), [Beidi Chen](https://md.research.elamur.ai/free/en/authors/beidi-chen/) of Carnegie Mellon University, [Zhiqiang Xie](https://md.research.elamur.ai/free/en/authors/zhiqiang-xie/), [Clark Barrett](https://md.research.elamur.ai/free/en/authors/clark-barrett/), [Joseph E. Gonzalez](https://md.research.elamur.ai/free/en/authors/joseph-e-gonzalez/) and [Percy Liang](https://md.research.elamur.ai/free/en/authors/percy-liang/). - [Simran Arora](https://md.research.elamur.ai/free/en/authors/simran-arora/) and [Azalia Mirhoseini](https://md.research.elamur.ai/free/en/authors/azalia-mirhoseini/) of Stanford University are among the KernelBench authors. - [Huazuo Gao](https://md.research.elamur.ai/free/en/authors/huazuo-gao/), [Damai Dai](https://md.research.elamur.ai/free/en/authors/damai-dai/), [Zhenda Xie](https://md.research.elamur.ai/free/en/authors/zhenda-xie/), [Chong Ruan](https://md.research.elamur.ai/free/en/authors/chong-ruan/), [Lean Wang](https://md.research.elamur.ai/free/en/authors/lean-wang/), [Liang Zhao](https://md.research.elamur.ai/free/en/authors/liang-zhao/) and [Zhengyan Zhang](https://md.research.elamur.ai/free/en/authors/zhengyan-zhang/) are among the Native Sparse Attention team. ## Institutions and places Most authors are in the United States and China. [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) and [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) appear through FlexGen and KernelBench; [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) and [DeepSeek](https://md.research.elamur.ai/free/en/companies/deepseek/) through FlexGen and sparse attention; French institutions such as the [Centre National de la Recherche Scientifique](https://md.research.elamur.ai/free/en/labs/centre-national-de-la-recherche-scientifique/) through the current affiliations of the Faiss authors. Where the authors on this topic work now Johnson Space Center is the current home of authors of 2 papers on this topic; counted are the authors with a page here, by their latest institution. **The numbers** | Institution | Papers | | --- | --- | | [Johnson Space Center](https://ror.org/04xx4z452) | 2 papers | | [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) | 2 papers | | [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) | 2 papers | | [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) | 1 paper | | [Center for Drug Evaluation and Research](https://ror.org/00yf3tm42) | 1 paper | | [Centre National de la Recherche Scientifique](https://md.research.elamur.ai/free/en/labs/centre-national-de-la-recherche-scientifique/) | 1 paper | | [Chinese Academy of Sciences](https://md.research.elamur.ai/free/en/government/chinese-academy-of-sciences/) | 1 paper | | [Cornell University](https://md.research.elamur.ai/free/en/universities/cornell-university/) | 1 paper | | [Dalian University of Technology](https://md.research.elamur.ai/free/en/universities/dalian-university-of-technology/) | 1 paper | | [DeepSeek](https://md.research.elamur.ai/free/en/companies/deepseek/) | 1 paper | Countries of the authors on this topic United States comes first: authors working there now wrote 3 papers on this topic. **The numbers** | Country | Papers | | --- | --- | | United States | 3 papers | | China | 2 papers | | India | 2 papers | | France | 1 paper | | Hungary | 1 paper | ## Open questions and neighbours The papers leave clear questions: whether models can generate kernels that are correct as well as fast without heavy feedback loops, how far offloading and quantisation can go before quality drops, and whether sparse attention trained from scratch generalises across tasks and hardware generations. These connect the topic to [Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/), [Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) and [Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/). The shelves this topic stands on Computing & GPUs: 4 of the 4 papers here. **The numbers** | Shelf | Papers | | --- | --- | | [Computing & GPUs](https://md.research.elamur.ai/free/en/shelves/computing/) | 4 papers | | [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/) | 1 paper | | [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) | 1 paper | ## All 4 papers on this topic 1. [**Billion-scale similarity search with GPUs**Jeff Johnson, Matthijs Douze, Hervé Jégou*2017 · 4 687 citations*](https://md.research.elamur.ai/free/en/papers/johnson-2017-billion-scale-similarity-search-gpus/) 2. [**FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU**Ying Sheng, Lianmin Zheng, Binhang Yuan, et al.*2023 · 399 citations*](https://md.research.elamur.ai/free/en/papers/sheng-2023-flexgen-high-throughput-generative-inference/) 3. [**KernelBench: Can LLMs Write Efficient GPU Kernels?**Anne Ouyang, Simon Guo, Simran Arora, et al.*2025 · 170 citations*](https://md.research.elamur.ai/free/en/papers/ouyang-2025-kernelbench-llms-write-efficient-gpu/) 4. [**Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention**Jingyang Yuan, Huazuo Gao, Damai Dai, et al.*2025 · 135 citations*](https://md.research.elamur.ai/free/en/papers/yuan-2025-native-sparse-attention-hardware/) - [← previous topic Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) - [next topic → Grammatical error correction & learner language](https://md.research.elamur.ai/free/en/topics/grammatical-error-correction/) - [↑ back to topics](https://md.research.elamur.ai/free/en/topics/)