---
title: "Characterizing WebGPU Dispatch Overhead for LLM Inference"
description: "WebGPU offers portable GPU access, including inside browsers. 35 references. Among them: 0."
authors: ["Jędrzej Maczan"]
year: 2026
venue: "arXiv"
source: "https://arxiv.org/abs/2604.02344"
arxiv: "2604.02344"
licence: "CC BY 4.0 (arXiv OAI)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/maczan-2026-characterizing-webgpu-dispatch-overhead-llm/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/maczan-2026-characterizing-webgpu-dispatch-overhead-llm/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/27/) / Characterizing WebGPU Dispatch Overhead for LLM Inference Across Four GPU Vendors, Three Backends, and Three Browsers

# Characterizing WebGPU Dispatch Overhead for LLM Inference Across Four GPU Vendors, Three Backends, and Three Browsers

**Authors:** Jędrzej Maczan · **Year:** 2026 · **Venue:** arXiv · **Licence:** [CC BY 4.0 (arXiv OAI)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2604.02344) · [PDF](https://arxiv.org/pdf/2604.02344) · [arXiv 2604.02344](https://arxiv.org/abs/2604.02344) · [OpenAlex](https://openalex.org/W7150732884)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary

WebGPU offers portable GPU access, including inside browsers. Its safety-oriented design, however, validates and submits every operation separately. For LLM inference, which issues many tiny kernels, the real cost of this has not been measured carefully. This paper measures it at batch size 1 with Qwen2.5-0.5B and Qwen2.5-1.5B. Coverage spans GPUs from NVIDIA, AMD, Apple and Intel, the native implementations Dawn and wgpu-native, and the browsers Chrome, Safari and Firefox.

The author times long chains of sequential dispatches rather than isolated single operations. This shows that the isolated approach inflates overhead about twentyfold. The author also built torch-webgpu, an out-of-tree PyTorch backend (PrivateUse1) with an FX-to-WGSL compiler. It is compared against CUDA, MPS, CPU, ONNX Runtime and WebLLM.

The key finding is that per-operation overhead, not kernel quality, limits decoding speed. Kernel fusion speeds up Vulkan by roughly half but gives CUDA nothing. This matters for on-device and in-browser deployment and motivates spec changes such as graph capture/replay. All code and data are released openly.

## Key points

- Measured WebGPU API cost per dispatch is about 24–36 µs on Vulkan and about 32–71 µs on Metal. CUDA's launch cost is about 7.4 µs. Including Python/framework cost, total overhead is about 95–99 µs per operation.
- On Vulkan, fusion cuts dispatches for Qwen2.5-0.5B from 876 to 564, raising throughput by 53%. Fusion brings no gain on CUDA.
- torch-webgpu (RTX 5090, Dawn/Vulkan, float32) reaches 21.0 tok/s for 0.5B and 17.9 tok/s for 1.5B, about 11–12% of CUDA float16.
  - With both sides in float32, a mobile RTX PRO 2000 under CUDA is only 1.4× faster.
- Implementation matters: Safari Metal (31.7 µs) is about 2.2× cheaper than wgpu-native Metal (71 µs). Firefox shows about 1040 µs per dispatch on every platform, possibly due to rate limiting.
- Scope is limited:
  - torch-webgpu was validated only on RTX 5090/Dawn, at batch size 1, in float32.
  - WebLLM with q4f16 reaches 46–51 tok/s in Chrome.
  - Code is at github.com/jmaczan/torch-webgpu.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,888 works · [🔌 Hardware and Architecture](https://md.research.elamur.ai/free/en/shelves/computer-science/hardware-and-architecture/) · 183 works · [🖥️ Computer Graphics and Computer-Aided Design](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-graphics-and-computer-aided-design/) · 19 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 4,145 works · [⚡ GPU & parallel computing](https://md.research.elamur.ai/free/en/topics/gpu-parallel-computing/) · 44 works

## Cite

Cite as: Jędrzej Maczan. 2026. Characterizing WebGPU Dispatch Overhead for LLM Inference Across Four GPU Vendors, Three Backends, and Three Browsers. arXiv.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{maczan2026characterizing,
  title = {{Characterizing WebGPU Dispatch Overhead for LLM Inference Across Four GPU Vendors, Three Backends, and Three Browsers}},
  author = {Jędrzej Maczan},
  year = {2026},
  eprint = {2604.02344},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.02344}
}
```

- [← previous Character-aware Transformers Learn an Irregular Morphological Pattern Yet None Generalize Like Humans](https://md.research.elamur.ai/free/en/papers/ramarao-2026-character-aware-transformers-learn-irregular/)
- [next → Chart-MRAG: Benchmarking Multimodal Retrieval Augmented Generation on Chart-based Documents](https://md.research.elamur.ai/free/en/papers/ymyang-2026-chart-mrag-benchmarking-multimodal-retrieval/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/27/)

## References

35 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. [GQA: Training generalized multi-query transformer models from multi-head checkpoints](https://md.research.elamur.ai/free/en/papers/ainslie-2023-gqa-training-generalized-multi-query/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023.
2. Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, et al. [PyTorch 2: Faster machine learning through dynamic Python bytecode transformation and graph compilation](https://md.research.elamur.ai/free/en/papers/ansel-2024-pytorch-2-faster-machine-learning/). In Proceedings of the 29th ACM International…
3. Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, Carlos Guestrin, and Arvind Krishnamurthy. [TVM: An automated end-to-end optimizing compiler for deep learning](https://md.research.elamur.ai/free/en/papers/chen-2018-tvm-automated-end-end-optimizing/). In Proceedings of the 13th USENIX Symposium on Operating S…
4. Zhiyang Chen, Yun Ma, Haiyang Shen, and Mugeng Liu. WeInfer: Unleashing the power of WebGPU on LLM inference in web browsers. In Proceedings of the ACM Web Conference (WWW), 2025. doi: 10.1145/3696410.3714553. 3.76× speedup over WebLLM via buffer reuse and asynchronous pipelines. — [doi:10.1145/3696410.3714553](https://doi.org/10.1145/3696410.3714553) [find](https://scholar.google.com/scholar?q=Zhiyang+Chen%2C+Yun+Ma%2C+Haiyang+Shen%2C+and+Mugeng+Liu%2E+WeInfer%3A+Unleashing+the+power+of+WebGPU+on+LLM+inference+in+web+browsers%2E+In+Proceedings+of+the+ACM+Web+Conference+%28WWW%29%2C+2025%2E+doi%3A+10%2E1145%2F3696410)
5. Tri Dao. [FlashAttention-2: Faster attention with better parallelism and work partitioning](https://md.research.elamur.ai/free/en/papers/dao-2023-flashattention-2-faster-attention-better/). In International Conference on Learning Representations (ICLR), 2024.
6. Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, 2022. [find](https://scholar.google.com/scholar?q=Tri+Dao%2C+Daniel+Y%2E+Fu%2C+Stefano+Ermon%2C+Atri+Rudra%2C+and+Christopher+R%C3%A9%2E+FlashAttention%3A+Fast+and+memory%2Defficient+exact+attention+with+IO%2Dawareness%2E+In+Advances+in+Neural+Information+Processing+Systems+)
7. Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. LLM.int8(): 8-bit matrix multiplication for transformers at scale. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, 2022. [find](https://scholar.google.com/scholar?q=Tim+Dettmers%2C+Mike+Lewis%2C+Younes+Belkada%2C+and+Luke+Zettlemoyer%2E+LLM%2Eint8%28%29%3A+8%2Dbit+matrix+multiplication+for+transformers+at+scale%2E+In+Advances+in+Neural+Information+Processing+Systems+%28NeurIPS%29%2C+volum)
8. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training quantization for generative pre-trained transformers. In International Conference on Learning Representations (ICLR), 2023. [find](https://scholar.google.com/scholar?q=Elias+Frantar%2C+Saleh+Ashkboos%2C+Torsten+Hoefler%2C+and+Dan+Alistarh%2E+GPTQ%3A+Accurate+post%2Dtraining+quantization+for+generative+pre%2Dtrained+transformers%2E+In+International+Conference+on+Learning+Representat)
9. Georgi Gerganov. llama.cpp: LLM inference in C/C++. [https://github.com/ggerganov/llama.cpp](https://github.com/ggerganov/llama.cpp), 2023. Cross-platform CPU and GPU inference with quantization support. Accessed February 2026. gfx-rs Community. wgpu: Safe and portable GPU abstraction in Rust. https://wgpu.rs/, 2024. Pure-Rust WebGPU implem… [↗](https://github.com/ggerganov/llama.cpp)
10. Google. Dawn: WebGPU implementation for Chromium. [https://dawn.googlesource.com/dawn](https://dawn.googlesource.com/dawn), 2024a. Accessed February 2026. [↗](https://dawn.googlesource.com/dawn)
11. Google. MediaPipe: Cross-platform, customizable ML solutions for live and streaming media. [https://ai.google.dev/edge/mediapipe/solutions/guide](https://ai.google.dev/edge/mediapipe/solutions/guide), 2024b. Supports WebGPU for browser deployment. Accessed February 2026. [↗](https://ai.google.dev/edge/mediapipe/solutions/guide)
12. Google. TensorFlow.js. [https://www.tensorflow.org/js](https://www.tensorflow.org/js), 2024c. Accessed February 2026. [↗](https://www.tensorflow.org/js)
13. Google. XLA: Optimizing compiler for machine learning. [https://www.tensorflow.org/xla](https://www.tensorflow.org/xla), 2024d. Graph-level compilation with operator fusion and memory planning. Accessed February 2026. [↗](https://www.tensorflow.org/xla)
14. Masatoshi Hidaka. webgpu-blas: WebGPU-accelerated BLAS operations. [https://github.com/](https://github.com/) milhidaka/webgpu-blas, 2024. Reports 376 GFLOP/s for 4096×4096 matmul on Windows. Accessed February 2026. [↗](https://github.com/)
15. Hugging Face. Transformers.js: State-of-the-art machine learning for the web. [https://huggingface.co/](https://huggingface.co/) docs/transformers.js, 2024. JavaScript library for running transformer models in the browser with WebGPU support via ONNX Runtime Web. Accessed February 2026. [↗](https://huggingface.co/)
16. Khronos Group. Vulkan 1.3 specification. [https://registry.khronos.org/vulkan/specs/1.3/html/](https://registry.khronos.org/vulkan/specs/1.3/html/), 2024. Explicit GPU API designed to reduce CPU overhead vs OpenGL. Accessed February 2026. [↗](https://registry.khronos.org/vulkan/specs/1.3/html/)
17. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. [Efficient memory management for large language model serving with PagedAttention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/). In Proceedings of the 29th ACM Symposium on Operating Systems Principles (SOSP), 2023. doi:…
18. Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. AWQ: Activation-aware weight quantization for on-device LLM compression and acceleration. In Proceedings of Machine Learning and Systems (MLSys), 2024. [find](https://scholar.google.com/scholar?q=Ji+Lin%2C+Jiaming+Tang%2C+Haotian+Tang%2C+Shang+Yang%2C+Wei%2DMing+Chen%2C+Wei%2DChen+Wang%2C+Guangxuan+Xiao%2C+Xingyu+Dang%2C+Chuang+Gan%2C+and+Song+Han%2E+AWQ%3A+Activation%2Daware+weight+quantization+for+on%2Ddevice+LLM+compres)
19. Microsoft. ONNX runtime. [https://onnxruntime.ai/](https://onnxruntime.ai/), 2024. Accessed February 2026. [↗](https://onnxruntime.ai/)
20. MLC-AI. WebLLM: High-performance in-browser LLM inference engine. [https://github.com/mlc-ai/](https://github.com/mlc-ai/) web-llm, 2024. WebGPU-accelerated LLM inference in the browser. Version used: 0.2.78 (February 2026). [↗](https://github.com/mlc-ai/)
21. Zach Nuss. Optimizing a WebGPU matmul kernel for 1TFLOP+ performance. [https://www.nuss-and-bolts.com/p/optimizing-a-webgpu-matmul-kernel](https://www.nuss-and-bolts.com/p/optimizing-a-webgpu-matmul-kernel), 2024. Achieves ∼1 TFLOP/s (∼17% of M2 Pro peak) via 2D tiling, loop unrolling, and vectorized FMA. Accessed February 2026. [↗](https://www.nuss-and-bolts.com/p/optimizing-a-webgpu-matmul-kernel)
22. NVIDIA. CUDA graphs. [https://developer.nvidia.com/blog/cuda-graphs/](https://developer.nvidia.com/blog/cuda-graphs/), 2024a. Graph capture and replay mechanism to amortize kernel launch overhead. CUDA kernel launch latency typically 3–10 µs; graph replay reduces to \\<1 µs. Accessed February 2026. [↗](https://developer.nvidia.com/blog/cuda-graphs/)
23. NVIDIA. CUDA C++ programming guide. [https://docs.nvidia.com/cuda/](https://docs.nvidia.com/cuda/) cuda-c-programming-guide/, 2024b. Section on asynchronous concurrent execution details kernel launch overhead and hardware queue management. Accessed February 2026. [↗](https://docs.nvidia.com/cuda/)
24. NVIDIA. TensorRT: High-performance deep learning inference optimizer and runtime. [https://developer.nvidia.com/tensorrt](https://developer.nvidia.com/tensorrt), 2024c. Accessed February 2026. [↗](https://developer.nvidia.com/tensorrt)
25. Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. In Proceedings of Machine Learning and Systems (MLSys), 2023. [find](https://scholar.google.com/scholar?q=Reiner+Pope%2C+Sholto+Douglas%2C+Aakanksha+Chowdhery%2C+Jacob+Devlin%2C+James+Bradbury%2C+Jonathan+Heek%2C+Kefan+Xiao%2C+Shivani+Agrawal%2C+and+Jeff+Dean%2E+Efficiently+scaling+transformer+inference%2E+In+Proceedings+of+)
26. Qwen Team. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024. [find](https://scholar.google.com/scholar?q=Qwen+Team%2E+Qwen2%2E5+technical+report%2E+arXiv+preprint+arXiv%3A2412%2E15115%2C+2024%2E)
27. Noam Shazeer. GLU variants improve transformer. arXiv preprint arXiv:2002.05202, 2020. [find](https://scholar.google.com/scholar?q=Noam+Shazeer%2E+GLU+variants+improve+transformer%2E+arXiv+preprint+arXiv%3A2002%2E05202%2C+2020%2E)
28. Philippe Tillet, H. T. Kung, and David Cox. [Triton: An intermediate language and compiler for tiled neural network computations](https://md.research.elamur.ai/free/en/papers/tillet-2019-triton-intermediate-language-compiler-tiled/). In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages (MAPL), 2019. doi: 10.1145/3315508.3329973.
29. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems (NeurIPS), volume 30, 2017.
30. W3C GPU for the Web Working Group. WebGPU shading language (WGSL) specification. W3c working draft, World Wide Web Consortium, 2024a. URL [https://www.w3.org/TR/WGSL/](https://www.w3.org/TR/WGSL/). Accessed February 2026. [↗](https://www.w3.org/TR/WGSL/)
31. W3C GPU for the Web Working Group. WebGPU specification. W3c working draft, World Wide Web Consortium, 2024b. URL [https://www.w3.org/TR/webgpu/](https://www.w3.org/TR/webgpu/). Accessed February 2026. [↗](https://www.w3.org/TR/webgpu/)
32. W3C Web Machine Learning Working Group. Web neural network API. W3c candidate recommendation, World Wide Web Consortium, 2024. URL [https://www.w3.org/TR/webnn/](https://www.w3.org/TR/webnn/). Accessed February 2026. [↗](https://www.w3.org/TR/webnn/)
33. Samuel Williams, Andrew Waterman, and David Patterson. [Roofline: An insightful visual performance model for multicore architectures](https://md.research.elamur.ai/free/en/papers/williams-2009-roofline-insightful-visual-performance-model/). Communications of the ACM, 52(4):65–76, 2009. doi: 10.1145/1498765. 1498785.
34. Biao Zhang and Rico Sennrich. Root mean square layer normalization. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019. [find](https://scholar.google.com/scholar?q=Biao+Zhang+and+Rico+Sennrich%2E+Root+mean+square+layer+normalization%2E+In+Advances+in+Neural+Information+Processing+Systems+%28NeurIPS%29%2C+volume+32%2C+2019%2E)
35. Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. [SGLang: Efficient execution of structured language model programs](https://md.research.elamur.ai/free/en/papers/zheng-2023-sglang-efficient-execution-structured-language/). In Advances in Neural Information Processing Systems (Neu…

*Updated 3 November 2026*