---
title: "Microscaling Data Formats for Deep Learning — Rouhani 2023"
description: "GPT-style models trained entirely in 6-bit MX reach FP32 loss with unchanged hyperparameters… 18 references, cited by 70 papers here."
authors: ["Bita Darvish Rouhani", "Ritchie Zhao", "Ankit More", "Mathew Hall", "Alireza Khodamoradi", "Summer Deng", "Dhruv Choudhary", "Marius Cornea", "Eric Dellinger", "Kristof Denolf"]
year: 2023
venue: "arXiv 2023 (OCP MX consortium)"
source: "https://arxiv.org/abs/2310.10537"
arxiv: "2310.10537"
licence: "arXiv — non-exclusive distribution license (arXiv OAI)"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/rouhani-2023-microscaling-data-formats-deep-learning/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/rouhani-2023-microscaling-data-formats-deep-learning/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/9/) / Microscaling Data Formats for Deep Learning

# Microscaling Data Formats for Deep Learning

**Authors:** Bita Darvish Rouhani, Ritchie Zhao, Ankit More, Mathew Hall, Alireza Khodamoradi, Summer Deng, Dhruv Choudhary, Marius Cornea, Eric Dellinger and Kristof Denolf · **Year:** 2023 · **Venue:** arXiv 2023 (OCP MX consortium) · **Licence:** [arXiv — non-exclusive distribution license (arXiv OAI)](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2310.10537) · [PDF](https://arxiv.org/pdf/2310.10537) · [arXiv 2310.10537](https://arxiv.org/abs/2310.10537) · [OpenAlex](https://openalex.org/W4387725429)

**Citations:** 9 citations (OpenAlex, 2026-10-14) · [70](https://md.research.elamur.ai/free/en/papers/rouhani-2023-microscaling-data-formats-deep-learning/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Low-precision number formats lower the compute and memory cost of deep learning. However, formats narrower than 8 bits cannot cover the value range of a whole tensor with a single scale factor. This paper, from an industry group including Microsoft, AMD, Intel, Meta, NVIDIA and Qualcomm, evaluates the Microscaling (MX) family from the open OCP standard. In MX, small blocks of 32 values share one power-of-two scale, and each value is stored as a tiny float or integer. The authors emulate MX on GPUs with a PyTorch CUDA extension. They test four settings across more than two dozen benchmarks in translation, text encoding, vision, speech, recommendation and LLMs, including GPT3-175B and LLaMA-7B:

- direct casting of FP32 models
- a post-training quantization method
- quantization-aware finetuning
- training from scratch

MXINT8 works as a drop-in substitute for FP32 inference. MXFP6 nearly matches FP32 after finetuning. GPT-style models trained entirely in 6-bit MX reach FP32 loss with unchanged hyperparameters, which the authors present as the first such sub-8-bit training result. This matters for designing cheaper training and inference hardware.

## Key points
- **Formats:** all concrete formats use blocks of 32 elements and an 8-bit exponent-only scale (E8M0). The element types are:
  - MXFP8 (E4M3/E5M2)
  - MXFP6 (E2M3/E3M2)
  - MXFP4 (E2M1)
  - MXINT8
- **Compute flow:** matmuls and convolutions in both forward and backward passes take MX inputs. Normalization, Softmax, activations and residual adds run in Bfloat16 or FP32, and an FP32 master copy of the weights is updated.
- **Direct-cast inference:** for MXFP8 and MXFP6, the variant with more mantissa bits did better. On LLaMA-7B, wikitext perplexity is:
  - 9.488 for FP32
  - 9.504 for MXINT8
  - 27.201 when everything is MXFP4
- **Training GPT models (20M–1.5B parameters):**
  - MXFP6_E3M2 for weights, activations and gradients gives loss nearly equal to FP32 (2.75 vs 2.74 at 1.5B).
  - MXFP4 weights with MXFP6 activations and gradients give 2.76.
- **Post-training quantization:** the PTQ method ("error diffusion") is derived from GPFQ. It makes one pass over a small calibration set and needs no tuning parameters.

## Interesting ideas
- **Offset the shared exponent:** the block's shared exponent is set from the block's largest magnitude, then shifted so that this value lands in the element format's top binade. This uses the full range of a tiny format, and the same rule could serve any block-wise quantizer, such as embedding or KV-cache compression.
- **Separate scale from element type:** scale format, element format and block size are defined independently, so one framework covers many bit-widths. A modular spec like this makes extending or comparing formats straightforward.
- **Plan for transposes:** quantizing and transposing do not commute, because the scale is tied to one axis. Training code must therefore keep separately quantized copies of a weight and its transpose, a subtle point for anyone implementing block-scaled kernels.
- **Reuse the FP32 recipe:** judging a format by whether it trains with unchanged FP32 hyperparameters directly measures "user friction." This is a practical evaluation criterion beyond plain accuracy.
- **Grade by adaptation effort:** comparing direct-cast, PTQ, finetuning and full training shows how much effort each format needs. This tiered protocol is useful for evaluating any compression scheme.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works · [🧮 Computational Theory and Mathematics](https://md.research.elamur.ai/free/en/shelves/computer-science/computational-theory-and-mathematics/) · 321 works · [⚙️ Engineering](https://md.research.elamur.ai/free/en/shelves/engineering/) · 188 works · [⚙️ Electrical and Electronic Engineering](https://md.research.elamur.ai/free/en/shelves/engineering/electrical-and-electronic-engineering/) · 33 works

## Cite

Cite as: Bita Darvish Rouhani, Ritchie Zhao, Ankit More, Mathew Hall, Alireza Khodamoradi, Summer Deng, Dhruv Choudhary, Marius Cornea, Eric Dellinger and Kristof Denolf. 2023. Microscaling Data Formats for Deep Learning. arXiv 2023 (OCP MX consortium).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{rouhani2023microscaling,
  title = {{Microscaling Data Formats for Deep Learning}},
  author = {Bita Darvish Rouhani, Ritchie Zhao, Ankit More, Mathew Hall, Alireza Khodamoradi, Summer Deng, Dhruv Choudhary, Marius Cornea, Eric Dellinger and Kristof Denolf},
  year = {2023},
  eprint = {2310.10537},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.10537}
}
```

- [← previous LEGO-Prover: Neural Theorem Proving with Growing Libraries](https://md.research.elamur.ai/free/en/papers/wang-2023-lego-prover-neural-theorem-proving/)
- [next → Multilingual Language Models are not Multicultural: A Case Study in Emotion](https://md.research.elamur.ai/free/en/papers/havaldar-2023-multilingual-language-models-are-not/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/9/)

## References

18 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Paulius Micikevicius, Stuart Oberman, Pradeep Dubey, Marius Cornea, Andres Rodriguez, Ian Bratt, Richard Grisenthwaite, Norm Jouppi, Chiachen Chou, Amber Huffman, Michael Schulte, Ralph Wittig, Dharmesh Jani, and Summer Deng. OCP 8-bit Floating Point Specification (OFP8). Open Compute Project, 2023. [find](https://scholar.google.com/scholar?q=Paulius+Micikevicius%2C+Stuart+Oberman%2C+Pradeep+Dubey%2C+Marius+Cornea%2C+Andres+Rodriguez%2C+Ian+Bratt%2C+Richard+Grisenthwaite%2C+Norm+Jouppi%2C+Chiachen+Chou%2C+Amber+Huffman%2C+Michael+Schulte%2C+Ralph+Wittig%2C+Dharme)
2. Mario Drumond, Tao Lin, Martin Jaggi, and Babak Falsafi. Training DNNs with Hybrid Block Floating Point. Advances in Neural Information Processing Systems (NeurIPS), 31, 2018. [find](https://scholar.google.com/scholar?q=Mario+Drumond%2C+Tao+Lin%2C+Martin+Jaggi%2C+and+Babak+Falsafi%2E+Training+DNNs+with+Hybrid+Block+Floating+Point%2E+Advances+in+Neural+Information+Processing+Systems+%28NeurIPS%29%2C+31%2C+2018%2E)
3. Bita Darvish Rouhani, Daniel Lo, Ritchie Zhao, Ming Liu, Jeremy Fowers, Kalin Ovtcharov, Anna Vinogradsky, Sarah Massengill, Lita Yang, Ray Bittner, Alessandro Forin, Haishan Zhu, Taesik Na, Prerak Patel, Shuai Che, Lok Chand Koppaka, XIA SONG, Subhojit Som, Kaustav Das, Saurabh T, Steve Reinhardt,… [find](https://scholar.google.com/scholar?q=Bita+Darvish+Rouhani%2C+Daniel+Lo%2C+Ritchie+Zhao%2C+Ming+Liu%2C+Jeremy+Fowers%2C+Kalin+Ovtcharov%2C+Anna+Vinogradsky%2C+Sarah+Massengill%2C+Lita+Yang%2C+Ray+Bittner%2C+Alessandro+Forin%2C+Haishan+Zhu%2C+Taesik+Na%2C+Prerak+Pa)
4. Steve Dai, Rangha Venkatesan, Mark Ren, Brian Zimmer, William Dally, and Brucek Khailany. VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inference. Machine Learning and Systems (MLSys, 3:873–884, 2021. [find](https://scholar.google.com/scholar?q=Steve+Dai%2C+Rangha+Venkatesan%2C+Mark+Ren%2C+Brian+Zimmer%2C+William+Dally%2C+and+Brucek+Khailany%2E+VS%2DQuant%3A+Per%2Dvector+Scaled+Quantization+for+Accurate+Low%2DPrecision+Neural+Network+Inference%2E+Machine+Learning)
5. Bita Darvish Rouhani, Ritchie Zhao, Venmugil Elango, Rasoul Shafipour, Mathew Hall, Maral Mesmakhosroshahi, Ankit More, Levi Melnick, Maximilian Golub, Girish Varatkar, Lei Shao, Gaurav Kolhe, Dimitry Melts, Jasmine Klar, Renee L’Heureux, Matt Perry, Doug Burger, and Eric Chung. With Shared Microexp… [find](https://scholar.google.com/scholar?q=Bita+Darvish+Rouhani%2C+Ritchie+Zhao%2C+Venmugil+Elango%2C+Rasoul+Shafipour%2C+Mathew+Hall%2C+Maral+Mesmakhosroshahi%2C+Ankit+More%2C+Levi+Melnick%2C+Maximilian+Golub%2C+Girish+Varatkar%2C+Lei+Shao%2C+Gaurav+Kolhe%2C+Dimitry)
6. Bita Darvish Rouhani, Nitin Garegrat, Tom Savell, Ankit More, Kyung-Nam Han, Mathew Zhao, Ritchie amd Hall, Jasmine Klar, Eric Chung, Yuan Yu, Michael Schulte, Ralph Wittig, Ian Bratt, Nigel Stephens, Jelena Milanovic, John Brothers, Pradeep Dubey, Marius Cornea, Alexander Heinecke, Andres Rodriguez… [find](https://scholar.google.com/scholar?q=Bita+Darvish+Rouhani%2C+Nitin+Garegrat%2C+Tom+Savell%2C+Ankit+More%2C+Kyung%2DNam+Han%2C+Mathew+Zhao%2C+Ritchie+amd+Hall%2C+Jasmine+Klar%2C+Eric+Chung%2C+Yuan+Yu%2C+Michael+Schulte%2C+Ralph+Wittig%2C+Ian+Bratt%2C+Nigel+Stephens%2C)
7. Microscaling PyTorch Library. 2023. URL [https://github.com/microsoft/microxcaling](https://github.com/microsoft/microxcaling). [↗](https://github.com/microsoft/microxcaling)
8. Jinjie Zhang, Yixuan Zhou, and Rayan Saab. Post-training quantization for neural networks with provable guarantees. arXiv:2201.11113, 2022. — [arXiv:2201.11113](https://arxiv.org/abs/2201.11113) [find](https://scholar.google.com/scholar?q=Jinjie+Zhang%2C+Yixuan+Zhou%2C+and+Rayan+Saab%2E+Post%2Dtraining+quantization+for+neural+networks+with+provable+guarantees%2E+arXiv%3A2201%2E11113%2C+2022%2E+%E2%80%94+%5BarXiv%3A2201%2E11113%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2201%2E11113%29)
9. Transformer For PyTorch. URL [https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Translation/Transformer](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Translation/Transformer). [↗](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Translation/Transformer)
10. GNMT v2 For PyTorch. URL [https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Translation/GNMT](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Translation/GNMT). [↗](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Translation/GNMT)
11. NVIDIA/Megatron-LM: Ongoing research training transformer. URL [https://github.com/NVIDIA/Megatron-LM](https://github.com/NVIDIA/Megatron-LM). [↗](https://github.com/NVIDIA/Megatron-LM)
12. Data-Efficient architectures and training for Image classification. facebookresearch/deit. [find](https://scholar.google.com/scholar?q=Data%2DEfficient+architectures+and+training+for+Image+classification%2E+facebookresearch%2Fdeit%2E)
13. Convolutional Network for Image Classification in PyTorch. URL [https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Classification/ConvNets](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Classification/ConvNets). [↗](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Classification/ConvNets)
14. Torchvision MobileNetV2. URL [https://github.com/pytorch/vision](https://github.com/pytorch/vision). [↗](https://github.com/pytorch/vision)
15. wav2vec 2.0. URL [https://github.com/facebookresearch/fairseq/tree/main/examples/wav2vec](https://github.com/facebookresearch/fairseq/tree/main/examples/wav2vec). [↗](https://github.com/facebookresearch/fairseq/tree/main/examples/wav2vec)
16. Deep Learning Recommendation Model for Personalization and Recommendation Systems. URL [https://github.com/facebookresearch/dlrm](https://github.com/facebookresearch/dlrm). [↗](https://github.com/facebookresearch/dlrm)
17. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
18. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. [Scaling Laws for Neural Language Models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). arXiv preprint arXiv:2001.08361, 2020. URL [https://github.com/](https://github.com/) [↗](https://github.com/)

*Updated 2 November 2026*