---
title: "One weird trick for parallelizing convolutional neural networks"
description: "He therefore proposes a hybrid scheme: Convolutional layers use data parallelism: each GPU processes its own… Cited by 126 papers here."
authors: ["Alex Krizhevsky"]
year: 2014
venue: "arXiv:1404.5997"
source: "https://arxiv.org/abs/1404.5997"
doi: "10.48550/arxiv.1404.5997"
arxiv: "1404.5997"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/krizhevsky-2014-one-weird-trick-parallelizing-convolutional/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/krizhevsky-2014-one-weird-trick-parallelizing-convolutional/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2014](https://md.research.elamur.ai/free/en/papers/2014/) / One weird trick for parallelizing convolutional neural networks

# One weird trick for parallelizing convolutional neural networks

**Authors:** [Alex Krizhevsky](https://md.research.elamur.ai/free/en/authors/alex-krizhevsky/) · **Year:** 2014 · **Venue:** arXiv:1404.5997 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1404.5997) · [PDF](https://arxiv.org/pdf/1404.5997) · [DOI 10.48550/arxiv.1404.5997](https://doi.org/10.48550/arxiv.1404.5997) · [arXiv 1404.5997](https://arxiv.org/abs/1404.5997) · [OpenAlex](https://openalex.org/W1598866093)

**Citations:** 919 citations (OpenAlex, 2026-10-14) · [126](https://md.research.elamur.ai/free/en/papers/krizhevsky-2014-one-weird-trick-parallelizing-convolutional/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Training large convolutional networks on several GPUs is limited by communication. Pure data parallelism must exchange many parameters, and pure model parallelism must exchange many activations. Krizhevsky notes that the two layer types behave very differently. Convolutional layers do most of the computation but hold few weights. Fully-connected layers hold most of the weights but little computation.

He therefore proposes a hybrid scheme:
- **Convolutional layers** use data parallelism: each GPU processes its own batch.
- **Fully-connected layers** use model parallelism: the last convolutional activities are shared across GPUs.

He describes three ways to exchange these activities, some of which overlap communication with computation. He also describes a variant that updates the fully-connected weights more often. This variant uses a smaller effective batch in those layers and no longer exactly reproduces synchronous SGD.

The experiments use a single-tower version of the ILSVRC 2012 winning network, trained on ImageNet (ILSVRC 2012). On 8 GPUs, training runs about 6.2 times faster with only a small loss in accuracy. The work offers a simple, practical recipe for scaling convnet training across GPUs without redesigning the architecture.

## Key points
- Layer profile motivating the method:
  - Convolutional layers: about 90–95% of computation and about 5% of parameters.
  - Fully-connected layers: about 5–10% of computation and about 95% of parameters.
- Speedups over 1 GPU (98.05 h): 1.95x on 2 GPUs, 3.74x on 4 GPUs and 6.25x on 8 GPUs (15.68 h).
- Top-1 validation error rose from 42.33% on 1 GPU to 43.28% on 8 GPUs with batch 1024. Keeping the fully-connected batch at 128 reduced the 8-GPU error to 42.86%.
- In practice, the best learning-rate heuristic was to scale the rate linearly with batch size. This worked better than the square-root scaling suggested by the author's analysis.
- The model uses 1000 independent logistic output units instead of a softmax, which makes parallelization easier. Every run was trained for 90 epochs.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works · [🔌 Hardware and Architecture](https://md.research.elamur.ai/free/en/shelves/computer-science/hardware-and-architecture/) · 167 works

## Cite

Cite as: Alex Krizhevsky. 2014. One weird trick for parallelizing convolutional neural networks. arXiv:1404.5997.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{krizhevsky2014one,
  title = {{One weird trick for parallelizing convolutional neural networks}},
  author = {Alex Krizhevsky},
  year = {2014},
  doi = {10.48550/arxiv.1404.5997},
  eprint = {1404.5997},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1404.5997}
}
```

- [← previous A SICK cure for the evaluation of compositional distributional semantic models](https://md.research.elamur.ai/free/en/papers/marelli-2014-sick-cure-evaluation-compositional-distributional/)
- [next → Findings of the 2014 Workshop on Statistical Machine Translation](https://md.research.elamur.ai/free/en/papers/bojar-2014-findings-2014-workshop-statistical-machine/)
- [↑ back to papers of 2014](https://md.research.elamur.ai/free/en/papers/2014/)

*Updated 2 November 2026*