--- title: "Distilling the Knowledge in a Neural Network — Hinton 2015" description: "Ensembles and very large regularized networks generalize well, but they are often too slow and expensive to serve to many users." authors: ["Geoffrey Hinton", "Oriol Vinyals", "Jeff Dean"] year: 2015 venue: "arXiv:1503.02531" source: "https://arxiv.org/abs/1503.02531" doi: "10.48550/arxiv.1503.02531" arxiv: "1503.02531" licence: "arXiv — non-exclusive licence to distribute" licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2015](https://md.research.elamur.ai/free/en/papers/2015/) / Distilling the Knowledge in a Neural Network # Distilling the Knowledge in a Neural Network **Authors:** [Geoffrey Hinton](https://md.research.elamur.ai/free/en/authors/geoffrey-hinton/), [Oriol Vinyals](https://md.research.elamur.ai/free/en/authors/oriol-vinyals/), [Jeff Dean](https://md.research.elamur.ai/free/en/authors/jeff-dean/) · **Year:** 2015 · **Venue:** arXiv:1503.02531 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/) **Links:** [Publisher's page](https://arxiv.org/abs/1503.02531) · [PDF](https://arxiv.org/pdf/1503.02531) · [DOI 10.48550/arxiv.1503.02531](https://doi.org/10.48550/arxiv.1503.02531) · [arXiv 1503.02531](https://arxiv.org/abs/1503.02531) · [OpenAlex](https://openalex.org/W1821462560) **Citations:** 12884 citations (openalex-snapshot · 2026-10-14) · [2142](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary Ensembles and very large regularized networks generalize well, but they are often too slow and expensive to serve to many users. Hinton, Vinyals and Dean propose "distillation", in which a compact student network learns to reproduce the class probabilities of a big "cumbersome" teacher. Raising the softmax temperature exposes the small probabilities the teacher gives to wrong classes, which reveal how it generalizes. The student's loss combines cross-entropy on these softened targets, scaled by T², with a lower-weighted cross-entropy on true labels. The authors show that Caruana-style logit matching is the high-temperature limit of this scheme. On MNIST, a small net recovers most of a dropout-trained large net's accuracy and can even recognize digits missing from its transfer set. For an acoustic model related to Android voice search, most of an ensemble's gain fits into one normal-sized network, and soft targets strongly regularize training on scarce data. The paper also introduces specialist networks for easily confused classes on JFT, which train in parallel, unlike mixtures of experts. ## Key points - **MNIST baseline result:** - The teacher (2×1200 ReLU units, dropout, two-pixel jitter) made 67 test errors. - An unregularized 2×800 net made 146 errors. - The same 2×800 net trained on soft targets at T=20 made only 74 errors. - **Temperature choice:** Students with at least 300 units per layer performed similarly at any T above 8. Very small students (30 units per layer) worked best at T between 2.5 and 4. - **Unseen classes:** - With every 3 removed from the transfer set, the student made 206 errors, 133 of them on the 1010 test threes. - Raising the 3-class bias by 3.5 cut this to 109 errors, with 98.6% of threes correct. - Training only on 7s and 8s gave 47.3% error, which fell to 13.2% after bias adjustment. - **Soft targets as regularizer:** For the 85M-parameter speech model trained on 3% of the data (about 20M examples): - Hard targets overfit and peaked at 44.5% test frame accuracy. - Soft targets reached 57.0%, without early stopping. - The full-data baseline reached 58.9%. - **Specialists on JFT:** Class subsets are chosen from the generalist's confusion matrix, and other classes go into a single dustbin class. Accuracy gains grew as more specialists covered a class. Distilling the specialists back into one net is left for future work. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Mathematics & statistics](https://md.research.elamur.ai/free/en/shelves/mathematics/) ## Cited by Cited by 2142 papers of our library cite it (of the 140884 whose bibliographies we read); 107 of them have a page here, the first 100 below, 2035 not yet. - [Towards a Human-like Open-Domain Chatbot](https://md.research.elamur.ai/free/en/papers/adiwardana-2020-towards-human-like-open-domain/) - [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://md.research.elamur.ai/free/en/papers/agarwal-2023-policy-distillation-language-models-learning/) - [Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning](https://md.research.elamur.ai/free/en/papers/aghajanyan-2021-intrinsic-dimensionality-explains-effectiveness-language/) - [Rationale-Guided Distillation for E-Commerce Relevance Classification: Bridging Large Language Models and Lightweight Cross-Encoders](https://md.research.elamur.ai/free/en/papers/agrawal-2025-rationale-guided-distillation-e-commerce/) - [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://md.research.elamur.ai/free/en/papers/ainslie-2023-gqa-training-generalized-multi-query/) - [CANDLE: CTC-based Arabic Noisy-character Deduplication using a Lightweight Encoder](https://md.research.elamur.ai/free/en/papers/alasmary-2026-candle-ctc-based-arabic-noisy/) - [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://md.research.elamur.ai/free/en/papers/ben-2021-zaken-bitfit-simple-parameter-efficient-fine/) - [Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence](https://md.research.elamur.ai/free/en/papers/bianchi-2021-pre-training-is-hot-topic/) - [Interpreting Pretrained Contextualized Representations via Reductions to Static Embeddings](https://md.research.elamur.ai/free/en/papers/bommasani-2020-interpreting-pretrained-contextualized-representations-reductions/) - [On the Opportunities and Risks of Foundation Models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/) - [Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision](https://md.research.elamur.ai/free/en/papers/burns-2023-weak-strong-generalization-eliciting-strong/) - [Emerging Properties in Self-Supervised Vision Transformers](https://md.research.elamur.ai/free/en/papers/caron-2021-emerging-properties-self-supervised-vision/) - [UNITER: UNiversal Image-TExt Representation Learning](https://md.research.elamur.ai/free/en/papers/chen-2019-uniter-universal-image-text-representation/) - [A Survey of Domain Adaptation for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/chu-2018-survey-domain-adaptation-neural-machine/) - [Rethinking embedding coupling in pre-trained language models](https://md.research.elamur.ai/free/en/papers/chung-2020-rethinking-embedding-coupling-pre-trained/) - [Semi-Supervised Sequence Modeling with Cross-View Training](https://md.research.elamur.ai/free/en/papers/clark-2018-semi-supervised-sequence-modeling-cross/) - [BAM! Born-Again Multi-Task Networks for Natural Language Understanding](https://md.research.elamur.ai/free/en/papers/clark-2019-bam-born-again-multi-task/) - [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://md.research.elamur.ai/free/en/papers/comanici-2025-gemini-2-5-pushing-frontier/) - [Security and Privacy Challenges of Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/das-2024-security-privacy-challenges-large-language/) - [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/deepseek-2025-ai-deepseek-r1-incentivizing-reasoning-capability/) - [Stronger Baselines for Trustable Results in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/denkowski-2017-stronger-baselines-trustable-results-neural/) - [TinyStories: How Small Can Language Models Be and Still Speak Coherent English?](https://md.research.elamur.ai/free/en/papers/eldan-2023-tinystories-small-language-models-still/) - [Learning to Parse and Translate Improves Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/eriguchi-2017-learning-parse-translate-improves-neural/) - [MBR and QE Finetuning: Training-time Distillation of the Best and Most Expensive Decoding Methods](https://md.research.elamur.ai/free/en/papers/finkelstein-2023-mbr-qe-finetuning-training-time/) - [IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages](https://md.research.elamur.ai/free/en/papers/gala-2023-indictrans2-towards-high-quality-accessible/) - [Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval](https://md.research.elamur.ai/free/en/papers/gao-2022-unsupervised-corpus-aware-language-model/) - [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-2024-team-gemini-1-5-unlocking-multimodal/) - [Stance Detection in COVID-19 Tweets](https://md.research.elamur.ai/free/en/papers/glandt-2021-stance-detection-covid-19-tweets/) - [Re2G: Retrieve, Rerank, Generate](https://md.research.elamur.ai/free/en/papers/glass-2022-re2g-retrieve-rerank-generate/) - [ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving](https://md.research.elamur.ai/free/en/papers/gou-2023-tora-tool-integrated-reasoning-agent/) - [MiniLLM: On-Policy Distillation of Large Language Models](https://md.research.elamur.ai/free/en/papers/gu-2023-minillm-policy-distillation-large-language/) - [The False Promise of Imitating Proprietary LLMs](https://md.research.elamur.ai/free/en/papers/gudibande-2023-false-promise-imitating-proprietary-llms/) - [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://md.research.elamur.ai/free/en/papers/howard-2017-mobilenets-efficient-convolutional-neural-networks/) - [Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes](https://md.research.elamur.ai/free/en/papers/hsieh-2023-distilling-step-step-outperforming-larger/) - [Harnessing Deep Neural Networks with Logic Rules](https://md.research.elamur.ai/free/en/papers/hu-2016-harnessing-deep-neural-networks-logic/) - [Large Language Models Can Self-Improve](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-can-self/) - [Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation](https://md.research.elamur.ai/free/en/papers/kasai-2020-deep-encoder-shallow-decoder-reevaluating/) - [ALADAN at IWSLT24 Low-resource Arabic Dialectal Speech Translation Task](https://md.research.elamur.ai/free/en/papers/kheder-2024-aladan-iwslt24-low-resource-arabic/) - [I-BERT: Integer-only BERT Quantization](https://md.research.elamur.ai/free/en/papers/kim-2021-i-bert-integer-only-bert/) - [DistiLLM: Towards Streamlined Distillation for Large Language Models](https://md.research.elamur.ai/free/en/papers/ko-2024-distillm-towards-streamlined-distillation-large/) - [Symphonym: Universal Phonetic Embeddings for Cross-Script Name Matching](https://md.research.elamur.ai/free/en/papers/langstream-2026-gadd-symphonym-universal-phonetic-embeddings-cross/) - [Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement](https://md.research.elamur.ai/free/en/papers/lee-2018-deterministic-non-autoregressive-neural-sequence/) - [Pretrained Language Models for Biomedical and Clinical Tasks: Understanding and Extending the State-of-the-Art](https://md.research.elamur.ai/free/en/papers/lewis-2020-pretrained-language-models-biomedical-clinical/) - [Improving Stance Detection with Multi-Dataset Learning and Knowledge Distillation](https://md.research.elamur.ai/free/en/papers/li-2021-improving-stance-detection-multi-dataset/) - [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://md.research.elamur.ai/free/en/papers/li-2022-blip-bootstrapping-language-image-pre/) - [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://md.research.elamur.ai/free/en/papers/li-2023-camel-communicative-agents-mind/) - [Distilling Calibrated Knowledge for Stance Detection](https://md.research.elamur.ai/free/en/papers/li-2023-distilling-calibrated-knowledge-stance-detection/) - [EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees](https://md.research.elamur.ai/free/en/papers/li-2024-eagle-2-faster-inference-language/) - [EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty](https://md.research.elamur.ai/free/en/papers/li-2024-eagle-speculative-sampling-requires-rethinking/) - [FastBERT: a Self-distilling BERT with Adaptive Inference Time](https://md.research.elamur.ai/free/en/papers/liu-2020-fastbert-self-distilling-bert-adaptive/) - [大语言模型的幻觉问题研究综述](https://md.research.elamur.ai/free/en/papers/liu-2024-hallucinations-large-language/) - [Investigating Memorization in Language Models Trained via Knowledge Distillation](https://md.research.elamur.ai/free/en/papers/macking-2026-investigating-memorization-language-models-trained/) - [Teaching Small Language Models to Reason](https://md.research.elamur.ai/free/en/papers/magister-2023-teaching-small-language-models-reason/) - [Merging Models with Fisher-Weighted Averaging](https://md.research.elamur.ai/free/en/papers/matena-2021-merging-models-fisher-weighted-averaging/) - [Generating Training Data with Language Models: Towards Zero-Shot Language Understanding](https://md.research.elamur.ai/free/en/papers/meng-2022-generating-training-data-language-models/) - [Improved Knowledge Distillation via Teacher Assistant](https://md.research.elamur.ai/free/en/papers/mirzadeh-2019-improved-knowledge-distillation-teacher-assistant/) - [Domain Robustness in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/muller-2020-domain-robustness-neural-machine-translation/) - [DINOv2: Learning Robust Visual Features without Supervision](https://md.research.elamur.ai/free/en/papers/oquab-2023-dinov2-learning-robust-visual-features/) - [F-coref: Fast, Accurate and Easy to Use Coreference Resolution](https://md.research.elamur.ai/free/en/papers/otmazgin-2022-f-coref-fast-accurate-easy/) - [Continual Lifelong Learning with Neural Networks: A Review](https://md.research.elamur.ai/free/en/papers/parisi-2018-continual-lifelong-learning-neural-networks/) - [Carbon Emissions and Large Neural Network Training](https://md.research.elamur.ai/free/en/papers/patterson-2021-carbon-emissions-large-neural-network/) - [Regularizing Neural Networks by Penalizing Confident Output Distributions](https://md.research.elamur.ai/free/en/papers/pereyra-2017-regularizing-neural-networks-penalizing-confident/) - [Discovering Language Model Behaviors with Model-Written Evaluations](https://md.research.elamur.ai/free/en/papers/perez-2022-discovering-language-model-behaviors-model/) - [Using the Output Embedding to Improve Language Models](https://md.research.elamur.ai/free/en/papers/press-2017-output-embedding-improve-language-models/) - [Learning Compact Metrics for MT](https://md.research.elamur.ai/free/en/papers/pu-2021-learning-compact-metrics-mt/) - [Why do Large Language Models Fail in Low-resource Translation? Unraveling the Token Dynamics of Large Language Models for Machine Translation](https://md.research.elamur.ai/free/en/papers/qian-2026-why-do-large-language-models/) - [Pre-trained Models for Natural Language Processing: A Survey](https://md.research.elamur.ai/free/en/papers/qiu-2020-pre-trained-models-natural-language/) - [RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering](https://md.research.elamur.ai/free/en/papers/qu-2021-rocketqa-optimized-training-approach-dense/) - [DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale](https://md.research.elamur.ai/free/en/papers/rajbhandari-2022-deepspeed-moe-advancing-mixture-experts/) - [Normalizing without Modernizing: Keeping Historical Wordforms of Middle French while Reducing Spelling Variants](https://md.research.elamur.ai/free/en/papers/rubino-2024-normalizing-without-modernizing-keeping-historical/) - [Masked Language Model Scoring](https://md.research.elamur.ai/free/en/papers/salazar-2020-masked-language-model-scoring/) - [Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference](https://md.research.elamur.ai/free/en/papers/schick-2021-exploiting-cloze-questions-few-shot/) - [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/schick-2021-it-s-not-just-size/) - [Membership Inference Attacks against Machine Learning Models](https://md.research.elamur.ai/free/en/papers/shokri-2016-membership-inference-attacks-against-machine/) - [Distilling Reasoning Capabilities into Smaller Language Models](https://md.research.elamur.ai/free/en/papers/shridhar-2023-distilling-reasoning-capabilities-into-smaller/) - [DeepBlueAI at WANLP-EACL2021 task 2: A Deep Ensemble-based Method for Sarcasm and Sentiment Detection in Arabic](https://md.research.elamur.ai/free/en/papers/song-2021-deepblueai-wanlp-eacl2021-task-2/) - [Synthetic Instruction Generation for Low-Resource Nordic Languages: Viability and Limitations in LLM Instruction-Tuning](https://md.research.elamur.ai/free/en/papers/stenlund-2026-synthetic-instruction-generation-low-resource/) - [Patient Knowledge Distillation for BERT Model Compression](https://md.research.elamur.ai/free/en/papers/sun-2019-patient-knowledge-distillation-bert-model/) - [MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices](https://md.research.elamur.ai/free/en/papers/sun-2020-mobilebert-compact-task-agnostic-bert/) - [The CUHKSZ System for the IWSLT 2026 Low-Resource Speech-to-Text Task](https://md.research.elamur.ai/free/en/papers/sun-2026-cuhksz-system-iwslt-2026-low/) - [Efficient Transformers: A Survey](https://md.research.elamur.ai/free/en/papers/tay-2020-efficient-transformers-survey/) - [Training data-efficient image transformers & distillation through attention](https://md.research.elamur.ai/free/en/papers/touvron-2020-training-data-efficient-image-transformers/) - [Reason-to-Learn (R2L): Multi-Agent Knowledge Distillation for Lightweight LLMs in Sentiment Analysis](https://md.research.elamur.ai/free/en/papers/tu-2026-reason-learn-r2l-multi-agent/) - [Well-Read Students Learn Better: On the Importance of Pre-training Compact Models](https://md.research.elamur.ai/free/en/papers/turc-2019-well-read-students-learn-better/) - [Parsing Tweets into Universal Dependencies](https://md.research.elamur.ai/free/en/papers/ud-2018-liu-parsing-tweets-into-universal-dependencies/) - [DyLoRA: Parameter-Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation](https://md.research.elamur.ai/free/en/papers/valipour-2023-dylora-parameter-efficient-tuning-pre/) - [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/) - [Structured Pruning of Large Language Models](https://md.research.elamur.ai/free/en/papers/wang-2020-structured-pruning-large-language-models/) - [MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning](https://md.research.elamur.ai/free/en/papers/wang-2023-mathcoder-seamless-code-integration-llms/) - [Symbolic Knowledge Distillation: from General Language Models to Commonsense Models](https://md.research.elamur.ai/free/en/papers/west-2022-symbolic-knowledge-distillation-general-language/) - [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://md.research.elamur.ai/free/en/papers/wortsman-2022-model-soups-averaging-weights-multiple/) - [Pay Less Attention with Lightweight and Dynamic Convolutions](https://md.research.elamur.ai/free/en/papers/wu-2019-pay-less-attention-lightweight-dynamic/) - [Scalable Zero-shot Entity Linking with Dense Entity Retrieval](https://md.research.elamur.ai/free/en/papers/wu-2020-scalable-zero-shot-entity-linking/) - [Connective Prediction for Implicit Discourse Relation Recognition via Knowledge Distillation](https://md.research.elamur.ai/free/en/papers/wu-2023-connective-prediction-implicit-discourse-relation/) - [Structured Pruning Learns Compact and Accurate Models](https://md.research.elamur.ai/free/en/papers/xia-2022-structured-pruning-learns-compact-accurate/) - [Unsupervised Data Augmentation for Consistency Training](https://md.research.elamur.ai/free/en/papers/xie-2019-unsupervised-data-augmentation-consistency-training/) - [RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation](https://md.research.elamur.ai/free/en/papers/xu-2023-recomp-improving-retrieval-augmented-lms/) - [ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers](https://md.research.elamur.ai/free/en/papers/yao-2022-zeroquant-efficient-affordable-post-training/) - [ZeroGen: Efficient Zero-shot Learning via Dataset Generation](https://md.research.elamur.ai/free/en/papers/ye-2022-zerogen-efficient-zero-shot-learning/) - [GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation](https://md.research.elamur.ai/free/en/papers/yoo-2021-gpt3mix-leveraging-large-scale-language/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{hinton2015distilling, title = {{Distilling the Knowledge in a Neural Network}}, author = {Hinton, Geoffrey and Vinyals, Oriol and Dean, Jeff}, year = {2015}, doi = {10.48550/arxiv.1503.02531}, eprint = {1503.02531}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.1503.02531} } ``` - [← previous Discourse parsing for multi-party chat dialogues](https://md.research.elamur.ai/free/en/papers/afantenos-2015-discourse-parsing-multi-party-chat/) - [next → Do Supervised Distributional Methods Really Learn Lexical Inference Relations?](https://md.research.elamur.ai/free/en/papers/levy-2015-do-supervised-distributional-methods-really/) - [↑ back to papers of 2015](https://md.research.elamur.ai/free/en/papers/2015/) ## References 9 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. C. Buciluǎ, R. Caruana, and A. Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’06, pages 535–541, New York, NY, USA, 2006. ACM. [find](https://scholar.google.com/scholar?q=C%2E+Bucilua%CC%8C%2C+R%2E+Caruana%2C+and+A%2E+Niculescu%2DMizil%2E+Model+compression%2E+In+Proceedings+of+the+12th+ACM+SIGKDD+International+Conference+on+Knowledge+Discovery+and+Data+Mining%2C+KDD+%E2%80%9906%2C+pages+535%E2%80%93541%2C+New+Y) 2. J. Dean, G. S. Corrado, R. Monga, K. Chen, M. Devin, Q. V. Le, M. Z. Mao, M. Ranzato, A. Senior, P. Tucker, K. Yang, and A. Y. Ng. Large scale distributed deep networks. In NIPS, 2012. [find](https://scholar.google.com/scholar?q=J%2E+Dean%2C+G%2E+S%2E+Corrado%2C+R%2E+Monga%2C+K%2E+Chen%2C+M%2E+Devin%2C+Q%2E+V%2E+Le%2C+M%2E+Z%2E+Mao%2C+M%2E+Ranzato%2C+A%2E+Senior%2C+P%2E+Tucker%2C+K%2E+Yang%2C+and+A%2E+Y%2E+Ng%2E+Large+scale+distributed+deep+networks%2E+In+NIPS%2C+2012%2E) 3. T. G. Dietterich. Ensemble methods in machine learning. In Multiple classifier systems, pages 1–15. Springer, 2000. [find](https://scholar.google.com/scholar?q=T%2E+G%2E+Dietterich%2E+Ensemble+methods+in+machine+learning%2E+In+Multiple+classifier+systems%2C+pages+1%E2%80%9315%2E+Springer%2C+2000%2E) 4. G. E. Hinton, L. Deng, D. Yu, G. E. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N Sainath, and B. Kingsbury. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. Signal Processing Magazine, IEEE, 29(6):82–97, 2012. [find](https://scholar.google.com/scholar?q=G%2E+E%2E+Hinton%2C+L%2E+Deng%2C+D%2E+Yu%2C+G%2E+E%2E+Dahl%2C+A%2E+Mohamed%2C+N%2E+Jaitly%2C+A%2E+Senior%2C+V%2E+Vanhoucke%2C+P%2E+Nguyen%2C+T%2E+N+Sainath%2C+and+B%2E+Kingsbury%2E+Deep+neural+networks+for+acoustic+modeling+in+speech+recognition%3A+T) 5. G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580, 2012. [find](https://scholar.google.com/scholar?q=G%2E+E%2E+Hinton%2C+N%2E+Srivastava%2C+A%2E+Krizhevsky%2C+I%2E+Sutskever%2C+and+R%2E+R%2E+Salakhutdinov%2E+Improving+neural+networks+by+preventing+co%2Dadaptation+of+feature+detectors%2E+arXiv+preprint+arXiv%3A1207%2E0580%2C+2012%2E) 6. R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991. [find](https://scholar.google.com/scholar?q=R%2E+A%2E+Jacobs%2C+M%2E+I%2E+Jordan%2C+S%2E+J%2E+Nowlan%2C+and+G%2E+E%2E+Hinton%2E+Adaptive+mixtures+of+local+experts%2E+Neural+computation%2C+3%281%29%3A79%E2%80%9387%2C+1991%2E) 7. A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, pages 1097–1105, 2012. [find](https://scholar.google.com/scholar?q=A%2E+Krizhevsky%2C+I%2E+Sutskever%2C+and+G%2E+E%2E+Hinton%2E+Imagenet+classification+with+deep+convolutional+neural+networks%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+1097%E2%80%931105%2C+2012%2E) 8. J. Li, R. Zhao, J. Huang, and Y. Gong. Learning small-size dnn with output-distribution-based criteria. In Proceedings Interspeech 2014, pages 1910–1914, 2014. [find](https://scholar.google.com/scholar?q=J%2E+Li%2C+R%2E+Zhao%2C+J%2E+Huang%2C+and+Y%2E+Gong%2E+Learning+small%2Dsize+dnn+with+output%2Ddistribution%2Dbased+criteria%2E+In+Proceedings+Interspeech+2014%2C+pages+1910%E2%80%931914%2C+2014%2E) 9. N. Srivastava, G.E. Hinton, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929–1958, 2014. [find](https://scholar.google.com/scholar?q=N%2E+Srivastava%2C+G%2EE%2E+Hinton%2C+A%2E+Krizhevsky%2C+I%2E+Sutskever%2C+and+R%2E+R%2E+Salakhutdinov%2E+Dropout%3A+A+simple+way+to+prevent+neural+networks+from+overfitting%2E+The+Journal+of+Machine+Learning+Research%2C+15%281%29%3A192)