--- title: "End-to-End Object Detection with Transformers — Carion 2020" description: "Conventional object detectors solve detection indirectly." authors: ["Nicolas Carion", "Francisco Massa", "Gabriel Synnaeve", "Nicolas Usunier", "Alexander Kirillov", "Sergey Zagoruyko"] year: 2020 venue: "arXiv:2005.12872" source: "https://arxiv.org/abs/2005.12872" doi: "10.48550/arxiv.2005.12872" arxiv: "2005.12872" licence: "CC0 1.0 (arXiv)" licence_url: "http://creativecommons.org/publicdomain/zero/1.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/carion-2020-end-end-object-detection-transformers/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/carion-2020-end-end-object-detection-transformers/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/2/) / End-to-End Object Detection with Transformers # End-to-End Object Detection with Transformers **Authors:** Nicolas Carion, [Francisco Massa](https://md.research.elamur.ai/free/en/authors/francisco-massa/), [Gabriel Synnaeve](https://md.research.elamur.ai/free/en/authors/gabriel-synnaeve/), Nicolas Usunier, [Alexander Kirillov](https://md.research.elamur.ai/free/en/authors/alexander-kirillov/), Sergey Zagoruyko · **Year:** 2020 · **Venue:** arXiv:2005.12872 · **Licence:** [CC0 1.0 (arXiv)](http://creativecommons.org/publicdomain/zero/1.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2005.12872) · [PDF](https://arxiv.org/pdf/2005.12872) · [DOI 10.48550/arxiv.2005.12872](https://doi.org/10.48550/arxiv.2005.12872) · [arXiv 2005.12872](https://arxiv.org/abs/2005.12872) · [OpenAlex](https://openalex.org/W3030520226) **Citations:** 16417 citations (openalex-snapshot · 2026-10-14) · [344](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Conventional object detectors solve detection indirectly. They regress and classify relative to proposals, anchors or grid centers, then depend on hand-tuned parts such as anchor design and non-maximum suppression (NMS) to remove duplicates. Carion et al. (Facebook AI) instead frame detection as predicting a set directly. Their model, DETR (DEtection TRansformer), combines a standard CNN backbone (ResNet) with a transformer encoder-decoder. Starting from a small fixed set of learned object queries, it emits all detections in parallel. Training relies on a set-level loss: the Hungarian algorithm computes a one-to-one assignment between predictions and ground-truth boxes, and leftover slots must output a "no object" class. On COCO, DETR matches a strong, heavily tuned Faster R-CNN baseline in accuracy and speed. It is notably better on large objects but weaker on small ones, and it needs a very long training schedule. A lightweight mask head extends it to panoptic segmentation, where it outperforms established baselines. The work matters because it removes much detection-specific engineering and requires no custom layers. ## Key points - Main components: a bipartite-matching (Hungarian) loss over a fixed set of N predictions, plus non-autoregressive parallel decoding with a transformer. The design uses no anchors and no NMS. - In the loss, the classification term for "no object" slots is down-weighted by a factor of 10 to offset class imbalance. Auxiliary decoding losses help training. - For panoptic segmentation on COCO (80 things + 53 stuff categories), a mask head is trained for 25 epochs on top of a frozen DETR. Per-pixel argmax merges the masks without overlap heuristics. - On COCO val, DETR-R101 reaches 45.1 PQ versus 44.1 for a retrained PanopticFPN++ (R101), and 37.0 versus 33.6 PQ on stuff classes. It reaches 46 PQ on the COCO test set, although its mask AP trails the baselines by up to 8 points. - Training code and pretrained models are released at github.com/facebookresearch/detr. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/) ## Cited by Cited by 344 papers of our library cite it (of the 140673 whose bibliographies we read); 13 of them have a page here, 331 not yet. - [Flamingo: a Visual Language Model for Few-Shot Learning](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/) - [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](https://md.research.elamur.ai/free/en/papers/chen-2023-shikra-unleashing-multimodal-llm-s/) - [Perceiver: General Perception with Iterative Attention](https://md.research.elamur.ai/free/en/papers/jaegle-2021-perceiver-general-perception-iterative-attention/) - [Segment Anything](https://md.research.elamur.ai/free/en/papers/kirillov-2023-segment-anything/) - [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://md.research.elamur.ai/free/en/papers/liu-2021-swin-transformer-hierarchical-vision-transformer/) - [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://md.research.elamur.ai/free/en/papers/liu-2023-grounding-dino-marrying-dino-grounded/) - [Prompt for Extraction? PAIE: Prompting Argument Interaction for Event Argument Extraction](https://md.research.elamur.ai/free/en/papers/ma-2022-prompt-extraction-paie-prompting-argument/) - [Transformers learn in-context by gradient descent](https://md.research.elamur.ai/free/en/papers/oswald-2022-transformers-learn-context-gradient-descent/) - [AraSpell: A Deep Learning Approach for Arabic Spelling Correction](https://md.research.elamur.ai/free/en/papers/salhab-2024-araspell-deep-learning-approach-arabic/) - [HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face](https://md.research.elamur.ai/free/en/papers/shen-2023-hugginggpt-solving-ai-tasks-chatgpt/) - [Efficient Transformers: A Survey](https://md.research.elamur.ai/free/en/papers/tay-2020-efficient-transformers-survey/) - [Training data-efficient image transformers & distillation through attention](https://md.research.elamur.ai/free/en/papers/touvron-2020-training-data-efficient-image-transformers/) - [Multimodal Chain-of-Thought Reasoning in Language Models](https://md.research.elamur.ai/free/en/papers/zhang-2023-multimodal-chain-thought-reasoning-language/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{carion2020end, title = {{End-to-End Object Detection with Transformers}}, author = {Carion, Nicolas and Massa, Francisco and Synnaeve, Gabriel and Usunier, Nicolas and Kirillov, Alexander and Zagoruyko, Sergey}, year = {2020}, doi = {10.48550/arxiv.2005.12872}, eprint = {2005.12872}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2005.12872} } ``` - [← previous End-to-End Bias Mitigation by Modelling Biases in Corpora](https://md.research.elamur.ai/free/en/papers/mahabadi-2020-end-end-bias-mitigation-modelling/) - [next → End-to-End Slot Alignment and Recognition for Cross-Lingual NLU](https://md.research.elamur.ai/free/en/papers/xu-2020-end-end-slot-alignment-recognition/) - [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/2/) ## References 3 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. 1. Al-Rfou, R., Choe, D., Constant, N., Guo, M., Jones, L.: Character-level language modeling with deeper self-attention. In: AAAI Conference on Artificial Intelligence (2019) 2. Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. In: ICLR (2015)… [find](https://scholar.google.com/scholar?q=1%2E+Al%2DRfou%2C+R%2E%2C+Choe%2C+D%2E%2C+Constant%2C+N%2E%2C+Guo%2C+M%2E%2C+Jones%2C+L%2E%3A+Character%2Dlevel+language+modeling+with+deeper+self%2Dattention%2E+In%3A+AAAI+Conference+on+Artificial+Intelligence+%282019%29+2%2E+Bahdanau%2C+D%2E%2C+Cho%2C+K%2E) 2. PAMI (2019) 6. Chan, W., Saharia, C., Hinton, G., Norouzi, M., Jaitly, N.: Imputer: Sequence modelling via imputation and dynamic programming. arXiv:2002.08926 (2020) 7. Cordonnier, J.B., Loukas, A., Jaggi, M.: On the relationship between self-attention and convolutional layers. In: ICLR (2020) 8. D… [find](https://scholar.google.com/scholar?q=PAMI+%282019%29+6%2E+Chan%2C+W%2E%2C+Saharia%2C+C%2E%2C+Hinton%2C+G%2E%2C+Norouzi%2C+M%2E%2C+Jaitly%2C+N%2E%3A+Imputer%3A+Sequence+modelling+via+imputation+and+dynamic+programming%2E+arXiv%3A2002%2E08926+%282020%29+7%2E+Cordonnier%2C+J%2EB%2E%2C+Loukas%2C+A%2E%2C+) 3. PAMI (2015) 38. Rezatofighi, H., Tsoi, N., Gwak, J., Sadeghian, A., Reid, I., Savarese, S.: Generalized intersection over union. In: CVPR (2019) 39. Rezatofighi, S.H., Kaskman, R., Motlagh, F.T., Shi, Q., Cremers, D., Leal-Taixé, L., Reid, I.: Deep perm-set net: Learn to predict sets with unknown p… [find](https://scholar.google.com/scholar?q=PAMI+%282015%29+38%2E+Rezatofighi%2C+H%2E%2C+Tsoi%2C+N%2E%2C+Gwak%2C+J%2E%2C+Sadeghian%2C+A%2E%2C+Reid%2C+I%2E%2C+Savarese%2C+S%2E%3A+Generalized+intersection+over+union%2E+In%3A+CVPR+%282019%29+39%2E+Rezatofighi%2C+S%2EH%2E%2C+Kaskman%2C+R%2E%2C+Motlagh%2C+F%2ET%2E%2C+Shi%2C)