--- title: "Relevance-guided Supervision for OpenQA with ColBERT (2021)" description: "The paper presents ColBERT-QA, which adapts the ColBERT late-interaction retriever to… 2.3 points. 3 references, cited by 46 papers here." authors: ["Omar Khattab", "Christopher Potts", "Matei Zaharia"] year: 2021 venue: "Transactions of the Association for Computational Linguistics" source: "https://aclanthology.org/2021.tacl-1.55/" doi: "10.1162/tacl_a_00405" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/khattab-2021-relevance-guided-supervision-openqa-colbert/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/khattab-2021-relevance-guided-supervision-openqa-colbert/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) / Relevance-guided Supervision for OpenQA with ColBERT # Relevance-guided Supervision for OpenQA with ColBERT **Authors:** [Omar Khattab](https://md.research.elamur.ai/free/en/authors/omar-khattab/), [Christopher Potts](https://md.research.elamur.ai/free/en/authors/christopher-potts/) and [Matei Zaharia](https://md.research.elamur.ai/free/en/authors/matei-zaharia/) · **Year:** 2021 · **Venue:** [Transactions of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/tacl/) ([Transactions of the Association for Computational Linguistics (TACL) 2021](https://md.research.elamur.ai/free/en/venues/tacl/2021/)) · **Publisher:** [The MIT Press](https://md.research.elamur.ai/free/en/publishers/the-mit-press/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2021.tacl-1.55/) · [PDF](https://aclanthology.org/2021.tacl-1.55.pdf) · [DOI 10.1162/tacl\_a\_00405](https://doi.org/10.1162/tacl_a_00405) · [OpenAlex](https://openalex.org/W3038831159) **Citations:** 71 citations (OpenAlex, 2026-10-14) · [46](https://md.research.elamur.ai/free/en/papers/khattab-2021-relevance-guided-supervision-openqa-colbert/cited-by/) in our library, of the 141,840 papers whose bibliographies we read ## Summary Open-domain question answering (OpenQA) systems pair a retriever with a reader. Learned retrievers such as DPR, ORQA and REALM compress each passage and question into one vector. The authors argue this is too coarse for complex questions, and that current supervision is costly or brittle. Some methods need hand-labeled positives, some rely on BM25 sampling, and some re-index the corpus repeatedly or freeze the document encoder. The paper presents ColBERT-QA, which adapts the ColBERT late-interaction retriever to OpenQA. ColBERT compares every question token embedding with every passage token embedding through MaxSim, yet still scales to millions of passages. Training uses relevance-guided supervision (RGS). A weak retriever, initially BM25, fetches top-k passages for each training question. A simple heuristic labels them as positives or negatives, and these labels train a stronger retriever. The cycle runs two to three times, needing only one or two corpus re-indexings and keeping the document encoder trainable. On Natural Questions, TriviaQA and SQuAD, ColBERT-QA achieves the best retrieval and extractive OpenQA results reported at the time. This shows that richer interaction and cheap self-generated supervision both pay off. ## Key points - RGS raises Success@20 by up to 2.3 points for ColBERT and up to 3.2 points for a single-vector retriever. Overall, ColBERT-QA beats prior retrievers by more than five Success@20 points. - The third round, ColBERT-QA 3, is significantly better than round 1 under a Wilcoxon signed-rank test with Bonferroni correction. Rounds 2 and 3 show no significant difference. - End-to-end exact match with a BERT-large (WWM) reader: - NQ: 47.8 - TriviaQA: 70.1 - SQuAD: 54.7 on the Dec 2018 Wikipedia dump and 58.7 on the 2016 dump These results surpass DPR, REALM and the generative RAG (BART-large). - With a BERT-base reader, ColBERT-QA scores 42.3 EM on NQ, 64.6 on TriviaQA and 47.7 on SQuAD. It beats all BERT-base systems and also outperforms RAG on TriviaQA. - Code is released at github.com/stanfordnlp/ColBERT-QA. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 310 works · 284,480 citations 📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 523 works 🧭 Topics (1): [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 145 works ## Cite Cite as: Omar Khattab, Christopher Potts and Matei Zaharia. 2021. Relevance-guided Supervision for OpenQA with ColBERT. Transactions of the Association for Computational Linguistics. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @article{khattab2021relevance, title = {{Relevance-guided Supervision for OpenQA with ColBERT}}, author = {Omar Khattab, Christopher Potts and Matei Zaharia}, journal = {Transactions of the Association for Computational Linguistics}, year = {2021}, doi = {10.1162/tacl\_a\_00405}, url = {https://doi.org/10.1162/tacl_a_00405} } ``` - [← previous Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task](https://md.research.elamur.ai/free/en/papers/tang-2021-improving-speech-translation-understanding-learning/) - [next → Intrinsic Bias Metrics Do Not Correlate with Application Bias](https://md.research.elamur.ai/free/en/papers/tarrant-2021-intrinsic-bias-metrics-do-not/) - [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) ## References 3 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Zeynep Akkalyoncu Yilmaz, Wei Yang, Haotian Zhang, and Jimmy Lin. 2019. Cross-domain modeling of sentence-level evidence for document retrieval. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Pr… [find](https://scholar.google.com/scholar?q=Zeynep+Akkalyoncu+Yilmaz%2C+Wei+Yang%2C+Haotian+Zhang%2C+and+Jimmy+Lin%2E+2019%2E+Cross%2Ddomain+modeling+of+sentence%2Dlevel+evidence+for+document+retrieval%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Meth) 2. Akari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi, Richard Socher, and Caiming Xiong. 2020. [Learning to retrieve reasoning paths over wikipedia graph for question answering](https://md.research.elamur.ai/free/en/papers/asai-2019-learning-retrieve-reasoning-paths-over/). 3. ICLR 2020. \#\# Conclusion Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. [Semantic parsing on Freebase from question-answer pairs](https://md.research.elamur.ai/free/en/papers/berant-2013-semantic-parsing-freebase-question-answer/). In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533–1544, Seattle, Washington, USA. Association for Co… *Updated 24 October 2026*