---
title: "NEAT-IR — paper by Sukherman et al. (2026)"
description: "The authors define 26 classical features (statistical scoring such as BM25, TF-IDF and IDF aggregates, plus lexical overlap… 24 references."
authors: ["Lev Sukherman", "Artem Frenk", "Nina Klimenkova", "Connor Jason"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)"
source: "https://aclanthology.org/2026.acl-srw.21/"
doi: "10.18653/v1/2026.acl-srw.21"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sukherman-2026-neat-ir-neural-explainable-analysis/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sukherman-2026-neat-ir-neural-explainable-analysis/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/43/) / NEAT-IR: Neural Explainable Analysis Tool for Information Retrieval

# NEAT-IR: Neural Explainable Analysis Tool for Information Retrieval

**Authors:** Lev Sukherman, Artem Frenk, Nina Klimenkova and Connor Jason · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-srw.21/) · [PDF](https://aclanthology.org/2026.acl-srw.21.pdf) · [DOI 10.18653/v1/2026.acl-srw.21](https://doi.org/10.18653/v1/2026.acl-srw.21) · [OpenAlex](https://openalex.org/W7166805891)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary

NEAT-IR treats ColBERT as a black box and asks how much of its behaviour can be reproduced using familiar IR signals, without inspecting the network itself. The authors define 26 classical features (statistical scoring such as BM25, TF-IDF and IDF aggregates, plus lexical overlap, term positions and document properties). From these they train surrogate models that either regress ColBERT v2.0 scores or learn its per-query ordering. The data are 48,250 query–passage pairs from MS MARCO v1.1.

The central observation, which the authors call a "score–rank gap", is a strong mismatch between two outcomes:

* **Ordering:** the surrogates almost perfectly reproduce which passages ColBERT puts on top.
* **Scores:** they account for only about a quarter of the variance in raw scores.

Attribution analysis explains the split. Query-level signals mainly set the score scale, while document–query matching features decide the order within a query.

The work matters for model auditing and for designing hybrid classical-plus-neural pipelines. The authors caution that MS MARCO's lexical bias and its tiny 5-passage candidate pools probably inflate the result.

## Key points

- **Data and setup:** MS MARCO v1.1 with 9,650 queries × 5 passages, using ColBERT v2.0 outputs as targets. Evaluation uses 5-fold GroupKFold grouped by query_id.
- **Features:** 26 classical features in four groups (11 statistical, 9 lexical, 5 positional, 4 document properties). Elbow-based selection keeps 10 of them and still reaches R² = 0.273.
- **Score regression:** Random Forest reaches R² ≈ 0.278 and XGBoost ≈ 0.262 (MSE ≈ 6.2). Per-query rank agreement is moderate (τ = 0.38, ρ = 0.44).
- **Rank preservation:** the XGBoost Ranker attains NDCG@5 = 0.9914, compared with roughly 0.59 for a random ranker.
- **Feature attribution (SHAP):**
  - The regressor depends most on avg_query_idf and query_length (both ≈ 0.45).
  - The ranker relies mainly on cosine_similarity_tfidf (0.09) and idf_sum_matched (0.07).
  - The authors propose that ColBERT may adjust scores for query specificity.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Worcester Polytechnic Institute](https://md.research.elamur.ai/free/en/universities/worcester-polytechnic-institute/) — University, Worcester, United States · 12 works · 380 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works

🧭 Topics (1): [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 552 works

## Cite

Cite as: Lev Sukherman, Artem Frenk, Nina Klimenkova and Connor Jason. 2026. NEAT-IR: Neural Explainable Analysis Tool for Information Retrieval. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{sukherman2026neat,
  title = {{NEAT-IR: Neural Explainable Analysis Tool for Information Retrieval}},
  author = {Lev Sukherman, Artem Frenk, Nina Klimenkova and Connor Jason},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-srw.21},
  url = {https://doi.org/10.18653/v1/2026.acl-srw.21}
}
```

- [← previous Near-Miss: Latent Policy Failure Detection in Agentic Workflows](https://md.research.elamur.ai/free/en/papers/rabinovich-2026-near-miss-latent-policy-failure/)
- [next → NeedleChain: Measuring Intact Context Comprehension Capability of Large Language Models](https://md.research.elamur.ai/free/en/papers/moon-2026-needlechain-measuring-intact-context-comprehension/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/43/)

## References

24 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Avishek Anand, Lijun Lyu, Maximilian Idahl, Yumeng Wang, Jonas Wallat, and Zijian Zhang. 2022. Explainable information retrieval: A survey. arXiv preprint arXiv:2211.02405. — [arXiv:2211.02405](https://arxiv.org/abs/2211.02405) [find](https://scholar.google.com/scholar?q=Avishek+Anand%2C+Lijun+Lyu%2C+Maximilian+Idahl%2C+Yumeng+Wang%2C+Jonas+Wallat%2C+and+Zijian+Zhang%2E+2022%2E+Explainable+information+retrieval%3A+A+survey%2E+arXiv+preprint+arXiv%3A2211%2E02405%2E+%E2%80%94+%5BarXiv%3A2211%2E02405%5D%28https%3A)
2. Avishek Anand, Procheta Sen, Sourav Saha, Manisha Verma, and Mandar Mitra. 2023. Explainable information retrieval. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 3448– 3451. [find](https://scholar.google.com/scholar?q=Avishek+Anand%2C+Procheta+Sen%2C+Sourav+Saha%2C+Manisha+Verma%2C+and+Mandar+Mitra%2E+2023%2E+Explainable+information+retrieval%2E+In+Proceedings+of+the+46th+International+ACM+SIGIR+Conference+on+Research+and+Develo)
3. Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M. Voorhees. 2020. [Overview of the TREC 2019 deep learning track](https://md.research.elamur.ai/free/en/papers/craswell-2020-overview-trec-2019-deep-learning/). In Proceedings of the 28th Text REtrieval Conference (TREC 2019).
4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
5. Finale Doshi-Velez and Been Kim. 2017. [Towards a rigorous science of interpretable machine learning](https://md.research.elamur.ai/free/en/papers/doshi-velez-2017-towards-rigorous-science-interpretable-machine/). arXiv preprint arXiv:1702.08608.
6. Zeon Trevor Fernando, Jaspreet Singh, and Avishek Anand. 2019. A study on the interpretability of neural retrieval models using DeepSHAP. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1005–1008. [find](https://scholar.google.com/scholar?q=Zeon+Trevor+Fernando%2C+Jaspreet+Singh%2C+and+Avishek+Anand%2E+2019%2E+A+study+on+the+interpretability+of+neural+retrieval+models+using+DeepSHAP%2E+In+Proceedings+of+the+42nd+International+ACM+SIGIR+Conference+)
7. Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. 2021. A white box analysis of ColBERT. In Proceedings of the 43rd European Conference on Information Retrieval (ECIR), pages 257–263. [find](https://scholar.google.com/scholar?q=Thibault+Formal%2C+Benjamin+Piwowarski%2C+and+St%C3%A9phane+Clinchant%2E+2021%2E+A+white+box+analysis+of+ColBERT%2E+In+Proceedings+of+the+43rd+European+Conference+on+Information+Retrieval+%28ECIR%29%2C+pages+257%E2%80%93263%2E)
8. Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. 2022. Match your words! a study of lexical matching in neural information retrieval. In Proceedings of the 44th European Conference on Information Retrieval (ECIR), pages 120–127. [find](https://scholar.google.com/scholar?q=Thibault+Formal%2C+Benjamin+Piwowarski%2C+and+St%C3%A9phane+Clinchant%2E+2022%2E+Match+your+words%21+a+study+of+lexical+matching+in+neural+information+retrieval%2E+In+Proceedings+of+the+44th+European+Conference+on+Inf)
9. Luyu Gao, Zhuyun Dai, Tongfei Chen, Zhen Fan, Benjamin Van Durme, and Jamie Callan. 2021. Complement lexical retrieval model with semantic residual embeddings. In Proceedings of the 43rd European Conference on Information Retrieval (ECIR), pages 146–160. [find](https://scholar.google.com/scholar?q=Luyu+Gao%2C+Zhuyun+Dai%2C+Tongfei+Chen%2C+Zhen+Fan%2C+Benjamin+Van+Durme%2C+and+Jamie+Callan%2E+2021%2E+Complement+lexical+retrieval+model+with+semantic+residual+embeddings%2E+In+Proceedings+of+the+43rd+European+Conf)
10. Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. [Dense passage retrieval for open-domain question answering](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781.
11. Omar Khattab and Matei Zaharia. 2020. [Colbert: Efficient and effective passage search via contextualized late interaction over BERT](https://md.research.elamur.ai/free/en/papers/khattab-2020-colbert-efficient-effective-passage/). In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 39–48. lightweight late interaction. In Proce…
12. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. [BERT rediscovers the classical NLP pipeline](https://md.research.elamur.ai/free/en/papers/tenney-2019-bert-rediscovers-classical-nlp-pipeline/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593– 4601.
13. Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Proceedings of the NeurIPS 2021 Datasets and Benchmarks Track. [find](https://scholar.google.com/scholar?q=Nandan+Thakur%2C+Nils+Reimers%2C+Andreas+R%C3%BCckl%C3%A9%2C+Abhishek+Srivastava%2C+and+Iryna+Gurevych%2E+2021%2E+BEIR%3A+A+heterogeneous+benchmark+for+zero%2Dshot+evaluation+of+information+retrieval+models%2E+In+Proceedings+of+)
14. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems, volume 30, pages 5998–6008.
15. Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. 2021. Pretrained Transformers for Text Ranking: BERT and Beyond. Synthesis Lectures on Human Language Technologies. Springer Nature. [find](https://scholar.google.com/scholar?q=Jimmy+Lin%2C+Rodrigo+Nogueira%2C+and+Andrew+Yates%2E+2021%2E+Pretrained+Transformers+for+Text+Ranking%3A+BERT+and+Beyond%2E+Synthesis+Lectures+on+Human+Language+Technologies%2E+Springer+Nature%2E)
16. Puxuan Yu, Daniel Cohen, Hemank Lamba, Joel R. Tetreault, and Alejandro Jaimes. 2025. Explain then rank: Scale calibration of neural rankers using natural language explanations from LLMs. In Findings of the Association for Computational Linguistics: ACL 2025, pages 22716–22730. [find](https://scholar.google.com/scholar?q=Puxuan+Yu%2C+Daniel+Cohen%2C+Hemank+Lamba%2C+Joel+R%2E+Tetreault%2C+and+Alejandro+Jaimes%2E+2025%2E+Explain+then+rank%3A+Scale+calibration+of+neural+rankers+using+natural+language+explanations+from+LLMs%2E+In+Findings+)
17. Scott M Lundberg and Su-In Lee. 2017. [A unified approach to interpreting model predictions](https://md.research.elamur.ai/free/en/papers/lundberg-2017-unified-approach-interpreting-model-predictions/). In Advances in Neural Information Processing Systems, volume 30, pages 4765–4774.
18. Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. 2015. Object detectors emerge in deep scene CNNs. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Bolei+Zhou%2C+Aditya+Khosla%2C+Agata+Lapedriza%2C+Aude+Oliva%2C+and+Antonio+Torralba%2E+2015%2E+Object+detectors+emerge+in+deep+scene+CNNs%2E+In+International+Conference+on+Learning+Representations%2E)
19. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. [MS MARCO: A human generated machine reading comprehension dataset](https://md.research.elamur.ai/free/en/papers/bajaj-2016-ms-marco-human-generated/). arXiv preprint arXiv:1611.09268.
20. Saran Pandian, Debasis Ganguly, and Sean MacAvaney. 2024. Evaluating the explainability of neural rankers. In Proceedings of the 46th European Conference on Information Retrieval (ECIR), pages 369–383. [find](https://scholar.google.com/scholar?q=Saran+Pandian%2C+Debasis+Ganguly%2C+and+Sean+MacAvaney%2E+2024%2E+Evaluating+the+explainability+of+neural+rankers%2E+In+Proceedings+of+the+46th+European+Conference+on+Information+Retrieval+%28ECIR%29%2C+pages+369%E2%80%93383)
21. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. [“why should i trust you?” explaining the predictions of any classifier](https://md.research.elamur.ai/free/en/papers/ribeiro-2016-why-should-i-trust-explaining/). In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1135–1144.
22. Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389. [find](https://scholar.google.com/scholar?q=Stephen+Robertson+and+Hugo+Zaragoza%2E+2009%2E+The+probabilistic+relevance+framework%3A+BM25+and+beyond%2E+Foundations+and+Trends%C2%AE+in+Information+Retrieval%2C+3%284%29%3A333%E2%80%93389%2E)
23. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. [A primer in BERTology: What we know about how BERT works](https://md.research.elamur.ai/free/en/papers/rogers-2020-primer-bertology-what-we-know/). Transactions of the Association for Computational Linguistics, 8:842–866.
24. Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. 2022. Col-BERTv2: Effective and efficient retrieval via [find](https://scholar.google.com/scholar?q=Keshav+Santhanam%2C+Omar+Khattab%2C+Jon+Saad%2DFalcon%2C+Christopher+Potts%2C+and+Matei+Zaharia%2E+2022%2E+Col%2DBERTv2%3A+Effective+and+efficient+retrieval+via)

*Updated 1 November 2026*