---
title: "Confidence-Aware Ranker Ensembles for Robust In-Context"
description: "They propose FWE-IKE, which treats LLM-, BERT- and MLP-based rankers as experts, in the spirit of mixture-of-experts. 30 references."
authors: ["Tejal Nair", "Mahmud Wasif Nafee", "Maiqi Jiang", "Ashley Gao", "Haipeng Chen", "Yanfu Zhang"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.1750/"
doi: "10.18653/v1/2026.findings-acl.1750"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/nair-2026-confidence-aware-ranker-ensembles-robust/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/nair-2026-confidence-aware-ranker-ensembles-robust/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/25/) / Confidence-Aware Ranker Ensembles for Robust In-Context Knowledge Editing

# Confidence-Aware Ranker Ensembles for Robust In-Context Knowledge Editing

**Authors:** Tejal Nair, Mahmud Wasif Nafee, Maiqi Jiang, [Ashley Gao](https://md.research.elamur.ai/free/en/authors/ashley-gao/), Haipeng Chen and [Yanfu Zhang](https://md.research.elamur.ai/free/en/authors/yanfu-zhang/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.1750/) · [PDF](https://aclanthology.org/2026.findings-acl.1750.pdf) · [DOI 10.18653/v1/2026.findings-acl.1750](https://doi.org/10.18653/v1/2026.findings-acl.1750) · [OpenAlex](https://openalex.org/W7166828673)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Large language models can keep producing outdated or wrong facts. In-context knowledge editing (IKE) addresses this at inference time by putting corrected facts and demonstrations into the prompt, without changing weights, which makes it suitable for black-box APIs. However, current IKE pipelines pick demonstrations with a single retriever whose scores are not calibrated. The authors find that different rankers surface different useful evidence, and that the most reliable one varies from query to query. They propose FWE-IKE, which treats LLM-, BERT- and MLP-based rankers as experts, in the spirit of mixture-of-experts. Each ranker's score distribution is first made comparable through top-K truncation and a temperature tuned to a target entropy. Cheap confidence features then feed a small meta-model that predicts per-query weights for fusion in logit space. When one expert clearly dominates, a cautious margin gate routes the query to it alone. On CounterFact, the ensemble outperforms every individual retriever and comes close to the oracle ceiling. This suggests adaptive ranker combination makes gradient-free editing more robust.

## Key points
- On a 300-query CounterFact evaluation, FWE-IKE reaches 88.33% Edit-Success Rate. This beats BERT (85.33%), LLM (84.00%) and MLP (82.67%) rankers, a +3.0 point gain over the best one, against a 91.00% oracle bound.
- The confidence features are normalized entropy, top-1 margin, top-1 mass and cross-retriever agreement. They are computed from each ranker's raw scores over a shared COPY/UPDATE/RETAIN candidate pool.
- In ablations, both score calibration and logit-space fusion help, and the small gate adds robustness at little cost.
- Experiments use Llama-3.2-3B because of GPU limits. Some baseline comparisons were run on reduced samples.
- Stated limitations:
  - gains depend on the experts actually disagreeing;
  - only a single benchmark is used;
  - the gate relies on shallow signals that may fail out of distribution.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Rensselaer Polytechnic Institute](https://md.research.elamur.ai/free/en/universities/rensselaer-polytechnic-institute/) — University, Troy, United States · 21 works · 4,933 citations
- [Bangladesh University of Engineering and Technology](https://md.research.elamur.ai/free/en/universities/bangladesh-university-of-engineering-and-technology/) — University, Dhaka, Bangladesh · 7 works · 308 citations
- [William & Mary](https://md.research.elamur.ai/free/en/universities/william-mary/) — University, Williamsburg, United States · 5 works · 357 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works

🧭 Topics (1): [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 473 works

## Cite

Cite as: Tejal Nair, Mahmud Wasif Nafee, Maiqi Jiang, Ashley Gao, Haipeng Chen and Yanfu Zhang. 2026. Confidence-Aware Ranker Ensembles for Robust In-Context Knowledge Editing. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{nair2026confidence,
  title = {{Confidence-Aware Ranker Ensembles for Robust In-Context Knowledge Editing}},
  author = {Tejal Nair, Mahmud Wasif Nafee, Maiqi Jiang, Ashley Gao, Haipeng Chen and Yanfu Zhang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1750},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1750}
}
```

- [← previous Confidence Leaps in LLM Reasoning: Early Stopping and Cross-Model Transfer](https://md.research.elamur.ai/free/en/papers/tikhonov-2026-confidence-leaps-llm-reasoning-early/)
- [next → Confidence-Weighted Token Set Cover for Early Hypothesis Pruning in Self-Consistency](https://md.research.elamur.ai/free/en/papers/sultan-2026-confidence-weighted-token-set-cover/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/25/)

## References

30 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Keqin Bao, Ming Yan, Yang Zhang, Jizhi Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He. 2024. Real-time personalization for llm-based recommendation with customized in-context learning. [find](https://scholar.google.com/scholar?q=Keqin+Bao%2C+Ming+Yan%2C+Yang+Zhang%2C+Jizhi+Zhang%2C+Wenjie+Wang%2C+Fuli+Feng%2C+and+Xiangnan+He%2E+2024%2E+Real%2Dtime+personalization+for+llm%2Dbased+recommendation+with+customized+in%2Dcontext+learning%2E)
2. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). Preprint, arXiv:2110.14168.
3. Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. 2023. Evaluating the ripple effects of knowledge editing in language models. [find](https://scholar.google.com/scholar?q=Roi+Cohen%2C+Eden+Biran%2C+Ori+Yoran%2C+Amir+Globerson%2C+and+Mor+Geva%2E+2023%2E+Evaluating+the+ripple+effects+of+knowledge+editing+in+language+models%2E)
4. Gordon Cormack, Charles Clarke, and Stefan Büttcher. 2009. [Reciprocal rank fusion outperforms condorcet and individual rank learning methods](https://md.research.elamur.ai/free/en/papers/cormack-2009-reciprocal-rank-fusion-outperforms/). pages 758–759.
5. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, and 181 others. 2025. Deepseek-v3 technical report. [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2C+Aixin+Liu%2C+Bei+Feng%2C+Bing+Xue%2C+Bingxuan+Wang%2C+Bochao+Wu%2C+Chengda+Lu%2C+Chenggang+Zhao%2C+Chengqi+Deng%2C+Chenyu+Zhang%2C+Chong+Ruan%2C+Damai+Dai%2C+Daya+Guo%2C+Dejian+Yang%2C+Deli+Chen%2C+Dongjie+Ji%2C+Erhan)
6. William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Preprint, arXiv:2101.03961. [find](https://scholar.google.com/scholar?q=William+Fedus%2C+Barret+Zoph%2C+and+Noam+Shazeer%2E+2022%2E+Switch+transformers%3A+Scaling+to+trillion+parameter+models+with+simple+and+efficient+sparsity%2E+Preprint%2C+arXiv%3A2101%2E03961%2E)
7. Yujie Feng, Zexin Lu, Bo Liu, Liming Zhan, and Xiao-Ming Wu. 2023. Towards llm-driven dialogue state tracking. [find](https://scholar.google.com/scholar?q=Yujie+Feng%2C+Zexin+Lu%2C+Bo+Liu%2C+Liming+Zhan%2C+and+Xiao%2DMing+Wu%2E+2023%2E+Towards+llm%2Ddriven+dialogue+state+tracking%2E)
8. Runyu He, Anyu Ying, and Xiaoyu Hu. 2024. Improving opendevin: Boosting code generation llm through advanced memory management. Applied and Computational Engineering, 68:313–320. [find](https://scholar.google.com/scholar?q=Runyu+He%2C+Anyu+Ying%2C+and+Xiaoyu+Hu%2E+2024%2E+Improving+opendevin%3A+Boosting+code+generation+llm+through+advanced+memory+management%2E+Applied+and+Computational+Engineering%2C+68%3A313%E2%80%93320%2E)
9. Robert Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton. 1991. Adaptive mixtures of local experts. Neural Computation, 3:79–87. [find](https://scholar.google.com/scholar?q=Robert+Jacobs%2C+Michael+Jordan%2C+Steven+Nowlan%2C+and+Geoffrey+Hinton%2E+1991%2E+Adaptive+mixtures+of+local+experts%2E+Neural+Computation%2C+3%3A79%E2%80%9387%2E)
10. Michael Jordan and Robert Jacobs. 1994. Hierarchical mixtures of experts and the. Neural computation, 6:181–. [find](https://scholar.google.com/scholar?q=Michael+Jordan+and+Robert+Jacobs%2E+1994%2E+Hierarchical+mixtures+of+experts+and+the%2E+Neural+computation%2C+6%3A181%E2%80%93%2E)
11. To Eun Kim and Fernando Diaz. 2025. Ltrr: Learning to rank retrievers for llms. Preprint, arXiv:2506.13743. — [arXiv:2506.13743](https://arxiv.org/abs/2506.13743) [find](https://scholar.google.com/scholar?q=To+Eun+Kim+and+Fernando+Diaz%2E+2025%2E+Ltrr%3A+Learning+to+rank+retrievers+for+llms%2E+Preprint%2C+arXiv%3A2506%2E13743%2E+%E2%80%94+%5BarXiv%3A2506%2E13743%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2506%2E13743%29)
12. Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020. Gshard: Scaling giant models with conditional computation and automatic sharding. Preprint, arXiv:2006.16668. [find](https://scholar.google.com/scholar?q=Dmitry+Lepikhin%2C+HyoukJoong+Lee%2C+Yuanzhong+Xu%2C+Dehao+Chen%2C+Orhan+Firat%2C+Yanping+Huang%2C+Maxim+Krikun%2C+Noam+Shazeer%2C+and+Zhifeng+Chen%2E+2020%2E+Gshard%3A+Scaling+giant+models+with+conditional+computation+and)
13. Xiaonan Li, Kai Lv, Hang Yan, Tianyang Lin, Wei Zhu, Yuan Ni, Guotong Xie, Xiaoling Wang, and Xipeng Qiu. 2023. [Unified demonstration retriever for in-context learning](https://md.research.elamur.ai/free/en/papers/li-2023-unified-demonstration-retriever-context-learning/).
14. Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021. [What makes good in-context examples for gpt-3?](https://md.research.elamur.ai/free/en/papers/liu-2022-what-makes-good-context-examples/)
15. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. [Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity](https://md.research.elamur.ai/free/en/papers/lu-2022-fantastically-ordered-prompts-where-find/). Preprint, arXiv:2104.08786.
16. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2023. Locating and editing factual associations in gpt. [find](https://scholar.google.com/scholar?q=Kevin+Meng%2C+David+Bau%2C+Alex+Andonian%2C+and+Yonatan+Belinkov%2E+2023%2E+Locating+and+editing+factual+associations+in+gpt%2E)
17. Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. 2023a. Mass-editing memory in a transformer. [find](https://scholar.google.com/scholar?q=Kevin+Meng%2C+Arnab+Sen+Sharma%2C+Alex+Andonian%2C+Yonatan+Belinkov%2C+and+David+Bau%2E+2023a%2E+Mass%2Dediting+memory+in+a+transformer%2E)
18. Dehai Min, Nan Hu, Rihui Jin, Nuo Lin, Jiaoyan Chen, Yongrui Chen, Yu Li, Guilin Qi, Yun Li, Nijun Li, and Qianren Wang. 2024. Exploring the impact of table-to-text methods on augmenting llm-based question answering with domain hybrid data. [find](https://scholar.google.com/scholar?q=Dehai+Min%2C+Nan+Hu%2C+Rihui+Jin%2C+Nuo+Lin%2C+Jiaoyan+Chen%2C+Yongrui+Chen%2C+Yu+Li%2C+Guilin+Qi%2C+Yun+Li%2C+Nijun+Li%2C+and+Qianren+Wang%2E+2024%2E+Exploring+the+impact+of+table%2Dto%2Dtext+methods+on+augmenting+llm%2Dbased+que)
19. Shuyang Yu, Runxue Bao, Parminder Bhatia, Taha Kass-Hout, Jiayu Zhou, and Cao Xiao. 2025. Dynamic uncertainty ranking: Enhancing retrieval-augmented in-context learning for long-tail knowledge in llms. [find](https://scholar.google.com/scholar?q=Shuyang+Yu%2C+Runxue+Bao%2C+Parminder+Bhatia%2C+Taha+Kass%2DHout%2C+Jiayu+Zhou%2C+and+Cao+Xiao%2E+2025%2E+Dynamic+uncertainty+ranking%3A+Enhancing+retrieval%2Daugmented+in%2Dcontext+learning+for+long%2Dtail+knowledge+in+llms)
20. Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. [Calibrate before use: Improving few-shot performance of language models](https://md.research.elamur.ai/free/en/papers/zhao-2021-calibrate-before-use-improving-few/). Preprint, arXiv:2102.09690.
21. Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. 2023a. Can we edit factual knowledge by in-context learning? Preprint, arXiv:2305.12740. [find](https://scholar.google.com/scholar?q=Ce+Zheng%2C+Lei+Li%2C+Qingxiu+Dong%2C+Yuxuan+Fan%2C+Zhiyong+Wu%2C+Jingjing+Xu%2C+and+Baobao+Chang%2E+2023a%2E+Can+we+edit+factual+knowledge+by+in%2Dcontext+learning%3F+Preprint%2C+arXiv%3A2305%2E12740%2E)
22. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023b. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). Preprint, arXiv:2306.05685.
23. Chenghao Zhu, Nuo Chen, Yufei Gao, Yunyi Zhang, Prayag Tiwari, and Benyou Wang. 2025. Is your llm outdated? a deep look at temporal generalization. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologi… [find](https://scholar.google.com/scholar?q=Chenghao+Zhu%2C+Nuo+Chen%2C+Yufei+Gao%2C+Yunyi+Zhang%2C+Prayag+Tiwari%2C+and+Benyou+Wang%2E+2025%2E+Is+your+llm+outdated%3F+a+deep+look+at+temporal+generalization%2E+In+Proceedings+of+the+2025+Conference+of+the+Nations)
24. Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. 2022. Fast model editing at scale. [find](https://scholar.google.com/scholar?q=Eric+Mitchell%2C+Charles+Lin%2C+Antoine+Bosselut%2C+Chelsea+Finn%2C+and+Christopher+D%2E+Manning%2E+2022%2E+Fast+model+editing+at+scale%2E)
25. Seyed Mahed Mousavi, Simone Alghisi, and Giuseppe Riccardi. 2024. Dyknow: Dynamically verifying time-sensitive factual knowledge in llms. [find](https://scholar.google.com/scholar?q=Seyed+Mahed+Mousavi%2C+Simone+Alghisi%2C+and+Giuseppe+Riccardi%2E+2024%2E+Dyknow%3A+Dynamically+verifying+time%2Dsensitive+factual+knowledge+in+llms%2E)
26. Mahmud Wasif Nafee, Maiqi Jiang, Haipeng Chen, and Yanfu Zhang. 2025. Dynamic retriever for in-context knowledge editing via policy optimization. Preprint, arXiv:2510.21059. [find](https://scholar.google.com/scholar?q=Mahmud+Wasif+Nafee%2C+Maiqi+Jiang%2C+Haipeng+Chen%2C+and+Yanfu+Zhang%2E+2025%2E+Dynamic+retriever+for+in%2Dcontext+knowledge+editing+via+policy+optimization%2E+Preprint%2C+arXiv%3A2510%2E21059%2E)
27. Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and beyond. volume 3, pages 333–389. [find](https://scholar.google.com/scholar?q=Stephen+Robertson+and+Hugo+Zaragoza%2E+2009%2E+The+probabilistic+relevance+framework%3A+Bm25+and+beyond%2E+volume+3%2C+pages+333%E2%80%93389%2E)
28. Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Lee Boyd-Graber, and Lijuan Wang. 2023. [Prompting GPT-3 to be reliable](https://md.research.elamur.ai/free/en/papers/si-2022-prompting-gpt-3-reliable/). In The Eleventh International Conference on Learning Representations.
29. Changyue Wang, Weihang Su, Qingyao Ai, Yichen Tang, and Yiqun Liu. 2025. Knowledge editing through chain-of-thought. [find](https://scholar.google.com/scholar?q=Changyue+Wang%2C+Weihang+Su%2C+Qingyao+Ai%2C+Yichen+Tang%2C+and+Yiqun+Liu%2E+2025%2E+Knowledge+editing+through+chain%2Dof%2Dthought%2E)
30. Liang Wang, Nan Yang, and Furu Wei. 2024. [Learning to retrieve in-context examples for large language models](https://md.research.elamur.ai/free/en/papers/wang-2024-learning-retrieve-context-examples-large/).

*Updated 1 November 2026*