---
title: "Rerank Before You Reason: Analyzing Reranking Tradeoffs"
description: "Scaling this test-time computation is costly, and the role of reranking in such pipelines… 55.97% accuracy. 16 references, cited by 1 paper here."
authors: ["Sahel Sharifymoghaddam", "Jimmy Lin"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.1289/"
doi: "10.18653/v1/2026.findings-acl.1289"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sharifymoghaddam-2026-rerank-before-you-reason-analyzing/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sharifymoghaddam-2026-rerank-before-you-reason-analyzing/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/9/) / Rerank Before You Reason: Analyzing Reranking Tradeoffs through Effective Token Cost in Deep Search Agents

# Rerank Before You Reason: Analyzing Reranking Tradeoffs through Effective Token Cost in Deep Search Agents

**Authors:** [Sahel Sharifymoghaddam](https://md.research.elamur.ai/free/en/authors/sahel-sharifymoghaddam/) and [Jimmy Lin](https://md.research.elamur.ai/free/en/authors/jimmy-lin/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.1289/) · [PDF](https://aclanthology.org/2026.findings-acl.1289.pdf) · [DOI 10.18653/v1/2026.findings-acl.1289](https://doi.org/10.18653/v1/2026.findings-acl.1289) · [OpenAlex](https://openalex.org/W7166832699)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 149,439 papers whose bibliographies we read

## Summary
Deep search agents answer hard multi-hop questions by alternating between issuing queries and reasoning. Scaling this test-time computation is costly, and the role of reranking in such pipelines has received little study. The authors use the BrowseComp-Plus benchmark, which runs agents against a fixed, human-verified corpus, to measure how reranking interacts with model size and reasoning effort. They pair gpt-oss-20b and gpt-oss-120b agents with the qwen3-embedding-8b retriever and insert zero-shot listwise reranking at depths of 10, 20 or 50 candidates. They also propose Effective Token Cost (ETC), which weights non-cached input, cached input and output tokens differently to reflect hardware or API pricing. Reranking improved both retrieval and final answer accuracy in every configuration tested. Moderate reranking combined with medium reasoning often matched or exceeded high-reasoning agents without reranking, while costing fewer effective tokens and less time. The results suggest that reranking is a cheaper way to improve deep research systems than adding more search-time reasoning.

## Key points
- **Setup:** The experiments cover all 830 BrowseComp-Plus queries. The agents run at low, medium and high reasoning effort, with output caps of 2k, 8k and 16k tokens. Reranking uses RankLLM with a window of 20, plus a sliding window with stride 10 at d = 50. In every case the agent receives only the top-5 documents, each truncated to 512 tokens.
- **ETC metric:** ETC = Input_nc + α·Input_c + β·Output. The cache discount α was varied over {0.1, 0.3, 0.5} and the output premium β over {3, 5, 7}. A further term, γ, extends the metric to setups where the reranker is a different model from the agent.
- **One-shot reranking:** Reranking raised NDCG@5 from 19.72 for the retriever alone to 46.05 with oss-120b at medium effort and d = 50. Increasing the depth d helped more than increasing reasoning effort. Once normalized by ETC, low-effort reranking was the most efficient choice and was adopted for the full pipeline.
- **End-to-end results:**
  - oss-20b at high effort improved from 41.38% to 55.97% accuracy at d = 50.
  - oss-120b at high effort improved from 51.20% to 63.35%.
  - oss-120b at medium effort with d = 50 reached 57.97%, which beats high reasoning without reranking.
- **Heterogeneous setup:** The cross-encoder qwen3-reranker-0.6b (γ = 0.32 relative to gpt-oss-20b) kept most of the accuracy gains at lower ETC. For example, oss-20b at high effort and d = 50 reached 52.91% accuracy, compared with 55.97% using listwise reranking.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Waterloo](https://md.research.elamur.ai/free/en/universities/university-of-waterloo/) — University, Waterloo, Canada · 80 works · 15,030 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works

🧭 Topics (2): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works · [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 552 works

## Cite

Cite as: Sahel Sharifymoghaddam and Jimmy Lin. 2026. Rerank Before You Reason: Analyzing Reranking Tradeoffs through Effective Token Cost in Deep Search Agents. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{sharifymoghaddam2026rerank,
  title = {{Rerank Before You Reason: Analyzing Reranking Tradeoffs through Effective Token Cost in Deep Search Agents}},
  author = {Sahel Sharifymoghaddam and Jimmy Lin},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1289},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1289}
}
```

- [← previous Remedy-R: Generative Reasoning for Machine Translation Evaluation without Error Annotations](https://md.research.elamur.ai/free/en/papers/tan-2026-remedy-r-generative-reasoning-machine/)
- [next → Retrieving to Recover: Towards Incomplete Audio-Visual Question Answering via Semantic-consistent Purification](https://md.research.elamur.ai/free/en/papers/zhang-2026-retrieving-recover-incomplete-audio-visual/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/9/)

## References

16 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. [Self-RAG: Learning to retrieve, generate, and critique through self-reflection](https://md.research.elamur.ai/free/en/papers/asai-2023-self-rag-learning-retrieve-generate/). In Proceedings of the 12th International Conference on Learning Representations (ICLR 2024), Vienna, Austria.
2. Zijian Chen, Xueguang Ma, Shengyao Zhuang, Ping Nie, Kai Zou, Andrew Liu, Joshua Green, Kshama Patel, Ruoxi Meng, Mingyi Su, Sahel Sharifymoghaddam, Yanxi Li, Haoran Hong, Xinyu Shi, Xuye Liu, Nandan Thakur, Crystina Zhang, Luyu Gao, Wenhu Chen, and Jimmy Lin. 2025. BrowseComp-Plus: A more fair and… — [arXiv:2508.06600](https://arxiv.org/abs/2508.06600) [find](https://scholar.google.com/scholar?q=Zijian+Chen%2C+Xueguang+Ma%2C+Shengyao+Zhuang%2C+Ping+Nie%2C+Kai+Zou%2C+Andrew+Liu%2C+Joshua+Green%2C+Kshama+Patel%2C+Ruoxi+Meng%2C+Mingyi+Su%2C+Sahel+Sharifymoghaddam%2C+Yanxi+Li%2C+Haoran+Hong%2C+Xinyu+Shi%2C+Xuye+Liu%2C+Nandan+)
3. In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. 2024. [Prompt cache: Modular attention reuse for low-latency inference](https://md.research.elamur.ai/free/en/papers/gim-2023-prompt-cache-modular-attention-reuse/). In Proceedings of the 6th Annual Conference on Machine Learning and Systems (MLSys 2024), pages 325–338, Santa Clara, CA, USA.
4. Yuxuan Huang, Yihang Chen, Haozheng Zhang, Kang Li, Huichi Zhou, Meng Fang, Linyi Yang, Xiaoguang Li, Lifeng Shang, Songcen Xu, Jianye Hao, Kun Shao, and Jun Wang. 2025. Deep research agents: A systematic examination and roadmap. arXiv:2506.18096. — [arXiv:2506.18096](https://arxiv.org/abs/2506.18096) [find](https://scholar.google.com/scholar?q=Yuxuan+Huang%2C+Yihang+Chen%2C+Haozheng+Zhang%2C+Kang+Li%2C+Huichi+Zhou%2C+Meng+Fang%2C+Linyi+Yang%2C+Xiaoguang+Li%2C+Lifeng+Shang%2C+Songcen+Xu%2C+Jianye+Hao%2C+Kun+Shao%2C+and+Jun+Wang%2E+2025%2E+Deep+research+agents%3A+A+system)
5. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. [Retrieval-augmented generation for knowledge-intensive NLP tasks](https://md.research.elamur.ai/free/en/papers/lewis-2020-retrieval-augmented-generation-knowledge-intensive/). In Proceedings of the 34th Annual Conf…
6. Rodrigo Nogueira and Kyunghyun Cho. 2019. [Passage re-ranking with BERT](https://md.research.elamur.ai/free/en/papers/nogueira-2019-passage-re-ranking-bert/). arXiv:1901.04085. pretrained sequence-to-sequence model. In Findings of the association for computational linguistics: EMNLP 2020, pages 708–718.
7. Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, Adam Khoja, Ryan Kim, Richard Ren, Jason Hausenloy, Oliver Zhang, Mantas Mazeika, and 1093 others. 2025. Humanity’s last e… — [Humanity's Last Exam](https://md.research.elamur.ai/free/en/papers/phan-2025-humanity-s-last-exam/)
8. Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. [In-context retrieval-augmented language models](https://md.research.elamur.ai/free/en/papers/ram-2023-context-retrieval-augmented-language-models/). Transactions of the Association for Computational Linguistics, 11:1316–1331.
9. Sahel Sharifymoghaddam, Ronak Pradeep, Andre Slavescu, Ryan Nguyen, Andrew Xu, Zijian Chen, Yilin Zhang, Yidi Chen, Jasper Xian, and Jimmy Lin. 2025. RankLLM: A Python package for reranking with LLMs. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Inform… [find](https://scholar.google.com/scholar?q=Sahel+Sharifymoghaddam%2C+Ronak+Pradeep%2C+Andre+Slavescu%2C+Ryan+Nguyen%2C+Andrew+Xu%2C+Zijian+Chen%2C+Yilin+Zhang%2C+Yidi+Chen%2C+Jasper+Xian%2C+and+Jimmy+Lin%2E+2025%2E+RankLLM%3A+A+Python+package+for+reranking+with+LLMs%2E)
10. Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. 2023. [Is ChatGPT good at search? investigating large language models as re-ranking agents](https://md.research.elamur.ai/free/en/papers/sun-2023-chatgpt-good-search-investigating-large/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023…
11. Fali Wang, Hui Liu, Zongyu DAI, Jingying Zeng, Zhiwei Zhang, Zongyu Wu, Chen Luo, Zhen Li, Xianfeng Tang, Qi He, and Suhang Wang. 2025. AgentTTS: Large language model agent for test-time compute-optimal scaling strategy in complex tasks. In Proceedings of the 39th Annual Conference on Neural Informa… [find](https://scholar.google.com/scholar?q=Fali+Wang%2C+Hui+Liu%2C+Zongyu+DAI%2C+Jingying+Zeng%2C+Zhiwei+Zhang%2C+Zongyu+Wu%2C+Chen+Luo%2C+Zhen+Li%2C+Xianfeng+Tang%2C+Qi+He%2C+and+Suhang+Wang%2E+2025%2E+AgentTTS%3A+Large+language+model+agent+for+test%2Dtime+compute%2Doptim)
12. Yunjia Xi, Jianghao Lin, Yongzhao Xiao, Zheli Zhou, Rong Shan, Te Gao, Jiachen Zhu, Weiwen Liu, Yong Yu, and Weinan Zhang. 2025. A survey of LLM-based deep search agents: Paradigm, optimization, and challenges. arXiv:2508.05668. — [arXiv:2508.05668](https://arxiv.org/abs/2508.05668) [find](https://scholar.google.com/scholar?q=Yunjia+Xi%2C+Jianghao+Lin%2C+Yongzhao+Xiao%2C+Zheli+Zhou%2C+Rong+Shan%2C+Te+Gao%2C+Jiachen+Zhu%2C+Weiwen+Liu%2C+Yong+Yu%2C+and+Weinan+Zhang%2E+2025%2E+A+survey+of+LLM%2Dbased+deep+search+agents%3A+Paradigm%2C+optimization%2C+and+c)
13. Yuan-An Xiao, Pengfei Gao, Chao Peng, and Yingfei Xiong. 2025. Improving the efficiency of LLM agent systems through trajectory reduction. arXiv:2509.23586. — [arXiv:2509.23586](https://arxiv.org/abs/2509.23586) [find](https://scholar.google.com/scholar?q=Yuan%2DAn+Xiao%2C+Pengfei+Gao%2C+Chao+Peng%2C+and+Yingfei+Xiong%2E+2025%2E+Improving+the+efficiency+of+LLM+agent+systems+through+trajectory+reduction%2E+arXiv%3A2509%2E23586%2E+%E2%80%94+%5BarXiv%3A2509%2E23586%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F)
14. Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, and 1 others. 2025. [Qwen3 embedding: Advancing text embedding and reranking through foundation models](https://md.research.elamur.ai/free/en/papers/zhang-2025-qwen3-embedding-advancing-text/). arXiv:2506.05176.
15. Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020. Document ranking with a *Table 5: Hardware configuration for (a) reranking-only and (b) end-to-end deep research experiments.* Reranking-only Experiments Model Reasoning GPU Type oss-20b oss-120b Low, Medium Low, Medium H200 H200 \\… [find](https://scholar.google.com/scholar?q=Rodrigo+Nogueira%2C+Zhiying+Jiang%2C+Ronak+Pradeep%2C+and+Jimmy+Lin%2E+2020%2E+Document+ranking+with+a+%2ATable+5%3A+Hardware+configuration+for+%28a%29+reranking%2Donly+and+%28b%29+end%2Dto%2Dend+deep+research+experiments%2E%2A+Rera)
16. (a) End-to-end Deep Research Experiments Model Reasoning Reranking GPU Type oss-20b oss-20b oss-120b oss-120b low, med, high low, med, high low, med, high low, med, high N/A (d = 0) listwise (d \> 0) N/A (d = 0) listwise (d \> 0) H200 H200 H200 H200 \\\#GPUs 1 2 1 2 oss-20b low, med, high cross-encoder… [find](https://scholar.google.com/scholar?q=%28a%29+End%2Dto%2Dend+Deep+Research+Experiments+Model+Reasoning+Reranking+GPU+Type+oss%2D20b+oss%2D20b+oss%2D120b+oss%2D120b+low%2C+med%2C+high+low%2C+med%2C+high+low%2C+med%2C+high+low%2C+med%2C+high+N%2FA+%28d+%3D+0%29+listwise+%28d+%5C%3E+0%29+)

*Updated 2 November 2026*