---
title: "BrowseComp — paper by Wei et al. (2025) — summary and ideas"
description: "OpenAI introduces BrowseComp (short for \"Browsing Competition\"), a set of 1,266 fact-seeking… 24 references, cited by 112 papers here."
authors: ["Jason Wei", "Zhiqing Sun", "Spencer Papay", "Scott McKinney", "Jeffrey Han", "Isa Fulford", "Hyung Won Chung", "Alex Tachard Passos", "William Fedus", "Amelia Glaese"]
year: 2025
venue: "arXiv:2504.12516"
source: "https://arxiv.org/abs/2504.12516"
doi: "10.48550/arxiv.2504.12516"
arxiv: "2504.12516"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wei-2025-browsecomp-simple-yet-challenging-benchmark/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wei-2025-browsecomp-simple-yet-challenging-benchmark/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/) / BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents

# BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents

**Authors:** [Jason Wei](https://md.research.elamur.ai/free/en/authors/jason-wei/), [Zhiqing Sun](https://md.research.elamur.ai/free/en/authors/zhiqing-sun/), [Spencer Papay](https://md.research.elamur.ai/free/en/authors/spencer-papay/), Scott McKinney, Jeffrey Han, Isa Fulford, [Hyung Won Chung](https://md.research.elamur.ai/free/en/authors/hyung-won-chung/), Alex Tachard Passos, [William Fedus](https://md.research.elamur.ai/free/en/authors/william-fedus/) and [Amelia Glaese](https://md.research.elamur.ai/free/en/authors/amelia-glaese/) · **Year:** 2025 · **Venue:** arXiv:2504.12516 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2504.12516) · [PDF](https://arxiv.org/pdf/2504.12516) · [DOI 10.48550/arxiv.2504.12516](https://doi.org/10.48550/arxiv.2504.12516) · [arXiv 2504.12516](https://arxiv.org/abs/2504.12516) · [OpenAlex](https://openalex.org/W4417089504)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [112](https://md.research.elamur.ai/free/en/papers/wei-2025-browsecomp-simple-yet-challenging-benchmark/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Most earlier retrieval and question-answering benchmarks ask for facts a person could find quickly, so current language models have mostly maxed them out. OpenAI introduces BrowseComp (short for "Browsing Competition"), a set of 1,266 fact-seeking questions. It tests whether an agent can keep searching the web, rephrase its queries in creative ways, and piece together clues spread across many pages. Human trainers built each question backwards. They started from a known fact and added several constraints that each cover a huge search space. The result is a short answer that is easy to check but very expensive to find. Every question had to stump GPT-4o, o1 and an early Deep Research model. A model-based grader checks whether a predicted answer means the same as the reference string. Human trainers working without AI assistants gave up on about seven questions in ten after two hours. Deep Research's accuracy rises steadily as it gets more test-time compute, and it rises further with parallel sampling. The benchmark is deliberately narrow: it does not test long answers or how ambiguous requests are handled. Still, it gives a cheap and reliable measure of how persistently an agent can browse.

## Key points
- **Dataset:** 1,266 short-answer questions, released in the `openai/simple-evals` repository with a canary string so the data can be filtered out of training corpora. The largest topics are TV shows & movies (205), science & technology (173), art, history and sports. The set started with 1,287 questions. The authors reviewed the 118 that Deep Research never solved and removed 21 with flawed answers.
- **Difficulty checks during creation:** each question had to defeat GPT-4o (with and without browsing), o1 and early Deep Research. The answer could not appear on the first results page of five Google searches. If a second trainer solved a writer's questions more than 40% of the time, the writer had to revise them.
- **Human baseline:** trainers attempted 1,255 questions. They solved 367 (29.2%) and gave up on 70.8% after two hours. When they did find an answer, it matched the reference 86.4% of the time.
- **Using more compute:** Deep Research produced 64 samples per question. Majority voting, confidence-weighted voting and best-of-N all scored 15–25% higher than a single attempt, and best-of-N did best. This suggests the model can often tell when its answer is right, even though its confidence scores are poorly calibrated.
- **How hard individual tasks are:** over 64 trials, Deep Research solved 16% of tasks every time and 14% never. When the model was given the correct answer for the never-solved tasks, it could usually find supporting evidence on the web. So those tasks are very hard, not impossible.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works

🧭 Topics (1): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works

## Cite

Cite as: Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus and Amelia Glaese. 2025. BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents. arXiv:2504.12516.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{wei2025browsecomp,
  title = {{BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents}},
  author = {Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus and Amelia Glaese},
  year = {2025},
  doi = {10.48550/arxiv.2504.12516},
  eprint = {2504.12516},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2504.12516}
}
```

- [← previous LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models](https://md.research.elamur.ai/free/en/papers/zhang-2025-lmms-eval-reality-check-evaluation/)
- [next → DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments](https://md.research.elamur.ai/free/en/papers/zheng-2025-deepresearcher-scaling-deep-research-reinforcement/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/)

## References

24 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. E. Dinan, S. Roller, K. Shuster, A. Fan, M. Auli, and J. Weston. [Wizard of wikipedia: Knowledge-powered conversational agents](https://md.research.elamur.ai/free/en/papers/dinan-2018-wizard-wikipedia-knowledge-powered-conversational/). ICLR, 2019.
2. A. Fan, Y. Jernite, E. Perez, D. Grangier, J. Weston, and M. Auli. [ELI5: Long form question answering](https://md.research.elamur.ai/free/en/papers/fan-2019-eli5-long-form-question-answering/). In ACL, 2019.
3. D. Ferrucci, E. Brown, J. Chu-Carroll, J. Fan, D. Gondek, A. Kalyanpur, A. Lally, J. W. Murdock, E. Nyberg, J. Prager, N. Schlaefer, and C. Welty. Building Watson: An overview of the DeepQA project. AI Magazine, 2010. [find](https://scholar.google.com/scholar?q=D%2E+Ferrucci%2C+E%2E+Brown%2C+J%2E+Chu%2DCarroll%2C+J%2E+Fan%2C+D%2E+Gondek%2C+A%2E+Kalyanpur%2C+A%2E+Lally%2C+J%2E+W%2E+Murdock%2C+E%2E+Nyberg%2C+J%2E+Prager%2C+N%2E+Schlaefer%2C+and+C%2E+Welty%2E+Building+Watson%3A+An+overview+of+the+DeepQA+project%2E+A)
4. Google. Try Deep Research and our new experimental model in Gemini, your AI assistant. Google blog, 2024. [find](https://scholar.google.com/scholar?q=Google%2E+Try+Deep+Research+and+our+new+experimental+model+in+Gemini%2C+your+AI+assistant%2E+Google+blog%2C+2024%2E)
5. K. Guu, K. Lee, Z. Tung, P. Pasupat, and M.-W. Chang. [REALM: Retrieval-augmented language model pre-training](https://md.research.elamur.ai/free/en/papers/guu-2020-realm-retrieval-augmented-language-model/). In ICML, 2020.
6. M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. [TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/). In ACL, 2017.
7. K. Lee, M.-W. Chang, and K. Toutanova. [Latent retrieval for weakly supervised open domain question answering](https://md.research.elamur.ai/free/en/papers/lee-2019-latent-retrieval-weakly-supervised-open/). In ACL, 2019.
8. P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. tau Yih, T. Rocktäschel, S. Riedel, and D. Kiela. [Retrieval-augmented generation for knowledge-intensive NLP tasks](https://md.research.elamur.ai/free/en/papers/lewis-2020-retrieval-augmented-generation-knowledge-intensive/). In NeurIPS, 2020.
9. J. Menick, T. Miller, T. Ribeiro, L. G. Brigato, A. Bakhtin, P. Chatelain, M. Moore, J. Kramár, A. Joulin, K. Shuster, P. Stenetorp, and D. Kiela. [Teaching language models to support answers with verified quotes](https://md.research.elamur.ai/free/en/papers/menick-2022-teaching-language-models-support-answers/). arXiv preprint arXiv:2203.11147, 2022.
10. G. Mialon, C. Fourrier, T. Wolf, Y. LeCun, and T. Scialom. [GAIA: a benchmark for general AI assistants](https://md.research.elamur.ai/free/en/papers/mialon-2023-gaia-benchmark-general-ai-assistants/). In ICLR, 2023.
11. R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, X. Jiang, K. Cobbe, T. Eloundou, G. Krueger, K. Button, M. Knight, B. Chess, and J. Schulman. [WebGPT: Browser-assisted question-answering with human feedback](https://md.research.elamur.ai/free/en/papers/nakano-2021-webgpt-browser-assisted-question-answering/). arXiv, 2021.
12. OpenAI. Introducing deep research. OpenAI blog, 2025a. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Introducing+deep+research%2E+OpenAI+blog%2C+2025a%2E)
13. OpenAI. Introducing operator. OpenAI blog, 2025b. perplexity.AI. Introducing perplexity deep research. perplexity.AI blog, 2025. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Introducing+operator%2E+OpenAI+blog%2C+2025b%2E+perplexity%2EAI%2E+Introducing+perplexity+deep+research%2E+perplexity%2EAI+blog%2C+2025%2E)
14. F. Petroni, A. Piktus, A. Fan, P. Lewis, M. Yazdani, N. De Cao, J. Thorne, Y. Jernite, V. Karpukhin, J. Maillard, V. Plachouras, T. Rocktäschel, and S. Riedel. [KILT: a benchmark for knowledge intensive language tasks](https://md.research.elamur.ai/free/en/papers/petroni-2020-kilt-benchmark-knowledge-intensive-language/). In NAACL, 2021.
15. L. Phan, A. Gatti, Z. Han, N. Li, J. Hu, H. Zhang, C. B. C. Zhang, M. Shaaban, J. Ling, S. Shi, et al. Humanity’s last exam. arXiv, 2025. — [Humanity's Last Exam](https://md.research.elamur.ai/free/en/papers/phan-2025-humanity-s-last-exam/)
16. D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman. [GPQA: A graduate-level google-proof Q&A benchmark](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/). In COLM, 2024.
17. T. Schick, J. Dwivedi-Yu, R. Dessı̀, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom. [Toolformer: Language models can teach themselves to use tools](https://md.research.elamur.ai/free/en/papers/schick-2023-toolformer-language-models-can-teach/). NeurIPS, 2023.
18. Y. Song, K. Thai, C. M. Pham, Y. Chang, M. Nadaf, and M. Iyyer. BEARCUBS: A benchmark for computer-using web agents. arXiv, 2025. [find](https://scholar.google.com/scholar?q=Y%2E+Song%2C+K%2E+Thai%2C+C%2E+M%2E+Pham%2C+Y%2E+Chang%2C+M%2E+Nadaf%2C+and+M%2E+Iyyer%2E+BEARCUBS%3A+A+benchmark+for+computer%2Dusing+web+agents%2E+arXiv%2C+2025%2E)
19. A. Srivastava, D. Kleyko, and Z. Wu. Beyond the imitation game: Quantifying and extrapolatingthe capabilities of language models. TMLR, 2023. [find](https://scholar.google.com/scholar?q=A%2E+Srivastava%2C+D%2E+Kleyko%2C+and+Z%2E+Wu%2E+Beyond+the+imitation+game%3A+Quantifying+and+extrapolatingthe+capabilities+of+language+models%2E+TMLR%2C+2023%2E)
20. J. Thorne, A. Vlachos, C. Christodoulopoulos, and A. Mittal. [FEVER: a large-scale dataset for fact extraction and verification](https://md.research.elamur.ai/free/en/papers/thorne-2018-fever-large-scale-dataset-fact/). In NAACL, 2018.
21. J. Wei, N. Karina, H. W. Chung, Y. J. Jiao, S. Papay, A. Glaese, J. Schulman, and W. Fedus. [Measuring short-form factuality in large language models](https://md.research.elamur.ai/free/en/papers/wei-2024-measuring-short-form-factuality-large/). arXiv, 2024. x.AI. Grok 3 beta — the age of reasoning agents. x.AI blog, 2025.
22. Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. Cohen, R. Salakhutdinov, and C. D. Manning. [HotpotQA: A dataset for diverse, explainable multi-hop question answering](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/). In EMNLP, 2018.
23. S. Yao, H. Chen, J. Yang, and K. Narasimhan. [Webshop: Towards scalable real-world web interaction with grounded language agents](https://md.research.elamur.ai/free/en/papers/yao-2022-webshop-towards-scalable-real-world/). In NeurIPS, 2022.
24. S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao. [ReAct: Synergizing reasoning and acting in language models](https://md.research.elamur.ai/free/en/papers/yao-2022-react-synergizing-reasoning-acting-language/). In ICLR, 2023.

*Updated 2 November 2026*