---
title: "Evaluating Token-Level and Passage-Level Dense Retrieval"
description: "Math Information Retrieval (MIR) is difficult because formulas are highly structured. 49 references, cited by 9 papers here."
authors: ["Wei Zhong", "Jheng-Hong Yang", "Yuqing Xie", "Jimmy Lin"]
year: 2022
venue: "Findings of the Association for Computational Linguistics: EMNLP 2022"
source: "https://aclanthology.org/2022.findings-emnlp.78/"
doi: "10.18653/v1/2022.findings-emnlp.78"
arxiv: "2203.11163"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhong-2022-evaluating-token-level-passage-level/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhong-2022-evaluating-token-level-passage-level/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/6/) / Evaluating Token-Level and Passage-Level Dense Retrieval Models for Math Information Retrieval

# Evaluating Token-Level and Passage-Level Dense Retrieval Models for Math Information Retrieval

**Authors:** Wei Zhong, [Jheng-Hong Yang](https://md.research.elamur.ai/free/en/authors/jheng-hong-yang/), [Yuqing Xie](https://md.research.elamur.ai/free/en/authors/yuqing-xie/) and [Jimmy Lin](https://md.research.elamur.ai/free/en/authors/jimmy-lin/) · **Year:** 2022 · **Venue:** [Findings of the Association for Computational Linguistics: EMNLP 2022](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2022](https://md.research.elamur.ai/free/en/venues/findings/2022/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.findings-emnlp.78/) · [PDF](https://aclanthology.org/2022.findings-emnlp.78.pdf) · [DOI 10.18653/v1/2022.findings-emnlp.78](https://doi.org/10.18653/v1/2022.findings-emnlp.78) · [arXiv 2203.11163](https://arxiv.org/abs/2203.11163) · [OpenAlex](https://openalex.org/W4385573344)

**Citations:** 14 citations (OpenAlex, 2026-10-14) · [9](https://md.research.elamur.ai/free/en/papers/zhong-2022-evaluating-token-level-passage-level/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Math Information Retrieval (MIR) is difficult because formulas are highly structured. The same concept can appear in very different notation, and formulas interact with the text around them. The strongest systems still rely on hand-engineered structure matching. This paper asks whether bi-encoder dense retrievers can fill in what structure search misses. The authors adapt two representative models to the math domain through pretraining and fine-tuning: DPR, which scores whole passages, and ColBERT, which scores individual tokens. They evaluate both on NTCIR-12 Wiki-Formula and ARQMath-2. They then merge each dense run with the Approach0 structure search engine, mainly through linear score interpolation. On their own, the dense models trail cross encoders. Combined with structure search, however, they set new state-of-the-art results. The two signals help each other in different ways. Dense retrieval finds relevant documents that share no formula substructure with the query. Structure search suppresses dense hits that look similar but are mathematically different. The released checkpoints and pipeline give a useful baseline for a domain short on resources.

## Key points
- The authors describe this as the first evaluation of a DPR model for MIR. Their fine-tuned ColBERT clearly beats earlier ColBERT-style retrievers reported in this domain.
- Benchmarks: NTCIR-12 Wiki-Formula, with 20 concrete queries over roughly 591,000 indexed formulas, and ARQMath-2.
- On ARQMath-2, the Approach0 + ColBERT linear fusion reaches NDCG' 0.449, MAP' 0.217, P'@10 0.279 and BPref 0.204. Borda Count, CombSUM, ISR, log-ISR and RRF (k = 60) do no significantly better than this, as computed with the polyfuse tool.
- Per-topic analysis shows large gains from fusion:
  - Topic A.287: P@10 rises from 0 to 0.8 compared with Approach0 alone.
  - Topic A.219: P@10 rises from 0.1 to 0.5 compared with ColBERT alone.
- The stated limitations are the engineering overhead of maintaining two retrieval stacks, the lack of match highlighting for DPR, and ColBERT's heavy storage and GPU demands. Code and models are available at github.com/approach0/math-dense-retrievers.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Waterloo](https://md.research.elamur.ai/free/en/universities/university-of-waterloo/) — University, Waterloo, Canada · 85 works · 15,188 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,888 works · [🧮 Computational Theory and Mathematics](https://md.research.elamur.ai/free/en/shelves/computer-science/computational-theory-and-mathematics/) · 496 works

🧭 Topics (2): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,394 works · [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 561 works

## Cite

Cite as: Wei Zhong, Jheng-Hong Yang, Yuqing Xie and Jimmy Lin. 2022. Evaluating Token-Level and Passage-Level Dense Retrieval Models for Math Information Retrieval. Findings of the Association for Computational Linguistics: EMNLP 2022.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhong2022evaluating,
  title = {{Evaluating Token-Level and Passage-Level Dense Retrieval Models for Math Information Retrieval}},
  author = {Wei Zhong, Jheng-Hong Yang, Yuqing Xie and Jimmy Lin},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2022},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.78},
  eprint = {2203.11163},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2022.findings-emnlp.78}
}
```

- [← previous DiscoGeM: A Crowdsourced Corpus of Genre-Mixed Implicit Discourse Relations](https://md.research.elamur.ai/free/en/papers/scholman-2022-discogem-crowdsourced-corpus-genre-mixed/)
- [next → Text-based NP Enrichment](https://md.research.elamur.ai/free/en/papers/elazar-2022-text-based-np-enrichment/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/6/)

## References

49 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. [Sci-BERT: A pretrained language model for scientific text](https://md.research.elamur.ai/free/en/papers/beltagy-2019-scibert-pretrained-language-model-scientific/). arXiv:1903.10676.
2. Pankaj Dadure, Partha Pakray, and Sivaji Bandyopadhyay. 2021. BERT-based embedding model for formula retrieval. In CLEF. [find](https://scholar.google.com/scholar?q=Pankaj+Dadure%2C+Partha+Pakray%2C+and+Sivaji+Bandyopadhyay%2E+2021%2E+BERT%2Dbased+embedding+model+for+formula+retrieval%2E+In+CLEF%2E)
3. Kenny Davila and Richard Zanibbi. 2017. Layout and semantics: Combining representations for mathematical formula search. In SIGIR. [find](https://scholar.google.com/scholar?q=Kenny+Davila+and+Richard+Zanibbi%2E+2017%2E+Layout+and+semantics%3A+Combining+representations+for+mathematical+formula+search%2E+In+SIGIR%2E)
4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv:1810.04805.
5. Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. 2021. [SPLADE v2: Sparse lexical and expansion model for information retrieval](https://md.research.elamur.ai/free/en/papers/formal-2021-splade-v2-sparse-lexical/). arXiv:2109.10086.
6. Dallas Fraser, Andrew Kane, and Frank Tompa. 2018. Choosing math features for BM25 ranking with Tangent-L. In DocEng. [find](https://scholar.google.com/scholar?q=Dallas+Fraser%2C+Andrew+Kane%2C+and+Frank+Tompa%2E+2018%2E+Choosing+math+features+for+BM25+ranking+with+Tangent%2DL%2E+In+DocEng%2E)
7. Liangcai Gao, Zhuoren Jiang, Yue Yin, Ke Yuan, Zuoyu Yan, and Zhi Tang. 2017. Preliminary exploration of formula embedding for mathematical information retrieval: Can mathematical formulae be embedded like a natural language? arXiv:1707.05154. — [arXiv:1707.05154](https://arxiv.org/abs/1707.05154) [find](https://scholar.google.com/scholar?q=Liangcai+Gao%2C+Zhuoren+Jiang%2C+Yue+Yin%2C+Ke+Yuan%2C+Zuoyu+Yan%2C+and+Zhi+Tang%2E+2017%2E+Preliminary+exploration+of+formula+embedding+for+mathematical+information+retrieval%3A+Can+mathematical+formulae+be+embedded)
8. Luyu Gao and Jamie Callan. 2021. [Condenser: A pre-training architecture for dense retrieval](https://md.research.elamur.ai/free/en/papers/gao-2021-condenser-pre-training-architecture-dense/). arXiv:2104.08253.
9. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. arXiv:2004.10964. [find](https://scholar.google.com/scholar?q=Suchin+Gururangan%2C+Ana+Marasovic%CC%81%2C+Swabha+Swayamdipta%2C+Kyle+Lo%2C+Iz+Beltagy%2C+Doug+Downey%2C+and+Noah+A%2E+Smith%2E+2020%2E+Don%E2%80%99t+stop+pretraining%3A+Adapt+language+models+to+domains+and+tasks%2E+arXiv%3A2004%2E10964%2E)
10. Michael Kohlhase, Bogdan A Matican, and Corneliu-Claudiu Prodescu. 2012. MathWebSearch 0.5: Scaling an open formula search engine. In CICM. [find](https://scholar.google.com/scholar?q=Michael+Kohlhase%2C+Bogdan+A+Matican%2C+and+Corneliu%2DClaudiu+Prodescu%2E+2012%2E+MathWebSearch+0%2E5%3A+Scaling+an+open+formula+search+engine%2E+In+CICM%2E)
11. Giovanni Yoko Kristianto, Goran Topic, and Akiko Aizawa. 2016. MCAT math retrieval system for NTCIR-12 MathIR task. In NTCIR. [find](https://scholar.google.com/scholar?q=Giovanni+Yoko+Kristianto%2C+Goran+Topic%2C+and+Akiko+Aizawa%2E+2016%2E+MCAT+math+retrieval+system+for+NTCIR%2D12+MathIR+task%2E+In+NTCIR%2E)
12. Xiaoyan Lin, Liangcai Gao, Xuan Hu, Zhi Tang, Ying-nan Xiao, and Xiaozhong Liu. 2014. A mathematics retrieval system for formulae in layout presentations. In SIGIR. [find](https://scholar.google.com/scholar?q=Xiaoyan+Lin%2C+Liangcai+Gao%2C+Xuan+Hu%2C+Zhi+Tang%2C+Ying%2Dnan+Xiao%2C+and+Xiaozhong+Liu%2E+2014%2E+A+mathematics+retrieval+system+for+formulae+in+layout+presentations%2E+In+SIGIR%2E)
13. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv:1711.05101. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2017%2E+Decoupled+weight+decay+regularization%2E+arXiv%3A1711%2E05101%2E)
14. Behrooz Mansouri, Douglas W Oard, and Richard Zanibbi. 2020. DPRL systems in the CLEF 2020 ARQMath lab. In CLEF. [find](https://scholar.google.com/scholar?q=Behrooz+Mansouri%2C+Douglas+W+Oard%2C+and+Richard+Zanibbi%2E+2020%2E+DPRL+systems+in+the+CLEF+2020+ARQMath+lab%2E+In+CLEF%2E)
15. Behrooz Mansouri, Douglas W. Oard, and Richard Zanibbi. 2021a. DPRL systems in the CLEF 2021 ARQMath lab: Sentence-BERT for answer retrieval, learning-to-rank for formula retrieval. In CLEF. [find](https://scholar.google.com/scholar?q=Behrooz+Mansouri%2C+Douglas+W%2E+Oard%2C+and+Richard+Zanibbi%2E+2021a%2E+DPRL+systems+in+the+CLEF+2021+ARQMath+lab%3A+Sentence%2DBERT+for+answer+retrieval%2C+learning%2Dto%2Drank+for+formula+retrieval%2E+In+CLEF%2E)
16. Behrooz Mansouri, Shaurya Rohatgi, Douglas W. Oard, Jian Wu, C Lee Giles, and Richard Zanibbi. 2019. Tangent-CFT: An embedding model for mathematical formulas. In SIGIR. [find](https://scholar.google.com/scholar?q=Behrooz+Mansouri%2C+Shaurya+Rohatgi%2C+Douglas+W%2E+Oard%2C+Jian+Wu%2C+C+Lee+Giles%2C+and+Richard+Zanibbi%2E+2019%2E+Tangent%2DCFT%3A+An+embedding+model+for+mathematical+formulas%2E+In+SIGIR%2E)
17. Behrooz Mansouri, Richard Zanibbi, Douglas W. Oard, and Anurag Agarwal. 2021b. Overview of ARQMath-2 (2021): Second CLEF lab on answer retrieval for questions on math (working notes version). In CLEF. [find](https://scholar.google.com/scholar?q=Behrooz+Mansouri%2C+Richard+Zanibbi%2C+Douglas+W%2E+Oard%2C+and+Anurag+Agarwal%2E+2021b%2E+Overview+of+ARQMath%2D2+%282021%29%3A+Second+CLEF+lab+on+answer+retrieval+for+questions+on+math+%28working+notes+version%29%2E+In+CLEF%2E)
18. Bruce R. Miller and Abdou Youssef. 2003. Technical aspects of the digital library of mathematical functions. In AMAI. [find](https://scholar.google.com/scholar?q=Bruce+R%2E+Miller+and+Abdou+Youssef%2E+2003%2E+Technical+aspects+of+the+digital+library+of+mathematical+functions%2E+In+AMAI%2E)
19. Yin Ki Ng. 2021. Dowsing for math answers: Exploring MathCQA with a math-aware search engine. Master’s thesis, University of Waterloo. [find](https://scholar.google.com/scholar?q=Yin+Ki+Ng%2E+2021%2E+Dowsing+for+math+answers%3A+Exploring+MathCQA+with+a+math%2Daware+search+engine%2E+Master%E2%80%99s+thesis%2C+University+of+Waterloo%2E)
20. Yoshinori Hijikata, Hideki Hashimoto, and Shogo Nishida. 2009. Search mathematical formulas by mathematical formulas. In SHI (Symposium on Human Interface). [find](https://scholar.google.com/scholar?q=Yoshinori+Hijikata%2C+Hideki+Hashimoto%2C+and+Shogo+Nishida%2E+2009%2E+Search+mathematical+formulas+by+mathematical+formulas%2E+In+SHI+%28Symposium+on+Human+Interface%29%2E)
21. Yin Ki Ng, Dallas Fraser, Besat Kassaie, and Frank Tompa. 2021. Dowsing for answers to math questions: Ongoing viability of traditional MathIR. In CLEF. [find](https://scholar.google.com/scholar?q=Yin+Ki+Ng%2C+Dallas+Fraser%2C+Besat+Kassaie%2C+and+Frank+Tompa%2E+2021%2E+Dowsing+for+answers+to+math+questions%3A+Ongoing+viability+of+traditional+MathIR%2E+In+CLEF%2E)
22. Sebastian Hofstätter, Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy Lin, and Allan Hanbury. 2021. [Efficiently teaching an effective dense retriever with balanced topic aware sampling](https://md.research.elamur.ai/free/en/papers/hofstatter-2021-efficiently-teaching-effective-dense-retriever/). In SIGIR.
23. Yin Ki Ng, Dallas J. Fraser, Besat Kassaie, George Labahn, Mirette S Marzouk, Frank Tompa, and Kevin Wang. 2020. Dowsing for math answers with Tangent-L. In CLEF. [find](https://scholar.google.com/scholar?q=Yin+Ki+Ng%2C+Dallas+J%2E+Fraser%2C+Besat+Kassaie%2C+George+Labahn%2C+Mirette+S+Marzouk%2C+Frank+Tompa%2C+and+Kevin+Wang%2E+2020%2E+Dowsing+for+math+answers+with+Tangent%2DL%2E+In+CLEF%2E)
24. Hervé Jégou, Romain Tavenard, Matthijs Douze, and Laurent Amsaleg. 2011. Searching in one billion vectors: Re-rank with source coding. In IEEE ICASSP. [find](https://scholar.google.com/scholar?q=Herv%C3%A9+J%C3%A9gou%2C+Romain+Tavenard%2C+Matthijs+Douze%2C+and+Laurent+Amsaleg%2E+2011%2E+Searching+in+one+billion+vectors%3A+Re%2Drank+with+source+coding%2E+In+IEEE+ICASSP%2E)
25. Vít Novotnỳ, Michal Štefánik, Dávid Lupták, Martin Geletka, Petr Zelina, and Petr Sojka. 2021. Ensembling ten math information retrieval systems. In CLEF. [find](https://scholar.google.com/scholar?q=V%C3%ADt+Novotny%CC%80%2C+Michal+%C5%A0tef%C3%A1nik%2C+D%C3%A1vid+Lupt%C3%A1k%2C+Martin+Geletka%2C+Petr+Zelina%2C+and+Petr+Sojka%2E+2021%2E+Ensembling+ten+math+information+retrieval+systems%2E+In+CLEF%2E)
26. Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. [Dense passage retrieval for open-domain question answering](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/). arXiv:2004.04906.
27. Omar Khattab and Matei Zaharia. 2020. [ColBERT: Efficient and effective passage search via contextualized late interaction over BERT](https://md.research.elamur.ai/free/en/papers/khattab-2020-colbert-efficient-effective-passage/). In SIGIR.
28. Iadh Ounis, Gianni Amati, Vassilis Plachouras, Ben He, Craig Macdonald, and Douglas Johnson. 2005. Terrier information retrieval platform. In ECIR. [find](https://scholar.google.com/scholar?q=Iadh+Ounis%2C+Gianni+Amati%2C+Vassilis+Plachouras%2C+Ben+He%2C+Craig+Macdonald%2C+and+Douglas+Johnson%2E+2005%2E+Terrier+information+retrieval+platform%2E+In+ECIR%2E)
29. Shuai Peng, Ke Yuan, Liangcai Gao, and Zhi Tang. 2021. MathBERT: A pre-trained model for mathematical formula understanding. arXiv:2105.00377. — [arXiv:2105.00377](https://arxiv.org/abs/2105.00377) [find](https://scholar.google.com/scholar?q=Shuai+Peng%2C+Ke+Yuan%2C+Liangcai+Gao%2C+and+Zhi+Tang%2E+2021%2E+MathBERT%3A+A+pre%2Dtrained+model+for+mathematical+formula+understanding%2E+arXiv%3A2105%2E00377%2E+%E2%80%94+%5BarXiv%3A2105%2E00377%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2105%2E00377%29)
30. Anja Reusch, Maik Thiele, and Wolfgang Lehner. 2021a. An ALBERT-based similarity measure for mathematical answer retrieval. In SIGIR. [find](https://scholar.google.com/scholar?q=Anja+Reusch%2C+Maik+Thiele%2C+and+Wolfgang+Lehner%2E+2021a%2E+An+ALBERT%2Dbased+similarity+measure+for+mathematical+answer+retrieval%2E+In+SIGIR%2E)
31. Wei Zhong and Richard Zanibbi. 2019. Structural similarity search for formulas using leaf-root paths in operator subtrees. In ECIR. [find](https://scholar.google.com/scholar?q=Wei+Zhong+and+Richard+Zanibbi%2E+2019%2E+Structural+similarity+search+for+formulas+using+leaf%2Droot+paths+in+operator+subtrees%2E+In+ECIR%2E)
32. Anja Reusch, Maik Thiele, and Wolfgang Lehner. 2021b. TU_DBS in the ARQMath lab 2021, CLEF. In CLEF. [find](https://scholar.google.com/scholar?q=Anja+Reusch%2C+Maik+Thiele%2C+and+Wolfgang+Lehner%2E+2021b%2E+TU%5FDBS+in+the+ARQMath+lab+2021%2C+CLEF%2E+In+CLEF%2E)
33. Wei Zhong, Xinyu Zhang, Ji Xin, Jimmy Lin, and Richard Zanibbi. 2021. Approach Zero and Anserini at the CLEF-2021 ARQMath track: Applying substructure search and BM25 on operator tree path tokens. In CLEF. [find](https://scholar.google.com/scholar?q=Wei+Zhong%2C+Xinyu+Zhang%2C+Ji+Xin%2C+Jimmy+Lin%2C+and+Richard+Zanibbi%2E+2021%2E+Approach+Zero+and+Anserini+at+the+CLEF%2D2021+ARQMath+track%3A+Applying+substructure+search+and+BM25+on+operator+tree+path+tokens%2E+In+)
34. Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. 2021. [Col-BERTv2: Effective and efficient retrieval via lightweight late interaction](https://md.research.elamur.ai/free/en/papers/santhanam-2022-colbertv2-effective-efficient-retrieval-lightweight/). arXiv:2112.01488.
35. Thomas Schellenberg, Bo Yuan, and Richard Zanibbi. 2012. Layout-based substitution tree indexing and retrieval for mathematical expressions. In DRR. [find](https://scholar.google.com/scholar?q=Thomas+Schellenberg%2C+Bo+Yuan%2C+and+Richard+Zanibbi%2E+2012%2E+Layout%2Dbased+substitution+tree+indexing+and+retrieval+for+mathematical+expressions%2E+In+DRR%2E)
36. Qi Zhu, Yuxian Gu, Lingxiao Luo, Bing Li, Cheng Li, Wei Peng, Minlie Huang, and Xiaoyan Zhu. 2021. When does further pre-training MLM help? An empirical study on task-oriented dialog pre-training. In EMNLP Insights Workshop. [find](https://scholar.google.com/scholar?q=Qi+Zhu%2C+Yuxian+Gu%2C+Lingxiao+Luo%2C+Bing+Li%2C+Cheng+Li%2C+Wei+Peng%2C+Minlie+Huang%2C+and+Xiaoyan+Zhu%2E+2021%2E+When+does+further+pre%2Dtraining+MLM+help%3F+An+empirical+study+on+task%2Doriented+dialog+pre%2Dtraining%2E+In+)
37. Petr Sojka and Martin Líška. 2011. The art of mathematics retrieval. In DocEng. [find](https://scholar.google.com/scholar?q=Petr+Sojka+and+Martin+L%C3%AD%C5%A1ka%2E+2011%2E+The+art+of+mathematics+retrieval%2E+In+DocEng%2E)
38. Yujin Song and Xiaoyu Chen. 2021. Searching for mathematical formulas based on graph representation learning. In CICM. [find](https://scholar.google.com/scholar?q=Yujin+Song+and+Xiaoyu+Chen%2E+2021%2E+Searching+for+mathematical+formulas+based+on+graph+representation+learning%2E+In+CICM%2E)
39. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In NIPS.
40. Keisuke Yokoi and Akiko Aizawa. 2009. An approach to similarity search for mathematical expressions using MathML. In DML (Digital Mathematics Library). [find](https://scholar.google.com/scholar?q=Keisuke+Yokoi+and+Akiko+Aizawa%2E+2009%2E+An+approach+to+similarity+search+for+mathematical+expressions+using+MathML%2E+In+DML+%28Digital+Mathematics+Library%29%2E)
41. Abdou Youssef. 2005. Search of mathematical contents: Issues and methods. In IASSE. [find](https://scholar.google.com/scholar?q=Abdou+Youssef%2E+2005%2E+Search+of+mathematical+contents%3A+Issues+and+methods%2E+In+IASSE%2E)
42. Richard Zanibbi, Akiko Aizawa, Michael Kohlhase, Iadh Ounis, Goran Topic, and Kenny Davila. 2016a. [NTCIR-12 MathIR task overview](https://md.research.elamur.ai/free/en/papers/zanibbi-2016-ntcir-12-mathir-task-overview/). In NTCIR.
43. Richard Zanibbi and Dorothea Blostein. 2012. Recognition and retrieval of mathematical expressions. In IJDAR. [find](https://scholar.google.com/scholar?q=Richard+Zanibbi+and+Dorothea+Blostein%2E+2012%2E+Recognition+and+retrieval+of+mathematical+expressions%2E+In+IJDAR%2E)
44. Richard Zanibbi, Kenny Davila, Andrew Kane, and Frank Tompa. 2015. The Tangent search engine: Improved similarity metrics and scalability for math formula search. arXiv:1507.06235. — [arXiv:1507.06235](https://arxiv.org/abs/1507.06235) [find](https://scholar.google.com/scholar?q=Richard+Zanibbi%2C+Kenny+Davila%2C+Andrew+Kane%2C+and+Frank+Tompa%2E+2015%2E+The+Tangent+search+engine%3A+Improved+similarity+metrics+and+scalability+for+math+formula+search%2E+arXiv%3A1507%2E06235%2E+%E2%80%94+%5BarXiv%3A1507%2E06235)
45. Richard Zanibbi, Kenny Davila, Andrew Kane, and Frank Tompa. 2016b. Multi-stage math formula search: Using appearance-based similarity metrics at scale. In SIGIR. [find](https://scholar.google.com/scholar?q=Richard+Zanibbi%2C+Kenny+Davila%2C+Andrew+Kane%2C+and+Frank+Tompa%2E+2016b%2E+Multi%2Dstage+math+formula+search%3A+Using+appearance%2Dbased+similarity+metrics+at+scale%2E+In+SIGIR%2E)
46. Wei Zhong. 2015. A novel similarity-search method for mathematical content in LaTeX markup and its implementation. University of Delaware. [find](https://scholar.google.com/scholar?q=Wei+Zhong%2E+2015%2E+A+novel+similarity%2Dsearch+method+for+mathematical+content+in+LaTeX+markup+and+its+implementation%2E+University+of+Delaware%2E)
47. Wei Zhong and Hui Fang. 2016. OPMES: A similarity search engine for mathematical content. In ECIR. [find](https://scholar.google.com/scholar?q=Wei+Zhong+and+Hui+Fang%2E+2016%2E+OPMES%3A+A+similarity+search+engine+for+mathematical+content%2E+In+ECIR%2E)
48. Wei Zhong and Jimmy Lin. 2021. PyA0: A Python toolkit for accessible math-aware search. In SIGIR. [find](https://scholar.google.com/scholar?q=Wei+Zhong+and+Jimmy+Lin%2E+2021%2E+PyA0%3A+A+Python+toolkit+for+accessible+math%2Daware+search%2E+In+SIGIR%2E)
49. Wei Zhong, Shaurya Rohatgi, Jian Wu, Lee Giles, and Richard Zanibbi. 2020. Accelerating substructure similarity search for formula retrieval. In ECIR. [find](https://scholar.google.com/scholar?q=Wei+Zhong%2C+Shaurya+Rohatgi%2C+Jian+Wu%2C+Lee+Giles%2C+and+Richard+Zanibbi%2E+2020%2E+Accelerating+substructure+similarity+search+for+formula+retrieval%2E+In+ECIR%2E)

*Updated 3 November 2026*