---
title: "NTCIR-12 MathIR Task Overview — paper by Zanibbi et al. (2016)"
description: "This overview describes the third NTCIR math retrieval evaluation, NTCIR-12 MathIR. 17 references, cited by 14 papers here."
authors: ["Richard Zanibbi", "Akiko Aizawa", "Michael Kohlhase", "Iadh Ounis", "Goran Topić", "Kenny Davila"]
year: 2016
venue: "NTCIR-12"
source: "https://ir.webis.de/anthology/2016.ntcir_conference-2016.52/"
licence: "NTCIR — free access, read only"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zanibbi-2016-ntcir-12-mathir-task-overview/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zanibbi-2016-ntcir-12-mathir-task-overview/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/4/) / NTCIR-12 MathIR Task Overview

# NTCIR-12 MathIR Task Overview

**Authors:** Richard Zanibbi, [Akiko Aizawa](https://md.research.elamur.ai/free/en/authors/akiko-aizawa/), [Michael Kohlhase](https://md.research.elamur.ai/free/en/authors/michael-kohlhase/), [Iadh Ounis](https://md.research.elamur.ai/free/en/authors/iadh-ounis/), [Goran Topić](https://md.research.elamur.ai/free/en/authors/goran-topic/) and Kenny Davila · **Year:** 2016 · **Venue:** [NTCIR-12](https://md.research.elamur.ai/free/en/venues/ntcir/) ([NTCIR 2016](https://md.research.elamur.ai/free/en/venues/ntcir/2016/)) · **Licence:** NTCIR — free access, read only

**Links:** [Publisher's page](https://ir.webis.de/anthology/2016.ntcir_conference-2016.52/) · [PDF](https://research.nii.ac.jp/ntcir/workshop/OnlineProceedings12/pdf/ntcir/OVERVIEW/01-NTCIR12-OV-MathIR-ZanibbiR.pdf) · [OpenAlex](https://openalex.org/W2594866016)

**Citations:** 55 citations (OpenAlex, 2026-10-14) · [14](https://md.research.elamur.ai/free/en/papers/zanibbi-2016-ntcir-12-mathir-task-overview/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Most search engines cannot handle mathematical formulae, though formulae are central to scientific communication. This overview describes the third NTCIR math retrieval evaluation, NTCIR-12 MathIR. It offers a shared benchmark for systems that search documents using keywords, formulae, or both. The organizers reused the NTCIR-11 arXiv collection, split into paragraph-level units. They also built a new English Wikipedia collection that targets non-expert users, such as students. Four subtasks were defined:
- arXiv-main (keywords plus formulae)
- arXiv-simto, an experimental task testing a new "similar-to" region operator inside formula queries
- Wiki-main
- Wiki-formula, which uses isolated formula queries and graded relevance instead of recall of known targets

Student assessors judged pooled results in the SEPIA interface. The strongest systems, MCAT and ICST, combined text retrieval with formula retrieval and used surrounding context. Still, every run fell well short of the ideal ranking derived from the pools. The authors conclude that combining formula appearance with formula semantics looks more promising than choosing one encoding over the other.

## Key points
- The arXiv corpus contains 105,120 articles converted to HTML+MathML by KWARC. It is split into 8,301,578 paragraph units holding roughly 60 million formulae.
- The new Wikipedia corpus has 319,689 articles and over 590,000 formulae in LaTeX, Presentation MathML and Content MathML. About 10% of the articles contain math markup, and the remaining 90% are text-only distractors.
- There were 107 topics in total: arXiv-main 29, arXiv-simto 8, Wiki-main 30 and Wiki-formula 40. Half of the Wiki-formula queries contain wildcards.
- Teams could submit up to four runs per task, with 1,000 results per query. Inter-assessor agreement (Fleiss' κ) was higher for arXiv than for Wikipedia and lowest for Wiki-formula.
- In Wiki-formula, MCAT slightly beat RITUW but was slower. Unification-based re-ranking helped ranking quality but cost speed.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,888 works · [🧮 Computational Theory and Mathematics](https://md.research.elamur.ai/free/en/shelves/computer-science/computational-theory-and-mathematics/) · 496 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,948 works

## Cite

Cite as: Richard Zanibbi, Akiko Aizawa, Michael Kohlhase, Iadh Ounis, Goran Topić and Kenny Davila. 2016. NTCIR-12 MathIR Task Overview. NTCIR-12.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zanibbi2016ntcir,
  title = {{NTCIR-12 MathIR Task Overview}},
  author = {Richard Zanibbi, Akiko Aizawa, Michael Kohlhase, Iadh Ounis, Goran Topić and Kenny Davila},
  booktitle = {NTCIR-12},
  year = {2016},
  url = {https://ir.webis.de/anthology/2016.ntcir_conference-2016.52/}
}
```

- [← previous Word Sense Clustering and Clusterability](https://md.research.elamur.ai/free/en/papers/mccarthy-2016-word-sense-clustering-clusterability/)
- [next → rstWeb - A Browser-based Annotation Interface for Rhetorical Structure Theory and Discourse Relations](https://md.research.elamur.ai/free/en/papers/zeldes-2016-rstweb-browser-based-annotation-interface/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/4/)

## References

17 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. A. Aizawa, M. Kohlhase, and I. Ounis. [NTCIR-10 Math pilot task overview](https://md.research.elamur.ai/free/en/papers/aizawa-2013-ntcir-10-math-pilot-task/). In N. Kando and K. Kishida, editors, NTCIR Workshop 10 Meeting, pages 1–8, Tokyo, Japan, 2013.
2. A. Aizawa, M. Kohlhase, I. Ounis, and M. Schubotz. [NTCIR-11 math-2 task overview](https://md.research.elamur.ai/free/en/papers/aizawa-2014-ntcir-11-math-2-task/). In NTCIR. National Institute of Informatics (NII), 2014.
3. K. Davila, R. Zanibbi, A. Kane, and F. Tompa. Tangent-3 at the NTCIR-12 MathIR task. In Proc. NTCIR-12, 2016. [find](https://scholar.google.com/scholar?q=K%2E+Davila%2C+R%2E+Zanibbi%2C+A%2E+Kane%2C+and+F%2E+Tompa%2E+Tangent%2D3+at+the+NTCIR%2D12+MathIR+task%2E+In+Proc%2E+NTCIR%2D12%2C+2016%2E)
4. L. Gao, K. Yuan, Y. Wang, Z. Jiang, and Z. Tang. The math retrieval system of ICST for NTCIR-12 MathIR task. In Proc. NTCIR-12, 2016. [find](https://scholar.google.com/scholar?q=L%2E+Gao%2C+K%2E+Yuan%2C+Y%2E+Wang%2C+Z%2E+Jiang%2C+and+Z%2E+Tang%2E+The+math+retrieval+system+of+ICST+for+NTCIR%2D12+MathIR+task%2E+In+Proc%2E+NTCIR%2D12%2C+2016%2E)
5. F. Guidi and C. S. Coen. A survey on retrieval of mathematical knowledge. In M. K. et al., editor, Proc. CICM, volume 9150 of LNAI, pages 296–315, 2015. [find](https://scholar.google.com/scholar?q=F%2E+Guidi+and+C%2E+S%2E+Coen%2E+A+survey+on+retrieval+of+mathematical+knowledge%2E+In+M%2E+K%2E+et+al%2E%2C+editor%2C+Proc%2E+CICM%2C+volume+9150+of+LNAI%2C+pages+296%E2%80%93315%2C+2015%2E)
6. M. Kohlhase. Formats for topics and submissions for the MathIR task at NTCIR-12. Technical report, NTCIR, 2015. [find](https://scholar.google.com/scholar?q=M%2E+Kohlhase%2E+Formats+for+topics+and+submissions+for+the+MathIR+task+at+NTCIR%2D12%2E+Technical+report%2C+NTCIR%2C+2015%2E)
7. G. Y. Kristianto, G. Topić, and A. Aizawa. MCAT math retrieval system for NTCIR-12 MathIR task. In Proc. NTCIR-12, 2016. [find](https://scholar.google.com/scholar?q=G%2E+Y%2E+Kristianto%2C+G%2E+Topic%CC%81%2C+and+A%2E+Aizawa%2E+MCAT+math+retrieval+system+for+NTCIR%2D12+MathIR+task%2E+In+Proc%2E+NTCIR%2D12%2C+2016%2E)
8. M. Lı́ška, P. Sojka, and M. Rûžička. Combining text and formula queries in math information retrieval: Evaluation of query results merging strategies. In Proc. Int’l Work. Novel Web Search Interfaces and Systems, pages 7–9, New York, 2015. [find](https://scholar.google.com/scholar?q=M%2E+L%C4%B1%CC%81s%CC%8Cka%2C+P%2E+Sojka%2C+and+M%2E+Ru%CC%82z%CC%8Cic%CC%8Cka%2E+Combining+text+and+formula+queries+in+math+information+retrieval%3A+Evaluation+of+query+results+merging+strategies%2E+In+Proc%2E+Int%E2%80%99l+Work%2E+Novel+Web+Search+Interfa)
9. L. Page, S. Brin, R. Motwani, and T. Winograd. [The PageRank citation ranking: Bringing order to the web](https://md.research.elamur.ai/free/en/papers/page-1999-pagerank-citation-ranking-bringing/). Technical Report 1999-66, Stanford InfoLab, November 1999.
10. M. Rûžička, P. Sojka, and M. Lı́ška. Math indexer and searcher under the hood: Fine-tuning query expansion and unification strategies. In Proc. NTCIR-12, 2016. [find](https://scholar.google.com/scholar?q=M%2E+Ru%CC%82z%CC%8Cic%CC%8Cka%2C+P%2E+Sojka%2C+and+M%2E+L%C4%B1%CC%81s%CC%8Cka%2E+Math+indexer+and+searcher+under+the+hood%3A+Fine%2Dtuning+query+expansion+and+unification+strategies%2E+In+Proc%2E+NTCIR%2D12%2C+2016%2E)
11. M. Schubotz, M. Leich, N. Meuschke, and B. Gipp. Exploring the one-brain barrier: a manual contribution to the NTCIR-12 Math task. In Proc. NTCIR-12, 2016. [find](https://scholar.google.com/scholar?q=M%2E+Schubotz%2C+M%2E+Leich%2C+N%2E+Meuschke%2C+and+B%2E+Gipp%2E+Exploring+the+one%2Dbrain+barrier%3A+a+manual+contribution+to+the+NTCIR%2D12+Math+task%2E+In+Proc%2E+NTCIR%2D12%2C+2016%2E)
12. M. Schubotz, A. Youssef, V. Markl, and H. S. Cohl. Challenges of mathematical information retrieval in the NTCIR-11 Math Wikipedia Task. In Proc. ACM SIGIR, pages 951–954, 2015. [find](https://scholar.google.com/scholar?q=M%2E+Schubotz%2C+A%2E+Youssef%2C+V%2E+Markl%2C+and+H%2E+S%2E+Cohl%2E+Challenges+of+mathematical+information+retrieval+in+the+NTCIR%2D11+Math+Wikipedia+Task%2E+In+Proc%2E+ACM+SIGIR%2C+pages+951%E2%80%93954%2C+2015%2E)
13. sepia: Standard evaluation package for information access systems. [https://code.google.com/p/sepia/](https://code.google.com/p/sepia/). [↗](https://code.google.com/p/sepia/)
14. P. Sojka and M. Lı́ška. The Art of Mathematics Retrieval. In Proc. ACM DocEng, pages 57–60, Mountain View, CA, Sept. 2011. [find](https://scholar.google.com/scholar?q=P%2E+Sojka+and+M%2E+L%C4%B1%CC%81s%CC%8Cka%2E+The+Art+of+Mathematics+Retrieval%2E+In+Proc%2E+ACM+DocEng%2C+pages+57%E2%80%9360%2C+Mountain+View%2C+CA%2C+Sept%2E+2011%2E)
15. A. Thanda, A. Agarwal, K. Singla, A. Prakash, and A. Gupta. A document retrieval system for math queries. In Proc. NTCIR-12, 2016. [find](https://scholar.google.com/scholar?q=A%2E+Thanda%2C+A%2E+Agarwal%2C+K%2E+Singla%2C+A%2E+Prakash%2C+and+A%2E+Gupta%2E+A+document+retrieval+system+for+math+queries%2E+In+Proc%2E+NTCIR%2D12%2C+2016%2E)
16. Common evaluation measures. In E. M. Voorhees and L. P. Buckland, editors, The Sixteenth Text REtrieval Conference (TREC 2007) Proceedings, number SP 500-274 in NIST Special Publication, 2007. [find](https://scholar.google.com/scholar?q=Common+evaluation+measures%2E+In+E%2E+M%2E+Voorhees+and+L%2E+P%2E+Buckland%2C+editors%2C+The+Sixteenth+Text+REtrieval+Conference+%28TREC+2007%29+Proceedings%2C+number+SP+500%2D274+in+NIST+Special+Publication%2C+2007%2E)
17. R. Zanibbi and D. Blostein. Recognition and retrieval of mathematical expressions. Int. J. Doc. Analysis and Recognition, 15(4):331–357, 2012. [find](https://scholar.google.com/scholar?q=R%2E+Zanibbi+and+D%2E+Blostein%2E+Recognition+and+retrieval+of+mathematical+expressions%2E+Int%2E+J%2E+Doc%2E+Analysis+and+Recognition%2C+15%284%29%3A331%E2%80%93357%2C+2012%2E)

*Updated 3 November 2026*