--- title: "NTCIR-11 Math-2 Task Overview — paper by Aizawa et al. (2014)" description: "Mainstream search engines generally cannot retrieve mathematical formulae. 9 references, cited by 12 papers here." authors: ["Akiko Aizawa", "Michael Kohlhase", "Iadh Ounis", "Moritz Schubotz"] year: 2014 venue: "NTCIR-11" source: "https://research.nii.ac.jp/ntcir/workshop/OnlineProceedings11/NTCIR/toc_ntcir.html" licence: "NTCIR — free access, read only" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/aizawa-2014-ntcir-11-math-2-task/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/aizawa-2014-ntcir-11-math-2-task/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2014](https://md.research.elamur.ai/free/en/papers/2014/page/2/) / NTCIR-11 Math-2 Task Overview # NTCIR-11 Math-2 Task Overview **Authors:** [Akiko Aizawa](https://md.research.elamur.ai/free/en/authors/akiko-aizawa/), [Michael Kohlhase](https://md.research.elamur.ai/free/en/authors/michael-kohlhase/), [Iadh Ounis](https://md.research.elamur.ai/free/en/authors/iadh-ounis/) and [Moritz Schubotz](https://md.research.elamur.ai/free/en/authors/moritz-schubotz/) · **Year:** 2014 · **Venue:** [NTCIR-11](https://md.research.elamur.ai/free/en/venues/ntcir/) ([NTCIR 2014](https://md.research.elamur.ai/free/en/venues/ntcir/2014/)) · **Licence:** NTCIR — free access, read only **Links:** [Publisher's page](https://research.nii.ac.jp/ntcir/workshop/OnlineProceedings11/NTCIR/toc_ntcir.html) · [PDF](https://research.nii.ac.jp/ntcir/workshop/OnlineProceedings11/pdf/NTCIR/OVERVIEW/01-NTCIR11-OV-MATH-AizawaA.pdf) · [OpenAlex](https://openalex.org/W2401312310) **Citations:** 50 citations (OpenAlex, 2026-10-14) · [12](https://md.research.elamur.ai/free/en/papers/aizawa-2014-ntcir-11-math-2-task/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Mainstream search engines generally cannot retrieve mathematical formulae. Research on math information retrieval (MIR) has been held back by a lack of large structured datasets and agreed evaluation procedures. Building on the NTCIR-10 Math Pilot, the NTCIR-11 Math-2 Task provides a shared ad-hoc retrieval benchmark that combines formula and keyword queries. The organizers converted about 105,120 arXiv preprints from LaTeX to HTML+MathML using LaTeXML. They split each document into paragraph-sized retrieval units, which makes relevance judgment manageable. Each of the 50 topics combines keywords with formulae, which may include wildcard query variables, encoded in presentation MathML, content MathML and LaTeX. Submitted runs were pooled and judged by humans in the SEPIA tool, with MAP, P@5, P@10 and Bpref as metrics. An optional free subtask used English Wikipedia with automatic scoring. Eight teams took part, and their approaches differed in how they handled formula structure, query variables and search engines. The judged collection is a first comprehensive MIR test set and was scheduled for release by NTCIR in 2015. ## Key points - Corpus: 8,301,578 paragraph units containing about 60 million formulae. The arXiv categories include math, cs, stat, hep-th and others. Total size is about 174GB, stored as HTML5 and XHTML5. - Topics: 50 formula/keyword queries, with 0–8 query variables each (``, written as ?x). - Participation: 8 groups, including 4 new ones, submitted 20 runs. The groups were ICST, IFISB, FSE, KWARC, MCAT, MIRMU, RIT and TUW-IMP. - Assessment: 50 pooled units per topic gave 2,500 judgments in total. Each unit was assessed by at least two evaluators. - The Wikipedia subtask used 100 automatically generated queries and immediate automatic evaluation. Four teams joined it. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* 📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,888 works · [🧮 Computational Theory and Mathematics](https://md.research.elamur.ai/free/en/shelves/computer-science/computational-theory-and-mathematics/) · 496 works 🧭 Topics (2): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,948 works · [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 561 works ## Cite Cite as: Akiko Aizawa, Michael Kohlhase, Iadh Ounis and Moritz Schubotz. 2014. NTCIR-11 Math-2 Task Overview. NTCIR-11. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{aizawa2014ntcir, title = {{NTCIR-11 Math-2 Task Overview}}, author = {Akiko Aizawa, Michael Kohlhase, Iadh Ounis and Moritz Schubotz}, booktitle = {NTCIR-11}, year = {2014}, url = {https://research.nii.ac.jp/ntcir/workshop/OnlineProceedings11/NTCIR/toc_ntcir.html} } ``` - [← previous Latent Trees for Coreference Resolution](https://md.research.elamur.ai/free/en/papers/fernandes-2014-latent-trees-coreference-resolution/) - [next → Tharwa: A Large Scale Dialectal Arabic - Standard Arabic - English Lexicon](https://md.research.elamur.ai/free/en/papers/diab-2014-tharwa-large-scale-dialectal-arabic/) - [↑ back to papers of 2014](https://md.research.elamur.ai/free/en/papers/2014/page/2/) ## References 9 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Proceedings of the 11th NTCIR Conference, Tokyo, Japan, 2014. [find](https://scholar.google.com/scholar?q=Proceedings+of+the+11th+NTCIR+Conference%2C+Tokyo%2C+Japan%2C+2014%2E) 2. A. Aizawa, M. Kohlhase, and I. Ounis. [NTCIR-10 Math pilot task overview](https://md.research.elamur.ai/free/en/papers/aizawa-2013-ntcir-10-math-pilot-task/). In N. Kando and K. Kishida, editors, NTCIR Workshop 10 Meeting, pages 18, Tokyo, Japan, 2013. 3. J. M. Gonzalez Pinto, S. Barthel, and W.-T. Balke. QUALIBETA at the NTCIR-11 Math 2 task: An attempt to query math collections. [1]. [find](https://scholar.google.com/scholar?q=J%2E+M%2E+Gonzalez+Pinto%2C+S%2E+Barthel%2C+and+W%2E%2DT%2E+Balke%2E+QUALIBETA+at+the+NTCIR%2D11+Math+2+task%3A+An+attempt+to+query+math+collections%2E+%5B1%5D%2E) 4. M. Kohlhase. Formats for topics and submissions for the Math2 task at NTCIR-11. Technical report, NTCIR, 2014. [find](https://scholar.google.com/scholar?q=M%2E+Kohlhase%2E+Formats+for+topics+and+submissions+for+the+Math2+task+at+NTCIR%2D11%2E+Technical+report%2C+NTCIR%2C+2014%2E) 5. M. Kohlhase, R. Hambasan, and C.-C. Prodescu. MathWebSearch at NTCIR-11. [1]. [find](https://scholar.google.com/scholar?q=M%2E+Kohlhase%2C+R%2E+Hambasan%2C+and+C%2E%2DC%2E+Prodescu%2E+MathWebSearch+at+NTCIR%2D11%2E+%5B1%5D%2E) 6. G. Y. Kristianto, G. Topi¢, F. Ho, and A. Aizawa. The MCAT math retrieval system for NTCIR-11 Math track. [1]. [find](https://scholar.google.com/scholar?q=G%2E+Y%2E+Kristianto%2C+G%2E+Topi%C2%A2%2C+F%2E+Ho%2C+and+A%2E+Aizawa%2E+The+MCAT+math+retrieval+system+for+NTCIR%2D11+Math+track%2E+%5B1%5D%2E) 7. J. R. Landis and G. G. Koch. The measurement of observer agreement for categorical data. Biometrics, 33:159174, 1977. [find](https://scholar.google.com/scholar?q=J%2E+R%2E+Landis+and+G%2E+G%2E+Koch%2E+The+measurement+of+observer+agreement+for+categorical+data%2E+Biometrics%2C+33%3A159%15174%2C+1977%2E) 8. L. H. Liangcai Gao, Yuehan Wang and Z. Tang. Icst math retrieval system for ntcir-11 math-2 task. [1]. [find](https://scholar.google.com/scholar?q=L%2E+H%2E+Liangcai+Gao%2C+Yuehan+Wang+and+Z%2E+Tang%2E+Icst+math+retrieval+system+for+ntcir%2D11+math%2D2+task%2E+%5B1%5D%2E) 9. A. Lipani, L. Andersson, F. Piroi, M. Lupu, and [find](https://scholar.google.com/scholar?q=A%2E+Lipani%2C+L%2E+Andersson%2C+F%2E+Piroi%2C+M%2E+Lupu%2C+and) *Updated 3 November 2026*