---
title: "NTCIR-10 Math Pilot Task Overview — Aizawa 2013"
description: "Most search systems ignore mathematical formulae in documents. 13 references, cited by 12 papers here."
authors: ["Akiko Aizawa", "Michael Kohlhase", "Iadh Ounis"]
year: 2013
venue: "NTCIR-10"
source: "https://research.nii.ac.jp/ntcir/workshop/OnlineProceedings10/NTCIR/toc_ntcir.html"
licence: "NTCIR — free access, read only"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/aizawa-2013-ntcir-10-math-pilot-task/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/aizawa-2013-ntcir-10-math-pilot-task/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2013](https://md.research.elamur.ai/free/en/papers/2013/page/2/) / NTCIR-10 Math Pilot Task Overview

# NTCIR-10 Math Pilot Task Overview

**Authors:** [Akiko Aizawa](https://md.research.elamur.ai/free/en/authors/akiko-aizawa/), [Michael Kohlhase](https://md.research.elamur.ai/free/en/authors/michael-kohlhase/) and [Iadh Ounis](https://md.research.elamur.ai/free/en/authors/iadh-ounis/) · **Year:** 2013 · **Venue:** [NTCIR-10](https://md.research.elamur.ai/free/en/venues/ntcir/) ([NTCIR 2013](https://md.research.elamur.ai/free/en/venues/ntcir/2013/)) · **Licence:** NTCIR — free access, read only

**Links:** [Publisher's page](https://research.nii.ac.jp/ntcir/workshop/OnlineProceedings10/NTCIR/toc_ntcir.html) · [PDF](https://research.nii.ac.jp/ntcir/workshop/OnlineProceedings10/pdf/NTCIR/OVERVIEW/01-NTCIR10-OV-MATH-AizawaA.pdf) · [OpenAlex](https://openalex.org/W2400955359)

**Citations:** 46 citations (OpenAlex, 2026-10-14) · [12](https://md.research.elamur.ai/free/en/papers/aizawa-2013-ntcir-10-math-pilot-task/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Most search systems ignore mathematical formulae in documents. This is partly because there were no large structured formula collections, well-defined tasks or agreed evaluation methods. The NTCIR-10 Math Pilot Task was the first shared evaluation dedicated to math information access.

The organizers built a test collection of about 100,000 arXiv papers from mathematics, physics and computer science, converted by the arXMLiv project. In it, each formula is stored as presentation MathML, content MathML and LaTeX. Two subtasks were offered:

* **Math Retrieval**: formula search with wildcard query variables, combined keyword-and-formula search, and open free-text queries.
* **Math Understanding**: extracting textual descriptions of formulae.

Submitted runs were pooled per query and judged on a per-formula basis by mathematically trained assessors in the SEPIA system. Six of sixteen registered teams took part. Open-query runs could not be judged because they lacked formula identifiers. The only Math Understanding system reached high precision but modest recall.

The task created reusable topics, relevance judgments and tools. It also helped bring together a cross-disciplinary math IR community.

## Key points
- Dataset is roughly 63 GiB, with about 35.5 million formulae and 297 million subformulae (the units targeted by formula search).
- Mathematicians supplied 56 topics: 22 formula search (FS), 15 full-text (FT) and 19 open (OMIR).
  - FS-17 was dropped as ill-formed.
  - No OMIR topics were evaluated because runs reported only document URIs.
- Pooling collected at least 100 formulae per query from all runs.
  - Assessors from Zentralblatt Math and Jacobs University students labelled hits as relevant, partially relevant or not relevant.
  - 2,128 formulae were judged for FS topics alone.
- Participants were BRKLY, FSE, KWARC, MCAT, MIRMU and NAK, with up to four runs per team. All six teams entered Math Retrieval.
- In Math Understanding, MCAT paired noun phrases with formulae and classified the pairs with an SVM.
  - Best strict-matching F1 was 50.77 (short descriptions).
  - Best soft-matching F1 was 62.18, with precision 87.25 and recall 48.30.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [National Institute of Informatics](https://md.research.elamur.ai/free/en/labs/national-institute-of-informatics/) — Research institute, Tokyo, Japan · 40 works · 3,317 citations
- [University of Glasgow](https://md.research.elamur.ai/free/en/universities/university-of-glasgow/) — University, Glasgow, United Kingdom · 15 works · 804 citations

📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,888 works · [🧮 Computational Theory and Mathematics](https://md.research.elamur.ai/free/en/shelves/computer-science/computational-theory-and-mathematics/) · 496 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,948 works

## Cite

Cite as: Akiko Aizawa, Michael Kohlhase and Iadh Ounis. 2013. NTCIR-10 Math Pilot Task Overview. NTCIR-10.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{aizawa2013ntcir,
  title = {{NTCIR-10 Math Pilot Task Overview}},
  author = {Akiko Aizawa, Michael Kohlhase and Iadh Ounis},
  booktitle = {NTCIR-10},
  year = {2013},
  url = {https://research.nii.ac.jp/ntcir/workshop/OnlineProceedings10/NTCIR/toc_ntcir.html}
}
```

- [← previous Latent Anaphora Resolution for Cross-Lingual Pronoun Prediction](https://md.research.elamur.ai/free/en/papers/hardmeier-2013-latent-anaphora-resolution-cross-lingual/)
- [next → Character-to-Character Sentiment Analysis in Shakespeare's Plays](https://md.research.elamur.ai/free/en/papers/nalisnick-2013-character-character-sentiment-analysis-shakespeare/)
- [↑ back to papers of 2013](https://md.research.elamur.ai/free/en/papers/2013/page/2/)

## References

13 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. arxiv.org e-Print archive. web page at [http://www.arxiv.org.seen](http://www.arxiv.org.seen) November 2012. [↗](http://www.arxiv.org.seen)
2. arXMLiv: Translating the Math Archives to XML+MathML. web page at [http://trac.kwarc.info/arXMLiv/](http://trac.kwarc.info/arXMLiv/). seen April 2010. [↗](http://trac.kwarc.info/arXMLiv/)
3. Hiroya Hagino and Hiroaki Saito. Partial-match retrieval with structure-reflected indices at the ntcir-10 math task. In NTCIR-10 [NTC13]. this volume. [find](https://scholar.google.com/scholar?q=Hiroya+Hagino+and+Hiroaki+Saito%2E+Partial%2Dmatch+retrieval+with+structure%2Dreflected+indices+at+the+ntcir%2D10+math+task%2E+In+NTCIR%2D10+%5BNTC13%5D%2E+this+volume%2E)
4. Michael Kohlhase. Topics for the ntcir-10 math task; math retrieval subtask. Technical report, NTCIR, 2012. [find](https://scholar.google.com/scholar?q=Michael+Kohlhase%2E+Topics+for+the+ntcir%2D10+math+task%3B+math+retrieval+subtask%2E+Technical+report%2C+NTCIR%2C+2012%2E)
5. Michael Kohlhase and Corneliu-Claudiu Prodescu. Mathwebsearch at ntcir-10. In NTCIR-10 [NTC13]. this volume. [find](https://scholar.google.com/scholar?q=Michael+Kohlhase+and+Corneliu%2DClaudiu+Prodescu%2E+Mathwebsearch+at+ntcir%2D10%2E+In+NTCIR%2D10+%5BNTC13%5D%2E+this+volume%2E)
6. Ray R. Larson, Chloe Reynolds, and Fredric Gey. The abject failure of keyword ir for mathematics search: Berkeley at ntcir-10 math. In NTCIR-10 [NTC13]. this volume. [find](https://scholar.google.com/scholar?q=Ray+R%2E+Larson%2C+Chloe+Reynolds%2C+and+Fredric+Gey%2E+The+abject+failure+of+keyword+ir+for+mathematics+search%3A+Berkeley+at+ntcir%2D10+math%2E+In+NTCIR%2D10+%5BNTC13%5D%2E+this+volume%2E)
7. Martin Lśka, Petr Sojka, and MichalRůžička. Mirmu at the ntcir-10 math task: Similarity search for mathematics. In NTCIR-10 [NTC13]. this volume. [find](https://scholar.google.com/scholar?q=Martin+Ls%CC%81ka%2C+Petr+Sojka%2C+and+MichalRu%CC%8Az%CC%8Cic%CC%8Cka%2E+Mirmu+at+the+ntcir%2D10+math+task%3A+Similarity+search+for+mathematics%2E+In+NTCIR%2D10+%5BNTC13%5D%2E+this+volume%2E)
8. NTCIR Workshop 10 Meeting, Tokyo, Japan, 2013. this volume. [find](https://scholar.google.com/scholar?q=NTCIR+Workshop+10+Meeting%2C+Tokyo%2C+Japan%2C+2013%2E+this+volume%2E)
9. NTICR Pilot Task: Math Task. [http://ntcir-math.nii.ac.jp/](http://ntcir-math.nii.ac.jp/), 2013. [↗](http://ntcir-math.nii.ac.jp/)
10. sepia: Standard evaluation package for information access systems. [https://code.google.com/p/sepia/](https://code.google.com/p/sepia/). [↗](https://code.google.com/p/sepia/)
11. Heinrich Stamerjohanns, Michael Kohlhase, Deyan Ginev, Catalin David, and Bruce Miller. [Transforming large collections of scientific publications to XML](https://md.research.elamur.ai/free/en/papers/stamerjohanns-2010-transforming-large-collections-scientific-publications/). Mathematics in Computer Science, 3(3):299–307, 2010.
12. Moritz Schubotz, Marcus Leich, and Volker Markl. Querying large collections of mathematical publications. In NTCIR-10 [find](https://scholar.google.com/scholar?q=Moritz+Schubotz%2C+Marcus+Leich%2C+and+Volker+Markl%2E+Querying+large+collections+of+mathematical+publications%2E+In+NTCIR%2D10)
13. Goran Topic, Giovanni Yoko Kristianto, Minh-Quoc Nghiem, and Akiko Aizawa. The mcat math retrieval system for ntcir-10 math track. In NTCIR-10 [NTC13]. this volume. [find](https://scholar.google.com/scholar?q=Goran+Topic%2C+Giovanni+Yoko+Kristianto%2C+Minh%2DQuoc+Nghiem%2C+and+Akiko+Aizawa%2E+The+mcat+math+retrieval+system+for+ntcir%2D10+math+track%2E+In+NTCIR%2D10+%5BNTC13%5D%2E+this+volume%2E)

*Updated 3 November 2026*