---
title: "The First Multilingual Surface Realisation Shared Task (SR'18) — paper by Mille et al. (2018)"
description: "The report presents the first multilingual shared task on surface realisation, the step that turns an unordered… 14 references, cited by 40 papers here."
authors: ["Simon Mille", "Anja Belz", "Bernd Bohnet", "Yvette Graham", "Emily Pitler", "Leo Wanner"]
year: 2018
venue: "Proceedings of the First Workshop on Multilingual Surface Realisation"
source: "https://aclanthology.org/W18-3601/"
doi: "10.18653/v1/w18-3601"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/mille-2018-first-multilingual-surface-realisation-shared/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mille-2018-first-multilingual-surface-realisation-shared/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/4/) / The First Multilingual Surface Realisation Shared Task (SR'18): Overview and Evaluation Results

# The First Multilingual Surface Realisation Shared Task (SR'18): Overview and Evaluation Results

Paper · 2018 · Simon Mille, Anja Belz, Bernd Bohnet et al.

**Authors:** [Simon Mille](https://md.research.elamur.ai/free/en/authors/simon-mille/), [Anja Belz](https://md.research.elamur.ai/free/en/authors/anja-belz/), [Bernd Bohnet](https://md.research.elamur.ai/free/en/authors/bernd-bohnet/), [Yvette Graham](https://md.research.elamur.ai/free/en/authors/yvette-graham/), [Emily Pitler](https://md.research.elamur.ai/free/en/authors/emily-pitler/) and [Leo Wanner](https://md.research.elamur.ai/free/en/authors/leo-wanner/) · **Year:** 2018 · **Venue:** Proceedings of the First Workshop on Multilingual Surface Realisation **Link:** [aclanthology.org/W18-3601](https://aclanthology.org/W18-3601/) · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)

[**46** citations](https://openalex.org/works?filter=cites:W4252316627) OpenAlex, 2026-10-14 · [**40** citations in our library](https://md.research.elamur.ai/free/en/papers/mille-2018-first-multilingual-surface-realisation-shared/cited-by/) of the 141,840 papers whose bibliographies we read · [Proceedings of the First Workshop on Multilingual Surface Realisation](https://md.research.elamur.ai/free/en/venues/msr/) · [Workshop on Multilingual Surface Realisation 2018](https://md.research.elamur.ai/free/en/venues/msr/2018/) · published by [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · [licence](https://creativecommons.org/licenses/by/4.0/)

- [Publisher's page](https://aclanthology.org/W18-3601/)
- [PDF](https://aclanthology.org/W18-3601.pdf)
- [DOI 10.18653/v1/w18-3601](https://doi.org/10.18653/v1/w18-3601)
- [OpenAlex](https://openalex.org/W4252316627)

**Authors (6)**

1. [Simon Mille](https://openalex.org/A5077016111) [ORCID](https://orcid.org/0000-0002-8852-2764) · [Semantic Scholar](https://www.semanticscholar.org/author/2738095) · h 19 · 1,372 citations · 132 works
2. [Anja Belz](https://openalex.org/A5010118897) [ORCID](https://orcid.org/0000-0002-0552-8096) · [Semantic Scholar](https://www.semanticscholar.org/author/1792458) · h 23 · 2,059 citations · 135 works
3. [Bernd Bohnet](https://openalex.org/A5066427994) [Semantic Scholar](https://www.semanticscholar.org/author/1678747) · h 26 · 4,245 citations · 103 works
4. [Yvette Graham](https://openalex.org/A5020500064) [ORCID](https://orcid.org/0000-0001-6741-4855) · [Semantic Scholar](https://www.semanticscholar.org/author/2000709) · h 28 · 4,264 citations · 110 works
5. [Emily Pitler](https://openalex.org/A5089519482) [Semantic Scholar](https://www.semanticscholar.org/author/2585932) · h 24 · 3,809 citations · 55 works
6. [Leo Wanner](https://openalex.org/A5087323271) [ORCID](https://orcid.org/0000-0002-9446-3748) · [Semantic Scholar](https://www.semanticscholar.org/author/1758440) · h 27 · 2,955 citations · 246 works

Citations and author metrics: OpenAlex, 2026-10-14; authors with the same name are not always told apart.

## What it is about

The report presents the first multilingual shared task on surface realisation, the step that turns an unordered dependency tree into a sentence. Inputs come from Universal Dependencies treebanks with word order removed and tokens lemmatised. The shallow track covers ten languages. The deep track goes further and also removes function words and surface morphology, and it covers three languages. Systems were scored with automatic intrinsic metrics (BLEU-type scores) and by human judges on readability and meaning similarity. The organisers report high participation for a first run and give per-language tables of both kinds of score. The paper is an evaluation report: the systems themselves are described in other papers of the same volume.

## Ideas

- I2. Score outputs with more than one measure, as the shared task does with automatic scores and human judgements. For the translator, keep exact match and chrF, and add a small hand-judged sample for meaning similarity. Check whether the measures agree on dev before any claim.
- I3. Use the deep-track idea as a test for open problem 4: give the encoder meta tokens with function words removed and measure how many of them it restores. The gap is the bit cost of the function words the meta-world does not yet carry.

## For Mova

- I1. Split the linearizer's test into two levels, as the shallow and deep tracks do. Feed `en/src/lin.rs` the same trees with and without function words and score each separately. A large drop in the second level shows which information the realiser depends on.

The shared task is a benchmark, not a method, so the value is in its test design. It fits `tools/translate` (`measure.rs`), `en/src/lin.rs` and `tools/roundtrip`. The shallow and deep inputs map onto the translator's residual classes (word order, function words, morphology). Use the track split as a measurement design, and keep the test set held out until the once-per-release run.

## Institutions

[🏢**Google** Mountain View · United States 353 works · 364,806 citations](https://md.research.elamur.ai/free/en/companies/google/) · [🎓**Dublin City University** Dublin · Ireland 50 works · 8,937 citations](https://md.research.elamur.ai/free/en/universities/dublin-city-university/) · [🎓**Universitat Pompeu Fabra** Barcelona · Spain 36 works · 11,433 citations](https://md.research.elamur.ai/free/en/universities/universitat-pompeu-fabra/) · [🎓**University of Brighton** Brighton · United Kingdom 10 works · 2,521 citations](https://md.research.elamur.ai/free/en/universities/university-of-brighton/) · [🔬**ADAPT - Centre for Digital Content Technology** Dublin · Ireland 6 works · 951 citations](https://md.research.elamur.ai/free/en/labs/adapt-centre-for-digital-content-technology/) · [🤝**Institució Catalana de Recerca i Estudis Avançats** Barcelona · Spain 5 works · 445 citations](https://md.research.elamur.ai/free/en/ngos/institucio-catalana-de-recerca-i-estudis-avancats/)

📚 Shelves 4[Computer Science · 9,086 works](https://md.research.elamur.ai/free/en/shelves/computer-science/) · [Artificial Intelligence · 8,889 works](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · [Arts and Humanities · 9,296 works](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · [Language and Linguistics · 9,206 works](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/)

🧭 Topics 3[Machine translation · 958 works](https://md.research.elamur.ai/free/en/topics/machine-translation/) · [Multilingual & cross-lingual NLP · 467 works](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · [Text generation · 145 works](https://md.research.elamur.ai/free/en/topics/text-generation/)

## Appears in

Articles of ours that cite this paper.

Agglomerations[Barcelona](https://md.research.elamur.ai/free/en/agglomerations/barcelona/) · [Dublin](https://md.research.elamur.ai/free/en/agglomerations/dublin/)

## Cite

Cite as Simon Mille, Anja Belz, Bernd Bohnet, Yvette Graham, Emily Pitler and Leo Wanner. 2018. The First Multilingual Surface Realisation Shared Task (SR'18): Overview and Evaluation Results. Proceedings of the First Workshop on Multilingual Surface Realisation.

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.18653/v1/w18-3601](https://doi.org/10.18653/v1/w18-3601)

```bibtex
@inproceedings{mille2018first,
  title = {{The First Multilingual Surface Realisation Shared Task (SR'18): Overview and Evaluation Results}},
  author = {Simon Mille, Anja Belz, Bernd Bohnet, Yvette Graham, Emily Pitler and Leo Wanner},
  booktitle = {Proceedings of the First Workshop on Multilingual Surface Realisation},
  year = {2018},
  doi = {10.18653/v1/w18-3601},
  url = {https://doi.org/10.18653/v1/w18-3601}
}
```

- [← previous Sequence-to-Sequence Data Augmentation for Dialogue Language Understanding](https://md.research.elamur.ai/free/en/papers/hou-2018-sequence-sequence-data-augmentation-dialogue/)
- [next → Neural Coreference Resolution with Deep Biaffine Attention by Joint Mention Detection and Mention Clustering](https://md.research.elamur.ai/free/en/papers/zhang-2018-neural-coreference-resolution-deep-biaffine/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/4/)

## References

14 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anja Belz and Eric Kow. 2011. Discrete vs. continuous rating scales for language evaluation in NLP. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (ACL-HLT’11). [find](https://scholar.google.com/scholar?q=Anja+Belz+and+Eric+Kow%2E+2011%2E+Discrete+vs%2E+continuous+rating+scales+for+language+evaluation+in+NLP%2E+In+Proceedings+of+the+49th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28ACL%2DHLT%E2%80%99)
2. Anja Belz, Michael White, Dominic Espinosa, Eric Kow, Deirdre Hogan, and Amanda Stent. 2011. The first surface realisation shared task: Overview and evaluation results. In Proceedings of the 13th European Workshop on Natural Language Generation, ENLG ’11, pages 217–226, Stroudsburg, PA, USA. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/belz-2011-first-surface-realisation-shared-task/)
3. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. 2017a. Findings of the 2017 conference on machine translation (WMT’17). In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers, pages 169– 214, Copenhagen, Denmark. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/bojar-2017-findings-2017-conference-machine-translation/)
4. Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. Creating training corpora for micro-planners. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, Canada. Association for Computational Linguistics. [↗](https://doi.org/10.5281/zenodo.827393)
5. Yvette Graham, George Awad, and Alan Smeaton. 2017. Evaluation of Automatic Video Captioning Using Direct Assessment. ArXiv e-prints. [↗](https://doi.org/10.1371/journal.pone.0202789)
6. Yvette Graham and Timothy Baldwin. 2014. Testing for significance of increased correlation with human judgment. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 172–176, Doha, Qatar. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/graham-2014-testing-significance-increased-correlation-human/)
7. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2016. Can machine translation systems be evaluated by the crowd alone. Natural Language Engineering, FirstView:1–28. [↗](https://doi.org/10.1017/s1351324915000339)
8. Jonathan May and Jay Priyadarshi. 2017. Semeval-2017 task 9: Abstract meaning representation parsing and generation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 534–543, Vancouver, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jonathan+May+and+Jay+Priyadarshi%2E+2017%2E+Semeval%2D2017+task+9%3A+Abstract+meaning+representation+parsing+and+generation%2E+In+Proceedings+of+the+11th+International+Workshop+on+Semantic+Evaluation+%28SemEval%2D2)
9. Adam Meyers, R. Reeves, C. Macleod, R. Szekely, V. Zielinska, B. Young, and R. Grishman. 2004. The NomBank project: An interim report. In HLT-NAACL 2004 Workshop: Frontiers in Corpus Annotation, Boston, MA, May 2004, pages 24–31. [our card](https://md.research.elamur.ai/free/en/papers/meyers-2004-nombank-project-interim-report/)
10. Simon Mille, Roberto Carlini, Ivan Latorre, and Leo Wanner. 2017. Upf at epe 2017: Transduction-based deep analysis. In Shared Task on Extrinsic Parser Evaluation (EPE 2017), pages 80–88, Pisa, Italy. [find](https://scholar.google.com/scholar?q=Simon+Mille%2C+Roberto+Carlini%2C+Ivan+Latorre%2C+and+Leo+Wanner%2E+2017%2E+Upf+at+epe+2017%3A+Transduction%2Dbased+deep+analysis%2E+In+Shared+Task+on+Extrinsic+Parser+Evaluation+%28EPE+2017%29%2C+pages+80%E2%80%9388%2C+Pisa%2C+Italy%2E)
11. Joakim Nivre and Marie-Catherine de Marneffe et al. 2016. Universal dependencies v1: A multilingual treebank collection. In Proceedings of LREC, Portorož, Slovenia. [our card](https://md.research.elamur.ai/free/en/papers/nivre-2016-universal-dependencies-v1-multilingual-treebank/)
12. Jekaterina Novikova, Ondrej Dušek, and Verena Rieser. 2017. The E2E dataset: New challenges for end-to-end generation. In Proceedings of the 18th Annual Meeting of the Special Interest Group on Discourse and Dialogue, Saarbrücken, Germany. ArXiv:1706.09254. [↗](https://arxiv.org/abs/1706.09254) [our card](https://md.research.elamur.ai/free/en/papers/novikova-2017-e2e-dataset-new-challenges-end/)
13. Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The proposition bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1):71–105. [↗](https://doi.org/10.1162/0891201053630264) [our card](https://md.research.elamur.ai/free/en/papers/palmer-2005-proposition-bank-annotated-corpus-semantic/)
14. K. Papineni, S. Roukos, T. Ward, and W. j. Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Proc. 40th Annual Meeting on Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania. [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)

Updated 24 October 2026