---
title: "The 2020 Bilingual, Bi-Directional WebNLG+ — Ferreira 2020"
description: "WebNLG+ 2020 extends the earlier WebNLG benchmark, which pairs sets of DBpedia RDF triples with texts."
authors: ["Thiago Castro Ferreira", "Claire Gardent", "Nikolai Ilinykh", "Chris van der Lee", "Simon Mille", "Diego Moussallem", "Anastasia Shimorina"]
year: 2020
venue: "Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+)"
source: "https://aclanthology.org/2020.webnlg-1.7/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/ferreira-2020-2020-bilingual-bi-directional-webnlg/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/ferreira-2020-2020-bilingual-bi-directional-webnlg/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2020](https://md.research.elamur.ai/free/papers/2020/page/4/) / The 2020 Bilingual, Bi-Directional WebNLG+ Shared Task: Overview and Evaluation Results (WebNLG+ 2020)

# The 2020 Bilingual, Bi-Directional WebNLG+ Shared Task: Overview and Evaluation Results (WebNLG+ 2020)

**Authors:** [Thiago Castro Ferreira](https://md.research.elamur.ai/free/authors/thiago-castro-ferreira/), [Claire Gardent](https://md.research.elamur.ai/free/authors/claire-gardent/), Nikolai Ilinykh, [Chris van der Lee](https://md.research.elamur.ai/free/authors/chris-van-der-lee/), [Simon Mille](https://md.research.elamur.ai/free/authors/simon-mille/), Diego Moussallem, [Anastasia Shimorina](https://md.research.elamur.ai/free/authors/anastasia-shimorina/) · **Year:** 2020 · **Venue:** Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.webnlg-1.7/) · [PDF](https://aclanthology.org/2020.webnlg-1.7.pdf) · [OpenAlex](https://openalex.org/W3131933120)

**Citations:** 62 citations (openalex-snapshot · 2026-10-14) · [101](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
WebNLG+ 2020 extends the earlier WebNLG benchmark, which pairs sets of DBpedia RDF triples with texts. The original task was English generation only. This edition adds Russian and the reverse direction, giving four tracks: RDF-to-English, RDF-to-Russian, English-to-RDF and Russian-to-RDF. The organisers released WebNLG 3.0 for English, with cleaned texts, more categories and new test sets that separate seen categories, unseen entities and unseen categories. The Russian data was machine-translated from WebNLG 2.0 and then post-edited by crowdworkers using entity-alignment links. Fifteen teams took part. Generation systems were scored with automatic metrics and human judgments. Semantic parsers were scored with precision, recall and F1 at several matching levels.

The main findings are:
- Fine-tuned pretrained models such as T5 and BART produced the most fluent output, and several systems were clustered with the human references.
- Rule-based systems matched humans on adequacy.
- Embedding-based metrics correlated better with human ratings than n-gram overlap metrics.
- Every system lost quality on unseen data.

The resource matters because it is an aligned bilingual testbed connecting NLG and Semantic Web research.

## Key points
- **English WebNLG 3.0 size:**
  - Train: 13,211 triple sets, 35,426 texts.
  - Dev: 1,667 triple sets.
  - Test: 1,779 triple sets for D2T and 752 for SP.
  - Coverage: 16 training categories (Company is new) and three unseen test categories (Film, Scientist, MusicalWork).
  - Unseen properties: 39 of the 220 properties in the D2T test set never appear in training or dev.
- **Russian WebNLG:**
  - Covers nine categories, with 5,573 training triple sets and 14,239 texts.
  - Built by MT (Sennrich et al.), then two-step post-editing on Yandex.Toloka.
  - Entity links between the languages come from DBpedia sameAs or were added manually.
  - The test set contains seen categories only.
- **Participation:** 17 teams submitted 48 runs. After two withdrawals, 15 teams with 46 runs remained. Only bt5 entered all four tracks.
- **Metric findings:** BERTScore and BLEURT correlated better with human ratings than BLEU and METEOR. BART-based systems lost data coverage, while T5-based ones stayed close to rule-based systems on that criterion.
- **Text-to-RDF:** All English parsers beat the Stanford CoreNLP OpenIE baseline, with Amazon AI (Shanghai) and bt5 strongest. Performance dropped on unseen entities and categories.

## Interesting ideas
- **Three-way test split (seen categories / unseen entities / unseen categories):** This separates memorisation from real generalisation. It would suit any Ukrainian data-to-text or knowledge-base task.
- **MT plus crowd post-editing with entity pointers:** Translators were given pre-aligned entity names, for example via sameAs links, so names stay consistent across the corpus. This is a cheap way to build a parallel resource in a new language, such as Ukrainian, from an English dataset.
- **Releasing results anonymously first:** Teams could withdraw before names were attached. This lowers the barrier to joining a shared task.
- **Reporting several metric families against human ratings:** Testing n-gram and learned metrics side by side shows which automatic scores can be trusted for a given language or task.
- **Graded parsing evaluation (Exact, Ent Type, Partial, Strict):** Scoring triple extraction at several strictness levels gives partial credit and helps diagnose errors better than one exact-match score.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Machine translation](https://md.research.elamur.ai/free/shelves/machine-translation/) · [Natural language processing](https://md.research.elamur.ai/free/shelves/nlp/)

## Cited by

Cited by

101 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 96 not yet.

- [Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced Language Model Pre-training](https://md.research.elamur.ai/free/papers/agarwal-2021-knowledge-graph-based-synthetic-corpus/)
- [The jsRealB Text Realizer: Organization and Use Cases -- Revised version](https://md.research.elamur.ai/free/papers/lapalme-2020-jsrealb-text-realizer-organization-use/)
- [A Comprehensive Overview of Large Language Models](https://md.research.elamur.ai/free/papers/naveed-2023-comprehensive-overview-large-language-models/)
- [UL2: Unifying Language Learning Paradigms](https://md.research.elamur.ai/free/papers/tay-2022-ul2-unifying-language-learning-paradigms/)
- [GLM-130B: An Open Bilingual Pre-trained Model](https://md.research.elamur.ai/free/papers/zeng-2022-glm-130b-open-bilingual-pre/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ferreira20202020,
  title = {{The 2020 Bilingual, Bi-Directional WebNLG+ Shared Task: Overview and Evaluation Results (WebNLG+ 2020)}},
  author = {Ferreira, Thiago Castro and Gardent, Claire and Ilinykh, Nikolai and Lee, Chris van der and Mille, Simon and Moussallem, Diego and Shimorina, Anastasia},
  booktitle = {Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+)},
  year = {2020},
  doi = {10.5281/zenodo.6552784},
  url = {https://doi.org/10.5281/zenodo.6552784}
}
```

- [← previous TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP](https://md.research.elamur.ai/free/papers/morris-2020-textattack-framework-adversarial-attacks-data/)
- [next → The ADAPT Enhanced Dependency Parser at the IWPT 2020 Shared Task](https://md.research.elamur.ai/free/papers/barry-2020-adapt-enhanced-dependency-parser-iwpt/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/papers/2020/page/4/)

## References

41 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Oshin Agarwal, Mihir Kale, Heming Ge, Siamak Shakeri, and Rami Al-Rfou. 2020. Machine translation aided bilingual data-to-text generation and semantic parsing. In Proceedings of the 3rd WebNLG Workshop on Natural Language Generation from the Semantic Web (WebNLG+ 2020), Dublin, Ireland (Virtual). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Oshin+Agarwal%2C+Mihir+Kale%2C+Heming+Ge%2C+Siamak+Shakeri%2C+and+Rami+Al%2DRfou%2E+2020%2E+Machine+translation+aided+bilingual+data%2Dto%2Dtext+generation+and+semantic+parsing%2E+In+Proceedings+of+the+3rd+WebNLG+Worksho)
2. Olga Babko-Malaya. 2005. Guidelines. [find](https://scholar.google.com/scholar?q=Olga+Babko%2DMalaya%2E+2005%2E+Guidelines%2E)
3. Propbank Annotation David Batista. 2018. Named-entity evaluation metrics based on entity-level. Retrieved from [http://www.davidsbatista.net/blog/](http://www.davidsbatista.net/blog/) 2018/05/09/Named_Entity_Evaluation/ on November 14, 2020. [↗](http://www.davidsbatista.net/blog/)
4. Pavel Blinov. 2020. Semantic triples verbalization with generative pre-training model. In Proceedings of the 3rd WebNLG Workshop on Natural Language Generation from the Semantic Web (WebNLG+ 2020), Dublin, Ireland (Virtual). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Pavel+Blinov%2E+2020%2E+Semantic+triples+verbalization+with+generative+pre%2Dtraining+model%2E+In+Proceedings+of+the+3rd+WebNLG+Workshop+on+Natural+Language+Generation+from+the+Semantic+Web+%28WebNLG%2B+2020%29%2C+Du)
5. Boxing Chen and Colin Cherry. 2014. A systematic comparison of smoothing techniques for sentence-level BLEU. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 362–367, Baltimore, Maryland, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Boxing+Chen+and+Colin+Cherry%2E+2014%2E+A+systematic+comparison+of+smoothing+techniques+for+sentence%2Dlevel+BLEU%2E+In+Proceedings+of+the+Ninth+Workshop+on+Statistical+Machine+Translation%2C+pages+362%E2%80%93367%2C+Bal) [our card](https://md.research.elamur.ai/free/papers/chen-2014-systematic-comparison-smoothing-techniques-sentence/)
6. Oriol Domingo Roig, David Bergés Lladó, Roser Can-tenys Sabà, Roger Creus Castanyer, and José Adrián Rodrı́guez Fonollosa. 2020. The upc rdf-to-text system at webnlg challenge 2020. In Proceedings of the 3rd WebNLG Workshop on Natural Language Generation from the Semantic Web (WebNLG+ 2020), Dublin, Ireland (Virtual). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Oriol+Domingo+Roig%2C+David+Berge%CC%81s+Llado%CC%81%2C+Roser+Can%2Dtenys+Saba%CC%80%2C+Roger+Creus+Castanyer%2C+and+Jose%CC%81+Adria%CC%81n+Rodr%C4%B1%CC%81guez+Fonollosa%2E+2020%2E+The+upc+rdf%2Dto%2Dtext+system+at+webnlg+challenge+2020%2E+In+Proceeding)
7. Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017a. Creating training corpora for NLG micro-planners. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 179–188, Vancouver, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Claire+Gardent%2C+Anastasia+Shimorina%2C+Shashi+Narayan%2C+and+Laura+Perez%2DBeltrachini%2E+2017a%2E+Creating+training+corpora+for+NLG+micro%2Dplanners%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+) [our card](https://md.research.elamur.ai/free/papers/gardent-2017-creating-training-corpora-nlg-micro/)
8. Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017b. The WebNLG challenge: Generating text from RDF data. In Proceedings of the 10th International Conference on Natural Language Generation, pages 124–133, Santiago de Compostela, Spain. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Claire+Gardent%2C+Anastasia+Shimorina%2C+Shashi+Narayan%2C+and+Laura+Perez%2DBeltrachini%2E+2017b%2E+The+WebNLG+challenge%3A+Generating+text+from+RDF+data%2E+In+Proceedings+of+the+10th+International+Conference+on+Nat) [our card](https://md.research.elamur.ai/free/papers/gardent-2017-webnlg-challenge-generating-text-rdf/)
9. Qipeng Guo, Zhijing Jin, Ning Dai, Xipeng Qiu, Xiangyang Xue, David Wipf, and Zheng Zhang. 2020a. [find](https://scholar.google.com/scholar?q=Qipeng+Guo%2C+Zhijing+Jin%2C+Ning+Dai%2C+Xipeng+Qiu%2C+Xiangyang+Xue%2C+David+Wipf%2C+and+Zheng+Zhang%2E+2020a%2E)
10. Improving data-to-text generation by pre-training and planning. In Proceedings of the 3rd WebNLG Workshop on Natural Language Generation from the Semantic Web (WebNLG+ 2020), Dublin, Ireland (Virtual). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Improving+data%2Dto%2Dtext+generation+by+pre%2Dtraining+and+planning%2E+In+Proceedings+of+the+3rd+WebNLG+Workshop+on+Natural+Language+Generation+from+the+Semantic+Web+%28WebNLG%2B+2020%29%2C+Dublin%2C+Ireland+%28Virtual%29)
11. Qipeng Guo, Zhijing Jin, Xipeng Qiu, Weinan Zhang, David Wipf, and Zheng Zhang. 2020b. Cyclegt: Unsupervised graph-to-text and text-to-graph generation via cycle training. In Proceedings of the 3rd WebNLG Workshop on Natural Language Generation from the Semantic Web (WebNLG+ 2020), Dublin, Ireland (Virtual). Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2006.04702)
12. Zdenek Kasner and Ondrej Dusek. 2020. Train hard, finetune easy: Multilingual denoising for rdf-to-text generation. In Proceedings of the 3rd WebNLG Workshop on Natural Language Generation from the Semantic Web (WebNLG+ 2020), Dublin, Ireland (Virtual). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Zdenek+Kasner+and+Ondrej+Dusek%2E+2020%2E+Train+hard%2C+finetune+easy%3A+Multilingual+denoising+for+rdf%2Dto%2Dtext+generation%2E+In+Proceedings+of+the+3rd+WebNLG+Workshop+on+Natural+Language+Generation+from+the+Se)
13. Natthawut Kertkeidkachorn and Hiroya Takamura. 2020. Text-to-text pre-training model with plan selection for rdf-to-text generation. In Proceedings of the 3rd WebNLG Workshop on Natural Language Generation from the Semantic Web (WebNLG+ 2020), Dublin, Ireland (Virtual). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Natthawut+Kertkeidkachorn+and+Hiroya+Takamura%2E+2020%2E+Text%2Dto%2Dtext+pre%2Dtraining+model+with+plan+selection+for+rdf%2Dto%2Dtext+generation%2E+In+Proceedings+of+the+3rd+WebNLG+Workshop+on+Natural+Language+Gener)
14. Guy Lapalme. 2020. Rdfg: a symbolic approach for generating text from rdf triples. In Proceedings of the 3rd WebNLG Workshop on Natural Language Generation from the Semantic Web (WebNLG+ 2020), Dublin, Ireland (Virtual). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Guy+Lapalme%2E+2020%2E+Rdfg%3A+a+symbolic+approach+for+generating+text+from+rdf+triples%2E+In+Proceedings+of+the+3rd+WebNLG+Workshop+on+Natural+Language+Generation+from+the+Semantic+Web+%28WebNLG%2B+2020%29%2C+Dublin)
15. Alon Lavie and Abhaya Agarwal. 2007. METEOR: An automatic metric for MT evaluation with high levels of correlation with human judgments. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 228–231, Prague, Czech Republic. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Alon+Lavie+and+Abhaya+Agarwal%2E+2007%2E+METEOR%3A+An+automatic+metric+for+MT+evaluation+with+high+levels+of+correlation+with+human+judgments%2E+In+Proceedings+of+the+Second+Workshop+on+Statistical+Machine+Tr) [our card](https://md.research.elamur.ai/free/papers/lavie-2007-meteor-automatic-metric-mt-evaluation/)
16. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.acl-main.703) [our card](https://md.research.elamur.ai/free/papers/lewis-2020-bart-denoising-sequence-sequence-pre/)
17. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. [↗](https://doi.org/10.1162/tacl_a_00343)
18. Edward Loper and Steven Bird. 2002. NLTK: the natural language toolkit. CoRR, cs.CL/0205028. [↗](https://doi.org/10.48550/arxiv.cs/0205028)
19. Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55–60, Baltimore, Maryland. Association for Computational Linguistics. [↗](https://doi.org/10.3115/v1/p14-5010) [our card](https://md.research.elamur.ai/free/papers/manning-2014-stanford-corenlp-natural-language-processing/)
20. Simon Mille, Stamatia Dasiopoulou, Beatriz Fisas, and Leo Wanner. 2019a. Teaching FORGe to verbalize DBpedia properties in Spanish. In Proceedings of the 12th International Conference on Natural Language Generation, pages 473–483, Tokyo, Japan. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Simon+Mille%2C+Stamatia+Dasiopoulou%2C+Beatriz+Fisas%2C+and+Leo+Wanner%2E+2019a%2E+Teaching+FORGe+to+verbalize+DBpedia+properties+in+Spanish%2E+In+Proceedings+of+the+12th+International+Conference+on+Natural+Langu)
21. Simon Mille, Stamatia Dasiopoulou, and Leo Wanner. 2019b. A portable grammar-based nlg system for verbalization of structured data. In Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing, pages 1054–1056. ACM. [find](https://scholar.google.com/scholar?q=Simon+Mille%2C+Stamatia+Dasiopoulou%2C+and+Leo+Wanner%2E+2019b%2E+A+portable+grammar%2Dbased+nlg+system+for+verbalization+of+structured+data%2E+In+Proceedings+of+the+34th+ACM%2FSIGAPP+Symposium+on+Applied+Computing)
22. Sebastien Montella, Betty Fabre, Lina Maria Rojas-Barahona, Johannes Heinecke, and Tanguy Urvoy. 2020. Denoising pre-training and data augmentation strategies for enhanced rdf verbalization with transformers. In Proceedings of the 3rd WebNLG Workshop on Natural Language Generation from the Semantic Web (WebNLG+ 2020), Dublin, Ireland (Virtual). Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2012.00571)
23. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. [↗](https://doi.org/10.3115/1073083.1073135) [our card](https://md.research.elamur.ai/free/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
24. Nivranshu Pasricha, Mihael Arcan, and Paul Buitelaar. 2020. Nuig-dsi at the webnlg+ challenge: Leveraging transfer learning for rdf-to-text generation. In Proceedings of the 3rd WebNLG Workshop on Natural Language Generation from the Semantic Web (WebNLG+ 2020), Dublin, Ireland (Virtual). Association for Computational Linguistics. [↗](https://doi.org/10.13025/21325)
25. Maja Popović. 2017. chrF++: words helping character n-grams. In Proceedings of the Second Conference on Machine Translation, pages 612–618, Copenhagen, Denmark. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Maja+Popovic%CC%81%2E+2017%2E+chrF%2B%2B%3A+words+helping+character+n%2Dgrams%2E+In+Proceedings+of+the+Second+Conference+on+Machine+Translation%2C+pages+612%E2%80%93618%2C+Copenhagen%2C+Denmark%2E+Association+for+Computational+Linguist) [our card](https://md.research.elamur.ai/free/papers/popovic-2017-chrf-words-helping-character-n/)
26. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. [find](https://scholar.google.com/scholar?q=Peng+Qi%2C+Yuhao+Zhang%2C+Yuhui+Zhang%2C+Jason+Bolton%2C+and+Christopher+D%2E+Manning%2E+2020%2E+Stanza%3A+A+python+natural+language+processing+toolkit+for+many+human+languages%2E) [our card](https://md.research.elamur.ai/free/papers/ud-2020-qi-stanza-python-natural-language-processing/)
27. A. Radford, Jeffrey Wu, R. Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. [find](https://scholar.google.com/scholar?q=A%2E+Radford%2C+Jeffrey+Wu%2C+R%2E+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E)
28. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67. [↗](https://doi.org/10.48550/arxiv.1910.10683)
29. Ehud Reiter and Robert Dale. 2000. Building natural language generation systems. Cambridge university press. [↗](https://doi.org/10.1017/cbo9780511519857)
30. Isabel Segura-Bedmar, Paloma Martı́nez, and Marı́a Herrero-Zazo. 2013. SemEval-2013 task 9 : Extraction of drug-drug interactions from biomedical texts (DDIExtraction 2013). In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 341–350, Atlanta, Georgia, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Isabel+Segura%2DBedmar%2C+Paloma+Mart%C4%B1%CC%81nez%2C+and+Mar%C4%B1%CC%81a+Herrero%2DZazo%2E+2013%2E+SemEval%2D2013+task+9+%3A+Extraction+of+drug%2Ddrug+interactions+from+biomedical+texts+%28DDIExtraction+2013%29%2E+In+Second+Joint+Conference)
31. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Thibault+Sellam%2C+Dipanjan+Das%2C+and+Ankur+Parikh%2E+2020%2E+BLEURT%3A+Learning+robust+metrics+for+text+generation%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+)
32. Rico Sennrich, Alexandra Birch, Anna Currey, Ulrich Germann, Barry Haddow, Kenneth Heafield, Antonio Valerio Miceli Barone, and Philip Williams. 2017. The University of Edinburgh’s Neural MT Systems for WMT17. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers, Copenhagen, Denmark. [↗](https://doi.org/10.48550/arxiv.1708.00726) [our card](https://md.research.elamur.ai/free/papers/sennrich-2017-university-edinburgh-s-neural-mt/)
33. Anastasia Shimorina, Elena Khasanova, and Claire Gardent. 2019. Creating a corpus for Russian data-to-text generation using neural machine translation and post-editing. In Proceedings of the 7th Workshop on Balto-Slavic Natural Language Processing, pages 44–49, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Anastasia+Shimorina%2C+Elena+Khasanova%2C+and+Claire+Gardent%2E+2019%2E+Creating+a+corpus+for+Russian+data%2Dto%2Dtext+generation+using+neural+machine+translation+and+post%2Dediting%2E+In+Proceedings+of+the+7th+Works)
34. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In In Proceedings of Association for Machine Translation in the Americas, pages 223–231. [find](https://scholar.google.com/scholar?q=Matthew+Snover%2C+Bonnie+Dorr%2C+Richard+Schwartz%2C+Linnea+Micciulla%2C+and+John+Makhoul%2E+2006%2E+A+study+of+translation+edit+rate+with+targeted+human+annotation%2E+In+In+Proceedings+of+Association+for+Machine+T) [our card](https://md.research.elamur.ai/free/papers/snover-2006-study-translation-edit-rate-targeted/)
35. Marco Antonio Sobrevilla Cabezudo and Thiago Alexandre Salgueiro Pardo. 2020. Nilc at webnlg+: Pretrained sequence-to-sequence models on rdf-to-text generation. In Proceedings of the 3rd WebNLG Workshop on Natural Language Generation from the Semantic Web (WebNLG+ 2020), Dublin, Ireland (Virtual). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Marco+Antonio+Sobrevilla+Cabezudo+and+Thiago+Alexandre+Salgueiro+Pardo%2E+2020%2E+Nilc+at+webnlg%2B%3A+Pretrained+sequence%2Dto%2Dsequence+models+on+rdf%2Dto%2Dtext+generation%2E+In+Proceedings+of+the+3rd+WebNLG+Worksh)
36. Trung Tran and Dang Tuan Nguyen. 2020. Webnlg 2020 challenge: Semantic template mining for generating references from rdf. In Proceedings of the 3rd WebNLG Workshop on Natural Language Generation from the Semantic Web (WebNLG+ 2020), Dublin, Ireland (Virtual). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Trung+Tran+and+Dang+Tuan+Nguyen%2E+2020%2E+Webnlg+2020+challenge%3A+Semantic+template+mining+for+generating+references+from+rdf%2E+In+Proceedings+of+the+3rd+WebNLG+Workshop+on+Natural+Language+Generation+from)
37. Xintong, Aleksandre Maskharashvili, Symon Jory Stevens-Guille, and Michael White. 2020. Leveraging large pretrained models for webnlg 2020. In Proceedings of the 3rd WebNLG Workshop on Natural Language Generation from the Semantic Web (WebNLG+ 2020), Dublin, Ireland (Virtual). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Xintong%2C+Aleksandre+Maskharashvili%2C+Symon+Jory+Stevens%2DGuille%2C+and+Michael+White%2E+2020%2E+Leveraging+large+pretrained+models+for+webnlg+2020%2E+In+Proceedings+of+the+3rd+WebNLG+Workshop+on+Natural+Languag)
38. Zixiaofan Yang, Arash Einolghozati, Hakan Inan, Angela Fan, Pinar Donmez, and Sonal Gupta. 2020. Improving text-to-text pre-trained models for the graph-to-text task. In Proceedings of the 3rd WebNLG Workshop on Natural Language Generation from the Semantic Web (WebNLG+ 2020), Dublin, Ireland (Virtual). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Zixiaofan+Yang%2C+Arash+Einolghozati%2C+Hakan+Inan%2C+Angela+Fan%2C+Pinar+Donmez%2C+and+Sonal+Gupta%2E+2020%2E+Improving+text%2Dto%2Dtext+pre%2Dtrained+models+for+the+graph%2Dto%2Dtext+task%2E+In+Proceedings+of+the+3rd+WebNLG+)
39. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. [↗](https://doi.org/10.48550/arxiv.1904.09675)
40. Chao Zhao, Marilyn Walker, and Snigdha Chaturvedi. 2020. Bridging the structural gap between encoding and decoding for data-to-text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, volume 1. [find](https://scholar.google.com/scholar?q=Chao+Zhao%2C+Marilyn+Walker%2C+and+Snigdha+Chaturvedi%2E+2020%2E+Bridging+the+structural+gap+between+encoding+and+decoding+for+data%2Dto%2Dtext+generation%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Associa)
41. Giulio Zhou and Gerasimos Lampouras. 2020. Webnlg challenge 2020: Language agnostic delexicalisation for multilingual rdf-to-text generation. In Proceedings of the 3rd WebNLG Workshop on Natural Language Generation from the Semantic Web (WebNLG+ 2020), Dublin, Ireland (Virtual). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Giulio+Zhou+and+Gerasimos+Lampouras%2E+2020%2E+Webnlg+challenge+2020%3A+Language+agnostic+delexicalisation+for+multilingual+rdf%2Dto%2Dtext+generation%2E+In+Proceedings+of+the+3rd+WebNLG+Workshop+on+Natural+Langu)