--- title: "Multi-document multilingual summarization corpus preparation, Part 2 — Elhadad 2013" description: "This is the second part of the report on the MultiLing 2013 corpus, which covers ten languages; this part covers… 12 references, cited by 6 papers here." authors: ["Michael Elhadad", "Sabino Miranda-Jiménez", "Josef Steinberger", "George Giannakopoulos"] year: 2013 venue: "Proceedings of the MultiLing 2013 Workshop on Multilingual Multi-document Summarization" source: "https://aclanthology.org/W13-3102/" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/elhadad-2013-multi-document-multilingual-summarization-corpus/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/elhadad-2013-multi-document-multilingual-summarization-corpus/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2013](https://md.research.elamur.ai/free/en/papers/2013/page/2/) / Multi-document multilingual summarization corpus preparation, Part 2: Czech, Hebrew and Spanish # Multi-document multilingual summarization corpus preparation, Part 2: Czech, Hebrew and Spanish **Authors:** [Michael Elhadad](https://md.research.elamur.ai/free/en/authors/michael-elhadad/); Sabino Miranda-Jiménez; Josef Steinberger; George Giannakopoulos · **Year:** 2013 · **Venue:** Proceedings of the MultiLing 2013 Workshop on Multilingual Multi-document Summarization, pages 13–19 **Link:** [aclanthology.org/W13-3102](https://aclanthology.org/W13-3102/) · library `aclbulk-2013-elhadad-multi-document-multilingual-summarization-corpus` · **Licence:** [CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) — summary only [**14** citations](https://openalex.org/works?filter=cites:W2250833996) OpenAlex, 2026-10-14 · [**6** citations in our library](https://md.research.elamur.ai/free/en/papers/elhadad-2013-multi-document-multilingual-summarization-corpus/cited-by/) of the 140,884 papers whose bibliographies we read · [Proceedings of the MultiLing 2013 Workshop on Multilingual Multi-document Summarization](https://aclanthology.org/volumes/W13-31/) · [licence](https://creativecommons.org/licenses/by-nc-sa/3.0/) - [Publisher's page](https://aclanthology.org/W13-3102/) - [PDF](https://aclanthology.org/W13-3102.pdf) - [OpenAlex](https://openalex.org/W2250833996) **Authors (4)** 1. [Michael Elhadad](https://openalex.org/A5013658527) [ORCID](https://orcid.org/0000-0002-5629-2351) · [Semantic Scholar](https://www.semanticscholar.org/author/1754680) · h 25 · 3,571 citations · 137 works 2. Sabino Miranda-Jiménez 3. [Josef Steinberger](https://openalex.org/A5025269577) [ORCID](https://orcid.org/0000-0003-1707-1895) · [Semantic Scholar](https://www.semanticscholar.org/author/3292561) · h 22 · 1,632 citations · 74 works 4. [George Giannakopoulos](https://openalex.org/A5048495248) [ORCID](https://orcid.org/0000-0003-2459-589X) · [Semantic Scholar](https://www.semanticscholar.org/author/144274464) · h 22 · 1,378 citations · 104 works Citations and author metrics: OpenAlex, 2026-10-14; authors with the same name are not always told apart. ## What it is about This is the second part of the report on the MultiLing 2013 corpus, which covers ten languages; this part covers Czech, Hebrew and Spanish. A topic is a set of news documents describing an event sequence. Each language needed source documents (translated from English where needed), human summaries of 240–250 words, and human grades of the summaries for overall responsiveness on a 1–5 scale. For Hebrew, five clusters of ten articles were translated and summarised by three people; per article, reading took 2.5 minutes, translation 30 minutes and proofreading 18.5 minutes, and the average grades of the three summarisers were 4.80, 4.40 and 4.13. For Spanish, 15 topics of ten documents were translated by 30 students (104.5 man-hours), and the average human grade was 3.8 out of 5. The authors record that ordering texts by date gave the most fluent summaries, that translation needs expert knowledge, and that machine-translated drafts slowed proofreading. ## For Mova - I1. Hebrew spelling variants: the team left plene and deficient spellings, and foreign words that have both a transliterated and a Hebrew form, to Wikipedia's usage. A rule for `languages/he` could map plene and deficient forms to one lemma for lookup while keeping both as surface forms. - I2. Date-ordered event chains: the summarisers found that ordering by publication date gave the most fluent text, but background facts sometimes sat in a later document. In `world`, order the event chain by date, and let a rule move a background event to the front when the first event refers to it. - I3. Translate from scratch when building a gold set: the paper found that editing machine output took longer than translating by hand with dictionaries, and that the first text of each cluster took longest. Any Hebrew or Aramaic gold set Mova builds should follow the same rule, and budget the first items of each topic higher. I1 goes into [languages/he/02-script-and-spelling.md](https://mova.elamur.ai/free/en/docs/languages/he/02-script-and-spelling/). I2 is an idea for the event chain in `world`, with the date rule noted in [languages/he/10-historical-layers.md](https://mova.elamur.ai/free/en/docs/languages/he/10-historical-layers/) only as background. I3 goes into [languages/he/12-mova.md](https://mova.elamur.ai/free/en/docs/languages/he/12-mova/) as a gold-set policy. ## Institutions [🎓**University of West Bohemia in Pilsen** Pilsen · Czechia 8 works · 38,657 citations](https://md.research.elamur.ai/free/en/universities/university-of-west-bohemia-in-pilsen/) · [🎓**Ben-Gurion University of the Negev** Beersheba · Israel 34 works · 2,748 citations](https://md.research.elamur.ai/free/en/universities/ben-gurion-university-of-the-negev/) · [🔬**National Centre of Scientific Research "Demokritos"** Athens · Greece 3 works · 1,677 citations](https://md.research.elamur.ai/free/en/labs/national-centre-of-scientific-research-demokritos/) Topics[Summarization](https://md.research.elamur.ai/free/en/topics/summarization/) · [Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · [Hebrew & Aramaic](https://md.research.elamur.ai/free/en/topics/hebrew-aramaic/) · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) ## Cite Cite as Michael Elhadad, Sabino Miranda-Jiménez, Josef Steinberger and George Giannakopoulos. 2013. Multi-document multilingual summarization corpus preparation, Part 2: Czech, Hebrew and Spanish. Proceedings of the MultiLing 2013 Workshop on Multilingual Multi-document Summarization. The paper itself, not this card. BibTeX from the library's record of the work. ```bibtex @inproceedings{elhadad2013multi, title = {{Multi-document multilingual summarization corpus preparation, Part 2: Czech, Hebrew and Spanish}}, author = {Michael Elhadad, Sabino Miranda-Jiménez, Josef Steinberger and George Giannakopoulos}, booktitle = {Proceedings of the MultiLing 2013 Workshop on Multilingual Multi-document Summarization}, year = {2013}, url = {https://aclanthology.org/W13-3102/} } ``` - [← previous What excludes an Alternative in Coherence Relations?](https://md.research.elamur.ai/free/en/papers/webber-2013-what-excludes-alternative-coherence-relations/) - [next → Multimodality and Dialogue Act Classification in the RoboHelper Project](https://md.research.elamur.ai/free/en/papers/chen-2013-multimodality-dialogue-act-classification-robohelper/) - [↑ back to papers of 2013](https://md.research.elamur.ai/free/en/papers/2013/page/2/) ## References 12 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Hoa Trang Dang and K. Owczarzak. 2009. Overview of the tac 2009 summarization track, Nov. [find](https://scholar.google.com/scholar?q=Hoa+Trang+Dang+and+K%2E+Owczarzak%2E+2009%2E+Overview+of+the+tac+2009+summarization+track%2C+Nov%2E) 2. Michael Elhadad, Sabino Miranda-Jiménez, Josef Steinberger, and George Giannakopoulos. 2013. Multi-document multilingual summarization corpus preparation, part 2: Czech, hebrew and spanish. In MultiLing 2013 Workshop in ACL 2013, Sofia, Bulgaria, August. [our card](https://md.research.elamur.ai/free/en/papers/elhadad-2013-multi-document-multilingual-summarization-corpus/) 3. Brigitte Endres-Niggemeyer and Elisabeth Wansorra. 2004. Making cognitive summarization agents work in a real-world domain. In Proceedings of NLUCS Workshop, pages 86–96. Citeseer. [find](https://scholar.google.com/scholar?q=Brigitte+Endres%2DNiggemeyer+and+Elisabeth+Wansorra%2E+2004%2E+Making+cognitive+summarization+agents+work+in+a+real%2Dworld+domain%2E+In+Proceedings+of+NLUCS+Workshop%2C+pages+86%E2%80%9396%2E+Citeseer%2E) 4. Brigitte Endres-Niggemeyer. 2000. Human-style WWW summarization. Technical report. [find](https://scholar.google.com/scholar?q=Brigitte+Endres%2DNiggemeyer%2E+2000%2E+Human%2Dstyle+WWW+summarization%2E+Technical+report%2E) 5. G. Giannakopoulos, M. El-Haj, B. Favre, M. Litvak, J. Steinberger, and V. Varma. 2011. TAC 2011 MultiLing pilot overview. In TAC 2011 Workshop, Maryland MD, USA, November. [find](https://scholar.google.com/scholar?q=G%2E+Giannakopoulos%2C+M%2E+El%2DHaj%2C+B%2E+Favre%2C+M%2E+Litvak%2C+J%2E+Steinberger%2C+and+V%2E+Varma%2E+2011%2E+TAC+2011+MultiLing+pilot+overview%2E+In+TAC+2011+Workshop%2C+Maryland+MD%2C+USA%2C+November%2E) 6. E. Hovy, C. Y. Lin, L. Zhou, and J. Fukumoto. 2005. Basic elements. [find](https://scholar.google.com/scholar?q=E%2E+Hovy%2C+C%2E+Y%2E+Lin%2C+L%2E+Zhou%2C+and+J%2E+Fukumoto%2E+2005%2E+Basic+elements%2E) 7. Lei Li, Corina Forascu, Mahmoud El-Haj, and George Giannakopoulos. 2013. Multi-document multilingual summarization corpus preparation, part 1: Arabic, english, greek, chinese, romanian. In MultiLing 2013 Workshop in ACL 2013, Sofia, Bulgaria, August. [find](https://scholar.google.com/scholar?q=Lei+Li%2C+Corina+Forascu%2C+Mahmoud+El%2DHaj%2C+and+George+Giannakopoulos%2E+2013%2E+Multi%2Ddocument+multilingual+summarization+corpus+preparation%2C+part+1%3A+Arabic%2C+english%2C+greek%2C+chinese%2C+romanian%2E+In+MultiLing+2) 8. C. Y. Lin. 2004. Rouge: A package for automatic evaluation of summaries. Proceedings of the Workshop on Text Summarization Branches Out (WAS 2004), pages 25–26. [our card](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/) 9. Annie Louis and Ani Nenkova. 2012. Automatically assessing machine summary content without a gold standard. Computational Linguistics, 39(2):267–300, Aug. [our card](https://md.research.elamur.ai/free/en/papers/louis-2013-automatically-assessing-machine-summary-content/) 10. Annie Piolat, Thierry Olive, and Ronald T Kellogg. 2005. Cognitive effort during note taking. Applied Cognitive Psychology, 19(3):291–312. [find](https://scholar.google.com/scholar?q=Annie+Piolat%2C+Thierry+Olive%2C+and+Ronald+T+Kellogg%2E+2005%2E+Cognitive+effort+during+note+taking%2E+Applied+Cognitive+Psychology%2C+19%283%29%3A291%E2%80%93312%2E) 11. Horacio Saggion and Sandra Szasz. 2012. The concisus corpus of event summaries. In LREC, pages 2031–2037. [find](https://scholar.google.com/scholar?q=Horacio+Saggion+and+Sandra+Szasz%2E+2012%2E+The+concisus+corpus+of+event+summaries%2E+In+LREC%2C+pages+2031%E2%80%932037%2E) 12. H. Saggion, J. M. Torres-Moreno, I. Cunha, and E. SanJuan. 2010. Multilingual summarization evaluation without human models. In Proceedings of the 23rd International Conference on Computational Linguistics: Posters, page 1059– 1067. [find](https://scholar.google.com/scholar?q=H%2E+Saggion%2C+J%2E+M%2E+Torres%2DMoreno%2C+I%2E+Cunha%2C+and+E%2E+SanJuan%2E+2010%2E+Multilingual+summarization+evaluation+without+human+models%2E+In+Proceedings+of+the+23rd+International+Conference+on+Computational+Lingui)