---
title: "Similarity or deeper understanding? Analyzing the TED-Q dataset"
description: "The paper asks whether TED-Q, a crowdsourced collection of questions that TED talks implicitly raise in listeners, captures… 27 references."
authors: ["Matthijs Westera", "Jacopo Amidei", "Laia Mayol"]
year: 2020
venue: "Proceedings of the 28th International Conference on Computational Linguistics"
source: "https://aclanthology.org/2020.coling-main.439/"
doi: "10.18653/v1/2020.coling-main.439"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/westera-2020-similarity-or-deeper-understanding-analyzing/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/westera-2020-similarity-or-deeper-understanding-analyzing/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/8/) / Similarity or deeper understanding? Analyzing the TED-Q dataset of evoked questions

# Similarity or deeper understanding? Analyzing the TED-Q dataset of evoked questions

**Authors:** [Matthijs Westera](https://md.research.elamur.ai/free/en/authors/matthijs-westera/), Jacopo Amidei and [Laia Mayol](https://md.research.elamur.ai/free/en/authors/laia-mayol/) · **Year:** 2020 · **Venue:** [Proceedings of the 28th International Conference on Computational Linguistics](https://md.research.elamur.ai/free/en/venues/coling/) ([COLING 2020](https://md.research.elamur.ai/free/en/venues/coling/2020/)) · **Publisher:** [International Committee on Computational Linguistics (ICCL)](https://md.research.elamur.ai/free/en/publishers/international-committee-on-computational-linguistics-iccl/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.coling-main.439/) · [PDF](https://aclanthology.org/2020.coling-main.439.pdf) · [DOI 10.18653/v1/2020.coling-main.439](https://doi.org/10.18653/v1/2020.coling-main.439) · [OpenAlex](https://openalex.org/W3117480590)

**Citations:** 2 citations (OpenAlex, 2026-10-14)

## Summary
The paper asks whether TED-Q, a crowdsourced collection of questions that TED talks implicitly raise in listeners, captures real discourse interpretation or only surface association between a text and its follow-up questions. The authors turn TED-Q into a binary task. Each item pairs a three-sentence context with a question, and a model must decide whether that question was actually evoked at that point. Negatives reuse the same context with a question elicited three to four sentences away. A parallel task is mined from dialogue quotations in the BookCorpus, using explicit questions instead of implicit ones. Random forests built on four similarity measures, alone and combined, are compared with a fine-tuned BERT classifier and with two human annotators. BERT beats every similarity model and their combination, and it scores roughly as well as the humans. This suggests that recognizing evoked questions requires more than lexical or semantic closeness. The finding supports using such data to ground Question Under Discussion (QUD) theories of discourse coherence empirically.

## Key points
- **TED-Q data:** 6 TED talks, 111 crowd workers and 2421 evoked questions, about 5 per sentence. The derived task has 4824 items, balanced between positives and negatives.
- **BookCorpus task:** about 11,000 novels yielded roughly 2 million dialogues and 3.8 million items. Positive items are turns that open with a question and follow a non-question turn.
- **TED-Q results (MCC):**
  - LEMMA OVERLAP 0.47
  - GLEU 0.24
  - MEAN COS (GloVe) 0.14
  - BERTSCORE 0.33
  - ALL SIMS 0.46
  - BERT (BERT-base-cased) 0.55
  - Two human annotators, on 195 items: 0.55 and 0.60, with Cohen's κ = 0.66.
- **Training on BookCorpus:**
  - LEMMA OVERLAP transfers to TED-Q almost unchanged.
  - BERT falls to 0.43 on TED-Q, and the share of its errors that are false positives jumps from 61% to 84%.
  - Explicit BookCorpus questions are harder overall, and BERT's advantage over the similarity models is larger there.
- **Question type matters:** wh-questions are handled far better than polar questions. BERT scores 0.61 vs 0.30, and LEMMA OVERLAP 0.53 vs 0.17.

## Interesting ideas
- Hard negatives come from the same context paired with a question from a few sentences away, a distance tuned by hand so the task is difficult but still solvable; this forces models to use position-specific meaning rather than topic.
- A ladder of similarity-only baselines, running from lemma overlap to contextual embeddings, works as a diagnostic: if a strong model cannot beat them clearly, the dataset may reward association rather than understanding.
- To prevent leakage when contexts overlap heavily, the authors split data in contiguous blocks of 20 contexts and drop test contexts that have training neighbours within 3 sentences.
- Explicit questions in fiction dialogue serve as a cheap, unannotated stand-in for implicit questions, and the finding that lemma reuse dominates in both sources argues against a crowdsourcing artifact.
- Evoked questions behave like candidate QUDs that the next utterance should address, and wh-questions relate to their context differently from polar ones.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Mova's evaluation with negative controls could build distractors from nearby sentences of the same text, so that frame or answer checks cannot pass on topical overlap alone.
- **I2** (learn): before trusting a new silver set or test suite, Mova could run cheap overlap baselines on it and flag items or sets those baselines already solve.
- **I3** (learn): training on treebank or story data with sliding windows should use block-wise splits with neighbour removal, so that evaluation scores are not inflated.
- **I4** (frames): question turns mined from dialogue could supply data on which open questions a statement raises, supporting implicature and coherence modelling in prag.
- **I5** (world): the world of a text could keep a list of open questions, each tied to its triggering entity or event, then mark them resolved when later sentences fill the gap, giving an explainable signal of coherence.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Universitat Pompeu Fabra](https://md.research.elamur.ai/free/en/universities/universitat-pompeu-fabra/) — University, Barcelona, Spain · 36 works · 11,433 citations
- [Leiden University](https://md.research.elamur.ai/free/en/universities/leiden-university/) — University, Leiden, Netherlands · 16 works · 936 citations
- [The Open University](https://md.research.elamur.ai/free/en/universities/the-open-university/) — University, Milton Keynes, United Kingdom · 4 works · 258 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

## Cite

Cite as: Matthijs Westera, Jacopo Amidei and Laia Mayol. 2020. Similarity or deeper understanding? Analyzing the TED-Q dataset of evoked questions. Proceedings of the 28th International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{westera2020similarity,
  title = {{Similarity or deeper understanding? Analyzing the TED-Q dataset of evoked questions}},
  author = {Matthijs Westera, Jacopo Amidei and Laia Mayol},
  booktitle = {Proceedings of the 28th International Conference on Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.coling-main.439},
  url = {https://doi.org/10.18653/v1/2020.coling-main.439}
}
```

- [← previous Quranic Studies Made in Austria: Approaching Quantitative Arabic Linguistics](https://md.research.elamur.ai/free/en/papers/elmaz-2020-quranic-studies-made-austria-approaching/)
- [next → The jsRealB Text Realizer: Organization and Use Cases — Revised version](https://md.research.elamur.ai/free/en/papers/lapalme-2020-jsrealb-text-realizer-organization-use/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/8/)

## References

27 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Davide Chicco and Giuseppe Jurman. 2020. The advantages of the matthews correlation coefficient (mcc) over f1 score and accuracy in binary classification evaluation. BMC genomics, 21(1):6. [↗](https://doi.org/10.1186/s12864-019-6413-7)
2. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. [QuAC: Question answering in context](https://md.research.elamur.ai/free/en/papers/choi-2018-quac-question-answering-context/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2174–2184, Brussels, Belgium, October-November. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1808.07036)
3. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46. [↗](https://doi.org/10.1177/001316446002000104)
4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June. Association for Computational Linguistics.
5. Wei-Jen Ko, Te-Yuan Chen, Yiyan Huang, Greg Durrett, and Junyi Jessy Li. 2020. [Inquisitive question generation for high level text comprehension](https://md.research.elamur.ai/free/en/papers/ko-2020-inquisitive-question-generation-high-level/). In Proceedings of EMNLP.
6. J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1):159–174. [↗](https://doi.org/10.2307/2529310)
7. William C Mann and Sandra A Thompson. 1988. [Rhetorical structure theory: Toward a functional theory of text organization](https://md.research.elamur.ai/free/en/papers/mann-1988-rhetorical-structure-theory-functional/). Text, 8(3):243–281. [↗](https://doi.org/10.1515/text.1.1988.8.3.243)
8. Brian W Matthews. 1975. Comparison of the predicted and observed secondary structure of t4 phage lysozyme. Biochimica et Biophysica Acta (BBA)-Protein Structure, 405(2):442–451. [↗](https://doi.org/10.1016/0005-2795(75)90109-9)
9. Edgar Onea. 2013. Potential questions in discourse and grammar. Manuscript, University of Göttingen. [↗](https://doi.org/10.13140/rg.2.1.1888.1128)
10. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830. [↗](https://doi.org/10.48550/arxiv.1201.0490)
11. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. [Glove: Global vectors for word representation](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/). In Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. [↗](https://doi.org/10.3115/v1/d14-1162)
12. Martin J Pickering and Simon Garrod. 2004. Toward a mechanistic psychology of dialogue. Behavioral and brain sciences, 27(2):169–190. [↗](https://doi.org/10.1017/s0140525x04000056)
13. Valentina Pyatkin, Ayal Klein, Reut Tsarfaty, and Ido Dagan. 2020. [Qadiscourse–discourse relations as qa pairs: Representation, crowdsourcing and baselines](https://md.research.elamur.ai/free/en/papers/pyatkin-2020-qadiscourse-discourse-relations-as-qa/). arXiv preprint arXiv:2010.02815. [↗](https://arxiv.org/abs/2010.02815)
14. Peng Qi, Yuhao Zhang, and Christopher D Manning. 2020. Stay hungry, stay focused: Generating informative and specific questions in information-seeking conversations. arXiv preprint arXiv:2004.14530. [↗](https://arxiv.org/abs/2004.14530)
15. Sudha Rao and Hal Daumé III. 2018. Learning to ask good questions: Ranking clarification questions using neural expected value of perfect information. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2737–2746, Melbourne, Australia, July. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sudha+Rao+and+Hal+Daume%CC%81+III%2E+2018%2E+Learning+to+ask+good+questions%3A+Ranking+clarification+questions+using+neural+expected+value+of+perfect+information%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the)
16. Sudha Rao and Hal Daumé III. 2019. Answer-based Adversarial Training for Generating Clarification Questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 143–155, Minneapolis, Minnesota, June. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1904.02281)
17. Arndt Riester. 2019. Constructing qud trees. In Questions in Discourse, pages 164–193. Brill. [↗](https://doi.org/10.1163/9789004378322_007)
18. Craige Roberts. 1996. Information structure in discourse: Toward a unified theory of formal pragmatics. The Ohio State University Working Papers in Linguistics, 49:91–136. [find](https://scholar.google.com/scholar?q=Craige+Roberts%2E+1996%2E+Information+structure+in+discourse%3A+Toward+a+unified+theory+of+formal+pragmatics%2E+The+Ohio+State+University+Working+Papers+in+Linguistics%2C+49%3A91%E2%80%93136%2E)
19. Luise Schricker and Tatjana Scheffler. 2019. Ranking of potential questions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 143–148. [find](https://scholar.google.com/scholar?q=Luise+Schricker+and+Tatjana+Scheffler%2E+2019%2E+Ranking+of+potential+questions%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%3A+Student+Research+Workshop%2C+page)
20. Jan Van Kuppevelt. 1995. Discourse structure, topicality and questioning. Journal of linguistics, pages 109–147. [↗](https://doi.org/10.1017/s002222670000058x)
21. Christiane Von Stutterheim and Wolfgang Klein. 1989. Referential movement in descriptive and narrative discourse. In North-Holland Linguistic Series: Linguistic Variations, volume 54, pages 39–76. Elsevier. [↗](https://doi.org/10.1016/b978-0-444-87144-2.50005-7)
22. Bonnie Webber, Rashmi Prasad, Alan Lee, and Aravind Joshi. 2019. The penn discourse treebank 3.0 annotation manual. [find](https://scholar.google.com/scholar?q=Bonnie+Webber%2C+Rashmi+Prasad%2C+Alan+Lee%2C+and+Aravind+Joshi%2E+2019%2E+The+penn+discourse+treebank+3%2E0+annotation+manual%2E)
23. Matthijs Westera, Laia Mayol, and Hannah Rohde. 2020. [Ted-q: Ted talks and the questions they evoke](https://md.research.elamur.ai/free/en/papers/westera-2020-ted-q-ted-talks-questions/). In Proceedings of The 12th Language Resources and Evaluation Conference, pages 1118–1127.
24. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. [↗](https://arxiv.org/abs/1609.08144)
25. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675. [↗](https://arxiv.org/abs/1904.09675)
26. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. [Aligning books and movies: Towards story-like visual explanations by watching movies and reading books](https://md.research.elamur.ai/free/en/papers/zhu-2015-aligning-books-movies-towards-story/). In The IEEE International Conference on Computer Vision (ICCV), December. A Supplementary material for “Similarity or deeper understanding? Analyzing the TED-Q dataset of evoked questions” The following lists of examples are not exhaustive, but exemplify some of the patterns we observed, as referenced in the paper. False positives by BERT, coded correctly by both annotators. (1)
27. It was first suggested by Lyman Spitzer, the father of the space telescope, in 1962, and he took his inspiration from an eclipse. You’ve all seen that. That’s a solar eclipse. Question: Do eclipses have any other effects? (2) I feel so fortunate that my first job was working at the Museum of Modern Art on a retrospective of painter Elizabeth Murray. I learned so much from her. Question: What did she think of those early works? (3) [find](https://scholar.google.com/scholar?q=It+was+first+suggested+by+Lyman+Spitzer%2C+the+father+of+the+space+telescope%2C+in+1962%2C+and+he+took+his+inspiration+from+an+eclipse%2E+You%E2%80%99ve+all+seen+that%2E+That%E2%80%99s+a+solar+eclipse%2E+Question%3A+Do+eclipses+ha)

*Updated 24 October 2026*