--- title: "Semantic Equivalence Detection: Are Interrogatives Harder than" description: "The paper asks whether detecting semantic equivalence is harder for questions than for statements. 10 references, cited by 3 papers here." authors: ["João Rodrigues", "Chakaveh Saedi", "António Branco", "João Silva"] year: 2018 venue: "Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)" source: "https://aclanthology.org/L18-1513/" licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)" licence_url: "https://creativecommons.org/licenses/by-nc/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/rodrigues-2018-semantic-equivalence-detection-are-interrogatives/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/rodrigues-2018-semantic-equivalence-detection-are-interrogatives/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/6/) / Semantic Equivalence Detection: Are Interrogatives Harder than Declaratives? # Semantic Equivalence Detection: Are Interrogatives Harder than Declaratives? **Authors:** João Rodrigues, Chakaveh Saedi, [António Branco](https://md.research.elamur.ai/free/en/authors/antonio-branco/) and João Silva · **Year:** 2018 · **Venue:** [Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)](https://md.research.elamur.ai/free/en/venues/lrec/) ([LREC 2018](https://md.research.elamur.ai/free/en/venues/lrec/2018/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/) **Links:** [Publisher's page](https://aclanthology.org/L18-1513/) · [PDF](https://aclanthology.org/L18-1513.pdf) · [OpenAlex](https://openalex.org/W2806885203) **Citations:** 6 citations (OpenAlex, 2026-10-14) · 3 in our library, of the 141,840 papers whose bibliographies we read ## Summary The paper asks whether detecting semantic equivalence is harder for questions than for statements. Paraphrase detection between declaratives and Duplicate Question Detection (DQD) are both forms of Semantic Text Similarity (STS), but the authors note that synonymy works differently for each. Two statements are equivalent when swapping them keeps truth conditions intact. Two questions are equivalent when every good answer to one also answers the other. To compare the cases fairly, they build equal-sized, class-balanced subsets from four English corpora: MSRPC (declaratives), Quora (interrogatives), DupStack (mixed), and PIT (tweets). They then run the established DQD approaches from their earlier work on each subset, with DCNN emerging as the strongest. Three findings stand out: - Duplicate question pairs share less surface material than paraphrase pairs, as measured with BLEU. - Resolvers score clearly lower on questions than on statements. - Tweets are the hardest segment type of all. They also show that adding cheaply harvested question pairs to declarative training data improves declarative paraphrase detection, but only if the original declarative data stays in the mix. This matters when comparing scores across datasets of different segment types. ## Key points - Each of the four corpora (MSRPC, Quora, DupStack, PIT) contributes 7,800 pairs, split evenly into 3,900 duplicates and 3,900 non-duplicates. The data is divided 80/20 into training and test sets. - MSRPC has only 1,901 genuine negatives. The authors created the remaining 1,999 by randomly pairing sentences drawn from different pairs. - In cross-corpus experiments, DCNN performs best overall. Interrogatives prove harder than declaratives, and tweets are the hardest. - Results on MSRPC test data (accuracy) by training set: | Training data | Accuracy | |---|---| | MSRPC alone | 78.42 | | Quora100k alone | 74.42 | | MSRPC + Quora100k | 79.36 | | All four sets merged | 79.42 | - Merging datasets helps resolvers for declaratives but not resolvers for interrogatives. ## Interesting ideas - Equivalence of statements and equivalence of questions need different tests: substitution that preserves truth for statements, interchangeable answers for questions. - Before claiming one input type is harder than another, the authors equalize dataset size and class balance across corpora, so differences reflect the input type rather than the data conditions. - When natural negative examples are scarce, extra negatives can be made by recombining sentences from unrelated pairs. - Measuring surface overlap (BLEU) between the two members of a pair is a quick diagnostic of how much a task rewards shallow matching. - A small hand-annotated set can be enlarged with a large free byproduct dataset of a related type, but only while the in-type data stays in the mix; the out-of-type data alone performed worse. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (answer): Two user questions could be treated as the same when they resolve to the same answer set in the world of the text, instead of comparing their wording. - **I2** (learn): Evaluations that compare genres (fables, school tests, word problems) should use matched-size, balanced samples so the scores are actually comparable. - **I3** (learn): This is a cheap way to generate negative controls for frame or coreference matchers, though such easy negatives should be reported separately from hard ones. - **I4** (back): The same overlap measure could check that text → graph → text paraphrases change the wording while keeping the meaning, so the system is not just copying the input. - **I5** (learn): When adding silver or out-of-genre data to gold treebanks or frame annotations, the gold data should always be kept in training, and a gold-only baseline should be tracked. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [University of Lisbon](https://md.research.elamur.ai/free/en/universities/university-of-lisbon/) — University, Lisbon, Portugal · 20 works · 3,477 citations Also on the byline: [University of Minho](https://ror.org/037wpkx04). 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works ## Cite Cite as: João Rodrigues, Chakaveh Saedi, António Branco and João Silva. 2018. Semantic Equivalence Detection: Are Interrogatives Harder than Declaratives?. Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{rodrigues2018semantic, title = {{Semantic Equivalence Detection: Are Interrogatives Harder than Declaratives?}}, author = {João Rodrigues, Chakaveh Saedi, António Branco and João Silva}, booktitle = {Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)}, year = {2018}, doi = {10.63317/337qsjbk3jsg}, url = {https://doi.org/10.63317/337qsjbk3jsg} } ``` - [← previous GenDR: A Generic Deep Realizer with Complex Lexicalization](https://md.research.elamur.ai/free/en/papers/lareau-2018-gendr-generic-deep-realizer-complex/) - [next → Universal Morpho-Syntactic Parsing and the Contribution of Lexica: Analyzing the ONLP Lab Submission to the CoNLL 2018 Shared Task](https://md.research.elamur.ai/free/en/papers/seker-2018-universal-morpho-syntactic-parsing-contribution/) - [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/6/) ## References 10 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Agirre, E., Gonzalez-Agirre, A., Cer, D., and Diab, M. (2012). [Semeval-2012 task 6: A pilot on semantic textual similarity](https://md.research.elamur.ai/free/en/papers/agirre-2012-semeval-2012-task-6-pilot/). In Proceedings of the 1st Joint Conference on Lexical and Computational Semantics (*SEM 2012), pages 385–393. 2. Bogdanova, D., dos Santos, C. N., Barbosa, L., and Zadrozny, B. (2015). Detecting semantically equivalent questions in online user forums. In Proceedings of the 19th Conference on Computational Natural Language Learning (CoNLL), pages 123–131. [find](https://scholar.google.com/scholar?q=Bogdanova%2C+D%2E%2C+dos+Santos%2C+C%2E+N%2E%2C+Barbosa%2C+L%2E%2C+and+Zadrozny%2C+B%2E+%282015%29%2E+Detecting+semantically+equivalent+questions+in+online+user+forums%2E+In+Proceedings+of+the+19th+Conference+on+Computational+Natura) 3. Nakov, P., Marquez, L., Moschitti, A., Magdy, W., Mubarak, H., Freihat, A. A., Glass, J., and Randeree, B. (2016). [Semeval-2016 task 3: Community question answering](https://md.research.elamur.ai/free/en/papers/nakov-2016-semeval-2016-task-3-community/). In Proceedings of the 11th International Conference on Semantic Evaluation (SemEval), pages 27–48. 4. Rodrigues, J., Saedi, C., Maraev, V., Silva, J., and Branco, A. (2017). Ways of asking and replying in duplicate question detection. In Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (*SEM 2017). [find](https://scholar.google.com/scholar?q=Rodrigues%2C+J%2E%2C+Saedi%2C+C%2E%2C+Maraev%2C+V%2E%2C+Silva%2C+J%2E%2C+and+Branco%2C+A%2E+%282017%29%2E+Ways+of+asking+and+replying+in+duplicate+question+detection%2E+In+Proceedings+of+the+6th+Joint+Conference+on+Lexical+and+Computati) 5. Saedi, C., Rodrigues, J., Silva, J., Branco, A., and Maraev, V. (2017). Learning profiles in duplicate question detection. In Proceedings of the IEEE 16th International Conference on Information Reuse and Integration (IEEE IRI 2017). [find](https://scholar.google.com/scholar?q=Saedi%2C+C%2E%2C+Rodrigues%2C+J%2E%2C+Silva%2C+J%2E%2C+Branco%2C+A%2E%2C+and+Maraev%2C+V%2E+%282017%29%2E+Learning+profiles+in+duplicate+question+detection%2E+In+Proceedings+of+the+IEEE+16th+International+Conference+on+Information+Reuse) 6. Wu, Y., Zhang, Q., and Huang, X. (2011). Efficient near-duplicate detection for Q&A forum. In Proceedings of the 5th International Joint Conference on Natural Language Processing (IJCNLP), pages 1001–1009. 10. [find](https://scholar.google.com/scholar?q=Wu%2C+Y%2E%2C+Zhang%2C+Q%2E%2C+and+Huang%2C+X%2E+%282011%29%2E+Efficient+near%2Dduplicate+detection+for+Q%26A+forum%2E+In+Proceedings+of+the+5th+International+Joint+Conference+on+Natural+Language+Processing+%28IJCNLP%29%2C+pages+1001%E2%80%93) 7. Language Resource References Dolan, B. and Brockett, C. (2005). [Automatically constructing a corpus of sentential paraphrases](https://md.research.elamur.ai/free/en/papers/dolan-2005-automatically-constructing-corpus-sentential-paraphrases/). In Proceedings of the 3rd International Workshop on Paraphrasing (IWP 2005). 8. Hoogeveen, D., Verspoor, K. M., and Baldwin, T. (2015). CQADupStack: A benchmark data set for community question-answering research. In Proceedings of the 20th Australasian Document Computing Symposium, pages 3:1–3:8. [find](https://scholar.google.com/scholar?q=Hoogeveen%2C+D%2E%2C+Verspoor%2C+K%2E+M%2E%2C+and+Baldwin%2C+T%2E+%282015%29%2E+CQADupStack%3A+A+benchmark+data+set+for+community+question%2Danswering+research%2E+In+Proceedings+of+the+20th+Australasian+Document+Computing+Symposiu) 9. Iyer, S., Dandekar, N., and Csernai, K. (2017). First Quora dataset release: Question pairs. [find](https://scholar.google.com/scholar?q=Iyer%2C+S%2E%2C+Dandekar%2C+N%2E%2C+and+Csernai%2C+K%2E+%282017%29%2E+First+Quora+dataset+release%3A+Question+pairs%2E) 10. Xu, W., Callison-Burch, C., and Dolan, B. (2015). [Semeval-2015 task 1: Paraphrase and semantic similarity in Twitter (PIT)](https://md.research.elamur.ai/free/en/papers/xu-2015-semeval-2015-task-1-paraphrase/). In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval), pages 1– 11. *Updated 24 October 2026*