---
title: "Syntactic Constraints on Paraphrases — Callison-Burch 2008"
description: "This work addresses a weakness of the pivot-based paraphrasing approach of Bannard and Callison-Burch (2005)."
authors: ["Chris Callison-Burch"]
year: 2008
venue: "Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/D08-1021/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/burch-2008-syntactic-constraints-paraphrases-extracted-parallel/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/burch-2008-syntactic-constraints-paraphrases-extracted-parallel/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2008](https://md.research.elamur.ai/free/papers/2008/) / Syntactic Constraints on Paraphrases Extracted from Parallel Corpora

# Syntactic Constraints on Paraphrases Extracted from Parallel Corpora

**Authors:** [Chris Callison-Burch](https://md.research.elamur.ai/free/authors/chris-callison-burch/) · **Year:** 2008 · **Venue:** Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/D08-1021/) · [PDF](https://aclanthology.org/D08-1021.pdf)

**Citations:** [92](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
This work addresses a weakness of the pivot-based paraphrasing approach of Bannard and Callison-Burch (2005). In that approach, two English phrases count as paraphrases when they share a foreign translation in a word-aligned parallel corpus. Because phrase extraction heuristics allow unaligned words at phrase edges, many extracted "paraphrases" are just longer or shorter versions of the original phrase. Swapping these into a sentence usually breaks grammar.

The author parses the English side of the parallel corpora and changes the extraction algorithm (called SBP) so that each bilingual phrase pair also carries a syntactic label. Paraphrase probabilities are then computed only within the same syntactic category. To keep coverage for word sequences that are not constituents, the method adds complex syntactic labels. Constraints are applied during extraction and, optionally, when substituting paraphrases into parsed test sentences.

A manual evaluation on out-of-domain news commentary sentences shows about a 19% absolute gain in paraphrases judged correct in both meaning and grammar. This makes the resource more usable for tasks such as generation, where output must be grammatical.

## Key points
- The baseline, trained on Europarl, produced 3.7 million paraphrases for 400,000 of 900,000 test phrases. 34% of these were sub- or super-strings of the source phrase, and such strings were the top-ranked paraphrase 73% of the time.
- Under the strict criterion (meaning and grammar both correct), the baseline scored .30. The syntactically constrained variants scored up to .55, and grammaticality rose from .35 to as much as .68.
- Adding a language model made output 7–9% more grammatical. However, it cut correct meaning by 12% for the baseline, against only 1–2% for the constrained versions.
- Substitution-time constraints reduced coverage to 876 of 1194 sentences (73%). Extraction-only constraints kept wider coverage with nearly the same meaning scores.
- The manual evaluation covered eight conditions:
  - Exact agreement on the 5-point scale gave Kappa 0.33.
  - Thresholded (correct/incorrect) agreement gave Kappa 0.61.
- The author released the following at http://cs.jhu.edu/~ccb/:
  - the paraphrase sets (3.7M baseline, 3.5M constrained);
  - the code;
  - 10 word-aligned parallel corpora with English parses;
  - the human judgments.

## Interesting ideas
- **Conditioning on a syntactic label.** Making the pivot probabilities depend on the label splits the paraphrase space by category. This cheaply removes cross-category noise (nouns paraphrasing adjectives, for instance) and could carry over to any pivot- or alignment-based lexicon induction.
- **Parsing only one side.** Only the English side of the parallel data needs a parser, so the foreign side requires no syntactic tools. This is useful when only one language in a pair has a good parser.
- **Labels for non-constituents.** Building composite syntactic labels for arbitrary word spans keeps the coverage of purely phrase-based methods while still adding syntax. This compromise is relevant wherever strict constituency would throw away too much data.
- **Diagnosing errors by counting substrings.** Measuring how often outputs are sub- or super-strings of the input is a quick, automatic check for boundary artifacts caused by alignment. It could serve as a sanity metric for other extraction pipelines.
- **Reporting agreement at two granularities.** Giving agreement both on the full scale and on the binary thresholds actually used for scoring is a more honest picture of annotation reliability. It suits any graded human evaluation.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Machine translation](https://md.research.elamur.ai/free/shelves/machine-translation/) · [Natural language processing](https://md.research.elamur.ai/free/shelves/nlp/)

## Cited by

Cited by

92 papers of our library cite it (of the 140673 whose bibliographies we read); 14 of them have a page here, 78 not yet.

- [Collecting Highly Parallel Data for Paraphrase Evaluation](https://md.research.elamur.ai/free/papers/chen-2011-collecting-highly-parallel-data-paraphrase/)
- [Paraphrase-Driven Learning for Open Question Answering](https://md.research.elamur.ai/free/papers/fader-2013-paraphrase-driven-learning-open-question/)
- [Adversarial Example Generation with Syntactically Controlled Paraphrase Networks](https://md.research.elamur.ai/free/papers/iyyer-2018-adversarial-example-generation-syntactically-controlled/)
- [A Continuously Growing Dataset of Sentential Paraphrases](https://md.research.elamur.ai/free/papers/lan-2017-continuously-growing-dataset-sentential-paraphrases/)
- [Generating Phrasal and Sentential Paraphrases: A Survey of Data-Driven Methods](https://md.research.elamur.ai/free/papers/madnani-2010-generating-phrasal-sentential-paraphrases-survey/)
- [Paraphrasing Revisited with Neural Machine Translation](https://md.research.elamur.ai/free/papers/mallinson-2017-paraphrasing-revisited-neural-machine-translation/)
- [Improved Statistical Machine Translation Using Monolingually-Derived Paraphrases](https://md.research.elamur.ai/free/papers/marton-2009-improved-statistical-machine-translation-monolingually/)
- [FrameNet+: Fast Paraphrastic Tripling of FrameNet](https://md.research.elamur.ai/free/papers/pavlick-2015-framenet-fast-paraphrastic-tripling-framenet/)
- [PPDB 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification](https://md.research.elamur.ai/free/papers/pavlick-2015-ppdb-2-0-better-paraphrase/)
- [Realization of Discourse Relations by Other Means: Alternative Lexicalizations](https://md.research.elamur.ai/free/papers/prasad-2010-realization-discourse-relations-other-means/)
- [Dear Sir or Madam, May I Introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer](https://md.research.elamur.ai/free/papers/rao-2018-dear-sir-or-madam-may/)
- [Fluency, Adequacy, or HTER? Exploring Different Human Judgments with a Tunable MT Metric](https://md.research.elamur.ai/free/papers/snover-2009-fluency-adequacy-or-hter-exploring/)
- [Sentence Simplification by Monolingual Machine Translation](https://md.research.elamur.ai/free/papers/wubben-2012-sentence-simplification-monolingual-machine-translation/)
- [Paraphrasing for Style](https://md.research.elamur.ai/free/papers/xu-2012-paraphrasing-style/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{callisonburch2008syntactic,
  title = {{Syntactic Constraints on Paraphrases Extracted from Parallel Corpora}},
  author = {Callison-Burch, Chris},
  booktitle = {Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing},
  year = {2008},
  url = {https://aclanthology.org/D08-1021/}
}
```

- [← previous Statistical Glossing - Language Independent Analysis in Bible Translation](https://md.research.elamur.ai/free/papers/riding-2008-statistical-glossing-language-independent-analysis/)
- [next → Syntactic Models for Structural Word Insertion and Deletion during Translation](https://md.research.elamur.ai/free/papers/menezes-2008-syntactic-models-structural-word-insertion/)
- [↑ back to papers of 2008](https://md.research.elamur.ai/free/papers/2008/)

## References

31 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Colin Bannard and Chris Callison-Burch. 2005. Paraphrasing with bilingual parallel corpora. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Colin+Bannard+and+Chris+Callison%2DBurch%2E+2005%2E+Paraphrasing+with+bilingual+parallel+corpora%2E+In+Proceedings+of+ACL%2E) [our card](https://md.research.elamur.ai/free/papers/bannard-2005-paraphrasing-bilingual-parallel-corpora/)
2. Regina Barzilay and Kathleen McKeown. 2001. Extracting paraphrases from a parallel corpus. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Regina+Barzilay+and+Kathleen+McKeown%2E+2001%2E+Extracting+paraphrases+from+a+parallel+corpus%2E+In+Proceedings+of+ACL%2E) [our card](https://md.research.elamur.ai/free/papers/barzilay-2001-extracting-paraphrases-parallel-corpus/)
3. Dan Bikel. 2002. Design of a multi-lingual, parallel-processing statistical parsing engine. In Proceedings of HLT. [find](https://scholar.google.com/scholar?q=Dan+Bikel%2E+2002%2E+Design+of+a+multi%2Dlingual%2C+parallel%2Dprocessing+statistical+parsing+engine%2E+In+Proceedings+of+HLT%2E)
4. Chris Callison-Burch, Philipp Koehn, and Miles Osborne. 2006. Improved statistical machine translation using paraphrases. In Proceedings of HLT/NAACL. [find](https://scholar.google.com/scholar?q=Chris+Callison%2DBurch%2C+Philipp+Koehn%2C+and+Miles+Osborne%2E+2006%2E+Improved+statistical+machine+translation+using+paraphrases%2E+In+Proceedings+of+HLT%2FNAACL%2E)
5. Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2007. (Meta-) evaluation of machine translation. In Proceedings of the Second Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Chris+Callison%2DBurch%2C+Cameron+Fordyce%2C+Philipp+Koehn%2C+Christof+Monz%2C+and+Josh+Schroeder%2E+2007%2E+%28Meta%2D%29+evaluation+of+machine+translation%2E+In+Proceedings+of+the+Second+Workshop+on+Statistical+Machine+T) [our card](https://md.research.elamur.ai/free/papers/burch-2007-meta-evaluation-machine-translation/)
6. Jean Carletta. 1996. Assessing agreement on classification tasks: The kappa statistic. Computational Linguistics, 22(2):249–254. [↗](https://doi.org/10.48550/arxiv.cmp-lg/9602004) [our card](https://md.research.elamur.ai/free/papers/carletta-1996-assessing-agreement-classification-tasks-kappa/)
7. David Chiang. 2005. A hierarchical phrase-based model for statistical machine translation. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=David+Chiang%2E+2005%2E+A+hierarchical+phrase%2Dbased+model+for+statistical+machine+translation%2E+In+Proceedings+of+ACL%2E) [our card](https://md.research.elamur.ai/free/papers/chiang-2005-hierarchical-phrase-based-model-statistical/)
8. Heidi J. Fox. 2002. Phrasal cohesion and statistical machine translation. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Heidi+J%2E+Fox%2E+2002%2E+Phrasal+cohesion+and+statistical+machine+translation%2E+In+Proceedings+of+EMNLP%2E)
9. Michel Galley, Mark Hopkins, Kevin Knight, and Daniel Marcu. 2004. What’s in a translation rule? In Proceedings of HLT/NAACL. [↗](https://doi.org/10.21236/ada460212) [our card](https://md.research.elamur.ai/free/papers/galley-2004-what-s-translation-rule/)
10. Bryant Huang and Kevin Knight. 2006. Relabeling syntax trees to improve syntax-based machine translation quality. In Proceedings of HLT/NAACL. [find](https://scholar.google.com/scholar?q=Bryant+Huang+and+Kevin+Knight%2E+2006%2E+Relabeling+syntax+trees+to+improve+syntax%2Dbased+machine+translation+quality%2E+In+Proceedings+of+HLT%2FNAACL%2E)
11. Ali Ibrahim, Boris Katz, and Jimmy Lin. 2003. Extracting structural paraphrases from aligned monolingual corpora. In Proceedings of the Second International Workshop on Paraphrasing (ACL 2003). [↗](https://doi.org/10.21236/ada460577)
12. David Kauchak and Regina Barzilay. 2006. Paraphrasing for automatic evaluation. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=David+Kauchak+and+Regina+Barzilay%2E+2006%2E+Paraphrasing+for+automatic+evaluation%2E+In+Proceedings+of+EMNLP%2E) [our card](https://md.research.elamur.ai/free/papers/kauchak-2006-paraphrasing-automatic-evaluation/)
13. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proceedings of HLT/NAACL. [↗](https://doi.org/10.21236/ada461156)
14. Philipp Koehn. 2005. A parallel corpus for statistical machine translation. In Proceedings of MT-Summit, Phuket, Thailand. [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2E+2005%2E+A+parallel+corpus+for+statistical+machine+translation%2E+In+Proceedings+of+MT%2DSummit%2C+Phuket%2C+Thailand%2E)
15. J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33:159–174. [find](https://scholar.google.com/scholar?q=J%2E+Richard+Landis+and+Gary+G%2E+Koch%2E+1977%2E+The+measurement+of+observer+agreement+for+categorical+data%2E+Biometrics%2C+33%3A159%E2%80%93174%2E)
16. Dekang Lin and Colin Cherry. 2002. Word alignment with cohesion constraint. In Proceedings of HLT/NAACL. [find](https://scholar.google.com/scholar?q=Dekang+Lin+and+Colin+Cherry%2E+2002%2E+Word+alignment+with+cohesion+constraint%2E+In+Proceedings+of+HLT%2FNAACL%2E)
17. Dekang Lin and Patrick Pantel. 2001. Discovery of inference rules from text. Natural Language Engineering, 7(3):343–360. [find](https://scholar.google.com/scholar?q=Dekang+Lin+and+Patrick+Pantel%2E+2001%2E+Discovery+of+inference+rules+from+text%2E+Natural+Language+Engineering%2C+7%283%29%3A343%E2%80%93360%2E)
18. Nitin Madnani, Necip Fazil Ayan, Philip Resnik, and Bonnie Dorr. 2007. Using paraphrases for parameter tuning in statistical machine translation. In Proceedings of the ACL Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Nitin+Madnani%2C+Necip+Fazil+Ayan%2C+Philip+Resnik%2C+and+Bonnie+Dorr%2E+2007%2E+Using+paraphrases+for+parameter+tuning+in+statistical+machine+translation%2E+In+Proceedings+of+the+ACL+Workshop+on+Statistical+Mach)
19. Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19–51. [↗](https://doi.org/10.1162/089120103321337421)
20. Franz Josef Och and Hermann Ney. 2004. The alignment template approach to statistical machine translation. Computational Linguistics, 30(4):417–449. [↗](https://doi.org/10.1162/0891201042544884)
21. Karolina Owczarzak, Declan Groves, Josef Van Genabith, and Andy Way. 2006. Contextual bitext-derived paraphrases in automatic MT evaluation. In Proceedings of the SMT Workshop at HLT-NAACL. [find](https://scholar.google.com/scholar?q=Karolina+Owczarzak%2C+Declan+Groves%2C+Josef+Van+Genabith%2C+and+Andy+Way%2E+2006%2E+Contextual+bitext%2Dderived+paraphrases+in+automatic+MT+evaluation%2E+In+Proceedings+of+the+SMT+Workshop+at+HLT%2DNAACL%2E)
22. Bo Pang, Kevin Knight, and Daniel Marcu. 2003. Syntax-based alignment of multiple translations: Extracting paraphrases and generating new sentences. In Proceedings of HLT/NAACL. [find](https://scholar.google.com/scholar?q=Bo+Pang%2C+Kevin+Knight%2C+and+Daniel+Marcu%2E+2003%2E+Syntax%2Dbased+alignment+of+multiple+translations%3A+Extracting+paraphrases+and+generating+new+sentences%2E+In+Proceedings+of+HLT%2FNAACL%2E) [our card](https://md.research.elamur.ai/free/papers/pang-2003-syntax-based-alignment-multiple-translations/)
23. Chris Quirk, Arul Menezes, and Colin Cherry. 2005. Dependency treelet translation: Syntactically informed phrasal smt. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Chris+Quirk%2C+Arul+Menezes%2C+and+Colin+Cherry%2E+2005%2E+Dependency+treelet+translation%3A+Syntactically+informed+phrasal+smt%2E+In+Proceedings+of+ACL%2E) [our card](https://md.research.elamur.ai/free/papers/quirk-2005-dependency-treelet-translation-syntactically-informed/)
24. Stefan Riezler, Alexander Vasserman, Ioannis Tsochantaridis, Vibhu Mittal, and Yi Liu. 2007. Statistical machine translation for query expansion in answer retrieval. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Stefan+Riezler%2C+Alexander+Vasserman%2C+Ioannis+Tsochantaridis%2C+Vibhu+Mittal%2C+and+Yi+Liu%2E+2007%2E+Statistical+machine+translation+for+query+expansion+in+answer+retrieval%2E+In+Proceedings+of+ACL%2E)
25. Mark Steedman. 1999. Alternative quantier scope in ccg. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Mark+Steedman%2E+1999%2E+Alternative+quantier+scope+in+ccg%2E+In+Proceedings+of+ACL%2E)
26. Andreas Stolcke. 2002. SRILM - an extensible language modeling toolkit. In Proceedings of the International Conference on Spoken Language Processing, Denver, Colorado, September. [find](https://scholar.google.com/scholar?q=Andreas+Stolcke%2E+2002%2E+SRILM+%2D+an+extensible+language+modeling+toolkit%2E+In+Proceedings+of+the+International+Conference+on+Spoken+Language+Processing%2C+Denver%2C+Colorado%2C+September%2E)
27. Idan Szpektor, Eyal Shnarch, and Ido Dagan. 2007. Instance-based evaluation of entailment rule acquisition. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Idan+Szpektor%2C+Eyal+Shnarch%2C+and+Ido+Dagan%2E+2007%2E+Instance%2Dbased+evaluation+of+entailment+rule+acquisition%2E+In+Proceedings+of+ACL%2E)
28. Dekai Wu. 1997. Stochastic inversion transduction grammars and bilingual parsing of parallel corpora. Computational Linguistics, 23(3). [find](https://scholar.google.com/scholar?q=Dekai+Wu%2E+1997%2E+Stochastic+inversion+transduction+grammars+and+bilingual+parsing+of+parallel+corpora%2E+Computational+Linguistics%2C+23%283%29%2E) [our card](https://md.research.elamur.ai/free/papers/wu-1997-stochastic-inversion-transduction-grammars-bilingual/)
29. Shiqi Zhao, Cheng Niu, Ming Zhou, Ting Liu, and Sheng Li. 2008a. Combining multiple resources to improve SMT-based paraphrasing model. In Proceedings of ACL/HLT. [find](https://scholar.google.com/scholar?q=Shiqi+Zhao%2C+Cheng+Niu%2C+Ming+Zhou%2C+Ting+Liu%2C+and+Sheng+Li%2E+2008a%2E+Combining+multiple+resources+to+improve+SMT%2Dbased+paraphrasing+model%2E+In+Proceedings+of+ACL%2FHLT%2E)
30. Shiqi Zhao, Haifeng Wang, Ting Liu, and Sheng Li. 2008b. Pivot approach for extracting paraphrase patterns from bilingual corpora. In Proceedings of ACL/HLT. [find](https://scholar.google.com/scholar?q=Shiqi+Zhao%2C+Haifeng+Wang%2C+Ting+Liu%2C+and+Sheng+Li%2E+2008b%2E+Pivot+approach+for+extracting+paraphrase+patterns+from+bilingual+corpora%2E+In+Proceedings+of+ACL%2FHLT%2E)
31. Liang Zhou, Chin-Yew Lin, and Eduard Hovy. 2006. Re-evaluating machine translation results with paraphrase support. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Liang+Zhou%2C+Chin%2DYew+Lin%2C+and+Eduard+Hovy%2E+2006%2E+Re%2Devaluating+machine+translation+results+with+paraphrase+support%2E+In+Proceedings+of+EMNLP%2E)