---
title: "ParaFusion — paper by Jayawardena and Yapa (2024)"
description: "Jayawardena and Yapa build ParaFusion from roughly 750,000 source sentences pooled from MRPC… 2 references, cited by 2 papers here."
authors: ["Lasal Jayawardena", "Prasan Yapa"]
year: 2024
venue: "arXiv:2404.12010"
source: "https://arxiv.org/abs/2404.12010"
doi: "10.5121/csit.2024.140418"
arxiv: "2404.12010"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/jayawardena-2024-parafusion-large-scale-llm-driven/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/jayawardena-2024-parafusion-large-scale-llm-driven/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/13/) / ParaFusion: A Large-Scale LLM-Driven English Paraphrase Dataset Infused with High-Quality Lexical and Syntactic Diversity

# ParaFusion: A Large-Scale LLM-Driven English Paraphrase Dataset Infused with High-Quality Lexical and Syntactic Diversity

**Authors:** [Lasal Jayawardena](https://md.research.elamur.ai/free/en/authors/lasal-jayawardena/) and Prasan Yapa · **Year:** 2024 · **Venue:** arXiv:2404.12010 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://doi.org/10.5121/csit.2024.140418) · [PDF](https://arxiv.org/pdf/2404.12010) · [DOI 10.5121/csit.2024.140418](https://doi.org/10.5121/csit.2024.140418) · [arXiv 2404.12010](https://arxiv.org/abs/2404.12010) · [OpenAlex](https://openalex.org/W4392345938)

**Citations:** 3 citations (OpenAlex, 2026-10-14) · 2 in our library, of the 149,439 papers whose bibliographies we read

## Summary
Paraphrase corpora commonly used to train generation models tend to yield outputs that barely differ from their sources in wording and structure. They also carry noise, offensive material and stray non-English text. Jayawardena and Yapa build ParaFusion from roughly 750,000 source sentences pooled from MRPC, the paraphrase-labelled part of Quora, PAWSWiki and a selection from ParaNMT, ParaBank1 and ParaBank2. They remove about 50,000 sentences flagged by OpenAI's Moderation Endpoint, then ask gpt-3.5-turbo for five varied rewrites of each remaining sentence, with small prompt adjustments per source. Quality is judged on four axes: meaning preservation, lexical variety, syntactic variety and grammaticality. The judging combines automatic metrics, human Likert ratings and a gpt-4 judge that receives the annotators' instructions. The authors report diversity gains of at least 25% for every source corpus, while semantic similarity holds steady or rises slightly. The work suggests that LLM augmentation can stand in for costly human-written paraphrase pairs, and it argues for evaluation that goes beyond lexical-overlap scores.

## Key points
- Sources: MRPC, Quora (only pairs labelled as paraphrases) and PAWSWiki. PAWSQQP was skipped because its sources duplicate Quora. For cost reasons, ParaNMT/ParaBank1/ParaBank2 contributed only their shared source sentences plus an extra 250,000.
- Pipeline: about 750k source sentences, about 50k removed by the moderation filter (hate, harassment, self-harm, sexual, violence categories), then 5 paraphrases per source from gpt-3.5-turbo.
- Human evaluation on a 1–5 scale, original vs ParaFusion:
  - lexical diversity 2.29 → 3.09
  - syntactic diversity 2.44 → 3.40
  - grammar 4.37 → 4.79
  - semantic similarity 4.36 → 4.46
- gpt-4 evaluation on the same items with the same rubric:
  - lexical diversity 1.75 → 3.34
  - syntactic diversity 2.02 → 3.84
  - semantic similarity 4.49 → 4.94
  - grammar 4.75 → 4.99
- Stated limitations: English only, possible residual noise or offensive output, and errors and quality drift inherited from the generating LLM.

## Interesting ideas
- A paraphrase counts as good only if it scores well on four separate axes (same meaning, new words, new structure, correct grammar), which prevents a single overlap number from hiding failures.
- The same written rubric was given to human annotators and to gpt-4, and the two sets of ratings were then checked for agreement, which validates the LLM judge against people before trusting it.
- Inputs are screened with a moderation classifier before any expensive generation runs, which saves money and keeps unwanted content out of the derived data.
- When mixing corpora, the authors dropped a subset whose source sentences duplicated another corpus, and they sampled the sentences shared by overlapping corpora, which avoids redundancy while keeping a variety of lengths and styles.
- Generating several meaning-preserving rewrites per sentence is a cheap way to produce many surface variants of the same content for augmentation.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (back): score the text → graph → text compressor's outputs on these four axes separately, measuring structural change by comparing the UD trees of input and output.
- **I2** (learn): calibrate each LLM teacher tier on a small human-rated subset with an identical rubric before letting it label silver data at scale.
- **I3** (learn): run a cheap pre-filter (language ID, toxicity, malformed text) over raw sentences before sending them to witness parsers or LLM teachers.
- **I4** (tree): deduplicate source sentences across treebanks before merging them for the perceptron parser or gparse, so repeated sentences neither inflate training nor leak into evaluation.
- **I5** (answer): paraphrase school-test and fable questions several ways and require the reader to give the same answer each time, as a robustness check on the frame-based pipeline.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Informatics Institute of Technology](https://md.research.elamur.ai/free/en/universities/informatics-institute-of-technology/) — University, Colombo, Sri Lanka · 3 works · 160 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,379 works

## Cite

Cite as: Lasal Jayawardena and Prasan Yapa. 2024. ParaFusion: A Large-Scale LLM-Driven English Paraphrase Dataset Infused with High-Quality Lexical and Syntactic Diversity. arXiv:2404.12010.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{jayawardena2024parafusion,
  title = {{ParaFusion: A Large-Scale LLM-Driven English Paraphrase Dataset Infused with High-Quality Lexical and Syntactic Diversity}},
  author = {Lasal Jayawardena and Prasan Yapa},
  year = {2024},
  doi = {10.5121/csit.2024.140418},
  eprint = {2404.12010},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.5121/csit.2024.140418}
}
```

- [← previous Old English emotion is temperature: Cultural influences on a universal experience](https://md.research.elamur.ai/free/en/papers/diaz-vera-2024-old-english-emotion-temperature-cultural/)
- [next → ReBERT at HSD-2Lang 2024: Fine-Tuning BERT with AdamW for Hate Speech Detection in Arabic and Turkish](https://md.research.elamur.ai/free/en/papers/yagci-2024-rebert-hsd-2lang-2024-fine/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/13/)

## References

2 references, 2 of them with a card of ours. ↗ leads to the work.

1. 1. K. R. McKeown, “Paraphrasing using given and new information in a question-answer system,” in Proceedings of the 17th annual meeting on Association for Computational Linguistics -. La Jolla, California: Association for Computational Linguistics, 1979, p. 67. [Online]. Available: http://portal.acm.org/citation.cfm?doid=982163.982182 2. M. Meteer and V. Shaked, “Strategies for effective paraphrasing,” in Proceedings of the 12th conference on Computational linguistics -, vol. 2. Budapest, Hungry: Association for Computational Linguistics, 1988, pp. 431–436. [Online]. Available: http://portal.acm.org/citation.cfm?doid=991719.991724 3. R. Kozlowski, K. F. McCoy, and K. Vijay-Shanker, “Generation of single-sentence paraphrases from predicate/argument structure using lexico-grammatical resources,” in Proceedings of the second international workshop on Paraphrasing -, vol. 16. Sapporo, Japan: Association for Computational Linguistics, 2003, pp. 1–8. [Online]. Available: http://portal.acm.org/citation.cfm?doid=1118984.1118985 4. I. A. Bolshakov and A. Gelbukh, “Synonymous Paraphrasing Using WordNet and Internet,” in Natural Language Processing and Information Systems, D. Hutchison, T. Kanade, J. Kittler, J. M. Kleinberg, F. Mattern, J. C. Mitchell, M. Naor, O. Nierstrasz, C. Pandu Rangan, B. Steffen, M. Sudan, D. Terzopoulos, D. Tygar, M. Y. Vardi, G. Weikum, F. Meziane, and E. Métais, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2004, vol. 3136, pp. 312–323, series Title: Lecture Notes in Computer Science. [Online]. Available: http://link.springer.com/10.1007/978-3-540-27779-8 27 5. J. Zhou and S. Bhat, “Paraphrase Generation: A Survey of the State of the Art,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih, Eds. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics, Nov. 2021, pp. 5075–5086. [Online]. Available: https://aclanthology.org/2021.emnlp-main.414 6. OpenAI, “[GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/),” 2023, publisher: arXiv Version Number: 3. [Online]. Available: https://arxiv.org/abs/2303.08774 7. J. Ganitkevitch, B. Van Durme, and C. Callison-Burch, “PPDB: The Paraphrase Database,” in Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Atlanta, Georgia: Association for Computational Linguistics, Jun. 2013, pp. 758–764. [Online]. Available: https://aclanthology.org/N13-1092 8. W. Lan, S. Qiu, H. He, and W. Xu, “A Continuously Growing Dataset of Sentential Paraphrases,” in Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Copenhagen, Denmark: Association for Computational Linguistics, 2017, pp. 1224–1234. [Online]. Available: http://aclweb.org/anthology/D17-1126 9. A. Fader, L. Zettlemoyer, and O. Etzioni, “Paraphrase-Driven Learning for Open Question Answering,” in Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Sofia, Bulgaria: Association for Computational Linguistics, Aug. 2013, pp. 1608–1618. [Online]. Available: https://aclanthology.org/P13-1158 10. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft COCO: Common Objects in Context,” in Computer Vision – ECCV 2014, D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, Eds. Cham: Springer International Publishing, 2014, vol. 8693, pp. 740–755, series Title: Lecture Notes in Computer Science. [Online]. Available: http://link.springer.com/10.1007/978-3-319-10602-1 48 11. W. B. Dolan and C. Brockett, “Automatically Constructing a Corpus of Sentential Paraphrases,” in Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005. [Online]. Available: https://aclanthology.org/I05-5002 12. S. Iyer, N. Dandeka, and K. Csernai, “First Quora Dataset Release: Question Pairs,” 2017. [Online]. Available: https://quoradata.quora.com/First-Quora-Dataset-Release-Question-Pairs 13. J. Wieting and K. Gimpel, “ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations,” in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Melbourne, Australia: Association for Computational Linguistics, 2018, pp. 451–462. [Online]. Available: http://aclweb.org/anthology/P18-1042 14. J. E. Hu, R. Rudinger, M. Post, and B. Van Durme, “ParaBank: Monolingual Bitext Generation and Sentential Paraphrasing via Lexically-constrained Neural Machine Translation,” Jan. 2019, arXiv:1901.03644 [cs]. [Online]. Available: http://arxiv.org/abs/ 1901.03644 15. J. E. Hu, A. Singh, N. Holzenberger, M. Post, and B. Van Durme, “Large-Scale, Diverse, Paraphrastic Bitexts via Sampling and Clustering,” in Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL). Hong Kong, China: Association for Computational Linguistics, 2019, pp. 44–54. [Online]. Available: https://www.aclweb.org/anthology/K19-1005 16. Y. Zhang, J. Baldridge, and L. He, “PAWS: Paraphrase Adversaries from Word Scrambling,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019, pp. 1298–1308. [Online]. Available: https://aclanthology.org/N19-1131 17. OpenAI, “Moderation,” 2023. [Online]. Available: https://platform.openai.com/docs/guides/ moderation 18. OpenAI, “New and Improved Embedding Model,” 2023, publisher: OpenAI. [Online]. Available: https://openai.com/blog/new-and-improved-embedding-model 19. T. Gao, X. Yao, and D. Chen, “SimCSE: Simple Contrastive Learning of Sentence Embeddings,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics, Nov. 2021, pp. 6894–6910. [Online]. Available: https://aclanthology.org/2021.emnlp-main.552 20. Y. Jiang, L. Zhang, and W. Wang, “Improved Universal Sentence Embeddings with Prompt-based Contrastive Learning and Energy-based Learning,” in Findings of the Association for Computational Linguistics: EMNLP 2022. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics, Dec. 2022, pp. 3021–3035. [Online]. Available: https://aclanthology.org/2022.findings-emnlp.220 21. N. Reimers and I. Gurevych, “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Nov. 2019. [Online]. Available: http://arxiv.org/abs/1908.10084 22. P. Qi, Y. Zhang, Y. Zhang, J. Bolton, and C. D. Manning, “Stanza: A Python Natural Language Processing Toolkit for Many Human Languages,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 2020. 23. S. Bird, E. Loper, and E. Klein, Natural Language Processing with Python. [↗](https://doi.org/10.1007/978-3-540-27779-8)
2. O’Reilly Media Inc., 2009. 24. M. Pawlik and N. Augsten, “Efficient Computation of the Tree Edit Distance,” ACM Transactions on Database Systems, vol. 40, no. 1, pp. 1–40, Mar. 2015. [Online]. Available: https://dl.acm.org/doi/10.1145/2699485 25. F. M. Zanzotto, A. Santilli, L. Ranaldi, D. Onorati, P. Tommasino, and F. Fallucchi, “KERMIT: Complementing Transformer Architectures with Encoders of Explicit Syntactic Interpretations,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Online: Association for Computational Linguistics, Nov. 2020, pp. 256–267. [Online]. Available: https://www.aclweb.org/anthology/2020.emnlp-main. 18 26. M. Post, “A Call for Clarity in Reporting BLEU Scores,” in Proceedings of the Third Conference on Machine Translation: Research Papers. Belgium, Brussels: Association for Computational Linguistics, Oct. 2018, pp. 186–191. [Online]. Available: https://www.aclweb.org/anthology/W18-6319 27. Google, “Google Research,” 2023. [Online]. Available: https://github.com/google-research/ google-research/tree/master 28. C. Van Der Lee, A. Gatt, E. Van Miltenburg, S. Wubben, and E. Krahmer, “Best practices for the human evaluation of automatically generated text,” in Proceedings of the 12th International Conference on Natural Language Generation. Tokyo, Japan: Association for Computational Linguistics, 2019, pp. 355–368. [Online]. Available: https://www.aclweb.org/anthology/W19-8643 29. Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu, “[G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://md.research.elamur.ai/free/en/papers/liu-2023-g-eval-nlg-evaluation-gpt/),” 2023, publisher: arXiv Version Number: 3. [Online]. Available: https://arxiv.org/abs/2303.16634 [↗](https://doi.org/10.1145/2699485)

*Updated 29 October 2026*