---
title: "Texygen: A Benchmarking Platform for Text — Zhu 2018"
description: "Comparing models for open-domain text generation, especially GAN-based ones, is unreliable for three reasons: no single metric captures model quality…"
authors: ["Yaoming Zhu", "Sidi Lu", "Lei Zheng", "Jiaxian Guo", "Weinan Zhang", "Jun Wang", "Yong Yu"]
year: 2018
venue: "arXiv:1802.01886"
source: "https://arxiv.org/abs/1802.01886"
doi: "10.48550/arxiv.1802.01886"
arxiv: "1802.01886"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhu-2018-texygen-benchmarking-platform-text-generation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhu-2018-texygen-benchmarking-platform-text-generation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/4/) / Texygen: A Benchmarking Platform for Text Generation Models

# Texygen: A Benchmarking Platform for Text Generation Models

**Authors:** Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, [Weinan Zhang](https://md.research.elamur.ai/free/en/authors/weinan-zhang/), Jun Wang, [Yong Yu](https://md.research.elamur.ai/free/en/authors/yong-yu/) · **Year:** 2018 · **Venue:** arXiv:1802.01886 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1802.01886) · [PDF](https://arxiv.org/pdf/1802.01886) · [DOI 10.48550/arxiv.1802.01886](https://doi.org/10.48550/arxiv.1802.01886) · [arXiv 1802.01886](https://arxiv.org/abs/1802.01886) · [OpenAlex](https://openalex.org/W2785896739)

**Citations:** 148 citations (openalex-snapshot · 2026-10-14) · [338](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Comparing models for open-domain text generation, especially GAN-based ones, is unreliable for three reasons: no single metric captures model quality, many authors don't release code, and gains in output quality often just come from giving up diversity. Texygen addresses this as an open-source benchmarking platform built on TensorFlow. It re-implements a set of baseline generators and puts several automatic metrics behind one shared API. The metrics cover three things: how closely outputs resemble real text, how well they fit the data in likelihood terms, and how diverse they are. The authors run the baselines in two settings. The first is synthetic: a randomly initialised LSTM "oracle" produces the training data. The second is real: captions from COCO. The runs show trade-offs that a single score would hide. LeakGAN converges fastest and keeps its embedding similarity after adversarial training starts, yet its outputs are also the least diverse. The plain MLE model gives the most varied text. A shared, reproducible setup like this makes later work on text generation easier to compare.

## Key points
- **Baselines:** vanilla MLE language model, SeqGAN, MaliGAN, RankGAN, TextGAN, GSGAN (Gumbel-Softmax) and LeakGAN. The code is at github.com/geek-ai/Texygen. Oracles can be LSTM-, GRU- or SRU-based.
- **Metrics:** BLEU, NLL-oracle, and three that the authors propose here: EmbSim (compares word-embedding similarity matrices built with skip-gram), NLL-test (fit to real test data) and Self-BLEU (how much generated sentences overlap with each other, used as a diversity proxy where higher means more mode collapse).
- **Setup:** The synthetic data has a 5,000-word vocabulary and 10,000 oracle sentences of 20 tokens each. The real data is 20,000 COCO captions split evenly into training and test sets. Training uses 80 epochs of generator pretraining, then 80 of discriminator pretraining, then 100 adversarial epochs.
- **Self-BLEU-5:** LeakGAN 0.780 is the highest (least diverse); MLE 0.408 is the lowest. On BLEU-5 against training data, TextGAN (0.523) beats LeakGAN (0.470).
- **Excluded or unhelpful:** GSGAN produced no meaningful sentences on the real data. A CFG-based grammar metric could not tell the models apart and tended to favour mode-collapsed models.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

338 papers of our library cite it (of the 140673 whose bibliographies we read); 19 of them have a page here, 319 not yet.

- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://md.research.elamur.ai/free/en/papers/agarwal-2023-policy-distillation-language-models-learning/)
- [Generation and Evaluation of English Grammar Multiple-Choice Cloze Exercises](https://md.research.elamur.ai/free/en/papers/donati-2024-generation-evaluation-english-grammar-multiple/)
- [DiffuSeq-v2: Bridging Discrete and Continuous Text Spaces for Accelerated Seq2Seq Diffusion Models](https://md.research.elamur.ai/free/en/papers/gong-2023-diffuseq-v2-bridging-discrete-continuous/)
- [Impossible Distillation: from Low-Quality Model to High-Quality Dataset & Model for Summarization and Paraphrasing](https://md.research.elamur.ai/free/en/papers/jung-2023-impossible-distillation-low-quality-model/)
- [Understanding the Effects of RLHF on LLM Generalisation and Diversity](https://md.research.elamur.ai/free/en/papers/kirk-2023-understanding-effects-rlhf-llm-generalisation/)
- [Pretraining Language Models with Human Preferences](https://md.research.elamur.ai/free/en/papers/korbak-2023-pretraining-language-models-human-preferences/)
- [Hurdles to Progress in Long-form Question Answering](https://md.research.elamur.ai/free/en/papers/krishna-2021-hurdles-progress-long-form-question/)
- [On Generative Spoken Language Modeling from Raw Audio](https://md.research.elamur.ai/free/en/papers/lakhotia-2021-generative-spoken-language-modeling-raw/)
- [GUM-SAGE: A Novel Dataset and Approach for Graded Entity Salience Prediction](https://md.research.elamur.ai/free/en/papers/lin-2025-gum-sage-novel-dataset-approach/)
- [Okay, Let's Do This! Modeling Event Coreference with Generated Rationales and Knowledge Distillation](https://md.research.elamur.ai/free/en/papers/nath-2024-okay-let-s-do-this/)
- [Red Teaming Language Models with Language Models](https://md.research.elamur.ai/free/en/papers/perez-2022-red-teaming-language-models-language/)
- [MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers](https://md.research.elamur.ai/free/en/papers/pillutla-2021-mauve-measuring-gap-between-neural/)
- [Children's English Reading Story Generation via Supervised Fine-Tuning of Compact LLMs with Controllable Difficulty and Safety](https://md.research.elamur.ai/free/en/papers/shen-2026-children-s-english-reading-story/)
- [BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model](https://md.research.elamur.ai/free/en/papers/wang-2019-bert-has-mouth-must-speak/)
- [On Temperature-Constrained Non-Deterministic Machine Translation: Potential and Evaluation](https://md.research.elamur.ai/free/en/papers/wang-2026-temperature-constrained-non-deterministic-machine/)
- [Symbolic Knowledge Distillation: from General Language Models to Commonsense Models](https://md.research.elamur.ai/free/en/papers/west-2022-symbolic-knowledge-distillation-general-language/)
- [Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models](https://md.research.elamur.ai/free/en/papers/wu-2021-polyjuice-generating-counterfactuals-explaining-evaluating/)
- [Paraphrasing as Zero-shot Translation with Feature-guided Diversity Enhancement](https://md.research.elamur.ai/free/en/papers/yan-2026-paraphrasing-as-zero-shot-translation/)
- [ZeroGen: Efficient Zero-shot Learning via Dataset Generation](https://md.research.elamur.ai/free/en/papers/ye-2022-zerogen-efficient-zero-shot-learning/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zhu2018texygen,
  title = {{Texygen: A Benchmarking Platform for Text Generation Models}},
  author = {Zhu, Yaoming and Lu, Sidi and Zheng, Lei and Guo, Jiaxian and Zhang, Weinan and Wang, Jun and Yu, Yong},
  year = {2018},
  doi = {10.48550/arxiv.1802.01886},
  eprint = {1802.01886},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1802.01886}
}
```

- [← previous TextWorld: A Learning Environment for Text-based Games](https://md.research.elamur.ai/free/en/papers/cote-2018-textworld-learning-environment-text-based/)
- [next → The 2018 Shared Task on Extrinsic Parser Evaluation: On the Downstream Utility of English Universal Dependency Parsers](https://md.research.elamur.ai/free/en/papers/ud-2018-fares-2018-shared-task-extrinsic-parser/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/4/)

## References

17 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Martín Abadi et al. 2016. TensorFlow: A System for Large-Scale Machine Learning.. In OSDI, Vol. 16. 265–283. [find](https://scholar.google.com/scholar?q=Mart%C3%ADn+Abadi+et+al%2E+2016%2E+TensorFlow%3A+A+System+for+Large%2DScale+Machine+Learning%2E%2E+In+OSDI%2C+Vol%2E+16%2E+265%E2%80%93283%2E)
2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014). [find](https://scholar.google.com/scholar?q=Dzmitry+Bahdanau%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2014%2E+Neural+machine+translation+by+jointly+learning+to+align+and+translate%2E+arXiv+preprint+arXiv%3A1409%2E0473+%282014%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
3. Tong Che, Yanran Li, Ruixiang Zhang, R Devon Hjelm, Wenjie Li, Yangqiu Song, and Yoshua Bengio. 2017. Maximum-Likelihood Augmented Discrete Generative Adversarial Networks. arXiv preprint arXiv:1702.07983 (2017). — [arXiv:1702.07983](https://arxiv.org/abs/1702.07983) [find](https://scholar.google.com/scholar?q=Tong+Che%2C+Yanran+Li%2C+Ruixiang+Zhang%2C+R+Devon+Hjelm%2C+Wenjie+Li%2C+Yangqiu+Song%2C+and+Yoshua+Bengio%2E+2017%2E+Maximum%2DLikelihood+Augmented+Discrete+Generative+Adversarial+Networks%2E+arXiv+preprint+arXiv%3A1702%2E0)
4. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in neural information processing systems. 2672–2680. [find](https://scholar.google.com/scholar?q=Ian+Goodfellow%2C+Jean+Pouget%2DAbadie%2C+Mehdi+Mirza%2C+Bing+Xu%2C+David+Warde%2DFarley%2C+Sherjil+Ozair%2C+Aaron+Courville%2C+and+Yoshua+Bengio%2E+2014%2E+Generative+adversarial+nets%2E+In+Advances+in+neural+information+pr)
5. Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Jun Wang, and Yong Yu. 2017. Long Text Generation via Adversarial Training with Leaked Information. arXiv preprint arXiv:1709.08624. — [arXiv:1709.08624](https://arxiv.org/abs/1709.08624) [find](https://scholar.google.com/scholar?q=Jiaxian+Guo%2C+Sidi+Lu%2C+Han+Cai%2C+Weinan+Zhang%2C+Jun+Wang%2C+and+Yong+Yu%2E+2017%2E+Long+Text+Generation+via+Adversarial+Training+with+Leaked+Information%2E+arXiv+preprint+arXiv%3A1709%2E08624%2E+%E2%80%94+%5BarXiv%3A1709%2E08624%5D%28h)
6. Ferenc Huszár. 2015. How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary? arXiv preprint arXiv:1511.05101 (2015). — [arXiv:1511.05101](https://arxiv.org/abs/1511.05101) [find](https://scholar.google.com/scholar?q=Ferenc+Husz%C3%A1r%2E+2015%2E+How+%28not%29+to+Train+your+Generative+Model%3A+Scheduled+Sampling%2C+Likelihood%2C+Adversary%3F+arXiv+preprint+arXiv%3A1511%2E05101+%282015%29%2E+%E2%80%94+%5BarXiv%3A1511%2E05101%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1511%2E05101%29)
7. Fred Jelinek, Robert L Mercer, Lalit R Bahl, and James K Baker. 1977. Perplexity - a measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America 62, S1 (1977), S63–S63. [find](https://scholar.google.com/scholar?q=Fred+Jelinek%2C+Robert+L+Mercer%2C+Lalit+R+Bahl%2C+and+James+K+Baker%2E+1977%2E+Perplexity+%2D+a+measure+of+the+difficulty+of+speech+recognition+tasks%2E+The+Journal+of+the+Acoustical+Society+of+America+62%2C+S1+%28197)
8. Matt J Kusner and José Miguel Hernández-Lobato. 2016. Gans for sequences of discrete elements with the gumbel-softmax distribution. arXiv preprint arXiv:1611.04051 (2016). — [arXiv:1611.04051](https://arxiv.org/abs/1611.04051) [find](https://scholar.google.com/scholar?q=Matt+J+Kusner+and+Jos%C3%A9+Miguel+Hern%C3%A1ndez%2DLobato%2E+2016%2E+Gans+for+sequences+of+discrete+elements+with+the+gumbel%2Dsoftmax+distribution%2E+arXiv+preprint+arXiv%3A1611%2E04051+%282016%29%2E+%E2%80%94+%5BarXiv%3A1611%2E04051%5D%28https%3A%2F)
9. Jiwei Li, Will Monroe, Tianlin Shi, Alan Ritter, and Dan Jurafsky. 2017. Adversarial learning for neural dialogue generation. arXiv preprint arXiv:1701.06547 (2017). [find](https://scholar.google.com/scholar?q=Jiwei+Li%2C+Will+Monroe%2C+Tianlin+Shi%2C+Alan+Ritter%2C+and+Dan+Jurafsky%2E+2017%2E+Adversarial+learning+for+neural+dialogue+generation%2E+arXiv+preprint+arXiv%3A1701%2E06547+%282017%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2017-adversarial-learning-neural-dialogue-generation/)
10. Kevin Lin, Dianqi Li, Xiaodong He, Zhengyou Zhang, and Ming-Ting Sun. 2017. Adversarial Ranking for Language Generation. arXiv preprint arXiv:1705.11001 (2017). — [arXiv:1705.11001](https://arxiv.org/abs/1705.11001) [find](https://scholar.google.com/scholar?q=Kevin+Lin%2C+Dianqi+Li%2C+Xiaodong+He%2C+Zhengyou+Zhang%2C+and+Ming%2DTing+Sun%2E+2017%2E+Adversarial+Ranking+for+Language+Generation%2E+arXiv+preprint+arXiv%3A1705%2E11001+%282017%29%2E+%E2%80%94+%5BarXiv%3A1705%2E11001%5D%28https%3A%2F%2Farxiv%2Eorg%2F)
11. Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černockỳ, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Eleventh Annual Conference of the International Speech Communication Association. [find](https://scholar.google.com/scholar?q=Tom%C3%A1%C5%A1+Mikolov%2C+Martin+Karafi%C3%A1t%2C+Luk%C3%A1%C5%A1+Burget%2C+Jan+%C4%8Cernocky%CC%80%2C+and+Sanjeev+Khudanpur%2E+2010%2E+Recurrent+neural+network+based+language+model%2E+In+Eleventh+Annual+Conference+of+the+International+Speech+Commu)
12. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In ACL. 311–318. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+BLEU%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+ACL%2E+311%E2%80%93318%2E) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
13. Jun Wang, Lantao Yu, Weinan Zhang, Yu Gong, Yinghui Xu, Benyou Wang, Peng Zhang, and Dell Zhang. 2017. Irgan: A minimax game for unifying generative and discriminative information retrieval models. In SIGIR. ACM, 515–524. [find](https://scholar.google.com/scholar?q=Jun+Wang%2C+Lantao+Yu%2C+Weinan+Zhang%2C+Yu+Gong%2C+Yinghui+Xu%2C+Benyou+Wang%2C+Peng+Zhang%2C+and+Dell+Zhang%2E+2017%2E+Irgan%3A+A+minimax+game+for+unifying+generative+and+discriminative+information+retrieval+models%2E+In)
14. Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning 8, 3-4 (1992), 229–256. [find](https://scholar.google.com/scholar?q=Ronald+J+Williams%2E+1992%2E+Simple+statistical+gradient%2Dfollowing+algorithms+for+connectionist+reinforcement+learning%2E+Machine+learning+8%2C+3%2D4+%281992%29%2C+229%E2%80%93256%2E)
15. Kelvin Xu, Jimmy Lei Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. 2015. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. arXiv preprint arXiv:1502.03044. [find](https://scholar.google.com/scholar?q=Kelvin+Xu%2C+Jimmy+Lei+Ba%2C+Ryan+Kiros%2C+Kyunghyun+Cho%2C+Aaron+Courville%2C+Ruslan+Salakhutdinov%2C+Richard+S%2E+Zemel%2C+and+Yoshua+Bengio%2E+2015%2E+Show%2C+Attend+and+Tell%3A+Neural+Image+Caption+Generation+with+Visual) [our card](https://md.research.elamur.ai/free/en/papers/xu-2015-show-attend-tell-neural-image/)
16. Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. 2017. SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient.. In AAAI. 2852–2858. [find](https://scholar.google.com/scholar?q=Lantao+Yu%2C+Weinan+Zhang%2C+Jun+Wang%2C+and+Yong+Yu%2E+2017%2E+SeqGAN%3A+Sequence+Generative+Adversarial+Nets+with+Policy+Gradient%2E%2E+In+AAAI%2E+2852%E2%80%932858%2E)
17. Yizhe Zhang, Zhe Gan, Kai Fan, Zhi Chen, Ricardo Henao, Dinghan Shen, and Lawrence Carin. 2017. Adversarial Feature Matching for Text Generation. arXiv preprint arXiv:1706.03850 (2017). *Table 2: BLEU score on test data SeqGAN MaliGAN RankGAN BLEU-2 0.745 0.673 0.743 BLEU-3 0.498 0.432 0.467 BLEU-4… — [arXiv:1706.03850](https://arxiv.org/abs/1706.03850) [find](https://scholar.google.com/scholar?q=Yizhe+Zhang%2C+Zhe+Gan%2C+Kai+Fan%2C+Zhi+Chen%2C+Ricardo+Henao%2C+Dinghan+Shen%2C+and+Lawrence+Carin%2E+2017%2E+Adversarial+Feature+Matching+for+Text+Generation%2E+arXiv+preprint+arXiv%3A1706%2E03850+%282017%29%2E+%2ATable+2%3A+BLEU)