---
title: "PLATO: Pre-trained Dialogue Generation Model with — Bao 2020"
description: "PLATO addresses why general-purpose pre-trained models like BERT transfer poorly to open-domain dialogue generation."
authors: ["Siqi Bao", "Huang He", "Fan Wang", "Hua Wu", "Haifeng Wang"]
year: 2020
venue: "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/2020.acl-main.9/"
doi: "10.18653/v1/2020.acl-main.9"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/bao-2020-plato-pre-trained-dialogue-generation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/bao-2020-plato-pre-trained-dialogue-generation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/2/) / PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable

# PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable

**Authors:** Siqi Bao, Huang He, [Fan Wang](https://md.research.elamur.ai/free/en/authors/fan-wang/), [Hua Wu](https://md.research.elamur.ai/free/en/authors/hua-wu/), [Haifeng Wang](https://md.research.elamur.ai/free/en/authors/haifeng-wang/) · **Year:** 2020 · **Venue:** Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.acl-main.9/) · [PDF](https://aclanthology.org/2020.acl-main.9.pdf) · [DOI 10.18653/v1/2020.acl-main.9](https://doi.org/10.18653/v1/2020.acl-main.9) · [OpenAlex](https://openalex.org/W3035451444)

**Citations:** 220 citations (OpenAlex, 2026-10-14) · [99](https://md.research.elamur.ai/free/en/papers/bao-2020-plato-pre-trained-dialogue-generation/cited-by/) in our library, of the 140884 papers whose bibliographies we read

## Summary
PLATO addresses why general-purpose pre-trained models like BERT transfer poorly to open-domain dialogue generation. The authors identify three causes: conversational language differs from general text, generation runs left to right while BERT encodes in both directions, and a single context can admit many valid replies. Their framework starts from BERT weights and continues pre-training on large Reddit and Twitter conversation data. Following UniLM, it uses self-attention masks so one transformer can encode bi-directionally and decode uni-directionally. A K-way categorical latent variable, interpreted as an unsupervised "latent speech act," captures reply diversity. Two complementary tasks share the same parameters: response generation conditioned on context and latent value, and latent act recognition, which infers that value from a context–response pair. Training combines NLL, bag-of-words, and response selection losses. The model was evaluated on chit-chat, knowledge-grounded dialogue, and conversational QA, including Persona-Chat and DSTC7-AVSD, and beat prior state-of-the-art systems. The code and models are public, offering a reusable foundation for diverse, coherent dialogue systems.

## Key points
- Each input token's representation sums four embeddings: token, role, turn, and position. Turn indices are relative, so the response is always turn 0. Special [EOU] and [BOU] tokens mark utterance boundaries.
- The latent act recognition and response selection objectives read the final hidden state of a special [M] mask token. Response selection scores also rank multiple candidate replies at inference time.
- The paper reports a new best result on DSTC7-AVSD. A reported upper bound, obtained by picking the best of K generated candidates, suggests further headroom.
- Persona-Chat ablations used 12-layer models trained on 1k, 5k, and 9k dialogues and compared them by perplexity. Flexible attention outperformed directly fine-tuning BERT or GPT-2. Further pre-training on Twitter/Reddit helped, and the latent variable improved results again.
- Code and pre-trained models are released at github.com/PaddlePaddle/Research (NLP/Dialogue-PLATO).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/)

## Institutions

- [Baidu](https://md.research.elamur.ai/free/en/companies/baidu/) — Company, Beijing, China · 26 works · 11 874 citations

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{bao2020plato,
  title = {{PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable}},
  author = {Bao, Siqi and He, Huang and Wang, Fan and Wu, Hua and Wang, Haifeng},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.9},
  url = {https://doi.org/10.18653/v1/2020.acl-main.9}
}
```

- [← previous Probing Pretrained Language Models for Lexical Semantics](https://md.research.elamur.ai/free/en/papers/vulic-2020-probing-pretrained-language-models-lexical/)
- [next → Explainable Automated Fact-Checking for Public Health Claims](https://md.research.elamur.ai/free/en/papers/kotonya-2020-explainable-automated-fact-checking-public/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/2/)

## References

5 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. 2015. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325. [find](https://scholar.google.com/scholar?q=Xinlei+Chen%2C+Hao+Fang%2C+Tsung%2DYi+Lin%2C+Ramakrishna+Vedantam%2C+Saurabh+Gupta%2C+Piotr+Dolla%CC%81r%2C+and+C+Lawrence+Zitnick%2E+2015%2E+Microsoft+coco+captions%3A+Data+collection+and+evaluation+server%2E+arXiv+preprint+ar) [our card](https://md.research.elamur.ai/free/en/papers/chen-2015-microsoft-coco-captions-data-collection/)
2. Huda Alamri, Vincent Cartillier, Abhishek Das, Jue Wang, Anoop Cherian, Irfan Essa, Dhruv Batra, Tim K Marks, Chiori Hori, Peter Anderson, et al. 2019. Audio visual scene-aware dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7558–7567. [find](https://scholar.google.com/scholar?q=Huda+Alamri%2C+Vincent+Cartillier%2C+Abhishek+Das%2C+Jue+Wang%2C+Anoop+Cherian%2C+Irfan+Essa%2C+Dhruv+Batra%2C+Tim+K+Marks%2C+Chiori+Hori%2C+Peter+Anderson%2C+et+al%2E+2019%2E+Audio+visual+scene%2Daware+dialog%2E+In+Proceedings+)
3. Siqi Bao, Huang He, Fan Wang, Rongzhong Lian, and Hua Wu. 2019. Know more about each other: Evolving dialogue strategy via compound assessment. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5382– 5391. [find](https://scholar.google.com/scholar?q=Siqi+Bao%2C+Huang+He%2C+Fan+Wang%2C+Rongzhong+Lian%2C+and+Hua+Wu%2E+2019%2E+Know+more+about+each+other%3A+Evolving+dialogue+strategy+via+compound+assessment%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Associa)
4. Boxing Chen and Colin Cherry. 2014. A systematic comparison of smoothing techniques for sentence-level bleu. In Proceedings of the 9th Workshop on Statistical Machine Translation, pages 362–367. [find](https://scholar.google.com/scholar?q=Boxing+Chen+and+Colin+Cherry%2E+2014%2E+A+systematic+comparison+of+smoothing+techniques+for+sentence%2Dlevel+bleu%2E+In+Proceedings+of+the+9th+Workshop+on+Statistical+Machine+Translation%2C+pages+362%E2%80%93367%2E) [our card](https://md.research.elamur.ai/free/en/papers/chen-2014-systematic-comparison-smoothing-techniques-sentence/)
5. Chaotao Chen, Jinhua Peng, Fan Wang, Jun Xu, and Hua Wu. 2019. Generating multiple diverse responses with multi-mapping and posterior mapping selection. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 4918–4924. \#\# Conclusion [find](https://scholar.google.com/scholar?q=Chaotao+Chen%2C+Jinhua+Peng%2C+Fan+Wang%2C+Jun+Xu%2C+and+Hua+Wu%2E+2019%2E+Generating+multiple+diverse+responses+with+multi%2Dmapping+and+posterior+mapping+selection%2E+In+Proceedings+of+the+28th+International+Joint+)