---
title: "ChatGPT vs Human-authored Text — Liu 2023"
description: "Liu and Demberg examine how closely ChatGPT's output matches human writing when the model is asked to control style, and how often it distorts content…"
authors: ["Dongqi Liu", "Vera Demberg"]
year: 2023
venue: "arXiv:2306.07799"
source: "https://arxiv.org/abs/2306.07799"
doi: "10.48550/arxiv.2306.07799"
arxiv: "2306.07799"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/liu-2023-chatgpt-vs-human-authored-text/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/liu-2023-chatgpt-vs-human-authored-text/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2023](https://md.research.elamur.ai/free/papers/2023/page/2/) / ChatGPT vs Human-authored Text: Insights into Controllable Text Summarization and Sentence Style Transfer

# ChatGPT vs Human-authored Text: Insights into Controllable Text Summarization and Sentence Style Transfer

**Authors:** Dongqi Liu, [Vera Demberg](https://md.research.elamur.ai/free/authors/vera-demberg/) · **Year:** 2023 · **Venue:** arXiv:2306.07799 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2306.07799) · [PDF](https://arxiv.org/pdf/2306.07799) · [DOI 10.48550/arxiv.2306.07799](https://doi.org/10.48550/arxiv.2306.07799) · [arXiv 2306.07799](https://arxiv.org/abs/2306.07799) · [OpenAlex](https://openalex.org/W4380715271)

**Citations:** 5 citations (openalex-snapshot · 2026-10-14) · [37](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Liu and Demberg examine how closely ChatGPT's output matches human writing when the model is asked to control style, and how often it distorts content while doing so. They test gpt-3.5-turbo zero-shot on two tasks. The first is summarizing scientific articles for experts versus lay readers, using the ELIFE dataset. The second is converting sentences between formal and informal register, using GYAFC. They compare the model's outputs with human-written references using readability scores (FRE, CLI, DCR), ROUGE, n-gram novelty, and faithfulness checks (SummaC, named-entity hallucination, dependency arc entailment).

ChatGPT beats earlier state-of-the-art systems on automatic metrics, but its stylistic range is much narrower than that of humans. Its lay and expert summaries differ only modestly in readability, and its word-type distributions diverge from human texts. Giving the model an example of the target human style reduces this gap. The model also introduces subtle factual errors and hallucinations, and these accumulate when a text is rewritten repeatedly. The work cautions against treating LLM text as a substitute for human-authored data in audience- or style-sensitive settings.

## Key points
- **Setup:** gpt-3.5-turbo was queried through the OpenAI API between 15 Feb and 30 Apr 2023.
  - Settings: temperature 0, top_p 1, frequency and presence penalties of 0.2, and a 512-token cap for summaries.
  - Conversation history was cleared before every query.
- **Prompt variants:** For summarization, the authors tried several prompt wordings ("simple", "technical", "in-depth", etc.).
  - FRE ranged from 31 to 38 for layman-oriented prompts and from 28 to 37 for expert-oriented ones.
  - This overlap shows only weak audience differentiation.
- **Faithfulness tools:** The named-entity hallucination check used BERT-based semantic similarity instead of exact string matching. Sentence-level factuality was assessed with dependency arc entailment (DAE).
- **Manual case study:** The authors inspected 15 random samples from each GYAFC branch.
  - About 18% of ChatGPT's style transfers showed noticeable meaning changes, often from altered content words.
  - ChatGPT also scored lower than humans on DAE and on named-entity recall.
- **Stated limitations:** API cost restricted the study to two datasets and default hyperparameters. Results also depend on prompt choice and may not carry over to later ChatGPT versions.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Machine translation](https://md.research.elamur.ai/free/shelves/machine-translation/) · [Natural language processing](https://md.research.elamur.ai/free/shelves/nlp/) · [AI & machine learning](https://md.research.elamur.ai/free/shelves/ai/)

Topics: [Named entity recognition](https://md.research.elamur.ai/free/topics/named-entity-recognition/)

## Cited by

Cited by

37 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 34 not yet.

- [A Survey on Evaluation of Large Language Models](https://md.research.elamur.ai/free/papers/chang-2023-survey-evaluation-large-language-models/)
- [Evaluating LLM Style Transfer Through Readability-Based Age Assessments](https://md.research.elamur.ai/free/papers/maro-2026-evaluating-llm-style-transfer-through/)
- [RSA-Control: A Pragmatics-Grounded Lightweight Controllable Text Generation Framework](https://md.research.elamur.ai/free/papers/wang-2024-rsa-control-pragmatics-grounded-lightweight/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{liu2023chatgpt,
  title = {{ChatGPT vs Human-authored Text: Insights into Controllable Text Summarization and Sentence Style Transfer}},
  author = {Liu, Dongqi and Demberg, Vera},
  year = {2023},
  doi = {10.48550/arxiv.2306.07799},
  eprint = {2306.07799},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.07799}
}
```

- [← previous ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://md.research.elamur.ai/free/papers/gilardi-2023-chatgpt-outperforms-crowd-workers-text/)
- [next → ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPT](https://md.research.elamur.ai/free/papers/wei-2023-chatie-zero-shot-information-extraction/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/papers/2023/page/2/)

## References

77 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Reinald Kim Amplayo, Stefanos Angelidis, and Mirella Lapata. 2021. Aspect-controllable opinion summarization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6578–6593, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Reinald+Kim+Amplayo%2C+Stefanos+Angelidis%2C+and+Mirella+Lapata%2E+2021%2E+Aspect%2Dcontrollable+opinion+summarization%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C)
2. Tal August, Katharina Reinecke, and Noah A. Smith. 2022. Generating scientific definitions with controllable complexity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8298–8317, Dublin, Ireland. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tal+August%2C+Katharina+Reinecke%2C+and+Noah+A%2E+Smith%2E+2022%2E+Generating+scientific+definitions+with+controllable+complexity%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computational+)
3. Nikolay Babakov, David Dale, Varvara Logacheva, and Alexander Panchenko. 2022. A large-scale computational study of content preservation measures for text style transfer and paraphrase generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 300–321, Dublin, Ireland. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nikolay+Babakov%2C+David+Dale%2C+Varvara+Logacheva%2C+and+Alexander+Panchenko%2E+2022%2E+A+large%2Dscale+computational+study+of+content+preservation+measures+for+text+style+transfer+and+paraphrase+generation%2E+In+)
4. David Baidoo-Anu and Leticia Owusu Ansah. 2023. Education in the era of generative artificial intelligence (ai): Understanding the potential benefits of chatgpt in promoting teaching and learning. Available at SSRN 4337484. [↗](https://doi.org/10.61969/jai.1337500)
5. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023a. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. ArXiv, abs/2302.04023. [↗](https://arxiv.org/abs/2302.04023) [our card](https://md.research.elamur.ai/free/papers/bang-2023-multitask-multilingual-multimodal-evaluation-chatgpt/)
6. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023b. A multi-task, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023. [↗](https://arxiv.org/abs/2302.04023) [our card](https://md.research.elamur.ai/free/papers/bang-2023-multitask-multilingual-multimodal-evaluation-chatgpt/)
7. Meng Cao, Yue Dong, and Jackie Cheung. 2022. Hallucinated but factual! inspecting the factuality of hallucinations in abstractive summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3340–3354, Dublin, Ireland. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2109.09784)
8. Yixin Cao, Ruihao Shui, Liangming Pan, Min-Yen Kan, Zhiyuan Liu, and Tat-Seng Chua. 2020. Expertise style transfer: A new task towards better communication between experts and laymen. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1061–1071, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yixin+Cao%2C+Ruihao+Shui%2C+Liangming+Pan%2C+Min%2DYen+Kan%2C+Zhiyuan+Liu%2C+and+Tat%2DSeng+Chua%2E+2020%2E+Expertise+style+transfer%3A+A+new+task+towards+better+communication+between+experts+and+laymen%2E+In+Proceedings+o)
9. Fredrik Carlsson, Joey Öhman, Fangyu Liu, Severine Verlinden, Joakim Nivre, and Magnus Sahlgren. 2022. Fine-grained controllable text generation using non-residual prompting. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6837– 6857, Dublin, Ireland. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Fredrik+Carlsson%2C+Joey+%C3%96hman%2C+Fangyu+Liu%2C+Severine+Verlinden%2C+Joakim+Nivre%2C+and+Magnus+Sahlgren%2E+2022%2E+Fine%2Dgrained+controllable+text+generation+using+non%2Dresidual+prompting%2E+In+Proceedings+of+the+60t)
10. Jeanne Sternlicht Chall and Edgar Dale. 1995. Readability revisited: The new Dale-Chall readability formula. Brookline Books. [find](https://scholar.google.com/scholar?q=Jeanne+Sternlicht+Chall+and+Edgar+Dale%2E+1995%2E+Readability+revisited%3A+The+new+Dale%2DChall+readability+formula%2E+Brookline+Books%2E)
11. Hou Pong Chan, Lu Wang, and Irwin King. 2021. Controllable summarization with constrained Markov decision process. Transactions of the Association for Computational Linguistics, 9:1213–1232. [↗](https://doi.org/10.1162/tacl_a_00423)
12. Yu Cheng, Zhe Gan, Yizhe Zhang, Oussama Elachqar, Dianqi Li, and Jingjing Liu. 2020. Contextual text style transfer. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2915– 2924, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yu+Cheng%2C+Zhe+Gan%2C+Yizhe+Zhang%2C+Oussama+Elachqar%2C+Dianqi+Li%2C+and+Jingjing+Liu%2E+2020%2E+Contextual+text+style+transfer%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2020%2C+pages+291)
13. Meri Coleman and Ta Lin Liau. 1975. A computer readability formula designed for machine scoring. Journal of Applied Psychology, 60(2):283. [↗](https://doi.org/10.1037/h0076540)
14. Ning Dai, Jianze Liang, Xipeng Qiu, and Xuanjing Huang. 2019. Style transformer: Unpaired text style transfer without disentangled latent representation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5997– 6007, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ning+Dai%2C+Jianze+Liang%2C+Xipeng+Qiu%2C+and+Xuanjing+Huang%2E+2019%2E+Style+transformer%3A+Unpaired+text+style+transfer+without+disentangled+latent+representation%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+t) [our card](https://md.research.elamur.ai/free/papers/dai-2019-style-transformer-unpaired-text-style/)
15. Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2019a. Plug and play language models: A simple approach to controlled text generation. arXiv preprint arXiv:1912.02164. [↗](https://arxiv.org/abs/1912.02164) [our card](https://md.research.elamur.ai/free/papers/dathathri-2019-plug-play-language-models-simple/)
16. Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2019b. Plug and play language models: A simple approach to controlled text generation. ArXiv, abs/1912.02164. [↗](https://arxiv.org/abs/1912.02164) [our card](https://md.research.elamur.ai/free/papers/dathathri-2019-plug-play-language-models-simple/)
17. Sam Earle, Maria Edwards, Ahmed Khalifa, Philip Bontrager, and Julian Togelius. 2021. Learning controllable content generators. In 2021 IEEE Conference on Games (CoG), pages 1–9. IEEE. [find](https://scholar.google.com/scholar?q=Sam+Earle%2C+Maria+Edwards%2C+Ahmed+Khalifa%2C+Philip+Bontrager%2C+and+Julian+Togelius%2E+2021%2E+Learning+controllable+content+generators%2E+In+2021+IEEE+Conference+on+Games+%28CoG%29%2C+pages+1%E2%80%939%2E+IEEE%2E)
18. Angela Fan, David Grangier, and Michael Auli. 2018. Controllable abstractive summarization. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 45–54, Melbourne, Australia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Angela+Fan%2C+David+Grangier%2C+and+Michael+Auli%2E+2018%2E+Controllable+abstractive+summarization%2E+In+Proceedings+of+the+2nd+Workshop+on+Neural+Machine+Translation+and+Generation%2C+pages+45%E2%80%9354%2C+Melbourne%2C+Aus) [our card](https://md.research.elamur.ai/free/papers/fan-2018-controllable-abstractive-summarization/)
19. T. Fischer, C. Biemann, Informatik und Naturwissenschaften Universität Hamburg Fakultät für Mathematik, and Universität Hamburg Fachbereich Informatik. 2021. Finding Factual Inconsistencies in Abstractive Summaries. Universität Hamburg. [find](https://scholar.google.com/scholar?q=T%2E+Fischer%2C+C%2E+Biemann%2C+Informatik+und+Naturwissenschaften+Universit%C3%A4t+Hamburg+Fakult%C3%A4t+f%C3%BCr+Mathematik%2C+and+Universit%C3%A4t+Hamburg+Fachbereich+Informatik%2E+2021%2E+Finding+Factual+Inconsistencies+in+Abstrac)
20. Sebastian Gehrmann, Zachary Ziegler, and Alexander Rush. 2019. Generating abstractive summaries with finetuned language models. In Proceedings of the 12th International Conference on Natural Language Generation, pages 516–522, Tokyo, Japan. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sebastian+Gehrmann%2C+Zachary+Ziegler%2C+and+Alexander+Rush%2E+2019%2E+Generating+abstractive+summaries+with+finetuned+language+models%2E+In+Proceedings+of+the+12th+International+Conference+on+Natural+Language+)
21. Aidan Gilson, Conrad W Safranek, Thomas Huang, Vimig Socrates, Ling Chi, Richard Andrew Taylor, David Chartash, et al. 2023. How does chatgpt perform on the united states medical licensing examination? the implications of large language models for medical education and knowledge assessment. JMIR Medical Education, 9(1):e45312. [find](https://scholar.google.com/scholar?q=Aidan+Gilson%2C+Conrad+W+Safranek%2C+Thomas+Huang%2C+Vimig+Socrates%2C+Ling+Chi%2C+Richard+Andrew+Taylor%2C+David+Chartash%2C+et+al%2E+2023%2E+How+does+chatgpt+perform+on+the+united+states+medical+licensing+examination)
22. Peng Cui and Le Hu. 2021. Topic-guided abstractive multi-document summarization. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1463–1472, Punta Cana, Dominican Republic. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2110.11207)
23. Tomas Goldsack, Zhihao Zhang, Chenghua Lin, and Carolina Scarton. 2022. Making science simple: Corpora for the lay summarisation of scientific literature. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10589–10604, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2022.emnlp-main.724)
24. Haixing Dai, Zhengliang Liu, Wenxiong Liao, Xiaoke Huang, Zihao Wu, Lin Zhao, Wei Liu, Ninghao Liu, Sheng Li, Dajiang Zhu, et al. 2023. Chataug: Leveraging chatgpt for text data augmentation. arXiv preprint arXiv:2302.13007. [↗](https://arxiv.org/abs/2302.13007) [our card](https://md.research.elamur.ai/free/papers/dai-2023-auggpt-leveraging-chatgpt-text-data/)
25. Tanya Goyal and Greg Durrett. 2020. Evaluating factuality in generation with dependency-level entailment. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3592–3603, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tanya+Goyal+and+Greg+Durrett%2E+2020%2E+Evaluating+factuality+in+generation+with+dependency%2Dlevel+entailment%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2020%2C+pages+3592%E2%80%933603%2C+Onl) [our card](https://md.research.elamur.ai/free/papers/goyal-2020-evaluating-factuality-generation-dependency-level/)
26. Yuxuan Gu, Xiaocheng Feng, Sicheng Ma, Lingyuan Zhang, Heng Gong, and Bing Qin. 2022. A distributional lens for multi-aspect controllable text generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1023–1043, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yuxuan+Gu%2C+Xiaocheng+Feng%2C+Sicheng+Ma%2C+Lingyuan+Zhang%2C+Heng+Gong%2C+and+Bing+Qin%2E+2022%2E+A+distributional+lens+for+multi%2Daspect+controllable+text+generation%2E+In+Proceedings+of+the+2022+Conference+on+Empi)
27. Junxian He, Wojciech Kryscinski, Bryan McCann, Nazneen Rajani, and Caiming Xiong. 2022. CTRLsum: Towards generic controllable text summarization. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5879–5915, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Junxian+He%2C+Wojciech+Kryscinski%2C+Bryan+McCann%2C+Nazneen+Rajani%2C+and+Caiming+Xiong%2E+2022%2E+CTRLsum%3A+Towards+generic+controllable+text+summarization%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+Met)
28. Francis Heylighen and Jean-Marc Dewaele. 1999. Formality of language: definition, measurement and behavioral determinants. Interner Bericht, Center “Leo Apostel”, Vrije Universiteit Brüssel, 4. [find](https://scholar.google.com/scholar?q=Francis+Heylighen+and+Jean%2DMarc+Dewaele%2E+1999%2E+Formality+of+language%3A+definition%2C+measurement+and+behavioral+determinants%2E+Interner+Bericht%2C+Center+%E2%80%9CLeo+Apostel%E2%80%9D%2C+Vrije+Universiteit+Br%C3%BCssel%2C+4%2E)
29. Zhiqiang Hu, Roy Ka-Wei Lee, and Charu C. Aggarwal. 2021. Syntax matters! syntax-controlled in text style transfer. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 566–575, Held Online. INCOMA Ltd. [find](https://scholar.google.com/scholar?q=Zhiqiang+Hu%2C+Roy+Ka%2DWei+Lee%2C+and+Charu+C%2E+Aggarwal%2E+2021%2E+Syntax+matters%21+syntax%2Dcontrolled+in+text+style+transfer%2E+In+Proceedings+of+the+International+Conference+on+Recent+Advances+in+Natural+Languag)
30. Zhiting Hu and Li Erran Li. 2021. A causal lens for controllable text generation. Advances in Neural Information Processing Systems, 34:24941–24955. [↗](https://doi.org/10.48550/arxiv.2201.09119)
31. Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is chatgpt a good translator? a preliminary study. arXiv preprint arXiv:2301.08745. [↗](https://arxiv.org/abs/2301.08745) [our card](https://md.research.elamur.ai/free/papers/jiao-2023-chatgpt-good-translator-yes-gpt/)
32. Di Jin, Zhijing Jin, Zhiting Hu, Olga Vechtomova, and Rada Mihalcea. 2022. Deep learning for text style transfer: A survey. Computational Linguistics, 48(1):155–205. [↗](https://doi.org/10.1162/coli_a_00426)
33. Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. 2019. Ctrl: A conditional transformer language model for controllable generation. ArXiv, abs/1909.05858. [↗](https://arxiv.org/abs/1909.05858) [our card](https://md.research.elamur.ai/free/papers/keskar-2019-ctrl-conditional-transformer-language-model/)
34. J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Technical report, Naval Technical Training Command Millington TN Research Branch. [↗](https://doi.org/10.21236/ada006655)
35. Sachin Kumar, Biswajit Paria, and Yulia Tsvetkov. 2022. Gradient-based constrained sampling from language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2251–2277, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sachin+Kumar%2C+Biswajit+Paria%2C+and+Yulia+Tsvetkov%2E+2022%2E+Gradient%2Dbased+constrained+sampling+from+language+models%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+in+Natural+Language+Process)
36. Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. 2022. SummaC: Re-visiting NLI-based models for inconsistency detection in summarization. Transactions of the Association for Computational Linguistics, 10:163–177. [↗](https://doi.org/10.1162/tacl_a_00453)
37. Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Huu Nguyen. 2023. Chatgpt beyond english: Towards a comprehensive evaluation of large language models in multilingual learning. arXiv preprint arXiv:2304.05613. [↗](https://arxiv.org/abs/2304.05613) [our card](https://md.research.elamur.ai/free/papers/lai-2023-chatgpt-beyond-english-comprehensive-evaluation/)
38. Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsunori B Hashimoto. 2022. Diffusion-lm improves controllable text generation. Advances in Neural Information Processing Systems, 35:4328– 4343. [↗](https://doi.org/10.48550/arxiv.2205.14217) [our card](https://md.research.elamur.ai/free/papers/li-2022-diffusion-lm-improves-controllable-text/)
39. Xiao Li, Guanyi Chen, Chenghua Lin, and Ruizhe Li. 2020. DGST: a dual-generator network for text style transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7131–7136, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Xiao+Li%2C+Guanyi+Chen%2C+Chenghua+Lin%2C+and+Ruizhe+Li%2E+2020%2E+DGST%3A+a+dual%2Dgenerator+network+for+text+style+transfer%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+Processi)
40. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chin%2DYew+Lin%2E+2004%2E+ROUGE%3A+A+package+for+automatic+evaluation+of+summaries%2E+In+Text+Summarization+Branches+Out%2C+pages+74%E2%80%9381%2C+Barcelona%2C+Spain%2E+Association+for+Computational+Linguistics%2E)
41. Xiaochen Liu, Yang Gao, Yu Bai, Jiawei Li, Yinan Hu, Heyan Huang, and Boxing Chen. 2022a. PSP: Pre-trained soft prompts for few-shot abstractive summarization. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6355–6368, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2204.04413)
42. Yizhu Liu, Qi Jia, and Kenny Zhu. 2022b. Length control in abstractive summarization by pretraining information selection. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6885– 6895, Dublin, Ireland. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yizhu+Liu%2C+Qi+Jia%2C+and+Kenny+Zhu%2E+2022b%2E+Length+control+in+abstractive+summarization+by+pretraining+information+selection%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computationa)
43. Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2022. Readability controllable biomedical document summarization. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4667–4680, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2022.findings-emnlp.343)
44. Eric Malmi, Aliaksei Severyn, and Sascha Rothe. 2020. Unsupervised text style transfer with padded masked language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8671–8680, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Eric+Malmi%2C+Aliaksei+Severyn%2C+and+Sascha+Rothe%2E+2020%2E+Unsupervised+text+style+transfer+with+padded+masked+language+models%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Languag)
45. Philip M McCarthy and Scott Jarvis. 2010. Mtld, vocd-d, and hd-d: A validation study of sophisticated approaches to lexical diversity assessment. Behavior research methods, 42(2):381–392. [↗](https://doi.org/10.3758/brm.42.2.381)
46. Remi Mir, Bjarke Felbo, Nick Obradovich, and Iyad Rahwan. 2019. Evaluating style transfer for text. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 495–504, Minneapolis, Minnesota. Association for Computational Linguistics. Linguistics (Volume 1: Long Papers), pages 416–431, Dublin, Ireland. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1904.02295) [our card](https://md.research.elamur.ai/free/papers/mir-2019-evaluating-style-transfer-text/)
47. Nasim Nouri. 2022. Text style transfer via optimal transport. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2532–2541, Seattle, United States. Association for Computational Linguistics. [↗](https://doi.org/10.48448/7pbj-g251)
48. Sudha Rao and Joel Tetreault. 2018. Dear sir or madam, may I introduce the GYAFC dataset: Corpus, benchmarks and metrics for formality style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 129–140, New Orleans, Louisiana. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sudha+Rao+and+Joel+Tetreault%2E+2018%2E+Dear+sir+or+madam%2C+may+I+introduce+the+GYAFC+dataset%3A+Corpus%2C+benchmarks+and+metrics+for+formality+style+transfer%2E+In+Proceedings+of+the+2018+Conference+of+the+Nort) [our card](https://md.research.elamur.ai/free/papers/rao-2018-dear-sir-or-madam-may/)
49. OpenAI. 2023. Gpt-4 technical report. abs/2303.08774. [↗](https://arxiv.org/abs/2303.08774) [our card](https://md.research.elamur.ai/free/papers/openai-2023-gpt-4-technical-report/)
50. ArXiv, Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744. [find](https://scholar.google.com/scholar?q=ArXiv%2C+Long+Ouyang%2C+Jeffrey+Wu%2C+Xu+Jiang%2C+Diogo+Almeida%2C+Carroll+Wainwright%2C+Pamela+Mishkin%2C+Chong+Zhang%2C+Sandhini+Agarwal%2C+Katarina+Slama%2C+Alex+Ray%2C+et+al%2E+2022%2E+Training+language+models+to+follow+in)
51. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. [↗](https://doi.org/10.3115/1073083.1073135) [our card](https://md.research.elamur.ai/free/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
52. Keqin Peng, Liang Ding, Qihuang Zhong, Li Shen, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. 2023. Towards making the most of chatgpt for machine translation. Available at SSRN 4390455. [↗](https://doi.org/10.13140/rg.2.2.24416.97283) [our card](https://md.research.elamur.ai/free/papers/peng-2023-making-most-chatgpt-machine-translation/)
53. Dongqi Pu, Xudong Hong, Pin-Jie Lin, Ernie Chang, and Vera Demberg. 2022. Two-stage movie script summarization: An efficient method for low-resource long document summarization. In Proceedings of The Workshop on Automatic Summarization for Creative Writing, pages 57–66, Gyeongju, Republic of Korea. Association for Computational Linguistics. [↗](https://doi.org/10.22028/d291-38833)
54. Dongqi Pu and Khalil Sima’an. 2022. Passing parser uncertainty to the transformer: Labeled dependency distributions for neural machine translation. In Proceedings of the 23rd Annual Conference of the European Association for Machine Translation, pages 41–50, Ghent, Belgium. European Association for Machine Translation. [find](https://scholar.google.com/scholar?q=Dongqi+Pu+and+Khalil+Sima%E2%80%99an%2E+2022%2E+Passing+parser+uncertainty+to+the+transformer%3A+Labeled+dependency+distributions+for+neural+machine+translation%2E+In+Proceedings+of+the+23rd+Annual+Conference+of+the+)
55. Dongqi Pu, Yifan Wang, and Vera Demberg. 2023. Incorporating distributions of discourse structure for long document abstractive summarization. arXiv preprint arXiv:2305.16784. [↗](https://arxiv.org/abs/2305.16784)
56. Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt a general-purpose natural language processing task solver? arXiv preprint arXiv:2302.06476. [↗](https://arxiv.org/abs/2302.06476) [our card](https://md.research.elamur.ai/free/papers/qin-2023-chatgpt-general-purpose-natural-language/)
57. Abhinav Ramesh Kashyap, Devamanyu Hazarika, Min-Yen Kan, Roger Zimmermann, and Soujanya Poria. 2022. So different yet so alike! constrained unsupervised text style transfer. In Proceedings of the 60th Annual Meeting of the Association for Computational Emily Reif, Daphne Ippolito, Ann Yuan, Andy Coenen, Chris Callison-Burch, and Jason Wei. 2022. A recipe for arbitrary text style transfer with large language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 837–848, Dublin, Ireland. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2205.04093) [our card](https://md.research.elamur.ai/free/papers/reif-2021-recipe-arbitrary-text-style-transfer/)
58. Jürgen Rudolph, Samson Tan, and Shannon Tan. 2023. Chatgpt: Bullshit spewer or the end of traditional assessments in higher education? Journal of Applied Learning and Teaching, 6(1). [↗](https://doi.org/10.37074/jalt.2023.6.1.9)
59. Ritesh Sarkhel, Moniba Keymanesh, Arnab Nandi, and Srinivasan Parthasarathy. 2020. Interpretable multi-headed attention for abstractive summarization at controllable lengths. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6871–6882, Barcelona, Spain (Online). International Committee on Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ritesh+Sarkhel%2C+Moniba+Keymanesh%2C+Arnab+Nandi%2C+and+Srinivasan+Parthasarathy%2E+2020%2E+Interpretable+multi%2Dheaded+attention+for+abstractive+summarization+at+controllable+lengths%2E+In+Proceedings+of+the+28t)
60. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073– 1083, Vancouver, Canada. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1704.04368)
61. Chenhui Shen, Liying Cheng, Lidong Bing, Yang You, and Luo Si. 2022a. SentBS: Sentence-level beam search for controllable summarization. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10256–10265, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chenhui+Shen%2C+Liying+Cheng%2C+Lidong+Bing%2C+Yang+You%2C+and+Luo+Si%2E+2022a%2E+SentBS%3A+Sentence%2Dlevel+beam+search+for+controllable+summarization%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+in+N)
62. Chenhui Shen, Liying Cheng, Ran Zhou, Lidong Bing, Yang You, and Luo Si. 2022b. MReD: A meta-review dataset for structure-controllable text generation. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2521–2535, Dublin, Ireland. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chenhui+Shen%2C+Liying+Cheng%2C+Ran+Zhou%2C+Lidong+Bing%2C+Yang+You%2C+and+Luo+Si%2E+2022b%2E+MReD%3A+A+meta%2Dreview+dataset+for+structure%2Dcontrollable+text+generation%2E+In+Findings+of+the+Association+for+Computational)
63. Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303.17580. [↗](https://arxiv.org/abs/2303.17580) [our card](https://md.research.elamur.ai/free/papers/shen-2023-hugginggpt-solving-ai-tasks-chatgpt/)
64. Dominik Sobania, Martin Briesch, Carol Hanna, and Justyna Petke. 2023. An analysis of the automatic bug fixing performance of chatgpt. arXiv preprint arXiv:2301.08653. [↗](https://arxiv.org/abs/2301.08653)
65. Ewoenam Kwaku Tokpo and Toon Calders. 2022. Text style transfer for bias mitigation using masked language modeling. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Student Research Workshop, pages 163–171, Hybrid: Seattle, Washington + Online. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2201.08643)
66. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+Advances+in+neural+information+processing+)
67. Jiaan Wang, Yunlong Liang, Fandong Meng, Zhixu Li, Jianfeng Qu, and Jie Zhou. 2023a. Cross-lingual summarization via chatgpt. arXiv preprint arXiv:2302.14229. [↗](https://arxiv.org/abs/2302.14229)
68. Jiaan Wang, Yunlong Liang, Fandong Meng, Beiqi Zou, Zhixu Li, Jianfeng Qu, and Jie Zhou. 2023b. Zero-shot cross-lingual summarization via large language models. [↗](https://doi.org/10.13140/rg.2.2.10074.72645)
69. Colin G West. 2023. Ai and the fci: Can chatgpt project an understanding of introductory physics? arXiv preprint arXiv:2303.01067. [↗](https://arxiv.org/abs/2303.01067)
70. Chien-Sheng Wu, Linqing Liu, Wenhao Liu, Pontus Stenetorp, and Caiming Xiong. 2021. Controllable abstractive dialogue summarization with sketch supervision. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 5108–5122, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chien%2DSheng+Wu%2C+Linqing+Liu%2C+Wenhao+Liu%2C+Pontus+Stenetorp%2C+and+Caiming+Xiong%2E+2021%2E+Controllable+abstractive+dialogue+summarization+with+sketch+supervision%2E+In+Findings+of+the+Association+for+Computat)
71. Kexin Yang, Dayiheng Liu, Wenqiang Lei, Baosong Yang, Mingfeng Xue, Boxing Chen, and Jun Xie. 2022. Tailor: A prompt-based approach to attribute-based controlled text generation. arXiv preprint arXiv:2204.13362. [↗](https://arxiv.org/abs/2204.13362)
72. Xianjun Yang, Yan Li, Xinlu Zhang, Haifeng Chen, and Wei Cheng. 2023a. Exploring the limits of chatgpt for query or aspect-based text summarization. arXiv preprint arXiv:2302.08081. [↗](https://arxiv.org/abs/2302.08081)
73. Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. 2023b. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303.11381. [↗](https://arxiv.org/abs/2303.11381)
74. Hanqing Zhang and Dawei Song. 2022. DisCup: Discriminator cooperative unlikelihood prompt-tuning for controllable text generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3392–3406, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2022.emnlp-main.223)
75. Hanqing Zhang, Haolin Song, Shaoyu Li, Ming Zhou, and Dawei Song. 2022. A survey of controllable text generation using transformer-based pre-trained language models. arXiv preprint arXiv:2201.05337. [↗](https://arxiv.org/abs/2201.05337)
76. Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, et al. 2023. A comprehensive survey on pretrained foundation models: A history from bert to chatgpt. arXiv preprint arXiv:2302.09419. [↗](https://arxiv.org/abs/2302.09419)
77. Wanrong Zhu, Xin Wang, Tsu-Jui Fu, An Yan, Pradyumna Narayana, Kazoo Sone, Sugato Basu, and William Yang Wang. 2021. Multimodal text style transfer for outdoor vision-and-language navigation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1207–1221, Online. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2007.00229)