---
title: "ChangJuan — paper by Yang et al. (2026) — summary and ideas"
description: "The authors present ChangJuan, which they describe as the first benchmark for evaluating book-length Chinese fiction. 15 references."
authors: ["Dingyi Yang", "Mingshuo Wang", "Qin Jin"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.2044/"
doi: "10.18653/v1/2026.findings-acl.2044"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yang-2026-changjuan-comprehensive-benchmark-book-length/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yang-2026-changjuan-comprehensive-benchmark-book-length/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/26/) / ChangJuan: A Comprehensive Benchmark for Book-Length Chinese Story Evaluation

# ChangJuan: A Comprehensive Benchmark for Book-Length Chinese Story Evaluation

**Authors:** [Dingyi Yang](https://md.research.elamur.ai/free/en/authors/dingyi-yang/), Mingshuo Wang and [Qin Jin](https://md.research.elamur.ai/free/en/authors/qin-jin/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.2044/) · [PDF](https://aclanthology.org/2026.findings-acl.2044.pdf) · [DOI 10.18653/v1/2026.findings-acl.2044](https://doi.org/10.18653/v1/2026.findings-acl.2044) · [OpenAlex](https://openalex.org/W7166797328)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Automatically judging the quality of full-length novels is still poorly studied, especially for languages other than English. The authors present ChangJuan, which they describe as the first benchmark for evaluating book-length Chinese fiction. It covers 300 novels published in the last five years, with the newest titles held out for testing to reduce contamination risk. For each novel the benchmark provides metadata, average ratings and user reviews collected from Douban. Individual reviews are subjective, so the authors built an LLM-based pipeline that turns them into consensus pros and cons, grouped under a unified taxonomy of evaluation aspects. They also propose an improved summary-based evaluator. It draws on summaries that balance length against detail, plus selected excerpts, and it weights aspects by genre before producing per-aspect reviews and an overall score. This approach matches or beats methods that read the entire book. Training an 8B model, CLEM, within this framework clearly improves agreement with human ratings. The work therefore provides a reproducible and culturally specific resource for long-form story evaluation.

## Key points
- **Dataset scale:** ChangJuan's novels average about 292K tokens. The benchmark contains 97.5K aspect-guided reviews and 22.1K distilled general viewpoints. For copyright reasons, only summaries are released, not the full texts.
- **Viewpoint distillation pipeline:**
  - Deepseek-Chat extracts viewpoints from the reviews.
  - High-vote and long-tail reviews are processed separately, which keeps minority opinions in the final set.
  - All viewpoints are mapped onto eight major aspects.
- **Viewpoint quality:** Automatic checks measure precision at 0.946 and recall at 0.932. Human verification gives 0.91.
- **CLEM results:** CLEM outperforms open-source baselines. It lifts Qwen3's Kendall's tau with human judgments from 24.8 to 34.1.
- **Genre differences:** Aspect importance varies by genre. Historical and military fiction depend most on world-building, while mystery and suspense depend most on creativity and plot.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Nanyang Technological University](https://md.research.elamur.ai/free/en/universities/nanyang-technological-university/) — University, Singapore, Singapore · 184 works · 27,484 citations
- [Renmin University of China](https://md.research.elamur.ai/free/en/universities/renmin-university-of-china/) — University, Beijing, China · 139 works · 17,752 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works

🧭 Topics (1): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works

## Cite

Cite as: Dingyi Yang, Mingshuo Wang and Qin Jin. 2026. ChangJuan: A Comprehensive Benchmark for Book-Length Chinese Story Evaluation. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yang2026changjuan,
  title = {{ChangJuan: A Comprehensive Benchmark for Book-Length Chinese Story Evaluation}},
  author = {Dingyi Yang, Mingshuo Wang and Qin Jin},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.2044},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.2044}
}
```

- [← previous Chameleons and Guardians: Unveiling the Divergence in Personality Plasticity and Cognitive Resistance across LLMs](https://md.research.elamur.ai/free/en/papers/wu-2026-chameleons-guardians-unveiling-divergence-personality/)
- [next → Character Iconicity vs. Arbitrariness: An Arabic NLP Perspective](https://md.research.elamur.ai/free/en/papers/alomari-2026-character-iconicity-vs-arbitrariness-arabic/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/26/)

## References

15 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jian Guan, Zhexin Zhang, Zhuoer Feng, Zitao Liu, Wenbiao Ding, Xiaoxi Mao, Changjie Fan, and Minlie Huang. 2021. Openmeva: A benchmark for evaluating open-ended story generation metrics. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internati… [find](https://scholar.google.com/scholar?q=Jian+Guan%2C+Zhexin+Zhang%2C+Zhuoer+Feng%2C+Zitao+Liu%2C+Wenbiao+Ding%2C+Xiaoxi+Mao%2C+Changjie+Fan%2C+and+Minlie+Huang%2E+2021%2E+Openmeva%3A+A+benchmark+for+evaluating+open%2Dended+story+generation+metrics%2E+In+Proceeding)
2. Yapei Chang, Kyle Lo, Tanya Goyal, and Mohit Iyyer. 2024. [Booookscore: A systematic exploration of book-length summarization in the era of llms](https://md.research.elamur.ai/free/en/papers/chang-2023-booookscore-systematic-exploration-book-length/). In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.
3. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net. \#\#…
4. Maurice G Kendall. 1938. A new measure of rank correlation. Biometrika, 30(1/2):81–93. [find](https://scholar.google.com/scholar?q=Maurice+G+Kendall%2E+1938%2E+A+new+measure+of+rank+correlation%2E+Biometrika%2C+30%281%2F2%29%3A81%E2%80%9393%2E)
5. Zhen Li, Xiaohan Xu, Tao Shen, Can Xu, Jia-Chen Gu, Yuxuan Lai, Chongyang Tao, and Shuai Ma. 2024. [Leveraging large language models for nlg evaluation: Advances and challenges](https://md.research.elamur.ai/free/en/papers/li-2024-leveraging-large-language-models-nlg/). Preprint, arXiv:2401.07103.
6. Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. 2024. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, arXiv:2405.04434. [find](https://scholar.google.com/scholar?q=Aixin+Liu%2C+Bei+Feng%2C+Bin+Wang%2C+Bingxuan+Wang%2C+Bo+Liu%2C+Chenggang+Zhao%2C+Chengqi+Dengr%2C+Chong+Ruan%2C+Damai+Dai%2C+Daya+Guo%2C+et+al%2E+2024%2E+Deepseek%2Dv2%3A+A+strong%2C+economical%2C+and+efficient+mixture%2Dof%2Dexperts+l)
7. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022.… — [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
8. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: A method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA, pages 311–318. ACL.
9. Karl Pearson. 1895. Vii. note on regression and inheritance in the case of two parents. proceedings of the royal society of London, 58(347-352):240–242. [find](https://scholar.google.com/scholar?q=Karl+Pearson%2E+1895%2E+Vii%2E+note+on+regression+and+inheritance+in+the+case+of+two+parents%2E+proceedings+of+the+royal+society+of+London%2C+58%28347%2D352%29%3A240%E2%80%93242%2E)
10. Charles Spearman. 1961. The proof and measurement of association between two things. The American Journal of Psychology, 15:72–101. [find](https://scholar.google.com/scholar?q=Charles+Spearman%2E+1961%2E+The+proof+and+measurement+of+association+between+two+things%2E+The+American+Journal+of+Psychology%2C+15%3A72%E2%80%93101%2E)
11. Danqing Wang, Kevin Yang, Hanlin Zhu, Xiaomeng Yang, Andrew Cohen, Lei Li, and Yuandong Tian. 2024. Learning personalized story evaluation. Preprint, arXiv:2310.03304. — [arXiv:2310.03304](https://arxiv.org/abs/2310.03304) [find](https://scholar.google.com/scholar?q=Danqing+Wang%2C+Kevin+Yang%2C+Hanlin+Zhu%2C+Xiaomeng+Yang%2C+Andrew+Cohen%2C+Lei+Li%2C+and+Yuandong+Tian%2E+2024%2E+Learning+personalized+story+evaluation%2E+Preprint%2C+arXiv%3A2310%2E03304%2E+%E2%80%94+%5BarXiv%3A2310%2E03304%5D%28https%3A%2F%2Farx)
12. Zhuohan Xie, Trevor Cohn, and Jey Han Lau. 2023. The next chapter: A study of large language models in storytelling. Preprint, arXiv:2301.09790. [find](https://scholar.google.com/scholar?q=Zhuohan+Xie%2C+Trevor+Cohn%2C+and+Jey+Han+Lau%2E+2023%2E+The+next+chapter%3A+A+study+of+large+language+models+in+storytelling%2E+Preprint%2C+arXiv%3A2301%2E09790%2E)
13. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jin… — [Qwen3 Technical Report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/)
14. Dingyi Yang and Qin Jin. 2025. What matters in evaluating book-length stories? a systematic study of long story evaluation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16375– 16398, Vienna, Austria. Association for Computa… [find](https://scholar.google.com/scholar?q=Dingyi+Yang+and+Qin+Jin%2E+2025%2E+What+matters+in+evaluating+book%2Dlength+stories%3F+a+systematic+study+of+long+story+evaluation%2E+In+Proceedings+of+the+63rd+Annual+Meeting+of+the+Association+for+Computation)
15. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q%2E+Weinberger%2C+and+Yoav+Artzi%2E+2020%2E+Bertscore%3A+Evaluating+text+generation+with+BERT%2E+In+8th+International+Conference+on+Learning+Representations%2C+ICLR+2)

*Updated 2 November 2026*