---
title: "What Makes Writing Great? First Experiments on Article Quality"
description: "They build a corpus from the New York Times Annotated Corpus. 41 references, cited by 29 papers here."
authors: ["Annie Louis", "Ani Nenkova"]
year: 2013
venue: "Transactions of the Association for Computational Linguistics"
source: "https://aclanthology.org/Q13-1028/"
doi: "10.1162/tacl_a_00232"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/louis-2013-what-makes-writing-great-first/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/louis-2013-what-makes-writing-great-first/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2013](https://md.research.elamur.ai/free/en/papers/2013/) / What Makes Writing Great? First Experiments on Article Quality Prediction in the Science Journalism Domain

# What Makes Writing Great? First Experiments on Article Quality Prediction in the Science Journalism Domain

**Authors:** [Annie Louis](https://md.research.elamur.ai/free/en/authors/annie-louis/) and [Ani Nenkova](https://md.research.elamur.ai/free/en/authors/ani-nenkova/) · **Year:** 2013 · **Venue:** [Transactions of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/tacl/) ([Transactions of the Association for Computational Linguistics (TACL) 2013](https://md.research.elamur.ai/free/en/venues/tacl/2013/)) · **Publisher:** [The MIT Press](https://md.research.elamur.ai/free/en/publishers/the-mit-press/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/Q13-1028/) · [PDF](https://aclanthology.org/Q13-1028.pdf) · [DOI 10.1162/tacl\_a\_00232](https://doi.org/10.1162/tacl_a_00232) · [OpenAlex](https://openalex.org/W2184410296)

**Citations:** 74 citations (OpenAlex, 2026-10-14) · [29](https://md.research.elamur.ai/free/en/papers/louis-2013-what-makes-writing-great-first/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Search and recommendation tools can already judge spelling, grammar, coherence, topical relevance and readability. They cannot yet recognize writing that is genuinely excellent: engaging, vivid and well crafted. Louis and Nenkova address this gap for science journalism. They build a corpus from the New York Times Annotated Corpus. Articles by writers featured in "The Best American Science Writing" anthologies are labeled VERY GOOD, and science articles by other authors are labeled TYPICAL. A topic-matched version of the corpus pairs each VERY GOOD article with similar TYPICAL ones.

The authors design genre-specific features for six facets of science writing: visual imagery, focus on people, beautiful language, sub-genre, sentiment, and depth of research description. They combine these with established readability, discourse-coherence, fiction-interest and bag-of-words features in SVM classifiers.

The genre-specific features alone beat general writing measures. Merging writing and content features reaches about 82–84% accuracy. This suggests that quality at this level can be modeled automatically and that topic and style contribute complementary signals.

## Key points
- The corpus covers NYT articles from 1999–2007. Of 23,709 science articles, 3,530 are VERY GOOD (from 63 anthology seed articles by about 40 authors) and 20,242 are TYPICAL. The topic-paired corpus has 35,300 pairs.
- There are two tasks: "any topic" classification on a balanced test set, and "same topic" pairwise choice of the better article. The chance baseline for both is 50%.
- The 41 interest-science features score 75.3% on any topic and 68.0% on same topic. This exceeds the readability (Pitler and Nenkova, 2008), Entity Grid/discourse, and McIntyre and Lapata (2009) fiction-interest feature sets.
- All 102 writing features together score 76.7% and 74.7%. Bag-of-words with 1000 words scores 81.2% and 82.1%.
- Combining writing and word features in one vector gives 82.6% and 84.0%, a significant gain over words alone. An oracle combination bound reaches 87.6% and 93.8%.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Pennsylvania](https://md.research.elamur.ai/free/en/universities/university-of-pennsylvania/) — University, Philadelphia, United States · 221 works · 63,234 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🤝 General Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/general-social-sciences/) · 227 works

## Cite

Cite as: Annie Louis and Ani Nenkova. 2013. What Makes Writing Great? First Experiments on Article Quality Prediction in the Science Journalism Domain. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{louis2013what,
  title = {{What Makes Writing Great? First Experiments on Article Quality Prediction in the Science Journalism Domain}},
  author = {Annie Louis and Ani Nenkova},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2013},
  doi = {10.1162/tacl\_a\_00232},
  url = {https://doi.org/10.1162/tacl_a_00232}
}
```

- [← previous SemEval-2013 Task 9 : Extraction of Drug-Drug Interactions from Biomedical Texts (DDIExtraction 2013)](https://md.research.elamur.ai/free/en/papers/bedmar-2013-semeval-2013-task-9-extraction/)
- [next → Learning Latent Personas of Film Characters](https://md.research.elamur.ai/free/en/papers/bamman-2013-learning-latent-personas-film-characters/)
- [↑ back to papers of 2013](https://md.research.elamur.ai/free/en/papers/2013/)

## References

41 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. C. F. Baker, C. J. Fillmore, and J. B. Lowe. 1998. [The berkeley framenet project](https://md.research.elamur.ai/free/en/papers/baker-1998-berkeley-framenet/). In Proceedings of COLING-ACL, pages 86–90.
2. Z. Bao, B. Kimelfeld, and Y. Li. 2011. A graph approach to spelling correction in domain-centric search. In Proceedings of ACL-HLT, pages 905–914. [find](https://scholar.google.com/scholar?q=Z%2E+Bao%2C+B%2E+Kimelfeld%2C+and+Y%2E+Li%2E+2011%2E+A+graph+approach+to+spelling+correction+in+domain%2Dcentric+search%2E+In+Proceedings+of+ACL%2DHLT%2C+pages+905%E2%80%93914%2E)
3. R. Barzilay and M. Lapata. 2008. [Modeling local coherence: An entity-based approach](https://md.research.elamur.ai/free/en/papers/barzilay-2008-modeling-local-coherence-entity-based/). Computational Linguistics, 34(1):1–34.
4. R. Barzilay, N. Elhadad, and K. McKeown. 2002. Inferring strategies for sentence ordering in multi-document summar ization. Journal of Artificial Intelligence Research, 17:35–55. [find](https://scholar.google.com/scholar?q=R%2E+Barzilay%2C+N%2E+Elhadad%2C+and+K%2E+McKeown%2E+2002%2E+Inferring+strategies+for+sentence+ordering+in+multi%2Ddocument+summar+ization%2E+Journal+of+Artificial+Intelligence+Research%2C+17%3A35%E2%80%9355%2E)
5. D.M. Blei, A.Y. Ng, and M.I. Jordan. 2003. [Latent dirichlet allocation](https://md.research.elamur.ai/free/en/papers/blei-2003-latent-dirichlet-allocation/). the Journal of machine Learning research, 3:993–1022. D. Blum, M. Knudson, and R. M. Henig, editors. 2006. A field guide for science writers: the official guide of the national association of science writers.
6. Oxford University Press, New York. E. Brill and R.C. Moore. 2000. [An improved error model for noisy channel spelling correction](https://md.research.elamur.ai/free/en/papers/brill-2000-improved-error-model-noisy-channel/). In Proceedings of ACL, pages 286–293.
7. S. F. Chen and J. Goodman. 1996. [An empirical study of smoothing techniques for language modeling](https://md.research.elamur.ai/free/en/papers/chen-1996-empirical-study-smoothing-techniques-language/). In Proceedings of ACL, pages 310–318.
8. S. Cucerzan and E. Brill. 2004. Spelling correction as an iterative process that exploits the collective knowledge of web users. In Proceedings of EMNLP, pages 293– 300. [find](https://scholar.google.com/scholar?q=S%2E+Cucerzan+and+E%2E+Brill%2E+2004%2E+Spelling+correction+as+an+iterative+process+that+exploits+the+collective+knowledge+of+web+users%2E+In+Proceedings+of+EMNLP%2C+pages+293%E2%80%93+300%2E)
9. R. Dale and A. Kilgarriff. 2010. Helping our own: text massaging for computational linguistics as a new shared task. In Proceedings of INLG, pages 263–267. [find](https://scholar.google.com/scholar?q=R%2E+Dale+and+A%2E+Kilgarriff%2E+2010%2E+Helping+our+own%3A+text+massaging+for+computational+linguistics+as+a+new+shared+task%2E+In+Proceedings+of+INLG%2C+pages+263%E2%80%93267%2E)
10. M. C. De Marneffe, B. MacCartney, and C. D. Manning. 2006. [Generating typed dependency parses from phrase structure parses](https://md.research.elamur.ai/free/en/papers/marneffe-2006-generating-typed-dependency-parses-phrase/). In Proceedings of LREC, volume 6, pages 449–454.
11. P. Diederich. 1974. Measuring Growth in English. National Council of Teachers of English. [find](https://scholar.google.com/scholar?q=P%2E+Diederich%2E+1974%2E+Measuring+Growth+in+English%2E+National+Council+of+Teachers+of+English%2E)
12. M. Elsner, J. Austerweil, and E. Charniak. 2007. [A unified local and global model for discourse coherence](https://md.research.elamur.ai/free/en/papers/elsner-2007-unified-local-global-model-discourse/). In Proceedings of NAACL-HLT, pages 436–443.
13. J. R. Finkel, T. Grenager, and C. Manning. 2005. [Incorporating non-local information into information extraction systems by gibbs sampling](https://md.research.elamur.ai/free/en/papers/finkel-2005-incorporating-non-local-information-into/). In Proceedings of ACL, pages 363–370.
14. R. Flesch. 1948. A new readability yardstick. Journal of Applied Psychology, 32:221 – 233. [find](https://scholar.google.com/scholar?q=R%2E+Flesch%2E+1948%2E+A+new+readability+yardstick%2E+Journal+of+Applied+Psychology%2C+32%3A221+%E2%80%93+233%2E)
15. A.C. Graesser, D.S. McNamara, M.M. Louwerse, and Z. Cai. 2004. Coh-Metrix: Analysis of text on cohesion and language. Behavior Research Methods Instruments and Computers, 36(2):193–202. [find](https://scholar.google.com/scholar?q=A%2EC%2E+Graesser%2C+D%2ES%2E+McNamara%2C+M%2EM%2E+Louwerse%2C+and+Z%2E+Cai%2E+2004%2E+Coh%2DMetrix%3A+Analysis+of+text+on+cohesion+and+language%2E+Behavior+Research+Methods+Instruments+and+Computers%2C+36%282%29%3A193%E2%80%93202%2E)
16. H. Ji and D. Lin. 2009. Gender and animacy knowledge discovery from web-scale n-grams for unsupervised person name detection. In Proceedings of PACLIC. [find](https://scholar.google.com/scholar?q=H%2E+Ji+and+D%2E+Lin%2E+2009%2E+Gender+and+animacy+knowledge+discovery+from+web%2Dscale+n%2Dgrams+for+unsupervised+person+name+detection%2E+In+Proceedings+of+PACLIC%2E)
17. D. Klein and C.D. Manning. 2003. [Accurate unlexicalized parsing](https://md.research.elamur.ai/free/en/papers/klein-2003-accurate-unlexicalized-parsing/). In Proceedings of ACL, pages 423–430.
18. S.M. Kosslyn. 1980. Image and mind. [find](https://scholar.google.com/scholar?q=S%2EM%2E+Kosslyn%2E+1980%2E+Image+and+mind%2E)
19. Harvard University Press. M. Lapata. 2003. [Probabilistic text structuring: Experiments with sentence ordering](https://md.research.elamur.ai/free/en/papers/lapata-2003-probabilistic-text-structuring-experiments-sentence/). In Proceedings of ACL, pages 545–552.
20. C. Lin and E. Hovy. 2000. [The automated acquisition of topic signatures for text summarization](https://md.research.elamur.ai/free/en/papers/lin-2000-automated-acquisition-topic-signatures-text/). In Proceedings of COLING, pages 495–501.
21. D. Lin, K. W. Church, H. Ji, S. Sekine, D. Yarowsky, S. Bergsma, K. Patil, E. Pitler, R. Lathbury, V. Rao, K. Dalwani, and S. Narsale. 2010. New tools for web-scale n-grams. In Proceedings of LREC. [find](https://scholar.google.com/scholar?q=D%2E+Lin%2C+K%2E+W%2E+Church%2C+H%2E+Ji%2C+S%2E+Sekine%2C+D%2E+Yarowsky%2C+S%2E+Bergsma%2C+K%2E+Patil%2C+E%2E+Pitler%2C+R%2E+Lathbury%2C+V%2E+Rao%2C+K%2E+Dalwani%2C+and+S%2E+Narsale%2E+2010%2E+New+tools+for+web%2Dscale+n%2Dgrams%2E+In+Proceedings+of+LREC%2E)
22. N. McIntyre and M. Lapata. 2009. Learning to tell tales: A data-driven approach to story generation. In Proceedings of ACL-IJCNLP, pages 217–225. [find](https://scholar.google.com/scholar?q=N%2E+McIntyre+and+M%2E+Lapata%2E+2009%2E+Learning+to+tell+tales%3A+A+data%2Ddriven+approach+to+story+generation%2E+In+Proceedings+of+ACL%2DIJCNLP%2C+pages+217%E2%80%93225%2E)
23. R. J. Mooney and L. Roy. 2000. Content-based book recommending using learning for text categorization. In Proceedings of the fifth ACM conference on Digital libraries, pages 195–204. [find](https://scholar.google.com/scholar?q=R%2E+J%2E+Mooney+and+L%2E+Roy%2E+2000%2E+Content%2Dbased+book+recommending+using+learning+for+text+categorization%2E+In+Proceedings+of+the+fifth+ACM+conference+on+Digital+libraries%2C+pages+195%E2%80%93204%2E)
24. A. Nakhimovsky. 1988. [Aspect, aspectual class, and the temporal structure of narrative](https://md.research.elamur.ai/free/en/papers/nakhimovsky-1988-aspect-aspectual-class-temporal-structure/). Computational Linguistics, 14(2):29–43, June.
25. M. Pazzani, J. Muramatsu, and D. Billsus. 1996. Syskill & Webert: Identifying interesting web sites. In Proceedings of AAAI, pages 54–61. [find](https://scholar.google.com/scholar?q=M%2E+Pazzani%2C+J%2E+Muramatsu%2C+and+D%2E+Billsus%2E+1996%2E+Syskill+%26+Webert%3A+Identifying+interesting+web+sites%2E+In+Proceedings+of+AAAI%2C+pages+54%E2%80%9361%2E)
26. E. Pitler and A. Nenkova. 2008. [Revisiting readability: A unified framework for predicting text quality](https://md.research.elamur.ai/free/en/papers/pitler-2008-revisiting-readability-unified-framework-predicting/). In Proceedings of EMNLP, pages 186–195.
27. E. Pitler and A. Nenkova. 2009. [Using syntax to disambiguate explicit discourse connectives in text](https://md.research.elamur.ai/free/en/papers/pitler-2009-syntax-disambiguate-explicit-discourse-connectives/). In Proceedings of ACL-IJCNLP, pages 13–16.
28. R Development Core Team, 2011. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing. [find](https://scholar.google.com/scholar?q=R+Development+Core+Team%2C+2011%2E+R%3A+A+Language+and+Environment+for+Statistical+Computing%2E+R+Foundation+for+Statistical+Computing%2E)
29. D. Ramage, D. Hall, R. Nallapati, and C.D. Manning. 2009. [Labeled lda: A supervised topic model for credit attribution in multi-labeled corpora](https://md.research.elamur.ai/free/en/papers/ramage-2009-labeled-lda-supervised-topic-model/). In Proceedings of EMNLP, pages 248–256.
30. A. Rozovskaya and D. Roth. 2010. Generating confusion sets for context-sensitive error correction. In Proceedings of EMNLP, pages 961–970. [find](https://scholar.google.com/scholar?q=A%2E+Rozovskaya+and+D%2E+Roth%2E+2010%2E+Generating+confusion+sets+for+context%2Dsensitive+error+correction%2E+In+Proceedings+of+EMNLP%2C+pages+961%E2%80%93970%2E)
31. E. Sandhaus. 2008. The new york times annotated corpus. Corpus number LDC2008T19, Linguistic Data Consortium, Philadelphia. [find](https://scholar.google.com/scholar?q=E%2E+Sandhaus%2E+2008%2E+The+new+york+times+annotated+corpus%2E+Corpus+number+LDC2008T19%2C+Linguistic+Data+Consortium%2C+Philadelphia%2E)
32. S. Schwarm and M. Ostendorf. 2005. [Reading level assessment using support vector machines and statistical language models](https://md.research.elamur.ai/free/en/papers/schwarm-2005-reading-level-assessment-support-vector/). In Proceedings of ACL, pages 523– 530.
33. H.A. Semetko and P.M. Valkenburg. 2000. Framing european politics: A content analysis of press and television news. Journal of communication, 50(2):93–109. [find](https://scholar.google.com/scholar?q=H%2EA%2E+Semetko+and+P%2EM%2E+Valkenburg%2E+2000%2E+Framing+european+politics%3A+A+content+analysis+of+press+and+television+news%2E+Journal+of+communication%2C+50%282%29%3A93%E2%80%93109%2E)
34. V. Spandel. 2004. Creating Writers Through 6-Trait Writing: Assessment and Instruction. [find](https://scholar.google.com/scholar?q=V%2E+Spandel%2E+2004%2E+Creating+Writers+Through+6%2DTrait+Writing%3A+Assessment+and+Instruction%2E)
35. Allyn and Bacon, Inc. S. H. Stocking. 2010. [find](https://scholar.google.com/scholar?q=Allyn+and+Bacon%2C+Inc%2E+S%2E+H%2E+Stocking%2E+2010%2E)
36. The New York Times Reader: Science and Technology. CQ Press, Washington DC. P. J. Stone, J. Kirsh, and Cambridge Computer Associates. 1966. The General Inquirer: A Computer Approach to Content Analysis. [find](https://scholar.google.com/scholar?q=The+New+York+Times+Reader%3A+Science+and+Technology%2E+CQ+Press%2C+Washington+DC%2E+P%2E+J%2E+Stone%2C+J%2E+Kirsh%2C+and+Cambridge+Computer+Associates%2E+1966%2E+The+General+Inquirer%3A+A+Computer+Approach+to+Content+Analysi)
37. MIT Press. J. R. Tetreault and M. Chodorow. 2008. [The ups and downs of preposition error detection in esl writing](https://md.research.elamur.ai/free/en/papers/tetreault-2008-ups-downs-preposition-error-detection/). In Proceedings of COLING, pages 865–872.
38. L. von Ahn and L. Dabbish. 2004. Labeling images with a computer game. In Proceedings of CHI, pages 319– 326. [find](https://scholar.google.com/scholar?q=L%2E+von+Ahn+and+L%2E+Dabbish%2E+2004%2E+Labeling+images+with+a+computer+game%2E+In+Proceedings+of+CHI%2C+pages+319%E2%80%93+326%2E)
39. R. L. Weide. 1998. The cmu pronunciation dictionary, release 0.6. [http://www.speech.cs.cmu.edu/cgi-bin/cmudict](http://www.speech.cs.cmu.edu/cgi-bin/cmudict). [↗](http://www.speech.cs.cmu.edu/cgi-bin/cmudict)
40. T. Wilson, J. Wiebe, and P. Hoffmann. 2005. [Recognizing contextual polarity in phrase-level sentiment analysis](https://md.research.elamur.ai/free/en/papers/wilson-2005-recognizing-contextual-polarity-phrase-level/). In Proceedings of HLT-EMNLP, pages 347–354.
41. M. Wilson. 1988. MRC psycholinguistic database: Machine-usable dictionary, version 2.00. Behavior Research Methods, 20(1):6–10. [find](https://scholar.google.com/scholar?q=M%2E+Wilson%2E+1988%2E+MRC+psycholinguistic+database%3A+Machine%2Dusable+dictionary%2C+version+2%2E00%2E+Behavior+Research+Methods%2C+20%281%29%3A6%E2%80%9310%2E)

*Updated 30 October 2026*