---
title: "Annotating Shakespeare's Sonnets with Appraisal Theory to Detect Irony (2019)"
description: "The paper asks whether Appraisal Theory (Martin and White, 2005), a linguistic framework for… 65% agreement. 16 references, cited by 1 paper here."
authors: ["Nicolò Busetto", "Rodolfo Delmonte"]
year: 2019
venue: "Proceedings of the Sixth Italian Conference on Computational Linguistics (CLiC-it 2019)"
source: "https://aclanthology.org/2019.clicit-1.11/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/busetto-2019-annotating-shakespeare-s-sonnets-appraisal/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/busetto-2019-annotating-shakespeare-s-sonnets-appraisal/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/7/) / Annotating Shakespeare's Sonnets with Appraisal Theory to Detect Irony

# Annotating Shakespeare's Sonnets with Appraisal Theory to Detect Irony

**Authors:** [Nicolò Busetto](https://md.research.elamur.ai/free/en/authors/nicolo-busetto/) and [Rodolfo Delmonte](https://md.research.elamur.ai/free/en/authors/rodolfo-delmonte/) · **Year:** 2019 · **Venue:** [Proceedings of the Sixth Italian Conference on Computational Linguistics (CLiC-it 2019)](https://md.research.elamur.ai/free/en/venues/clicit/) ([Italian Conference on Computational Linguistics 2019](https://md.research.elamur.ai/free/en/venues/clicit/2019/)) · **Publisher:** [CEUR Workshop Proceedings](https://md.research.elamur.ai/free/en/publishers/ceur-workshop-proceedings/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2019.clicit-1.11/) · [PDF](https://aclanthology.org/2019.clicit-1.11.pdf) · [OpenAlex](https://openalex.org/W2989976963)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper asks whether Appraisal Theory (Martin and White, 2005), a linguistic framework for describing evaluation in text, can separate ironic, sarcastic and other pragmatically marked poems among Shakespeare's 154 Sonnets. The authors first built a gold standard from published literary criticism (Frye, Melchiori, Weiser, Serpieri and others). Each sonnet received structured fields: thematic sequence, main theme, proposed action, central metaphor, negative evaluation, positive evaluation, and the key contrast. Pragmatic labels came from these sources: irony, satire, parody, sarcasm, or blank. Next, every sonnet was hand-annotated in XML with only the Attitude branch of Appraisal: Affect, Judgement and Appreciation, each marked positive or negative. The counts of these six classes were then compared with the critics' labels. Each pragmatic category showed its own profile. Sarcasm is dominated by negative Judgement and overall negativity. Irony leans positive, with Appreciation prominent. Unlabelled sonnets are dominated by Affect. The reported match with the critics' views is 80%. This suggests that fine-grained evaluation categories, and not just plain sentiment, can give an interpretable signal for figurative stance.

## Key points
- Of the 154 sonnets, critics give some pragmatic label to 98 (63.64%), and 54 of these carry a double label. Table 1 counts: Blank 57, Irony 73, Satire 20, Parody 4, Sarcasm 47.
- The annotation has 1186 Attitude spans in total, 607 positive and 579 negative. They are marked with `<apprsl>` tags inside `<text>`/`<p>`/`<s>` (sonnet, stanza, line).
- Agreement was never formally measured. One annotator corrected about 35% of the other's labels, which the authors read as roughly 65% agreement.
- The 44 highest-contrast sonnets, read as sarcastic, have negative Judgement as the largest class (sum 122) and more negative than positive evaluations (224 vs 141). The roughly 50 ironic sonnets have more positive than negative ones (262 vs 183), led by positive Appreciation.
- Judgement tends to cluster in the closing couplet. The overall match between the Appraisal-based classification and the critics is reported as 80%.

## Interesting ideas
- Each pragmatic category (sarcasm, irony, neutral) gets a distinct "fingerprint": which Attitude subtype dominates, plus the balance of positive and negative polarity. This makes the classification explainable rather than a black-box score.
- Sarcasm is operationalised as heavy negative moral judgement co-occurring with positive aesthetic or emotional evaluation. This is a concrete, checkable form of the incongruity idea.
- The gold standard records structured expert fields (theme, action, metaphor, negative/positive evaluation, contrast), not just a single label, so disagreements can be traced to a specific field.
- Evaluation is located positionally: Judgement concentrates in the final couplet, so where an evaluation appears carries meaning.
- Span rules handle lists: two or three coordinated items with the same type and polarity become one unit, while longer runs are split, and only the most evaluative part of a long phrase is marked.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (frames): the irony module in prag could compute per-text counts of affect, judgement and appreciation frames with polarity, then justify an irony or sarcasm verdict by pointing to the dominant subtype and the contrast.
- **I2** (frames): when one entity in a frame graph receives both a negative Judgement and a positive Appreciation or Affect, prag could flag it as an incongruity candidate and pass it to implicature reasoning.
- **I3** (answer): QA on fables and tales could use a similar per-text card (moral, action, contrast) as structured gold, making it possible to test which part of a reader's interpretation went wrong.
- **I4** (world): the world model could tag evaluations with discourse position, letting a late judgement that reverses earlier praise be treated as the text's final stance.
- **I5** (frames): similar rules could keep evaluative frames compact and consistent when coordination in the UD tree produces many conjuncts carrying the same attitude.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [📚 Literature and Literary Theory](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/literature-and-literary-theory/) · 340 works

🧭 Topics (1): [😊 Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) · 608 works

## Cite

Cite as: Nicolò Busetto and Rodolfo Delmonte. 2019. Annotating Shakespeare's Sonnets with Appraisal Theory to Detect Irony. Proceedings of the Sixth Italian Conference on Computational Linguistics (CLiC-it 2019).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{busetto2019annotating,
  title = {{Annotating Shakespeare's Sonnets with Appraisal Theory to Detect Irony}},
  author = {Nicolò Busetto and Rodolfo Delmonte},
  booktitle = {Proceedings of the Sixth Italian Conference on Computational Linguistics (CLiC-it 2019)},
  year = {2019},
  url = {https://aclanthology.org/2019.clicit-1.11/}
}
```

- [← previous Rater attitude towards emerging varieties of English: a new rater effect?](https://md.research.elamur.ai/free/en/papers/hsu-2019-rater-attitude-towards-emerging-varieties/)
- [next → Crowdsourced Hedge Term Disambiguation](https://md.research.elamur.ai/free/en/papers/ulinski-2019-crowdsourced-hedge-term-disambiguation/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/7/)

## References

16 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Salvatore Attardo. 1994. Linguistic Theories of Humor. Mouton de Gruyter, Berlin – New York. [find](https://scholar.google.com/scholar?q=Salvatore+Attardo%2E+1994%2E+Linguistic+Theories+of+Humor%2E+Mouton+de+Gruyter%2C+Berlin+%E2%80%93+New+York%2E)
2. Salvatore Attardo. 2000. Irony as relevant inappropriateness. Journal of Pragmatics, 84(32). [↗](https://doi.org/10.1016/s0378-2166(99)00070-3)
3. Dario Calimani. 2009. William Shakespeare, I sonetti della menzogna. Carrocci, Roma. [find](https://scholar.google.com/scholar?q=Dario+Calimani%2E+2009%2E+William+Shakespeare%2C+I+sonetti+della+menzogna%2E+Carrocci%2C+Roma%2E)
4. Rodolfo Delmonte and Giulia Marchesini. 2017. A semantically-based approach to the annotation of narrative style. In Proceedings of the 13th Joint ISO-ACL Workshop on Interoperable Semantic Annotation (ISA-13), pages 14–25, Stroudsburg, PA, USA. ACL. [find](https://scholar.google.com/scholar?q=Rodolfo+Delmonte+and+Giulia+Marchesini%2E+2017%2E+A+semantically%2Dbased+approach+to+the+annotation+of+narrative+style%2E+In+Proceedings+of+the+13th+Joint+ISO%2DACL+Workshop+on+Interoperable+Semantic+Annotation)
5. R.L. Eagle. 1916. New light on the enigmas of Shakespeare’s Sonnets. John Long Limited, London. [find](https://scholar.google.com/scholar?q=R%2EL%2E+Eagle%2E+1916%2E+New+light+on+the+enigmas+of+Shakespeare%E2%80%99s+Sonnets%2E+John+Long+Limited%2C+London%2E)
6. Northrop Frye. 1957. Anatomy of Criticism: Four Essays. Princeton University Press. [find](https://scholar.google.com/scholar?q=Northrop+Frye%2E+1957%2E+Anatomy+of+Criticism%3A+Four+Essays%2E+Princeton+University+Press%2E)
7. Maria Antonietta Marelli. 2015. William Shakespeare, I Sonetti – con testo a fronte. Garzanti. [find](https://scholar.google.com/scholar?q=Maria+Antonietta+Marelli%2E+2015%2E+William+Shakespeare%2C+I+Sonetti+%E2%80%93+con+testo+a+fronte%2E+Garzanti%2E)
8. J. Martin and P.R. White. 2005. Language of Evaluation, Appraisal in English. Palgrave Macmillan, London and New York. [find](https://scholar.google.com/scholar?q=J%2E+Martin+and+P%2ER%2E+White%2E+2005%2E+Language+of+Evaluation%2C+Appraisal+in+English%2E+Palgrave+Macmillan%2C+London+and+New+York%2E)
9. Giorgio Melchiori. 1971. Adriatica Editrice, Bari. Shakespeare s Sonnets. [find](https://scholar.google.com/scholar?q=Giorgio+Melchiori%2E+1971%2E+Adriatica+Editrice%2C+Bari%2E+Shakespeare+s+Sonnets%2E)
10. J. Read and J. Carrol. 2012. Annotating expressions of appraisal in english. Language Resources and Evaluation, 46:421–447. [find](https://scholar.google.com/scholar?q=J%2E+Read+and+J%2E+Carrol%2E+2012%2E+Annotating+expressions+of+appraisal+in+english%2E+Language+Resources+and+Evaluation%2C+46%3A421%E2%80%93447%2E)
11. Michael Schoenfeldt. 2010. Cambridge introduction to Shakespeare’s poetry. Cambridge University Press, Cambridge. [find](https://scholar.google.com/scholar?q=Michael+Schoenfeldt%2E+2010%2E+Cambridge+introduction+to+Shakespeare%E2%80%99s+poetry%2E+Cambridge+University+Press%2C+Cambridge%2E)
12. Alessandro Serpieri. 2002. Polifonia Shakespeariana. Bulzoni, Roma. [find](https://scholar.google.com/scholar?q=Alessandro+Serpieri%2E+2002%2E+Polifonia+Shakespeariana%2E+Bulzoni%2C+Roma%2E)
13. Michele Stingo and Rodolfo Delmonte. 2016. Annotating satire in italian political commentaries with appraisal theory. In Natural Language Processing meets Journalism - Proceedings of the Workshop, NLPMJ-2016, pages 74–79, Stroudsburg, PA, USA. ACL. [find](https://scholar.google.com/scholar?q=Michele+Stingo+and+Rodolfo+Delmonte%2E+2016%2E+Annotating+satire+in+italian+political+commentaries+with+appraisal+theory%2E+In+Natural+Language+Processing+meets+Journalism+%2D+Proceedings+of+the+Workshop%2C+NLP)
14. M. Taboada and J. Grieve. 2004. Analyzing appraisal automatically. In Proceedings of the AAAI Spring Symposium on Exploring Attitude and Affect in Text: Theories and Applications, pages 158–161. AAAI Press. [find](https://scholar.google.com/scholar?q=M%2E+Taboada+and+J%2E+Grieve%2E+2004%2E+Analyzing+appraisal+automatically%2E+In+Proceedings+of+the+AAAI+Spring+Symposium+on+Exploring+Attitude+and+Affect+in+Text%3A+Theories+and+Applications%2C+pages+158%E2%80%93161%2E+AAAI+)
15. David K. Weiser. 1983. [http://www.jstor.org/stable/43343552](http://www.jstor.org/stable/43343552) Shakespearean irony: The ’sonnets’. Neuphilologische Mitteilungen, 84(4):456–469. [↗](http://www.jstor.org/stable/43343552)
16. David K. Weiser. 1987. Mind in Character – Shakespeare s Speaker in the Sonnets. The University of Missouri Press. [find](https://scholar.google.com/scholar?q=David+K%2E+Weiser%2E+1987%2E+Mind+in+Character+%E2%80%93+Shakespeare+s+Speaker+in+the+Sonnets%2E+The+University+of+Missouri+Press%2E)

*Updated 24 October 2026*