---
title: "CLAUDETTE — paper by Lippi et al. (2018) — summary and ideas"
description: "Online Terms of Service (ToS) often contain clauses that EU consumer law (Directive… 80% precision. 27 references, cited by 77 papers here."
authors: ["Marco Lippi", "Przemyslaw Palka", "Giuseppe Contissa", "Francesca Lagioia", "Hans-Wolfgang Micklitz", "Giovanni Sartor", "Paolo Torroni"]
year: 2018
venue: "arXiv:1805.01217 · Artif. Intell. Law 27 (2019) 117-139"
source: "https://arxiv.org/abs/1805.01217"
doi: "10.1007/s10506-019-09243-2"
arxiv: "1805.01217"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lippi-2018-claudette-automated-detector-potentially-unfair/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lippi-2018-claudette-automated-detector-potentially-unfair/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/3/) / CLAUDETTE: an Automated Detector of Potentially Unfair Clauses in Online Terms of Service

# CLAUDETTE: an Automated Detector of Potentially Unfair Clauses in Online Terms of Service

**Authors:** Marco Lippi, Przemyslaw Palka, [Giuseppe Contissa](https://md.research.elamur.ai/free/en/authors/giuseppe-contissa/), Francesca Lagioia, Hans-Wolfgang Micklitz, Giovanni Sartor and [Paolo Torroni](https://md.research.elamur.ai/free/en/authors/paolo-torroni/) · **Year:** 2018 · **Venue:** arXiv:1805.01217 · Artif. Intell. Law 27 (2019) 117-139 · **Publisher:** [Springer Science+Business Media](https://md.research.elamur.ai/free/en/publishers/springer-science-business-media/) · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://doi.org/10.1007/s10506-019-09243-2) · [PDF](https://arxiv.org/pdf/1805.01217) · [DOI 10.1007/s10506-019-09243-2](https://doi.org/10.1007/s10506-019-09243-2) · [arXiv 1805.01217](https://arxiv.org/abs/1805.01217) · [OpenAlex](https://openalex.org/W2798838283)

**Citations:** 169 citations (OpenAlex, 2026-10-14) · [77](https://md.research.elamur.ai/free/en/papers/lippi-2018-claudette-automated-detector-potentially-unfair/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Online Terms of Service (ToS) often contain clauses that EU consumer law (Directive 93/13) could treat as unfair. Consumers rarely read these contracts, and consumer protection bodies lack the resources to review them manually. The authors automate one step of that review, locating potentially unfair clauses, so that lawyers can spend their time on the legal assessment itself.

They built a corpus of 50 ToS from major online platforms, totalling more than 12,000 sentences. Each document was annotated in XML with eight clause categories and a three-level fairness grade. They compared several learners: bag-of-words SVMs, recent deep learning text classifiers, and a structured SVM that models the order of sentences in a document. A second stage then assigns each detected clause to a category.

An ensemble performed best at detection, recovering over 80% of potentially unfair clauses at roughly 80% precision. Categorisation turned out to be easier, with F1 above 74% for every tag. Simple lexical features held up well against more complex models. The system is available as the CLAUDETTE web server, a practical aid for consumers and enforcement agencies.

## Key points
- **Corpus:** 50 English ToS from large platforms (e.g. Dropbox, Facebook, Google, Netflix, Spotify), using the latest EU-facing version of each. This scales up an earlier 20-document study by the same group.
- **Eight categories:** the clause types covered are unilateral change, unilateral termination, limitation of liability, choice of law, jurisdiction, contract by using, content removal, and arbitration.
  - The first five come from Loos and Luzak; the last three were added by the authors.
- **Fairness grades:** each tag carries a value, where 1 = clearly fair, 2 = potentially unfair and 3 = clearly unfair. Nested tags mark text relevant to several categories, and multi-paragraph clauses are tagged paragraph by paragraph.
- **Detection results:** the best result came from an ensemble, at about 80% recall and 80% precision. The bag-of-words approach performed comparatively strongly.
- **Categorisation results:** F1 exceeded 0.93 for jurisdiction, choice of law, limitation of liability and contract by using. Content removal was the weakest category, at 0.745.
- **Deployment:** the CLAUDETTE server runs only the single classifier C2 rather than the ensemble, for efficiency. It outputs HTML, XML, JSON or plain text, with detected sentences highlighted and labelled by type.

## Interesting ideas
- **Graded labels instead of a binary flag.** Annotating fairness on a three-step scale preserves legal uncertainty in the data. This is useful in any domain where experts themselves disagree or where the final verdict depends on context.
- **Two-stage pipeline.** A detector first finds suspicious sentences, and a separate classifier, trained only on positive examples, assigns their type. Splitting a hard, imbalanced problem this way lets each stage be tuned and evaluated on its own.
- **Framing the tool as triage, not judgement.** The authors map the lawyers' workflow and automate only the document-mining step. This sets realistic goals and makes moderate precision and recall acceptable when the alternative is reading entire documents.
- **Accuracy versus cost in deployment.** The authors use the ensemble to report the best scores but ship a cheaper single model in the public service. This pragmatic trade-off is worth copying whenever research prototypes become user-facing tools.
- **Checking simple baselines first.** Bag-of-words competed well with deep and structured models on a small specialised corpus. This is a reminder to benchmark lexical baselines before investing in heavier architectures for low-resource or niche-domain classification.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Yale University](https://md.research.elamur.ai/free/en/universities/yale-university/) — University, New Haven, United States · 101 works · 13,302 citations
- [University of Bologna](https://md.research.elamur.ai/free/en/universities/university-of-bologna/) — University, Bologna, Italy · 19 works · 2,067 citations
- [University of Modena and Reggio Emilia](https://md.research.elamur.ai/free/en/universities/university-of-modena-and-reggio-emilia/) — University, Modena, Italy · 4 works · 249 citations

Also on the byline: [European University Institute](https://ror.org/0031wrj91).

📚 Shelves (6): [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,954 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 626 works · [🤝 General Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/general-social-sciences/) · 254 works · [🤝 Political Science and International Relations](https://md.research.elamur.ai/free/en/shelves/social-sciences/political-science-and-international-relations/) · 206 works

🧭 Topics (1): [⚖️ Legal AI](https://md.research.elamur.ai/free/en/topics/legal-ai/) · 150 works

## Cite

Cite as: Marco Lippi, Przemyslaw Palka, Giuseppe Contissa, Francesca Lagioia, Hans-Wolfgang Micklitz, Giovanni Sartor and Paolo Torroni. 2018. CLAUDETTE: an Automated Detector of Potentially Unfair Clauses in Online Terms of Service. arXiv:1805.01217 · Artif. Intell. Law 27 (2019) 117-139.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{lippi2018claudette,
  title = {{CLAUDETTE: an Automated Detector of Potentially Unfair Clauses in Online Terms of Service}},
  author = {Marco Lippi, Przemyslaw Palka, Giuseppe Contissa, Francesca Lagioia, Hans-Wolfgang Micklitz, Giovanni Sartor and Paolo Torroni},
  year = {2018},
  doi = {10.1007/s10506-019-09243-2},
  eprint = {1805.01217},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1007/s10506-019-09243-2}
}
```

- [← previous Reasoning with Sarcasm by Reading In-Between](https://md.research.elamur.ai/free/en/papers/tay-2018-reasoning-sarcasm-reading-between/)
- [next → Few-Shot and Zero-Shot Multi-Label Learning for Structured Label Spaces](https://md.research.elamur.ai/free/en/papers/rios-2018-few-shot-zero-shot-multi/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/3/)

## References

27 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Aletras N, Tsarapatsanis D, Preoiuc-Pietro D, Lampos V (2016) [Predicting judicial decisions of the European Court of Human Rights: a natural language processing perspective](https://md.research.elamur.ai/free/en/papers/aletras-2016-predicting-judicial-decisions-european/). PeerJ Computer Science 2:e93
2. Ashley K (2017) Artificial Intelligence and Legal Analytics: New Tools for Law Practice in the Digital Age. [find](https://scholar.google.com/scholar?q=Ashley+K+%282017%29+Artificial+Intelligence+and+Legal+Analytics%3A+New+Tools+for+Law+Practice+in+the+Digital+Age%2E)
3. Cambridge University Press Ashley KD, Walker VR (2013) Toward constructing evidence-based legal arguments using legal decision documents and machine learning. In: Proceedings of the Fourteenth International Conference on Artificial Intelligence and Law, ACM, pp 176–180 [find](https://scholar.google.com/scholar?q=Cambridge+University+Press+Ashley+KD%2C+Walker+VR+%282013%29+Toward+constructing+evidence%2Dbased+legal+arguments+using+legal+decision+documents+and+machine+learning%2E+In%3A+Proceedings+of+the+Fourteenth+Interna)
4. Bakos Y, Marotta-Wurgler F, Trossen DR (2014) Does anyone read the fine print? consumer attention to standard-form contracts. The Journal of Legal Studies 43(1):1–35 [find](https://scholar.google.com/scholar?q=Bakos+Y%2C+Marotta%2DWurgler+F%2C+Trossen+DR+%282014%29+Does+anyone+read+the+fine+print%3F+consumer+attention+to+standard%2Dform+contracts%2E+The+Journal+of+Legal+Studies+43%281%29%3A1%E2%80%9335)
5. Biagioli C, Francesconi E, Passerini A, Montemagni S, Soria C (2005) Automatic semantics extraction in law documents. In: Proceedings of ICAIL, ACM, pp 133–140 [find](https://scholar.google.com/scholar?q=Biagioli+C%2C+Francesconi+E%2C+Passerini+A%2C+Montemagni+S%2C+Soria+C+%282005%29+Automatic+semantics+extraction+in+law+documents%2E+In%3A+Proceedings+of+ICAIL%2C+ACM%2C+pp+133%E2%80%93140)
6. Collins M, Duffy N (2002) New ranking algorithms for parsing and tagging: Kernels over discrete structures, and the voted perceptron. In: Proceedings of the 40th Annual Meeting of the ACL, ACL, pp 263–270 [find](https://scholar.google.com/scholar?q=Collins+M%2C+Duffy+N+%282002%29+New+ranking+algorithms+for+parsing+and+tagging%3A+Kernels+over+discrete+structures%2C+and+the+voted+perceptron%2E+In%3A+Proceedings+of+the+40th+Annual+Meeting+of+the+ACL%2C+ACL%2C+pp+263)
7. Department of Commerce (2010) Commercial data privacy and innovation in the Internet economy: A dynamic policy framework. Tech. rep., Department of Commerce Internet Policy Task Force, URL [https://www.ntia.doc.gov/files/ntia/publications/iptf_privacy_greenpaper_12162010.pdf](https://www.ntia.doc.gov/files/ntia/publications/iptf_privacy_greenpaper_12162010.pdf) [↗](https://www.ntia.doc.gov/files/ntia/publications/iptf_privacy_greenpaper_12162010.pdf)
8. Fabian B, Ermakova T, Lentz T (2017) Large-scale readability analysis of privacy policies. In: Proceedings of the International Conference on Web Intelligence, ACM, pp 18–25 [find](https://scholar.google.com/scholar?q=Fabian+B%2C+Ermakova+T%2C+Lentz+T+%282017%29+Large%2Dscale+readability+analysis+of+privacy+policies%2E+In%3A+Proceedings+of+the+International+Conference+on+Web+Intelligence%2C+ACM%2C+pp+18%E2%80%9325)
9. Graves A, Schmidhuber J (2005) Framewise phoneme classification with bidirectional lstm and other neural network architectures. Neural Networks 18(5):602–610 [find](https://scholar.google.com/scholar?q=Graves+A%2C+Schmidhuber+J+%282005%29+Framewise+phoneme+classification+with+bidirectional+lstm+and+other+neural+network+architectures%2E+Neural+Networks+18%285%29%3A602%E2%80%93610)
10. Harkous H, Fawaz K, Lebret R, Schaub F, Shin KG, Aberer K (2018) Polisis: Automated analysis and presentation of privacy policies using deep learning. arXiv preprint arXiv:180202561 [find](https://scholar.google.com/scholar?q=Harkous+H%2C+Fawaz+K%2C+Lebret+R%2C+Schaub+F%2C+Shin+KG%2C+Aberer+K+%282018%29+Polisis%3A+Automated+analysis+and+presentation+of+privacy+policies+using+deep+learning%2E+arXiv+preprint+arXiv%3A180202561)
11. Joachims T (1998) [Text categorization with support vector machines: Learning with many relevant features](https://md.research.elamur.ai/free/en/papers/joachims-1998-text-categorization-support-vector-machines/). ECML 98 pp 137–142
12. Kim Y (2014) [Convolutional neural networks for sentence classification](https://md.research.elamur.ai/free/en/papers/kim-2014-convolutional-neural-networks-sentence-classification/). In: Moschitti A, Pang B, Daelemans W (eds) Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, Doha, Qatar, ACL, pp 1746–1751
13. Leopold E, Kindermann J (2002) Text categorization with support vector machines. How to represent texts in input space? Machine Learning 46(1-3):423–444 [find](https://scholar.google.com/scholar?q=Leopold+E%2C+Kindermann+J+%282002%29+Text+categorization+with+support+vector+machines%2E+How+to+represent+texts+in+input+space%3F+Machine+Learning+46%281%2D3%29%3A423%E2%80%93444)
14. Lippi M, Torroni P (2016a) Argumentation mining: State of the art and emerging trends. ACM Trans Internet Technol 16(2):10:1–10:25 [find](https://scholar.google.com/scholar?q=Lippi+M%2C+Torroni+P+%282016a%29+Argumentation+mining%3A+State+of+the+art+and+emerging+trends%2E+ACM+Trans+Internet+Technol+16%282%29%3A10%3A1%E2%80%9310%3A25)
15. Lippi M, Torroni P (2016b) Margot: A web server for argumentation mining. Expert Systems with Applications 65(C):292–303, DOI 10.1016/j.eswa.2016.08.050 [↗](https://doi.org/10.1016/j.eswa.2016.08.050)
16. Lippi M, Palka P, Contissa G, Lagioia F, Micklitz H, Panagis Y, Sartor G, Torroni P (2017) Automated detection of unfair clauses in online consumer contracts. In: Wyner AZ, Casini G (eds) Legal Knowledge and Information Systems - JURIX 2017: The Thirtieth Annual Conference, Luxembourg, 13-15 December 2017, IOS Press, Frontiers in Artificial Intelligence and Applications, vol 302, pp 145–154 [find](https://scholar.google.com/scholar?q=Lippi+M%2C+Palka+P%2C+Contissa+G%2C+Lagioia+F%2C+Micklitz+H%2C+Panagis+Y%2C+Sartor+G%2C+Torroni+P+%282017%29+Automated+detection+of+unfair+clauses+in+online+consumer+contracts%2E+In%3A+Wyner+AZ%2C+Casini+G+%28eds%29+Legal+Knowle)
17. Lippi M, Lagioia F, Contissa G, Sartor G, Torroni P (2018) Claim detection in judgments of the EU Court of Justice. In: Artificial Intelligence and the Complexity of Legal Systems, VI International Workshop (AICOL), selected revised papers. Lecture Notes in Artificial Intelligence, Springer, forthcoming Loos M, Luzak J (2016) Wanted: a bigger stick. On unfair terms in consumer contracts with online service providers. Journal of Consumer Policy 39(1):63–90 [find](https://scholar.google.com/scholar?q=Lippi+M%2C+Lagioia+F%2C+Contissa+G%2C+Sartor+G%2C+Torroni+P+%282018%29+Claim+detection+in+judgments+of+the+EU+Court+of+Justice%2E+In%3A+Artificial+Intelligence+and+the+Complexity+of+Legal+Systems%2C+VI+International+Wo)
18. McDonald A, Cranor L (2008) The cost of reading privacy policies. I/S: A Journal of Law and Policy for the Information Society 4(3), URL [http://moritzlaw.osu.edu/students/groups/is/files/2012/02/Cranor_Formatted_Final.pdf](http://moritzlaw.osu.edu/students/groups/is/files/2012/02/Cranor_Formatted_Final.pdf), issue: 2008 Privacy Year in Review Micklitz HW, Reich N (2014) The court and sleeping beauty: The revival of the unfair contract terms directive (uctd). Common Market Law Review 51(3):771–808 [↗](http://moritzlaw.osu.edu/students/groups/is/files/2012/02/Cranor_Formatted_Final.pdf)
19. Micklitz HW, Pa lka P, Panagis Y (2017) The empire strikes back: Digital control of unfair terms of online services. Journal of Consumer Policy pp 1–22 [find](https://scholar.google.com/scholar?q=Micklitz+HW%2C+Pa+lka+P%2C+Panagis+Y+%282017%29+The+empire+strikes+back%3A+Digital+control+of+unfair+terms+of+online+services%2E+Journal+of+Consumer+Policy+pp+1%E2%80%9322)
20. Mikolov T, Chen K, Corrado G, Dean J (2013) [Efficient estimation of word representations in vector space](https://md.research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/). arXiv preprint arXiv:13013781
21. Moens MF, Boiy E, Palau RM, Reed C (2007) Automatic detection of arguments in legal texts. In: Proceedings of the 11th international conference on Artificial intelligence and law, ACM, pp 225–230 [find](https://scholar.google.com/scholar?q=Moens+MF%2C+Boiy+E%2C+Palau+RM%2C+Reed+C+%282007%29+Automatic+detection+of+arguments+in+legal+texts%2E+In%3A+Proceedings+of+the+11th+international+conference+on+Artificial+intelligence+and+law%2C+ACM%2C+pp+225%E2%80%93230)
22. Moschitti A (2006) Efficient convolution kernels for dependency and constituent syntactic trees. In: Frnkranz J, Scheffer T, Spiliopoulou M (eds) ECML 2006, Springer Berlin Heidelberg Nebbia P (2007) Unfair contract terms in European law: A study in comparative and EC law. [find](https://scholar.google.com/scholar?q=Moschitti+A+%282006%29+Efficient+convolution+kernels+for+dependency+and+constituent+syntactic+trees%2E+In%3A+Frnkranz+J%2C+Scheffer+T%2C+Spiliopoulou+M+%28eds%29+ECML+2006%2C+Springer+Berlin+Heidelberg+Nebbia+P+%282007%29+)
23. Bloomsbury Publishing Obar JA, Oeldorf-Hirsch A (2016) The biggest lie on the internet: Ignoring the privacy policies and terms of service policies of social networking services. In: TPRC 44: The 44th Research Conference on Communication, Information and Internet Policy Reich N, Micklitz HW, Rott P, Tonner K (2014) European consumer law. [find](https://scholar.google.com/scholar?q=Bloomsbury+Publishing+Obar+JA%2C+Oeldorf%2DHirsch+A+%282016%29+The+biggest+lie+on+the+internet%3A+Ignoring+the+privacy+policies+and+terms+of+service+policies+of+social+networking+services%2E+In%3A+TPRC+44%3A+The+44th)
24. Intersentia Schulte-Nölke H, Twigg-Flesner C, Ebers M (2008) EC consumer law compendium: The consumer acquis and its transposition in the member states. [find](https://scholar.google.com/scholar?q=Intersentia+Schulte%2DNo%CC%88lke+H%2C+Twigg%2DFlesner+C%2C+Ebers+M+%282008%29+EC+consumer+law+compendium%3A+The+consumer+acquis+and+its+transposition+in+the+member+states%2E)
25. Walter de Gruyter Sebastiani F (2002) [Machine learning in automated text categorization](https://md.research.elamur.ai/free/en/papers/sebastiani-2002-machine-learning-automated-text-categorization/). ACM Comput Surv 34(1):1–47
26. Shulayeva O, Siddharthan A, Wyner A (2017) Recognizing cited facts and principles in legal judgements. Artificial Intelligence and Law 25(1):107–126 [find](https://scholar.google.com/scholar?q=Shulayeva+O%2C+Siddharthan+A%2C+Wyner+A+%282017%29+Recognizing+cited+facts+and+principles+in+legal+judgements%2E+Artificial+Intelligence+and+Law+25%281%29%3A107%E2%80%93126)
27. Tsochantaridis I, Joachims T, Hofmann T, Altun Y (2005) Large margin methods for structured and interdependent output variables. Journal of Machine Learning Research 6:1453–1484 [find](https://scholar.google.com/scholar?q=Tsochantaridis+I%2C+Joachims+T%2C+Hofmann+T%2C+Altun+Y+%282005%29+Large+margin+methods+for+structured+and+interdependent+output+variables%2E+Journal+of+Machine+Learning+Research+6%3A1453%E2%80%931484)

*Updated 2 November 2026*