---
title: "FactSheets — paper by Arnold et al. (2018) — summary and ideas"
description: "The paper looks at how to build trust in AI services. 18 references, cited by 12 papers here."
authors: ["Matthew Arnold", "Rachel K. E. Bellamy", "Michael Hind", "Stephanie Houde", "Sameep Mehta", "Aleksandra Mojsilovic", "Ravi Nair", "Karthikeyan Natesan Ramamurthy", "Darrell Reimer", "Alexandra Olteanu", "David Piorkowski", "Jason Tsay", "Kush R. Varshney"]
year: 2018
venue: "arXiv:1808.07261"
source: "https://arxiv.org/abs/1808.07261"
doi: "10.48550/arxiv.1808.07261"
arxiv: "1808.07261"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/arnold-2018-factsheets-increasing-trust-ai-services/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/arnold-2018-factsheets-increasing-trust-ai-services/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/2/) / FactSheets: Increasing Trust in AI Services through Supplier's Declarations of Conformity

# FactSheets: Increasing Trust in AI Services through Supplier's Declarations of Conformity

**Authors:** [Matthew Arnold](https://md.research.elamur.ai/free/en/authors/matthew-arnold/), Rachel K. E. Bellamy, [Michael Hind](https://md.research.elamur.ai/free/en/authors/michael-hind/), Stephanie Houde, Sameep Mehta, Aleksandra Mojsilovic, Ravi Nair, [Karthikeyan Natesan Ramamurthy](https://md.research.elamur.ai/free/en/authors/karthikeyan-natesan-ramamurthy/), Darrell Reimer, [Alexandra Olteanu](https://md.research.elamur.ai/free/en/authors/alexandra-olteanu/), [David Piorkowski](https://md.research.elamur.ai/free/en/authors/david-piorkowski/), Jason Tsay and Kush R. Varshney · **Year:** 2018 · **Venue:** arXiv:1808.07261 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1808.07261) · [PDF](https://arxiv.org/pdf/1808.07261) · [DOI 10.48550/arxiv.1808.07261](https://doi.org/10.48550/arxiv.1808.07261) · [arXiv 1808.07261](https://arxiv.org/abs/1808.07261) · [OpenAlex](https://openalex.org/W2911599914)

**Citations:** 425 citations (OpenAlex, 2026-10-14) · [12](https://md.research.elamur.ai/free/en/papers/arnold-2018-factsheets-increasing-trust-ai-services/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary

The paper looks at how to build trust in AI services. These are cloud-based functions accessed through an API, and they often combine several models and datasets. The authors, from IBM Research, argue that where errors are costly (lending, the courts, medicine), accuracy alone is not enough. Consumers need to know about a service's safety, fairness, explainability, security and provenance, but there is still no generally accepted way to communicate this. The authors model their solution on suppliers' declarations of conformity (SDoC), which manufacturers in telecommunications, transportation and other industries draw up voluntarily. They propose a similar document for AI, called a FactSheet. It describes the service's purpose, its performance, its robustness to attacks and to dataset shift, how it was trained and tested, and its ethical risks. An appendix gives a full list of items and two sample FactSheets for fictional services. FactSheets are meant to be adopted voluntarily at first, with a possible later move to independent certification for the highest-risk domains. The paper laid the groundwork for practices of documenting AI for the sake of transparency.

## Key points

- Unlike datasheets for datasets (Gebru et al.), data statements (Bender and Friedman), the dataset nutrition label (Holland et al.) and model cards (Mitchell et al.), a FactSheet describes the whole finished service rather than a single model or dataset.
- The authors give three arguments for this approach. Developers work with the service itself. There is an expertise gap between its creators (data scientists) and its users (developers). Moreover, a system built from reliable components can still be unreliable.
- A FactSheet is intended as an immutable artifact: new test results are only appended to it. Any change to the service means a new version of the document, and the recorded metrics help monitor the service after deployment.
- The appendix contains two example FactSheets for fictional services: fingerprint verification and detection of trending topics on social media.
- The expected evolution runs from voluntary declarations to a de facto standard along the lines of Energy Star, and to independent certification for the highest-risk applications. The authors compare this to the way digital certificates once spurred the growth of the internet.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 2,009 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 665 works · [⚙️ Engineering](https://md.research.elamur.ai/free/en/shelves/engineering/) · 206 works · [⚙️ Safety, Risk, Reliability and Quality](https://md.research.elamur.ai/free/en/shelves/engineering/safety-risk-reliability-and-quality/) · 18 works

## Cite

Cite as: Matthew Arnold, Rachel K. E. Bellamy, Michael Hind, Stephanie Houde, Sameep Mehta, Aleksandra Mojsilovic, Ravi Nair, Karthikeyan Natesan Ramamurthy, Darrell Reimer, Alexandra Olteanu, David Piorkowski, Jason Tsay and Kush R. Varshney. 2018. FactSheets: Increasing Trust in AI Services through Supplier's Declarations of Conformity. arXiv:1808.07261.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{arnold2018factsheets,
  title = {{FactSheets: Increasing Trust in AI Services through Supplier's Declarations of Conformity}},
  author = {Matthew Arnold, Rachel K. E. Bellamy, Michael Hind, Stephanie Houde, Sameep Mehta, Aleksandra Mojsilovic, Ravi Nair, Karthikeyan Natesan Ramamurthy, Darrell Reimer, Alexandra Olteanu, David Piorkowski, Jason Tsay and Kush R. Varshney},
  year = {2018},
  doi = {10.48550/arxiv.1808.07261},
  eprint = {1808.07261},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1808.07261}
}
```

- [← previous Fast Lexically Constrained Decoding with Dynamic Beam Allocation for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/post-2018-fast-lexically-constrained-decoding-dynamic/)
- [next → Unsupervised Statistical Machine Translation](https://md.research.elamur.ai/free/en/papers/artetxe-2018-unsupervised-statistical-machine-translation/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/2/)

## References

18 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. E. E. Levine, T. B. Bitterly, T. R. Cohen, and M. E. Schweitzer, “Who is trustworthy? Predicting trustworthy intentions and behavior,” Journal of Personality and Social Psychology, vol. 115, no. 3, pp. 468–494, Sep. 2018. [find](https://scholar.google.com/scholar?q=E%2E+E%2E+Levine%2C+T%2E+B%2E+Bitterly%2C+T%2E+R%2E+Cohen%2C+and+M%2E+E%2E+Schweitzer%2C+%E2%80%9CWho+is+trustworthy%3F+Predicting+trustworthy+intentions+and+behavior%2C%E2%80%9D+Journal+of+Personality+and+Social+Psychology%2C+vol%2E+115%2C+no%2E+3%2C+pp)
2. M. K. Lee, “Understanding perception of algorithmic decisions: Fairness, trust, and emotion in response to algorithmic management,” Big Data & Society, vol. 5, no. 1, Jan.–Jun. 2018. [find](https://scholar.google.com/scholar?q=M%2E+K%2E+Lee%2C+%E2%80%9CUnderstanding+perception+of+algorithmic+decisions%3A+Fairness%2C+trust%2C+and+emotion+in+response+to+algorithmic+management%2C%E2%80%9D+Big+Data+%26+Society%2C+vol%2E+5%2C+no%2E+1%2C+Jan%2E%E2%80%93Jun%2E+2018%2E)
3. K. R. Varshney and H. Alemzadeh, “On the safety of machine learning: Cyber-physical systems, decision sciences, and data products,” Big Data, vol. 5, no. 3, pp. 246–255, Sep. 2017. [find](https://scholar.google.com/scholar?q=K%2E+R%2E+Varshney+and+H%2E+Alemzadeh%2C+%E2%80%9COn+the+safety+of+machine+learning%3A+Cyber%2Dphysical+systems%2C+decision+sciences%2C+and+data+products%2C%E2%80%9D+Big+Data%2C+vol%2E+5%2C+no%2E+3%2C+pp%2E+246%E2%80%93255%2C+Sep%2E+2017%2E)
4. N. Papernot, P. McDaniel, S. Jha, M. Fredrikson, Z. B. Celik, and A. Swami, “The limitations of deep learning in adversarial settings,” in Proceedings of the IEEE European Symposium on Security and Privacy, Saarbrucken, Germany, Mar. 2016, pp. 372–387. [find](https://scholar.google.com/scholar?q=N%2E+Papernot%2C+P%2E+McDaniel%2C+S%2E+Jha%2C+M%2E+Fredrikson%2C+Z%2E+B%2E+Celik%2C+and+A%2E+Swami%2C+%E2%80%9CThe+limitations+of+deep+learning+in+adversarial+settings%2C%E2%80%9D+in+Proceedings+of+the+IEEE+European+Symposium+on+Security+and+Pr)
5. N. Moller, “The concepts of risk and safety,” in Handbook of Risk Theory, S. Roeser, R. Hiller-brand, P. Sandin, and M. Peterson, Eds. Dordrecht, Netherlands: Springer, 2012, pp. 55–85. [find](https://scholar.google.com/scholar?q=N%2E+Moller%2C+%E2%80%9CThe+concepts+of+risk+and+safety%2C%E2%80%9D+in+Handbook+of+Risk+Theory%2C+S%2E+Roeser%2C+R%2E+Hiller%2Dbrand%2C+P%2E+Sandin%2C+and+M%2E+Peterson%2C+Eds%2E+Dordrecht%2C+Netherlands%3A+Springer%2C+2012%2C+pp%2E+55%E2%80%9385%2E)
6. National Institute of Standards and Technology, “The use of supplier’s declaration of conformity,” [https://www.nist.gov/document-6075](https://www.nist.gov/document-6075). [↗](https://www.nist.gov/document-6075)
7. American National Standards Institute, “U. S. conformity assessment system: 1st party conformity assessment,” [https://www.standardsportal.org/usa](https://www.standardsportal.org/usa) en/-conformity assessment/suppliers declaration.-aspx. [↗](https://www.standardsportal.org/usa)
8. T. Gebru, J. Morgenstern, B. Vecchione, J. W. [19] [Online]. Available: Vaughan, H. Wallach, H. Daumé, III, and [https://www.cpsc.gov/Business--Manufacturing/Testing-Certific](https://www.cpsc.gov/Business--Manufacturing/Testing-Certific) K. Crawford, “Datasheets for datasets,” in Pro-Available: ceedings of the Fairness, Accountability, and [20] [Online]. http:/… [↗](https://www.cpsc.gov/Business--Manufacturing/Testing-Certific)
9. E. M. Bender and B. Friedman, “Data statements for NLP: Toward mitigating system bias [22] [Online]. Available: and enabling better science,” Transactions of the [https://www.spratings.com/documents/20184/774196/Guide](https://www.spratings.com/documents/20184/774196/Guide) to ACL, forthcoming. [23] [Online]. Available: http://www.trusteddigitalrepository.… [↗](https://www.spratings.com/documents/20184/774196/Guide)
10. S. Holland, A. Hosny, S. Newman, J. Joseph, and K. Chmielinski, “The dataset nutrition la-Available: bel: A framework to drive higher data quality [24] [Online]. [https://www.coretrustseal.org/about/standards,”](https://www.coretrustseal.org/about/standards,”) arXiv:1805.03677, May 2018. [25] A. K. Ghosh and G. McGraw, “An approach for [↗](https://www.coretrustseal.org/about/standards,”)
11. M. Mitchell, S. Wu, A. Zaldivar, P. Barnes, certifying security in software components,” in L. Vasserman, B. Hutchinson, E. Spitzer, I. D. the 21st National Information Systems Security Raji, and T. Gebru, “Model cards for model re-Conference, 1998. porting,” in Proceedings of the ACM Conference on… [find](https://scholar.google.com/scholar?q=M%2E+Mitchell%2C+S%2E+Wu%2C+A%2E+Zaldivar%2C+P%2E+Barnes%2C+certifying+security+in+software+components%2C%E2%80%9D+in+L%2E+Vasserman%2C+B%2E+Hutchinson%2C+E%2E+Spitzer%2C+I%2E+D%2E+the+21st+National+Information+Systems+Security+Raji%2C+and+T%2E+G)
12. M. Loukides, H. Mason, and D. Patil, “Of oaths and checklists,” [27] P. A. Currit, M. Dyer, and H. D. Mills, “Certifying the reliability of software,” IEEE Transac-https://www.oreilly.com/ideas/of-oaths-and-tions on Software Engineering 1, pp. 3–11, 1986. checklists, Jul. 2018. [find](https://scholar.google.com/scholar?q=M%2E+Loukides%2C+H%2E+Mason%2C+and+D%2E+Patil%2C+%E2%80%9COf+oaths+and+checklists%2C%E2%80%9D+%5B27%5D+P%2E+A%2E+Currit%2C+M%2E+Dyer%2C+and+H%2E+D%2E+Mills%2C+%E2%80%9CCertifying+the+reliability+of+software%2C%E2%80%9D+IEEE+Transac%2Dhttps%3A%2F%2Fwww%2Eoreilly%2Ecom%2Fideas%2Fof%2Doat)
13. M. Hind, S. Mehta, A. Mojsilović, R. Nair, [28] D. Port and J. Wilf, “The value of certifying software release readiness: an exploratory study of K. N. Ramamurthy, A. Olteanu, and K. R. certification for a critical system at jpl,” in 2013 Varshney, “Increasing trust in AI services ACM/IEEE Internat… — [arXiv:1808.07261](https://arxiv.org/abs/1808.07261) [find](https://scholar.google.com/scholar?q=M%2E+Hind%2C+S%2E+Mehta%2C+A%2E+Mojsilovic%CC%81%2C+R%2E+Nair%2C+%5B28%5D+D%2E+Port+and+J%2E+Wilf%2C+%E2%80%9CThe+value+of+certifying+software+release+readiness%3A+an+exploratory+study+of+K%2E+N%2E+Ramamurthy%2C+A%2E+Olteanu%2C+and+K%2E+R%2E+certification)
14. C. O’Neil, “What is a data audit?” [http://www.oneilrisk.com/articles/2017/1/24/-](http://www.oneilrisk.com/articles/2017/1/24/-) [29] P. Heck, M. Klabbers, and M. van Eekelen, “A software product certification model,” Software what-is-a-data-audit, Jan. 2017. Quality Journal, vol. 18, no. 1, 2010. [↗](http://www.oneilrisk.com/articles/2017/1/24/-)
15. R. Carrier, “AI safety— the [30] [Online]. Available: concept of independent audit,” [https://www.sei.cmu.edu/research-capabilities/all-work/display](https://www.sei.cmu.edu/research-capabilities/all-work/display). https://www.forhumanity.center/independent-audit-1/. [31] A. D. Selbst, “Disparate impact in big data policing,” Georgia Law Review, vol. 52, no. 1, [↗](https://www.sei.cmu.edu/research-capabilities/all-work/display)
16. The European Commission’s High-Level Expert pp. 109–195, Feb. 2017. Group on Artificial Intelligence, “Draft ethics guidelines for trustworthy AI,” Brussels, Bel- [32] J. M. Sims, “A brief review of the Belmont report,” Dimensions of Critical Care Nursing, gium, Dec. 2018. vol. 29, no. 4, pp. 173–17… [find](https://scholar.google.com/scholar?q=The+European+Commission%E2%80%99s+High%2DLevel+Expert+pp%2E+109%E2%80%93195%2C+Feb%2E+2017%2E+Group+on+Artificial+Intelligence%2C+%E2%80%9CDraft+ethics+guidelines+for+trustworthy+AI%2C%E2%80%9D+Brussels%2C+Bel%2D+%5B32%5D+J%2E+M%2E+Sims%2C+%E2%80%9CA+brief+review+of+t)
17. [Online]. Available: [http://standards.ieee.org/develop/overview.html](http://standards.ieee.org/develop/overview.html) [33] K. R. Varshney, “Data science of the people, for the people, by the people: A viewpoint on an emerging dichotomy,” in Data for Good Ex- [↗](http://standards.ieee.org/develop/overview.html)
18. [Online]. Available: change Conference, New York, USA, Sep. 2015. — [Towards A Rigorous Science of Interpretable Machine Learning](https://md.research.elamur.ai/free/en/papers/doshi-velez-2017-towards-rigorous-science-interpretable-machine/) [https://www.iso.org/home.html](https://www.iso.org/home.html) [34] X. Li, T. J. Hess, and J. S. Valacich, “Why [44] C. Rudin, “Algorithms for interpretable machine do we trust new technology? a study of initial learning,” in Proceedings of the ACM SIGKDD trust forma… [↗](https://www.iso.org/home.html)

*Updated 3 November 2026*