---
title: "Model evaluation for extreme risks — Shevlane 2023"
description: "Rather than introducing a dataset or benchmark, the authors propose a conceptual framework that… 48 references, cited by 50 papers here."
authors: ["Toby Shevlane", "Sebastian Farquhar", "Ben Garfinkel", "Mary Phuong", "Jess Whittlestone", "Jade Leung", "Daniel Kokotajlo", "Nahema Marchal", "Markus Anderljung", "Noam Kolt", "Lewis Ho", "Divya Siddarth", "Shahar Avin", "Will Hawkins", "Been Kim", "Iason Gabriel", "Vijay Bolina", "Jack Clark", "Yoshua Bengio", "Paul Christiano", "Allan Dafoe"]
year: 2023
venue: "arXiv:2305.15324"
source: "https://arxiv.org/abs/2305.15324"
doi: "10.48550/arxiv.2305.15324"
arxiv: "2305.15324"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shevlane-2023-model-evaluation-extreme-risks/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shevlane-2023-model-evaluation-extreme-risks/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/) / Model evaluation for extreme risks

# Model evaluation for extreme risks

**Authors:** Toby Shevlane, [Sebastian Farquhar](https://md.research.elamur.ai/free/en/authors/sebastian-farquhar/), Ben Garfinkel, Mary Phuong, Jess Whittlestone, Jade Leung, Daniel Kokotajlo, Nahema Marchal, Markus Anderljung, Noam Kolt, Lewis Ho, Divya Siddarth, Shahar Avin, Will Hawkins, [Been Kim](https://md.research.elamur.ai/free/en/authors/been-kim/), [Iason Gabriel](https://md.research.elamur.ai/free/en/authors/iason-gabriel/), Vijay Bolina, [Jack Clark](https://md.research.elamur.ai/free/en/authors/jack-clark/), [Yoshua Bengio](https://md.research.elamur.ai/free/en/authors/yoshua-bengio/), [Paul Christiano](https://md.research.elamur.ai/free/en/authors/paul-christiano/) and Allan Dafoe · **Year:** 2023 · **Venue:** arXiv:2305.15324 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2305.15324) · [PDF](https://arxiv.org/pdf/2305.15324) · [DOI 10.48550/arxiv.2305.15324](https://doi.org/10.48550/arxiv.2305.15324) · [arXiv 2305.15324](https://arxiv.org/abs/2305.15324) · [OpenAlex](https://openalex.org/W4378474292)

**Citations:** 54 citations (OpenAlex, 2026-10-14) · [50](https://md.research.elamur.ai/free/en/papers/shevlane-2023-model-evaluation-extreme-risks/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary

This position paper, from authors at Google DeepMind, OpenAI, Anthropic, GovAI and academic institutions, argues that general-purpose AI models may develop harmful abilities their creators never targeted. Examples include cyber-offense and manipulation, which could lead to harm on a catastrophic scale. Rather than introducing a dataset or benchmark, the authors propose a conceptual framework that expands evaluation practice in two directions. First, *dangerous capability evaluations* test what a model is able to do. Second, *alignment evaluations* test whether the model tends to use those abilities harmfully.

The paper explains how evaluation results should inform four governance areas: decisions about training, decisions about deployment, transparency toward stakeholders, and information security. It lists candidate dangerous capabilities and sets out design criteria for evaluations. It also acknowledges limitations, such as models being trained to pass the tests superficially and harms caused by the testing itself. The main contribution is a theory of change that places such evaluations inside company policy and regulation. This matters because it shaped how frontier labs and governments later approached pre-deployment safety testing.

## Key points

- **Scope of "extreme" risk:** The authors define it as harm on the order of tens of thousands of deaths or hundreds of billions of dollars in damage, or as severe disruption of social and political order. They cite a 2022 survey in which 36% of AI researchers considered a catastrophe comparable to nuclear war plausible this century.
- **Dangerous capabilities (Table 1):** The non-exhaustive list includes cyber-offense, deception, persuasion and manipulation, political strategy, weapons acquisition, long-horizon planning, AI development and situational awareness.
- **Alignment evaluation targets:** These include pursuing unintended long-term goals, power-seeking, resisting shutdown, colluding with other AI systems, and how well the model resists misuse by malicious users.
- **Unintended harms of evaluation:** One risk is overfitting to evaluations, so the authors suggest keeping some held-out tests private. Another is harm during testing itself, illustrated by ARC's GPT-4 test, in which the model produced deceptive messages for a TaskRabbit worker.
- **Recommendations:** Developers should invest in evaluation research, set internal policies, support outside researchers and educate policymakers. Policymakers should track capabilities, fund external evaluation, mandate audits and block deployment of models that pose extreme risks.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Cambridge](https://md.research.elamur.ai/free/en/universities/university-of-cambridge/) — University, Cambridge, United Kingdom · 256 works · 123,373 citations
- [University of Toronto](https://md.research.elamur.ai/free/en/universities/university-of-toronto/) — University, Toronto, Canada · 110 works · 179,062 citations
- [University of Oxford](https://md.research.elamur.ai/free/en/universities/university-of-oxford/) — University, Oxford, United Kingdom · 122 works · 160,277 citations
- [Université de Montréal](https://md.research.elamur.ai/free/en/universities/universite-de-montreal/) — University, Montreal, Canada · 87 works · 91,597 citations
- [OpenAI](https://md.research.elamur.ai/free/en/companies/openai/) — Company, San Francisco, United States · 23 works · 27,874 citations
- [Mila - Quebec Artificial Intelligence Institute](https://md.research.elamur.ai/free/en/labs/mila-quebec-artificial-intelligence-institute/) — Research institute, Montreal, Canada · 65 works · 14,425 citations
- [Anthropic](https://md.research.elamur.ai/free/en/companies/anthropic/) — Company, San Francisco, United States · 8 works · 1,908 citations
- [Centre for the Governance of AI](https://md.research.elamur.ai/free/en/ngos/centre-for-the-governance-of-ai/) — Nonprofit, Oxford, United Kingdom · 2 works · 302 citations
- [American Anthropological Association](https://md.research.elamur.ai/free/en/labs/american-anthropological-association/) — Research institute, Arlington, United States · 2 works · 225 citations
- [Anthropological Survey of India](https://md.research.elamur.ai/free/en/government/anthropological-survey-of-india/) — Government body, Kolkata, India · 2 works · 225 citations
- [Institute for Anthropological Research](https://md.research.elamur.ai/free/en/labs/institute-for-anthropological-research/) — Research institute, Zagreb, Croatia · 2 works · 225 citations
- [Laboratoire d'Anthropologie Sociale](https://md.research.elamur.ai/free/en/labs/laboratoire-d-anthropologie-sociale/) — Research institute, Paris, France · 2 works · 225 citations

Also on the byline: [The Collective Intelligence Project](https://ror.org/02190e883).

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,954 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 626 works · [⚙️ Engineering](https://md.research.elamur.ai/free/en/shelves/engineering/) · 188 works · [⚙️ Safety, Risk, Reliability and Quality](https://md.research.elamur.ai/free/en/shelves/engineering/safety-risk-reliability-and-quality/) · 14 works

🧭 Topics (1): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works

## Cite

Cite as: Toby Shevlane, Sebastian Farquhar, Ben Garfinkel, Mary Phuong, Jess Whittlestone, Jade Leung, Daniel Kokotajlo, Nahema Marchal, Markus Anderljung, Noam Kolt, Lewis Ho, Divya Siddarth, Shahar Avin, Will Hawkins, Been Kim, Iason Gabriel, Vijay Bolina, Jack Clark, Yoshua Bengio, Paul Christiano and Allan Dafoe. 2023. Model evaluation for extreme risks. arXiv:2305.15324.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{shevlane2023model,
  title = {{Model evaluation for extreme risks}},
  author = {Toby Shevlane, Sebastian Farquhar, Ben Garfinkel, Mary Phuong, Jess Whittlestone, Jade Leung, Daniel Kokotajlo, Nahema Marchal, Markus Anderljung, Noam Kolt, Lewis Ho, Divya Siddarth, Shahar Avin, Will Hawkins, Been Kim, Iason Gabriel, Vijay Bolina, Jack Clark, Yoshua Bengio, Paul Christiano and Allan Dafoe},
  year = {2023},
  doi = {10.48550/arxiv.2305.15324},
  eprint = {2305.15324},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.15324}
}
```

- [← previous LENS: A Learnable Evaluation Metric for Text Simplification](https://md.research.elamur.ai/free/en/papers/maddela-2023-lens-learnable-evaluation-metric-text/)
- [next → Transformers for Tabular Data Representation: A Survey of Models and Applications](https://md.research.elamur.ai/free/en/papers/badaro-2023-transformers-tabular-data-representation-survey/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/)

## References

48 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anthropic. Strengthening U.S. AI innovation through an ambitious investment in NIST. Technical report, 2023. [find](https://scholar.google.com/scholar?q=Anthropic%2E+Strengthening+U%2ES%2E+AI+innovation+through+an+ambitious+investment+in+NIST%2E+Technical+report%2C+2023%2E)
2. ARC Evals. Update on ARC’s recent eval efforts. [https://evals.alignment.org/blog/](https://evals.alignment.org/blog/) 2023-03-18-update-on-recent-evals/ , Mar. 2023. Accessed: 2023-3-20. [↗](https://evals.alignment.org/blog/)
3. E. Bluemke, T. Collins, B. Garfinkel, and A. Trask. Exploring the relevance of data Privacy-Enhancing technologies for AI governance use cases. Mar. 2023. [find](https://scholar.google.com/scholar?q=E%2E+Bluemke%2C+T%2E+Collins%2C+B%2E+Garfinkel%2C+and+A%2E+Trask%2E+Exploring+the+relevance+of+data+Privacy%2DEnhancing+technologies+for+AI+governance+use+cases%2E+Mar%2E+2023%2E)
4. M. Brundage, S. Avin, J. Clark, H. Toner, P. Eckersley, B. Garfinkel, A. Dafoe, P. Scharre, T. Zeitzoff, B. Filar, H. Anderson, H. Roff, G. C. Allen, J. Steinhardt, C. Flynn, S. Ó. hÉigeartaigh, S. Beard, H. Belfield, S. Farquhar, C. Lyle, R. Crootof, O. Evans, M. Page, J. Bryson, R. Yampolskiy, and… [find](https://scholar.google.com/scholar?q=M%2E+Brundage%2C+S%2E+Avin%2C+J%2E+Clark%2C+H%2E+Toner%2C+P%2E+Eckersley%2C+B%2E+Garfinkel%2C+A%2E+Dafoe%2C+P%2E+Scharre%2C+T%2E+Zeitzoff%2C+B%2E+Filar%2C+H%2E+Anderson%2C+H%2E+Roff%2C+G%2E+C%2E+Allen%2C+J%2E+Steinhardt%2C+C%2E+Flynn%2C+S%2E+%C3%93%2E+h%C3%89igeartaigh%2C+S%2E+Be)
5. M. Brundage, S. Avin, J. Wang, H. Belfield, G. Krueger, G. Hadfield, H. Khlaaf, J. Yang, H. Toner, R. Fong, T. Maharaj, P. W. Koh, S. Hooker, J. Leung, A. Trask, E. Bluemke, J. Lebensold, C. O’Keefe, M. Koren, T. Ryffel, J. B. Rubinovitz, T. Besiroglu, F. Carugati, J. Clark, P. Eckersley, S. de Haas… [find](https://scholar.google.com/scholar?q=M%2E+Brundage%2C+S%2E+Avin%2C+J%2E+Wang%2C+H%2E+Belfield%2C+G%2E+Krueger%2C+G%2E+Hadfield%2C+H%2E+Khlaaf%2C+J%2E+Yang%2C+H%2E+Toner%2C+R%2E+Fong%2C+T%2E+Maharaj%2C+P%2E+W%2E+Koh%2C+S%2E+Hooker%2C+J%2E+Leung%2C+A%2E+Trask%2C+E%2E+Bluemke%2C+J%2E+Lebensold%2C+C%2E+O%E2%80%99Keefe%2C+)
6. M. Brundage, K. Mayer, T. Eloundou, S. Agarwal, S. Adler, G. Krueger, J. Leike, and P. Mishkin. Lessons learned on language model safety and misuse. [https://openai.com/research/](https://openai.com/research/) language-model-safety-and-misuse , Mar. 2022. Accessed: 2023-4-17. [↗](https://openai.com/research/)
7. S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, H. Nori, H. Palangi, M. T. Ribeiro, and Y. Zhang. [Sparks of artificial general intelligence: Early experiments with GPT-4](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/). Mar. 2023.
8. C. Burns, H. Ye, D. Klein, and J. Steinhardt. Discovering latent knowledge in language models without supervision. Dec. 2022. [find](https://scholar.google.com/scholar?q=C%2E+Burns%2C+H%2E+Ye%2C+D%2E+Klein%2C+and+J%2E+Steinhardt%2E+Discovering+latent+knowledge+in+language+models+without+supervision%2E+Dec%2E+2022%2E)
9. J. Carlsmith. Is Power-Seeking AI an existential risk? June 2022. [find](https://scholar.google.com/scholar?q=J%2E+Carlsmith%2E+Is+Power%2DSeeking+AI+an+existential+risk%3F+June+2022%2E)
10. A. Chan, R. Salganik, A. Markelius, C. Pang, N. Rajkumar, D. Krasheninnikov, L. Langosco, Z. He, Y. Duan, M. Carroll, M. Lin, A. Mayhew, K. Collins, M. Molamohammadi, J. Burden, W. Zhao, S. Rismani, K. Voudouris, U. Bhatt, A. Weller, D. Krueger, and T. Maharaj. Harms from increasingly agentic algori… [find](https://scholar.google.com/scholar?q=A%2E+Chan%2C+R%2E+Salganik%2C+A%2E+Markelius%2C+C%2E+Pang%2C+N%2E+Rajkumar%2C+D%2E+Krasheninnikov%2C+L%2E+Langosco%2C+Z%2E+He%2C+Y%2E+Duan%2C+M%2E+Carroll%2C+M%2E+Lin%2C+A%2E+Mayhew%2C+K%2E+Collins%2C+M%2E+Molamohammadi%2C+J%2E+Burden%2C+W%2E+Zhao%2C+S%2E+Rismani%2C+K)
11. N. Emery-Xu, A. Park, and R. Trager. Uncertainty, information, and risk in international technology races. Unpublished manuscript, 2023. [find](https://scholar.google.com/scholar?q=N%2E+Emery%2DXu%2C+A%2E+Park%2C+and+R%2E+Trager%2E+Uncertainty%2C+information%2C+and+risk+in+international+technology+races%2E+Unpublished+manuscript%2C+2023%2E)
12. T. Everitt, R. Carey, E. Langlois, P. A. Ortega, and S. Legg. Agent incentives: A causal perspective. Feb. 2021. [find](https://scholar.google.com/scholar?q=T%2E+Everitt%2C+R%2E+Carey%2C+E%2E+Langlois%2C+P%2E+A%2E+Ortega%2C+and+S%2E+Legg%2E+Agent+incentives%3A+A+causal+perspective%2E+Feb%2E+2021%2E)
13. D. Ganguli, D. Hernandez, L. Lovitt, N. DasSarma, T. Henighan, A. Jones, N. Joseph, J. Kernion, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, D. Drain, N. Elhage, S. El Showk, S. Fort, Z. Hatfield-Dodds, S. Johnston, S. Kravec, N. Nanda, K. Ndousse, C. Olsson, D. Amodei, D. Amodei, T. Brown, J. K… — [Predictability and Surprise in Large Generative Models](https://md.research.elamur.ai/free/en/papers/ganguli-2022-predictability-surprise-large-generative-models/)
14. L. Gao, J. Schulman, and J. Hilton. [Scaling laws for reward model overoptimization](https://md.research.elamur.ai/free/en/papers/gao-2022-scaling-laws-reward-model-overoptimization/). Oct. 2022.
15. A. Glaese, N. McAleese, M. Trębacz, J. Aslanides, V. Firoiu, T. Ewalds, M. Rauh, L. Weidinger, M. Chadwick, P. Thacker, L. Campbell-Gillingham, J. Uesato, P.-S. Huang, R. Comanescu, F. Yang, A. See, S. Dathathri, R. Greig, C. Chen, D. Fritz, J. S. Elias, R. Green, S. Mokrá, N. Fernando, B. Wu, R. Fo… — [Improving alignment of dialogue agents via targeted human judgements](https://md.research.elamur.ai/free/en/papers/glaese-2022-improving-alignment-dialogue-agents-targeted/)
16. D. Hadfield-Menell, A. Dragan, P. Abbeel, and S. Russell. The Off-Switch game. Nov. 2016. [find](https://scholar.google.com/scholar?q=D%2E+Hadfield%2DMenell%2C+A%2E+Dragan%2C+P%2E+Abbeel%2C+and+S%2E+Russell%2E+The+Off%2DSwitch+game%2E+Nov%2E+2016%2E)
17. E. Jones, A. Dragan, A. Raghunathan, and J. Steinhardt. [Automatically auditing large language models via discrete optimization](https://md.research.elamur.ai/free/en/papers/jones-2023-automatically-auditing-large-language-models/). Mar. 2023.
18. Z. Kenton, R. Kumar, S. Farquhar, J. Richens, M. MacDermott, and T. Everitt. Discovering agents. Aug. 2022. [find](https://scholar.google.com/scholar?q=Z%2E+Kenton%2C+R%2E+Kumar%2C+S%2E+Farquhar%2C+J%2E+Richens%2C+M%2E+MacDermott%2C+and+T%2E+Everitt%2E+Discovering+agents%2E+Aug%2E+2022%2E)
19. H. Khlaaf, P. Mishkin, J. Achiam, G. Krueger, and M. Brundage. A hazard analysis framework for code synthesis large language models. July 2022. [find](https://scholar.google.com/scholar?q=H%2E+Khlaaf%2C+P%2E+Mishkin%2C+J%2E+Achiam%2C+G%2E+Krueger%2C+and+M%2E+Brundage%2E+A+hazard+analysis+framework+for+code+synthesis+large+language+models%2E+July+2022%2E)
20. V. Krakovna and J. Kramar. Power-seeking can be probable and predictive for trained agents. Apr. 2023. [find](https://scholar.google.com/scholar?q=V%2E+Krakovna+and+J%2E+Kramar%2E+Power%2Dseeking+can+be+probable+and+predictive+for+trained+agents%2E+Apr%2E+2023%2E)
21. P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar, B. Newman, B. Yuan, B. Yan, C. Zhang, C. Cosgrove, C. D. Manning, C. Ré, D. Acosta-Navas, D. A. Hudson, E. Zelikman, E. Durmus, F. Ladhak, F. Rong, H. Ren, H. Yao, J. Wang, K. Santhanam, L. Or… [find](https://scholar.google.com/scholar?q=P%2E+Liang%2C+R%2E+Bommasani%2C+T%2E+Lee%2C+D%2E+Tsipras%2C+D%2E+Soylu%2C+M%2E+Yasunaga%2C+Y%2E+Zhang%2C+D%2E+Narayanan%2C+Y%2E+Wu%2C+A%2E+Kumar%2C+B%2E+Newman%2C+B%2E+Yuan%2C+B%2E+Yan%2C+C%2E+Zhang%2C+C%2E+Cosgrove%2C+C%2E+D%2E+Manning%2C+C%2E+R%C3%A9%2C+D%2E+Acosta%2DNavas%2C+D%2E)
22. I. McKenzie, A. Lyzhov, A. Parrish, A. Prabhu, A. Mueller, N. Kim, S. Bowman, and E. Perez. Inverse scaling prize. [https://github.com/inverse-scaling/prize](https://github.com/inverse-scaling/prize) , 2022. Accessed: 2023-4-7. [↗](https://github.com/inverse-scaling/prize)
23. J. Michael, A. Holtzman, A. Parrish, A. Mueller, A. Wang, A. Chen, D. Madaan, N. Nangia, R. Y. Pang, J. Phang, and S. R. Bowman. What do NLP researchers believe? results of the NLP community metasurvey. Aug. 2022. [find](https://scholar.google.com/scholar?q=J%2E+Michael%2C+A%2E+Holtzman%2C+A%2E+Parrish%2C+A%2E+Mueller%2C+A%2E+Wang%2C+A%2E+Chen%2C+D%2E+Madaan%2C+N%2E+Nangia%2C+R%2E+Y%2E+Pang%2C+J%2E+Phang%2C+and+S%2E+R%2E+Bowman%2E+What+do+NLP+researchers+believe%3F+results+of+the+NLP+community+metasurve)
24. M. Mitchell, S. Wu, A. Zaldivar, P. Barnes, L. Vasserman, B. Hutchinson, E. Spitzer, I. D. Raji, and T. Gebru. Model cards for model reporting. Oct. 2018. [find](https://scholar.google.com/scholar?q=M%2E+Mitchell%2C+S%2E+Wu%2C+A%2E+Zaldivar%2C+P%2E+Barnes%2C+L%2E+Vasserman%2C+B%2E+Hutchinson%2C+E%2E+Spitzer%2C+I%2E+D%2E+Raji%2C+and+T%2E+Gebru%2E+Model+cards+for+model+reporting%2E+Oct%2E+2018%2E)
25. J. Mökander, J. Schuett, H. R. Kirk, and L. Floridi. Auditing large language models: a three-layered approach. Feb. 2023. [find](https://scholar.google.com/scholar?q=J%2E+M%C3%B6kander%2C+J%2E+Schuett%2C+H%2E+R%2E+Kirk%2C+and+L%2E+Floridi%2E+Auditing+large+language+models%3A+a+three%2Dlayered+approach%2E+Feb%2E+2023%2E)
26. N. Nanda, L. Chan, T. Lieberum, J. Smith, and J. Steinhardt. [Progress measures for grokking via mechanistic interpretability](https://md.research.elamur.ai/free/en/papers/nanda-2023-progress-measures-grokking-mechanistic-interpretability/). Jan. 2023.
27. R. Ngo, L. Chan, and S. Mindermann. [The alignment problem from a deep learning perspective](https://md.research.elamur.ai/free/en/papers/ngo-2022-alignment-problem-deep-learning-perspective/). Aug. 2022.
28. C. Olah, N. Cammarata, L. Schubert, G. Goh, M. Petrov, and S. Carter. Zoom in: An introduction to circuits. Distill, 5(3), Mar. 2020. [find](https://scholar.google.com/scholar?q=C%2E+Olah%2C+N%2E+Cammarata%2C+L%2E+Schubert%2C+G%2E+Goh%2C+M%2E+Petrov%2C+and+S%2E+Carter%2E+Zoom+in%3A+An+introduction+to+circuits%2E+Distill%2C+5%283%29%2C+Mar%2E+2020%2E)
29. OpenAI. GPT-4 system card. Mar. 2023a. [find](https://scholar.google.com/scholar?q=OpenAI%2E+GPT%2D4+system+card%2E+Mar%2E+2023a%2E)
30. OpenAI. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). Mar. 2023b.
31. T. Ord. Lessons from the development of the atomic bomb. Technical report, Centre for the Governance of AI, Nov. 2022. [find](https://scholar.google.com/scholar?q=T%2E+Ord%2E+Lessons+from+the+development+of+the+atomic+bomb%2E+Technical+report%2C+Centre+for+the+Governance+of+AI%2C+Nov%2E+2022%2E)
32. L. Orseau and S. Armstrong. Safely interruptible agents. [https://www.deepmind.com/](https://www.deepmind.com/) publications/safely-interruptible-agents , 2016. Accessed: 2023-4-26. [↗](https://www.deepmind.com/)
33. A. Pan, C. J. Shern, A. Zou, N. Li, S. Basart, T. Woodside, J. Ng, H. Zhang, S. Emmons, and D. Hendrycks. Do the rewards justify the means? measuring Trade-Offs between rewards and ethical behavior in the MACHIAVELLI benchmark. Apr. 2023. [find](https://scholar.google.com/scholar?q=A%2E+Pan%2C+C%2E+J%2E+Shern%2C+A%2E+Zou%2C+N%2E+Li%2C+S%2E+Basart%2C+T%2E+Woodside%2C+J%2E+Ng%2C+H%2E+Zhang%2C+S%2E+Emmons%2C+and+D%2E+Hendrycks%2E+Do+the+rewards+justify+the+means%3F+measuring+Trade%2DOffs+between+rewards+and+ethical+behavior+in)
34. E. Perez, S. Huang, F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, and G. Irving. [Red teaming language models with language models](https://md.research.elamur.ai/free/en/papers/perez-2022-red-teaming-language-models-language/). Feb. 2022a.
35. E. Perez, S. Ringer, K. Lukošiūtė, K. Nguyen, E. Chen, S. Heiner, C. Pettit, C. Olsson, S. Kundu, S. Kadavath, A. Jones, A. Chen, B. Mann, B. Israel, B. Seethor, C. McKinnon, C. Olah, D. Yan, D. Amodei, D. Amodei, D. Drain, D. Li, E. Tran-Johnson, G. Khundadze, J. Kernion, J. Landis, J. Kerr, J. M… — [Discovering Language Model Behaviors with Model-Written Evaluations](https://md.research.elamur.ai/free/en/papers/perez-2022-discovering-language-model-behaviors-model/)
36. I. D. Raji, A. Smart, R. N. White, M. Mitchell, T. Gebru, B. Hutchinson, J. Smith-Loud, D. Theron, and P. Barnes. Closing the AI accountability gap: Defining an End-to-End framework for internal algorithmic auditing. Jan. 2020. [find](https://scholar.google.com/scholar?q=I%2E+D%2E+Raji%2C+A%2E+Smart%2C+R%2E+N%2E+White%2C+M%2E+Mitchell%2C+T%2E+Gebru%2C+B%2E+Hutchinson%2C+J%2E+Smith%2DLoud%2C+D%2E+Theron%2C+and+P%2E+Barnes%2E+Closing+the+AI+accountability+gap%3A+Defining+an+End%2Dto%2DEnd+framework+for+internal+algor)
37. I. D. Raji, I. Elizabeth Kumar, A. Horowitz, and A. D. Selbst. The fallacy of AI functionality. June 2022a. [find](https://scholar.google.com/scholar?q=I%2E+D%2E+Raji%2C+I%2E+Elizabeth+Kumar%2C+A%2E+Horowitz%2C+and+A%2E+D%2E+Selbst%2E+The+fallacy+of+AI+functionality%2E+June+2022a%2E)
38. I. D. Raji, P. Xu, C. Honigsberg, and D. E. Ho. Outsider oversight: Designing a third party audit ecosystem for AI governance. June 2022b. [find](https://scholar.google.com/scholar?q=I%2E+D%2E+Raji%2C+P%2E+Xu%2C+C%2E+Honigsberg%2C+and+D%2E+E%2E+Ho%2E+Outsider+oversight%3A+Designing+a+third+party+audit+ecosystem+for+AI+governance%2E+June+2022b%2E)
39. S. Rismani, R. Shelby, A. Smart, E. Jatho, J. Kroll, A. Moon, and N. Rostamzadeh. From plane crashes to algorithmic harm: Applicability of safety engineering frameworks for responsible ML. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, number Article 2 in CHI ’23, p… [find](https://scholar.google.com/scholar?q=S%2E+Rismani%2C+R%2E+Shelby%2C+A%2E+Smart%2C+E%2E+Jatho%2C+J%2E+Kroll%2C+A%2E+Moon%2C+and+N%2E+Rostamzadeh%2E+From+plane+crashes+to+algorithmic+harm%3A+Applicability+of+safety+engineering+frameworks+for+responsible+ML%2E+In+Proceedi)
40. R. Shah, V. Varma, R. Kumar, M. Phuong, V. Krakovna, J. Uesato, and Z. Kenton. Goal misgeneralization: Why correct specifications aren’t enough for correct goals. Oct. 2022. [find](https://scholar.google.com/scholar?q=R%2E+Shah%2C+V%2E+Varma%2C+R%2E+Kumar%2C+M%2E+Phuong%2C+V%2E+Krakovna%2C+J%2E+Uesato%2C+and+Z%2E+Kenton%2E+Goal+misgeneralization%3A+Why+correct+specifications+aren%E2%80%99t+enough+for+correct+goals%2E+Oct%2E+2022%2E)
41. T. Shevlane. Sharing powerful AI models. [https://www.governance.ai/post/](https://www.governance.ai/post/) sharing-powerful-ai-models , Jan. 2022a. Accessed: 2023-3-18. [↗](https://www.governance.ai/post/)
42. T. Shevlane. Structured access: An emerging paradigm for safe AI deployment. Feb. 2022b. [find](https://scholar.google.com/scholar?q=T%2E+Shevlane%2E+Structured+access%3A+An+emerging+paradigm+for+safe+AI+deployment%2E+Feb%2E+2022b%2E)
43. A. Turner, L. Smith, R. Shah, A. Critch, and P. Tadepalli. Optimal policies tend to seek power. Adv. Neural Inf. Process. Syst., 34:23063–23074, Dec. 2021. [find](https://scholar.google.com/scholar?q=A%2E+Turner%2C+L%2E+Smith%2C+R%2E+Shah%2C+A%2E+Critch%2C+and+P%2E+Tadepalli%2E+Optimal+policies+tend+to+seek+power%2E+Adv%2E+Neural+Inf%2E+Process%2E+Syst%2E%2C+34%3A23063%E2%80%9323074%2C+Dec%2E+2021%2E)
44. J. Wei, N. Kim, Y. Tay, and Q. V. Le. Inverse scaling can become u-shaped. Nov. 2022a. [find](https://scholar.google.com/scholar?q=J%2E+Wei%2C+N%2E+Kim%2C+Y%2E+Tay%2C+and+Q%2E+V%2E+Le%2E+Inverse+scaling+can+become+u%2Dshaped%2E+Nov%2E+2022a%2E)
45. J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, E. H. Chi, T. Hashimoto, O. Vinyals, P. Liang, J. Dean, and W. Fedus. [Emergent abilities of large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/). June 2022b.
46. J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou. [Chain-of-Thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Jan. 2022c.
47. J. Whittlestone and J. Clark. Why and how governments should monitor AI development. Aug. 2021. [find](https://scholar.google.com/scholar?q=J%2E+Whittlestone+and+J%2E+Clark%2E+Why+and+how+governments+should+monitor+AI+development%2E+Aug%2E+2021%2E)
48. D. M. Ziegler, S. Nix, L. Chan, T. Bauman, P. Schmidt-Nielsen, T. Lin, A. Scherlis, N. Nabeshima, B. Weinstein-Raun, D. de Haas, B. Shlegeris, and N. Thomas. Adversarial training for High-Stakes reliability. May 2022. [find](https://scholar.google.com/scholar?q=D%2E+M%2E+Ziegler%2C+S%2E+Nix%2C+L%2E+Chan%2C+T%2E+Bauman%2C+P%2E+Schmidt%2DNielsen%2C+T%2E+Lin%2C+A%2E+Scherlis%2C+N%2E+Nabeshima%2C+B%2E+Weinstein%2DRaun%2C+D%2E+de+Haas%2C+B%2E+Shlegeris%2C+and+N%2E+Thomas%2E+Adversarial+training+for+High%2DStakes+relia)

*Updated 2 November 2026*