---
title: "Michelangelo — paper by Attaluri; and Olszewska (2024)"
description: "The authors propose the Latent Structure Queries (LSQ) framework. 30 references, cited by 35 papers here."
authors: ["Kiran Vodrahalli; Santiago Ontanon; Nilesh Tripuraneni; Kelvin Xu; Sanil Jain; Rakesh Shivanna; Jeffrey Hui; Nishanth Dikkala; Mehran Kazemi; Bahare Fatemi; Rohan Anil; Ethan Dyer; Siamak Shakeri; Roopali Vij; Harsh Mehta; Vinay Ramasesh; Quoc Le; Ed Chi; Yifeng Lu; Orhan Firat; Angeliki Lazaridou; Jean-Baptiste Lespiau; Nithya Attaluri;", "Kate Olszewska"]
year: 2024
venue: "arXiv:2409.12640"
source: "https://arxiv.org/abs/2409.12640"
doi: "10.48550/arxiv.2409.12640"
arxiv: "2409.12640"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/vodrahalli-2024-michelangelo-long-context-evaluations-haystacks/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/vodrahalli-2024-michelangelo-long-context-evaluations-haystacks/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/9/) / Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries

# Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries

**Authors:** Kiran Vodrahalli; Santiago Ontanon; Nilesh Tripuraneni; Kelvin Xu; Sanil Jain; Rakesh Shivanna; Jeffrey Hui; Nishanth Dikkala; Mehran Kazemi; Bahare Fatemi; Rohan Anil; Ethan Dyer; Siamak Shakeri; Roopali Vij; Harsh Mehta; Vinay Ramasesh; Quoc Le; Ed Chi; Yifeng Lu; Orhan Firat; Angeliki Lazaridou; Jean-Baptiste Lespiau; Nithya Attaluri; and [Kate Olszewska](https://md.research.elamur.ai/free/en/authors/kate-olszewska/) · **Year:** 2024 · **Venue:** arXiv:2409.12640 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2409.12640) · [PDF](https://arxiv.org/pdf/2409.12640) · [DOI 10.48550/arxiv.2409.12640](https://doi.org/10.48550/arxiv.2409.12640) · [arXiv 2409.12640](https://arxiv.org/abs/2409.12640) · [OpenAlex](https://openalex.org/W4403713491)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [35](https://md.research.elamur.ai/free/en/papers/vodrahalli-2024-michelangelo-long-context-evaluations-haystacks/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Widely used long-context tests, such as Needle-in-a-Haystack, mainly check whether a model can locate one or a few facts. That reveals little about whether it can combine information scattered across a very large input. Existing reasoning benchmarks have their own problems: short contexts, costly manual scaling, data likely leaked into training, or shortcuts that let a model skip most of the context.

The authors propose the Latent Structure Queries (LSQ) framework. A long input conceals a structure among irrelevant material, and the model must strip away the noise and answer questions about that hidden structure. Using LSQ, they build Michelangelo, a compact synthetic benchmark that can be regenerated automatically and scored easily. It has three tasks set in code and natural language: Latent List, MRCR and IDK.

Testing ten frontier models up to 128K tokens, and the Gemini 1.5 family up to 1M, reveals a steep early drop, often before 32K, followed by a flatter curve. Long-context synthesis therefore remains largely unsolved. Michelangelo offers a leakage-resistant way to track progress on it.

## Key points
- **Latent List:** the model tracks a short Python list through operations (append, insert, pop, remove, sort, reverse) hidden among no-op filler. It then reports a slice, sum, min, max or length of the result. Complexity levels are 1, 5 or 20 relevant operations, using values drawn from [−4000, 4000].
- **MRCR and IDK:** MRCR (multi-round coreference resolution) requires reproducing a specific earlier part of the context while respecting order. IDK tests whether the model can recognize that the context does not contain the answer.
- **Scale of evaluation:** ten frontier models were evaluated up to 128K tokens, and Gemini 1.5 Pro/Flash up to 1M. GPT and Claude models stay non-trivial up to 128K, while Gemini models generalize as far as 1M.
- **Family resemblance:** models from the same family (Gemini, GPT, Claude) show parallel MRCR curves, most strikingly Claude 3.5 Sonnet and Claude 3 Opus.
- **Recommendation:** the authors propose MRCR as a default replacement for Needle-in-a-Haystack. It uses a simple metric, is robust to prompt choice, and gives a useful signal even for smaller and non-post-trained models.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 492 works · 440,462 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

## Cite

Cite as: Kiran Vodrahalli; Santiago Ontanon; Nilesh Tripuraneni; Kelvin Xu; Sanil Jain; Rakesh Shivanna; Jeffrey Hui; Nishanth Dikkala; Mehran Kazemi; Bahare Fatemi; Rohan Anil; Ethan Dyer; Siamak Shakeri; Roopali Vij; Harsh Mehta; Vinay Ramasesh; Quoc Le; Ed Chi; Yifeng Lu; Orhan Firat; Angeliki Lazaridou; Jean-Baptiste Lespiau; Nithya Attaluri; and Kate Olszewska. 2024. Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries. arXiv:2409.12640.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{attaluri2024michelangelo,
  title = {{Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries}},
  author = {Kiran Vodrahalli; Santiago Ontanon; Nilesh Tripuraneni; Kelvin Xu; Sanil Jain; Rakesh Shivanna; Jeffrey Hui; Nishanth Dikkala; Mehran Kazemi; Bahare Fatemi; Rohan Anil; Ethan Dyer; Siamak Shakeri; Roopali Vij; Harsh Mehta; Vinay Ramasesh; Quoc Le; Ed Chi; Yifeng Lu; Orhan Firat; Angeliki Lazaridou; Jean-Baptiste Lespiau; Nithya Attaluri; and Kate Olszewska},
  year = {2024},
  doi = {10.48550/arxiv.2409.12640},
  eprint = {2409.12640},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2409.12640}
}
```

- [← previous MambaByte: Token-free Selective State Space Model](https://md.research.elamur.ai/free/en/papers/wang-2024-mambabyte-token-free-selective-state/)
- [next → On the Diversity of Synthetic Data and its Impact on Training Large Language Models](https://md.research.elamur.ai/free/en/papers/chen-2024-diversity-synthetic-data-impact-training/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/9/)

## References

30 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. R. Agarwal, A. Singh, L. M. Zhang, B. Bohnet, L. Rosias, S. Chan, B. Zhang, A. Anand, Z. Abbas, A. Nova, J. D. Co-Reyes, E. Chu, F. Behbahani, A. Faust, and H. Larochelle. [Many-shot in-context learning](https://md.research.elamur.ai/free/en/papers/agarwal-2024-many-shot-context-learning/), 2024. URL https://arxiv.org/abs/2404.11018 . [↗](https://arxiv.org/abs/2404.11018)
2. M. Akter, N. Bansal, and S. K. Karmaker. Revisiting automatic evaluation of extractive summarization task: Can we do better than ROUGE? In S. Muresan, P. Nakov, and A. Villavicencio, editors, Findings of the Association for Computational Linguistics: ACL 2022, pages 1547–1560, Dublin, Ireland, May 2… [find](https://scholar.google.com/scholar?q=M%2E+Akter%2C+N%2E+Bansal%2C+and+S%2E+K%2E+Karmaker%2E+Revisiting+automatic+evaluation+of+extractive+summarization+task%3A+Can+we+do+better+than+ROUGE%3F+In+S%2E+Muresan%2C+P%2E+Nakov%2C+and+A%2E+Villavicencio%2C+editors%2C+Findings)
3. R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen, E. Chu, J. H. Clark, L. E. Shafey, Y. Huang, K. Meier-Hellstern, G. Mishra, E. Moreira, M. Omernick, K. Robinson, S. Ruder, Y. Tay, K. Xiao, Y. Xu, Y. Zhang, G. H. Abrego, J. Ahn, J. Austin,… — [PaLM 2 Technical Report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/)
4. Anthropic. Model Card and Evaluations for Claude Models, 2023. [find](https://scholar.google.com/scholar?q=Anthropic%2E+Model+Card+and+Evaluations+for+Claude+Models%2C+2023%2E)
5. Anthropic. The Claude 3 Model Family: Opus, Sonnet, Haiku, 2024a. [find](https://scholar.google.com/scholar?q=Anthropic%2E+The+Claude+3+Model+Family%3A+Opus%2C+Sonnet%2C+Haiku%2C+2024a%2E)
6. Anthropic. Claude 3.5 Sonnet Model Card Addendum, 2024b. [find](https://scholar.google.com/scholar?q=Anthropic%2E+Claude+3%2E5+Sonnet+Model+Card+Addendum%2C+2024b%2E)
7. S. Arora, S. Eyuboglu, A. Timalsina, I. Johnson, M. Poli, J. Zou, A. Rudra, and C. Ré. [Zoology: Measuring and improving recall in efficient language models](https://md.research.elamur.ai/free/en/papers/arora-2023-zoology-measuring-improving-recall-efficient/), 2023. URL [https://arxiv.org/](https://arxiv.org/) abs/2312.04927 . [↗](https://arxiv.org/)
8. Y. Bai, X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang, Z. Du, X. Liu, A. Zeng, L. Hou, et al. [Longbench: A bilingual, multitask benchmark for long context understanding](https://md.research.elamur.ai/free/en/papers/bai-2023-longbench-bilingual-multitask-benchmark/). arXiv preprint arXiv:2308.14508, 2023.
9. B. Bohnet, A. Nova, A. T. Parisi, K. Swersky, K. Goshvadi, H. Dai, D. Schuurmans, N. Fiedel, and H. Sedghi. Exploring and benchmarking the planning capabilities of large language models, 2024a. URL https://arxiv.org/abs/2406.13094 . — [arXiv:2406.13094](https://arxiv.org/abs/2406.13094) [↗](https://arxiv.org/abs/2406.13094)
10. B. Bohnet, K. Swersky, R. Liu, P. Awasthi, A. Nova, J. Snaider, H. Sedghi, A. T. Parisi, M. Collins, A. Lazaridou, O. Firat, and N. Fiedel. Long-span question-answering: Automatic question generation and qa-system ranking via side-by-side evaluation, 2024b. URL https://arxiv.org/abs/2406. 00179 . [↗](https://arxiv.org/abs/2406)
11. Y. Chang, K. Lo, T. Goyal, and M. Iyyer. [Booookscore: A systematic exploration of book-length summarization in the era of LLMs](https://md.research.elamur.ai/free/en/papers/chang-2023-booookscore-systematic-exploration-book-length/). In The Twelfth International Conference on Learning Representations, 2024. URL [https://openreview.net/forum?id=7Ttk3RzDeu](https://openreview.net/forum?id=7Ttk3RzDeu) . [↗](https://openreview.net/forum?id=7Ttk3RzDeu)
12. A. Cohan and N. Goharian. Revisiting summarization evaluation for scientific articles. In N. Calzolari, K. Choukri, T. Declerck, S. Goggi, M. Grobelnik, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, and S. Piperidis, editors, Proceedings of the Tenth International Conference on Language Res… [find](https://scholar.google.com/scholar?q=A%2E+Cohan+and+N%2E+Goharian%2E+Revisiting+summarization+evaluation+for+scientific+articles%2E+In+N%2E+Calzolari%2C+K%2E+Choukri%2C+T%2E+Declerck%2C+S%2E+Goggi%2C+M%2E+Grobelnik%2C+B%2E+Maegaard%2C+J%2E+Mariani%2C+H%2E+Mazo%2C+A%2E+Moreno%2C+J%2E)
13. O. Goldman, A. Jacovi, A. Slobodkin, A. Maimon, I. Dagan, and R. Tsarfaty. Is it really long context if all you need is retrieval? towards genuinely difficult long context nlp, 2024. URL https://arxiv.org/abs/2407.00402 . [↗](https://arxiv.org/abs/2407.00402)
14. Google, P. Georgiev, V. I. Lei, R. Burnell, L. Bai, A. Gulati, G. Tanzer, D. Vincent, Z. Pan, S. Wang, S. Mariooryad, Y. Ding, X. Geng, F. Alcober, R. Frostig, M. Omernick, L. Walker, C. Paduraru, C. Sorokin, A. Tacchetti, C. Gaffney, S. Daruki, O. Sercinoglu, Z. Gleicher, J. Love, P. Voigtlaender,… — [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/)
15. C.-P. Hsieh, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, Y. Zhang, and B. Ginsburg. Ruler: What’s the real context size of your long-context language models?, 2024. URL — [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://md.research.elamur.ai/free/en/papers/hsieh-2024-ruler-s-real-context-size/) [https://arxiv.org/](https://arxiv.org/) abs/2404.06654 . [↗](https://arxiv.org/)
16. URL [https://github.com/gkamradt/LLMTest_NeedleInAHaystack/](https://github.com/gkamradt/LLMTest_NeedleInAHaystack/) blob/main/README.md . M. Karpinska, K. Thai, K. Lo, T. Goyal, and M. Iyyer. One thousand and one pairs: A "novel" challenge for long-context language models, 2024. URL https://arxiv.org/abs/2406.16264 . [↗](https://github.com/gkamradt/LLMTest_NeedleInAHaystack/)
17. D. Kaushik and Z. C. Lipton. [How much reading does reading comprehension require? a critical investigation of popular benchmarks](https://md.research.elamur.ai/free/en/papers/kaushik-2018-how-much-reading-does-reading/). In E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5010–5015,…
18. Y. Kim, Y. Chang, M. Karpinska, A. Garimella, V. Manjunatha, K. Lo, T. Goyal, and M. Iyyer. Fables: Evaluating faithfulness and content selection in book-length summarization, 2024. URL https://arxiv.org/abs/2404.01261 . — [arXiv:2404.01261](https://arxiv.org/abs/2404.01261) [↗](https://arxiv.org/abs/2404.01261)
19. T. Kočiskỳ, J. Schwarz, P. Blunsom, C. Dyer, K. M. Hermann, G. Melis, and E. Grefenstette. [The narrativeqa reading comprehension challenge](https://md.research.elamur.ai/free/en/papers/kocisky-2018-narrativeqa-reading-comprehension-challenge/). Transactions of the Association for Computational Linguistics, 6:317–328, 2018.
20. Y. Kuratov, A. Bulatov, P. Anokhin, I. Rodkin, D. Sorokin, A. Sorokin, and M. Burtsev. [Babilong: Testing the limits of llms with long context reasoning-in-a-haystack](https://md.research.elamur.ai/free/en/papers/kuratov-2024-babilong-testing-limits-llms-long/), 2024. URL [https://arxiv.org/](https://arxiv.org/) abs/2406.10149 . [↗](https://arxiv.org/)
21. J. Lee, A. Chen, Z. Dai, D. Dua, D. S. Sachan, M. Boratko, Y. Luan, S. M. R. Arnold, V. Perot, S. Dalmia, H. Hu, X. Lin, P. Pasupat, A. Amini, J. R. Cole, S. Riedel, I. Naim, M.-W. Chang, and K. Guu. [Can long-context language models subsume retrieval, rag, sql, and more?](https://md.research.elamur.ai/free/en/papers/lee-2024-long-context-language-models-subsume/), 2024. URL [https://arxiv.org…](https://arxiv.org…) [↗](https://arxiv.org…)
22. M. Levy, A. Jacoby, and Y. Goldberg. [Same task, more tokens: the impact of input length on the reasoning performance of large language models](https://md.research.elamur.ai/free/en/papers/levy-2024-same-task-more-tokens-impact/). arXiv preprint arXiv:2402.14848, 2024.
23. M. Li, S. Zhang, Y. Liu, and K. Chen. Needlebench: Can llms do retrieval and reasoning in 1 million context window?, 2024. URL https://arxiv.org/abs/2407.11963 . — [arXiv:2407.11963](https://arxiv.org/abs/2407.11963) [↗](https://arxiv.org/abs/2407.11963)
24. Magic. Hashhop: Long context evaluation. [https://github.com/magicproduct/hash-hop](https://github.com/magicproduct/hash-hop) , 2024. [↗](https://github.com/magicproduct/hash-hop)
25. M. Nasr, N. Carlini, J. Hayase, M. Jagielski, A. F. Cooper, D. Ippolito, C. A. Choquette-Choo, E. Wallace, F. Tramèr, and K. Lee. [Scalable extraction of training data from (production) language models](https://md.research.elamur.ai/free/en/papers/nasr-2023-scalable-extraction-training-data-production/). arXiv preprint arXiv:2311.17035, 2023.
26. OpenAI. [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/), Mar. 2023. URL https://arxiv.org/abs/2303.08774 . [↗](https://arxiv.org/abs/2303.08774)
27. OpenAI. [GPT-4o System Card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/), 2024.
28. Y. Tay, M. Dehghani, S. Abnar, Y. Shen, D. Bahri, P. Pham, J. Rao, L. Yang, S. Ruder, and D. Metzler. [Long range arena: A benchmark for efficient transformers](https://md.research.elamur.ai/free/en/papers/tay-2020-long-range-arena-benchmark-efficient/). arXiv preprint arXiv:2011.04006, 2020.
29. Wikipedia. Blind men and an elephant — Wikipedia, the free encyclopedia. [http://en.wikipedia.org/w/index.php?title=Blind%20men%20and%20an%20elephant&oldid=1242776766](http://en.wikipedia.org/w/index.php?title=Blind%20men%20and%20an%20elephant&oldid=1242776766) , 2024. [Online; accessed 18-September-2024]. [↗](http://en.wikipedia.org/w/index.php?title=Blind%20men%20and%20an%20elephant&oldid=1242776766)
30. X. Zhang, Y. Chen, S. Hu, Z. Xu, J. Chen, M. K. Hao, X. Han, Z. L. Thai, S. Wang, Z. Liu, et al. ∞-bench: Extending long context evaluation beyond 100k tokens. arXiv preprint arXiv:2402.13718, 2024. — [arXiv:2402.13718](https://arxiv.org/abs/2402.13718) [find](https://scholar.google.com/scholar?q=X%2E+Zhang%2C+Y%2E+Chen%2C+S%2E+Hu%2C+Z%2E+Xu%2C+J%2E+Chen%2C+M%2E+K%2E+Hao%2C+X%2E+Han%2C+Z%2E+L%2E+Thai%2C+S%2E+Wang%2C+Z%2E+Liu%2C+et+al%2E+%E2%88%9E%2Dbench%3A+Extending+long+context+evaluation+beyond+100k+tokens%2E+arXiv+preprint+arXiv%3A2402%2E13718%2C+2024%2E+%E2%80%94)

*Updated 2 November 2026*