---
title: "Dolomites: Domain-Specific Long-Form Methodical Tasks (2025)"
description: "Professionals such as clinicians, lawyers and teachers often produce structured documents (care plans… 60 references, cited by 9 papers here."
authors: ["Chaitanya Malaviya", "Priyanka Agrawal", "Kuzman Ganchev", "Pranesh Srinivasan", "Fantine Huot", "Jonathan Berant", "Mark Yatskar", "Dipanjan Das", "Mirella Lapata", "Chris Alberti"]
year: 2025
venue: "Transactions of the Association for Computational Linguistics"
source: "https://aclanthology.org/2025.tacl-1.1/"
doi: "10.1162/tacl_a_00727"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/malaviya-2025-dolomites-domain-specific-long-form/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/malaviya-2025-dolomites-domain-specific-long-form/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/7/) / Dolomites: Domain-Specific Long-Form Methodical Tasks

# Dolomites: Domain-Specific Long-Form Methodical Tasks

**Authors:** [Chaitanya Malaviya](https://md.research.elamur.ai/free/en/authors/chaitanya-malaviya/), [Priyanka Agrawal](https://md.research.elamur.ai/free/en/authors/priyanka-agrawal/), [Kuzman Ganchev](https://md.research.elamur.ai/free/en/authors/kuzman-ganchev/), Pranesh Srinivasan, [Fantine Huot](https://md.research.elamur.ai/free/en/authors/fantine-huot/), [Jonathan Berant](https://md.research.elamur.ai/free/en/authors/jonathan-berant/), [Mark Yatskar](https://md.research.elamur.ai/free/en/authors/mark-yatskar/), [Dipanjan Das](https://md.research.elamur.ai/free/en/authors/dipanjan-das/), [Mirella Lapata](https://md.research.elamur.ai/free/en/authors/mirella-lapata/) and [Chris Alberti](https://md.research.elamur.ai/free/en/authors/chris-alberti/) · **Year:** 2025 · **Venue:** [Transactions of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/tacl/) ([Transactions of the Association for Computational Linguistics (TACL) 2025](https://md.research.elamur.ai/free/en/venues/tacl/2025/)) · **Publisher:** [The MIT Press](https://md.research.elamur.ai/free/en/publishers/the-mit-press/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.tacl-1.1/) · [PDF](https://aclanthology.org/2025.tacl-1.1.pdf) · [DOI 10.1162/tacl\_a\_00727](https://doi.org/10.1162/tacl_a_00727) · [OpenAlex](https://openalex.org/W4406237254)

**Citations:** 3 citations (OpenAlex, 2026-10-14) · [9](https://md.research.elamur.ai/free/en/papers/malaviya-2025-dolomites-domain-specific-long-form/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Professionals such as clinicians, lawyers and teachers often produce structured documents (care plans, petitions, lesson plans) by following a fairly fixed routine. The authors call these "methodical tasks." No existing benchmark captured them, so they built DoLoMiTes. They recruited 266 qualified experts via Prolific, covering 25 fields, and asked each to describe routine writing tasks in a shared template: objective, procedure, input sections, output sections and notes. After manual filtering, 519 task descriptions remained. To make the tasks testable, the authors first retrieved web documents and had a language model draft example instances from them. The contributing experts then substantially corrected these drafts, yielding 1,857 input–output examples. Models are evaluated as long-form generators: given a task description and an input, they must write the output. Current LMs leave considerable room for improvement, since the tasks demand domain knowledge and multi-step reasoning. Standard automatic metrics also proved weak at judging such outputs, while LM-based judges matched human preferences best. The benchmark grounds LM evaluation in realistic expert use cases.

## Key points
- **Scale:** 519 tasks from 266 experts in 25 fields. Law contributes the most tasks (38). Experts needed formal training in their field plus at least 3 years of work experience.
- **Task structure:** On average, a task has about 2.78 input sections and about 2.82 output sections. Tasks are text-only, recurring and time-consuming, taking from roughly half an hour to a few days.
- **Independent validation:** A separate group of experts reviewed the tasks. About 76% were judged likely to be performed regularly by experts, and about 63% of reviewers said an AI writing assistant would help with them.
- **Examples:** 1,857 instances were created through retrieval plus LM generation, with up to 10 drafts per task post-edited by the experts.
- **Evaluation:** Pairwise judgments by Claude-3 Opus agreed with human labels 67% of the time with ties and 77% without ties, outperforming other metrics. The benchmark is English-only and has one reference output per example.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 430 works · 424,363 citations
- [University of Pennsylvania](https://md.research.elamur.ai/free/en/universities/university-of-pennsylvania/) — University, Philadelphia, United States · 221 works · 63,234 citations
- [Google DeepMind](https://md.research.elamur.ai/free/en/companies/google-deepmind/) — Company, London, United Kingdom · 56 works · 78,880 citations
- [Google (United Kingdom)](https://md.research.elamur.ai/free/en/companies/google-united-kingdom/) — Company, London, United Kingdom · 10 works · 1,009 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

## Cite

Cite as: Chaitanya Malaviya, Priyanka Agrawal, Kuzman Ganchev, Pranesh Srinivasan, Fantine Huot, Jonathan Berant, Mark Yatskar, Dipanjan Das, Mirella Lapata and Chris Alberti. 2025. Dolomites: Domain-Specific Long-Form Methodical Tasks. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{malaviya2025dolomites,
  title = {{Dolomites: Domain-Specific Long-Form Methodical Tasks}},
  author = {Chaitanya Malaviya, Priyanka Agrawal, Kuzman Ganchev, Pranesh Srinivasan, Fantine Huot, Jonathan Berant, Mark Yatskar, Dipanjan Das, Mirella Lapata and Chris Alberti},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.1162/tacl\_a\_00727},
  url = {https://doi.org/10.1162/tacl_a_00727}
}
```

- [← previous Decoding Decoded: Understanding Hyperparameter Effects in Open-Ended Text Generation](https://md.research.elamur.ai/free/en/papers/arias-2025-decoding-decoded-understanding-hyperparameter-effects/)
- [next → Empowering GraphRAG with Knowledge Filtering and Integration](https://md.research.elamur.ai/free/en/papers/guo-2025-empowering-graphrag-knowledge-filtering-integration/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/7/)

## References

60 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anthropic. The claude 3 model family: Opus, sonnet, haiku. [Accessed on March 4, 2024]. [find](https://scholar.google.com/scholar?q=Anthropic%2E+The+claude+3+model+family%3A+Opus%2C+sonnet%2C+haiku%2E+%5BAccessed+on+March+4%2C+2024%5D%2E)
2. Abeba Birhane, Atoosa Kasirzadeh, David Leslie, and Sandra Wachter. 2023. Science in the age of large language models. Nature Reviews Physics, 5(5):277–280. https://doi.org/10.1038/s42254-023-00581-4 — [doi:10.1038/s42254-023-00581-4](https://doi.org/10.1038/s42254-023-00581-4) [↗](https://doi.org/10.1038/s42254-023-00581-4)
3. Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language P… [find](https://scholar.google.com/scholar?q=Jesse+Dodge%2C+Maarten+Sap%2C+Ana+Marasovic%CC%81%2C+William+Agnew%2C+Gabriel+Ilharco%2C+Dirk+Groeneveld%2C+Margaret+Mitchell%2C+and+Matt+Gardner%2E+2021%2E+Documenting+large+webtext+corpora%3A+A+case+study+on+the+colossal+cl)
4. Alex Calderwood, Vivian Qiu, Katy Ilonka Gero, and Lydia B. Chilton. 2020. How novelists use generative language models: An exploratory user study. In HAI-GEN+ user2agent@ IUI. [find](https://scholar.google.com/scholar?q=Alex+Calderwood%2C+Vivian+Qiu%2C+Katy+Ilonka+Gero%2C+and+Lydia+B%2E+Chilton%2E+2020%2E+How+novelists+use+generative+language+models%3A+An+exploratory+user+study%2E+In+HAI%2DGEN%2B+user2agent%40+IUI%2E)
5. Cheng-Han Chiang and Hung-yi Lee. 2023. Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15607–15631, Toronto, Canada. Association for Computational Linguistics.… [find](https://scholar.google.com/scholar?q=Cheng%2DHan+Chiang+and+Hung%2Dyi+Lee%2E+2023%2E+Can+large+language+models+be+an+alternative+to+human+evaluations%3F+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Vo)
6. Yanai Elazar, Akshita Bhagia, Ian Helgi Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Evan Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, Hannaneh Hajishirzi, Noah A. Smith, and Jesse Dodge. 2024. What’s in my big data? In The Twelfth International Conference on Learning R… [find](https://scholar.google.com/scholar?q=Yanai+Elazar%2C+Akshita+Bhagia%2C+Ian+Helgi+Magnusson%2C+Abhilasha+Ravichander%2C+Dustin+Schwenk%2C+Alane+Suhr%2C+Evan+Pete+Walsh%2C+Dirk+Groeneveld%2C+Luca+Soldaini%2C+Sameer+Singh%2C+Hannaneh+Hajishirzi%2C+Noah+A%2E+Smith%2C)
7. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). Journal of Machine Learning Research, 24(240):1–113.
8. Tyna Eloundou, Sam Manning, Pamela Mishkin, and Daniel Rock. 2024. Gpts are gpts: Labor market impact potential of llms. Science, 384(6702):1306–1308. https://doi.org/10.1126/science.adj0998, PubMed: 38900883 — [doi:10.1126/science.adj0998](https://doi.org/10.1126/science.adj0998) [↗](https://doi.org/10.1126/science.adj0998)
9. Cohere. 2024. Introducing Command R+: A scalable LLM built for business. [Accessed on April 16, 2024]. [find](https://scholar.google.com/scholar?q=Cohere%2E+2024%2E+Introducing+Command+R%2B%3A+A+scalable+LLM+built+for+business%2E+%5BAccessed+on+April+16%2C+2024%5D%2E)
10. Douglas C. Engelbart. 2023, Augmenting human intellect: A conceptual framework. In Augmented Education in the Global Age, pages 13–29. Routledge. [find](https://scholar.google.com/scholar?q=Douglas+C%2E+Engelbart%2E+2023%2C+Augmenting+human+intellect%3A+A+conceptual+framework%2E+In+Augmented+Education+in+the+Global+Age%2C+pages+13%E2%80%9329%2E+Routledge%2E)
11. Fabrizio Dell’Acqua, Edward McFowland, Ethan R. Mollick, Hila Lifshitz-Assaf, Katherine Kellogg, Saran Rajendran, Lisa Krayer, François Candelon, and Karim R. Lakhani. 2023. Navigating the jagged technological frontier: Field experimental evidence of the effects of ai on knowledge worker productivi… — [doi:10.2139/ssrn.4573321](https://doi.org/10.2139/ssrn.4573321) [find](https://scholar.google.com/scholar?q=Fabrizio+Dell%E2%80%99Acqua%2C+Edward+McFowland%2C+Ethan+R%2E+Mollick%2C+Hila+Lifshitz%2DAssaf%2C+Katherine+Kellogg%2C+Saran+Rajendran%2C+Lisa+Krayer%2C+Franc%CC%A7ois+Candelon%2C+and+Karim+R%2E+Lakhani%2E+2023%2E+Navigating+the+jagged+tec)
12. Constanza Fierro, Reinald Kim Amplayo, Fantine Huot, Nicola De Cao, Joshua Maynez, Shashi Narayan, and Mirella Lapata. 2024. Learning to plan and generate text with citations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11… [find](https://scholar.google.com/scholar?q=Constanza+Fierro%2C+Reinald+Kim+Amplayo%2C+Fantine+Huot%2C+Nicola+De+Cao%2C+Joshua+Maynez%2C+Shashi+Narayan%2C+and+Mirella+Lapata%2E+2024%2E+Learning+to+plan+and+generate+text+with+citations%2E+In+Proceedings+of+the+62)
13. Hajishirzi. 2024. OLMo: Accelerating the science of language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15789–15809, Bangkok, Thailand. Association for Computational Linguistics. https://doi.org/10 .18653/v1/2024.… [find](https://scholar.google.com/scholar?q=Hajishirzi%2E+2024%2E+OLMo%3A+Accelerating+the+science+of+language+models%2E+In+Proceedings+of+the+62nd+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+Long+Papers%29%2C+pages+15789%E2%80%93158)
14. Scott L. Fleming, Alejandro Lozano, William J. Haberkorn, Jenelle A. Jindal, Eduardo Reis, Rahul Thapa, Louis Blankemeier, Julian Z. Genkins, Ethan Steinberg, Ashwin Nayak, Birju Patel, Chia-Chun Chiang, Alison Callahan, Zepeng Huo, Sergios Gatidis, Scott Adams, Oluseyi Fayanju, Shreya J. Shah, Thom… [find](https://scholar.google.com/scholar?q=Scott+L%2E+Fleming%2C+Alejandro+Lozano%2C+William+J%2E+Haberkorn%2C+Jenelle+A%2E+Jindal%2C+Eduardo+Reis%2C+Rahul+Thapa%2C+Louis+Blankemeier%2C+Julian+Z%2E+Genkins%2C+Ethan+Steinberg%2C+Ashwin+Nayak%2C+Birju+Patel%2C+Chia%2DChun+Chia)
15. Neel Guha, Julian Nyarko, Daniel Ho, Christopher Ré, Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory Dickinson, Haggai Porat, Jason Hegland, Jessica Wu, Joe Nudell, Joel Ni… — [LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models](https://md.research.elamur.ai/free/en/papers/guha-2023-legalbench-collaboratively-built-benchmark-measuring/)
16. Jens Frankenreiter and Julian Nyarko. 2022. Natural language processing in legal tech. Legal Tech and the Future of Civil Justice (David Engstrom ed.) Forthcoming. https://doi.org/10.2139/ssrn.4027030 — [doi:10.2139/ssrn.4027030](https://doi.org/10.2139/ssrn.4027030) [↗](https://doi.org/10.2139/ssrn.4027030)
17. Deep Ganguli, Nicholas Schiefer, Marina Favaro, and Jack Clark. 2023. Challenges in evaluating AI systems. [Accessed on October 4, 2023]. [find](https://scholar.google.com/scholar?q=Deep+Ganguli%2C+Nicholas+Schiefer%2C+Marina+Favaro%2C+and+Jack+Clark%2E+2023%2E+Challenges+in+evaluating+AI+systems%2E+%5BAccessed+on+October+4%2C+2023%5D%2E)
18. Hiroaki Hayashi, Prashant Budania, Peng Wang, Chris Ackerson, Raj Neervannan, and Graham Neubig. 2021. [Wikiasp: A dataset for multi-domain aspect-based summarization](https://md.research.elamur.ai/free/en/papers/hayashi-2021-wikiasp-dataset-multi-domain-aspect/). Transactions of the Association for Computational Linguistics, 9:211–225. https://doi.org/10.1162/tacl_a_00362 [↗](https://doi.org/10.1162/tacl_a_00362)
19. Katy Ilonka Gero, Vivian Liu, and Lydia Chilton. 2022. Sparks: Inspiration for science writing using language models. In Proceedings of the 2022 ACM Designing Interactive Systems Conference, pages 1002–1019. https://doi.org/10.1145/3532106.3533533 [↗](https://doi.org/10.1145/3532106.3533533)
20. Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill… — [Measuring Massive Multitask Language Understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/)
21. Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022. [TRUE: Re-evaluating factual consistency evaluation](https://md.research.elamur.ai/free/en/papers/honovich-2022-true-re-evaluating-factual-consistency/). In Proceedings of the 2022 Conference of the North American Chapter of the Associa…
22. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. [Natural…](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)
23. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Emma+Bou+Hanna%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guill)
24. Mina Lee, Percy Liang, and Qian Yang. 2022. Coauthor: Designing a human-ai collaborative writing dataset for exploring language model capabilities. In Proceedings of the 2022 CHI conference on human factors in computing systems, pages 1–19. https://doi.org/10 .1145/3491102.3502030 [find](https://scholar.google.com/scholar?q=Mina+Lee%2C+Percy+Liang%2C+and+Qian+Yang%2E+2022%2E+Coauthor%3A+Designing+a+human%2Dai+collaborative+writing+dataset+for+exploring+language+model+capabilities%2E+In+Proceedings+of+the+2022+CHI+conference+on+human+f)
25. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. [What disease does this patient have? A large-scale open domain question answering dataset from medical exams](https://md.research.elamur.ai/free/en/papers/jin-2020-disease-this-patient-have-large/). Applied Sciences, 11(14). https://doi.org/10 .3390/app11146421
26. Peter Lee, Sebastien Bubeck, and Joseph Petro. 2023. Benefits, limits, and risks of gpt-4 as an ai chatbot for medicine. New England Journal of Medicine, 388(13):1233–1239. https://doi.org/10.1056/NEJMsr2214184, PubMed: 36988602 — [doi:10.1056/nejmsr2214184](https://doi.org/10.1056/nejmsr2214184) [↗](https://doi.org/10.1056/NEJMsr2214184)
27. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. [Pubmedqa: A dataset for biomedical research question answering](https://md.research.elamur.ai/free/en/papers/jin-2019-pubmedqa-dataset-biomedical-research-question/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proce…
28. Zhuoyan Li, Chen Liang, Jing Peng, and Ming Yin. 2024. The value, benefits, and concerns of generative ai-powered assistance in writing. In Proceedings of the CHI Conference on Human Factors in Computing Systems, CHI ’24, New York, NY, USA. Association for Computing Machinery. https://doi.org/10 .11… [find](https://scholar.google.com/scholar?q=Zhuoyan+Li%2C+Chen+Liang%2C+Jing+Peng%2C+and+Ming+Yin%2E+2024%2E+The+value%2C+benefits%2C+and+concerns+of+generative+ai%2Dpowered+assistance+in+writing%2E+In+Proceedings+of+the+CHI+Conference+on+Human+Factors+in+Comput)
29. Bill Yuchen Lin, Khyathi Chandu, Faeze Brahman, Yuntian Deng, Abhilasha Ravichander, Valentina Pyatkin, Ronan Le Bras, and Yejin Choi. 2024. [Wildbench: Benchmarking llms with challenging tasks from real users in the wild](https://md.research.elamur.ai/free/en/papers/lin-2024-wildbench-benchmarking-llms-challenging-tasks/).
30. J. Peter Kincaid, Robert P. Fishburne Jr., Richard L. Rogers, and Brad S. Chissom. 1975. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. https://doi.org/10.21236/ADA006655 [↗](https://doi.org/10.21236/ADA006655)
31. Chin-Yew Lin. 2004. [ROUGE: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
32. Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021. [Hurdles to progress in long-form question answering](https://md.research.elamur.ai/free/en/papers/krishna-2021-hurdles-progress-long-form-question/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4940–4957, Online. Association for Computat…
33. Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. [How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation](https://md.research.elamur.ai/free/en/papers/liu-2016-how-not-evaluate-your-dialogue/). In Proceedings of the 2016
34. Conference on Empirical Methods in Natural Language Processing, pages 2122[–2132, Austin, Texas](https://md.research.elamur.ai/free/en/papers/novikova-2017-why-we-need-new-evaluation/). Association for Computational Linguistics. https://doi.org/10.18653/v1/D16-1230 NLG. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2241–2252, Copenhagen… [↗](https://doi.org/10.18653/v1/D16-1230)
35. Chaitanya Malaviya, Subin Lee, Sihao Chen, Elizabeth Sieber, Mark Yatskar, and Dan Roth. 2024. [ExpertQA: Expert-curated questions and attributed answers](https://md.research.elamur.ai/free/en/papers/malaviya-2024-expertqa-expert-curated-questions-attributed/). In 2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics. https://doi.org/10 .18653/v1/2024.naacl-…
36. Shakked Noy and Whitney Zhang. 2023. Experimental evidence on the productivity effects of generative artificial intelligence. Science, 381(6654):187–192. https://doi.org/10 .1126/science.adh2586, PubMed: 37440646 [find](https://scholar.google.com/scholar?q=Shakked+Noy+and+Whitney+Zhang%2E+2023%2E+Experimental+evidence+on+the+productivity+effects+of+generative+artificial+intelligence%2E+Science%2C+381%286654%29%3A187%E2%80%93192%2E+https%3A%2F%2Fdoi%2Eorg%2F10+%2E1126%2Fscience%2Eadh2586%2C+PubM)
37. OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). ArXiv, abs/2303.08774v6.
38. James Manyika and Sissie Hsiao. 2023. An overview of bard: An early experiment with generative ai. AI. Google Static Documents, 2. [find](https://scholar.google.com/scholar?q=James+Manyika+and+Sissie+Hsiao%2E+2023%2E+An+overview+of+bard%3A+An+early+experiment+with+generative+ai%2E+AI%2E+Google+Static+Documents%2C+2%2E)
39. Siru Ouyang, Shuohang Wang, Yang Liu, Ming Zhong, Yizhu Jiao, Dan Iter, Reid Pryzant, Chenguang Zhu, Heng Ji, and Jiawei Han. 2023. The shifted and the overlooked: A task-oriented investigation of user-GPT interactions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language P… [find](https://scholar.google.com/scholar?q=Siru+Ouyang%2C+Shuohang+Wang%2C+Yang+Liu%2C+Ming+Zhong%2C+Yizhu+Jiao%2C+Dan+Iter%2C+Reid+Pryzant%2C+Chenguang+Zhu%2C+Heng+Ji%2C+and+Jiawei+Han%2E+2023%2E+The+shifted+and+the+overlooked%3A+A+task%2Doriented+investigation+of+use)
40. Mistral. 2024. Au large. Accessed on March 26, 2024. [find](https://scholar.google.com/scholar?q=Mistral%2E+2024%2E+Au+large%2E+Accessed+on+March+26%2C+2024%2E)
41. Ethan R. Mollick and Lilach Mollick. 2023. Using ai to implement effective teaching strategies in classrooms: Five strategies, including prompts. Including Prompts (March 17, 2023). https://doi.org/10.2139/ssrn.4391243 — [doi:10.2139/ssrn.4391243](https://doi.org/10.2139/ssrn.4391243) [↗](https://doi.org/10.2139/ssrn.4391243)
42. Shashi Narayan, Joshua Maynez, Reinald Kim Amplayo, Kuzman Ganchev, Annie Louis, Fantine Huot, Anders Sandholm, Dipanjan Das, and Mirella Lapata. 2023. [Conditional generation with a question-answering blueprint](https://md.research.elamur.ai/free/en/papers/narayan-2023-conditional-generation-question-answering-blueprint/). Transactions of the Association for Computational Linguistics, 11:974–996. [https://doi.…](https://doi.…) [↗](https://doi.…)
43. Brian Owens. 2023. How nature readers are using chatgpt. Nature. https://doi.org/10 .1038/d41586-023-00500-8 [find](https://scholar.google.com/scholar?q=Brian+Owens%2E+2023%2E+How+nature+readers+are+using+chatgpt%2E+Nature%2E+https%3A%2F%2Fdoi%2Eorg%2F10+%2E1038%2Fd41586%2D023%2D00500%2D8)
44. Anusri Pampari, Preethi Raghavan, Jennifer Liang, and Jian Peng. 2018. [emrQA: A large corpus for question answering on electronic medical records](https://md.research.elamur.ai/free/en/papers/pampari-2018-emrqa-large-corpus-question-answering/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2357–2368, Brussels, Belgium. Association for Computatio…
45. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. [Ms marco: A human generated machine reading comprehension dataset](https://md.research.elamur.ai/free/en/papers/bajaj-2016-ms-marco-human-generated/). In CoCo@NIPS, volume 1773 of CEUR Workshop Proceedings. CEUR-WS.org.
46. Arjun Panickssery, Samuel R. Bowman, and Shi Feng. 2024. Llm evaluators recognize and favor their own generations. arXiv preprint arXiv:2404.13076v1. [find](https://scholar.google.com/scholar?q=Arjun+Panickssery%2C+Samuel+R%2E+Bowman%2C+and+Shi+Feng%2E+2024%2E+Llm+evaluators+recognize+and+favor+their+own+generations%2E+arXiv+preprint+arXiv%3A2404%2E13076v1%2E)
47. Joel Niklaus, Veton Matoshi, Pooja Rani, Andrea Galassi, Matthias Stürmer, and Ilias Chalkidis. 2023. [LEXTREME: A multi-lingual and multi-task benchmark for the legal domain](https://md.research.elamur.ai/free/en/papers/niklaus-2023-lextreme-multi-lingual-multi-task/). In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3016–3054, Singapore. Association for Comput…
48. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of Machine Learning Research, 21(140):1–67.
49. Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017. [Why we need new evaluation metrics for David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/). Bowman. 2024. GPQA: A graduate-level google-proo…
50. Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind… [find](https://scholar.google.com/scholar?q=Jennifer+Dumas%2C+Yanai+Elazar%2C+Valentin+Hofmann%2C+Ananya+Jha%2C+Sachin+Kumar%2C+Li+Lucy%2C+Xinxi+Lyu%2C+Nathan+Lambert%2C+Ian+Magnusson%2C+Jacob+Morrison%2C+Niklas+Muennighoff%2C+Aakanksha+Naik%2C+Crystal+Nam%2C+Matthew+Pe)
51. David Rolnick, Alan Aspuru-Guzik, Sara Beery, Bistra Dilkina, Priya L. Donti, Marzyeh Ghassemi, Hannah Kerner, Claire Monteleoni, Esther Rolf, Milind Tambe, and Adam White. 2024. Position: Application-driven innovation in machine learning. In Proceedings of the 41st International Conference on Machi… [find](https://scholar.google.com/scholar?q=David+Rolnick%2C+Alan+Aspuru%2DGuzik%2C+Sara+Beery%2C+Bistra+Dilkina%2C+Priya+L%2E+Donti%2C+Marzyeh+Ghassemi%2C+Hannah+Kerner%2C+Claire+Monteleoni%2C+Esther+Rolf%2C+Milind+Tambe%2C+and+Adam+White%2E+2024%2E+Position%3A+Application)
52. Natalie Schluter. 2017. [The limits of automatic summarisation according to ROUGE](https://md.research.elamur.ai/free/en/papers/schluter-2017-limits-automatic-summarisation-according-rouge/). In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 41–45, Valencia, Spain. Association for Computational Linguistics. https://…
53. Gemini Team. 2023. [Gemini: A family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805v4.
54. Gemini Team. 2024. [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/). arXiv preprint arXiv:2403.05530v4.
55. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. [BLEURT: Learning robust metrics for text generation](https://md.research.elamur.ai/free/en/papers/sellam-2020-bleurt-learning-robust-metrics-text/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1 /202…
56. George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R. Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos, et al. 2015. An overview of the bioasq large-scale biomedical semantic indexing and question answer… [find](https://scholar.google.com/scholar?q=George+Tsatsaronis%2C+Georgios+Balikas%2C+Prodromos+Malakasiotis%2C+Ioannis+Partalas%2C+Matthias+Zschunke%2C+Michael+R%2E+Alvers%2C+Dirk+Weissenborn%2C+Anastasia+Krithara%2C+Sergios+Petridis%2C+Dimitris+Polychronopoulos%2C)
57. Chenhui Shen, Liying Cheng, Xuan-Phi Nguyen, Yang You, and Lidong Bing. 2023. [Large language models are not yet human-level evaluators for abstractive summarization](https://md.research.elamur.ai/free/en/papers/shen-2023-large-language-models-are-not/). In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 4215–4233, Singapore. Association for Computational Li…
58. Hanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexin Huang, Ziming Liu, Payal Chandak, Shengchao Liu, Peter Van Katwyk, Andreea Deac, et al. 2023. Scientific discovery in the age of artificial intelligence. Nature, 620(7972):47–60. https://doi.org/10 .1038/s41586-023-06221-2, PubMed: 37532811 [find](https://scholar.google.com/scholar?q=Hanchen+Wang%2C+Tianfan+Fu%2C+Yuanqi+Du%2C+Wenhao+Gao%2C+Kexin+Huang%2C+Ziming+Liu%2C+Payal+Chandak%2C+Shengchao+Liu%2C+Peter+Van+Katwyk%2C+Andreea+Deac%2C+et+al%2E+2023%2E+Scientific+discovery+in+the+age+of+artificial+intel)
59. Zejiang Shen, Kyle Lo, Lauren Yu, Nathan Dahlberg, Margo Schlanger, and Doug Downey. 2022. Multi-lexsum: Real-world summaries of civil rights lawsuits at multiple granularities. Advances in Neural Information Processing Systems, 35:13158–13173. [find](https://scholar.google.com/scholar?q=Zejiang+Shen%2C+Kyle+Lo%2C+Lauren+Yu%2C+Nathan+Dahlberg%2C+Margo+Schlanger%2C+and+Doug+Downey%2E+2022%2E+Multi%2Dlexsum%3A+Real%2Dworld+summaries+of+civil+rights+lawsuits+at+multiple+granularities%2E+Advances+in+Neural+Inf)
60. Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and Zhifang Sui. 2024. Large language models are not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Sc… [find](https://scholar.google.com/scholar?q=Peiyi+Wang%2C+Lei+Li%2C+Liang+Chen%2C+Zefan+Cai%2C+Dawei+Zhu%2C+Binghuai+Lin%2C+Yunbo+Cao%2C+Lingpeng+Kong%2C+Qi+Liu%2C+Tianyu+Liu%2C+and+Zhifang+Sui%2E+2024%2E+Large+language+models+are+not+fair+evaluators%2E+In+Proceedings+o)

*Updated 30 October 2026*