---
title: "Quantifying the Impact of Structured Output Format on Large"
description: "They replace simple metric comparisons with a causal-inference framework built on randomized controlled trials and statistical… 65 references."
authors: ["Han Yuan", "Yue Zhao", "Li Zhang", "Wuqiong Luo", "Zheng Ma"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: EACL 2026"
source: "https://aclanthology.org/2026.findings-eacl.91/"
doi: "10.18653/v1/2026.findings-eacl.91"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yuan-2026-quantifying-impact-structured-output-format/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yuan-2026-quantifying-impact-structured-output-format/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/8/) / Quantifying the Impact of Structured Output Format on Large Language Models through Causal Inference

# Quantifying the Impact of Structured Output Format on Large Language Models through Causal Inference

**Authors:** Han Yuan, [Yue Zhao](https://md.research.elamur.ai/free/en/authors/yue-zhao/), [Li Zhang](https://md.research.elamur.ai/free/en/authors/li-zhang/), Wuqiong Luo and [Zheng Ma](https://md.research.elamur.ai/free/en/authors/zheng-ma/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: EACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-eacl.91/) · [PDF](https://aclanthology.org/2026.findings-eacl.91.pdf) · [DOI 10.18653/v1/2026.findings-eacl.91](https://doi.org/10.18653/v1/2026.findings-eacl.91) · [OpenAlex](https://openalex.org/W7140120480)

**Citations:** 2 citations (OpenAlex, 2026-10-14)

## Summary
Earlier studies disagree on whether forcing LLMs to answer in structured formats helps or hurts output quality. The authors argue these studies tested few tasks, compared prompts that were not equivalent, and relied only on aggregate score differences. They replace simple metric comparisons with a causal-inference framework built on randomized controlled trials and statistical tests. The framework treats instruction, output format and the model's final answer as variables. Under one assumed and two guaranteed constraints, the possible causal graphs reduce to five DAGs, ranging from colliders (with or without m-bias) to full independence. The study covers seven public reasoning benchmarks plus a new one, ELLC. It compares JSON, XML and YAML against plain text using matched templates, with persona paraphrases serving as the instruction intervention. Raw metrics suggest gains, losses or no change depending on the task. The causal analysis instead finds that format usually has no causal effect on GPT-4o's answers. The work matters for regulated industries that need documented impact assessments of pipeline components, and the pipeline can be reused for modules other than output format.

## Key points
- Across 48 scenarios on GPT-4o, 43 show no causal impact of output format. In 3 of the remaining 5, the causal structure is more complex and depends on the specific instructions.
- ELLC (Enhanced Last Letter Concatenation) extends LLC by requiring the extracted letters to be rearranged into a valid English word. GPT-4o scores only 8.2% on its hardest sub-task.
- OpenAI-o3 is consistently robust to format changes. GPT-4.1 behaves much like GPT-4o, and both remain vulnerable on some tasks.
- Small language models are more format-sensitive, with causal effects in 7 of 24 DAGs and success rates sometimes falling below 50%. GPT-oss-20B is the most robust among them.
- With JSON obtained via function calling, independence remains the dominant DAG, and answer quality beats format-restricting prompts on 5 of 8 datasets. The code and notebook are available on GitHub.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

Also on the byline: [American Express](https://ror.org/05rckx884).

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Han Yuan, Yue Zhao, Li Zhang, Wuqiong Luo and Zheng Ma. 2026. Quantifying the Impact of Structured Output Format on Large Language Models through Causal Inference. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yuan2026quantifying,
  title = {{Quantifying the Impact of Structured Output Format on Large Language Models through Causal Inference}},
  author = {Han Yuan, Yue Zhao, Li Zhang, Wuqiong Luo and Zheng Ma},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.91},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.91}
}
```

- [← previous Quantifying and Predicting Disagreement in Graded Human Ratings](https://md.research.elamur.ai/free/en/papers/zhang-2026-quantifying-predicting-disagreement-graded-human/)
- [next → QuCo-RAG: Quantifying Uncertainty from the Pre-training Corpus for Dynamic Retrieval-Augmented Generation](https://md.research.elamur.ai/free/en/papers/min-2026-quco-rag-quantifying-uncertainty-pre/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/8/)

## References

65 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Marah Abdin, Jyoti Aneja, Hany Awadalla, et al. 2024. [Phi-3 technical report: A highly capable language model locally on your phone](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-3-technical-report-highly/). arXiv.
2. Orlando Ayala and Patrice Bechard. 2024. Reducing hallucination in structured outputs via retrieval-augmented generation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics, pages 228– 238. [find](https://scholar.google.com/scholar?q=Orlando+Ayala+and+Patrice+Bechard%2E+2024%2E+Reducing+hallucination+in+structured+outputs+via+retrieval%2Daugmented+generation%2E+In+Proceedings+of+the+2024+Conference+of+the+North+American+Chapter+of+the+Ass)
3. Guangsheng Bao, Hongbo Zhang, Cunxiang Wang, et al. 2025. [How likely do LLMs with CoT mimic human reasoning?](https://md.research.elamur.ai/free/en/papers/bao-2025-how-likely-do-llms-cot/) In Proceedings of the 31st International Conference on Computational Linguistics, pages 7831–7850.
4. Bradley Brown, Jordan Juravsky, Ryan Ehrlich, et al. 2024. [Large language monkeys: Scaling inference compute with repeated sampling](https://md.research.elamur.ai/free/en/papers/brown-2024-large-language-monkeys-scaling-inference/). arXiv.
5. Robin Chan, Katsiaryna Mirylenka, Thomas Gschwind, et al. 2024. Adapting LLMs for structured natural language API integration. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 991–1000. [find](https://scholar.google.com/scholar?q=Robin+Chan%2C+Katsiaryna+Mirylenka%2C+Thomas+Gschwind%2C+et+al%2E+2024%2E+Adapting+LLMs+for+structured+natural+language+API+integration%2E+In+Proceedings+of+the+2024+Conference+on+Empirical+Methods+in+Natural+Lan)
6. Sherman Chann. Non-determinism in gpt-4 is caused by sparse moe. Accessed: 2025-01-01. Available at: [https://152334h.github.io/blog/non-deter](https://152334h.github.io/blog/non-deter) minism-in-gpt-4/. [↗](https://152334h.github.io/blog/non-deter)
7. Jiaao Chen and Diyi Yang. 2023. Controllable conversation generation with conversation structures via diffusion models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 7238– 7251. [find](https://scholar.google.com/scholar?q=Jiaao+Chen+and+Diyi+Yang%2E+2023%2E+Controllable+conversation+generation+with+conversation+structures+via+diffusion+models%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL+2023%2C+pages+72)
8. Jinpeng Chen, Jianxiang He, Huan Li, et al. 2025a. Hierarchical intent-guided optimization with pluggable llm-driven semantics for session-based recommendation. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1655–1665. [find](https://scholar.google.com/scholar?q=Jinpeng+Chen%2C+Jianxiang+He%2C+Huan+Li%2C+et+al%2E+2025a%2E+Hierarchical+intent%2Dguided+optimization+with+pluggable+llm%2Ddriven+semantics+for+session%2Dbased+recommendation%2E+In+Proceedings+of+the+International+ACM)
9. Xinyun Chen, Ryan Andrew Chi, Xuezhi Wang, et al. 2024. Premise order matters in reasoning with large language models. In Proceedings of the International Conference on Machine Learning, pages 6596–6620. [find](https://scholar.google.com/scholar?q=Xinyun+Chen%2C+Ryan+Andrew+Chi%2C+Xuezhi+Wang%2C+et+al%2E+2024%2E+Premise+order+matters+in+reasoning+with+large+language+models%2E+In+Proceedings+of+the+International+Conference+on+Machine+Learning%2C+pages+6596%E2%80%9366)
10. Yingjian Chen, Haoran Liu, Yinhong Liu, et al. 2025b. GraphCheck: Breaking long-term text barriers with extracted knowledge graph-powered fact-checking. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, pages 14976–14995. [find](https://scholar.google.com/scholar?q=Yingjian+Chen%2C+Haoran+Liu%2C+Yinhong+Liu%2C+et+al%2E+2025b%2E+GraphCheck%3A+Breaking+long%2Dterm+text+barriers+with+extracted+knowledge+graph%2Dpowered+fact%2Dchecking%2E+In+Proceedings+of+the+63rd+Annual+Meeting+of+th)
11. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, et al. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv.
12. William G Cochran. 1950. The comparison of percentages in matched samples. Biometrika, 37(3):256– 266. [find](https://scholar.google.com/scholar?q=William+G+Cochran%2E+1950%2E+The+comparison+of+percentages+in+matched+samples%2E+Biometrika%2C+37%283%29%3A256%E2%80%93+266%2E)
13. William G Cochran. 1954. Some methods for strengthening the common χ 2 tests. Biometrics, 10(4):417– 451. [find](https://scholar.google.com/scholar?q=William+G+Cochran%2E+1954%2E+Some+methods+for+strengthening+the+common+%CF%87+2+tests%2E+Biometrics%2C+10%284%29%3A417%E2%80%93+451%2E)
14. Yijiang Dong, Tiancheng Hu, and Nigel Collier. 2024. Can llm be a personalized judge? In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 10126–10141. [find](https://scholar.google.com/scholar?q=Yijiang+Dong%2C+Tiancheng+Hu%2C+and+Nigel+Collier%2E+2024%2E+Can+llm+be+a+personalized+judge%3F+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2024%2C+pages+10126%E2%80%9310141%2E)
15. Ronald Aylmer Fisher. 1919. The correlation between relatives on the supposition of mendelian inheritance. Earth and Environmental Science Transactions of the Royal Society of Edinburgh, 52(2):399–433. [find](https://scholar.google.com/scholar?q=Ronald+Aylmer+Fisher%2E+1919%2E+The+correlation+between+relatives+on+the+supposition+of+mendelian+inheritance%2E+Earth+and+Environmental+Science+Transactions+of+the+Royal+Society+of+Edinburgh%2C+52%282%29%3A399%E2%80%93433)
16. Kanishk Gandhi, Denise H J Lee, Gabriel Grand, et al. 2024. Stream of search (sos): Learning to search in language. In Proceedings of the Conference on Language Modeling. [find](https://scholar.google.com/scholar?q=Kanishk+Gandhi%2C+Denise+H+J+Lee%2C+Gabriel+Grand%2C+et+al%2E+2024%2E+Stream+of+search+%28sos%29%3A+Learning+to+search+in+language%2E+In+Proceedings+of+the+Conference+on+Language+Modeling%2E)
17. Jinglong Gao, Xiao Ding, Bing Qin, and Ting Liu. 2023. Is ChatGPT a good causal reasoner? a comprehensive evaluation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 11111– 11126, Singapore. [find](https://scholar.google.com/scholar?q=Jinglong+Gao%2C+Xiao+Ding%2C+Bing+Qin%2C+and+Ting+Liu%2E+2023%2E+Is+ChatGPT+a+good+causal+reasoner%3F+a+comprehensive+evaluation%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2023%2C+pages+11)
18. Ahmad Ghazal, Tilmann Rabl, Minqing Hu, et al. 2013. Bigbench: towards an industry standard benchmark for big data analytics. In Proceedings of the 2013 [find](https://scholar.google.com/scholar?q=Ahmad+Ghazal%2C+Tilmann+Rabl%2C+Minqing+Hu%2C+et+al%2E+2013%2E+Bigbench%3A+towards+an+industry+standard+benchmark+for+big+data+analytics%2E+In+Proceedings+of+the+2013)
19. ACM SIGMOD International Conference on Management of Data, page 1197–1208. GPT-oss. Introducing gpt-oss. Accessed: 2025-08-05. Available at: [https://openai.com/index/intro](https://openai.com/index/intro) ducing-gpt-oss. [↗](https://openai.com/index/intro)
20. Xinyu Guan, Li Lyna Zhang, Yifei Liu, et al. 2025. [rstar-math: Small LLMs can master math reasoning with self-evolved deep thinking](https://md.research.elamur.ai/free/en/papers/guan-2025-rstar-math-small-llms-can/). In Proceedings of the International Conference on Machine Learning.
21. Biyang Guo, He Wang, Wenyilin Xiao, et al. 2024. Sample design engineering: An empirical study on designing better fine-tuning samples for information extraction with LLMs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 573–594. [find](https://scholar.google.com/scholar?q=Biyang+Guo%2C+He+Wang%2C+Wenyilin+Xiao%2C+et+al%2E+2024%2E+Sample+design+engineering%3A+An+empirical+study+on+designing+better+fine%2Dtuning+samples+for+information+extraction+with+LLMs%2E+In+Proceedings+of+the+2024+)
22. Guangfu Guo, Kai Zhang, Bryan Hoo, et al. 2025. Structured outputs enable general-purpose llms to be medical experts. arXiv. [find](https://scholar.google.com/scholar?q=Guangfu+Guo%2C+Kai+Zhang%2C+Bryan+Hoo%2C+et+al%2E+2025%2E+Structured+outputs+enable+general%2Dpurpose+llms+to+be+medical+experts%2E+arXiv%2E)
23. Bo Hu, Han Yuan, Vlad Pandelea, Wuqiong Luo, Yingzhu Zhao, and Zheng Ma. 2025. Extract, match, and score: An evaluation paradigm for long question-context-answer triplets in financial analysis. In ICLR 2025 Workshop on Advances in Financial AI: Opportunities, Innovations and Responsible AI. [find](https://scholar.google.com/scholar?q=Bo+Hu%2C+Han+Yuan%2C+Vlad+Pandelea%2C+Wuqiong+Luo%2C+Yingzhu+Zhao%2C+and+Zheng+Ma%2E+2025%2E+Extract%2C+match%2C+and+score%3A+An+evaluation+paradigm+for+long+question%2Dcontext%2Danswer+triplets+in+financial+analysis%2E+In+ICL)
24. Tiancheng Hu and Nigel Collier. 2024. [Quantifying the persona effect in llm simulations](https://md.research.elamur.ai/free/en/papers/hu-2024-quantifying-persona-effect-llm-simulations/). In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 10289–10307.
25. Myeongjun Erik Jang and Fran Silavong. 2025. Insta-judge: Aligning judgment bias of llm-as-judge with humans in industry applications. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1158–1172. [find](https://scholar.google.com/scholar?q=Myeongjun+Erik+Jang+and+Fran+Silavong%2E+2025%2E+Insta%2Djudge%3A+Aligning+judgment+bias+of+llm%2Das%2Djudge+with+humans+in+industry+applications%2E+In+Proceedings+of+the+Conference+on+Empirical+Methods+in+Natural+)
26. Zhijing Jin, Yuen Chen, Felix Leeb, et al. 2023. CLadder: A benchmark to assess causal reasoning capabilities of language models. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Zhijing+Jin%2C+Yuen+Chen%2C+Felix+Leeb%2C+et+al%2E+2023%2E+CLadder%3A+A+benchmark+to+assess+causal+reasoning+capabilities+of+language+models%2E+In+Advances+in+Neural+Information+Processing+Systems%2E)
27. Zhijing Jin, Jiarui Liu, Zhiheng Lyu, et al. 2024. Can large language models infer causation from correlation? In Proceedings of the International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Zhijing+Jin%2C+Jiarui+Liu%2C+Zhiheng+Lyu%2C+et+al%2E+2024%2E+Can+large+language+models+infer+causation+from+correlation%3F+In+Proceedings+of+the+International+Conference+on+Learning+Representations%2E)
28. Mohammad Abdullah Matin Khan, M Saiful Bari, Xuan Long Do, et al. 2024. [XCodeEval: An execution-based large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval](https://md.research.elamur.ai/free/en/papers/khan-2024-xcodeeval-execution-based-large-scale/). In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pag…
29. Will Kurt. Say what you mean: A response to ’let me speak freely’. Accessed: 2025-01-01. Available at: [https://blog.dottxt.co/say-what-you-mean](https://blog.dottxt.co/say-what-you-mean). html. [↗](https://blog.dottxt.co/say-what-you-mean)
30. Hao Li, Chenghao Yang, An Zhang, et al. 2025. [Hello again! llm-powered personalized agent for long-term dialogue](https://md.research.elamur.ai/free/en/papers/li-2025-hello-again-llm-powered-personalized/). In Proceedings of the Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics, pages 5259–5276.
31. Xinze Li, Yixin Cao, Liangming Pan, et al. 2024a. Towards verifiable generation: A benchmark for knowledge-aware language model attribution. In Findings of the Association for Computational Linguistics: ACL 2024, pages 493–516. [find](https://scholar.google.com/scholar?q=Xinze+Li%2C+Yixin+Cao%2C+Liangming+Pan%2C+et+al%2E+2024a%2E+Towards+verifiable+generation%3A+A+benchmark+for+knowledge%2Daware+language+model+attribution%2E+In+Findings+of+the+Association+for+Computational+Linguistic)
32. Yinghao Li, Rampi Ramprasad, and Chao Zhang. 2024b. A simple but effective approach to improve structured language model output for information extraction. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 5133–5148. [find](https://scholar.google.com/scholar?q=Yinghao+Li%2C+Rampi+Ramprasad%2C+and+Chao+Zhang%2E+2024b%2E+A+simple+but+effective+approach+to+improve+structured+language+model+output+for+information+extraction%2E+In+Findings+of+the+Association+for+Computati)
33. Aixin Liu, Bei Feng, Bing Xue, et al. 2024a. Deepseekv3 technical report. arXiv. [find](https://scholar.google.com/scholar?q=Aixin+Liu%2C+Bei+Feng%2C+Bing+Xue%2C+et+al%2E+2024a%2E+Deepseekv3+technical+report%2E+arXiv%2E)
34. Michael Xieyang Liu, Frederick Liu, Alexander J Fiannaca, et al. 2024b. We need structured output: Towards user-centered constraints on large language model output. In The ACM CHI Conference on Human Factors in Computing Systems, pages 1–9. [find](https://scholar.google.com/scholar?q=Michael+Xieyang+Liu%2C+Frederick+Liu%2C+Alexander+J+Fiannaca%2C+et+al%2E+2024b%2E+We+need+structured+output%3A+Towards+user%2Dcentered+constraints+on+large+language+model+output%2E+In+The+ACM+CHI+Conference+on+Human+)
35. Yuhe Liu, Changhua Pei, Longlong Xu, et al. 2025. Op-seval: A comprehensive benchmark suite for evaluating large language models’ capability in it operations domain. In Proceedings of the ACM International Conference on the Foundations of Software Engineering, page 503–513. [find](https://scholar.google.com/scholar?q=Yuhe+Liu%2C+Changhua+Pei%2C+Longlong+Xu%2C+et+al%2E+2025%2E+Op%2Dseval%3A+A+comprehensive+benchmark+suite+for+evaluating+large+language+models%E2%80%99+capability+in+it+operations+domain%2E+In+Proceedings+of+the+ACM+Internat)
36. Robert A McLean, William L Sanders, and Walter W Stroup. 1991. A unified approach to mixed linear models. The American Statistician, 45(1):54–64. [find](https://scholar.google.com/scholar?q=Robert+A+McLean%2C+William+L+Sanders%2C+and+Walter+W+Stroup%2E+1991%2E+A+unified+approach+to+mixed+linear+models%2E+The+American+Statistician%2C+45%281%29%3A54%E2%80%9364%2E)
37. Quinn McNemar. 1947. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153–157. [find](https://scholar.google.com/scholar?q=Quinn+McNemar%2E+1947%2E+Note+on+the+sampling+error+of+the+difference+between+correlated+proportions+or+percentages%2E+Psychometrika%2C+12%282%29%3A153%E2%80%93157%2E)
38. Thomas Mesnard, Cassidy Hardin, Robert Dadashi, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv. [find](https://scholar.google.com/scholar?q=Thomas+Mesnard%2C+Cassidy+Hardin%2C+Robert+Dadashi%2C+et+al%2E+2024%2E+Gemma%3A+Open+models+based+on+gemini+research+and+technology%2E+arXiv%2E)
39. Erik Miehling, Michael Desmond, Karthikeyan Natesan Ramamurthy, et al. 2025. Evaluating the prompt steerability of large language models. In Proceedings of the Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics, pages 7874–7900. [find](https://scholar.google.com/scholar?q=Erik+Miehling%2C+Michael+Desmond%2C+Karthikeyan+Natesan+Ramamurthy%2C+et+al%2E+2025%2E+Evaluating+the+prompt+steerability+of+large+language+models%2E+In+Proceedings+of+the+Conference+of+the+Nations+of+the+America)
40. Seyed Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, et al. 2025. [GSM-symbolic: Understanding the limitations of mathematical reasoning in large language models](https://md.research.elamur.ai/free/en/papers/mirzadeh-2024-gsm-symbolic-understanding-limitations-mathematical/). In Proceedings of the International Conference on Learning Representations.
41. Robert Osazuwa Ness. 2025. Causal AI. Manning Publications Co. [find](https://scholar.google.com/scholar?q=Robert+Osazuwa+Ness%2E+2025%2E+Causal+AI%2E+Manning+Publications+Co%2E)
42. OpenAI. Introducing structured outputs in the api. Accessed: 2025-01-01. Available at: [https://openai.com/index/introducing-structured-outputs](https://openai.com/index/introducing-structured-outputs) -in-the-api/. [↗](https://openai.com/index/introducing-structured-outputs)
43. Judea Pearl and Dana Mackenzie. 2018. The Book of Why: The New Science of Cause and Effect. Basic Books, Inc. [find](https://scholar.google.com/scholar?q=Judea+Pearl+and+Dana+Mackenzie%2E+2018%2E+The+Book+of+Why%3A+The+New+Science+of+Cause+and+Effect%2E+Basic+Books%2C+Inc%2E)
44. Karl Pearson. 1900. X. on the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. Philosophical Magazine, 50(302):157– 175. [find](https://scholar.google.com/scholar?q=Karl+Pearson%2E+1900%2E+X%2E+on+the+criterion+that+a+given+system+of+deviations+from+the+probable+in+the+case+of+a+correlated+system+of+variables+is+such+that+it+can+be+reasonably+supposed+to+have+arisen+fr)
45. Zhenting Qi, Mingyuan MA, Jiahang Xu, et al. 2025. Mutual reasoning makes smaller LLMs stronger problem-solver. In Proceedings of the International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Zhenting+Qi%2C+Mingyuan+MA%2C+Jiahang+Xu%2C+et+al%2E+2025%2E+Mutual+reasoning+makes+smaller+LLMs+stronger+problem%2Dsolver%2E+In+Proceedings+of+the+International+Conference+on+Learning+Representations%2E)
46. Adam Roegiest, Radha Chitta, Jonathan Donnelly, et al. 2023. Questions about contracts: Prompt templates for structured answer generation. In Proceedings of the Natural Legal Language Processing Workshop 2023, pages 62–72. [find](https://scholar.google.com/scholar?q=Adam+Roegiest%2C+Radha+Chitta%2C+Jonathan+Donnelly%2C+et+al%2E+2023%2E+Questions+about+contracts%3A+Prompt+templates+for+structured+answer+generation%2E+In+Proceedings+of+the+Natural+Legal+Language+Processing+Works)
47. Donald B Rubin. 1974. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of educational Psychology, 66(5):688. [find](https://scholar.google.com/scholar?q=Donald+B+Rubin%2E+1974%2E+Estimating+causal+effects+of+treatments+in+randomized+and+nonrandomized+studies%2E+Journal+of+educational+Psychology%2C+66%285%29%3A688%2E)
48. Shuai Wang, Harrisen Scells, Bevan Koopman, et al. 2025. Reassessing large language model boolean query generation for systematic reviews. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 3296–3305. [find](https://scholar.google.com/scholar?q=Shuai+Wang%2C+Harrisen+Scells%2C+Bevan+Koopman%2C+et+al%2E+2025%2E+Reassessing+large+language+model+boolean+query+generation+for+systematic+reviews%2E+In+Proceedings+of+the+International+ACM+SIGIR+Conference+on+R)
49. Jason Wei, Xuezhi Wang, Dale Schuurmans, et al. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In Advances in Neural Information Processing Systems, pages 24824–24837.
50. Haolun Wu, Ye Yuan, Liana Mikaelyan, et al. 2024. Learning to extract structured entities using language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6817–6834. [find](https://scholar.google.com/scholar?q=Haolun+Wu%2C+Ye+Yuan%2C+Liana+Mikaelyan%2C+et+al%2E+2024%2E+Learning+to+extract+structured+entities+using+language+models%2E+In+Proceedings+of+the+2024+Conference+on+Empirical+Methods+in+Natural+Language+Processi)
51. Shenghan Wu, Yimo Zhu, Wynne Hsu, et al. 2025a. From personas to talks: Revisiting the impact of personas on llm-synthesized emotional support conversations. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 5439–5453. [find](https://scholar.google.com/scholar?q=Shenghan+Wu%2C+Yimo+Zhu%2C+Wynne+Hsu%2C+et+al%2E+2025a%2E+From+personas+to+talks%3A+Revisiting+the+impact+of+personas+on+llm%2Dsynthesized+emotional+support+conversations%2E+In+Proceedings+of+the+Conference+on+Empiri)
52. Yunxiao Shi, Haoning Shang, Xing Zi, et al. 2025. Answering narrative-driven recommendation queries via a retrieve–rank paradigm and the ocg-agent. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 13192–13213. [find](https://scholar.google.com/scholar?q=Yunxiao+Shi%2C+Haoning+Shang%2C+Xing+Zi%2C+et+al%2E+2025%2E+Answering+narrative%2Ddriven+recommendation+queries+via+a+retrieve%E2%80%93rank+paradigm+and+the+ocg%2Dagent%2E+In+Proceedings+of+the+Conference+on+Empirical+Method)
53. Yilin Wu, Han Yuan, Li Zhang, and Zheng Ma. 2025b. Natural language inference as a judge: Detecting factuality and causality issues in language model self-reasoning for financial analysis. In Proceedings of the Tenth Workshop on Financial Technology and Natural Language Processing. [find](https://scholar.google.com/scholar?q=Yilin+Wu%2C+Han+Yuan%2C+Li+Zhang%2C+and+Zheng+Ma%2E+2025b%2E+Natural+language+inference+as+a+judge%3A+Detecting+factuality+and+causality+issues+in+language+model+self%2Dreasoning+for+financial+analysis%2E+In+Proceedi)
54. Samuel A Stouffer, Edward A Suchman, Leland C DeVinney, et al. 1949. The american soldier: Adjustment during army life. Princeton Univ. Press. [find](https://scholar.google.com/scholar?q=Samuel+A+Stouffer%2C+Edward+A+Suchman%2C+Leland+C+DeVinney%2C+et+al%2E+1949%2E+The+american+soldier%3A+Adjustment+during+army+life%2E+Princeton+Univ%2E+Press%2E)
55. An Yang, Baosong Yang, Beichen Zhang, et al. 2024. Qwen2. 5 technical report. arXiv. [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+et+al%2E+2024%2E+Qwen2%2E+5+technical+report%2E+arXiv%2E)
56. Ben Swanson, Kory Mathewson, Ben Pietrzak, Sherol Chen, and Monica Dinalescu. 2021. Story centaur: Large language model few shot learning as a creative writing tool. In Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics, pages 244–256. [find](https://scholar.google.com/scholar?q=Ben+Swanson%2C+Kory+Mathewson%2C+Ben+Pietrzak%2C+Sherol+Chen%2C+and+Monica+Dinalescu%2E+2021%2E+Story+centaur%3A+Large+language+model+few+shot+learning+as+a+creative+writing+tool%2E+In+Proceedings+of+the+Conference+o)
57. Zhi Rui Tam, Cheng-Kuang Wu, Yi-Lin Tsai, et al. 2024. Let me speak freely? a study on the impact of format restrictions on large language model performance. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 1218–1236. [find](https://scholar.google.com/scholar?q=Zhi+Rui+Tam%2C+Cheng%2DKuang+Wu%2C+Yi%2DLin+Tsai%2C+et+al%2E+2024%2E+Let+me+speak+freely%3F+a+study+on+the+impact+of+format+restrictions+on+large+language+model+performance%2E+In+Proceedings+of+the+2024+Conference+on+E)
58. Xiangru Tang, Yiming Zong, Jason Phang, et al. 2024. Struc-bench: Are large language models good at generating complex structured tabular data? In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics, pages 12–34. [find](https://scholar.google.com/scholar?q=Xiangru+Tang%2C+Yiming+Zong%2C+Jason+Phang%2C+et+al%2E+2024%2E+Struc%2Dbench%3A+Are+large+language+models+good+at+generating+complex+structured+tabular+data%3F+In+Proceedings+of+the+2024+Conference+of+the+North+Ameri)
59. Hugo Touvron, Thibaut Lavril, Gautier Izacard, et al. 2023. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv.
60. Peiyi Wang, Lei Li, Zhihong Shao, et al. 2024. [Math-shepherd: Verify and reinforce LLMs step-by-step without human annotations](https://md.research.elamur.ai/free/en/papers/wang-2024-math-shepherd-verify-reinforce-llms/). In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 9426–9439.
61. Han Yuan. 2025. Agentic large language models for healthcare: current progress and future opportunities. Medicine Advances, 3(1):37–41. [find](https://scholar.google.com/scholar?q=Han+Yuan%2E+2025%2E+Agentic+large+language+models+for+healthcare%3A+current+progress+and+future+opportunities%2E+Medicine+Advances%2C+3%281%29%3A37%E2%80%9341%2E)
62. Han Yuan, Li Zhang, and Zheng Ma. 2025. Exploring the reliability of self-explanation and its relationship with classification in language model-driven financial analysis. In ICLR 2025 Workshop on Advances in Financial AI: Opportunities, Innovations and Responsible AI. [find](https://scholar.google.com/scholar?q=Han+Yuan%2C+Li+Zhang%2C+and+Zheng+Ma%2E+2025%2E+Exploring+the+reliability+of+self%2Dexplanation+and+its+relationship+with+classification+in+language+model%2Ddriven+financial+analysis%2E+In+ICLR+2025+Workshop+on+Adv)
63. Dan Zhang, Sining Zhoubian, Ziniu Hu, et al. 2024. [Rest-mcts*: Llm self-training via process reward guided tree search](https://md.research.elamur.ai/free/en/papers/zhang-2024-rest-mcts-llm-self-training/). In Advances in Neural Information Processing Systems, volume 37, pages 64735– 64772.
64. Jun Zhao, Can Zu, Xu Hao, et al. 2024a. LON-GAGENT: Achieving question answering for 128ktoken-long documents through multi-agent collaboration. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16310–16324. [find](https://scholar.google.com/scholar?q=Jun+Zhao%2C+Can+Zu%2C+Xu+Hao%2C+et+al%2E+2024a%2E+LON%2DGAGENT%3A+Achieving+question+answering+for+128ktoken%2Dlong+documents+through+multi%2Dagent+collaboration%2E+In+Proceedings+of+the+2024+Conference+on+Empirical+Meth)
65. Qingfei Zhao, Ruobing Wang, Yukuo Cen, et al. 2024b. LongRAG: A dual-perspective retrieval-augmented generation paradigm for long-context question answering. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 22600–22632. [find](https://scholar.google.com/scholar?q=Qingfei+Zhao%2C+Ruobing+Wang%2C+Yukuo+Cen%2C+et+al%2E+2024b%2E+LongRAG%3A+A+dual%2Dperspective+retrieval%2Daugmented+generation+paradigm+for+long%2Dcontext+question+answering%2E+In+Proceedings+of+the+2024+Conference+on+E)

*Updated 31 October 2026*