--- title: "TreeCut — paper by Ouyang (2025) — summary and ideas" description: "LLMs score near human level on math word problem benchmarks such as GSM8K. 2 references, cited by 4 papers here." authors: ["Jialin Ouyang"] year: 2025 venue: "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)" source: "https://aclanthology.org/2025.acl-short.84/" doi: "10.18653/v1/2025.acl-short.84" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/ouyang-2025-treecut-synthetic-unanswerable-math-word/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ouyang-2025-treecut-synthetic-unanswerable-math-word/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/11/) / TreeCut: A Synthetic Unanswerable Math Word Problem Dataset for LLM Hallucination Evaluation # TreeCut: A Synthetic Unanswerable Math Word Problem Dataset for LLM Hallucination Evaluation **Authors:** Jialin Ouyang · **Year:** 2025 · **Venue:** [Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2025](https://md.research.elamur.ai/free/en/venues/acl/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2025.acl-short.84/) · [PDF](https://aclanthology.org/2025.acl-short.84.pdf) · [DOI 10.18653/v1/2025.acl-short.84](https://doi.org/10.18653/v1/2025.acl-short.84) · [OpenAlex](https://openalex.org/W4412889500) **Citations:** 1 citations (OpenAlex, 2026-10-14) · 4 in our library, of the 149,439 papers whose bibliographies we read ## Summary LLMs score near human level on math word problem benchmarks such as GSM8K. However, they often give confident answers to problems that cannot be solved. Earlier unanswerable-problem sets were built by editing existing benchmarks, which brings contamination risk, small scale and little structural control. TreeCut instead generates problems synthetically. Each problem is a tree: nodes are item prices, edges from the root assign values, and other edges are linear relations between two items. Deleting one edge on the path to the asked variable makes the problem unsolvable, while the intact tree serves as an answerable control. Generation parameters set the number of variables, the depth of the target, whether item names are composite, and the cut's position. Six models were tested: Llama 3.1, Qwen2.5, GPT-4o and o3-mini. All showed substantial hallucination, and most could solve the answerable versions far better than they could spot the unanswerable ones. The dataset gives a contamination-free and controllable way to probe whether models recognize missing information. ## Key points - **Generator design:** prices are integers from 5 to 15, and coefficients are non-zero integers from −3 to 3. The controllable parameters are numVars, ansDepth, compositeName and cutDepth. Code and samples are at github.com/j-bagel/treecut-math. - **Zero-shot worst cases:** GPT-4o hallucinated on 64% of problems at ansDepth = 8, and o3-mini on 44% at ansDepth = 2. In the shallow case, o3-mini tended to invent assumptions. Llama-8B and both Qwen models failed almost entirely, with hallucination rates around 80–95%. - **Few-shot prompting** (3 answerable + 3 unanswerable examples) lowered o3-mini to about 3%. Rates for the other five models stayed roughly the same, which suggests an intrinsic weakness. - **Solving vs. detecting:** GPT-4o solved 84% of answerable problems at ansDepth = 8 but recognized only 36% of the matching unanswerable ones. - **What raises hallucination (GPT-4o, 500 problems per configuration):** deeper target variables, extra unrelated conditions in the tree, composite item names, and a cut placed near the middle of the path. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Columbia University](https://md.research.elamur.ai/free/en/universities/columbia-university/) — University, New York, United States · 241 works · 54,053 citations 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works 🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works ## Cite Cite as: Jialin Ouyang. 2025. TreeCut: A Synthetic Unanswerable Math Word Problem Dataset for LLM Hallucination Evaluation. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{ouyang2025treecut, title = {{TreeCut: A Synthetic Unanswerable Math Word Problem Dataset for LLM Hallucination Evaluation}}, author = {Jialin Ouyang}, booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)}, year = {2025}, doi = {10.18653/v1/2025.acl-short.84}, url = {https://doi.org/10.18653/v1/2025.acl-short.84} } ``` - [← previous Towards Context-Robust LLMs: A Gated Representation Fine-tuning Approach](https://md.research.elamur.ai/free/en/papers/zeng-2025-context-robust-llms-gated-representation/) - [next → Uncertainty Distillation: Teaching Language Models to Express Semantic Confidence](https://md.research.elamur.ai/free/en/papers/hager-2025-uncertainty-distillation-teaching-language-models/) - [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/11/) ## References 2 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Figure 3: Hallucination percentage versus cutDepth. Left* panel has ansDepth = 7. Right panel has ansDepth = 8. [find](https://scholar.google.com/scholar?q=Figure+3%3A+Hallucination+percentage+versus+cutDepth%2E+Left%2A+panel+has+ansDepth+%3D+7%2E+Right+panel+has+ansDepth+%3D+8%2E) 2. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774. *Updated 30 October 2026*