---
title: "Asking language models how to represent data for fine-tuning"
description: "When small language models handle tables, database schemas or graphs, the structure must first be turned into text. 17 references."
authors: ["Usneek Singh", "Ananya Singha", "Abhijeet Awasthi", "Sumit Gulwani", "Aditya Kanade", "Vu Le", "Mukul Singh", "Gust Verbruggen"]
year: 2026
venue: "Proceedings of the First Workshop on Structured Understanding, Retrieval, and Generation in the LLM Era (SURGeLLM 2026)"
source: "https://aclanthology.org/2026.surgellm-1.13/"
doi: "10.18653/v1/2026.surgellm-1.13"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/singh-2026-asking-language-models-how-represent/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/singh-2026-asking-language-models-how-represent/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/22/) / Asking language models how to represent data for fine-tuning

# Asking language models how to represent data for fine-tuning

**Authors:** Usneek Singh, Ananya Singha, Abhijeet Awasthi, [Sumit Gulwani](https://md.research.elamur.ai/free/en/authors/sumit-gulwani/), Aditya Kanade, Vu Le, [Mukul Singh](https://md.research.elamur.ai/free/en/authors/mukul-singh/) and Gust Verbruggen · **Year:** 2026 · **Venue:** [Proceedings of the First Workshop on Structured Understanding, Retrieval, and Generation in the LLM Era (SURGeLLM 2026)](https://md.research.elamur.ai/free/en/venues/surgellm/) ([Workshop on Structured Understanding, Retrieval, and Generation in the LLM Era 2026](https://md.research.elamur.ai/free/en/venues/surgellm/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.surgellm-1.13/) · [PDF](https://aclanthology.org/2026.surgellm-1.13.pdf) · [DOI 10.18653/v1/2026.surgellm-1.13](https://doi.org/10.18653/v1/2026.surgellm-1.13) · [OpenAlex](https://openalex.org/W7166826483)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
When small language models handle tables, database schemas or graphs, the structure must first be turned into text. Developers usually pick that serialization by intuition or through costly repeated fine-tuning runs. Singh et al. ask whether the choice still matters once a model is fine-tuned, and whether it can be made cheaply.

They test six tasks with Phi-3, Mistral and CodeLlama, each adapted with LoRA:
- NL-to-pandas code (Arcade)
- NL-to-Excel formulas
- text-to-SQL (Spider)
- three graph problems: cycle detection, flow estimation and shortest path

To find candidate formats, they truncate a prompt just before the data structure, let the base model fill it in, and classify the completions with regular expressions.

Fine-tuning does not make models indifferent to format, since some serializations stay clearly stronger. More importantly, how formats rank under few-shot inference on the base model largely carries over after fine-tuning. A format can therefore be selected through inference alone, which saves substantial compute for anyone fine-tuning on structured inputs.

## Key points
- **Models and training:** Phi-3-mini-4k-instruct (3.8B), Mistral-7B-Instruct-v0.2 and CodeLlama-7b-hf were trained with LoRA (rank 64) for 10 epochs on one A100 GPU. Results are scored with pass@5.
- **Format gaps after fine-tuning:** The spread between best and worst format averages 0.03 pass@5 on text-to-code tasks and 0.33 on graph tasks. It reaches up to 0.50 in the largest case.
- **Base model predicts the winner:** In 16 of 18 model–task settings, the best format on the base model stays among the top candidates after fine-tuning (mean Kendall's τ = 0.67). In 17 of 18 settings it lands in the top-2.
- **Model-suggested formats:** Models most often proposed the column-dictionary format for dataframes and the closed-bracket style for SQL schemas. For graphs, edge lists and adjacency dicts dominated. However, how often a model suggests a format is not a reliable indicator of how well that format performs.
- **Limitations:** Fine-tuning gave little gain for Phi-3 on text-to-Python and for CodeLlama on text-to-SQL. The analysis also covers only one structured element per prompt.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/) — Company, Redmond, United States · 325 works · 142,931 citations
- [Georgia Institute of Technology](https://md.research.elamur.ai/free/en/universities/georgia-institute-of-technology/) — University, Atlanta, United States · 133 works · 24,497 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Usneek Singh, Ananya Singha, Abhijeet Awasthi, Sumit Gulwani, Aditya Kanade, Vu Le, Mukul Singh and Gust Verbruggen. 2026. Asking language models how to represent data for fine-tuning. Proceedings of the First Workshop on Structured Understanding, Retrieval, and Generation in the LLM Era (SURGeLLM 2026).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{singh2026asking,
  title = {{Asking language models how to represent data for fine-tuning}},
  author = {Usneek Singh, Ananya Singha, Abhijeet Awasthi, Sumit Gulwani, Aditya Kanade, Vu Le, Mukul Singh and Gust Verbruggen},
  booktitle = {Proceedings of the First Workshop on Structured Understanding, Retrieval, and Generation in the LLM Era (SURGeLLM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.surgellm-1.13},
  url = {https://doi.org/10.18653/v1/2026.surgellm-1.13}
}
```

- [← previous Ask Me Again Differently: GRAS for Measuring Bias in Vision Language Models on Gender, Race, Age, and Skin Tone](https://md.research.elamur.ai/free/en/papers/malik-2026-ask-me-again-differently-gras/)
- [next → Aspect-based sentiment analysis in the accommodation domain with the BERTimbau model](https://md.research.elamur.ai/free/en/papers/pereira-2026-analise-de-sentimento-baseada-em/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/22/)

## References

17 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ziquan Liu, Yi Xu, Yuanhong Xu, Qi Qian, Hao Li, Xiangyang Ji, Antoni Chan, and Rong Jin. 2022. Improved fine-tuning by better leveraging pre-training data. Advances in Neural Information Processing Systems, 35:32568–32581. [find](https://scholar.google.com/scholar?q=Ziquan+Liu%2C+Yi+Xu%2C+Yuanhong+Xu%2C+Qi+Qian%2C+Hao+Li%2C+Xiangyang+Ji%2C+Antoni+Chan%2C+and+Rong+Jin%2E+2022%2E+Improved+fine%2Dtuning+by+better+leveraging+pre%2Dtraining+data%2E+Advances+in+Neural+Information+Processing+S)
2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
3. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374.
4. Xi Fang, Weijie Xu, Fiona Anting Tan, Jiani Zhang, Ziqing Hu, Yanjun Jane Qi, Scott Nickleach, Diego Socolinsky, Srinivasan Sengamedu, Christos Faloutsos, et al. 2024. Large language models (llms) on tabular data: Prediction, generation, and understanding-a survey. [find](https://scholar.google.com/scholar?q=Xi+Fang%2C+Weijie+Xu%2C+Fiona+Anting+Tan%2C+Jiani+Zhang%2C+Ziqing+Hu%2C+Yanjun+Jane+Qi%2C+Scott+Nickleach%2C+Diego+Socolinsky%2C+Srinivasan+Sengamedu%2C+Christos+Faloutsos%2C+et+al%2E+2024%2E+Large+language+models+%28llms%29+on+)
5. Bahare Fatemi, Jonathan Halcrow, and Bryan Perozzi. 2023. Talk like a graph: Encoding graphs for large language models. arXiv preprint arXiv:2310.04560. [find](https://scholar.google.com/scholar?q=Bahare+Fatemi%2C+Jonathan+Halcrow%2C+and+Bryan+Perozzi%2E+2023%2E+Talk+like+a+graph%3A+Encoding+graphs+for+large+language+models%2E+arXiv+preprint+arXiv%3A2310%2E04560%2E)
6. Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. 2023. [Text-to-sql empowered by large language models: A benchmark evaluation](https://md.research.elamur.ai/free/en/papers/gao-2023-text-sql-empowered-large-language/). arXiv preprint arXiv:2308.15363.
7. Usneek Singh, José Cambronero, Sumit Gulwani, Aditya Kanade, Anirudh Khatry, Vu Le, Mukul Singh, and Gust Verbruggen. 2024. An empirical study of validating synthetic data for formula generation. arXiv preprint arXiv:2407.10657. [find](https://scholar.google.com/scholar?q=Usneek+Singh%2C+Jos%C3%A9+Cambronero%2C+Sumit+Gulwani%2C+Aditya+Kanade%2C+Anirudh+Khatry%2C+Vu+Le%2C+Mukul+Singh%2C+and+Gust+Verbruggen%2E+2024%2E+An+empirical+study+of+validating+synthetic+data+for+formula+generation%2E+arXi)
8. Ananya Singha, José Cambronero, Sumit Gulwani, Vu Le, and Chris Parnin. 2023. Tabular representation, noisy operators, and impacts on table structure understanding tasks in llms. arXiv preprint arXiv:2310.10358. — [arXiv:2310.10358](https://arxiv.org/abs/2310.10358) [find](https://scholar.google.com/scholar?q=Ananya+Singha%2C+Jos%C3%A9+Cambronero%2C+Sumit+Gulwani%2C+Vu+Le%2C+and+Chris+Parnin%2E+2023%2E+Tabular+representation%2C+noisy+operators%2C+and+impacts+on+table+structure+understanding+tasks+in+llms%2E+arXiv+preprint+arXiv%3A)
9. Ananya Singha, Bhavya Chopra, Anirudh Khatry, Sumit Gulwani, Austin Henley, Vu Le, Chris Parnin, Mukul Singh, and Gust Verbruggen. 2024. Semantically aligned question and code generation for automated insight generation. In Proceedings of the 1st International Workshop on Large Language Models for C… [find](https://scholar.google.com/scholar?q=Ananya+Singha%2C+Bhavya+Chopra%2C+Anirudh+Khatry%2C+Sumit+Gulwani%2C+Austin+Henley%2C+Vu+Le%2C+Chris+Parnin%2C+Mukul+Singh%2C+and+Gust+Verbruggen%2E+2024%2E+Semantically+aligned+question+and+code+generation+for+automated)
10. Yuan Sui, Mengyu Zhou, Mingjie Zhou, Shi Han, and Dongmei Zhang. 2024. Table meets llm: Can large language models understand structured table data? a benchmark and empirical study. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, pages 645–654. [find](https://scholar.google.com/scholar?q=Yuan+Sui%2C+Mengyu+Zhou%2C+Mingjie+Zhou%2C+Shi+Han%2C+and+Dongmei+Zhang%2E+2024%2E+Table+meets+llm%3A+Can+large+language+models+understand+structured+table+data%3F+a+benchmark+and+empirical+study%2E+In+Proceedings+of+t)
11. Heng Wang, Shangbin Feng, Tianxing He, Zhaoxuan Tan, Xiaochuang Han, and Yulia Tsvetkov. 2024a. Can language models solve graph problems in natural language? Advances in Neural Information Processing Systems, 36. [find](https://scholar.google.com/scholar?q=Heng+Wang%2C+Shangbin+Feng%2C+Tianxing+He%2C+Zhaoxuan+Tan%2C+Xiaochuang+Han%2C+and+Yulia+Tsvetkov%2E+2024a%2E+Can+language+models+solve+graph+problems+in+natural+language%3F+Advances+in+Neural+Information+Processing+)
12. Zilong Wang, Hao Zhang, Chun-Liang Li, Julian Martin Eisenschlos, Vincent Perot, Zifeng Wang, Lesly Miculicich, Yasuhisa Fujii, Jingbo Shang, Chen-Yu Lee, et al. 2024b. Chain-of-table: Evolving tables in the reasoning chain for table understanding. arXiv preprint arXiv:2401.04398. [find](https://scholar.google.com/scholar?q=Zilong+Wang%2C+Hao+Zhang%2C+Chun%2DLiang+Li%2C+Julian+Martin+Eisenschlos%2C+Vincent+Perot%2C+Zifeng+Wang%2C+Lesly+Miculicich%2C+Yasuhisa+Fujii%2C+Jingbo+Shang%2C+Chen%2DYu+Lee%2C+et+al%2E+2024b%2E+Chain%2Dof%2Dtable%3A+Evolving+tables)
13. Ruosong Ye, Caiqi Zhang, Runhui Wang, Shuyuan Xu, Yongfeng Zhang, et al. 2023a. Natural language is all a graph needs. arXiv preprint arXiv:2308.07134, 4(5):7. [find](https://scholar.google.com/scholar?q=Ruosong+Ye%2C+Caiqi+Zhang%2C+Runhui+Wang%2C+Shuyuan+Xu%2C+Yongfeng+Zhang%2C+et+al%2E+2023a%2E+Natural+language+is+all+a+graph+needs%2E+arXiv+preprint+arXiv%3A2308%2E07134%2C+4%285%29%3A7%2E)
14. Yunhu Ye, Binyuan Hui, Min Yang, Binhua Li, Fei Huang, and Yongbin Li. 2023b. Large language models are versatile decomposers: Decomposing evidence and questions for table-based reasoning. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrie… [find](https://scholar.google.com/scholar?q=Yunhu+Ye%2C+Binyuan+Hui%2C+Min+Yang%2C+Binhua+Li%2C+Fei+Huang%2C+and+Yongbin+Li%2E+2023b%2E+Large+language+models+are+versatile+decomposers%3A+Decomposing+evidence+and+questions+for+table%2Dbased+reasoning%2E+In+Proceedi)
15. Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, et al. 2022. Natural language to code generation in interactive data science notebooks. arXiv preprint arXiv:2212.09248. [find](https://scholar.google.com/scholar?q=Pengcheng+Yin%2C+Wen%2DDing+Li%2C+Kefan+Xiao%2C+Abhishek+Rao%2C+Yeming+Wen%2C+Kensen+Shi%2C+Joshua+Howland%2C+Paige+Bailey%2C+Michele+Catasta%2C+Henryk+Michalewski%2C+et+al%2E+2022%2E+Natural+language+to+code+generation+in+int)
16. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. 2018. [Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task](https://md.research.elamur.ai/free/en/papers/yu-2018-spider-large-scale-human-labeled/). arXiv preprint arXiv:1809.08887.
17. Alex Zhuang, Ge Zhang, Tianyu Zheng, Xinrun Du, Junjie Wang, Weiming Ren, Stephen W Huang, Jie Fu, Xiang Yue, and Wenhu Chen. 2024. Structlm: Towards building generalist models for structured knowledge grounding. arXiv preprint arXiv:2402.16671. — [arXiv:2402.16671](https://arxiv.org/abs/2402.16671) [find](https://scholar.google.com/scholar?q=Alex+Zhuang%2C+Ge+Zhang%2C+Tianyu+Zheng%2C+Xinrun+Du%2C+Junjie+Wang%2C+Weiming+Ren%2C+Stephen+W+Huang%2C+Jie+Fu%2C+Xiang+Yue%2C+and+Wenhu+Chen%2E+2024%2E+Structlm%3A+Towards+building+generalist+models+for+structured+knowledg)

*Updated 1 November 2026*