--- title: "łlinstruct — paper by Ghosh and Chan (2024)" description: "Writing items for English Language Proficiency Assessments (ELPA) by hand is slow, and general instruction-tuned LLMs often… 4 references." authors: ["Debanjan Ghosh", "Sophia Chan"] year: 2024 venue: "arXiv:2410.09314" source: "https://arxiv.org/abs/2410.09314" doi: "10.48550/arxiv.2410.09314" arxiv: "2410.09314" licence: "CC BY 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/ghosh-2024-llinstruct-instruction-tuned-model-english/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ghosh-2024-llinstruct-instruction-tuned-model-english/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/15/) / łlinstruct: An Instruction-tuned model for English Language Proficiency Assessments # łlinstruct: An Instruction-tuned model for English Language Proficiency Assessments **Authors:** [Debanjan Ghosh](https://md.research.elamur.ai/free/en/authors/debanjan-ghosh/) and [Sophia Chan](https://md.research.elamur.ai/free/en/authors/sophia-chan/) · **Year:** 2024 · **Venue:** arXiv:2410.09314 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2410.09314) · [PDF](https://arxiv.org/pdf/2410.09314) · [DOI 10.48550/arxiv.2410.09314](https://doi.org/10.48550/arxiv.2410.09314) · [arXiv 2410.09314](https://arxiv.org/abs/2410.09314) · [OpenAlex](https://openalex.org/W4403564562) **Citations:** 0 citations (OpenAlex, 2026-10-14) ## Summary Writing items for English Language Proficiency Assessments (ELPA) by hand is slow, and general instruction-tuned LLMs often do poorly on narrow educational tasks. The authors built LL-INSTRUCT, a domain-specific instruction dataset and model for ELPA content generation. They started from 130 hand-written seed tuples, each with an instruction, input, output and explanation, derived from public sample test items. They then used GPT-4 in a self-instruct-style bootstrapping loop to expand these to about 70K tuples. Off-topic tuples were removed by a second LLM acting as a judge, along with keyword filters and ROUGE-L deduplication. Llama-3 8B was fine-tuned on 17K, 50K and 70K subsets and compared by human raters against Dolly-2, Mistral, base Llama-3 and GPT-3.5 on 200 unseen instructions. The 70K model gave the most test-ready outputs, on par with GPT-3.5, and much more usable explanations. This suggests a small, specialized 8B model can compete with a large general one. Expert revision is still needed for a sizeable share of outputs. ## Key points - The seed set has 130 tuples built by the authors from publicly available ELPA sample items covering reading, writing, speaking and listening. A single item is often split into several tuples, for example one for the correct indirect reply and one for three distractors. - Each generation prompt holds four example tuples: three seeds plus one previously generated tuple. Seeds are mixed as two short and one long task, or the reverse, and GPT-4 returns ten new tuples per call. - Filtering has three parts: - an LLM discriminator rejects non-ELPA items, such as trivia questions; - keyword filters remove tasks that mention images, videos, graphs or similar media; - a ROUGE-L similarity cap of 0.75 removes near-duplicate instructions. - Llama-3 8B was fine-tuned into SFT-17K, SFT-50K and SFT-70K. SFT-70K and GPT-3.5 each exceeded 60% valid, ready outputs. Usable explanations reached 80.5% for SFT-70K versus 42% for GPT-3.5. - About 20–30% of outputs still needed edits by subject matter experts. Figurative language, such as idioms, was the hardest task type, while grammar and vocabulary tasks were the easiest. ## Interesting ideas - Each training tuple carries an explanation field that justifies the output, which makes generated data auditable and directly useful to people, not just to the trainer. - One exam item is decomposed into separate tasks: produce the correct answer, then produce wrong-but-plausible distractors, each with a reason it fails. - A second model acts as a gatekeeper that discards off-domain generations, layered with cheap keyword filters and a ROUGE-L near-duplicate threshold. - Prompts deliberately mix short and long task types, and seed instructions use varied phrasings for the same task, to fight the diversity collapse seen when training on synthetic text. - Error analysis found figurative language to be the consistent weak spot even after fine-tuning, while grammar-level tasks were easy. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (learn): Silver examples from the LLM teacher tiers could require a short rationale, so disagreements between witness parsers can be traced and turned into repair rules. - **I2** (answer): School-test readers could generate distractors along with a reason for each, giving negative controls that check whether the reader rejects options for the right reason. - **I3** (learn): The same three-layer filter fits LLM-generated training sentences: topic judge, banned-content keywords, then surface-similarity dedup to keep the silver corpus varied. - **I4** (tree): When asking a teacher LLM for synthetic sentences for gparse, mixing short and long constructions in each prompt could widen syntactic coverage beyond the teacher's favourite patterns. - **I5** (frames): Idioms and non-literal replies, such as the indirect-answer items, would make a natural dedicated test set for the implicature and irony layer in prag. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Educational Testing Service](https://md.research.elamur.ai/free/en/ngos/educational-testing-service/) — Nonprofit, Princeton, United States · 27 works · 2,212 citations 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works 🧭 Topics (1): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,263 works ## Cite Cite as: Debanjan Ghosh and Sophia Chan. 2024. łlinstruct: An Instruction-tuned model for English Language Proficiency Assessments. arXiv:2410.09314. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{ghosh2024llinstruct, title = {{łlinstruct: An Instruction-tuned model for English Language Proficiency Assessments}}, author = {Debanjan Ghosh and Sophia Chan}, year = {2024}, doi = {10.48550/arxiv.2410.09314}, eprint = {2410.09314}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2410.09314} } ``` - [← previous Yemeni medicine students’ competence in medical English collocations](https://md.research.elamur.ai/free/en/papers/al-eryani-2025-yemeni-medicine-students-competence-medical/) - [next → ”You should probably read this”: Hedge Detection in Text](https://md.research.elamur.ai/free/en/papers/katerenchuk-2024-should-probably-read-this-hedge/) - [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/15/) ## References 4 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610–623. [↗](https://doi.org/10.1145/3442188.3445922) 2. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). arXiv preprint arXiv:2005.14165. [↗](https://arxiv.org/abs/2005.14165) 3. Jill Burstein, Geoffrey T LaFlair, Antony John Kunnan, and Alina A von Davier. 2021. A theoretical assessment ecosystem for a digital-first assessment—the duolingo english test. DRR-21-04. [↗](https://doi.org/10.46999/kiqf4328) 4. Limitation Cheng-Han Chiang and Hung-yi Lee. 2023. Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15607–15631, Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Limitation+Cheng%2DHan+Chiang+and+Hung%2Dyi+Lee%2E+2023%2E+Can+large+language+models+be+an+alternative+to+human+evaluations%3F+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+Computational+Ling) *Updated 1 November 2026*