---
title: "Heterogeneity in Formal Linguistic Competence of Language"
description: "In intervention runs, they replace 1% of the corpus with synthetic documents generated by GPT-OSS 120B. 17 references."
authors: ["H S V N S Kowndinya Renduchintala", "Sumit Bhatia"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.224/"
doi: "10.18653/v1/2026.findings-acl.224"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/renduchintala-2026-heterogeneity-formal-linguistic-competence-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/renduchintala-2026-heterogeneity-formal-linguistic-competence-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/32/) / Heterogeneity in Formal Linguistic Competence of Language Models: Is Data the Real Bottleneck?

# Heterogeneity in Formal Linguistic Competence of Language Models: Is Data the Real Bottleneck?

**Authors:** H S V N S Kowndinya Renduchintala and [Sumit Bhatia](https://md.research.elamur.ai/free/en/authors/sumit-bhatia/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.224/) · [PDF](https://aclanthology.org/2026.findings-acl.224.pdf) · [DOI 10.18653/v1/2026.findings-acl.224](https://doi.org/10.18653/v1/2026.findings-acl.224) · [OpenAlex](https://openalex.org/W7166898994)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Language models score well on BLiMP overall, yet on certain paradigms they stay below chance even after massive pre-training, and adding more tokens does not reliably fix this. The authors ask whether these gaps come from architectural limits or from rare exposure to the relevant constructions in web text. They train GPT-2 Small (124M) on 100M FineWeb tokens. In intervention runs, they replace 1% of the corpus with synthetic documents generated by GPT-OSS 120B. Each document is a natural-sounding text in a randomly chosen genre and subgenre. It must contain several sampled frequent lemmas and embeds a sentence that follows the syntactic template of one target BLiMP paradigm. The targeted runs are compared with an unmodified baseline and with a control that uses random synthetic data. Targeted injection improves 8 of the 9 weakest paradigms, and aggregate BLiMP results stay the same or rise slightly. The findings suggest that, at human-like data budgets, what the training data contains can matter more than its volume. This is relevant to BabyLM-style research.

## Key points
- **Training setup:** GPT-2 Small 124M with FlashAttention-2 and a 1024-token context, trained on 100M tokens for 20 epochs. Intervention runs use 99M FineWeb tokens plus 1M synthetic tokens. Only about 3K of the synthetic sentences actually instantiate the target paradigm.
- **Largest gains:**
  - only_npi_scope rose from 20.9% to 69.4% (+48.5).
  - principle_A_reconstruction rose from 37.2% to 78.9% (+41.7). This beats Llama-3 70B (49.6%), which was trained on more than 15T tokens.
- **Resistant case:** principle_A_c_command dropped from 49.0% to 46.0%. It stayed below chance at every injection level tested.
- **Ablation over 0.01%, 0.1% and 1% injection:**
  - Most paradigms improve steadily as the share grows. For example, coordinate_structure_constraint_complex_left_branch went 46.5 → 52.7 → 59.3 → 62.4.
  - For principle_A_reconstruction, 0.01% (about 10K tokens, roughly 30 examples) already reached 72.3%.
  - only_npi_scope and existential_there_quantifiers_2 behaved non-monotonically.
- **Limitations:**
  - Only one architecture was tested.
  - Only 9 of the 67 BLiMP paradigms were targeted.
  - No injection shares above 1% were tried.
  - Effects on non-targeted paradigms can be positive or negative.
  - Code is available on GitHub.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: H S V N S Kowndinya Renduchintala and Sumit Bhatia. 2026. Heterogeneity in Formal Linguistic Competence of Language Models: Is Data the Real Bottleneck?. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{renduchintala2026heterogeneity,
  title = {{Heterogeneity in Formal Linguistic Competence of Language Models: Is Data the Real Bottleneck?}},
  author = {H S V N S Kowndinya Renduchintala and Sumit Bhatia},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.224},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.224}
}
```

- [← previous HeteroCache: A Dynamic Retrieval Approach to Heterogeneous KV Cache Compression for Long-Context LLM Inference](https://md.research.elamur.ai/free/en/papers/shi-2026-heterocache-dynamic-retrieval-approach-heterogeneous/)
- [next → Hidden Sentiments: The Impact of Low-level Adversarial Perturbations on Arabic Sentiment Analysis Services](https://md.research.elamur.ai/free/en/papers/abdelkader-2026-hidden-sentiments-impact-low-level/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/32/)

## References

17 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
2. Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, and 1 others. 2025. [gpt-oss-120b & gpt-oss-20b model card](https://md.research.elamur.ai/free/en/papers/agarwal-2025-gpt-oss-120b-gpt-oss/). arXiv preprint arXiv:2508.10925.
3. Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíček, Agustín Piqueres Lajarín, Vaibhav Srivastav, and 1 others. 2025. [Smollm2: When smol goes big–data-centric training of a small language model](https://md.research.elamur.ai/free/en/papers/allal-2025-smollm2-smol-goes-big-data/). arXiv preprint ar…
4. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
5. Michael Y Hu, Aaron Mueller, Candace Ross, Adina Williams, Tal Linzen, Chengxu Zhuang, Ryan Cotterell, Leshem Choshen, Alex Warstadt, and Ethan Gotlieb Wilcox. 2024. Findings of the second babylm challenge: Sample-efficient pretraining on developmentally plausible corpora. arXiv preprint arXiv:2412.… [find](https://scholar.google.com/scholar?q=Michael+Y+Hu%2C+Aaron+Mueller%2C+Candace+Ross%2C+Adina+Williams%2C+Tal+Linzen%2C+Chengxu+Zhuang%2C+Ryan+Cotterell%2C+Leshem+Choshen%2C+Alex+Warstadt%2C+and+Ethan+Gotlieb+Wilcox%2E+2024%2E+Findings+of+the+second+babylm+chal)
6. Cameron Robert Jones, Ishika Rathi, Sydney Taylor, and Benjamin K Bergen. 2025. People cannot distinguish gpt-4 from a human in a turing test. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency, pages 1615–1639. [find](https://scholar.google.com/scholar?q=Cameron+Robert+Jones%2C+Ishika+Rathi%2C+Sydney+Taylor%2C+and+Benjamin+K+Bergen%2E+2025%2E+People+cannot+distinguish+gpt%2D4+from+a+human+in+a+turing+test%2E+In+Proceedings+of+the+2025+ACM+Conference+on+Fairness%2C+Ac)
7. Shalom Lappin and Stuart M Shieber. 2007. Machine learning theory and practice as a source of insightinto universal grammar. Journal of Linguistics, 43(2):393– 427. [find](https://scholar.google.com/scholar?q=Shalom+Lappin+and+Stuart+M+Shieber%2E+2007%2E+Machine+learning+theory+and+practice+as+a+source+of+insightinto+universal+grammar%2E+Journal+of+Linguistics%2C+43%282%29%3A393%E2%80%93+427%2E)
8. Kyle Mahowald, Anna A Ivanova, Idan A Blank, Nancy Kanwisher, Joshua B Tenenbaum, and Evelina Fedorenko. 2023. Dissociating language and thought in large language models: a cognitive perspective. arXiv preprint arXiv:2301.06627, 2(3). [find](https://scholar.google.com/scholar?q=Kyle+Mahowald%2C+Anna+A+Ivanova%2C+Idan+A+Blank%2C+Nancy+Kanwisher%2C+Joshua+B+Tenenbaum%2C+and+Evelina+Fedorenko%2E+2023%2E+Dissociating+language+and+thought+in+large+language+models%3A+a+cognitive+perspective%2E+arXi)
9. Kanishka Misra and Kyle Mahowald. 2024. [Language models learn rare phenomena from less rare phenomena: The case of the missing AANNs](https://md.research.elamur.ai/free/en/papers/misra-2024-language-models-learn-rare-phenomena/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 913–929, Miami, Florida, USA. Association for Computational Linguisti…
10. Peter Norvig. n-grams. [https://norvig.com/](https://norvig.com/) ngrams/. Accessed: 2026-01-06. [↗](https://norvig.com/)
11. Team Olmo, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, and 1 others. 2025. [Olmo 3](https://md.research.elamur.ai/free/en/papers/ettinger-2025-olmo-3/). arXiv preprint arXiv:2512.13961.
12. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
13. Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro Von Werra, Thomas Wolf, and 1 others. 2024. [The fineweb datasets: Decanting the web for the finest text data at scale](https://md.research.elamur.ai/free/en/papers/penedo-2024-fineweb-datasets-decanting-web-finest/). Advances in Neural Information Processing Systems, 37:30811–30849.
14. Tri Dao. 2023. [Flashattention-2: Faster attention with better parallelism and work partitioning](https://md.research.elamur.ai/free/en/papers/dao-2023-flashattention-2-faster-attention-better/). arXiv preprint arXiv:2307.08691.
15. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and 1 others. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+and+1+others%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E)
16. Jill Gilkerson, Jeffrey A Richards, Steven F Warren, Judith K Montgomery, Charles R Greenwood, D Kimbrough Oller, John HL Hansen, and Terrance D Paul. 2017. Mapping the early language environment using all-day recordings and automated analysis. American journal of speech-language pathology, 26(2):24… [find](https://scholar.google.com/scholar?q=Jill+Gilkerson%2C+Jeffrey+A+Richards%2C+Steven+F+Warren%2C+Judith+K+Montgomery%2C+Charles+R+Greenwood%2C+D+Kimbrough+Oller%2C+John+HL+Hansen%2C+and+Terrance+D+Paul%2E+2017%2E+Mapping+the+early+language+environment+usin)
17. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R Bowman. 2020. [Blimp: The benchmark of linguistic minimal pairs for english](https://md.research.elamur.ai/free/en/papers/warstadt-2020-blimp-benchmark-linguistic-minimal-pairs/). Transactions of the Association for Computational Linguistics, 8:377–392.

*Updated 1 November 2026*