--- title: "Recurrent Networks Are (Linguistically) Better? An (Ongoing) Experiment on Small-LM — Fusco 2024" description: "The authors examine what small language models (SLMs) can learn when given roughly as much Italian speech as a child hears. Cited by 3 papers here." authors: ["Achille Fusco", "Matilde Barbini", "Maria Letizia Piccini Bianchessi", "Veronica Bressan", "Sofia Neri", "Sarah Rossi", "Tommaso Sgrizzi", "Cristiano Chesi"] year: 2024 venue: "Proceedings of the Tenth Italian Conference on Computational Linguistics (CLiC-it 2024)" source: "https://aclanthology.org/2024.clicit-1.46/" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/fusco-2024-recurrent-networks-are-linguistically-better/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/fusco-2024-recurrent-networks-are-linguistically-better/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/) / Recurrent Networks Are (Linguistically) Better? An (Ongoing) Experiment on Small-LM Training on Child-Directed Speech in Italian # Recurrent Networks Are (Linguistically) Better? An (Ongoing) Experiment on Small-LM Training on Child-Directed Speech in Italian **Authors:** [Achille Fusco](https://md.research.elamur.ai/free/en/authors/achille-fusco/), [Matilde Barbini](https://md.research.elamur.ai/free/en/authors/matilde-barbini/), [Maria Letizia Piccini Bianchessi](https://md.research.elamur.ai/free/en/authors/maria-letizia-piccini-bianchessi/), [Veronica Bressan](https://md.research.elamur.ai/free/en/authors/veronica-bressan/), [Sofia Neri](https://md.research.elamur.ai/free/en/authors/sofia-neri/), [Sarah Rossi](https://md.research.elamur.ai/free/en/authors/sarah-rossi/), [Tommaso Sgrizzi](https://md.research.elamur.ai/free/en/authors/tommaso-sgrizzi/) and [Cristiano Chesi](https://md.research.elamur.ai/free/en/authors/cristiano-chesi/) · **Year:** 2024 · **Venue:** Proceedings of the Tenth Italian Conference on Computational Linguistics (CLiC-it 2024) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2024.clicit-1.46/) · [PDF](https://aclanthology.org/2024.clicit-1.46.pdf) **Citations:** 3 in our library, of the 140,884 papers whose bibliographies we read ## Summary The authors examine what small language models (SLMs) can learn when given roughly as much Italian speech as a child hears. To this end, they compiled a corpus of about 3 million tokens. Its sources are the Italian section of CHILDES (adult speech only), subtitles of children's films from OpenSubtitles, Zecchino D'Oro songs, VoLIP telephone conversations, and fairy tales that are already out of copyright. In addition to the standard BPE and FastWordPiece, they tested their own tokenizer, MorPiece. It splits words into pseudo-morphemes according to the Tolerance and Sufficiency Principle. They also proposed eMG-RNN, an LSTM with modified gates that are separately responsible for the "merge" of a new token and for "move", i.e. keeping an element in memory. The models were evaluated on COnVERSA, a BLiMP-style set of minimal pairs. The single-layer eMG-RNN has a somewhat worse training loss than standard LSTMs, yet it better distinguishes linguistically relevant contrasts. The authors conclude from this that loss and accuracy are poor indicators of a model's linguistic competence. None of the models trained on this corpus came close to the performance of 7-year-old native-speaker children. ## Key points - **Corpus:** 2,973,879 tokens before cleaning, 2,431,038 after. The largest portions are fairy tales and subtitles. - **MorPiece:** its vocabulary has 55,049 units versus 96,028 for BPE. The segmentation is linguistically more natural: "farlo" → "far" + "+lo". However, the loss is slightly worse, and the tokenizer has almost no effect on the minimal pairs. - **Models and accuracy on COnVERSA:** GroNLP GPT-2 small (121M, pretrained on a much larger corpus): 0.73; eMG-RNN (73M): 0.61; LSTMx1 (36M): 0.52; LSTMx2 (65M): 0.48; BERT (113M): 0.43. - **Data feeding regime:** a fixed window of 60 tokens works best. Training plateaus after about 10–12 epochs, and the COnVERSA score stops improving after just 3 epochs. - **Infrastructure:** IUSS HPC with A100 GPUs. PyTorch 2.3.1, Transformers 4.42.4, lm-eval 0.4.3. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) Topics: [🧒 Psycholinguistics & language acquisition](https://md.research.elamur.ai/free/en/topics/psycholinguistics/) ## Cite Cite as: Achille Fusco, Matilde Barbini, Maria Letizia Piccini Bianchessi, Veronica Bressan, Sofia Neri, Sarah Rossi, Tommaso Sgrizzi and Cristiano Chesi. 2024. Recurrent Networks Are (Linguistically) Better? An (Ongoing) Experiment on Small-LM Training on Child-Directed Speech in Italian. Proceedings of the Tenth Italian Conference on Computational Linguistics (CLiC-it 2024). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{fusco2024recurrent, title = {{Recurrent Networks Are (Linguistically) Better? An (Ongoing) Experiment on Small-LM Training on Child-Directed Speech in Italian}}, author = {Achille Fusco, Matilde Barbini, Maria Letizia Piccini Bianchessi, Veronica Bressan, Sofia Neri, Sarah Rossi, Tommaso Sgrizzi and Cristiano Chesi}, booktitle = {Proceedings of the Tenth Italian Conference on Computational Linguistics (CLiC-it 2024)}, year = {2024}, url = {https://aclanthology.org/2024.clicit-1.46/} } ``` - [← previous Rasid at StanceEval: Fine-tuning MARBERT for Arabic Stance Detection](https://md.research.elamur.ai/free/en/papers/alshenaifi-2024-rasid-stanceeval-fine-tuning-marbert/) - [next → Semantically Corrected Amharic Automatic Speech Recognition](https://md.research.elamur.ai/free/en/papers/adnew-2024-semantically-corrected-amharic-automatic-speech/) - [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/)