--- title: "Same Spelling, Different Functions Mah:Evaluating Language" description: "Singlish sentence-final particles such as lah , hor or leh change their pragmatic function depending on tone. 1 reference." authors: ["Chan Young Jung"] year: 2025 venue: "Proceedings of the 39th Pacific Asia Conference on Language, Information and Computation" source: "https://aclanthology.org/2025.paclic-1.2/" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/jung-2025-same-spelling-different-functions-mah/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/jung-2025-same-spelling-different-functions-mah/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/18/) / Same Spelling, Different Functions Mah:Evaluating Language Models' Understanding of Singlish Particles # Same Spelling, Different Functions Mah:Evaluating Language Models' Understanding of Singlish Particles **Authors:** Chan Young Jung · **Year:** 2025 · **Venue:** [Proceedings of the 39th Pacific Asia Conference on Language, Information and Computation](https://md.research.elamur.ai/free/en/venues/paclic/) ([PACLIC 2025](https://md.research.elamur.ai/free/en/venues/paclic/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2025.paclic-1.2/) · [PDF](https://aclanthology.org/2025.paclic-1.2.pdf) ## Summary Singlish sentence-final particles such as *lah*, *hor* or *leh* change their pragmatic function depending on tone. Writing does not mark tone, so a text-only model has to work out the intended function from context. This paper asks how well language models manage that. The author mined the English part of the NUS-SMS corpus with regular expressions and kept 10 frequent, well-documented particles. Instances of tonally ambiguous particles were hand-labelled with their most likely functional variant. The particle functions were then split into four semantic groups, chosen so that no group holds near-synonymous options. The test is a cloze task: the particle is masked and the model must pick the right one from its group. Models tested: - BERT-base-uncased - BERT-base-multilingual-uncased - SingBERT - GPT-4.1 with zero-shot, definition-based and few-shot prompts Domain pretraining helped clearly: SingBERT reached 56.2% against 30.1% for general English BERT. Still, no setup did well in absolute terms. This suggests context cannot fully stand in for prosody, which matters for any contact language. ## Key points - The data come from the NUS-SMS corpus (over 55K messages). After cleaning (deduplication, dropping anonymization tokens, removing messages under three words, excluding pure substrate-language text), 4,910 particle occurrences remained, across 10 particles and 14 tone/function variants. - Four particles show tonal polysemy. *one* and *what* were left out because they share spelling with ordinary English words. - SingBERT scored 56.2% overall versus 30.1% for BERT-base. GPT-4.1 ranged from 23.8% to 66.4% depending on group and prompt type. - For GPT-4.1, definition prompting averaged 53.7%, few-shot 49.5% and zero-shot 45.0%. In Group 3, definitions raised accuracy from 23.8% to 46.7%, while few-shot reached only 30.0%. - Results by group: - Group 1 (question markers) was the easiest and most stable. - Group 2 defeated the general English models (15.0% and 18.9%). - Group 3 (mixed functions) kept every model below 47%. - Group 4 (persuasive/confirmatory) reached 60.4% with SingBERT. ## Interesting ideas - Restricting the cloze choices to groups whose members do not overlap in function avoids punishing a model for confusing labels that are near-synonyms anyway, so evaluation stays fair when the label set is fuzzy. - One written form is split into several tone-specific function entries, and each corpus instance is assigned to the nearest variant using context and syntax (statement, imperative, question). - Hiding a discourse particle and asking the model to restore it is a cheap probe of whether pragmatic knowledge is present, without needing full semantic annotation. - Giving an LLM the label definitions worked better than giving it worked examples, and was much steadier across groups. - Particles that share spelling with standard words (*one*, *what*) were excluded deliberately to prevent false matches when extracting by surface form. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (learn): when testing speech-act or implicature labels, we could score inside such non-overlapping candidate groups so that real errors are separated from arguable confusions. - **I2** (frames): prag could store sentence-final tags like "right?" or "eh" as several speech-act variants and choose one from clause type, keeping the choice explainable. - **I3** (learn): masking tag questions or hedges in our texts and checking whether prag predicts the right stance-marker class would give a quick regression test for pragmatics. - **I4** (learn): prompts for the LLM teacher tiers should lead with explicit definitions of frames or speech acts rather than relying mostly on few-shot examples. - **I5** (tokens): a dialect-particle lexicon in the en crate should flag such homographs and leave them to the tagger and parser rather than matching them by string. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* 📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works 🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works ## Cite Cite as: Chan Young Jung. 2025. Same Spelling, Different Functions Mah:Evaluating Language Models' Understanding of Singlish Particles. Proceedings of the 39th Pacific Asia Conference on Language, Information and Computation. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{jung2025same, title = {{Same Spelling, Different Functions Mah:Evaluating Language Models' Understanding of Singlish Particles}}, author = {Chan Young Jung}, booktitle = {Proceedings of the 39th Pacific Asia Conference on Language, Information and Computation}, year = {2025}, url = {https://aclanthology.org/2025.paclic-1.2/} } ``` - [← previous Rule-based Surface Realization of Romanian Weak Pronouns](https://md.research.elamur.ai/free/en/papers/gerstenberger-2025-rule-based-surface-realization-romanian/) - [next → Sarc7: Evaluating Sarcasm Detection and Generation with Seven Types and Emotion-Informed Techniques](https://md.research.elamur.ai/free/en/papers/xiong-2025-sarc7-evaluating-sarcasm-detection-generation/) - [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/18/) ## References 1 references, 1 of them with a card of ours. ↗ leads to the work. 1. David Ifeoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D’souza, Julia Kreutzer, Constantine Lignos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijhwani, Sebastian Ruder, and 1 others. 2021. [Masakhaner: Named entity recognition for african languages](https://md.research.elamur.ai/free/en/papers/adelani-2021-masakhaner-named-entity-recognition-african/). Transactions of the Association for Computational Linguistics, 9:1116–1131. *Updated 2 November 2026*