---
title: "SurreyCTS at BEA 2026 Shared Task 1: Semantic Funnelling"
description: "The University of Surrey team entered the open track of the BEA 2026 shared task. 12 references, cited by 1 paper here."
authors: ["Georgina Willoughby", "Jordan Painter", "Diptesh Kanojia", "Emily Wells", "Constantin Orasan"]
year: 2026
venue: "Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026)"
source: "https://aclanthology.org/2026.bea-1.70/"
doi: "10.18653/v1/2026.bea-1.70"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/willoughby-2026-surreycts-bea-2026-shared-task/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/willoughby-2026-surreycts-bea-2026-shared-task/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/17/) / SurreyCTS at BEA 2026 Shared Task 1: Semantic Funnelling and Entropy-based Multilingual Lexical Difficulty Prediction

# SurreyCTS at BEA 2026 Shared Task 1: Semantic Funnelling and Entropy-based Multilingual Lexical Difficulty Prediction

**Authors:** Georgina Willoughby, [Jordan Painter](https://md.research.elamur.ai/free/en/authors/jordan-painter/), [Diptesh Kanojia](https://md.research.elamur.ai/free/en/authors/diptesh-kanojia/), Emily Wells and [Constantin Orasan](https://md.research.elamur.ai/free/en/authors/constantin-orasan/) · **Year:** 2026 · **Venue:** [Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026)](https://md.research.elamur.ai/free/en/venues/bea/) ([BEA 2026](https://md.research.elamur.ai/free/en/venues/bea/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.bea-1.70/) · [PDF](https://aclanthology.org/2026.bea-1.70.pdf) · [DOI 10.18653/v1/2026.bea-1.70](https://doi.org/10.18653/v1/2026.bea-1.70) · [OpenAlex](https://openalex.org/W7166882984)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/willoughby-2026-surreycts-bea-2026-shared-task/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The University of Surrey team entered the open track of the BEA 2026 shared task. The task asks systems to predict how hard English vocabulary items are for learners whose first language is Spanish, German or Chinese. The targets are continuous GLMM (Rasch-type) difficulty estimates from the Extended KVL Dataset, which is based on British Council vocabulary lists and roughly 3.3 million responses from over 100,000 learners.

The authors started from multilingual encoders (XLM-R, mDeBERTa, COMET, RemBERT) and built a hybrid regressor. It mean-pools target-word states from several encoder layers ("semantic funnelling") and adds string-similarity, subword-ratio, attention-entropy and language-ID signals. These go into a deep MLP head trained with Huber loss. A redesigned, difficulty-stratified validation split that pools all three L1s proved essential for choosing models.

The final entry was a five-model ensemble weighted by inverse squared RMSE. It beat the organisers' open baseline for every L1 and ranked fifth among open-track teams. The work suggests that a few well-chosen linguistic features still help even with strong pretrained encoders.

## Key points
- **Best single models** (internal 500-item validation set):
  - prod-H (RemBERT + funnelling + language-ID prefix) had the lowest RMSE, 0.8122.
  - prod-G (COMET with minimal features) had the highest Pearson, 0.8968.
  - Vanilla RemBERT reached RMSE 0.8415.
- **Official test RMSE**, ensemble vs. open baseline:
  - Spanish: 1.034 vs. 1.198
  - German: 0.945 vs. 1.166
  - Chinese: 0.861 vs. 1.034
- **Validation split redesign:**
  - Encoder-only models stalled near RMSE 1.00.
  - The official 2,031-item dev set did not separate configurations reliably.
  - The authors split labelled data into 50 GLMM quantile buckets and took the 10 top-scoring items from each, giving a 500-item dev set biased toward harder items and 19,804 training items.
  - Mixing L1s in training forced entry into the open track.
- **Ablation findings:**
  - Targeted feature additions beat the full feature bundle.
  - TSA was the most useful entropy feature.
  - COMET worked best with minimal features, while RemBERT gained from the additions. The authors suggest COMET's smaller embedding size (d = 512 vs. 1152) may explain this.
- **L1 differences:**
  - For Spanish (−0.24, −0.27) and German (−0.34, −0.35) learners, normalised Levenshtein and Jaccard show an inverse relationship with difficulty.
  - For Chinese learners these correlations are near zero, even after pinyin romanisation.
  - Subword ratio behaves consistently across all three L1s.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Surrey](https://md.research.elamur.ai/free/en/universities/university-of-surrey/) — University, Guildford, United Kingdom · 22 works · 677 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 882 works

## Cite

Cite as: Georgina Willoughby, Jordan Painter, Diptesh Kanojia, Emily Wells and Constantin Orasan. 2026. SurreyCTS at BEA 2026 Shared Task 1: Semantic Funnelling and Entropy-based Multilingual Lexical Difficulty Prediction. Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{willoughby2026surreycts,
  title = {{SurreyCTS at BEA 2026 Shared Task 1: Semantic Funnelling and Entropy-based Multilingual Lexical Difficulty Prediction}},
  author = {Georgina Willoughby, Jordan Painter, Diptesh Kanojia, Emily Wells and Constantin Orasan},
  booktitle = {Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.bea-1.70},
  url = {https://doi.org/10.18653/v1/2026.bea-1.70}
}
```

- [← previous Surprisal and Metaphor Novelty Judgments: Moderate Correlations and Divergent Scaling Effects Revealed by Corpus-Based and Synthetic Datasets](https://md.research.elamur.ai/free/en/papers/momen-2026-surprisal-metaphor-novelty-judgments-moderate/)
- [next → sutta at SemEval-2026 Task 12: A Multi-Perspective Retrieve-Verify-Aggregate Framework for Abductive Event Reasoning](https://md.research.elamur.ai/free/en/papers/zou-2026-sutta-semeval-2026-task-12/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/17/)

## References

12 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Hyung Won Chung, Thibault Fevry, Henry Tsai, Melvin Johnson, and Sebastian Ruder. 2021. [Rethinking embedding coupling in pre-trained language models](https://md.research.elamur.ai/free/en/papers/chung-2020-rethinking-embedding-coupling-pre-trained/). In International Conference on Learning Representations.
2. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. [Unsupervised cross-lingual representation learning at scale](https://md.research.elamur.ai/free/en/papers/conneau-2020-unsupervised-cross-lingual-representation-learning/). In Proceedings of the 58th Annual Meeting of the Association fo…
3. Paul De Boeck. 2008. Random item irt models. Psychometrika, 73(4):533–559. [find](https://scholar.google.com/scholar?q=Paul+De+Boeck%2E+2008%2E+Random+item+irt+models%2E+Psychometrika%2C+73%284%29%3A533%E2%80%93559%2E)
4. Karen J. Dunn. 2024. Random-item rasch models and explanatory extensions: A worked example using l2 vocabulary test item responses. Research Methods in Applied Linguistics, 3(3):100143. [find](https://scholar.google.com/scholar?q=Karen+J%2E+Dunn%2E+2024%2E+Random%2Ditem+rasch+models+and+explanatory+extensions%3A+A+worked+example+using+l2+vocabulary+test+item+responses%2E+Research+Methods+in+Applied+Linguistics%2C+3%283%29%3A100143%2E)
5. Namrata Bhalchandra Patil Gurav, Akashdeep Ranu, Archchana Sindhujan, and Diptesh Kanojia. 2026. Domain-specific quality estimation for machine translation in low-resource scenarios. In Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026), pages 630– 650, R… [find](https://scholar.google.com/scholar?q=Namrata+Bhalchandra+Patil+Gurav%2C+Akashdeep+Ranu%2C+Archchana+Sindhujan%2C+and+Diptesh+Kanojia%2E+2026%2E+Domain%2Dspecific+quality+estimation+for+machine+translation+in+low%2Dresource+scenarios%2E+In+Proceedings+of)
6. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. [DeBERTa: Decoding-enhanced BERT with disentangled attention](https://md.research.elamur.ai/free/en/papers/he-2020-deberta-decoding-enhanced-bert-disentangled/). In International Conference on Learning Representations.
7. Peter J. Huber. 1964. Robust estimation of a location parameter. The Annals of Mathematical Statistics, 35(1):73–101. [find](https://scholar.google.com/scholar?q=Peter+J%2E+Huber%2E+1964%2E+Robust+estimation+of+a+location+parameter%2E+The+Annals+of+Mathematical+Statistics%2C+35%281%29%3A73%E2%80%93101%2E)
8. Diptesh Kanojia, Malhar Kulkarni, Pushpak Bhattacharyya, and Gholamreza Haffari. 2020. Challenge dataset of cognates and false friend pairs from Indian languages. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 3096–3102, Marseille, France. European Language Resourc… [find](https://scholar.google.com/scholar?q=Diptesh+Kanojia%2C+Malhar+Kulkarni%2C+Pushpak+Bhattacharyya%2C+and+Gholamreza+Haffari%2E+2020%2E+Challenge+dataset+of+cognates+and+false+friend+pairs+from+Indian+languages%2E+In+Proceedings+of+the+Twelfth+Languag)
9. Diptesh Kanojia, Prashant Sharma, Sayali Ghodekar, Pushpak Bhattacharyya, Gholamreza Haffari, and Malhar Kulkarni. 2021. Cognition-aware cognate detection. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3281–3292, On… [find](https://scholar.google.com/scholar?q=Diptesh+Kanojia%2C+Prashant+Sharma%2C+Sayali+Ghodekar%2C+Pushpak+Bhattacharyya%2C+Gholamreza+Haffari%2C+and+Malhar+Kulkarni%2E+2021%2E+Cognition%2Daware+cognate+detection%2E+In+Proceedings+of+the+16th+Conference+of+the)
10. Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022. Searching for COMETINHO: The little metric that could. In Proceedings of the 23rd Annual Conference of the European Association for Machine Tran… [find](https://scholar.google.com/scholar?q=Ricardo+Rei%2C+Jos%C3%A9+G%2E+C%2E+de+Souza%2C+Duarte+Alves%2C+Chrysoula+Zerva%2C+Ana+C+Farinha%2C+Taisiya+Glushkova%2C+Alon+Lavie%2C+Luisa+Coheur%2C+and+Andr%C3%A9+F%2E+T%2E+Martins%2E+2022%2E+Searching+for+COMETINHO%3A+The+little+metric+t)
11. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. [COMET: A neural framework for MT evaluation](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702. Association for Computational Linguistics.
12. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593– 4601, Florence, Italy. Association for Computational Linguistics. A [find](https://scholar.google.com/scholar?q=Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+4593%E2%80%93+4601%2C+Florence%2C+Italy%2E+Association+for+Computational+Linguistics%2E+A)

*Updated 2 November 2026*