---
title: "Recognizing English Learners' Native Language from Their"
description: "This paper describes a system for the first Native Language Identification (NLI) shared… 0.773 accuracy. 19 references, cited by 2 papers here."
authors: ["Baoli Li"]
year: 2013
venue: "Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications"
source: "https://aclanthology.org/W13-1715/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/li-2013-recognizing-english-learners-native-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/li-2013-recognizing-english-learners-native-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2013](https://md.research.elamur.ai/free/en/papers/2013/page/3/) / Recognizing English Learners' Native Language from Their Writings

# Recognizing English Learners' Native Language from Their Writings

**Authors:** [Baoli Li](https://md.research.elamur.ai/free/en/authors/baoli-li/) · **Year:** 2013 · **Venue:** [Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications](https://md.research.elamur.ai/free/en/venues/bea/) ([BEA 2013](https://md.research.elamur.ai/free/en/venues/bea/2013/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W13-1715/) · [PDF](https://aclanthology.org/W13-1715.pdf) · [OpenAlex](https://openalex.org/W2250623617)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · 2 in our library, of the 149,439 papers whose bibliographies we read

## Summary
This paper describes a system for the first Native Language Identification (NLI) shared task, NLI2013. The task is to guess an English learner's first language from an essay, which can help second language teaching and forensic authorship profiling. The data is TOEFL11, a corpus of TOEFL essays covering 11 L1s, released by ETS through the LDC. The author treats the problem as plain supervised text classification. Each essay becomes a TFIDF-weighted vector, and a linear SVM (Libsvm) is trained on it. The features are deliberately shallow: lowercased tokens, word 2–3-grams, character 2–5-grams, simple style counts and the essay metadata. Syntactic features were left out on purpose, so the study measures how far these traditional cues go on the first large open dataset. The full-feature run reached 0.773 accuracy, placing 18th of 29 teams in the closed track. Ablations show that the feature types help different L1s unevenly, which points toward class-specific feature sets. Character n-grams emerge as cheap but strong features, while style and metadata features add almost nothing.

## Key points
- TOEFL11 is described as 12,100 essays of roughly 300–400 words, split 9,900 train / 1,100 dev / 1,100 test. The eleven native languages span five Asian ones (Chinese, Japanese, Korean, Hindi and Telugu), four European ones (French, German, Italian and Spanish), plus Arabic and Turkish, and every essay is also tagged with a proficiency level (low/medium/high) and a Prompt ID.
- Four official runs were submitted:
  - HAUTCS-1, all features: 0.773.
  - HAUTCS-2, without character n-grams: 0.758.
  - HAUTCS-3, which also drops style: 0.760.
  - HAUTCS-4, words plus metadata only: 0.756.
- HAUTCS-1 was accidentally trained without the dev set. Retraining it on train+dev gave 0.781 after the evaluation.
- Per-language F1 for the best run peaked on German (about 0.90) and was lowest on Hindi (about 0.69). 10-fold cross-validation on train+dev gave 0.799, higher than the test score.
- A words-only run (HAUTCS-5) matched HAUTCS-4 at 0.756, so the metadata gave no measurable gain. Adding style features slightly lowered accuracy.

## Interesting ideas
- Character n-grams are taken over the whole sentence, with spaces replaced by a marker symbol and explicit start/end symbols added, so the n-grams capture affixes and cross-word spellings without a morphological analyzer.
- Feature groups are added run by run and per-class F1 is checked, revealing that each group helps only some classes (character n-grams suit Spanish, plain words suit Telugu).
- A words-only run is compared against words+metadata to test whether the metadata carries any signal, and it turns out to carry none.
- Paragraph-based style features were exposed to a train/test format mismatch, because the original test release had paragraph breaks stripped.
- Cross-validation accuracy (0.799) ran noticeably above held-out test accuracy (0.773), a reminder that in-distribution folds can be optimistic.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tokens): A boundary-aware character n-gram model could serve as a backoff signal for the tagger and lemmatizer on unknown, misspelled or learner-influenced word forms.
- **I2** (learn): Evaluation reports should break gains down per category, e.g. per dependency relation or per frame type, rather than relying on a single headline accuracy.
- **I3** (learn): This is a cheap negative control to adopt: retrain with one feature source removed and confirm the gain is real before keeping that source.
- **I4** (tokens): Keep paragraph and sentence boundary conventions identical across all corpora fed to the tokenizer, and flag inputs where structural markers are missing.
- **I5** (learn): Silver-data training should always be scored on a truly separate test set as well as internal folds, and the gap should be tracked.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 1,552 works · [💭 Social Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/social-psychology/) · 306 works

🧭 Topics (1): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,263 works

## Cite

Cite as: Baoli Li. 2013. Recognizing English Learners' Native Language from Their Writings. Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{li2013recognizing,
  title = {{Recognizing English Learners' Native Language from Their Writings}},
  author = {Baoli Li},
  booktitle = {Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications},
  year = {2013},
  url = {https://aclanthology.org/W13-1715/}
}
```

- [← previous Contemporary Amharic Oral Poetry from Gojjam: Classification and a sample Analysis](https://md.research.elamur.ai/free/en/papers/gelaye-2013-contemporary-amharic-oral-poetry-gojjam/)
- [next → (Pre-)Annotation of Topic-Focus Articulation in Prague Czech-English Dependency Treebank](https://md.research.elamur.ai/free/en/papers/mirovsky-2013-pre-annotation-topic-focus-articulation/)
- [↑ back to papers of 2013](https://md.research.elamur.ai/free/en/papers/2013/page/3/)

## References

19 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ahn, C. S. 2011. Automatically Detecting Authors' Native Language. Master's thesis, Naval Postgraduate School, Monterey, CA. [find](https://scholar.google.com/scholar?q=Ahn%2C+C%2E+S%2E+2011%2E+Automatically+Detecting+Authors%27+Native+Language%2E+Master%27s+thesis%2C+Naval+Postgraduate+School%2C+Monterey%2C+CA%2E)
2. Bergsma, S., Post, M., and Yarowsky, D. 2012. Stylometric analysis of scientific articles. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 327–337, Montréal, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Bergsma%2C+S%2E%2C+Post%2C+M%2E%2C+and+Yarowsky%2C+D%2E+2012%2E+Stylometric+analysis+of+scientific+articles%2E+In+Proceedings+of+the+2012+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Ling)
3. Blanchard, D., Tetreault, J., Higgins, D., Cahill, A., and Chodorow, M. 2013. TOEFL11: A Corpus of Non-Native English. Technical report, Educational Testing Service. [↗](https://doi.org/10.1002/j.2333-8504.2013.tb02331.x)
4. Brooke, J. and Hirst, G. 2012a. Measuring interlanguage: Native language identification with l1-influence metrics. In Calzolari, N., Choukri, K., Declerck, T., Dogan, M. U., Maegaard, B., Mariani, J., Odijk, J., and Piperidis, S., editors, Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC-2012), pages 779–784, Istanbul, Turkey Brooke, J. and Hirst, G. 2012b. Robust, Lexicalized Native Language Identification. In Proceedings of COLING 2012, pages 391-408, Mumbai, India. [find](https://scholar.google.com/scholar?q=Brooke%2C+J%2E+and+Hirst%2C+G%2E+2012a%2E+Measuring+interlanguage%3A+Native+language+identification+with+l1%2Dinfluence+metrics%2E+In+Calzolari%2C+N%2E%2C+Choukri%2C+K%2E%2C+Declerck%2C+T%2E%2C+Dogan%2C+M%2E+U%2E%2C+Maegaard%2C+B%2E%2C+Mariani%2C+J%2E%2C)
5. Bykh, S. and Meurers, D. 2012. Native Language Identification using Recurring n-grams - Investigating Abstraction and Domain Dependence. In Proceedings of COLING 2012, pages 425-440, Mumbai, India. [find](https://scholar.google.com/scholar?q=Bykh%2C+S%2E+and+Meurers%2C+D%2E+2012%2E+Native+Language+Identification+using+Recurring+n%2Dgrams+%2D+Investigating+Abstraction+and+Domain+Dependence%2E+In+Proceedings+of+COLING+2012%2C+pages+425%2D440%2C+Mumbai%2C+India%2E)
6. Chang, C.-C. and Lin C.-J. 2011. LIBSVM : a library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2:3:27:1-27. [find](https://scholar.google.com/scholar?q=Chang%2C+C%2E%2DC%2E+and+Lin+C%2E%2DJ%2E+2011%2E+LIBSVM+%3A+a+library+for+support+vector+machines%2E+ACM+Transactions+on+Intelligent+Systems+and+Technology%2C+2%3A3%3A27%3A1%2D27%2E)
7. Crossley, S. A. and McNamara, D. 2012. Detecting the First Language of Second Language Writers Using Automated Indices of Cohesion, Lexical Sophistication, Syntactic Complexity and Conceptual Knowledge. In Jarvis, S. and Crossley, S. A., editors, Approaching Language Transfer through Text Classification, pages 106-126. Multilingual Matters. [↗](https://doi.org/10.2307/jj.27195494.7)
8. Jarvis, S., Castañeda-Jiménez, G., and Nielsen, R. 2012. Detecting L2 Writers' L1s on the Basis of Their Lexical Styles. In Jarvis, S. and Crossley, S. A., editors, Approaching Language Transfer through Text Classification, pages 34-70. Multilingual Matters. [↗](https://doi.org/10.2307/jj.27195494.5)
9. Jarvis, S. and Paquot, M. 2012. Exploring the Role of n-Grams in L1 Identification. In Jarvis, S. and Crossley, S. A., editors, Approaching Language Transfer through Text Classification, pages 71-105. Multilingual Matters. [↗](https://doi.org/10.2307/jj.27195494.6)
10. Kochmar, E. 2011. Identification of a writer’s native language by error analysis. Master’s thesis, University of Cambridge. [find](https://scholar.google.com/scholar?q=Kochmar%2C+E%2E+2011%2E+Identification+of+a+writer%E2%80%99s+native+language+by+error+analysis%2E+Master%E2%80%99s+thesis%2C+University+of+Cambridge%2E)
11. Koppel, M., Schler, J., and Zigdon, K. 2005. Determining an author’s native language by mining a text for errors. In Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining, pages 624–628, Chicago, IL. ACM. [find](https://scholar.google.com/scholar?q=Koppel%2C+M%2E%2C+Schler%2C+J%2E%2C+and+Zigdon%2C+K%2E+2005%2E+Determining+an+author%E2%80%99s+native+language+by+mining+a+text+for+errors%2E+In+Proceedings+of+the+eleventh+ACM+SIGKDD+international+conference+on+Knowledge+discov)
12. Koppel, M., Schler, J., and Argamon, S. 2008. Computational methods in authorship attribution. Journal of the American Society for information Science and Technology, 60(1):9–26. [↗](https://doi.org/10.1002/asi.20961)
13. Li, B., and Vogel, C. 2010. Improving Multiclass Text Classification with Error-Correcting Output Coding and Sub-class Partitions. In Proceedings of the 23rd Canadian Conference on Artificial Intelligence, pages 4-15, Ottawa, Canada. [find](https://scholar.google.com/scholar?q=Li%2C+B%2E%2C+and+Vogel%2C+C%2E+2010%2E+Improving+Multiclass+Text+Classification+with+Error%2DCorrecting+Output+Coding+and+Sub%2Dclass+Partitions%2E+In+Proceedings+of+the+23rd+Canadian+Conference+on+Artificial+Intellig)
14. Swanson, B. and Charniak, E. 2012. Native language detection with tree substitution grammars. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 193–197, Jeju Island, Korea. [find](https://scholar.google.com/scholar?q=Swanson%2C+B%2E+and+Charniak%2C+E%2E+2012%2E+Native+language+detection+with+tree+substitution+grammars%2E+In+Proceedings+of+the+50th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+2%3A+Shor)
15. Tetreault, J., Blanchard, D., and Cahill, A. 2013. A report on the first native language identification shared task. In Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications. Atlanta, GA, USA. [find](https://scholar.google.com/scholar?q=Tetreault%2C+J%2E%2C+Blanchard%2C+D%2E%2C+and+Cahill%2C+A%2E+2013%2E+A+report+on+the+first+native+language+identification+shared+task%2E+In+Proceedings+of+the+Eighth+Workshop+on+Innovative+Use+of+NLP+for+Building+Educati)
16. Tofighi, P.; Köse, C.; and Rouka, L. 2012. Author’s native language identification from web-based texts. International Journal of Computer and Communication Engineering. 1(1):47-50 [find](https://scholar.google.com/scholar?q=Tofighi%2C+P%2E%3B+K%C3%B6se%2C+C%2E%3B+and+Rouka%2C+L%2E+2012%2E+Author%E2%80%99s+native+language+identification+from+web%2Dbased+texts%2E+International+Journal+of+Computer+and+Communication+Engineering%2E+1%281%29%3A47%2D50)
17. Torney, R.; Vamplew, P.; and Yearwood, J. 2012. Using psycholinguistic features for profiling first language of authors. Journal of the American Society for Information Science and Technology. 63(6):1256-1269. [↗](https://doi.org/10.1002/asi.22627)
18. Wong, S.-M. J. and Dras, M. 2011. Exploiting Parse Structures for Native Language Identification. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1600–1610, Edinburgh, Scotland, UK. [find](https://scholar.google.com/scholar?q=Wong%2C+S%2E%2DM%2E+J%2E+and+Dras%2C+M%2E+2011%2E+Exploiting+Parse+Structures+for+Native+Language+Identification%2E+In+Proceedings+of+the+2011+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+1600%E2%80%93)
19. Wong, S.-M. J., Dras, M., and Johnson, M. 2012. Exploring Adaptor Grammars for Native Language Identification. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 699–709, Jeju Island, Korea. [find](https://scholar.google.com/scholar?q=Wong%2C+S%2E%2DM%2E+J%2E%2C+Dras%2C+M%2E%2C+and+Johnson%2C+M%2E+2012%2E+Exploring+Adaptor+Grammars+for+Native+Language+Identification%2E+In+Proceedings+of+the+2012+Joint+Conference+on+Empirical+Methods+in+Natural+Language+Proc)

*Updated 1 November 2026*