---
title: "Rater attitude towards emerging varieties of English: a new rater"
description: "The paper asks whether raters' attitudes towards a World Englishes (WE) variety affect the scores they give in a speaking test… 82 references."
authors: ["Tammy Huei-Lien Hsu"]
year: 2019
venue: "Language Testing in Asia 9"
source: "http://link.springer.com/article/10.1186/s40468-019-0080-0"
doi: "10.1186/s40468-019-0080-0"
licence: "CC BY (journal licence per DOAJ; Language Testing in Asia)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/hsu-2019-rater-attitude-towards-emerging-varieties/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/hsu-2019-rater-attitude-towards-emerging-varieties/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/7/) / Rater attitude towards emerging varieties of English: a new rater effect?

# Rater attitude towards emerging varieties of English: a new rater effect?

**Authors:** Tammy Huei-Lien Hsu · **Year:** 2019 · **Venue:** Language Testing in Asia 9 · **Publisher:** [Springer Science+Business Media](https://md.research.elamur.ai/free/en/publishers/springer-science-business-media/) · **Licence:** [CC BY (journal licence per DOAJ; Language Testing in Asia)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://doi.org/10.1186/s40468-019-0080-0) · [PDF](https://link.springer.com/content/pdf/10.1186/s40468-019-0080-0.pdf) · [DOI 10.1186/s40468-019-0080-0](https://doi.org/10.1186/s40468-019-0080-0) · [OpenAlex](https://openalex.org/W2938229689)

**Citations:** 4 citations (OpenAlex, 2026-10-14)

## Summary
The paper asks whether raters' attitudes towards a World Englishes (WE) variety affect the scores they give in a speaking test, which would be a new kind of rater effect. Hsu uses Indian English as the case. 96 teacher raters first scored six IELTS speech samples from Indian examinees on the four analytic IELTS criteria. They then filled in an online questionnaire, the "rater attitude instrument" (RAI), which measures how they view WE. The analysis used correlations between RAI scores and ratings, a MANOVA comparing attitude groups, Tukey contrasts, and a check of whether the RAI predicts scores. Raters with more favourable attitudes consistently gave higher scores than unfavourable ones on every criterion, and the correlations were moderate to strong. The RAI turned out to be a significant predictor of the scores awarded. This matters because IELTS formally accepts all standard varieties of English. A judge's hidden stance towards a variety can therefore threaten score fairness and validity. The author suggests treating this as fairness evidence to gather before testing, not only afterwards.

## Key points
- Participants and data: 96 teacher raters, six IELTS speech samples by Indian examinees, and the RAI (from Hsu 2016) completed online.
- Statistics: correlation analysis, MANOVA across the four analytic rating criteria, and Tukey contrasts between attitude groups.
- The group with the positive attitude scored higher than the group with the negative attitude on all four criteria.
- Rater attitude accounted for roughly 30% of the variance in test scores.
- Limitations named by the author:
  - no rigorous rater training;
  - Indian English was the only stimulus variety;
  - only descriptive tasks were used;
  - purely quantitative data;
  - a possible order effect, because raters' task order online was not enforced.

## Interesting ideas
- Profile the judges separately from the items: give every evaluator an independent attitude instrument, then use that profile to predict and explain how they score.
- Compare evaluator groups criterion by criterion rather than on one overall score, since in earlier work (Kim 2005) the bias appeared in analytic scores but not in holistic ones.
- Keep the stimuli fixed and vary only the judge, so that any score difference can be attributed to the judge's stance and not to the input.
- Collect the judgments first and the attitude questionnaire afterwards to avoid priming, and treat an unenforced order as a confound.
- Treat fairness towards language varieties as evidence gathered before deployment, not as a post-hoc check.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Each witness parser or LLM teacher tier could get a measured "disposition" on non-standard English inputs, so systematic disagreement in silver data can be traced to a known judge bias instead of being averaged away.
- **I2** (learn): Evaluation reports could break results down by separate dimensions (attachment, labels, roles), so a bias that cancels out in a single headline metric still becomes visible.
- **I3** (learn): This works as a negative control, where the same sentences are labelled by different teachers and divergence beyond chance flags teacher-specific bias.
- **I4** (learn): When an LLM teacher both labels data and answers meta-questions about a dialect, the labelling should come first in a fixed, logged order.
- **I5** (tree): Before release, the parser could be tested on outer-circle English texts (for example Indian English) to see whether attachment accuracy drops for these varieties.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

Also on the byline: [Fu Jen Catholic University](https://ror.org/04je98850).

📚 Shelves (2): [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [English](https://md.research.elamur.ai/free/en/topics/english/) · 1,373 works

## Cite

Cite as: Tammy Huei-Lien Hsu. 2019. Rater attitude towards emerging varieties of English: a new rater effect?. Language Testing in Asia 9.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{hsu2019rater,
  title = {{Rater attitude towards emerging varieties of English: a new rater effect?}},
  author = {Tammy Huei-Lien Hsu},
  booktitle = {Language Testing in Asia 9},
  year = {2019},
  doi = {10.1186/s40468-019-0080-0},
  url = {https://doi.org/10.1186/s40468-019-0080-0}
}
```

- [← previous Neural Lemmatization of Multiword Expressions](https://md.research.elamur.ai/free/en/papers/schmitt-2019-neural-lemmatization-multiword-expressions/)
- [next → Annotating Shakespeare's Sonnets with Appraisal Theory to Detect Irony](https://md.research.elamur.ai/free/en/papers/busetto-2019-annotating-shakespeare-s-sonnets-appraisal/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/7/)

## References

82 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abeywickrama, P. (2013). Why not non-native varieties of English as listening comprehension test input? RELC Journal, 44, 59– 74. [↗](https://doi.org/10.1177/0033688212473270)
2. Ajzen, I., & Fishbein, M. (1980). Understanding attitudes and predicting social behavior. Englewood Cliffs: Prentice Hall. [find](https://scholar.google.com/scholar?q=Ajzen%2C+I%2E%2C+%26+Fishbein%2C+M%2E+%281980%29%2E+Understanding+attitudes+and+predicting+social+behavior%2E+Englewood+Cliffs%3A+Prentice+Hall%2E)
3. Ajzen, I., & Timko, C. (1986). Correspondence between health attitudes and behavior. Basic and Applied Social Psychology, 7(4), 259–276. [↗](https://doi.org/10.1207/s15324834basp0704_2)
4. Albarracin, D., Johnson, B. T., & Zanna, M. P. (Eds.). (2005). Handbook of attitudes. Mahwah: Erlbaum. [find](https://scholar.google.com/scholar?q=Albarracin%2C+D%2E%2C+Johnson%2C+B%2E+T%2E%2C+%26+Zanna%2C+M%2E+P%2E+%28Eds%2E%29%2E+%282005%29%2E+Handbook+of+attitudes%2E+Mahwah%3A+Erlbaum%2E)
5. Bayyurt, Y., & Altinmakas, D. (2012). A WE-based English communication skills course at a Turkish university. In A. Matsuda (Ed. ), Principles and practices of teaching English as an international language (pp. 169–182). Bristol: Multilingual Matters. [↗](https://doi.org/10.2307/jj.27195482.15)
6. Berger, C. R., & Bradac, J. J. (1982). Language and social knowledge: Uncertainty in interpersonal relations. London: Arnold. [find](https://scholar.google.com/scholar?q=Berger%2C+C%2E+R%2E%2C+%26+Bradac%2C+J%2E+J%2E+%281982%29%2E+Language+and+social+knowledge%3A+Uncertainty+in+interpersonal+relations%2E+London%3A+Arnold%2E)
7. Bhatt, R. M. (2001). World Englishes [Electronic version]. Annual Review of Anthropology, 30, 527–550. [find](https://scholar.google.com/scholar?q=Bhatt%2C+R%2E+M%2E+%282001%29%2E+World+Englishes+%5BElectronic+version%5D%2E+Annual+Review+of+Anthropology%2C+30%2C+527%E2%80%93550%2E)
8. Brown, J. D. (2014). The future of World Englishes in language testing. Language Assessment Quarterly, 11, 5–26. [↗](https://doi.org/10.1080/15434303.2013.869817)
9. Browne, K., & Fulcher, G. (2017). Pronunciation and intelligibility in assessing spoken fluency. In T. Isaacs & P. Trofimovich (Eds.), Second language pronunciation assessment (pp. 37–53). Bristol: Multilingual Matters. [find](https://scholar.google.com/scholar?q=Browne%2C+K%2E%2C+%26+Fulcher%2C+G%2E+%282017%29%2E+Pronunciation+and+intelligibility+in+assessing+spoken+fluency%2E+In+T%2E+Isaacs+%26+P%2E+Trofimovich+%28Eds%2E%29%2C+Second+language+pronunciation+assessment+%28pp%2E+37%E2%80%9353%29%2E+Bristol%3A+Mu)
10. Canagarajah, S. (2006). Changing communicative needs, revised assessment objectives: testing English as an international language. Language Assessment Quarterly, 3(3), 229–242. [↗](https://doi.org/10.1207/s15434311laq0303_1)
11. Cargile, A. C., Giles, H., Ryan, E. B., & Y Bradac, J. J. (1994). Language attitudes as a social process: A conceptual model and new directions. Language & Communication, 14, 211–236. [↗](https://doi.org/10.1016/0271-5309(94)90001-9)
12. Chalhoub-Deville, M. (1995). Deriving oral assessment scales across different tests and rater groups. Language Testing, 12(1), 16–33. [↗](https://doi.org/10.1177/026553229501200102)
13. Chalhoub-Deville, M., & Wigglesworth, G. (2005). Rater judgment and English language speaking proficiency. World Englishes, 24(3), 383–391. [↗](https://doi.org/10.1111/j.0083-2919.2005.00419.x)
14. Cluver, A. D. (2000). Changing language attitudes: The stigmatization of Khoekhoegowap in Namibia. Language Problems and Language Planning, 24(10), 77–100. Council of Europe. (2001). Common European framework of reference for languages: Learning, teaching, assessment. Cambridge: Press Syndicate of the University of Cambridge. [find](https://scholar.google.com/scholar?q=Cluver%2C+A%2E+D%2E+%282000%29%2E+Changing+language+attitudes%3A+The+stigmatization+of+Khoekhoegowap+in+Namibia%2E+Language+Problems+and+Language+Planning%2C+24%2810%29%2C+77%E2%80%93100%2E+Council+of+Europe%2E+%282001%29%2E+Common+European+f)
15. D’ Angelo, J. (2012). WE-informed EIL curriculum at Chuko: Towards a functional, educated, multilingual outcome. In A. Matsuda (Ed.), Principles and practices of teaching English as an international language (pp. 121–136). Bristol: Multilingual Matters. [find](https://scholar.google.com/scholar?q=D%E2%80%99+Angelo%2C+J%2E+%282012%29%2E+WE%2Dinformed+EIL+curriculum+at+Chuko%3A+Towards+a+functional%2C+educated%2C+multilingual+outcome%2E+In+A%2E+Matsuda+%28Ed%2E%29%2C+Principles+and+practices+of+teaching+English+as+an+international+l)
16. Davidson, F. (1993). Testing English across cultures: summary and comments. World Englishes, 13(1), 113–115. [↗](https://doi.org/10.1111/j.1467-971x.1993.tb00013.x)
17. Davidson, F. (2006). World Englishes and test construction. In B. Kachru, Y. Kachru & C. Nelson (Eds.), The handbook of world Englishes (pp. 709–717). Malden: Blackwell Publishes Ltd. [find](https://scholar.google.com/scholar?q=Davidson%2C+F%2E+%282006%29%2E+World+Englishes+and+test+construction%2E+In+B%2E+Kachru%2C+Y%2E+Kachru+%26+C%2E+Nelson+%28Eds%2E%29%2C+The+handbook+of+world+Englishes+%28pp%2E+709%E2%80%93717%29%2E+Malden%3A+Blackwell+Publishes+Ltd%2E)
18. Davies, A., Hamp-Lyons, L., & Kemp, C. (2003). Whose norms? International proficiency tests in English. World Englishes, 22(4), 571–584. [find](https://scholar.google.com/scholar?q=Davies%2C+A%2E%2C+Hamp%2DLyons%2C+L%2E%2C+%26+Kemp%2C+C%2E+%282003%29%2E+Whose+norms%3F+International+proficiency+tests+in+English%2E+World+Englishes%2C+22%284%29%2C+571%E2%80%93584%2E)
19. Derwing, T., & Munro, M. (1997). Accent, intelligibility and comprehensibility: evidence from four L1s. Studies in Second Language Acquisition, 19, 1–16. [find](https://scholar.google.com/scholar?q=Derwing%2C+T%2E%2C+%26+Munro%2C+M%2E+%281997%29%2E+Accent%2C+intelligibility+and+comprehensibility%3A+evidence+from+four+L1s%2E+Studies+in+Second+Language+Acquisition%2C+19%2C+1%E2%80%9316%2E)
20. Eagly, A. H., & Chaiken, S. (1993). The psychology of attitude. Fort Worth: Harcourt Brace Jovanovich. [find](https://scholar.google.com/scholar?q=Eagly%2C+A%2E+H%2E%2C+%26+Chaiken%2C+S%2E+%281993%29%2E+The+psychology+of+attitude%2E+Fort+Worth%3A+Harcourt+Brace+Jovanovich%2E)
21. Educational Testing Service. (2014). ETS standards for quality and fairness. Princeton: Educational Testing Service. [find](https://scholar.google.com/scholar?q=Educational+Testing+Service%2E+%282014%29%2E+ETS+standards+for+quality+and+fairness%2E+Princeton%3A+Educational+Testing+Service%2E)
22. Elder, C., & Davies, A. (2006). Assessing English as a lingua franca. Annual Review of Applied Linguistics, 26, 282–301. [↗](https://doi.org/10.1017/s0267190506000146)
23. Elder, C., & Harding, L. (2008). Language testing and English as an international language: constraints and contributions. In Sharifian, F. and M. Clyne (eds.): Australian Review of Applied Linguistics (Special forum issue), 31(3), 34.1–34.11. [↗](https://doi.org/10.2104/aral0834)
24. Fazio, R. H., Powell, M. C., & Williams, C. J. (1989). The role of attitude accessibility in the attitude-to-behavior process. Journal of Consumer Research, 16(3), 280–288. [↗](https://doi.org/10.1086/209214)
25. Fazio, R. H., & Towles-Schwen, T. (1999). The MODE model of attitude-behavior processes. In S. Chaiken & Y. Trope (Eds.), Dual process theories in social psychology (pp. 97–116). New York: Guilford. [find](https://scholar.google.com/scholar?q=Fazio%2C+R%2E+H%2E%2C+%26+Towles%2DSchwen%2C+T%2E+%281999%29%2E+The+MODE+model+of+attitude%2Dbehavior+processes%2E+In+S%2E+Chaiken+%26+Y%2E+Trope+%28Eds%2E%29%2C+Dual+process+theories+in+social+psychology+%28pp%2E+97%E2%80%93116%29%2E+New+York%3A+Guilford%2E)
26. Field, J. (2003). Promoting perception: Lexical segmentation in L2 listening. ELT Journal, 57(4), 325–334. [↗](https://doi.org/10.1093/elt/57.4.325)
27. Fulcher, G. (2003). Testing second language speaking. London: Longman. [find](https://scholar.google.com/scholar?q=Fulcher%2C+G%2E+%282003%29%2E+Testing+second+language+speaking%2E+London%3A+Longman%2E)
28. Galloway, N., & Rose, H. (2015). Introducing Global Englishes. Abingdon: Routledge. [↗](https://doi.org/10.4324/9781315734347)
29. Garrett, P. (2010). Attitudes to language. key topics in sociolinguistics. Cambridge: Cambridge University Press. [find](https://scholar.google.com/scholar?q=Garrett%2C+P%2E+%282010%29%2E+Attitudes+to+language%2E+key+topics+in+sociolinguistics%2E+Cambridge%3A+Cambridge+University+Press%2E)
30. Giles, H., & Billings, A. C. (2004). Assessing language attitudes: Speaker evaluation studies. In A. Davies & C. Elder (Eds.), The handbook of applied linguistics (pp. 187–209). Malden: Blackwell. [↗](https://doi.org/10.1002/9780470757000.ch7)
31. Gu, L., & So, Y. (2014). Voices from stakeholders: What makes an academic English test “international”? Journal of English for Academic Purposes, 18, 9–24. [find](https://scholar.google.com/scholar?q=Gu%2C+L%2E%2C+%26+So%2C+Y%2E+%282014%29%2E+Voices+from+stakeholders%3A+What+makes+an+academic+English+test+%E2%80%9Cinternational%E2%80%9D%3F+Journal+of+English+for+Academic+Purposes%2C+18%2C+9%E2%80%9324%2E)
32. Hamid, M. O. (2014). World Englishes in international proficiency tests. World Englishes, 33(2), 263–277. [↗](https://doi.org/10.1111/weng.12084)
33. Hamp-Lyons, L., & Zhang, B. W. (2001). World Englishes: issues in and from academic writing assessment. In L. Flowerdew & M. Peacock (Eds.), Research perspectives on English for academic purposes (pp. 101–116). Cambridge: Cambridge University Press. [↗](https://doi.org/10.1017/cbo9781139524766.010)
34. Harding, L. (2008). The use of speakers with L2 accents in academic English listening assessment: A validation study. Unpublished doctoral dissertation, The University of Melbourne, Melbourne. [find](https://scholar.google.com/scholar?q=Harding%2C+L%2E+%282008%29%2E+The+use+of+speakers+with+L2+accents+in+academic+English+listening+assessment%3A+A+validation+study%2E+Unpublished+doctoral+dissertation%2C+The+University+of+Melbourne%2C+Melbourne%2E)
35. Henson, R. K. (2001). Understanding internal consistency reliability estimates: a conceptual primer on coefficient alpha. Measurement and Evaluation in Counseling and Development, 34, 177–189. [↗](https://doi.org/10.1080/07481756.2002.12069034)
36. Hrubes, D., Ajzen, I., & Daigle, J. (2001). Predicting hunting intentions and behavior: An application of the theory of planned behavior. Leisure Sciences, 23(3), 165–178. [↗](https://doi.org/10.1080/014904001316896855)
37. Hsu, T. H. L. (2016). Removing bias towards World Englishes: The development of a Rater Attitude Instrument using Indian English as a stimulus. Language Testing, 33 (3), 367–389. [↗](https://doi.org/10.1177/0265532215590694)
38. Huberty, C. J., & Petoskey, M. D. (2000). Multivariate analysis of variance and covariance. In H. Tinsley & S. Brown (Eds.), Handbook of applied multivariate statistics and mathematical modeling (pp. 183–208). New York: Academic Press. [↗](https://doi.org/10.1016/b978-012691360-6/50008-2)
39. Jenkins, J. (2000). The phonology of English as an international language: New models, new norms, new goals. Oxford: Oxford University Press. [find](https://scholar.google.com/scholar?q=Jenkins%2C+J%2E+%282000%29%2E+The+phonology+of+English+as+an+international+language%3A+New+models%2C+new+norms%2C+new+goals%2E+Oxford%3A+Oxford+University+Press%2E)
40. Jenkins, J. (2006). Current perspectives on teaching world Englishness and English as a lingua franca. TESOL Quarterly, 40(1), 157–181. [find](https://scholar.google.com/scholar?q=Jenkins%2C+J%2E+%282006%29%2E+Current+perspectives+on+teaching+world+Englishness+and+English+as+a+lingua+franca%2E+TESOL+Quarterly%2C+40%281%29%2C+157%E2%80%93181%2E)
41. Jenkins, J. (2015). Global Englishes: a resource book for students. Abingdon: Routledge. [find](https://scholar.google.com/scholar?q=Jenkins%2C+J%2E+%282015%29%2E+Global+Englishes%3A+a+resource+book+for+students%2E+Abingdon%3A+Routledge%2E)
42. Kachru, B. (1992). The other tongue (2nd ed.). Urbana: University of Illinois press. [find](https://scholar.google.com/scholar?q=Kachru%2C+B%2E+%281992%29%2E+The+other+tongue+%282nd+ed%2E%29%2E+Urbana%3A+University+of+Illinois+press%2E)
43. Kachru, B. (2001). World Englishes. In R. Mesthrie (Ed.), Concise encyclopedia of sociolinguistics (pp. 519–524). New York: Elsevier. [find](https://scholar.google.com/scholar?q=Kachru%2C+B%2E+%282001%29%2E+World+Englishes%2E+In+R%2E+Mesthrie+%28Ed%2E%29%2C+Concise+encyclopedia+of+sociolinguistics+%28pp%2E+519%E2%80%93524%29%2E+New+York%3A+Elsevier%2E)
44. Kachru, B., Kachru, Y., & Nelson, C. (2006). The handbook of world Englishes. Oxford: Blackwell. [find](https://scholar.google.com/scholar?q=Kachru%2C+B%2E%2C+Kachru%2C+Y%2E%2C+%26+Nelson%2C+C%2E+%282006%29%2E+The+handbook+of+world+Englishes%2E+Oxford%3A+Blackwell%2E)
45. Kang, O. (2008). Ratings of L2 oral performance in English: Relative impact of rater characteristics and acoustic measures of accentedness. Spaan Fellow Working Papers in Second or Foreign Language Assessment, 6, 181–205. [find](https://scholar.google.com/scholar?q=Kang%2C+O%2E+%282008%29%2E+Ratings+of+L2+oral+performance+in+English%3A+Relative+impact+of+rater+characteristics+and+acoustic+measures+of+accentedness%2E+Spaan+Fellow+Working+Papers+in+Second+or+Foreign+Language+As)
46. Kang, O., & Moran, M. (2014). Functional loads of pronunciation features in non-native speakers’ oral assessment. TESOL Quarterly, 48(1), 176–187. [find](https://scholar.google.com/scholar?q=Kang%2C+O%2E%2C+%26+Moran%2C+M%2E+%282014%29%2E+Functional+loads+of+pronunciation+features+in+non%2Dnative+speakers%E2%80%99+oral+assessment%2E+TESOL+Quarterly%2C+48%281%29%2C+176%E2%80%93187%2E)
47. Kim, H. J. (2005). World Englishes and language testing: the influence of rater variability in the assessment process of English oral proficiency. Unpublished doctoral dissertation. Iowa city: University of Iowa. [find](https://scholar.google.com/scholar?q=Kim%2C+H%2E+J%2E+%282005%29%2E+World+Englishes+and+language+testing%3A+the+influence+of+rater+variability+in+the+assessment+process+of+English+oral+proficiency%2E+Unpublished+doctoral+dissertation%2E+Iowa+city%3A+Univers)
48. Kirkpatrick, A. (2012). English as an Asian lingua franca: the “ Lingua Franca Approach” and implications for language education policy. Journal of English as a Lingua Franca, 1(1), 121–139. [find](https://scholar.google.com/scholar?q=Kirkpatrick%2C+A%2E+%282012%29%2E+English+as+an+Asian+lingua+franca%3A+the+%E2%80%9C+Lingua+Franca+Approach%E2%80%9D+and+implications+for+language+education+policy%2E+Journal+of+English+as+a+Lingua+Franca%2C+1%281%29%2C+121%E2%80%93139%2E)
49. Krueger, J. (2001). Null hypothesis significance testing: On the survival of a flawed method. The American Psychologist, 56, 16–26. [↗](https://doi.org/10.1037//0003-066x.56.1.16)
50. Kubota, R. (2001). Teaching world Englishes to native speakers of English in the USA. World Englishes, 20(1), 47–64. [↗](https://doi.org/10.1111/1467-971x.00195)
51. Kunnan, A. J. (2004). Test fairness. In M. Milanvoic, C. Weir, & S. Bloton (Eds.), European year of language conference papers, Barcelona (pp. 27–48). Cambridge, U.K.: CUP. [find](https://scholar.google.com/scholar?q=Kunnan%2C+A%2E+J%2E+%282004%29%2E+Test+fairness%2E+In+M%2E+Milanvoic%2C+C%2E+Weir%2C+%26+S%2E+Bloton+%28Eds%2E%29%2C+European+year+of+language+conference+papers%2C+Barcelona+%28pp%2E+27%E2%80%9348%29%2E+Cambridge%2C+U%2EK%2E%3A+CUP%2E)
52. Lee, H. (2012). World Englishes in a high school English case: A case from Japan. In A. Matsuda (Ed.), Principles and practices of teaching English as an international language (pp. 154–168). Bristol: Multilingual Matters. [find](https://scholar.google.com/scholar?q=Lee%2C+H%2E+%282012%29%2E+World+Englishes+in+a+high+school+English+case%3A+A+case+from+Japan%2E+In+A%2E+Matsuda+%28Ed%2E%29%2C+Principles+and+practices+of+teaching+English+as+an+international+language+%28pp%2E+154%E2%80%93168%29%2E+Bristol%3A)
53. Leung, C., & Lewkowicz, J. (2006). Expanding horizons and unresolved conundrums: language testing and assessment. TESOL Quarterly, 40(1), 211–234. [↗](https://doi.org/10.2307/40264517)
54. Linacre, J. M. (1989). Many-faceted Rasch measurement. Chicago: MESA. [find](https://scholar.google.com/scholar?q=Linacre%2C+J%2E+M%2E+%281989%29%2E+Many%2Dfaceted+Rasch+measurement%2E+Chicago%3A+MESA%2E)
55. Lindemann, S. (2002). Listening with an attitude: A model of native-speaker comprehension of non-native speakers in the United States of America. Language in Society, 31, 419–441. [↗](https://doi.org/10.1017/s0047404502020286)
56. Lowenberg, P. H. (2002). Assessing English proficiency in the expanding circle. World Englishes, 21(3), 431–435. [↗](https://doi.org/10.1111/1467-971x.00261)
57. Luoma, S. (2004). Assessing speaking. New York: Cambridge University Press. [find](https://scholar.google.com/scholar?q=Luoma%2C+S%2E+%282004%29%2E+Assessing+speaking%2E+New+York%3A+Cambridge+University+Press%2E)
58. Matsuda, A., & Friedrich, P. (2011). English as an international language: a curriculum blueprint. World Englishes, 30(3), 332–344. [↗](https://doi.org/10.1111/j.1467-971x.2011.01717.x)
59. McArthur, T. (2003). The Oxford guide to world Englishes. Oxford: Oxford University Press. [find](https://scholar.google.com/scholar?q=McArthur%2C+T%2E+%282003%29%2E+The+Oxford+guide+to+world+Englishes%2E+Oxford%3A+Oxford+University+Press%2E)
60. McKay, S. L. (2012). Teaching materials for English as an international language. In A. Matsuda (Ed.), Principles and practices of teaching English as an international language (pp. 70–83). Bristol: Multilingual Matters. [↗](https://doi.org/10.2307/jj.27195482.9)
61. McNamara, T. (1996). Measuring second language performance. London: Longman. [find](https://scholar.google.com/scholar?q=McNamara%2C+T%2E+%281996%29%2E+Measuring+second+language+performance%2E+London%3A+Longman%2E)
62. McNamara, T., & Roever, C. (2006). Language Testing: The Social Dimension. Oxford: Blackwell. [↗](https://doi.org/10.1111/j.1473-4192.2006.00117.x)
63. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13–103). New York: American Council on Education & Macmillan. [find](https://scholar.google.com/scholar?q=Messick%2C+S%2E+%281989%29%2E+Validity%2E+In+R%2E+L%2E+Linn+%28Ed%2E%29%2C+Educational+measurement+%283rd+ed%2E%2C+pp%2E+13%E2%80%93103%29%2E+New+York%3A+American+Council+on+Education+%26+Macmillan%2E)
64. Munro, M. J., & Derwing, T. M. (2006). The functional load principle in ESL pronunciation instruction: An exploratory study. System, 34, 520–531. [↗](https://doi.org/10.1016/j.system.2006.09.004)
65. Nunnally, J. C. (1978). Psychometric theory (2nd ed.). New York: McGraw-Hill. [find](https://scholar.google.com/scholar?q=Nunnally%2C+J%2E+C%2E+%281978%29%2E+Psychometric+theory+%282nd+ed%2E%29%2E+New+York%3A+McGraw%2DHill%2E)
66. Plonsky, L., & Oswald, F. L. (2014). How big is “big”? Interpreting effect size in L2 research. Language Learning, 64, 878–912. [find](https://scholar.google.com/scholar?q=Plonsky%2C+L%2E%2C+%26+Oswald%2C+F%2E+L%2E+%282014%29%2E+How+big+is+%E2%80%9Cbig%E2%80%9D%3F+Interpreting+effect+size+in+L2+research%2E+Language+Learning%2C+64%2C+878%E2%80%93912%2E)
67. Powers, D. E., Scheldi, M. A., Leung, S. W., & Butler, F. A. (1999). Validating the revised test of spoken English against a criterion of communicative success. TOEFL Research Report, 99(5), 63. [↗](https://doi.org/10.1177/026553229901600401)
68. Rubin, D. L. (1992). Nonlanguage factors affecting undergraduates’ judgments of nonnative English-speaking teaching assistants. Research in Higher Education, 33, 511–531. [↗](https://doi.org/10.1007/bf00973770)
69. Saito, K. & Shintani, N. (2016). Do native speakers of North American and Singapore English differentially percieve comprehensibility in second language speech? TESOL Quarterly, 50(2), 421–446. [find](https://scholar.google.com/scholar?q=Saito%2C+K%2E+%26+Shintani%2C+N%2E+%282016%29%2E+Do+native+speakers+of+North+American+and+Singapore+English+differentially+percieve+comprehensibility+in+second+language+speech%3F+TESOL+Quarterly%2C+50%282%29%2C+421%E2%80%93446%2E)
70. Saito, K., Trofimovich, P., Isaacs, T., & Webb, S. (2017). Re-examining phonological and lexical correlates of second language comprehensibility: the role of rater experience. In T. Isaacs & P. Trofimovich (Eds.), Second language pronunciation assessment (pp. 141–156). Bristol: Multilingual Matters. [find](https://scholar.google.com/scholar?q=Saito%2C+K%2E%2C+Trofimovich%2C+P%2E%2C+Isaacs%2C+T%2E%2C+%26+Webb%2C+S%2E+%282017%29%2E+Re%2Dexamining+phonological+and+lexical+correlates+of+second+language+comprehensibility%3A+the+role+of+rater+experience%2E+In+T%2E+Isaacs+%26+P%2E+Trofim)
71. Sewell, A. (2013). Language testing and international intelligibility: a Hong Kong case study. Language Assessment Quarterly, 10, 423–443. [↗](https://doi.org/10.1080/15434303.2013.824974)
72. Sharfian, F., & Marlina, R. (2012). English as an international language (EIL): An innovative academic program. In A. Matsuda (Ed.), Principles and practices of teaching English as an international language (pp. 140–153). Bristol: Multilingual Matters. [find](https://scholar.google.com/scholar?q=Sharfian%2C+F%2E%2C+%26+Marlina%2C+R%2E+%282012%29%2E+English+as+an+international+language+%28EIL%29%3A+An+innovative+academic+program%2E+In+A%2E+Matsuda+%28Ed%2E%29%2C+Principles+and+practices+of+teaching+English+as+an+international+la)
73. Shrout, P. E., & Yager, T. J. (1989). Reliability and validity of screening scales: Effective of reducing scale length. Journal of Clinical Epidemiology, 42, 69–78. [find](https://scholar.google.com/scholar?q=Shrout%2C+P%2E+E%2E%2C+%26+Yager%2C+T%2E+J%2E+%281989%29%2E+Reliability+and+validity+of+screening+scales%3A+Effective+of+reducing+scale+length%2E+Journal+of+Clinical+Epidemiology%2C+42%2C+69%E2%80%9378%2E)
74. Smith, L. E. (1992). Spread of English and issues of intelligibility. In B. B. Kachru (Ed.), The other tongue: English across cultures (2nd ed., pp. 75–90). Urbana: University of Illinois Press. [find](https://scholar.google.com/scholar?q=Smith%2C+L%2E+E%2E+%281992%29%2E+Spread+of+English+and+issues+of+intelligibility%2E+In+B%2E+B%2E+Kachru+%28Ed%2E%29%2C+The+other+tongue%3A+English+across+cultures+%282nd+ed%2E%2C+pp%2E+75%E2%80%9390%29%2E+Urbana%3A+University+of+Illinois+Press%2E)
75. Steenkamp, J. B. E., De Jong, M., & Baumgartner, H. (2010). Socially desirable response tendencies in survey research. Journal of Marketing Research, 47, 199–214. [↗](https://doi.org/10.1509/jmkr.47.2.199)
76. Streiner, D. L., & Norman, G. R. (2000). Health measurement scales. A practical guide to their development and use. New York: Oxford University Press. [find](https://scholar.google.com/scholar?q=Streiner%2C+D%2E+L%2E%2C+%26+Norman%2C+G%2E+R%2E+%282000%29%2E+Health+measurement+scales%2E+A+practical+guide+to+their+development+and+use%2E+New+York%3A+Oxford+University+Press%2E)
77. Taylor, L. (2002). Assessing learners’ English: but whose/which English (es)? Research Notes 10. Cambridge: University of Cambridge ESOL Examinations. [find](https://scholar.google.com/scholar?q=Taylor%2C+L%2E+%282002%29%2E+Assessing+learners%E2%80%99+English%3A+but+whose%2Fwhich+English+%28es%29%3F+Research+Notes+10%2E+Cambridge%3A+University+of+Cambridge+ESOL+Examinations%2E)
78. Taylor, L. (2005). Linguistic diversity: Language varieties and their implications for testing and assessment. Berlin: Paper presented at the Association of Language Testers in Europe (ALTE) Conference. [find](https://scholar.google.com/scholar?q=Taylor%2C+L%2E+%282005%29%2E+Linguistic+diversity%3A+Language+varieties+and+their+implications+for+testing+and+assessment%2E+Berlin%3A+Paper+presented+at+the+Association+of+Language+Testers+in+Europe+%28ALTE%29+Conferenc)
79. Villarreal, D., Loring, A., & Evans, K. (2014). Teaching World Englishes to undergraduates: tensions and pedagogical insights. Portland: Paper presented at American Association for Applied Linguistics Roundtable. March 22–25. [find](https://scholar.google.com/scholar?q=Villarreal%2C+D%2E%2C+Loring%2C+A%2E%2C+%26+Evans%2C+K%2E+%282014%29%2E+Teaching+World+Englishes+to+undergraduates%3A+tensions+and+pedagogical+insights%2E+Portland%3A+Paper+presented+at+American+Association+for+Applied+Linguistics)
80. Wigglesworth, G. (2001). Influences on performance in task-based oral assessment. In M. Bygate, P. Skehan, & M. Swain (Eds.), Researching pedagogic tasks, second language learning, teaching and testing (pp. 186–209). Harlow: Longman. [find](https://scholar.google.com/scholar?q=Wigglesworth%2C+G%2E+%282001%29%2E+Influences+on+performance+in+task%2Dbased+oral+assessment%2E+In+M%2E+Bygate%2C+P%2E+Skehan%2C+%26+M%2E+Swain+%28Eds%2E%29%2C+Researching+pedagogic+tasks%2C+second+language+learning%2C+teaching+and+testin)
81. Xi, X., & Mollau, P. (2011). Using raters from India to score a large-scale speaking test. Language Learning, 61(4), 1222–1255. [↗](https://doi.org/10.1111/j.1467-9922.2011.00667.x)
82. Zahn, C. J., & Hopper, R. (1985). Measuring language attitudes: the speech evaluation instrument. Journal of Language and Social Psychology, l4(2), 113–123. [↗](https://doi.org/10.1177/0261927x8500400203)

*Updated 24 October 2026*