Friday, January 22, 2016

Test Validity (or Justification) and Beneficial/Negative Consequences

Almost finished a submission on a major theme of my Lit Rev: current paradigms in Language Test Validity. I'm looking at what Lyle, (Bachman & Palmer, 2010) has to say about "unintended detrimental consequences". I've got to reframe that for the language test score = indefinite leave to remain discourse. A language test like this is unlike any other. 

For example, say a student needs IELTS 7.5 to get her place at a UK University. Let's also agree, just for now, that IELTS scores have been fully justified/validated, and that the University admissions department knows full well, using past experience, that anyone with less than 7.5 will be unable to keep up, submit essays of the required standard, and will not graduate. Maybe they know that because until a few years ago the bar was set at 7.0, and they had a whole load of mushkila situations with L2 students. 

So our student gets 7.0. This looks like a "detrimental consequence" for her. She really, really wanted that particular course at that particular University. Now she has to take shit from her parents, teachers and friends, and make do with a place at a less prestigious University. But it ISN'T a detrimental consequence for her in the framework we're using. If she'd gone to her 1st choice University, she would have struggled, failed, and wasted at least a year of her life. Whereas, she'll graduate from her second best option, maybe an L1 University. Beneficial consequences for all stakeholders. 

Whereas. With language tests whose consequences are leave to remain or no leave to remain. You can see the beneficial consequences for those who reach the cut-score. But those who fail? Does their lower language ability mean they would have failed as a new Scot? That's the crux of it. I need to dig into the policy and politics of it, to see what arguments were advanced for a B1 cut-score, and how someone who didn't meet that requirement, an A2 learner perhaps, would be seen as sure to fail as a new Scot. 

Bachman, L. F., & Palmer, A. S. (2010). Language assessment in practice. Oxford, UK: Oxford University Press.

Kane, M. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50, 1–73