Harmer on Testing

In this post, I’d like to suggest that Jeremy Harmer’s public pronouncements on testing fail to say anything new or interesting and demonstrate a regrettable lack of knowledge of the matters discussed.

First Presentation

At the 2015 IATEFL conference, Harmer gave a talk which you can see by clicking its title here: An uncertain and approximate business? Why teachers should love testing. Harmer’s basic thesis is that teachers should love testing because it’s a necessary part of their job, but it’s an opinion that’s brashly asserted rather than a proposition that’s reasonably argued. Harmer begins by listing objections to testing:

  • Tests don’t measure creativity.
  • Chomsky says “testing is an anathema.”
  • Some people on Facebook don’t like testing.
  • Testing 4 year olds is weird.
  • Testing is only a snapshot.
  • Some people are good at testing, some aren’t.

Note that none of these points is developed and that no coherent argument is attempted.

Harmer then gives reasons why teachers should love testing:

  • He got a Grade 1 in playing the tuba because there was a test, and he performed badly in a concert because there wasn’t a test. Testing is thus a powerful motivator.
  • Neurosurgeons and pilots must be tested. So we need tests.
  • Tests tell us where students are. ”A test if it’s well done will tell you how well your students have done.”
  • Tests are getting better. “The Pearson test of academic English is bloody wonderful. I’m saying that because I believe it, not just because they pay me.” The designers claim that their speech-recognition software evaluates speech “as reliably and accurately as any human being can. And I have no reason to doubt that, because the research behind it is er.., er.., massive.”
  • Lots of tests are bad. If you want to change testing you can moan or do something; so learn about tests and do something. .

I suggest that this talk makes no worthwhile contribution to our understanding of language testing and that we should expect more than this from a presentation held at prime time in the biggest room in the entire conference centre, and streamed live on the conference website.

Second Presentation

Harmer’s second presentation was a videoconference given as part of of the 2015 TOBELTA Online Conference. This is something of a volte-face, since here Harmer asks “Should teachers love tests or hate them?” and begins by confiding that the question is so knotty that it drives him “to schizophrenia”.

Harmer devotes the first 20 minutes of his talk to saying that while he agrees with Luke Meddings that testing is badly-affected by big business, and that the commodification of language is a bad thing, he still thinks that neurosurgeons and pilots should be properly assessed. Harmer spends the rest of the hour variously stating the view that teachers need to become “test literate” experts in the field of testing. At one point Harmer says that teachers need to know about concepts of validity, reliability, and test item types, and at another point he says that knowledge of the two “profound concepts” of content validity and construct validity is vital if teachers are to “get inside the test.”

One other point that can be identified in the talk is that teachers and students should explore testing together. Students and teachers should “discuss together what it is they need to do and want to do with the full understanding of how a test works.”

Harmer concludes:

How do you stop a huge corporation dominating the testing world? How do you stop tests being designed that are absurd and ridiculous? And, guess what? I have no easy answer to that… but I know perfectly well that there’s no merit in, or virtue in complaining about this in private, and, by the way, I say this absolutely genuinely, the reason why listening to Luke and others is so important is that it was not a private event, it was a public event and the more of us who are public about what we think, the greater the opportunity is that, er, things might change.

In evidence here is Harmer’s hallmark combination of gushing sincerity and jibberish.

I suggest that we should expect more than these empty words from an invited speaker at an international conference. The presentation is poorly-structured, seriously lacking in coherence and cohesion, and very low in substantial content. Watching the video, it becomes clear that Harmer doesn’t have a good grasp of even the basic vocabulary of testing, and that he’s unable to offer anything informative or well-considered to a discussion of the uses and abuses of language testing.

Harmer’s Website

Finally, Harmer’s website offers some thoughts on testing in the post Testophile or Testophobe? I leave it the viewer to decide on its merits.

Conclusion

To seriously address the question of the pros and cons of language testing, we have to look not just at how tests are designed, but at how they’re used, an area which Harmer hardly mentions. I would argue that classroom teaching should be 100% test-free, while the use of large-scale tests should be carefully restricted. As Fulcher (2009) says, large-scale testing is a social tool used to ration limited resources and opportunities, and it’s currently being used “to carry a larger social burden than it can reasonably bear.” It should not, for example, as Fulcher (2011) argues, be used to implement immigration policies, to evaluate teachers, or to rank order schools. A place remains for testing, of course, but, on the basis of his latest offerings, Harmer is unlikely to be of much help in deciding what that place might be.

References

The 2 references to Fulcher can be found on Glenn Fulcher’s excellent website here: http://languagetesting.info/gf/glennfulcher.php Scroll down 4 pages till you come to “Selected Papers”. I particularly recommend his 2009 article “Test use and political philosophy” which you can download from his website. At the end of the article, Fulcher proposes an “effect-driven test architecture” which he hopes can serve “as a method for testers to proscribe unintended uses of their tests.”

Advertisements

19 thoughts on “Harmer on Testing

  1. Ok, well that’s sorted, then.

    But is it fair to condemn Jeremy for failing to develop his argument when you throw out this tantalizingly undeveloped morsel: ‘I would argue that classroom teaching should be 100% test-free’. Ok, argue it. Seriously, I’d like to hear your argument, not least because you once dismissed my own qualms about testing as ‘drivel’. Here: https://scottthornbury.wordpress.com/2013/02/10/o-is-for-outcomes/#comment-9177

    Likewise, ‘A place remains for testing, of course’. That ‘of course’ presupposes we agree with you and that we know what that place is. Speaking for myself, we don’t.

    Like

  2. Hi Scott,

    1. I think it’s fair to criticise Jeremy for failing to develop ANY argument in the talk.
    2. I don’t think it’s fair to accuse me of not arguing the case for not including testing in classroom teaching right then at the end of the post, but I’ll be pleased to do so, now you’ve asked.
    3. This is the new PC me, so I apologise for calling your post about testing “drivel”. Having looked at my comment I also apologise for it’s unnecessary length and borrishness.
    4. I don’t think the “of course” presupposes that we know what the precise place of testing is, just that it has some place. I don’t claim to know what that place is, although, as a quick answer I’d agree with Glenn Fulcher’s view that testing assumes that scores are indexical of the linguistic abilities required for a real world communicative purpose. To be valid, such scores should be genuinely useful for making decisions about an individual’s likely communicative success. As Fulcher (2011) says “This protects the individual: “Don’t go to college now, do some more academic English so you don’t fail.” And it protects society: “You aren’t ready to practice as an air traffic controller yet.”

    Like

  3. Pingback: Harmer’s Online Presentation on Testing | aplinglink

  4. I have long worried whether a language isn’t ‘commodified’, turned into a commodity, the minute someone charges a fee to teach it, and that there is something meretricious, therefore, about our profession as such. I felt the same about critical thinking during the few years when I was asked to teach that. Surely, helping each other to communicate, like subjecting each other’s arguments to critical scrutiny, is itself a normal part of communication and, as such, should be a normal part of our social lives together, not a good to be traded at market. The natural reply, it seems to me, is that if all those of us who are at present professional language teachers had to make our living doing something else then the availability of whatever help it is that we provide in language acquisition would be much less than it is now. Anyway, this is the best answer I can think of. It does not involve, though, denying that charging a fee to teach a language entails treating it as a commodity. If this reply is correct, then, it would seem that the charge of ‘commodification’ is not of itself a decisively damning one.

    Like

    • Hi Patrick,

      Yes the construct “commodification” is certainly shot through with political bias, but it’s a term that I think Luke Meddings uses to effect in his (very slow, but finally interesting) talk at the conference where Harmer gave Presentation 2, reviewed above. You can see it here: https://www.youtube.com/watch?v=gc7R24IzCs0

      Meddings says a bit more about Dogme’s attitude towards the commodification of everyday life on his blog here http://lukemeddings.com/2014/03/a-catalyst-for-change/ and I agree with everything he says.

      Like

      • I must say, having spent the last hour watching Meddings’ talk right the way through, and then watching part of one of his talks about Dogme, that he’s a poor public speaker: he looks and sounds as if he’s about to fall asleep. He should ask his co-author to give him a few tips. Tip 1: Liven up: put some enthusiasm into what you’re saying.

        Like

      • Hmmm. I’ve read the Luke Meddings article you helpfully sent me a link to. Thanks. It’s certainly very likeable. There are a few things I’m inclined to say, all of them still somewhat underdeveloped. I’ll confine myself for the time being to what is exercising me most, the paragraph that begins,

        ‘Because it’s a model which says that economics is everything …’

        I don’t think anyone thinks that ‘economics is everything.’ Everyone acknowledges that there is love and friendship and jokes and music and spirituality and so on (ELT coursebooks, for all their failings, cannot be accused of neglecting these topics, except maybe the last one – oh, and now I think about it, also the first one since it involves the last one). Still, what ails large sections of the population in Greece and Spain and elsewhere is that insistence on austerity is leading to widespread unemployment, is giving rise to the perverse consequence that, while there is work urgently needing to be done there are able adults who cannot find employment. Analysing why this is so and how it might be changed is, erm, economics. To be fair, Meddings does go on to emphasise the importance of economic critique but the fact that he begins his paragraph by apparently bemoaning an over-emphasis on economics suggests, to me, a degree of confusion on the issue. I’m inclined to think that when he says that the problem is a model which says that economics is everything what he means is that the problem is a model that says the market is everything that we need to think about when we think about economics. Would you say that this is fair? Would you agree with me that if we wish to address inequality and suffering we need to think about economics more, not less?

        Like

      • I really need to stop doing this but I can’t help it. Towards the end of his article, Meddings bemoans our excessive consumerism. Left-leaning economists such as Krugman, Stiglitz and Varoufakis, if I have understood them correctly, diagnose our present malaise as one of insufficient consumption and recommend, as an antidote to the suffering imposed by austerity, measures that will increase, not decrease, consumption.

        Like

      • Hi Patrick,

        I agree with you that Meddings needs to be more careful with his use of the word “economics.” I take him to be referring to advanced capitalism’s tendency to rob individuals of directly lived experiences and to substitute it with the consumption of commodities. I think Meddings is suggesting that education in general, and ELT in particular is being increasingly viewed as one more commodity, which is packaged in such a way that it can be marketed and sold, with maximum profit being the prime objective. He sometimes refers to the Situationists, whose key construct was the “Society of the Spectacle”, a society characterised by social alienation and commodity fetishism I have a great deal of sympathy with such views, but I agree that we need to be clear about what we’re talking about, and that we also need to think realistically about the realities of economic life today.

        This is

        Like

      • Hi Geoff

        We’ve come some way from the topic of testing, but I’m intrigued, remembering your dislike of Derrida, by your invocation of ‘directly lived experiences.’ Hasn’t Derrida, following Heidegger’s investigations into fundamental ontology, done more than anyone to explore what might be meant by ‘directly lived experiences’? I realise I’m now quite a long way off topic.

        Like

      • To put things another way, it cannot have escaped readers of your work that one of your own key constructs is the construct ‘construct.’ I’m interested to know whether or not you think that ‘directly lived experience’ is a construct?

        Like

      • Hi Patrick,

        I don’t agree with Derrida that we must be “responsible guardians of the heritage of transcendental idealism” (Rogues,2005, p. 134); I don’t share Derrida’s objective of undermining the Western concept of rationality and all the presuppositions that underlie our ideas of science, common sense, reality, etc., and I don’t think Derrida’s method of discourse analysis (popularly called “deconstructing texts”) has led to many good results, especially when attempted by fans of his who study sociolinguistics.

        To answer your question, no, I don’t think Derrida has done more than anyone to explore what we mean by “directly lived experiences”. I used this phrase in reference to Debord’s Society of the Spectacle, which begins “In societies where modern conditions of production prevail, all of life presents itself as an immense accumulation of spectacles. Everything that was directly lived has moved away into a representation” (https://www.marxists.org/reference/archive/debord/society.htm). You may say that this has a lot in common with existentialism in general and with Heidegger in particular. Maybe Heidegger’s concern with the self, with “authenticity” as the norm of self-identity, and with defining oneself in terms of choice, and commitment can be seen as influences on Debord, but Derrida obviously didn’t influence Debord, and, to the limited extent that I understand him, I don’t think Derrida explores Heidegger’s work as well as Marx or Camus, for example . .

        In order for “directly-lived experience” to be a construct, it would have to be used as a key term in a theory or hypothesis. I’m not aware of Derrida having used the expression in this way, although I’m sure you’ll tell us if he did.

        Like

      • Debord appears to be using the expression ‘everything that was directly lived’ as a key term in a theory. Still, the topic is remote from, and only tangentially related to, your post. Sorry. I really shouldn’t have brought it up.

        Like

      • Hi Patrick,

        Please don’t apologise just because we might be the only 2 people interested in this – blogs are not subject to the democratic will!

        I think you’re right; it is a key construct for Debord, but he doesn’t define it in any strict way because he doesn’t follow the norms of academic discourse.

        Like

      • Or because it cannot be defined in any strict way, though it is the implicit grounds for all of our definitions of anything. Intentional experience is surely preupposed in any realist, scientistic ontology, though it cannot be accomodated within any such ontology. Fodor, apparently, has come to feel that even the theory of natural selection founders on the rocks of intentionality.

        Like

  5. I have not read Fulcher but your quotes suggest that he takes a pragmatic, utilitarian view of testing, a view which must, surely, be right. A question arises, and this is at the nub of my disagreement with Steve in his recent post about performativity; what do we do when the criteria being employed (which, in Scotland, as Steve explains, are indeed often woeful) are really not especially valid (in that their usefulness is questionable) but where better criteria are not available and where the criteria being employed may be, at least, better than nothing?

    Like

  6. Hi Patrick,

    I suppose the answer is to insist on the use of better criteria and to look for alternatives until its replaced by a better test. From a pragmatic point of view, if you have to work with the test despite all its shortcomings, you should, ideally, use it while at the same time organising resistance to it and drawing attention to whatever injustices it causes.

    Like

  7. Good recent post by Achilleas Kostoulas here, I think, which illustrates your point about combining resistance with a pragmatic perspective: http://wp.me/p2aFDK-199 Inspiring to me as example of how an educator can build potential impact/washback of test into decision-making on whether or not and how to implement – not just doing it because you’re supposed to & ignoring consequences

    Like

  8. Pingback: The 2015 Quiz: A Round-up of High and Low Moments in ELT, 2015. | aplinglink

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s