Assessment of papers

Probably the most important, yet one of the most neglected, aspects of academic work is how academic staff mark papers submitted by postgraduates students. There has been little research done on the processes involved in that assessment (Tinkler & Jackson, 2000) and it seems evident that academic staff need assistance with the assessment of postgraduate work. An interesting article by Mullins and Kiley (2002 ‘It’s a PhD, not a Nobel Prize’: how experienced examiners assess research theses. Studies in Higher Education, 27, 4) addresses the following issues:

• What steps do examiners go through in the process of reading a thesis?
• What criteria do examiners use to assess a thesis? Are these criteria derived from institutional policies or are they based on the individual examiner’s understanding of what is required for a PhD or a master’s by research?
• Do examiners use different criteria for different groups of students (e.g. international students from non-English speaking backgrounds)?
• Are there influences on the examiners arising from their knowledge of the university, department or supervisor?
• What evidence do examiners collect as they read a thesis with a view to the formulation of their written reports?
• Are there critical points in the process of making judgements about a thesis which significantly influence the examiner’s final evaluation of the thesis?

I should emphasise that while a lot of what is discussed below relates to PhD work, I think it is almost equally applicable to MA dissertation work.

The authors review previous studies of examiners’ marks and comments on MA and PhD dissertations and then report on their own study which involved interviews with 35 academics responsible for marking PhD theses. While I can’t give any proper summary of their findings, I can report the following:

The questions that examiners have in mind as they read include the following:

• How would they have tackled the problem set out in the abstract and the title?
• What questions would they like answers to?
• Do the conclusions follow on from the introduction?
• How well does the candidate explain what he/she is doing?
• Is the bibliography up to date and substantial enough?
• Are the results worthwhile?
• How much work has actually been done?
• What is the intellectual depth and rigour of the thesis?
• Is this actually ‘research’—is there an argument?

But what most caught my attention was this: the authors came to the “overwhelming conclusion” that what counted most were examiners’ first impressions. The report suggests that examiners decide very early in the process whether assessment of a particular thesis is likely to be ‘hard work’ or ‘an enjoyable read’. The initial impressions of the quality of the thesis are usually formed by the end of the second or third chapter of the thesis—often by the end of the literature review. While the ‘traditional’ format of a thesis—introduction, literature review, methodology, results, conclusions is no longer universally accepted as appropriate, none of our interviewees indicated that the theses they read departed much from this model. Two examples given by the authors, verbatim comments from their interviewees, are typical of the influence of first impressions:

1. “A good indicator is the way the candidate reviews the literature and their overall grasp of what’s going on. If it looks as if the student grasps the problem then I read the rest with much more of a sympathetic view and I feel I can relax. If chapter 2 is not good, then I read the rest much more critically.”

2. “It is unusual that if someone does a poor job of the literature review that they will suddenly improve, or vice versa.”

So, take note!

On a more downbeat note, one of the most common descriptors of a poor thesis was ‘sloppiness’. Sloppiness might be demonstrated by typographical errors, or mistakes in calculations, referencing and footnotes. The concern with sloppiness was that examiners considered it was an indicator that the research itself might not be rigorous and the results and conclusions could not be trusted.

Characteristics of a poor thesis were:
• lack of coherence;
• lack of understanding of the theory;
• lack of con confidence;
• researching the wrong problem;
• mixed or confused theoretical and methodological perspectives;
• work that is not original;
• not being able to explain at the end of the thesis what had actually been argued in the thesis.

In contrast, a term used frequently to describe positive theses was ‘scholarship’, described by interviewees as originality, coherence, and a sense of student autonomy or independence: ‘The student makes the ideas their own’; ‘The original use of a concept or theoretical framework’.

The development of a well-structured argument was highly valued in a thesis. Within this term we clustered argument, conceptualisation, conclusion, design, logic and structure.

Most examiners looked for sufficient quantity as well as quality of work, with the frequent use of the word ‘substantial’.

When asked to comment on what they thought set apart a ‘good’ thesis from the rest, a unifying responses was the use of the artistic metaphor. For example, words and phrases such as the following were used to describe a good thesis:
• an artistic endeavour where the student is designing the work and there is elegance of design, of the synthesis, and executions;
• creativity;
• design—where it all fts together;
• elegant;
• a well-sculpted piece of work.

The use of the artistic metaphor extended to such terms as ‘elan’, ‘passion’, ‘excitement’ and ‘sparkle’. Clearly, these experienced examiners believed there was a level of ‘art’ involved in producing a particularly good thesis.

So, in brief, the authors’ research suggests the following:

Positive indicators:
• ‘sparkle, elan and sense of confidence with the material;
• cohesiveness and clarity;
• a student who makes the ideas his/her own, with some originality of presentation;
• professionalism—as demonstrated by mature comments, and the accuracy of the logic;
• style and sophistication.

The negative indicators are:
• poor references (“This is usually a sign of a poor thesis—the two go hand in hand”);
. Irritating things in the thesis such as typos and other careless textual mistakes that indicate a lack of attention to detail. Sloppiness in the text indicates sloppy research.

The final, substantive judgement is determined by:
• the student’s confidence and independence;
• a creative view of the topic;
• the structure of the argument;
• the coherence of theoretical and methodological perspectives; and
• evidence of critical self-assessment by the student.

And what about feedback? What are the criteria for markers concerning good feedback on postgrad. work?

I suggest the following:

1. Comment more on content than format.

2. Make sure that what you say is easily understood and refers clearly to the comment made.

3. Fully recognise and comment on the good elements.

4. Be objective. Especially in A.L there are very different views of most central concerns. Don’t start arguing with the student because you hold a different view.

5. Suggest ways that future work can be improved.

Some tips for students:

* Check the guidebook or handbook to your course and find the section that deals with evaluation. Read it carefully and ask your tutor for clarification if necessary. The question is: What do the markers see as the difference between an “A” and a “C”?

* Markers of postgrad. papers are particularly fussed about citations and reference lists. Make sure your papers use the system expected in your university. Make the OWL site a Favorite and constantly check. http://owl.english.purdue.edu/owl/

* Pay special attention to coherence and cohesion. Make use of numbered sections and sub-sections and of cohesive advices which tell the reader where you’re going. Always include an overview in the Introduction. Tell a story. I know this is all very postmodern talk, but try to see your paper as a narrative which leads the reader through a tell-me-more beginning, a really meaty main argument, and a good resolution. If you like music, see your paper as a sonata: statement; development; recapitulation.

* Demonstrate enthusiasm.

* Avoid generalisations and make limited use of anecdotes. Support any claims you make with references. Don’t use too many secondary references.

* Focus. Don’t take on too much in one paper. A single main argument, supported by a lit. review and 4 main points is ideal for a normal MA paper. And focus is even more important in dissertations: it’s just so easy to wander down tempting blind alleys.

* Don’t limit your paper to description – there must be evidence of a critical evaluation of the matters you describe.

* Proofread your paper carefully and ask a friend or expert colleague to read it before you submit it.

To return to the original discussion, I wonder, as a marker myself, what students like and dislike most about the feedback they get on their submitted work. Please tell us. I know replying to a blog is intimidating for many, but this is an area where we really should hear from as many people as possible. Scott Thornbury has proposed Dogme, Adrian Underhill and Jim Scrivener have proposed Demand High; I think I’m going to do a Manifesto proposing “Don’t Mark My Words”.

6 thoughts on “Assessment of papers

  1. Hi

    So I am doing an (online) MA and the marker passed my paper because he said (and I am paraphrasing here) that he thought that my tutor must have seen a draft of the paper and felt her had no choice but to pass the paper….I was a bit annoyed but mostly hurt and then really embarrassed….it seems that my tutor and the marker were not looking for the same thing….and what is the point of a tutor if the final marker does not feel that the paper you have submitted to said tutor is good enough to get more than a bare pass? This has happened to me twice…I must say that marks improved in my last assignment but I do not know, these two Cs will forever haunt me…I have never seen a mark scheme (though I have asked for one) and perhaps it is all a matter of perceptions which makes me wonder just how valid my grades are…

    Like

    • Hi Lucabella,

      I’m sorry to hear of your bad experiences in the assessments of your papers.

      There are 4 main points here, so let’s deal with them 1 by 1.

      1. Help from your tutor.

      Most universities discourage tutors from commenting on near-finished drafts. Tutors are there to help with reading, academic writing, clarifying tasks, selecting topics, organising topics, and monitoring outlines of papers. They should not really go further than that; the problem with tutors commenting on nearly-finished drafts is that the student rightly gets the feeling that the paper now has a kind of “seal of approval”. So tutors should not say to tutees “I think this is a good paper and it should get a B”, for example. Tutors should do everything they can to help tutees improve their marks, partly by discussing papers that have already been marked.

      2. Comments made by markers.

      In my opinion, it is completely unacceptable for a marker to make the sort of comments you report. No marker should criticise a colleague to students for obvious reasons; no marker should say they “feel they have no choice”. Markers should explain every negative comment they make in terms of the paper itself and with reference to a published list of criteria for awarding A,B,C and D grades.

      3. Agreement on criteria for marking among markers.

      Most tutors also do marking. In any good university there are regular norming sessions among markers to make sure everybody interprets the criteria in the same way. At least 30% of all papers should be second marked by a small team of senior academic staff, so that mistakes, poor comments, and discrepancies are spotted and addressed before students get their final grades and comments.

      4. There should be an opportunity for all students to appeal against their marks or comments made by markers.

      The Student Handbook should clearly state the appeal procedure, and students should be able to ask their tutor, or the head of department, for help.

      Thanks for sharing your experiences with us, Lucabella, and I wish you well with the rest of your studies.

      Best,

      Geoff

      Like

  2. Hello Geoff: I am rereading this post after what we have seen on the blog at Leicester University lately. Your last comment above “there should be an opportunity for all students to appeal against their marks or comments made by markers”. Why does Leicester forbid any type of appeal?

    Like

  3. This may be slightly off-topic but I find not getting any feedback at all on entries in the VLE discussions is dispiriting. Especially when regular feedback has been promised. I’m not sure whether vague feedback is worse than no feedback. But I know that when a number of us said we weren’t sure what to do on a task to be told “it’s as simple as can be” was worse than no comment at all. I really don’t like not having questions answered too. I would like to have an extended discussion on feedback even by email, but I get the sense it’s not welcome.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s