Probably the most important, yet one of the most neglected, aspects of academic work is how academic staff mark papers submitted by postgraduates students. There has been little research done on the processes involved in that assessment (Tinkler & Jackson, 2000) and it seems evident that academic staff need assistance with the assessment of postgraduate work. An interesting article by Mullins and Kiley (2002 ‘It’s a PhD, not a Nobel Prize’: how experienced examiners assess research theses. Studies in Higher Education, 27, 4) addresses the following issues:
• What steps do examiners go through in the process of reading a thesis?
• What criteria do examiners use to assess a thesis? Are these criteria derived from institutional policies or are they based on the individual examiner’s understanding of what is required for a PhD or a master’s by research?
• Do examiners use different criteria for different groups of students (e.g. international students from non-English speaking backgrounds)?
• Are there influences on the examiners arising from their knowledge of the university, department or supervisor?
• What evidence do examiners collect as they read a thesis with a view to the formulation of their written reports?
• Are there critical points in the process of making judgements about a thesis which significantly influence the examiner’s final evaluation of the thesis?
I should emphasise that while a lot of what is discussed below relates to PhD work, I think it is almost equally applicable to MA dissertation work.
The authors review previous studies of examiners’ marks and comments on MA and PhD dissertations and then report on their own study which involved interviews with 35 academics responsible for marking PhD theses. While I can’t give any proper summary of their findings, I can report the following:
The questions that examiners have in mind as they read include the following:
• How would they have tackled the problem set out in the abstract and the title?
• What questions would they like answers to?
• Do the conclusions follow on from the introduction?
• How well does the candidate explain what he/she is doing?
• Is the bibliography up to date and substantial enough?
• Are the results worthwhile?
• How much work has actually been done?
• What is the intellectual depth and rigour of the thesis?
• Is this actually ‘research’—is there an argument?
But what most caught my attention was this: the authors came to the “overwhelming conclusion” that what counted most were examiners’ first impressions. The report suggests that examiners decide very early in the process whether assessment of a particular thesis is likely to be ‘hard work’ or ‘an enjoyable read’. The initial impressions of the quality of the thesis are usually formed by the end of the second or third chapter of the thesis—often by the end of the literature review. While the ‘traditional’ format of a thesis—introduction, literature review, methodology, results, conclusions is no longer universally accepted as appropriate, none of our interviewees indicated that the theses they read departed much from this model. Two examples given by the authors, verbatim comments from their interviewees, are typical of the influence of first impressions:
1. “A good indicator is the way the candidate reviews the literature and their overall grasp of what’s going on. If it looks as if the student grasps the problem then I read the rest with much more of a sympathetic view and I feel I can relax. If chapter 2 is not good, then I read the rest much more critically.”
2. “It is unusual that if someone does a poor job of the literature review that they will suddenly improve, or vice versa.”
So, take note!
On a more downbeat note, one of the most common descriptors of a poor thesis was ‘sloppiness’. Sloppiness might be demonstrated by typographical errors, or mistakes in calculations, referencing and footnotes. The concern with sloppiness was that examiners considered it was an indicator that the research itself might not be rigorous and the results and conclusions could not be trusted.
Characteristics of a poor thesis were:
• lack of coherence;
• lack of understanding of the theory;
• lack of con confidence;
• researching the wrong problem;
• mixed or confused theoretical and methodological perspectives;
• work that is not original;
• not being able to explain at the end of the thesis what had actually been argued in the thesis.
In contrast, a term used frequently to describe positive theses was ‘scholarship’, described by interviewees as originality, coherence, and a sense of student autonomy or independence: ‘The student makes the ideas their own’; ‘The original use of a concept or theoretical framework’.
The development of a well-structured argument was highly valued in a thesis. Within this term we clustered argument, conceptualisation, conclusion, design, logic and structure.
Most examiners looked for sufficient quantity as well as quality of work, with the frequent use of the word ‘substantial’.
When asked to comment on what they thought set apart a ‘good’ thesis from the rest, a unifying responses was the use of the artistic metaphor. For example, words and phrases such as the following were used to describe a good thesis:
• an artistic endeavour where the student is designing the work and there is elegance of design, of the synthesis, and executions;
• design—where it all fts together;
• a well-sculpted piece of work.
The use of the artistic metaphor extended to such terms as ‘elan’, ‘passion’, ‘excitement’ and ‘sparkle’. Clearly, these experienced examiners believed there was a level of ‘art’ involved in producing a particularly good thesis.
So, in brief, the authors’ research suggests the following:
• ‘sparkle, elan and sense of confidence with the material;
• cohesiveness and clarity;
• a student who makes the ideas his/her own, with some originality of presentation;
• professionalism—as demonstrated by mature comments, and the accuracy of the logic;
• style and sophistication.
The negative indicators are:
• poor references (“This is usually a sign of a poor thesis—the two go hand in hand”);
. Irritating things in the thesis such as typos and other careless textual mistakes that indicate a lack of attention to detail. Sloppiness in the text indicates sloppy research.
The final, substantive judgement is determined by:
• the student’s confidence and independence;
• a creative view of the topic;
• the structure of the argument;
• the coherence of theoretical and methodological perspectives; and
• evidence of critical self-assessment by the student.
And what about feedback? What are the criteria for markers concerning good feedback on postgrad. work?
I suggest the following:
1. Comment more on content than format.
2. Make sure that what you say is easily understood and refers clearly to the comment made.
3. Fully recognise and comment on the good elements.
4. Be objective. Especially in A.L there are very different views of most central concerns. Don’t start arguing with the student because you hold a different view.
5. Suggest ways that future work can be improved.
Some tips for students:
* Check the guidebook or handbook to your course and find the section that deals with evaluation. Read it carefully and ask your tutor for clarification if necessary. The question is: What do the markers see as the difference between an “A” and a “C”?
* Markers of postgrad. papers are particularly fussed about citations and reference lists. Make sure your papers use the system expected in your university. Make the OWL site a Favorite and constantly check. http://owl.english.purdue.edu/owl/
* Pay special attention to coherence and cohesion. Make use of numbered sections and sub-sections and of cohesive advices which tell the reader where you’re going. Always include an overview in the Introduction. Tell a story. I know this is all very postmodern talk, but try to see your paper as a narrative which leads the reader through a tell-me-more beginning, a really meaty main argument, and a good resolution. If you like music, see your paper as a sonata: statement; development; recapitulation.
* Demonstrate enthusiasm.
* Avoid generalisations and make limited use of anecdotes. Support any claims you make with references. Don’t use too many secondary references.
* Focus. Don’t take on too much in one paper. A single main argument, supported by a lit. review and 4 main points is ideal for a normal MA paper. And focus is even more important in dissertations: it’s just so easy to wander down tempting blind alleys.
* Don’t limit your paper to description – there must be evidence of a critical evaluation of the matters you describe.
* Proofread your paper carefully and ask a friend or expert colleague to read it before you submit it.
To return to the original discussion, I wonder, as a marker myself, what students like and dislike most about the feedback they get on their submitted work. Please tell us. I know replying to a blog is intimidating for many, but this is an area where we really should hear from as many people as possible. Scott Thornbury has proposed Dogme, Adrian Underhill and Jim Scrivener have proposed Demand High; I think I’m going to do a Manifesto proposing “Don’t Mark My Words”.