Why is it not 52% or 55% or 45%? What is the scientific rationale for using a score of 50%? Unfortunately, the answer to all these questions is the same: We don’t know.
Scarpa Schoeman
Introduction
I started working as an academic in 2009 and from day 1 I was hooked on the idea that I could do things in the classroom that might help students to learn better. I spent years working on learning tasks, experimenting with different tools, changing my assessment plans, reading the literature, attending conferences, and talking to like-minded colleagues. I paid attention. I thought about pedagogical problems all the time but it took me almost 10 years to question the fundamental assumption that a passing grade should be a fixed percentage (i.e. 50%) across all assessment tasks.
If it’s true that:
- Not all assessments are the same.
- Not all assessors are the same.
- Not all patients are the same.
How is it possible that all pass marks can be the same?
Podcast
Schoeman, S. (2017). Standard setting. SAAHE podcast.
Scarpa Schoeman has published a variety of peer-reviewed articles and presented at international conferences on the topic of medical education and assessment. His research interests include assessment and standard setting, as well as the educational environment for medical students. His clinical interests and practice focus on Emergency Medicine. He also acts as Assessment consultant to the Colleges of Physicians, Obstetrics and Gynaecology, Paediatricians and Anaesthetists of South Africa. He is a Fellow of the Higher Education Academy in the United Kingdom and is a part-time tutor in Assessment and Standard setting for the CME at Dundee University.
Article
Norcini, J. J. (2003). Setting standards on educational tests. Medical Education, 37(5), 464–469.
Objective: This instalment in the series on professional assessment provides an introduction to methods of setting standards. Method: A standard is a special score that serves as a boundary between those who perform well enough and those who do not. The practical steps in selecting it include: deciding on the type of standard; deciding the method for setting it; selecting the judges; holding the meeting; calculating the cutpoint, and deciding what to do afterwards. Four of the more popular methods are illustrated for both written and clinical examinations. Results: The most important criteria for selecting a method for setting standards are whether it is consistent with the purpose of the test, based on expert judgement, informed by data, supported by research, transparent, and requires due diligence. The credibility of the standard will rely largely on the nature of the standard setters and the selection of a broadly representative and knowledgeable group is essential. After the standard has been set, it is important to ensure that stakeholders view the results as credible and that the pass rates have sensible relationships with other markers of competence. Conclusions: A standard is an expression of professional values in the context of a test’s purpose and content, the ability of the examinees, and the wider social or educational setting. Because standards are an expression of values, methods for setting them are systematic ways of gathering value judgements, reaching consensus and expressing that consensus as a single score on a test.