
Comparative Assessment In Pharmacology Multiple Choice Questions Versus Essay With Focus On Gender Differences
Correspondence Address :
Dr. Preethi G Pai
Department of Pharmacology
Kasturba Medical College, P.B. No.53,
Light House Hill Road, Hampanakatta, Mangalore-575001 Karnataka, (India)
E-mail: meddocpai2@yahoo.com;
Tele: Office: 91-0824-242227; Resi-91-9880750040 Fax: 91 824 2428183
Background: Assessment is a driver of innovation and changes in education. In this study, medical student performances in pharmacology short essay questions (SEQ) and multiple choice questions (MCQ) were assessed and compared and gender differences were also explored.
Methods: A total of 712 second year students with records of scores in SEQ and MCQ papers were analyzed. The performances of students who scored > 60% marks in theory papers were compared with those who scored > 60% marks in MCQ and vice versa. Gender differences in performances were also studied. Statistical analysis was done using McNemar, Chi-Square and Pearson’s co-efficient of correlation. P values < 0.05 were considered to be significant.
Results: The analysis showed that the probability of students scoring > 60% marks in SEQ papers was higher when they had scored > 60% marks in MCQ papers (P < 0.001). Their SEQ performance positively correlated with their MCQ performance (r= 0.768, r= 0.599 and r=0.767 for the first, second and third tests respectively). Females scored higher than males in all tests.
Conclusion: Literature indicates that MCQ as an assessment tool is debatable. This study indicates: performance in MCQ is an effective tool to predict SEQ paper performance; and correlates with global data on better performances by female counterparts.
Pharmacology, assessment, essays, multiple choice questions
Introduction
Review of the medical school curriculum and the evolutionary changes in curricula based on these reviews are compulsory requirements in medical schools across the globe. Some reasons which necessitate these innovations are: changing disease patterns; translation of medical education in health care professionals to nation building; exponential rise in medical student numbers globally and their optionless requirement of the highest quality of medical education; besides the desire for change as a primary requirement in man’s endeavors. To stand the test of purposeful effective change that translates to better patient care, these educational innovations need to be assessed at the student level, to know if the service delivery component has been conveyed appropriately through the curriculum. Care has been taken that all faculty involved in delivery of the curriculum to students have been trained at the Medical Education Unit of the university, ensuring proper teaching skills.
Assessment is a powerful driver of innovative changes in education and defines goals for learners and teachers (1). Student learning is driven by assessment and assessment is important to the student’s experience (2). In this century, momentum in every module of activity has hastened, leaving little time for answering essay questions meticulously as an assessment. Other reasons cited for its gradual disappearance include the lowest correlation with overall performance versus multiple choice question (MCQ) / clinical examination (3),(4) and low levels of reliability with generalizability (5). The process of structuring and answering MCQ and objectively structured short essay questions (SEQ) has come of age in a majority of the medical schools in India. This is of relevance in developing countries as these modules are cost effective on overstretched staff time (not withstanding a requirement of breadth in their expertise) and institutional finances. Gender differences in student performances have been reported. Studies show that in undergraduate medical examinations, female students performed significantly better than male students (6),(8).
The objective of this study was to evaluate student performances in MCQ and SEQ and to assess the predictive value of each one on their performance in the other component. Undergraduates who fetched more teaching time competing with lesser paid attention to parallel postgraduates formed the cohort of this study. The more powerful of these tools may then be used to predict later grades and its transition to clinical skills. Performances between genders have also been studied.
The study was an observational, retrospective study of written pen and paper type assessment at formative examinations that utilized a convenience sample of 200-250 second year medical students, at each of three consecutive academic years 2004-2005, 2005-2006, and 2006-2007. Student performance data (percentage marks) for each assessment module – MCQ and SEQ of these years were acquired from the central student records. Percentages for each component of these modules at summative examinations of the university were not available and hence, were not included in the study. The pharmacology course in Manipal University is offered during the second year of the students’ MBBS curriculum whose duration spans a total of 18 months and is divided into three semesters; each of 24 weeks. The course contents include: general and clinical pharmacology, autonomic nervous system, cardiovascular system and diuretics in the first semester (hours allotted -40); central nervous system, gastrointestinal tract, blood, autacoids and respiratory system in the second semester (hours allotted -47); endocrine system, chemotherapy, immunomodulators, rational drug use, essential drug list and P drug in the third semester (hours allotted -45). The students received didactic lectures with teaching aides three hours a week and practical sessions inclusive of tutorials and seminars two hours a week.
The written theory paper consisted of two components: part 1 MCQ where four options are given for a question and the single best answer was to be picked; part 2 theory paper of short essay questions (SEQ). The MCQ was timed at 30 minutes for 40 questions with 20 marks. (0.5 marks for the correct answer and 0.166 marks deducted for a wrong answer). The SEQ was timed at 150 minutes for 60 marks, which included questions for 2, 3, 5 and 9 marks. Every effort was made to eliminate examiner bias when the theory paper was evaluated by the faculty. The MCQ answer sheet was evaluated by an optical scanner.
Scores obtained in MCQ papers and SEQ papers were analyzed for: primary outcome – correlation between the high score group (> 60%) in SEQ, who were also in the high score group, (> 60%) for MCQ and vice versa; and secondary outcome - gender differences in written examinations.
All scores were analyzed using the Mc Nemar test. Correlation coefficients (‘r’ value) between the MCQ and SEQ components were estimated by using Pearson’s product moment method. In all analyses, P-values of less than 0.05 were considered to be significant. Statistical package SPSS (version 11.5) was used for the analysis.
The University’s principles and procedures on research ethics were adhered to throughout the study, which was approved by the Institutional Ethics Committee. Data on student performances were presented, such that identification of individual performances was impossible, complying with the requirements of the Data Protection Act.
A total of 683, 653, and 707 second MBBS students who had appeared for the MCQ and SEQ components of the first, second and third formative examinations between the years 2004 to 2007 were included for the study. Pearson’s correlation coefficient (r) in the high – score group (> 60%), in the SEQ, who were also in the high – score group (>60%) for MCQ and vice versa were studied for the level of significance.
The number and (percentage) of the students who scored more than 60% marks in the SEQ and who had also scored more than 60% marks in MCQ at three formative examinations is given in (Table/Fig 1).The performances of the high score group in the SEQ positively correlated with the high score group for MCQ (r = 0.768, 0.599, 0.767). P values < 0.001 were indicative of very high statistical significance.
The number and (percentage) of the students who scored more than 60% marks in the MCQ and who had also scored more than 60% marks in SEQ at three formative examinations is given in (Table/Fig 2). The performances of the high score group in the MCQ positively correlated with the high score group for SEQ (r =0.294, 0.167, 0.445). P values < 0.001 were indicative of very high statistical significance. However, the r values were lower, indicating a lesser degree of positive correlation between the high score group of SEQ in their ability to be in the high score group for MCQ as well.
Gender differences were analyzed for 712 students (406 males and 306 females). In the first, second and third formative examinations, 57.35%, 58% and 57.5% of female students scored more than 60% marks in the MCQ and SEQ components (P<0.001) (Table/Fig 3) Female students consistently scored better than their male counterparts.
(1) DFES (Department for education and skills) Towards a Unified e-Learning Strategy: consultation document, Nottingham: DFES Publications, 2003.
(2) Rust C. The impact of assessment on student learning: how can the research literature practically help to inform the development of departmental assessment strategies and learner-centered assessment practices? Active Learning in Higher Education 2002;3(2):145-58.
(3) Robinowitz HK, Hojat M. A comparison of modified essay question and multiple choice question formats- the relationship to clinical performance. Fam Med 1989; 21 (5): 364-7.
(4) McCloskey DI, Holland RA. A comparison of student performances in answering essay- type and multiple choice questions. Med Educ 1976; 10 (5): 382-5.
(5) Tombleson P. Demise of the essay question. Occas Pap R Coll Gen Pract 1990; 46:23-4.
(6) Tyrer SP, Leung WC, Smalls J, Katona C. The relationship between medical school of training, age, gender and success in the MRCPsych examinations. Psychiatric Bulletin 2002; 26: 257-263.
(7) Pritchard, DJ. Effects of sex and alphabetical listing on examination performance of medical students. Med Educ 1988; 22(3): 205-210.
(8) Krueger PM. Do women medical students outperform men in obstetrics and gynecology? Acad Med 1998; 73(1): 101-102.
(9) Woodford K, Bancroft P. Multiple choice questions not considered harmful. ACM International Conference Proceeding Series; Vol. 106 Proceedings of the 7th Australasian conference on Computing education- Volume 42. NewCastle, Australia;2005.p.109-116.
(10) Prout RES, Hoy TG. University examination performance of Dental students’ correlation between entry qualifications and non clinical and clinical examinations .Br Dent J 1976; 141: 141 – 5.
(11) Shittu LA, Zachariah MP, Adesanya OA, Izegbu MC & Ashiru OA. The differential impact of various assessment parameters on the medical students’ performance in the professional anatomy examination. Scientific Research and Essay 2006;1 (1):14-19.
(12) Jain S, Alkhawajah A, Larbi E, Al-Ghamdi M & Al-Mustafa Z.Evaluation of student performance in written examination in Medical Pharmacology. Scientific J of King Faisal University (Basic and Applied sciences) 2005; 6(1):1426-35.
(13) Lugman W, Ibrahim E. The outcome for some methods of evaluation of clinical students. Mater Med Pol 1987; 19: 60-1.
(14) Robinowitz HK. The modified essay question- An evaluation of its use in a family medicine clerkship. Med educ 1987; 21 (2): 114-8.
(15) Edelstein RA, Reid HM, Usatine R, Wilker MS. A comparative study of measures to evaluate medical students’ performance. Acad Med 2000; 75 (8): 825-33.
(16) Williams JB. Assertion- reason multiple choice testing as a tool for deep learning: a qualitative analysis. Assessment & Evaluation in higher education 2006; 31(3):287-301.
(17) Epstein RM. Assessment in medical education. N Engl J Med 2007;356:387-96.
(18) Burton RF. Quantifying the effects of chance in multiple choice and true/false tests: question selection and guessing of answers, Assessment and Evaluation in Higher Education 2001; 26(1): 41–50.
(19) De Vita G. Cultural equivalence in the assessment of home and international business management students: a UK exploratory study. Studies in Higher Education 2002; 27(2): 221–231.
(20) Leamnson R. Thinking about teaching and learning. Sterling, VA, Stylus Publishing;1999
(21) Vyas R, Supe A. Multiple choice questions: A literature review on the optimal number of options. Natl Med J India 2008; 21 (3):130-33.
(22) Hettiaratchi ES. A comparison of student performance in two parallel physiology tests in multiple choice and short answer forms. Med Educ 1978; 12 (4): 290-6.
(23) Wakeford RE, Robert S. Short answer questions in an undergraduate qualifying examination: a study of examiner variability. Med Educ 1984; 18 (3): 168-73.
(24) Webber RH. Structured short answer questions: an alternative examination method. Med Educ1992; 26 (1): 58-62 .
(25) McManus I C, Richards P, Winder PC, Sproston KA. Final examination performance of medical students from ethnic minorities. Med Educ 1996;30: 195-200.
(26) Acheson AG. Do male students face prejudice? Lancet1997; 350: 964.
(27) McDonough CM, Horgan A, Codd MB, Casey. Gender differences in the results of the final medical examination of University College Dublin. Med Educ 2000; 34: 30-34.
(28) Matsubayashi K. Sex and examination results. Lancet 1997; 350: 1711.
1) Authentic assessment is inseparably linked to constructivist learning.The MCQ format of assessment is an effective, time saving tool of assessment in Pharmacology. The judicious use of MCQ can give a better insight into the student’s understanding, application and the synthesis of knowledge.
2) The probability of students scoring > 60% marks in SEQ was higher when they had scored > 60% marks in the MCQ papers (P<0.001). The SEQ performance positively correlated with the MCQ performance.
3) The essay question format correlated poorly when compared with the MCQ pattern of assessment. Factors which were not in favour of essays included the idiosyncratic behaviours of examiners in the distribution of marks which were awarded for essays, language problems, the emotional maturity of students, poor, illegible handwriting, etc.
4) Female students consistently have outperformed their male counterparts and the data correlated with the data across the globe on gender differences in student performances.
We would like to acknowledge and convey our gratitude to Kasturba Medical College, Mangalore, Manipal University for providing us the necessary data during the conduction of this study.
- Emerging Sources Citation Index (Web of Science, thomsonreuters)
- Index Copernicus ICV 2017: 134.54
- Academic Search Complete Database
- Directory of Open Access Journals (DOAJ)
- Embase
- EBSCOhost
- Google Scholar
- HINARI Access to Research in Health Programme
- Indian Science Abstracts (ISA)
- Journal seek Database
- Popline (reproductive health literature)
- www.omnimedicalsearch.com