Learning Sessions on Critical Appraisal of Randomised Clinical Trials: Student Feedback
Correspondence Address :
Dr. P. Ravi Shankar. Department of Pharmacology, Manipal College of Medical Sciences, P.O. Box 155, Deep Heights, Pokhara, Nepal. Tel.: 00977-61-440600; fax: 00977-61-440260; e-mail: ravi.dr.shankar@gmail.com
Context:Critical appraisal skills are essential for students to become rational prescribers. Evidence-based medicine requires the ability to appraise articles critically. Manipal College of Medical Sciences, Pokhara, Nepal, mainly admits students from Nepal, India and Sri Lanka for the undergraduate medical (MBBS) course. Pharmacology is taught during the first four semesters. The department recently started learning sessions on the critical analysis of published randomised clinical trials for the third- and fourth-semester students.
Aims: The study is aimed to obtain information on the student attitudes towards the sessions and note the association, if any, of the attitudes with demographic characteristics of the respondents.
Settings and design: The study was conducted among the third- and fourth-semester basic science students of the institution. The design was a prospective questionnaire-based study.
Methods: Basic demographic information about the respondents was collected, and the degree of agreement with a set of 16 statements was noted. The agreement was noted using a modified Likert-type scale. The total score was calculated for each respondent. The scores of some statements were reversed while calculating the total score. Statistical analysis: The median total score was compared among different subgroups of respondents. The level of statistical significance for all analyses was set at p < 0.05 using two-tailed comparisons.
Results: A total of 116 students participated (overall response rate was 81.8%). Nepalese and Indians were the major nationalities. The median score (interquartile range) was 54 (50.5–56.5). No significant difference in scores was seen among different subgroups. The students were of the opinion that the sessions made them aware of the technique of critical analysis of a journal article.
Conclusions: Student opinion about the sessions was positive. Lack of knowledge of statistics among the students was a limitation. A student elective on reading and writing science can be considered.
Critical appraisal, medical students, randomised clinical trial
In the early 1990s, the concept of evidence-based medicine (EBM) started gaining importance. EBM states that well-formulated questions about medical management can be answered by carrying out randomised, high-quality, controlled clinical trails, tracing all the available evidence, systematically analysing the evidence, determining the validity and usefulness of the evidence and applying the evidence to the management of the individual patient (1).
EBM has been defined as ‘the conscientious, explicit and judicious use of current evidence in making decisions about the care of individual patients’ (2). A critical skill for the practice of EBM is the ability to appraise articles critically (3). EBM has been included as a part of the undergraduate medical curriculum in medical schools all over the world (4),(5),(6). In Nepal, however, the concept of EBM is still in its infancy.
Critical appraisal skills are crucial for students to become rational prescribers (7). Medical students should be equipped with skills to critically review not only medical literature but also promotional material and the prescribing patterns of fellow clinicians (7). The Department of Pharmacology at the Manipal College of Medical Sciences teaches medical students to critically analyse drug advertisements and other promotional material (8). The department has been able to introduce students to the concept of rational use of medicines (RUM) and influence students during the first 2 years of their training, but the influence gets eroded once students enter the clinical phase of training (9). Manipal College of Medical Sciences, Pokhara, Nepal, is affiliated to the Kathmandu University for the undergraduate medical (MBBS) course and mainly admits students from Nepal, India and Sri Lanka. A few students are from other countries. This year, the department started learning sessions on the critical analysis of published randomised clinical trials for the third- and fourth-semester students. The students were informed of the topic of the learning session a week in advance.
The faculty members (PRS and PS) introduced the students to published scientific research papers. The different parts of a paper, Abstract, Introduction, Materials and Methods, Results and Discussion, were briefly discussed. The IMRaD format was highlighted. The tips for efficient reading described in ‘Guide to good prescribing’ were highlighted. Then we discussed with the students the various phases of clinical trials and the process of new drug development and its progress through pre-clinical and clinical testing. The gold standard of the randomised, double-blind, placebo-controlled, clinical trial was discussed. With the involvement of the students, a checklist to assess published clinical trials was drawn up.
Then each group was given a published clinical trial from a scientific journal to analyse using the developed checklist. The homework given to the students was to familiarise themselves with all the trials given to the various groups. The student group of 37 or 38 students was divided into five groups of seven or eight students each for the session.
In the next practical session, the various groups first described their allotted trial in brief. Then they analysed the strengths and weaknesses of the trial using the checklist. Then the discussion was opened to the house. After all the five groups presented their findings, the facilitators elaborated points that they felt were not covered adequately.
Student feedback about the sessions was obtained using a questionnaire. Basic demographic information was obtained, and student attitude towards the sessions was studied.
At the end of the second learning session, student feedback was collected using a questionnaire (see the Appendix-(Table/Fig 1)). The students were explained the objectives of the study and were invited to participate. Verbal consent to participate was obtained from all participants.
Basic demographic information like sex, nationality, method of financing of medical education, occupation of parents, semester of study and place of family residence were noted. The degree of agreement of the respondents with a set of 16 statements was noted using a modified Likert-type scale (Appendix).
The total score was calculated by adding the scores of the individual statements. Statements 3, 5, 6, 7 and 12 were negative, and their scores were reversed while calculating the total score. The median total score was calculated. Scores derived by using a Likert-type scale do not generally follow a ‘normal’ distribution. The median scores were compared among different subgroups of respondents. Mann–Whitney test was used to compare the median total score when there were two subgroups of respondents and the Kruskal–Wallis test when the subgroups were more than two. A p-value less than 0.05 were taken as statistically significant.
Free text comments from the respondents regarding the sessions were elicited and the common comments were noted.
A total of 116 students participated in the study. Fifty-two of the 75 third-semester (69.3%) and 64 of the 68 fourth-semester (94.1%) students participated. The overall response rate was 81.8%. Seventy-one students were male. Nepalese and Indians were the major nationalities. The majority of students was self-financing, did not have doctor parents and was from urban areas.
The median score (interquartile range) was 54 (50.5–56.5). The maximum possible score was 80. (Table/Fig 2)shows the median scores of different subgroups of respondents. The median score of the Sri Lankans was lower compared to other nationalities. No other significant difference in scores was seen between different subgroups.
(Table/Fig 3) shows the median scores of individual statements. The students were of the opinion that the sessions made them aware of the technique of critical analysis of a journal article. They agreed that clinical research is a big growth area in South Asia. They were in agreement with the statement that Introduction is the first section to look at when reading a published trial. The students were not in agreement with the statement that doctors do not require knowledge of statistics. They also showed a high degree of agreement with statement 9 that the abstract is the first part of a research paper to be read. They were in favour of similar sessions in the future.
The common free text comments were grouped together. The students were not in favour of this exercise being included in the assessment examination. They were in favour, however, of more such interactive, interesting and short PBL sessions.
The overall student opinion regarding the sessions was positive. The students were in favour of similar sessions in the future.
In our institution, the MBBS course is divided into nine semesters. Pharmacology is taught during the first four semesters, as already detailed. Clinical subjects are taught during the last five semesters. Community medicine learning starts from the first semester and continues till the seventh semester. The compulsory internship training of 1 year’s duration follows the ninth semester.
The medical literature is a commonly used information source for students and practicing health-care professionals trying to keep up-to-date. Critical appraisal skills are important for medical students (7). Medical literature must be analysed critically to confirm the validity of the results and conclusions. Students must learn to form their own conclusions after careful, critical review and must learn to challenge exaggerated and unjustified claims (7). The skill of efficient reading has been emphasised in the ‘Guide to good prescribing’ (10). The students should be taught to identify at an early stage articles that are worth reading.
The steps of efficient reading were discussed with the students. The first step is to look at the title of the paper to decide whether it would be of interest to the reader. Then the abstract should be read. If the reader thinks the paper may be of interest then the Materials and Methods section should be considered. The method of the study is important in deciding about the validity of the findings.
We had instructed the respondents to respond to all the statements in the questionnaire. However, two students did not denote their agreement with certain statements. Some respondents did not fill in all the demographic details. If the respondent did not indicate his/her degree of agreement with a particular statement we gave the statement a value of 0. This may have accounted for the lower score of the Sri Lankan students regarding the sessions. The number of Sri Lankans was low (four).
A major problem we faced was the lack of knowledge of biostatistics among the student respondents. The students are only taught the measures of central tendency and the measures of dispersion and the remainder of the biostatistics course is covered in the seventh semester. We are discussing with the department of Community Medicine the possibility of rescheduling some of the biostatistics classes to the second semester. We did not keep statistical aspects in the assessment checklist, which was developed with the students.
The students agreed with the statement, ‘The sessions made me aware of the technique of critical analysis of a journal article’. However, we spent a lot of time discussing the basic parts of a scientific paper and of the methodology of a randomised trial. The two sessions only served as an introduction to the complex topic of critical analysis of information. We discussed only clinical trials and did not consider meta-analysis and systematic review, which are important means of synthesising information. The student agreement with statement 3 is a matter of concern. We discussed the steps of efficient reading with the students and emphasised that they should first look at the Abstract and then the Methods section. This and the low-agreement statement 5 are worrying. However, the students showed good agreement with statement 9. The results are conflicting. We plan to emphasise critical analysis of clinical trials again in a future session.
The students did not agree with the first point that conclusions from clinical trials carried out in the West can be applied to South Asian patients. The Indian clinical trials market of US $30–35 million is projected to grow 8–10 times to US $250–300 million by 2010 (11). In 2005, the government of India enacted a rule that allo
Student opinion about the sessions was positive. The sessions served to introduce students to critical appraisal. Lack of knowledge of statistics among students was a major problem in teaching critical appraisal. Restructuring of the teaching of statistics can be considered, with more of the subject being taught during the second semester. A student elective on research methodology and critical analysis can be a possibility.
The authors acknowledge the encouragement and support of Dr. Archana Saha, Head of the Department of Pharmacology. We thank all the students who participated in the study.
- Emerging Sources Citation Index (Web of Science, thomsonreuters)
- Index Copernicus ICV 2017: 134.54
- Academic Search Complete Database
- Directory of Open Access Journals (DOAJ)
- Embase
- EBSCOhost
- Google Scholar
- HINARI Access to Research in Health Programme
- Indian Science Abstracts (ISA)
- Journal seek Database
- Popline (reproductive health literature)
- www.omnimedicalsearch.com