Selection of PGs in medical education should have the potential to explore and identify the required competencies for the specific specialty program. The graduate should explore the skill and specialty they want to move ahead, at least by the end of the internship program of their medical curriculum. In our opinion, having clarity for the specialisation program, graduates would move further on with full satisfaction and less stress [1].
Globally, the medical residency program has shown marked variation across countries [1]. Multiple assessment tools are introduced in the United States of America, considers standardised licensing examinations scores, and past academic records, curriculum-vitae, personal statements, personal portfolio, referees’ reports, Dean’s letters and letters of recommendations other than basic and clinical cognitive, psychomotor and affective skill assessment depending upon the type of residency program graduate wants to apply [2-5]. Similarly, the United Kingdom and Australia have robust selection methods using multiple formats either associated with low fidelity methods e.g., written or video scenario-based test or high fidelity method like simulations which authenticate job-related tasks [6-8]. Internationally, selection tests vary as per the PG specialty program. Literature evidence identified several selection methods, including Multiple Mini-Interviews (MMIs), Situational Judgment Test (SJTs), Clinical Problem-Solving Tests (CPSTs), simulations (low or high fidelity, depending on the specialty) and assessment centers [6,7,9,10].
In India before April 2016, central government affiliated medical institutes, state and private medical institutes had independent examination criteria, either a specific assessment tool or graduates’ academic score for selecting a graduate for the PG residency program, after completion of compulsory internship program [11,12]. To improve the selection procedure in India the Medical Council of India (MCI), in 2010, replaced the selection test with a single eligibility cum entrance test for which a proposal was passed by the Government of India, on and before December 2010 [13,14]. This proposal had to pass through various levels of scrutiny at different levels of government and Supreme Court [15-17]. The Supreme Court of India on 11th April 2016 announced the reintroduction of “The NEET-PG”, hence the nationalised examination proposal was approved [18].
Educationists in India, at various forums, had argued for and against this assessment at that time and still are continuing at various platforms [19-23]. The journey of this nationalised examination started; despite large debate by conducting authorities also as per the logistics and cost. Appearing for different PG examinations at different exam centers at different timeframes was a mental and financial burden to the graduates. A single nationwide test (NEET-PG) probably decreased this burden to a great extent decreasing the travel and financial stress [24].
The NEET-PG is conducted once in a year, for three hours and 30 minutes having 300 Multiple Choice Questions (MCQ) in English only [25]. Thus, this test focuses only on the cognitive aspect of learned knowledge disregarding the required psychomotor, communication skills or professionalism aspect of the medical graduate which is the prime aspect of a physician rather than the rote learned content tick marked on OMR sheet [24]. A single exam of three and a half hour will not be sufficient to express the multiple levels of complex information understood during their graduate program [25].
This, nationalised examination, is supposed to allow medical students, throughout the country, to be assessed with uniformity. Limited literature is available regarding what graduates think about this type of, one attempt per year, selection test with single type of assessment method [22], and the same criteria for different specialty programs, be it surgical or medicinal or basic science. Hence, present study was performed with the following objectives:
Materials and Methods
An exploratory qualitative thematic analytic study was conducted for duration of six months (December 2018-June 2019). The participants were selected through purposeful sampling strategies, and the sampling continued until data saturation. To do this, 11 interns and 10 first-year PG residents from a private medical college in India (SVIEC/UN/Medi/RP/18022; dated: 17th December 2018), were selected after procuring the ethical approval.
The sample collection was nonrandomised and purposive. Out of all the interns (92) and first-year residents (81), 11 interns and 10 first-year PG residents were selected for the study. The purposive sampling selection from the voluntary intern participants was done as per gender and their academic score for final year medical exam results (percentage scores above 65% and between 55%-65%). All the selected voluntary participants were contacted telephonically (since the majority of the participants were not using e-mail on a routine basis). Similarly, voluntary first-year PG resident participants were selected as per gender and the clinical PG program (surgical and allied specialty, medicine, and allied specialty PG program).
Study Procedure
The SSI was conducted for 10 PG residents and 11 interns to have five intern participants each, as per gender and their grades. Unknowingly, one participant for a higher grade was missed, so an extra intern was chosen as a participant. No new idea or information emerged from the participants after this sample size thus considered that the ‘saturation’ was achieved for the prepared SSI questions. The majority of interviews took 40 to 45 minutes except for two where participants expressed their thoughts for 75-80 minutes.
Data were collected through semi-structured and face-to-face interviews. This methodology allowed for many unexpected stories or ideas to be unfolded. SSI questions were prepared and validated by the experts. The Content Validity Ratio (CVR), after input from three qualified experts for each item for accuracy, was +1 for the entire item. A pilot test was carried out, with one intern and first-year PG resident, to validate the SSI. The pilot test allowed the author to reframe some probing questions for the subsequent interviews. The pilot test participants were not the part of the main study. Chosen participants were requested to read the Participant Information Sheet (PIS) again for any further queries so that they would sign the consent form before initiating the interview. A face to face interview was conducted and audio-recorded by a voice recorder and note-taking was done for the data recording method.
Complete confidentiality at all levels was maintained. All the selected voluntary participants were given a number that was used during the interview. Data storage was in the form of participant numbers rather than names. The data has been securely stored in the author’s personal password-protected computer. The notes taken at the time of the interview are well secured in the author’s locker with participant numbers on it rather than the name.
Statistical Analysis
For the present study, Braun & Clarke’s 6-step thematic analysis was used as framework. Starting from becoming familiar with the data, generating initial codes, followed by a search for themes; review of themes; defined themes and lastly writing these down [26]. After transcribing the voice recordings the data was printed and read at least twice to generate the initial codes. In case of any confusion, about the transcript, the participant would have been contacted to verify the codes or theme as per their interview recording. Primarily ten codes emerged from the transcribed data out of which three themes emerged after discussion with other researchers, as an external validate i.e., building validity, proposed changes, and learning by assessment for the proposed research question.
Results
The demographic distribution of the participants is represented in [Table/Fig-1]. The participants are coded as I=Intern, R=Resident, M=Male, F=Female, H=Intern with percentage grades >65% in their final MBBS exam, L=Intern with percentage grades between 55%-65% in their final MBBS exam, Sa=resident from Surgical and allied departments, Ma=resident from Medicine and allied departments.
The demographics of the 21 interviewees (11 Interns and 10 Residents).
Participants | Gender | Percentage marks in final MBBS |
---|
Male (M) | Female (F) | L (55 to 65%) | H (>65%) |
---|
Interns (I) | 5 | 6 | 5 | 6 |
Participants | Male (M) | Female (F) | Surgical and Allied Department (Sa) | Medicine and Allied Department (Ma) |
First-year resident (R) | 5 | 5 | 5 | 5 |
Participant codes I1FH and R1MSa were the pilot interview participants, which are not part of the study. Study participant codes for Interns are from I2 to I12 and residents are from R2 to R11.
Analysis of Interviews
The three final themes [Appendix-1] emerging from the transcribed data are presented under the following headings.
Building Validity
The majority of interns and many residents welcomed this nationalised exam as they thought it was better than the previous selection test. There were clear ideas that the validity of the assessment could be increased. Different approaches were suggested by the participants to improve the knowledge and skill aspect for the selection test and offer a better selection of the PG residents.
“So first thing which I agree with NEET is that they have made it very universal as per the nation. There is no differentiation or bias which I felt earlier. Then what another thing which I agree with is that it is a single day exam so in this way also it is also completely unbiased exam.” (I10FH)
Few interns and residents questioned this type of assessment as per the timeframe of conducting the test.
“I personally feel that one exam cannot decide your calibre. One exam cannot decide about you how much you know and you do not.” (I4FH)
“It is not a fair exam because you get one shot to decide the destiny of the student but still you have to wait till next year.” (R8MMa)
One of the residents quoted that “It is good exam but it is not a great exam.” (R11FSa)
Few of the interns did mention this assessment was based on the luck factor.
“MCQs depend on the luck also. Some of the students like few of my friends who had not prepared at all. They appeared in the exam as a trial, so they answered 298 questions out of 300. They attempted so many questions. There is equal possibility of their selection too. If by luck they opted for correct answer. So they can score well too.” (I4FH)
The query for question pattern: Most of the participants reflected that the MCQ based assessment pattern would be a good way to improve performance in the selection test while few thought the other way around.
“The standard of question which is asked in the NEET exam is way beyond the university exam. When you know all the subjects then only you will be able to solve MCQs and those are one point with 4 options which are so close.” (R8MMa)
Although, many residents and few interns expected modification of the MCQs, allowing them to analyse and develop a critical thinking approach.
“First of all there is mugging up of stuff. Something’s are really not important like the name of scientist and all that.” (I9FL)
“Name of scientists is asked which no one can solve. They are more of memorising questions.” (R3MMa)
Proposed modifications
Various modifications were proposed by interns as well as residents starting from Question pattern to exam pattern
Modification for MCQs: Residents and interns had a varied view about the patter of MCQ’s asked from the applicants. Some expected to have more case-based questions while few expected patient-based exam which probably means the same in case of objective structured questions.
“I think it should have more of clinical scenario and less of memorisation aspect.” (I11MH)
“I think NEET-PG should also start having long theory type questions exam. It will be very helpful.” (I4FH)
“Although, we are having case scenario and audio visual patients but virtual patient will be much better because we feel that we are diagnosing the patient or treating a patient. That gives a better feeling.” (R10MSa)
Biannual assessment: Many residents, as well as few interns, suggested having this assessment on a biannual basis rather than annually.
“It should be two times a year.” (I6FL)
“….. this exam can be held twice a year so that the students who could not perform well would not have to wait for the whole year to reappear for the exam.” (R9FMa)
Negative marking: Few residents and interns did mention to have no negative grading for the incorrect options.
“...to remove this negative marking for the question incorrectly attempted. Due to this negative marking the chance of decrease in our rank occurs.” (I8FL)
“I would have liked to go for common paper including all the subjects but without negative marking.” (R5FSa)
Clinical skill assessment: Many interns proposed to have clinical skill assessment along with that of the MCQ exam.
“If you are dealing with the patients in the exam then you will know about what kind of situation you are going to face in future. So, they should have a practical base.” (I5FH)
Although, some had concern for the logistic to conduct clinical skill assessment in a populated country likes India leading to bias approach.
“But frankly speaking I don’t think that it is very feasible to do in India, because universal assessment will not be done. There will be different teachers; different places and I feel that universal assessment has to be done. So, like that practical exam cannot be conducted.” (I10FH)
Many residents too had an optimistic response for the clinical skill assessment.
“Clinical based is more important. I would like to give some weightage to internship. ……. There must be some clinical testing but this should be done at higher level. ….. The examiners should be from NEET-PG authority, not on college based as they might give benefit to their own student.” (R3MMa)
“If all these exams are associated with clinical exam, then some of the students like myself, who are not prepared well in internship or who are not good in clinical skills they will then only qualify the first phase then they will not be qualified in skill exam.” (R5FSa)
Specialty based assessment: Few of the interns suggested having subject specialty assessment or choosing the subject of residency at the time of filling the form for the NEET-PG exam.
“They should divide the student, which branch they are interested. If student is interested in clinical branch then they should have clinical exam.” (I4FH)
“I am interested in paediatrics then I will write before appearing for the exam on the form itself ‘Ped’. Then my exam should have some general questions with some related to the chosen subject. ….. And accordingly I will be asked more about my choose subject. If it would be in NEET-PG form then it will be good.” (I8FL)
Few of the residents had similar opinions.
“If anyone wants to get Obstetrics and gynaecology then that is all together a completely different branch. You do not need to have knowledge of a subject like may be let us say…. Ophthalmology. …. So, I think that should be narrowed down you know… according to the field of interest.” (R7FSa)
Two-step selection: Few interns and residents suggested having a selection test in two phases.
“It should be conducted in parts. Like once you are in your first and second year then you can have exam for those two years and then for last two years a separate paper. Probable that would make the exam lesser hectic and less stressful for the students to appear because you are really fresh at that time.” (I11MH)
“There should be an exam that is divided into two levels may be where one can be a subjective questionnaire while other once you are clearing first then there should be a clinical round or something.” (R7FSa)
Additional assessment tools: Few interns mentioned having different assessment tools along with the basic tool for selecting an applicant for the residency program.
“...to make the procedure more selective they can go for interviews. Or something like that if they want best out of.. best students for their course.” (I12ML)
“I think like they do in other countries some recommendation letters. They also check the performance of the applicant during the MBBS program or any previous program they were in. such things may be a reference letter from a Professor or the faculty.” (R9FM)
“…interviews of the applicant only after getting admission in a particular subject can be kept. Even a letter of recommendation can also be introduced but it should be a proper white process.” (R10MSa)
Learning by the Assessment
Varied responses were registered by two groups of participants. Few interns agreed that they had a better learning approach compared to learning from the undergraduate assessment method.
“During UG I had to memorise everything what was in the topic. But I know the concept; I can answer the question if anyone is asking me about the patient. I can mention that my critical approach has improved.” (I5FH)
Although, few interns had different opinion for the same.
“I think that when the students are prepared well for the concepts not for the questions they can crack n number of exams they give. But they are not preparing for the concepts, they are preparing for questions.” (I7MH)
“I didn’t think much of learning or acquiring knowledge towards the patient care, but I just wanted to crack that exam so according to the exam pattern I studied.” (I11MH)
Similarly, residents also had a varied opinion for the learning aspect from this assessment process.
“I personally feel that NEET-PG is not a conceptual learning. Because in NEET-PG we have very limited time and we have to read that this is the thing and this is the answer.” (R10MSa)
“In MBBS we just mug up the things and write the answer. But in NEET-PG we have to understand the thing.” (R5FSa)
Discussion
The participants in the study agreed on having NEET-PG, nationalised exam, to uphold equivalent standards throughout the country, although, majority of them highlighted the need for improvement. Pardeshi G, study mentioned that 52% of interns agreed for a nationwide single assessment system leading to uniformity as per assessment questions [22].
Despite maintaining uniformity nationally this test, still, is based on a single cognitive MCQs solving approach, mostly centered on “knows” or “knows how” of Miller’s assessment pyramid [27]. This would make students MCQ solvers, rather than learners with an analytic problem-solving approach, which is in concurrence with Singh T et al., study [24]. This type of single assessment system will produce untrained graduates, as per psychomotor or affective domains, thus raising serious concern for patient safety, as per the author’s opinion. The researcher in a study mentioned that interns, unfortunately, misused the internship period as “one year paid leave” using it to prepare for PG entrance examinations or for other exams [28]. Although, internship duration can be well used by the graduates to figure out what they like or don’t, so they can prepare themselves for the specialty residency program of their choice [29]. Other than this they can gain experience and develop inter-professional and communication skills and be able to learn professional workplace operates [29].
The majority of the participants mentioned having clinical based MCQs which is well accepted by many researchers [9,24]. The researcher in their study mentioned Clinical Problem-Solving Tests (CPSTs) presenting clinical scenarios for applicants to apply their clinical knowledge to solve a problem associated with the case [9].
A systematic review study showed that the academic records, MMIs, aptitude tests, SJTs and Selection Center (SCs) are more effective selection methods [30]. Those methods are considered better than traditional interviews, references, and personal statements [30]. SCs are based on a multi-trait, multi-method design; they may comprise a large number of elements in different combinations and orders [30].
Assessment of all the learning domains with varied tools would be a better selection approach compared to a single assessment tool for a single learning domain, for NEET-PG exam, by interns [22].
A study showed that 91% of participants out of 117 agreed to have clinical skill assessment stating “…assessment would be holistic as entrance rather than focus on the theory” and “…would be helpful to the interns to identify their mistakes and correct it” [22].
Another study proposed a list of limitations to single assessment tests and suggestions for improvement for those limitations [24]. They pointed to introduce clinical reasoning questions in MCQ format for NEET-PG selection, other than including tools to test higher-order thinking skills, aptitude and ethical judgment, and introduction of clinical assessment skills [24].
Research shows that candidates prefer selection methods that are job-relevant and offer them the opportunity to demonstrate their ability, as well as treating them sympathetically [31]. Researchers internationally are trying to design and develop a selection method which would be specific for a specialty PG training program [32]. Indian PG selection authorities should look into the global evidence for revising the selection method for the graduates shortly. As well suggested by some of the interns and residents that there should be a course-specific PG selection method, so that graduates are sure which specialty to opt for, rather than wait what the merit list has in for their destiny.
A recent study mentioned that a specialty based tests would be quite challenging in the Indian set-up, due to the large number of applicants applying for the specialty program with limited resources to conduct clinical skill assessment [33]. Moreover, limited applicants will be applying for pre or paraclinical specialty courses [34].
As per the authors’ opinion, a modification for selection test in Indian set-up should be done as per the feasibility. Moreover, the applicants should be aware of the modification at the time they join the graduation program, so that they have sufficient time to prepare for the selection test, rather than announce it mid-way through the program progression. The study has laid the path to further research opportunities. Primarily, this type of study should be conducted in the government-funded institutes so that the opinion of the interns and residents in that setup could also be explored. Secondly, faculty and administrative stakeholder’s opinion should be taken into consideration before coming up to the outcome.
Limitation(s)
The literature search for the topic was the most difficult part since the researcher couldn’t find any original article related to what stakeholders think about the method of NEET-PG examination. The result of the present study reflects the views of graduates from a private medical institute (non-state funded) and those views may not be echoed by participants from government-funded institutes.
Conclusion(s)
National Eligibility Cum Entrance Test for Postgraduates exam was designed to maintain uniformity for the entry of the graduates to the PG specialty program but applicants have raised various questions on why this selection test is being conducted. Out of which the most important the introduction of multiple levels of the exam and using various assessment tools to assess the competency of the graduate to enter a specialty program. Modification of MCQ from absolute rote to critically analysing one would reflect the assessment of the higher domain of cognition. Biannual assessment with no negative marking would allow the applicants to appear the test second time without having to wait for the next 12 months for assessment/examination.
Selection authorities should take an account of the voice of stakeholders who are being affected maximally by the change in policies i.e., graduate applicants, before modifying the selection procedures within the nation.