|Year : 2014 | Volume
| Issue : 2 | Page : 49-53
Formative assessment using direct observation of single-patient encounters in ophthalmology residency
Kavita Bhatnagar, OK Radhakrishnan, Abhay Lune, K Sandhya
Department of Ophthalmology, Pad. Dr. D Y Patil Medical College, Hospital and Research Center, Pune, Maharashtra, India
|Date of Web Publication||6-Feb-2015|
B4/21, Brahma Angan, Salunke Vihar Road, Pune - 411 048, Maharashtra
Background: There has been a growing concern that trainees are infrequently observed, assessed, and given feedback during their workplace-based education. This has led to an increasing interest in a variety of formative assessment methods that require observation and offer the opportunity for feed. Mini-clinical evaluation exercise (mini-CEX) is an observation tool that facilitates the assessment of skills that are essential for good clinical care and provision of immediate feedback. The aim of this article is to sensitize academician-clinician in developing countries to mini-CEX. Materials and Methods: This observational study was conducted on 10 postgraduate students who were assessed on clinical skills, communication skills, professionalism and ethics in different clinical settings using a validated mini-CEX global rating scale. Data analysis was done. Student's scores in the first encounter and last encounter were compared to see an improvement in their performance. Results: A total of 216 encounters was observed and rated on a nine-point Likert scale. Mean score and maximum improvement were seen for Humanistic qualities/professionalism. About 100% residents found mini-CEX exercise useful. Faculty feedback was more useful than multiple encounters being observed. Over 90% patients were highly satisfied with resident behavior. The mean score for faculty satisfaction with mini-CEX was 5.33, and for the students, it was 5.48. Conclusion: Mini-CEX is an examination tool which permits evaluation based on a much broader set of clinical settings and patients. It also gives an opportunity to observe and communicate with the students in real life settings. It can be used as a tool to identify and highlight deficient areas in individual performance for further improvement.
Keywords: Clinical competence, communication skills, mini-clinical evaluation exercise, professionalism/ethics, workplace based assessments
|How to cite this article:|
Bhatnagar K, Radhakrishnan O K, Lune A, Sandhya K. Formative assessment using direct observation of single-patient encounters in ophthalmology residency. Sudanese J Ophthalmol 2014;6:49-53
|How to cite this URL:|
Bhatnagar K, Radhakrishnan O K, Lune A, Sandhya K. Formative assessment using direct observation of single-patient encounters in ophthalmology residency. Sudanese J Ophthalmol [serial online] 2014 [cited 2020 Dec 1];6:49-53. Available from: https://www.sjopthal.net/text.asp?2014/6/2/49/150994
| Introduction|| |
Workplace based assessments are being used increasingly during postgraduate medical training as a method of assessing competence. , There has been a concern that trainees are infrequently observed, assessed, and given feedback during their workplace-based education that is in outpatient (OP), inpatient and emergency department. This has led to an increasing interest in a variety of formative assessment methods that require observation and offer the opportunity for feedback. (Norcini 2007). 
There are many methods including direct observation of procedural skills, mini-clinical evaluation exercise (mini-CEX) and case-based discussions. All these methods are used in Ophthalmology in western countries but are still in their infancy in India.
The mini-CEX is widely used for assessment in single work-based encounters of clinical competence at the top of Miller's pyramid the "does" level. Currently, assessment of clinical competence is receiving increasing attention, particularly in postgraduate training (Wass et al. 2001), , and assessment of authentic performance is considered the main challenge. Reliable and valid performance measurements that can serve as a gold standard for clinical assessment have as yet not been achieved (Wass et al. 2001). , Developed for the evaluation of a multitude of clinical competencies (Norcini et al. 1995), , the mini-CEX is a single-encounter instrument to be used by professionals in conducting work-based assessment of actual clinical performance. It was originally developed by American Board of Internal Medicine in 1995 in the USA for the evaluation of internal medicine residents' clinical skills (Norcini et al. 1995, 2003) , and its principal characteristics are direct observation of real patient encounters, easy and instant use in day-to-day practice, applicability in a broad range of settings and immediate feedback to the learner after the encounter. These characteristics make the mini-CEX an educational tool that can help learners to gain insight into the strengths and weaknesses of their clinical performance.  It can be used to assess multiple competencies, such as communication and professionalism. Typically, the mini-CEX and similar instruments use global assessment scales, provide space for narrative comments and allow for feedback presented by a moderator in a postencounter review session. Previous data on mini-CEX suggest that it is a valid and reliable evaluation tool. ,,,,,,,,,,,
Our college was established in 1996, and postgraduate course in Ophthalmology was started in the year 2002 with two Masters in Ophthalmology (MS) and one Diploma in Ophthalmology (DOMS) intake annually. Today, we have an intake of six MS and two DOMS students every year so at time we are training 22 residents in the department that is a big number. After passing out, majority will start private practice where they will be treating patients independently.
We realized that observing trainees during their daily practice in OP or inpatient department is a time-efficient way of determining their level of competence. By giving great importance to the concept of feedback, trainees will be encouraged to reflect upon the learning experience and can, therefore, improve their clinical and communication skills including professionalism. Mini-CEX was introduced as part of formative assessment in the department of ophthalmology of our college with subsequent inclusion in university examination with time and experience.
| Materials and methods|| |
This Observational Study was conducted on 10 Postgraduate students in the department of Ophthalmology of a medical college. Institutional Ethical Committee clearance was obtained before starting the study. Residents were informed that they will be assessed using this new method of assessment as part of internal assessment.
All participating faculties were trained by the first author in this new method of assessment and art of giving effective feedback. This included a combination of didactic lectures and videos of mini-CEX on simulated patients. They were also given literature on mini-CEX for further reading.
Students were assessed on their history taking skills, physical examination skills, clinical judgment skills, communication skills, professionalism and ethics, and organizational ability in different clinical settings using a prevalidated mini-CEX global rating scale and a check list for communication skills by different examiners. Minimum six resident-patient encounters were recorded per resident by four faculty members with more than 10 years of clinical experience. Each encounter was of 15-20 min duration and 5 min were assigned for feedback to the students. The problems observed varied in difficulty levels and also during busy OP department (OPD) days as well as lighter OPDs.
For each encounter, the examiner recorded the date, the complexity of the patient's problem on a 3-point scale (low, moderate, and high), the sex of the patient, the type of visit (new or follow up), the setting (OP, inpatient, or emergency department), the number of minutes spent observing the encounter, and the number of minutes spent giving feedback. The examiner also noted whether the focus of the encounter was data gathering, diagnosis, treatment or counseling.
Using a nine-point scale (in which 1-3 were "unsatisfactory," four was "marginal," five and six were "satisfactory," and 7-9 were "superior"), the examiner rated the resident on history taking, physical examination, professionalism, clinical judgment, counseling, organization and efficiency, and overall competence. The examiner also rated his or her own satisfaction with the method as a valid and efficient assessment device on a nine-point scale in which one was "dissatisfied" and nine was "very satisfied." For all items, the examiner could select "not applicable" where appropriate. In addition, a total score was calculated as the mean of the six component ratings.
| Statistical analysis|| |
Data were collected and recorded in Microsoft Excel spreadsheet. They were verified and completed according to the mini-CEX rating forms and feedback questionnaires. They were presented in the form of central tendencies. Appropriate tables were projected for data.
| Results|| |
A total of four senior faculties and 10 postgraduate students posted in author's unit in the Department of Ophthalmology were enrolled for the study. Out of 10 students, five were in 2 nd year of residency and five in 3 rd year of residency. The total number of resident-patient encounters observed were 216. Mean score was highest for Humanistic qualities/professionalism (5.48) followed by counseling skills (5.45) and lowest scores for Organization/efficiency (4.81) followed by physical examination skills (5.14) on a nine-point Likert scale [Table 1]. Maximum improvement between the first encounter and last encounter was seen in Humanistic qualities/professionalism while minimum improvement was seen in physical examination skills.
Hundred percent residents said that the mini-CEX exercise was useful for them. The mean score for the extent of its usefulness was 5.47 (78.14%) on a scale of 1-7. Majority of students opined that faculty feedback was more useful than multiple encounters being observed with mean scores of 5.9 (84.2%) and 5.6 (80%) respectively. The feedback on extent of improvement of individual skills was most satisfying for Humanistic qualities/professionalism with a mean score of 5.7 (81.42%) and was least satisfying for physical examination skills with a mean score of 4.9 (70%) on a scale of 1-7 [Table 2].
Patients feedback on attending resident's communication behaviors was very satisfying with more than 90% of them said that they treated patient and his/her family with respect and dignity, encouraged patient to ask questions, listened to them without interrupting and discussed management options with them [Table 3].
|Table 3: Patients ratings of attending resident's communication behaviors |
Click here to view
The mean score for faculty satisfaction with mini-CEX was 5.33, and for the students, it was 5.48. 100% faculty felt that there was an improvement in resident's clinical skills with special mention of Humanistic qualities/professionalism since the time we introduced mini-CEX in our department.
| Discussion|| |
There has been a growing concern among medical educators about quality of medical graduates and postgraduates trained in various medical colleges in our country.  The clinical skills of medical interviewing, physical examination, and counseling remain vital to effective care of patients, yet research continues to document serious deficiencies in clinical skills among students and residents. ,
This pilot study was clear in its purpose. It was meticulously planned and prepared in careful, systematic details. The objectives were defined, blueprinting was done, and adequate attention was paid to the response process.
We introduced mini-CEX as part of formative assessment for postgraduates at our medical college in the year 2013-2014. We found as have others , that mini-CEX as a method of examining clinical skills is superior to the oral examination in several ways, and was preferred by both faculty and students in our institution.
Mean score was highest for Humanistic qualities/professionalism (5.48) followed by counseling skills (5.45) and lowest scores for Organization/efficiency (4.81) followed by Physical examination skills (5.14) on a nine-point Likert scale. Maximum improvement between the first encounter and last encounter was seen in Humanistic qualities/professionalism while minimum improvement was seen in physical examination skills which corroborate with findings of other studies. 
Hundred percent residents said that the mini-CEX exercise was useful for them. The mean score for the extent of its usefulness was 5.47 (78.14%) on a scale of 1-7. Majority of students opined that faculty feedback was more useful than multiple encounters being observed with mean scores of 5.9 (84.2%) and 5.6 (80%), respectively. According to Holm et al., the involvement by senior doctors is unstructured and haphazard, and minimum attention has been paid to the educational needs of the trainee (Holm, 2002).  In our study, senior faculty involvement helped in giving the desirable feedback in an effective way with more student satisfaction.
There has been a move in recent years toward competency-based medical training because patients expect doctors to diagnose, plan management, carry out practical procedures, and behave in a reasonable way demonstrating a caring and humanistic attitude.  In this study, the feedback on extent of improvement of individual skills was most satisfying for Humanistic qualities/professionalism with a mean score of 5.7 (81.42%) and was least satisfying for physical examination skills with a mean score of 4.9 (70%) on a scale of 1-7.
Patient's feedback on attending resident's communication behaviors was very satisfying with more than 90% of them rating them as good to excellent.
Hundred percent faculties felt that there was an improvement in resident's clinical skills with special mention of Humanistic qualities/professionalism since the time we introduced mini-CEX in our department.
| Conclusion|| |
In conclusion, we believe that the Mini-CEX is an examination tool, which permits evaluation based on a much broader set of clinical settings and patients. It also gives an opportunity to observe and communicate with the students in real life settings. It can be used as a tool to identify and highlight deficient areas in individual performance for further improvement. Yet there were certain limitations. Number of encounters observed was small. Scoring variability among raters as found by Hawkins RE that acts as a confounding factor in validity of mini-CEX conduction.  More research is needed to determine whether scoring related issues limit the utility of mini-CEX for providing the feedback to the students. Malhotra S has shown that students consider mini-CEX to be anxiety provoking due to their dual nature of the examination focusing on assessment and education. We also had to remind students from time to time that the focus is on learning, and these scores will not be included in university exams. 
| Acknowledgments|| |
Our sincere thanks to the faculties at Institute of Medical Education Technology and Teachers Training, Pune for their valuable inputs.
| References|| |
Southgate L. Professional competence in medicine. Hosp Med 1999;60:203-5.
Saedon H, Saedon MH, Aggarwal SP. Workplace-based assessment as an educational tool: Guide supplement 31.3-viewpoint. Med Teach 2010;32:e369-72.
Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE Guide No 31. Med Teach 2007;29:855-71.
Pelgrim EA, Kramer AW, Mokkink HG, van den Elsen L, Grol RP, van der Vleuten CP. In-training assessment using direct observation of single-patient encounters: A literature review. Adv Health Sci Educ Theory Pract 2011;16:131-42.
Wass V, Van der Vleuten C, Shatzer J, Jones R. Assessment of clinical competence. Lancet 2001;357:945-9.
Norcini JJ, Blank LL, Arnold GK, Kimball HR. The mini-CEX (clinical evaluation exercise): A preliminary investigation. Ann Intern Med 1995;123:795-9.
Norcini JJ, Blank LL, Duffy FD, Fortna GS. The mini-CEX: A method for assessing clinical skills. Ann Intern Med 2003;138:476-81.
Holmboe ES, Hawkins RE, Huot SJ. Effects of training in direct observation of medical residents' clinical competence: A randomized trial. Ann Intern Med 2004;140:874-81.
Durning SJ, Cation LJ, Markert RJ, Pangaro LN. Assessing the reliability and validity of the mini-clinical evaluation exercise for internal medicine residency training. Acad Med 2002;77:900-4.
Holmboe ES, Huot S, Chung J, Norcini J, Hawkins RE. Construct validity of the miniclinical evaluation exercise (miniCEX). Acad Med 2003;78:826-30.
Alves de Lima A, Barrero C, Baratta S, Castillo Costa Y, Bortman G, Carabajales J, et al.
Validity, reliability, feasibility and satisfaction of the Mini-Clinical Evaluation Exercise (Mini-CEX) for cardiology residency training. Med Teach 2007;29:785-90.
Anderson CI, Jentz AB, Harkema JM, Kareti LR, Apelgren KN, Slomski CA. Assessing the competencies in general surgery residency training. Am J Surg 2005;189:288-92.
Burch VC, Seggie JL, Gary NE. Formative assessment promotes learning in undergraduate clinical clerkships. S Afr Med J 2006;96:430-3.
Cook DA, Beckman TJ. Does scale length matter? A comparison of nine- versus five-point rating scales for the mini-CEX. Adv Health Sci Educ Theory Pract 2009;14:655-64.
Cook DA, Dupras DM, Beckman TJ, Thomas KG, Pankratz VS. Effect of rater training on reliability and accuracy of mini-CEX scores: A randomized, controlled trial. J Gen Intern Med 2009;24:74-9.
Cruess R, McIlroy JH, Cruess S, Ginsburg S, Steinert Y. The Professionalism Mini-evaluation Exercise: A preliminary investigation. Acad Med 2006;81:S74-8.
Donato AA, Pangaro L, Smith C, Rencic J, Diaz Y, Mensinger J, et al.
Evaluation of a novel assessment form for observing medical residents: A randomised, controlled trial. Med Educ 2008;42:1234-42.
Dowson C, Hassell A, Training Sub-Committee of the West Midlands Rheumatology Services and Training Committee (WMRSTC). Competence-based assessment of specialist registrars: Evaluation of a new assessment of out-patient consultations. Rheumatology (Oxford) 2006;45:459-64.
Golnik KC, Goldenhar L. The ophthalmic clinical evaluation exercise: Reliability determination. Ophthalmology 2005;112:1649-54.
Golnik KC, Goldenhar LM, Gittinger JW Jr, Lustbader JM. The Ophthalmic Clinical Evaluation Exercise (OCEX). Ophthalmology 2004;111:1271-4.
Bhatnagar KR, Saoji VA, Banerjee AA. Objective structured clinical examination for undergraduates: Is it a feasible approach to standardized assessment in India? Indian J Ophthalmol 2011;59:211-4.
Wilkinson TJ, Wade WB, Knock LD. A blueprint to assess professionalism: Results of a systematic review. Acad Med 2009;84:551-8.
Nair BR, Alexander HG, McGrath BP, Parvathy MS, Kilsby EC, Wenzel J, et al.
The mini clinical evaluation exercise (mini-CEX) for assessing clinical performance of international medical graduates. Med J Aust 2008;189:159-61.
Carr SJ. Assessing clinical competency in medical senior house officers: How and why should we do it? Postgrad Med J 2004;80:63-6.
Carr S. The Foundation Programme assessment tools: An opportunity to enhance feedback to trainees? Postgrad Med J 2006;82:576-9.
Holm HA. Postgraduate education. In: Norman GR, van der Vleuten CP, Newble DI, editors. International Handbook of Research in Medical Education. Dordrecht: Kluwer; 2002. p. 381-413.
Hawkins RE, Margolis MJ, Durning SJ, Norcini JJ. Constructing a validity argument for the mini-Clinical Evaluation Exercise: A review of the research. Acad Med 2010;85:1453-61.
Malhotra S, Hatala R, Courneya CA. Internal medicine residents' perceptions of the Mini-Clinical Evaluation Exercise. Med Teach 2008;30:414-9.
[Table 1], [Table 2], [Table 3]