Home Print this page Email this page
Users Online: 198


 
 Table of Contents  
ORIGINAL ARTICLE
Year : 2014  |  Volume : 6  |  Issue : 2  |  Page : 49-53

Formative assessment using direct observation of single-patient encounters in ophthalmology residency


Department of Ophthalmology, Pad. Dr. D Y Patil Medical College, Hospital and Research Center, Pune, Maharashtra, India

Date of Web Publication6-Feb-2015

Correspondence Address:
Kavita Bhatnagar
B4/21, Brahma Angan, Salunke Vihar Road, Pune - 411 048, Maharashtra
India
Login to access the Email id

DOI: 10.4103/1858-540X.150994

Rights and Permissions
  Abstract 

Background: There has been a growing concern that trainees are infrequently observed, assessed, and given feedback during their workplace-based education. This has led to an increasing interest in a variety of formative assessment methods that require observation and offer the opportunity for feed. Mini-clinical evaluation exercise (mini-CEX) is an observation tool that facilitates the assessment of skills that are essential for good clinical care and provision of immediate feedback. The aim of this article is to sensitize academician-clinician in developing countries to mini-CEX. Materials and Methods: This observational study was conducted on 10 postgraduate students who were assessed on clinical skills, communication skills, professionalism and ethics in different clinical settings using a validated mini-CEX global rating scale. Data analysis was done. Student's scores in the first encounter and last encounter were compared to see an improvement in their performance. Results: A total of 216 encounters was observed and rated on a nine-point Likert scale. Mean score and maximum improvement were seen for Humanistic qualities/professionalism. About 100% residents found mini-CEX exercise useful. Faculty feedback was more useful than multiple encounters being observed. Over 90% patients were highly satisfied with resident behavior. The mean score for faculty satisfaction with mini-CEX was 5.33, and for the students, it was 5.48. Conclusion: Mini-CEX is an examination tool which permits evaluation based on a much broader set of clinical settings and patients. It also gives an opportunity to observe and communicate with the students in real life settings. It can be used as a tool to identify and highlight deficient areas in individual performance for further improvement.

Keywords: Clinical competence, communication skills, mini-clinical evaluation exercise, professionalism/ethics, workplace based assessments


How to cite this article:
Bhatnagar K, Radhakrishnan O K, Lune A, Sandhya K. Formative assessment using direct observation of single-patient encounters in ophthalmology residency. Sudanese J Ophthalmol 2014;6:49-53

How to cite this URL:
Bhatnagar K, Radhakrishnan O K, Lune A, Sandhya K. Formative assessment using direct observation of single-patient encounters in ophthalmology residency. Sudanese J Ophthalmol [serial online] 2014 [cited 2019 Sep 20];6:49-53. Available from: http://www.sjopthal.net/text.asp?2014/6/2/49/150994


  Introduction Top


Workplace based assessments are being used increasingly during postgraduate medical training as a method of assessing competence. [1],[2] There has been a concern that trainees are infrequently observed, assessed, and given feedback during their workplace-based education that is in outpatient (OP), inpatient and emergency department. This has led to an increasing interest in a variety of formative assessment methods that require observation and offer the opportunity for feedback. (Norcini 2007). [3]

There are many methods including direct observation of procedural skills, mini-clinical evaluation exercise (mini-CEX) and case-based discussions. All these methods are used in Ophthalmology in western countries but are still in their infancy in India.

The mini-CEX is widely used for assessment in single work-based encounters of clinical competence at the top of Miller's pyramid the "does" level. Currently, assessment of clinical competence is receiving increasing attention, particularly in postgraduate training (Wass et al. 2001), [4],[5] and assessment of authentic performance is considered the main challenge. Reliable and valid performance measurements that can serve as a gold standard for clinical assessment have as yet not been achieved (Wass et al. 2001). [4],[5] Developed for the evaluation of a multitude of clinical competencies (Norcini et al. 1995), [6],[7] the mini-CEX is a single-encounter instrument to be used by professionals in conducting work-based assessment of actual clinical performance. It was originally developed by American Board of Internal Medicine in 1995 in the USA for the evaluation of internal medicine residents' clinical skills (Norcini et al. 1995, 2003) [6],[7] and its principal characteristics are direct observation of real patient encounters, easy and instant use in day-to-day practice, applicability in a broad range of settings and immediate feedback to the learner after the encounter. These characteristics make the mini-CEX an educational tool that can help learners to gain insight into the strengths and weaknesses of their clinical performance. [8] It can be used to assess multiple competencies, such as communication and professionalism. Typically, the mini-CEX and similar instruments use global assessment scales, provide space for narrative comments and allow for feedback presented by a moderator in a postencounter review session. Previous data on mini-CEX suggest that it is a valid and reliable evaluation tool. [9],[10],[11],[12],[13],[14],[15],[16],[17],[18],[19],[20]

Our college was established in 1996, and postgraduate course in Ophthalmology was started in the year 2002 with two Masters in Ophthalmology (MS) and one Diploma in Ophthalmology (DOMS) intake annually. Today, we have an intake of six MS and two DOMS students every year so at time we are training 22 residents in the department that is a big number. After passing out, majority will start private practice where they will be treating patients independently.

We realized that observing trainees during their daily practice in OP or inpatient department is a time-efficient way of determining their level of competence. By giving great importance to the concept of feedback, trainees will be encouraged to reflect upon the learning experience and can, therefore, improve their clinical and communication skills including professionalism. Mini-CEX was introduced as part of formative assessment in the department of ophthalmology of our college with subsequent inclusion in university examination with time and experience.


  Materials and methods Top


This Observational Study was conducted on 10 Postgraduate students in the department of Ophthalmology of a medical college. Institutional Ethical Committee clearance was obtained before starting the study. Residents were informed that they will be assessed using this new method of assessment as part of internal assessment.

All participating faculties were trained by the first author in this new method of assessment and art of giving effective feedback. This included a combination of didactic lectures and videos of mini-CEX on simulated patients. They were also given literature on mini-CEX for further reading.

Students were assessed on their history taking skills, physical examination skills, clinical judgment skills, communication skills, professionalism and ethics, and organizational ability in different clinical settings using a prevalidated mini-CEX global rating scale and a check list for communication skills by different examiners. Minimum six resident-patient encounters were recorded per resident by four faculty members with more than 10 years of clinical experience. Each encounter was of 15-20 min duration and 5 min were assigned for feedback to the students. The problems observed varied in difficulty levels and also during busy OP department (OPD) days as well as lighter OPDs.

For each encounter, the examiner recorded the date, the complexity of the patient's problem on a 3-point scale (low, moderate, and high), the sex of the patient, the type of visit (new or follow up), the setting (OP, inpatient, or emergency department), the number of minutes spent observing the encounter, and the number of minutes spent giving feedback. The examiner also noted whether the focus of the encounter was data gathering, diagnosis, treatment or counseling.

Using a nine-point scale (in which 1-3 were "unsatisfactory," four was "marginal," five and six were "satisfactory," and 7-9 were "superior"), the examiner rated the resident on history taking, physical examination, professionalism, clinical judgment, counseling, organization and efficiency, and overall competence. The examiner also rated his or her own satisfaction with the method as a valid and efficient assessment device on a nine-point scale in which one was "dissatisfied" and nine was "very satisfied." For all items, the examiner could select "not applicable" where appropriate. In addition, a total score was calculated as the mean of the six component ratings.


  Statistical analysis Top


Data were collected and recorded in Microsoft Excel spreadsheet. They were verified and completed according to the mini-CEX rating forms and feedback questionnaires. They were presented in the form of central tendencies. Appropriate tables were projected for data.


  Results Top


A total of four senior faculties and 10 postgraduate students posted in author's unit in the Department of Ophthalmology were enrolled for the study. Out of 10 students, five were in 2 nd year of residency and five in 3 rd year of residency. The total number of resident-patient encounters observed were 216. Mean score was highest for Humanistic qualities/professionalism (5.48) followed by counseling skills (5.45) and lowest scores for Organization/efficiency (4.81) followed by physical examination skills (5.14) on a nine-point Likert scale [Table 1]. Maximum improvement between the first encounter and last encounter was seen in Humanistic qualities/professionalism while minimum improvement was seen in physical examination skills.
Table 1: Mean scores in the components of mini-CEX


Click here to view


Hundred percent residents said that the mini-CEX exercise was useful for them. The mean score for the extent of its usefulness was 5.47 (78.14%) on a scale of 1-7. Majority of students opined that faculty feedback was more useful than multiple encounters being observed with mean scores of 5.9 (84.2%) and 5.6 (80%) respectively. The feedback on extent of improvement of individual skills was most satisfying for Humanistic qualities/professionalism with a mean score of 5.7 (81.42%) and was least satisfying for physical examination skills with a mean score of 4.9 (70%) on a scale of 1-7 [Table 2].
Table 2: Student feedback on their experience with mini-CEX


Click here to view


Patients feedback on attending resident's communication behaviors was very satisfying with more than 90% of them said that they treated patient and his/her family with respect and dignity, encouraged patient to ask questions, listened to them without interrupting and discussed management options with them [Table 3].
Table 3: Patients ratings of attending resident's communication behaviors


Click here to view


The mean score for faculty satisfaction with mini-CEX was 5.33, and for the students, it was 5.48. 100% faculty felt that there was an improvement in resident's clinical skills with special mention of Humanistic qualities/professionalism since the time we introduced mini-CEX in our department.


  Discussion Top


There has been a growing concern among medical educators about quality of medical graduates and postgraduates trained in various medical colleges in our country. [21] The clinical skills of medical interviewing, physical examination, and counseling remain vital to effective care of patients, yet research continues to document serious deficiencies in clinical skills among students and residents. [22],[23]

This pilot study was clear in its purpose. It was meticulously planned and prepared in careful, systematic details. The objectives were defined, blueprinting was done, and adequate attention was paid to the response process.

We introduced mini-CEX as part of formative assessment for postgraduates at our medical college in the year 2013-2014. We found as have others [19],[20] that mini-CEX as a method of examining clinical skills is superior to the oral examination in several ways, and was preferred by both faculty and students in our institution.

Mean score was highest for Humanistic qualities/professionalism (5.48) followed by counseling skills (5.45) and lowest scores for Organization/efficiency (4.81) followed by Physical examination skills (5.14) on a nine-point Likert scale. Maximum improvement between the first encounter and last encounter was seen in Humanistic qualities/professionalism while minimum improvement was seen in physical examination skills which corroborate with findings of other studies. [24]

Hundred percent residents said that the mini-CEX exercise was useful for them. The mean score for the extent of its usefulness was 5.47 (78.14%) on a scale of 1-7. Majority of students opined that faculty feedback was more useful than multiple encounters being observed with mean scores of 5.9 (84.2%) and 5.6 (80%), respectively. According to Holm et al., the involvement by senior doctors is unstructured and haphazard, and minimum attention has been paid to the educational needs of the trainee (Holm, 2002). [26] In our study, senior faculty involvement helped in giving the desirable feedback in an effective way with more student satisfaction.

There has been a move in recent years toward competency-based medical training because patients expect doctors to diagnose, plan management, carry out practical procedures, and behave in a reasonable way demonstrating a caring and humanistic attitude. [26] In this study, the feedback on extent of improvement of individual skills was most satisfying for Humanistic qualities/professionalism with a mean score of 5.7 (81.42%) and was least satisfying for physical examination skills with a mean score of 4.9 (70%) on a scale of 1-7.

Patient's feedback on attending resident's communication behaviors was very satisfying with more than 90% of them rating them as good to excellent.

Hundred percent faculties felt that there was an improvement in resident's clinical skills with special mention of Humanistic qualities/professionalism since the time we introduced mini-CEX in our department.


  Conclusion Top


In conclusion, we believe that the Mini-CEX is an examination tool, which permits evaluation based on a much broader set of clinical settings and patients. It also gives an opportunity to observe and communicate with the students in real life settings. It can be used as a tool to identify and highlight deficient areas in individual performance for further improvement. Yet there were certain limitations. Number of encounters observed was small. Scoring variability among raters as found by Hawkins RE that acts as a confounding factor in validity of mini-CEX conduction. [27] More research is needed to determine whether scoring related issues limit the utility of mini-CEX for providing the feedback to the students. Malhotra S has shown that students consider mini-CEX to be anxiety provoking due to their dual nature of the examination focusing on assessment and education. We also had to remind students from time to time that the focus is on learning, and these scores will not be included in university exams. [28]


  Acknowledgments Top


Our sincere thanks to the faculties at Institute of Medical Education Technology and Teachers Training, Pune for their valuable inputs.

 
  References Top

1.
Southgate L. Professional competence in medicine. Hosp Med 1999;60:203-5.  Back to cited text no. 1
    
2.
Saedon H, Saedon MH, Aggarwal SP. Workplace-based assessment as an educational tool: Guide supplement 31.3-viewpoint. Med Teach 2010;32:e369-72.  Back to cited text no. 2
    
3.
Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE Guide No 31. Med Teach 2007;29:855-71.  Back to cited text no. 3
    
4.
Pelgrim EA, Kramer AW, Mokkink HG, van den Elsen L, Grol RP, van der Vleuten CP. In-training assessment using direct observation of single-patient encounters: A literature review. Adv Health Sci Educ Theory Pract 2011;16:131-42.  Back to cited text no. 4
    
5.
Wass V, Van der Vleuten C, Shatzer J, Jones R. Assessment of clinical competence. Lancet 2001;357:945-9.  Back to cited text no. 5
    
6.
Norcini JJ, Blank LL, Arnold GK, Kimball HR. The mini-CEX (clinical evaluation exercise): A preliminary investigation. Ann Intern Med 1995;123:795-9.  Back to cited text no. 6
    
7.
Norcini JJ, Blank LL, Duffy FD, Fortna GS. The mini-CEX: A method for assessing clinical skills. Ann Intern Med 2003;138:476-81.  Back to cited text no. 7
    
8.
Holmboe ES, Hawkins RE, Huot SJ. Effects of training in direct observation of medical residents' clinical competence: A randomized trial. Ann Intern Med 2004;140:874-81.  Back to cited text no. 8
    
9.
Durning SJ, Cation LJ, Markert RJ, Pangaro LN. Assessing the reliability and validity of the mini-clinical evaluation exercise for internal medicine residency training. Acad Med 2002;77:900-4.  Back to cited text no. 9
    
10.
Holmboe ES, Huot S, Chung J, Norcini J, Hawkins RE. Construct validity of the miniclinical evaluation exercise (miniCEX). Acad Med 2003;78:826-30.  Back to cited text no. 10
    
11.
Alves de Lima A, Barrero C, Baratta S, Castillo Costa Y, Bortman G, Carabajales J, et al. Validity, reliability, feasibility and satisfaction of the Mini-Clinical Evaluation Exercise (Mini-CEX) for cardiology residency training. Med Teach 2007;29:785-90.  Back to cited text no. 11
    
12.
Anderson CI, Jentz AB, Harkema JM, Kareti LR, Apelgren KN, Slomski CA. Assessing the competencies in general surgery residency training. Am J Surg 2005;189:288-92.  Back to cited text no. 12
    
13.
Burch VC, Seggie JL, Gary NE. Formative assessment promotes learning in undergraduate clinical clerkships. S Afr Med J 2006;96:430-3.  Back to cited text no. 13
    
14.
Cook DA, Beckman TJ. Does scale length matter? A comparison of nine- versus five-point rating scales for the mini-CEX. Adv Health Sci Educ Theory Pract 2009;14:655-64.  Back to cited text no. 14
    
15.
Cook DA, Dupras DM, Beckman TJ, Thomas KG, Pankratz VS. Effect of rater training on reliability and accuracy of mini-CEX scores: A randomized, controlled trial. J Gen Intern Med 2009;24:74-9.  Back to cited text no. 15
    
16.
Cruess R, McIlroy JH, Cruess S, Ginsburg S, Steinert Y. The Professionalism Mini-evaluation Exercise: A preliminary investigation. Acad Med 2006;81:S74-8.  Back to cited text no. 16
    
17.
Donato AA, Pangaro L, Smith C, Rencic J, Diaz Y, Mensinger J, et al. Evaluation of a novel assessment form for observing medical residents: A randomised, controlled trial. Med Educ 2008;42:1234-42.  Back to cited text no. 17
    
18.
Dowson C, Hassell A, Training Sub-Committee of the West Midlands Rheumatology Services and Training Committee (WMRSTC). Competence-based assessment of specialist registrars: Evaluation of a new assessment of out-patient consultations. Rheumatology (Oxford) 2006;45:459-64.  Back to cited text no. 18
    
19.
Golnik KC, Goldenhar L. The ophthalmic clinical evaluation exercise: Reliability determination. Ophthalmology 2005;112:1649-54.  Back to cited text no. 19
    
20.
Golnik KC, Goldenhar LM, Gittinger JW Jr, Lustbader JM. The Ophthalmic Clinical Evaluation Exercise (OCEX). Ophthalmology 2004;111:1271-4.  Back to cited text no. 20
    
21.
Bhatnagar KR, Saoji VA, Banerjee AA. Objective structured clinical examination for undergraduates: Is it a feasible approach to standardized assessment in India? Indian J Ophthalmol 2011;59:211-4.  Back to cited text no. 21
[PUBMED]  Medknow Journal  
22.
Wilkinson TJ, Wade WB, Knock LD. A blueprint to assess professionalism: Results of a systematic review. Acad Med 2009;84:551-8.  Back to cited text no. 22
    
23.
Nair BR, Alexander HG, McGrath BP, Parvathy MS, Kilsby EC, Wenzel J, et al. The mini clinical evaluation exercise (mini-CEX) for assessing clinical performance of international medical graduates. Med J Aust 2008;189:159-61.  Back to cited text no. 23
    
24.
Carr SJ. Assessing clinical competency in medical senior house officers: How and why should we do it? Postgrad Med J 2004;80:63-6.  Back to cited text no. 24
    
25.
Carr S. The Foundation Programme assessment tools: An opportunity to enhance feedback to trainees? Postgrad Med J 2006;82:576-9.  Back to cited text no. 25
    
26.
Holm HA. Postgraduate education. In: Norman GR, van der Vleuten CP, Newble DI, editors. International Handbook of Research in Medical Education. Dordrecht: Kluwer; 2002. p. 381-413.  Back to cited text no. 26
    
27.
Hawkins RE, Margolis MJ, Durning SJ, Norcini JJ. Constructing a validity argument for the mini-Clinical Evaluation Exercise: A review of the research. Acad Med 2010;85:1453-61.  Back to cited text no. 27
    
28.
Malhotra S, Hatala R, Courneya CA. Internal medicine residents' perceptions of the Mini-Clinical Evaluation Exercise. Med Teach 2008;30:414-9.  Back to cited text no. 28
    



 
 
    Tables

  [Table 1], [Table 2], [Table 3]



 

Top
 
 
  Search
 
Similar in PUBMED
   Search Pubmed for
   Search in Google Scholar for
 Related articles
Access Statistics
Email Alert *
Add to My List *
* Registration required (free)

 
  In this article
Abstract
Introduction
Materials and me...
Statistical analysis
Results
Discussion
Conclusion
Acknowledgments
References
Article Tables

 Article Access Statistics
    Viewed5671    
    Printed143    
    Emailed0    
    PDF Downloaded1224    
    Comments [Add]    

Recommend this journal


[TAG2]
[TAG3]
[TAG4]