Illinois — Targeted Issue

Author(s)/Presenter(s)

Adler MD, Vozenilek JA, Trainor JL, Eppich WJ, Wang EE, Beaumont JL, Aitchison PR, Pribaz PJ, Erickson T, Edison M, McGaghie WC

Abstract/Description

PURPOSE: To compare the psychometric performance of two rating instruments used to assess trainee performance in three clinical scenarios.
METHODS: This study was part of a two-phase, randomized trial with a wait-list control condition assessing the effectiveness of a pediatric emergency medicine curriculum targeting general emergency medicine residents. Residents received 6 hours of instruction either before or after the first assessment. Separate pairs of raters completed either a dichotomous checklist for each of three cases or the Global Performance Assessment Tool (GPAT), an anchored multidimensional scale. A fully crossed person×rater×case generalizability study was conducted. The effect of training year on performance is assessed using multivariate analysis of variance.
RESULTS: The person and person×case components accounted for most of the score variance for both instruments. Using either instrument, scores demonstrated a small but significant increase as training level increased when analyzed using a multivariate analysis of variance. The inter-rater reliability coefficient was >0.9 for both instruments.
CONCLUSIONS: We demonstrate that our checklist and anchored global rating instrument performed in a psychometrically similar fashion with high reliability. As long as proper attention is given to instrument design and testing and rater training, checklists and anchored assessment scales can produce reproducible data for a given population of subjects. The validity of the data arising for either instrument type must be assessed rigorously and with a focus, when practicable, on patient care outcomes.

Author(s)/Presenter(s)

Adler MD, Vozenilek JA, Trainor JL, Eppich WJ, Wang EE, Beaumont JL, Aitchison PR, Erickson T, Edison M, McGaghie WC

Abstract/Description

PURPOSE: The infrequency of severe childhood illness limits opportunities for emergency medicine (EM) providers to learn from real-world experience. Simulation offers an evidence-based educational approach to develop and practice clinical skills.
METHODS: This was a two-phase, randomized trial with a wait-list control condition. The development phase (2005-2006) involved systematic curriculum and rating checklist creation, producing a six-case, simulation-based curriculum linked to three evaluation cases. In the validation phase (2006-2007), the authors randomized 69 residents from two EM residencies to either an intervention group that received the curriculum one month before the first assessment of all participants or a wait-list control group that received the identical curriculum three months later. A final assessment of all residents followed one month after that. Two raters evaluated all residents. Primary outcome measures are percentages of items completed correctly. The authors assessed rater agreement using intraclass correlation (ICC) and compared group performance using mixed-model analysis of variance.
RESULTS: ICCs surpassed 0.78. The instructional intervention produced a statistically significant effect for two of three evaluation cases for the validation phase of the study, a case x occasion interaction. Training year was significantly associated with better performance. In a multivariate analysis, training year and session correlated with score, but study group did not.
CONCLUSION: A one-day, simulation-based pediatric EM curriculum produced limited results. The evaluation approach is reasonable and reproducible for the population studied. Instructional dose strength and factors may have limited curriculum effectiveness. Focused, frequent, and effortful instructional interventions are necessary to achieve substantial performance improvements.