This Article Full Text (PDF) All Versions of this Article: 0163278708324444v1 31/4/419    most recent References Alert me when this article is cited Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Add to Saved Citations Download to citation manager Request Permissions Request Reprints Add to My Marked Citations Citing Articles Citing Articles via Google Scholar Citing Articles via Scopus Google Scholar Articles by Fernandez, S. A. Articles by Jarjoura, D. Search for Related Content PubMed PubMed Citation Articles by Fernandez, S. A. Articles by Jarjoura, D. Social Bookmarking                         What's this?

Reliability of Surgical Skills Scores in Otolaryngology Residents

Analysis Using Generalizability Theory

Ohio State University, Columbus, Soledad.Fernandez{at}osumc.edu

Gregory J. Wiet

Columbus Children's Hospital, Columbus, Ohio Ohio State University

Nancy N. Butler

Ohio State University, Columbus

Bradley Welling

Ohio State University, Columbus

David Jarjoura

Ohio State University, Columbus

Assessments of temporal bone dissection performance among otolaryngology residents have not been adequately developed. At the Ohio State College of Medicine, an instrument (Welling Scale, Version 1 [WS1]) is used to evaluate residents' end-product performance after drilling a temporal bone. In this study, the authors evaluate the components that contribute to measurement error using this scale. Generalizability theory was used to reveal components of measurement error that allow for better understanding of test results. A major component of measurement error came from inconsistency in performance across the two cadaveric test bones each resident was assigned. In contrast, ratings of performance using the WS1 were highly consistent across raters and rating sessions within raters. The largest source of measurement error was caused by residents' inconsistent performance across bones. Rater disagreement introduced only small error into scores. The WS1 provides small measurement error, with two raters and two bones for each participant.

Key Words: measurement error variance • mixed models • performance measures

This version was published on December 1, 2008

Evaluation & the Health Professions, Vol. 31, No. 4, 419-436 (2008)
DOI: 10.1177/0163278708324444


CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter    What's this?