|
Sign In to gain access to subscriptions and/or personal tools.
|
Convergence Between Cluster Analysis and the Angoff Method for Setting Minimum Passing Scores on Credentialing Examinations
Brian Hess
American Board of Internal Medicine
Raja G. Subhiyah
National Board of Medical Examiners
Carolyn Giordano
National Board of Medical Examiners
Cluster analysis can be a useful statistical technique for setting minimum passing scores on high-stakes examinations by grouping examinees into homogenous clusters based on their responses to test items. It has been most useful for supplementing data or validating minimum passing scores determined from expert judgment approaches, such as the Ebel and Nedelsky methods. However, there is no evidence supporting how well cluster analysis converges with the modified Angoff method, which is frequently used in medical credentialing. Therefore, the purpose of this study is to investigate the efficacy of cluster analysis for validating Angoff-derived minimum passing scores. Data are from 652 examinees who took a national credentialing examination based on a content-by-process test blueprint. Results indicate a high degree of consistency in minimum passing score estimates derived from the modified Angoff and cluster analysis methods. However, the stability of the estimates from cluster analysis across different samples was modest.
Key Words: standard setting • Angoff method • cluster analysis • high-stakes examinations
References
- Anderberg, M.R. (1973). Cluster analysis for applications. New York: Academic Press.
- Angoff, W.H. (1971). Scales, norms, and equivalent scores. In R. L. Thorndike (Ed.), Educational measurement (2nd ed., pp. 508-600). Washington, DC: American Council on Education.
- Chinn, R.N., & Hertz, N.R. (2002). Alternative approaches to standard setting for licensure and certification examinations. Applied Measurement in Education, 15, 1-14.[CrossRef][Web of Science]
- Cizek, G.J. (2001). Conjectures on the rise and call of standard setting: An introduction to context and practice. In G. J. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 1-17). Mahwah, NJ: Lawrence Erlbaum.
- Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 10, 37-46.[Medline]
[Order article via Infotrieve]
- Downing, S.M., Tekian, A., & Yudkowsky, R. (2006). Procedures for establishing defensible absolute passing scores on performance examinations in health professions education. Teaching and Learning in Medicine, 18, 50-57.
- Ebel, R.L. (1979). Essentials of educational measurement (3rd. ed.). Englewood Cliffs, NJ: Prentice Hall.
- Engelhard, G., & Stone, G.E. (1998). Evaluating the quality of ratings obtained from standard-setting judges. Educational and Psychological Measurement, 58, 179-196.[Abstract]
- Hurtz, G.M., & Auerbach, M.A. (2003). A meta-analysis of the effects of modifications to the Angoff method on cutoff scores and judgment consensus. Educational and Psychological Measurement, 63, 584-601.[Abstract/Free Full Text]
- Livingston, S.A., & Zieky, M.J. (1989). A comparative study of standard-setting methods. Applied Measurement in Education, 2, 121-141.[CrossRef]
- Meara, K.P., Hambleton, R.K., & Sireci, S.G. (2001). Setting and validating standards on professional licensure and certification exams: A survey of current practices. Clear Exam Review, 12, 17-23.
- Nedelsky, L. (1954). Absolute grading standards for objective tests. Educational and Psychological Measurement, 14, 3-19.[Medline]
[Order article via Infotrieve]
- Plake, B.S., Hambleton, R.K., & Jaeger, R.M. (1997). A new standard-setting method for performance assessments: The dominant profile judgment method and some field tests results. Educational and Psychological Measurement, 57, 400-411.[Abstract]
- Ramsey, P.G., Carline, J.D., Inui, T.S., Larson, E.B., LoGerfo, J.P., Norcini, J.J., et al. (1991). Changes over time in the knowledge base of practicing internists. Journal of the American Medical Association, 266, 1103-1107.[CrossRef]
- Searle, J. (2000). Defining competency: The role of standard setting. Medical Education, 34, 363-366.[CrossRef][Web of Science][Medline]
[Order article via Infotrieve]
- Sireci, S.G. (2001). Standard setting using cluster analysis. In G. J. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 339-354). Mahwah, NJ: Lawrence Erlbaum.
- Sireci, S.G., Hambleton, R.K., & Pitoniak, M.J. (2004). Setting passing scores on licensure examinations using direct consensus. Clear Exam Review, 15, 21-25.
- Sireci, S.G., Robin, F., & Patelis, T. (1999). Using cluster analysis to facilitate standard setting. Applied Measurement in Education, 12, 301-325.[CrossRef][Web of Science]
- Truxillo, D.M., Donahue, L.M., & Sulzer, J.L. (1996). Setting cutoff scores for personnel selection tests: Issues, illustrations, and recommendations. Human Performance, 9, 275-295.[CrossRef][Web of Science]
- Violato, C., Marini, A., & Lee, C. (2003). A validity study of expert judgment procedures for setting cutoff scores on high-stakes credentialing examinations using cluster analysis. Evaluation and the Health Professions, 26, 59-72.[Abstract/Free Full Text]
- Wang, N. (2003). Use of the Rasch IRT model in standard setting: An item-mapping method. Journal of Educational Measurement, 40, 231-253.[CrossRef][Web of Science]
Evaluation & the Health Professions, Vol. 30, No. 4,
362-375 (2007)
DOI: 10.1177/0163278707307904
 CiteULike  Complore  Connotea  Del.icio.us  Digg  Reddit  Technorati  Twitter What's this?
|
|
