Evaluation & the Health Professions recent issues

Analyzing Longitudinal Data With the Linear Mixed Models Procedure in SPSS

West, B. T. — Tue, 25 Aug 2009 03:00:12 PDT

Many applied researchers analyzing longitudinal data share a common misconception: that specialized statistical software is necessary to fit hierarchical linear models (also known as linear mixed models [LMMs], or multilevel models) to longitudinal data sets. Although several specialized statistical software programs of high quality are available that allow researchers to fit these models to longitudinal data sets (e.g., HLM), rapid advances in general purpose statistical software packages have recently enabled analysts to fit these same models when using preferred packages that also enable other more common analyses. One of these general purpose statistical packages is SPSS, which includes a very flexible and powerful procedure for fitting LMMs to longitudinal data sets with continuous outcomes. This article aims to present readers with a practical discussion of how to analyze longitudinal data using the LMMs procedure in the SPSS statistical software package.

A Review of Software for Sample Size Determination

Dattalo, P. — Tue, 25 Aug 2009 03:00:12 PDT

The size of a sample is an important element in determining the statistical precision with which population values can be estimated. This article identifies and describes free and commercial programs for sample size determination. Programs are categorized as follows: (a) multiple procedure for sample size determination; (b) single procedure for sample size determination; and (c) Web-based. Programs are described in terms of (a) cost; (b) ease of use, including interface, operating system and hardware requirements, and availability of documentation and technical support; (c) file management, including input and output formats; and (d) analytical and graphical capabilities.

The Role of Education in Improving Physicians' Professional Use of Economic Evaluations of Health Interventions: Some Evidence From a Cross-Sectional Survey in Italy

Tue, 25 Aug 2009 03:00:12 PDT

A cross-sectional survey was carried out on a random sample of Italian physicians through a self-administered questionnaire to describe knowledge, attitudes, and professional behavior toward economic evaluations of health interventions. A response rate of 74.1% was achieved (760 questionnaires). Although many physicians show a positive attitude toward cost-minimization and, to a lesser extent, to cost-effectiveness analysis, they rated their methodological knowledge as unsatisfactory, and the professional use of the economic evaluations of the health interventions in clinical practice is quite low. Multiple logistic regression analysis showed that adequate knowledge and positive attitudes are associated with increased physicians’ use of health economic evaluations, as well as time dedicated to continuing medical education and previous training experience about health economics and management. Education and specific training may play an important role in promoting a more cost-conscious behavior of physicians.

The Relationship of Fennell Phases to Symptoms Among Patients With Chronic Fatigue Syndrome

Reynolds, N. L., Brown, M. M., Jason, L. A. — Tue, 25 Aug 2009 03:00:12 PDT

The Fennell Phase Inventory (FPI) is an instrument designed to measure phases of the illnesses known as Myalgic Encephalomyelitis/Chronic Fatigue Syndrome (ME/CFS). The current study explored how the FPI was related to physical and psychological functioning as well as coping style. Based on FPI scores, 111 adults with ME/CFS were placed in one of three groups: crisis, stabilization, or resolution. Results showed that the crisis group demonstrated significantly worse functioning than at least one other group for depression, quality of life, mental functioning, anxiety, and self-efficacy; and utilized less adaptive coping styles. These results indicate that patients with ME/CFS who are in the crisis phase tend to experience more severe psychological and physical symptoms and utilize poorer coping strategies. Those in the resolution phase maintain the most adaptive coping strategies. Implications for these findings are discussed.

Medical Record Review Conduction Model for Improving Interrater Reliability of Abstracting Medical-Related Information

Engel, L., Henderson, C., Fergenbaum, J., Colantonio, A. — Tue, 25 Aug 2009 03:00:12 PDT

Medical record review (MRR) is often used in clinical research and evaluation, yet there is limited literature regarding best practices in conducting a MRR, and there are few studies reporting interrater reliability (IRR) from MRR data. The aim of this research was twofold: (a) to develop a MRR abstraction tool and standardize the MRR process and (b) to examine the IRR from MRR data. This study introduces the MRR-Conduction Model, which was used to implement a MRR, and examines the IRR between two abstractors who collected preinjury medical and psychiatric, incident-related medical and postinjury head symptom information from the medical records of 47 neurologically injured workers. Results showed that the percentage agreement was ≥85% and the unweighted statistic was ≥.60 for most variables, indicating substantial IRR. An effective and reliable MRR to abstract medical-related information requires planning and time. The MRR-Conduction Model is proposed to guide the process of creating a MRR.

Ethnocultural Empathy Among Students in Health Care Education

Rasoal, C., Jungert, T., Hau, S., Edvardsson Stiwne, E., Andersson, G. — Tue, 25 Aug 2009 03:00:12 PDT

In a multicultural society, ethnocultural empathy has become an important element in most health settings and development of this capacity has become a central component for health care professionals in their interactions with patients and clients. In this study, differences in basic empathy and ethnocultural empathy were explored in a sample of 365 undergraduate students at the beginning and end of four master’s programs in health care (medicine, psychology, nursing, and social work). Results showed that it was mainly psychology students in the first semester who had significantly higher general empathic skills and ethnocultural empathic skills compared to students in the other study programs. Few signs of differences between students in their first and in later semesters were obtained. The observed differences may be explained by (a) levels of admission grades and applications requirements or (b) different cultures and expectations from the surrounding milieus in the investigated study programs.

Bridging the Gap: Knowledge Seeking and Sharing in a Virtual Community of Emergency Practice

Curran, J. A., Murphy, A. L., Abidi, S. S. R., Sinclair, D., McGrath, P. J. — Tue, 25 Aug 2009 03:00:12 PDT

Disparities exist between rural and urban emergency departments with respect to knowledge resources such as online journals and clinical specialists. As knowledge is a critical element in the delivery of quality care, a web-based learning project was proposed to address the knowledge needs of emergency clinicians. One objective of this project was to evaluate the effectiveness of the online environment for knowledge exchange among rural and urban emergency clinicians. Descriptive and content analysis of the online discussion board revealed 202 postings with rural participants contributing the largest number of postings (75%; 152/202). Postings were used to establish a clinical presence (87/202), seek clinical information (52/202), and share clinical information (63/202). Postintervention survey results indicate that this modality introduced participants to new clinical experts and resources. The results provide direction for design of a virtual community of practice, which may reduce current knowledge resource disparities.

The Impact of Federal Bioterrorism Funding Programs on Local Health Department Preparedness Activities

Avery, G. H., Zabriskie-Timmerman, J. — Fri, 15 May 2009 13:52:00 PDT

Using the 2005 National Association of County and City Health Officers Profile of Local Health Departments data set, bivariate probit and Heckman selection models were used to test the hypothesis that the level of federal funding received for bioterrorism preparedness is related to the preparedness activities undertaken by local health departments. Overall budget, leadership, and crisis experience are found to be the most important determinants of local preparedness activity, but Centers for Disease Control and Prevention preparedness funding plays a mediating role by building capacity through the hiring of one key leadership position, the emergency preparedness coordinator. Additional research is needed to determine the potential impact of these funds on other aspects of the local public health system, such as the scope of services delivered, to determine secondary effects of the program.

A Multisource and Repeated Measure Approach to Assessing Patient--Physician Relationship and Patient Satisfaction

Weng, H.-C. — Fri, 15 May 2009 13:52:00 PDT

The object of this study is to compare the multisource assessments of patient— physician relationship with assessments by their patients at two time points. In this observational study, 1,747 outpatients nested under 64 internists and 70 surgeons are surveyed by face-to-face interview at initial patient— physician visits and then in a telephone interview 2 weeks later. On the first evaluation, physicians' self-assessments are not correlated their patients' assessments. At follow-up, physicians' self-assessments correlated with the perceived improvements in patients' health status (p < .05). We also find a positive association (p < .05) between patient satisfaction with their surgeons and perceived improvements of health status at the 2-week follow-up, suggesting that patient satisfaction may be a proxy for symptom or functional improvement. Although most of the ratings of nursing directors, physician peers, administrators, and nonclinical observers are positively associated with the patients' first ratings, the significance of that association disappear by the 2-week follow-up.

Health Care Workers in the Dominican Republic: Self-Perceived Role in Smoking Cessation

Fri, 15 May 2009 13:52:00 PDT

A Dominican Republic (DR)—based multi-community trial of smoking cessation viewed health care workers (HCWs) as potential interventionists. Effectively engaging them requires a clear understanding of their attitudes and practices regarding smoking. A Rapid Assessment Procedure, conducted among HCWs in six economically disadvantaged communities, included physicians, nurses, other health professionals, paraprofessionals, and lay workers. Attitudes and practices about smoking were consistent across the 82 HCWs and mostly reflected community views. HCWs lacked proactiveness related to smoking cessation and had a limited view of their role, attributing clients' quitting successes to personal will. Prior cessation training was limited, although interest was generally high. Material resources about smoking cessation were virtually absent. DR HCWs' views represented features both distinct from and common to HCWs elsewhere. Any intervention with HCWs must first raise awareness before addressing their role in smoking cessation, discussing implementation barriers, and include training and materials about risks and effective interventions.

Construct Validity and Generalizability of Pediatrics Clerkship Evaluation at a Problem-Based Medical School, Bahrain

Al-Mahroos, F. — Fri, 15 May 2009 13:52:00 PDT

The aim of this study is to identify the sources of construct validity evidence and determine generalizability of the scores given to medical students on pediatrics clerkship evaluation at the College of Medicine and Medical Sciences, Arabian Gulf University (CMMS-AGU). The CMMS-AGU is a problem-based medical school in Bahrain. The sample was composed of 140 students. Validity evidence and generalizability theory were sought. Results indicated that pediatrics clerkship evaluation at CMMS-AGU has content validity. Some limitations may have occurred in response process, but the students performance was observed directly by the tutors. There was evidence of range restriction and ``halo'' effect. Interrater reliability was 0.74, and the generalizability coefficient for three raters was 0.61. Students' contribution to the variance component of the global clerkship evaluation scores was 24%. Decision study indicated that seven tutors are required to achieve a generalizability coefficient of about 0.80. In conclusion, this study indicated that the pediatrics clerkship evaluation at the CMMS-AGU has, overall, some acceptable content and internal structure evidence of validity and that the raters are the major source of error. To improve the construct validity, there is a need to increase the number of raters and to train raters about the proper process of clerkship evaluation.

A Multi-Method Process Evaluation for a Skin Cancer Prevention Diffusion Trial

Escoffery, C., Glanz, K., Hall, D., Elliott, T. — Fri, 15 May 2009 13:52:00 PDT

This article describes process evaluation methods for the Pool Cool diffusion trial across 4 years. Pool Cool is a skin cancer prevention program that was found to improve behaviors and environments for sun protection at swimming pools in a randomized efficacy trial, which was followed by a national diffusion trial. The process evaluation focus shifted from measuring program satisfaction to assessing widespread program implementation, barriers and facilitators to implementation, and program maintenance and sustainability. Data collection methods include training surveys, database tracking, field coordinator activity logs, e-mails, surveys of parents, lifeguards and pool managers, and process evaluation interviews and site visits. The data revealed high levels of implementation of major program components when disseminated in the diffusion trial, including sun safety lessons, sun safety signs, and sunscreen use. This article describes program features and participant factors that facilitated local implementation, maintenance and sustainability across dispersed pools such as linkage agents, a packaged program, and adaptations of program elements.

Outcomes of Multimodal Training for Healthcare Professionals at an AIDS Education and Training Center

Cook, P. F., Friedman, R., Lord, A., Bradley-Springer, L. A. — Tue, 17 Feb 2009 05:28:50 PST

There is an ongoing need for continuing professional education (CPE) in the rapidly changing field of HIV care, but the best instructional methods remain a subject of debate. This study assessed the effects of training at an AIDS Education and Training Center (AETC) over an 18-month period. Health care professionals (HCP) who attended more than one training event showed small but significant improvements over time in HIV-related clinical practice behaviors. The type of training also predicted self-reported practice behavior, with interactive trainings and individual consultations associated with greater change, and intensive clinical training activities associated with a faster rate of change but not better scores on the self-reported behavior measure. Participants also reported high levels of satisfaction, knowledge improvement, and intention to change after each training event; however, these results were unrelated to whether trainees actually reported improved practice behavior.

Using Mixed Methods to Evaluate the Pediatric Lead Assessment Network Education Training Program (PLANET)

Polivka, B. J., Chaudry, R. V., Sharrock, T. — Tue, 17 Feb 2009 05:28:50 PST

The Pediatric Lead Assessment Network Education Training Program (PLANET) is a peer-to-peer in-person 1-hr lead poisoning prevention educational program for health professionals. This evaluation was designed to determine the impact of the PLANET program. Evaluation methods included analyzing data from PLANET sign-in sheets, evaluation forms, pre/postknowledge tests, claims data, and focus groups (FGs) and interviews (IVs) with PLANET attendees and nonattendees. Claims data were used to compare blood lead testing rates for physicians attending and those not attending a PLANET program. Over 2,000 health professionals attended the 192 PLANET presentations delivered between June 2001 and December 2006; most were registered nurses or physicians. Written evaluations were overwhelmingly positive. Posttests indicated increased provider knowledge about childhood lead poisoning prevention, and assessment of blood lead testing rates showed higher testing rates for PLANET attendees. FG and IV participants suggesting improvements including using alternative delivery modes.

Tricks of the Trade in Community Mental Health Research: Working With Mental Health Services and Clients

Dadich, A., Muir, K. — Tue, 17 Feb 2009 05:28:50 PST

The complexities of research in the community mental health sector are seldom acknowledged in existing literature; this article attempts to address this void. It presents the methodological challenges experienced in the longitudinal evaluation of the Housing and Accommodation Support Initiative—a program that supports people with chronic mental illness toward long-term recovery. The evaluation provides a case study for understanding methodological problems in community mental health research, which include working with organizations that experience high staff turnover; staff members who have large caseloads; and clients who have chronic mental illness. Although not applicable to all research designs, the suggested strategies highlight the importance of innovation, flexibility, and balance between research theory and practical limitations when conducting community mental health research.

Community Health Program Evaluation Using Accreditation as a Framework

Hahn Severance, J. — Tue, 17 Feb 2009 05:28:50 PST

Increasingly, health system leaders seek to determine whether community health interventions make a difference to individuals in the community. However, community health improvement is difficult to measure, and health system staff may not be familiar with evaluation research methods. Health care organizations can improve their evaluation efforts relatively easily by building on what they already know: the Joint Commission accreditation process. By using accreditation as a framework, community health evaluation may be seen as more approachable when viewed through that lens. This article provides a framework for practical approaches to program planning, evaluation, and sustainability. Joint Commission accreditation functions (chapters) are similar to health program goals. Standards are similar to program objectives. Elements of performance are similar to activities or methods. Scoring comparisons are similar to measures.

Adherence to Practice Guidelines, Clinical Outcomes, and Costs Among Medicaid Enrollees With Severe Mental Illnesses

Stiles, P. G., Boothroyd, R. A., Dhont, K., Beiler, P. F., Green, A. E. — Tue, 17 Feb 2009 05:28:50 PST

The treatment of Medicaid enrollees diagnosed with depression or schizophrenia was examined to determine whether adherence to treatment guideline was associated with health care financing strategy, clinical outcomes, and cost-effectiveness. Individuals in a fee-for-service condition were significantly more likely to receive treatment consistent with guidelines than those in managed care. Mental health costs were higher for individuals diagnosed with schizophrenia, individuals in an acute phase of illness at intake into the study, and those receiving treatment consistent with practice guidelines. Being in an acute phase of illness and having treatment that comported with recommended practice guidelines were associated with higher total social costs. Policy implications of the findings and recommendations for future research are discussed.

The Burroughs Wellcome Career Award in the Biomedical Sciences: Challenges to and Prospects for Estimating the Causal Effects of Career Development Programs

Pion, G. M., Cordray, D. S. — Tue, 18 Nov 2008 22:16:32 PST

The Career Award in Biomedical Sciences (CABS) was launched by the Burroughs Wellcome Fund to assist talented scientists to cross the "bridge" that connects postdoctoral training with independent investigator status. This study examines the causal effects of CABS, using both conventional methods and a newer method (propensity score analysis) in attempting to account for selection biases (e.g., selecting the "best and the brightest") in estimating the award's impact. The results indicated that grantees outperformed unsuccessful applicants in obtaining a faculty position, competing for an R01 grant, and publishing in prestigious journals. However, it was nearly impossible to remove all the effects of selection bias from estimates of program effects. Because selection bias is a pervasive problem in assessing these programs, recommendations for improving evaluations of similar merit-based programs are offered.

Selective Reporting of Adjusted Estimates in Observational Epidemiology Studies: Reasons and Implications for Meta-analyses

Peters, J., Mengersen, K. — Tue, 18 Nov 2008 22:16:32 PST

For meta-analyses of observational epidemiology studies, unadjusted and adjusted study estimates are often extracted. However, there is evidence of selective reporting of adjusted study estimates. We investigate adjustment reporting bias, examining the reasons why some studies do not contribute an adjusted estimate to a meta-analysis. Ten published meta-analyses were re-analysed to assess evidence of adjustment reporting bias and over 100 primary studies were read to investigate why they did not contribute an adjusted estimate to a meta-analysis. Selective reporting of adjusted estimates may lead to a bias in some meta-analyses when adjusted study estimates are not reported because univariate analyses indicated a non-significant effect. We recommend that unadjusted and adjusted study estimates be extracted for a meta-analysis. If adjusted estimates cannot be obtained, the reasons for this should be investigated and sensitivity analyses could be used to assess the impact of this on the meta-analysis.

Mental Health Clinicians' Beliefs About Medicines, Attitudes, and Expectations of Improved Medication Adherence in Patients

Byrne, M. K., Deane, F. P., Caputi, P. — Tue, 18 Nov 2008 22:16:32 PST

Nonadherence to antipsychotic medications remains a major factor in poor clinical outcomes. This study sought to identify clinician beliefs about patients who do not adhere to treatment, the clinicians' own beliefs about medicines, and the impact of beliefs on efforts to enhance patient adherence. In total, 292 clinicians responded to an anonymous questionnaire that included questions about their beliefs and their efforts to enhance adherence. Results indicated that clinicians' beliefs about their own adequacy to enhance adherence significantly predicted actual efforts to enhance adherence. Both pessimism about outcomes and empathy for the patient predicted outcome expectancy. It was concluded that enhancing clinicians' beliefs about working with nonadherent patients is a potentially important ingredient in efforts to improve patient adherence.

Relationship of Candidate Communication and Organization Skills to Oral Certification Examination Scores

Houston, J. E., Smith, E. V. — Tue, 18 Nov 2008 22:16:32 PST

This study investigated the relationship between candidate's proficiency in communication/organization and ability measures derived from judges' oral examination ratings on a medical specialty certification examination. Judges who rated candidates on the oral examination also provided a separate rating for candidates' communication/organization skills. ANOVA was used to examine differences among levels of communication/organizational skills with respect to candidates' ability measures on the oral examination. There was a statistically significant increase in oral examination measures for candidates with higher levels of communication/organization skills. This supports the hypothesis that candidates who were more proficient in organizing and presenting their responses were more likely to have better performance on the oral examination. Given communication and oral examination ratings were provided by the same judges, future research should investigate whether these preliminary findings generalize to situations in which communication ratings are obtained from independent ratings during and outside the oral examination.

Reliability of Surgical Skills Scores in Otolaryngology Residents: Analysis Using Generalizability Theory

Fernandez, S. A., Wiet, G. J., Butler, N. N., Welling, B., Jarjoura, D. — Tue, 18 Nov 2008 22:16:32 PST

Assessments of temporal bone dissection performance among otolaryngology residents have not been adequately developed. At the Ohio State College of Medicine, an instrument (Welling Scale, Version 1 [WS1]) is used to evaluate residents' end-product performance after drilling a temporal bone. In this study, the authors evaluate the components that contribute to measurement error using this scale. Generalizability theory was used to reveal components of measurement error that allow for better understanding of test results. A major component of measurement error came from inconsistency in performance across the two cadaveric test bones each resident was assigned. In contrast, ratings of performance using the WS1 were highly consistent across raters and rating sessions within raters. The largest source of measurement error was caused by residents' inconsistent performance across bones. Rater disagreement introduced only small error into scores. The WS1 provides small measurement error, with two raters and two bones for each participant.