Original Research

Do HCAHPS doctor communication scores reflect the communication skills of the attending on record? A cautionary tale from a tertiary-care medical service

Abstract

BACKGROUND

Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) scores measure patient satisfaction with hospital care. It is not known if these reflect the communication skills of the attending physician on record. The Four Habits Coding Scheme (4HCS) is a validated instrument that measures bedside physician communication skills according to 4 habits, namely: investing in the beginning, eliciting the patient’s perspective, demonstrating empathy, and investing in the end.

OBJECTIVE

To investigate whether the 4HCS correlates with provider HCAHPS scores.

METHODS

Using a cross-sectional design, consenting hospitalist physicians (n = 28), were observed on inpatient rounds during 3 separate encounters. We compared hospitalists’ 4HCS scores with their doctor communication HCAHPS scores to assess the degree to which these correlated with inpatient physician communication skills. We performed sensitivity analysis excluding scores returned by patients cared for by more than 1 hospitalist.

RESULTS

A total of 1003 HCAHPS survey responses were available. Pearson correlation between 4HCS and doctor communication scores was not significant, at 0.098 (-0.285, 0.455; P = 0.619). Also, no significant correlations were found between each habit and HCAHPS. When including only scores attributable to 1 hospitalist, Pearson correlation between the empathy habit and the HCAHPS respect score was 0.515 (0.176, 0.745; P = 0.005). Between empathy and overall doctor communication, it was 0.442 (0.082, 0.7; P = 0.019).

CONCLUSION

Attending-of-record HCAHPS scores do not correlate with 4HCS. After excluding patients cared for by more than 1 hospitalist, demonstrating empathy did correlate with the doctor communication and respect HCAHPS scores. Journal of Hospital Medicine 2017;12:421-427. © 2017 Society of Hospital Medicine

© 2017 Society of Hospital Medicine

Communication is the foundation of medical care.1 Effective communication can improve health outcomes, safety, adherence, satisfaction, trust, and enable genuine informed consent and decision-making.2-9 Furthermore, high-quality communication increases provider engagement and workplace satisfaction, while reducing stress and malpractice risk.10-15

Direct measurement of communication in the healthcare setting can be challenging. The “Four Habits Model,” which is derived from a synthesis of empiric studies8,16-20 and theoretical models21-24 of communication, offers 1 framework for assessing healthcare communication. The conceptual model underlying the 4 habits has been validated in studies of physician and patient satisfaction.1,4,25-27 The 4 habits are: investing in the beginning, eliciting the patient’s perspective, demonstrating empathy, and investing in the end. Each habit is divided into several identifiable tasks or skill sets, which can be reliably measured using validated tools and checklists.28 One such instrument, the Four Habits Coding Scheme (4HCS), has been evaluated against other tools and demonstrated overall satisfactory inter-rater reliability and validity.29,30

The Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey, developed under the direction of the Centers for Medicare and Medicaid Services (CMS) and the Agency for Healthcare Research and Quality, is an established national standard for measuring patient perceptions of care. HCAHPS retrospectively measures global perceptions of communication, support and empathy from physicians and staff, processes of care, and the overall patient experience. HCAHPS scores were first collected nationally in 2006 and have been publicly reported since 2008.31 With the introduction of value-based purchasing in 2012, health system revenues are now tied to HCAHPS survey performance.32 As a result, hospitals are financially motivated to improve HCAHPS scores but lack evidence-based methods for doing so. Some healthcare organizations have invested in communication training programs based on the available literature and best practices.2,33-35 However, it is not known how, if at all, HCAHPS scores relate to physicians’ real-time observed communication skills.

To examine the relationship between physician communication, as reported by global HCAHPS scores, and the quality of physician communication skills in specific encounters, we observed hospitalist physicians during inpatient bedside rounds and measured their communication skills using the 4HCS.

METHODS

Study Design

The study utilized a cross sectional design; physicians who consented were observed on rounds during 3 separate encounters, and we compared hospitalists’ 4HCS scores to their HCAHPS scores to assess the correlation. The study was approved by the Institutional Review Board of the Cleveland Clinic.

Population

The study was conducted at the main campus of the Cleveland Clinic. All physicians specializing in hospital medicine who had received 10 or more completed HCAHPS survey responses while rounding on a medicine service in the past year were invited to participate in the study. Participation was voluntary; night hospitalists were excluded. A research nurse was trained in the Four Habits Model28 and in the use of the 4HCS coding scheme by the principal investigator. The nurse observed each physician and ascertained the presence of communication behaviors using the 4HCS tool. Physicians were observed between August 2013 and August 2014. Multiple observations per physician could occur on the same day, but only 1 observation per patient was used for analysis. Observations consisted of a physician’s first encounter with a hospitalized patient, with the patient’s consent. Observations were conducted during encounters with English-speaking and cognitively intact patients only. Resident physicians were permitted to stay and conduct rounds per their normal routine. Patient information was not collected as part of the study.

Measures

HCAHPS. For each physician, we extracted all HCAHPS scores that were collected from our hospital’s Press Ganey database. The HCAHPS survey contains 22 core questions divided into 7 themes or domains, 1 of which is doctor communication. The survey uses frequency-based questions with possible answers fixed on a 4-point scale (4=always, 3=usually, 2=sometimes, 1=never). Our primary outcome was the doctor communication domain, which comprises 3 questions: 1) During this hospital stay, how often did the doctors treat you with respect? 2) During this hospital stay, how often did the doctors listen to you? and 3) During this hospital stay, how often did the doctors explain things in a language you can understand? Because CMS counts only the percentage of responses that are graded “always,” so-called “top box” scoring, we used the same measure.

The HCAHPS scores are always attributed to the physician at the time of discharge even if he may not have been responsible for the care of the patient during the entire hospital course. To mitigate contamination from patients seen by multiple providers, we cross-matched length of stay (LOS) data with billing data to determine the proportion of days a patient was seen by a single provider during the entire length of stay. We stratified patients seen by the attending providers to less than 50%, 50% to less than 100%, and at 100% of the LOS. However, we were unable to identify which patients were seen by other consultants or by residents due to limitations in data gathering and the nature of the database.

The Four Habits Model.

Figure 1

The Four Habits. The Four Habits are: invest in the beginning, elicit the patient’s perspective, demonstrate empathy, and invest in the end (Figure 1). Specific behaviors for Habits 1 to 4 are outlined in the Appendix, but we will briefly describe the themes as follows. Habit 1, invest in the beginning, describes the ability of the physician to set a welcoming environment for the patient, establish rapport, and collaborate on an agenda for the visit. Habit 2, elicit the patient’s perspective, describes the ability of the physician to explore the patients’ worries, ideas, expectations, and the impact of the illness on their lifestyle. Habit 3, demonstrate empathy, describes the physician’s openness to the patient’s emotions as well as the ability to explore, validate; express curiosity, and openly accept these feelings. Habit 4, invest in the end, is a measure of the physician’s ability to counsel patients in a language built around their original concerns or worries, as well as the ability to check the patients’ understanding of the plan.2,29-30

4HCS. The 4HCS tool (Appendix) measures discreet behaviors and phrases based on each of the Four Habits (Figure 1). With a scoring range from a low of 4 to a high of 20, the rater at bedside assigns a range of points on a scale of 1 to 5 for each habit. It is an instrument based on a teaching model used widely throughout Kaiser Permanente to improve clinicians’ communication skills. The 4HCS was first tested for interrater reliability and validity against the Roter Interaction Analysis System using 100 videotaped primary care physician encounters.29 It was further evaluated in a randomized control trial. Videotapes from 497 hospital encounters involving 71 doctors from a variety of clinical specialties were rated by 4 trained raters using the coding scheme. The total score Pearson’s R and intraclass correlation coefficient (ICC) exceeded 0.70 for all pairs of raters, and the interrater reliability was satisfactory for the 4HCS as applied to heterogeneous material.30

STATISTICAL ANALYSIS

Physician characteristics were summarized with standard descriptive statistics. Pearson correlation coefficients were computed between HCAHPS and 4HCS scores. All analyses were performed with RStudio (Boston, MA). The Pearson correlation between the averaged HCAHPS and 4HCS scores was also computed. A correlation with a P value less than 0.05 was considered statistically significant. With 28 physicians, the study had a power of 88% to detect a moderate correlation (greater than 0.50) with a 2-sided alpha of 0.05. We also computed the correlations based on the subgroups of data with patients seen by providers for less than 50%, 50% to less than 100%, and 100% of LOS. All analyses were conducted in SAS 9.2 (SAS Institute Inc., Cary, NC).36

RESULTS

There were 31 physicians who met our inclusion criteria. Of 29 volunteers, 28 were observed during 3 separate inpatient encounters and made up the final sample. A total of 1003 HCAHPS survey responses were available for these physicians. Participants were predominantly female (60.7%), with an average age of 39 years. They were in practice for an average of 4 years (12 were in practice more than 5 years), and 9 were observed on a teaching rotation.

Overall 4HCS Score Distribution

Table 1

The means of the overall 4HCS scores per observation were 17.39 ± 2.33 for the first, 17.00 ± 2.37 for the second, and 17.43 ± 2.36 for third bedside observation. The mean 4HCS scores per observation, broken down by habit, appear in Table 1. The ICC among the repeated scores within the same physician was 0.81. The median number of HCAHPS survey returns was 32 (range = [8, 85], with mean = 35.8, interquartile range = [16, 54]). The median overall HCAHPS doctor communication score was 89.6 (range = 80.9-93.7). Participants scored the highest in the respect subdomain and the lowest in the explain subdomain. Median HCAHPS scores and ranges appear in Table 2.

Overall HCAHPS Score Distribution

Table 2


Because there were no significant associations between 4HCS scores or HCAHPS scores and physician age, sex, years in practice, or teaching site, correlations were not adjusted. Figure 2A and 2B show the association between mean 4HCS scores and HCAHPS scores by physician. There was no significant correlation between overall 4HCS and HCAHPS doctor communication scores (Pearson correlation coefficient 0.098; 95% confidence interval [CI], -0.285, 0.455). The individual habits also were not correlated with overall HCAHPS scores or with their corresponding HCAHPS domain (Table 3).
4HCS vs. HCAHPS: Pearson Correlations, CI, and P Values for Each Strata of Hospitalist Involvement. All returns; <50%, 50%-<100%, and 100% LOS

Table 3

For 325 patients, 1 hospitalist was present for the entire LOS. In sensitivity analysis limiting observations to these patients (Figure 2C, Figure 2D, Table 3), we found a moderate correlation between habit 3 and the HCAHPS respect score (Pearson correlation coefficient 0.515; 95% CI, 0.176, 0.745; P = 0.005), and a weaker correlation between habit 3 and the HCAHPS overall doctor communication score (0.442; 95% CI, 0.082, 0.7; P = 0.019). There were no other significant correlations between specific habits and HCAHPS scores.

Figure 2

Figure 2

DISCUSSION

In this observational study of hospitalist physicians at a large tertiary care center, we found that communication skills, as measured by the 4HCS, varied substantially among physicians but were highly correlated within patients of the same physician. However, there was virtually no correlation between the attending physician of record’s 4HCS scores and their HCAHPS communication scores. When we limited our analysis to patients who saw only 1 hospitalist throughout their stay, there were moderate correlations between demonstration of empathy and both the HCAHPS respect score and overall doctor communication score. There were no trends across the strata of hospitalist involvement. It is important to note that the addition of even 1 different hospitalist to the LOS removes any association. Habits 1 and 2 are close to significance in the 100% subgroup, with a weak correlation. Interestingly, Habit 4, which focuses on creating a plan with the patient, showed no correlation at all with patients reporting that doctors explained things in language they could understand.

Development and testing of the HCAHPS survey began in 2002, commissioned by CMS and the Agency for Healthcare Research and Quality for the purpose of measuring patient experience in the hospital. The HCAHPS survey was endorsed by the National Quality Forum in 2005, with final approval of the national implementation granted by the Office of Management and Budget later that year. The CMS began implementation of the HCAHPS survey in 2006, with the first required public reporting of all hospitals taking place in March 2008.37-41 Based on CMS’ value-based purchasing initiative, hospitals with low HCAHPS scores have faced substantial penalties since 2012. Under these circumstances, it is important that the HCAHPS measures what it purports to measure. Because HCAHPS was designed to compare hospitals, testing was limited to assessment of internal reliability, hospital-level reliability, and construct validity. External validation with known measures of physician communication was not performed.41 Our study appears to be the first to compare HCAHPS scores to directly observed measures of physician communication skills. The lack of association between the 2 should sound a cautionary note to hospitals who seek to tie individual compensation to HCAHPS scores to improve them. In particular, the survey asks for a rating for all the patient’s doctors, not just the primary hospitalist. We found that, for hospital stays with just 1 hospitalist, the HCAHPS score reflected observed expression of empathy, although the correlation was only moderate, and HCAHPS were not correlated with other communication skills. Of all communication skills, empathy may be most important. Almost the entire body of research on physician communication cites empathy as a central skill. Empathy improves patient outcomes1-9,13-14,16-18,42 such as adherence to treatment, loyalty, and perception of care; and provider outcomes10-12,15 such as reduced burnout and a decreased likelihood of malpractice litigation.

It is less clear why other communication skills did not correlate with HCAHPS, but several differences in the measures themselves and how they were obtained might be responsible. It is possible that HCAHPS measures something broader than physician communication. In addition, the 4HCS was developed and normed on outpatient encounters as is true for virtually all doctor-patient coding schemes.43 Little is known about inpatient communication best practices. The timing of HCAHPS may also degrade the relationship between observed and reported communication. The HCAHPS questionnaires, collected after discharge, are retrospective reconstructions that are subject to recall bias and recency effects.44,45 In contrast, our observations took place in real time and were specific to the face-to-face interactions that take place when physicians engage patients at the bedside. Third, the response rate for HCAHPS surveys is only 30%, leading to potential sample bias.46 Respondents represent discharged patients who are willing and able to answer surveys, and may not be representative of all hospitalized patients. Finally, as with all global questions, the meaning any individual patient assigns to terms like “respect” may vary.

Our study has several limitations. The HCAHPS and 4HCS scores were not obtained from the same sample of patients. It is possible that the patients who were observed were not representative of the patients who completed the HCAHPS surveys. In addition, the only type of encounter observed was the initial visit between the hospitalist and the patient, and did not include communication during follow-up visits or on the day of discharge. However, there was a strong ICC among the 4HCS scores, implying that the 4HCS measures an inherent physician skill, which should be consistent across patients and encounters. Coding bias of the habits by a single observer could not be excluded. High intra-class correlation could be due in part to observer preferences for particular communication styles. Our sample included only 28 physicians. Although our study was powered to rule out a moderate correlation between 4HCS scores and HCAHPS scores (Pearson correlation coefficient greater than 0.5), we cannot exclude weaker correlations. Most correlations that we observed were so small that they would not be clinically meaningful, even in a much larger sample.

CONCLUSIONS

Our findings that HCAHPS scores did not correlate with the communication skills of the attending of record have some important implications. In an environment of value-based purchasing, most hospital systems are interested in identifying modifiable provider behaviors that optimize efficiency and payment structures. This study shows that directly measured communication skills do not correlate with HCAHPS scores as generally reported, indicating that HCAHPS may be measuring a broader domain than only physician communication skills. Better attribution based on the proportion of care provided by an individual physician could make the scores more useful for individual comparisons, but most institutions do not report their data in this way. Given this limitation, hospitals should refrain from comparing and incentivizing individual physicians based on their HCAHPS scores, because this measure was not designed for this purpose and does not appear to reflect an individual’s skills. This is important in the current environment in which hospitals face substantial penalties for underperformance but lack specific tools to improve their scores. Furthermore, there is concern that this type of measurement creates perverse incentives that may adversely alter clinical practice with the aim of improving scores.46

Training clinicians in communication and teaming skills is one potential means of increasing overall scores.15 Improving doctor-patient and team relationships is also the right thing to do. It is increasingly being demanded by patients and has always been a deep source of satisfaction for physicians.15,47 Moreover, there is an increasingly robust literature that relates face-to-face communication to biomedical and psychosocial outcomes of care.48 Identifying individual physicians who need help with communication skills is a worthwhile goal. Unfortunately, the HCAHPS survey does not appear to be the appropriate tool for this purpose.

Disclosure

The Cleveland Clinic Foundation, Division of Clinical Research, Research Programs Committees provided funding support. No funding source had any role in the study design; in the collection, analysis, and interpretation of data; in the writing of the report; or in the decision to submit the article for publication. The authors have no conflicts of interest for this study.

References

Online-Only Materials

AttachmentSize
File Velez_Appendix_1_and_2.docx317.37 KB
   Comments ()

Related Articles from PubMed

J Hosp Med. 2013 Jun;8(6):315-20. doi: 10.1002/jhm.2041. Epub 2013 Apr 3.
J Gen Intern Med. 2016 Jul;31(7):755-61. doi: 10.1007/s11606-016-3597-2. Epub 2016 Feb 26.