Original Research

Cognitive interview techniques reveal specific behaviors and issues that could affect patient satisfaction relative to hospitalists

Abstract

Abstract

BACKGROUND:

There is a trend toward patient‐centered care as a means of improving patient satisfaction. The Centers for Medicare and Medicaid Services (CMS) have made this concept more significant with plans to link reimbursement to patient satisfaction measures such as the Hospital Consumer Assessment of Healthcare Providers and Systems survey (HCAHPS).

OBJECTIVES:

To generate hypotheses for reasons underlying diminished HCAHPS patient satisfaction survey ratings, with reference to hospitalists.

DESIGN, SETTING, PATIENTS:

Observational study conducted using a cognitive interview (CI) technique in a 180‐bed community hospital on adult medical, surgical, and critical care inpatients.

MEASUREMENTS:

Mixed qualitative and quantitative study using both standard responses and open‐ended responses. The standard responses were compiled into raw numbers and percentages and the qualitative responses were evaluated for common themes and other useful information.

RESULTS:

Notable factors that may affect satisfaction of patients include ability to have all of their questions answered, incomplete discussion of medication side effects, and failure of physicians to listen and form personal connections with them.

CONCLUSION:

Cognitive interview techniques can be used to provide additional detail regarding patient satisfaction beyond that provided by standard surveys. Journal of Hospital Medicine 2009;4:E1–E6. © 2009 Society of Hospital Medicine.

Copyright © 2009 Society of Hospital Medicine

Patient satisfaction is an important issue for hospitals, as it may affect the decision to seek care at one institution over another, but it may soon have direct implications for hospital reimbursement with the recent proposals for Value Based Purchasing (VBP) models by the Centers for Medicare and Medicaid Services (CMS). Up to 5% of inpatient Medicare reimbursement would be linked to performance measures, 40% of which could come from percentile outcomes on the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) alone.1 HCAHPS scores are now available for individual hospitals at the Hospital Compare website maintained by CMS (http://www.hospitalcompare.hhs.gov). Hospitalists will likely be held accountable by administrators for poor performance on the survey.

While the information garnered from the HCAHPS provides an external perception of hospital quality, the questions are broad and do not identify specific reasons for reduced satisfaction. Many have suggested that the incorporation of surveys already administered at hospitals may be required for successful HCAHPS administration in order to overcome the limitations inherent in its design.2

In order to identify explanatory factors for low HCAHPS scores, we decided to incorporate a technique known as a cognitive interview (CI). The CI is widely used as an evaluative tool for survey questions because of its ability to allow the interviewer to discern the processes that lead to responses.3 Up to this point, the focus of this CI method has been on the ability of the subject to comprehend and answer the questions.4 However, when a CI subject is answering a question, there is a large amount of information presented to the interviewer about the topic that is typically regarded as supplementary because of the focus on these specific issues.5 This study reports the supplemental information that may provide insight as to why patients answered as they did. Our goal was to gain further understanding of factors that may underlie HCAHPS responses.

Materials and Methods

Overview

The premise behind every step in the development and implementation of the interview process was to increase the comfort level of the patients with the process as well as take as much of the cognitive burden off of the patients as possible while maintaining the integrity of the questioning.

The study was developed and conducted in May and June 2008. The duration of the study and number of participants were based on time constraints on personnel and limited funding for the study.

Development of the Interview

We used the HCAHPS questions as a starting point to launch the development of our interview. Because the purpose of this study was to generate hypotheses regarding issues affecting patient satisfaction, we used a loose approach to questioning that gave the interview subject a greater degree of freedom with their answers than in a measurement study. This freedom of response eliminated the need to take minor comprehension issues into account, which would have been cause for concern using a standard survey.5

We focused on what the patients thought or felt in order to avoid making them feel overburdened for factual recall. There were some questions for which this was not possible (eg, the questions about medication). After the questions were reworded, their comprehension level was confirmed to be appropriate for the local community (grade 3‐6) using the Flesch Reading Ease and Flesch‐Kincaid grade level equations (data not shown).

The questions were then grouped from least cognitively intense to most cognitively intense surrounding any particular issue, which was generally least specific to most specific. This process of ordering questions has been shown to increase the accuracy of self reports,6 which is advantageous in our situation because it increases the amount of potentially useful information the patients include in their perception analysis. Every major section change started with at least 1 general question without specific aim in order to facilitate free response (full interview is included as Appendix 1).

For classic quantitative analysis, the 4 HCAHPS categories (always, usually, sometimes, never) were presented for most of the questions. They are reported in this work as Top Box, (ie, always), or Not Top Box (any other response). Top Box responses are the ratings most widely reported in HCAHPS surveys, and this dichotomy makes it easier to code and review answers. However, when we felt that quantitatively we wanted a more superficial inquiry, a 2‐category yes/no system was used. The interviews were also designed to be less than 1 hour in length, which has been shown to be appropriate in other kinds of cognitive studies in the medical field.7

The questions in the About You section of the HCAHPS were largely excluded with the exception of the education demographic question (for a full HCAHPS survey see Goldstein et al.).2 The community that we studied is quite homogeneously Caucasian and non‐Hispanic. Education level is the only remaining HCAHPS demographic variable, and our subjects had education levels comparable to those reported in HCAHPS for this hospital. Our data are primarily qualitative and were not weighted as the HCAHPS data were.8

Conducting the Interviews

The interviews were conducted by the principal researcher (A.B.) (the use of a single interviewer eliminates potential bias between interviewers on interview delivery) on medical and surgical inpatients in general inpatient and intensive care units who were at least 18 years old, not suffering significant discomfort, and able to comprehend questions and provide meaningful answers. The nurse managers on the inpatient units were asked to identify patients who fit these criteria. Of the 50 patients typically present on medical/surgical services, about 10 would be suitable for interview. The researcher would then select up to 5 patients each weekday from those actually present in their rooms, explain that he was doing research for the hospital, and ask if the patient would participate. The range of the number of selected patients depended upon the willingness of the available patients to participate and the amount of time it took to conduct the interviews. If the patient accepted, the researcher then emphasized that participation was completely optional and would have no effect on the medical care the patient would receive, that the identity of the patient would never be shared, and that the information provided would be used internally and possibly for anonymous external reporting. The researcher then asked if the patient would still like to continue. This protocol was approved by the hospital Institutional Review Board.

The patients were then instructed to think aloud as they answered questions and given Think Aloud Training using established methods.9 There were 4 instructions or questions in the training that were used sequentially, as follows: (1) Try to picture the place where you live, and think about how many windows there are in that place. As you count up the windows, tell me what you are seeing and thinking about;9 (2) I want you to think about the last school you personally went to and studied at. Think about yourself walking into the main building. Tell me what you are seeing and thinking about as you walk through the doors; (3) Now I want you to think about your favorite food. Can you tell me about it? and (4) I want you to think about a pot of flowers. Tell me about what you think and see as you walk up and smell the flowers. While the patients were thinking, the interviewer followed up with spontaneous general verbal probes (eg, Can you tell me what you are thinking?). After the patients were responding adequately to the training instructions, we proceeded to the interview.

The interviews were conducted by asking the patients the designed questions using a Think Aloud CI technique and spontaneous verbal probing in the instance of a seeming lack of information, inconsistency in information offered by the patient, indicative body language (eg, seeming uncomfortable), hesitation indicating the patient was not rendering all of the information that he or she was thinking, or if a common specific issue had been previously identified. Because we were looking for issues adversely affecting patient satisfaction, we decided to optimize our effort by focusing on questions to which patients responded negatively or hesitantly. However, as time permitted we also gave attention to questions patients answered immediately and positively.

Data Collection

Data were collected on interview sheets containing the questions asked to the patients with the Top Box/Not Top Box coding method described above, as well as a section for qualitative data gleaned from the cognitive elements of the interview. Recordings and full transcriptions were not performed to avoid raising patient concerns, which held the potential to negatively impact participation. All records of participant names were kept solely by the principal investigator to avoid duplicate interviews and were destroyed at the conclusion of the interview process.

Analysis

The data was collected from the interview sheets and then compiled into a spreadsheet. The classic quantitative data was compiled into raw totals and percentages. The qualitative data from the cognitive portion was then considered both separately, based on the question, and as a group. Both authors reviewed the data and identified significant items in both the qualitative and quantitative data, based on their ability to provide useful hypotheses regarding higher or lower levels of satisfaction. Also, because of the nature of the CI, useful information can come from as few as 1 respondent,9 so potentially useful individual comments were pulled out of the qualitative data as well.

It is important to note that this study is not a means of measuring patient satisfaction, but rather a means of determining elements and specific issues affecting patient satisfaction. Answers in this study cannot be considered equivalent to answers on the HCAHPS or any other patient satisfaction survey.

Results

Response Rate

There were 57 eligible patients who were asked to participate in this study. Availability of the patients identified as eligible limited the number of participants. Of the 57, 50 accepted. Of the 50 who accepted, 1 was rejected because the patient had only been at the hospital for a few hours and had not seen a physician yet, and 5 more opted to stop before the interview was completed. The 1 that was rejected was not included in the analysis, but the 5 that were left incomplete were included for the questions that had been answered.

Responsiveness to Questions

Patients were asked Did you get all of your questions answered? (Table 1). The responsiveness to questions by our staff was largely satisfactory, but the qualitative data yielded an interesting finding in that 3 of the 6 patients who rendered Not Top Box answers cited problems with question delivery. One patient said, The doctor was just in and out, and I didn't have time to ask questions. Another said, I think of questions 20 minutes after the doctor leaves. In other parts of the interview, 2 more patients mentioned not having questions ready when their attending physician was there, and then not being able to ask them, 1 of them saying, It was my own fault (when I did not get information I wanted) because I didn't ask all of the questions.

Standardized Responses to Questions
Questions Asked of PatientsResponsesNumberQualitative Subtheme
  • NOTE: These are the responses to the questions based on the preestablished coding categories and with qualitative subthemes. There are also examples given with each of the qualitative subthemes. There was no qualitative subtheme or example given for the question Did anyone tell you what [the new medication] was before they gave it to you? because none could be gathered from the data.

Did you get all of your questions answered?Always41Difficulty with question delivery: (eg, I think of questions 20 minutes after the doctor leaves.)
Never, sometimes, usually6
Total47
Did anyone tell you what [the new medicine] was before they gave it to you?Yes29
No2
Total31
Did anyone tell you about any side effects of the medicine?Yes17Lack of communication regarding medicines with few major side effects: (eg, Tylenol or Advil)
No13
Total31
Do you feel the doctors spend enough time with you?Yes37For example: (My doctor) didn't consult me and he didn't inform me.
No8
Total45
Do you feel the doctors know you as a person?Yes29Physician behaviors may influence: (eg, He treats me as a person.)
No16
Total45
Do you feel the doctors treat you with respect?Always40Physician behaviors may influence: (eg, I feel talked down too, like I can't handle the answers.)
Never, sometimes, usually6
Total46
Do you think the doctors listen carefully to you?Always33For example: He had so much on his mind.
Never, sometimes, usually13
Total46

Communication Regarding Medication

There are 2 important questions regarding this issue. Patients who reported taking new medication during their current hospital stay were asked, Did anyone tell you what [the new medication] was for before they gave it to you? and then, Did anyone tell you about any side effects of the medicine? (Table 1). From this we can see that, generally, patients are informed about new medications, but there seems to be a lack of informing the patients regarding side effects of new medications. Interestingly, 8 of the patients who reported that they were not told about side effects raised concerns about medications that have few major side effects. It is important to note that this issue was not initially known and the kind of medication the patients were given was seldom freely offered. We began specifically probing for it after a patient offered the information without prompting, so the actual number of these instances could be much higher.

Perception of Time Spent

We asked our subjects Do you feel the doctors spend enough time with you? (Table 1). The sensitivity of the quantitative data was decreased by using a 2‐category approach, but frequency estimation about an issue of time seemed an unnecessarily awkward approach considering that the more valuable information comes from the qualitative section of the interview.

The perception of enough time spent by physicians was not felt by all of the subjects. Three of the patients who answered No had mentioned a specific need that was not met, all 3 of which were issues with information and communication. The hospitalist changed all of my medicationshe didn't consult me, and he didn't inform me, said 1 patient. Another said, I don't really know who my doctor is, in reference to his attending hospitalist. There was another patient who answered No who felt that the physicians were just mechanically trying to treat his condition and discharge him as soon as possible. I think the hospital is getting the doctors to push people outthey need the beds. Also, 2 patients who answered Yes acknowledged that the doctors spent all the time they needed to for the patients' situations, and 1 patient who answered Yes mentioned a specific personal accommodation the physician made, specifically that the physician prays with the patient.

Personal Connections

This question was, Do you feel the doctors know you as a person? and was included to attempt to probe into the depth of the personal connections formed by our physicians. The standard quantitative data (Table 1) shows there were a significant number of patients who did not feel their physicians knew them as a person. Six of the patients who answered No said they felt the physicians were just doing their job. What else are you besides a body and a diagnosis? 1 patient asked. They are doing their job, said another. Six patients who answered Yes cited how their physicians acted toward them as the reason. One patient responded, One doctor carries on extra conversation with me. He knows me as a person. Another patient plainly said, He treats me as a person. Yet another said, They seem to respect my situation and my need for their services.

Nonclinical Competency

Two questions elicited significant information relevant to hospitalists on this issue, the first being Do you feel the doctors treat you with respect? and the second being Do you think the doctors listen carefully to you? (Table 1).

The standard quantitative data for the first question shows that, generally, our patients felt as though the doctors were treating them with respect. Interestingly, the qualitative portion of the study shows that all 6 patients who rendered Not Top Box answers mentioned either being talked down to or not being responded to by their physicians. The hospitalist treated me as I expected, as [a generic 50‐year‐old], remarked 1 patient about her hospitalist, by whom she felt ignored. I feel talked down too, like I can't handle the answers. I feel like I don't always get the full truth, said another patient regarding her physician telling her about her medical condition. It seems like the doctor put me off, said a third patient regarding his physician's willingness to treat him.

The quantitative data for the second question shows much weaker performance with regard to the perception of the doctors listening to the patients. The qualitative portion of the study shows that 3 patients mentioned how much their physicians had on their minds. One patient said, He was thinking so many things at the same time, and another plainly said, He had so much on his mind. Four patients mentioned not feeling responded to by their physicians. A patient commented, I would say something (to the doctor), and then my dad would [sic] re‐say it, and the doctor would respond when my dad said it, but not when I did. Another patient responded, I would ask (the doctors) to do something, and they wouldn't do it.

Discussion

Because of the limitations inherent in typical survey measures of investigation, we employed a CI technique. The objective was to generate hypotheses as to why patients may report reduced satisfaction with various aspects of hospitalization. The data set gleaned from this study was sizable, and held information pertinent to all parts of the hospital. The results reported in this article are focused on hospitalists.

The quantity of time hospitalists spend with patients may not be as important to the patients as the quality of interaction. Other studies have shown similar results, illustrating that the key factor in a patients' opinion of a visit with a physician was the perception of being taken seriously.10 The perception of being talked down to (as noted by Levy11) and the perception of not being responded to or focused on are the key negative factors for patients. By interacting with patients in a way that makes them feel valued, focused on, and responded to, physicians may improve patient satisfaction without requiring increased amounts of time spent with them.

Patient complaints that physicians do not respond to them may partially reflect inability to ask all of their questions. Hospitals may wish to consider methods to improve the ability of the patients to ask questions in the small amount of time available for physicians to talk with them (eg, placing notebooks by patients' bedsides and repeatedly encouraging them to write their questions down). This concept has also been endorsed as a means of improving the quality of care and reducing medical mistakes.12

Hospitalists and other physicians may positively impact the communication about medication issues by establishing a protocol for communicating major as well as minor side effects or even the lack of side effects of medication to patients. The challenge lies in determining the level of incidence that is significant enough (eg, 10%, 5%, or 1%) to warrant explaining minor side effects.

The findings of this study are limited by its small size, qualitative nature, nonrepresentative population, and loose approach to questioning. The patient responses were transcribed manually by the interviewer, leaving the possibility that some responses were not accurately recorded. Our survey instrument is not validated, and the results may not generalize to other settings. Instead of taking these findings as conclusive, we present them as suggestions for improvement that may be validated by further research.

Acknowledgements

The authors Tom Quinn and Toni Murdough for their contributions in developing the survey instrument.

Appendix

Appendix 1. Interview questions in the order they were presented to the patients

Background Questions

  • What is your name?

  • About how many hospital visits have you had in the last 6 months?

  • What is the highest grade or level of school that you have completed?

General Hospital Questions

  • What are the most important things you want in a hospital?

  • What do you think about the reputation of the hospital?

  • How has your stay here at the hospital been?

  • While you've been here, have you had any pain? Were you given any pain medicine? How well was your pain controlled? (The first two questions were for screening, the last question was for coding)

  • During this hospital stay, were you given any medicine that you had not taken before? Did anyone tell you what it was for before they gave it to you? Did anyone tell you about any side effects of the medicine? (First question for screening, last two for coding)

  • Do you think your room and bathroom are clean? Has it been that way the whole time you've been here?

  • Do you think this hospital has all the equipment needed for your treatment?

  • Do you feel you've been able to rest while you've been here?

  • Have you ever pressed the call button? How fast did the help come? Did it come as soon as you wanted it? (The first question was for screening, the second for insight, the third for coding)

  • While you were here at the hospital, did you ever need help getting to the bathroom or using a bedpan? How fast did you get help? Did help come as soon as you wanted it? (The first question was for screening, the second for insight, the third for coding)

  • Do you feel the staff paid attention to you?

  • Did you get all of your questions answered?

  • Have you had any problems here at the hospital? Did anybody help you with it? (The first question was for screening, the second for coding)

Nursing Staff Questions

  • What do you think about your nurses?

  • Do you feel the nurses treat you with respect?

  • Do you think the nurses listen carefully to you?

  • Did the nurses explain things in a way you could understand?

  • Did you get all the information you wanted from the nurses about your tests or treatments?

  • Do the nurses make you feel calm and safe while you are here at the hospital?

  • Do you feel the nurses know you as a person?

  • Do the nurses seem interested in doing things for you?

  • Do you feel the nurses spend enough time with you?

  • Have you ever had a change of nurses while you were here? Do you feel like the new nurse knew what the other nurse knew about you? Did the new nurse seem to know your situation? (The first question is for screening, the next two for coding)

  • What do you feel about the skill of your nurses here at the hospital?

Physician Staff Questions

  • What do you think about your doctors?

  • Do you feel the doctors treat you with respect?

  • Do you think the doctors listen carefully to you?

  • Did the doctors explain things in a way you could understand?

  • Did you get all the information you wanted from the doctors about your tests or treatments?

  • Do the doctors make you feel calm and safe while you are here at the hospital?

  • Do you feel the doctors know you as a person?

  • Did the doctors seem interested in doing things for you?

  • Do you feel the doctors spend enough time with you?

  • Have you ever been switched between doctors here at the hospital? Do you feel like the new doctor knew what the other doctor knew about you? Did the new doctor seem to know your situation? (The first question was for screening, the next two for coding)

  • Were you ever switched from your primary doctor to a doctor here at the hospital? Do you feel like the new doctor knew what your primary doctor knew about your medical situation? (The first question was for screening, the second question for coding)

  • What do you feel about the skill of your doctors here at the hospital?

Cumulative Hospital Questions

  • Did you see any hospital staff disagree with each other about your medical treatment?

  • On a scale of 1‐10, how would you rate the hospital?

  • Is there anything that we haven't talked about that would keep you from rating us a 10?

  • Would you recommend this hospital to your friends and family?

  • What can we do to make this the best hospital around?

   Comments ()