Brief Reports

The carewell in hospital questionnaire: A measure of frail elderly inpatient experiences with individualized and integrated hospital care


Given our aging society with an increasing number of frail elderly patients, we must provide integrated care tailored to their complex needs regarding health and well‐being. The aim of this study was to develop and validate a questionnaire designed to assess how frail hospitalized elderly patients experience several important aspects of individualized and integrated care. An 8‐item questionnaire was developed using input from a panel representing the target group and administered to patients age ≥70 years from surgical, medical, and geriatric departments to measure data characteristics, internal consistency, test‐retest reliability, construct validity, and responsiveness. A total of 470 questionnaires were returned, including 78 for test‐retest reliability. Data were missing from 1.7% to 7.0% within the individual questions. The percentage of questions answered with “don't know” ranged 3.8% to 21.9%. Cronbach's α for internal consistency was 0.70. Test‐retest intraclass correlation was 0.75. Achievement of goals during the hospital stay was significantly correlated with the questionnaire score. Scores did not differ significantly between departments or between the before and after measurements related to an innovative intervention study in healthcare delivery. The CareWell in Hospital questionnaire has good content validity, internal consistency, and test‐retest reliability and warrants further research to explore responsiveness. Journal of Hospital Medicine 2014;9:324–329. © 2014 Society of Hospital Medicine

© 2014 Society of Hospital Medicine

Patient‐reported quality of care is currently an important outcome measure. Ideally, quality of care is assessed by measuring patient's experiences rather than patient satisfaction, as most patients are satisfied with the care they receive, even if the quality is poor.[1] Within the study of the CareWell in Hospital (CWH) program[2]which aims to improve quality of care for frail inpatients age 70 yearswe aimed to assess experiences using a questionnaire to determine the quality of hospital care from the perspective of elderly inpatients. This questionnaire should specifically address whether individualized, integrated care is delivered, with an emphasis on autonomy and maintaining patient independence as well as integrating well‐being into hospital care, all of which are aims of the CWH program. In this, it follows the perspective of integrated care as enabling the achievement of common goals and optimal care results from the patients' view: Care should be sensitive to the characteristics and needs of individual patients.[3]

In the Netherlands, a patient questionnaire to measure experiences with hospital care was carefully developed (partially based on the Consumer Assessment of Healthcare Providers and Systems) and is used to obtain information for national benchmarking: the Consumer Quality Index (CQI).[4] However, we considered this questionnaire containing 78 core questions as well as the time between discharge and measurement (often several months) too long for frail elderly patients, as they have complex, multidisciplinary needs and may have difficulty communicating their needs and reporting their experienced quality of care.

Here, we report on the development and validation of a questionnaire that is based on the CQI and can be used to measure the quality of individualized and integrated hospital care as experienced by inpatients age 70 years.



The predefined criteria for the questionnaire were that it should be brief, thereby reducing the burden placed on frail elderly persons; cover the aims of CWH; and measure experiences rather than satisfaction.

Ten categories were initially formulated to match CWH's goals of autonomy, independence, well‐being, individualized care, communication, coordination of care, continuity of care, patient safety, and competence of physicians and nurses. Items from the CQI questionnaire database[5] were selected for each category. Ten members of a panel representing the elderly target group were invited to select the 3 most important questions in each category (first Delphi round). This panel is an important party within a regional network of care and well‐being organizations and involved in discussing the various regional care and/or well‐being projects when it concerns their content and value for elderly persons. They represent elderly persons through their position in elderly‐care or informal care organizations or from personal experiences. During a second Delphi round, they determined whether the individual items of the concept questionnaire were clearly stated, comprehensible to frail elderly patients, represent quality of care, have appropriate answer categories, and so forth. The final questionnaire was edited to match the reading level of a 12‐year‐old and approved by the panel in a face‐to‐face meeting. By this process, content validity was ensured.[6]

Data Collection

The final questionnaire was mailed to both frail and nonfrail medical and surgical inpatients who were included in the CWH before‐after study (January 2011 to July 2012) 1 week after their discharge, by a research assistant (see Supporting Information, Appendix A, in the online version of this article for a description of the study and CWH program).

Patients in the CWH study who returned the questionnaire during the postimplementation measurement period were asked to participate in the test‐retest reliability study until a predetermined sample size of 75 was reached (March 2012 to November 2012). The target interval between returning the first and second questionnaire was 2 to 14 days.[7]

In addition, patients admitted to the geriatrics departmentand therefore assumed to be frailreceived the questionnaire upon discharge (February 2012 to April 2013). The geriatrics department administered the questionnaire anonymously for evaluation and quality‐improvement purposes, as part of usual care. The secretary included the questionnaire in all patient files, and a nurse provided the questionnaire to patients together with other important discharge documents. This questionnaire also included a question regarding goal attainment, as this reflects whether what is important to the most frail elderly patients was accomplished.

Validation and Analysis

Data were analyzed using the statistical software program SPSS version 18.0 (SPSS Inc., Chicago, IL.).


Characteristics of (non)responders, levels of missing data, and measurement range were assessed using descriptive statistics.


Internal consistency was assessed by calculating Cronbach's for all available questionnaires with complete data. The answer categories were recoded to a 010 scale; 10 represents the highest quality of care. Test‐retest reliability[6] was assessed by calculating Cohen's for individual questions and intraclass correlation (ICC) for the questionnaire's mean score.


The following hypotheses were tested in order to assess construct validity: lower scores for female patients[8] and for patients who rate their health lower,[9] and with higher education[8, 9]; higher scores for patients who had an elective admission[8] and whose treatment goals were achieved (own reasoning). Finally, whether patients answered the questionnaire independently or with help should not affect scores (own reasoning). The Spearman was calculated for nonparametric and ordinal data.

In addition, we performed a Kruskal‐Wallis analysis to test the hypothesis that patients admitted to different departments have different scores. Second, we used the Mann‐Whitney U test to detect differences before and after implementation of the CWH program.

For all these analyses, only questionnaires with complete data were included.



The selected answers within the categories communication and competence of nurses and physicians by the panel overlapped with questions from the other 8 categories; thus, the final questionnaire contains 8 core questions (Table 1) (see Supporting Information, Appendix B, in the online version of this article).

The 8 Core Questions of the CareWell in Hospital Questionnaire
  • NOTE: The questionnaire for the geriatrics department included 1 additional question: Within a few days of your hospital admission, a doctor discussed the goal of the admission with you. Did you achieve your goal(s) satisfactorily? (no, not at all; yes, partially; yes, completely; don't know; doctor did not discuss my goals). See Supporting Information, Appendix A, in the online version of this article for the entire questionnaire, including the answer categories.

1. Were you informed sufficiently by your doctor regarding the various options for treating your health problems?
2. Were you able to indicate which treatment and/or care you preferred?
3. During your hospital stay, could you co‐decide what was important to your care?
4. During your hospital stay, were you supported in keeping busy and finding social contacts and activities?
5. Did you know to whom you can go within the hospital with questions, problems, or complaints?
6. Before discharge, did you talk with a member of the hospital staff regarding the care you would need after discharge?
7. Did a member of the hospital staff inform the key people and/or care providers of your discharge from the hospital?
8. During your hospital stay, did you experience 1 or more of the following events?
Did you fall?
Did you become confused?
Did you develop pressure ulcers?
Did medication errors occur?
Did you develop a urinary tract infection?
Did you develop a wound infection?
Did you experience complications with your surgery and/or treatment?

Data Collection

Figure 1 shows a flowchart of the questionnaires.

Figure 1

Flowchart of the available questionnaires returned by elderly inpatients. Abbreviations: CWH, CareWell in Hospital.

Table 2 presents data of responders compared with nonresponders who were included in the CWH study (N = 293). Patients were age 70 years and admitted 48 hours. Patients responded 14.8 11.3 days after discharge (n = 265). Response rate was 75.8%. From 18 responders no baseline characteristics were available, as only the questionnaire was collected from them to reach n = 75 for test‐retest purposes.

Characteristics of the Responding (n = 293) and Nonresponding (n = 88) Patients Included in the CareWell in Hospital Before‐After Study
No. Responders No. Nonresponders P Value
  • NOTE: Data on baseline characteristics from 18 patients in the post‐CWH measurement period are missing, and from those patients only the CareWell in Hospital questionnaires were gathered in order to reach n = 75 for test‐retest purposes. CIRS‐G ranging from 0 to 56 (with a higher score indicating more comorbidity).[14] MMSE ranging from 0 to 30 (with 30 representing the best score). Length of stay is defined as the time between admission to a CWH study department and discharge from a CWH study department. Abbreviations: CIRS‐G, Cumulative Illness Rating ScaleGeriatrics; CWH, CareWell in Hospital; MMSE, Mini‐Mental State Examination; SD, standard deviation.

Age, y SD 275 76.9 5.2 88 77.3 5.5 0.701
Male sex, n (%) 275 156 (56.7) 88 52 (59.1) 0.696
CIRS‐G, score SD 274 12.8 5.0 88 13.9 5.0 0.071
MMSE admission, score SD 264 26.7 3.7 82 25.1 4.8 0.001
MMSE discharge, score SD 230 26.9 3.7 66 25.8 4.4 0.026
Length of stay, days SD 275 8.2 7.4 88 9.6 9.7 0.322
Department, surgical (%) 275 170 (61.8) 88 56 (63.6) 0.759
Admission type, n (%) 275 88 0.343
Emergency 82 (29.8) 22 (25.0)
Elective 138 (50.2) 52 (59.1)
From other hospital or other department 55 (20.0) 14 (15.9)
Marital status, alone (%) 273 187 (68.5) 84 50 (59.5) 0.128
Discharge destination, n (%) 275 88 0.000
Home 197 (71.6) 54 (61.4)
Other hospital 69 (25.1) 20 (22.7)
Care facility 9 (3.3) 14 (15.9)
Readmission, n (%) 275 38 (13.8) 88 7 (8.0) 0.146
Readmission <1 mo, n (%) 275 28 (10.2) 88 14 (15.9) 0.144
Death <3 mo following discharge, n (%) 274 9 (3.3) 86 5 (5.8) 0.233
Received CWH intervention 149 43 (28.9) 33 15 (45.5) 0.064

Patients in the geriatrics department responded in 10.5 15.0 days (n = 111). Mean length of stay was 9.0 7.2 days (n = 116). Data regarding other baseline characteristics and response rate were unavailable due to privacy concerns.

Data Characteristics

Table 3 summarizes data of all 470 questionnaires. Response rates to the answer options ranged from 3.8% to 66.8%. Missing data among the questions ranged from 1.7% within question 8 to 7.0% within question 4. Upon combining the answer categories I don't know and missing, 7/8 questions had >10% missing data; the questions 2 and 3 had the highest percentage of missing data due to the I don't know answer option. The reasons stated by the respondents for why they could not answer these questions included cognitive disabilities; the perception that, because there was only one option (eg, in case of emergency admissions), the question did not apply to them; and/or that the patients preferred not to co‐decide because they felt that the physician knows best and can decide what is best.

Data Quality and Range and Test‐Retest Reliability of All Questionnaires Received
Data (n = 470) Test‐Retest (n = 78)
No. % No.
  • NOTE: For adverse events, the minimum amount of missing data was 1.7%. Sum scores range from 0 to 80. Mean scores range from 0 to 10. = Cohen's . Abbreviations: DK, don't know; ICC, intraclass correlation coefficient; Max, maximum; MIS, missing.

Sufficiently informed regarding treatment options 65 0.278
Not at all 23 4.9
Sometimes 90 19.1
Often 115 24.5
Every time 191 40.6
Don't know 29 6.2
Missing 21 4.7
Treatment and care preferences discussed 59 0.415
Not at all 89 18.9
Sometimes 78 16.6
Often 61 13.0
Every time 111 23.6
Don't know 103 21.9
Missing 28 6.0
Co‐decide regarding important issues 56 0.295
Not at all 75 16.0
Sometimes 86 18.3
Often 67 14.3
Every time 112 23.8
Don't know 98 20.9
Missing 32 6.8
Supported in finding (social) activities 73 0.533
Not at all 72 15.3
A little 66 14.0
Good 109 23.2
Very good 36 7.7
Not applicable 130 27.7
Don't know 24 5.1
Missing 33 7.0
Knows relevant person for questions, problems, complaints 77 0.652
Yes 279 59.4
No 107 22.8
Don't know 67 14.3
Missing 17 3.6
Discussed postdischarge care needs 75 0.574
Yes, sufficient 311 66.2
Yes, but insufficient 26 5.5
No 99 20.3
I don't know/I don't remember 18 3.8
Missing 19 4.0
Hospital informed other important people/providers of discharge 69 0.405
No 45 9.6
Some were informed 54 11.5
Yes 314 66.8
Don't know 38 8.1
Missing 19 4.0
Adverse events during hospital admission DK MIS 78 0.816
Fall, confusion, pressure ulcer, medication error, bladder infection, wound infection, complication of surgery/treatment Max 9.1% Max 4.3%
Sum Mean No. ICC
Mean score on the total questionnaire, complete cases (n = 222) 51.9 18.3 6.5 2.3 39 0.745


Of the 470 questionnaires, 222 (47.2%) had complete data and were used to analyze internal consistency. Cronbach's for the 8‐item questionnaire was 0.70 (good internal consistency).

Seventy‐eight questionnaires were available to measure test‐retest reliability. The interval between test‐retest was 8.7 4.8 days; 94.7% was returned within the targeted 14 days. Thirty‐eight patients had complete data for both measurements: ICC on the mean score of the questionnaire was 0.75 (95% confidence interval [CI]: 0.56‐0.86), which indicates good test‐retest reliability (Table 3). Including patients with incomplete data (1 to 2 missing items) yielded an ICC >0.70. Among the individual questions, Cohen's ranged from 0.28 to 0.82.


The mean questionnaire score was significantly correlated with goals achieved while hospitalized (Table 4).

Construct Validity of the CareWell in Hospital Questionnaire Based on All Questionnaires With Complete Data on Both the Variable and the Questionnaire Score
Variable Response No.a Score SD Correlation
  • NOTE: Mean scores range from 0 to 10. Abbreviations: F, female; M, male; SD, standard deviation.

  • The number differs per analysis. Education level was not known for every patient; this variable was extracted from a different questionnaire. Admission type includes only emergency admission and elective admission; patients could also be transferred from another department or hospital, but this was not included as a category as this might include emergency as well as elective admissions. Goal of admission was only available for patients from the geriatrics department, whereas educational level and admission type were not available for patients from the geriatrics department.

  • Correlation (Spearman ) is significant at the 0.01 level (1‐tailed for goal achieved).

Sex M 114 6.3 2.3 0.080
F 108 6.7 2.3
Health status Excellent 1 0.071
Very good 5 7.9 2.0
Good 52 6.7 2.4
Fair 120 6.5 2.2
Poor 28 6.2 2.1
Education level <6 grades primary school 4 4.9 1.2 0.068
Primary school 19 6.4 2.5
Higher than primary school 6 7.6 1.2
Practical training 27 6.0 2.2
Secondary vocational training 41 6.1 2.5
Pre‐university education 2 7.2 4.0
University/higher education 20 6.8 2.2
Admission type Emergency 31 6.5 2.6 0.015
Elective 61 6.6 2.0
Goal of admission achieved Yes 33 7.6 1.7 0.319b
Partially 24 6.6 2.1
No 6 4.7 2.8
Respondent Patient only 117 6.7 2.2 0.063
Patient with help 59 5.9 2.3
Other person 41 6.7 2.4

Mean scores did not differ significantly between departments (geriatrics: 6.8 2.2, n = 88; cardiothoracic surgery and lung diseases: 6.5 2.4, n = 54; internal medicine: 6.3 2.5, n = 30; general surgery: 6.0 2.2, n = 50; P = 0.234).

In addition, mean scores did not differ significantly before (6.5 2.2, n = 53) and after (6.1 2.4, n = 67) implementation of the CWH study (P = 0.320).


The CareWell in Hospital patient questionnaire is a brief 8‐item questionnaire to assess the experiences of elderly patients regarding integrated hospital care. It showed good internal consistency and test‐retest reliability, and low responsiveness. Here we discuss some issues related to the preset criteria of the questionnaire.

First, a panel representing the elderly target population was used to develop the questionnaire in order to ensure content validity, which was confirmed by good internal consistency. Yet, with respect to individualized, integrated care for frail elderly patients, we recommend including a question regarding the involvement of informal caregivers during the hospital stay, as they are important partners in healthcare.[10]

Second, the questionnaire was kept short because it should not be a burden and feasible for frail patients to complete. Nonetheless, some of the questions had a high nonresponse rate, and many patients answered I don't know, particularly to the questions 2 and 3. It does not necessarily mean that these questions are poor in quality; it could also indicate that offering individualized care is not yet embedded in the culture of elderly patients and care professionals, such that patients consider such questions to be irrelevant.[11, 12] Nevertheless, we suggest to further explore the feasibility of the questionnaire and potential additional methods for the most frail elderly,[13] who might have been excluded from the CWH study sample at this point (Table 2).

Third, the questionnaire measures experiences rather than satisfaction. Patient‐satisfaction scores are generally tightly correlated with the age, sex, education level, health status, and the person completing the questionnaire.[8] In our study, the correlation did not reach statistical significance. Nevertheless, the achievement of preset goals was correlated significantly with mean CWH scores (Table 4). These findings may indicate that individualized care experiences can indeed be assessed better using this questionnaire. Test‐retest reliability also supports validity, as we expectedand, indeed, sawhigher reliability among the more objective questions (eg, question 8). The most valuing question is question 1, which also had the lowest reliability; the word sufficiently should perhaps be removed in the next version in order to increase its reliability and objectivity.

Finally, scores did not differ between before and after implementation of the CWH program, which suggests either that the questionnaire is unable to detect change or that the program was not sufficiently effective to invoke change yet. The latter option seems plausible, as changes in the provision of individualized care were ongoing. In addition, the items on which favorable differences can be seen for CWH are in fact the items that could be most directly influenced by the CWH interventionists, questions 4, 6, and 7 (see Supporting Information, Appendix C, in the online version of this article). Lastly, we performed an extra analysis concerning the discriminating property of the questionnaire in a subgroup of frail elderly patients; we do see a significant difference in scores between the frail patients in the geriatrics department and the frail patients who received the CWH intervention: 6.8 (n = 88) vs 4.8 (n = 13) for complete data, respectively, P = 0.013; and 6.8 (n = 155) vs 5.7 (n = 37) for incomplete data (2 items missing), P = 0.017 (Mann‐Whitney U test). This may indicate that the questionnaire can measure differences in quality of care for specifically the frail elderly patients between departments. However, these issuesincluding validity and reliability characteristics per specific patient subgroupwarrant further research using a larger sample.


In conclusion, the CareWell in Hospital patient questionnaire is a feasible and reliable tool for assessing experiences of frail elderly inpatients in the provision of individualized, integrated care. To improve the questionnaire, we recommend to add a question regarding the participation of informal caregivers during the hospital stay, investigate the response rate to questions regarding participation and shared decision‐making, and study responsiveness issues further.


The authors thank Gerda van Straaten, Anne Kuijpers, and Thijs Cauven for their support with data collection. We thank all members of the ZOWEL Study Group and the panel representing the elderly target group.

Disclosures: The work was made possible by grant 60‐6190‐098‐272 and grant 60‐61900‐98‐129 of the National Programme for Elderly Care, coordinated and sponsored by ZonMw, The Netherlands, Organization of Health Research and Development. The authors report no conflicts of interest.


Online-Only Materials

   Comments ()