Original Research

Derivation of a Clinical Model to Predict Unchanged Inpatient Echocardiograms

Abstract

BACKGROUND: Transthoracic echocardiography (TTE) is one of the most commonly ordered tests in healthcare. Repeat TTE, defined as a TTE done within 1 year of a prior TTE, represents 24% to 42% of all studies. The purpose of this study was to derive a clinical prediction model to predict unchanged repeat TTE, with the goal of defining a subset of studies that are unnecessary.

METHODS: Single-center retrospective cohort study of all hospitalized patients who had a repeat TTE between October 1, 2013, and September 30, 2014.

RESULTS: Two hundred eleven of 601 TTEs were repeat studies, of which 78 (37%) had major changes. Five variables were independent predictors of major new TTE changes, including history of intervening acute myocardial infarction, cardiothoracic surgery, major new electrocardiogram (ECG) changes, prior valve disease, and chronic kidney disease. Using the β-coefficient for each of these variables, we defined a clinical prediction model that we named the CAVES score. The acronym CAVES stands for chronic kidney disease, acute myocardial infarction, valvular disease, ECG changes, and surgery (cardiac). The prevalence of major TTE change for the full cohort was 35%. For the group with a CAVES score of −1, that probability was only 5.6%; for the group with a score of 0, the probability was 17.7%; and for the group with a score ≥1, the probability was 55.3%. The bootstrap corrected C statistic for the model was 0.78 (95% confidence interval, 0.72-0.85), indicating good discrimination.

CONCLUSIONS: Overall, the CAVES score had good discrimination and calibration. If further validated, it may be useful to predict repeat TTEs that are unlikely to have major changes.

© 2017 Society of Hospital Medicine

Transthoracic echocardiography (TTE) is one of the most commonly ordered diagnostic tests in healthcare. Studies of Medicare beneficiaries, for example, have shown that each year, approximately 20% undergo at least 1 TTE, including 4% who have 2 or more.1 TTE utilization rates increased dramatically in the 1990s and early 2000s. Between 1999 and 2008, for example, the rate of use of TTE per Medicare beneficiary nearly doubled.2 In 2014, echocardiography accounted for 10% of all Medicare spending for imaging services, or approximately $930 million.3 In response to concerns about the possible unnecessary use of TTE, the American Heart Association and American Society of Echocardiography developed Appropriate Use Criteria (AUC) in 2007 and 2011, which describe appropriate versus inappropriate indications for TTE.4 Subsequent studies have shown that rather than rooting out inappropriate studies, the vast majority of ordered studies appear to be appropriate according to the AUC criteria.5 The AUC criteria have also been criticized for being based on expert opinion rather than clinical evidence.6 Repeat TTE, defined as TTE done within 1 year of a prior TTE, represents 24% to 42% of all studies,7-9 and 31% of all Medicare beneficiaries who have a TTE get a repeat TTE within 1 year.10 In the present study, we reviewed all inpatient TTE performed over 1 year and described the group that have had a prior TTE within the past year (“repeat TTE”). We then derived a clinical prediction model to predict unchanged repeat TTE, with the goal of defining a subset of studies that are potentially unnecessary.

METHODS

The West Haven Connecticut Veteran’s Administration Hospital (WHVA), located outside New Haven, Connecticut, is a 228-bed tertiary care center affiliated with Yale University School of Medicine. Potential subjects were identified from review of the electronic medical records of all inpatients who had an inpatient echocardiogram between October 1, 2013, and September 30, 2014. Patient’s records were reviewed by using a standardized data extraction form for demographics, comorbidity, cardiovascular risk factors, service ordering the TTE, intensive care unit (ICU) location, prior TTE abnormalities, TTE indication, AUC category, time between TTEs, technical quality of TTE, electrocardiogram (ECG) abnormalities, history of intervening acute coronary syndrome, cardiac surgery, and revascularization. Candidate predictors included any variables suspected by the authors as being potentially associated with the primary outcome of changed repeat TTE. All patients who had an inpatient TTE and a prior TTE within the Veterans Affairs (VA) system within the past year were included in the study. Repeat studies from the same admission were only counted as 1 TTE and patients had to have had a prior TTE from a different admission or a prior outpatient TTE to be included. Patients who did not have a prior TTE within the past year or who had only a transesophageal echocardiogram or stress echocardiography were excluded. Suboptimal studies were included but noted as limited quality. The study was approved by the WHVA Institutional Review Board. The Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis statement was used in planning and reporting this study.11

TTEs were classified as normal, mildly abnormal, or with a major abnormality based on previously published definitions.12-14 Any abnormality was defined as any left ventricle (LV) dysfunction (left ventricular ejection fraction [LVEF] <55%), any aortic or mitral valve stenosis, any regional wall motion abnormality, any right ventricular dysfunction, any pulmonary hypertension, mild or greater valvular regurgitation, any diastolic dysfunction, moderate or greater pericardial effusion, any ventricular hypertrophy, or any other significant abnormality including thrombus, vegetation, or tamponade. Major abnormality was defined as moderate or greater LV dysfunction (LVEF <45%), moderate or greater valvular regurgitation, moderate or greater valvular stenosis (aortic or mitral valve area <1.5 cm²), any regional wall motion abnormality, right ventricular dysfunction, moderate or greater pulmonary hypertension, moderate or greater diastolic dysfunction, moderate or greater pericardial effusion, or any other major abnormality including thrombus, vegetation, tumor, or tamponade. Repeat TTEs were classified as changed or unchanged. Changed TTEs were divided into any new abnormality or improvement or a new major abnormality or improvement. Any new abnormality or improvement was defined as any new TTE abnormality that had not previously been described or in which there was a change of at least 1 severity grade from a prior TTE, including improvement by 1 grade. A new major TTE abnormality or improvement was defined as any new major TTE abnormality that had previously been normal, or if there had been a prior abnormality, a change in at least 1 severity grade for LVEF or 2 severity grades for abnormal valvular, pericardial, or prior pulmonary hypertension, including improvement by 2 severity grades. A change from mild to moderate mitral regurgitation therefore was classified as a nonmajor change, whereas a change from mild to severe was classified as major. All TTE classifications were based on the electronic TTE reports and were reviewed by 2 independent investigators (CG and JC) blinded to the patients’ other clinical characteristics. For TTE studies in which the investigators agreed, that determination was the final classification. Disagreements were reviewed and the final classification was determined by consensus.

In an analogous manner, ECGs were classified as normal, mildly abnormal, or with a major abnormality based on previous definitions in the literature.15 Major abnormality was defined as atrial fibrillation or flutter, high-degree atrioventricular blocks, left bundle-branch block, right bundle-branch block, indeterminate conduction delay, q-wave myocardial infarction, isolated ischemic abnormalities, left ventricular hypertrophy with ST-T abnormalities, other arrhythmias including supraventricular tachycardia (SVT) or ventricular tachycardia (VT), low voltage (peak-to-peak QRS amplitude of <5 mm in the limb leads and/or <10 mm in the precordial leads), paced rhythm, sinus tachycardia (heart rate [HR] >100) or bradycardia (HR <50). Mild ECG abnormality was defined as low-grade atrioventricular blocks, borderline prolonged ventricular excitation, prolonged ventricular repolarization, isolated minor Q and ST-T abnormalities, left ventricular hypertrophy without ST-T abnormalities, left atrial enlargement, atrial or ventricular premature beats, or fascicular blocks. New major ECG abnormalities were any of the listed major ECG abnormalities that were not present on ECGs prior to the admission during which the repeat TTE was performed.

Other study definitions included intervening acute myocardial infarction (AMI), which was defined by any intervening history of elevated troponins, regardless of symptoms or ECG changes and including demand ischemia. Chronic kidney disease (CKD) was defined as an abnormal serum creatinine on 2 or more occasions 3 months apart. Active cancer was defined as receiving chemotherapy or palliative care for advanced cancer. Valvular heart disease was defined as prior moderate or severe valvular stenosis or regurgitation.

For analysis, we first compared patients with repeat TTE with major changes with those without major changes. For comparison of dichotomous variables, χ2 or Fisher exact tests were used. For continuous variables, Student t test or the Mann-Whitney U test were performed. Because many of the frequencies of individual AUC criteria were small, related AUC criteria were grouped for analysis as grouped by the tables of the American College of Cardiology Foundation Appropriate Use Criteria Task Force, American Society of Echocardiography, American Heart Association, American Society of Nuclear Cardiology, Heart Failure Society of America, Heart Rhythm Society, Society for Cardiovascular Angiography and Interventions, Society of Critical Care Medicine, Society of Cardiovascular Computed Tomography, and Society for Cardiovascular Magnetic Resonance (ACCF/ASE/AHA) Guideline.4 Criteria groupings that were significantly less likely to have major TTE changes on analysis were classified as low risk and criteria that were significantly more likely were classified as high risk. Criteria groupings that were not significantly associated with TTE change were classified as average risk. All variables with P values less than 0.05 on bivariate analysis were then entered into a multivariate logistic regression analysis with major TTE change as the dependent variable, using backward stepwise variable selection with entry and exit criteria of P < 0.05 and P > 0.10, respectively. Scores were derived by converting the regression coefficients of independently predictive variables in the logistic regression model into corresponding integers. A total score was calculated for each patient by summing up the points for each independently significant variable. Model performance was described by calculating a C statistic by creation of a receiver operating characteristic curve to assess discrimination, and by performing the Hosmer and Lemeshow test to assess calibration. Internal validation was assessed by calculating the C statistic using the statistical method of bootstrapping in which the data were resampled multiple times (n = 200) and the average resultant C statistic reported. The bootstrap analysis was performed using R version 3.1 (R Foundation for Statistical Computing, Vienna, Austria). All other analyses were performed using SPSS version 21.0 (IBM, Armonk, New York). P values <0.05 were considered significant.

RESULTS

During the 1-year study period, there were 3944 medical/surgical admissions for 3266 patients and 845 inpatient TTEs obtained on 601 patients. Of all patients who were admitted, 601/3266 (18.4%) had at least 1 inpatient TTE. Of these 601 TTEs, 211 (35%) had a TTE within the VA system during the prior year. Of the 211 repeat TTEs, 67 (32%) were unchanged, 66 (31%) had minor changes, and 78 (37%) had major changes. The kappa statistic for agreement between extractors for “major TTE change” was 0.91, P < 0.001. The 10 most common AUC indications for TTE, which accounted for 72% of all studies, are listed in Table 1. The initial AUCs assigned by reviewers were the same in 187 of 211 TTEs (kappa 0.86, P < 0.001). Most indications were not associated with TTE outcome, although studies ordered for AUC indications 1 and 2 were less likely be associated with major changes and AUC indications 22 and 47 were more likely to be associated with major changes. Table 2 shows the comparison of the 78 patients that had repeat TTE with major changes compared with the 133 patients that did not. Nine variables were significantly different between the 2 groups; repeat TTEs with major changes were more likely to have dementia, be ordered by the surgery service, be located in an ICU, have major new ECG changes, have had prior valvular heart disease, have had an intervening AMI or cardiac surgery, or be in a high-risk AUC category. Patients with CKD were less likely to have major changes. Table 3 shows the results of the multivariate analysis; CKD, intervening AMI, prior valvular heart disease, major new ECG changes, and intervening cardiac surgery all independently predicted major changes on repeat TTE. Based on the β-coefficient for each variable, a point system was assigned to each variable and a total score calculated for each patient. Most variables had β-coefficients close to 1 and were therefore assigned a score of 1. CKD was associated with a lower risk of major TTE abnormality and was assigned a negative score. Intervening AMI was associated with a β-coefficient of 2.2 and was assigned a score of 2. Based on the points assigned to each variable and its presence or absence for each patient, a total score, which we named the CAVES score, was calculated. The acronym CAVES stands for CKD, AMI, valvular disease, ECG changes, and surgery (cardiac). Table 4 shows the frequencies of each score for each patient, ranging from patients with CKD and no other risk factors who scored −1 to patients without CKD who had all 4 of the other variables who scored 5. The prevalence of major TTE change for the full cohort was 37%. For the group with a CAVES score of −1, the probability was only 5.6%; for the group with a score of 0, the probability was 17.7%; and for the group with a score ≥1, the probability was 55.3%.

The only missing data were for the variables of admission or baseline ECG, which were missing for 13 patients (6.1%). Ten of these 13 were patients referred for cardiac surgery or revascularization from nonlocal VA hospitals and hence had no prior ECGs in our electronic records. We included these patients and assumed for analysis that their ECGs were unchanged.

The bootstrap corrected C statistic for the model was 0.78 (95% confidence interval, 0.72-0.85), indicating good discrimination. The Hosmer and Lemeshow test showed nonsignificance, indicating good calibration (χ2 = 5.20, df = 6, P = 0.52).

DISCUSSION

In this retrospective study, we found that approximately 18% of all patients admitted to the hospital had an inpatient TTE performed, and that approximately 35% of this group had a prior TTE within the past year. Of the group with prior TTEs within the past year, 37% had a major new change and 63% had either minor or no changes. Prior studies have reported similar high rates of repeat TTE7-9 and of major changes on repeat TTE.8,14,16 On multivariate analysis, we found that 5 variables were independent predictors of new changes on TTE—absence of CKD, intervening AMI, intervening cardiac surgery, history of valvular heart disease, and major new ECG changes. We developed and internally validated a risk score based on these 5 variables, which was found to have good overall accuracy as measured by the bootstrap corrected C statistic. The simplified version of the score divides patients into low, intermediate, and high risk for major changes on TTE. The low-risk group, defined as the group with no risk factors, had an approximately 6% risk of a major TTE change; the intermediate risk group, defined as a score of 0, had an 18% risk of major TTE change; and the high-risk group, defined as a score of 1 or greater, had a 55% chance of major TTE change. We believe that this risk score, if further validated, will potentially allow hospital-based clinicians to estimate the chance of a major change on TTE prior to ordering the study. For the low-risk group, this may indicate that the study is unnecessary. Conversely, for patients at high risk, this may offer further evidence that it will be useful to obtain a repeat TTE.

The primary limitation of the study is that it was relatively small and derived at a single institution and will thus need to be externally validated prior to adoption. Although there are no widely accepted criteria for calculating study sizes for clinical prediction models, a small study increases the chance of overfitting, as does the lack of external validation. Because of the relatively small size, it is possible that important variables were found to lack association with the outcome because of their rarity. Many of the individual AUC indications, for example, were infrequent. Another limitation is the lack of female patients, which may limit generalizability. Finally, although the overall performance of the model was good, the lowest-risk group was only 8.5% of the cohort, which may limit its ability to decrease the number of repeat TTE. The intermediate-risk group represented a much larger proportion of 37% but still had an 18% risk of major TTE changes.

Strengths of the study included the careful definitions of study variables, particularly of AUC, major TTE, and ECG changes. The 5 variables in the final model are clinically plausible, with the possible exception of CKD, which was associated with a lower risk of having a changed repeat TTE, possibly because of the nonspecificity of edema in patients with CKD. There were also minimal missing data, which only occurred in 6% of patients, and for only 1 variable, baseline ECG.

In summary, we have developed a simple score to predict the likelihood of major changes on repeat TTEs for hospitalized patients. The CAVES score identified 8.5% of patients as being low risk for changed repeat TTE, 37% at intermediate risk, and 54% at high risk for major changes. We believe that the CAVES score, if further validated, may be used to risk stratify patients for ordering TTE and to potentially avoid unnecessary repeat studies.

Disclosure

The authors indicated no conflicts of interest.

References

   Comments ()