Original Research

Data that drive: Closing the loop in the learning hospital system


The learning healthcare system describes a vision of US healthcare that capitalizes on science, information technology, incentives, and care culture to drive improvements in the quality of health care. The inpatient setting, one of the most costly and impactful domains of healthcare, is an ideal setting in which to use data and information technology to foster continuous learning and quality improvement. The rapid digitization of inpatient medicine offers incredible new opportunities to use data from routine care to generate new discovery and thus close the virtuous cycle of learning. We use an object lesson—sepsis care within the 21 hospitals of the Kaiser Permanente Northern California integrated healthcare delivery system—to offer insight into the critical elements necessary for developing a learning hospital system. We then describe how a hospital‐wide data‐driven approach to inpatient care can facilitate improvements in the quality of hospital care. Journal of Hospital Medicine 2016;11:S11–S17. © 2016 Society of Hospital Medicine

© 2016 Society of Hospital Medicine

In the landmark Best Care at Lower Cost report, the Institute of Medicine presents a compelling vision of a US healthcare system where science, information technology, incentives, and care culture are brought together seamlessly to produce high‐quality healthcare.[1] At the center of this transformation is the learning healthcare system, a system characterized by its ability to leverage data arising from care provision to drive rapid improvements in care delivery.[2] When steeped within the right organizational milieu, these data help to close the virtuous cycle of continuous learning moving from science to evidence to care and back to new science. The anticipated end result is a healthcare system that can provide Americans with superior care at lower cost.

Hospital‐based practitioners will recognize the inpatient setting as an ideal demonstration opportunity for continuous learning. Hospital care is costly, accounting for more than 30% of all US healthcare costs[3]; intensive care alone accounts for a notable proportion of the US gross domestic product.[4] Inpatient care is associated with significant mortality and morbidity, and its use is often greatly increased in patients' last days.[5, 6] Fortunately, the inpatient setting also offers an ideal opportunity to leverage high‐quality data to help inform and improve care. The digitization of medicine means that far more data are now available through electronic health records, medical devices, and tests.[7] This is particularly true for inpatients, for whom a large volume of data are produced even over relatively short hospital stays.

Whereas the challenge to improve hospital care is daunting, there is an incredible opportunity to advance the quality of inpatient care through realizing the vision of the learning hospital system. In the sections that follow, we use an object lessonsepsis care within hospitals of the Kaiser Permanente Northern California (KPNC) integrated healthcare delivery systemto evaluate the challenges and insights gleaned from working toward building a learning hospital system. Then, we describe further steps that could enhance the use of inpatient data to drive improved care.


Best Care at Lower Cost notes a fundamental paradox in US healthcare: although we have witnessed a dramatic expansion in biomedical knowledge, innovative therapies and surgical procedures, and clinical treatments to extend survival, US healthcare persistently falls short on the basic dimensions of quality, outcomes, cost, and equity.[1] The proposed path forward lies in building the learning healthcare system, a system characterized by continuous knowledge development, improvement, and application. Figure 1 shows the critical nodes in the framework for continuous learning, which include: (1) the development of new scientific knowledge (science), (2) the translation of science into clinical evidence of efficacy (evidence), and (3) the application of efficacious interventions through effective care delivery (care). In healthcare today, transitions between these nodes are rife with missed or wasted opportunities like delays in applying high‐quality evidence or poorly managed insights arising from scientific discovery. If such opportunities could be recovered, however, the quality of healthcare could be improved dramatically.[8]

Figure 1

Schematic of a continuously learning healthcare system, adapted from the Institute of Medicine's Best Care at Lower Cost report.

The pursuit of continuous learning is aided by rapid changes in the quality and quantity of biomedical data available over the past decade, especially through the use of electronic health records, novel biomolecular tools, and digital sensors.[2, 7, 9] The Internet has ushered in a new era of data connectivity, for example, allowing for highly engaged communication between patients and providers as well as collaboration between professional or citizen scientists on data of unprecedented scale.[10] New methodologic approaches, including data mining and machine learning, increasingly leverage commodity hardware to conduct previously computationally intractable analyses.[9] Moreover, the development of domain ontologies fosters the discovery of meaningful insights from data of heterogeneous types.[11]

Ultimately, however, improvements in data alone are inadequate to achieve continuous learning. As shown in Figure 1, whereas data form the channels that allow for transitions from science to evidence to care, novel insights need to be steeped within the right culture, motivated by the right incentives, and supported by the right leaders.[1, 12] Within the sustainable learning healthcare system, knowledge generation feeds practice change with the support and guidance of system leadership; improved practice, in turn, generates new knowledge and completes the virtuous cycle of learning.


The hospital is an ideal setting in which to foster continuous learning because advances in inpatient care have the potential to substantially improve healthcare quality and value.[8] Americans were hospitalized roughly 37 million times in 2012; in total, these episodes cost $378 billion.[3] Over 700,000 patients die in US hospitals annually, with reports showing that many patients utilize greatly increased inpatient and critical care services near the end of their lives in a manner that appears misaligned with their preferences.[11, 13] Hospital care is also highly variable in quality and cost; this heterogeneity is not closely associated with improved outcomes.[14, 15] Preventable harm and medical injury occur commonly in hospitals and are now recognized to be a leading cause of inpatient death.[16] Finally, emerging research illuminates the substantial toll that acute care has on patients and families resulting in new comorbidity, functional or neuropsychiatric impairment, rehospitalization, and financial burden that persist long after patients are discharged.[17]

Fortunately, inpatient care also exhibits several qualities that improve the likelihood that continuous learning can be achieved. Although it is clear that hospitalizations occur within the arc of a patient's larger health trajectory, these distinct episodes offer the potential to observe patient trajectories and treatments evolving within relatively compressed time intervals; over that same interval, a large volume of data are produced. Stored within comprehensive electronic health records, these granular data now allow inpatient episodes to be digitally recapitulated with high fidelity, bolstering their use in driving care improvements.[18]


Translating Science to Evidence in Sepsis

Although sepsis has attracted great attention in modern hospital care, sepsis was described long ago by Hippocrates to describe the process by which wounds fester.[19] Recast after the confirmation of germ theory, sepsis came to be known primarily as the blood poisoning resulting from pathogenic organisms.[20] However, with the advent of antibiotics, numerous scientific studies now recognize that sepsis actually results from the dysregulated host immune response to systemic infection, which can also cause organ dysfunction.[21] Based on this knowledge, landmark translational and clinical studies in the 2000s provided strong evidence that early identification of sepsis patients and aggressive infection control and resuscitation were associated with improved mortality (Figure 2, step 1).[22]

Figure 2

Closing the continuous learning loop in sepsis care. Schematic representation of the continuous learning hospital system generating new discovery in sepsis care.

Translating Evidence to Care in Sepsis at KPNC

In 2007, the leadership of KPNC initiated a regional effort to improve the quality of care and reduce the variability in performance at its medical centers (Table 1).[23] Reviewing data from nearly 1000 inpatientsthe last 50 consecutive hospital deaths from each of 19 medical centersa mortality diagnostic based on Institute for Healthcare Improvement recommendations[24] revealed that sepsis had a major impact on hospital outcomes. For example, even though sepsis patients were still relatively under‐recognized at the time, accounting for fewer than 3% of hospitalizations, they contributed to one‐quarter of hospital deaths. In light of these compelling data, senior regional leadership identified reducing sepsis mortality as a key performance improvement goal (Figure 2, step 2).

Timeline of Elements in the Kaiser Permanente Northern California Sepsis Performance Improvement Effort
Time Period Event Summary
  • NOTE: Specific elements related to data infrastructure and analysis are indicated in italics. Abbreviations: ARISE, Australasian Resuscitation in Sepsis Evaluation; CMS SEP‐1, Centers for Medicare and Medicaide Services Severe Sepsis and Septic Shock: Management Bundle; ED, emergency department; EGDT, early goal‐directed therapy; KP, Kaiser Permanente; ProCESS, Protocolized Care for Early Septic Shock; ProMISe, Protocolized Care for Early Septic Shock.

2007 Variability in hospital standardized mortality ratio observed, indicating an opportunity to drive improved outcomes.
Initiation of staggered implementation of unified electronic medical record across all KP sites (starting in 2006 and ending in 2009).
Spring 2008 Mortality diagnostic chart review completed identifying sepsis and infection‐related causes as key factors in hospital outcomes.
May 2008 Regional Mortality Summit held with a focus on patient safety and mortality reduction efforts through performance improvement. Executive regional and local leadership alignment to focus on sepsis performance improvement.
Summer 2008 Sepsis Steering Committee evaluates best available evidence, develops treatment algorithms, and plans for medical center pilots.
Fall 2008 Pilot intervention deployed at 2 medical centers.
November 2008 First Regional Sepsis Summit: development of sepsis performance improvement playbook, training materials, implementation plans, and measurement strategy.
November 2008 All medical centers begin to form multidisciplinary sepsis teams and performance improvement committees, obtain equipment and supplies including assembly of a sepsis cart. Multidisciplinary teams included ED physician champion, ED nurse champion, improvement advisor, hospitalists, intensivists, quality improvement personnel, nurse educators, and even resident physicians.
January 2009 Performance data collection begins on EGDT processes and outcomes. Initiation of 2 key elements to enhance screening for and detection of sepsis: (1) concomitant ordering of serum lactic acid along with blood cultures, and (2) definition of lactate >2.0 as a critical lab value.
Use of manual chart review for case finding and central database entry because of ongoing implementation of electronic medical record and limited sepsis‐specific data infrastructure.
March 2009 Regional train the trainer sessions occur and local educational spread efforts begin including: collaborative calls, in‐person training events, and medical center site visits.
August 2009 Grant funding from the Gordon and Betty Moore Foundation begins with a planned 2‐year duration providing funding for improvement advisors with performance improvement expertise and data infrastructure development.
November 2009 Second Regional Sepsis Summit. Identification of intermediate lactate sepsis patients having significant mortality.
January 2010 Initiate measurement of performance for intermediate lactate sepsis patients with a focus on lactate clearance as an outcome measure of interest.
2010 Development of an intranet Web‐based data abstraction tool to identify cases and auto‐populate specific fields for review. Facilities were responsible for review of cases at the local level to foster rapid feedback cycles for local performance improvement. Standardized data query tools were deployed to foster local medical center engagement and system‐level evaluation.
Accompanying development of a sepsis performance improvement scorecard allowing for comparison of longitudinal performance metrics across all facilities. Scorecard elements included: proportion of lactates drawn following ED blood culture, EGDT‐specific bundle elements (ie, number of EGDT cases, antibiotics within 1 hour, first central venous pressure within 2 hours of EGDT start, target mean arterial pressure achievement), repeat lactate elements, balancing measures for central line placement (ie, pneumothorax, central line infection), and overall sepsis statistics.
April 2011 Third Regional Sepsis Summit. Refinement of EGDT bundle and further development of intermediate lactate bundle approach, including piloting specific treatment bundles targeting this population. Collaborative performance improvement environment in which successful strategies at 1 site were rapidly disseminated to other sites including the Sepsis Alert and the Sepsis Clock.
May 2012 Research analysis of fluid volume and lactate clearance in intermediate lactate sepsis population begins.
February 2013 Fourth Regional Sepsis Summit. Regional spread of intermediate lactate bundle including the use of fluids, antibiotics, and repeat lactate measurements.
May 2013 Research analysis of the contribution of sepsis to hospital deaths (within KP and in national sample) as well as post‐sepsis resource utilization and mortality
March 2014 Publication of ProCESS randomized clinical trial, requiring systemic reevaluation of EGDT‐based sepsis strategy. Subsequent publications of ARISE and ProMISe trials confirming findings from ProCESS. Updated approach under consideration and informally disseminated to practitioners.
October 2014 Updated sepsis treatment guidelines and data capture strategy fully implemented moving away from a catheter‐based strategy for all EGDT‐eligible patients.
October 2015 Sixth Regional Sepsis Summit held to adjust sepsis treatment and data measurement strategy to align more closely with CMS SEP‐1 guidelines.

Based on the principles of performance improvement methodology, clinical and operational leaders established an environment with aligned culture, incentives, and leadership around sepsis care. The effort was launched in late 2008 at a Sepsis Summit, bringing together a multidisciplinary group of stakeholders (eg, hospitalist, emergency department, and intensive care chiefs of staff and nursing managers; medical center and nursing executive and operational leadership) and providing sepsis care pathways based on the best available evidence.[23] Regional investments in the digital infrastructure to support implementation resulted in the provision of granular data within monthly sepsis scorecards quantifying each medical center's performance and trends for a diverse set of sepsis bundle metrics.

The resulting changes in sepsis care were substantial. For example, improved early recognition of infected patients meeting the criteria for sepsis resulted in large changes in the standardized diagnostic criteria used to label patients (Figure 3A). Implementing screening strategies using serum lactate testing for any patient receiving blood cultures resulted in a roughly 10‐fold increase in the use of lactate testing in the emergency department (Figure 3B). Earlier recognition of sepsis also increased the number of patients receiving early antibiotics and receiving central venous catheters for quantitative resuscitation.[23]

Figure 3

(A) Changes in the diagnosis patterns among infected patients following program‐wide implementation of a sepsis performance improvement project at 21 hospitals in the Kaiser Permanente Northern California system. The lower portion of the figure displays the proportion of infection diagnosis codes among all hospital admissions, grouped into mutually exclusive and hierarchical categories (sorted in order as sepsis, respiratory, genitourinary, abdominal, other, skin/soft tissue, device‐related, vascular, or CNS). For example, patients with a diagnosis code of sepsis and respiratory infection would be coded exclusively within the sepsis category; the proportion of all infected patients coded as having sepsis increased rapidly after the sepsis program implementation. The upper portion of the figure shows the overall proportion of all hospital admissions that had at least 1 infection diagnosis codes (red solid line); the dotted grey line at 30% shows that, over the same period, there was only a modest increase in the proportion of all hospitalized patients with infection. (B) Changes in the number and mean values of serum lactate laboratory testing conducted in Kaiser Permanente Northern California emergency departments before and after sepsis performance improvement program implementation. Each point indicates a half‐year total. The blue line shows the total number of emergency department lactate values drawn following sepsis program implementation; there was a rapid rise in the number of lactate values checked, which plateaued after 2010. The black line shows the mean value of lactates and confidence limits (dotted lines show 95% confidence intervals) over time; the mean value of lactates decreased significantly as the volume of lactate testing increased. Abbreviations: CI, confidence interval; CNS, central nervous system; ED, emergency department.


Leveraging timely and actionable data steeped within an aligned organizational milieu resulted in large‐scale changes across a heterogeneous set of hospitals. However, to realize the true vision of a learning hospital system, a looming question remained: Could the data generated as the byproduct of routine care now be used to complete the virtuous cycle and drive new scientific discovery (Figure 2, step 3)?

Confirming Concordance in the Impact of Sepsis Nationally

The heightened identification of sepsis patients through program implementation revealed that the impact of sepsis on hospital mortality was greater than originally estimated; based on improved patient identification, sepsis now accounted for upward of 1 in 2 hospital deaths.[25] This sobering statistic confirmed that the investments in standardizing best sepsis care following the mortality diagnostic were critical. However, were similar estimates of sepsis‐attributable mortality consistent outside of the KPNC system? To study this, we examined US hospitalizations occurring across >1000 hospitals and involving >6 million hospital stays to estimate corresponding prevalence.[25] In this national sample, sepsis contributed to as many as half of hospital deaths in the United States in 2010, lending strong support to ongoing international and state‐based efforts to improve sepsis care. These studies also paved the way to use these data drawn from our large sepsis population to inform updated international consensus definitions for sepsis and septic shock.[26, 27, 28]

Identifying New Avenues for Reducing the Toll of Sepsis

A major concern of sepsis program leaders was the prior findings that sepsis hospitalizations among Medicare beneficiaries were associated with substantial new cognitive and functional disability.[29] This lingering toll of sepsis had been termed a hidden public health disaster.[30] To further understand the posthospital impact of sepsis and to begin investigating new avenues to reduce this impact, a cohort of patients was followed for 1 year following sepsis hospitalization.[31] Over that period, nearly half of sepsis survivors were rehospitalized. When compared with their presepsis levels of healthcare utilization, middle‐aged and elderly sepsis patients experienced a 3‐fold increase in their days using facility‐based care. Subsequent studies in other populations outside of KPNC have confirmed these concerning findings, resulting in new efforts to address postsepsis survivorship care.[32, 33]

Phenotyping New Targets for Standardized Sepsis Care

At its outset, the sepsis improvement program applied the best available evidence to treat patients with the most severe forms of sepsisseptic shock. However, once the initial implementation phase had succeeded, clinicians and operational leaders quickly realized from the emerging data that there was a far larger group of sepsis patients for whom treatment guidelines were poorly defined.[25, 34, 35] These were severe sepsis patients with so‐called intermediate lactate values between 2 mmol/L and 4 mmol/L; they comprised a substantial proportion of all sepsis patients dying in the hospital. Using data generated from the routine care of sepsis patients treated across 21 hospitals, the sepsis leadership group was able to rapidly assemble a cohort of intermediate lactate sepsis patients up to 20‐ to 100‐fold larger than that reported in prior studies and evaluate their outcomes.[34, 35]

The data used to evaluate these intermediate lactate sepsis patients now spurred a new implementation program in 2013 for a group of patients in whom there was essentially no existing evidence to guide care. Rapidly implemented within a mature sepsis performance improvement program, evaluations at the 6‐month and 1‐year intervals demonstrated significant decreases in mortality.[36] Importantly, to allay the justified concerns of clinicians, these evaluations also clearly showed no evidence of harm from more aggressive fluid resuscitation (eg, increased transfer to intensive care, increased rates of mechanical ventilation). Again, driven by clinician input, subgroup analyses further revealed that the implementation program was only associated with reduced mortality in patients who could be at risk for iatrogenic fluid overload (ie, those with a history of congestive heart failure or chronic kidney disease).[36] Spurred by these provocative findings, operational and clinical leaders are currently considering how to guide future care in these patients, especially with the emerging use of noninvasive methods to quantify patients' fluid responsiveness.


The object lesson of using data to drive improved sepsis care and further new scientific discovery offers some important insights for continuous learning.

Building a Digital Infrastructure for Utilizing Granular Hospital Data

As described above, current transitions between the nodes of the learning framework are rife with missed opportunities. Perhaps one of the most glaring is the inability to use highly granular data already collected within the electronic health record (eg, trajectories and trends across vital signs or laboratory results, large‐scale medication administration records to evaluate multidrug interactions). An essential starting point for continuous learning is investing in the digital infrastructure to improve the use of data beyond traditional claims (administrative dataadmission source codes, disposition codes, diagnoses, and procedures). As shown in Table 2, the first key step is incorporating laboratory data into the quality assessment/emmprovement process. In addition, using these data to automate severity of illness and risk adjustment metrics fosters use of similar comparison cohorts across time or disease types.[18, 37, 38, 39, 40]

Data Types Necessary to Actualize the Learning Hospital System Graded by Contents and the Degree of Difficulty Necessary to Access and Analyze the Data
Data Type Contents Degree of Difficulty in Accessing Degree of Difficulty in Analyzing
Administrative Traditional claims data, diagnostic or procedural codes Low Low to moderate
Standard cohort profiling Limited instances of vitals signs, laboratory, diagnostic testing, or treatment data Low to moderate Low to moderate
Metrics reporting for care improvement Standard cohort identification, aggregated achievement of treatment targets, scorecard dissemination Moderate Moderate
Advanced cohort profiling Time series of physiologic data, inpatient triage and treatment data within short temporal intervals Moderate to high High
Research‐grade discovery Data with breadth (representative sample size) and depth (highly granular physiologic and treatment data) High Very high
Patient‐reported outcomes Quality of life, functional and cognitive disability Very high High

Employing Novel Methods to Address the Limitations of Using Real‐World Data

The rapid digitization of medicine through the use of electronic medical records offers tremendous opportunities to facilitate continuous learning. However, these opportunities are accompanied by important limitations.[41] Data collected as a byproduct of real‐world care can be vulnerable to many forms of bias and confounding, potentially clouding the validity and robustness of corresponding analytic results. Fortunately, advanced methods including causal inference are now used routinely to address some limitations.[42] In the context of a learning healthcare system, other opportunities for improved study design including cluster randomized trials or stepped wedge implementation can also be employed to preserve the statistical rigor of subsequent analyses.[43] Finally, emerging methods employing randomization through the electronic medical record alongside adaptive trial design offer great potential to increase the efficiency of continuous learning.[44]

Evaluating the Hospital as a Single System

Advances in contemporary hospital care require seamless transitions of patient care, screening strategies, and therapeutic approaches across multiple hospital domains and with diverse providers; these interventions also need to happen rapidly. Many traditional approaches to inpatient care have taken a bottom‐up approach (eg, studying a specific disease within a specific hospital ward like the intensive care unit) that have proven useful but may limit generalizability when applied to a real‐world hospital operating with Pareto optimality (ie, the trade‐off scenario where new resource allocation to 1 area also requires resource withdrawal from another area). In certain cases, an empiric approach, without initial preference for any specific ward or disease, can aid decision making by hospital operational and clinical leaders by providing a global picture of impact and value.

Focusing on Early Detection in Hospital Settings as Secondary Prevention

Once patients have been admitted to the hospital, a race against the clock begins. Each additional hour of hospitalization increases the risks of iatrogenic injury or medical harm manifested by immobility, disorientation and delirium, nosocomial infections, or medication errors, among others. In this context, detection systems that use granular hospital data to focus on the earliest detection of risk can aid critical approaches to secondary prevention (Although the hospitalization for sepsis cannot be avoided, careful attention to mobility can limit the risk of developing delirium. In turn, preventing delirium can limit the risk of new functional disability).

Contextualizing Hospital Care Within a Longitudinal Trajectory

Although we described the benefit of hospital episodes having well‐demarcated beginning and ending points, it remains essential to recognize that the harms associated with hospitalization extend well beyond discharge. In this context, hospitalizations can serve as waypoints in patients' health trajectories as well as an opportunity to achieve patient‐centered care including discussing and aligning goals of care with actual care provision. Furthermore, although we have seen steady declines in hospital mortality over time, it is highly likely that we will reach a nadir in mortality where additional metrics of hospital outcomes will need to include postdischarge events like readmission, long‐term mortality, quality of life, and the prevention of disability or decline.


Hospitalizations in the United States are costly and associated with high mortality and morbidity; the toll of hospitalization also extends well beyond hospital discharge. The promise of the learning hospital system has marked improvements in the quality of hospital care, especially where healthcare systems can steep critical investments in data and digital infrastructure within the right culture, incentives, and leadership. Where continuous learning is achieved, data generated during routine care offer the potential to yield new scientific discovery and drive further improvements in hospital care.


As part of our agreement with the Gordon and Betty Moore Foundation, we made a commitment to disseminate our findings in articles such as this one. However, the Foundation and its staff played no role in how we actually structured our articles, nor did they review or preapprove any of the manuscripts submitted as part of the dissemination component. None of the authors has any conflicts of interest to declare of relevance to this work, which was funded by a combination of funding from the Gordon and Betty Moore Foundation, The Permanente Medical Group, Inc., and Kaiser Foundation Hospitals, Inc. VXL was supported by NIH K23GM112018.


Online-Only Materials

   Comments ()