Nearly one in three patients who dies in the hospital has sepsis. Early detection and treatment increase the chance of survival; yet predicting who will develop sepsis remains challenging. The situation has created a robust market for devices to aid clinical detection of sepsis. Indeed, more than half of U.S. hospitals today employ alert technology built into their electronic health record (EHR) systems.
Researchers at the University of Michigan evaluated the effectiveness of one such clinical prediction tool, the widely used Epic Sepsis Model (ESM), which is embedded in EHR systems sold by Epic Systems Corporation. They concluded, in a study published in the August JAMA InternalMedicine, that the algorithms used by the device were "poor" at detecting sepsis-missing 67% of cases-and also generated a significant number of false alerts.
According to the manufacturer, the ESM was developed using data from 405,000 patient encounters in three hospital systems from 2013 to 2015. But because the company did not share its data, the alert system was never independently validated before hundreds of U.S. hospitals deployed it.
The Michigan team undertook a retrospective study that included 27,697 patients who had 38,455 hospitalizations between December 6, 2018, and October 20, 2019. Sepsis occurred in 2,552, or 7%, of these hospitalizations. The researchers found that the ESM had a hospitalization-level area under the curve of 0.63, meaning that 63% of the time the model correctly differentiated, or sorted, patients according to their risk of sepsis-a performance "substantially worse" than the manufacturer's reported rate of 76% to 83%, the researchers reported.
Of the 2,552 hospitalizations with sepsis, the ESM identified 183 (7%) that were missed by clinicians, as indicated by patients who did not receive antibiotics prior to or within three hours of sepsis onset. The ESM failed to identify 1,709 hospitalized patients with sepsis (67%), although 1,030 (60%) of them still received timely antibiotics because clinicians spotted the developing sepsis. Meanwhile, the ESM generated false-positive alerts for 18% of hospitalized patients-a rate considered high enough to create alert fatigue in clinical staff.
The researchers called on professional organizations to develop national guidelines for use of commercial products unverified by independent experts. "The increase and growth in deployment of proprietary models has led to an underbelly of confidential, non-peer-reviewed model performance documents that may not accurately reflect real-world model performance," they wrote.
An editorial accompanying the study advises that such commercial tools "be incorporated into care with caution." Nurses and other clinicians should maintain "a culture of independent clinical thinking" so that device results "inform but do not supplant" clinical judgment.-Joan Zolot, PA