Thank you to the editors of Quality Management in Health Care for inviting me to comment on this important work. Having this opportunity is a great honor, and I am mindful of the words I am using because, unlike other forms of text, the authors are unable to respond. So, I must be careful with my words. And this, I believe, is the principal merit of their study: Words matter.
In this study, the authors use natural language processing (NLP) from patient-reported experience measures to determine whether the words of patients can be processed in a manner that is more meaningful than rating an experience on some version of a Likert scale (eg, 4 out of 5 stars). Patient feedback provided via narrative, they argue, has the potential to be more specific and actionable than star ratings. Where Likert scales are perhaps most limited is in their subjectivity. For example, to determine what might be required to deliver care to an individual patient in a manner that would get 5 out of 5 stars, it must first be determined what 5 out of 5 stars means to that specific patient. Further, it must then be determined whether those expectations are reasonable, actionable, etc, and hope that the organization is capable of delivering care in that manner. Despite their apparent ease of interpretation, ratings are by their nature subjective, and a 5-star experience for one patient is not the same as it is for other patients. It is known, for example, that age, minority status, and health status influence patients' satisfaction with their care.1 In addition, the authors do a good job of establishing that other factors, such as social desirability bias and the need to be perceived as generous or altruistic, may limit the legitimacy of these ratings. And so, health care organizations are forced to focus their patient experience improvement efforts with limited, and perhaps faulty, information.
Where the authors find Likert scale ratings of patient satisfaction most troubling, however, is in their ceiling effects. "Ceiling effects" is a statistical term, and it is used to indicate that the distribution of responses does not adhere to a statistically normal distribution and instead the distribution of responses is skewed positive. The authors argue that because of these ceiling effects, there is little motivation for action and the threat of complacency toward the improvement of patient experience initiatives is prone to set in. They provide an example of similarly high ratings for ridesharing and posit how limited these ratings are as an impetus for improvement.
On this point, the authors and I disagree.
First, health care delivery is not the same as ridesharing. The stakes are different. Analogies aside, the authors and I disagree as to whether ceiling effects present a problem in the context of health care delivery. While it is certainly not a good thing for health care organizations to become complacent in patient experience improvement efforts, suggesting that patient-reported outcomes should be statistically normal is to suggest that the desired outcomes represent a wide distribution of experiences. I assure you that the decades of work on care quality improvement and centering the evaluation of care quality on patient experiences does not want the distribution of patient experiences to adhere to a statistically normal distribution.
Assuredly, for example, no one desires for it to be as likely for a patient to have a negative outcome as it is for them to have a positive outcome. Or, more specifically to patient experience ratings, why would we desire for it to be as likely for a patient to have a 1-star experience as a 5-star experience? The fact that patient-reported outcomes are achieving a ceiling effect is worthy of praise, not admonition. The presence of ceiling effects tell us that most patients are mostly satisfied with their care most of the time. Isn't this what we want?
This is a victory of the health care quality, patient-centered care, and patient experience movements. Decades of policy, practice, and scholarship have brought us to this important moment in which patients are empowered as consumers2 and evaluators.3 We should all want and desire and even encourage a positively skewed patient experience of care. That said, the authors and I are in complete agreement that historical success of patient experience improvement should not be reason to rest on our laurels and that health care organizations must continually find ways to improve patient experience and care quality.
There is a need, as the authors suggest, to induce action through the measurement and reporting of patient satisfaction. While patient satisfaction scores are publicly reported as an organizational outcome at the hospital level, there is much to be gained by reporting patient satisfaction outcomes based on patient population, patient demographics, disease states, or germane to specific hospital units. There are also instances in which patient dyads are the more appropriate unit of analysis for care quality evaluation, such as in childbirth or in transplants. In addition, the advancement of technology has created new ways for satisfaction surveys to be administered and reported. For example, some hospitals have initiated online feedback mechanisms to enable immediate response to patient feedback. The authors and I agree that NLP is another promising way forward.
Patient experience scholars and practitioners have known for some time now that patients are more likely to access Yelp reviews than HCAHPS (Hospital Consumer Assessment of Healthcare Providers and Systems) scores, and that Yelp reviews' ability to capture aspects of the care experience such as compassion and cultural competency exceeds that of their Likert-scaled counterparts.4 More recent scholarship has begun to use NLP to try to unlock the power of text feedback.5,6 These studies contribute to an exciting application of machine learning and its ability to improve patient experiences.
In my view, NLP is most promising in its ability to emphasize the patient voice. The power is in the narrative. The conduit for change thus is the patient voice. Looking ahead, I believe that NLP will be more widely utilized and emphasized in the evaluation of patient experiences, and the technology to capture these narratives and give them meaning in terms of actionable feedback will only continue to improve. But there are other ways to capture patient voice that I hope are not forsaken.
Talking with (and listening to) patients remains the catalyst for meaningful change in patient care delivery improvement. NLP and other machine learning capabilities are feedback devices that still regress to an average patient experience, but there is no such thing as an average patient.
There is no such thing as an average patient.
It's worth repeating because we so often forget. The potential for harm that is created by converting a patient's experience to numbers remains. Numbers can be averaged-humans cannot be averaged. Extreme numerical values can be omitted as outliers to ease statistical calculations. But these extremes might hold valuable narratives that would be abandoned for the sake of reducing analytic friction. A concern of mine, and one that should be shared by all who consider themselves to be patient advocates, is that the further we move toward data analytics, machine learning, and the like, the more likely we are to forget that the numbers on our various reports are human beings.
By their nature, numbers, and our affinity for easily processed data, have the potential to dehumanize patients.7 It has been noted that an overreliance on metrics, which are easy to measure, might lead organizations to disproportionately value the things that are easiest to measure.7 However, numbers don't have dreams, aspirations, purpose, significant relationships, spouses, parents, or children. So, as analytically exciting as it is to have new ways to highlight and aggregate patient voices via NLP and other capabilities, it is most important to remember that what might be one of a thousand narratives in a report is the only narrative for that patient. It is for this reason that it has been suggested that metrics are most meaningful (and most actionable) when they are used in complement with patient narratives.7
I am excited by this study as I think it highlights patient experiences and hopes to improve health systems' ability to enact changes to improve the patient experience with greater precision. Furthermore, it enables the ability to do so in a manner that is more responsive to the emotional experiences and needs of patients. However, whether the capabilities introduced by NLP and machine learning will be meaningful will depend not only the power of the analytic capabilities introduced by machine learning, but also on our ability to remember what the true value of the numbers are, the patients.
REFERENCES