Everyone's talking about monitoring, measuring, and improving the quality of care. "Show me the quality!!" is progressing beyond mere talk. The interest of payers and health plans in "pay for performance" is heating up.
What, however, is the proper measure of performance quality? What are the functions of different measures? Which constituencies want-and can make best use of-which measures? And finally, which measures are most and least practical to obtain and utilize?
The Joint Commission on Accreditation of Healthcare Organizations (JCAHO) has been reporting select data on hospitals' compliance with standards for some 10 years. Beginning in the summer of 2004, however, the JCAHO started releasing data on hospitals' clinical quality performance. Hospitals will report on up to 10 "core measures" (such as percentage of time aspirin is given to patients with myocardial infarction [MI] within 24 hours of arrival). These performance measures will closely parallel Centers for Medicare, and Medicaid Services' (CMS's) "starter set" of publicly released data on 10 measures from its "National Voluntary Hospital Reporting Initiative." Moreover, CMS is presently attempting to develop a patient satisfaction measure that can be added to hospitals' existing surveys. An increasing number of state hospital associations and health boards are asking member institutions to measure and report publicly on various clinical issues as well as their patient satisfaction. Some business and purchasing coalitions are asking for such data from hospitals with which they contract. Last year, close to half of California's hospitals participated in the second release of patient satisfaction data by the PEP-C (Patients' Evaluation of Performance in California) initiative.
These are relatively new developments. Health care providers have typically been reluctant to make quality measures public. The outcry (over a decade ago) that accompanied Health Care Financing Administration's (HCFA's) (CMS's former name) publication of mortality figures put hospitals on the defensive about releasing performance data. Moreover, the need to risk-adjust various measures complicates performance calculations and makes the resulting data even more difficult for the general public to understand or judge appropriately.
Although the call for public measures is growing, there has been little consistency in the justification, development, selection, or utilization of quality measuring methods. This situation underlay the formation of the National Quality Forum (NQF), whose laudable goal has been the identification of standardized quality measures that would meet the needs of all constituencies. The NQF, however, has yet to clarify whether the focus of its proposed measures is on improving the quality of care or aiding consumers in the selection of providers. These 2 different functions sound complementary but may require very different formats. This confusion underlies the recent controversy surrounding AHRQ's (Agency for Healthcare Research and Quality) development of a standardized patient satisfaction measure. Dubbed "HCAHPS," it is still being revised and has become the focus of debate about the nature and function of quality measurement.
This article is a preliminary exercise in organizing and categorizing health care quality measures. Much more discussion will be needed. Measures clearly differ in form, function, and feasibility. We need to become more familiar with the differences and develop strategies for adopting, discarding, and utilizing them appropriately.
"Quality of care" is not a one-dimensional phenomenon. There are a number of components to it that could be measured, and these can be interpreted and bundled in different ways. Each indicator or measure has a subject (what it's measuring). Each also has a function (what it does-its purpose or use).
In general, most quality measures can be placed into 2 general subject groupings. These include process measures and outcome measures. Measures can also be distinguished by their function. These include quality improvement (QI) measures and accountability measures.
PROCESS MEASURES
Process measures reflect the quality of activities (preparations, interactions, and interventions) that occur prior to and during care. The process of care thus includes the infrastructure as well as the direct delivery of care to patients. Any act of care necessarily results from technical, procedural, and organizational preparations prior to actual contact with the patient. Staffing, interaction, medication, and surgical protocols are already in place before the patient arrives. The injection, intervention, interaction, instruction, or manipulation (direct delivery) is merely a last step in the care process. Process measures fall into 3 general categories:
* Infrastructural processes (technical and organizational) that occur prior to contact with the patient.
* Care delivery processes (technical and organizational) that occur during contact with the patient.
* Patient satisfaction with the processes of care.
The infrastructure and formal organization of care
This is a reflection of a provider's general preparations for the care of patients, indicating adherence to (externally established) policies and practices. These would include sterilization protocols, equipment maintenance schedules, type and number of intensive care unit (ICU) staff, pharmacy policies, nurse training guidelines, etc. Hospitals can be scored on the number or percentage of such processes that meet externally established standards.
An existing measure here would be the JCAHO accreditation evaluation, which is derived from a lengthy on-site visit by examiners. Because of the time lag between evaluations (typically 3 years, but likely to be more frequent in the future), JCAHO accreditation scores would best serve as a supplement to more frequent evaluation methods. Much can change in 2 or 3 years.
Risk-adjustment considerations
By and large, infrastructural process protocols are universal and generic. Staff training requirements, proper disposal of sharps and waste, etc hold for all institutions under all normal conditions and should not be subject to adjustment. However, it is possible that hospital size may mitigate some standards (such as presence of certain specialized staffing or equipment).
Target users
Infrastructural grading will likely be of little interest to the general public. For the present, JCAHO, other oversight entities, and payers will be the only likely users. It is possible that with considerable public education, JCAHO accreditation scores and other similar assessments may come to have more popular interest.
Standardized technical and organizational processes that occur during the delivery of care
This includes surgical, medical, or interpersonal interventions deemed appropriate to the ailment and physical condition. Essentially, this type of process measure would involve recording the content and frequency of occurrence of specified, desired interventions. What is the percentage of time that aspirin is given on arrival for acute MI, and how does this compare with other hospitals? Were appropriate instructions recited or handed out to the patient with diabetes prior to discharge? Technical care protocols are typically established by outside entities. Data are obtained from direct observation, staff notes, and medical records and typically reported out as the percentage of relevant patients receiving the desired intervention. The preliminary measures in CMS's proposed "quality initiative" would be representative of technical process measures.
Patients themselves are generally unaware of appropriate clinical procedural protocols. Even those that they experience directly can be perceived, recalled, and evaluated differently by different patients with differing expectations, physical condition, pain tolerances, personalities, etc. Therefore, it is inappropriate to attempt to obtain procedural data directly from patients. Accurate, meaningful process data can be obtained only from staff notes, patient records, and observation by professionally trained investigators.
Risk-adjustment considerations
Care delivery protocols typically reflect broad industry agreements on appropriate interventions for specific conditions or situations. As these are not outcomes, but specified interventions, they are not usefully risk adjusted, aside from consideration of counterindications to the intervention (ie, counterindications for aspirin in a patient with acute myocardial infarction [AMI]).
Target users
Because process measures are strictly "facility centered" (the criteria for proper performance are established and judged by the caregiver or external professionals, not the patient), they are primarily meaningful to entities such as providers, state hospital associations, accreditors, and large purchasers. Patients themselves are unlikely to judge or select a provider on the basis of a list of statistics on condition-specific interventions. This is especially the case if risk-adjustment calculations complicate the data.
Patient satisfaction with the experience of care
The subjective experience of-and personal reaction to-the process of care is measured by patient satisfaction surveys.
Patient satisfaction is affected by the interaction of patients with the context and givers of care, if not the specific content of care itself. The patient may be unaware that a beta blocker is the appropriate medication to be given at a specific time following admission. However, the patient is aware of the content of explanations (understandable? "makes sense"? condescending?), the tone, empathy, and body language of the caregiver (friendly? rushed? professional looking?), the caregiver's apparent concern for the patient's comfort, etc. These, too, are as much a part of the care process as are proper drug dosages.
Unlike clinical process measures, which are strictly facility centered, patient satisfaction is a "patient-centered" process measure. It reflects the patient's personal response to, and evaluation of, care (as opposed to the hospital's view of what is appropriate). Patient satisfaction is the only available measure of the personal impact of the full spectrum of the care process.
Risk-adjustment considerations
Like measures of errors (and as opposed to mortality rates, which are heavily dependent upon patient condition), satisfaction data from individual patients or aggregates of patients cannot justifiably be risk adjusted. Different categories of patients may indeed differ in satisfaction levels (older patients are typically more satisfied than are younger patients). However, risk adjusting upward the satisfaction scores of certain categories of patients (such as the younger patients) does not make them more satisfied and would divert attention from discovering and modifying processes that contribute to these patients experiencing lower satisfaction during care. This is no different from varying a medication or dosage to fit the patient's disease, weight, etc (as opposed to a "one size fits all" approach).
Similarly, institutional level satisfaction scores are not typically candidates for risk adjustment. Although larger, teaching hospitals tend to exhibit lower patient satisfaction scores than do smaller institutions, the justification for adjusting their scores upwards is ambiguous. It may be true that patient/staff ratios and/or the clinical inexperience of residents, medical, and nursing students contribute to lower patient satisfaction in a teaching hospital. However, whether this is due to lack of technical or interpersonal skills is unclear. Admittedly, nurse and physician training necessarily requires that inexperienced students treat and/or interact with patients. At the same time, risk adjustment for lower patient satisfaction in teaching hospitals implies acceptance of substandard treatment (both technical and interpersonal). In general, hospitals (particularly nonteaching) that satisfy their patients less must simply work harder to identify those processes that negatively affect the experience of care.
Target users
Certainly, providers derive significant benefit from satisfaction monitoring, as it is a direct pipeline to quality of care experienced by the patient. At the same time-and in spite of wishful statements to the contrary-there is little hard evidence that the general public would pay attention to published patient satisfaction scores (or technical process scores, for that matter) in selecting a hospital or doctor. Rather, studies suggest that personal experiences of care and word-of-mouth reports of satisfaction have the most impact on selecting or returning to providers.
OUTCOME MEASURES
Outcome measures reflect the result of care (whether intended or unintended). This result could manifest at any time during or after the patient's stay. Outcome measures fall into 3 broad categories:
1. Adverse events or negative results of the technical process
2. The desired (intended), documentable result of care
3. Patient reports of healing
Adverse events or negative physical impact of the care process
This is actually a "reverse" quality measure. Adverse events and undesirable results of processes may manifest either while the patient is still under care or at any time after discharge. Essentially, we're talking about quantification of rates of errors, complications, nosocomial infections, unexpected returns to the operating room (OR), mortality, etc. Data are obtained primarily from hospital records, including medical and risk management notes and incident reports. Some data may also be obtained through direct patient complaints.
Self-reporting of errors by staff (many of whom tend to view incident reports as self-incriminating) is still tentative in spite of national calls for intensified nonpunitive monitoring of error. Many errors go unnoted, as voluntary reporting is highly dependent upon departmental and hospital culture. As a result, in-house-manifest error rate is not yet a reliable measure of quality. At the same time, such reports are essential to the development of error-reduction processes.
Undesirable/adverse outcomes that manifest themselves after hospitalization can vary from undiscernable to major. Moreover, patients may fail to recognize undesirable or unexpected outcomes. Or they may fail to interpret them as direct results of care received days, weeks, or months earlier (indeed, these undesirable outcomes may easily have been caused by posthospitalization events). Even if patients recognize a problem as being caused by the care that they received while in hospital, they may fail to report it. Thus, postdischarge complications or adverse results are likely grossly underreported.
Postdischarge undesirable outcomes (other than death) could theoretically be assessed through (a) voluntary physician reports (assuming patients complain to them); (b) patient complaints made directly to the hospital; or (c) patient surveys that specifically request information on discernable negative outcomes. None of these methods is likely to yield complete data. Moreover, an "adverse outcome" survey is highly unlikely to be recommended by a hospital's risk manager!! Perhaps, patients would respond more easily to such a survey from a state or national regulatory entity.
Risk-adjustment considerations
Complications, length of stay, and death are logically adjustable for patient condition. Error rates, however, are never candidates for adjustment. Moreover, if a patient's characteristics make adverse events more likely (ie, the likelihood of older patients falling; the likelihood of non-English-speaking patients misunderstanding instructions or questions), adjustments to care should be obvious and automatic.
Mortality rates, while relatively easy to monitor, are difficult to interpret without complex (and contentious) risk adjusting. Many hospital leaders roundly criticize mortality rates as not reflective of actual quality of care. Teaching hospitals, for example, claim that their patients are sicker, often "dumped" by smaller institutions that lack higher staffing ratios or the most sophisticated equipment.
Target users
If a meaningful, reliable error rate/adverse event measure can be developed, all constituencies will find it useful. It will form the backbone of a hospital QI program. The general public will likely consider it in selecting providers, as it is easily understood and appreciated. Accreditors, payers, and regulators will use such measures to pressure providers for improvement.
Documentable healing (a measure of desired outcome)
To what extent is the ulcer healed? The heart rate stabilized? The cancer gone? The hip useable and pain free? Healing, after all, is the ultimate goal of care. In theory, this is the "true" outcome. But can it be measured? What would its components be? There are at least 2 components: (1) degree of remission/repair of the disease or injury and (2) degree to which patients are able to resume desired activities.
Monitoring remission or repair would be difficult and costly. In most instances, final outcome can be measured only after the patient leaves the hospital or medical office. It is unlikely that physicians' and therapists' records will be voluntarily or readily accessible, let alone complete, with respect to the "ultimate" degree of healing. Frequently, final contact with the provider (including final medical notes) occurs before full healing has been achieved. Logically, only professional reexamination and testing at a specified time after treatment would yield documentable data on degree of healing.
At present, reexaminations for the purpose of determining degree of healing are typically part of longitudinal studies designed to gauge outcomes of specific drug therapies (such as hormone replacement) or procedures (eg, seed radiation vs prostatectomies). It is unlikely that a practical, general, objective measure of healing outcome will be available any time soon.
Risk-adjustment considerations
Whatever its form, a measure of healing will undoubtedly have some form of risk adjustment to account for age, seriousness of the problem, condition of the patient, etc. In addition, some further adjustment based on impact of the patient's lifestyle on the healing process (income differentials, daily habits, etc) would undoubtedly be necessary. This need for complex risk adjustment poses immense barriers to accurate measurement of care-related healing.
Target users
At present, it's wishful thinking. However, if a practical, affordable measure could be developed, all constituencies would be interested in the data.
Patient reports of healing
Theoretically, one could obtain outcome information on the extent to which healing has occurred directly from patients. Patients could be surveyed (at an appropriate time after treatment) for personal reports on their health or the resumption of activities interrupted by disease or injury. However, it must be recognized that such evaluations by the patient are highly subjective. Moreover, it may be difficult or impossible to clearly differentiate the degree of "care-related healing" from postdischarge social, psychological, and economic events that interact with it in shaping the patient's self-evaluation of being healed or returning to "normal" activities. This is one reason why various self-reported health status surveys (such as the Short Form [SF] series) have demonstrated little utility so far. Here, especially, results would have to be adjusted for lifestyle and a host of other highly subjective and contentious variables (that would also have to be obtained from patients).
Risk-adjustment considerations
Adjusting for patient age and general physical condition would be hard enough. But how could we control for lifestyle or personality?
Target users
As with technical outcome evaluation, essentially all constituencies would find patient reports of healing to be of great utility-if a way could be found to gather meaningful data. The prognosis is very doubtful.
OUTCOME MEASURES
Any subject measure can also be used to fill one or both of 2 functions. These include the following:
Quality improvement
QI measures yield data for internal use. They allow the provider to identify specific areas for improvement and monitor the progress of QI programs. Obviously, to be of any use, such data must be able to identify the performance of individual nursing units, shifts, medical specialties, departments, or staff members. Moreover, a useful QI measure will generate data by select categories of patients (sex, age, condition, etc).
As quality of care is an ongoing concern, QI measures should be ongoing. Monitoring of processes and outcomes should be more or less continual throughout the year, so as to monitor the progress of new programs and to identify developing problems with processes or staff.
Accountability
These measures provide data for consumption by entities external to the provider. Referred to by many as report cards, they offer comparative information on the relative quality of providers. Typical users include local or national accrediting entities, state hospital or health associations, insurers, business coalitions (that represent a significant number of insured employees), etc. Various individuals and official entities (eg, AHRQ) have also gone on record claiming that the general public will find such measures useful in their selection of providers.
It would be difficult for a single subject measure to fill both QI and accountability functions. Typically, much more detailed information (unit, shift, specialty, department, physician, etc) is required for improvement efforts. For public accountability, it is usually sufficient to limit data reporting to the level of a medical specialty and/or the entire institution (hospital, clinic, etc). Third parties will not likely be interested in internal performance differences between nursing units.
For accountability purposes, measures need be released no more than once a year. More frequent reports don't give a hospital time to develop programs to deal with quality issues that continually arise.
DISCUSSION
The measures described above differ in many ways. Some are too technical and specific for the average lay person to appreciate. Some are difficult or costly to conduct. Some require impractical levels of risk adjustment.
Of all quality indicators available today, the most easy, comprehensive, and cost-effective to obtain are process measures.
Data on errors are less reliable at present and will not be truly useful until mechanisms are developed to ensure full, honest reporting. The advantage of technical process (and, perhaps eventually, error) measures lies in their objectivity and ease of judging them against established (national) standards of quality. Data are readily obtainable in-house through observation, interviews, and records. Moreover, process and error measures generally require no risk adjustment and thus are relatively easy to understand and report.
Data on unintended outcomes in general offer a mixed bag of utility. Nosocomial infection rates can be considered objective, with little justification for risk-adjustment. Other unintended outcomes, however (eg, complications manifest while the patient is still in-house or under care, unintended returns to the OR), are logical candidates for risk-adjustments that account for condition, severity, age, etc. They thus become more difficult to interpret.
Accurate patient satisfaction data are also easy to obtain and are not justifiably risk adjustable. In contrast with other process measures, however, raw satisfaction data are not available in-house. Patients must be surveyed at home after discharge and this entails an expense. Moreover, for accurate, useful satisfaction data, patients must be contacted no more than a week or two after their hospital experience. Otherwise, recollections may be vague and misleading.
Given the highly subjective nature of patient perception of, response to, and recollection of the experience of care, patients' evaluations of care must be interpreted in light of what they are-subjective evaluations (ie, satisfaction), not accurate reports of care processes. These subjective evaluations nonetheless are extremely important indicators of care as personally experienced by patients. These experiential aspects of care can have a significant impact on the patient's attitude toward (and response to) both treatment and providers while care is ongoing. Satisfaction levels during care can affect compliance, trust, stress, and other reactions to medical management.
Regardless of who else might find the information useful, all data from these methods can be used by the provider for quality improvement so long as there is sufficient information to identify the origins and causes of problems. Again, it must be stressed that if hospitals or clinics cannot trace data back to their point of origin (eg, nursing unit) the information is essentially useless for quality improvement. At the same time, if an accountability survey (report card) is too long or too detailed to be easily understood or evaluated, its purpose will be defeated.
Thus, the intended function of any measure must be clearly stated and the measure designed to yield the kind of information desired.
The importance of failing to clearly identify the function of a measure is evident in AHRQ's recent attempt to develop a hospital patient satisfaction survey for CMS. Initially, AHRQ was unclear as to whether the "HCAHPS" survey would be a QI or public-reporting instrument. Its first version was 64-items long, generating almost universal criticism of functional ambiguity. Although AHRQ now claims that HCAHPS will be a public report card (to help consumers choose among hospitals), its latest iteration with 24 items is still viewed by many as far too long and detailed to function effectively as such.
CONCLUSIONS
The present national concern for monitoring health care quality is a very positive development. State departments of health, accreditation organizations, business coalitions, and other organizations exhibit a growing interest in objective measures of provider quality. At the same time, there is still no hard data indicating that the general public will pay attention to report cards in selecting hospitals or physicians. Studies indicate that the public pay far more attention to their own experience and that of significant others in evaluation and selection of hospitals and physicians.
Nonetheless, there may be public interest building in the utilization of limited types of data for selecting providers to treat very specific diseases. Procedure volume reports (that assume higher volume means better outcome) and magazine lists of "best hospitals" or "best doctors" for treating this or that condition are increasingly popular. This may suggest that we are in an early stage in the evolution of public utilization of quality indicators. It will be at least a year or two before we have data on whether the JCAHO's release of its core measures, or CMS's Quality Initiative (both with similar content) has an impact on public selection of providers. Begun as a voluntary report by hospitals on 10 clinical process measures, CMS's Hospital Quality Initiative may soon add a number of patient satisfaction items. A more immediate impact would likely be pressure on providers from purchasing groups and insurers to sustain or improve their quality.
In the meantime, all parties will be best served through more precise definition of both the content and function of proposed quality measures.