DAHDAH ET AL (this issue) present an article that they describe as a comparative effectiveness study of traumatic brain injury (TBI) rehabilitation. In this study of 6975 persons with TBI drawn from the National Institute on Disability and Rehabilitation Research-funded TBI Model Systems National Database, they examined changes in Functional Independence Measure, Disability Rating Scale, and Glasgow Outcome Scale-Extended from admission to inpatient rehabilitation to discharge and to 1 year postinjury. These change scores were treated as outcomes (dependent variables) in regression models using 10 demographic, injury characteristic, and other variables as predictors. The authors used these regression models to predict expected outcomes for all participants and next calculated mean differences between observed outcomes and expected outcomes separately for all 21 centers that contributed data to the database. Identities of these centers were masked. The authors went on to interpret positive differences as indicating that centers achieved better than expected outcomes and negative differences as indicating that centers achieved outcomes poorer than they could have given the nature of the patients they treated. Although they acknowledged "...that there may have been other unmeasured factors that have an important impact on patient outcome," they speculated that differences between centers were caused by "... variations in intensity or frequency of certain therapeutic activities ..., differences in specialists available and training and experience in staff...." They concluded that "These findings may be related to differences in institutional structures, resources, and clinical practices between centers."

The Institute of Medicine defines comparative effective research (CER) as the "... generation and synthesis of evidence that compares the benefits and harms of alternative methods to prevent, diagnose, treat and monitor a clinical condition, or to improve the delivery of care." 1^(p203) The Institute of Medicine goes on to say that "The purpose of CER is to assist consumers, clinicians, purchasers, and policy makers to make informed decisions that will improve health care...."1^(p203) We do not believe that the investigation by Dahdah et al constitutes CER as defined by the Institute of Medicine. The authors treat the centers as nominal geographically defined entities. Moreover, they do not formally test for a center effect in their statistical model. Even if the centers were individually identifiable, this would have limited utility for consumers who could not all be treated at the "better performing" centers or for providers wishing to emulate the best center. To be truly useful, one would need to know something about the institutional structures, treatment assignment processes, and/or specific treatments used in the "better" centers to give direction to other facilities wishing to improve their outcomes. The article by Dahdah et al provides no description whatsoever of actual differences in care among the 21 sites.

Beyond this failure to meet the basic definition of CER, we have a number of other concerns with the methodology and conclusion of the investigation. We believe that the authors' conclusion that differences in outcomes that were unexplained by their limited set of case mix predictor variables reflected differences in treatment effectiveness is flawed. The set of baseline predictors is quite limited and does not include many factors know to affect outcomes, such as preinjury functioning, environmental supports, and early cognitive status. When patient characteristics are pitted against type, quantity, and quality of treatment in prediction models, it is generally found that patient characteristics account for a greater proportion of the variance. For instance, in the SCIRehab study, patient characteristics explained 65% (discharge) and 51% (1 year) of Motor Functional Independence Measure. Depending on the interventions considered (occupational therapy, physical therapy, nursing), adding a fairly long list of treatments and the hours of these provided changed the percentage explained to 70% to 77% (discharge) and 53% to 61% (1 year). Adding site identity for each of the 6 sites involved boosted this to 73% to 78% and 54% to 62%, respectively.2-5 Thus, unexplained outcome variability in the article by Dahdah et al may be due to missing baseline covariates or poorly performing covariates rather than differences in treatments provided.

The authors attempted to control for baseline (rehabilitation admission) differences on Functional Independence Measure and Disability Rating Scale scores by calculating change (difference) scores at rehabilitation discharge and 1-year follow-up. There is general consensus that in cases in which baseline scores differ among groups, it is preferable to use baseline scores as covariates rather than calculating difference scores.6 Note that many possible causes of baseline differences such as local differences in referral patterns, acute medical care practices, insurance coverage, and so forth are not captured in the TBIMS data set.

Dahdah et al appear not to have considered the possibility that baseline participant characteristics might interact with characteristics of the centers. As an example, an inpatient rehabilitation facility with a waiting list for admission might move patients through the rehabilitation process more rapidly, transitioning them to alternative care settings for additional services to open up beds for patients on the waiting list. Such a center would have poorer outcomes at discharge than other centers with different treatment philosophies or obligations to empty medical/surgical beds in affiliated hospitals. Adjustment by covariates such as acute rehabilitation length of stay might moderate the degree to which these facilities' outcomes are poorer but not completely account for this difference. Similarly, centers treating large numbers of patients from rural areas where few posthospital treatment options are available might have different models of rehabilitative care than centers with mostly urban patients who have easy access to posthospital services. The approach to analysis used in the article by Dahdah et al would have limited or no ability to account for the possible differences in treatment models across centers that developed in response to such local variation in injury severity, access to follow-up care, scarcity of rehabilitation beds, and so forth.

Characterization of treatments provided in rehabilitation is an extremely complex enterprise. All treatment that is provided is highly individualized and thus confounded with the patient's clinical condition. Questions regarding how to measure type, quantity, and quality of rehabilitation care that were sidestepped by Dahdah et al have not been begun to be resolved,5 so that even if one conceives of their article as a preliminary step toward conducting actual CER, much remains to be done before this can be achieved.

The goal of the article by Dahdah et al is laudable and the results, if presented differently, could have added to our knowledge of the importance of the predictor variables they studied. Unfortunately, true CER that would provide specific guidance to consumers, clinicians, and others remains elusive.

REFERENCES