HOW WELL DOES A TEST PREDICT AN OUTCOME?

We have several good measures of efficacy of predictive or screening tests not addressed by the article "Predicting Licensure Success with a Computerized Comprehensive Nursing Exam," in the May/June 1999 issue of Computers in Nursing. This response to that article uses those authors' data to demonstrate how to use measures of the accuracy of predictive or screening tests. Neither Dr. Lauchner and her coauthors1 nor I address the reliability of such tests.

Please note that these efficacy measures have a long history of use in public health. Many readers are already familiar with these measures. Those who are not will find a more detailed explanation in epidemiology textbooks, or see pages xxix through xxxi in Guide to Clinical Preventive Services, published in 1989 by the United States Preventive Services Task Force2.

The response will first summarize the data collected by Dr. Lauchner and her coauthors as well as the conclusions drawn from those data. Using certain assumptions to estimate missing data will allow us to illustrate the value of four measures of a test's predictive accuracy: sensitivity, specificity, positive predictive value, and negative predictive value. The conclusion will be to agree with the authors that we can predict success fairly well. However, we cannot predict failure from the test studied by Dr. Lauchner and her coauthors or by most other available tests. Once again, the best data we have about probable success on the NCLEX exam may be our clinical judgment of students' intelligence, competence, and motivation gained in 2 years of teaching.

PREDICTING SUCCESS

Doctors Lauchner, Newman, and Britt performed a service for nursing education by identifying the need for an exam that is similar in form and content to the NCLEX and that also is computerized so that it provides immediate and detailed feedback. This discussion addresses only the RN tests and student data and focuses only on the pieces of data from the article needed to help calculate sensitivity, specificity, positive predictive value, and negative predictive value. The authors were able to collect the following data comparing outcomes on the HESI Exit Exam (E²) developed by Health Education Systems, Inc. with the NCLEX-RN.

From these data the authors conclude that, "The E² was found to be highly predictive of students' success on the licensing exam."

PREDICTING FAILURE

While we nurses value health and success, we must also understand and predict disease and failure. The value of a mammogram is that it can identify lesions suspicious of being cancerous early enough to increase the chances of a cure. Likewise, the value of a pretest taken before the real licensing exam is that one can identify students in danger of failing in time for them to study more in hopes of increasing their chances of passing. No disease screening test or exam pretest is 100% accurate. In each situation, the test administrator and the person taking the test must weigh possible harm and benefit. Is unnecessary worry for many who are mistakenly told that they may have a disease or that they may fail the real exam worth experiencing in order to correctly identify a large number of people that have a reason to worry soon enough to take corrective action?

Discussing pretests to predict success or failure instead of tests used to predict the presence of disease requires some thought and attention to the definition of what we mean by a "positive test." In disease screening, a positive test means that the test predicts that disease is present. By analogy, in licensing pretests, a positive test must mean that failure is expected. With this clarification, we can set up a table with four cells to enter the number of students who have each of the four outcomes on the NCLEX-RN: (a) correctly predicted failures; (b) incorrectly predicted failures; (c) incorrectly predicted successes; and (d) correctly predicted successes. If one remembers that in this example a failure on the NCLEX is comparable to the presence of disease, one can more easily accept referring to failure as a "positive" test outcome. Table 1 represents this four-cell table. Only the data in cells "c" and "d" are provided by Lauchner et al. Estimating the numbers in cells "a" and "b" requires making an assumption about the overall failure rate for RNs on the NCLEX. Table 1 assumes the failure rate on NCLEX-RN is 10%, which is the median percentage for all Texas RN schools (Texas Board of Nurse Examiners Web Site http://www.bne.state.tx.us/NCLEX-RN Pass Rate by School). Table 2 calculates the various criteria of pretest efficacy based on the data in Table 1.

Table 1 Predictions of HESI and Outcomes on the NCLEX-RN

Table 2 Measures of Pretest Accuracy

In order to understand how the prevalence of a condition affects these various criteria of test efficacy, Tables 3 and 4 change the assumption of the prevalence of failures on the NCLEX-RN. Table 3 assumes it is only 5%; Table 4 assumes it is 20%. Table 5 only confirms that varying the ratio of false negatives to true positives does not affect any of the efficacy criteria strongly.

Table 3 Measures of Pretest Accuracy with Assumption of 5% NCLEX Failures

Table 4 Measures of Pretest Accuracy with Assumption of 20% NCLEX Failures

Table 5 Measures of Pretest Accuracy with Assumption of 20% NCLEX Failures and Keeping the Proportion of False Negatives (Predicted to Pass, but Failed NCLEX-RN) to True Positives at 3:20 as in

The sensitivity and the positive predictive value of the pretest are directly related to the prevalence of the condition (failure to pass the licensing exam in our examples) in the population. As there are more students who fail the NCLEX-RN, one can better predict who will fail by giving a pretest. When only 5% of students fail, only 7% of those predicted to fail by the HESI pretest may actually fail. When 20% of students fail the NCLEX-RN, one-third of those predicted to fail do fail.

Conclusion and Recommendations

As Lauchner et al concluded, when most people pass a licensing exam, the HESI and probably other tests can reassure those most likely to pass that they will pass. However, these tests are very weak at correctly identifying those likely to fail. When the failure rate is 10%-a common rate for RN schools-more than 90% of those predicted to fail will pass. We do not know if prediction of failure from the HESI motivates students to study and thus pass. Alternatively, some students could be so frightened by a prediction of failure that they would perform poorly on the licensing exam despite a good knowledge base and decision-making skills. A researcher would have to administer the HESI immediately before the NCLEX to control for the confounding due to learning that takes place between the pretest and the licensing exam.

Nursing faculty must decide what they can expect a pretest to do for the school and for the students. Pretests that help students to practice using the test-taking skills required for the actual licensing exam and that give immediate and detailed feedback to make their review for the licensing exam more focused should be valuable. Neither students nor faculty can expect a pretest to accurately predict who will fail unless we admit so many unqualified students and/or provide such low-quality education that more than 20% of the students fail the licensing exam.

Carole Hanks, DrPh, RN

Associate Professor, Baylor School of Nursing

REFERENCES