More than 30,000 new nurse practitioners (NPs) completed their academic programs in the academic year 2018-2019 (American Association of Colleges of Nursing, 2020). Current estimates report more than 290,000 NPs are licensed in the United States. This number jumped from an estimated 270,000 NPs in January 2019, with continuing growth (American Association of Nurse Practitioners, 2020). In conjunction with the APRN Consensus Model (National Council of State Boards of Nursing, 2014), state boards of nursing are more uniformly requiring family nurse practitioner (FNP) graduates to successfully pass the American Nurses Credentialing Center or American Association of Nurse Practitioners certification exams in order to practice. To ensure that FNP graduates are fully prepared for these certification exams, nursing schools can utilize proprietary exit exams (referred to in this study as diagnostic readiness tests or diagnostic readiness tests [DRTs]); examples include the Health Education Systems, Inc. (HESI) exam and the Advanced Practice Education Associates (APEA) exam.

Information about the reliability and validity of the FNP HESI and APEA DRTs is scant, as both are relatively new proprietary products; moreover, there is a paucity of published literature regarding the graduate-level DRT. Even a recent convenience study that reported that DRT posttest scores were improvements over pretest scores reached only the general conclusion that DRTs help facilitate FNP students to transition to the NP role (Coppa & Barcelos Winchester, 2019) and should be integrated into NP curricula (Coppa & Barcelos Winchester, 2019; Sosa & Sethares, 2015). The Coppa and Barcelos Winchester study did not reach any conclusions about how DRTs may be best integrated into FNP curricula to effectively facilitate that transition. This research aims to look further into HESI and APEA DRT score relationships and FNP students' perceptions of these tests in order to better understand student perceptions of their usefulness in the FNP curriculum and their effectiveness for preparing students to pass certification exams.

METHOD

The researchers received approval for this study through the review board of a large health science university. The study used an exploratory quantitative research design to: 1) ascertain any correlations between FNP HESI and APEA scores; 2) ascertain any statistical significance in the change, if any, in DRT scores over time; and 3) elicit student perceptions of the HESI and APEA DRTs' usefulness as preparatory tools for succeeding on certification exams.

Five hypotheses underlay the study: 1) APEA2 scores will be higher than APEA1 scores because information about weak performance areas from APEA1 results offers targets for study in the interim between APEA1 and APEA2; 2) HESI1 scores will be higher than APEA1 scores because of the extra time allotted to a more thorough preparation for the APEA2 test and availability of information about a base knowledge score attained from APEA1; 3) HESI1 and APEA2 postpredictor scores will be highly correlated because both exams are comprehensive exit exam products; 4) HESI2 scores will be higher than HESI1 scores because of the extra time available for preparation for the HESI2 exam and the availability of remediation materials and information about base knowledge scores attained for APEA1 and APEA2 exams as well as for the HESI1 exam; 5) FNP students will perceive the two exit exams as useful preparation for certification exams leading to licensure (e.g., the American Nurses Credentialing Center and American Association of Nurse Practitioners exams).

Sample

Subjects were a convenience sample of graduate-level FNP students enrolled at a large health science university school of nursing, enrolled from August 2016 to December 2016. A standard recruitment script with electronic consent to participate was emailed to students immediately following completion of the FNP HESI or postremediation HESI (if necessary) exams. This population completed FNP HESI and APEA DRT exams administered during the fall 2016 semester; sets of exam scores were available for analysis. Seventy-eight students took the pre-APEA and post-APEA exams. Of the 78, 77 also took the HESI1; 41 students took both APEA exams and both HESI exams. Only students who completed the FNP HESI and APEA DRTs were able to access the online survey link. Of those with access, eight completed the survey. All students responded voluntarily, and all student responses were anonymous. The results generated in the online survey system have been presented as an aggregate, and no one student can be linked to any specific response.

Data Collection and Analysis

EXAM SCORES

As part of their clinical courses, FNP students complete APEA1 and APEA2 exams and HESI1 and HESI2 exams near the end of the FNP program to gauge their preparedness for the certification exam following graduation. Data management of exam scores was completed by pairing individual HESI1/HESI2 and APEA1/APEA2 scores in a database and removing all identifiers from the database. In the case of remediation, HESI1 and HESI2 scores were paired in a database, and all identifiers were removed. Analysis of the deidentified test score databases employed statistical analysis using SPSS software.

QUESTIONNAIRES

After finishing these exams, the FNP students were asked to complete a Likert-type survey via an online system using an emailed invitation that included consent. Item data from the survey were analyzed employing SPSS for descriptive statistical analysis.

RESULTS

HESI/APEA Comparisons

Overall, the four study hypotheses about the comparative results of APEA and HESI exam mean scores were sometimes not borne out by data (see Supplemental Content for Tables 1 and 2, available at http://links.lww.com/NEP/A234). However, in cases where mean scores increased from initial test to subsequent test, the results were likely not due to chance (null hypothesis rejected); in cases where mean scores declined or remained virtually the same from pretest to subsequent tests (not hypothesized), chance could not be rejected as an explanation for the lower scores. All Pearson correlations involving APEA and HESI scores were moderate, positive, and significant at the .01 level, two-tailed.

HYPOTHESIS 1 (APEA2/APEA1)

Hypothesis 1 was not confirmed: The APEA posttest mean score was 2.83 points lower than the APEA pretest mean. Statistical analysis showed, however, that the null hypothesis could not be rejected (results could be a matter of chance).

HYPOTHESIS 2 (HESI1/APEA1)

Hypothesis 2 was confirmed: HESI1 mean scores were higher than and correlated to APEA1 mean scores. HESI1 mean scores were 3.17 points higher than APEA1 mean scores (null hypothesis could be rejected), and HESI1 and APEA1 scores were correlated with moderate strength.

HYPOTHESIS 3 (APEA2/HESI1)

Hypothesis 3 was only partially supported: APEA2 mean scores were lower than HESI1 scores (-5.70). The Pearson correlation of HESI1 to APEA2 showed moderate positive strength.

HYPOTHESIS 4 (HESI2/HESI1)

Hypothesis 4 was confirmed: The mean score of the HESI2 was 3.72 points higher than that of the HESI1 (null hypothesis rejected).

Students' Perceptions: Likert Results

Overall, students' perceptions of APEA and HESI DRTs being useful as preparation for passing certification exams were mixed but leaned toward agreement (see Supplemental Content for Table 3, available at http://links.lww.com/NEP/A235). For HESI, the mean was 4.625. A total of 50 percent somewhat (37.5 percent) or strongly (12.5 percent) agreed the exam was good preparation. For APEA, the mean was slightly lower, 4.50, but 75 percent of respondents were on the "agree" side of the scale: 50 percent said "somewhat agree," and 25 percent said "agree." These were the two highest mean scores for any of the 10 Likert questions.

The lowest mean scores (3.625 for each) applied equally to the APEA and the HESI: Students were only moderately confident that either exam provided useful feedback. For both HESI and APEA, only 37.5 percent were distributed equally under "somewhat agree," "agree," or "strongly agree." As to the exams being a good reflection of their knowledge, only 37.5 percent fit under the agreement categories. For HESI, the 37.5 percent all said "somewhat agree"; for APEA, 25 percent said "agree." Nonetheless, the mean score for recommending continued use of both exams for future students was higher: 3.75. For APEA, 50 percent (four of eight participants) "agreed" they would recommend continued use for future students; for HESI, the results were more varied: Only 37.5 percent "somewhat agreed" or "agreed" about continued use.

DISCUSSION

Because of limitations in the size of the sample population, especially for the questionnaire (n = 8), results from analysis of data in this descriptive study should be seen mainly as suggestive of possible directions for more systematically controlled investigations. Still, salient paths for further investigation emerged.

Research hypotheses that posttest scores would be higher than pretest scores were only partially borne out by the study data. Important research assumptions in this study were: 1) the availability and scheduling of tests would afford students test-taking practice and valuable experience with minor test design differences and 2) valuable test feedback indicating areas of performance strength and weakness, as well as having a time interval for using feedback, would contribute to higher posttest scores. These assumptions were supported to a degree: APEA and HESI exam results (in all combinations) did demonstrate at least moderate and positive correlations, indicating that, as the scores of one test increase, so too will those of subsequent tests, and this supports Coppa's general findings in the 2019 study. However, a nonintuitive result of this study might be the most interesting: Students' perceptions of the correlations between pre- and postexams were not always parallel to the statistical correlations; that is, statistical results were actually better than students' perceptions of the APEA and HESI DRTs.

This research, though small in scope, suggests a few immediate implications for FNP faculty: 1) the need to optimize student understanding of "performance feedback"; 2) to build strategies into course design that help students more effectively use pretest feedback and collaborate with students to maximize valuable use of interim time between pre- and posttests; and 3) based on more definitive research, to experiment, in course design, with the timing and order of presentation of DRTs.

Student perceptions elicited in this study suggest possible focal points for further research. Specific questions might be as follows: 1) Why are test-taker perceptions of the preparatory value of APEA and HESI exams varied and often not in sync with statistical results of test effectiveness? 2) Why do students recommend these tests to future students preparing for certification exams when they do not necessarily see them as valuable preparatory vehicles for passing their own certification exams? 3) If performance feedback and accurate indicator of student knowledge of subject matter are not key to student perceptions of test effectiveness, what do student perceptions of test effectiveness hinge upon? 4) How is APEA and HESI feedback understood by students, and are they using it efficiently? Clearer answers to questions such as these could help guide the strategies used to integrate DRTs into FNP curricula.

CONCLUSION

If it is true that nursing boards of certification are becoming more involved in judging the adequacy and quality of academic programs, thus encroaching upon the territory of academic accreditation bodies (Baker & Lockwood, 2016), graduate nursing programs should be especially mindful of how DRTs are designed into nursing curricula. The overall picture that emerges from this limited study is similar to that in a recent study by Coppa & Barcelos Winchester (2019): that DRTs are generally effective tools for preparing nursing students for certification exams and their transition to NP roles; however, that effectiveness might be dependent on the curricular design in which they appear. Moreover, that design may hinge primarily on the yet-to-be-understood criteria by which students judge DRT effectiveness.

REFERENCES