Recruitment of participants for research studies using a survey design is a challenge for many investigators, yet boards of nursing and educational programs depend on robust studies to influence policy and practice. Moreover, a recent review of the three highest impact journals in medical education found that more than half of research studies used survey methodology.1 Likewise, 60% of the published research in major higher education journals utilizes survey data.2

Although 80% is the conventional standard for acceptable responses to surveys based on the minimum methodology requirements for federally funded projects,3 there is evidence that there has been a decline in overall participation in survey research in the past 5 years.4 Most health profession education research reports response to surveys between 40% and 50%, and few report their sampling frame.1,5

The literature on survey methodology is extensive related to best practices and suggestions for improving survey response rates. These methods, however, can be fiscally or time prohibitive and may yield less than satisfactory results. Increasingly, social media is being used to increase size and representativeness of samples, as well as reduce cost. Therefore, during the conduct of a national study describing the nature of clinical experiences in nurse practitioner education, when sample size targets were at risk for not being met using traditional methods, the research team turned to social media. A search of the literature showed that in healthcare research social media (eg, Facebook, Twitter, LinkedIn) was increasingly being used to recruit subjects for research studies. This approach was predominantly used for specific health issues, such as smoking, and although most were descriptive studies, a few were interventional. Yet, no studies were found that used social media to recruit health professional students for research. Therefore, the purpose of this article is to (1) describe how social media was used in recruiting a large and hard-to-reach national sample of family nurse practitioner (FNP) students enrolled in their final clinical course and (2) report the impact social media had on reaching aggressive survey response targets.

Review of Literature

According to the Pew Research Center,6 84% of people 18 to 29 years old use at least one form of social media daily, whereas approximately 45% of people 65 years and older report using social media. Social media use has increased substantively over the past decade, whereas membership in the different platforms has ebbed and flowed for different demographics. In general, however, social media presence has remained consistent; therefore, it has been increasingly used to recruit participants for research studies.

Recruitment Representativeness

In one of the first systematic reviews exploring social media recruitment approaches, the authors examined 110 studies that used Facebook7 and found that 57% addressed a physical health or disease issue. Of the studies that reported characteristics of their sample, 86% said samples derived from Facebook were similar to traditional recruitment methods (email, printed advertisements/flyers, radio, television). In contrast, a scoping review of 30 studies published the same year found that of the 14 studies reporting demographics, only two found samples were comparable to traditional methods.8 In 2017, a systematic review of 35 studies using Facebook for recruitment showed success in reaching potential participants who were considered difficult to reach.8 The review compared Facebook with traditional recruitment methods and found improved participant selection completed in a shorter time; however, representativeness decreased with overrepresentation of young, White women.9 Oesterle et al10 reported the challenge of recruiting diverse samples. Conversely, Gu et al11 found that samples from social media recruitment were more diverse and representative of the overall population than those using traditional recruitment methods from both scope of reach and snowball effect. In a 2020 systematic review of social media recruitment in mental health research, Sanchez et al12 found social media to be a promising recruitment outlet, providing access to populations that are potentially difficult to reach, particularly with targeted advertising.12 This review also found that social media recruitment performed similarly to, if not better than, traditional recruitment. Topolovec-Vranic and Natarajan8 also found social media effective for six of eight studies where the target population was hard to reach, with three using Facebook only and three using multiple social media platforms. Leighton and colleagues13 reported incorporating snowball sampling into social media recruitment by posting on sites affiliated with their research and asking members of those groups to both post the study information on their own personal social media sites and also forward the information to potential study participants to get wide exposure from diverse populations.

Recruitment Cost

Stuart and Moore14 used Facebook ads over a 7-month period to recruit US licensed nurses for their study, resulting in 536 participants at a cost of $1.78 per completed survey. Carter-Harris et al15 recruited long-term smokers eligible for lung cancer screening for a survey. The Facebook-targeted advertisements were viewed 56 621 times over an 18-day campaign. Of the final sample of 361, 92% were recruited by Facebook at a cost of $1.51 per completed survey versus 30 participants recruited by a 3-day newspaper advertisement at a cost of $40.80 per completed survey. Sample characteristics were similar in both groups, and the research team found Facebook a highly effective method to recruit participants who met very specific inclusion criteria. Whereas Stuart and Moore14 and Carter-Harris et al15 had low cost per participant, Sanchez et al12 cited that 55.6% of the studies in their review cited social media recruitment as the more cost-effective option over traditional methods, with a median final per recruited study participant cost of $19.47. As an example, Kayrouz et al16 used Facebook to recruit hard-to-reach populations for mental health research. With 86% of their sample recruited via Facebook, the cost-effectiveness at $37 per participant was not significantly different from traditional strategies such as media releases, emails, and print advertisements (cost of $40 per participant).

Target Samples

Within recent years, social media recruitment is being used more widely in healthcare professional and student research. In 2021, Bethel et al17 reviewed three different studies using nurse participants recruited via social media. These studies all found success posting nonpaid posts in closed groups on Facebook obtaining 10% to 32% of their sample within hours of their initial post. Leighton et al13 conducted snowball nonprobability convenience sampling of nursing students using multiple social media platforms, including LinkedIn, Twitter, Instagram, blogs, and Web sites. Using this strategy, they were able to survey respondents in nine countries, with 84% of the 208 responses occurring in the first and last 15 days of the collection period. Different social media platforms performed in different ways. For example, LinkedIn activity began high and decreased throughout the study, whereas Twitter activity remained consistently high at the beginning and end of the study. Neither study discussed cost-effectiveness.

METHODS

Study Design

A cross-sectional, observational, complex-samples survey design was used to collect data from students enrolled in accredited FNP programs. This survey was intended to describe the nature of clinical experiences during FNP education, and the sample was students in FNP programs across the United States, inclusive of traditional MSN, post-master's certificates, BSN-DNP, and DNP-FNP programs. At the time, FNP program enrollments were estimated at more than 42 000 full- and part-time students, representing 67.2% of all nurse practitioner program enrollments in the United States.18 To ensure a representative sample of this population was included in the study, an aggressive recruitment plan was designed. The intent was to design a student sampling frame and strategy targeted to the population that captured the state of clinical education in FNP programs, thus ensuring that the findings were rigorous enough to inform future interventions for research, education, practice, and regulation. To that end, the target population comprised students enrolled in any type of FNP program in accredited US nursing programs.

There are a variety of educational pathways to becoming an FNP in the United States. According to the American Association of Colleges of Nursing (AACN),18 the most common type of program is the post-BSN master's-level FNP program followed by the BSN-DNP program with an FNP focus, the post-Master's FNP certificate program for nurses already holding a master's degree in nursing, and the post-master's DNP program with an FNP focus. Table 1 describes the enrollment of each type of program from fall 2014. These figures sum to a population estimate of students enrolled in FNP programs across all program types and from all 50 US states, of N = 49 980, with 44% (n = 21 477) enrolled full-time and 56% (n = 27 503) enrolled part-time.18

Table 1 AACN

Power Analysis

Power analysis for noninterventional, complex-samples survey research is a complicated and inexact endeavor.19 A key consideration in this study was that we sought to recruit from the entire population of students enrolled in accredited FNP programs. Although the exact N is unknown because of the dynamic nature of student enrollments, well-grounded estimates were figured based on rigorously conducted AACN annual school surveys. The a priori power analysis presented here was constructed to power the multilevel, multivariable analysis of student data per the study design. Using a conservative a priori power analysis approach to figure the desired study respondent sample size,20 for [eta tonos] = 0.25, [alpha] = .05, 1 - [beta] = .95, k = 125, and covxy = 5, 1594 subjects were needed. To account for responses with missing data or unusable data (eg, spurious), oversampling by 25% was desired, raising the total desired N = 1992. To provide as close to population-level estimates as possible, the aim then was to recruit up to N = 4000 currently enrolled students to participate in the study. This figure could represent approximately 10% of students enrolled in FNP programs in the United States, again assuming that not all enrolled students will be near the end of their programs and thus eligible to be invited to participate in the study. However, these figures were estimates based on many unknowns, including challenges with nonresponse in certain geographic areas of the country.

Following institutional review board approval, a list of schools with FNP in the US programs from 2013 was obtained online.21 Because this list was several years old even at the launch of the study, and to avoid systematically omitting either new or newly accredited programs, a separate list of all schools with graduate programs in nursing accredited by either Commission on Collegiate Nursing Education or Accreditation Commission for Education in Nursing was assembled using text data mining of information posted on the accreditors' Web sites. Because some schools offer multiple accredited programs (and sometimes hold accreditations from both accreditors), when the counts were combined and duplicates eliminated, there were 535 distinct schools offering 902 accredited graduate-level nursing programs. Realizing that not all of these programs offer FNP tracks (and because there is no feasible way to otherwise identify the programs that do), these figures did not constitute the final possible N of schools; rather, they represented a starting point for soliciting each school's assistance with forwarding the study invitation to eligible students. Program administrators were contacted directly and asked to forward the information via email to their FNP at a specified time and the dates (during the upcoming 1-year period) when students were expected to graduate/complete the FNP program. Based on the responses received, students were invited via email at approximately 6 weeks prior to the completion of their program to participate in the study.

Given the wide variability in how individual FNP programs are organized and delivered, and consequently, at what point during the calendar year that students complete their programs, data were collected in multiple stages over a 24-month study window. Three main waves of survey administration were anticipated within the study window. The first wave occurred in the spring semester of the first year of the study, the second wave in the end of the subsequent summer, and the third wave at the end of the fall semester of the next academic year. The cycle was repeated for two academic years to meet recruitment targets.

RESULTS

Findings With Traditional Recruitment

After 6 months of recruiting student respondents using direct email invitations to program administrators, it was clear that the recruitment rate was too sluggish to meet the demanding a priori sample size goal of 4000 students. After a strong initial response with the first wave of recruitment in April (n = 630) and May (n = 209) of 2017, the volume of responses from June through September 2017 was dramatically lower (n = 60), despite having previously identified that between one-fourth and one-third of programs listed summer 2017 program graduation or completion dates on their program Web sites. In addition, there was some imbalance in key demographic and educational characteristics of student respondents. Using regional geographic definitions used by AACN in their annual enrollment reports (see, for example, Fang et al.18), student respondents from the northeastern United States were overrepresented in the sample when compared with the AACN benchmark (29.1% vs 19.9%) and those from the midwestern United States were underrepresented (27.2% vs 34.3%). The proportions of student respondents from the southern and western United States were close to the benchmark figures from AACN.18 Likewise, students enrolled in traditional BSN-to-MSN FNP programs were the vast majority of respondents (78.2%) through September 2017, when benchmark data suggested that post-master's graduate certificate and direct entry MSN FNP students should make up a larger portion of the whole than was observed (4.4% and 4.0%, respectively). Similar still, students in fully online (18.8%) and students in mostly online, hybrid (50.7%) programs appeared to be slightly overrepresented as well. Given that just 14 months remained in the funding period after September 2017, the need to significantly boost student participant numbers was critical. Thus, after obtaining an institutional review board modification approval, a new student recruitment strategy was embarked upon using social media to contact potential study participants directly. Facebook was selected based on affordable targeted advertisements compared with other social media platforms at the time of the study.

Social Media Recruitment

Potential participants were made aware of the study directly through advertisements placed on Facebook (Figure 1). The Facebook ad platform allowed for development of targeted advertisements that showed up on a user's News/Events feed. Estimates at the launch of social media recruitment indicated a potential audience of 50 000 people with the term "Family Nurse Practitioner" either in their job title or in the interest area of their Facebook profile. Targeted Facebook users would see an ad, targeted to appear in their news feeds, based on Facebook algorithms for frequency, placement, and so on. Clicking on the ad took the user to the landing page for a presurvey verification process.

FIGURE 1. Distribution of student respondents by recruitment channel and month/year of recruitment (n = 3940).

The landing page for the "Self-enrollment Survey" contained information about the study and the criteria: either current enrollment in the final clinical course/clinical practicum experience in an FNP program or recent completion (within 90 days) of an FNP program and currently not working as an FNP. Respondents who indicated they were currently enrolled in their final FNP clinical course or had recently completed their FNP programs were provided with information on how to verify their enrollment or FNP program completion status with the study team, thereby confirming their eligibility to participate in the primary study (where an incentive was provided upon completion). Verification of eligibility was important to protect the survey incentives awarded automatically upon completion of the survey. The system was designed to award incentives for respondents recruited through verified, school-controlled channels. By providing a pathway for verification of study eligibility, student respondents from social media sites were able to participate in the study with minimal additional effort. These strategies were used to reduce and allow the research team to detect bots or automated survey takers/fraudsters.

To verify their eligibility to participate in the study, respondents had two options. The first was to upload a document such as a current course syllabus or other dated course-related document (for currently enrolled students) or a convocation program, completion letter, or other similar non-Family Educational Rights and Privacy Act protected documentation (for recent program completers) directly to the study team. Documents were reviewed, and if suitable to confirm eligibility, subjects received a link via email to participate in the study. The second option was to request confirmation from FNP program faculty by forwarding the text of an email the study team had previously sent to FNP program faculty members or program directors. Potential subjects included our study email address on the communication to the faculty member or program director so that the team would be copied on the email confirmation. When the FNP program faculty member replied to the student and study team using the email reply-all function, the study team directly received the email and could then quickly send the eligible subject a link to complete in the study questionnaires. Respondents who were enrolled in an FNP program but had not yet reached their final course were given the option to provide contact information so that the study team could follow up with them in the future.

Continuing Assurances of Confidentiality and Anonymity With Social Media Recruitment

Although the social media recruitment process involved the collection of limited contact information to facilitate verification of participant eligibility for the study and receive the study incentive of a $10 electronic gift card, the contact information was collected by a separate survey collector with a different URL, which was not linked to the primary study survey database. Documentation regarding study eligibility was checked by a member of the research team after which identifiers were removed from all databases. Only email addresses were kept in the separate incentive database to document the distribution of incentives for budgetary purposes. This level of documentation was required by the funder for audit purposes. Thus, a "wall" existed between these two survey portals, no data passed from one to the other, and anonymity of survey responses was maintained. In addition, participant responses to the primary study remained protected by the "Anonymize Responses" feature available in Qualtrics. This feature allowed for participants to be invited via an individual link (a survey link sent directly to an email address from a Qualtrics contact list) to participate in the study survey-but no identifying information (eg, IP address, email address, unique ID, etc) was associated with recorded participant responses. Moreover, this audit process enabled identification of individuals who attempted to take the survey more than once.

Findings With Social Media Recruitment

After initiating the social media recruitment plan, student enrollments in the study increased. From November 2017 through January 2018, targeted Facebook study advertisements were shown 602 389 times to 77 410 unique Facebook users. Each individual was served ads 7.68 times, on average. During this same period, 1709 students were recruited into the study, with the bulk (74%, n = 1264) recruited through social media advertising. Regular study invitation emails continued to be sent to program leaders throughout the study, providing for a small but nonnegligible recruiting base rate. After April 2018, monthly social media recruitment rates diminished as funding for social media advertisements was exhausted. Some students continued to be recruited through social media because of organic interest in the study generated through study information postings made by the research team in two Facebook groups that comprised FNP students.

In total, of the 3940 students recruited into the study, 46% (n = 1811) were recruited through social media, with the remainder (54%, n = 2129) recruited through email invitations to program administrators. The distribution of students across the AACN geographic regions closely matched figures reported by AACN. Small analytic survey weights (0.093-1.05) were applied to the cases within the data, correcting for the small undersampling or oversampling from the AACN geographic regions. Students recruited through social media advertising were unable to be accurately disaggregated from those recruited through social media snowballing or regular study information posts made to Facebook because potential participants undertook the same procedure to enroll in the study regardless of where or how on social media they learned about the study. However, using the total number of students recruited through social media as a basis for calculation, the cost per recruited student was $3.03. In comparison to mean per-subject Facebook recruitment costs of $17.48 (SD, $23.06) reported by Thornton et al7 and $19.27 (Canadian) reported by Wozney et al,22 costs for our study were substantially less.

IMPLICATIONS

Large, representative studies have clinical and policy implications for the discipline of nursing worldwide. Using evidence to drive policy has been a mantra within all aspects of healthcare in the 21st century. Therefore, a robust research methodology and sampling strategy are critical to ensure that findings are representative and can be used by stakeholders to influence change and establish standards. The importance of obtaining adequate samples is particularly critical when results have educational, clinical, and policy implications. In establishing nursing practice and education policy, decisions must be driven by strong evidence generated from quality research. Although it is common to see survey responses of less than 50% being reported in the literature, innovative sampling strategies, such as the use of social media, can dramatically increase survey participation and stratify findings. Although this challenges the convention of accepting low survey response rates as normal, it also provides a successful, cost-effective, recruitment strategy to obtain higher response rates.

LIMITATIONS

There were several limitations to this work. First, the study was limited to the United States because of the historical nature of FNP education and the large availability of nursing programs and students. The impact of seeking participants from other countries with FNP education remains unknown. Further research is also needed to determine the impact of response bias from participants recruited using social media and the efficacy of this modality over time, as new platforms of communication and social interaction continue to expand.

CONCLUSION

Despite a strong initial recruitment plan using traditional methods, the needed sample size for this national study was not initially met. Although the response rates using traditional methods could have been defended as acceptable given reported average rates of between 6% and 70% in physician populations,23 the study team desired a rate closer to 80% to represent the populations of interest more fully. Recognizing traditional methods were not achieving these goals, a dynamic plan, leveraging the power and scope of Facebook, was implemented. Using this strategy, our target for a large, geographically disperse and representative sample of students was met. Health professions and educational researchers should consider this approach, because Internet access, historically a limiting factor associated with social media use, is rapidly diminishing as technology advances and access continue to grow worldwide.

Acknowledgments

The authors thank Darrell Spurlock, Jr, PhD, RN, NEA-BC, ANEF, for his support and contribution.

References