Abstract
Background: In meta-analysis, researchers combine the results of individual studies to arrive at cumulative conclusions. Meta-analysts sometimes include "grey literature" in their evidential base, which includes unpublished studies and studies published outside widely available journals. Because grey literature is a source of data that might not employ peer review, critics have questioned the validity of its data and the results of meta-analyses that include it.
Objective: To examine evidence regarding whether grey literature should be included in meta-analyses and strategies to manage grey literature in quantitative synthesis.
Methods: This article reviews evidence on whether the results of studies published in peer-reviewed journals are representative of results from broader samplings of research on a topic as a rationale for inclusion of grey literature. Strategies to enhance access to grey literature are addressed.
Results: The most consistent and robust difference between published and grey literature is that published research is more likely to contain results that are statistically significant. Effect size estimates of published research are about one-third larger than those of unpublished studies. Unfunded and small sample studies are less likely to be published. Yet, importantly, methodological rigor does not differ between published and grey literature.
Conclusions: Meta-analyses that exclude grey literature likely (a) over-represent studies with statistically significant findings, (b) inflate effect size estimates, and (c) provide less precise effect size estimates than meta-analyses including grey literature. Meta-analyses should include grey literature to fully reflect the existing evidential base and should assess the impact of methodological variations through moderator analysis.
Meta-analysis is a powerful tool for summarizing research (Cooper & Hedges, 1994). It can provide a scientific evidence base for nursing practice and guide future nursing research (Conn & Armer, 1996). However, only the studies chosen for inclusion in a meta-analysis can determine its results.
Studies published in peer-reviewed scientific journals are the most widely accepted source of research but meta-analysts sometimes include so-called grey literature. Grey literature refers to studies with limited distribution (i.e., those not included in computerized bibliographic retrieval systems), unpublished reports, dissertations, articles in obscure journals, some online journals, conference abstracts, policy documents, reports to funding agencies, rejected or unsubmitted manuscripts, non-English language articles, and technical reports (Cook et al., 2001;Dickersin, 1994;Eysenbach, Tuische, & Diepgen, 2001;McAuley, Pham, Tugwell, & Moher, 2000). Serious concerns have been raised about the inclusion of grey literature. Although meta-analyses are increasingly common in nursing, the methodological issue of inclusion of grey literature has not been addressed in nursing journals. This article examines issues related to inclusion of grey literature and strategies for handling grey literature in meta-analyses.