Transparency is the basis for accountability, and peer-reviewed publications are major scientific products for providing transparency about scientific output and priorities and for gauging the value of publicly funded science and research. Not only do they invite challenge and scrutiny from other scientists but they also represent an accounting of applicable public funding; directly engage an independent, objective, and scientific viewpoint into issues of public importance; and via the mechanism of accountability, both directly and indirectly, involve segments of American society in the process of governance. At agencies, where science and scientific evidence are the cornerstones of their public health programs, accountability can be strengthened with an enterprise information system for reviewing and approving (clearing) agency scientific products when that system also yields operational data useful for assessment of the agency's strategic and operational scientific performance. We discuss the value of such an enterprise information system for monitoring an agency's output of peer-reviewed publications that goes beyond a simple numerical accounting of scientific products (eg, peer-review publications) and envision how this type of information system might be used to enhance important issues of science quality and excellence. We believe that information about peer-reviewed publications, along with other scientific products such as patents, data sets, or guidelines, could improve our evaluation and understanding of the health impact of public health science and research. This pathway from knowledge to impact is the cornerstone of scientific credibility, and it logically starts with an internal awareness about the importance of measuring one's scientific merit.
The Role of Peer-Reviewed Publications in Evaluating Enterprise Science and Research
Scientific agencies have both a clear interest and value in evaluating science and research. Typically, research itself is not executed as an independent or isolated activity so that we could track the relationship between the independent variable (ie, the research) and the dependent variable (ie, core products of research). Making this relationship even more complicated is a portfolio of research in which more than one research project yields results contributing to a single core product without clear indication for how to attribute the contribution of the various lines of research into the single core product.1 The problem of attribution is in the accumulation of scientific knowledge over time so that the downstream, time-separated events might appear for a variety of reasons unrelated to a particular or small number of discrete research projects or even a broader research portfolio, but rather to the accumulation of findings that made significant outcomes possible. Nevertheless, peer-reviewed publications still are an important currency for gauging scientific merit and an important link in logical relationships to outcomes, impact, and other events distal to the research itself. Usually the quantification of scientific merit is based on one or more different metrics for citation analyses such as journal impact factor,2 h-index,3 and measures provided by citation services such Google Scholar,4 Thomson Reuters,5 or the Faculty of 1000.6 Obviously, a starting point for using any of these metrics is enterprise awareness of the peer-reviewed publications authored by an agency's scientists.
A Dual Role for Enterprise Information Systems: Monitoring Scientific Productivity and Scientific Quality as Well as Operational Performance
Scientific documents produced and intended for release to the public must be scientifically reviewed and cleared and that process is generally codified in an agency's clearance policy.7,8 A clearance policy is a set of requirements guiding an agency's internal peer-review process with the ultimate goal of assuring quality scientific documents; it usually includes a matrix of agency scientists, or clearance officials, in the form of a hierarchy with increasing decision-making authority. Electronic clearance (ie, e-clearance) is adoption of an enterprise-wide information system for all of the processes and data necessary to clear manuscripts consistent with the agency's clearance policy. The workflows based on the clearance matrix are federated in the sense that they devolve to each "chain of supervision" (ie, a cascading hierarchy of organizational units that reflects the corresponding decision-making authority in the clearance process). The e-clearance system manages both the federated workflows within organizational units as well as workflows enterprise-wide that are associated with clearance. Enterprise-wide, e-clearance workflows (cross clearance) arise when the topic or content of the scientific document is shared by multiple program areas so that scientific review and clearance also is a shared responsibility. Thus, under a wide variety of circumstances, e-clearance workflows must be managed both locally and centrally, and clearance per se is an internal version of peer review designed to improve the scientific quality of the product.
Implementing e-clearance can be justified with the following 5 principles: (1) stewardship-e-clearance would be a good business practice that optimizes use of the public's resources; (2) efficiency--an enterprise-wide information system would be more efficient for scientists via standardization of the clearance process across the agency and would minimize information system redundancy by using one technical solution; (3) collaboration-e-clearance could promote collaboration by enhancing researchers and scientific leaders' access to information about work related to their own interests or areas of responsibility; (4) measurement-an e-clearance system would contribute to measurement of work related to achieving agency priorities; and (5) pipeline-an e-clearance system would support the quantification and characterization of a research pipeline more easily. Stewardship and efficiency relate to tactical (or operational) benefits of the information system, and the remaining 3 principles address its strategic value, because they represent high-level outcomes describing opportunities to accomplish the agency's scientific vision.
Enterprise-Wide Information Systems
To illustrate the dual roles-operational and strategic-of an enterprise-wide information system for clearance, we refer to the implementation of Documentum* at the US Centers for Disease Control and Prevention. Documentum is a commercial-off-the-shelf product.9 Its customization and enhancement is an ongoing adaptive and complex process requiring a sustained investment by the agency, and the customizations results in the actual implementation of the evolving e-clearance processes. An enterprise-wide information system-like Documentum has 2 fundamental distinguishing features: implementation of special purpose workflows and repositing data unique to e-clearance. Because of the ongoing customization necessary to implement e-clearance requirements, it can serve as the enterprise-wide information system for managing all of the agency's clearance workflows; it supports clearance, because clearance is not just about "sending" (ie, tracking, managing, reporting, etc.) documents electronically or manually around the agency-clearance also is about the actual, hard, time-consuming reviewer work and the revisions by the authors. Importantly, Documentum also is a database of document data and metadata. One could imagine a repository of documents and other static data associated with those documents, but this would not serve the dual purposes-data and workflow-provided by Documentum. Therefore, Documentum per se is neither clearance nor e-clearance; it is an enterprise-wide information system that can provide both tactical and strategic value to the agency. Questions about science quality and excellence, a research portfolio, or outcomes and impact leverage both the tactical and strategic decision-making principles referenced earlier, and these 3 questions are cross-cutting drivers for implementing e-clearance. Cross-cutting drivers support evaluation of agency processes associated with science quality and excellence leading to a research portfolio view for the agency's science that in turn can support assessment of the contribution of the agency's science to public health via outcomes and impact. E-clearance binds both the operational and strategic constructs in a way that requires both to be present for the agency to address issues and important questions of science quality and excellence. In fact, without a valid and reliable measurement-"counting"-infrastructure, it becomes nearly impossible to address strategic priorities with a systematic, scientific approach to move from "counting" to "accountability."
Leveraging Business Process Data for Monitoring Operational Performance
Very practical, operational questions can be answered with e-clearance data, such as time to clearance (TTC), which is among the most important and frequently requested performance metrics. It is the number of elapsed workdays that accrue between the date of submission of a document into e-clearance and the date of approval by the final clearing official. It would not be surprising if TTC varied depending on the complexity and length of the document, authors and reviewer's schedules, the number of clearing officials, and so on. The really important interest is less about pinpointing where the TTC differences exist, and more about (1) why the differences exist and (2) how to close the gaps between actual and target TTC to possibly identify factors associated with these differences and nudge agency performance toward a performance target. Because the ultimate goal of e-clearance should be to ensure scientific quality, TTC should be addressed in a manner that does not negatively impact science quality. Some factors regarding TTC are more amenable to improvement than others: the agency could support more easily business process development and training to improve efficiency than drawing down the size of an organizational unit or increasing publication productivity. These factors suggest that performance data arise for a host of factors naturally present in the enterprise, and these "naturally" collected data can be repurposed for analyses leading to adjustments in e-clearance execution. While this approach of using data from normal business practices to improve performance is common in the business world, here we focus on a system that primarily supports scientific quality. An important point is that this enterprise-wide information system makes it possible to monitor and evaluate agency performance so that implementation of its clearance policy does wind up creating a bottleneck for the dissemination of its science, and this operational value is embedded in our second principle of efficiency.
The Value of Creating a Logical Pathway With Enterprise Information Systems: Supporting Assessment of the Role of Agency's Science in Public Health Practice, Research, and Learning
Because author identification and affiliation are the keys for linking and tracing inputs through to impact (via process, outputs, and outcomes), we came to realize that clearance and its associated enterprise-wide information system were the lynchpin for creating a virtual, enterprise-wide research tracking and evaluation system (VIEW) (Figure) that can be composed of discrete information systems already existing in an agency, other governmental agencies, and even in the private sector. At the national level, this type of logical linking of existing, discrete information systems can create a "line of sight" (LoS) to reflect how health benefit is created from inputs and subsequently to guide the direction of the government in terms of policy priorities.10 This type of virtual information system can arise from a service-oriented architecture (SOA), because a SOA-based solution delivers a complete set of services that are required by the business or enterprise.11 Because Documentum is located in the continuum of business processes between inputs and impact, it is naturally situated to link upstream and downstream performance metrics in a way that would assist the agency with an enterprise view of its scientific projects and how they benefit the agency and society. We illustrate this pivotal point of interoperability (capability of different systems to exchange data) on the basis of sharing common data and information needs with clearance supported by Documentum and show how this approach to enterprise architecture could address important issues of science quality and excellence.
Documentum sits at the pivotal point in VIEW, which is the nexus between scientific processes and scientific products. The e-clearance submission function automatically provides the data required for citation retrieval from a publication database such as MEDLINE,12 when the cleared document is subsequently published and a citation record is created in MEDLINE, or when using services provided by other information systems, such as ScholarOne,13 the Chemical Abstracts Service,14 or "Others," as illustrated in the figure. When Documentum also serves as a repository for the agency's scientific publications, it is the basis for a Science Nexus System (SNS) connecting the outputs and outcomes of science to upstream resources and research as well as other to services and data via SOA. Tracking all of the agency's scientific output, generally not available in a single, agency-level database, now becomes possible as bibliographic citations for all agency's publications will appear in SNS, because Documentum reposits the associated publication data required to retrieve the bibliographic citation from the publication database where the manuscript is recorded (eg, MEDLINE or Chemical Abstracts Service). Then, to the extent possible and consistent with Federal copyright laws, the actual publications can be reposited in SNS, making it the system of record and a single, comprehensive, and transparent repository of the agency's peer-reviewed publications available to researchers, public health professionals, and the general public. This view of Documentum as an SNS providing an enterprise view of agency public health science and research also supports our collaboration principle. Thus, the linkage of SNS, which is Documentum augmented with bibliographic citation data and the actual publications, to other enterprise information systems creates a virtual system that permits the agency to link together (1) research inputs, (2) the research itself, and (3) the research outputs (eg, use in practice guidelines or recommendations, development of patents). This linkage and the information it provides is the very first step toward being able to assess the health impact of the enterprise scientific data. However, implementation of such enterprise architecture vision will require an ongoing and sustained investment by the agency.
There is no underestimating the importance of timely and appropriately cleared manuscripts to the agency's scientists and science. However, in the process of achieving such a tactical outcome, a well-architected information system can be invaluable to the agency's strategic scientific goals, and most importantly, to its fiduciary responsibility of serving public health. To make these points, we focused on the importance of leveraging an enterprise-wide information system for clearing peer-reviewed publications as one of the important tools for gauging the value of public health science. We envisioned how repurposing basic operational data from this enterprise-wide information system could strengthen agencywide efforts to ensure science quality and excellence. The information system underpinning e-clearance appears increasingly important for measuring operational and strategic performance and for disseminating agency scientific knowledge and also could be just as beneficial for establishing the value of publicly funded research.
REFERENCES