This issue of Critical Care Nurse (CCN) warmly welcomes a valuable enhancement to the content provided to our readers: ongoing publication of summaries of Cochrane Collaboration research reviews relevant to critical care nursing. The Cochrane Collaboration, an international, independent, nonprofit organization of more than 28000 people from more than 100 countries, was established in 1993 to ensure that unbiased, current, and accurate information regarding the effectiveness of health care interventions would be readily available worldwide.1  The Collaboration produces, updates, and disseminates systematic reviews of research through the online Cochrane Library2  and a number of cooperating organizations.

When critical care nurses consult Cochrane Reviews, they can trust that the evidence found there is valid, reliable, and clinically useful.

The Cochrane Nursing Care Field (CNCF; cncf.cochrane.org/home) supports the dissemination and use of Cochrane systematic review findings in a number of ways, including preparation of summaries of reviews that have implications for nursing practice.3  The CNCF is coordinated by the Joanna Briggs Institute (JBI) at the University of Adelaide in Australia, an international, not-for-profit organization that collaborates with more than 70 organizations around the world to promote and support the synthesis, appraisal, transfer, and use of evidence-based care.4  The Cochrane Collaboration employs Archie Cochrane’s original definition of evidence-based health care as the conscientious use of current best evidence in making decisions about the care of individual patients or the delivery of health services. Current best evidence is up-to-date information from relevant, valid research. . .5 

The Cochrane Review procedures incorporate rigorous attention not only in examination of the results reported by each study, but also in determining whether the methodology employed to derive those results was appropriate and free of bias. Even a cursory perusal of the Cochrane Handbook6  chapters related to assessment of internal and external validity, reliability, and risk of bias attest to the high threshold those attributes must reach before the Cochrane Collaboration allows “data” and “findings” to be construed as “evidence” for making health care decisions. As a result, the value of the CNCF summaries lies in both their succinct distillation of outcomes from numerous sources into a coherent, bottom-line synopsis as well as in their verification of the confidence that health care professionals can place in those outcomes. When critical care nurses consult Cochrane Reviews, they can trust that the evidence found there is valid, reliable, and clinically useful.

As fortunate as we are to have the Cochrane Library, however, a few caveats shared previously with CCN readers7  need to be kept in mind: the Cochrane Reviews focus solely on the effectiveness of health care interventions, not on the entire field of health care and not specifically on nursing care, and, although the Cochrane Database8  includes 5000 or more systematic reviews, few are specific to critical care, acute care or progressive care patient populations. As a result, critical care nurses will often need to examine other sources to search for the evidence needed for evidence-based patient care. Considering the sheer volume of health care–related information, how can critical care nurses distinguish which of these sources are credible and trustworthy?

Becoming a discriminating judge of the reliability and applicability of health care information is extremely important for critical care nurses in many ways:

  • As direct consumers, who use evidence to make personal health decisions for themselves and their family

  • As health care professionals, who apply evidence in their own patient care and in developing and updating unit algorithms, policies, procedures, and standards of nursing practice

  • As peer reviewers for professional nursing books and journals, who use their own expertise and literature search skills to assist in appraising the suitability of manuscripts for publication

  • As nurse authors, who identify, compare, and summarize evidence in review papers, case reports, quality improvement projects, or research reports that will be shared online, in print or via other media with nurse colleagues around the world

Before we can apply evidence from the health care literature for any of these purposes, we need to know that it is trustworthy because—quite literally—lives depend on it. Because lives may depend on how effectively we vet health care information, we have an obligation to not take information at face value, but to verify that the information is accurate, current, and free of bias. Although critically appraising health care information from any source might initially appear overwhelming and time-consuming, I’d like to suggest a few fairly straightforward guidelines to support critical care nurses in this effort.

The following guidelines may assist critical care nurses in appraising the quality of information that may serve as “evidence” for nursing practice.

1. Search Only Reputable Sources of Health Information

Valid and reliable health care information is most likely to be generated and published by reputable sources. Reputable purveyors of health information are typically located within the federal government or its state or local agencies; at institutions of higher learning; not-for-profit health care systems, organizations, and professional associations; and even at some newspapers in the health columns offered by the New York Times or the Wall Street Journal. What makes these sources credible is their consistent use of standard operating procedures that include established measures (such as fact-checking, double-blinded peer review, editing and verification) aimed at ensuring maximal quality control and minimal potential for bias before disseminating information. Websites for these sources often identify an editorial board that includes members who are readily recognized as authors and authorities in their respective field. Many of the best places to search for evidence on any health care issue can be launched at one or more of the following sources:

2. Search Health Information Sources Recommended by Recognized Experts in Biomedical Information Resources

The Consumer and Patient Health Information Section of the Medical Library Association9  recommends the following websites of medical information for health care professionals:

3. Exercise Skepticism for Material Found at Online Locations Not Universally Regarded as Reputable for Health Care Information

The Internet affords a venue for a wide array of information ranging from useless minutiae and fabrications to extremely informative. This material exists in many forms and formats and is prepared for widely disparate purposes (to inform, teach, sell, humor, entertain, share, frighten, mollify, etc). In contrast to traditional print sources of professional or scientific literature, which are typically reviewed by experts in that field, corrected for errors or omissions, and edited and held to established publication standards before dissemination, there is no oversight required for much of the material posted to the Internet. For this reason, those who search the Internet for information, including sources of health information, must assume responsibility for appraising information found there to determine its value, quality, applicability, and suitability.

Although everyone needs to follow certain basic admonitions when searching the Internet for health care information,10  additional issues need to be scrutinized for research studies. One helpful device designed for critically evaluating Internet research sources is CARS (Figure 1), with the acronym reflecting the 4 attributes included in this appraisal: credibility, accuracy, reasonableness, and support.11  As CARS suggest, when the accuracy or credibility of information lacks support via reference citations or other means of identifying the source; when the author’s education, background, and affiliations are unknown; when information is cursory or incomplete; and when original publication and revision dates are lacking, those in search of evidence need to look elsewhere.12 

4. Avoid Using Information Sources of Dubious or Unknown Reliability

When information is conveyed without identification of its author, its publication or posting date, or its origin, or when it presents only one side of a multidimensional issue, relies heavily on anecdotes and/or opinions, neglects to include documentation or links to its own sources, cannot be corroborated by other references, or is clearly outdated, it is not appropriate to use as evidence and does not belong in any nurse’s reference list.13 

5. Avoid Using Sources That Demonstrate a Concerning Level of Potential Bias

Ideally, no commercial advertising related to content should be included. If advertisements are visible, these should be clearly labeled to distinguish them from other content. If such material is not clearly differentiated from other content (as in infomercials) or if management of a health problem recommends one specific therapy by name without consideration of other plausible alternatives, be especially vigilant for other indicators that information originates from a commercial or otherwise biased source.

6. Use Information From the Highest Level of Evidence Available

Numerous hierarchical systems have been developed to classify and rank health care information based on the level of evidence they provide and the quality of the work. The levels of evidence typically relate to a generally accepted hierarchy of research study designs that vary in the degree to which they control for bias. In this ranking, meta-analyses of randomized controlled trials are generally considered the least susceptible to bias, so are accorded the highest level, followed by less rigorously controlled studies, quasi-experimental studies, nonexperimental studies, and expert consensus and expert opinion. The American Association of Critical-Care Nurses’ 6 levels of evidence reflect such a system (see Table).14 

A multitude of systems for classifying health care studies that lack standardization in levels, definitions, purpose, or attributes exist. For example, the JBI’s FAME system employs 4 levels of evidence across 5 attributes (Feasibility, Appropriateness, Meaningfulness, Effectiveness, and Economic Evidence).15  Reviewing several of these systems16,17  and selecting one that works for your projects is important.

7. Use Sources That Demonstrate the Attributes of Good Research

Good research reflects a genuine desire to determine what is true. It respects the established procedures of science and describes these fully so others might replicate the study. It is careful in addressing relevant issues, adroit in integrating relationships among factors, clear in identifying results, and cautious in drawing conclusions.12 Figure 2 captures most of the attributes of good research and formats them as a checklist, which can be used to critique research reports. Keep in mind that the reason for completing such a critique is not just to appraise the quality of the study, but to determine whether the results are valid, reliable, and potentially applicable to your practice.

Critical care nurses who have limited or no experience in critically appraising the quality of research studies might enlist assistance from a colleague with that expertise to mentor their development or pursue learning opportunities such as the JBI Rapid Appraisal Protocol Internet Database User Guide18  to develop that capability.

8. Avoid Using Sources That Manifest the Indicators of “Bad Research”

It might seem obvious, but is worth mentioning that articles that display attributes of “bad research” should set off red flags that alert readers about faulty project design and/or methodologic omissions or transgressions. When these attributes are detected, further examination of the report is not warranted because their findings, conclusions, and related claims cannot be considered valid or reliable. Some of the signs of bad research include the following12 :

  • May start with a specious argument that because no random controlled trials (or mountain of research) have been completed on the subject, nothing is known about it.

  • Introduction starts with a (foregone) conclusion, followed by selected data plucked from its original context to support that conclusion.

  • Phrases research questions so these are directed toward specific conclusion(s).

  • Specific research questions are not stated explicitly anywhere in the report.

  • Design methodology is inconsistent with or inappropriate for the questions of interest.

  • Data analysis is inappropriate or faulty for the design, data, and/or comparison desired.

  • Methodologic limitations are minimally or not addressed.

  • The relative strength of associations between variables is not quantified, but described in only generic terms.

  • Associations between variables are mischaracterized or implied to be causal in nature.

  • Implications are overstated or lack support by data.

  • Ignores or neglects data and alternative interpretations that do not support the emphasized conclusion(s).

  • Data analysis, discussion, and/or conclusions include leaps in logic, spurious explanations, and one or more nonsequiturs.

  • References are dated and/or incomplete and/or drawn from sources outside the sphere of highly regarded, nonpartisan, peer-reviewed academic, health care, or professional organizations.

9. Avoid “Potholes” That Confound the Conduct of Good Research

Sources that demonstrate the 60 “methodological potholes” that have confounded the conduct of good research for centuries19  include use of “straw men” constructions that pose extreme positions on a topic, ad-hoc and post-hoc hypotheses, carry-over effect, control failure, contradiction blindness, data neglect, instrument decay, expectancy effect, order effect, over-generalization, regression artifacts, smorgasbord thinking, testing effect as well as various forms of bias: attrition bias, performance bias, detection bias, confirmation bias, cultural bias, egocentric bias, hindsight bias, reporting bias, selection bias or sampling bias.

10. Always Use the Most Authoritative Sources of Evidence Available

When searching for information on a specific topic, searches need to be tailored so they remain within the scope of that specialty area’s field and audience. For example, a search for information related to the criteria for a specific medical diagnosis needs to be drawn from medical sources authored by knowledgeable physicians, whereas a search for information related to drug administration or management of drug toxicity is best taken from pharmacology literature authored by expert clinical pharmacologists. Although members of the critical care team share and collaborate on many aspects of their responsibilities for patient care, in general, it is more appropriate to use medical references as citations for medical practice, nursing literature as references for nursing practice, etc. A few corollaries also apply here:

  • Textbooks and academic references written for students are not appropriate sources of information for experienced staff in those professions.

  • When citing study findings, the most valid and reliable evidence to cite is the primary or original source of that information—that is, research study that included that finding. Secondary sources are one step removed from the original, provided by different authors, and may reinterpret or mischaracterize the original finding, so are not equivalent to that source. Tertiary sources are 2 steps removed from the original and often appear many years later in compilations or summaries of findings from numerous studies. Of these, only primary sources represent evidence.20 

  • Generic dictionaries of the English language are not appropriate sources for definitions of medical, nursing, pharmacologic, or scientific terms.

  • Generic inquiry websites such as Wikipedia.com are not appropriate sources for academic papers, much less for research reports or papers intended for publication in a professional journal because neither the content nor related discussions at these sites is subject to editorial oversight using standards comparable to those used for peer-review of professional or scientific articles. In addition, content at such sites can be contributed by persons who lack any background, education, experience, or expertise in the topic area, so information may not be accurate, complete, or current.2123 

11. Whenever Possible, Get a Second (and Third) Opinion

Always seek concurrence on research findings from multiple sources rather than relying on any single source. Highly controversial or complex issues frequently require multiple sources to encompass all perspectives, but corroboration on all important aspects of evidence helps to ensure the validity and reliability of those findings to a much greater extent than any single set of findings could provide. Concurrence across both print and online sources of information increases the likelihood that at least some of those sources use peer-review or comparable means to minimize bias and ensure the integrity of that information.21 

12. Verify the Age of Findings So the Most Current Evidence Is Used

Although the recency of research findings has no necessary relationship to its accuracy or reliability, the age of information is important to ensure that it is not outdated. Some information becomes obsolete very rapidly, so it is important to know the time sensitivity of the topic being searched. Assuming that different investigators perform comparably thorough literature reviews before completing their study, a more recent study should encompass findings reported from older ones, so should reflect more informed evidence than older studies. Those assumptions are not always true, however, so careful notation of when the study data were actually collected and the timeframe spanned by the references cited in the study can be more revealing than the publication date for determining the true age of the findings.

Being a discriminating consumer of health care evidence requires a liberal and enduring sense of skepticism when perusing material that might influence nursing practice. As you read research reports, quality improvement project reports, case studies, or even overviews of clinical topics, challenge the source(s) and implications by asking: Who says so? On what basis does the author make this claim? Where did this information come from? Is this a statement of fact, an opinion, an inference, a deduction, an assumption, or merely an assertion? Did this study generate findings that support these conclusions? What statistical test results support that assertion?

As critical care nurses, we have an obligation to verify that health information is accurate, reliable, and current before we judge whether it applies to nursing practice, so consider it your duty to not take any health-related information at face value. Strive to be as discriminating and selective as possible in searching for and appraising the health care information you find. Your own life, those of your family members and your patients may depend on it. CCN

1
Cochrane Collaboration
. . Accessed June 7, 2013.
2
Cochrane Library
. . Accessed June 7, 2013.
3
Cochrane Nursing Care Field
. . Accessed June 7, 2013.
4
Joanna Briggs Institute
.
Affiliated Groups
. . Accessed June 5, 2013.
5
Cochrane
AL
.
Effectiveness and Efficiency: Random Reflections on Health Services
.
London, United Kingdom
:
Nuffield Provincial Hospitals Trust
;
1972
. . Accessed June 4, 2013.
6
Higgins
JPT
,
Green
S
, eds.
Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0
.
The Cochrane Collaboration
;
2011
. . Accessed June 10, 2013.
7
Alspach
JG
.
Readers: our inspiration and design driving force
.
Crit Care Nurse
.
2013
;
33
(
2
):
10
12
.
9
Consumer and Patient Health Information Section of the Medical Library Association (CAPHIS)
.
CAPHIS Top 100 For Health Professionals
. . Accessed June 7, 2013.
10
MedlinePlus Guide to Healthy Web Surfing
.
US National Library of Medicine and National Institutes of Health
,
Bethesda, MD
. Updated April 2012. . Accessed June 4, 2013.
11
Harris
R
.
Evaluating Internet Research Sources
.
Virtual Salt
,
11
-
20
-
2010
. . Accessed June 5, 2013.
12
Litman
T
.
Evaluating research quality: guidelines for scholarship
.
Victoria Transport Policy Institute
;
2012
. . Accessed June 7, 2013.
13
University of Sydney, College of Humanities & Social Sciences
.
The Write Site: Module 2 Unit 2: Evaluating Sources—Potentially Questionable Sources
. . Accessed June 7, 2013.
14
Armola
R
,
Bourgault
A
,
Halm
M
, et al
.
AACN levels of evidence: what’s new?
Crit Care Nurse
.
2009
;
29
(
4
):
70
73
.
15
Joanna Briggs Institute
.
Levels of evidence: FAME
. . Accessed June 5, 2013.
16
American Nurses Association
.
Research Toolkit: Appraising the Evidence
. . Accessed June 5, 2013.
17
Higgins
JP
,
Altman
DG
,
Gøtzsche
PC
, et al
.
The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials
.
BMJ
.
2011
;
343
:
d5928
. . Accessed June 5, 2013.
18
Joanna Briggs Institute
.
Rapid Appraisal Protocol Internet Database (RAPid) User Guide
.
Adelaide
,
Australia
;
2006
. . Accessed June 7, 2013.
19
Huron
D
.
Sixty Methodological Potholes
.
Columbus, OH
:
Ohio State University
. . Accessed June 5, 2013.
20
University of Maryland
.
University Libraries: Primary, Secondary and Tertiary Sources
. . Accessed June 7, 2013.
21
Forsyth
A
.
A guide for students preparing written theses, research papers, or planning projects
,
2011
. . Accessed June 7, 2013.
22
Wikipedia
.
About
. . Accessed June 7, 2013.
23
Wikipedia
.
Help: Editing
. . Accessed June 7, 2013.

Footnotes

PS. Please let me know if the guidelines provided are helpful in your work. If you have more guidelines to suggest, please send them to CCN at [email protected] so we can share them with CCN readers.