All ICUs Are Not Created Equal: Evaluating Pilot Studies Performed in Different Environments

This variability challenges us as critical readers trying to interpret pilot studies. When we read another hospital’s study, we often find ourselves asking questions and struggling with the multitude of ICU organizational varieties that have developed in the United States and in other countries. Because manuscript descriptions are not uniform, it may be difficult or even impossible to assume much about the day-to-day activity of the reporting ICU with respect to its materials, patients, and outcomes.

Unfortunately, journal editors have not made this situation any easier. Authors preparing pilot study manuscripts will quickly discover that there is no standardization across journals regarding how to report the structure of the ICU or facility in which the research was performed. What usually happens, it seems, is that, in dialogue with one another, reviewers and editors advise authors about where further elaboration of the ICU environment will aid readers in their interpretation of the study’s relevance and value. Authors who comply, offering more details about the environment in which they work, do readers of the published study a tremendous service.

How much variability in ICU structure and organization is there worldwide? Consider the variation in professional staffing models. In ICUs in the United States, a 1:2 nurse to patient ratio predominates, whereas in many parts of Europe a 1:1 ratio is standard.¹ A distinct professional role for a respiratory therapist is commonly found in ICUs in the United States. In other countries, a nurse or “physiotherapist” may handle the types of care typically delivered by a US respiratory therapist. The physiotherapist also has responsibilities that are similar to the assessment and delivery of hands-on therapies performed by US physical therapists.

Variations in ICU staffing do not end there: other differences might include management personnel and physicians. The impact of hospital funding is another important consideration. This facet could be measured in many ways. One important parameter could be the organization’s ratio of overall employees to adult patients. Standards have not been clearly established across continents; in fact, there is considerable controversy and a wide spectrum of practice for ICU-related medical professionals and hospital employees.

With respect to ICU reporting, patient acuity is an area in which some standardization does exist. Patient acuity scoring tools, such as the Therapeutic Intervention Scoring System, the Acute Physiology and Chronic Health Evaluation, and the Sequential Organ Failure Assessment Score, help readers better understand the patient population receiving care by any 2 ICUs in the world. However, the patient acuity score may account for only a small part of the outcome variability reported in a single ICU’s pilot study. Socioeconomic factors, such as patients’ attitudes toward seeking medical care from a hospital sufficiently large to maintain an ICU, may vary from hospital to hospital, city to city, or country to country. In light of these variables, including the risk of underreporting, we must be cautious about planning interventions for our own ICUs based on pilot data from São Paulo, Brazil; Seoul, South Korea; Wellington, New Zealand; Barcelona, Spain; Montreal, Canada—even from an ICU across town.

Health professionals involved in comparisons of human drug therapies or care interventions are sometimes called “clinical trialists.” Authors of pilot studies should be seen as clinical trialists as well, because these investigators seek to answer questions about day-to-day care.

Clinical trialists begin the research process either by observing that there is a practice variation or by recognizing that a new intervention is available that lacks sufficient or current evidence about benefits and risks. The clinical trialist can employ a tool called the randomized controlled trial (RCT) to establish reliable data about those benefits and risks. Use of RCTs is one means for reducing confounding factors that may bias the outcome in smaller studies. A real strength of the multicenter RCT is that many hospitals participate by enrolling patients, resulting in larger sample sizes that theoretically offset or even negate hospital-to-hospital variation that exists between centers.

Many RCTs are based on data and findings first established in pilot studies. Pilot studies whose interventions are clearly described are more useful to clinicians and researchers. Equally important, as mentioned earlier, is a clear description of the ICU structure in which the intervention was carried out; ICU professionals who wish to duplicate the intervention are aided by such information. A problem for authors, however, is that no consensus exists about what to include in such a standardized report.

After they’ve read the methods section in a clinical study, readers need to know what differences there might be in the delivery of care between the “reporting” hospital and other hospitals. But no organized criteria exist for such an administrative description. Several have been suggested, however, including (1) reporting the compliance rate to internal ICU or hospital policies, (2) reporting how many daily awakenings are performed, (3) asking if patients with acute respiratory distress syndrome were managed with a low-tidal volume that was calculated after the patient’s height was measured, (4) asking if the charge nurse cares for patients directly while supervising the entire unit, (5) asking what physicians are in-house at night, and so on.

In RCTs, some types of variability are addressed under the auspices of an institutional review board (IRB). An IRB is formally charged with approving, monitoring, and reviewing biomedical and behavioral research involving human subjects to protect those subjects’ rights and guard their welfare. Interestingly, there have been calls for centralized IRBs for multicenter RCTs to reduce variability. Nevertheless, the argument most frequently used to support the local IRB structure is that “local” is the important part of the formula. Nuances of patient behavior or cultural attributes would not be obvious to a central IRB and probably are handled more effectively by local IRBs. Although local or centralized IRBs provide improvement and uniformity about the ethical conduct of a trial, no such centralized oversight body or guidelines exist for reporting ICU structural details.

Through their published instructions for authors, medical and nursing journals often provide helpful suggestions for pilot study and RCT structure, offering tips for manuscript development. In addition, the Web site of the International Committee of Medical Journal Editors (http://www.icmje.org) can be helpful. Useful information is also available at the CONSORT (Consolidated Standards of Reporting Trials) Web site, http://www.consortstatement.org/index.aspx?o=1011. At the site there is a 22-item checklist, though the level of ICU detail is not stipulated per se. With respect to reporting information about ICU structure in research, the site notes, “This description should provide enough information that readers can judge whether the results of the trial are relevant to their own setting.”

These sites represent quick sources of solid information that have led to improvements in trial reporting and ethical behavior. The problem is that they leave the description of ICU detail reporting to future documents.

Large multicenter trials about ICU practices and interventions will provide insight into geographic differences in disease epidemiology, help to describe regional practice variability, and enroll large numbers of patients in a brief amount of time compared with single-site studies. The future will bring additional improvement in international standards for optimal clinical trial structure, ethical participation, and transparent analyses. Author guidelines that call for more specificity about the sites in which research was conducted, especially if that research takes the form of a pilot study, would be welcomed by investigators and readers. Information on ICU structure and future clarification of ICU reporting guidelines will help to speed acceptance of new ideas and will allow ICU staff to feel more confident in their interpretation of the literature. Achieving such goals may be one way to accelerate the adoption of new care practices and to accomplish more rapid improvements in related ICU outcomes.