Measuring Performance on the ABCDEF Bundle During Interprofessional Rounds via a Nurse-Based Assessment Tool

This project was performed at 2 ICUs at UPMC, a tertiary referral center in Pittsburgh, Pennsylvania, and an academic affiliate of the University of Pittsburgh. One ICU has 12 beds and specializes in abdominal transplant patients, and the second ICU has 22 beds and specializes in surgical trauma patients. Both ICUs conduct multidisciplinary rounds that typically include some combination of the following provider types: attending intensivists, fellows, residents, critical care nurses, pharmacists, nutritionists, respiratory therapists, physical therapists, and students. The data for this project were collected as part of an ongoing effort to improve hospital performance; thus, the project was designated as a quality improvement initiative by UPMC’s Quality Improvement Review Committee (approval #1792). Additional contributions by D.L.M. were determined to be non–human subject research by the University of Pennsylvania institutional review board (#833938).

A paper-based assessment tool was developed for the observers to use when assessing the discussion of various evidence-based practices during rounds. The practices considered were those included in the ABCDEF bundle¹² : A = assess, prevent, and manage pain; B = both spontaneous awakening trial (SAT) and spontaneous breathing trial (SBT); C = choice of analgesia and sedation; D = delirium: assess, prevent, and manage; E = early mobility and exercise; and F = family engagement and empowerment.

To develop a tool that met our project needs, we convened an in-person development workshop to split each of the ABCDEF bundle elements into 1 or more binary (yes/no) statements representing whether that element (or portion of an element) was discussed during rounds. For example, element A (assess, prevent, and manage pain) was split into 3 nested questions: Was pain discussed? If so, was pain present? If so, was a plan for addressing pain made? We used an iterative consensus process to draft an initial version of the tool, followed by an in-person workshop where attendees used the draft tool to assess 2 transcribed discussions of patients. The tool was then revised on the basis of feedback from the workshop. The workshop was followed by a field pilot test where 2 nurses used the revised tool during rounds for 1 pilot observation day. Final revisions to the tool were made, resulting in the assessment tool shown in Figure 1.

For nested items, we created 6 rules to determine agreement and disagreement. Observers agreed when (1) both observers reached the item and selected yes; (2) both observers reached the item and selected no; (3) neither observer reached the item; or (4) 1 observer reached the item and selected no and the other observer did not reach the item. Conversely, observers disagreed when (5) both observers reached the item and disagreed on whether it was yes or no; or (6) 1 observer reached the item and selected yes and the other observer did not reach the item.

Two nurse observers (K.M.P. and J.B.S.) performed in-person observations of multidisciplinary morning rounds on 15 observation days in the fall of 2021. One unit and rounding team was chosen for observation each day, with no requirements made for the number or type of team members present. By design, 4 of the observation days occurred on the weekend, when rounds are usually shorter and involve fewer people. Handoff discussions were not observed. During an assessment, both nurse observers were instructed to watch and listen to the rounding discussion while independently completing the assessment tool. To isolate the content of the discussions, they were instructed not to look at electronic health record data or into patient rooms for visual cues (eg, delivery of invasive mechanical ventilation). To maintain independence of their responses, they were also asked to distance themselves from each other and not confer. All of a rounding team’s discussions during a data collection day were assessed except when the nurse observers were asked by the care team to not observe a discussion (eg, a difficult end-of-life discussion with the patient’s family), a discussion occurred in a patient’s room (rather than in the hallway), or either of the nurse observers had to leave the unit before rounds ended because of scheduling constraints.

On the basis of the measured reliability of our 2 nurse observers, we used generalizability theory to estimate what the reliability would be for each item if there were only 1 nurse observer. Generalizability theory holds that the observed variance in reliability can be divided into different component parts in order to make inferences about how reliability changes when altering 1 of the parts.¹⁴  This approach enables determination of the expected reliability with various numbers of hypothetical raters, assuming that each rater has similar training and skill.¹⁴  For example, generalizability theory has been used to determine how many radiologists would be needed to evaluate each image when creating a reliable reference standard for evaluating a machine learning model that extracts information from chest radiographs.¹⁸ 

We used a single-part crossed design (discussion× observer), enabling separation of the observed variance into 3 component parts: discussion (σ²_discussion), observer (σ²_observer), and residual (σ²_residual).¹⁴  The σ²_discussion is equal to the true variance. The sum of σ²_observer and σ²_residual equals the variance due to error. We then used σ²_discussion and σ²_residual to calculate the generalizability coefficient for a single observer (ρ₁).¹⁸  Consistent with past work, we considered a generalizability coefficient of 0.70 or greater to be sufficient support for having a single rater perform the assessments by themselves in the future.^18,19  In addition to the generalizability coefficient, we also calculated the Cohen κ statistic,²⁰  which is a more commonly reported reliability statistic, but it may underestimate the reliability of the observers.¹⁸ 

All statistical analyses were conducted in R.²¹  We describe patient characteristics using standard summary statistics based on data obtained from an existing registry of ICU patients. The Cohen κ and 95% CIs were calculated in R version 3.6.3 using the Kappa. test function from version 0.7.2 of the “fmsb” package. Generalizability coefficients were calculated by using R version 3.5.3 with the gstudy and dstudy functions from version 0.1.2 of the “gtheory” package. Confidence intervals for κ and the generalizability coefficient were produced using the fmsb package and a nonparametric bootstrap with 1000 repetitions,²²  respectively.

The characteristics of the ICU patients whose rounding discussions were observed during this project are shown in Table 1. In total, 53 different patients were observed, with 33 of them receiving invasive mechanical ventilation. The patients were split almost evenly between the surgical trauma and transplant ICUs. Because ICU admissions often last for multiple days, many patients had multiple discussions observed. In total, the nurse observers used the assessment tool when assessing 118 patient discussions. These dually observed discussions are the basis for calculating reliability and agreement.

The 17 items on the assessment tool are each related to 1 or more of the elements of the ABCDEF bundle. Each item’s reliability is shown in Table 2, along with an estimate of the number of observers needed to reach the targeted generalizability coefficient, 0.70. Eleven items (65%) had single-observer reliability (defined as generalizability coefficient per observer ≥0.70), and 14 items (82%) had a substantial agreement (defined as κ ≥ 0.61).²⁰  The reliable items have broad coverage across the bundle’s elements, with the partial exception of “B: Both SAT and SBT” because data showed that the generalizability coefficient for a single observer of the “SAT discussed” item was less than our predefined threshold. This means that a hypothetical observer with similar training and experience who is using this assessment tool cannot reliably assess whether an SAT was discussed (ie, this item is unacceptably inconsistent among different hypothetical observers).

The distribution of responses across all items is shown in Figure 2. Although agreement occurred for the majority of responses, the most common response type (both observers = yes or both observers = no) varied across items. For example, items involving pain were often discussed, whereas “agitation score discussed” was almost never answered yes. The conditional branching on the assessment tool is also clearly represented in the data, where some items were applicable only if the preceding item was responded to with a particular answer. For example, the item “pain present” was applicable only if the parent item “pain discussed” was answered yes.

We found substantial agreement between 2 independent nurses as to whether elements of an ABCDEF bundle rounding checklist were addressed during rounds. These results suggest that a single independent observer, with similar training and experience, of multidisciplinary ICU rounding discussions can reliably assess adherence to an ABCDEF bundle rounding checklist. Our findings have several important implications for performance improvement and quality measurement in the ICU.

First, our results provide supporting evidence that nurses can identify when a rounding checklist element has been addressed and, therefore, might not need to be repeated during a readout of the checklist. This added flexibility enables a shorter, more precise checklist for each patient, which might make using a checklist for every patient more palatable to the team.⁶ 

Second, as with empowering the bedside nurse to perform checklist customizations, our results show that critical care nurses are ideal candidates to be independent checklist prompters during rounds.⁹  Although some quality improvement projects that have included prompts for evidence-based practices have found reductions in mortality,¹⁰  others have not.²³  The difference may be due in part to how prompting was implemented: during rounds versus later in the day.²⁴ 

Third, our results support a novel approach for measuring and improving performance with the ABCDEF bundle via audit and feedback.²  The assessment tool we created can be used as the basis for occasional strategic use of an independent nurse observer to measure team performance. Similar tools may also be useful when measuring team performance during other events where team communication is essential, such as emergency response and bedside shift handoff.

The generalizability of these results is limited by a few factors. First, the nurse observers were involved in the creation of the assessment tool. Such involvement may result in them having more insight into how to apply the tool than an outside nurse. Therefore, additional work may be needed to understand the need for user training and the validity of application in practice. This issue is particularly important because conclusions drawn from generalizability theory assume that observers have similar training and experience.¹⁴  These concerns were partly countered by the clarifying instructions that the observers added to the assessment tool after the field pilot testing. Furthermore, the observers were instructed not to discuss the tool with each other anymore after it was finalized.

Second, although the items on the assessment tool were inspired by the ABCDEF bundle, our assessment tool divides the 6 bundle elements into 17 items that were relevant to our local needs. Using only the 6-element bundle as the assessment tool was considered but rejected because doing so would reduce the granularity of the data collected. Future users of this tool or similar tools should give careful thought to the information desired when selecting items for inclusion. This could mean including additional items, such as infection control measures,^25,26  that are not part of the ABCDEF bundle.

Third, only rounding discussions that occurred in the hallway were observed. Owing to both institutional norms and the presence of COVID-19, most rounding discussions occurred in the hallway rather than the patient rooms, but the results may differ for units with different rounding practices.

A final consideration is that the units observed in this project were heterogeneous in their use of rounding checklists. One unit uses a custom rounding checklist that includes the ABCDEF bundle elements plus 9 additional elements. The other unit has a unitwide, nurse-led evidence-based practice checklist, but it is not typically referenced during multidisciplinary rounds. Our sample size did not enable subanalyses with sufficient power to be conducted on each unit; however, the diversity of rounding practices can be seen as a strength.

Using a paper-based assessment tool, 1 trained ICU nurse can reliably assess the discussion of elements of the ABCDEF bundle during multidisciplinary rounds. The presence of a few less-reliable items, such as “SAT discussed,” indicates room for improving the assessment tool and a need to further investigate how evidence-based practices are conceptualized among members of the multidisciplinary ICU team.