Background

Patients who receive prolonged endotracheal intubation (> 48 hours) are at risk for dysphagia. Nurses should conduct swallowing assessments after extubation because of the high likelihood of aspiration pneumonia developing. No valid and reliable postextubation dysphagia screening tools are available.

Objectives

To establish content validity, analyze interrater reliability, and determine sensitivity and specificity of an evidence-based postextubation dysphagia screening tool developed by a multidisciplinary team.

Methods

A prospective nonexperimental study was conducted in 4 medical-surgical intensive care units in 4 hospitals. The study was conducted in 3 phases: (1) establishing content validity with clinical experts who participated in a Delphi survey, (2) establishing inter-rater reliability by agreement with nurses who simultaneously and independently completed the tool, and (3) establishing sensitivity and specificity with speech language pathologists and nurses who independently and blindly completed the tool for eligible patients.

Results

Individual item scores were > 0.82 and the overall content validity index was 0.93, indicating content validity. Interrater reliability was established (Cohen κ = 0.92). In 66 eligible patients, the prevalence of postextubation dysphagia was 56%, sensitivity of the postextubation dysphagia screening tool was 81%, and specificity was 69%.

Conclusion

The reliability and validity of a postextubation dysphagia screening tool that can help nurses determine an extubated patient’s ability to swallow after prolonged endotracheal intubation were established.

Notice to CE enrollees

Notice to CE enrollees

This article has been designated for CE contact hour(s). The evaluation demonstrates your knowledge of the following objectives:

  1. Compare and contrast dysphagia screening and dysphagia evaluation.

  2. Identify 4 criteria that constitute a good dysphagia screening tool.

  3. Describe the sensitivity and specificity of the postextubation dysphagia screening tool.

To complete the evaluation for CE contact hour(s) for this article #A1827021, visit www.ajcconline.org and click the “CE Articles” button. No CE evaluation fee for AACN members. This expires on March 1, 2021.

The American Association of Critical-Care Nurses is an accredited provider of continuing nursing education by the American Nurses Credentialing Center’s Commission on Accreditation. AACN has been approved as a provider of continuing education in nursing by the State Boards of Registered Nursing of California (#01036) and Louisiana (#LSBN12).

Endotracheal intubation (ETI) to support mechanical ventilation is often required for critically ill patients. ETI is a life-sustaining technique; however, complications, including dysphagia, can manifest after extubation. Postextubation dysphagia (PED) is attributed to mechanical and cognitive factors that interfere with the ability to execute an efficient and safe swallow. The incidence of PED in mixed populations of medical-surgical patients is from 3% to 62%.1  Patients who receive prolonged ETI, defined as longer than 48 hours, have the highest frequency of dysphagia compared with other diagnostic subtypes.1  One report2  revealed that the risk of PED developing on the first day of ETI was 25% and the risk doubled after 2 days of ETI. Studies2,3  have shown that prolonged ETI is an independent predictor of PED.

The act of swallowing is complex: It involves sensory and motor nerves, more than 30 muscle groups, 2 brainstem centers, occurs in 4 phases, and is voluntary and involuntary.4  PED can develop through several mechanisms, including injury from ETI,5  muscle weakness,6  dysfunctional oropharyngeal or laryngeal sensation,7  impaired sensorium,8  gastroesophageal reflux,9  or dyssynchronous breathing and swallowing.7  Complications of PED include dehydration, malnutrition, aspiration of oral secretions, and aspiration pneumonia.1  These consequences are associated with poor outcomes and high financial cost owing to longer stays in the intensive care unit and additional medical costs related to the need for antibiotics and chest radiographs.4,10  The annual cost in the United States for PED is estimated to be more than $500 million.11 

Swallowing should be assessed in recently extubated patients who were intubated > 48 hours.

Given the likelihood of serious medical complications of PED and the associated high costs, Skoretz et al,1  in a systematic review, recommended that a swallowing evaluation be conducted on all patients who have received prolonged ETI. The American Association of Critical-Care Nurses issued a Practice Alert for the prevention of aspiration in adults.12  One of the 7 recommendations made in the Practice Alert was to “consult with [a] provider about obtaining a swallowing evaluation before oral feedings are started for recently extubated patients who had been intubated for more than 2 days.”12(pe20)

Donovan et al13  stress the importance of differentiating between dysphagia screening and dysphagia evaluation (clinical or instrumental). Dysphagia screening is defined as a “pass/fail procedure to identify individuals who require a comprehensive referral to other professional and/or medical services.”14(p10) Screening procedures identify patients who need a complete dysphagia evaluation, including a clinical evaluation of swallowing mechanisms and function that uses different food and liquid consistencies.13 

Speech language pathologists (SLPs) perform evaluations to diagnose and manage dysphagia. Because SLPs may be available only during standard weekday working hours, the definitive diagnosis by SLPs may not be available for 24 to 48 hours after ETI. In these situations, nurses have to provide initial screening for PED. If the initial PED screening determines the patient is at risk for dysphagia, the patient is allowed nothing by mouth until the SLP can complete a diagnostic evaluation.15  With nurses available 24/7, enhancing their skills in identifying dysphagia is logical and necessary.16 

Evidence indicates that nurses can successfully complete dysphagia screenings17,18  using valid and reliable dysphagia screening tools for patients who have had a stroke,15,18,19  medical patients in acute care,16  and patients with thermal burn injuries.20  To our knowledge, no valid and reliable screening tool is available for nurses to use to assess for PED. Dysphagia screening tools that have been developed for a specific population of patients cannot be used with other populations because the pathophysiological mechanisms of dysphagia are different.16  The result is that patients who are extubated after prolonged ETI either are allowed oral intake with no dysphagia screening or are still allowed nothing by mouth for up to 48 hours pending SLP evaluation. Thus, a PED screening tool for nurses is needed.

Using a prospective nonexperimental design, we evaluated the validity, reliability, and value of a PED screening tool for nurses. The specific aims of the study were to (1) assess content validity of the postextubation dysphagia screening (PEDS) tool; (2) analyze the interrater reliability of the PEDS tool; and (3) establish accuracy and overall value of the PEDS tool by determining its sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV).

Methods

The study was approved by the institutional review board at Banner Health and conducted according to ethical standards set forth in the Helsinki Declaration of 1975. The study took place in 4 medical-surgical intensive care units in 4 Banner Health medical centers located in Phoenix, Arizona.

Development of the PEDS Tool

To develop our PEDS tool, we began with our health system’s existing dysphagia screening tool for patients who have had a stroke. This tool was developed by a multidisciplinary team from our system and had been in use for more than 5 years. The team that developed the tool reported a sensitivity of 64%, specificity of 99%, PPV of 92%, and NPV of 93%. The screening tool consists of 5 sections of nursing assessments: SLP evaluation, level of alertness, symptoms and tubes, new-onset aspiration symptoms, and contraindications for trial feedings. Based on those assessments, nurses are directed to take subsequent actions.

A review of the literature revealed different pathophysiological mechanisms and risk factors for dysphasia after ETI versus after stroke. Therefore, we modified 2 sections of the stroke dysphagia screening tool to reflect risk factors for dysphagia after ETI. We replaced new-onset aspiration symptoms with a section titled “Assessment of respiratory status” and modified the tubes and symptoms section to reflect criteria specifically relevant to patients who are extubated after prolonged ETI.

Published evidence supports assessment of respiratory factors, including tachypnea, and indicates that inability to maintain adequate oxygen saturation without oxygen therapy can contribute to PED.1,10,21  Accordingly, we modified the existing tool to include the following: (1) ability to remain without continuous airway pressure or bilevel positive airway pressure for at least 15 minutes, (2) ability to maintain oxygen saturation without a nonrebreather or Venturi mask for at least 15 minutes (oxygen saturation measured by pulse oximeter remains > 90% and/or does not decrease > 10% from baseline), and (3) patient’s respiratory rate is less than 30/min. If the patient does not meet any one of these criteria, the nurse stops the assessment and reassesses the patient in 24 hours or when respiratory status improves. If the answer is yes to all 3 questions, the nurse is directed to proceed to the assessments in the next sections.

In the final section, the nurse confirms that the patient has no contraindications for trial feedings. Once that is confirmed, the nurse is directed to the oral intake trial. Note that the trial takes place only if the patient has an order from the provider for oral intake. The oral intake trial is based on the gold standard for screening: the 3-oz (90-mL) water swallow test. This test has been used by members of various disciplines for screening in more than 3000 patients, including critically ill patients with heterogeneous diagnoses, and has been validated against fiberoptic endoscopic evaluation.2224  The screening is rated as pass or fail, with a failure occurring if any screening item is scored as yes. Patients who do not pass the screening are allowed nothing by mouth until SLP evaluation.

Different populations of patients need different dysphagia screening tools because the pathophysiological mechanisms of dysphagia differ.

To establish validity and reliability of our PEDS tool, we conducted a study in 3 phases: phase 1 established content validity, phase 2 established interrater reliability, and phase 3 established sensitivity, specificity, PPV, and NPV.

Phase 1: Establish Content Validity

We recruited 16 clinical experts within our system who provide care regularly for patients with PED to serve as content experts. Experts included intensive care unit nurses, stroke coordinators, intensivists, and SLPs. The experts were sent an email invitation to complete 2 rounds of Delphi surveys. The survey asked them to rate the relevance of each section of the PEDS tool on a 4-point scale, as follows: 4 (very relevant with no revision needed), 3 (relevant with minor revision needed), 2 (unable to assess relevance without revision), and 1 (not relevant). They were asked if each section was clearly written and, if it was not, were asked to provide suggestions to improve the clarity. They were asked, “Are there any additional items you think are relevant to include for the next round?” Expert panelist responses, statistics, and summaries were distributed to the panelists after each round. Each round was open for 2 weeks. After suggestions were incorporated into the PEDS tool, the modified tool was sent to experts for additional review.

The PEDS tool has a sensitivity of 81% and a specificity of 69%.

Based on consensus, items were retained or modified. We analyzed the relevance of items using means and percentage agreement. Consensus for retaining an item was satisfied by a percentage agreement greater than 70% (ie, ratings of 3 or 4) or a mean response of at least 3.0. A content validity index (CVI), as outlined by Lynn,25  was calculated for each item and for the whole scale with a goal of attaining a CVI of 0.80.

Respondents (n = 11) to round 1 of the Delphi survey were 3 intensive care unit clinical nurses, 4 intensivists, 3 nurse stroke coordinators, and 1 SLP. There was high agreement that all items were highly relevant for clinical nurses to screen for PED. The overall content validity index in round 1 was 0.92. Based on the respondents’ feedback, 2 items were modified to provide greater clarity: “Patient is awake, alert, and able to follow commands” was changed to “Patient is awake, alert, and able to follow simple commands.” And “Patient has a feeding tube” was changed to “Patient has an oral-gastric, nasal-gastric, and/or surgically placed tube (including percutaneous endoscopic gastrostomy).”

The PEDS tool reduces the risk of a patient starting oral intake inappropriately or unsafely after prolonged intubation.

The 7 experts who responded to round 2 of the Delphi survey included 2 intensive care unit clinical nurses, 2 intensivists, 2 stroke coordinators, and 1 SLP. Item CVIs with expert ratings of 3 to 4 were all greater than 0.82, with an overall CVI of 0.93. No additional Delphi survey rounds were conducted, because content validity of the PEDS tool was established (see Figure).

Phase 2: Establish Interrater Reliability

To establish interrater reliability, a nurse member of the research team and a clinical nurse simultaneously and independently assessed an eligible patient (Table 1) for dysphagia using the PEDS tool. Interrater reliability was verified by agreement between these 2 nurses who independently and simultaneously (within 5 minutes) completed the PEDS tool. The Cohen g statistic was used to measure interrater agreement for categorical items (ie, dysphagia present or not present).

Phase 3: Establish Sensitivity, Specificity, PPV, and NPV

A clinical nurse and SLP member of the research team independently completed a dysphagia assessment on eligible patients within 24 hours of extubation (Table 1). They were blind to the results of each other’s independent assessments. The SLP dysphagia evaluation was performed by certified SLPs within 16 hours of the nurse’s assessment, using a standardized swallowing evaluation including a physical assessment with cognitive screening, a complete oral motor examination, and an assessment of oral and pharyngeal muscle function. Initial swallowing trials were performed using water and crackers to allow for assessment of airway safety during feeding and success of strategies in feeding.

A 2 × 2 contingency table was used to calculate sensitivity, specificity, PPV, and NPV.

Results

Interrater Reliability

A nurse member of the research team and a clinical nurse simultaneously and independently assessed 25 eligible patients for PED using the PEDS tool. The Cohen κ was +0.92.

Sensitivity, Specificity, PPV, and NPV

A total of 66 patients were evaluated by a clinical nurse and an SLP for dysphagia. Most of the patients in the sample were male (73%), with a mean age of 59.4 (SD, 14.8; range, 26–87) years. More than half of the patients (52%) had a primary diagnosis of acute respiratory failure, followed by sepsis or septic shock (15%; Table 2). More than half of the sample (56%; n = 37) had dysphagia.

The 2 × 2 contingency table (Table 3) revealed a sensitivity of 81%, specificity of 69%, PPV of 77%, and NPV of 74%.

Discussion

This study was designed to evaluate a screening tool for nurses to assess patients at risk for PED after prolonged ETI (defined as longer than 48 hours). A multidisciplinary team modified an existing dysphagia screening tool used by nurses to assess patients after a stroke. The modified tool includes evidence-based assessment criteria specifically relevant to patients who are extubated after prolonged ETI.

Content validity was established in 2 rounds of a Delphi survey with content experts. The number of rounds in the modified Delphi method can be as few as 2 if the experts achieve consensus.26  For a scale to be judged as having excellent content validity, Lynn25  recommended it should be composed of items that have item CVIs greater than 0.78, with an overall average CVI greater than 0.90. The PEDS tool item CVIs were all greater than 0.82, and overall CVI was 0.93.

Interrater reliability was established with a Cohen κ of +0.92. Cohen κ is a measure of the difference between observed agreement and expected agreement, standardized to a −1 to +1 scale, where perfect agreement is 1; 0 is what would be expected by chance, and negative values indicate less than chance.27  Values greater than 0.75 represent very good agreement.28 

In phase 3 of this study, the prevalence of dysphagia was 56%. This prevalence is within the range reported by others in mixed medical-surgical patient populations (3%–62%).1  These results provide additional evidence that patients who have received prolonged ETI for more than 48 hours are at risk of having PED develop.

Sensitivity and specificity are important criteria to evaluate screening tools. The PEDS tool identified the majority of patients who were determined by the SLP evaluation to have PED and identified the majority of patients who did not have PED. The necessity for more extensive evaluations and their associated costs can be avoided in patients with negative findings.29 

Sensitivity is the ability of a tool to identify a case correctly; in this study, this meant to screen “in” PED when PED actually existed. Sensitivity measures “true positives,” or the proportion of those patients who screened positive for PED by the nurse and who later had PED diagnosed by the SLP. It is important for screening instruments to have a high sensitivity so patients with PED are not missed by the nursing assessment. The sensitivity of the PEDS tool is consistent with the sensitivity of other dysphagia screening tools reported in the literature (29%–90%) that were validated in other populations of patients.13,15,16,1820,30,31 

Specificity is the ability of a screening tool to screen out those patients who do not have the condition (ie, the “true negatives”). In our study, this was the proportion of patients who screened negative for PED by the nurse and who were later evaluated by the SLP and found not to have dysphagia. The specificity of the PEDS tool is consistent with the specificity of other dysphagia screening tools reported in the literature (52%–90%) that were validated in other populations of patients.13,15,16,1820,30,31 

Emphasis for dysphagia screening tools should be on high sensitivity, not high specificity; specificity ranging from 50% to 90% is acceptable.16  A good screening tool needs to have a high sensitivity and a high negative predictive value.32  The PEDS tool meets these criteria with a sensitivity of 81% and an NPV of 74%, which are consistent with published data from other studies on dysphagia screening tools for patients after stroke.13,15,16,1820,30,31  High sensitivity is especially important in conditions such as dysphagia in which the risk of being undetected can lead to the serious consequences of pneumonia, malnutrition, and death.29 

Donovan et al13  identified criteria that constitute a good dysphagia screening tool: (1) The tool should evaluate dysphagia risk, suitability for oral feedings, and the need for further evaluation by an SLP (content validity); (2) various people should be able to administer the screening with similar results (interrater reliability); (3) the screening tool should identify patients at risk of dysphagia (sensitivity); and (4) the tool should rule out patients who are not at risk for dysphagia (specificity). The PEDS tool met these criteria.

Implications

The information that nurses need to complete the PEDS tool is gathered quickly, allowing swift identification of PED and timely referral to SLP care. One strength of the PEDS tool is that it screens for PED rather than aspiration and, as such, reduces the risk of a patient commencing oral intake inappropriately or unsafely after prolonged ETI. The PEDS tool is a swallow screening tool for nurses and does not replace a complete swallowing evaluation by an SLP.

When the nurse uses the PEDS tool and the assessment reveals that the patient is not at risk for PED, oral intake can be initiated and patients do not have to wait for an SLP evaluation. This initial risk assessment by nurses is especially advantageous during nights and weekends when SLPs may not be available and allows patients to eat and drink as soon as clinically able.

Limitations

The small sample size of this study may limit generalizability of the findings to other acute care settings. Although the study was conducted in 4 medical-surgical intensive care units in 4 medical centers in 1 large city, the findings may not be generalizable to other locations with different organizational and personnel characteristics. Additional well-designed controlled studies are needed to support use of PEDS tool in other health care settings.

Although it is logical to assume that PED screening may decrease patients’ hospital length of stay and complications, we did not measure these outcomes. Further research with larger sample sizes is needed to analyze the effect of the use of the PEDS tool on outcomes and health economic benefits, including length of stay, costs associated with dysphagia evaluation, and aspiration pneumonia rates.

Conclusion

This study established the content validity and interrater reliability of the PEDS tool. Its high sensitivity combined with the high negative predictive value demonstrate that the PEDS tool is a valid screening tool for nurses to use to assess for PED in patients who have received prolonged ETI.

Acknowledgments

This study was performed at Banner University Medical Center – Phoenix, Arizona; Banner Estrella Medical Center, Phoenix, Arizona; Banner Baywood Medical Center Mesa, Arizona; and Banner Desert Medical Center, Mesa, Arizona. We thank members of the research team: Robechard Halaghay, rn, msn (Banner Estrella Medical Center); Sara Nichols, ms, ccc-slp, Barbara Janine Crandall-Litvak, rn, bsn, rd, ccrn, ocn, Amy Merril, rn, ccrn (Banner Desert Medical Center); Timothy Jackson, rn, bsn, Lorena Bachelier, rn, bsn (Banner Baywood Medical Center); and Casey Thompson, rn, ba, bsn, ccrn, Christine M. Meyer, ms, ccc-slp, and Jaclyn Knoll, ms, ccc-slp (Banner University Medical Center Phoenix).

REFERENCES

REFERENCES
1
Skoretz
SA
,
Flowers
HL
,
Martino
R
.
The incidence of dysphagia following endotracheal intubation: a systematic review
.
Chest
.
2010
;
137
(
3
):
665
673
.
2
Kwok
AM
,
Davis
JW
,
Cagle
KM
,
Sue
LP
,
Kaups
KL
.
Postextubation dysphagia in trauma patients: its hard to swallow
.
Am J Surg
.
2013
;
206
(
6
):
924
927
.
3
Barker
J
,
Martino
R
,
Reichardt
B
,
Hickey
EJ
,
Ralph-Edwards
A
.
Incidence and impact of dysphagia in patients receiving prolonged endotracheal intubation after cardiac surgery
.
Can J Surg
.
2009
;
52
(
2
):
119
124
.
4
Macht
M
,
King
C
,
Wimbish
T
, et al
.
Post-extubation dysphagia is associated with longer hospitalization in survivors of critical illness with neurological impairment
.
Crit Care
.
2013
;
17
:
R119
.
5
Stauffer
J
,
Olson
D
,
Petty
T
.
Complications and consequences of endotracheal intubation and tracheotomy: a prospective study of 150 critically ill adult patients
.
Am J Med
.
1981
;
70
:
65
76
.
6
Devita
MA
,
Spierer-Rundback
L
.
Swallowing disorders in patients with prolonged orotracheal intubation or tracheostomy tubes
.
Crit Care Med
.
1990
;
18
(
12
):
1328
1330
.
7
Hermans
G
,
De Jonghe
B
,
Bruninck
,
Van Den Berghe
G
.
Critical illness polyneuropathy and myopathy
.
Crit Care
.
2008
;
12
(
6
):
238
250
.
8
Leder
S
,
Suiter
D
,
Lisitiano
W
.
Answering orientation questions and following single-step verbal commands: effect on aspiration status
.
Dysphagia
.
2009
;
24
(
3
):
290
295
.
9
Methany
N
,
Clouse
R
,
Chang
Y
,
Stewart
B
,
Oliver
D
,
Kollef
M
.
Tracheobronchial aspiration of gastric contents in critically ill tube-fed patients: frequency, outcomes and risk factors
.
Crit Care Med
.
2006
;
15
:
360
369
.
10
Macht
M
,
Wimbish
T
,
Clark
BB
, et al
.
Postextubation dysphagia is persistent and associated with poor outcomes in survivors of critical illness
.
Crit Care
.
2011
;
15
:
R231
.
11
Altman
KW
,
Wimbish
T
,
Clark
BJ
.
Consequence of dysphagia in the hospitalized patient: impact on prognosis and hospital resources
.
Arch Otolaryngol Head Neck Surg
.
2010
;
136
:
2756
2789
.
12
American Association of Critical Care Nurses
.
Practice alert: Prevention of aspiration in adults
.
Crit Care Nurse
.
2016
;
36
(
1
):
e20
e24
.
13
Donovan
NK
,
Daniels
SK
,
Edmiaston
J
,
Weinhardt
J
,
Summers
D
,
Mitchell
PH
,
on behalf of the AHA Council on Cardiovascular Nursing and Stroke
.
Dysphagia screening: state of the art
.
Stroke
.
2013
;
44
:
e24
e31
.
14
American Speech Language Hearing Association
.
Preferred Practice Patterns of the Profession of Speech Language Pathology
.
2004
. . Accessed November 26, 2017.
15
Cummings
J
,
Soomans
D
,
O’Laughlin
J
, et al
.
Sensitivity and specificity of a nurse dysphagia screen in stroke patients
.
Medsurg Nurs
.
2015
;
24
(
4
):
219
263
.
16
Cichero
J
,
Heaton
S
,
Bassett
L
.
Triaging dysphagia: nurse screening for dysphagia in an acute hospital
.
J Clin Nurs
.
2009
;
18
:
1649
1659
.
17
Perry
J
.
Screening swallowing function of patients with acute stroke: detailed evaluation of the tool used by nurses
.
J Clin Nurs
.
2001
;
10
:
474
481
.
18
Edmiaston
J
,
Connor
LT
,
Loehr
L
,
Nassief
A
.
Validation of a dysphagia screening tool in acute stroke patients
.
Am J Crit Care
.
2010
;
19
(
4
):
357
364
.
19
Bravata
DM
,
Dagget
VS
,
Woodward-Hagg
H
, et al
.
Comparison of two approaches to screen for dysphagia among acute ischemic stroke patients: nursing admission screening tool versus National Institutes of Health Stroke Scale
.
J Rehabil Res Dev
.
2009
;
46
(
9
):
1127
1134
.
20
Rumbach
A
,
Ward
E
,
Heaton
S
,
Bassett
L
,
Webster
A
,
Muller
M
.
Validation of predictive factors of dysphagia risk following thermal burns: a prospective cohort study
.
Burns
.
2014
;
40
(
4
):
744
750
.
21
Starks
B
,
Herbert
C
.
Aspiration prevention protocol: decreasing postoperative pneumonia in heart surgery patients
.
Crit Care Nurs
.
2011
;
31
(
5
):
38
45
.
22
DePippo
KL
,
Holas
MA
,
Reding
MJ
.
Validation of the 3 ounce water swallow screen test for aspiration following stroke
.
Arch Neurol
.
1992
;
49
:
1259
1261
.
23
Suiter
DM
,
Leder
SB
.
Clinical utility of the 3 ounce water swallow test
.
Dysphagia
.
2008
;
23
:
244
250
.
24
Leder
S
,
Suiter
D
,
Warner
H
,
Kaplan
L
.
Initiating safe oral feeding in critically ill intensive care unit and step down unit patients based on a 3 ounce water swallow challenge
.
J Trauma
.
2011
;
70
(
5
):
1203
1207
.
25
Lynn
M
.
Determination and quantification of content validity
.
Nurs Res
.
1986
;
35
(
6
):
382
385
.
26
Keeney
S
,
Hasson
F
,
McKenna
JP
.
A critical review of the Delphi technique as a research methodology for nursing
.
Int J Nurs Stud
.
2001
;
38
(
2
):
195
200
.
27
Viera
AJ
,
Garrett
JM
.
Understanding interobserver agreement: the Kappa statistic
.
Fam Med
.
2005
;
37
(
5
):
360
363
.
28
Polit
DF
,
Beck
CT
(eds).
Nursing Research: Generating and Assessing Evidence for Nursing Research
. 10th ed.
Philadelphia, PA
:
Wolters Kluwer
;
2017
:
306
.
29
Martino
R
,
Silver
F
,
Teasell
R
, et al
.
The Toronto bedside swallow screening test
.
Stroke
.
2009
;
40
:
555
561
.
30
Daniels
SK
,
Anderson
JA
,
Willson
PC
.
Valid items for screening dysphagia risk in patients with stroke. A systematic review
.
Stroke
.
2012
;
43
(
30
):
892
987
.
31
Schepp
SK
,
Tirschwell
DL
,
Miller
RM
,
Longstretch
ST
.
Swallowing screens after acute stroke: a systematic review
.
Stroke
.
2012
;
43
(
3
):
869
871
.
32
Swet
JA
.
The science of choosing the right decision threshold in high stakes diagnostics
.
Am Psychol
.
1992
;
47
:
522
532
.

Footnotes

To purchase electronic or print reprints, contact American Association of Critical-Care Nurses, 101 Columbia, Aliso Viejo, CA 92656. Phone, (800) 899-1712 or (949) 362-2050 (ext 532); fax, (949) 362-2049; email, reprints@aacn.org.