Reliability and validity of a risk assessment tool for patients with kidney failure.
In the past, much of the research on risk characteristics of patients with kidney failure focused on predictors of mortality. This emphasis was consistent with the concern about high rates of premature death in this population and the goal of extending the duration of dialysis patients' lives. More recently, researchers have expanded the study of risk factors of patients with kidney failure to identify predictors of risk for a broader range of adverse outcomes, including hospitalization rates and reduced quality of life (Merkus et al., 2000).
Today, much more is known than in the past about the factors that place patients with kidney failure at risk of experiencing several important adverse outcomes. However, as yet, there is no single risk assessment tool that nurses and other professionals can use to screen the dialysis population for their risk status. The purpose of this research is to describe the development and testing of a brief, evidence-based instrument to prospectively identify patients on hemodialysis who have a greater likelihood of a number of adverse outcomes, including hospitalization, poor quality of life, poor symptom management, and death. This initial test of the instrument examined its ability to predict patients at greater risk for two outcomes, hospitalization and poor quality of life.
Several standardized instruments have been developed and tested to measure risk of adverse outcomes, such as hospitalization and mortality in chronically ill patients. Tools that predict repeated hospitalizations have been of particular interest in light of the significant opportunities for prevention and cost-savings. The Probability of Repeat Admissions ([P.sub.ra]) questionnaire, for example, has been used extensively to identify older persons at risk for rehospitalization within 4 years (Paccala, Boult, & Boult, 1995). The Community Assessment Risk Screen (CARS) extends beyond hospitalization risk and identifies older adults who are at higher risk of emergency department use as well as hospitalization in the next year (Shelton, Sager, & Schraeder, 9000). Similarly, there are a number of tools that measure increased risk for death. The Charlson Comorbidity Index is commonly used in populations with multiple chronic illnesses to screen for mortality risk (Charlson, Pompei, Ales, & MacKenzie, 1987).
The majority of standardized risk assessment instruments, such as the [P.sub.ra], or Charlson Comorbidity Index, attempt to predict risk of a single adverse outcome, such as hospitalization or mortality, and are intended to do so in diverse populations comprised of patients with multiple diagnoses. These instillments use a relatively small set of risk predictors, such as previous hospitalizations or self-perceived health, to segment patients into low and high-risk groups. For more homogeneous populations with advanced and serious illnesses, such as kidney failure, research findings suggest that a broader scope of physical, psychological, social, and clinical factors may be required to measure risk at acceptable levels of reliability and validity. Findings of research on predictors of important adverse outcomes in the hemodialysis population are highlighted in the following section.
Predictors of mortality. There has been extensive research on the clinical risk factors associated with mortality in patients on hemodialysis. Several studies have demonstrated a significant relationship between age, incidence, and severity of co-morbid conditions and mortality (Chadna, Schultz, Lawrence, Greenwood, & Farrington, 1999; Davies, Russell, Bryan, Phillips, & Russell, 1995; Khan, 1998; Mailloux et al., 1996). Cardiovascular disease is the primary cause of death in patients with kidney failure. Older individuals (over the age of 70) and those with other comorbid conditions including diabetes and obstructive lung disease also have a greater probability of dying. Poor functional status and clinical indicators of inadequate nutritional status, such as reduced serum albumin, also have been found to be significant predictors of mortality (DeOreo, 1997; Maillonx et al., 1996; McClellan, Anson, &Tuttle, 1991; Owen, Lew, Liu, Lowrie, & Lazarus, 1993).
The relationship between psychosocial factors, such as lack of social support or depression, and mortality in the hemodialysis population is less well understood. Kimmel et al. (1998) found significant relationships between poor social support, non compliance, negative perceptions of the effects of illness, and mortality. However, Kuttner et al. (1994) showed that when functional status variables were in the predictive model, other social and psychological variables, such as life satisfaction and depression, did not significantly predict mortality.
Predictors of hospitalization. While the body of research on predictors of hospitalization has been rapidly expanding for many different populations with chronic illness, there has been less research on patients undergoing hemodialysis. We know that patients with kidney failure have higher rates of hospitalization with longer lengths of stay than the general adult population (Khan et al., 2002). The emotional and economic burdens of repeated hospitalizations to patients, their families, and the health care system also are well documented (Naylor, 2002). To date, however, relatively little is known about the constellation of factors that predisposes patients on dialysis to being admitted to the hospital or to prolonged stays. Early research findings point to likely predictors of hospitalization risk, including poor quality of life, reduced functional status, and the incidence of co-morbid conditions (Beddhu, Bruns, Saul, Seddon, & Zeidel, 2000; Chadna et al., 1999; DeOreo, 1997).
Predictors of poor quality of life. There has been limited research on predictors of reduced quality in life in the hemodialysis population. Most of the studies in this area have been conducted in the past 5 years. Initial findings suggest that co morbid conditions, number and severity of syrup toms, and extent of anemia may predict quality of life (Khan, 1998; Merkus et al., 1997; Merkns et al., 1999). Rocco et al. (1997) found a significant relationship between the degree of renal dysfunction, measured by glomerular filtration rate (GFR) and quality of life. More recently, Patel and colleagues (2002) found significant relationships between several psychosocial variables, including perception of illness, depression, social support, satisfaction with physician care, and quality of life. The Kidney Outcomes Prediction and Evaluation (KOPE) study, now underway, proposes to examine clinical, social, and psychological characteristics influencing quality of life and other outcomes for patients with kidney failure (Sevick et al., 1998).
Overall, previous research has identified several common, overlapping risk factors for mortality, hospitalization, and poor quality of life in patients with kidney failure. Not unexpectedly, these risk factors cross psychosocial, physical, and functional domains. Most commonly, co-morbid conditions, functional status, and selected physiological indicators of declining kidney performance have been found to be significantly associated with a range of adverse outcomes, including mortality, hospitalization, and reduced quality of life. There also is growing support for psychosocial variables as predictors of risk in the hemodialysis population. This is consistent with research findings on risk assessment in other populations with chronic illness. Findings of common risk factors across outcomes suggest that it is feasible to successfully predict risk of a range of adverse outcomes with a single instrument.
Instrument development. The investigators conducted a comprehensive literature review to identify the set of factors known to be associated with risk of adverse health outcomes in patients with kidney failure. The review encompassed multiple domains of patient status and prognosis including prior medical history, current clinical conditions, functional status, social support, psychological stares, and individual health monitoring behavior. From the set of factors identified, individual risk assessment items were then developed or selected from available tools.
The primary goal for the instrument was optimal discrimination between high, moderate, and low risk patients. Other desired features were considered during instrument development including: (a) ease and speed of administration; (b) coverage across relevant domains of patient status and care; and (c) ease of integration with other clinical processes and tools. The tool was intended for use in the clinical setting by busy clinicians. We believed that attention to these additional criteria would increase the likelihood of systematic and regular application of the instrument.
This approach resulted in the development of 16 individual risk assessment items. Since the literature on predictors of risk does not yet identify specific cutoff points for high risk in the hemodialysis population, an expert panel of nephrology physicians, nurses, and other health care professionals with expertise in kidney failure identified initial thresholds for defining high versus low/moderate risk for each item. Each item was scored as 0 or 1, with a value of 1 rep resenting high future risk of adverse health outcomes. The total possible score ranged from 0 to 16. Although the use of different weights for each item was considered, it was decided that equal weighting of items was more appropriate for this initial stage of instrument development and testing. The 16 item Risk for Outcomes Adverse to Dialysis (ROAD) assessment instrument is shown in Figure 1.
Source population. Study participants for testing of the ROAD were selected from the Anemia Management Demonstration Project (AMDP) funded by Amgen, Inc. The AMDP was a multi-center intervention study among 6,277 hemodialysis patients from 89 clinical sites in which select ed anemia management practices and outcomes were measured. The primary objective of the AMDP was to improve anemia status in hemodialysis patients and to implement best practices in anemia management. The AMDP is part of a systematic program of research designed to understand clinical practice patterns and interventions to improve quality of life and survival rates in patients with chronic kidney disease. Secondary objectives were to determine the impact of adequacy of dialysis, vascular access procedures, comorbid conditions, quality of life, and hospitalizations in anemia management. All AMDP participants were 18 years of age or older, with chronic kidney disease Stage 5 on in-center hemodialysis for at least 3 months, and receiving EPOGEN[R] (Epoetin Alfa; Amgen, Inc: Thousand Oaks) and no other erythropoietic agent for treatment of anemia. Exclusion criteria included life expectancy less than 6 months, peritoneal dialysis, home hemodialysis, and return to hemodialysis after a failed renal transplant.
At baseline, demographics, clinical history, psychosocial information, use of medications, quality of life, and patient satisfaction were assessed. During a prospective 18-month follow-up period, patient comorbidities, vascular access information, use of medications, hospitalizations, insurance coverage, EPOGEN[R] dosing, iron dosing, and hemoglobin status were documented at quarterly intervals. Patient satisfaction and quality of life were documented at 6-month follow-up intervals.
Embedded within the AMDP was a case management intervention that included periodic patient risk assessment by use of the ROAD assessment instrument, and coordinated use of a 23-item Multidisciplinary Action Plan (MAP) (Hartigan et al., 2003). The MAP was used to document physiologic, psychologic, and treatment-related target outcomes in study patients, and to document variances from these outcomes and corrective actions implemented. Within this framework, the ROAD assessment was completed at study entry and every 6 months thereafter, unless triggered by specific patient variances from target outcomes on the MAP that was completed monthly for high-risk patients and quarterly for low/moderate risk patients.
The AMDP study protocol was approved by the Institutional Review Boards (IRB) of all participating clinical centers. Each patient participating in the research signed an informed consent. A study coordinator at each site obtained the consent.
Study population. Within the full AMDP cohort, a random subsample of 253 participants at 5 of the 89 participating clinical centers was selected to assess the reliability and predictive validity of the ROAD assessment instrument. A power analysis showed that a sample size of 250 was needed to detect a moderate correlation between the ROAD scores and the two outcomes, hospitalization and quality of life. The five sites participating in the validation study were asked to submit actual item responses comprising the total ROAD score.
The mean age of the 253 study participants was 57 [+ or -] 16 years (range 18 to 88), 47% were female, 54% were of African American race, 30% of Caucasian race, and the remaining 16% of other races. The leading primary causes of kidney failure were hypertension (38%) and diabetes (25%). The distribution of study participants by number of years on hemodialysis were 14% for less than 1 year, 21% for one to less than 2 years, 23% for 2 to less than 4 years, and 42% for 4 years or more. Overall, participants in the reliability and validity sample were comparable to all participants in the AMDP study.
Statistical analysis. For each study participant, a single ROAD assessment was selected for use in the analysis (completed between the period June 2001 and February 2002). When more than one ROAD assessment was completed on a study participant, the last assessment was selected for analysis. The lower bound of the internal reliability coefficient of the individual ROAD assessment items was estimated by use of Cronbach's alpha coefficient (Cronbach, 1951). The predictive validity of individual ROAD assessment scores was evaluated in relation to incident hospitalization (as documented by participating clinical site personnel on a quarterly basis) and physical and mental quality of life scores (as determined every 6 months by use of the Kidney Disease Quality of Life (KDQOL)--Dialysis version (Hays, Kallich, Mapes, Coons, & Carter, 1994), each of which was determined prospectively during a period 1 to 6 months after completion of the ROAD assessment. The KDQOL is a self-report measure consisting of 19 scales that include a 36 item health survey supplemented with multi-item scales targeted at particular concerns of individuals with chronic kidney disease and on dialysis (e.g. cognitive function, quality of social interaction), and other multi-item scales (e.g., social support, patient satisfaction). Relative risk of hospitalization by ROAD assessment scores (categorized as 0 to 1, 2 to 4, or 5 or more), and 95% confidence intervals were calculated by the Mantel Haenszel method. Pearson correlation coefficients were calculated with bivariate scatter plots developed to depict relationships between individual ROAD assessment scores and physical and mental component quality of life scores.
Given the preliminary nature of the reliability and validity assessment, scoring thresholds for selected risk assessment items were modified in an attempt to increase over-all internal reliability consistency. Similarly, the potential utility of reducing one or more of the original 16 items to achieve a more parsimonious set of items was investigated by iteratively recalculating internal reliability coefficients. Exploratory and confirmatory factor analyses were conducted to investigate the underlying factor structure of the ROAD assessment instrument
Internal reliability. The psychometric properties of the individual ROAD assessment instrument items are listed in Table 1. Using the instrument in its original form, the prevalence of items with high risk of future adverse outcomes (scored as 1) ranged from a low of 5.1% (Item 14: "In the last 3 months, patient has lost more than 5% of his/her estimated dry weight.") to a high of 71.5% (Item 7: "During the past 4 weeks, patient has either never or rarely: (a) actively looked for information about kidney disease and its treatments; (b) watched care received to be sure the right thing was being done; or (c) decided which symptoms to report to the doctor or nurse and which ones to handle personally."). Correlations of the individual items with the total 16-item risk assessment score ranged from 0.12 (Item 8: "Patient age [greater than or equal to] 70 years.") to 0.54 (Item 3: "Health in climbing one flight of stairs, walking one block, or bathing or dressing is currently limited a lot."). Cronbach's internal reliability coefficient of precision for the 16 items was 0.53. The mean risk assessment score was 3.7 [+ or -] 2.2 (see Figure 2 [left side]).
[FIGURE 2 OMITTED]
The revised instrument included modified scoring thresholds for Items 5, 6, and 7, as well as removal of Item 8 (see Table 1). This resulted in a narrowed range of the prevalence of items with high risk of future adverse outcomes from 5.1% (Item 14) to 41.5% (Item 1). Correlations of the individual items with the total 15-item risk assessment score improved modestly from 0.20 (Item 9) to 0.55 (Item 3), as did the internal reliability coefficient of precision from 0.53 to 0.61. The mean risk assessment score was 3.0 [+ or -] 2.3 (see Figure 2 [right side]). Mean risk assessment scores and internal reliability coefficients were of similar magnitude across the five participating clinical centers.
Construct validity. A single dominant factor (eigenvalue = 2.54) emerged in the exploratory factor analysis using principal components analysis with varimax rotation. Half of the ROAD items (Items 1 through 7) had strong loadings on this first dominant factor. All but one of remaining items had factor loadings over .40 on three less dominant factors (eigenvalues 1.47, 1.36, 1.24, respectively). This included a factor with Items 13 and 14; a factor with Items 11, 12, and 16, and a factor with Items 9 and 15. This pattern of factor loadings indicates that the ROAD may contain more than one dimension within the general construct of risk for adverse outcomes. Confirmatory factor analysis showed similar findings.
Predictive validity. Using the ROAD assessment instrument in its original form, high risk scores were strongly associated with incident hospitalization 1 to 6 months after the risk assessment was performed (see Table 2). Compared to patients with a total risk assessment score of 0 or 1, those with a score of between 2 to 4 were estimated to be at 4.32 times higher risk of hospitalization (95% confidence interval 1.07 - 17.37, P = 0.02), and the risk associated with a score of 5 or higher was estimated to be 8.42 fold (9.5% confidence interval 2.13 33.35, P < 0.0001). When evaluated as a continuous variable, each unit increase in the total risk assessment score was associated with an estimated 1.37 times higher risk of hospitalization (95% confidence interval 1.19 - 1.58, P < 0.0001). Similar results were observed using the revised ROAD assessment instrument (see Table 2). With each unit increase in the total risk assessment score, the rate of hospitalization increased from 32% at a score of 2 or higher to 51.5% at a risk assessment score of 6 or higher (see Table 3).
Scores from the ROAD assessment instrument in its original form were inversely correlated with both physical (r = 0.36, P < 0.001) and mental (r = 0.37, P < 0.001) components of quality of life (see Figure 3). The ROAD assessment scores from the revised instrument were also inversely correlated with physical and mental components of quality of life (r = 0.30, P < 0.001; r = -0.43, P < 0.001, respectively). The associations between risk assessment scores and incident risk of hospitalization and overall poorer quality of life were consistently observed across subgroups including age (< 50 years, 50 to 65 years, > 65 years), gender, race (African American, Caucasian, Other), and number of years on dialysis (<1 year, 1 to < 2 years, 2 to < 4 years, > 4 years).
[FIGURE 3 OMITTED]
The initial testing of a new instrument to predict risk of adverse outcomes in patients with kidney failure was promising. The ROAD instrument demonstrated moderate internal consistency reliability and strong predictive validity. Items most strongly correlated with the overall risk score, self-perceived health and functional performance, are similar to those on more general risk assessment tools.
The reliability coefficient of 0.61, measuring internal consistency of the revised ROAD instrument, while usually considered lower than optimal in internally homogeneous instruments, reflects a number of intended features of the instrument. The ROAD was developed for ease of use in the clinical setting. All individual items of the ROAD assessment instrument are scored in a binary manner (0 or 1), rather than using a more extensive rating scale. The use of binary scoring results in less variation than scales with more than two levels. Lower variation across instrument items has been shown to reduce the magnitude of the reliability coefficient (Crocker & Algina, 1986).
As noted earlier, the ROAD was developed to be a composite risk assessment instrument that purposely and systematically incorporated predictors of several adverse outcomes. Unlike many of the risk assessment tools that seek to predict a single type of risk, such as risk for hospitalization or death, the ROAD was intended to measure a more global concept of risk, that is, the patient's general status of risk for a set of common adverse outcomes. While there was some overlap in predictors of risk for the different adverse events in patients with kidney failure, the ROAD contained items that, to date, have been linked to only one or a small number of the adverse outcomes. Thus, not all of the items may have been expected to be as highly correlated with one another as in an instrument attempting to measure risk for a single adverse event. This interpretation is consistent with the results of the factor analyses, which showed a single dominant factor and additional weaker factors. Given the intent and the item structure of the ROAD, an alternative measure of reliability, such as test-retest, may be more appropriate in future evaluations of instrument performance.
Tests of predictive validity of the ROAD showed moderate inverse correlations between the ROAD score and physical and mental quality of life as measured by the KDQOL -- Dialysis version. A small number of the ROAD items (Items 1,2,3,5, and 6) were totally or partially contained in the 39-item KDQOL. Differences in scoring of the ROAD and the KDQOL mediate against the likelihood that the modest overlap in items accounted for the magnitude of the correlation between the two instruments. Items on the ROAD, including those with multiple subparts, such as Items 2 and 3, are scored either as 0 or 1. In contrast, all items on the KDQOL are scored separately and, thus, have a broader range of possible scores. In addition, correlations calculated for the tests of predictive validity were based on total KDQOL scores. It is not likely that the overlap of 5 of 39 items would substantially influence the magnitude of the correlation between the ROAD and the KDQOL at a later time. From the standpoint of clinical utility, it is noteworthy that the brief ROAD instrument predicted subsequent global quality of life. Future tests of predictive validity of the ROAD with the KDQOL might be carried out with and without the few overlap ping items for comparison.
Overall, initial psychometric testing indicated that the ROAD instrument has substantial promise for achieving the criteria set out for its development. The instrument is relatively brief, with items that are easily retrievable from patient interview or record review. The scoring algorithm is simple and may be completed by the majority of clinical staff working with patients with kidney failure. The total score on the instrument is significantly correlated with risk for poor outcomes.
Several further refinements may be considered in future tests of the ROAD including adjustments in the cutoff points for risk as well as weighting of items to enhance the sensitivity and specificity of prediction. In this initial testing of the ROAD instrument, for instance, the rate of hospitalization increased from 32% to 51.5% as the total score increased from 2 or higher to 6 or higher and then stabilized with a score of 7 of higher. This suggests that a total score of 6 may be a reasonable cutoff point for differentiating high-risk patients from those at lower risk, particularly if hospitalization is a significant concern. However, the selection of cutoff points also must take into account the clinical implications of incorrectly categorizing high risk patients as low risk and, thereby, missing opportunities for intervention or conversely, categorizing lower risk patients as high risk and possibly deploying unneeded resources in their behalf. Choosing the most appropriate cutoff point is situation specific and should be based on the above clinical considerations as well as available resources for intervening with high risk patients (Iezzoni, 1997).
Weighting of ROAD items pro vides another opportunity to enhance instrument performance. Several of the general risk tools incorporate regression weights into their scoring algorithms. The drawback of this refinement, most obviously, is the added complexity and time required for calculating the risk score. The benefits in predictive accuracy must be carefully weighed against the costs to clinical utility.
The results of this study indicate that the ROAD assessment instrument is a promising tool to identify patients on hemodialysis at risk of poor outcomes. The instrument is short, easily scored, and relies on clinical and laboratory information commonly gathered on dialysis patients. High risk scores on the ROAD were significantly related with subsequent hospitalization and poor quality of life ratings. In light of the number of hemodialysis patients at substantial risk for adverse outcomes, we encourage nurses and other professionals to consider use of the ROAD to identify their patients at risk and to assist them in matching interventions and resources to patient needs and desired goals.
Figure 1 Risk for Outcomes Adverse to Dialysis (ROAD) Instrument Risk for Outcomes Adverse to Dialysis Patients (ROAD) Assessment Interview with Patient Date: -- Patient Name: -- 1. In general, would you say your health is: Excellent  Very good  Good  Fair  Poor  1a. Risk: Mark "X" in the Yes box if answer is fair or poor Y  N  2. During the past four weeks, to what extent were you bothered by each of the following? Not at all Somewhat Moderately bothered bothered bothered a. Itchy skin    b. Washed out or drained    c. Cramps    d. Nausea    Very much Extremely bothered bothered a. Itchy skin   b. Washed out or drained   c. Cramps   d. Nausea   2a. Risk: Mark "X" in the Yes box if any symptom is rated very much bothered or extremely bothered Y  N  3. Does your health now limits you in these activities? If so, how much? Yes, Yes, No, not limited limited limited a lot a little at all a. Climbing one flight of stairs    b. Walking one block    c. Bathing or dressing yourself    3a. Rick: Mark "X" in the Yes box if any activity is rated Yes, limited a lo Y  N  4. Would you have difficulty getting a friend, relative, or neighbor to take care of you for a few days it necessary? 4a. Risk: Mark "X" in the Yes or No box Y  N  5. How much of the time during the past 4 weeks have you felt downhearted or blue? All of the time  Most of the time  A good bit of the time  Some of the time  A little of the time  None of the time  5a. Risk: Mark "X" in the Yes box if answer is all of the time or most of the time Y  N  6. During the past four weeks, how much of the time has your physical health or emotional problems interfered with your social activities (like visiting friends, relatives, doing the things you like to do, etc.)? All of the time  Most of the time  A good bit of the time  Some of the time  A little of the time  None of the time  6a. Risk: Mark "X" in the Yes box if answer is all or most of the time Y  N  7. During the past four weeks, how often have you done the following? Y  N  Never Rarely Sometimes Often Always a. Actively looked for      information about kidney disease and its treatments (e.g., library, internet, etc.) b. Watched the care you      received to be sure the right thing was being done c. Decided which      symptoms to report to the doctor or nurse and which ones to handle on our own 7a. Risk: Mark "X" in the Yes box if any activity rated Never or Rare. Y  N  Medical Record 8. Patient date of birth 8a. Risk: Mark "X" in the Yes box if age [greater than or equal to] 70 Y  N  9. Length of time on Dialysis 9a. Risk: Mark "X" in the Yes box if time on dialysis is less than one year Y  N  10. Does the patient have any history of the following: a. Diabetes Y  N  b. Angina Y  N  c. MI (heart attack) Y  N  d. Heart Failure Y  N  e. Emphysema/Chronic Bronchitis Y  N  f. Stroke Y  N  g. Peripheral Vascular Disease Y  N  10a. Risk: Mark "X" in the Yes box if 3 or more condition: Y  N  11. Has the patient missed one or mole dialysis treatments in the past four weeks? Y  N  Did the patient have his/her dialysis treatment in another setting? Y  N  11a. Risk: Mark "X" in the Yes box if patient missed one or more dialysis treatments the past four weeks and he/she did not have the treatment in another setting Y  N  12. In the previous 6 months, has the patient stayed overnight in a hospital? 12a. Risk: Mark "X" in the Yes or No box Y  N  Laboratory Values/Weight 13. In how many of the last three months, has the patient's average * hemoglobin been less than 11 g/dL? * Calculate the patient's average hemoglobin for each month. Add all hemoglobin levels for the month and divide by the total number of times the hemoglobin level was collected during that month. Repeat this calculation for each of the preceding three months. None  Once  Twice  Three Times  13a. Risk: Mark "X" in the Yes if one or more times Y  N  14. In the last three months, has the patient lost more than 5% of his/her estimated dry weight (EDW)? 14a. Risk: Mark "X" in the Yes or No box Y  N  15. Is the patient's serum albumin less than 3.5 g/dL? Y  N  15a. Risk: Mark "X" in the Yes or No box Y  N  16. In how many of the last three months, has the patient's Kt/V been less than 1.2? Y  N  None  Once  Twice  Three Times  16a. Risk: Mark "X" in the Yes if one or more times Y  N  Calculation of Risk Score Count all the Yes answers (Highest possible is 16 points) Total ROAD: -- HIGH-RISK: 10 or more points Table 1 Psychometric Properties of Individual ROAD Instrument Items (N = 253) Original Instrument Risk Assessment Item Prevalence R (a) 1. Self-reported health (reported as Fair or 41.5% .45 Poor) 2. Bothered by various physical symptoms in the 32.4% .53 past 4 weeks (reported as Very Much or Extremely Bothered) 3. Health limited by various activities 36.4% .54 (reported as Limited a Lot) 4. Difficulty getting a friend, relative, or 9.5% .28 neighbor for help with care (reported as Yes) 5. Time during the past 4 weeks felt downhearted or blue (reported as All or Most of the Time) 9.5% .40 (reported All, Most, or a Good Bit of the Time) 6. Time during the past 4 weeks physical health or emotional problems interfered with social activities (reported as All or Most of the 9.9% .38 Time) (reported All, Most, or a Good Bit of the Time) 7. Time during the past 4 weeks actively monitored kidney disease information and care (reported as Never or Rarely) 71.5% .32 (reported as Never or Rarely -- excluded actively looking for kidney information) 8. Date of birth (70 years or older) 27.7% .12 Item removed from assessment 9. Length of time on dialysis (< 1 year) 7.9% .24 10. Having a past history of various clinical 15.4% .30 conditions (reported as [greater than or equal to] 3 past conditions) 11. Missed one or more dialysis treatments in the 9.9% .17 past 4 weeks and did not have treatment in another setting (reported as Yes) 12. Stayed overnight in a hospital in the 27.3% .50 previous 6 months (reported as Yes) 13. Number of times in the past 3 months that 33.2% .38 average monthly hemoglobin < 11 g/dL (reported as one or more times) 14. Lost more than 5% of estimated dry weight in 5.1% .38 the past 3 months (reported as Yes) 15. Serum albumin < 3.5 g/dL (reported as Yes) 11.9% .28 16. Number of times in the past 3 months 19.4% .32 Kt/V < 1.2 (reported as one or more times) Coefficient Alpha for All Items 0.53 Revised Instrument Risk Assessment Item Prevalence R (b) 1. Self-reported health (reported as Fair or 41.5% .47 Poor) 2. Bothered by various physical symptoms in the 32.4% .52 past 4 weeks (reported as Very Much or Extremely Bothered) 3. Health limited by various activities 36.4% .55 (reported as Limited a Lot) 4. Difficulty getting a friend, relative, or 9.5% .30 neighbor for help with care (reported as Yes) 5. Time during the past 4 weeks felt downhearted or blue (reported as All or Most of the Time) (reported All, Most, or a Good Bit of the 17.0% .51 Time) 6. Time during the past 4 weeks physical health or emotional problems interfered with social activities (reported as All or Most of the 15.4% .54 Time) (reported All, Most, or a Good Bit of the Time) 7. Time during the past 4 weeks actively monitored kidney disease information and care (reported as Never or Rarely) 12.2% .29 (reported as Never or Rarely -- excluded actively looking for kidney information) 8. Date of birth (70 years or older) -- -- Item removed from assessment 9. Length of time on dialysis (< 1 year) 7.9% .20 10. Having a past history of various clinical 15.4% .30 conditions (reported as [greater than or equal to] 3 past conditions) 11. Missed one or more dialysis treatments in the 9.9% .20 past 4 weeks and did not have treatment in another setting (reported as Yes) 12. Stayed overnight in a hospital in the 27.3% .53 previous 6 months (reported as Yes) 13. Number of times in the past 3 months that 33.2% .38 average monthly hemoglobin < 11 g/dL (reported as one or more times) 14. Lost more than 5% of estimated dry weight in 5.1% .39 the past 3 months (reported as Yes) 15. Serum albumin < 3.5 g/dL (reported as Yes) 11.9% .26 16. Number of times in the past 3 months 19.4% .35 Kt/V < 1.2 (reported as one or more times) Coefficient Alpha for All Items 0.61 Note: (a) Pearson correlation with total risk assessment score: P < 0.0001 for all items except Item 8 (P = 0.05) and Item 11 (p = 0.007). (b) Pearson correlation with total revised risk assessment score: P < 0.0001 for all items except Item 9 (P = 0.001) and Item 11 (p = 0.001) Table 2 Relationship Between ROAD Assessment Scores and Incidence of Hospitalization 1 to 6 Months After Risk Assessment (N = 241) (a) ROAD Score Relative (Original Scale) N Incidence Risk 0 or 1 40 5.0% 1.0 2 to 4 125 21.6% 4.32 5 or higher 76 42.1% 8.42 P-value for trend (b) (Revised Scale) 0 or 1 66 7.6% 1.0 2 to 4 118 27.1% 3.58 5 or higher 57 42.1% 5.56 P-value for trend (b) 95% ROAD Score confidence (Original Scale) interval P-value 0 or 1 -- -- 2 to 4 1.07-17.37 0.02 5 or higher 2.13-33.35 < 0.0001 P-value for trend (b) < 0.0001 (Revised Scale) 0 or 1 -- -- 2 to 4 1.47-8.74 0.002 5 or higher 2.27-13.62 < 0.0001 P-value for trend (b) < 0.0001 Note: (a) Excludes 12 patients without complete follow-up 1 to 6 months after completion of the ROAD assessment. (b) ROAD assessment score modeled as a continuous variable. Table 3 Relationship Between Risk Assessment Scores and Incident Hospitalization 1 to 6 Months After Risk Assessment (N = 241) (a) Risk Assessment Rate of Score Prevalence Hospitalization Sensitivity 2 or higher 72.6% 32.0% 91.8 3 or higher 54.8% 35.6% 77.0 4 or higher 37.8% 39.6% 59.0 5 or higher 23.7% 42.1% 39.3 6 or higher 13.7% 51.5% 27.9 7 or higher 8.3% 50.0% 16.4 Risk Assessment Relative 95% Confidence Score Specificity Risk Interval 2 or higher 33.9 4.22 1.77-10.08 3 or higher 52.8 2.77 1.62-4.76 4 or higher 69.4 2.37 1.53-3.68 5 or higher 81.7 2.09 1.38-3.18 6 or higher 91.1 2.44 1.60-3.71 7 or higher 94.4 2.17 1.31-3.57 Note: (a) Excludes 12 patients without complete follow-up 1 to 6 months after completion of the ROAD assessment. Risk assessment scores are cumulative and represent different possible cutpoints.
Beddhu, S., Bruns, F.J., Saul, M., Seddon, P., & Zeidel, M. (2000). A simple comorbidity scale predicts outcomes and costs in dialysis patients. The American Journal of Medicine, 108, 609-613.
Chadna, S.M, Schultz, J., Lawrence, C., Greenwood, R.N., & Farrington, K. (1999). Is there a rationale for rationing chronic dialysis? A hospital-based cohort study of factors affecting survival and morbidity. British Medical Journal, 318, 217-223.
Charlson, M.E., Pompei, P., Ales, K., & MacKenzie, C.R. (1987). A new method of classifying prognostic comorbidity in longitudinal studies: Development and validation. Journal of Chronic Diseases, 40, 373 383.
Crocker, L., & Algina, J. (1986). Procedures for estimating reliability. In L. Crocker & J. Algina (Eds.), Introduction to classical and modern test theory (pp. 131-156). Orlando, FL: Harcourt Brace Jovanovich.
Cronbach, LJ. (195i). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297-334.
Davies, S.J., Russell, L., Bryan, J., Phillips, L., & Russell, G. (1995). Comorbidity, urea, Kinetics, and appetite in continuous ambulatory dialysis patients, their interrelationships and predictors of survival. American Journal of Kidney Diseases, 26, 353-361.
DeOreo, P. (1997). Hemodialysis patient-assessed functional health status predicts continued survival, hospitalization, and dialysis attendance compliance. American Journal of Kidney Diseases, 30, 204-212.
Hartigan, M., Cesta, T., Mapes, D., Bun-ows-Hudson, S., Prathikanti, R., Lamb, G., & Zazworksy, D. (2003). AMDP: The Anemia Management Demonstration Project. Care Management, 8, 19-30.
Hays, R.D., Kallich, J.D., Mapes, D.L., Coons, S.J., & Carter, W.B. (1994). Development of the kidney disease quality of life (KDQOL) instrument. Quality of Life Research, 3, 329-338.
Iezzoni, L.I. (1997). Risk adjustment for measuring healthcare outcomes. Second edition. Chicago, IL: Health Administration Press.
Khan, I. (1998). Comorbidity: The major challenge for survival and quality of life in end-stage renal disease. Nephrology, Dialysis and Transplantation, 13, S76-79.
Khan, S.S., Kazmi, WH., Abichandani, R., Tighiouart, H., Pereira, B.J., & Kautz, A.T. (2002). Health care utilization among patients with chronic kidney disease. Kidney International, 62, 229-236.
Kimmel, P., Peterson, C., Weirs, K, Simmens, S.J., Alleyne, S., Cruz, I., & Veis, J.H. (1998). Psychosocial factors, behavioral compliance, and survival in urban hemodialysis patients. Kidney International, 54, 245-254.
Kuttner, N., Lin, L., Fielding, B., Grogan, D., & Hail, D. (1994). Continued survival of older hemodialysis patients: Investigations of psychosocial predictors. American Journal of Kidney Diseases, 24, 42-49.
Lamb, G., Shelton, P., & Zazworsky, D. (2002). Disease management. In T.G. Cesta & H.M. Tahan (Eds.), The case. manager's survival guide: Winning strategies for clinical practice (2nd edition) (pp. 134-152). St. Louis: Mosby.
Mailloux, L., Napolitano, B., Bellucci, A., Mossey, R., Vernace, M., & Wilkes, B. (1996). The impact of co-morbid risk factors at the start of dialysis upon the survival of ESRD patients. ASAIO Journal, 164-169.
McClellan, W., Anson, C., & Tuttle, E. (1991). Functional status and quality of life: Predictors of early mortality among patients entering treatment for end-stage renal disease. Journal of Clinical Epidemiology, 44, 83-89.
Merkus, M.P., Jager, K.J., Dekker, F.W., Boeschoten, E.W., Stevens, P., Krediet, R.T. & The Necosad Study Group. (1997). Quality of life in patients on chronic dialysis: Self-assessment 3 months after the start of treatment. American Journal of Kidney Diseases, 29, 584-592.
Merkus, M.P., Jager, K.J., Dekker, F.W., deHaan, R.J., Boetschoten, E.W., & Krediet, R.J. (1999). Physical symptoms and quality of life in patients on chronic dialysis: Results of The Netherlands Cooperative Study on Adequacy of Dialysis. Nephrology Dialysis Transplantation, 14, 1163-1170.
Merkus, M., Jager, K., Dekker, F., deHaan, R., Boeschoten, E., & Krediet, R. (2000). Predictors of poor outcome in chronic dialysis patients: The Netherlands Cooperative Study on the adequacy of dialysis. American Journal of Kidney Diseases, 35, 69-79.
Owen, W., Lew, N., Liu, Y., Lowfie, E., & Lazarus, J. (1993). The urea reduction ratio and serum albumin concentration as predictors of mortality in patients undergoing hemodialysis. New England Journal of Medicine, 329, 1001-1006.
Naylor, M.D. (2002). Transitional care of older adults. In P. Archbold & B. Stewart (Eds.), Annual review of nursing research (Vol. 20, pp. 127-147). New York: Springer.
Paccala, J., Boult, C., & Boult, L (1995). Predictive validity of a questionnaire that identifies older persons at risk for hospital admission. JAGS, 43, 374-377.
Patel, S.S., Shah, V.S., Peterson, R.A., & Kimmel, P.L. (2002). Psychosocial variables, quality of life, and religious beliefs in ESRD patients treated with hemodialysis. American Journal of Kidney Diseases, 40, 1013-1022.
Rocco, M.V., Gassmau, J.J., Wang, S.R., & Kaplan, R.M. (1997). Modification of diet in renal disease study group. Cross-sectional study of quality of life and symptoms in chronic renal disease patients: The modification of diet in renal disease study. American Journal of Kidney Diseases, 29, 888-896.
Sevick, M.A., Tell, G.S., Shumaker, S.A., Rocco, M.V., Burkart, J.M., Rushing, J.T., Levine, D.W., Chen, G.J., Bradham, D.D., Pierce, J.J., & James, M.K. (1998). The Kidney Outcomes Prediction and Evaluation (KOPE) study: A prospective cohort investigation of patients undergoing hemodialysis. Annals of Epidemiolagy, 8, 192-200.
Shelton, P., Sager, M., & Schrader, C. (2000). The Community Assessment Risk Screen (CARS): Identifying elderly persons at risk for hospitalization or emergency department visit. American Journal of Managed Care, 6, 925-933.
Wagner, E.H., Austin, B.T., Davis, C., Hindmarsh, M., Schaefer, J., & Bonomi, A. (2001). Improving chronic illness care: Translating evidence into action. Health Affairs, 10, 64-78.
Gerri Lamb, PhD, RN, FAAN, is Associate Dean and Associate Professor, University of Arizona. College of Nursing, Member, Case Manager Solutions, LLC, Tucson, AZ.
Kevin E. Kip, PhD, is Assistant Professor, University of Pittsburgh, Department of Epidemiology, Pittsburgh, PA.
Donna Mapes, DNSc, RN, is Principal, Donna Mopes and Associates, and Adjunct Professor. University of California, San Francisco, San Francisco, CA; she is a member of ANNA's Northern California Chapter.
Sally Burrows-Hudson, MS, RN, CNN, is Senior Associate Director of Nephrology Medical Affairs, Amgen, Inc., and Adjunct Professor, University of California, San Francisco, San Francisco. CA. She is a Past President of ANNA and a member of the Northern California Chapter.
Toni Cesta, PhD, RN, FAAN, is Director of Case Management, St. Vincents Hospital New York, NY, and Member, Case Manager Solutions, LLC, Tucson, AZ.
Donna Zazworsky, MS, RN, FAAN, is Director, Community Nursing and Outreach, St. Elizabeth of Hungary Clinic, Tucson, AZ, and Member, Case Manager Solutions, LLC, Tucson, AZ.
Note: Funding for the research was provided by Amgen, Inc., Thousand Oaks, CA. Figure 1 in this article is reprinted with the permission of Amgen, Inc.
|Printer friendly Cite/link Email Feedback|
|Author:||Lamb, Gerri; Kip, Kevin E.; Mapes, Donna; Burrows-Hudson, Sally; Cesta, Toni; Zazworsky, Donna|
|Publication:||Nephrology Nursing Journal|
|Date:||May 1, 2004|
|Previous Article:||Caring through the end of life.|
|Next Article:||Challenges for nephrology nurses in the management of children with chronic kidney disease.|