A psychometric study of empowerment and confidence among veterans with psychiatric disabilities.
Empowerment is an often cited recovery domain that has been linked empirically with participation in peer support (Burti et al., 2005; Dumont & Jones, 2002; Resnick & Rosenheck, 2008; Rogers et al., 2007), working for pay, and participation in family psycho-education (Resnick, Rosenheck, & Lehman, 2004). Rogers et al. (1997) using a mixed-methods approach, created a tool to measure empowerment, and identified five subordinate factors: self-esteem/self-efficacy, power-powerlessness, community activism and autonomy, optimism and control over the future, and righteous anger. Carpinello et al. (2000) identified a related concept, confidence, with similar components to those identified by Rogers et al.: optimism, coping, and advocacy, suggesting overlap between the operationalization of empowerment by Rogers et al. and that of confidence by Carpinello.
The current study is an evaluation of the psychometric properties of these two measures and their interrelationships. We examine the internal consistency and test-retest reliability for each measure and examine convergent and discriminant validity of both the total scale and subscales in a sample of veterans receiving community-based outpatient mental health services.
[FIGURE 1 OMITTED]
Participants consisted of 296 veterans with severe mental illness who were admitted to the Community Reintegration Program, a community-based outpatient program at the Errera Community Care Center of the V.A. Connecticut Healthcare System between 2002 and 2006, and who agreed to participate in a quasi-experimental efficacy study of a peer education program for veterans. In the parent study, participants were recruited in two cohorts, but are pooled into a single sample for the present study, and thus represent both control and experimental groups (Resnick & Rosenheck, 2008).
As summarized in Table 1, respondents were predominantly male (95%) and white (66%), averaging 48.5 years of age and 12.6 years of education. One-third (36%) indicated regular full- or part-time employment. Although a comparable number (34%) reported unemployment due to disability, only one in five (19%) were receiving service-connected disability payments from the Veterans Administration for either medical or psychiatric reasons. PTSD symptom severity was high (mean [+ or -] SD = 46.4 [+ or -] 17.0), and--despite high levels on the ADL scale (mean [+ or -] SD = 4.1 [+ or -] 0.7)--global assessment of functioning (GAF) was quite low (mean [+ or -] SD = 37.1 [+ or -] 5.5). As is common in this population, participants reported high rates of homelessness (34%), problems with alcohol (46%), and arrest histories (18%).
Most subjects (91%) returned for their one-month follow-up. Four out of five (79%) participated in the three month follow-up, and two-thirds (67%) completed the nine month assessment.
To avoid recruiting individuals who did not return for treatment after their initial screening, eligibility criteria for the study required participation in the Community Reintegration Program for at least one full week. There were no other inclusion or exclusion criteria. The Community Reintegration program is adjunctive to psychiatric treatment, providing individualized treatment with access to a range of therapeutic, rehabilitative and psychoeducational groups.
The parent study was observational only; veterans received usual care, and those in the second cohort could choose to attend or not attend peer education services. Referrals came from Community Reintegration Program clinicians after informing clients about the study, and from program-wide presentations at community meetings. Upon verifying eligibility and giving informed consent, veterans completed a baseline interview conducted by an experienced independent rater, who also conducted follow-up interviews at one, three and nine months after baseline assessment. The one-month interview was an abbreviated version of the assessment done at other time points, primarily intended to encourage study participation. All measures were administered in interview format, with the interviewer reading the item aloud and using visual aids to describe scale responses.
The protocol was approved by institutional review boards of the VA Connecticut Healthcare System and Yale Medical School.
Interviews assessed numerous domains, including empowerment and confidence; functional status; substance use; symptom severity; trauma history; and life satisfaction, as detailed fully elsewhere (Resnick & Rosenheck, 2008). This paper focuses on measures of empowerment and confidence, selected for inclusion in the parent study due to their hypothesized relationship to participation in peer education, and potential sensitivity to change over time. They were selected for the current study due to their theoretical relationship to the concept of recovery.
The 28-item Empowerment Scale (ES) measures general empowerment as defined by consumers of mental health services (Rogers, Chamberlin, Ellison, & Crean, 1997). The instrument's subscales address self-efficacy, power, activism, optimism and righteous anger. Rogers et al. report a high degree of internal consistency for the full scale (alpha=.86). Responses range from 1 to 4, with items scored so that higher values reflect greater levels ofempowerment. The scale is reported as the average of all items. In a sample of 261 participants in self-help groups, the Empowerment Scale had a mean [+ or -] SD score of 2.94 [+ or -] 0.32 (Rogers et al.).
The Mental Health Confidence Scale (MHCS) is a sixteen-item measure of general confidence and self-efficacy designed for use in populations with mental disorders (Carpinello, Knight, Markowitz, & Pease, 2000). The measure has high internal consistency (alpha=.94 for the full scale, and .91, .90 and .80 for the subscales of optimism, coping and self-advocacy, respectively). Items are rated on a six-point scale where 1="very nonconfident" and 6="very confident"; thus, higher scale scores indicate greater confidence. Carpinello and colleagues calculate a total score as a sum of all items. The first fifteen of the sixteen items were administered in the present study (the last item was omitted due to a clerical error), and thus, scale scores were calculated as the average of all items multiplied by sixteen, which is equivalent to mean substitution.
Scores on the ES and MCHS resembled those observed in previous studies of mental health self-help service consumers. Whereas Rogers and colleagues (Rogers et a1.,1997) reported a mean [+ or -] SD of 2.94 [+ or -] 0.32 in their validation study of the ES, the present sample yielded 2.96 [+ or -] 0.33 (t=0.72, p=.47). The study by Carpinello and colleagues (2000) does not cite an overall mean on the MHCS, but reports means of 68 and 65.3 (SD not available) for those who did and did not attend their self-help program in the past twelve months, respectively. The mean in the present study, 66.11 [+ or -] 15.26, fell between those who had not attended their program recently (t=0.91, p=.36) and those who had (t=2.13, p=.03), indicating that those in the present study were statistically less confident than this latter group. Nevertheless, our mean falls within the range represented by the validation study's two subgroups.
Both measures displayed good levels of internal consistency reliability at all four study time points (see Table 2). Cronbach's standardized item alpha for MCHS was consistently excellent at all observation periods (.92, .93, .95 and .93 for the baseline, one month, three month and nine month interviews, respectively). The ES also displayed good reliability, with alphas of .79, .82, .85 and .84, respectively at baseline, one-, three- and nine-months.
The differing time intervals between follow-ups allowed test-retest reliability, as measured by the Pearson correlation coefficient, to be calculated for intervals of 30, 60, 90, 180, 240 and 270 days. As expected, reliability declined as the time-interval increased (see Figure 2). Test-retest reliability on both scales was highest for 30 days (between baseline and the one-month interview), although, lower than the corresponding Cronbach's alpha. This is typical, as alpha is calculated on contemporaneous rather than lagged data. For example, 30-day test-retest reliability for the MHCS was .74, compared with alphas in the .9 range. Findings were similar for the ES (.75 vs.79).
Convergent and Discriminant Validity
For establishing validity, Cohen (1988) recommended the criteria of r [greater than or equal to] .70 for convergent validity (i.e., measures of the same construct) and r [less than or equal to] .40 for discriminant validity (i.e., measures of unrelated constructs). In this study, correlations between the MHCS and ES were .70 at baseline, .65 at one month, .70 at three months, and .72 at nine months. Three of these four correlations thus meet Cohen's criterion for convergent validity.
Correlations of the ES and MHCS with theoretically different constructs are shown in Table 3. The scales were selected from among those available in the parent study as measuring variables that were conceptually distinct from empowerment and confidence. PTSD symptom severity (PTSD), the Brief Psychiatric Rating Scale (BPRS), General Assessment of Functioning (GAF), general life satisfaction, and activities of daily living (ADL) generally correlate .40 or less with both empowerment scales, thus indicating good discriminant validity. Exceptions include the anxiety-depression subscale of the BPRS, which correlates at--0.43 with the ES, and -.52 with the MHCS (p<.001 for both), indicating that greater symptom severity is associated with less empowerment and confidence. Similarly, the MHCS also correlates above .40 with PTSD symptom severity (r = -.45, p<.001).
The remainder of Table 3 shows the association of the ES and MHCS with various background characteristics in order to determine if any potential bias in empowerment or confidence can be attributed to respondents' backgrounds. All correlations are smaller in magnitude than r [less than or equal to] .40. Two small but statistically significant correlations (p<.05) occur, between ES and education (r = .14) and MHCS and money spent on alcohol in the past 30 days (r = -.13). However, the magnitude of these correlations is quite low--indicating that the variables share less than 2% of their variance and may reflect over-powered significance tests due to the sample size (n=296).
[FIGURE 2 OMITTED]
Subscales of the ES and MHCS
In order to examine the nuanced conceptual similarities and differences between the ES and MHCS, the baseline subscales of both measures were examined. Of the five ES subscales, only the first, measuring self-esteem and self-efficacy, had acceptable reliability in this sample (Cronbach's standardized-item alpha = 0.87 at baseline). The other four ES subscales (power-powerlessness; community activism and autonomy; optimism and control over the future; and righteous anger) had alphas ranging from 0.51 to 0.62. The first two of three subscales on the MHCS has good reliability: Optimism (alpha=0.91) and factors coping (alpha=.90). However, the advocacy factor had a lower reliability of 0.56.
The intercorrelations between these subscales are presented in Table 4. The average correlation among ES subscales was r=0.20, with a range of -0.17 to 0.48. The three MHCS subscales correlated with each other, on average, at r=0.29, with a range of 0.30 to 0.67.
The ES and MHCS subscales that correlated most highly with each other were also those with reliability coefficients above 0.80: The ES self-esteem subscale correlated with the MHCS optimism and factors coping subscales at r=0.71 and 0.61, respectively (p<.001 for both). The MHCS optimism and factors coping subscales correlated at r=0.67 (p<.001) with each other. Interestingly, the optimism subscales of the ES and MHCS correlated with each other at r=0.43 (p<.001); this, however, may reflect the low internal consistency we observed on the ES optimism subscale (alpha=0.54).
Data presented here suggest that the two empowerment measures are reliable and valid when administered to veterans with psychiatric disabilities. The internal consistency of the measures was good to excellent, and the sample means on the MHCS and ES were similar to those reported in the original validation studies in similar treatment samples of seriously mentally ill consumers, increasing confidence in the validity of these measures.
Measures of constructs that vary as life circumstances change, such as those examined here, would be expected to have reasonable test-retest reliability when evaluated across short intervals, but reliability coefficients should decrease in magnitude as the interval between assessment points increases. This pattern was observed in the present study, with the 30-day test-retest reliability coefficients ranging between .64 and .75 for the measures but, as expected, these correlation coefficients decreased over time.
The relationships between the ES and MHCS were consistently around .70, suggesting that these measures are closely related and possibly measure the same construct. This differs from another study in which the correlation between these same two scales among individuals with psychiatric disabilities in the Netherlands was somewhat lower, at .61 and the authors concluded that the two instruments are measuring different aspects of empowerment (Castelein, van der Gaag, Bruggeman, van Busschbach, & Wiersma, 2008).
An examination of the subscales suggests that much of the overlap between the two scales observed in the current study is driven by the Self-Esteem subscale of the ES, and the Optimism and Coping subscales of the MHCS. Correlations between these subscales were r = 0.71 (Self-Esteem with Optimism) and r = 0.61 (Self-Esteem with Coping). We thus conclude that the Self-Esteem subscale of the ES and the Optimism subscales of the MHCS are measuring the same construct, while the other subscales are related with varying magnitude, and thus measure similar but not identical constructs. Interestingly, the two Optimism subscales were only related at r = 0.43, which does not meet criteria for convergent validity, and thus these two subscales are likely measuring two distinct but related constructs.
This variation in language is reflective of the larger literature. There are many conceptualizations of the recovery construct, and disagreements as to where these various constructs fit into the larger conceptualization. The continued growth in this area has led to a diverse array of measures with uncertain interrelations. Instruments have been developed to measure domains thought to be elements of recovery, including self-esteem, hope and optimism, life satisfaction, knowledge, and quality of life, as well as some that attempt to measure recovery more globally. There have been at least two major efforts to gather these recovery measures into a single source (Campbell-Orde, Chamberlin, Carpenter, & Left, 2005; Ralph, Kidder, & Phillips, 2000). However, to date, there is no consensus in the literature on a definition or defining ingredients of recovery, nor has the relationship between recovery-oriented constructs been systematically examined for convergent and divergent validity.
There has been some suggestion in the literature that it is possible that the quest to empirically determine the distinct critical ingredients of recovery is unnecessary. In this perspective all subjective evaluation in community mental health can be measured by one global factor, reflecting a tendency to respond in a positive or negative manner, influenced by mood (Fakhoury, Kaiser, Roeder-Wanner, & Priebe, 2002; Fakhoury & Priebe, 2002; Hansson, Bjorkman, & Priebe, 2007), perhaps making the need for additional construct development unnecessary. However, the results of the current study do not support the idea of a single all-encompassing dimension, given that many of the interrelationships between the three primary measures of interest and the measure of general life satisfaction were well below .70, Cohen's criterion for convergent validity. However, our findings do support the idea that anxiety and depression significantly influence self-ratings. The Anxiety and Depression subscale of the BPRS had a moderate and significantly negative association with both measures, while those with the Thought Disturbance subscale were small, and with the exception of the ES, not statistically significant. The substantial relationship between symptoms and recovery oriented measures has also been reported from a different sample using a different measure of recovery (Resnick, Rosenheck, Dixon, & Lehman, 2005).
The study presented here has several limitations that should be noted. First, we did not employ post hoc corrections for the large number of correlations, and so the risk of Type I error is increased. Because confidentiality limitations precluded collecting comparison data from those who declined to participate, it was not possible to role out potential threats to external validity. It is furthermore unknown to what extent veterans in CT resemble the general population of U.S. veterans. Furthermore, they may not generalize to other populations of people with psychiatric disabilities.
Empowerment and confidence are believed to play important roles in recovery from mental illness. Two measures of these recovery-oriented concepts were examined for reliability and validity in a sample of 296 veterans with psychiatric disabilities enrolled in a peer education program. Both measures displayed good internal consistency and test-retest reliability; correlated moderately with each other, indicating the cohesion of convergent validity without being overly redundant; and did not correlate excessively with unrelated measures (i.e., good divergent validity). We conclude that these measures are both reliable and valid for use among veterans with psychiatric disabilities.
Burti, L., Amaddeo, F., Ambrosi, M., Bonetto, C., Cristofalo, D., Ruggeri, M., et al. (2005). Does additional care provided by a consumer self-help group improve psychiatric outcome? A study in an Italian community-based psychiatric service. Community Mental Health Journal, 41, 705-720.
Campbell-Orde, T., Chamberlin, J., Carpenter, J., & Leff, H. S. (2005). Measuring the promise: A compendium of recovery measures, Volume II: The Evaluation Center @ HSRI.
Carpinello, S. E., Knight, E. L., Markowitz, F. E., & Pease, E. (2000). The development of the Mental Health Confidence Scale: A measure of self-efficacy in individuals diagnosed with mental disorders. Psychiatric Rehabilitation Journal, 23, 236-243.
Castelein, S., van der Gaag, M., Bruggeman, R., van Busschbach, J. T., & Wiersma, D. (2008). Measuring empowerment among people with psychotic disorders: A comparison of three instruments. Psychiatric Services, 59, 1338-1342.
Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences. New York: Psychology Press.
Dumont, J. M., & Jones, K. (2002). Findings from a consumer/survivor defined alternative to psychiatric hospitalization. Outlook, Spring, 4-6.
Fakhoury, W. K. H., Kaiser, W., Roeder-Wanner, U.-U., & Priebe, S. (2002). Subjective evaluation: Is there more than one criterion? Schizophrenia Bulletin, 28, 319-327.
Fakhoury, W. K. H., & Priebe, S. (2002). Subjective quality of life: Its association with other constructs. International Review of Psychiatry, 14, 219-224.
Hansson, L., Bjorkman, T., & Priebe, S. (2007). Are important patient-rated outcomes in community mental health care explained by only one factor? Acta Psychiatrica Scandinavica, 116, 113-118.
Mancini, A. D. (2008). Self-determination theory: A framework for the recovery paradigm. Advances in Psychiatric Treatment, 14, 358-365.
President's New Freedom Commission on Mental Health. (2003). Achieving the promise. Transforming mental health care in America. Washington, D.C.
Ralph, R. O., Kidder, K., & Phillips, D. (2000). Can we measure recovery? A compendium of recovery and recovery-related instruments: The Evaluation Center @ HSRI.
Resnick, S. G., Fontana, A., Lehman, A. F., & Rosenheck, R. A. (2005). An empirical conceptualization of the recovery orientation. Schizophrenia Research, 75, 119-128.
Resnick, S. G., & Rosenheck, R. A. (2008). Integrating peer support and mental health services: A quasi-experimental study of recovery orientation, confidence and empowerment. Psychiatric Services, 59, 1307-1314.
Resnick, S. G., Rosenheck, R. A., Dixon, L., & Lehman, A. F. (2005). Correlates of family contact with the mental health system: Allocation of a scarce resource. Mental Health Services Research, 7, 113-121.
Resnick, S. G., Rosenheck, R. A., & Lehman, A. F. (2004). An exploratory analysis of recovery and its correlates in people with schizophrenia. Psychiatric Services, 55, 540-547.
Rogers, E. S., Chamberlin, J., Ellison, M. L., & Crean, T. (1997). A consumer-constructed scale to measure empowerment among users of mental health services. Psychiatric Services, 48, 1042-1047.
Rogers, E. S., Teague, G. B., Lichenstein, C., Campbell, J., Lyass, A., Chen, R., et al. (2007). Effects of participation in consumer-operated service programs on both personal and organizationally mediated empowerment: Results of multisite study. Journal of Rehabilitation Research and Development, 44, 785-800.
SAMHSA. (2004). National Consensus Statement on Mental Health Recovery. National Consensus Conference on Mental Health Recovery and Mental Health Systems Transformation from http://download.ncadi.samhsa.gov/ken/pdf/SMA054129/trifold.pdf
Silverstein, S. M., & Bellack, A. S. (2008). A scientific agenda for the concept of recovery as it applies to schizophrenia. Clinical Psychology Review, 28, 1108-1124.
Yale University School of Medicine
Robert A. Rosenheck
Yale University School of Medicine
Sandra G. Resnick
Yale University School of Medicine
Sandra Resnick, Yale School of Medicine, 333 Cedar Street, New Haven, CT 06510: Email:
Table 1. Baseline characteristics and study participation of veterans with psychiatric disabilities (N=296) Characteristic N Demographics Male 281 95% White 194 66% Age (M [+ or -] SD) 48.5 [+ or -] 8.7 Education (M [+ or -] SD years) 12.6 [+ or -] 2.1 Marital status Divorced 129 44% Never married 82 28% Housing and employment Homeless in the past 30 days 100 34% Days worked in the past 30 (M [+ or -] SD) 0.2 [+ or -]0.4 Total income in the past month $885.50 [+ or -] 1131.68 (M [+ or -] SD dollars) Usual employment pattern, past 30 days Disabled 100 34% Regular full- or part-time 105 36% Irregular/student/volunteer/retired 48 16% Other 44 15% Service, disability and mental health Ever served in a war zone 89 30% VA service-connected disability benefits 56 19% Percent service-connected psychiatric 15.5 [+ or -] 31.3 Age first psychiatric admission 32.6 [+ or -] 12.9 Number of psychiatric admissions (lifetime) 5.4 [+ or -] 14.9 Health and functioning GAF (M [+ or -] SD score) (1) 37.1 [+ or -] 5.5 Activities of Daily Living Scale 4.1 [+ or -] 0.7 (M [+ or -] SD score) (2) General life satisfaction (M [+ or -] SD 4.2 [+ or -] 1.5 score) (3) BPRS (M [+ or -] SD total score) (4) 1.3 [+ or -] 0.5 PTSD symptom severity (M [+ or -] SD 46.4 [+ or -] 17.0 score) (5) Substance Abuse and Legal Money spent on alcohol, past 30 days $14.50 [+ or -] 45.40 (M [+ or -] SD dollars) Days used alcohol, past 30 (M 2.6 [+ or -] 6.8 [+ or -] SD days) Money spent on drugs, past 30 days $34.70 [+ or -] 163.50 (M [+ or -] SD dollars) Days used drugs, past 30 (M [+ or -] 2.1 [+ or -] 6.7 SD days) Ever arrested 54 18% Follow-up rates One month 269 91% Three months 235 79% Nine months 199 67% (1) Global Assessment of Functioning. Possible scores range from 1 to 99, with higher scores indicating greater functioning. (2) Possible scores range from 1 to 5, with higher scores indicating greater functioning. (3) As measured by the Lehman Quality of Life scale. Possible scores range from 1 to 7, with higher scores indicating greater life satisfaction. (4) Brief Psychiatric Rating Scale. Possible scores range from 0 to 6, with higher scores indicating greater symptom severity. (5) Post-traumatic stress disorder is measured by the PTSD Check List-Stressor Specific Version. Possible scores range from 17 to 85, with higher scores indicating greater PTSD symptom severity. Table 2. Internal Consistency and Test-Retest Reliability of Two Measures of Empowerment/Confidence among Veterans with Severe Mental Illness Internal consistency (Cronbach's alpha) Interview Scale Baseline 1 month 3 month 9 Month (N=296) (N=269) (N=235) (N=199) Empowerment scale (ES) 0.79 0.82 0.85 0.84 Mental Health Confidence Scale (MHCS) 0.92 0.93 0.95 0.93 Test-Retest Reliability (Pearson r) Time interval (in days) * Scale 30 60 90 180 240 270 Empowerment scale (ES) 0.75 0.71 0.68 0.59 0.63 0.57 Mental Health Confidence Scale (MHCS) 0.74 0.67 0.71 0.72 0.56 0.62 * Time intervals reflect differing intervals between survey pairs: 30 days=baseline vs. 1 month; 60=1 month vs. 3 month; 90=baseline vs. 3 month; 180=3 month vs. 9 month; 240=1 month vs. 9 month; 270=baseline vs. 9 month. Table 3. Association of the Empowerment and Confidence Scales with Baseline Characteristics (N=296) Empowerment MH Confidence Correlations Scale (ES) Scale (MHOS) r r PTSD symptom severity (a) -0.27 *** -0.45 *** Brief Psychiatric Rating Scale -0.23 *** -0.32 *** (BPRS) (a) BPRS anxiety-depression -.43 *** -.52 *** subscale (a) BPRS thought disturbance -.14 * -.09 subscale (a) General Assessment of 0.17 ** 0.19 ** Functioning (GAF) (a) General life satisfaction (b) 0.30 *** 0.40 *** Activities of Daily Living (b) 0.17 ** 0.18 ** Age -0.07 0.00 Education 0.14 * 0.04 Employment income -0.06 -0.07 Age at first mental health -0.03 -0.06 hospitalization Lifetime # of psych admissions -0.02 0.04 Days drunk alcohol, past 30 -0.04 -0.10 Money spent on alcohol, past 30 -0.08 -0.13 Money spent on drugs, past 30 0.04 0.10 Lifetime # of arrests 0.00 0.00 TX: psych admissions (lifetime) -0.09 -0.05 Attended -0.10 -0.08 t t Gender 0.16 0.65 White 0.70 -0.27 Married 0.62 0.44 Never married -1.00 -1.66 Once married 0.46 1.19 Ever seen combat 1.38 0.69 Usual employment regular -0.98 -0.06 full- or part-time Usual employment disability 0.98 0.52 Service-connected psych >0% -1.23 -1.35 (a) Higher scores indicate greater severity. (b) Higher scores indicate better health functioning. * p<.05 ** p<.01 *** p<.001 Table 4. Correlation of ES and MHCS subscales at baseline (N=296) Empowerment Scale Self Power Activism Optimism esteem (lessness) Empowerment scale (ES) Self esteem -- Power(lessness) 0.11 -- Activism 0.48 *** -0.02 -- Optimism 0.57 *** 0.06 0.51 *** -- Righteous anger -0.17 ** 0.50 *** -0.11 -0.14 * Mental Health Confidence Scale (MHCS) Optimism 0.71 *** 0.21 *** 0.36 *** 0.43 *** Coping 0.61 *** 0.28 *** 0.27 *** 0.41 *** Advocacy 0.37 *** 0.19 ** 0.27 *** 0.22 *** Empowerment Scale Mental Health Confidence Scale Righteous Optimism Coping Advocacy anger Empowerment scale (ES) Self esteem Power(lessness) Activism Optimism Righteous anger -- Mental Health Confidence Scale (MHCS) Optimism -0.04 -- Coping 0.04 0.67 *** -- Advocacy 0.02 0.31 *** 0.30 *** --