Printer Friendly

Use of multilevel covariance structure analysis to evaluate the multilevel nature of theoretical constructs.

Hierarchical linear modeling (HLM) techniques have had significant impact on the evaluation of multilevel theoretical models in a variety of disciplines (that is, educational research, criminology, organizational psychology, economics, and family therapy; see Bryk & Raudenbush, 1987,1992; Draper, 1995; Hoeksma & Koomen, 1992; Hox & Kreft, 1994; Kreft & Leeuw, 1998; Moritz & Watson, 1998; Raudenbush, Brennan, & Barnett, 1995; Rogosa & Saner, 1995a, 1995b; Sampson, Raudenbush, & Earls, 1997; Thum, 1997; Vancouver, 1997). Despite its effect, HLM does not allow researchers to examine covariance structure models, such as factor analysis, path analysis, and structural equation models. Recognizing this limitation, researchers (for example, Harnqvist, 1978; Muthen, 1994; Muthen & Satorra, 1989) have extended HLM to the analysis of multilevel structural equation models, which is referred to as multilevel covariance analysis (MCA). This article illustrates the use of MCA by evaluating the multilevel nature of students' perceptions of school safety. The article provides readers with an overview of MCA, discuss its application to social work research, and demonstrates how Mplus 1.0, a structural equation model (SEM) program, can be used to analyze MCA models.

Multilevel Covariance Analysis--Definition and Uses

MCA represents an extension of hierarchical linear multivariate modeling to those situations that involve covariance structure models, such as factor analysis, path analysis, and structural equation modeling. The research and theory used to achieve this extension is based on the work of Muthen and colleagues (Muthen, 1994; Muthen & Muthen, 1998; Muthen & Satorra, 1989).

Given the hierarchical nature of social and organizational behavior, social workers have a keen interest in multilevel analysis techniques. For example, these techniques help social workers gain a better understanding of the effects of both community characteristics and individual characteristics on residents' perceptions of neighborhood safety (see, for example, Sampson et al., 1997).

MCA has several advantages and potential uses for social workers. Like HLM, the primary advantage is found in its ability to simultaneously estimate both within- and between-setting variance, and therefore more accurately estimate individual's outcomes within the setting (Muthen, 1994). MCA allows social workers to study the contribution to the variations in individuals' behavior that are associated with between-context variations and idiosyncrasies of individuals or their experiences. For example, it allows school social workers to examine the contribution that differences in school climate, compared to parent's cultural capital, play in student's academic progress. Like SEM, MCA makes use of two methodological traditions: Thurstonian factor analysis and simultaneous equation (path analytic) modeling (Kaplan & Elliott, 1997). These two methodologies allow social workers to develop statistical models that acknowledge the relationship among predictors and their direct and indirect relationships to outcome variables. In other words, MCA allows a social worker to examine the role that various community resources have on activism when both the relationships among community resources and individual's temperament are taken into account. In addition, MCA allows social workers to represent complex theoretical constructs as latent variables. For example, social workers studying sibling social behavior could represent social behavior as a latent variable, that is, composed of indicators of adaptive and maladaptive social functioning. In addition, the relationship among predictors of social functioning, for example, parental discipline and conflict resolution styles, could be studied.

Advantages of MCA

MCA seeks to incorporate SEM's ability to capture the complexity of multivariate relationships with HLM's ability to capture the nested nature of those relationships, both of which are needed to develop accurate statistical models of social phenomena. Although HLM addresses the limitations of conventional regression analysis to accurately estimate multivariate models under conditions of multilevel sampling, HLM, like conventional regression analysis techniques, cannot incorporate latent variables into the representation of theoretical variables, and it cannot model the interrelationship among independent variables. Like OLS regression the independent variables are assumed to be uncorrelated. Meanwhile, MCA allows one to incorporate latent variables and model the interrelationships among independent variables. Unlike OLS regression, the independent variables can be correlated.

Development of MCA

Muthen (1989, 1994) developed a procedure for modeling the multilevel phenomena found in MCA. The Muthen model assumes that sampling occurs at two levels, between groups (that is, between social structures) and within groups (that is, between individuals). As a result, we can consider that the population's total covariance matrix ([[Sigma].sub.T]) contains both the between-group covariance ([[Sigma].sub.B]) and within-group covariance matrixes ([[Sigma].sub.W]). In other words, [[Sigma].sub.T] = [[Sigma].sub.B] + [[Sigma].sub.W]. In single-level covariance analysis (or conventional SEM), which ignores the clustering of individuals' observations and assumes the independence of observations, the between-group covariance matrix equals zero (that is, [[Sigma].sub.W] = 0). In the multilevel situation the between-group covariance matrix is nonzero. Building on these assumptions (that is, [[Sigma].sub.T] = [[Sigma].sub.B] + [[Sigma].sub.W] and [[Sigma].sub.W] = 0, when the single-level analysis involves a single-level covariance structure), Muthen (1994) developed a five-step process for performing MCA: (1) conventional factor analysis of the total structure using the sample total covariance matrix; (2) estimation of the between-group variation; (3) estimation of the within-group structure; (4) estimation of the between-group structure; and (5) estimation of the MCA model using both the between-group and within-group covariance matrixes. The latter process can be achieved using the multiple-group capability of conventional SEM software (that is, AMOS, EQS, or LISREL). Duncan, Alpert, and Duncan (1998) provided an example for those interested in MCA using LISREL and software that can decompose the total covariance matrix into the between- and within-group covariance matrix (for example, SOURCEBW; Muthen, 1997). Using the above-mentioned software to estimate the MCA method is problematic in two ways: (1) Conventional SEM using maximum-likelihood estimation provides biased standard errors and chi-squares in the unbalanced situation (in those situations where the number of observations between clusters are not equal), and (2) fitting the necessary constraints between the within- and between-group models can often be problematic.

Muthen and Muthen (1998) developed Mplus 1.0, an SEM program that implements MCA and provides unbiased standard error and chi-square estimates. Mplus 1.0 uses the maximum-likelihood-fitting function with robust standard errors and a mean-adjusted chi-square statistic (MLM). Using Mplus 1.0, I demonstrate the use of MCA to study the potential multilevel nature of students' perceptions of school safety.

AN EMPIRICAL EXAMPLE: THE MULTILEVEL NATURE OF STUDENTS' PERCEPTIONS OF SCHOOL SAFETY

This example derives from research focusing on students' perceptions of school safety. Existing research suggests that school safety is a multilevel construct that has yet to be examined. It is hypothesized that students' perceptions of the safety of a school can be described at two levels: (1) personal safety, which refers to students' perceptions of the extent to which they have experienced unsafe situations while in school or their assessment of their personal risk of having unsafe experiences; and (2) collective safety, which refers to students' judgments of the extent to which classmates are likely to engage in unsafe acts while in school. Although there is an interaction between personal and collective safety, they both exist as features of a school operating at different levels.

Support for conceptualizing perceptions of school safety into personal and collective can be found in research on school bullying and in the sociology of crime and delinquency literature (Austin, Woolever, & Baba, 1994; Byrne, 1997; Rigby, 1997; Simpson, 1996). School bullying research supports the assessment of multiple stakeholders (that is, victims, bullies, teachers, parents, and so forth) who reported on their individual experiences/perceptions and on the larger safety/ victimization climate of the school as necessary for the development and evaluation of schoolwide intervention programs (Colvin, Tobin, Beard, Hagan, & Spraque, 1998; Olweus, 1991). Sociologically, safety and dangerousness can be viewed as intersubjective (Simpson, 1996). These concepts are "the product of social construction, collective agreement and socialization" (Simpson, 1996, p. 549). In addition, organizational researchers have assessed perceptions of safety at both individual and collective levels (Cheyne, Cox, Oliver, & Tomas, 1998; Clarke, 1999).

Data Source

Data from the base year (eighth grade) data collection phase of the National Educational Longitudinal Survey of 1988 (NELS:88) were used in this study. NELS:88 is an ongoing study that aims to provide trend data about critical transitions experienced by U.S. secondary school students. The NELS:88 survey design is based on a two-stage, stratified probability sample. The sampling strategy sought to attain a sample that would be nationally representative of public, private, and parochial school eighth graders (Ingels et al., 1994; Kaufman, Bradby, & Owings, 1992). The base-year sample consisted of 24,599 eighth graders, enrolled in 1,052 public and private high schools across the nation. Students were the basic unit of analysis. List-wise detention of cases with missing data or multiple responses reduced the sample to 23,427 students clustered in 1,050 schools. There was an average of 22 students in a school.

School Safety Indicators

Figure 1 shows the measurement model of students' perceptions of personal and collective school safety as distinct dimensions of a two-factor construct. Collective school safety assesses students' perceptions of the extent to which physical conflicts, vandalism, robbery or theft, and weapon possession are problems at their school. Higher scores on these items indicate that students perceive that their school has problems with physical conflict, vandalism, robbery or theft, and weapon possession. Personal safety assesses the frequency of times that a student had something stolen from him or her at school, been threatened by another student, had physical fights at school, or been asked to buy drugs at school. As scores increase, students' unsafe experiences increase. (Appendix 1 provides a full description of each of the items.)

[Figure 1 ILLUSTRATION OMITTED]

To illustrate the use of MCA, I hypothesized that the two-factor model of school safety can be used to represent perceptions of safety at both the individual and school levels.

METHOD

Model Estimation and Evaluations

The estimation method for the conventional and multilevel SEM uses maximum-likelihood (ML) and maximum-likelihood-fitting function with robust standard errors and mean-adjusted chi-square statistics (MLM), respectively. With the exception of the multiple level model, the evaluation of the model fit focuses on the comparative fit index (CFI) and standardized root mean squared residual (SRMR) statistics. Following the recommendations of Hu and Bentler (1999), CFI equal to or exceeding .95 and SRMR equal to or less than .09 are considered indicators of a good-fitting model. Because of the use of a quasi-likelihood-fitting function to estimate MCA, the only fit information available to evaluate multilevel models is the chi-square and [R.sup.2] (Muthen & Muthen, 1998). Standard fit statistics like SRMR and CFI that need likelihood values from a true maximum-likelihood function cannot be estimated with MLM estimators (Muthen & Muthen, 1998). In addition, the chi-square is presented and interpreted cautiously. Both the size of the study's sample and the multilevel bias introduced into the conventional SEM analysis support cautious interpretation of the chi-square statistics. The sample size makes the power of the chi-square test high, thus rejection at the 5 percent level might reflect trivial deviations from the model (Bentler & Weeks, 1980; Bollen, 1989; Muthen, 1994). Also, the hierarchical nature of the data, which is ignored in this conventional SEM analysis, inflates the value of the chi-square and the [R.sup.2] (Muthen, 1989, 1994). Therefore, evaluation of the MCA model focuses on the [R.sup.2] and size of the residuals for the between- and within-correlation matrix. Residual analysis focuses on the discrepancy between the observed correlations and the MCA model produced correlations. Small differences in the average absolute value of the discrepancy between the two matrixes are considered support for a model (Hu & Bentler, 1995).

RESULTS

Step 1: Conventional Factor Analysis of the Total Structure ([S.sub.T]).

This step allows the researcher to examine the hypothesized measurement and structural model that will be later evaluated in the between- and within-level components. The analysis during this phase follows the two-step SEM building procedures proposed by James, Mulaik, and Brett (1982). This step seeks to provide a rough estimate of the fit of the model, which is the focus of the MCA analysis. Because this analysis is based on the total covariance matrix, the model's fit usually is inflated, especially in those situations when there are large intraclass correlations and large group sizes (Duncan et al., 1998). Therefore, sequential models are not expected to provide larger fit statistics (Duncan et al., 1998).

This step of the analysis provides a rough estimation of the quality of fit that is likely to be achieved. Following the recommendations of Muthen (1994), standard errors of estimates are not given in this article because all models presented show the parameter significantly different from zero due to the large sample size.

Because the focus of the study is on the measurement of school safety, this analysis focuses on the measurement model. (Table 1 contains the total correlation matrix, item means, standard deviations, and intraclass correlations.) The analysis is based on the total and decomposed covariance matrixes (within and between matrixes), which are available from the author upon request. (Table 2 contains the standardized factor loadings for the hypothesized model.) The resulting model fits the ST matrix well: [chi square](19, N= 23,427) = 1,920, p [is less than] .01; CFI = .96; SRMR = .03, given the large sample size. The [R.sup.2], which gives lower bound estimates of the variables reliabilities, is lower for the personal than the collective safety items. The implications of this analysis are that there are two related dimensions to students' perceptions of safety--individual exposure to unsafe behavior and the collective safety perceptions.

TABLE 1 -- Total Correlation Matrix of Variables Affecting Students Perceptions of School Safety (N = 23,427)
 Factor

Factor 1 2 3

1. Number of times you have gotten
 into a fight with
 another student 1.00
2. Number of times you have
 had something stolen
 at school 0.14 1.00
3. Someone offered to sell 0.23 0.13 1.00
 me drugs at school
4. Someone threatened to hurt 0.24 0.25 0.18
 me at school
5. Physical conflicts among 0.12 0.10 0.11
 students a problem
6. Robbery or theft a 0.04 0.22 0.10
 problem at school
7. Vandalism of school 0.06 0.12 0.10
 property is a problem
8. Student possession of 0.13 0.10 0.18
 weapons a problem

M 1.70 1.43 1.91
SD 0.54 0.64 0.40
Intraclass correlations 0.03 0.06 0.04
Design effects(a) 1.63 2.26 1.84

 Factor

Factor 4 5 6

1. Number of times you have gotten
 into a fight with
 another student
2. Number of times you have
 had something stolen
 at school
3. Someone offered to sell
 me drugs at school
4. Someone threatened to hurt 1.00
 me at school
5. Physical conflicts among 0.14 1.00
 students a problem
6. Robbery or theft a 0.11 0.60 1.00
 problem at school
7. Vandalism of school 0.10 0.57 0.68
 property is a problem
8. Student possession of 0.13 0.54 0.63
 weapons a problem

M 1.74 2.71 3.02
SD 0.58 1.02 1.01
Intraclass correlations 0.03 0.10 0.08
Design effects(a) 1.63 3.10 2.68

 Factor

Factor 7 8

1. Number of times you have gotten
 into a fight with
 another student
2. Number of times you have
 had something stolen
 at school
3. Someone offered to sell
 me drugs at school
4. Someone threatened to hurt
 me at school
5. Physical conflicts among
 students a problem
6. Robbery or theft a
 problem at school
7. Vandalism of school 1.00
 property is a problem
8. Student possession of 0.62 1.00
 weapons a problem

M 3.01 3.22
SD 1.13 1.04
Intraclass correlations 0.09 0.08
Design effects(a) 2.89 2.68


(a) Design effect = 1 + (average cluster size - 1) x intraclass correlation.

TABLE 2--Two-Factor CFA Model with the Total Covariance Matrix ([S.sub.T) and Pooled Within-Covariance Matrix (Shy) Matrixes of Students' Perceptions of School Safety

 Total
 Covariance
 Matrix ([S.sub.T])
Factor
 Loading E [R.sup.2]

Personal safety problems

1. Number of times you have
 gotten into a fight
 with another student. .44 .90 .19
2. Number of times you have had
 something stolen
 at school. .42 .91 .18
3. Someone offered to sell me .40 .85 .16
 drugs at school.
4. Someone threatened to hurt .53 .85 .28
 me at school.
 Collective safety problems
5. Physical conflicts among .71 .70 .51
 students a problem
6. Robbery or theft a problem .83 .55 .70
 at school.
7. Vandalism of school property .81 .58 .66
 is a problem.
8. Student weapon possession .76 .65 .58
 is a problem

Covariance between personal
and collective school safety .32 -- --

 Within-
 Covariance
 Matrix ([S.sub.PW])

Factor
 Loading E [R.sup.2]
Personal safety problems

1. Number of times you have
 gotten into a fight
 with another student. .43 90 .18
2. Number of times you have had
 something stolen
 at school. .41 .91 .17
3. Someone offered to sell me .38 .92 .15
 drugs at school.
4. Someone threatened to hurt .53 .90 .28
 me at school.
 Collective safety problems
5. Physical conflicts among .70 .71 .49
 students a problem
6. Robbery or theft a problem .84 .55 .70
 at school.
7. Vandalism of school property .82 .58 .67
 is a problem.
8. Student weapon possession .75 .66 .57
 is a problem

Covariance between personal
and collective school safety .26 -- --

 Between
 Covariance
 Matrix ([S.sub.B])

Factor
 Loading E [R.sup.2]
Personal safety problems

1. Number of times you have
 gotten into a fight
 with another student. .53 .85 .28
2. Number of times you have had
 something stolen
 at school. .47 .88 .22
3. Someone offered to sell me .54 .84 .29
 drugs at school.
4. Someone threatened to hurt .63 .76 .29
 me at school.
 Collective safety problems
5. Physical conflicts among .82 .57 .67
 students a problem
6. Robbery or theft a problem .73 .68 .54
 at school.
7. Vandalism of school property .72 .70 .51
 is a problem.
8. Student weapon possession .86 .51 .74
 is a problem

Covariance between personal
and collective school safety .72 -- --


NOTE: [S.sub.T]: [chi square] (19, N = 23,427) = 1,920, p < .01; CFI = .96; SRMR = .03; [S.sub.PW]: [chi square] (19, N = 22,377) = 1,544, p < .01; CFI = 97; SRMR = 03; [S.sub.B]: [chi square] (19, N = 22,377) = 1,544, p < .01; CFO = 97; SRMR = .03.

Step 2: Estimation of the Intraclass Correlations for Variables in the Model

This phase of the analysis seeks to determine whether a multilevel analysis is warranted. Multilevel analysis is only warranted when the hypothesis that [[Sigma].sub.B] = 0 is rejected. To determine if this hypothesis can be rejected, one needs to examine the intraclass correlations. The intraclass correlation for a given variable is the ratio of the between-group variance to the total variance. If they are close to zero for the majority of variables, the hypothesis that [[Sigma].sub.B] = 0 is likely not to be rejected. The estimates of the intraclass correlations can be attained from Mplus 1.0 (Muthen & Muthen, 1998). (Appendix 2 contains the Mplus program code needed to estimate the intraclass correlations and the within- and between-covariance matrixes from the raw data file).

When considering all 1,050 schools in the sample, the average cluster size was 22, with a range from 1 to 63. The intraclass correlations ranged from .03 to .10 (Table 1). Given the average cluster size of 22, these intraclass correlations provide weak support for further investigation of the multilevel nature of the construct. Earlier research indicates that multilevel modeling also is indicated when design effects, which are a function of intraclass correlations and average cluster size, are greater than 2 (Johnson & Elliott, 1998). Five of the eight indicators had design effects greater than 2z (Table 1).

Step 3: Estimation of Within Structure ([S.sub.PW]).

This step involves analysis of the pooled within-covariance matrix ([S.sub.PW]). This analysis uses only the individual level parameters. Because all of the parameters used in this study are measured at the individual level, all of them are included in this analysis. Muthen (1994) noted that if the multilevel model is correct, a conventional factor analysis of [S.sub.PW] is the same as MCA with an unrestricted [[Sigma].sub.B] matrix. Analysis of the pooled within-covariance matrix provides estimates that are close to the within-level parameters of an MCA (Muthen, 1994). This analysis uses a sample size of N - G (sample size - number of clusters = 23,427 - 1,050 = 22,377) and uses the ML estimator. Because this analysis is not distorted by the between-level covariation, the fit is expected to be better than that found with the [S.sub.T] analysis (Muthen, 1994). In addition, it should be noted that the between loadings and [R.sup.2] are often larger than their corresponding within-level counterparts. Research indicates that this finding is reflective of the fact that between-level error variance is smaller because the consideration of outcomes aggregated to the higher level where errors tend to cancel out (Muthen, 1989). Professor Bengt Muthen (personal communication, University of California, Los Angeles, March 2, 2000) noted that this finding often is related to classic literature on ecological correlations. Robinson (1950) introduced the distinction between individual and ecological correlations. Individual correlation is the correlation in which the variables are measured at the individual unit of analysis; the variables in ecological correlation are measured at the group unit of analysis. Robinson's (1950) example of an ecological correlation is the correlation between the percentage of the population that is African American and the percentage of the population that is illiterate across all states. The variables are descriptive of the groups and not descriptive of individuals. Research has demonstrated that the magnitude of the relationship between variables operationalized at the group level often result in a larger effect size than these variables operationalized at the individual level (Duncan & Davis, 1953; Hammond, 1973; Kraemer, 1978; Robinson, 1950).

The two-factor model gives a good fit with the [S.sub.PW] matrix: [[chi square] (19, N= 22,377) = 1,544, p [is less than] .01; CFI = .97; SRMR = .03] (Table 2). Because the [S.sub.PW] is not distorted by the between-group variability, the fit of the model is expected to be better than that found when the total covariance matrix was used (that is, step 1). Unfortunately the chi-square parameter was the only fit indices indicating that the model that was performed on the [S.sub.PW] matrix resulted in a better fit relative to the [S.sub.T] matrix analysis. The [R.sup.2] when compared to those found in the conventional CFA, step 1 analysis, indicates very little biases in the reliabilities. The [S.sub.PW] analysis adjusts for differences in school means, whereas in the [S.sub.T] analysis the heterogeneity in the means across schools is expected to increase the reliable part of the variation, which inflates the reliabilities (that is, [R.sup.2]).

Step 4: Estimation of the Between Structure ([S.sub.B])

This step investigates the between structure using [S.sub.B] (the sample between-group covariance matrix). Duncan et al. (1998) noted that this phase of the analysis can be the most difficult. The difficulties result because it focuses not on individual-level data but on across-group variation. Such a focus may result in differences in interpretation of the data at the individual level. The difficulties are most problematic when the individual and higher level model are the same. In those situations the problems result from the fact that the individual (or within-level) component can have a different meaning at the between-group level. This phase of the analysis raises the question as to whether between school and between individual variations in perceptions of school safety are conceptually similar. The fit information for the two-factor model does not indicate an adequate fit: [[chi square] (19, N= 1,050) = 303, p [is less than] .01; CFI = .91; SRMR = .05 ]. Attempts to modify the two-factor model did not result in a model with acceptable fit. Failure at this stage weakens support for the MCA school safety model. Final examination of the MCA is needed to completely determine the appropriateness of this model.

Step 5: Estimation of the Multilevel Covariance Analysis

The last phase of the MCA involves analyzing the [S.sub.PW] and [S.sub.B] simultaneously. (Appendix 3 contains the Mplus program code needed to estimate the MCA model.) Descriptively, the factor loadings associated with the between-level (school) model are larger than those associated with the within-level (individual) model (Table 3). In addition, the covariation between the two factors, personal and collective safety problems, is larger in the between-level model when compared to the within-level model. Those findings along with the small level of between groups variability, which is indicated by the size of the intraclass correlations, suggest that the two-factor model at the school level is not needed and possibly a one-factor model is needed (Muthen, 1994).

TABLE 3--MCA School Safety Model
 Parameter Estimates

 Between (School) Level

 Residual
 (error)
Factor Loading variance [R.sup.2]

Personal safety problems

1. Number of times you have
 gotten into a fight with
 another student. .63 .60 .40
2. Number of times you have had
 something stolen at school. .48 .77 .23
3. Someone offered to sell me
 drugs at school. .67 .55 .45
4. Someone threatened to hurt
 me at school. .77 .41 .59

Collective safety problems

5. Physical conflicts
 among students
 a problem. .85 .27 .73
6. Robbery or theft a problem
 at school. .75 .43 .57
7. Vandalism of school property
 is a problem. .73 .47 .53
8. Student weapon possession
 is a problem. .88 .22 .78

Covariance between personal
 and collective school safety .94 -- --

 Parameter Estimates

 Within (Individual) Level

 Residual
 (error)
Factor Loading variance [R.sup.2]

Personal safety problems

1. Number of times you have
 gotten into a fight with
 another student. .43 .81 .18
2. Number of times you have had
 something stolen at school. .41 .83 .17
3. Someone offered to sell me
 drugs at school. .38 .85 .15
4. Someone threatened to hurt
 me at school. .53 .72 .28

Collective safety problems
5. Physical conflicts
 among students
 a problem. .70 .51 .49
6. Robbery or theft a problem
 at school. .84 .30 .70
7. Vandalism of school property
 is a problem. .82 .33 .67
8. Student weapon possession
 is a problem. .75 .43 .57

Covariance between personal
 and collective school safety .26 -- --


NOTES: MCA = multilevel covariance analysis. Between: average absolute covariance residuals = 0.063; average off-diagonal absolute covariance residuals = 0.068. Within: average absolute covariance residuals = 0.009; average off-diagonal absolute covariance residuals = 0.012. -- = not applicable.

An evaluation of the one-factor model at the school level with the corresponding two-factor model at the individual level did not improve the model's fit. These findings also provide additional evidence that between schools there is less variability in students' perceptions of safety than within schools and that the association between perceptions of personal and collective safety problems is strong within schools than between schools. Examination of the parameter estimates and the average absolute value of the residuals do not provide consistent support for the model's fit. If the MCA model better represents school safety than the conventional model, then the parameter estimates in the conventional model are biased. The MCA reveals very little bias in the parameter estimates compared to those found in the individual level of the MCA model. Furthermore, parameter estimates found at the individual level of the MCA analysis are not meaningfully different from those found in the conventional model (Tables 2 and 3).

When considering the residuals, if the MCA model is correct the average absolute value of the residuals should be small. The average absolute value of residuals was .063 and .009 for the between and within matrixes, respectively. Unfortunately, there were large residuals for three indicators. The largest discrepancy between observed and reproduced correlations, among all the correlations, was found among the collective safety indicators (robbery or theft a problem at school and vandalism of school property) and the individual safety indicator (someone offered to "sell me drugs" at school). The residuals were between .35 and .36.

Overall the results support the conceptualization of school safety as a multilevel theoretical construct that consists of two components, perceptions of individual and collective safety. The results also suggest that students' perceptions of safety contain more variability within schools than between schools.

DISCUSSION

MCA allows social work researchers to model both within (individual level) and between (higher level or contextual factors) parameters that provide greater understanding of the ecology of social phenomena. In addition, MCA allows researchers to examine the extent to which a theoretical construct can be said to be a characteristic of a setting and the extent to which that setting's characteristic influences individuals who operate within the setting.

Implications for Social Work Researchers

Multilevel analysis methods provide social workers with the means to directly and empirically evaluate ecological models and more efficiently address a variety of other important research questions related to the evaluation of social work practice. Multilevel analysis provides opportunities for the evaluation of individual change involving experimental, quasi-experimental, and correlation designs by developing growth curve models. By using multilevel analysis methods, limitations of conventional multivariate repeated measures methods (that is, MANOVA) can be overcome.

The flexibility of multilevel modeling techniques could allow for greater integration of case and program level evaluations. For example, multilevel modeling procedures could be used to integrate single-subject outcome data across different practitioners in the same agency. This can be achieved whether the client outcomes are measured at the interval or ordinal or nominal level given the availability of latent class analysis (Muthen, 1998). The analysis simultaneously could be used to take into account the differences in client resources, the elements of the practice process, and treatment components when determining the effectiveness of the agency. For example, consider an agency attempting to study the factors related to the agency's effectiveness in treating depression. If the evaluation makes use of multilevel modeling it can take into account the trajectory of client's depression over time and predictors of that trajectory, such as the client's social supports and the components of the intervention when the intervention is being evaluated.

Multilevel analysis methods allow for the evaluation of multisite programs where differences between sites and individuals can be adequately taken into account when estimating program effects (Kreft & Leeuw, 1998). MCA extends the benefits of HLM by allowing for the modeling of theoretical constructs as latent variables. This allows the researcher to refine the operationalization of theoretical constructs when incorporating measurement error and multiple indicators into the process.

MCA Limitations

As noted by Hox (1995) and Muthen and Muthen (1998), unlike HLM (that is, Bryk & Raudenbush, 1992), MCA does not allow for the examination of effects of higher level factors on the relationship between lower level predictors and outcomes (that is, the regression slopes). Both HLM and MCA techniques require the specification of a hierarchical system of regression equations. At each level of the hierarchy, outcomes and predictors can be specified and only in HLM can variables at higher levels be used as predictors of the relationship between predictors and outcomes at the lower levels. In MCA the higher level variable can only be used as predictors of outcomes at the lower levels. For example, in HLM the effect that contextual variables have on the relationship between individual risk factors and outcomes can be examined. In MCA only the relationship between contextual variables in individual outcomes can be examined. MCA is not a small sample technique. Muthen (1989) recommended that the group size should be from 50 to 100, with at least two students nested within each group.

Currently, whereas conventional SEM software can be used to evaluate MCA models, the input specifications are cumbersome and the analysis may break down because of singularity of between-sample covariance matrices. In addition, only Mplus 1.0 presently implements Muthen's maximum-likelihood-fitting function with robust standard errors and mean-adjusted chi-squared statistic, which provides unbiased standard error and chi-square estimates in the unbalanced situations.

Social workers using multilevel modeling techniques have the opportunity to answer important questions with an analysis strategy that can incorporate the multilevel nature of social behavior. Use of this strategy will advance the field of social work and allow social workers to use analytical strategies that are consistent with the ecological modeling of social environments.

REFERENCES

Austin, D. M., Woolever, C., & Baba, Y. (1994). Crime and safety-related concerns in a small community. American Journal of Criminal Justice, 19, 79-97.

Bentler, P. M., & Weeks, D. G. (1980). Linear structural equations with latent variables. Psychometrika, 45, 289-307.

Bollen, K. A. (1989). Structural equations with latent variables. New York: John Wiley & Sons.

Bryk, A. S., & Raudenbush, S. W. (1987). Application of hierarchical linear models to assessing change. Psychological Bulletin, 101, 147-158.

Bryk, A. S., & Raudenbush, S. W. (1992). Hierarchical linear models: Applications and data analysis models (Vol. 1). Newbury Park, CA: Sage Publications.

Byrne, B. J. (1997). Bullying: A community approach. Irish Journal of Psychology, 18, 258-266.

Cheyne, A., Cox, S., Oliver, A., & Tomas, J. M. (1998). Modeling safety climate in the prediction of levels of safety activity. Work & Stress, 12, 255-271.

Clarke, S. (1999). Perceptions of organizational safety: Implications for the development of safety culture. Journal of Organizational Behavior, 20, 185-189.

Colvin, G., Tobin, T, Beard, K., Hagan, S., & Sprague, J. (1998). The school bully: Assessing the problem, developing interventions, and furore research directions. Journal of Behavioral Education, 8, 293-319

Draper, D. (1995). Inference and hierarchical modeling in the social sciences. Journal of Educational and Behavioral Statistics, 20, 115-147.

Duncan, O. D., & Davis, B. (1953). An alternative to ecological correlation. American Sociological Review, 18, 665-666.

Duncan, T. E., Alpert, A., & Duncan, S. C. (1998). Multilevel covariance structure analysis of siblings antisocial behavior. Structural Equation Modeling, 5, 211-228.

Hammond, J. L. (1973). Two sources of error in ecological correlations. American Sociological Review, 38, 764-778.

Harnqvist, K. (1978). Primary mental abilities at collective and individual levels. Journal of Educational Psychology, 67, 706-716.

Hoeksma, J. B., & Koomen, H.M.Y. (1992). Multilevel models in developmental psychological research: Rationales and applications. Early Development & Parenting, 1, 157-167.

Hox, J. J. (1995). Applied multilevel analysis (2nd ed.). Amsterdam: TT-Publikaties.

Hox, J. J., & Kreft, I.G.G. (1994). Multilevel analysis models. Sociological Methods and Research, 22, 283-299.

Hu, L., & Bentler, P. M. (1995). Evaluating model fit. In R. H. Hoyle (Ed.), Structural equation modeling (pp. 76-99). Thousand Oaks, CA: Sage Publications.

Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55.

Ingels, S. J., Dowd, K. L., Baldridge, J. D., Stipe, J. L., Bartot, V. H., & Frankel, M. R. (1994). National Education Longitudinal Study of 1988: Second follow-up student component data file user's manual (NCES 94-374). Washington, DC: National Center for Education Statistics.

James, L. R., Mulaik, S. A., & Brett, J. M. (1982). Causal analysis: Assumptions, models, and data. Beverly Hills, CA: Sage Publications.

Johnson, D. R. & Elliott, L. A. (1998). Sampling design effects: Do they affect the analyses of data from the National Survey of Families and Households? Journal of Marriage and the Family, 60, 993-1001.

Kaplan, D., & Elliott, P. R. (1997). A didactic example of multilevel structural equation modeling applicable to the study of organizations. Structural Equation Modeling, 4(1), 1-24.

Kaufman, E, Bradby, D., & Owings, J. (1992). National Education Longitudinal Study of 1988: Characteristics of at-risk students in NELS:88. (NCES 92-042). Washington, DC: U.S. Department of Education, Office of Educational Research and Improvement.

Kraemer, H. C. (1978). Individual and ecological correlation in a general context. Behavioral Science, 23, 67-72.

Kreft, I., & Leeuw, J. D. (1998). Introducing multilevel modeling. Thousand Oaks, CA: Sage Publications.

Moritz, S. E., & Watson, C. B. (1998). Levels of analysis issues in group psychology: Using efficacy as an example of a multilevel model. Group Dynamics, 2, 285-298.

Muthen, B. O. (1989). Latent variable modeling in heterogeneous populations presidential address to the psychometric society. Psychometrika, 54, 557-585.

Muthen, B. O. (1994). Multilevel covariance structure analysis. Sociological Methods and Research, 22, 376-398.

Muthen, B. O. (1997). Latent variable modeling with longitudinal and multilevel data. In A. Raftery (Ed.). Sociological methodology 1997 (pp. 453-480). Boston: Blackwell.

Muthen, B. O. (1998). Second-generation structural equation modeling with a combination of categorical and continuous latent variables: New opportunities for latent class/latent growth modeling. In A. Sayer & L. Collins (Eds.), New methods for the analysis of change. Washington, DC: American Psychological Association.

Muthen, B. O., & Satorra, A. (1989). Multilevel aspects of varying parameters in structural models. In D. Bock (Ed.), Multilevel analysis of educational data. San Diego: Academic Press.

Muthen, L. K., & Muthen, B. O. (1998). Mplus: The comprehensive modeling program for applied researchers. Los Angeles: Muthen and Muthen.

Olweus, D. (1991). Bully/victim problems among school children: Basic facts and effects of a school-based intervention program, In D. Pepler & K, Rubin (Eds.), The development and treatment of childhood aggression (pp. 411-418). Hillsdale, NJ: Lawrence Erlbaum.

Raudenbush, S. W., Brennan, R. T., & Barnett, R. C. (1995). A multivariate hierarchical model for studying psychological change within married couples. Journal of Family Psychology, 9, 161-174.

Rigby, K, (1997). Attitudes and beliefs about bullying among Australian school children. Irish Journal of Psychology, 18, 202-220.

Robinson, W. S. (1950). Ecological correlations and the behavior of individuals. American Sociological Review, 15, 351-357.

Rogosa, D., & Saner, H. (1995a). Longitudinal data analysis examples with random coefficient models. Journal of Educational & Behavioral Statistics, 20, 149-170.

Rogosa, D., & Saner, H. (1995b). Reply to discussants: Longitudinal data analysis examples with random coefficient models. Journal of Educational & Behavioral Statistics, 20, 234-238.

Sampson, R. J., Raudenbush, S. W., & Earls, F. (1997). Neighborhoods and violent crime: A multilevel study of collective efficacy. Science, 277, 918-924.

Simpson, R. (1996). Neither clear nor present: The social construction of safety and danger. Sociological Forum, II, 549-562.

Thum, Y. M. (1997). Hierarchical linear models for multivariate outcomes. Journal of Educational & Behavioral Statistics, 22, 77-108.

Vancouver, J. B. (1997). The application of HLM to the analysis of the dynamic interaction of environment, person and behavior. Journal of Management, 23, 795-818.

An earlier version of this article was presented at the Third Annual Conference of the Society for Social Work and Research, January 1999, Austin, Texas. Data for this study were taken from the National Educational Longitudinal Study of 1988, which was conducted by the U.S. Department of Education's National Center for Education Statistics.

Original manuscript received February 17, 1999

Final revision received April 18, 2000

Accepted April 27, 2000

G. Lawrence Farmer, MSW, PhD Assistant Professor School of Social Work Rutgers, The State University of New Jersey New Brunswick, NJ 08903 e-mail: glfarmer@rci.rutgers.edu
COPYRIGHT 2000 National Association of Social Workers
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2000 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Author:Farmer, G. Lawrence
Publication:Social Work Research
Article Type:Statistical Data Included
Date:Sep 1, 2000
Words:6491
Previous Article:Factors associated with the discharge of patients from a long-term state psychiatric hospital.
Next Article:Community activism, effects of the economy and state policies on welfare rolls, and research methodologies.
Topics:

Terms of use | Privacy policy | Copyright © 2018 Farlex, Inc. | Feedback | For webmasters