A Powerful, Potential Outcomes Method for Estimating Any Estimand across Multiple Groups.
To read the full text of this article, click here: http://www.eric.ed.gov/ERICWebPortal/contentdelivery/servlet/ERICServlet?accno=ED563037
Author(s): Pattanayak, Cassandra W.; Rubin, Donald B.; Zell, Elizabeth R.
In educational research, outcome measures are often estimated across separate studies or across schools, districts, or other subgroups to assess the overall causal effect of an active treatment versus a control treatment. Students may be partitioned into such strata or blocks by experimental design, or separated into studies within a meta-analysis. In non-randomized studies, students may be partitioned into subclasses based on key covariates or estimated propensity scores to improve observed covariate balance across treatment groups (e.g., Rosenbaum & Rubin, 1983). Procedures designed to estimate any estimand in the presence of strata, including a simple t-test for the difference in mean outcomes (Neyman, Iwaszkiewicz, & Kolodziejczyk, 1935), rely on implicit assumptions about the unknowable correlation between potential outcomes under active treatment and control treatment. For binary outcomes, the standard procedures used to estimate overall odds ratios in the presence of strata were introduced by Cochran (1954), who first proposed a hypothesis test for the difference in proportions across strata. Mantel and Haenszel (1959) proposed a very similar test and introduced an estimator for a common odds ratio. Consider the following hypothetical studies designed to estimate the causal effect of an existing program on high school graduation: (1) Within each of several school districts, half of the schools are randomized to participate in the program, and half are randomized not to participate in the program. (2) Within each of several cities, schools participating in the program are compared to schools not participating in the program, though participation was not randomized. (3) Several separate evaluations of the program are collected, to be combined in a meta-analysis. In each hypothetical study, a binary outcome (graduation) must be measured over strata (1. school districts, 2. cities, 3. evaluations). The effect of the program on graduation rates may be measured by a difference in proportions, odds ratio, or some other quantity. A hypothesis test will be conducted and a confidence interval constructed for the chosen estimand. The authors propose tests and intervals that can be more powerful for any finite population estimand than traditional tests and intervals, including t-tests for the difference in means or Cochran-Mantel- Haenszel procedures for the odds ratio.
ERIC Descriptors: Computation; Outcome Measures; Statistical Analysis; Graduation; Outcomes of Education; Influences; Simulation; Bayesian Statistics
Publisher: Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; Fax: 202-640-4401; e-mail: firstname.lastname@example.org; Web site: http://www.sree.org
Source: Society for Research on Educational Effectiveness
ERIC Number: ED563037
Record Type: Non-Journal
Publication Type: Reports - Research
|Printer friendly Cite/link Email Feedback|
|Author:||Pattanayak, Cassandra W.; Rubin, Donald B.; Zell, Elizabeth R.|
|Date:||Jan 1, 2013|
|Previous Article:||Making Sense of the "Zero Effect" of Comprehensive Teacher Induction Programs: A Mediational Analysis.|
|Next Article:||Accounting for One-Group Clustering in Effect-Size Estimation.|