Achieving alignment: the need for standards and assessments to work together to guide student learning has never been greater. Here's how schools can investigate and demonstrate alignment, and plan for and use the results of alignment studies.

More and more, districts and states are focusing on the need for alignment between the standards that exemplify expectations of student learning and the assessments used to measure that learning. This growing concern about alignment owes much to the No Child Left Behind Act of 2001 and, more generally, to the escalating use of student assessment results for schools, teachers and students.

Faced with myriad new assessment and accountability requirements, districts and states must address head-on many alignment issues that have previously received less attention. This article describes the many ways to conceptualize, investigate and demonstrate alignment--particularly those brought to the fore by NCLB. It ends with advice to decision-makers about how to plan for and use the results of targeted alignment studies.

What is alignment?

Simply stated, alignment means agreement. In this case, it refers to the degree to which standards, assessments and other important elements in an education system are complementary and work together to effectively guide student learning (Webb, 1997). As such, the concept of alignment is intricately connected to test validity, because the notion that accurate inferences can be drawn from test scores assumes proper alignment of standards and assessment.

What does NCLB say and suggest about alignment?

NCLB requires states to adopt grade-specific content standards, to do annual testing for grades 3 through 8, and to report assessment results by standards. While the law allows states to select or develop an assessment of their choosing (including designing their own criterion-referenced test, augmenting existing norm-referenced tests, or using comparable local assessment tools), the caveat is that whatever assessment, are used must be aligned with the state's academic standards. These requirements underscore the need for meaningful alignment of standards to assessment and articular of both across grade levels.

For states electing to use a norm-referenced test, the NCLB regulations require an independent alignment study relative to the state's content standards to identify missing content. If there is missing content, the assessment must be augmented accordingly.

NCLB's accountability provisions also raise interesting alignment issues. Briefly, NCLB requires states to set up a single, statewide accountability system in which, by the end of 12 years, all students at all grade levels will be expected to score a state-defined "proficient" level on the state assessment. Furthermore, in moving towards this goal of 100 percent proficiency, each state must show a gradual improvement in student performance occurring in equal increments over time.

In planning compliance, many states are considering how to establish appropriate passing scores on assessments for different grade levels. Doing so requires serious attention to alignment. Misalignment between standards across different grades or between standards and assessments within particular grade could result in a state setting inappropriate passing scores, leading to variable proficiency rates for students at different grade levels, having widely different proficiency rates a different grade levels would be hard for a state or district to explain or defend.

Another significant alignment issue is raised by NCLB's required use of the National Assessment of Educational Progress as a point of comparison for state's results on its own test. Beginning in 2002-03, a sample of fourth and eight graders in each state must participate in NAEP reading and math every other year. Although states will not be sanctioned for low NAEP performance, such results could raise questions about the quality of a state's testing program, particularly if the NAEP results differ significantly from the state test results.

Implementation of NCLB raises significant challenges for those seeking alignment between their assessment and accountability systems (Carlson, 2002). For example, NCLB presents conflicting timelines for assessment and accountability requirements. The law mandates that states establish their accountability baseline measurement (i.e., adequate yearly progress) 2001-02, yet it does not require states to begin implementing any new assessment until 2005-06. This means states must establish baseline measurements using existing assessments, then switch to the new measures midstream as they pursue their goal of 100 percent student proficiency within 12 years. In light of this switch, the consistency and continuity of accountability decisions over time (e.g., which schools get identified as successful or in need of improvement) may be difficult to maintain and defend.

What types of alignment studies are decision-makers asking for?

Alignment studies can be formal or informal, exploratory or confirmatory in nature, and they can be designed to serve any of a variety of purposes. When districts and states ask for an alignment study, they want it to accomplish one or some combination of the

* Identify areas of vulnerability. Faced with ambitious timelines for implementation of assessment and accountability systems under NCLB or other mandates, several districts and states have sought targeted, short-term alignment studies. These studies are geared to quickly providing answers to questions such as: Where are we vulnerable? Where are our content gaps and other potential shortcoming? Many education agencies are willing to forego complex alignment research designs and detailed final reports in exchange for obtaining global information about their content gaps that allows them to quickly begin strategizing next steps.

* Inform restructuring of an existing assessment or accountability system. Whereas some alignment studies are designed only to signal possible problems, others are intended to provide more specific and detailed information to help districts and states decide whether to restructure their existing systems, or how to go about restructuring them. Evidence of strong alignment might argue for slight changes to an existing system, whereas evidence of weak alignment might argue for major restructuring or even building a new system from scratch.

* Compare own standards and assessments to others. Districts and states sometimes want to know how their standards and assessments compare to those in other systems, building such comparisons into their alignment studies. If comparisons are desired, districts or states must decide on the most appropriate basis for comparison. In some instances, they want to know how they compare to standards and assessments that are known to be exemplary. In other instances, districts want to know how their local standards and assessments compare with those of their state. Armed with such information, a district can, for example, decide whether there is any content covered by the state standards and assessments that the district wants to reinforce or supplement with its own local standards and assessments. Similarly, a district may want to conduct a study to ensure that its instruction and curriculum are sufficiently aligned to the state standards and assessments so that students are properly prepared for the statewide assessments.

* Inform future assessment item development activities. Alignment studies can serve the practical purpose of guiding the augmentation of assessment item pools. When designed appropriately, an alignment study can identify any gaps or surpluses in item pools relative to targeted standards as specified by assessment plans or blueprints. Such information can be extremley helpful to a district or state in planning its future item development activities.

* Provide evidence of content validity from an external source. Sometimes districts and states want to obtain confirmatory evidence of the content validity of their assessments from an independent expert source--someone other than their own staff or development contractors. They may seek this evidence for their own purposes or because it's been requested by others, such as the state legislature or the board of education. A common reason for commissioning an alignment study of standards and assessments is to slow compliance with district, state or federal mandate.

How is alignment achieved?

Alignment can be achieved through use of sound standards- and assessment-development practices, with a focus on alignment in each step in the development process. For example, one way to build in alignment during standards development is to design exemplary assessment items for each standard. These exemplary items then serve as concrete examples of the desired relationship of a given standard to its assessment. These items could then be used to inform subsequent assessment development, effectively bridging the standards- and assessment-development process.

Test blueprints and item specifications can help ensure alignment by clearly stipulating the connections of standards and objectives to assessment content and format. As an alternative to test blueprints and item specifications, some advocate use of a test map, which is much more informative. A test map explains in detail for teachers and test developers what is "fair game" to be included in the test, enumerating content and cognition limits, assessment formats, how test items will be sampled and what subscores will be developed and reported (Schafer, in progress).

In preparation for actual assessment development, item writers must become intimately familiar with the targeted standards and test blueprint (or test map). When writing items, they should code their draft items to the appropriate standards. During the item review process, content experts should also evaluate the closeness of the relationship between each assessment item and the standard(s) it is intended to measure. Item found to have weak or superficial connections with the targeted standards should be revised or dropped, once as assessment is fully developed, a formal alignment analysis should be conducted.

What is sufficient alignment between standards and assessments?

Conventional wisdom holds that sufficient alignment between standards and custom-developed criterion referenced tests is nearly guaranteed. However, this assumption does not always stand up under scrutiny. In fact, a number of researchers now assert that states have a lot of work to do to bring their standards and CRTs into alignment (Webb, 1999). That said, there is no hard and fast rule about what constitutes sufficient alignment. Defining sufficient alignment is complicated by a number of factors.

First, when articulating expectations for what students should learn (what they should know and be able to do), it is common for states to have different levels of statements, ranging from more global statements (California's domains) to narrower, more targeted statements clustered under the broader statement (California's content standards). Some assessment are designed to align global-level expectations, while other are designed to align to the narrower and more specific expectation statements.

If a district or state has designed its assessment to align to standards at a very specific level, an alignment study focusing on the relationship between standards and assessment at a global level will not allow for meaningful conclusions about sufficient alignment.

Non-alignment may also come up as an issue when standards are not amenable to measurement. For example, "students develop appreciation for reading" may be a worthwhile standard around which to design curriculum, but it is a difficult standard to measure. Similarly, some standards that call for students to produce extensive work products or projects (laboratory experiments, case studies) may not be accommodated easily in large-scale assessment, but may be appropriate for performance-based classroom assessment.

Weak alignment of standards and assessments can also result from flaws in the standards, the assessments, or both. Many alignment studies have uncovered deficiencies in standards--such as lack of clarity in particular standards or redundancy of content across different standards--that make it difficult to develop items that are sufficiently aligned. Similarly, it is often difficult to evaluate the match of ill-constructed assessment items to particular content standards.

Tests for multiple grade levels

Another complication in determining whether alignment is sufficient comes up when tests are intentionally designed to align to standards at more than one grade level. For example, high school exit exams are intended for high school students, yet many of these exams include items targeted to standards at both the high-school level and lower-grade levels (e.g., eight grade computation). Sometimes states deliberately opt for non-alignment to grade-level standards when field test data shows that large numbers of students would otherwise fail the high school exit exam. In these cases, a decision is made to remove harder item (e.g., algebra, geometry), even though the result is a test less aligned to grade-level content standards.

Finally, as Porter (2002) points out, tests comprise a sample of items from a domain, whereas standards make up the domain. "Thus, perfect alignment should not be expected." Most sets of content standards cover far more content than any given student could be expected to master. In fact, in many states, content standards were developed primarily to guide and evaluate instruction at the classroom level, not to serve as the learning. The comprehensiveness of such standards places real limitations on the extent to which standards and assessments can be aligned.

Given these complicating factors and the general lack of agreement about what constitutes sufficient alignment, determining whether or not system components are sufficiently aligned is somewhat arbitrary. Pursuing fuller alignment often involves strategic trade-offs that may require more testing (due to increased test length) and resources (money for developing and scoring open-response items) that are readily available to a district or state.

How is alignment studied?

In the past, many alignment studies, using different and sometimes unclear methodologies, were criticized as being arbitrary and subjective. However, new methodologies that show great promise are now emerging (Council of Chief State School Officers, 2002). Most entail a systematic review of the standards by content experts, followed by their systematic review of assessment items and tasks. These experts are trained to judge alignment against a specific set of alignment criteria and decision rules. More than one reviewer is used to judge alignment between any given set of standards and assessments, and agreement among reviewers is monitored.

Webb (1997, 1999), for example, has written extensively about an alignment analysis process that calls for content experts to use the following alignment criteria:

* Categorical concurrence: How similar are the categories of content in the standards and assessments?

* Depth-of-knowledge consistency: To what extent is the knowledge tapped by the assessment as cognitively demanding as that reflected in the standards?

* Range-of-acknowledge consistency: To what extent is the span of knowledge represented in the assessments comparable to that represented in the standards?

* Balance of representation: How evenly are the assessment items distributed across the various objectives within a standard?

The approach that Achieve, Inc. takes to examining alignment uses criteria that overlap with Webb's, including: content centrality, performance centrality, challenge, balance and range (Achieve, 2002). In addition to providing qualitative and quantitative analysis on the alignment of a state's assessments to its content standards, Achieve offers additional services, such as comparison of norm-referenced tests to a state's content standards.

The Council for Basic Education also has its own alignment process where reviewers work in pairs, applying a scoring rubric and exemplars to evaluate the degree of match between test items and standards (or benchmarks). The four alignment criteria they focus on are content, content balance, rigor and item response type.

WestEd's approach to alignment studies is a highly collaborative process. WestEd actively engages the district or state whose standards and assessments are being studied in determining alignment criteria and decision rules. This helps ensure that the results of the study will provide the specific types of information that each particular district or state needs in order to improve its systems, consistent with the stated purposes of the educational reform model its standards and assessment were developed to support.

Whereas most approaches to alignment studies do not allow for direct state-to-state comparisons of how well standards and assessments are aligned, Porter (2002) offers innovative and promising alignment indices that are expressly used to compare alignment within and between states. These indices allow such comparisons by mapping standards and assessments from different states on to a common content language.

When considering possible alignment approaches and criteria, states and districts need to determine for themselves whether comparisons to other states are relevant for their particular alignment purposes.

Using alignment studies

In conclusion, designers and consumers of alignment studies should consider the following steps that cover both the planning of an alignment study and use of its results to inform practice:

* Be clear about the purpose(s) of the alignment study and how the results will be used.

* Select a methodology that matches your purpose and resources, remembering that not all methods lead to the same conclusions.

* Determine appropriate alignment criteria and decision rules.

* Review the results of your alignment study to determine if there is evidence of sufficient alignment.

* Secure resources and approvals necessary to implement changes to your system based on the findings of your alignment study.

* Refine your system by building alignment practices into front-end processes, and monitor alignment throughout the life of your system.

(This article is adapted from Ananda's 2003 WestEd Knowledge Brief entitled "Rethinking Issues of Alignment Under No Child Left Behind.")

References

Achieve, Inc. (2002). Benchmarking. www.achieve.org/achieve.nsf/Benchmarking.

Carlson, D. (2002). "Toward a structure for evaluating and improving the validity and reliability of state accountability systems." In S. Erpenbach (Ed.), Making valid and reliable decisions in the determination of adequate yearly progress. Washington, D.C.: Council of Chief State School Officers.

Council of Chief State School Officers. (2002). Models for alignment analysis and assistance to states. Online at www.ccsso/pdfs/AlignmentModels.pdf.

La Marca, P. M. (2001). "Alignment of standards and assessments as an accountability criterion." Practical Assessment, Research & Evaluation, 7(21). Online at ericae.net/pare/getvn.asp?v=7&n=21.

Messick, S. (1989). "Validity." In R. L. Linn (Ed.), Educational measurement (3rd Edition). New York: American Council on Education. Macmillan Publishing.

Porter, A. C. (2002). "Measuring the content of instruction: Uses in research and practice." Educational Researcher, 31 (7).

Schafer, W.D. (2002). Describing assessments for teaching and learning. Conference on Optimizing State and Classroom Tests: Implications of Cognitive Research for Assessment of Higher Order Reasoning in Subject-Matter Domains, University of Maryland.

Smith, M. S., & O'Day, J. (1991). "Systemic school reform." In S. H. Fuhrman & B. Malen (Eds.). The politics of curriculum and testing: The 1990 yearbook of the Politics of Education Association. Bristol, PA: Taylor & Francis.

Webb, N. L. (1997). Research monograph no. 6: Criteria for alignment of expectations and assessments in mathematics and science education. Washington, D.C.: Council of Chief State School Officers.

Webb, N. L. (1999). Alignment of science and mathematics standards and assessments in four states. Washington, D.C.: Council of Chief State School Officers.

WestEd. (2002). A comparison of NAEP content to reading, writing, and mathematics content standards for Arizona, California, Nevada, and Utah. San Francisco: Author.

Sri Ananda is chief development officer at WestEd.