Recommendations for evaluating accreditation guideline outcomes assessment methods for accredited environmental health programs in the United States.

Introduction

A Brief History of EHAC 

The National Environmental Health Science and Protection Accreditation Council (EHAC) is an independent organization whose mission is to accredit environmental health programs offered in national and regional institutions of higher education. These programs prepare students for entry into careers in environmental health, environmental protection, and industrial hygiene (EHAC, 2003).

EHAC was established in 1967 as the National Accreditation Council for Environmental Health Curricula. It is an autonomous organization that is related to and works closely with the National Environmental Health Association, from which it evolved. The purpose of the council is to enhance the education and training of students who intend to become environmental health science and protection professionals. The council is composed of highly qualified professionals representing academic, government, and industrial communities. Over its 28-year history, it has consistently worked to upgrade the quality of education and training delivered by the programs it accredits (EHAC, 2003).

EHAC is the only accrediting agency in the United States for environmental health programs. The council has accredited 27 undergraduate and graduate programs in both public and private institutions across the United States. Since 1989, accredited programs have placed over 1,500 well-qualified graduates into the environmental health workforce. Graduates of accredited programs hold positions with federal, state, and local health agencies; the military; the U.S. Public Health Service; various universities; and numerous companies, including many Fortune 500 companies (EHAC, 2003).

Accreditation indicates that an academic program has the curriculum, faculty, facilities, and institutional support necessary to provide quality education in environmental health science and protection. Only students from accredited programs are eligible to participate as sanitarians in the Commission Officer Student Extern Training Program (COSTEP) of the U.S. Public Health Service (USPHS). Another benefit for students who graduate from an accredited program is that those students have priority in USPHS hiring practices. Many states recognize graduation from an accredited program as meeting a specific standard needed to register environmental health specialists or sanitarians (EHAC, 2003).

Numerous accrediting agencies throughout the United States are "recognized" (see definitions below) either by the U.S. Department of Education (USDE) or by the Council for Higher Education Accreditation (CHEA). Currently. EHAC is not recognized by either entity. This paper was written to provide guidance to EHAC on outcomes assessment and other areas related to EHAC's consideration of pursuing recognition by either USDE or CHEA. In recent years, EHAC has been working to obtain comprehensive and systematic information about key elements related to environmental health science and protection and the accreditation process.

Definition of Accreditation and Recognition

For the purposes of this article, two important terms are defined: accreditation and recognition.

"Accreditation" is a process of external quality review used by the higher education community to scrutinize colleges, universities, and educational programs for quality assurance and quality improvement. In the United States, accreditation is carried out by private, nonprofit organizations designed specifically for this purpose (CHEA, 2001).

"Recognition" is a process of external quality review of accrediting organizations to affirm their quality and effectiveness. In the United States, recognition is carried out by a federal agency, the U.S. Department of Education (USDE), and by a private organization, the Council for Higher Education Accreditation (CHEA) (CHEA, 2001).

USDE Requirements for Outcomes Assessment

For more information on the USDE procedures and criteria for the recognition of accrediting agencies, visit the agency's Web site at http://www.ed.gov/admins/finaid/accred/index.html and click on the "Secretary's Recognition of Accrediting Agencies" link. The site does not have a section that addresses outcomes assessment specifically; rather, references to outcomes assessment are woven in with discussions of other criteria.

CHEA Requirements for Outcomes Assessment

CHEA is a private nonprofit national organization that coordinates accreditation activity in the United States. It represents more than 3,000 colleges and universities and 60 national, regional, and specialized accreditors. As of 2000-2001, 57 accreditors were recognized by USDE, 59 accreditors were recognized by CHEA or undergoing a CHEA recognition review, and 38 of these accreditors are recognized by both USDE and CHEA or are undergoing a CHEA review (CHEA, 2001; Ewell, 2001).

To view CHEA's recognition standards, visit its Web site at http://www.chea.org/About/Recognition.cfm#11.

CHEA emphasizes that the accrediting agency must have expectations of individual programs that can be assessed. That is, expectations should be measurable. As on the USDE Web site, the CHEA site does not have a section that addresses outcomes assessment specifically; rather, references to outcomes assessment are woven in with discussions of other criteria.

Current EHAC Program Outcomes Assessment Methods

At the time of re-accreditation, institutions are required to survey program graduates and employers using EHAC's outcome assessment tools. (To receive a copy of these assessment tools, please contact the corresponding author. Contact information is located at the end of the article.) All students who have graduated since the last accreditation must be in the pool of those to be surveyed. The completed tools are gathered by the institution and forwarded to the EHAC executive director six months prior to the annual meeting of the council. EHAC supplies a summary of the information gathered to all accredited programs on an annual basis. The purpose of these surveys is to determine the adequacy of the accreditation process mandates to the needs of the professional practice of environmental health. The information gathered by an institution through the outcomes assessment process is not used as part of a self-study for re-accreditation purposes for a given institution. Rather, EHAC uses the information compiled from all institutions undergoing re-accreditation to evaluate and modify the requirements for accreditation (EHAC, 2003). The graduate and employer surveys are a key component in a successful outcomes assessment strategy, and they are an innovative approach to collecting critical information that will be beneficial to individual programs and to EHAC as an accrediting agency.

What Are Other Accrediting Agencies Doing?

In assessing EHAC's accreditation standards for outcomes assessment, it is important to look at what other accrediting agencies are doing. The authors chose two agencies currently recognized by USDE--the Council for Education on Public Health (CEPH) and the Transnational Association of Christian Colleges and Schools (TRACS).

Council for Education on Public Health

CEPH was chosen in large part because of the authors' familiarity with this accrediting agency. For more information on CEPH's criteria for accreditation procedures, visit its Web site at http://www.ceph.org. The site does not have a section that addresses outcomes assessment specifically; rather, references to outcomes assessment are integrated among the criteria.

CEPH requires institutions to adhere to two standards in particular:

* Institutions are required to have clear, measurable learning objectives defining the competencies students should obtain by completing the program.

* Institutions are required to have a method for assessing students' level of attainment in comparison with stated program competencies; in other words, the degree of attainment by the student should be measured against the expected program competencies.

Assessing students' level of attainment in comparison with stated program competencies is a formidable task. The goal of an outcomes assessment process is appropriate documentation that can demonstrate to a third party that each of the stated objectives and measures has been met and to what degree, with any variance explained. One of the more comprehensive methods for achieving this goal is to monitor and report student pass rates and scores on national certification or licensure examinations.

Outcomes assessment is a complicated issue in a field such as public health, in which the profession per se is not licensed. Graduates of public health programs may elect to pursue various certifications (e.g., Certified Health Education Specialist, among others), but this arrangement is far different from that of professions such as nursing that require licensure prior to practice. In public health, the only licensure requirement is in environmental health, and that license is required only by a few states. California, Colorado, and Illinois are among the few states that require a professional examination in environmental health. In these states, outcomes assessment procedures could require the degree-granting academic institution to examine, assess, and report the pass rates on the state registration or licensing examination, although the data may be somewhat difficult to obtain.

Some of the most common outcome assessment techniques utilized by CEPH-accredited programs are the assessment and reporting of graduation, degree, and attrition rates and of professional employment rates and locations. If these rates are lower than 80 percent of the program goal, an explanation is required. CEPH chose the value of 80 percent not to reflect a standard of "good or bad" but to establish a point for explanation. For example, a part-time-only program of study could explain legitimately high attrition and slower graduation rates.

Any USDE- or CHEA-recognized accrediting body will agree that meeting the reporting requirements for securing and maintaining recognition is a tremendous task. CEPH has a budget sufficient to maintain a core full-time staff necessary to provide an ongoing professional accreditation body.

Transnational Association of Christian Colleges and Schools

The second accrediting agency that was contacted was the Transnational Association of Christian Colleges and Schools (TRACS). Located in Forest, Virginia, TRACS is highly recognized for meeting the rigorous guidelines of national accrediting boards such as USDE and CHEA. TRACS recently completed a successful recognition review by USDE. Its outcomes assessment methods were highly praised by USDE review officers.

TRACS is a national accrediting body for Christian institutions, colleges, universities, and seminaries. The board is a voluntary, nonprofit, self-governing organization that provides accreditation to Christian post-secondary institutions offering certificates, diplomas, or degrees. TRACS has established a comprehensive document called Benchmarks for Excellence to serve as a) a frame of reference in the evaluation of institutional compliance with the TRACS standards. b) an aid to each institution in the self-evaluation process, and c) an indicator of quality characteristics in critical areas. It took approximately five years of intensive work to refine the standards, now referred to as "benchmarks" (TRACS, 1995). A psychometric expert was employed to conduct the ongoing process and the study of reliability and validity used for accreditation. The benchmarks include the establishment of behavioral examples to measure the levels of performance. According to the TRACS handbook, two categories that fall under the standards profile are a) foundational standards and b) operational standards. These categories comprise 26 subcategories that are measured in the TRACS Institutional Evaluation Profile. The following five ranking measures are used to evaluate each of the 26 subcategories: 5--Far Exceeds the Standard, 4--Exceeds the Standard, 3--Meets the Standard, 2--Falls Below the Standard, and 1--Fails to Meet the Standard.

In addition to goals and objectives that TRACS has established for each subcategory, it also imposes the use of "must" statements in its policies and procedures manuals, as opposed to "should" statements. Thus, the likelihood of compliance with individual goals and objectives is predicted or ensured. USDE is interested in process, but its primary concern is the outcome of the educational process.

One practical approach to conducting outcome assessments is collaborative planning. According to Dr. Russell Fitzgerald of TRACS, "Cooperation, vision, and resources among the various departments and colleges of the university are crucial in the planning and assessment process." Included in the planning process is the determination of goals and objectives. The goals and objectives are the cornerstone from which the assessment will be derived. When one undertakes to assess outcomes of student preparedness or qualifications, one should expect that the process, if it is to be effective, will be comprehensive and ongoing.

Discussion

The goal of accreditation of undergraduate environmental health science and protection programs is to enhance the education and training of students who intend to become environmental health science and protection professionals (EHAC, 2003). The focus is twofold, on 1) students as future environmental health practitioners and 2) professional preparation.

An important question for accrediting agencies is how to determine if the program and institution have adequately prepared or qualified students to enter the profession Questions about program quality and student preparedness can be answered with methods such as process and outcome evaluation or assessment. The terms evaluation and assessment are used interchangeably in this article.

Process evaluation is briefly discussed in this paper to distinguish it from the outcome assessment endorsed by USDE. Process evaluation procedures enable the review of instruction components before, during, and after the instruction is completed. For example, in environmental health, the instruction is reviewed from various angles such as the quality and quantity of materials used, the consistency and quality of instructors, and the social and environment conditions conducive to learning. Observation and immediate feedback, or process evaluation tools, provide the opportunity to make changes in materials, instruction, and environment before the closing stages of the instruction are reached (Simons-Morton, Greene, & Gottlieb, 1995). Changes made in these areas also will affect the outcome of measurement because the program is modified to better attain the stated goals.

Comments About USDE

USDE regards "success with respect to student achievement" as the single most important portion of accreditation standards for recognized accrediting agencies (Kershenstein, 1998).

USDE-recognized accrediting agencies have some common features (Kershenstein, 1998):

* The agencies either strongly encourage or require a variety of approaches to outcomes assessment by the institutions.

* A fairly heavy emphasis is placed by the agencies on the use of completion rates, job placement rates, and pass rates on state licensing exams in the assessment of an individual institution's or program's success with respect to student achievement.

* The agencies conduct fairly extensive monitoring of these outcomes rates throughout the accreditation period, with follow-up as warranted.

* The agencies place a heavy emphasis on follow-up among graduates from the institution or program.

* The agencies are actively involved in the analysis of the outcomes rates agencywide, using the aggregate data in making a decision about a particular institution or program.

USDE-recognized accrediting agencies that are in need of strengthening their compliance with respect to outcomes assessment also have some common features (Kershenstein, 1998):

* lack of true standards for outcomes assessment,

* lack of clear standards for outcomes assessment,

* lack of appropriate outcomes measures, and

* lack of multiple outcomes assessment measures.

In general, USDE appears to view pass rates on licensing or certification exams as a significant measure for outcomes assessment, particularly if programs evaluate subscores for specific topic areas (i.e., whether the curriculum was perhaps adequately preparing students in one area but not in others) (Kershenstein, 1998).

USDE looks favorably upon periodic "micro" and "macro" reviews of accreditation standards. The micro review is an indepth look (usually annually) at the existing standards to find out what's working, what's not working, what's well understood by the institutions and programs, and what's often misunderstood. Often the micro review includes an in-depth analysis of all team reports from the past year, looking for commonalities in problems identified. The macro review of accreditation standards takes a broader view. Periodically, agencies should ask themselves, "What does it really mean today to have a quality curriculum in field X, or what does it really mean today to assess success with respect to student achievement in our niche of the accrediting world?" (Kershenstein, 1998).

USDE makes the following recommendations for accrediting agencies (Kershenstein, 1998):

* a standards review committee established in the bylaws,

* a systematic and ongoing process for review of accreditation standards (as opposed to major overhauls on an infrequent basis),

* provision of multiple opportunities for input to accreditation standards by the relevant constituencies,

* the use of multiple surveys targeted to a specific audience,

* self-studies of the agencies themselves, and

* trained evaluators accompanying site teams to evaluate the effectiveness of the on-site review process.

Comments About CHEA

The task of addressing student learning outcomes has no single or simple answers; however, it is imperative that accreditors address them. What does this process require? Three lines of collective action seem warranted (Ewell, 2001).

First, the accrediting community needs to develop a coherent and understandable way to explain its collective approach toward student learning outcomes to outside stakeholders. This rationale must make it clear that while it is appropriate for differences to exist in how accreditors choose to engage student learning, all accreditors should be doing so in appropriate and rigorous ways. It also must make clear that accreditors are requiring some measures of direct student attainment--not merely proxies for it--in their various approaches (Ewell, 2001).

Second, accreditors need a language with which to talk to one another about what they are doing. Such discussion may or may not require the development of a common vocabulary. It does require a common conceptual framework that allows various members of the community to understand the key distinctions and similarities among their approaches (Ewell, 2001).

Third, all would benefit from the development of common resources. These might include sharing review approaches and techniques like standards of evidence or ways of auditing institutional or program assessment efforts. They also might include more fundamental research and development efforts directed toward the creation of better tools for examining, learning, or identifying best practices. There is a growing fund of experience to build on here, both within and outside the accreditation community. The growing body of experience has not yet been effectively "rounded up," though, for use by the community in common (Ewell, 2001).

Student learning outcomes are rapidly taking center stage as the principal gauge of higher education's effectiveness. Accrediting agencies have responded to the growing salience of learning outcomes in a variety of ways. Virtually all now include explicit references to student learning in their standards for accreditation. Most also require institutions or programs to examine student achievement or institutional effectiveness as part of their self-study or review process--usually in the form of some kind of assessment. What is meant by assessment often varies greatly, ranging from job placement and student satisfaction to self-reported gains in skill and knowledge on the part of students and former students. None of these constitute direct evidence of student learning outcomes, however. Accrediting agencies need to address at least five key questions in order to effectively incorporate student learning outcomes into accreditation review and in the determination of institutional and programmatic quality (Ewell, 2001):

1. What is a student learning outcome?

2. What counts as evidence of student learning?

3. At what level should evidence of student-learning outcomes be sought?

4. To what extent should particular student-learning outcomes be specified by accreditors?

5. What models are available to accreditors when they are choosing an approach?

Table 1 provides a comparison of the various components of outcome assessment and shows how these components differ among constituents.

Recommendations

The following recommendations were compiled by the authors on the basis of their review of EHAC, CEPH, TRACS, USDE, and CHEA. No rank-ordered priority is implied.

* Conduct a detailed agency self-study. Both USDE and CHEA specify numerous recognition criteria that accrediting agencies must satisfy before recognition is provided. On a periodic basis, EHAC should conduct a detailed agency self-study to ensure that current EHAC accreditation criteria satisfy the recognition criteria specified by USDE, CHEA, or both.

* Require programs to provide degree completion rates and attrition rates. The current EHAC outcomes assessment criteria do not set expectations in two vital professional-degree program areas as identified by CEPH: 1) degree completion rates and 2) job placement rates. If the outcome measures selected by the program do not include degree completion rates and job placement rates, then data for these two indicators must be provided, including experiential data over the last three years. If degree completion rates in the normal time period for degree completion are less than 80 percent, an explanation must be provided.

* Require programs to provide external validation (e.g., national certification pass rates). The one most consistently lauded form of outcomes assessment is student (or recent alumni) pass rates on state or national certification or licensing examinations.

* Require programs to identify student outcome competencies. Both CEPH and TRACS, throughout their accreditation criteria, require programs to define programmatic goals, objectives, competencies, and sub-competencies. Programs are then asked to demonstrate how these are measured. The method of assessment should measure the student's level of attainment compared with stated program competencies.

* Hire a permanent full-time director and demonstrate sufficient monetary resources for an ongoing core professional accreditation staff. Compared with accrediting agencies such as CEPH and TRACS. EHAC is relatively small and underfunded. It seems unlikely that EHAC will achieve USDE or CHEA recognition until these two items are addressed.

* Combine with an existing accrediting agency. EHAC may wish to consider in detail the advantages and disadvantages of joining with another accrediting agency, particularly one that is already recognized by USDE, CHEA, or both.

* Quantify expectations of programs. What does EHAC expect of accredited environmental health programs? The level of expectation needs to be made clear in all areas of the accreditation criteria. In addition, programs should be required to demonstrate that the curriculum is aligned with university and profession competencies and objectives.

TABLE 1 A Taxonomy of Terms Commonly Used in Connection with the
Assessment of Student Learning Outcomes*

Units of     Ways of Looking
Analysis     at Performance

Institution  Efficiency, effectiveness, productivity
Program      Output, productivity
Student      Outcome

Units of     Ways of Looking          Ways to Review
Analysis     at Outcomes              Performance

Institution  Behaviors (e.g.,         Evaluation
             employment, further
             education, career
             mobility, income)
Program      Satisfaction             Measurement, indicator, assessment
Student      Learning (e.g.,          Evidence of achievement (e.g.,
             knowledge, skill,        examination, performances, student
             ability, attitude,       work)
             disposition),
             attainment, development

* Based on terms discussed in Accreditation and Student Learning
Outcomes: A Proposed Point of Departure (Ewell, 2001).

Acknowledgements: This paper was commissioned by the National Environmental Health Science and Protection Accreditation Council (EHAC) in response to its call for proposals (Request for Proposals Part III: Tools Useful in Evaluating Accreditation Guideline Outcomes), which resulted from funding provided by the Centers for Disease Control and Prevention National Centers for Environmental Health. The purpose was to obtain comprehensive and systematic information about key elements related to appropriate environmental health science and protection accreditation. The paper was presented at the March 2002 mid-year meeting of EHAC in New Orleans, Louisiana.

REFERENCES

Council for Higher Education Accreditation. (2001). Fact sheet #1: Overview of accreditation. http://www.chea.org/Research/index.cfm (24 July 2003).

Ewell, P.T. (2001). Accreditation and student learning outcomes: A proposed point of departure. Council for Higher Education Accreditation. http://www.chea.org/Research/CHEA%20pubs%20order%20form.pdf (24 July 2003).

Kershenstein, K.W. (1998, March). Validity and reliability--Flexing within the rules. Speech given at the annual meeting of the Association of Specialized and Professional Accreditors, Chicago, IL. National Environmental Health Science and Protection Accreditation Council. (2003). Guidelines for the accreditation of environmental health science & protection baccalaureate programs. http://www.ehacoffice.org (24 July 2003).

Simons-Morton, B.G., Greene, W.H., & Gottlieb, N.H. (1995). Introduction to health education and health promotion (2nd ed.). Prospect Heights, IL: Waveland Press.

Transnational Association of Christian Colleges and Schools (2002). http://www.tracs.org (24 July 2003).

Transnational Association of Christian Colleges and Schools (1995). Benchmarks for excellence: A frame of reference in the evaluation of institutional compliance with the TRACS standards. http://www.tracs.org/benchmarks.htm (24 July 2003).

Stephen D. Arnold, Ph.D.

Charles T. Kozel, Ph.D.

Lily D. Velarde, Ph.D.

Corresponding author: Stephen D. Arnold, Department of Health Science, MSC 3HLS. New Mexico State University, Las Cruces, NM, 88003. E-mail: [e-mail].