Comprehensive guidance and counseling programs: the evolution of accountability.

This article traces the evolution of accountability from the 1920s to 2003. Attention is given to expressions of concern about the need for accountability as well as recommendations for school counselors about how to be accountable. Then a sampling of empirical studies that provide evidence of the impact of guidance and counseling programs is presented. The article closes with the challenge of accountability for today and tomorrow and presents several themes that have appeared consistently in the literature that identify prerequisite conditions that must exist if accountability is to be achieved.

**********

Today the issue of accountability is in the forefront of professional dialogue (Dahir & Stone, 2003; Gysbers & Henderson, 2000; Isaacs, 2003; Johnson & Johnson, 2003; Myrick, 2003). School counselors, working within the framework of comprehensive guidance and counseling programs, increasingly are being asked to demonstrate that their work contributes to student success, particularly student academic achievement. Not only are school counselors being asked to tell what they do, they also are being asked to demonstrate how what they do makes a difference in the lives of students.

Is the focus on accountability a new phenomenon or has our profession always been concerned about assessing the effects of the work of school counselors? The purpose of this article is to answer this question by tracing the evolution of accountability as documented in professional literature. The story begins in the 1920s, soon after guidance and counseling was introduced in the schools as vocational guidance in the early 1900s. A sampling of literature in each decade from 1920 through 2003 was reviewed, first for evidence of expressions of concern about the need for accountability, and then, for any recommendations that school counselors could follow to be accountable. The results of this review are presented in the first part of the article. Then, a sampling of literature for the same period of time was reviewed for empirical studies that provided evidence that comprehensive guidance and counseling programs have had an impact on the lives of students. The results of this review are presented in the second part of this article. Finally, the last part of the article identifies several themes from the literature that describe the necessary prerequisite conditions for accountability to occur.

CONCERNS AND RECOMMENDATIONS ABOUT ACCOUNTABILITY

The 1920s

Before the 1920s, the work of professionals focused on establishing guidance and counseling (then called vocational guidance) in the schools. Rapid progress was made and this progress carried over into the 1920s. By the 1920s, however, concern about accountability was beginning to be expressed in the literature as indicated by this statement by Payne (1924):

      What method do we have of checking the
   results of our guidance? For particular groups
   was it guidance, misguidance, or merely a contributing
   experience? We simply must work
   out some definite method of testing and
   checking the results of our work. If we do not,
   some other group will, with possibly disastrous
   results for our work. (p. 63)

During the same year, Edgerton and Herr (1924) described the efforts of school districts in 143 cities across the United States to provide organized educational and vocational guidance activities in their respective school systems. They did not describe any studies that had been done to assess the impact of these activities, but they did identify some outcomes they thought were achieved.

      As an outcome of these serious endeavors to
   meet current and changing demands for purposeful
   instruction and systematic guidance, it
   is found that marked increases in interest,
   ambition, and school attendance often follow
   the inauguration and development of suitable
   courses of study and their accompanying guidance
   programs. (p. 5)

Much of the work on accountability in the 1920s focused on establishing standards for judging whether or not a guidance and counseling program was complete. Myers (1926) was one of the first to suggest such standards. He identified four, including completeness as measured by the number of guidance and counseling activities, distribution of emphasis as shown by the time devoted to each activity, thoroughness as revealed by the kinds and quality of the work completed, and consistency of organization. Later Edgerton (1929) presented data that indicated that a vocational guidance program needed to contain seven guidance activities to be claimed as complete.

The 1930s

The 1930s witnessed intensive work on the issue of accountability, picking up on efforts begun in the 1920s. One line of investigation focused on the work of Myers (1926) and Edgerton (1929) to establish standards to judge which guidance and counseling activities, when brought together, would constitute a complete program. The need to develop standards to judge the completeness of a program arose because of the wide array of activities being conducted under the banner of guidance and counseling by this time. Proctor (1930) made this point as follows:

      One of the great needs in the field of guidance
   is some fairly objective means of comparing
   the guidance activities of one secondary
   school system with that of another. It is only
   in this manner that we shall ever arrive at an
   estimate of what constitutes a standard setup
   for the carrying out of a guidance program.
   (p. 58)

To respond to this concern Proctor (1930) developed a score card system designed to assess whether or not certain guidance and counseling activities were in place and functioning the way they should. This system was the forerunner of today's concept of program evaluation (program audit). References to this aspect of accountability continued to appear in the literature in the 1930s. It was important work because of the need to develop a generally accepted notion of what constituted a complete program of guidance and counseling in the schools.

While work continued on establishing standards to measure the completeness of guidance and counseling programs, several writers began expressing the need to also focus on results. Hedge and Hutson (1931), for example, worried that individuals involved in guidance work were "still so occupied with the establishment of procedures of guidance that we have not yet attained the critical attitude which engenders attempts at objective measurement of results" (p. 508). This focus on results caused a number of writers to begin identifying what they felt were the desirable outcomes of guidance and counseling programs. For example, Christy, Stewart, and Rosecrance (1930), Hinderman (1930), and Rosecrance (1930) identified the following student outcomes:

* Fewer pupils dropping out of school

* Increase in the standard of scholarship

* Better morale in the student body

* Better all-round school life

* Fewer student failures and subject withdrawals

* Young people better informed about the future

* Satisfactory adjustment of graduates to community life and vocation and to a college or university

* Fewer disciplinary cases

* Fewer absences

* More intelligent selection of subjects

* Better study habits

Other writers, picking up on the results theme, began to explore what might be involved in measuring results. In a series of articles from 1932 to 1934, Kefauver and his colleagues (Kefauver, 1932; Kefauver & Davis, 1933; Kefauver & Hand, 1932, 1934; Kefauver, Hand, & Block, 1934) described possible outcomes for guidance. They also presented different approaches to conducting evaluation studies. Later, Hutson (1935) stared that the need for measuring the results of guidance was beginning to receive recognition. "These are days ... when all school activities are subject to the sharpest scrutiny, and the administrator is called upon to justify every expenditure of time and money in the operation of the school." (p. 21)

Treacy (1937) offered a series of questions for administrators to use to review their guidance programs. One of the questions was "Is there a constant effort to evaluate the effectiveness of the guidance program?" (p. 30). Alstetter (1938) stated that no program in schools was more difficult to evaluate than the guidance service. Finally, Becker (1937) listed a number of criteria that could be used to judge the effectiveness of guidance. She also identified a number of ways that these criteria could be measured.

The 1940s

The literature of the 1940s continued to emphasize the need for the evaluation of guidance. Wrenn (1940), substituting the phrase student personnel work for guidance, urged that more studies were needed. He recommended:

      Let us have more developmental studies
   that will follow over a period of years counseled
   students and students exposed to certain
   personnel procedures; that will use control
   groups matched upon the more intangible
   aspects of personality as well as upon such
   objective measures as test scores and grades;
   that will evaluate the success of the work done
   with students in terms of broad areas of life
   adjustment, and that will indicate the extent to
   which the assistance the student receives is
   effective in an after-institutional environment
   as well as within the school walls. (p. 414)

Schultz (1941) reviewed the literature on guidance in five educational magazines covering the period 1934 to 1939. He replicated a similar study conducted by Kefauver and Davis (1933) covering the time period 1927 to 1932. He found that in these five magazines for the time period covered there were no articles based on investigations which involved "the projection of as complete a program of guidance as possible, following a group of students through this service, and making a careful measure of the results." (p. 494)

In the 1940s, the issue of what kind of training should school counselors receive was receiving increasing attention. Jager (1948) pointed out that little or no mention of training in the evaluation of guidance programs can be found in the literature. He indicated that such training in evaluation should take two forms: "that of the program as a whole and of the techniques, staff, and administrative provisions with which it operates; and that of its results as evidenced in the counselees." (p. 481)

A landmark document on evaluation appeared in the 1940s authored by Froehlich (1949). He reviewed and classified 173 studies according to the following system:

1. External criteria, the do-you-do-this? method.

2. Follow-up, the what-happened-then? method.

3. Client opinion, the what-do-you-think? method.

4. Expert opinion, the "Information Please" method.

5. Specific techniques, the little-little method.

6. Within-group changes, the before-and-after method.

7. Between-group changes, the what's-the-difference? method. (p. 2)

Froehlich (1949) concluded his review by stating that much more work on evaluation was needed. He was concerned about the lack of suitable evaluation criteria and the need for methods that met acceptable research standards but yet could be used by practitioners.

"Since its inception the guidance movement has been concerned primarily with the establishment of philosophies, the development of instruments and techniques, and the organization of programs--to the exclusion of the evaluation of results." (Wilson, 1945, p. 1) This was the opening sentence in Wilson's book, Procedures in Evaluating a Guidance Program. The remainder of her book focused on different evaluation procedures including the use of surveys, questionnaires, interviews, observations, and self-evaluations. She provided detailed discussions concerning the design and uses of each of these procedures.

Later in the 1940s Wrenn (1947), again spoke to the need for evaluating personnel services. (In this article Wrenn was using the words personnel services synonymously with the word guidance.) He stated that "Neither they [personnel workers] nor the public have been aware of the needs for evaluation, but as the public becomes more intelligent about what we are doing it will demand proof of value and we must be prepared to supply it." (p. 512)

One final article is worth noting from the 1940s. Travers (1949), in a lengthy article that reviewed a number of issues surrounding the evaluation of guidance, stated that:

      Progress will be slow until guidance workers
   come to recognize guidance as a learning situation
   which can be investigated by the methods
   developed for investigating other learning
   situations. These methods involve the specification
   of the objectives of learning that are to
   be achieved, the specification of the means of
   achieving these objectives, the selection of criteria
   for determining whether the learning
   objectives have been achieved, and provision
   for the control of relevant variables. Until
   more studies of guidance are undertaken following
   these steps, there will be very little certain
   knowledge of what guidance is actually
   accomplishing. (p. 223)

The 1950s

Interest in the evaluation of school guidance and counseling programs continued unabated during the 1950s. Calls for more and better evaluation of guidance programs continued to be heard (Cottle, 1957; Jones, 1951; Mahoney, 1950). Such calls stressed the need to establish better criteria for measuring the results of guidance in schools.

Guidance and counseling activities in the 1950s were reviewed three times in the Review of Educational Research. Wagner, Arbuckle, & Carnes (1951) noted that while there had been an increase in the number of studies of guidance over the 3 years of their review period, the focus of these studies had been limited to specific parts of guidance. They stressed the need "to evaluate total programs as well as specific or isolated phases" (p. 106). Later, McDaniel (1954), in his review of 3 years of literature, pointed out that the research conducted during that period had been mainly on the process aspects of guidance programs and that more research was needed on the effectiveness of various organizational structures for guidance. Finally Cottle (1957) reported on several studies that indicated total programs of guidance in the schools had had an impact on students' lives.

The 1960s

In 1958, the National Defense Education Act (NDEA) became law. Not since the passage of the Vocational Education Act of 1946 (often referred to as the George-Barden Act) did a federal law have as much impact on the field of guidance and counseling. NDEA provided funding for state supervisors of guidance, statewide testing programs, and training for individuals to become school counselors through summer and year-long institutes. Propelled by NDEA, the 1960s began a period of rapid expansion for guidance in schools, particularly at the elementary level.

The 1960s also witnessed the emergence of the accountability movement in education. As education was being held accountable for its outcomes, so too was guidance. It was clear that it would be necessary for school counselors to state guidance objectives in measurable terms and then show how these objectives related to the goals of education. It was also clear that the value of guidance programs was increasingly going to be judged based on their impact on students.

In 1961, Wellman and Twiford prepared a bulletin for the U.S. Office of Education (USOE) titled Guidance Counseling and Testing Program Evaluation. This bulletin was a response to a requirement in Title V-A of NDEA that required states to review and evaluate annually, local programs of guidance and counseling. The bulletin summarized the recommendations of the participants of a series of workshops held in 1959 concerning the evaluation of school guidance programs. It provided some desired student outcomes of guidance programs, offered suggestions for data collection, and presented suggested procedural methods that could be used in studies of students. The student outcomes identified were:

1. Do students develop greater understanding of their abilities, aptitudes, and interests?

2. Are students, and their parents, fully aware of opportunities and requirements for education and careers?

3. Do students select courses, and achieve in them, in line with their abilities, aptitudes, interests, and opportunities?

4. Do those students who are able to do so finish secondary school?

5. Do those students who are capable of doing so continue education beyond the secondary school?

6. Are those students who continue their education beyond the secondary school successful in their educational pursuits?

7. Are significant numbers of the especially able students getting more extensive background in mathematics, science, and the foreign languages? (Wellman & Twiford, 1961, p. 26)

The USOE continued its interest in the evaluation of guidance and counseling by sponsoring research seminars at the University of Georgia in 1961 and at the University of Michigan in 1962. The focus of these seminars was on problems in evaluating the effectiveness of guidance. In 1963, the USOE initiated a request for a proposal to evaluate the effectiveness of guidance focusing on outcomes using the recommendations from these seminars. Charles Neidt (1965) was awarded a contract to develop the research design while Fred Proff (1965) was awarded a contract to do a literature review.

In his report, Neidt (1965) recommended that the purpose of the proposed National Study of Guidance should be "to identify factors of the guidance process that are uniquely related to changes in the behavior of students" (p. 2). As reported in Wellman and Moore (1975, p. 5), the research design Neidt suggested had four phases:

1. The development of taxonomies and operational definitions of variables to be included in each of the four variable domains, i.e., process, criterion, student, and situational.

2. Instrumentation and field testing of instruments.

3. Sample selection.

4. Data collection and analysis.

(Wellman & Moore)

Only phase one of the proposed National Study was completed with the work being done at the University of Missouri-Columbia between 1966 and 1968 (Wellman & Moore, 1975).

One result of the work of Wellman and his colleagues was the development of a systems model for evaluation accompanied by a taxonomy of guidance objectives classified in the three domains of educational, vocational, and social development. This model and its companion taxonomy of objectives served as a basis for a number of evaluation models that began appearing in the late 1960s and early 1970s. A Process Guide for the Development of Objectives, originally published by the California State Department of Education in 1970 and later by the California Personnel and Guidance Association (Sullivan & O'Hare, 1971), was an example of one such model.

In a series of reviews of the evaluation of guidance and counseling published in the 1960s in the Review of Educational Research by Rothney and Farwell (1960), Patterson (1963), Strowig and Farwell (1966), and Gelatt (1969), discussion centered on the need for evaluation and the lack of evidence that it was occurring. Rothney and Farwell stated that "Guidance services, like many others in education, are still offered largely on the basis of hope and faith" (p. 168). Strowig and Farwell were particularly concerned about the lack of total program research. Gelatt, repeating the concerns of previous writers over the years, expressed his concern about the confusion and vagueness that existed concerning guidance objectives and outcomes.

In a report of research conducted in Minnesota, Tamminen and Miller (1968) discussed the lack of attention to outcomes research. To Rothney and Farwell's comment about accepting guidance in the schools based on hope and faith, Tamminen and Miller added charity.

      Faith, hope, and charity have characterized
   the American attitude toward guidance programs--faith
   in their effectiveness, hope that
   they can meet important if not always clearly
   specified need, and charity in not demanding
   more evaluative evidence that the faith and
   hope are justified. (p. 3)

The 1970s

In the early 1970s the accountability movement intensified. Concurrently, interest in the development of comprehensive systematic approaches to guidance program development and management continued to increase. The convergence of these movements in the 1970s served as a stimulus to continue the task of defining guidance developmentally in measurable individual outcome terms--as a program in its own right rather than as services ancillary to other programs.

On the West Coast, McDaniel (1970) proposed a model for guidance called Youth Guidance Systems. It was organized around goals, objectives, programs, implementation plans, and designs for evaluation. Closely related to this model was the Comprehensive Career Guidance System (CCGS) developed by personnel at the American Institutes for Research (Jones, Helliwell, Ganschow, & Hamilton, 1971; Jones, Hamilton, Ganschow, Helliwell, & Wolff, 1972). The CCGS was designed to systematically plan, implement, and evaluate guidance programs. At about the same time, personnel at the National Center for Vocational and Technical Education designed a behavioral model for career guidance based on a systems approach focusing on evaluation (Campbell et al., 1971). Later, the American College Testing Program (1976) created the River City Guidance Model which also stressed the evaluation of the results of the program.

In addition to these approaches, a systematic approach to guidance was being advocated in the PLAN (Program of Learning in Accordance with Needs) System of Individualized Education at about the same time (Dunn, 1972). Guidance was seen as a major component of PLAN and was treated as an integral part of the regular instructional program. According to Dunn the guidance program in PLAN "to be effective, must be predicated on empirical evidence." (p. 8)

Concurrent with these efforts, a national effort was begun to assist the states in developing and implementing state models or guides for career guidance, counseling, and placement. In 1971, the University of Missouri-Columbia was awarded a U.S. Office of Education grant to assist each state, the District of Columbia, and Puerto Rico in developing models or guides for implementing and evaluating career guidance, counseling, and placement programs in local schools. This project was the next step in a program of work begun as a result of a previous project at the university, a project that conducted a national conference on career guidance, counseling, and placement in October 1969, and regional conferences across the country during the spring of 1970. All 50 states, the District of Columbia, and Puerto Rico were involved in the 1971 project, and by the time the project ended in 1974, 44 states had developed some type of guide or model for career guidance, counseling, and placement.

As a part of the assistance provided to the states, project staff conducted a national conference in January 1972 and developed a manual (Gysbers & Moore, 1974) to be used by the states as they developed their own guides. The manual described how to develop, implement, and evaluate a comprehensive guidance program. The program concept described in the manual was evaluation-based, focusing both on process and outcome evaluation. Four questions were asked. What do we want to accomplish? What kind of delivery system is needed? What did we provide and do? What was the impact?

As the movement toward planning and implementing systematic developmental and accountable guidance programs in the early 1970s became more sophisticated, theoretical models began to be translated into practical, workable models to be implemented in the schools. One example of this occurred in Mesa, Arizona. The guidance staff in the Mesa Public Schools felt the need to reorient their guidance program to make it more accountable.

      Our main objective was, briefly stated, to
   reduce the size of our "universe" down to
   manageable size and then--within the parameters
   of this "new" definition of guidance--be
   responsible, i.e., accountable. We were committed
   to move toward a model of accountability--based
   not only upon what counselors
   did--but rather based on results or outcomes
   in terms of observable student behaviors.
   (McKinnon, n.d., p. iii)

In 1974, the American Institutes for Research began work on bringing together program planning efforts previously undertaken by the Pupil Personnel Division of the California State Department of Education and their own Youth Development Research Program in Mesa, Arizona, and elsewhere (Jones, Helliwell, & Ganschow, 1975). This resulted in the development of 12 competency-based staff development modules on developing comprehensive career guidance programs K-12. Module 3, titled Assessing Desired Outcomes (Dayton, n.d.), focused on the need for programs to be accountable by starting with desired student outcomes spelled out in "concrete, measurable terms, not vague statements" (Dayton, p. 7).

In addition to these local, state, and national efforts to establish guidance as a program and make it accountable, there was substantial discussion of these issues in professional literature. For example, in a book titled Research and the School Counselor, Cramer, Herr, Morris, and Frantz (1970) devoted a chapter to the evaluation of guidance programs. They pointed to the "increasing pressure on school counselors to document and justify the efficacy of their services" (p. 87). They described possible methodology to accomplish the task whether the focus is on total guidance program evaluation or it is only on specific aspects of the program.

Another example of attention to the topic of accountability occurred when one journal, Measurement and Evaluation in Guidance, devoted an entire issue (Volume 8, Number 3, October, 1975) to evaluation. Pine (1975), writing in this issue, opened his article with this statement: "In this age of accountability the evaluation of school counseling is of paramount concern to all counselors regardless of their theoretical and philosophical biases" (p. 136). The other writers in this special issue all echoed this concern (Bardo & Cody, 1975; Buckner, 1975; Helliwell & Jones, 1975; Lasser, 1975; & Miller & Grisdale, 1975).

Pine (1975) went on in his article to identify criteria typically used to establish that behavior change in students has occurred at the elementary school level as a result of being involved in counseling.

* Academic achievement

* Increase in grade point average

* Improvement in reading

* Peer relations

* Personal adjustment

* School attendance

* School adjustment

* School attitudes

* School anxiety

* Self-concept

* Self-esteem

* Self-understanding

* Teacher-pupil relationships

* Reduction of inappropriate behavior

* Intelligence test scores

* Setting realistic goals (p. 138)

Pine (1975) also identified methods typically used for evaluating the effectiveness of elementary school counseling programs. These were:

* The experimental approach--"after-only" design, the "before-and-after" design, and the "before-and-after-with-control-group design

* The tabulation approach--the number of clients, the number of counseling sessions, the nature and kinds of problems discussed, the number of parental contacts

* The follow-up approach

* The expert opinion, the "information-please" method--a subjective evaluation by experts

* The client opinion ("what-do-you-think" method) characterized by opinion surveys of counselees

* The external criteria, the "do you do this?" method--the first step is to set up standards against which the program to be evaluated is compared

* Opinion surveys of teachers, parents, and employers

* The descriptive approach--counseling practices are analyzed and described

* The case-study approach--a longitudinal view of each client (p. 139)

Concern about accountability in the 1970s was also evident in articles in a number of other journals (Arkinson, Furlong, & Janoff, 1979; Bardo, Cody, & Bryson, 1978; Carr, 1977; Crabbs & Crabbs, 1977; Gamsky, 1970; Gerler, 1976; Gubser, 1974; Krumboltz, 1974; Thompson, & Borsari, 1978). In addition, Wellman and Gysbers (1971) in the title of their article asked the question that many others had asked previously, "Did the program make a difference?" They asked this question because they pointed out that federal and state funding for guidance placed responsibility on professionals to demonstrate program effectiveness. They contended that outcomes must be stated in behavioral terms that would permit measurement. Finally, they suggested a variety of designs for outcome evaluation including baseline comparison group, within group design, and experimental design.

Campbell (1978), Herr (1978), and Mitchell (1978) writing in New Imperatives for Guidance all stressed the need for and importance of accountability for guidance. Campbell pointed out that simply demonstrating that a guidance program is needed is not enough; that decisionmakers want documentation of the results. Herr reviewed definitions of the terms research, evaluation, and accountability. He focused his attention mainly on the issue of research examining the need for research and how research forms that basis for accountability. Mitchell emphasized the importance of prioritizing student needs and being accountable for student outcomes based on those needs. In her discussion of the evaluation of guidance she made an interesting observation concerning the nature of such studies.

   Evaluation studies need not be exhaustive,
   scientifically oriented, statistically embellished.
   On the other hand, they should be something
   more than "warm puppy studies." Warm
   puppy studies focus on how happy everyone is
   with the program, how much they like it. Such
   studies, although helpful for ego-deficient
   program leaders, do little to identify the
   strengths and weaknesses of the program, or
   to suggest direction for change in order to
   enhance the likelihood of effectiveness.
   (p. 127)

In the mid 1970s, the College Entrance Examination Board (1978) developed the Career Skills Assessment Program. John Krumboltz was the principal consultant. Six content modules consisting of an exercise booklet, self-scorable and machine-scorable response sheets, and a self-instructional guide were created as follows:

* Self-evaluation and development skills

* Career awareness skills

* Career decision-making skills

* Employment-seeking skills

* Work effectiveness skills

* Personal economics skills

In the Career Skills Assessment Program manual (College Entrance Examination Board, 1978), Section IV described various uses of the instruments including conducting evaluation and research studies of career guidance programs. The point was made that few evaluation tools existed to measure student achievement in career guidance programs. It was suggested that the instruments could help assess program effectiveness from both a formative and summative basis.

The 1980s

The concern about accountability did not lessen in the 1980s, rather it increased. Due to budget cuts at federal, state, and local levels, the theme that school counselor survival depended upon accountability was prevalent (Hayden & Pohlmann, 1981; Shay, 1981; Wiggins, 1981). Shay quoting Thurow (1980) said that "the theme for the 1980s will be: All stress, much strain" (p. 74).

As a result of these conditions numerous articles were written about the need for accountability in guidance and the lack of work being done to make programs accountable (Froehle & Fuqua, 1981; Wilson & Rotter, 1982; Wilson, 1985). A number of other writers during the 1980s provided ideas about how to do program evaluation (Lewis, 1983; Lombana, 1985; Pine, 1981; Wheeler & Loesch, 1981). Fairchild and Zins (1986) reported on a national survey of accountability practices. Of the 239 respondents (239 out of 500), 55 percent indicated they were collecting accountability data. The remainder stated they lacked knowledge of accountability procedures and time was a major problem for them.

In 1981, the California State Department of Education published Guidelines for Developing Comprehensive Guidance Programs in California Public Schools: Kindergarten Through Adult School. In this document formative and summative evaluation were described using product data, process data, and context data. It was stated that formative evaluation answers the question "How are we doing" while summative evaluation answers the question "How did we do?"

1990s

Continued expressions about the lack of research concerning the impact of guidance and counseling were apparent as the decade of the 1990s began. For example, Lee and Workman (1992) noted that "Compared to other areas of the profession, school counseling seems to have little empirical evidence to support claims that it has a significant impact on the development of children and adolescents" (p. 15). Fairchild (1993) stated that while there had been an increase in work on accountability by school counselors since the survey Fairchild and Zins (1986) had conducted earlier, there were still many practitioners who did not collect accountability data.

Ways of proceeding with school guidance program evaluation were also described in the 1990s. Johnson and Whitfield (1991) presented an overall plan to evaluate school guidance programs. In the opening sentence of the introduction to their edited monograph they stated that "Evaluation is an integral part of every program and when considered during the program development state, assures clear, measurable goals" (p. 1).

Gysbers, Hughey, Starr, and Lapan (1992) described the overall evaluation framework that guided Missouri's efforts to evaluate comprehensive school guidance programs. Two of the five questions that guided the evaluation process focused on the measurement of students' mastery of guidance competencies and the possible impact of the program on the climate and goals of the school.

During this same period of time, Borders and Drury (1992) described components of effective programs. One of the components was program evaluation. In this component it was suggested that "evaluation plans should focus on program results rather than program services" (p. 493). They suggested an evaluation plan that would focus on results as well as a variety of evaluation methods that could be used to accomplish this task.

Later in the 1990s, Whiston (1996) outlined a number of approaches to research that could be used in many settings including school settings. She pointed out that school counselors were encountering increased pressure to be accountable, and hence, needed to be more active in outcome research. Then in 1998, Whiston and Sexton (1998) presented a review of school counseling outcome research published between 1988 and 1995. In "their opening sentence they stated that "In this era of accountability, school counselors increasingly are asked to provide information to parents, administrators, and legislators on the effectiveness of school counseling activities" (p. 412).

2000

The emphasis on accountability that began in the 1920s has continued with renewed vigor in this the first decade of the 21st century. Trevisan and Hubert (2001) reiterated statements made over the past 20 years concerning the importance of program evaluation and of obtaining accountability data regarding student results. Foster, Watson, Meeks, and Young (2002) also reiterated the need for accountability for school counselors and offered the single-subject research design as a way to demonstrate effectiveness. Lapan (2001) stressed the importance of comprehensive programs of guidance and counseling "conceptualized as results-based systems" (p. 289). In his article he described a framework for guidance program planning and evaluation. Hughes and James (2001) noted the importance of using accountability data with site-based management teams and other school personnel. In addition, articles by Myrick (2003), Johnson and Johnson (2003), and Dahir and Stone (2003) in the February 2003 issue of Professional School Counseling all emphasized the need for accountability.

EMPIRICAL STUDIES

Given 80 years of discussion about the importance of accountability for guidance and counseling, the question is, have there been empirical studies conducted to evaluate the impact of guidance and counseling on students? The answer is yes. These studies have taken two forms; namely, evaluating the impact of specific guidance and counseling activities and services and evaluating the impact of total programs of guidance and counseling. Both types of evaluation are important. For purposes of this article however, only selected total program evaluation studies are presented. Three of the early empirical longitudinal studies are described in some detail first because of their historical importance while the remainder of the studies are presented in the Table.

Kefauver and Hand

In the fall of 1934, Kefauver and Hand (1941) undertook a study involving junior high school students over a 3-year period, supported by a grant from the Carnegie Foundation for the Advancement of Teaching. For the study, two junior high schools from Oakland, California, and two junior high Schools from Pasadena, California, were selected. Students entering the seventh grade in the fall of 1934 served as the subjects. One school in each city was chosen as an experimental school while the second was chosen as a control school. Six tests and two inventories developed by Kefauver and Hand were administered to the experimental and control group students at the beginning and end of the study.

Before highlighting some of the findings, it is important to share the researchers' use of the words experimental and control.

      It may be desirable to caution against interpreting
   the use of the term "control" as referring
   to groups or schools without any guidance
   service. One cannot find a school without
   some form of guidance. The comparisons
   between experimental and control groups are
   actually comparisons between schools with
   different forms of guidance service and different
   amounts of emphasis on guidance. The
   added emphasis on guidance in the two so-called
   experimental schools took the form, for
   the most part, of an increase in the amount of
   school time given to guidance activities by students,
   increase in staff time for guidance work,
   and extension in the amount of material and
   tests over and above that generally employed
   in the regular guidance programs in the two
   cities. The comparisons reported in the study,
   then, do not freeport to show how students
   develop with or without guidance. Instead,
   they indicate some of the major outcomes
   yielded by the two "plus" programs. (Kefauver
   & Hand, 1941, p. 168)

What were the findings of this study? Kefauver and Hand (1941) reported that there were small beneficial effects in favor of the experimental schools civic guidance information. Larger beneficial effects were noted in health guidance information. Even larger beneficial effects were noted in imparting vocational guidance information and information about false guidance. Of particular interest to us today was that "students in the experimental schools typically made slightly greater gains in mean scores on the Stanford Achievement Test that did the students (1) in the corresponding control situations, or (2) those who had been graduated by the two experimental schools prior to the 3-year period during which the study reported in this section of the volume was conducted" (Kefauver & Hand, 1941, p. 215)

Rothney and Roens

Another major study of guidance began in the school year 1936-1937 in Arlington, Massachusetts (Rothney & Roens, 1950). Eighth graders were divided into a guided group (experimentals) and an unguided group (controls). At the beginning of the study there were 129 students in each group. The experimental group received intensive assistance by counselors while the control group received no assistance other than routine help in selecting courses and making educational and vocational plans that had been available previously.

Comparisons were made between the two groups at the end of the senior year (June 1941). There were 81 students in the guided group and 90 students in the unguided group at graduation time in June 1941. The comparisons were made on the following criteria: drop-outs, subject failures, grade failures, curriculum changes, graduation standing (an over-all estimate of school achievement), and admission to college. The findings were as follows:

1. The incidence of drop-out in the two groups was approximately the same. Guidance apparently had no effect with respect to the retention of students.

2. The rate of subject failures and the average number of failures per subject decreased faster in the guided group than in the unguided group.

3. The rate of grade failure was higher in the unguided group in grade 10, the critical first year of senior high school. It seems reasonable to assume that the lower rate of the guided group was partly the result of guidance.

4. Fewer students in the guided group made curriculum changes, and the total number of changes they made was less than in the unguided group. More of the guided students who were in the college curriculum remained in it. The relatively more permanent decisions made by the guided group may be attributed in part to guidance.

5. The mean scholastic rating of the guided group was higher than that of the unguided group by a statistically reliable margin.

6. A significantly larger percent of the guided group than of the unguided group was admitted to institutions of higher learning. (Rothney & Roens, 1950, pp. 216-217)

Additional results from this study were gathered by comparing the answers to questions administered to the guided and unguided students during their senior year (85 guided and 94 unguided) and 8 months later (85 guided and 82 unguided). The questions dealt with education and employment. What were the results? The responses indicated that students in the guided group were better equipped with information about their goals, better adjusted and more confident, more familiar with community agencies, more secure about their future, and more satisfied with their choice of school or employment.

Rothney

The major study of the 1950s concerning the effects of guidance in the schools occurred in the state of Wisconsin. It is known as the Wisconsin Counseling Study (Rothney, 1958). The full details of the study were published in his book Guidance Practices and Results. All 870 sophomores in the schools of the four communities in Wisconsin were placed in either an experimental group or a control group. The experimental group received an intensive guidance program while the controls did not.

Since the control and experimental subjects attended the same school in the same city, questions were raised about contamination. Rothney (1958) acknowledged that contamination probably occurred, but given the fact that this was a natural setting, it could not be avoided. He stated that he had "even observed an experimental boy with his arm around a control-group girl while off on an evening's excursion and it was assumed that some 'contamination' might have resulted" (p. 61).

On graduation day in June 1951, there were 690 graduates. Three follow-ups took place: one 6 months after high school graduation, one 2 and one-half years after graduation, and one 5 years after graduation in 1956. One hundred percent of the students (685) who were living participated in the final follow-up.

Here are the findings of this landmark study. Students who received counseling

1. Achieved slightly higher academic records in high school and post-high school education.

2. Indicated more realism about their own strengths and weaknesses at the time they were graduated from high school.

3. Were less dissatisfied with their high school experiences.

4. Had different vocational aspirations.

5. Were more consistent in expression of, entering into, and remaining in their vocational choices, classified by areas.

6. Made more progress in employment during the 5-year period following high school graduation.

7. Were more likely to go on to higher education, to remain to graduate, and to plan for continuation of higher education.

8. Were more satisfied with their post-high school education.

9. Expressed greater satisfaction with their status 5 years after high school and were more satisfied in retrospect with their post-high school experiences.

10. Participated in more self-improvement activities after completing high school.

11. Looked back more favorable on the counseling they had obtained. (Rothney, 1958, pp. 479480)

Rothney (1950) offered the following conclusion to the study:

      When so many small and a few large differences
   in the directions hypothesized by guidance
   workers can be obtained under representative
   high school counseling conditions, it
   seems likely that greater differences would
   appear if counseling were done under more
   ideal circumstances. Such circumstances
   would seem to require more acceptance of
   counseling as a regular part of secondary
   school experience, more enthusiastic support
   by parents and school personnel, and better
   techniques of evaluation. (pp. 482-483)

ACCOUNTABILITY IS AN ONGOING RESPONSIBILITY

Why has accountability been a topic of long standing concern? By this time one might think that this topic no longer requires professional attention because empirical studies have demonstrated that guidance and counseling does make a difference in the lives of students. I believe that the topic keeps reappearing because accountability is not a one-time phenomenon. Accountability is an ongoing responsibility of the profession at the national, state, and local levels.

If accountability is never over, what can be learned from the extensive literature on accountability of the past 80 years that can help school counselors meet their accountability obligations today? Space does not permit a detailed presentation concerning how the specific accountability techniques and methods of the past can be applied to today's world. However there are several dominant themes that have appeared consistently in the accountability literature that speak to necessary prerequisite conditions that must exist if accountability is to be achieved.

The first theme deals with mindset that individuals have about accountability. Some see it as a threat. The literature makes it clear that it is important to rid the mind of the phobia of accountability, of the persistent fear of accountability that often leads to a compelling desire to avoid it. What is required is a mindset that being accountable is simply a part of the guidance and counseling work that is done in schools every day. It is a way that this work can be improved and its effectiveness demonstrated. It is important to begin each school year, semester, month, week, and day by being results oriented. When providing guidance and counseling activities and services, always begin by first identifying the results anticipated.

The second theme focuses on the results of guidance and counseling work in the schools. If questions arise concerning which results are important to focus on, review the local school district or local building improvement plan. Those plans contain outcomes that a local district has deemed important. Most plans contain outcomes sought to which guidance and counseling programs and/or specific activities and services can contribute. There is extensive language describing possible outcomes for guidance and counseling available in the literature dating back to the 1930s. These outcomes, presented in the literature review in the first part of the article, are as applicable to today's work as they were for the work in the past.

The third and final theme is that accountability talk is not enough. It is important to remember that expressing concern about accountability is necessary but it is not sufficient. It is time for action. It is time for school counselors and their leaders at all levels to accept the challenge of accountability. The past has much to offer us concerning this challenge and how to address it. Let us use the wisdom of the past to address the challenge of accountability today and tomorrow.

Selected Empirical Studies That Evaluated the Impact of Guidance
and Counseling

Researchers             Year   Type of Study

Cantoni                 1954   Longitudinal experimental/control
                               study, high school students

Wellman &               1975   Experimental/control study,
Moore                          elementary school

Lapan, Gysbers,         1997   Comparisons of students in high
& Sun                          schools with more fully implemented
                               guidance programs with students
                               in less fully implemented programs

Nelson,                 1998   Comparisons of students in high
Gardner, & Fox                 schools with more hilly implemented
                               guidance programs with students in
                               less fully implemented programs

Lapan, Gysbers,         2001   Comparisons of students in middle
& Petroski                     schools with more fully implemented
                               guidance programs with students in
                               less fully implemented programs

Sink & Stroh            2003   Comparison of elementary students
                               enrolled for several years in
                               well-established comprehensive
                               school counseling program schools
                               with students enrolled in non-
                               comprehensive school counseling
                               program schools

Researchers             Major Findings

Cantoni                 Follow-up data indicated experimental
                        group had markedly better
                        adjustment in
                        * educational achievement
                        * occupational level
                        * emotional stability

Wellman &               Experimental group had higher
Moore                   academic achievement

Lapan, Gysbers,         Students in high schools with more
& Sun                   fully implemented programs reported:
                        * they had earned higher grades
                        * their education was better
                          preparing them for the tenure
                        * their schools had a more positive
                          climate

Nelson,                 Students in high schools with
Gardner, & Fox          more fully implemented programs:
                        * took more advanced math and
                          science courses
                        * took more vocational/technical
                          courses
                        * had higher ACT scores on every
                          scale of the test

Lapan, Gysbers,         Students in middle schools with more
& Petroski              fully implemented programs reported:
                        * they earned higher grades
                        * school was more relevant
                        * they had positive relationships
                          with teachers
                        * they were more satisfied with their
                          education
                        * they felt safer in school

Sink & Stroh            Elementary students (Grades 3 & 4)
                        enrolled in schools with a
                        comprehensive school counseling
                        program produced higher
                        achievement test scores on the
                        Iowa Tests of Basic Skills--Form M
                        and the Washington Assessment
                        of Student Learning

A version of this article was presented at the A CES/ASCA School Counseling Research Summit on June 28-29, 2003, in St. Louis, MO.

References

Alstetter, M. L. (1938). Guidance service in two hundred secondary schools. Occupations 16, 513-520.

American College Testing Program. (1976). River City high school guidance services: A conceptual model. Iowa City, IA: Author.

Atkinson, D. R., Furlong, M. J., & Janoff, D. S. (1979). A four-component model for proactive accountability in school counseling. The School Counselor, 26, 222-228.

Bardo, H. R., & Cody, J. J. (1975). Minimizing measurement concerns in guidance evaluation. Measurement and Evaluation of Guidance, 8, 175-179.

Bardo, H. R., & Cody, J. J., & Bryson, S. L. (1978). Evaluation of guidance programs: Call the question. Personnel and Guidance Journal, 57, 204-208.

Backer, E. G. (1937). How shall guidance be evaluated? High Points, 19, 5-10.

Borders, L. D., & Drury, S. M. (1992). Comprehensive school counseling programs: A review for policymakers and practitioners. Journal of Counseling and Development, 70, 487-501.

Buckner, E. T. (1975). Accountable to whom? The counselor's dilemma. Measurement and Evaluation of Guidance, 8, 187-192.

California State Department of Education. (1981). Guidelines for developing comprehensive guidance programs in California public schools: Kindergarten through adult school. Sacramento, CA: Author.

Cantoni, L. J. (1954). Guidance: 4 students 10 years later. The Clearing House, 28, 474-478.

Campbell, R. E. (1978). Developing exemplary guidance programs. In G. R. Walz & L. Benjamin (Eds.), New imperatives for guidance (pp. 30-62). Ann Arbor, MI: ERIC Counseling and Personnel Services Clearinghouse.

Campbell, R. E., Dworkin, E. P., Jackson, D. P., Hoeltzel, K. E., Parsons, G. E., & Lacey, D. W. (1971). The systems approach: An emerging behavioral model for career guidance. Columbus, OH: The Center for Vocational and Technical Education.

Carr, R. (1977). The counselor or the counseling program as the target of evaluation? Personnel and Guidance Journal, 56, 112-118.

Christy, E. B., Stewart, F. J., & Rosecrance, F. C. (1930). Guidance in the senior high school. The Vocational Guidance Magazine, 9, 51-57.

College Entrance Examination Board. (1978). Implementing the career skills assessment program. New York: Author.

Cottle, W. C. (1957). The evaluation of guidance services. Review of Educational Research, 37, 229-235.

Crabbs, S. K., & Crabbs, M.A. (1977). Accountability: Who does what to whom, when, where, and how? The School Counselor, 25, 104-109.

Cramer, S. H., Herr, E. L., Morris, C. N., & Frantz, T. T. (1970). Research and the school counselor. Boston, MA: Houghton Mifflin.

Dahir, C. A., & Stone, C.B. (2003). Accountability: A M.E.A.S.U.R.E. of the impact school counselors have on student achievement. ProfessionaI School Counseling, 6, 214-221.

Dayton, C. (n.d.). Module 3 Assessing desired outcomes. Palo Alto, CA: American Institutes for Research.

Dunn, J. A. (1972). The guidance program in the plan system of individualized education. Palo Alto, CA: American Institutes for Research.

Edgerton, A. H. (1929). Evaluating the effectiveness of guidance practices. The Nation's Schools, 3, 38-42.

Edgerton, A. H., & Herr, L. A. (1924). Present status of guidance activities in public schools. The twenty-third yearbook, National Society for the Study of Education, Part II, Vocational guidance and vocational education for the industries. Bloomington, IL: Public School Publishing.

Fairchild, T. N. (1993). Accountability practices of school counselors: 1990 national survey. The School Counselor, 40, 363-374.

Fairchild, T. N., & Zins, J. E. (1986). Accountability practices of school counselors: A national survey. Journal of Counseling and Development, 65, 196-199.

Foster, L. H., Watson, T. S., Meeks, C., & Young, T. S. (2002). Single-subject research design for school counselors: Becoming an applied researcher. Professional School Counseling, 6, 146-154.

Froehle, T. C., & Fuqua, D. R. (1981). Systematic inquiry in the school context. Personnel and Guidance Journal, 59, 509-514.

Froehlich, C. P. (1949). Evaluating guidance procedures: A review of the literature. Washington, DC: Federal Security Agency, Office of Education.

Gamsky, N. R. (1970). Action research and the school counselor. The School Counselor, 18, 36-42.

Gelatt, H. B. (1969). School guidance programs. Review of Educational Research, 39, 141-153.

Gerler, E. R. (1976). New directions for school counseling. The School Counselor, 23, 247-251.

Gubser, M. M. (1974). Performance-based counseling: Accountability or liability? The School Counselor, 21, 296-302.

Gysbers, N. D., Hughey, K. F., Starr, M., & Lapan, R. T. (1992). Improving school guidance programs: A framework for program personnel and results evaluation. Journal of Counseling and Development, 70, 565-570.

Gysbers, N. C., & Henderson, P. (2000). Developing and managing your school guidance program. Alexandria, VA: American Counseling Association.

Gysbers, N. C., & Moore, E. J. (1974). Career guidance, counseling and placement:. Elements of an illustrative program guide. Columbia, MO: Career Guidance, Counseling and Placement Project, University of Missouri-Columbia.

Hayden, C., & Pohlmann, N. (1981). Accountability and evaluation: Necessary for the survival of guidance programs? NASSP Bulletin, 65, 60-63.

Hedge, J. W., & Hutson, P. W. (1931). A technique for evaluating guidance activities. School Review, 39, 508-519.

Helliwell, C. B., & Jones, G. B. (1975). Reality considerations in guidance program evaluation. Measurement and Evaluation of Guidance, 8, 155-168.

Herr, E. L. (1978). Research in guidance. In G. R. Walz & L. Benjamin (Eds.), New imperatives for guidance (pp. 63-112). Ann Arbor, MI: ERIC Counseling and Personnel Services Clearinghouse.

Hedge, J. W., & Hutson, P. W. (1931). A technique for evaluating guidance activities. School Review, 39, 508-519.

Hinderman, R. A. (1930). Evaluating and improving guidance services. Nation's Schools, 5, 47-52.

Hughes, D. K., & James, S. H. (2001). Using accountability data to protect a school counseling program: One counselor's experience. Professional School Counseling, 4, 306-309.

Hutson, P. W. (1935). Testing the guidance program. Nation's Schools, 15, 21-23.

Isaacs, M. L. (2003). Data-driven decision making: The engine of accountability. Professional School Counseling, 6, 288-295.

Jager, H. A. (1948). Trends in counselor training. Occupations, 26, 480-482.

Johnson, S., & Johnson, C. D. (2003). Results based guidance: A systems approach to student support programs. Professional School Counseling, 6, 180-184.

Johnson, S. K., & Whitfield, E. A. (Eds.). (1991). Evaluating guidance programs: A practitioner's guide. Iowa City, IA: American College Testing Program and the National Consortium of State Career Guidance Supervisors.

Jones, A. J. (1951). Principles of guidance (4th ed.). New York: McGraw-Hill.

Jones, G. B., Hamilton, J. A., Ganschow, L. H., Helliwell, C. B., & Wolff, J. M. (1972). Planning, developing, and field testing career guidance programs: A manual and report. Polo Alto, CA: American Institutes for Research.

Jones, G. B., Helliwell, C. B., & Ganschow, L. H. (1975). A planning model for career guidance. Vocational Guidance Quarterly, 23, 220-226.

Jones, G. B., Helliwell, C. B., & Ganschow, L. H., & Hamilton, J. A. (1971). Development and evaluation of a comprehensive career guidance system. Palo Alto, CA: American Institutes for Research.

Kefauver, G. N. (1932). Achievements of guidance in secondary schools. The Vocational Guidance Magazine, 10, 199-201.

Kefauver, G. N., & Hand, H. C. (1932). Measurement of outcomes of guidance in secondary schools. Teachers College Record, 33, 324-334.

Kefauver, G. N., & Davis, A. M. (1933). Investigations in guidance. Occupations, 12, 17-25.

Kefauver, G. N., & Hand, H. C. (1934). Evaluation of guidance programs. Occupations, 12, 106-114.

Kefauver, G. N., & Hand, H. C. (1941). Appraising guidance in secondary schools. New York: Macmillan.

Kefauver, G. N., & Hand, H. C., & BIock, V.L. (1934). Objectives of experimental guidance programs. Occupations, 13, 226-227.

Krumboltz, J. D. (1974). An accountability model for counselors. Personnel and Guidance Journal, 52, 639-646.

Lasser, B. R. (1975). An outcomes-based approach to counseling evaluation. Measurement and Evaluation of Guidance, 8, 169-174.

Lapan, R. T. (2001). Results-based comprehensive guidance and counseling programs: A framework for planning and evaluation. Professional School Counseling, 4, 289-299.

Lapan, R. T., Gysbers, N. C., & Petroski, G. (2001). Helping 7th graders be safe and academically successful: A statewide study of the impact of comprehensive guidance programs. Journal of Counseling and Development, 79, 320-330.

Lapan, R. T., Gysbers, N. C., & Sun, Y. (1997). The impact of more fully implemented guidance programs on the school experiences of high school students: A statewide evaluation study. Journal of Counseling and Development, 75, 292-302.

Lee, C. C., & Workman, D. J. (1992). School counselors and research: Current status and future direction. The School Counselor, 40, 15-19.

Lewis, J. D. (1983). Guidance program evaluation: How to do it. The School Counselor, 31, 111-119.

Lombana, J. H. (1985). Guidance accountability: A new look at an old problem. The School Counselor, 32, 340-346.

Mahoney, H. J. (1950). 10 years of evaluating guidance services. Occupations, 29, 194-197.

McDaniel, H. B. (1954). Organization and administration of guidance in elementary and secondary schools. Review of Educational Research, 34, 109-112.

McDaniel, H. B. (1970). Youth guidance systems. Palo Alto, CA: College Entrance Examination Board.

McKinnon, B. E. (n.d.). Toward accountability: A report an the Mesa approach to career guidance, counseling, and placement. Mesa AZ: Mesa Public Schools.

Miller, J. V., & Grisdale, G. A. (1975). Guidance program evaluation: What's out there? Measurement and Evaluation of Guidance, 8, 145-154.

Mitchell, A. M. (1978). The design, development, and evaluation of systematic guidance programs. In G. R. Walz & L. Benjamin (Eds.), New imperatives for guidance (pp. 113-148). Ann Arbor, MI: ERIC Counseling and Personnel Services Clearinghouse.

Myers, C. E. (1926). Some tentative standards for judging a comprehensive guidance plan. Pittsburgh, PA: Department of Vocational Guidance, Pittsburgh public Schools.

Myrick, R. D. (2003). Accountability: Counselors count. Professional School Counseling, 6, 174-179.

National Defense Education Act of 1958. Pub. L. No. 85-864, 72, Part 1, Stat. 1580 (1958).

Neidt, C. O. (1965). Relation of guidance practices to student behavioral outcomes (OE-5-99-222). U.S. Department of Health, Education, and Welfare. (Mimeographed).

Nelson, D. E., Gardner, J. L., & Fox D. G. (1998). An evaluation of the comprehensive guidance program in Utah public schools. Salt Lake City: Utah State Office of Education.

Patterson, C. H. (1963). Program evaluation. Review of Educational Research, 33, 214-224.

Payne, A. F. (1924). Problems in vocational guidance. National Vocational Guidance Association Bulletin 2, 61-63.

Pine, G. J. (1975). Evaluating school counseling programs: Retrospect and prospect. Measurement and Evaluation in Guidance, 8, 136-144.

Pine, G. J. (1981). Collaborative action research in school counseling: The integration of research and practice. The Personnel and Guidance Journal, 59, 495-501.

Proctor, W. M. (1930). Evaluating guidance activities in high schools. The Vocational Guidance Magazine, 9, 58-66.

Proff, F. (1965). Research and counseling. Contractor's Report, U.S. Office of Education. (Mimeographed)

Rosecrance, F. C. (1930). Organizing guidance for the larger school system. The Vocational Guidance Magazine, 9, 243-249.

Rothney, J. W. M. (1958). Guidance practices and results. New York: Harper.

Rothney, J. W. M., & Farwell, G. F. (1960). The evaluation of guidance and personnel services. Review of Educational Research, 30, 168-175.

Rothney, J. W. M., & Roens, B. A. (1950). Guidance of American youth: An experimental study. Cambridge, MA: Harvard University.

Schultz, F. G. (1941). A re-check of articles on guidance in five educational magazines. Occupations, 19, 492-495.

Shay, M. J. (1981). A zero-sum game: Are today's economics crunching counselor services? NASSP Bulletin, 65, 10-16.

Sink, C. A., & Stroh, H. R. (2003). Raising achievement test scores of early elementary school students through comprehensive school counseling programs. Professional School Counseling, 6, 350-364.

Strowig, R. W., & Farwell, G. F. (1966). Programmatic research. Review of Educational Research, 36, 327-334.

Sullivan, H. J., & O'Hare, R. W. (1971). Accountability in pupil personnel services: A process guide for the development of objectives. Fullerton, CA: California Personnel and Guidance Association.

Tamminen, A. W., & Miller, G. D. (1968). Guidance programs and their impact on students, Research Project No. OE-5-85-035. St. Paul, MN: Department of Education.

Thompson, D. L., & Borsari, L. R. (1978). An overview of management by objectives for guidance and counseling services. The School Counselor, 25, 172-177.

Thurow, L. C. (1980, September 27). No progress without pain. Time, p. 74.

Travers, R. M. W. (1949). A critical review of techniques for evaluating guidance. Educational and Psychological Measurement, 9, 211-225.

Treacy, J. P. (1937). Analyzing a guidance program. The American School Board Journal, 44, 29-30.

Trevisan, M. S., & Hubert, M. (2001). Implementing comprehensive guidance program evaluation support: Lessons learned. Professional School Counseling 4, 225-228.

Vocational Education Act of 1946, Pub. L. No. 586, 60, Part 1, Stat. 775 (1946).

Wagner, E. E., Arbuckle, D. S., & Carnes, E. F. (1951). Programs of guidance. Review of Educational Research, 31, 106-114.

Wellman, F. E., & Gysbers, N. C. (1971). Main question is: Did the program make a difference? American Vocational Journal, 46, 47-50.

Wellman, F. E., & Moore, E. J. (1975). Pupil personnel services: A handbook for program development and evaluation. Washington, DC: U.S. Department of Health, Education, and Welfare.

Wellman, F. E., & Twiford, D. D. (1961). Guidance Counseling and Testing Program Evaluation Suggestions for Secondary Schools Title V-A National Defense Education Act of 1958. Washington DC: United States Government Printing Office.

Wheeler, P.T., & Loesch, L. (1981). Program evaluation and counseling: Yesterday, today, and tomorrow. The Personnel and Guidance Journal, 59, 573-577.

Whitson, S. C. (1996). Accountability through action research: Research methods for practitioners. Journal of Counseling and Development, 74, 616-623.

Whitson, S. C., & Sexton, T.L. (1998). A review of school counseling outcome research: Implications for practice. Journal of Counseling and Development, 76, 412-426.

Wiggins, J. D. (1981). For counselors only: Steps to take in evaluating a school's guidance program. NASSP Bulletin, 65, 29-33.

Wilson, F. M. (1945). Procedures in evaluating a guidance program. New York Teachers College, Columbia University.

Wilson, N. H., & Rotter, J. C. (1982). School counseling: A look into the future. The Personnel and Guidance Journal, 60, 353-357.

Wilson, N. S. (1985). School counselors and research: Obstacles and opportunities. The School Counselor, 33, 111-119.

Wrenn, C. G. (1940). The evaluation of student personnel work: Critique of the "guidance movement." School and Society, 52, 409-414.

Wrenn, C. G. (1947). Trends and predictions in vocational guidance. Occupations, 25, 503-513.

Norman C. Gysbers, Ph.D., is a professor in the Department of Educational, School, and Counseling Psychology, University of Missouri-Columbia. E-mail: [e-mail].