Establishing the validity of employment standards.

A law enforcement manager learns that a Federal court has ruled that his department discriminated against minorities in hiring police officers. The court based its finding on the department's use of a written cognitive test on which the pass rate for minorities is significantly lower than the pass rate for nonminorities. The ruling astonishes the manager because he knows that an industrial psychologist under contract to the department created the test. This psychologist testified that the test is a valid selection instrument.

The psychologist's report indicates that a correlation of .09 exists between scores on the test and supervisors' ratings of officers serving on the force. The psychologist assured the court that this result is statistically significant and serves as evidence that the test is useful in predicting the future performance of officer candidates. Unfortunately, the court's review of the test to assess its validity as a selection instrument was more extensive than mere consideration of the opinion of the department's expert.

This article discusses the standards used by courts to evaluate the legality of employment tests,(1) which have a disparate impact on groups of persons based upon their race, color, national origin, religion, or sex. It begins with a brief discussion of the legal concepts of "disparate impact" and "business necessity." It then examines in detail "validation," a scientific method that courts have adopted as a guide for assessing the business necessity of tests. The article concludes with recommendations for managers required to navigate this complicated overlap of discrimination law, industrial/organizational psychology, and personnel practices.

DISPARATE IMPACT AND BUSINESS NECESSITY

In 1971, a unanimous Supreme Court issued its opinion in the case of Griggs v. Duke Power Co.,(2) holding that an employer's use of a high school diploma requirement and two standardized written tests, each of which disqualified a higher percentage of blacks than whites, for purposes of hiring and assigning employees to laborer positions violated Title VII of the Civil Rights Act of 1964.(3) Under Griggs, a person claiming that an employment standard has a disparate impact based on race, color, sex, national origin, or religion must demonstrate factually a disparity of legal consequence before the law will require an employer to demonstrate "business necessity."(4) A person who proves such a disparity establishes a "prima facie" case of discrimination.

In evaluating whether an employment standard has a disparate impact, a statistical assessment must be made of a particular group's success rate in regard to the standard, as compared to the success rate of other groups. Where the standard creates no disparity, no demonstration of business necessity is required.

For example, in Drake v. City of Fort Collins,(5) an unsuccessful police officer candidate challenged the legality of the department's requirement of 2 years of college credits, alleging that the standard had a disparate impact on blacks. Assessment of the department's statistics revealed that the standard eliminated only 12.5 percent of black candidates, compared to elimination of 16 percent of candidates who were not black. The court held that no assessment of the educational requirements business necessity was needed in the absence of a showing of statistical disadvantage.

However, the detection of some statistical disparity requires a determination as to whether the disparity is legally significant. Because some degree of disparity is probably inherent in almost any standard, the rule of four-fifths has become a "rule of thumb" for measuring the legal significance of detected disparities. This rule provides that when the success rate of a group is less than 80 percent of that of the most successful group, then the less successful group is disadvantaged to a legally significant extent.(6)

Announcing business necessity as the legal yardstick for assessing the legality of employment standards, the Griggs Court held that an employment practice was prohibited if it operated to exclude blacks and could not be shown to be related to job performance. The Court did not provide additional guidance regarding the meaning of the phrase "business necessity," other than later stating that "any given requirement must have a manifest relationship to the employment in question."(7)

In 1975, in Albemarle Paper Co. v. Moody,(8) the U.S. Supreme Court stated, "Discriminatory tests are impermissible unless shown, by professionally accepted methods, to be 'predictive of or significantly correlated with important elements of work behavior which comprise or are relevant to the job or jobs for which candidates are being evaluated.'"(9) The Court quoted and relied on the Uniform Guidelines on Employee Selection Procedures adopted by the Equal Employment Opportunity Commission (EEOC).

In Moody, the employer hired an industrial psychologist to validate its use of a cognitive test in its promotional process. The expert's research indicated that candidates' test scores were predictive of performance if promoted.

Despite this testimony, the Court held that the employer's use of the tests constituted illegal discrimination. The failure of the employer's expert to comply with the EEOC's uniform guidelines was a basis used by the Court to discount the expert's opinion and the evidence that his study produced regarding the validity of the employer's use of tests. The Moody decision made the uniform guidelines an extremely important tool for evaluating validation studies.

In evaluating the legality of tests that have a disparate impact, courts speak of job-relatedness and business necessity. In discussing the scientific evaluation of tests, industrial/organizational psychologists, and the uniform guidelines, courts use the terms "validity" and "reliability." It is not surprising that confusion occasionally results.

A review of the case law makes it quite clear that an expert' s opinion on the reliability and validity of a test does not necessarily equate to a judicial finding of business necessity.(10) It also is true that the law does not require that every test creating a disparate impact be validated by professional means in order to meet the judicial test of business necessity.(11) Consequently, it is quite important that law enforcement managers, as consumers of the services of industrial/organizational psychologists, be aware of the scrutiny courts are likely to give the validation evidence supporting an employer's use of a test.

JOB ANALYSIS

The EEOC's uniform guidelines state that "[a]ny validity study should be based upon a review of information about the job for which the selection procedure is to be used."(12) Common sense dictates that a person making a selection for a baseball team would benefit not only from a knowledge of the game of baseball but also from knowledge about the demands of the particular position for which a player is being selected.

A job analysis is the generally accepted starting point for an industrial/organizational psychologist seeking to develop a selection procedure. "Job analysis is a means for identifying the human behaviors necessary for adequate job performance. Based on the identification of such behaviors, theories about the kinds of people the job requires (usually in terms of KSAs - knowledge, skills or abilities) can be formulated and procedures (tests, exercises, interviews) for identifying such people can be developed. The procedures can then be submitted to a test of their effectiveness."(13)

The importance of a thorough job analysis for scientific, practical, and legal reasons cannot be overstated. The failure of the employer's expert in the Moody case to perform a job analysis constituted a key factor in the judicial rejection of the expert's findings.(14)

Because the job of police officer is quite complex, a thorough job analysis is likely to be both lengthy and expensive.(15) Nonetheless, it is quite valuable in developing a selection procedure for police officer candidates, because it frequently identifies important aspects of the job that might otherwise be overlooked. By analogy, the job of a baseball catcher, at first glance, seems simple. It is only with some study that the complexity of the catcher's job, such as the signs used to reach agreement with the pitcher regarding the pitch to throw, is revealed.

TYPES OF VALIDATION EVIDENCE

Once the knowledge, skills, and abilities necessary to perform the job successfully have been identified through job analysis, an industrial/organizational psychologist either will create or choose from available tests components that identify individuals who possess these qualities and quantify the extent to which the individuals possess the qualities. For a police officer, the job analysis might identify the ability to communicate effectively with others using the spoken word as critical to performing the job successfully. A role-playing exercise might be developed to evaluate a candidate's skill and ability in this area.

Once a selection test has been devised, it should be evaluated to determine its usefulness. The EEOC's uniform guidelines address three strategies for such an evaluation - criterion-related validation, content validation, and construct validation.(16)

Criterion-related Validation

Criterion-related validity involves a statistical comparison of performance on the selection test with some measure of job performance. For example, an officer's score on the oral communication role-playing exercise might be compared to measures of on-the-job performance, such as the number of arrests the officer made each month or a structured evaluation of work performance by a supervisor.(17)

Generally, criterion-related validity studies are the preferred means of evaluating tests, but these studies present employers and their experts with certain difficult challenges. The first challenge involves the selection of a suitable criterion measure to quantify job performance.

To be suitable, the criterion chosen should reflect accurately job performance and distinguish individuals on the basis of that performance. Thus, in a law enforcement agency in which a vast majority of officers receive a "superior" rating on their annual performance evaluations, the annual rating would not be a suitable criterion because it fails to meaningfully distinguish performance among the department's officers.

This is a common failing as a criterion measure of most employers' supervisory performance appraisals. Too often, the vast majority of employees receive the same rating.

With some frequency, industrial/organizational psychologists performing criterion-related validity studies find it necessary to create criterion measures because no acceptable measures are in use. Although creating a criterion measure requires additional work and expense, it often is essential.(18)

A second challenge associated with criterion-related validity is that it involves statistical analysis, which produces results that are difficult for a layperson to interpret and requires the assessment of fairly large groups of people in order for the results to be used. Consequently, a police department with fewer than 100 employees working in the job to be studied is unlikely to benefit from a criterion-related study.

Finally, once the criterion-related validity results are in hand, an employer must determine how the results should be used. If, for example, a study indicates that an officer's scores on the oral communication exercise correlates with supervisory evaluations with a coefficient of .18, a determination needs to be made as how much weight the employer should place on the exercise results. The employer should be aware that a number of courts have expressed reluctance to accept tests that demonstrate correlation coefficients below .30, because lower correlations indicate a test result has quite limited value in predicting performance.(19)

Content Validation

The EEOC's uniform guidelines explain content validity as follows: "Evidence of the validity of a test or other selection procedure by a content validity study should consist of data showing that the content of the selection procedure is representative of important aspects of performance on the job for which the candidates are to be evaluated."(20) Frequently, selection procedures supported by content validity involve a structured evaluation of a candidate's performance on an exercise that simulates a task or tasks performed on the job. An oral communication role-playing exercise for police officer selection, where the candidate is asked to interview a person playing the role of an assault victim and gather pertinent information quickly, despite the fact that the victim is quite upset, would be an example of a test with some content validity.

A challenge frequently encountered in the use of content validation is the need to evaluate a comprehensive sample of the skills required to perform the job in question. Choosing an officer exclusively based on an oral communication exercise probably would not guarantee selection of a truly competent officer because the position requires many other skills and abilities. Content valid procedures also frequently require trained evaluators, which make them workpower intensive.

Construct Validation

The uniform guidelines state, "A construct validity study should consist of data showing that the procedure measures the degree to which candidates have identifiable characteristics which have been determined to be important in successful performance in the job for which the candidates are to be evaluated."(21) With regard to selecting law enforcement officers, it might be determined that the ability to quickly comprehend a series of seven numerals and/or alpha characters is important to successful performance, such as comprehending a vehicle license plate at a glance. A test would then be developed for the selection process that determined within .01 of a second how long candidates took to read and remember a string of characters.

Unfortunately, the scientific research required to identify measurable constructs for most jobs makes this type of validation impracticable. One authority states, "The complexity of construct validation as well as its inherently time-consuming nature make it an infrequently used procedure in the selection context."(22)

RECOMMENDATIONS

The law, and the science, of employment selection is complex, and most employers require expert assistance to succeed. It is to every employer's advantage to be an informed consumer when seeking such assistance. In the example at the beginning of this article, an informed employer should have been suspicious of a selection process that had a correlation with successful performance of only .09, not only out of a concern about its legality but also its utility.

Because of the complexity of a police officer's job, it is advantageous to evaluate as wide a variety of needed skills and abilities in the selection process as possible. Police managers should encourage their employment experts to be imaginative and resourceful in seeking appropriate selection instruments. Law enforcement employers would benefit by moving away from relying primarily on cognitive testing to the exclusion of assessment of other skills and abilities that are more difficult to assess and quantify.

Law enforcement managers also will benefit from the collection, analysis, and retention of validity evidence for the tests and procedures used in selecting and promoting police officers. In this regard, the choice of validity strategies necessarily should not be an either/or choice. As was noted previously in the example of the oral communication exercise that involved the simulated interview of an assault victim, such an exercise must be supported by both criterion-related and content validity evidence. In addition, the exercise might be structure to measure additional critical skills and abilities, such as poise and attention to detail.

The validation process should not be looked upon as merely a legal obligation. It provides an opportunity to examine critically the selections procedures to enhance effectiveness. Selection systems that can be scientifically shown to produce highly qualified candidates in a fair manner are most likely to withstand legal scrutiny. Such systems also produce candidates most likely to effectively serve and protect the community.

Endnotes

1 This article, and the courts, use the term "test" in a very broad sense, meaning any form of collecting information on individuals when that information is used as a basis for making employment decisions. Benjamin Schneider and Neal Schmitt, Staffing Organizations, 2d ed. (Scott, Foresman and Company, 1986), 13. This text is an excellent resource for the layperson attempting to understand the concepts of industrial/organizational psychology discussed in this article.

2 401 U.S. 424 (1971).

3 42 U.S.C., sec. 2000e-1 et seq. (1991).

4 The Civil Rights Act of 1991 amended Title VII, incorporating disparate impact liability into the statute, and placing the burden of proof regarding business necessity on the employer-defendant. 42 U.S.C. sec. 2000e-2(k)(1)(A)(i) (1991).

5 927 F.2d 1156 (10th Cir. 1991).

6 EEOC, Uniform Guidelines on Employee Selection Procedures, 29 C.F.R. 1607.4(D) (1988).

7 401 U.S. at 432.

8 422 U.S. 405 (1975).

9 422 U.S. at 431 (quoting 29 C.F.R. Section 1607.4(c)). See also, Contreras v. City of Los Angeles, 656 F.2d 1267, 1280 (9th Cir. 1981), cert. denied, 455 U.S. 1021 (1982).

10 See, e.g., Bouman v. Block, 940 F.2d 1211 (9th Cir. 1991).

11 See 29, C.F.R. sec. 1607.6B(2), which states, "When a formal and scored selection procedure is used which has an adverse impact, the validation techniques contemplated by these guidelines usually should be followed if technically feasible. Where the user cannot or need not follow the validation techniques anticipated by these guidelines, the user should either modify the procedure to eliminate adverse impact or otherwise justify continued use of the procedure in accord with Federal law (emphasis added). See also, Sauls, "Proving Business Necessity: The Disparate Impact Challenge," FBI Law Enforcement Bulletin, April, 1995, pp. 26-32.

12 29, C.F.R. 1607.14(A).

13 Benjamin Schneider and Neal Schmitt, Staffing Organizations, 2d ed. (Scott, Foresman and Company, 1986), 24.

14 Compare Albemarle Paper v. Moody, 422 U.S. 405, 432 (1975), with Curtin v. Office of Personnel Management, 846 F.2d 1373, (Fed.Cir. 1988).

15 Benjamin Schneider and Neal Schmitt, Staffing Organizations, 2d ed. (Scott, Foresman and Company, 1986), 29-57.

16 See 29, C.F.R. sec. 1607.5B.

17 For a discussion of criterion-related validation, see Ensley Branck of NAACP v. Seibels, 616 F.2d 812 (5th Cir.), cert. denied, 449 U.S. 1061 (1980).

18 For an interesting discussion of criterion measures in a practical context, see Hamer v. City of Atlanta, 872 F.2d 1521 (11th Cir. 1989).

19 See Bernard v. Gulf Oil Corp., 890 F.2d 735 (5th Cir. 1989); Brunet v. City of Columbus, 1 F.3d 390 (6th Cir. 1993).

20 29 C.F.R. 1607.5B.

21 29 C.F.R. 1607.5B.

22 Benjamin Schneider and Neal Schmitt, Staffing Organizations, 2d ed. (Scott, Foresman and Company, 1986), 249.

Law enforcement officers of other than Federal jurisdiction who are interested in this article should consult their legal advisor. Some police procedures ruled permissible under Federal constitutional law are of questionable legality under State law or are not permitted at all.