Applying learning principles to development of multimedia for addressing bias in street-level public decision-making.

JI. of Educational Multimedia and Hypermedia (2003) 12(2) 163-183

This article describes a specific effort to create a multimedia training technology (A Child Protective Service Investigation Simulator or CPS Simulator) for a representative group of street-level public servants--child protective service (CPS) workers. The article first provides some background on why bias in this area is a significant problem, then the basic features of the technology are outlined and a rationale for the development of the software within the current environment for developing educational technology is presented. The bulk of the article discusses the research on which the initial design of the CPS Simulator was based, and how the ultimate design of technology incorporates characteristics of situated learning environments and constructivist approaches to learning.

**********

Street-level bureaucrats, such as police officers, welfare workers, and schoolteachers are employees of public agencies who have regular direct contact with members of the public. By definition, street-level bureaucrats are the ultimate implementers of public policy, and the outcome of their decisions often determines the distribution of public goods, benefits, sanctions, and services. While decision makers presume that their wishes are being precisely fulfilled by these public servants, in reality street-level workers typically possess a high level of decision-making discretion within which personal bias can play a major role (Lipsky, 1980). Unfortunately, many street-level bureaucrats are poorly trained and the training that does occur may be ineffective in addressing bias. However, opportunities exist to create training technologies specifically designed to help street-level workers learn to overcome bias.

Bias is closely connected to learning. It is generally recognized that new information only becomes meaningful when it is related to knowledge already existing in the mind of the learner (Piaget, 1959; Ausubel, 1968; Anderson, 1983) and when the context for learning is highly personal and adaptive to the learner (Novak, 1985). Existing biases can form either bridges or obstacles to further learning depending on whether the bias in question helps or hinders the establishment of a link to new knowledge. Unfortunately, in many cases bias acts as a barrier to learning and good judgment. Overcoming bias represents a particularly higher order learning challenge. People form emotional attachments to their beliefs and will often regard ideas or experiences that challenge those beliefs as threats to the self. As a result, simply providing information that contradicts one's beliefs or biases is typically ineffective as a training strategy (Head & Sutton, 1985). More immersive learning experiences are needed. Similar to teachers whose beliefs about teaching are often closely linked to their experiences as students (Kagan, 1992), the decision-making of many street level bureaucrats is likely to be based on their personal experiences rather than on a rational decision-making process (Ross, 1977).

Biases in decision-making also occur because mental schema and belief systems cause only certain environmental events or stimuli to be selected out for further processing. (Kahneman, 1973). This selective attention or encoding effect and its impact on cause-effect judgments appears to be particular salient in social situations (Holland, Holyoack, Nisbett, & Thagard, 1986). Essentially, trainees in these situations will often place too much emphasis on the personalities of those they are interacting with.

These processes (e.g., attribution error, selective encoding, etc.) tend to be automatic, operating outside of conscious awareness and control (Schneider, Dumais, & Shiffrin, 1984). Hence biased decision-making can remain undetected. Because most training does not attempt to address the personal experience that underlies bias, normal training technologies are likely to leave these barriers to good decision-making intact. For example, in normal training situations workers are often encouraged to analyze and verbalize their beliefs. This practice could potentially lead to as reinforcement of "previously established, automatic patterns of thinking" (Shepard, 1995). Similar effects may also be occurring when trainees are required to participate in unsupervised observations. Moreover, even in supervised observations and training, the potential for group biases or belief systems to reinforce bias exists (Kagan, 1992). Hollingsworth (1989) and others have suggested that "change" in beliefs or mental schema is more likely to occur when trainees encounter persons or situations that force a comparison of beliefs. Hence, situations that pair trainees with supervisors or cotrainees who possess dissimilar or antagonistic beliefs or mental schemas to their own are more likely to result in a change or in truly reflective decision-making. Unfortunately, ordinary training situations do not assure that these fortuitous challenges to existing bias will be part of the training event.

Training can be designed to effectively overcome bias. Such training usually includes high dosages of role playing and human-to-human interaction (e.g., a large portion of the training budget at the Federal Law Enforcement Training Center, for example, is dedicated to employing actors for role plays). (Role plays can be much more demanding than similarly crafted paper and pencil exercises because role playing requires that the learners retrieve and analyze several types of information (e.g., auditory, visual, body language and eye movements, moods, etc.) in addition to text, that they think on their feet, and that they be producers of questions and the directors of an inquiry--rather than simply reacting to a series of pre-set questions as is the case in the customary multiple-choice exercises.) Unfortunately, the broad impact of role-playing-based training is typically limited because of its high expense, the degree of trainer skills needed, and the difficulty in uniformly conducting and thoroughly assessing role play experiences. As an alternative to live role play training, some public agencies with sufficient training budgets have begun to use video-based training technologies to help teach skills such as the use of a firearm in crisis situations. While such technologies may be marginally less expensive and more controllable than live role plays, the expense of producing video clips and the inability to make adjustments in a taped scenario limits the viability of this approach.

Designing a remedy for the specific case of child protective services (CPS): Simulations and decision support technologies (S & DST) have long been recognized as some of the most promising ways in which to deliver training in areas that demand decision-making in complex problem areas. S & DST effectively shorten feedback loops as well as allow decision makers to work out the implications of numerous "what-if" scenarios. In this way, learners can achieve both explicit and tacit understanding. These technologies are particularly helpful in situations where making the wrong decision is costly or dangerous, where observations are too rare to achieve certain knowledge, and where the impacts of one's decisions are not well understood until one has arrived at some distant point in the future (Spencer Roundtable on Simulations, 2000). Psychological studies of social interactions in avatar-based, 3D environments also suggest that the potential for simulations to effectively play human roles in the learning environment may exist. (This ability is important because people believe they get more help from talking to other people than to computers). That is, preliminary research indicates that people's experience and behavior within virtual environments employing only primitive avatar figures is substantially similar to their experience and social behavior within real-life settings. (Loomis, Blascovich, & Beall, 1999). Finally, S & DSTs have been shown to be highly effective in specific training situations (Alavi, 1994).

Faced with the problem of bias affecting the decisions of street-level public employees and the potential to help redesign the training of a specific group of public employees (child protective service workers), faculty from the Carl Vinson Institute of Government supported by a National Science Foundation Digital Government grant have developed a multimedia training technology--the CPS Simulator. While this technology has yet to be field tested, understanding the rationale and research basis for the development of the software and how the technology incorporates situated learning and constructivist principles could be useful to other learning technology developers. The development of simulation technology for CPS training is particularly appropriate because on-the-job training in this field is too risky to allow. Similarly, traditional classroom training has proved to be very expensive because of the high turnover rate and is often ineffective because trainees are rarely faced with people, settings, or situations that are truly like those that they will confront once they are on the job. Well-designed simulations can provide a low-cost, low-risk mean of presenting new workers with a wider range of challenges than is otherwise possible. They could also provide opportunities for trainees to experience the "logical consequences" of different case management approaches to the same or similar cases (e.g., trainees can try and see the results of taking decision path A, C, D and compare the results with decision path A, C, B.) Moreover, trainees could also learn from seeing how slight variations in case circumstances might call for large variations in case management strategies. Most importantly, simulations have the potential to identify and correct bias in a field that is consistently threatened by its effects. That is, despite continued agency efforts to achieve standardized responses to possible child maltreatment through policy making and training, success has been elusive. Bias in decision-making can result in CPS workers acting too precipitously or not quick enough against parents suspected of abuse and neglect (Besharov, 1989; Finkelhor, 1990). Bias can have this effect because workers possess substantial discretion at all stages of the process from the screening out of initial reports, to decisions about child removal versus family remediation. Bias can have this effect because workers possess substantial discretion at all stages of the process from the screening out of initial reports, to decisions about the intensity of the investigation, to those about appropriate intervention and ultimately to those about child removal versus family remediation. To make appropriate decisions in the real world, child protective workers must apply abstract guidelines to complex and richly nuanced events that rarely will fit neatly into state guidelines. In particular, judgment is often colored by personal perceptions, affective states, and past experiences. As they go about filling in the gaps in the guidelines, CPS workers bring their own personal standards (e.g., for appropriate parent-child interaction, house cleanliness, etc.), racial and cultural biases, and interpretative framework for understanding these guidelines. The consequence can be substantial variation in the treatment of similar cases across individual workers or groups of worker that can potentially subject agencies to liability (e.g., with respect to equal protection claims), unwanted public scrutiny, loss of citizen confidence in the fairness of the system, and a weakening of workers' confidence and morale.

Unfortunately, bias, poor judgment, or unwarranted variation in the management of abuse or neglect cases is nearly impossible to detect. CPS caseworkers or supervisors can usually argue that their decisions were appropriate to the unique features of the cases with which they dealt. (Historically, attempts to identify bias in judges' sentencing of defendants through statistical sentencing data has consistently been rejected by judges who have argued that harsher sentences for the same crime to different groups of defendants was based on unique case circumstances). What is needed to force recognition of bias in case treatment is a way to control the high levels of variation in case contexts. Structured simulations (e.g., Crime & Punishment) have provided such a means of controlling case context variation in the area of judicial sentencing (Portney & Goldman, 2002). In the Crime & Punishment simulations, a controller presents subjects with scenarios in which only a single factor is changed (e.g., the race, class, or gender of the defendant) while leaving all the other case particulars the same. Because of the need to control for the influence of contextual or informal factors, such simulations are multimedia in nature. Although the current CPS Simulator project follows the Crime & Punishment model, it is designed to take the process further by allowing for natural language processing, (1) voice recognition, artificial intelligence, text-to-speech and real-time character lip synching, expert system feedback, rapid scenario authoring, and web-based deployment and statistical processing. While Crime & Punishment scenarios had to be individually developed by video and computer animation specialists, deployed on the desktop, with researchers hand gathering data, the CPS Simulator scenarios can be developed by nonexperts, run on the World Wide Web (WWW or Web), with the results automatically processed on a web server with a statistical module.

Basic operation of the simulator. CPS Trainees are presented with a child abuse report and are then tasked with managing a case investigation. The key task for the trainee is to make five decisions related to the existence of maltreatment, the degree of risk for the child, the closure or nonclosure of the case, the removal of the child, and the placement option for the child. To make these decisions, the trainees review records, documents, photos, and animated scenes of family interaction. However, the key information gathering occurs as trainees interview a series of computer characters (with speech capabilities) that play various roles in the case (e.g., parent, teachers, relative, neighbor, etc.) All the trainees' questions, actions, information gathered, and decisions are recorded and are compared to best practices in case management, and a report is provided comparing the trainee's case management with best practices. Additionally, trainees can be challenged to use the case authoring tools to construct new case for other trainees to use.

HOW KNOWLEDGE INFORMS THE DESIGN OF THE SIMULATOR

Three inter-related areas of knowledge informed the architecture and design of the CPS Simulator: knowledge about situated learning environments, knowledge about constructivist approaches to learning, and specific knowledge about bias in decision-making. The following describes how knowledge in these areas affected the software design. While identifying and correcting bias in decision-making were the initial goals of the CPS Simulator design, the developers came to understand that addressing bias represented a special instance of learning that would be best achieved were the simulator environment one that was concretely situated and provided trainees with a personal experience of trying of constructing truth.

Situated Learning

A strong relationship between situated learning and higher order thinking (such as the thinking needed to identify and correct bias in one's judgment) can be identified in the literature. Ellis (2001) reports that multimedia-enhanced educational products can facilitate a greater depth of learning, and Herrington and Oliver (1999) found that students' talk when using interactive multimedia programs designed on principles of situated learning reflects features of higher-order thinking. The CPS Simulator incorporates a number of the features of situated learning, which according to Herrington and Oliver possesses nine characteristics, namely: an authentic context; complex authentic activities; multiple perspectives; expert performances; coaching and scaffolding; opportunities for collaboration, reflection and articulation; and authentic assessment. The ultimate multimedia for situated learning is a well-designed and purposeful virtual reality environment. The following discussion identifies areas of congruency between the CPS Simulator design and the characteristics of situated learning.

Authentic context and complex authentic activities. The design of the CPS Simulator grew out of a series of interviews with both trainers of child protective service investigators and front-line investigators. A job analysis conducted from interview data indicated that a child protective service worker essentially performs the following types of actions as part of every investigation themselves:

* Interview and pose questions to persons who might have information to share about an alleged incident of maltreatment.

* Identify information in a variety of archival record sources (e.g., social services, medical, school, juvenile court, law enforcement, and psychological records).

* Make appropriate statements to persons being interviewed (e.g., explaining the investigator's role, the goals of protective services, and what to expect in the investigative process).

* View action scenes involving multiple characters in a family interaction situation (e.g., scenes such as:

- the parents getting the child to stop grapping at items in the grocery store;

- the child refusing to go to school or do homework; or

- the parents fighting with each other.

* Ask for help from other providers (e.g., from Medical providers in a sudden infant death situation).

* Ask for permission (e.g., to interview a child at school).

* Notify other of certain key events (e.g., state policy requires that parents be notified immediately after a school-based interview with a child).

* View that child's body and physical appearance or view the household condition.

* Make key decisions based on a preponderance of evidence and weighing of multiple risk factors (e.g., every protective service worker must make decisions about response time, immediate safety, whether the alleged maltreatment occurred, whether a case will be kept open, whether the child is at risk, and whether the risk is sufficient to warrant removal of the child, and if so, what kind of removal is appropriate).

While these actions may appear relatively simple on the surface, over 270 core case issues that feed into a variety of analytical and risk assessment frameworks for making protective services decisions were identified. Similarly, training guidelines suggest that a protective service investigator employs a variety of case strategy and interview techniques (e.g., interviewing case participants in certain sequences under certain situations, mixing open- and closed-ended questions, judicious use of ice-breakers, avoidance of certain types of questions such as "why" questions, etc.). Learning the state-established CPS policies and practices represents the first step in developing good protective service skills, but informants indicated that higher order CPS skills involve being able to employ these practices under stressful, authentic circumstances (e.g., a high profile sexual abuse case with a time limit on taking decisive action). The CPS Simulator allows one to craft highly authentic cases with numerous dynamic elements (e.g., characters can be programmed to change moods upon being asked a particular question). Similarly, with respect to detecting bias, the Simulator can be programmed to present the exact same case but using a different set of characters to play the various roles (e.g., the suspected parent might be African-American in one version, while Caucasian in another).

Multiple perspectives. Perspective taking has long been identified as a key attribute of mature learners. Identifying and working with multiple perspectives is at the heart of the CPS Simulator as trainees must gather much of the case information from 3-D characters representing various roles in the case scenario (e.g., the mother, the child, a teacher, a neighbor, etc.) The simulator is designed to be programmed so as to have different characters know different pieces of information or respond to similar questions in a different manner. Moreover, the case author can choose to have a character lie about certain issues. For example, the mother might lie about the child's emotional state or physical condition if she believes that she might be suspected of maltreatment were these conditions discovered. Typically, in this instance the simulation case author would have other characters (or case documents) provide the correct information. Students, however, would be challenged to attempt to sort out the truth from within multiple perspectives of the actors.

The design principle of multiple perspectives is also evidenced by how the simulator handles open-ended questions. Experience informs us that human beings answer open-ended questions in a variety of different ways. Some people respond positively and profusely to open-ended questions, while others tend to clam up, perhaps because the question is too abstract or fails to provide sufficient cues as to what the questioner is really interested in knowing. The CPS Simulator is designed to allow the case author to set whether or not the animated characters will provide some, none, or a great deal of information as a result of an open-ended question.

Another way in which multiple perspectives are emphasized in the simulator is in the incorporation of character moods and body language. Each animated character in the simulation can express happiness, sadness, sarcasm/disinterest, and fear, and can turn its head in three different positions relative to the trainee or to other characters in the case scenario. Similarly, the case author can shape the character's voice in terms of pitch, pauses, emphasis, whispers, and, to a more limited extent, tone. Also, because the case author can create dramatic scenes involving multiple characters, students can be challenged to understand how one character may behave differently when interacting with different characters or sets of characters. Finally, with respect to helping the trainee identify bias, the CPS Simulator can be used to have the trainee compare how they managed what is essentially the same case with a variation in one or two elements (e.g., the gender of the suspected abuser is changed).

Expert performances. At any time in the simulation, the student can choose to view how key decisions required in the simulation would have been made by an expert. When this option is chosen, the simulator analyzes the underlying case facts specified by the case author, determines the correct decision for each decision area, provides an explanation for why that decision was made, and outlines the data requirements for making that decision. In future versions of the software, the project team hopes to incorporate case-specific advice from expert or model caseworkers as well as feedback on how a particular bias (e.g., the stereotype that abusers come from lower-class backgrounds) might have impacted the trainee's decision. (Also, see discussion of Opportunities for Collaboration).

Coaching and scaffolding. Coaching and scaffolding are particularly important in the context of simulations or situated learning environments that involves natural language processing and multiple perspectives. As other researchers have suggested, the situated learning environment should be as complex as need be to achieve authenticity, but not so complex as to create a cognitive overload for the student (Kashihara, Kinshuk, Oppermann, Rashev, & Simm, 2000). Unfortunately, being able to use a natural language interface can potentially add complexity to the environment. This is the case because satisfactory general purpose natural language interfaces for computers are still beyond current computer science capability (Berg, 2000; Shneiderman, 1995), due to the infinite number of ways in which people can express the same or highly similar meaning (Chomsky, 1957). Nevertheless, for the development of domain-limited situated learning environments, natural language interfaces, if well-built, should substantially enhance the authenticity of the learning experience. That is, while having a student recognize and choose from a drop list of available responses might represent an appropriate way in which to test a student's knowledge at the lower level of learning, it is not adequate to assess skill at the higher levels of comprehension, application, analysis, and synthesis (Bloom, 1956). To assess understanding at these higher levels, it is necessary for the student to be able to generate their own queries and responses. Natural language interfaces facilitate an active role in that they require trainees to form their own questions without extensive cues or the provision of preset options.

Unfortunately, in a limited-domain natural language interface, a paradox exists: because new students of the domain do not really understand the domain boundaries, they will continually be asking questions or using language that cannot be processed because the language falls outside the domain. Such students will sometimes assume that the computer is like a human being, and until they learn the limits of the domain they will not be able to use the natural language interface effectively. While this is an annoying paradox, the situation for multimedia designers is not entirely hopeless for two reasons. First, the program can be designed (as the CPS Simulator is) to attempt language pattern matching at different levels. That is, upon not achieving a perfect match, the program can look for the best imperfect match. In the CPS Simulator, we have taken this one step further: upon failing to meet the criteria for an imperfect match, the user's question is sent to a chatterbot component (A.L.I.C.E., 2002) that can provide 40,000 different natural language responses on numerous subjects. Visual and verbal cues are provided to indicate to the user that these responses are meant to be conversational rather than accurate reflections of key case facts. In essence, this chatterbot component acts as a scaffolding device meant to maintain conversational continuity or the illusion of human-to-human interaction. Second, judicious use of a natural language interface would typically mean that the instructors not use this technology in a first-stage learning activity. Rather, they would introduce it later in the sequence of learning activities, after the students have learned the boundaries of the domain in a more traditional manner (e.g., through reading). In this manner, use of the natural language interface would act as a demonstration that the student can apply knowledge in a performance or situated learning context.

Because a natural language interface potentially places excess cognitive demands on a student (i.e., the student must remember to stay within the boundaries of the domain), technologies employing these interfaces should probably incorporate ways to scaffold this learning about the domain. The CPS Simulator includes an ontology-based system for facilitating the generation of actionable questions about the facts of a protective services case. Trainees generate a question by choosing sentence elements such as actor, action, or object of the action from drop-down lists. The choices available in one drop-down list are based on the previous choices made from the other lists. This system allows a student to begin to use the simulator and learn the important questions to ask before they fully understand the limits of the domain itself. At the same time, the student must still have a pretty good idea of what they want to ask before being able to build their query.

Finally, the challenge of using a natural language interface is further complicated as one incorporates multiple character perspectives and capabilities. Students will typically become frustrated when the "machine" fails to provide an answer. Within the multiple character world of the CPS Simulator, machine failures to answer questions are artificially increased by the fact that only certain characters will possess certain information. Fortunately, it may also be the case that students will be less frustrated overall when they realize the failure of the character being addressed to reply is due to the character's knowledge limits rather than to the limits of the system overall. Typically, case authors will designate one character such as a CPS Supervisor to act as a coach or "all knowing" character. Hence, if the student wants to fill in a gap in their knowledge about the case or to be sure about their guess that a certain character is lying, they can ask the CPS Supervisor character.

Opportunities for collaboration. As with many single-player simulations, the primary way in which the CPS Simulator offers opportunities for collaboration among users is through the instructions on how to play the simulation within a group decision-making setting. In addition, however, trainees can use the CPS Simulator authoring tools to collaborate. For example, in one graduate class testing the simulator, students were instructed to work together to develop simulated child abuse and neglect cases and to specify the learning objectives that the case simulation would be designed to achieve when played. Groups of students could then experience the simulated cases that were developed by other groups. The two groups (the case creators and those experiencing the case) then work together to see if the specified objectives were achieved and if the case context provided sufficient clues to the case solution without being too obvious.

The long term goals for the CPS Simulator include development of a data warehouse that would hold the recorded plays (i.e., a complete dataset of all the questions, actions, and decisions of all the persons who have played the simulation) of numerous trainees as well as the plays of select groups of policy makers or experts. The purpose of such a data warehouse would be three-fold: first, it would allow the agency to better define and clarify policy with regard to the conditions under which certain decisions should be made. (Currently, protective service workers from different cultural backgrounds will interpret the same information about home conditions, family member interactions, adequacy of hygiene, nutrition, supervision, and severity of discipline in a different manner, leading to potential bias in crucial decisions about children's lives. With the CPS Simulator, policy makers will be able to identify the degree of cultural bias in decision-making based on an analysis of hundreds of runs of the same simulation by different case-workers.) Second, as a consequence, training designers will then be able to identify the areas of weakness or misunderstanding among specific groups of students. Third, the data warehouse will enable trainees to compare their simulation experiences to any number of representative groups (e.g., to those of their classmates, to those with the same level of education and experience, to state experts, or to their own earlier play). The degree to which such comparisons act in a similar manner to real-time collaboration will depend on the level of analysis the can be incorporated into the comparison. For example, it may become possible to turn on a "virtual collaborator" who is known to represents a particular group's historical way of looking at a case. In this way, for instance, a Caucasian caseworker might be able to get feedback from a Latino virtual caseworker who provides ongoing advice on case decisions and strategies as the simulation progresses and/or who provides a summary commentary about the student's play at the end of the simulation. In these ways cultural bias may be more effectively identified.

Reflection and articulation. Herrington and Standen (2000) and others (Boud, Keogh, & Walker, 1985; Collins & Brown, 1988; Kemmis, 1985) suggested that to provide opportunities for students to reflect on their learning, multimedia programs need to provide nonlinear navigation to enable them to return to any element of the program. The CPS Simulator is highly nonlinear in design in that a student can take any of the major actions (e.g., ask a question, view documents, view scenes, ask for help, ask for permission, notify, etc.) at any time during the simulation. While reflection is enhanced by nonlinear navigation, it is also furthered by an ability to review the history and the flow of one's actions and decisions. What pieces of information or interviews led to the decision to set a 24 hour rather than a 5 day reaction time? Why did one delay in making a decision about the substantiation of maltreatment? To answer these kinds of questions, one needs to be able to review the sequence of questions, answers, and actions. The CPS Simulator allows the student to view a log of all these simulated events, questions, responses, and actions at any time during the simulation. Instructors are encouraged to spend time processing the simulation by discussing the flow of the case management activities (as demonstrated in the log) with the students.

Authentic assessment. At the end of the session, the student receives from the CPS Simulator an assessment of their interview, case management and safety management skills, including evaluation of the:

* Appropriateness of the sequence of the trainee's:

- Data gathering activities (e.g., interviews, documents, etc.)

- Decisions

- Requests for permission, authority, and help

- Notifications

* Appropriateness of notifications (e.g., of law enforcement, school administrators, parents, etc.)

* Appropriateness of decisions (e.g., to mediate, to use force, to call for help, to enter a dangerous situations alone, etc.)

* Accuracy of risk assessment

* Completeness of the trainee's statements and information gathered (e.g., what was gathered in the process of making a decision and had the trainee gathered sufficient information to make the decision in question)

* Concepts discussed (e.g., how thorough was the trainee in terms of addressing the broad areas of concern that might extend from a more narrowly defined issue).

* Speech act types used: To what degree did the trainee use the following in their case management process:

- Open-ended questions

- Why questions

- Ice-breaking statements

- Reflective questions

- Closed-ended questions

A key purpose of the CPS Simulator is to act as a knowledge management tool, aggregating the wisdom of multiple experts in the field. As this list of assessment dimensions suggests, the CPS Simulator provides students with a great deal of detailed feedback. Were an instructor to take the time to provide this level of feedback to each student, there would likely be little time left for any other work by the instructor. However, by providing the assessment details, the Simulator frees the instructor to do what teachers do much better than computers--provide encouragement, celebrate key capabilities and highlight weaknesses that need further work.

Enhancing the Constructivist Learning Environment

The nine attributes of situated learning outlined by Herrington and Oliver (1999) underpin a constructivist approach to learning. Within this framework, learners are conceived as active, motivated, self-directed and heavily impacted by a rich social, media, and informational context. (Bruning, Schraw, & Ronning, 1999; Jonassen, 1990). In its current stage of development, the CPS Simulator provides a shell or foundation for creating highly contextual learning environments that can potentially be customized to different learning groups. Moreover, the constructivist perspective suggests that a potentially productive use of the CPS Simulator would be to have students become authors as well as players of a simulated case (e.g., of child abuse). In such instances, the student would become a constructivist learner in the purist sense of the term. While this suggestion is very much in line with the concept that one learns more by teaching than by almost any other activity, the current format of education is not particularly conducive to this activity. Essentially, having students learn to build their own simulation involves some time on the task of learning to use software authoring tools. While the software industry has created some educational technology authoring tools that allow nonprogrammers to easily and quickly build simple interactive applications, authoring next generation simulations may demand greater time on the part of students. This is mainly due to an increase in the capabilities and options in this software. For example, The CPS Simulator scenario authoring tools allow a nonprogrammer to:

* Specify up to 300 core case facts, develop new case records and documents, and create multi-character dramatic scenes.

* Develop new customized dialog (question-response) items. These customized items can be set so that particular responses are dependent on prior events, moods, or interactions with other characters (e.g., getting a truthful answer from the parent about whether she had instructed the child to visit the doctor only after the CPS worker has interviewed the doctor on that issue). Similarly, one can have a customized dialog item trigger the imputation of a core case fact or the change in a character's mood.

* Specify that the trainee receive different (e.g., progressively more/less polite, truthful or informative) responses to their asking the a question.

* To specify the amount of information that a trainee will receive when asking open-ended questions.

While the authoring tools developed as part of the CPS Simulator project make it possible for non-programmers to quickly build the core facts of a case by selecting options from menus, there is still some instructional time that has to be taken away from the subject matter content to the developing of new computer skills. In learning environments where all available instructional time must be devoted to subject matter content exclusively, having students become case or simulation constructors may be difficult to justify. The project team discovered, for example, that a college student in compressed summer course on child welfare typically did not have the time to devote to learning how to build the customized dialog (question-recognition-response patterns) that enhance the context of the case. Learning to develop new customized dialog (question-response) items is a particularly complex task as it involves the creation of keyword-structure-recognition templates that can potentially conflict with existing language recognition elements designed for the core case facts of a typical child maltreatment case. Despite these limitations, however, the project team is confident that service learning opportunities for students to use the authoring tools in this manner will become available.

Specific Knowledge about the Sources of Bias

Sources of bias in decision-making can be categorized as perceptual, social psychological, or organizational. The CPS Simulator was designed to begin to help trainees to identify each of these sources. Perceptual distortions negatively impacting decisions include:

* Making judgments based on personal importance or protected values without attention to quantity or possible value exchanges (Kemp & Willetts, 1995). In CPS cases, for example, household cleanliness may be a personal value that could impact judgments about child treatment.

* Making judgments about the future without being consistently sensitive to the impact of time. Such judgments are particularly important in child protective services where psychological age is crucial to a child's experience.

* Making decisions without appropriate recognition of marginal utility. Decision makers may inappropriately choose policies that are "fair" or allocate resources equally among competing values without recognizing that an alternative allocation may provide much greater marginal utility (Ubel, DeKay, Baron, & Asch, 1996).

* Making decisions based on group identification. While judging ideally involves objectivity and neutrality, judgments about allocations of rights in child protective service cases typically involve complex trade-offs among persons of unequal distance from the decision makers, (e.g., the relatives with whom a child might be placed may come from a different culture or social background than the CPS worker).

Decision theorists suggest perceptual distortions in decision-making can also lead to a number of classic decision failures at the level of social psychology, including tendencies to:

* Favor the status quo over alternatives. This tendency can help explain some of the stickiness of possibly beneficial trading, including possible exchanges between, for instance, a social service worker and a caretaker under suspicion of child neglect.

* Favor choices where "no action" is taken over potentially less harmful or more beneficial choices where action is required. That is, decision-makers have a bias toward omission or "do no harm" strategies.

* Negotiate based on an assumption that there is a "fixed pie" size (Bazerman, Magliozzi, & Neale, 1985). Negotiations that are stuck in a "fixed pie," status quo state tend to fail because negotiators are unable to see opportunities for tradeoffs. Unfortunately, the more integrative the approach taken to resolving a conflict, the more computationally complex negotiations can become. Decision support tools can potentially assist in handling these complexities.

At the level of organizational theory, decisions are impacted by situational factors that include organizational trust levels, stakeholder characteristics, administrative capacity, and community capacity (Lubell, 2000; Lubell & Scholz, 2001). Also transaction cost theory suggests that the costs associated with the coordination of multiple stakeholders in complex decision environments would be among the prime inhibitors of sound decision-making (Taylor & Singleton, 1993).

The CPS Simulator has features that allow it to address most of these sources of biases. For reasons of space only a few illustrations are possible. First, the CPS Simulator is designed to allow for a high degree of manipulation of quantity/quality factors. The case author, for example, can include a large number of case facts (e.g., about the condition of a child's clothes) that are designed to trigger a caseworker's personal biases. In CPS cases, however, inadequate or socially unacceptable clothing are not significant enough (in terms of state policy) to trigger a substantiation of maltreatment by themselves. Hence, through feedback trainees can potentially learn to appropriately address the quantity/quality source of bias. Second, the application can be programmed to have a small time period simulate a larger time period (e.g., 1 minute = 5 hours), thereby challenging the trainee's bias toward "no action" by speeding up the time at which "no action" becomes unacceptable. Third, a trainee's bias toward making decisions based on group identification can be challenged by having the trainee handle what is essentially the same case--after an appropriate interlude--with only the race or gender of the characters changed. Fourth, following, Scholz and Schneider (1998), the project team is using the transaction cost framework to help guide further development of the CPS Simulator. That is, the Simulator algorithms should reflect some of the case design and decision-impact relationships in the decision environment that Scholz and Schneider are discovering: for example, that scarcity of resources is likely to increase the level of agreement, that agreements will be more difficult when the resiliency of the ecosystem (or family system) is low, that high search and implementation costs and high numbers of stakeholders can decrease the likelihood of an agreement, while communication networks can help increase the extent of agreement (Provan & Milward, 1995). While the project team has not attempted to simulate all of these relationships in the current prototype, plans exist to help social workers to experience, over time, many of these contingencies within the CPS Simulator environment so as to provide them with the experience of handling difficult as well as customary cases.

CONCLUSIONS AND IMPLICATIONS FOR PRACTICE AND POLICY

Effective and appropriate decision-making by street-level public employees can be undermined by cultural, psychological, and other biases. Existing training technologies are often ineffective, too "thin" in terms of context, or too expensive to address the higher order skills that are needed to address bias in new or current workers. The next generation of training technologies, designed around principles of situated learning and crafted so as to identify potential sources of bias, may hold some promise for enhancing critical thinking by front-line public servants. However, achieving the authenticity of a working natural language interface in the next generation of multimedia software and developing scaffolding techniques for this type of authenticity will continue to present substantial challenges.

Note

(1) While multimedia, simulation-based learning is often touted as key to the future of education, the adoption of these technologies for the enhancement of higher order skills has been slow (Herrington & Oliver, 1999) possibly due to the dominance of early-stage learning activities in most time-limited learning contexts (Lewis & Smith, 1993; Ohl, 2001), the higher expense of developing software for higher order skills (e.g., the U.S. Army, for example, has awarded a multi-million dollar contract for a training simulation very similar in basic in capability to the CPS Simulator) and the ability of educators to rely on on-the-job training to supply the needed higher-order learning challenges. On-the-job learning can provide many of the benefits associated with "situated learning" as outlined by Herrington & Oliver (1999). Situated learning (discussed in more detail below) provides students with perplexing situations and requires them to "rearrange and extend" classroom knowledge to achieve a purpose. In the process, the circumstances for higher order thinking are believed to be created. Where interactive multimedia applications for higher-order problem solving have been developed, they have tended to focus on the motor and reaction skills of the trainee (e.g., as in flight simulations, hand-gun use simulations, and surgery simulations), rather than on higher-order thinking skills per se. This focus on kinetic skill is based in part on the traditional strengths and limitations of computer technologies. These technologies have generally advanced much more in the area of sensing and measuring motor skills than they have in sensing, "understanding," and appropriately reacting to human language. This proved to be the case with the CPS Simulator as the most difficult part of the development process involved creation of an ontology for this domain (or a language framework for understanding the key actor, actions, and objects in this world that potentially allows for a natural language interface with the world).

References

Alavi, M. (1994 June) Computer-mediated collaborative learning: An empirical evaluation. MIS Quarterly, 18(2), 159-175.

A.L.I.C.E, (2002). Promoting the adoption and development of Alicebot and AIML software. The A.L.I.C.E. AI Foundation. Retrieved May 28, 2003 from: http://www.alicebot.org/

Anderson, J.R. (1983). The architecture of cognition. Cambridge, MA: Harvard University Press.

Ausubel, D.P. (1968). Educational psychology: A cognitive view (2nd ed.). New York: Holt, Rinehart, and Winston.

Bazerman, M. H., Magliozzi, T., & Neale, M. A. (1985). The acquisition of an integrative response in a competitive market. Organizational Behavior and Human Performance, 34, 294-313.

Berg, G.A. (2000). Human-computer interaction (hci) in educational environments: Implications of understanding computers as media. Journal of Educational Multimedia and Hypermedia, 9(4), 349-70.

Besharov, D.J. (1988). The need to narrow the grounds for state intervention: Protecting children from abuse and neglect: Policy and practice. Springfield, Ill: C. C. Thomas.

Bloom, B.S. (Ed.) (1956). Taxonomy of educational objectives: The classification of educational goals: Handbook I, cognitive domain. New York; Toronto: Longman, Green

Boud, D., Keogh, R., & Walker, D. (1985). Promoting reflection in learning: A model. In D. Boud, R. Keogh, & D. Walker (Eds.), Reflection: Turning experience into learning (pp. 18-40). London: Kogan Page.

Bruning, R., Schraw, G., & Ronning, H. (1999). Cognitive psychology and instruction (3rd ed.). Columbus, OH: Merrill, Prentice Hall.

Chomsky, N. (1957). Syntactic structures. The Hague, The Netherlands: Mouton & Company.

Collins, A., & Brown, J.S. (1988). The computer as a tool for learning through reflection. In H. Mandl & A. Lesgold (Eds.), Learning issues for intelligent tutoring systems (pp. 1-18). New York: Springer-Verlag.

Ellis, T. J. (2001). Multimedia enhanced educational products as a tool to promote critical thinking in adult students. Journal of Educational Multimedia and Hypermedia, 10(2), 107-23.

Finkelhor, David. (1990). Is child abuse overreported? Public Welfare, 48(1), 22-30.

Head, J.O., & Sutton, C.R. (1985). Language, understanding, and commitment. In L.H.T. West & A.L. Pines (Eds.) Cognitive structure and conceptual change (pp. 91-100). Orlando, FL: Academic Press.

Herrington, J., & Oliver R. (1999). Using situated learning and multimedia to investigate higher-order thinking. Journal of Educational Multimedia and Hypermedia, 8(4) 401-421.

Herrington, J., & Standen, P. (2000). Moving from an instructivist to a constructivist multimedia learning environment. Journal of Educational Multimedia and Hypermedia, 9(3), 195-205.

Holland, J. H., Holyoak, K. J., Nisbett, R. E., & Thagard, P. R. (1986). Induction: Processes of inference, learning, and discovery. Cambridge, MA: MIT Press

Hollingsworth, S. (1989). Prior beliefs and cognitive change in learning to teach. American Educational Research Journal, 26, 160-189.

Jonassen, D.H. (1990). Thinking technology: Toward a constructivist view of instructional design. Educational Technology, 30(9), 32-34.

Kagan, D.M. (1992). Implications of research on teacher belief. Educational Psychologist, 27(1), 65-90.

Kahneman, D. (1973). Attention and effort. Englewood Cliffs, NJ: Prentice-Hall.

Kashihara, A., Kinshuk, Oppermann, R., Rashev, R., & Simm, H. (2000). A cognitive load reduction approach to exploratory learning and its application to an interactive simulation-based learning system. Journal of Educational Multimedia and Hypermedia, 9(3), 253-276.

Kemmis, S. (1985). Action research and the politics of reflection. In D. Boud, R. Keogh, & D. Walker (Eds.), Reflection: Turning experience into learning (pp. 139-163). London: Kogan Page.

Kemp, S., & Willetts, K. (1995). Rating the value of government-funded services: Comparison of methods, Journal of Economic Psychology; 16(1), 1-21.

Lewis, A., & Smith, D. (1993). Defining higher order thinking. Theory into Practice, 32(3), 131-137.

Lipsky, M. (1980). Street-level bureaucracy: Dilemmas of the individual in public services. New York: Russell Sage.

Loomis, J.M., Blascovich, J.J., & Beall, A.C. (1999). Immersive virtual environments as a basic research tool in psychology. Behavior Research Methods, Instruments, and Computers, 31(4), 557-564.

Lubell, M. (2000). Cognitive conflict and consensus building in the national estuary program. American Behavioral Scientist, 44(4), 628-647.

Lubell, M., & Scholz, J. T. (2001). Cooperation, reciprocity, and the collective-action heuristic. American Journal of Political Science, 45, 160-178.

Novak, J. (1985). Meta-learning and meta-knowledge strategies to help students learn how to learn. In L. West & L. Pines (Eds.), Cognitive Structure and Conceptual Change. Orlando, FL: Academic Press, 189-209.

Ohl, T.M. (2001). An interaction-centric learning model. Journal of Educational Multimedia and Hypermedia, 10(4), 311-332.

Piaget, J. (1959). The language and thought of the child. London: Routledge and Kegan Paul.

Portney, K., & Goldman, J. (2002). Crime and punishment: An interactive multimedia simulation of the criminal sentencing practice in American felony courts. Retrieved May 28, 2003 from: http://ase.tufts.edu/polsci/ and: http://www.ed.gov/offices/OPE/FIPSE/96ProgBk/tufts.html

Provan, K.G., & Milward, H. B. (1995). A preliminary theory of interorganizational network effectiveness: A comparative study of four community mental health systems, Administrative Science Quarterly, 40, 1-33.

Ross, L. (1977). The intuitive psychologist and his shortcomings: Distortions in the attribution process. In L. Berkowitz (Ed.), Advances in experimental social psychology Vol. 10, (pp. 173-220). New York: Academic Press.

Schneider, W., Dumais, S.T., & Shiffrin, R.M. (1984). Automatic and control processing and attention. In R. Parasuraman, R. Davies, & R. J. Beatty (Eds.), Varieties of attention (pp. 1-27). New York: Academic Press.

Scholz, J.T., & Schneider, M. (1998). National estuary program: NSF grant proposal. Retrieved May 28, 2003 from: http://www.sinc.sunysb.edu/Class/jscholz/SEA/our-res/project.htm

Shepard, R. (1995). Subjectivity in teacher decision-making: Underlying cognitive processes. Education, 115(4), 509-516.

Shneiderman, B. (1995). Looking for the bright side of user interface agents. ACM Interactions, 2(1), 13-15.

Spencer roundtable on assessing instructional simulations of social decisions (Invited Participant Conference: Tufts University, 2000).

Taylor, M., & Singleton, S. (1993). The communal resource: Transaction costs and the solution of collective action problems. Politics and Society, 21, 195-214.

Ubel P.A., DeKay M.L., Baron J., & Asch D.A. (1996) Public preferences for efficiency and equity in kidney transplant allocation decisions. Transplant Proceedings, 28, 2975-2980.

University of Georgia 

USA

[e-mail]