Reliability and factor structure of the Attitude Toward Tutoring Agent Scale (ATTAS).

Pedagogical agents are gaining acceptance as effective learning tools (Baylor & Ryu, 2003; Moreno, Mayer, Spires & Lester 2001; Moreno, 2004). The increase in the use of agents highlights the need for standardized measurements for evaluating user performance in these environments. While learning gains are a primary variable of interest in such environments, the role of affective variables may be at least as important as learning gains (Anderson, 1995; Bardwell, 1984; Cognition and Technology Group at Vanderbilt, 1992; Kort, Reilly, & Picard, 2001). The purpose of this research was to design and validate an instrument, the Attitude Toward Tutoring Agent Scale (ATTAS), to measure users' perception of pedagogical agents who use conversational dialog to teach (i.e., as tutors). Items were developed from existing higher education teacher rating scales. Scale items were administered to 129 participants from three large urban universities in the south and northwest after interactions with AutoTutor, an animated pedagogical agent designed to teach conceptual physics. Results of factor analysis indicate a scale with three constructs: (a) conversation/pedagogy, (b) attitude toward student, and (c) student interest/attention. Reliability analyses showed strong reliability coefficients for each construct (alphas of .84, .87 and .89, respectively). Scales may be used independently or together in pedagogical agent tutoring environments.

**********

STATEMENT OF THE PROBLEM

Research has shown that individualized instruction and one-to-one tutoring that encourages students to provide in-depth explanations of their answers promotes learning gains (Chi, de Leeuw, Chiu, & LaVancher, 1994; Bloom, 1984). One of the goals of computer-based instruction is to recreate these types of tutoring environments. One of the more promising ways to do this is through the use of pedagogical agent environments. Pedagogical agents allow designers to create an environment in which learners can interact with a computer-based conversational partner to get advice, feedback, or instruction. These agents can take the form of humans, animals, and/or inanimate objects (e.g., the Microsoft Paperclip), or fantastic creatures (e.g., genies or space aliens). With the advent of more powerful computer technology, these agents are being designed to interact with the learner in much the same way as in a human teacher-student relationship, which allows designers to address the social aspects of human-computer interaction (HCI) while tapping the benefits of individualized instruction.

Reeves and Nass (1996) have presented evidence that people apply human social interaction rules to computer characters. This research has particular relevance for the design and evaluation of pedagogical agent environments; the extent to which a pedagogical agent can be effective will be largely influenced by its ability to mimic and support the application of those social rules in the learning environment. Because pedagogical agents are becoming more prevalent (Baylor, 2000; Baylor & Ryu, 2003; Graesser, Van Lehn, Jordan, Rose, & Harter, 2001; Johnson, 2004; Lester et al., 1997; Moreno, 2004; Moreno, Mayer, Spires, & Lester, 2001), it is of increasing importance to researchers to be able to evaluate the user perceptions of the pedagogical agents' ability to effectively reproduce a teacher-student environment.

While learning gains are an important variable of interest in the study of any pedagogical tool, the role that affective variables such as mood, motivation, attitude toward instruction, and attitude toward content can play in the learning process has been well documented, and many believe that these factors are at least as important as direct measures of learning gains (Anderson, 1995; Bardwell, 1984; Cognition and Technology Group at Vanderbilt, 1992; Kort, Reilly, & Picard, 2001; Lent, Brown, & Larkin, 1984; Lepper & Chabay, 1987; Marsh, Cairns, Relich, Barnes, & Debus, 1984; Picard, 1997; Sedighian & Sedighian, 1996; Shaw & Costanza, 1970; Smead & Chase, 1981). This is largely the rationale behind a current practice in the evaluation of college teachers: having students rate instructors. While learners may not necessarily be the best judges of teaching efficacy, we recognize that their perceptions of efficacy reflect their attitudes, and that these attitudes are important components of the learning process. In particular, if we expect our pedagogical agent environments to accurately represent the human-tutor/instructor environment, then we must be able to assess the degree to which our agents are able to mimic and support the same social/affective processes and characteristics we expect of human-tutors/instructors. The importance of these characteristics is reflected in our routine process of having students rate their instructors on openness, willingness to answer questions, encouragement of discussion, and solicitation of multiple points of view.

Some researchers have begun to examine these kinds of affective variables with pedagogical agents (Baylor & Ryu, 2003; 2004; Lester et al., 1997; Lester & Stone, 1997). When addressing the affective impact of these agents, researchers ask a variety of questions covering issues such as the perceived intelligence and likeability of the agent, its ability to respond in natural language, and learner engagement in the task. Because there is no established research tool for assessing the affective impact of pedagogical agents, these questions are often the result of "common sense" approaches, with each researcher re-inventing the wheel. This article will describe the initial development, validation, and reliability assessment of an instrument to measure the perceived efficacy of pedagogical agents as tutors, the Attitude Toward Tutoring Agent Scale (ATTAS).

REVIEW OF THE LITERATURE

Pedagogical Agent Environments

Early history of agents. The roots of the modern pedagogical agent lie in the early development of software agents. Initially, an agent was defined as software programmed with complex algorithms designed for effective communication of statements back and forth from agent to program (Genesereth, 1994). As such, these agents did not communicate with human users, but with other software in the program. In this context, agents were best used as "software interoperators" (p. 53), for example, as communication tools for the integration of databases. No thought was given to communication with human users.

Other researchers at this time were discussing the benefits of designing an environment containing animated characters, which interacted with users in a believable fashion through the expression of emotional responses (Bates, 1994). These emotional responses were thought to create "an illusion of life" (p. 124), allowing for a more engaging experience for the user. The initial attempts at creating believable characters indicated that emotional responses, even ones that were exaggerated and not considered human-like, could increase the likeability of the characters and thereby increase the motivation to interact with the agents.

At this initial stage of development, intelligent agent programmers had created software agents that were able to aid in complex mechanical tasks or prepare itineraries through interactions with human users. Even then, some researchers were beginning to consider what role these agents might play in future human-computer interaction. In his 1994 essay, Norman speculated on the psychology of interaction with these agents and proposed that many issues in the affective domain could come into play when humans were interacting with intelligent agents.

Norman's (1994) essay focused on the expectations users place on intelligent agents. He contended that users will anthropomorphize any agent that appears even the least bit intelligent and that this may cause exaggerated expectations of the agent's capabilities. It is important to emphasize that this projection of human-like qualities applies not just to agent characters, but to any system which, through its actions and responses, appears to respond even somewhat intelligently. He also pointed out that users may also have increased expectations when agents can engage in conversation. "Develop a system that recognizes words, and people assume it has full language understanding, which is not at all the same thing" (p. 69). Such expectations can obviously have an impact on learner attitude, and being able to measure such attitudes accurately and consistently is important, especially with the increased use of pedagogical agents in educational settings.

Agents in educational settings. More recently in the field of education, the focus of pedagogical agent research has been on the effectiveness of animated pedagogical agents as a method for delivering computer-based instruction. This new breed of intelligent agent is defined as a combination of knowledge-based engineering principles and animated interface agents with the ability to combine the roles of animated and autonomous agents (Johnson, Rickel, & Lester, 2000). By combining domain knowledge with believable animated characters, an engaging learning environment can be created for the user, one that closely mimics the student-human teacher paradigm. An added benefit of using agents as an educational tool is the users' perception of having an animated partner in learning, who is often presented as a seasoned instructor or tutor.

Johnson et al. (2000) described how agents can be effectively used as pedagogical advisors. As an example, the authors describe how Steve, an agent developed to aid military training, can demonstrate how to perform complex mechanical tasks. Agents can also lead learners through an environment and prevent confusion when the task is to navigate through a large and/or complex structure. Agents can also use nonverbal communication as part of the teaching process, just as human teachers do. Although the authors briefly address affective concerns through emotional behavior of the agent in these environments, they cite the need for more research in this specific area.

Baylor (1999) also considered the role of intelligent agents in learning environments. She described intelligent agents both as computer programs (with or without animated character components) that help users with certain tasks and as cognitive tools, "mental and computational devices that support, guide, and extend the thinking process of students" (p. 37). As cognitive tools, she argued that agents can be used as coaches, allowing students to monitor their learning, or as tutors through a model-building approach. This kind of coaching advisement has been shown to be effective in teaching statistics (Dempsey, Litchfield, & Van Eck, 2002) and in promoting transfer (Van Eck & Dempsey, 2002).

Baylor (2000) further identified three equally important affordances that must be included for creating agents as educational mentors: (a) regulated intelligence, (b) existence of a persona, and (c) pedagogical control. By regulated intelligence, Baylor meant that the agent should be created with moderate intelligence, and that learners should be made aware of the intelligence level so that they do not expect more than is delivered. The existence of a persona allows users to attach the perception of a character to the animated agent. By pedagogical control, she meant that the agent should be dynamic so it can adapt to any learner and be programmed to perform minimal intervention so that the user carries the load of learning and the agent acts only as a guide.

An instrument to measure learner attitudes and perceptions could help assess these three areas. There is an existing instrument that focuses on measuring the perception of the agent's persona (Baylor & Ryu, 2004). While it is useful to measure users' perception of an agent's persona (i.e., likeability), it is equally important to assess users' perceptions of the agent in terms of teacher-like qualities, given the nature of pedagogical agent learning environments. By measuring these qualities, designers can create agents that conform to the desired instructional role of the pedagogical agent. For example, the extent to which the agent acts as a guide rather than the "sage on the stage" can be adjusted to make the learner feel more accomplished and in control, in turn creating a more meaningful learning experience. Such issues are separate from issues of persona and personality, which are not related to the teaching process or role.

An animated pedagogical agent tutor that uses natural dialog. The use of animated pedagogical agents as instructional delivery devices has been most successfully manifested in the framework of intelligent tutoring systems. Researchers at the University of Memphis have been working on the implementation of a pedagogical agent tutor that uses dialog management algorithms and natural language understanding tools such as Latent Semantic Analysis (LSA) to create a dynamic, conversational interface (Graesser, Wiemer-Hastings, Wiemer-Hastings, Kreuz, & Tutoring Research Group, 1999; Graesser et al., 2001; Graesser et al., 2003). This tutor has been used to teach basic computer literacy and conceptual physics at the college level.

The underlying goal in creating this agent environment was to create dynamic tutoring session conversations and interactions based on research in human tutoring interactions. The conversations are currently designed to mimic the actions of natural human tutors, with whom interactions have been shown to increase learning by as much as 2 standard deviations (Bloom, 1984; Chi et al., 1994). While some question the generalizability of these claims, there is little doubt that individualized instruction is among the more effective modes of instruction. This pedagogical agent environment, AutoTutor, provides a simulated tutoring environment that elicits deep processing of information by engaging the learning in a conversation on the domain of interest. The inclusion of a dynamic conversational component in the pedagogical agent environment allows it to serve in the same capacity as a human teacher.

AutoTutor works by asking learners a question designed to promote deep processing and the synthesis of several conceptual elements. For example, the conceptual physics tutor might pose the following question: "The sun exerts a gravitational force on the earth as the earth moves in orbit around the sun. Does the earth pull equally on the sun? Explain why." The student and tutor then engage in a tutoring dialog to work towards an acceptable answer. LSA (Landaur, Foltz, & Latham, 1998) uses mathematical processes to compare student responses to tutor questions and statements as well as to content-relevant documents, and in turn provides information to the system to determine the next move. When the assessment module of the tutoring system determines that the student has successfully articulated a high level of domain knowledge, the tutoring system goes on to the next problem. Along the way, students are provided with pedagogically appropriate feedback and questioning strategies previously found to be effective in "live" tutoring sessions (Graesser & Person, 1994). Figure 1 shows a screen shot of the AutoTutor environment.

The Importance of Affect in Agent Environments

Picard (1997) stated, "To date, researchers trying to create intelligent computers have focused on problem solving, reasoning, learning, perception, language, and other cognitive tasks considered essential to intelligence. Most of them have not been aware that emotion influences these functions in humans" (p. 47).

The role of affect is an important component in agent environments. As seen in the previous writings, new technologies and developers are moving toward the use of agents in a dynamic conversational interface. Several studies directly addressing the affective aspect of pedagogical agent learning environments have been conducted (Baylor & Ryu, 2003; 2004; Lester et al., 1997; Lester & Stone, 1997). Early educational uses of agents have centered on the agent as a guide or mentor for students. This is consistent with the tenets of learner-centered, rather than content-centered, education. While some may question the value of examining attitudes towards agents in isolation, it is nevertheless important to be able to reliably assess attitudinal responses to computerized agents so that researchers who want to measure affective components in addition to learning gains (and whatever other variables are of interest to the researcher) have a valid, reliable means of doing so.

[FIGURE 1 OMITTED]

Lester et al. (1997), examined the motivational effects of introducing an animated pedagogical agent (Herman the Bug, who taught botany) into a middle school classroom. During the experiment, the researchers found a "Persona Effect," defined as a positive effect on student's perception of the learning experience. They went on to probe these subjects by asking for ratings of affective dimensions when fullness of communication from the agent was varied. They felt this affect was important for improving the motivation of students. Their findings indicated that the most communicative agent showed the greatest learning gain and was more highly rated by the subjects. This has been further confirmed by the description of a social agency effect (Moreno et al., 2001). This effect was found in empirical studies of the beneficial properties of pedagogical agent environments. Participants reported that the most motivating and engaging properties of the environment were the social aspects of interactions with pedagogical agents.

Baylor and Ryu (2003) conducted another empirical study that attempted to measure the role of animation in agent environments and the affective impact this has on the learning experience. Their purpose was to investigate the speculation that anthropomorphic agents enhance the positive perception of the learning experience and that this effect is enhanced with the presence of animation. Specifically, they feel the agent must be person-like, engaging, credible and instructor-like, similar to the roles described in Baylor (2000). In their experiment, participants were asked to rate their perceptions of four types of agents (non-animated parrot, animated parrot, non-animated genie, and animated genie) while performing a learning task. The characteristics were measured as participants interacted with the agents. In order to measure the presence or absence of these characteristics, the researchers asked Likert-type and open-ended questions. The researchers found that the presence of an animated image further enhanced the perceptions of credibility and engagement of the learner with the agent.

The Need for a Pedagogical Agent Scale

Even with the limited number of empirical studies, it appears that agents can serve as effective mentors or guides for students. The superficial appearance and manifestation of these agents (human, animal, fantastic, etc.) also seems less important than the appearance of intelligence, responsiveness, and emotive qualities of the agent. If, as it appears, one of the central goals of designers of these agent environments is to create student-centered learning environments that make use of more constructivist strategies and approaches, then designers will have to attend to affective variables such as emotion, motivation, and attitude. The literature to date shows that this is beginning to happen (Baylor & Ryu, 2004). With the advent of faster computers and more intuitive, powerful authoring tools, it has become much easier and more practical to create dynamic, emotional conversational agents that can act as mentors for students. These agent environments are wide open for further research because of the rapid pace of emerging technologies that allow for natural language processing and characters that possess human-like verbal and nonverbal communication. What is needed now is a valid, reliable instrument to assess the affective components of these new pedagogical agents. It is currently impossible to evaluate findings across multiple studies because each researcher's questions are individually determined, and despite their similarity and adherence to common sense, there is no empirical way to evaluate the validity or reliability of these assessment questions. It was the purpose of this study to develop a scale with constructs similar to those used in human evaluations that can be used by researchers to measure student attitude toward agents. Long-term validation of this instrument will be established through future studies comparing human tutor to agent tutor conditions.

METHODOLOGY

Construction of Scale

Often, instruments such as those proposed in this study are initially generated by the researchers from compatible existing scales that are reliable and valid. For example, the Computer Game Attitude Scale (Chappell & Taylor, 1997) was generated from the Computer Attitude Scale (Lloyd & Gressard, 1984). There are no candidates for scales for attitude toward agents, however, and particularly in terms of teacher-like qualities. The closest analogs available are instruments that measure attitude toward human instructors, which are regularly used in higher education. Rather than creating items from scratch, the researchers felt it would be better to examine existing teacher rating scales for possible items and to select and modify those for use with pedagogical agent environments. Teachers at the higher education level have been evaluated in part by student ratings of teaching efficacy as part of the tenure and promotion process for decades, and research on these scales is readily available.

In theory, a conversational agent engaged in a teaching dialog should function identically to a human teacher or tutor. In developing an agent attitude scale, however, each institution uses it's own instructor rating scale; there is no single widely accepted or widely used scale upon which to base items for a pedagogical agent rating scale. In the course of our review of this literature, a study of student-instructor rating scales gathered from several different major sources was discovered (Abrami, d'Apollonia, & Rosenfield, 1997). The list of items compiled from different rating scales had been put through a rigorous factor analysis designed to extract the most efficient means of determining effective teaching. The researchers felt that the items and constructs identified by Abrami et al. was the best place to begin constructing items for this instrument.

The researchers began by examining all of the questions and constructs listed by Abrami et al. (1997). Any questions that seemed applicable to a computer-based tutoring situation were initially selected, regardless of the category. For example, questions such as, "The instructor made helpful comments" or "The instructor provided meaningful feedback," were selected, while others like "The instructor was often late to class" or "The instructor always dressed appropriately" were not. Items were selected and modified as little as possible to make them appropriate for tutoring agents rather than human teachers. In most cases, this consisted only of changing the term "teacher" to "tutor." The goal of the researchers was to find items that were not specifically related to the classroom environment or items that addressed qualities beyond those relevant to the classroom. Table 1 lists and describes each construct found in Abrami et al. (1997) and the rationale for using all, some, or none of the items contained in each construct. (See Appendix A for a complete listing of items.)

Since the decision to include items was based solely on the item's appropriateness for pedagogical agent environments, the final items represented 10 of the constructs identified by Abrami et al. (1997). Selecting items in this fashion makes it impossible to assume that the constructs will still be valid in the new scale, especially as they were represented by a smaller number of items than in the original factor analysis and because the number of items from each construct varied. The resulting items were instead examined to determine if the original constructs could be collapsed, combined, or renamed into a smaller number appropriate to ITS environments. The initial scale consisted of 20 items divided into four constructs (Pedagogy, Conversation, Tutor's Attitude Toward Teaching, Tutor's Attitude Toward Content) with each item being scored on a six-point Likert-type scale (1 = "strongly disagree," 6 = "strongly agree"). Constructs for the initial scale were created to complement the existing constructs in Abrami et al. (1997), but with consideration of the constraints of the AutoTutor environment. Appendix A lists each original question with its corresponding construct. Table 2 shows how the constructs proposed by Abrami et al. (1997) correspond with the initial constructs created by the scale authors.

Participants

Participants consisted of 129 undergraduate students at two large public urban universities and one private urban college in the South, and a large northeastern urban university. Participants were completing a course in conceptual physics and were offered extra course credit for participation. There were 68 (53%) women and 61 (45%) men in the sample, ranging in age from 16 to 75, with an average age of 21 years. Approximately 26% were freshmen, with 20% sophomores, 21% juniors, 29% seniors, and 4% graduate students.

Study Design

Conventional wisdom for valid principal components analyses suggests the need for between 100 and 300 cases (Gorsuch, 1983; Hatcher, 1994; Hutcheson, Graeme, & Sofroniou 1999). To collect enough data for the analysis, data were collected over several experiments to capture as many participants interacting with the pedagogical agent as possible. Data collection took place over the course of four large-scale experiments beginning in March, 2001 and ending in August, 2003. Although the individual research questions of these experiments varied, the overall purpose of the experiments was to evaluate the effectiveness of AutoTutor, a natural language intelligent tutoring system, in comparison with other methods of teaching conceptual physics. Four experiments were conducted using the same content to compare AutoTutor's animated tutor with other systems used to teach conceptual physics (i.e., WhyAtlas natural language tutor [Graesser et al., 2001], human-human tutoring and textbook summaries). Because the proposed scale is meant to measure student perception of pedagogical agents acting as tutors, only data from participants interacting with the AutoTutor system in each of the four experiments were used in this analysis. Participants were randomly assigned to all conditions, ensuring that the participants in the AutoTutor conditions were representative of the sample as a whole.

In each of the four experiments, participants worked through the same pre-selected training problems on conceptual physics. Each of these problems included subsets of specific principles, which are thought to encompass the important ideas (expectations) within the concept being taught (for example, "Horizontal and vertical velocity are independent of one another"). The participants were considered successful once they had articulated each expectation associated with a problem.

In the first two experiments (n = 24, n = 23), data were collected from 47 volunteers recruited from introductory conceptual physics courses at each institution. On the first day, participants were given a pretest with questions taken from the Force Concept Inventory (FCI) (Hestenes, Wells, & Swackhammer, 1992). They then worked through five physics problems similar to the sun-earth gravitation problem described earlier. Participants returned approximately one week later for a second session of five problems covering the same principles as in the previous session, and for the posttest immediately after those problems (using different questions from the FCI). The scale was administered to each of the participants after they had completed their second session of tutoring and the posttest.

In the last two experiments, (n=25, n=57), participants, recruitment, and data collection were identical to the first two studies, with the exception that each participant received fewer physics problems. In these latter two studies, participants completed the same pretest (FCI) and worked through the five problems as described earlier, then returned for the posttest one week later but did not work through the additional five isomorphic tutoring problems as had the participants in the first studies. All other procedures and measures relevant to the proposed scale were identical.

After all data collection took place, a principal components factor analysis was run on the items from the ATTA scale. A principal components analysis determines relationships between items used in a given measure (see the following section for a more detailed explanation of principal components analyses). The analysis was used to evaluate the construct validity of the initial items and to determine whether each item was strongly enough related to its construct to justify its inclusion. The results of the initial item analysis and the steps taken to revise and construct the final version of the scale are discussed in the following section.

RESULTS

Principal Components Analysis

A principal components analysis was used to determine the overall structure of the proposed measurement. This analysis uses correlation matrices and the rotation of these matrices to determine patterns among variables, or in this case, items measuring certain criteria (Darlington, 2004). In the case of the proposed scale, the researchers used principal components analyses to discover items that were correlated with each other to create constructs that measure the desired characteristics of pedagogical agents as tutors. Item responses are entered and the principal components analysis procedure begins with the construction of a correlation matrix and the rotation of the matrix to discover groups of items with reliable relationships. These are then referred to as dimensions, or constructs.

Suggestions for the interpretation of principal components analyses results include an examination of the resulting eigenvalues to determine the amount of reliable groups (constructs) found in the data set. Eigenvalues indicate the amount of variance explained by a group of similar items. To eliminate the possibility of increased error variance, only groups of items with eigenvalues above one were selected as reliable factors, as suggested by Kaiser (1960). The results of the principal components analysis using Oblimin with Kaiser Normalization rotation indicated four factors in the initial scale with eigenvalues greater than one (8.69, 2.67, 1.33, and 1.09, respectively). These values explained 68.91% of the total variance of the initial items and indicated four reliable factors used in the proposed scale (Table 3).

The analysis showed nine individual items in the first factor, eight in the second, six in the third, and seven in the fourth. The strength of association (i.e., correlation) between item and subscales were then examined to determine if any items should be moved from their original constructs or deleted from the scale as superfluous. The criteria set for inclusion in one of the four factors was an [r.sup.2] greater than .65. As a result, the following items were deleted from the initial scale as unrelated to the measure: 11, 10, 16, and 17 (see Appendices A and B for initial and final scale items).

A further examination of the results showed that three items correlated higher on different factors than anticipated in the initial construction of the scale (item 5 correlated with factor 1; items 1 and 2 correlated with factor 2). These items were moved to their more strongly related factors. It was also observed that factor 3 was comprised solely of one item (item 12). The researchers did not feel that this single item contributed enough to the overall utility of the scale to warrant developing additional items, and it was accordingly dropped. This resulted in three factors (see Appendix B for the resulting three factors and their items).

Closer examination of these new constructs revealed that factor 1 was comprised equally of items measuring perceptions of conversation and pedagogy. Given that in the case of intelligent tutoring systems, the conversation IS the pedagogy (i.e., tutoring dialog), we felt this new construct was valid, and the items were all retained. Factor 2 was comprised primarily of items from the initial "Attitude Toward Teaching" construct, with the exception of items 1 and 2 from the initial Pedagogy construct. Further examination of these two items revealed that they focus on coaching the learner to be more independent in the learning process, which may in turn be related to attitude toward the learner. It is possible that learners interpreted these two questions as attitudinal because of the inclusion of the word "encouraged" in each question, given that item 14 uses the word discourage. Likewise, the additional three items focused primarily on items measuring perceptions of the tutor's attitude toward the learner, which may explain why all five loaded on the same factor. This scale was thus renamed "Attitude Toward the Student."

Factor 3 in the final scale was comprised of items from the initial "Attitude Toward Content" construct with the exception of item 5, which was originally on the "Pedagogy" construct. This question focused on the ability of the tutor to sense (and presumably act accordingly) when the learner needed help. As the remaining items concern student interest and attention in the subject, that is, being sensitive to when the learner needs help and providing that help, seemed related to maintaining student interest and attention. Given that this item was highly correlated with the new factor (.76), it was retained, and the construct was renamed "Student Interest/Attention."

To confirm the structure of the revised scale, a second principal components analysis was run on the revised version of the scale (with superfluous items removed from the data set). Results confirmed the researchers' conception of the revised scale by showing three components with eigenvalues greater than 1, explaining 72.36% of the total variance of the revised scale items (Table 4).

Researchers retained the [r.sup.2] greater than .65 criteria in the second principal components analysis. An examination of the results showed that five items loaded at greater than .65 on the first factor, 5 items loaded at greater than .65 on the second, and six items loaded at greater than .65 on the third factor. This resulted in a 5-item scale measuring perceptions of "Conversation/Pedagogy" a 5-item scale measuring perceptions of "Attitude Toward Student," and a 5-item scale measuring perceptions of "Student Interest/Attention." See Table 5 for the structure matrix values for correlations of each item and its related construct. See Appendix B for a final listing of scale constructs and items.

Reliability Analysis

Results from the principal components analyses indicates a valid list of items to measure perceptions of pedagogical agents as tutors. However, an analysis of the reliability of initial scale responses is called for to examine the overall relation of each item to all the other items and to provide information about how this inter-item correlation may have changed from the first version to the final version of the scale. To measure this, item responses collected from the experiment previously described were used to compute and analyze reliability of the scale. First, an overall alpha coefficient was determined for the subscale items in the first conception of the instrument (i.e., before removal and reassignment of items). The initial standardized alpha coefficient was .92. Reliability analysis for the revised instrument was .91, indicating that the reliability of the instrument was not impacted by the revisions.

Reliability analyses were then run on each of the revised subscale items. Although there were some low item-subtest correlations (i.e., below .5) for each of the subscales, the item-subtest correlation coefficients were all significant at the .001 level. The mean correlation for the "Conversation/Pedagogy" subscale was .51 with a variance of .01.

The standardized alpha coefficient for this particular subscale was .84. The mean correlation for the "Attitude Toward Student" subscale was .57 with a variance of .01. Standardized alpha coefficient for this particular subscale was .87. The mean correlation for the "Student Interest/Attention" subscale was .61 with a variance of less than .01. Standardized alpha coefficient for this particular subscale was .89. While further analysis of the reliability of the scale items is necessary through multiple applications of the scale with different populations and environments, the initial results are encouraging and provide evidence that the instrument is appropriate for such testing.

DISCUSSION

The rationale for the development of this scale was to provide a valid, reliable way to measure the perceived efficacy of a conversational pedagogical agent acting as a tutor. The finalized version of the Attitude Toward Tutoring Agent Scale is composed of 15 items in three subscales. The three subscales are defined as Conversational/Pedagogy (five items), "Attitude Toward Student" (5 items), and "Student Interest/Attention" (5 items).

The proposed scale is designed to be administered after interaction with the agent environment. Items 1 through 5 make up the "Conversation/Pedagogy" scale. Items 6 through 10 make up the "Attitude Toward Student" scale, and items 11 through 15 make up the "Student Interest/Attention" scale. Scales may be interpreted and/or administered independently. Scores are obtained by averaging responses to the Likert-type scale for each individual item (1 = "strongly disagree," 6 = "strongly agree").

Recommendations for Future Studies

Principal components analysis and reliability analysis provide evidence for the validity and reliability of this instrument and its subscales. It was hoped that by basing the scales on established constructs from the literature on instructor ratings, the initial constructs would have at least face validity. Additional studies should be conducted to further validate the scale. Participants in this study were college-age physics students interacting with AutoTutor on conceptual physics; it is recommended that the scale be administered in a variety of agent environments and with differing populations and content to see how the constructs and reliability hold up. Additional studies should be conducted with human tutor conditions as well to determine how affective instructor ratings on this scale may differ. This scale should also be compared to existing instructor rating scales for concurrent validity assessment. Finally, additional items for each sub-scale should be developed, as the current number makes individual application of sub-scales not as robust as they could be.

Summary

Technological advances and research appear to be focused on using pedagogical agents in the role of instructors or advisors in instructional settings. When employing agents as teachers, it is important to attend to affective measures as well as learning gains. Current thinking supports attention to this affective dimension as we delve further into human-computer conversational interfaces. To evaluate affective claims across multiple studies of pedagogical agents, a common measure of affective ratings is desirable. The ATTAS provides a reliable measure of perceived learning efficacy, which will allow researchers in this area to build upon each other's work.

APPENDIX A

Initial items for Perception of Pedagogical Agents as Tutors Scale

Pedagogy

1. The tutor encouraged me to think for myself.

2. The tutor encouraged the development of my knowledge.

3. The tutor made helpful comments.

4. The tutor provided helpful feedback

5. The tutor sensed when I needed help.

Conversation

6. The tutor answered my questions in a clear and concise manner.

7. The tutor responded effectively to my questions.

8. The tutor encouraged questions and answers

9. The tutor encouraged discussion.

10. There was nothing accomplished in conversations.

Attitude Towards Teaching

11. The tutor was concerned with whether I learned the material.

12. The tutor was enthusiastic when presenting the material

13. The tutor seemed friendly towards me

14. The tutor seemed discouraging towards me.

15. The tutor seemed impatient with me.

Attitude Towards Content

16. The tutor presents material in an interesting manner

17. The tutor seemed interested in the content

18. The tutor increased my interest in the subject.

19. The teaching style of the tutor held my interest

20. The tutor knows how to hold my attention when presenting material.

APPENDIX B

Final items for Attitude Toward Tutoring Agent Scale (ATTAS)

Conversation/Pedagogy

1. The tutor provided helpful feedback

2. The tutor answered my questions in a clear and concise manner.

3. The tutor responded effectively to my questions.

4. The tutor encouraged questions and answers

5. The tutor encouraged discussion.

Attitude Toward Student

6. The tutor encouraged me to think for myself.

7. The tutor encouraged the development of my knowledge.

8. The tutor seemed friendly towards me

9. The tutor seemed discouraging towards me.

10. The tutor seemed impatient with me.

Student Interest/Attention

11. The tutor made helpful comments.

12. The tutor sensed when I needed help.

13. The tutor increased my interest in the subject.

14. The teaching style of the tutor held my interest

15. The tutor knows how to hold my attention when presenting material.

Table 1 Abrami et al. (1997) Initial Constructs, Items, and Rationale
for Exclusion/Inclusion in ATTAS

                     Number                            Initial Construc
Abrami Construct     of items  Decision & Rationale    Question #

Answering questions   6        3 questions used,       Conversation: 6,
                               modified for checking   7, 8
                               natural language or
                               scripted properties in
                               intelligent tutoring
                               systems
Appropriate use of    1        Not used. Relevant to
methods/materials              classroom environment
                               only
Availability          4        Not used. Relevant to
                               human instructor only
Choice of required    4        Not used. Relevant to
materials                      classroom environment
                               only, and implies a
                               flexibility of
                               information delivery
                               not appropriate for
                               tutors
Choice of             2        Not used. Relevant to
supplementary                  classroom environment
materials                      only, and implies a
                               flexibility of
                               information delivery
                               not appropriate for
                               tutors
Clarity of           15        Not used. Has to do
instruction                    with the presentation
                               style of instruction
                               through lectures,
                               labs, etc.; delivery
                               of clear concise,
                               understandable,
                               accurate instruction,
                               Did not feel this is
                               relevant because ITSs
                               are more focused on
                               having students
                               construct knowledge
                               rather than getting
                               knowledge through
                               information delivery.
                               Also, clarity ismore a
                               function of the human
                               who designs the
                               instruction than of
                               the agent who delivers
                               it
Concern for           6        1 question used;        Attitude towards
students                       modified, used to       teaching: 11
                               check for perceptions
                               of tutoring system's
                               attitude towards
                               teaching in general.
                               Other questions not
                               appropriate to
                               tutoring system
Dramatic delivery     3        Not used. Not
                               appropriate for ITS
                               because delivery of
                               material is a function
                               of the underlying
                               technology
Enthusiasm for        3        1 question used;        Attitude toward
subject                        modified, used to       teaching: 12
                               check for perceptions
                               of tutoring system's
                               attitude toward
                               teaching in general
Enthusiasm for        9        Not used. Has to do
students                       predominantly with
                               interactions outside
                               of the classroom
Enthusiasm for        3        Not used. Relevant to
teaching                       human teachers only
                               (i.e., considered
                               teaching a chore)
Evaluation            8        Not used. Relevant to
                               materials produced by
                               instructor to evaluate
                               student performance.
                               Did not feel this was
                               applicable to tutoring
                               systems as primary
                               evaluation is through
                               student modeling.
Feedback              5        2 questions used,       Pedagogy: 3, 4
                               modified; used to
                               check student
                               perceptions of
                               tutor's pedagogical
                               efficacy
Friendly classroom    6        2 questions used,       Attitude towards
climate                        modified; used to       teaching: 13, 14
                               check if students
                               perceived the tutor'sl
                               attitude toward
                               teaching in genera
General attitudes     3        Not used. These items
                               were appropriate for
                               humans only
General knowledge     2        Not used. Specific to
and cultural                   human instructor only
attainment
High-level           11        4 items used,           Pedagogy: 1, 2
cognitive outcomes             modified. Used to       Attitude toward
                               check perceptions of    student: 6, 7
                               pedagogical efficacy
Interaction and       6        2 items used, modified  Conversation: 9,
discussion                     for checking natural    10
                               language or scripted
                               properties in
                               intelligent tutoring
                               systems
Knowledge of domain   1        Not used. This item
                               measured whether
                               instructor needed
                               notes or had knowledge
                               of the domain, not
                               appropriate for ITS
                               environments
Knowledge of          1        Not used. Measured
teaching and                   ability to handle
students                       students, not
                               appropriate for ITS
                               environments
Low level cognitive  None      Items used to measure
outcomes                       instructors ability to
                               promote low level
                               learning. Because
                               there are no specific
                               items listed in
                               Abrami, et al., none
                               were used
Management style     10        Not used. Items were
                               specific to classroom
                               encounters, not
                               relevant to ITS
                               environments
Monitoring learning   5        1 question used to      Pedagogy: 5
                               measure perceptions of
                               pedagogical efficacy
Motivating students   9        Not used. Measured
to greater effort              student perceptions of
                               instructors ability to
                               motivate further
                               study, none used in
                               this evaluation
                               because this type of
                               interaction is beyond
                               the scope of our ITS
Objectives            4        Not used. These items
                               measured specific
                               course objectives,
                               not used in ITS
Overall course        5        Not used. These items
                               measured overall
                               course satisfaction,
                               because the ITS is
                               only being used for
                               part of a course
                               (i.e., supplemental
                               tutoring) this was no
                               deemed relevant
Overall instructor   12        Not used. These items
                               measured overall
                               teacher satisfaction.
                               There are currently
                               measures for
                               perceptions of tutor's
                               effectiveness, etc.
Overall learning     None      Items used to measure
                               overall learning
                               satisfaction. Because
                               there were no specific
                               items listed in Abrami
                               et al., none were used
Personal              5        Not used. Personal
appearance, health,            appearance of tutor
attire                         not relevant for ITS
Personality          20        Not used. Personality
characteristics and            of tutor not relevant
peculiarities                  for ITS
Preparation and       8        Not used. Not relevant
organization                   for ITS
Relevance of          7        Not used. Focused
instruction                    mainly on instructors'
                               use of personal
                               examples, thought to
                               be too specific for
                               assessing attitude
                               towards pedagogical
                               agent as tutor
Research              2        Not used. Not relevant
productivity and               for ITS, extends
reputation                     evaluation to outside
                               activities of
                               instructor
Respect for others   15        1 item used, modified;  Attitude toward
                               used to measure         teaching: 15
                               tutor's attitude
                               toward teaching
Stimulation of       13        5 questions used,       Attitude towards
interest in the                modified. Used to rate  content: 16, 17,
course                         student perceptions of  18, 19, 2015
                               the tutor's generaL
                               attitude toward
                               teaching content
Supervision and       1        Not used. Not relevant
disciplinary                   for ITS
actions
Time management      None      Not used. Not relevant
                               for ITS
Tolerance and         6        Not used. Not relevant
diversity                      for ITS
Vocal delivery        5        Not used. Are existing
                               measures for these
Workload             None      Not used. Not relevant
                               for limited tutoring
                               sessions, concentrates
                               on take home workload,
                               and none specified by
                               Abrami et al.

Table 2 Comparison of Constructs from the ATTAS and Constructs from
Abrami, et al.

Construct (Abrami et al.)              Construct (Initial ATTAS)

High-Level Cognitive Outcomes          Pedagogy
Monitoring Learning
Feedback
Answering Questions                    Conversation
Interaction and Discussion
Concern for Students                   Attitude Towards Teaching
Enthusiasm for Subject
Friendly Classroom Climate
Stimulation of Interest in the Course  Attitude Towards Content

Table 3 Eigenvalues of Principal Components Analysis

        Initial Eigenvalues
Factor  Total                % of Variance  Cumulative %

1       8.69                 43.47          43.47
2       2.67                 13.36          56.82
3       1.33                  6.66          63.48
4       1.09                  5.43          68.91

Table 4 Eigenvalues of Principal Factor Analysis

        Initial Eigenvalues
Factor  Total                % of Variance  Cumulative %

1       7.00                 46.66          46.66
2       2.53                 16.84          63.51
3       1.08                  7.20          70.70

Table 5 Structure Matrix Values in Revised Scale

Revised Construct           Original Item No.  Correlation Value

Conversation/Pedagogy        4                 .74
                             6                 .81
                             7                 .84
                             8                 .71
                             9                 .74
Attitude Toward Student      1                 .72
                             2                 .69
                            13                 .78
                            14                 .88
                            15                 .84
Student Interest/Attention   3                 .74
                             5                 .77
                            18                 .81
                            19                 .88
                            20                 .87

Acknowledgements

This research was supported by grants from the National Science Foundation (SBR 9720314 and REC 0106965) and the Department of Defense Multidisciplinary University Research Initiative (MURI) administered by the Office of Naval Research under grant N00014-00-1-0600. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of ONR or NSF.

The authors also wish to acknowledge the assistance of Dr. Corinna Ethington and Dr. Steve Ross at the University of Memphis, who provided valuable feedback along the way.

References

Abrami, P.C., d'Apollonia, S., & Rosenfield, S. (1997). The dimensionality of student ratings of instruction; What we know and what we do not. In R.P.S. Perry & J.C. Smart (Eds.), Effective teaching in higher education: Research and practice. New York: Agathon Press.

Anderson, J.R. (1995). Cognitive psychology and its implications (4th ed.). New York: W. H. Freeman.

Bardwell, R. (1984). The development and motivational function of expectations. American Educational Research Journal, 21(2), 461-472.

Bates, J. (1994). The role of emotion in believable agents. Communications of the ACM, 37(7),22-125.

Baylor, A. (1999). Intelligent agents as cognitive tools for education. Educational Technology, 39(2), 36-40.

Baylor, A. (2000). Beyond butlers: Intelligent agents as mentors. Journal of Educational Computing Research, 22(4), 373-382.

Baylor, A., & Ryu, J. (2003). Does the presence of image and animation enhance pedagogical agent persona? Journal of Educational Computing Research, 28(4), 373-395.

Baylor, A.L., & Ryu, J. (June, 2004). The API (Agent Persona Instrument) for assessing pedagogical agent persona. Paper presented at the ED-MEDIA conference, Honolulu, Hawaii 

Bloom, B.S. (1984). The 2-sigma problem: The search for methods of group instruction as effective as one-to-one tutoring. Educational Researcher, 13(6), 4-16.

Chappell, K.K., & Taylor, C.S. (1997). Evidence for the reliability and factorial validity of the "computer game attitude scale." Journal of Educational Computing Research, 71(1), 67-77.

Chi, M.T.H., de Leeuw, N., Chiu, M., & LaVancher, C. (1994). Eliciting self-explanations improves understanding. Cognitive Science, 18, 439-477.

Cognition and Technology Group at Vanderbilt. (1992). The Jasper series as an example of anchored instruction: Theory, program description, and assessment data. Educational Psychologist, 27(3), 291-315.

Darlington, R. (2004). Factor analysis. Retrieved June 10, 2004, from http://comp9.psych.cornell.edu/Darlington/factor.htm

Dempsey, J., Litchfield, B., & Van Eck, R. (2002, December). Use of pedagogical advisement in technology-assisted learning environments. Proceedings of the annual meeting of the International Conference on Computers in Education, Aukland, New Zealand. ICCE Society Press.

Genesereth, M.R. (1994). Software agents. Communications of the ACM, 37(7), 48-53.

Gorsuch, R.L. (1983). Factor analysis. Hillsdale, NJ: Lawrence Erlbaum. (Original work published 1974)

Graesser, A., Moreno, K.N., Marineau, J.C., Adcock, A.B., Olney, A.M., & Person, N.K. (2003). AutoTutor improves deep learning of computer literacy: Is it the dialog or the talking head? Paper presented at the International Conference of Artificial Intelligence in Education, Sydney, Australia.

Graesser, A.C., & Person, N.K. (1994). Question asking during tutoring. American Educational Research Journal, 31, 104-137.

Graesser, A.C., VanLehn, K., Rose, C., Jordan, P., & Harter, D. (2001). Intelligent tutoring systems with conversational dialogue. Al Magazine, 22(4), 39-50.

Graesser, A.C., Wiemer-Hastings, K., Wiemer-Hastings, P., Kreuz, R., & Tutoring Research Group. (1999). AutoTutor: A simulation of a human tutor. Journal of Cognitive Systems Research, 1, 35-51.

Hatcher, L. (1994). A step-by-step approach to using the SAS system for factor analysis and structural equation modeling. Cary, NC: SAS Institute. Focus on the CALIS procedure.

Hestenes, D., Wells, M., & Swackhammer, G. (1992). Force concept inventory. The Physics Teacher, 30, 141-158.

Hutcheson, G., & Sofroniou, N. (1999). The multivariate social scientist: Introductory statistics using generalized linear models. Thousand Oaks, CA: Sage.

Johnson, W.L. (2004). Motivational effects of socially intelligent pedagogical agents. Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA.

Johnson, W., Rickel, J.W., and Lester, J.C. (2000). Animated pedagogical agents: Face-to-face interaction in interactive learning environments. The International Journal of Artificial Intelligence in Education, 11, 47-78.

Kaiser, H.F. (1960). The application of electronic computers to factor analysis. Educational and Psychological Measurement, 20, 141-151.

Kort, B., Reilly, R., & Picard, R.W. (2001). An affective model of interplay between emotions and learning: Reengineering educational pedagogy-building a learning companion. Proceedings of ICALT 2001, 43-48.

Landauer, T.K., Foltz, P.W., & Latham, D. (1998). An introduction to latent semantic analysis. Discourse Processes, 25(2-3), 259-284.

Lent, R.W., Brown, S.D., & Larkin, K.E. (1984). Relation of self-efficacy expectations to academic achievement and persistence. Journal of Counseling Practice, 31(3), 356-362.

Lepper, M.R., & Chabay, R.W. (1987). Socializing the intelligent tutor: Bringing empathy to computer tutors. In H. Mandl & A. Lesgold (Eds.), Learning issues in intelligent tutoring systems (pp. 242-257). New York: Springer Verlag.

Lester, J.C., Converse, S.A., Kahler, S.E., Barlow, S.T., Stone, B.A. & Bhoga, R.S. (1997). The persona effect: Affective impact of animated pedagogical agents. Association of Computing Machinery. Retrieved, 2001, from http://www.acm.org/sigchi/chi97/proceedings/paper/j1.htm

Lester, J.C. & Stone, B.A. (1997). Increasing believability in animated pedagogical agents. Paper presented at the First International Conference on Autonomous Agents, Marina del Ray, CA.

Lloyd, B.H., & Gressard, C. (1984). Reliability and factorial validity of computer attitude scales. Educational and Psychological Measurement, 44, 501-505.

Marsh, H.W., Cairns, L., Relich, J., Barnes, J., & Debus, R.L. (1984). The relationship between dimensions of self-attribution and dimension of self-concept. Journal of Educational Psychology, 76(1), 3-32.

Moreno, R. (2004). Agent-based methods for multimedia learning environments: What works and why? Paper presented at the Annual Meeting of the American Educational Research Association, San Diego, CA.

Moreno, R., Mayer, R., Spires, H.A., & Lester, J.C. (2001). The case for social agency in computer-based teaching: Do students learn more deeply when they interact with animated pedagogical agents? Cognition and Instruction, 19(2), 177-213.

Norman, D. (1994). How people might interact with agents. Communications of the ACM, 37(7), 68-71.

Picard, R. (1997). Affective computing. Cambridge: The MIT Press.

Reeves, B., & Nass, C. (1996). The media equation: How people treat computers, television, and new media like real people and places. New York: Cambridge University Press.

Sedighian, K., & Sedighian, A. S. (1996). Can educational computer games help educators learn about the psychology of learning mathematics in children? Paper presented at the 18th Annual Meeting of the International Group for the Psychology of Mathematics Education--the North American Chapter, Florida, USA. Retrieved September, 1998, from http://www.cs.ubc.ca/nest/egems/papers.html

Shaw, M.E., & Costanza, P.R. (1970). Theories of social psychology. New York: McGraw-Hill.

Smead, V.S., & Chase, C.I. (1981). Student expectations as they relate to achievement in eighth grade mathematics. Journal of Educational Research, 75, 115-120.

Van Eck, R., & Dempsey, J.V. (2002). The effect of competition and contextualized advisement on the transfer of mathematics skills in a computer-based instructional simulation game. Educational Technology Research and Development, 50(3), 23-41.

AMY B. ADCOCK

Old Dominion University, Norfolk, VA USA

[e-mail]

RICHARD N. VAN ECK

University of North Dakota, Grand Forks, ND USA

[e-mail]