A factor analysis of the NETS performance profiles: searching for constructs of self-concept and technology professionalism.

A factor analysis was conducted on data (n=956) from surveys based on the National Educational Technology Standards (NETS) Performance Profiles to see how well student responses matched the six-factor NETS structure. Findings show that the performance profiles, when used directly as an assessment instrument, do not factor to the six-factor standard. Instead, two composite factors emerged: (a) Technology Self-Concept and (b) Policy and Professionalism. The authors conclude that self-assessments of educational technology using NETS are problematic at both the individual and institutional level. Findings also support the notion of early training in educational technology to provide individuals with a multidimensional concept of educational technology sufficient for more meaningful field experiences later in the teacher education program.

**********

In response to national concerns regarding the technology competence of teachers, the U.S. Department of Education, as well as state education agencies, have provided extensive funding for programs to promote professional development with technology. As the teaching profession undergoes numerous reform-driven changes to its definition of technology competence, there exists a need to better understand the ways in which those standards are enacted in teacher education programs. National and international professional organizations have developed definitions and standards of competence with technology that create for teacher education institutions an imperative to implement the standards within programs and experiences of future teachers. For individual teacher education students, this institutional imperative translates into a burden of self-efficacy or self-concept. In this study, a factor analysis of the dominant set of technology standards, NETS, is conducted in pursuit of an understanding of individual and programmatic response to the competencies these standards promote. While the fullest description of a future teacher's competence relies upon multiple indices and lenses, including portfolios, this study is focused on one measurement for technology competence that has had broad implementation across the country.

BACKGROUND

The Preparing Tomorrow's Teachers to use Technology (PT3) initiative has drawn significant interest and resources to meet the challenge of changing the practice of teachers. Since the first grants were awarded in 1999, PT3 has become an institution unto itself in terms of its effect within institutions and between them. It can be argued that PT3 has become the language for technology reform in teacher education in the United States.

The National Educational Technology Standards (NETS) Project is an ongoing initiative of the International Society for Technology in Education (ISTE). Following the National Council on Accreditation of Teacher Education's (NCATE) adoption, NETS has emerged as the predominant standards set for educational technology for K-12 and teacher education institutions. As of March of 2003, 30 states have adopted NETS for Teachers and 2 others reference it as the basis for their own standards (ISTE, 2003). Yet, as with any set of standards, implementing them in meaningful ways or even using them as a benchmark for assessing current readiness is problematic.

Because of the efficiency of using web-based surveys and a national standard, there have emerged a number of instruments for assessing individual and/or program compliance with the national standards. One instrument for assessing NETS for Teachers, Profiler (South Central Regional Technology in Education Consortium [SCRTEC1], 1999) has seen widespread use by a number of PT3 institutions (Grandgenett, Jones, Pawloski, Timms, & Ostler, 2002; Jolly & Clark, 2002). Profiler was originally developed by the South Central Regional Technology in Education Consortium (SCRTEC, 1999) to be a web-based tool for measurement of program or individual deficits and strengths in relation to the NETS performance profiles at four distinct levels in their educational program: General Preparation, Professional Preparation, Student Teaching, and First Year Teacher.

In light of the widespread adoption of the NETS standards, our study sought to examine self-report instruments such as Profiler to determine if the NETS Performance Profiles were a valid instrument for measuring self-reports of competency with educational technology. We developed a series of similar web-based self-report instruments that asked the preservice teachers to rate themselves on the leveled NETS indicators. Our investigation is driven to answer three central questions related to students' self-reported levels of educational technology competency:

1. Do students' responses to the NETS performance profiles match the six-factor structure proposed in the standard model?

2. Do preservice teachers at NETS Levels 1 and 2 differ in their conceptual representations for educational technology competency?

3. In what settings did students claim to primarily acquire specific expertise related to NETS?

METHOD

Participants

All participants were students enrolled in the teacher education program at a medium-sized midwestern university (approximately 18,000 students) with a historically large teacher education program (approximately 4000 students). The participants included in this report were those individuals who were classified as Level 1 or Level 2 in the NETS profiling strategy. Level 1 participants (General Preparation) were those students who were in line to be education majors, but had not yet completed all the necessary requirements to be officially enrolled in the teacher preparation program. These participants were generally enrolled in core courses that were prerequisites to the education methods and pedagogical content courses. In this study, 256 students completed the Level 1 survey for the NETS Teacher standards. Of that total, 126 completed the materials between April 2001 and November 2001, while 130 completed the materials between December 2001 and April 2002. A change involving the inclusion of further requests for information from the participants (see Materials section for detail) in what we refer to as the "second data capture" accounts for the uncommon point at which these two samples were split.

Level 2 participants (Professional Preparation) were those students who were officially enrolled in the teacher preparation program, but had not yet started their student teaching experience. These students generally were taking content classes, or engaged in courses related to program endorsements during the semester they completed the surveys. Students in both secondary and elementary education preparation programs were included in the sample. In this study, data from 709 students at the NETS for Teachers Level 2 were available. Of those, 374 completed the materials between April 2001 and November 2001, while 335 completed the Level 2 Profile-It survey between December 2001 and April 2002.

To support the confidentiality of responses obtained from a wide variety of courses, the online data entry environment was intentionally lean in gathering demographic information. Hence, there are no reliable data regarding gender, race, or other commonly reported variables for these participants. However, the participants were a representative sample of the population, which is predominately white, traditional college student population with a larger number of females than males.

Materials

The surveys used in this study were constructed from the level-oriented NETS for Teachers performance profiles provided by ISTE (2001). While Profiler worked very well as a direct instrument of the NETS performance profiles, we decided to develop a modified version of this survey system that included contextual questions. Since the Profiler survey used the published NETS performance profiles for teachers, our instrument used the same core questions but with a different database system running on our own server with other instrument changes subsequently discussed.

In the survey, the participant is asked to rate her or his skill or competency for each listed profile within the appropriate preparation level. The four levels are: (a) General Preparation, for those students who are on target to enter a teacher preparation program but have not begun formal educational training; (b) Professional Preparation, for those students already enrolled in core teacher education classes; (c) Student Teaching/Internship; and (d) First Year teacher. These levels and descriptions were also drawn directly from the NETS materials provided by ISTE.

While Profiler uses the following scale: 1) Not at all; 2) Minimally; 3) Confidently; and 4) Able to teach others, the surveys used in this study asked participants to respond to each item using a 5-point scale: 1) unable; 2) basic; 3) developing; 4) proficient; and 5) advanced. For this investigation, we have isolated our focus to the General Preparation and Professional Preparation populations to enable the depth and breadth of analyses required to conduct the forthcoming analyses. There were simply an insufficient number of participants who completed the Level 3 and 4 versions to enable the quantitative model testing proposed in this design.

Based on preliminary analyses from early 2001 data, we decided to probe into the programmatic explanations to the kinds of responses we were receiving. Revised versions of the surveys were constructed and used for the December 2001 through April 2002 data collection period. In addition to the NETS statements, the participants were asked to reply to the question "How did you learn this skill? (self-study, working with a friend, at work, working with a professor, a particular course, etc.)." The reply location for these follow-up queries was an open-ended text box falling below each of the forced-choice proficiency self-ratings. In our findings, we refer to students' explanations for how they learned each competency as the attributed learning context.

Procedures

This instrument was delivered through a web-enabled database portal that ensured all participants were current students in the program. Students conducted these surveys in the context of selected teacher education courses in which the instructor guided the student as to which survey to take. Because some courses were composed of students at various points in the program, there was an additional level of control over guaranteeing data in this report were from relevant stakeholders. The students indicated their current course and academic status (freshman, sophomore, etc.). In cases where there was a clear contradiction in these classifications, the participants' data were dropped from all analyses (e.g., a first year teacher taking Level 1 or 2 surveys).

Analyses

To answer the three primary questions, both quantitative and qualitative methodologies were employed. First, to examine the structural representations of educational technology competency for students at these two levels of NETS integration, we made use of exploratory and confirmatory factor analyses. For both levels, the first step was to examine the structure of responses to items on the survey through exploratory factor analysis using the first data capture (Spring 2001-November 2001), employing the oblimin rotation to allow for correlations among the factors and cross loadings. This oblimin rotation was selected due to the level of overlap in the NETS proficiency statements, suggesting that several items should load on more than one of the six proposed factors. After identifying the likely factor structure with the exploratory factor analysis, a confirmatory factor analysis (CFA) was conducted to further validate that structure solution using the data from the second round of data collection (December 2001-April 2002). The factor loadings for the CFA were used to identify the items comprising each factor.

Once the items comprising the factors were determined, three groups were formed based upon the self-reported scores on each factor. These groups were established to enable comparisons of responses to the "where did you learn this" open-ended queries regarding educational technology based on self-reported proficiency levels. The low-proficiency group included participants' whose average self-rating on the items for the factor fell be-low 2.33 on the 5-point scale. The moderate-proficiency group included those whose scores ranged from 2.34 to 3.66, and the high-proficiency group included those whose scores ranged from 3.67 to 5.0.

The factor analytic procedure also provides information relevant to our second question regarding differences among Level 1 and Level 2 preservice teachers' conceptual representations of educational technology proficiency. In this sample, variation in the number or structure of factors could be an indication of differential personal representations as students move from Level 1 to Level 2.

Finally, the third question regarding "how" the students in Levels 1 and 2 gained their level of proficiency required qualitative research methodologies to synthesize the open-ended responses (only available for the December 2001-April 2002 data capture). This was accomplished through a grounded theory framework, attempting to identify and isolate relevant concepts represented in the data (Glaser, 2002). Following the guidelines of prevailing theories of qualitative research, we employed a recursive, or iterative, procedure to establishing a cadre of concepts or categories that represent students' explanations for how they acquired these skills and competencies.

To create a representative list of attributions for acquired skills, we relied upon our experiences with the teacher education program and students to generate a list of categories. Second, we explored the students' responses to the question on how they gained each of the identified skills in the two leveled surveys. A point of clarification for these analyses is that students were asked after each item to report how they learned that specific skill. Thus, the responses provide specific information for relatively discrete domains of activity related to educational technology. After we examined a representative sample of the available data, we generated a highly specific list of explanations provided by participants (for instance, making a distinction between computer science and educational technology courses). The extended list is displayed in Table 1. Once these categories, or concepts, were generated, the process of coding the responses was undertaken by one of the authors. Discussions among the authors and other informed colleagues helped establish a reliable and valid set of coding criteria. In all coding decisions, the primary goal was to isolate the key or primary experience that the student referred to as essential to her or his development of that skill or ability. When that was not possible, a separate code for "multiple experiences" was entered. To increase the levels of parsimony and coherence in our theory building, and to avoid what Glaser (2002) referred to as "conceptual foppery," we have used the data and our experiences to collapse initially distinct concepts into relevant groups (see Table 1, column 1, for final list).

RESULTS

The results have been organized to present information about NETS Levels 1 and 2 sequentially. For each level, we describe the exploratory factor analysis (EFA), followed by the confirmatory factor analysis (CFA) approach. The initial attempt at factor analysis for each NETS level will not be presented in full detail due to the overlap with the current process. The first factor analysis was a CFA design used to confirm the six primary standards embedded in the performance profile at each level. However, the initial 6-factor model did not provide a reasonable fit for the data (as will be seen in the patterns demonstrated by the exploratory models), even using the oblimin rotation that allows cross loadings as required by the representation of multiple levels for most proficiency indicators.

Following the presentation of the CFA results, a discussion of the qualitative responses to the question regarding where the competency was learned is provided. In addition to identifying the most common student attributions for learning educational technology, we explore different patterns of identified educational contexts that supported learning about educational technology based upon overall ratings of competency.

Level 1: General Preparation

Factor structure. The General Preparation survey was examined through an exploratory principal components analysis, using data from the April 2001-November 2001 sampling pool (n = 126). The results of the factor analysis demonstrated there was one clear factor, although the exploratory model produced two potential factors using the eigenvalue > 1.0 strategy. Notice that there was no evidence that there were six interrelated factors represented in these data, as we initially intended to investigate to connect the items directly to NETS.

The single-factor solution was clear in examining both the unrotated and rotated solution matrices (see Table 2 for pattern matrix with oblimin rotation), where the factor loadings for all variables were high on the first factor (eigenvalue = 9.44, % variance = 55.51), while no variables loaded significantly on the second factor (eigenvalue = 1.23, %variance = 7.23).

The confirmatory factor analysis (CFA) for the second data capture period (n = 130) confirmed that there was one dominant factor, explaining 61.43% of the variance in participants' responses (eigenvalue = 10.44; see Table 2). We have referred to this single factor solution as "Technology Self-Concept" as the items address several different issues within educational technology competency, but at least center around a notion of self-perceived ability to use technology in a variety of settings.

Attributed learning context. Content analyses of the students' attributed learning context for each of the items in the General Preparation survey revealed that students in the core preparation phase leading to the teacher education program identified self-instruction as the overwhelmingly most common method for learning educational technology competencies. Overall, the second most common response to the open-ended query targeting the students' attributions for learning educational technology was through university course experience.

We explored these attributional statements more deeply by investigating differences among the patterns of responses provided by participants in the low-, moderate-, or high-proficiency groups for this measure. Table 3 reveals that there were relatively similar patterns among the groups, with self-study occurring as the most common explanation for having learned skills on the "Technology Self-Concept" factor for all three groups. A chi-square analysis was conducted to compare just the high- and low-proficiency groups. Restriction to the two extreme groups was established to limit misrepresentation of group status due to measurement error. Chi-square analyses revealed there was no significant difference in the distribution of participants' responses for the General Preparation open-ended responses, [chi square](5) = 9.52, p = .09.

Level 2: Professional Preparation

Factor structure. Replicating the procedures used for the Level 1 survey, exploratory principal components analysis was conducted on the April 2001-November 2001 sample (n = 374). The results of the factor analysis demonstrated there was one dominant factor (eigenvalue = 13.62, accounting for 56.76% of the variance) with two additional factors meeting the eigenvalue greater than 1.0 criteria. The second factor (eigenvalue = 1.14, 4.77% of variance) contained several items that loaded meaningfully, while the third factor (eigenvalue = 1.00, 4.17%) only appeared to represent one item, which also loaded heavily on the first factor. Correlations among the three factors demonstrated strong positive correlations between factors 1 and 2 (r = .73), with weak correlations with factor 3 (Factor 1: r = .16; Factor 2: r = .17). Given the tenuous representation of the second and third factors, consideration for a single factor model was made. However, a two-factor solution appeared to be more telling of the data when examining the items represented in the second factor for the exploratory analysis.

Using the second round of available data (November 2001-April 2002) in a confirmatory factor analysis (n = 335), we constrained the model to create a two-factor solution, once again employing the oblimin rotation to allow for high cross loadings and correlations between the factors. Table 4 demonstrates that this solution provided a meaningful pattern of results.

The dominant first factor, which we refer to again as "Technology Self-Concept" (eigenvalue = 13.63, 56.78% of the variance) is primarily composed of items addressing the ability to use technology in education, and to solve problems with hardware and software. The second factor, which we have dubbed "Policy and Professionalism" (eigenvalue = 1.26, 5.24%) primarily targets issues of using technology in an ethical and legal manner, and using technology to promote professional growth and personal success unrelated to teaching with technology per se. The correlation between the two factors was again significant, r = .80, p < .001.

Although the two factors demonstrate statistically unique representations of educational technology self-concept constructs, there is clearly a high degree of overlap across these two factors, suggesting that one is a subset of the other or that there is a hierarchically superior factor that was not identified in our measurement model. Given the conceptual differentiation between the items in the two factors, we have continued to explore them as separate representations of perceived self-competence with technology.

Attributed learning context. Content analyses of the students' attributed learning context for each of the items in the two factors of the Professional Preparation survey revealed that students claimed far lower levels of self-instruction in the various educational technology competencies, referring significantly more often to university courses. Students completing the Professional Preparation level of the profiling tool differed from those completing the General Preparation level primarily on status in their educational preparation.

Replicating the comparison made between the high, moderate, and low groups on the level 1 factor of Technology Self-Concept with this set of data required examining group differences between the high- and low-proficient responders for the two different factors representing the Professional Participation level scale. Once again, chi-square analyses were restricted to the responses provided by individuals in the high and low groups only to limit the effects of erroneous group placement created by measurement error. The chi-square analysis for the first factor (Technology Self-Concept) revealed a statistically significant difference in the distribution of participants' responses, [/.sup.2](5) = 79.08, p < .001. Examination of the noted differences between the high and low group responses suggests these differences were primarily due to the high rate of indicating having received "No Training" by the low-proficient group, and the higher rate of indicating having learned these technology competencies in university courses by the high-proficient group (Table 5). The pattern of responses for the second factor (Professional Issues) also produced a significant chi-square value, [chi square](4) = 57.19, p < .001. Once again, the differences are primarily in the low group's assertion that they had no opportunity for training, while the high group claimed most training occurred in university courses.

For both factors the moderate-proficiency groups' responses most closely mirrored the high-proficiency group, suggesting that the differences between the high and low groups is better explained through a deficit model, where the low-proficiency group represented those students who did not recognize opportunities to learn educational technology competencies in the university curriculum. This deficit cannot be explained by any explicit differentiation in access to course opportunities, given that comparisons of the rate of subjects' enrollment in technology-based endorsements or minors did not differ among the three groups for the Technology Self-Concept factor, [chi square](2) = 2.46, p = .29, or Professional Issues, [chi square](2) = 3.78, p = .15.

DISCUSSION

We have examined the factor structure of the NETS competencies for Levels 1 and 2, and there is evidence that Level 1 captures only one factor (Technology Self-Concept). These are general preparation students only, and those who replied to the question addressing where they learned their skills referred most often to self-directed learning processes. These students were also more likely to refer back to their high school experiences than those participants completing the Level 2 (Professional Preparation) instrument. Our data suggest that the overall level of preparation in the areas of educational technology was low in the core courses that are precursors to the teacher education program. This stands to reason, as the students had not yet enrolled in the courses most directly impacted by the PT3 initiatives or in advanced educational technology courses.

The participants already in the Professional Preparation phase (Level 2) of the teacher education process reported dramatically higher levels of access to educational technology training in their university coursework. The particular courses most commonly identified in these responses included courses in the Educational Technology program, required education courses ranging from educational psychology to pedagogical methods courses, and a standalone computer science course for educators. In addition to differential attributions for learning educational technology competencies, responses to the Professional Preparation phase survey indicated that there are two factors (Technology Self-Concept; Policy and Professionalism) contained within the NETS standards.

We propose that our data raise pertinent and timely issues related to self-assessments for educational technology, at both the individual and institutional levels. First, we found no evidence for the existence of six distinct domains of competency that are present in all four levels of NETS for teachers. However, it would be misleading to suggest that these data disconfirm such a structure. Three basic explanations for this pattern of results are most reasonable based on the data we have reviewed.

Problematic Measurement

The first explanation is a simple argument based on the conditions of measurement underlying these data. Given the disparate standards and competencies listed for the four levels of teacher preparation, measuring overall individual growth and development in the conceptual representation of educational technology encompassing multiple domains is quite difficult. Given the NETS focus on a progressive building of technology comfort and use, through a process that is not necessarily bound to the principles of continuity in development, making comparisons across levels with the existing materials is fraught with confounding effects. This explanation is initially compelling due to the overwhelming strength of single factors in all of the factor analyses conducted. Although the second factor in the Level 2 analyses was significant and meaningful, it provides limited explanatory power for the respondent's educational technology competencies.

Concept Formation

The second proposition that may explain the changed factor structure from Level 1 to Level 2 is centered on the nature of concept development (Mayer, 2003). When learners employ elaborate rehearsal strategies to embed information or skills into the long-term memory system, their existing cognitive frameworks (i.e., concepts, schemas, knowledge bases) have a large impact on the structure of knowledge representation (Dole & Sinatra, 1998), the likelihood of accurate storage and retrieval (Mayer, 2003), interpretations of ambiguous or ill-defined content (Crick & Dodge, 1994), and their ability to adjust their current models of understanding (Chinn & Brewer, 1993). In the preservice teachers' experiences, the task of becoming competent in the six broad dimensions of educational technology standards requires well-developed conceptual representations of both education and technology. Novices in any domain generally have overly simplistic representations of given domains of knowledge or competency. In the case of preservice teachers in this study, our data and observations of students have demonstrated that a critical mass of undergraduate students at the beginning of the teacher education process have an overly-simplistic view of both of these critical concepts.

Through the process of preparing future teachers, the standards-based teacher education movement (e.g., INTASC) has heightened attention to the need to help students gain an awareness of the multiple dimensions of teacher behavior, ranging from classroom management and pedagogy to communicating with parents and professional behavior. This is the process we see as the next logical step in developing and implementing programs that promote standards in educational technology. However, this should not be taken as an endorsement for creating introductory educational technology courses where the primary goal is to teach NETS to beginning preservice teachers. Such a practice would not likely induce the creation of stable and useful conceptual representations by the students. Simply teaching students what the competencies are is a "surface level processing" approach to learning, which is known to produce limited gains in nearly all domains of learning (Craik & Lockhart, 1972). To promote the development of coherent educational technology conceptual representations that are aligned with the learner's developing representation for teaching and learning, early experiences that allow the students to use technology within meaningful educational contexts is more desirable. Such an approach allows learners to develop their understanding of how to use technology in education as they are building their views of pedagogy, planning, and instruction. This process of simultaneously developing related conceptual representations enhances the probability that these future teachers will view the concept "educational technology" as a fully integrated partner with "education" rather than as an add-on. The add-on view is particularly evident in professional development work, because the teachers have already developed sophisticated representations for the concept "education" and asking for the full integration of a new concept into a well-developed conceptual model is far more difficult that building the integrated model from the base.

The Role of Self-Concept & Self-Efficacy

Self-efficacy is the individual's determination regarding her or his ability to effectively complete a defined task (Bandura, 1989). The level of specificity in self-efficacy judgments is very high, suggesting that most of the indicators used in our study address a more global self-perception, such as self-concept for technology (Cassady, 2002). This third explanation for our results is theoretically tied to both of the other offered interpretations. That is, the nature of this self-report methodology necessitates an indirect and imprecise measurement of the targeted underlying NETS competencies. In addition, if the learner has not formed a complex cognitive representation of the concept in question (i.e., educational technology), then it is not possible for that individual to identify differential skills, competencies, or experiences in multiple domains for that concept.

As the learner develops his or her conceptual representation for educational technology, the level of knowledge and expertise in that area increases, leading to two common self-concept trends. First, learners become more cognizant of their limitations and lack of knowledge for that domain (Marsh, 1990b). That is, as they develop a more diversified conception of "technology," they are more likely to recognize several skills or abilities that they do not possess. This tends to spark the second trend, which is the process of recognizing that the target concept (i.e., educational technology) is comprised of multiple domains of expertise or ability (Marsh, 1990a). It is precisely this trend that we have observed in our data. As the students progress from Level 1 to Level 2, they are demonstrating an awareness of the multidimensional nature of educational technology. It is reasonable to believe that these learners who were as yet only at the professional preparation level will continue to develop more domain-specific self-conceptions of educational technology, perhaps eventually approximating the six-factor structure implied by NETS.

Two additional pieces of evidence support this position. First, the average ratings provided by the participants for both Level 1 and Level 2 items was at the middle option for the response scale ("developing"), with the response means for all 41 items represented on the two levels ranging from 2.6 to 3.4 when translating the response scale options to a 1 to 5 numerical representation. There were no observed increases in level of reported ability despite a clear increase in exposure to explicit training based on the open-ended comments. In addition, the Level 1 respondents were not able to identify as many relevant and structured learning experiences for developing educational technology competencies, relying most often on "self-instruction." Tying these data together provides early evidence that the Level 1 responses may provide unrealistically high self-ratings of competency, and as the students progress to Level 2, their advanced training is offset statistically by their increased awareness of skill deficiencies that they were oblivious to in the early phase of learning.

CONCLUSION AND FUTURE DIRECTIONS

These findings suggest that, during early exposure to educational technology in the teacher education program, students do not progress in a linear growth pattern as the performance profiles imply. Rather, there is a plateau of development in which the students construct new definitions of technology and their own efficacy with it. True development returns when more authentic experience is provided that supports the new constructs of technology integration and the teaching practice.

These findings support the new model for teacher education employed at the institution. Starting in the Fall of 2002, freshmen took an introductory course that provided a foundation for the study of the profession of teaching as well as instruction on the creation of a digital teaching portfolio. Students engaged in web publishing and digital media construction in their very first semester. Some program areas even required a separate technology course in the freshman year.

Students appear to be entering the university with operational computer skills (e.g. word processing, e-mail, web browsing). And, indeed, most own their own computer. However, advanced competencies, those formed from skills that historically were not viewed as part of the teaching profession, appear to find little resonance with students in teacher education programs, unless they are enrolled in the advanced, yet elective, educational technology courses. But while the PT3 project in this institution has promoted changes to the teacher education curriculum to provide a greater level of awareness and integration of the NETS standards, the latency between program redesign and matriculation of students appears as students' low self-assessment during their professional participation phase. This demonstrates that PT3 (via NETS) has raised the bar of expectations of individuals and programs. However, without an instrument to measure how high that bar is, teacher education institutions are prone to using the standards themselves (as has been the case with the performance profiles) despite evidence from this study has shows that those profiles do not meaningfully factor to the six-standard structure.

This study demonstrates the need for an improved instrument for assessing the individual and/or a program. The use of different measurement instruments over time to assess the same fundamental attribute is an additional barrier to effective measurement of change and growth. Given that the assessment instrument is in a continuous state of change through the teacher's development, tracking programmatic change with the profiling tools based explicitly on the four levels of NETS is a tenuous process.

We are currently examining the psychometric function of a revised single assessment instrument for all levels, ranging from Level 1 to beyond the current NETS Level 4. We are hoping to create this tool in such a fashion that experienced teachers who have reached levels of exemplary implementation of educational technology resources and pedagogy can be assessed on the same instrument as preservice teachers at Level 1. Our first step in this process is classic scale development, using all the items from the four NETS levels in data collection efforts, and identifying items with high levels of discriminatory power at varying levels of teacher development (Cassady, 2002). We see this tool as an essential development in the evaluation of programs and individuals. Such a tool would have made the examination more productive for all teachers, regardless of their year in the program or their career.

When we constructed our survey from a nationally recognized one, we did so for comparative purposes. A single instrument is needed that is sensitive to naive early concepts as well as robust enough to reveal growth along a common scale is needed. We do not suggest a problem of validity of the NETS standards themselves; rather, we argue that measuring individual or programmatic success using a series of self-report instruments across the four identified levels of NETS, that do not have shared items upon which to compare snapshots in time, poses a critical threat to validity. Our data suggest that two simpler core constructs--technology self-concept and policy and professionalism of technology--are the factors that the performance indicators in Levels 1 and 2 reveal, not the competence in relation to the six NETS standards themselves.

Our experience has demonstrated that the process of learning to implement sound educational technology practices is a gradual process that is largely replicated across populations. Our data are consistent with this theoretical view, specifically we believe that individuals start with a very broad view of what educational technology is, and therefore have a very broad view of their level of competency with that concept. As the learner becomes more familiar with the variability of skills, competencies, and activities embedded within the broad framework of educational technology, they begin to differentiate their interpretations of their own abilities and skills--leading to a multidimensional educational technology self-concept construct that can continue to be refined and modified with experience.

Table 1 Categories for Contexts in which Participants Reported Learning
NETS

Final        Initial
Category     Categories           Description

University Course Experience:
             Computer Science     A given class or assignment as the
             Course               primary source of learning.
             Educational Tech.    A particular course or assignment.
             Course
             Teacher Education    A particular course in the core
             Course               teacher education program, but not be
                                  a computer science or educational
                                  technology course.
             General University   Any course at the University level not
             Course               included in early categories.
             Tech. Endorsement    A particular endorsement, minor, or
             Program              credential program, not a specific
                                  class.
High School  High School          Learning through experience with a
Experience                        high school course, teacher, or
                                  project.
Self-Study   Self-Study           Learning through trial and error or
                                  personal exploration.
Working with Others:
             Family or Friends    Learning from any friend, relative, or
                                  peer.
             Working with a       Directly working with a professor
             Professor            (either named or general reference),
                                  but not when tied directly to a class.
             Work                 Learned on the job.
Practicum    Experience in K-12   Learning from experience, practicum,
Experience   Setting              volunteering, etc. in a K-12 setting.
Multiple     Multiple Categories  No single category noted as primary
Categories                        cause to understanding.
No training  No Training          No experience or training received.

Table 2 Principle Components Analyses Factor Loadings on Profile-It
Level 1

                                                             EFA     CFA
"I am able to ..."                               Factor:   1     2    1

Observe and experience the use of technology in my major  .75  -.31  .77
field of study.
Use technology to process data and report results.        .76  -.45  .78
Exhibit positive attitudes toward technology that         .74  -.30  .78
supports lifelong learning, collaboration, personal
pursuits, and productivity.
Construct technology-enhanced models, prepare             .76  -.07  .80
publications, and produce other creative works.
Discuss health and safety issues related to technology    .68   .38  .71
use.
Use technology to locate, evaluate and collect            .65  -.35  .75
information from a variety of sources.
Demonstrate a sound understanding of the nature and       .81  -.11  .83
operation of technology systems.
Use technology tools and resources to increase            .74  -.23  .80
productivity, promote creativity, and facilitate
learning.
Demonstrate competency with common input and output       .80   .10  .79
devices; solve routine hardware and software problems;
make informed choices about technology systems,
resources and services.
Evaluate and select new information resources and         .83   .02  .84
technological innovations based on their appropriateness
to specific tasks.
Use content-specific tools to support learning and        .71   .23  .76
research.
Discuss diversity issues related to digital media.        .74   .46  .75
Use technology in the development of strategies for       .77   .14  .82
solving problems in the real world.
Use a variety of media and formats, including             .79   .10  .79
telecommunications to collaborate, publish, and interact
with peers, experts, and others.
Demonstrate understanding of the legal, ethical,          .69   .32  .77
cultural and social issues related to technology.
Use technology to facilitate higher order thinking,       .74   .25  .81
including problem solving, critical thinking, decision
making, knowledge construction, and creativity.
Use technology tools and resources for managing and       .70  -.17  .76
communicating information

Table 3 Participants Explanations for Learning Level 1 Competencies

                       Technology Self-Concept Factor Self-Rating
Explanation            Low    Moderate  High

Self-Study             50%    44%       58%
University Course(s)   26%    22%       18%
High School             7%    11%        6%
Work with others       15%    14%       10%
Practicum               0.5%   1%        0
No Training             1.5%   8%        7%

Note. Percentages are based on percentage of groups' responses to all
items falling in that response category, not percentage of members from
group using that response.

Table 4 Principle Components Analyses Factor Loadings on Profile-It
Level 2

                                        EFA Factor       CFA Factor
"I am able to ..."                     1     2     3      1    2

Identify benefits of technology to    .74  -.01   .08    .93  -.22
maximize learning and facilitate
higher order thinking.
Identify technology resources         .44   .14   .61    .86  -.08
available in schools and analyze
how accessibility to them affects
planning for instruction.
Identify, select, and use hardware    .58   .11   .44    .81  -.04
and software technology resources
specially designed for use by
PK-12 students to meet specific
teaching and learning objectives.
Differentiate between appropriate    -.14   .77   .32    .63   .15
and inappropriate uses of
technology for teaching and
learning while using electronic
resources to design and implement
activities.
Plan for management of electronic     .59   .21   .18    .68   .18
resources within a lesson by
identifying potential problems and
planning for solutions.
Design and teach                      .84  -.02   .09    .75  -.05
technology-enriched learning
activities that connect content
standards with student technology
standards and meet the diverse
needs of students.
Identify specific technology          .66   .22   .03    .66   .23
applications and resources that
maximize student learning, address
learner needs, and affirm
diversity.
Plan and teach student-centered       .89  -.14   .05    .70   .13
learning activities and lessons in
which students apply technology
tools and resources.
Design and peer teach a lesson        .83  -.02  -.06    .40   .43
that meets content area standards
and reflects the best practices in
teaching with technology.
Research and evaluate the             .54   .24  -.02    .52   .30
accuracy, relevance, bias, and
comprehensiveness of information
resources used by students.
Discuss technology-based              .81   .10  -.21    .37   .48
assessment and evaluation
strategies.
Examine multiple strategies for       .85   .08  -.12    .49   .41
evaluating technology-based
student products and the processes
used to create those products.
Examine technology tools used to      .78   .09  -.03    .55   .30
collect, analyze, interpret,
represent, and communicate student
performance data.
Integrate technology-based            .83   .05  -.26    .62   .26
assessment strategies and tools
for evaluating learning
activities.
Develop a portfolio of                .55   .18  -.01    .37   .38
technology-based products from
coursework, including the
assessment tools.
Identify and engage in                .73  -.06   .26    .27   .57
technology-based professional
education and lifelong learning,
including distance education.
Apply online and technology           .70   .09   .001   .43   .45
resources to support problem
solving and related decision
making for maximizing student
learning.
Participate in online professional    .10   .59   .25    .11   .61
collaboration.
Use productivity tools to complete    .37   .35   .06    .18   .57
required professional tasks.
Identify technology-related legal     .17   .63   .04    .19   .57
and ethical issues, including
copyright, privacy, and security
of systems, data, and information.
Examine acceptable use policies       .11   .78  -.16    .00   .84
for the use of technology in
schools, including strategies for
addressing threats to security of
technology systems, data, and
information.
Identify issues related to            .24   .67  -.15   -.05   .89
equitable access to technology in
school, community, and homes.
Identify safety and health issues     .15   .76  -.23   -.17   .96
related to technology use in
schools.
Identify and use assistive            .03   .79  -.03    .06   .76
technologies to meet the special
physical needs of students.

Table 5 Participants' Explanations for Learning Level 2 Competencies

               Technology Self-Concept    Policy and Professionalism
Explanation    Low    Moderate  High      Low  Moderate  High

Self-Study     28%    22%       21%       40%  27%       26%
University
  Course(s)    36%    54%       61%       27%  45%       56%
High School     0.4%   0.1%      0.2%      0    0.6%      0
Others         12%    14%       13%        9%  14%       12%
Practicum       6%     3%        2%        5%   2%        5%
No Training    17%     6%        2%       19%  12%        1%

Note. Percentages are based on percentage of groups' responses to all
items falling in that response category, not percentage of members from
group using that response. Further, self-reported factor level (low,
moderate, high) was determined separately for each factor.

Note

1. SCRTEC is now High Plains Regional Technology in Education Consortium (HPRTEC). See Profiler at http://profiler.hprtec.org

References

Bandura, A. (1989). Human agency in social cognitive theory. American Psychologist, 44, 1175-1184.

Cassady, J. C. (2002). Learner outcomes in the affective domain. In J. Johnston & L. Toms Barker (Eds.), Assessing the impact of technology on teaching and learning: A sourcebook for evaluators (pp. 35-66) Ann Arbor: University of Michigan, Institute for Social Research.

Chinn, C., & Brewer, W. (1993). The role of anomalous data in knowledge acquisition: A theoretical framework and implications for science instruction. Review of Educational Research, 63, 1-49.

Craik, F.I.M., & Lockhart, R.S. (1972). Levels of processing: A framework for memory research. Journal of Verbal Learning and Verbal Behavior, 11, 671-684.

Crick, N.R. & Dodge, K.A. (1994). A review and reformulation of social information-processing mechanisms in children's social adjustment. Psychological Bulletin, 115, 74-101.

Dofe, J., & Sinatra, G. (1998). Reconceptualizing change in the cognitive construction of knowledge. Educational Psychologist, 33(2/3), 109-128.

Dye, J.F., Schatz, I.M., Rosenberg, B.A., & Coleman, S.T. (2000, January). Constant comparison method: A kaleidoscope of data [24 paragraphs]. The Qualitative Report [Online serial], 4(1/2). http://www.nova.edu/ssss/QR/QR3-4/dye.html

Glaser, B. (2002). Conceptualization: On theory and theorizing using grounded theory. International Journal of Qualitative Methods [Online serial], 1(2). Retrieved March 5, 2003 from http://www.ualberta.ca/~ijqm/backissues/1_2Final/html/glaswer.html

Grandgenett, N., Jones, J., Pawloski, B., Timms, M., & Ostler, E. (2002). Online and ontarget: Strategies for assessing the educational technology competency of preservice teachers. Presentation at the Society for Information Technology and Teacher Education International Conference 2002, Nashville, TN.

High Plains Regional Technology in Education Consortium (2001). Technology integration survey for faculty. Retrieved March 6, 2003, from http://profiler.scrtec.org

International Society for Technology in Education (2001). ISTE NETS essential conditions for teacher preparation. Retrieved March 6, 2003, from http://cnets.iste.org

International Society for Technology in Education (2003). Use of NETS by State. Retrieved March 6, 2003, from http://cnets.iste.org/docs/States_using_NETS.pdf

Jolly, D. & Clark, F. (2002). Intergenerational learning: A model for teacher preparation. Proceedings of the Society for Information Technology and Teacher Education International Conference 2002, 1, 1349-1350. Retrieved from http://www.aace.org/dl/index.cfm/fuseaction/View/paperID/11101

Marsh, H.W. (1990a). Causal ordering of academic self-concept and academic achievements: A multiwave longitudinal panel analysis. Journal of Educational Psychology, 82, 646-663.

Marsh, H.W. (1990b). A multidimensional, hierarchical model of self-concept: Theoretical and empirical justification. Educational Psychology Review, 2(2), 77-172.

Mayer, R.E. (2003). Memory and information processing. In W.M. Reynolds & G.E. Miller (Vol. Ed.), Handbook of psychology: Volume 7 educational psychology (I.B. Weiner, Series Ed.), 47-58. Hoboken, NJ: Wiley.

MATTHEW STUVE AND JERRELL CASSADY

Ball State University

Muncie, IN USA

[e-mail]

[e-mail]