Using developmental research to design, develop, and evaluate an Urban Education Portal.

The Teacher Education Network (TEN) PT3 Catalyst Grant, in conjunction with its partner, the Urban Network to Improve Teacher Education (UNITE), created an Urban Education Portal for its members that provided interactive and personalized access to a variety of digital resources and tools. The process used to create the portal and manage the digital resources is presented, using Richey and Nelson's (1996) Type I developmental research as the conceptual framework. A general analysis of the design, development, and evaluation process is followed by the results of three rounds of usability testing by typical users.

**********

BACKGROUND

UNITE is an outgrowth and subset of the Holmes Partnership, a network of local universities and schools in collaboration with national professional associations and community agencies that has had a long history of addressing educational reform issues. The Holmes Partnership is committed to creating high quality professional development and significant school renewal. UNITE is comprised of 31 urban teacher preparation programs in the United States that are committed to preparing future teachers for urban schools and that focus explicitly on fostering equity and social justice in urban teacher preparation. As a national initiative of urban education partnerships, UNITE is engaged in a thoughtful redesign of teacher preparation and induction that begins with an understanding of and commitment to urban communities and that is responsive to these communities. UNITE's members are informed and dedicated urban teaching professionals who provide research and resources to support education practitioners in the unique and dynamic urban environment.

The Teacher Education Network (TEN) is a Preparing Tomorrow's Teachers to Use Technology (PT3) Catalyst Grant. Awarded in 2000, this grant has six goals: to develop, strengthen, and disseminate Web-based tools for personal learning, planning, and digital portfolios for preservice teachers; to develop and nationally disseminate to preservice programs instructional materials and strategies for technology-enhanced contextual learning; to create online graduate programs in urban teacher preparation for preservice teachers and non-licensed or under-prepared practicing educators nationally; to develop a national Web-based clearinghouse on PT3 strategies and materials; to revise National Staff Development Standards to address quality in distance-based teacher development; and to institutionalise TEN as a growing national collaboration on technology-facilitated preservice reform and technology in preservice preparation.

Members of the TEN partnership include the Holmes Partnership; UNITE; the National Education Association (NEA); the National Staff Development Council (NSDC); the International Society for Technology in Education (ISTE); the Great Cities Universities Urban Educator Corps, a national consortium of 19 large urban universities with teacher preparation programs; the Association of Teacher Educators (ATE); the Quality of Life project at New England College, a group that uses the Contextual Teaching and Learning (CT & L) model to help current and future teachers design and engage in standards-based community learning projects; the Society for Information Technology and Teacher Education (SITE); and the National Institute on Community Innovations (NICI), a nine-state network of 30 Professional Development Schools (PDSs) committed to infusing technology into K-12 education. NICI also provided programming and technical support for the UNITE/TEN team.

Over the past three years, TEN actively engaged its various partners in nominating and reviewing high quality online resources that were customized to their particular audiences. TEN created a set of interoperable Educational Reform Portals at http://edreform.net, and is currently working to establish strong partnerships for each portal effort to ensure that the contents are highly valued and effective. NICI's collaboration with UNITE and the Holmes Partnership focuses on supporting the efforts of post-secondary educators to use technologies effectively for teaching, learning, and collaboration, and to infuse equity resources and strategies into all aspects of their practices. The Urban Education Portal project grew out of the need to make available exemplary curricular materials and information on effective strategies for transforming teacher preparation, produced or reviewed by UNITE members. The design goal of the portal was to create a set of resources that changed over time, engaged the professional community that makes and uses the resources, and customizes itself to the user.

CONCEPTUAL FRAMEWORK

The conceptual framework that describes the process of creating the Urban Education Portal and managing the digital resources is based on developmental research, as identified by Richey and Nelson (1996). Developmental research is "the systematic study of designing, developing and evaluating instructional programs, processes, and products that must meet the criteria of internal consistency and effectiveness" (Seels & Richey, 1994, p. 127). In our case, we will be describing the development of the urban education portal and simultaneously analysing the process.

Richey and Nelson (1996) recognize two types of developmental research. Type 1 developmental research "involves situations in which the product development process used in a particular situation is described and analysed, and the final product is evaluated" (p. 1216), whereas Type 2 developmental research requires a general analysis of the design, development, or evaluation processes either as a whole or as a single component. For the purpose of our research we followed the Type 1 developmental research methodology. Type 1 developmental research helped us structure the different components of our project by identifying our product focus, process focus, use context, tools and techniques, research methods, and nature of our conclusions. The outcomes of our ongoing study will be lessons learned from developing the Urban Education Portal and an analysis of the conditions that facilitate their use.

PRODUCT FOCUS

The Urban Education Portal, located at http://urban.edreform.net, is an online environment that provides organized information and offers resources and tools to users on the topics of preservice teacher preparation, teacher induction, community-based urban education, equity and social justice, and research and policy. The Urban Education Portal is more than a Web site that archives information on the four topics related to UNITE's mission; it organizes and personalizes the resources for easy search and retrieval. Once resources are entered into any of the portal categories, UNITE/TEN reviewers then assess their validity using a set of standards. These standards include relevancy to urban teacher education, high quality, research-based, and significance to the field.

PROCESS FOCUS

The process focus of the portal involved the research phase, development phase, resource review, and pilot test.

Research Phase

During the research phase, a group of TEN PT3 Grant consultants explored the Dublin Core (DC) project and other global metatagging projects (Stephens, 2001). That led the group to build a catalog or metatagging vocabulary that was interoperable with other global databases that are using similar standards. These projects exist mostly in Europe, but there are also some in the United States. One example of such a project is the Education Reform Portals, which encourages the education reform community to develop metadata for disseminating educational research findings (Kurowski, Knapp, McLaughlin, & Gibson, 2003). The application underlying http://www.edreform.net demonstrates the value of the Semantic Web approach to cataloguing and retrieving digital information.

A portal that uses Semantic Web technology is different from a standard server-based database in several ways. One important difference is that the portal can "poll" the WWW for new updates of any records in any other metatagged set, and can allow a constant two-way communication to occur. One example of this is when users customize a page on CNN (http://www.cnn.com) or Excite (http://www.excite.com), and then every time the user logs in, the site tells that user the local weather. Programs such as these use Semantic Web technology to go to the WWW and get the latest weather update. The program also selects the weather topics to be displayed on the user's screen.

A second important difference is that Semantic Web technology allows applications to develop "Web services" that share data sources, even if they were built independently and remotely from one another. For example, an instruction and assessment application in California can send essays to Colorado, where they are picked up and scored by people in Vermont, and then returned for display. Semantic Web applications can help build digital catalogs of resources that take advantage of a decentralized network of experts. Intelligent routing of those resources can then respond to queries that express the essay score, a multidimensional score from a survey, and other profiles of a user's strengths, interests, and aspirations. Human advisors and teachers can utilize new forms of network-based assessments to provide guidance to learners and validation of learning, resulting in highly personalized instruction, guidance, and assessment applications.

Development Phase

The product development phase consisted of programming, metatagging, and resource review procedure identification.

Programming. The programming component of the development phase was accomplished by the TEN programming team. As part of its National Web-Based Clearinghouse activities, the team began developing the system architecture by building a Digital Equity Portal for the PT3 Program Office, located at http://digitalequity.edreform.net. The Digital Equity Portal presents a collection of educational resources addressing inequitable access to technology tools, computers, and the Internet. Once the underlying tool for cataloguing portal resources had been created, it was then a natural step to begin to apply the tool to new content areas. Some early usability issues involved respecting privacy and considering appropriateness when representing people as resources, and the problem of people's familiarity with keyword searching getting in the way of seeing the resource collection in a new way. The TEN programming team is still working on solutions for these two issues, even after several revisions of the portal architecture. A record of the architecture development can be found at http://sourceforge.net/projects/liber/ with the latest version at http://leeber.sourceforge.net/.

The issue of people as resources comes up because when portal users are looking for resources, they usually search for information texts, Web sites, articles, and other online documents. When the TEN programming team presented portal users with people who had tagged themselves as being able to help, facilitate, offer services, or advise others, our users said, "Don't give us those." The team is now working on a way to allow the user to decide whether or not to mix "people resources" with text and multimedia resources.

The familiarity of a keyword search is a well-ingrained search habit for users of the Web, online libraries, search engines like Google and Yahoo, and even commercial sites with catalog stores. The Urban Education Portal offered a keyword search as a touchstone of familiarity, but the portal had been more powerfully organized around a different searching metaphor. A selected resource could be used to find similar documents, where groups of documents were already bound together into collections in which a portal editor (or editorial board) made a professional judgment to select works and tag them into certain categories. This makes the NICI collection more of a hybrid between a "browsing" collection where a user should roam around and look, sample, and select items as relevant and appropriate for their specific needs and a "searchable" collection that only uses keywords. The TEN programming team is still working on how best to allow both types of search processes to be supported in one environment.

The portal will eventually contain data gathering tools that are employed by both the user and the site. An example of a user-selected tool might be a self-assessment, the results of which are analyzed and displayed immediately, and which form a user profile. The profile acts as an agent on behalf of the user to go into the resource collection and build a customized set of those resources that fit the assessment profile. It enables the user to describe his or her interests and expertise, search the tagged resources, and have the portal display a ranked list of relevant online resources. An example of a portal selected tool is the user model. A user model might evolve over time as the user visits the portal to select resources or add new resources. That updated profile can then act as an agent for the user, going into the collection to build a customized set of resources in response to the actions taken, and articles already ordered.

The TEN programming team is also working on a hierarchical representation system, which will allow a more sophisticated cross-relationship of resources and provide an essential boundary feature needed for evolution of the portal's resource set. The boundary function brings up the possibility of self-organized portals and feedback loops relating one level of a portal to another, or displaying several portals inside of another portal. These possibilities allow a level of complexity that the team members are just beginning to grasp.

Metatagging. Metadata can be thought of as data about data. Collections organized by metatags share but go beyond the features of traditional library indexing and referencing systems. Digital references using metadata are put together using a basic semantic structure that relates a resource to a structured vocabulary system called an ontology (Berners-Lee, Hendler, & Lassila, 2001). The vocabulary or metadata ontology is, in a sense, the evolving structure of digitally represented knowledge as developed by the World Wide Web Commission (W3C). The W3C has been working for years to develop agreements about the structure and parts of the ontology as well as protocols for its further development and use. The metadata ontology of the DC (see http://dublincore.org/) global project contains the notions that all resources will have an "author," "date of creation," and so forth. Metadata are applied to resources in triplets. That is, a resource acts like the subject of a sentence, a metadata element (e.g., "author") acts as a predicate, and the specific information exemplifying that element for that resource acts as the object. This fundamental schema is the foundation for the Semantic Web.

The predicates or metadata elements are also called metatags, and they come in two types: (1) those with open text fields, and (2) those with controlled vocabularies. An example of an open text field is the field for the tag "author," where we need a number of different, essentially innumerable answers to the question of who authored a resource. An example of a controlled vocabulary is the field for "resource type," since there is a much smaller number of types of digital resources, including film, books, articles, videos, audio files, graphics, and the like. Alternative conceptions of controlled vocabularies can co-exist without harming the basic semantic relationships. For example, one community might want "resource type" to include more or fewer words than in the DC list of elements. The "subject area" is another example for which one might use both or either the Library of Congress or Dewey Decimal system subject headings.

The identification and development of the metatag structure for the Urban Education Portal were carried out by the UNITE/TEN team. The team identified four primary threads: preservice curriculum, recruitment, induction, and continuing professional growth. Within each thread, other hierarchical metatagging structures were included, such as learning and development, teaching, assessing, planning, field experience, advocacy and equity, partnerships, mentoring/coaching, communities of practice, and the like. These threads are not static because the more users navigate through the portal searching and sharing resources, the more they will be engaged in creating their own customized channel for accessing the online resources in the portal.

Resource Review Procedure Identification. The UNITE/TEN team identified the procedure for review of nominated resources for the Urban Education Portal. After a resource has been submitted, the portal editor accepts or rejects the resource. If the resource is accepted, then it is automatically sent to one of five technology liaisons, representing each of the five UNITE work groups. The technology liaisons then select three people from their list of review volunteers to review the resource using the following UNITE/TEN evaluation standards:

* Is the article relevant to urban teacher education?

* Has it been peer reviewed by a trusted source?

* Does it inform practice, policy, and/or program?

* Is it grounded in practice, theory, and/or research?

* Is it current/timely?

The reviewer has one month to complete the review. The technology liaisons have the ability to check the status of the evaluation via the portal. It should be noted that a resource becomes available through the portal after its acceptance by the portal editor. However, for it to receive the UNITE recommendation, it must first go through the review process. Resources to be evaluated by reviewers may fall into the following three categories: research study, article, or paper; documents from a teacher preparation program; or book review.

Pilot Test

During the weekend of October 18-20, 2002, UNITE hosted its semiannual meeting in Orlando, Florida. At the opening session, members of the UNITE/TEN team presented the shell for the newly launched UNITE Web site, the Urban Education Portal, the UNITE Online Campus, and the other online resources available by request for UNITE members through the TEN PT3 Grant. One of the UNITE/TEN team members then handed out a suggestion form and asked all UNITE members to identify digital curricular materials that could be linked to the portal in order to strengthen the urban preservice curriculum. UNITE members were asked to provide the title of the resource and a brief description, list the URL or contact person information, and to select a category that would be used to catalog the resource: educator guide, lesson plan, reference, research study, study guide, unit of instruction, or "other." Finally, they were asked to check which of the four primary threads applied to the resource, including eight secondary threads for preservice curriculum.

The following day, the UNITE/TEN team, with representatives from the Holmes Partnership, met to review and update its members on the development of the portal and the other online tools, and to discuss the submission and a possible refereeing process for materials to be placed on the Urban Education Portal. Prior to the meeting in Orlando, the team had received a set of online resources from one of its members to "seed" the discussion about nominating, cataloguing, and reviewing resources. It became clear that the original eight secondary threads for preservice curriculum were not necessarily appropriate for cataloguing online resources within the four primary threads presented at the opening meeting.

The original metatagging scheme was based on 12 interrelated strategies for urban education used by Great Cities Universities/Urban Educator Corps and described at http://www.gcu-uec.org/home.html. But during the meeting in Orlando, the emphasis changed to reflect the expertise from people who lead preservice teacher education. Their focus was on the "pipeline" of teacher preparation, induction, retention, and ongoing professional development. Since this involved a change in scope of the metatagging process, the bulk of the day's meeting was spent in brainstorming, evaluating, and reaching consensus on a new metatagging scheme that would be used for the resources to be nominated by the UNITE members. The four primary threads remained the same, but primary and secondary threads now comprised a more extensive set. Several tertiary threads were also identified, but more refinement was warranted before the portal editor would be able to catalog the nominated resources efficiently.

Once the review process was approved, the TEN programming team then designed and tested an application to formalize and automate the editorial review process using criteria of relevance, quality, and usefulness. The metatagging procedure and the resource review procedures were both tested and worked as planned.

Testing the Metatagging Procedure. In a follow-up activity, one of the UNITE/TEN team members e-mailed 10 sample documents to the entire UNITE/TEN team and asked them to read and evaluate one article from each of the three categories: research study, article, or paper; documents from a teacher preparation program; and book review. One of the reviewers sent a message to the team the next day, suggesting that the set of criteria be applied while reviewing the documents. Two members of the team reviewed all of these resources.

Testing the Resource Review Procedure. The following week, one of the UNITE/TEN team members sent an e-mail message to the entire UNITE/TEN team with directions for submitting a resource to the Urban Education Portal. She requested that each member submit one resource to test the directions for uploading resources, as well as the functionality of the portal, and to let her know by the end of the week if anything was unclear or not working. The directions were clear and explicit, and the uploading process worked as planned. Once the uploading process was tested, the director of UNITE sent a message to the entire UNITE membership, describing the portal and encouraging each member to contribute one online resource to the portal before the next semi-annual meeting, scheduled for February 2003 in Washington D.C.

USE CONTEXT

Portal materials were cataloged with the purpose of allowing teachers, teacher union members, teacher educators, administrators, and students to access, use, and share effective resources. UNITE members nominated seven resources via the paper suggestion forms during the Orlando meeting. More resources were received by e-mail over the next month, bringing the total to 20 new resources that were added but not cataloged. The number of nominated resources increased over time. As of April 2003, there were over 100 resources "pointed to" by the Urban Education Portal, including resources that had not yet been accepted by the portal editor. All resources that were added after April 2003 underwent the quality review process.

TOOLS AND TECHNIQUES

The UNITE/TEN team used a variety of tools and techniques to accomplish our developmental research, including face-to-face meetings, e-mail discussions, conference calls, interviews, surveys, and artifact analysis.

RESEARCH METHODS

The two members of the TEN PT3 Grant who served on the UNITE/TEN team were asked a set of structured questions via e-mail and provided extensive information about their progress to date. Their responses provided a major contribution to this article.

Members of RMC Research Corporation, serving as the external evaluation team for the TEN PT3 Grant, acted as participant observers at all UNITE/TEN meetings and participated in conference calls with team members to assess progress on the programming, metatagging, and resource review procedure development.

During the fall 2002 and spring 2003 UNITE meetings, UNITE members were asked to complete a survey about their familiarity, use, and perceptions of appropriate audiences for each of the TEN online tools, including the portal. They were also asked to provide any suggestions for improvement of the online tools. Usability testing was then carried out on the portal, and the results were used for continuous improvement and portal development.

RESULTS TO DATE

Familiarity and Use

At the semi-annual meeting of UNITE in October 2002, 100 UNITE members were surveyed regarding their perceptions of the portal and all of the tools developed by NICI for the TEN project. Twenty-six members who completed the survey (26%) were generally in the "emergent awareness" stage, having just learned about the newly developed Urban Education Portal at the October meeting. The only four members who reported that they "examined but haven't used it" were members of the UNITE/TEN team who were advising the programming team about the metatagging process and the document review process. The survey was repeated at the following UNITE meeting in February 2003, using the same instrument. Eleven UNITE members (11%) completed the follow up survey. The results indicated that the respondents were beginning to move from the "emergent awareness" to the "early exploration" phase. Survey results also indicated that usability was an issue that needed to be addressed; therefore, user feedback was used for continuous improvement by the programming team. Because of the low response rate and the large number of items in the survey instrument, it was not possible to carry out a sophisticated data analysis of the initial and follow-up survey results.

Respondents felt that the portal audience could comprise preservice teachers in a four-year teacher preparation program, preservice teachers in a fifth year teacher preparation program, preservice teachers at a Professional Development School, preservice teachers in an alternative certification program, inservice teachers in a post-baccalaureate program, inservice teachers seeking professional development, teacher educators at a college or university, supervisors of field placement and/or student teaching, and faculty in other academic departments where aspiring teachers take courses.

Usability Testing

In fall 2002, the TEN programming team hired an external usability consultant to test the portal interface. Her suggestions were implemented, and all of the Education Reform Portals were redesigned by April 2003.

UNITE then performed two rounds of usability testing: a pilot test in April 2003 with a group of 16 typical users in the course entitled, "Using Technology with Adult Learners," at the University of Wisconsin at Milwaukee, and a field test with 18 UNITE members from 14 different member institutions in June 2003. All questions used a rating scale of 1=low to 5=high. The results are presented in Table 1.

Suggestions for improving the ease of use of the portal included requesting e-mail announcements of new, interesting, or relevant resources; suggesting ways of translating what works in one venue to another; requesting specific guidance for using the portal; and being sure that documents were useful to teacher educators. One respondent wished to view the main information categories on each page. When asked what types of resources they would want to look for on the portal, respondents requested research studies, articles, papers, documents from a teacher preparation program, and book reviews. Some ways in which respondents reported they utilized the information they accessed included:

* Browsed and reviewed resources;

* Shared with colleagues;

* Applied to current instructional planning;

* Applied to current programs; and

* Promoted professional growth.

Using Data for Continuous Improvement

In early November 2003, the portal interface was redesigned, using feedback from the pilot testers and field testers. Users were now presented with a set of four dimensions of educational reform, thereby enabling them to "browse the shelves" of the portal's collection of online resources using the metatag searching scheme. However, they could still choose to "search the catalog" using a standard keyword search scheme, if they so desired. Additional online resources were nominated, reviewed, and cataloged. A "my collection" button was also added in order to begin the next phase of portal evolution by asking new users to register and fill out a self-assessment survey regarding their needs, interests, and level of knowledge about urban teacher preparation. The portal is also ready and able to connect an online survey or assessment information and use the results to configure a person's unique collection at a portal site. As far as UNITE is concerned, NICI can implement a survey for it at any time if UNITE surveys on the very dimensions that they use to catalog materials. For example, if a survey question's answer amounted to someone saying "I want to learn more about best practices concerning communities of practice," then the survey would perform that search and return the resources marked that way.

In December 2003, a letter was sent out to all UNITE members requesting that they respond to another usability survey for the redesigned portal. The same survey instrument was used. Fifteen UNITE members from 13 institutions completed the online survey. All but three of the respondents were from institutions that had not participated in the initial field test and were not involved in the initial portal design conversations. The results of the third round of usability testing were compared with the results from the initial pilot and field tests and are presented in Table 2.

An interesting pattern emerged from the second field test. The changes in these ratings were probably not great enough to be considered significant, as supported by the responses to the open-ended questions. The feedback regarding the portal's design and navigation was generally positive and included:

* Very professional and quick;

* It was easy to get into;

* Very text-based--graphics are nice;

* A little too busy, lists of beneficiaries too long, and

* I feel this is improved over past offerings.

Nearly all respondents indicated that they would like to look for research studies (100%) or articles (93%), while about half of the respondents wished to see more papers (47%), documents (53%), and book reviews (40%). Respondents were also interested in seeing bibliographies of urban education articles, video demonstrations of strategies to use in urban classrooms, and newsletters with current information. Suggestions for further improvements included adding a description of the criteria for resource selection and "how to" offer additions; implementing the user-specific collections feature; a way to search for a particular need; making navigation possible within each subcategory without having to use the back button; speeding up the loading time for the site; adding blank lines between items in a long list of resources; subdividing lists of resources by type such as article, Web site, or bibliography; and indicating which resources the user had already looked at.

Some ways in which respondents reported they utilized the information they accessed included:

* Browsed and reviewed resources;

* Used for mentoring research;

* Used to write professional publications and grants;

* Shared with colleagues, action research group, and preservice educators;

* Applied to current instructional planning and teaching strategies;

* Used to learn about urban education programs; and

* Used for personal edification and professional growth.

Current Status

The Urban Education Portal is up and running, and it empowers users to do complex searches using a simple interface. The interface has undergone usability testing and has evolved from a programmer-friendly to a more user-friendly appearance. Clear directions orient the user to the difference between typical keyword-searching and the more powerful metatag-based searching used in the portal. The automatic review process has been implemented, and new digital resources are being added. Site statistics, presented in Table 3, indicate that usage of the portal has increased continuously since its initial redesign in April 2003.

However, at this time, keyword-searching produces more resources than browsing the catalog, unless the user wishes to perform a fairly broad search. For example, when someone visits the catalog page, that person is presented with several lists, each one representing a specific kind of categorization scheme (e.g., "subject" or "author"). To perform a simple search, the user selects one item from a list and then clicks the "Find Resources" button at the bottom of the page. To find resources about administration, the user goes to the catalog page, scrolls down to the list labelled "Subject," and clicks on "Administration." After hitting the button at the bottom of the page, the user is presented with a list of resources addressing administration.

Significance of the Study

The "portal story" is just part of a larger picture that comprises all of the technology-based tools and supports that the TEN PT3 Grant has been attempting to offer to UNITE. For example, http://edreform.net is actually an umbrella mega-portal that houses an integrated, interoperable array of Web portals, each providing digital content showcasing best practices in a single critical educational reform dimension rather than a single clearinghouse of educational resources. Likewise, TEN customized and provided responsive technical support, training, dissemination, and system improvement for each organization that pilot tested its portal with its own unique audience and requested additional features, such as a customized self-assessment survey to create a user profile for the Great Cities Universities-Urban Educator Corps' (GCU-UEC) Preservice Technology Infusion Portal, which is located at: http://preservicetech.edreform.net. Like UNITE, GCU-UEC is vitally interested in issues of recruitment, preparation, retention, and professional development for urban educators, and TEN is a partner in GCU-UEC's own PT3 Catalyst grant. Recently, GCU-UEC and the TEN programming team worked with ISTE to redesign the Preservice Technology Infusion Portal's user assessment survey and to align it with the ISTE NETS Essential Conditions for Teacher Preparation.

The ability to create new portals for other educational and professional organizations, based on the programming efforts and the developmental research carried out by the UNITE/TEN team will greatly facilitate sustainability of the entire array of portals, should user interest continue to grow. NICI is also beginning to use the portal application to construct new digital collections within the graduate programs. This effort is quite young, but the goal is to have masters and doctoral students build online collections in content such as critical theory, best practices, learning theory, etc., and to have them make regular contributions across the existing portals.

One of the members of UNITE worked closely with TEN in presenting the TEN PT3 Grant's resources to each of the five UNITE work groups, training the technology liaisons from each work group, customizing resources for them, and asking for feedback. These activities were crucial to the success of the portal activities, although they progressed more slowly than the UNITE/TEN team would have liked. One reason for this is the fact that national networks may operate differently from traditional organizations in their ability and speed of technology infusion. Another reason is that developing a hierarchical metatagging structure is not a well-defined process, which thus leads to complex, sometimes tangled relationships among resources, people who can help others use the resources, and the highly variable contexts of usefulness of the resources. Clearly, the development of such collections as the Urban Education Portal requires an iterative process. The process started with phases of teacher preparation and then went deeper into the elements important at each stage. The Urban Education Portal has been through three phases. At each TEN meeting more people become interested in what was being done and provided more input. The Urban Education Portal will be fine-tuned at least one or two more times. The hierarchical metatagging structure has never been done before by anyone in the world of education. The NICI programmers are creating something that is new. That is one reason the portals are so special.

The UNITE/TEN effort is part of a larger effort to create a rich and complex evolving hierarchy that can allow for multiple contexts for a resource to be found and used. For example, suppose there is an electronic resource about the Civil War. It might be useful in a history class, a class about social justice, or a class about literature and media. Each of these contexts will have a specialized vocabulary and purpose for that resource, with a host of different ancillary pieces of information to enhance the resource's use in that context. Such a resource might appear in more than one portal, with different surrounding materials and in different parts of the ontology or hierarchy of each portal. That is part of the challenge of envisioning and building a cataloging structure like http://www.edreform.net, within which the UNITE/TEN portal has been created.

CONCLUSION

The UNITE/TEN example illustrates an approach taken by NICI to developing a collection of online resources. The approach starts by working with a professional community that understands the needs of its clients and audience as well as the resources (print, media, and human) needed to address those needs. The professional vocabulary specific to an area of expertise such as educational reform sits in contrast to the vocabulary of professional librarians who catalog resources for general audiences, making the resources more relevant to the current and changing needs of the professional community. The professionals also organize online resources in a way that promotes their values and perspectives, in contrast with general cataloging approaches that employ a structural approach such as using an ontological standard like the Dewey Decimal system. The goal of engaging user audiences in addition to professional catalogers in the work of building collections and forming metatagging terms and relationships is, therefore, to create more interactive and personalized access that is relevant and timely for the profession while honouring time-tested structures of knowledge.

Table 1 Initial Usability Test Results

Usability Criterion              Pilot Test (N=16)     Field Test (N=18)

Relevance of Resources           4.2                   4.0
Clarity of Resources             4.1                   3.9
Timeliness of Resources          4.5                   3.9
Usefulness of Resources          4.4                   3.7
Appropriateness of Resources     4.4                   4.1
Ease of Use of Portal            4.3                   4.3
Portal Navigation                4.3                   4.1
Portal Interface                 4.0                   4.2

Table 2 Usability Test Results for Initial and Redesigned Portals

                                                      Field Test with
                              Pilot Test  Field Test  Typical Users
Usability Criterion           (N=16)      (N=18)      (N=15)

Relevance of Resources        4.2         4.0         4.3
Clarity of Resources          4.1         3.9         4.3
Timeliness of Resources       4.5         3.9         4.5
Usefulness of Resources       4.4         3.7         4.2
Appropriateness of Resources  4.4         4.1         4.3
Ease of Use of Portal         4.3         4.3         4.1
Portal Navigation             4.3         4.1         3.9
Portal Interface              4.0         4.2         4.1

Table 3 Site Statistics for Urban Education Portal

Month      Number of Visits

April        71
May         130
June        733
July       1370
August     1168
September  2138
October    2930
November   2982

Acknowledgment 

This developmental research project was supported by a 2000 Preparing Tomorrow's Teachers to Use Technology (PT3) Catalyst Grant to the Teacher Education Network.

References

Berners-Lee, T., Hendler, J. & Lassila, O. (2001). The semantic web: A new form of web content that is meaningful to computers will unleash a revolution of new possibilities. Scientific American, 284, 5, 34-43.

Kurowski, B., Knapp, M., McLaughlin, R. & Gibson, D. (2003). An application of the semantic Web for school improvement. National Institute for Community Innovations (NICI): Unpublished manuscript. Available from [e-mail]

Richey, R.C. & Nelson, W.A. (1996). Developmental research. In D.H. Jonassen (Ed.), Handbook of Research for Educational Communications and Technology, pp. 1213-1246. NY: Macmillan.

Seels, B.B. & Richey, R. C. (1994). Instructional technology: The definition and domains of the field. Washington, DC: Association for Educational Communications and Technology.

Stephens, R.T. (2001). Improving Web maintenance with code documentation. Retrieved December 16, 2003 from: http://www.truststudy.com/Research/Course/Spring2001/Web%20Documentation.pdf

SIMONE CONCEICAO

UNITE, University of Wisconsin, Milwaukee USA

[e-mail]

LORRAINE SHERRY 

RMC Research Corporation, Denver, Colorado USA

[e-mail]

DAVID GIBSON 

TEN PT3 Grant, Montpelier, Vermont USA

[e-mail]