What's a "hit"? an analysis of a web-based learning environment.

The purpose of this study was to conduct a qualitative and quantitative analysis of usage patterns at a virtual museum, a web-based learning environment. Specifically, this study sought to examine extant documents to profile users interaction patterns at a SJSU Virtual Museum. The quantitative portion of the study dealt with interactivity patterns; that is, electronic "browsing," "grazing," and "pogo sticking." The qualitative portion involved an analysis of e-mail correspondence directed to the site's webmaster. This analysis dealt with correspondence in six broad categories: (a) editorial assistance, (b) hyperlinking, (c) suggestions for expansion of the site, (d) acknowledgment of web site utility, (e) virtual scholarship requests, and (f) recalling personal histories.

**********

The World Wide Web (WWW or Web) or Internet has become an integral tool in educational communities for dissemination of information, research, teaching, and communication. While many position papers have been written about web-based learning environments, little research has been conducted into the effectiveness of this media or milieu. Indeed, Windschitl (1998, p. 28) pointed out that "unfortunately, the uncritical, popular attention given to Internet initiatives is rapidly becoming disproportional to the amount of substantive classroom research on learning derived from these projects." Available literature has focused on the development of models for distance education using the Web or Internet (Chang, Lin, Hao, Suarez, & St. Lawrence, 1997; Fryer, 1997; Saltzberg & Polyson, 1995), teaching and learning in cyberspace (Khan, 1997; Polyson, Saltzberg, & Goodwin-Jones, 1996; Yakimovicz & Murphy, 1995), improvement of "traditional" instruction using Internet or web-based ancillaries (Dodge, 1995; Kaye, 1996), l ayout and design of web sites (Slough & McGrewZoubi, 1996; Wilkinson, Bennett, & Oliver, 1997), development of communities of virtual scholars (Mende, 1996), levels of user interactivity in electronic learning environments (Campbell, 1995), and using web-based learning to meet the needs of workplace literacy (Peraya, 1995; Rosen, 1996; Wagner, 1995).

Empirical studies which have investigated usage patterns at web-based information sources (Descy, 1997; He & Jacobson, 1996; Romenesko, 1996) have focused on the number of electronic visitors to a web location, without inquiring as to what users do and/or learn during their electronic visits. This study seeks to address questions associated with what students learn in an electronic environment, while establishing baseline data and methods with which to analyze similar web-based resources.

The site investigated in this study, the SJSU Virtual Museum (http:// www.sjsu.edu/depts/Museum/aamenu.html), was designed as an electronic reference source to provide teachers, students, and parents with information about the lives of famous scientists and mathematicians. The current study sought to investigate user information-seeking behaviors at this location. Additionally, this study raises discussion about methods of determining user logon patterns and factors affecting web site usage.

Significance for this study rests in timeliness, research that impacts a large population, establishing baseline data regarding the use of web-based information sources, and establishing tools and techniques with which to evaluate the effectiveness of Internet resources.

DESIGN, DATA COLLECTION, AND ANALYSIS

This study is what Richey and Nelson (1996, p. 1223) would classify as Type I Developmental Research Investigation. This study was designed to provide information (a) to improve an instructional product, (b) to investigate the conditions that promote successful use of a product, and (c) to provide insights into the efficient design, development and/or evaluation of an instructional product. To interpret the level and type of user interactivity, the following substantive research questions were addressed:

1. How frequently was the web site accessed?

2. Is the "hit" useful in measuring user or logon patterns?

3. Did usage vary over time? If so, how?

4. What information were users seeking?

5. What is the nature of user correspondence to the virtual museum webmaster?

Data used in this study were obtained from two sources, the web usage log (stored on the server) and e-mail letters addressed to the webmaster. Descy (1997, p. 50) pointed out that every time a user sends e-mail, Telnets, FTPs a computer, or accesses a web site they are leaving information at the destination. This information about the domain of the user (also called a "cookie"), logon date, duration of logon, and names of files accessed is automatically recorded on the server log every time a site is "hit". In the current study these raw data were "decommaed" and parsed (put into tabular format) using Microsoft Excel[TM]. The data were exported into Statview[TM] for statistical analysis.

E-mail letters addressed to the SJSU Virtual Museum webmaster were saved as they arrived and were used as an additional data source. During the year that data were collected, 108 e-mail letters where received. This represents correspondence from 1.28% of the web site visitors (8,390 logins were recorded on the user log during the year of the study).

As part of this study, a survey was undertaken of webmasters at 12 Silicon Valley corporations to establish a meter for comparison. The webmasters surveyed reported that commercial web sites, offering "freebies" or tangible rewards for logging on to a web site, typically expect correspondence from 3 to 5% of their electronic visitors. Thus, the correspondence rate at this web site was within parameters that might be expected from an Internet location that offers no tangible rewards to the user.

SUBSTANTIVE RESEARCH QUESTIONS

In determining the characteristics of user interactivity at the SJSU Virtual Museum, a series of substantive or working research questions were formulated and addressed using the data.

Substantive Question #1: How Frequently Was this Web Site Accessed?

In addressing this first substantive or working research question, data from the usage log were analyzed. Results showed that the number of browsers outnumbered grazers in a ratio of about 3:1. During its first year of operation the SJSU Virtual_Museum was accessed 8,390 times, as measured by the number of domain names recorded in the usage log. The majority of hits (73.85% or 6,196 hits) were single usage visitors, that is, individuals who accessed only one of the 50 files during their electronic trip to the virtual museum. These individuals, those who logon to a site and rapidly logoff, may be thought of as electronic "browsers" or gawkers.

A smaller number of visitors (26.15% or 2,194 individuals) opened multiple files during their visits. These multiple usage visitors opened an average of 4.57 files per visit with a range of 2 to 17 files. Repeat visitors, individuals who visited the SJSU Virtual Museum more than one time during the year, were detected by counting the number of duplicate domain names from the user log. These electronic users may be thought of as electronic "grazers," those who visit a site and explore it in detail.

Substantive Question #2: Is the "Hit" Useful in Measuring User or Logon Patterns? Why or why not?

Because the current method of counting a "hit" can result in artificially raised counts as well as omission of "hits" directly to secondary pages, "hit" count should not be the sole source of measuring user or logon patterns. The "hit" or a logon to an Internet home page is considered an "industry standard" for measuring the effectiveness of web-based advertising or dissemination of web-based information. "Hits" are typically recorded as the number of visitors who logon to the home page of a particular web site. In the case of the & ISU Virtual Museum, a counter on the home page (filename: a menu) would have recorded 2,391 visitors to the web site during its first year of operation since this the number of times the main menu file was opened. The 5,999 individuals who logged directly on to secondary pages of the web site would not have been "counted" by most web counting loops. A counter attached to the main menu of this site would have recorded only 28.49% of the actual usage, a large under reporting of the electronic traffic to the web site. In an age of web browsers, when individuals can directly access a portion of a site without logging on to the main menu, the practice of counting home page "hits" should be reexamined.

A second phenomena, that of electronic "pogo sticking" on the part of the users, also impacts web counter accuracy. Electronic "pogo sticking" or moving from the main menu to branches of a web site and back to the main menu, also called hyperlinking, can alter the number of recorded visits as measured by a home page counter. From the usage log at the SJSU Virtual Museum it was noted that 2,194 visitors used the home page as a menu to branch to other displays at the web site. Electronic "pogo sticking" may artificially raise the number of clicks recorded by a home page or main menu counter.

Substantive Question #3: Did Usage Vary Over Time? If so, How?

Although usage increased, it is not clear if this was due to registering the site with search engines or the increased demand caused by the start of a new school year. To address this question, data from the web usage log (stored on the server) were examined.

It is apparent that web access to the electronic museum increased in a linear fashion during the first year of operation. The SJSU Virtual Museum was installed on a server in January, 1996. During the first eight months of operation, the site was unregistered. In August, the URL (web address) of the site was registered with three search engines (Yahoo, AltaVista, and InfoSeek). The number of visits to the site increased dramatically during the last four months of the year. It is unclear whether registering the site with these search engines or the opening of a new school year (teachers and their students were the largest users of the site according to addresses of the) accounted for the increase in electronic traffic.

Substantive Question #4: What Information were Users Seeking?

To get a clear picture of the type of information sought by users, data regarding access to the home page and copyright information was removed. In addressing this question, data from the user log were examined. The SJSU Virtual Museum provides bibliographical sketches and photographs of famous scientists and mathematicians.

While the user log provides information about the total web site usage, the inclusion of information about home page access and access to copyright information skews the data in terms of the type of information that users sought. Table 1 provides usage patterns with the influence of home page and copyright files removed.

Substantive Question #5: What is the Nature of Users Correspondence with the Virtual Museum Webmaster?

By far, the largest number of e-mail letters to the webmaster were suggestions to expand the web site. One hundred eight (108) letters e-mailed to the webmaster during the first year of operation were collected and analyzed. The content of those letters is distributed across six broad categories: (a) editorial assistance, (b) hyperlinking, (c) suggestions for expansion of the site, (d) acknowledgment of web site utility, (e) virtual scholarship requests, and (f) personal histories. A distribution of those comment categories is shown in Table 2.

The largest number of e-mail letters to the webmaster (n = 49 or 45.37% of the correspondence) addressed suggestions to include biographies of other scientists and mathematicians to the list of those currently displayed at the site. These comments included statements such as: "You might consider adding George Dantzig to your list" and "Include more disabled Americans at your site." In total, the names of 35 different scientists and mathematicians were forwarded from users for consideration for inclusion at the site. Humor characterized some of the correspondence in this category, one apparently young writer penned the following request, "Please include my teacher, Mr. J_____ at your virtual museum, he's a real astronaut."

The second category of correspondence addressed to the webmaster was concerned with hyperlinking requests (n15 or 13.89% of the e-mail notes). Correspondence in this category included requests to link the virtual museum to existing sites "You might add a link from Mendel to Mendel-Web." Other correspondence in this category included requests to link existing sites to the SJSU Virtual Museum. These electronic notes included statements such as, "Permission is requested to link your site to..."

The third most popular type of correspondence was in the area of personal relationships or stories. Eleven of 108 letters sent to the web master (12.96%) addressed individual's personal interactions with scientists and mathematicians. Those individuals had a personal experience in dealing with the scientist and wanted that information recorded as part of the information at the web site. Correspondence in this category included statements such as "In a personal correspondence with Linus Pauling I learned..., and I feel this information should be included at your site." and "I knew Grace Hopper when..."

The search for grammar, punctuation, and spelling errors, and historical accuracy was of concern to 13 of the 108 individuals who wrote to the webmaster (12.03% of the notes). These comments were sometimes accurate "I noticed that you have the incorrect birth date for Grace Hopper." At other times the information was not accurate "You misspelled Susan LaFlesche's name, it should be Susan Lafletch."

The value of the web site as a resource for research was the fifth largest category of correspondence. Ten of the 108 notes to the webmaster (9.25%) addressed the value of the site in providing information on the lives of scientists and mathematicians. Information in this category included comments such as, "Thank you for highlighting the work of Barbara McClintock."

The final category of correspondence (n=7 or 6.48%) was concerned with virtual scholarship. Correspondence in this category included requests from doctoral students to analyze the site for their dissertation research (n=4), questions about ways to conduct qualitative and quantitative analysis of web sites, and a request for collegial contact regarding processes involved in developmental research.

CONCLUSION

The Internet or Web holds tremendous potential to provide users with information in nontraditional educational settings. The virtual museum investigated in this study is an information resource, one that may be used anytime, anywhere educational settings.

An analysis of the server log provided insights into the usage patterns at this web site. First, users at this site tended to be "browsers"; that is, single time, single file users. "Grazers," multi-file and/or multi-time visitors, accounted for a smaller portion of the electronic visitors. Second, while "counters" attached to a home page provide some information about visitor usage patterns, they may not give an accurate picture of web site usage. At this particular web site, a counter attached to the home page would have produced misleading information since a large number of electronic visitors entered secondary pages rather than the main menu or home page. Further study should be given toward developing a more effective method of accurately measuring user patterns. A suggestion may be to investigate user patterns of an entire web site instead of focusing on individual webpages. Third, web site visitation increases over time, whether that was attributable to registration of the site or seasonal demand is uncertain at this time. It may be beneficial to further investigate the impact of registering a web site on web site usage. This may be accomplished by comparing similar web sites; one registered the other not registered. An alternative approach would be to investigate web Site usage over a period of time when the web site in not registered versus when the web site is registered.

Finally, a small percentage of electronic visitors chose to correspond with the webmaster. Feedback for this site, an information resource, was provided by approximately 1% of the users. This correspondence rate is less than that experienced by commercial web sites which offer incentives to correspondents. Correspondence to the webmaster was typically concerned with (a) suggestions to improve the site, (b) requests for hyperlinks to other sites, (c) shared personal histories, (d) editorial assistance, (e) critique of the value of the web site, and (f) requests for virtual scholarship.

While this study has provided models for qualitative and quantitative analysis of web sites, much remains to be done to assess the effectiveness of web-based learning resources. Procedures used in the analysis of this site should be replicated at other sites, and finally, other methods should be developed to analyze web-based learning, to provide information about students learning in web-based environments.

Table 1

Frequency of File Access at Virtual Museum

File name                # hits  % use

alv - Luis Alvarez          406  2.50%
arm - Neil Armstrong        350  2.16%
bab - Charles Babbage       574  3.54%
bac - Roger Bacon           491  3.02%
bai - Liberty Bailey        293  1.81%
bei - Alexander G. Bell     294  1.81%
ber - M. Berthelot          224  1.38%
bla - E. Blackwell          421  2.59%
blu - Buion Bluford         293  1.81%
boh - Niels Bohr            309  1.90%
bun - Robert Bunsen         113  0.69%
bur - Luther Burbank        365  2.24%
cars - Rachel Carson        588  3.62%
carv - G.W.Carver           294  1.81%
dav - Humphry Davy          251  1.55%
fer - Enrico Fermi          281  1.73%
foc - Jean Focault          307  1.89%
gag - Yuri Gagarin          281  1.73%
ham - Alice Hamilton        448  2.76%
hop - Grace Hopper         1106  6.82%
hyd - Ida Hyde              364  2.24%
hyp - Hypatia               281  1.73%
kau - Joyce Kaufman         251  1.55%
kel - William Kelvin        168  1.04%
kep - Johannes Kepler       167  1.03%
koc - Robert Koch           197  1.21%
laf - Susan LaFlesche       252  1.55%
lam - Berthe Lamme          280  1.72%
lat - Lewis Lafimer         195  1.20%
lea - Henrietta Leavitt     281  1.73%
lim - Robert Lim            252  1.55%
lis - Joseph Lister         167  1.04%
lye - Charles Lyell         197  1.21%
mau - Antionia Maury        253  1.55%
mcc - B. McClintock         195  1.21%
men - Gregor Mendel         115  0.70%
opp - R. Oppenheimer        237  1.47%
pas - Blaise Pascal         250  1.55%
pee - Florence Peebles      143  0.87%
per - Gertrude Perlmann     139  0.86%
pla - Max Planck            140  0.86%
ree - Walter Reed           168  1.04%
sal - Jonas Salt            196  1.21%
she - Alan Shepard          140  0.86%
ter - V. Tereshkova         336  2.07%
wat - James Watt            138  0.86%
wri - Dorothy Wrinch        226  1.38%
yal - Rosalyn Yalow         308  1.89%
Table 2

Distribution of E-mail Correspondence (n=108) to Webmaster

Category of Correspondence      Number of  Percentage of
                                Responses    Responses

Suggestions for site expansion     49         45.37%
Hyperlinking requests              15         13.89%
Personal histories                 14         12.96%
Editorial assistance               13         12.03%
 (grammatical errors)
Web site utility                   10          9.26%
Virtual scholarship                 7          6.48%

References

Campbell, J.O. (1995). Interactive distance learning: Issues and current work. Journal of Instruction Delivery Systems, 9(3), 32-35.

Chang, I.F., Lin, L.C., Hao, X., Suarez, H., & St. Lawrence, J. (1997). Cyberspace assisted responsive education implemented on the Internet (I-CARE). Journal of Asynchronous Learning Networks, I(1). Retrieved from the World Wide Web April 1, 2002, from: http://explorer.scrtec.org

Descy, D.E. (1997). The Internet and education: Some lessons on privacy and pitfalls. Educational Technology, 37(3), 48-51.

Dodge, B. (1995). WebQuests: A technique for Internet-based learning. Distance Educator, I(2), 10-13.

Fryer, B. (1997). Better WBT: Beating the traffic jam. Inside Technology Training, I(8), 28-32.

He, P.W., & Jacobson, T.E. (1996). What are they doing with the Internet? A study of user information seeking behaviors. Internet Reference Services Quarterly, I(1), 31-51.

Kaye, J.C. (1996). Characteristics of effective networking environments. Paper presented at the Annual Meeting of the American Educational Research Association, New York. (ERIC Document Reproduction Service No. ED 394 502)

Khan, B.H. (1997). Web-based instruction (WBI): What is it and why is it? In B.H. Khan (Ed.), Web-based instruction (pp. 5-18). Englewood Cliffs, NJ: Educational Technology Publications.

Mende, R. (1996). Building global learning communities through the Internet. Paper presented at the Annual Conference of the Association of Canadian Community College, Toronto, Ontario, Canada. (ERIC Document Reproduction Service No. ED 397 928)

Peraya, D. (1995). Distance education and the WWW. Education at a Distance, 9(7), 20-23.

Polyson, S., Saltzberg, S., & Goodwin-Jones, R. (1996). A practical guide to teaching with the World Wide Web. Syllabus, 10(2), pp. 12-16.

Richey, R.C., & Nelson, W.A. (1996). Developmental research. In D.H. Jonassen (Ed.), Handbook of research for educational communications and technology: A project of the Association for Educational Communications and Technology (pp. 1213-1245). New York: Simon & Schuster Macmillan.

Rosen, D.J. (1996). How easy is it for adult educators to use the Information Superhighway? (ERIC Document Reproduction Service No. ED 392 964)

Saltzberg, S. & Polyson. S. (1995). Distributed learning on the World Wide Web. Retrieved from the World Wide Web April 8, 2002, from: http:// www.syllabus.com/archive/syll95/O7_sept95fdistrlrngwwWeb.txt.

Slough, S.W., & McGrew-Zoubi, R. (1996). Area under constructivisism: A pilot study using a World Wide Web home page to assess professional development. Paper presented at the Annual Meeting of the National Association for Research in Science Teaching, St. Louis, MO. (ERIC Document Reproduction Service No. ED 396 919)

Wagner, J.O. (1995). Locating vocational education curriculum and instructional materials. (ERIC Document Reproduction Service No. ED 385 777)

Wilkinson, G.L., Bennett, L.T., & Oliver, K.M. (1997) Evaluation criteria and indicators of quality for Internet resources. Educational Technology, 37(3), 52-58.

Windschitl, M. (1998). The WWW and classroom research: What path should we take? Educational Researcher, 27(1), 28-33.

Yakimovicz, A.D., & Murphy, K.L. (1995). Constructivism and collaboration on the Internet: Case study of a graduate class experience. Computers & Education, 24(3), 203-209.