User-based data collection techniques for evaluating networked information services.
Networked information services are increasingly being developed for a range of network users and potential users. The passage of the High Performance Computing Act of 1991 (P.L. 102-194) authorized the development of the National Research and Education Network (NREN). The Clinton Administration's National Information Infrastructure: Agenda for Action (Office of the President, 1993) will promote even greater development of information services over the networks. But the development and implementation of those services have not kept pace with ongoing evaluation of networked services. Increased attention must be given to the evaluation of networked information services. Moreover, the evaluation of these services must be user-based as opposed to system-based. To date, there have been very few formal attempts reported in the literature of user-based assessments of networked information services.
The notion of networked information services is an evolving one. Such services can be offered by individuals, libraries, computer centers, publishers, networks, government agencies, or a host of other organizations and groups with access to the Internet and the evolving NREN. Networked information services comprise bulletin boards; e-mail; listservs; remote access to distant databases, software, and high speed computing; and collaborative efforts among geographically dispersed individuals--to name but a few (LaQuey, 1992). A key aspect of "networked information services" is that there are numerous providers--local and remote; there is a range of electronic information services available to users; and access to and use of these services continues to increase.
Evaluation is the process of identifying and collecting data about specific services or activities, establishing criteria by which their success can be assessed, and determining both the quality of the service or activity and the degree to which the service or activity accomplishes stated goals and objectives (Van House et al., 1990). As such, evaluation is a decision-making tool that is intended primarily to: (1) ensure that the highest quality services are provided to intended users of that service, and (2) assist decision makers in allocating necessary resources to those activities and services that best facilitate the accomplishment of organizational goals and objectives (Hernon McClure, 1990). Unfortunately, many networked information services are designed without user input and, worse, are inadequately (if at all) evaluated by those for whom the service was originally intended.
User-based evaluation and determination of user needs should be considered as part of the strategic planning process for the development of networked information services. Thus, developers of networked information services constantly need to ask:
* Who are the users of the service and how well are they able to identify and access a particular service?
* To what degree do networked information services enhance or detract from users' ability to accomplish specific tasks?
* What information resources and services are "most" important for network clientele and how well does the network deliver these services?
* What are the costs and benefits of specific networked information services and to what degree do these services meet the objectives of both the provider and the user?
* What are the specific strengths and weaknesses of the information services and how do these services affect different user groups?
* Would the provider of the networked information service receive more or better benefits by reallocating resources to new or different information services?
While this list is not intended to be comprehensive, it suggests that user-based evaluation of networked information services should accompany the design and implementation of such services. Overall, we need a better understanding of how well networked information services meet (and anticipate) user information needs.
The purpose of this article is to: (1) provide an overview of the importance of user-based evaluations of networked information services, (2) review a number of data collection techniques that provide a user perspective when assessing networked information services, and (3) offer practical suggestions and guidelines for using such techniques. The data collection techniques discussed in this article have been used by the author in a number of studies related to electronic networking (McClure et al., 1994). A key theme throughout the article is that ongoing evaluation--as part of the strategic planning process--is essential in the design and successful operation of networked information services.
NEED FOR A USER PERSPECTIVE
If the Internet/NREN and other new electronic systems and services are to be successful, they must be integrated into the working lives of users in those communities they are meant to serve. Such integration depends upon identifying and addressing a number of social and behavioral issues related to the use of networks by the various users (McClure et al., 1991). A user perspective should consider the culture of the communities and subcommunities involved; the relationship between community norms and the use of electronic networks; effects of networks on collaboration and scholarly communication; definitions of eligible users and acceptable uses; relationships among users in academia, government, and the private sector; and the training and support of onsite and remote users of networked facilities.
A user-based evaluation perspective considers issues such as: How can the use of electronic networks facilitate the tasks and goals of particular communities of users?
* What problems do particular groups of users face in attempting to exploit networks for the accomplishment of those tasks and goals?
* What design, management, and policy strategies can alleviate those problems and maximize network use and effectiveness?
These and similar questions can be approached by developing and implementing ongoing user-based evaluations of networked information services.
A user perspective assumes that information technologies should not be designed and implemented according to technical criteria alone but should take into account the particular communication behavior, information use patterns, and work environments of potential users. This perspective will help network designers, managers, and users:
* avoid conflicts;
* understand and estimate the impact and benefits of network use;
* choose appropriate network designs, features, and services;
* devise appropriate strategies for marketing network services and
promoting network use;
* develop effective policies for network management and use;
* develop appropriate mechanisms for user training and support;
* evaluate the effects of network implementation.
Adopting a user perspective for evaluating networked information services offers a useful model to better understand the role, importance, and impact of networked information services in a range of organizational settings (McClure, 1992).
IMPORTANCE OF USER-BASED EVALUATION
It is important to differentiate the notion of user-based evaluation of networked information services from a systems or technology perspective of evaluation. The systems perspective might consider total number of packets transported over the network, the number of log-ins to a particular server, or the accuracy with which a particular router moves messages from one system to another. While such assessments are useful, they do not address the degree to which users of the service have their particular needs met, the impact that use of the service may have made on the user, or the ease of access encountered in using a particular service. Systems or technology based criteria for a "successful" networked information service do not ensure successful use of the service from a user's point of view.
Approaches for evaluating networked information services can be based on:
* Extensiveness: How much of the service has been provided--e.g., number of users logging-in per week on a bulletin board or the number of participants of a particular listserv.
* Efficiency: The use of resources in providing or accessing networked information services--e.g., cost per session in providing access to remote users of an online catalog or average time required to successfully telnet to a remote database.
* Effectiveness: How well the networked information service met the objectives of the provider or of the user--e.g., success rate of identifying and accessing the information needed by the user.
* Impact: How a service made a difference in some other activity or situation--e.g., the degree to which faculty network users increased their research productivity or teaching effectiveness by use of networked information services.
Although evaluations of networked information services need to consider extensiveness and efficiency criteria, much more attention needs to be given to effectiveness and impact measures.
Because networked information services are multidimensional, the type of evaluation needed will be multidimensional as well. A single measure provides only one "snapshot" of a particular service; multiple "snapshots" of measures are needed. Moreover, evaluators of networked information services will need to know what type of evaluation approach and data collection techniques will be appropriate for what types of services. The key point is that we need to develop evaluation strategies that are user-based--that is, they examine networked information services from the point-of-view of the user.
Providers of networked information services must not accept as a "given" that their services, resources, and technical procedures are effective; rather, they must test their assumptions about the quality of networked information services through an ongoing process of evaluation. Ongoing evaluation activities are essential to support the provider's planning process. Planning and evaluation are two sides of the same coin. Each will be more successful when the other is part of the overall services design and implementation approach.
USER-BASED DATA COLLECTION TECHNIQUES
Although the focus of this article is on user-based data collection techniques, it is well to remember that these techniques evolve within a larger context of evaluation research. A discussion of evaluation research designs and social science research methods is beyond the scope of this article. Additional information on these topics can be found in a number of useful texts including Rossi and Freeman (1993), Babbie (1992), and Marshall and Rossman (1989). The intent of this section is to highlight a number of data collection techniques useful in developing a user-based evaluation of networked information services. Specifics for using these techniques can be found in textbooks listed earlier or in other research methods texts.
This qualitative data collection technique is an extremely valuable one for obtaining naturalistic insights on how individuals perceive networks and networked information resources (Morgan, 1993). In this technique, the evaluator identifies a particular group of individuals (usually five to eleven people) that meet certain criteria (e.g., very knowledgeable about accessing and using government bulletin boards). The individuals (who typically do not know each other) are brought together and discuss aspects of the topic at hand. The session typically lasts from one to two hours, is conducted in a conference room setting, and there are usually a moderator and a note taker from the study team participating in the session (Krueger, 1988).
A focus group session differs from a group interview in that the participants in the focus group determine for themselves the topics to be discussed and the conversations "evolve" during the session with little guidance and direction by the moderator. The point, in fact, is to allow the topics and discussions to be those of most interest to the participants rather than forcing the group to talk about the topics that the moderator believes to be important. This approach encourages a user perspective, provides data based on the topics that users believe most important, and allows the data collection to inform the evaluator about additional topics that might need attention which otherwise would not have been identified.
Critical Incident Technique
To better understand user perspectives, sometimes it is helpful to have users recall a specific recent experience or incident that occurred regarding use of the network or network information resources. For example, the evaluator is especially interested in difficulties that users might be having in telneting to remote databases. Either through an individual interview or a survey format, the evaluator asks the user to recall the most recent time when he or she experienced a problem in telneting to a remote database.
First the person describes the general experience he or she had with this particular situation, and then the evaluator either probes (if an interview) or asks a standard list of exploratory questions (if a survey) which the user then answers. The preference of this writer is in using individual interviews. When using the interview format, the evaluator has much greater flexibility in probing and follow-up on specific experiences which cannot be done via a survey. The critical incident technique is an excellent approach to focus a user's attention on a particular type of experience.
Another very useful approach is to have users maintain a log that records: (1) the nature of their activities regarding some related network activity, (2) the amount of time spent on that particular activity, and (3) the user's assessment of the usefulness or success of that particular activity. A user log can be designed to collect information on a range of network activities, or it might focus on a particular activity of special interest to the evaluator. It is important that the participants maintaining the log have a high degree of commitment or are provided with some rewards for engaging in this data-collection technique since maintaining the log can be somewhat time consuming. Examples of such logs can be found in Doty, Bishop, and McClure (1992).
Typically, the evaluator will identify specific individuals to maintain the user log over some period of time. The users may be segmented to obtain data from specific user types--i.e., naive users versus experienced users. And, depending on the nature of the study, the specific types of information to be collected on the log may vary. User logs tend to be more useful if maintained over an extended time period--i.e., a month--so that patterns in use may be more easily identified by the evaluator. User logs are an especially important technique as they mirror actual behavior rather than asking the user to describe his or her behavior--behavior which may not be easily recalled or might be skewed in light of other factors.
Network-Based Data Collection
One excellent approach to obtain evaluative information about users, networks, and networked information resources is to use the network itself. The evaluator can establish an online conference on the network about a particular topic and invite selected individuals to participate in the conference. Participants are informed that the discussion on the conference will be used as data and input for the evaluation study. Software is available that organizes the conference into particular topics, encourages individuals to send messages to one or more members of the conference, and otherwise manages the operation of the conference.
An interesting aspect of this approach is that the moderator can play virtually no role in the development of the conference or he or she can take a very active role in the conference by participating and directing the conversations into certain topics or otherwise ensuring that everyone participates and offers their opinion. Moreover, this data collection technique allows participants' views to evolve and inform each other as the conference proceeds. Another benefit of this approach is that users can participate in the conference at times best for them rather than at times determined by the moderator.
Another related technique is to use the network as a means of administering a survey or a set of discussion items. For example, someone might put a short questionnaire on PACS-L asking for responses (to the evaluator and not the list). This technique, while having the merits of being easy to do, has a number of possible problems. First, the evaluator has no control over who will respond or if they will respond at all. Second, there are so many messages on lists that some receivers of the electronic survey may perceive it as junk and discard it immediately--or worse, be put off that you sent the survey out to them at all. In short, response rates on this approach may be so low as to invalidate the results obtained.
Of course, one of the old standbys for data collection is an interview. Interviews can be done with individuals or with groups. The questions posed in the interviews can range from unstructured (little predetermination of topics to be covered) to structured (complete determination of the topics to be covered). The success of this technique is largely dependent on the interviewer's skills as a moderator and facilitator. Interviews have the advantage of allowing the evaluator to probe into topics which cannot be done on surveys. They have the disadvantage of requiring considerable time in both organizing the interviews, conducting the interviews, transcribing the interviews, and analyzing the data resulting from the interviews.
Group Process Surveys
A group process survey is halfway between a survey and an interview. In this technique, the evaluator selects a particular set of participants to examine a topic or issue. In preparation for the meeting, the evaluator has developed a set of discussion topics as a hand-out to participants. During the one to two hour meeting, as the group discusses a particular topic, each participant writes on the hand-out their view of the topic. The moderator can ask that participants write their thoughts on the topic as the discussion is in progress, after the discussion occurs, and before moving on to the next topic, or both before and after the actual discussion.
There are a number of advantages to using this technique. First, and perhaps most importantly, the participants write, in their own words, their views on the topic being discussed so that the moderator does not have to do so from his or her notes at a later time. Second, this approach allows participants to be informed by the discussion and to modify their views in light of how the discussion evolves. Finally, writing one's views on the handout usually results in a 100 percent response rate from participants--which rarely occurs in surveys or in group discussions where a small number of participants can dominate the conversations.
Site visits are similar to a case study approach (Yin, 1989) except that site visits are not likely to be as time consuming and detailed. Generally, case studies have some longitudinal dimension to them since they are conducted over a period of time. A site visit, however, entails less time and is a bit more informal. A site visit generally is planned to obtain first-hand information from tours of specific facilities and services, interviews with individuals or groups, or observation of specific activities at the site. In addition, another aspect of the site visit is also to obtain reports, brochures, and examples of products or services made available at the site. An interesting aspect of site visits is the potential to directly compare and contrast different types of data collection techniques from different sources on the same topic.
In site visits, it is not always possible to predict in advance the range of data collection activities in which the researcher might engage. Clearly, some of the data collection strategies can and should be planned in advance of the site visit--i.e., scheduling interview times, tours, and so on. Additional data collection opportunities, however, may arise as the site visit progresses. Indeed, the evaluator should be extremely conscious of opportunities to meet with individuals or groups that perhaps he or she could not have known about until the site visit occurred. Two major benefits of such site visits are the opportunity of: (1) having first-hand information about users or activities in a particular setting, and (2) evolving the data collection strategies on site depending on the topics the evaluator deems important to probe for obtaining additional information.
An interesting but underutilized data collection technique is scenario development. This technique can be done either as a group or as an individual process. The basic idea with this approach is to have participants discuss "what if..." types of questions and construct a scenario or likely series of events that would need to occur if a particular vision or goal is to be accomplished. Scenario development is an especially useful technique for having participants consider possible future events, speculating about what key assumptions may drive the development of future events, and suggesting what designers of networked services and resources might need to consider if they are to be successful in a particular future scenario.
There are a number of methods for using scenario development as a successful data collection technique (see Amara & Lipinski, 1983). One approach used successfully by this writer is to first carefully define the nature of the scenario to be explored, develop a one-page written draft of an example scenario to use with the group (making sure it is pretested and revised before use), and identifying appropriate topics and questions that need to be explored. As an example, the scenario might be that there are T3 lines into all the branches of the public library. Given that scenario, a number of discussion questions might be used with a group: What services might be provided by the library? How would increased remote access to the library affect the management of the library? and so forth.
The views of group participants when discussing the implications and assumptions for a scenario (or group of scenarios) can provide very useful and insightful perspectives on what users think might or should happen in the future. From the evaluator's point-of-view, these perspectives and insights can be used to identify issues and possible policies that might be needed to deal with the issues.
An important and useful idea within a user-based perspective is describing the activities of users, being able to know what network users do, the amount of time they are engaged in various activities, the tasks for which they use the network, and to learn how they actually go about using particular services and resources. Observing users (in a range of situations or in the use of various services/ equipment) is an extremely valuable approach for obtaining a user perspective.
Observation can be either obtrusive (the user knows that he or she is being observed) or unobtrusive (the user is unaware that he or she is being observed). There are trade-offs and issues to consider for selecting one over the other (Hernon & McClure, 1986). There are a number of useful texts that provide suggestions for conducting a formal observation as part of a data collection technique (Epstein Tripoldi, 1977, pp. 42-54). But one of the most important aspects of using this type of data collection is having a well-developed data-collection form upon which the evaluator can easily and quickly summarize the activities observed, the length of time in which the user was engaged in that activity, and any comments the observer might have at the time of the observation.
And finally, there is the well-known, but increasingly difficult to use, survey technique. In recent times, successful survey research has become quite difficult because of problems in obtaining adequate response rates. Nonetheless, a well-developed and carefully designed survey can oftentimes be used successfully in obtaining user perspectives on networked information resources and services. The experience of this writer, however, is that other types of data collection techniques should be considered prior to using a survey approach.
Surveys have the advantage of being relatively inexpensive to develop and distribute to the intended target audience. They can be designed to be easily analyzed. Their primary drawbacks are obtaining an adequate response rate and ensuring that responses from the intended target audience are, in fact, the ones that were sought. Moreover, people are suspicious of filling out such surveys and are increasingly concerned about confidentiality and privacy issues. Given these concerns, it is essential to have participants who are committed and interested in the study and to provide some rewards (either tangible or intangible) to those who participate in the study.
KEY ISSUES FOR SUCCESSFUL USER-BASED EVALUATIONS
The above section offers an introduction to selected user-based data collection techniques of networked information services. But in the use of these techniques there are a number of key issues that evaluators of networked information resources should keep in mind. In the experience of this author, evaluators should consider these issues carefully as a means of increasing the likelihood of a successful user-based evaluation.
Know Your Audience
When conducting an evaluation of networked information services, it is important to recognize who the audience will be for the evaluation results prior to designing the evaluation and determining what data collection techniques will be used. Potential audiences might be the users themselves; network managers; providers of the service; organizational administrators; government policy makers; or others. A concern, however, is that different audiences may require different evaluation data collection techniques.
Thus, part of the evaluator's responsibility is to understand the information needs of the audience for whom the evaluation is being done. While it is likely that the evaluator will be asked simply to evaluate" a networked information service, some thought should be given to the measures and thus the data collection techniques that might be of special interest to that specific audience.
Carefully Decide What Exactly will be Evaluated
Evaluators will not have the luxury of being able to collect all the data they might want about a particular networked information service due to lack of time, limited budget, inability to acquire the needed information, and a host of other reasons. Thus they will have to have clearly defined objectives of what is to be evaluated. Usually those aspects of the networked information services to be evaluated will be those that are "actionable"--i.e., interventions or strategies could be put in place to improve and modify that aspect of the service. Thus the evaluator will usually target specific types of data to be collected and make certain that they provide the needed information to make the required evaluation assessment.
Develop Appropriate Measures
Another aspect of this issue is recognizing that performance measures for a particular networked information service may have to be developed. For example, in the assessment of the information made available to organizational members from a particular remote file server, the measure "references to file-server information in organizational research reports" might be established. To use this performance measure, however, the evaluator will have to carefully define and operationalize key terms such as "reference to file server information" and "organizational research reports." Then data collection techniques (from those listed above) would have to be considered in light of how well they would provide information on these two data elements.
Determine Costs and Schedule
For user-based evaluations to be successful, they must be done in a timely fashion and with a clear sense of the costs needed to complete the study. In preparation for the evaluation, costs associated with standard budget items--e.g., personnel, supplies, travel, equipment, contract services--should be identified. There is no use in initiating a user-based evaluation for which there are inadequate resources available to complete it. Indeed, it is better to complete a smaller less costly evaluation than to do none at all.
Equally important is to develop a schedule for the completion of the project and detail the key tasks that will have to be done over the duration of the study. There are many types of GANTT tasking charts and project management software programs currently available that can assist the evaluator in scheduling the evaluation. Such scheduling ensures that everyone involved in the project knows what tasks are to be completed when. Further, scheduling allows the evaluator to monitor the progress of the evaluation more effectively and identify possible problems while they can still be resolved.
Identify the Correct Study Participants
One problem of ten encountered by new evaluators is attempting to obtain assessments from sample participants that may not have the necessary information. For example, in the evaluation of a particular government bulletin board, simply collecting data from a random sample of network users may not produce enough participants that have actually used or know about the bulletin board. The general rule of thumb is to not expect users to provide you with information about things that they know nothing about.
This concern is especially important in focus group sessions. If some members of the focus group are extremely knowledgeable about a particular networked service and others are not, the group dialogue can be extremely skewed. One strategy is to carefully consider whether you need information from naive, beginning, or expert network users. Another is to use a filter question in interviews and surveys to determine the type of user and his or her background before you proceed with collecting the information you require.
Develop, Pretest, and Refine Data Collection Instruments
No data collection instrument should be administered without it first being carefully developed, pretested, refined, and oftentimes pretested a second time. User-based data collection techniques require data collection instruments that make sense to that particular group of users. One useful approach is to have the data collection instruments reviewed by: (1) someone with experience in the data collection technique you wish to employ, and (2) a group of individuals who are members of the user group from whom you will be obtaining data. To ensure reliable and valid data, pretesting and refining of data collection instruments is essential (Kirk & Miller, 1986).
Administer Instruments Successfully
The logistics associated with administering data collection instruments can oftentimes be formidable. And since the notion of a user perspective is to make the study participants at ease and able to relate social and behavioral concerns in arranging for a focus group session, one has to not only identify and obtain the cooperation of participants, one also has to (among other things):
* arrange for a pleasant setting to conduct the focus group and,
typically, provide some refreshments and amenities;
* consider the order and development of topics to be discussed in
* have a technique for recording and analyzing the content of the
session while it is occurring;
* manage and moderate the session in a positive and productive
* provide follow-up thank you notes. Similar logistical concerns affect the use of other data collection techniques such as surveys, transaction logs, interviews, and so on. Once again, it is essential that these logistical concerns are considered and resolved as part of the data collection process.
Presenting Study Results and Findings
Evaluations incorporating user-based data collection techniques typically fall under the heading of "action research"--i.e., research that is intended to assist in the decision-making process or assist in policymaking. Thus, if the evaluation is to be successful, the findings have to be presented to decision makers in such a manner that: (1) the decision makers are aware of the findings, (2) the findings are adequately explained and made understandable, and (3) specific implications and recommendations are made explicit.
INTEGRATING EVALUATION INTO THE PLANNING PROCESS
One of the most important challenges facing the development of successful networked information services is to assess these services in light of a user perspective. One of the best possible strategies for meeting this challenge lies in the development of a comprehensive strategic planning approach that integrates evaluation with strategic planning. Such a comprehensive and integrated approach to network development is essential if evaluators are to provide leadership in accessing, managing, and disseminating networked information to users effectively in the future.
Strategic planning is a disciplined effort to produce fundamental decisions and actions that shape and guide the networked information services being provided currently as well as those being designed. At its best, strategic planning requires broad-scale information gathering, an exploration of alternatives, and an emphasis on the future implications of present decisions (Bryson, 1988). Strategic planning and ongoing evaluation is a critical management process for the development of networked information services if they are to be effective, if designers are to allocate resources wisely, and if user information needs are to be met. More specifically, strategic planning and evaluation:
* provides a rational response to uncertainty and change: We will
never have complete certainty about the future, but we can minimize
some of the risks.
* focuses attention on organizational outcomes: It is essential that
networked information providers identify and assess the services
and products that are placed on the network and how users use
* establishes priorities for resource allocation decisions. There are
inadequate resources to provide all the networked information
services one might like; the most important services for the most
important target audiences must be identified.
* provides a basis for accountability: Providers of networked
information services must be able to justify and be accountable
for what they offer and how well it meets user information needs;
such may be the basis for future funding requests.
* encourages the development of management information systems
to support the planning process: The collection and use of planning
and evaluation data requires that the information is managed
successfully--both analysis and reporting.
* educates providers and users about factors affecting the success
of particular information services: The planning and evaluation
process can serve as an excellent vehicle for staff training and
* informs governing boards and external communities about the
success of the provider: A provider can neither be isolated nor
have its governing board be ignorant of what it does--their
activities must be supported by making certain that its governing
board and clientele know what it does, and that it is doing a good
* forces informational input into the organization from clients and
other stakeholders: Providers of networked information must
receive ongoing information and knowledge about its users and
nonusers as a basis for program development.
* orients the organization to identify opportunities and be future-oriented
rather than responding primarily to daily problems: The
providers of networked information services that will survive and
flourish will be those that identify problems and concerns with
current services and create future strategies to deal with these
concerns; if the provider is unaware of problems with a service,
it is unlikely that those problems will be resolved. The benefits of strategic planning and evaluation contribute to the overall effectiveness and impact of networked information services on clientele. Without a plan, without ongoing evaluation, providers will engage in "crisis management" and be so busy dealing with day-to-day problems that they cannot develop strategies to flourish in the future.
But strategic planning and evaluation have yet to receive the they deserve in the networked information environment. Information providers seem to find reasons not to plan and engage in evaluation rather than to do it, they talk more about planning and evaluation than actually doing the planning process and, frequently, after a plan or an evaluation approach is developed, they do not implement or evaluate it. But strategic planning and evaluation are much more than going through a process that produces a written plan and evaluation results; the process makes networked information services providers think about the success of their current services and then create future plans to support the innovative development of enhanced or innovative information services.
A commentator recently noted that the future was really important, "since I plan on spending the rest of my life there." And indeed, thinking about and creating future network strategies is a primary responsibility for providers of networked information. Strategic planning and evaluation forces us to move beyond the day-to-day responsibilities and the day-to-day crises to address two key questions: (1) how successful are the existing networked information services we provide, and (2) what services should we provide in the future? Strategic planning and a program of regular user-based evaluation of networked information services will be essential to answer these questions.
Amara, R., & Lipinski, A. J. (1983). Business planning for an uncertain future: Scenarios & strategies. Elmsford, NY: Pergamon Press. Babbie, E. (1992). The practice of social research (6th ed.). Belmont, CA: Wadsworth Publishing. Bryson, J. M. (1988). Strategic planning for public and nonprofit organizations. San Francisco, CA: Jossey-Bass. Doty, P.; Bishop, A. P.; & McClure, C. R. (1992). Evaluation of NYSERNET's new connections program. In Networks, telecommunications, and the networked information resource revolution (proceedings of the ASIS 1992 Mid-Year Meeting) (pp. 216-240). Silver Spring, MD: American Society for Information Science. Epstein, I., & Tripodi, T. (1977). Research techniques for program planning, monitoring, and evaluation. New York: Columbia University Press. Hernon, P., & McClure, C. R. (1990). Evaluation and library decision making. Norwood, NJ: Ablex Publishing. Hernon, P., & McClure, C. R. (1986). Unobtrusive testing and quality of reference service. Norwood, NJ: Ablex Publishing. Kirk, J., & Miller, M. L. (1986). Reliability and validity in qualitative research Qualitative Research Methods Series). Beverly Hills, CA: Sage Publications. Krueger, R. A. (1988). Focus groups: A practical guide for applied research. Newbury Park, CA: Sage Publications. LaQuey, T with Ryer, J. C. (1993). The Internet companion: A beginner's guide to global networking. Reading, MA: Addison-Wesley. McClure, C. R. (1992). A user perspective on developing Internet services. Computers in Libraries, 12(4), 53-55. McClure, C. R.; Bishop, A. P.; Doty, P.; & Rosenbaum, H. (1991). The National Research and Education Network (NREN): Research and policy perspectives. Norwood, NJ: Ablex Publishing. McClure, C. R.; Moen, W. E.; & Ryan, J. (1993). Libraries and the Internet/NREN. Perspectives, issues, and challenges. Westport, CT: Meckler Publishing. Marshall, C., & Rossman, G. B. (1989). Designing qualitative research. Newbury Park, CA: Sage Publications. Morgan, D. L. (Ed.). (1993). Successful focus groups. Advancing the state of the art. Newbury Park, CA: Sage Publications. Office of the President. Information Infrastructure Task Force. (1993). The National Information Infrastructure. Agenda for action. Washington, DC: Department of Commerce. Rossi, P. H., & Freeman, H. E. (1993). Evaluation: A systematic approach (5th ed.). Newbury Park, CA: Sage Publications. U.S. Senate. Senate Committee on Commerce, Science, and Transportation. (1991). High-Performance Computing Act of 1991 (P.L. 102-194; Report 102-57). 102nd Congress, 1st session. Washington, DC: USGPO. Van House, N. A.; Weil, B. T.; & McClure, C. R. (1990). Measuring academic library performance: A practical approach. Chicago, IL: American Library Association. Yin, R. K. (1989). Case study research: Design and methods. Newbury Park, CA: Sage Publications. (*) This is a revised version of a paper presented at "Evaluating Library Programs Services," a conference sponsored by the U.S. Department of Education and held at the School of Library and Information Studies, Madison, WI, May 24, 1993.
|Printer friendly Cite/link Email Feedback|
|Title Annotation:||Libraries and the Internet: Education, Practice & Policy|
|Author:||McClure, Charles R.|
|Date:||Mar 22, 1994|
|Previous Article:||The whole cost of libraries.|
|Next Article:||The Internet: educational issues.|