Developing effective electronic information seekers.

Abstract:

As electronic sources of information expand and build depth, there is an increasing reliance on information seeking to support work activities and decision-making. Despite the growing importance of these information sources, little is known about the process of information seeking, and the ways in which skills may be improved. This paper describes a series of studies that seek to identify factors which enhance information seeking, and, in particular, to explore the relationship between motivational influences and information search strategy. The results indicate the need for reviews of organisational training and infrastructure to ensure workers are sufficiently skilled in basic information seeking.

Keywords:

INFORMATION SEEKING; TRAINING; ORGANISATIONAL LEARNING; SELF-EFFICACY; TECHNOLOGICAL INFRASTRUCTURE.

1. Introduction

Research on decision-making is an important area of organisational research. Within this field, there is increasing discussion of how people use sources of information to make informed choices (e.g. Payne, Bettman & Johnson 1993). Allied to this is the growing recognition of the challenges which occur when the problems are ill-defined and/or highly complex. Electronic information seeking is one form of decision-making which is becoming more complex due to the burgeoning range of avenues of information (Sanders 1999). The advent of electronic storage of data, the World Wide Web and intranets has enabled the cultivation of deep, rich sources of information which can be accessed by the information seeker. However, it is important that businesses ensure that workers are knowledgeable, informed and up-to-date in sourcing, capturing and applying information to core business processes (Broadbent, Weill & St Clair 1999; Martiny 1998).

2. Searching for Information

Electronic information seeking can be a highly complex and ill-defined decision process. Information seekers are faced with a wide array of information sources and options which they must access, reject or interrogate during their initial selection of information sources. Cues drawn from these information sources, and from other motivational sources, such as self-regulatory responses, performance feedback and environmental influences, may all lead to changing strategies as the task is continued. This can lead to significant reductions in strategy, as the desire to limit decision parameters overtakes the desire to create a quality decision. As with many other forms of decision-making, the challenges are particularly significant for those who are novice searchers attempting to build good decision strategies and decision outcomes conjointly (Heath & Tindale 1994; Payne et al. 1993; Salterio 1996).

There is an increasing recognition of the complexities of information search (Bystrom & Jarvelin 1995; Marchionini 1995; Wood, George-Falvy & Debowski 2001). This type of information processing reflects all three elements of task complexity, as defined by Wood (1986). First, it requires high levels of component complexity. This relates to the number of acts and information cues which are required to interpret a particular decision task. Second, it is co-ordinatively complex, in that the information seeker must integrate a number of cues, outcomes and acts to process simultaneously the diverse information sources and outcomes. This creates significant processing loads. Third, the process of information seeking can be dynamically complex, due to the constant change in outcomes which occur, as the result of each step taken. In the case of database searching, for example, the simple choice of a different keyword or connector can create an entirely different search outcome, and provide very different cues for the next step of the search process. These three types of complexity lead to interrogative search processes which constantly change the information set. This type of decision-making can be described as `strategy search dominated' in that it does not provide strong cues to assist strategy, relies on cognitive effort to achieve performance outcomes, may comprise a number of different strategies, and provides few means of testing the efficacy of prior search strategies (Earley, Connolly & Ekegren 1989). Electronic search is also made more complex by the expansiveness of its boundaries. Five different searchers may seek answers from similar sources, and arrive at different sets of information to develop their solutions (Collantes 1995). It can be very difficult to verify the completeness of the information set, or the credibility of the sources accessed.

Research on difficulties experienced by electronic information seekers illustrates the reasons for many poor quality searches (Debowski 2001; Kulthau 1999; Marchionini 1995). Problems can relate to the choosing of keywords to describe the search request, creating coherent command structures to input into the system, and in linking the various search elements into a final, cohesive search process. Many searchers are overwhelmed by the sheer magnitude of the outputs which are returned. Others become irritated and frustrated at a significant lack of success (Wood, George-Falvy & Debowski 2001). Unsuccessful searchers are likely to experience decreased self-efficacy and lower commitment toward using the system again, or fail to persevere when poor quality outcomes are obtained. Instead, they may revert to their own, limited, knowledge on which to base decisions.

3. Research on Electronic Information Search

Discussions of the overall search processes people undertake when information seeking have generally failed to analyse the actual iterative process which is followed during the information interrogation phase. For example, many studies focus on the acquisition of procedural, rather than strategic skills, when exploring information systems (e.g. Arthur, Young, Jordan & Shelbiske 1996; Dyck & Smither 1996; Van der Linden, Sonnentag, Frese & Van Dyck 2001). This research neglects the real concerns of how people build skills to search in a complex information environment where results are hard to evaluate and the cues relating to strategy quality are not available. Some studies have investigated the effects of information seeking on performance outcomes (e.g. McJunkin 1995; Saracevic & Kantor 1988). However, there is still little understanding of how people reach these outcomes, and how they could improve their methods and performance. A final limitation of previous research relates to the limited recognition of the impact of affective responses on search quality and outcomes. Thus, there is a strong need for further research on information seeking, particularly in relation to the building of search strategy and related processes.

4. Australian Research on Information Seeking

Information search is gaining increasing attention in Australian research, with work being undertaken on such aspects as search query formulation (Bruza, McArthur & Dennis 2002) and air traffic controller decision processes (Griffin, Neal & Neale 2000). The series of studies reported in this paper extend this understanding by exploring complex electronic information search processes and the ways in which they impact on quality search outcomes. The research aims to explore three issues. First, how information seeking can be operationalised to accurately capture the information processes which are enacted. Second, the types of training which can assist learners to gain mastery of these critical skills. Third, other forms of support which can assist in building competent and confident information seekers. Each of these research issues is discussed in turn. The research was undertaken in laboratory settings, with each experimental group comprising 12-20 participants. The volunteers were university students who had little or no prior experience with the search process. The Silverplatter ERIC database was employed for the reported studies as the search source. This database offers a complex information environment, with provision for thesaurus and index searching and term explosion, in addition to standard search strategy building. It also allows for the nesting of search commands, and the use of brackets and Boolean connectors. As such, it reflects the complexity of database structures commonly found in complex business systems.

5. Measuring Information Search Behaviours

A key element of the studies has been the development of a number of measures to analyse information search behaviours. Three main measures have been tested and refined through the studies, to reflect the quantity and quality of search activity which is undertaken during the interrogative phase. A measure of task-focused effort is based on the time spent on the activity and the numbers of inputs developed in the search. A second effort measure, wasted effort, examines the activities which impede the completion of the task. Factors such as repetitive or redundant lines of search, the different sets of enquiries which were generated, and errors, contribute to this assessment. While these activities may sometimes prove fruitful and informative, they may also distract attention from the achievement of the final information set, and can serve to create much greater search complexity for novice searchers. Search quality is a composite measure of search strategy comprising elements such as search breadth, depth and sequence. Search breadth relates to the number of concepts incorporated, search depth explores the number of different terms used to describe each concept, and search sequence reflects the number of sequential steps entered in a single series. New measures of search self-efficacy and satisfaction were also devised. The studies also incorporated a range of established measures such as search performance, intrinsic motivation and mood. A range of background measures were taken in each study, including gender, age, years of university study, prior computer experience and confidence, and information search experience and background skills.

6. Training Electronic Searchers

It has been claimed that exploratory learning is a more conducive learning environment (e.g. Dorman & Frese 1994) than the more structured guided mastery programmes advocated by Bandura (1986, 1997), Gist (1989) and others. Exploratory learning is perceived to be a self-managing learning tool which enables the building of a cognitive framework based on personal experience. Guided mastery, on the other hand, offers opportunities for enactive mastery. The first form of training builds on trial and error as a tool for learning, while the second encourages the building of effective strategies through structured practice. The first study tested the effectiveness of these two forms of learning when applied to a highly complex information search context (Debowski, Wood & Bandura 2001). University students were trained using a guided exploration or an enactive exploration learning mode. During the instructional phase of the training, all participants were provided with the same content and instructional materials. Both groups were then provided with a structured set of practice tasks, ranging from simple, two concept tasks to more complex, three concept tasks. However, enactive exploration participants were not required to adhere to this practice programme. Instead, they were encouraged to explore the information environment, using the tasks as a framework. Posters exhorted these participants to see errors and wrong strategies as cues to learning. Guided exploration participants were required to follow a regulated process in their practice component, using the practice tasks as a progressive set of activities. No feedback was provided to any participant. At the conclusion of the training and practice segment, participants completed two search tasks--one moderately complex, two-concept task, which was similar to the practice tasks, and the other a highly complex, four-concept task which required significantly stronger application of the training model to successfully complete the search. The guided exploration programme led to significantly higher self-efficacy and satisfaction, lower wasted effort, higher search quality and better performance. The study demonstrated that initial learning of new complex strategies was more effectively managed through the provision of a structured guided mastery approach. The complexity of the search context may be the reason for this outcome, since the expansive range of cues, outcomes and activities, and the limited capacity to interpret quality and performance, made the exploratory learning process less informative at this particular stage of complex skill acquisition.

The second study (Wood, Kakabeeke, Debowski & Frese 2000) tested the effect of enactive exploration on intrinsic motivation, strategy and performance following initial mastery of the basic search skills. Participants were first trained to search using the same guided mastery approach. Following this, they were provided with practice opportunities which incorporated an enactive exploration segment, or no further encouragement. Participants then completed two highly challenging transfer tasks. The results showed that the combination of guided mastery, followed by enactive exploration led to higher levels of intrinsic motivation and better retrieval of records. However, there was no impact on the quality of search, with all participants demonstrating improved strategies as they worked on the two transfer tasks.

7. Other Forms of Support for Information Seeking: The Role of Feedback

Participants in these initial studies were not provided with feedback other than task-related information they personally derived while undertaking the tasks. Both studies ignored the role of feedback as a performance enhancer. Feedback is an important source of guidance in complex decision environments, since it can help to direct and (re)focus attention to the decision context. This is particularly important in complex information environments where naturally occurring feedback is not available. External feedback can provide this guidance on standards to be achieved, the impact of the strategies on performance, and it can also enhance self-efficacy (Goodman 1998; Cervone, Jiwani & Wood 1991).

These external feedback effects were explored within a complex decision environment in two further experiments. The first feedback study tested the effects of comparative feedback on search activity and performance. As with other comparative feedback studies (e.g. Krueger & Dickson 1994), a standard for performance and knowledge of results was provided to enable the monitoring of future performance. Participants were provided with either positive, negative or no comparative feedback at two points: after the first transfer task, and a week later, prior to the final, third, transfer task. This comparative feedback provided a numerical comparison with other participants on their retrieval outcomes, with those in the positive condition told their scores were 42% better than others, and the negative condition participants told they were 42% worse than other participants. At the conclusion of the study, those receiving negative feedback were debriefed and provided with accurate feedback. These feedback cues tend to be motivational, rather than informational, leading to improved or reduced effort and perseverance, depending on the feedback sign. The results showed that the comparative performance-related feedback influenced self-efficacy and task-focused effort. While initial negative feedback resulted in high levels of effort, this was not transferred across to the final delayed task which occurred a week later. Positive feedback was more resilient in sustaining effort to the next week. High self-efficacy individuals were spurred to work longer on the tasks if they were provided with either form of feedback. This first feedback study illustrated the effects of comparative feedback on volitional effort, but failed to influence the participant's search activities in any tangible manner. This could be attributed to the focus on performance, rather than process, which limited the value of the information for improving the search strategy. A follow up study which provides comparative feedback on process skills is in progress.

The second feedback study (Wood, Debowski & Goodman, unpublished) sought to test the value of process-related feedback in a complex strategy context. This study extended previous research on this form of feedback provision (e.g. Benson & Onkal 1992; Schunk & Rice 1991). In this design, feedback related to the strategies participants had demonstrated during the baseline task. Participants in one group received strategy validity information which provided a detailed evaluation of the strategies used by the participant. The second group received the same feedback, but with an additional element of task information, in the form of a model of an effective search. The final group received no feedback on the baseline task. Feedback which supplied strategy validity information as well as task information led to significantly lower levels of wasted effort on non-productive search, and better search quality compared with the control group. Both feedback interventions boosted self-efficacy, with their effects increasing as experience with the task grew. The combined task and strategy validity feedback most strongly enhanced search strategy, leading to a significantly better emphasis on building good information searching compared with the other two groups. However, these skills were still not sufficiently developed to achieve higher levels of final performance.

These four studies raise a number of issues for further consideration. They demonstrate the complex nature of electronic information search, and the difficulties searchers face in undertaking effective complex searching without effective training and follow up. Studies 1 and 2 demonstrated the value of guided exploration, followed by an enactive exploration phase in creating quality searches which also build self-efficacy and perseverance. Study 4 illustrates the value of including both strategy validity information and task information to those who are still continuing to build their knowledge and understanding of these search processes. It also illustrated the value of providing models and continued guidance during the skill development phase, after initial instruction. The relationship between search strategy and motivational processes still requires further investigation, as these studies serve to illustrate the complexity of the interactions whilst working through complex decisions. All four studies demonstrated the significant challenges associated with creating effective performance in these searchers. Despite quite extended training sessions in some cases, participants retrieved low numbers of records on their topics. The studies highlight the challenge of ensuring those searching electronic information sources are sufficiently skilled to gain the information they require. Answers to these questions could help to clarify the way in which we learn and develop complex information seeking skills.

8. Conclusion

There are a number of challenges which also arise from this research to date. First, the participants in these four studies were university students. The studies are being continued with members of the wider community, and with older age groups to identify other aspects of search which need to be considered. This is increasingly important as the age of retirement continues to rise, leading to an ageing workforce with limited technological skills, but escalating information needs (Bates, Holton & Seyler 1996). Second, the work has been extended into the realm of Internet search, where the retrieval capture is even less predictable. This is creating some challenging issues relating to the provision of feedback and task information to learners. Third, the length of training and the testing of skill consolidation has been extended to explore longer term impacts of these instructional and feedback modes.

The research also identifies some important issues for organisations. First, organisational audits of skill deficits should include a review of information search competencies (see Hesketh & Ivancic 1999). An evaluation of the existing organisational training and learning mechanisms could also indicate that little support is offered to information users (Dobbs 2000). A new service which provides guided exploration training, and enactive exploration follow-up to support the emergent searcher could be a valuable addition to the organisational learning framework. Second, the provision of expert support to provide further task information and strategic validity information could escalate skill development (e.g. Keefer & Karabenick 1998). Third, there is a need to consider how electronic support services can support information seekers in electronic work environments (Debowski 2000). With the increasing focus on employee knowledge as an organisational priority, the basic interrogation and search expertise of the individual will become a major strategic issue, and needs to be considered in terms of training, support, encouragement and quality outcomes.

References

Arthur, W., Young, B., Jordan, J.A. & Shebilske, W.L. 1996, `Effectiveness of individual and dyadic training protocols: The influence of trainee interaction anxiety', Human Factors, vol. 38, no. 1, pp. 79-86.

Bandura, A. 1986, Social Foundations of Thought and Action: A Social Cognitive Theory, Prentice-Hall, Englewood Cliffs, New Jersey.

Bandura, A. 1997, Self-Efficacy: The Exercise of Control, Freeman, New York.

Bates, R.A., Holton, E.F. & Seyler, D.L. 1996, `Principles of CBI design and the adult learner: The need for further research', Performance Improvement Quarterly, vol. 9, no. 2, pp. 3-24.

Benson P.G. & Onkal, D. 1992, `The effects of feedback and training on the performance of probability forecasters', International Journal of Forecasting, vol. 8, no. 4, pp. 559-73.

Broadbent, M., Weill, P. & St Clair, D. 1999, `The implications of information technology infrastructure for business process redesign', MIS Quarterly, vol. 23, no. 2, pp. 159-81.

Bruza, P., McArthur, R. & Dennis, S. 2002, `Interactive Internet search: keyword, directory and query reformulation mechanisms compared', http:/humanfactors.uq.edu.au/research/prism.pdf

Bystrom, K. & Jarvelin, K. 1995, `Task complexity affects information seeking and use', Information Processing and Management, vol. 31, no. 2, pp. 191-213.

Cervone, D. Jiwani, N. & Wood, R. 1991, `Goal setting and the differential influence of self-regulatory processes on complex decision-making performance', Journal of Personality and Social Psychology, vol. 61, no. 2, pp. 257-66.

Collantes, L.Y. 1995, `Degree of agreement in naming objects and concepts for information retrieval', Journal of the American Society for Information Science, vol. 46, no. 2, pp. 116-32.

Debowski, S. 2000, `The hidden user: Providing an effective service to users of electronic information sources', OCLC Systems and Services, vol. 16, no. 4, pp. 175-80.

Debowski, S. 2001, `Wrong Way, Go Back! An Exploration of novice search behaviours while conducting an information search', The Electronic Library, vol. 19, no. 6, pp. 371-82.

Debowski, S., Wood, R.E. & Bandura, A. 2001, `The impact of guided exploration and enactive exploration on self-regulatory mechanisms and information acquisition through electronic enquiry', Journal of Applied Psychology, vol. 86, no. 6. pp. 1129-41.

Dobbs, K. 2000, `Who's in charge of e-learning?', Training, June, pp. 55-8.

Dorman, T. & Frese, M. 1994, `Error training: Replication and the function of exploratory behaviour', International Journal of Human and Computer Interaction, vol. 6, pp. 365-72.

Dyck, J.L. & Smither, J.A.A. 1996, `Older adults' acquisition of word processing: The contribution of cognitive abilities and computer anxiety', Computers in Human Behavior, vol. 12, no. 1, pp. 107-19.

Earley, P.C., Connolly, T. & Ekegren, G. 1989, `Goals, strategy development and task performance: some limits on the efficacy of goal setting', Journal of Applied Psychology, vol. 74, no. 1, pp. 24-33.

Gist, M.E. 1989, `The influence of training method on self-efficacy and idea generation among managers', Personnel Psychology, vol. 42, no. 4, pp. 787-805.

Goodman, J. 1998, `The interactive effects of task and external feedback on practice performance and learning', Organisational Behavior and Human Decision Processes, vol. 76, pp. 223-52.

Griffin, M.A., Neal, A. & Neale, M. 2000, `The contribution of task performance and contextual performance to effectiveness: Investigating the role of situational constraints', Applied Psychology: An International Review, vol. 49, no. 3, pp. 517-33.

Heath, L. & Tindale, R.S. 1994, `Heuristics and biases in applied settings; an introduction' in Applications of Heuristics and Biases to Social Issues, ed. L. Heath, Plenum Press, New York, pp. 1-12.

Hesketh, B. & Ivancic, K. 1999, `Training for transfer: New directions from industrial/organisational and cognitive psychology', Training and Performance, ed. J. Langan-Fox, Australian Psychological Society, Melbourne.

Keefer, J.A. & Karabenick, S.A. 1998, `Help seeking in the information age', in Strategic Help Seeking: Implications for Learning and Teaching, ed. S. A. Karabenick, Lawrence Erlbaum Associates, Mahwah, NJ, pp. 219-50.

Krueger, N. & Dickson, P.R. 1994, `How believing in ourselves increases risk taking: Perceived self-efficacy and opportunity recognition', Decision Sciences, vol. 25, no. 3, pp. 385-400.

Kulthau, C. 1999, `The role of experience in the information search process of an early career information worker: Perceptions of uncertainty, complexity, construction and sources', Journal of the American Society for Information Science, vol. 50, no. 5, pp. 399-412.

Marchionini, G. 1995, Information Seeking in Electronic Environments, Cambridge University Press, Cambridge.

Martiny, M. 1998, `Knowledge management at HP consulting', Organizational Dynamics, vol. 27, no. 2, pp. 71-7.

McJunkin, M.C. 1995, `Precision and recall in title keyword searches', Information Technology and Libraries, September, pp. 161-71.

Payne, J.W., Bettman, J.R. & Johnson, E.J. 1993, The Adaptive Decision Maker, Cambridge, Cambridge University Press.

Salterio, S. 1996, `Decision support and information search in a complex environment: Evidence from archival data in auditing', Human Factors, vol. 38, no. 3, pp. 495-505.

Sanders, R.L. 1999, `Searching for relevance in the virtual files of the future', Information Management Journal, vol. 33, no. 2, pp. 64-72.

Saracevic, T. & Kantor, P. 1988, `A study of information seeking and retrieving: Users, questions and effectiveness', Journal of the American Society for Information Science, vol. 39, no. 3, pp. 177-96.

Schunk, D.H. & Rice, J.M. 1991, `Learning goals and process feedback during reading comprehension instruction', Journal of Reading Behavior, vol. 23, no. 3, pp. 351-64.

Van der Linden, D., Sonnentag, S., Frese, M. & Van Dyck, C. 2001, `Exploration strategies, performance, and error consequences when learning a complex computer task', Behaviour and Information Technology, vol. 20, no. 3, pp. 189-98.

Wood, R.E. 1986, `Task complexity: Definition of the construct', Organizational Behavior and Human Decision Processes, vol. 37, pp. 60-82.

Wood, R.E., Debowski, S. & Goodman, J. unpublished, `Feedback effects on electronic search task', (manuscript submitted for publication).

Wood, R.E., George-Falvy, J. & Debowski, S. 2001, `Motivation and information search on complex tasks', in Work Motivation in the Context of a Globalising Economy, ed. M. Erez, U. Klienbeck, & H. Thierry, Lawrence Erlbaum Associates, Mahwah, NJ. pp.27-48.

Wood, R.E., Kakabeeke, B.M., Debowski, S. & Frese, M. 2000, `The impact of enactive exploration on intrinsic motivation, strategy and performance in electronic search', Applied Psychology: An International Review, vol. 49, no. 2, pp. 263-83.

(Date of receipt of final transcript: December, 2001. Accepted by Sharon

Parker and Robert Wood, Special Issue Editors.)

Shelda Debowski, Murdoch Business School, Murdoch University, South St, Murdoch, WA 6150. Email: [e-mail]

Shelda Debowski is a Senior Lecturer at the Murdoch University Business School where she teaches in the areas of human resource management and training and development. Her PhD from the University of Western Australia explored the issues relating to social cognitive, training and feedback effects on information seeking. This research has been extended over the subsequent years across a more diverse learner population, and into the Internet decision environment. Shelda also researches in the area of training technologies and knowledge management.