Printer Friendly

Study of effectiveness of implicit indicators and their optimal combination for accurate inference of users interests.

ABSTRACT: Retrieval and filtering systems may apply relevance feedback to gain information on users' needs in order to improve their ad-hoc queries or long term profiling. Explicit relevance involves explicit ratings of documents or terms by the users and disrupts their normal patterns of browsing and searching. The alternative non-disruptive method is implicit feedback inferring users' needs and interests by monitoring their regular interaction with the system. Some implicit indicators of interest, such as reading time, have been investigated in previous studies and were found indicative to the relevance of documents but not sufficiently accurate to substitute explicit ratings. In this paper we present several new relative implicit feedback indicators and examine their effectiveness as well as the effect of combining several implicit indicators. The paper describes a large-scale user study on which users' searches were observed by a specially developed browser that recorded their behavior (implicit indicators) as well as their explicit ratings. The relationship between implicit indicators and explicit ratings was analyzed and found that a certain combination of implicit indicators achieved higher correlation with the explicit ratings than any of the individual indicators. We have also found that the newly suggested relative indicators are more indicative to the level of interest of the user in an information item than the non-relative indicators.

Categories and Subject Descriptors

H.3.3 [Information Search and Retrieval]: Relevance feedback; H.3.4 [Systems and Software]: User profile and alert services

General Terms

User profiling, Query formulation

Keywords: Relevance feedback, Implicit indicators, User studies, User Modeling

1. Introduction

Relevance feedback (RF) is typically used to acquire information about users for obtaining their accurate information needs. RF is used for building or updating user models in information filtering or recommender systems, and for expanding or modifying ad-hoc queries (Salton and Buckley, 1990). Explicit relevance feedback requires the user to explicitly provide feedback by marking or rating documents and terms for their relevancy. One main problem with this traditional feedback technique is the disruption to the regular work of users. To provide explicit ratings users are required to change their normal browsing and searching patterns and to perform additional activities (e.g., a few more mouse clicks). Since the benefit of providing feedback is not easily perceived, many users tend not to provide evaluations, thus resulting in lower effectiveness of the systems that rely on these evaluations (Shapira et. al., 2001).

Implicit relevance feedback techniques obtain information on users by monitoring their natural interaction with the system without interruption. Every interaction is recorded to infer users' interests and preferences. Analysis of users' behavior consists of examination of relevance indicators such as reading and scrolling time and activities such as printing, bookmarking, etc.

Despite the clear advantage of implicit feedback due to the removal of cost to the user, implicit feedback is still known to be less accurate than explicit rating in predicting users' interests (Nichols 1997). Although there is a growing body of studies related to implicit feedback, the optimal set of implicit indicators and their relative importance for an accurate derivation of a user profile has not yet been determined (Kelly and Teevan, 2003, White et. al., 2002, Clyapool et. al., 2001 and others). From the vast literature on implicit feedback one can conclude that not all implicit measures are equally effective and it might very well be that some are effective only when combined with others (Kelly and Teevan, 2003). In this paper we investigate the effectiveness of implicit indicators. We describe a user study that examine the usefulness of known and newly suggested implicit indicators and determine the optimal combination of these indicators for an accurate inference of users' interest.

We first review related research in section 2. Section 3 describes the goals of this the experiment designed to achieve them. Section 4 describes the results while section 5 is a discussion of the results and future research issues.

2. Related Research

Implicit feedback techniques observe user behavior for short or long term inference of user interests and preferences. Oard and kim (1998) presented a classification of observable implicit indicators according to their behavior categories: Examination, Retention and Reference (See Table 1). Examination refers to behaviors indicating that the user is examining an object (such as reading time), retention refers to behaviors indicating intention of future use of the object (such as bookmarking), and reference is a type of behavior intended to link the relevant object to other objects. In a later paper Oard and Kim (2001) categorized the implicit indicators using two axes: behavior and minimum scope, where minimum scope (Segment, Object, Class) refers to the minimal scope of the item for which a certain behavior might be acted upon. In a recent survey of implicit feedback related studies Kelly and Teevan (2003) enhanced the categories to include a new "create" behavior. In the current research we have used the original categorization of behaviors as presented in Oard and Kim (1998).

Table 2 summarizes the main studies related to the effect of implicit indicators and their findings. We present on that table the indicators examined (within their category), the context of the study (task, type of collection etc.), the findings, and specific comments.

It is notable from Table 2 that reading time was found correlative to explicit ratings in most studies but the duration of reading as a positive indicator varies between studies, and might depend on the task, thus, although useful, the reading time indicator cannot serve as a single indicator. Printing was found to be indicative when it was observed, however, users do not print very often so it is hard to infer meaningful conclusions about the interests of users only from data about their printing activities. There is no agreed conclusion regarding scrolling. All other examined implicit indicators were not indicative when applied as single indicators for interest in a page. Regarding combination of implicit indicators, the only study referring to the usefulness of combining indicators was conducted by Claypool et al., (2001), who examined the effect of combining reading and scrolling time and found that it was more effective than any of the indicators alone. In the current study we examine the combination of all indicators in order to determine the optimal indicators to combine.

3. Experiment

3.1 Goals of the study

The current research has three main goals:

1) To re-examine known implicit indicators due contradicting results on previous studies.

2) To suggest and examine new relative indicators.

3) To investigate the effect of combining implicit indicators on the accuracy of the prediction of users' needs.

Since former studies revealed that not all implicit indicators are equally useful, it might well be that some will only be useful when combined or related to others. In addition to an examination of useful combinations of indicators, we suggest relative indicators, i.e., indicators normalized by other factors. Our intuition was that the interest of user in a page might be better predicted when the indicators are normalized by properties of the data (such as their length) or by other implicit indicators. For example, reading time normalized by the length of the page being read might indicate better the interest of the user assuming that longer pages naturally require a longer time to read. Another example is the relation between reading time and mouse movements. Assuming that mouse movement on a page indicates active interaction, it is possible to assume that mouse movement during most of the reading time indicates higher interest in a page than where only a few mouse movements are observed. The user might have remained on the page for a long time inactive because of a coffee-break or phone call.

The new suggested (single and relative) indicators are:

1) Time of mouse movement relative to reading time; (relative indicator)

2) Scrolling time relative to reading time; (relative indicator)

3) Reading time normalized by page size; (relative indicator)

4) Number of links visited on a page. We predicted that this indicator might be helpful for navigation, i.e., will correlate with the explicit rating of "helpful for navigation"--rating 3. (single indicator)

5) Number of links visited on a page relative to the number of existing links on the page; (relative indicator)

6) Level of interaction on a page where interaction is considered one of the following user's actions: Print, bookmark, copy & paste, mouse movement, scroll, visit links. The maximal interaction level is six, indicating that the user performed all these operations, and the minimal is zero, if no operations were performed. The assumption is that the more interaction observed, the level of relevancy of the page for the user is more significant (single indicator).

In addition, we have re-examined the following known single indicators:

Examination: Reading time, mouse movement, and scrolling.

Retention: bookmark, print, copy & paste.

3.2 Experiment

An experiment was conducted to examine the effect of the single, relative and combination of implicit indicators compared to explicit ratings. During the experiment, a specially developed browser was installed for four months in an Internet station (a PC dedicated for Internet searches) in a large high-tech firm. The firm produces confidential military products. For security reasons the personal stations of employees are not connected to the Internet. In order to search the Internet the employees need to use the dedicated Internet station, and to log on when using the station. Twenty five computer engineers, workers of the company, participated in the experiment and were asked to use the browser when searching the Web for their occasional professional needs. The age of the majority of the participants (23 of 25) ranged between 30-45 years. They all had at least five years experience of working with computers. These workers performed their regular searches, only that they used our browser that had similar functionality to the regular browser on the station. Their behavior on every page they accessed was recorded by the browser. The participants were also asked to explicitly rate the relevancy of each page (before leaving it) which was a minimal disruption to their regular work, but necessary for the experiment.

The workers were introduced to the browser and were given explanation regarding the experiment in a special seminar. The rating ranged from 0-5 according to the following key:

0- No opinion--Sometimes the user may not be very sure of his or her answer regarding the relevancy of a specific web page. In this case he or she could choose: 0- No opinion.

1- Not interesting--The page is not relevant to the participant's searching needs.

2- Somewhat interesting--The user found information on the page which is in the general domain of the search but is not related directly to his or her specific search.

3- Helps navigation--pages with non-interesting content that contains useful links assisting the user to reach interesting pages. 4- Interesting--The participant found on the page most of the information he or she has searched for but not the complete information.

5- Very interesting--The user has found the page which contains exactly the information he or she needed. No further search is required.

Figure 1 presents the browser interface with the explicit rating window.


The following is the list of users' behaviors (implicit indicators) gathered by the browser for each user on each access to a page:

* URL of the accessed page--gathered, not analyzed.

* User ID--the workers are regularly required to login in order to use the Internet station.

* Current time--gathered, but not analyzed

* Time spent on page--measured in seconds.

* Number of links on the page.

* Number of visited links.

* Total text size--measured in bytes (the non-textual content was filtered).

* Total time of mouse movement on a page--measured in milliseconds.

* Total scroll time--vertical and horizontal scrolling--measured in milliseconds.

* Print--y/n--indication of any print operations performed on the page.

* Bookmark--y/n--indication of addition of the page to the bookmark list.

* Copy/paste--y/n--indication of any copy/paste operations performed on the page.

In addition the users were asked to explicitly rate the relevancy of pages they visited (on a 0-5 scale as detailed above). 1758 pages were collected from which 198 pages were removed since the participants had no opinion about their relevancy (rated them as 0). Six pages were identified as outliers and removed since their reading time was higher than 500 seconds (while the maximal reading time of 90% of the pages was 48 seconds). Thus, 1554 pages were analyzed. The graph on figure 2 presents the rating distribution where it is notable that users mostly rated pages as very interesting (5) or not interesting (1).


3.3 Evaluation Methods and Measures

For every implicit indicator we examined whether it correlates with explicit ratings. Our prediction was that there should be significant differences in the values of implicit indicators among the explicit rating groups (1-5). Once our prediction was verified, for each indicator examined, we used descriptive statistics to study the trend. The common statistical test examining whether there exists a significant difference between at least one of the groups means is ANOVA. However, ANOVA assumes normal distribution of the dependent variables, equality of variances (homoscedacity), and independent observations. In cases of violation of these assumptions it is possible to apply a-parametric tests, such as Kruskal-Wallis. We found that the assumption of equality of variances was violated. Since most indicators had a heavy-tailed distribution we also applied the Kruskal-Wallis test for each implicit indicator using the group medians rather than the means, since means are known to be affected by outliers. For each implicit interest indicator we performed a Kruskal-Wallis test on the implicit indicator as the dependent variable and the explicit rating as the independent variable. Only for those variables where the Kruskal-Wallis test rejected the null hypothesis (that the median values are the same), we performed a post-hoc multiple comparison using the Dunnet test (Montgomery, 2001) to evaluate the degree of difference (variation) between the groups. We later use the results of this test to rank the implicit indicators by their effectiveness. For these significant implicit indicators we also present box-plots to illustrate visually the relation between the implicit indicator and the explicit rating.

The effectiveness of combining implicit indicators in predicting users' interests was tested using stepwise linear regression. Regression Analysis is a statistical technique for modeling and investigating the relationship between one dependent variable to one or more independent variables (e.g., between an implicit indicator and explicit rating) (Montgomery and Runger, 1999). Stepwise regression is a technique for screening candidate variables to obtain a regression model that contains the "best" subset of regressor variables. In our experiment we are interested in choosing the variables, i.e., the set of "best" implicit indicators, to be included in the regression model, i.e., the set of implicit indicators that are most significantly related to the explicit rating.

4. Results

In this section we present only results for indicators found to be significant. Results are shown in graphs on which the upper dashed line shows the average implicit indicator. In addition we present a box-plot where the boxes show the range of values from the bottom quartile (25%) to the top quartile (75%) and the lines (in the boxes) show the medians of the implicit indicator.

Results for Single Non-Relative Indicators:

Reading Time (time spent on page):

Our results confirmed previous results showing that reading time is an adequate but not very accurate indicator for user's interest in a page. The Kruskal-Wallis test rejected the null hypothesis (p<0.001) signifying that the median implicit values for at least one of the explicit rating groups was different. The box-plot of the time spent on a page versus explicit rating (presented in figure 3) shows that there is a substantial increase in median for the reading time toward the explicit rating groups (positive linear trend). Applying the Dunnet post-hoc test we found significant differences only between 50% of the groups stemming mainly from the differences between the non sequential groups. For example, the differences in reading time between the explicit rating groups: e.g., 1 and 3, 2 and 5, 3 and 4, were significantly different. It can therefore be concluded that reading time might differentiate only between three levels of users' interest in a page: not interesting, partially interesting, and very interesting.

Time Moving Mouse:

The results demonstrate that the time spent moving the mouse is a good indicator for the level of interest in a page. The Kruskal-Wallis test rejected the null hypothesis (p<0.001) indicating that the median values for at least one of the explicit rating groups was different. Observing figure 4 showing the box-plot of the time moving mouse versus explicit rating, it can be seen that there is a positive linear trend of the time moving mouse towards the explicit rating. The Dunnet post-hoc test revealed that there are significant differences between 70% of the groups. The differences were mainly between the extreme groups to the other groups (1 to the other groups, and 5 to the other groups). The time moving mouse seems to be an even better indicator than the time spent on page (reading time).

The following three indicators, namely Scrolling time, Number of visited links on a page, and the level of interaction on a page revealed the same pattern of results. The Kruskal Wallis test for each of these indicators rejected the null hypothesis, i.e., the median implicit values for at least one of the explicit rating groups were significantly different from the others. A Dunnet post-hoc test revealed only two major levels of significantly different rating groups, i.e. these indicators should be considered only as binary indicators of interest in a page (interesting, not interesting). Due to space limitations not all the graphs are shown.

For the retention category indicators, namely, Print, Bookmark, and Copy&paste, few data were collected as users do not often perform these activities. Actually, for the copy&paste indicator no data at all was collected.

Out of the 1554 rated pages only eight were bookmarked and 46 pages were printed. The printed and bookmarked pages were rated 4 or 5 (interesting and very interesting), where the ratings of the non-bookmarked and non-printed pages were uniformly distributed among the rating groups. We therefore conclude that print and bookmark are indicators of interesting and very interesting pages.

Results for Single Relative indicators:

The only single relative indicator found not significant when applying the Kruskal Wallis test was the relative scrolling. The following are the results for the other single relative indicators.

Time Moving Mouse Normalized by Reading Time

The results presented in figure 5 demonstrate that moving mouse normalized by reading time is the best of the examined indicators. After the null hypothesis was rejected by the Kruskal-Wallis test (p<0.001), the Dunnet post-hoc test showed significant differences between 80% of the groups. From the Dunnet test as shown in figure 5 four different levels of rating were identified (1, 2, 3-4, 5).

Reading Time Normalized by Page Size:

We predicted that this indicator would be more indicative than absolute reading time. However, although the Kruskal Wallis test showed significant difference between at least one of the rating groups, the results of the Dunnet post-hoc revealed that this indicator has no advantage over absolute reading time with respect to the percentage of difference between rating groups.

Number of Visited Links on a Page Relative to the Number of Links on a Page

We predicted that this indicator would be indicative to the navigational interest in a page, expressed in the explicit rating 3, since users tend to follow many links on navigational pages. The Kruskal Wallis test showed significant difference between at least one of the groups, and a post-hoc Dunnet test resulted with a significant difference between the rating group 3 to the other rating groups. There was no significant difference within the other groups. This proved our hypothesis.

Combination of Implicit Indicators

We were interested in examining whether a combination of indicators is more effective in predicting users' interest in a page than any of the single indicators. We compared the correlation between each of the single indicators to the explicit rating with the correlation between the combined implicit indicators and the explicit ratings. To select the best indicators for the combination we applied stepwise regression which selects only the variables whose contribution to the model is significant. These variables were the reading time, relative mouse movement, print, and the number of visited links indicators. It should be noted that the stepwise regression selected only one of the single and the corresponding relative indicators. For example, mouse movement was not selected while the mouse movement normalized by reading time was selected. This can be explained by the dependency between the relative and its corresponding single indicators. The inclusion of two mutually dependent indicators usually results in an insignificant marginal contribution by one of them to the model. Table 3 presents the correlation coefficient r for each of the single implicit indicators that showed significant positive correlation with the explicit rating. It can be seen that although the correlation was significant, the correlation coefficients were low. In addition, we show the correlation coefficient obtained for the combination of indicators selected according to the stepwise regression results. This correlation coefficient was significantly higher than any of the correlation coefficients of the single indicators.

5. Discussion and Future Research Issues

In this work we investigated the effectiveness of implicit indicators resulting in the following findings:

1) For the single non-relative indicators we were able to prove positive correlation to implicit indicators. However, these indicators are not effective in distinguishing the level of interest in a page. Even mouse moving and reading time which were found to be the best of this group of indicators are not able to distinguish between all levels of users' interest in a page which is necessary for accurate derivation of users' interest. One reason for the inaccuracy of these indicators is that they do not differentiate between the time spent on a page when not interacting (such as coffee break times) and the active interaction time that is more indicative to the interest on a page. This problem is handled by the relative indicators.

The other (new) single indicators, namely, the number of visited links and the level of interaction in a page were found to be binary indicators, i.e., they may distinguish only between interesting and not interesting pages. As for printing and bookmarking, they are indicative only for interesting pages, but users do not perform these operations very often. To sum up, the single relative indicators are effective in giving some indication regarding the level of user interest in a page, but are not sufficiently accurate. These findings confirm previous studies results regarding the questionable usefulness of some indicators and the reasonable quality of reading time, while they add evidence regarding the usefulness of the newly suggested indicators.

2) For the relative indicators we found that the relative mouse movement (i.e., mouse movement relative to reading time) Indicator Correlation coefficient(r) is the best single indicator. It is more accurate than reading time or mouse movement. This indicator is able to distinguish accurately between all levels of interest and is the recommended indicator if no combination of indicators is applied.

Reading time relative to page size was not found to be more effective than absolute reading time. We believe that this is because users notice immediately if they are interested in a page or not, so for the no interest in a page indication, the length of the page does not change the time. An explanation for the lack of indication of interest in a page might be that when users are interested in a large sized page they read only the sections of interest to them. This issue requires further investigation. Another finding of this research is the ability of the new indicator "number of visited links relative to the number of links on a page" to accurately indicate a navigational page. The other relative indicators were not significantly better than the corresponding non-relative indicators. We believe that these findings point to the potential of the relative indicators to significantly improve the accuracy of implicit indicators. There exist other relations within users' behaviors and between behaviors and attributes of the page being viewed that should be defined and examined in a continuing study, such as reading time relative to the percentage of text or images on a page, etc.

3) Combination. The best correlation with explicit indicators was obtained when some indicators were combined. The optimal selection of the indicators to be included in the combination was performed by applying stepwise regression. These positive results regarding the ability to predict users' interest with implicit indicators are encouraging in respect to the future development of collaborative and other user-based systems that suffer from lack of provision of evaluations by users that do not like to disrupt their regular work in order to explicitly rate items (Avery and Zeckhauser, 1997).

However, although relative and combined indicators significantly improved the accuracy of the prediction obtained by using implicit indicators, the highest correlation coefficient with explicit rating was not high. One explanation is the distribution of the data within the rating groups and many outliers that are known to affect linear correlation (Montgomery 2001).

Another explanation relates to the results of some studies claiming that explicit indicators do not always demonstrate accurately user's interest (Shapira, et. al., 2005). There is a difference between the perceived and the real value of users' feedback. A natural continuing study would be a to measure users' or systems' performance in a personalized system when various implicit indicators are used to examine their real effect, not only as compared to explicit ratings.

Received 8 Mar. 2006; Revised 20 June 2006; Accepted 30 June 2006


Avery, C., Zeckhauser, R (1997). Recommender systems for evaluating computer messages, Communications of the ACM, 40 (7) 88-89.

Claypool, M., Brown, D., Le, P., Waseda, M (2001). Inferring user interest . In: Proceedings of the 6th Intl. Conference on Intelligent User Interfaces (IUI 01), USA, p. 33-40.

Kelly, D., Teevan, J (2003). Implicit feedback for inferring user preferences, SIGIR Forum 27 (2) 18-28.

Kim, J., Oard, D.W., Romanik, K (2000). Using implicit feedback for user modeling in internet and intranet searching. University of Maryland CLIS Technical Report 00-01.

Konstan, J., Miller, B. N., Maltz, D., Herlocker, J.L., Gordon, L.R., Riedl, J (1997). Grouplens: Applying collaborative filtering to usenet news. Communications of the ACM , 40(7), p. 77-87.

Montgomery, D.C., Runger, G.C (1999). Applied Statistics and Probability for Engineers. John Wiley & Sons, Inc. ed.

Morita, M., Shinoda, Y (1994). Information filtering based on user behavior analysis and best match text retrieval. In: Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '94), Ireland, p. 272-281. Montgomery, D.C (2001). Design and Analysis of Experiments. Wiley & Sons, Inc. ed.

Oard, D.W., Kim, J (1998). Implicit feedback for recommender systems. In: Proceedings of the AAAI Workshop on Recommender Systems.

Oard, D.W., Kim.J (2001). Modeling information content using observable behavior. In proceeding of the 64th Annual meeting of the American Society for Information Science and Technology, USA, p. 38-45.

Rafter, R., Smyth, B (2001). Passive profiling from server logs in an online recruitment environment. In: Proceedings of the IJCAI Workshop on Intelligent Techniques for Web Personalization (ITWP 2001), USA, p. 35-41.

Salton, G., Buckley, C (1990). Improving retrieval performance by relevance feedback. Journal of the American Society for Information Science, 41(4) p. 288-297

Shapira, B., Kantor , P.B., Melamed, B (2001). The effect of extrinsic motivation on user behavior in a collaborative information finding system, Journal of the American Society for Information Science, 52 (11) 879-887.

Shapira, B., Taieb-Maimon M., Nemeth, Y (2005). Subjective and Objective Evaluation of interactive and automatic query expansion Online Information Review, 29 (4) 374-390.

White, R.W., Ruthven, I., Jose, J.M (2002). The use of implicit evidence for relevance feedback in web retrieval, Proceedings of the 24th BCS-IRSG European Colloquium on IR Research: Advances in Information.

Zhang, B.T., Seo, Y.W (2001). Personalized Web documents filtering using reinforcement learning. Applied Artificial Intelligence 15 (7) 665-685.

Bracha Shapira *

Department of Information Systems Engineering

Ben-Gurion University, Beer-Sheva, P.O.B. 653, Israel

Meirav Taieb-Maimon

Department of Information Systems Engineering

Ben-Gurion University, Beer-Sheva, P.O.B. 653, Israel

Anny moskowitz

Department of Information Systems Engineering

Ben-Gurion University, Beer-Sheva, P.O.B. 653, Israel

* To whom all correspondence should be addressed

(1) A two pages version of this document was accepted to the ACM Symposium of Applied Computing (SAC) 2006 conference

Bracha Shapira is currently a senior lecturer at the department of Information Systems Engineering at Ben-Gurion University of the Negev in Israel. She holds an M.Sc. degree in computer science form the Hebrew University in Jerusalem and a Ph.D. in Information Systems from Ben-Gurion University. She spent two years at Rutgers University in New-Jersey where she has conducted post-doc research. Her research interests include information filtering, user-profiles, recommender systems and privacy, as well as formal model of IR systems. Bracha's articles have been published in IR--related referred Journals (such as JASIST, DSS, CACM and more), and she has presented her work in professional conferences. She also worked for telecommunication companies in Israel as a system engineer for real-time applications.
Table 1. Classification of behaviors for implicit indicators

Category Observable behavior

Examination Selection, Duration, Edit, Repetition,
 Purchase (object or subscription), scrolling

Retention Save, print, delete, bookmark

Reference Object-> object (forward), portion->

Table 2. Summary of Findings Related to Implicit Indicators

Category and Indicator Reference Context of experiment

Examination Reading Morita and Examination of
time Shinoda 1994 Correlation between
Retention Copy, save reading time and
 interests of
 Usenet News

Examination Reading White et. al. summaries of Web
 2002 document to derive
 expansion terms for

Examination reading Claypool Browsing Web
time, scrolling, mouse et. al., 2001 documents,
clicks and movement

Examination Reading Konstan et. al., Usenet newsgroup
time 1997

Examination reading Zhang and User modeling for a
time, scrolling Seo 2001 filtering system
Retention Bookmark
Reference Hyperlinks

Examination reading time Rafter and User modeling for using
(enhanced-calculated as Smyth 2001 accessing job postings
the standard deviation
from the "normal reading

Examination Reading Kim et., al Reading academic
time Retention Print 2000 papers

Category and Indicator Finding

Examination Reading Reading time is indicative
time to the interest, save and
Retention Copy, save copy not indicative

Examination Reading User's performance using
 implicit or explicit
 feedback was not
 significantly different

Examination reading Reading time, scrolling-useful
time, scrolling, mouse indicators, Mouse
clicks and movement clicks and movement-not

Examination Reading Reading time-useful
time indicator

Examination reading Reading time and
time, scrolling book marking-useful
Retention Bookmark indicator, Scrolling and
Reference Hyperlinks hyperlinks not useful

Examination reading time Reading time was found
(enhanced-calculated as useful- enhanced reading
the standard deviation time - better
from the "normal reading

Examination Reading Reading time only a binary
time Retention Print indicator, printing useful
 (only 14 cases checked)

Category and Indicator Comments

Examination Reading
Retention Copy, save

Examination Reading

Examination reading Compared implicit indicators
time, scrolling, mouse to explicit feedback. Most
clicks and movement effective threshold of
 interest is about 20 seconds

Examination Reading Most effective threshold of
time interest--20 seconds

Examination reading Compared implicit
time, scrolling indicators to explicit
Retention Bookmark feedback
Reference Hyperlinks

Examination reading time Most effective threshold
(enhanced-calculated as of interest 50 seconds
the standard deviation
from the "normal reading

Examination Reading
time Retention Print

Table 3. Correlation Coefficient of the Significantly Correlated

Indicator Correlation coefficient(r)

Single Indicators

Reading time 0.18
Mouse movement 0.25
Scrolling 0.07
Print 0.19
Bookmark 0.06
No. of visited links 0.17
Level of Interaction 0.1

Relative Indicators

Mouse movement normalized by Reading time 0.27
Visited links normalized to no. of links 0.08

Combined Indicator

Combined Indicators according to 0.4

Figure 3. Box Plot-Time Spent (Reading time)

1 12.23 5
2 17.71 5
3 20.75 13
4 26.64 14
5 27.49 16

Note: Table made from bar graph.

Figure 4. Time of Mouse Moving

1 1.99 0.62
2 2.68 0.72
3 3.54 2.53
4 4.09 2.62
5 5.88 3.32

Note: Table made from bar graph.

Figure 5. Mouse Moving Time Normalized by Reading Time

1 11.67 5.3
2 15.15 12.5
3 21.34 17.15
4 20.05 16.75
5 23.52 21.77

Note: Table made from bar graph.
COPYRIGHT 2006 Digital Information Research Foundation
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2006 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Author:Shapira, Bracha; Taieb-Maimon, Meirav; Moskowitz, Anny
Publication:Journal of Digital Information Management
Date:Sep 1, 2006
Previous Article:On the use of ontologies for an optimal representation and exploration of the web.
Next Article:Peer-to-peer content search supported by a distributed index in a publication/search model.

Terms of use | Privacy policy | Copyright © 2019 Farlex, Inc. | Feedback | For webmasters