Printer Friendly

Urologists in cyberspace: A review of the quality of health information from American urologists' websites using three validated tools.

Author(s): Lih-Ming Wong, MD, Hanmu Yan, MD, David Margel, MD, Neil E. Fleshner, MD, FRCSC


Face-to-face consultation with a doctor is traditionally the way health information has been conveyed. However, the Internet, with its quick and easy access to information, is changing the society we live in and how we get information.

Online searches for health information are commonplace. A 2010 survey found that of the 74% of adults who use the Internet in the United States, 80% look for health information,[sup.1] which translates to 59% of all adults. Older surveys reported figures ranging from 32% to 71%.[sup.2,3] As society becomes Internet savvy, perceptive clinicians, researchers and hospitals have developed websites to engage users.

How do potential patients determine the legitimacy of information and credentials of the provider? Also, what guidance do doctors, such as urologists, have to follow to provide an ethical, educational and legal website for users? The Health on the Net Foundation (HON) code,[sup.4] DISCERN Plus instrument[sup.5] and LIDA tool[sup.6] are all freely available online and are designed to help consumers assess the quality of medical information. These sites also provide a checklist for doctors to follow when creating a website.

There is little doubt that the health care system in the U.S. is one of the most commercialized in the world. In 2007, health care was a $2 trillion industry, supported by private insurance companies, where the needs of patients are balanced against investors' requirements for profit.[sup.7] Many U.S. physicians manage their medical practices as businesses and having a website is a way to compete in a commercialized environment. In this study, we have used the HONcode, DISCERN Plus and LIDA tools to review the quality of health information on urologists' websites across the U.S. This assessment can provide valuable information and guidance for others who want to create or improve the quality of their own websites.


After looking at ecological data of urologist density in the U.S., we found that urologists (and their websites) are most likely in metropolitan counties.[sup.8] To identify a range urologists' websites, we sourced the 10 most populous cities from the U.S. Census Bureau (Table 1).[sup.9] Using com,[sup.10] we searched "urologist + city" and identified the first 10 hits in each location.

The three tools used to assess the websites are freely available online. HON is a non-governmental organization that developed the HONcode of conduct to encourage the dissemination of quality health information for patients, professionals and the general public.[sup.4] The HONcode consists of 15 principles (Table 2) to guide users in assessing and creating a reliable and credible website. Websites can freely apply for evaluation by HON and attain a HONcode certification seal. The DISCERN tool,[sup.5] an online project funded by the National Health Service (NHS) Executive Research and Development Programme, helps users to judge the quality of written information, but also aims to increase involvement in treatment decisions by raising issues to discuss with health professionals (Table 3). It consists of 16 questions scored 1 to 5 for a total of 80 points. Questions 1 to 8 address the reliability of the publication, questions 9 to 15 assess information provided on treatment choices and question 16 is an overall rating. The LIDA tool, designed by a health-care consultancy (Minervation), measures the accessibility, usability and reliability of health care websites. The online version of the LIDA tool,[sup.6] upon entering a website's URL, will automatically assess the accessibility, usability and reliability of the site (Table 4). The overall score is given as a percentage. Although LIDA is a tool provided by a private consultancy firm and there are potential conflicts of interest, the tool itself is freely available and is currently the only tool that checks HTML and metadata for errors.

Websites were categorized as academic, group or solo practices. Use of medical design website services and use of interactive audiovisual media and social media marketing tools (such as Facebook and Twitter) were also noted.

Each website was scored using HONcode, DISCERN and LIDA by a single researcher. The median score was used to dichotomize the cohort for each instrument; multivariable logistic regression was used to identify independent predictors of higher scores. Analysis for correlation between scores of the three systems was performed. Statistical analysis was performed with SPSS (Statistical Package for the Social Sciences, Cary, NC).


A total of 100 websites were considered. After eliminating links to duplicate sites, web-directory sites and a cybersquatter (a site using a domain name with bad faith intent to profit from the goodwill of a trademark), we analyzed 78 websites in total. There were 18 academic institutions, 43 group and 17 solo practices. A medical website design service was by 18 websites. The HONcode seal and American Urological Association (AUA) logo were posted on 3 websites (3/78, 4%). Social media tools were used by 16 websites. We tallied the various interactive online tools used (Table 5).

The HONcode principles

For each principle, a score of 1 was given if it was present (total 15). The median HONcode score was 5.5 (interquartile range [IQR] 5-7, range: 1-10). We tallied the websites based the the HONcode principles (Table 2). Websites were good at giving author(s) credentials (94.9%), providing a mission statement (97.4%) and identifying the intended audience (89.7%). Attribution (bibiliographic references 30.4%, last modification date 33.3%) and justification (8.6%) of information was poor, as was financial disclosure (2.6%). Advertising was present on 26 sites (7 sites directly and 19 via links to external websites marketing medical products), but none of these 26 sites had an advertising policy. A medical legal disclaimer was present in 56% of websites and a privacy policy in 68%.


The median total DISCERN score was 36/80 (IQR 30-45) (Table 3). The questions from DISCERN that scored high were: "Is it relevant;" "Does it provide details of additional sources of support and information;" "Does it describe how each treatment works;" and "Does it describe the benefits of each treatment." Websites were poor at declaring sources of information (other than the author), dating when information used was produced, and describing risks of treatment or what would happen if no treatment is used. There were no websites that addressed the following two DISCERN questions: "Does it describe how the treatment choices affect overall quality of life" or "Does it provide support for shared decision-making."

LIDA tool

The overall median LIDA score was 66/100 (range: 50-65) (Table 4). Accessibility (81/100) and usability (58/100) were reasonable; however, reliability (27/100) of most sites was poor.

Multivariate logistical regression

We tallied the multivariate logistical regression for predictors of high HON, DISCERN and LIDA scores (Table 6). These include predictors for HON, including academic centres (odds ratio [OR] 6.79, confidence interval [CI] 1.24-37.32, p = 0.03) and the use of medical website design services (OR 17.24, CI 3.80-78.11, p [less than or equal to] 0.01); DISCERN, including academic centres (OR 23.13, p [less than or equal to] 0.01, CI 3.15-169.99 and group practices (OR 7.19, p = 0.02, CI 1.33-38.93). For the LIDA tool, there were no predictors of higher scores. Pearson correlation did not show any correlation between the three scores.


With the emergence of the Internet, doctors need to stay abreast of shifts in how society communicates and searches for information. A website is more than the simple directory where you can look up a urologist. Websites enable patients to access their doctors more easily, and research their medical conditions and the doctors. For doctors, websites allow them to increase their exposure to the community, market their services and streamline the administrative burden of running a practice. Despite the pressures of an increasingly competitive, commercial environment, doctors need to present information ethically, balanced, unbiased and medico-legally safe.

There are many ways to assess the quality of online health information.[sup.11]-[sup.13] These include codes of conduct, third-party certification/quality assurance marking, logos, quality seals, specially designed health search engines, evaluation instruments, user guides and filters.[sup.14] In our review of 78 websites using three different instruments, we outline several areas that were consistently poor and could be improved.

Clear aim for the website

A good website not only has a mission statement (77/78), but should clearly state the purpose of the site (3/78). The author(s) of the website and the site's target audience (i.e., patients, medical professionals) should be clearly identified. The function of the website should also be stated. These functions could include to provide information on urologic diseases, to introduce the attending urologists in the practice or to take bookings.


The HONcode system includes a medical legal disclaimer (principle 7) and a privacy policy (principle 6). A medical legal disclaimer, present in 56% of websites reviewed, generally states that the medical information provided is not medical advice, which should be obtained from a medical consultation. It may also provide that no warranties are given in relation to the medical information supplied on the website, and that no liability will accrue to the website owner in the event that a user suffers loss as a result of reliance upon the information.[sup.15] Templates for medical legal disclaimers are available, and often used by medical website designers, however formal legal advice is always advisable. A sufficiently misleading statement, though unintentional, can potentially nullify a disclaimer. A privacy policy, present in 68% of websites reviewed, may be as simple as stating that personal information is not tracked when visiting the website. For websites that collect personal information, The Health Insurance Portability and Accountability Act of 1996 Privacy and Security Rules available on the U.S. Department of Health and Human Services website, protects the privacy of individually identifiable health information and is the national standard for security of electronic protected health information.[sup.16]

Referencing of information

Many websites (69/78) had health information on urological diseases and treatment. All three tools used include adequate referencing of health information in their scoring systems. Only 28% (21/69) of websites acknowledged the use of external references in compiling their health information and 33.3% (23/69) of sites included the date of most recent update, or when the information was written.

Presence of advertising

In the websites we examined, 33% (26/78) had advertising; 7 sites had direct ads, while the other 19 had links to external websites marketing medical products and devices. It is crucial that sites make a clear distinction between health information and advertising. An advertising policy should be present and it should be clear whether payment is received from advertisements. It should also be clear that the doctor does not endorse or recommend any products advertised and that the doctor is not responsible for the effectiveness of the product. If no advertising is present, a statement affirming that the website does not accept or host any advertising is suggested by the HONcode.[sup.4]

Being balanced and unbiased

The DISCERN instrument rewards websites that provide fair and impartial information. A range of treatment options should be discussed, including "no treatment," such as active surveillance for prostate cancer. We found that some statements ("The best cure rate for localized prostate cancer is with the use of radical prostatectomy" or "Radiation is generally not thought of as a curative procedure" [for prostate cancer]) are clearly misleading. These statements are personal opinions without scientific base and, in the end, harm the profession as a whole.

Our analysis found academic centre websites were associated with higher HON and DISCERN scores. These larger centres have greater resources and access to expertise. Use of a medical website designer was associated with higher HON score. One of the medical website designers used by several websites was certified with the HONcode seal, and thus was familiar with HON principles. A good medical website designer will not only provide technical expertise in creating a customizable website, but have a checklist to ensure the HONcode and the other tools are being used (Table 2, Table 3, Table 4). While the HONcode, DISCERN and LIDA are helpful guides, ultimately the reliability of the information depends on the clinician.

There have been other articles discussing the quality of urology websites. Lawrentschuk and colleagues evaluated multilingual uro-oncology websites for the HONcode accreditation in 2009. Of these websites, 11% to 29% of sites had the HONcode accreditation seal.[sup.17] Fast and colleagues used both the HONcode seal and DISCERN Plus instrument to assess 3 pediatric urology topics.[sup.18] They found the HONcode seal in 25% to 30% of websites; the average DISCERN score was 45-60 with no correlation between mean DISCERN score and presence of HONcode seal. In our sample, only 3 websites had the HONcode seal. It is important to note that absence of the HONcode seal does not necessarily imply a poor quality website. The HONcode seal is obtained by voluntary application for certification. A website may still rate highly when scored using the HON principles, but not have the seal because they were unaware of its existence. There is evidence recognizing that patients seek health information and that this information can improve medical outcomes and reduce anxiety.[sup.19] However, there is no data describing what patients actually want in a health care website.

What guidance is available to create a website that accomplishes the things above, but is also ethical, legal and contemporary? We would suggest that governance should come from the Medical Boards or licensing authorities of doctors from each state or nation (e.g., College of Physicians and Surgeons of Ontario). The Medical Board is usually responsible for guiding good practice, registering and regulating health professionals and protecting health care consumers. Furthermore, legal requirements often vary regionally. A policy, specifically for electronic media, would be ideal.

Our paper does have limitations. We are not formally trained in using the HONcode, DISCERN or LIDA tools. However, these tools are meant for general public use and there will be differences in interpretation of criteria, particularly with DISCERN, as there is a range of points (2,3 or 4) for a partial response. Each instrument has its own focus, with DISCERN dedicating half of its questions to treatment and LIDA calculating accessibility of HTML coding. In our experience, the HONcode provided the best overall guidance. Ease of readability is important for medical websites as many of them target the general population. This was not assessed directly by any of the three tools used. However in studies using LIDA and the Flesch Reading Ease score, an observed correlation between the two was seen.[sup.20]-[sup.22] Readability ease is calculated using a mathematical formula considering average sentence length and average number of syllables per word.[sup.23] Inter-observer variation in scoring was minimized by having a single researcher perform all web-site assessments. We used a single Internet search engine (Google) to identify websites. Google has over 80% of the desktop search engine share[sup.24] and it is unlikely that other search engines would identify websites with differing quality. The urologist websites were identified by most populous city, but other methods, such as highest urologist density, urologic subspecialty, condition or procedure, could potentially result in a different sample of websites. As our sample of websites was mostly from metropolitan urologists, with a high proportion of academic institutions, the results may not be generalizable to non-metropolitan urologists. However, in our sample several of the highest scores were achieved by solo and group practices; this demonstrates that good quality websites can be made without the resources and backing of an academic centre. Furthermore, the benefit of the Internet is that patients and physicians in rural communities, regardless of geographical location, will be able to access the same urological information. Finally, while urologists may not represent all types of doctors in cyberspace, a trend in the literature, across multiple specialties has called for an improvement in the quality of medical websites.[sup.20]-[sup.22,25]


As society evolves in the way it communicates and shares information, doctors should be leaders in the provision of health information. Websites are the forefront. Urologists are out there in cyberspace, but as a profession we can improve. Using three validated tools for appraising online health information, we found a wide variation in quality of urologists' websites in the United States. Guidance for doctors and other health professionals should come from national or state health regulatory bodies. With added increased awareness of available resources, such as the HONcode, DISCERN and LIDA tools, the quality of health information on doctors' websites can be improved.

Competing interests: None declared.

This paper has been peer-reviewed.


1.. Fox S. The Social Life of Health Information, 2011. Pew Research Center's Internet and American Life Project. Accessed March 18, 2013.

2.. Harris Interactive. 160 million Americans have gone online for health info. August 13, 2007. Accessed March 18, 2013.

3.. The Center for Studying Health System Change. More Americans Seeking Health Information, Especially Online.August21 2008 Accessed March 18, 2013.

4.. Health On the Net Foundation. HON Code of Conduct (HONcode) for medical and health Web sites. Accessed March 18, 2013.

5.. Radcliffe Online. Discern online. Accessed March 18, 2013.

6.. Tomlin A, Badenoch D. The Minervation validation instrument for healthcare websites (LIDA tool). Accessed March 18, 2013.

7.. Relman AS. A Second Opinion: Rescuing America's Health Care. New York: Public Affairs; 2007.

8.. Odisho AY, Fradet V, Cooperberg MR, et al. Geographic distribution of urologists throughout the United States using a county level approach. J Urol 2009;181:760-5

9.. US Census Bureau. American Fact Finder. 2011 Accessed March 18, 2013.

10.. Google Incorporated. Accessed March 18, 2013.

11.. Jadad A, Gagliardi A. Rating health information on the Internet: navigating to knowledge or to Babel? JAMA 1998;279:611-4.

12.. Eysenbach G, Powell J, Kuss O, et al. Empirical Studies Assessing the Quality of Health Information for Consumers on the World Wide Web. JAMA 2002;287:2691-700

13.. Bomba D. Evaluating the Quality of Health Web Sites: Developing a Validation Method and Rating Instrument.38th Hawaii International Conference on System Sciences. Hawaii: Institute of Electrical and Electronics Engineers; 2005:139-48

14.. Wilson P. How to find the good and avoid the bad or ugly: a short guide to tools for rating quality of health information on the Internet. BMJ 2002;324:598-602

15.. SEQ Legal LLP. Website Law. 2011. Accessed March 18, 2013.

16.. United States Department of Health and Human Services. Health Information Privacy. 2011 Accessed March 18, 2013.

17.. Lawrentschuk N, Abouassaly R, Hackett N, et al. Health Information Quality on the Internet in Urological Oncology: A Multilingual Longitudinal Evaluation. Urology 2009;74:1058-63

18.. Fast AM, Deibert CM, Hruby GW, et al. Evaluating the quality of Internet health resources in pediatric urology. J Pediatr UrolEpub 2012 Jan 24.

19.. NHS. Patient information. 2010. Accessed March 18, 2013.

20.. Grewal P, Williams B, Alagaratnam S, et al. Quality of vascular surgery Web sites on the Internet. J Vasc Surg 2012;56:1461-7

21.. Soobrah R, Clark SK. Your patient information website: how good is it? Colorectal Dis 2012;14:e90-4

22.. Kirthi VMB. Coronary angioplasty and the internet: what can patients searching online expect to find? J Interv Cardiol 2012;25:476-81

23.. Flesch R. A new readability yardstick. J Appl Psychol 1948;32:221-33

24.. Net Applications. Desktop Search Engine Share. 2011. Accessed March 18, 2013.

25.. Muthukumarasamy S, Osmani Z, Sharpe A, et al. Quality of information available on the World Wide Web for patients undergoing thyroidectomy: review. J Laryngol Otol 2012;126:116-9


Table 1.: Ten most populous cities in the United States used to search for urologists' websites* [Table omitted]

Table 2.: HONcode principles[sup.#] [Table omitted]

Table 3.: DISCERN criteria[sup.#] [Table omitted]

Table 4.: LIDA tool [Table omitted]

Table 5.: Interactive Internet tools [Table omitted]

Table 6.: Multivariate logistical regression for predictors of high HON, DISCERN and LIDA scores [Table omitted]

Author Affiliation(s):

[1] Division of Uro-oncology, Department of Surgical Oncology, Princess Margaret Hospital, University of Toronto, Toronto, ON

Correspondence: Dr. Lih-Ming Wong, Division of Uro-oncology, Department of Surgical Oncology, Princess Margaret Hospital, 610 University Ave., Toronto, ON M5G 2M9;
COPYRIGHT 2013 Canadian Urological Association
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2013 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Title Annotation:Original Research
Author:Wong, Lih-Ming; Yan, Hanmu; Margel, David; Fleshner, Neil E.
Publication:Canadian Urological Association Journal (CUAJ)
Article Type:Report
Geographic Code:1USA
Date:Apr 1, 2013
Previous Article:Alteration of renal cyclooxygenase expression due to partial unilateral ureteral obstruction in neonatal.
Next Article:Paging Dr. Google.

Terms of use | Privacy policy | Copyright © 2019 Farlex, Inc. | Feedback | For webmasters