Confirming the reliability of a theory-based questionnaire.Abstract: The purpose of this paper is to describe the process of confirming the reliability of the Survey to Predict Adolescent Athletes' Dietary Supplement Noun 1. dietary supplement - something added to complete a diet or to make up for a dietary deficiency diet - a prescribed selection of foods vitamin pill - a pill containing one or more vitamins; taken as a dietary supplement Use (SPAADSU). The SPAADSU is based on the Theory of Reasoned Action The theory of reasoned action (TRA), developed by Martin Fishbein and Icek Ajzen (1975, 1980), derived from previous research that started out as the theory of attitude, which led to the study of attitude and behavior. , which has three major constructs: 1) attitude towards the behavior in question; 2) subjective norm; and 3) behavioral intention. This paper focuses on the subjective norm scale as an example and the use of SPSS A statistical package from SPSS, Inc., Chicago (www.spss.com) that runs on PCs, most mainframes and minis and is used extensively in marketing research. It provides over 50 statistical processes, including regression analysis, correlation and analysis of variance. 11.0 as the statistical software to run the reliability analysis. ********** Health professionals interested in health pro. motion/education commonly use questionnaires to collect data. In fact, the entry-level health educator should possess the skill to "employ or develop appropriate data-gathering instruments" (National Center for Health Education and Credentialing Credentialing is the administrative process for validating the qualifications of licensed professionals, organizational members or organizations, and assessing their background and legitimacy. , 1996). Graduate level competencies include the ability to "develop valid and reliable data collection instruments" (American Association American Association refers to one of the following professional baseball leagues:
NCHEC National Center for Home Equity Conversion Mortgage , & Society for Health Education, 1999). The purpose of this paper is to describe the process of confirming the reliability of a theory-based questionnaire. Consider the following scenario. As you peruse pe·ruse tr.v. pe·rused, pe·rus·ing, pe·rus·es To read or examine, typically with great care. [Middle English perusen, to use up : Latin per-, per- the professional literature you come across an article describing a study that was successful in changing behavior by applying a specific psychosocial psychosocial /psy·cho·so·cial/ (si?ko-so´shul) pertaining to or involving both psychic and social aspects. psy·cho·so·cial adj. Involving aspects of both social and psychological behavior. theory. The questionnaire the authors employed to document that success was stated to be reliable. You think it would be a great idea to conduct and evaluate the same intervention in your community. Since the questionnaire is not in the public domain, you contact the authors of the study and get permission to use the survey as is or with simple modifications. Following the recent guidance from O'Rourke (2001) in the American Journal of Health Studies (AJHS AJHS American Jewish Historical Society AJHS Australian Jewish Historical Society AJHS Apache Junction High School (Ajo, AZ) AJHS Andrew Jackson High School AJHS Aptakisic Junior High School (Grove, Illinois) ) you may be able to improve the format of the questionnaire for your particular population. Of course your community is different from the one in the original article. A questionnaire that was reliable with one population may not be with another. How can you establish that the questionnaire employed in the original study (which had such great results) would be reliable when used with your population? Ideally, you could begin by pilot testing the questionnaire with a smaller representative group from the priority population. A reliability analysis on the data from the pilot test can tell you how well the questionnaire would work in a full study of your own community, and how you might fine-tune it to improve its performance. Or, if budget constraints A Budget Constraint represents the combinations of goods and services that a consumer can purchase given current prices and his income. Consumer theory uses the concepts of a budget constraint and a preference ordering to analyze consumer choices. force you to skip the pilot test and go straight to the main study, reliability analysis can still assess the adequacy of the questionnaire and suggest improvements that can be made even at that stage. RELIABILITY There are a vast number of definitions and descriptions of reliability (see Anastasi, 1969, Cronbach, 1970, or Windsor, Baranowski, Clark, & Cutter cutter, small, one-masted sailing vessel, with a rig similar to that of a sloop except that it usually has a sliding bowsprit and a topmast. From 1800 to 1830 cutters were in service between England and France. , 1994, among others); however, ultimately and simply reliability refers to the degree of self-consistency among the scores earned by an individual (Ghiselli, 1964). From a questionnaire composed of several question items (say k in number) all of which are designed to assess the same trait trait (trat) 1. any genetically determined characteristic; also, the condition prevailing in the heterozygous state of a recessive disorder, as the sickle cell trait. 2. a distinctive behavior pattern. or characteristic, reliability addresses the degree to which a person's observed scores on each item agrees with his/her overall or universal score (Cronbach, 1970). If the item scores all agree closely with the universal score (often measured as the average score on the questionnaire), then the questionnaire would be considered reliable. If the item scores agree very closely with the universal score, then they should also be very closely related to one another as well. This way of assessing reliability is referred to as internal consistency In statistics and research, internal consistency is a measure based on the correlations between different items on the same test (or the same subscale on a larger test). It measures whether several items that propose to measure the same general construct produce similar scores. , and is the approach presented in this paper. SPAADSU FRAMEWORK Perko (1999) reported the development of the Survey to Predict Adolescent Athletes' Dietary Supplement Use (SPAADSU). Perko used a combination of methods to examine the measurement properties (reliability and validity) of the SPAADSU. The present paper aims to build on Perko's work, without duplicating those efforts. Perko (1995) conducted a factor analysis to show that items designed to reflect intention, attitude, and subjective norm in regard to the use of dietary supplements did indeed inter-correlate in subsets corresponding to the respective constructs. Factor analysis is a popular method for determining the specific scale structure (constructs) present in a set of data. In general, factor analysis is a complex method often requiring the assistance of a statistician to be employed correctly. This approach is not the focus of this paper. Instead, we address situations where the literature has established a specific scale structure for a set of theoretical constructs, as Perko (1999) has done (or else such a structure has been assumed or proposed), and where the objective is to assess the reliability of empirical measures In probability theory, an empirical measure is a random measure arising from a particular realization of a (usually finite) sequence of random variables. The precise definition is found below. Empirical measures are relevant to mathematical statistics. of that structure by examining their internal consistency. The SPAADSU is based on the Theory of Reasoned Action (TRA TRA Training TRA Transfer TRA Transition TRA Tennessee Regulatory Authority TRA Telecommunications Regulatory Authority (Oman) TRA Tax Reform Act (1976, 1984, or 1986) TRA Teachers Retirement Association ; see Fishbein & Ajzen, 1975). The TRA has three major constructs: 1) attitude towards the behavior in question--one can hold either a positive or negative attitude toward a behavior; 2) subjective norm--the perception of what others think about the behavior and the motivation to comply with that perception; and 3) behavioral intention--a person's subjective probability of performing the behavior (Fishbein & Ajzen, 1975). The SPAADSU consists of 44 items. Eight items measure standard demographics The attributes of people in a particular geographic area. Used for marketing purposes, population, ethnic origins, religion, spoken language, income and age range are examples of demographic data. , athletic/recreational participation, and dietary supplement use. Attitude toward dietary supplement use is measured with a 10-item scale assessing the adolescents' positive or negative evaluation of engaging in the behavior. Subjective norm is measured with a 13-item scale for the adolescents' perception of the social pressure to engage in supplement use and their motivation to comply with influential others. Intention is measured with a 13-item scale of the adolescents' willingness to take dietary supplements in a number of situations. Composite scores are calculated for attitude, subjective norm, and intention. This paper will focus on the subjective norm scale (see Appendix) as an example and the use of SPSS 11.0 as the statistical software to run the reliability analysis. The data used for this "practical perspective" have been reported previously (Bartee, 2000; Bartee, et al., 2004). Please see these documents for details on the methods of data collection and a description of the sample. RELIABILITY ANALYSIS PROCEDURES To test the internal consistency of the SPAADSU scales, the reliability procedure was run using SPSS for Windows 11.0. Inter-item correlations, Cronbach's alpha Cronbach's  (alpha) has an important use as a measure of the reliability of a psychometric instrument. It was first named as alpha by Cronbach (1951), as he had intended to continue with further instruments. , the standardized standardizedpertaining to data that have been submitted to standardization procedures. standardized morbidity rate see morbidity rate. standardized mortality rate see mortality rate. item alpha, and alpha if item deleted will all be presented and discussed. To obtain these reliability indices, open your data file in SPSS and select analyze from the menu bar; then select scale and click on reliability analysis. Next, highlight the variables to be analyzed an·a·lyze tr.v. an·a·lyzed, an·a·lyz·ing, an·a·lyz·es 1. To examine methodically by separating into parts and studying their interrelations. 2. Chemistry To make a chemical analysis of. 3. . In this example, thirteen variables (Q24-Q36) were selected and put into the Items field. Select the alpha model, and within the Statistics section choose scale if item deleted (the key results of which will appear in the output as alpha if item deleted). In the Inter-Item section choose correlations (which will be called inter-item correlations in the output). It is important to note that the reliability procedure does not in fact create any scales and add them to your data file. Guided by the reliability test results, you will then have to accomplish those steps using the transform menu and the compute To perform mathematical operations or general computer processing. For an explanation of "The 3 C's," or how the computer processes data, see computer. dialog box A movable window that is displayed on screen in response to the user selecting a menu option. It provides the current status and available options for a particular feature in the program. . INTERPRETATION Why were these specific tests performed, what exactly do they tell the health professional, and what are their limitations? These questions will be addressed for each reliability index below. Discussion will revolve around Verb 1. revolve around - center upon; "Her entire attention centered on her children"; "Our day revolved around our work" center, center on, concentrate on, focus on, revolve about the SPSS Output (see Table 1). Inter-item correlation matrix Noun 1. correlation matrix - a matrix giving the correlations between all pairs of data sets statistics - a branch of applied mathematics concerned with the collection and interpretation of quantitative data and the use of probability theory to estimate population . The purpose of the inter-item correlations is to get some idea about how well the items relate to one another. First and foremost, it is essential to examine the inter-item correlations to insure that all or nearly all of them are positive. Negative correlations Noun 1. negative correlation - a correlation in which large values of one variable are associated with small values of the other; the correlation coefficient is between 0 and -1 indirect correlation , if any, should be very low and widely scattered Scattered Used for listed equity securities. Unconcentrated buy or sell interest. in the matrix. If any item has a consistent negative correlation with all or most of the other items, then clearly that item, as scored, is not measuring the same thing as the other items. Indeed, it may be measuring exactly the opposite of what the others are measuring. This situation can arise if the researcher has, for the sake of variety in the questionnaire, reversed the wording of some items so that a "strongly agree" response on one item corresponds in meaning to a "strongly disagree" response on another item. In that case, the scoring of the reverse-worded items should be recoded so that the highest numerical score is assigned to "strongly agree" on some items but to "strongly disagree" on others, as appropriate. After this recoding Noun 1. recoding - converting from one code to another coding, steganography, cryptography, secret writing - act of writing in code or cipher , all or nearly all of the inter-item correlations should be positive; if not, there is a bad item in the mix. As indicated at the outset, reliability implies internal consistency, which in turn requires that the items should be very closely related to one another. This would be seen through their "fairly" high, positive correlations Noun 1. positive correlation - a correlation in which large values of one variable are associated with large values of the other and small with small; the correlation coefficient is between 0 and +1 direct correlation with one another. If one or more of the items were correlated cor·re·late v. cor·re·lat·ed, cor·re·lat·ing, cor·re·lates v.tr. 1. To put or bring into causal, complementary, parallel, or reciprocal relation. 2. to a much lower degree than the remaining body of items (or negatively), this would suggest the elimination of such items from the scale. Looking at the Correlation Matrix in Table 1, it becomes apparent that three variables, Q34-Q36, are correlated to a much lower degree with the others than these others are with one another. This raises the question of whether to eliminate these items. The first reaction may be "yes"; however, the practitioner should be knowledgeable of the theory the questionnaire is based upon. In this case, the theoretical construct "subjective norm" of the Theory of Reasoned Action is known to have two components--normative beliefs and motivation to comply. Variables Q34-36 represent the second component. So, the practitioner might decide to retain these variables in the scale, pending the results on other reliability statistics to be discussed below. As an alternative, one could consider dividing the scale into two subscales, and running reliability separately on each component. (In results not shown from the present example, Cronbach's alpha on Q34-Q36 alone was .79, a very respectable level of reliability that indicates the subscales could indeed be analyzed separately if desired.) Cronbach's alpha The purpose of the Cronbach's alpha is to provide a lower bound estimate for how much variance in the empirical scale would be explained by (or shared with) an imaginary perfect measure of the same theoretical construct. The overall Cronbach's alpha for the subjective norm scale (variables Q24-Q36) was .89, a very high level of reliability. In other words Adv. 1. in other words - otherwise stated; "in other words, we are broke" put differently , at least 89 percent of the variance in our empirical scale would be held in common with a perfect measure of the same construct. Many researchers regard an alpha of .60 or better as sufficient to justify using a scale. For those familiar with the split-half method for assessing reliability, alpha has an additional interpretation worth noting. In the split-half approach (available as an option in the SPSS reliability procedure) the items in a scale are randomly split into two subscales, and the correlation between the two subscales is used to estimate their "true score" variance. In essence, Cronbach's alpha calculates this true score variance from the collection of all possible splits of the items. Since it considers all possible splits, Cronbach's alpha can be considered a much better estimate of reliability than the reliability obtained from split-half. The "standardized item alpha" is also given in the SPSS output (Table 1). This is a confusingly con·fuse v. con·fused, con·fus·ing, con·fus·es v.tr. 1. a. To cause to be unable to think with clarity or act with intelligence or understanding; throw off. b. labeled statistic statistic, n a value or number that describes a series of quantitative observations or measures; a value calculated from a sample. statistic a numerical value calculated from a number of observations in order to summarize them. ; it would be more informative to call it "alpha for standardized items." As long as all of the items in your scale are scored on the same rubric RUBRIC, civil law. The title or inscription of any law or statute, because the copyists formerly drew and painted the title of laws and statutes rubro colore, in red letters. Ayl. Pand. B. 1, t. 8; Diet. do Juris. h.t. (say, 1 to 5), with the same range and similar standard deviations In statistics, the average amount a number varies from the average number in a series of numbers. (statistics) standard deviation - (SD) A measure of the range of values in a set of numbers. , the numerical values of alpha and the "standardized item alpha" will be very similar, as they are here. However, if some items are measured on one rubric (1 to 5) and others are measured on a different rubric (perhaps zero to 100), these two versions of alpha can differ substantially. The "standardized item alpha" tells you what the alpha would be for your scale if--before constructing the scale--you first convert all of the scores on every item into standardized scores. (These are also called z-scores, and are computed as the difference between each respondent's original score and the sample mean score, with the difference then divided by the sample standard deviation. The SPSS descriptives procedure will add z-scores for any or all variables to your data file.) If the "standardized item alpha" is considerably larger than alpha, then standardizing the items before constructing the scale is a good idea. In that case, you should first rerun re·run n. The act or an instance of rebroadcasting a recorded movie or a recorded television performance. tr.v. re·ran , re·run, re·run·ning, re·runs To present a rerun of. the reliability procedure on the new, standardized items, to maximize the usefulness of the results to be discussed next. Alpha if item deleted. Lastly, the column labeled alpha if item deleted is used as a diagnostic tool to identify items that are candidates for elimination from the scale. For each row of the table, the value in this column tells you what alpha would be for a scale consisting of all the other variables except the variable that corresponds to that row. Ordinarily or·di·nar·i·ly adv. 1. As a general rule; usually: ordinarily home by six. 2. In the commonplace or usual manner: ordinarily dressed pedestrians on the street. , alpha tends to go down as the number of items in a scale goes down, because scales made up of multiple indicators are usually superior to measures using only one or a few items. Hence, the practitioner should scan down the "alpha if item deleted" column and take special note if any entries are greater than the overall alpha for the entire scale that appears at the bottom of the table. (The relevant comparison is to the overall alpha, not to the standardized item alpha.) If alpha would increase if a particular item were deleted from the scale, then the indication is that this item is measuring something different from what the other items measure (and the greater the increase, the greater the difference). This would violate the basic principle of internal consistency, suggesting that this item might not belong in the same scale with the others. As before, however, such judgments should be based not only on the empirical results, but also on the practitioner's knowledge of the theoretical basis for the questionnaire. In the present example, alpha would be increased by the deletion deletion /de·le·tion/ (de-le´shun) in genetics, loss of genetic material from a chromosome. de·le·tion n. Loss, as from mutation, of one or more nucleotides from a chromosome. of any of items Q34-Q36, but only slightly. Again, it might be recommended that these variables be retained in the scale with the acknowledgement that they do indeed represent a different component of the subjective norm construct. With these results in hand, the practitioner is in a position to make sound decisions about the remaining research: Does the pilot study indicate the questionnaire will be sufficiently reliable to support a full field study? Which items from the pilot test can or should be deleted for the full study? Do the results from the full study confirm sufficient reliability to justify further data analysis? Can some items be deleted from the scales before further analysis to enhance reliability? These and related questions can be addressed, systematically but with relative ease, using the SPSS reliability procedure as outlined in this paper. APPENDIX The following are questions representing the subjective norm construct on the Survey to Predict Adolescent Athletes Dietary Supplement Use. All questions were asked in a 5-point Likert scale Likert scale A subjective scoring system that allows a person being surveyed to quantify likes and preferences on a 5-point scale, with 1 being the least important, relevant, interesting, most ho-hum, or other, and 5 being most excellent, yeehah important, etc format (Strongly agree, Agree, No Opinion, Disagree, Strongly disagree). 24. My coach would support my using dietary supplements to improve sports performance. 25. My coach would support my using dietary supplements for general health reasons. 26. My parent(s) or guardian would support my using dietary supplements for general health reasons. 27. My parent(s) or guardian would support my using dietary supplements for better sports performance. 28. My teammates would support my using dietary supplements for better sports performance. 29. My teammates would support my using dietary supplements for general health reasons. 30. My doctor would support my using dietary supplements to improve sports performance. REFERENCES American Association for Health Education, National Commission for Health Education Credentialing & Society for Public Health Education. (1999). A competency-based framework for graduate-level health educators. Allentown, PA: NCHEC. Anastasi, A. (1969). Psychological Testing psychological testing Use of tests to measure skill, knowledge, intelligence, capacities, or aptitudes and to make predictions about performance. Best known is the IQ test; other tests include achievement tests—designed to evaluate a student's grade or performance . 3rd ed. London: Collier-Macmillan Limited Bartee, R.T. (2000). Predicting the use of dietary supplements marketed as nutritional ergogenic aids Ergogenic aids are any external influences which can positively affect physical or mental performance. These include mechanical aids, pharmacological aids, physiological aids, nutritional aids, and psychological aids. among high school athletes. Unpublished doctoral dissertation dis·ser·ta·tion n. A lengthy, formal treatise, especially one written by a candidate for the doctoral degree at a university; a thesis. dissertation Noun 1. . The University of Alabama The University of Alabama (also known as Alabama, UA or colloquially as 'Bama) is a public coeducational university located in Tuscaloosa, Alabama, USA. Founded in 1831, UA is the flagship campus of the University of Alabama System. , Tuscaloosa. Bartee, R.T., Grandjean, B.D., Dunn, M.S., Perko, M.A., Eddy, J.M., & Wang, M.Q. (2004). Predictors of dietary supplement use among adolescent athletes. Pediatric pediatric /pe·di·at·ric/ (pe?de-at´rik) pertaining to the health of children. pe·di·at·ric adj. Of or relating to pediatrics. Exercise Science, 16, 250-264. Cronbach, L.J. (1970). Essentials of Psychological Testing. 3rd ed. New York New York, state, United States New York, Middle Atlantic state of the United States. It is bordered by Vermont, Massachusetts, Connecticut, and the Atlantic Ocean (E), New Jersey and Pennsylvania (S), Lakes Erie and Ontario and the Canadian province of : Harper & Row. Fishbein, M., & Ajzen, I. (1975). Belief, attitude, intention and behavior: An introduction to theory and research. Reading, MA: Addison-Wesley. Ghiselli, E. E. (1964). Theory of Psychological Measurement. New York: McGraw-Hill. National Commission for Health Education Credentialing. (1996).A competency-based framework for professional development of certified See certification. health education specialists. New York: NCHEC. Perko, M. A. (1995). Development of an instrument to assess intentions, attitudes, and beliefs of adolescent athletes regarding dietary supplement use. Unpublished doctoral dissertation, The University of Alabama, Tuscaloosa. Perko, M. A. (1999). Development of a theory-based instrument regarding adolescent athletes and dietary supplements. American Journal of Health Studies, 15, 71-80. O'Rourke, T. (2001). Techniques to improve questionnaire format. American Journal of Health Studies, 17, 36-38. Windsor, R., Baranowski, T., Clark, N., & Cutter, G. (1994). Evaluation of Health Promotion, Health Education, and Disease Prevention programs. 2"a ed. Mountain View, CA: Mayfield Publishing Company. R. Todd Bartee, Ph.D. Burke D. Grandjean, Ph.D. Stephen L. Bieber, Ph.D. R. Todd Bartee, Ph.D. is an Assistant Professor in the Division of Kinesiology kinesiology Study of the mechanics and anatomy of human movement and their roles in promoting health and reducing disease. Kinesiology has direct applications to fitness and health, including developing exercise programs for people with and without disabilities, preserving and Health and Adjunct adjunct (aj´ungkt), n a drug or other substance that serves a supplemental purpose in therapy. adjunct Clinical Faculty in the Division of Public Health and Medical Education at the University of Wyoming UW is a national research university prominent in the fields of environment and natural resource research, specializing in agriculture, energy, geology, and water resource related fields. . Burke D. Grandjean, Ph.D. is Executive Director of the Wyoming Survey & Analysis Center and a Professor of Statistics and Sociology at the University of Wyoming. Stephen L. Bieber, Ph.D. is the Department Chair and Professor of Statistics at the University of Wyoming. Address all correspondence to R. Todd Bartee, Ph.D., Assistant Professor, Division of Kinesiology and Health, Adjunct Clinical Faculty, Division of Public Health and Medical Education, University of Wyoming, 1000 E. University Avenue, Dept. 3196, Laramie, WY 82071-3196, PHONE: 307.766.4191, FAX: 307.766.4098, E-MAIL e-mail: see electronic mail. in full electronic mail Messages and other data exchanged between individuals using computers in a network. : tbartee@uwyo.edu
Table 1. SPSS 11.0 Output for Subjective Norm Scale Reliability
Analysis
RELIABILITY ANALYSIS--SCALE (ALPHA)
Correlation Matrix
Q24 Q25 Q26 Q27 Q28
Q24 1.0000
Q25 .5446 1.0000
Q26 .4116 .7036 1.0000
Q27 .5838 .4991 .6058 1.0000
Q28 .5716 .5176 .4545 .6208 1.0000
Q29 .4364 .6294 .6251 .4724 .6162
Q30 .5428 .4488 .4548 .6263 .5200
Q31 .350 .5548 .6052 .4192 .4051
Q32 .6220 .552 .4931 .6280 .6551
Q33 .4524 .6207 .5880 .4786 .5182
Q34 .0527 .1185 .1198 .0972 .0795
Q35 .2019 .1898 .1592 .2128 .2524
Q36 .1050 .1562 .1630 .1226 .1458
Q29 Q30 Q31 Q32 Q33
Q29 1.0000
Q30 .4509 1.0000
Q31 .5774 .5218 1.0000
Q32 .5223 .6301 .4753 1.0000
Q33 .6506 .4751 .6550 .6481 1.0000
Q34 .1374 .1104 .1473 .1124 .1538
Q35 .2360 .2029 .1570 .2654 .2285
Q36 .1938 .1206 .2126 .1687 .2506
Q34 Q35 Q36
Q34 1.0000
Q35 .5715 1.0000
Q36 .5957 .5094 1.0000
N of Cases = 1713.0
RELIABILITY ANALYSIS--SCALE (ALPHA)
Item-total Statistics
Scake Scale
Mean Variance Corrected Squared Alpha if
if Item if Item Item-Total Multiple Item
Deleted Deleted Correlation Correlation Deleted
Q24 32.4542 71.7527 .6050 .5194 .8839
Q25 32.6334 71.1798 .6940 .6275 .8800
Q26 32.6474 70.2845 .6718 .6566 .8806
Q27 32.3859 69.7593 .6722 .6267 .8804
Q28 32.6959 70.7223 .6700 .5980 .8808
Q29 32.6947 71.1573 .6955 .6111 .8799
Q30 32.2709 71.6556 .6382 .5446 .8824
Q31 32.6328 71.5409 .6304 .5600 .8827
Q32 32.5797 70.4120 .7277 .6580 .8783
Q33 32.6708 70.8927 .7181 .6409 .8789
Q34 32.8424 76.7999 .2798 .4621 .9006
Q35 32.7642 74.8415 .3934 .4177 .8946
Q36 33.0922 76.5335 .3409 .4189 .8962
Reliability 13 items Alpha=.8927 Standardized item
Coefficients alpha=.8955
|
|
||||||||||||||||||
Printer friendly
Cite/link
Email
Feedback
Reader Opinion