Suitability of the perceived motivational climate in sport questionnaire-2 for dance research: a think aloud approach.
The concept of the "motivational climate" is originally derived from achievement goal theory (AGT). (12,13) AGT proposes that an important prerequisite for motivated behaviour is a desire to feel competent. (13) According to AGT, individuals perceive their competence in achievement activities in relation to two goal perspectives, task and ego. Individuals with a more task-involved goal perspective see themselves as successful when they try their best and improve their performance. (14) In contrast, more ego-involved individuals compare their performance to others and feel successful only when their performance is superior. (13)
AGT proposes a number of factors that influence whether an individual becomes task- or ego-involved in different settings. An important factor
is the prevailing goal structure that individuals perceive to be operating in situations, more widely known in the literature as the "motivational climate." Ames (12) conceptualized the motivational climate as represented by a hierarchical framework, with two higher order dimensions, task-involving and ego-involving features. A task-involving motivational climate is one in which individuals perceive self-referenced judgments of competence to be encouraged. (12) For example, a task-involving climate is characterized by significant others (e.g., the teacher) emphasizing self-development, encouraging learning and mastery of tasks, and recognizing effort and personal progression. (4,12) The focus on self-referenced judgments of ability promotes the adoption of a more task-focused goal involvement in which mastery and improvement are considered criteria for success. (13) Task-involved individuals are more likely to exhibit adaptive achievement behaviours such as working hard and persevering in challenging situations, regardless of the performer's level of ability. (13,14) Hence, task-involving climates have been found to be predictive of benefits, such as greater self-determined motivation for engagement in physical education (15) and higher levels of positive affect in vocational dancers. (3)
In contrast, an ego-involving climate is one in which individuals perceive a normative conception of competence to be emphasized with individuals' performances being assessed relative to others. (12) For example, in an ego-involving motivational climate, the teacher may encourage competition, rivalry, and social comparison among individuals. (4,12) Ego-involving climates are characterized by a hierarchy based on ability, with the most able receiving the most attention. (14) As a result of the focus on normative comparison when construing levels of competence, over time ego-involving motivational climates foster the development of athletes, and students' ego-focused goal involvement. Ego-involved individuals are more likely to exhibit maladaptive achievement behavior such as exerting little effort and avoiding challenges. (13) Hence, ego-involving motivational climates have been found to be associated with negative outcomes, including diminished self-esteem, (15) elevated levels of anxiety, (6) and greater inclination to drop out. (16)
Development of the Perceived Motivational Climate in Sport Questionnaire-2
The Perceived Motivational Climate in Sport Questionnaire (PMCSQ) (17) was originally developed to measure athletes' perceptions of the motivational climate created by their coach. In line with Ames' (12) conceptual framework, the PMCSQ comprises scales tapping both task-involving and ego-involving dimensions. Initially psychometric testing supported the concurrent validity of the measure when implemented in physical education settings. (18) However, Walling, Duda, and Chi (19) proposed that the measure could be improved and made more congruent with Ames' conceptualization of the motivational climate as having a hierarchical structure by including subscales underlying the higher order task- and ego-involving dimensions. Newton and colleagues (4) took on this task and developed the PMCSQ-2 for use in competitive team sport contexts. They extended and revised the PMCSQ to include the following subscales: (Task) Effort and Improvement, Cooperative Learning, and Important Role; (Ego) Unequal Recognition, Punishment for Mistakes, and Intra-Team Member Rivalry. The reliability and factorial validity of the PMCSQ-2 has been supported in a variety of competitive team sports. (4-6) The Intra-Team Member Rivalry subscale has, however, been reported in some studies (4,6) as exhibiting inadequate internal consistency ([alpha] =.54; [alpha] =.67, respectively). Newton and colleagues suggested that issues of reliability may be due to the low number of items within the IntraTeam Member Rivalry subscale and the language used, which may require refining. Despite potential limitations with items on the Intra-Team Member Rivalry subscale, the content validity of the PMCSQ-2 items has not been examined in any context since the initial validation of the measure. (4)
Application of the PMCSQ-2 in Dance Contexts
Interest in the motivational climate in dance contexts began in 2003 with the work of Carr and Wyon, (8) who examined the impact of perceptions of the motivational climate on dance students' achievement goals, trait anxiety, and perfectionism. They measured the climate via the PMCSQ-2 with minor rewording to suit the vocational dance setting. The PMCSQ-2 was also independently reworded by Quested and Duda, (7) who altered items to make them suitable for elite and vocational dance environments. For example, "On this team" was replaced with "In this dance school/company," "coach" with "teacher," "player" with "dancer," and "game" with "performance." As the PMCSQ-2 was originally designed for use in competitive team sport contexts, the phrasing and examples used in some of the items may not easily translate to dance. Thus, certain items had to be further adapted. For example, the item "On this team, only the players with the best 'stats' get praise" from the Unequal Recognition subscale of the original PMCSQ-2 is not directly transferable to dance contexts. Generally dancers do not have an objective measure by which they rank performance and ability (i.e., game "stats"). Therefore, Quested and Duda removed the reference to "stats," leaving the item to read: "Only the best dancers get praise." Deleting the reference to "stats" leaves open to interpretation what participants use as their gauge of "the best dancer" when completing this item. The lack of a clear criterion with which to interpret who the best dancers are creates ambiguity in the question, which could lead to dancers having trouble answering this item.
When employing measures with new subgroups or in different contexts, it is critical to assess whether the measure is viable for the new target population of interest as a scale may not always be valid across all situations. (20) To date, only two studies have attempted to examine the suitability of the PMCSQ-2 for dance. (3,9) Both of these took a quantitative approach and tested the factorial validity of the measure when it has been employed with dancers. In their work with vocational dancers, Quested and Duda (3) conducted confirmatory factor analysis (CFA) and found certain subscales of the PMCSQ-2 (i.e., Unequal Recognition, Important Role, and IntraTeam Member Rivalry) to have poor construct validity. Two of the items, namely one from the Unequal Recognition subscale ("The teachers think that only the lead dancers contribute to the success of a performance") and another from the Important Role subscale ("Each dancer feels as if he/she is an important team member") loaded poorly onto the intended factors and were removed from analyses. These investigators also report that two of the items measuring the dancers' perceptions of "Intra-Team member rivalry" ("Dancers are encouraged to outperform the other dancers," and "Dancers are 'fired up' [positively excited] when they perform better than their fellow dancers in a performance") loaded onto other factors. Quested and Duda propose these cross loadings may be attributable to the applicability of item content to dance, due to the dancers having trouble with "identifying an emphasis on outperforming." Quantitative indicators of performance (e.g., height, speed, distance, and goals scored) that can be used to judge performance relative to others are not as evident in dance as they are in sport. Thus, dancers may have difficulty judging whether they have outperformed others as performance judgments may often be based more on quality. They also assert that Intra-Team rivalry may be less relevant to vocational dancers than to sports team members. However, more recent research with dancers (21) reports that one reason dancers dropped out of England's Centres for Advanced Training (CATs) program was poor friendships, the cause of which was often related to competition or rivalry between dancers and teacher favoritism. Thus, one of the aims of the current study was to explore whether the item content within the IntraTeam Member Rivalry subscale and the other sub-dimensions captured by the measure are understandable, meaningful, and relevant in dance settings.
Following CFA, Quested and Duda (3) removed the Intra-Team Member Rivalry dimension from the analyses in their study. The Intra-Team Member Rivalry subscale was also omitted in subsequent research with dancers in CATs (9,10) and community dance. (11) Without the Intra-Team Member Rivalry items, Nordin-Bates and associates (9) reported that with minor modifications (the addition of three correlations between error terms), CFA supported the construct validity of the PMCSQ-2 with CAT dancers. On the one hand, removal of items that were not accurately measuring the constructs they were intended to measure would have strengthened the validity of the PMCSQ-2 with the sample of dance students in Quested and Duda's study. (3) However, by dropping the Intra-Team rivalry construct researchers have changed the measure so that it cannot assess the motivational climate as originally intended. (9-11) The arbitrary selection of factors increases the indeterminacy (lack of clarity and continuity) in the factors (22) and hence weakens the validity of the measure. Moreover, by omitting the Intra-Team Member Rivalry subscale these studies have failed to measure an important feature originally revealed to contribute to the prevalence of ego-involving climates in competitive team sport contexts. (4) It should be noted, however, that the underlying AGT dimensions and constructs identified by Newton and colleagues (4) were derived from research with athletes participating in competitive team sports. Although the six subscales of the PMCSQ-2 are conceptually similar to the dimensions of the motivational climate suggested by Ames (12) and Seifriz and colleagues, (17) deviations from the proposed dimensions did emerge. (4) These deviations, for example, rivalry being "Intra-Team," may be specific to competitive team sports and thus may not be directly applicable to other activities (e.g., dance). Hence, the present study aimed to examine whether the item content within the subscales of the PMCSQ-2 are relevant and meaningful in dance.
Purpose and Aims
The results of research are only as valid as the measures used. (20) Therefore, it is of critical importance that measurement of the motivational climate in dance be robust and appropriate in order for future researchers to confidently examine key social-environmental predictors of dancers' optimal development, performance, and well-being. Taking a critical realist perspective, (23) this study seeks to gain insight into what we are actually measuring with the PMCSQ-2 in dance settings. Critical realism places strong importance on attaining an adequate conceptualization of the phenomena under investigation. Different methodological approaches can provide diverse perspectives and insight into the same material. Previous quantitative psychometric assessments of the factorial validity of the PMCSQ-2 (such as the CFA conducted by Quested and Duda (3)) have highlighted the items, factors, and constructs that did not perform as intended. However, qualitative evaluation of the suitability of the PMCSQ-2 for use in dance settings enables examination as to why this may be the case. Thus, the goal of the current study was not to seek empirical generalization of findings to a population. Instead, the qualitative nature of the study lends itself to the goal of examining the content validity of the measure and developing an in-depth understanding of the problems dancers may experience when responding to the PMCSQ-2.
In summary, psychometric assessments, such as the PMCSQ-2, rely on respondents interpreting the meaning of the questions as intended. (24) However, previous research (3) has questioned whether the wording, content, and subscales of the adapted PMCSQ-2 are contextually relevant to dance. Thus, the first aim of the present study was to examine the content validity of the 33-item PMCSQ-2 (as adapted for dance by Quested and Duda (7)) in dance contexts. More specifically, the focus was to identify any problematic items and the nature and frequency of such problems experienced by dancers while completing the measure. Secondly, this study aimed to determine whether the content of the items within the subscales of the PMCSQ-2 are understandable, relevant, and meaningful when applied to dance.
Participants were an opportunity sample of 21 dancers (10 male, 11 female) from the UK. Dancers ranged from 12 to 36 years of age (Mean: 22.71, SD = 6.34). The inclusion criteria were that dancers needed to be at least 10 years old and able to verbalise their thoughts. A heterogeneous sample was recruited to gain a view of the dance population as a whole. Five classifications of participants were included as follows: recreational non-performance (N = 3; attend open dance classes but do not take part in any dance performances or exams), recreational performance (N = 4; attend a private dance school or group and take part in dance exams or performances), academic (N = 3; study dance at college or university), vocational (N = 6; study dance at a performing arts school or conservatory), and professional (N = 5; perform professional work either freelance or for a company on a regular basis). The majority of dancers (81%) reported that they participated or performed in more than one dance style, including ballet, contemporary, jazz, tap, street, hip hop, modern, African, and tango. They had danced for an average of 12.57 years (range = 2 to 24, SD = 6.14) and spent on average 18.08 hours (range = 2 to 40, SD = 13.89) dancing per week.
To achieve the aims of this study, a "think aloud" (25,26) method was employed. This method requests participants to verbalise all their thoughts overtly as they complete a questionnaire. (26) The participants are not asked to explain the reasoning behind their thoughts but just to say aloud what they are thinking about as they work through the items. The process underlying individuals' responses to items when completing a questionnaire is assumed to be multi-dimensional, consisting of a number of component phases. (27) The integrated model of the components of response categories, as outlined by Holland and coworkers (28) (Table 1), combines all five component phases (understanding, interpretation, retrieval, judgment, and responding) originally identified by Tourangeau and associates. (27) The "think aloud" responses of participants could be used to investigate the processes by which participants completed the PMCSQ-2, identifying problems experienced by respondents within each of the component phases of response.
The think aloud method has been used by a number of researchers to explore difficulties experienced by individuals while completing self-report questionnaires. (28-31) For example, Holland and coworkers (28) used the think aloud method to investigate young athletes' (aged 14 to 17) experiences of completing the Behavioural Regulation in Sport Questionnaire. (32) They found that participants had problems with understanding some of the terms in the items (e.g., "obligated"). Specifically, the wording of some of the items was found to be beyond the reading ability of the young participants. In another study, (29) researchers used the think aloud method to examine the nature and extent of problems that adults experience when completing a theory of planned behaviour (TPB) questionnaire. (33) French and colleagues (29) report the main problems experienced by participants while completing the TPB questionnaire included trouble with retrieval of relevant information and answers addressing different questions from those intended by the researcher. These insights into the problems of questionnaire response would not have been gained through the use of quantitative psychometric assessment of the measure alone. The findings can then be used to enhance wording and construction of instruments, to improve the reliability of the measure (e.g., by refining ambiguously worded items), and validity (e.g., by addressing the relevance and clarity of items; Knafl and associates (34)).
Prior to commencement of the study, ethical approval was gained from the ethics board of a large UK university. Potential participants were given information sheets, including written indications that participation was voluntary, that interviews would be audio-recorded, and that all data would be kept confidential and any quotes included in analysis would be made anonymous. All participants completed written consent forms before taking part in the study. For dancers less than 16 years of age, parental consent was obtained prior to inviting the dancer to take part. Demographic information (including age, gender, dance styles studied, years of dance experience, and hours spent in dance classes each week) was obtained from all participants via completion of a short questionnaire.
All think aloud interviews were conducted in a quiet room at a convenient location for the participant. Before beginning the first investigator read each participant standardized instructions adapted from Green and Gilhooly. (26) The instructions explained that participants should verbalize their thoughts constantly while completing the questionnaire, they should not plan what to say but just act as if they were alone in the room talking to themselves, and that there was no right or wrong way to think. Dancers were then given a practice task to familiarize them with the think aloud method, which required them to complete five items while thinking out loud. The five items were randomly selected from another measure of the social-environment (the Health Care Climate Questionnaire, (35) adapted for dance (3)), which has a similar response format to the PMCSQ-2. The researcher dealt with any questions at that time. If they did not do so during the practice, dancers were asked to read the questions aloud first before answering. To minimize influence the researcher sat out of the line of sight of the participants and remained silent unless the participant stopped talking for longer than 20 seconds, at which time the researcher asked the participant to "please keep talking." The interviews lasted approximately 30 minutes each.
Materials and Measures
Dancers completed the 33-item PMCSQ-2 (4) adapted for dance. (7) The stem "In this dance school/company/ group ..." was used. The PMCSQ-2 has six sub-scales: three measuring task-involving aspects of the motivational climate (Important Role, e.g., "Each dancer has an important role"; Cooperative Learning, e.g., "Dancers help each other learn"; and Effort and Improvement, e.g., "Dancers feel successful when they improve") and three measuring ego-involving features (Punishment for Mistakes, e.g., "The teachers get mad when a dancer makes a mistake"; Intra-Team Member Rivalry, e.g., "The teachers praise dancers only when they outperform other dancers"; and Unequal Recognition, e.g., "The teachers have their own favorites"). Dancers responded to items on a 5-point Likert scale from 1 (strongly disagree) to 5 (strongly agree).
Firstly, descriptive statistics (Table 2) were calculated to determine whether the participants' responses were comparable to those of previous research and that the think aloud process did not interfere with or change the way participants responded to the questionnaire. Secondly, the audio recordings of all interviews were transcribed verbatim and segmented into text relating to each item in the PMCSQ-2. In total there were 639 segments of text (21 segments per item). Participants' responses were analyzed thematically using QSR NVIVO software (version 10). Following recommendations by Someren, Barnard, and Sandberg, (36) each item segment was individually analyzed to identify if the participant experienced any problems in answering. The integrated model of the components of response, as outlined by Holland and colleagues (28) (see Table 1), was used as a coding framework. Each segment of text was coded as having either 1. no problems, 2. an issue that could be categorized into one of the five component phases of response, or 3. insufficient thinking aloud (the participant seems to experience problems when answering the item but does not give enough information to identify the reason). For each segment of text, there could be more than one problem recorded. For example, if a participant interpreted the meaning and content of the question in a way that was not as it was originally intended, then this was noted as a problem with interpretation. Within the same segment of text if the participant had trouble selecting an appropriate number on the provided Likert-scale, this would be categorized as a problem with responding. Consistent with previous think aloud protocols, (30,31) the item response category selected (i.e., agree or disagree) on the questionnaire was taken into account during analysis of participants' think aloud responses. All transcripts were coded by the main investigator. To establish reliability of the coding system, five think aloud transcripts were selected at random for independent coding by the second investigator and another trained rater. Inter-coder reliability for the five think aloud transcripts was determined using Krippendorffs alpha. (37) Krippendorffs alpha coefficient is a statistical measure of the extent of agreement among coders, which can be used with any number of coders, variables, and levels of measurement. (38) Krippendorff recommends a > .800, with [alpha] [greater than or equal to] .667 being considered but only tentative conclusions drawn. In this case, Krippendorff's alpha ranged from .73 to .84, suggesting that inter-rater reliability was generally considered acceptable. Following the independent coding, any segments of text that more than one coder noted as a specific problem were discussed and a code agreed upon by all raters.
Once all segments of the text had been coded, the frequency and nature of the problems experienced by participants while completing the PMCSQ-2 was examined. The total number of participants experiencing problems, the number of problems by component phase of response, and the total frequency of problems were calculated (Table 3). The subscale with the highest average number of participants experiencing problems per items was identified, and the total overall number of problems experienced by dancers while completing the measure was recorded. The data were further analyzed thematically (39) to identify recurrent and salient themes in relation to why particular items may not be performing as intended and whether the item content within the subscales of the PMCSQ-2 are understood, relevant, and meaningful in dance contexts. Themes are reported in the Results section in relation to each component phase of questionnaire response. Quotes were selected to illustrate the themes. The participant's number (e.g., P1), dance level (e.g., vocational), and age were presented to contextualize each quote.
Descriptive statistics (Table 2) revealed that the dancers generally perceived the motivational climate created by their teachers to be more task-involving than ego-involving. All mean scores for task-involving subscales were above the mid-point, and all mean scores for ego-involving subscales were below the mid-point. Both higher-order scales and all three task-involving subscales demonstrated acceptable internal consistency, with Cronbach's alpha >.70. In terms of the ego-involving subscales, the Unequal Recognition subscale was found to be internally consistent, but the alphas for the Intra-Team Member Rivalry ([alpha] =.67) and Punishment for Mistakes ([alpha] =.67) subscales were considered to be within the lower level of acceptability for established scales with few items. (40) The directions of the bivariate correlations between the PMCSQ-2 subscales and higher order factors correspond to those reported by Newton and coworkers. (4) The findings revealed that the participants responded to the measure in a way comparable to previous research, with mean scores for the subscales, alpha values, and bivariate correlations being similar to those of previous research with dancers. (3,7,11)
Think Aloud Interview Results
Analysis of the think aloud interviews revealed that out of the 693 responses to the items from participants, the majority (72%) presented no problems. However, no participant completed the whole measure without experiencing some difficulty or confusion. The number of problems per person ranged from 1 to 25 (Mean: 8.81). Problems identified are presented in relation to the five component phases of questionnaire response (understanding, interpretation, retrieval, judgment, and responding). Interpretation of items was the most frequent problem, and understanding the item was the least common problem experienced.
Table 3 presents the frequency and nature of problems identified by item. The most problematic item was "Dancers are not selected for the best roles if they make mistakes," which resulted in a total of 17 problems experienced by 13 (62%) of the participants. There was no item that was found to be completely unproblematic. Of all the subscales, Intra-Team Member Rivalry was found to have the highest frequency of problems per item, with an average of nine dancers (43%) experiencing problems with items on this subscale.
Less than 1% of problems identified were due to dancers having difficulty comprehending the language used. Only two participants, both of whom spoke English as a second language, experienced problems with understanding. For example, in response to the item "The teacher praises dancers only when they outperform other dancers," a professional dancer (P23, aged 34) said: "When they outperform other dancers. I don't understand this question so I skip."
Half of all problems identified (50%) were related to the misinterpretation of items. Thirteen dancers had trouble interpreting items in relation to the general motivational climate operating in their dance school, company, or group, as instructed in the stem. The most problematic item, with six dancers not interpreting it in relation to their current dance situation, was "If you want to be cast for the best roles you must be one of the best dancers." Four participants interpreted this item in relation to other dance contexts (e.g., previous dance groups or schools with which they had performed). Other items (1, 6, and 9 in Table 3) were also appraised by dancers (N = 3) in relation to contexts other than the dancers' current one. For example, a recreational non-performance dancer interpreted the item "Dancers at all skill levels have an important role in performances" in relation to her experiences from when she was younger and in a different dance setting from her present one: "Yes, definitely. I mean, when I used to dance in a school when I was really young ... I was like four or something and I still had something to do [Agree]" (P17, recreational non-performance, aged 20).
Six dancers expressed their own opinions and values rather than reflecting on those emphasized by the teacher-created climate. For example, in response to the item "Dancers feel good when they try their best," a dancer (P1, recreational performance, aged 27 years) said, "Do you mean in terms of myself, so I would feel good when I try my best, or the teacher makes us feel good when we try our best? OK so, I'm going to take it that that means based on me." Two dancers interpreted the items in relation to their own teaching practices rather than the climate in which they participate as a dancer: ". when I am teaching, it doesn't matter how technical you are. It doesn't matter how good you are in the studio. You always give attention to the student who is trying to be their best. So yes, the teachers always emphasize trying your best. I think that's a good thing and that, yes, I agree [Agree]" (P23, professional, aged 34).
Dancers also experienced problems with the interpretation of certain words and phrases, such as "important role " (N = 6), "rewarded" (N = 6), "best dancers" (N = 5), and "mistakes" (N = 3).
Six dancers had trouble with interpretation of the construct of important role in dance contexts. Of those, five dancers' responses highlighted issues of discriminant validity, with their answers being more related to the construct of cooperative learning. They misinterpreted the question to be about working cooperatively with others and helping other dancers out, as opposed to every dancer having an important role. For example, in response to the item "Each dancer contributes in some important way," a professional dancer (P27, aged 26) responded: "Yes, definitely. Everyone helps everyone. Everyone learns from everyone [Strongly Agree]." Furthermore, one of the six dancers questioned how to interpret the concept of important role: "So ... I'm wondering what do you mean with important role, for what? For the ensemble or for ... so um, I guess for the group dynamic?" (P15, academic).
Six dancers had problems interpreting the item "Trying hard is rewarded in rehearsals and performances." Two dancers questioned the meaning and context surrounding the term rewarded: "I wonder rewarded in which way?" (P15, academic, aged 24). Three of the dancers referred to providing themselves with reward rather than being rewarded by the climate in which they were operating. For example, one dancer said: "I think that the reward comes from yourself. So, you end up feeling good [Agree]" (P1, recreational performance, aged 27). A young vocational dancer (P33, aged 12) interpreted the item in relation to the long-term extrinsic rewards associated with effort and improvement in a professional dance career: "I strongly agree with that one as then, if you are in the corp de ballet you may be offered a soloist job if you try your best everyday [Strongly Agree]." While potentially relevant to a task-involving climate, both these items highlight that the dancers were not formulating their responses based on whether the dance school specifically encouraged effort and improvement, as the item intends.
Six dancers had trouble with interpretation of the "best" or "top" dancers. For example, in response to an item from the Punishment for Mistakes subscale ("If you want to be cast for the best roles you must be one of the best dancers"), a vocational dancer questioned, "What is the best dancer? Every piece or performance suits someone different. So you might be the best technician but not necessarily the best suited to a particular piece or casting. Um ye ... I'll put that in the middle [Neutral]" (P9, vocational, aged 23). Another dancer in response to the Unequal Recognition item "Only the best dancers get praise" said: "I think, best, the best dancer is not really, um, the right way of describing, because what is the best dancer? What, what defines a good dancer anyway? So this question is strange. But, yes dancers that seem to be serious about their work, whatever it is, they get, um, acknowledged and recognized I guess. But, I refuse to answer this question as I don't know what you mean with best dancer [left blank]" (P15, academic, aged 24).
One dancer interpreted "best dancers" to mean those that try the hardest. For example, in response to the item "Only the best dancers get praise," a young vocational dancer (P31, aged 12) said, "They when they, um, do really, really well like noticeably well, then the teacher will give you loads of praise because she's happy that you've worked hard and got better. Um, 4 [Agree]." The item "Only the best dancers get praise" is intended to tap the construct of unequal recognition, a dimension of ego-involving climates. However, in the above example, the young dancer appears to misinterpret the item to capture the concept of effort and improvement, which is relevant to task-involving climates.
Three participants had trouble interpreting the purpose of the use of the word "mistakes" within the item context due to multiple possible meanings. For example, in response to the item "Dancers are afraid to make mistakes," a vocational dancer thought aloud, "To make mistakes I think ... I don't understand so much, err, making mistakes. Does it mean like physical or does it mean, er, like general like. I'm not sure [Agree]" (P19, vocational, aged 22). The same dancer when answering the item "Dancers are not selected for the best roles if they make mistakes" explained ".well in a dance school environment you can make loads of mistakes. You can like make mistakes by not turning up every day. But the people that didn't turn up every day still managed to get the roles so. kind of ... kind of doesn't apply really like. no, so I would disagree with that one [Strongly Disagree]."
Nine percent of the problems were due to dancers having trouble gathering the necessary information from their memory or value system to form an appropriate answer. The majority (64%) of issues with retrieval were experienced by recreational non-performance dancers. For these dancers, problems with retrieval appeared to be mainly due to a lack of experience in dance performance. For example, three dancers had trouble retrieving an appropriate memory when answering the item "If you want to be cast for the best roles you must be one of the best dancers," as they had not experienced being cast for roles in their dance school, company, or group. One dancer stated, "If that's a question about my dancing now, we aren't necessarily being cast for any roles. Oh. I don't know. Oh well [Strongly Agree]" (P3, recreational performance, aged 23).
Nineteen percent of problems were due to participants having trouble using the information retrieved from their memory or value system to make a judgment as to what response to give. The most problematic item in terms of judgment was "Dancers are 'fired up' (positively excited) when they perform better than their fellow dancers in a performance." Three dancers struggled to make a judgment due to the varied nature of performance in dance. For example, a young vocational dancer (P33, aged 12) said, "I think that one's neutral, as it all depends what sort of performance you are in really [Neutral]." Another dancer offered, "If you are solo, you don't compete with anybody. So, dancers get fired up when they perform better in classes than the other ones, yes, you get a little 'yes I'm doing good,' but not in performances, so I would say that I don't have an answer to that question so that's a 3 [Neutral]" (P29, vocational, aged 22). Three dancers experienced trouble making a judgment about whether they outperformed other dancers because of not being able to see their fellow dancers perform. For example, one dancer stated, "Normally, you wouldn't know [be]cause you're on stage at the same time" (P9, vocational, aged 23), and another also commented, "You aren't able to kind of see the other person's technique to be able to be excited, fired up [Neutral]" (P5, recreational performance, aged 19). Two recreational non-performance dancers had trouble making a judgment as they felt that the context of the question was not relevant for them. One thought aloud: "I mean, I don't perform, so it's hard to say, but I'm sure it's a bit of a team effort. So a bit neutral on that one [Neutral]" (P17, recreational non-performance, aged 20).
Thirteen percent of problems were due to participants experiencing difficulty with the questionnaire's response format and not selecting an appropriate number on the provided Likert scale. Sixty-six percent of these problems were encountered by one particular dancer. His choice of category on the Likert scale did not match up with what he verbalized for 22 of the items. For example, in response to the item "Dancers are not selected for the best roles if they make mistakes," he said, "So I will put strongly agree" but selected strongly disagree on the questionnaire. Four other dancers' choice of response category also did not match up with what they thought aloud. Some dancers found the response categories to be restricting, with three participants selecting both Neutral and Agree and another dancer thinking aloud "I would like to say 3 and a half but as I cannot say it then, let's just say 3 [Neutral]" (P29, vocational aged 22).
Employing a "think aloud" methodology, (25,26) the current study aimed to examine the content validity of the 33-item PMCSQ-2 as adapted for dance. (7) More specifically, the first aim was to identify any problematic items and the nature and frequency of such problems experienced by dancers while completing the measure. Think aloud interviews revealed 72% of the participant responses to the items presented no problems, indicating the measure to have a degree of content validity. However, in the case of 28% of item responses, problems were encountered in relation to at least one of the five component phases (understanding, interpretation, retrieval, judgment, and responding) identified by Tourangeau and associates. (27)
The second aim of the study was to examine whether the item content of the sub-dimensions of task- and ego-involving climates as captured in the subscales of the PMCSQ-2 are understood, meaningful, and relevant in dance contexts. Overall, the results supported the relevance of the subdimensions. The findings revealed that the majority of dancers are able to relate to and understand the constructs of cooperative learning, effort and improvement, unequal recognition, and punishment for mistakes. However, the wording of some of the items may not be appropriate for accurately capturing the key constructs of important role and Intra-Team member rivalry as construed in dance. Although this study focused specifically on the use of the PMCSQ-2 in dance settings, only a small percentage (5%) of the problems identified were purely dance specific. This is encouraging in terms of evaluating the potential utility of sport-based measures in dance or at least this particular sport-referenced instrument.
Misinterpretation of Context
The main problem experienced by participants, while completing the measure, was the misinterpretation of items. Many participants (62%) interpreted items in relation to their own opinions or their experiences in or views regarding the status quo in other settings. In the case of the PMCSQ-2 this was not appropriate, as the questionnaire items were specifically intended to provide an assessment of the general motivational climate currently operating in the dancers' school, company, or group (i.e., the focus of the stem). In order for researchers concerned with motivational processes in achievement settings to examine relations between individuals' perceptions of the motivational climate and variables of interest, it is of critical importance that participants interpret questions in relation to the particular context being examined. Otherwise, the assessments taken and subsequent results of such studies are compromised. Consider, for example, a study that aims to examine correlations between individuals' goal orientation and their perception of the motivational climate. If participants respond to PMCSQ-2 items in relation to their own preferences and beliefs as opposed to those operating in their surroundings, this could lead to inflated correlations between goal orientations and climate perceptions.
A possible explanation for the misinterpration of items is that the instructions may not have been stated clearly and adamently enough. Although the stem, instructing dancers to interpret items in relation to the general motivational climate operating in their dance school, company, or group, was repeated at the top of each page, it may be that stronger directions clearly indicating that all items refer to actions and attitudes of the teachers and leaders are needed. Another possibility is that although the PMCSQ-2 aims to capture the motivational climate created by the coach and teacher, the items do not always clearly relate to the "leader." Only 14 of the 33 items in the PMCSQ-2 contain the words "the coach/ teacher." This may explain why in the current study some of the dancers did not know whether to interpret items in relation to the behaviour of their teacher(s) or their own views and opinions. For example, in reponse to the item "In this dance/school/ company or group dancers feel good when they try their best," one dancer questioned whether the item meant in relation to herself (i.e., whether she feels good when she tries her best) or the teacher (i.e., the teacher makes her feel good when she tries her best). The dancer made the decision to interpret the item as based on herself, thus misinterpreting it and not capturing the teacher-created motivational climate.
The lack of clarity in relation to the items tapping the teacher-created climate may also be the possible cause of dancers experiencing problems with regard to interpreting certain terms such as "reward." For example, when answering the item "Trying hard is rewarded in rehearsals and performances," three dancers misinterpreted the item to refer to providing themselves with reward rather than being rewarded by the climate in which they were operating. Thus, the dancers were not responding based on whether the teacher(s) specifically encouraged effort and improvement, as the item intends. It is recommended, therefore, that future research employing the PMCSQ-2 provide strong, explicit, and repeated directions or reword the PMCSQ-2 items to start with "My teacher(s)/coach(es)/leader(s). ..."
Relevance of Item Content
For three of the dancers, the misinterpretation of items was due to the content of the question not being relevant for them. The PMCSQ-2 was originally designed for use in competitive team sport settings and then reworded by Quested and Duda (7) for elite and vocational dance contexts. For example, in their revision, the word "game" was replaced with "performance," as taking part in performances is an integral part of elite and vocational dancers' training. Hence, the PMCSQ-2 (adapted for dance) contains nine performance-related items. It is important to note, however, that dance is a diverse activity that can be individualistic, group or team based, and competitive or non-competitive. Furthermore, the occurrence and nature of performances vary depending upon the situation in which they take place. The findings of this study revealed that recreational non-performance dancers have trouble interpreting items related to performance within their range of interest, and they also experienced problems retrieving relevant memories. Instead, they tended to answer in relation to past performance experiences in other dance contexts, or what they believed might happen when dancers do perform. Based on the findings of this study, it would seem that the performance-related items (e.g., "Dancers at all skill levels have an important role in performances.") are not relevant or appropriate for community, social, or recreational dance groups or classes, where public performance is not an integral focus. One possible solution to this issue for the six items that refer to performance (2, 3, 9, 10, 17, and 23 in Table 3) could be to remove the reference to rehearsals and performances and capture the relevant feature of the environment without contextualizing it within performance. For example, "Trying hard is rewarded in rehearsals and performances" could simply read, "Trying hard is rewarded." However, three of the items that refer to performance (1, 5, and 19 in Table 3; e.g., "If you want to be cast for the best roles you must be one of the best dancers.") can only be contextualized in performance-focused dance contexts, and it may not be possible to alter these and retain relevance to all dance environments. For example, in community, social, or recreational dance settings dancers are not cast for roles within the group or class. To address this issue, future research could employ focus groups with dancers from community and recreational non-performance activities to generate possible alternative items that more accurately reflect the underlying constructs (i.e., Punishment for Mistakes and Unequal Recognition) in nonperformance centered dance settings.
It is important to note that the results of previous research (11) employing the PMCSQ-2 (adapted for dance) with community dancers should be interpreted with caution. Although the alpha values were acceptable for the higher order factors (task-involving a =.89, ego-involving a = .86), the quantitative nature of the study does not qualitatively capture any difficulties (misunderstandings, etc.) experienced by respondents. As the participants in the Norfield and Nordin-Bates (11) study were community dancers, some may not take part in performances, and thus might have experienced problems responding to performance-related items. Furthermore, researchers adapting the PMCSQ-2 for use in other contexts (41) that are not always competitive or team-based in nature (i.e., exercise, physical activity, individual sports) should consider the relevance of items, in particular those contextualized within team game and performance situations.
Dancers experienced problems with regard to interpreting certain terms such as "reward," "important role," "mistake," and the "best" or "top" dancers. This may be due to "vague concepts" (27) that could have multiple meanings. For example, the wording of some of the items from the Unequal Recognition (e.g., "Only the best dancers get praise.") and Punishment for Mistakes (e.g., "If you want to be cast for the best roles, you must be one of the best dancers.") subscales leaves the interpretation of what constitutes the "best" dancer up to the respondent. In this study, the ambiguity of the term led to six dancers not being clear what the question was asking, with one dancer refusing to answer the question as a result of not knowing what criterion to use to interpret "best dancer." The measure is assessing dancers' perceptions of the teacher-created motivational climate; hence, it is the dancers' perceptions of who the teacher favors or considers to be the "best" dancer that is of interest. Thus, it could be argued that it does not matter how the dancer interprets "best" dancer as long as the criterion is one that the teacher could use to create a hierarchy of unequal recognition. For example, one dancer interpreted the "best" dancer from the item "Only the best dancers get praise" to mean those that tried the hardest. The ambiguity of the term could lead to the item being misinterpreted to capture the construct of effort and improvement as opposed to unequal recognition based on ability differences. However, if the teacher indeed creates a hierarchy based on effort (or whatever criterion the dancer uses to interpret "best" dancer), then issues of unequal recognition could still be present. In this case, the participant's thoughts articulated out loud did not make clear whether she felt that the teacher created a hierarchy based on effort. Follow-up questions (probes) from the interviewer could have clarified the respondent's interpretation of the item. However, the use of verbal probing is considered more invasive and artificial compared to the traditional think-aloud protocol in which participants answer the questionnaire by themselves. (42) Therefore, probes were not used in the current study to minimize any social contact between the researcher and participant that could interrupt or effect the respondent's ongoing thinking process. (43) The traditional think-aloud protocol can provide insight into what problems participants experienced while completing the questionnaire but not always why. Verbal probing is advantageous in that it gives the researcher control to focus the discussion and further explore why particular items or issues may be potential sources of response error. (43) Thus, future research using the think aloud method could consider the use of scripted or unscripted probes to gain greater insight into the questionnaire response process. (44)
In the current study, a number of items from the Important Role subscale appeared to lack conceptual clarity. For example, five dancers' responses were related to the construct of cooperative learning, rather than capturing the extent to which dancers' feel important and that their contributions are valued within their groups. This finding is in line with previous research undertaken with vocational dancers (3) and athletes, (4) which has reported the item "Each dancer feels as if he/she is an important team member" to be problematic, as it loaded poorly onto the Important Role factor and cross-loaded with Co-operative Learning. In the original development of the PMCSQ-2, the aforementioned item was added to the Important Role subscale without first testing its face validity. (4) Newton and colleagues (4) suggested that it makes conceptual sense that this item could also be construed as an aspect of cooperative learning because it captures the value of being "an important team member." To address this issue, future research could conduct focus groups with both athletes and artistic performers that would explore the different ways in which the construct important role is operationalized in various achievement settings. Alternative items could then be generated based on the findings of the focus groups. The content and construct validity of the items should be examined, and the item found to be the most valid could then be used in future versions of the PMCSQ-2.
Intra-Team Member Rivalry
The results revealed the Intra-Team Member Rivalry subscale to be the most problematic in terms of frequency of problems per participant. The item "Dancers are 'fired up' (positively excited) when they perform better than their fellow dancers in a performance" was found to be particularly challenging. It appeared from the think aloud data that participants struggled with making a judgment as to what response to give to the item due to the varied and subjective nature of dance performance. For example, a young vocational dancer stated, "it all depends what sort of performance you are in really." Other responses indicated that in some dance contexts (e.g., a solo performance), there may not be any fellow dancers to compare their performance against. Also, in group performance, dancers may not be able to see their fellow dancers due to being off stage or in a different section of the performance, whereas in team sports players are all visible to each other during a game.
Previous studies with vocational dancers (3) report that the "fired up" item and another from the Intra-Team Member Rivalry subscale, "Dancers are encouraged to outperform the other dancers," cross-load onto other factors. The findings of the current study suggest that these issues of cross-loading may be due to the content of the Intra-Team Member Rivalry subscale items not being fully understood, relevant, or meaningful within dance settings. This finding is in contrast to Quested and Duda's (3) proposition that these cross-loadings may be attributable to dancers having trouble identifying when they outperform other dancers. In the present study, for example, a recreational performance dancer articulated out loud that she got "fired up" when she performed better than her fellow dancers "in class, but not in performance" (P3, recreational performance dancer, aged 23). Hence, rivalry between dancers may occur if teachers encourage comparison with others in class. (21) The subscale of Intra-Team Member Rivalry is, then, relevant and meaningful to dance, but the findings of the present study suggest that the wording of the items in the PMCSQ-2 do not accurately capture the construct of rivalry and interpersonal comparison in dance.
Issues with the content validity of items on the Intra-Team Member Rivalry subscale may also apply to other achievement contexts. Previous research in sport (4,6) and physical education (45) has reported that the Intra-Team Member Rivalry items lack internal consistency. Thus, future research investigating how rivalry is manifested in the different achievement contexts and whether the IntraTeam Member Rivalry items in the PMCSQ-2 fully capture this construct as it is construed in sport, physical education, exercise, and dance would be valuable. Such research could employ focus groups to explore what words or phrases are used in achievement contexts to refer to rivalry or interpersonal comparison between individuals. The focus groups could also look into when athletes and dancers feel a sense of rivalry and under what sort of circumstances (e.g., does the occurrence of rivalry differ between class, rehearsal, audition, and performance contexts?). Furthermore, it would be interesting to know what criteria individuals use for normative comparison (e.g., most artistic or expressive, most athletic, most technical) and how this plays out in the social environment. A review of the physical, psychological, social, and interpersonal factors relevant to talent identification in dance proposes that talent should be considered from a multi-dimensional perspective, as strengths in some areas may compensate for weaknesses in others. (46) Thus, normative comparison may be a dynamic process in which a range of criteria are used to judge success, talent, or performance. Findings of the focus groups may aid the generation of alternative wording that is more appropriate for capturing the construct of Intra-Team member rivalry and interpersonal comparison in achievement contexts.
Limitations and Future Directions
This is the first study to employ a "think aloud" methodology to examine the appropriateness of any dance psychology related measure. Hence, it provides important insights into the process of questionnaire response; however, it is not without limitations. A heterogeneous sample was recruited to reflect the fact that the PMCSQ-2 (adapted for dance) has been employed with dancers from a variety of dance situations. Although a total sample size of 15 to 25 participants is considered to be appropriate for think aloud studies, (43) we recognize that there was a small representation within each classification of participation. Research with larger sample sizes within each dance setting would need to be conducted before robust conclusions about the content validity of the PMCSQ-2 with different populations of dancers can be reached.
With the majority (72%) of dancers' responses to items presenting no problem and as only a small percentage (5%) of the problems identified were dance specific, the findings provide potential support for the utility of the PMCSQ-2 in dance settings. With any questionnaire data, there is likely to be "noise" based on each participant interpreting one or more items incorrectly. Thus, a few issues of minor magnitude are expected with any questionnaire. The potential impact of this "noise" on the results can be reduced by the use of very large samples. However, the development and validation of measures is an ongoing process, and the findings of this study have also highlighted a number of potentially problematic areas that warrant further attention.
The main problem that arose was participants not interpreting items in relation to the motivational climate created by the teachers in their current dance, school, company, or group. The findings also brought into question the relevance of the content of some of the items for non-performance contexts and the ability of the wording of a few items to accurately capture the constructs of important role and rivalry and interpersonal comparison. This study has implications for the interpretation of previous research and the conduct of future research employing the PMCSQ-2 in dance as well other achievement activities (e.g., sport).
In order to improve the validity and reliability of measures for examining the relationships between motivational climate and individuals' optimal functioning, development, and well-being, it is vital to reduce any problems that participants may experience while completing those measures. Based on the findings of the present study, it is recommended that future research employing the PMCSQ-2 provide strong, explicit, and repeated directions, or rewording of the PMCSQ-2 items to start with "My teacher/coach/leader ..." to reinforce the context in which the item should be interpreted. In addition, further research is required to examine issues with item content causing a lack of conceptual clarity within the Important Role and IntraTeam Member Rivalry subscales. Such research could employ focus groups with both athletes and performers from a variety of participation levels and settings. These groups could be used to explore the different ways in which the AGT constructs (e.g., Important Role and Intra-team member rivalry) are operationalized within and across different types of achievement settings, potentially generating alternative items that more generally capture the underlying constructs.
Jennie E. Hancox, M.Sc., Ph.D., and Eleanor Quested, M.Sc., Ph.D., School of Psychology and Speech Pathology, Faculty of Health Sciences, Curtin University, Perth, Australia. Joan L. Duda, Ph.D., School of Sport, Exercise and Rehabilitation Sciences, University of Birmingham, Edgbaston, United Kingdom.
This research was conducted while Jennie Hancox, M.Sc., Ph.D., and Eleanor Quested, M.Sc., Ph.D., were in the School of Sport, Exercise and Rehabilitation Sciences, University of Birmingham, Edgbaston, United Kingdom.
Correspondence: Jennie Hancox, M.Sc., Ph.D., School of Psychology and Speech Pathology, Faculty of Health Sciences, Curtin University, Perth, WA 6845, Australia; firstname.lastname@example.org.
(1.) Boyce A, Gano-Overway LA, Campbell AL. Perceived motivational climate's influence on goal orientations, perceived competence, and practice strategies across the athletic season. J Appl Sport Psychol. 2009;21(4):381-94.
(2.) Ntoumanis N, Biddle S. A review of motivational climate in physical activity. J Sports Sci. 1999 Aug;17(8):643-65.
(3.) Quested E, Duda JL. Exploring the social-environmental determinants of well- and ill-being in dancers: a test of Basic Needs Theory. J Sport Exerc Psychol. 2010 Feb;32(1):39-60.
(4.) Newton M, Duda JL, Yin ZN. Examination of the psychometric properties of the Perceived Motivational Climate in Sport Questionnaire-2 in a sample of female athletes. J Sports Sci. 2000 Apr;18(4):275-90.
(5.) Reinboth M, Duda JL. Perceived motivational climate, need satisfaction and indices of well-being in team sports: a longitudinal perspective. Psychol Sport Exerc. 2006;7(3):269-86.
(6.) Vazou S, Ntoumanis N, Duda JL. Predicting young athletes' motivational indices as a function of their perceptions of the coach- and peer-created climate. Psychol Sport Exerc. 2006;7(2):215-33.
(7.) Quested E, Duda JL. Perceptions of the motivational climate, need satisfaction, and indices of well- and ill-being among hip hop dancers. J Dance Med Sci. 2009;13(1):10-9.
(8.) Carr S, Wyon M. The impact of motivational climate on dance students' achievement goals, trait anxiety, and perfectionism. J Dance Med Sci. 2003;7(4):105-14.
(9.) Nordin-Bates SM, Quested E, Walker IJ, Redding E. Climate change in the dance studio: Findings from the UK Centres for Advanced Training. Sport Exerc Perform Psychol. 2012;1(1):3-16.
(10.) Walker IJ, Nordin-Bates SM, Redding E. Characteristics of talented dancers and age group differences: findings from the UK Centres for Advanced Training. High Abil Stud. 2011;22(1):43-60.
(11.) Norfield J, Nordin-Bates S. How community dance leads to positive outcomes: a self-determination theory perspective. J Applied Arts Health. 2012;2(3):257-72.
(12.) Ames C. Achievement goals and the classroom motivational climate. In: Meece J, Schunk D (eds): Students Perceptions in the Classroom: Causes and Consequences. Hillsdale, NJ: Erlbaum, 1992, pp. 327-348.
(13.) Nicholls JG. The Competitive Ethos and Democratic Education. London: Harvard University Press, 1989.
(14.) Duda JL. Achievement goal research in sport: pushing the boundaries and clarifying some misunderstandings. In: Roberts GC (ed.): Advances in Motivation in Sport and Exercise. Leeds, UK: Human Kinetics, 2001, pp. 129-182.
(15.) de Bruin AP, Bakker FC, Oudejans RRD. Achievement goal theory and disordered eating: relationships of disordered eating with goal orientations and motivational climate in female gymnasts and dancers. Psychol Sport Exerc. 2009;10(1):72-9.
(16.) Sarrazin P, Vallerand R, Guillet E, et al. Motivation and dropout in female handballers: a 21-month prospective study. Eur J Soc Psychol. 2002;32(3):395-418.
(17.) Seifriz JJ, Duda JL, Chi L. The relationship of perceived motivational climate to intrinsic motivation and beliefs about success in basketball. J Sport Exerc Psychol. 1992;14(4):375-91.
(18.) Duda JL, Whitehead J. Measurement of goal perspectives in the physical domain. In: Duda JL (ed): Advances in Sport and Exercise Psychology Measurement. Morgantown, WV: Fitness Information Technology, 1998, pp. 21-48.
(19.) Walling MD, Duda JL, Chi L. The Perceived Motivational Climate in Sport Questionnaire: construct and predictive validity. J Sport Exerc Psychol. 1993;15(2):172-83.
(20.) Messick S. Validity of psychological assessment: validation of inferences from persons' responses and performances as scientific inquiry into score meaning. Am Psychol. 1995;50(9):741-9.
(21.) Walker IJ, Nordin-Bates SM, Redding E. A mixed methods investigation of dropout among talented young dancers findings from the UK Centres for Advanced Training. J Dance Med Sci. 2012;16(2):65-73.
(22.) Velicer WF, Jackson DN. Component Analysis versus Common Factor Analysis: some issues in selecting an appropriate procedure. Multivariate Behav Res. 1990;25(1):1-28.
(23.) Bhaskar R. A Realist Theory of Science (2nd ed). Brighton, Sussex: Harvester Press, 1978.
(24.) Hughes AK. Comparing pretesting methods: cognitive interviews, respondent debriefing, and behavior coding. Presented at the Annual Meeting of the Federal Committee on Statistical Methodology. Arlington, Virginia, November 17-19, 2003. Available at: http://www.census.gov/srd/papers/pdf/rsm2004-02. pdf. Accessed October 5, 2015.
(25.) Ericsson KA, Simon HA. Protocol Analysis: Verbal Reports as Data. Cambridge, MA: MIT Press, 1993.
(26.) Green C, Gilhooly K. Protocol analysis: practical implementation. In: Richardson TE (ed): Handbook of Qualitative Research Methods for Psychology and Social Sciences. Oxford: BPS Blackwell, 1996, pp. 55-74.
(27.) Tourangeau R, Rips LJ, Rasinki K. The Psychology of Survey Response. Cambridge: Cambridge University Press, 2000.
(28.) Holland MJ, Sharp LA, Woodcock C, et al. Validity and reliability of the Behavioral Regulation in Sport Questionnaire (BRSQ) with youth athletes. J Sport Exerc Psychol. 2010;32:S175-6.
(29.) French DP, Cooke R, McLean N, et al. What do people think about when they answer theory of planned behaviour questionnaires? A 'think aloud' study. J Health Psychol. 2007 Jul;12(4):672-87.
(30.) Darker CD, French DP. What sense do people make of a theory of planned behaviour questionnaire?: a think-aloud study. J Health Psychol. 2009 Oct;14(7):861-71.
(31.) Kaklamanou D, Armitage CJ, Jones CR. A further look into compensatory health beliefs: a think aloud study. Br J Health Psychol. 2013 Feb;18(1):139-54.
(32.) Lonsdale C, Hodge K, Rose EA. The development of the Behavioral Regulation in Sport Questionnaire (BRSQ): instrument development and initial validity evidence. J Sport Exerc Psychol. 2008 Jun;30(3):323-55.
(33.) Ajzen I. The theory of planned behavior. Organ Behav Hum Decis Process. 1991;50(2):179-211.
(34.) Knafl K, Deatrick J, Gallo A, et al. The analysis and interpretation of cognitive interviews for instrument development. Res Nurs Health. 2007 Apr;30(2):224-34.
(35.) Williams GC, Grow VM, Freedman ZR, et al. Motivational predictors of weight loss and weight-loss maintenance. J Pers Soc Psychol. 1996 Jan;70(1):115-26.
(36.) Someren MWV, Barnard YF, Sandberg JAC. The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes. London: Academic Press, 1994.
(37.) Krippendorff K. Content Analysis: An Introduction to Its Methodology (2nd ed). Thousand Oaks, CA: SAGE, 2004.
(38.) Hayes AF, Krippendorff K. Answering the call for a standard reliability measure for coding data. Commun Methods Meas. 2007;1(1):77-89.
(39.) Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77-101.
(40.) Hair JF Jr, Black WC, Babin BJ, et al. Multivariate Data Analysis (6th ed). Upper Saddle River, NJ: Prentice Hall, 2006.
(41.) Huddleston H, Fry MD, Brown TC. Corporate fitness members' perceptions of the environment and their intrinsic motivation. Revista de Psicologia del Deporte. 2012;21(1):15 23.
(42.) Ericsson KA, Simon HA. Verbal reports as data. Psychol Rev. 1980;87(3):215-51.
(43.) Willis GB. Cognitive Interviewing--A Tool for Improving Questionnaire Design. Thousand Oaks, CA: SAGE, 2005.
(44.) Beatty PC, Willis GB. Research synthesis: the practice of cognitive interviewing. Publ Opin Q. 2007;71(2):287-311.
(45.) Gonzalez-Cutre D, Sicilia A, Moreno JA, Fernandez-Balboa JM. Dispositional flow in physical education: relationships with motivational climate, social goals, and perceived competence. J Teach Phys Educ. 2009;28(4):422-40.
(46.) Walker IJ, Nordin-Bates SM, Redding E. Talent identification and development in dance: a review of the literature. Research in Dance Education. 2010;11(3):167-91.
Table 1 Integrated Model of the Components of Response as Outlined by Holland and Colleagues (28) Coding Component Description Number 0 No problems The participant does not appear to experience any problems while completing the item. 1 Understanding The participant has problems comprehending the language used. For example, the participant is unable to read or understand the words and phrases used in the question. This may be due to a difficult grammatical construction or an unfamiliar word or phrase. 2 Interpretation The participant has trouble correctly comprehending sentences used in the question. For example, he or she incorrectly interprets the meaning and content of the question as it was originally intended (in line with the construct the item is trying to capture). Is the participant clear as to what the question is actually asking? Is the question misinterpreted or does the answer address a different question than the one that was asked? Is the content of the question relevant? 3 Retrieval The participant has problems gathering the necessary information from his or her memory or value system in order to form an appropriate answer. 4 Judgment The participant has trouble using the information retrieved from her or his memory or value system to make a judgment as to what response should be given to the question. 5 Responding The participant experiences difficulty with the questionnaire's response format and has trouble selecting an appropriate number on the provided Likert-scale. 6 Insufficient The participant seems to experience Thinking Aloud problems when answering the item but does not give enough information to identify the reason. The integrated model combines all five component phases (understanding, interpretation, retrieval, judgment, and responding) identified by Tourangeau and associates. (27) Table 2 Descriptive Statistics: Mean, SD, Cronbach's Alpha Coefficient, and Correlations Between the PMCSQ-2 Subscales and Second-Order Factors M SD [alpha] 1 2 1. Effort/Improvement 4.33 .75 .86 2. Important Role 4.02 .80 .75 .70 ** 3. Cooperative Learning 4.18 .77 .72 .77 ** .87 ** 4. Punishment for 2.79 .78 .67 -.27 -.50 ** Mistakes 5. Unequal Recognition 2.86 .90 .85 -.34 -.33 6. Intra-team 2.18 .83 .67 -.01 -.20 Member Rivalry 7. Task Climate 4.43 .75 .92 .91 ** .91 ** 8. Ego Climate 2.72 .74 .84 -.25 -.41 3 4 5 6 1. Effort/Improvement 2. Important Role 3. Cooperative Learning 4. Punishment for -.64 ** Mistakes 5. Unequal Recognition -.42 .64 ** 6. Intra-team -.28 .59 ** .67 ** Member Rivalry 7. Task Climate .93 ** -.43 -.37 -.10 8. Ego Climate -.49 * .86 ** .93 ** .79 ** 7 1. Effort/Improvement 2. Important Role 3. Cooperative Learning 4. Punishment for Mistakes 5. Unequal Recognition 6. Intra-team Member Rivalry 7. Task Climate 8. Ego Climate -.38 Table 3 Number of Participants Experiencing Problems, Types of Problems Identified for the 21 Respondents, and Frequency Total Displayed Per Item Number of Problems by Component Phase of Response (coding numbers Number of correspond to Participants those in Table 1) Experiencing Item Subscale Problems 1 2 3 4 1. Dancers are not PM 13 0 7 4 2 selected for the best roles if they make mistakes. 2. Dancers are IMR 13 1 5 0 8 "fired up" (positively excited) when they perform better than their fellow dancers in a performance. 3. Trying hard is EI 10 0 8 1 0 rewarded in rehearsals and performances. 4. The teachers IMR 8 1 5 0 3 praise dancers only when they outperform other dancers. 5. If you want to be PM 8 0 7 3 4 cast for the best roles you must be one of the best dancers. 6. The teachers yell PM 8 0 3 2 4 at dancers for messing up. 7. Only the top UR 8 0 5 0 3 dancers "get noticed" by the teachers. 8. Each dancer has IR 8 0 6 1 1 an important role. 9. Dancers at all IR 7 0 7 1 1 skill levels have an important role in performances. 10. The teachers IR 7 1 4 2 1 believe that all of us are crucial to the success of a performance. 11. Each dancer IR 7 0 5 1 0 contributes in some important way. 12. Dancers are IMR 7 0 5 0 0 encouraged to outperform the other dancers. 13. The teachers EI 7 0 5 0 1 want us to try new skills/movements/ expressions. 14. Each dancer IR 6 0 5 1 1 feels as if they are an important team member. 15. Dancers are PM 6 0 4 0 1 punished when they make a mistake. 16. Dancers are PM 6 0 5 0 0 afraid to make mistakes. 17. The dancers CL 6 0 4 1 0 really "work together" as a team when it comes to performances. 18. Dancers feel EI 5 0 5 0 1 good when they try their best. 19. The teachers UR 5 0 4 1 1 think that only the lead dancers contribute to the success of a performance. 20. The teachers get PM 5 0 3 0 1 mad when a dancer makes a mistake. 21. Only the best UR 5 0 2 0 2 dancers get praise. 22. The teachers UR 4 0 1 2 1 make it clear who they think are the best dancers. 23. The focus is to EI 4 1 2 0 1 improve each class/ rehearsal/ performance. 24. The dancers help CL 4 0 2 1 1 each other to get better and excel. 25. The teachers UR 4 0 1 0 1 have their own favorites. 26. The teachers UR 4 0 2 0 2 favor some dancers more than others. 27. The teachers EI 4 0 3 0 0 emphasize always trying your best. 28. The teachers EI 3 0 1 0 2 make sure dancers improve on skills or movements they're not good at. 29. Dancers are EI 3 0 2 0 0 encouraged to work on their weaknesses. 30. The teachers UR 3 0 1 0 2 give most of their attention to the "stars." 31. Dancers help CL 2 0 0 0 0 each other learn. 32. The teachers CL 1 0 0 1 1 encourage dancers to help each other. 33. Dancers feel EI 1 0 0 0 1 successful when they improve. TOTAL 4 119 22 47 Number of Problems by Component Phase of Response (coding numbers correspond to those in Table 1) Item 5 6 Total 1. Dancers are not 2 2 17 selected for the best roles if they make mistakes. 2. Dancers are 0 2 16 "fired up" (positively excited) when they perform better than their fellow dancers in a performance. 3. Trying hard is 1 0 10 rewarded in rehearsals and performances. 4. The teachers 1 1 11 praise dancers only when they outperform other dancers. 5. If you want to be 0 0 14 cast for the best roles you must be one of the best dancers. 6. The teachers yell 2 0 11 at dancers for messing up. 7. Only the top 2 0 10 dancers "get noticed" by the teachers. 8. Each dancer has 1 1 10 an important role. 9. Dancers at all 0 2 11 skill levels have an important role in performances. 10. The teachers 1 1 10 believe that all of us are crucial to the success of a performance. 11. Each dancer 1 1 8 contributes in some important way. 12. Dancers are 2 1 8 encouraged to outperform the other dancers. 13. The teachers 1 1 8 want us to try new skills/movements/ expressions. 14. Each dancer 0 0 7 feels as if they are an important team member. 15. Dancers are 1 0 6 punished when they make a mistake. 16. Dancers are 1 0 6 afraid to make mistakes. 17. The dancers 1 0 6 really "work together" as a team when it comes to performances. 18. Dancers feel 0 0 6 good when they try their best. 19. The teachers 0 0 6 think that only the lead dancers contribute to the success of a performance. 20. The teachers get 2 0 6 mad when a dancer makes a mistake. 21. Only the best 2 1 7 dancers get praise. 22. The teachers 1 1 6 make it clear who they think are the best dancers. 23. The focus is to 1 0 5 improve each class/ rehearsal/ performance. 24. The dancers help 1 0 5 each other to get better and excel. 25. The teachers 2 0 4 have their own favorites. 26. The teachers 0 0 4 favor some dancers more than others. 27. The teachers 1 0 4 emphasize always trying your best. 28. The teachers 1 0 4 make sure dancers improve on skills or movements they're not good at. 29. Dancers are 1 0 3 encouraged to work on their weaknesses. 30. The teachers 0 0 3 give most of their attention to the "stars." 31. Dancers help 1 1 2 each other learn. 32. The teachers 1 0 3 encourage dancers to help each other. 33. Dancers feel 1 0 2 successful when they improve. TOTAL 32 15 239 CL = Cooperative Learning, IR = Important Role, EI = Effort- Improvement, PM = Punishment for Mistakes, UR = Unequal Recognition, IMR = Intra-Team Member Rivalry. Coding numbers correspond to those in Table 1 (0 = No problems, 1 = Understanding, 2 = Interpretation, 3 = Retrieval, 4 = Judgment, 5 = Responding, 6 = Insufficient Thinking Aloud).
|Printer friendly Cite/link Email Feedback|
|Author:||Hancox, Jennie E.; Quested, Eleanor; Duda, Joan L.|
|Publication:||Journal of Dance Medicine & Science|
|Date:||Oct 1, 2015|
|Previous Article:||Epidemiological review of injury in pre-professional ballet dancers.|
|Next Article:||Trunk muscle activation and estimating spinal compressive force in rope and harness vertical dance.|